DataLLM LabKevin Fan
I build DataLLM Lab - a gateway that routes to 300+ language models through one OpenAI- and Anthropic-compatible endpoint - and I write everything published on this blog. This page exists so you can judge whether to trust the numbers here: what I actually test, how I test it, what tooling I use, and how to tell me I got something wrong.
What I do
I run DataLLM Lab, an LLM gateway. Customers point their existing OpenAI or Anthropic SDK at one endpoint and get access to 300+ models without maintaining a separate integration, key, and billing relationship for each provider. That is the product, and it is also why this blog exists: to route traffic well, I have to know what the models actually cost and what they can actually do, and almost nobody publishes that in a form you can check.
The writing output to date is 198 articles on this blog and 341 model reference pages covering individual models' pricing, context windows, and capabilities. The blog is not a content-marketing arm bolted onto the product - it is the research I needed for the product, published.
The short version of my bias: I make money when you route traffic through the gateway. I do not make money from which model you choose, and the gateway bills at provider list price. So I have a commercial interest in you using an aggregator, and no commercial interest in steering you toward any particular model. Where an article recommends a model, that recommendation is the test result. Judge accordingly - and see the methodology page for how the tests are constructed.
How I test models
When an article on this site says a model was tested, it means the model was called and the output was executed. Not vendor-reported scores, not leaderboard scraping, not vibes.
The coding harness works like this: nine Python function-writing tasks, each with a specification and a set of assert statements the model never sees. The model gets the spec, returns code, and the code is run against the asserts. It either passes all of them or it does not. There is no partial credit and no human grading the output, so there is nothing for me to fudge.
The nine tasks:
| Task | Signature | What it exercises |
|---|---|---|
two_sum | two_sum(nums, target) | Baseline - hash-map lookup, negative numbers |
valid_parentheses | valid_parentheses(s) | Stack logic, empty-string edge case |
merge_intervals | merge_intervals(intervals) | Sorting plus merge, empty input |
roman_to_int | roman_to_int(s) | Subtractive notation (IV, IX, MCMXCIV) |
lcs_len | lcs_len(a, b) | Dynamic programming |
flatten | flatten(d) | Recursion over nested structures |
top_k_words | top_k_words(text, k) | Counting plus tie-breaking order |
token_bucket | token_bucket(times, capacity, refill_per_sec) | Stateful simulation - refill over time, capped at capacity |
parse_csv_line | parse_csv_line(line) | Quoted fields, escaped quotes - fiddly spec-following |
Every model runs at temperature: 0 with a 4,000-token ceiling. The calls go through OpenRouter's OpenAI-compatible endpoint - deliberately not through my own gateway, so the numbers do not depend on my infrastructure and you can reproduce them without being my customer. Cost is computed from the real token usage recorded for each call multiplied by the model's list price, normalized to dollars per 1,000 tasks - measured tokens, not a guessed token count. Latency is wall-clock, averaged across the nine calls. Each task gets exactly one scored attempt: the harness retries only when the API itself errors out, never when the model returns a wrong answer, so nothing gets a second chance to pass. A response that comes back empty - a refusal or a content filter - counts as an API failure and is dropped from the denominator rather than scored as a wrong answer, because a bug that did score them as wrong answers once cost a model five of its nine tasks.
The limits of this harness, stated plainly. Nine short Python functions is a narrow test. It measures whether a model can follow a precise spec and produce runnable code on the first try. It does not measure long-context reasoning, multi-file refactoring, agentic tool use, or anything non-Python. A model that scores 9/9 here is not thereby "the best coding model" - it is a model that reliably nails small, well-specified functions cheaply. I try to say so in every article that cites these numbers. The 4,000-token ceiling is itself a limit: a model that needs more room than that to reason gets cut off and scored as a miss, which is a constraint of the harness rather than a verdict on the model.
What the tests found
Thirteen models have been through the current harness on identical tasks. The headline result is not which model won - it is how little the price tells you:
| Measure | Result |
|---|---|
| Models tested on the current harness | 13 |
| Models scoring a perfect 9/9 | 10 |
| Cheapest perfect score | Qwen3-Coder-Next - $0.10 per 1,000 tasks |
| Most expensive perfect score | GPT-5.5 - $8.83 per 1,000 tasks |
| Cost spread between them | 88x, for identical output |
| Fastest model | Mistral Medium 3.5 - 2.9 s average, also 9/9 |
Ten of thirteen models produced identical passing output across an 88x price range. That single finding is the thesis of most of what I write: for a large share of real workloads, model choice is a cost decision that has been dressed up as a capability decision. The full run, per-model, is in the coding benchmark article, and the raw scores feed the individual model reviews.
Tooling and AI disclosure
Google's own guidance asks publishers whether their use of automation is self-evident to readers. Here is mine, without euphemism.
I use AI tooling to research and draft. Articles on this site are researched and drafted with LLM assistance - which is not a confession, it is the obvious thing for a person who runs an LLM gateway to do. What matters is what that tooling is and is not permitted to do:
- It never invents a number. Every first-party figure published here - score, cost, latency - must exist verbatim in the benchmark data files before it can appear in an article. Numbers that cannot be traced to a recorded run get cut, not rounded into place.
- Every claim about a third party is checked against a primary source. Model pricing, context windows, and release dates come from the provider's own documentation, and get re-checked when a provider changes them.
- Drafts get an adversarial review pass whose job is to find the wrong claim, not to approve the article. Corrections are only applied when the replacement fact is independently verified - a confidently wrong "fix" is worse than the original error.
- No article gets an AI byline. Everything here is published under my name because I am accountable for it. If a number on this site is wrong, that is my error, not a tool's.
The benchmarks themselves are not AI-generated in any sense - they are executed programs. A model writes code, a Python interpreter runs it against fixed assertions, and the pass/fail comes back from the interpreter.
What I am not
I am a practitioner who runs inference infrastructure and publishes what the meter says. I am not an academic ML researcher, I do not train models, and I have no inside access to any provider's roadmap. When I write about why a model behaves a certain way, that is inference from observed behavior, and I try to mark it as such rather than dress it up as architectural knowledge I do not have.
I also do not accept payment, sponsorship, or review units from model providers for coverage. No article on this site has been paid for by the vendor whose model it discusses.
Corrections and contact
Prices change, models get deprecated, and I make mistakes. If you find a number on this site that is wrong or stale, tell me and I will fix it and note what changed. The fastest route is GitHub (@kevinwowo), or the contact details on the about page.
Substantive corrections update the article's modified date. Mechanical edits - fixing a link, adding a related-article card - deliberately do not, so that an "updated" date on this site always means the content actually changed.
FAQ
Who actually writes the articles on DataLLM Lab?
I do - Kevin Fan, the founder of the gateway. Articles are researched and drafted with AI assistance and published under my name, which means I am accountable for every claim in them. No article is bylined to an AI persona or a fictitious author.
What does "we tested this model" mean on this site?
It means the model was called and its output was executed. For coding claims, that is nine Python tasks scored by running the code against assertions the model never sees, at temperature 0 with a 4,000-token ceiling, called through OpenRouter rather than through my own gateway. It never means a vendor-reported benchmark score was copied over.
Are the cost figures estimates or real?
Measured, not modelled. Cost per 1,000 tasks is the actual token usage recorded for those specific runs multiplied by the model's list price - not a guess at how many tokens a task takes. That is why a model with a low sticker price but verbose, reasoning-heavy output can cost more per task than a pricier, terser one.
Do model providers pay for coverage here?
No. There is no sponsorship, affiliate arrangement, or paid placement with any model provider. The commercial interest is in you routing traffic through the gateway, which bills at provider list price and is indifferent to which model you pick.
Why should I trust a benchmark run by someone selling API access?
Start with the fact that the benchmark does not run on my product. Calls go through OpenRouter, not the DataLLM Lab gateway, so nothing about the numbers depends on my infrastructure and you can reproduce them without being my customer. Then note the incentive does not point the way you might expect - the gateway earns the same whether you use the $0.10 model or the $8.83 one, and the headline finding is that the cheap one often matches the expensive one. Beyond that, trust it only as far as it is checkable: the tasks, the settings, and the scoring rule are documented on the methodology page, so you can run it yourself and tell me if you get a different answer.
How current are the numbers?
Model scores are re-run when a provider ships a new version, and prices are re-checked against provider documentation. Each article shows its publication date, and an "updated" date only when the substance changed. If you find something stale, the corrections route is above.