DataLLM LabLLM Evaluation: What Actually Makes an Eval Trustworthy (2026)
Ten of the thirteen models in our core sweep scored an identical 9/9. Their measured cost ranged from $0.10 to $8.83 per 1,000 tasks — an 88x spread across models that were, on this workload, indistinguishable on correctness. A pass-rate leaderboard would have told you almost nothing. That result is not a fact about those models so much as a fact about evals: what you decide to measure decides what your eval can tell you. This page is not a survey of evaluation frameworks. It is the seven design decisions inside a harness we actually built and ran, what each one buys, and what each one costs — including the two that can score a good model wrong.
Most LLM evaluation advice ends in one of two places. Either pick a benchmark, which hands the hard problem to whoever built the benchmark, or use an LLM as a judge, which hands it to a second model with its own failure modes. Both moves push the difficulty somewhere you can no longer see it. Neither answers the question that decides whether anyone should believe your numbers.
The one property that makes an eval trustworthy
An eval is trustworthy when the scoring cannot be argued with.
That is the whole test. Not sample size, not how many models are on the chart, not whether the tasks are hard. If a reasonable person can look at a scored item and say I would have marked that differently, the number is an opinion with a decimal point on it.
Ours cannot be argued with, and it is worth being precise about why. The model receives a function signature and a prose specification. It returns code. A subprocess runs that code against assertions the model never saw. The Python interpreter says pass or fail. There is no rubric, no grader prompt, no partial credit, and no place for anyone's judgement to enter. If you disagree with a score, you are disagreeing with python3.
This property is expensive. Buying it restricts you to domains where correctness is machine-checkable — code that runs, SQL that returns the right rows, JSON that validates against a schema, arithmetic that has one answer. It buys you nothing for summarisation quality, tone, or helpfulness. Most published eval advice skips this trade because it is a genuine loss, not a free win. We took the loss deliberately: a narrow eval you can defend beats a broad one you cannot.
Seven design decisions and what each one costs
Every eval framework page lists features. Almost none of them explain what a feature costs you, because the cost is where the argument is. Here are the seven decisions inside our harness, each with the thing it buys and the thing it gives up. The right-hand column is the one to read.
| Decision | What it buys | What it costs |
|---|---|---|
| Hidden tests — the model gets the signature and a prose spec, never the assertions | The model cannot write to the test. It has to satisfy a description, which is what a developer does | You must write specs precise enough to be satisfiable without seeing the tests. Ambiguous prose becomes a false failure |
| Execution over LLM-judging — code runs against asserts, interpreter decides | No grader drift, no rubric argument, reproducible by anyone with the same tasks | Restricts you to verifiable-answer domains. Writing quality, tone and judgement are simply out of scope |
| Temperature 0 | Cuts run-to-run variance so a score difference is about the model, not the dice | Not how production traffic runs. Most real deployments sample above 0, so this is a lower bound on real-world variance |
| One scored attempt; retry up to 3 times only on an API error | A miss is a miss. No second chance at a wrong answer, so the score reflects first-attempt reliability | Says nothing about whether the model self-corrects when told it failed — which is how coding agents actually operate |
| Fixed 4,000-token ceiling for every model | Simple, identical, no per-model tuning that could be accused of favouritism | A verbose or heavy-reasoning model can be truncated mid-answer and scored a miss. This is a real limitation, not a neutral choice |
12-second timeout in an isolated python3 -I subprocess | A hung or looping generation cannot stall the run, and generated code cannot touch the host environment | A timeout is a scoring rule wearing an infrastructure disguise. A correct but slow solution scores identically to a wrong one, and the threshold is a number we picked |
| Cost as reported tokens × list price | Auditable line by line. Anyone can recompute it from the token counts and a public price sheet | Not a vendor invoice. Ignores caching, batch discounts, negotiated rates and any provider-side rounding |
Two of those rows deserve more than a table cell, because they are the ones most likely to make our numbers wrong in a specific direction.
The two decisions that can score a good model wrong
The 4,000-token ceiling. Every model gets the same max_tokens. That is the fair version and it is also the version that can punish a model for thinking out loud. Reasoning tokens count against the budget. On our run, Gemini 3.6 Flash spent 2,615 reasoning tokens on parse_csv_line alone — a task it still passed, but one where roughly two thirds of the ceiling went to reasoning before a line of the answer was emitted. A model that reasons harder on that task, or is simply more verbose, gets cut off and scored a miss. The miss looks identical to a wrong answer in the results table. It is not the same thing.
We keep the ceiling anyway, because the alternative — tuning max_tokens per model — introduces a knob that someone can accuse us of turning to favour a result. A fixed ceiling is defensible and imperfect. Per-model ceilings would be more accurate and less credible. If you build your own harness, pick one and say which.
The 12-second subprocess timeout. This one surprises people. There is nothing adversarial in our nine tasks — two_sum, valid_parentheses, merge_intervals, roman_to_int, lcs_len, flatten, top_k_words, token_bucket, parse_csv_line. They are the kind of thing a working developer writes before lunch. And model-written code still hangs: an unterminated while, a recursive flatten with no base case for a self-referencing structure, a generator consumed twice. If your eval has no timeout, one bad generation stops your entire run and you find out at 3am.
The cost is that a timeout is a scoring decision disguised as an infrastructure detail. Twelve seconds is generous for these nine tasks and would be far too tight for anything with real I/O. Whatever you pick, publish it — an eval that does not state its timeout has an undisclosed correctness threshold.
-I — isolated mode, no user site directory, no environment-derived paths. You are executing text a language model produced. Treat it exactly like you would treat a code sample from a stranger, because that is what it is. The same instinct applies to agents that run tools: see AI agent traps.Cost: measured, not invoiced
Our cost column is computed. We take the token counts the API reports for each call and multiply by that model's published list price. That is a measured cost, and it is deliberately not a vendor total.
The reason is auditability. A vendor dashboard total is a single number you cannot decompose: it silently folds in caching, promotional credits, free-tier allowances and whatever rounding the provider applies. You cannot check it and neither can we. Reported tokens times list price is arithmetic anyone can redo from two public inputs.
The honest cost of that choice: our figure will be higher than a real bill for anyone using prompt caching or batch pricing, and it will drift the moment a provider changes list prices. It is a comparable number, not a forecast of your invoice. If you want the forecast, plug your own volumes into the cost calculator, and the estimating method is laid out in how to estimate LLM API costs.
One more disclosure that belongs in any eval writeup: all calls go through OpenRouter's OpenAI-compatible endpoint, deliberately not through the DataLLM Lab gateway. We sell a gateway. Benchmarking our own models through our own infrastructure would make every number here a marketing asset instead of a measurement. Routing through a third party means you do not have to trust us about latency, and you do not need to be our customer to reproduce the run.
A live example we did not test: VibeThinker-3B
We have no first-party number for this model. It is not available on OpenRouter, so we could not run it through our harness. Everything in this section is vendor-reported or third-party reporting, and it is here as an illustration of why eval design matters — not as a result of ours.
Weibo released VibeThinker-3B in June 2026, an MIT-licensed 3-billion-parameter model, with inference and evaluation code on GitHub. Weibo's own technical report puts it at 94.3 on AIME 2026 — level with DeepSeek V3.2 at 671B parameters — and at 123 of 128 first-attempt submissions on unseen LeetCode contests from late April to late May 2026. VentureBeat, in June 2026, put that 94.3 ahead of Gemini 3 Pro's reported 91.7 — and noted that every one of these figures is self-reported, measured on the authors' own harness with their own decoding settings, with no independent lab having reproduced them at the time of writing.
That last clause is why the release set the AI world arguing about benchmarks again, and the argument maps directly onto the seven decisions above. A 3B model matching a 671B model on a competition-maths benchmark invites exactly one question: did the model see the assertions, or something close enough to them, during training? That question is unanswerable from the outside for any public benchmark. It is answerable for a private harness with hidden tests, which is the entire reason we wrote our own tasks instead of importing a public set.
The second question is attempt count. First-attempt pass rate and any-attempt pass rate are different metrics that look identical in a headline. Weibo's figure is stated as first-attempt, which is the stronger claim and the right one to state. Ours is also single-attempt, with retries only on transport errors — see the checklist below for why that line item is worth hunting for in every eval you read.
What our harness actually found
Thirteen models, one methodology, one sitting. Ten of the thirteen scored a perfect 9/9. Three missed exactly one task each. Here is the full field, and the useful column is not the score.
| Model | Score | Measured cost / 1k tasks | Latency | Reasoning tokens | Missed |
|---|---|---|---|---|---|
| Qwen3 Coder Next | 9/9 | $0.10 | 7.0 s | 0 | — |
| DeepSeek V4-Flash | 9/9 | $0.13 | 14.5 s | 568 | — |
| DeepSeek V4-Pro | 8/9 | $0.74 | 18.2 s | 732 | parse_csv_line |
| Mistral Medium 3.5 | 9/9 | $0.87 | 2.9 s | 0 | — |
| MiniMax M3 | 9/9 | $0.90 | 13.4 s | 623 | — |
| Nemotron 3 Ultra | 9/9 | $1.07 | 8.1 s | 373 | — |
| Kimi K2.7 Code | 9/9 | $1.34 | 10.4 s | 272 | — |
| Claude Sonnet 5 | 9/9 | $1.67 | 7.2 s | 0 | — |
| Grok 4.3 | 8/9 | $1.75 | 8.4 s | 482 | flatten |
| GLM 5.2 | 9/9 | $1.99 | 12.3 s | 559 | — |
| StepFun Step 3.7 Flash | 8/9 | $2.66 | 19.2 s | 450 | valid_parentheses |
| Claude Opus 4.8 | 9/9 | $4.05 | 6.1 s | 0 | — |
| GPT-5.5 | 9/9 | $8.83 | 10.5 s | 176 | — |
Read the score column and you learn that most current models can write nine short Python functions. That is a true statement and a useless one. Read the cost column and the decision appears.
Gemini 3.6 Flash was run later on the same harness and also scored 9/9, at $8.02 per 1,000 tasks, 6.5 s average, and an average of 933 reasoning tokens per call — more reasoning than any model in the 13-model core sweep, where the top was 732. Same correctness as the $0.10 model on this set. It is a separate run, not part of the thirteen, and we count it separately for that reason. Per-model detail is in the Gemini 3.6 Flash review.
The four models that emitted zero reasoning tokens — Qwen3 Coder Next, Mistral Medium 3.5, Claude Sonnet 5, Claude Opus 4.8 — all scored 9/9. On this workload, reasoning tokens bought billable output rather than correctness. That is a claim about nine short Python tasks and nothing wider.
Here is the design lesson, and it is the reason the seven decisions above are worth arguing about. Our eval separates models on cost and latency because it deliberately refused to separate them on judged quality. Had we scored with an LLM judge on a rubric, we would have produced a spread of scores that looked more informative and meant less. The flat 9/9 column is the eval telling the truth: on this workload, these models are the same, and the decision lives elsewhere.
What this eval does not measure
Stating this is not a disclaimer. It is the part that makes the rest credible.
Single-turn only. One prompt, one scored attempt. The harness does not run agents, does not use tools, does not do multi-turn correction. Anything you read here about loops or agents is arithmetic on a per-call number, not a measurement of a loop.
Out of scope entirely: long-context reasoning, multi-file refactoring, agentic and multi-turn tool use, any language other than Python, and vision. Nine short functions do not test architecture, and we do not claim they do. Context-length behaviour degrades in ways short tasks cannot see — that is covered separately in context rot.
The ceiling can produce a false miss. Restating it because it belongs in this list: 4,000 tokens is fixed, reasoning tokens count against it, and a truncated verbose model scores as a miss indistinguishable from a wrong answer.
Not tested, and never presented as ours: any Gemini other than 3.6 Flash, GPT-5-mini and GPT-5-nano, Ollama or any locally-run model, Grok 3, Grok 4, Grok 4.5, Grok 4.20, grok-build-0.1, Claude Haiku 4.5, Claude Opus 4.7, Claude Sonnet 4.6, and VibeThinker-3B. If a number for any of those appears on this site attributed to our harness, it is an error.
A checklist for auditing any eval you are shown
Six questions. Run them against a vendor benchmark, a leaderboard, an academic paper, or this page. Anything that cannot answer all six is an impression, however many models are on the chart.
| Question | Why it decides the number | Our answer |
|---|---|---|
| Who wrote the tests? | A vendor scoring itself on tasks it chose is a product page. Third-party or private tasks are a different claim | We did, privately, and they are not published as a public set |
| Did the model see them? | Any public benchmark may be in training data. Contamination inflates scores and is invisible from the outside | No — the model gets a signature and prose spec, never the assertions |
| Who or what grades? | A human rubric drifts between graders; an LLM judge drifts between versions and cannot be audited | The Python interpreter. All asserts pass or the task fails. No partial credit |
| How many attempts? | Best-of-5 and first-attempt are different metrics that look the same in a headline | One scored attempt. Retries up to 3 times only on an API error, never on a wrong answer |
| What is the token ceiling? | An undisclosed or per-model ceiling silently rewards or truncates particular models | 4,000 tokens, fixed for every model, temperature 0 |
| Is cost measured or estimated? | A vendor total cannot be decomposed; a missing cost column means the eval never asked the question that decides procurement | Computed: reported tokens × list price. Auditable, and not a vendor invoice |
The sixth question is the one most evals fail, and it is the one our own run shows matters most. An 88x cost spread sat underneath a completely flat score column. If your eval has no cost column, it cannot see the only variable that separated ten of our thirteen models. Cheaper candidates specifically are covered in best cheap LLM for coding, the full run writeup is the LLM coding cost benchmark, and instrumenting this in production rather than in a harness is LLM observability.
One last note on scope. Everything above is about offline evaluation — a fixed task set, run deliberately. It is not a substitute for measuring your own traffic, and the two answer different questions. An eval tells you which candidates are worth deploying. Production telemetry tells you which one you actually deployed well. If you are choosing a model to put inside a loop, best LLM for AI agents covers the dimensions our single-turn harness cannot reach.
Run your own eval against 300+ models
One OpenAI-compatible endpoint, one key. Swap the model id, rerun your task set, compare measured cost — the only eval that predicts your bill is yours.
FAQ
Is an LLM judge ever the right choice?
Yes, when the output has no machine-checkable correct answer — summarisation, tone, helpfulness. The rule is to be honest about what you have bought: an LLM judge produces a score you cannot falsify, because the grader is itself a model with a version, a temperature and its own biases. If you use one, pin the judge version, publish the grader prompt, and hand-score a sample to measure judge-human agreement. And never present judge scores next to executed scores as if they were the same kind of number.
Why hide the tests instead of publishing them?
Because a published test set is a training target. Once assertions are public, any model trained after that date may have seen them, and you can no longer tell a model that solved the problem from one that memorised the answer. Hidden tests also stop the subtler version: a model that writes code shaped to satisfy the specific asserts rather than the specification. The cost is reproducibility — you cannot rerun our exact tasks — which is why we publish the method, the settings and every raw number instead.
Nine tasks is a small eval. Why not more?
It is small, and we say so. Nine executed tasks across thirteen models is enough to separate them on cost and latency, and not enough to rank the ten that all scored 9/9. The value here is not statistical power — it is that every scored attempt was decided by running code rather than by an opinion. A larger set graded by a rubric would look more authoritative and prove less. Scale the task count when you build your own; keep the scoring rule.
Did you test VibeThinker-3B?
No. It is not available on OpenRouter, so we have no first-party number for it and will not imply one. The 94.3 on AIME 2026 and the 123 of 128 first-attempt LeetCode figures are self-reported in Weibo's June 2026 release and the reporting around it, not measured by us. If it becomes callable through an endpoint we use, it goes through the same nine tasks as everything else and we publish whatever the interpreter says.
Why temperature 0 if production does not run that way?
To make the score about the model rather than the sampling. At temperature 0 a rerun gives you close to the same output, so a difference between two models is more likely to be real. The cost is realism: most production traffic samples above 0, so our variance figures are a lower bound and our scores are a best case for determinism. If your deployment runs hot, run your own eval at your own temperature — the harness setting is a choice, not a law.
What single change would improve most evals I read?
Add a cost column. Ten of our thirteen models scored an identical 9/9 while their measured cost ranged from $0.10 to $8.83 per 1,000 tasks. Without cost, that eval reports a tie and the reader learns nothing. With cost, it reports an 88x decision. Latency is the close second, for the same reason.