DataLLM LabHow We Test LLMs
Most LLM articles repeat vendor benchmark numbers. We don't - when we say a model scored a certain way, we ran it ourselves. This page documents exactly how our coding benchmark works, so you can judge our numbers, reproduce them, and know precisely what they do and don't measure. Transparency about method is part of trusting the result.
Why we run our own tests
Vendor benchmark numbers are marketing. They're not worthless, but they're chosen and framed to flatter the model, and they rarely tell you the thing you actually need: what it costs to get a real task right. So across our model reviews we run a first-party benchmark — the same tasks, executed and scored the same way, with the real bill recorded — and we publish the method here so you can judge the numbers for yourself.
How the benchmark works
The harness is deliberately simple and reproducible:
- Identical prompts. Every model gets the exact same task prompt, asking for a single solution in one code block.
- Call via OpenRouter. We call each model through OpenRouter's OpenAI-compatible endpoint, which returns the real billed cost per call.
- Execute the code. We extract the code from the response and run it against hidden test cases — including edge cases the prompt doesn't spell out.
- Score pass/fail. A task passes only if every hidden test passes. An execution timeout guards against solutions that loop forever (some bug-fix tasks are designed to expose this).
- Record the numbers. For each call we log correctness, real cost (token usage × list price), wall-clock latency, and reasoning-token count.
The tasks
The tasks are chosen to separate models — each has a common failure mode that trips weaker or careless answers:
| Task | What it tests |
|---|---|
| Two-sum | Hash-map pairing; must handle duplicates and negative numbers |
| Valid parentheses | Stack matching across three bracket types, including the empty string |
| Merge intervals | Sort then merge overlaps; must handle touching and out-of-order inputs |
| Roman to integer | Subtractive notation (IV, IX, XL, CM) |
| Longest common subsequence | Classic DP — a subsequence, not a substring |
| Flatten a nested dict | Join key paths with dots; non-dict values are leaves |
| Top-k words | Frequency count with an alphabetical tie-break, case-sensitive |
| Token-bucket limiter | Stateful timing: refill over time, capped at capacity |
| Parse a CSV line | Quoted fields with commas, and two double-quotes escaping one literal quote |
They're standard problems on purpose: deterministic to grade, and revealing of the small correctness lapses (edge cases, quote-escaping, tie-breaking, stateful timing) that separate a model that "looks right" from one that is right.
What we measure
- Pass rate — how many tasks the model got fully correct, by execution.
- Real cost — token usage × the model list price, extrapolated to cost per 1,000 tasks. This exposes the gap between sticker price and real spend: per-task cost = price × tokens generated, so a cheap-per-token model that writes verbosely can cost more per task than a pricier, terser one.
- Latency — wall-clock seconds per task.
- Reasoning tokens — how much the model "thinks" before answering, which drives both latency and cost.
Our principles
First-party
- Pass rates, costs and latencies are measured by us — not copied from a vendor deck.
Labeled sources
- Vendor figures are marked "vendor-reported"; third-party numbers (e.g. Artificial Analysis) are attributed.
No fabrication
- Every number is sourced or measured. If we can't verify it, we say so or leave it out.
Honest even against us
- We report when a cheap model wins — accurate routing matters more than any one sale.
What it doesn't measure
Being clear about scope is the point. This benchmark measures one-shot correctness on standard coding tasks plus real cost and latency. It does not measure agentic or multi-file work, repo-scale engineering, tool use, long-horizon planning, or non-coding ability. The tasks are well-known algorithms, so they may appear in training data. We treat the results as a cost-efficiency reality check, and we lean on independent benchmarks and your own task suite for the rest. Results vary run-to-run because models are non-deterministic — so the ratios between models, not any single figure, are the signal.
Test these models yourself — one key
Every model we benchmark is callable through DataLLM Lab's OpenAI-compatible endpoint, so you can run them on your own tasks and route to whichever wins.
FAQ
How does DataLLM Lab test LLMs?
Identical prompts via OpenRouter, then we execute each solution against hidden tests for pass/fail, recording real billed cost, latency, and reasoning tokens. An execution timeout guards infinite-loop tasks.
Are your numbers independent?
Yes — pass rates, costs, and latencies are first-party. Vendor figures are labeled "vendor-reported"; third-party numbers (e.g. Artificial Analysis) are attributed.
What does it measure — and not?
One-shot coding correctness + real cost/latency. Not agentic, repo-scale, tool-use, or non-coding ability. Standard tasks may be in training data — a cost-efficiency check, not a full ranking.
How is cost per 1,000 tasks calculated?
Sum real billed cost over the test ÷ number of tasks × 1,000. It captures that per-task cost = price × tokens generated — verbose models cost more than their sticker price implies.
Can I reproduce it?
Yes — fixed tasks, deterministic grading, identical prompts. Models are non-deterministic, so we treat between-model ratios as the durable signal.
Why OpenRouter?
One OpenAI-compatible endpoint across many models, with a real billed cost per call — the same gateway routing model DataLLM Lab uses, so the prices reflect what a real user pays.
Do you take payment for rankings?
No — rankings come only from measured results, including when a cheap model beats an expensive one.
One API, every model.
Get a single API key for Claude Opus 4.7, GPT-5.4, and 300+ more — with automatic price comparison and routing to the best model for every request.