AI Coding Ranking 2026: Four Rankings of the Same Models (They Disagree)

We ran 13 models through the same executed coding benchmark in one sitting and then ranked them four different ways. Ranked by pass rate, 10 of the 13 tie at 9/9 — the ranking is almost flat and tells you almost nothing. Ranked by measured cost to finish the same nine tasks, the same field spans $0.10 to $8.83 per 1,000 tasks, an 88x gap. Ranked by speed it reorders again: 2.9 s to 19.2 s, and the fastest model is neither the cheapest nor the priciest. Three rankings, one set of results, and they do not agree on a winner. A leaderboard position is not a purchasing decision.

Most coding rankings you can find are a single ordered list, and the number they sort on came from somewhere else. This one is four ordered lists over one set of results we executed ourselves, and the interesting part is where they contradict each other.

Read this before the rankings

This ranks nine short, self-contained Python functions. That is the whole scope. Each task is a signature plus a prose spec; the model returns code; the code is executed against assertions it never sees.

It says nothing about long-context work, multi-file refactoring, agentic or multi-turn tool use, or any language other than Python. It is single-turn. A model that ranks low here may be exactly right for work this harness never touches, and a model that ranks first here may fall apart on a 200,000-token codebase — we would not know, because we did not test it.

We are putting that at the top rather than in a footnote because it is the only reason these numbers are worth anything. A ranking whose limits are invisible is a ranking you cannot check. Full boundaries are in what we did not measure and on the methodology page.

Four rankings, one field, side by side

Rows are sorted by measured cost, cheapest first. Read across a row and watch the rank numbers disagree.

Three things fall out of that table immediately. The pass-rate column is a wall of ties. The cost column is a clean 1-to-13 ordering that spans two orders of magnitude. And the speed column agrees with neither: the fastest model is 4th on cost, and the 2nd-fastest is 12th on cost.

Gemini 3.6 Flash sits on its own row because it was run later on the same harness, not in the 13-model sweep. It is not ranked inside the 13 for that reason. More on it below.

Ranking 1: by pass rate, and why it is nearly useless

10 of the 13 models scored 9/9. Ten-way tie for first. The remaining three scored 8/9 and each missed exactly one task: DeepSeek V4-Pro missed parse_csv_line, Grok 4.3 missed flatten, Step 3.7 Flash missed valid_parentheses. No model scored below 8/9.

That is a ranking with two rungs. If you sorted a purchase decision by it, you would be picking essentially at random among ten models whose real-world cost to you differs by up to 88x.

This is not a flaw in the scoring — the scoring is strict. All assertions pass or the task fails, no partial credit, no judge model, one scored attempt each. It is a fact about the workload: nine short Python functions are now inside the competent range of nearly every serious coding model shipping in 2026. The differentiation moved somewhere else, and the other three rankings are where it went.

One consequence worth stating plainly: a single miss on nine tasks is one data point. We would not tell you DeepSeek V4-Pro cannot parse CSV lines. We would tell you it did not on this attempt, at temperature 0, with a 4,000-token ceiling, and we did not buy it a retry. The V4-Pro against V4-Flash comparison has the fuller picture.

Ranking 2: by measured cost per 1,000 tasks

This is the ranking that separates the field. Same nine tasks, same settings, cost computed from the token counts the API reported multiplied by each model's list price.

In that 13-model sweep the cheapest perfect score cost $0.10 per 1,000 tasks and the priciest cost $8.83 — 88x for the same 9/9. We have since measured DeepSeek V3.2 lower still, at $0.08. Both models did exactly the same amount of verified work on exactly the same nine problems.

The gap is not a rounding error at any real volume. Projected forward from those measured per-task numbers — a projection, not a bill we received — a steady thousand tasks a day of roughly this size, at these prices, works out to about $36 a year against about $3,200 a year for output that scored identically on this set. Put either model inside a loop that makes twenty calls per run and the same projection multiplies again — the arithmetic is in AI agent traps.

Notice also that cost order is not price-tier order. DeepSeek V4-Pro is cheaper per task than Mistral Medium 3.5 despite scoring one task lower, because it is cheap enough per token to absorb the miss. And GLM-5.2 at $1.99 lands above Claude Sonnet 5 at $1.67 even though its list price per token is lower — Sonnet 5 emitted zero reasoning tokens and GLM-5.2 emitted 559. What you pay is tokens burned multiplied by rate, and models differ on both factors at once. If you want to run that arithmetic on your own token mix, the cost calculator does it.

Ranking 3: by speed

Mean wall-clock per task across the nine tasks, same run, same tasks. It reorders the field a third time.

Mistral Medium 3.5 finished in 2.9 s. Step 3.7 Flash took 19.2 s. That is 6.6x between the ends. Mistral Medium 3.5 is more than twice as fast as the next model in line, Claude Opus 4.8 at 6.1 s — a 3.2 s step, and it sits at the very top of the ranking where it is most visible. Only one gap in the field is wider, the 3.7 s between DeepSeek V4-Flash at 14.5 s and DeepSeek V4-Pro at 18.2 s, and that one is down at the slow end.

The full speed order: Mistral Medium 3.5 2.9 s, Claude Opus 4.8 6.1 s, Qwen3 Coder Next 7.0 s, Claude Sonnet 5 7.2 s, Nemotron 3 Ultra 8.1 s, Grok 4.3 8.4 s, Kimi K2.7 Code 10.4 s, GPT-5.5 10.5 s, GLM-5.2 12.3 s, MiniMax M3 13.4 s, DeepSeek V4-Flash 14.5 s, DeepSeek V4-Pro 18.2 s, Step 3.7 Flash 19.2 s.

Two disagreements are worth naming. DeepSeek V4-Flash is 2nd on cost and 11th on speed — it is cheap partly because it takes its time and partly because its tokens are cheap. Claude Opus 4.8 is 2nd on speed and 12th on cost, which is the opposite trade: you are buying latency, not correctness, because on this set correctness was already tied.

Latency also tracks reasoning tokens loosely but not reliably. The four models that emitted zero reasoning tokens — Qwen3 Coder Next, Mistral Medium 3.5, Claude Sonnet 5, Claude Opus 4.8 — occupy the top four positions on speed. But GPT-5.5 emitted only 176 reasoning tokens and still landed 8th at 10.5 s, so thinking time is not the only thing in the wall clock.

A cheap sticker price is not a cheap job

Gemini 3.6 Flash is the sharpest illustration on this page. It was run later on the same harness, on 2026-07-28, and scored 9/9.

Its list price is $1.50 input / $7.50 output per 1M tokens. GPT-5.5 lists at $5 / $30 — four times the output rate. On sticker price alone, Gemini 3.6 Flash looks like a budget option next to it.

The measured job cost was $8.02 per 1,000 tasks against GPT-5.5's $8.83. A quarter of the output rate, and it landed within 10% of the same bill. It is also about 80x the cheapest 9/9 in the field.

The reason is in the token counts. Gemini 3.6 Flash averaged 933 reasoning tokens per call across the nine tasks — more than any model in the 13-model core sweep, where the top was 732 — and reasoning tokens bill at the output rate. GPT-5.5 emitted 176. The expensive model thought briefly; the cheap-per-token model thought at length, and paid for it at the output rate on every single call.

Its per-task spread shows where the tokens went.

The hardest task burned 7.7x the reasoning tokens of the easiest — 2,615 on parse_csv_line against 341 on two_sum. So the cost of a reasoning model is not a flat per-call tax you can budget once. It scales with how awkward your prompts happen to be, and you find out afterwards. A model that emits zero reasoning tokens has no such variance, which is a real property and not just a cheaper number.

None of this makes Gemini 3.6 Flash a bad model. It scored 9/9 in 6.5 s, faster than GPT-5.5, and its million-token context window is a capability this harness does not exercise at all. It makes the sticker price a bad predictor of the bill. Our fuller read is in the Gemini 3.6 Flash review.

Ranking 4: our opinionated pick per tier

The first three rankings are arithmetic on measured numbers. This one is a judgement call, so the reasoning is stated and you can reject it.

Best overall value: Qwen3 Coder Next. 9/9 at $0.10 per 1,000 tasks, 7.0 s, and zero reasoning tokens. It is the cheapest perfect score in the field, it is 3rd fastest, and the zero reasoning count means its cost does not inflate when a task gets awkward. On this workload there is no argument for paying more, and the burden of proof is on the model that wants 88x. See also the cheap coding model roundup.

Best when latency is the product: Mistral Medium 3.5. 2.9 s, 9/9, $0.87. If a human is watching a spinner — an IDE completion, an inline fix, a chat that has to feel instant — a 2.9 s response against 7.0 s is a different product, and $0.77 per 1,000 tasks is a cheap price for that difference. Details in the Mistral Medium 3.5 review.

Best name-brand default: Claude Sonnet 5. 9/9 at $1.67, 7.2 s, zero reasoning tokens. It costs 16.7x the Qwen number and we still list it, for one honest reason: if you need a model for work well beyond nine Python functions — long context, refactors across files, tool use — you will be choosing on evidence this harness does not provide, and Sonnet 5 is the cheapest frontier-tier model in our field that also went 9/9 without burning reasoning tokens. That is a defensible hedge, not a measured win. Sonnet against Opus covers the step up.

Best premium: Claude Opus 4.8. $4.05, 6.1 s, 9/9, zero reasoning tokens. If you are buying at the top of the market anyway, this is the fastest model in the field above $2 and the only expensive model here that did not spend tokens thinking.

Hardest to justify from this data: GPT-5.5 at $8.83 and Gemini 3.6 Flash at $8.02. Both scored 9/9. So did a model costing $0.10. Neither result says they are weak — it says nine short Python functions are far below the level where their price starts earning anything, and you should be choosing them on capabilities we did not test. Long-horizon agent work is one place they may separate.

The three 8/9 models — DeepSeek V4-Pro, Grok 4.3, Step 3.7 Flash — each missed one task out of nine. That is a single data point per model, not a verdict. We report it because we do not hide misses, not because we think one miss ranks a model.

How these numbers were produced

Nine executed Python tasks: two_sum, valid_parentheses, merge_intervals, roman_to_int, lcs_len, flatten, top_k_words, token_bucket, parse_csv_line.

The model receives a function signature and a prose spec. It never sees the tests. The returned code is executed against hidden assertions in an isolated python3 -I subprocess with a 12-second timeout. Every assertion passes or the task fails — no partial credit, no human grader, no LLM judge.

Temperature 0, max_tokens 4000. One scored attempt per task. The harness retries up to three times on an API error and never on a wrong answer, which is why a miss stays a miss.

Cost is computed: the token counts the API reported, multiplied by that model's list price. It is a measured cost, not a vendor invoice. Calls go through OpenRouter's OpenAI-compatible endpoint, deliberately not through the DataLLM Lab gateway, so nothing here depends on our infrastructure and you do not have to be our customer to reproduce it.

The 13-model sweep ran in one sitting under one methodology. Gemini 3.6 Flash ran later on the same harness, on 2026-07-28, which is why it is reported separately rather than folded into the 13.

What we did not measure

Not measured at all: long-context reasoning, multi-file refactoring, agentic or multi-turn tool use, any non-Python language, and vision. The harness is single-turn. It does not run agents and does not use tools.

A known scoring artefact: the 4,000-token ceiling can truncate a very verbose model mid-answer, and a truncated answer scores as a miss. That penalises verbosity, which is a real cost in production but is not the same thing as being wrong.

Cost caveat: list prices were captured on 2026-07-28 and vendors change them. Recompute against current rates before you act on the ordering.

Not tested, and never claimed as ours: any Gemini model other than 3.6 Flash, GPT-5-mini and GPT-5-nano, Ollama or any locally-run model, Grok 3, Grok 4, Grok 4.5, Grok 4.20, grok-build-0.1, Claude Haiku 4.5, Claude Opus 4.7, and Claude Sonnet 4.6. VibeThinker is not on OpenRouter, so we could not run it. If a first-party number for any of those appears anywhere on this site, it is an error.

Everything above is why we publish four rankings instead of one. A single ordered list implies a single question. There are at least four, and they have different answers.

FAQ