LLM Coding Cost Benchmark: Dated Runs and Exact Costs

The original thirteen-model July 2026 coding experiment is a dated cost and pass-count comparison, not a current model ranking. Ten configurations scored 9/9 on nine short functions. This page now also houses the later archive's filtering and unrounded cost calculation, kept separate from that original experiment. No model was rerun for this consolidation.

What we tested

We gave 13 models the same nine coding tasks, executed every answer against hidden test cases, and recorded the measured cost, latency, and reasoning tokens for each call. This is a fresh, single-harness run in July 2026 — every model measured the same way, so the numbers are directly comparable. The line-up spans the closed frontier, the open-weights field, and the cheap coding specialists:

• Closed frontier: GPT-5.5 (OpenAI), Claude Opus 4.8 and Claude Sonnet 5 (Anthropic), Grok 4.3 (xAI)
• Open-weights flagships: GLM 5.2 (Z.ai), Nemotron 3 Ultra (Nvidia), Mistral Medium 3.5, MiniMax M3, Kimi K2.7-Code (Moonshot), StepFun 3.7 Flash
• Cheap coding specialists: Qwen3 Coder Next (Alibaba), DeepSeek V4-Flash and V4-Pro

The nine tasks mix classic interview problems with edge-case-heavy parsing and stateful logic, so a model has to handle detail, not just recognise a pattern: two-sum (hash-map pairing), valid-parentheses (stack matching), merge-intervals (sort then merge), Roman-to-integer (subtractive notation), longest-common-subsequence (DP), nested-dict flatten (path-joined keys), top-k words (frequency with an alphabetical tiebreak), a token-bucket rate-limiter (stateful timing), and a CSV-line parser (quoted fields with escaped double-quotes). Each is given as a signature plus spec — no example tests — and graded only by running the code.

Results: almost everyone passed

Ten of thirteen solved all nine. That is the real 2026 story: on bounded, well-specified coding tasks the frontier has converged, and a perfect score no longer separates a $8.83 model from a $0.10 one. The three misses were single slips, each on a different task:

There was no single task that broke everyone — the misses spread across three unrelated problems — which reads more like a one-off slip than a capability wall. Notably, the two priciest models (GPT-5.5, Claude Opus 4.8) and the two cheapest (Qwen3 Coder Next, DeepSeek V4-Flash) all landed on the same perfect 9/9.

Results: the 88x cost gap

Here is where the models stop looking alike. Same nine tasks, same score for most of them — wildly different bills. This is the measured cost to run all nine, scaled to 1,000 tasks:

Historical latency observations, not a causal experiment

In the original thirteen-model sample, Mistral Medium 3.5 had the lowest recorded mean at 2.9 seconds per task. Qwen3 Coder Next recorded 7.0 seconds and Sonnet 5 recorded 7.2 seconds. Those are the dated table's observations, not current service guarantees.

The experiment did not isolate network, load, implementation or internal reasoning as causes. A reported zero reasoning-token field does not establish that no internal deliberation occurred, or explain why an endpoint was faster. Preserve the measurement without claiming a causal result the harness did not test.

Reported reasoning usage has an observability limit

The original table records approximately 732 reasoning tokens per task for DeepSeek V4-Pro and 623 for MiniMax M3. Other rows expose zero. These are fields reported through the tested route, not a complete trace of each model's internal work.

Use the provider's reported billable input and output counts at the captured rates for the historical calculation. Do not add a reasoning subcount to an output total that already includes it. Missing categories cannot be reconstructed from a latency number or an architecture label.

What the small test can and cannot establish

Ten of the original thirteen configurations passed all nine short functions. That is a saturated score on this task set, not proof that general coding quality converged or that the answers were identical.

The harness did not evaluate repository edits, tool calls, multi-turn recovery, long context or a production routing policy. It used one scored attempt per task; familiar problems may also overlap training material. Do not use a nine-task tie as a broad failure-rate estimate.

The useful result is a dated record of pass counts, usage-derived base costs and elapsed time under the stated settings. The methodology supplies the shared contract. Workload selection requires a separate acceptance set.

The October 3 archive is a different, larger cohort

The original thirteen-model July experiment above stays intact. A later published archive contains 110 entries: 96 not marked excluded, 78 with a complete 9/9, and 75 paid complete configurations after three free entries are removed. They span different run and price dates. This is not a new simultaneous 110-model experiment.

The former nine-task cheapest-model page is consolidated here. Its useful contribution is the filter and exact cost calculation, not another claim that a perfect score ranks general capability.

Calculate before rounding the display

derived USD per 1,000 tasks =
  (suite input tokens * captured input rate per million
   + suite output tokens * captured output rate per million) / 9,000

Solar Mini 4's 1,009 input and 938 output tokens at the October 2 rates of $0.05/$0.20 give $0.02645. Fugu Ultra v2's September 15 captured rates and recorded 60,645/7,011 tokens give $57.061666.... Dividing rounded displays $57.06 by $0.03 produces the old 1,902x figure; unrounded amounts produce approximately 2,157.34x. Neither is a savings guarantee or a new quality result.

Keep incomplete, excluded and free configurations outside the paid complete comparison. The following selected archive rows retain their own dates; they do not replace the original thirteen-row cohort:

Download the offline archive arithmetic kit. It contains the decimal calculation, a dated copy of the public evidence and tests. It never fetches prices or calls a model. Costs are normalized from nine tasks and omit billing categories absent from the source; they are not invoices.

Evaluate routing as a third configuration

A cheaper baseline followed by escalation is a proposed workflow, not a measured outcome of this one-shot benchmark. If you evaluate it, count the initial failed attempt, escalation call, tool work, retries and review time. Compare completed work against all-baseline and all-frontier configurations under the same acceptance tests.

Do not carry a July model ID or a historical token rate into a new integration without checking current support. Use the routing guide for route behavior and the provider's current contract for charges. This page does not establish an 88x production saving or an agent-quality winner.

FAQ

Are the original thirteen models the same cohort as the later archive?

No. The original July experiment and the later 110-entry archive are separate dated cohorts. The archive has 75 paid complete 9/9 configurations after filtering.

Were 1,000 tasks run for each model?

No. The complete configurations contain nine tasks. Costs are normalized from recorded usage at dated rates, not invoices for 1,000 executions.

Why does rounding change the historical cost ratio?

Dividing rounded displays can lose substantial precision at very low costs. Calculate from original token counts and historical rates, then round the final value.

Does a perfect score establish equal agent capability?

No. The short single-function test does not cover repository edits, tool loops, long context or multi-turn recovery.