DataLLM LabCodestral Review (25.08): Eight of Nine, Beaten by Its Own Sibling
This Codestral review comes down to one task. On our executed Python benchmark Codestral 25.08 scored 8 out of 9 at $0.11 per 1,000 tasks (priced 2026-10-02), with a 2.5-second mean and zero reasoning tokens. The run was complete — all nine tasks returned code — so the 8/9 is a real score, not an API accident. The one it got wrong was parse_csv_line, the same task that tripped Qwen3 Coder Plus and Qwen3 Coder Flash in the same sweep. Meanwhile Devstral 2, from the same vendor, solved all nine at $0.23 and was faster, at 2.1 seconds. Codestral is not the Mistral model to pick for writing whole functions from a spec, and Mistral, to be fair, never said it was.
A model with “code” in its name is supposed to win a coding benchmark. On ours, the dedicated code-completion model from Mistral finished one task behind Mistral's agentic coding model, and that gap tells you something about what each was trained to do.
The result
| Metric | Codestral 25.08 | Devstral 2 |
|---|---|---|
| Score | 8/9 | 9/9 |
| Task missed | parse_csv_line (wrong answer) | none |
| Measured cost / 1,000 tasks (priced 2026-10-02) | $0.11 | $0.23 |
| Mean latency | 2.5s | 2.1s |
| Reasoning tokens per call | 0 | 0 |
| Tokens across the suite | 588 in / 950 out | 696 in / 911 out |
| List price in / out, per 1M (2026-10-02) | $0.3 / $0.9 | $0.4 / $2 |
| Context window | 256,000 | 262,144 |
| Measured | 2026-10-02 | 2026-10-02 |
Both models were measured on the same day against the same nine prompts. Neither emits reasoning tokens; both answer straight away. Codestral returned code for every task, with no empty bodies and no API-layer failures, so its run is complete and 8/9 is a valid score. It is one wrong function, not a missing one.
Devstral 2 ranked 11th cheapest and 2nd fastest of the 75 models that had scored 9 out of 9 as of 2026-10-02. The only 9/9 model ahead of it on speed was Solar Mini 4, at 2 seconds. Codestral has no rank in that list because it did not clear the suite.
The miss: parse_csv_line
parse_csv_line asks for a parser that splits one CSV line into fields. The prose spec covers quoted fields, delimiters that appear inside quotes, and quotes escaped inside a quoted field, and the hidden asserts test each of those clauses. Nothing in it is algorithmically hard. It punishes a model that writes the obvious split-on-comma parser, or a half-correct quote handler, and stops reading the spec early. We have written about why this task catches models of every size in our Llama 4 Scout review.
Codestral had company on 2026-10-02:
| Model (2026-10-02 sweep) | Score | Missed | Cost / 1k tasks (priced 2026-10-02) | Latency |
|---|---|---|---|---|
| Codestral 25.08 | 8/9 | parse_csv_line | $0.11 | 2.5s |
| Qwen3 Coder Flash | 8/9 | parse_csv_line | $0.12 | 2s |
| Qwen3 Coder Plus | 8/9 | parse_csv_line | $0.41 | 2s |
| GLM 5.3 Prime | 7/9 | token_bucket, parse_csv_line | $11.52 | 14.2s |
Every scored model in that table missed parse_csv_line, and they range from 11 cents to $11.52 per thousand tasks. Price did not buy the fix. A fifth endpoint, the free stealth model Space Bunny Alpha, also got it wrong in one of its two runs, but both of those runs were incomplete, so it is excluded and has no valid score; it does not belong in the table beside the others, and as a free endpoint it is barred from any cost comparison anyway.
A second view from the same day. When we pushed the nine tasks through TypeSafe's Jev Router, a model-picking router, it sent parse_csv_line to Claude Opus 5.5, and that single task was 56% of the summed cost of the probe (cost as reported by the API per call, 2026-10-02). The router, too, treated this as the one task worth paying up for.
What Codestral is built for
The fair reading of an 8/9 depends on what the model was designed to do. Per Mistral's own announcement (third-party, read 2026-10-02), Codestral 25.08 was released on July 30, 2025 and is built for high-precision fill-in-the-middle (FIM) completion: the editor sends the code before and after the cursor, and the model fills the gap. Mistral's headline claims for this release are about that use case — 30% more accepted completions and 50% fewer runaway generations — with a smaller 5% instruction-following gain claimed for chat mode.
Our harness tests something different. We send a function signature and a prose spec through the ordinary chat route, the same way for every model, and ask for the whole function. There is no surrounding code to anchor on, and the spec is the only source of truth. That is a chat-mode, instruction-following job, which is the part of Codestral Mistral says improved by 5%, not the part it says improved by 30%. We measured Codestral off its home ground. It still solved eight of nine, which is a decent result for a completion model, but it is not evidence of how it performs as an autocomplete engine in an IDE.
Devstral 2 is the opposite case. TechCrunch reported on December 9, 2025 (third-party, read 2026-10-02) that Devstral 2 is a 123-billion-parameter coding model released under a modified MIT licence. It is pitched at writing and changing code from instructions — much closer to what our harness asks for.
Codestral vs Devstral 2
The surprising part of the comparison is how similar the two runs look apart from the score. Codestral wrote 950 output tokens across the suite; Devstral 2 wrote 911, 39 fewer. Neither spent a single token on reasoning. So the cost gap is almost entirely the rate card: Devstral 2 charges $2 per million output tokens against Codestral's $0.9 (both priced 2026-10-02), 2.2 times more (2 ÷ 0.9), and its measured cost per thousand tasks comes out 2.1 times higher (0.23 ÷ 0.11).
In absolute terms that multiple is 12 cents per thousand tasks ($0.23 − $0.11, priced 2026-10-02). For the extra 12 cents Devstral 2 got parse_csv_line right and answered 0.4 seconds faster at the mean. If your workload is generating functions from written requirements, that is an easy trade. Codestral being cheaper does not help much when the thing it saves you money on is a wrong answer you then have to find.
The input-token difference — 588 for Codestral, 696 for Devstral 2 on identical prompts — is 108 tokens. We did not investigate why; tokenizer or template differences between the two endpoints are plausible, but we have not confirmed either.
Where $0.11 sits
Two comparisons from the chart matter for anyone choosing on price. Solar Mini 4 — the cheapest and fastest 9/9 in our data as of 2026-10-02 — scored 9/9 at $0.03 and 2 seconds, so Codestral costs 3.7 times as much (0.11 ÷ 0.03) and scored lower. Qwen3 Coder scored 9/9 at $0.13 in 2.7 seconds, two cents more than Codestral (both priced 2026-10-02) for the task Codestral missed. On this suite, Codestral sits in an awkward spot: not cheap enough to be the budget pick, not complete enough to be the safe one.
Keep the scale of all this in proportion. Nine self-contained Python functions cannot separate a frontier model from a competent small one — 75 models have cleared the whole suite. A one-task difference on a single run is a real measurement, but it is a thin one. For the wider cheap end, our cheap coding roundup has the field, and our Mistral Medium 3.5 review covers Mistral's general-purpose line.
Who should use it
- Autocomplete in an editor. This is what Codestral is designed for, and nothing in our data argues against it there. We simply did not test it.
- Generating whole functions from a written spec. Use Devstral 2. On our run it cost 12 cents more per thousand tasks (priced 2026-10-02), was faster, and did not miss.
- Lowest cost for a complete pass on tasks like ours. Neither Mistral model. Solar Mini 4 and Qwen3 Coder both scored 9/9 for less than Devstral 2.
How these numbers were produced
Nine Python tasks, each a function signature plus a spec and no example tests. The generated code is executed against hidden asserts in an isolated python3 -I subprocess with a 12-second timeout. Temperature 0, max_tokens 4000, one scored attempt per task. An API-layer failure is recorded separately from a wrong answer, which is why Codestral's miss counts against its score while excluded runs get no score at all. Cost is derived: measured input and output token counts multiplied by the list price captured on 2026-10-02 — it is not a billing statement, and prices move. The Jev Router figure is the exception: there the cost is what the API reported per call. All runs go through OpenRouter, not through the DataLLM Lab gateway. Full method on the methodology page.
What we did not measure
- Fill-in-the-middle. The use case Codestral is built for. We never sent it a prefix and suffix; every call was a chat-style request for a whole function.
- The exact bug. We know Codestral's
parse_csv_linefailed our hidden asserts. We have not published which clause of the spec it got wrong, so we will not guess here. - Repeat runs. One scored attempt per task at temperature 0. Whether Codestral misses
parse_csv_lineagain on a re-run, or Devstral 2 misses it once, is untested. - Other languages. Mistral pitches Codestral across many programming languages. We tested Python only.
- The context window. 256,000 tokens for Codestral, 262,144 for Devstral 2. Our prompts are short, so neither number was exercised.
- Multi-file and agentic work, which is where Devstral 2 is pitched. Nine single-turn functions say nothing about it.
Evidence: dated measurements and catalogue fields; method, latency statistics and limitations. Measurements describe these runs, not all providers or future versions.