A Content Filter Scored 4/9 in Our Benchmark (The Bug Was in Our Harness, Not the Model)

On 2026-07-30 we ran Claude Fable 5 through our executed coding harness. It came back 4/9, with zero reasoning tokens. A flagship model missing five of nine short Python functions is not a result, it is a fault. We opened one failing call and found an HTTP 200 response carrying finish_reason: content_filter, 8 output tokens, and an empty content body. Our runner had handed that empty body to the scorer, watched it fail the assertions, and recorded a missed task. An API-side filter on one route was being rendered as a model that cannot code. This page is the bug, the fix, the re-check of every result it could have touched, and the one thing that changed.

Every number on this site comes out of one harness: nine short Python functions, executed against hidden assertions, temperature 0. On 2026-07-30 that harness produced a number that was wrong, and the fault was entirely ours. This is what it was and what we did.

The short version: our runner treated only HTTP and transport errors as call failures. A 200 response with an empty body went straight to the scorer, failed, and was written down as a missed task. So an API-side content filter looked identical to a model getting the answer wrong. Had we published the run, the headline would have been that Anthropic's most capable model scores 4/9 on nine basic functions — false, specific, and extremely quotable.

The run that did not look like a result

Claude Fable 5 came back 4/9 with 0 reasoning tokens per task. Both halves of that were odd. Five misses would put a $10 / $50 flagship below every open-weight model we have run, and a model that produces no reasoning tokens while failing more than half the set is not thinking its way into wrong answers — it is not producing anything.

So we stopped and inspected a single failing call directly, rather than filing the score. The response was an HTTP 200. Inside it:

• finish_reason: content_filter
• usage reporting 8 output tokens
• an empty content body

Repeating the call per task showed the shape of it. two_sum returned a normal 233-character answer. lcs_len, token_bucket and valid_parentheses all came back empty with content_filter. Nothing about the prompts explains that — they are function signatures and one-sentence specs for a longest-common-subsequence length, a token bucket, and a bracket matcher. Whatever tripped the filter on that route, it was not the content of the request.

One detail worth recording because it shapes the fix: the filter did not hit the same tasks twice. In the diagnostic probe, valid_parentheses came back empty. In the later re-run under the fixed harness, valid_parentheses returned a scorable answer and flatten and parse_csv_line came back empty instead. Only lcs_len and token_bucket were empty in both. A per-task blocklist would have been the wrong mental model; this is a probabilistic gate on a route, and a retry is a legitimate response to it.

The bug: an empty body scored as a wrong answer

Here is the actual defect, in one sentence. The runner's error path caught curl failures, JSON parse failures and any response containing an error object — and nothing else. A 200 with choices[0].message.content equal to the empty string fell through as a successful call. It then went into the extractor, produced empty code, was written to a temp file, run under python3 -I, raised a NameError on the first assertion, and was recorded as a miss.

Every stage of that behaved correctly in isolation. The scorer is supposed to fail code that does not define the function. The problem is that the harness had no category for a call that succeeded and returned nothing, so a non-answer and a wrong answer collapsed into the same bucket. That is a measurement bug, not a scoring bug, and it is invisible in the output: a miss looks like a miss.

The consequence generalises past our own harness. Any evaluation that scores a non-answer as a wrong answer will systematically understate models whose routes are filtered, rate-limited, or truncated, and it will do so in a direction that always flatters the models on clean routes. It is also silent — there is no exception, no log line, no anomaly in the aggregate unless someone looks at a single number and thinks it is too low to be real.

The fix, in two parts

The first part was the obvious one, and it was not enough on its own.

1. An empty or whitespace-only body now returns an error. The call is treated as failed, which means it retries — the harness already retried API errors, never wrong answers, so the empty response simply joins the category it always belonged in. Three consecutive empties on the same task are recorded as an API failure, not a wrong answer, and the finish_reason is carried into the error string so the cause is visible in the run log.

That fixes the misclassification. It does not fix the arithmetic, because a task that never returned still sat in the denominator.

2. API failures are excluded from the denominator. The score is now pass / scored rather than pass / 9, where scored is nine minus the number of tasks that never returned a body. The measured cost, the mean latency and the mean reasoning tokens all divide by the tasks that actually returned, so a filtered task cannot silently drag an average toward zero. The failed task names are stored in an apiFail field, which is the part that matters: the exclusion is written down next to the score rather than inferred from a denominator that does not equal nine.

Re-running Claude Fable 5 under the fixed harness produced 4/5, with four tasks recorded as API failures — lcs_len, flatten, token_bucket and parse_csv_line. That is an honest number and a useless one. It shares no denominator with a 9-task score, so it cannot be ranked against anything else in the dataset, and five tasks is too thin to characterise a model. So we excluded Claude Fable 5 from the dataset entirely, with the reason stored alongside the entry, rather than publishing a figure that looks comparable and is not. What we know about that model is vendor-sourced and marked as such in our Fable 5 writeup.

The cheap, complete re-check

This is the reason to publish the incident rather than just patch it. A model that scored full marks cannot have been filtered, because a filtered task cannot pass. An empty body never defines the function, and the assertions never run. So the contamination surface of this bug is exactly the set of results below full marks — and that set is small.

That makes the surface enumerable rather than a matter of judgement. There is no sampling question about which results looked suspicious, because the list of results that could have been touched is fixed in advance by the scores themselves. Four models went back through the fixed harness, including the one that exposed the bug. Here is what each one showed — and the suspect results we did not re-run are named in what we did not measure.

Two of the three comparable results came back identical, down to the same missed task. Gemini 2.5 Pro did not. It moved from 7/9 to 6/9, at temperature 0, on the same nine prompts. No task returned empty, so this is not the filter bug — it is genuine run-to-run variance on a model that spends 2,671 reasoning tokens per task, by far the heaviest reasoning load in our dataset. Temperature 0 constrains sampling; it does not make a long reasoning trace deterministic.

That is worth more than the bug it fell out of. A one-task swing on a nine-task set is 11 percentage points of apparent capability, from nothing but re-running it. Our published figure for Gemini 2.5 Pro is the post-fix 6/9, at a measured $28.58 per 1,000 tasks and 25.3 s mean latency, priced 2026-07-30 — and anyone treating that single run as definitive, ours included, is over-reading it. The comparison in Gemini 2.5 Pro versus Claude should be read with that band in mind.

The nine results that were ever suspect

For completeness, here is the entire re-check surface — every result below full marks across the 40 usable models in the dataset. 31 of the 40 scored 9/9 and are structurally immune to this bug. These nine are not.

There are 12 individual missed tasks across all 40 models, and six of the 12 are parse_csv_line — the one task in the set with a genuinely fiddly spec, where two double-quotes inside a quoted field mean one literal quote character. flatten accounts for three more. That concentration is itself a check on the fix: if empty responses were still being scored as misses, the misses would be scattered across tasks at random rather than piling onto the two hardest ones.

Nothing in this table is a verdict on any model. Nine short functions do not separate serious 2026 coding models — the field sorts on cost and latency, not correctness, which is the argument the AI coding ranking works through. Among the 31 models that scored 9/9, measured cost runs from $0.08 to $14.70 per 1,000 tasks — a 184x spread for an identical result (DeepSeek V3.2 priced 2026-07-30, Gemini 3.1 Pro priced 2026-07-29).

Four questions to ask of any benchmark

Nothing in a published leaderboard tells you whether its authors checked for this. The output of a harness with this bug and one without it look the same: a table of scores. So here are the four questions that separate them, stated without moralising, because we failed the first one ourselves for as long as the harness existed.

1. Does it distinguish an empty response from a wrong one? If a 200 with no content is scored as a failed task, every filtered or truncated route reads as incapability. Ask specifically about finish_reason — content_filter, length and a truncated tool call are all non-answers that a naive scorer marks wrong.
2. What is the denominator? A score of X/N is meaningless unless you know what N counts. If failed calls stay in N, the number is depressed; if they are dropped without saying so, two models in the same table are being divided by different things.
3. Are excluded runs disclosed? Exclusion is often the right call — ours was. The question is whether the exclusion and its reason are published, or whether the model just quietly does not appear.
4. Was anything re-run? A single run at temperature 0 is one sample, not a measurement. Gemini 2.5 Pro moved a full task between two runs of identical prompts. If a leaderboard reports no variance, the most likely explanation is that nobody measured any.

The same logic applies to your own evaluation harness, where the stakes are higher because the output feeds a routing or procurement decision rather than a blog table. The failure modes worth instrumenting are covered in how to evaluate an LLM, and the case for logging raw responses rather than just scores in LLM observability.

How these numbers were produced

Nine executed Python tasks: two_sum, valid_parentheses, merge_intervals, roman_to_int, lcs_len, flatten, top_k_words, token_bucket, parse_csv_line.

The model receives a function signature and a prose spec. It never sees the assertions. The returned code runs against hidden assertions in an isolated python3 -I subprocess with a 12-second timeout. Every assertion passes or the task fails — no partial credit, no human grader, no LLM judge.

Temperature 0, max_tokens 4000. One scored attempt per task, retrying only on an API error — which, since the fix, includes an empty or whitespace-only response body. Calls go through OpenRouter's OpenAI-compatible endpoint, deliberately not through the DataLLM Lab gateway. Nothing on this page depends on our infrastructure.

Cost is derived, not invoiced. It is the token counts the API reported multiplied by that model's list price on a stated date, which is why every cost figure here carries one. List prices move — 49 of roughly 396 catalogue models changed price in the twelve days to 2026-07-29 — so an undated cost figure is not a fact, an argument we make at length in LLM price volatility.

On the counting. The core sweep was 13 models run in one sitting. The rest ran later on the same harness under the same settings. The dataset now holds 41 entries: 40 usable and 1 excluded, the exclusion being Claude Fable 5 for the reason above. Where this page says 40 models, that is the usable set. Author and disclosures: Kevin Fan.

What we did not measure

Not measured at all: long-context reasoning, multi-file refactoring, agentic or multi-turn tool use, any non-Python language, and vision. The harness is single-turn and API-hosted. It does not run agents, it does not use tools, and it does not run anything locally.

We did not measure how often the filter fires. We observed it on one model on one route on one day. We do not know its rate, whether it is provider-specific, whether it varies by region or by time, or whether it has since changed. Four empties out of nine tasks in one run is an observation, not a rate.

We re-ran four models, not all nine suspect results. Claude Opus 4.7, MiniMax M2.5 and Gemini 2.5 Pro went back through the fixed harness, along with Claude Fable 5 itself. The other six sub-perfect results — GPT-OSS-120B, MiMo V2.5, DeepSeek V4-Pro, Grok 4.3, Step 3.7 Flash and Gemini 3.5 Flash — still carry their pre-fix scores. Each of those six missed a single task, or two in MiMo V2.5's case, which is a long way from the four-empties-out-of-nine shape the filter produced on Fable 5. That is an inference from the shape of the data, not a re-run, and we are not going to call it verification.

We did not re-run the 31 models that scored 9/9. The argument that they cannot have been affected is logical, not empirical: a filtered task returns no function definition, so it cannot pass an assertion. That argument is sound for this bug specifically. It says nothing about run-to-run variance in the other direction — a model that scored 9/9 could plausibly score 8/9 on a second run, and we have not checked.

We ran the re-check once, not repeatedly. Gemini 2.5 Pro is now two samples: 7/9 and 6/9. That is enough to establish that variance exists and not enough to characterise it. We are not going to convert two runs into a confidence interval.

A known scoring artefact remains: the 4,000-token ceiling can truncate a very verbose model mid-answer, and a truncated answer still scores as a miss. That is a different failure from the one fixed here — the body is not empty, it is incomplete — and it penalises verbosity, which is a real production cost but is not the same thing as being wrong.

FAQ