Context Rot: Why LLMs Degrade Long Before the Window Fills

A 1M-token window is not 1M tokens of usable attention. In July 2025 Chroma Research ran 18 frontier models through 194,480 calls and found the same thing in every one: reliability falls as input grows, long before the window is anywhere near full — and how information is arranged matters more than whether it is present. This is the phenomenon Chroma named "context rot." This guide explains the mechanism, the specific failure modes Chroma isolated, why the needle-in-a-haystack test gives you a false all-clear, and a matrix mapping each cause to the fix that removes it.

The short answer

A context window is the maximum you can pass, not the amount the model uses well. In July 2025 Chroma Research published Context Rot: How Increasing Input Tokens Impacts LLM Performance, testing 18 models across 194,480 LLM calls, and found that every model degrades as input length grows — well before the window fills. The degradation is driven by input length, distractors, how closely the target fact matches the query, and how the surrounding text is structured. "Context rot" is Chroma's coined term for this, echoed by Anthropic, and it reframes long context from a capacity number into an engineering constraint.

Why the window lies

The marketing number is capacity; the useful number is attention. Anthropic frames the cause plainly: a model has a finite attention budget, and "every new token introduced depletes this budget." Transformer attention requires every token to attend to every other token, which creates n-squared pairwise relationships — 1,000 tokens is a million pairs, 100,000 tokens is ten billion. As the context grows, that finite budget gets stretched thin. Anthropic explicitly acknowledges the effect: "as the number of tokens in the context window increases, the model's ability to accurately recall information from that context decreases." Part of the reason is that training data skews toward shorter sequences, so models see far fewer long-context examples in training.

Position compounds length. Liu et al.'s Lost in the Middle established a U-shaped curve: accuracy is highest when the key information sits at the very start or the very end of the input, and degrades by more than 30% when the same fact is buried in the middle — replicated across multiple model families. So two identical prompts carrying the same fact can score very differently based only on where the fact sits. That is the first hint that presence is not usability, and it is why anyone tempted to dump their whole knowledge base into a large Claude context window should treat the window size as a ceiling, not a promise.

What Chroma actually found

Chroma extended the classic needle-in-a-haystack setup along the axes that resemble real work, running 8 input lengths and 11 needle positions with over 99% agreement between LLM-based and human judgment. Four results stand out:

• Length alone degrades reliability. Even on simple retrieval and a trivial text-replication task, performance slid as inputs grew — evidence this is not a reasoning-difficulty artifact but something about length itself.
• Distractors hurt, non-uniformly. Adding semantically similar but incorrect passages degraded accuracy, and different distractor types hurt by different amounts. A single needle with distractors is much harder than a clean one.
• Low needle-question similarity accelerates decay. As the semantic match between the target fact and the query weakened, degradation with length got steeper — tested across five embedding models.
• Structure matters, counterintuitively. Models often did better on a randomly shuffled haystack than on a logically coherent one. Read this as a specific experimental result about haystack coherence, not a prescription to shuffle your data.

The through-line: models do not use their context uniformly, and how information is presented matters more than whether it is present. That single sentence is the whole finding.

Why NIAH is a false all-clear

The benchmark that reassures teams is the wrong test. A classic needle-in-a-haystack run hides one distinctive sentence in filler and asks the model to find it. Models pass this handily even at a million tokens, and vendors quote that as proof long context "works." It is the version of the problem models are best at: a single, verbatim, high-similarity needle with no competing candidates.

Chroma's contribution is showing how quickly that all-clear evaporates. Introduce distractors, weaken the wording match, require joining two or three facts, or bury the target mid-document, and accuracy falls — sometimes steeply. Real workloads are the hard version by default: support histories full of near-duplicate tickets, codebases where the relevant function looks like ten irrelevant ones, contracts where the operative clause is phrased nothing like your question. So a green NIAH result tells you the floor is fine and says almost nothing about the ceiling you actually operate at. This is the same trap behind unreliable recall in retrieval systems, and it feeds directly into hallucination — a model that silently grabs the wrong-but-similar passage will state it with full confidence.

Causes, symptoms, and fixes

Most write-ups either restate the headline ("performance drops with length") or list generic RAG tips. The useful move is to connect each cause Chroma isolated to the symptom you see and the specific fix that removes it. The fixes map cleanly onto Anthropic's context-engineering techniques:

Reranking earns the highlight because it targets the single nastiest failure — distractors — that raw retrieval and bigger windows both leave untouched. A cheap model such as DeepSeek V4 Flash makes a serviceable reranker so you narrow a pre-filtered top-k down to the passages that matter before your flagship ever sees them. For the retrieval side of this table, best LLM for RAG covers how grounding-heavy workloads reward different models than open-ended chat, and best LLM for summarization covers the closely related case where you must compress a long source without losing the middle.

Retrieve, stuff, or compact?

The decision rule falls out of the table. Retrieve when the task needs a handful of relevant passages out of a large corpus — which is most of the time. Retrieval keeps the working context short and focused, which is exactly the regime where models are reliable, and it is far cheaper: 2025-2026 RAG reviews put long-context stuffing at roughly 8-82x more expensive than retrieval for typical workloads. Treat that range as directional, not an exact benchmark. Stuff the whole document only when the task genuinely needs global coverage — a full-contract review, a single long transcript — and you have verified accuracy holds at that length on your own eval. Compact when a session or agent history keeps growing: summarize the older turns and reinitialize before you drift into the rotting zone, rather than letting the transcript balloon.

How much room do you actually have? Practitioners estimate the high-accuracy zone for 2M-window models at roughly 150K-400K tokens — well below the marketed limit. That figure comes from secondary practitioner blogs, not Chroma's paper, so use it as a rule of thumb to provoke your own measurement, not a hard line. The durable habit is to eval at the length you actually run at, with realistic distractors, because the number is model-specific and moves with every release. This is the same discipline that separates a real coding-cost benchmark from a vendor slide: measure on your data, not the marketing. A gateway makes that a config change — point the same long-context eval at Claude Opus 4.8, Qwen3 Coder Next, or GLM-5.2, and keep whichever resists rot best on your task, at your length.

FAQ