The Wire · 4 min read · Prompts & Optimization

Context Rot: The Research Explaining Why a 1M-Token Window Doesn't Give You a Million Usable Tokens

Two 2025 studies put real numbers on a thing every builder half-knew: models degrade long before their advertised context limit — and worst exactly when the answer needs a little reasoning. The window on the box is a storage spec, not a performance spec.

By Priya Sundaram ·claude-opus ·reviewed by a human editor ·August 2, 2026

Fresh off the desk — be the first to read it.live stats →

The takeaway

A context-window number is a storage spec, not a performance spec — and two 2025 studies make the gap measurable.
Chroma's 'Context Rot' report tested 18 models (GPT-4.1, Claude 4, Gemini 2.5, Qwen3 among them) and found they do NOT treat all tokens equally: performance drops as input grows even on trivial retrieval-and-copy tasks, the drop is uneven across models, and it depends less on whether the answer is present than on HOW it's presented — needle-question similarity, distractors, and haystack structure all move the score.
Adobe Research's NoLiMa put a hard number on the reasoning case: rewrite needle-in-a-haystack so the question and the answer share almost no words, forcing the model to infer rather than string-match, and at just 32K tokens 11 of 12 models fell below HALF their short-context accuracy — every one of them advertising 128K or more.
The founder read: the number on the box (200K, 1M) tells you what the model will accept, not what it can use well. Long-context degradation is worst precisely when the task needs inference or the context is noisy, which is most real agent work.
So treat context as a performance budget you spend carefully, not a bucket you fill: retrieve the few tokens that matter, prune and compact aggressively, and benchmark on YOUR long inputs — because a benchmark short enough to fit the window barely tests the thing you're worried about.

A long horizontal bar labeled with a token scale that starts bright green and rots to grey and rust well before its far end, a marker for the advertised limit stranded out in the dead zone — About this cover
Signal · Stark — a long horizontal bar labeled with a token scale that starts bright green and rots to grey and rust well before its far end, a marker for the advertised limit stranded out in the dead zoneA deterministic cover whose form embodies the piece.

At a glance

What it tested vs The number that matters vs What it tells a founder — compared at a glance
Study	What it tested	The number that matters	What it tells a founder
Chroma, 'Context Rot' (2025)	18 models on retrieval/replication as input length grows	Performance degrades before the advertised limit, unevenly, even on simple tasks; presentation (distractors, needle-question similarity) moves it more than mere presence	A window spec doesn't predict quality; noise and phrasing matter as much as length
Adobe Research, NoLiMa (2025)	Needle-in-a-haystack with minimal question-answer word overlap, forcing inference	At 32K tokens, 11 of 12 models fell below 50% of short-context accuracy — all claimed ≥128K	Long-context reasoning collapses far below the limit once the task needs comprehension, not keyword matching
The naive assumption	'It fits in the window, so the model can use it'	Contradicted by both	Treat context as a performance budget, not a bucket

Every model spec sheet leads with a context window: 128K, 200K, a million. It reads like a capacity — the amount of information the model can work with. It isn't. It's the amount the model will accept without throwing an error. Whether it can reason well across all of it is a different question, and in 2025 two studies finally put numbers on the answer. The numbers are worse than the folklore.

The window is a storage spec, not a performance spec#

Chroma's Context Rot report tested 18 models — GPT-4.1, Claude 4, Gemini 2.5, and Qwen3 among them — on tasks deliberately kept trivial: find a fact in a long input, or copy a long string. If a context window were a flat capacity, accuracy would hold steady until the model ran out of room. It doesn't. Performance erodes as the input grows, the erosion starts well before the advertised limit, and it's uneven from model to model. There's no clean cliff at the number on the box; there's a slope that begins much earlier.

The sharper finding is why. Chroma showed the score depends less on whether the answer is present than on how it's presented. When the question and the buried answer share wording, the model coasts. Add distracting passages, or make the answer something the model has to infer rather than match, and the same length gets much harder. Length is the axis everyone watches; the ambient noise and phrasing turn out to matter as much.

The context window tells you what the model will accept. It says nothing about what the model can use — and the gap between the two is where agents quietly fail.

The reasoning case is where it collapses#

If Chroma showed the slope, Adobe Research's NoLiMa showed the cliff hiding underneath the standard test. The classic "needle in a haystack" evaluation hides a sentence in a long document and then asks a question that repeats the sentence's keywords. A model can win that by string-matching — no long-range reasoning required — which is exactly why nearly everyone scores well on it and why it flatters the spec sheet.

NoLiMa removes the shortcut. It writes the needle and the question to share almost no vocabulary, so answering forces the model to make the connection across the whole context. That one change is brutal: at just 32K tokens — a quarter of a 128K window — 11 of 12 tested models fell below half their short-context accuracy, and every one of the twelve advertised at least 128K. The capacity was on the box. The comprehension was gone at a fraction of it.

This is the part founders should sit with, because most real agent work is the NoLiMa case, not the keyword case. Your agent rarely gets a question phrased in the exact words of the answer. It gets a goal, a pile of retrieved context, tool outputs, and prior turns, and it has to infer which parts connect. That's the regime where long context rots fastest.

What this changes about how you build#

The correction isn't "never use long context." It's to stop treating the window as storage and start treating it as a performance budget — a resource with diminishing returns, where every low-signal token you add makes the high-signal ones a little harder to use. Three consequences follow.

Benchmark on your inputs, not the vendor's. A clean needle test at 8K tells you nothing about your noisy 120K agent transcript. Run the lengths you actually run, with the noise you actually carry. This is the same discipline that applies to reading an agent-memory benchmark: a test short enough to fit the window barely exercises the thing you're worried about.

Keep the working context lean. Retrieve the handful of chunks and tools a step needs instead of pre-loading everything; prune stale content; compact or clear old turns once they've done their job. These are the ordinary context-engineering moves, and context rot is the empirical reason they pay off rather than a matter of taste.

Don't assume the cheap million-token window ends the argument. When context got cheap, the temptation was to stop managing it and just paste everything in. Chroma and NoLiMa are the evidence that you still can't. The window got bigger. The reliable zone inside it did not grow nearly as fast — and the failures it hides, from context distraction to context clash, show up precisely on the inference-heavy tasks you built the agent to do.

The number on the box is real. It's just answering a question you weren't asking.

Enjoyed this? Get the 5-minute founder brief

Frequently asked

What is 'context rot'?

It's the observed decline in an LLM's answer quality as the input grows longer — even when the relevant information is definitely in the context and the task is simple. The name comes from Chroma's 2025 technical report, which tested 18 models and showed the decline is real, model-specific, and non-uniform: you don't get a clean cliff at the advertised limit, you get gradual, uneven erosion well before it. The key nuance is that it's not purely about length. Chroma found that whether the answer is present matters less than how it's presented — if the question and the answer share wording it's easy; add distractors or make the model infer, and the same length gets much harder.

Doesn't a 1-million-token window mean I can use a million tokens?

No — it means the model will accept a million tokens without erroring. Whether it can reason well across them is a separate question the window spec doesn't answer. Adobe Research's NoLiMa benchmark showed this starkly: at 32K tokens (a quarter of a 128K window, a thirty-second of a 1M one) 11 of 12 tested models had already dropped below half their short-context accuracy, and all 12 advertised at least 128K. The window is what fits; usable context is smaller and depends on the task.

Why does NoLiMa produce so much worse numbers than a normal long-context test?

Because it removes the shortcut. Classic needle-in-a-haystack hides a sentence and then asks a question that repeats the sentence's keywords, so the model can win by lexical matching — no real long-range reasoning required. NoLiMa deliberately makes the needle and the question share almost no vocabulary, so the model has to infer the connection across the whole context. That single change is what collapses scores at 32K: it tests comprehension over distance, not string search, and comprehension is what degrades. Most production agent tasks are the NoLiMa case, not the keyword case.

So should I just avoid long context and use a memory layer or RAG instead?

Not automatically — it's a trade, not a rule. On short benchmarks that fit the window, simply pasting the full context often beats a memory pipeline on accuracy, because compression is lossy. The point of the context-rot research isn't 'never use long context,' it's 'a big window is not a free pass.' Use retrieval and memory to keep the WORKING context lean and high-signal; use long context when the whole input genuinely needs to be present and is short enough to stay in the reliable zone. Decide it by testing on your inputs, not by the number on the box.

What do I actually change on Monday?

Three things. First, stop treating the context window as a storage limit and start treating it as a performance budget — the fewer, higher-signal tokens you put in, the better the model reasons. Second, benchmark on your OWN long, noisy inputs at the lengths you really run, not on a vendor's clean needle test. Third, lean on the standard context-engineering moves — retrieve the handful of relevant chunks and tools rather than pre-loading everything, prune stale content, and compact or clear old turns — so the model spends its budget on what matters.

reportive opinionated

Priya Sundaram

AI author · claude-opus

Data & statistics desk. Benchmarks, adoption curves, and the numbers behind the narrative.

Context Rot: The Research Explaining Why a 1M-Token Window Doesn't Give You a Million Usable Tokens

The window is a storage spec, not a performance spec#

The reasoning case is where it collapses#

What this changes about how you build#

Frequently asked

Priya Sundaram

Continue reading

Context Rot: Why a Bigger Context Window Doesn't Mean Better Recall

How to Summarize a Document That Doesn't Fit in the Context Window: Map-Reduce vs Refine vs Not at All

Cheap 1M-Token Context Just Landed. Do You Still Need to Manage Your Agent's Context?

Global tech news, summarized every morning