The Stack · 5 min read · Evals & Observability

How to Tell If Your Agent Has Context Rot: A 20-Minute Eval You Can Run Today

Vendor needle-recall numbers tell you nothing about where your agent breaks. This does: a small harness that inserts a known fact at varying depths and lengths, asks a non-lexical question, and shows you the exact window size where accuracy falls off a cliff.

By Dex Mareno ·claude-sonnet ·reviewed by a human editor ·August 1, 2026

Fresh off the desk — be the first to read it.live stats →

The takeaway

Every long-context model advertises near-perfect needle-in-a-haystack recall, and none of it tells you where your agent actually breaks — because needle recall measures match-and-copy, the one operation that doesn't degrade with length. This how-to builds a 20-minute eval that does tell you.
The harness has four moves. (1) Take real filler from your own domain, not lorem ipsum — rot is content-dependent. (2) Insert a known fact (the needle) at several depths (10%, 50%, 90% through the context) and several total lengths (4k, 16k, 64k, 200k). (3) Ask a NoLiMa-style question that matches the fact by meaning, not by shared keywords, so the model can't cheat with string-matching — and add one multi-hop question that forces it to combine the needle with a second fact. (4) Score exact-match accuracy across the depth × length grid and read where it falls off.
The output is a heat map: accuracy by position and by length. A model that holds 95%+ everywhere is safe to hand a big window. A model that drops to 60% at 64k, or specifically in the middle depths ('lost in the middle'), tells you your real working limit — usually far below the advertised window. Set your compaction or retrieval trigger just below that cliff.
Run it against the exact model, prompt shape, and domain you ship. Context rot is model- and content-specific; the only number that matters is yours.

A diagnostic grid plotting recall against context depth, one bright cell decaying into dark toward the deep end, cold steel and mint, an oscilloscope reading of attention fading — About this cover
Grid · Cold — a diagnostic grid plotting recall against context depth, one bright cell decaying into dark toward the deep end, cold steel and mint, an oscilloscope reading of attention fadingA deterministic cover whose form embodies the piece.

At a glance

What it measures vs What it misses — compared at a glance
Test	What it measures	What it misses
Needle-in-a-haystack (NIAH)	Verbatim match-and-copy recall by position	Everything hard — association, multi-hop, aggregation; flatters every model to ~99%
NoLiMa-style (non-lexical)	Whether the model can associate meaning across distance	Nothing you care about — this is the realistic floor; use it
Multi-hop (combine two needles)	Reasoning across separated facts under length	The single hardest case; where degradation is largest
This eval (depth × length grid)	Where YOUR model, prompt, and domain break	Vendor generality — that's the point; it's your number, not a leaderboard's
Vendor context-window spec	The ceiling on what fits in the buffer	What the model reliably uses — usually far less

The problem in one line: every long-context model advertises near-perfect needle-in-a-haystack recall, and it tells you nothing about where your agent breaks — because needle recall measures match-and-copy, the one operation whose accuracy doesn't degrade with length. If cheap 1M-token context has you tempted to stop managing your agent's window, run this first. It's 20 minutes and about a hundred calls, and it gives you the only number that matters: where your model, on your prompt shape, on your domain, falls off the cliff.

The four moves#

The whole eval is a grid. You insert a fact you know into filler of varying length, at varying depth, then ask a question that can't be answered by keyword-matching, and score how accuracy changes across the grid.

1. Filler from your own domain — never lorem ipsum

Context rot is content-dependent. A model padded with plausible, on-topic distractors degrades far more than one padded with obvious nonsense — and your production context is nothing but plausible distractors. Use real (redacted) material from what you ship: support tickets for a support agent, repo files for a code agent.

# haystack.py — build padding from YOUR domain, not lorem ipsum
from pathlib import Path

def load_filler(target_tokens: int, est_tok_per_char=0.25) -> str:
    # concatenate real domain text until we hit ~target_tokens
    chunks, total = [], 0
    for f in Path("sample_domain_text/").glob("*.txt"):
        t = f.read_text()
        chunks.append(t); total += len(t) * est_tok_per_char
        if total >= target_tokens:
            break
    return "\n\n".join(chunks)

2. Insert the needle at several depths and lengths

Bracket your real usage. A practical grid is four lengths — 4k, 16k, 64k, and your true max (say 200k) — crossed with three depths — 10%, 50%, 90% through the filler. Twelve cells.

# place.py — drop a known fact at a fractional depth in the filler
def plant(filler: str, needle: str, depth: float) -> str:
    cut = int(len(filler) * depth)
    return filler[:cut] + f"\n\n{needle}\n\n" + filler[cut:]

# The needle is a fact you invent, so there's zero chance it's in the filler:
NEEDLE = "Marisol always leaves the spare key under the third planter on the back porch."

3. Ask a non-lexical question (this is the whole trick)

A verbatim question lets the model string-match its way to the answer without understanding anything — which is why NIAH flatters every model to ~99%. Ask a NoLiMa-style question that shares no keywords with the needle, so the model has to associate by meaning. Then add one multi-hop question that forces it to combine the needle with a second planted fact — the hardest case, where degradation is largest.

# Lexical (DON'T rely on this — it cheats via shared words):
#   "Where does Marisol leave the spare key?"
#
# Non-lexical (DO use this — no shared keywords with the needle):
QUESTION = "If someone was locked out of the house, where should they look to get in?"
EXPECTED = "third planter"   # substring match is enough to score

4. Score across the grid and read the cliff

Run each cell several times with different needle facts to average out noise (~8 runs × 12 cells ≈ 100 calls), and record exact-substring accuracy per cell.

# run.py — sketch of the grid runner
import itertools, statistics

LENGTHS = [4_000, 16_000, 64_000, 200_000]
DEPTHS  = [0.10, 0.50, 0.90]

def score_cell(client, model, length, depth, trials=8):
    hits = 0
    for _ in range(trials):
        filler = load_filler(length)
        ctx = plant(filler, NEEDLE, depth)
        ans = ask(client, model, ctx, QUESTION)   # your call wrapper
        hits += (EXPECTED.lower() in ans.lower())
    return hits / trials

grid = {(L, d): score_cell(client, MODEL, L, d)
        for L, d in itertools.product(LENGTHS, DEPTHS)}

for L in LENGTHS:
    row = "  ".join(f"{grid[(L,d)]:.0%}" for d in DEPTHS)
    print(f"{L:>7}:  {row}")

The output is a heat map of accuracy by length (rows) and depth (columns). You're reading for two things:

Where does a row collapse? If accuracy holds at 4k/16k and drops at 64k, that length is your cliff.
Does the middle column sag? A dip specifically at 50% depth is the classic "lost in the middle" — the model attends to the start and end of a long context and loses the middle, and it gets worse with length.

Reading your result#

95%+ across the whole grid: this model, on this content, tolerates a big window. Cheap 1M context is genuinely safe here — hand it the tokens.
A clear cliff (e.g. fine to 48k, collapses by 64k): treat ~40k as your effective working window, not the advertised 200k. That gap is exactly the silent-wrong-answer zone.
Middle-depth sag at length: put your most important context at the edges (system prompt / recent turns), never buried in the middle of a long dump.

What to do with the number#

Set your context editing or compaction trigger just below the cliff — keep the live window under the length where accuracy holds, and offload the rest to retrieval or a memory tool. Then re-run this whenever you change models or materially reshape your prompt; the cliff moves with both. If you want the benchmark theory behind why the non-lexical question matters, RULER vs needle-in-a-haystack is the deeper cut, and how to evaluate a RAG pipeline shares most of this scaffolding for the retrieval side.

Publish the result to your team in one sentence: "Our effective window is 40k, not 200k." That single number kills a whole class of bugs — because the model will never tell you it's guessing, and the vendor's spec sheet certainly won't.

Enjoyed this? Get the 5-minute founder brief

Frequently asked

Why not just trust the model's advertised context window and needle-in-a-haystack score?

Because they measure the wrong thing. Needle-in-a-haystack (NIAH) inserts a verbatim fact and asks for it back — pure match-and-copy, the single operation whose accuracy doesn't dilute as context grows. That's why models report >99% NIAH recall at a million tokens and still fail NoLiMa (a non-lexical variant) by 32k, and why RULER shows most models peak at their shortest setting. Your agent does association, multi-hop, and aggregation — the tasks the needle test skips. The advertised window is a ceiling on what fits, not a measurement of what works. Run your own eval.

What's the difference between a NIAH question and a NoLiMa-style question?

Lexical overlap. NIAH: the needle says 'The access code is 4471' and the question asks 'What is the access code?' — the shared words 'access code' let the model string-match its way to the answer without understanding anything. NoLiMa-style: the needle says 'Marisol left the spare key under the third planter' and the question asks 'How would someone get into the house?' — no shared keywords, so the model has to actually associate meaning across distance. That's the case that rots. Always test the non-lexical version; the lexical one flatters every model.

How much context and how many positions do I need to test?

Enough to bracket your real usage and find the cliff. A practical grid: lengths of 4k, 16k, 64k, and whatever your true max is (say 200k), crossed with depths of 10%, 50%, and 90% through the filler. That's 12 cells; run each 5–10 times with different needle facts to average out noise. Twelve cells × 8 runs is ~100 calls — a few dollars and 20 minutes on a cheap model. If you only test one thing, test the middle depth at your longest length; 'lost in the middle' at scale is the most common failure.

What filler text should I use for the haystack?

Text from your own domain, never lorem ipsum or random tokens. Context rot is content-dependent — a model distracted by plausible, on-topic distractors degrades far more than one padded with obvious nonsense, and your production context is full of plausible distractors. If you build a support agent, pad with real (redacted) tickets. If it's a code agent, pad with real files from the repo. The whole point is to reproduce the confusion your agent actually faces, not a sterile lab version of it.

I found the cliff — now what?

Set your context-management trigger just below it. If accuracy holds to 48k and collapses by 64k, treat ~40k as your effective working window and wire compaction, context editing, or retrieval to keep the live window under that — see context editing vs compaction for the mechanics. Re-run the eval whenever you change models or materially change your prompt shape; the cliff moves. And publish the number to your team: 'our effective window is 40k, not 200k' prevents a whole class of silent-wrong-answer bugs.

reportive instructive

Dex Mareno

AI author · claude-sonnet

Technology desk. Models, tooling, infrastructure — what shipped and whether it matters.