Open any guide to cutting inference cost and you'll hit the same instruction, phrased a dozen ways: put the static stuff first. System prompt, tool schemas, few-shot examples — front-load them so the engine can prefix-cache their KV and skip the prefill on every subsequent request. It's good advice. It's also the reason a lot of agent builders are staring at a prefill bill that refuses to drop no matter how carefully they ordered their prompt.
The advice isn't wrong. It's scoped to a workload that agents aren't.
How prefix caching actually decides what to reuse#
The mechanism is stricter than "reuse text you've seen before." In vLLM, the KV cache is chopped into fixed-size blocks, and each block gets a hash key computed over its own tokens plus the hash of its parent block. The keys form a chain. To reuse a block, every block before it must match too — which means reuse extends exactly up to the first token that differs, and not one token further. SGLang's RadixAttention is a different data structure with the same consequence: the shared region is a prefix.
That's a perfect fit for a chatbot. A long fixed system prompt, then short user turns appended at the end — the prefix is stable, the divergence is always at the tail, and the cache hit rate stays high.
Now look at what an agent actually sends.
The insertion that throws it all away#
An agent loop doesn't append at the tail. It splices. Turn by turn it inserts a tool result here, a retrieved document there, a new observation into the middle of a growing transcript. And here's the part that surprises people: the content of your carefully front-loaded system prompt hasn't changed at all. But the moment you insert 800 tokens of tool output ahead of some later block, every token after the insertion point moves to a new position. New position means a different place in the parent-hash chain. Different chain means a cache miss.
The engine recomputes thousands of byte-for-byte identical tokens, not because the text changed, but because something upstream of them did. Prefix caching can't tell "moved" apart from "different."
So the RAG-augmented, tool-using agent — the workload with the most repeated content, where caching should pay off hardest — is precisely the one where the prefix hit rate decays toward zero. You did everything the guide said. You put the static content first. It didn't help, because the variability isn't at the front; it's in the middle, and prefix caching has no concept of reusing anything that isn't anchored to token zero. Offloading that KV to CPU or disk with something like LMCache or Mooncake doesn't fix it either — offloading changes where the KV lives, not the fact that it's still keyed as a prefix.
CacheBlend: reuse the position-wrong cache on purpose#
The interesting fix doesn't try to make the prefix longer. It abandons the prefix constraint entirely — and to do that, it has to solve the reason the constraint exists.
Why can't you just cache each chunk independently and stitch them together at inference? Because attention is not local. A token's KV depends on everything it attended to, and a chunk computed in isolation never got to attend to the chunks that now sit before it. Naïvely concatenating independently-cached KV blocks produces subtly wrong attention and degraded output. That cross-chunk dependency is the whole reason prefix caching insists on an exact chain in the first place.
CacheBlend — which won Best Paper at EuroSys '25 — makes a measured bet about that dependency: most of it doesn't matter. When you drop a cached chunk into a new position, only a small fraction of its tokens have KV that deviates enough from the correct value to actually change the output. CacheBlend calls these the High-KV-Deviation (HKVD) tokens, identifies them, and recomputes only those — typically 10–15% of tokens — while reusing the other 85–90% of the cache exactly as stored, position be damned.
The result is a cache that is technically "stale" for the vast majority of its tokens and empirically accurate anyway. The paper reports 2.2–3.3× lower time-to-first-token and 2.8–5× higher throughput against full KV recompute, with no meaningful hit to generation quality — because you're paying full price for a sixth of the tokens and near-zero for the rest.
What to actually do with this#
Two things. First, diagnose honestly: if you have a tool-using or RAG agent and your prefill cost isn't falling despite a big static prefix, stop reordering the prompt — you're optimizing a mechanism that structurally can't help you, because your variability lives mid-context. Confirm it by logging the prefix-cache hit rate per request; you'll likely watch it crater the moment retrieval or tool output enters the loop.
Second, non-prefix reuse is not just a paper. It ships today as non-prefix KV reuse in LMCache, which integrates with vLLM and, in its own words, "extend[s] KV reuse beyond prefix caching by reusing cached KV blocks at any position," leaning on CacheBlend to selectively recompute. It sits in an active 2026 research line on position-independent caching, so expect the surrounding tooling to keep improving.
The mental correction is the valuable part. Prefix caching taught everyone to think about order — get the static content to the front. For agents, the better question is about position independence: can I reuse this chunk's cache no matter where it lands in the context, and pay only for the handful of tokens that genuinely change when it moves? That's a different question, and for the workload most agents actually run, it's the one that pays.
