You have been watching the wrong number. Token count tells you how much text your agent moved; it does not tell you what that text cost, because two prompts with identical token counts can bill 10x apart. The number that actually decides your agent's bill — and its latency — is the share of your prompt that hits cache.

Here is the whole mechanic in one screen: a model computes key/value (KV) tensors for every token in your prompt. Prompt caching stores those tensors for the prefix, so on the next turn only the newly appended tokens are computed fresh and the rest is reused. On Claude Sonnet, that reused input runs about $0.30 per million tokens against roughly $3.00 uncached — a 10x gap on the exact same content (Anthropic). The Manus team, after shipping a production agent, put it plainly: KV-cache hit rate is the single most important metric for an agent in production, because it moves cost and latency at the same time (Manus).

The catch: one token kills it#

The prefix is brittle by design. Caching only works if the prefix is byte-identical to last time — so a single changed token invalidates the cache from that point to the end of the prompt. Everything below the change gets reprocessed at full price.

That's what makes this sneaky. Nothing errors. The agent runs, the output looks right, and your hit rate is quietly sitting near zero while you pay uncached rates every turn. The damage is invisible until you look at the bill.

Token count measures how much you sent. Cache-hit rate measures how much you paid to send it again. Optimize the second one.

The four rules#

Everything that protects the cache comes down to never touching the prefix once it's set. Four concrete rules, straight from what production agent teams learned the expensive way:

1. Keep the prompt prefix stable. The classic self-inflicted wound is a timestamp with second-level precision in the system prompt — it changes on every request, so nothing before it ever hits cache. If the agent needs the time, put it in the turn's user message, or coarsen it to the hour or day. Audit your system prompt for anything that varies per call.

2. Make context append-only. Never rewrite earlier turns in place. Any edit below the cache point forces a reprocess of everything after it. Add new context to the end; treat history as immutable. This is also why serialization matters — if you re-serialize a JSON object and its keys come out in a different order, the model sees new tokens even though the meaning is identical. Serialize deterministically, with a stable key order.

3. Don't edit tools mid-conversation. Tool definitions sit early in the prompt, so adding or removing one invalidates the entire conversation after it — and it can leave dangling references the model trips over. Keep the full tool set fixed and control availability by masking logits so the model can't select a disabled tool. That's a decoding-time filter, not a prompt edit; the tool-swap mechanics are their own how-to, and caching the tool-definition block protects the most expensive part of the prefix.

4. Append retrieval late. If you inject retrieved chunks near the top of the prompt, you shift the whole prefix every turn. Put the stable system prompt and tools first, and add retrieved content after them, so the cacheable part stays put.

Where the context levers fit#

This is also the honest lens on the context-management debate. Every technique for keeping a long-running agent in its window pays a cache tax, and knowing the size of that tax is how you choose:

The full division of labor across editing, compaction, and memory is worth having next to this, because the right context strategy is partly a cache decision.

Get the four rules right and you cut cost and latency without changing a single thing the agent does. That's the rare optimization with no tradeoff — you're not making the agent worse to make it cheaper, you're just stopping it from paying twice for the same tokens.