Open your LLM billing dashboard and the number that hurts is almost never output. It's input — and most of that input is the same context, re-retrieved and re-sent on every single request. Every turn of an agent re-ships the system prompt, the tool definitions, the documents you retrieved, and the whole conversation so far, and you pay full input price for all of it, again. A 20-turn agent carrying 15k tokens of context can re-send that 15k up to twenty times.

The short version: Stop paying full price for context you've already assembled. Four techniques, in order of leverage: prompt caching (a config change, ~90% off a repeated prefix), compile-ahead context (precompute your knowledge once instead of searching raw docs every turn), rerank-then-trim (send fewer chunks without losing recall), and context editing (drop stale tool results in long loops). Turn on the first today.

1. Prompt caching — the fastest win, the one most teams skip#

If you send the same system prompt, the same tool list, or the same shared document on repeated requests, cache it. On Anthropic's API, cached prefix tokens read back at roughly 0.1× base input price — about 90% off the cached portion — while the initial write costs ~1.25× (5-minute cache) or ~2× (1-hour). Two requests against the same prefix already break even. OpenAI and Google (Gemini) offer the same idea under different names.

The one rule that decides whether it works: caching is a prefix match. Any byte that changes anywhere in the prefix invalidates everything after it. So put stable content first — a frozen system prompt, a deterministically-sorted tool list — and push anything volatile (a timestamp, a per-request ID, the user's actual question) to the very end. If your cache-read count is stuck at zero across near-identical requests, you have a silent invalidator up front: a datetime.now() in the system prompt, JSON serialized without sorted keys, or a tool set that varies per user. Diff the rendered prompt bytes between two calls and you'll find it.

2. Compile-ahead context — stop searching raw docs every turn#

Classic RAG runs a fresh semantic search over your raw documents on every request and stuffs the top chunks into context. You pay the retrieval and the tokens each time, for information that barely changed.

Compile-ahead flips it: you precompute your sources into a structured, queryable knowledge layer once, and at runtime the agent queries that compiled layer instead of re-searching raw text. This is the shift Pinecone's Nexus productizes — retrieval moves from search-at-runtime to compile-ahead-of-time — and Pinecone reports up to ~90% lower token spend with task completion above 90%. The tradeoff is honest: you take on a build step and you own the staleness, recompiling when the underlying data changes. For a large corpus queried by many agents, that trade is usually worth it. For a tiny, fast-changing dataset, it isn't.

3. Rerank-then-trim — send fewer chunks, keep the recall#

The reflex with RAG is to send more chunks "just in case," which inflates input tokens on every call. Invert it: retrieve a wide candidate set, run it through a reranker, and send only the top few. You keep recall — the reranker, not the context window, does the filtering — while cutting the tokens you actually ship. A top-20 retrieval reranked down to a top-4 send is a large, free-of-recall-loss reduction on a hot path you hit every request.

4. Context editing — don't re-ship a 40-turn history at full price#

Long agent loops accumulate junk: tool results from twelve turns ago you'll never reference again, completed thinking blocks, dead-end branches. Left alone, all of it re-ships at full input price on every subsequent step, and the cost grows with the loop. Context editing clears stale tool results and old thinking from the transcript as the agent runs — pruning rather than summarizing — so a 40-turn session doesn't carry its entire history forward at full weight. Reach for this specifically when the loop is long and the transcript is mostly exhaust.

The order that matters#

Don't do all four at once. Turn on prompt caching first — it's a configuration change, not a rebuild, and it's the biggest same-day win. Add rerank-then-trim if you're shipping large retrieved contexts. Reach for a compile-ahead layer when you have a big, relatively stable corpus that many agents hit. Add context editing only when long loops are the problem. Each one attacks the same root cause from a different angle: the cheapest token is the one you don't send twice at full price.

For where this fits in the wider shift toward compiled knowledge and agent control planes, see our Founder's Wire on the governance-and-knowledge layer, and for the retrieval-quality side of the tradeoff, Agentic RAG vs Naive RAG.