The pitch is seductive because the numbers are true. Vendor write-ups in 2026 advertise semantic caching as a 40–80% cut to your LLM bill. Redis LangCache reports up to 73% cost reduction on high-repetition workloads. Cache hits come back in sub-millisecond to 50ms against the 3–10 seconds a live model call takes; one RAG pipeline's retrieval dropped from 6,504ms to 1,919ms. If you run an agent and you're watching the invoice climb, that reads like free money.

Before you wire GPTCache around your agent loop, sit with two words in that 73% claim: high-repetition. That is not a hedge. It is the entire condition under which the number exists — and it describes a workload most agents don't have.

Two different things wear the same word#

The confusion at the center of this is that "caching" names two mechanisms that behave nothing alike.

Prefix caching — what OpenAI does automatically for prompts over 1,024 tokens, what Anthropic exposes as prompt caching with roughly 90% savings on cache reads — matches on identical leading tokens. Same prefix, reuse the computation already done on it, produce the same output you would have produced anyway. It is deterministic. It cannot return a wrong answer, because it isn't returning an answer at all — it's skipping recomputation of a shared prefix. The savings are mechanical and correctness-neutral.

Semantic caching — GPTCache from Zilliz, Redis LangCache, the gateway-native caches — matches on embedding similarity. It takes your incoming query, embeds it, finds a past query whose vector is close enough, and returns that past query's answer. The two queries are not the same. The system is betting that "close in embedding space" means "same correct answer." For a support bot fielding the thousandth phrasing of "how do I reset my password," that bet pays off beautifully — the answer genuinely is the same. That's where the 73% lives.

Prefix caching skips work you already did. Semantic caching guesses that a different question has the same answer. Only one of those can be wrong.

Why agents are the worst-case input#

Now look at what an agent actually feeds a cache. Its steps are textually repetitive — "search the docs for X," "call the pricing tool," "summarize the results" — which is exactly the surface pattern that makes semantic similarity light up. But the meaning of each step is loaded with context the embedding barely encodes: which task, which user, which point in a multi-step plan, what the previous tool returned. Two "call the pricing tool" steps can be near-identical vectors and require completely different results.

So the failure mode isn't a slightly-off answer. It's a semantic cache hit that hands your agent a stale tool result from an hour ago, or a plan fragment from a different task that happened to phrase itself similarly. The agent then reasons forward on a false premise, confidently, because nothing in the trace says "this came from a cache." High-repetition FAQ traffic is forgiving of a bad hit — the user rephrases and moves on. An agent compounds the bad hit through three more tool calls before anyone notices.

The headline savings and the agent risk are the same property viewed from two sides. Semantic caching wins big precisely when many inputs should collapse to one output. Agent correctness depends on inputs that look alike staying distinct. You cannot have both from the same cache.

The pattern that actually holds#

None of this means agents should pay full price. It means matching the cache type to the layer:

Use prefix/prompt caching aggressively — it's the real agent win. Agents carry a long, stable system prompt and a growing context on every single step. That repeated prefix is the ideal target for deterministic caching, and it's often automatic (OpenAI) or ~90% cheaper on reads (Anthropic). This is where your money is, and it's safe by construction.

Confine semantic caching to the outermost turn. The one place embedding-similarity caching fits an agent is the very first user-facing question — before any tools run, before any plan exists. Two users asking the same thing in different words is the FAQ pattern semantic caching was built for. Put it there, and only there.

Gate it like it can be wrong, because it can. A high similarity threshold (err toward misses), plus a scope key that segments the cache by user or tenant and a freshness window that expires entries, is the difference between a cost optimization and a silent correctness bug. Never let a semantic cache sit between the agent and a tool call, and never let it short-circuit intermediate reasoning.

The 73% is real. It's just the answer to a question — "how much can I save on repetitive FAQ traffic?" — that your agent isn't asking. Take the deterministic savings that fit its actual shape, and treat the embedding-similarity kind as a sharp tool you point only at the one layer that can absorb a wrong guess.