You're Measuring Your Semantic Cache Wrong: Hit Rate Hides the False Positives

Semantic caching is the kind of optimization that demos beautifully and audits poorly. The pitch is irresistible: instead of paying for an LLM call, embed the incoming query, search a vector store for a previous query that means roughly the same thing, and if you find one, hand back the answer you already have. Teams stand this up, watch a 40% hit rate roll in, multiply by their per-call cost, and write the savings into a slide. The slide is usually wrong, and the reason is a metric almost nobody puts on it.

The hit rate is the wrong headline#

A classic exact-match cache has one failure mode: it misses, and you pay for the call you were always going to pay for. A semantic cache has a second, quieter failure mode. Because it matches on similarity rather than equality, it can decide two genuinely different questions are the same and return the wrong stored answer — no error, no exception, just a confident response to a question the user didn't ask.

That is a false hit, and it changes the entire accounting. A cache miss costs you money. A false hit costs you correctness, and it does so invisibly. So the number that governs whether a semantic cache is safe to ship is not the hit rate everyone reports; it is the false-positive rate: of the requests you served from cache, what fraction returned an answer that was actually wrong for the query.

A cache miss costs you a call. A false hit costs you a wrong answer served with a straight face — and it never throws.

The academic literature has been blunt about where false hits come from. The 2026 paper From Exact Hits to Close Enough frames semantic caching as a similarity-threshold problem: set the threshold loose and your hit rate climbs while false positives climb with it; set it tight and you suppress false hits by throwing away most of the hits you wanted. The embedding model is the other lever — a weak general-purpose embedder maps "cancel my subscription" and "why was I charged after I cancelled" into the same neighborhood, and the cache cheerfully conflates them.

The fix eats the win#

The standard remedy is well known: don't trust the vector search alone. Put a cross-encoder reranker or an LLM-as-judge behind the cache to verify that a candidate hit truly answers the incoming query before you serve it. This works. It also quietly dismantles the economics.

Walk the path. You installed the cache to avoid an expensive model call and shave latency. To make the cache safe, you now run an embedding model on every request, a vector search, and then a second model — the reranker — to double-check each hit. On a true hit you've replaced one big call with two smaller ones plus a lookup; on a miss you've added the embed-and-search overhead to a call you were going to make anyway. The savings survive only if hits are frequent enough and the verification is cheap enough to stay under the cost of the call you skipped. That is a real band, but it is narrower than the slide assumes.

The viability band#

Put the two forces together and semantic caching turns out to pay off in a specific corner of the space: high query redundancy and high tolerance for approximate answers. Customer-support FAQs, repeated agent sub-questions, retrieval queries that recur across users — these have the redundancy to fill a cache and the slack to absorb the occasional near-miss. Practitioner guidance in 2026 lands on false-positive tolerances of roughly 2% for ordinary systems and closer to 0.5% for regulated ones. Below those thresholds you are forced into aggressive verification, and the verification is where your savings go to die.

Outside that corner, the honest answer is usually the boring one. Prompt caching — the provider-side, exact-prefix kind — reuses computation only when the start of your prompt is byte-for-byte identical, which means it can never return a wrong answer. It saves less per hit, but it has zero false-positive risk and no reranker tax. For the long shared system prompts and repeated context that dominate agent workloads, exact-prefix caching quietly captures most of the achievable win with none of the correctness exposure.

Which of the two to reach for is a separate buyer's-guide question; this piece is about the instrument you point at whichever one you pick. Semantic caching is not a scam; it is a sharp tool with a specific edge. But if the only number in your evaluation is the hit rate, you have measured how often the cache fires and not once measured whether it was right to. Instrument the false-positive rate first. If you can't hold it under your domain's tolerance without a reranker that erases the savings, you don't have a caching win — you have a slower, cheaper-looking way to be occasionally, confidently wrong.