Add semantic caching when your traffic is repetitive and a near-duplicate answer is genuinely acceptable — FAQ bots, documentation Q&A, support triage, anything where "how do I reset my password" and "I forgot my password, help" should return the same thing. It will burn you the moment small wording changes are supposed to change the answer: dates ("Q2" vs "Q3"), names ("my order" vs "John's order"), prices, or anything per-user and stateful. There, two queries can sit millimeters apart in embedding space while demanding completely different answers, and a semantic cache will hand back the wrong one with total confidence. This is the whole trade: you swap a small, real correctness risk for a large cost-and-latency win, and it is only safe on some workloads.

Two kinds of caching people confuse#

Semantic caching is something you build. You store past answers keyed by the embedding of the query, and when a new query is close enough to an old one, you return the stored answer and never call the model. That's the 30–90% savings — you skipped inference entirely.

Prompt caching (Anthropic's cache_control, OpenAI's automatic prefix cache) is a provider feature. It discounts re-processing an identical prompt prefix — a big system prompt, a long document — so you pay ~0.1× on the cached input tokens instead of full price. The model still runs and still generates fresh output. Anthropic requires you to mark breakpoints and matches an exact byte prefix; OpenAI caches automatically for prompts ≥1024 tokens and gives a ~50% input discount on the cached prefix. If you're weighing the two purely on cost and accuracy, we go deeper in semantic caching vs prompt caching.

Semantic caching skips the model; prompt caching makes running the model cheaper. Use both — they solve different problems and stack cleanly.

Start with the free win: exact-match caching#

Before anything clever, cache on the exact query string. It's free, it's safe, and it never returns a wrong answer:

_exact = {}

def answer(query, call_llm):
    key = query.strip().lower()
    if key in _exact:
        return _exact[key]          # exact hit — zero risk
    result = call_llm(query)
    _exact[key] = result
    return result

This catches identical retries, refreshes, and duplicated requests. In production, back it with Redis and a TTL. Exact-match is your baseline; semantic caching is what you add when paraphrases are common enough that exact matching leaves money on the table.

The semantic-cache algorithm#

The whole thing is five steps:

  1. Embed the incoming query into a vector.
  2. Search your store for the nearest stored query vector.
  3. Compute similarity (cosine).
  4. If similarity > threshold, it's a hit — return the stored answer.
  5. Otherwise it's a miss — call the model, then store (embedding, answer).

That's it. Every design decision — which embedding model, what threshold, what store, what TTL — is tuning on top of this loop.

A real implementation#

Local embeddings via Sentence Transformers (no API key, runs on CPU), in-memory search with numpy. Normalizing the vectors makes cosine similarity a plain dot product.

# pip install sentence-transformers numpy
import time
import numpy as np
from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer("all-MiniLM-L6-v2")  # 384-dim, local

class SemanticCache:
    def __init__(self, threshold=0.85, ttl_seconds=3600):
        self.threshold = threshold        # cosine similarity, not distance
        self.ttl = ttl_seconds
        self.entries = []                 # (vector, answer, stored_at)

    def _embed(self, text):
        return embedder.encode(text, normalize_embeddings=True)

    def get(self, query):
        q = self._embed(query)
        now = time.time()
        self.entries = [e for e in self.entries if now - e[2] < self.ttl]  # drop expired
        best_sim, best_answer = 0.0, None
        for vec, answer, _ in self.entries:
            sim = float(np.dot(q, vec))   # normalized -> cosine similarity
            if sim > best_sim:
                best_sim, best_answer = sim, answer
        if best_sim >= self.threshold:
            return best_answer, best_sim  # HIT
        return None, best_sim             # MISS

    def set(self, query, answer):
        self.entries.append((self._embed(query), answer, time.time()))

cache = SemanticCache(threshold=0.85)

def answer(query, call_llm):
    hit, _ = cache.get(query)
    if hit is not None:
        return hit                        # served without a model call
    result = call_llm(query)              # miss -> pay for inference
    cache.set(query, result)
    return result

A linear scan is fine up to a few thousand entries. Past that, use a vector index. Here's the same logic on Redis vector search (RediSearch), which gives you persistence, cross-process sharing, and native KNN. Redis returns cosine distance, so similarity is 1 - distance:

# pip install redis numpy
import numpy as np, redis
from redis.commands.search.field import VectorField, TextField
from redis.commands.search.indexDefinition import IndexDefinition, IndexType
from redis.commands.search.query import Query

r = redis.Redis(host="localhost", port=6379)
DIM = 384

def ensure_index():
    try:
        r.ft("llm_cache").create_index(
            [TextField("answer"),
             VectorField("embedding", "HNSW",
                         {"TYPE": "FLOAT32", "DIM": DIM, "DISTANCE_METRIC": "COSINE"})],
            definition=IndexDefinition(prefix=["cache:"], index_type=IndexType.HASH),
        )
    except redis.ResponseError:
        pass  # index already exists

def to_bytes(vec):
    return np.asarray(vec, dtype=np.float32).tobytes()

def get(query_vec, threshold=0.85):
    q = (Query("*=>[KNN 1 @embedding $vec AS dist]")
         .sort_by("dist").return_fields("answer", "dist").dialect(2))
    res = r.ft("llm_cache").search(q, query_params={"vec": to_bytes(query_vec)})
    if not res.docs:
        return None
    similarity = 1 - float(res.docs[0].dist)      # COSINE distance -> similarity
    return res.docs[0].answer if similarity >= threshold else None

def set(query, query_vec, answer, ttl=3600):
    key = f"cache:{abs(hash(query))}"
    r.hset(key, mapping={"answer": answer, "embedding": to_bytes(query_vec)})
    r.expire(key, ttl)                            # TTL = invalidation

Don't want to wire it yourself? GPTCache packages this pattern — embedding function, vector store, and a SearchDistanceEvaluation threshold:

from gptcache import cache
from gptcache.embedding import Onnx
from gptcache.manager import CacheBase, VectorBase, get_data_manager
from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation

onnx = Onnx()
data_manager = get_data_manager(CacheBase("sqlite"),
                                VectorBase("faiss", dimension=onnx.dimension))
cache.init(embedding_func=onnx.to_embeddings,
           data_manager=data_manager,
           similarity_evaluation=SearchDistanceEvaluation())

Picking the threshold (and evaluating it)#

The threshold is the entire safety story. Too low, and unrelated queries collide — false positives, wrong answers. Too high, and you barely ever hit. Around 0.85 cosine similarity is a reasonable start, but don't ship a guess.

Build a small eval set: pairs that should hit (real paraphrases of the same intent) and pairs that should not (near-miss queries that differ in a load-bearing detail — a different date, a different account). Sweep the threshold and measure false-positive rate (wrong hits served), not just hit rate. A 70% hit rate that serves one customer another customer's balance is a bug, not a win. Weight toward precision: a miss costs you a model call; a false positive costs you trust.

TTL and invalidation#

Cached answers go stale. Set a TTL that matches how fast the underlying truth moves — hours for docs, minutes for anything with live data — as shown with expire() above. When source content changes (a doc edit, a policy update), you need targeted invalidation: tag cache entries by source document and evict by tag on update, or version your cache namespace and bump the version on each content release so the old cache is abandoned wholesale. Never cache responses that embed user-specific or session state unless the cache key includes that state.

When NOT to use it#

Skip semantic caching entirely when small wording changes are supposed to change the answer: anything with dates, names, IDs, prices, quantities, or per-user context; anything transactional or stateful; anything where being subtly wrong is expensive (medical, legal, financial). The failure mode is concrete and ugly — a support bot answering the wrong user's question because their two queries embedded close together. If your workload is repetitive reads over stable content, semantic caching is one of the highest-leverage cost cuts available. If it isn't, exact-match caching plus provider prompt caching gets you most of the savings with none of the correctness risk.