Here's a bill nobody audits: the text you embed over and over. You rebuild your RAG index and re-embed 100,000 chunks, 95,000 of which are byte-for-byte identical to last week's. Your chunker overlaps windows and embeds the same overlapping sentences twice. Ten thousand documents share the same legal footer, and you embed it ten thousand times. None of that is a similarity problem or a threshold problem. It's the same text, and the same text always produces the same vector.

An embedding cache fixes it in about fifteen lines, and it is the optimization people mean to reach for when they reach for a "semantic cache" and accidentally take on a correctness risk they didn't need. Ship this one first.

Embedding cache ≠ semantic cache#

The two get conflated constantly, so pin the distinction down before you build:

Different layer, different payload, different risk. The embedding cache is the free lunch; take it before you touch the one with sharp edges.

The whole thing, in one function#

The key is a content hash of the text — plus the model name. That second part is the only place people go wrong, and it's the whole safety story (more below).

import hashlib, json, sqlite3

MODEL = "text-embedding-3-small"   # name AND version live in the key

db = sqlite3.connect("emb_cache.db")
db.execute("CREATE TABLE IF NOT EXISTS emb (k TEXT PRIMARY KEY, v BLOB)")

def _key(text: str) -> str:
    # model first, so a model change is automatically a cache miss
    return hashlib.sha256(f"{MODEL}\n{text}".encode()).hexdigest()

def embed_cached(text: str) -> list[float]:
    k = _key(text)
    row = db.execute("SELECT v FROM emb WHERE k=?", (k,)).fetchone()
    if row:
        return json.loads(row[0])            # cache hit — no API call
    vec = embed_api(text)                    # your embedding provider call
    db.execute("INSERT OR REPLACE INTO emb VALUES (?,?)", (k, json.dumps(vec)))
    db.commit()
    return vec

Swap SQLite for a dict in a script, or Redis for anything shared across workers. The value is just the vector; you do not need a vector database for the cache — the vector DB is downstream, where you search embeddings by similarity. The cache is a plain key-value lookup by exact text.

The one rule: the model name is part of the key#

Embeddings from different models live in different geometric spaces and are not comparable. Mix vectors from text-embedding-3-small and its successor in one index and your nearest-neighbor search returns confident nonsense — with no error to warn you. So the cache key must be hash(model + "\n" + text), never hash(text) alone.

Do that and a model upgrade becomes an automatic cache miss: every lookup under the new model name misses, you re-embed cleanly, and the old vectors simply age out. That is also your entire invalidation strategy.

Cache invalidation is famously hard. Embeddings are the exception: the input is immutable, so a (model, text) pair maps to one vector forever. You only ever evict for space — or when you change models.

Where the money actually is#

Point the cache at the three places re-embedding hides:

  1. Full re-indexes. You rebuild the index; almost none of the source text changed. With a cache, a "rebuild" only pays for the deltas.
  2. Chunk overlap. Overlapping windows share text by design. Cache it once, reuse it for every window that contains it.
  3. Boilerplate. Nav bars, license headers, legal footers — the same block across thousands of documents. Embed it a single time.

In all three, most of what you're "embedding" is text you already embedded. The cache turns that from a recurring line item into a one-time cost.

Ship it, then reach for the risky layer#

An embedding cache is the rare optimization with no downside: deterministic, trivially invalidated, no threshold to tune, no wrong answers to chase. Add it in an afternoon, watch your embedding spend drop by whatever fraction of your text is repeated — which, for most indexing workloads, is most of it.

Then, if your traffic is repetitive question-answering rather than indexing, layer a semantic cache on top to skip whole model calls — carefully, with a tuned threshold, because that one can be wrong. But that's the second move. This is the first.