Short version: Object-storage vector stores — Turbopuffer, embedded LanceDB, Chroma Cloud — are cheap because the index lives on S3/GCS, not in RAM. The bill for that is the cold read: the first query to an uncached namespace has to fetch from object storage and stalls, while a warm query is fast. Turbopuffer's own numbers put that gap at roughly 500ms cold vs. 14ms warm at 1M vectors. You don't fix this by paying for a pricier store; you fix it by hiding the cold read — warming the namespace before your agent's latency-sensitive query needs it. Here are the five strategies that actually move the number.

If you're still choosing a store, we compared two of these head-to-head in LanceDB vs. Turbopuffer for agent retrieval. This piece is what you do after you've picked one and the first query is slow.

Why the cold read exists#

The thing that makes these stores cheap is the same thing that makes them cold. Instead of holding every vector in RAM — the expensive, memory-resident model — they keep the index durably on object storage, with NVMe SSD and RAM as a cache tier. Storage costs collapse (both Turbopuffer and Chroma quote around $0.02/GB-month), but the first query to a namespace that isn't cached pays for a round-trip to S3.

The thing that makes these stores cheap is the same thing that makes them cold. Don't pay it away — schedule it away.

For a background or async agent, a 500ms cold query is invisible. For a synchronous tool call the user is watching, it's a stall in the loop. So the goal is never "make the cold read fast" — it's "make sure the cold read already happened by the time it matters."

1. Pre-warm the moment you know the namespace#

This is the highest-leverage move. The instant you can predict which namespace an agent will hit, warm it — don't wait for the query that needs to be fast.

The trick is that you almost always know early. A per-tenant agent knows the tenant at login. A session knows its namespace at session start. A tool-using agent knows the target when the tool is selected, one step before the retrieval call. Fire a cheap warming query — or the vendor's explicit warm hint, if it has one — at that moment:

# On session start, warm the namespace off the critical path.
# A tiny top-k query pulls the index into cache; the result is discarded.
async def warm(store, namespace):
    try:
        await store.query(namespace=namespace, vector=ZERO_VEC, top_k=1)
    except Exception:
        pass  # warming is best-effort; never fail the session on it

# ... later, the real interactive query lands warm:
hits = await store.query(namespace=namespace, vector=embedding, top_k=8)

Turbopuffer exposes cache-warming hints for exactly this — warm all of a user's namespaces at session start so the first real query is warm. The pattern is portable even where the hint isn't: a discarded top-1 query does the same job.

2. Keep a hot set resident#

Traffic is almost always skewed — a small fraction of tenants drive most queries. Identify that hot set and keep its namespaces resident in cache continuously, rather than letting them age out between requests and re-paying the cold read each time.

Chroma Cloud does this tiering for you, reading through RAM → SSD → object storage hottest-first. With Turbopuffer or an embedded LanceDB deployment you manage residency more explicitly — a periodic keep-warm query per hot namespace, or holding the working set in your own process. You pay to keep idle-but-frequent tenants warm; for interactive agents with repeat users, that's a bargain against a cold spike on every return visit.

3. Co-locate compute with the bucket#

This one is free and people forget it: run your agent in the same region as the object-storage bucket. A cross-region cold read adds network latency to an already-slow path and can add egress cost on top. Same-region deployment shrinks the cold read itself, before any caching. If your store lets you pick the bucket region (Turbopuffer, LanceDB Enterprise, Chroma Cloud all do), match it to where your agents run.

4. Isolate async work from the live path#

Mixed workloads sabotage each other's cache. A nightly re-index or a batch eval that cold-reads a namespace your live agent depends on will evict warm data and leave the next interactive query cold. Split the paths: route cold-tolerant work — bulk indexing, offline evals, enrichment — off the interactive budget, and reserve the warm cache for synchronous tool calls. If a batch job has to touch a live namespace, warm it back when the job finishes.

5. Scan fewer bytes#

The cold read's cost is proportional to the data it moves, so move less. Tighter metadata filters cut the candidate set before the vector scan; a quantized or compressed index (Turbopuffer's SPFresh, LanceDB's IVF-PQ product quantization) shrinks what a cold read has to pull. Both trade against recall, so tune them against your own eval set — but on large namespaces, fewer bytes scanned is a direct cut to cold latency and to per-query cost.

Measure cold and warm separately#

The mistake that hides all of this is reporting one averaged latency. Blend a 14ms warm query with a 500ms cold one and you get a middling number that describes neither — and it moves unpredictably as your cache hit rate drifts. Instrument the first query to a namespace distinctly from the rest, and track p50 and p99 for cold and warm independently. Then load a representative slice of your own corpus, replay real agent traffic, and watch whether your warming actually removed the spike.

Every latency figure here is vendor self-reported, so treat them as the shape of the problem, not your numbers. The shape, though, is universal to this class of store: cheap because it's cold, fast because you warmed it first.