Everyone's first instinct for scaling a database is the one that fails hardest here. You reach for the relational playbook: pick a shard key, hash it, route each write to a shard and each read straight back to it. It works for orders and users because the thing you query by is the thing you sharded by. It does not work for vectors, and the reason is worth internalizing before you provision a single extra node.

A vector's nearest neighbors are defined by geometry, not by any key. Two rows with adjacent IDs can sit at opposite ends of the embedding space; two semantically identical documents can have IDs that share nothing. There is no key you can hash such that "close in key space" means "close in vector space." So the relational move — route the query to the one shard that owns it — has no equivalent. You don't know which shard owns the answer until you've searched, and the whole point of scaling was to avoid searching everything.

That single fact generates the entire design space. As the distributed-systems literature frames it, every billion-scale vector database is answering three questions at once: how do we split the vectors across machines, how do we route each query to the right machines, and how do we merge the partial results correctly. Split, route, aggregate. The interesting decisions all live in route.

First, delay the problem#

Before distributing anything, know your ceiling. A billion 1536-dimensional float32 vectors is about 6 TB of raw data — 1e9 × 1536 × 4 bytes — before you add the HNSW graph, which is not free. That won't fit in one machine's memory, which is why billion-scale forces distribution. But hundreds of millions often can be delayed, and delay is cheaper than a cluster.

Two levers buy runway on a single node. Quantization compresses vectors by splitting them into sub-vectors and clustering each — product quantization can cut memory by roughly 90%, at a recall cost you should measure rather than assume. Disk-based indexes like DiskANN keep most of the index on SSD and trade RAM for a few extra reads per query. If a compressed or disk-backed index fits your latency budget, you have just avoided the hardest class of bug in this whole space: a distributed one.

When you do distribute, the routing decision splits into two honest options, and picking between them is the scaling problem.

Scatter-gather shards randomly (or by hash), then sends every query to every shard. Each shard returns its local top-k; a coordinator merges them into a global top-k. Recall is as high as it gets — nothing was skipped — but every node does work on every query, so latency and cost climb with shard count. You bought parallelism and paid for it with fan-out.

Centroid routing clusters the whole dataset into coarse regions, assigns each region to a shard, and sends a query only to the shards whose centroids are nearest it. Now a query touches a handful of shards instead of all of them, which is dramatically cheaper. The catch is exactly the geometry problem again: if a true neighbor happened to land just across a cluster boundary, in a shard you didn't query, you never see it. Recall becomes a function of how many shards you're willing to probe.

You do not scale a vector database by storing more vectors. You scale it by deciding how many shards each query is allowed to skip — and recall is the bill for skipping them.

This is why Milvus, Qdrant, and Weaviate look different under the hood even when their APIs feel similar. Milvus separates compute and storage and shards a collection into segments across index nodes, with proxies routing and aggregating; Qdrant does automatic sharding and replication with Raft-based consensus and configurable replication factors; Weaviate builds its segment index in memory per shard. They are different answers to split, route, aggregate — not different products solving different problems. (If you're still choosing one, that's a separate comparison.)

Recall, latency, cost — pick two, honestly#

The uncomfortable truth is that distributed vector search is an optimization problem over a triangle: recall, latency, cost. Wider search (more shards probed, larger ef, more replicas) buys recall and spends latency and money. Tighter routing buys latency and money and spends recall. There is no configuration that maximizes all three, and any vendor claiming otherwise is quietly holding one of them fixed.

Two consequences fall out of that:

The teams that scale vector search well aren't the ones with the most nodes. They're the ones who know, for every query, which shards they are choosing not to search — and exactly what recall that choice costs them.