You added a reranker because a blog post said it would fix your RAG, and one of two things happened: quality barely moved, or your p95 latency doubled. Both are almost always the same bug — two numbers set by copying someone else's defaults instead of measuring your own corpus. Those numbers are fetch-k (how many candidates you retrieve before reranking) and keep-n (how many you keep after). Get them right and a reranker is the cheapest precision upgrade in RAG. Get them wrong and it's latency you paid for and didn't get anything back.

What a reranker is actually doing#

Retrieval and reranking are a two-stage funnel. Your vector search is a bi-encoder: it embeds the query and the documents separately and compares vectors, which is fast and cheap and lets you search millions of chunks — but it never sees the query and a passage together, so its ordering is approximate. A reranker is a cross-encoder: it takes the (query, passage) pair jointly and scores true relevance, which is far more accurate and far more expensive per item. So you use each for what it's good at: retrieval casts a wide cheap net, the reranker reorders the catch.

That division is the whole reason the two numbers exist. Fetch-k is how wide the cheap net is; keep-n is how much of the reordered catch you keep.

Fetch-k: retrieve wide, because the reranker can't conjure a missing passage#

Fetch-k is your recall ceiling. The reranker can only promote a passage that retrieval actually surfaced — it reorders the candidate set, it doesn't reach back into the index for something you didn't fetch. So if the passage that answers the question isn't in your top-k, no reranker on earth recovers it. That argues for fetching wide: a common working range is 50–100 candidates.

But wide isn't free, and there's a real ceiling:

So don't just crank fetch-k. Measure recall@fetch-k on a labeled sample — the fraction of questions whose supporting passage appears anywhere in the retrieved set — and raise fetch-k only until that recall plateaus. Once it stops climbing, more candidates just cost you latency and add noise. If recall is low even at k=100, your problem is retrieval, not ranking: fix embeddings, chunking, or add hybrid BM25 + dense retrieval before you touch the reranker. A reranker amplifies good retrieval; it cannot rescue bad recall.

Keep-n: keep narrow, because more context can hurt#

Keep-n is your answer-quality and prompt-cost dial — the top_n you pass the reranker, the passages that actually reach the model. The typical range is 5–10, and the failure modes point in opposite directions:

Tune it by sweeping n ∈ {3, 5, 8, 10} and measuring answer faithfulness (does the answer stay grounded in the retrieved passages?). Pick the smallest n where faithfulness stops improving — that's the point where you're keeping every passage that helps and none that hurt, at the lowest token cost.

The score threshold: let counts flex, and let "no answer" mean nothing#

A fixed keep-n has a blind spot: it always returns n passages, even when none are relevant. For a query your corpus can't answer, top-n dutifully hands the model n irrelevant chunks, and the model dutifully hallucinates an answer from them. That's the worst failure mode in RAG, and it's built into fixed-n.

The fix is a score threshold: keep only passages whose reranker relevance score clears a bar, and drop the rest regardless of count. Now an easy query might keep 2 passages, a hard one 8 — and a query with no good match keeps zero, which is exactly what lets your app answer "I don't have that information" instead of confabulating. In code (Cohere's Rerank shown, but every cross-encoder returns comparable scores):

reranked = cohere.rerank(
    model="rerank-v3.5",
    query=query,
    documents=candidates,     # your fetch-k=50–100 retrieved passages
    top_n=10,                 # keep-n ceiling
)
KEEP = [r for r in reranked.results if r.relevance_score >= 0.30]  # the threshold
context = [candidates[r.index] for r in KEEP]   # may be 0 passages — that's the point

Two cautions. Reranker scores are not calibrated across models or even query types, so 0.30 is not a universal number — set it on your data, and re-tune it whenever you swap rerankers. And keep a small floor if your product must always answer something; the threshold is for products that would rather say "I don't know" than guess.

Set all three from data, not from a blog#

Here's the calibration loop, and it's the same discipline as reading any RAG benchmark — trust your numbers, on your corpus:

  1. Build a small labeled set: queries with their known supporting passages.
  2. Sweep fetch-k (20 → 100); pick the smallest k where recall@k plateaus.
  3. Sweep keep-n (3 → 10); pick the smallest n where faithfulness plateaus.
  4. If query difficulty varies or "no answer" is a real case, add a score threshold and set it to the value that cuts hallucinations on unanswerable queries without dropping good passages.
  5. Watch p95 latency the whole time — it's the budget fetch-k spends.

The right values are dataset-specific; anyone who gives you universal numbers is guessing. What's universal is the shape: fetch wide enough to catch the answer, rerank, keep narrow enough to stay grounded, and threshold so counts flex with the query.

Where this fits#

Reranking is one stage of a retrieval pipeline, and it only pays off if the stages around it are sound. If you haven't chosen a reranker yet, start with the best-reranker-for-rag rundown and know how to evaluate one on your own data before you trust its scores. If your recall is the real problem, that traces back to your embedding model and your vector store — fix those first, because a reranker's two numbers can only ever reorder what those two stages hand it.