The short version: a RAG benchmark number is a promise about one labeled dataset — not about your corpus, your queries, or your users. A model can top MTEB, post a great recall@k, and score 0.9 on RAGAS faithfulness while your users still get wrong answers. Reading the number correctly means knowing exactly which promise it makes and which it quietly skips. Here's the field guide, one metric at a time.
Retrieval metrics measure the retriever — on labeled data you probably don't have#
Recall@k, precision@k, MRR, and nDCG all score the same thing: did the right documents come back, and in what order. Two facts govern how to read them:
- They need ground-truth relevance labels. Every one of these metrics is computed against per-query judgments of which documents are relevant. Public benchmarks like BEIR ship those labels; your production corpus almost never does. A recall@k you compute without real labels is measuring your guesses, not your retriever.
- Recall and precision are order-blind. A relevant hit at rank 1 scores identically to one at rank 4 — but position is exactly what the generator attends to. If you care about ranking (you do), read nDCG or MRR alongside them, not recall@k alone.
So recall@k is a necessary retrieval signal, never a sufficient product signal. It tells you the chunk was in the pile; it says nothing about whether the model used it.
RAGAS splits retrieval from generation — on purpose#
The most common reading error is treating one end-to-end score as "how good is my RAG." RAGAS deliberately refuses to: it separates context precision / context recall (which grade retrieval) from faithfulness / answer relevancy (which grade the generated answer). That separation is the whole point — because the two halves disagree constantly:
- You can retrieve the perfect chunk and still generate an unsupported answer (retrieval good, faithfulness bad).
- You can retrieve mostly noise and still land a correct answer from parametric knowledge (retrieval bad, answer good).
If you only watch a blended score, you can't tell which half to fix. Read the retrieval metrics and the generation metrics as two separate dials.
The catch with RAGAS: several of those dials are an LLM's opinion#
Faithfulness and answer relevancy are typically computed by an LLM judge, and LLM judges carry documented biases — position (which option came first), verbosity (longer reads as better), self-preference (favoring output from the same model family), and plain prompt sensitivity (small wording changes flip the verdict). None of that makes RAGAS useless; it makes it a relative instrument. Use it to catch regressions and rank candidates, calibrate it against a human-labeled slice, and never quote a lone faithfulness number as if it were accuracy.
Embedding leaderboards: rank is not destiny#
When you pick an embedding model off MTEB or BEIR, remember what those benchmarks are built to measure. BEIR is explicitly a zero-shot, out-of-distribution suite — 18 datasets chosen to test whether retrieval generalizes, not whether it's great in one domain. And the MTEB paper's own headline finding is that no single model is best across tasks; a leaderboard rank is a summary, not a verdict.
The cleanest cautionary result: DisastIR measured the rank correlation between MTEB and a disaster-management retrieval benchmark and got 0.225 — effectively no correlation. Models that topped MTEB underperformed in-domain, and models that lagged on MTEB led in-domain.
Leaderboard rank tells you which models are worth trying. It does not tell you which one wins on your documents.
How to actually read one — the checklist#
- Scope the promise. Ask what dataset the number is on and how relevance was labeled. A number without a named, labeled dataset under it is decoration.
- Split the pipeline. Never accept one end-to-end score; demand a retrieval number and an answer number separately.
- Discount the judge. If a score is LLM-judged, treat it as relative and calibrate against humans on a small slice.
- Distrust cross-domain transfer. A leaderboard rank is a shortlist for your own eval, not a substitute for it.
- Build the 50–100. A small set of real queries and real documents with human labels beats every public benchmark for deciding what to ship — the same discipline that separates a real retrieval eval from a vanity metric.
The number that predicts your production is the one you measured on your production. Everything on a leaderboard is a hypothesis until then. And once your retrieval is honest, the next question is whether it even needs to fire on every query — which is exactly what Self-RAG, CRAG, and Adaptive-RAG argue about.
