Short version: Before you can trust any recall@k or MRR number, you need labeled (question -> relevant chunk) pairs, and almost nobody built them. Generate them: loop over the chunks you already index, and for each chunk ask a cheap model (Claude Haiku) to write one natural question that chunk answers. The (question, chunk_id) pair is your relevance label — save the whole set to a JSONL file so you pay for generation exactly once. Then embed the corpus and the questions with sentence-transformers, rank chunks per question by cosine similarity, and compute two numbers in a dozen lines of numpy: recall@k (did the gold chunk make the top-k?) and MRR (the mean of 1 / rank of the gold chunk). That is a complete, offline, repeatable retriever benchmark you can run before every embedding-model swap or chunking change.

When to use this: reach for it the moment you're about to change something upstream of the LLM — a new embedding model, a different chunk size, hybrid search, a reranker — and want to know whether retrieval got better or worse without eyeballing answers. It is a relative instrument: great for comparing configuration A to configuration B on the same synthetic set, weaker as an absolute quality claim (see the caveats). If you already have real query logs with known-good chunks, use those instead — this is the bootstrap for when you don't.

Step 0: install and set a key#

Three libraries. The generation step needs an Anthropic key; everything after it runs offline.

pip install sentence-transformers anthropic numpy
export ANTHROPIC_API_KEY=sk-ant-...

Step 1: generate the golden set from your own chunks#

This is the part people skip because it feels like it needs a labeling team. It doesn't — it needs one model call per chunk. Replace the toy CORPUS with your actual indexed chunks (in production, load them from your vector store). For each chunk, Claude writes a single realistic question, and we cache the results so a second run costs nothing.

import json, os, anthropic

# In production, load these from your vector store instead.
CORPUS = [
    "Mandate error E4012 fires when an agent's spend cap is exceeded before settlement clears.",
    "Agentic checkout registers an agent identity tied to a sponsor's verified email address.",
    "Webhooks emit mandate.charge.pending with a 30-second veto window before the charge captures.",
    "Refunds above the per-agent daily cap require a human approver in the dashboard.",
    "The retrieval service embeds chunks with all-MiniLM-L6-v2 and stores them in sqlite-vec.",
    # ... hundreds more in a real corpus
]

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY

PROMPT = (
    "Write exactly one natural question a real user might type that is answered "
    "by the passage below. Do NOT quote the passage or reuse its distinctive "
    "wording — paraphrase as a real user would. Output only the question.\n\n"
    "Passage:\n{chunk}"
)

def gen_question(chunk: str) -> str:
    msg = client.messages.create(
        model="claude-haiku-4-5",   # cheap + fast: right tool for bulk generation
        max_tokens=100,
        messages=[{"role": "user", "content": PROMPT.format(chunk=chunk)}],
    )
    return next(b.text for b in msg.content if b.type == "text").strip()

def build_goldenset(path: str = "goldenset.jsonl") -> list[dict]:
    if os.path.exists(path):                       # idempotent: don't pay twice
        return [json.loads(l) for l in open(path)]
    rows = []
    for chunk_id, chunk in enumerate(CORPUS):
        rows.append({"question": gen_question(chunk), "gold_id": chunk_id})
    with open(path, "w") as f:
        for r in rows:
            f.write(json.dumps(r) + "\n")
    return rows

goldenset = build_goldenset()

Two deliberate choices here. Haiku, not Opus — writing a question from a passage is a simple, high-volume task; the cheap model is the correct engineering call, and swapping to a stronger model for trickier corpora is a one-line change. gold_id is just the chunk's index — one relevant chunk per question keeps the labels unambiguous and makes recall@k equal to hit@k, which is exactly what you want as a floor.

Step 2: embed the corpus and the questions#

all-MiniLM-L6-v2 is a small, fast, widely-used model that returns 384-dimensional unit vectors, so a dot product of two embeddings is their cosine similarity. Encode the corpus once and the questions once.

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

corpus_emb = model.encode(CORPUS, normalize_embeddings=True)          # (N, 384)
questions  = [r["question"] for r in goldenset]
gold_ids   = [r["gold_id"]  for r in goldenset]
q_emb      = model.encode(questions, normalize_embeddings=True)       # (Q, 384)

Step 3: rank, then compute recall@k and MRR#

For each question, score every chunk by cosine similarity, sort chunks best-first with argsort, and find where the gold chunk landed. Recall@k is 1 if that rank is within k; MRR is 1 / rank. Both are averaged over all questions.

def evaluate(corpus_emb, q_emb, gold_ids, ks=(1, 3, 5, 10)):
    recall = {k: [] for k in ks}
    rr = []
    for i, q in enumerate(q_emb):
        scores = corpus_emb @ q                 # cosine sims (unit vectors)
        ranked = np.argsort(-scores)            # chunk ids, best first
        rank = int(np.where(ranked == gold_ids[i])[0][0]) + 1   # 1-indexed
        rr.append(1.0 / rank)
        for k in ks:
            recall[k].append(1.0 if rank <= k else 0.0)
    out = {f"recall@{k}": round(float(np.mean(v)), 3) for k, v in recall.items()}
    out["mrr"] = round(float(np.mean(rr)), 3)
    return out

print(evaluate(corpus_emb, q_emb, gold_ids))

Run the whole file and you'll see something like this (numbers depend entirely on your corpus and are only meaningful in comparison to another run):

{'recall@1': 0.71, 'recall@3': 0.89, 'recall@5': 0.94, 'recall@10': 0.98, 'mrr': 0.80}

That's the instrument. Change the embedding model to a bigger one, or re-chunk, re-run, and watch the curve move. If recall@5 climbs from 0.94 to 0.97, the change helped the half of the pipeline the generator can't fix — a missed chunk is the unrecoverable failure no downstream LLM can undo.

Recall@k answers a yes/no question — is the evidence even in the window? — and it is the ceiling on everything the generator can do. Optimize a reranker all you like; it can only reorder chunks retrieval already fetched.

The caveat that keeps you honest#

Synthetic questions have one failure mode you must design around: if the model echoes the chunk's exact vocabulary, the question and the passage share tokens, and any retriever scores near-perfect — you've measured your prompt, not your retriever. Three defenses, all in the recipe above: the prompt explicitly forbids quoting and asks for paraphrase; you use a corpus of hundreds of chunks so a top-5 hit is non-trivial; and you read the numbers as a relative baseline for comparing configurations, never as an absolute "our retrieval is 94% good." The instant you have real user queries with known-good chunks, fold them in — synthetic labels are the scaffold, not the building.

For the precise definitions and when each metric earns its keep, see recall@k vs MRR vs nDCG; for the stage after recall — fusing keyword and dense hits — see hybrid search with reciprocal rank fusion.

Ship checklist#