You did everything the guides said. You rewrote the tool descriptions as prompts, not API docs, you stopped shipping your whole API as tools, you tuned the "when to use this" lines. The agent feels better. But "feels better" is not a number, and the next prompt tweak — or the next model upgrade — can silently undo all of it and you won't know until a user files a bug.

The fix is a tool-selection eval: a small labeled test set that measures exactly one thing — given a prompt, does your agent reach for the right tool. It's the cheapest high-value eval you can build, and almost nobody builds it.

The one number that matters#

Tool selection is the highest-frequency agent failure, and it's measurable. The RAG-MCP stress test made this concrete: with every tool injected into the prompt, tool-selection accuracy was 13.62%; retrieving only the relevant tools lifted it to 43.13% — 3× better from selection changes alone. Your agent has a number like that right now. You just aren't looking at it.

A selection eval grades the decision, not the result. So you stub the tools out: run the model, capture the first tool it wants to call, compare that to the label, throw away everything after. One model call per case. No live services, no flakiness, no environment to stand up.

Three parts#

1. A labeled set — small and deliberate. 20–50 cases beats 500 random ones. Cover the three things that actually break:

{"prompt": "where's my order from last tuesday?", "expected": "search_orders"}
{"prompt": "do you sell the black one in a large?", "expected": "search_products"}
{"prompt": "what's your return window?", "expected": "none"}

Grow this file from production misses — every wrong-tool call you see becomes a case. The eval should be a record of real failures, not imagined ones.

2. A runner that captures the decision. Force a choice with tool_choice: "auto", read the first tool_use block, and never execute it:

def picked_tool(client, prompt, tools):
    resp = client.messages.create(
        model="claude-opus-5", max_tokens=1024, tools=tools,
        tool_choice={"type": "auto"},
        messages=[{"role": "user", "content": prompt}],
    )
    for block in resp.content:
        if block.type == "tool_use":
            return block.name        # the decision — stop here
    return "none"                    # answered directly, called nothing

cases = [json.loads(l) for l in open("selection.jsonl")]
results = [(c["expected"], picked_tool(client, c["prompt"], TOOLS)) for c in cases]
acc = sum(e == g for e, g in results) / len(results)
print(f"selection accuracy: {acc:.1%}")

That's the whole harness. (A dedicated eval runner like promptfoo can assert on tool calls if you'd rather not hand-roll it, but the 15 lines above are enough to start today.)

**3. A confusion matrix — the part that tells you why.** Accuracy alone says you're wrong; the matrix says which tool it picks instead, and that names the fix:

from collections import Counter
m = Counter((e, g) for e, g in results if e != g)
for (expected, got), n in m.most_common():
    print(f"{n:>3}  wanted {expected:<16} got {got}")
  6  wanted search_orders   got search_products
  3  wanted none            got get_faq

Six cases confusing search_orders with search_products isn't a mystery — the two descriptions overlap. Tighten the "when NOT to use this" line on both, or make the parameters expressive enough that only one fits (well-named enums carry intent). Three cases where the agent should have stayed silent and instead called get_faq means that tool's trigger is too eager. Fix that one pair, rerun, watch the cell cool — and confirm you didn't light up a new one.

Then gate on it#

A selection eval you run once is a curiosity. A selection eval you run on every prompt edit and every model swap is a guardrail. Wire it into CI next to your other checks so a "harmless" wording change that quietly drops selection accuracy from 90% to 70% fails the build instead of shipping.

This is the eval to build first — before the full trajectory evals that grade arguments, recovery, and task outcome. Most "the agent did the wrong thing" bugs are wrong-tool bugs in disguise, and this catches them for one model call apiece. Stop guessing which tool your agent picks. Measure it, and make the number go up.