Every team that connects an agent to more than a handful of tools eventually hits the same wall — dozens of MCP servers, hundreds of function schemas, a context window that fills with API definitions before the model has read the user's request. The standard fix is retrieval: embed every tool's description, retrieve the top-K most relevant for each query, and hand the model a shortlist instead of the phone book. And the standard way to check that it's working is to measure recall, or Success@K — is the correct tool somewhere in the retrieved set?
It almost always is. Retrieve fifty tools out of three hundred and the right one is in the list well above ninety-nine percent of the time. That number is the reassurance the whole pattern runs on. It is also, according to two 2026 papers, measuring almost nothing.
The tool was in the list. So would a random one have been.#
The trap is chance. If you show the model a wide enough shortlist, the correct tool is present by luck — a randomly chosen set of K tools out of N contains it most of the time too. So a 99% recall doesn't tell you your retriever is good; it tells you your shortlist is big relative to the registry. The two are easy to confuse and the leaderboard rewards the confusion.
The 99% Success Paradox makes the gap literal with a chance-corrected metric called Bits-over-Random (BoR) — how much the shortlist actually narrows the choice versus a random set of the same size. On a standard retrieval task, BM25 and SPLADE both clear 99% Success@K=100 while scoring BoR ≈ 0: statistically indistinguishable from random selectivity. On MS MARCO, a thirteen-point recall gap between BM25 (85.7%) and the far fancier SimLM (98.7%) shrinks to 0.20 bits once you correct for chance, with all forty-one systems clustered against the same ceiling. The recall leaderboard was ranking noise.
Recall answers "could the model have found the right tool?" An agent lives or dies on "did it pick it?" Those two questions diverge exactly where you deploy — big registries, sprawling MCP fleets.
The collapse even has a threshold. Define the expected coverage ratio λ = K·R̄/N — roughly, how many relevant items you'd expect a random shortlist to sweep up. Once λ climbs past 3–5, the random baseline dominates and selectivity collapses. The paper points at production systems that present about 58 tools to a model: those sit at λ ≈ 4.0, deep in the collapse zone, where a theoretically perfect selector buys roughly 0.02 bits over guessing. You can build a beautiful retriever and, at that depth, it is decoration.
The surplus tools don't just fail to help. They cost you.#
If wide shortlists were merely uninformative, you could shrug and over-retrieve for safety. You can't, because the extra tools actively degrade the decision. In the same study, widening retrieval from K=10 to K=100 pushed Success@K to a perfect 100% — and dropped the model's downstream classification accuracy by 10–16% while multiplying token cost tenfold. More candidates in context means more near-duplicate descriptions competing for attention, the failure mode people have started calling context rot. You paid ten times the tokens to make the model worse and called it a recall win. This is the same economics behind dynamic tool management for long-running agents: what you load into context is a running tax, not a one-time convenience.
Fewer — but adaptive, not a magic number#
The companion paper, How Many Tools Should an LLM Agent See?, turns BoR into a policy: keep the shortlist only as deep as the query justifies. On BFCL's 370-tool suite, the adaptive policy presented about 7 tools on average yet matched the query-coverage of always showing 50 (90.3% vs 90.8%). And downstream — the part that matters — a shorter list made the model choose better: Claude Sonnet 4.6 picked the correct tool 93.1% of the time from the adaptive shortlist versus 87.1% when always handed a fixed five. Fewer tools, higher accuracy, a fraction of the tokens. (RAG-MCP reported the same direction more dramatically last year: retrieval-narrowed selection more than tripled tool-choice accuracy, 13.6% → 43.1%, while halving prompt tokens.)
The honest catch is that "fewer" is not a constant. On ToolBench's 3,251 tools, a fixed cut of five won higher aggregate coverage (64.7% vs 61.9%) but found nothing on the hard queries where the correct tool ranked sixth to twentieth — the adaptive policy recovered 16.7% of those by searching deeper only when it needed to. So the answer to "how many tools should my agent see" is not five, and not fifty. It is as few as this particular request requires, and no fewer — a function of the query, not a line in your config.
What to change on Monday#
Stop reporting Success@K as if it were a grade. It measures whether the ceiling is reachable, not whether your retriever earned its keep, and it saturates precisely in the large-registry regime where you deployed retrieval to begin with. Report something chance-corrected. Bind your shortlist depth to the query instead of a global top-K. And treat every tool description you drop into the context as a cost to be justified — because past the point the task needs, each one is buying you worse decisions at ten times the price, behind a number that says everything is fine.
