If you read one line: A tool-use score is three tests hiding under one word — did the model emit the right call, does that call actually run, and does the whole task complete reliably — and the reliability number (pass^k), not the headline average (pass^1), is the one that predicts your production agent.
Every model that wants to be your agent backend now ships a function-calling number: a rank on the Berkeley Function-Calling Leaderboard, a τ-bench score, a "tool-use accuracy" percentage in the launch post. Founders read one of them and pick the model that runs their agent loop. But tool-use benchmarks fail differently than the coding benchmarks we walked through last week — the traps are in what the metric quietly leaves out. Here's how to read them so the leaderboard informs your choice instead of making it for you.
Rule 1: "function-calling accuracy" is three different tests#
The word "accuracy" hides which question a benchmark actually answered.
- BFCL's AST categories compare the model's generated call to a reference call as an abstract syntax tree — right function name, right arguments, right types — using a static tree match. This is fast and scales to thousands of functions, but note what it does not do: it never runs the call. A syntactically perfect match can still throw the moment you execute it against a real API.
- BFCL's Execute categories actually run the function and check the result. Closer to reality — but still an isolated single call, not an agent carrying state.
- τ-bench asks a different question entirely: dropped into a multi-turn retail or airline conversation with an LLM-simulated user and a policy to follow, did the agent complete the task correctly? (Yao et al.)
A high AST-match score sitting on top of calls that would error on execution is a real and common gap. When a vendor writes "90% function-calling accuracy," your first question is which of these three it measured. If they don't say, treat it as AST-match — the most flattering and least predictive of the three.
Rule 2: pass^1 is what they show you; pass^k is what predicts production#
This is the number founders miss, and it's the most important one on the page.
The headline is almost always pass^1 — average success on a single attempt. The metric that predicts an unsupervised agent is pass^k: the probability the agent solves the same task on every one of k tries. τ-bench measured exactly this gap and it is brutal — an agent above 60% average success (pass^1) fell below 25% at pass^8 (Sierra). Run the same task eight times and it went a perfect eight-for-eight less than a quarter of the time.
What it means: doing the task once is not the job. An agent that books the wrong flight one time in four isn't a 75% agent, it's a liability. Read pass^1 as "can the model do this at all" and pass^k as "can I leave it running." If a benchmark reports only pass^1, its reliability is unknown — which is the same discipline behind why one tokens-per-second number lies to you.
Rule 3: single-turn is not multi-turn#
Most function-calling leaderboards started single-turn: one prompt, one set of tools, one correct call. Real agents don't work that way — they carry state across many turns, remember what they already did, and sometimes have to decide no tool applies. BFCL v3 added multi-turn, state-tracked categories (plus "missing function" and long-context cases) precisely because single-turn scores overstate how a model behaves inside a stateful loop.
Match the benchmark's shape to your agent's shape. If your agent makes one clean call per request, a single-turn score is fair. If it runs a ten-step workflow that has to stay coherent, a single-turn #1 tells you almost nothing about the step where it forgets the order ID it looked up three turns ago. This is the tool-use twin of matching a memory benchmark to your memory workload — the discipline we laid out in how to read an agent-memory benchmark.
Rule 4: a tool-use score is a model plus a schema plus a tool count#
The same weights post different tool-use scores depending on two things the leaderboard rarely foregrounds:
- How the tools are described. JSON Schema, TypeScript-style signatures, or a compact DSL are not equivalent for the model — the format shift alone can move the score, sometimes more than the gap between two models. (It's the same surface as JSON mode vs. function calling vs. constrained decoding.)
- How many tools are in scope. A model that's near-perfect choosing among 6 functions can degrade sharply picking the right one from 60 — or correctly deciding none of them fits.
Vendors report the format and tool-set that flatter their model. You run your schemas and your tool count. That's why a BFCL rank rarely survives contact with your real tool belt, and why the deciding question is never "which model is #1" but "which model is #1 with tools shaped like mine — and is that even the same as MCP vs. plain function calling for my stack."
What to actually do#
- Match the shape. Single well-specified calls → a single-turn score is fair. Long stateful workflow → only a multi-turn benchmark (BFCL v3, τ-bench) is honest.
- Read pass^k, not pass^1. If reliability isn't reported, it's unknown — and unknown reliability is a red flag for anything unsupervised.
- Know which "accuracy" you're reading. AST-match ("can it emit the call") is not execution ("does the call work") is not task completion ("did the job get done").
- Run a private eval with your real tools. Pull 20–50 real tasks, wire in your tool schemas and your tool count, and run each finalist several times — recording both pass^1 and pass^k. It takes an afternoon and measures the only thing BFCL and τ-bench can't: how the model behaves with your tools, in your loop. The private-eval playbook we wrote for coding models ports directly to tool use.
Public tool-use benchmarks are a filter for deciding who gets into that eval. They were never meant to be the decision — and once you read them as which test, which metric, which schema, the weekly "best at function calling" headlines stop being confusing and start being useful. The alternative is reading a benchmark exactly the way it's built to be misread.
