The short version: an agent's real cost is tokens-per-task × price × a retry penalty, and only one of those three — price — is on the pricing page. So when a cheaper model ships and the group chat says "switch," the rate card can't tell you whether your bill goes down. Twenty minutes can: freeze a handful of your real tasks, run them through both models, and compare completed-task cost, not dollars-per-million-tokens. Sometimes the cheaper sticker is the more expensive model. Here's the test.

The three-factor cost, and why two are hidden#

A chat completion is one prompt in, one answer out — there, price per token basically is the cost. An agent is nothing like that. It reads files, calls tools, inspects output, retries, and self-corrects, so a single task can burn one to several million tokens. Your bill is:

task_cost = tokens_per_task × price_per_token

and if a task fails, you also pay the re-run. Price is the visible factor. Tokens-per-task lives in your traces. The retry penalty lives in your pass/fail rate. A model with a lower sticker but a higher token count, or a worse pass rate, quietly costs you more — and the rate card will never show it.

The test#

  1. Freeze 15–20 real tasks. Pull them from your agent's actual history — recent jobs you can confidently label pass or fail against a known-good outcome. Not toy prompts; you want the retry-and-recover behavior where models actually diverge.
  2. Run each through the identical scaffold on the old model and the candidate. Same tools, same system prompt, same retry budget. Change only the model string — you're isolating the model, not redesigning the agent.
  3. Record two numbers per task: did it pass, and total input + output tokens burned. Every major SDK returns per-call usage metadata (Anthropic, OpenAI, and Google all expose input/output counts on the response) — sum it across the task's turns.
  4. Rank on completed-task cost:
completed_task_cost = (input_tokens  × input_price
                     + output_tokens × output_price)
                     / pass_rate

The lowest completed-task cost at an acceptable pass rate wins.

The term everyone drops: pass rate#

Dividing by pass rate is what separates this from a naive token comparison, because a failed task isn't free — it costs the tokens it burned plus the human time to notice and the tokens of the re-run. Amortize the failures across the successes and the picture flips:

The "expensive" model is actually cheaper, and reliably so. Drop the pass-rate term and you'll pick the flaky one every time — then pay for it in re-runs and lost trust. This is the general form of the argument in why cheap models fail silently in long agent loops: the failures don't announce themselves on the invoice, but you pay for them.

Worked example: Gemini 3.6 Flash vs 3.5 Flash#

Google's 3.6 Flash is the textbook case, because its two cost cuts sit in two different places. On the rate card, one number moved — output from $9.00 to $7.50 per million tokens, with input unchanged at $1.50. On a realistic input-heavy agent task that's worth only about 5%, since output is a thin slice of the bill.

But Google also claims 3.6 Flash finishes tasks in fewer tokens — "up to 65% cheaper" on long-horizon agentic work (VentureBeat). That saving is tokens-per-task, and it never appears on the pricing page. Which number you get — 5% or 65% — depends entirely on how much churn your old setup had. The test settles it: because the input price and API are unchanged, migrating a running 3.5 Flash agent is nearly free, so you run the A/B to size the win, not to decide whether to take it. We priced the models themselves in Gemini 3.6 Flash: do the math before you switch; this is how you do that math on your own workload.

The rule#

Never migrate on a price table. A sticker cut is a hypothesis about your bill, not a measurement — and the two factors that decide it, tokens-per-task and retry penalty, are both invisible on the rate card. The same test prices any change that moves the token count, not just a model swap: trimming a bloated prompt, adding caching, rewriting tool schemas, switching frameworks. That last one is exactly how Deep Agents v0.7 cut input tokens 65% with no price change at all — same model, same rate card, different loop. Twenty minutes of A/B on your own tasks turns every one of these from a group-chat opinion into a number. Run it before you switch.