Your model's pricing page shows a rate. Your invoice shows a result. The gap between them is where founders lose money on agents — and neither tokens per second (a speed number) nor price per token (a rate) tells you how wide it is.

The only number that lands on an invoice is cost per completed task: total dollars spent to finish one task, divided by the tasks that actually succeeded, counting every token from every attempt. This walkthrough gives you a ~60-line harness that logs it, and the three rules that keep it honest.

If you read one line: a model that's 3x cheaper per token but succeeds 60% of the time on the first try is often more expensive per completed task than a pricier model that finishes first attempt. You can't see that on a pricing page. You can see it in about an hour of measurement.

The three rules that make the number honest#

  1. Count every attempt, including failures. A failed attempt that gets retried still cost tokens. Worse, if you leave the failed attempt in context (most agents do), it inflates the input of the retry. Sum them all.
  2. The denominator is completed tasks — not attempts, not requests. Dollars ÷ successes. Everything else is a rate, not a cost.
  3. Price cached input at the cache-read rate. Cache reads run ~90% cheaper than full input (Claude Haiku 4.5: ~$0.10 vs $1.00 per 1M). Charging cached tokens at full rate overstates any cacheable multi-agent workload.

1. Define the model prices you actually pay#

Prices are per 1M tokens (July 2026 list rates). Keep them in one place so the arithmetic is auditable.

# dollars per 1M tokens: (input, output, cached_input)
PRICES = {
    "gemini-3.6-flash": (1.50, 7.50, 1.50),   # cache rates unconfirmed → full rate
    "claude-haiku-4.5": (1.00, 5.00, 0.10),   # cache read is the lever
    "gpt-5-mini":       (0.25, 2.00, 0.025),  # ~10% cached input
}

def cost_usd(model, in_tok, out_tok, cached_tok=0):
    pin, pout, pcache = PRICES[model]
    fresh_in = max(0, in_tok - cached_tok)
    return (fresh_in * pin + cached_tok * pcache + out_tok * pout) / 1_000_000

2. Track tokens across every attempt#

The key move is to record one Attempt object per model call and hang every attempt off a single Task, along with a succeeded flag that only flips true when the task genuinely finishes. Because the failed attempts stay in the list, they stay in the sum — which is exactly the point. A retry that burned tokens and then failed still cost you real money, and the harness has to see it. Keep the structure dumb and additive: append on every call, never overwrite, and let the totals fall out at the end.

from dataclasses import dataclass, field

@dataclass
class Attempt:
    in_tok: int
    out_tok: int
    cached_tok: int = 0

@dataclass
class Task:
    model: str
    attempts: list = field(default_factory=list)
    succeeded: bool = False

    def cost(self):
        return sum(cost_usd(self.model, a.in_tok, a.out_tok, a.cached_tok)
                   for a in self.attempts)

3. Wrap your agent run and record the usage#

Your provider returns token usage on every response (usage.input_tokens / output_tokens, and a cached field). Push an Attempt for each call — success or failure — and set succeeded only when the task's own check passes (tests green, valid JSON, correct answer — whatever "done" means for you).

def run_task(model, prompt, max_retries=3):
    task = Task(model=model)
    for _ in range(max_retries):
        resp = call_model(model, prompt)          # your provider call
        u = resp.usage
        task.attempts.append(Attempt(
            in_tok=u.input_tokens,
            out_tok=u.output_tokens,
            cached_tok=getattr(u, "cached_input_tokens", 0),
        ))
        if task_is_complete(resp):                # YOUR success check
            task.succeeded = True
            break
    return task

4. Divide by successes — the number that matters#

def report(tasks):
    total_cost = sum(t.cost() for t in tasks)
    wins = sum(1 for t in tasks if t.succeeded)
    per_success = total_cost / wins if wins else float("inf")
    print(f"model:            {tasks[0].model}")
    print(f"tasks run:        {len(tasks)}")
    print(f"succeeded:        {wins} ({wins/len(tasks):.0%})")
    print(f"total spend:      ${total_cost:.4f}")
    print(f"COST PER SUCCESS: ${per_success:.4f}")

Run the same fixed task set through two or three models and put the COST PER SUCCESS lines next to each other. That comparison — not the vendor's pricing page — is what should pick your default.

What the number catches that the pricing page hides#

Say GPT-5 mini is ~3x cheaper per token than Gemini 3.6 Flash. If mini succeeds first-try 60% of the time (each failure burns tokens and a retry) while Flash's token efficiency lets it finish first-try 90% of the time with fewer output tokens, the pricier-per-token model can land lower per completed task. The pricing page shows you the 3x. Only this harness shows you the crossover.

That's the same lesson as why one tokens-per-second number is lying to you — the vanity metric and the paid metric diverge exactly where the money is. Once you have cost-per-success, the Gemini 3.6 Flash vs Claude Haiku 4.5 vs GPT-5 mini decision stops being a guess. And if you want to push it further — per-customer margins rather than per-task cost — see how to track LLM cost per customer.