The first thing almost everyone reaches for to control an AI agent's cost is max_tokens. It's right there in every SDK, it has "max" in the name, and setting it feels like installing a limiter. It is a limiter — of the wrong thing. max_tokens caps the length of one model reply. Agent bills do not blow up on the length of one reply. They blow up on the number of replies, and that number is exactly what max_tokens says nothing about.

An agent is a loop, and cost lives in the loop#

A single chat completion has a naturally bounded cost: prompt in, one reply out, done. An agent is a while loop wrapped around that call — think, act, observe, think again — and it keeps calling the model until it decides it's finished. The total cost is the sum over every iteration, and the number of iterations is a runtime decision the model makes, not a constant you set.

Now stack on the detail that makes it worse than linear. Most agent loops resend the full conversation each turn, so the input on step 20 contains everything from steps 1 through 19 — every tool result, every intermediate thought. Token spend per step climbs as context accumulates, which is why agent costs scale super-linearly in the number of steps rather than tracking it one-for-one. A run that goes twice as many rounds doesn't cost twice as much; it costs more.

So picture the failure that actually empties a budget: a fail-open loop. A tool returns a malformed result, the agent decides to "try a different approach," calls the model again, gets confused, tries again — a hundred short, individually cheap completions, each dragging a fatter context than the last. Every one of them respects your max_tokens perfectly. The bill is enormous anyway, because max_tokens was guarding a door the money never walked through.

max_tokens bounds the reply. The reply was never the problem. The loop is the problem, and the loop is unbounded.

Cap the run, not the token#

The control you actually want is a per-run ceiling, and it has two dimensions, because either one can run away on its own:

The budget dimension is the one people skip, and it's the one that catches the growing-context problem: an iteration cap alone won't notice that each of your 15 allowed steps got three times more expensive than you assumed. Count real dollars, across the whole run, and abort when the meter crosses the line.

Two properties make this a cap rather than a wish. First, it has to be enforced outside the model call — in the agent runtime or an LLM gateway that sees every request — because that's the only place with a running total across iterations. A single request can't know it's the ninetieth. Second, it has to fail closed: when the ceiling is hit, the next call is refused with a hard error, not softened into a warning the loop ignores.

This is not theoretical plumbing; it's shipping. LiteLLM's agent iteration budgets are exactly this shape. You attach a session id to a run (via the x-litellm-trace-id header or metadata.session_id), and configure max_iterations and max_budget_per_session. LiteLLM accumulates that session's spend against a shared counter and, once a run crosses the line, returns a budget_exceeded error with HTTP 429 — the loop is stopped at the gateway no matter what the agent's own code intended to do. Because the counter lives in the gateway, it holds even across restarts and parallel workers, and it composes with the coarser max_budget ceilings LiteLLM already offers per key, user, and team. One layer bounds a single run; the outer layers bound the account.

Where the cap must not live: the prompt#

There's a tempting shortcut worth naming so you don't take it: telling the agent, in its system prompt, "do not spend more than \$2." This does nothing you can rely on. The model can't see its own running total — it has no meter — and even if it could, a prompt is a suggestion it's free to reason past. Budget language in the prompt is at best a nudge toward brevity. The enforceable cap belongs in the runtime that issues the calls and counts the tokens, because that layer can say no and mean it.

Two caps, then, aimed at two different targets. Keep max_tokens (or max_completion_tokens) to bound the size and latency of any single reply — that's a real job and it does it well, as Anthropic and OpenAI both document. But put the ceiling that protects your bill one level up: a per-run iteration and dollar budget, enforced in the gateway, failing closed. Pair it with real loop-termination logic so a healthy agent stops on its own — and treat the budget cap as the backstop for the day it doesn't. Cap the loop, not the token, because the loop is where the money is.