There's a line of code that shows up in every production agent, and it looks like diligence: retry(3, { backoff: "exponential" }). Every model call gets it. Timeouts get retried, 429s get retried, 529s get retried. It reads as the responsible thing to do.
It is also, under the one condition that matters, the thing that takes you down.
Retries multiply#
The condition is correlated failure — the provider is slow or rate-limiting you, and it's happening to everyone at once. That's precisely when the reflex to retry does the most damage, because retries don't add load, they multiply it.
Google's SRE book has the canonical illustration, and the number is worth memorizing. Three layers, each with its own retry policy allowing up to four attempts, and a single request at the top can generate 4 × 4 × 4 = 64× the load at the bottom — arriving at the exact moment the dependency is least able to handle it. You rarely have three explicit layers, but you have more layers than you think: your application retries, your gateway or proxy retries, the vendor SDK retries. Stack a 3× app retry on a 3× gateway retry on a 2× SDK retry and one user action becomes eighteen calls into a system that's already gasping.
The SRE book states the mechanism in one sentence: retries can amplify low error rates into higher levels of traffic, leading to cascading failures. A dependency at 5% errors, hit by a fleet that retries everything, does not recover. It gets a traffic spike on top of the degradation that caused the errors. The brownout becomes a blackout, and the retries are the reason.
A retry is not free insurance. Under correlated failure it is the fastest path from "slow" to "down."
Backoff is necessary and not sufficient#
The usual defense is exponential backoff, and you should absolutely use it — with jitter. But understand exactly what it buys, because it's less than people assume.
Backoff controls one client's spacing: wait 1s, then 2s, then 4s, so you're not hammering back-to-back. It does nothing to cap the total number of retries, and nothing to coordinate across clients. Without jitter it's actively dangerous: a fleet that all failed at the same instant all backs off by the same schedule and all retries at the same instant — a synchronized thundering herd, which is why AWS's guidance is backoff plus randomized jitter, not backoff alone. Even then, backoff only changes when the 64× lands, not whether it does.
To bound the total, you need a different primitive.
The budget#
A retry budget treats retries as a shared, exhaustible resource and caps how much of your traffic is allowed to be them. Google's SRE practice is concrete: each client tracks the ratio of requests that are retries, and only retries while that ratio stays under about 10%. Cross the line and you stop retrying — you fail closed — because past that point you're amplifying, not recovering.
The mechanism is adaptive throttling. The client counts requests and accepts over a rolling window and rejects a new attempt with probability:
max(0, (requests − K · accepts) / (requests + 1))
with K around 2. While the dependency accepts your calls, the numerator stays negative and you throttle nothing. As acceptances collapse, the rejection probability climbs smoothly toward 1 — you back off proportionally to how badly things are going, without a human flipping a switch. Layer a per-request cap on top (three attempts, then bubble the failure up) and, if you want a hard ceiling, a server-wide limit like 60 retries per minute per process. gRPC bakes the same idea in as a token-bucket retry throttle specifically to prevent retry storms.
Note what this is not. It's not a circuit breaker. A breaker is a binary gate — tripped open, everything to the downstream is rejected until a cooldown. A budget is a proportional throttle that degrades gracefully while the dependency is merely slow. They cover different failures; run both. The breaker handles "it's down." The budget handles "don't make it worse while it's struggling."
The agent-specific rule#
There's one more constraint that general retry advice doesn't carry, and it's the one that bites hardest in an agent.
A retry budget has to be tool-aware. Retrying a model completion wastes tokens; retrying a non-idempotent tool call — a charge, an email, a POST that isn't safe to repeat — doesn't just add load, it duplicates the side effect. A budget that blindly retries a settled payment isn't protecting you from an outage; it's manufacturing a second charge. So the budget draws only from calls that are retryable by construction — idempotent reads, or writes guarded by an idempotency key — and everything else fails closed on first error. Google Cloud's own retry guidance gates retries on idempotency for exactly this reason.
Which reframes the question you started with. It was never "should I retry this call?" It's "what fraction of my traffic am I willing to let be retries?" Pick that number — 10% is a defensible default — enforce it with adaptive throttling, exempt the non-idempotent tools, and the 64× can't happen. Leave it unbounded, and every retry(3) in your codebase is a small loaded contribution to the outage you'll eventually cause yourself.
