Your agent has a good turn. The model looks at the request, decides it needs eight things, and emits eight tool calls in a single assistant message. You loop over them, fire all eight, and — if any of them hit an external API — you get a wall of 429 Too Many Requests. The single-call version worked fine for weeks. The parallel version breaks the moment it succeeds.

Here's the thing the feature's name hides: the model emitting parallel tool calls and your runtime running them concurrently are two different decisions. The API makes the first one. The second one — how many actually run at once, and what happens when they fail — is entirely your code, and the naive version is a self-inflicted outage.

The failure: a burst with no brakes#

Fire N tool calls simultaneously and you send a burst of N downstream requests with zero spacing. A tool that calls a service rated at 5 requests/second is perfectly happy one call at a time and returns 429 the instant eight land together. Worse, the obvious "fix" — retry the failures — has every failed call waking up after the same delay and firing again at the same instant: a thundering herd that re-triggers the very limit you just hit.

Two levers fix this: bound the concurrency, and back off with jitter.

Bound it with a semaphore#

Cap concurrency to the slowest downstream limit, not the fastest. If the tightest tool hits a service that allows 5 in flight, your ceiling is 5 — no matter that the model asked for 20. A semaphore enforces exactly that: it admits the first few and queues the rest until a slot frees.

import asyncio, random

SEM = asyncio.Semaphore(5)          # size to the slowest downstream limit

async def run_one(call):            # call = one tool_use block
    async with SEM:                 # at most 5 in flight, ever
        for attempt in range(5):
            try:
                result = await execute(call.name, call.input)
                return {"type": "tool_result", "tool_use_id": call.id,
                        "content": result}
            except RateLimited as e:
                # honor Retry-After if the service sent one; else full jitter
                delay = e.retry_after or random.uniform(0, 2 ** attempt)
                await asyncio.sleep(delay)
        return {"type": "tool_result", "tool_use_id": call.id,
                "content": "rate limited, no result", "is_error": True}

async def run_turn(tool_calls):
    return await asyncio.gather(*(run_one(c) for c in tool_calls))

(In TypeScript, p-limit is the one-line equivalent of the semaphore.) Start the cap at 4–8, watch for 429s, and raise it only if the downstream service's published limits actually have room. Unbounded is never the production answer.

Back off the right way#

Two rules, in order:

  1. Honor Retry-After. A rate-limited response usually tells you exactly how long to wait — Anthropic and OpenAI both send it, alongside x-ratelimit-* headers. Obeying the header beats guessing.
  2. **No header? Exponential backoff with full jitter** — sleep a random amount up to a growing cap (random.uniform(0, 2attempt)), not a fixed 2attempt. The randomness is the whole point: it spreads retries out instead of resynchronizing them into the next spike.

Note the SDK already retries 429/5xx for its own calls — this backoff is for the downstream services your tools hit, which the SDK knows nothing about.

Don't lose the wiring#

Concurrency makes it easy to mangle three things the loop depends on:

Only fan out what's independent#

Bounded concurrency is only safe for calls that don't depend on each other. If call B needs A's output, if they share mutable state, if order matters (write-then-read, a transaction), run them sequentially — parallelizing a hidden dependency doesn't just risk a rate limit, it computes results against stale or missing inputs and you get a wrong answer with no error at all. And wrap each call in a timeout so one hung request can't hold its semaphore slot forever.

Prove independence, cap the concurrency, honor the headers, jitter the retries, and return everything in one message. Thirty lines stand between "the agent got faster" and "the agent took down the API it was calling."