Adding a fallback model is the first reliability upgrade every agent team makes, and it's a good one. Your primary model gets rate-limited or falls over, the request retries down a chain to a backup, and the user never sees the outage. The chain does exactly what it promises.

That's the problem. It did exactly what it promised — it turned a failure into a success — and in doing so it deleted the one signal that told you anything was wrong.

The failure mode nobody instruments#

Think about what a fallback actually is: a rule that says when the good model can't answer, let a different model answer instead. If your primary is a frontier model and your fallback is a cheaper, weaker one — which is the usual setup, because the whole point is that the fallback is more available — then every successful failover is a request that quietly got a worse answer. No exception. No 5xx. A clean 200 OK with a plausible response that happens to be dumber.

This is the same shape as the most dangerous class of agent bug: the failure that returns 200 OK. An error you can see is a problem you can fix. A regression that hides inside a success is a problem that ships. A fallback chain, left uninstrumented, is a machine for manufacturing exactly that kind of hidden regression — and it runs hardest during an incident, when your primary is struggling and failover rate spikes, which is precisely when you're least able to notice quality quietly cratering.

Availability and correctness are different axes. A fallback trades one for the other on every request it catches — and by default it never tells you it made the trade.

The chain itself is the easy part#

Don't overthink the mechanism; the libraries have solved it. In LiteLLM, the Router takes an ordered map:

from litellm import Router

router = Router(
    model_list=[
        {"model_name": "primary",   "litellm_params": {"model": "gpt-primary"}},
        {"model_name": "backup",    "litellm_params": {"model": "claude-backup"}},
        {"model_name": "budget",    "litellm_params": {"model": "cheap-backup"}},
    ],
    # tried in order; every name here must exist in model_list
    fallbacks=[{"primary": ["backup", "budget"]}],
)

LiteLLM even splits fallbacks by error class — content_policy_fallbacks for moderation refusals and context_window_fallbacks for context-length overflows — so a 400-because-too-long doesn't get routed the same way as a 429. With OpenRouter, you drop the single model and pass a models array in priority order:

resp = client.chat.completions.create(
    extra_body={"models": ["primary-model", "backup-model", "budget-model"]},
    messages=[...],
)
served_by = resp.model   # which model actually answered — log this

OpenRouter tries each in turn, only bills the run that completes, and — crucially — returns a model field telling you which one answered. That field is the thread you pull to make the whole thing observable.

Four habits that make the trade visible#

The mechanism gives you availability for free. Correctness-under-failover is the part you have to build, and it's cheap:

1. Log which model served every request. Not the model you asked for — the one that answered. OpenRouter's resp.model, LiteLLM's response metadata. If you can't answer "what fraction of yesterday's traffic was served by a non-primary model?" you are flying blind on your own quality.

2. Alert on fallback rate, not just error rate. Your error dashboard is green because the fallbacks are working. Add a separate metric: the share of requests served by anything other than the primary. A slow creep from 0.5% to 15% is an incident that your error rate will never show you.

3. Order the chain by capability-adjacency, not just availability. A fallback from a frontier model straight to a tiny budget model is a cliff. Put a comparable-tier model next in line and the budget option last, so a single failover is a step down, not a fall. You're designing a gradient, not a trapdoor.

4. Run a quality canary on fallback outputs. Sample the requests that were served by a non-primary model and score them with the same eval you already run in CI — an LLM-judge rubric or a golden-set comparison. You are not trying to make the fallback match the primary; you're trying to know the gap and get paged when it widens. This is the missing half of every retries-and-fallbacks setup: the retry logic guards availability, the canary guards correctness.

There's a cost dimension hiding here too. When a request silently falls back to a cheaper model, its cost lands under the wrong line unless you attribute spend by the model that actually served it — so a rash of failovers can make your primary look cheaper than it is while your quality quietly drops. Same root cause, different symptom: the fallback hid the truth.

The one-sentence version to take to your next reliability review: a fallback chain is not free reliability, it's a loan against quality, and the interest is invisible until you instrument it. Add the four habits and the trade becomes a dial you control. Skip them and the dial turns itself — always toward cheaper, always toward worse, always with the status light glowing a steady, reassuring green.