The short version: A fallback model chain sends every request to a cheap model first, and only calls an expensive frontier model when the cheap one fails — exceptions, timeouts, malformed output, or low-confidence answers all count as "failed." The pattern is: try the cheap model with a short timeout and a couple of retries for transient errors, validate what comes back, and escalate to the frontier model if any of those checks don't pass. Done right, 90%+ of traffic never touches the expensive model, but every request still gets a correct answer.

Why this beats picking one model#

Cheap, often open-weight models (Llama, DeepSeek, Mistral-class models on cheap-inference providers) can run 10-25x cheaper per token than frontier models like GPT-5.x or Claude Opus-class models, but they're flakier: more rate limits on cheap-tier infra, more malformed JSON, more outright wrong answers on hard prompts. A fallback chain lets you have the cheap price on the easy 90% of requests and the frontier reliability on the hard 10%, without a human ever choosing which is which.

The building blocks are all standard: try/except, a timeout, a backoff loop, and a validation function. Nothing here requires a specialized library — though we'll cover the built-in alternative at the end.

Setup: one client, two models, one base_url pattern#

Most inference providers (OpenAI, Together, Fireworks, Groq, local vLLM/Ollama servers) expose an OpenAI-compatible /chat/completions endpoint, so a single OpenAI client with a swapped base_url and api_key works for both your cheap and frontier calls.

import os
from openai import OpenAI

# Cheap / open-weight model, e.g. served via Together, Fireworks, or a local vLLM box
cheap_client = OpenAI(
    base_url=os.environ.get("CHEAP_BASE_URL", "https://api.together.xyz/v1"),
    api_key=os.environ["CHEAP_API_KEY"],
    timeout=10.0,      # short — a slow cheap model isn't saving you anything
    max_retries=0,     # we handle retries ourselves, see below
)

# Frontier backstop
frontier_client = OpenAI(
    base_url=os.environ.get("FRONTIER_BASE_URL", "https://api.openai.com/v1"),
    api_key=os.environ["FRONTIER_API_KEY"],
    timeout=60.0,
    max_retries=0,
)

CHEAP_MODEL = "meta-llama/Llama-3.3-70B-Instruct-Turbo"
FRONTIER_MODEL = "gpt-5.6-terra"

Note the client-level timeout and max_retries — these are real OpenAI() constructor arguments. The SDK's own default timeout is 10 minutes and default retries is 2, which is far too patient for a fallback trigger, so set both explicitly.

Step 1: basic try/except fallback#

The floor of any fallback chain is: try the cheap model, and if it raises, call the frontier model instead.

import openai

def complete_basic(prompt: str) -> str:
    try:
        resp = cheap_client.chat.completions.create(
            model=CHEAP_MODEL,
            messages=[{"role": "user", "content": prompt}],
        )
        return resp.choices[0].message.content
    except (openai.APIConnectionError, openai.APITimeoutError, openai.RateLimitError, openai.APIStatusError) as e:
        print(f"cheap model failed ({type(e).__name__}), falling back to frontier")
        resp = frontier_client.chat.completions.create(
            model=FRONTIER_MODEL,
            messages=[{"role": "user", "content": prompt}],
        )
        return resp.choices[0].message.content

APIConnectionError, APITimeoutError, RateLimitError, and the generic APIStatusError (covers 4xx/5xx) are all real exception classes in the OpenAI Python SDK, so this except clause catches network failures, timeouts, rate limits, and server errors in one shot. This alone catches maybe half of real-world failures — the rest are transient and shouldn't trigger an expensive escalation at all, which is what retries are for.

Step 2: retry with exponential backoff first#

Rate limits and brief network blips on a cheap-tier provider are common and usually resolve in a couple of seconds. Escalating to the frontier model on the first 429 wastes the whole point of the cheap tier — retry first, escalate second.

import time
import random

def call_with_retry(client, model, messages, max_retries=2, base_delay=0.5):
    last_exc = None
    for attempt in range(max_retries + 1):
        try:
            return client.chat.completions.create(model=model, messages=messages)
        except (openai.APIConnectionError, openai.APITimeoutError, openai.RateLimitError) as e:
            last_exc = e
            if attempt == max_retries:
                break
            delay = base_delay * (2 ** attempt) + random.uniform(0, 0.25)
            print(f"attempt {attempt + 1} failed ({type(e).__name__}), retrying in {delay:.2f}s")
            time.sleep(delay)
    raise last_exc

Two retries with jittered exponential backoff (0.5s, then ~1s) is enough to ride out a rate-limit blip without adding noticeable latency. APIStatusError for hard 4xx errors (bad request, auth failure) is deliberately not retried here — those won't resolve by waiting, so they should escalate immediately rather than burn retry budget.

Step 3: a validation gate for malformed or low-quality output#

This is the trigger most people skip, and it's the one that matters most for agent workloads: the cheap model can return 200 OK with content that's flat-out unusable — broken JSON, a missing required field, an empty string, or a refusal where you expected a tool call. (This is the same class of silent failure we covered in tool-call error handling, where the most dangerous failure returns 200 OK — a fallback chain's validation gate is where you catch it.)

import json

class ValidationError(Exception):
    pass

def validate_json_output(text: str, required_keys: list[str]) -> dict:
    if not text or not text.strip():
        raise ValidationError("empty response")
    # strip common markdown code fences before parsing
    fence = "`" * 3
    cleaned = text.strip().removeprefix(fence + "json").removeprefix(fence).removesuffix(fence).strip()
    try:
        data = json.loads(cleaned)
    except json.JSONDecodeError as e:
        raise ValidationError(f"invalid JSON: {e}")
    missing = [k for k in required_keys if k not in data]
    if missing:
        raise ValidationError(f"missing required keys: {missing}")
    return data

Treat ValidationError exactly like a network failure in the fallback logic — it's a signal that this response is not usable, regardless of whether the HTTP call "succeeded." If you have logprobs or a self-reported confidence field, the same gate can check a threshold (e.g. if data.get("confidence", 1.0) < 0.6: raise ValidationError(...)) and escalate on low confidence too.

Step 4: tying it together#

def complete_with_fallback(
    prompt: str,
    required_keys: list[str] | None = None,
    cheap_retries: int = 2,
) -> dict | str:
    messages = [{"role": "user", "content": prompt}]

    try:
        resp = call_with_retry(cheap_client, CHEAP_MODEL, messages, max_retries=cheap_retries)
        text = resp.choices[0].message.content
        if required_keys:
            return validate_json_output(text, required_keys)
        return text
    except (openai.APIConnectionError, openai.APITimeoutError, openai.RateLimitError,
            openai.APIStatusError, ValidationError) as e:
        print(f"cheap model chain exhausted ({type(e).__name__}: {e}) — escalating to frontier")

    # Frontier backstop: one attempt, minimal retry, must succeed
    resp = call_with_retry(frontier_client, FRONTIER_MODEL, messages, max_retries=1)
    text = resp.choices[0].message.content
    if required_keys:
        return validate_json_output(text, required_keys)
    return text

Call it the same way regardless of what's underneath:

result = complete_with_fallback(
    "Return JSON with keys 'summary' and 'sentiment' for: 'Great support call today.'",
    required_keys=["summary", "sentiment"],
)

A fallback chain isn't a nicer error message — it's a second, more expensive attempt at the exact same task, gated behind a definition of "failed" that includes bad output, not just bad status codes.

Built-in alternatives: litellm and provider SDKs#

You don't have to hand-roll all of this. litellm's Router takes a model_list of deployments plus a fallbacks mapping and handles retry/backoff and escalation internally:

from litellm import Router

router = Router(
    model_list=[
        {"model_name": "cheap", "litellm_params": {"model": "together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo"}},
        {"model_name": "frontier", "litellm_params": {"model": "gpt-5.6-terra"}},
    ],
    fallbacks=[{"cheap": ["frontier"]}],
)

response = router.completion(
    model="cheap",
    messages=[{"role": "user", "content": "Hello"}],
)

This gets you the exception-and-retry layer (steps 1-2) for free, with cooldowns and load balancing across deployments built in — but you still need your own validation gate (step 3), since litellm has no idea what "good output" means for your task. Several provider SDKs also expose their own retry/timeout knobs at the client level (as shown above for OpenAI-compatible clients); check yours before rebuilding what already ships.

The decision#

Use a plain try/except fallback the moment you put a cheap model anywhere near production — it's five lines and prevents total outages. Add retry-with-backoff as soon as you see rate limits in logs; it's nearly free and stops you from escalating on noise. Add the validation gate as soon as the cheap model's output feeds anything structured — an agent's tool call, a JSON field, a database write — because malformed output is the failure mode that silently corrupts data instead of loudly erroring. Reach for litellm's Router once you're juggling more than two models or providers and don't want to maintain the retry loop yourself; keep your own validation gate either way, because no library can know what "correct" means for your prompt.