How to Shadow-Test a Cheaper LLM on Your Real Traffic Before You Switch

The advice everyone's giving right now — "reserve the expensive model for the hard calls, route the rest to something cheaper" — is correct. The problem is the switch. A cheaper model that's 90% as good on a demo can be quietly 30% worse on the specific, weird, real prompts your users actually send. Swap it in blind and you find out from churn.

Shadow testing is how you find out first. You run the challenger model against your real production traffic — same prompts, same context — while your users keep getting answers from your current model. Nobody sees the challenger's output; you just log it and grade it. By the time you flip the switch, you're not guessing. You have a win-rate, a cost delta, and a latency profile from your traffic.

Here's the whole thing, end to end. The code is Python with an OpenAI-compatible client, but the pattern is portable to any language and any gateway.

1. Put a gateway in front of every model call#

You can't swap a model you've hardcoded. The prerequisite for everything below is that your model is a config value routed through one interface, not a string buried in ten call sites. An OpenAI-compatible gateway (OpenRouter, LiteLLM, your own thin proxy) gets you this for free: every model speaks the same request/response shape, so the model name is the only thing that changes.

from openai import OpenAI

client = OpenAI(
    base_url="https://openrouter.ai/api/v1",  # or your own gateway
    api_key=OPENROUTER_API_KEY,
)

PRIMARY = "anthropic/claude-opus-4.8"       # what you serve today
CHALLENGER = "some-vendor/cheaper-model"     # the one you're evaluating

def complete(model, messages):
    return client.chat.completions.create(model=model, messages=messages)

If you take one thing from this guide, take this step. Even with no shadow testing, being able to change PRIMARY in one place is what turns "migrate to a cheaper model" from a two-week project into a one-line diff.

2. Serve from the primary, shadow the challenger#

On each real request, you do two things: return the primary's answer to the user now, and fire the challenger in the background where it can't add latency or affect the response. The user's experience is untouched; you're just quietly running a second model on the side.

import asyncio, time

async def shadow(model, messages, request_id):
    """Fire-and-forget: never blocks the user, never raises into the request path."""
    t0 = time.monotonic()
    try:
        resp = await aclient.chat.completions.create(model=model, messages=messages)
        log_shadow(request_id, model,
                   text=resp.choices[0].message.content,
                   usage=resp.usage,
                   latency_ms=(time.monotonic() - t0) * 1000)
    except Exception as e:
        log_shadow(request_id, model, error=str(e))

def handle_request(messages, request_id):
    answer = complete(PRIMARY, messages)          # served to the user
    log_primary(request_id, PRIMARY, answer)
    asyncio.create_task(shadow(CHALLENGER, messages, request_id))  # observed only
    return answer

Two rules make this safe: the shadow call is fire-and-forget (wrap it so an error or timeout can never bubble into the user's request), and you sample it. You don't need 100% of traffic — 5–10% is plenty of signal and keeps your shadow spend to a rounding error. Skip shadowing entirely on anything user-visibly latency-critical or on requests carrying data you're not comfortable sending to a second vendor.

3. Log both responses against the same request#

The comparison is only meaningful if both models saw the exact same input. Key every log line to a shared request_id and capture the three things you'll actually decide on: the output, the cost, and the latency.

def log_shadow(request_id, model, text=None, usage=None, latency_ms=None, error=None):
    record = {
        "request_id": request_id,
        "model": model,
        "role": "shadow",
        "text": text,
        "prompt_tokens": getattr(usage, "prompt_tokens", None),
        "completion_tokens": getattr(usage, "completion_tokens", None),
        "latency_ms": latency_ms,
        "error": error,
    }
    sink.write(record)   # your warehouse, an eval tool, or newline-delimited JSON

Write it wherever you already look at data — a table, a plain JSONL file, or a dedicated LLM-observability tool like Langfuse, which is built to store paired traces and grade them. The schema matters more than the destination: one row per (request, model), with text + tokens + latency, is everything the next two steps need.

4. Grade the shadow responses offline#

Now that you have paired outputs, grade the challenger. How you grade depends on the task:

• Structured / verifiable output (classification, extraction, JSON, SQL, a specific format): grade programmatically. Exact match, regex, JSON-schema validation, or "does this SQL run and return the right shape." This is cheap, deterministic, and where cheaper models most often quietly win.
• Open-ended output (summaries, replies, rewrites): use an LLM-as-judge. Give a strong model both answers without telling it which is which and a rubric, and ask which better satisfies the task. Randomize the order to cancel position bias, and spot-check a sample by hand to make sure the judge agrees with you.

JUDGE_PROMPT = """You are grading two AI answers to the same request.
Rubric: correctness, completeness, follows instructions, tone.
Answer A:
{a}
Answer B:
{b}
Reply with only "A", "B", or "TIE" for which better satisfies the rubric."""

def judge(primary_text, shadow_text):
    # randomize which slot each answer goes in, then map the verdict back
    ...
    verdict = complete(JUDGE, [{"role": "user", "content": prompt}])
    return verdict  # -> "primary" | "challenger" | "tie"

Then compute the three numbers that make the decision for you, across your logged sample:

• Win/tie rate of the challenger vs. the primary (a challenger that ties on 95% of real traffic is a switch, not a downgrade).
• Cost delta — sum the token costs; this is the whole reason you're here, so put a real dollar figure on the monthly saving.
• p95 latency — averages lie; a cheaper model that's occasionally very slow can be worse UX than a pricier steady one.

5. Roll out behind a flag, and keep the escape hatch#

If the numbers clear your bar, don't flip 100% at once. Route a percentage of live traffic to the challenger as the real primary and watch the same metrics — plus your actual product signals (thumbs, retries, escalations) — for a few days. Ramp 5% → 25% → 100% as it holds.

def pick_model(request_id):
    return CHALLENGER if bucket(request_id) < ROLLOUT_PCT else PRIMARY

Because everything still runs through the gateway from Step 1, rolling back is a single config change, not an incident. That's the property that lets you move fast: the switch is cheap in both directions.

The takeaway#

"Route the cheap work to a cheaper model" is only advice until you can prove the cheaper model clears your bar on the traffic you actually have. Shadow testing is that proof: serve from your current model, quietly run the challenger on a sample of real requests, grade the paired outputs, and decide on win-rate, cost, and p95 latency instead of a demo. Keep the gateway in front of it all, and every future model swap — and the price war is going to keep handing you cheaper options — becomes a measured, reversible config change instead of a leap of faith.