Anthropic just made model tiering a visible control in Claude voice mode — Haiku by default, Opus for the hard turn. In your own agent, that same decision is code: run the cheap model first and escalate only when you must. The escalation ladder is the how. This is the missing whether — the afternoon eval that tells you if tiering will actually save you money on your traffic, before you ship it.

Here's the trap: tiered routing is only cheaper than a flat mid-tier if the cheap model clears most of your requests. If it doesn't, you pay the cheap call and the expensive one on every hard query, plus the latency of both. The only way to know which case you're in is to measure the escalation rate on real queries. Most teams never do — they route by vibes and assume Haiku is fine.

Step 1: Build a small golden set#

Pull 50–100 real queries that look like production, and pair each with a known-good answer or an acceptance rule. Representative beats large.

# golden.jsonl — one row per query
# {"q": "...", "gold": "...", "rule": "contains:refund"}
import json
golden = [json.loads(l) for l in open("golden.jsonl")]

Step 2: Run the cheap model and grade every answer#

Call the cheap tier once per query and grade it with the same check you'll use in production — a hard rule where you can write one, an LLM judge where you can't.

import anthropic
client = anthropic.Anthropic()

CHEAP = "claude-haiku-4-5"   # use your provider's exact model id

def run(model, q):
    r = client.messages.create(
        model=model, max_tokens=512,
        messages=[{"role": "user", "content": q}],
    )
    return r.content[0].text

def passes(out, row):
    kind, _, arg = row["rule"].partition(":")
    if kind == "contains":
        return arg.lower() in out.lower()
    if kind == "equals":
        return out.strip() == arg.strip()
    return judge(out, row["gold"])   # fall back to an LLM judge

results = [(row, passes(run(CHEAP, row["q"]), row)) for row in golden]
escalations = sum(1 for _, ok in results if not ok)
rate = escalations / len(results)
print(f"escalation rate: {rate:.0%}")

If you need a judge, keep the rubric tight and calibrate it against your own labels on ~20 examples before you trust the other 80:

def judge(out, gold):
    verdict = run("claude-sonnet-5",
        f"Gold answer: {gold}\n\nCandidate: {out}\n\n"
        "Is the candidate acceptable to ship? Answer only PASS or FAIL.")
    return verdict.strip().upper().startswith("PASS")

Step 3: Compute the break-even#

Let the cheap call cost c and the expensive call cost e (blended input+output for a typical request). A flat expensive tier costs e every time. The ladder costs c always, plus e on the fraction r you escalate:

ladder = c + r * e        # you always pay the cheap probe
flat   = e
# routing wins when:  c + r*e < e   →   r < 1 - c/e

So if Haiku costs roughly a tenth of Opus, c/e ≈ 0.1 and the ladder wins whenever you escalate less than ~90% of the time — but the margin only gets meaningful when your escalation rate is well under half. Plug in your measured rate from Step 2 and your real per-request costs. If rate sits above the break-even, stop here: a flat mid-tier like Sonnet is cheaper and far simpler than a ladder that escalates constantly.

Step 4: Promote the validator into the live gate#

The payoff of doing this as an eval: the grader you just built is your production escalation gate. Ship the exact passes() check inline — run cheap, gate, escalate on fail — and keep the golden set as a regression test for the day a model or prompt changes underneath you.

def answer(q, row_rule):
    out = run(CHEAP, q)
    if passes_live(out, row_rule):
        return out
    return run("claude-opus-5", q)   # escalate only on a real miss

Then log the escalation rate in production forever. It's the one number that tells you whether the ladder is still paying off — and the first thing that moves when a prompt regresses or your traffic mix shifts. A router you don't measure is just a more expensive way to call two models. For the cost-cap and A/B side of this, see the model router playbook.