The AI bill crosses some line — a scary line item on the invoice, a margin conversation with a co-founder — and the reflex kicks in: switch to a cheaper model. It's the wrong first move. Swapping the model is the one lever that can quietly degrade the thing your customers actually experience, and it's usually not even where most of the money is.

Here's the reframe that matters: the same request costs wildly different amounts depending on how you send it, not which model you send it to. The demand-side price war we covered this week makes intelligence cheaper at the source, but you capture almost none of that if your calls are structured wastefully. Most of the waste is structural, invisible to users, and yours to fix today.

Five moves, ordered by return on effort. Four of them don't change a single output.

The 30-second version#

Move 1: Turn on prompt caching — the free 90%#

This is the largest win almost nobody has turned on, and it costs you nothing in quality.

Every call your app makes carries a big chunk that never changes: the system prompt, your tool definitions, the retrieved documents, the few-shot examples. Without caching, you pay full input price to re-process all of it on every single request. Prompt caching stores that processed prefix and charges a fraction to reuse it — on Claude, a cache read is 0.1x the base input price (a ~90% discount on the cached part), with a small one-time write premium of 1.25x for the standard 5-minute cache. OpenAI's current models cache automatically for any prompt over 1,024 tokens with no code change; Gemini 2.5+ does implicit caching by default.

The output is byte-for-byte identical. You are simply refusing to pay full price for the same tokens twice.

The one thing you have to do is order your prompt so the stable content comes first and the variable content comes last — caching works on the shared prefix, so if the user's unique question sits at the top, nothing after it can be cached.

from openai import OpenAI
client = OpenAI(base_url="http://0.0.0.0:4000", api_key="sk-...")  # gateway; see Move 2

resp = client.chat.completions.create(
    model="claude-haiku-4-5",
    messages=[
        # STABLE, cacheable prefix — same on every call, so it's charged ~once:
        {"role": "system", "content": SYSTEM_PROMPT + TOOL_DOCS + FEWSHOT_EXAMPLES},
        # VARIABLE part goes LAST so the whole prefix above stays a cache hit:
        {"role": "user", "content": user_question},
    ],
)
If you do exactly one thing after reading this, do this one. It is the rare optimization with a large upside and no downside — the model, the prompt, and the answer are unchanged; only the bill moves.

Takeaway: Reorder your prompt (stable first, variable last), confirm caching is on for your provider, and watch the input cost of your repeated prefix fall by roughly an order of magnitude.

Move 2: Route through an OpenAI-compatible gateway#

Before the moves that involve changing where calls go, make where they go a config value instead of a code path. Nearly every serious model now speaks the OpenAI Chat Completions format, so if you point your existing OpenAI SDK at a gateway, the provider and model become strings you can change without touching code.

from openai import OpenAI

client = OpenAI(
    base_url="http://0.0.0.0:4000",  # a self-hosted LiteLLM gateway (or a hosted marketplace)
    api_key="sk-...",
)
# model is now just a string — "claude-haiku-4-5", "gpt-5.6-terra", "grok-4.5"...

A gateway like LiteLLM gives you one API key, one bill, one spend dashboard, and — critically — one place to add caching (Move 4), routing (Move 3), and a spend cap (Move 5) without editing your app. If you're pre-launch and don't want the extra hop yet, that's fine; just keep your calls OpenAI-shaped so adding the gateway later is a base_url change, not a migration. (We wrote the full keep-your-stack-swappable playbook if you want the mechanics.)

Takeaway: One base_url change now turns the next three moves from rewrites into config edits.

Move 3: Tier your models — but prove it with an eval set#

Now the move with real leverage and real risk. Not every call needs your best model. Classification, extraction, formatting, tagging, first-draft generation — a lot of production traffic is "good-enough" work that a cheap tier handles fine. And because output tokens cost 3–6x input (Claude Opus 4.8 is $5 in / $25 out per million; the cheap tiers land near $1 in / $5–6 out), the savings are largest exactly on the generation-heavy calls you can most afford to route down.

The discipline that makes this safe: keep a 20–50 example eval set of your real tasks, and only route a call type to a cheaper model once it passes. "Cheaper" is only cheaper if it clears your bar; a leaderboard can't tell you that about your data. This is the one move here that can degrade the product, so it's the one you gate behind evidence.

# Route by task difficulty, not by habit.
model = "claude-haiku-4-5" if task in CHEAP_OK else "claude-opus-4-8"

Takeaway: Send the flagship only the calls that fail on the cheap tier — and let a small eval set, not a guess, decide which those are.

Move 4: Cache answers, not just prompts#

Move 1 stops you paying twice to send the same context. Semantic caching stops you paying to answer the same question twice. When a new query is close enough to a past one, a semantic cache returns the stored answer and skips the model call entirely — a large win on repetitive read-only traffic (support, FAQ, docs search), where reported hit rates run 40–60% and vendor case studies claim cost cuts north of 60%.

The catch is correctness. "Close enough" is a similarity threshold you have to tune, and a too-loose threshold serves the wrong answer confidently. So: turn it on for your most repetitive, least personalized endpoint first, set the threshold conservatively, scope the cache per-user for anything that depends on identity, and never cache a call whose answer depends on live state. It's plumbing in your gateway (LiteLLM supports Redis and semantic caching directly), not an app rewrite.

Takeaway: On a genuinely repetitive endpoint, an answer cache can erase a large fraction of calls — as long as you tune the threshold and scope it so it never serves a stale answer as a fresh one.

Move 5: Batch the non-urgent, then cap the whole thing#

Two finishers.

Batch anything that doesn't need to be real-time. Nightly summaries, backfills, evals, bulk enrichment, content generation — the batch APIs run these asynchronously (up to ~24h) for 50% off both input and output. If a job doesn't have a user waiting on it, it shouldn't pay real-time prices.

Put a hard spend cap in the gateway. This one doesn't lower your steady bill; it caps the blast radius of the failures that produce five-figure surprises — a retry loop, a leaked key, a bad deploy. A gateway budget per key or team fails calls with a 429 once spend crosses the line, instead of quietly running up the tab. (For agent-shaped workloads specifically, the cap that matters is per-run, not per-reply — we go deep on that in how to cap an agent's spend per run.)

Takeaway: Move the patient work to batch for half price, and let the gateway — not your on-call rotation at 2am — be what stops a runaway loop.

The order of operations#

The mistake is starting at the bottom of this list. Downgrading the model is the most visible move and the one customers feel, yet it's rarely where most of the waste lives. Work top-down instead:

  1. Prompt caching — free, invisible, ~90% off your repeated prefix. Do it today.
  2. Gateway — makes everything below a config change.
  3. Batch + spend cap — half-price patient work, a ceiling on disasters. No quality cost.
  4. Semantic caching — big on repetitive traffic, gated by a threshold.
  5. Model tiering — the largest lever and the only one with quality risk, so it goes last and behind evals.

Exhaust the moves your users can't feel before you reach for the one they can. If the tools rather than the mechanics are what you're after, we mapped the cost-control tools founders should know — but most of the savings above are yours for free, in code you already control.