The reasoning always sounds airtight. API tokens cost money on every call; a rented GPU is a flat rate; therefore, past some volume, self-hosting an open model has to be cheaper. So the founder rents a card, stands up a serving stack, and waits for the savings.

They don't come — not because the math is wrong, but because it compares the wrong two numbers.

Two different kinds of cost#

An API bills you per token you actually generate. Serve nothing this hour and you pay nothing. Serve a million tokens and you pay for a million. The cost tracks your usage exactly.

A rented GPU bills you per hour you hold it — pinned at 95% or idling at 3%, the invoice is identical. In 2026, an on-demand H100 runs about $2–3/GPU-hour at GPU-first clouds like Lambda and RunPod (and $7 or more at the hyperscalers). Held continuously, that's roughly $1,460–2,190 a month for one GPU — a fixed cost that does not care how much traffic you have.

So the honest comparison isn't "$6 per million tokens" versus "the open model is free." The open model isn't free; the GPU under it costs the same fifteen hundred dollars whether you serve ten requests or ten million.

The break-even is utilization, not tokens#

A GPU bills the same at 3% utilization and 95%. Your real cost per token is the fixed monthly price divided by the tokens you actually serve — and you only reach the low number if you keep the meter pinned.

Here's the ceiling. A single H100 can sustain, very roughly, about 1,500 output tokens per second on a mid-size open model with decent batching — call it 3.9 billion tokens a month, if you keep it saturated 24/7. Divide $1,460–2,190 by 3.9 billion and the per-token cost is genuinely tiny. That's the dream.

Now divide the same fixed cost by what you really serve. A spiky product that's busy during business hours and dead overnight might average 15% utilization. Your effective per-token cost just went up almost 7×, because you paid for a full month and used a sixth of it. At low or bursty volume, you're renting a mostly-empty meter, and the pay-per-token API — which charged you nothing for all those idle hours — wins by a wide margin.

The rule of thumb every 2026 cost calculator converges on: below roughly 100–500 million tokens a month of steady usage, APIs are almost always cheaper. Self-hosting only flips at high, predictable, sustained volume.

The costs the GPU quote leaves out#

Even when the utilization math looks favorable, the sticker price hides four lines that reliably wreck the spreadsheet:

That last one is the killer. The API's price includes someone else's on-call rotation. Yours doesn't.

The honest default#

For almost every early-stage builder, the answer in 2026 is: stay on the API. Pick a cheap tier — GPT-5.6 Luna is $1 in / $6 out per million tokens, DeepSeek and Gemini Flash go lower — layer in prompt caching, and you'll spend less than a single idle GPU costs, with zero ops burden.

Revisit self-hosting only when one of three things is true:

  1. Measured, sustained, high utilization — you've watched your traffic and the GPU would stay near capacity, not a hunch that tokens "feel expensive." (If you're there, where you serve the open model becomes the next decision.)
  2. A hard requirement the API can't meet — data residency, air-gapped deployment, or a regulatory line that forbids sending data to a third party.
  3. A specialized or fine-tuned model no hosted provider will serve for you.

Outside those, the seductive part of the self-hosting pitch — the flat rate — is exactly the part that hurts you. A flat rate is only a bargain if you keep the meter running. Measure your utilization before you rent the card, not after.