Here is the entire decision in one sentence: a rented GPU is a fixed cost and a per-token API is a variable cost, and the only question is whether your traffic is big enough and steady enough to make the fixed cost the cheaper one. Everything else — which card, which provider, which model — is downstream of that.
If you read one line: a specialty-cloud H100 costs about $1,460/month whether it runs or sits idle; an open-model API costs ~$0.20–$0.90 per million tokens and nothing when idle. Multiply your real monthly token volume by the API rate, compare it to $1,460, and you have your answer. Most early products are nowhere near the crossover.
The two cost shapes#
A rented GPU bills by the clock. At the specialty-cloud rate of ~$2/hr for an H100 (see the full price map), that's:
$2.00/hr × 24 × ~30.4 days ≈ $1,460/monthThat number does not move with usage. Serve one request or a million — the bill is $1,460. The GPU's idle hours cost exactly as much as its busy ones.
A per-token API bills by the token. Open models in 2026 run roughly $0.20–$0.90 per million tokens: Groq lists Llama 3.3 70B at $0.59 in / $0.79 out per million; Together AI lists the same model at ~$0.88 flat and DeepSeek V3.1 at $0.60/$1.70; DeepSeek's own API runs $0.14/$0.28 for V3.2. When no one is calling, you pay zero.
Finding your crossover#
The break-even is where variable cost meets fixed cost. Rearranged, it's a token volume:
break-even tokens/month = $1,460 ÷ (API $ per token)
At Together's ~$0.88 / 1M tokens for Llama 3.3 70B:
$1,460 ÷ ($0.88 / 1,000,000) ≈ 1.66 billion tokens / monthSo on paper you'd need to push ~1.66 billion tokens a month through that 70B model before the rented H100 breaks even against the API — and that's before you account for the fact that a single H100 serving a 70B model realistically caps out around a few hundred million tokens a month at healthy utilization. In other words, for many workloads a single card can't even reach its own break-even; you'd need several cards busy around the clock, which raises the fixed cost in lockstep.
Run the number for your volume, not the sticker. The API's headline rate looks expensive per million tokens until you multiply by how few million tokens you actually serve — at which point the $1,460 fixed rental is the expensive option.
Cheaper or smaller models move the crossover, but rarely in the direction you'd hope: an 8B model on an API drops to $0.05–$0.12 per million tokens, which makes the API even harder to beat with rented metal.
When renting actually wins#
The GPU is the right call in specific, recognizable situations — none of which are "to save money on traffic I don't have yet":
- Steady, high-volume 24/7 load. If the card stays genuinely busy, it amortizes its fixed cost across enough tokens to undercut the API. This is the amortization the whole bet rests on.
- Data isolation or compliance. Some workloads can't send tokens to a third-party endpoint. Then the question isn't cost — it's control, and you rent.
- A custom fine-tune that must stay warm. If you've trained weights that no API hosts, and you need them always-on with low latency, you own the serving.
- Latency/throughput you can't get shared. Dedicated hardware gives predictable tail latency that a shared API endpoint may not.
If you do rent, keep the card busy: run a modern inference server (vLLM 0.25 or SGLang), batch aggressively, and watch utilization like it's the meter it is. And weigh the full token-cost picture — inference is only one line of the bill.
The takeaway#
The rent-vs-API choice has a number, and the number is your monthly token volume. Below the crossover — where most early products live — the per-token API is cheaper, simpler, and carries zero idle cost. Rent a GPU when you have steady high-volume load, a hard data-isolation requirement, or custom weights that must stay warm — not because per-million-token pricing looks expensive next to a $2/hr sticker. Do the multiplication first; it almost always argues for the API until you've genuinely outgrown it.
