If you are training, fine-tuning, or running batch jobs on rented H100s and paying the on-demand rate, you are very likely overpaying by 60–90%. Spot instances — AWS's name; "preemptible" on Google Cloud, "interruptible" on the GPU marketplaces — are the same silicon sold out of spare capacity at a steep discount, on one condition: the provider can take the machine back when it needs the capacity. On AWS, the 8×H100 p5.48xlarge that costs $98.32/hr on-demand runs about $19.66/hr on spot — roughly an 80% cut for the identical box.

That is the headline, and it is real. But the headline number is not the one that decides whether spot works for your job. This one is: how much warning do you get before the machine disappears?

The discount is uniform. The notice window is not.#

Every provider that sells spare capacity will reclaim it. What varies — by an order of magnitude — is how much runway you get when they do.

Your checkpoint interval and your choice of provider are not two decisions. They are the same decision.

If you checkpoint every 10 minutes and your provider gives 15 seconds of warning, you will routinely lose up to 10 minutes of compute per reclaim. If you checkpoint every 30 seconds, you have throttled your own training to survive a machine you chose for its price. The notice window sets the floor on how much work a single reclaim can destroy — so pick the provider and the interval together, sized to the shortest window you'll actually see, not the average.

Interruptions got rarer, which changes the math#

The old objection to spot — "it gets yanked constantly" — is dated for the current generation of accelerators. Thunder Compute's 2026 interruption data puts H100 spot interruptions on AWS under 5%, against 15–20% for the older A100 fleet. Newer, higher-demand parts are held more tightly by the provider precisely because they're scarce, but once you're on one, the reclaim probability per hour is low.

Low, not zero. And that's the entire game: a job that checkpoints cheaply pays almost nothing for a rare reclaim, while a job that can't checkpoint pays for the whole run every time. Which is why the correct way to budget spot is not the sticker rate. It's:

effective cost ≈ spot rate × (1 + expected recompute overhead)

For a well-checkpointed training run on a sub-5% part, that overhead rounds to noise and you keep almost the full 80%. For an un-checkpointed 6-hour job that loses an average of three hours per reclaim, the "80% discount" can quietly turn into paying more than on-demand once you count the wasted GPU-hours. The discount isn't a rate. It's a rate times how resumable your job is.

The decision, in one line each#

Use spot when the work is fault-tolerant and resumable:

Stay on-demand when a stop is a failure:

Making a job spot-safe (the 20-minute version)#

  1. Checkpoint to durable storage, not local disk. S3, GCS, or a network volume that survives the instance. Local NVMe dies with the box.
  2. Save on a fixed interval, sized to your worst-case notice. On a 15-second-warning marketplace host, "checkpoint at the warning" is not a strategy — you need recent state already on disk.
  3. Resume by default. On boot, look for the latest checkpoint and continue; don't restart from zero. This is what turns a reclaim from a disaster into a hiccup.
  4. Trap the reclaim signal. Poll the metadata endpoint (AWS) or handle the provider's termination hook, and force a final checkpoint on the way out. Free insurance when you do get two minutes.
  5. Fall back, don't fail. If capacity is gone, requeue onto another region or provider — or a temporary on-demand box — rather than stalling the pipeline.

Do those five things and spot stops being a gamble and becomes what it actually is: the same H100 you were already renting, at a fraction of the price, with a small, well-understood tax for the rare interruption.

Where it fits in the rental map#

Spot is one axis of the GPU-cost decision, not the whole thing. Which provider and which chip is the CoreWeave vs Lambda vs Nebius comparison; what the raw hourly rates are across H100/H200/B200 is the August 2026 rental price map; whether to rent at all versus serve an open model per-token is its own build-or-buy call. Spot sits on top of all of them: once you've picked a provider and a chip, interruptible-vs-on-demand is the last lever, and for anything that checkpoints, it's the biggest one on the board.

The instinct to reach for the guaranteed machine is the expensive instinct. Most of what a founder runs on a GPU — the training, the sweeps, the overnight batch — was never latency-bound in the first place. It was just easier to click "on-demand." That click is the 80%.