Every engineer who has tried to write a golden-file test against an LLM has hit the same wall. You pin the model, you pin the seed, you set temperature=0, you run it twice — and the two outputs don't match. The internet's answer is a shrug: GPUs are nondeterministic, live with it. That shrug is wrong, and in 2025 Mira Murati's Thinking Machines Lab published the first rigorous account of why — along with a fix that gets you bitwise-identical output across a thousand runs.
The starting demonstration is bracing. Sample the same prompt from Qwen3-235B 1,000 times at temperature 0 — greedy decoding, no sampling randomness at all — and you get 80 distinct completions. They agree for a while and then fork; in their run, the divergence appeared as early as token 103. Temperature 0 was doing its job perfectly. Something underneath it was not.
The comfortable explanation is mostly wrong#
Ask around and you'll be told the cause is concurrency plus atomics: thousands of GPU threads accumulate partial sums into shared memory, the operating system schedules them in a different order each time, and because floating-point addition isn't associative — (a + b) + c can differ from a + (b + c) in the last bit — you get slightly different numbers, which compound into different tokens.
That story is real for some operations, mostly in the training backward pass. But it does not explain inference. The forward pass that serves your request is dominated by matrix multiplications, and those run in a fixed, reproducible order. Run one matmul twice, on the same GPU, on the same batch, and you get the same bits every time. If atomics-driven chaos were the culprit, you couldn't. So the nondeterminism is entering somewhere the folklore isn't looking.
The output for your request is bitwise-stable. What's unstable is which batch your request happened to land in.
The real culprit: batch invariance#
Here is the finding that reframes the whole problem. A kernel is batch-invariant if the result computed for your tokens is identical no matter how many other requests are batched alongside them. Most high-performance kernels are not batch-invariant — and that's not a bug, it's an optimization. To stay fast across wildly different batch sizes, kernels like RMSNorm, matrix multiplication, and attention change their internal reduction strategy — how they split a dimension into chunks and sum the chunks — based on the shape of the batch. Different split, different summation order, different last bit.
Now connect that to how inference actually runs. Your request doesn't get the GPU to itself. It's packed into a dynamic batch with whatever other requests arrived in the same window. The batch size is a function of concurrent server load — a variable you neither see nor control. So the same prompt, with the same weights at temperature 0, can return one answer when the server is quiet and a subtly different one under heavy traffic, because the kernel chose a different reduction path for a batch of 4 than for a batch of 128. The hidden input to your "deterministic" call is how busy the datacenter was.
That's why the nondeterminism feels haunted. It isn't correlated with anything in your code, because the thing that changed wasn't in your code.
The fix, and what it's worth#
The remedy is unglamorous and total: rewrite the offending kernels so their reduction order is fixed regardless of batch size. Thinking Machines did exactly this for those three kernels and got 100% bitwise-identical completions across 1,000 runs — the 80-way fork collapsed to a single answer. They shipped it as an open-source PyTorch library, batch-invariant-ops, with drop-in replacements, and demonstrated it as a deterministic mode in vLLM.
It isn't free. Batch-invariant kernels forfeit some of the load-adaptive tricks that maximize throughput, so you pay in tokens-per-second. That's why it's an opt-in mode, not the default: you flip it on where reproducibility earns its keep and leave it off where latency rules.
And reproducibility earns its keep in more places than test suites. Two matter most:
- Evals you can trust. If a model's score on your eval wobbles because the eval harness batched requests differently on Tuesday, you can't tell a real regression from numerical noise. Bitwise determinism makes an eval delta mean something — and it's a prerequisite for the reliability metrics, like pass^k, that only work if repeated runs are truly comparable.
- RL that's genuinely on-policy. Reinforcement learning assumes you're training on samples drawn from the current policy. If the inference engine that generated the samples and the training engine that consumes them disagree in the last bits, your "on-policy" updates are quietly off-policy — a bias that vanishes when sampling and training share identical numerics.
The lesson is bigger than one library. We keep treating LLM inference as a pure function of (prompt, weights, temperature) and are surprised when it behaves like a function of the datacenter's mood. It always was. The fix is to stop optimizing the one property — batch invariance — that we quietly needed all along.
