If you read one line: Speculative decoding buys lower latency by spending idle GPU compute — so it wins at low batch size and hurts once batching has used that idle compute up. vLLM's own numbers: up to 2.8× faster at low QPS, but 1.4×–1.8× slower at high QPS. If your endpoint got slower when you enabled it, you're serving above your crossover. Benchmark on-vs-off at your real concurrency and route by load.
Someone enables EAGLE, Medusa, or SGLang's new DSpark, sees the demo's 2–3× number, ships it to a busy endpoint, and watches tokens/sec go down. Then they file a bug. It isn't a bug. It's the technique doing exactly what it's designed to do, in a regime it was never designed for.
The one mechanism that explains all of it#
Decoding tokens one at a time on a large model is memory-bandwidth-bound. The GPU spends most of each step waiting to stream the model's weights from memory; its compute units — the FLOPs — sit largely idle. That idle compute is the entire opportunity speculative decoding exploits.
A small draft model proposes the next several tokens; the big target model verifies all of them in a single forward pass. Every token it accepts is a token you produced without a separate memory-bandwidth-bound decode step. As vLLM's docs put it, when there's a single sequence in the batch, decoding is so memory-bound that "large numbers of FLOPs can be consumed in the verification step... without significantly increasing the latency per decoding step" (vLLM docs). You're paying with compute you weren't using anyway.
Now add concurrency. Batch enough sequences together and the GPU stops waiting on memory and starts being compute-bound — the FLOPs are busy doing real decode work across all those streams. The idle compute speculative decoding was spending is gone. And now the drafting-and-verifying work — including every draft token you reject, which is pure waste — competes with useful requests. Same feature, inverted economics.
Speculative decoding doesn't make decoding cheaper. It moves work from the memory-bandwidth bill to the compute bill — a great trade when compute is free and a terrible one when it's the thing you're out of.
The crossover is real, and vLLM measured it#
This isn't hand-waving. vLLM's speculative-decoding launch numbers show up to 2.8× speedup at low QPS — and, at high QPS on Llama3-70B with 4×H100, roughly a 1.4× slowdown on ShareGPT and 1.8× on CNN DailyMail. The docs name the reason directly: at larger batch sizes verification "becomes compute-bound," and the tokens verified per sequence per step must be "more tightly controlled to avoid... entering the regime where speculative decoding becomes unprofitable."
So there is a batch size — a QPS — where the two throughput curves cross. Below it, spec decode wins. Above it, spec decode is a tax you're paying to go slower.
Where your crossover sits (the four dials)#
The crossover isn't a constant; it moves with:
- Draft acceptance rate. The fraction of draft tokens the target accepts. High acceptance means fewer wasted verifications, which pushes the crossover to a higher batch size. A poorly-matched draft model can put your crossover at batch size two.
- Draft tokens per step. Verify more tokens per pass and you win bigger at low batch — and lose bigger at high batch, because rejected tokens cost more. This is the dial vLLM says to "tightly control" under load, and the reason adaptive schemes exist: DSpark sizes the verify window from the draft's own confidence, wasting less on hard spans and nudging the crossover higher — but not to infinity.
- Model and hardware. A bigger model on bandwidth-starved hardware stays memory-bound longer, so its crossover is higher. Fast-memory accelerators become compute-bound sooner.
- Your prefill/decode mix. If your traffic is prefill-heavy (long prompts, short outputs), decode — the part spec decode accelerates — is a smaller slice of the bill, so the whole technique matters less either way.
How to find your own crossover (the only honest method)#
Don't inherit someone else's batch-size-1 benchmark. Run two servers, identical except for the flag:
- Server A: your model, speculative decoding off.
- Server B: same model, speculative decoding on (EAGLE/Medusa/DSpark, your draft config).
- Sweep concurrency — 1, 2, 4, 8, 16, 32, up to your real peak — and at each level record tokens/sec and p50 / p99 latency, using the discipline in how to benchmark LLM inference.
- Plot both throughput curves. The batch size where B drops below A is your crossover. The latency curves tell you how much interactive users gain below it.
Then act on the graph:
- Interactive, low-concurrency serving (a coding agent, a single-user copilot) lives below the crossover — leave spec decode on and enjoy the latency win.
- Busy multi-tenant endpoints live above it — turn spec decode off, or cap draft tokens hard, and get your wins from batching and prefix reuse instead.
- Spiky traffic that crosses the line both ways is the case for load-based routing: speculate when the queue is short, stop when it fills. That's a real lever now that the crossover is a measured number on your own graph, not a vibe.
The takeaway a founder can act on#
Speculative decoding is marketed as "up to N× faster," and that number is true — in the regime it was measured in. It is a latency optimization for latency-bound serving, and it can cost you real throughput the moment your GPU is busy. Before you ship it to production, spend one afternoon running the two-server sweep. The crossover you find is worth more than any headline multiplier, because it's the one that describes your traffic — and it tells you not just whether to turn the feature on, but exactly when to turn it off.
