Two releases landed a day apart this week, and if you self-host LLM inference you probably run one of them. vLLM v0.25.0 shipped July 11; SGLang v0.5.15 shipped July 10. Each has a headline throughput win. And here is the part worth noticing: they are the same win, reached from opposite ends of the stack.

The disease both of them treated#

For years the mental model of "make inference faster" was "write a hotter kernel." That era is mostly over. The core matmuls are close to saturated on modern hardware, and the big algorithmic ideas — paged KV cache, continuous batching, tree speculative decoding — are mature and widely implemented. So where is the throughput still hiding?

In the stall. Before each generation step, the host (CPU) does bookkeeping — building input tensors, scheduling the next batch, preparing draft candidates for verification. The accelerator can't start until that work arrives, and often the host can't prepare the next step until the current one returns. That handoff is a synchronization point, and at the small-to-medium batch sizes that real agent traffic produces — one user, a few concurrent tool calls, a stream of short completions — the GPU spends a genuinely embarrassing fraction of its time idle, waiting on the host.

Kernels are saturated and the algorithms are mature. The throughput left on the table in mid-2026 is a GPU standing still, waiting for a CPU to hand it the next batch. That's the fight both engines picked this week.

Same fight, two different organs#

vLLM went after it in the model runner. v0.25 finishes making Model Runner V2 the default for all dense models. MRv2's whole thesis is an invariant: zero CPU–GPU synchronization. Input preparation runs on the GPU with Triton kernels, and the scheduler prepares step N+1 while the device is still executing step N, with a buffer discipline that removes the races so no barrier is needed. The host work and the device work stop taking turns.

SGLang went after it in the speculative-decoding scheduler. SGLang already made Spec V2 the default path back in v0.5.13; v0.5.15 makes that path zero-overhead. The draft-extend step became CUDA-graphable, the device-to-host and host-to-device syncs were dropped, and the metadata operations were fused — worth roughly +11% end-to-end tokens per second on top of the acceptance-rate win speculation already delivers, per the release. Speculative decoding is a loop of draft-then-verify handoffs; every handoff was a chance to stall, and v0.5.15 closes them.

Two independent teams, one week, the same root cause. When two of the most-watched projects in a field converge on the same fix from different directions, that's not a coincidence — it's the field telling you where the frontier moved.

Which one moves your bill#

This is not a "switch engines" moment. vLLM and SGLang trade the lead release to release, and the right engine is still decided mostly by how much context your requests share and whether you're throughput- or latency-bound. But if you're deciding what to re-benchmark this week:

And if you buy inference from a hosted provider rather than running an engine yourself, none of this is a task on your board — but it's not nothing either. You inherit these gains as slightly lower latency or price whenever your provider upgrades. The lever you actually have is a question: which engine and which version are you serving me on? Anyone spending real money on tokens is entitled to ask, and the answer tells you how fast improvements like this week's reach your invoice.

The models get the headlines. But the reason your token bill drifts down quarter over quarter isn't a new model — it's two serving teams, an ocean apart, both deciding this month that the most expensive object in the datacenter is a GPU that's waiting.