Ask a developer to name an LLM router and you'll get LiteLLM, Portkey, OpenRouter, maybe RouteLLM. They're all the same shape: an application-layer HTTP proxy you call instead of a model, which then picks which model to hit based on a rule, a price ceiling, or a small routing model. Useful, well-understood, and — architecturally — a box that sits beside your app.
The vLLM Semantic Router is not that box. It runs one layer down, inside the proxy data plane, and it routes on a question none of the others ask: should this request reason at all?
Reasoning as a routing decision#
Here's the finding that makes the project worth your attention, from its underlying paper, When to Reason: Semantic Router for vLLM (IBM Research). On MMLU-Pro, deciding per request whether to enable reasoning — instead of always leaving it on — produced +10.2 percentage points of accuracy while cutting latency 47.1% and token consumption 48.5%, measured against plain vLLM inference.
Read that twice, because it breaks the usual mental model. Routing is supposed to be a cost lever you pull at the expense of quality: cheaper model, worse answer. Here quality went up as cost went down. The reason is that reasoning models overthink simple questions. Forced chain-of-thought on an easy prompt can reason a model right out of the correct answer, so "always reason" quietly taxes accuracy on the easy slice of your traffic. Turn it off there and you win twice.
"Always reason" is not a safe default. It's a setting that costs you both tokens and, on easy questions, correctness.
The decision is made by a fine-tuned ModernBERT classifier that reads each request's intent and complexity — across the fourteen MMLU-Pro domains in the eval, and generalizing to your own categories — and picks both a model and a reasoning mode.
Why it lives in the data plane#
The architectural choice that makes this deployable is where it runs. The Semantic Router is an Envoy external-processing (ext_proc) gRPC filter. Envoy intercepts the request, hands it to the router over gRPC, and the router classifies, applies policy, and mutates headers to steer the request — all before it reaches any backend. There's no application code in the path. The classifier core is written in Rust with Go bindings into Envoy, so the per-request overhead is a small CPU classification, not another GPU call; the heavy lifting stays in the vLLM backends the router points at.
Living in the data plane buys two things beyond routing. First, it's Kubernetes- and service-mesh-native by construction — the same place your other Envoy filters already run. Second, it makes inline safety free: jailbreak detection, PII and sensitive-data screening run in the same pass as intent classification, before the prompt ever hits a model, rather than as a bolted-on middleware hop.
Where it fits — and where it doesn't#
This is not a drop-in replacement for a LiteLLM import. If you want multi-provider access from a few lines of Python, the app-layer gateways are still the fast path, and we've mapped how semantic routing compares to plain LLM routing and where a cascade beats a router. The Semantic Router earns its complexity when you're already running Envoy or a service mesh, you serve a mixture of models that genuinely diverge on cost, quality, latency, and privacy, and you want routing plus safety enforced as infrastructure rather than as code every team has to remember to call.
The cost is real: you run Envoy and a classifier, and you own a young project — the releases are landing steadily since January 2026, but the ecosystem is still forming. Treat it as early-adopter infrastructure with an unusually strong core idea.
That core idea is the part worth taking even if you never deploy the router: the most valuable thing your gateway can decide isn't which model to call. It's whether the expensive part of the model — the reasoning — should run this time at all. On a lot of your traffic, the honest answer is no, and saying so makes your agents both cheaper and, on the easy questions, right more often.
