The bug always shows up in production, never in the demo. Your agent has parsed the model's JSON cleanly a thousand times, and then one afternoon it emits a trailing comma, or wraps the object in ``` `json ``, or renames total_price to totalPrice, and your json.loads` throws — mid-transaction, in the one code path you didn't wrap in a retry. If you self-host your model, you don't have to live with this. Constrained decoding makes invalid output structurally impossible, and on modern backends it's faster than the prompt-and-pray approach you're replacing.

The idea in one paragraph#

At each generation step, a language model produces a probability over its whole vocabulary and samples one token. Constrained decoding (also called guided or structured decoding) inserts a step in between: it computes a mask of which tokens could still lead to a valid output — valid against your JSON Schema, regex, or grammar — and sets every other token's probability to zero before sampling. The model can only ever pick a token that keeps the output legal. There is no invalid string to backtrack from because the invalid strings were never reachable. XGrammar, the default engine in vLLM and SGLang, computes that mask for a JSON schema in tens of microseconds, so the overhead is negligible.

vLLM: the online server#

Start vLLM's OpenAI-compatible server as usual (XGrammar is the default structured-outputs backend, so there's nothing extra to configure):

vllm serve Qwen/Qwen3-8B-Instruct --port 8000

Now constrain the output. The cleanest path is the OpenAI-standard response_format with a JSON Schema — and the ergonomic way to build that schema is a Pydantic model:

from openai import OpenAI
from pydantic import BaseModel

client = OpenAI(base_url="http://localhost:8000/v1", api_key="-")
model = client.models.list().data[0].id

class Invoice(BaseModel):
    vendor: str
    total_cents: int
    currency: str
    paid: bool

completion = client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "Extract the invoice: 'Acme billed us $412.50, USD, unpaid.'"}],
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "invoice", "schema": Invoice.model_json_schema()},
    },
)

invoice = Invoice.model_validate_json(completion.choices[0].message.content)

That model_validate_json will never raise on a malformed shape, because the decoder could not produce one. For the other constraint types, vLLM exposes an extra_body field called structured_outputs:

# Classification — output is exactly one of the choices, nothing else:
completion = client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "Sentiment of: 'this shipped late again'"}],
    extra_body={"structured_outputs": {"choice": ["positive", "negative", "neutral"]}},
)

# Fixed format — output matches a regex:
completion = client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "Give a support ticket id"}],
    extra_body={"structured_outputs": {"regex": r"TICKET-\d{6}"}, "stop": ["\n"]},
)

(On older vLLM builds these same options are spelled guided_json, guided_choice, guided_regex, and guided_grammar inside extra_body — the semantics are identical; check your version's structured outputs page.)

vLLM: offline batch inference#

If you're running vLLM in-process for a batch job rather than behind the OpenAI-compatible server, the same constraints live directly on SamplingParams. You build a StructuredOutputsParams object carrying your schema and hand it to the sampler, and every prompt in the batch is then decoded under that same constraint with no per-request wiring. Here is the offline equivalent of the invoice extractor above:

from vllm import LLM, SamplingParams
from vllm.sampling_params import StructuredOutputsParams

llm = LLM(model="Qwen/Qwen3-8B-Instruct")

params = SamplingParams(
    structured_outputs=StructuredOutputsParams(json=Invoice.model_json_schema())
)
out = llm.generate("Extract the invoice: 'Acme billed us $412.50, USD, unpaid.'", params)
print(out[0].outputs[0].text)  # guaranteed to match the Invoice schema

StructuredOutputsParams accepts json, regex, choice, grammar, structural_tag, and whitespace_pattern — the same menu as the server.

SGLang: the same guarantee, same engine#

SGLang serves the same OpenAI-compatible response_format, and it also uses XGrammar as its grammar backend under the hood. That means a JSON Schema you validated against vLLM ports over to SGLang completely unchanged — the guarantee and the wire format are identical, and only the server behind the port is different. Point your client at the SGLang endpoint and send the very same request:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="-")

completion = client.chat.completions.create(
    model="Qwen/Qwen3-8B-Instruct",
    messages=[{"role": "user", "content": "Extract the invoice: 'Acme billed us $412.50, USD, unpaid.'"}],
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "invoice", "schema": Invoice.model_json_schema()},
    },
)

SGLang additionally exposes regex and EBNF grammar fields in extra_body for the cases a plain schema can't express.

The one thing that trips people up: recursion#

A regex or an FSM-based backend can validate flat objects all day, but it cannot represent an arbitrarily nested structure — a comment tree, a nested category, an expression AST — because a finite-state machine has, by definition, finite states. FSM-based backends handle a recursive JSON Schema by flattening it to a fixed depth or rejecting it. If your shape is genuinely recursive, you need a context-free grammar backend — XGrammar and llguidance both qualify — and you express the shape as a CFG (grammar=...) rather than a JSON Schema. (If you're choosing between engines rather than just using vLLM's default, the tradeoffs are laid out in Outlines vs XGrammar vs llguidance.) This is the single most common reason a "constrained decoding doesn't work for my schema" complaint turns out to be a backend mismatch, not a bug.

When to reach for which constraint#

Use choice for classification and routing labels — it's the tightest guarantee and the cheapest to compile. Use regex for rigid formats like IDs, dates, and phone numbers. Use a JSON Schema (json / response_format) for the common case of a typed object feeding typed downstream code — this is what most agent tool-calls and extraction pipelines want. Save the raw grammar for recursion or a real DSL. And if you notice quality dropping, you're probably constraining too early: let the model reason in free text (or a dedicated reasoning string field) and constrain only the final answer. Structure the output, not the thinking.