The short version: gpt-oss-120b is a 117B-parameter mixture-of-experts model with only 5.1B active parameters, released under Apache 2.0. It fits on one 80GB GPU — an H100 or an MI300X — because OpenAI post-trained it with MXFP4 quantization on the MoE weights, shrinking the weight footprint to about 60GB. Serve it with vllm serve openai/gpt-oss-120b, point your existing OpenAI-compatible agent code at the endpoint, and you have a private model backend with function calling built in. The one mistake that will burn an afternoon: the model only works in the harmony response format, so never feed it raw strings.

This is the model teams still reach for when they want an OpenAI-shaped model on their own metal — no per-token bill, no data leaving the building. If you want the head-to-head on which server to run it under, we split that out in vLLM vs llama.cpp for serving gpt-oss. This piece is the setup.

1. Check the memory math before you rent the GPU#

The headline "117B parameters" is misleading in two useful directions. First, it's a sparse MoE: only 5.1B parameters fire per token, so inference is far faster than 117B dense. Second, the MoE weights ship in MXFP4 (a 4-bit floating-point format), which is why the weights land near 60GB instead of 234GB at bf16.

That leaves the arithmetic simple for an 80GB card:

~60 GB   MXFP4 weights
~15 GB   KV cache + activations (your --max-model-len × batch)
--------
~75 GB   → fits an 80GB H100 / MI300X with a little headroom

What it means: the KV cache is the variable you control. If you set a long context and a big batch, you'll OOM even though the weights fit. Start with a modest --max-model-len, confirm it loads, then grow it. The smaller gpt-oss-20b (21B/3.6B active) is the sibling for 16GB consumer cards and edge boxes when you don't need the 120b's reasoning.

2. Serve it with vLLM (the OpenAI-compatible path)#

vLLM is the production route because it gives you an OpenAI-compatible server — your agent code doesn't change, only its base_url does.

# vLLM with native gpt-oss / MXFP4 support
uv pip install --pre "vllm[gptoss]"

vllm serve openai/gpt-oss-120b \
  --quantization mxfp4 \
  --kv-cache-dtype fp8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90

Now point any OpenAI SDK at it:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="local")

resp = client.chat.completions.create(
    model="openai/gpt-oss-120b",
    messages=[{"role": "user", "content": "List 3 uses for a raspberry pi."}],
    reasoning_effort="low",   # low | medium | high
)
print(resp.choices[0].message.content)

What it means: vLLM applies the harmony format for you, so the /v1/chat/completions and function-calling calls you already wrote against OpenAI keep working. A single node lands around 58 tokens/sec on MXFP4 in published benchmarks — plenty for a background agent, and it scales across nodes if you need concurrency. Prefer a one-line local test first? ollama pull gpt-oss:120b then ollama run gpt-oss:120b gets you talking to it in minutes, and Ollama also exposes an OpenAI-compatible endpoint.

3. Respect harmony — or get silent garbage#

This is the failure that isn't in the error log. Both models were trained on the harmony response format and only work with it. The symptoms of ignoring it are null content, leaked reasoning channels, or incoherent tokens — not an exception.

You almost never format harmony by hand: let the server do it. vLLM, Ollama, and the Transformers chat template all apply it. Only if you're calling model.generate directly do you need the openai-harmony package to build the token sequence yourself:

# Only needed for raw generation — servers handle this for you
from openai_harmony import load_harmony_encoding, HarmonyEncodingName
enc = load_harmony_encoding(HarmonyEncodingName.HARMONY_GPT_OSS)

What it means: if you're seeing empty responses from a self-hosted gpt-oss, the cause is almost always a missing harmony template, not the weights. Route through a server and the problem disappears.

4. Wire the tool calls — it's built for agents#

gpt-oss was post-trained for function calling, structured outputs, and tool use — the reference repo even ships browser (search/open/find) and Python tools. Because vLLM speaks the OpenAI tools schema, your agent loop is unchanged:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get weather for a city",
        "parameters": {"type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"]},
    },
}]

resp = client.chat.completions.create(
    model="openai/gpt-oss-120b",
    messages=[{"role": "user", "content": "Weather in Lagos?"}],
    tools=tools,
    reasoning_effort="medium",
)
tool_calls = resp.choices[0].message.tool_calls

What it means: the model returns standard tool_calls; you execute them and feed results back exactly as you would against a hosted model. The one gpt-oss-specific lever worth using in the loop is reasoning_effort — drop it to low on turns that only dispatch a tool, raise it to high for planning. Reasoning tokens are the variable cost once the GPU is paid for, so gating them per turn is your biggest bill control, the same way one tokens-per-second number can lie to you about real cost.

The whole loop, honestly: rent an 80GB GPU, vllm serve openai/gpt-oss-120b, repoint your base_url, keep effort low by default. You get a private, Apache-2.0, tool-calling model backend with no per-token meter running — and the only real trap, harmony, is handled the moment you serve through vLLM instead of generating raw. Next question is throughput versus simplicity, which is a server choice: vLLM vs llama.cpp.