The short version: the best way to deploy an LLM in production in 2026 is usually to not self-host it — use a managed API until you hit a concrete reason to run your own, and when you do, the default stack is vllm serve on a rented GPU behind a gateway. This is the whole path, in the order you actually make the decisions, with the exact commands and the places each one bites a small team. If you need a working /v1 endpoint this week, start at Decision 1 and stop the moment a managed API answers your need.

Here's the map:

  1. Managed API vs self-host — the decision that saves the most time.
  2. Which model, and at what precision — weights and quality.
  3. How big a GPU — weights plus KV cache.
  4. Which serving engine — vLLM is the default.
  5. Where to rent the hardware — by the hour, usually.
  6. Make it production-grade — scaling, batching, and guardrails.

Decision 1: Don't self-host until you have to#

The fastest deployment is an API key. A managed endpoint from OpenAI, Anthropic, Google, or an open-model host like Fireworks or Together gives you a production-grade /v1 endpoint with autoscaling, upgrades, and uptime someone else owns. At low-to-moderate volume it is also cheaper than self-hosting once you count your own hours — a dedicated GPU bills 24/7 whether or not you send it traffic.

Self-host when you hit a specific trigger, not on principle:

If none of those is true yet, your "deployment" is an API key plus a gateway (Decision 6). Come back when one becomes true.

Decision 2: Pick the model and the precision#

Choose the smallest open model that passes your evals, not the biggest one you can fit. A well-chosen 8B–32B model that clears your task is cheaper and faster to serve than a 70B you picked for the leaderboard. Run your own eval set before you commit — public benchmarks don't measure your task.

Precision is the lever that decides how much GPU you need. Weights in FP16 cost ~2 bytes per parameter; FP8 or a pre-quantized AWQ/GPTQ checkpoint roughly halves that, letting a smaller card serve a bigger model. On H100/H200-class hardware, FP8 is the common default. The catch: quantization can quietly regress quality on hard tasks, so measure your eval at the precision you'll actually ship, not just at FP16.

Decision 3: Size the GPU — weights first, then KV cache#

Two numbers decide the card:

We keep a dedicated walkthrough of how much VRAM it takes to serve an LLM and a card-by-card comparison in H100 vs H200 vs A100 vs L40S and B200 vs H200 vs H100 — read those before you rent, because guessing the card wrong is the most expensive mistake on this list.

Decision 4: Choose the serving engine — vLLM is the default#

For most teams, vLLM is the answer. It's an inference server that does the two hard things — continuous (in-flight) batching and paged KV-cache attention — so you get high throughput without writing an inference loop, and it exposes an OpenAI-compatible API, which means you migrate an app off the OpenAI SDK by changing one base_url.

Install and serve:

pip install vllm

# Stands up an OpenAI-compatible server on http://localhost:8000/v1
vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 2 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90

Call it with the OpenAI client you already have — only the base_url changes:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="local")

resp = client.chat.completions.create(
    model="meta-llama/Llama-3.3-70B-Instruct",
    messages=[{"role": "user", "content": "Ship it."}],
)
print(resp.choices[0].message.content)

Prefer a container in production — the official image pins CUDA and drivers for you:

docker run --gpus all -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen3-32B --gpu-memory-utilization 0.90

Reach past vLLM only for a reason: TensorRT-LLM for the lowest possible latency on NVIDIA (at the cost of a heavier build), or a different runtime if you're on non-NVIDIA silicon. The full trade-off is in vLLM vs TensorRT-LLM vs TGI; if you're serving many fine-tunes of one base model, see multi-LoRA serving. For where a higher-level framework like BentoML or Ray Serve fits on top, see BentoML vs Ray Serve vs KServe.

Decision 5: Rent the GPU — usually by the hour#

You almost never buy hardware to start. Hourly GPU rental from CoreWeave, Lambda, Nebius, RunPod, or Together gets you an H100/H200 in minutes, and you only own the box for as long as your load justifies it. Owning or reserving pays off only under sustained 24/7 load over many months.

The rates move constantly, so check them the day you deploy: our GPU rental price map tracks live H100/H200/B200 pricing, and CoreWeave vs Lambda vs Nebius plus where to rent a GPU to serve an open model compare the providers on price, availability, and cold-start. And keep an eye on the supply chain: the hyperscalers are vertically integrating custom silicon, which is part of why the rent-vs-own math keeps favoring rent for a small team.

Decision 6: Make it production-grade#

A bare vllm serve is a working endpoint, not a production one. Three additions turn it into something you can put behind a paying product:

The one-screen checklist#

  1. Can a managed API do it? If yes, stop — use it behind a gateway.
  2. Pick the smallest open model that passes your evals, at the precision you'll ship (measure quantization loss).
  3. Size the GPU: weights (~2 bytes/param FP16) + KV cache headroom; 70B needs 2+ cards or quantization.
  4. vllm serve <model> → OpenAI-compatible /v1; containerize it.
  5. Rent by the hour; check live prices; own only under sustained load.
  6. Gateway + right-sized scaling + guardrails. Never leave a GPU idle at 24/7.

Deploy the boring version first — one model, one GPU, one gateway — get real traffic on it, and let the bottleneck you actually hit (cost, latency, or concurrency) tell you which decision to revisit. Every step above is one you can defer until the traffic demands it.