You already decided why serverless: an agent or a low-traffic product doesn't keep a GPU busy, so paying by the hour for a rented card burns money on idle. (If you're still weighing platforms, that's the cost decision we just ran — RunPod, Modal, and Baseten, and the per-second-vs-per-minute detail that decides your invoice.) This piece is the how: two concrete paths to a scale-to-zero endpoint on RunPod, one with no code and one with a handler you own.

The one-screen version: for a standard open LLM, deploy RunPod's prebuilt vLLM worker from the Docker registry and configure it with env vars — no handler code. For anything custom, write a thin handler.py that turns event['input'] into a result, bake it into a Docker image, and deploy. In both cases, set minimum workers to 0 so the endpoint scales to zero and costs nothing when idle.

Path 1: the vLLM worker (no code)#

If you're serving a standard open-weights model and an OpenAI-compatible API is enough, you don't write any Python. RunPod maintains an official vLLM worker image; you configure it entirely with environment variables.

In the RunPod console:

  1. Serverless → New Endpoint → deploy from Docker registry, using RunPod's vLLM worker image.
  2. Set the endpoint type to queue-based.
  3. Add an environment variable pointing at your model, e.g. MODEL_NAME=Qwen/Qwen3-8B (a Hugging Face repo).
  4. Tune the engine with more env vars — the worker auto-discovers any environment variable that matches a vLLM engine argument, uppercased (context length, dtype, tensor-parallel size, and so on).
  5. Set min workers to 0 for scale-to-zero, max workers to your ceiling.

You get an OpenAI-compatible endpoint. Call it like any chat API, with your endpoint ID and API key:

curl -X POST https://api.runpod.ai/v2/<ENDPOINT_ID>/openai/v1/chat/completions \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-8B", "messages": [{"role": "user", "content": "Say hi"}]}'

That's the whole deploy. Confirm the current image tag and env-var names in RunPod's docs — the worker moves fast — but the shape has been stable: a prebuilt image, configured by env vars, no handler.

Path 2: a custom handler (your own model or logic)#

When you need a custom model, non-LLM inference, pre/post-processing, or your own request logic, you write a handler. The RunPod serverless contract is small: a function that takes an event and returns a JSON-serializable result, plus one line to hand it to the worker runtime.

handler.py:

import runpod

# Load once, at module import — this runs while the worker warms,
# not on every request. Heavy model loads belong here.
# model = load_your_model()

def handler(event):
    # RunPod delivers your request body under event["input"].
    payload = event["input"]
    prompt = payload.get("prompt", "")

    # ...run your inference here...
    result = f"echo: {prompt}"

    # Return anything JSON-serializable; it becomes the job output.
    return {"output": result}

# Hand the handler to RunPod's worker runtime and block.
runpod.serverless.start({"handler": handler})

That's the entire core — define handler(event), return a dict, call runpod.serverless.start. The platform owns the queue, autoscaling, and per-second billing; your job is only to turn an input dict into an output dict.

Bake it into an image. Dockerfile:

FROM runpod/base:0.6.2-cuda12.4.1

COPY requirements.txt /requirements.txt
RUN pip install --no-cache-dir -r /requirements.txt

# Bake weights into the image (fast cold start, bigger image), or
# pull them at load time in handler.py (small image, slower first boot).
COPY handler.py /handler.py

CMD ["python3", "-u", "/handler.py"]

requirements.txt at minimum:

runpod
# + your inference deps: vllm, transformers, torch, etc.

Build, push to a registry, and create the endpoint from that image exactly as in Path 1 — queue-based, min workers 0, max workers to taste.

Scale-to-zero and cold starts: the setting that saves the money#

The whole reason you're here is the minimum worker count:

RunPod's FlashBoot cuts cold-start time on recently-active endpoints (sub-200ms advertised). Two levers you control matter as much: bake weights into the image instead of pulling them at boot (faster first request, larger image), and do all heavy loading at module import in handler.py — outside handler() — so it happens once while the worker warms, not on every call. For a deeper treatment of the trade, see our piece on scale-to-zero and GPU cold starts.

Test it#

Short, synchronous calls use /runsync; long jobs use /run and then poll /status/{id}:

# synchronous — returns the result directly
curl -X POST https://api.runpod.ai/v2/<ENDPOINT_ID>/runsync \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"input": {"prompt": "hello"}}'

The body is always {"input": {...}} — exactly what your handler reads from event["input"] (or, for the vLLM worker, an OpenAI-style payload). Endpoint ID and API key come from the console.

That's the deploy. You now have a GPU endpoint that costs nothing at rest and bills by the second under load — which was the entire point. When you're deciding which platform to run this pattern on, the RunPod vs Modal vs Baseten cost breakdown is the companion to this how-to.

RunPod's images, base tags, and console flow change over time — treat the specific tags and variable names here as illustrative and confirm the current ones in RunPod's docs (linked below) before you ship.