The one-line version: vLLM offloading is two separate mechanisms with one confusing name. --cpu-offload-gb offloads model weights to CPU — it does nothing for your KV cache. To offload the KV cache, you attach a KV connector, and the documented, stable path is LMCache on CPU RAM: vllm serve <model> --enable-prefix-caching --kv-transfer-config '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}' with LMCACHE_LOCAL_CPU=True and LMCACHE_MAX_LOCAL_CPU_SIZE=<GB>. Size that <GB> from 2 × layers × kv_heads × head_dim × dtype_bytes per token, and prove it's working with prefix-cache hit rate, median TTFT, and the p99 miss tail — not vibes.

This is the hands-on companion to two overviews we ran: the three 0.26 serving knobs worth turning and why the memory hierarchy is the fight between vLLM 0.26 and SGLang 0.5.16. Those tell you what shipped and why it matters. This one is for the person who read them, opened a terminal, and typed --cpu-offload-gb 40 expecting their KV cache to spill — and got nothing. Here is what to actually type, how to size it, and how to read the meters.

First, the trap: --cpu-offload-gb is not KV offload#

The single most expensive mistake here costs you an afternoon. --cpu-offload-gb (default 0) is a weights offload flag. Set it to 10 on a 24 GB card and vLLM treats you as if you had a 34 GB card — it keeps part of the model in CPU RAM and streams it into HBM on each forward pass. That's useful for fitting a model that otherwise wouldn't load. It is not what makes your KV cache spill when 100 concurrent long sessions overflow VRAM.

# This offloads WEIGHTS, not KV cache. It will not fix KV pressure.
vllm serve your-model --cpu-offload-gb 40

KV-cache offloading is a connector. vLLM exposes it through --kv-transfer-config, and the connector does the storing and fetching of KV blocks across a tier below HBM.

The minimal working KV-offload command#

The mature, documented option is LMCache. Attach it as the KV connector and give it a CPU-RAM budget:

export LMCACHE_LOCAL_CPU=True
export LMCACHE_MAX_LOCAL_CPU_SIZE=80      # GB of CPU RAM for the KV tier
export LMCACHE_CHUNK_SIZE=256             # tokens per stored chunk (LMCache default)

vllm serve your-model \
    --enable-prefix-caching \
    --kv-transfer-config '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}'

Three things to notice:

If you deploy through the vLLM production-stack Helm chart, the same thing is expressed as lmcacheConfig.enabled: true with cpuOffloadingBufferSize, which resolves to the kv_both connector under the hood.

The sizing math (do this before you pick a number)#

Guessing the tier size is how you either waste RAM or spill a working set that never gets reused. The math is small. Start with the per-token KV footprint:

kv_bytes_per_token = 2 × n_layers × n_kv_heads × head_dim × dtype_bytes

The 2 is for K and V. Use n_kv_heads (the grouped-query head count), not the attention head count — on modern GQA models that's much smaller, and getting it wrong inflates your estimate several-fold. dtype_bytes is 2 for fp16/bf16, 1 for an fp8 KV cache (--kv-cache-dtype fp8).

Worked example — a 32-layer model, 8 KV heads, head_dim 128, fp16:

2 × 32 × 8 × 128 × 2  =  131,072 bytes/token  ≈  128 KiB/token

Now scale it to a session and to your fleet:

bytes_per_session = kv_bytes_per_token × avg_context_tokens
# 128 KiB × 8,192  ≈  1 GiB per 8k-token session

warm_set = bytes_per_session × sessions_you_want_kept_warm
# 1 GiB × 200 warm sessions  =  200 GiB

Then subtract what HBM already gives you for free. On an 80 GB card at --gpu-memory-utilization 0.90, after ~40 GB of weights and a few GB of activation overhead, you have roughly ~28 GB left for KV — about 28 warm 8k sessions on the GPU. So:

offload_tier_GB ≈ (warm_set − hbm_kv_budget) × 1.15   # ~15% headroom
              ≈ (200 − 28) × 1.15  ≈  198 GB

Set LMCACHE_MAX_LOCAL_CPU_SIZE=200. The rule of thumb: size the offload tier to your target warm working set minus the HBM KV budget, plus ~15%. If that number is negative, your load fits in HBM and you don't need offload at all — leave it off (see the decision table above; recompute/evict is the right call for small steady load).

How to tell it's helping — three meters, in order#

Offload is a trade: you avoid recompute, you pay a copy. Whether that's a win is an empirical question, and vLLM exposes the meters to answer it. Turn on Prometheus metrics (/metrics) and watch these, in this order.

1. Prefix-cache hit rate — is anything being reused? This is the metric that gates everything else; we argued it's the number that decides your agent's bill. Derive it from vLLM's prefix-cache query/hit counters:

hit_rate = increase(vllm:prefix_cache_hits_total)
         / increase(vllm:prefix_cache_queries_total)

If the hit rate is near zero, offload cannot help — there's no reuse, so you're spilling blocks nobody asks for again. Stop and check that prefix caching is on and that your traffic actually shares prefixes. If the hit rate rises after you enable offload, blocks that used to be evicted are now surviving. Good sign — go to meter 2.

2. TTFT p50 — is the reuse actually saving prefill? A prefix-cache hit skips prefill, so a real win shows up as lower median time-to-first-token:

histogram_quantile(0.50, rate(vllm:time_to_first_token_seconds_bucket[5m]))

Hit rate up and TTFT p50 down = offload is doing its job. Hit rate up but TTFT flat = the copy cost is eating the prefill savings; your chunks may be too small or the interconnect too slow.

3. The miss tail — is offload hurting the unlucky requests? Every offload miss adds a fetch-and-copy before prefill can proceed. That lands in the tail, not the median. Watch TTFT p99:

histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m]))

vLLM 0.26 also added offload-specific meters for exactly this: basic offloading metrics, a CPU-cache read/write gauge split, and — the important one — a tiering-lookup-delay split into sync and async histograms (release-note-described; treat exact metric names as vendor-stated until you see them on your own /metrics). The async-delay histogram is your offload-miss latency signal. A bounded tail means the trade is paying off. A fat, growing tail means you're spilling a working set that doesn't fit even in the offload tier and thrashing it — shrink the warm-set target or add tier capacity.

The one-line checklist: helping = hit rate ↑, TTFT p50 ↓, p99 / async-lookup tail bounded. Hurting = hit rate ~0 (no reuse), or p99 tail blowing out (thrash). Nothing in between needs a meeting.

The tier below CPU RAM (0.26), and where docs run out#

The reason 0.26 is the version to write this against is that it matured the layers beneath CPU RAM. The release notes describe an object-store secondary tier with workload identity, DP-replica-aware tiering, tier-owned BlockStored events, and a blocks_per_chunk knob for heterogeneous KV groups. That's the difference between "spill to the RAM on this box" and "spill to a shared tier every replica can pull from."

Be honest about the maturity gap. The CPU-RAM path above is documented and stable. The object-store tier internals are, at time of writing, release-note-described rather than fully documented — so treat the specific object-store configuration as vendor-stated, benchmark it on your own traffic before you size a cluster around it, and don't invent flag names to fill the gap. For almost everyone reading this, single-node LMCache on CPU RAM is the 80/20, and the sizing-and-measurement loop above is the same regardless of which tier sits underneath.

Start with the CPU-RAM connector. Compute the tier size instead of guessing it. Then let the three meters tell you whether to keep it — because on a self-hosted stack, this is the layer that now sets your cost per concurrent user.