Kimi K3 has roughly 2.8 trillion total parameters, yet it stays cheap to serve because it almost never uses most of them at once. Three choices do the work. First, it is a sparse Mixture-of-Experts — Moonshot's "Stable LatentMoE" framework — so only about 16 of its 896 experts fire per token, which means the compute you pay per token tracks the small slice of active parameters, not the 2.8T headline. Second, Kimi Delta Attention (KDA), a linear/delta-rule attention variant, keeps long-context decode from exploding: Moonshot credits it (as reported) with up to ~6.3× faster decoding at million-token context, because a bounded recurrent state replaces the KV cache that would otherwise grow with every token. Third, Attention Residuals is a training-time change credited with ~25% higher training efficiency at under ~2% extra compute — it lowers the cost of building the model, not of running it. In short: sparse routing cuts per-token FLOPs, KDA cuts long-context memory and latency, Attention Residuals cut the training bill. Which model to actually pick is a separate question — we cover that comparison here — this piece is about why the serving economics work.

Every number here is reported by Moonshot or secondary coverage as of 2026-07-27 and should be read as "as reported." Weight sizes in particular conflict across sources — verify against the official weights and read the license file yourself before you provision or ship.

Sparse MoE: you pay for the experts you use, not the ones you own#

The instinct with a 2.8T-parameter model is to assume 2.8T-worth of math per token. That is a dense-model instinct, and K3 is not dense. In a sparse MoE, each layer holds a large pool of expert sub-networks — 896 here — and a router picks a handful per token. Only the ~16 selected experts run; the other ~880 sit idle. So the active parameter count that drives compute is a small fraction of the total, even though the whole set still has to live in memory across your cluster.

That split — total parameters set your memory bill, active parameters set your compute bill — is the entire trick. It is also why which experts fire depends on routing, one reason LLM inference isn't bit-for-bit deterministic once you batch requests.

What it means: your cost-to-serve per token is governed by the ~16 active experts, not the 2.8T total — but you still need enough aggregate GPU memory to hold every expert resident, which is why this is a cluster model, not a single-card model.

Kimi Delta Attention: bounding the long-context tax#

Standard softmax attention has a well-known long-context problem: it keeps a KV cache that grows linearly with the sequence, so at a million tokens each decode step drags an enormous, ever-expanding cache through memory to emit one token. That is the memory-bound wall that makes long-context serving expensive.

KDA is Moonshot's answer — a linear / delta-rule attention variant. The principle behind that whole family is simple: instead of storing every past key and value, you maintain a fixed-size recurrent state updated as tokens arrive. The state does not grow with context, so the per-step decode cost stops scaling with how deep into the window you are — the mechanism behind the ~6.3× faster decoding at 1M context Moonshot reports.

Full attention remembers everything and pays for it every token. Delta attention keeps a fixed-size summary — the context can grow to a million tokens and the per-step cost barely notices.

Be precise about the claim: the intuition — bounded state beats an unbounded KV cache — is the established reason linear/delta schemes decode faster. The exact internal formulation of KDA and the 6.3× figure are Moonshot's, reported, and worth treating as such until independently reproduced. A single "tokens/sec" number will still mislead you, because prefill and decode are different workloads and KDA mainly attacks the decode side.

What it means: at long context, KDA is what keeps latency and memory from ballooning — the practical unlock is that 1M-token serving stays tractable instead of pricing itself out. On the tooling side, vLLM 0.7.0+ reportedly ships native KDA support, so you are not hand-rolling the kernel.

Attention Residuals: a training-time win, not a serving one#

It is easy to lump every "K3 is efficient" claim together, so keep this one in its own box. Attention Residuals reportedly buys ~25% higher training efficiency for under ~2% additional compute. That is a fantastic trade — but you cash it at training time. It helps explain how Moonshot built a 2.8T frontier model economically; it is not the reason your inference bill is low. KDA and sparse MoE are the serving-time levers; Attention Residuals is the build-time one.

What it means: when you are estimating cost-to-serve, ignore this number — it does not touch your inference math. It matters only as context for why K3 exists at this quality-per-dollar.

What this actually costs you to run#

The architecture makes K3 efficient per token; it does not make it small. The MXFP4 open weights are reported at around 1.4 TB on disk, though some sources cite ~594 GB for a different quantization — resolve that conflict before provisioning, not after. Either way, no single GPU holds it. Moonshot reportedly recommends 64+ accelerators for a full deployment, with smaller quantized (INT4) ~256K-context setups around 8×H100, served via vLLM, SGLang, or TensorRT-LLM.

That is the fork in the road. Hosted, Moonshot's API is reported at $3 per 1M input and $15 per 1M output tokens, dropping to ~$0.30/1M on a cache hit, reportedly with no long-context premium above 200K — notable, given the whole KDA story is cheap long context. Owning the cluster only beats that if you keep 64 accelerators genuinely busy. The full rent-vs-own math — including the managed third-party hosts between DIY and the first-party API — is its own build-out guide.

What it means: the per-token efficiency is real, but self-hosting is still a cluster commitment with a terabyte-scale footprint. If your utilization is spiky or your volume modest, the hosted API's cache-hit pricing and flat long-context rate are hard to beat; self-hosting wins on sustained, high-utilization workloads.

The one-line model to keep#

Strip it down: sparse MoE decouples your compute bill from the parameter count, KDA decouples your long-context cost from sequence length, and Attention Residuals decoupled Moonshot's training bill from model size. Three different decouplings doing three different jobs — conflating them is how people get K3's economics wrong. The one line for your cost model: the 2.8T number sets your memory footprint, the ~16 active experts set your compute per token, and KDA keeps the million-token context from taxing both. For the wider context on the open-weights drop, see the founders' wire roundup.