Moonshot dropped Kimi K3's full weights on Sunday, July 27, and the first thing most self-hosters do with a new open model is reach for a quantizer. With K3, that reflex is wrong — and so is the second one, about hardware. Both mistakes come from the same misunderstanding: the ~1.56TB you just downloaded is not a compressed copy of a bigger model. It is the model.
The weights are 4-bit because the model was trained that way#
K3 was quantization-aware trained (QAT) in MXFP4 across the entire post-training stage — both supervised fine-tuning and reinforcement learning. The network learned to compute in low-precision numerics from the start, so there is no train-versus-inference mismatch to paper over and, crucially, no BF16 checkpoint sitting behind the release. The MXFP4 weights are the reference model.
That single fact rewrites the standard playbook. Normally, open weights arrive in BF16 and you run AWQ or GPTQ to INT4 so the thing fits your GPUs, accepting a small, well-studied quality hit measured against the original. Do that to K3 and you are post-training-quantizing a model that already lives at its trained 4-bit floor, with a method it never saw — and you have nothing to measure the damage against, because the higher-precision version you would compare to was never released.
When the released weights are the reference model, "quantize it to fit" isn't a tradeoff — it's spending days of engineering to risk making a frontier model quietly worse.
Not everything is 4-bit — and that's the point#
The layout is mixed on purpose. Only the mixture-of-experts expert weights are MXFP4. Activations run in MXFP8, and the non-expert path — attention and the rest — stays in higher precision. That is why the download is ~1.56TB and not a clean quarter of the ~5.6TB a full BF16 K3 would weigh: you are paying for real scales, metadata, and the higher-precision components, not four bits flat.
For serving, it means the memory you provision is for the full expert set — you hold every expert even though each token only routes to a few — plus KV cache and activations. Size the cluster for the whole weight file, not the active-parameter count.
The hardware mistake: MXFP4 on Hopper#
Here is the part that catches teams reusing last year's fleet. MXFP4 is a native datatype on NVIDIA Blackwell (B200) and AMD MI355-class accelerators. On Hopper — the H100 and H200 most rented clusters are built from — it is not. The runtime will happily serve K3 there, but it upcasts the 4-bit experts to a supported precision to do the math.
You keep the accuracy. You lose the reason the weights are 4-bit in the first place: the memory footprint and bandwidth savings evaporate the moment they're upcast. If your case for self-hosting K3 rested on fitting it into fewer GPUs, running it on Hopper quietly deletes that argument. Match the format to the silicon or don't bother with the format.
What to actually do#
vLLM shipped day-0 support with a dedicated image, vllm/vllm-openai:kimi-k3. The clean path is short:
# Serve the released MXFP4 weights as-is — no re-quantization step.
# Target Blackwell (B200) or AMD MI355-class for native MXFP4.
docker run --gpus all \
-v /models/kimi-k3:/models/kimi-k3 \
vllm/vllm-openai:kimi-k3 \
--model /models/kimi-k3 \
--tensor-parallel-size <gpus-per-node> \
--served-model-name kimi-k3Shard the ~1.56TB across your accelerators, serve an OpenAI-compatible endpoint, and point your client at it. K3 lands at 93.4% on SWE-bench Verified — third among frontier models, behind GPT-5.6 Sol and Claude Fable 5 — with a 1M-token context and native vision, so the model is worth serving correctly.
Reach for further quantization only after you have written down your own quality bar and measured against it. And before you commit a cluster at all, the self-host-versus-API math is the first decision — for most founders the API wins, and the cheapest quantization strategy is the one you never have to run. If you do self-host: serve the weights as shipped, on hardware that speaks MXFP4, and leave the quantizer in the drawer.
