How to Run Kimi K3 Locally (and When You Shouldn't)

Moonshot AI released the open weights for Kimi K3 on July 27, and the download counter on Hugging Face is already close to 100,000. The pitch is obvious: a 2.8 trillion parameter model that beat Claude Opus 4.8 on every benchmark Moonshot published, and now you can host it yourself.

The catch is also obvious once you look at the numbers. Full precision inference needs 1.57 TB of disk. Even the released MXFP4 weights are a 594 GB download. This is a model you can own, but “local” means something different at this scale than it did for an 8B Llama.

This guide covers what it takes to run K3 on your own hardware, what the community has managed on consumer machines, and how to wire a self-hosted K3 endpoint into your API workflow with Apidog once it’s serving.

What you’re downloading

First, the shape of the thing. If you want the full background, start with What Is Kimi K3?; the short version:

• 2.8T total parameters, 104B activated per token. K3 is a Mixture-of-Experts model with 896 experts. Each token routes through 16 selected experts plus 2 shared ones, so the compute per token is a fraction of the headline number.
• 93 layers: 69 Kimi Delta Attention (KDA) layers and 24 Gated MLA layers. The KDA design is why the 1-million-token context window is usable at all.
• Native vision via a 401M-parameter MoonViT-V2 encoder. The released weights handle text, images, and video input.
• MXFP4 weights, MXFP8 activations. Moonshot did quantization-aware training, so the 4-bit release is the intended serving format, not an afterthought. This also means the weights compress poorly beyond that; the low-bit headroom is already spent.
• Thinking-only. K3 always reasons before answering, with low, high, and max effort levels. There is no instant mode.

The weights are gated behind the Kimi K3 License on the Hugging Face repo. Accept the license, then pull with huggingface-cli. On a 1 Gbps connection, budget about 80 to 90 minutes for the 594 GB.

Option 1: datacenter-class serving with vLLM or SGLang

Moonshot recommends three engines: vLLM, SGLang, and TokenSpeed. Their KDA prefill-cache contribution shipped in vLLM alongside the weights, so vLLM is the path of least resistance:

vllm serve moonshotai/Kimi-K3 \
  --tensor-parallel-size 8 \
  --max-model-len 131072

Notes from the field:

• Hardware. Moonshot evaluated on H20 clusters. Realistically you need an 8-GPU node with tensor parallelism as the floor. On B200-class hardware, throughput above 100 tokens/s is achievable.
• Context. The model supports up to 1,048,576 tokens, but the KV cache at full context is around 27 GB on its own. Start at 131K and raise it only if your workload needs it.
• Sampling. Moonshot’s defaults are temperature 1.0 and top-p 0.95. For agentic workloads, keep temperature at 1.0 and move top-p to 1.0.

This is “local” in the data-sovereignty sense: your infrastructure, your logs, your compliance story. It is not local in the laptop sense, and no amount of quantization changes that for interactive use.

Option 2: GGUF quants on a big workstation

Unsloth published GGUF conversions for llama.cpp users, and their dynamic quants are the only realistic way to shrink K3 below the official release:

The working rule: your RAM plus VRAM should roughly equal the quant size. Fall short and llama.cpp will still run through offloading, but every missing gigabyte costs you speed. A Mac Studio chained to a 128 GB machine, or a DGX Station, sits at the practical low end.

A minimal llama.cpp invocation, including the vision projector:

./llama.cpp/llama-cli \
    --model unsloth/Kimi-K3-GGUF/UD-IQ1_S/Kimi-K3-UD-IQ1_M-00001-of-00015.gguf \
    --mmproj unsloth/Kimi-K3-GGUF/mmproj-F16.gguf \
    --temp 1.0 \
    --top-p 0.95

If your hardware tops out below this, don’t force it. The best local LLMs of 2026 list has open models that fit in 24 to 128 GB and answer in real time; K3 at 1-bit on inadequate RAM will not.

The M1 Max experiment: yes, but 16 seconds per token

A Hacker News thread this week documented K3 running on a 64 GB M1 Max by streaming weights from a 2 TB SSD instead of holding them in memory. The numbers explain both why it works and why you wouldn’t use it:

• K3 carries roughly 115 GB of dense parameters that every token touches, plus about 25 GB of routed expert weights per token. The dense part alone exceeds the machine’s RAM, so the SSD becomes slow-motion memory.
• Result: around 16 seconds per token. Some configurations reported over a minute per token. That’s a paragraph per hour.
• Disk throughput is the whole game. M1-era SSDs read far slower than current Apple silicon, and streaming experts over a network is slower still.

As a proof that MoE sparsity plus mmap can run a 2.8T model on a laptop, it’s a genuinely fun result. As a way to use K3, it isn’t one. If you want K3 answers on a MacBook, the free tiers or the hosted API will serve you better.

Wiring your local K3 into an API workflow

Whether you serve through vLLM or llama.cpp’s server mode, you end up with the same thing: an OpenAI-compatible HTTP endpoint on localhost. From here it’s an API like any other, and the same workflow we use for testing local LLMs as APIs applies:

1. Point Apidog at the endpoint. Create an environment with base_url set to http://localhost:8000/v1 (vLLM’s default) and swap it against Moonshot’s hosted endpoint later. Same requests, two backends, one variable.
2. Inspect the thinking stream. K3 is thinking-only, so responses carry reasoning content before the answer. Apidog’s SSE debugging view renders the stream as it arrives, which makes it much easier to see what the reasoning-effort levels change.
3. Assert on structure, not vibes. Add automated tests that validate response schema, latency budgets, and token usage fields, so a quant swap or engine upgrade that degrades output shows up in a failing test instead of a user report.
4. Mock K3 while the GPUs are busy. A 594 GB model takes a while to load. Record real responses once, then let a mock server return them so frontend work never waits on the inference box. Download Apidog to set this up free; the mock and test tooling both work against any OpenAI-compatible server.

The request format itself matches what we covered in the Kimi K3 API guide, so tests written against the hosted API transfer directly to your local deployment.

So should you run it locally?

A quick decision table:

The honest summary: K3’s open weights matter because you can audit, fine-tune, and self-host a frontier-class model, not because most people should. For teams with the hardware, the vLLM path works today and performs well. For everyone else, the open release still pays off indirectly, through cheaper hosted access and third-party providers competing to serve it.

Whichever side of that table you land on, the endpoint is where the model meets your code. Test it like one: schema checks, streaming inspection, and mocks that keep development moving while the model thinks.