How to Run GLM-5.3-Flash Locally

GLM-5.3-Flash is a 320-billion-parameter model released under the MIT license. Those two facts pull in opposite directions: the license says run it however you like, and the parameter count says you will need serious hardware to do it.

The interesting part is that 18 billion of those 320 billion parameters are active per token, and quantized builds exist. That combination puts this model within reach of setups far smaller than the 8x H200 node most guides assume.

This post walks the hardware tiers honestly, from a full-precision production node down to a quantized build on a workstation, and covers when running it yourself makes sense at all.

What you are actually loading

The mixture-of-experts design is what makes this feasible. All 320B parameters have to be resident in memory, but only 18B participate in any given token, so compute demand is far below what the total suggests. Memory is your binding constraint, not FLOPs.

Z.ai also reports a KV cache roughly 4.4 times smaller than GLM-5.3, which matters enormously for long-context work. KV cache is what consumes memory as your context fills, and on a 1M-token window that is normally the thing that kills you.

Tier 1: full precision on a production node

For serving at full quality with real concurrency, the reference configuration is an 8x H200 node (141GB each, about 1,128GB total). An 8x H20 node works as well.

Rough numbers: the weights alone want somewhere in the region of 700 to 800GB depending on precision, and you need headroom on top for KV cache and runtime overhead. Rented cloud capacity for a node like this runs somewhere around $24 to $48 per day.

vLLM

vLLM is the common default and has the broadest ecosystem support. Tensor parallel size should be a power of two:

vllm serve zai-org/GLM-5.3-Flash \
  --tensor-parallel-size 8 \
  --max-model-len 1048576 \
  --trust-remote-code

Start with a smaller --max-model-len while you are validating the setup. Asking for the full million-token window immediately means allocating KV cache for it, and a failure there looks like an out-of-memory error rather than a configuration problem.

SGLang

SGLang had day-zero support for this model, with published recipes for H100, H200, B200, B300, GB200, and GB300, including multimodal serving. Z.ai used an SGLang-based stack for its own pre-launch serving.

python -m sglang.launch_server \
  --model-path zai-org/GLM-5.3-Flash \
  --tp 8 \
  --context-length 1048576

SGLang tends to win on structured output and high-concurrency agentic workloads. If you are serving a coding agent rather than a chat interface, it is worth benchmarking against vLLM rather than defaulting.

Both stacks need a tool-call parser configured if you want function calling to work properly. Check current flags in each project’s documentation, since parser names change between releases.

Tier 2: quantized on smaller hardware

This is the tier that most coverage skips, and it is the one that matters for anyone without a data center.

Quantized GGUF builds are published at unsloth/GLM-5.3-Flash-GGUF, going down to aggressive 1-bit and 2-bit formats such as IQ1_S and IQ2_XXS. A 2-bit quantization of a 320B model brings the weights into a range that a high-memory workstation or a multi-GPU consumer rig can hold, particularly with CPU offload.

Two honest caveats:

Aggressive quantization costs quality. IQ1_S is a long way from full precision. On a 320B MoE the degradation is often gentler than the same treatment applied to a dense model, because there is more redundancy to lose, but “runs” and “runs well” are different claims. Test it on your own tasks before concluding anything.

Unsloth’s documentation for this model is marked work in progress. Quant availability and recommended settings are still moving. Check what is actually published before planning a build around a specific format.

For CPU-heavy and hybrid setups, KTransformers is designed for exactly this case, keeping MoE experts in system RAM and moving only what is needed onto the GPU. On a MoE model with 18B active parameters, that architecture fits unusually well. TokenSpeed is also listed among supported runtimes.

Our guide to running GLM-4.7-Flash locally covers the smaller-model version of this workflow, and running GLM-5 locally for free covers the general local-GLM setup.

Working out your memory budget

Two numbers determine whether a configuration fits.

Weights. At roughly 2 bytes per parameter in BF16, 320B parameters is about 640GB before overhead. FP8 roughly halves that. A 4-bit quantization brings it near 160GB, and the aggressive 2-bit formats go lower still at a real cost in quality.

KV cache. This scales with context length and concurrency, and it is what surprises people. A configuration that loads fine at 8K context can fail at 128K because the cache grew, not the weights. Z.ai’s reported 4.4x reduction versus GLM-5.3 helps a lot here, but the scaling is still linear in tokens.

The practical implication is to size for your real context length, not the maximum the model advertises. Very few applications need the full million tokens, and provisioning for a window you never use is the most common way to make this model look unaffordable.

If you were following the earlier open-weights story for this family, our GLM-5.3 self-hosting post was written ahead of the drop. The weights have now landed for Flash under MIT, so the guidance here supersedes it.

Fine-tuning

The MIT license permits fine-tuning and redistribution, which is unusual at this capability level and is the strongest reason to hold the weights yourself.

Be realistic about the cost. Full fine-tuning of a 320B model is out of reach for most teams. Parameter-efficient methods such as LoRA are the practical path, and on a mixture-of-experts model there is an additional design question of whether you adapt the router, the experts, or the attention layers. That is an active area with less settled guidance than for dense models.

If your goal is domain adaptation rather than new capability, test prompting and retrieval against the base model first. On a model with a 1M-token context window, putting your domain knowledge in the prompt is often cheaper and better than training it in.

Sampling settings

Z.ai publishes different recommendations by task:

The model also supports three reasoning modes through reasoning_effort, with values low, high, and max. Max is the default. On local hardware this matters more than it does on the API, because reasoning tokens are generation you are paying for in wall-clock time rather than dollars. If your rig generates slowly, low is the difference between usable and not.

Does self-hosting make financial sense?

Usually not, and the API price is why.

At list pricing, GLM-5.3-Flash costs $0.15 per million input tokens. A rented 8x H200 node at roughly $1,000 per month buys you about 6.7 billion input tokens of API usage. Sustaining volume above that, continuously, is a large operation.

The node also costs the same whether it is saturated or idle, while the API bills only what you use. Unless your utilization is genuinely high around the clock, the fixed cost loses.

So the reasons to self-host are not cost:

• Data residency and privacy. Nothing leaves your infrastructure.
• No rate limits. Your capacity is your capacity.
• Availability guarantees. No dependency on a vendor’s uptime or pricing decisions.
• The MIT license. You can modify, fine-tune, and redistribute. That is unusual for a model at this capability level and it is the strongest argument in the list.
• Hardware you already own. If the GPUs are bought and idle, the marginal cost is electricity, and the whole calculation inverts.

Our pricing breakdown works through the API side of this comparison in more detail.

Verifying your deployment

Both vLLM and SGLang expose OpenAI-compatible endpoints, so the same request shape works against your local server and against Z.ai:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zai-org/GLM-5.3-Flash",
    "messages": [{"role": "user", "content": "reply with OK"}]
  }'

Worth testing beyond a smoke check: long-context behavior at the length you actually need, image input if you are serving multimodal, tool calling with your real schemas, and throughput under concurrency rather than single-request latency.

This is where a saved test collection earns its keep. Point Apidog at both your local server and the Z.ai endpoint with the base URL as an environment variable, run the same suite against each, and compare. You will find out quickly whether your quantized build still handles the tool schemas your application depends on, which is the failure mode people discover in production instead.

FAQ

What is the minimum hardware? For full precision, an 8x H200-class node. For quantized GGUF builds, considerably less, though quality drops with quantization level.

Do I need all 320B parameters in memory? Yes. Only 18B are active per token, but the full set must be resident. Memory is the constraint; compute is not.

Which is better, vLLM or SGLang? SGLang had day-zero support with published multimodal recipes and often wins on concurrency and structured output. vLLM has broader ecosystem support. Benchmark both on your workload.

Can I run it on a single GPU? Not at full precision. With aggressive quantization and CPU offload through KTransformers, a high-memory single-GPU system plus a lot of system RAM is plausible. Expect slow generation.

Is the license really MIT? Yes. The weights are published as MIT, which permits commercial use, modification, and redistribution.