Model setup recipes
GLM-5.3-Flash: serving a server-scale 320B MoE
If the 18B active parameter count led you to look for a single-GPU recipe for GLM-5.3-Flash, first check where all 320B weights will fit. The official server setup here uses multiple accelerators. Keep it separate from compressed-model experiments on high-memory Macs, and check loading, distribution and responses in that order.
Serving recipe
Settings for your hardware
For one user
Choose your hardware to see the launch command and settings you can adjust.
Apple Mac Studio M5 Ultra (256GB)
Experimental setupRecommended runtime: Compression runtime verification required (Q4 Compression weight (artifact/runtime verification required)) · Experimental setup (needs validation)
Profile context length
4,096 tokens
Estimated decode speed
81.3 ~ 90.3 tok/s
Estimate for one user
Estimated TTFT
15.0 ~ 26.1 s
Prefill: 157 ~ 273 tok/s
Estimated memory use
About 190.21GB / 236GB
Remaining margin: approx. 45.8GB
Server address
http://127.0.0.1:1234
Local only · 127.0.0.1
Estimates use the calculator’s recommended quantization and acceleration with 4,096tokens. They are not measurements of the configuration above.
Settings in this profile
| Value | Why this setting |
|---|---|
| Quantization and model filesQ4 quantized weights (verify the artifact and runtime) · $MODEL_PATH | A starting point that reduces weight transfers and memory use for single-user decoding. |
| Starting context length4,096 tokens | a value that reserves space for weights, KV cache, and runtime buffer instead of immediately allocating native 1024K |
| Bind address127.0.0.1:1234 | Bind only to loopback to prevent local servers without authentication from being exposed to external networks. |
| GPU offload and splittingSingle integrated memory full load verification | Keep weights in fast memory to avoid CPU/RAM offload bottlenecks. |
| Flash AttentionActivate after checking model compatibility in optional runtime | Reduces attention memory use during prefill and with long contexts. |
| KV cacheSeparate verification of q8_0 after loading with default precision | Reduce memory use, using only precision formats supported by the runtime. |
| Prefill batch and micro-batchCheck runtime settings | Do not use llama.cpp's batch-size and ubatch-size values as substitutes for MLX settings. |
| Number of concurrent requests1 | Prioritize time to first token and decode latency for 1 user over total server throughput. |
Fitting in memory and having runtime support are different. Copyable launch commands are not shown until the compressed model files and runtime compatibility are confirmed.
Things to watch for
- The official GLM serving setup and local Q4 conversions are different. Launch commands are not provided until the actual model files and runtime compatibility are confirmed.
Adjust one item at a time
Speed tuning, step by step
If several values are changed at once, it is difficult to find the cause.
- STEP 1
Save a baseline
Record first-input and cache-reuse runs separately, three times each. Compare prefill, decode and peak memory together.
When to stop: Do not move on if there are errors or less than 2GB of memory headroom.
- STEP 2
Tune prefill chunk size
Increase in steps of 1,024 → 2,048 → 4,096, comparing TTFT and peak memory on long inputs.
When to stop: Revert to the previous value if TTFT does not improve or peak memory rises sharply.
- STEP 3
KV Cache Tuning
Only when you need more context, compare Q8 cache against the F16/BF16 baseline using the same question.
When to stop: Keep the default precision if the output changes or you already have enough context.
- STEP 4
MTP/speculative decoding
Enable it only for supported models. Test code and prose separately, measuring acceptance rates and actual decode tok/s.
When to stop: Turn it off if the median of three runs does not improve on the baseline.
- STEP 5
Context expansion
Double context length at each step until you reach what you need. Check retrieval from the middle of the input and whether swapping occurs.
When to stop: Reduce by one step if retrieval accuracy drops or swapping or memory compression starts.
Submit a measurement from my hardware
Import a JSON file with at least three runs under the same conditions. The file stays in this browser until you submit it.
Include only hardware, model, runtime settings and measurements. Do not include prompts, responses, raw logs or file paths.
If you do not have a measurement file, use the tool with your running local server. It requires Node.js 20 or later and does not upload results automatically.
These are reviewed community submissions, not measurements made by this site.
Keep official weights and compressed versions separate
Running the official zai-org/GLM-5.3-Flash checkpoint with SGLang or vLLM requires a server that can hold the full model. The active parameter count describes computation per token; it does not shrink the files to the size of an 18B model.
Loading a community-quantized file into a large unified-memory system is a separate setup. Even if the file fits, confirm that the app supports the architecture and produces answers. Do not read the official server commands as instructions for running it on a Mac.
If the hardware selector marks a combination as unsupported, do not force the command to run just because the memory estimate looks sufficient. First find a reproducible working record for that setup.
Match accelerator count and starting context to the hardware
Set GPU_COUNT below to match your hardware and the parallelism supported by the runtime. The right card count is not enough if individual cards lack memory. Check the actual interconnects and communication paths as well.
Rather than starting at the model's full native context, test loading at a reduced length such as the 16K example. Even that is not universally safe: calculate space for the weights and working memory first. Do not simply push the memory limit to its maximum.
Bind the server to 127.0.0.1 first. For use across an organization, configure authentication, access restrictions and a log-handling policy separately. Running a model locally does not, by itself, restrict access by other people.
GPU_COUNT=8 # 실제 물리 GPU 수로 수정
python -m sglang.launch_server --model-path zai-org/GLM-5.3-Flash --tp "$GPU_COUNT" --host 127.0.0.1 --port 8000 --context-length 16384GPU_COUNT=8 # 실제 물리 GPU 수로 수정
vllm serve zai-org/GLM-5.3-Flash --tensor-parallel-size "$GPU_COUNT" --host 127.0.0.1 --port 8000 --max-model-len 16384 --gpu-memory-utilization 0.90Check each accelerator, not just the server average
When the server is ready, send a short question and check that streaming works and the answer completes. Inspect each GPU's memory use and logs for local shortages or communication delays. An overall average can hide a problem on one node.
After warming up, take three measurements and record the median time to first token, decode speed and peak memory. Do not combine fresh-input and cache-reuse results. Support for long context does not mean speed stays constant at every length. Measure the lengths you need separately.
If the server will support multiple users, verify one request first, then increase concurrency. Check whether individual requests wait longer even as total throughput improves. Peak speed for one user and overall server performance are different measurements.
curl -N http://127.0.0.1:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "zai-org/GLM-5.3-Flash",
"messages": [
{"role": "user", "content": "대규모 분산 추론 환경의 이점을 정리해줘."}
],
"max_tokens": 512,
"stream": true
}'
Separate model-loading failures from long-context failures
If the weights do not fit at startup, reducing context a little may not solve it. First check file precision, how the weights are split, and memory on each card. If loading succeeds but long inputs fail, investigate the cache and temporary prefill memory.
When increasing beyond 16K, place a fact with a known answer in the middle of the input and check that the model retrieves it. Completing a request does not mean the answer used the information you needed. Record answer accuracy alongside memory use.
If a change causes a failure, restore the last working configuration and compare one setting at a time. Finding one cause is a better route to a reproducible improvement than changing several accelerators' environments together.
Be clear about why you need a model this large
Buying the hardware does not end the work of running a large model: versions and the distributed setup still need maintenance. The cost makes sense only if this model solves specific tasks that smaller models could not handle well.
For a personal document or coding tool, a smaller model may be enough. If this model clearly improves a particular team task, use that task as your test case. Once the server starts successfully, the next step is to finish a real piece of work.
Running an NVFP4 checkpoint
Requires the GLM5-next loader and mixed-precision MoE kernels. This is not the standard GLM-5.3 checkpoint.
SGLang · RadixArk/GLM-5.3-Flash-NVFP4
Change log
These entries record changes to the site's guidance. They do not automatically check your installed engine or model version.
Added saving and reopening hardware profiles
Runnable profiles can now be saved with their hardware selection in this browser, up to five entries, and reopened from the guide directory. A notice prompts you to recheck before running if the published settings have changed or the profile is no longer available.
Added links to runtime explanations
Engine names in the settings now link to the corresponding runtime guide. Paths without an identified engine are not linked to a guessed program.