Model setup recipes

GLM-5.3-Flash Serving Guide: A 320B Server-Scale MoE Recipe

GLM-5.3-Flash Setup: Mac and Multi-GPU Servers

GLM-5.3-Flash is a language model that answers questions about your documents and helps write code. There are two main local paths: distribute the official model across several GPUs, or run a quantized MLX¹ conversion on a Mac with ample memory. This guide explains how to choose a path, start a server, and receive your first response.

For a Mac used by one person, start with the oMLX path. For a GPU² server shared by several people, consider SGLang or vLLM. Once it works, increase input length and concurrency to see how well it responds to your actual workload.

Contents

Requirements and key details

Use SGLang or vLLM for the official model, and oMLX for an MLX conversion on Mac.
The model has 320B total parameters, with about 18B active per token.
Start the GPU server at 16K context and send a short API request first.

Jump to hardware-specific settings

Choose a runtime path for your hardware

GLM-5.3-Flash is an MoE³ model that selects some expert networks for each calculation. It has 320B total parameters and activates about 18B to produce a token⁴. Running it requires memory for the full weights. Serve the official zai-org/GLM-5.3-Flash model across multiple GPUs with SGLang or vLLM.

On Mac, use a community-quantized file converted to MLX. oMLX 0.7.0 supports Lightning MTP⁵ and DFlash2 for GLM-5.3-Flash. For DFlash2, also prepare the incoai/GLM-5.3-Flash-DFlash2 draft model. The two methods require different model files, so choosing the acceleration method first makes setup easier.

Two local paths for GLM-5.3-Flash
Hardware	Model file	Runtime
Multi-GPU server	Official zai-org/GLM-5.3-Flash	SGLang or vLLM
Mac with ample memory	Community quantized MLX conversion	oMLX 0.7.0

A green box with documents and a dark block box sit above a row of large and small blocks. — Choose the model and runtime for your hardware.

Start an MLX-VLM server on Mac and verify the API

The Mac path uses the mlx-community/GLM-5.3-Flash-4bit conversion with mlx-vlm. Its model card lists a 190 GiB file and says a Mac with at least 256 GB of memory is required. The architecture is documented for mlx-vlm, not mlx-lm, so match the runtime⁶ to the checkpoint⁷.

Start the server on 127.0.0.1:8080 and check that /v1/models returns a model ID. If the list shows mlx-community/GLM-5.3-Flash-4bit, send a short code question using the same ID. A sentence in choices[0].message.content explaining the first element and empty-input behavior confirms the basic text path.

If the command is not found, check that mlx-vlm is installed in the same Python environment. For a checkpoint-loading error, verify the mlx-vlm version and exact model ID. For a memory error, check unified memory⁸ used by other running apps and consider the model card’s 190 GiB file and 256 GB-or-more requirement.

Install MLX-VLM and start the local server

python -m pip install -U mlx-vlm
mlx_vlm.server --model mlx-community/GLM-5.3-Flash-4bit --host 127.0.0.1 --port 8080

Load the conversion through the MLX-VLM path on Apple Silicon and bind the server to loopback.

Check server status and model ID

curl --fail-with-body http://127.0.0.1:8080/v1/models

Check the model identifier to use from the model list.

Send a short code question to the Mac server

curl --fail-with-body http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"mlx-community/GLM-5.3-Flash-4bit","messages":[{"role":"user","content":"Explain this function empty-input behavior in one sentence: def first_item(items): return items[0]"}],"temperature":0,"max_tokens":128}'

Confirm that the response body returns a complete explanation of the function behavior.

Start the GPU server at 16K context

Run one of the commands below for your chosen runtime. GPU_COUNT=8 is an example using eight physical GPUs. Adjust it to the installed GPU count and the tensor-parallel configurations supported by the runtime. Different memory capacities across GPUs can limit how evenly the model is distributed.

Both commands set a 16K context. Memory must remain for KV cache⁹ and workspace after loading the model, so this is a useful starting length. Verify a response to a short question, then increase to the document lengths you plan to use.

The server address is 127.0.0.1:8000. It is reachable from the same computer, not from external machines. Configure authentication and access controls before sharing it with teammates. If requests contain internal material, also define what logs are stored and for how long.

Start a multi-GPU server with SGLang

GPU_COUNT=8 # Change this to the number of physical GPUs
python -m sglang.launch_server   --model-path zai-org/GLM-5.3-Flash   --tp "$GPU_COUNT"   --host 127.0.0.1   --port 8000   --context-length 16384

Set TP to the actual GPU count and start the server at 127.0.0.1:8000.

Start a multi-GPU server with vLLM

GPU_COUNT=8 # Change this to the number of physical GPUs
vllm serve zai-org/GLM-5.3-Flash   --tensor-parallel-size "$GPU_COUNT"   --host 127.0.0.1   --port 8000   --max-model-len 16384   --gpu-memory-utilization 0.90

Set tensor parallelism to the actual GPU count and open the local server.

Get the first response and check its speed

Once the model is loaded and the server is ready for requests, open a new terminal and run the command below. If the answer appears progressively, streaming is working. Wait for it to finish and inspect memory usage on each GPU and the server logs.

After a warm-up run, send the same question three times. Record median time to first token¹⁰, decode¹¹ speed, and peak memory to compare the next configuration. Keep first-time inputs separate from requests that reuse cache: reading a different amount of text changes the waiting time.

If you want a reference for response speed on Mac, consider the published M5 Ultra result.

The published Mac was an M5 Ultra 256 GB/80-core GPU. It ran a community MLX oQ4 MTP conversion in oMLX 0.7.0 with code_python input, an output budget of 128 tokens, and temperature 0. KV cache used TurboQuant 4-bit and the memory guard was aggressive. The table shows results from that configuration.

Published GLM-5.3-Flash results on M5 Ultra 256 GB
Input length	Prefill	Decode
4K tokens	2,270 tok/s	64.0 tok/s
16K tokens	2,313 tok/s	83.6 tok/s

A server shared by developers also needs a concurrency test. If four teammates ask questions at once, send four requests together and record first-response and completion times for each. Record the total tokens processed too, so you can evaluate both individual waiting time and team throughput¹².

Send a short question

curl -N http://127.0.0.1:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "zai-org/GLM-5.3-Flash",
    "messages": [
      {"role": "user", "content": "대규모 분산 추론 환경의 이점을 정리해줘."}
    ],
    "max_tokens": 512,
    "stream": true
  }'

Send a streaming request to 127.0.0.1:8000 and verify its completion.

A laptop connects along a curved path to eight servers, each marked with a green check. — Inspect per-GPU memory and communication to confirm each accelerator participates in requests.

Tune for long documents and concurrent requests

For long documents, check whether the model finds the information you need. Put a verifiable fact such as a date or exception in the middle, then ask about it. Each time you increase input beyond 16K, record answer accuracy alongside memory use.

Change context length¹³, cache settings, and MTP one at a time to identify which setting improves speed. Per-GPU memory records and server logs help diagnose errors. If short questions work but long inputs fail, first check whether memory for KV cache and workspace is insufficient.

Choose a model using your actual tasks

Make the final model choice with tasks you do frequently. Prepare a few verifiable questions, such as finding where a function is used, summarizing a configuration-change commit, or explaining differences between two files. Give the same material to a smaller model and GLM, then compare accuracy, response time, and memory use.

If a smaller model gives the needed answers and handles concurrent users, it lets you run a lighter setup. If GLM performs an important task better, use that improvement to guide your choice. Checking whether waiting times remain acceptable as users increase also helps choose hardware.

Save a working setup with the model identifier, runtime version, GPU configuration, context length, and concurrency. Test the next update with the same questions to compare performance directly. You also have a previous configuration to return to if problems arise.

Colored blocks in a cabinet lead to a conveyor and dashboard, beside a large block group and color-coded document trays. — Choose a model by comparing answer quality and waiting time on frequent tasks.

Use the GLM5-next loader for NVFP4

For RadixArk/GLM-5.3-Flash-NVFP4, choose an SGLang path that includes the GLM5-next model loader and mixed-precision MoE kernels. This file format needs the dedicated loader. Check that your SGLang branch includes the loader and kernels before loading the model.

Terminology notes

MLX — Apple’s machine-learning framework, which runs models using memory shared by the CPU and GPU on Apple Silicon.
Back to the text
GPU — A processor designed to handle many calculations in parallel. It performs model computations during AI inference.
Back to the text
MoE — A model architecture that selects some of several expert subnetworks for each input. Total parameters can differ from the number activated for one token.
Back to the text
Token — A unit into which a model divides input or output for processing.
Back to the text
MTP — A training objective or model component for predicting multiple future tokens at once. In speculative decoding, those predictions can be proposed as candidate tokens.
Back to the text
Runtime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.
Back to the text
Checkpoint — A file containing saved model weights and related state. Versions or tasks in one model family may use different checkpoints.
Back to the text
Unified memory — An architecture where the CPU and GPU share one physical memory pool.
Back to the text
KV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.
Back to the text
Time to First Token — The time from sending a request until its first output token arrives.
Back to the text
Token decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the text
Throughput — The amount of work processed or generated over time. Comparisons need the unit, such as tokens per second or requests per second.
Back to the text
Context window — The token span of input and generated content a model can handle in one request. The supported limit and memory use depend on the model and runtime settings.
Back to the text

Frequently asked questions

What should you check first in this article?: GLM-5.3-Flash is a language model that answers questions about your documents and helps write code. There are two main local paths: distribute the official model across several GPUs, or run a quantized MLX conversion on a Mac with ample memory. This guide explains how to choose a path, start a server, and receive your first response.
How much memory does it need?: The Mac path uses the `mlx-community/GLM-5.3-Flash-4bit` conversion with `mlx-vlm`. Its model card lists a 190 GiB file and says a Mac with at least 256 GB of memory is required. The architecture is documented for `mlx-vlm`, not `mlx-lm`, so match the runtime to the checkpoint.
Where can you check the conditions and sources?: If the command is not found, check that `mlx-vlm` is installed in the same Python environment. For a checkpoint-loading error, verify the `mlx-vlm` version and exact model ID. For a memory error, check unified memory used by other running apps and consider the model card’s 190 GiB file and 256 GB-or-more requirement.

Serving recipe

Settings for your hardware

For one user

Choose your hardware to see the launch command and settings you can adjust.

Choose your hardware

Available memory: 236GB / Bandwidth: 1200GB/s

Profile

Apple Mac Studio M5 Ultra (256GB)

Memory fits

Recommended runtime: oMLX (Q4_K_M · MLX checkpoint) · Documented runtime settings

For your first run or reliable everyday use

Profile context length

16,384 tokens

Based on runtime documentationStudio M5 Ultra 256GB · Q4 · 1 request

A starting setup based on the runtime documentation. Follow the steps below and compare speed as you make changes.

Estimated decode speed

66.9 ~ 100.3 tok/s

Estimate for one user

Estimated TTFT

5.9 to 8.9 seconds

Prefill: 1850.4 ~ 2775.6 tok/s

Estimated memory use

About 192.94GB / 236GB

Remaining margin: approx. 43.1GB

Server address

http://127.0.0.1:8000

Local only · 127.0.0.1

Estimates use the calculator’s recommended quantization and acceleration with 16,384tokens. They are not measurements of the configuration above.

Settings in this profile

Value	Why this setting
WeightsQ4_K_M · MLX checkpoint	Use an MLX model directory. This is a different file format from Q4_K_M GGUF.
Check input length16,384	Check the input limit in model settings before sending a request. This launch command does not set the context length.
KV cacheCheck model settings	This launch command does not set the KV cache precision.
Prefill batch sizeCheck runtime settings	Do not use llama.cpp's batch-size and ubatch-size values as substitutes for MLX settings.
Concurrent requests1	Measure the speed of one request, separately from the combined throughput of multiple requests.
Memory headroom2 GB or more	After loading the model, check actual memory use and whether the system is swapping.

Studio M5 Ultra 256GB Run command (oMLX)

Apple Silicon · MLX Model · Memory Guard

MODEL_DIR="$HOME/.omlx/models"
omlx serve \
  --model-dir "$MODEL_DIR" \
  --host 127.0.0.1 \
  --port 8000 \
  --max-concurrent-requests 1 \
  --memory-guard balanced

Record a baseline with this setup, then follow the tuning steps below, changing one setting at a time.

Checks after startup

01Check the startup log for completed model loading and the actual context length.
02After warming up, send one request at a time. Use three different inputs of the same length to record fresh-input processing time.
03Record repeated-input results separately as cache-reuse runs. Do not combine them with fresh-input results.
04Record prefill tok/s, decode tok/s and peak memory together, then change one setting at a time.

Trade-offs

Conservative context and concurrency settings may produce less than the hardware's maximum throughput.

See speed with recommended settings Studio M5 Ultra 256GB View price, power, and sales configuration

Adjust one item at a time

Speed tuning, step by step

If several values are changed at once, it is difficult to find the cause.

STEP 1
Save a baseline
Record first-input and cache-reuse runs separately, three times each. Compare prefill, decode and peak memory together.
When to stop: Do not move on if there are errors or less than 2GB of memory headroom.
STEP 2
Tune prefill chunk size
Increase in steps of 1,024 → 2,048 → 4,096, comparing TTFT and peak memory on long inputs.
When to stop: Revert to the previous value if TTFT does not improve or peak memory rises sharply.
STEP 3
KV Cache Tuning
Only when you need more context, compare Q8 cache against the F16/BF16 baseline using the same question.
When to stop: Keep the default precision if the output changes or you already have enough context.
STEP 4
MTP/speculative decoding
Enable it only for supported models. Test code and prose separately, measuring acceptance rates and actual decode tok/s.
When to stop: Turn it off if the median of three runs does not improve on the baseline.
STEP 5
Context expansion
Double context length at each step until you reach what you need. Check retrieval from the middle of the input and whether swapping occurs.
When to stop: Reduce by one step if retrieval accuracy drops or swapping or memory compression starts.

Submit a measurement from my hardware

Import a JSON file with at least three runs under the same conditions. The file stays in this browser until you submit it.

Include only hardware, model, runtime settings and measurements. Do not include prompts, responses, raw logs or file paths.

If you do not have a measurement file, use the tool with your running local server. It requires Node.js 20 or later and does not upload results automatically.

Read the measurement instructions Download the measurement tool Download the hardware and settings template

Import measurement JSON (up to 32KB)

These are reviewed community submissions, not measurements made by this site.

Report a setup issue

Tell us where this setup failed. Only the site administrator can read your report.

Setup being reported · Studio M5 Ultra 256GB · GLM-5.3-Flash (MoE) · Balanced

Change log

These entries record changes to the site's guidance. They do not automatically check your installed engine or model version.

2026-09-06
Added saving and reopening hardware profiles
Runnable profiles can now be saved with their hardware selection in this browser, up to five entries, and reopened from the guide directory. A notice prompts you to recheck before running if the published settings have changed or the profile is no longer available.
2026-09-06
Added links to runtime explanations
Engine names in the settings now link to the corresponding runtime guide. Paths without an identified engine are not linked to a guessed program.