GLM-5.3-Flash: serving a server-scale 320B MoE

Hardware-specific speed tuning and serving recipe summary

Provides weights, KV cache, prefill batch, and concurrent request with acceleration functionality, separated for balanced performance, speed, and long-context profiling.

On the detail page, you can check the command per profile, applied value, stopping conditions, and measurement order. The execution command is displayed only for combinations of model file and execution program that have been verified.

Distinguish the original weights from quantized versions

The path to executing the official checkpoint of zai-org/GLM-5.3-Flash with SGLang or vLLM presupposes a server environment that will contain a large model. The active size is a per-token calculation number and does not mean the entire file will be as small as the 18B model.

Uploading the community quantization file to the large integrated memory is a separate route. Even if the size is right, you need to make sure your app supports and actually answers your model structure. Do not read the official server commands by translating them directly into Mac execution methods.

If the device selector shows an unsupported combination, it only reports estimated memory and does not force the command to run. You must first obtain a reproduction record of that route.

Choose the accelerator count and initial context together

The GPU_COUNT below should match your actual configuration and parallel support of your runtime. If the number is correct and the capacity for each card is insufficient, it will not be loaded. It also checks the actual connections and communication paths between the cards.

Rather than turning on the large native context right away, check the loading under lowered conditions, such as in the 16K example. That length is also not guaranteed to be universally safe, so the sum of the weights and workspace must be calculated first. Avoid raising the memory limit all the way.

Put the server first on 127.0.0.1 only. If you plan to share within your organization, you must prepare separate authentication/access restrictions and log processing policies. Just because it's running locally doesn't automatically limit other people's access.

GPU_COUNT=8 # 실제 물리 GPU 수로 수정
python -m sglang.launch_server   --model-path zai-org/GLM-5.3-Flash   --tp "$GPU_COUNT"   --host 127.0.0.1   --port 8000   --context-length 16384

GPU_COUNT=8 # 실제 물리 GPU 수로 수정
vllm serve zai-org/GLM-5.3-Flash   --tensor-parallel-size "$GPU_COUNT"   --host 127.0.0.1   --port 8000   --max-model-len 16384   --gpu-memory-utilization 0.90

See if all the accelerators are doing their job

Once the server is ready, ask a short question to confirm answers and streaming. Look at the memory and execution log of each GPU together to see if one side is insufficient or has communication delays. If you only look at the overall average, you may miss problems in specific nodes.

After the preparation run, the median of three identical inputs is left and the first token, decode, and peak memory are separated. Even though it is a structure that processes long contexts, the speed is not constant at any length. Results must be measured separately for each required length.

If receiving multiple users, confirm a single request and then increase concurrent requests. We separately see whether the waiting time for individual requests becomes longer even though total throughput improves. The maximum speed of one person and the overall performance of the server are different records.

curl -N http://127.0.0.1:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "zai-org/GLM-5.3-Flash",
    "messages": [
      {"role": "user", "content": "대규모 분산 추론 환경의 이점을 정리해줘."}
    ],
    "max_tokens": 512,
    "stream": true
  }'

Separate load failures from long context failures

If weight is not included from the beginning, simply lowering the context a little may not solve the problem. First check the file precision, partition path, and memory for each card. If it only fails on long inputs after the primary load, look at the cache and prefill temporary space.

When increasing from 16K to a longer input, we insert information that knows the correct answer in the middle to check whether it actually finds it. The fact that the request has been completed is different from the fact that the required content has been utilized. Along with your memory, record the accuracy of your results.

If changing the settings fails, restore the previous normal configuration and compare items one by one. Finding a single cause is closer to reproducible improvement than changing the environment of multiple accelerators simultaneously.

I leave you with the reasons why you need this size

The operation of large models requires managing versions and distributed states even after purchasing the equipment. You can only account for its costs by making sure that this model solves the specific task that the smaller model lacked.

If you're just starting out as a one-person documentation and coding tool, a smaller model may suffice. Conversely, if there is a clear difference in results for a specific task in your team, use that task as the basis for verification. After successfully turning on the server, the next step is to complete one actual task.