Executable programs and extensions

Run and compare GLM-5.3 Flash recipes on two, three, and four Sparks

Single-request decode measured 60.4, 77.6, and 95.2 tok/s on two, three, and four Sparks; four-Spark tp4 remains experimental.

MIA's GLM-5.3-Flash Spark recipe adds 96 patches to TensorFold 0.6.0. With its default EXL3 4-bit setup, single-request decode¹ measured 60.4 tok/s on two Sparks, 77.6 on three, and 95.2 on four. Aggregate throughput² was 108.8 tok/s at four requests on two Sparks, 146.2 at four on three, and 222.4 at eight on four. The four-Spark run uses the experimental tp4 branch.

Contents

Requirements and key details

Single request: 60.4 tok/s on two, 77.6 on three, and 95.2 on four Sparks.
Aggregate throughput: 108.8 tok/s on two (4 requests), 146.2 on three (4), and 222.4 on four (8). Keep these separate from per-request rates.
Four-Spark tp4 is experimental and does not include all later fixes.

Start with the scope of the two-Spark recipe

GLM-5.3-Flash is a different model from full GLM-5.3 744B. This guide covers the Flash model's two-rank TensorFold recipe. MIA's default run places patches on TensorFold v0.6.0, with one Spark per rank.

Connect the ConnectX-7 ports directly and configure RoCE v2. The three-Spark experiment needs a triangle layout and rank order; four Sparks use a switch or ring and the separate tp4 branch procedure.

A dark monitor, a compact desktop computer, and a PC case with several visible fans — Compare single-request and concurrent results when adding Spark machines.

Compare one request and concurrent throughput separately

The one-request column is the generation speed of one answer. The aggregate column includes its request count: four concurrent requests for two or three Sparks, and eight for four Sparks. Keep these workloads distinct when comparing shared-server capacity.

Scroll sideways to compare all columns.

Prose decode speed and concurrent throughput
Sparks	One request	Aggregate (request count)	Status
2	60.4 tok/s	108.8 tok/s (4 requests)	Base recipe
3	77.6 tok/s	146.2 tok/s (4 requests)	Patched experiment
4	95.2 tok/s	222.4 tok/s (8 requests)	tp4 branch · experimental

The two-Spark figures were measured with sparkDash at the defaults: 1M context, FP8³ KV cache⁴, 4-bit dense weights, DFlash2 and copy drafts, vision on, and GPU⁵ clocks capped at 2,200 MHz. The three-Spark run also used its default four streams, 1M window, FP8 KV, and DFlash2 plus copy drafts. Compare model version and request count alongside the table.

Two compact desktop computers connected to a monitor showing streams of particles — Check what each server rank does across the link.

Prepare the machines and repository, then start the server

Each Spark uses a GB10 with 128 GB unified memory⁶ and needs about 110 GiB free when the server starts. For the two-Spark recipe, install Docker and the NVIDIA Container Toolkit on both machines and verify key-based SSH from head to worker. Reserve disk for the checkpoint⁷, DFlash2, and container image.

Clone the repository on the head and copy scripts/local.sh.example to scripts/local.sh. Save the following settings in the matching entries of scripts/local.sh with a file editor. WORKER is the worker's management-LAN SSH address; FABRIC_PEER is its CX7 address when needed. Check the saved file, then verify key-based SSH using the actual address before starting the server.

Prepare the two-Spark repository and settings file

git clone https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold.git
cd GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold
cp scripts/local.sh.example scripts/local.sh

Save these settings in scripts/local.sh with a file editor

WORKER=user@192.0.2.2
FABRIC_PEER=192.0.2.12

Check the saved settings and worker SSH from the head

grep -nE 'WORKER=|FABRIC_PEER=' scripts/local.sh
# Replace the reserved address with the actual worker SSH target.
ssh-copy-id user@192.0.2.2
ssh -o BatchMode=yes user@192.0.2.2 true

On first run, the two-Spark starter prepares the container, checkpoint, and DFlash2, then smoke-tests both ranks. Success prints GLM-5.3-Flash-EXL3 is now LIVE! on port 8888. Three Sparks use start-tp3.sh; four use the tp4 branch's switchless-ring procedure. The four-Spark path is experimental, so verify detected cabling and rank order in a dry run first.

Start the two-Spark server

./start.sh

Check the model list and first request after LIVE

curl -fsS http://127.0.0.1:8888/v1/models
curl -fsS http://127.0.0.1:8888/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"GLM-5.3-Flash-EXL3","messages":[{"role":"user","content":"Say READY in one word."}],"max_tokens":8,"temperature":0}'

Start the three-Spark TP3 experiment

./start-tp3.sh

4 Spark ring dry run

git clone --branch tp4 https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold.git ../glm-flash-tp4
cd ../glm-flash-tp4
DRY_RUN=1 ./start-tp4-switchless.sh

Start four-Spark tp4 after checking the wiring

./start-tp4-switchless.sh

Check the four-Spark server response

curl -fsS http://127.0.0.1:8888/v1/models
curl -fsS http://127.0.0.1:8888/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"GLM-5.3-Flash-EXL3","messages":[{"role":"user","content":"Say READY in one word."}],"max_tokens":8,"temperature":0}'

Requirements, startup command, and API address

Treat the four-Spark run as a separate validation scope

The progression from two to four Sparks shows decode gains as machines are added. The three-Spark setup adds recipe patches, while four Sparks use an experimental branch that has not received the same breadth of exactness checks as the three-Spark setup. Treat the four-Spark figure as a reproduction target for a specific topology and branch, not a guaranteed result on any setup.

If concurrent throughput is your goal, record both aggregate and per-request tok/s. With only one or two users, single-request speed and TTFT⁸ matter more. More concurrent requests can still mean longer waits per user, so measure using your own prompts⁹ and reply lengths.

Printed reports and an open notebook around a laptop displaying documents and charts — Validate the final choice with your real request count and reply length.

Terminology notes

Token decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the text
Throughput — The amount of work processed or generated over time. Comparisons need the unit, such as tokens per second or requests per second.
Back to the text
FP8 — A floating-point format that represents each value with 8 bits.
Back to the text
KV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.
Back to the text
GPU — A processor designed to handle many calculations in parallel. It performs model computations during AI inference.
Back to the text
Unified memory — An architecture where the CPU and GPU share one physical memory pool.
Back to the text
Checkpoint — A file containing saved model weights and related state. Versions or tasks in one model family may use different checkpoints.
Back to the text
Time to First Token — The time from sending a request until its first output token arrives.
Back to the text
Prompt — The input text sent to a model. For generative models it describes the desired result and strongly shapes the output.
Back to the text

Frequently asked questions

What should you check first in this article?: MIA's GLM-5.3-Flash Spark recipe adds 96 patches to TensorFold 0.6.0. With its default EXL3 4-bit setup, single-request decode measured 60.4 tok/s on two Sparks, 77.6 on three, and 95.2 on four. Aggregate throughput was 108.8 tok/s at four requests on two Sparks, 146.2 at four on three, and 222.4 at eight on four. The four-Spark run uses the experimental tp4 branch.
How much memory does it need?: Each Spark uses a GB10 with 128 GB unified memory and needs about 110 GiB free when the server starts. For the two-Spark recipe, install Docker and the NVIDIA Container Toolkit on both machines and verify key-based SSH from head to worker. Reserve disk for the checkpoint, DFlash2, and container image.