Executable programs and extensions

GLM-5.3 Flash on two RTX PRO 6000s: speed and power limits

The two-GPU 300 W setup measured 238.3 tok/s for one request and 448.7 combined at four; four GPUs at 250 W measured 265.9 and 1,063.6 respectively.

The two-GPU recipe uses TensorFold v0.6.5, an EXL3 expert 4-bit checkpoint¹, 4-bit dense layers, and an FP8² KV cache³. Two RTX PRO 6000 Blackwell GPUs were measured at 300 W each, with 238.3 tok/s for one prose request and 448.7 combined for four. The four-GPU recipe measured 265.9 tok/s for one request and 1,063.6 combined for four at 250 W per card. Both GPU⁴ count and per-card power change, so the ratio is not a power-efficiency or retention measure.

This recipe serves GLM-5.3 Flash locally using two RTX PRO 6000 Blackwell GPUs. Send a prompt⁵ and receive an answer, then compare single-request speed and aggregate throughput⁶ using the public records below. Four-GPU results are included with their own power limits and runtime⁷ settings.

Fix the model and GPU setup first

This guide compares two Linux recipes serving GLM-5.3-Flash on RTX PRO 6000 Blackwell GPUs. Each uses one host with either two or four cards, and the guide separates single-request speed from concurrent throughput. The two-GPU recipe is based on TensorFold v0.6.5; the two results use different power and runtime conditions.

A dark monitor, a compact desktop computer, and a PC case with several visible fans — Single-request speed and the aggregate for four requests measure different things.

Separate per-request speed from aggregate throughput

The two-GPU run was measured with sparkDash v1.8.9 at temperature 0, top_p 1, thinking off, and 400-token replies, with each GPU at 300 W. One prose request measured 238.3 tok/s; four requests reached 448.7 combined. The median per-request rate in that four-request test was 114.9 tok/s, so aggregate and individual response flow are different readings.

Scroll sideways to compare all columns.

Prose decode comparison for RTX PRO setups
Setup	Power per GPU	One request	Four-request aggregate
Two RTX PRO 6000s	300W	238.3 tok/s	448.7 tok/s
Four RTX PRO 6000s	250W	265.9 tok/s	1,063.6 tok/s

The four-GPU page reports 265.9 tok/s for one prose request and 1,063.6 combined for four. Each card was at 250 W, on a different TensorFold and recipe version. Because GPU count and per-card power both differ from the two-card 300 W run, a simple ratio cannot show that performance is retained at twice the power.

Start the two-GPU recipe on a Linux host

The two-GPU recipe requires two 96 GB RTX PRO 6000 Blackwell cards in one Linux x86_64 host. Free at least 90,000 MiB on each GPU and confirm CUDA⁸ peer access. It also requires Docker, NVIDIA Container Toolkit, Bash, Python 3, Git, GNU patch, curl, gzip, and flock.

The default recipe uses the DFlash2 drafter under a non-commercial license. If those terms do not fit your use, choose DRAFTER=mtp ./start.sh before starting; that path serves one request at a time. The 238.3 tok/s and four-request figures below are from the default DFlash2 recipe.

The official Aevonix start.sh builds a pinned TensorFold commit with 86 patches, downloads the checkpoint and DFlash2, then runs kernel autotuning and a smoke test. Start with idle GPUs; on success it reports LIVE on port 8022. DRY_RUN=1 ./start.sh prints the steps without using Docker or GPUs.

Start the two-GPU recipe from its official repository

git clone https://github.com/Aevonix/GLM-5.3-Flash-EXL3-2x-RTX-PRO-6000-TensorFold.git
cd GLM-5.3-Flash-EXL3-2x-RTX-PRO-6000-TensorFold
DRY_RUN=1 ./start.sh
./start.sh

Two-GPU TensorFold recipe and launch command

A desk with a large monitor showing code-like lines, a desktop tower, and a laptop — The startup script checks both GPUs and their connection.

Read the power warning and first errors in the log

Send a request with model ID glm-5.3-flash and confirm the first reply. If W_POWER appears, check the configured GPU power limit and record each card's setting with the comparison.

Check the first response through the port 8022 API

curl -fsS http://127.0.0.1:8022/v1/models
curl -fsS http://127.0.0.1:8022/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"glm-5.3-flash","messages":[{"role":"user","content":"Say READY in one word."}],"temperature":0,"max_tokens":32}'

E_P2P indicates a CUDA peer-access problem between the GPUs. Check the driver, PCIe topology, ACS, and IOMMU settings. For E_GPU_MEMORY, stop other GPU work and verify at least 90,000 MiB free on each card.

Printed reports and an open notebook around a laptop displaying documents and charts — Recording power limits and error codes makes later comparisons easier.

Terminology notes

Checkpoint — A file containing saved model weights and related state. Versions or tasks in one model family may use different checkpoints.
Back to the text
FP8 — A floating-point format that represents each value with 8 bits.
Back to the text
KV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.
Back to the text
GPU — A processor designed to handle many calculations in parallel. It performs model computations during AI inference.
Back to the text
Prompt — The input text sent to a model. For generative models it describes the desired result and strongly shapes the output.
Back to the text
Throughput — The amount of work processed or generated over time. Comparisons need the unit, such as tokens per second or requests per second.
Back to the text
Runtime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.
Back to the text
CUDA — A software platform for general-purpose computing on NVIDIA GPUs.
Back to the text

Frequently asked questions

What should you check first in this article?: The two-GPU recipe uses TensorFold v0.6.5, an EXL3 expert 4-bit checkpoint, 4-bit dense layers, and an FP8 KV cache. Two RTX PRO 6000 Blackwell GPUs were measured at 300 W each, with 238.3 tok/s for one prose request and 448.7 combined for four. The four-GPU recipe measured 265.9 tok/s for one request and 1,063.6 combined for four at 250 W per card. Both GPU count and per-card power change, so the ratio is not a power-efficiency or retention measure.
Where can you check the conditions and sources?: The default recipe uses the DFlash2 drafter under a non-commercial license. If those terms do not fit your use, choose `DRAFTER=mtp ./start.sh` before starting; that path serves one request at a time. The 238.3 tok/s and four-request figures below are from the default DFlash2 recipe.