Local AI Guide

See the speed before choosing hardware

Executable programs and extensions

GLM-5.3 744B on two M5 Ultras: memory and link requirements

The full GLM-5.3 model is split across two Macs, and each needs 256 GB of memory and a wired connection.

TensorFold 1.0.8 runs the full GLM-5.3 744B model across two M5 Ultras with kindlingai's trellis-coded EXLR8 home profile. Each machine has 256 GB unified memory1; download the checkpoint2 by rank and connect the machines over a cable. At context 8192 and parallel 4, each rank needs 159.6 GB. A single stream decoded at 31–32 tok/s with MTP3 and 21 tok/s without it; prefill4 was about 200 tok/s.

Contents
Requirements and key details
  • Both machines need to be M5 Ultras with 256 GB unified memory.
  • At context 8192 and parallel 4, each rank needs 159.6 GB.
  • MTP raised single-stream decode from 21 to 31–32 tok/s.

Split one model across two Mac ranks

TensorFold's GLM-5.3 EXLR8 path splits checkpoint experts and attention5 heads across two ranks. One Mac does not hold the complete 744B model. Each rank works on the same request and computes its share of the model.

The figures here apply to the full GLM-5.3 744B home profile. The Flash model uses a different checkpoint and runtime6 path, so their memory and speed figures are kept separate. Before setup, confirm the model name, memory capacity, and TensorFold version on both Macs.

Two compact desktop computers connected to a monitor showing streams of particles
Each Mac rank works on the same request over the cable link.

Check per-rank memory and MTP figures

Each Mac downloads half of the expert weights (139.5 GiB) plus 11 GB of whole files shared by each rank. These are checkpoint file sizes. Separately, a running rank uses 159.6 GB at context 8192 and parallel 4. File size and runtime memory measure different things and use the source's different units; leave room for the OS, apps, and cache even on a 256 GB Mac.

Two-Mac conditions for the GLM-5.3 home profile
SettingValueHow to read it
ModelFull GLM-5.3 744B · EXLR8 homeFlash excluded
HardwareTwo M5 Ultras · 256 GB eachConnect ranks over a cable
Conditioncontext 8192 · parallel 4159.6 GB per rank
decodeMTP on 31–32 · off 21 tok/sSingle stream

On two M5 Ultras, a single stream decoded at 21 tok/s with MTP off and 31–32 tok/s with it on. Prompt7 prefill was about 200 tok/s. These are results for one stream's decode8 and prompt processing, not concurrent-user throughput9 or a long-run average.

Prepare both ranks and check the link

Install TensorFold 1.0.8 and pull the same checkpoint revision on both Macs with --node 0/2 and --node 1/2. These numbers select the download shard. For serving, start rank 1 first and start rank 0 within five minutes.

Pull rank 0 shard
tensorfold pull kindlingai/glm-5.3-exlr8-k2-k3-k4@f459f5d0272a529403dd9b3fd32f771f4250a0d4 --exlr8 home --node 0/2 --dir ~/models/glm-5.3-exlr8-home
Pull rank 1 shard
tensorfold pull kindlingai/glm-5.3-exlr8-k2-k3-k4@f459f5d0272a529403dd9b3fd32f771f4250a0d4 --exlr8 home --node 1/2 --dir ~/models/glm-5.3-exlr8-home

First prepare the network on both Macs. Follow the MCDMA Thunderbolt fabric setup to enable RDMA in Recovery, then check the cable-connected interface and peer address. This setup uses the built-in Thunderbolt RDMA path and does not use the separate security changes for Mac and ConnectX drivers.

From the MCDMA repository root, run the build below in a regular macOS terminal. When build/rpc/libmcdma-fabric.dylib is created and passes the official fabric connectivity check, prepare the rank JSON files.

Build the fabric library on both Macs
git clone https://github.com/ashhart/MCDMA.git
cd MCDMA
make -C rpc
ls -l build/rpc/libmcdma-fabric.dylib
printf '%s\n' "$PWD/build/rpc/libmcdma-fabric.dylib"

Use ~/models/glm-5.3-exlr8-home on both Macs. Save rank0.json there on Mac A and rank1.json on Mac B. Set library to the absolute library path printed earlier, and device and via to the actual RDMA device and cable-peer address. Replace 192.0.2.x and /path/to with actual values, then start rank 1 followed by rank 0. When both logs are ready, send requests to Mac A on port 8080.

Mac A: $HOME/models/glm-5.3-exlr8-home/rank0.json
{
  "rank": 0,
  "library": "/path/to/MCDMA/build/rpc/libmcdma-fabric.dylib",
  "links": [{"peer": 1, "device": "rdma_en4", "via": "en4/192.0.2.2", "port": 7490, "name": "speedup"}]
}
Mac B: $HOME/models/glm-5.3-exlr8-home/rank1.json
{
  "rank": 1,
  "library": "/path/to/MCDMA/build/rpc/libmcdma-fabric.dylib",
  "links": [{"peer": 0, "device": "rdma_en4", "via": "en4/192.0.2.1", "port": 7490, "name": "speedup"}]
}
Start rank 1 on Mac B first
tensorfold serve ~/models/glm-5.3-exlr8-home --speed-up "$HOME/models/glm-5.3-exlr8-home/rank1.json" --context 8192 --parallel 4 --temperature 0
Start rank 0 on Mac A within five minutes
tensorfold serve ~/models/glm-5.3-exlr8-home --speed-up "$HOME/models/glm-5.3-exlr8-home/rank0.json" --context 8192 --parallel 4 --temperature 0 --name glm-5.3
Send an API request after both logs are ready
curl -fsS http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"glm-5.3","messages":[{"role":"user","content":"Say READY in one word."}],"max_tokens":8,"temperature":0}'
A desk with a large monitor showing code-like lines, a desktop tower, and a laptop
Check the server log to confirm that both ranks are ready.

Set the working limits after the first response

When both ranks are ready and return a reply, the link is working. Change context or parallel one at a time. A longer context expands the state held for each request; more parallel requests require more memory at once. Record response delay and memory pressure together to identify a suitable setting.

Printed reports and an open notebook around a laptop displaying documents and charts
After the first reply, adjust context and concurrent requests for your workload.

Terminology notes

  1. Unified memory — An architecture where the CPU and GPU share one physical memory pool.

    Back to the text

  2. Checkpoint — A file containing saved model weights and related state. Versions or tasks in one model family may use different checkpoints.

    Back to the text

  3. MTP — A training objective or model component for predicting multiple future tokens at once. In speculative decoding, those predictions can be proposed as candidate tokens.

    Back to the text

  4. Prefill — The stage where an LLM reads the input prompt and computes representations for its tokens. Longer prompts contain more tokens to process.

    Back to the text

  5. Attention — A neural-network operation that calculates relationships between input tokens and combines information from each position into the current representation.

    Back to the text

  6. Runtime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.

    Back to the text

  7. Prompt — The input text sent to a model. For generative models it describes the desired result and strongly shapes the output.

    Back to the text

  8. Token decoding — The stage that generates successive output tokens using the input and the response produced so far.

    Back to the text

  9. Throughput — The amount of work processed or generated over time. Comparisons need the unit, such as tokens per second or requests per second.

    Back to the text

Frequently asked questions

What should you check first in this article?
TensorFold 1.0.8 runs the full GLM-5.3 744B model across two M5 Ultras with kindlingai's trellis-coded EXLR8 home profile. Each machine has 256 GB unified memory; download the checkpoint by rank and connect the machines over a cable. At context 8192 and parallel 4, each rank needs 159.6 GB. A single stream decoded at 31–32 tok/s with MTP and 21 tok/s without it; prefill was about 200 tok/s.
How much memory does it need?
Each Mac downloads half of the expert weights (139.5 GiB) plus 11 GB of whole files shared by each rank. These are checkpoint file sizes. Separately, a running rank uses 159.6 GB at context 8192 and parallel 4. File size and runtime memory measure different things and use the source's different units; leave room for the OS, apps, and cache even on a 256 GB Mac.