What should you check first in this article?

Strata's multi-GPU mode assigns contiguous model layers to different cards using pipeline/layer parallelism. Select cards with --gpus 0,2 or configure gpu and layer_split. In the RTX 5080 + 3090 Coder results, prompt throughput rose about 18–20%, while decode results differed between story and code tasks. NVLink is not required, but adding a slower card does not guarantee a speedup.

Executable programs and extensions

Strata Multi-GPU Layer Splitting: Setup and Results

The GPUs take different layer stages; their memory is not simply pooled.

Strata's multi-GPU mode assigns contiguous model layers to different cards using pipeline/layer parallelism. Select cards with --gpus 0,2 or configure gpu¹ and layer_split. In the RTX 5080 + 3090 Coder results, prompt² throughput³ rose about 18–20%, while decode⁴ results differed between story and code tasks. NVLink is not required, but adding a slower card does not guarantee a speedup.

Contents

Requirements and key details

Layer splitting assigns contiguous model ranges to different GPUs.
NVIDIA multi-GPU does not require NVLink; handoffs pass through host memory.
Compare prompt processing and answer decode separately while holding model and test conditions constant.

Layer splitting assigns a range of model layers to each card

Strata's multi-GPU layer split assigns contiguous ranges of one model's layers to different cards. The first GPU processes early layers and hands results to the next GPU, which processes later layers. Each card keeps an expert cache for its own layers; the last GPU also runs the output head and draft/MTP layer.

This is layer/pipeline parallelism. Each GPU handles different layers; unlike tensor parallelism⁵, it does not split one layer’s matrices across GPUs.

Token⁶ data crosses between cards once per verify window through host memory. NVLink and GPU peer-to-peer support are not required, so PCIe x4 or x1 can work.

A black tower PC, a blank monitor, and RAM modules beside it — Each GPU handles a different range of model layers.

Another card helps when one card cannot cache enough experts

When two cards split the layers, each caches experts for its own stage. A second card can reduce CPU⁷ work during decode when one card cannot hold enough of the frequently used experts. Every card also needs dense weights, the full context's KV state, a verify window, and prompt buffers, so summed VRAM⁸ alone does not determine whether a model fits.

Adding a card also adds PCIe transfer overhead, so a slower GPU does not always make the run faster. With two cards, Strata tests split points and chooses the layout that fits the most expert profiles.

If the cards use different architectures, check runtime⁹ support and build requirements. NVIDIA and AMD cards cannot be mixed in one run.

The setup tool flags a second card with less than 8 GB of VRAM. Each card needs room for dense weights and the prompt buffer, so limited VRAM can constrain the whole layout.

Before adding a card, check cache hit rate and CPU use with the current single-card setup. Decide which bottleneck the second card should reduce, then test it.

Per-card memory and requirements

Choose GPUs through setup or the config file

On Windows, run START-HERE.bat --gpus 0,2 from the Strata folder to select and remember two cards. On Linux, use ./setup.sh --gpus 0,2. NVIDIA device numbers follow nvidia-smi. --gpus all selects every supported card; --gpu 0 selects one card for that start only.

In the config, set the gpu array and layer_split. For example, gpu [0, 2] with layer_split auto enables automatic placement. A manual layer_split of 26 means the second card starts at layer 26. For three cards, enter the first layer of each later stage in ascending order, such as 18,32. These are boundary indices, not layer counts.

Strata reports the selected card order in its startup log. The later main/0.1.41 line of the docs and setup.py says auto ordering places the faster card last. That auto-sort and the gpu_order setting are not in v0.1.40.2, so for that version use the order shown by setup and the startup log.

A graphics card, RAM, and an NVMe SSD on a cream-colored wood desk — Account for each card's memory and PCIe path when adding a GPU.

The 5080 + 3090 showed larger gains in prompt processing

These Coder results used a Ryzen 9 9950X3D and 32K context. They separate a single RTX 5080 from RTX 5080 + 3090 automatic and manual splits, and report prompt processing at two lengths separately from story and code decode.

In a separate setup, adding a slower RTX 2080 Ti as a third card lowered story and code decode to 68 and 90 tok/s. Check which workload bottleneck another card would solve rather than relying on card count or summed VRAM.

Scroll sideways to compare all columns.

Coder results: single RTX 5080 and 5080 + 3090 split
Setup	16K prompt tok/s	28K prompt tok/s	Story decode tok/s	Code decode tok/s
RTX 5080 alone	1,726–2,017	1,970	83–87	88–105
5080 + 3090 · auto K=22	2,037	2,073	80	109
5080 + 3090 · manual K=26	2,039	2,357	84	110

Source: the Coder results table in Strata v0.1.40.2 MULTI_GPU.md. Prompt throughput and answer decode are different workloads; read the columns separately.

RTX 5080 and 3090 split benchmark

Two graphics cards installed inside an assembled PC case — Two graphics cards are installed inside the PC.

Terminology notes

GPU — A processor designed to handle many calculations in parallel. It performs model computations during AI inference.
Back to the text
Prompt — The input text sent to a model. For generative models it describes the desired result and strongly shapes the output.
Back to the text
Throughput — The amount of work processed or generated over time. Comparisons need the unit, such as tokens per second or requests per second.
Back to the text
Token decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the text
Tensor parallelism — A method that splits matrices and computations within a model layer across devices for joint execution. Devices communicate to combine partial results.
Back to the text
Token — A unit into which a model divides input or output for processing.
Back to the text
CPU — The central processor that runs the operating system and general-purpose instructions; in AI workloads it also handles tasks such as preprocessing and data movement.
Back to the text
VRAM — Memory used by a graphics card’s GPU to store model weights and intermediate values.
Back to the text
Runtime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.
Back to the text

Frequently asked questions

What should you check first in this article?: Strata's multi-GPU mode assigns contiguous model layers to different cards using pipeline/layer parallelism. Select cards with --gpus 0,2 or configure gpu and layer_split. In the RTX 5080 + 3090 Coder results, prompt throughput rose about 18–20%, while decode results differed between story and code tasks. NVLink is not required, but adding a slower card does not guarantee a speedup.
How much memory does it need?: Token data crosses between cards once per verify window through host memory. NVLink and GPU peer-to-peer support are not required, so PCIe x4 or x1 can work.
When is it better not to choose this?: When two cards split the layers, each caches experts for its own stage. A second card can reduce CPU work during decode when one card cannot hold enough of the frequently used experts. Every card also needs dense weights, the full context's KV state, a verify window, and prompt buffers, so summed VRAM alone does not determine whether a model fits.