Executable programs and extensions

Running local LLMs across two or more GPUs

Can two 24GB GPUs replace one 48GB GPU? Although the sum of memory appears to be the same, the path the calculation takes is different. Before buying another used card, let's check how the program I'm using divides the work between the two cards.

Detecting two GPUs is not the same as using them together

Even if the operating system recognizes two GPUs, if an app only uses one, the remaining memory will not benefit that model. The executor must split the weights and cache. Each card must also have its own workspace, so the entire total cannot be filled with model files.

Therefore, rather than looking for how to set up the card first, check whether the desired model/type supports multiple GPUs first. Even in the same app, the splitting path may be different depending on the model structure. Already verified running examples are important in narrowing down your purchase candidates.

Comparison illustration of placing requests of different lengths into cache blocks on a single GPU and dual GPUs communicating with each other. — Increasing GPUs can increase memory and throughput, but introduces new variables in how devices communicate and request placement.

Split the model's earlier and later layers

You can consider splitting the layers by putting the front part of the model on one card and the back part on another card. Since the calculation of one token goes through the cards one after the other, intermediate results must be passed. The two cards are not always busy at the same time.

If you have multiple requests, there is room to take advantage of the empty time between steps, but the effect may be different when you get one response alone. That's why we split the capacity benefits and single request rate benefits of this approach.

Illustration of dividing one large model into two GPU memories and combining partial calculation results — The runtime must explicitly split layers or tensors to utilize the memory of two GPUs for one model.

You can also split a layer of calculations together

Tensor parallelism divides the calculations in one layer between multiple GPUs and exchanges the results. Communication is repeated at the expense of being able to reduce computations to parallelism. If you have a slow connection, the benefit of splitting the computation may be overshadowed by the transfer time.

Check the PCIe generation, actual number of lanes, and connection structure between GPUs. NVLink support also depends on the card and configuration. This difference is not visible by calculating that the token speed is doubled by adding the two theoretical bandwidths.

Illustration of the speed difference between two GPUs exchanging intermediate calculation results with narrow legs and wide legs — Even if each GPU is fast, if PCIe or the interconnect is slow, result exchange becomes a bottleneck and the speed does not increase as much as the number of cards.

Loading a copy on each device is a different approach

If the model fits on one GPU, you can load a copy on each card and distribute requests between them. That can help serve more users. It does not combine their memory for a single request, nor does it make one answer twice as fast.

You need to decide whether your goal is one large, personal model, or whether you want to handle requests from multiple apps simultaneously. Even if you buy two of the same card, what the second card does depends on whether you split or duplicate.

Check the host PC requirements missing from the GPU price tag

Even if there are two slots, they may not be installed together due to the thickness of the cards. You should also check the power supply, cables, cooling and case space. Noise and power in the space where it will operate for a long time are also costs outside of the card price.

If you absolutely have to run large models, there is a reason to tolerate this complexity. If you're just looking to speed up an already one-person model, compare the overall cost to a single-card alternative. Having the right amount of memory and creating a computer that is comfortable to use every day are two different things.