Executable programs and extensions

What actually improves when you connect two DGX Sparks?

The model was launched with just one Spark. However, the response is slower than expected. Will connecting one more device of the same device solve the problem? The answer depends on what tasks the second device will share. From holding a larger model to reducing the wait for one person, we'll break it down for you.

When the model will not fit on one device

The first reason is relatively obvious. This is a case where the desired model and conversation cache do not fit into one memory. If an executable program can split the model across two nodes, it can expand its executable range with space on a second machine.

The total notation of 256GB is the sum of two 128GB memories. Not just any program is using 256GB in one chunk. You need a distributed execution path that determines how to divide the model and how much cache to leave on each machine.

Illustration showing how one and two DGX Spark configurations expand the model storage space and processing passages differently.
The most obvious benefit of two connections is larger model load and the leeway to handle concurrent requests, rather than doubling the single response rate.

If it already fits on one device, ask a different question

Now suppose the model already fits comfortably on one device. Splitting it across two can reduce the computation each device performs. But they must exchange intermediate results over the network and wait for each other. The computation saved has to outweigh that extra communication time before a single answer becomes faster.

There are also differences depending on the division method. The communication timing is different when dividing the front and back of a model and when dividing one layer of calculation together. This is why the decode speed cannot be calculated simply by adding the bandwidth figures of the two devices.

Illustration showing the connection section where calculation results are exchanged and synchronized between two AI nodes
Dividing the model into two nodes adds communication and synchronization at every step, making it impossible to sum up the equipment performance as is.

What do I mean by optimal settings?

If you see fast results, you should look not only at the number of devices, but also the model files, quantization, MTP, input length and number of concurrent requests. If one device has the default settings and the two devices use MTP and a different compression version, the difference cannot be attributed entirely to the effect of the second device.

The order of comparison is to first decide on the settings available on one device and then view them side by side with the paths of two devices of the same model. If you can use a feature on two devices that is not available on one device, that is an important advantage, but you must leave a record of what has changed so that you can understand the results even after the next update.

Illustration of a server throughput-oriented configuration in which two AI nodes divide multiple requests and process them simultaneously.
The throughput advantage of two nodes becomes more evident when processing requests from multiple users together rather than in one long answer.

The story changes when you do multiple things at the same time.

If your server is receiving requests from both document processing and coding agents at the same time, rather than just chat, overall throughput becomes important. You can duplicate each model to handle different requests, or configure a distributed model to handle more requests. Which one is correct depends on model size and runtime.

At this time, individual requests may be delayed even if the total number of tokens increases. You should record the time of the first token of a request divided by the decode speed and the number of requests completed in that same amount of time. If you are in a family or team-sharing environment, don’t forget the experience of the person who waited the longest.

What do you want the second device to buy you?

If you really need a larger model and can't run it with just one, the increased capacity itself is a reason to buy it. On the other hand, if your goal is to quickly have a short conversation alone, you should pay the cost of managing and connecting two devices to see how much waiting you can reduce.

Compare the estimated experience of one device and two on this site. The comparison is not a measurement of every model using the best result from a particular patch or runtime. If the difference is small, consider tuning the device you already have. If two devices make a new workload possible, judge the upgrade by that workload. The second device should have a clear job to do.