Executable programs and extensions
Last updated
What actually improves when you connect two DGX Sparks?
Two DGX Spark systems add memory and room for concurrent work
With two DGX Sparks, you can split a large model across the machines or let each one handle different requests. The appropriate setup depends on whether the model fits on one machine and how many people will use it at once.
Consider model splitting if one machine lacks enough memory, or replication if requests from several people create a long queue. These approaches assign different jobs to the machines. First check which setups the runtime1 supports, then compare performance for the problem you want to solve.
Identify why a second system is needed
If one system lacks enough available memory for the weights and conversation KV cache2, a supported runtime can partition the model across two nodes. The first benefit is running a model that would not fit on one node, not necessarily generating an answer faster.
Each Spark is specified with 128 GB, for 256 GB nominally across two systems. This does not create one 256 GB device in the application. The runtime must distribute weights and cache and exchange results between nodes. If components or workspace are concentrated on one node, that node can run out first even when aggregate memory remains; check per-node use and placement during execution.
If the model fits on one unit, first define what the second unit should do. The setup differs depending on whether you split one answer across two nodes or replicate the model to handle separate requests.
Partitioning one answer adds communication to computation
Splitting a model can reduce the computation assigned to each node, but intermediate results must be transferred for the next step and the nodes must synchronize. A single user's answer gets faster only if the saved compute time outweighs communication and waiting.
Placing earlier and later layers on different nodes differs from splitting computation within a layer: the timing and volume of data exchange change. That is why network bandwidth or device specifications cannot simply be added to predict single-response speed. Prefill3 for a long input and token-by-token decode4 also have different patterns, so one phase's figure does not describe an entire document task.
Match comparison conditions to isolate the configuration difference
When you see a claim that two systems were faster, check whether the model file, quantization5, runtime version, MTP6 setting, input length, and concurrency were held constant. If only the two-system setup used a different quantization or MTP, the result may compare real purchase configurations, but it does not isolate the effect of adding one system.
Run two comparisons: first use shared settings to isolate node count and model partitioning; then apply each system's practical optimized settings. Record the model file and format, input/output conditions, time to first token7, and completion time. Mark any option available on only one side as a configuration difference.
Also verify packages for Spark's ARM64 environment, a supported QSFP cable, inter-node communication settings, and that the program recognizes the partitioned execution path. Without these checks, an unsupported setup can be confused with the inherent cost of two-node communication.
With multiple requests, replication and partitioning behave differently
If a model fits on one system, identical copies can be placed on both nodes. If the scheduler sends request A to one and request B to the other, the systems may serve different users concurrently. The relevant measures then include completed requests over time and queueing, not just the speed of one answer.
By contrast, when request A is partitioned across two nodes, both nodes work together on A. Do not assume request B will automatically be assigned independently to a free node. Partitioning a model that cannot fit on one system and replicating a model to distribute requests solve different problems.
Time to first token, generation speed, and multi-request throughput8 are distinct metrics. Aggregate throughput can improve while an individual request or the longest-waiting user still sees delay. A single-request comparison does not replace a concurrent-load test.
Decide what work the second system will handle
If one system cannot fit the desired model and cache, a second Spark may expand what you can run. If the required model already works well and single-user waits are acceptable, consider whether another system has enough work to justify its connection and management overhead. Actual request queues or a need for a larger model provide a concrete reason.
Diagnose the stage of the bottleneck. If loading fails, inspect memory placement and supported paths. If the model loads but long inputs or concurrent requests wait too long, check what partitioning, replication, or request scheduling features are available. Then compare the target model and runtime directly.
The source comparison tool estimates the experience of one versus two Spark systems with Qwen3.8 27B and a 4K input. Recommended formats and acceleration can differ by device, and this is not a concurrent-load test. Toggle whether prefill is included to inspect one user's wait, then measure the actual target configuration before buying.
Terminology notes
Runtime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.
Back to the textKV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.
Back to the textPrefill — The stage where an LLM reads the input prompt and computes representations for its tokens. Longer prompts contain more tokens to process.
Back to the textToken decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the textQuantization — A way to represent model weights or values with fewer bits, reducing storage and memory use. The coarser representation can also affect accuracy.
Back to the textMTP — A training objective or model component for predicting multiple future tokens at once. In speculative decoding, those predictions can be proposed as candidate tokens.
Back to the textTime to First Token — The time from sending a request until its first output token arrives.
Back to the textThroughput — The amount of work processed or generated over time. Comparisons need the unit, such as tokens per second or requests per second.
Back to the text
Frequently asked questions
- What should you check first in this article?
- With two DGX Sparks, you can split a large model across the machines or let each one handle different requests. The appropriate setup depends on whether the model fits on one machine and how many people will use it at once.
- How much memory does it need?
- Each Spark is specified with 128 GB, for 256 GB nominally across two systems. This does not create one 256 GB device in the application. The runtime must distribute weights and cache and exchange results between nodes. If components or workspace are concentrated on one node, that node can run out first even when aggregate memory remains; check per-node use and placement during execution.