What should you check first in this article?

Executable programs and extensions

Two RTX 3090s with NVLink: how much it actually changes inference

In inference NVLink was about a 5 percent difference, and the default setup works without a bridge at all.

The RTX 3090 has an NVLink 3.0 connector, but a bridge must be physically installed and most consumer setups do not have one. In a 2x RTX 3090 vLLM tensor-parallel community benchmark, NVLink led at 85.0 vs 81.1 tok/s, about 5 percent, with a steadier p95 of 239.1 vs 277.6 seconds. The default multi-GPU path runs on PCIe with P2P off.

Contents

Requirements and key details

In the community benchmark, NVLink raised decode throughput about 5 percent and cut p95 latency.
Tensor parallelism works over PCIe without a bridge, and the default setup leaves P2P off.
Whether P2P is really on has to be checked at three layers: driver, NCCL and vLLM.

What an NVLink bridge does

The RTX 3090 supports NVLink 3.0 on GA102, so a pair of cards can carry four links of 112.5 Gbit/s each for 112.5 GB/s bidirectional in total, 56.25 GB/s per direction. PCIe 4.0 x16 provides about 32 GB/s per direction, so the dedicated path between cards is much wider.

Having an NVLink connector on the 3090 is not the same as having a bridge installed. The reference notes state that a bridge has to be physically installed and that most consumer setups do not have one. Without it, multi-card setups run over PCIe.

Two graphics cards mounted side by side inside a case with an NVLink bridge connecting them — A bridge changes the path between cards; it adds neither compute nor capacity.

Measured: NVLink on and off at TP=2

A community benchmark ran the same model on two 3090s with vLLM tensor parallelism¹ (TP=2). With Devstral Small 2505 Q8_0, 50 code generation prompts² and responses averaging about 1650 tokens³, NVLink gave 85.0 tok/s against 81.1 tok/s over PCIe alone, roughly a 5 percent difference.

The tail latency stands out more: the 95th percentile response time was 239.1 seconds with NVLink against 277.6 seconds without, so NVLink was steadier. If variance on long requests matters more than average throughput⁴, that difference counts. The author concluded that for inference⁵ alone the money is better spent on VRAM⁶ or a card than on a bridge.

2x RTX 3090 vLLM TP=2 community measurement (Devstral Small 2505 Q8_0, 50 prompts, responses averaging about 1650 tokens)
Setup	Throughput	p95 response
NVLink enabled	85.0 tok/s	239.1 s
PCIe only	81.1 tok/s	277.6 s

The default is PCIe with P2P off

The 2x3090 operating notes describe the default dual and multi-card path as PCIe-only with P2P off: NCCL_P2P_DISABLE=1 and custom all-reduce disabled, chosen so it works out of the box on any consumer rig without tuning.

The same notes say enabling P2P over PCIe can add a workload-dependent few percent up to about 20 percent. The emphasis is on order: confirm the default PCIe path works, then read the topology and enable P2P to compare, which makes problems easier to isolate.

Three layers to check whether P2P is on

P2P is not one question but three stacked ones: whether the driver grants direct GPU-to-GPU access, whether NCCL⁷ uses that path for transfers, and whether vLLM's extra custom all-reduce kernel is active. Conflating them makes it impossible to explain why P2P looks enabled yet does not speed anything up.

vLLM's gate checks for NVLink by name, but that check is a cheap pre-filter ahead of an expensive one. As a result, rigs with three or more cards over PCIe keep custom all-reduce off by design, because they lack a fully connected NVLink mesh. That is intended behaviour rather than misconfiguration, and P2P benefits can still flow through the NCCL layer.

The three P2P layers
Layer	How to check
Driver	`nvidia-smi topo -p2p rw` reports OK
NCCL	`NCCL_P2P_LEVEL` setting with driver permission
vLLM custom all-reduce	check the startup log for a disabled notice

Two graphics cards stacked inside an open-frame case with power cables attached to each — Splitting across two cards puts power and heat on each and adds a communication cost.

What TP=2 costs and requires

In vLLM, tensor parallelism shards the weights across the cards. The vLLM documentation notes that this leaves more KV cache⁸ space per card but can add synchronization overhead, so context length⁹ and concurrency have to be tuned alongside it.

Enabling chunked prefill¹⁰ splits compute-bound prefill and bandwidth-bound decode¹¹ so both resources get used at once. Power limits apply per card, so two cards mean watching the combined draw and case heat. As the chart above shows, lowering a 3090's power limit lowers decode speed with it.

TP=2 example with P2P checks

nvidia-smi topo -m
nvidia-smi topo -p2p rw

CUDA_VISIBLE_DEVICES=0,2 CUDA_DEVICE_ORDER=PCI_BUS_ID \
vllm serve /models/model.gguf \
  --tensor-parallel-size 2 \
  --max-num-seqs 4 --max-model-len 64000 \
  --gpu-memory-utilization 0.95 \
  --max-num-batched-tokens 16384 --enable-chunked-prefill \
  --quantization gguf

The flags follow the conditions of the measurement above. Adjust them for your cards, model and context length, and read the P2P and all-reduce notices in the startup log.

So should you buy a bridge

For inference, a bridge is the last item to consider. The measured gain was about 5 percent throughput and a better tail, while larger differences come from whether a model can be split across two 24 GB cards or must live on one larger card. With two cards, confirm the default PCIe path works first.

Work that keeps card-to-card traffic running, such as training or fine-tuning, is a different story. The common assessment is that NVLink's bandwidth advantage shows up more in training than in inference, and the benchmark above reached the same conclusion for inference: about 5 percent. Decide whether your use is inference or training before committing.

A bar chart comparing RTX 3090 decode speed of 68.9, 65.3, 62.3 and 51.0 tok/s against measured GPU power of 347.7, 298.7, 274 and 249.5 W at 350, 300, 275 and 250 W limits — Tensor parallelism applies a power limit per card, so with two cards watch the combined draw.

Terminology notes

Tensor parallelism — A method that splits matrices and computations within a model layer across devices for joint execution. Devices communicate to combine partial results.
Back to the text
Prompt — The input text sent to a model. For generative models it describes the desired result and strongly shapes the output.
Back to the text
Token — A unit into which a model divides input or output for processing.
Back to the text
Throughput — The amount of work processed or generated over time. Comparisons need the unit, such as tokens per second or requests per second.
Back to the text
Inference — The process of using a trained model to compute an output for an input. Here, local inference means running the model on the user’s device.
Back to the text
VRAM — Memory used by a graphics card’s GPU to store model weights and intermediate values.
Back to the text
NCCL — A collective-communication library for exchanging and combining data across NVIDIA GPUs.
Back to the text
KV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.
Back to the text
Context window — The token span of input and generated content a model can handle in one request. The supported limit and memory use depend on the model and runtime settings.
Back to the text
Prefill — The stage where an LLM reads the input prompt and computes representations for its tokens. Longer prompts contain more tokens to process.
Back to the text
Token decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the text

Frequently asked questions

What should you check first in this article?: The RTX 3090 has an NVLink 3.0 connector, but a bridge must be physically installed and most consumer setups do not have one. In a 2x RTX 3090 vLLM tensor-parallel community benchmark, NVLink led at 85.0 vs 81.1 tok/s, about 5 percent, with a steadier p95 of 239.1 vs 277.6 seconds. The default multi-GPU path runs on PCIe with P2P off.