2× NVIDIA A40 NVLink 96GB Configuration to run the model across two devices · Total memory 96GB
Enterprise dual workstation configuration with 2× 48GB ECC VRAM (96GB combined, 1,392 GB/s combined local bandwidth) and NVLink (112.5 GB/s). Runs 70B to 120B models with tensor parallelism.
Speed and memory for the same model
Hardware core specifications
- Available memory
- 92GB
- Memory bandwidth
- 1392GB/s
- Estimated sustained LLM load
- About 700W
- Korean price range (KRW)
- 1,260 × KRW 10,000–1,260 × KRW 10,000
Who it fits
- Utilize 92GB of available memory
- Two-Device Distributed Inference
- GLM-5.3-Flash (MoE) model comparison
Skip it if
Even if you connect two devices, the memories are not automatically merged into one. Distributed runtime support and interconnect bottlenecks must be checked together.
Recommended format for each device 4K input
The model you will experience with this equipment
K2 Horizon 0.9B
Required 1.03GB · spare time
llama.cpp CUDA · Q4_K_M · GPU split into 2 pieces
Decode
1285.9 ~ 1765 tok/s
First token
0.19 to 0.44 seconds
GLM-5.3-Flash (MoE)
recommendedRequired 190.21GB · tight
llama.cpp CUDA · Q4_K_M · GPU split into 2 pieces
Decode
3.4 ~ 3.4 tok/s
First token
6.3 to 14.0 seconds
Actual speed varies with runtime version, cooling, context length and quantization file.
On sales pages, look at memory before chips.
The model and speed assumptions above change if the product is not the 96GB configuration.
Frequently asked questions
- What LLM can be run locally using 2× A40 96GB (NVLink)?
- K2 Horizon 0.9B, GLM-5.3-Flash (MoE) and others with a 4K input and the recommended format. Required memory varies by model and context length.
- What is the power consumption of a local LLM using 2× A40 96GB (NVLink)?
- Sustained load is about 700 W, with an actual range of 550–850 W.
- Are the displayed token speeds ground truth?
- This is a range aligned to collected benchmark measurements under the same conditions. Combinations without a matching measurement are estimated from the nearest result and vary with runtime, cooling, quantization file and context length.
Equipment to compare together
similar memory
Studio M4 Max 128GB
Suits people who want to compare a wide range from 70B-class to high-precision models in a single finished machine.
similar memory
Ryzen AI Max+ 395 128GB
It is attractive when you have large amounts of integrated memory at a low price and can handle runtime settings directly.
similar memory
DGX Spark 128GB
This is for handling a large model on one machine without splitting it across several graphics cards. You should have a reason to buy 128GB. If the model you need now fits in 24GB or 32GB, do not assume Spark will be faster just because it has more memory.