Response speed and acceleration

LLM memory bandwidth vs compute performance

GPU's AI calculation performance is much higher, but local models are not as fast as expected. Did I read the spec sheet wrong? Even if you are ready to compute, you can still wait for the model weights to arrive. If you divide ‘how much you calculate’ and ‘how quickly you get it’, you can see this difference.

The processor can be waiting too

The model reads many weights and creates the next token. When each person receives tokens one by one, the computation unit has to retrieve data from memory to process, which can limit speed. The amount of data that can be retrieved per second is memory bandwidth.

For the same reason, we must also distinguish between memory capacity and bandwidth. The larger the capacity, the easier it is to store the model, but the speed of reading the weights does not increase at the same time. The advantage of running a large model and the advantage of quick response must be checked separately.

Illustration of how model weights flow from memory to a compute device each time you create one output token. — Single-user decode is significantly affected by memory bandwidth because it reads the same weights repeatedly.

Let's get a feel for the numbers.

Simplifying that the weight of 20 GB is read once per token and the effective bandwidth is 400 GB/s, the read alone takes about 0.05 seconds. The reciprocal is about 20 times per second. These are not actual product measurements, but rather hypothetical calculations that demonstrate why bandwidth is a limitation.

Calculation, cache access, and program costs are omitted here, and there is also a structure like MoE that does not use all weights every time. Therefore, the actual tok/s cannot be determined in this way. You should also avoid promising purchased performance by putting theoretical bandwidth as is.

Illustration of a prefill matrix calculation that processes many input tokens at once on a wide calculation grid. — Prefill processes input tokens in parallel, making better use of the computing power of the compute unit for long inputs.

Calculations look different when reading long documents

Prefills can process multiple input tokens together. There is greater room to utilize GPU resources by tying calculations together than by creating single tokens sequentially. At this time, differences in computational performance may be more evident.

Servers that batch multiple requests also have different conditions. So equipment that is similar for short conversations may show differences when processing long documents. This is why it is difficult to use TOPS or memory bandwidth as a representative score for all tasks.

Illustration of a bottleneck in which the slower of the memory carrier and compute unit limits overall speed. — Actual speed is determined by whichever memory supply or calculation processing reaches its limit first, so comparison cannot be made based on TOPS alone.

Quantizing reduces the amount to read

Storing the weights in lower bits allows less to be moved from memory. However, the processing and kernel efficiency of using compressed representations for computation also have an impact. You can't just calculate that the speed is doubled because the file is halved.

The manufacturer's theoretical specifications and actual utilization rates are also different. If the model structure, execution program, quantization format, or cooling conditions change, the results will vary even with the same equipment. The specification table is the starting point for understanding the candidate, and the execution results under the same conditions are the next confirmation material.

Decide which number to look at first

If you have a lot of tasks that require long answers alone, look at the decode after loadability. If you are reading long documents or handling multiple requests, you should also look at prefill and throughput separately. It is also important not to mix recordings with acceleration turned on and basic recordings.

If your expensive GPU isn't as fast as you'd like, see where you're waiting before you blindly look for a higher tier. What you need might be more memory or a better execution path. Finding the bottleneck will determine where you look next in the spec sheet.