Response speed and acceleration

LLM memory bandwidth vs compute: how to find the speed bottleneck

Memory bandwidth and compute performance reduce different kinds of waiting.

Memory bandwidth¹ is the amount of data that can move from memory to the compute units each second. GPU² compute performance is the amount of computation those units can perform on that data. Generating an answer token³ by token and reading a long input involve different workloads, so the two specifications affect them differently.

When summarizing a document, distinguish a late first response from an answer that continues slowly after it starts. The first part involves prefill⁴, which processes the input; the second involves decode⁵, which generates the answer. Knowing where you wait helps you decide which hardware specifications to compare first.

Contents

Requirements and key details

Memory capacity is space for data; bandwidth is the rate at which data moves.
Single-user decode can be strongly affected by how fast weights are supplied.
GPU compute performance can matter more for long inputs and multiple requests processed together.

Generating tokens one by one repeatedly reads the weights.

An LLM⁶ uses weights stored in the model to calculate the next token. For one person's answer, each next token follows the preceding token. The required weights are repeatedly fetched from memory. If the compute units are waiting for data, reducing that wait matters before adding more compute units.

Capacity and bandwidth serve different roles here. Enough capacity is needed to hold the model and KV cache⁷, while bandwidth supplies data for computation quickly. If the model fits but answers are slow, inspect actual decode speed and the data path rather than simply adding more capacity.

Model weights flowing from memory to the compute units — Token-by-token generation repeatedly supplies the required weights.

Let's calculate the time to read 20 GB.

Suppose the weights needed for one token occupy 20 GB and the actual transfer rate is 400 GB/s. Reading them takes 20÷400=0.05 seconds. Repeating that read for one second gives 1÷0.05=20 reads. Less data or a higher transfer rate reduces the waiting time.

This example calculates only weight-reading time. Actual tok/s also includes computation, KV-cache access, and runtime⁸ overhead. We assumed an actual transfer rate rather than a product's peak bandwidth; selective architectures such as MoE⁹ also change how much is read. Use the formula to understand bandwidth, and compare hardware using results for the same model.

Reading a long document processes many tokens together.

After receiving a document, the model processes the input tokens before answering. Prefill can group the tokens already provided, creating more opportunity to use the compute units. If a long contract produces a long wait before the first answer, compare prefill throughput¹⁰ and time to first token¹¹.

Batching requests from several people can reuse the same weights across more computations. The server's aggregate token rate may increase while individual requests wait in a queue. For personal chat, look at one answer's speed; for a shared server, examine individual latency and aggregate throughput together.

A prefill matrix computation processing many input tokens across a wide grid — Prefill can use more compute resources by processing a long input together.

Quantization and context length also change data movement.

Quantization¹² stores weights in fewer bits and can reduce the amount read. Efficient kernels for that representation can help both capacity and speed. Record the format and runtime even for the same original model so you can identify which combination reduced the time.

As a conversation grows, the KV cache storing previous context grows too. Consulting that cache during generation can add work, making short-question and long-document speeds different. Fix the input length and record output tokens to reduce differences caused by changing workloads.

A bottleneck between the memory conveyor and the compute device — Reducing the slower of data supply and computation shortens the actual wait.

Compare hardware for the stage that is slow.

First check that the model and actual context fit in memory. Then use the same document and question to record time to first token, generation rate, and completion time. Separating them shows which part of your work a higher-spec machine can shorten.

Use the slow stage to choose what to investigate next.
Slow stage	Compare first
The answer continues slowly.	Same-model decode, bandwidth, and actual execution path
The first answer is late after a long input.	Prefill and TTFT at the same input length
Several requests queue up.	Individual latency and aggregate throughput under concurrency

Terminology notes

Memory bandwidth — The amount of data transferable between memory and a processor per unit time, usually expressed in bytes per second (B/s).
Back to the text
GPU — A processor designed to handle many calculations in parallel. It performs model computations during AI inference.
Back to the text
Token — A unit into which a model divides input or output for processing.
Back to the text
Prefill — The stage where an LLM reads the input prompt and computes representations for its tokens. Longer prompts contain more tokens to process.
Back to the text
Token decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the text
Large language model — A model trained on large text datasets to learn language patterns and process or generate text in context.
Back to the text
KV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.
Back to the text
Runtime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.
Back to the text
MoE — A model architecture that selects some of several expert subnetworks for each input. Total parameters can differ from the number activated for one token.
Back to the text
Throughput — The amount of work processed or generated over time. Comparisons need the unit, such as tokens per second or requests per second.
Back to the text
Time to First Token — The time from sending a request until its first output token arrives.
Back to the text
Quantization — A way to represent model weights or values with fewer bits, reducing storage and memory use. The coarser representation can also affect accuracy.
Back to the text

Frequently asked questions

What should you check first in this article?: Memory bandwidth is the amount of data that can move from memory to the compute units each second. GPU compute performance is the amount of computation those units can perform on that data. Generating an answer token by token and reading a long input involve different workloads, so the two specifications affect them differently.
How much memory does it need?: An LLM uses weights stored in the model to calculate the next token. For one person's answer, each next token follows the preceding token. The required weights are repeatedly fetched from memory. If the compute units are waiting for data, reducing that wait matters before adding more compute units.