Executable programs and extensions

FlashAttention vs PagedAttention: what each one changes

FlashAttention reduces data movement; PagedAttention manages KV-cache placement

FlashAttention¹ divides attention² computation into tiles to reduce repeated writes and reads of intermediate data in memory. PagedAttention manages KV cache³ for multiple requests in fixed-size blocks, allocating and reclaiming what each needs. One is a computation kernel and the other is per-request memory management; they have different roles and can be supported together.

A request to read a long code repository goes through prefill⁴, which computes relationships among input tokens⁵; during answer generation, KV cache stores state needed to continue. FlashAttention addresses memory movement in attention computation, while PagedAttention handles cache placement across requests.

FlashAttention tiles attention computation for long inputs

The model uses attention to compute which input tokens should inform one another. Longer inputs can require more intermediate results to be written to and read from GPU⁶ memory. FlashAttention processes small tiles and avoids repeatedly storing large intermediate arrays, reducing data movement.

It uses the full input in attention computation while reducing storage and movement of intermediate results. If the first token takes a long time for a lengthy input, record model loading and prefill separately and check runtime⁷ logs to see whether a FlashAttention kernel was selected.

A diagram of a large attention matrix divided into small tiles and processed repeatedly close to the compute units — FlashAttention computes in tiles instead of storing the full intermediate matrix far from the compute unit, reducing memory traffic.

PagedAttention manages per-request KV-cache space

When requests of different lengths arrive, each conversation keeps input and generated-token state in KV cache. PagedAttention manages the cache in fixed-size blocks, allocating what each request needs and reclaiming it when finished. This can reduce unused space caused by reserving large contiguous regions for each request.

When a short request B finishes while request A continues, B's blocks can be reclaimed and assigned to C. A's conversation state remains until A finishes. This shares space more efficiently between requests within the same GPU memory.

More concurrent requests or longer documents still require cache for active states. When input prefill shares GPU compute with ongoing decode⁸, token intervals can lengthen; runtime support for chunking and scheduling long prefill varies. More efficient cache placement still requires physical memory for active state.

Requests of different lengths placed with reduced gaps into fixed-size KV-cache pages — PagedAttention manages cache by assigning needed pages rather than reserving large contiguous regions.

Their different roles allow them to work together

FlashAttention changes how data moves through memory during attention computation; PagedAttention assigns each request's KV cache to blocks. If a runtime supports both, they can optimize prefill computation and request-memory management together. A slow prefill for long code and scarce cache space under team concurrency require separate checks.

Support varies by GPU, model architecture, quantization⁹, and application version, and a runtime may choose a path automatically. Do not infer use from an option name in a menu; check logs for the selected kernel and cache path. Record the working defaults before forcing settings.

Comparison of attention repeatedly moving data to memory versus reducing transfers through tiled processing — The key is not only operation count but how often intermediate data is written to and read from slower memory.

Measure one request and team-server behavior separately

For one short function question, few cache states are concurrent, so PagedAttention's server-memory placement benefit may be limited. FlashAttention also need not have the same effect on prefill and token-by-token decode. Record time to first token¹⁰ and token intervals during generation separately.

For a team server, establish a single-request baseline, then overlap a short question and another analysis with a long code review. Record time to first token, generation interval, peak memory, per-request latency, and aggregate throughput¹¹ to see which kernel or cache change benefits users.

Test one supported runtime option at a time and confirm application in logs. If users cannot configure an option, do not invent one; inspect automatic selection. Check FlashAttention when input computation is the bottleneck and PagedAttention support when cache placement under concurrent requests is the issue.

Terminology notes

FlashAttention — An attention algorithm that computes in tiles, keeps intermediate values in fast on-chip memory, and reduces GPU-memory traffic without storing the full attention matrix.
Back to the text
Attention — A neural-network operation that calculates relationships between input tokens and combines information from each position into the current representation.
Back to the text
KV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.
Back to the text
Prefill — The stage where an LLM reads the input prompt and computes representations for its tokens. Longer prompts contain more tokens to process.
Back to the text
Token — A unit into which a model divides input or output for processing.
Back to the text
GPU — A processor designed to handle many calculations in parallel. It performs model computations during AI inference.
Back to the text
Runtime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.
Back to the text
Token decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the text
Quantization — A way to represent model weights or values with fewer bits, reducing storage and memory use. The coarser representation can also affect accuracy.
Back to the text
Time to First Token — The time from sending a request until its first output token arrives.
Back to the text
Throughput — The amount of work processed or generated over time. Comparisons need the unit, such as tokens per second or requests per second.
Back to the text

Frequently asked questions

What should you check first in this article?: FlashAttention divides attention computation into tiles to reduce repeated writes and reads of intermediate data in memory. PagedAttention manages KV cache for multiple requests in fixed-size blocks, allocating and reclaiming what each needs. One is a computation kernel and the other is per-request memory management; they have different roles and can be supported together.
How much memory does it need?: When requests of different lengths arrive, each conversation keeps input and generated-token state in KV cache. PagedAttention manages the cache in fixed-size blocks, allocating what each request needs and reclaiming it when finished. This can reduce unused space caused by reserving large contiguous regions for each request.