Executable programs and extensions
FlashAttention vs PagedAttention: what each one changes
You can see FlashAttention and PagedAttention in the settings list. If both are fast, can I just turn on one? Although they have similar names, one deals with data movement during computation and the other deals with memory placement for interactions. When you know which problem it solves, the options become much less confusing.
Cost of moving intermediate data from long input
Attention calculates how each part of the input refers to other parts. In long contexts, the cost of writing intermediate results to memory and reading them back can also be high. FlashAttention is a way to reduce these memory movements by grouping calculations and organizing their order.
This is different from SpecPrefill, which picks out important sentences and discards the rest. This is a path that attempts to perform the same attention calculation more efficiently. Therefore, it should not be considered a function such as input screening or quantization just because the name implies fast.
The problem of reducing empty space in multiple conversations
The server receives requests of different lengths. If you reserve a large contiguous space for each conversation in advance, you may end up with unused space or gaps that are difficult to place. PagedAttention is a method of dividing the KV cache into blocks and managing them as needed.
It's a similar idea to page management in an operating system, but it doesn't automatically make GPU memory infinite or use SSDs like VRAM. It is a function to utilize the remaining space more efficiently and is not a function to reduce the model weight itself.
One does not replace the other
Because the computational kernel and cache management solve different problems, they can be utilized together in one runtime. Which kernel to use for prefill and how to manage the cache during creation is a separate topic to consider. Comparisons that pick a winner between the two serve no purpose.
Support depends on GPU, model structure, quantization, and app version. Some paths may be automatically selected, so check the current log before forcing them. The actual execution path selected is more important than the fact that the name is in the menu.
It's different when you ask briefly on your own and on the server.
For a single short conversation, there may not be much space to save by managing the page. With multiple requests and long contexts, the value of holding more state in the same memory increases. The total throughput improvement of the server cannot be transferred to the speed multiplier of private chat.
FlashAttention also does not guarantee the same effect in long prefill and short decode. You can tell which areas have improved by dividing the first token time, peak memory, and throughput per concurrent request.
Before adding more options, I leave one result:
Save the results of your usual input with default settings that run normally. Then try changing the supported paths to see if speed and memory improve. If an execution fails or you get strange output, you should be able to go back to baseline.
When buying equipment, look at the results found on the model you want rather than the number of feature names. The goal is not to memorize all the new terms. If you can tell whether you're waiting on a long document or running out of memory for multiple conversations, that's a good starting point for finding the settings you need.