Executable programs and extensions

Continuous batching: throughput is not single-user speed

They say the server creates hundreds of tokens per second, but my response comes out much slower than that. Could this be a false record? The throughput can be the combined tokens of multiple people. To understand continuous batching, you need to break down what the entire server did and what each person waited for.

Use the space from the request that ended earlier

If you batch short and long answers together, the shorter ones will finish first. Fixed batches may not consume enough resources while waiting for remaining requests. Continuous Batching continues execution by subtracting completed requests and adding pending requests.

The goal is to reduce GPU idle time on servers where new requests are constantly coming in. It's not a switch that makes things go faster when you send a single question. This is a feature to consider first in environments where you have work to do at the same time.

An illustration of the scheduler continuously inserting requests that arrive at different times into empty slots in the running batch.
Continuous batching fills empty slots with new requests immediately, rather than waiting for all requests to arrive.

All 100 tokens and my 100 tokens are different

As a hypothetical example, if ten requests each receive 10 tokens per second, the total is 100 tokens per second. It's not the same as a situation where one person receives an answer at 100 tokens per second. Real servers may have different speeds for each request, but the principles that differentiate the two metrics are the same.

If you're a batch server that completes a lot of work, total throughput is important. If it is a tool for communicating with people, the time until the first token and the interval between tokens are also important. If chatting becomes frustrating even if the total tok/s increases, you should reconsider whether the improvement is suitable for your purpose.

Illustration of a congested throughput path that efficiently handles many requests versus a delay-first path that takes one request directly.
Even as the overall tok/s increases, individual requests may take turns, increasing the latency between the first and next token.

Shared weights, separate caches

KV state is required per conversation even if multiple requests use one model weight. Simultaneous requests with many long inputs can exhaust memory quickly. Setting maximum number of requests and maximum context length respectively is not a good starting point.

Paged cache management and common prefix reuse can help leverage slack. However, it does not create memory that does not exist. You should watch peak usage and queues and adjust limits to match actual requests.

Illustration of a completed sequence being pulled out of an execution slot and a new pending request immediately taking its place.
Reusing slots along the completion of requests of different lengths can reduce the empty compute resources that result from fixed batches.

If you copy one person's optimal settings,

The MTP or draft model creates candidates, verifies them, and updates the cache. You need to make sure that an implementation that works well for a single request supports the same path for multiple requests of different lengths. Even if it is turned on in the settings window, it may run in a different path in batch.

If you load chat and background document processing together on the same server, test for situations where the two tasks compete. The top speed measured alone cannot describe the experience when used together with family or colleagues.

See the moment users waited the longest

For server evaluation, we need to consider not only averages but also requests with high delays. Tools that are fast most of the time but sometimes pause for a long time can be inconvenient to use in conversation. You need to record the input length, output length, and number of concurrent requests to be able to recreate that moment.

If you are using a local AI alone, you can start with a small number of concurrent requests. Increase the limits and see the results when multiple agents actually work together. Rather than making the GPU as busy as possible, the role of the server is to ensure that users receive the answers they need in a timely manner.