read first

Prefill, TTFT and decode: three different measures of speed

It took a long time for the answers to come out, but once they started, the posts started pouring in quickly. Conversely, there are times when you start answering right away and then write slowly, line by line. Both have different speeds, but the waiting section is different. These two are called prefill and decode.

What is the model doing before the answer starts?

When you first run a local model, this waiting may seem like an error. I sent a question, but there are no words on the screen. Before we ask the same question again, let's first distinguish what happens before an answer comes out.

Let's say you put in a PDF and asked for a summary. The model cannot be written straight from the summary. Questions and documents must be addressed first. This input processing step is called prefill.

The longer the documents to be processed, the greater the burden of this step. So, equipment that quickly answers short questions may seem to stop for a while when facing long meeting minutes. Just because you don't see an answer yet doesn't mean the model is resting.

Time flow diagram of prefill, which reads long input all at once, and decode, which writes the answer after the first token.
The waiting time for the first token and the speed at which the answer follows the first token are determined in different intervals.

Does a 10-second wait for the first token mean 10 seconds of prefill?

Not necessarily. The time from sending a request until the first token arrives on the screen is called TTFT. In addition to prefilling, it can also include waiting for requests, turning input into tokens, and preparing for execution.

For the user, this entire time is the initial waiting time. On the other hand, the prefill time reported by the program may only measure the input processing section. If the numbers on different screens are different, check first what you are counting from.

A side-by-side comparison of the length of long input and short output operations, and short input and long output operations.
Document summaries involve a large proportion of input processing, while writing and coding involve a large proportion of output generation, so the indicators to look at are different.

The answer has now begun

Decoding is the step of creating an answer after the first token. The decode tok/s in the performance table indicates how many tokens are created per second. Tokens are not exactly the same unit as letters, but when comparing the same model, the higher the number, the faster the text continues.

You will feel this difference for a long time when writing long articles or printing a lot of code. Acceleration such as MTP is mainly used to reduce this section. You'll have to check depending on your model and launcher to see if you can turn it on and if it actually makes it faster.

Comparison illustration of a request that calculates the input from scratch and a request that reuses the stored cache to quickly issue the first token.
TTFT comparisons require matching cache states because matching caches can reduce re-reading common input.

Why does it go faster when I ask for the same document again?

I waited a long time at first, but when I asked again using the same document, I was able to get an answer quickly. It is possible that the prompt cache was at work, rewriting the results of previous input processing when the preceding part was the same.

It's not fair to compare these results to other devices that ran for the first time without cache. You must decide whether to compare tasks that are read for the first time or tasks that ask for the same document repeatedly, and then match both conditions.

There is also SpecPrefill, which attempts to reduce throughput by selecting some tokens from long input, but in a different way than a cache. The two features should not be considered the same just because the first answer is faster.

So what numbers should we look at?

If you're getting a short summary of a long document, look at the time to the first token first. Decode speed is also important if you are receiving long answers with short instructions. Either way, looking at the total time it takes to complete a task makes it easier to judge.

After looking at the results including prefill in the comparison screen, look at the token generation speed separately. Depending on whether the initial wait is long or the response speed is slow, the required equipment performance will also vary.

Think about the frustrating moments you had before looking for fast gear. Was it when I put in the document and nothing came out, or was it when I waited for a long time until the end even after the response started? Being able to tell the difference can help you find the performance you need instead of just a vague “better computer.”