Response speed and acceleration

MTP: how multi-token prediction works and when to use it

I see a story saying that it is the same equipment and model, but when MTP is turned on, it becomes faster. It's nice to be able to reduce waiting without having to change equipment. But why is the difference small for some questions? Let's start by looking at how many of the pre-created tokens are left over in the actual answer.

Instead of waiting for the next token

A typical generation looks at the sentences so far and creates the next token. Only when that token is determined can you create the next one. MTP leverages an additional prediction part that learns to predict multiple future tokens. The idea is to first prepare candidates for the next few tokens and have the model group them and check them.

At this time, the candidate is not immediately labeled as the correct answer. We accept the parts that have passed the verification from the beginning, and continue with the results of this model at the points where there is a discrepancy. This is not a technique aimed at lowering the quality of answers by shuffling small drafts.

Comparison of the basic method of generating tokens one by one and the accelerated method of proposing several candidates and then using only approved tokens. — The above is the basic method of creating one token at a time, and the below is a method that proposes several candidates and accepts only the correct sections.

Proposing three tokens does not mean keeping all three

Even if you create multiple candidates, if you make a mistake in the first part, you cannot use the second part. Generating and verifying candidates takes time, so if there are a lot of discarded parts, the benefit is reduced. Conversely, if multiple tokens are accepted in succession, the number of times the model repeats one token at a time can be reduced.

So setting the largest length to predict at once is not always optimal. You should look at not only the percentage of tokens accepted, but also how many tokens were left at a time on average and how many seconds the process took. Even for the same question, the balance changes depending on the setting.

An illustration of several future token prediction branches branching out from the main model and then merging back together with verification results. — The additional prediction part of the MTP pre-generates the second and third subsequent token candidates and prepares the bundle for verification.

Check the model file before the button

The model file must have weights for MTP and the executable must support that structure. If additional weights are missing during the conversion process, the general answer may not be MTP. You need to distinguish between what appears to be turned on on the settings screen and what actually runs that route.

Additional memory may also be required. If the model body can barely fit, you may run out of space after turning on MTP and long context together. This feature is not a replacement for insufficient memory, but rather an option to change the creation process in supported environments.

An illustration in which only depths that have been approved in succession from the beginning among several proposed tokens are included in the answer. — The more candidates are approved in a row, the further you go in one verification, and the greater the acceleration gain.

Compare MTP on and off with your usual questions

First, check if the same model answers normally without MTP. After the warmup run, sending the same input three times and comparing the medians can reduce the impact of a single good result by chance. If you're doing both code editing and free writing, it's better to check each one separately.

In addition to speed, check whether the answer was completed normally and whether the tool call format or important figures are abnormal. Even if token generation is fast, it is not a setting to maintain if repetitive output occurs or execution fails frequently. If the effect is small, turning it off is also a normal tuning result.

Before buying equipment

A good combination of MTPs can make the same equipment more useful for a longer period of time. However, do not calculate the highest multiplier for a particular question as the baseline performance for all models. The results confirmed from the model and app to be used now come first.

If it's already fast enough on my machine, there's less reason to look at something more expensive. Conversely, if the wait for reading a long document remains the same even when MTP is turned on, the problem may be prefill. Rather than saying it's faster, you need to know which wait has been reduced to make your next choice more accurately.