Response speed and acceleration

Speculative decoding: drafting tokens and verifying them

I like the large model's answer, but the speed of writing is frustrating. Smaller models are faster, but you can't expect the same response. What if we had the small model create candidates first and then have the big model check them? Speculative decoding is a way to reduce the generation time by dividing this role.

The fast draft is not sent straight to the user

The large bone model repeats the calculation each time it creates the next token. Once the small draft model prepares follow-up candidates, the model can be checked across multiple positions. Sections with well-matched candidates are accepted all at once to reduce the number of sequential calculations.

Incorrect candidates will not be included in the answer. We discard the ones that were out of line in verification and continue with the results of this model. Speculative decoding using exact verification is a way to preserve the output of the model, as opposed to the ability to shuffle answers from arbitrarily small models for speed.

Illustration where a small draft model creates a batch of candidate tokens and a large bone model verifies them all at once.
Speculative decoding allows the raw model to verify the proposals of a fast draft model, producing the same raw model's results with fewer iterations.

Start with a draft that is cheap to compute

In addition to writing a separate small model, you can use Prompt Lookup to find repeated phrases in the MTP head or context of this model. What they have in common is that they divide the work of creating candidates into the work of verifying the model. There is also a method without additional models.

Either way, the draft should be light enough. If it takes as much time as this model to create a good draft, the savings are small. The separate model approach also requires checking that additional weights and caches fit into memory.

An illustration of several candidate tokens being spread out like tree branches and only one path left for verification.
Implementations using tree-type candidates also ultimately adopt only the continuous paths allowed by the model and discard the rest.

Why can it be slow even with a high acceptance rate?

Even if many candidates are approved, the time it takes to create and verify candidates may be longer. The approval rate is an important clue, but the ultimate goal is to get the same answers done faster. You should look at the overall time taken, along with the average acknowledgment length and memory growth.

Especially with short answers, the output may end before the cost of preparation is recouped. In code with a lot of repetition, a setting that works well may have less effect when asking questions that write a new story. This is why we do not use the results of one task as a multiplier for all conversations.

A scale comparing the additional costs of creating and verifying drafts and the benefits gained from approval tokens.
It doesn't necessarily mean you'll be faster on every prompt, as light drafts and high candidate acceptance rates offset the extra work.

Before I get another small model

Check the main model/draft model combination recommended by the executor. The method of dividing tokens and the model structure must match, and it does not end with simply connecting any small model. First, I will leave this model's normal response as a standard.

After turning on acceleration, check not only the speed, but also the iteration output, format of tool calls and whether answers are complete. Implementation and setup issues may result in differences between theoretical objectives and actual results. If it's an important task, it's a good idea to check it by including questions for which you know the correct answer.

Where to use the additional memory

You can choose whether to spend the remaining memory on a draft model, leave it in a longer context, or increase the precision of the original model. The answer to which choice is better depends on the task. If the content is already good enough and the generation is slow, there is reason to experiment with speculative decoding.

Conversely, in situations where the model barely fits, stabilizing the basic execution comes first. Just because there are many acceleration features doesn't mean you have to turn them all on. If my question clearly reduces the wait, you can leave that setting alone.