Response speed and acceleration

Prompt Lookup and n-gram speculative decoding explained

I asked them to change just one function in the code, but they reprinted even the parts that didn't change. Isn't it possible to continue writing sentences that are already in the input more quickly? Prompt Lookup uses that repetition as a candidate. This is a different function from the RAM/SSD offload of the model table with a similar name.

Find the next part of an existing sentence

Look for patterns in the input or previous output, such as the few recently generated tokens. If there is a match, the token that followed is taken as a candidate. This model groups these candidates, verifies them, and uses only the matching parts.

At this time, a continuous bundle of tokens is called an n-gram. Since we are not running a separate language model to create a draft, no additional weights are needed. However, this does not mean that there is no work space and time required for inquiry and candidate verification.

Illustration of an n-gram lookup that finds the same set of tokens in the recent input and proposes a copy of the next candidate.
n-gram lookup finds repeated token bundles in the input and recent output and uses them as candidates without a separate draft model.

Consider it for tasks that reuse much of the source text

Modifying only a portion of a long code and reprinting it or reformatting an existing document may leave behind many original sentences. Since the next sentence is already in the input, we also have a chance to find a candidate.

Conversely, questions that create completely new explanations or stories may have fewer match sections. In this case, a search is conducted, but there are no candidates to use, or they are often discarded during verification. It is difficult to say that it is a common acceleration that makes all articles equally fast.

An illustration of finding repeated codes and tabular patterns in the input window and connecting them to the next section.
In tasks with a lot of repetition, such as code, tables, and forms, it is easy to find long matching sections, so even a light search can be beneficial.

Similar text is not simply copied into the answer

The model must check whether even sentences in the original text fit the current answer. If one variable or number in the code has changed, candidates beyond that point cannot be accepted as is. Using repetition does not mean reusing the original text without reviewing it.

If n is set small, it is easy to find a match, but it may be in the wrong context, and if n is set large, it may be difficult to find an accurate pattern. Don't blindly increase the candidate length, but start from the default and compare the average approval length and overall time.

Illustration comparing the method of using only the query window without an additional model and the method of loading a separate draft model into memory.
Prompt lookup has a small memory burden, but has the limitation that there are few candidates to suggest in creative sentences without repetition.

How is it different from n-gram moving to SSD?

PLE offloading, which places the large embedding tables of some models in RAM or SSD, is a technique that changes the storage location of the weights. Prompt Lookup, explained here, is a technology that creates the next candidate by looking up sentence records. Even if n-gram appears in the name, the problem it solves is different.

Turning on Prompt Lookup does not move the model's large weights off the GPU. So if your model won't load due to insufficient memory, this option is not a solution. In the off-road guide, you must separately check the location of the model and its execution path.

This is a feature to test without any additional purchase.

Prepare code you usually modify and free questions, and compare the on and off states. Rather than leaving only one best result for an example with many repetitions, check to see if there is a lot of loss in tasks where candidates are not common.

If it only helps with code modification, you can choose to use it only for that task. Rather than constantly changing equipment or models to find a feature that answers all your questions, reducing the wait for one thing you do frequently can be enough of an improvement.