Response speed and acceleration
Last updated
Prompt Lookup and n-gram speculative decoding explained
Use repeated source text as next-token candidates
Prompt Lookup1 finds repeated token2 sequences in the input and previous output, then uses their continuations as candidates. The main model verifies them and keeps matching tokens. It can reduce generation time when code edits or document rewrites preserve much of the source.
When renaming a function and reprinting its code, much of the answer already exists in the input. Prompt Lookup finds these repeated spans and proposes several tokens together. You can try it without downloading a separate draft model.
Find continuations of earlier tokens
An n-gram3 is a consecutive token sequence. If the input says the contract runs through March 31, 2027, and the answer repeats that sentence, tokens after a matching sequence can become candidates. It uses the input and generated history available to the model, rather than searching an entire contract separately.
Lookup creates candidates, then the model evaluates them. It outputs the consecutive span accepted for the current context and resumes generation at the first mismatch. Requests with long repeated spans make useful candidates easier to find.
If a document app sends only retrieved clauses, lookup can use only that material. Other pages must first be retrieved by the app. Prompt Lookup optimizes generation by reusing delivered text; the app still handles missing retrieval results.
It suits tasks that retain the source
Start with renaming a function while retaining its body, rewriting a table in the same format, or quoting a contract clause. Matching input and output tokens may produce longer accepted candidates. Include the source code or relevant clauses in the request.
New explanations or interpretations across clauses require new wording. There may be fewer reusable candidates and smaller gains. Compare repetitive rewrites separately from new interpretations to judge usefulness for your work.
Check changed code and tests
For example, rename get_renewal_date to find_renewal_date. A candidate with the old name can be rejected when the main model selects the new name. Unchanged input checks and date-parsing code afterward may still be reusable.
Acceptance logs explain the token-generation path. Use a diff and tests to check whether the code meets the request. Look for callers still using the old name or unintended changes to input validation.
Start with default n-gram and candidate lengths. Short sequences match often but may find different contexts; long sequences are more specific but tolerate fewer wording changes. Record a baseline, then change one setting at a time.
Separate it from PLE offloading
The term n-gram also appears in descriptions of Flash-Next's large PLE4 table. Choose by what the feature changes: Prompt Lookup finds output candidates, while PLE offloading5 changes where a model table is stored.
| Feature | Target | Main purpose |
|---|---|---|
| Prompt Lookup | Tokens in input and output history | Reduce repetitive generation time |
| PLE offloading | Model embedding table | Use RAM/SSD to reduce VRAM residency |
Compare on the same task
Find Prompt Lookup in your runtime6's supported features and start with a model that already works. Prepare function-renaming, clause-quoting, and new-explanation requests. Record enabled and disabled runs with identical inputs, keeping model-loading time separate.
After warm-up, repeat under the same conditions and compare medians. Record time to first token7, subsequent generation time, and output tokens to distinguish gains caused by shorter answers. Check code tests and missing quoted clauses as well.
Use it for repetitive tasks that get faster while retaining correct results. If memory prevents loading the model, first assess quantization8 or supported offloading. Resolve residency problems before comparing generation optimizations.
Terminology notes
N-gram lookup speculation — A method that finds repeated token sequences in the prompt or generation history and proposes them as next-token candidates. Context lookup supplies candidates without a separate draft model.
Back to the textToken — A unit into which a model divides input or output for processing.
Back to the textN-gram — A sequence of n consecutive tokens or items. N-grams may be used for context lookup or speculative generation, and may also form the entries of embedding tables in some models.
Back to the textPLE embedding table — Short for Per-Layer Embedding, a structure that adds embedding information at particular model layers. The Qwen Flash-Next family uses n-gram tables; supported runtimes can offload these tables to RAM or SSD.
Back to the textOffloading — Moving some model data from GPU memory to system RAM or storage when capacity is limited. This adds data transfer.
Back to the textRuntime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.
Back to the textTime to First Token — The time from sending a request until its first output token arrives.
Back to the textQuantization — A way to represent model weights or values with fewer bits, reducing storage and memory use. The coarser representation can also affect accuracy.
Back to the text
Frequently asked questions
- What should you check first in this article?
- Prompt Lookup finds repeated token sequences in the input and previous output, then uses their continuations as candidates. The main model verifies them and keeps matching tokens. It can reduce generation time when code edits or document rewrites preserve much of the source.
- How much memory does it need?
- Use it for repetitive tasks that get faster while retaining correct results. If memory prevents loading the model, first assess quantization or supported offloading. Resolve residency problems before comparing generation optimizations.