Response speed and acceleration
Last updated
MTP: how multi-token prediction works and when to use it
MTP proposes future tokens and the main model verifies the matching span
MTP1 prepares candidates for upcoming answer tokens2 and verifies multiple tokens together. Additional prediction heads propose candidates, and the main model incorporates the accepted consecutive sequence into the answer. Its speed depends on how well candidates match, so this guide compares it with standard generation on the same code modification request.
Suppose you request a code change that renames a function and handles empty input. Standard generation chooses a next token, then computes another token using that result. MTP prediction heads prepare likely future candidates so the main model can verify several positions together. Proposing three candidates does not mean all three appear in the answer.
The main model accepts a matching consecutive span of candidates
For example, if only the first of three candidates matches, it can be retained while the next two are discarded. The model continues from the mismatch instead of outputting rejected candidates, but proposing and checking them still used computation. If all three match in sequence, the main model may reduce some sequential steps.
A deeper proposal is therefore not always faster. Frequent mismatches increase discarded work, while frequently accepted consecutive candidates can reduce sequential computation. Compare completion time on the actual request rather than the number of proposed tokens.
- Prefill vs decode: should you look at TTFT or tok/s?
- GGUF vs MLX: which model format should you download?
- What is Lightning MTP? Faster answers with oMLX 0.7.1rc1
- Qwen3.8-Flash-Next PLE Offload: RAM and SSD Requirements and Setup
- Husky on Mac: Woof, MLX and the 730 tok/s result
- Gemma 4 26B-A4B MTP: How Speed Changes Between One and Many Requests
Check model files and runtime support before enabling a setting
The checkpoint3 must retain MTP weights, and the runtime4 must support that structure. Conversion can remove prediction weights even from a model with the same name. Fix the file and quantization5 and check runtime version and supported paths so you can connect result changes to whether MTP was applied.
If enabling MTP changes nothing, inspect logs to distinguish a fallback to standard generation because of an incompatible checkpoint or runtime from a working MTP path with few accepted candidates. Keep a known-working model as a baseline while checking support, and avoid changing several files at once.
Compare before and after MTP on the same code edit
First establish a baseline answer with MTP off using the same file, question, and request length. Then change only the MTP setting on the same model file. After warm-up, measure each state three times and compare medians; record model reload cost separately. Match conversation-cache state and other running apps across both tests.
First determine whether input processing or answer generation is slow. MTP primarily targets the generation stage. If the first answer is delayed after PDF retrieval or a long input, measure prefill6 and document delivery separately too.
Separate time to first token7 (TTFT), decode8 after that, and total completion. Calculate decode tok/s by dividing generated tokens after the first token by decode time; do not include loading or TTFT. Also check whether both answers contain a similar edit scope and token count.
Check the function name and call sites, empty-input error, and normal-input handling in the resulting code. Candidate-token verification does not test user requirements; if the edit is incomplete, reduced time is not faster completion of the same task.
When there is no speed benefit, adjust the failing stage
If the path is active but decode time is unchanged, no benefit was observed for this request. For a delayed first answer after a long PDF, inspect prefill and retrieval; for memory errors, check input length and cache first.
If repeated tests show MTP reduces actual completion time and the code meets requirements, you may be able to keep using the current device. If MTP is unsupported or results are similar, do not base a purchase on advertised peak acceleration; first address the other stage interrupting your work.
Terminology notes
MTP — A training objective or model component for predicting multiple future tokens at once. In speculative decoding, those predictions can be proposed as candidate tokens.
Back to the textToken — A unit into which a model divides input or output for processing.
Back to the textCheckpoint — A file containing saved model weights and related state. Versions or tasks in one model family may use different checkpoints.
Back to the textRuntime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.
Back to the textQuantization — A way to represent model weights or values with fewer bits, reducing storage and memory use. The coarser representation can also affect accuracy.
Back to the textPrefill — The stage where an LLM reads the input prompt and computes representations for its tokens. Longer prompts contain more tokens to process.
Back to the textTime to First Token — The time from sending a request until its first output token arrives.
Back to the textToken decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the text
Frequently asked questions
- What should you check first in this article?
- MTP prepares candidates for upcoming answer tokens and verifies multiple tokens together. Additional prediction heads propose candidates, and the main model incorporates the accepted consecutive sequence into the answer. Its speed depends on how well candidates match, so this guide compares it with standard generation on the same code modification request.