What is Lightning MTP? Faster answers with oMLX 0.7.1rc1

When an app that asks questions about PDFs connects to the oMLX API³, oMLX runs the model and returns an answer. In standard generation, the model selects one next token⁴, adds it to the context, and calculates the next token again. Multi-Token Prediction (MTP) uses an auxiliary prediction head trained with the model to propose several next-token candidates, which the main model checks in one pass. The main model includes consecutive candidates it confirms in the answer, discards candidates from the first mismatch onward, and continues with standard generation.

Proposing and checking candidates also requires computation. Models and requests that often accept proposed tokens can reduce sequential calculations, while a lower acceptance rate can reduce the savings. Compare total completion time and the answer for the same question instead of judging by draft depth alone.

To choose a supported combination, check the version and architecture in the oMLX release model list. Review the file format in the checkpoint instructions and confirm that the converted file retains MTP-head tensors. Then check whether the model settings show an MTP option for that checkpoint.

For an initial comparison, keep a file that returns a normal answer with standard generation as your baseline, then add one supported MTP checkpoint. If the model does not load, check the file path, memory, and checkpoint format before changing MTP settings. If both standard and MTP versions of the same model are available, check for MTP tensors as well as file size.

The table gathers the file-editing scope and completion results reported in the development release published on October 9, 2026. It measures time to finish a request, so to see how it applies to your work, compare with matched model and input conditions as described below.

To apply the result to your work, match the model and checkpoint, input, output length, and temperature. After a warm-up, alternate the same file-edit request between standard generation and MTP. Record total completion time and whether the resulting code meets the requirement. For PDF questions or short chats, test that workload directly instead of using file-edit results.

The 0.7.1rc1 release also improves the combination of a TurboQuant 4-bit KV cache⁵ and Lightning MTP. Qwen3.8-27B on M3 Ultra generated faster after 16K and 32K inputs. If you already use this model, this gives you a specific long-context configuration to compare.

M3 Ultra · Qwen3.8-27B · TurboQuant 4-bit KV + Lightning MTP · maintainer measurements against the code immediately preceding each change · repetitions and full sampling settings are not specified in the release table

At 16K, the measured gain was about 29 generated tokens per second. For your own work, reuse one document and fix the response length, then measure time to first token⁶ and full completion separately. TurboQuant changes KV-cache representation while Lightning MTP changes candidate generation, so record both settings.

If you test MTP with the PDF question introduced at the beginning, use the same file, page, and question. That workload differs from the file-editing results in the previous section, so measure a task you actually do. Time from document loading to the first token includes prefill⁷ and request wait; the rate of subsequent tokens shows generation. Also record total time through the final sentence. MTP changes the generation path, so track input-reading and initial model-loading time separately.

Run a warm-up for standard generation and MTP, then alternate three trials each and compare the medians to reduce the effect of brief background activity. Keep input-token count, answer length, temperature, KV-cache setting, and number of concurrent requests the same. Check that the answer preserves the evidence in the question. For a server handling several requests, measure one request and concurrent requests separately to see how load changes the results.

If generation remains slow with MTP enabled, inspect the selected Lightning MTP path, draft depth and acceptance behavior in the logs. If it falls back to ordinary generation, check the architecture and cache combination. If Lightning MTP runs with low acceptance, compare draft depth and checkpoints one at a time on the same request.

The KV cache retains computed information about processed tokens for subsequent generation. Reuse of common input across separate requests depends on the server’s prefix-cache settings. When comparing MTP, record fresh conversations and follow-ups on the same document separately, matching input lengths and request order.

Concurrent requests change engine batching and memory use. To evaluate MTP on a server, first record a normal response and throughput⁸ baseline with one request. Then reproduce the actual concurrent-request pattern and compare errors and total throughput with the same inputs. Run this measurement with the model and checkpoint you use.

If MTP is consistently faster and answer quality holds, use that checkpoint with the corresponding oMLX version. If you change models often or the converted file lacks an MTP head, use standard generation as your baseline. Check the actual runtime⁹ path in the server log, and record before-and-after time and answer accuracy for the same request.