Response speed and acceleration
Last updated
Long prompts can reverse a local LLM speed ranking
For long documents, generation and completion rankings can differ
A language model takes text, processes its content, and then generates an answer. A device with twice the decode1 rate can still finish a summary later if it waits longer before the first token2 on a long document. Using a 16K meeting record and 300-token answer, this guide adds prefill3 wait and generation time to compare completion.
When summarizing meeting notes in a few paragraphs, the user's wait lasts until the final sentence appears. It includes both the wait before the first token while the model reads the document and the time to generate the summary. Tokens per second, commonly shown in device tables, describes generation after the answer begins—not total completion time.
For a short question, the faster decoder finishes first
Assume the same model produces an answer of about 300 tokens. Hypothetical device A takes 4 seconds to first token and then generates at 60 tok/s; B takes 6 seconds and generates at 30 tok/s. These are assumptions for illustrating the calculation, not product measurements.
Generation alone takes about 5 seconds for A (300÷60) and 10 seconds for B (300÷30). Adding first-token wait gives about 9 seconds for A and 16 for B, so A finishes first on this short task. Even with twice the generation speed, total completion is not exactly half.
This illustrative calculation assumes a constant token rate and excludes model loading and queueing. First-token wait is added separately from decode speed.
For a long input, prefill wait can reverse the ranking
Keep the same model and 300-token answer, but use a 16K meeting record. In this hypothetical example, time to first token4 is 40 seconds for A and 10 seconds for B; generation still takes 5 seconds for A and 10 for B. Total time is about 45 seconds for A and 20 for B, so B finishes first despite A's twice-faster generation.
Prefill is the stage in which the model processes a long input. TTFT covers the whole wait for the first token, including prefill, queuing, and execution preparation. The calculation changes only TTFT to 40 and 10 seconds to show how total completion time changes with a longer input.
The hardware did not change; the input length of the compared task did. A can lead on a short question while B finishes a long meeting record first.
| Workload | Illustrative device A · 60 tok/s | Illustrative device B · 30 tok/s |
|---|---|---|
| Short input · about 300 output tokens | 4 s + 5 s = about 9 s | 6 s + 10 s = about 16 s |
| 16K input · about 300 output tokens | 40 s + 5 s = about 45 s | 10 s + 10 s = about 20 s |
Match answer length and cache state too
If you only ask for dates and owners, the short answer makes first-token wait a large share of total time. A long draft extends the generation stage, so decode differences continue to matter. Comparison prompts5 should therefore match both the source document and expected answer length.
Separate a meeting record opened for the first time from one already processed. A repeated request with cached input can have a different processing profile from the first request. Keeping a process running and reusing an input cache are also distinct conditions, so record them separately.
Compare devices by completion time for your workload
In the comparison view, select both 4K and a 16K input closer to your usual documents while keeping the model fixed. Review estimated end-to-end flow including prefill, then compare with generation-only to see which wait causes the difference. Account for different recommended formats or acceleration settings across devices.
Before buying, if possible, run the model file and runtime6 you plan to use with your meeting-note input and answer length. Record first-token time, completion, and generated token count separately to distinguish estimates from measurements.
Even while A remains at 60 tok/s and B at 30 tok/s, a longer first-token wait at 16K can change the end-to-end ranking. If A leads on your real 16K task and preserves the decisions you need, that may justify a purchase. If your current device finishes on time with adequate quality, a higher ranking alone may not justify the expense.
Terminology notes
Token decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the textToken — A unit into which a model divides input or output for processing.
Back to the textPrefill — The stage where an LLM reads the input prompt and computes representations for its tokens. Longer prompts contain more tokens to process.
Back to the textTime to First Token — The time from sending a request until its first output token arrives.
Back to the textPrompt — The input text sent to a model. For generative models it describes the desired result and strongly shapes the output.
Back to the textRuntime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.
Back to the text
Frequently asked questions
- What should you check first in this article?
- A language model takes text, processes its content, and then generates an answer. A device with twice the decode rate can still finish a summary later if it waits longer before the first token on a long document. Using a 16K meeting record and 300-token answer, this guide adds prefill wait and generation time to compare completion.