Executable programs and extensions
Last updated
How to read Strata speed benchmarks: short chats and long prompts
Prompt-reading speed and answer-generation speed measure different work.
Strata accepts a prompt1 and conversation history, then generates an answer with Qwen3.8-Flash-Next. Prompt throughput2 measures input processing; output tok/s measures answer generation. The RTX 5070 Q2_0 figures of 94, 76, and 2,650 tok/s are separate engine 0.1.36 measurements for a short chat, a 128K context, and a 32K prompt. AMD results come from a different PC, so they are not a direct GPU3 ranking; 0.1.43 figures also require their hardware and setup conditions.
Strata speed tables separate the stage that reads a prompt from the stage that generates an answer. Distinguishing the wait for the first character after sending long code or a document from the rate at which the answer appears helps identify the bottleneck in your work.
This guide includes older PC-level measurements and changes reported in 0.1.43 on October 11. Check the model, quantization4, GPU, runtime5, and prompt length listed for each table, then compare before/after results with matching conditions.
Separate prompt processing from answer generation
Prompt throughput is the rate at which the model reads tokens6 in the question and conversation history. Output speed is the rate at which it generates the answer after the first token. Both use tok/s: prompt throughput affects the wait for a response, while output speed affects how quickly the answer continues.
TTFT7 (time to first token) is the time from sending a request until the first output token arrives. It can include app-side request preparation and server queueing as well as model prompt processing. If you want to know how long a long prompt delays the first answer, record request time and first-token time in the actual app instead of relying only on throughput.
Compare the RTX 5070 and RX 9070 XT by PC
The figures below are historical published Q2_0 measurements from an RTX 5070 and RX 9070 XT. They place three different workloads side by side: a short answer, answer generation with a 128K context, and processing a 32K prompt. Even with the same unit, each row represents a different input and generation task.
| Workload | RTX 5070 PC | RX 9070 XT PC |
|---|---|---|
| Short-chat answer | 94 tok/s | 60 tok/s |
| Answer with 128K context | 76 tok/s | 48 tok/s |
| 32K prompt processing | 2,650 tok/s | 1,160 tok/s |
The NVIDIA PC used an RTX 5070 12 GB, Ryzen 5 7600, 64 GB DDR5-5200 RAM8, Windows, and Strata 0.1.36. The AMD PC used an RX 9070 XT 16 GB, Ryzen 9 3900X, 47 GB RAM, and Linux. Q2_0 on the AMD system exceeded setup's RAM estimate and was installed with --model Q2_0 --yes. These are results from separate PCs, not a controlled test that changed only the GPU.
A long-prompt calculation covers only part of first-token time
Dividing 32,000 tokens by the RTX 5070's 2,650 tok/s prompt throughput gives about 12.1 seconds for input processing. Using the RX 9070 XT's 1,160 tok/s gives about 27.6 seconds. The calculation is prompt-token count divided by throughput, an estimate of the model's input-reading stage.
The wait for the first token also includes the app's request preparation and server queue. Reusing a prompt cache reduces the input that the model must read again. Send the same prompt through the app and record the request time and first-token arrival to measure TTFT including these stages.
Record answer length and cache state too
Output tok/s measures ongoing answer generation, so a 400-token answer takes longer than a 40-token answer at the same rate. To compare completion time, record the first-token wait, generated-token count, and time of the final token. Speculative decoding9 can also change output speed because the number of accepted draft tokens varies by answer.
Separate a prompt sent in a new conversation from one sent as a follow-up in the same conversation. If the prompt cache reuses earlier context, the amount of input the model rereads can change. Record the Strata version, model and quantization, prompt and output lengths, GPU and CPU10, context, whether it was a first request or follow-up, and cache state.
Default Windows RTX 5070 decode11 in 0.1.43 matches 0.1.42. Linux NVIDIA enables adaptive overlap and automatic PCIe sharing by default. Results for the AMD R9700 and Arc Pro B70 are listed by model, cache, and context. Find the row matching your setup in the relevant GPU guide, then compare the same workload before and after updating.
Terminology notes
Prompt — The input text sent to a model. For generative models it describes the desired result and strongly shapes the output.
Back to the textThroughput — The amount of work processed or generated over time. Comparisons need the unit, such as tokens per second or requests per second.
Back to the textGPU — A processor designed to handle many calculations in parallel. It performs model computations during AI inference.
Back to the textQuantization — A way to represent model weights or values with fewer bits, reducing storage and memory use. The coarser representation can also affect accuracy.
Back to the textRuntime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.
Back to the textToken — A unit into which a model divides input or output for processing.
Back to the textTime to First Token — The time from sending a request until its first output token arrives.
Back to the textSystem RAM — System memory that temporarily holds data while programs run.
Back to the textSpeculative decoding — A generation method in which a draft model, MTP head, or repeated-context lookup proposes output candidates for the main model to verify and accept.
Back to the textCPU — The central processor that runs the operating system and general-purpose instructions; in AI workloads it also handles tasks such as preprocessing and data movement.
Back to the textToken decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the text
Frequently asked questions
- What should you check first in this article?
- Strata accepts a prompt and conversation history, then generates an answer with Qwen3.8-Flash-Next. Prompt throughput measures input processing; output tok/s measures answer generation. The RTX 5070 Q2_0 figures of 94, 76, and 2,650 tok/s are separate engine 0.1.36 measurements for a short chat, a 128K context, and a 32K prompt. AMD results come from a different PC, so they are not a direct GPU ranking; 0.1.43 figures also require their hardware and setup conditions.