Qwen3.8-27B settings by hardware: oMLX on Mac and llama.cpp on RTX

Qwen3.8 27B · 4,096 input tokens¹ · Estimated for one user. These are the same calculated results as the speed experience, not a measured run.

Time to first token² is the wait before the answer starts. Token generation speed describes how quickly the rest follows. For long documents, consider both.

Quantization³: Q4_K_M · Acceleration: none · Token generation: 5.1 ~ 5.7 tok/s · First token: 16.4 to 35.5 seconds

Prompt processing: 116 ~ 252 tok/s · Memory required: 17.6GB · Available memory: 27GB

Quantization: Q4_K_M · Acceleration: none · Token generation: 12.3 ~ 13.7 tok/s · First token: 7.8 to 14.3 seconds

Prompt processing: 289 ~ 530 tok/s · Memory required: 17.6GB · Available memory: 41GB

Quantization: Q4_K_M · Acceleration: none · Token generation: 38.3 ~ 51.9 tok/s · First token: 2.6 to 3.6 seconds

Prompt processing: 1,141 ~ 1,591 tok/s · Memory required: 17.6GB · Available memory: 22.5GB

Start by selecting the computer you actually plan to use. This guide separates the Mac path using oMLX from the RTX path using llama.cpp; the desktop starting point is a compatible Q4 GGUF⁴ or MLX⁵ conversion. Even with the same model name, different file formats and apps lead the selector to provide different commands. Match your device and format before copying someone else's settings.

The roughly 17 GB figure is an approximate guide when looking for a Q4 file, not the total memory needed at runtime⁶. After downloading it, check its exact size and whether enough room remains for the operating system, app, and cache during execution. A model file being present on disk is not the same as the model being loaded into memory and ready to answer.

The path that passes the official BF16⁷ model ID to a server is separate as well. A simple calculation for the 27B weights alone gives roughly 54 GB, with runtime workspace needed on top. The vLLM example below therefore assumes an NVIDIA environment capable of running the official server weights; it cannot be reused as a Q4 command for a 24 GB GPU⁸. Even under the same model name, the required environment changes with the distribution file and precision.

Establish a text-question baseline first; it will be easier to compare later when you add images or MTP⁹. If you change the Q4 file, vision input, and MTP all at once, it is hard to tell which of the three caused a failure. In this example, first reach a state where a text answer completes, then add only the extra features supported by the selected runtime.

Apply the command produced by the selector for Mac oMLX or RTX llama.cpp, and send one request at a time. First get a short coding question to complete at the indicated context. The goal is not a high score but to establish that the basic path works from model loading through the answer. Starting with a much larger context changes cache use and makes it harder to establish a baseline.

Once a normal answer appears, record that configuration. After a preparation run, note prefill¹⁰, time to first token, decode¹¹, and peak memory across three runs so you are less likely to judge from one unusually fast result. Record a repeat of the same input separately from the first request. An input cache may remain for the second run, so do not interpret the difference as a fixed speed difference between models or devices.

For example, if you include a function that fails on empty input along with its caller, note which files were included in the request and which conditions the answer identified. Before producing performance figures, this record helps confirm that both settings were given the same task. If someone else used a shorter question or files that were already open, you cannot directly apply their tok/s to the completion time for your work.

The vLLM command below is a different configuration from the desktop selector. It is an example for an NVIDIA server that can hold the official BF16 weights, so do not mix it with the Q4 commands for Mac or a 24 GB RTX GPU. A client sending requests to 127.0.0.1 does not prove that the server is inaccessible from external networks. Check the actual binding and firewall; prepare authentication and access controls before allowing other devices to connect.

For an API¹² test, use the model identifier actually shown in the model list. The port must also match the runtime you launched. This guide's commands use 8000 for oMLX and 8080 for llama.cpp; repeating the example curl against port 8000 will not help if no server is listening there. First check that the address responds, then verify that a short chat request completes.

Check whether the response arrives in streaming chunks, and if the app separates reasoning from the final answer, inspect how both stages appear. When the runtime is set to emit a long reasoning trace, the final sentence may appear late even if input processing is not the cause. Looking at the display from the first token through the final answer alongside server logs helps distinguish where the time was spent.

A name appearing in the model list confirms that the server knows an identifier; it does not verify that your app requested the intended model or that the context setting took effect. An incorrect name may cause a rejected request or route the answer elsewhere. Verify the actual model identifier and server logs in a short test response before moving to document input; this reduces confusion later between a slow answer caused by loading and one caused by targeting a different model.

Once the text question completes, send code or a document you actually plan to use. For example, include the function to change and nearby callers, then check whether the answer addresses the requested file scope. A high tok/s on a short greeting cannot stand in for the first-answer and completion times after a long code input, so use your own task as the comparison baseline.

If the first answer is late after sending code or a document, check the input-token count, prefill chunk, and cache state. If generation is slow after the answer has started, inspect memory use, swapping, and whether some weights have moved to the CPU¹³. The former may be an input-processing bottleneck and the latter a generation bottleneck; increasing MTP steps to fix both can miss the cause.

The oMLX 0.7.0 release includes Lightning MTP and DFlash2 paths for Qwen3.8-27B. They are different speculative-decoding¹⁴ methods, so select one at a time in the UI rather than enabling both, and check that the checkpoint¹⁵ and draft model match the chosen path. Comparing the baseline decode with one acceleration path on the same prompt shows what actually changed.

The Qwen3.8-27B oQ4e comparison in PR #3958 generated 512 tokens from a long code_python prompt on an M3 Ultra with an 80-core GPU and 512 GB of unified memory¹⁶. At temperature 1, top_p 0.95, and top_k 20, Lightning MTP reached 74.1, 64.9, and 49.6 tok/s at 8K, 16K, and 64K context; DFlash2 reached 79.4, 84.7, and 59.9 tok/s. ANE prefill was off, and each row compares the PR branch with main. These are PR validation results, not a fresh repeated test of stable oMLX 0.7.0 under the same conditions, so repeat the comparison on your Mac with the same model, input, and context before applying them.

PR #3958 does not list a memory-guard tier or KV-cache¹⁷ precision in its results table. In oMLX 0.7.0, balanced leaves about 8% of memory (3–8 GB) free for other apps, while aggressive leaves about 2% (1.5–4 GB). Start with balanced for everyday use; consider aggressive only after checking stability with other apps closed. Record KV-cache precision and MTP as separate conditions so you can tell which setting changed the result.

Also validate the code result, not just speed. For example, send the same function-change request with the default and MTP settings, then check whether the function was renamed as requested, the empty-input branch remains, and the existing return format is preserved. If a test fails, record the difference to see whether the faster answer actually completed the task. This checks the code requirements; it is not the same test as whether candidate-token verification works.

If memory runs short only after increasing the context, first reduce the request count and input length to find a range that answers reliably. Check whether the limit recurs with documents you use every day, and consider a larger-memory configuration only if needed. Rather than declaring the device inadequate after one failed long prompt, compare the text baseline with the actual workload to find what changed.

If you plan to use the model for both code changes and document summaries, keep one representative prompt for each. A prefill batch or cache setting that helps one task may change memory pressure or waiting time at another input length. Record more than answer speed: compare whether the code runs and whether important document conditions remain in the summary against a reference answer.

In the final record, include the model filename, app version, context length¹⁸, request count, and acceleration options. If an update changes execution, you can return to this state and identify which setting changed. First establish a setup that completes your usual task and returns the necessary information accurately. Once that baseline is stable, adjust one item and compare whether waiting actually decreased.

For example, if your usual questions complete reliably with the Q4 file but only long inputs fail, first check how the selector configured that context and prefill before buying new hardware. Conversely, if the model will not load even with a short input, inspect the file/runtime combination and available memory before cache precision or MTP steps. Locating the problem makes it easier to choose what to change.

Requires a compatible NVFP4¹⁹ kernel. Spark measurements and RTX estimates are kept separate.

vLLM / SGLang · Inferact/Qwen3.8-27B-NVFP4