Model setup recipes

Qwen3.8-27B setup by hardware: Mac oMLX and RTX llama.cpp

Match the model format and request, then run the same Qwen3.8-27B workload on Mac and RTX

For a Q4 checkpoint¹, use the MLX² format with oMLX on an Apple Silicon Mac and GGUF³ with llama.cpp on an RTX. Confirm that each server answers the same request, then record first-token and completion times along with memory use. The 0.7.1.dev1 INT8 activation-prefill results apply separately to Qwen3.8-27B oQ8e on an M5 Max.

Qwen3.8-27B is a 27B dense language model that takes questions or code and returns text answers. First confirm that a short request works end to end, then adjust context length⁴ and generation options for your environment.

Contents

Requirements and key details

oMLX uses MLX checkpoints; llama.cpp uses GGUF.
Separate the first model load, which includes setup and caching, from warm repeated runs.
INT8 activation prefill applies to the prefill stage on a supported oQ8e checkpoint.
Change MTP and KV-cache precision one at a time after confirming a normal baseline response.

Jump to hardware-specific settings

Match the checkpoint and Mac or GPU memory first

oMLX on a Mac runs an MLX-converted checkpoint, while llama.cpp on an RTX reads GGUF. Files are not interchangeable just because they share the Qwen3.8-27B name. Use mlx-community/Qwen3.8-27B-4bit on the Mac and Q4_K_M from ggml-org/Qwen3.8-27B-GGUF on the RTX as baseline files. The Q4_K_M file is about 17.6 GB, but actual memory use also includes the KV cache⁵ and working buffers.

Start with a 4,096-token context on an RTX with 24 GB, and leave memory for a browser or editor you keep open. On a Mac, check total unified memory⁶ and memory pressure. Use the same request on both systems to compare their runs.

Green and yellow paths branch from document blocks toward a laptop and a GPU computer. — Even within the same model family, Mac and RTX use different checkpoint formats.

Load the MLX model with oMLX on a Mac

After opening oMLX on an Apple Silicon Mac, choose the model folder and start the server. Download the MLX checkpoint mlx-community/Qwen3.8-27B-4bit, then open http://localhost:8000/admin/chat in a browser. Select the model and send this request: Write a Python function that accepts a path string, returns None if the file is missing, and reads UTF-8 text if it exists. Explain the input and return types.

A successful response shows the Python code and an explanation of its input and return types in the chat. If it fails, check the model status and download completion in the management UI, then inspect memory pressure and the first server-log error. If the model is missing from the list, check the MLX checkpoint format and selected model folder.

oMLX installation and quick start

A laptop with a file icon, a shelf of blocks, and a graphics card connected by colored paths — Prepare the MLX path for Mac and CUDA with GGUF for RTX.

Start a CUDA llama.cpp server on RTX

On an RTX PC with the CUDA⁷ Toolkit installed, use the commands below to build CUDA-enabled llama.cpp and start its server. The final line uses -hf to select the GGUF repository and Q4_K_M; -ngl all places as many model layers as possible on the GPU⁸. -c 4096 sets a 4,096-token context. The ./build/bin/llama-server path below is for a Linux shell.

Build llama.cpp with CUDA and serve Q4_K_M

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -t llama-server
./build/bin/llama-server -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M -c 4096 -ngl all --host 127.0.0.1 --port 8080

Start the server at the local address after the CUDA build finishes.

When the server is ready, open http://localhost:8080 in a browser and select the model. Send the same Python request used in the Mac section.

When the response displays code and its explanation, the model load and request path are working. If the build fails, check whether CMake can find the CUDA Toolkit. If the server opens but the model does not start, check download progress, repository name, and free VRAM⁹, then retry with a 2,048-token context.

llama.cpp official CUDA build guide

On Windows CUDA, locate the built server executable

For a Windows build, prepare the Visual Studio 2022 C++ tools and CUDA Toolkit, then open the repository in a Visual Studio Developer PowerShell. The official build guide uses the same CMake CUDA option as Linux, while the server guide builds the llama-server target. The commands below build the server, locate the generated executable, and start it with the selected model.

Build with CUDA and start a Q4_K_M server on Windows

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -t llama-server
$server = Get-ChildItem -Path .\build -Filter llama-server.exe -Recurse | Select-Object -First 1 -ExpandProperty FullName
& $server -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M -c 4096 -ngl all --host 127.0.0.1 --port 8080

Start the found executable with the selected Q4_K_M model and local port.

When the server starts, open http://localhost:8080, select the model, and send the Python request from the Mac section unchanged.

A normal response shows the code and its input/return explanation, confirming model loading and the request path. If no executable was generated, inspect the first build error and CUDA Toolkit detection. If the server starts but model loading fails, check the first model error in the log and available VRAM.

Record first-run and repeated-run waits separately

First confirm that each server answers the Python request above. For a speed comparison, record the initial model-load time separately, warm up once, then send the same request three times. Keep context and answer limit fixed; record request start, first token¹⁰, and completion, along with VRAM on RTX or unified-memory pressure on Mac.

To increase input length, append a real code excerpt from your work to the same Python request and send identical text to both systems. A longer request can change prefill¹¹ time and KV-cache use. Mac's MLX Q4 and RTX's GGUF Q4_K_M differ in format and quantization¹², so record the model file to distinguish hardware effects from checkpoint differences.

Items to record when running the same workload on different hardware
Recorded value	Where to check	Why it matters
Model format and quantization	Model card and runtime log	Distinguishes Mac and RTX file paths
First-token and completion times	Request timer	Separates input processing from generation wait
Peak memory	Activity Monitor and GPU monitor	Checks room for other apps

Five colored blocks beside a laptop and a small device connect by lines to a note with a check mark. — Change input length and settings one at a time, and record both answers and wait times.

Test 0.7.1.dev1 INT8 prefill separately on supported models

oMLX 0.7.1.dev1, published on October 9, 2026, includes an INT8 activation-prefill improvement for Qwen3.8-27B oQ8e. Enable this path under Experimental Features in model settings. The release measurement used an M5 Max and the oQ8e checkpoint, a different file from the Q4 model mlx-community/Qwen3.8-27B-4bit in the Mac setup above. The table below lists the input lengths, measurement conditions, and figures.

To test INT8 prefill, prepare a supported oQ8e file and 0.7.1.dev1, then confirm a normal response with a short input. Enable INT8 activation prefill for that model, process an input with the same token count three times, and compare the median prefill speed or TTFT¹³. The published M5 Max gain concerns prefill, not generation, so record decode¹⁴ tok/s separately.

If you see a memory error or the first token takes longer, shorten the context and check the model-specific settings and the selected path in the server log. INT8 activation changes the representation used for computation, so also check output numbers and answer quality. Turn the option off and resend the same request to compare its effect.

oMLX release measurements for Qwen3.8-27B oQ8e INT8 activation prefill
M5 Max input	Published change	Stage compared
4K tokens	Prefill +25%	Prefill
16K tokens	Prefill +38%	Prefill

Published conditions for oMLX 0.7.1.dev1

Change MTP and KV-cache precision after recording a baseline

For a Qwen3.8-27B checkpoint that supports MTP¹⁵, you can compare Lightning MTP for that model. MTP affects token generation; KV-cache precision affects the memory and computation used to store conversation state. First record a response and memory use with both at their defaults. Finish a comparison with only MTP enabled, then change only the KV-cache setting. Use the same checkpoint and question at each step.

Version 0.7.1.dev1 is a development release. If stability matters more, stay on 0.7.0 and check new features in a separate test environment. If requests often fail, inspect server logs, context length, and VRAM or unified-memory status before adjusting MTP or the KV cache.

Development-build features and stable-release guidance

Appendix: Running BF16 vLLM on a separate NVIDIA server

Multiplying 27B parameters by two bytes for BF16¹⁶ gives about 54 GB for the weights alone. This calculation excludes runtime¹⁷ buffers and cache. The commands below are for a separate NVIDIA server that can hold BF16 weights, not the RTX 24 GB Q4_K_M recipe above. If you are following the main recipe, skip this appendix and use the earlier llama.cpp command.

Start BF16 vLLM on a separate server

vllm serve Qwen/Qwen3.8-27B --host 127.0.0.1 --port 8000 --max-model-len 8192 --gpu-memory-utilization 0.90

An example for a separate server that can hold BF16 weights.

After running the first command, check that the server process remains active. If it does not respond or exits, start with the first terminal error and GPU memory status. Once the server is running, use the second command to check the registered model ID for requests.

Query the model ID reported by the server

curl -N http://127.0.0.1:8000/v1/models

Use the model ID returned in the response for your request.

Official Qwen3.8-27B model card

Terminology notes

Checkpoint — A file containing saved model weights and related state. Versions or tasks in one model family may use different checkpoints.
Back to the text
MLX — Apple’s machine-learning framework, which runs models using memory shared by the CPU and GPU on Apple Silicon.
Back to the text
GGUF — A file format for model data, widely used by llama.cpp-based tools.
Back to the text
Context window — The token span of input and generated content a model can handle in one request. The supported limit and memory use depend on the model and runtime settings.
Back to the text
KV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.
Back to the text
Unified memory — An architecture where the CPU and GPU share one physical memory pool.
Back to the text
CUDA — A software platform for general-purpose computing on NVIDIA GPUs.
Back to the text
GPU — A processor designed to handle many calculations in parallel. It performs model computations during AI inference.
Back to the text
VRAM — Memory used by a graphics card’s GPU to store model weights and intermediate values.
Back to the text
Token — A unit into which a model divides input or output for processing.
Back to the text
Prefill — The stage where an LLM reads the input prompt and computes representations for its tokens. Longer prompts contain more tokens to process.
Back to the text
Quantization — A way to represent model weights or values with fewer bits, reducing storage and memory use. The coarser representation can also affect accuracy.
Back to the text
Time to First Token — The time from sending a request until its first output token arrives.
Back to the text
Token decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the text
MTP — A training objective or model component for predicting multiple future tokens at once. In speculative decoding, those predictions can be proposed as candidate tokens.
Back to the text
BF16 — A 16-bit floating-point format for model values. Its exponent has the same width as FP32, giving it a similarly wide value range.
Back to the text
Runtime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.
Back to the text

Frequently asked questions

What should you check first in this article?: For a Q4 checkpoint, use the MLX format with oMLX on an Apple Silicon Mac and GGUF with llama.cpp on an RTX. Confirm that each server answers the same request, then record first-token and completion times along with memory use. The 0.7.1.dev1 INT8 activation-prefill results apply separately to Qwen3.8-27B oQ8e on an M5 Max.
How much memory does it need?: oMLX on a Mac runs an MLX-converted checkpoint, while llama.cpp on an RTX reads GGUF. Files are not interchangeable just because they share the Qwen3.8-27B name. Use `mlx-community/Qwen3.8-27B-4bit` on the Mac and Q4_K_M from `ggml-org/Qwen3.8-27B-GGUF` on the RTX as baseline files. The Q4_K_M file is about 17.6 GB, but actual memory use also includes the KV cache and working buffers.

Serving recipe

Settings for your hardware

For one user

Choose your hardware to see the launch command and settings you can adjust.

Choose your hardware

Available memory: 56GB / Bandwidth: 307GB/s

Profile

Apple MacBook Pro M5 Pro (64GB)

Memory fits

Recommended runtime: oMLX (4-bit MLX (a different format from GGUF Q4_K_M)) · Documented runtime settings

For your first run or reliable everyday use

Profile context length

16,384 tokens

Based on runtime documentationMBP M5 Pro 64GB · Q4 · 1 request

A starting setup based on the runtime documentation. Follow the steps below and compare speed as you make changes.

Estimated decode speed

13.9 ~ 15.4 tok/s

Estimate for one user

Estimated TTFT

36.0 ~ 41.9 s

Prefill: 392 ~ 456 tok/s

Estimated memory use

About 21.21GB / 56GB

Remaining margin: approx. 34.8GB

Server address

http://127.0.0.1:8000

Local only · 127.0.0.1

Estimates use the calculator’s recommended quantization and acceleration with 16,384tokens. They are not measurements of the configuration above.

Settings in this profile

Value	Why this setting
Weights4-bit MLX (a different format from GGUF Q4_K_M)	Use an MLX model directory. This is a different file format from Q4_K_M GGUF.
Check input length16,384	Check the input limit in model settings before sending a request. This launch command does not set the context length.
KV cacheCheck model settings	This launch command does not set the KV cache precision.
Prefill batch sizeCheck runtime settings	Do not use llama.cpp's batch-size and ubatch-size values as substitutes for MLX settings.
Concurrent requests1	Measure the speed of one request, separately from the combined throughput of multiple requests.
Memory headroom2 GB or more	After loading the model, check actual memory use and whether the system is swapping.

MBP M5 Pro 64GB Run command (oMLX)

Apple Silicon · MLX Model · Memory Guard

MODEL_DIR="$HOME/.omlx/models"
omlx serve \
  --model-dir "$MODEL_DIR" \
  --host 127.0.0.1 \
  --port 8000 \
  --max-concurrent-requests 1 \
  --memory-guard balanced

Record a baseline with this setup, then follow the tuning steps below, changing one setting at a time.

Checks after startup

01Check the startup log for completed model loading and the actual context length.
02After warming up, send one request at a time. Use three different inputs of the same length to record fresh-input processing time.
03Record repeated-input results separately as cache-reuse runs. Do not combine them with fresh-input results.
04Record prefill tok/s, decode tok/s and peak memory together, then change one setting at a time.

Trade-offs

Conservative context and concurrency settings may produce less than the hardware's maximum throughput.

See speed with recommended settings MBP M5 Pro 64GB View price, power, and sales configuration

Adjust one item at a time

Speed tuning, step by step

If several values are changed at once, it is difficult to find the cause.

STEP 1
Save a baseline
Record first-input and cache-reuse runs separately, three times each. Compare prefill, decode and peak memory together.
When to stop: Do not move on if there are errors or less than 2GB of memory headroom.
STEP 2
Tune prefill chunk size
Increase in steps of 1,024 → 2,048 → 4,096, comparing TTFT and peak memory on long inputs.
When to stop: Revert to the previous value if TTFT does not improve or peak memory rises sharply.
STEP 3
KV Cache Tuning
Only when you need more context, compare Q8 cache against the F16/BF16 baseline using the same question.
When to stop: Keep the default precision if the output changes or you already have enough context.
STEP 4
MTP/speculative decoding
Enable it only for supported models. Test code and prose separately, measuring acceptance rates and actual decode tok/s.
When to stop: Turn it off if the median of three runs does not improve on the baseline.
STEP 5
Context expansion
Double context length at each step until you reach what you need. Check retrieval from the middle of the input and whether swapping occurs.
When to stop: Reduce by one step if retrieval accuracy drops or swapping or memory compression starts.

Submit a measurement from my hardware

Import a JSON file with at least three runs under the same conditions. The file stays in this browser until you submit it.

Include only hardware, model, runtime settings and measurements. Do not include prompts, responses, raw logs or file paths.

If you do not have a measurement file, use the tool with your running local server. It requires Node.js 20 or later and does not upload results automatically.

Read the measurement instructions Download the measurement tool Download the hardware and settings template

Import measurement JSON (up to 32KB)

These are reviewed community submissions, not measurements made by this site.

Report a setup issue

Tell us where this setup failed. Only the site administrator can read your report.

Setup being reported · MBP M5 Pro 64GB · Qwen3.8 27B · Balanced

Change log

These entries record changes to the site's guidance. They do not automatically check your installed engine or model version.

2026-09-06
Corrected model formats and runtime settings for Mac
Separated the Mac MLX path from GGUF Q4_K_M and aligned the engine name, model format and command. KV-cache precision and prefill batching that the command does not specify are now marked for checking in the model or runtime settings.
Also removed MLX speed profiles that implied a larger batch despite leaving the command unchanged.
2026-09-06
Removed another engine's options from Qwen 27B Spark settings
Prevented llama.cpp-specific speculative-decoding flags from appearing in single- and dual-Spark SGLang settings. The DFlash2 and DSpark configurations retain their respective models and runtime paths.
2026-09-06
Separated uncached-input and cache-reuse records
Updated the checks to record first-time inputs separately from repeated inputs. Cache effects are not folded into uncached prefill throughput or hardware differences.
2026-09-06
Added saving and reopening hardware profiles
Runnable profiles can now be saved with their hardware selection in this browser, up to five entries, and reopened from the guide directory. A notice prompts you to recheck before running if the published settings have changed or the profile is no longer available.
2026-09-06
Added links to runtime explanations
Engine names in the settings now link to the corresponding runtime guide. Paths without an identified engine are not linked to a guessed program.