Model setup recipes

Qwen3.8-27B settings by hardware: oMLX on Mac and llama.cpp on RTX

If Qwen3.8-27B runs slower than the results others have shared, the setup may be different. The same model name can still mean a different file, context length or runtime. This guide separates oMLX settings for Mac from llama.cpp settings for RTX, then helps you find where the wait occurs and adjust one setting at a time.

Serving recipe

Settings for your hardware

For one user

Choose your hardware to see the launch command and settings you can adjust.

Choose your hardware

Available memory: 56GB / Bandwidth: 307GB/s

Profile

Apple MacBook Pro M5 Pro (64GB)

Memory fits

Recommended runtime: oMLX (4-bit MLX (a different format from GGUF Q4_K_M)) · Documented runtime settings

For your first run or reliable everyday use

Profile context length

16,384 tokens

Based on runtime documentationMBP M5 Pro 64GB · Q4 · Request 1个

A starting setup based on the runtime documentation. Follow the steps below and compare speed as you make changes.

Estimated decode speed

13.9 ~ 15.4 tok/s

Estimate for one user

Estimated TTFT

36.0 ~ 41.9 s

Prefill: 392 ~ 456 tok/s

Estimated memory use

About 21.21GB / 56GB

Remaining margin: approx. 34.8GB

Server address

http://127.0.0.1:8000

Local only · 127.0.0.1

Estimates use the calculator’s recommended quantization and acceleration with 16,384tokens. They are not measurements of the configuration above.

Settings in this profile

Value	Why this setting
Model weights4-bit MLX (a different format from GGUF Q4_K_M)	Use an MLX model directory. This is a different file format from Q4_K_M GGUF.
Check input length16,384	Check the input limit in model settings before sending a request. This launch command does not set the context length.
KV cacheCheck model settings	This launch command does not set the KV cache precision.
Prefill batch sizeCheck runtime settings	Do not use llama.cpp's batch-size and ubatch-size values as substitutes for MLX settings.
Concurrent requests1	Measure the speed of one request, separately from the combined throughput of multiple requests.
Memory headroom2 GB or more	After loading the model, check actual memory use and whether the system is swapping.

MBP M5 Pro 64GB Run command (oMLX)

Apple Silicon · MLX Model · Memory Guard

MODEL_DIR="$HOME/.omlx/models"
omlx serve \
  --model-dir "$MODEL_DIR" \
  --host 127.0.0.1 \
  --port 8000 \
  --max-concurrent-requests 1 \
  --memory-guard balanced

Record a baseline with this setup, then follow the tuning steps below, changing one setting at a time.

Checks after startup

01Check the startup log for completed model loading and the actual context length.
02After warming up, send one request at a time. Use three different inputs of the same length to record fresh-input processing time.
03Record repeated-input results separately as cache-reuse runs. Do not combine them with fresh-input results.
04Record prefill tok/s, decode tok/s and peak memory together, then change one setting at a time.

Trade-offs

Conservative context and concurrency settings may produce less than the hardware's maximum throughput.

See speed with recommended settings MBP M5 Pro 64GB View price, power, and sales configuration

Adjust one item at a time

Speed tuning, step by step

If several values are changed at once, it is difficult to find the cause.

STEP 1
Save a baseline
Record first-input and cache-reuse runs separately, three times each. Compare prefill, decode and peak memory together.
When to stop: Do not move on if there are errors or less than 2GB of memory headroom.
STEP 2
Tune prefill chunk size
Increase in steps of 1,024 → 2,048 → 4,096, comparing TTFT and peak memory on long inputs.
When to stop: Revert to the previous value if TTFT does not improve or peak memory rises sharply.
STEP 3
KV Cache Tuning
Only when you need more context, compare Q8 cache against the F16/BF16 baseline using the same question.
When to stop: Keep the default precision if the output changes or you already have enough context.
STEP 4
MTP/speculative decoding
Enable it only for supported models. Test code and prose separately, measuring acceptance rates and actual decode tok/s.
When to stop: Turn it off if the median of three runs does not improve on the baseline.
STEP 5
Context expansion
Double context length at each step until you reach what you need. Check retrieval from the middle of the input and whether swapping occurs.
When to stop: Reduce by one step if retrieval accuracy drops or swapping or memory compression starts.

Submit a measurement from my hardware

Import a JSON file with at least three runs under the same conditions. The file stays in this browser until you submit it.

Include only hardware, model, runtime settings and measurements. Do not include prompts, responses, raw logs or file paths.

If you do not have a measurement file, use the tool with your running local server. It requires Node.js 20 or later and does not upload results automatically.

Read the measurement instructions Download the measurement tool Download the hardware and settings template

Import measurement JSON (up to 32KB)

These are reviewed community submissions, not measurements made by this site.

Report a setup issue

Tell us where this setup failed. Only the site administrator can read your report.

Setup being reported · MBP M5 Pro 64GB · Qwen3.8 27B · Balanced

Qwen3.8 27B · 4K

Qwen3.8 27B · 4,096 input tokens · Estimated for one user. These are the same calculated results as the speed experience, not a measured run.

Time to first token is the wait before the answer starts. Token generation speed describes how quickly the rest follows. For long documents, consider both.

Mac mini M4 32GB · Qwen3.8 27B

Quantization: Q4_K_M · Acceleration: none · Token generation: 5.1 ~ 5.7 tok/s · First token: 16.4 to 35.5 seconds

Prompt processing: 116 ~ 252 tok/s · Memory required: 17.6GB · Available memory: 27GB

Mac mini M4 Pro 48GB · Qwen3.8 27B

Quantization: Q4_K_M · Acceleration: none · Token generation: 12.3 ~ 13.7 tok/s · First token: 7.8 to 14.3 seconds

Prompt processing: 289 ~ 530 tok/s · Memory required: 17.6GB · Available memory: 41GB

RTX 4090 24GB · Qwen3.8 27B

Quantization: Q4_K_M · Acceleration: none · Token generation: 38.3 ~ 51.9 tok/s · First token: 2.6 to 3.6 seconds

Prompt processing: 1,141 ~ 1,591 tok/s · Memory required: 17.6GB · Available memory: 22.5GB

Match the model file to your runtime

Choose your hardware above to see the appropriate setup and commands. For desktop use, start with a compatible Q4 GGUF or MLX conversion. The file-size estimate of around 17GB is only a starting point: check the actual download size and memory use while running.

Passing the official BF16 model ID to the server does not load the same Q4 file. A simple estimate puts the 27B weights alone at about 54GB, with additional working memory needed. Do not copy the official server example below as though it were a Q4 command for a 24GB GPU.

First get a text question working, then check image input and additional acceleration. Changing the model file, vision support and MTP together on the first run makes failures harder to diagnose.

Keep a working baseline

Use the oMLX command for Mac or the llama.cpp command for RTX shown for your selected hardware. Start with one request at a time and the displayed context length. Before increasing the context or pushing memory limits, establish a setup that completes an answer correctly.

After warming up, record prefill, time to first token, decode and peak memory three times. For a new document, confirm that the input cache was not reused. Keep repeated-input results separate. Return to these records when changing chunk size or cache precision to see whether the change actually helps.

The vLLM command below is a separate setup for an NVIDIA server with enough memory for the official weights. Do not mix it with the desktop commands in the selector. Bind the server to 127.0.0.1 first; if other devices will connect, configure authentication and access restrictions separately.

Official vLLM server setup (BF16 weights)

vllm serve Qwen/Qwen3.8-27B   --host 127.0.0.1   --port 8000   --max-model-len 8192   --gpu-memory-utilization 0.90

Run on 127.0.0.1 on an NVIDIA server with room for the official BF16 weights and runtime overhead.

A running server still needs a reply test

Check the model list and use the loaded model's identifier in your request. Match the address to the command you ran: this guide uses port 8000 for oMLX and 8080 for llama.cpp. If that differs from the example below, update the address before sending a short question.

Check that the answer streams in chunks and that the app separates reasoning from the final answer. If the model spends a long time generating reasoning tokens, the final text may appear later. Check the server logs too, so that this time is not mistakenly counted as prefill alone.

Once a basic question completes, try a document you normally use. Its time to first response and completion time tell you more about everyday use than the highest tok/s from a short greeting.

Check the API endpoint and test a chat request

# 1. 로드된 모델 목록 확인
curl -N http://127.0.0.1:8000/v1/models

# 2. OpenAI 호환 챗 완성 API 호출
curl -N http://127.0.0.1:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "<local-model-identifier>",
    "messages": [
      {"role": "user", "content": "로컬 LLM 서빙의 장점을 세 가지로 요약해줘."}
    ],
    "temperature": 0.6,
    "max_tokens": 1024,
    "stream": true
  }'

These examples use port 8000 for oMLX and 8080 for llama.cpp. Match the address to the runtime you started.

Find where the wait occurs

If the wait is long before the answer starts, check prefill chunk size, input length and cache state. If generation is slow after it starts, look for weights offloaded to the CPU, memory pressure or swapping. Trying to solve both problems only by changing MTP steps can hide the real cause.

Enable MTP only with supported files and runtimes, then compare it with the baseline using the same question. A short average accepted length or high verification overhead can erase the benefit. Do not change cache precision and MTP together.

If the context you need runs out of memory, first reduce request count and length to recover a working baseline. Consider more memory after checking whether the limit keeps affecting work you regularly need to do.

Keep settings you can use every day, not just a fast result

If you use both code editing and document summarization, keep test questions for each. Some settings help only one of them. Alongside speed, check for mistakes in the output and failed runs.

Finally, save the model file, app version, context length and acceleration options. That gives you a working setup to restore after a troublesome update. It is easier to maintain a setup that finishes your work and then make it faster than to chase the highest number from the start.

Keep settings you can use every day, not just a fast result

Running an NVFP4 checkpoint

Requires a compatible NVFP4 kernel. Spark measurements and RTX estimates are kept separate.

vLLM / SGLang · Inferact/Qwen3.8-27B-NVFP4

Change log

These entries record changes to the site's guidance. They do not automatically check your installed engine or model version.

2026-09-06
Corrected model formats and runtime settings for Mac
Separated the Mac MLX path from GGUF Q4_K_M and aligned the engine name, model format and command. KV-cache precision and prefill batching that the command does not specify are now marked for checking in the model or runtime settings.
Also removed MLX speed profiles that implied a larger batch despite leaving the command unchanged.
2026-09-06
Removed another engine's options from Qwen 27B Spark settings
Prevented llama.cpp-specific speculative-decoding flags from appearing in single- and dual-Spark SGLang settings. The DFlash2 and DSpark configurations retain their respective models and runtime paths.
2026-09-06
Separated uncached-input and cache-reuse records
Updated the checks to record first-time inputs separately from repeated inputs. Cache effects are not folded into uncached prefill throughput or hardware differences.
2026-09-06
Added saving and reopening hardware profiles
Runnable profiles can now be saved with their hardware selection in this browser, up to five entries, and reopened from the guide directory. A notice prompts you to recheck before running if the published settings have changed or the profile is no longer available.
2026-09-06
Added links to runtime explanations
Engine names in the settings now link to the corresponding runtime guide. Paths without an identified engine are not linked to a guessed program.