Qwen3.8-27B settings by hardware: oMLX on Mac and llama.cpp on RTX

Hardware-specific speed tuning and serving recipe summary

Provides weights, KV cache, prefill batch, and concurrent request with acceleration functionality, separated for balanced performance, speed, and long-context profiling.

On the detail page, you can check the command per profile, applied value, stopping conditions, and measurement order. The execution command is displayed only for combinations of model file and execution program that have been verified.

Match the model file to your runtime

Choose your hardware above to see the appropriate setup and commands. For desktop use, start with a compatible Q4 GGUF or MLX conversion. The file-size estimate of around 17GB is only a starting point: check the actual download size and memory use while running.

Passing the official BF16 model ID to the server does not load the same Q4 file. A simple estimate puts the 27B weights alone at about 54GB, with additional working memory needed. Do not copy the official server example below as though it were a Q4 command for a 24GB GPU.

First get a text question working, then check image input and additional acceleration. Changing the model file, vision support and MTP together on the first run makes failures harder to diagnose.

Keep a working baseline

Use the oMLX command for Mac or the llama.cpp command for RTX shown for your selected hardware. Start with one request at a time and the displayed context length. Before increasing the context or pushing memory limits, establish a setup that completes an answer correctly.

After warming up, record prefill, time to first token, decode and peak memory three times. For a new document, confirm that the input cache was not reused. Keep repeated-input results separate. Return to these records when changing chunk size or cache precision to see whether the change actually helps.

The vLLM command below is a separate setup for an NVIDIA server with enough memory for the official weights. Do not mix it with the desktop commands in the selector. Bind the server to 127.0.0.1 first; if other devices will connect, configure authentication and access restrictions separately.

vllm serve Qwen/Qwen3.8-27B   --host 127.0.0.1   --port 8000   --max-model-len 8192   --gpu-memory-utilization 0.90

A running server still needs a reply test

Check the model list and use the loaded model's identifier in your request. Match the address to the command you ran: this guide uses port 8000 for oMLX and 8080 for llama.cpp. If that differs from the example below, update the address before sending a short question.

Check that the answer streams in chunks and that the app separates reasoning from the final answer. If the model spends a long time generating reasoning tokens, the final text may appear later. Check the server logs too, so that this time is not mistakenly counted as prefill alone.

Once a basic question completes, try a document you normally use. Its time to first response and completion time tell you more about everyday use than the highest tok/s from a short greeting.

# 1. 로드된 모델 목록 확인
curl -N http://127.0.0.1:8000/v1/models

# 2. OpenAI 호환 챗 완성 API 호출
curl -N http://127.0.0.1:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "<local-model-identifier>",
    "messages": [
      {"role": "user", "content": "로컬 LLM 서빙의 장점을 세 가지로 요약해줘."}
    ],
    "temperature": 0.6,
    "max_tokens": 1024,
    "stream": true
  }'

Find where the wait occurs

If the wait is long before the answer starts, check prefill chunk size, input length and cache state. If generation is slow after it starts, look for weights offloaded to the CPU, memory pressure or swapping. Trying to solve both problems only by changing MTP steps can hide the real cause.

Enable MTP only with supported files and runtimes, then compare it with the baseline using the same question. A short average accepted length or high verification overhead can erase the benefit. Do not change cache precision and MTP together.

If the context you need runs out of memory, first reduce request count and length to recover a working baseline. Consider more memory after checking whether the limit keeps affecting work you regularly need to do.

Keep settings you can use every day, not just a fast result

If you use both code editing and document summarization, keep test questions for each. Some settings help only one of them. Alongside speed, check for mistakes in the output and failed runs.

Finally, save the model file, app version, context length and acceleration options. That gives you a working setup to restore after a troublesome update. It is easier to maintain a setup that finishes your work and then make it faster than to chase the highest number from the start.