Qwen3.8-27B setup: oMLX on Mac and llama.cpp on RTX

Hardware-specific speed tuning and serving recipe summary

Provides weights, KV cache, prefill batch, and concurrent request with acceleration functionality, separated for balanced performance, speed, and long-context profiling.

On the detail page, you can check the command per profile, applied value, stopping conditions, and measurement order. The execution command is displayed only for combinations of model file and execution program that have been verified.

Match the model file to the runtime first

Select your current device from the device selector to see the path and commands it will use. On desktop, it checks against compatible Q4 GGUF or MLX renditions. The file size guide of approximately 17GB is only a starting point, and you must check the actual download and execution usage.

The path to passing the official BF16 model ID to the server is not to open the same Q4 file. A simple calculation of the 27B weight alone is approximately 54GB and requires more work space. Please do not copy the official server example below as the Q4 command for 24GB GPU.

After running the text question normally, check image input and additional acceleration. If you change the model file and Vision/MTP support at the same time on the first load, it is difficult to find out which part failed.

Keep one configuration that works

The commands in Mac oMLX and RTX llama.cpp are used according to the above device selector. Leave the request as one and start from the indicated context. Rather than jumping straight into larger context or aggressive memory settings, leave a normal answer first.

Records prefill, time to first token, decode and peak memory. After executing the preparation, use the median of three identical inputs as a standard. When we later change the prefill chunk or cache precision, we return to this record to determine the actual benefit.

The vLLM command below is a separate NVIDIA server path that can contain the formula weights. Do not mix with desktop commands in selectors. First, leave the server at 127.0.0.1 and prepare separately for authentication and access restrictions if you plan to access it from other devices.

vllm serve Qwen/Qwen3.8-27B   --host 127.0.0.1   --port 8000   --max-model-len 8192   --gpu-memory-utilization 0.90

A running server is not yet a working answer

Checks the model list and puts the actually loaded identifier in the request. You must use the address of the selected instruction, such as 8000 in this guide for oMLX and 8080 for llama.cpp. If it is different from the example address below, please correct the address and send a short question.

Make sure your answers arrive in pieces and that your thought process and final answer are separated in the app. If you produce thought tokens for a long time, the final sentence may appear late on the screen. I also look at the server logs to make sure I'm not incorrectly logging this time as pure prefill.

Once you've completed the basic questions, send your usual documentation. The first reply and completion time of that document are closer to actual usage than the best tok/s from a short greeting.

# 1. 로드된 모델 목록 확인
curl -N http://127.0.0.1:8000/v1/models

# 2. OpenAI 호환 챗 완성 API 호출
curl -N http://127.0.0.1:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "<local-model-identifier>",
    "messages": [
      {"role": "user", "content": "로컬 LLM 서빙의 장점을 세 가지로 요약해줘."}
    ],
    "temperature": 0.6,
    "max_tokens": 1024,
    "stream": true
  }'

If it's slow, check which section it is first.

If you wait a long time after entering input, it checks the prefill chunk, input length, and cache status. If the response is slow after starting, check to see if weight has been transferred to the CPU or if memory pressure or swap has occurred. If you try to solve both problems with one MTP step count, you might miss the cause.

After you turn on MTP on supported files and runtimes, compare them to the default by asking the same questions. If the average acknowledgment length is low or verification costs are high, there may be no benefit in turning it on. Do not change cache precision and MTP at once.

If you run out of memory for the long context you want, first reduce the number and length of requests to have a baseline to return to. Verify that the limit is repeated only for tasks that continue to require it, and then review the upper memory configuration.

Keep a setup you can use every day, not just a good score

If you're both editing code and summarizing documentation, ask two types of questions. There are also settings that make it faster on only one side. In addition to speed, you should check the actual results for errors and execution failures.

Finally, save the model file, app version, context, and acceleration options. Even if a problem occurs after the update, the normal configuration can be restored. Rather than finding the fastest number from the beginning, it is easier to maintain a setting that gets my work done until the end and then increase the speed.