Qwen3.6 35B-A3B: vLLM setup and the qwen3 parser

Hardware-specific speed tuning and serving recipe summary

Provides weights, KV cache, prefill batch, and concurrent request with acceleration functionality, separated for balanced performance, speed, and long-context profiling.

On the detail page, you can check the command per profile, applied value, stopping conditions, and measurement order. The execution command is displayed only for combinations of model file and execution program that have been verified.

On 24GB, what remains after loading the model matters

The compatible Q4 conversion takes the guidance of around 20GB as a starting point, but confirms the actual file. Model weights, 4K to 8K context cache, and execution space must be included. Just having a message loaded on the GPU does not guarantee a long conversation.

Even above 32GB, you should make sure you have room for other apps and cache. The official checkpoint has a different capacity than the Q4 conversion. The multi-GPU example below and the command to open the desktop file have different paths.

First, match the support of the model you are receiving with the running app. If you have a file that was working normally, you should keep it and test the new path so you have a standard to compare it to if a problem occurs.

The number of GPUs in the server example is not a purchase recommendation

The number of parallels in the official vLLM example must match the installed equipment and memory/support requirements. Don't read the 8 in the comments to mean that eight GPUs are all you need. If your intention is to run a desktop Q4, use LM Studio or the path in the device selector above.

vLLM checks which version supports the model and the qwen3 inference parser. This example takes the vLLM 0.19.0 or later instructions as a starting point, but the actual execution condition is whether it is supported by the currently installed version. The server first places only local addresses.

MTP is added from the supported combinations after the standard answer comes out normally. Rather than copying multiple options at once, a record of default values ​​and changed items is better for finding the cause in your environment.

GPU_COUNT=8 # 공식 예시는 8 GPU, 실제 구성에 맞춰 수정
vllm serve Qwen/Qwen3.6-35B-A3B   --host 127.0.0.1   --port 8000   --tensor-parallel-size "$GPU_COUNT"   --max-model-len 8192   --reasoning-parser qwen3   --gpu-memory-utilization 0.92

lms ls
lms load <qwen-3.6-35b-identifier> --gpu=max --context-length=8192
lms server start --port 1234

Distinguish between thinking time and writing time.

The inference parser helps apps distinguish between the thought process and the final result. Make sure you're already generating thought tokens when the final sentence appears late on the screen. Recording all of this as prefill will result in a misreading of the input processing performance.

Compare the input length, maximum output, and thought settings for the same question. After the prepare run, we leave the median of the three runs and the actual number of output tokens. Comparing only the completion times of two runs with significantly different answer lengths makes it difficult to separate the generation rates.

The test below is for the 8000 address of vLLM. If you used LM Studio, match 1234 with the model identifier you loaded. If the addresses are different, it may not be a matter of the model being slow, but a question of asking a different server.

curl -N http://127.0.0.1:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "Qwen/Qwen3.6-35B-A3B",
    "messages": [
      {"role": "user", "content": "파이썬으로 이진 탐색 함수를 작성하고 시간 복잡도를 설명해줘."}
    ],
    "max_tokens": 1024,
    "stream": true
  }'

Increase long context and acceleration one by one

If 24GB is limited, try limiting the request to one and keeping the context short. We then check the cache and peak memory when increasing the input length to the required length. If the speed drops suddenly, first check to see if any weights have been transferred to swap or CPU.

Turn on supported MTP to see if the average acknowledgment length and decode actually improve. Don't apply the scaling that works well for one type of code to general writing. Cache precision and prefill chunks should also be compared individually.

If waiting for long input is the only issue, there is reason to look at input processing and caching rather than decode acceleration. By writing down which sections are slow, you can narrow down your next experiment without having to add more settings.

Once it runs, try finishing your usual work

Don't assume that testing binary search code alone is sufficient. Enter the code or document you actually want to modify and check the results. It is a setting to use only when both the answer you want and whether it is worth waiting for are correct.

Save the model file, runtime, context and acceleration in that state. These are also the criteria for comparison when looking at larger equipment. If you're already good enough, continuing to use your current configuration instead of a new GPU is what this recipe can give you.