Model setup recipes

Qwen3.6 35B-A3B: vLLM setup and the qwen3 parser

The 3B active parameter count can make Qwen3.6 35B-A3B sound like a small model. Yet on 24GB hardware, loading all its Q4 weights may leave little room. First get the memory setup right and separate reasoning from the final answer, then look for settings that speed up your own questions.

Serving recipe

Settings for your hardware

For one user

Choose your hardware to see the launch command and settings you can adjust.

Choose your hardware

Available memory: 56GB / Bandwidth: 307GB/s

Profile

Apple MacBook Pro M5 Pro (64GB)

Memory fits

Recommended runtime: oMLX (4-bit MLX (a different format from GGUF Q4_K_M)) · Documented runtime settings

For your first run or reliable everyday use

Profile context length

16,384 tokens

Based on runtime documentationMBP M5 Pro 64GB · Q4 · Request 1个

A starting setup based on the runtime documentation. Follow the steps below and compare speed as you make changes.

Estimated decode speed

22.6 ~ 54.1 tok/s

Estimate for one user

Estimated TTFT

8.9 ~ 21.2 s

Prefill: 774 ~ 1,854 tok/s

Estimated memory use

About 21.91GB / 56GB

Remaining margin: approx. 34.1GB

Server address

http://127.0.0.1:8000

Local only · 127.0.0.1

Estimates use the calculator’s recommended quantization and acceleration with 16,384tokens. They are not measurements of the configuration above.

Settings in this profile

Value	Why this setting
Model weights4-bit MLX (a different format from GGUF Q4_K_M)	Use an MLX model directory. This is a different file format from Q4_K_M GGUF.
Check input length16,384	Check the input limit in model settings before sending a request. This launch command does not set the context length.
KV cacheCheck model settings	This launch command does not set the KV cache precision.
Prefill batch sizeCheck runtime settings	Do not use llama.cpp's batch-size and ubatch-size values as substitutes for MLX settings.
Concurrent requests1	Measure the speed of one request, separately from the combined throughput of multiple requests.
Memory headroom2 GB or more	After loading the model, check actual memory use and whether the system is swapping.

MBP M5 Pro 64GB Run command (oMLX)

Apple Silicon · MLX Model · Memory Guard

MODEL_DIR="$HOME/.omlx/models"
omlx serve \
  --model-dir "$MODEL_DIR" \
  --host 127.0.0.1 \
  --port 8000 \
  --max-concurrent-requests 1 \
  --memory-guard balanced

Record a baseline with this setup, then follow the tuning steps below, changing one setting at a time.

Checks after startup

01Check the startup log for completed model loading and the actual context length.
02After warming up, send one request at a time. Use three different inputs of the same length to record fresh-input processing time.
03Record repeated-input results separately as cache-reuse runs. Do not combine them with fresh-input results.
04Record prefill tok/s, decode tok/s and peak memory together, then change one setting at a time.

Trade-offs

Conservative context and concurrency settings may produce less than the hardware's maximum throughput.

See speed with recommended settings MBP M5 Pro 64GB View price, power, and sales configuration

Adjust one item at a time

Speed tuning, step by step

If several values are changed at once, it is difficult to find the cause.

STEP 1
Save a baseline
Record first-input and cache-reuse runs separately, three times each. Compare prefill, decode and peak memory together.
When to stop: Do not move on if there are errors or less than 2GB of memory headroom.
STEP 2
Tune prefill chunk size
Increase in steps of 1,024 → 2,048 → 4,096, comparing TTFT and peak memory on long inputs.
When to stop: Revert to the previous value if TTFT does not improve or peak memory rises sharply.
STEP 3
KV Cache Tuning
Only when you need more context, compare Q8 cache against the F16/BF16 baseline using the same question.
When to stop: Keep the default precision if the output changes or you already have enough context.
STEP 4
MTP/speculative decoding
Enable it only for supported models. Test code and prose separately, measuring acceptance rates and actual decode tok/s.
When to stop: Turn it off if the median of three runs does not improve on the baseline.
STEP 5
Context expansion
Double context length at each step until you reach what you need. Check retrieval from the middle of the input and whether swapping occurs.
When to stop: Reduce by one step if retrieval accuracy drops or swapping or memory compression starts.

Submit a measurement from my hardware

Import a JSON file with at least three runs under the same conditions. The file stays in this browser until you submit it.

Include only hardware, model, runtime settings and measurements. Do not include prompts, responses, raw logs or file paths.

If you do not have a measurement file, use the tool with your running local server. It requires Node.js 20 or later and does not upload results automatically.

Read the measurement instructions Download the measurement tool Download the hardware and settings template

Import measurement JSON (up to 32KB)

These are reviewed community submissions, not measurements made by this site.

Report a setup issue

Tell us where this setup failed. Only the site administrator can read your report.

Setup being reported · MBP M5 Pro 64GB · Qwen3.6 35B-A3B (MoE) · Balanced

Qwen3.6 35B-A3B (MoE) · 4K

Qwen3.6 35B-A3B (MoE) · 4,096 input tokens · Estimated for one user. These are the same calculated results as the speed experience, not a measured run.

If you already own this device, try the configuration below first. If the wait is acceptable, speed alone may not justify replacing it.

Mac mini M4 32GB · Qwen3.6 35B-A3B (MoE)

Quantization: Q4_K_M · Acceleration: none · Token generation: 7.8 ~ 22.5 tok/s · First token: 5.5 to 15.7 seconds

Prompt processing: 263 ~ 757 tok/s · Memory required: 21.3GB · Available memory: 27GB

DGX Spark 128GB · Qwen3.6 35B-A3B (MoE)

Quantization: Q4_K_M · Acceleration: none · Token generation: 17 ~ 42.7 tok/s · First token: 1.5 to 2.8 seconds

Prompt processing: 1,520 ~ 2,782 tok/s · Memory required: 21.3GB · Available memory: 116GB

2× DGX Spark 256GB · Qwen3.6 35B-A3B (MoE)

Quantization: Q4_K_M · Acceleration: none · Token generation: 17 ~ 61.4 tok/s · First token: 0.91 to 2.4 seconds

Prompt processing: 1,763 ~ 4,563 tok/s · Memory required: 21.3GB · Available memory: 232GB

Two devices can split a model between them. More memory does not mean a single answer generates twice as fast. An interconnect and distributed configuration are required.

On 24GB, what remains after loading the model matters

Treat the roughly 20GB estimate for a compatible Q4 conversion as a starting point, and check the actual file. Memory must hold the weights, the cache for a 4K–8K context and runtime working space together. A successful GPU-load message does not guarantee that a long conversation will fit.

Even at 32GB or more, check the room left for other apps and the cache. The official checkpoint is a different size from a Q4 conversion. The multi-GPU example below and the desktop model-loading command are separate setups.

First confirm that the app supports the model you plan to download. Keep any model file that already works while testing a new setup, so you have a baseline if something goes wrong.

On 24GB, what remains after loading the model matters

The example's GPU count is not a shopping list

The parallelism in the official vLLM example must match your installed hardware, memory and runtime support. The 8 in the comment does not mean any eight GPUs will work. For desktop Q4 use, choose LM Studio or the setup shown in the hardware selector above.

Check that your vLLM version supports the model and the qwen3 reasoning parser. This example starts from the guidance for vLLM 0.19.0 or later, but support in the version you actually installed is what matters. Bind the server to the local address first.

Add MTP on a supported combination only after the baseline produces a correct answer. A record of the baseline and each changed setting is more useful for diagnosing your setup than a peak result obtained by copying many options at once.

Official checkpoint with vLLM 0.19.0+ (multi-GPU)

GPU_COUNT=8 # 공식 예시는 8 GPU, 실제 구성에 맞춰 수정
vllm serve Qwen/Qwen3.6-35B-A3B   --host 127.0.0.1   --port 8000   --tensor-parallel-size "$GPU_COUNT"   --max-model-len 8192   --reasoning-parser qwen3   --gpu-memory-utilization 0.92

For the official checkpoint, set tensor parallel size to match the installed GPU count.

Desktop serving with LM Studio (Q4 GGUF)

lms ls
lms load <qwen-3.6-35b-identifier> --gpu=max --context-length=8192
lms server start --port 1234

Use LM Studio to start a local daemon on a Mac with at least 32GB of memory or a high-memory GPU.

Separate reasoning time from answer generation

The reasoning parser helps the app separate reasoning from the final answer. If final text is slow to appear, check whether the model is already generating reasoning tokens. Counting all that time as prefill would misrepresent input-processing performance.

Compare the same question with matched input length, output limit and reasoning settings. After warming up, record the median of three runs and the actual output token count. Comparing only completion times when answer lengths differ substantially does not isolate generation speed.

The test below uses vLLM on port 8000. For LM Studio, use port 1234 and the identifier of the loaded model. A wrong address can mean you are asking a different server, not that your model is slow.

Test a reasoning chat completion request

curl -N http://127.0.0.1:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "Qwen/Qwen3.6-35B-A3B",
    "messages": [
      {"role": "user", "content": "파이썬으로 이진 탐색 함수를 작성하고 시간 복잡도를 설명해줘."}
    ],
    "max_tokens": 1024,
    "stream": true
  }'

Check the answer and reasoning tokens at the 127.0.0.1:8000 endpoint.

Adjust context length and acceleration one at a time

If 24GB leaves little headroom, keep one request at a time and a short context. As you increase to the input length you need, check cache use and peak memory. A sudden speed drop is a reason to check for swapping or weights offloaded to the CPU first.

When supported MTP is enabled, check whether average accepted length and decode speed actually improve. Do not assume a speedup seen on one coding task also applies to prose. Compare cache precision and prefill chunk size separately as well.

If only the wait on long inputs is a problem, investigate input processing and caching before decode acceleration. Noting which stage is slow helps narrow the next test without piling on more settings.

Once it runs, try finishing your usual work

Do not stop at the binary-search test. Try the code or document you actually need to work on and check the result. A useful setup needs both an answer that meets your needs and a wait you can accept.

Save the model file, runtime, context length and acceleration settings that worked. Use the same conditions when considering more powerful hardware. If the current setup is already enough, deciding to keep it instead of buying a new GPU is a useful outcome too.

Once it runs, try finishing your usual work

Running an NVFP4 checkpoint

The Spark recipe uses Marlin for MoE and Triton for MTP. Selecting NVFP4 does not enable MTP automatically.

vLLM · nvidia/Qwen3.6-35B-A3B-NVFP4

Change log

These entries record changes to the site's guidance. They do not automatically check your installed engine or model version.

2026-09-06
Corrected model formats and runtime settings for Mac
Separated the Mac MLX path from GGUF Q4_K_M and aligned the engine name, model format and command. KV-cache precision and prefill batching that the command does not specify are now marked for checking in the model or runtime settings.
Also removed MLX speed profiles that implied a larger batch despite leaving the command unchanged.
2026-09-06
Separated uncached-input and cache-reuse records
Updated the checks to record first-time inputs separately from repeated inputs. Cache effects are not folded into uncached prefill throughput or hardware differences.
2026-09-06
Added saving and reopening hardware profiles
Runnable profiles can now be saved with their hardware selection in this browser, up to five entries, and reopened from the guide directory. A notice prompts you to recheck before running if the published settings have changed or the profile is no longer available.
2026-09-06
Added links to runtime explanations
Engine names in the settings now link to the corresponding runtime guide. Paths without an identified engine are not linked to a guessed program.