What should you check first in this article?

Qwen3.6-35B-A3B is an MoE model that takes text input and generates answers. On one DGX Spark, use the NVIDIA ARM64/Blackwell vLLM container together with the official NVFP4 checkpoint. This guide loads the model, verifies a first response through the loopback API, and then adjusts memory and context options.

Model setup recipes

Qwen3.6 35B-A3B setup: NVFP4 on Mac, RTX, and DGX Spark

Run Qwen3.6 35B-A3B with vLLM on DGX Spark

Qwen3.6-35B-A3B is an MoE¹ model that takes text input and generates answers. On one DGX Spark, use the NVIDIA ARM64/Blackwell vLLM container together with the official NVFP4² checkpoint³. This guide loads the model, verifies a first response through the loopback API⁴, and then adjusts memory and context options.

Match the NVFP4 weight format, container version, and model ID. NVFP4 is a weight format, while FP8⁵ KV cache⁶ is a separate setting for storing context state. First verify the endpoint with TP1, an 8K context, and one request; then test longer context, concurrency, or MTP⁷ one at a time.

Jump to hardware-specific settings

Separate speed estimates from server measurements

The speed pages for the three systems below are calculated estimates for a 4,096-token input and one user, not measurements from a running server. Keep them separate from results you record after deployment.

For Mac mini M4 32 GB with Q4_K_M and no acceleration, the explorer shows generation at 7.8–22.5 tok/s, first token⁸ at 5.5–15.7 seconds, input processing at 263–757 tok/s, 21.3 GB needed, and 27 GB available. For DGX Spark 128 GB with NVFP4 and MTP, it shows 61.2–111.1 tok/s, 1.5–3.1 seconds, 1,341–2,661 tok/s, 24.3 GB needed, and 116 GB available.

For two DGX Spark systems with 256 GB, Q4_K_M, and no acceleration, the estimated values are 17–61.4 tok/s generation, 0.91–2.4 seconds to first token, 1,763–4,563 tok/s input processing, 21.3 GB needed, and 232 GB available. The combined memory can be used to split a model, but answer speed does not automatically double; networking and distributed configuration are separate requirements.

A scene showing the Qwen3.6 35B-A3B MoE model and a 4K input setting — Separate calculated estimates for 4K single-request speed from server tests.

Match the container and model as one configuration

On DGX Spark, use the NVIDIA vLLM recipe for Spark rather than a container intended for image generation or a different architecture. This guide covers one GB10 node. A recipe that configures Ray or tensor parallel⁹ across multiple systems is a separate deployment with different networking and distributed processes.

The single-Spark NVFP4 profile specifies the vLLM v0.28.0 container and the nvidia/Qwen3.6-35B-A3B-NVFP4 repository. NVIDIA's recipe also has profiles for RTX 5090 and RTX PRO 6000, but do not copy this GB10 configuration to those cards. Check device-specific kernels, memory settings, and model-file access in the matching path.

NVFP4 is the precision used for model weights, while --kv-cache-dtype fp8 sets the format for request-context state. Do not add FP8 KV cache, MTP, and context options all at once; first verify that the official single-Spark configuration starts an endpoint.

Open a loopback endpoint and verify the first response

Start the basic check with container v0.28.0, TP1, an 8K context, and one concurrent request. Keep FlashInfer, Marlin, FP8 KV, and memory utilization 0.5 from the official GB10 path; this profile lowers only context and request count. Even with host networking, bind the server to 127.0.0.1 for local access only.

After logs show that weights are loaded and the engine is ready, request /v1/models. If the list contains qwen3.6-35b-a3b, use that name in the chat completion model field. A response in choices[0].message.content confirms the API connection. This short function explanation is a smoke test, not a benchmark.

Minimal Spark NVFP4 vLLM endpoint

docker run --rm --gpus all --ipc=host --network host -v "$HOME/.cache/huggingface:/root/.cache/huggingface" vllm/vllm-openai:v0.28.0 nvidia/Qwen3.6-35B-A3B-NVFP4 --served-model-name qwen3.6-35b-a3b --host 127.0.0.1 --port 8000 --tensor-parallel-size 1 --trust-remote-code --kv-cache-dtype fp8 --attention-backend flashinfer --moe-backend marlin --gpu-memory-utilization 0.5 --max-model-len 8192 --max-num-seqs 1

Keep the official Spark profile's kernels and FP8 KV. The container runs vllm serve, so place the model ID after the image name.

Check the model list and send a short request

curl --fail-with-body http://127.0.0.1:8000/v1/models
curl --fail-with-body http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen3.6-35b-a3b","messages":[{"role":"user","content":"Explain this function's empty-input behavior in one sentence: def first_item(items): return items[0]"}],"temperature":0,"max_tokens":64}'

Both requests use the loopback address. Use the returned answer only to verify connectivity.

Change options according to where a problem occurs

After the server responds, adjust the existing recipe's FP8 KV, context limit, and concurrent-sequence settings as needed. Long input increases per-request cache use, and concurrent requests need multiple caches; test one value at a time rather than raising both together.

If the container does not start, inspect logs for the image tag, model ID and file access, and architecture support. If the model loads but a request hits a memory error, lower either context or concurrent sequences. First distinguish startup failures from request-time failures rather than adding options absent from the playbook.

Do not combine a Ray setup that splits a model across two Sparks with the single-node command. If the goal is personal responsiveness, first check whether answer time and memory on one Spark are sufficient. Multiple Sparks are a separate operational choice requiring networking and distributed execution.

Evaluate results with the same real task

After the short function check, send a task you actually do. For code changes, include input/output requirements and tests; for document questions, ask about a verifiable date or detail. Record the launch command, container tag, model revision, prompt¹⁰, and output limit so you can compare option changes.

For your own measurements, exclude warm-up and record end-to-end time for repeated requests, GPU¹¹ memory, and answer accuracy. Official support information and calculated speed estimates are different from runtime¹² measurements. If using the Qwen3 reasoning parser, check that vLLM 0.19.0 or later actually supports the parser and model; the version number alone does not establish this.

A published oMLX 0.7.0 result on M5 Ultra is a reference from a separate platform. On M5 Ultra 256 GB/80-core GPU, the Qwen3.6-35B-A3B 4-bit code_python test used a 128-token output budget and temperature 0: 4K prefill¹³ was 9,126 tok/s and decode¹⁴ 158.3 tok/s. Version 0.7.0rc1 showed 7,931 tok/s and 158.2 tok/s respectively; for this item, prefill rose while decode was nearly unchanged. This table alone does not show that MTP accelerates Qwen3.6 decode.

The binary-search request below uses a generic model ID. If you started the server with the Spark command above, change the request's model value to qwen3.6-35b-a3b. For another server, read its actual name from /v1/models. With LM Studio, use port 1234 and the identifier of the model loaded in the app.

Check the reasoning-response streaming API

curl -N http://127.0.0.1:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "Qwen/Qwen3.6-35B-A3B",
    "messages": [
      {"role": "user", "content": "파이썬으로 이진 탐색 함수를 작성하고 시간 복잡도를 설명해줘."}
    ],
    "max_tokens": 1024,
    "stream": true
  }'

Send a request to vLLM port 8000 and inspect how reasoning and the final answer arrive in the stream.

oMLX 0.7.0 Qwen3.6 result and conditions

A scene representing completion of a real task after verifying the run — Check not only connectivity but whether your regular code or document task completes.

Increase context and concurrency one at a time

For a Q4 conversion on a 24 GB desktop, around 20 GB is a starting estimate for the weight file. Check the actual download size and leave space for KV cache and runtime. This uses a different file format and runtime from the Spark NVFP4 path explained here.

If 24 GB leaves little headroom, first verify a baseline response with one request and short context. Increase input length in stages, checking cache and peak memory each time; record the server limit separately from actual input tokens. System instructions and prior conversation can also count toward input length.

After adding MTP, check whether acceptance length and decode time after the first token change for the same code request. Do not generalize code-generation results to general writing. Avoid changing cache precision, prefill chunk, and MTP at once; test the interval that was actually slow first.

If a short baseline works but long context causes OOM or swapping, first lower context or concurrent sequences. If the target itself fails to load or even short requests remain memory-bound, revisit the weight file and runtime memory requirements. Consider larger memory only in response to a confirmed bottleneck.

A scene about inspecting memory after loading the model file on a 24 GB system — After loading the model, check headroom for KV cache and real requests.

Recheck versions in the official runtime references

The references below document the single-Spark NVFP4 container and official vLLM checkpoint paths. Before running commands, recheck the current image tag, model revision, and supported options.

For LM Studio, download the Q4 file and confirm it with lms ls, then select it in the prompt opened by lms load. Its API alias is qwen36-local. Connect LM Studio using port 1234 and this alias; keep the earlier Spark vLLM requests on port 8000 with their existing server alias.

Official checkpoint with vLLM 0.19.0+ (multi-GPU)

GPU_COUNT=8 # The official example uses 8 GPUs; change this to match your system
vllm serve Qwen/Qwen3.6-35B-A3B   --host 127.0.0.1   --port 8000   --tensor-parallel-size "$GPU_COUNT"   --max-model-len 8192   --reasoning-parser qwen3   --gpu-memory-utilization 0.92

This official server example sets GPU count and model path for the deployment system.

Desktop serving with LM Studio (Q4 GGUF)

lms ls
lms load --gpu=max --context-length=8192 --identifier qwen36-local
lms server start --port 1234

An example of serving a Q4 file through a local daemon on a 32 GB+ Mac or high-memory GPU.

Terminology notes

MoE — A model architecture that selects some of several expert subnetworks for each input. Total parameters can differ from the number activated for one token.
Back to the text
NVFP4 — A 4-bit floating-point data format defined by NVIDIA. Support depends on GPU generation, model, and software implementation.
Back to the text
Checkpoint — A file containing saved model weights and related state. Versions or tasks in one model family may use different checkpoints.
Back to the text
API — A defined interface that lets other code call a program’s functions.
Back to the text
FP8 — A floating-point format that represents each value with 8 bits.
Back to the text
KV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.
Back to the text
MTP — A training objective or model component for predicting multiple future tokens at once. In speculative decoding, those predictions can be proposed as candidate tokens.
Back to the text
Token — A unit into which a model divides input or output for processing.
Back to the text
Tensor parallelism — A method that splits matrices and computations within a model layer across devices for joint execution. Devices communicate to combine partial results.
Back to the text
Prompt — The input text sent to a model. For generative models it describes the desired result and strongly shapes the output.
Back to the text
GPU — A processor designed to handle many calculations in parallel. It performs model computations during AI inference.
Back to the text
Runtime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.
Back to the text
Prefill — The stage where an LLM reads the input prompt and computes representations for its tokens. Longer prompts contain more tokens to process.
Back to the text
Token decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the text

Frequently asked questions

What should you check first in this article?: Qwen3.6-35B-A3B is an MoE model that takes text input and generates answers. On one DGX Spark, use the NVIDIA ARM64/Blackwell vLLM container together with the official NVFP4 checkpoint. This guide loads the model, verifies a first response through the loopback API, and then adjusts memory and context options.
How much memory does it need?: For Mac mini M4 32 GB with Q4_K_M and no acceleration, the explorer shows generation at 7.8–22.5 tok/s, first token at 5.5–15.7 seconds, input processing at 263–757 tok/s, 21.3 GB needed, and 27 GB available. For DGX Spark 128 GB with NVFP4 and MTP, it shows 61.2–111.1 tok/s, 1.5–3.1 seconds, 1,341–2,661 tok/s, 24.3 GB needed, and 116 GB available.

Serving recipe

Settings for your hardware

For one user

Choose your hardware to see the launch command and settings you can adjust.

Choose your hardware

Available memory: 56GB / Bandwidth: 307GB/s

Profile

Apple MacBook Pro M5 Pro (64GB)

Memory fits

Recommended runtime: oMLX (4-bit MLX (a different format from GGUF Q4_K_M)) · Documented runtime settings

For your first run or reliable everyday use

Profile context length

16,384 tokens

Based on runtime documentationMBP M5 Pro 64GB · Q4 · 1 request

A starting setup based on the runtime documentation. Follow the steps below and compare speed as you make changes.

Estimated decode speed

22.6 ~ 54.1 tok/s

Estimate for one user

Estimated TTFT

8.9 ~ 21.2 s

Prefill: 774 ~ 1,854 tok/s

Estimated memory use

About 21.91GB / 56GB

Remaining margin: approx. 34.1GB

Server address

http://127.0.0.1:8000

Local only · 127.0.0.1

Estimates use the calculator’s recommended quantization and acceleration with 16,384tokens. They are not measurements of the configuration above.

Settings in this profile

Value	Why this setting
Weights4-bit MLX (a different format from GGUF Q4_K_M)	Use an MLX model directory. This is a different file format from Q4_K_M GGUF.
Check input length16,384	Check the input limit in model settings before sending a request. This launch command does not set the context length.
KV cacheCheck model settings	This launch command does not set the KV cache precision.
Prefill batch sizeCheck runtime settings	Do not use llama.cpp's batch-size and ubatch-size values as substitutes for MLX settings.
Concurrent requests1	Measure the speed of one request, separately from the combined throughput of multiple requests.
Memory headroom2 GB or more	After loading the model, check actual memory use and whether the system is swapping.

MBP M5 Pro 64GB Run command (oMLX)

Apple Silicon · MLX Model · Memory Guard

MODEL_DIR="$HOME/.omlx/models"
omlx serve \
  --model-dir "$MODEL_DIR" \
  --host 127.0.0.1 \
  --port 8000 \
  --max-concurrent-requests 1 \
  --memory-guard balanced

Record a baseline with this setup, then follow the tuning steps below, changing one setting at a time.

Checks after startup

01Check the startup log for completed model loading and the actual context length.
02After warming up, send one request at a time. Use three different inputs of the same length to record fresh-input processing time.
03Record repeated-input results separately as cache-reuse runs. Do not combine them with fresh-input results.
04Record prefill tok/s, decode tok/s and peak memory together, then change one setting at a time.

Trade-offs

Conservative context and concurrency settings may produce less than the hardware's maximum throughput.

See speed with recommended settings MBP M5 Pro 64GB View price, power, and sales configuration

Adjust one item at a time

Speed tuning, step by step

If several values are changed at once, it is difficult to find the cause.

STEP 1
Save a baseline
Record first-input and cache-reuse runs separately, three times each. Compare prefill, decode and peak memory together.
When to stop: Do not move on if there are errors or less than 2GB of memory headroom.
STEP 2
Tune prefill chunk size
Increase in steps of 1,024 → 2,048 → 4,096, comparing TTFT and peak memory on long inputs.
When to stop: Revert to the previous value if TTFT does not improve or peak memory rises sharply.
STEP 3
KV Cache Tuning
Only when you need more context, compare Q8 cache against the F16/BF16 baseline using the same question.
When to stop: Keep the default precision if the output changes or you already have enough context.
STEP 4
MTP/speculative decoding
Enable it only for supported models. Test code and prose separately, measuring acceptance rates and actual decode tok/s.
When to stop: Turn it off if the median of three runs does not improve on the baseline.
STEP 5
Context expansion
Double context length at each step until you reach what you need. Check retrieval from the middle of the input and whether swapping occurs.
When to stop: Reduce by one step if retrieval accuracy drops or swapping or memory compression starts.

Submit a measurement from my hardware

Import a JSON file with at least three runs under the same conditions. The file stays in this browser until you submit it.

Include only hardware, model, runtime settings and measurements. Do not include prompts, responses, raw logs or file paths.

If you do not have a measurement file, use the tool with your running local server. It requires Node.js 20 or later and does not upload results automatically.

Read the measurement instructions Download the measurement tool Download the hardware and settings template

Import measurement JSON (up to 32KB)

These are reviewed community submissions, not measurements made by this site.

Report a setup issue

Tell us where this setup failed. Only the site administrator can read your report.

Setup being reported · MBP M5 Pro 64GB · Qwen3.6 35B-A3B (MoE) · Balanced

Change log

These entries record changes to the site's guidance. They do not automatically check your installed engine or model version.

2026-09-06
Corrected model formats and runtime settings for Mac
Separated the Mac MLX path from GGUF Q4_K_M and aligned the engine name, model format and command. KV-cache precision and prefill batching that the command does not specify are now marked for checking in the model or runtime settings.
Also removed MLX speed profiles that implied a larger batch despite leaving the command unchanged.
2026-09-06
Separated uncached-input and cache-reuse records
Updated the checks to record first-time inputs separately from repeated inputs. Cache effects are not folded into uncached prefill throughput or hardware differences.
2026-09-06
Added saving and reopening hardware profiles
Runnable profiles can now be saved with their hardware selection in this browser, up to five entries, and reopened from the guide directory. A notice prompts you to recheck before running if the published settings have changed or the profile is no longer available.
2026-09-06
Added links to runtime explanations
Engine names in the settings now link to the corresponding runtime guide. Paths without an identified engine are not linked to a guessed program.