Qwen3.8-Flash-Next tuning: SGLang on a single DGX Spark

Hardware-specific speed tuning and serving recipe summary

Provides weights, KV cache, prefill batch, and concurrent request with acceleration functionality, separated for balanced performance, speed, and long-context profiling.

On the detail page, you can check the command per profile, applied value, stopping conditions, and measurement order. The execution command is displayed only for combinations of model file and execution program that have been verified.

The same model name does not mean the same setup

This setup uses a specific NVFP4 checkpoint of Qwen3.8-Flash-Next. That does not mean every tensor is stored at 4-bit precision. Parts such as attention and MTP retain higher precision, so check the full file size and actual memory use after loading.

Within a single Spark's unified memory, moving weights to the CPU does not create more physical memory. The key is a dedicated mmap patch that keeps the large PLE table in an NVMe file and reads the needed portions. This is different from simply adding flags to stock SGLang.

Review the experimental files and scripts, including their versions, and prepare them separately from your working environment. Allow temporary space for downloads and conversion too. The setup below calls for roughly 140GB or more of free space, but check the total required by the files currently distributed.

A starting point for reproducing the 32K result

The current published script starts with TP 1, memory fraction 0.85 and prefill chunk size 2,048. It uses Triton for prefill and trtllm_mha for decode, with NEXTN MTP set to 3 steps, top-k 1 and 4 draft tokens. These defaults do not establish every condition used for the published measurements.

The published decode results distinguish English coding responses at about 41.5 tok/s from Spanish prose at about 22.8 tok/s. This does not establish the same speed for Korean answers. The previously cited 1,910 tok/s prefill figure has been removed from comparisons: the author reported contamination from input-cache reuse, and repeated cold-cache measurements are not yet complete. Do not use that figure as the input-processing speed for a new document.

Once preparation and verification pass, send one request at a time to establish a baseline. The server's maximum request limit is not the same as the number sent concurrently during a measurement. Also check that the Docker port is published only on 127.0.0.1.

git clone https://github.com/hashd1ve/qwen38-flash-next-one-dgx-spark.git
cd qwen38-flash-next-one-dgx-spark

./scripts/download.sh
./scripts/prepare.sh
sed -i.bak 's/-p "$PORT":30000/-p "127.0.0.1:$PORT:30000"/' scripts/serve.sh
MEMFRAC=0.85 PREFILL=2048 CTX=32768 ./scripts/serve.sh
python3 verify.py

# Docker 포트는 127.0.0.1:30000에만 게시
--tp-size 1
--prefill-attention-backend triton
--decode-attention-backend trtllm_mha
--quantization modelopt_fp4
--ple-offload-embedding
--mamba-radix-cache-strategy extra_buffer
--mem-fraction-static 0.85
--chunked-prefill-size 2048
--max-running-requests 4
--speculative-algorithm NEXTN
--speculative-num-steps 3
--speculative-eagle-topk 1
--speculative-num-draft-tokens 4
--speculative-draft-model-quantization unquant

Do not increase the context setting in isolation

For long contexts, memory fraction 0.79 and prefill chunk size 1,024 are starting values chosen to leave headroom. The published long-context measurement used memory fraction 0.85 and one request, so the command below does not reproduce that measurement exactly. Being able to set the context to 262,144 does not guarantee reliable processing of your documents.

First verify a short input, then increase in steps such as 8K, 32K and 128K. At each length, place a fact with a known answer in the middle and check that the model can retrieve it. Reduce other GPU work and measure memory use and time to first token with one request at a time.

Smaller chunks can reduce peak memory, but may also lower throughput on short inputs. There is no need to keep the largest context setting if you do not use it. Compare results at your usual input length to decide which profile to keep.

# 앞의 다운로드·준비 및 로컬 포트 설정을 마친 뒤 실행
# 서버 주소: http://127.0.0.1:30000
cd qwen38-flash-next-one-dgx-spark
sed -i.bak 's/--max-running-requests 4/--max-running-requests 1/' scripts/serve.sh
MEMFRAC=0.79 PREFILL=1024 CTX=262144 ./scripts/serve.sh
python3 verify.py

What to check when published results are faster

First check the checkpoint, patch and container versions, and confirm that the PLE file is on NVMe. Then use the logs to verify the selected prefill and decode backends and MTP weight precision. Do not start by blindly increasing the draft steps.

Keep initial disk access separate from repeated access that benefits from the page cache. Results just after setup may differ from everyday use with several apps open. Record MTP off and on under matched conditions, testing code and prose separately.

If runs fail frequently, restore a stable baseline before tuning for speed. Restrict access if the port is exposed beyond the machine. Before turning it into a shared server, configure authentication and a firewall.

What do you want this setup to achieve?

Fitting a large model on one machine with a specialized patch is a worthwhile experiment. But reproducing a setup is not the same as having a tool that is convenient every day. Test it on your documents and code all the way to a completed task.

If one machine already handles your work reliably, there is less reason to buy a second. If you need longer contexts or multiple requests, compare a two-machine setup under those conditions. Base a purchase on repeatable results in your environment, not a published peak speed.