Qwen3.8-Flash-Next: a single DGX Spark SGLang recipe

Hardware-specific speed tuning and serving recipe summary

Provides weights, KV cache, prefill batch, and concurrent request with acceleration functionality, separated for balanced performance, speed, and long-context profiling.

On the detail page, you can check the command per profile, applied value, stopping conditions, and measurement order. The execution command is displayed only for combinations of model file and execution program that have been verified.

The same model name alone does not mean the same configuration

This path uses a specific NVFP4 checkpoint in Qwen3.8-Flash-Next. This doesn't mean that all tensors are 4 bits in batches. There are parts that are maintained with higher precision, such as attention and MTP, so you need to check the entire file and the actual load.

Within a single Spark's unified memory, simply moving the weight toward the CPU does not increase physical space. The key is a dedicated mmap patch that places a large PLE table in an NVMe file and looks it up as needed. Distinguish this from a path that just adds flags to the default SGLang.

Experiment files and scripts are prepared separately from the normal environment by reviewing their versions and contents. Temporary space for downloading and conversion is also required. Please refer to the instructions for freeing up approximately 140GB or more in the executable bundle below, but double-check the total space required for the current distribution file.

The starting point for reproducing 32K records

The reported 32K configuration uses TP 1, a memory fraction of 0.85 and a prefill chunk size of 4,096. It uses Triton for prefill and trtllm_mha for decode. MTP uses NEXTN with 3 steps, top-k 1 and 4 draft tokens.

Community reported values are divided into a median code decode of approximately 41.5 tok/s and a general sentence approximately 22.8 tok/s. This is not universal performance measured directly by the site. The prefill of approximately 1,910 tok/s must also check the input and cache conditions, and should not be read as a promise that applies to other questions as well.

After passing the preparation script and verification, actual requests are sent one by one to leave a standard. The server's maximum request tolerance and the number of concurrently sent requests during measurement are different. Also make sure that ports outside of Docker are published only on 127.0.0.1.

git clone https://github.com/hashd1ve/qwen38-flash-next-one-dgx-spark.git
cd qwen38-flash-next-one-dgx-spark

./scripts/download.sh
./scripts/prepare.sh
sed -i.bak 's/-p "$PORT":30000/-p "127.0.0.1:$PORT:30000"/' scripts/serve.sh
MEMFRAC=0.85 PREFILL=4096 CTX=32768 ./scripts/serve.sh
python3 verify.py

# Docker 포트는 127.0.0.1:30000에만 게시
--tp-size 1
--prefill-attention-backend triton
--decode-attention-backend trtllm_mha
--quantization modelopt_fp4
--ple-offload-embedding
--mamba-radix-cache-strategy extra_buffer
--mem-fraction-static 0.85
--chunked-prefill-size 4096
--max-running-requests 4
--speculative-algorithm NEXTN
--speculative-num-steps 3
--speculative-eagle-topk 1
--speculative-num-draft-tokens 4
--speculative-draft-model-quantization unquant

Do not increase the context setting in isolation

On long context paths, we lower the memory ratio to 0.79 and reduce the prefill chunks to 1,024 to leave temporary space. The fact that a setting of 262,144 is acceptable and the fact that my documents of that length will be processed reliably are different.

First, check normal operation at small inputs and then gradually increase to 8K, 32K, and 128K. In each length, insert a sentence in the middle where you know the answer to see if you actually find it. Reduce other GPU activity and view memory and first token time together on a single request basis.

Lowering memory peaks by reducing chunks can result in lower throughput for short inputs. There is no reason to keep the largest setting unless you need a long context. Come back with the results at your usual length and decide which profile to leave behind.

# 서버 주소: http://127.0.0.1:30000
cd qwen38-flash-next-one-dgx-spark
MEMFRAC=0.79 PREFILL=1024 CTX=262144 ./scripts/serve.sh
python3 verify.py

What to check when your numbers are slower than others

Check the checkpoint and patch/container version first, and make sure the PLE file is in the actual NVMe path. Next, check the log to see if the precision of the prefill/decode backend and MTP weights were selected as intended. Don’t just increase it from the candidate stage.

A distinction must also be made between initial disk accesses and repeated accesses with page cache remaining. The number immediately after preparation and the number with multiple apps turned on may be different. The values ​​for turning MTP off and turning it on are recorded under the same conditions in each code and general sentence.

If execution fails frequently, recover a stable baseline before speed. If the port is open to the outside, the access range must be restricted, and when changing to a shared server, separate authentication and firewall must be prepared.

What results do you want to achieve with this recipe?

Putting together a large model with a special patch is a meaningful experiment. However, the achievement of recreating a setup and the result of obtaining a tool you can comfortably use every day can be different. Get the documentation and code you need right through to the end.

If you already have one reliable setup that does what you want, there's less reason to buy two. If you absolutely need a longer context or multiple requests, compare the two on those terms. Use repeatable results in your environment rather than recorded top speed to make purchasing decisions.