LocalAI

See the speed before choosing hardware

Model setup recipes

Qwen3.8-Flash-Next tuning: SGLang on a single DGX Spark

Reports of Flash-Next running quickly on a single Spark can make the same command tempting to try. But this setup takes more than downloading the model. It needs the specific compressed checkpoint, PLE SSD patch, MTP and backend settings together. Start by checking exactly which setup you are trying to reproduce.

Serving recipe

Settings for your hardware

For one user

Choose your hardware to see the launch command and settings you can adjust.

Available memory: 114GB / Bandwidth: 546GB/s

Profile

Apple Mac Studio M4 Max (128GB)

Memory fits

Recommended runtime: oMLX (4-bit MLX (a different format from GGUF Q4_K_M)) · Documented runtime settings

For your first run or reliable everyday use

Profile context length

8,192 tokens

Based on runtime documentationStudio M4 Max 128GB · Q4 · Request 1个

A starting setup based on the runtime documentation. Follow the steps below and compare speed as you make changes.

Estimated decode speed

24.7 ~ 48.2 tok/s

Estimate for one user

Estimated TTFT

8.8 ~ 17.1 s

Prefill: 479 ~ 936 tok/s

Estimated memory use

About 107.85GB / 114GB

Remaining margin: approx. 6.2GB

Server address

http://127.0.0.1:8000

Local only · 127.0.0.1

Estimates use the calculator’s recommended quantization and acceleration with 8,192tokens. They are not measurements of the configuration above.

Settings in this profile

ValueWhy this setting
Model weights4-bit MLX (a different format from GGUF Q4_K_M)Use an MLX model directory. This is a different file format from Q4_K_M GGUF.
Check input length8,192Check the input limit in model settings before sending a request. This launch command does not set the context length.
KV cacheCheck model settingsThis launch command does not set the KV cache precision.
Prefill batch sizeCheck runtime settingsDo not use llama.cpp's batch-size and ubatch-size values as substitutes for MLX settings.
Concurrent requests1Measure the speed of one request, separately from the combined throughput of multiple requests.
Memory headroom2 GB or moreAfter loading the model, check actual memory use and whether the system is swapping.

Studio M4 Max 128GB Run command (oMLX)

Apple Silicon · MLX Model · Memory Guard
MODEL_DIR="$HOME/.omlx/models"
omlx serve \
  --model-dir "$MODEL_DIR" \
  --host 127.0.0.1 \
  --port 8000 \
  --max-concurrent-requests 1 \
  --memory-guard balanced
Record a baseline with this setup, then follow the tuning steps below, changing one setting at a time.

Checks after startup

  1. 01Check the startup log for completed model loading and the actual context length.
  2. 02After warming up, send one request at a time. Use three different inputs of the same length to record fresh-input processing time.
  3. 03Record repeated-input results separately as cache-reuse runs. Do not combine them with fresh-input results.
  4. 04Record prefill tok/s, decode tok/s and peak memory together, then change one setting at a time.

Trade-offs

  • Conservative context and concurrency settings may produce less than the hardware's maximum throughput.

Things to watch for

  • It even resides in a built-in n-gram table, so context must be taken based on the total loaded memory shown.
See speed with recommended settings Studio M4 Max 128GB View price, power, and sales configuration

Adjust one item at a time

Speed tuning, step by step

If several values are changed at once, it is difficult to find the cause.

  1. STEP 1

    Save a baseline

    Record first-input and cache-reuse runs separately, three times each. Compare prefill, decode and peak memory together.

    When to stop: Do not move on if there are errors or less than 2GB of memory headroom.

  2. STEP 2

    Tune prefill chunk size

    Increase in steps of 1,024 → 2,048 → 4,096, comparing TTFT and peak memory on long inputs.

    When to stop: Revert to the previous value if TTFT does not improve or peak memory rises sharply.

  3. STEP 3

    KV Cache Tuning

    Only when you need more context, compare Q8 cache against the F16/BF16 baseline using the same question.

    When to stop: Keep the default precision if the output changes or you already have enough context.

  4. STEP 4

    MTP/speculative decoding

    Enable it only for supported models. Test code and prose separately, measuring acceptance rates and actual decode tok/s.

    When to stop: Turn it off if the median of three runs does not improve on the baseline.

  5. STEP 5

    Context expansion

    Double context length at each step until you reach what you need. Check retrieval from the middle of the input and whether swapping occurs.

    When to stop: Reduce by one step if retrieval accuracy drops or swapping or memory compression starts.

Submit a measurement from my hardware

Import a JSON file with at least three runs under the same conditions. The file stays in this browser until you submit it.

Include only hardware, model, runtime settings and measurements. Do not include prompts, responses, raw logs or file paths.

If you do not have a measurement file, use the tool with your running local server. It requires Node.js 20 or later and does not upload results automatically.

Read the measurement instructionsDownload the measurement toolDownload the hardware and settings template

These are reviewed community submissions, not measurements made by this site.

Report a setup issue

Tell us where this setup failed. Only the site administrator can read your report.

Setup being reported · Studio M4 Max 128GB · Qwen3.8-Flash-Next · Balanced

The same model name does not mean the same setup

This setup uses a specific NVFP4 checkpoint of Qwen3.8-Flash-Next. That does not mean every tensor is stored at 4-bit precision. Parts such as attention and MTP retain higher precision, so check the full file size and actual memory use after loading.

Within a single Spark's unified memory, moving weights to the CPU does not create more physical memory. The key is a dedicated mmap patch that keeps the large PLE table in an NVMe file and reads the needed portions. This is different from simply adding flags to stock SGLang.

Review the experimental files and scripts, including their versions, and prepare them separately from your working environment. Allow temporary space for downloads and conversion too. The setup below calls for roughly 140GB or more of free space, but check the total required by the files currently distributed.

The same model name does not mean the same setup
The same model name does not mean the same setup

A starting point for reproducing the 32K result

The current published script starts with TP 1, memory fraction 0.85 and prefill chunk size 2,048. It uses Triton for prefill and trtllm_mha for decode, with NEXTN MTP set to 3 steps, top-k 1 and 4 draft tokens. These defaults do not establish every condition used for the published measurements.

The published decode results distinguish English coding responses at about 41.5 tok/s from Spanish prose at about 22.8 tok/s. This does not establish the same speed for Korean answers. The previously cited 1,910 tok/s prefill figure has been removed from comparisons: the author reported contamination from input-cache reuse, and repeated cold-cache measurements are not yet complete. Do not use that figure as the input-processing speed for a new document.

Once preparation and verification pass, send one request at a time to establish a baseline. The server's maximum request limit is not the same as the number sent concurrently during a measurement. Also check that the Docker port is published only on 127.0.0.1.

Run the 32K configuration on one DGX Spark
git clone https://github.com/hashd1ve/qwen38-flash-next-one-dgx-spark.git
cd qwen38-flash-next-one-dgx-spark

./scripts/download.sh
./scripts/prepare.sh
sed -i.bak 's/-p "$PORT":30000/-p "127.0.0.1:$PORT:30000"/' scripts/serve.sh
MEMFRAC=0.85 PREFILL=2048 CTX=32768 ./scripts/serve.sh
python3 verify.py
Requires Docker and roughly 140GB or more of free NVMe space.
Key SGLang options in the published script
# Docker 포트는 127.0.0.1:30000에만 게시
--tp-size 1
--prefill-attention-backend triton
--decode-attention-backend trtllm_mha
--quantization modelopt_fp4
--ple-offload-embedding
--mamba-radix-cache-strategy extra_buffer
--mem-fraction-static 0.85
--chunked-prefill-size 2048
--max-running-requests 4
--speculative-algorithm NEXTN
--speculative-num-steps 3
--speculative-eagle-topk 1
--speculative-num-draft-tokens 4
--speculative-draft-model-quantization unquant
These flags alone will not reproduce the memory layout on stock SGLang without the PLE mmap patch.

Do not increase the context setting in isolation

For long contexts, memory fraction 0.79 and prefill chunk size 1,024 are starting values chosen to leave headroom. The published long-context measurement used memory fraction 0.85 and one request, so the command below does not reproduce that measurement exactly. Being able to set the context to 262,144 does not guarantee reliable processing of your documents.

First verify a short input, then increase in steps such as 8K, 32K and 128K. At each length, place a fact with a known answer in the middle and check that the model can retrieve it. Reduce other GPU work and measure memory use and time to first token with one request at a time.

Smaller chunks can reduce peak memory, but may also lower throughput on short inputs. There is no need to keep the largest context setting if you do not use it. Compare results at your usual input length to decide which profile to keep.

Run the 262K configuration on one DGX Spark
# 앞의 다운로드·준비 및 로컬 포트 설정을 마친 뒤 실행
# 서버 주소: http://127.0.0.1:30000
cd qwen38-flash-next-one-dgx-spark
sed -i.bak 's/--max-running-requests 4/--max-running-requests 1/' scripts/serve.sh
MEMFRAC=0.79 PREFILL=1024 CTX=262144 ./scripts/serve.sh
python3 verify.py
Reducing the prefill chunk size may also reduce throughput on short inputs.
Do not increase the context setting in isolation
Do not increase the context setting in isolation

What to check when published results are faster

First check the checkpoint, patch and container versions, and confirm that the PLE file is on NVMe. Then use the logs to verify the selected prefill and decode backends and MTP weight precision. Do not start by blindly increasing the draft steps.

Keep initial disk access separate from repeated access that benefits from the page cache. Results just after setup may differ from everyday use with several apps open. Record MTP off and on under matched conditions, testing code and prose separately.

If runs fail frequently, restore a stable baseline before tuning for speed. Restrict access if the port is exposed beyond the machine. Before turning it into a shared server, configure authentication and a firewall.

What do you want this setup to achieve?

Fitting a large model on one machine with a specialized patch is a worthwhile experiment. But reproducing a setup is not the same as having a tool that is convenient every day. Test it on your documents and code all the way to a completed task.

If one machine already handles your work reliably, there is less reason to buy a second. If you need longer contexts or multiple requests, compare a two-machine setup under those conditions. Base a purchase on repeatable results in your environment, not a published peak speed.

What do you want this setup to achieve?
What do you want this setup to achieve?

Running an NVFP4 checkpoint

File size depends on how PLE is stored. A single Spark needs a dedicated patched serving path.

vLLM / SGLang · nvidia/Qwen3.8-Flash-Next-NVFP4

Change log

These entries record changes to the site's guidance. They do not automatically check your installed engine or model version.

  1. Aligned Flash-Next article and profile settings

    The article now uses the published script's prefill chunk size of 2,048, and the long-context command limits requests to 1. The published measurements' language conditions are distinguished from conservative starting settings.

  2. Corrected the single-Spark Flash-Next prefill claim

    Removed the claim that 1,910 tok/s, which included cache reuse, represented prefill for an uncached input. The entry now calls for a new measurement without cache reuse.

  3. Adjusted the Flash-Next long-context request limit

    Reduced the running-request limit from four to one in the single-Spark 262K-context command. Added checks for incorrect output caused by older QSA patches.

  4. Corrected model formats and runtime settings for Mac

    Separated the Mac MLX path from GGUF Q4_K_M and aligned the engine name, model format and command. KV-cache precision and prefill batching that the command does not specify are now marked for checking in the model or runtime settings.

    Also removed MLX speed profiles that implied a larger batch despite leaving the command unchanged.

  5. Separated uncached-input and cache-reuse records

    Updated the checks to record first-time inputs separately from repeated inputs. Cache effects are not folded into uncached prefill throughput or hardware differences.

  6. Added saving and reopening hardware profiles

    Runnable profiles can now be saved with their hardware selection in this browser, up to five entries, and reopened from the guide directory. A notice prompts you to recheck before running if the published settings have changed or the profile is no longer available.

  7. Added links to runtime explanations

    Engine names in the settings now link to the corresponding runtime guide. Paths without an identified engine are not linked to a guessed program.