Local AI Guide

See the speed before choosing hardware

Executable programs and extensions

One or two DGX Sparks for Qwen3.8 Flash Next?

The one-Spark setup measured 64.4 tok/s for one request and 200.9 combined at eight; two Sparks with NVFP4 measured 67.4 for one request and 390.1 combined at sixteen.

The one-Spark recipe uses TensorFold's Zig engine and an INT4-AutoRound checkpoint1. At its default 262,144-token window, it measured 64.4 tok/s for one request and 200.9 combined at eight. The two-Spark recipe links two ranks over ConnectX-7, offers either NVFP42 or INT4-AutoRound, and uses a 1,048,576-token window. NVFP4 measured 67.4 tok/s for one request and 390.1 combined at sixteen. Checkpoint, context, and parallelism also change with machine count, so compare the results under their recipe conditions.

Compare two ways to send prompts3 to Qwen3.8 Flash-Next on DGX Spark. The single-Spark recipe uses INT4-AutoRound; two Sparks connect through ConnectX-7 and offer NVFP4 or INT4-AutoRound. Read single-request speed separately from aggregate throughput4, then follow the server setup.

Contents
Requirements and key details
  • One Spark: INT4-AR, 262K, 64.4 tok/s single and 200.9 combined at eight requests.
  • Two Sparks with NVFP4: 1M, 67.4 tok/s single and 390.1 combined at sixteen requests.
  • Two-Spark INT4-AutoRound reached 89.3 tok/s for one request, with a different checkpoint and quantization.

The one- and two-Spark recipes use different setups

The one-Spark recipe serves Qwen3.8 Flash Next with TensorFold's Zig engine. Its defaults are an INT4-AutoRound checkpoint, FP85 KV cache6, image input, context 262,144, and parallel 8. The two-Spark recipe assigns one Spark to each rank and connects them over ConnectX-7, offering a larger context window7 and two checkpoint choices.

These Spark figures come from MIA's recipe-specific TensorFold Zig builds and settings. As with other MIA recipes that patch TensorFold 0.6.x, do not transfer these results to TensorFold 1.0.8's native qualification table. Model, branch, or patch changes alter the conditions even on the same GPU8.

These are separate recipes. Do not read the one-Spark result as a two-Spark test with the same checkpoint and context; distinguish the effect of the extra machine from quantization9 differences.

A dark monitor, a compact desktop computer, and a PC case with several visible fans
Start by comparing a one-machine and two-machine server setup.

Single-request speed and aggregate speed change with request count

The table reports prose decode10. It separates the single-request result from the aggregate across concurrent requests. More simultaneous requests can raise total throughput while reducing tokens11 per request and increasing first-token delay.

Scroll sideways to compare all columns.

Default prose decode comparison
SetupOne requestConcurrent aggregateMeasured parallelism
1 Spark · INT4-AR64.4 tok/s200.9 tok/s8
2 Sparks · NVFP467.4 tok/s390.1 tok/s16
2 Sparks · INT4-AR89.3 tok/s483.6 tok/s16

For personal use, compare the one-request column. For a shared server, also read aggregate throughput and the measured request count. The two-Spark NVFP4 and INT4-AutoRound rows use different checkpoints, so retain that distinction when choosing a setup.

Two compact desktop computers connected to a monitor showing streams of particles
The two Spark ranks use a dedicated link and network configuration.

Install the one-Spark recipe and check the first response

For the one-Spark setup, prepare a DGX Spark with NVIDIA GPU drivers, Docker and the NVIDIA Container Toolkit, and about 160 GB of free disk. The README's start.sh fetches the container and checkpoint and prepares the server with defaults.

Clone and start the single-Spark recipe
git clone https://github.com/MiaAI-Lab/Qwen3.8-Flash-Next-Single-DGX-Spark-TensorFold.git
cd Qwen3.8-Flash-Next-Single-DGX-Spark-TensorFold
./start.sh

Follow the repository's current Quick start procedure. When setup completes, the server reports LIVE on port 8888. Set the API12 base to the head machine's address followed by /v1 and use model ID Qwen3.8-Flash-Next. After curl receives a reply, enter the same address and model in your client.

Check the API and send a first request on the head
curl -s http://127.0.0.1:8888/v1/models
curl -s http://127.0.0.1:8888/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"Qwen3.8-Flash-Next","messages":[{"role":"user","content":"Write a Python fibonacci function."}],"max_tokens":1000}'

For the two-device recipe, first create scripts/local.sh with the command below. The next settings block belongs in that file: edit its WORKER and FABRIC_PEER entries with a file editor and save, rather than running the block in the terminal. WORKER is the management-LAN SSH target; FABRIC_PEER is the worker's CX7 address when needed. Replace the reserved 192.0.2.x examples with actual addresses. Both devices need Docker and NVIDIA Container Toolkit, and key-based SSH from the head. The default copy mode stores the checkpoint on both devices, so allow disk space on each.

Configure the worker and start the two-Spark recipe
git clone https://github.com/MiaAI-Lab/Qwen3.8-Flash-Dual-DGX-Sparks-TensorFold.git
cd Qwen3.8-Flash-Dual-DGX-Sparks-TensorFold
cp scripts/local.sh.example scripts/local.sh
Save these settings in scripts/local.sh using a file editor
# Replace both reserved documentation addresses with your actual values.
WORKER=user@192.0.2.2       # worker SSH target on the management LAN
FABRIC_PEER=192.0.2.12      # worker CX7 address when needed
Check the saved settings on the head
grep -nE 'WORKER=|FABRIC_PEER=' scripts/local.sh
Verify worker SSH from the head
ssh-copy-id user@192.0.2.2
ssh -o BatchMode=yes user@192.0.2.2 true
Start the two-Spark server
./start.sh
Check the model list and first response after LIVE
curl -fsS http://127.0.0.1:8888/v1/models
curl -fsS http://127.0.0.1:8888/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"Qwen3.8-Flash-Next","messages":[{"role":"user","content":"Write a Python fibonacci function."}],"max_tokens":1000}'

Choose after matching prefill and context too

The two-Spark recipe offers up to a 1M context and defaults to parallel 16, but that is not a promise that every request can fill both limits at once. Each Spark needs about 102 GiB of free memory, and concurrent requests share the KV pool. Choose based on whether you need both 1M context and sixteen concurrent requests, or mostly shorter one- or two-request sessions.

Prefill13 is the stage that reads the prompt before producing the first token. The one-Spark result was 2,630 tok/s at a 32K prompt; the two-Spark results near that length were about 2,575 tok/s for NVFP4 and 3,285 tok/s for INT4-AR. First-token delay also depends on prompt length, so evaluate it separately from decode speed.

Printed reports and an open notebook around a laptop displaying documents and charts
Check concurrency and prompt length against your workload.

Terminology notes

  1. Checkpoint — A file containing saved model weights and related state. Versions or tasks in one model family may use different checkpoints.

    Back to the text

  2. NVFP4 — A 4-bit floating-point data format defined by NVIDIA. Support depends on GPU generation, model, and software implementation.

    Back to the text

  3. Prompt — The input text sent to a model. For generative models it describes the desired result and strongly shapes the output.

    Back to the text

  4. Throughput — The amount of work processed or generated over time. Comparisons need the unit, such as tokens per second or requests per second.

    Back to the text

  5. FP8 — A floating-point format that represents each value with 8 bits.

    Back to the text

  6. KV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.

    Back to the text

  7. Context window — The token span of input and generated content a model can handle in one request. The supported limit and memory use depend on the model and runtime settings.

    Back to the text

  8. GPU — A processor designed to handle many calculations in parallel. It performs model computations during AI inference.

    Back to the text

  9. Quantization — A way to represent model weights or values with fewer bits, reducing storage and memory use. The coarser representation can also affect accuracy.

    Back to the text

  10. Token decoding — The stage that generates successive output tokens using the input and the response produced so far.

    Back to the text

  11. Token — A unit into which a model divides input or output for processing.

    Back to the text

  12. API — A defined interface that lets other code call a program’s functions.

    Back to the text

  13. Prefill — The stage where an LLM reads the input prompt and computes representations for its tokens. Longer prompts contain more tokens to process.

    Back to the text

Frequently asked questions

What should you check first in this article?
The one-Spark recipe uses TensorFold's Zig engine and an INT4-AutoRound checkpoint. At its default 262,144-token window, it measured 64.4 tok/s for one request and 200.9 combined at eight. The two-Spark recipe links two ranks over ConnectX-7, offers either NVFP4 or INT4-AutoRound, and uses a 1,048,576-token window. NVFP4 measured 67.4 tok/s for one request and 390.1 combined at sixteen. Checkpoint, context, and parallelism also change with machine count, so compare the results under their recipe conditions.
How much memory does it need?
The two-Spark recipe offers up to a 1M context and defaults to parallel 16, but that is not a promise that every request can fill both limits at once. Each Spark needs about 102 GiB of free memory, and concurrent requests share the KV pool. Choose based on whether you need both 1M context and sixteen concurrent requests, or mostly shorter one- or two-request sessions.