Executable programs and extensions
Last updated
Basalt at 665 tok/s? Qwen3.8 Flash-Next Setup and GPU Requirements
How can you run Qwen3.8 Flash-Next faster with Basalt?
Basalt is an inference1 engine for running Qwen3.8-Flash-Next on Linux x86-64 with a Blackwell (sm_120) GPU2. It takes a .basalt model pack and a prompt3, then returns a text response through an OpenAI- or Anthropic-compatible API4. Expert weights that do not fit in VRAM5 are read from system RAM6.
Connect a ready-made Qwen3.8-Flash-Next model pack to the Basalt server on a Linux PC and send a prompt; the answer comes back through a local API. This guide checks the requirements, downloads the IQ3_XXS pack, builds the server, and follows the steps through the first response.
A Strata Fork with a Tuned Token Drafter
Basalt is derived from Strata. Its IQ3_XXS, IQ3_S and Q2_0 packs include an MTP7 drafter additionally trained on Qwen3.8 Flash-Next's own output. The drafter proposes several upcoming tokens8 for the main model to verify; higher acceptance can improve generation speed.
Basalt is an engine derived from Strata that provides a server and compatible API for running this model. To use a chat interface, connect an OpenAI- or Anthropic-compatible client.
RAM and VRAM Hold Different Parts of the Expert Weights
Qwen3.8-Flash-Next is a mixture-of-experts (MoE9) model. For each input token, it uses selected experts rather than computing every expert. Basalt keeps frequently used experts in a GPU-memory (VRAM) cache and places expert weights that do not fit in VRAM in system RAM, retrieving them as needed. This lets the model run without loading the entire model file into VRAM, but the system needs enough RAM to hold the remaining weights.
The IQ3_XXS file is 74.4 GiB, including 40.0 GiB of expert weights. IQ3_S is 81.0 GiB with 46.8 GiB of experts. Reserve enough storage for the full file, then budget runtime10 memory for the GPU cache, experts remaining in RAM and the operating system.
The Dual-GPU Run Reached 354 tok/s for Prose and 665 for Structured Output
The figures below are single-request generation speeds for Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS. The reported results with one RTX 5090 were 316 tok/s for prose and 585 tok/s for structured output; adding an RTX 5060 Ti as a peer expert tier raised those figures to 354 and 665 tok/s.
| GPU setup | Prose generation | Structured output |
|---|---|---|
| RTX 5090 + RTX 5060 Ti | 354 tok/s | 665 tok/s |
| RTX 5090 | 316 tok/s | 585 tok/s |
The dual-GPU setup used a Ryzen 9 9950X, 32 GB RAM, and a 400 W GPU power limit. A separate concurrency result in the model card reports IQ3_S prose throughput11 of 338 tok/s for one request and 623 tok/s combined across eight requests. The 623 tok/s figure is aggregate throughput, not per-request speed, and uses a different quantized pack from the IQ3_XXS table. These speeds are measurements published by the developer.
Check Linux and GPU Requirements Before Installing
Building and serving Basalt requires Linux x86-64, an AVX2 CPU12, an NVIDIA Blackwell GPU with compute capability 12.0 (sm_120), CUDA13 13, CMake 3.24 or later, and a C++20 compiler. The Python server also requires Python 3.10 or later and the packages in the repository's requirements.txt.
System RAM holds expert weights that do not fit in VRAM. Check that enough RAM remains for the rest of the selected model pack and the operating system.
From the repository root, build the Release binaries and prepare the Python server environment. The build creates bin/basalt.
git clone https://github.com/jesdga95/basalt.git
cd basalt
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j
python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txtDownload the IQ3_XXS Pack into the Repository
Basalt can read a finished .basalt file from Hugging Face directly, so you do not need to convert the Qwen weights yourself. This walkthrough uses the IQ3_XXS pack. The file is 74.4 GiB, so make sure your storage has that much free space before downloading it.
From the repository root, create a models directory and run the command below. When it finishes, the file should appear there with the same name.
mkdir -p models
curl -fL --output models/Qwen3.8-FN-GSQ-RCO-IQ3_XXS.basalt https://huggingface.co/jesdga/Qwen3.8-Flash-Next-Basalt/resolve/main/Qwen3.8-FN-GSQ-RCO-IQ3_XXS.basaltStart the Server on Localhost First
Start with the server listening only on 127.0.0.1. If another computer needs to connect, enable authentication by passing --api-key when launching the server. The API key is a command-line-only option and cannot be stored in the config file.
Put the Model and Engine Options in One Configuration File
The model file does not contain engine options. The current repository README says the built-in defaults are a 4096-token context, fp1614 KV cache15, no speculative decoding16, and no expert cache, so configure these explicitly when you want the published setup. Put the model path and server port under server, and cache, prefill17, speculative decoding, and context length18 under engine.
Save the JSON below as basalt_config.json at the repository root. Begin with max-context 16384 and automatic expert-cache and prefill. spec: 7 proposes up to seven draft tokens; spec-min-p: 0.5 sets the draft probability threshold. Tune context and cache after confirming a response.
Check the applied settings in the console's [server] engine: line. Add resident-experts only when you have sufficient RAM for resident expert weights; otherwise start with the automatic cache. After confirming a response, adjust max-context and cache settings one at a time so you can track their effects.
{
"server": {
"model": "models/Qwen3.8-FN-GSQ-RCO-IQ3_XXS.basalt",
"alias": "Qwen 3.8 FN",
"gpu": "0",
"host": "127.0.0.1",
"port": 8080
},
"engine": {
"expert-cache": "auto",
"prefill": "auto",
"spec": 7,
"spec-min-p": 0.5,
"max-context": 16384,
"kv": "int8",
"vram-reserve-mib": 700
}
}bin/basalt serve --config basalt_config.json
Check the First Response and Common Startup Errors
From another terminal, check the /v1/models path for the model ID, then send a simple request and confirm that a response arrives. In this example the model ID is Qwen 3.8 FN. The server response contains model-generated text for your message. Open http://127.0.0.1:8080/ in a browser to see the statistics page with throughput and request logs.
The first call succeeded if Qwen 3.8 FN appears in the model list and the second request returns JSON with choices[0].message.content.
If the model list is empty, first check whether the model path in your config points to the actual file and whether its download has finished. For a CUDA error or a message saying no GPU was found, check that the driver, CUDA 13, and the card's compute capability meet the requirements. If the server starts but the model does not load, inspect the first console warning for RAM shortages, the VRAM reserve, and space taken by the KV cache; then try starting again with a shorter context length.
curl http://127.0.0.1:8080/v1/models
curl http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"Qwen 3.8 FN","messages":[{"role":"user","content":"Explain what a mixture-of-experts model is in one sentence."}],"max_tokens":80}'Supported Hardware and Related Ways to Run Models
If you have Linux, a supported GPU, and enough RAM for the model, start with the IQ3_XXS file and run the localhost API. After a successful request, check whether the context and cache remain stable for your workload, then adjust the settings.
Use the related guides below to compare other Strata-family setups and GPU configurations, find models for an RTX 5090, or review published speeds across inference engines.
Terminology notes
Inference — The process of using a trained model to compute an output for an input. Here, local inference means running the model on the user’s device.
Back to the textGPU — A processor designed to handle many calculations in parallel. It performs model computations during AI inference.
Back to the textPrompt — The input text sent to a model. For generative models it describes the desired result and strongly shapes the output.
Back to the textAPI — A defined interface that lets other code call a program’s functions.
Back to the textVRAM — Memory used by a graphics card’s GPU to store model weights and intermediate values.
Back to the textSystem RAM — System memory that temporarily holds data while programs run.
Back to the textMTP — A training objective or model component for predicting multiple future tokens at once. In speculative decoding, those predictions can be proposed as candidate tokens.
Back to the textToken — A unit into which a model divides input or output for processing.
Back to the textMoE — A model architecture that selects some of several expert subnetworks for each input. Total parameters can differ from the number activated for one token.
Back to the textRuntime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.
Back to the textThroughput — The amount of work processed or generated over time. Comparisons need the unit, such as tokens per second or requests per second.
Back to the textCPU — The central processor that runs the operating system and general-purpose instructions; in AI workloads it also handles tasks such as preprocessing and data movement.
Back to the textCUDA — A software platform for general-purpose computing on NVIDIA GPUs.
Back to the textFP16 — A 16-bit floating-point format. It uses the same bit width as BF16 but allocates bits differently between exponent and significand.
Back to the textKV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.
Back to the textSpeculative decoding — A generation method in which a draft model, MTP head, or repeated-context lookup proposes output candidates for the main model to verify and accept.
Back to the textPrefill — The stage where an LLM reads the input prompt and computes representations for its tokens. Longer prompts contain more tokens to process.
Back to the textContext window — The token span of input and generated content a model can handle in one request. The supported limit and memory use depend on the model and runtime settings.
Back to the text
Frequently asked questions
- What should you check first in this article?
- Basalt is an inference engine for running Qwen3.8-Flash-Next on Linux x86-64 with a Blackwell (sm_120) GPU. It takes a .basalt model pack and a prompt, then returns a text response through an OpenAI- or Anthropic-compatible API. Expert weights that do not fit in VRAM are read from system RAM.
- How much memory does it need?
- Qwen3.8-Flash-Next is a mixture-of-experts (MoE) model. For each input token, it uses selected experts rather than computing every expert. Basalt keeps frequently used experts in a GPU-memory (VRAM) cache and places expert weights that do not fit in VRAM in system RAM, retrieving them as needed. This lets the model run without loading the entire model file into VRAM, but the system needs enough RAM to hold the remaining weights.
- When is it better not to choose this?
- Basalt can read a finished .basalt file from Hugging Face directly, so you do not need to convert the Qwen weights yourself. This walkthrough uses the IQ3_XXS pack. The file is 74.4 GiB, so make sure your storage has that much free space before downloading it.