Executable programs and extensions
Last updated
Run Qwen3.8 Locally with NInfer: RTX 5090, MTP, and DFlash2
Download an 11.18 GB NInfer artifact and serve prompts on an RTX 5090.
NInfer is a CUDA1 inference2 engine that runs .ninfer model files on NVIDIA GPUs and returns answers through an OpenAI-compatible API3. Run Underdog Saluki in a Docker server, send a short query, and compare MTP4 and DFlash2 draft generation using the same request.
NInfer serves model files through an API
NInfer is an inference engine that computes NInfer-format model files on an NVIDIA GPU5 and returns replies to chat requests sent to its server. Its inputs are a model artifact and a text request; the output is a reply through an OpenAI-compatible API. We will serve Underdog Saluki, a low-bit Qwen3.8-27B derivative, on one RTX GPU and send it a short prompt6.
The converted file preserves the source GGUF7 tensor blocks and also contains an MTP head and DFlash2 adapter. It uses a mixed-precision layout of about 2 bits per weight and is 11,180,119,296 bytes (10.41 GiB). A stock NInfer build cannot read its gguf_* formats, so it needs a build from the consolidated repository.
MTP (Multi-Token Prediction) uses a separate next-token head to propose several candidates for the main model to verify. DFlash2 uses a separate draft adapter to propose candidates that the main model checks. Accepted proposals can emit more tokens8 per verification step, but the speed difference depends on the prompt, draft count, and GPU.
Download the model and start the Docker server
On Linux, install Docker, NVIDIA driver 580 or newer, and NVIDIA Container Toolkit. The Docker image used here runs on CUDA 13. Use the Toolkit's official sample workload to check that the container sees the GPU, then install the Hugging Face CLI9 command hf. Leave storage for the 11.18 GB model and server image, and enough GPU memory for both the model and KV cache10.
The download command saves the 11.18 GB .ninfer file in models. If hf download is unavailable, install the Hugging Face Hub package's CLI and retry.
The server accepts requests only on 127.0.0.1:8080. It starts with a 65,536-token context and three MTP drafts. The current directory's models folder is mounted at /models inside the container.
mkdir -p models
hf download satoshifukimoto/Underdog-Saluki-27B-1.0-IQ2-mix-NInfer-v3 Underdog-Saluki-27B-1.0-IQ2-mix-ninfer-v3.ninfer --local-dir modelsdocker run --rm --gpus all -p 127.0.0.1:8080:8080 --ulimit memlock=-1 \
-v "$PWD/models:/models" -v ninfer-cache:/cache \
ghcr.io/iamwavecut/ninfer-all serve /models/Underdog-Saluki-27B-1.0-IQ2-mix-ninfer-v3.ninfer \
--model-id underdog-saluki --max-context 65536 --kv-capacity 65536 --kv-dtype rk8v4 \
--gdn-state-fp16 --spec mtp --draft-tokens 3 --lm-head-draft --vision
Send the first prompt and inspect the reply
Keep the server terminal running and open another one. If underdog-saluki appears in the model list, the model is ready. The request below sends a short English prompt to the OpenAI-compatible chat endpoint with a 128-token output limit.
A successful response contains the model's reply in choices[0].message.content. Connection refused usually means the server is not ready or the port was not mapped; a model-ID error means the request's model differs from the server's --model-id.
curl -sS http://127.0.0.1:8080/v1/models
curl -sS http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"underdog-saluki","messages":[{"role":"user","content":"Explain why a local model server uses a context window in two sentences."}],"max_tokens":128}'MTP and DFlash2 generate draft candidates differently
After the first request works, compare draft modes using the same model and prompt. MTP uses the model's next-token head; DFlash2 uses the separate adapter in the same artifact. Change one setting at a time and restart the server with --spec dflash2 --draft-tokens 5 instead of --spec mtp --draft-tokens 3 --lm-head-draft.
More drafts can offer more tokens per verification round, but the engine may also spend work on candidates the model rejects. Five DFlash2 drafts can suit short chats, while MTP often does well with three to five; reported MTP acceptance for Saluki ranged from 30% to 100% by request. Different prose, code, and languages can change the result, so compare repeated runs of the same prompt rather than the fastest single run.
Hold prompt and answer lengths constant when comparing decode11 speed. Restart the same server and send the same prompt several times, recording output tokens and elapsed time so cache and GPU temperature do not silently change the comparison. To return to ordinary generation, remove --spec, --draft-tokens, and --lm-head-draft, then restart.
RTX 5090 community results
The table separates Saluki decode and prefill12 speed from its memory and load time.
| Workload | Community result |
|---|---|
| 512-token output · 3 MTP drafts | 223 tok/s |
| 512-token output · 5 DFlash2 drafts | 209 tok/s |
| Prefill for ~9.7K input tokens | 3,955 tok/s |
| Idle memory with 65K KV · 3 MTP drafts | ~14.3 GB |
| Model load | 20.2 seconds |
Single community run on October 9, 2026: Windows RTX 5090 (sm_120a build), rk8v4 KV, thinking off. Decode used a 512-token output, prefill used an approximately 9.7K-token prompt, and idle memory used 65K KV with MTP-3.
Check the artifact format and GPU connection first
For unsupported model or block-format errors, check that you are using the NInfer-all engine. This IQ2-mix artifact uses gguf_* blocks specific to NInfer-all and will not open in a stock NInfer build. Check container GPU access with the official NVIDIA Container Toolkit sample workload, then verify the artifact path and .ninfer extension.
A no CUDA device or GPU-architecture error points to the container runtime13 or build target. The supported targets 86, 89, and 120a correspond to GPU architectures. The Windows RTX 5090 result in this guide used an sm_120a build, so reproducing it on a 5090 requires that architecture. A native Windows build requires CUDA 13, CMake, Visual Studio C++ tools, and vcpkg dependencies; follow the repository's current build procedure.
If loading or a long request fails for lack of memory, lower the context limit below 65,536 or reduce --kv-capacity to reserve less KV cache. Increasing context while GPU memory is tight can cause the model to unload or the server to fail. Stop the container with Ctrl+C in its terminal; the model file remains on disk, so you can restart with adjusted options.
Try a short run alongside your existing API server
Keep your current model server and test NInfer at the local address 127.0.0.1:8080. Check the model list and a short reply, then expand to Korean prompts, code, and longer documents while recording answer quality and memory use. Compare task accuracy and tool calls14 alongside speed.
NInfer artifacts and ordinary GGUF use different runtime formats. To convert another checkpoint15, check NInfer-all's model-specific recipe and support for its MTP, DFlash2, or Vision components. Run the prepared Saluki file with the command above.
After choosing a fast setup, compare its replies with a fixed set of prompts you use often. Record omissions in summaries, code execution results, and tool arguments so the next model update can be checked on the same tasks.
For another inference engine, see the llama.cpp guide below. The Saluki guide covers standard GGUF setup and choosing a quantized file.
Terminology notes
CUDA — A software platform for general-purpose computing on NVIDIA GPUs.
Back to the textInference — The process of using a trained model to compute an output for an input. Here, local inference means running the model on the user’s device.
Back to the textAPI — A defined interface that lets other code call a program’s functions.
Back to the textMTP — A training objective or model component for predicting multiple future tokens at once. In speculative decoding, those predictions can be proposed as candidate tokens.
Back to the textGPU — A processor designed to handle many calculations in parallel. It performs model computations during AI inference.
Back to the textPrompt — The input text sent to a model. For generative models it describes the desired result and strongly shapes the output.
Back to the textGGUF — A file format for model data, widely used by llama.cpp-based tools.
Back to the textToken — A unit into which a model divides input or output for processing.
Back to the textCLI — Short for Command-Line Interface: operating a program by entering commands in a terminal.
Back to the textKV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.
Back to the textToken decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the textPrefill — The stage where an LLM reads the input prompt and computes representations for its tokens. Longer prompts contain more tokens to process.
Back to the textRuntime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.
Back to the textTool call — A structured request from a model for an external function such as reading a file, searching, or running a command. The agent runtime and its permission settings decide whether the request is actually executed.
Back to the textCheckpoint — A file containing saved model weights and related state. Versions or tasks in one model family may use different checkpoints.
Back to the text
Frequently asked questions
- What should you check first in this article?
- NInfer is a CUDA inference engine that runs `.ninfer` model files on NVIDIA GPUs and returns answers through an OpenAI-compatible API. Run Underdog Saluki in a Docker server, send a short query, and compare MTP and DFlash2 draft generation using the same request.
- How much memory does it need?
- Keep your current model server and test NInfer at the local address `127.0.0.1:8080`. Check the model list and a short reply, then expand to Korean prompts, code, and longer documents while recording answer quality and memory use. Compare task accuracy and tool calls alongside speed.