Model setup recipes
Last updated
How Can You Run Qwen3.8-27B Fastest on a CMP 170HX?
In the measured single-card setup, vLLM DFlash2 had the higher decode mean on a 3.2K-token prompt.
Running Qwen3.8-27B on a CMP 170HX means loading GGUF1 Q4_K_M or W4A16 AutoRound weights in a serving engine and returning answers to text requests. In an individual approximately 3.2K-token request from the October 10 single-card measurement, vLLM DFlash2 reached 109.42 tok/s and open-jet MTP2 d=4 reached 77.20 tok/s. Separately, the greedy mean across five prompts3 was 181.8 and 94.1 tok/s, respectively.
For fast generation with short inputs, start by examining the vLLM DFlash2 setup. For longer documents, open-jet has a published completed run with 18,976 input tokens4. Below we compare generation speed, time to the first answer, and how to reproduce the setups.
Read Decode and Prefill Results Separately
Prefill5 covers processing the input up to the first token; decode6 is the speed at which the answer continues after that token. In the October 10 comparison at 180W, vLLM DFlash2 k=7 averaged 181.8 tok/s across five greedy prompts, versus 94.1 tok/s for open-jet megakernel MTP d=4. The four-prompt mean with thinking enabled was also higher for vLLM: 156.3 versus 75.6 tok/s.
| Workload and input | vLLM DFlash2 k=7 | open-jet MTP d=4 |
|---|---|---|
| Single chat · 3,163 / 3,158 input tokens | 109.42 tok/s · TTFT 1.6515 s · 58/128 output tokens | 77.20 tok/s · TTFT 2.0662 s · without MTP: 48.74 tok/s |
| Greedy mean · 5 prompts | 181.8 tok/s | 94.1 tok/s |
| Thinking mean · 4 prompts | 156.3 tok/s | 75.6 tok/s |
This comparison served one request at a time on a CMP with 64GiB, 74 SMs, HBM at 1,728MHz, and PCIe Gen2 x16, capped at 180W with forced airflow. vLLM 0.27.1 used stock guest driver 580.178.04 and host unlock commit 88e39ce; open-jet used published commit 93c2b9a.
Keeping the Server Running or Starting It Again
Both runs used 4-bit Qwen3.8-27B weights, but the files differed. open-jet used Qwen3.8-27B-Q4_K_M.gguf from a pinned revision; vLLM used a W4A16 AutoRound model with a DFlash2 drafter. Quantization7 format and checkpoint8 can affect output and speed, so the table compares these two specific setups.
The vLLM server became ready after 427 seconds, including CUDA9 graph capture and DFlash2 setup, while open-jet started in seconds. If a server starts once per day and stays running, startup is a one-time daily cost; during development with frequent restarts, that wait can recur. Once a server stays online, decode speed is the more direct per-request comparison.
For Reproduction, Follow the Published vLLM Recipe Pins
The measured setup used vLLM 0.27.1. Start from commit 69ba4d0688c6ae76cb9d3c4a5c3b36445e1b040c and requirements.lock in the linked syv.ai recipe. The server used SPEC=dflash2, CTX=fast, MAX_SEQS=1, DFLASH_TOKENS=7, GPU_UTIL=0.90, and BF1610 KV cache11. Context was configured to 65,536; reproduce.ipynb and recipe.json describe model files and execution steps.
Check the CUDA compiler installation path when reproducing this in a fresh environment. In the published notebook, nvidia-cuda-nvcc 13.4.92 was installed under nvidia/cu13/, causing import nvidia.cuda_nvcc to fail. The notebook appendix records the researcher's path fix.
To try open-jet, check out the measured commit and build for GA100. Run the commands below after preparing CUDA 12.8.93 and the repository's dependencies. Follow the linked reproduction notebook for model downloads and server startup.
git clone https://github.com/L-Forster/open-jet.git
cd open-jet
git checkout 93c2b9abee50ea2ea41981c406cfd201989e4d58
make ARCH=sm_80
Reassess the Result at Your Own Context Length
If you want the highest decode speed on short prompts, vLLM DFlash2 is the leading starting point in the published single-card setup. If server restart time matters more or you need a path with a short startup, open-jet's quick start may also matter. Because the 180W open-jet and vLLM runs used different checkpoint formats, compare them with the same prompts and response settings before deciding.
Results differed by request type when inputs were increased. vLLM completed a 6,603-token /v1/completions task, but exited with Xid 31 on a 9,472-token chat request. open-jet completed inputs up to 18,976 tokens. For long-document work, use these completed runs to choose an engine, then test with your own documents.
Before deciding, fix the prompt and answer lengths that are common in your work. Record whether the engine completes each request, time to first token12, generation speed, and completion status separately. If you need longer inputs, increase context step by step with the engine and checkpoint you use rather than jumping to the configured maximum; mark any length that fails as outside the validated range.
Terminology notes
GGUF — A file format for model data, widely used by llama.cpp-based tools.
Back to the textMTP — A training objective or model component for predicting multiple future tokens at once. In speculative decoding, those predictions can be proposed as candidate tokens.
Back to the textPrompt — The input text sent to a model. For generative models it describes the desired result and strongly shapes the output.
Back to the textToken — A unit into which a model divides input or output for processing.
Back to the textPrefill — The stage where an LLM reads the input prompt and computes representations for its tokens. Longer prompts contain more tokens to process.
Back to the textToken decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the textQuantization — A way to represent model weights or values with fewer bits, reducing storage and memory use. The coarser representation can also affect accuracy.
Back to the textCheckpoint — A file containing saved model weights and related state. Versions or tasks in one model family may use different checkpoints.
Back to the textCUDA — A software platform for general-purpose computing on NVIDIA GPUs.
Back to the textBF16 — A 16-bit floating-point format for model values. Its exponent has the same width as FP32, giving it a similarly wide value range.
Back to the textKV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.
Back to the textTime to First Token — The time from sending a request until its first output token arrives.
Back to the text
Frequently asked questions
- What should you check first in this article?
- Running Qwen3.8-27B on a CMP 170HX means loading GGUF Q4_K_M or W4A16 AutoRound weights in a serving engine and returning answers to text requests. In an individual approximately 3.2K-token request from the October 10 single-card measurement, vLLM DFlash2 reached 109.42 tok/s and open-jet MTP d=4 reached 77.20 tok/s. Separately, the greedy mean across five prompts was 181.8 and 94.1 tok/s, respectively.