How much memory does it need?

Record completion time, generated tokens, draft/accepted lengths in speculative statistics, and memory. Accepted-token counts alone do not prove user wait decreased; compare decode and total completion for the same request. If draft initialization or memory fails, remove the draft, confirm the baseline server answers again, then inspect file, backend, and block limit one at a time.

Response speed and acceleration

Run DFlash and DSpark with llama.cpp: A Qwen3-4B Walkthrough

DFlash and DSpark propose tokens for the main model to verify

Speculative decoding¹ is a generation method in which a smaller draft model proposes future tokens² and the main model verifies them, accepting a matching consecutive span. With Qwen3-4B in llama.cpp, prepare a compatible DFlash or DSpark draft GGUF³ and compare them one at a time.

If a code-review answer is slow after it begins, generation is a useful stage to test. If the wait is long before the first text appears, PDF retrieval, input processing, or memory may be responsible; adding draft candidates alone may not reduce it.

Contents

Requirements and key details

DFlash and DSpark require separate draft GGUFs matched to the Qwen3-4B target. MTP is a different path using supported heads in the main model.
Match draft candidate length to the trained block size and verify the actual value in logs.
Compare baseline and accelerated runs with the same greedy request, and inspect accepted tokens and completion time.

Verify a baseline server with Qwen3-4B first

Send the same request to baseline and draft runs: edit one repository function and handle both empty and normal input. Check the diff and tests against the requested behavior. Draft proposals and multi-position verification also cost computation, so judge the result on the actual task.

Run these commands after preparing a llama.cpp build supporting the model and convert_hf_to_gguf.py in its source directory. From that directory, download the Hugging Face model, convert it to GGUF, and start the target server. Fix the baseline at 4,096 context tokens and one request. If the baseline does not answer, check the model path, build support, and memory first.

Qwen3-4B server without a draft

hf download Qwen/Qwen3-4B --local-dir ./hf-models/Qwen3-4B
python convert_hf_to_gguf.py ./hf-models/Qwen3-4B --outfile ./models/Qwen3-4B.gguf --outtype bf16
llama-server -m ./models/Qwen3-4B.gguf --alias qwen3-4b --host 127.0.0.1 --port 8080 --ctx-size 4096 --parallel 1 --n-gpu-layers auto

Prepare and convert the Hugging Face source, then launch the server. Check GPU placement in startup logs.

Check readiness and send a short request

curl -sS http://127.0.0.1:8080/health
curl -sS http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen3-4b","messages":[{"role":"user","content":"Reply with one short sentence."}],"temperature":0,"max_tokens":64,"stream":false}'

Check server readiness, then send a short prompt to test the baseline connection.

A small draft model proposes tokens for a larger target model to verify together. — Only consecutive draft proposals verified by the target enter the answer.

Choose DFlash or DSpark matched to the target

In the official example, z-lab/Qwen3-4B-DFlash is paired with the Qwen/Qwen3-4B target. DFlash uses a block-diffusion draft; DSpark adds a Markov head to the draft structure. Do not substitute an arbitrary small model—use a draft trained for the target.

The commands assume both model source directories are already prepared. Keep the Qwen3-4B target fixed and attach one converted draft at a time. Example draft limits are 15 for DFlash and 7 for DSpark; the trained block size may impose a lower limit. Check the model card and conversion output for other models.

Run one server at a time for each comparison; they all use port 8080. Stop the baseline server with Ctrl+C in its terminal, then start DFlash. When it is ready, send the same prompt⁴ from another terminal and record the result.

Stop DFlash, then test DSpark in the same way. Download and convert both draft models before starting the servers.

Attach DFlash or DSpark draft to the target

hf download z-lab/Qwen3-4B-DFlash --local-dir ./hf-models/Qwen3-4B-DFlash
python convert_hf_to_gguf.py ./hf-models/Qwen3-4B-DFlash --target-model-dir ./hf-models/Qwen3-4B --outtype bf16 --outfile ./models/Qwen3-4B-DFlash.gguf
# Stop the previous server with Ctrl+C before starting this one.
llama-server -m ./models/Qwen3-4B.gguf -md ./models/Qwen3-4B-DFlash.gguf --spec-type draft-dflash --spec-draft-n-max 15 -fa on --jinja --host 127.0.0.1 --port 8080 --ctx-size 4096 --parallel 1

hf download deepseek-ai/dspark_qwen3_4b_block7 --local-dir ./hf-models/dspark_qwen3_4b_block7
python convert_hf_to_gguf.py ./hf-models/dspark_qwen3_4b_block7 --target-model-dir ./hf-models/Qwen3-4B --outtype bf16 --outfile ./models/Qwen3-4B-DSpark.gguf
# Stop the previous server with Ctrl+C before starting this one.
llama-server -m ./models/Qwen3-4B.gguf -md ./models/Qwen3-4B-DSpark.gguf --spec-type draft-dspark --spec-draft-n-max 7 -fa on --jinja --host 127.0.0.1 --port 8080 --ctx-size 4096 --parallel 1

Change one draft type at a time with the same target and request. Model downloads and conversion setup are separate prerequisites.

Branching token candidates narrow to one path after verification. — Tree proposals also retain only the consecutive path accepted by the target.

Check logs and responses to confirm the draft path

Check each server log for the draft file opening and DFlash or DSpark initialization. Do not treat a run where only the main model answered as an acceleration comparison. Once ready, send the same request, confirm a response in choices[0].message.content, and inspect model identifier and error logs.

Keep model file, prompt, sampling, context, output limit, and device state the same as the baseline. After warm-up, repeat runs and record median and range; distinguish a fresh server start from runs with the model loaded. Compare answers using a greedy request and check code diff and tests for edit scope.

Record completion time, generated tokens, draft/accepted lengths in speculative statistics, and memory. Accepted-token counts alone do not prove user wait decreased; compare decode⁵ and total completion for the same request. If draft initialization or memory fails, remove the draft, confirm the baseline server answers again, then inspect file, backend, and block limit one at a time.

Weigh added memory against gains on the real task

A draft model uses additional memory. If code-answer generation repeatedly improves, the requested edit and tests remain correct, and memory is sufficient, record that target/device/backend combination. If first-token wait on a long input is the bottleneck or memory headroom is tight, adding a draft may have limited value.

Do not assume the same result on another target, quantization⁶, or device. Whenever the model or runtime⁷ changes, recheck official support, load logs, and completion time for the same request.

Spark speed depends on the serving setup — Qwen3.8-27B · NVFP4 · 128 input/output tokens · 1 request — Published community results for complete configurations. Engines and model artifacts also differ; these are not isolated MTP gains.

A balance compares draft-and-verification cost with gains from accepted tokens. — A lightweight draft and well-accepted proposals can offset extra computation.

Terminology notes

Speculative decoding — A generation method in which a draft model, MTP head, or repeated-context lookup proposes output candidates for the main model to verify and accept.
Back to the text
Token — A unit into which a model divides input or output for processing.
Back to the text
GGUF — A file format for model data, widely used by llama.cpp-based tools.
Back to the text
Prompt — The input text sent to a model. For generative models it describes the desired result and strongly shapes the output.
Back to the text
Token decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the text
Quantization — A way to represent model weights or values with fewer bits, reducing storage and memory use. The coarser representation can also affect accuracy.
Back to the text
Runtime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.
Back to the text

Frequently asked questions

What should you check first in this article?: Speculative decoding is a generation method in which a smaller draft model proposes future tokens and the main model verifies them, accepting a matching consecutive span. With Qwen3-4B in llama.cpp, prepare a compatible DFlash or DSpark draft GGUF and compare them one at a time.
How much memory does it need?: Record completion time, generated tokens, draft/accepted lengths in speculative statistics, and memory. Accepted-token counts alone do not prove user wait decreased; compare decode and total completion for the same request. If draft initialization or memory fails, remove the draft, confirm the baseline server answers again, then inspect file, backend, and block limit one at a time.