llama.cpp: finding which settings are slowing you down

For summarizing a long technical memo, keep the GGUF file and request fixed. Check the server response with a short request, then submit the whole memo. Separating input processing from answer generation shows where you wait.

Direct execution also means managing the executable, drivers, updates, and commands. That effort is useful when you need conditions unavailable in an application or must reproduce a build-specific issue.

Low GPU utilization alone does not indicate a faulty card. A CPU-only executable will not use a GPU backend even when a GPU is present. NVIDIA CUDA⁶, Apple Silicon Metal⁷, and Vulkan have different setup paths. Check detected devices and selected backend in startup logs; changing placement options cannot enable an undetected backend.

GGUF is a distribution format, not a guarantee that every build supports every model architecture. For a new architecture, check executable support and the conversion's source model. If you need image input, separately verify multimodal implementation and required files; a text GGUF alone does not provide an image path.

After preparing a model file, replace ./models/model.gguf with its actual path. The local-guide alias, 4,096 context budget, and one parallel slot are baseline comparison settings. Context is shared by input and generated output, so 4,096 does not mean the full budget can be used for the memo input. Separate first-load time from subsequent requests.

Start GPU-layer placement at auto and inspect logs and per-GPU memory for actual placement. An auto option does not mean the entire model is on GPU. If the executable differs from expectations, check its version and --help, then save the smallest working command and its logs. A command with unsupported options removed is not the same condition as the original example.

Record the GGUF filename and quantization⁸, executable version, backend, 4,096 context, and one slot. Keep the model and runtime⁹ conditions fixed so you can change batch or GPU placement one item at a time.

After the server process starts, check that /health reports readiness and send a short prompt¹⁰. The first command checks status; the second requests text generation. Verify the alias and basic connection before submitting the full memo to isolate load and connectivity issues.

An API¹¹ success does not verify that a client-specific template or tool-calling format works. The curl request below waits for the full response, so it can show round-trip time but does not measure time to first token¹² or prefill and decode separately.

If the first answer is delayed after a long memo, prefill may be responsible; if tokens¹³ arrive slowly after generation starts, inspect decode. Long input also uses more context memory, so compare against the 4,096-context baseline and check GPU headroom and KV cache¹⁴.

Increasing prefill batch may help long-input processing, but it does not imply decode speeds up by the same factor. Increasing concurrency can affect aggregate throughput¹⁵ without reducing one user's completion time. For individual summaries, keep one slot as the baseline and change only the relevant batch, context, or GPU-placement setting indicated by logs.

For comparisons, keep the GGUF file and quantization, input and output lengths, backend, and concurrent request count fixed. After a warm-up run, send the same request at least three times and record whether the model is resident and whether the KV cache is reused.

A run is not a cold start if you could not clear the cache. If one result is an outlier, keep it and check warm-up state and background load before summarizing with the median.

A curl request that waits for the full answer measures round-trip time; it is not a benchmark that separates prefill and decode. Once the target task completes reliably, decide whether direct management provides enough benefit. If Ollama or LM Studio offers the needed options, you can keep using it.

llama.cpp b11413 improves the Vulkan execution path for some long-context models. It targets sparse-attention architectures such as QSA in Qwen3.8-Flash-Next, which select only needed positions. With quantized K/V caches, the new path reads those selected positions instead of processing the entire context.

The PR author's Radeon R9700 and RX 7900 XT measurements report average decode improvements of 19% at 64K and 15% at 128K with q8_0 KV caches. Shorter contexts can have little benefit or regress, so the runtime selects the path. These figures do not apply to f16 caches or prefill. Use the following command after preparing a Vulkan build and a compatible model.

Released on October 5, 2026, llama.cpp 0.6.0 adds /v1/systemone, an API that returns decisions and probabilities. It can route inquiries to teams or classify documents. The local decision-model example explains how to select a supported model and send the input and options. Official 0.6.0 release

Performance-related changes include Qwen4Exp MTP¹⁶, Metal Flash Attention¹⁷ for F16 KV, and CUDA NVFP4/MXFP4 computation paths. Compare the old and new versions using the same question and a feature supported by your model. Existing speed records retain the version used for their measurement so they can be compared with new records.

Starting with llama.cpp b11507, MoE¹⁸ expert weights kept on the CPU¹⁹ can be cached across multiple GPUs. The cache reduces repeated transfers of needed experts to the GPU. It is worth testing when a large MoE model is split between system RAM²⁰ and GPUs.

The current --moe-cache-mib specifies the total cache budget in MiB, distributed across GPUs according to --tensor-split. It is disabled by default with a value of zero. Check for the flag in llama-server --help, then compare both first-token wait and decode speed on the same document with and without the cache.

Build b11512 also fixes DFlash output-head sharing. When updating a DFlash setup, keep the main and draft models unchanged, send the same input and check that generation completes. b11507 and b11512 are October 8 rolling builds, a separate update series from the fixed 0.6.0 release.