Executable programs and extensions

llama.cpp: finding which settings are slowing you down

You use the same GGUF but get slower results than someone else. After several changes in the app, you no longer remember the original settings. Running llama.cpp directly is not automatically faster; it makes the file and execution conditions easier to see and record.

What direct execution gives you—and leaves to you

llama.cpp is an inference project for running models across different hardware. llama-server exposes those models over HTTP. Related engines also run inside desktop apps, but launching the server directly makes it easy to record the file, context and concurrency in a command.

In return, you take responsibility for choosing the executable and drivers and managing updates. There is little reason to move if your current app exposes all the settings you need. Direct execution becomes useful when you need to inspect hidden conditions or reproduce a problem with a particular build.

Having a GPU is not enough; the build must support it

Setup differs between backends such as CUDA for NVIDIA, Metal for Apple Silicon and Vulkan on other supported hardware. Low GPU utilization after launching a CPU build does not establish a graphics-card problem. Check the detected devices and selected backend in the startup log.

GGUF is a model file format, not a guarantee that every architecture is supported. For a new architecture, check build support and the converted file's details together. Image input also requires checking multimodal support and any additional files; one text GGUF is not enough for every feature.

Start with one model and one request

The example assumes you already have llama-server and a GGUF file. Replace ./models/model.gguf with the actual file path. It explicitly sets the alias to local-guide, the context to 4,096 tokens and one concurrent slot. This is a baseline for comparing changes, not a claim about the fastest settings for your hardware.

Leave GPU placement on auto and inspect the result. If the executable's help does not list an option, your installed build differs from this example. Check the version and help first. Record any change in conditions caused by removing unsupported options.

Start a local server with a downloaded GGUF

llama-server -m ./models/model.gguf --alias local-guide --host 127.0.0.1 --port 8080 --ctx-size 4096 --parallel 1 --n-gpu-layers auto

This terminal example assumes the executable is on PATH. If you built it yourself, use its actual location. First check the GGUF path and --help for that build.

A running server is not always a ready model

While the model is loading, it may not be ready to answer. Check that /health reports readiness before sending a short request. The first command below checks status only; the second requests an actual generated answer.

Even a valid HTTP response can fail in an app if the chat template or tool-call format does not match. Check the API alias and a short sentence before adding features. This example receives the full answer, so it is not a measurement of pure prefill time or time to first token.

Check readiness and the response separately

curl -sS http://127.0.0.1:8080/health
curl -sS http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"local-guide","messages":[{"role":"user","content":"Reply with one short sentence."}],"max_tokens":128,"stream":false}'

Leave the server terminal running and use another terminal. Confirm that the model is ready before sending the second request.

Find the slow part, then change one thing

First inspect how the model is split between CPU and GPU. Check whether more can fit on the GPU or whether context allocation is causing memory pressure. Then separate input processing, or prefill, from output generation, or decode. A gain from larger input batches does not imply the same proportional gain in generation speed.

Keep concurrency fixed when testing one person's experience. Higher aggregate throughput does not necessarily finish your answer sooner. Match the file, quantization and input/output lengths, warm up, then record at least three runs. If maintaining the setup costs more than the tuning gains, returning to Ollama or LM Studio is reasonable. The aim is reliable answers, not more commands.

Change log

These entries record changes to the site's guidance. They do not automatically check your installed engine or model version.

  1. Added llama.cpp server-start and recovery guidance

    Added examples for starting a server with a prepared GGUF file and checking its local API. It explains how to change GPU offloading, context length and request count one at a time, then return to a working configuration if memory runs short.