Executable programs and extensions

Serving locally with vLLM: fast replies and high throughput are different

When connecting a local model to a coding tool, you will often come across vLLM. Yet the high token rate in an introduction may not match what you see on screen. Start by deciding what you want to make faster, verify one small server, then add only the settings your work needs.

1. Decide which wait matters to you

Imagine asking a local model questions while you write code alone. You want the reply to begin promptly and the code to keep appearing smoothly. A server processing many documents overnight may instead benefit from finishing the whole queue earlier, even if individual requests wait longer. Those goals can call for different settings in the same engine.

vLLM schedules requests together to use the GPU and manages the KV cache needed during generation. This can produce a high combined output rate across users. Reading that number as the speed of a single chat window leads to the wrong expectation. For your comparison, record first-token latency and generation rate with concurrency set to one.

2. NVIDIA hardware does not all use the same build

The commands here assume vLLM is installed in a compatible Linux NVIDIA CUDA environment. Use a separate virtual environment or a container built for the hardware rather than replacing packages in an existing workspace. PyTorch, CUDA and the GPU driver must work together. A successful installation message does not yet establish that GPU inference works.

For DGX Spark, check both its ARM64 CPU architecture and Blackwell GPU support; an image built for a typical x86 PC is not interchangeable. Macs take another path. Native macOS CPU execution and the community vllm-metal plugin, which uses MLX, are distinct from CUDA execution. Metal support does not imply identical quantization or MTP support.

3. Use a small model to check the connection

Starting with a large target model and acceleration options together makes failures harder to diagnose. The command below starts Qwen2.5-1.5B-Instruct as an installation check, with a 4,096-token context limit and one concurrent request. It is not a recommendation for your main model. If the files are not cached, the first launch downloads them; do not include that preparation in answer-speed measurements.

Once the server reports that it is ready, send a short question from another terminal. Keep the model name and port consistent between the two commands. The address 127.0.0.1 is accessible only on this computer. After a successful reply, an app can use http://127.0.0.1:8000/v1 as its server address. Save this working setup so you have a known point to return to if later changes fail.

Connection-check server in a CUDA environment

vllm serve Qwen/Qwen2.5-1.5B-Instruct \
  --host 127.0.0.1 --port 8000 \
  --max-model-len 4096 --max-num-seqs 1

Run after installing vLLM. Spark requires an ARM64 build with Blackwell support first.

Send a question from another terminal

curl --fail-with-body http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen2.5-1.5B-Instruct","messages":[{"role":"user","content":"Explain what a local AI server does in two sentences."}],"max_tokens":64,"temperature":0}'

This question checks connectivity. Do not treat this single response as a benchmark record.

4. Filling memory is not the goal

After switching to your main model, first fix the files, quantization format and input length. NVFP4 and GGUF Q4 are not interchangeable settings simply because both use four-bit representations. Execution support depends on the GPU generation and model architecture. Long documents still need room for KV cache and working memory after the weights have loaded.

Start with --max-model-len set to the length you actually need and --max-num-seqs 1. Test higher concurrency afterward. Increasing --gpu-memory-utilization may leave more capacity for cache, but less room for other processes. If requests are repeatedly preempted and recomputed, reduce context and concurrency to isolate the cause rather than pushing the allocation to its limit.

5. Check the model before enabling MTP

MTP proposes upcoming tokens and verifies them. It requires a supported target model, the necessary weights and a compatible vLLM version; one option cannot be attached to every model. Current vLLM takes the configuration through --speculative-config. Instead of copying NEXTN or --spec-type from another engine, check the supported path in the recipe for your model.

Compare acceleration off and on with the same input and output lengths. Code tasks with a high acceptance rate can behave differently from open-ended writing. If the task has a long input and a short answer, the wait before generation may matter more than faster decoding. Separate first-time inputs from repeated, cached inputs so you do not attribute cache effects to the wrong optimization.

6. Keep a working configuration to return to

If memory runs out while loading, check model-file size and other GPU processes first. If failure occurs only after a request, reduce context and concurrency. If it starts after adding MTP, remove acceleration and retest. For failures confined to compilation or CUDA graph capture, --enforce-eager can help diagnosis, but leaving it enabled afterward can change steady-state generation performance.

Keep the working version, model revision and launch command together. For updates, retain the old environment and test a short question and a typical long document in a new one. Recovery is much easier that way. If single-user speed is already sufficient, there is no need to move to a complex multi-GPU setup. Decide whether you need more memory or more concurrent work before comparing hardware.

Change log

These entries record changes to the site's guidance. They do not automatically check your installed engine or model version.

  1. Added vLLM runtime-requirement and request-handling guidance

    Added short server and API examples for CUDA, while distinguishing the paths for a conventional PC, ARM64 Spark and Mac. It separates single-request generation speed from total server throughput and calls for model-specific quantization and MTP compatibility checks.