Executable programs and extensions

Serving local models with SGLang: separate caching from acceleration

When asking several questions about the same document, the first may take a while and the next may start quickly. The model may be reusing previously processed input rather than suddenly generating faster. Understanding that difference helps you decide whether SGLang fits your work and which settings are worth keeping.

1. Repeated questions and new documents are different workloads

Consider repeatedly asking for code changes after a long project description. If the beginning of each input is identical, previously computed input can sometimes be reused. SGLang uses its Radix cache to exploit shared prefixes. A workflow that constantly introduces new documents may not gain the same benefit. Combining both cases into one prefill rate gives a poor estimate of your wait.

Caching does not establish answer quality. Check how the app sends earlier conversation and new documents as well. Separate fresh inputs from repeated ones, and inspect output tok/s independently. Aggregate server throughput under concurrent load is not the output rate one person reads. That is why document-batch and interactive-chat settings should not share a single ranking.

2. Do not paste a Spark command unchanged onto a PC or Mac

The usual NVIDIA path requires a compatible combination of Linux, CUDA, PyTorch and SGLang GPU-kernel packages. Install in an isolated environment and record the combination that works. DGX Spark needs a build supporting both ARM64 and Blackwell. Results from a model-specific container or patch are not automatically reproducible with a standard installation.

There is also an MLX-based Metal serving path documented for Macs. Enabling it with SGLANG_USE_MLX=1 does not make it the same installation as CUDA. Check MLX-converted models and supported features separately. It is also distinct from the PyTorch MPS path used by SGLang Diffusion for images and video. The launch command below is for NVIDIA CUDA, not a Mac launch recipe.

3. Connect a server and a question before adding acceleration

After installation, use the small Qwen2.5-0.5B-Instruct model to check that the server accepts a question. The 4,096-token context limit and concurrency of one are connection-test settings, not peak-speed settings for the hardware. The first launch downloads model files if they are absent. Wait until the model loads and the server is ready, then run the second command in another terminal.

After a successful reply, point the app to http://127.0.0.1:30000/v1. If it cannot connect, check server readiness, port and model name before changing the listening address to 0.0.0.0. The small model’s answer quality is not a basis for choosing your main model. This step tests the connection between engine and client.

Start an SGLang CUDA server

python3 -m sglang.launch_server \
  --model-path Qwen/Qwen2.5-0.5B-Instruct \
  --host 127.0.0.1 --port 30000 \
  --context-length 4096 --max-running-requests 1

A connection-check example to run after installing SGLang for your NVIDIA hardware.

Send a short question to the local server

curl --fail-with-body http://127.0.0.1:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen2.5-0.5B-Instruct","messages":[{"role":"user","content":"Explain what a local AI server does in two sentences."}],"max_tokens":64,"temperature":0}'

Use the same model name in the server and request. This public example does not require a personal document.

4. Does it fail while reading or while writing?

When moving to your main model, check loader and GPU-kernel support even if the quantization label looks familiar. --mem-fraction-static controls the share used for weights and the KV cache, but computation needs additional space. Copying a high value to fill every free byte can lead to runtime failures. A setting that works for the small test model may not work for a larger one.

If memory runs out during prefill of a long document, reduce --chunked-prefill-size to process input in smaller pieces. This can also change latency. If it fails only during generation, lower --max-running-requests first. Distinguishing weights that do not fit from insufficient working memory tells you whether you need a smaller quantized file or whether a settings change is enough.

5. Do not combine MTP and NGRAM by name alone

SGLang runs MTP through a model-specific speculative-decoding path. You will encounter --speculative-algorithm and settings for candidate depth and width, but built-in MTP differs from methods that require a separate draft model. Current documentation describes NEXTN as an alias of EAGLE. That does not make one command valid for every Qwen, Gemma or GLM checkpoint.

NGRAM speculative decoding finds candidates in previous tokens. It is not PLE offloading, which places a large n-gram embedding table in RAM or on SSD. When adding acceleration, check memory needed for candidate verification and the constraints of the current backend. Record a baseline without it, then assess fresh and repeated documents separately after enabling it.

6. Save the working setup before the next experiment

For kernel or library errors, return to the last working environment instead of repeatedly replacing packages in the existing one. Remove acceleration first if memory failures begin after adding MTP. An unsupported-option message can indicate a version-command mismatch rather than a model-performance problem. Do not substitute another engine’s --gpu-memory-utilization for SGLang’s memory option.

Finally, keep the model revision, runtime and kernel versions, acceleration state and actual input length together. After a warmup, check at least three runs under the same conditions and separate first-token and decode timing. With two Sparks, splitting a model for capacity and distributing requests serve different purposes. You can wait for evidence that your usual work improves before adding hardware.

Change log

These entries record changes to the site's guidance. They do not automatically check your installed engine or model version.

  1. Added SGLang cache and model-specific option guidance

    Added local-server connection examples and steps for adjusting memory at the prefill and decode stages. It separates cache effects from repeated inputs and explains that NGRAM candidate lookup and placing PLE tables in RAM or on SSD are different features.