Executable programs and extensions
Last updated
How to measure local LLM speed with sparkDash
Measure single-request speed and concurrent throughput separately in sparkDash, then compare results with hardware, model, and workload conditions recorded.
sparkDash monitors DGX Spark and NVIDIA GPU1 hosts, local LLM2 servers, and runs decode3, prefill4, and quality benchmarks in one dashboard. Decode tests select prose, code, structured, or JSON across concurrent request counts. Prefill sweeps context sizes from 1K to 300K and records TTFT5 and prefill tok/s. Different request counts and input lengths represent different workloads, so compare matching engine, model, GPU power, prompt6, and reply length.
Choose the workload and server first
sparkDash's decode benchmark measures token7 speed while a server generates replies. Its prefill benchmark records prompt processing throughput8 and time until the first token. If your work begins with long documents, focus on prefill and TTFT; if replies are long, focus on decode.
The current README documents sparkDash 2.40.0. Remote GPU hosts can be added over SSH, and each LLM server address and port can be registered for backend and model detection. Before adding a local LLM, note its server port; for a remote unit, prepare an account that sparkDash can reach over SSH.
- Current sparkDash README and setup
- DGX Spark 128GB local LLM baseline
- Context length, RoPE and sliding-window attention
- Prefill vs decode: should you look at TTFT or tok/s?
- Set Up a Local vLLM Server: From a Small Model to Adaptive Verification in 0.30
- Hardware for Local AI Agents: Choose by Workload, Context, and Uptime
Start the dashboard from the official repository
Clone the repository and start the production Docker Compose stack. Its default bind is loopback-only, so open http://127.0.0.1:5555 on that host. The dashboard monitors servers and displays benchmark results; it does not install or download LLM models.
git clone https://github.com/MiaAI-Lab/sparkDash.git
cd sparkDash
docker compose up --build -dSelect This host in Add unit, or enter a remote machine's LAN address and SSH user. Add the TensorFold default port 8888 or the actual port used by the running API9. For remote Linux collection, the SSH public-key setup must be visible inside the Docker container. To view the dashboard from another computer, use a loopback SSH tunnel rather than exposing it directly.
ssh -N -L 5555:127.0.0.1:5555 user@sparkdash-host
Vary request count in the decode benchmark
Open the benchmark on the LLM unit, choose prose, structured, code, or JSON, then set concurrent request count and reply length. The lab protocol uses temperature 0 and thinking off. During the run, sparkDash measures tok/s from the streams and saves the result. When you increase concurrency, read both aggregate tok/s and per-request speed.
For example, if four requests each average 25 tok/s, the server total is 100 tok/s. Check the speed experienced by an individual through per-request token speed and time to first token (TTFT). Compare prose requests with prose and structured requests with structured ones.
Record prefill and hardware conditions for comparison
The prefill benchmark varies context size from 1K to 300K and records tok/s and TTFT. It uses a unique prefix at each prompt size, helping distinguish first-time prompt processing from cache hits. Read the time to the first token alongside the 32K prefill rate.
Record GPU model and power limit, TensorFold and model versions, quantization10, context, KV cache11, and parallelism beside the result. If the backend reports them, the health panel also shows KV use and running or waiting requests. If results differ sharply, check the unit's GPU and API connection before repeating the same workload.
Terminology notes
GPU — A processor designed to handle many calculations in parallel. It performs model computations during AI inference.
Back to the textLarge language model — A model trained on large text datasets to learn language patterns and process or generate text in context.
Back to the textToken decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the textPrefill — The stage where an LLM reads the input prompt and computes representations for its tokens. Longer prompts contain more tokens to process.
Back to the textTime to First Token — The time from sending a request until its first output token arrives.
Back to the textPrompt — The input text sent to a model. For generative models it describes the desired result and strongly shapes the output.
Back to the textToken — A unit into which a model divides input or output for processing.
Back to the textThroughput — The amount of work processed or generated over time. Comparisons need the unit, such as tokens per second or requests per second.
Back to the textAPI — A defined interface that lets other code call a program’s functions.
Back to the textQuantization — A way to represent model weights or values with fewer bits, reducing storage and memory use. The coarser representation can also affect accuracy.
Back to the textKV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.
Back to the text
Frequently asked questions
- What should you check first in this article?
- sparkDash monitors DGX Spark and NVIDIA GPU hosts, local LLM servers, and runs decode, prefill, and quality benchmarks in one dashboard. Decode tests select prose, code, structured, or JSON across concurrent request counts. Prefill sweeps context sizes from 1K to 300K and records TTFT and prefill tok/s. Different request counts and input lengths represent different workloads, so compare matching engine, model, GPU power, prompt, and reply length.