memory and model
Last updated
MLX-LM Server Cache: Reusing Shared Prompts and Testing Concurrency
Reuse prefix computation when repeatedly reading the same policy.
Prompt1 caching reuses computation state from the beginning of an already processed input, or prefix, in later requests. A server receiving shared instructions or the same document repeatedly can reduce input processing before the first answer. This guide sends a return policy twice using MLX-LM2 to examine input reuse, memory, and waiting time.
Product support often attaches the same return policy while changing the customer's question. The server can reuse policy computation and process the new question and answer. Because each answer is generated anew, also check that the policy and question were transmitted correctly.
Start a baseline server on an Apple Silicon Mac
This exercise uses MLX-LM and mlx-community/Qwen3-4B-4bit on an Apple Silicon Mac. The model card's 2.26GB is the download size. Execution also needs unified memory3 for weights, caches, and temporary computation.
Prepare a Python tooling environment and uv, then run the commands below. The first run downloads the model, so wait for the server-ready log. This example uses default context settings and keeps the same server process. If the server is already running, send requests to that process.
uv tool install mlx-lm
mlx_lm.server --model "mlx-community/Qwen3-4B-4bit" --host 127.0.0.1 --port 8080Keep the policy and change only the question
The system message below contains an example return policy: unopened accessories may be returned within 30 days with proof of purchase; opened items qualify only if they arrived damaged. The first user asks whether an opened item can be returned after a change of mind. Check that the answer cites the relevant policy sentence and reason.
For the second request, change only the user's content to ask what happens if the item arrived damaged. Keep the model, message roles and order, system content, template, and context settings. The questions differ, but the policy tokens4 at the beginning can be shared.
If response usage includes usage.prompt_tokens_details.cached_tokens, record it for both requests. It indicates reused input length. If your version does not return this field, inspect server logs and available usage information. Record first-token time, total completion time, and output token count to compare actual waiting times. This example uses stream:false and returns the completed answer, so obtain TTFT5 from server timing metrics or a separate streaming request.
curl -sS http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"mlx-community/Qwen3-4B-4bit","messages":[{"role":"system","content":"Return the relevant policy sentence and one short reason. Policy: Unopened accessories may be returned within 30 days with proof of purchase. Opened accessories are not returnable unless they arrived damaged."},{"role":"user","content":"A customer opened an accessory and changed their mind. Can they return it?"}],"max_tokens":128,"stream":false}'
Text before shared content also affects the cache
The cache reuses the beginning of the final token sequence sent to the model. A date or user ID before the policy can change the input from that point onward. Check the difference between shared instructions followed by the customer's question and customer-specific instructions placed first.
Changing the model or message template also changes reuse conditions. Preserve required permissions and safety instructions when adjusting order for caching. Next, alternate different long policies and return to the original to check whether its earlier state remains.
If cached tokens are zero for apparently identical content, check actual system content, the server process, cache entry retention, and the model's cache trimming support in turn. Compare logs to see which request state was removed and why it was recomputed.
Measure individual latency separately from throughput
First send inquiries one at a time to the prepared server. Keep the same model and policy, recording first token, completion time, input and output tokens, and cache hits. Separate initial model loading after restart and compare prepared requests with each other.
Next send independent inquiries concurrently. Record median request latency, slow requests, total tokens per second, and memory. Test concurrent shared prefixes separately from mixed user-specific prefixes to distinguish caching and concurrency effects.
As an illustration, suppose one request takes 30 seconds and four concurrent requests each finish in 36 seconds. Individual latency increases by 6 seconds, or 20%. Sequential processing takes 30×4=120 seconds, while concurrent completion takes about 36 seconds of wall-clock time. The sum of individual concurrent times, 36×4=144 seconds, differs from elapsed time. Prioritize individual latency for immediate answers or completion count and elapsed time for inquiry batches.
| Condition | Key records | What it reveals |
|---|---|---|
| Sequential warm requests | TTFT, completion time, cached tokens | Shared-input reuse effect |
| Cold start | Model loading, first request, memory | Server preparation cost |
| Concurrent requests | Individual latency, throughput, memory | Latency and capacity under concurrency |
| Mixed prefixes | Hit/miss, requests after eviction | Cache retention scope |
What to check when reducing cache size
Retained state continues to use memory. Model weights, active request KV caches, and stored entries share that space. If memory is tight, first test reducing concurrency and actual input length.
Find cache limits in the current version's mlx_lm.server --help. Options such as --prompt-cache-bytes limit stored-entry size, whereas --max-kv-size changes the cache structure for long conversations. Verify support and application in help and startup logs, then change one option at a time and compare hits, TTFT, and peak memory.
After establishing stability with short context and one request, increase to match your workload. If the first token improves but generation does not, compare decode6 next. If greater concurrency exceeds acceptable waiting time, reduce it and retain a cache size that preserves reuse of frequent shared documents.
Keep settings matched to the version and workload
MLX-LM v0.31.0 includes server cache and size-limit improvements; v0.31.2 improves system and user message caching and the batch generator. v0.31.3 adds fixes for batch KV cache7 and parallel tool calls8. On updates, recheck options and logs, then compare previous settings with the same requests.
For work repeatedly using the same instructions and documents, tune according to actual hits and first-response differences. If you read each new document once and your Mac responds adequately, larger caches may be unnecessary. Add concurrency tests when multiple applications begin connecting at once.
Terminology notes
Prompt — The input text sent to a model. For generative models it describes the desired result and strongly shapes the output.
Back to the textMLX-LM — A package for loading, running, and fine-tuning language models with MLX.
Back to the textUnified memory — An architecture where the CPU and GPU share one physical memory pool.
Back to the textToken — A unit into which a model divides input or output for processing.
Back to the textTime to First Token — The time from sending a request until its first output token arrives.
Back to the textToken decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the textKV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.
Back to the textTool call — A structured request from a model for an external function such as reading a file, searching, or running a command. The agent runtime and its permission settings decide whether the request is actually executed.
Back to the text
Frequently asked questions
- What should you check first in this article?
- Prompt caching reuses computation state from the beginning of an already processed input, or prefix, in later requests. A server receiving shared instructions or the same document repeatedly can reduce input processing before the first answer. This guide sends a return policy twice using MLX-LM to examine input reuse, memory, and waiting time.
- How much memory does it need?
- This exercise uses MLX-LM and mlx-community/Qwen3-4B-4bit on an Apple Silicon Mac. The model card's 2.26GB is the download size. Execution also needs unified memory for weights, caches, and temporary computation.