Model setup recipes
A safe local starting point for MiniCPM5-2B Q4_K_M
MiniCPM5-2B is a dense 2.51675648B-parameter model with official GGUF, 4-bit MLX and GPTQ releases. This guide uses the official Q4_K_M GGUF and llama.cpp's llama-server as a practical baseline for a typical computer. The site's speed and memory figures are estimates derived from hardware specifications, so validate them with the short local test below.
Serving recipe
Settings for your hardware
For one user
Choose your hardware to see the launch command and settings you can adjust.
Profile
Apple MacBook Pro M5 Pro (64GB)
Memory fitsRecommended runtime: LM Studio (4-bit MLX (a different format from GGUF Q4_K_M)) · Documented runtime settings
For your first run or reliable everyday use
Profile context length
16,384 tokens
A starting setup based on the runtime documentation. Follow the steps below and compare speed as you make changes.
Estimated decode speed
148.3 ~ 164.8 tok/s
Estimate for one user
Estimated TTFT
5.7 to 12.3 seconds
Prefill: 1,331 ~ 2,890 tok/s
Estimated memory use
About 2.59GB / 56GB
Remaining margin: approx. 53.4GB
Server address
http://127.0.0.1:1234
Local only · 127.0.0.1
Estimates use the calculator’s recommended quantization and acceleration with 16,384tokens. They are not measurements of the configuration above.
Settings in this profile
| Value | Why this setting |
|---|---|
| Model weights4-bit MLX (a different format from GGUF Q4_K_M) | Use an MLX model directory. This is a different file format from Q4_K_M GGUF. |
| Check input length16,384 | Check both the context-length setting in lms load and the context actually loaded. |
| KV cacheCheck model settings | This launch command does not set the KV cache precision. |
| Prefill batch sizeCheck runtime settings | Do not use llama.cpp's batch-size and ubatch-size values as substitutes for MLX settings. |
| Concurrent requests1 | Measure the speed of one request, separately from the combined throughput of multiple requests. |
| Memory headroom2 GB or more | After loading the model, check actual memory use and whether the system is swapping. |
MBP M5 Pro 64GB Run command (LM Studio)
# lms ls에서 확인한 MLX 4비트 모델 식별자를 입력
MODEL_ID=""
lms ls
: "${MODEL_ID:?MLX 모델 식별자를 입력하세요}"
lms load "$MODEL_ID" --gpu=max --context-length=16384
lms server start --port 1234Checks after startup
- 01Check the startup log for completed model loading and the actual context length.
- 02After warming up, send one request at a time. Use three different inputs of the same length to record fresh-input processing time.
- 03Record repeated-input results separately as cache-reuse runs. Do not combine them with fresh-input results.
- 04Record prefill tok/s, decode tok/s and peak memory together, then change one setting at a time.
Trade-offs
- Conservative context and concurrency settings may produce less than the hardware's maximum throughput.
Adjust one item at a time
Speed tuning, step by step
If several values are changed at once, it is difficult to find the cause.
- STEP 1
Save a baseline
Record first-input and cache-reuse runs separately, three times each. Compare prefill, decode and peak memory together.
When to stop: Do not move on if there are errors or less than 2GB of memory headroom.
- STEP 2
Tune prefill chunk size
Increase in steps of 1,024 → 2,048 → 4,096, comparing TTFT and peak memory on long inputs.
When to stop: Revert to the previous value if TTFT does not improve or peak memory rises sharply.
- STEP 3
KV Cache Tuning
Only when you need more context, compare Q8 cache against the F16/BF16 baseline using the same question.
When to stop: Keep the default precision if the output changes or you already have enough context.
- STEP 4
MTP/speculative decoding
Enable it only for supported models. Test code and prose separately, measuring acceptance rates and actual decode tok/s.
When to stop: Turn it off if the median of three runs does not improve on the baseline.
- STEP 5
Context expansion
Double context length at each step until you reach what you need. Check retrieval from the middle of the input and whether swapping occurs.
When to stop: Reduce by one step if retrieval accuracy drops or swapping or memory compression starts.
Submit a measurement from my hardware
Import a JSON file with at least three runs under the same conditions. The file stays in this browser until you submit it.
Include only hardware, model, runtime settings and measurements. Do not include prompts, responses, raw logs or file paths.
If you do not have a measurement file, use the tool with your running local server. It requires Node.js 20 or later and does not upload results automatically.
These are reviewed community submissions, not measurements made by this site.
Report a setup issue
Tell us where this setup failed. Only the site administrator can read your report.
Setup being reported · MBP M5 Pro 64GB · MiniCPM5-2B · Balanced
Download the exact official Q4_K_M file
The official Q4 filename is MiniCPM5-2B-Q4_K_M.gguf. The command below downloads only that file into models/minicpm5-2b, avoiding the extra space required by the F16 and Q8 files in the same repository. When it finishes, verify that the downloaded path and filename match the command.
hf download openbmb/MiniCPM5-2B-GGUF MiniCPM5-2B-Q4_K_M.gguf --local-dir ./models/minicpm5-2b
Start a loopback-only server at 4,096 tokens
After installing a current llama.cpp build, run the command below from the repository root. --host 127.0.0.1 keeps the server reachable only from the same computer, while -c 4096 fixes the first test at a 4,096-token context. This is not the model's native limit of 131,072 tokens. Increase context gradually only after a short request completes reliably.
Once the server starts, open http://127.0.0.1:8080 in a browser on the same computer. -ngl 99 requests GPU offload, so check the startup log for a CUDA- or Metal-enabled build. The first request can be slower because it includes file loading and kernel preparation. Send a warm-up request, then three different questions of the same length and record prefill and decode separately. Repeating only the same question can mistake cache reuse for intrinsic model performance. If hf is unavailable, install the Hugging Face CLI first; if llama-server is unavailable, check the path to the llama.cpp executable.
llama-server -m ./models/minicpm5-2b/MiniCPM5-2B-Q4_K_M.gguf -ngl 99 -c 4096 --host 127.0.0.1 --port 8080
MLX is a separate model and runtime path
On Apple Silicon, you can choose the official 4-bit openbmb/MiniCPM5-2B-MLX release. It is not a GGUF file and should not be passed to the llama-server model path. Use that model ID with an MLX-compatible runner, and hold context length and prompt constant when comparing it with GGUF. Because the formats and kernels differ, a measured speed from one path is not a measured result for the other.
DSpark requires a separate draft model
The official MiniCPM5-2B instructions provide a separate MiniCPM5-2B-DSpark draft checkpoint and a compatible SGLang path. This is not generic MTP built into the target model, nor a named switch that can be enabled on the Q4_K_M GGUF server. Establish a baseline first, then match the DSpark checkpoint, SGLang version and launch command together. This site does not apply a DSpark speedup until device-specific measurements are verified.
Change log
These entries record changes to the site's guidance. They do not automatically check your installed engine or model version.
MiniCPM5 setup recipe added
The guide distinguishes official GGUF launch commands from the MLX release. DSpark speedups are not applied until device-specific measurements are verified.