Model setup recipes
Last updated
A safe local starting point for MiniCPM5-2B Q4_K_M
Summarize meeting notes locally with MiniCPM5-2B
MiniCPM5-2B is a language model that takes text and returns summaries or details such as owners and dates. Download the official GGUF1 Q4_K_M file and run a llama.cpp server with a 4,096-token context on a local address, then check whether the meeting-note summary matches the source.
Given a three-page meeting note, the model reads it and generates a summary and possible action items. Comparing whether decisions are preserved and owners and due dates actually appear in the source gives you a small baseline task.
Download just one official GGUF file
MiniCPM5-2B is a dense model with 2.51675648B parameters. Official distributions include GGUF, 4-bit MLX2 for Apple Silicon, and GPTQ; this starting path pairs MiniCPM5-2B-Q4_K_M.gguf with llama.cpp's llama-server. Keeping format and runtime3 fixed makes issues easier to diagnose.
Downloading only Q4_K_M instead of the whole repository avoids storing duplicate F16 and Q8 variants. After the download, match the file path and name to the server command below. Confirm model loading and request readiness in startup logs, then send a question.
hf download openbmb/MiniCPM5-2B-GGUF MiniCPM5-2B-Q4_K_M.gguf --local-dir ./models/minicpm5-2b
Start a local server with a 4,096-token context
In current llama.cpp, set the model path and use -c 4096 as the initial context size. This runtime setting is shared by input and output; it differs from the model's native context limit of 131,072 tokens4. Increase it for real document lengths after a short note works reliably.
--host 127.0.0.1 restricts access to the same computer. Once the server is listening, open http://127.0.0.1:8080 in that computer's browser. The -ngl 99 option requests GPU5 loading, so use a CUDA6 or Metal7 build and check the backend and number of GPU-placed layers in the logs.
llama-server -m ./models/minicpm5-2b/MiniCPM5-2B-Q4_K_M.gguf -ngl 99 -c 4096 --host 127.0.0.1 --port 8080
Check startup wait and summary accuracy separately
The first run reads model files and loads them into memory, so measure server startup separately from response time. After one warm-up request, submit the meeting note and distinguish initial setup, first request, and follow-ups on a ready server. Record whether you are measuring cache effects in one session or starting a new process for each request.
For speed, fix the question, input length, output budget, and backend, run the same request three times, and record the median. Separate prefill8, which reads input, from decode9, which generates the answer. Separately verify that the answer adds no unsupported decisions and that owners and due dates appear in the note.
MLX and DSpark use dedicated runtime paths
On Apple Silicon, you can choose the openbmb/MiniCPM5-2B-MLX 4-bit distribution, but it is not a GGUF file and should not be passed to llama-server. Use its model ID in an MLX-compatible runtime. For comparison, keep the note, question, context length10, and backend recorded; different formats and kernels make one path's speed inapplicable to another.
DSpark is a separate distribution requiring a dedicated MiniCPM5-2B-DSpark checkpoint11 and compatible SGLang path. Establish a baseline with the GGUF server first, then test the checkpoint, engine version, and command together. Do not apply acceleration multipliers without verified hardware-specific figures.
Terminology notes
GGUF — A file format for model data, widely used by llama.cpp-based tools.
Back to the textMLX — Apple’s machine-learning framework, which runs models using memory shared by the CPU and GPU on Apple Silicon.
Back to the textRuntime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.
Back to the textToken — A unit into which a model divides input or output for processing.
Back to the textGPU — A processor designed to handle many calculations in parallel. It performs model computations during AI inference.
Back to the textCUDA — A software platform for general-purpose computing on NVIDIA GPUs.
Back to the textMetal — Apple’s low-level technology for graphics and parallel GPU computation.
Back to the textPrefill — The stage where an LLM reads the input prompt and computes representations for its tokens. Longer prompts contain more tokens to process.
Back to the textToken decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the textContext window — The token span of input and generated content a model can handle in one request. The supported limit and memory use depend on the model and runtime settings.
Back to the textCheckpoint — A file containing saved model weights and related state. Versions or tasks in one model family may use different checkpoints.
Back to the text
Frequently asked questions
- What should you check first in this article?
- MiniCPM5-2B is a language model that takes text and returns summaries or details such as owners and dates. Download the official GGUF Q4_K_M file and run a llama.cpp server with a 4,096-token context on a local address, then check whether the meeting-note summary matches the source.
Serving recipe
Settings for your hardware
For one user
Choose your hardware to see the launch command and settings you can adjust.
Profile
Apple MacBook Pro M5 Pro (64GB)
Memory fitsRecommended runtime: LM Studio (4-bit MLX (a different format from GGUF Q4_K_M)) · Documented runtime settings
For your first run or reliable everyday use
Profile context length
16,384 tokens
A starting setup based on the runtime documentation. Follow the steps below and compare speed as you make changes.
Estimated decode speed
148.3 ~ 164.8 tok/s
Estimate for one user
Estimated TTFT
5.7 to 12.3 seconds
Prefill: 1,331 ~ 2,890 tok/s
Estimated memory use
About 2.59GB / 56GB
Remaining margin: approx. 53.4GB
Server address
http://127.0.0.1:1234
Local only · 127.0.0.1
Estimates use the calculator’s recommended quantization and acceleration with 16,384tokens. They are not measurements of the configuration above.
Settings in this profile
| Value | Why this setting |
|---|---|
| Weights4-bit MLX (a different format from GGUF Q4_K_M) | Use an MLX model directory. This is a different file format from Q4_K_M GGUF. |
| Check input length16,384 | Check both the context-length setting in lms load and the context actually loaded. |
| KV cacheCheck model settings | This launch command does not set the KV cache precision. |
| Prefill batch sizeCheck runtime settings | Do not use llama.cpp's batch-size and ubatch-size values as substitutes for MLX settings. |
| Concurrent requests1 | Measure the speed of one request, separately from the combined throughput of multiple requests. |
| Memory headroom2 GB or more | After loading the model, check actual memory use and whether the system is swapping. |
MBP M5 Pro 64GB Run command (LM Studio)
# lms ls에서 확인한 MLX 4비트 모델 식별자를 입력
MODEL_ID=""
lms ls
: "${MODEL_ID:?MLX 모델 식별자를 입력하세요}"
lms load "$MODEL_ID" --gpu=max --context-length=16384
lms server start --port 1234Checks after startup
- 01Check the startup log for completed model loading and the actual context length.
- 02After warming up, send one request at a time. Use three different inputs of the same length to record fresh-input processing time.
- 03Record repeated-input results separately as cache-reuse runs. Do not combine them with fresh-input results.
- 04Record prefill tok/s, decode tok/s and peak memory together, then change one setting at a time.
Trade-offs
- Conservative context and concurrency settings may produce less than the hardware's maximum throughput.
Adjust one item at a time
Speed tuning, step by step
If several values are changed at once, it is difficult to find the cause.
- STEP 1
Save a baseline
Record first-input and cache-reuse runs separately, three times each. Compare prefill, decode and peak memory together.
When to stop: Do not move on if there are errors or less than 2GB of memory headroom.
- STEP 2
Tune prefill chunk size
Increase in steps of 1,024 → 2,048 → 4,096, comparing TTFT and peak memory on long inputs.
When to stop: Revert to the previous value if TTFT does not improve or peak memory rises sharply.
- STEP 3
KV Cache Tuning
Only when you need more context, compare Q8 cache against the F16/BF16 baseline using the same question.
When to stop: Keep the default precision if the output changes or you already have enough context.
- STEP 4
MTP/speculative decoding
Enable it only for supported models. Test code and prose separately, measuring acceptance rates and actual decode tok/s.
When to stop: Turn it off if the median of three runs does not improve on the baseline.
- STEP 5
Context expansion
Double context length at each step until you reach what you need. Check retrieval from the middle of the input and whether swapping occurs.
When to stop: Reduce by one step if retrieval accuracy drops or swapping or memory compression starts.
Submit a measurement from my hardware
Import a JSON file with at least three runs under the same conditions. The file stays in this browser until you submit it.
Include only hardware, model, runtime settings and measurements. Do not include prompts, responses, raw logs or file paths.
If you do not have a measurement file, use the tool with your running local server. It requires Node.js 20 or later and does not upload results automatically.
These are reviewed community submissions, not measurements made by this site.
Report a setup issue
Tell us where this setup failed. Only the site administrator can read your report.
Setup being reported · MBP M5 Pro 64GB · MiniCPM5-2B · Balanced
Change log
These entries record changes to the site's guidance. They do not automatically check your installed engine or model version.
MiniCPM5 setup recipe added
The guide distinguishes official GGUF launch commands from the MLX release. DSpark speedups are not applied until device-specific measurements are verified.