Keeping models ready on your Mac with oMLX: choosing cache and acceleration settings

Start by checking your model files

oMLX runs MLX models on Apple Silicon Macs and makes them available to other apps through an API. GGUF and MLX conversions are not interchangeable just because the model name matches. Check the repository's format, quantization and required configuration files first.

You can use the official macOS app or a separately installed CLI. This guide assumes oMLX and your model are already installed. Place each model in its own subfolder inside the model directory, and the server discovers it for the model list.

Connect with one model and one request at a time

Start with one request at a time and the memory guard set to safe. This is a baseline for checking connectivity and replies, not a maximum-speed preset. Replace the model directory below with your actual path. If the app is already running a server, stop it first.

Check its status from another terminal. healthy means the server is ready; it does not verify every model's answers. loading may mean a pinned model is still being loaded. Once ready, choose a model at 127.0.0.1:8000/admin and try a short question.

omlx serve --model-dir "$HOME/.omlx/models" --host 127.0.0.1 --port 8000 --memory-guard safe --max-concurrent-requests 1

curl http://127.0.0.1:8000/health

Keep only your regular models loaded

Pin a model you use regularly, and set an idle timeout, or TTL, for occasional models. oMLX can also unload less recently used models when memory runs low. Pinning everything leaves less room to switch to another model.

A KV cache stores calculations for input the model has already read. Frequently used blocks stay in memory, while others can be kept on SSD and reused when the same prefix appears again. Measure the first question about a new document separately from a follow-up about the same document.

Flash-Next PLE SSD offload differs from this KV cache. It is a separate loader path that places the model's large embedding table on SSD and can be configured alongside Lightning MTP. Space requirements and speed depend on actual file sizes and page-cache state.

If memory warnings appear, shorten the context first

Your Mac shares unified memory with other apps. The oMLX memory guard considers current headroom and limits, and may delay new input processing, unload models or abort requests under heavy pressure. Disabling the guard does not reduce the memory your workload needs.

If a long input stalls, first reduce the context limit, concurrent requests and pinned models. Increasing the SSD cache alone cannot cover all the memory needed by model weights and active computation.

MTP and SpecPrefill target different parts of the wait

Lightning MTP drafts and verifies upcoming tokens to accelerate answer generation. It requires a supported model architecture, actual MTP weights and the setting enabled for that model. MTP in the name is not enough, and a greater draft depth is not always faster.

SpecPrefill is an experimental feature that uses a small draft model to select parts of a long input for processing. It needs a compatible draft model with the same tokenizer, and the eligible input remaining after cache reuse and other exclusions must exceed the configured threshold. Do not expect the same benefit for short questions or image inputs.

Native MTP and the Lightning path for a merged Gemma assistant can be configured separately from SpecPrefill. External-assistant VLM MTP, however, cannot currently run alongside SpecPrefill. Identify the execution path, not just the model name.

Being able to enable both settings does not mean every request benefits equally. Compare logs, time to first token, generation speed and answer content with each setting alone and both enabled. Check that selecting only part of the input does not omit important details.

What to record before switching servers

When comparing oMLX and MTPLX, first match the model conversion, context and acceleration settings. Comparing a repeated question on one with a first question on the other can mistake a cache advantage for an engine advantage. Ease of switching models and the speed of a single reply are also separate considerations.

Repeat tests with your own documents and code, recording the model file, runtime version, settings and timings. One peak result does not guarantee the next task's performance. Once connectivity works, continue to the model-specific recipe for settings relevant to your Mac.