Set up Qwen3.8-27B prefill and caching with Splash 1.3.0

Starting Splash with the Qwen3.8-27B model ID prepares the weights and matching DFlash2 drafter. Once ready, it handles questions sent to the local address and returns answers.

When you send long code or a document, the model reads the input before producing its first token. This input-processing stage is called prefill. Decode⁸ then generates the answer token by token. For supported dense models, Splash 1.3.0 uses the Neural Engine for long prefill; Qwen3.8-27B answer generation runs on the GPU.

If the first token is slow after a long prompt, compare Neural Engine prefill; if the answer streams slowly, inspect decode speed separately. At startup, Splash calibrates for the Mac and model, then adjusts the split by prompt length.

You need an Apple M3 or newer Mac, macOS 26.4 or later, and Homebrew. The Qwen3.8-27B 4-bit example needs at least 36 GB of unified memory⁹; 48 GB is recommended. The first run downloads model and draft weights, so allow for disk space and an internet connection.

Install Splash with Homebrew, then start the server in a first Terminal window. Leave the server command running and wait for Ready. Run the API¹⁰ check below in a second Terminal window.

When the server terminal prints Ready, open http://127.0.0.1:8000 in a browser. Splash's chat page appears; send a prompt and receive an answer. By default, the server listens on a local address on this Mac. Closing the terminal or pressing Ctrl+C stops it.

Send the API request below from a second terminal to check the model connection. If the returned JSON's choices array contains answer text, the request was processed.

In oMLX 0.7.1rc1, Splash can do inference while oMLX keeps the model list and API. Install Splash 1.3.0 or later and open Settings → Integrations → External engines. Once the detected launcher path and version appear, prepare the Qwen3.8-27B model directory and the z-lab/Qwen3.8-27B-DFlash2 drafter.

For 1.3.0, use an affine 4-bit, group-64 MLX¹¹ checkpoint¹² such as mlx-community/Qwen3.8-27B-4bit. Version 1.3.1 also accepts mixed-bit checkpoints. For the oMLX external engine, select a supported MLX checkpoint rather than an older incoai/*-Splash prebuilt package.

Enable Use external engine in Model Settings → Advanced and select Splash. oMLX manages the loopback port and API key for the separate process. Send a short prompt and check prefill progress and generation speed on the dashboard. If it fails to start, check the launcher path in Integrations, the model format, then the drafter files.

Adjust the memory limit, KV precision, SSD cache and Neural Engine prefill in the Splash integration settings. Splash uses its own DFlash2 and cache settings. Leave Lightning MTP¹³ or TurboQuant settings for other oMLX models unchanged and compare TTFT¹⁴ and completion time for the same document on Splash and the default engine.

When you omit --disable-ane, Splash 1.3.0 can split FFN computation between the GPU and Neural Engine for supported dense targets. At startup it calibrates the split for the Mac and model, chooses prompt lengths that benefit, and remembers the result. If evaluation fails, times out, or produces a non-finite value, it reruns the affected chunk on the GPU.

The Splash 1.3.0 release records Qwen3.8-27B measurements comparing prefill and time to first token (TTFT) with MLX 4-bit weights, one request, and a cold 14,096-token prompt. M5 Max took 14.09 seconds on GPU alone and 12.36 seconds with GPU plus Neural Engine; M5 Pro took 28.28 and 21.57 seconds, respectively. TTFT covers input processing through the first token; compare it separately from answer-generation speed.

Prefill splitting uses some memory that could otherwise hold the KV cache¹⁵. The context limit for long inputs can be lower than with GPU-only operation, so check the startup summary and /status field maximum_context_tokens on your Mac. Compare decode throughput¹⁶ separately for the speed of streaming answers.

A follow-up prompt that repeats the same document or instructions may require the model to read the opening again. Splash 1.3.0 can reuse a completed prefill checkpoint across conversations with a long shared prefix, resuming near the point where the second conversation diverges. In a published 19K-token shared-prefix case, the second conversation's TTFT fell from 39.6 to 6.3 seconds, reusing 16,384 tokens.

In that record, one additional reclaimable restore state for the 27B model used 187 MiB. With disk caching enabled, states evicted from memory can be written to the SSD. Set a quota such as --max-cache-disk 16G to use SSD caching for the current server session; add --persistent-cache to keep it across restarts.

Start without disk options and send two conversations with the same long document to check shared-prefix reuse. Add disk caching only if memory remains constrained or states are being evicted, after checking available SSD space. SSD caching does not increase the context limit, and a quota smaller than the working set can cause repeated reads and writes.

How much cache is reused depends on the length of the shared prefix. Send the same model, document, and question twice, then record TTFT for the first and repeated requests separately.

First match prefix and SSD cache reuse conditions. Record TTFT for the same model and long input with GPU plus Neural Engine, restart with --disable-ane, then measure GPU alone. Separate initial and repeated runs in each configuration to identify calibration and compilation costs.

The first Neural Engine startup can take longer due to program compilation and split calibration. In /status, inspect ane_ffn for split state, share, minimum input-chunk size, evaluations, and GPU reruns. For short prompts that do not benefit, Splash can select the GPU-only path automatically.

If the server does not become ready, check the first terminal error and macOS version. If the browser cannot connect, check for Ready and whether port 8000 is available. If the model cannot start because of context length¹⁷, inspect the automatic limit in the startup summary and /status, then lower it with --max-context if needed. Use --disable-ane to return to GPU-only prefill if Neural Engine operation is unstable or you need a fixed baseline.

Keep the default path when Neural Engine reduces TTFT for your long prompt and the available context still fits your work. Choose GPU-only with --disable-ane if context becomes too short or results are unstable.

The first splash serve downloads the selected Qwen weights and their matching DFlash2 draft weights from Hugging Face. The Qwen3.8-27B 4-bit example can use up to about 21 GB for both downloads. If progress appears stalled, check the last progress line in Terminal and your internet connection before stopping the server. A later start reuses files already downloaded.

If Terminal says it cannot find the model repository, check the value passed to --model. The MLX ID in this guide is mlx-community/Qwen3.8-27B-4bit. Private or gated repositories require you to sign in to Hugging Face or set HF_TOKEN. Correct the model name or access first, then rerun the same command.

If the weights finish downloading but the memory budget check fails, you do not need to download them again. The selected model and context length exceed the Mac's available memory budget. Restart with a lower --max-context; on a 24 GB Mac, choose a smaller supported GGUF¹⁸ variant from the model list.