new model · 2026.09.20
Last updated
Bonsai 2 27B: what to check before the 6 GB headline
Run Bonsai 2 27B with an engine matching its compressed format
Bonsai 2 27B is a compressed Qwen3.8-27B-based model that answers questions in text. It reduces file size by representing most weights with three values. Run it in a dedicated llama.cpp or MLX1 runtime2 and add vision components for image understanding.
First choose whether to test a coding question or image understanding, then send a short prompt3 through a runtime supported by the official demo. Even if the 5.95 GB PTQ1_0 language-weight file fits on disk, KV cache4 and runtime workspace are additional. Being able to store a file is not the same as being able to generate answers reliably.
Distinguish file size from the complete runtime setup
Bonsai 2 27B is based on Qwen3.8-27B and represents most weights as −1, 0, or +1. It uses groups of 128 and Hadamard rotations to reduce bits while aiming to preserve accuracy. The complete model has 27.36B parameters: a 24.35B language backbone, 2.54B embeddings and output head, and a 0.46B vision tower.
The smallest PTQ1_0 language weights are 5.95 GB; the official demo defaults to PQ2_0 at 7.21 GB. The MLX bundle is 8.60 GB including a 0.92 GB vision tower, while the GGUF5 vision projector is a separate file of about 0.63 GB. These are disk sizes of files or bundles, not total runtime memory.
As prompts are processed, KV cache grows to hold context; Apple Silicon also uses unified memory6 for the model buffers and operating system. A model file fitting on the SSD7 of an 8 GB device does not establish that conversations will run reliably. Image tasks also need vision components and a runtime that connects them, beyond the language file.
Start with a short text prompt in the dedicated runtime
Bonsai 2 applies a Hadamard rotation to activations before linear layers. The runtime must handle this transform together with the compressed weights to produce valid output. A Q2 label does not mean the GGUF file can be opened by any Q2 loader.
The official demo installer prepares a patched llama.cpp build with the required support. Standard llama.cpp may reject the file or load it while producing corrupted text. MLX also requires a dedicated loader that understands the Bonsai conversion. Before downloading again or changing GPUs, verify that the checkpoint8 and runtime form a supported pair.
Begin with a short text question and verify that the answer is coherent; only then add a long document or image. Passing the text test first makes later failures easier to diagnose as memory issues, vision setup, or runtime incompatibility.
Separate prompt processing from generation in vendor timings
PrismML publishes separate PQ2_0, batch-1 figures for generating 128 tokens9 and processing a 512-token prompt. Generation tok/s is not time to first token10, so both measurements are needed to understand latency from submitting a prompt through response completion.
These vendor tests exclude the vision tower. Results on a personal setup vary with operating system, concurrent requests, actual prompts, and model configuration. Test a typical short question and long document separately, distinguishing time to first token from completion. For inputs of 4K tokens or more, the speed display shows a broader range.
| Device | 128-token generation (tok/s) | 512-token prefill (tok/s) |
|---|---|---|
| RTX 5090 | 129.9 | 3,893 |
| RTX 4090 | 81.2 | 3,124 |
| RTX PRO 6000 Blackwell | 124.8 | 4,020 |
| Apple M5 Pro | 28.1 | 387 |
Validate compression and public averages on your own tasks
PTQ1_0 is 1.75 bits/weight and 5.95 GB; PQ2_0 is 2.13 bits/weight and 7.21 GB. The files differ by 1.26 GB, but a smaller file does not always decode11 faster.
On the published devices, PTQ1_0 generated at a similar or slightly higher rate, but PQ2_0 processed prompts faster on every device tested. Start with PQ2_0 when more than 8 GB of memory is available; compare PTQ1_0 on the same prompts when memory is tight or you need a smaller file.
PrismML reports averages across 14 reasoning evaluations of 84.78 for PQ2_0 Bonsai 2 and 86.32 for FP1612 Qwen3.8-27B, claiming 98.2% by the ratio of average scores. This is a vendor evaluation, not a guarantee of 98.2% quality on every prompt. Compare typical code reviews, bug fixes, or documents under the same conditions and record correctness, omissions, consistency, time to first token, and completion time.
If the official demo's PQ2_0 and patched runtime work on your computer and answer real prompts adequately, keep that setup. If memory is the actual bottleneck, test PTQ1_0 on the same ten questions and compare answer quality with the space saved. Image tasks need a separate check including the vision projector.
Terminology notes
MLX — Apple’s machine-learning framework, which runs models using memory shared by the CPU and GPU on Apple Silicon.
Back to the textRuntime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.
Back to the textPrompt — The input text sent to a model. For generative models it describes the desired result and strongly shapes the output.
Back to the textKV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.
Back to the textGGUF — A file format for model data, widely used by llama.cpp-based tools.
Back to the textUnified memory — An architecture where the CPU and GPU share one physical memory pool.
Back to the textSSD — A data storage device that uses flash memory and retains data without power.
Back to the textCheckpoint — A file containing saved model weights and related state. Versions or tasks in one model family may use different checkpoints.
Back to the textToken — A unit into which a model divides input or output for processing.
Back to the textTime to First Token — The time from sending a request until its first output token arrives.
Back to the textToken decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the textFP16 — A 16-bit floating-point format. It uses the same bit width as BF16 but allocates bits differently between exponent and significand.
Back to the text
Frequently asked questions
- What should you check first in this article?
- Bonsai 2 27B is a compressed Qwen3.8-27B-based model that answers questions in text. It reduces file size by representing most weights with three values. Run it in a dedicated llama.cpp or MLX runtime and add vision components for image understanding.
- How much memory does it need?
- The smallest PTQ1_0 language weights are 5.95 GB; the official demo defaults to PQ2_0 at 7.21 GB. The MLX bundle is 8.60 GB including a 0.92 GB vision tower, while the GGUF vision projector is a separate file of about 0.63 GB. These are disk sizes of files or bundles, not total runtime memory.
Change log
These entries record changes to the site's guidance. They do not automatically check your installed engine or model version.
Added the dedicated Bonsai 2 runtime path and published performance
The guide now separates Ternary PQ2_0 from ordinary Q2 GGUF and requires a Hadamard-aware runtime. Vendor-published prefill and decode measurements are connected only to matching speed experiences.