Bonsai 2 27B: what to check before the 6 GB headline

Bonsai 2 27B is derived from Qwen3.8-27B. Most weights are represented with the three values -1, 0 and +1, while groups of 128 and a Hadamard rotation reduce quantization error. The smallest PTQ1_0 language weights are 5.95 GB; the official demo defaults to the 7.21 GB PQ2_0 pack. The MLX bundle is 8.60 GB including the vision tower. So the 6 GB claim is real, but it does not describe total working memory. Longer documents grow the KV cache, and the operating system and runtime buffers use the same memory. A file fitting on an 8 GB machine is not the same as a comfortable conversation.

Loading the packed weights is only half of the computation. Bonsai 2 rotates activations before each linear layer, so the kernel must understand that transform. The setup script in the official demo downloads patched llama.cpp binaries with this support. Stock llama.cpp may reject the file, and some combinations can load it yet produce meaningless text. MLX likewise needs a Bonsai-aware loader. On the first run, use a short text prompt before a long document or vision input. Checking the runtime and checkpoint pair is usually more useful than assuming the model file is damaged.

PrismML published cross-platform results using the same PQ2_0 pack at batch 1. For 128 generated tokens, RTX 5090 reached 129.9 tok/s, RTX 4090 81.2, RTX PRO 6000 Blackwell 124.8 and Apple M5 Pro 28.1. Prompt processing at 512 tokens measured 3,893, 3,124, 4,020 and 387 tok/s respectively. These are vendor measurements without the vision tower, so longer prompts, operating systems and concurrent requests will differ. Our speed experience uses these points directly on matching hardware and widens the range for inputs of 4K tokens or more. That makes both first-token delay and ongoing generation visible instead of copying one peak number.

PTQ1_0 is smaller at 1.75 bits per weight, but it is not always faster. Heavier packing adds unpack work during inference, so decode can be similar or only slightly faster depending on the device. PQ2_0 was faster for prompt processing on every published comparison system. If you read long documents or use RAG, saving 1.26 GB may cost more time before the first answer. PQ2_0 is therefore the better starting point when the machine has more than 8 GB of usable headroom. Try PTQ1_0 only when memory is genuinely tight, and compare ten identical prompts for both answer consistency and first-token latency.

Across the vendor's fourteen thinking-mode evaluations, Bonsai 2 PQ2_0 averaged 84.78 while FP16 Qwen3.8-27B averaged 86.32, or 98.2% in aggregate. That is notable, but it does not promise 98.2% quality for every Korean prompt or coding task. An average blends strong and weak areas, and your prompts are not the benchmark set. Before choosing hardware, collect the questions, long documents and code edits you use most, then compare them beside the original or a Q4 build. Bonsai 2 is most valuable when a smaller file makes hardware you already own useful again. Preserve that advantage by proving it on your work before paying for another machine.