What is Crane? Run LLM, OCR and audio models behind a local Rust API

Imagine a chat server working in its virtual environment until an OCR package conflicts with a dependency, while a speech demo wants another Python version. The friction is not the models themselves, but the separate execution environments. Crane takes a different approach: manage text inference, image input, OCR and audio code within one project built around Rust and Candle.

Rust catches many errors during compilation and packages programs as executable binaries. Candle supplies tensor operations and model components in Rust, letting Crane keep application code and inference in the same language ecosystem. That is an implementation and deployment advantage, not a promise that every model will run faster automatically.

Ollama simplifies downloading a model and using its local API. MLX is a machine-learning framework for Apple silicon, used by applications such as oMLX. Crane brings Candle's Rust model implementations together with several device backends. It is worth considering for inference inside a Rust app or for running the same server program with different models. The basic setup loads one model per process. To use chat and speech together, run the required servers separately on different ports.

You point Crane's crane-serve at a model, then send OpenAI-shaped chat requests from an existing program. This is less about replacing the convenience features of a desktop app and more about adding a server layer you can control between an application and its model. The project's Metal support on Mac does not imply every model uses Metal; specific checkpoints have their own device support, as with Bonsai 2 below.

The steps below target the Crane revision reviewed on September 23, 2026. Cloning the repository and checking out that commit makes the commands and model support less likely to shift under later main changes. Prepare the Rust toolchain, Xcode Command Line Tools on macOS, and a compatible CUDA toolkit on Linux when compiling for NVIDIA CUDA.

Run only one of the three build commands below: Metal and Accelerate for Mac, CUDA for Linux with an NVIDIA GPU, or CPU for a test without GPU acceleration. Model weights are not part of the build; download them in the next step. The official install.sh helps choose these features, but explicit Cargo commands make the selected configuration visible.

For the first run, use the official Hugging Face repository Qwen/Qwen3.5-0.8B. Crane's README lists Qwen 3.5 0.8B as supported, and the model card provides the safetensors repository and license. This checkpoint includes a vision encoder, so pass crane-serve's --text-only option when testing text chat without the vision component.

Download the model using the Hugging Face CLI. If uv is already installed, use uvx hf below; uv manages the tool's isolated environment, so you do not need to create one yourself. Otherwise choose a suitable installation method in the CLI guide linked below. Once the download finishes, check that the model directory contains its configuration, tokenizer and weight files.

After downloading the model, start crane-serve. --model-name crane-local sets the name used in API requests, and --context 4096 sets the context limit for this test. Passing --host 127.0.0.1 restricts access to clients on the same computer. Crane's default host is 0.0.0.0, so keep this option in the command.

When the server starts, check the address in the terminal. If the built-in page does not open automatically, visit http://127.0.0.1:8080/ in your browser. --ui serves Crane's built-in browser interface. With the text-only model loaded, start with a short prompt and make sure the response completes.

A direct HTTP request lets you inspect the connection between the server and an app. curl -N displays streamed output without buffering, and the request's model value must match the crane-local name from the launch command. For supported Qwen templates, chat_template_kwargs.enable_thinking turns thinking mode off.

In an OpenAI-compatible app, set the base URL to http://127.0.0.1:8080/v1 and model name to crane-local. The example above is local-only with API authentication disabled. If the app requires an API-key field, a placeholder such as local can be used; it does not secure the server. Once connected, try an everyday question in your language or a short code explanation.

Once chat works, restart the server with the model type for your next task. Crane's repository lists PaddleOCR variants, Qwen3-ASR and Qwen3-TTS, while the crane-serve docs describe audio transcription and speech-generation paths. Audio servers use different model types and inputs from chat, so take the options and request format from the corresponding README section and test with one sample.

Before sending real files or work material, try short, non-sensitive inputs. For OCR, see whether text in a photo is captured; for ASR, whether Korean speech is transcribed; for TTS, whether it speaks the language you need. Even when each task uses a different model, you can reuse the same server pattern in your app.

Bonsai 2's Prism release is a GGUF file containing PTQ1_0 or PQ2_0 ternary weights. Crane was updated to recognize those dedicated GGUF types and use its TernaryLinear execution path for the affected weights. When downloading, choose the specified Bonsai 2 Prism file rather than a generic Q4 GGUF.

The official support note names CPU and NVIDIA CUDA for this model. That differs from Qwen3.5's Metal path, so building Crane with Metal on a Mac does not mean Bonsai 2 receives the same GPU acceleration. Do not estimate memory from the 27B parameter label alone; check the selected Prism file size and available memory with the 4096-token context setting.

This guide does not include a direct Crane benchmark. On your machine, separate model loading and warm-up from measurement, then record the median of three matching requests. Track time to first token, subsequent generation speed and total completion time to see whether input processing or answer generation changed.

An identical question is not a fair comparison if only one engine enables thinking or uses a different output limit. Match the model, quantization, context and output cap; separate fresh requests from cached follow-ups. Record format differences such as MLX versus GGUF. A faster answer that omits important content is not sufficient reason to switch. The useful outcome is less environment upkeep while reliably completing your everyday task.