Local AI Guide

See the speed before choosing hardware

Model setup recipes

Qwen3.8 27B in 7.89 GB? Install Saluki and Plan for an 8 GB GPU

Before switching to a smaller model, consider smaller weights for a 27B model.

Underdog Saluki 27B 1.0 compresses Qwen3.8-27B with roughly two-bit mixed quantization1. Load its 7.89 GB GGUF2 in llama.cpp to answer questions locally and serve replies to connected apps. Conversation state needs memory alongside the weights, so start with a short context and expand it after checking available memory.

Contents
Requirements and key details
  • Leave memory for KV cache and the runner alongside the 7.89 GB GGUF.
  • Start with standard llama.cpp; NVIDIA's NInfer uses a separate converted artifact.
  • Confirm the setup with a short prompt, then try your documents and tool calls.

Saluki puts Qwen3.8 27B into a smaller file

If memory has kept you on smaller models despite wanting to try Qwen3.8-27B, Saluki offers another option. It keeps the 27 billion parameters while reducing the bits used to store their weights. The download is a 7.89 GB GGUF, a format supported by many local runners.

Here we load the model in llama.cpp and start a server on port 8080 of your own computer. Once ready, it can connect to web chat or an OpenAI-compatible app. The weights are released under Apache 2.0, supporting personal workflows and services that follow the license terms.

A laptop, black PC, connected external SSD, and notebook on a desk
Try a compact 27B GGUF on the computer you already have.

Budget for conversation state on an 8 GB GPU

A 7.89 GB file looks close to an 8 GB graphics card's capacity, but running it uses more than the weights. KV cache3 for the conversation, intermediate calculations, and display applications share GPU4 memory. A long document can exhaust the remaining space early.

The example below limits context to 4,096 tokens5 and chooses the GPU load automatically. Parts that do not fit in GPU memory can use system RAM6, which must also have room for the model and operating system. Start with your existing 8GB GPU, then use Task Manager or Activity Monitor to check RAM and GPU memory use and the time to the first answer.

On Apple Silicon, the CPU7 and GPU share unified memory8. Close other apps and start with a short prompt9 to see how much room your Mac has. If longer conversations sharply increase swap or slow replies, reduce the context.

A dual-fan graphics card, external SSD, and open notebook
Leave room for conversation state and computation after loading the model.

Prepare a folder and download the model

On Mac or Linux, prepare Python 3, Git, and CMake. Mac also needs Xcode Command Line Tools. If you use Homebrew, brew install git cmake installs the build tools; run xcode-select --install and complete its installation dialog for Xcode tools.

These commands create a Python environment inside saluki-local and save the model under models. The weights alone download nearly 8 GB; leave additional disk space for the runner and temporary files. Run the following steps from the same saluki-local folder.

Mac/Linux: working folder and GGUF download
mkdir -p saluki-local
cd saluki-local
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -U huggingface_hub
hf download ConwayResearch/Underdog-Saluki-27B-1.0 Underdog-Saluki-27B-1.0-IQ2-mix.gguf --local-dir models
The IQ2-mix.gguf file appears in models when the download finishes.

Use Metal on Mac and CUDA on NVIDIA

Building llama.cpp gives you a runner for your device. Mac builds enable the Metal10 GPU backend by default. After the Mac commands finish, llama.cpp/build/bin/llama-server is the executable for the next step.

On Linux with an NVIDIA GPU, install the driver and CUDA11 Toolkit, then use the CUDA build commands instead of the Mac commands. Check that nvidia-smi lists the GPU and nvcc --version reports the toolkit before building.

On Windows, extract a CUDA build from the llama.cpp releases and run llama-server.exe in PowerShell. Replace the executable path in the server command below. A source build also needs Visual Studio C++ tools and the CUDA Toolkit.

Mac: build the Metal runner
git clone https://github.com/ggml-org/llama.cpp.git
cmake -S llama.cpp -B llama.cpp/build
cmake --build llama.cpp/build --config Release --target llama-server -j 4
NVIDIA Linux: build the CUDA runner
git clone https://github.com/ggml-org/llama.cpp.git
cmake -S llama.cpp -B llama.cpp/build -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release --target llama-server -j 4

Start the server and send a first prompt

Start the server in one terminal. Wait for model loading and the listening message, then open a second terminal. Once /health reports a healthy state, send the prompt below. Binding to 127.0.0.1 keeps access on the current computer.

For this short setup check, thinking is disabled so the answer can arrive directly. A sentence in choices[0].message.content confirms generation. Next, replace the question with an excerpt from a document you normally summarize. For tasks needing longer reasoning, set enable_thinking to true and allow more output tokens.

Terminal 1: local server
./llama.cpp/build/bin/llama-server -m models/Underdog-Saluki-27B-1.0-IQ2-mix.gguf --alias saluki27b --jinja -ngl auto -fa on -c 4096 --host 127.0.0.1 --port 8080
Leave this terminal running. Ctrl+C stops the server.
Terminal 2: health check and first request
curl -sS http://127.0.0.1:8080/health
curl -sS http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"saluki27b","messages":[{"role":"user","content":"Explain local AI in two sentences."}],"max_tokens":128,"temperature":0,"chat_template_kwargs":{"enable_thinking":false}}'

Test tool calls and math tasks separately

If you want Saluki to power an agent as well as summarize documents, test ordinary replies and tool calls12 separately. In a tool call, the model selects a function name and arguments: for example, a search phrase for a notes function, followed by a reply based on the results returned by the app.

In the developer's 120-task tool-call run, Saluki passed 88 tasks and the original Qwen3.8-27B passed 84. For complex mathematical reasoning, compare solution quality alongside memory savings. Saluki scored 79.2 on AIME 2025; the original model's public reference score was 96.7.

Tool-call tasks passed by Saluki and the original
ModelTasks passed
Underdog Saluki 27B 1.088 / 120
Qwen3.8-27B84 / 120

Developer evaluation · Tool calls: 120 UnderdogBench tasks, thinking off, temperature 0. AIME 2025: Saluki average@4; the original's public reference uses a different evaluation setup.

Start with read-only tools such as notes search rather than deletion or payments. Keep roughly ten prompts from your real work so you can compare search phrases, argument formatting, and final replies when changing models.

A landscape photo, documents, and pen beside a laptop
A fixed set of real prompts makes replies and tool calls easier to compare.

Add a separate file for image input

Once text prompts work, you can add image input. Download an mmproj file beside the weights and pass it to the server so images can be encoded for the model. The Q8_0 file is 629 MB; F16 is 928 MB. Q8_0 is a smaller starting option.

After downloading, stop the running server with Ctrl+C and append --mmproj models/mmproj-Underdog-Saluki-27B-1.0-Q8_0.gguf to its command. Restart, attach a simple screenshot in the web chat if image upload is available, and ask for a summary. Add this after checking the memory left by text-only operation, since image processing also consumes memory.

Download the image projection file
hf download ConwayResearch/Underdog-Saluki-27B-1.0 mmproj-Underdog-Saluki-27B-1.0-Q8_0.gguf --local-dir models

Resolve the first-run problems in this order

For hf: command not found, reactivate the environment with source .venv/bin/activate. If the command is still missing, run python -m pip install -U huggingface_hub again. For a missing model file, check that the current folder is saluki-local and the path begins with models.

If auto is not recognized or the model format fails to load, update llama.cpp to a current release first. For GPU memory errors, close GPU-using apps and try a context below 4,096. If loading still fails, replace -ngl auto with -ngl 0 to test from system RAM; the extra CPU work can make replies slower.

For connection refused, check that the server terminal is still running and using port 8080. If another app occupies that port, change both --port and the request URL. Ctrl+C stops the server when you finish; the model stays on disk for the next run.

For faster answer generation on NVIDIA, continue with the NInfer guide. If long documents delay the first answer on Mac, see the Splash guide for ways to reduce input-processing time.

Terminology notes

  1. Quantization — A way to represent model weights or values with fewer bits, reducing storage and memory use. The coarser representation can also affect accuracy.

    Back to the text

  2. GGUF — A file format for model data, widely used by llama.cpp-based tools.

    Back to the text

  3. KV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.

    Back to the text

  4. GPU — A processor designed to handle many calculations in parallel. It performs model computations during AI inference.

    Back to the text

  5. Token — A unit into which a model divides input or output for processing.

    Back to the text

  6. System RAM — System memory that temporarily holds data while programs run.

    Back to the text

  7. CPU — The central processor that runs the operating system and general-purpose instructions; in AI workloads it also handles tasks such as preprocessing and data movement.

    Back to the text

  8. Unified memory — An architecture where the CPU and GPU share one physical memory pool.

    Back to the text

  9. Prompt — The input text sent to a model. For generative models it describes the desired result and strongly shapes the output.

    Back to the text

  10. Metal — Apple’s low-level technology for graphics and parallel GPU computation.

    Back to the text

  11. CUDA — A software platform for general-purpose computing on NVIDIA GPUs.

    Back to the text

  12. Tool call — A structured request from a model for an external function such as reading a file, searching, or running a command. The agent runtime and its permission settings decide whether the request is actually executed.

    Back to the text

Frequently asked questions

What should you check first in this article?
Underdog Saluki 27B 1.0 compresses Qwen3.8-27B with roughly two-bit mixed quantization. Load its 7.89 GB GGUF in llama.cpp to answer questions locally and serve replies to connected apps. Conversation state needs memory alongside the weights, so start with a short context and expand it after checking available memory.
How much memory does it need?
If memory has kept you on smaller models despite wanting to try Qwen3.8-27B, Saluki offers another option. It keeps the 27 billion parameters while reducing the bits used to store their weights. The download is a 7.89 GB GGUF, a format supported by many local runners.
Where can you check the conditions and sources?
Here we load the model in llama.cpp and start a server on port 8080 of your own computer. Once ready, it can connect to web chat or an OpenAI-compatible app. The weights are released under Apache 2.0, supporting personal workflows and services that follow the license terms.