Executable programs and extensions

Qwen3.8-27B on a 16 GB GPU: choosing ExLlamaV3 and EXL3

The default 16 GB profile is EXL3 2.5 bpw with about 176K context and image input. The 2.0 bpw option offers a longer 229K window with a different quality trade-off.

MIA's serving kit serves Qwen3.8-27B EXL3 on NVIDIA GPUs with at least 12 GB VRAM¹ and compute capability 7.5. Setup reads VRAM and proposes a bits-per-weight level, context, and image support. The default 16 GB profile is 2.5 bpw with images at 176K; the default 24 GB profile is 4.0 bpw with images at 262K. A 32 GB card shows the same menu. The profile planner prints the selected file size and planned context.

EXL3 adjusts storage size and weight precision together

ExLlamaV3 is an inference² engine for NVIDIA GPUs, and EXL3 bpw describes the average bits per weight used to store model weights. Lower bpw reduces the file and weight memory, leaving room for a longer context, while moving farther from the original BF16³ representation. Choose VRAM, input length, and image support as one profile.

The kit requires at least 12 GB VRAM, compute capability 7.5 or newer, and NVIDIA driver 570 or newer. Prepare Python 3.11 or newer and run the installer for Windows or Linux. You do not need to install the CUDA⁴ Toolkit or packages into system Python.

A desk with a large monitor showing code-like lines, a desktop tower, and a laptop — Choose bpw, context, and image use together in the VRAM profile.

Compare default profiles and measured limits by VRAM

The profile menu calculates combinations after reserving memory for the driver. On 16 GB, 3.5 bpw offers 78K text-only, 3.0 bpw offers 118K with images, 2.5 bpw offers 176K with images, and 2.0 bpw offers 229K with images. The default does not simply select the highest bpw; it also accounts for image support and a useful context length⁵.

VRAM profile options from the official planner
GPU VRAM	bpw	Context and images
12GB	2.0	33K · text-only
16GB	2.5 default · 3.0 · 3.5 · 2.0	176K images · 118K images · 78K text · 229K images
24GB	4.0 default · 5.0 · 6.0	262K images · 180K images · 84K text
32GB+	Same menu as 24 GB	Capped at measured prefill limits

The 24 GB and 32 GB+ menu includes 6.0 bpw text-only at 84K, 5.0 bpw with images at 180K, and 4.0 bpw with images at 262K. The planner prints the actual download size and context, so you can preview the 16 GB profiles before setup. The selected profile writes the repository, revision, context, KV cache⁶, GPU⁷ memory budget, and vision state to .env.

VRAM profiles and context table

A dark monitor, a compact desktop computer, and a PC case with several visible fans — The profile planner calculates context options for the card's VRAM.

Choose a download with the profile planner and serve it

First use Code → Download ZIP in the repository and extract the kit into Downloads. The paths below assume Downloads/Qwen3.8-27B-16gb-NVIDIA-GPUs-one-click-install-main; replace that path if you extract elsewhere. Run the following commands from the kit root, where tools/profiles.py, linux, and windows are visible.

Enter the extracted kit folder on Linux

cd "$HOME/Downloads/Qwen3.8-27B-16gb-NVIDIA-GPUs-one-click-install-main"
ls tools/profiles.py linux/setup.sh windows/START-HERE.bat

Enter the kit folder in Windows PowerShell

Set-Location "$HOME\Downloads\Qwen3.8-27B-16gb-NVIDIA-GPUs-one-click-install-main"
Get-Item .\tools\profiles.py, .\linux\setup.sh, .\windows\START-HERE.bat

To inspect profiles before installation, run tools/profiles.py from the kit folder with system Python; this query needs no extra packages. Start installation with windows\START-HERE.bat on Windows or ./linux/setup.sh on Linux. After setup, the kit contains .venv⁸ and the model files.

Preview profiles for a 16 GB card

python3 tools/profiles.py --list --vram 16
# Windows PowerShell: py -3 tools\profiles.py --list --vram 16

After installation, start the server with ./linux/start.sh on Linux or windows\start.bat on Windows. The local API⁹ is at http://127.0.0.1:8888/v1. Read the actual model ID from /v1/models, then send a short request with that ID from Bash or Windows PowerShell and check the response content. Node 22.19 or newer also enables the DeepSeek Harness UI.

Start installation on Linux

bash linux/setup.sh

Start installation on Windows

& .\windows\START-HERE.bat

Start the serving server on Linux

bash linux/start.sh

Start the serving server on Windows

& .\windows\start.bat

Read the model ID, then send a first API request

curl -fsS http://127.0.0.1:8888/v1/models
# Copy an actual id from the response into MODEL_ID.
MODEL_ID='paste-the-id-returned-by-v1-models'
curl -fsS http://127.0.0.1:8888/v1/chat/completions -H 'Content-Type: application/json' -d "{\"model\":\"$MODEL_ID\",\"messages\":[{\"role\":\"user\",\"content\":\"Say READY in one word.\"}],\"max_tokens\":8,\"temperature\":0}"

Send the first request from Windows PowerShell

$models = Invoke-RestMethod http://127.0.0.1:8888/v1/models
$id = $models.data[0].id
$body = @{ model = $id; messages = @(@{ role = 'user'; content = 'Say READY in one word.' }); max_tokens = 8; temperature = 0 } | ConvertTo-Json -Depth 6
Invoke-RestMethod -Uri http://127.0.0.1:8888/v1/chat/completions -Method Post -ContentType 'application/json' -Body $body

Installation, profiles, and OpenAI API procedure

Check VRAM and model state after the first error

If the server reports Insufficient VRAM in split for model and cache, lower CONTEXT_SIZE or GPU_MEM_GB in .env and restart. If the image tower does not fit alongside the context, the Ready state shows Images: off. Reduce context or select a profile that includes images.

If the model list is empty, inspect download state with simplex models. If the error persists, simplex doctor checks Python, driver, ports, required .env values, model files, and free disk. One request runs at a time by default, so additional requests wait for the current reply.

Profile troubleshooting and simplex doctor

A laptop with a document window connected by cable to a phone displaying a page — Use the error log to check the memory budget and server state.

Terminology notes

VRAM — Memory used by a graphics card’s GPU to store model weights and intermediate values.
Back to the text
Inference — The process of using a trained model to compute an output for an input. Here, local inference means running the model on the user’s device.
Back to the text
BF16 — A 16-bit floating-point format for model values. Its exponent has the same width as FP32, giving it a similarly wide value range.
Back to the text
CUDA — A software platform for general-purpose computing on NVIDIA GPUs.
Back to the text
Context window — The token span of input and generated content a model can handle in one request. The supported limit and memory use depend on the model and runtime settings.
Back to the text
KV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.
Back to the text
GPU — A processor designed to handle many calculations in parallel. It performs model computations during AI inference.
Back to the text
Python virtual environment — An isolated space for installing Python packages per project, helping reduce package-version conflicts.
Back to the text
API — A defined interface that lets other code call a program’s functions.
Back to the text

Frequently asked questions

What should you check first in this article?: MIA's serving kit serves Qwen3.8-27B EXL3 on NVIDIA GPUs with at least 12 GB VRAM and compute capability 7.5. Setup reads VRAM and proposes a bits-per-weight level, context, and image support. The default 16 GB profile is 2.5 bpw with images at 176K; the default 24 GB profile is 4.0 bpw with images at 262K. A 32 GB card shows the same menu. The profile planner prints the selected file size and planned context.
How much memory does it need?: ExLlamaV3 is an inference engine for NVIDIA GPUs, and EXL3 bpw describes the average bits per weight used to store model weights. Lower bpw reduces the file and weight memory, leaving room for a longer context, while moving farther from the original BF16 representation. Choose VRAM, input length, and image support as one profile.