Local speech recognition & synthesis
Last updated
A real-time voice assistant at home on an AMD Radeon: Ming Voice and its 20 GB VRAM budget
24 GB for bf16, 16 GB with int8 experts; below that the model does not fit.
Ming-3.0-audio-tiny is a 9.3B MoE1 model that answers speech with speech. We cover the 20.5 GB bf162 and 12.5 GB int8 splits, the conditions behind the 0.23 s figure, the install commands, and the Home Assistant link.
It answers with speech as soon as you stop
Ming Voice answers out loud when you speak into a microphone. Instead of chaining separate recognition, language, and synthesis models, it loads one speech-in, speech-out model, Ming-3.0-audio-tiny, onto the GPU3 and runs the conversation through it.
Smart Turn v3 decides whether you have finished from intonation and wording. If you keep talking after the answer starts, it withdraws that answer and treats everything as one turn. On speakerphone it filters out its own voice and stops only for yours.
VRAM decides which precision fits
The model is an MoE with 9.3B parameters in total and 2.7B active. Even so, the bf16 weights take 20.5 GB, which calls for a 24 GB card. On cards under 23 GB, the int8 expert path brings the footprint down to 12.5 GB.
A 12 GB card or laptop integrated graphics cannot hold it. Supported cards are Radeon RX 6800 and newer, and RTX 30 series and newer. At 16 GB the remaining headroom after the display, driver, and other apps is thin, so running it alongside a game or video export is difficult.
| Precision | VRAM | Card |
|---|---|---|
| bf16 | 20.5GB | 24GB |
| int8 experts | 12.5GB | 16GB |
Install on WSL2 or Linux
On Linux the installer runs from one command. It sets up Python and PyTorch4 for your card and downloads the 20 GB model. You do not install ROCm5 separately; PyTorch brings 7.2 with it. On Windows, run the installer from PowerShell and it prepares WSL2 as well.
The weights are about a 20 GB download, and install plus caches use roughly 30 GB of disk. The first launch after install spends one to two minutes compiling kernels; later launches load the model in about 30 seconds. After that you start talking from the browser.
# Linux / WSL2
curl -fsSL https://raw.githubusercontent.com/JMS1717/ming-voice/main/install.sh | bash
# Windows 11 (PowerShell)
irm https://raw.githubusercontent.com/JMS1717/ming-voice/main/install.ps1 | iexWhat conditions produced 0.23 s?
The project author measured it on a Radeon RX 7900 XTX with 24 GB. The median from end of speech to first audio was 0.23 s. The reply is written during the 0.2 s end-of-turn pause and is ready to play when that pause ends.
Do not expect the same figure on the int8 path of a 16 GB card. Lower precision lightens the math, but card generation and driver change the result a lot. Measure the time to first audio on your own machine, and shorten replies if it passes a second.
Connecting the devices in your home
Connecting Home Assistant lets you control lights, plugs, music, and climate by voice. Commands like "turn off the living room lights" go straight through without the model, and scheduled versions work too. Jellyfin playback connects the same way.
Other apps can reach it through the OpenAI-compatible API6 or the Wyoming7 bridge. With tools on, only search terms and place names leave the machine; turn tools off and it runs fully offline. The audio itself never leaves.
It does not run on a Mac
Apple Silicon is not supported. The kernels assume Triton with ROCm or CUDA8 and have no Metal9 path, so a Mac cannot load them. Building a voice assistant on a Mac means choosing a different model that runs on MLX10.
Supported languages are English and Chinese. Korean needs a different model, and because the code is MIT you can modify and redistribute it freely. The model holds VRAM11 while loaded, so sharing that GPU with other work is difficult.
Terminology notes
MoE — A model architecture that selects some of several expert subnetworks for each input. Total parameters can differ from the number activated for one token.
Back to the textBF16 — A 16-bit floating-point format for model values. Its exponent has the same width as FP32, giving it a similarly wide value range.
Back to the textGPU — A processor designed to handle many calculations in parallel. It performs model computations during AI inference.
Back to the textPyTorch — A software framework for building and running AI models. Check the compatible PyTorch version and hardware support along with the model.
Back to the textROCm — AMD’s software platform for AI and high-performance computing on GPUs. Support depends on the combination of GPU, operating system, driver, and framework versions.
Back to the textAPI — A defined interface that lets other code call a program’s functions.
Back to the textWyoming — The protocol Home Assistant and voice services use to exchange audio and text. It has no authentication or encryption, so it is used on trusted networks only.
Back to the textCUDA — A software platform for general-purpose computing on NVIDIA GPUs.
Back to the textMetal — Apple’s low-level technology for graphics and parallel GPU computation.
Back to the textMLX — Apple’s machine-learning framework, which runs models using memory shared by the CPU and GPU on Apple Silicon.
Back to the textVRAM — Memory used by a graphics card’s GPU to store model weights and intermediate values.
Back to the text
Frequently asked questions
- What should you check first in this article?
- Ming-3.0-audio-tiny is a 9.3B MoE model that answers speech with speech. We cover the 20.5 GB bf16 and 12.5 GB int8 splits, the conditions behind the 0.23 s figure, the install commands, and the Home Assistant link.
- How much memory does it need?
- The model is an MoE with 9.3B parameters in total and 2.7B active. Even so, the bf16 weights take 20.5 GB, which calls for a 24 GB card. On cards under 23 GB, the int8 expert path brings the footprint down to 12.5 GB.