Local speech recognition & synthesis
Last updated
What is FluidAudio? Local transcription, speakers, and speech on Mac
Add the transcription, speaker diarization, and speech generation your Mac app needs.
FluidAudio is a Swift SDK for running speech models on Apple devices. It can add ASR1, speaker diarization2, and TTS3 to local apps. Choose the supported language and model, then start by running the one feature you need.
Start by deciding what result you need from one short recording. Use ASR to read the words and diarization to separate speaking intervals by person. Each feature has its own models and language support, so choose from your actual audio and language. FluidAudio
Choose the language before running transcription
FluidAudio's default Parakeet v3 supports 25 European languages, while v2 is English-only. For Korean transcription, you can start with a multilingual WhisperKit model. Check that the model's supported-language list includes your language. Speech models
Prepare an Apple Silicon Mac and the Xcode Swift environment, matching the tools version in the source Package.swift for the CLI4 below. Put a short English audio.wav in the repository folder and run transcription first. The initial run includes model download and preparation.
git clone --depth 1 https://github.com/FluidInference/FluidAudio.git
cd FluidAudio
swift run fluidaudiocli transcribe ./audio.wav
Read FluidAudio's ANE path alongside each model's language list
The FluidAudio README describes offloading5 local inference6 to the ANE7 on Apple devices. ASR language support varies by model. Some Parakeet models support 25 European languages and Japanese, while Parakeet EOU provides real-time transcription for English.
TTS language support also varies by model. Check the model table for supported languages and real-time processing. FluidAudio README · API documentation
Diarization separates speaking intervals
Speaker diarization groups sections spoken by the same person and assigns a speaker ID. Match those IDs to the participants' names and align the time segments with the ASR transcript to create minutes showing who said what.
For a recorded meeting, start with offline mode to separate speakers. The command below finds each speaker's time segments in audio.wav. Compare overlapping speech and similar-sounding speakers with the recording to check that the segments are separated correctly. CLI guide
swift run fluidaudiocli process ./audio.wav --mode offline --threshold 0.6Generate speech from text
TTS converts a script into audio. FluidAudio offers paths such as Kokoro and PocketTTS. The example below saves a short English announcement to en.wav with Kokoro ANE v3. On macOS 15 or later, v3 uses a fixed hybrid route across CPU8, ANE, and GPU9.
swift run -c release fluidaudiocli tts "The meeting starts at four." \
--backend kokoro-ane --kokoro-version v3 --voice af_heart --output en.wavAfter it finishes, play en.wav in the working folder. Listen for the voice and pronunciation, then choose the variant and voice for your language. The Japanese example uses --variant ja --voice jf_alpha. Match the model and voice before supplying a script in another language.
Measure voice-assistant waiting time by stage
A voice assistant uses VAD10 to find speech, ASR to turn it into text, an LLM11 to produce the answer, and TTS to read it aloud. Record time for VAD, ASR, the LLM, and TTS separately to find where replies are delayed.
For file transcription, record total file completion time. For conversation, record the interval from the end of speech to the first audible answer. Run the actual app combination so GPU LLM and speech resource sharing is represented. Separate initial model preparation and check recognition and pronunciation on several familiar sentences.
Integrate only the features you need
Once CLI output is correct, add FluidAudio through Swift Package Manager and call the managers your app needs. Keep models loaded for repeated use, and separate model preparation from user requests to avoid unexpected downloads during a task.
For an offline-only app, prepare the models and set ModelHub.offlineMode = true. Missing files cause an error, so connect the exact model folder your distributed app will use and test a real request. Get transcription working in the required language before adding diarization or TTS one at a time to make failures easier to locate.
Terminology notes
Automatic Speech Recognition — Technology that recognizes spoken language in audio and converts it to text.
Back to the textSpeaker diarization — The task of dividing a recording into who-spoke-when segments and labeling them by speaker.
Back to the textText-to-Speech — Technology that converts text into spoken audio with pronunciation and prosody.
Back to the textCLI — Short for Command-Line Interface: operating a program by entering commands in a terminal.
Back to the textOffloading — Moving some model data from GPU memory to system RAM or storage when capacity is limited. This adds data transfer.
Back to the textInference — The process of using a trained model to compute an output for an input. Here, local inference means running the model on the user’s device.
Back to the textNeural Engine — Dedicated hardware in Apple Silicon for neural-network computation.
Back to the textCPU — The central processor that runs the operating system and general-purpose instructions; in AI workloads it also handles tasks such as preprocessing and data movement.
Back to the textGPU — A processor designed to handle many calculations in parallel. It performs model computations during AI inference.
Back to the textVoice Activity Detection — A process that detects which parts of a signal contain speech.
Back to the textLarge language model — A model trained on large text datasets to learn language patterns and process or generate text in context.
Back to the text
Frequently asked questions
- What should you check first in this article?
- FluidAudio is a Swift SDK for running speech models on Apple devices. It can add ASR, speaker diarization, and TTS to local apps. Choose the supported language and model, then start by running the one feature you need.