Agents and automation

Let an AI answer your phone on a Mac: installing ai-phone and choosing 4B or 9B

Memory decides the model: 4B at 16 GB, 9B at 32 GB.

On Apple Silicon the speech recognition and synthesis run locally through MLX¹, and only the call text reaches the model. Setup picks the model from detected memory, and the latest local run measured a 1.50 s median first-audible response.

Contents

Requirements and key details

It runs only on Apple Silicon Macs, and speech recognition and synthesis stay local through MLX.
Setup recommends a model from detected memory: 2B at 8 GB, 4B at 16 GB, 9B at 32 GB.
The latest local call measured a 1.50 s median to first audio, short of the one-second target.

The Mac answers the call and keeps the conversation going

ai-phone lets a Mac answer SIP calls and hold the conversation. Speech recognition turns the caller's words into text, a language model decides the reply, and speech synthesis reads it back. Those three steps repeat for the length of the call.

Speech recognition uses Parakeet and synthesis uses Phonon or Kokoro, both through MLX. Only the model that writes the replies is selectable: a local MLX model, a Claude Code or Codex subscription, or an OpenAI-compatible endpoint. Choosing a remote model sends the call text and instructions to that provider.

Lines connect a compact computer to a browser, calendar, and chat window — One call runs recognition, the model, and synthesis in sequence.

You need a Mac and a SIP line

You need an Apple Silicon Mac with Git, Homebrew, and uv. The installer prepares the Python packages and the native SIP library. Intel Macs, Linux, and Windows are not supported.

Receiving calls needs a SIP address, a SIP trunk, or a Telnyx or Twilio account. The other side can use an ordinary phone; the call still connects over SIP and the carrier charges apply. For outgoing calls only, start the program when needed and stop it afterwards.

Installing and choosing the model

Clone the repository, run the install script, then start setup. Setup asks whether the Mac should receive or place calls, which line and voice to use, and recommends a model for the detected memory before downloading the weights. The model files take several gigabytes.

The default is the 4B model; the 9B option uses more memory. An 8 GB Mac is offered the 2B model. Running setup again changes the line, voice, or model, and the daemon must be restarted afterwards.

Suggested model by memory size
Memory	Suggested model	Use
8GB	2B	Short calls on a tight machine
16GB	4B (default)	Default setup
32GB	9B	Longer calls, better phrasing

install

git clone https://github.com/oxnw/ai-phone.git
cd ai-phone
./scripts/install.sh
uv run ai-phone setup
uv run ai-phone run

Choose the line and model in setup, then start listening with run.

Placing and receiving calls

An outgoing call starts with one prompt², and the transcript comes back when the call ends. Every outgoing call opens with a disclosure that an AI is speaking.

Relay mode lets an agent drive each turn itself: connect with call, then alternate say and listen, which works without any model in the loop. An unanswered relay turn ends after 25 seconds.

call

uv run ai-phone call +15555550100 --json \
  --prompt "Ask whether they have a table for two at eight tonight."

Include the country code in the number, and put the purpose of the call in the prompt.

A person at a laptop is surrounded by connected document, spreadsheet, folder, image, settings, and robot panels — Setup picks the model from detected memory, so choose the Mac by memory first.

A 1.50 s median to first audio, short of the one-second target

The latest local call measurement for this project put the median first-audible response at 1.50 s. It ran under heavy memory pressure and missed the one-second target. The figure covers the time from the end of the caller's speech through model generation and synthesis to playback.

Call latency tracks model size and available memory. A 9B model on a 32 GB Mac improves phrasing but can delay the first audio. For short booking or confirmation calls, start with 4B and measure latency on real calls.

English calls only, one at a time

Support currently covers English and a single call at a time. Korean calls would need the recognition, synthesis, and the model all swapped to Korean, and this project does not offer that setup out of the box.

The code is AGPL-3.0-or-later, and the bundled PJSIP interface files are GPLv2-or-later. Carrier charges and subscription limits apply separately. Keeping the line open means leaving the Mac on with the daemon running.

A user looks at progress indicators and a document-processing window on a desktop computer — After changing the line or model, restart the daemon for it to apply.

Terminology notes

MLX — Apple’s machine-learning framework, which runs models using memory shared by the CPU and GPU on Apple Silicon.
Back to the text
Prompt — The input text sent to a model. For generative models it describes the desired result and strongly shapes the output.
Back to the text

Frequently asked questions

What should you check first in this article?: On Apple Silicon the speech recognition and synthesis run locally through MLX, and only the call text reaches the model. Setup picks the model from detected memory, and the latest local run measured a 1.50 s median first-audible response.
How much memory does it need?: Clone the repository, run the install script, then start setup. Setup asks whether the Mac should receive or place calls, which line and voice to use, and recommends a model for the detected memory before downloading the weights. The model files take several gigabytes.