LFM3-Audio-2B: a 1 GB model that listens and speaks

Input is text or 16 kHz audio; output is text or 24 kHz speech, and the two can interleave, so an answer can be spoken while the text stays on screen.

It ships an in-house audio encoder and the LiquidCodec tokenizer. The encoder is fully causal, so it can prefill³ while you are still speaking, which is where the low latency comes from.

Speech compresses 80 ms into 12 tokens, so filling 32K with audio alone lasts a little over three minutes and longer meetings need splitting.

The GGUF files are 1.05 GB at Q4_0, 1.1 GB at Q4_K_M and 1.85 GB at Q8_0, plus the mmproj audio encoder at 0.34 GB for Q8_0.

The 4-bit file and the encoder together come to about 1.4 GB, which leaves headroom on an 8 GB Mac with a browser open.

Loading it in Python through Transformers pulls 4 GB of bf16⁴ weights; the 1.4 GB figure belongs to the GGUF plus encoder path, so mixing the two paths breaks the memory estimate.

Name the language in the system prompt⁵, as in Perform ASR in Korean., and it transcribes in that language; write only Perform ASR. and the model infers it.

Live transcription is English-only for now: feed it 0.96 s chunks and text appears while the speaker talks, and past 30 seconds you reopen the text context.

Batch file transcription and the live stream use different code: the live path passes state into the next call and takes a result per chunk.

from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "LiquidAI/LFM3-Audio-2B"
model = AutoModelForMultimodalLM.from_pretrained(
    model_id, trust_remote_code=True).eval()
processor = AutoProcessor.from_pretrained(
    model_id, trust_remote_code=True)

conversation = [
    {"role": "system", "content": "Perform ASR in Korean."},
    {"role": "user", "content": [{"type": "audio", "audio": "meeting.wav"}]},
]
inputs = processor.apply_chat_template(
    conversation, add_generation_prompt=True,
    tokenize=True, return_dict=True, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=1024,
    suppress_mixed=True, suppress_audio_out=True,
    return_dict_in_generate=True)
text = processor.decode(
    out.sequences[0, inputs["input_ids"].shape[1]:],
    skip_special_tokens=True)
print(text)

Synthesise by naming the language and voice in the system prompt and pre-filling the assistant turn with <|audio_start|>, as in Perform TTS in American English using a female voice..

Sampling uses audio_temperature 1.0, audio_top_k 128, repetition penalty 1.1 and a 4096 window; for text-only output turn on suppress_mixed and suppress_audio_out.

import soundfile as sf

codec = processor.audio_tokenizer.eval()

conversation = [
    {"role": "system", "content": "Perform TTS in Korean using a female voice."},
    {"role": "user", "content": "오늘 회의는 세 시에 시작합니다."},
    {"role": "assistant", "content": "<|audio_start|>"},
]
inputs = processor.apply_chat_template(
    conversation, continue_final_message=True,
    tokenize=True, return_dict=True, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=256,
    audio_temperature=1.0, audio_top_k=128,
    audio_repetition_penalty=1.1, audio_repetition_window=4096,
    return_dict_in_generate=True)
waveform = codec.decode(
    out.audio_codes[:1, :, :out.audio_lengths[0]]).audio_values[0]
sf.write("tts.wav", waveform.float().cpu().numpy(), 24_000)

The causal encoder prefills while you speak, cutting the wait after you stop; the first token can arrive in as little as 50 ms.

LiquidCodec turns tokens into speech at about 87 times real time on a single M4 Max CPU⁶ thread. Because 80 ms compresses into 12 tokens predicted in parallel, generation and playback overlap.

Ask out loud and it returns text and speech together, streaming so audio starts before the sentence ends.

A spoken request can become a tool call⁷: asking to turn off a light produces a structured call, and the result is read back as speech.

A long tool result means more speech to generate, so a system prompt that keeps answers short pays off directly in latency.

To wire tools in, put Respond with function calls, and interleaved text and audio. in the system prompt and append the tool list; leave the assistant turn empty so the model picks between a call and speech.

For transcription alone, dedicated models lead: English WER is 6.23 percent against 5.51 for Voxtral Mini 3B.

Tool calling at 58.50 trails Gemma 4 E4B at 72.40, and audio understanding at 50.60 trails Qwen2.5-Omni 3B at 54.41.

What it wins is VoiceBench at 73.13 and synthesis quality at 3.83, which is where the value sits: one model carrying both listening and speaking.