Fish Audio Drama 3 int4: 41.5 GB down to 22 GB

Fish Audio Drama 3 is a multi-speaker speech model that computes 3B of its 35B per token². The original needs 41.5 GB in bf16³, which no single 24 GB card holds; this checkpoint⁴ folds the weights to 4 bits and lands at 22.1 GB.

What got folded matters more than the size. Only the routed experts and some attention⁵ projections drop to int4, while the shared expert every token passes through and the audio decoder that builds the waveform stay in bf16, which is why it is labelled a W4A16 hybrid.

The licence is the Fish Audio community licence, so check its terms before commercial use.

The weights are 20.8 GB, but serving memory is much larger, which is why a 24 GB card stalls at start-up. On an A100 80 GB the engine already held 57.3 GB at start and peaked at 57.9 GB across five generations.

The rest is KV cache⁶ and engine overhead. Longer lines and more speakers grow the cache, so headroom counted from weights alone disappears quickly.

It needs Ampere or newer, because the 4-bit kernel uses GPTQ-Marlin. Turing and earlier cards do not open that path.

The runtime⁷ is SGLang Omni: clone it, build a Python 3.12 virtualenv, then install the fishaudio extra with pre-releases allowed.

After downloading the checkpoint, use the same sgl-omni serve command as the original. The checkpoint carries its quantization⁸ settings, so the loader needs no patch.

git clone https://github.com/sgl-project/sglang-omni.git
cd sglang-omni
uv venv -p 3.12 .venv && . .venv/bin/activate
uv pip install --prerelease=allow -e ".[fishaudio]"
hf download akhaliq/drama-3-int4

sgl-omni serve \
  --model-path akhaliq/drama-3-int4 \
  --port 8000

The API⁹ is unchanged: post input to /v1/audio/speech, attach a reference clip through references, and the [sighs] and [laughs] tags, multi-speaker scenes and previous_takes editing all behave as on the original.

curl -s localhost:8000/v1/audio/speech \
  -H 'content-type: application/json' \
  -d '{"input":"오늘 회의는 여기까지입니다. [sighs] 정리해 둘게요.",
       "references":[{"audio":"ref.wav","text":"참조 음성 대본"}]}' \
  --output out.wav

The run covers 1,088 Seed-TTS English prompts¹⁰ on one H200, concurrency 8, non-streaming, with WER measured by transcribing the output with Qwen3-ASR-1.7B.

Latency grows about 8 percent per request and throughput¹¹ falls from 6.47 to 5.99, while WER came out lower than the original at 1.63 percent.

That gap is not a quality gap: a second run of the same FP8 checkpoint measured 1.365 s and 2.02 percent, a run-to-run spread as large as the difference between the two.

A second evaluation on one A100 ran the same eight prompts against an int8 dense quantisation built for 32 GB cards.

On the long paragraph the comparison checkpoint hallucinated an ending and scored 0.667 against 0.028 here, while speaker similarity tied at 0.846.

Both scored 1.0 on the Chinese prompt although the transcripts nearly matched, a tokenisation artifact of unspaced Chinese rather than a difference between models.

Do not compare this table with the Seed-TTS table above: both the dataset and the transcription model differ.

If memory binds first, take int4: 22 GB of weights opens up 24 and 32 GB cards.

If latency binds first, take the FP8 original; 8 percent per request shows up in tight real-time conversation.

Both sit on pre-release ground: sglang-omni moves fast, so the install command and supported GPUs can change between releases. Listen to one sentence before putting it into production.