Prefill and token generation speed
2× RTX 5090 64GB (PCIe) with Qwen3.8-Flash-Next: prefill and generation speed
Last updated
Strata 0.1.41 · IQ3_S · MTP-6 · INT8 KV · SSD PLE · GPU 1/2
What you need
Expand settings SettingsIQ3_S · 4K · Strata MTPCollapse
Strata 0.1.41 · IQ3_S · MTP-6 · INT8 KV · SSD PLE · GPU 1/2
Experts remain in RAM, with frequently used experts cached on the GPU. The PLE table is read from an NVMe SSD.
What changes with a longer document?Expand input length, waiting time, speed and memory
Run configuration
Strata 0.1.41 · IQ3_S · MTP-6 · INT8 KV · SSD PLE · GPU 1/2
Experts remain in RAM, with frequently used experts cached on the GPU. The PLE table is read from an NVMe SSD.
Self-built 0.1.41, Ryzen 9800X3D with 120GB DDR5-6000, original IQ3_S, native configured limit 262144 but the included fresh prompt is 8059 actual tokens. INT8 KV with 32768 resident cells, MTP spec6/min-p0.70, prefill16384, pcie-frac0, CPU vision. One reported sample; output count and HTTP TTFT unreported. Host CPU/RAM capacity transfer gets a wider scenario band, no RAM speed bonus. 4K–32K are bounded projections only; the configured 262K limit is not a measured prompt axis. Fully reused 8054/8059-token ~230 tok/s artifacts and short-prose 162.5 tok/s without a prompt axis are excluded. No global 0.1.41 speed multiplier is applied. Single-GPU configuration transferred to one identical RTX 5090 on the dual-card host. The other GPU is idle: no pooled 64GB VRAM, pipeline gain or second-GPU timing is claimed. 문맥 사이 값은 대기 시간 보간, 범위 밖은 보수적 외삽입니다. 표시 범위는 반복 측정 신뢰구간이 아닌 시나리오 범위입니다.
Popular configurations
Speed simulations by device and model
Studio M3 Ultra 512GB × Qwen3.8-Flash-Next
50 ~ 70.3 tok/s · 3.3 to 4.5 seconds
Qwen Flash Next on Mac Studio M5 Ultra 512GB
103.5 ~ 184.1 tok/s · 0.65 to 1.2 seconds
RTX PRO 6000 96GB × Qwen3.8-Flash-Next
175.2 ~ 340.0 tok/s · 1.2 to 2.2 seconds
Qwen Flash Next on two DGX Sparks: what changes?
16.7 ~ 33.8 tok/s · 1.7 to 4.3 seconds
