Prefill and token generation speed

2× RTX 5090 64GB (PCIe) with Qwen3.8-Flash-Next: prefill and generation speed

Last updated

3584 GB/s combined

Strata 0.1.41 · IQ3_S · MTP-6 · INT8 KV · SSD PLE · GPU 1/2

Active 6B / Sangju 180B
195.8 tok/sDecode speed(baseline: No matching baseline)
PLE table NVMe SSD mmap + RAM partition (VRAM 30.5GB + RAM 52GB + SSD 29GB)
First token (TTFT)
0.46 to 0.94 secondsContext 4,096 input tokens included
Prefill speed
4375.9 ~ 9044.5 tok/s0.45 to 0.94 seconds (Estimate based on published benchmarks)
Total time (E2E)
0.90 to 1.9 seconds117 output tokens included
Press Play to process the input and generate the answer.
Ready
Expand settings IQ3_S · 4K · Strata MTP
QuantizationMemory 82.5GB + SSD 29GB
Input context4K tokens
Strata MTP
Baseline
PLE n-gram table: NVMe SSD — placed automatically. Applied alongside decode optimization.
Sample question:
117 output tokens

Strata 0.1.41 · IQ3_S · MTP-6 · INT8 KV · SSD PLE · GPU 1/2

Experts remain in RAM, with frequently used experts cached on the GPU. The PLE table is read from an NVMe SSD.

What changes with a longer document?Expand input length, waiting time, speed and memory
Run configuration

Strata 0.1.41 · IQ3_S · MTP-6 · INT8 KV · SSD PLE · GPU 1/2

Experts remain in RAM, with frequently used experts cached on the GPU. The PLE table is read from an NVMe SSD.

Self-built 0.1.41, Ryzen 9800X3D with 120GB DDR5-6000, original IQ3_S, native configured limit 262144 but the included fresh prompt is 8059 actual tokens. INT8 KV with 32768 resident cells, MTP spec6/min-p0.70, prefill16384, pcie-frac0, CPU vision. One reported sample; output count and HTTP TTFT unreported. Host CPU/RAM capacity transfer gets a wider scenario band, no RAM speed bonus. 4K–32K are bounded projections only; the configured 262K limit is not a measured prompt axis. Fully reused 8054/8059-token ~230 tok/s artifacts and short-prose 162.5 tok/s without a prompt axis are excluded. No global 0.1.41 speed multiplier is applied. Single-GPU configuration transferred to one identical RTX 5090 on the dual-card host. The other GPU is idle: no pooled 64GB VRAM, pipeline gain or second-GPU timing is claimed. 문맥 사이 값은 대기 시간 보간, 범위 밖은 보수적 외삽입니다. 표시 범위는 반복 측정 신뢰구간이 아닌 시나리오 범위입니다.

Popular configurations

View all 20