Response speed and acceleration
Last updated
What is ReplaySSM? Faster decoding for Nemotron and Qwen servers
Reduce repeated state writes while generating answers for concurrent requests.
ReplaySSM is an inference1 optimization that changes how SSM layers retain past input. It stores a checkpoint2 separately from recent inputs, reducing full-state writes at each token3. It is relevant to hybrid-model servers handling concurrent requests.
When several agents send requests to a Qwen or Nemotron server, the engine repeatedly reads and updates each conversation’s state. SSM layers summarize previous context in a fixed-size state. ReplaySSM reduces repeated state-memory traffic to help the server generate tokens faster.
Buffer recent inputs instead of writing the state every step
An SSM layer incorporates new information into its previous state when processing a token. With concurrent requests, reading and writing a state for every conversation creates more memory traffic. ReplaySSM keeps a checkpoint and buffers recent inputs. It computes outputs from both, then folds the buffered inputs into the checkpoint when the buffer fills.
Hybrid Nemotron and Qwen models combine SSM and attention4 layers. ReplaySSM reduces the cost of reading and writing the SSM state. It can coexist with weight quantization5, while the file-size setting is selected separately.
Rollback after verifying MTP drafts becomes cheaper
When MTP6 or a separate drafter proposes several tokens, the target model accepts a prefix. After discarding the rest, its conversation state must match the accepted tokens. ReplaySSM adjusts the end of the recent-input buffer to retain that prefix, reducing the need to save a large state snapshot for every draft token.
This optimization changes state computation and storage inside the engine. LiLiCorr instead improves the proposals likely to be accepted. Recording draft acceptance length separately from state-verification cost helps identify which server change was useful.
Results differ between one request and large batches
NVIDIA and Princeton researchers compared four Mamba-2 and GDN models in vLLM. The table covers standard decoding7 without speculative acceleration. Separating kernel speedup from speedup of the entire decode step makes the server-level effect clearer.
| Condition | Result |
|---|---|
| Batch 1 | No speedup in these tests |
| Batch 256 · SSM kernel | 1.46–1.80× |
| Batch 256 · complete decode step | 1.20–1.47× |
4B models: one H100; 120B-class: one B300 · no tensor parallelism8 · FP329 SSM state · window L=16 · tuned vLLM baseline
Published tests showed no standard-decoding speedup at batch size10 1 and gains at batch size 256. On a personal PC, measure one request’s TTFT11 and generation speed first. For a team server, reproduce actual concurrency and compare per-request latency and total throughput12.
What to record before adopting it on a server
ReplaySSM is integrated into TensorRT-LLM, vLLM and FlashInfer. Check that your version supports the model’s SSM path and start on a test server. Fix the checkpoint, input and output lengths, state precision and drafter setting, then record the current engine’s baseline. After changing the engine, compare TTFT, completion time, peak memory and failures using the same requests.
If your goal is a faster personal PC, first compare supported MTP, DFlash or more efficient quantization for the model. For a server shared by several agents, add ReplaySSM to the candidates and evaluate request completion during busy periods before switching.
Terminology notes
Inference — The process of using a trained model to compute an output for an input. Here, local inference means running the model on the user’s device.
Back to the textCheckpoint — A file containing saved model weights and related state. Versions or tasks in one model family may use different checkpoints.
Back to the textToken — A unit into which a model divides input or output for processing.
Back to the textAttention — A neural-network operation that calculates relationships between input tokens and combines information from each position into the current representation.
Back to the textQuantization — A way to represent model weights or values with fewer bits, reducing storage and memory use. The coarser representation can also affect accuracy.
Back to the textMTP — A training objective or model component for predicting multiple future tokens at once. In speculative decoding, those predictions can be proposed as candidate tokens.
Back to the textToken decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the textTensor parallelism — A method that splits matrices and computations within a model layer across devices for joint execution. Devices communicate to combine partial results.
Back to the textFP32 — A 32-bit floating-point format. It uses more memory per value than BF16 and can represent values more precisely.
Back to the textBatch size — The number of inputs or requests processed together in one batch.
Back to the textTime to First Token — The time from sending a request until its first output token arrives.
Back to the textThroughput — The amount of work processed or generated over time. Comparisons need the unit, such as tokens per second or requests per second.
Back to the text
Frequently asked questions
- What should you check first in this article?
- ReplaySSM is an inference optimization that changes how SSM layers retain past input. It stores a checkpoint separately from recent inputs, reducing full-state writes at each token. It is relevant to hybrid-model servers handling concurrent requests.
- How much memory does it need?
- ReplaySSM is integrated into TensorRT-LLM, vLLM and FlashInfer. Check that your version supports the model’s SSM path and start on a test server. Fix the checkpoint, input and output lengths, state precision and drafter setting, then record the current engine’s baseline. After changing the engine, compare TTFT, completion time, peak memory and failures using the same requests.