memory and model
Last updated
Why a 35B MoE can be faster than a smaller dense model
35B MoE and 27B Dense differ in computation per token.
MoE1 selects only some experts for computation. A model with 35B total and about 3B active weights is large to store but can reduce per-token work. Hardware that executes this path efficiently can produce combinations where it generates faster than a smaller 27B Dense model.
Let's compare two models fixing a function that fails on an empty string. Giving both the same code and request lets you compare first response and generation speed. Including the time until tests pass shows which model helps more with your coding task.
MoE computes only selected expert paths.
Dense uses most weights in each layer for a new token2. MoE routes the token to selected experts and shared components. With 35B total and about 3B active, a smaller part of the full model participates for that token. A larger total model can therefore do less per-token work.
If the runtime3 supplies selected weights quickly and processes expert operations efficiently, less work becomes higher decode4 speed. Routing and irregular data access also cost time, so examine actual results on the same hardware.
Prepare a setup that can access all weights.
The next token may select different experts. The runtime must be able to access all weights, so check the actual distribution size and quantization5. Leave room for KV cache6, the runtime, editor, and browser to compare under your usual working conditions.
A model entirely in VRAM7 and one partly offloaded to RAM8 have different data paths. Offloading9 expands the runnable size but adds transfer time. Record filenames and placement to understand reduced computation alongside data movement.
Compare the first response with the same code and input conditions.
Send the same function, related callers, and error logs to both models. Specify the scope, such as handling empty input safely while preserving the return format. Start each from the same baseline commit or copy rather than the first model's modified files, and keep logs and diffs separately.
Record model files, quantization, runtime, thinking mode, input, output limit, and concurrency. Separate prefill10 from decode and record time to first token11 and total completion. Tokenization differs by model, so compare tok/s alongside task completion time.
For a personal coding assistant, start with one request. For a team server, separately test concurrent requests: different inputs can select different experts and change memory access and scheduling. Record each user's latency as well as aggregate throughput12.
Record time until the fix passes its tests.
Test empty and normal inputs and existing callers. Check that the return format is preserved and the error is fixed. If tests fail, send the logs to the same model, review the revised diff, and rerun the tests. Including that follow-up time shows whether your task actually became faster.
The site's comparison screen lets you keep the hardware and input length while switching between 35B MoE and 27B Dense. Recommended formats and acceleration can differ by model, so use it to explore the setups you would actually use. Then choose from candidates that fit with cache space and produce the required fix on your code.
Terminology notes
MoE — A model architecture that selects some of several expert subnetworks for each input. Total parameters can differ from the number activated for one token.
Back to the textToken — A unit into which a model divides input or output for processing.
Back to the textRuntime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.
Back to the textToken decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the textQuantization — A way to represent model weights or values with fewer bits, reducing storage and memory use. The coarser representation can also affect accuracy.
Back to the textKV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.
Back to the textVRAM — Memory used by a graphics card’s GPU to store model weights and intermediate values.
Back to the textSystem RAM — System memory that temporarily holds data while programs run.
Back to the textOffloading — Moving some model data from GPU memory to system RAM or storage when capacity is limited. This adds data transfer.
Back to the textPrefill — The stage where an LLM reads the input prompt and computes representations for its tokens. Longer prompts contain more tokens to process.
Back to the textTime to First Token — The time from sending a request until its first output token arrives.
Back to the textThroughput — The amount of work processed or generated over time. Comparisons need the unit, such as tokens per second or requests per second.
Back to the text
Frequently asked questions
- What should you check first in this article?
- MoE selects only some experts for computation. A model with 35B total and about 3B active weights is large to store but can reduce per-token work. Hardware that executes this path efficiently can produce combinations where it generates faster than a smaller 27B Dense model.
- How much memory does it need?
- For a personal coding assistant, start with one request. For a team server, separately test concurrent requests: different inputs can select different experts and change memory access and scheduling. Record each user's latency as well as aggregate throughput.