How much memory does it need?

Routing, irregular memory access, expert load, and kernels also consume time. Offloading some weights to system RAM adds transfer time. To retain the benefit of a small active size, inspect memory placement and the supported execution path for that model.

memory and model

MoE vs dense: why low active parameters still need memory

MoE computes with selected experts while keeping all the weights available.

MoE¹ is a model architecture that selects some expert blocks to calculate the next token². Dense models use most of their weights for each token. An MoE labeled '35B total, 3B active' distinguishes the stored model size from the size participating in one token's computation.

Downloading the model requires its complete weight files, while speed depends on the computation path actually used. Understanding why these numbers differ helps you avoid ruling out a model just because its name is large or underestimating memory from its active count.

The router selects experts for each token.

An expert in MoE is a computation block inside the model. A learned router selects blocks from the input, and their results feed the subsequent computation. Rather than imagining small models manually assigned to Korean or coding, think of paths chosen for each token.

In Dense, most weights in each layer participate. In MoE, shared components and selected experts participate, and the next token may select different experts. Experts unused for one token therefore still need to remain accessible for later computation.

Several expert modules with only selected modules on a token's route — The router chooses expert paths for each token.

35B total and 3B active serve different roles.

B means a billion parameters. The 35B total includes shared components and expert weights; 3B active refers to the scale participating in one token's computation path. That is why downloads and memory start with the total count, while per-token computation starts with the active count.

Consider a simplified example storing everything in 4 bits. 35 billion×4 bits÷8 is 17.5 billion bytes, or 17.5 GB in decimal units. Counting only 3B active gives 1.5 GB, but the full model also needs the remaining experts. Actual Q4 files add quantization³ parameters, metadata, and components kept at other precisions, so check their published sizes.

Use the two counts for different decisions.
Label	Meaning	Use
35B total	All stored model weights	Checking files and loading memory
3B active	Scale participating in each token's computation	Understanding computation and execution paths

Less computation creates opportunities for higher speed.

If an MoE has a smaller active size than a Dense model of similar total size, it can reduce computation per token. Efficiently fetching and computing selected expert weights creates a decode⁴ advantage. This is why total B counts alone are insufficient for ranking speed.

Routing, irregular memory access, expert load, and kernels also consume time. Offloading⁵ some weights to system RAM⁶ adds transfer time. To retain the benefit of a small active size, inspect memory placement and the supported execution path for that model.

Differently colored tokens routed to selected experts — The next token may select different experts.

Compare code fixes through the completed tests.

For example, fix a function that fails on an empty string. Give both models the same code and error description and ask them to handle empty input while preserving the return format. Tests for normal input, empty input, and existing callers show whether the requested behavior is preserved.

Record time to first answer, code generation speed, and the number of additional fixes needed to pass the tests. A fast-answering model may take longer overall if it needs several revisions. Comparing the result required by your code alongside waiting time makes selection easier.

All stored experts compared with the subset active for one token — Loading requires the full weights and room for execution.

Choose hardware by checking files, context, and results.

Check the desired Q4 file on the distribution page and leave memory for the runtime⁷, other apps, and KV cache⁸. Start with a short question to check loading, then test your usual project input length to check memory under the actual workload.

Then compare the code result, prefill⁹, and decode. An MoE that fits reliably and produces the required fix can be a good option on your current machine. If it fits but waits are long, inspect bandwidth and runtime options; if loading fails, check total capacity and offloading support.

Terminology notes

MoE — A model architecture that selects some of several expert subnetworks for each input. Total parameters can differ from the number activated for one token.
Back to the text
Token — A unit into which a model divides input or output for processing.
Back to the text
Quantization — A way to represent model weights or values with fewer bits, reducing storage and memory use. The coarser representation can also affect accuracy.
Back to the text
Token decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the text
Offloading — Moving some model data from GPU memory to system RAM or storage when capacity is limited. This adds data transfer.
Back to the text
System RAM — System memory that temporarily holds data while programs run.
Back to the text
Runtime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.
Back to the text
KV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.
Back to the text
Prefill — The stage where an LLM reads the input prompt and computes representations for its tokens. Longer prompts contain more tokens to process.
Back to the text

Frequently asked questions

What should you check first in this article?: MoE is a model architecture that selects some expert blocks to calculate the next token. Dense models use most of their weights for each token. An MoE labeled '35B total, 3B active' distinguishes the stored model size from the size participating in one token's computation.
How much memory does it need?: Routing, irregular memory access, expert load, and kernels also consume time. Offloading some weights to system RAM adds transfer time. To retain the benefit of a small active size, inspect memory placement and the supported execution path for that model.