memory and model
MoE vs dense LLMs: total and active parameters
Model descriptions come with numbers like 35B full and 3B active. Does this mean that it is a large model when downloaded but a small model when run? This is half true, but if you choose equipment based solely on this statement, you may run out of memory. Let’s read separately the size of storage and the size of calculation each time.
Dense uses most weights
Dense models undergo most of the weighting when creating tokens. If the series and conditions are the same, it can be understood that as the model grows, the space to be contained and the work to be processed also increase. Of course, actual speed is also affected by quantization and executable program.
MoE has several expert blocks and selects a portion for each token. However, this does not mean that each expert is in charge of a subject determined by a person, such as Korean or mathematics. This is a structure in which a learned router selects a calculation path.
The two numbers answer different questions
The total parameter is the size of the expert and shared parts combined. The active parameter is the size of the actual writing portion of a token. The former is a clue to understanding files and memory, and the latter is a clue to understanding the computational burden per token.
You might be thinking, why not just save the active part? However, in the next token, a different expert may be selected. Make sure you keep all of your weights in an accessible location, and even factor in retrieval time if you leave some in slower storage locations.
Why it can be fast even with large numbers
If the parts to read and compute in a token are small and the expert processing at runtime is efficient, it can be produced faster than Dense, which has a smaller total scale. So, if you determine the speed order only by the B number in the model name, it may not be correct.
This is not always the case. There are routing and random memory access costs, and it also affects how well the device bundles computations. There remains a step to directly compare prefill and decode results for the same quantization and operation.
More total parameters do not guarantee a better answer
It cannot be assumed that having more experts will always give you an advantage when it comes to summarizing documents or modifying code. Training data, model design and the ability to follow instructions work together. Don't read either the total or active numbers as if they were intelligence scores.
One model may be fast but have many errors, and another model may be a little slower but require less correction. If possible, check your results with questions you'll actually use and then compare speeds. After choosing a model, the process of finding equipment that comfortably fits that model is less confusing.
If you have plenty of memory but it’s slow
Even if you solve the loading problem with a large unified memory, the bandwidth or execution path may limit the generation speed. Conversely, even fast GPUs can lose the benefit of offloading if the weight exceeds the VRAM. Capacity and speed are conditions that must be passed separately.
Rather than using the name MoE itself as a reason to purchase, look at the entire files, cache, and actual creation results of the model you want. The purpose of understanding the two numbers is not to memorize complex specifications, but to avoid choosing the wrong equipment based on small active numbers.