memory and model

Why a 35B MoE can be faster than a smaller dense model

It seems that the larger the number in the model name, the slower it will be. However, there are cases where the 35B model answers faster than the 27B model. Did I misread the big number? In fact, the two numbers only refer to the size of the model to be stored, and do not mean that they all calculate the same amount when creating one token.

Give both models the same job

Before comparing just the speed of 35B and 27B, you need to match the question and answer length, quantization, and executable program. If the numbers come from different tasks, it is difficult to see structural differences. Here, let's consider a situation where the same type of request is sent from the same device.

Dense models use most of the weight when creating new tokens. As the model grows, the weights that need to be read tend to increase. So the expectation that larger Dense models would be slow is understandable as a starting point.

Illustration of the workshop of the MoE model, with only some of the several expert weight drawers opened to process tokens. — MoE keeps the entire expert in memory, but only uses a selected subset of experts when calculating a single token.

MoE keeps the whole and selects some

The MoE has several expert blocks. From each token, the router selects a portion and computes it. Since not all experts work together every time, the portion used by one token may be small even if the overall parameters are large.

For example, a model that says about 3B active for a total of 35B would need to read the storage size and per-token compute size separately. These numbers are illustrative structure examples and do not promise the same speed or quality as the 3B dense model. Shared floor and routing costs also remain.

Illustration comparing the dense model that passes through all blocks and the MoE model that passes only some expert paths. — Even MoEs with larger total parameters can result in combinations that decode faster than smaller dense models if the amount of active computation is small.

Why do answers get faster when calculations are reduced?

When one person is receiving an answer, the speed of retrieving weights from memory can be as important as the computational power of the GPU. If the path that only reads and computes the selected weights is efficient, the next token can be created with less work than a smaller Dense model.

However, if the executor's MoE processing is inefficient, this benefit will not be fully achieved. There is also the cost of selecting experts and accessing scattered memories. Knowing the model structure can explain the possibility of it being fast, but the actual tok/s cannot be confirmed based on the structure alone.

Illustration of a structure in which only a few active expert paths lead quickly in a large memory cabinet. — MoE device selection requires separate consideration of the memory into which the entire weight will be placed and the generation rate of the actual active path.

Fast models don't necessarily fit into small memory

This is where the most unfortunate misunderstanding occurs. This is a case where you buy a small GPU just for the active part, but the model file is not included. Since we do not know which expert the next token will choose, the entire weight must be in accessible memory or storage.

Even when a runtime can keep some weights in RAM or on an SSD, fetching them takes time. That transfer time can outweigh the benefit of reduced computation. First establish that the model can be loaded, then compare the speed of the working configurations.

We also find reasons to use larger models in our work.

Just because MoE is fast does not automatically mean it is ahead in terms of answer accuracy. I need to check which model's results are more useful from the code modifications I made or the document summary. If you get the wrong answer quickly, your review time may actually increase.

If you feel overwhelmed by the number 35B, try dividing it by actual file size and active size. Conversely, if you thought it was light just by looking at the active 3B, look at the overall payload again. Instead of the impression given by the name, the goal is to leave a model that matches three things: memory, speed, and work results.