Skip to content
Published on

MoE Routing — How an Expert Gets Picked

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

Introduction

In the previous post we counted the parameters of a dense model from its config.json. But open the config of a modern large model and unfamiliar fields appear, such as num_experts or n_routed_experts. This is the mixture-of-experts, or MoE, layer.

The idea is simple. Split the single FFN in each layer into many smaller FFNs, and run only a few of them per token. This post is about how that "only a few" is written into the config, and what it does and does not save.

All figures were verified directly against papers, official reports, and config.json files on 2026-08-12. Models get updated, so check the originals again.

The Simplest Form: Mixtral

Pulling only the MoE-related fields from the Mixtral-8x7B config gives this.

{
  "num_local_experts": 8,
  "num_experts_per_tok": 2,
  "intermediate_size": 14336,
  "router_aux_loss_coef": 0.02
}

There are 8 experts and each token uses 2. The Mixtral paper describes this structure as giving each token access to 47B parameters while using only 13B active parameters per token (arXiv:2401.04088). The same paper explains that the router chooses two experts at every layer for every token, and that different experts can be selected at each timestep.

Active Parameters Versus Total Parameters

Here is the misconception worth settling first. What shrinks with active parameters is compute, not memory. Since you never know which expert will be called, all of them must sit in memory. The Mixtral paper states plainly that the memory cost of serving Mixtral is proportional to its sparse parameter count of 47B.

Let us compute this directly for Qwen3-30B-A3B. The config gives hidden_size 2048, 48 layers, 128 experts, num_experts_per_tok 8, and moe_intermediate_size 768.

One expert (gate/up/down, 3 matrices)
  = 3 x 2,048 x 768 = 4,718,592

All experts in one layer : 128 x 4,718,592 = 603,979,776
Actually executed        :   8 x 4,718,592 =  37,748,736

Total parameters  (computed) = 30,532,122,624  -> matches the published value exactly
Active parameters (computed) =  3,353,032,704  -> about 3.35B, matching the A3B in the name

Ratio = 3.35B / 30.53B = about 11%

You do 11 percent of the compute but need 100 percent of the memory. The essence of MoE is not trading memory for compute. It is spending more memory to hold more knowledge at the same compute.

Shared Experts and Dense Early Layers

DeepSeek-V3 uses a slightly different structure. According to the model hyper-parameters section of its technical report, there are 61 layers, each MoE layer consists of 1 shared expert and 256 routed experts, and 8 of the routed experts activate per token. The expert intermediate dimension is 2048. Of 671B total parameters, 37B activate per token (arXiv:2412.19437).

A shared expert is one that every token passes through regardless of routing. Concentrating the common functionality that every token needs into that expert frees the routed experts to take on more specialized roles.

Another field worth noting is first_k_dense_replace. DeepSeek-V3 sets it to 3, and the report likewise states that all FFNs except those in the first three layers were substituted with MoE layers. Kimi K2 sets the same field to 1. Early layers build general representations rather than token-specific specialization, so keeping them dense is the more stable choice.

Load Balancing: Two Approaches

Left alone, a router will funnel tokens to a handful of experts. The rest never train, and load piles up unevenly across parallel devices. There are broadly two remedies.

The first is an auxiliary loss. You add a loss term that pushes the load toward uniformity. The router_aux_loss_coef in the config is its coefficient: 0.02 for Mixtral and 0.001 for the Qwen3 MoE models. The Qwen3 report states that a global-batch load balancing loss was adopted to encourage expert specialization (arXiv:2505.09388). The problem is that this loss competes with the real objective and shaves off a little quality.

The second approach avoids an auxiliary loss entirely. DeepSeek-V3 keeps a bias term per expert and adds it to the affinity scores only when selecting the top k. The report notes that the bias is used only for routing, and that at the end of each step the bias is decreased for overloaded experts and increased for underloaded ones. The bias update speed was 0.001 for the first 14.3T tokens and switched to 0.0 for the remaining 500B tokens. A sequence-wise balance loss guards against extreme imbalance and is kept very small, with a coefficient of 0.0001.

The contrast is clear. An auxiliary loss is simple to implement but contaminates the objective, while the bias scheme leaves the objective alone at the cost of extra hyperparameters and a schedule to tune during training.

How Far to Push Sparsity

Kimi K2 raised the expert count to 384. Table 2 of its report lines the model up against DeepSeek-V3. Layer count is the same at 61, total parameters rose from 671B to 1.04T, yet active parameters fell from 37B to 32.6B. Experts went from 256 to 384 while active experts per token stayed at 8 (arXiv:2507.20534).

The same report defines sparsity as the ratio of total experts to activated experts, and reports that holding activated parameters fixed while increasing the total number of experts consistently lowered both training and validation loss. At a validation loss of 1.5, it states that sparsity 48 reached the same point with 1.69x, 1.39x, and 1.15x fewer FLOPs than sparsity levels 8, 16, and 32 respectively.

The cost appears in the same paragraph. Raising sparsity increases infrastructure complexity, and the authors say they chose 48 to balance performance against cost. As experts multiply, both all-to-all communication volume between devices and the difficulty of batch scheduling rise with them.

Limiting Routing Scope

The DeepSeek-V3 config contains n_group 8 and topk_group 4. This groups experts and caps how many groups a single token can reach. The report likewise states that each token is ensured to be sent to at most 4 nodes. It is a design that puts a ceiling on communication cost. Kimi K2, by contrast, explicitly lists expert grouping as not used in Table 2. The same problem solved on different infrastructure assumptions.

Closing

For an MoE config you only need four things: how many experts there are, how many are used per token, whether there is a shared expert, and whether balance is enforced by a loss or by a bias. From those four values you can compute active parameters, and active parameters tell you the compute cost while total parameters tell you the memory cost. Reading those two numbers separately is where understanding MoE begins.

References

Try It Yourself

Series

Discuss on Twitter • View on GitHub