Skip to content
←

필사 모드: Normalization and Activation — Keeping Training From Falling Apart

English
0%
정확도 0%
💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction

The previous posts dealt with parameters and memory. This one has a different subject. Normalization and activation functions contribute almost nothing to parameter count, but get them wrong and training itself stops. On a job where thousands of GPUs run for weeks, a diverging loss is the most expensive failure there is.

The clues visible in the config are rms_norm_eps, hidden_act, and in recent models use_qk_norm. Let us look at the design sitting behind them.

All figures were verified directly against papers, official reports, and config.json files on 2026-08-12. Models get updated, so check the originals again.

Why RMSNorm Became the Default

Every config examined here contains rms_norm_eps. Qwen3, DeepSeek-V3, and Kimi K2 use 1e-06, while Mixtral and GLM-4.5 use 1e-05. RMSNorm without exception.

The original layer normalization subtracts the mean and divides by the variance. RMSNorm drops the mean-subtraction step and divides only by the root mean square. Compute goes down, and more importantly the extra reduction needed to compute a mean disappears, which lowers communication and synchronization cost in distributed training.

rms_norm_eps is the small value that prevents division by zero. It looks trivial, but in low-precision training a value too small makes the arithmetic unstable while one too large blurs the normalization.

Pre-Norm and Post-Norm

Where you put the normalization strongly determines stability. The original transformer normalized after adding the residual connection, which is post-norm. Today nearly everyone uses pre-norm, normalizing before adding to the residual. The Qwen3 report explicitly states that it uses RMSNorm with pre-normalization (arXiv:2505.09388).

The difference lies in the residual path. Under pre-norm an identity path runs from input to output without passing through a normalization, so gradients propagate well even as layers stack up. There is a cost. Pre-norm tends to let the magnitude of deep-layer outputs accumulate, which is why a final normalization must be added separately. The 4,096 added at the end of the parameter count in the first post is exactly that final normalization.

The SwiGLU Family

A hidden_act of silu signals the SwiGLU family. Among the models checked, Qwen3, Qwen2.5, Mixtral, DeepSeek-V3, and GLM-4.5 all use silu. Table 3 of the Llama 3 report likewise lists the activation function as SwiGLU (arXiv:2407.21783).

The gated structure builds two linear projections, applies a nonlinearity to one of them, and multiplies element-wise. The relevant paper reports that testing gated linear unit variants in the feed-forward sublayers of the Transformer found some of them yielding quality improvements over the typically used ReLU or GELU activations (Shazeer, arXiv:2002.05202).

There is a practically important cost here. The number of matrices goes from two to three. To hold the same parameter budget you must therefore shrink the FFN inner dimension. Llama 3.1 8B pairs a width of 4096 with an FFN of 14336, a ratio of 3.5, and Qwen3-8B pairs 4096 with 12288, a ratio of 3. This is why the old convention of 4 times, from the days before gating, no longer holds.

When Attention Logits Explode

Recent reports share a failure mode. As training proceeds, attention logits keep growing. When the softmax input becomes too large the distribution collapses onto a single point and gradients vanish, and at low precision it leads to numerical overflow.

The Qwen3 answer is QK-Norm. The report states that the QKV bias used in Qwen2 was removed and QK-Norm was introduced into the attention mechanism to ensure stable training (arXiv:2505.09388). The approach normalizes queries and keys, putting a ceiling on the magnitude of their dot product.

You can see it in the config too. When counting the parameters of Qwen3-8B there were 128 x 2, or 256, extra parameters per layer. Those are the query normalization and key normalization, each the size of the head dimension. GLM-4.5 exposes a use_qk_norm field set to true outright. The cost is 256 parameters per layer plus a little compute, which is to say almost free.

QK-Clip: A Different Approach

Kimi K2 solved the same problem from the optimizer side. According to its report, the team proposes MuonClip, which uses the token-efficient Muon optimizer while mitigating its instabilities with QK-Clip, and reports pre-training on 15.5 trillion tokens without a single loss spike (arXiv:2507.20534).

QK-Clip rescales the query and key projection weights after the update to bound the growth of attention logits. The report notes that in a mid-scale training run attention logits rapidly exceeded 1000, that K2 training used a threshold of 100, and that maximum logits rose quickly to the capped value and only decayed to a stable range after roughly 30 percent of the training steps.

The notable design choice is minimizing intervention. In practice only a small subset of heads exhibits exploding logits, so a per-head scaling factor is computed and applied to just those heads.

When More Normalization Is Added

DeepSeek-V3 added extra normalization because of its compression structure. The report states that additional RMSNorm layers are employed after the compressed latent vectors, and that additional scaling factors are multiplied in at the width bottlenecks (arXiv:2412.19437).

There is a general lesson here. Compressing a representation to save memory tends to distort the distribution of magnitudes at that point, so normalization has to be patched in. This is the concrete shape of what the third post described as MLA giving up compute and complexity.

Stability Also Lives in the Learning Rate Schedule

Structure alone does not secure stability. The DeepSeek-V3 report states that the learning rate was raised linearly from 0 to 2.2e-4 over the first 2K steps, held there until the model consumed 10T tokens, then decayed to 2.2e-5 over 4.3T tokens following a cosine curve. The gradient clipping norm is 1.0, and the batch size was gradually increased from 3072 to 15360 over the first 469B tokens.

The peak learning rates in Table 3 of Llama 3 are 3e-4 for 8B, 1.5e-4 for 70B, and 8e-5 for 405B. The larger the model, the lower the learning rate. The rule of thumb that bigger models are more prone to divergence shows up directly in the numbers.

Closing

Normalization and activation are the least conspicuous part of a config, yet they occupy the most pages in the reports. Now that RMSNorm, pre-norm, and SwiGLU are effectively standard, what reports actually discuss is the next problem: exploding attention logits. Qwen3 solved it structurally with QK-Norm, Kimi K2 solved it in the optimizer with QK-Clip. When you read a new model report, looking at what was added here tells you what that team struggled with.

References

Try It Yourself

Series

현재 단락 (1/33)

The previous posts dealt with parameters and memory. This one has a different subject. Normalization...

작성 글자: 0자원문 글자: 6,784자작성 단락: 0/33