Start with the toy vector [2, −1, 4, 1]. Change its overall size and shift it, then compare what each normalization does. The gain sliders stand for the learned weights, which are separate from the factor computed from the input.
Controls
Toy calculation· 4 numbersepsilon (ε)
Input
Toy calculationRMSNorm (used by Qwen)
no mean subtraction- mean(x²) = 5.5000
- denominator = √(mean(x²) + ε) = 2.3452
- x / denominator, then × gain
LayerNorm (for comparison)
subtracts the mean- mean = 1.5000; x − mean = [0.50, -2.50, 2.50, -0.50]
- denominator = √(variance + ε) = 1.8028
- (x − mean) / denominator, then × gain (+ bias 0)
What to notice (computed live)
- RMSNorm output differs from the output for the original [2, −1, 4, 1] by at most 0.0000. Scaling alone barely changes it: the denominator scales with the input.
- LayerNorm output mean: 0.0000 (with unit gains it is always about 0). RMSNorm output mean: 0.6396.
- The gain multiplies after dividing: the learned weights g are fixed numbers, while the denominator (2.345) is recomputed for every input.
- ε = 1e-6: tiny, so it only matters for vectors that are almost all zeros. Try scale × 0.05 with ε = 4.
Recorded Qwen3-8B-Base RMSNorm gains (not the toy sliders)
Recorded Qwen measurement- Captured
- input_layernorm.weight and post_attention_layernorm.weight in each block, read from the stored checkpoint
- Shape
- [4,096] per norm, 36 blocks
- Shown
- mean of the 4,096 gain values in each block
Qwen uses ε = 1e-6 and computes RMSNorm in float32. Block 1's gain averages 0.0114; the final norm's averages 2.01. Q/K RMSNorm is a separate operation over 128-number head vectors with its own gains.
The formulas
RMSNorm(x) = g ⊙ x / √((1/d) Σ x_i² + ε)
LayerNorm(x) = g ⊙ (x − μ) / √((1/d) Σ (x_i − μ)² + ε) + b
If the mean μ is 0, the two coincide (apart from the bias). RMSNorm is cheaper and keeps the mean direction of the vector.