RMSNorm (and Q/K norm)
A · 10-second intuition RMSNorm rescales a vector to a standard size, keeping its direction, before a sub-layer reads it.
Predict before you read
You multiply a vector by 10 before RMSNorm. What happens to the output?
Not started