RoPE does not add a position number to the token. It rotates the query and key vectors by an angle that grows with position, so the attention score between two tokens depends on how far apart they are.
One rotating pair
Toy calculation· 2 numbers, 0.5 rad per positionScore q·k after rotation: 0.8150 (offset t − s = 2).
Score after moving both by 0: 0.8150: the same, because only the offset t − s matters.
The full model rotates 64 pairs at different speeds
Recorded Qwen measurement· exact angles from the recorded configurationEach 128-number Q or K head vector is split into 64 pairs (dimension i with i + 64). Pair i turns by θi = 1,000,000−2i/128 radians per position. Slow pairs barely move over a sentence; fast pairs spin quickly.
Angles are computed exactly from the verified θ = 1,000,000 and head dimension 128; wavelengths range from 6.28 to about 5,063,256 tokens. Rotation happens inside attention after Q/K RMSNorm, for Q and K only.
Where RoPE acts in Qwen3
Recorded Qwen measurement- Captured
- order verified in the installed transformers Qwen3Attention source
- Shape
- Q: [B, 32, T, 128], K: [B, 8, T, 128]
- Shown
- the operation order, not tensor values
- q = q_norm(q_proj(x)), k = k_norm(k_proj(x))
- q, k = rotate(q, k, positions 0 … T−1)
- scores = q kᵀ / √128
- v = v_proj(x): never rotated
Why rotations give relative position
(R(t)q)·(R(s)k) = qᵀR(t)ᵀR(s)k = qᵀR(s − t)k, because rotations are orthogonal and angles add. This holds for each pair, so it holds for the whole head vector.