The residual stream
A · 10-second intuition Each position carries one 4,096-number vector up through the model, and every sub-layer only adds an update to it.
Predict before you read
What happens to the residual stream when an attention sub-layer's output is all zeros?