The MLP (SwiGLU)
A · 10-second intuition Attention moves information between positions; the MLP transforms the features within each position.
Predict before you read
Does the MLP at position 4 read the hidden state of position 2?
Not started
A · 10-second intuition Attention moves information between positions; the MLP transforms the features within each position.
Does the MLP at position 4 read the hidden state of position 2?