A tower of thinking floors
The model is like a tall tower with many floors.
On every floor, each piece does two things.
First it looks around at the other pieces. That is the purple step.
Then it thinks on its own. That is the orange step.
It writes what it found in its blue notebook, and climbs up.
Qwen has 36 floors. At the top, the notebook turns into guesses.
Your turn
Press Climb, and ride the piece to the top of the tower.
Toy calculation Recorded Qwen measurementOn the ground floor.
- Look around at the other pieces (attention)
- Think on its own (MLP)
- Write it in the notebook, then climb
The real Qwen3-8B has 36 floors, and its notebook for each piece holds 4,096 numbers.
A model is a tower of the same kind of floor, stacked many times.
Grown-up corner: the precise version
- floor = transformer block (layer)
- looking around = attention
- thinking on its own = MLP
- notebook = residual stream
Each Qwen3 block: RMSNorm, attention, add to the residual stream; RMSNorm, SwiGLU MLP, add. Every floor has the same shape but its own weights.
The guesses at the top are the recorded Qwen3-8B-Base probabilities for the next token.