A flashlight called attention
Some words point back to other words.
The dog chased the ball because it was hungry. Who was hungry?
The dog! Balls don't get hungry.
A model has a flashlight called attention. Each word shines it on the earlier words that help it.
Change the last word, and watch the flashlight move.
Your turn
Tap a different last word. Where does the flashlight shine?
Toy calculation· the real formula with hand-picked numbersThe thick lines show where "it" shines its flashlight. The numbers are its shares out of 100.
The dog chased the ball because it was hungry. What was hungry?
Attention lets each word look back at the words that matter.
Grown-up corner: the precise version
- flashlight = attention
Attention scores each earlier position by query · key / √d, turns the scores into shares with softmax, and mixes those positions' value vectors. A word can only look back, never forward (the causal mask).
In this toy the keys and queries have two hand-picked numbers so the idea is visible; the formula is the real one. Qwen3-8B has 32 attention heads in each of its 36 blocks.