Getting better by practising
A real model learns by practising, like you learn to ride a bike.
It guesses. Then it checks the right answer.
If it was surprised, it turns its knobs a tiny bit.
Press "Practise" and watch the surprise get smaller.
But if the steps are too big, it wobbles!
Your turn
Press Practise. Then try huge steps.
Toy calculation· real gradient descent on four wordsThe kitten drinks ?
- milk11 in 100
- water11 in 100
- rocks49 in 100
- crayons30 in 100
Next example to practise: "The kitten drinks milk." (The examples say milk 3 times and water 2 times.)
Surprise: 2.22 after 0 practice steps
The line is how surprised the machine is by the examples. Practice pushes it down. It can never reach zero, because sometimes the answer is milk and sometimes water.
Models learn by guessing, checking, and changing a tiny bit.
Grown-up corner: the precise version
- knobs = parameters (weights)
- surprise = loss (cross-entropy)
- step size = learning rate
Each step lowers the cross-entropy loss, -ln p(correct word), by gradient descent. For next-word scores the gradient is simply p - onehot(correct word).
The examples say "milk" three times and "water" twice, so the best possible answer is 60% milk, 40% water. Each practice step uses one example, so tiny steps settle very close to that and good steps hover nearby; huge steps overshoot back and forth, which is why real training uses small learning rates.
Qwen3-8B has 8,190,735,360 parameters, all trained this way (by backpropagation through every layer) on about 36 trillion tokens.