Gradients and gradient descent
The gradient says which way is uphill for the loss. Training takes small steps the opposite way, downhill.
Play with it
Press Step to move the ball downhill on the loss curve. Change the learning rate and see what too small and too large do.
- w
- 3.20
- loss
- 2.84
- slope
- 3.70
- steps
- 0
w ← w − 0.15 × slope
Try 0.05 (slow), 0.15 (good), 1.0 (overshoots). Starting on the right, small steps settle in the local dip, not the deeper one on the left.
The idea
The derivative of the loss with respect to one weight is the slope: how much the loss changes if that weight is nudged. The gradient is that slope for every weight at once, 8 billion of them for Qwen3-8B.
Gradient descent updates each weight by w ← w − learning_rate × slope. Too small a learning rate and training crawls; too large and it overshoots and can blow up.
Real training uses AdamW, which keeps a running average of past gradients and scales each weight's step individually, plus a learning-rate schedule (warm-up, then decay). The core idea is the same small step downhill.
Using a model does not train it. During a chat, no gradients are computed and no weights change; learning only happens in a separate training run.
Check yourself
The slope at w is +3 and the learning rate is 0.1. What is the new w?
w − 0.3. Positive slope means the loss rises to the right, so step left.
What happens with a learning rate that is far too large?
The steps overshoot the minimum, the loss bounces or grows, and training can diverge.