Backpropagation: the chain rule at scale
Backprop computes every gradient in one backward sweep, by multiplying local slopes along the path from the loss back to each weight.
Play with it
Move the inputs. The top numbers are the forward values; the coloured numbers are the gradients flowing back from the loss.
∂L/∂e = 2e = 2.00
∂L/∂w = ∂L/∂e · 1 · 1 · x = 4.00
∂L/∂x = ∂L/∂e · w = 3.00
White: forward values. Amber ∂: gradient of L with respect to that node.
The idea
A network is a chain of simple operations. The chain rule says: the effect of a weight on the loss = (effect of the weight on the next value) × (effect of that value on the loss).
So we go forward once to get every value, then backward once from the loss, multiplying local slopes. Each operation only needs to know its own derivative: for a + b both inputs get the incoming gradient; for a × b each input gets the gradient times the other input.
PyTorch's autograd does exactly this for tensors. In the Build track you write it yourself on single numbers in about 60 lines.
Backprop is not the learning rule. It only computes gradients; the optimizer (gradient descent, AdamW) uses them to change the weights.
Check yourself
z = x × y with x = 3, y = −2, and dL/dz = 5. What is dL/dx?
dL/dz × y = 5 × (−2) = −10.
For z = a + b, what gradient does a receive?
Exactly the incoming gradient dL/dz: addition passes gradients through unchanged.