Neural networks learn by fitting
A neural network is matrices with simple nonlinear functions between them. Gradient descent bends it to fit the data.
Play with it
Press Train and watch a tiny network (one hidden layer of 16 neurons) learn the curve. Switch the activation off to see why nonlinearity matters.
- activation
- SiLU
- steps
- 0
- loss (MSE)
- 0.434
- hidden neurons
- 16
Blue dots: the data. Amber: the network's current guess. 16 neurons, trained with Adam on the squared error.
The idea
One neuron computes a weighted sum of its inputs plus a bias, then applies an activation such as SiLU. A layer is many neurons side by side: a matrix multiply plus the activation.
Stack linear layers without activations and the result is still a single linear map, no matter how deep. The nonlinearity is what lets networks build curves, edges and, at scale, language.
Training repeats the loop you met in the last three modules: predict, measure the loss, backprop, step downhill. An LLM is this loop with a transformer as the network and next-token prediction as the task.
Neurons in artificial networks are not models of brain neurons. The name is historical; each is just a weighted sum and a simple function.
Check yourself
Why does a network need a nonlinear activation?
Without one, all layers multiply together into one linear map, which can only fit straight lines (or flat planes).
What are the four steps of a training loop?
Forward pass (predict), loss, backward pass (gradients), optimizer step (update weights).