Loss: measuring how wrong
Cross-entropy loss is −log(probability the model gave to the right answer). Confident and right: near 0. Confident and wrong: huge.
Play with it
Slide the probability the model gives to the correct next token. Watch the loss and the perplexity.
- loss = −ln p
- 0.92
- perplexity = e^loss
- 2.5
- as bits
- 1.32
- knows nothing (Qwen)
- 11.9
The idea
For every position in the training text, the model outputs a distribution over the next token. The loss looks only at the probability p it gave to the token that actually came next, and scores it −ln p.
p = 1 costs 0. p = 0.5 costs 0.69. p = 0.01 costs 4.6. The curve punishes confident mistakes much more than hesitant ones, which pushes the model toward honest probabilities.
Averaged over many tokens this is the number training drives down. e^loss is the perplexity: “as unsure as choosing uniformly among this many tokens”. A model that knows nothing about Qwen's 151,936 entries scores ln(151,936) ≈ 11.9.
Low training loss is not the same as being right or truthful. It means the model predicts its training text well, which can include predicting common mistakes.
Check yourself
The model gave the right token probability 1. What is the loss?
0, because −ln 1 = 0.
Loss 2.3 nats per token. Roughly what perplexity?
e^2.3 ≈ 10: as unsure as choosing among about 10 equally likely tokens.