Predicting the next token
A language model gives a probability to every possible next token. Generating text is sampling one, appending it, and repeating.
Play with it
This model only counts which character follows which in a short text. Click a character to see its next-character probabilities, or generate text.
the cat sat on the mat. the dog sat on the log. a cat saw a dog. the dog ran to the cat. the cat ran up a tree. a bird sat in the tree and sang. the dog and the cat sat on the mat and slept.
the dog. cathee in d sathe thee. tra trd biraw thee cat. the athee. can t bire re t ca upth
Locally plausible, globally nonsense: it never sees more than one character back.
The idea
Text becomes tokens, and the model is trained on one task: given all the tokens so far, predict the next one. Every position in every document is a training example.
The simplest model looks at only the previous character and counts. It already has the full machinery: a vocabulary, a probability for each next token, a loss, and a sampling loop.
Its weakness is obvious when it writes: it has no memory beyond one character. Everything in a transformer exists to make the prediction use the WHOLE context, well.
An LLM does not plan a whole answer and then type it out. It produces one token at a time; anything that looks like a plan has to be represented in its hidden states as it goes.
Check yourself
How many training examples does one 1,000-token document give?
About 1,000: every position predicts the token after it.
What is the bigram model's big limitation?
It only sees the previous token, so it cannot use anything earlier in the text.