Reading a giant library
How does a model learn all this?
It reads a giant library, more than one person could read in thousands of lifetimes.
While it reads, it plays the guessing game again and again, and turns its knobs each time.
Bigger models have more knobs. Some have more than a trillion!
Some giant models work like a school full of helpers. For each piece, only a few helpers answer.
Your turn
Pick a model and count its knobs. Then tap a word and see which helpers answer.
Toy calculation· knob counts from papers and model cardsCount the knobs
GPT-2 has about 1.5 billion knobs.
One dot is one billion knobs. Sources: model papers and model cards (see the grown-up corner and the timeline).
A school of 512 helpers
Qwen3.8 has 512 helpers on each floor. For every piece, a little router picks 10 helpers, plus 1 helper who always answers. Tap a word:
Which helpers light up is pretend. The numbers 512, 10 and 1 are from Qwen3.8's model card.
Models learn by reading a huge amount, and giant ones share the work among helpers.
Grown-up corner: the precise version
- knobs = parameters
- school of helpers = mixture of experts
- reading the library = pretraining
Qwen3 was pretrained on about 36 trillion tokens. A person reading 250 words a minute, 8 hours a day, every day for 80 years reads about 3.5 billion words, roughly 4.7 billion tokens at about 1.3 tokens per English word. 36 trillion tokens is about 7,700 such lifetimes.
In a mixture-of-experts model a small router picks a few expert MLPs for each token. Qwen3.8 has 512 experts and uses 10 routed experts plus 1 shared expert per token, so only 95B of its 2.4T parameters work on any one token. Which helpers light up here is a toy; the counts are from the model card.