Chapter 2 of 14 · Words and numbers

Words are made of pieces

Before the computer can guess, it cuts the words into pieces.

Short, common words stay whole, like "the" and "cat".

Long or rare words are cut into a few pieces, like building blocks.

Every piece has its own number.

Tap a word to see how a real model cuts it.

Your turn

Tap a word card. Count its blocks.

Recorded Qwen measurement· real Qwen3 tokenizer output

First, guess: how many blocks will unbelievably be cut into?

Open three cards to earn the sticker. (0 of 3)

What you learned

The computer reads pieces of words, not whole words.

Finish "Your turn" to earn the Block builder sticker.
Grown-up corner: the precise version
  • piece = token
  • piece number = token id

Qwen3 uses a byte-level BPE tokenizer. Frequent strings are single tokens; rare words split into several. The space before a word is part of the token, so " notebook" and "notebook" have different ids.

Every split shown here is the real Qwen3 tokenizer output, recorded by LatentLens.