Chapter 3 of 14 · Words and numbers

Every piece gets a secret code

Computers love numbers.

So every piece gets its own list of numbers, like a secret code.

Words that mean similar things get similar codes.

That is how the computer can tell that a puppy is a bit like a kitten.

Make a code with the sliders, and see which words live nearby.

Your turn

Move the sliders. Who lives next to your word?

Toy calculation Recorded Qwen measurement

Challenge: make a code that lives right next to the elephant

Your code is [0.50, 0.50, 0.50]. Its nearest neighbours: 🌳 tree, 🚗 car, 🎂 cake.
animalsthings to eatsmall ←→ big🐱kitten🐶puppy🐭mouse🦁lion🐘elephant🐋whale🍎apple🍌banana🥕carrot🍕pizza🎂cake🚲bike🚗car🚌bus🚀rocket🌳treeyour word
Up means more like an animal, down means more like food, right means bigger. The dashed lines go to the three closest codes.

Our toy code has 3 numbers. A real model, Qwen, gives every piece 4,096 numbers! Here are the first 8 for the piece "The":

[-0.039, 0.016, -0.003, 0.022, -0.026, 0.019, 0.006, 0.042, … 4,088 more]

What you learned

Every piece becomes a list of numbers, and similar words get similar lists.

Finish "Your turn" to earn the Code maker sticker.
Grown-up corner: the precise version
  • secret code = embedding vector

Each token id selects one row of the embedding matrix: 151,936 rows of 4,096 numbers in Qwen3-8B. The rows are learned in training.

The three sliders are a toy. A real model's 4,096 numbers are not labelled "alive" or "big"; meaning is spread across many directions, and nobody chose them by hand.