Level 2 · Build it

Write an LLM yourself.

Eight short coding steps in learn/build/. In each you fill in a few marked blocks and run its checks, which compare your code with PyTorch's own. By step 6 you have a working transformer with Qwen3's block design that learns to write.

How it works

1
Open a step

Read the top comment, then find each raise NotImplementedError("TODO"). The comment above it gives the formula.

2
Run its checks

From learn/build: ../../.venv/Scripts/python -m pytest -k s01. Each failing check says what is wrong.

3
Run the file

Every step prints something worth seeing. After step 6, train_tiny_gpt.py trains your own model.

The eight steps

  1. 1
    Autograd from scratchs01_micrograd.py

    A Value that records how it was computed, five operations with their derivatives, and backward(). Then train a tiny network on XOR with it.

    Atlas: How every box learns →Foundations first →
  2. 2
    A counting language models02_bigram.py

    Count which character follows which, turn counts into probabilities, score text with cross-entropy, and sample.

    Atlas: Logits, Softmax → sample →Foundations first →
  3. 3
    A byte-level BPE tokenizers03_bpe.py

    Pair counting, merging, training merges, encoding and decoding, with a perfect round trip on any Unicode text.

    Atlas: Tokenizer, Token IDs →
  4. 4
    Attention by hands04_attention.py

    A numerically stable softmax, the causal mask, scaled dot-product attention and GQA's repeat_kv, checked against PyTorch.

    Atlas: Inside attention →Foundations first →
  5. 5
    RMSNorm, RoPE, SwiGLUs05_parts.py

    Qwen3's three distinctive parts, with checks that RoPE keeps lengths and makes scores depend only on distance.

    Atlas: RoPE →
  6. 6
    A tiny Qwen-style GPTs06_tiny_gpt.py

    Assemble the block and the full model with Hugging Face's Qwen3 module names, then train it to write sentences.

    Atlas: The whole diagram →
  7. 7
    Sampling and the KV caches07_generate.py

    Temperature, top-k and top-p, plus a KV cache whose logits match the uncached model exactly.

    Atlas: Keys and values →
  8. 8
    LoRA fine-tunings08_lora.py

    A low-rank add-on that starts as a no-op, trains a tiny fraction of the weights and merges back exactly.

    Atlas: Wq → Q →

What step 6 gets you

A real run of the reference solution: a 796,800-parameter model trained on a small synthetic story corpus. The loss falls from 3.581 to 0.163 nats per character.

cd learn/build
../../.venv/Scripts/python -m pytest -k s06
../../.venv/Scripts/python train_tiny_gpt.py
Before training

The tlSZbeTolwbrwIMniyLSLTTglbTiIgp.vZBureydiSkmipodp.ZdrbAgwdBeM.ihvnmor,hwcpck,

After training

The fox was curious, so it slept. Sam was hungry and the bear was curious. Zoe was brave and the dog was brave.

Stuck?

  • solutions/ has the reference answers. Try for 20 minutes first: the struggle is where the learning happens.
  • python make_exercises.py --force resets every exercise and keeps your old files as .py.bak.
  • BUILD_IMPL=solutions python -m pytest shows what passing looks like.