Qwen3-8B-Base · every size verified from the checkpoint

The transformer,
in one picture.

Start with the whole model. Click any part to see what it does, what shape its data has and how to remember it. Then zoom into a block, and into attention and the MLP. The shapes and sizes are Qwen3-8B's real ones.

parameters
8.19B
blocks
36
numbers per token
4,096
query / KV heads
32 / 8
vocabulary rows
151,936
The whole model
Text“The researcher opened the notebook”Tokenizerbyte-level BPEToken IDsintegersEmbedding lookup151,936 × 4,096 tableTransformer blockAttentionMLP→each adds an update · tap to openFinal RMSNormmodel.normLM head4,096 → 151,936Logitsone score per entrySoftmax → sampleprobabilities“.”next token, appended
  • Residual stream
  • Normalization
  • Attention
  • MLP
  • Input / output
Remember it

The whole model is four verbs.

Hang everything else on these. The middle two repeat 36 times, and every one of them writes onto the same running vector per token.

  1. 1
    Look up
    Embedding

    Each token id picks one row of a 151,936 × 4,096 table.

  2. 2×36
    Mix
    Attention

    Tokens look back at earlier tokens and borrow what they need.

  3. 3×36
    Think
    MLP

    Each token is transformed alone: expand, gate, contract.

  4. 4
    Read out
    LM head

    Score all 151,936 entries, softmax, pick one, repeat.

Ten-second recall

Answer in your head before you open each one. Pulling a fact back out is what makes it stick.

Where is the only place tokens exchange information?

Attention. The MLP, the norms and the residual additions all work on each position separately.

What does a block do to the residual stream?

It adds two updates, one from attention and one from the MLP. It never overwrites or normalizes the stream itself; the sub-layers read a normalized copy.

Logits or probabilities: what comes out of the LM head?

Logits, one raw score per vocabulary entry. Softmax turns them into probabilities, and a decoding rule picks one token.

Which of Q, K and V get rotated by RoPE?

Only Q and K. Position enters through how well a query matches a key; V passes content and is never rotated or normalized.

Go deeper

Level 0 · start here
Foundations

New to the maths? Vectors, matrices, softmax, loss, gradients and backprop, each as something you can play with.

Level 2 · code
Build it

Write autograd, a tokenizer, attention and a tiny Qwen-style GPT yourself. Automatic checks tell you when it is right.

Level 3 → 4
Mastery and security

Training, inference, RAG, agents and MCP, then the AI-security path from threat models to red teaming.

Everything else, including the Storybook, labs, research and professor mode, is under More in the top bar.