Four toy tokens, three input numbers each, one head with d_head = 2. Change one token's input and follow the effect through every step. The recorded mode shows real Qwen3-8B-Base attention probabilities, with only the controls that recording supports.
Inputs
Toy calculation| row | x0 | x1 | x2 |
|---|---|---|---|
| A | 1.00 | 0.00 | 0.50 |
| B | 0.00 | 1.00 | 0.50 |
| C | 1.00 | 1.00 | 0.00 |
| D | 0.50 | -1.00 | 1.00 |
1. Projections
Q = XW_q, K = XW_k, V = XW_v| row | q0 | q1 |
|---|---|---|
| A | 1.25 | 0.25 |
| B | 0.25 | 1.25 |
| C | 1.00 | 1.00 |
| D | 1.00 | -0.50 |
| row | k0 | k1 |
|---|---|---|
| A | 1.50 | 0.25 |
| B | 0.50 | 0.75 |
| C | 1.00 | 1.50 |
| D | 1.50 | -1.25 |
| row | v0 | v1 |
|---|---|---|
| A | 1.25 | 0.25 |
| B | 0.25 | 1.25 |
| C | 1.00 | 1.00 |
| D | 1.00 | -0.50 |
2–4. Scores, scaling, mask, shares
rows = queries, columns = keys| row | A | B | C | D |
|---|---|---|---|---|
| A | 1.94 | 0.81 | 1.63 | 1.56 |
| B | 0.69 | 1.06 | 2.13 | -1.19 |
| C | 1.75 | 1.25 | 2.50 | 0.25 |
| D | 1.38 | 0.13 | 0.25 | 2.13 |
| row | A | B | C | D |
|---|---|---|---|---|
| A | 1.37 | 🔒 | 🔒 | 🔒 |
| B | 0.49 | 0.75 | 🔒 | 🔒 |
| C | 1.24 | 0.88 | 1.77 | 🔒 |
| D | 0.97 | 0.09 | 0.18 | 1.50 |
| row | A | B | C | D |
|---|---|---|---|---|
| A | 100% | 🔒 | 🔒 | 🔒 |
| B | 43% | 57% | 🔒 | 🔒 |
| C | 29% | 21% | 50% | 🔒 |
| D | 28% | 12% | 13% | 48% |
Query D shares: A 28.1%, B 11.6%, C 12.7%, D 47.7% (before your edits: A 28.1%, B 11.6%, C 12.7%, D 47.7%). Each row adds up to 100%.
5. Weighted sum of values
Toy calculationout(D) = 0.28·V_A + 0.12·V_B + 0.13·V_C + 0.48·V_D
Dashed outline: before your edits. Filled bar: now.
Grouped-query attention in Qwen
Qwen3-8B has 32 query heads but only 8 key/value heads: each K/V head is shared by 4 query heads, which makes the KV cache 4 times smaller. Each query head still has its own pattern.