Learn Mode

How does Qwen actually work?

From text to the next token, then how the weights were learned. Real values come from Qwen3-8B-Base and are marked as measured; small worked examples are marked as toys.

2Learn

Follow one token through Qwen

Choose a stage. Each one is explained the same way: intuition, exact mechanism, tensor shapes, why it exists, and a common confusion. Panels marked Recorded Qwen measurement come from one real forward pass of the sentence "The researcher opened the notebook".

Text

1Intuition: The model never sees letters directly; it starts from a string that must be turned into integers.
2Exact mechanism: A Python string, for example "The researcher opened the notebook". Nothing has been computed yet.
3Tensor shapes: one string
4Why it exists: Language models are trained on text, but neural networks compute on numbers.
5Common confusion: The model does not read words one character at a time. Its unit of input is the token.

The running example

"The researcher opened the notebook"

3Learn

Transformer block microscope

One complete Qwen3 block, drawn from the order of operations verified in the installed implementation. The vertical rail on the left is the residual stream; it is only ever added to. Click any box.

Attention sub-layer (mixes positions)
MLP sub-layer (each position separately)

Block RMSNorm #1

input_layernorm

1Intuition: Rescale each position's vector to a standard size before attention reads it.
2Exact mechanism: Per position: divide by the root-mean-square over all 4096 coordinates, then multiply by a learned gain vector g (4096 numbers). Computed in float32. No mean subtraction, no bias.
RMSNorm(x) = g * x / sqrt((1/4096) sum_i x_i^2 + 1e-6)
3Tensor shapes: [B, T, 4096] -> [B, T, 4096]
4Why it exists: Keeps the input to attention on a stable scale no matter how large the residual stream has grown.
5Common confusion: This normalizes over hidden_size. The Q/K RMSNorm inside attention normalizes 128-dimensional head vectors: a different operation.

4Learn

RMSNorm lab

LayerNorm centers a vector and rescales it. RMSNorm only rescales it, by its root-mean-square magnitude, and then multiplies by a learned gain. Qwen3 uses RMSNorm everywhere, in two different roles that must not be confused.

RMSNorm step by step

Toy calculation· Toy vector, d = 4

x =gain g =

1. squares [4, 1, 16, 1]
2. mean squared magnitude (1/4) · 22 = 5.5
3. square root (with eps = 1e-6) RMS = sqrt(5.5 + 1e-6) = 2.3452
4. divide by RMS x / RMS = [0.853, -0.426, 1.706, 0.426] (its RMS is now 1)
5. learned gain g ⊙ (x / RMS) = [0.853, -0.426, 1.706, 0.426]

RMS(x) = sqrt( (1/d) Σ x_i² + ε )
RMSNorm(x) = g ⊙ x / RMS(x)

LayerNorm on the same x (gain 1, bias 0)

mean = 1.5
x - mean = [0.5, -2.5, 2.5, -0.5]
std = 1.8028
output = [0.277, -1.387, 1.387, -0.277]

What changed and what was kept

cosine(x, RMSNorm(x)) = 1 (direction kept exactly when the gain is uniform)
cosine(x, LayerNorm(x)) = 0.7687 (mean subtraction changes the direction)
Try the two buttons: scaling x leaves both outputs unchanged; adding a constant changes the RMSNorm output but not the LayerNorm output.

Recorded Qwen measurement

The stream grows by orders of magnitude; the normalized copy follows the learned gain

Last token of "The researcher opened the notebook". The residual stream entering each block grows by orders of magnitude; the input that attention actually reads has RMS set by the learned gain of that block (compare the gain chart), not by the size of the stream. The stream itself is never normalized between blocks.

Recorded Qwen measurement· Stored weights

Learned gain g, mean over its 4096 entries

The gains are learned, not fixed at 1. Block 1's input gain averages 0.0114; the final norm's gain averages 2.01. Because the gain is a vector, RMSNorm can change the relative sizes of coordinates; a negative gain entry would flip a coordinate's sign. The smallest input_layernorm gain entry in any block is -0.13672.

Block RMSNorm and Q/K RMSNorm are different operations

Recorded Qwen measurement· Layer 1, last token

	Block RMSNorm	Q / K RMSNorm
Modules	input_layernorm, post_attention_layernorm, model.norm	self_attn.q_norm, self_attn.k_norm
Normalizes	the whole hidden vector of a position	one head vector (Q or K) of a position
Dimension	`4096`	`128`
Where	before attention, before the MLP, after the last block	inside attention, after q_proj/k_proj and reshape, before RoPE
Gain shared across	nothing (one 4096-vector per norm)	all 32 query heads (q_norm) or all 8 KV heads (k_norm) of a layer
V normalized?	No. Values are neither normalized nor rotated.
Measured, block 1	residual RMS 0.0268 → normalized input RMS 0.0115	query head RMS 0.023 to 0.092 → 0.88 to 1.96 across the 32 heads

RMSNorm is applied to a copy of the stream that feeds a sub-layer. In a pre-norm block, x' = x + Attn(RMSNorm(x)): the unnormalized x is what gets added to. That is why the stream can keep growing while every sub-layer still sees a well-scaled input.

5Learn

Attention, GQA and RoPE

Attention is the only computation that moves information between positions. Qwen3 adds two details to the textbook version: Q/K RMSNorm and grouped-query attention.

Q, K and V are three linear projections of the same normalized input

Q = X W_q K = X W_k V = X W_v

Q, query. "What am I looking for?"

K, key. "What do I contain or advertise?"

V, value. "What should be passed on if I am selected?"

These phrases are intuitive analogies, not literal semantics. The model learns three matrices that make the computation below useful for predicting the next token.

Projection	Weight shape	Output per position	After reshape and transpose
q_proj	`[4096, 4096]`	`4096`	`[B, 32, T, 128]`
k_proj	`[4096, 1024]`	`1024`	`[B, 8, T, 128]`
v_proj	`[4096, 1024]`	`1024`	`[B, 8, T, 128]`

A linear projection is a matrix multiply: each output number is a weighted sum of the 4096 inputs. Reshape splits the 4096 query numbers of each position into 32 heads of 128; transpose moves the head axis before the position axis so that each head can be processed as its own [T, 128] matrix. No biases (attention_bias = false).

Grouped-query attention

Recorded Qwen measurement· Head counts verified

K/V 0

K/V 1

K/V 2

K/V 3

K/V 4

K/V 5

K/V 6

K/V 7

32 query heads; 8 key/value heads; 4 query heads per KV head. KV cache per token (all 36 layers, bf16): 144 KiB. Qwen3-8B's verified value is 144 KiB, 4x smaller than full multi-head attention.

Why: during generation the K and V of every past token are stored. Fewer KV heads means less memory and less memory bandwidth per generated token, while keeping 32 distinct query heads, each with its own attention pattern.

Where Q/K RMSNorm sits

q = q_proj(RMSNorm₁(x)) → view [B, T, 32, 128]
q = q_norm(q) over the last 128 dims, per head
k = k_norm(k_proj(...).view(...)) same for keys
v = v_proj(...).view(...) no norm
q, k = RoPE(q, k)
scores = q kᵀ · 0.0884 (= 1/sqrt(128))

Order verified in the installed transformers source. Normalizing q and k bounds the size of their dot products.

Rotary position embedding (RoPE)

Toy calculation· 2D toy, angle 0.5 rad per position

Query position t = 3Key position s = 1

Position 3 rotates q by 3 × θ = 1.5 rad; position 1 rotates k by 0.5 rad (dashed: before rotation).

Score q·k after rotation: 0.815. Move both positions forward by 4: 0.815, the same, because the score depends only on t - s = 2.

RoPE does not add a position vector to the token. It rotates Q and K, so the attention score carries their relative position. V and the residual stream are never rotated.

In the real model (verified): each 128-dimensional Q and K head vector is split into 64 pairs, dimension i paired with i + 64 ("rotate half"). Pair i rotates at frequency θ_i = 1,000,000^-2i/128, so wavelengths range from 6.28 tokens (fastest pair) to about 5,063,256 tokens (slowest). It is applied after Q/K RMSNorm.

Causal attention on four tokens

Toy calculation· Toy Q, K, V with d_head = 2

query \ key	A	B	C	D
A	0.707	0	0.707	0.283
B	0.354	0.707	-0.354	0.566
C	1.061	0.707	0.354	0.849
D	0	-0.707	0.707	-0.424

The visible pattern is lower-triangular: at position t the model cannot use positions after t. Training predicts every next token in parallel, so without the mask position t could simply read token t+1. A masked score of -infinity becomes a softmax weight of exactly 0.

Real attention weights

Recorded Qwen measurement· Qwen3-8B-Base, eager attention

block query head

Block 9, head 0: "The researcher opened the notebook"

query \ key	<eot>	The	␣researcher	␣opened	␣the	␣notebook
<eot>	1.00
The	0.91	0.09
␣researcher	0.83	0.15	0.02
␣opened	0.39	0.20	0.03	0.38
␣the	0.18	0.03	0.05	0.64	0.11
␣notebook	0.27	0.03	0.03	0.46	0.19	0.02

Each row sums to 1. Hatched cells are future positions the causal mask hides.

In block 9, 12 of 32 heads put more than half of their weight (averaged over query positions 1 to 5) on position 0, the document-start token. Weight concentrating on the first position is a commonly reported pattern (often called an attention sink); reading it as "the head ignores the sentence" would be an over-interpretation. Only the blocks listed were exported.

From head outputs to residual addition #1

1. Each head returns a weighted sum of values: [B, 32, T, 128]
2. Transpose and concatenate the heads: [B, T, 4096]
3. Output projection o_proj: [B, T, 4096] → [B, T, 4096]
4. Residual addition #1: x' = x + attention update

Recorded Qwen measurement Block 1, last token: ‖x‖ = 1.71, ‖attention update‖ = 3.61, RMS of x' = 0.065.

Attention(Q, K, V) = softmax( Q Kᵀ / sqrt(d_head) + M_causal ) V

6Learn

SwiGLU MLP and the residual stream

After attention has moved information between positions, the MLP transforms what each position holds. Its update is added to the stream: residual addition #2.

The exact path in Qwen3MLP

Recorded Qwen measurement· Dimensions verified

gate = gate_proj(x) [B, T, 4096] → [B, T, 12288]
up = up_proj(x) [B, T, 4096] → [B, T, 12288]
activated_gate = SiLU(gate) elementwise
combined = activated_gate * up elementwise
output = down_proj(combined) [B, T, 12288] → [B, T, 4096]

MLP(x) = W_down ( SiLU(W_gate x) ⊙ (W_up x) )

Three weight matrices, no biases: 150,994,944 parameters per block, 78% of the block.

The distinction to remember

Why can the MLP change the meaning of a token if it never looks at other positions? Because by the time it runs, attention has already copied information from earlier positions into this position's vector. The MLP then transforms that combined vector. Mixing (attention) and transforming (MLP) alternate 36 times.

One gate unit

Toy calculation

gate = gate_proj(x) = 1.50up = up_proj(x) = 0.80

SiLU(gate) = gate · sigmoid(gate) = 1.2264; combined = SiLU(gate) · up = 0.9811

A strongly negative gate gives SiLU near 0 and switches the feature off, whatever up says. That multiplicative gate is the "GLU" in SwiGLU; down_proj then mixes all intermediate features back into the hidden size.

Recorded Qwen measurement

Size of each update, last token, every block

"The researcher opened the notebook", last token "notebook". Updates are added, so the stream after block k is the embedding plus the sum of all 2 × k updates so far.

Residual addition #2 in block 36

Recorded Qwen measurement

‖x′‖ after attention	1716.3
‖MLP update‖	2180.1
‖x″ = x′ + update‖	2101.2
cosine(block input, block output)	0.338

In this sentence the last block's MLP update is larger than the stream it is added to, so the final block changes the direction of the last token's vector substantially. One sentence is an illustration, not a general finding.

Shapes: [B, T, 4096] in and out; the 12288-wide intermediate exists only inside the MLP.

7aLearn · inference

Final RMSNorm, LM head, logits and decoding

After the last block, one more RMSNorm and a linear layer turn each position's 4,096-number vector into 151,936 scores.

h = model.norm(x) [B, T, 4096] → [B, T, 4096]
z = lm_head(h) [B, T, 4096] → [B, T, 151,936] (separate weights, not tied)
p = softmax(z) over the last axis

p_i = exp(z_i) / Σ_j exp(z_j)

LOGIT ≠ PROBABILITY. In the table below the top logit is 32.3 but its probability is 0.162. Adding the same constant to every logit would leave every probability unchanged.

Decoding lab

Recorded Qwen measurement· Real top-50 logits; exact normalizers

temperature T

token	logit	p (T=1)	sampling p
.	32.25	0.162	0.162
.↵	31.75	0.098	0.098
,	31.63	0.087	0.087
␣and	31.63	0.087	0.087
↵	31.38	0.067	0.067
␣to	31.38	0.067	0.067
␣The	30.63	0.032	0.032
↵↵	30.50	0.028	0.028
␣the	30.50	0.028	0.028
␣for	30.50	0.028	0.028
.↵↵	30.38	0.025	0.025
␣on	29.88	0.015	0.015
's	29.88	0.015	0.015
␣opened	29.75	0.013	0.013
␣notebook	29.50	0.010	0.010

Logits are divided by T before softmax. T < 1 sharpens the distribution, T > 1 flattens it. The 50 exported tokens hold 86.5% of the probability at T = 1; the rest is spread over the other 151,886 rows. These are inference-time decoding strategies. They are not how the model was pretrained.

Intermediate diagnostic projection of the last position ("␣notebook")

Recorded Qwen measurement

Each row applies Qwen's own final RMSNorm and LM head to the residual stream after a block. This is a diagnostic, not "Qwen's prediction at layer k": the final norm and LM head were trained to read only the last residual. The model's actual prediction is the last row.

depth	top projected tokens (diagnostic probability)	rank of final choice "."
0 (embeddings)	␣circ 0.251 emm 0.195 阐明 0.032	74,658
1	切れ 0.103 艇 0.031 scopes 0.028	110,413
2	␣advisers 0.040 公共服务 0.040 ␣sergeant 0.033	56,690
3	公共服务 0.163 services 0.099 ␣vouchers 0.034	59,733
4	方式进行 0.232 时代的 0.059 奎 0.052	62,167
5	-scripts 0.047 acher 0.039 -span 0.034	34,068
6	-record 0.053 式的 0.032 -span 0.023	25,817
7	edImage 0.071 aja 0.059 里的 0.030	44,957
8	ypad 0.056 /screen 0.036 ␣Ink 0.025	34,178
9	-span 0.204 /screen 0.102 aud 0.085	31,211
10	电脑 0.067 folders 0.052 ipt 0.034	82,355
11	电脑 0.155 袋 0.065 里的 0.035	26,220
12	袋 0.072 限り 0.068 bag 0.039	11,777
13	限り 0.222 iesz 0.041 /email 0.030	21,808
14	限り 0.153 iesz 0.053 _ang 0.047	15,966
15	党工委 0.072 iesz 0.056 限り 0.039	52,454
16	党工委 0.069 iesz 0.034 /Table 0.032	55,131
17	�� 0.053 iesz 0.021 ês 0.014	77,325
18	␣Bible 0.037 /lab 0.027 .old 0.017	93,247
19	�� 0.069 acci 0.037 _reading 0.035	94,468
20	/language 0.053 限り 0.034 itian 0.015	80,270
21	ator 0.248 /Table 0.017 限り 0.010	112,231
22	ator 0.039 ␣([[ 0.032 _hp 0.030	139,140
23	限り 0.054 _reading 0.033 _question 0.033	88,564
24	_reading 0.060 _visible 0.053 /lab 0.032	43,987
25	扉 0.059 ␣containing 0.046 写字 0.038	10,977
26	电脑 0.077 /book 0.064 ␣containing 0.044	5,146
27	␣lid 0.137 ␣drawer 0.083 扉 0.065	5,894
28	␣halfway 0.255 ␣containing 0.069 ␣drawer 0.039	4,511
29	␣halfway 0.173 ␣lid 0.077 ␣containing 0.072	5,296
30	␣notebook 0.691 ␣opened 0.057 ␣opener 0.042	2,601
31	␣notebook 0.862 ␣opened 0.062 -opening 0.014	2,725
32	␣notebook 0.531 ␣opened 0.322 ␣opening 0.072	1,062
33	␣notebook 0.583 ␣opened 0.243 ␣open 0.079	332
34	␣notebook 0.557 ␣opened 0.383 ␣opening 0.017	32
35	␣notebook 0.171 ␣opened 0.118 .↵ 0.098	8
36 · actual output	. 0.162 .↵ 0.098 ␣and 0.087	1

KV cache

Recorded Qwen measurement· Sizes computed from the verified config

Without a cache, generating token n+1 would rerun the whole prefix: every previous position's K and V would be recomputed in every layer, for every new token.

With a cache, K and V of past positions are stored once per layer. A new token computes only its own Q, K and V, appends its K and V, and attends over cached + new keys.

prompt → compute K, V for all positions → cache
new token → its Q, K, V → append K, V
attend: Q_new against [K_cache; K_new], weights on [V_cache; V_new]

Queries are never cached: only the newest position asks a question. Keys are cached after RoPE.

Context length: 4,096 tokens

variant	KV heads	per token	at 4,096 tokens
GQA (Qwen3-8B)	8	144 KiB	576 MiB
MHA	32	576 KiB	2.25 GiB
MQA	1	18.0 KiB	72.0 MiB

2 (K and V) × 8 × 128 × 36 layers × 2 bytes = 147,456 bytes

Per sequence, bfloat16. Fewer KV heads means a proportionally smaller cache.

7bLearn · training, kept separate from inference

How Qwen learns

Training and inference use the same forward pass but differ in everything around it. This section is only about training.

tokens
forward pass
logits
cross-entropy
loss
backpropagation
gradients
optimizer
weight update

L = −(1/T) Σ_t log p_θ(x_(t+1) | x_≤t)

All 8,190,735,360 parameters change during pretraining. The model does not store documents in a database: what it learns is spread across these weights, which is why it can sometimes reproduce famous text (see the Recall Microscope) without holding a copy that can be looked up.

Teacher forcing

Toy calculation· Toy sequence

position	0	1	2	3
input (true tokens)	A	B	C	D
target (next token)	B	C	D	E

All four predictions are made in one parallel forward pass. Each position sees the TRUE previous tokens (never its own earlier guesses), and the causal mask stops it from seeing its own target.

The training objective on our sentence

Recorded Qwen measurement· Forward pass only, no weights changed

context	target	p(target)	rank	loss −log p	model's top 3
<\|endoftext\|>	The	0.014	10	4.29	Human What How
The	␣researcher	2.3e-8	56,550	17.59	The ↵ 的
The␣researcher	␣opened	2.1e-5	1,415	10.78	's ␣is ␣will
The␣researcher␣opened	␣the	0.192	1	1.65	␣the ␣a ␣their
The␣researcher␣opened␣the	␣notebook	9.5e-4	99	6.96	␣door ␣researcher ␣discussion

Cross-entropy averaged over the 5 targets: 8.254 nats. During pretraining this number, averaged over trillions of tokens, is what gradient descent pushes down. A surprising word ("␣researcher") costs the most.

Backprop explorer

Toy calculation· Two-weight network

Network: h = w₁ · x, ŷ = w₂ · h, loss = (ŷ − y)². Input x = 1.5, target y = 3, learning rate 0.05.

forward h = 0.5 × 1.5 = 0.75
forward ŷ = 0.8 × 0.75 = 0.6
loss (0.6 − 3)² = 5.76
derivative ∂L/∂ŷ = 2(ŷ − y) = -4.8
chain rule ∂L/∂w₂ = ∂L/∂ŷ · h = -3.6
chain rule ∂L/∂w₁ = ∂L/∂ŷ · w₂ · x = -5.76
update w ← w − lr · ∂L/∂w → w₁ = 0.788, w₂ = 0.98

Loss after each step: 5.76

A gradient says how the loss would change if a parameter were nudged. Training Qwen3-8B applies the same chain rule to all 8,190,735,360 parameters at once (automatic differentiation), and an optimizer such as AdamW turns gradients into updates.

Training vs inference

	Training	Inference
Weights	updated	frozen
Inputs	true tokens (teacher forcing)	the model's own previous outputs
Positions	all at once, in parallel	one new token per step
Output used	loss on every position	last position only, then a decoding rule
KV cache	not needed	used to avoid recomputation

Base model vs instruction model

Base: trained mainly by causal language modeling (the loss above) on a large corpus. The checkpoint card lists its training stage as "Pretraining". Instruct: the base model plus post-training (instruction tuning, preference or reinforcement learning).

LatentLens starts with the base model because we want representations of the pretrained language model without making assistant alignment the main confounder. This does not mean the base model is "neutral": its pretraining data and recipe also shape what it represents.

Decoding rules (greedy, temperature, top-k, top-p) and the KV cache belong to inference. Teacher forcing, the loss, gradients and the optimizer belong to training.

Learn · teach-back

Test my understanding

Answer out loud first, then reveal the explanation. If an answer surprises you, return to the section it came from.

residual

What is the residual stream?

rmsnorm

Why does Qwen normalize before each sub-layer?

rmsnorm

What is the difference between LayerNorm and RMSNorm?

attention

What does Q represent?

attention

What does K represent?

attention

What does V represent?

attention

Why does Qwen3-8B have fewer KV heads than query heads?

rope

Where is RoPE applied?

attention

Why is causal masking needed?

mlp

Does the MLP mix different token positions?

residual

Why use residual connections at all?

output

What does the LM head output?

output

What is the difference between a logit and a probability?

inference

What is cached during autoregressive generation?

training

What changes during training?

training

Why is inference different from training?