Understand modern AI from the inside out.

Model card (Pretraining): 8.2B parameters, 6.95B non-embedding, 36 layers, 32 for Q and 8 for KV, context 32,768. Consistent with the counts above.

• ✓ Block is pre-norm: input_layernorm is applied before self_attn, and the residual is added after it

  residual = hidden_states
  hidden_states = self.input_layernorm(hidden_states)
  hidden_states, _ = self.self_attn(
  hidden_states = residual + hidden_states
  residual = hidden_states
  hidden_states = self.post_attention_layernorm(hidden_states)
  hidden_states = self.mlp(hidden_states)
  hidden_states = residual + hidden_states

• ✓ Second sub-layer: post_attention_layernorm, then MLP, then the second residual addition

  hidden_states = self.post_attention_layernorm(hidden_states)
  hidden_states = self.mlp(hidden_states)

• ✓ q_norm and k_norm are RMSNorms over head_dim (one head vector at a time), not over hidden_size

  self.q_norm = Qwen3RMSNorm(self.head_dim, eps=config.rms_norm_eps)  # unlike olmo, only on the head dim!
  self.k_norm = Qwen3RMSNorm(self.head_dim, eps=config.rms_norm_eps)  # thus post q_norm does not need reshape

• ✓ q_norm / k_norm are applied after the projection and reshape to heads, and before RoPE

  query_states = self.q_norm(self.q_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
  key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
  value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
  query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)

• ✓ V is not normalized and not rotated

  value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
  key_states, value_states = past_key_values.update(key_states, value_states, self.layer_idx)

• ✓ RoPE rotates Q and K only (applied inside attention, not added to the residual stream)

  query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)

• ✓ RoPE uses the rotate-half pairing (dimension i with i + head_dim/2), not interleaved pairs

  x1 = x[..., : x.shape[-1] // 2]
  x2 = x[..., x.shape[-1] // 2 :]
  return torch.cat((-x2, x1), dim=-1)

• ✓ Attention scores are scaled by head_dim ** -0.5

  self.scaling = self.head_dim**-0.5

• ✓ Grouped-query attention: K/V heads are shared by num_attention_heads // num_key_value_heads query heads

  self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads

• ✓ Attention is causal

  self.is_causal = True

• ✓ MLP is SwiGLU: down_proj(act(gate_proj(x)) * up_proj(x))

  down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
  return down_proj

• ✓ RMSNorm computes in float32: x * rsqrt(mean(x^2) + eps), then multiplies by a learned gain (no bias, no mean subtraction)

  hidden_states = hidden_states.to(torch.float32)
  variance = hidden_states.pow(2).mean(-1, keepdim=True)
  hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
  return self.weight * hidden_states.to(input_dtype)

• ✓ A final RMSNorm (model.norm) follows the last decoder layer

  hidden_states = self.norm(hidden_states)