LatentLens · Professor mode

Recall from the inside: tracing tokens through Qwen

A 5 to 7 minute walkthrough. Recorded measurements, professor-reported observations, hypotheses and pending work are labelled separately.

1 of 9 · under 1 min

From outputs to internals

Last time I was analyzing Qwen mainly from its outputs. After our discussion, I wanted to understand what happens inside the model and make those internal states measurable.

  1. Text
  2. Tokenizer
  3. Token IDs
  4. Embeddings
  5. Transformer blocks
  6. Final RMSNorm
  7. LM head
  8. Logits
  9. Next token
  • Model: Qwen/Qwen3-8B-Base, pinned revision 49e3418f, a decoder-only causal model with 8,190,735,360 parameters stored in bfloat16.
  • Byte-level BPE tokenizer with 151,669 entries; the embedding table and LM head have 151,936 rows and are separate (untied) matrices. No BOS token is added.
  • Recorded on a 4 GB laptop GPU by streaming one block at a time; streamed outputs were checked against a full offloaded run.