LatentLens journey · about 10 minutes
How does this piece of text become the next token?
plain language and concrete examples
Example sentence · selected token
Scene 1 of 8
Text becomes tokens
What does the model actually read?
Recorded Qwen measurement· real Qwen tokenizer
The researcher opened the notebook
34 characters · 5 words · 5 tokens
These splits were produced by the real Qwen3 tokenizer and saved. Typing new text here is not supported because the tokenizer does not run in this page; run build_tokenizer_examples.py --text "…" to add an example.
Keys: [ and ] change scenes
Progress is stored only in this browser and records which scenes you visited and which checks you answered. It is not a measure of mastery.