12Recall Microscope

At what depth does a known continuation begin to dominate?

Teacher-forced next-token recall on a famous public-domain text and on an original paragraph the model cannot have seen. For every position we record the true token's rank, probability and margin at the output, and at every depth through an intermediate diagnostic projection.

Mistake, or another version?

Recorded Qwen measurement· teacher-forced recall, bfloat16; analysis of recorded predictions

9 of Qwen's 73 misses on the Gettysburg Address are the wording of another Lincoln manuscript.

Lincoln wrote out five copies of the speech, and they differ. The text scored here is the Bliss copy. Where Qwen's top prediction was "wrong", it sometimes produced exactly what the Nicolay, Hay, Everett, Bancroft copies say at that point.

top-1 recall against the Bliss copy

76.9%

if any of the five manuscripts' wording counts

79.7%

scored positions

316

243 hits, 73 misses

What the 73 misses are

punctuation or formatting: 17
another Lincoln manuscript's wording: 9
other: 47

…en years ago our fathers brought forth
Bliss copy
␣on
Qwen's top guess
␣upon
- Nicolay copy: …our fathers brought forth, upon this continent, a new nat…
- Hay copy: …our fathers brought forth, upon this continent, a new nat…
- Everett copy: …our fathers brought forth, upon this continent, a new nat…
… continent, a new nation, conceived in
Bliss copy
␣Liberty
Qwen's top guess
␣liberty
- Nicolay copy: …new nation, conceived in liberty, and dedicated to the …
…dedicated, can long endure. We are met
Bliss copy
␣on
Qwen's top guess
␣here
- Hay copy: …endure. We are met here on a great battlefield of…
…can long endure. We are met on a great
Bliss copy
␣battle
Qwen's top guess
␣battlefield
- Hay copy: …on a great battlefield of that war. We ha…
…tion of that field, as a final resting
Bliss copy
␣place
Qwen's top guess
-place
- Everett copy: …as a final resting-place for those who here gave …
…who fought here have thus far so nobly
Bliss copy
␣advanced
Qwen's top guess
␣carried
- Hay copy: …thus far, so nobly carried on. It is rather for u…
… devotion to that cause for which they
Bliss copy
␣gave
Qwen's top guess
␣here
- Nicolay copy: …cause for which they here, gave the last full measu…
- Everett copy: …cause for which they here gave the last full measur…
…evotion -- that we here highly resolve
Bliss copy
␣that
Qwen's top guess
␣these
- Nicolay copy: …we here highly resolve these dead shall not have died…
…ad shall not have died in vain -- that
Bliss copy
␣this
Qwen's top guess
␣the
- Nicolay copy: …died in vain; that the nation, shall have a new b…

Why this matters for recall plateaus. A recall score is measured against one reference text. For a famous text that exists in several authentic versions, some "errors" are disagreements between versions, so the measurable ceiling can sit below 100% even for a model that has seen all of them. Before comparing two models' recall (for example around 86%), it is worth checking which transcript was used and whether the misses coincide with known variants.

One passage. Exploratory: it shows which misses coincide with another authentic wording, not why the model produced them.
Transcriptions of the copies come from one secondary source and differ in punctuation.
A prediction that matches another copy may still have other causes (for example, the same phrase in quotations or paraphrases seen in training).

Manuscript transcriptions: Abraham Lincoln Online, 'The Gettysburg Address' (transcriptions of the five manuscript copies), retrieved 2026-10-16 (https://www.abrahamlincolnonline.org/lincoln/speeches/gettysburg.htm).

Loading the recall artifact…

Prepared, not run

Comparing model sizes: same final recall, different trajectories?

Professor Regan observed that a ~27B Qwen model and an older ~14B model reach almost the same recall rate (about 86.3%) on a famous speech. Final recall alone cannot show whether the two models get there the same way. LatentLens can compare them on relative depth (block / number of blocks) once both are run with the same script.

What would be compared

true-token rank by relative depth, and top-token identity by relative depth
depth at which the prediction stabilizes
activation RMS by depth, prediction margin and entropy
tokenizer differences (the same text can have different token counts)
bf16 vs fp32 sensitivity of the recall rate itself

How to run it

python scripts/latentlens/run_recall_analysis.py --model <model A> \
    --loader full --input speech.txt --no-store-text --out A.json
python scripts/latentlens/run_recall_analysis.py --model <model B> \
    --loader full --input speech.txt --no-store-text --out B.json
python scripts/latentlens/compare_recall_models.py A.json B.json

Model ids are left open on purpose: LatentLens does not assume which checkpoints Professor Regan used. Nothing is downloaded automatically. Only Qwen3-8B-Base (36 blocks) has been run so far.