Phase 2 · LLMs · proposed approach

Proposed experiment: explaining the 14B → ~27B recall plateau

From final recall to layerwise recall trajectories

Proposal · not yet runNo results on this page

Professor-reported observationFrom Professor Regan's email; not reproduced yet.

Older Qwen ~14B · MLK speech

86.31%

recall

Larger Qwen ~27B · MLK speech

86.34%

recall

about 40 more tokens in the larger model's run
some long words were segmented into tokens differently
both models correctly recalled the preceding US place names
one continuation where the older model prefers we will … and the larger model we shall not …

This plan does not assume that scaling has hit a wall. Nearly identical final recall may come from several different underlying causes.

Do the two models achieve similar final recall through the same internal trajectory, or do their hidden representations evolve differently across transformer depth?

Pipeline

1Professor's existing run
2Reproduce exact recall
3Compare tokenization
4Extract layerwise hidden states
5Intermediate diagnostic projection
6Compare true-token trajectories
7Inspect disagreement cases
8Test plateau hypothesis

Read left to right, top to bottom. Each step depends on the one before it.

Five stages

Stage 1

Reproduce the professor's baseline exactly

Reproduce first. Interpret second.

If the original metric cannot be reproduced, any deeper comparison risks answering a different question.

Use exactly his

input text
scripts
preprocessing
model checkpoints
tokenizer for each model
recall definition
numerator and denominator
handling of special and non-word tokens

Target

86.31% (older ~14B) and 86.34% (larger ~27B), as closely as possible

Stage 2

Separate tokenization from model behavior

The larger model's run reportedly had about 40 more tokens, so raw token-level recall percentages are not automatically aligned.

Record for each model

total token count
token boundaries
special and non-word tokens
words split differently
locations where tokenization diverges

Two reporting views

A. Model-native token recall: each model against its own token sequence
B. Aligned text-span analysis: where meaningful, align the underlying text at word, span or character level, so token boundaries can be inspected separately from recall behavior

Caveat

No alignment method is final until Professor Regan confirms the comparison he intends.

How much of the observed difference is caused by tokenization rather than internal recall behavior?

Stage 3

Trace the known next token through transformer depth

This is the central LatentLens experiment. The text supplies a known next token at every position, so its support can be followed through every layer.

At the model output, per position

correct next token
top predicted token
true-token rank
true-token logit
true-token probability
top-token probability
margin
entropy
top-1 correctness

At every depth (intermediate diagnostic projection)

diagnostic rank of the known next token
diagnostic score (logit)
top projected token
margin against the strongest competitor

Intermediate diagnostic projection

hidden state after block k
final normalization (Qwen3: model.norm, to confirm per checkpoint)
LM head
diagnostic vocabulary logits

These are diagnostic projections, not the model's predictions at those layers. Only the last layer is the model's output.

Main graph: true next-token diagnostic rank vs relative transformer depth

Relative depth = layer number ÷ total transformer layers, so models with different numbers of layers share one axis.

Latent-state track

At early, middle, late, final depths, measure the representation itself:

residual-state RMS
L2 norm
cosine similarity between consecutive layer states, if useful
hidden-state trajectory statistics
pre-RMSNorm state
post-RMSNorm state, where feasible

Block RMSNorm: the whole residual vector of a token (before attention, before the MLP, final norm). The pre/post comparison above uses this one.

Q/K RMSNorm: each query or key head separately, inside attention. Measured separately, never mixed with block-norm results.

Representation magnitude is reported as magnitude; it is not treated as an amount of information.

Does the trajectory change materially before versus after RMSNorm?

The Qwen3-8B trace in the Professor demo is an instrumentation demonstration that follows the model's own final top output. This experiment is different: Professor Regan's text supplies a known next token at every position.

When does the correct continuation become strongly supported in each model?

Stage 4

Analyze agreement and disagreement cases

Aggregate recall may hide qualitatively different behavior, so positions are partitioned before trajectories are compared.

A
Both models correct
B
14B correct, larger model wrong
C
Larger model correct, 14B wrong
D
Both wrong, same continuation
E
Both wrong, different continuations

Compare the layerwise trajectories within each category.

Case study: will vs shallPending checkpoints and model access

…from every state and every city,14Bwe will …~27Bwe shall not …

Plot against relative depth, for each model:

rank of “will”
rank of “shall”
score of “will”
score of “shall”
margin between them

If the two tokenizers split this span differently, the relevant text span is aligned instead and the limitation is stated.

At what point do the two candidate continuations begin to diverge?

Stage 5

Test whether the apparent plateau is real

Do not decide on one aggregate percentage. Compare the full passage on several measures, then frame the conclusion conservatively.

Across the full passage

final top-1 recall
mean and median true-token rank
average negative log-likelihood, if appropriate
prediction margin
entropy
stabilization depth
disagreement rate
tokenization-adjusted results where possible

Uncertainty

Paired bootstrap resampling over aligned positions or spans, if appropriate.
No significance test is run until the alignment and independence assumptions have been checked.

Possible outcomes (not predetermined)

ASame final recall, similar internal trajectories

Consistent with: The task may genuinely be near saturation across these model sizes.

BSame final recall, different internal trajectories

Consistent with: The final metric hides meaningful representational differences.

CRecall similarity changes substantially after accounting for tokenization

Consistent with: Part of the apparent plateau was a measurement or alignment issue.

DDifferences concentrate at particular token types or contexts

Consistent with: The phenomenon may be localized rather than a global scaling plateau.

Central comparison

Measurement	Older ~14B	Larger ~27B
Final recall	86.31%	86.34%
Checkpoint	to confirm	to confirm
Token count	to confirm	to confirm (reported: about 40 more)
Tokenizer	to confirm	to confirm
True-token rank trajectory	to measure	to measure
Prediction margin	to measure	to measure
Entropy	to measure	to measure
Stabilization depth	to measure	to measure
Residual RMS	to measure	to measure
Pre / post RMSNorm	to measure	to measure

■ professor-reported■ to confirm with Professor Regan■ to measure

Why this approach?

It preserves Professor Regan's existing metric rather than replacing it.
It separates tokenization effects from model-behavior effects.
It turns the recall problem into a layerwise representation question.
It can explain cases that aggregate accuracy hides.
It produces a reusable pipeline for future model comparisons.
The same instrumentation can later be extended to human-vs-AI representation analysis.

Before running this, I need to confirm

exact 14B checkpoint
exact larger-model checkpoint
tokenizer for each
exact recall definition
treatment of extra and non-word tokens
preferred latent-state capture point
whether relative-depth comparison makes sense to him
token-level only, or an aligned word/span comparison too
whether “will” vs “shall” is the best first case study
which result he would consider most useful

In one sentence

“I would first reproduce your recall experiment exactly, then separate the tokenization effect, and finally trace the known next token through the hidden states of both models to see whether nearly identical final recall is hiding a different internal trajectory.”

Related: Professor demo (one recorded Qwen3-8B trace) · Recall Microscope (the per-layer true-token rank already recorded on Qwen3-8B-Base, Gettysburg Address).