A research prototype · held-out Lichess games

MoveLens

Three ways to understand a chess move.

Statistics, language models, and human chess behavior on the same positions.

I take the same chess position and study it using a statistical model, a general-purpose language model, and a state-of-the-art human chess model.

held-out human decisions: 12,863; held-out human decisions
held-out player-games: 605; held-out player-games
controlled engine-assisted variants: 9,075; controlled engine-assisted variants
models + Stockfish 19 reference: 3 + 1; models + Stockfish 19 reference

Statistical model

How unusual does the move look from historical and engine-derived evidence?

Qwen3-8B

What move does a general-purpose language model prefer?

Maia-3 79M

What move would a human of this rating most likely play?

Same position. Three different views.

MoveLens is an independent research prototype. Its statistical model is reconstructed from publicly documented concepts and is not Professor Kenneth W. Regan's production cheating-detection system.

Position

1. Statistical Model

Can historical and engine-derived evidence tell us whether a move is unusual?

How it works

1. Stockfish 19 evaluates the position's best five moves.
2. The position is described by measurable features: rating, evaluation, gap between the best moves, number of near-optimal moves, number of legal moves.
3. A model fitted on thousands of human decisions from training games estimates how hard the position is, i.e. how often a player of this rating finds the engine's first move.
4. A table of rating band × difficulty gives the expected behavior: chance of the engine's first move, of a top-3 move, expected loss.
5. The played move is compared with that expectation. Over a whole game, observed and expected engine matches give a z-score.

Note. This is an independent reconstruction based on publicly documented ideas and is not Professor Regan's production analyzer. It uses no k-nearest-neighbor search; “similar positions” means the same rating band and difficulty decile of the training data.

Played move

24… h6

Player Elo

1956

Move

24 · Black

Precomputed result

Black arrow: played move. Blue arrow: Stockfish's first choice (♕e6).

1
Position received
FEN, side to move (black), player Elo 1956 vs 1945.
2
Features extracted
Stockfish evaluation +0.09; gap between the best two moves 1 cp; 5 near-optimal moves (within 25 cp); 31 legal moves; evaluation swing 10 cp; middlegame.
3
Statistical evidence evaluated
Difficulty 0.82 (0 = easy, 1 = hard). For this rating and difficulty, players chose Stockfish's first move 17.0% of the time and a top-3 move 42.4%; expected loss 38 cp.
4
Played move compared
h6: not among Stockfish's top 5 lines. No engine match.
5
Final score
Probability the model gave to what actually happened: 83.0%.

Show technical details

Reading rule: probability the statistical model assigned to what happened (engine #1 played or not): unusual below 25%, somewhat unexpected below 50%, otherwise expected (presentation thresholds, not a test)
Implementation: logistic-regression difficulty model + Elo band x difficulty decile projection table (training split), kappa-scaled z-scores (validation split). Not KNN. (services/analyzer/models/projection.py)
Engine: Stockfish 19, depth 12, MultiPV 5, 1 thread
Stockfish lines: #1 ♕e6 +0.09 · #2 ♕f6 +0.08 · #3 g6 +0.07 · #4 h5 +0.02 · #5 ♕c7 0.00
Projected P(top-3 move): 42.4%
Whole game: this player's original game: z(top-1) = +0.20 over all eligible decisions
Position: li_aVMHaGzT:47

2. Qwen3-8B

Can a general-purpose language model express a useful preference over chess moves?

Qwen is not a chess engine. I give it a structured description of the position and score every legal candidate move.

Nothing is generated and Qwen is never asked for a verdict. For each legal move, I measure how likely Qwen finds that move as the next piece of text, then normalize these likelihoods over the legal moves into a ranking.

The model is Qwen3-8B-Base, the base (non-chat) checkpoint, pinned to one revision. Note that the prompt includes Stockfish's top-5 evaluations, the same engine evidence the statistical model uses.

Input

Chess position (FEN)
player and opponent Elo
Stockfish top-5 evaluations
all legal moves

Model

Qwen3-8B-Base

Output

score for every legal move
ranked candidate moves

Played move

24… h6

Player Elo

1956

Move

24 · Black

Precomputed Qwen3-8B result

Black arrow: played move. Red arrow: Qwen's first choice. No live inference runs in this page.

1
Position
Black to move, 31 legal moves.
2
Prompt construction
A fixed text template: both ratings, side to move, the FEN, Stockfish's top-5 evaluations and all legal moves. The played move is never marked.
3
Qwen evaluation
Each legal move is appended as the answer; Qwen's log-probability of every token of that move is summed.
4
Move ranking
Scores normalized over the 31 legal moves; the human move is #3.

Show technical details

Model: Qwen/Qwen3-8B-Base @ 49e3418fbbbc
Prompt: frozen template, SHA-256 7f4373cf34140539…
Candidate text: “ <uci>\n” appended after “Move:”; every token's log-probability summed (no length normalization)
Normalization: softmax over the legal moves
Temperature / sampling: none: nothing is generated
This result: precomputed locally (laptop CPU with CPU-RAM + disk offload (accelerate device_map='auto'; modules per device {'cpu': 16, 'disk': 24}; no layers on the 4 GB RTX 3050 Ti), bf16, candidate batch 4); scope: 7 demo positions only; consistency vs full-sequence scoring 8.3e-2 nats
Held-out evaluation: not yet run (needs a cloud GPU); no accuracy or AUC exists for Qwen
Caveat: the prompt contains Stockfish's evaluations; section 6 shows how Qwen's ranking changes without them
Played move score: log-prob -4.62 over 5 tokens

Show the exact prompt for this position

Chess move prediction. The position below is from a rated online game (Lichess, rapid or classical time control).

Player Elo: 1956
Opponent Elo: 1945
Side to move: Black

Position FEN:
4r1k1/p3qppp/1p6/2p5/2P5/1P5P/P1Q2PP1/3R2K1 b - - 0 24

Engine analysis: Stockfish 19, fixed depth 12, MultiPV 5. engine_eval is the evaluation in pawns from the point of view of the side to move; n/a means the move is not among the engine's top lines.

Legal candidate moves:
a7a5 | engine_eval=n/a
a7a6 | engine_eval=n/a
b6b5 | engine_eval=n/a
e7b7 | engine_eval=n/a
e7c7 | engine_eval=+0.00
e7d6 | engine_eval=n/a
e7d7 | engine_eval=n/a
e7d8 | engine_eval=n/a
e7e1 | engine_eval=n/a
e7e2 | engine_eval=n/a
e7e3 | engine_eval=n/a
e7e4 | engine_eval=n/a
e7e5 | engine_eval=n/a
e7e6 | engine_eval=+0.09
e7f6 | engine_eval=+0.08
e7f8 | engine_eval=n/a
e7g5 | engine_eval=n/a
e7h4 | engine_eval=n/a
e8a8 | engine_eval=n/a
e8b8 | engine_eval=n/a
e8c8 | engine_eval=n/a
e8d8 | engine_eval=n/a
e8f8 | engine_eval=n/a
f7f5 | engine_eval=n/a
f7f6 | engine_eval=n/a
g7g5 | engine_eval=n/a
g7g6 | engine_eval=+0.07
g8f8 | engine_eval=n/a
g8h8 | engine_eval=n/a
h7h5 | engine_eval=+0.02
h7h6 | engine_eval=n/a

Predict the move most likely to be chosen by a human player of approximately this strength.

Move:

3. Maia-3 79M

What would a human of this strength most likely play?

Maia-3 is a chess transformer designed specifically to predict human moves across skill levels.

It is not a language model. Because it is conditioned on the players' ratings, it can say what a 1200 or a 2000 player would typically do in the same position, which is exactly the reference I need for human behavior.

Unlike Stockfish, Maia is not trying to find the strongest possible move. It is trying to model what a human player is likely to do.

Input

Position
player rating
move history (up to 7 previous positions)

Model

Maia-3 79M

Output

human move probability distribution

Played move

24… h6

Player Elo

1956

Move

24 · Black

Precomputed result

Black arrow: played move. Green arrow: Maia's most likely move.

1
Position + rating
Board, up to seven previous positions, player 1956 vs opponent 1945.
2
Maia-3 79M
A Chessformer-based transformer trained on human games across skill levels.
3
Human move distribution
A probability for each of the 31 legal moves.

Why not Stockfish?

Stockfish is optimized for objective playing strength. Maia is optimized for modeling human move behavior. Stockfish is still used here as the objective reference: engine rank, evaluation and the controlled engine substitutions.

Show technical details

Checkpoint: UofTCSSLab/Maia3-79M @ a107d6ceb7b2
Paper: Monroe et al., “Chessformer: A Unified Architecture for Chess Modeling”, ICLR 2026
Output used: softmax of the policy over all legal moves
Entropy: 1.34 nats

4. Same move. Three interpretations.

The same position, the same played move, three different questions.

Played move

24… h6

Player Elo

1956

Move

24 · Black

Statistical model

Played move: h6
Engine match: no
Model expected this: 83.0%

How unusual is this statistically?

expected

Qwen3-8B

Played move rank: #3 of 31
Score: 14.7%

What does the general LLM prefer?

high preference

Maia-3 79M

Probability: 38.6%
Rank: #2 of 31

How plausible is it for a human of this rating?

high human plausibility

Agreement / disagreement

Statistical model: expected
Qwen: high preference
Maia: high human plausibility
Stockfish (reference): the played move is not in its top 5 lines; its first choice is ♕e6.

Here the three systems broadly agree about the played move.

So what am I actually testing?

Does the LLM contain useful information that is not already captured by classical statistical evidence or a specialized human chess model?

Each model provides a different signal. The research question is whether combining those signals gives us additional information.

Important limitations

This is not Professor Regan's production cheating detector.
A strong or engine-like move is not proof of cheating.
Maia predicts human behavior, not misconduct.
Qwen is a general-purpose language model, not a chess cheating detector.
Controlled substitutions are experimental proxies.
Model disagreement is evidence for analysis, not accusation.

Methods View full results View provenance Earlier detailed views

Projected versus actual

5. From one move to a whole game

A single move proves nothing. The statistical method accumulates evidence over a whole game: how many engine matches were expected, and how many happened?

The same held-out game as Example 1: all 36 eligible moves by the Black player (rated 1956). Each square is one move; a filled disc means the move matched Stockfish's first choice.

Moves replaced

engine first choiceother movereplaced by the experiment

Hover or focus a square to see that move.

Expected matches

13.4

Actual matches

z (top-1)

+0.20

Among the 605 presumed-clean held-out games, 3.3% exceed z = 2 (20 games) and 0.17% exceed z = 3 (1 game); a standard normal would give 2.3% and 0.14%.

The original game: 14 engine matches where 13.4 were expected (z = +0.20, well inside the range of clean games).

Random and selective (hardest-move) substitution give identical z here at every level: each replacement adds exactly one engine match, and a count statistic cannot see which moves were replaced. Telling them apart needs position-aware evidence, such as the hard-position statistic or the human-move model.

Show technical details

Formula: z = (actual - projected) / sqrt(kappa * sum p(1-p)); kappa estimated on validation original games
κ (top-1): 1.289
Variance: Σ p(1−p) = 6.61 for this game
Check: z(top-1) recomputed from the per-decision projections equals the stored value for every variant (|diff| < 1e-6)
Game: li_aVMHaGzT (black), held-out test split; substitutions evaluated in the original positions (not replayed)

z(top-1) by share of moves replaced
Replaced	Engine #1 subs: actual / expected / z	Subtle subs: actual / expected / z
none	14 / 13.4 / +0.20	14 / 13.4 / +0.20
5%	16 / 13.4 / +0.89	15 / 13.4 / +0.55
10%	18 / 13.4 / +1.57	15 / 13.4 / +0.55
20%	21 / 13.4 / +2.60	15 / 13.4 / +0.55
40%	28 / 13.4 / +5.00	19 / 13.4 / +1.92
70%	36 / 13.4 / +7.74	18 / 13.4 / +1.57

Checks a statistician would ask for

6. Can these numbers be trusted?

Before comparing models, check that the statistical yardstick is calibrated, know how often it misfires on honest games, and know what each model is really reading.

Are the projections calibrated on games the model never saw?

Hover or focus a mark for its values.

Show table

Calibration by projection bin
Bin	Decisions	Projected	Actual
0.1-0.2	1,215	17.1%	15.6%
0.2-0.3	3,955	24.6%	25.8%
0.3-0.4	3,386	34.9%	34.3%
0.4-0.5	1,789	45.5%	47.1%
0.5-0.6	448	58.7%	59.2%
0.6-0.7	792	61.3%	61.9%
0.8-0.9	1,278	84.0%	85.2%

Calibration by rating band
Rating band	Decisions	Projected	Actual
1000-1399	2,397	33.3%	35.6%
1400-1799	2,942	37.0%	37.0%
1800-2199	4,435	39.7%	39.5%
2200-2699	3,089	43.6%	44.2%

Across 12,863 held-out human decisions the model projected 38.8% engine-first matches; 39.3% happened. In every bin with at least 30 decisions, projected and actual rates differ by at most 1.7 percentage points.

Source: regan_checks.json · calibration (held-out test split, original moves)

How often would an honest player be flagged?

605 held-out gamesstandard normal, same total

Hover or focus a mark for its values.

Show table

Tail fractions
Threshold	Games above	Observed	Standard normal
z > 2	20	3.31%	2.27%
z > 2.5	5	0.83%	0.62%
z > 3	1	0.17%	0.14%

z on clean games has mean +0.02 and SD 1.04. 20 of 605 games (3.3%) exceed z = 2, where a standard normal predicts about 14 (2.3%), so the tail is a little heavier than normal and thresholds need some margin.

Source: regan_checks.json · clean_z (605 presumed-clean held-out player-games)

How much substitution does the statistic need to notice?

Engine first-choice substitutesSubtle substitutes (≤ 30 cp from best)

Hover or focus a mark for its values.

Show table

Share of games flagged
Replaced	Engine #1: z>2	Engine #1: z>3	Subtle: z>2	Subtle: z>3
none	3.3%	0.2%	3.3%	0.2%
5%	8.9%	1.0%	3.3%	0.3%
10%	15.7%	3.5%	4.6%	0.3%
20%	39.7%	14.9%	6.6%	1.0%
40%	87.4%	62.5%	17.5%	2.3%
70%	100.0%	97.5%	42.5%	10.9%

Selective (hardest-move) substitution gives the same values as random here, for the reason given in section 5.

With 20% of moves replaced by engine first choices, 39.7% of games exceed z = 2. Replace them with near-best moves that are not the first choice and only 6.6% do, against 3.3% of untouched games. Counting engine first-choice matches is largely insensitive to subtle help.

Source: regan_checks.json · screening (605 held-out games per cell)

Does the human-move model track playing strength?

1000-139953.4%

1400-179956.3%

1800-219958.1%

2200-269958.8%

Top-1 accuracy on original held-out moves (n = 12,863).

Maia-3 79M predicts the exact human move 53.4% of the time for 1000–1399 players and 58.8% for 2200+; it is the rating conditioning that lets it act as a human-behavior control.

Source: three_model_move_prediction.json · Maia-3 79M by rating band

Is Qwen just echoing the engine lines it is shown?

The main Qwen prompt contains Stockfish's top-5 evaluations. If Qwen's preference is mostly a copy of those numbers, it cannot add information beyond the engine. The same frozen prompt without the engine lines separates the two.

Qwen's top move with and without Stockfish lines in the prompt
Played	Engine #1	Qwen top, with engine lines	Qwen top, position only	Played move's Qwen rank (with / without)
♗f3	O-O	O-O= engine #1	♗c3	#3 / #5
hxg3	hxg3	d5	♘g5	#4 / #16
♖c7	♔f2	g3	e5	#33 / #24
♖xg5	♖xg5	f3	a4	#5 / #19
h6	♕e6	♕e6= engine #1	♕d6	#3 / #11
fxe4	♖ce1	♖c5	fxe4	#16 / #1
♘f6	♘f6	♘f6= engine #1	♘c3	#1 / #13

With the engine lines in the prompt, Qwen's top move was Stockfish's first choice in 3 of 7 demo positions; without them, in 0 of 7. The median rank of the engine's first choice moves from 2 to 16, and of the human move from 4 to 13. On these positions the engine numbers drive much of Qwen's ranking; any added value must be measured against that. Seven positions are a qualitative check, not a test; the held-out ablation is planned.

Source: regan_checks.json · qwen_ablation (7 demo positions, local precomputed Qwen3-8B-Base runs with two frozen prompts)

What is not shown yet

The held-out Qwen evaluation (12,863 positions) and the paired test of whether Qwen adds information beyond the statistical model and Maia.
Sensitivity to engine depth and version (all analysis is Stockfish 19 at depth 12).
Real, adjudicated cheating cases; only controlled substitutions are used.
Over-the-board and classical games; the data are one month of online Lichess rapid/classical games.
An intrinsic performance rating: the reconstruction does not fit per-player skill parameters.
Comparison with Professor Regan's actual analyzer on the same games.