How MoveLens measures the gap

MoveLens in one minute

What the home page computes, and where every number on it comes from. Downloads are listed under Research artifacts.

Statistical model

Logistic-regression difficulty model on Stockfish position features and rating (training split) → rating-band × difficulty-decile table of engine-match rates → κ-scaled z-score over a game (κ from the validation split). Not a k-nearest-neighbor model. Code: services/analyzer/models/projection.py. An independent reconstruction from public descriptions, not Professor Regan's analyzer.

Engine reference

Stockfish 19, fixed depth 12, MultiPV 5, one thread (bit-reproducible).

Maia-3

UofTCSSLab/Maia3-79M @ a107d6ceb7b2: a Chessformer-based human-move transformer (not a language model), conditioned on both ratings and up to seven previous positions; softmax over legal moves.

Qwen3-8B-Base

Qwen/Qwen3-8B-Base @ 49e3418fbbbc, base checkpoint. Frozen prompt (FEN, ratings, side to move, Stockfish top-5 evaluations, all legal moves; SHA-256 7f4373cf3414…). Each legal move's continuation “ <uci>\n” is scored by its summed token log-probability, then softmax over legal moves. Nothing is generated.

Qwen runs shown

Local, demo scope only: the 7 MoveLens positions, bf16, laptop CPU with CPU-RAM + disk offload (accelerate device_map='auto'; modules per device {'cpu': 16, 'disk': 24}; no layers on the 4 GB RTX 3050 Ti); cached vs full-sequence scoring differ by at most 0.083 nats. A second run used the frozen position-only prompt (no engine lines). The held-out Qwen evaluation has not been run.

Data

2,230 Lichess rapid/classical games (August 2026), one game per player, split 60/20/20 by connected components of players. Held-out test split: 321 source games, 605 player-games, 12,863 eligible original decisions; 9,075 controlled variants (decision-level substitutions evaluated in the original positions).

Reading rules

Statistical: probability the model gave to what happened (engine #1 played or not); “unusual” below 25%, “somewhat unexpected” below 50%, otherwise “expected” (presentation thresholds, not a test). Qwen: “high preference” if the played move is in its top 3. Maia: “high human plausibility” if it is Maia's #1 or has ≥ 30%, “moderate” if in the top 3.

Example selection

Rules fixed in experiments/20_export_curated_examples.py and 22_export_movelens_demo.py, using Stockfish / Maia / statistical outputs; the “Qwen and Maia disagree” example was chosen after the Qwen run by the stated rule (largest Jensen–Shannon divergence among the remaining scored positions).

Whole game & checks

experiments/24_export_regan_checks.py: z(top-1) recomputed from per-move projections and asserted equal to the stored values; calibration, clean-game z, screening and Maia-by-band read from held-out artifacts with no refitting.

Model comparison

The central question is whether a general-purpose LLM can solve the same underlying prediction problem as a statistical human-move model: given a position, the player's rating and the legal moves, which move will this human choose? Three approaches answer it for the same held-out decisions. The Regan-inspired reconstruction and Maia-3 79M have been evaluated; the held-out Qwen3-8B-Base run is the next experiment (method built and tested; seven MoveLens demo positions were scored locally with the 8B model, which is not an evaluation).

• Regan-inspired public reconstruction (R). The existing projection model: position difficulty from Stockfish features and Elo, an Elo-band × difficulty-decile table fitted on training decisions, κ from validation. Per decision it outputs projected probabilities of matching the engine's first line and top three lines; per game, z(top-1). It does not output a probability for every legal move, so move-level metrics are reported as “not defined” and R is compared on the engine-first-move and engine-top-3 events, which every approach predicts.
• Qwen3-8B-Base (Q). A base (non-chat) LLM used as a scorer, never as a judge. For each legal move the frozen prompt is followed by the continuation “ <uci>\n”; the summed log-probability of every token of that continuation is the move's score, and a softmax over the legal moves gives the distribution. No text is generated, no confidence is requested, no few-shot examples are used.
• Maia-3 79M (M). The released 79M Chessformer checkpoint, a human-move transformer (not a language model), with the same input construction as the existing 5M pipeline. The 5M checkpoint used inside hybrid systems C/D is reported as a secondary row and never relabelled.

Qwen checkpoint

—

Qwen status

—

Prompt SHA-256 (Regan-aligned)

—

Maia-3 79M

—

Analysis plan

three-model-plan.v1, SHA-256 58637622b3a88123… (frozen before test evaluation)

Task 1 data

12,863 original test decisions, 321 source games

Bootstrap

paired source-game cluster bootstrap, 2,000 resamples, no refitting, no multiplicity correction

Task 1 uses only original (presumed-human) decisions of the test split: top-1 / top-3 accuracy, mean negative log-likelihood, multi-class Brier score (Σ over legal moves of (pᵢ − yᵢ)²) and mean rank (1 + number of moves with strictly higher probability). Task 2 reuses the existing controlled benchmark unchanged; the pre-registered game-level score is the mean surprisal −ln P(move) of the evaluated moves for Q and M, and z(top-1) for R. Low-FPR thresholds come from validation original player-games, and the realized test FPR is always shown next to recall.

Information policy. The prompt contains only what is known before the decision: both ratings, side to move, FEN, the legal moves in lexicographic order, and the Stockfish MultiPV values that the statistical model also uses. The played move is never marked; when it lies outside the five engine lines its own engine value (which is derived from the target) is not shown. Prompts are built from position fields only, and automated tests check that changing the played move or any assistance field leaves the prompt byte-identical.

Numerics. bfloat16 weights, float32 log-softmax, candidates scored through one shared prompt key/value cache (verified against independent full-sequence scoring: 2.5×10⁻⁵ nats in float32). In bfloat16 the two paths differ by rounding (up to 0.16 nats on the small pipeline-check model), so the candidate batch composition is fixed for reproducibility.

Overview of the hybrid pipeline

MoveLens looks at every decision from three complementary views. Stockfish says what is objectively strong; Maia-3 gives a model-estimated human plausibility — the predicted probability of the move for a player of this rating; a Regan-inspired statistical projection (our empirical approximation, not Regan's analyzer) says what we expect overall from this level in these positions. The views share inputs, so they are not independent. Logistic/ridge baselines, XGBoost and SHAP fuse them into a score and an explanation.

Classifier outputs are reported as similarity to controlled synthetic substitution. Regression outputs are a synthetic-equivalent assistance intensity: the fraction of eligible decisions that would have to be substituted by engine moves to produce a comparable pattern. Neither is a probability that a player cheated.

Terminology: original (in figures sometimes “clean”) means the game's own moves with no assistance injected by us. These Lichess games are presumed human; that they contain no real assistance is not established.

The defensible contribution is added predictive signal from learned human-move features on our documented controlled benchmark. We make no claim to outperform Professor Regan's analyzer, which was not run or compared here.

Exact pipeline

1. Select genuine human games: Lichess open database, standard rated, August 2026 (first ~420 MB of the dump); rated rapid/classical, normal or time-forfeit terminations, both ratings 1000–2699, ≥ 50 plies, one game per player, stratified by Elo band.
2. Split original games into train / validation / test by player-connected components (before any variant exists).
3. Stockfish MultiPV analysis of every position from move 8 on; Maia-3 policy for every position.
4. Build per-decision candidate tables (every MultiPV line plus the played move) and apply the eligibility rule.
5. Fit the difficulty model and projection table on original training decisions; estimate κ on original validation games.
6. Generate controlled decision-level substitution variants (3 strategies × 5 levels, plus the original) for every player-game.
7. Compute nested feature groups A ⊂ B ⊂ C ⊂ D for every variant.
8. Train classifiers and regressors on train, early-stop on validation, evaluate once on test; compute SHAP.
9. Export the JSON shown on this site; every number on the site is read from that export.
10. Post hoc, on the saved test predictions only: paired cluster bootstrap of AUC differences; write the provenance record.

Relationship to Regan's work

The left column lists only concepts found in Regan's public materials, each with its source and page. Source: Professor Kenneth W. Regan's public materials: the 8 April 2026 talk “Chess Research and Public Interests”, the public IR analyzer user guide, and public CSE 702 seminar notes (full references under Methodology › Sources). The middle column is what MoveLens actually implements in their place; the right column is new. Where the sources differ (the 2026 talk names s, c, h; the user guide names s, c and an equal-value parameter ev), both are stated.

Regan-inspired baseline

This is not Kenneth W. Regan's model. His analyzer scales Multi-PV engine move utilities across depths and derives a probability for every legal move from Elo-trained parameters s, c and h [Regan, Apr 2026 talk, pp. 10–12]; his reference calibration is not used here. We use a simpler, fully documented empirical projection with the same actual-versus-projected statistical shape:

1. Difficulty model. A logistic regression of P(top-1 match) on Stockfish-only position features and the player's Elo, fitted on original (presumed-human) decisions of the training split only. Position difficulty is one minus the predicted top-1 probability.
2. Projection table. Cells of Elo band × difficulty decile (40 cells), each holding the mean and variance of top-1 match, EV-style equal-value match, top-3 match and capped move loss among training human decisions (cells with fewer than 30 decisions use the pooled difficulty decile).
3. Projected vs observed. For a player-game with eligible decisions i = 1…n, projected top-1 = Σ p_i with variance Σ p_i(1 − p_i) under independence, and z = (observed − projected) / √(κ · variance). The same construction is used for equal-value match (value equal to the best line at our depth — an analogue of Regan's EV-match), top-3 match, unscaled capped move loss (positive z = less loss than projected; this is not ASD, which scales differences) and Maia-3 surprisal (expectation = Maia entropy). Top-1 match restricted to the hardest third of positions is our difficulty-conditioned statistic; it is not EWN, which is an optional position-weighting scheme in Regan's analyzer (unit weighting is used in most analyses) [IR user guide, pp. 1, 11]. Accounting for position difficulty is part of Regan's model and is not our contribution [IR user guide, p. 10].

κ is a variance-inflation factor estimated on validation original games so that original-game z has unit variance, because decisions within one game are not independent. Exported values: κ_t1 = 1.289, κ_t3 = 1.324, κ_loss = 1.864, κ_{t1_hard} = 1.082, κ_maia = 1.503, κ_ev = 1.463. Hard and ordinary positions are the top and bottom terciles of training difficulty (thresholds 0.732 and 0.577).

Stockfish

Each analyzed position is searched by Stockfish 19 at fixed depth 12 with MultiPV 5, one thread and a fresh hash per game, so re-analysis is reproducible. Scores are in centipawns from the side to move, mate scores are mapped to a large constant, and evaluations are clipped to ±10.00 pawns before losses are computed. Plies beyond 160 are not analyzed.

If the played move is outside the MultiPV list, its evaluation is taken as minus the best evaluation of the resulting position (same depth), capped to be no better than the last listed line, and its rank is censored as 6+ (shown as “6+”).

Maia-3

Maia-3 is a family of Chessformer-based transformer models for rating-conditioned human move prediction — an encoder-only transformer over board squares, not a language model [Chessformer (ICLR 2026); CSSLab/maia3]. The model used here is Maia3 5M (checkpoint UofTCSSLab/Maia3-5M, revision b6559de), whose model card states it was trained on Lichess human games from January 2023 to July 2025 [Maia3-5M model card]; our games are from August 2026.

It is run through the official CSSLab inference package (commit 1e13597), reproducing the input construction of the official UCI engine in history mode: the current position plus up to seven previous positions, conditioned on the side-to-move Elo and the opponent Elo. Instead of the top-k moves exposed by the UCI interface, the policy logits are read directly, giving a predicted probability for every legal move. The predicted probability, Maia rank and surprisal (−ln p, in nats) of the chosen move are exact model outputs. They are model estimates — “Maia assigns a predicted probability of 0.3%” — not observed frequencies of players.

Eligible decisions

A player decision (one ply by that player) is eligible if and only if all of the following hold:

1. it is not in the opening: the first 8 moves per side (16 plies) are skipped as book-like;
2. the side to move has at least two legal moves (the move is not forced);
3. the game is not already decided: |Stockfish evaluation before the move| ≤ 3.00 pawns;
4. Stockfish returned at least two principal variations.

A quick depth-8 single-line search is run first; positions beyond ±5.00 pawns are treated as decided, only the quick line is stored and the decision is ineligible. A move is near-optimal if it is within 25 cp of the best line.

Controlled decision-level substitutions

Alternative moves are evaluated in the original positions. Later positions remain those of the original game; these are not replayed assisted games.

The unit is a player-game: one player's n eligible decisions in one game. For a nominal level α in 0%, 5%, 10%, 20%, 40%, 70%, m = max(1, round(α·n)) decisions are substituted (for α > 0): at each, a different, engine-chosen move is analyzed in place of the original game move. A decision where the original move already is the engine's choice cannot be substituted. If fewer than m decisions are substitutable, all are substituted. The ground truth is the intervention label (substituted variant or original) and the realized substituted fraction α = (number substituted) / n, which is also the regression target.

Random

Alterable = decisions where the human move is not Stockfish's #1; choose m uniformly at random and substitute Stockfish's #1.

Selective

Same alterable set and substitute as Random, but choose the m decisions with the highest position difficulty (1 − projected top-1 probability for this Elo).

Subtle

Substitute a move drawn uniformly from Stockfish's top-3 lines within 30 cp of the best line, excluding the human move; alterable decisions are those with such an option; choose m uniformly at random.

Selections are nested across levels: the 10% set contains the 5% set, and so on. Every move-level feature of a variant is an exact function of the original Stockfish and Maia-3 analyses. The synthetic-equivalent assistance intensity estimated from these variants is a resemblance measure on this benchmark; it is distinct from, and not convertible into, a probability of cheating.

Features and models

Four nested feature groups; each model sees everything the previous one saw.

Each learned model has a classifier (original vs substituted) and a regressor (realized α). Hyper-parameters are fixed in advance; early stopping uses the validation split only. Game-level SHAP values explain both. A separate decision-level XGBoost model scores individual decisions for resemblance to a substitution and is used only to rank and explain moves; its per-move SHAP values are not a decomposition of the game-level classifier.

Hyper-parameters

Fixed before evaluation; read from the training scripts at export time.

Train / validation / test isolation

Original games are split before any synthetic variant exists. Games are grouped into connected components of the graph “two games are linked if they share a player name”, and whole components are assigned to a split (train 60%, validation 20%, test 20%, seed 20261003). No game and no player appears in more than one split, and variants inherit the split of their source game. Automated leakage tests check these invariants.

Test units are nested and must not be read as independent samples: 448 games were assigned to test, of which 321 source games contribute 605 original player-games (players with ≥ 10 eligible decisions), which yield 9,075 injected variants — 9,680 test rows including the originals. All intervals resample whole source games.

Evaluation and uncertainty

• Classes are balanced by sample weights in training and in ROC-AUC, PR-AUC and Brier on the test split.
• Operating thresholds for a target false-positive rate (1%, 5%) are chosen on validation original scores; recall (TPR) and the realized FPR are then measured on test, and the realized FPR differs by system: A 32.1% at 1.0% · B 37.1% at 1.7% · C 40.9% at 1.0% · D 40.4% at 0.7% (1% target). Recall at different realized FPRs does not rank systems.
• Each system's ROC-AUC interval is a 95% cluster bootstrap (200 resamples of test source games).
• Incremental differences between nested systems are tested with a paired cluster bootstrap (2,000 resamples of 321 source games, seed 20261012): each resample of source games is applied to both systems' saved held-out predictions, and the 2.5/97.5 percentiles of the AUC difference form the interval (experiments/09_paired_bootstrap.py). This is post hoc, without refitting or tuning, without multiplicity correction, and it does not capture variation from refitting the models.
• Overlapping individual intervals do not show that there is no gain, and non-overlapping individual intervals are a visual comparison, not a paired difference test. The site uses the paired interval for statistical statements and reports numerical changes separately.
• Per-cell results compare all original test player-games with the substituted variants of one strategy and level.
• The Regan-inspired z(top-1) (A0) uses no labels at all.
• Baseline z-statistics are described on original held-out games by mean, SD and tail shares. Mean ≈ 0 and SD ≈ 1 are largely implied by how κ is estimated and do not validate normal tails: z(top-1, hard positions) exceeds 3 for 0.99% of original test player-games versus 0.14% under N(0,1).
• Pseudo-events (expert detail only) pool k games by k different players of one Elo band. This exploratory analysis does not validate longitudinal detection for a single player.

AIF (Analysis Interchange Format)

In plain terms: AIF stores the chess game together with detailed engine analysis for each position.

Technically: an AIF file is a PGN-compatible game with extra tags (engine ID and settings, threads, hash, MultiPV, depth range), followed by one block per move with the FEN, legal moves, engine evaluations by depth, MultiPV candidates and the principal variation. It is the input format of Regan's IR analyzer (K. W. Regan & T. Biswas, AIF v0.5, described in “A Computer Chess Analysis Interchange Format”, Gödel’s Lost Letter and P=NP, 20 Jan 2015.). MoveLens stores its analysis as JSON/Parquet and offers an experimental AIF-compatible export; compatibility with Regan's current analyzer has not been tested.

Limitations

• Decision-level substitution does not re-simulate the game: an engine move is judged in the original position, and the opponent's later moves are those of the real game.
• The same engine and depth are used for analysis and for constructing assistance. This inflates top-1 matching for Random and Selective (the substitute is by definition the analysis engine's first choice); a real assistant may use a different engine, depth or time.
• Games are online rapid Lichess games, not over-the-board play; behaviour and time pressure differ.
• Per-game samples are small (mean 21.3 eligible decisions per player-game), so single-game scores are noisy. Every player contributes one game, so nothing here validates detection across a single player's games.
• Original games are presumed human, not verified clean; undetected real assistance would make them look more engine-like and reduce measured separation.
• Maia-3 was trained on Lichess games (January 2023 – July 2025, per its model card) — the same platform as our August 2026 test games, but an earlier period; behaviour elsewhere, e.g. over the board, may differ.
• The baseline's upper tail is heavier than normal for the hard-position statistic, so fixed z thresholds would over-flag original games; tail estimates rest on a few games.
• Our statistical baseline is an empirical approximation. Nothing here compares against, or claims to outperform, Regan's actual analyzer.
• Scores measure similarity to these controlled synthetic patterns only. They are not evidence that a real player cheated and must not be used for adjudication.

Reproducibility and provenance

Everything below is read from provenance.json (experiments/10_provenance.py), which records only what can be checked on disk. Fields that could not be established are marked as such rather than filled in. Reproduction has not been re-run end to end for this record.

Consistency checks (recomputed when the record is written)

• Passed — Saved test predictions reproduce exported ROC-AUC: max |difference| over 5 systems = 0.0e+00
• Passed — Current variants.parquet equals the evaluated test feature table: 48 columns × 9680 test rows compared exactly
• Passed — Raw dump prefix has the documented size: 419430400 bytes on disk; range 0–419430399 = 419430400 bytes
• Passed — Lockfile versions match versions recorded in the export: xgboost, scikit-learn, shap, python-chess, torch, maia3 commit

curl -r 0-419430399 -o data/raw/lichess_2026-08_head.pgn.zst https://database.lichess.org/standard/lichess_db_standard_rated_2026-08.pgn.zst
python experiments/01_select_games.py --per-band 600 --max-bytes 4000000000
python experiments/02_stockfish.py --workers 16
python experiments/03_maia.py
python experiments/04_build_dataset.py
python experiments/05_train_eval.py
python experiments/05b_decision_model.py
python experiments/06_export_web.py
python experiments/07_results_md.py
python experiments/08_figures.py
python experiments/09_paired_bootstrap.py
python experiments/10_provenance.py
pytest tests

• No end-to-end re-run of the pipeline was performed when this record was written; it documents the existing artifacts.
• data/processed/* were rebuilt after training (EV-match display addition); check 2 confirms the test features are unchanged.
• Training used a CUDA torch wheel for Maia-3; the live analyzer image uses the CPU wheel of the same version.

Research artifacts

Every displayed result maps to one of these machine-readable files, written by the listed script; the JSON files carry their own generator, timestamp and plan hash.

• MoveLens positions

  movelens_demo.json
  The three curated positions on the home page: Stockfish lines, the statistical model's features and outputs, Maia-3 79M and local Qwen3-8B-Base results, with the selection rules. Script: experiments/22_export_movelens_demo.py.
• Whole game & checks

  regan_checks.json
  Projected vs actual over one game (z recomputed and asserted), held-out calibration, clean-game z distribution, screening power, Maia by band, Qwen engine-echo ablation. Script: experiments/24_export_regan_checks.py.
• Raw local Qwen scores

  qwen_local_scores.json
  Summed token log-probabilities for every legal move of the 7 demo positions, with and without engine lines, plus hardware and consistency-check records. Not an evaluation. Script: experiments/23_qwen_local_demo.py → 24.
• Task 1: move prediction

  three_model_move_prediction.json
  Top-k accuracy, NLL, Brier, rank, engine-event log-loss on 12,863 original test decisions, with source-game bootstrap CIs. Script: experiments/14_three_model_eval.py.
• Task 2: controlled substitutions

  three_model_assistance_results.json
  ROC-AUC / PR-AUC / low-FPR recall of the reconstruction and Maia-3 on 9,680 test rows (Qwen entries pending). Script: experiments/14_three_model_eval.py.
• Paired bootstrap (three-model)

  three_model_bootstrap.json
  Paired source-game bootstrap differences (2,000 resamples) for every model pair; Qwen comparisons pending. Script: experiments/14_three_model_eval.py.
• Paired bootstrap (A/B/C/D)

  paired_bootstrap.json
  Paired source-game bootstrap of AUC differences between nested feature sets on saved held-out predictions (no refit). Script: experiments/09_paired_bootstrap.py.
• Curated examples

  curated_examples.json
  Six rule-selected real test decisions (+ two Qwen slots pending) with Stockfish, Maia-3 and projection outputs. Script: experiments/20_export_curated_examples.py.
• Prepared player-game

  prepared_three_model_demo.json
  36 decisions of one test player-game with every legal move's model outputs and the controlled substitutions. Script: experiments/16_export_three_model_demo.py.
• Provenance record

  three_model_provenance.json
  Model revisions, prompt hash, engine binary hash, analysis-plan hash, split counts and consistency checks. Script: experiments/18_three_model_provenance.py.
• Hybrid experiment metrics

  metrics.json
  A/B/C/D ROC-AUC, low-FPR recall, calibration, SHAP and configuration of the validated hybrid experiment. Script: experiments/05_train_eval.py + 06_export_web.py.
• Results summary (CSV)

  three_model_results_summary.csv
  Task 1, Task 2 and paired-difference tables as one flat CSV, generated from the JSON files above. Script: experiments/21_export_results_csv.py.

Limitations of the model comparison

Sources

Every external statement on this site cites one of these. Each document was retrieved and read; page numbers refer to the PDF.

1. K. W. Regan, “Chess Research and Public Interests”, New Faculty Academy, University at Buffalo, 8 April 2026 (slides; page numbers refer to the no-pause PDF). https://cse.buffalo.edu/~regan/Talks/ChessResearchApr2026np.pdf
2. K. W. Regan, “User Guide for IR Chess Statistical Analyzer” (public PDF, CSE 702 seminar materials; file dated 13 Feb 2024). https://cse.buffalo.edu/~regan/ChessSeminar/UserGuideIRlite.pdf
3. K. W. Regan, “CSE702 Week 6A: Cross-Validation and Things Out of Whack, and Some Experiments” (public seminar notes; file dated 7 Mar 2024). https://cse.buffalo.edu/~regan/ChessSeminar/CSE702Week6A.pdf
4. K. W. Regan & T. Biswas, AIF v0.5, described in “A Computer Chess Analysis Interchange Format”, Gödel’s Lost Letter and P=NP, 20 Jan 2015. https://rjlipton.com/2015/01/20/a-computer-chess-analysis-interchange-format/
5. D. Monroe, G. Eilender, P. Chalmers, Z. Tang & A. Anderson, “Chessformer: A Unified Architecture for Chess Modeling”, ICLR 2026; arXiv:2605.19091. https://arxiv.org/abs/2605.19091
6. CSSLab, maia3 — official inference code and documentation (GitHub). Pinned commit 1e13597 is used here. https://github.com/CSSLab/maia3
7. UofTCSSLab/Maia3-5M model card (Hugging Face): encoder-only transformer; trained on Lichess human games, January 2023 – July 2025. https://huggingface.co/UofTCSSLab/Maia3-5M

Not sourced, and therefore not claimed: whether EWN or any particular setting is used in a given official analysis, and any performance comparison with Regan's analyzer.

Independent research note

MoveLens was independently developed from publicly available research, documentation, open-source chess engines and released models. It does not use Professor Regan's private reference datasets, private analysis files, private adjudication records, or proprietary calibrations, and it has not been run against his analyzer. The objective is to investigate whether modern learned human-behavior signals can complement the publicly documented statistical framework.