Where the models disagree
Real held-out decisions, selected by fixed rules, not by how well any model does. The goal is not to show that one model wins but to see what signal each one carries: where Stockfish, the human-move model and the statistical baseline agree, and where they pull apart. Qwen3-8B-Base has not been run on the held-out test split yet, so its column here reads “pending” and the two Qwen-specific slots are reserved (a local run on seven demo positions is shown on the home page).
- Human played: hxg3
- The human move, Stockfish's first choice and Maia-3's top move are the same move.
- Maia-3's top move is Stockfish's first line.
- No disagreement among the available signals.
What it means. When the engine's best move is also the move humans of this strength overwhelmingly play, engine agreement carries little information: matching Stockfish here is expected. The Qwen comparison for this position will be added when real Qwen3-8B-Base scores exist.
Sentences are generated from the stored numbers by fixed rules (no language model writes them).
| Move | Engine rank | Engine eval | Statistical baseline | Qwen3-8B | Maia-3 79M |
|---|---|---|---|---|---|
| hxg3played | #1 | +0.20 | P(engine #1) 47.3% | pending | 95.0%#1 |
| fxg3 | #2 | +0.02 | top-3 line; P(top-3) 74.8% | pending | 4.9%#2 |
| h3 | >5 | — | no per-move value | pending | 0.1%#3 |
| O-O-O | >5 | — | no per-move value | pending | <0.1%#4 |
| h4 | >5 | — | no per-move value | pending | <0.1%#5 |
| e5 | >5 | — | no per-move value | pending | <0.1%#6 |
| Qd1 | #5 | −9.31 | no per-move value | pending | <0.1%#12 |
| d5 | #3 | −8.94 | top-3 line; P(top-3) 74.8% | pending | 0.0%#30 |
| Ne5 | #4 | −9.19 | no per-move value | pending | 0.0%#26 |
Why this example was selectedHide why this example was selected
- Position id
- li_ItVPTIJx:20
- Rule
- played == Stockfish #1 == Maia-3 #1; >= 25 legal moves; |eval| <= 150 cp; Maia-3 probability of the played move in [0.5, 0.95] (not a forced reply); highest such probability
- Source
- test split, original move
- Maia-3 entropy
- 0.207 nats
- Reconstruction difficulty
- 0.547
How to read these examples
- The statistical baseline is an independent reconstruction from publicly documented concepts, not Professor Regan's production system or calibration.
- Controlled engine substitutions are not identical to real-world cheating: substitutes are Stockfish lines inserted into otherwise human decisions and evaluated in the original positions (games are not replayed).
- Maia-3 models human move preference, not cheating. A low Maia probability means a move is unusual for that rating, nothing more.
- Qwen3-8B-Base is a general language model, not a chess-specialized detector. It provides a move-preference signal that can be evaluated for predictive value; it does not detect cheating.
- Tokenization and the way moves are written (UCI strings, ordering, prompt wording) can change LLM likelihoods; the prompt is frozen and hashed but other formulations were not explored.
- Results depend on the dataset (one month of online Lichess rapid/classical games, presumed human), the engine settings (Stockfish 19, depth 12, MultiPV 5) and the substitution protocol; bootstrap intervals condition on the frozen models and apply no multiplicity correction.
- Classification performance or correlation is not proof of misconduct and must not be used alone for adjudication.
This is an independent reconstruction based on publicly documented concepts and is not Professor Regan's production analyzer.