Research engineering · October 21, 2026
Can you reproduce the result—and trust the restart?
Two engineering boundaries behind this project: independently recalculating reported metrics from saved outputs, and resuming a long transformer extraction without binding stale vectors to new documents.
Recalculate the saved evidence
The minimal CPU verifier checks frozen input hashes and recomputes top-1/top-3 accuracy, mean negative log-likelihood and mean rank for two Maia models on 12,863 paired positions. Eight model intervals and four paired intervals use 2,000 shared source-game resamples. They match the stored references within the predeclared absolute tolerance of 1e-10.
A separate optional command requires the exact frozen, locally supplied split/variant/prediction snapshots. It checks recorded player/game separation and independently reproduces the hybrid ROC-AUC points and seven paired intervals within 1e-9. Those private input files are not bundled with this website or available from a fresh clone.
| Saved score | Recomputed ROC-AUC |
|---|---|
| Top-1 statistical z-score | 0.784662 |
| Statistical features | 0.819662 |
| Stockfish features | 0.826225 |
| Stockfish + Maia | 0.873027 |
| Sequence extension | 0.879741 |
Reports retain exact values, intervals, input/verifier/contract hashes and numerical runtime. These are snapshots of executed local checks, not a live health monitor or a claim that CI ran remotely.
Make interrupted extraction safe to resume
The old extractor skipped a chunk if its filename existed. A deterministic CLI fixture changed a source document and demonstrated stale vectors under rewritten metadata. A second fixture exposed mixing a new split with an existing run under a changed numerical runtime. These are reproduced code defects; they do not establish that published experiments encountered them.
- Bind the shared model revision, configuration, numerical runtime, execution device and code identity before adding any split.
- Bind each split's ordered text, tokenization, metadata and chunk layout before publishing metadata.
- Reuse a chunk only after its completion receipt, hash, exact keys, dimensions, precision and valid values agree.
- Publish data atomically, then its completion receipt. Recompute unfinished work; reject corrupt completed work.
- Serialize writers with an OS-held lock and validate final split/run identities before downstream reading.
Failure/recovery tests cover changed content/order/settings, interrupted writes, process-death lock release, corrupted artifacts and clean-versus-resumed parity. Real cached-model regressions separately check extraction, streaming, hooks and probing. Historical artifacts stay intact; old unverified chunks require a new output directory.
What this evidence establishes
It establishes saved-output arithmetic, recorded split-identifier consistency, and tested local extraction integrity. It does not regenerate the original model predictions, establish true human identities, prove real-world cheating detection, complete the pending held-out Qwen evaluation, or measure a production speedup. The Regan-inspired reconstruction remains independent of Professor Regan's production system.
Checkpoint coordination is for a single host with normal filesystem locking and atomic replacement. Hashes detect accidental drift; they do not defend against an authorized operator rewriting both data and receipts. Final consolidation still loads arrays in memory.