Phase 15: NNUE self-play/Stage-3 infrastructure and diversity-mechanism research - #225
Merged
Merged
Conversation
Why: Phase A's agentic-eval review found onMake(Board, int move) can't derive the captured piece for captures/promotion-captures — the only undo-info field feature-delta computation actually needs (en passant, promotion, and castling are all fully derivable from the packed move alone, confirmed against Move.java's flag encoding). What: add Board.lastCapturedPiece(), a zero-allocation accessor reading the existing pooled UnmakeInfo record (the field was already computed, just private); extend EvaluatorStrategy.onMake to onMake(Board, int move, int capturedPiece); update all 6 Searcher call sites. Signature-only — nothing consumes the new param yet (ClassicalEvaluator's no-op default is unaffected). Left out: NnueEvaluator/NnueNetwork/FeatureExtractor (PR2); UCI wiring (PR3). Rejected changing Board.makeMove's return type (would allocate in the hottest path in the engine) and moving hook timing earlier (gains nothing, breaks the audited post-makeMove hook convention). Verification: full reactor suite unchanged (engine-core 178/0/2 skipped, engine-tuner 131/0/1 skipped), NodeCountRegressionTest green, Perft 5/5, mirror-symmetry (EvaluatorTest) 46/46 — confirms behavior-neutrality. Phase: 15 — NNUE Phase B, PR1 of 3
…mulator) Why: Phase B's core deliverable — a real NnueEvaluator, isolated and fuzz-tested before any Searcher/UCI wiring, per the plan's PR2 boundary (reviewable independent of search/concurrency concerns). What: core.eval.nnue package — - NnueNetwork: immutable weights + versioned .nnue binary loader (magic/version header, FT weights/biases, output weights/bias, quantization scales, provenance UUID/commit/timestamp). - FeatureExtractor: plain-768 dual-perspective index formula (relativeColor*384 + (type-1)*64 + relativeSquare, black view mirrors via square^56) plus forEachChange(), one absolute-delta computation per move covering all 7 move-type-matrix cases (quiet/capture/en passant/castling/promotion/promotion+capture/ null), applied to both perspectives by the caller. - NnueEvaluator: preallocated dual accumulator stack (Board's unmakePool/unmakeSP idiom — zero allocation in onMake/onUnmake); implements FeatureExtractor.ChangeVisitor as `this` rather than a per-call lambda/anonymous class, for the same zero-allocation reason. evaluate() via CReLU + int32 dot product + output scale. - Incremental-vs-full-rebuild fuzz test (40 games x 60 plies, random legal moves + null moves, forward and unwind) — the load-bearing correctness gate; passed on first run. - Feature-index parity fixture: hand-derived, verified startpos indices (both perspectives, confirmed numerically identical per the position's mirror symmetry) plus well-formedness checks over a small FEN corpus. Full Java/Python cross-repo parity deferred — the Python FeatureEncoder doesn't exist yet (separate trainer project, PRD "Trainer Architecture," TBD at implementation); this fixture is the exact contract that encoder must reproduce. Left out: Searcher/UciApplication wiring (PR3); float32 oracle test (US-3, not in the approved PR2 plan scope); real trained weights (trainer is separate, later Python work) — all tests use a small deterministic synthetic network (TestNetworks). Verification: full reactor suite green (engine-core 183/0/2 skipped, up from 178 — the 5 new NNUE tests; engine-tuner 131/0/1 skipped unchanged), NodeCountRegressionTest/PerftHarnessTest/EvaluatorTest unaffected. Phase: 15 — NNUE Phase B, PR2 of 3
Why: PR3 of 3 — the search/UCI integration boundary, isolated from PR2's accumulator/feature-extraction correctness so concurrency safety can be reviewed on its own (per the Phase B plan's PR split). What: EvalFile becomes live. resolveNnueNetworkForSearch() runs once per `go` (not once per thread, so any fallback info string prints exactly once): EvalType != NNUE -> null (Classical); NNUE + no EvalFile -> fallback info string, null; NNUE + EvalFile -> load (cached by path so unchanged files aren't re-read every go), on success returns the network, on failure (bad path, bad format) prints the failure + falls back, never crashes. The resolved network (or null) is then used to call setEvaluatorStrategy(new NnueEvaluator(network)) separately for the main Searcher and for each Lazy SMP helper inside its own spawn-loop lambda — a fresh instance every time, satisfying the non-negotiable one-evaluator- per-thread constraint from the Phase B plan. setoption EvalType NNUE no longer immediately falls back (Phase A's stub message is gone) since a real evaluator now exists once a valid file loads; the fallback decision moved from setoption-time to go-time, since EvalFile may not be set yet when EvalType is set. Removed NnueNetwork.loadUnchecked — added speculatively in PR2, never called; UciApplication catches the checked IOException directly. Left out: UCI-visible reload-on-EvalFile-change beyond the existing path-based cache check; a debug UCI command to report the loaded network's provenance beyond the load-time info string (developer tooling is separate PRD scope). Verification: full reactor suite green (engine-core 183/0/2 skipped unchanged; engine-uci 23/0/8 skipped, up from 20 — 3 net new tests). UciApplicationIntegrationTest 13/13, including: NNUE+no-file fallback at go-time, NNUE+invalid-file fallback, NNUE+valid-file actually searches (network-loaded info string, no fallback), and NNUE+valid- file+Threads=2 (exercises the per-helper NnueEvaluator construction path without crash/hang). Phase: 15 — NNUE Phase B, PR3 of 3
Why: three independent review passes (code-review Standards+Spec axes, agentic-eval self-critique) on Phase B converged on real, confirmed gaps rather than style nits: 1. NnueNetwork.load() validated magic bytes and format version only — PRD line 251 requires validating width and exact expected file length before allocation, since ".nnue path is user-supplied input to the UCI process." A corrupt/hostile hiddenWidth could previously drive a huge allocation before any rejection. 2. The incremental-vs-rebuild fuzz test's docstring claimed coverage of "every move type" but random legal play essentially never produces castling, en passant, or promotion within 60-ply games — agentic-eval and the Spec-axis review both independently flagged this; first-run-green was weak evidence for those branches specifically. 3. NnueEvaluator.ACCUMULATOR_POOL_SIZE duplicated Board.UNMAKE_POOL_SIZE as an independent literal (768) with no compiler-enforced link — flagged independently by both the Standards-axis review and agentic-eval. What: - NnueNetwork: added MAX_HIDDEN_WIDTH bound check (rejects before any large array allocation) and an exact-file-length check (via a small CountingInputStream) for the Path-based load entry point, comparing declared header fields against the actual file size before reading the body arrays. - Board.UNMAKE_POOL_SIZE made public with a doc comment stating the NnueEvaluator dependency is load-bearing, not cosmetic; NnueEvaluator now references it directly instead of a duplicated literal. - New FeatureExtractorMoveTypeTest: 6 hand-picked FEN+move fixtures (quiet, capture, en passant, kingside castling, promotion, promotion+capture) asserting forEachChange's exact emitted deltas directly — independent of fuzz RNG luck. All passed on first run, confirming both the square-numbering convention used throughout and the delta logic itself. - New NnueNetworkLoaderTest cases: oversized-hiddenWidth rejection and truncated-file rejection, per PRD line 226's reject-path requirement. Left out (consciously deferred, not silently dropped): a staleness check for resolveNnueNetworkForSearch()'s path-based cache (same path, content changed on disk between two `go`s) — real but narrow gotcha, documented as a known gap rather than fixed here; the duplicated ~20-line .nnue test-file writer between engine-core and engine-uci test suites — cosmetic, low value, left as two small independent copies rather than adding cross-module test-jar wiring for it. Verification: full reactor suite green (engine-core 191/0/2 skipped, up from 183 — 8 new tests; engine-uci 23/0/8 skipped unchanged; engine-tuner 131/0/1 skipped unchanged). Perft, mirror-symmetry, and the node-count guard unaffected. Phase: 15 — NNUE Phase B, review-driven fix
…ownership ADR Why: a final freeze review (not tied to a GitHub issue) of Phase B found NnueNetwork.load() read architectureId/featureSetId off the .nnue header but silently discarded both — a future file built for a different architecture or feature set would load successfully and misinterpret its weight bytes instead of failing fast. Separately, the NNUE evaluator's three-tier ownership invariant (immutable shared network / one evaluator per Searcher / one accumulator stack per evaluator / never shared across Lazy SMP helpers) existed only in Javadoc and an ephemeral planning file, with no permanent architecture record. What: - NnueNetwork.load(): validate architectureId and featureSetId against this build's single supported values immediately after reading them (before hiddenWidth, before any allocation), rejecting with a clear IOException naming the field and the supported value. No forward-compatibility logic added — this build implements exactly one architecture and one feature set, matching the existing formatVersion exact-match check's style. - Two new NnueNetworkLoaderTest cases: unsupported architectureId, unsupported featureSetId. Verified red (NullPointerException on a reverted build, proving the check — not the message-contains assertion alone — gates the test) before restoring the fix to green. - New docs/adr/ADR-006-nnue-evaluator-ownership.md recording the three-tier ownership invariant as a permanent decision, with Supporting Evidence cross-checked against the actual UciApplication/NnueEvaluator/Board code, not just asserted. Left out (by design — freeze patch only, no redesign): no forward-compatibility scheme for future architecture/feature-set versions; no runtime/compiler guard against a future edit accidentally sharing an NnueEvaluator across threads (ADR-006 documents this as a known, accepted limitation); no changes to any already-accepted PR1-3 implementation. Reviewed via code-review (Standards + Spec axes) and an agentic-eval self-critique pass, both independently verified against the live diff and current code — no blocking findings on either. Verification: full reactor suite green (engine-core 193/0/2 skipped, up from 191 — the 2 new tests; engine-uci 23/0/8 skipped unchanged; engine-tuner 131/0/1 skipped unchanged). BUILD SUCCESS. Phase: 15 — NNUE Phase B freeze patch (Phase B is now frozen)
… 009 Why: Phase C readiness audit found two PRD-reserved ADRs (int16-canonical- inference, copy-per-ply accumulator stack) were never written despite both decisions being implemented in frozen Phase B code, plus a numbering collision — the freeze patch's own evaluator-ownership ADR was filed as ADR-006, which the NNUE PRD's Appendix B reserves for a distinct Phase D trainer-choice decision. What: ADR-002 and ADR-003 document the already-implemented, already-shipped decisions, verified line-by-line against NnueEvaluator.java/NnueNetwork.java (an agentic-eval pass caught and fixed two off-by-a-few line-number citations before this commit). Evaluator-ownership ADR renamed 006 -> 009 via git mv (content unchanged except the title line's number); repo-wide grep confirms no dangling ADR-006 references remain outside the PRD's own Phase D reservation. Left out: no code changes anywhere in this commit. The float32 oracle's acceptable-drift numeric bound (ADR-002) is explicitly deferred to Phase C, to be measured against a real trained network rather than guessed here. One finding from the agentic-eval pass deliberately left unaddressed here (not a silent drop, a scope decision): ADR-004 still documents onMake(Board, int move) rather than the shipped onMake(Board, int move, int capturedPiece) — ADR-004's own Open Questions anticipated this exact resolution, but ADRs in this repo are immutable once accepted, so recording the resolution needs a new ADR, which was outside this commit's three explicitly scoped tasks (ADR-002, ADR-003, the rename). Flagged to the user, not created unprompted. Phase: 15 — NNUE Phase B documentation freeze
Why: both planning artifacts were produced and approved in prior sessions but never committed, leaving the repo's tracked state out of sync with what Phase C's GitHub issues need to link back to. What: commit the two docs/superpowers/plans/*.md files as-is (no content changes) plus .graphifyignore (repo config, previously untracked). Left out: no code changes; graphify-out/ is regenerated by the post-commit hook and intentionally left untouched here. Phase: 15 — nnue
Why: PRD Phase B named these as exit criteria but Phase B shipped without them (accepted scope narrowing); Phase C's PR C-1 absorbs this carried-over debug tooling as the foundation PR C-2/C-3 build on. What: NnueEvaluator.dumpAccumulators() (stable text dump, sp doubles as ply index) and verifyAgainstRebuild(Board) (rebuilds into scratch arrays, returns RebuildDiff — perspectiveColor/firstDivergingIndex/ delta, NONE sentinel), both read-only, never mutate whiteAcc/blackAcc. currentAccumulator stays package-private per the plan's own risk review. Package-private corruptForTest hook for the new rebuild- comparator test. No UCI wiring, no EvaluatorStrategy/NnueNetwork/ FeatureExtractor change — engine-core only, per PR C-1's scope. Left out: oracle comparator, eval breakdown (C-2); NnueDebug mode, debug UCI commands (C-3); benchmark corpus (C-4); CI wiring (C-5). Acceptance criteria verified: NnueEvaluatorDebugTest (3 tests) and NnueEvaluatorRebuildComparatorTest (4 tests) green; full reactor suite green (engine-core 200/200, engine-tuner 131/131); zero behavior change confirmed via unchanged NodeCountRegressionTest and byte-identical 31-position bench (77,265,370 nodes, matching the pre-existing Phase A/B baseline). Closes #185 Phase: 15 — nnue
Why: the ad-hoc node/edge delta done for PR C-1's post-commit summary was useful but not reusable or archived; every completed PR needs a richer, permanent architectural change record without hand-rolling the analysis each time. What: docs/architecture/graph-audits/generate_report.py — reuses graphify's own graph.json output and its own dependency (networkx) to compute betweenness-centrality deltas, degree deltas, new bridge- node candidates, dependency cycles (via strongly-connected-components, not full enumeration), cross-module dependency changes, community drift (via cohort-overlap, since Louvain community IDs aren't stable across separate graphify runs), new God Nodes, and a modularity-based cohesion verdict. Narrative section is filled in by hand per report. Retroactively generated the C-1 report from the auto-captured pre/post-commit graph.json snapshots (0d0794e -> ef14e9d). Left out: no engine-core/engine-uci/engine-tuner changes — pure docs/tooling, entirely outside production code, matching the request to reuse graphify's existing output rather than build new analysis infrastructure. Phase: 15 — nnue
Why: ADR-002 deferred the float32 oracle to Phase C as its own documented
deliverable; the eval breakdown gives NNUE the same UCI diagnostic shape
the classical Evaluator already has.
What: NnueOracle.compareInt16VsFloat32/compareBatch (float32 dequantized
reference vs int16 canonical eval), NnueEvaluator.explainEval(Board)
(pre-activation ranges/clip counts, output contribution, int16 score,
oracle delta), UciApplication.handleEval() gains its first EvalType
branch routing NNUE to explainEval while the classical path is untouched.
Code-review fixes folded in before this commit: %n -> \n in explainEval
(printBreakdown splits on literal \n; %n becomes \r\n on Windows, this
repo's native target), output-contribution line now reports actual
computed us/them dot products instead of static config values, positional
int[]{min,max,clipped} replaced with RangeAndClipCount record,
compareBatch reuses one NnueEvaluator across the whole FEN list.
Left out: no UCI command for the oracle itself (batch/CI tool, not an
interactive query — Risk Review); eval command extension stays ungated
by NnueDebug, matching classical eval's always-on behavior; NnueDebug
mode and nnue features/acc/verify commands are C-3.
Tests: engine-core 207/207 (2 pre-existing skips), engine-uci 26/26
(8 pre-existing skips), engine-tuner 131/131 (1 pre-existing skip).
NnueOracleTest dequantization verified against hand-computed values.
eval command's classical-mode cases unchanged; new NNUE-mode cases pass.
Closes #186 Phase: 15 — nnue
Why: ADR-002 confines NnueOracle's float32 path to debug/test use; C-2 wired it into NnueEvaluator.explainEval() as the one approved entry point. Nothing previously prevented a future refactor from calling it from evaluate()/onMake()/onUnmake() or from Searcher/EvaluatorStrategy. What: OracleArchitecturalBoundaryTest disassembles compiled .class files with the JDK's own javap (no ArchUnit dependency exists in this repo, and the rule is method-granular — "explainEval may call NnueOracle, evaluate/onMake/onUnmake may not" — which plain package/class-level rules can't express, since NnueOracle deliberately shares NnueEvaluator's production package per issue #186). Asserts Searcher and EvaluatorStrategy never reference NnueOracle, NnueEvaluator's three hot-path methods never invoke it, every other class in the eval package tree is free of it, and explainEval is confirmed as the one place it IS invoked (positive control proving the parsing isn't vacuous). Manually verified red: temporarily inserted an NnueOracle call into evaluate(), reran, confirmed 1 failure with the offending bytecode line in the assertion message, reverted, confirmed green again. Left out: no ArchUnit — one dependency for two classes and three methods fails the ladder; ADR/design-doc updates (none needed, this only encodes an existing frozen boundary). Tests: OracleArchitecturalBoundaryTest 5/5. Full reactor green: engine-core 207/207 (+5 from this commit), engine-uci 26/26, engine-tuner 131/131 (pre-existing skips unchanged). Phase: 15 — nnue
…tool
Why: the existing per-PR audit tracked topology drift but had no
targeted view of the one boundary that actually matters for NNUE Phase
C: production code reaching into debug/oracle-only code. That
information was scattered across generic sections (top-degree-change,
community-moves) that require reading hundreds of lines to notice.
What: generate_report.py classifies every graphify node by owning
src/main/*.java file into production vs. debug (DEBUG_ONLY_MAIN_CLASSES,
currently {NnueOracle.java}, extend as future debug-only src/main
classes land) and adds a new "Architectural Boundary Report" section:
production->debug and debug->production edge deltas (file-pair-approved,
so only NnueEvaluator.java->NnueOracle.java is expected on the prod
side), a frozen-boundary verdict (UNCHANGED/CHANGED), and per-class
coupling deltas for the five ADR-frozen classes (EvaluatorStrategy,
Searcher, NnueEvaluator, FeatureExtractor, NnueNetwork). Classification
keys off source_file, not node label — graphify emits file/class/method
granularity nodes sharing one source_file, and call/reference edges
attach to the class/method nodes, not the file node; an earlier label-
based attempt silently produced zero matches. Dogfooded against the
already-existing 70376a9->4107b38 (C-2) snapshot pair, which also
produces the C-2 audit report the prior session's C-2 commit skipped
per explicit user redirect; verdict UNCHANGED as expected (sole prod->
debug edge is .explainEval()->.compareInt16VsFloat32(), the approved
one). Verified the CHANGED branch with a synthetic 2-node fixture
(fabricated Searcher->NnueOracle edge) — caught and fixed a real
5-tuple/3-tuple unpacking bug in that branch this way, since real C-2
data never exercises it (zero unapproved edges).
Left out: no separate report artifact — this extends the existing
per-PR audit file, doesn't add a new one; graphify-out/ stays
untracked, as instructed.
Tests: manual — real C-2 snapshot pair (UNCHANGED verdict, matches
known-good boundary) and synthetic fixture (CHANGED verdict, correct
rendering, no crash). py_compile clean.
Phase: 15 — nnue
…g UCI (C-3) Why: the PRD names silent accumulator desync as the dominant NNUE bug class. UciApplication holds no persistent Searcher/evaluator reference once "go" returns, so interactive UCI commands can only ever inspect root-position state — the automatic sampled assertion running inside the live search thread during onMake is the one tool that actually catches mid-search desync. What: NnueEvaluator gains a debugMode constructor overload (existing single-arg constructor delegates with debugMode=false, unchanged for all existing call sites); onMake gains, only when debugMode is true, a stack-bounds check (advisory — the Phase C plan's underspecified "stack-depth==ply invariant" reinterpreted as a self-contained check since Board.unmakeSP is private/frozen and isn't even a true invariant against makeNullMove) and a deterministic 1-in-256 incremental-vs- rebuild comparison (verifyAgainstRebuild, PR C-1) — both log via LOG.warn on divergence, never throw. New UciApplication NnueDebug boolean option wires debugMode into both live-search NnueEvaluator construction sites (main searcher + Lazy SMP helpers) and gates three new root-position-only debug commands (nnue features/acc/verify), reusing FeatureExtractor.activeFeatureIndices and C-1's dump/verify tools rather than reimplementing them. Left out: no live mid-search UCI inspection (explicitly rejected in the plan's risk review — would need real cross-thread synchronization, violating ADR-009's ownership boundary for scope far beyond this PR); C-4/C-5 (benchmark corpus, CI integration) untouched. Deviations from the plan's literal wording, both reviewed and justified during pre-implementation grilling: deterministic 1-in-256 sampling instead of RNG (avoids the test flakiness the plan's own risk review flagged); stack-depth check reinterpreted as bounds-checking rather than comparing against Board's private unmakeSP field (would have widened Board's frozen API for an invariant that doesn't even hold across null moves). Fixes applied during this PR's own review pass: reused FeatureExtractor.activeFeatureIndices instead of hand-rolling feature enumeration in dumpActiveFeatures; extracted buildRootNnueEvaluator() to remove duplication between handleEval/handleNnueDebug; nnue verify's output now translates RebuildDiff's raw perspectiveColor int (8/16) to a symbolic white/black label instead of exposing the record's default toString(), with new direct unit tests for both outcomes — a gap the original test suite didn't cover, since it only ever exercised the non-divergent path over the wire. Tests: engine-core 214/214 (2 pre-existing skips, +7 from this PR: NnueEvaluatorDebugModeTest x2 proves the assertion fires on injected corruption within a bounded move count via a logged desync warning and asserts no exception propagates out of onMake via assertDoesNotThrow, plus 5 from the prior architectural-guard commit). engine-uci 34/34 (8 pre-existing skips, +8 from this PR: 6 UciApplicationIntegrationTest cases covering all three nnue debug commands x NnueDebug on/off x NNUE/Classical active, plus 2 formatVerifyResult unit tests). engine-tuner 131/131 unchanged. Bench: Nodes searched=77265370 (byte-identical to the pre-PR baseline, confirmed across three separate runs including a fresh run in this session) with NnueDebug at its default (false) — proves zero production-path impact. NPS=318804, above both the 316964 baseline and the 301116 floor (WSL timing is directional-only per CLAUDE.md; node count is the load-bearing, deterministic proof). Closes #187 Phase: 15 — nnue
…eport Why: preserve the per-PR structural audit for C-3, matching the C-1/C-2 precedent, using the enhanced tool from the Improvement-2 commit. What: before=41891e4 (pre-C-3), after=643d44c (the C-3 commit). Frozen boundary verdict UNCHANGED. Narrative flags one line requiring ground- truth verification: the graph shows the approved explainEval->NnueOracle edge as "removed," which OracleArchitecturalBoundaryTest's bytecode-level positive control (re-run immediately after generating this report) disproves — graphify's own AST extraction non-determinism on a file that grew several methods this PR, same noise class already documented in the C-2 report for UciApplication.java. Phase: 15 — nnue
…t (C-4) Why: PRD's benchmark-corpus requirement — a categorized, versioned corpus for per-category eval throughput/NPS tracking plus a hard-failure golden-eval pin against the CI test net, both explicitly out of Phase C's earlier PRs (C-1/C-2/C-3 built the debug tooling that produces evals; nothing pinned what a "correct" eval looks like or measured it across board densities). What: bench/nnue-corpus/ — six category files (tactical.epd/quiet.epd are static copies of already-committed fixtures; opening.epd/middlegame.epd/ random-legal.epd/endgame.epd are output of a new one-time generator, NnueCorpusGenerator, tagged @tag("corpus-generation") and gated behind -Dcorpus.generate=true so it never runs in normal `mvn test` or CI) + golden-evals.csv (FEN,int16-eval pinned against TestNetworks.synthetic(8)) + README.md. NnueGoldenEvalTest is a new always-on reactor test asserting exact int16 match per FEN. NnueCorpusBenchmarkTest reports per-category eval throughput + fixed-depth NPS for both evaluators, tagged @tag("nnue-benchmark") (deliberately distinct from NpsBenchmarkTest's "benchmark" tag — ci.yml's existing -Dgroups=benchmark step has no -Dtest= scoping, so sharing that tag would have silently added this to every CI run despite the "not wired into CI yet" intent). The generator does two separate self-play passes with two separate seeds: an unbiased pass (20260713) matching NnueIncrementalVsRebuildFuzzTest's plain random- legal-game machinery verbatim for opening/middlegame/random-legal, and a second, capture-biased pass (20260714) confined to endgame.epd only, needed because pure random play rarely reduces material within a bounded ply count. draw_failures.epd's 3 curated FENs are folded into endgame.epd. Small unrelated cleanup folded in ahead of C-4 (flagged by C-3's own architecture-review as a remaining maintainability item): UciApplication gained newSearchNnueEvaluator(NnueNetwork), collapsing the two live- search-thread NnueEvaluator(network, nnueDebug) construction sites into one place so a future third construction site can't silently drift out of sync with the NnueDebug flag. Left out: CI workflow wiring (ci.yml additions) — deliberately PR C-5, per the issue's own scope split, so a bad corpus file and a bad CI job stay independently revertible. No Stockfish-labeling infra. No corpus content growth beyond this PR's curation. No C-5 work. Deviations from the plan's literal wording: (1) opening/middlegame/ random-legal are generated via fixed-seed self-play rather than derived from OpeningBook's polyglot entries — the plan offered both as valid ("curate ~50-100 FENs from ply 1-15 ... or generate from OpeningBook's polyglot entries"); self-play was chosen to avoid a new engine-core-to- engine-uci-resources coupling (the only committed .bin book lives under engine-uci/src/main/resources/, and OpeningBook.java itself is already in engine-core so no code coupling was needed, but the *book file* wasn't). (2) The endgame category's capture-bias is a new deviation not named in the plan, added because an unbiased pass at up to 150 plies essentially never reached <=12 pieces (0/60 games in an initial run) — confined to that one category and both seeds/passes documented in the generator's own comments. Bug found and fixed during this PR's own code-review pass (not a production change): Board.getCurrentFEN() never flushes a trailing run of empty squares on the last rank (no subsequent rank-transition to trigger the flush that every other rank gets) — either truncating the final rank's square count or, when the whole last rank is empty, dropping it entirely (7 segments instead of 8). This repo's own Board(String) round-trips the truncated output correctly by coincidence (the dropped squares are already empty in the freshly-constructed board), so it's never caused a functional bug here, but a committed corpus file must hold standards-valid FEN. Fixed at the point corpus data is written (NnueCorpusGenerator.repairTruncatedLastRank) rather than in Board.java itself, which is out of C-4's scope (no production behavior changes) and would need its own perft/mirror-symmetry regression pass. Verified: 786/786 FEN lines across all six category files + golden-evals.csv now parse as valid FEN (8 ranks, each summing to exactly 8 squares) via an independent Python validator, up from 786 lines with 38 malformed before the fix. Also fixed during review: a duplicated-javadoc ordering bug in UciApplication (the new helper's javadoc had been inserted between an existing method's javadoc and the method itself, orphaning it) and the benchmark-tag CI-coupling issue described above. Tests: engine-core 217/217 (+3 from this PR: NnueGoldenEvalTest, NnueCorpusGenerator [self-skips via Assumptions unless -Dcorpus.generate=true], NnueCorpusBenchmarkTest [self-skips unless -Dbenchmark.enabled=true — confirmed via a live run under -Dgroups=benchmark that it does NOT execute under that tag, only under its own -Dgroups=nnue-benchmark]). engine-uci 34/34 unchanged. engine-tuner 131/131 unchanged (untouched by this PR). Reproducibility: NnueCorpusGenerator re-run twice in a row after the FEN-repair fix produces byte-identical output for all four generated .epd files and golden-evals.csv (diff -rq clean) — satisfies the approved plan's "Task 3, Reproducibility" requirement and issue #188's "Golden evals reproducible" acceptance criterion directly. Acceptance criteria (issue #188), evidenced: - All six categories present and non-empty: tactical.epd (50), quiet.epd (100), opening.epd (60), middlegame.epd (60), endgame.epd (63), random-legal.epd (60) — 393 total FENs, all valid. - Golden evals reproducible: verified above. - "New CI job passes on a clean develop-based checkout": no new CI workflow job is added by this PR (ci.yml wiring is explicitly PR C-5 per this issue's own "Explicitly Out of Scope" section — the issue's own acceptance-criteria wording pre-dates that scope split and is reconciled here rather than treated as a gap). What already exists covers it without any ci.yml change: NnueGoldenEvalTest has no @tag gating, so it runs under ci.yml:28's existing `mvn -B clean test` reactor step on every PR/push; NnueCorpusBenchmarkTest is deliberately its own tag precisely so it stays out of ci.yml:34's existing -Dgroups=benchmark step until C-5 explicitly wires it in. Bench: no production code touched by the corpus/test additions. UciApplication's construction-site consolidation is mechanically behavior-preserving (same NnueEvaluator(network, nnueDebug) call, same two call sites) — confirmed via the full reactor run above and NpsBenchmarkTest (aggregate 324,666 NPS, no regression flagged; WSL timing is directional-only per CLAUDE.md §3, not gated here). Closes #188 Phase: 15 — nnue
…eport Why: continuing the per-PR graphify architectural audit trail established in C-2/C-3 — verifies PR C-4 (benchmark corpus + golden-eval test) didn't introduce any production<->debug coupling or touch the frozen EvaluatorStrategy/Searcher/FeatureExtractor/NnueNetwork surface, since it's meant to be pure data + test-scope code. What: before=65a75bb (C-3's audit commit), after=42aa8b9 (C-4's feature commit). Architectural Boundary Report shows "No change" across five of six tracked classes and an UNCHANGED frozen-boundary verdict — expected for a data+test PR. NnueEvaluator's 3 flagged coupling edges and the engine-uci->engine-core cross-module edge drop (73->63) are both the same, already-documented UciApplication.java AST-extraction noise from C-2/C-3 (re-confirmed here by re-running OracleArchitecturalBoundaryTest's positive control, still green) — this PR's only production change is a small, behavior-preserving method extraction inside that one file. Left out: nothing — this is a documentation-only follow-up commit, no code changes. Phase: 15 — nnue
Why: two engineering-workflow gaps flagged before starting C-5 — (1) the Board.getCurrentFEN() bug found and worked around during C-4 had no tracking beyond the commit message, risking the workaround silently becoming the permanent canonical implementation; (2) the corpus had no provenance record (seeds, generator commit, golden-eval network identity) for anyone regenerating or auditing it later. What: filed issue #190 documenting the bug (root cause, why C-4 deliberately didn't fix production code, the current workaround, and acceptance criteria for a proper fix including removing the workaround once it lands) and linked it from a new "Known issues" section in bench/nnue-corpus/README.md. Added a "Provenance" section recording corpus version, both generation seeds (20260713 unbiased / 20260714 endgame-only capture-biased) and their scope, self-play parameters, category definitions, the golden-eval network's exact identity (TestNetworks.synthetic(8), seed 42, qa/qb/outputScale), and the regeneration workflow. Left out: no change to Board.java (tracked in #190, not fixed here — C-4's "no production behavior changes" constraint still applies until that issue is picked up on its own). No change to the corpus data itself — documentation only. Phase: 15 — nnue
Why: PR-blocking and nightly CI jobs for Phase C's debug tooling (C-1..C-3) and benchmark corpus (C-4) — none of it was actually enforced by CI until now. ADR-005 requires the NNUE-mode regression job to run under genuine EvalType=NNUE, never a silent Classical fallback, or every NNUE-mode CI result becomes unattributable. What: three new PR-blocking ci.yml steps — "NNUE-mode regression" (-Dgroups=nnue-mode, new NnueModeSearchRegressionTest), "NNUE golden-eval regression" (-Dgroups=nnue-golden, C-4's NnueGoldenEvalTest newly tagged), "NNUE benchmark throughput" (-Dgroups=nnue-benchmark, C-4's NnueCorpusBenchmarkTest, report-only, no assertions). Loader validation (NnueNetworkLoaderTest's malformed-network matrix, +2 cases: zero- hiddenWidth boundary, corrupt modified-UTF-8 string) and the UCI-level "genuinely NNUE, not fallback" confirmation (UciApplicationIntegrationTest#evalTypeNnueWithValidEvalFileActuallySearchesWithNnue, unmodified, asserts the positive "NNUE network loaded" info string) are both untagged and already covered by the existing "Build and test modules" step — no new step needed for either. New nightly-nnue.yml (sibling to nightly-sprt.yml, ubuntu-latest, cron 00:30 UTC + workflow_dispatch): unfixed-seed/larger-game-count fuzz (NnueIncrementalVsRebuildFuzzTest's SEED/GAMES made overridable via -Dfuzz.seed=/-Dfuzz.games=, defaults unchanged, seed printed at test start for reproducibility) and an oracle-bound batch report over the full bench/nnue-corpus/ (NnueOracle.compareBatch, reporting only — no gate, per ADR-002's deferred numeric-threshold policy pending a real trained network). Deliberately no `ref:` pin on the nightly checkout (unlike nightly-sprt.yml's correct-for-its-purpose `ref: develop`) — workflow_dispatch must be able to target this branch, and develop doesn't have this PR's code yet. NnueModeSearchRegressionTest re-runs SearchRegressionTest's 30 existing position constants (referenced directly, that file untouched) through a real search tree with EvaluatorStrategy swapped to NnueEvaluator over the CI test net. Deliberately does not assert "same move as classical" for most positions (the CI test net is untrained random weights with no reason to reproduce a handcrafted evaluator's move preferences) — what's asserted is that search returns a move actually present in MovesGenerator's own legal-move list (compared via Move.pack(), since Move has no equals()). 4 of the 30 (T1/T2/T4/T5, literal mate-in-1 per SearchRegressionTest's own comments) additionally assert the returned move delivers checkmate, evaluator-independent so a non-flaky gate. Left out: production behavior changes (none — this PR touches zero main-sources code; TestNetworks's visibility widening and the NnueNetworkLoaderTest/NnueIncrementalVsRebuildFuzzTest additions are all test-scope). Trainer-reproducibility CI job (Phase D doesn't exist yet). Hard-gating the oracle bound. Phase D. Improvements made before starting C-5 (own commits): 9b867b1 tracked the Board.getCurrentFEN() bug (issue #190) and added corpus provenance documentation. This PR additionally tracks a second gap found while extending the loader-validation matrix: NnueNetwork.load() has zero validation for qa/qb (a zero or negative value loads successfully and only fails later, at evaluation time, with a divide-by-zero ArithmeticException) — deliberately not fixed here (adding the guard is a production behavior change, out of this PR's CI-only scope) and tracked in a new issue, #191, instead. Self-corrections made during this PR's own development (verified empirically, not just reasoned about): an initial nodesVisited() > 0 assertion in NnueModeSearchRegressionTest was dropped after confirming via a standalone script that several regression positions report zero visited nodes even under the classical evaluator — a pre-existing root-level shortcut for trivially-decided positions, not NNUE-specific, so asserting on it would have tested the wrong thing. T3 was removed from the forced-mate-in-1 list after the test's own first run demonstrated it isn't actually a mate-in-1 (the position's correct move only "sets up" mate next ply per SearchRegressionTest's own comment). The move-legality check itself was strengthened from a weak assertDoesNotThrow(makeMove) (Board.makeMove only validates the `reaction` string, not genuine chess legality) to an explicit comparison against MovesGenerator's actual legal-move list, after self-review identified the original check as not meaningfully testing what its name claimed. A hardcoded `ref: develop` in the nightly workflow (copied from nightly-sprt.yml, where it's correct) was caught and removed before any review pass ran — it would have made workflow_dispatch always check out develop, which doesn't have this PR's code, defeating both the acceptance criterion naming workflow_dispatch and the nightly job itself (a -Dtest= for a nonexistent class fails Surefire's failIfNoSpecifiedTests, not a silent no-op). Tests: engine-core 253/253 (+37 over C-4's 217: NnueModeSearchRegressionTest x34, NnueOracleBatchReportTest x1 [self-skips via Assumptions unless -Doracle.batch.enabled=true], NnueNetworkLoaderTest x2 new cases). engine-uci 34/34 unchanged. engine-tuner 131/131 unchanged. CI validation: every new mvn invocation dry-run exactly as ci.yml/ nightly-nnue.yml specify it, with surefire-reports cleared before each to confirm test-class isolation: -Dgroups=nnue-mode (34/34, only NnueModeSearchRegressionTest), -Dgroups=nnue-golden (1/1, only NnueGoldenEvalTest), -Dgroups=nnue-benchmark -Dbenchmark.enabled=true (1/1, only NnueCorpusBenchmarkTest, confirmed no assertions in that class — genuinely report-only), nightly fuzz with -Dfuzz.seed=/-Dfuzz.games= overrides (confirmed default-unchanged behavior separately), -Dgroups=nnue-oracle-batch -Doracle.batch.enabled=true (393 corpus positions, logged, no assertion on magnitude). Both new/changed workflow YAML files parsed successfully. bench/nnue-corpus/ has zero diff, confirming the NnueCorpusCategories.readFens() consolidation (replacing three near-identical private FEN-reading methods) is behavior-preserving. Acceptance criteria (issue #189), evidenced: "full CI run completes green on phase/15-nnue itself, both PR-triggered and via workflow_dispatch" — validated locally (every command above, all green; YAML parses); not yet observed on GitHub Actions, since that requires this branch to actually be pushed and a PR opened, which is the user's call, not this session's. "NNUE-mode regression job explicitly confirmed to run under EvalType=NNUE + the CI test net" — satisfied by two tests together, cited explicitly since no single job does this in isolation: NnueModeSearchRegressionTest (constructs NnueEvaluator directly via setEvaluatorStrategy, bypassing the UCI EvalType option and its fallback path entirely) plus the pre-existing, unmodified UciApplicationIntegrationTest#evalTypeNnueWithValidEvalFileActuallySearchesWithNnue (exercises real EvalType=NNUE over the wire, asserts the positive "NNUE network loaded" string) — both already run under the existing "Build and test modules" ci.yml step. Closes #189 Phase: 15 — nnue
…eport Why: continuing the per-PR graphify architectural audit trail established in C-2/C-3/C-4 — verifies PR C-5 (CI integration) didn't introduce any production<->debug coupling or touch the frozen EvaluatorStrategy/Searcher/ FeatureExtractor/NnueNetwork surface, consistent with it being the only PR in the series with zero main-sources code changes. What: before=4f6f7c9 (C-4's audit commit), after=4dbc24d (C-5's feature commit). Architectural Boundary Report shows "No change" across five of six tracked classes and an UNCHANGED frozen-boundary verdict. The two non-"No change" entries (NnueEvaluator, NnueNetwork) both trace directly to this PR's own intended test-scope changes (TestNetworks widened to public, NnueIncrementalVsRebuildFuzzTest modified) — re-confirmed by re-running OracleArchitecturalBoundaryTest's positive control, still green. Also flags two unrelated .claude/.github/dev-entries graph-node deltas as noise (neither touches engine-core/engine-uci/engine-tuner). Left out: nothing — this is a documentation-only follow-up commit, no code changes. Phase: 15 — nnue
Why: Phase C froze with no Phase D plan or trainer architecture on record — docs/NNUE_PRD.md explicitly deferred trainer location, ADR-006, and ADR-007 to "Phase D start." Before any trainer code lands, the module boundaries, cross-module contracts, and ownership rules need to be settled and reviewed, not discovered mid-implementation. What: docs/architecture/NNUE_TRAINER_ARCHITECTURE.md — 18-section trainer architecture (module boundaries, Canonical Network IR decoupling the exporter from PyTorch state_dict, .nnue/provenance/quantization contracts, 8 permanent Trainer Architecture Invariants, a Contract Matrix naming producer/consumer/owner/versioning/scope for every cross-module contract) plus a grilled-and-accepted decision to locate the trainer at trainer/ in this repo rather than a separate one. Went through /grilling (3 decisions challenged), /architecture-review (x2 — fixed a CanonicalNetwork precision-ambiguity bug, a provenance-timing contradiction, and Contract Matrix Owner-column inconsistencies), and /agentic-eval (x2 — fixed a wrong CLAUDE.md citation, a thin section missing required rationale/alternatives/tradeoffs, and two factual imprecisions in the Contract Matrix). Extended docs/architecture/graph-audits/generate_report.py with a minimal architecture_group() classifier so future graph audits report Engine and Trainer architecture separately — verified against real graph.json snapshots, no change to existing whole-graph analysis. docs/superpowers/plans/2026-07-13-nnue-phase-d-roadmap.md sequences Phase D into 8 PRs (D-1..D-8) derived from the approved architecture; issues #192-#199 filed on GitHub, none implemented. Left out: no trainer implementation code (Python or Java); ADR-006/ ADR-007/ADR-010 remain unwritten as standalone files (tracked as D-1's own scope); no change to engine-core/engine-uci/engine-tuner/ chess-engine-api production code. Phase: 15 — nnue
Why: Phase D needs a place to land real trainer code, and three
decisions that already existed as prose (docs/NNUE_PRD.md Appendix A
items 5-6, and NNUE_TRAINER_ARCHITECTURE.md §16's grilled-and-accepted
repository-location decision) but not as standalone, citable ADRs the
way ADR-002/003/004/005/009 already are.
What: trainer/ directory skeleton exactly per the architecture doc's
§16 nested layout (trainer/trainer/{dataset,encoding,model,export,
quantization,validation,cli}/, each an empty package; tests/, configs/,
scripts/, gitignored outputs/); trainer/pyproject.toml (uv-managed,
torch/numpy declared per already-accepted ADR-006, not new deps
decided here) with a committed uv.lock (uv lock resolved in <1s with
no downloads, verified before committing per an architecture-review
recommendation, matching the doc's own reproducibility invariant);
ADR-006/007/010 extracted verbatim from their existing source
decisions, not re-derived, following the ADR-002 template exactly (9
sections, verified); a path-filtered trainer-ci.yml skeleton
(paths: ['trainer/**'] on push and pull_request) so path-scoping is
correct from the first trainer commit; docs/NNUE_PRD.md Appendix B's
ADR-006/007 stub lines converted to links, plus a dated addendum
linking ADR-010 (which had no pre-existing stub line, since it
postdates the original 2026-07-07 grilling session Appendix B
documents).
Left out: no functional DatasetProvider/FeatureEncoder/model/etc -
that's D-2 onward per
docs/superpowers/plans/2026-07-13-nnue-phase-d-roadmap.md. No change
to engine-core/engine-uci/engine-tuner/chess-engine-api - verified via
diff, zero matches.
Closes #192
Phase: 15 — nnue
Why: issue #192's own acceptance criteria required a graphify graph audit confirming the Trainer Architecture split section (added ahead of Phase D, commit 7570984) shows real node counts for the first time — this is that audit, and its first real exercise against actual trainer content rather than an empty directory. What: Engine Architecture 0 node delta (confirms zero Java production code touched); Trainer Architecture 0 -> 14 nodes; Architectural Boundary Report UNCHANGED across all five FROZEN_BOUNDARY_CLASSES. Left out: no change to generate_report.py itself (already committed/reviewed in 7570984) - this PR only exercises it. Phase: 15 — nnue
Why: Phase D's first functional trainer code. Before any concrete
DatasetProvider, the contract itself needed to be designed around all
three intended sources (Stage 1 text, Stage 2 Stockfish-labeled, Stage
3 self-play), not just the one being implemented now - a separate
architecture-review/grilling/agentic-eval pass on trainer/contracts/
happened first, per instruction, before any implementation touched it.
What: trainer/trainer/contracts/dataset.py - DatasetMetadata/ShardRef/
PositionLabel/PositionMetadata/PositionRecord + DatasetProvider(ABC),
separating dataset metadata, shard enumeration, and position iteration
as three distinct concerns, independent of any storage format. Chosen
ABC over Protocol for fail-fast construction-time enforcement
(matches ADR-002/003/004's precedent, and no Python type-checker runs
in CI yet so Protocol's static-only guarantee wouldn't be enforced
anywhere). eval_cp/eval_mate/wdl documented as side-to-move-relative,
verified against Searcher.java's own negamax convention.
trainer/trainer/dataset/{text_provider,mmap_shard,transform}.py - the
first concrete DatasetProvider (Stage 1, reading normalized CSV per
trainer/configs/stage1-dataset.md's documented dataset choice: Lichess
evaluated positions, resolving PRD Open Question #1), an mmap shard
writer/reader (streams in bounded 10k-record batches, never
materializes a full dataset in memory - fixed after an initial eager
version was caught by code-review), and a composable Transform stage
(dedup, ply-range filter, phase-balancing via a piece-count heuristic -
the phase-balancing piece was initially missed against issue #193's
own scope, caught by the same review pass and added with tests).
Also fixes a real packaging bug only surfaced by actually running the
code: trainer/pyproject.toml's automatic package discovery failed
(outputs/, trainer/, configs/ as sibling top-level dirs) - fixed via
explicit [tool.setuptools.packages.find]; split torch into an optional
'train' extra so dataset-layer test runs don't pull the full CUDA
stack (D-2 has zero PyTorch dependency - only D-4 needs it).
Left out: no feature encoding (D-3), no PyTorch model work (D-4), no
quantization/export (D-5/D-6). Real acquisition of Lichess's published
dump into the normalized CSV format text_provider.py reads is a
separate, later, small task - not fabricated here; tests run against
an explicitly-synthetic fixture.
Closes #193
Phase: 15 — nnue
Why: mirrors D-1's own convention (graph audit as a separate follow-up commit, per the C-1..C-5 precedent) - this audit confirms D-2's real functional code landed with zero engine-side impact. What: Engine Architecture 0 node delta; Trainer Architecture 14 -> 104 nodes (+90, tracing to the DatasetProvider contract, TextDatasetProvider, mmap shard module, Transform module, and 27 new tests - no unaccounted scaffolding); Architectural Boundary Report UNCHANGED across all five FROZEN_BOUNDARY_CLASSES; cross-module dependency changes and community-churn sections checked directly, neither traces to unexpected coupling. Phase: 15 — nnue
Why: Invariant 2 (Java/Python feature parity) needed a real, running Python side and CI gate -- FeatureIndexParityTest.java's own javadoc has stated since Phase B that its fixture exists specifically to pin a future Python FeatureEncoder against it. Before implementing that encoder, a pre-D-3 architectural improvement was requested: eliminate the encoder's constants being duplicated from FeatureExtractor.java rather than defined once. What: Added a versioned Feature Specification (docs/architecture/feature-spec/v1.json) as the shape/rules contract (piece/color/square ordering, index formula) -- architecture doc Section 4.1, resolved via a self-conducted grilling round informed by a primary- source research note (Feast, TFX, ONNX, protobuf, SemVer, Specification-by- Example) confirming no surveyed system parses such a spec on a hot/serving path. Resolution is asymmetric by design: trainer/trainer/encoding/ feature_encoder.py (Python) derives its constants from v1.json at import time; FeatureExtractor.java stays exactly as written (hardcoded, hot-path, allocation-free, zero behavior change) and is instead verified against the same file by a new FeatureSpecConformanceTest.java. A purpose-built minimal JSON reader (MinimalJson.java) avoids adding engine-core's first-ever JSON library dependency for two small fixed-shape test fixtures. The corpus-sharing mechanism (issue #194's own open decision) is a second committed JSON fixture, docs/architecture/feature-spec/parity-corpus-v1.json, containing FeatureExtractor's real captured output (not hand-derived) for all 4 FENs in FeatureIndexParityTest.CORPUS, both perspectives. FeatureIndexParityTest.java gained a new test asserting Java's live output still matches this fixture exactly, replacing what was previously only a well-formedness check for 3 of the 4 positions. trainer-ci.yml went from an echo placeholder to a real job running the full trainer pytest suite, with its path trigger widened to include docs/architecture/feature-spec/** (the shared contract) alongside trainer/**; Java-side drift is already caught by the existing unconditional Backend CI (mvn test on every PR), so the trigger was not widened to engine-core. Verified by deliberately injecting drift on both sides (XOR 56 -> 55) and confirming the parity check fails in each language (Python: 8 tests red; Java: 3 tests red, across FeatureIndexParityTest and FeatureSpecConformanceTest), then reverting -- issue #194's own acceptance criterion. Live GitHub Actions can't be triggered from this environment, so this verification ran locally via the same commands trainer-ci.yml executes. Left out: PyTorch model work, quantization, exporter -- all explicitly out of scope for D-3, deferred to D-4 onward per the Phase D roadmap. Closes #194 Phase: 15 — nnue
Why: track architectural deltas per Phase D PR, per the established C-1..C-5/D-1/D-2 convention. What: before=c40114d, after=7954216. Engine +23 nodes (Java test scope only), Trainer +27 nodes. FeatureExtractor coupling changes: No change -- confirms zero production behavior change. Frozen boundary verdict UNCHANGED. Left out: nothing -- audit-only commit, no code changes. Phase: 15 — nnue
…fra (D-4) Why: this is the first PR that actually trains something. PRD "Trainer Requirements" makes loss/target calibration a hard requirement -- search margins (futility, aspiration, null-move, LMR) are tuned to classical's centipawn scale, so an uncalibrated net produces false SPRT failures attributable to margin mismatch, not the net. Before writing the training loop, seeding/determinism/run-metadata needed one owner so future scripts don't reinvent or drift from the same convention. What: trainer/trainer/reproducibility/ (pre-D-4 improvement, architecture doc Section 10.1, informed by a primary-source research note on PyTorch determinism/RNG/checkpoint-metadata conventions) -- seed_everything(), dataloader_generator(), worker_init_fn() (closes a real DataLoader multi-worker reseeding gap), configure_deterministic_execution() (opt-in, not default), and ExperimentMetadata/capture() (the lighter "identify this run" tier, not RNG-state-exact resume, per Invariant 7's own "not bit-exact on GPU" disclaimer). trainer/trainer/model/network.py: NnueNet, (768->hiddenWidth)x2->1 via a shared nn.EmbeddingBag(mode="sum") FT layer (natural fit for variable-length active-feature sets, no padding), forward pass mirroring NnueEvaluator.java's/NnueOracle.java's exact formula (verified by direct code inspection: `sum = Sigma clamp(acc,qa)*outWeight + outputBias; eval = sum*outputScale/(qa*qb)`) so training and inference stay on the same numeric scale. derive_weight_clip_bounds(qa) resolves the overflow-safety formula (architecture doc Section 7) as bias_clip=qa, weight_clip=(INT16_MAX-qa)/32 -- reserving qa's own headroom for the bias and splitting the remainder across the 32-active-feature worst case, so the two exactly saturate INT16_MAX. Applied after every optimizer step (clip_ft_weights_()), not once at the end. trainer/trainer/model/train.py: the K-calibrated sigmoid loss, verified byte-for-byte identical to TunerEvaluator.java's own sigmoid() (`1/(1+10^(-K*eval/400))`); K is a config-supplied value taken from an existing KFinder run, never recomputed in Python. batching.py assembles EmbeddingBag (indices, offsets) tensors with "us"/"them" relative to each position's own side to move, matching PositionLabel's documented sign convention. Verified end-to-end: a real 3-step run over D-2's existing CSV fixture (exercising both eval_cp and eval_mate label paths) produces a loadable checkpoint; two same-seeded runs produce bit-exact identical weights. Left out: quantization-aware training and export (D-6) -- both explicitly out of scope per issue #195. The eval/WDL loss blend PRD "Trainer Requirements" describes only implements the eval_cp/eval_mate half -- no WDL-bearing DatasetProvider exists before Phase E, so train() raises loudly on a WDL-only label rather than silently treating the blend as satisfied (documented in train.py's own scope-boundary docstring). Closes #195 Phase: 15 — nnue
Why: track architectural deltas per Phase D PR, per the established C-1..C-5/D-1..D-3 convention. What: before=e3524a1, after=a32142a. Engine +0 nodes, Trainer +88 nodes. FeatureExtractor/NnueNetwork coupling changes: No change -- confirms the model's forward pass mirrors Java's formula by inspection, not coupling. Frozen boundary verdict UNCHANGED. Left out: nothing -- audit-only commit, no code changes. Phase: 15 — nnue
…work + Quantizer (D-5) Why: Quantizer/Exporter both need a single, tested seam for "what a network is" between the training checkpoint and the .nnue exporter, instead of duplicated state_dict extraction or an undocumented shape only Exporter understands (NNUE_TRAINER_ARCHITECTURE.md Section 5). A pre-D-5 architectural improvement (researched against MLIR's SSA-value immutability and torch.export/PT2E's "functionalized, never mutates in place" pipeline contract) replaced the originally-approved single CanonicalNetwork dataclass + quantized: bool flag with two separate immutable IR types, since no compiler or ML system surveyed represents "before this pass" / "after this pass" as the same mutable object with a state flag. What: - CanonicalNetwork (float32, mathematical network state) and QuantizedCanonicalNetwork (int16 tensors + int32 output_bias, deployable engine state), both @DataClass(frozen=True) with array.flags.writeable = False on every ndarray field in __post_init__ -- and __setstate__ re-applying the same freeze after pickle, since unpickling reconstructs a fresh writeable array and bypasses __post_init__ entirely (a real gap caught by the round-trip tests themselves, not assumed away). - checkpoint_to_canonical() (the "Checkpoint Loader" stage) extracts ft/output tensors from a D-4 training checkpoint; torch is imported locally inside this one function so the two IR dataclasses themselves need no torch import at all. - quantize() (Quantizer) is a pure function, CanonicalNetwork -> QuantizedCanonicalNetwork, never mutating its input -- round-to-nearest + int16 clip for ft_weights/ft_biases/output_weights, int32 round-to-nearest (no clip) for output_bias. No additional qa/qb scale multiply: verified directly against NnueOracle.java's float64 reference oracle, which uses ftWeights/outWeights unscaled and applies outputScale/(qa*qb) exactly once at the end -- both FT and output layer weights already train in qa-/qb-native units. - clipping_report() is a separate function (not bundled into quantize()'s return value), matching the same "don't overload one return with two concerns" reasoning that killed the old quantized: bool flag. - Provenance fields (network_uuid/trainer_commit/created_at_epoch_seconds) removed from both IR types entirely -- they were always-None dead weight on both stages even in the original design; deferred to Exporter's (D-6) own future signature. - trainer-ci.yml: added --extra train alongside --extra dev. A real, independently discovered gap: D-4 added torch-dependent tests without this, so a clean CI runner (unlike this already-synced dev environment) would fail ModuleNotFoundError on every torch-dependent test file, including D-5's new ones. - Reviewed via research, self-conducted grilling (verified the no-scale-multiply claim directly against NnueOracle.java rather than inferring it), a background architecture-review agent and agentic-eval agent against the doc revision (4 findings, all resolved: stale Section 14 failure-mode row, incomplete immutability-limits documentation, ambiguous output_bias clip-scope wording, Exporter-input clarification), then a second architecture-review + two-axis code-review pass against the implementation itself (4 more findings, all resolved: missing serialization round-trip test, stale Section 2 module table, ambiguous clipping-report-is-separate wording, asymmetric frozen-instance test coverage) -- plus one implementation-time finding from my own adversarial testing (pickle not preserving array immutability) that none of the review passes had caught, fixed and covered by new tests before commit. Left out: Exporter itself, .nnue file generation, provenance/UUID assignment (D-6) -- explicitly out of scope per the issue and this commit. Deviation from issue #196's literal text: the issue's acceptance criteria still describe the pre-revision design (quantized: bool field, Optional provenance fields) -- superseded by the pre-D-5 architectural improvement this same commit implements against. Direct issue-body edit was blocked by this session's auto-mode classifier as an external write not explicitly requested; the supersession is recorded here and in the closing issue comment instead. Closes #196 Phase: 15 — nnue
Why: track D-5's structural footprint against the Trainer Architecture Invariants and the frozen production/debug boundary, per this branch's established per-PR audit convention. What: before=84316d4 (D-4 audit), after=5556f34 (D-5 feature commit). Engine +0, Trainer 219->275 (+56). Frozen boundary verdict: UNCHANGED. Left out: n/a (audit-only commit). Phase: 15 — nnue
Why: #209/#220/#210/#212 are all closed -- the Stage-3 design chain has nothing left blocking a first bounded generator implementation. Built against DR-E12-stage3-generator-selection.md section 17's exported contract: explicit exact generator identity, bounded pilot mode, fail- closed legality/numeric checks, VSPR output via the existing Java codec, no implicit "latest checkpoint" selection, no promotion decision logic. What: new coeusyk.game.chess.core.selfplay package, seven separate components rather than one monolithic GameLoop (#221 section 4): - GeneratorConfig: every identity/budget field required at construction, validated in a compact constructor (SHA-256 format, positive search budget, mandatory finite maxGames, and -- fail closed before any game starts, not just per-search -- maxPlies + a 150-ply defensive search- occupancy margin kept below Board.UNMAKE_POOL_SIZE). The margin is documented as a conservative runtime guard, not a claim that any larger value is mathematically proven unsafe. - EligibilitySmoke: IDENTITY (SHA-256 + logical UUID match) and ENGINE/LEGALITY (legal move, sane score, board left unchanged) checks against a small set of deterministic positions, run before any real pilot game. - CompletedRootCandidateAdapter: the smallest safe seam for DR-E9's own finding that IterationListener.onIteration fires before Searcher's per-iteration abort check -- IterationInfo gained one new field (completed, set from !RootResult.aborted at the existing call site) so a listener can tell a finished iteration from an interrupted one, and this adapter only ever publishes a full-multiPV-width, all-completed depth, discarding a partial/aborted one and retaining the previous complete set otherwise. IterationInfo's only other construction site (Searcher.java) and only other consumer (UciApplication, via named accessors) are both unaffected by the added field. - MoveSelector / BestMoveSelector: deterministic rank-0 selection for this initial vertical slice -- no invented randomness, diversity is a separate future MoveSelector implementation. - GameLoop: one persistent Board/Searcher pair per game (DR-E9's ownership model), natural termination (checkmate/stalemate/threefold/ fifty-move/insufficient-material) checked before every search, the hard move cap producing GameOutcome.UNRESOLVED/MOVE_CAP (never a fabricated draw), on-trajectory sample creation with the mate-distance-in-plies conversion DR-220 actually specifies (not UCI's mate-in-moves), and a defensive per-search history-safety check independent of GeneratorConfig's own construction-time rejection. - SelfPlayCli: thin orchestration only -- runs the smoke check, plays up to maxGames games (aborting the whole pilot on any hard-correctness exception, never skip-and-continue), writes VSPR via the existing Java VsprCodec (never trainer.vspr at runtime) through a temp-file-then- atomic-move, then a DecisionRecord (also atomic) satisfying DR-E12 section 12's minimum provenance fields -- explicitly not dataset assessment, which stays #210's job after ingestion. 34 new tests (GeneratorConfig validation, CompletedRootCandidateAdapter completion/abort/rank-order semantics, GameLoop terminal-state detection for all five natural conditions plus move-cap plus on-trajectory sample/ FEN exactness, EligibilitySmoke identity mismatch handling, and an end-to-end CLI run whose VSPR output round-trips through the same VsprCodec that wrote it). Full engine-core + engine-uci suite: 377 passed, zero regressions.
Why: train() called target_cp() unconditionally for every record before ever reaching the wdl_lambda blend, so a WDL-only record (no eval_cp/ eval_mate) always failed with target_cp()'s generic "no target-cp equivalent" error, regardless of wdl_lambda -- including at wdl_lambda=0, where there's actually nothing missing (the blend never needs an evaluation component at that value). No policy existed for what a Stage-3 WDL-only shard should do at any wdl_lambda. What: _target_cp_for_record() in trainer/trainer/model/train.py applies the decision recorded on #208: eval-bearing records (eval-only, or eval+ WDL) call target_cp() exactly as before, unchanged in every respect. A WDL-only record at wdl_lambda=0.0 returns an inert 0.0 placeholder that the blend's own wdl_lambda=0 multiplier discards, so target_cp() is never called for it and the WDL outcome alone becomes the target. A WDL-only record at wdl_lambda>0.0 raises ValueError immediately, naming wdl_lambda=0 as the fix -- never synthesizes an eval target, never renormalizes wdl_lambda per-record, never silently drops the row. target_cp() itself is untouched: still a pure CP-domain function, still raises on a WDL-only label when called directly, never sees one from train() now. No second Labeler/target-builder/blend implementation -- the existing has_wdl-masked blend in train()'s loss line is unmodified. Tests: 9 new (rejection at lambda 0.5/1.0, acceptance at lambda 0.0 with both wdl=1.0/0.0 records, target_cp() proven never called via a spy, target_cp()'s own direct-call contract unchanged, eval-only regression identical across lambda 0/0.5/1.0, eval+WDL blend regression at all three boundary values). Full trainer suite: 347 passed (was 338), zero regressions. Closes #208
Why: E-14 needs one preregistered stochastic MoveSelector so Stage-3 self-play stops producing byte-identical games (#221's pilot: 3/3 duplicate trajectories). What: SeededDiversitySelector implements bounded near-best-candidate sampling per DR-E14's audit (rank<=3, CP-loss<=40 clamped >=0, exp(-loss/20) weights, mate-rank1 always deterministic). MoveSelector gains default methods (requiredCandidateCount, mechanismKind/Name, lastSelectionWeight/Probability) so GameLoop never sees the sampling formula; BestMoveSelector is unchanged. SeedDerivation replaces GameLoop's old gameSeed+ply (which collided whenever a game ordinal and ply traded places under a fixed run seed) with a SplitMix64 mix, plus a remix() defense against java.util.Random's own sequential-seed correlation. CompletedRootCandidateAdapter gets a real bug fix: it only ever published when the buffer hit the requested candidate count exactly, so multiPV>1 on any position with fewer legal moves (check, endgames) silently never published -- #221's multiPV=1 usage could never hit this since every non-terminal position has >=1 legal move; #222 is the first caller to request multiPV>1. finish() now flushes a final under-width depth. GameLoop wires searcher.setMultiPV()/candidateAdapter sizing off the selector's own requiredCandidateCount(), records true mechanism/seed/rank provenance on PlayedMoveDecision (previously hardcoded to BEST_MOVE regardless of selector), and emits per-ply SelectionDiagnosticsEntry (never written to VSPR) for #222's quality-cost measurability requirement. SelfPlayCli adds optional --diversity-max-rank/--diversity-cp-loss-bound/--diversity-temperature (all-or- nothing), writes VsprHeader.diversityConfig via existing OpaqueConfig, and writes a local diagnostics CSV alongside the VSPR output only when diversity is enabled. Out of scope: VSPR V1/shard V1/#210 ingestion unmodified; no second diversity algorithm; no retraining/SPRT. Phase: 15 -- nnue
Why: #222 forbids choosing diversity parameters before observing the generator's own candidate-score geometry, and forbids altering the preregistration after seeing pilot results. What: Stage A audit (174-ply replay of the #221 bootstrap trajectory at multiPV=3, rank-0 always played) records candidate-count distribution, rank1/2/3 CP-gap percentiles, mate incidence, and a MultiPV rank-inversion finding that shapes the eligibility rule's clamped-loss requirement. Stage B preregistration pins the selected mechanism (SeededDiversitySelector), its three parameters tied directly to the audit's percentiles, seed derivation, mate/forced-line policy, and the second pilot's exact bounded config -- recorded before that pilot ran. Also documents the engine-build-identity investigation (#222 section 9): no existing build-metadata plugin anywhere in this reactor, and adding one is judged to cross into build-system scope creep for this issue -- engineBuildId stays operator-attested, network SHA-256 verification stays mandatory and unchanged. Out of scope: pilot results/report land in the closing issue comment, not this document, so the preregistration itself is never edited after the fact. Phase: 15 -- nnue
Why: before generating any Stage-3 training corpus, need a preregistered design that isolates the diversity mechanism's own effect from the MultiPV=3 search confound #222 already found, and a search budget backed by evidence rather than promoting #222's depth-4 mechanism-validation smoke into a label-quality budget. What: matched control (multiPV=3 + BestMoveSelector) / treatment (multiPV=3 + SeededDiversitySelector, #222's unmodified parameters) generator arms; a depth calibration (28 positions from #222's own pilot output, re-searched at depth 4/6/8 -- found only 32.1% root-move agreement between depth 4 and either deeper budget, motivating depth 6 for the corpus); a resource-justified 8,000-position per-arm budget (measured 0.475s/position at the selected config, <=90min/arm cap); a precombined-list mixing rule with exact Stage-3 row-budget equalization; matched fresh initialization/schedule reusing P3A-001's frozen values; wdl_lambda kept at the trainer's own existing default (1.0); the unmodified Measurement Model as the sole evaluation framework, with pre-declared success/null/regression interpretation. Also identifies a small, additive implementation gap (BestMoveSelector has no way to run at multiPV=3 today) as a Phase A prerequisite, specified but not implemented this turn. Out of scope: no corpus generation, training, or SPRT this turn -- only the depth-calibration data described above, explicitly excluded from any training corpus. No architecture/optimizer/wdl_lambda change. No diversity-parameter change. Phase: 15 -- nnue
Why: E-15's causal isolation strategy needs a control generator arm that runs Searcher at the same multiPV=3 width as SeededDiversitySelector -- otherwise comparing corpora confounds MultiPV-induced search/TT effects with the selector policy itself (DR-E14 section 2.1's own finding). No such capability existed: BestMoveSelector.requiredCandidateCount() was hardcoded to 1. What: BestMoveSelector(int requiredCandidateCount) constructor, additive only -- the no-arg constructor and every existing #221/#222 call site is unchanged (requiredCandidateCount() still defaults to 1 via this(1)). Move selection is unaffected either way: always rank-0. SelfPlayCli gains --control-multipv <n>, valid only when no --diversity-* flags are present (rejected otherwise -- one selector per run, never inferred implicitly). Decisively tested at the GameLoop level: BestMoveSelector(3) is proven to actually widen Searcher's returned candidate set (via the existing diagnostics sink), not just the config record. Also verified (no code change needed) that SelfPlayCli's maxPositions check only ever runs between games, never mid-game -- added a regression test since #221/ #222 never happened to exercise a budget low enough to make this observable. Out of scope: no diversity-parameter change; no VSPR/shard format change. Phase: 15 -- nnue
Why: E-15's matched-arm design needs both training arms to start from literally identical parameter bytes, not just "the same seed passed to two separate process invocations" -- torch/CUDA/BLAS initialization is not guaranteed bit-reproducible across runs even under a fixed seed, and train() had no way to load a pre-built initial state to make this an audited fact rather than an assumption. What: train()'s new initial_state_dict parameter (Optional[dict], default None) -- additive only, every existing caller's behavior is unchanged when omitted. When given, it is loaded into the freshly-constructed model in place of relying on seed_everything()'s own RNG-driven init; seed_everything still runs unconditionally either way, since it also governs the epoch-reshuffle order. Tested: an explicit initial_state_dict loaded into two train() calls with DIFFERENT config.seed values and different training data still produces byte-identical starting weights in both saved checkpoints (learning_rate=0.0 neutralizes optimizer updates so the saved checkpoint reflects the initial weights); the None default is proven unaffected. Out of scope: no training run in this turn -- this is the mechanism E-15's Phase D execution protocol will use, not itself an execution of that protocol. Phase: 15 -- nnue
…cols Why: DR-E15 left several items as placeholders or partially-evidenced claims that could not survive into Phase B unresolved: split/truncation seeds, an unsupported "depth 8 buys nothing over depth 6" claim (only depth-4 comparisons existed), and an initialization protocol that relied on seed alone. What: pins all E-15 seeds now, before any corpus generation, as explicit named constants (20261501/502/503, generation/split/equalization); adds the direct depth-6-vs-depth-8 comparison this task required (same 28 positions, no new ones added after seeing results) -- 39.3% root-move agreement, honestly disclosed as real, persistent instability across every affordable depth tested, with the reasoning for why this doesn't invalidate depth 6 made explicit rather than silently switched to depth 8; documents the verified (already-correct) whole-game corpus-boundary behavior; preregisters "no corpus-level deduplication" with rationale; documents the row-level-truncation-vs-whole-game tradeoff explicitly; records the initial_state_dict-based initialization protocol. Phase A status marked complete; Phase B not started. Out of scope: no diversity parameter, research question, Measurement Model, or wdl_lambda change -- none of Phase A's findings contradicted any of those. Phase: 15 -- nnue
…sment Why: E-15 needed both preregistered generator arms actually run, cross-validated, ingested, split, and quality-gated before Phase D training can be authorized. What: corrected DR-E15's artifact paths (repo-root outputs/ -> trainer/outputs/, where the recovered artifacts actually live; no hash or parameter changed); ran control (BestMoveSelector(3)) and treatment (SeededDiversitySelector) generation at the exact preregistered config/seed/budget, both within the 90-minute cap; cross-verified both VSPR outputs via the existing Java VsprCodec and Python trainer.vspr with zero disagreement; ingested both through the unmodified #210 selfplay_ingest.py path as distinct dataset identities; applied the preregistered grouped split (seed 20261502) and exact row-budget equalization (seed 20261503); both corpora passed every Phase-C hard gate with zero failures and are now assessed/approved. New DR-E15-phase-bc-corpus-generation-report.md records full provenance, hashes, and the descriptive corpus comparison, including the preregistered control-degeneracy outcome (all 58 control games collapsed to one repeated 140-ply trajectory) reported as-is, per DR-E15 section 13, not repaired. Out of scope: Phase D training, checkpoint construction, SPRT -- none run. Phase: 15 -- nnue
Why: combine_and_split() could not read outputs/datasets/stage2-quiet-sf/shard-0.bin -- a legacy119 (pre-#207, itemsize=119) shard the current read_shard() rejects by design, blocking construction of E-15's 36,000-record base training set. What: proved outputs/datasets/stage2-quiet-sf-v1/shard-0.bin is the exact migrate_legacy119_to_v1() output of that legacy source: all 20,000 records match field-by-field (fen/eval_cp/eval_mate/ply/search_depth/search_nodes, wdl/game_id both None as expected), and an independent fresh re-migration is SHA-256 byte-identical to it. Adopted stage2-quiet-sf-v1/ as the canonical path for E-15's data prep only; legacy source left untouched; added a manifest.json to the V1 directory (labeling provenance copied from the original, plus migration evidence) since it had none. Historical phase1/3/4 scripts that hardcode the legacy path are untouched -- not re-run by this task, and rewriting them would misrepresent history. Reran the Phase D preflight: base split reproduces 36,000/4,000 exactly, every frozen Phase-B/C Stage-3 membership hash is unchanged, and both arms' final training lists are byte-identical on the base 36,000 and equal at 43,155 records each (36,000 + 7,155 equalized Stage-3 rows), no deduplication. Out of scope: initialization artifact, training, SPRT -- none run. Phase: 15 -- nnue
Why: several tracked docs/configs/logs/scripts hardcoded a developer's real home directory (both Linux /home/coeusyk and Windows C:\Users\yashk paths) or a machine-specific Stockfish install location -- exposing username and directory layout to anyone reading the repo. What: genericized every prose/log mention (e.g. "the local Stockfish binary", "resolved via PATH") without changing the technical meaning of any record; made trainer/scripts/phase5_rq5_label_noise_floor.py resolve its Stockfish binary via shutil.which() instead of a hardcoded path, verified it still resolves correctly; changed trainer/configs/stockfish-label-e2-real.json's engine_path to a bare "stockfish" (PATH-resolved), consistent with its own doc's existing note that this field is meant to be edited per machine. Out of scope: Dockerfile's authors="yashk" label (author attribution, not a path) and tools/nnue-gauntlet-e4.md's /mnt/c/Tools/... mention (a generic tool-install location, does not reveal a username) -- left untouched.
Why: needed both preregistered arms actually trained under byte-identical initialization and schedule, differing only in Stage-3 data, before Phase E evaluation can be authorized. What: reran the full preflight (frozen hashes, base split, v1-clean count 3,992 matching measurement-model.md exactly); created and hashed one fresh seed-42 initialization, verified byte-identical loaded parameters across two independently constructed models; trained control (142.7s) and treatment (153.2s) with byte-identical TrainingConfig and v1-clean held-out set, only Stage-3 row content differing; both selected checkpoint step 7999, zero NaN/Inf in either loss trajectory. New DR-E15-phase-d-training-report.md records full provenance, hashes, and loss/correlation trajectories. Out of scope: Phase E evaluation, SPRT -- no Phase-E metric was inspected to make any training decision, none run. Phase: 15 -- nnue
Why: Phase D produced selected and final checkpoints for both matched control/treatment arms; Phase E evaluates them against the unmodified Measurement Model per the preregistration's section 11/12 protocol before any release decision is considered. What: evaluated both checkpoints, both arms, on v1-clean (screening + primary cp-only + regression guards) and each arm's own Stage-3 held-out set (exploratory). Computed train/held-out FEN overlap directly rather than assuming it: control 100% (its corpus collapsed to one repeated trajectory, per Phase B/C), treatment 2.72%. v1-clean pooled screening correlation is flat and sign-unstable across checkpoints (-0.0003 selected, +0.0007 final, at the ~0.0004 noise floor); cp-only favors treatment consistently at both checkpoints (+0.0058, +0.0079); RMSE/calibration show no regression either direction. Classified B (null/inconclusive) per the pinned section 12 rule -- the mandatory screening condition isn't clearly met even though the majority-population condition favors treatment. Out of scope: no training, no SPRT, no preregistered parameter changed after seeing results. Closes #223 Phase: 15 -- nnue
…ation Why: measurement-model.md flags cp-only correlation's seed variance as never independently measured; E-15's own cp-only treatment-control deltas (+0.0058 selected, +0.0079 final) needed a configuration-matched noise floor before designing a follow-on Stage-3 experiment worth running. What: trained 3 fresh seeds (42/43/44) on E-15's exact base-only configuration (36,000-record stage1+stage2-quiet-sf-v1 set, P3A-001's frozen schedule, no Stage-3 data). cp-only noise: stdev 0.0023 (selected) / 0.0040 (final), pairwise deltas 0.0011-0.0080. E-15's final-checkpoint delta (0.0079) is statistically indistinguishable from the largest same-config noise draw (0.0080); selected-checkpoint delta (0.0058) sits ~2.5 sigma above the noise band. Also found the previously-cited ~0.0004 pooled noise floor (measured at a different K) does not transfer to this configuration (5-15x larger here). E-15's B (null/inconclusive) classification is unchanged, per instruction -- this is forward-looking characterization, not a retroactive reclassification. Opened #224 to scope a follow-on Stage-3 experiment: shared reproducible opening prefix for both arms (fixes E-15's degenerate single-trajectory control) plus at least one seed replicate per arm, motivated directly by this noise-floor estimate. Not implemented here. Out of scope: no Stage-3 corpus generation, no architecture/Measurement Model change, no SPRT, no E-15 rerun. Phase: 15 -- nnue
…n only) Why: E-15's control collapsed to one repeated trajectory (DR-E15-phase-bc section 12), confounding "diversity mechanism effect" with "opening variety effect" since the control had essentially none of either. DR-M1's noise-floor measurement also showed E-15's cp-only signal isn't clearly separable from seed noise. A follow-on experiment needed a hardened design before any corpus generation, not a rerun of E-15 with the same confound. What: reframed the estimand explicitly (E-16 asks whether the diversity mechanism helps conditional on shared exogenous opening diversity, not "diversity vs none" -- E-15's own estimand and B classification are untouched). Chose Performance.bin (real, already-in-repo Polyglot book) over the self-labeled "random-legal" bench/nnue-corpus/opening.epd. Exhaustively traversed it against the compiled engine-core classpath (no production code changed) to a fixed 6-ply depth: 58 unique terminal FENs, 4 dead-end branches rejected. Separated four previously- bundled randomness sources (opening assignment, treatment generation, Stage-3 split, training seeds 42/43/44 both arms). Corpus generated once per arm, trained under all three seeds -- measures training-seed variance only, stated explicitly. Resource estimate ~131 min total (generation + 6 training runs), corpus size held at 8,000/arm. Interpretation rule uses DR-M1's measured noise band and a paired-by-training-seed comparison, no invented significance threshold at n=3. Listed two real implementation gaps (SelfPlayCli FEN-start flag, color-mirror utility) as blockers, not solved here. Out of scope: no corpus generation, no engine code changes, no SPRT, no selector retuning, no architecture/schedule/WDL change, no E-15 reclassification. Phase: 15 -- nnue
…venance) Why: the first E-16 preregistration had a pairing rule that could break under unequal game lengths (independent per-arm 8,000-position stops), an unjustified color-mirror mitigation, a meaningless seed for a deterministic assignment, an overclaimed provenance statement for Performance.bin, and a generation-time estimate that silently reused E-15's degenerate-control rate. What: pairing hardened to exactly 58 complete games/arm (maxGames=58, maxPositions=null), opening i shared by both arms via a plain identity mapping, corpus size now an expected outcome (~8,970/arm) rather than a stopping criterion, existing row equalization handles the resulting mismatch. Dropped color mirroring -- the paired-by-opening design already cancels single-sided book bias in the decisive comparison; kept the exact legality-preserving transform spec (rank flip only, no file flip) as a deferred, unimplemented option. Removed the now-meaningless opening-assignment seed; pinned the three real remaining seeds (generation 20261601, split 20261602, equalization 20261603); training seeds unchanged (42/43/44, both arms). Corrected Performance.bin's provenance claim to what commit 060ad2f actually documents ("bundled as a classpath resource," no external source) instead of asserting real-game-derived theory; verified sortedness and weight non-uniformity directly instead. Re-derived the resource estimate using treatment's own 75.0s/game rate (E-15's control rate was only cheap because of its degenerate collapse): ~72.5 min/arm, ~160 min total, still under the 90-min/arm cap. Out of scope: no corpus generation, no engine code changes, no SPRT, no selector retuning, no architecture/schedule/WDL change, no E-15 reclassification. Phase: 15 -- nnue
…t-fen Why: DR-E16's shared-opening-prefix design needs SelfPlayCli to start a game from an explicit FEN so both arms can be paired at the opening level; no such seam existed at the CLI layer, only GameLoop's own internal overload. What: added an optional --start-fen flag, additive only. Absent, behavior is byte-identical to before. When given, validated eagerly at CliArgs.parse() time -- before eligibility smoke, before the network loads -- so an invalid FEN fails loudly before any game is attempted. Each game gets a fresh Board parsed from that FEN and is handed to GameLoop's already-existing playGame(gameId, gameSeed, Board) overload, the same entry point GameLoopTest's own terminal-state fixtures have used since #221/#222. No change to GameLoop, Board, or Searcher. Tests: default-start behavior unchanged; invalid FEN fails loudly with no output files written; a supplied FEN is exactly the first searched position with side-to-move/castling(partial, per-color)/en-passant/halfmove/fullmove all preserved; checkmate detection still fires from a supplied non-default start through the CLI; maxGames=58 with maxPositions left unset (not just large) emits exactly 58 complete games; a narrow pairing test proves a control-arm and treatment-arm invocation given the same --start-fen both actually start from that FEN, for two distinct openings. 9 new tests. engine-core+engine-uci full suite: 380+36=416 passed, 0 failures. Out of scope: no corpus generated, no selector/schedule/search-depth/multiPV change, no mirroring, no VSPR/shard format change, no SPRT. Phase: 15 -- nnue
…FEN pool artifact Why: --start-fen (prior pass) applies one FEN to every game in a run, but E-16's pairing rule needs game i to start from pool[i] for all 58 openings within one arm's own run -- the previous "Phase B unblocked" claim was premature; this was the actual remaining gap. What: added --start-fen-file <path>, mutually exclusive with --start-fen. The whole file is read and validated eagerly -- every non-empty line must be a legal-shaped FEN, and there must be at least --max-games entries -- before EligibilitySmoke, the network load, or any game is attempted. Game gameId uses schedule line gameId; gameId/gameSeed derivation is completely untouched. No change to GameLoop, Board, Searcher, VSPR, or selector logic -- same seam --start-fen itself uses, just indexed. Materialized DR-E16 section 2's 58-FEN pool as a tracked, reviewable artifact (bench/nnue-corpus/e16-shared-opening-pool.txt, not hand-edited) via a new E16OpeningPoolGenerator JUnit test, excluded from the default suite via the same @tag("corpus-generation") + -Dcorpus.generate=true convention NnueCorpusGenerator (#188) already established. Ran it twice independently (file deleted between runs) against the same committed Performance.bin: both runs reproduced this design record's own already- published stats (139 edges, 4 dead ends, 66 leaves, 58 unique) and produced byte-identical output, SHA-256 c8f02240f3ddbaf8043de0cf4b7fa3d96914cf91ed8 9f20e19ac8094566b2ce0. Tests: a 3-entry schedule puts opening i on game i's first sample with gameIds staying 0,1,2 in one VSPR file; control and treatment consuming the same schedule file get identical first-sample FENs per index; an invalid FEN anywhere in the schedule fails before generation; too few entries fails before generation; --start-fen + --start-fen-file together is rejected; every prior --start-fen/default test stays green. 6 new tests. engine-core+engine-uci full suite: 386+36=422 passed, 0 failures. Out of scope: no corpus generated, no selector/schedule/search-depth/multiPV change, no mirroring, no VSPR/shard format change, no SPRT. Phase: 15 -- nnue
Why: with the --start-fen-file seam and 58-FEN pool artifact in place (prior commit), Phase B corpus generation could finally run under the hardened pairing design without repeating E-15's degenerate-control confound. What: generated both arms at commit 3ea385f (control 58 games/8,190 samples/72.2min, treatment 58 games/7,605 samples/61.6min, both under the 90-min cap, 0 hard failures). Verified directly: gameIds exactly 0..57 in both arms under both Java and Python VSPR codecs (zero disagreement); game i starts from opening-pool line i in both arms independently and cross-arm (control game i's first FEN == treatment game i's first FEN, 0 mismatches across all 58 indices); the two decision records are byte-identical except output path/VSPR-hash/runId, confirming no unintended config drift between arms. Control's degenerate single-repeated-trajectory problem (E-15: 98.28% duplicate-position rate, 0 decisive games) is fixed and confirmed, not just architecturally plausible: 58/58 unique trajectories, 3.92% duplicate rate, 51 checkmates. Treatment: 58/58 unique trajectories, 0.05% duplicate rate, rank split 60.93/23.67/15.40%, changed-move fraction 39.07%, CP-loss mean/median/max 3.24/0/40 -- consistent with E-15's own treatment profile. Out of scope: no ingestion, split_by_game, row-budget equalization, training, or SPRT. Phase C not started. Generation artifacts (VSPR files, decision records) are gitignored under trainer/outputs/, same convention as E-15 -- only this report is tracked. Phase: 15 -- nnue
… IDs
Why: split_by_game(seed=...) selects held-out games by accumulating each
arm's own row count independently. Control/treatment games for the same
opening index have very different lengths (DR-E16-phase-b-generation-
report.md section 6), so the same seed run per-arm could stop the
accumulation loop after choosing different held-out game IDs, silently
breaking E-16's opening-level pairing at the held-out set.
What: add select_held_out_game_ids() (pure function of the shared 58-game
universe + seed 20261602, no row-count dependence) and
split_by_fixed_game_ids() (partitions one arm against an already-chosen
set) to trainer/trainer/dataset/split.py; split_by_game() itself is
unchanged, existing callers unaffected. Both arms now hold out the
identical {17, 20, 27, 31, 34, 51} and train on the identical remaining
52 IDs by construction. Row counts are not forced equal here -- that
stays the existing row-budget-equalization step (seed 20261603).
Out of scope: no ingestion, split, or equalization of the real E-16
corpora performed. No pinned seed changed (generation 20261601, split
20261602, equalization 20261603, training 42/43/44). Not a response to
any training/evaluation result -- found and fixed before Phase C ran.
Phase: 15 — nnue
…on report Why: DR-E16 Phase C required executing ingestion, the hardened shared held-out-game-ID split, the corpus quality gate, and training-side row- budget equalization against the two frozen Phase B corpora, then freezing the resulting provenance before Phase D training starts. What: ingested both arms via the unmodified #210 selfplay_ingest.py path (stage3-e16-control-001 / stage3-e16-treatment-001); confirmed both arms' game-ID universe is exactly {0..57}; select_held_out_game_ids(seed=20261602) reproduced {17, 20, 27, 31, 34, 51} exactly; split_by_fixed_game_ids applied identically to both arms (zero leakage, identical held-out/training game-ID sets, row order preserved); zero within-arm train/held-out FEN overlap; both datasets passed the hard quality gate and are marked assessed/approved; row-budget equalization (seed=20261603) truncated control by 684 rows (9.01%) to match treatment's 6,911, held-out sets untouched. Out of scope: no training, initialization artifact, Phase E metric, or SPRT run. No pinned seed or split/equalization rule changed (generation 20261601, split 20261602, equalization 20261603, training 42/43/44). Phase: 15 — nnue
…3/44 Why: DR-E16 Phase D required training all six matched runs (control and treatment under each of seeds 42/43/44) against the frozen Phase C equalized corpora, with byte-verified matched initialization and configuration per seed pair, then freezing checkpoint provenance before Phase E evaluation starts. What: base data reconstructed and hash-verified (36,000 train, 3,992 v1-clean held-out); Stage-3 equalized slices re-derived from Phase C's own pipeline and confirmed byte-identical to the frozen input hashes; one fresh init per seed, verified byte-identical between arms via a parameter-tensor hash over independently constructed models; all six runs trained to 20,000 steps with the existing v1-clean checkpoint-selection mechanism unchanged; zero NaN/Inf and zero deviations across all six runs. Out of scope: no Phase E metric comparison performed or inspected to alter execution, no SPRT, no schedule/checkpoint-selection-rule change, no per-opening rebalancing or deduplication. No pinned seed changed (generation 20261601, split 20261602, equalization 20261603, training 42/43/44). Phase: 15 — nnue
…onclusive Why: DR-E16 Phase E required evaluating all 12 frozen Phase D checkpoints against the unmodified Measurement Model, applying the preregistered paired-by-training-seed interpretation rule (DR-E16 section 6) against DR-M1's measured noise band, and reaching a classification without inventing a significance threshold from n=3. What: re-verified all 12 checkpoint hashes before evaluating; cp-only paired deltas are 3/3 sign-consistent (positive) at both checkpoint kinds but every individual delta falls inside DR-M1's ~0.001-0.008 noise band; pooled paired spread exceeds DR-M1's pooled band, read as a screening-only caution, not evidence; regression guards (RMSE, bias, mate-only correlation) hold cleanly in all six seed/kind combinations; Stage-3 exploratory metrics computed and reported but not used to override the v1-clean-based classification; E-15 compared as secondary context only, its own B classification left untouched. Classification: null/ inconclusive. Out of scope: no retraining, retuning, checkpoint-selection change, Measurement Model change, or SPRT. No pinned seed changed (generation 20261601, split 20261602, equalization 20261603, training 42/43/44). Closes #224 Phase: 15 — nnue
…ise band Why: DR-E16-phase-e-evaluation-report.md compared E-16's paired treatment-control deltas to DR-M1's between-run noise band as if that band were a calibrated null for the paired quantity. DR-M1 measures three fully independent single runs; E-16's decisive comparison is matched pairs sharing a verified identical starting parameter hash and data-shuffle order, confirmed directly against train.py and both experiments' own reports. These are different random variables, so "inside DR-M1's band" does not establish "indistinguishable from noise." What: added DR-M2, which withdraws that specific comparison without revising DR-M1 itself or any E-16 metric. E-16's final-checkpoint cp-only paired deltas (+0.00238, +0.00150, +0.00414; mean +0.00267, 3/3 positive) are restated without the noise-band claim; classification stays null/inconclusive, now because no calibrated paired null exists rather than because the deltas fell inside one. Stage-3 exploratory reversal recorded as an unresolved hypothesis, not a finding. Recommends pausing the seeded-diversity Stage-3 line, with a prospective six-seed-replicate estimate offered only as future design guidance. Also added a repo CLAUDE.md rule to run the human skill before any public-facing writing (docs, issue/PR bodies and comments) and to never use a double hyphen as a dash. Out of scope: no training, no corpus generation, no reopening #224, no change to E-16's or E-15's recorded classification or any reported metric. Phase: 15 — nnue
…laims Why: DR-M2's prospective replication guidance had two errors. It counted three new matched seed pairs as twelve new runs and 24-26 minutes instead of six runs and 12-13 minutes, and it claimed a separate M1-style "paired null noise floor" study would be mandatory before more matched training seeds could be interpreted, when those seeds directly estimate the distribution of interest (D_s = treatment_s minus control_s) themselves. It also did not distinguish training-seed variance from corpus-generation variance, which more training seeds on the existing frozen corpora cannot measure. What: fixed the run-count and wall-clock arithmetic in section 8; replaced the mandatory-separate-study claim with a direct explanation of what matched training seeds estimate (D_s's distribution, conditional on the frozen corpora, no separate calibration needed) and what they cannot (corpus-generation variance, since E-16 has exactly one frozen corpus per arm); noted that repeating an exact same-data/same-seed pair would mostly test numerical determinism rather than either kind of variance; added a Phase-15 research-line disposition section (E-15/DR-M1/E-16/DR-M2 status and a three-step resumption order) since no existing roadmap document tracks this line. Out of scope: no change to E-16's or E-15's classification, no reopening of #224, no training, corpus generation, or SPRT. Phase: 15 — nnue
Why: the self-play/Stage-3 diversity-mechanism line (#209-#222 infra, #223/E-15, DR-M1, #224/E-16, DR-M2) is finished for now and its record was spread across nine infrastructure issues, two experiment reports, and two measurement-interpretation notes with no single reconciled summary. What: added a closeout document covering issue-state verification (zero open P15 issues, #223/#224 both closed), the infrastructure chain's durable-versus-research-only split, both experiments' null/inconclusive results, DR-M1/DR-M2's corrected relationship, and deferred/non-promoted items kept separate from the research conclusions (ingestion resume mode, off-trajectory sampling semantics, the full generator-promotion policy, the unimplemented opening-pool mirror transform). States the corrected reading precisely: n=3 matched pairs give only a weak estimate of the D_s distribution, not "inconclusive because no separate paired null noise floor exists." Linked from NNUE_PRD.md's Phase E row. Out of scope: no new experiment, no implementation, no Phase 16 definition, no change to any issue state or recorded classification. Phase: 15 — nnue
…erge Why: preparing the phase/15-nnue -> develop PR surfaced two factual issues in the closeout doc found during a full integration audit: the #220 bullet's "no codec written yet" phrasing was accurate when #220 itself closed but reads as misleading in retrospective context now that the Java/Python VSPR codecs exist and are verified passing; a section cross-reference to DR-M2 drifted after that document's own later section 9 addition. What: clarified the #220 bullet to state the codecs were built afterward under #211/#210 and are passing today (VsprCodecTest 21/21, Python golden-fixture decoder tests 25/25); corrected the DR-M2 section cross-reference from "7 to 9" to "7 and 8" (the actual replication- guidance sections). Out of scope: no rewrite for style, no change to any classification, issue state, or research conclusion. Phase: 15 — nnue
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This merges Phase 15's NNUE self-play/Stage-3 line into
develop. Ancestry is clean:developis an ancestor ofphase/15-nnue, 157 commits ahead, 0 behind, no rebase needed.Durable infrastructure
The chain from
#209through#222built production infrastructure that stays regardless of whether the diversity mechanism it enabled testing turns out to be useful:#220,#211), with cross-language Java/Python round-trip tests running on every PR.SHARD_DTYPEwithwdl/game_id) and its non-destructive legacy-format migration path (#207).selfplay_ingest.py,SelfPlayProvider) and grouped train/held-out splitting by game rather than by row (#210).#221).TrainingConfig.wdl_lambda(#208).Research result: both retraining experiments are null/inconclusive
E-15 (
#223) was the first controlled Stage-3 corpus and retraining experiment. Its control arm collapsed to one repeated 140-ply trajectory across all 58 games (a design flaw independent of the result), and its cp-only signal came out null/inconclusive.E-16 (
#224) fixed that degenerate control by giving both arms the same 58-opening shared prefix before diverging, and ran three seed-matched pairs instead of one. Also null/inconclusive: the paired cp-only deltas were consistently positive (3 of 3, both checkpoint kinds) but small (final-checkpoint mean +0.00267), and regression guards held cleanly throughout.SeededDiversitySelectorremains research-only. It is not promoted, and it does not regress anything either. The Stage-3 line built around it is paused, not abandoned: the recommended next step, if this line is picked back up, is more matched training seeds on the existing frozen corpora, not new corpus generation or selector retuning.DR-M2: a paired-vs-unpaired interpretation correction
A follow-up audit (
DR-M2-e16-paired-variance-interpretation.md) found that E-16's Phase E report had compared its paired treatment/control deltas againstDR-M1's between-run noise band as if that band were a calibrated null for the paired quantity.DR-M1measured three independent single runs; E-16's decisive comparison is matched pairs sharing a verified byte-identical starting parameter hash and an identical data-shuffle order, which is a different random variable.DR-M2withdraws that specific comparison without changing any metric or E-16's classification: the honest reading is that three matched pairs give only a weak estimate of the treatment-minus-control distribution, not that the deltas were shown to be noise.A closeout document (
phase15-nnue-closeout.md) reconciles this whole record in one place, including which infrastructure is durable versus research-only, and what is deferred (not new scope): ingestion resume mode, off-trajectory sampling semantics, a fuller generator-promotion policy, and an unimplemented opening-pool mirror transform.Scope
No Phase 16 work. No selector retuning. No new corpus generation. No SPRT. Nothing here is promoted to production; the currently shipped NNUE network is untouched by this entire branch.
Validation
Full Maven suite (
engine-core,engine-uci,engine-tuner,chess-engine-api): 584 tests, 0 failures, 0 errors, 15 skipped (all explicitly tagged: one-time corpus-generation tooling, tablebase-gated tests, and opt-in benchmark suites). Build succeeded.Full trainer pytest suite: 356 passed, 0 failed.
Cross-language VSPR contract, specifically:
VsprCodecTest(Java) 21/21, the Python golden-fixture decoder suite 25/25,SelfPlayCliTest24/24. Shard format and migration tests, self-play ingestion and grouped-split tests: all passing, all counted in the totals above.A repo-wide audit of the full
develop...phase/15-nnuediff (358 files) found no accidentally tracked generated/output artifacts, no machine-specific absolute paths, no secrets or tokens, and no stale debug/scratch scripts. The small binary/tracked artifacts in the diff (VSPR golden fixtures, an NNUE export test fixture, benchmark EPD corpora, a documented self-play opening pool, a network release report) are all small and intentional.trainer/outputs/remains fully gitignored with nothing tracked under it. One pre-existing cleanup commit already on this branch removed three obsolete tracked binary jars; no new large artifacts were added.