Add hint quality probe tool - #44
Merged
Merged
Conversation
plays 500 seeded deals per variant two ways — following every hint, and forward-only random moves as the calibration floor — and reports win rates and loss taxonomy against the baselines recorded in the readme. deterministic (no search deadlines) and parallel across cores, so a full pass takes about seven minutes and reproduces exactly. exits nonzero if any hint follower ever loops or yukon records a position revisit. the readme records acceptance gates for adding new variants and the findings from two retired tap-policy controls. compiled by run.sh against the ui-free game sources; not part of the app, tests, or ci.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Changed
Checks in the hint-quality probe harness at
tools/hint-probe/— the acceptance and regression instrument for the hint engines, previously an out-of-tree scratchpad script. Three files: the probe (main.swift), a compile-and-run script (run.sh), and a README that serves as the durable baseline ledger.Every variant gets identical treatment: 500 seeded deals played to completion by a bot following every hint, alongside a forward-only random control that calibrates each variant's deal universe. All players use the planners' deterministic entry points (no wall-clock deadlines), so every figure is exact and reproduces bit-for-bit across runs and machines. Deals run in parallel across cores (~7 minutes for a full pass; ~40 seconds for a single variant). Acceptance gates are machine-enforced: the probe exits nonzero if any hint follower ever loops, or if Yukon records a single position revisit.
Certified baselines (single run of this exact code, exit 0):
The README documents how to read these honestly (draw-1 random wins 39.4% because unlimited stock passes reward persistence — hint value there is efficiency, 133 vs 358 median moves), the acceptance checklist for adding a new variant, and the findings from two evaluated-and-retired tap-policy controls.
Why
The recorded win-rate baselines are load-bearing: they are the yardstick for any planner tuning or shared-code refactor, and they are only comparable on an identical deal generator and driver policy. Keeping the harness out-of-tree meant rebuilding it from memory notes — a rebuilt harness that differs subtly invalidates every recorded number silently. Checking it in pins the instrument to the baselines it produced.
Not part of the app target, the test suite, or CI:
tools/sits outside all Xcode synchronized folders, and the probe is a statistical instrument run deliberately, not a pass/fail check.Validation
tools/.