Skip to content

Add hint quality probe tool - #44

Merged
austin-smith merged 1 commit into
mainfrom
hint-probe-tool
Jul 12, 2026
Merged

austin-smith merged 1 commit into
mainfrom
hint-probe-tool

Conversation

@austin-smith

Copy link
Copy Markdown
Owner

What Changed

Checks in the hint-quality probe harness at tools/hint-probe/ — the acceptance and regression instrument for the hint engines, previously an out-of-tree scratchpad script. Three files: the probe (main.swift), a compile-and-run script (run.sh), and a README that serves as the durable baseline ledger.

Every variant gets identical treatment: 500 seeded deals played to completion by a bot following every hint, alongside a forward-only random control that calibrates each variant's deal universe. All players use the planners' deterministic entry points (no wall-clock deadlines), so every figure is exact and reproduces bit-for-bit across runs and machines. Deals run in parallel across cores (~7 minutes for a full pass; ~40 seconds for a single variant). Acceptance gates are machine-enforced: the probe exits nonzero if any hint follower ever loops, or if Yukon records a single position revisit.

Certified baselines (single run of this exact code, exit 0):

Run Following every hint Random (control)
yukon 62.0% 13.6%
klondike draw-1 44.4% 39.4%
klondike draw-3 24.0% 6.0%
freecell 99.8% 0.2%

The README documents how to read these honestly (draw-1 random wins 39.4% because unlimited stock passes reward persistence — hint value there is efficiency, 133 vs 358 median moves), the acceptance checklist for adding a new variant, and the findings from two evaluated-and-retired tap-policy controls.

Why

The recorded win-rate baselines are load-bearing: they are the yardstick for any planner tuning or shared-code refactor, and they are only comparable on an identical deal generator and driver policy. Keeping the harness out-of-tree meant rebuilding it from memory notes — a rebuilt harness that differs subtly invalidates every recorded number silently. Checking it in pins the instrument to the baselines it produced.

Not part of the app target, the test suite, or CI: tools/ sits outside all Xcode synchronized folders, and the probe is a statistical instrument run deliberately, not a pass/fail check.

Validation

  • Final certification run of the committed code: exit 0, zero gate violations, output matches the README table verbatim.
  • The hint-following column reproduced identically across five consecutive full runs, serial and parallel (parallelism only changes wall-clock time; each seeded game is self-contained and results aggregate in seed order).
  • The loop gate is proven live: during development it caught a follower misclassification (FreeCell's nudge-fallback circling on the one solver-unprovable deal) and correctly failed that run's exit code.
  • CLI rejects invalid inputs (zero deals, unsupported draw counts) with usage and exit 1.
  • App build verified unaffected after adding tools/.

plays 500 seeded deals per variant two ways — following every hint, and
forward-only random moves as the calibration floor — and reports win
rates and loss taxonomy against the baselines recorded in the readme.
deterministic (no search deadlines) and parallel across cores, so a full
pass takes about seven minutes and reproduces exactly. exits nonzero if
any hint follower ever loops or yukon records a position revisit. the
readme records acceptance gates for adding new variants and the findings
from two retired tap-policy controls. compiled by run.sh against the
ui-free game sources; not part of the app, tests, or ci.
@austin-smith
austin-smith merged commit 6168179 into main Jul 12, 2026
2 checks passed
@austin-smith
austin-smith deleted the hint-probe-tool branch July 12, 2026 19:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant