explore certify: per-pick verdicts for the curated finalists - #121
Merged
Conversation
CERTIFY exists to prove the curated finalists optimal at their own tradeoff, but the certificate only reported frontier-level facts: an aggregate count of NSGA points the exact overlay dominates, plus up to three examples. The user's actual question — "are the finalists I curated optimal?" — could only be answered by reading the persisted exact-run JSON off disk and comparing by hand. Found by driving every bundled runbook end to end. On shift_coverage_staffing BOTH curated picks turned out dominated and had to be re-curated from the overlay; nothing in the certify response said so. What changed: - `per_pick` on the certify payload — keyed by `content_signature` (the identifier curation already uses), one entry per finalist: `verdict` (optimal / dominated / inconclusive), `gap`, and `dominated_by` — the signature of the nearest certified point that beats it on every objective. Absent, not empty, when nothing is curated, the way `regret.curated` behaves. - Same dominance criterion as the frontier-level audit (`_dominates_min` against the cleaned exact front), so the two readings can never disagree. `gap` is the worst per-objective shortfall normalized by that objective's spread across both fronts — the box the certificate already reasons in — so it reads on the same 0–1 scale as regret; an optimal pick's gap is 0. - A pin whose stored values predate a current objective is reported `inconclusive` with `missing_objectives`, rather than scored against a fabricated 0.0 that would read as a spurious "optimal". - The `recommendation` leads with the tally and `next_steps` routes a dominated finalist to its certified replacement (match the signature in `explore solutions source="exact"`, curate it in its place) — verified end to end on the example that surfaced the defect. - Read side: solution_interpreter → "Reading the Certificate" gains the per-pick block; the certify tool docstring, the signatures-scope redirect, architecture.md's tools table + engine section, and the README principle line all name it. Tests: eight engine cases (optimal, dominated with its gap and certifying signature, nearest-of-several beaters, agreement with the dominance audit, absent without curation, stale pin, both prose paths) plus a wire-level case through `explore certify` — the defect was a payload gap, so an engine-only assertion would not have caught it. Full suite 888 passed / 2 skipped.
cafzal
force-pushed
the
claude/certify-per-pick-verdicts
branch
from
August 2, 2026 03:58
2f58920 to
7241f4f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect
CERTIFY's stated job is to prove the curated finalists optimal at their own tradeoff. The certificate didn't do that: it reported frontier-level facts only — an aggregate count of NSGA points the exact overlay dominates, plus up to three examples. The user's real question, "are the finalists I curated optimal?", could only be answered by reading the persisted exact-run JSON off disk and comparing by hand.
How it was found: driving every bundled runbook end to end. On
shift_coverage_staffing, BOTH curated picks turned out dominated and had to be re-curated from the overlay — and nothing in the certify payload said so.What changed
per_pickon the certify payload, keyed bycontent_signature(the identifier curation already uses). One entry per finalist:verdict(optimal/dominated/inconclusive),gap, and — on a dominated pick —dominated_by, the signature of the nearest certified point that beats it on every objective. Absent, not empty, when nothing is curated, the wayregret.curatedbehaves._dominates_minagainst the cleaned exact front), applied per pick, soper_pickanddominance_auditcan never tell different stories (asserted).gapis the distance to that nearest certifying point: the worst per-objective shortfall normalized by that objective's spread across both fronts — the box the certificate already reasons in — so it reads on the same 0–1 scale asregret. Anoptimalpick has nothing above it, so its gap is 0.inconclusivewithmissing_objectives, rather than scored against a fabricated0.0that would read as a spurious "optimal" — the same class of silent misreport this PR fixes.recommendationleads with the tally (curated picks: 1 of 2 optimal at their own tradeoff, 1 dominated);next_stepsroutes a dominated finalist to its certified replacement. Verified end to end onshift_coverage_staffing: thedominated_bysignature matches a point inexplore solutions source="exact", curating it works, and re-certifying readsoptimal.solution_interpreter→ Reading the Certificate gains the per-pick block (and its stale "Five blocks:" count is now accurate). Thecertifytool docstring, the signatures-scope redirect, architecture.md's tools table + engine section, and the README principle line all name it.Surface coherence
regret.curatedinexplore scenario_resultswas the nearest analogue and set the conventions: curated-only block,content_signaturekeys,custom_namealongside, absent when nothing is pinned. Placement is top-level (a lens of the same rank asdominance_audit/coverage/completeness) rather than nested underdominance_audit— burying the answer would defeat the point.per_pickis a pure map with no sibling metadata, mirroringcorner_sharpening; the explanation ridesrecommendationand the skill section.Tests
Eight engine cases in
tests/test_certify.py(optimal; dominated with its gap and certifying signature; nearest of several beaters; agreement with the dominance audit; absent without curation; stale pin; both prose paths) plus a wire-level case throughexplore certify— the defect was a payload gap, so an engine-only assertion wouldn't have caught it.Full suite: 891 passed, 2 skipped (882/2 before), rebased on
origin/main@ 1a76a96.