Skip to content

explore certify: per-pick verdicts for the curated finalists - #121

Merged
cafzal merged 1 commit into
mainfrom
claude/certify-per-pick-verdicts
Aug 2, 2026
Merged

explore certify: per-pick verdicts for the curated finalists#121
cafzal merged 1 commit into
mainfrom
claude/certify-per-pick-verdicts

Conversation

@cafzal

@cafzal cafzal commented Aug 2, 2026

Copy link
Copy Markdown
Owner

The defect

CERTIFY's stated job is to prove the curated finalists optimal at their own tradeoff. The certificate didn't do that: it reported frontier-level facts only — an aggregate count of NSGA points the exact overlay dominates, plus up to three examples. The user's real question, "are the finalists I curated optimal?", could only be answered by reading the persisted exact-run JSON off disk and comparing by hand.

How it was found: driving every bundled runbook end to end. On shift_coverage_staffing, BOTH curated picks turned out dominated and had to be re-curated from the overlay — and nothing in the certify payload said so.

What changed

  • per_pick on the certify payload, keyed by content_signature (the identifier curation already uses). One entry per finalist: verdict (optimal / dominated / inconclusive), gap, and — on a dominated pick — dominated_by, the signature of the nearest certified point that beats it on every objective. Absent, not empty, when nothing is curated, the way regret.curated behaves.
  • Same criterion as the frontier-level audit (_dominates_min against the cleaned exact front), applied per pick, so per_pick and dominance_audit can never tell different stories (asserted).
  • gap is the distance to that nearest certifying point: the worst per-objective shortfall normalized by that objective's spread across both fronts — the box the certificate already reasons in — so it reads on the same 0–1 scale as regret. An optimal pick has nothing above it, so its gap is 0.
  • A stale pin is declined, not guessed. A pin whose stored values predate a current objective is reported inconclusive with missing_objectives, rather than scored against a fabricated 0.0 that would read as a spurious "optimal" — the same class of silent misreport this PR fixes.
  • Prose stays a caption. The recommendation leads with the tally (curated picks: 1 of 2 optimal at their own tradeoff, 1 dominated); next_steps routes a dominated finalist to its certified replacement. Verified end to end on shift_coverage_staffing: the dominated_by signature matches a point in explore solutions source="exact", curating it works, and re-certifying reads optimal.
  • Read side: solution_interpreterReading the Certificate gains the per-pick block (and its stale "Five blocks:" count is now accurate). The certify tool docstring, the signatures-scope redirect, architecture.md's tools table + engine section, and the README principle line all name it.

Surface coherence

regret.curated in explore scenario_results was the nearest analogue and set the conventions: curated-only block, content_signature keys, custom_name alongside, absent when nothing is pinned. Placement is top-level (a lens of the same rank as dominance_audit / coverage / completeness) rather than nested under dominance_audit — burying the answer would defeat the point. per_pick is a pure map with no sibling metadata, mirroring corner_sharpening; the explanation rides recommendation and the skill section.

Tests

Eight engine cases in tests/test_certify.py (optimal; dominated with its gap and certifying signature; nearest of several beaters; agreement with the dominance audit; absent without curation; stale pin; both prose paths) plus a wire-level case through explore certify — the defect was a payload gap, so an engine-only assertion wouldn't have caught it.

Full suite: 891 passed, 2 skipped (882/2 before), rebased on origin/main @ 1a76a96.

CERTIFY exists to prove the curated finalists optimal at their own tradeoff,
but the certificate only reported frontier-level facts: an aggregate count of
NSGA points the exact overlay dominates, plus up to three examples. The user's
actual question — "are the finalists I curated optimal?" — could only be
answered by reading the persisted exact-run JSON off disk and comparing by
hand.

Found by driving every bundled runbook end to end. On shift_coverage_staffing
BOTH curated picks turned out dominated and had to be re-curated from the
overlay; nothing in the certify response said so.

What changed:

- `per_pick` on the certify payload — keyed by `content_signature` (the
  identifier curation already uses), one entry per finalist: `verdict`
  (optimal / dominated / inconclusive), `gap`, and `dominated_by` — the
  signature of the nearest certified point that beats it on every objective.
  Absent, not empty, when nothing is curated, the way `regret.curated` behaves.
- Same dominance criterion as the frontier-level audit (`_dominates_min`
  against the cleaned exact front), so the two readings can never disagree.
  `gap` is the worst per-objective shortfall normalized by that objective's
  spread across both fronts — the box the certificate already reasons in — so
  it reads on the same 0–1 scale as regret; an optimal pick's gap is 0.
- A pin whose stored values predate a current objective is reported
  `inconclusive` with `missing_objectives`, rather than scored against a
  fabricated 0.0 that would read as a spurious "optimal".
- The `recommendation` leads with the tally and `next_steps` routes a
  dominated finalist to its certified replacement (match the signature in
  `explore solutions source="exact"`, curate it in its place) — verified end
  to end on the example that surfaced the defect.
- Read side: solution_interpreter → "Reading the Certificate" gains the
  per-pick block; the certify tool docstring, the signatures-scope redirect,
  architecture.md's tools table + engine section, and the README principle
  line all name it.

Tests: eight engine cases (optimal, dominated with its gap and certifying
signature, nearest-of-several beaters, agreement with the dominance audit,
absent without curation, stale pin, both prose paths) plus a wire-level case
through `explore certify` — the defect was a payload gap, so an engine-only
assertion would not have caught it. Full suite 888 passed / 2 skipped.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant