Skip to content

Post-solve constraint verification + constraint provenance - #115

Merged
cafzal merged 4 commits into
mainfrom
claude/post-solve-verification
Jul 31, 2026
Merged

Post-solve constraint verification + constraint provenance#115
cafzal merged 4 commits into
mainfrom
claude/post-solve-verification

Conversation

@cafzal

@cafzal cafzal commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Two Phase-1 guardrail items: the constraint enforcement claim gets a verifier, and constraints get a place to record why they exist.

1.12 — Post-solve constraint verification

The README says nine hard constraint types are "respected by every plan the search returns." That was underwritten by the search enforcing constraints internally — the engine grading its own homework. Several mechanisms move a plan after that enforcement: whole-percent rounding (_round_weights_to_pct), the repair operator, an exact backend's solver tolerance.

optimizer.verify_run re-reads every returned plan against the problem's constraints and reports a constraint_check block on solve run, and per scenario on run_scenarios (each against its own overrides, since a scenario replaces the whole base constraint set). A sibling of frontier_quality, not a fourth gate inside it: those gates ladder usefulness, a breached hard rule is correctness. A violations_found outranks every other solve signal in the guidance pointer.

The primitive already existed: make_slate_scorer builds the scoring and constraint context once and already re-scores solutions for scenario regret. It only needed to stop collapsing the violation vector to a bool. Naming which constraint is _constraint_row_labels, which mirrors the G-row order of both _evaluate blocks — derived from the same parsed constraints that built the pymoo problem, length-checked against n_ieq_constr so a drifted mirror degrades to unlabeled rows rather than misattributing a violation.

Scope is stated, not implied. It shares the encoding with the search, so it catches search-side drift and cannot catch an error in that encoding. The docstring says so, and the skill says verified means the plans are legal — not optimal, not complete; frontier_complete and the exact overlay answer those.

kind separates the two causes (deliberately not severity, which the same response already spends on metrics.diagnostics). On the proportional path a solution records objective_values from the search's continuous x alongside allocations rounded to whole percents, so only objective_bound rows can trip on that gap — rounding, never failing the status, margin always reported. Every other type is judged on the allocations both writer and checker read, so a breach there is a real property of the recorded plan — violation. explorer._objective_bound_dips stays as the complementary check: it reads the value a solution stores (what a certified table prints); this re-derives from the stored allocations.

1.13 — Constraint provenance

Scenario.motivated_by and CuratedSolution.notes already record a why; all nine constraint types recorded none — precisely where binding_analysis can price a cap exactly and not say what it protects.

Optional motivated_by on every constraint type (same field name, same role — one term per concept), defaulted so existing bundles, saved problems, and tool calls parse unchanged. Echoed beside the constraint's cost in binding_analysis and carried into the stakeholder writeup. The engine transports it and never reasons over it.

Provenance is inert everywhere identity or staleness is computed. Two leaks found in review and closed in this PR:

  • explorer._constraint_key fell through to str(dict) for the two proportional types, so the annotation entered run-diff identity — annotating a rule would read in compare_runs as removing and re-adding it, and a pre-PR constraints_snapshot would diff against a post-PR one as a phantom criteria change.
  • _solve_fingerprint dumped constraints wholesale, so annotating a rule after a solve flipped results_stale — colliding with the skill flow this PR teaches (backfill the motive at Post-Solve Constraint Discovery / writeup time). _strip_motives now excludes the field per the fingerprint's own membership criterion ("fields that determine a solve's result"); this also covers Scenario.motivated_by, which was already in the hash with the same flaw.

Surfacing, per the feature template

New structured fields + one skill section (optimization_strategyConstraint Verification, under Examine) + one pointer (violations_found, outranking the quality signals) + one epistemic caption on rounding rows. Cross-references run both ways (solution_interpreterFrontier Quality points at the sibling block).

Review & user-testing

Two review passes over the diff plus the real-flow user-test per frontier-dev: a signal-level chain test confirming the healthy injection chain is untouched, and three agent-in-the-loop sessions on the live tools. The rounding agent read kind: rounding correctly and used margin to avoid leading with a bound-pinned plan; the clean-run agent kept verified silent ("a null result") but said it would lead with a violation; the provenance agent found motivated_by via the schemas section, kept the motive out of the scores unprompted, and saw it echoed beside the shadow price.

Tests

tests/test_constraint_check.py, 30 cases: the anti-drift gate (label count pinned against n_ieq_constr with every constraint type on both approaches, plus order guards where same-type rows sit adjacent), the rounding/violation split, the identity and fingerprint guards (each verified to fail without its fix), and the calibration gate — every bundled example's run, exact_run, and scenario runs verify clean (42 runs across 14 bundles). 879 passed under the CI invocation.

Deferred, with a reason

Attaching the check to explore curated format= needs design first: CuratedSolution records no scenario, and a scenario's constraint_overrides replace the whole base set, so verifying a scenario-curated pick against base rules would manufacture false violations. Scoped in .claude/plans/post-solve-verification-and-provenance-scoping.md.

@cafzal
cafzal merged commit 30d406f into main Jul 31, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant