Post-solve constraint verification + constraint provenance - #115
Merged
Conversation
…e violation discriminator
…gate covers exact and scenario runs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two Phase-1 guardrail items: the constraint enforcement claim gets a verifier, and constraints get a place to record why they exist.
1.12 — Post-solve constraint verification
The README says nine hard constraint types are "respected by every plan the search returns." That was underwritten by the search enforcing constraints internally — the engine grading its own homework. Several mechanisms move a plan after that enforcement: whole-percent rounding (
_round_weights_to_pct), the repair operator, an exact backend's solver tolerance.optimizer.verify_runre-reads every returned plan against the problem's constraints and reports aconstraint_checkblock onsolve run, and per scenario onrun_scenarios(each against its own overrides, since a scenario replaces the whole base constraint set). A sibling offrontier_quality, not a fourth gate inside it: those gates ladder usefulness, a breached hard rule is correctness. Aviolations_foundoutranks every other solve signal in the guidance pointer.The primitive already existed:
make_slate_scorerbuilds the scoring and constraint context once and already re-scores solutions for scenario regret. It only needed to stop collapsing the violation vector to a bool. Naming which constraint is_constraint_row_labels, which mirrors the G-row order of both_evaluateblocks — derived from the same parsed constraints that built the pymoo problem, length-checked againstn_ieq_constrso a drifted mirror degrades to unlabeled rows rather than misattributing a violation.Scope is stated, not implied. It shares the encoding with the search, so it catches search-side drift and cannot catch an error in that encoding. The docstring says so, and the skill says
verifiedmeans the plans are legal — not optimal, not complete;frontier_completeand the exact overlay answer those.kindseparates the two causes (deliberately notseverity, which the same response already spends onmetrics.diagnostics). On the proportional path a solution recordsobjective_valuesfrom the search's continuous x alongsideallocationsrounded to whole percents, so onlyobjective_boundrows can trip on that gap —rounding, never failing the status, margin always reported. Every other type is judged on the allocations both writer and checker read, so a breach there is a real property of the recorded plan —violation.explorer._objective_bound_dipsstays as the complementary check: it reads the value a solution stores (what a certified table prints); this re-derives from the stored allocations.1.13 — Constraint provenance
Scenario.motivated_byandCuratedSolution.notesalready record a why; all nine constraint types recorded none — precisely wherebinding_analysiscan price a cap exactly and not say what it protects.Optional
motivated_byon every constraint type (same field name, same role — one term per concept), defaulted so existing bundles, saved problems, and tool calls parse unchanged. Echoed beside the constraint's cost inbinding_analysisand carried into the stakeholder writeup. The engine transports it and never reasons over it.Provenance is inert everywhere identity or staleness is computed. Two leaks found in review and closed in this PR:
explorer._constraint_keyfell through tostr(dict)for the two proportional types, so the annotation entered run-diff identity — annotating a rule would read incompare_runsas removing and re-adding it, and a pre-PRconstraints_snapshotwould diff against a post-PR one as a phantom criteria change._solve_fingerprintdumped constraints wholesale, so annotating a rule after a solve flippedresults_stale— colliding with the skill flow this PR teaches (backfill the motive at Post-Solve Constraint Discovery / writeup time)._strip_motivesnow excludes the field per the fingerprint's own membership criterion ("fields that determine a solve's result"); this also coversScenario.motivated_by, which was already in the hash with the same flaw.Surfacing, per the feature template
New structured fields + one skill section (
optimization_strategy→ Constraint Verification, under Examine) + one pointer (violations_found, outranking the quality signals) + one epistemic caption on rounding rows. Cross-references run both ways (solution_interpreter→ Frontier Quality points at the sibling block).Review & user-testing
Two review passes over the diff plus the real-flow user-test per
frontier-dev: a signal-level chain test confirming the healthy injection chain is untouched, and three agent-in-the-loop sessions on the live tools. The rounding agent readkind: roundingcorrectly and usedmarginto avoid leading with a bound-pinned plan; the clean-run agent keptverifiedsilent ("a null result") but said it would lead with a violation; the provenance agent foundmotivated_byvia the schemas section, kept the motive out of the scores unprompted, and saw it echoed beside the shadow price.Tests
tests/test_constraint_check.py, 30 cases: the anti-drift gate (label count pinned againstn_ieq_constrwith every constraint type on both approaches, plus order guards where same-type rows sit adjacent), the rounding/violation split, the identity and fingerprint guards (each verified to fail without its fix), and the calibration gate — every bundled example'srun,exact_run, and scenario runs verify clean (42 runs across 14 bundles). 879 passed under the CI invocation.Deferred, with a reason
Attaching the check to
explore curated format=needs design first:CuratedSolutionrecords no scenario, and a scenario'sconstraint_overridesreplace the whole base set, so verifying a scenario-curated pick against base rules would manufacture false violations. Scoped in.claude/plans/post-solve-verification-and-provenance-scoping.md.