Repository navigation
feat(calibration): pre-registered recalibration policy + machine-enforced gate (v0.5.20) - #134
Merged
Merged
Conversation
…rced gate (v0.5.20) Lore: add the missing permission layer between v0.5.19 calibration-intake (descriptive join of predictions with lab actuals) and any future recalibration engine. The most dangerous failure mode for the wet-lab compression roadmap is silent recalibration: an agent sees real wet-lab data, decides the scorer "needs improvement," and rewrites weights to fit. The new gate makes that impossible without violating a pre-registered, machine-enforced policy. Policy contract (configs/recalibration_policy.yaml): - 7 minimum_conditions: cohort size, controls (pos/neg), orphans, confirmed actives/inactives, metrics availability. Every rule is listed in locked_changes; removing one requires policy_version bump + dated decision log entry. - 5 prohibited_actions (permanent floor): no toxicity relaxation, no hemolysis relaxation, no novelty relaxation, no dangerous pathogen optimization, no post-hoc success redefinition. Mirrors AGENTS.md / MISSION.md. The validator rejects any policy file that drops them. - 2 rate_limits: WEIGHT_CHANGE_L1_BUDGET (0.10), COOLDOWN_DAYS (14). - 3 required_reviewer_artefacts: intake JSON, intake Markdown, dated decision log entry. New modules: - openamp_foundry.calibration.policy: load_recalibration_policy parses, validates, and exposes the policy. Raises PolicyLoadError on missing fields, duplicate ids, unlocked rules, or missing canonical prohibited actions. - openamp_foundry.calibration.recalibration_gate: evaluate_recalibration_gate consumes an intake report + policy and returns a GateVerdict with may_recalibrate (bool), per-rule results, prohibited-action audit, rate-limit status, reviewer- artefact status, reasons, and summary. JSON + Markdown writers. CLI: - openamp-foundry recalibration-gate (exit 0 = may recalibrate, 3 = forbidden, 2 = input error). - Makefile targets: recalibration-gate-example, recalibration-gate INTAKE=... [DATE=...] [PREV=...] [L1=...]. Tests (39 new, total 1647 passing): - tests/test_recalibration_gate.py: policy loader happy + every rejection mode (missing fields, duplicate ids, unlocked rules, missing canonical prohibited actions, missing locked_changes entries, bad types), gate evaluator for every minimum condition, prohibited-action audit, rate-limit status (unknown/ok/exceeded for cooldown + L1 budget), reviewer-artefact status, writers, end-to-end CLI smoke. Docs: - docs/CALIBRATION_POLICY.md: human-readable overview. - CHANGELOG.md, docs/ROADMAP.md, docs/METRICS_CURRENT.md, AGENTS.md: v0.5.20 milestone. - docs/ARCHITECTURE.md, docs/PLAN.md: link policy into wet-lab compression loop and Phase 5 active-learning plan. Honest limitations: the gate does NOT trigger weight updates. A may_recalibrate=true verdict is a permission, not a command. The synthetic example correctly yields may_recalibrate=false (cohort size 4 < 5, one positive control failed, all reviewer artefacts missing). This is the expected outcome on tiny synthetic data and is itself a useful sanity check that the gate is enforcing the cohort floor. Co-authored-by: OmX <omx@openamp.local>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the missing permission layer between v0.5.19 calibration-intake (descriptive join) and any future recalibration engine. The most dangerous failure mode for the wet-lab compression roadmap is silent recalibration: an agent sees real wet-lab data, decides the scorer "needs improvement," and rewrites weights to fit. The new gate makes that impossible without violating a pre-registered, machine-enforced policy.
What ships
Policy contract (
configs/recalibration_policy.yaml)minimum_conditions(cohort size, controls, orphans, positives, negatives, metrics availability)prohibited_actionspermanent floor (no toxicity relaxation, no hemolysis relaxation, no novelty relaxation, no pathogen optimization, no post-hoc success redefinition)rate_limits(L1 weight budget 0.10, cooldown 14 days)required_reviewer_artefacts(intake JSON, intake Markdown, dated decision log entry)locked_changesentries, one per enforced ruleModules
openamp_foundry.calibration.policy:load_recalibration_policyparses and validates the policy. Rejects policy files that drop canonical prohibited actions or removelocked_changesentries.openamp_foundry.calibration.recalibration_gate:evaluate_recalibration_gatereturns aGateVerdictwithmay_recalibrate(bool), per-rule results, prohibited-action audit, rate-limit status, reviewer-artefact status, reasons, and summary. JSON + Markdown writers.CLI
openamp-foundry recalibration-gate(exit 0 = may recalibrate, 3 = forbidden, 2 = input error)recalibration-gate-example,recalibration-gate INTAKE=... [DATE=...] [PREV=...] [L1=...]Tests
39 new tests, total 1647 passing (was 1614). New file:
tests/test_recalibration_gate.pycovering policy loader (happy + every rejection mode), gate evaluator for every minimum condition, prohibited-action audit, rate-limit status (unknown/ok/exceeded), reviewer-artefact status, writers, end-to-end CLI smoke.Docs
docs/CALIBRATION_POLICY.md(new): human-readable overviewCHANGELOG.md,docs/ROADMAP.md,docs/METRICS_CURRENT.md,AGENTS.md: v0.5.20 milestonedocs/ARCHITECTURE.md,docs/PLAN.md: link policy into wet-lab compression loop and Phase 5Honest limitations (must read before relying on the gate)
may_recalibrate=trueverdict is a permission, not a command. The decision to apply a weight change still belongs to a human reviewer with a dated decision log entry.make validate-scoring,make bench-triage, and the selectivity benchmark — not by this policy.may_recalibrate=false(cohort size 4 < 5, one positive control failed, all reviewer artefacts missing). That is the expected outcome on tiny synthetic data and is itself a useful sanity check that the gate is enforcing the cohort floor.AGENTS.mdandMISSION.md. The validator rejects a policy file that drops any of them. Any relaxation of the source documents must happen in lockstep with the policy file.Verification
Co-authored-by: OmX omx@openamp.local