Skip to content

feat(calibration): pre-registered recalibration policy + machine-enforced gate (v0.5.20) - #134

Merged
cschanhniem merged 1 commit into
mainfrom
agent/oamp-loop-20260704x
Jul 4, 2026
Merged

cschanhniem merged 1 commit into
mainfrom
agent/oamp-loop-20260704x

Conversation

@cschanhniem

Copy link
Copy Markdown
Collaborator

Summary

Adds the missing permission layer between v0.5.19 calibration-intake (descriptive join) and any future recalibration engine. The most dangerous failure mode for the wet-lab compression roadmap is silent recalibration: an agent sees real wet-lab data, decides the scorer "needs improvement," and rewrites weights to fit. The new gate makes that impossible without violating a pre-registered, machine-enforced policy.

What ships

Policy contract (configs/recalibration_policy.yaml)

  • 7 minimum_conditions (cohort size, controls, orphans, positives, negatives, metrics availability)
  • 5 prohibited_actions permanent floor (no toxicity relaxation, no hemolysis relaxation, no novelty relaxation, no pathogen optimization, no post-hoc success redefinition)
  • 2 rate_limits (L1 weight budget 0.10, cooldown 14 days)
  • 3 required_reviewer_artefacts (intake JSON, intake Markdown, dated decision log entry)
  • 12 locked_changes entries, one per enforced rule

Modules

  • openamp_foundry.calibration.policy: load_recalibration_policy parses and validates the policy. Rejects policy files that drop canonical prohibited actions or remove locked_changes entries.
  • openamp_foundry.calibration.recalibration_gate: evaluate_recalibration_gate returns a GateVerdict with may_recalibrate (bool), per-rule results, prohibited-action audit, rate-limit status, reviewer-artefact status, reasons, and summary. JSON + Markdown writers.

CLI

  • openamp-foundry recalibration-gate (exit 0 = may recalibrate, 3 = forbidden, 2 = input error)
  • Makefile targets: recalibration-gate-example, recalibration-gate INTAKE=... [DATE=...] [PREV=...] [L1=...]

Tests

39 new tests, total 1647 passing (was 1614). New file: tests/test_recalibration_gate.py covering policy loader (happy + every rejection mode), gate evaluator for every minimum condition, prohibited-action audit, rate-limit status (unknown/ok/exceeded), reviewer-artefact status, writers, end-to-end CLI smoke.

Docs

  • docs/CALIBRATION_POLICY.md (new): human-readable overview
  • CHANGELOG.md, docs/ROADMAP.md, docs/METRICS_CURRENT.md, AGENTS.md: v0.5.20 milestone
  • docs/ARCHITECTURE.md, docs/PLAN.md: link policy into wet-lab compression loop and Phase 5

Honest limitations (must read before relying on the gate)

  1. The gate does NOT trigger any weight update. It only emits a verdict.
  2. A may_recalibrate=true verdict is a permission, not a command. The decision to apply a weight change still belongs to a human reviewer with a dated decision log entry.
  3. The gate evaluates cohort evidence, not pipeline calibration health. Benchmark regressions are caught by make validate-scoring, make bench-triage, and the selectivity benchmark — not by this policy.
  4. The synthetic example correctly yields may_recalibrate=false (cohort size 4 < 5, one positive control failed, all reviewer artefacts missing). That is the expected outcome on tiny synthetic data and is itself a useful sanity check that the gate is enforcing the cohort floor.
  5. The five canonical prohibited actions are duplicated from AGENTS.md and MISSION.md. The validator rejects a policy file that drops any of them. Any relaxation of the source documents must happen in lockstep with the policy file.

Verification

1647 passed, 7 skipped in 31.84s
ruff check src/openamp_foundry/calibration src/openamp_foundry/cli tests/test_recalibration_gate.py → All checks passed!

Co-authored-by: OmX omx@openamp.local

…rced gate (v0.5.20)

Lore: add the missing permission layer between v0.5.19 calibration-intake
(descriptive join of predictions with lab actuals) and any future
recalibration engine. The most dangerous failure mode for the
wet-lab compression roadmap is silent recalibration: an agent sees
real wet-lab data, decides the scorer "needs improvement," and
rewrites weights to fit. The new gate makes that impossible without
violating a pre-registered, machine-enforced policy.

Policy contract (configs/recalibration_policy.yaml):
- 7 minimum_conditions: cohort size, controls (pos/neg), orphans,
  confirmed actives/inactives, metrics availability. Every rule is
  listed in locked_changes; removing one requires policy_version
  bump + dated decision log entry.
- 5 prohibited_actions (permanent floor): no toxicity relaxation,
  no hemolysis relaxation, no novelty relaxation, no dangerous
  pathogen optimization, no post-hoc success redefinition. Mirrors
  AGENTS.md / MISSION.md. The validator rejects any policy file
  that drops them.
- 2 rate_limits: WEIGHT_CHANGE_L1_BUDGET (0.10), COOLDOWN_DAYS (14).
- 3 required_reviewer_artefacts: intake JSON, intake Markdown,
  dated decision log entry.

New modules:
- openamp_foundry.calibration.policy: load_recalibration_policy
  parses, validates, and exposes the policy. Raises PolicyLoadError
  on missing fields, duplicate ids, unlocked rules, or missing
  canonical prohibited actions.
- openamp_foundry.calibration.recalibration_gate:
  evaluate_recalibration_gate consumes an intake report + policy
  and returns a GateVerdict with may_recalibrate (bool), per-rule
  results, prohibited-action audit, rate-limit status, reviewer-
  artefact status, reasons, and summary. JSON + Markdown writers.

CLI:
- openamp-foundry recalibration-gate (exit 0 = may recalibrate,
  3 = forbidden, 2 = input error).
- Makefile targets: recalibration-gate-example,
  recalibration-gate INTAKE=... [DATE=...] [PREV=...] [L1=...].

Tests (39 new, total 1647 passing):
- tests/test_recalibration_gate.py: policy loader happy + every
  rejection mode (missing fields, duplicate ids, unlocked rules,
  missing canonical prohibited actions, missing locked_changes
  entries, bad types), gate evaluator for every minimum condition,
  prohibited-action audit, rate-limit status (unknown/ok/exceeded
  for cooldown + L1 budget), reviewer-artefact status, writers,
  end-to-end CLI smoke.

Docs:
- docs/CALIBRATION_POLICY.md: human-readable overview.
- CHANGELOG.md, docs/ROADMAP.md, docs/METRICS_CURRENT.md, AGENTS.md:
  v0.5.20 milestone.
- docs/ARCHITECTURE.md, docs/PLAN.md: link policy into wet-lab
  compression loop and Phase 5 active-learning plan.

Honest limitations: the gate does NOT trigger weight updates. A
may_recalibrate=true verdict is a permission, not a command. The
synthetic example correctly yields may_recalibrate=false (cohort
size 4 < 5, one positive control failed, all reviewer artefacts
missing). This is the expected outcome on tiny synthetic data and
is itself a useful sanity check that the gate is enforcing the
cohort floor.

Co-authored-by: OmX <omx@openamp.local>
@cschanhniem
cschanhniem merged commit 1abe8bb into main Jul 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant