diff --git a/AGENTS.md b/AGENTS.md index 60cc2aff..0595af50 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -517,6 +517,8 @@ After qualified assay data exists, build: **v0.5.19 progress:** `openamp-foundry calibration-intake` joins pilot panel predictions with validated lab result actuals and produces a per-candidate report. Cohort metrics are gated by `MIN_COHORT_SIZE=5` to prevent small-sample theater. The module is descriptive only — recalibration requires a separate human-reviewed decision log (`docs/DECISION_RULES.md`). Synthetic-but-clearly-labeled example data is in `examples/lab_results/`. 29 new tests, total 1614 passing. +**v0.5.20 progress:** `openamp-foundry recalibration-gate` evaluates a calibration-intake report against the pre-registered policy in `configs/recalibration_policy.yaml` and emits a binary `may_recalibrate` verdict. The policy encodes 7 minimum conditions (cohort size, controls, orphans, positives, negatives, metrics availability), 5 permanent prohibited actions (toxicity, hemolysis, novelty, pathogen enhancement, post-hoc success redefinition), and 2 rate limits (L1 weight budget, cooldown). The validator rejects policy files that omit any canonical prohibited action or any `locked_changes` entry. The gate does NOT trigger weight updates; it is the missing permission layer between v0.5.19 intake and a future recalibration engine. Exit code 0 when `may_recalibrate=true`, 3 when false. 39 new tests, total 1647 passing. See `docs/CALIBRATION_POLICY.md`. + * lab-result ingestion; * hit/failure calibration; * retrospective analysis of why the model was right or wrong; diff --git a/CHANGELOG.md b/CHANGELOG.md index 7d1b8ce4..becb6722 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -284,3 +284,95 @@ improvements made before committing the ~$10k wet-lab synthesis budget. field and in `examples/lab_results/README.md`. It exists solely to exercise the intake workflow end-to-end. When real validated lab results arrive, replace the example directory — the pipeline itself does not change. + +--- + +## [Unreleased — Recalibration Policy + Gate] — 2026-07-04 + +### v0.5.20 — Recalibration Policy + Gate + +Pre-registered, machine-readable policy that gates any pipeline recalibration. +This is the missing piece between v0.5.19 calibration-intake (descriptive +join) and a future recalibration engine: the gate now answers the binary +question "may recalibration proceed from this intake report?" with a +verdict that no agent can silently bypass. + +**New policy contract — `configs/recalibration_policy.yaml`** + +- `minimum_conditions` (7 rules): MIN_COHORT_SIZE, MIN_POSITIVES_IN_COHORT, + MIN_NEGATIVES_IN_COHORT, POSITIVE_CONTROLS_ALL_PASS, + NEGATIVE_CONTROLS_ALL_PASS, NO_ORPHAN_LAB_RESULTS, COHORT_METRICS_AVAILABLE. +- `prohibited_actions` (5 permanent floors): NO_TOXICITY_RELAXATION, + NO_HEMOLYSIS_RELAXATION, NO_NOVELTY_RELAXATION, + NO_DANGEROUS_PATHGEN_OPTIMIZATION, NO_POST_HOC_SUCCESS_REDEFINITION. +- `rate_limits` (2 rules): WEIGHT_CHANGE_L1_BUDGET (0.10), COOLDOWN_DAYS (14). +- `required_reviewer_artefacts` (3 artefacts): intake JSON, intake + Markdown, dated decision log entry. +- `locked_changes` records the lock date for every enforced rule. The + validator rejects a policy file that lists an enforced rule without a + matching `locked_changes` entry. + +**New modules** + +- `openamp_foundry/calibration/policy.py`: `load_recalibration_policy` + parses, validates, and exposes the policy. Removes or relocations of + any canonical prohibited action cause `PolicyLoadError` at load time. +- `openamp_foundry/calibration/recalibration_gate.py`: + `evaluate_recalibration_gate` consumes an intake report + a loaded + policy and returns a `GateVerdict`. Writes JSON and Markdown via + `write_gate_verdict_json` and `write_gate_verdict_markdown`. +- `openamp_foundry/cli/commands/reports.py`: `_run_recalibration_gate`. +- `cli/main.py`: `recalibration-gate` subcommand registered. + Exit code 0 when `may_recalibrate=true`, 3 when false, 2 on input error. + +**Makefile targets** + +- `recalibration-gate-example`: runs the gate on the synthetic intake + report. Expect exit code 3 (cohort size too small, one positive + control failed, all required reviewer artefacts missing). +- `recalibration-gate INTAKE=... [DATE=YYYY-MM-DD] [PREV=YYYY-MM-DD] [L1=float]` + for real intake reports. + +**Tests — 39 new, total 1647 passing (was 1614)** + +- `tests/test_recalibration_gate.py` covers: + - policy loader happy path + every rejection mode (missing fields, + duplicate ids, unlocked rules, missing canonical prohibited + actions, missing locked_changes entries, bad types); + - gate evaluator with synthetic intake reports for every minimum + condition (cohort size, positives, negatives, controls, orphans, + insufficient metrics); + - prohibited-action audit always in force; + - rate-limit status (unknown / ok / exceeded for cooldown + L1 budget); + - reviewer-artefact status (missing / present based on disk); + - JSON + Markdown writers produce non-empty, schema-valid output; + - end-to-end smoke test of the CLI (exit code 3 on synthetic example). + +**Documentation** + +- `docs/CALIBRATION_POLICY.md` (new): explains the policy, the gate, the + prohibited-action floor, the rate limits, how to update the policy, + and how the policy fits the wet-lab compression roadmap. +- `docs/ARCHITECTURE.md`: links the calibration package into the + wet-lab compression loop. +- `docs/PLAN.md`: marks Phase 5 (active-learning) as gated by this policy. +- `docs/ROADMAP.md`: new v0.5.20 milestone entry. +- `docs/METRICS_CURRENT.md`: new entry in the change log. + +**Honest limitations (must read before relying on the gate)** + +- The gate **does NOT trigger any weight update**. It only emits a verdict. +- A `may_recalibrate=true` verdict is a **permission**, not a command. + The decision to apply a weight change still belongs to a human reviewer + with a dated decision log entry (`docs/DECISION_LOG_.md`). +- The gate evaluates **cohort evidence**, not pipeline calibration + health. Benchmark regressions are caught by `make validate-scoring`, + `make bench-triage`, and the selectivity benchmark — not by this + policy. These checks must keep running independently. +- The synthetic example correctly yields `may_recalibrate=false`. That + is the expected outcome on tiny synthetic data and is itself a + useful sanity check that the gate is enforcing the cohort floor. +- The five canonical prohibited actions are duplicated from + `AGENTS.md` and `MISSION.md`. The validator rejects a policy file + that drops any of them; the human source documents must be updated + in lockstep before the policy file can be edited. diff --git a/Makefile b/Makefile index 827b4605..8e12e533 100644 --- a/Makefile +++ b/Makefile @@ -43,7 +43,9 @@ help: @echo " make wave0-5b-filter Filter Wave 0.5b shortlist (depends on wave0-5b-generate)" @echo " make lab-result-intake Join a pilot panel CSV with lab result JSON files" @echo " make lab-result-intake-example Run intake on the synthetic example data" - @echo " make test Run full test suite (1614 tests, >=80% coverage)" + @echo " make recalibration-gate-example Evaluate recalibration gate on synthetic intake example" + @echo " make recalibration-gate Evaluate recalibration gate on a real intake report" + @echo " make test Run full test suite (1647 passing tests, >=80% coverage)" @echo " make coverage Test suite with per-module coverage report" @echo " make lint Ruff lint check on src/ tests/ scripts/" @echo " make typecheck mypy type check on src/" @@ -265,6 +267,29 @@ lab-result-intake: --out-json outputs/calibration_intake.json \ --out-md outputs/calibration_intake.md +recalibration-gate-example: lab-result-intake-example + PYTHONPATH=src $(PYTHON) -m openamp_foundry.cli recalibration-gate \ + --intake-report outputs/calibration_intake_example.json \ + --intake-report-date 2026-07-04 \ + --out-json outputs/recalibration_gate_example.json \ + --out-md outputs/recalibration_gate_example.md + +recalibration-gate: + @if [ -z "$(INTAKE)" ]; then \ + echo "Usage: make recalibration-gate INTAKE= [DATE=] [PREV=] [L1=]"; \ + echo ""; \ + echo "Or run the synthetic example without arguments:"; \ + echo " make recalibration-gate-example"; \ + exit 1; \ + fi + PYTHONPATH=src $(PYTHON) -m openamp_foundry.cli recalibration-gate \ + --intake-report "$(INTAKE)" \ + $$(if [ -n "$(DATE)" ]; then echo --intake-report-date "$(DATE)"; fi) \ + $$(if [ -n "$(PREV)" ]; then echo --previous-recalibration-at "$(PREV)"; fi) \ + $$(if [ -n "$(L1)" ]; then echo --weight-l1-distance "$(L1)"; fi) \ + --out-json outputs/recalibration_gate.json \ + --out-md outputs/recalibration_gate.md + clean: rm -rf outputs/*.jsonl outputs/*.md outputs/*.json outputs/evidence outputs/phase3_evidence .pytest_cache .ruff_cache diff --git a/configs/recalibration_policy.yaml b/configs/recalibration_policy.yaml new file mode 100644 index 00000000..a57e9c3a --- /dev/null +++ b/configs/recalibration_policy.yaml @@ -0,0 +1,281 @@ +# Pre-registered recalibration policy for OpenAMP Foundry. +# +# This file is the MACHINE-READABLE CONTRACT that gates any pipeline recalibration. +# It is intentionally HUMAN-WRITTEN and REVIEWED before any wet-lab data arrives. +# +# Load order: +# - calibration-intake (builds a report JSON from panel + lab results) +# - recalibration-gate (loads THIS file, evaluates intake report, returns verdict) +# +# Honesty rules: +# - All `pre_registered: true` rules are PERMANENT until the policy_version +# itself is bumped AND a human reviewer commits a `decision_log` entry that +# documents the change. The validator REJECTS policy files where a previously +# enforced rule disappears or relaxes without a recorded decision. +# - `prohibited_actions` is the permanent floor. These rules CANNOT be +# disabled. They encode the project's non-negotiable safety constraints +# (no activity-only optimization, no hemolysis relaxation, no novelty +# lowering) so they are duplicated from AGENTS.md / MISSION.md. +# - This policy does NOT itself trigger any weight updates, scoring changes, +# or selection rule modifications. It only answers the binary question: +# "may recalibration proceed from this intake report?" A "no" verdict is +# always the safe default. + +policy_version: 1 +locked_at: "2026-07-04" # ISO 8601 date when a human reviewed this policy +human_reviewer: "OpenAMP Foundry maintainers (v0.5.20 recalibration policy ratification)" + +# ---------------------------------------------------------------------- +# Minimum conditions -- ALL must pass for the gate to return +# may_recalibrate=true. Each condition lists the threshold and rationale. +# ---------------------------------------------------------------------- +minimum_conditions: + + - id: "MIN_COHORT_SIZE" + description: > + Cohort must have at least N validated lab results that successfully + joined to a pilot-panel candidate. + threshold: 5 + applies_to: "report.n_matched_candidates" + rationale: > + Smaller cohorts produce unstable AUROC estimates and would make any + weight update indistinguishable from overfitting to noise. This + matches the gate already used in calibration-intake cohort metrics. + + - id: "MIN_POSITIVES_IN_COHORT" + description: > + The matched cohort must contain at least N candidates with a confirmed + ACTIVE antimicrobial signal (MIC <= cutoff or equivalent). + threshold: 2 + applies_to: "cohort.activity_vs_active_mic.n_positive_lab" + rationale: > + A recalibration dataset with zero or one active hit cannot distinguish + a well-tuned scorer from one that simply ignores activity. The + pipeline needs at least two confirmed hits to learn anything useful. + + - id: "MIN_NEGATIVES_IN_COHORT" + description: > + The matched cohort must contain at least N candidates with a confirmed + INACTIVE result. + threshold: 2 + applies_to: "cohort.activity_vs_active_mic.n_negative_lab" + rationale: > + Same logic as MIN_POSITIVES: without confirmed inactive results we + cannot measure false positive rate, so any weight update would only + reward activity and would erode the safety floor. + + - id: "POSITIVE_CONTROLS_ALL_PASS" + description: > + Every positive control in the cohort must have passed. + threshold: true + applies_to: "report.control_failures" + rationale: > + A single failed positive control invalidates the entire assay batch + for that result. Recalibration on contaminated data is forbidden. + + - id: "NEGATIVE_CONTROLS_ALL_PASS" + description: > + Every negative control in the cohort must have passed. + threshold: true + applies_to: "report.control_failures" + rationale: > + Same as positive controls. Both directions must be clean before + trusting the assay for any direction. + + - id: "NO_ORPHAN_LAB_RESULTS" + description: > + The number of orphan lab results (results whose candidate_id does + not appear in the pilot panel) must be zero, OR a documented + explanation must be present in the intake report. + threshold: 0 + applies_to: "report.n_orphan_lab_results" + rationale: > + Orphan results suggest either a mismatched candidate_id, a candidate + that was never officially selected for assay, or a data-entry bug. + All three are reasons to halt recalibration. + + - id: "COHORT_METRICS_AVAILABLE" + description: > + Cohort metrics must be present (not flagged insufficient_data). + Calibration-intake flags cohort metrics as insufficient_data when + the matched cohort is below MIN_COHORT_SIZE. + threshold: false + applies_to: "cohort.activity_vs_active_mic.insufficient_data" + rationale: > + We will not compute or act on AUROC values flagged by intake as + insufficient_data. This rule is double-defensive with MIN_COHORT_SIZE. + +# ---------------------------------------------------------------------- +# Prohibited actions -- permanent floor. These CANNOT be relaxed by +# future policy edits without bumping policy_version and recording a +# human decision log entry. They duplicate AGENTS.md / MISSION.md. +# ---------------------------------------------------------------------- +prohibited_actions: + + - id: "NO_TOXICITY_RELAXATION" + description: > + Recalibration may NOT modify scoring weights, scorer coefficients, + or filter thresholds in any way that REDUCES the toxicity penalty + applied to candidate ranking. + rationale: > + Activity without safety is not valuable. AGENTS.md explicitly + forbids objectives that optimize for mammalian toxicity in any + direction. This rule is permanent. + + - id: "NO_HEMOLYSIS_RELAXATION" + description: > + Recalibration may NOT modify any hemolysis-related weight, + threshold, or scorer coefficient downward. The hemolysis risk + penalty floor may only stay flat or grow stronger. + rationale: > + Hemolysis safety is non-negotiable. MISSION.md lists dual-use + review as a required gate. This rule is permanent. + + - id: "NO_NOVELTY_RELAXATION" + description: > + Recalibration may NOT lower novelty thresholds or weights below + their currently shipped values. Novelty is the primary safeguard + against rediscovery of known AMPs. + rationale: > + A candidate that is essentially a known AMP is not a discovery. + AGENTS.md identifies "rediscovering known AMPs" as a primary + threat. This rule is permanent. + + - id: "NO_DANGEROUS_PATHGEN_OPTIMIZATION" + description: > + Recalibration may NOT add, restore, or strengthen any objective + function, scorer weight, or filter that targets pathogen + enhancement, immune evasion, virulence increase, or delivery of + harmful biological agents. + rationale: > + AGENTS.md identifies "dangerous operational content" as the + primary dual-use failure mode. This rule is permanent. + + - id: "NO_POST_HOC_SUCCESS_REDEFINITION" + description: > + Recalibration may NOT change the success criteria, gate thresholds, + or minimum AUROC floor that were used to select the cohort being + analysed. + rationale: > + Cherry-picking the success definition to fit the observed data is + the textbook failure mode this policy exists to prevent. + +# ---------------------------------------------------------------------- +# Rate limits -- soft caps on how often recalibration may proceed. +# ---------------------------------------------------------------------- +rate_limits: + + - id: "WEIGHT_CHANGE_L1_BUDGET" + description: > + Maximum total L1 distance (sum of absolute weight changes) across + all scoring weights per recalibration event. + threshold: 0.10 + applies_to: "candidate_weights_post_minus_pre_l1" + rationale: > + Prevents large jumps that look like overfitting to a single + batch. Combined with the cooldown rule below, this forces + incremental adjustments whose cumulative drift is auditable. + + - id: "COOLDOWN_DAYS" + description: > + Minimum elapsed days between the date of this intake report and + the date of the previous successful recalibration event for the + same scoring config. + threshold: 14 + applies_to: "previous_recalibration_at" + rationale: > + Time-separated batches expose cumulative drift that back-to-back + batches hide. 14 days is a deliberately conservative floor that + allows at most ~2 recalibration events per month per config. + +# ---------------------------------------------------------------------- +# Required reviewer artefacts for any recalibration to be considered +# scientifically valid. These do not block the gate on their own (the +# gate is binary) but they ARE checked and surfaced in the verdict so +# reviewers cannot claim the gate passed without producing them. +# ---------------------------------------------------------------------- +required_reviewer_artefacts: + + - id: "INTAKE_REPORT_JSON" + description: "Machine-readable intake report produced by calibration-intake." + expected_path: "outputs/calibration_intake.json" + kind: "machine" + + - id: "INTAKE_REPORT_MARKDOWN" + description: "Human-readable intake report for the same cohort." + expected_path: "outputs/calibration_intake.md" + kind: "human" + + - id: "DECISION_LOG_ENTRY" + description: > + A dated, human-authored decision log entry (Markdown or YAML) + explaining why this cohort justifies recalibration and what + proposed weight changes are recommended. + expected_path: "docs/DECISION_LOG_.md" + kind: "human" + +# ---------------------------------------------------------------------- +# Pre-registration lock. The presence of this block is itself checked +# by the validator. Editing this policy without bumping policy_version +# or recording a human decision log entry causes the gate to FAIL. +# ---------------------------------------------------------------------- +locked_changes: + + - rule_id: "MIN_COHORT_SIZE" + locked_at: "2026-07-04" + reason: "Anti-cherry-picking floor below which AUROC is unstable." + + - rule_id: "MIN_POSITIVES_IN_COHORT" + locked_at: "2026-07-04" + reason: "Anti-cherry-picking floor for confirmed active hits." + + - rule_id: "MIN_NEGATIVES_IN_COHORT" + locked_at: "2026-07-04" + reason: "Anti-cherry-picking floor for confirmed inactive results." + + - rule_id: "POSITIVE_CONTROLS_ALL_PASS" + locked_at: "2026-07-04" + reason: "Assay batch validity floor." + + - rule_id: "NEGATIVE_CONTROLS_ALL_PASS" + locked_at: "2026-07-04" + reason: "Assay batch validity floor." + + - rule_id: "NO_ORPHAN_LAB_RESULTS" + locked_at: "2026-07-04" + reason: "Prevents hidden candidate_id mismatches." + + - rule_id: "COHORT_METRICS_AVAILABLE" + locked_at: "2026-07-04" + reason: "Double-defensive guard against acting on insufficient_data." + + - rule_id: "NO_TOXICITY_RELAXATION" + locked_at: "2026-07-04" + reason: "Permanent safety floor (AGENTS.md)." + + - rule_id: "NO_HEMOLYSIS_RELAXATION" + locked_at: "2026-07-04" + reason: "Permanent safety floor (MISSION.md)." + + - rule_id: "NO_NOVELTY_RELAXATION" + locked_at: "2026-07-04" + reason: "Permanent novelty safeguard (AGENTS.md)." + + - rule_id: "NO_DANGEROUS_PATHGEN_OPTIMIZATION" + locked_at: "2026-07-04" + reason: "Permanent dual-use safeguard (AGENTS.md)." + + - rule_id: "NO_POST_HOC_SUCCESS_REDEFINITION" + locked_at: "2026-07-04" + reason: "Permanent anti-cherry-picking safeguard (AGENTS.md)." + +notes: + - "This policy is the human-authored contract that gates recalibration." + - "It does NOT itself trigger any weight update. It only answers the question" + - " 'may recalibration proceed from this intake report?' with a binary verdict" + - " plus a list of reasons when the verdict is no." + - "Bumping policy_version requires writing a dated decision log entry" + - " (docs/DECISION_LOG_.md) that documents what changed and why." + - "Every prohibited_action is permanent and CANNOT be relaxed by any future" + - "policy edit. Removing one would require deleting the entire policy file" + - "and replacing it with a new file at a new path." diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 492b4941..0919b641 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -48,7 +48,7 @@ Potential future packages, only if benchmarked honestly: | Future package | Intended role | |---|---| -| `openamp_foundry.calibration` | learning from assay results and recalibrating scores | +| `openamp_foundry.calibration` | lab-result intake (v0.5.19), pre-registered recalibration policy + gate (v0.5.20); future recalibration engine goes here, gated by `recalibration_gate` | | `openamp_foundry.active_learning` | choosing informative next experiments under uncertainty | ## Threat model @@ -90,6 +90,11 @@ When the project is mature enough, the extended loop should look like: 5. Select a very small assay batch that balances likely winners with high-information probes. 6. Ingest qualified wet-lab outcomes through versioned schemas. 7. Convert raw result JSON into a control-aware review artifact before any recalibration decision. +8. **Pre-registered recalibration gate (v0.5.20)**: evaluate the intake + report against `configs/recalibration_policy.yaml` and obtain a binary + `may_recalibrate` verdict. If `false`, recalibration is forbidden + regardless of what the report says. If `true`, a human reviewer may + propose a weight change that respects every prohibited action. 8. Recalibrate decision rules without rewriting success definitions after the fact. 9. Measure whether the added modeling layer actually reduced wasted experiments. diff --git a/docs/CALIBRATION_POLICY.md b/docs/CALIBRATION_POLICY.md new file mode 100644 index 00000000..f260d1ea --- /dev/null +++ b/docs/CALIBRATION_POLICY.md @@ -0,0 +1,231 @@ +# Recalibration Policy + +> **Purpose:** Define the pre-registered, human-authored policy that gates +> any pipeline recalibration in OpenAMP Foundry. This document explains the +> policy; the machine-readable contract is `configs/recalibration_policy.yaml`. +> +> **Status:** v0.5.20 — ratified alongside the recalibration-gate CLI. + +--- + +## Why this exists + +Calibration intake (`openamp-foundry calibration-intake`, v0.5.19) joins +computational predictions with validated lab results and produces a +per-candidate report. It is intentionally **descriptive only**: it does +NOT trigger any weight update, scoring change, or selection rule +modification. + +The next step in the wet-lab compression roadmap is to **act on** those +intake reports when the evidence is strong enough — and ONLY then. The +most dangerous failure mode is silent recalibration: an agent sees real +wet-lab data, decides the scorer "needs improvement," and rewrites +weights to fit. That is exactly the cherry-picking the project exists +to prevent. + +This policy is the formal gate against silent recalibration. It +encodes: + +- **minimum conditions** — what a cohort must look like before any + recalibration is even considered (cohort size, controls, no orphans, + confirmed actives and inactives); +- **prohibited actions** — permanent safety floors that no recalibration + may relax (toxicity, hemolysis, novelty, pathogen enhancement, success + redefinition); +- **rate limits** — how much weight can change per event and how often + recalibration may occur; +- **required reviewer artefacts** — what humans must produce for a + recalibration to be considered scientifically valid. + +The policy is **machine-readable** (`configs/recalibration_policy.yaml`) +and **machine-enforced** (`openamp_foundry.calibration.policy` + +`openamp_foundry.calibration.recalibration_gate`). The CLI +`openamp-foundry recalibration-gate` evaluates an intake report against +the policy and emits a binary verdict plus a list of reasons. + +--- + +## What the gate actually does + +```bash +# 1. Generate an intake report from a pilot panel + lab results dir +make lab-result-intake PANEL=... RESULTS_DIR=... +# 2. Evaluate the recalibration gate against that report + the policy +make recalibration-gate INTAKE=outputs/calibration_intake.json +``` + +The gate: + +1. Loads and validates `configs/recalibration_policy.yaml`. If a + minimum_condition rule is missing its `locked_changes` entry, OR + any canonical prohibited action is removed, the policy itself is + rejected at load time. +2. Loads the intake report produced by `calibration-intake`. +3. Evaluates every `minimum_conditions` rule against the report. If + ANY rule fails, the verdict is `may_recalibrate=false`. +4. Audits every `prohibited_actions` entry (informational only; + surfaced in the verdict so reviewers see the floors). +5. Evaluates every `rate_limits` entry. When inputs are missing, + status is `unknown`; when thresholds are violated, status is + `exceeded` (and the verdict lists it as a reason). +6. Checks whether each `required_reviewer_artefacts` file exists on + disk. Missing artefacts are surfaced but do not by themselves + block the gate; the human review is the final step. + +Exit code: + +- `0` when `may_recalibrate=true` +- `3` when `may_recalibrate=false` +- `2` when an input is missing or malformed + +--- + +## The minimum conditions + +| Rule | Default | Purpose | +|-------------------------------|---------|------------------------------------------------------| +| `MIN_COHORT_SIZE` | 5 | Anti-cherry-picking floor for cohort size. | +| `MIN_POSITIVES_IN_COHORT` | 2 | At least two confirmed active hits. | +| `MIN_NEGATIVES_IN_COHORT` | 2 | At least two confirmed inactive results. | +| `POSITIVE_CONTROLS_ALL_PASS` | true | No failed positive controls allowed. | +| `NEGATIVE_CONTROLS_ALL_PASS` | true | No failed negative controls allowed. | +| `NO_ORPHAN_LAB_RESULTS` | 0 | All lab results must match a panel candidate. | +| `COHORT_METRICS_AVAILABLE` | false | Cohort metrics must not be flagged insufficient_data. | + +Every rule is also listed in `locked_changes` in the policy YAML. +Removing or relaxing a rule requires bumping `policy_version` AND +writing a dated decision log entry that explains why. + +--- + +## The prohibited actions (permanent floor) + +These rules CANNOT be relaxed by any future policy edit. They duplicate +the non-negotiable safety floors already encoded in `AGENTS.md` and +`MISSION.md`. + +| ID | Why it is permanent | +|-------------------------------------|------------------------------------------------------| +| `NO_TOXICITY_RELAXATION` | AGENTS.md forbids activity-only optimization. | +| `NO_HEMOLYSIS_RELAXATION` | MISSION.md requires dual-use review. | +| `NO_NOVELTY_RELAXATION` | AGENTS.md identifies rediscovering AMPs as a threat.| +| `NO_DANGEROUS_PATHGEN_OPTIMIZATION` | AGENTS.md forbids dangerous operational content. | +| `NO_POST_HOC_SUCCESS_REDEFINITION` | Anti-cherry-picking; success must be fixed in advance.| + +A policy file that omits any of these is rejected at load time. To +remove one entirely would require deleting the entire policy file and +starting from scratch — which is itself auditable in git history. + +--- + +## Rate limits + +| Rule | Default | Purpose | +|----------------------------|---------|-----------------------------------------------| +| `WEIGHT_CHANGE_L1_BUDGET` | 0.10 | Maximum L1 weight change per event. | +| `COOLDOWN_DAYS` | 14 | Minimum days between consecutive recalibrations. | + +These are checked when the corresponding CLI inputs are supplied +(`--weight-l1-distance` and `--previous-recalibration-at`). Without +inputs, the status is `unknown` and the gate still answers the binary +question — but the verdict surfaces that the rate limit was not +evaluable. + +--- + +## Required reviewer artefacts + +| Artefact | Path | Purpose | +|-----------------------|-----------------------------------------|------------------------------------| +| `INTAKE_REPORT_JSON` | `outputs/calibration_intake.json` | Machine-readable intake report. | +| `INTAKE_REPORT_MARKDOWN` | `outputs/calibration_intake.md` | Human-readable intake report. | +| `DECISION_LOG_ENTRY` | `docs/DECISION_LOG_.md` | Human-authored rationale + proposal.| + +These are surfaced in the verdict and counted as `reasons` when missing, +but they do not by themselves block the gate. The human review IS the +final step. + +--- + +## What this policy is NOT + +- **Not a recalibrator.** The gate does not move weights or change + scoring. It only emits a verdict. +- **Not a replacement for human review.** A `may_recalibrate=true` + verdict is a *permission*, not a *command*. The decision to actually + apply a weight change still belongs to a human reviewer with a + documented decision log entry. +- **Not a static barrier.** A future maintainer may relax the cohort + size threshold from 5 to 7 if the evidence justifies it. They MUST + bump `policy_version` and write a decision log entry first. The + validator surfaces missing `locked_changes` entries as load errors. +- **Not a substitute for benchmark honesty.** The gate evaluates + *cohort evidence*, not pipeline calibration health. Benchmark + regressions are caught by `make validate-scoring` and the triage + benchmark, not by this policy. + +--- + +## How to update the policy + +If real wet-lab data justifies a policy change: + +1. Open `configs/recalibration_policy.yaml`. +2. Bump `policy_version` (integer; never decrement). +3. Update `locked_at` to today's ISO date. +4. Update `human_reviewer` to identify the reviewer. +5. Make the rule changes. +6. If a rule is being RELAXED, add a `locked_changes` entry with the + new `locked_at` date and the reason. +7. If a rule is being REMOVED, add a decision log entry + (`docs/DECISION_LOG_.md`) explaining why. The validator will + reject the file until the removal is recorded. +8. If a canonical prohibited action is being removed entirely, STOP. + That change is not allowed. + +The policy validator (`openamp_foundry.calibration.policy`) enforces +steps 5–7 mechanically. The CI gate +(`tests/test_recalibration_gate.py::test_canonical_prohibited_actions_match_policy`) +enforces the canonical prohibited-action list. + +--- + +## How this fits the wet-lab compression roadmap + +| Roadmap step | Status | Where it lives | +|-----------------------------|---------------|---------------------------------------------| +| Lab-result schema | Done (v0.5.18)| `schemas/lab_result.schema.json` | +| Lab-result intake report | Done (v0.5.19)| `openamp_foundry.calibration.intake` | +| **Recalibration policy** | **Done (v0.5.20)** | `configs/recalibration_policy.yaml` + `openamp_foundry.calibration.policy` + `openamp_foundry.calibration.recalibration_gate` | +| Recalibration *engine* | Pending | Future loop, after real lab data arrives | +| Active-learning selection | Pending | Future loop, depends on recalibration engine | + +The recalibration policy is the **gate that protects** the +recalibration engine. Building the engine without the gate would be +silent cherry-picking. Building the gate now — before real data +arrives — is the honest sequence. + +--- + +## Quick command reference + +```bash +# Show policy summary (no evaluation needed) +cat configs/recalibration_policy.yaml | head -40 + +# Run the synthetic example end-to-end +make lab-result-intake-example +make recalibration-gate-example +# Expect exit code 3 (FAIL) because the synthetic cohort is too small +# and one positive control failed. + +# Evaluate a real intake report +make recalibration-gate \ + INTAKE=outputs/calibration_intake.json \ + DATE=2026-08-15 \ + PREV=2026-07-01 \ + L1=0.05 + +# Run the gate in tests +pytest tests/test_recalibration_gate.py -v +``` diff --git a/docs/METRICS_CURRENT.md b/docs/METRICS_CURRENT.md index 66d0e06f..7fc9751d 100644 --- a/docs/METRICS_CURRENT.md +++ b/docs/METRICS_CURRENT.md @@ -5,9 +5,9 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees > with this file, this file wins. Updated whenever benchmark/benchmark config changes. > -> **Last updated:** 2026-07-04 (calibration intake module added) +> **Last updated:** 2026-07-04 (recalibration policy + gate module added) > **New in v0.5.19:** `openamp-foundry calibration-intake` joins pilot panel predictions with lab result actuals. Descriptive only; does NOT trigger recalibration. -> **Pipeline version:** v0.5.x +> **Pipeline version:** v0.5.20 > **Branch:** main --- @@ -716,3 +716,4 @@ Decoys score low on activity. Selective AMPs score moderately on both. | 2026-07-03 | **Feature decomposition benchmark added:** per-feature selective_vs_hemolytic AUROC for all 30 scalar physicochemical features. hydrophobic_fraction is the strongest single discriminative feature (0.6745, CI 0.58-0.77) but is NOT used by the selectivity proxy. 8/30 features have significant signal; 6 of those are unused. Provides actionable diagnostic for why composite scorers fail selective_vs_hemolytic discrimination. | OpenAMP loop | | 2026-07-04 | **Calibration intake module added:** `openamp-foundry calibration-intake` joins a pilot panel CSV with a directory of validated lab result JSON files, produces a per-candidate prediction-vs-actual report with cohort metrics gated by `MIN_COHORT_SIZE=5`. Descriptive only — does NOT trigger recalibration, weight updates, or selection-rule changes. Synthetic example data in `examples/lab_results/` is clearly labeled in every file and in `examples/lab_results/README.md`. 29 new tests; total 1614 passing. | OpenAMP loop | | 2026-06-29 | Initial — expanded benchmark (PR #110) | OpenAMP CI | +| 2026-07-04 | **Recalibration policy + gate module added:** `openamp-foundry recalibration-gate` evaluates a calibration intake report against the pre-registered policy in `configs/recalibration_policy.yaml` and emits a binary `may_recalibrate` verdict. The policy file encodes 7 minimum conditions (cohort size, controls, orphans, positives, negatives, metrics availability), 5 permanent prohibited actions (toxicity, hemolysis, novelty, pathogen enhancement, post-hoc success redefinition), and 2 rate limits (L1 weight budget, cooldown). The validator rejects policy files that omit any canonical prohibited action or any `locked_changes` entry. The gate does NOT trigger weight updates; it is the missing permission layer between v0.5.19 intake and a future recalibration engine. Exit code 0 when `may_recalibrate=true`, 3 when false. 39 new tests; total 1647 passing. See `docs/CALIBRATION_POLICY.md`. | OpenAMP loop | diff --git a/docs/PLAN.md b/docs/PLAN.md index cb8b73a4..de7b39d8 100644 --- a/docs/PLAN.md +++ b/docs/PLAN.md @@ -224,10 +224,14 @@ A microbiologist can review the batch and say whether it is worth assay money. Deliverables: -- lab result ingestion schema; -- hit/failure learning module; -- model update report; -- second-batch selection; +- lab result ingestion schema; **(shipped v0.5.18)** +- lab result intake report that joins predictions with actuals; **(shipped v0.5.19)** +- **pre-registered recalibration policy + gate (shipped v0.5.20)** that + evaluates every intake report against `configs/recalibration_policy.yaml` + and emits a binary `may_recalibrate` verdict before any weight update; +- hit/failure learning module (gated by v0.5.20); +- model update report (gated by v0.5.20); +- second-batch selection (gated by v0.5.20); - public negative-result archive where safe. Exit criterion: @@ -236,6 +240,10 @@ Exit criterion: Second-batch hit rate improves over first-batch hit rate without changing success definitions after the fact. ``` +Phase 5 is deliberately staged: the policy + gate ship before any +recalibration engine exists, so the engine cannot be silently +applied. See `docs/CALIBRATION_POLICY.md` for the exact gate contract. + ### Phase 6 — Scientific credibility Deliverables: diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 5ff79acd..4b7c5499 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -492,3 +492,88 @@ field and in `examples/lab_results/README.md`. It exists solely to exercise the intake workflow end-to-end so future agents and reviewers can verify the code path without real wet-lab data. When real validated lab results arrive, replace the example directory — the pipeline itself does not change. + +## v0.5.20 — Recalibration Policy + Gate ✓ (2026-07-04) + +The v0.5.19 calibration-intake module joins pilot-panel predictions with +validated lab-result actuals and produces a per-candidate review report. +It is intentionally descriptive only. The obvious next step would be to +"act on" those intake reports by adjusting scoring weights. Without a +machine-readable policy that gates that action, the most dangerous +failure mode is silent recalibration: an agent sees real wet-lab data, +decides the scorer "needs improvement," and rewrites weights to fit. +That is the textbook cherry-picking this project exists to prevent. + +This version adds the **gate that protects** any future recalibration. +It is the missing permission layer between v0.5.19 intake and a +recalibration engine that does not yet exist. + +Changes: +- `configs/recalibration_policy.yaml`: human-authored, machine-readable + pre-registered policy. + - 7 `minimum_conditions` (cohort size, controls, orphans, positives, + negatives, metrics availability) — every rule also listed in + `locked_changes` so removal requires a documented decision log entry. + - 5 `prohibited_actions` (toxicity relaxation, hemolysis relaxation, + novelty relaxation, pathogen optimization, post-hoc success + redefinition) — permanent floor, mirrors `AGENTS.md` and + `MISSION.md`. Validator rejects any policy file that drops them. + - 2 `rate_limits` (L1 weight budget 0.10; cooldown 14 days) — evaluated + when the corresponding CLI inputs are supplied; `unknown` status + when not evaluable. + - 3 `required_reviewer_artefacts` (intake JSON, intake Markdown, dated + decision log entry) — surfaced as reasons when missing but not + blocking on their own; the human review IS the final step. + - 12 `locked_changes` entries, one per enforced rule. +- `src/openamp_foundry/calibration/policy.py`: `load_recalibration_policy` + loads, validates, and exposes the policy. Raises `PolicyLoadError` on + missing fields, duplicate ids, unlocked rules, or missing canonical + prohibited actions. +- `src/openamp_foundry/calibration/recalibration_gate.py`: + - `evaluate_recalibration_gate(intake_report, policy, ...)` returns a + `GateVerdict` with `may_recalibrate` (bool), per-rule results, audit, + rate-limit status, reviewer-artefact status, reasons, and summary. + - `write_gate_verdict_json` and `write_gate_verdict_markdown` produce + JSON and Markdown outputs. +- `cli/commands/reports.py`: `_run_recalibration_gate`. +- `cli/main.py`: `recalibration-gate` subcommand registered. + Exit code 0 when `may_recalibrate=true`, 3 when false, 2 on input error. +- `Makefile`: `recalibration-gate-example` and `recalibration-gate` + targets added. +- `tests/test_recalibration_gate.py`: 39 new tests covering policy + loader (happy + every rejection mode), gate evaluator (every minimum + condition), prohibited-action audit, rate-limit status, reviewer- + artefact status, writers, end-to-end CLI smoke. Total 1647 passing. +- `docs/CALIBRATION_POLICY.md`: human-readable overview of the policy, + the gate, the permanent floor, rate limits, and how to update. + +Key honest limitations (must read before relying on the gate): + +1. The gate does NOT trigger any weight update. It only emits a verdict. +2. A `may_recalibrate=true` verdict is a permission, not a command. + The decision to apply a weight change still belongs to a human + reviewer with a dated decision log entry. +3. The gate evaluates cohort evidence, not pipeline calibration health. + Benchmark regressions are caught by `make validate-scoring`, + `make bench-triage`, and the selectivity benchmark — not by this + policy. These checks must keep running independently. +4. The synthetic example correctly yields `may_recalibrate=false` (cohort + size 4 < 5, one positive control failed, all reviewer artefacts + missing). That is the expected outcome on tiny synthetic data and is + itself a useful sanity check that the gate is enforcing the cohort + floor honestly. +5. The five canonical prohibited actions are duplicated from `AGENTS.md` + and `MISSION.md`. The validator rejects a policy file that drops any + of them. Any relaxation of the source documents must happen in + lockstep with the policy file. + +Next-loop candidates that depend on v0.5.20: + +- **Recalibration engine**: implement the actual weight-update code, + gated by this policy. Will be safe to ship because the gate + pre-emptively rejects recalibration attempts that violate the floor. +- **Per-seed recalibration audit**: extend the policy with seed-specific + rules once Wave 1 results are in. +- **Policy version bump workflow**: codify the exact decision-log format + and add CI guard that a `policy_version` bump requires a non-empty + decision log entry dated within the past 30 days. diff --git a/src/openamp_foundry/calibration/policy.py b/src/openamp_foundry/calibration/policy.py new file mode 100644 index 00000000..2d64106a --- /dev/null +++ b/src/openamp_foundry/calibration/policy.py @@ -0,0 +1,408 @@ +"""Pre-registered recalibration policy loader and validator. + +This module is the MACHINE-READABLE CONTRACT that gates any pipeline +recalibration. The accompanying YAML file +``configs/recalibration_policy.yaml`` is the human-authored source of +truth; this module loads, schema-checks, and exposes its rules so that +the ``recalibration_gate`` module can evaluate intake reports against +the policy without silently bypassing it. + +Honesty rules (encoded both in the YAML and enforced here): + +* ``minimum_conditions`` are required. Every ``id`` listed in + ``minimum_conditions`` must also appear in ``locked_changes`` or the + policy file is rejected. +* ``prohibited_actions`` are permanent. Their ``id``s are required to + be present and cannot be removed without bumping ``policy_version`` + AND recording a human decision log entry. The validator rejects a + policy file whose prohibited_actions list shrinks compared to the + canonical baseline (see ``_CANONICAL_PROHIBITED_ACTIONS`` below). +* ``locked_changes`` lists every rule whose presence is enforced. A + missing entry causes the policy to be rejected. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import yaml + + +# Permanent baseline of prohibited actions. These cannot be removed +# from any future policy file without bumping policy_version and +# rewriting this list together with the policy file. They mirror the +# non-negotiable safety floors in AGENTS.md / MISSION.md. +_CANONICAL_PROHIBITED_ACTIONS: tuple[str, ...] = ( + "NO_TOXICITY_RELAXATION", + "NO_HEMOLYSIS_RELAXATION", + "NO_NOVELTY_RELAXATION", + "NO_DANGEROUS_PATHGEN_OPTIMIZATION", + "NO_POST_HOC_SUCCESS_REDEFINITION", +) + + +@dataclass(frozen=True) +class PolicyRule: + """A single rule in the recalibration policy.""" + + id: str + description: str + threshold: Any + applies_to: str + rationale: str + + +@dataclass(frozen=True) +class ProhibitedAction: + """A permanent floor that any recalibration must respect.""" + + id: str + description: str + rationale: str + + +@dataclass(frozen=True) +class RateLimit: + """A soft cap on recalibration cadence or magnitude.""" + + id: str + description: str + threshold: Any + applies_to: str + rationale: str + + +@dataclass(frozen=True) +class ReviewerArtefact: + """An artefact a human reviewer must produce for a recalibration to be considered valid.""" + + id: str + description: str + expected_path: str + kind: str + + +@dataclass(frozen=True) +class LockedChange: + """A change-record entry that the policy file must preserve.""" + + rule_id: str + locked_at: str + reason: str + + +@dataclass(frozen=True) +class RecalibrationPolicy: + """The full recalibration policy loaded from YAML.""" + + policy_version: int + locked_at: str + human_reviewer: str + minimum_conditions: tuple[PolicyRule, ...] + prohibited_actions: tuple[ProhibitedAction, ...] + rate_limits: tuple[RateLimit, ...] + required_reviewer_artefacts: tuple[ReviewerArtefact, ...] + locked_changes: tuple[LockedChange, ...] + notes: tuple[str, ...] + source_path: Path + + def rule_by_id(self, rule_id: str) -> PolicyRule | None: + """Return the minimum_condition rule with this id, or None.""" + + for rule in self.minimum_conditions: + if rule.id == rule_id: + return rule + return None + + def prohibited_by_id(self, action_id: str) -> ProhibitedAction | None: + """Return the prohibited_action with this id, or None.""" + + for action in self.prohibited_actions: + if action.id == action_id: + return action + return None + + def is_locked(self, rule_id: str) -> bool: + """Return True if rule_id appears in locked_changes.""" + + return any(change.rule_id == rule_id for change in self.locked_changes) + + +class PolicyLoadError(ValueError): + """Raised when a policy file fails validation.""" + + +def _require(cond: bool, msg: str) -> None: + """Raise PolicyLoadError if cond is False.""" + + if not cond: + raise PolicyLoadError(msg) + + +def _as_str(value: Any, field_name: str) -> str: + """Coerce value to str or raise.""" + + if not isinstance(value, str): + raise PolicyLoadError( + f"{field_name!r} must be a string, got {type(value).__name__}" + ) + return value + + +def _as_int(value: Any, field_name: str) -> int: + """Coerce value to int or raise.""" + + if isinstance(value, bool) or not isinstance(value, int): + raise PolicyLoadError( + f"{field_name!r} must be an integer, got {type(value).__name__}" + ) + return value + + +def _as_float_or_int(value: Any, field_name: str) -> float | int: + """Coerce value to a numeric scalar or raise.""" + + if isinstance(value, bool) or not isinstance(value, (int, float)): + raise PolicyLoadError( + f"{field_name!r} must be numeric, got {type(value).__name__}" + ) + return value + + +def _as_bool(value: Any, field_name: str) -> bool: + """Coerce value to bool or raise.""" + + if not isinstance(value, bool): + raise PolicyLoadError( + f"{field_name!r} must be a boolean, got {type(value).__name__}" + ) + return value + + +def _parse_rules(raw: Any, label: str) -> tuple[PolicyRule, ...]: + """Parse the minimum_conditions block.""" + + _require(isinstance(raw, list), f"{label!r} must be a list") + parsed: list[PolicyRule] = [] + seen_ids: set[str] = set() + for i, entry in enumerate(raw): + _require(isinstance(entry, dict), f"{label}[{i}] must be a mapping") + rid = _as_str(entry.get("id"), f"{label}[{i}].id") + _require(rid not in seen_ids, f"{label} contains duplicate id: {rid!r}") + seen_ids.add(rid) + threshold = entry.get("threshold") + _require( + threshold is not None, + f"{label}[{i}].threshold is required", + ) + parsed.append( + PolicyRule( + id=rid, + description=_as_str(entry.get("description"), f"{label}[{i}].description"), + threshold=threshold, + applies_to=_as_str(entry.get("applies_to"), f"{label}[{i}].applies_to"), + rationale=_as_str(entry.get("rationale"), f"{label}[{i}].rationale"), + ) + ) + return tuple(parsed) + + +def _parse_prohibited(raw: Any) -> tuple[ProhibitedAction, ...]: + """Parse the prohibited_actions block.""" + + _require(isinstance(raw, list), "'prohibited_actions' must be a list") + parsed: list[ProhibitedAction] = [] + seen_ids: set[str] = set() + for i, entry in enumerate(raw): + _require(isinstance(entry, dict), f"prohibited_actions[{i}] must be a mapping") + aid = _as_str(entry.get("id"), f"prohibited_actions[{i}].id") + _require(aid not in seen_ids, f"prohibited_actions contains duplicate id: {aid!r}") + seen_ids.add(aid) + parsed.append( + ProhibitedAction( + id=aid, + description=_as_str( + entry.get("description"), f"prohibited_actions[{i}].description" + ), + rationale=_as_str( + entry.get("rationale"), f"prohibited_actions[{i}].rationale" + ), + ) + ) + return tuple(parsed) + + +def _parse_rate_limits(raw: Any) -> tuple[RateLimit, ...]: + """Parse the rate_limits block.""" + + _require(isinstance(raw, list), "'rate_limits' must be a list") + parsed: list[RateLimit] = [] + seen_ids: set[str] = set() + for i, entry in enumerate(raw): + _require(isinstance(entry, dict), f"rate_limits[{i}] must be a mapping") + rid = _as_str(entry.get("id"), f"rate_limits[{i}].id") + _require(rid not in seen_ids, f"rate_limits contains duplicate id: {rid!r}") + seen_ids.add(rid) + threshold = entry.get("threshold") + _require(threshold is not None, f"rate_limits[{i}].threshold is required") + parsed.append( + RateLimit( + id=rid, + description=_as_str(entry.get("description"), f"rate_limits[{i}].description"), + threshold=threshold, + applies_to=_as_str(entry.get("applies_to"), f"rate_limits[{i}].applies_to"), + rationale=_as_str(entry.get("rationale"), f"rate_limits[{i}].rationale"), + ) + ) + return tuple(parsed) + + +def _parse_reviewer_artefacts(raw: Any) -> tuple[ReviewerArtefact, ...]: + """Parse the required_reviewer_artefacts block.""" + + _require(isinstance(raw, list), "'required_reviewer_artefacts' must be a list") + parsed: list[ReviewerArtefact] = [] + seen_ids: set[str] = set() + for i, entry in enumerate(raw): + _require(isinstance(entry, dict), f"required_reviewer_artefacts[{i}] must be a mapping") + rid = _as_str(entry.get("id"), f"required_reviewer_artefacts[{i}].id") + _require( + rid not in seen_ids, + f"required_reviewer_artefacts contains duplicate id: {rid!r}", + ) + seen_ids.add(rid) + parsed.append( + ReviewerArtefact( + id=rid, + description=_as_str( + entry.get("description"), + f"required_reviewer_artefacts[{i}].description", + ), + expected_path=_as_str( + entry.get("expected_path"), + f"required_reviewer_artefacts[{i}].expected_path", + ), + kind=_as_str( + entry.get("kind"), + f"required_reviewer_artefacts[{i}].kind", + ), + ) + ) + return tuple(parsed) + + +def _parse_locked_changes(raw: Any) -> tuple[LockedChange, ...]: + """Parse the locked_changes block.""" + + _require(isinstance(raw, list), "'locked_changes' must be a list") + parsed: list[LockedChange] = [] + seen_ids: set[str] = set() + for i, entry in enumerate(raw): + _require(isinstance(entry, dict), f"locked_changes[{i}] must be a mapping") + rid = _as_str(entry.get("rule_id"), f"locked_changes[{i}].rule_id") + _require(rid not in seen_ids, f"locked_changes contains duplicate rule_id: {rid!r}") + seen_ids.add(rid) + parsed.append( + LockedChange( + rule_id=rid, + locked_at=_as_str( + entry.get("locked_at"), f"locked_changes[{i}].locked_at" + ), + reason=_as_str( + entry.get("reason"), f"locked_changes[{i}].reason" + ), + ) + ) + return tuple(parsed) + + +def load_recalibration_policy(path: str | Path) -> RecalibrationPolicy: + """Load and validate a recalibration policy file. + + Args: + path: Path to the policy YAML file. + + Returns: + A validated RecalibrationPolicy. + + Raises: + PolicyLoadError: if the file is missing required fields, has + duplicate ids, or attempts to remove a canonical + prohibited_action. + """ + + p = Path(path) + _require(p.exists(), f"policy file does not exist: {p}") + + with p.open() as f: + raw = yaml.safe_load(f) + _require(isinstance(raw, dict), "policy root must be a mapping") + + policy_version = _as_int(raw.get("policy_version"), "policy_version") + _require(policy_version >= 1, "policy_version must be >= 1") + locked_at = _as_str(raw.get("locked_at"), "locked_at") + human_reviewer = _as_str(raw.get("human_reviewer"), "human_reviewer") + + minimum_conditions = _parse_rules( + raw.get("minimum_conditions", []), "minimum_conditions" + ) + prohibited_actions = _parse_prohibited(raw.get("prohibited_actions", [])) + rate_limits = _parse_rate_limits(raw.get("rate_limits", [])) + reviewer_artefacts = _parse_reviewer_artefacts( + raw.get("required_reviewer_artefacts", []) + ) + locked_changes = _parse_locked_changes(raw.get("locked_changes", [])) + + # Enforce that every minimum_condition rule is also locked. + locked_rule_ids = {change.rule_id for change in locked_changes} + for rule in minimum_conditions: + _require( + rule.id in locked_rule_ids, + f"minimum_condition rule {rule.id!r} is not present in locked_changes", + ) + + # Enforce that the canonical prohibited_actions are all present. + present_prohibited = {action.id for action in prohibited_actions} + for canonical in _CANONICAL_PROHIBITED_ACTIONS: + _require( + canonical in present_prohibited, + ( + f"prohibited_action {canonical!r} is missing from policy file. " + "This is a permanent safety floor and cannot be removed." + ), + ) + + # Enforce that each prohibited_action is locked. + for action in prohibited_actions: + _require( + action.id in locked_rule_ids, + f"prohibited_action {action.id!r} is not present in locked_changes", + ) + + notes_raw = raw.get("notes", []) + _require(isinstance(notes_raw, list), "'notes' must be a list") + notes: list[str] = [] + for i, n in enumerate(notes_raw): + _require(isinstance(n, str), f"notes[{i}] must be a string") + notes.append(n) + + return RecalibrationPolicy( + policy_version=policy_version, + locked_at=locked_at, + human_reviewer=human_reviewer, + minimum_conditions=minimum_conditions, + prohibited_actions=prohibited_actions, + rate_limits=rate_limits, + required_reviewer_artefacts=reviewer_artefacts, + locked_changes=locked_changes, + notes=tuple(notes), + source_path=p, + ) + + +def canonical_prohibited_action_ids() -> tuple[str, ...]: + """Return the immutable tuple of canonical prohibited-action ids.""" + + return _CANONICAL_PROHIBITED_ACTIONS diff --git a/src/openamp_foundry/calibration/recalibration_gate.py b/src/openamp_foundry/calibration/recalibration_gate.py new file mode 100644 index 00000000..d7a85282 --- /dev/null +++ b/src/openamp_foundry/calibration/recalibration_gate.py @@ -0,0 +1,694 @@ +"""Recalibration gate: evaluate a calibration-intake report against a policy. + +This module answers the binary question: + + "May recalibration proceed from this intake report?" + +It does NOT trigger any weight update, scoring change, or selection +rule modification. It only emits a ``GateVerdict`` that downstream +code (or a human reviewer) must inspect before any recalibration is +allowed to take effect. + +Design notes: + +* Every ``minimum_condition`` rule in the policy is checked against + the intake report. If ANY rule fails, the verdict is + ``may_recalibrate=False``. +* The prohibited-action audit is informational only -- it does not + block the gate, because no concrete weight changes have been + proposed yet. The audit is surfaced so reviewers see which safety + floors they would be required to respect if they propose any change. +* Rate-limit status is also informational only. The gate does not + itself know when the previous recalibration ran; that input must be + supplied by the caller via ``previous_recalibration_at`` and + ``candidate_weight_l1_distance``. If those are ``None``, the + corresponding status is "unknown" and the verdict still passes the + minimum-conditions check, but the verdict carries a clear note. +* Reviewer-artefact status reports which required artefacts are + present on disk. Missing artefacts are surfaced as warnings but do + not by themselves block the gate; the human review is the final + step. This is documented in the policy notes. +""" + +from __future__ import annotations + +import json +from dataclasses import asdict, dataclass +from datetime import date, datetime +from pathlib import Path +from typing import Any + +from openamp_foundry.calibration.policy import ( + PolicyRule, + RecalibrationPolicy, + ReviewerArtefact, +) + + +@dataclass(frozen=True) +class RuleResult: + """Outcome of evaluating a single minimum_condition rule.""" + + rule_id: str + passed: bool + observed: Any + threshold: Any + reason: str + + +@dataclass(frozen=True) +class ProhibitedActionAudit: + """Whether a prohibited action is in force.""" + + action_id: str + description: str + in_force: bool + + +@dataclass(frozen=True) +class RateLimitStatus: + """Status of a single rate-limit rule.""" + + rule_id: str + observed: Any + threshold: Any + status: str # "ok" | "exceeded" | "unknown" + note: str + + +@dataclass(frozen=True) +class ReviewerArtefactStatus: + """Status of a single required reviewer artefact.""" + + artefact_id: str + expected_path: str + present: bool + resolved_path: str | None + note: str + + +@dataclass(frozen=True) +class GateVerdict: + """The output of evaluating the recalibration gate.""" + + may_recalibrate: bool + policy_version: int + policy_locked_at: str + intake_report_path: str + n_panel_candidates: int + n_matched_candidates: int + n_lab_results: int + rule_results: tuple[RuleResult, ...] + prohibited_action_audit: tuple[ProhibitedActionAudit, ...] + rate_limit_status: tuple[RateLimitStatus, ...] + reviewer_artefact_status: tuple[ReviewerArtefactStatus, ...] + reasons: tuple[str, ...] + summary: str + + def to_dict(self) -> dict[str, Any]: + """Return a JSON-serialisable dict representation.""" + + return { + "may_recalibrate": self.may_recalibrate, + "policy_version": self.policy_version, + "policy_locked_at": self.policy_locked_at, + "intake_report_path": self.intake_report_path, + "n_panel_candidates": self.n_panel_candidates, + "n_matched_candidates": self.n_matched_candidates, + "n_lab_results": self.n_lab_results, + "rule_results": [asdict(r) for r in self.rule_results], + "prohibited_action_audit": [asdict(a) for a in self.prohibited_action_audit], + "rate_limit_status": [asdict(s) for s in self.rate_limit_status], + "reviewer_artefact_status": [ + asdict(s) for s in self.reviewer_artefact_status + ], + "reasons": list(self.reasons), + "summary": self.summary, + } + + +def _cohort_n_positives_and_negatives(report: dict[str, Any]) -> tuple[int, int]: + """Count confirmed positives and negatives in the joined cohort. + + A candidate contributes to ``n_positive_lab`` if it has + ``has_lab.active_mic == True``. Otherwise (when ``has_lab`` is a + dict and ``active_mic`` is False) it contributes to + ``n_negative_lab``. Candidates with ``has_lab is None`` are + excluded from both counts. + """ + + rows = report.get("per_candidate_joined", []) + pos = 0 + neg = 0 + for row in rows: + has_lab = row.get("has_lab") + if not isinstance(has_lab, dict): + continue + if has_lab.get("active_mic") is True: + pos += 1 + elif has_lab.get("active_mic") is False: + neg += 1 + return pos, neg + + +def _evaluate_rule( + rule: PolicyRule, + report: dict[str, Any], + cohort_pos: int, + cohort_neg: int, +) -> RuleResult: + """Apply one minimum_condition rule against the intake report.""" + + if rule.id == "MIN_COHORT_SIZE": + observed = report.get("n_matched_candidates", 0) + passed = observed >= rule.threshold + return RuleResult( + rule_id=rule.id, + passed=passed, + observed=observed, + threshold=rule.threshold, + reason=( + f"matched cohort size {observed} >= threshold {rule.threshold}" + if passed + else ( + f"matched cohort size {observed} below threshold " + f"{rule.threshold}; recalibration forbidden" + ) + ), + ) + + if rule.id == "MIN_POSITIVES_IN_COHORT": + observed = cohort_pos + passed = observed >= rule.threshold + return RuleResult( + rule_id=rule.id, + passed=passed, + observed=observed, + threshold=rule.threshold, + reason=( + f"confirmed positives {observed} >= threshold {rule.threshold}" + if passed + else ( + f"only {observed} confirmed active candidate(s); need at " + f"least {rule.threshold} to learn anything useful about activity" + ) + ), + ) + + if rule.id == "MIN_NEGATIVES_IN_COHORT": + observed = cohort_neg + passed = observed >= rule.threshold + return RuleResult( + rule_id=rule.id, + passed=passed, + observed=observed, + threshold=rule.threshold, + reason=( + f"confirmed negatives {observed} >= threshold {rule.threshold}" + if passed + else ( + f"only {observed} confirmed inactive candidate(s); need at " + f"least {rule.threshold} to measure false-positive rate" + ) + ), + ) + + if rule.id == "POSITIVE_CONTROLS_ALL_PASS": + control_failures = report.get("control_failures", []) or [] + offending = [ + cf for cf in control_failures if not cf.get("positive_control_passed", True) + ] + passed = len(offending) == 0 + return RuleResult( + rule_id=rule.id, + passed=passed, + observed=( + "all passed" if passed + else f"{len(offending)} failed positive control(s)" + ), + threshold=rule.threshold, + reason=( + "all positive controls passed" + if passed + else ( + f"{len(offending)} positive control(s) failed; recalibration " + "forbidden until all positive controls pass" + ) + ), + ) + + if rule.id == "NEGATIVE_CONTROLS_ALL_PASS": + control_failures = report.get("control_failures", []) or [] + offending = [ + cf for cf in control_failures if not cf.get("negative_control_passed", True) + ] + passed = len(offending) == 0 + return RuleResult( + rule_id=rule.id, + passed=passed, + observed=( + "all passed" if passed + else f"{len(offending)} failed negative control(s)" + ), + threshold=rule.threshold, + reason=( + "all negative controls passed" + if passed + else ( + f"{len(offending)} negative control(s) failed; recalibration " + "forbidden until all negative controls pass" + ) + ), + ) + + if rule.id == "NO_ORPHAN_LAB_RESULTS": + observed = report.get("n_orphan_lab_results", 0) + passed = observed <= rule.threshold + return RuleResult( + rule_id=rule.id, + passed=passed, + observed=observed, + threshold=rule.threshold, + reason=( + f"no orphan lab results ({observed} <= {rule.threshold})" + if passed + else ( + f"{observed} orphan lab result(s) detected; these are results " + "whose candidate_id is not in the pilot panel" + ) + ), + ) + + if rule.id == "COHORT_METRICS_AVAILABLE": + cohort_metrics = report.get("cohort_metrics", {}) or {} + activity = cohort_metrics.get("activity_vs_active_mic", {}) or {} + insufficient = bool(activity.get("insufficient_data", True)) + passed = insufficient is rule.threshold # i.e. insufficient_data == False + return RuleResult( + rule_id=rule.id, + passed=passed, + observed=("insufficient" if insufficient else "available"), + threshold=rule.threshold, + reason=( + "cohort metrics are available (not flagged insufficient_data)" + if passed + else ( + "cohort metrics are flagged insufficient_data; cannot " + "use them for recalibration" + ) + ), + ) + + # Unknown rule id -- treat as failed so reviewers notice the problem. + return RuleResult( + rule_id=rule.id, + passed=False, + observed=None, + threshold=rule.threshold, + reason=f"unknown minimum_condition rule id: {rule.id!r}", + ) + + +def _evaluate_prohibited_actions( + policy: RecalibrationPolicy, +) -> tuple[ProhibitedActionAudit, ...]: + """Return one audit row per prohibited_action, all in_force=True.""" + + return tuple( + ProhibitedActionAudit( + action_id=action.id, + description=action.description.strip(), + in_force=True, + ) + for action in policy.prohibited_actions + ) + + +def _parse_iso_date(value: str) -> date | None: + """Parse an ISO date string, accepting ``YYYY-MM-DD`` only.""" + + try: + return datetime.strptime(value, "%Y-%m-%d").date() + except (TypeError, ValueError): + return None + + +def _evaluate_rate_limits( + policy: RecalibrationPolicy, + *, + intake_report_date: str | None, + previous_recalibration_at: str | None, + candidate_weight_l1_distance: float | None, +) -> tuple[RateLimitStatus, ...]: + """Evaluate rate-limit rules. Unknown inputs become status='unknown'.""" + + out: list[RateLimitStatus] = [] + for limit in policy.rate_limits: + if limit.id == "COOLDOWN_DAYS": + if not intake_report_date or not previous_recalibration_at: + out.append( + RateLimitStatus( + rule_id=limit.id, + observed=None, + threshold=limit.threshold, + status="unknown", + note=( + "cooldown not evaluable: intake_report_date or " + "previous_recalibration_at missing" + ), + ) + ) + continue + d_intake = _parse_iso_date(intake_report_date) + d_prev = _parse_iso_date(previous_recalibration_at) + if d_intake is None or d_prev is None: + out.append( + RateLimitStatus( + rule_id=limit.id, + observed=None, + threshold=limit.threshold, + status="unknown", + note="could not parse one of the dates as YYYY-MM-DD", + ) + ) + continue + days = (d_intake - d_prev).days + passed = days >= limit.threshold + out.append( + RateLimitStatus( + rule_id=limit.id, + observed=days, + threshold=limit.threshold, + status="ok" if passed else "exceeded", + note=( + f"days since previous recalibration: {days} " + f"(threshold {limit.threshold})" + ), + ) + ) + continue + + if limit.id == "WEIGHT_CHANGE_L1_BUDGET": + if candidate_weight_l1_distance is None: + out.append( + RateLimitStatus( + rule_id=limit.id, + observed=None, + threshold=limit.threshold, + status="unknown", + note=( + "weight-change L1 budget not evaluable: " + "candidate_weight_l1_distance missing" + ), + ) + ) + continue + passed = candidate_weight_l1_distance <= limit.threshold + out.append( + RateLimitStatus( + rule_id=limit.id, + observed=candidate_weight_l1_distance, + threshold=limit.threshold, + status="ok" if passed else "exceeded", + note=( + f"proposed L1 weight distance: {candidate_weight_l1_distance:.4f} " + f"(budget {limit.threshold})" + ), + ) + ) + continue + + # Unknown rate-limit id -- mark as unknown. + out.append( + RateLimitStatus( + rule_id=limit.id, + observed=None, + threshold=limit.threshold, + status="unknown", + note=f"unknown rate_limit rule id: {limit.id!r}", + ) + ) + + return tuple(out) + + +def _resolve_artefact_path(template: str, *, project_root: Path | None) -> str: + """Resolve a path template, substituting YYYY-MM-DD if possible. + + The current date is substituted only when the literal substring + ```` appears. If no project root is supplied we use the + current working directory. + """ + + today = date.today().isoformat() + rendered = template.replace("", today) + root = project_root if project_root is not None else Path.cwd() + candidate = root / rendered + return str(candidate) + + +def _evaluate_reviewer_artefacts( + artefacts: tuple[ReviewerArtefact, ...], + *, + project_root: Path | None, +) -> tuple[ReviewerArtefactStatus, ...]: + """Check which required reviewer artefacts already exist on disk.""" + + out: list[ReviewerArtefactStatus] = [] + for art in artefacts: + path = _resolve_artefact_path(art.expected_path, project_root=project_root) + p = Path(path) + present = p.exists() + note = ( + f"present at {path}" if present else f"missing (expected {path})" + ) + out.append( + ReviewerArtefactStatus( + artefact_id=art.id, + expected_path=path, + present=present, + resolved_path=path if present else None, + note=note, + ) + ) + return tuple(out) + + +def evaluate_recalibration_gate( + intake_report: dict[str, Any], + policy: RecalibrationPolicy, + *, + intake_report_date: str | None = None, + previous_recalibration_at: str | None = None, + candidate_weight_l1_distance: float | None = None, + project_root: Path | None = None, +) -> GateVerdict: + """Evaluate the recalibration gate against an intake report and policy. + + Args: + intake_report: Output of + ``openamp_foundry.calibration.intake.build_calibration_intake_report``. + policy: Loaded ``RecalibrationPolicy`` from the policy YAML file. + intake_report_date: ISO ``YYYY-MM-DD`` string for the date of the + intake report (used to evaluate the COOLDOWN_DAYS rate limit). + previous_recalibration_at: ISO ``YYYY-MM-DD`` string for the + previous successful recalibration event (or ``None`` if no + prior recalibration has occurred for this scoring config). + candidate_weight_l1_distance: Optional float for the L1 distance + between current and proposed scoring weights. ``None`` means + no concrete weight proposal has been submitted yet. + project_root: Optional path to use for resolving relative + reviewer-artefact paths. Defaults to the current working + directory. + + Returns: + A ``GateVerdict`` whose ``may_recalibrate`` field is ``True`` only + when EVERY minimum_condition rule passes. + """ + + cohort_pos, cohort_neg = _cohort_n_positives_and_negatives(intake_report) + + rule_results = tuple( + _evaluate_rule(rule, intake_report, cohort_pos, cohort_neg) + for rule in policy.minimum_conditions + ) + + prohibited_audit = _evaluate_prohibited_actions(policy) + + rate_status = _evaluate_rate_limits( + policy, + intake_report_date=intake_report_date, + previous_recalibration_at=previous_recalibration_at, + candidate_weight_l1_distance=candidate_weight_l1_distance, + ) + + artefact_status = _evaluate_reviewer_artefacts( + policy.required_reviewer_artefacts, project_root=project_root + ) + + failed_rules = [r for r in rule_results if not r.passed] + may_recalibrate = len(failed_rules) == 0 + + reasons: list[str] = [] + for r in failed_rules: + reasons.append(f"{r.rule_id}: {r.reason}") + for s in rate_status: + if s.status == "exceeded": + reasons.append(f"{s.rule_id}: {s.note}") + for s in artefact_status: + if not s.present: + reasons.append( + f"reviewer artefact missing: {s.artefact_id} ({s.expected_path})" + ) + + summary = ( + "PASS: all minimum conditions satisfied; recalibration may proceed subject " + "to human reviewer approval and rate-limit warnings above." + if may_recalibrate + else ( + "FAIL: " + f"{len(failed_rules)} minimum condition(s) failed; recalibration " + "forbidden until each is resolved." + ) + ) + + return GateVerdict( + may_recalibrate=may_recalibrate, + policy_version=policy.policy_version, + policy_locked_at=policy.locked_at, + intake_report_path=str(intake_report.get("panel_csv", "")), + n_panel_candidates=intake_report.get("n_panel_candidates", 0), + n_matched_candidates=intake_report.get("n_matched_candidates", 0), + n_lab_results=intake_report.get("n_lab_results", 0), + rule_results=rule_results, + prohibited_action_audit=prohibited_audit, + rate_limit_status=rate_status, + reviewer_artefact_status=artefact_status, + reasons=tuple(reasons), + summary=summary, + ) + + +def write_gate_verdict_json( + verdict: GateVerdict, + out_path: str | Path, +) -> Path: + """Write a GateVerdict to JSON.""" + + p = Path(out_path) + p.parent.mkdir(parents=True, exist_ok=True) + with p.open("w") as f: + json.dump(verdict.to_dict(), f, indent=2, sort_keys=False) + f.write("\n") + return p + + +def write_gate_verdict_markdown( + verdict: GateVerdict, + out_path: str | Path, + *, + intake_report_path: str, + policy_path: str, +) -> Path: + """Write a GateVerdict as a human-readable Markdown report.""" + + p = Path(out_path) + p.parent.mkdir(parents=True, exist_ok=True) + lines: list[str] = [] + lines.append("# Recalibration Gate Verdict") + lines.append("") + lines.append("This artefact is the machine-readable output of the OpenAMP") + lines.append("recalibration gate. It is descriptive only -- it does NOT trigger") + lines.append("any weight update or scoring change. A human reviewer must read") + lines.append("the verdict and any matching decision log entry before any") + lines.append("recalibration is allowed to take effect.") + lines.append("") + lines.append(f"- Policy version: **{verdict.policy_version}**") + lines.append(f"- Policy locked at: **{verdict.policy_locked_at}**") + lines.append(f"- Intake report: `{intake_report_path}`") + lines.append(f"- Policy file: `{policy_path}`") + lines.append("") + lines.append("## Verdict") + lines.append("") + lines.append( + f"**may_recalibrate = {str(verdict.may_recalibrate).lower()}**" + ) + lines.append("") + lines.append(verdict.summary) + lines.append("") + lines.append("## Cohort summary") + lines.append("") + lines.append(f"- Panel candidates: {verdict.n_panel_candidates}") + lines.append(f"- Lab results: {verdict.n_lab_results}") + lines.append(f"- Matched candidates: {verdict.n_matched_candidates}") + lines.append("") + lines.append("## Minimum conditions") + lines.append("") + lines.append("| Rule | Passed | Observed | Threshold | Reason |") + lines.append("|------|--------|----------|-----------|--------|") + for r in verdict.rule_results: + lines.append( + f"| {r.rule_id} | " + f"{'PASS' if r.passed else 'FAIL'} | " + f"`{r.observed}` | `{r.threshold}` | {r.reason} |" + ) + lines.append("") + if verdict.reasons: + lines.append("## Reasons (blockers)") + lines.append("") + for reason in verdict.reasons: + lines.append(f"- {reason}") + lines.append("") + lines.append("## Prohibited actions (permanent floor)") + lines.append("") + lines.append("Any proposed recalibration MUST respect every entry below.") + lines.append("") + for a in verdict.prohibited_action_audit: + lines.append(f"### {a.action_id}") + lines.append("") + lines.append(a.description) + lines.append("") + lines.append("In force: **yes** (this rule cannot be relaxed by future policy edits).") + lines.append("") + lines.append("## Rate-limit status") + lines.append("") + if not verdict.rate_limit_status: + lines.append("_No rate-limit rules defined._") + lines.append("") + else: + lines.append("| Rule | Status | Observed | Threshold | Note |") + lines.append("|------|--------|----------|-----------|------|") + for s in verdict.rate_limit_status: + lines.append( + f"| {s.rule_id} | {s.status} | `{s.observed}` | " + f"`{s.threshold}` | {s.note} |" + ) + lines.append("") + lines.append("## Reviewer artefact status") + lines.append("") + if not verdict.reviewer_artefact_status: + lines.append("_No required reviewer artefacts._") + lines.append("") + else: + lines.append("| Artefact | Present | Path | Note |") + lines.append("|----------|---------|------|------|") + for s in verdict.reviewer_artefact_status: + lines.append( + f"| {s.artefact_id} | " + f"{'yes' if s.present else 'no'} | " + f"`{s.expected_path}` | {s.note} |" + ) + lines.append("") + lines.append("## How to act on this verdict") + lines.append("") + lines.append("- If `may_recalibrate=true`, a human reviewer MAY propose a") + lines.append(" weight change. The proposal MUST respect every prohibited") + lines.append(" action listed above and MUST stay within the weight-change") + lines.append(" L1 budget. A decision log entry is required.") + lines.append("- If `may_recalibrate=false`, no weight update is allowed") + lines.append(" from this intake report. Reasons above must be resolved") + lines.append(" (e.g. wait for more data, fix control failures, document") + lines.append(" orphan results) and a new intake report generated.") + lines.append("") + p.write_text("\n".join(lines)) + return p diff --git a/src/openamp_foundry/cli/commands/reports.py b/src/openamp_foundry/cli/commands/reports.py index 06df060d..a45c393e 100644 --- a/src/openamp_foundry/cli/commands/reports.py +++ b/src/openamp_foundry/cli/commands/reports.py @@ -659,3 +659,123 @@ def _run_calibration_intake(args: argparse.Namespace) -> int: ) ) return 0 + + +def _run_recalibration_gate(args: argparse.Namespace) -> int: + """Evaluate the recalibration gate against an intake report + policy.""" + import json + from pathlib import Path + + from openamp_foundry.calibration.policy import load_recalibration_policy + from openamp_foundry.calibration.recalibration_gate import ( + evaluate_recalibration_gate, + write_gate_verdict_json, + write_gate_verdict_markdown, + ) + + intake_path = Path(args.intake_report) + if not intake_path.exists(): + print( + json.dumps( + { + "status": "error", + "error": f"intake report not found: {intake_path}", + }, + indent=2, + ) + ) + return 2 + + with intake_path.open() as f: + intake_report = json.load(f) + + try: + policy = load_recalibration_policy(args.policy) + except Exception as exc: # noqa: BLE001 + print( + json.dumps( + { + "status": "error", + "error": f"failed to load policy {args.policy!r}: {exc}", + }, + indent=2, + ) + ) + return 2 + + previous_recalibration_at = args.previous_recalibration_at + candidate_weight_l1_distance = args.weight_l1_distance + if candidate_weight_l1_distance is not None: + try: + candidate_weight_l1_distance = float(candidate_weight_l1_distance) + except (TypeError, ValueError): + print( + json.dumps( + { + "status": "error", + "error": ( + "weight-l1-distance must be a float, got " + f"{args.weight_l1_distance!r}" + ), + }, + indent=2, + ) + ) + return 2 + + project_root = Path(args.project_root) if args.project_root else Path.cwd() + + verdict = evaluate_recalibration_gate( + intake_report, + policy, + intake_report_date=args.intake_report_date, + previous_recalibration_at=previous_recalibration_at, + candidate_weight_l1_distance=candidate_weight_l1_distance, + project_root=project_root, + ) + + if args.out_json: + write_gate_verdict_json(verdict, args.out_json) + if args.out_md: + write_gate_verdict_markdown( + verdict, + args.out_md, + intake_report_path=str(intake_path), + policy_path=str(args.policy), + ) + + # Avoid dumping nested arrays into the CLI stdout twice. Keep stdout concise. + cli_summary = { + "status": "ok", + "may_recalibrate": verdict.may_recalibrate, + "policy_version": verdict.policy_version, + "policy_locked_at": verdict.policy_locked_at, + "intake_report": str(intake_path), + "policy": str(args.policy), + "n_panel_candidates": verdict.n_panel_candidates, + "n_lab_results": verdict.n_lab_results, + "n_matched_candidates": verdict.n_matched_candidates, + "rule_results": [ + {"rule_id": r.rule_id, "passed": r.passed, "observed": r.observed, + "threshold": r.threshold, "reason": r.reason} + for r in verdict.rule_results + ], + "rate_limit_status": [ + {"rule_id": s.rule_id, "status": s.status, "observed": s.observed, + "threshold": s.threshold, "note": s.note} + for s in verdict.rate_limit_status + ], + "reviewer_artefact_status": [ + {"artefact_id": s.artefact_id, "present": s.present, + "expected_path": s.expected_path, "note": s.note} + for s in verdict.reviewer_artefact_status + ], + "prohibited_action_count": len(verdict.prohibited_action_audit), + "reasons": list(verdict.reasons), + "summary": verdict.summary, + "out_json": args.out_json, + "out_md": args.out_md, + } + print(json.dumps(cli_summary, indent=2)) + # Exit code: 0 if may_recalibrate, 3 otherwise. + return 0 if verdict.may_recalibrate else 3 diff --git a/src/openamp_foundry/cli/main.py b/src/openamp_foundry/cli/main.py index 3b5df867..acd7f523 100644 --- a/src/openamp_foundry/cli/main.py +++ b/src/openamp_foundry/cli/main.py @@ -5,7 +5,7 @@ from openamp_foundry.cli.commands.selection import _run_pilot_panel, _run_pilot_confident, _run_diversity_check from openamp_foundry.cli.commands.external import _run_external_predict, _run_external_consensus from openamp_foundry.cli.commands.qc import _run_synthesis_order, _run_presynth_qc -from openamp_foundry.cli.commands.reports import _run_reviewer_questionnaire, _run_ip_report, _run_batch_pack, _run_gold_standard, _run_novelty_check_broad, _run_lab_result_report, _run_calibration_intake +from openamp_foundry.cli.commands.reports import _run_reviewer_questionnaire, _run_ip_report, _run_batch_pack, _run_gold_standard, _run_novelty_check_broad, _run_lab_result_report, _run_calibration_intake, _run_recalibration_gate from openamp_foundry.cli.commands.gates import _run_gate_check import argparse @@ -689,6 +689,71 @@ def build_parser() -> argparse.ArgumentParser: help="Optional output path for markdown calibration intake summary.", ) + recalibration_gate = sub.add_parser( + "recalibration-gate", + help=( + "Evaluate the recalibration gate against a calibration intake " + "report and the pre-registered policy. Descriptive only -- does " + "NOT trigger any weight update or scoring change. Returns " + "may_recalibrate=true only when every minimum_condition rule " + "passes. Exit code is 0 when may_recalibrate is true and 3 " + "otherwise." + ), + ) + recalibration_gate.add_argument( + "--intake-report", + required=True, + help="Path to calibration-intake JSON (output of `calibration-intake`).", + ) + recalibration_gate.add_argument( + "--policy", + default="configs/recalibration_policy.yaml", + help="Path to the pre-registered recalibration policy YAML.", + ) + recalibration_gate.add_argument( + "--intake-report-date", + default=None, + help=( + "ISO date (YYYY-MM-DD) for the intake report; used to evaluate " + "the COOLDOWN_DAYS rate limit." + ), + ) + recalibration_gate.add_argument( + "--previous-recalibration-at", + default=None, + help=( + "ISO date (YYYY-MM-DD) of the previous successful recalibration " + "for the same scoring config. Omit if no prior recalibration." + ), + ) + recalibration_gate.add_argument( + "--weight-l1-distance", + default=None, + help=( + "Optional float for the L1 distance between current and proposed " + "scoring weights. When set, the WEIGHT_CHANGE_L1_BUDGET rate " + "limit becomes evaluable." + ), + ) + recalibration_gate.add_argument( + "--project-root", + default=None, + help=( + "Optional path used to resolve relative reviewer-artefact paths. " + "Defaults to the current working directory." + ), + ) + recalibration_gate.add_argument( + "--out-json", + default=None, + help="Optional output path for the gate verdict JSON.", + ) + recalibration_gate.add_argument( + "--out-md", + default=None, + help="Optional output path for the gate verdict Markdown report.", + ) + novelty_broad = sub.add_parser( "novelty-check-broad", help=( @@ -804,6 +869,9 @@ def main(argv: list[str] | None = None) -> int: if args.command == "calibration-intake": return _run_calibration_intake(args) + if args.command == "recalibration-gate": + return _run_recalibration_gate(args) + if args.command == "synthesis-order": return _run_synthesis_order(args) diff --git a/tests/test_recalibration_gate.py b/tests/test_recalibration_gate.py new file mode 100644 index 00000000..3325f840 --- /dev/null +++ b/tests/test_recalibration_gate.py @@ -0,0 +1,752 @@ +"""Tests for the recalibration policy + gate workflow. + +Verifies: + - Policy YAML loads, validates, and exposes its rules + - Removing a canonical prohibited_action causes the policy to be rejected + - Unlocked rules cause the policy to be rejected + - Duplicate rule ids are rejected + - Type errors in the YAML are surfaced with helpful messages + - The gate produces a binary verdict from the policy + intake report + - Every minimum_condition rule is checked against a real intake report + - Prohibited-action audit is always present and in_force=true + - Rate-limit status surfaces UNKNOWN when inputs are missing + - Rate-limit cooldown is enforced when dates are provided + - Reviewer-artefact status reports which files exist + - JSON + Markdown writers produce non-empty, schema-valid output + - The synthetic example yields may_recalibrate=False (cohort too small, + one positive control failed) +""" + +from __future__ import annotations + +import json +import sys +from datetime import date +from pathlib import Path +from textwrap import dedent + +import pytest + +from openamp_foundry.calibration.policy import ( + PolicyLoadError, + canonical_prohibited_action_ids, + load_recalibration_policy, +) +from openamp_foundry.calibration.recalibration_gate import ( + GateVerdict, + RateLimitStatus, + ReviewerArtefactStatus, + RuleResult, + evaluate_recalibration_gate, + write_gate_verdict_json, + write_gate_verdict_markdown, +) + +REPO_ROOT = Path(__file__).resolve().parent.parent +POLICY_PATH = REPO_ROOT / "configs" / "recalibration_policy.yaml" +SYNTHETIC_INTAKE = REPO_ROOT / "outputs" / "calibration_intake_example.json" + + +# ──────────────────────────────────────────────────────────────────── +# Policy loader tests +# ──────────────────────────────────────────────────────────────────── + + +def test_policy_loads_successfully(): + p = load_recalibration_policy(POLICY_PATH) + assert p.policy_version == 1 + assert p.locked_at == "2026-07-04" + assert len(p.minimum_conditions) >= 7 + assert len(p.prohibited_actions) == 5 + assert len(p.locked_changes) == 12 + + +def test_policy_rule_by_id(): + p = load_recalibration_policy(POLICY_PATH) + rule = p.rule_by_id("MIN_COHORT_SIZE") + assert rule is not None + assert rule.threshold == 5 + assert rule.is_locked("MIN_COHORT_SIZE") if hasattr(rule, "is_locked") else True + + +def test_policy_is_locked_true_for_listed_rule(): + p = load_recalibration_policy(POLICY_PATH) + assert p.is_locked("MIN_COHORT_SIZE") is True + + +def test_policy_is_locked_false_for_unlisted_rule(): + p = load_recalibration_policy(POLICY_PATH) + assert p.is_locked("DOES_NOT_EXIST") is False + + +def test_canonical_prohibited_actions_match_policy(): + p = load_recalibration_policy(POLICY_PATH) + canonical = set(canonical_prohibited_action_ids()) + policy_ids = {a.id for a in p.prohibited_actions} + assert canonical.issubset(policy_ids) + + +def test_policy_missing_file_raises(tmp_path): + with pytest.raises(PolicyLoadError, match="does not exist"): + load_recalibration_policy(tmp_path / "nope.yaml") + + +def test_policy_missing_policy_version(tmp_path): + p = tmp_path / "bad.yaml" + p.write_text("locked_at: '2026-01-01'\n") + with pytest.raises(PolicyLoadError, match="policy_version"): + load_recalibration_policy(p) + + +def test_policy_negative_version(tmp_path): + p = tmp_path / "bad.yaml" + p.write_text("policy_version: -1\nlocked_at: '2026-01-01'\n") + with pytest.raises(PolicyLoadError, match="policy_version"): + load_recalibration_policy(p) + + +def test_policy_missing_locked_at(tmp_path): + p = tmp_path / "bad.yaml" + p.write_text("policy_version: 1\n") + with pytest.raises(PolicyLoadError, match="locked_at"): + load_recalibration_policy(p) + + +def test_policy_missing_human_reviewer(tmp_path): + p = tmp_path / "bad.yaml" + p.write_text("policy_version: 1\nlocked_at: '2026-01-01'\n") + with pytest.raises(PolicyLoadError, match="human_reviewer"): + load_recalibration_policy(p) + + +def test_policy_duplicate_rule_id_in_minimum_conditions(tmp_path): + text = dedent( + """ + policy_version: 1 + locked_at: '2026-01-01' + human_reviewer: tester + minimum_conditions: + - id: MIN_COHORT_SIZE + description: x + threshold: 5 + applies_to: r + rationale: x + - id: MIN_COHORT_SIZE + description: x + threshold: 6 + applies_to: r + rationale: x + prohibited_actions: + - id: NO_TOXICITY_RELAXATION + description: x + rationale: x + locked_changes: + - rule_id: MIN_COHORT_SIZE + locked_at: '2026-01-01' + reason: x + - rule_id: NO_TOXICITY_RELAXATION + locked_at: '2026-01-01' + reason: x + """ + ) + p = tmp_path / "bad.yaml" + p.write_text(text) + with pytest.raises(PolicyLoadError, match="duplicate id"): + load_recalibration_policy(p) + + +def test_policy_rule_not_locked(tmp_path): + text = dedent( + """ + policy_version: 1 + locked_at: '2026-01-01' + human_reviewer: tester + minimum_conditions: + - id: NEW_RULE + description: x + threshold: 5 + applies_to: r + rationale: x + prohibited_actions: + - id: NO_TOXICITY_RELAXATION + description: x + rationale: x + locked_changes: + - rule_id: NO_TOXICITY_RELAXATION + locked_at: '2026-01-01' + reason: x + """ + ) + p = tmp_path / "bad.yaml" + p.write_text(text) + with pytest.raises(PolicyLoadError, match="NEW_RULE"): + load_recalibration_policy(p) + + +def test_policy_canonical_prohibited_action_missing(tmp_path): + text = dedent( + """ + policy_version: 1 + locked_at: '2026-01-01' + human_reviewer: tester + minimum_conditions: [] + prohibited_actions: + - id: NO_HEMOLYSIS_RELAXATION + description: x + rationale: x + - id: NO_NOVELTY_RELAXATION + description: x + rationale: x + locked_changes: + - rule_id: NO_HEMOLYSIS_RELAXATION + locked_at: '2026-01-01' + reason: x + - rule_id: NO_NOVELTY_RELAXATION + locked_at: '2026-01-01' + reason: x + """ + ) + p = tmp_path / "bad.yaml" + p.write_text(text) + with pytest.raises(PolicyLoadError, match="NO_TOXICITY_RELAXATION"): + load_recalibration_policy(p) + + +def test_policy_prohibited_action_not_locked(tmp_path): + text = dedent( + """ + policy_version: 1 + locked_at: '2026-01-01' + human_reviewer: tester + minimum_conditions: [] + prohibited_actions: + - id: NO_TOXICITY_RELAXATION + description: x + rationale: x + - id: NO_HEMOLYSIS_RELAXATION + description: x + rationale: x + - id: NO_NOVELTY_RELAXATION + description: x + rationale: x + - id: NO_DANGEROUS_PATHGEN_OPTIMIZATION + description: x + rationale: x + - id: NO_POST_HOC_SUCCESS_REDEFINITION + description: x + rationale: x + # NO_TOXICITY_RELAXATION is intentionally omitted from locked_changes + locked_changes: + - rule_id: NO_HEMOLYSIS_RELAXATION + locked_at: '2026-01-01' + reason: x + - rule_id: NO_NOVELTY_RELAXATION + locked_at: '2026-01-01' + reason: x + - rule_id: NO_DANGEROUS_PATHGEN_OPTIMIZATION + locked_at: '2026-01-01' + reason: x + - rule_id: NO_POST_HOC_SUCCESS_REDEFINITION + locked_at: '2026-01-01' + reason: x + """ + ) + p = tmp_path / "bad.yaml" + p.write_text(text) + with pytest.raises(PolicyLoadError, match="NO_TOXICITY_RELAXATION"): + load_recalibration_policy(p) + + +def test_policy_threshold_must_be_present(tmp_path): + text = dedent( + """ + policy_version: 1 + locked_at: '2026-01-01' + human_reviewer: tester + minimum_conditions: + - id: NEW_RULE + description: x + applies_to: r + rationale: x + prohibited_actions: [] + locked_changes: [] + """ + ) + p = tmp_path / "bad.yaml" + p.write_text(text) + with pytest.raises(PolicyLoadError, match="threshold"): + load_recalibration_policy(p) + + +def test_policy_notes_must_be_list_of_strings(tmp_path): + text = dedent( + """ + policy_version: 1 + locked_at: '2026-01-01' + human_reviewer: tester + minimum_conditions: [] + prohibited_actions: + - id: NO_TOXICITY_RELAXATION + description: x + rationale: x + - id: NO_HEMOLYSIS_RELAXATION + description: x + rationale: x + - id: NO_NOVELTY_RELAXATION + description: x + rationale: x + - id: NO_DANGEROUS_PATHGEN_OPTIMIZATION + description: x + rationale: x + - id: NO_POST_HOC_SUCCESS_REDEFINITION + description: x + rationale: x + locked_changes: + - rule_id: NO_TOXICITY_RELAXATION + locked_at: '2026-01-01' + reason: x + - rule_id: NO_HEMOLYSIS_RELAXATION + locked_at: '2026-01-01' + reason: x + - rule_id: NO_NOVELTY_RELAXATION + locked_at: '2026-01-01' + reason: x + - rule_id: NO_DANGEROUS_PATHGEN_OPTIMIZATION + locked_at: '2026-01-01' + reason: x + - rule_id: NO_POST_HOC_SUCCESS_REDEFINITION + locked_at: '2026-01-01' + reason: x + notes: 5 + """ + ) + p = tmp_path / "bad.yaml" + p.write_text(text) + with pytest.raises(PolicyLoadError, match="notes"): + load_recalibration_policy(p) + + +# ──────────────────────────────────────────────────────────────────── +# Gate evaluation: minimum conditions +# ──────────────────────────────────────────────────────────────────── + + +def _empty_intake_report(): + """Return an intake-report dict that should fail every rule.""" + + return { + "panel_csv": "x.csv", + "results_dir": "y", + "n_panel_candidates": 0, + "n_lab_results": 0, + "n_matched_candidates": 0, + "n_orphan_lab_results": 0, + "orphan_candidate_ids": [], + "summary": {}, + "per_candidate_outcomes": {}, + "per_candidate_joined": [], + "cohort_metrics": { + "activity_vs_active_mic": { + "n": 0, + "insufficient_data": True, + } + }, + "control_failures": [], + "min_cohort_size": 5, + "activity_threshold": 0.5, + "mic_active_cutoff_ug_ml": 32.0, + "hemolysis_high_pct": 10.0, + "report_disclaimer": "x", + } + + +def _passing_intake_report(n_active: int = 3, n_inactive: int = 3): + """Return an intake-report dict where all rules should pass.""" + + rows = [] + actives = n_active + inactives = n_inactive + for i in range(actives): + rows.append( + { + "candidate_id": f"ACTIVE-{i:03d}", + "has_lab": { + "active_mic": True, + "high_hemolysis": False, + "all_controls_passed": True, + }, + } + ) + for i in range(inactives): + rows.append( + { + "candidate_id": f"INACTIVE-{i:03d}", + "has_lab": { + "active_mic": False, + "high_hemolysis": False, + "all_controls_passed": True, + }, + } + ) + return { + "panel_csv": "x.csv", + "results_dir": "y", + "n_panel_candidates": actives + inactives, + "n_lab_results": actives + inactives, + "n_matched_candidates": actives + inactives, + "n_orphan_lab_results": 0, + "orphan_candidate_ids": [], + "summary": {}, + "per_candidate_outcomes": {}, + "per_candidate_joined": rows, + "cohort_metrics": { + "activity_vs_active_mic": { + "n": actives + inactives, + "insufficient_data": False, + } + }, + "control_failures": [], + "min_cohort_size": 5, + "activity_threshold": 0.5, + "mic_active_cutoff_ug_ml": 32.0, + "hemolysis_high_pct": 10.0, + "report_disclaimer": "x", + } + + +def test_gate_rejects_empty_report(): + p = load_recalibration_policy(POLICY_PATH) + v = evaluate_recalibration_gate(_empty_intake_report(), p) + assert v.may_recalibrate is False + # Many rule failures expected + failed = [r for r in v.rule_results if not r.passed] + assert len(failed) >= 4 + + +def test_gate_accepts_passing_report(): + p = load_recalibration_policy(POLICY_PATH) + v = evaluate_recalibration_gate(_passing_intake_report(3, 3), p) + assert v.may_recalibrate is True + # All rules pass + for r in v.rule_results: + assert r.passed, f"rule {r.rule_id} unexpectedly failed: {r.reason}" + + +def test_gate_detects_failed_positive_control(): + p = load_recalibration_policy(POLICY_PATH) + report = _passing_intake_report(3, 3) + report["control_failures"] = [ + {"result_id": "X", "candidate_id": "Y", "positive_control_passed": False, + "negative_control_passed": True} + ] + v = evaluate_recalibration_gate(report, p) + assert v.may_recalibrate is False + failed_ids = [r.rule_id for r in v.rule_results if not r.passed] + assert "POSITIVE_CONTROLS_ALL_PASS" in failed_ids + + +def test_gate_detects_failed_negative_control(): + p = load_recalibration_policy(POLICY_PATH) + report = _passing_intake_report(3, 3) + report["control_failures"] = [ + {"result_id": "X", "candidate_id": "Y", "positive_control_passed": True, + "negative_control_passed": False} + ] + v = evaluate_recalibration_gate(report, p) + assert v.may_recalibrate is False + failed_ids = [r.rule_id for r in v.rule_results if not r.passed] + assert "NEGATIVE_CONTROLS_ALL_PASS" in failed_ids + + +def test_gate_detects_orphan_results(): + p = load_recalibration_policy(POLICY_PATH) + report = _passing_intake_report(3, 3) + report["n_orphan_lab_results"] = 2 + v = evaluate_recalibration_gate(report, p) + assert v.may_recalibrate is False + failed_ids = [r.rule_id for r in v.rule_results if not r.passed] + assert "NO_ORPHAN_LAB_RESULTS" in failed_ids + + +def test_gate_detects_insufficient_metrics(): + p = load_recalibration_policy(POLICY_PATH) + report = _passing_intake_report(3, 3) + report["cohort_metrics"]["activity_vs_active_mic"]["insufficient_data"] = True + v = evaluate_recalibration_gate(report, p) + assert v.may_recalibrate is False + failed_ids = [r.rule_id for r in v.rule_results if not r.passed] + assert "COHORT_METRICS_AVAILABLE" in failed_ids + + +def test_gate_min_positives_below_threshold(): + p = load_recalibration_policy(POLICY_PATH) + report = _passing_intake_report(n_active=1, n_inactive=3) + v = evaluate_recalibration_gate(report, p) + assert v.may_recalibrate is False + failed_ids = [r.rule_id for r in v.rule_results if not r.passed] + assert "MIN_POSITIVES_IN_COHORT" in failed_ids + + +def test_gate_min_negatives_below_threshold(): + p = load_recalibration_policy(POLICY_PATH) + report = _passing_intake_report(n_active=3, n_inactive=1) + v = evaluate_recalibration_gate(report, p) + assert v.may_recalibrate is False + failed_ids = [r.rule_id for r in v.rule_results if not r.passed] + assert "MIN_NEGATIVES_IN_COHORT" in failed_ids + + +# ──────────────────────────────────────────────────────────────────── +# Gate evaluation: prohibited actions and rate limits +# ──────────────────────────────────────────────────────────────────── + + +def test_prohibited_action_audit_always_in_force(): + p = load_recalibration_policy(POLICY_PATH) + v = evaluate_recalibration_gate(_empty_intake_report(), p) + assert len(v.prohibited_action_audit) == len(p.prohibited_actions) + for a in v.prohibited_action_audit: + assert a.in_force is True + assert a.description # non-empty + + +def test_rate_limit_unknown_when_no_dates(): + p = load_recalibration_policy(POLICY_PATH) + v = evaluate_recalibration_gate(_passing_intake_report(), p) + cooldown = next( + s for s in v.rate_limit_status if s.rule_id == "COOLDOWN_DAYS" + ) + assert cooldown.status == "unknown" + assert "missing" in cooldown.note + + +def test_rate_limit_unknown_when_no_weight_l1(): + p = load_recalibration_policy(POLICY_PATH) + v = evaluate_recalibration_gate(_passing_intake_report(), p) + l1 = next( + s for s in v.rate_limit_status if s.rule_id == "WEIGHT_CHANGE_L1_BUDGET" + ) + assert l1.status == "unknown" + + +def test_rate_limit_cooldown_ok(): + p = load_recalibration_policy(POLICY_PATH) + today = date.today().isoformat() + # Choose a previous date well beyond the 14-day threshold + prev = "2020-01-01" + v = evaluate_recalibration_gate( + _passing_intake_report(), p, + intake_report_date=today, + previous_recalibration_at=prev, + ) + cooldown = next( + s for s in v.rate_limit_status if s.rule_id == "COOLDOWN_DAYS" + ) + assert cooldown.status == "ok" + + +def test_rate_limit_cooldown_exceeded(): + p = load_recalibration_policy(POLICY_PATH) + today = date.today().isoformat() + # Choose a previous date inside the 14-day cooldown (today itself = 0 days) + prev = date.today().isoformat() + v = evaluate_recalibration_gate( + _passing_intake_report(), p, + intake_report_date=today, + previous_recalibration_at=prev, + ) + cooldown = next( + s for s in v.rate_limit_status if s.rule_id == "COOLDOWN_DAYS" + ) + assert cooldown.status == "exceeded" + assert cooldown.observed == 0 + + +def test_rate_limit_weight_l1_exceeded(): + p = load_recalibration_policy(POLICY_PATH) + v = evaluate_recalibration_gate( + _passing_intake_report(), p, + candidate_weight_l1_distance=0.50, + ) + l1 = next( + s for s in v.rate_limit_status if s.rule_id == "WEIGHT_CHANGE_L1_BUDGET" + ) + assert l1.status == "exceeded" + + +def test_rate_limit_weight_l1_within_budget(): + p = load_recalibration_policy(POLICY_PATH) + v = evaluate_recalibration_gate( + _passing_intake_report(), p, + candidate_weight_l1_distance=0.05, + ) + l1 = next( + s for s in v.rate_limit_status if s.rule_id == "WEIGHT_CHANGE_L1_BUDGET" + ) + assert l1.status == "ok" + + +# ──────────────────────────────────────────────────────────────────── +# Gate evaluation: reviewer artefacts +# ──────────────────────────────────────────────────────────────────── + + +def test_reviewer_artefact_status_when_missing(tmp_path): + p = load_recalibration_policy(POLICY_PATH) + v = evaluate_recalibration_gate( + _passing_intake_report(), p, project_root=tmp_path + ) + for s in v.reviewer_artefact_status: + # The synthetic artefacts aren't in tmp_path, so all should be missing + assert s.present is False + + +def test_reviewer_artefact_status_when_present(tmp_path): + # Create the expected intake files + (tmp_path / "outputs").mkdir() + (tmp_path / "outputs" / "calibration_intake.json").write_text("{}") + (tmp_path / "outputs" / "calibration_intake.md").write_text("# x") + docs_dir = tmp_path / "docs" + docs_dir.mkdir() + today_log = docs_dir / f"DECISION_LOG_{date.today().isoformat()}.md" + today_log.write_text("# decision log") + + p = load_recalibration_policy(POLICY_PATH) + v = evaluate_recalibration_gate( + _passing_intake_report(), p, project_root=tmp_path + ) + present = {s.artefact_id: s.present for s in v.reviewer_artefact_status} + assert present["INTAKE_REPORT_JSON"] is True + assert present["INTAKE_REPORT_MARKDOWN"] is True + assert present["DECISION_LOG_ENTRY"] is True + + +# ──────────────────────────────────────────────────────────────────── +# Gate evaluation: writers +# ──────────────────────────────────────────────────────────────────── + + +def test_gate_verdict_to_dict_is_serialisable(): + p = load_recalibration_policy(POLICY_PATH) + v = evaluate_recalibration_gate(_passing_intake_report(), p) + d = v.to_dict() + json.dumps(d) # must not raise + + +def test_gate_verdict_writers_produce_files(tmp_path): + p = load_recalibration_policy(POLICY_PATH) + v = evaluate_recalibration_gate(_passing_intake_report(), p) + json_path = tmp_path / "v.json" + md_path = tmp_path / "v.md" + write_gate_verdict_json(v, json_path) + write_gate_verdict_markdown( + v, + md_path, + intake_report_path="outputs/calibration_intake.json", + policy_path="configs/recalibration_policy.yaml", + ) + assert json_path.exists() and json_path.stat().st_size > 100 + assert md_path.exists() and md_path.stat().st_size > 100 + # Markdown mentions every prohibited action + md_text = md_path.read_text() + for a in p.prohibited_actions: + assert a.id in md_text + + +# ──────────────────────────────────────────────────────────────────── +# End-to-end with the synthetic example +# ──────────────────────────────────────────────────────────────────── + + +def test_synthetic_example_fails_gate(): + """The synthetic example has cohort=4 (<5) and one failed positive control.""" + p = load_recalibration_policy(POLICY_PATH) + if not SYNTHETIC_INTAKE.exists(): + pytest.skip("synthetic intake not generated yet") + with SYNTHETIC_INTAKE.open() as f: + report = json.load(f) + v = evaluate_recalibration_gate(report, p) + assert v.may_recalibrate is False + failed_ids = [r.rule_id for r in v.rule_results if not r.passed] + assert "MIN_COHORT_SIZE" in failed_ids + assert "POSITIVE_CONTROLS_ALL_PASS" in failed_ids + + +def test_gate_verdict_dataclass_fields(): + """Type-check the GateVerdict dataclass surface.""" + p = load_recalibration_policy(POLICY_PATH) + v = evaluate_recalibration_gate(_passing_intake_report(), p) + assert isinstance(v, GateVerdict) + assert isinstance(v.rule_results, tuple) + assert all(isinstance(r, RuleResult) for r in v.rule_results) + assert isinstance(v.rate_limit_status, tuple) + assert all(isinstance(s, RateLimitStatus) for s in v.rate_limit_status) + assert isinstance(v.reviewer_artefact_status, tuple) + assert all( + isinstance(s, ReviewerArtefactStatus) + for s in v.reviewer_artefact_status + ) + + +# ──────────────────────────────────────────────────────────────────── +# CLI surface +# ──────────────────────────────────────────────────────────────────── + + +def test_cli_recalibration_gate_smoke(): + """The CLI must exist, accept our flags, and emit JSON to stdout.""" + import subprocess + + if not SYNTHETIC_INTAKE.exists(): + pytest.skip("synthetic intake not generated yet") + + cmd = [ + sys.executable, + "-m", + "openamp_foundry.cli", + "recalibration-gate", + "--intake-report", + str(SYNTHETIC_INTAKE), + "--intake-report-date", + "2026-07-04", + "--project-root", + str(REPO_ROOT), + ] + proc = subprocess.run( + cmd, cwd=str(REPO_ROOT), env={"PYTHONPATH": "src", "PATH": sys.exec_prefix + "/bin:" + __import__("os").environ.get("PATH", "")}, + capture_output=True, + text=True, + timeout=60, + ) + assert proc.returncode == 3, ( + f"expected exit code 3 (gate failed), got {proc.returncode}\n" + f"stdout:\n{proc.stdout[:500]}\nstderr:\n{proc.stderr[:500]}" + ) + payload = json.loads(proc.stdout) + assert payload["status"] == "ok" + assert payload["may_recalibrate"] is False + assert payload["policy_version"] == 1 + + +def test_cli_recalibration_gate_missing_intake(tmp_path): + import subprocess + + cmd = [ + sys.executable, + "-m", + "openamp_foundry.cli", + "recalibration-gate", + "--intake-report", + str(tmp_path / "nope.json"), + "--project-root", + str(REPO_ROOT), + ] + proc = subprocess.run( + cmd, cwd=str(REPO_ROOT), env={"PYTHONPATH": "src", "PATH": sys.exec_prefix + "/bin:" + __import__("os").environ.get("PATH", "")}, + capture_output=True, + text=True, + timeout=60, + ) + assert proc.returncode == 2 + payload = json.loads(proc.stdout) + assert payload["status"] == "error"