Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -517,6 +517,8 @@ After qualified assay data exists, build:

**v0.5.19 progress:** `openamp-foundry calibration-intake` joins pilot panel predictions with validated lab result actuals and produces a per-candidate report. Cohort metrics are gated by `MIN_COHORT_SIZE=5` to prevent small-sample theater. The module is descriptive only — recalibration requires a separate human-reviewed decision log (`docs/DECISION_RULES.md`). Synthetic-but-clearly-labeled example data is in `examples/lab_results/`. 29 new tests, total 1614 passing.

**v0.5.20 progress:** `openamp-foundry recalibration-gate` evaluates a calibration-intake report against the pre-registered policy in `configs/recalibration_policy.yaml` and emits a binary `may_recalibrate` verdict. The policy encodes 7 minimum conditions (cohort size, controls, orphans, positives, negatives, metrics availability), 5 permanent prohibited actions (toxicity, hemolysis, novelty, pathogen enhancement, post-hoc success redefinition), and 2 rate limits (L1 weight budget, cooldown). The validator rejects policy files that omit any canonical prohibited action or any `locked_changes` entry. The gate does NOT trigger weight updates; it is the missing permission layer between v0.5.19 intake and a future recalibration engine. Exit code 0 when `may_recalibrate=true`, 3 when false. 39 new tests, total 1647 passing. See `docs/CALIBRATION_POLICY.md`.

* lab-result ingestion;
* hit/failure calibration;
* retrospective analysis of why the model was right or wrong;
Expand Down
92 changes: 92 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -284,3 +284,95 @@ improvements made before committing the ~$10k wet-lab synthesis budget.
field and in `examples/lab_results/README.md`. It exists solely to exercise
the intake workflow end-to-end. When real validated lab results arrive,
replace the example directory — the pipeline itself does not change.

---

## [Unreleased — Recalibration Policy + Gate] — 2026-07-04

### v0.5.20 — Recalibration Policy + Gate

Pre-registered, machine-readable policy that gates any pipeline recalibration.
This is the missing piece between v0.5.19 calibration-intake (descriptive
join) and a future recalibration engine: the gate now answers the binary
question "may recalibration proceed from this intake report?" with a
verdict that no agent can silently bypass.

**New policy contract — `configs/recalibration_policy.yaml`**

- `minimum_conditions` (7 rules): MIN_COHORT_SIZE, MIN_POSITIVES_IN_COHORT,
MIN_NEGATIVES_IN_COHORT, POSITIVE_CONTROLS_ALL_PASS,
NEGATIVE_CONTROLS_ALL_PASS, NO_ORPHAN_LAB_RESULTS, COHORT_METRICS_AVAILABLE.
- `prohibited_actions` (5 permanent floors): NO_TOXICITY_RELAXATION,
NO_HEMOLYSIS_RELAXATION, NO_NOVELTY_RELAXATION,
NO_DANGEROUS_PATHGEN_OPTIMIZATION, NO_POST_HOC_SUCCESS_REDEFINITION.
- `rate_limits` (2 rules): WEIGHT_CHANGE_L1_BUDGET (0.10), COOLDOWN_DAYS (14).
- `required_reviewer_artefacts` (3 artefacts): intake JSON, intake
Markdown, dated decision log entry.
- `locked_changes` records the lock date for every enforced rule. The
validator rejects a policy file that lists an enforced rule without a
matching `locked_changes` entry.

**New modules**

- `openamp_foundry/calibration/policy.py`: `load_recalibration_policy`
parses, validates, and exposes the policy. Removes or relocations of
any canonical prohibited action cause `PolicyLoadError` at load time.
- `openamp_foundry/calibration/recalibration_gate.py`:
`evaluate_recalibration_gate` consumes an intake report + a loaded
policy and returns a `GateVerdict`. Writes JSON and Markdown via
`write_gate_verdict_json` and `write_gate_verdict_markdown`.
- `openamp_foundry/cli/commands/reports.py`: `_run_recalibration_gate`.
- `cli/main.py`: `recalibration-gate` subcommand registered.
Exit code 0 when `may_recalibrate=true`, 3 when false, 2 on input error.

**Makefile targets**

- `recalibration-gate-example`: runs the gate on the synthetic intake
report. Expect exit code 3 (cohort size too small, one positive
control failed, all required reviewer artefacts missing).
- `recalibration-gate INTAKE=... [DATE=YYYY-MM-DD] [PREV=YYYY-MM-DD] [L1=float]`
for real intake reports.

**Tests — 39 new, total 1647 passing (was 1614)**

- `tests/test_recalibration_gate.py` covers:
- policy loader happy path + every rejection mode (missing fields,
duplicate ids, unlocked rules, missing canonical prohibited
actions, missing locked_changes entries, bad types);
- gate evaluator with synthetic intake reports for every minimum
condition (cohort size, positives, negatives, controls, orphans,
insufficient metrics);
- prohibited-action audit always in force;
- rate-limit status (unknown / ok / exceeded for cooldown + L1 budget);
- reviewer-artefact status (missing / present based on disk);
- JSON + Markdown writers produce non-empty, schema-valid output;
- end-to-end smoke test of the CLI (exit code 3 on synthetic example).

**Documentation**

- `docs/CALIBRATION_POLICY.md` (new): explains the policy, the gate, the
prohibited-action floor, the rate limits, how to update the policy,
and how the policy fits the wet-lab compression roadmap.
- `docs/ARCHITECTURE.md`: links the calibration package into the
wet-lab compression loop.
- `docs/PLAN.md`: marks Phase 5 (active-learning) as gated by this policy.
- `docs/ROADMAP.md`: new v0.5.20 milestone entry.
- `docs/METRICS_CURRENT.md`: new entry in the change log.

**Honest limitations (must read before relying on the gate)**

- The gate **does NOT trigger any weight update**. It only emits a verdict.
- A `may_recalibrate=true` verdict is a **permission**, not a command.
The decision to apply a weight change still belongs to a human reviewer
with a dated decision log entry (`docs/DECISION_LOG_<date>.md`).
- The gate evaluates **cohort evidence**, not pipeline calibration
health. Benchmark regressions are caught by `make validate-scoring`,
`make bench-triage`, and the selectivity benchmark — not by this
policy. These checks must keep running independently.
- The synthetic example correctly yields `may_recalibrate=false`. That
is the expected outcome on tiny synthetic data and is itself a
useful sanity check that the gate is enforcing the cohort floor.
- The five canonical prohibited actions are duplicated from
`AGENTS.md` and `MISSION.md`. The validator rejects a policy file
that drops any of them; the human source documents must be updated
in lockstep before the policy file can be edited.
27 changes: 26 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,9 @@ help:
@echo " make wave0-5b-filter Filter Wave 0.5b shortlist (depends on wave0-5b-generate)"
@echo " make lab-result-intake Join a pilot panel CSV with lab result JSON files"
@echo " make lab-result-intake-example Run intake on the synthetic example data"
@echo " make test Run full test suite (1614 tests, >=80% coverage)"
@echo " make recalibration-gate-example Evaluate recalibration gate on synthetic intake example"
@echo " make recalibration-gate Evaluate recalibration gate on a real intake report"
@echo " make test Run full test suite (1647 passing tests, >=80% coverage)"
@echo " make coverage Test suite with per-module coverage report"
@echo " make lint Ruff lint check on src/ tests/ scripts/"
@echo " make typecheck mypy type check on src/"
Expand Down Expand Up @@ -265,6 +267,29 @@ lab-result-intake:
--out-json outputs/calibration_intake.json \
--out-md outputs/calibration_intake.md

recalibration-gate-example: lab-result-intake-example
PYTHONPATH=src $(PYTHON) -m openamp_foundry.cli recalibration-gate \
--intake-report outputs/calibration_intake_example.json \
--intake-report-date 2026-07-04 \
--out-json outputs/recalibration_gate_example.json \
--out-md outputs/recalibration_gate_example.md

recalibration-gate:
@if [ -z "$(INTAKE)" ]; then \
echo "Usage: make recalibration-gate INTAKE=<intake.json> [DATE=<YYYY-MM-DD>] [PREV=<YYYY-MM-DD>] [L1=<float>]"; \
echo ""; \
echo "Or run the synthetic example without arguments:"; \
echo " make recalibration-gate-example"; \
exit 1; \
fi
PYTHONPATH=src $(PYTHON) -m openamp_foundry.cli recalibration-gate \
--intake-report "$(INTAKE)" \
$$(if [ -n "$(DATE)" ]; then echo --intake-report-date "$(DATE)"; fi) \
$$(if [ -n "$(PREV)" ]; then echo --previous-recalibration-at "$(PREV)"; fi) \
$$(if [ -n "$(L1)" ]; then echo --weight-l1-distance "$(L1)"; fi) \
--out-json outputs/recalibration_gate.json \
--out-md outputs/recalibration_gate.md

clean:
rm -rf outputs/*.jsonl outputs/*.md outputs/*.json outputs/evidence outputs/phase3_evidence .pytest_cache .ruff_cache

Expand Down
Loading
Loading