Skip to content

v0.5.44 — Recalibration report with schema validation (Loop 23) #142

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cschanhniem merged 1 commit into from
Jul 6, 2026
Merged

v0.5.44 — Recalibration report with schema validation (Loop 23) #142

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions docs/50_LOOP_PLAN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# 50-Loop Execution Plan

> **Status:** Strategic roadmap. Updated v0.5.43.
> **Current state:** 1804 tests, pipeline AUROC 0.7792, calibration intake + gate + engine + dry-run + policy-version + synthetic generator shipped,
> Phase 0 complete (Loops 1–8), Phase 1 complete (Loops 9–17), Phase 2 Loops 18, 19, 20, 21, 22 complete,
> **Status:** Strategic roadmap. Updated v0.5.44.
> **Current state:** 1813 tests, pipeline AUROC 0.7792, calibration intake + gate + engine + dry-run + policy-version + synthetic generator + recalibration report shipped,
> Phase 0 complete (Loops 1–8), Phase 1 complete (Loops 9–17), Phase 2 Loops 18, 19, 20, 21, 22, 23 complete,
> cluster-split/selectivity/triage now gated in CI, Wave 0.5 panel ready (24 candidates, 15 families), no wet-lab data yet.
>
> Each loop = one focused PR: bottleneck identified → implemented → verified → merged.
Expand Down Expand Up @@ -73,7 +73,7 @@ Build the recalibration engine gated by the policy. Implement active-learning ba
| 20 ✅ | No lab-result simulator for testing the calibration loop without real data | `examples/lab_results_generator.py`: generates synthetic lab results at configurable cohort sizes, effect sizes, and noise levels. Clearly labeled SYNTHETIC in every file. v0.5.42. 7 assay types, 40/40 schema-valid, integrates with calibration-intake. | Produces schema-valid JSON files that drive engine tests; verified with `make generate-synthetic-lab-results` + `calibration-intake` (20 matched, 0 orphans) |
| 21 ✅ | **Recalibration engine**: the gate existed but no engine. Real weight-update code needs to be safe, auditable, and gated | `calibration/engine.py`: `compute_weight_update(intake_report, gate_verdict, current_weights, l1_budget)` returns `WeightUpdateProposal`. Control theory: delta = learning_rate × (observed_accuracy − target_accuracy). Conservative learning rate 0.05. L1 budget enforced. v0.5.36 | 1735 tests; CLI `recalibration-engine` exits 0 on proposal, 3 on violations, 2 on missing files; 12 engine tests |
| 22 ✅ | No "dry run" mode for recalibration engine. User can't preview what weights would change | `--dry-run` flag: compute and log proposed weight changes without applying them. Outputs comparison table (current vs proposed). v0.5.43. 1 test, Makefile target. | Dry run produces diff without side effects; verified with `make recalibration-engine-dry-run` + manual inspection |
| 23 | No weight-change report that a human reviewer can inspect | `reports/recalibration_report.py`: generates human-readable report with before/after weights, rationale, policy-rule results, and explicit "human review required" banner | Report validates against schema |
| 23 ✅ | No weight-change report that a human reviewer can inspect | `reports/recalibration_report.py`: generates human-readable report with before/after weights, rationale, policy-rule results, gate-verdict context, and explicit "human review required" banner. Schema validation via `schemas/recalibration_report.schema.json`. v0.5.44. 9 tests. | Report validates against schema; gate verdict details preserved; CLI uses combined report for JSON/MD output |
| 24 | No second-batch selection logic. The gate allows recalibration, but how do we pick the next 8–12 candidates? | `active_learning/selector.py`: implements uncertainty sampling + diversity + safety gate. Returns ranked candidates for batch 2 | Selection respects safety constraints; top-5 include at least 1 high-uncertainty probe |
| 25 | No active-learning benchmark on synthetic data. Can the selector recover known "hidden active" candidates better than random? | Synthetic benchmark: hide 3 known active AMPs from the panel, run selection, measure how many rounds to recover them | Recovery within N rounds (N pre-registered) |
| 26 | No integration between active-learning selector and the calibration pipeline | End-to-end flow: intake → gate (pass) → recalibration engine (dry-run) → selector → batch 2 manifest | Full chain runs on synthetic data, produces a batch-2 manifest |
Expand Down Expand Up @@ -177,7 +177,7 @@ If no data arrives, virtual assay scaffolding continues independently.
```
Phase 0: ✅ Complete (Loops 1–8)
Phase 1: ✅ Complete (Loops 9–17)
Phase 2: Loop 22 of 29 (dry-run shipped — Loop 23 next)
Phase 2: Loop 23 of 29 (recalibration report shipped — Loop 24 next)
Phase 3: Not started (Loops 30–39)
Phase 4: Not started (Loops 40–49)
```
Expand Down Expand Up @@ -232,4 +232,4 @@ Phase 4: Not started (Loops 40–49)
| 21 ✅ | Recalibration engine: the gate existed but no engine. Real weight-update code needs to be safe, auditable, and gated | `calibration/engine.py`: `compute_weight_update()` returns `WeightUpdateProposal`. Control theory: delta = learning_rate × (observed_accuracy − target_accuracy). Conservative LR 0.05. L1 budget enforced. v0.5.36 | 1735 tests; CLI recalibration-engine exits 0 on proposal, 3 on violations, 2 on missing files; 12 engine tests |
| 22 ✅ | No dry-run mode for recalibration engine. User can't preview what weights would change | `--dry-run` flag: compute and log proposed weight changes without applying them. Outputs comparison table (current vs proposed). v0.5.43. 1 test, Makefile target. | Dry run produces diff without side effects; verified with `make recalibration-engine-dry-run` + manual inspection |

**Next loop:** Loop 23 — Phase 2 (Weight-change report for human reviewer inspection).
**Next loop:** Loop 24 — Phase 2 (Second-batch selection logic).
28 changes: 28 additions & 0 deletions docs/ROADMAP.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,33 @@
# Roadmap

## v0.5.44 — Recalibration Report with Schema Validation ✓ (2026-07-06)

The existing weight-update proposal markdown writer was embedded in engine.py
and lacked gate-verdict context. This release adds a standalone recalibration
report module with a JSON Schema, validation, and a combined report that joins
gate verdict details with the weight-change proposal.

Changes:
- Added ``schemas/recalibration_report.schema.json`` — JSON Schema (Draft 2020-12)
for the combined recalibration report. Enforces: report_type discriminator,
schema_version, human_review_required=True, required gate-verdict fields
(may_recalibrate, cohort counts, summary), required proposal fields
(gate_passed, L1 totals, deltas). Each delta requires scorer, current_weight,
proposed_weight, delta, and rationale.
- Added ``reports/recalibration_report.py`` — ``build_recalibration_report()``
combines a ``WeightUpdateProposal`` and ``GateVerdict`` into a single report
dict. ``write_recalibration_report_json()`` and
``write_recalibration_report_markdown()`` produce the combined output.
``validate_recalibration_report()`` validates against the schema.
- The CLI ``recalibration-engine`` now uses the combined report for both
``--out-json`` and ``--out-md`` output. The ``--dry-run`` diff also includes
gate-verdict summary (rule count, prohibited action count).
- Updated ``reports/__init__.py`` to export the new symbols.
- Added 9 tests: report shape, gate context preservation, proposal shape,
delta field completeness, schema validation (valid + missing fields +
human_review_required=True enforcement), JSON write+validate round-trip,
markdown output with required sections.

## v0.5.43 — Dry-Run Mode for Recalibration Engine ✓ (2026-07-06)

The recalibration engine computed proposals but had no way to preview what
Expand Down
152 changes: 152 additions & 0 deletions schemas/recalibration_report.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,152 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "OpenAMP Recalibration Report",
"description": "Combined report joining a gate verdict with a weight-update proposal for human reviewer inspection.",
"type": "object",
"required": [
"report_type",
"schema_version",
"timestamp",
"policy_version",
"human_review_required",
"gate_verdict",
"proposal"
],
"properties": {
"report_type": {
"type": "string",
"enum": ["recalibration_report"],
"description": "Fixed discriminator for this report type."
},
"schema_version": {
"type": "string",
"pattern": "^\\d+\\.\\d+$",
"description": "Version of this schema (semver-like, e.g. 1.0)."
},
"timestamp": {
"type": "string",
"description": "ISO-8601 timestamp when the report was generated."
},
"policy_version": {
"type": "integer",
"minimum": 1,
"description": "Policy version in effect when the report was generated."
},
"human_review_required": {
"type": "boolean",
"enum": [true],
"description": "This report is a proposal only. A human must inspect, verify, and sign before any weight changes are applied."
},
"gate_verdict": {
"type": "object",
"required": [
"may_recalibrate",
"n_panel_candidates",
"n_matched_candidates",
"n_lab_results",
"summary"
],
"properties": {
"may_recalibrate": {
"type": "boolean",
"description": "Whether the gate allows recalibration."
},
"n_panel_candidates": {
"type": "integer",
"minimum": 0
},
"n_matched_candidates": {
"type": "integer",
"minimum": 0
},
"n_lab_results": {
"type": "integer",
"minimum": 0
},
"rule_results": {
"type": "array",
"items": {
"type": "object",
"required": ["rule_id", "passed", "reason"],
"properties": {
"rule_id": { "type": "string" },
"passed": { "type": "boolean" },
"observed": {},
"threshold": {},
"reason": { "type": "string" }
}
}
},
"prohibited_action_audit": {
"type": "array",
"items": {
"type": "object",
"required": ["action_id", "description", "in_force"],
"properties": {
"action_id": { "type": "string" },
"description": { "type": "string" },
"in_force": { "type": "boolean", "enum": [true] }
}
}
},
"rate_limit_status": {
"type": "array",
"items": {
"type": "object",
"required": ["rule_id", "status", "note"],
"properties": {
"rule_id": { "type": "string" },
"status": { "type": "string", "enum": ["ok", "exceeded", "unknown"] },
"observed": {},
"threshold": {},
"note": { "type": "string" }
}
}
},
"reasons": {
"type": "array",
"items": { "type": "string" }
},
"summary": { "type": "string" }
}
},
"proposal": {
"type": "object",
"required": [
"gate_passed",
"l1_total",
"l1_budget",
"l1_within_budget",
"deltas"
],
"properties": {
"gate_passed": { "type": "boolean" },
"l1_total": { "type": "number", "minimum": 0 },
"l1_budget": { "type": "number", "minimum": 0 },
"l1_within_budget": { "type": "boolean" },
"deltas": {
"type": "array",
"items": {
"type": "object",
"required": ["scorer", "current_weight", "proposed_weight", "delta", "rationale"],
"properties": {
"scorer": { "type": "string" },
"current_weight": { "type": "number" },
"proposed_weight": { "type": "number" },
"delta": { "type": "number" },
"precision": { "type": ["number", "null"] },
"recall": { "type": ["number", "null"] },
"n_positive": { "type": "integer", "minimum": 0 },
"n_negative": { "type": "integer", "minimum": 0 },
"rationale": { "type": "string" }
}
}
},
"notes": {
"type": "array",
"items": { "type": "string" }
}
}
}
}
}
54 changes: 37 additions & 17 deletions src/openamp_foundry/cli/commands/reports.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -837,6 +837,12 @@ def _run_recalibration_engine(args: argparse.Namespace) -> int:
summary=gate_data.get("summary", ""),
)

from openamp_foundry.reports.recalibration_report import (
build_recalibration_report,
write_recalibration_report_json,
write_recalibration_report_markdown,
)

try:
proposal = compute_weight_update(
intake_report=intake,
Expand All @@ -860,42 +866,56 @@ def _run_recalibration_engine(args: argparse.Namespace) -> int:
print(json.dumps(payload))
return 3

# Build the combined recalibration report (proposal + gate verdict)
report = build_recalibration_report(proposal, gate_verdict)
prop_section = report.get("proposal", {})

is_dry_run = getattr(args, "dry_run", False)
dry_run_label = " [DRY RUN — no changes applied]" if is_dry_run else ""

if is_dry_run:
# Print diff table instead of writing files
print(f"Recalibration Engine Proposal{dry_run_label}")
print(f"{'=' * 60}")
print(f" Timestamp: {proposal.timestamp}")
print(f" Gate passed: {proposal.gate_passed}")
print(f" Policy version: {proposal.policy_version}")
print(f" L1 total: {proposal.l1_total:.4f} / budget {proposal.l1_budget:.4f}")
print(f" L1 within budget: {proposal.l1_within_budget}")
print(f" Timestamp: {report['timestamp']}")
print(f" Report type: {report['report_type']}")
print(f" Schema version: {report['schema_version']}")
print(f" Policy version: {report['policy_version']}")
print(f" Gate passed: {prop_section.get('gate_passed')}")
print(f" L1 total: {prop_section.get('l1_total', 0):.4f} / budget {prop_section.get('l1_budget', 0):.4f}")
print(f" L1 within budget: {prop_section.get('l1_within_budget')}")
gv = report.get("gate_verdict", {})
n_rules = len(gv.get("rule_results", []))
n_actions = len(gv.get("prohibited_action_audit", []))
print(f" Gate rules: {n_rules} minimum conditions, {n_actions} prohibited actions")
print()
print(f" {'Scorer':<20} {'Current':>8} {'Proposed':>8} {'Delta':>8} {'Rationale'}")
print(f" {'-'*20} {'-'*8} {'-'*8} {'-'*8} {'-'*40}")
for d in proposal.deltas:
sign = "+" if d.delta >= 0 else ""
print(f" {d.scorer:<20} {d.current_weight:>8.4f} {d.proposed_weight:>8.4f} {sign}{d.delta:>7.4f} {d.rationale[:60]}")
for d in prop_section.get("deltas", []):
sign = "+" if d["delta"] >= 0 else ""
print(f" {d['scorer']:<20} {d['current_weight']:>8.4f} {d['proposed_weight']:>8.4f} {sign}{d['delta']:>7.4f} {d['rationale'][:60]}")
print()
print(f" N deltas: {len(proposal.deltas)}")
print(f" Notes: {', '.join(proposal.notes) if proposal.notes else '(none)'}")
print(f" N deltas: {len(prop_section.get('deltas', []))}")
notes = prop_section.get("notes", [])
print(f" Notes: {', '.join(notes) if notes else '(none)'}")
print(f"{'=' * 60}")
print(f"DRY RUN — no files written, no weight changes applied.")
else:
if args.out_json:
write_weight_update_proposal_json(proposal, args.out_json)
write_recalibration_report_json(report, args.out_json)
if args.out_md:
write_weight_update_proposal_markdown(proposal, args.out_md)
write_recalibration_report_markdown(report, args.out_md)

summary = {
"status": "ok",
"gate_passed": proposal.gate_passed,
"n_deltas": len(proposal.deltas),
"l1_total": round(proposal.l1_total, 4),
"l1_within_budget": proposal.l1_within_budget,
"timestamp": proposal.timestamp,
"report_type": "recalibration_report",
"schema_version": "1.0",
"gate_passed": prop_section.get("gate_passed"),
"n_deltas": len(prop_section.get("deltas", [])),
"l1_total": round(prop_section.get("l1_total", 0), 4),
"l1_within_budget": prop_section.get("l1_within_budget"),
"timestamp": report["timestamp"],
"human_review_required": True,
"dry_run": is_dry_run,
}
print(json.dumps(summary, indent=2))
Expand Down
11 changes: 11 additions & 0 deletions src/openamp_foundry/reports/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,12 @@
write_pilot_csv,
write_pilot_markdown,
)
from openamp_foundry.reports.recalibration_report import (
build_recalibration_report,
validate_recalibration_report,
write_recalibration_report_json,
write_recalibration_report_markdown,
)

__all__ = [
# batch_pack
Expand All @@ -55,4 +61,9 @@
# pilot_panel
"write_pilot_csv",
"write_pilot_markdown",
# recalibration_report
"build_recalibration_report",
"validate_recalibration_report",
"write_recalibration_report_json",
"write_recalibration_report_markdown",
]
Loading
Loading