Skip to content

Phase G G3 — calibration pipeline consistency audit - #767

Merged
cschanhniem merged 1 commit into
mainfrom
loop81-phase-g
Jul 9, 2026
Merged

cschanhniem merged 1 commit into
mainfrom
loop81-phase-g

Conversation

@cschanhniem

Copy link
Copy Markdown
Collaborator

Change

Adds openamp-foundry calibration-audit CLI that checks consistency across all calibration pipeline artifacts (intake report, gate verdict, engine proposal, recalibration report).

Why

Phase G G3: make calibration-audit — checks intake/gate/engine/report consistency. A human reviewer inspecting the calibration pipeline output needs to verify that all stages agree on candidate counts, gate verdicts, weight proposals, and timestamps. Previously there was no automated way to do this.

What

  • Schema: schemas/calibration_audit.schema.json — Draft 2020-12 JSON Schema for audit report output
  • Core module: src/openamp_foundry/calibration/audit.py — run_calibration_audit() with 12 consistency checks across the 4 artifacts
  • CLI: openamp-foundry calibration-audit with --intake-report, --gate-verdict, --engine-proposal, --recalibration-report, --out-json, --out-md
  • Makefile: make calibration-audit-example (synthetic example) and make calibration-audit (variable-based)
  • Tests: 18 pytest tests covering all check types, edge cases, and schema validation

Evidence

make calibration-audit-example  # produces outputs/calibration_audit_example.json + .md
.venv/bin/python -m pytest tests/calibration/test_calibration_audit.py -v  # 18/18 pass
.venv/bin/python -m pytest --collect-only -q 2>&1 | tail -1  # 2937 tests

Safety impact

None. This is a read-only consistency check that does not modify any pipeline state, scoring weights, or candidate selection.

Proof level

Level 0 (computational infrastructure). No biological claims.

Limitations

  • Checks consistency between artifacts, not biological validity
  • Does not validate chronological ordering between artifacts (gate must precede engine)
  • Does not validate each artifact's schema conformance (separate tests exist)

Human review needed

No.

CLI (openamp-foundry calibration-audit) that checks consistency across
calibration pipeline artifacts — intake report, gate verdict, engine
proposal, and recalibration report. Runs 12 consistency checks with
machine-readable JSON and human-readable Markdown output.

- schemas/calibration_audit.schema.json
- src/openamp_foundry/calibration/audit.py
- CLI registration and dispatch in main.py and reports.py
- make calibration-audit and make calibration-audit-example targets
- 18 pytest tests
- Docs: METRICS_CURRENT.md, ROADMAP.md, test baseline
@cschanhniem
cschanhniem merged commit 9a6c732 into main Jul 9, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant