Skip to content

feat: Phase C C1 benchmark registry -- 5 BMC- cards make benchmark governance machine-enforceable - #880

Merged
cschanhniem merged 1 commit into
mainfrom
feat/c1-benchmark-registry
Jul 10, 2026
Merged

cschanhniem merged 1 commit into
mainfrom
feat/c1-benchmark-registry

Conversation

@cschanhniem

Copy link
Copy Markdown
Collaborator

Summary

  • New module: src/openamp_foundry/evidence/benchmark_registry.py
  • 5 BenchmarkCard instances covering all major pipeline benchmarks:
    • BMC-0001: AMP Activity Precision@k (leakage_aware_split, 3 metrics, 3 limitations)
    • BMC-0002: Charge-Matched AMP Activity (charge_stratified, 2 cheap enemies)
    • BMC-0003: Calibration Accuracy (random_70_30, calibration_error+brier_score)
    • BMC-0004: Family-Stratified AMP Precision (family_stratified, 3 cheap enemies, 3 limitations)
    • BMC-0005: Cheap Enemy Comparison (4 cheap enemies including conjunction, anti-hype primary benchmark)
  • API: get_card(bmc_id) → card|None; validate_registry() → list[errors] (CI-callable)
  • All 5 cards: ≥2 cheap enemies, ≥2 known limitations, validate() clean
  • 63 tests covering registry structure, all-cards-valid, cheap enemy requirements, known limitations, get_card(), benchmark types covered, by-ID lookup
  • BASELINE 6709→6772
  • Closes Phase C C1 — benchmark governance is now machine-enforceable via validate_registry()

@cschanhniem
cschanhniem merged commit 4366982 into main Jul 10, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant