Skip to content

feat: Phase C C2 benchmark card schema (BMC-) -- prevents incomplete benchmark docs - #879

Merged
cschanhniem merged 1 commit into
mainfrom
feat/c2-benchmark-card-schema
Jul 10, 2026
Merged

cschanhniem merged 1 commit into
mainfrom
feat/c2-benchmark-card-schema

Conversation

@cschanhniem

Copy link
Copy Markdown
Collaborator

Summary

  • New schema: BenchmarkCard (BMC-) in src/openamp_foundry/evidence/benchmark_card.py
  • 12 fields: bmc_id, pipeline_version, benchmark_name, measurement_target, split_strategy, cheap_enemy_baselines, evaluation_metrics, known_limitations, deprecated, created_date, last_updated_date, notes
  • Controlled vocabs: VALID_MEASUREMENT_TARGETS (10), VALID_SPLIT_STRATEGIES (10), VALID_EVALUATION_METRICS (12)
  • cheap_enemy_baselines ≥1 required (warns if <2): benchmark without declared cheap enemies cannot prove method beats trivial rules
  • known_limitations ≥1 required: every benchmark has limitations; documenting zero is not allowed
  • deprecated=True requires non-empty notes (explanation of why deprecated)
  • 2 warnings: <2 cheap enemies, <2 known limitations
  • 63 tests covering: valid records, ID/version, measurement_target+split, cheap enemy rules, eval metrics+limitations, deprecated+dates, validate_dict edge cases
  • BASELINE 6646→6709
  • Closes Phase C C2 — benchmark documentation is now machine-checkable; incomplete benchmark docs are blocked at schema level

@cschanhniem
cschanhniem merged commit 3e5aa13 into main Jul 10, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant