[cschanhniem]

Chosen bottleneck

Bottleneck: No CI benchmark regression gate — AUROC can silently degrade as scorers or weights change. The committed outputs/metrics_snapshot.json existed but was never compared against in CI.

Why it matters

The scratch-built scoring pipeline's headline metric is standard.auroc = 0.7832. Without a regression gate, any PR that changes scorers, weights, or defaults can silently degrade this without being caught. The hidden-active gate (EF >= 1.5) catches a different failure mode. The AUROC gate catches general scoring quality regression.

Changes

New: scripts/benchmark_gate.py

• Runs all 6 benchmarks via build_metrics_snapshot() (same as make metrics-snapshot)
• Compares standard.auroc and phase3.auroc against committed baseline
• Exits 0 (PASS), 1 (FAIL — regression > tolerance), 2 (no baseline file)
• Prints human-readable diff report
• Exposed via make bench-gate with tolerance 0.02

New: tests/test_benchmark_gate.py (13 tests)

• Unit tests for run_benchmark_gate() comparison logic (9 tests)
• CLI integration tests with mocked snapshots (2 tests)
• CLI with real baseline (1 test, n_bootstrap=50)
• Missing baseline exits 2 (1 test)

Updated: .github/workflows/ci.yml

• New CI step runs make bench-gate between leakage check and hidden-active gate
• Fails PR if AUROC drops > 0.02

Updated: docs/50_LOOP_PLAN.md

• Loop 5 marked complete
• Current position advanced to Loop 6

Updated: Makefile

• bench-gate target added
• help text includes bench-gate description

Verification

• 1682 tests passing (+13 new)
• make bench-gate exits 0 (AUROC matches baseline exactly: 0.7832 → 0.7832)
• Full test suite passes with 3 pre-existing warnings

Risks / uncertainties

• The CI step adds ~10-30s to CI runtime (6 benchmarks with n_bootstrap=200)
• The gate uses n_bootstrap=200 (default) in CI; the committed baseline uses 2000. This is fine — the AUROC point estimate is stable at 200 iterations for this dataset size.
• If a legitimate refactor changes AUROC by > 0.02, the baseline must be updated intentionally (commit new outputs/metrics_snapshot.json). This is by design.

Recommended next loop

Loop 6: Deterministic make regenerate-all script that rebuilds all derived outputs from versioned inputs.