Repository navigation
feat(bench): expert-vs-ensemble ablation benchmark - #112
Merged
Merged
Conversation
… AUROC 0.736 < ensemble 0.783 Adds run_expert_ablation_benchmark() to benchmark/retrospective.py that scores all 95 AMPs + 96 decoys with both the simple ensemble and the 7-component expert composite (scoring/expert.py), computing AUROC + bootstrap CI for both plus per-component AUROC to attribute which axes carry signal. Key finding: the expert composite scores LOWER than the ensemble (delta -0.0472 AUROC). Three expert components are anti-signal: safety (0.349), serum_stability (0.223), synthesis (0.423) score known AMPs worse than random decoys because real AMPs have extreme biophysical properties these scorers penalise. Only activity (0.814) and selectivity_proxy (0.773) are signal-bearing. Per AGENTS.md §7.5 (No simulation theater): every added modeling layer must justify itself against simpler baselines. The expert composite does not earn its keep for AMP/non-AMP triage. The ensemble remains the primary synthesis gate. New CLI: 'openamp-foundry bench expert-ablation' New Makefile targets: bench-expert-ablation, bench-cluster-split 14 new tests in test_expert_ablation.py (1435 total, 7 skipped) Docs updated: METRICS_CURRENT.md, BENCHMARK_CARD.md, ROADMAP.md, ARCHITECTURE.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the first expert-vs-ensemble ablation benchmark, closing a benchmark honesty gap: the 7-component expert composite scorer was used for generation but never benchmarked against the simpler ensemble.
Key Finding
The expert composite scores LOWER than the ensemble (AUROC 0.736 vs 0.783, delta -0.047). Three components (safety 0.349, serum_stability 0.223, synthesis 0.423) are anti-signal: known AMPs score worse than random decoys because real AMPs have extreme biophysical properties these scorers penalise. The ensemble remains the primary synthesis gate.
Changes
Verification