feat(bench): expert-vs-ensemble ablation benchmark

Makefile

@@ -1,4 +1,4 @@
-.PHONY: help demo test lint ci clean bench-leakage bench-baseline bench-hidden-active generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-predict pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter
+.PHONY: help demo test lint ci clean bench-leakage bench-baseline bench-hidden-active bench-cluster-split bench-expert-ablation generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-predict pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter
 
 PYTHON := $(shell [ -f .venv/bin/python ] && echo .venv/bin/python || echo python3)
 PYTEST  := $(shell [ -f .venv/bin/pytest ] && echo .venv/bin/pytest || echo pytest)
@@ -28,6 +28,8 @@ help:
 	@echo "  make validate-scoring-strict  AUROC with scrambled-decoy strict benchmark"
 	@echo "  make bench-leakage            Check for near-duplicates between candidates and refs"
 	@echo "  make bench-baseline           Hidden-positive recovery benchmark (demo set)"
+	@echo "  make bench-cluster-split      Cluster-aware bootstrap CI (de-inflates near-duplicates)"
+	@echo "  make bench-expert-ablation    Expert composite vs ensemble ablation (honesty check)"
 	@echo "  make bench-hidden-active      Hidden-positive recovery on mixed benchmark set"
 	@echo ""
 	@echo "  make wave0-5-gate-check     Run Wave 0.5 gates W0.5-1 through W0.5-7"
@@ -134,6 +136,18 @@ validate-scoring-phase3:
 		--benchmark-type standard \
 		--out outputs/validate_scoring_report_phase3.json
 
+bench-cluster-split:
+	PYTHONPATH=src $(PYTHON) -m openamp_foundry.cli bench cluster-split 
+		--amp-csv examples/validation/known_amps.csv 
+		--decoy-csv examples/validation/random_background.csv 
+		--out outputs/cluster_split_report.json
+
+bench-expert-ablation:
+	PYTHONPATH=src $(PYTHON) -m openamp_foundry.cli bench expert-ablation 
+		--amp-csv examples/validation/known_amps.csv 
+		--decoy-csv examples/validation/random_background.csv 
+		--out outputs/expert_ablation_report.json
+
 validate-scoring-strict:
 	PYTHONPATH=src $(PYTHON) -m openamp_foundry.cli validate-scoring \
 		--amp-csv examples/validation/known_amps.csv \

docs/ARCHITECTURE.md

@@ -39,7 +39,7 @@ The purpose of the added layers is not simulation theater. It is to improve whic
 | `openamp_foundry.scoring` | activity, safety, novelty, synthesis, ensemble scoring |
 | `openamp_foundry.selection` | ranking and diversity selection |
 | `openamp_foundry.evidence` | JSON certificate generation and validation |
-| `openamp_foundry.benchmark` | leakage checks, cluster-split benchmark, and evaluation scaffolding |
+| `openamp_foundry.benchmark` | leakage checks, cluster-split benchmark, expert ablation benchmark, and evaluation scaffolding |
 | `openamp_foundry.generators` | safe, bounded toy candidate generation |
 | `openamp_foundry.simulation` | membrane/selectivity/stability proxy modeling (scaffolded) |
 

docs/BENCHMARK_CARD.md

@@ -72,6 +72,43 @@ indolicidin/analog/lys-analog, BMAP-27/PMAP-36/bmap-fragment, aurein-1/3, cecrop
 apidaecin-Ia/Ib, RsAFP-1/2, plectasin/eurocin, piscidin-1/3, buforin/buforin-II,
 dermaseptin-S1/S3-fragment, KWK-template/WKL.
 
+
+## Expert Ablation (added 2026-07-01)
+
+> Run: `make bench-expert-ablation`
+
+The expert composite scorer (`scoring/expert.py`, 7 components) was benchmarked
+against the simple ensemble to test whether its additional complexity earns its keep.
+
+| Scorer | AUROC | CI₉₅ | Delta vs ensemble |
+|--------|:-----:|:----:|:-----------------:|
+| **Ensemble** (pipeline.yaml) | **0.7832** | 0.717–0.8423 | — |
+| Expert composite | 0.7360 | 0.6604–0.8037 | **−0.0472** |
+| **Ensemble** (phase3.yaml) | **0.7448** | 0.6741–0.8118 | — |
+| Expert composite | 0.7360 | 0.6604–0.8037 | −0.0088 |
+
+**Per-component AUROC:**
+
+| Component | AUROC | Signal class |
+|-----------|:-----:|:------------:|
+| activity | 0.8137 | Signal-bearing |
+| selectivity_proxy | 0.7729 | Signal-bearing |
+| hinge_selectivity | 0.5180 | Near-zero |
+| boman_activity | 0.4620 | Near-zero |
+| synthesis | 0.4228 | Anti-signal |
+| safety | 0.3487 | Anti-signal |
+| serum_stability | 0.2231 | Anti-signal |
+
+**Verdict:** Expert composite does NOT improve AMP-vs-decoy discrimination. Three
+components (safety, serum_stability, synthesis) are anti-signal: known AMPs score
+worse than random decoys on these axes because real AMPs have extreme biophysical
+properties (high charge, high hydrophobic moment, many protease sites) that these
+scorers penalise. The ensemble remains the primary synthesis gate.
+
+**Caveat:** This tests binary AMP/non-AMP discrimination only. The expert components
+may add value for within-AMP ranking (selective vs hemolytic) — a within-AMP benchmark
+is the appropriate next test.
+
 ## Known Biases
 
 | Bias | Impact | Mitigation |

docs/METRICS_CURRENT.md

@@ -163,11 +163,77 @@ Machine-readable: `outputs/wave0_5_external_predict_results.csv`, `outputs/wave0
 
 ---
 
+---
+
+## Expert Ablation Benchmark (v0.5.x — added 2026-07-01)
+
+> The expert composite scorer (`scoring/expert.py`) adds four components beyond the
+> simple ensemble: selectivity, serum stability, helix-hinge, and k-mer motif novelty.
+> This ablation tests whether those additions improve binary AMP-vs-decoy discrimination
+> or are complexity that does not earn its keep.
+>
+> Run: `make bench-expert-ablation`
+
+| Metric | Pipeline (pipeline.yaml) | Phase3 (phase3.yaml) |
+|--------|:-----------------------:|:-------------------:|
+| Ensemble AUROC | 0.7832 | 0.7448 |
+| Ensemble CI₉₅ | 0.717–0.8423 | 0.6741–0.8118 |
+| Expert composite AUROC | 0.7360 | 0.7360 |
+| Expert CI₉₅ | 0.6604–0.8037 | 0.6604–0.8037 |
+| **Delta (expert − ensemble)** | **−0.0472** | **−0.0088** |
+| Verdict | Expert LOWER | Within ±0.02 |
+
+**Per-component AUROC** (pipeline config, n=191):
+
+| Component | AUROC | Above random | Classification |
+|-----------|:-----:|:------------:|:--------------:|
+| activity | 0.8137 | +0.3137 | **Signal-bearing** |
+| selectivity_proxy | 0.7729 | +0.2729 | **Signal-bearing** |
+| hinge_selectivity | 0.5180 | +0.0180 | Near-zero |
+| novelty | 0.5000 | 0.0000 | Near-zero |
+| motif_novelty | 0.5000 | 0.0000 | Near-zero |
+| boman_activity | 0.4620 | −0.0380 | Near-zero |
+| synthesis | 0.4228 | −0.0772 | **Anti-signal** |
+| safety | 0.3487 | −0.1513 | **Anti-signal** |
+| serum_stability | 0.2231 | −0.2769 | **Anti-signal** |
+
+**Key finding:** The expert composite scores **lower** than the simple ensemble on
+binary AMP-vs-decoy discrimination (delta = −0.0472). Three expert components are
+anti-signal: `safety`, `serum_stability`, and `synthesis` score random decoys higher
+than known AMPs. This is expected and not a bug: real AMPs have high hydrophobic
+moment and charge (features the safety scorer penalises), many interior protease
+sites (low serum stability), and often contain repeat runs or aggregation-prone
+segments (low synthesis feasibility). Random decoys drawn from Swiss-Prot
+frequencies tend to have more moderate biophysical properties.
+
+**What this means for the pipeline:**
+
+1. The expert composite should NOT replace the ensemble for AMP/non-AMP triage.
+2. The ensemble (activity + safety + synthesis + novelty + Boman) remains the
+   primary synthesis gate, as it has higher discriminative power.
+3. The expert components may still add value for **within-AMP ranking** (selectivity
+   and safety differentiation among candidates that already pass the activity gate)
+   — but this has not been demonstrated and should not be assumed.
+4. The `boman_activity` scorer (AUROC 0.462, below random) does not discriminate
+   AMPs from random decoys on the expanded benchmark. It contributes to the ensemble
+   only through the disagreement signal, not through independent discrimination.
+5. `motif_novelty` and `novelty` are 0.5 by construction (no k-mer index, no
+   references in this benchmark) — they are correctly neutral, not noise.
+
+**Honest limitation:** This benchmark measures binary AMP-vs-decoy discrimination
+only. The expert composite's selectivity, safety, and synthesis components are
+designed for within-AMP candidate differentiation, not for separating AMPs from
+non-AMPs. A within-AMP ranking benchmark (comparing selective vs hemolytic AMPs)
+would be a more appropriate test for those components and is a candidate for the
+next loop.
+
+---
+
 ## Test Suite
 
 | Metric | Value |
 |--------|-------|
-| Total tests | 1343 |
+| Total tests | 1435 |
 | Coverage (branch) | 99% (6 CLI guard lines only) |
 | Source modules at 100% | All pipeline, QC, scoring modules |
 
@@ -195,4 +261,5 @@ Machine-readable: `outputs/wave0_5_external_predict_results.csv`, `outputs/wave0
 | 2026-06-29 | Wave 0.5b: 23-candidate safety-optimized shortlist (SEED-020–024, no aromatics) | OpenAMP Wave 0.5b |
 | 2026-06-29 | External predictor results filled from wave05_combined_consensus.csv; all 7 gates PASS | OpenAMP Wave 0.5 |
 | 2026-06-29 | Wave 0.5 scaffold diversification — 24-candidate Wave 1 panel across 14 families | OpenAMP Wave 0.5 |
+| 2026-07-01 | Expert ablation benchmark added: expert composite AUROC 0.736 vs ensemble 0.7832 (delta −0.0472); 3 components anti-signal; ensemble remains primary gate | OpenAMP loop |
 | 2026-06-29 | Initial — expanded benchmark (PR #110) | OpenAMP CI |

docs/ROADMAP.md

@@ -136,6 +136,16 @@ Implemented during the pre-wet-lab improvement loop (PRs #31–#54):
 - Wave 0.5 gates W0.5-1 through W0.5-7 implemented (`src/openamp_foundry/gates/wave0_5_gate_checker.py`)
 - Discovery probability impact: from 7 correlated families → 14 independent families; reduces correlated-failure risk for wet-lab batch
 
+## v0.5.8 — Expert Ablation Benchmark ✓ (2026-07-01)
+
+- `run_expert_ablation_benchmark()` in `benchmark/retrospective.py` — scores all 95 AMPs + 96 decoys with both the simple ensemble and the 7-component expert composite
+- `bench expert-ablation` CLI command + `make bench-expert-ablation` target
+- Per-component AUROC attribution: identifies signal-bearing (activity 0.814, selectivity 0.773), near-zero (hinge, novelty, boman, motif), and anti-signal (safety 0.349, serum_stability 0.223, synthesis 0.423) components
+- **Key finding:** Expert composite AUROC 0.736 < ensemble AUROC 0.783 (delta −0.047). The added complexity does NOT improve AMP-vs-decoy discrimination. Three components are anti-signal because real AMPs have extreme biophysical properties that safety/stability/synthesis scorers penalise.
+- Ensemble remains primary synthesis gate. Expert composite may add value for within-AMP ranking (not yet demonstrated).
+- `make bench-cluster-split` Makefile target added (was missing despite CLI command existing)
+- 14 new tests in `test_expert_ablation.py`; 1435 total tests
+
 ## v1.0 — Validated dry-lab-to-wet-lab loop
 
 - independently reviewed assay batch (expert_review.yml GitHub issue template);