cschanhniem · Jun 30, 2026 · Jun 30, 2026 · Jun 30, 2026 · Jun 30, 2026
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
@@ -39,7 +39,7 @@ The purpose of the added layers is not simulation theater. It is to improve whic
 | `openamp_foundry.scoring` | activity, safety, novelty, synthesis, ensemble scoring |
 | `openamp_foundry.selection` | ranking and diversity selection |
 | `openamp_foundry.evidence` | JSON certificate generation and validation |
-| `openamp_foundry.benchmark` | leakage checks and evaluation scaffolding |
+| `openamp_foundry.benchmark` | leakage checks, cluster-split benchmark, and evaluation scaffolding |
 | `openamp_foundry.generators` | safe, bounded toy candidate generation |
 | `openamp_foundry.simulation` | membrane/selectivity/stability proxy modeling (scaffolded) |
 

diff --git a/docs/BENCHMARKING.md b/docs/BENCHMARKING.md
@@ -20,6 +20,7 @@ Common artifacts:
 | Random baseline | Prove the pipeline is not noise |
 | Simple physicochemical baseline | Prove models beat charge/hydrophobicity alone |
 | Cluster split | Prevent near-duplicate leakage |
+| Cluster-aware CI | Honest bootstrap when positives contain near-duplicate families |
 | Time split | Test future generalization where metadata allows |
 | Hidden-positive recovery | Check whether known actives are ranked high |
 | Toxicity down-ranking | Ensure predicted risky candidates are penalized |

diff --git a/docs/BENCHMARK_CARD.md b/docs/BENCHMARK_CARD.md
@@ -1,7 +1,7 @@
 # Benchmark Card — OpenAMP Foundry v0.5.x
 
 > **Purpose:** Single-page summary of benchmark methodology, data, and metrics.
-> **Last updated:** 2026-06-29 (PR #110 expanded benchmark)
+> **Last updated:** 2026-07-01 (cluster-split benchmark added)
 
 ---
 
@@ -45,6 +45,33 @@ AUROC computed via Wilcoxon-Mann-Whitney statistic (concordant-pair enumeration)
 AUPRC via trapezoidal integration of precision-recall curve (pessimistic tie-breaking).
 Confidence intervals: percentile bootstrap (2000 resamples, seed=0).
 
+
+## Cluster-Split Analysis
+
+> Added 2026-07-01. 33 of 95 AMPs fall into 14 near-duplicate clusters (sim >= 0.70).
+
+| Metric | Pipeline | Phase3 |
+|--------|:--------:|:------:|
+| Full AUROC | 0.7832 | 0.7448 |
+| Standard CI₉₅ | 0.717–0.8423 | 0.6741–0.8118 |
+| **Cluster-aware CI₉₅** | **0.7061–0.8526** | **0.6591–0.8237** |
+| Representative AUROC (76 clusters) | 0.7607 | 0.7196 |
+| Representative CI₉₅ | 0.6854–0.8301 | 0.6372–0.7985 |
+| Held-out AUROC (19 near-dup AMPs) | 0.8734 | 0.8454 |
+
+The cluster-aware bootstrap resamples clusters (not individual sequences), producing
+an honest CI when near-duplicate families exist in the positive set. The standard
+bootstrap underestimates variance by treating near-identical sequences as independent.
+
+**Verdict:** Signal survives de-inflation. Cluster-aware CI lower bound (0.7061)
+stays above 0.65. Representative-only CI lower bound (0.6854) dips below 0.70 —
+an honest limitation. The pipeline has real but modest discriminative power.
+
+Near-duplicate clusters: magainin-1/2/3, protegrin-1/2/3, tachyplesin-I/II/polyphemusin-I,
+indolicidin/analog/lys-analog, BMAP-27/PMAP-36/bmap-fragment, aurein-1/3, cecropin-A/B,
+apidaecin-Ia/Ib, RsAFP-1/2, plectasin/eurocin, piscidin-1/3, buforin/buforin-II,
+dermaseptin-S1/S3-fragment, KWK-template/WKL.
+
 ## Known Biases
 
 | Bias | Impact | Mitigation |

diff --git a/docs/METRICS_CURRENT.md b/docs/METRICS_CURRENT.md
@@ -3,7 +3,7 @@
 > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
 > with this file, this file wins. Updated whenever benchmark/benchmark config changes.
 >
-> **Last updated:** 2026-06-29 (PR #110 expanded benchmark)
+> **Last updated:** 2026-07-01 (cluster-split benchmark added)
 > **Pipeline version:** v0.5.x
 > **Branch:** main
 
@@ -27,6 +27,44 @@
 | Recall@43 | 0.4211 | 40/95 positives in top 43 |
 | Interpretation | **STRONG** | AUROC > 0.70 gate passed |
 
+
+### Cluster-Split Benchmark (near-duplicate de-inflation)
+
+> Added 2026-07-01. The standard benchmark treats all 95 AMPs as independent samples.
+> 33 of 95 AMPs are in 14 near-duplicate clusters (sim >= 0.70): magainin-1/2/3,
+> protegrin-1/2/3, tachyplesin-I/II/polyphemusin-I, indolicidin/analog/lys-analog, etc.
+> The cluster-aware bootstrap resamples clusters (not sequences) to produce an honest CI.
+
+| Metric | Pipeline (pipeline.yaml) | Phase3 (phase3.yaml) |
+|--------|:-----------------------:|:-------------------:|
+| Full AUROC | 0.7832 | 0.7448 |
+| Standard CI₉₅ | 0.717–0.8423 | 0.6741–0.8118 |
+| **Cluster-aware CI₉₅** | **0.7061–0.8526** | **0.6591–0.8237** |
+| Representative AUROC (1/cluster) | 0.7607 | 0.7196 |
+| Representative CI₉₅ | 0.6854–0.8301 | 0.6372–0.7985 |
+| Held-out AUROC (19 near-dup AMPs) | 0.8734 | 0.8454 |
+| Independent clusters | 76 / 95 | 76 / 95 |
+| AMPs in multi-member clusters | 33 / 95 | 33 / 95 |
+| Multi-member clusters | 14 | 14 |
+
+**Key finding:** The cluster-aware CI (0.7061–0.8526) is wider than the standard CI
+(0.717–0.8423) on the upper end but the lower bound drops below the standard CI
+(0.7061 vs 0.717). The representative-only AUROC (0.7607, CI: 0.6854–0.8301) confirms
+the signal is not entirely driven by near-duplicate redundancy — but the CI lower
+bound (0.6854) dips below the 0.70 synthesis gate threshold. The pipeline passes the
+cluster-aware gate (CI lo > 0.65) but with less margin than the standard benchmark
+suggested. The held-out AUROC (0.8734) is high because held-out near-duplicates share
+composition features with their cluster representatives — this is expected and not
+evidence of generalisation to novel sequence space.
+
+**Verdict:** Signal survives near-duplicate de-inflation. The synthesis gate (AUROC >
+0.70) holds on the full set and the cluster-aware CI lower bound stays above 0.65.
+The representative-only CI lower bound (0.6854) crossing below 0.70 is an honest
+limitation: the pipeline has real but modest discriminative power, and the headline
+CI was slightly overconfident.
+
+Run: `openamp-foundry bench cluster-split`
+
 ### Historical baselines
 
 | Point | Benchmark | AUROC | Phase3 AUROC | Source |

diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md
@@ -165,7 +165,7 @@ review (2026-06-28). Progress on these would materially raise breakthrough proba
 
 | Gap | Why it matters | Effort estimate |
 |-----|----------------|-----------------|
-| ~~Large-scale benchmark (≥ 500 AMPs vs composition-matched decoys, cluster-split)~~ | ~~Current AUROC 0.8420 measured on 43+44 demo set (n=87, CI₉₅: 0.76–0.91); may not generalise~~ | **Partial** (PR #110: expanded to 95 AMPs + 96 decoys, n=191, AUROC=0.7832, CI₉₅: 0.72–0.84; 52 new public-domain AMPs from 12 taxonomic classes; covers defensins, proline-rich, lantibiotics — a more honest estimate. 500+ target still deferred to v1.0+) |
+| ~~Large-scale benchmark (≥ 500 AMPs vs composition-matched decoys, cluster-split)~~ | ~~Current AUROC 0.8420 measured on 43+44 demo set (n=87, CI₉₅: 0.76–0.91); may not generalise~~ | **Partial** (PR #110: expanded to 95 AMPs + 96 decoys, n=191, AUROC=0.7832, CI₉₅: 0.72–0.84; 52 new public-domain AMPs from 12 taxonomic classes; covers defensins, proline-rich, lantibiotics — a more honest estimate. Cluster-split benchmark added: cluster-aware CI 0.7061–0.8526, representative AUROC 0.7607. 500+ target still deferred to v1.0+) |
 | External predictor ensemble adapters (CAMPR4, AMPScanner, dbAMP, AntiCP2, Macrel) | Independent second opinions on activity; required for scientific credibility. Manual web-submission checklist at `outputs/external_predict_checklist.md`. Macrel v1.6.0 CLI ONNX bug documented PR #77 — all sequences (incl. canonical AMPs magainin-2, LL-37) misclassified as NAMP; use web server at big-data-biology.org/software/macrel. AMPlify omitted: GPU/ONNX env incompatible with current deps | Medium (checklist generated; web submissions pending) |
 | ~~True novelty check against APD3, DRAMP v3.0, dbAMP~~ | ~~Current novelty scored against 45-sequence seed set only; may overestimate novelty~~ | **Done** (v0.5.7: BioPython BLOSUM62 local alignment vs 27,234 AMPs from APD6+DRAMP+UniProt; Wave 0.5 novelty corrected from 53/60 RELATED_NOVEL → 39 CLOSE_RELATIVE + 19 KNOWN_VARIANT + 1 RELATED_NOVEL) |
 | ~~AUPRC alongside AUROC~~ | ~~Better metric for class-imbalanced AMP datasets~~ | **Done** (PR #58; updated PR #72) — pipeline AUPRC = 0.8627 |