diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index a96484d3..287e21c4 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -39,7 +39,7 @@ The purpose of the added layers is not simulation theater. It is to improve whic | `openamp_foundry.scoring` | activity, safety, novelty, synthesis, ensemble scoring | | `openamp_foundry.selection` | ranking and diversity selection | | `openamp_foundry.evidence` | JSON certificate generation and validation | -| `openamp_foundry.benchmark` | leakage checks and evaluation scaffolding | +| `openamp_foundry.benchmark` | leakage checks, cluster-split benchmark, and evaluation scaffolding | | `openamp_foundry.generators` | safe, bounded toy candidate generation | | `openamp_foundry.simulation` | membrane/selectivity/stability proxy modeling (scaffolded) | diff --git a/docs/BENCHMARKING.md b/docs/BENCHMARKING.md index ca1313ab..699c8060 100644 --- a/docs/BENCHMARKING.md +++ b/docs/BENCHMARKING.md @@ -20,6 +20,7 @@ Common artifacts: | Random baseline | Prove the pipeline is not noise | | Simple physicochemical baseline | Prove models beat charge/hydrophobicity alone | | Cluster split | Prevent near-duplicate leakage | +| Cluster-aware CI | Honest bootstrap when positives contain near-duplicate families | | Time split | Test future generalization where metadata allows | | Hidden-positive recovery | Check whether known actives are ranked high | | Toxicity down-ranking | Ensure predicted risky candidates are penalized | diff --git a/docs/BENCHMARK_CARD.md b/docs/BENCHMARK_CARD.md index 7433472a..871b844a 100644 --- a/docs/BENCHMARK_CARD.md +++ b/docs/BENCHMARK_CARD.md @@ -1,7 +1,7 @@ # Benchmark Card — OpenAMP Foundry v0.5.x > **Purpose:** Single-page summary of benchmark methodology, data, and metrics. -> **Last updated:** 2026-06-29 (PR #110 expanded benchmark) +> **Last updated:** 2026-07-01 (cluster-split benchmark added) --- @@ -45,6 +45,33 @@ AUROC computed via Wilcoxon-Mann-Whitney statistic (concordant-pair enumeration) AUPRC via trapezoidal integration of precision-recall curve (pessimistic tie-breaking). Confidence intervals: percentile bootstrap (2000 resamples, seed=0). + +## Cluster-Split Analysis + +> Added 2026-07-01. 33 of 95 AMPs fall into 14 near-duplicate clusters (sim >= 0.70). + +| Metric | Pipeline | Phase3 | +|--------|:--------:|:------:| +| Full AUROC | 0.7832 | 0.7448 | +| Standard CI₉₅ | 0.717–0.8423 | 0.6741–0.8118 | +| **Cluster-aware CI₉₅** | **0.7061–0.8526** | **0.6591–0.8237** | +| Representative AUROC (76 clusters) | 0.7607 | 0.7196 | +| Representative CI₉₅ | 0.6854–0.8301 | 0.6372–0.7985 | +| Held-out AUROC (19 near-dup AMPs) | 0.8734 | 0.8454 | + +The cluster-aware bootstrap resamples clusters (not individual sequences), producing +an honest CI when near-duplicate families exist in the positive set. The standard +bootstrap underestimates variance by treating near-identical sequences as independent. + +**Verdict:** Signal survives de-inflation. Cluster-aware CI lower bound (0.7061) +stays above 0.65. Representative-only CI lower bound (0.6854) dips below 0.70 — +an honest limitation. The pipeline has real but modest discriminative power. + +Near-duplicate clusters: magainin-1/2/3, protegrin-1/2/3, tachyplesin-I/II/polyphemusin-I, +indolicidin/analog/lys-analog, BMAP-27/PMAP-36/bmap-fragment, aurein-1/3, cecropin-A/B, +apidaecin-Ia/Ib, RsAFP-1/2, plectasin/eurocin, piscidin-1/3, buforin/buforin-II, +dermaseptin-S1/S3-fragment, KWK-template/WKL. + ## Known Biases | Bias | Impact | Mitigation | diff --git a/docs/METRICS_CURRENT.md b/docs/METRICS_CURRENT.md index 7276868e..31a2e496 100644 --- a/docs/METRICS_CURRENT.md +++ b/docs/METRICS_CURRENT.md @@ -3,7 +3,7 @@ > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees > with this file, this file wins. Updated whenever benchmark/benchmark config changes. > -> **Last updated:** 2026-06-29 (PR #110 expanded benchmark) +> **Last updated:** 2026-07-01 (cluster-split benchmark added) > **Pipeline version:** v0.5.x > **Branch:** main @@ -27,6 +27,44 @@ | Recall@43 | 0.4211 | 40/95 positives in top 43 | | Interpretation | **STRONG** | AUROC > 0.70 gate passed | + +### Cluster-Split Benchmark (near-duplicate de-inflation) + +> Added 2026-07-01. The standard benchmark treats all 95 AMPs as independent samples. +> 33 of 95 AMPs are in 14 near-duplicate clusters (sim >= 0.70): magainin-1/2/3, +> protegrin-1/2/3, tachyplesin-I/II/polyphemusin-I, indolicidin/analog/lys-analog, etc. +> The cluster-aware bootstrap resamples clusters (not sequences) to produce an honest CI. + +| Metric | Pipeline (pipeline.yaml) | Phase3 (phase3.yaml) | +|--------|:-----------------------:|:-------------------:| +| Full AUROC | 0.7832 | 0.7448 | +| Standard CI₉₅ | 0.717–0.8423 | 0.6741–0.8118 | +| **Cluster-aware CI₉₅** | **0.7061–0.8526** | **0.6591–0.8237** | +| Representative AUROC (1/cluster) | 0.7607 | 0.7196 | +| Representative CI₉₅ | 0.6854–0.8301 | 0.6372–0.7985 | +| Held-out AUROC (19 near-dup AMPs) | 0.8734 | 0.8454 | +| Independent clusters | 76 / 95 | 76 / 95 | +| AMPs in multi-member clusters | 33 / 95 | 33 / 95 | +| Multi-member clusters | 14 | 14 | + +**Key finding:** The cluster-aware CI (0.7061–0.8526) is wider than the standard CI +(0.717–0.8423) on the upper end but the lower bound drops below the standard CI +(0.7061 vs 0.717). The representative-only AUROC (0.7607, CI: 0.6854–0.8301) confirms +the signal is not entirely driven by near-duplicate redundancy — but the CI lower +bound (0.6854) dips below the 0.70 synthesis gate threshold. The pipeline passes the +cluster-aware gate (CI lo > 0.65) but with less margin than the standard benchmark +suggested. The held-out AUROC (0.8734) is high because held-out near-duplicates share +composition features with their cluster representatives — this is expected and not +evidence of generalisation to novel sequence space. + +**Verdict:** Signal survives near-duplicate de-inflation. The synthesis gate (AUROC > +0.70) holds on the full set and the cluster-aware CI lower bound stays above 0.65. +The representative-only CI lower bound (0.6854) crossing below 0.70 is an honest +limitation: the pipeline has real but modest discriminative power, and the headline +CI was slightly overconfident. + +Run: `openamp-foundry bench cluster-split` + ### Historical baselines | Point | Benchmark | AUROC | Phase3 AUROC | Source | diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 1607cea9..4229adc1 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -165,7 +165,7 @@ review (2026-06-28). Progress on these would materially raise breakthrough proba | Gap | Why it matters | Effort estimate | |-----|----------------|-----------------| -| ~~Large-scale benchmark (≥ 500 AMPs vs composition-matched decoys, cluster-split)~~ | ~~Current AUROC 0.8420 measured on 43+44 demo set (n=87, CI₉₅: 0.76–0.91); may not generalise~~ | **Partial** (PR #110: expanded to 95 AMPs + 96 decoys, n=191, AUROC=0.7832, CI₉₅: 0.72–0.84; 52 new public-domain AMPs from 12 taxonomic classes; covers defensins, proline-rich, lantibiotics — a more honest estimate. 500+ target still deferred to v1.0+) | +| ~~Large-scale benchmark (≥ 500 AMPs vs composition-matched decoys, cluster-split)~~ | ~~Current AUROC 0.8420 measured on 43+44 demo set (n=87, CI₉₅: 0.76–0.91); may not generalise~~ | **Partial** (PR #110: expanded to 95 AMPs + 96 decoys, n=191, AUROC=0.7832, CI₉₅: 0.72–0.84; 52 new public-domain AMPs from 12 taxonomic classes; covers defensins, proline-rich, lantibiotics — a more honest estimate. Cluster-split benchmark added: cluster-aware CI 0.7061–0.8526, representative AUROC 0.7607. 500+ target still deferred to v1.0+) | | External predictor ensemble adapters (CAMPR4, AMPScanner, dbAMP, AntiCP2, Macrel) | Independent second opinions on activity; required for scientific credibility. Manual web-submission checklist at `outputs/external_predict_checklist.md`. Macrel v1.6.0 CLI ONNX bug documented PR #77 — all sequences (incl. canonical AMPs magainin-2, LL-37) misclassified as NAMP; use web server at big-data-biology.org/software/macrel. AMPlify omitted: GPU/ONNX env incompatible with current deps | Medium (checklist generated; web submissions pending) | | ~~True novelty check against APD3, DRAMP v3.0, dbAMP~~ | ~~Current novelty scored against 45-sequence seed set only; may overestimate novelty~~ | **Done** (v0.5.7: BioPython BLOSUM62 local alignment vs 27,234 AMPs from APD6+DRAMP+UniProt; Wave 0.5 novelty corrected from 53/60 RELATED_NOVEL → 39 CLOSE_RELATIVE + 19 KNOWN_VARIANT + 1 RELATED_NOVEL) | | ~~AUPRC alongside AUROC~~ | ~~Better metric for class-imbalanced AMP datasets~~ | **Done** (PR #58; updated PR #72) — pipeline AUPRC = 0.8627 | diff --git a/src/openamp_foundry/benchmark/retrospective.py b/src/openamp_foundry/benchmark/retrospective.py index ed66cdc3..c6c87adc 100644 --- a/src/openamp_foundry/benchmark/retrospective.py +++ b/src/openamp_foundry/benchmark/retrospective.py @@ -248,3 +248,310 @@ def run_retrospective_benchmark( "controls. Wet-lab validation remains mandatory." ), } + + +def _cluster_aware_bootstrap_auroc_ci( + pos_scores: list[float], + neg_scores: list[float], + cluster_assignments: list[int], + n_bootstrap: int = 2000, + seed: int = 0, +) -> dict: + """Bootstrap 95% CI for AUROC that resamples clusters, not individual sequences. + + Standard bootstrap resampling treats every sample as independent. When the + positive set contains near-duplicate sequences (e.g. magainin-1, magainin-2, + magainin-3), this underestimates variance and produces artificially narrow CIs. + + Cluster-aware bootstrap: + 1. Group positive scores by cluster ID. + 2. Resample clusters with replacement (not individual scores). + 3. Flatten the resampled cluster scores into a new positive score list. + 4. Compute AUROC on the resampled set. + + Args: + pos_scores: ensemble scores of positive (AMP) sequences. + neg_scores: ensemble scores of negative (decoy) sequences. + cluster_assignments: cluster ID for each positive sequence (same order as pos_scores). + Negative sequences are always treated as independent (random decoys have no + near-duplicate clusters by construction). + n_bootstrap: number of bootstrap resamples. + seed: RNG seed for reproducibility. + + Returns: + Dict with mean, ci_lo, ci_hi, n_bootstrap. + """ + import random as _random + rng = _random.Random(seed) + n_pos = len(pos_scores) + n_neg = len(neg_scores) + if n_pos == 0 or n_neg == 0: + return {"mean": 0.5, "ci_lo": 0.5, "ci_hi": 0.5, "n_bootstrap": 0} + + # Group positive scores by cluster + clusters: dict[int, list[float]] = {} + for score, cid in zip(pos_scores, cluster_assignments): + clusters.setdefault(cid, []).append(score) + + unique_cluster_ids = list(clusters.keys()) + n_clusters = len(unique_cluster_ids) + + samples = [] + for _ in range(n_bootstrap): + # Resample clusters with replacement + resampled_pos: list[float] = [] + for _ in range(n_clusters): + cid = rng.choice(unique_cluster_ids) + resampled_pos.extend(clusters[cid]) + # Resample negatives normally (they are independent) + neg_s = [rng.choice(neg_scores) for _ in range(n_neg)] + samples.append(_auc_wilcoxon(resampled_pos, neg_s)) + + samples.sort() + lo_idx = int(0.025 * n_bootstrap) + hi_idx = int(0.975 * n_bootstrap) + return { + "mean": round(sum(samples) / len(samples), 4), + "ci_lo": round(samples[lo_idx], 4), + "ci_hi": round(samples[hi_idx], 4), + "n_bootstrap": n_bootstrap, + } + + +def run_cluster_split_benchmark( + amp_csv: str | Path, + decoy_csv: str | Path, + config_path: str | Path = "configs/pipeline.yaml", + similarity_threshold: float = 0.70, + n_bootstrap: int = 2000, +) -> dict: + """Cluster-split retrospective benchmark: honest AUROC with leakage control. + + The standard benchmark (run_retrospective_benchmark) scores all AMPs against + all decoys and computes AUROC + bootstrap CI. This is the primary synthesis gate. + + However, the standard benchmark treats near-duplicate AMPs as independent + samples. When the positive set contains families of nearly identical sequences + (e.g. magainin-1/2/3 at 91-96% identity, protegrin-1/2/3 at 89-94% identity), + two problems arise: + + 1. **Inflated AUROC**: near-duplicates that share composition features will + all score similarly, so getting one right means getting all right. This + inflates the apparent concordance count. + 2. **Artificially narrow CI**: bootstrap resampling treats each sequence as + an independent draw. Resampling a cluster of 3 near-identical magainins + as 3 independent observations underestimates the true variance. + + This benchmark addresses both: + + - **Cluster-aware AUROC**: scores all AMPs and decoys, but computes the + bootstrap CI by resampling clusters (not individual sequences). + - **Held-out recovery**: clusters the AMP set, holds out near-duplicate + members, and reports AUROC on the held-out set vs decoys. This tests + whether the scorer generalises beyond the cluster representatives. + + The cluster-aware CI is the primary honesty improvement. The held-out + recovery is a secondary diagnostic. + + Args: + amp_csv: CSV of known AMPs (id, sequence, family, reference, label). + decoy_csv: CSV of decoy peptides (id, sequence, ...). + config_path: Pipeline config for scoring weights. + similarity_threshold: Normalized similarity threshold for clustering (default 0.70). + n_bootstrap: Bootstrap resample count. + + Returns: + Dict with full cluster-split benchmark results. + """ + from openamp_foundry.benchmark.splits import cluster_by_similarity + from openamp_foundry.features.physchem import compute_features + from openamp_foundry.scoring.activity import activity_likeness_score + from openamp_foundry.scoring.boman import boman_activity_score + from openamp_foundry.scoring.ensemble import ensemble_score + from openamp_foundry.scoring.novelty import novelty_score + from openamp_foundry.scoring.safety import safety_score + from openamp_foundry.scoring.synthesis import synthesis_feasibility_score + from openamp_foundry.config import load_config + + config = load_config(config_path) + weights = config["weights"] + + # Load and score all AMPs + amp_rows = [] + with open(amp_csv, newline="", encoding="utf-8") as f: + for row in csv.DictReader(f): + seq = row["sequence"].strip().upper() + seq_id = row["id"] + valid = all(aa in "ACDEFGHIKLMNPQRSTVWY" for aa in seq) + features = compute_features(seq) + act = activity_likeness_score(features) + safe = safety_score(features) + synth = synthesis_feasibility_score(features, valid_sequence=valid) + nov, _ = novelty_score(seq, []) + boman_act = boman_activity_score(seq) + raw_scores = { + "activity": act, "safety": safe, + "synthesis": synth, "novelty": nov, + "boman_activity": boman_act, + "disagreement": abs(act - boman_act), + } + raw_scores["ensemble"] = ensemble_score(raw_scores, weights) + amp_rows.append({ + "id": seq_id, + "sequence": seq, + "family": row.get("family", ""), + "label": 1, + "ensemble": raw_scores["ensemble"], + }) + + # Load and score all decoys + decoy_rows = [] + with open(decoy_csv, newline="", encoding="utf-8") as f: + for row in csv.DictReader(f): + seq = row["sequence"].strip().upper() + seq_id = row["id"] + valid = all(aa in "ACDEFGHIKLMNPQRSTVWY" for aa in seq) + features = compute_features(seq) + act = activity_likeness_score(features) + safe = safety_score(features) + synth = synthesis_feasibility_score(features, valid_sequence=valid) + nov, _ = novelty_score(seq, []) + boman_act = boman_activity_score(seq) + raw_scores = { + "activity": act, "safety": safe, + "synthesis": synth, "novelty": nov, + "boman_activity": boman_act, + "disagreement": abs(act - boman_act), + } + raw_scores["ensemble"] = ensemble_score(raw_scores, weights) + decoy_rows.append({ + "id": seq_id, + "sequence": seq, + "label": 0, + "ensemble": raw_scores["ensemble"], + }) + + # Cluster the AMP set + amp_seqs = [r["sequence"] for r in amp_rows] + clusters = cluster_by_similarity(amp_seqs, threshold=similarity_threshold) + + # Map each AMP index to its cluster ID + index_to_cluster: dict[int, int] = {} + for cluster_id, cluster_members in enumerate(clusters): + for member_idx in cluster_members: + index_to_cluster[member_idx] = cluster_id + + n_clusters = len(clusters) + multi_member_clusters = [c for c in clusters if len(c) > 1] + n_in_multi = sum(len(c) for c in multi_member_clusters) + + # Cluster-aware AUROC: full set, but CI resamples clusters + all_pos_scores = [r["ensemble"] for r in amp_rows] + all_neg_scores = [r["ensemble"] for r in decoy_rows] + full_auroc = round(_auc_wilcoxon(all_pos_scores, all_neg_scores), 4) + cluster_assignments = [index_to_cluster[i] for i in range(len(amp_rows))] + cluster_aware_ci = _cluster_aware_bootstrap_auroc_ci( + all_pos_scores, all_neg_scores, cluster_assignments, + n_bootstrap=n_bootstrap, + ) + + # Standard CI for comparison (to show the inflation) + standard_ci = _bootstrap_auroc_ci(all_pos_scores, all_neg_scores, n_bootstrap=n_bootstrap) + + # Held-out recovery: non-representative cluster members vs decoys + ref_indices = [c[0] for c in clusters] + test_indices = [i for c in clusters for i in c[1:]] + + held_out_auroc = None + held_out_ci = None + held_out_n_pos = 0 + if test_indices: + held_out_pos = [amp_rows[i]["ensemble"] for i in test_indices] + held_out_auroc = round(_auc_wilcoxon(held_out_pos, all_neg_scores), 4) + held_out_ci = _bootstrap_auroc_ci(held_out_pos, all_neg_scores, n_bootstrap=n_bootstrap) + held_out_n_pos = len(test_indices) + + # Representative-only AUROC: one per cluster vs decoys + rep_pos = [amp_rows[i]["ensemble"] for i in ref_indices] + rep_auroc = round(_auc_wilcoxon(rep_pos, all_neg_scores), 4) + rep_ci = _bootstrap_auroc_ci(rep_pos, all_neg_scores, n_bootstrap=n_bootstrap) + + # Interpretation based on cluster-aware CI lower bound + ci_lo = cluster_aware_ci["ci_lo"] + if full_auroc >= 0.70 and ci_lo >= 0.65: + interpretation = ( + "STRONG — AUROC > 0.70 and cluster-aware CI lower bound > 0.65. " + "Signal survives near-duplicate de-inflation." + ) + elif full_auroc >= 0.70: + interpretation = ( + "STRONG-AUROC-BUT-WIDE-CI — point estimate > 0.70 but cluster-aware CI " + f"lower bound = {ci_lo:.4f} < 0.65. Near-duplicate clusters inflate apparent " + "precision. Signal is real but less certain than standard CI suggests." + ) + elif full_auroc >= 0.55: + interpretation = ( + "WEAK — AUROC 0.55-0.70. Modest signal; proceed with caution." + ) + else: + interpretation = ( + "POOR — AUROC < 0.55. Model is near-random. Do NOT proceed to synthesis." + ) + + return { + "benchmark": "cluster_split_retrospective", + "n_positives": len(amp_rows), + "n_negatives": len(decoy_rows), + "n_total": len(amp_rows) + len(decoy_rows), + "similarity_threshold": similarity_threshold, + "n_clusters": n_clusters, + "n_multi_member_clusters": len(multi_member_clusters), + "n_amps_in_multi_member_clusters": n_in_multi, + "n_independent_amps": n_clusters, + "n_held_out_amps": held_out_n_pos, + # Full-set AUROC (same as standard benchmark) + "full_auroc": full_auroc, + # Standard bootstrap CI (treats all sequences as independent — inflated) + "standard_ci95_lo": standard_ci["ci_lo"], + "standard_ci95_hi": standard_ci["ci_hi"], + "standard_ci_note": ( + "Standard bootstrap treats near-duplicate sequences as independent. " + "When the positive set contains near-duplicate clusters, this CI is " + "artificially narrow." + ), + # Cluster-aware bootstrap CI (resamples clusters, not sequences) + "cluster_aware_ci95_lo": cluster_aware_ci["ci_lo"], + "cluster_aware_ci95_hi": cluster_aware_ci["ci_hi"], + "cluster_aware_ci_mean": cluster_aware_ci["mean"], + "cluster_aware_ci_note": ( + "Cluster-aware bootstrap resamples clusters (not individual sequences). " + "This is the honest CI when the positive set contains near-duplicate families." + ), + # Held-out recovery: near-duplicate AMPs held out, scored vs decoys + "held_out_auroc": held_out_auroc, + "held_out_ci95_lo": held_out_ci["ci_lo"] if held_out_ci else None, + "held_out_ci95_hi": held_out_ci["ci_hi"] if held_out_ci else None, + "held_out_n_positives": held_out_n_pos, + # Representative-only: one AMP per cluster vs decoys + "representative_auroc": rep_auroc, + "representative_ci95_lo": rep_ci["ci_lo"], + "representative_ci95_hi": rep_ci["ci_hi"], + "representative_n_positives": n_clusters, + "interpretation": interpretation, + "near_duplicate_clusters": [ + { + "cluster_id": idx, + "size": len(c), + "members": [amp_rows[i]["id"] for i in c], + "families": [amp_rows[i]["family"] for i in c], + } + for idx, c in enumerate(clusters) if len(c) > 1 + ], + "disclaimer": ( + "Cluster-split benchmark controls for near-duplicate contamination in the " + "positive set. The cluster-aware CI is the honest confidence interval. " + "The standard CI is shown for comparison to reveal the inflation. " + "AUROC > 0.70 does NOT imply nominated candidates are antimicrobial. " + "Wet-lab validation remains mandatory." + ), + } diff --git a/src/openamp_foundry/cli/__main__.py b/src/openamp_foundry/cli/__main__.py new file mode 100644 index 00000000..87b8030c --- /dev/null +++ b/src/openamp_foundry/cli/__main__.py @@ -0,0 +1,4 @@ +"""Allow running the CLI as `python -m openamp_foundry.cli`.""" +from openamp_foundry.cli import main + +raise SystemExit(main()) diff --git a/src/openamp_foundry/cli/commands/benchmark.py b/src/openamp_foundry/cli/commands/benchmark.py index 8afd7be6..84556bdc 100644 --- a/src/openamp_foundry/cli/commands/benchmark.py +++ b/src/openamp_foundry/cli/commands/benchmark.py @@ -81,3 +81,39 @@ def _run_validate_scoring(args: argparse.Namespace) -> int: return 0 + + +def _run_cluster_split_bench(args: argparse.Namespace) -> int: + """Run cluster-split retrospective benchmark with honest CI.""" + import json as _json + from openamp_foundry.benchmark.retrospective import run_cluster_split_benchmark + from openamp_foundry.utils.io import write_json + + result = run_cluster_split_benchmark( + amp_csv=args.amp_csv, + decoy_csv=args.decoy_csv, + config_path=args.config, + similarity_threshold=args.threshold, + n_bootstrap=args.n_bootstrap, + ) + if args.out: + write_json(args.out, result) + summary = { + "status": "ok", + "benchmark": "cluster_split_retrospective", + "n_positives": result["n_positives"], + "n_negatives": result["n_negatives"], + "n_clusters": result["n_clusters"], + "n_multi_member_clusters": result["n_multi_member_clusters"], + "n_amps_in_multi_member_clusters": result["n_amps_in_multi_member_clusters"], + "full_auroc": result["full_auroc"], + "standard_ci95": f"{result['standard_ci95_lo']}-{result['standard_ci95_hi']}", + "cluster_aware_ci95": f"{result['cluster_aware_ci95_lo']}-{result['cluster_aware_ci95_hi']}", + "representative_auroc": result["representative_auroc"], + "representative_ci95": f"{result['representative_ci95_lo']}-{result['representative_ci95_hi']}", + "held_out_auroc": result["held_out_auroc"], + "interpretation": result["interpretation"], + "out": args.out, + } + print(_json.dumps(summary, indent=2)) + return 0 diff --git a/src/openamp_foundry/cli/main.py b/src/openamp_foundry/cli/main.py index f4dc937e..bc4f9d63 100644 --- a/src/openamp_foundry/cli/main.py +++ b/src/openamp_foundry/cli/main.py @@ -1,7 +1,7 @@ from __future__ import annotations from openamp_foundry.cli.commands.core import _run_generate_batch -from openamp_foundry.cli.commands.benchmark import _run_bench, _run_validate_scoring +from openamp_foundry.cli.commands.benchmark import _run_bench, _run_validate_scoring, _run_cluster_split_bench from openamp_foundry.cli.commands.selection import _run_pilot_panel, _run_pilot_confident, _run_diversity_check from openamp_foundry.cli.commands.external import _run_external_predict, _run_external_consensus from openamp_foundry.cli.commands.qc import _run_synthesis_order, _run_presynth_qc @@ -64,6 +64,47 @@ def build_parser() -> argparse.ArgumentParser: baseline.add_argument("--config", default="configs/pipeline.yaml") baseline.add_argument("--out", required=False, help="Optional JSON output path.") + cluster_split = bench_sub.add_parser( + "cluster-split", + help=( + "Cluster-split retrospective AUROC benchmark with honest CI. " + "Clusters near-duplicate AMPs, reports cluster-aware bootstrap CI " + "(wider than standard CI when near-duplicates exist), and held-out " + "recovery AUROC for near-duplicate cluster members." + ), + ) + cluster_split.add_argument( + "--amp-csv", + default="examples/validation/known_amps.csv", + help="CSV of known AMPs (id, sequence, family, reference, label).", + ) + cluster_split.add_argument( + "--decoy-csv", + default="examples/validation/random_background.csv", + help="CSV of decoy peptides.", + ) + cluster_split.add_argument( + "--config", + default="configs/pipeline.yaml", + ) + cluster_split.add_argument( + "--threshold", + type=float, + default=0.70, + help="Similarity threshold for clustering (default: 0.70).", + ) + cluster_split.add_argument( + "--n-bootstrap", + type=int, + default=2000, + help="Bootstrap resample count (default: 2000).", + ) + cluster_split.add_argument( + "--out", + required=False, + help="Optional JSON output path.", + ) + generate = sub.add_parser( "generate-batch", help=( @@ -507,6 +548,8 @@ def main(argv: list[str] | None = None) -> int: return 0 if args.command == "bench": + if args.bench_command == "cluster-split": + return _run_cluster_split_bench(args) return _run_bench(args) if args.command == "generate-batch": diff --git a/tests/test_cluster_split_benchmark.py b/tests/test_cluster_split_benchmark.py new file mode 100644 index 00000000..7597cc5b --- /dev/null +++ b/tests/test_cluster_split_benchmark.py @@ -0,0 +1,321 @@ +"""Cluster-split retrospective benchmark tests — Phase 2 honesty requirement. + +AGENTS.md: "Cluster split — Pipeline still performs when near-duplicates are removed." +AGENTS.md: "Benchmarks must be adversarial — search for ways the pipeline may be +fooling itself." + +These tests verify that the cluster-split benchmark: +1. Correctly identifies near-duplicate clusters in the AMP positive set. +2. Reports a cluster-aware CI that is wider than the standard CI when near-duplicates exist. +3. Reports held-out recovery AUROC for near-duplicate cluster members. +4. Reports representative-only AUROC (one per cluster). +5. Produces honest interpretation that does not overclaim. + +All results are computational. No biological activity is implied. +""" +from __future__ import annotations + +from pathlib import Path + +import pytest + +from openamp_foundry.benchmark.retrospective import ( + _cluster_aware_bootstrap_auroc_ci, + run_cluster_split_benchmark, +) + + +AMP_CSV = "examples/validation/known_amps.csv" +DECOY_CSV = "examples/validation/random_background.csv" + + +class TestClusterAwareBootstrap: + """Unit tests for the cluster-aware bootstrap CI function.""" + + def test_single_cluster_resampling_is_deterministic(self): + """When all positives are in one cluster, cluster-aware CI should be very wide.""" + # 6 positives all in cluster 0, 6 negatives + pos = [0.9, 0.88, 0.91, 0.87, 0.89, 0.92] + neg = [0.3, 0.4, 0.35, 0.25, 0.45, 0.38] + clusters = [0, 0, 0, 0, 0, 0] + result = _cluster_aware_bootstrap_auroc_ci(pos, neg, clusters, n_bootstrap=500, seed=42) + # With 1 cluster, every bootstrap sample draws the same cluster → same scores + # CI should be very wide because we're resampling 1 cluster repeatedly + # Resampling 1 cluster means every bootstrap sample has identical positive scores + # So AUROC is the same every time → CI width = 0 + assert result["ci_hi"] == result["ci_lo"] == 1.0 + + def test_all_singletons_ci_similar_to_standard(self): + """When every positive is its own cluster, cluster-aware ≈ standard bootstrap.""" + pos = [0.9, 0.8, 0.7, 0.6, 0.85, 0.75] + neg = [0.3, 0.4, 0.35, 0.25, 0.45, 0.38] + clusters = [0, 1, 2, 3, 4, 5] # all singletons + result = _cluster_aware_bootstrap_auroc_ci(pos, neg, clusters, n_bootstrap=500, seed=42) + # All singletons → same as standard bootstrap + assert 0.0 <= result["mean"] <= 1.0 + assert result["ci_lo"] <= result["ci_hi"] + + def test_empty_positives_returns_chance(self): + result = _cluster_aware_bootstrap_auroc_ci([], [0.5, 0.6], [], n_bootstrap=100) + assert result["mean"] == 0.5 + assert result["n_bootstrap"] == 0 + + def test_empty_negatives_returns_chance(self): + result = _cluster_aware_bootstrap_auroc_ci([0.9, 0.8], [], [0, 1], n_bootstrap=100) + assert result["mean"] == 0.5 + assert result["n_bootstrap"] == 0 + + def test_perfect_separation_high_auroc(self): + """Clear separation should yield high AUROC mean.""" + pos = [0.9, 0.85, 0.88] + neg = [0.2, 0.15, 0.1] + clusters = [0, 0, 1] + result = _cluster_aware_bootstrap_auroc_ci(pos, neg, clusters, n_bootstrap=500, seed=0) + assert result["mean"] > 0.9 + + +class TestClusterSplitBenchmark: + """End-to-end tests for run_cluster_split_benchmark on the real validation set.""" + + @pytest.fixture(scope="class") + def benchmark_result(self): + """Run the cluster-split benchmark once for all tests in this class.""" + return run_cluster_split_benchmark(AMP_CSV, DECOY_CSV, n_bootstrap=500) + + def test_returns_required_keys(self, benchmark_result): + required = [ + "benchmark", "n_positives", "n_negatives", "n_clusters", + "full_auroc", "standard_ci95_lo", "standard_ci95_hi", + "cluster_aware_ci95_lo", "cluster_aware_ci95_hi", + "representative_auroc", "representative_ci95_lo", "representative_ci95_hi", + "held_out_auroc", "interpretation", "disclaimer", + "near_duplicate_clusters", + ] + for key in required: + assert key in benchmark_result, f"Missing key: {key}" + + def test_benchmark_name(self, benchmark_result): + assert benchmark_result["benchmark"] == "cluster_split_retrospective" + + def test_n_positives_matches_known_amps(self, benchmark_result): + assert benchmark_result["n_positives"] == 95 + + def test_n_negatives_matches_decoys(self, benchmark_result): + assert benchmark_result["n_negatives"] == 96 + + def test_finds_near_duplicate_clusters(self, benchmark_result): + """The benchmark set has 14 multi-member clusters at 0.70 threshold.""" + assert benchmark_result["n_multi_member_clusters"] >= 10, ( + f"Expected >= 10 multi-member clusters, got {benchmark_result['n_multi_member_clusters']}" + ) + assert benchmark_result["n_amps_in_multi_member_clusters"] >= 25, ( + f"Expected >= 25 AMPs in multi-member clusters, got " + f"{benchmark_result['n_amps_in_multi_member_clusters']}" + ) + + def test_n_clusters_less_than_n_positives(self, benchmark_result): + """With near-duplicates, independent clusters < total AMPs.""" + assert benchmark_result["n_clusters"] < benchmark_result["n_positives"], ( + "n_clusters should be < n_positives when near-duplicate clusters exist" + ) + + def test_full_auroc_matches_standard_benchmark(self, benchmark_result): + """Full AUROC should match the standard benchmark (0.7832 on expanded set).""" + assert benchmark_result["full_auroc"] == pytest.approx(0.7832, abs=0.01), ( + f"Full AUROC={benchmark_result['full_auroc']} should match standard benchmark ~0.7832" + ) + + def test_cluster_aware_ci_is_wider_or_equal(self, benchmark_result): + """Cluster-aware CI should be at least as wide as standard CI when near-dups exist.""" + standard_width = ( + benchmark_result["standard_ci95_hi"] - benchmark_result["standard_ci95_lo"] + ) + cluster_width = ( + benchmark_result["cluster_aware_ci95_hi"] - benchmark_result["cluster_aware_ci95_lo"] + ) + # Cluster-aware CI should be wider or at least not narrower + # (near-duplicates reduce effective sample size → wider CI) + assert cluster_width >= standard_width - 0.02, ( + f"Cluster-aware CI width ({cluster_width:.4f}) should be >= " + f"standard CI width ({standard_width:.4f}) when near-duplicates exist. " + "If not, near-duplicates may not be inflating the standard CI." + ) + + def test_cluster_aware_ci_lo_lower_than_standard(self, benchmark_result): + """The key honesty test: cluster-aware CI lower bound should be <= standard CI lo.""" + assert benchmark_result["cluster_aware_ci95_lo"] <= benchmark_result["standard_ci95_lo"] + 0.01, ( + f"Cluster-aware CI lo ({benchmark_result['cluster_aware_ci95_lo']}) should be <= " + f"standard CI lo ({benchmark_result['standard_ci95_lo']}) + 0.01. " + "If the cluster-aware CI lo is higher, the standard CI was not inflated by " + "near-duplicates — which would be a surprising but honest result." + ) + + def test_representative_auroc_below_full_auroc(self, benchmark_result): + """Removing near-duplicates should not increase AUROC.""" + assert benchmark_result["representative_auroc"] <= benchmark_result["full_auroc"] + 0.02, ( + f"Representative AUROC ({benchmark_result['representative_auroc']}) should be <= " + f"full AUROC ({benchmark_result['full_auroc']}) + 0.02. " + "Removing near-duplicates should not improve performance." + ) + + def test_representative_auroc_above_random(self, benchmark_result): + """Even with one-per-cluster, AUROC should beat random (0.5).""" + assert benchmark_result["representative_auroc"] > 0.60, ( + f"Representative AUROC ({benchmark_result['representative_auroc']}) > 0.60: " + "pipeline must have signal even after removing near-duplicate redundancy." + ) + + def test_held_out_auroc_above_random(self, benchmark_result): + """Held-out near-duplicate AMPs should still outrank decoys.""" + if benchmark_result["held_out_auroc"] is not None: + assert benchmark_result["held_out_auroc"] > 0.60, ( + f"Held-out AUROC ({benchmark_result['held_out_auroc']}) > 0.60: " + "near-duplicate AMPs should still score above decoys." + ) + + def test_held_out_n_positives_correct(self, benchmark_result): + """19 AMPs should be held out (non-representative cluster members).""" + assert benchmark_result["held_out_n_positives"] >= 10, ( + f"Expected >= 10 held-out AMPs, got {benchmark_result['held_out_n_positives']}" + ) + + def test_interpretation_is_string(self, benchmark_result): + assert isinstance(benchmark_result["interpretation"], str) + assert len(benchmark_result["interpretation"]) > 20 + + def test_disclaimer_present(self, benchmark_result): + assert "cluster-aware" in benchmark_result["disclaimer"].lower() + assert "wet-lab" in benchmark_result["disclaimer"].lower() + + def test_near_duplicate_clusters_have_members(self, benchmark_result): + """Each near-duplicate cluster entry should list its members.""" + for cluster in benchmark_result["near_duplicate_clusters"]: + assert "members" in cluster + assert "size" in cluster + assert len(cluster["members"]) == cluster["size"] + assert cluster["size"] >= 2 + + def test_magainin_cluster_detected(self, benchmark_result): + """The magainin family (MAG-001/002/003) should appear as a near-dup cluster.""" + all_members = [] + for cluster in benchmark_result["near_duplicate_clusters"]: + all_members.extend(cluster["members"]) + assert "REF-MAG-001" in all_members, "Magainin cluster should be detected" + assert "REF-MAG-002" in all_members, "Magainin-2 should be in same cluster" + + def test_protegrin_cluster_detected(self, benchmark_result): + """The protegrin family (PRG1/2/3) should appear as a near-dup cluster.""" + all_members = [] + for cluster in benchmark_result["near_duplicate_clusters"]: + all_members.extend(cluster["members"]) + assert "REF-PRG1-001" in all_members, "Protegrin cluster should be detected" + + def test_phase3_config_also_works(self): + """Cluster-split benchmark should also work with phase3.yaml weights.""" + result = run_cluster_split_benchmark( + AMP_CSV, DECOY_CSV, config_path="configs/phase3.yaml", n_bootstrap=300, + ) + assert result["full_auroc"] > 0.65, ( + f"Phase3 full AUROC ({result['full_auroc']}) should be > 0.65" + ) + assert result["representative_auroc"] > 0.60, ( + f"Phase3 representative AUROC ({result['representative_auroc']}) should be > 0.60" + ) + + def test_custom_threshold_changes_clustering(self): + """A stricter threshold should produce fewer clusters (more singletons).""" + result_loose = run_cluster_split_benchmark(AMP_CSV, DECOY_CSV, similarity_threshold=0.50, n_bootstrap=200) + result_strict = run_cluster_split_benchmark(AMP_CSV, DECOY_CSV, similarity_threshold=0.90, n_bootstrap=200) + # Lower threshold → more sequences co-clustered → fewer clusters + assert result_loose["n_clusters"] <= result_strict["n_clusters"], ( + f"Loose threshold (0.50) clusters={result_loose['n_clusters']} should be <= " + f"strict threshold (0.90) clusters={result_strict['n_clusters']}" + ) + + def test_ala_and_drs_are_exact_duplicate(self, benchmark_result): + """REF-ALA-001 and REF-DRS1-001 are identical sequences — must cluster together.""" + found = False + for cluster in benchmark_result["near_duplicate_clusters"]: + if "REF-ALA-001" in cluster["members"] and "REF-DRS1-001" in cluster["members"]: + found = True + break + assert found, ( + "REF-ALA-001 and REF-DRS1-001 are identical sequences and must be in the same cluster" + ) + + +class TestClusterSplitBenchmarkSmallData: + """Tests with small synthetic data to verify edge cases.""" + + def test_no_near_duplicates(self, tmp_path): + """When all AMPs are unique, cluster-split should report n_clusters = n_positives.""" + amp_csv = tmp_path / "amps.csv" + amp_csv.write_text( + "id,sequence,family,reference,label\n" + "AMP-001,KWKLFKKIGAVLKVL,test,t,1\n" + "AMP-002,GLFDIVKKVVGALGSL,test,t,1\n" + "AMP-003,RRWQWRMKKLG,test,t,1\n" + ) + decoy_csv = tmp_path / "decoys.csv" + decoy_csv.write_text( + "id,sequence,family,source_id,label\n" + "DEC-001,GGGGGGGGGGGGGGG,bg,bg,0\n" + "DEC-002,AAAAAAAAAAA,bg,bg,0\n" + "DEC-003,DEDEDEDEDEDEDE,bg,bg,0\n" + ) + result = run_cluster_split_benchmark(amp_csv, decoy_csv, n_bootstrap=100) + assert result["n_multi_member_clusters"] == 0 + assert result["n_clusters"] == 3 + assert result["n_held_out_amps"] == 0 + assert result["held_out_auroc"] is None + + def test_all_identical_amps(self, tmp_path): + """When all AMPs are identical, one cluster, all held out except one.""" + amp_csv = tmp_path / "amps.csv" + amp_csv.write_text( + "id,sequence,family,reference,label\n" + "AMP-001,KWKLFKKIGAVLKVL,test,t,1\n" + "AMP-002,KWKLFKKIGAVLKVL,test,t,1\n" + "AMP-003,KWKLFKKIGAVLKVL,test,t,1\n" + ) + decoy_csv = tmp_path / "decoys.csv" + decoy_csv.write_text( + "id,sequence,family,source_id,label\n" + "DEC-001,GGGGGGGGGGGGGGG,bg,bg,0\n" + "DEC-002,AAAAAAAAAAA,bg,bg,0\n" + ) + result = run_cluster_split_benchmark(amp_csv, decoy_csv, n_bootstrap=100) + assert result["n_clusters"] == 1 + assert result["n_multi_member_clusters"] == 1 + assert result["n_held_out_amps"] == 2 + + +class TestClusterSplitCLI: + """Test the CLI command bench cluster-split.""" + + def test_cli_cluster_split_runs(self, tmp_path): + from openamp_foundry.cli import main + out = str(tmp_path / "cluster_split.json") + ret = main(["bench", "cluster-split", "--out", out, "--n-bootstrap", "200"]) + assert ret == 0 + import json + data = json.loads(Path(out).read_text()) + assert data["benchmark"] == "cluster_split_retrospective" + assert "full_auroc" in data + assert "cluster_aware_ci95_lo" in data + assert "near_duplicate_clusters" in data + + def test_cli_cluster_split_custom_threshold(self, tmp_path): + from openamp_foundry.cli import main + out = str(tmp_path / "cluster_split.json") + ret = main(["bench", "cluster-split", "--threshold", "0.80", "--out", out, "--n-bootstrap", "100"]) + assert ret == 0 + import json + data = json.loads(Path(out).read_text()) + assert data["similarity_threshold"] == 0.80 + + def test_cli_cluster_split_no_out_file(self, tmp_path): + """CLI should work without --out (prints to stdout only).""" + from openamp_foundry.cli import main + ret = main(["bench", "cluster-split", "--n-bootstrap", "100"]) + assert ret == 0