diff --git a/Makefile b/Makefile index ddce579d..8ccb3953 100644 --- a/Makefile +++ b/Makefile @@ -260,6 +260,10 @@ wave0-5-panel: wave0-5-fill-external wave0-5-evidence: PYTHONPATH=src $(PYTHON) scripts/generate_wave0_5_evidence_certs.py +metrics-snapshot: + PYTHONPATH=src $(PYTHON) -m openamp_foundry.cli bench metrics-snapshot \ + --out outputs/metrics_snapshot.json + wave0-5b-generate: PYTHONPATH=src $(PYTHON) scripts/generate_wave0_5b_candidates.py diff --git a/docs/BENCHMARK_CARD.md b/docs/BENCHMARK_CARD.md index a7d2c36d..9fe38094 100644 --- a/docs/BENCHMARK_CARD.md +++ b/docs/BENCHMARK_CARD.md @@ -109,9 +109,9 @@ against the simple ensemble to test whether its additional complexity earns its | Scorer | AUROC | CI₉₅ | Delta vs ensemble | |--------|:-----:|:----:|:-----------------:| | **Ensemble** (pipeline.yaml) | **0.7832** | 0.717–0.8423 | — | -| Expert composite | 0.7360 | 0.6604–0.8037 | **−0.0472** | +| Expert composite | 0.7119 | 0.6295–0.7865 | **−0.0713** | | **Ensemble** (phase3.yaml) | **0.7448** | 0.6741–0.8118 | — | -| Expert composite | 0.7360 | 0.6604–0.8037 | −0.0088 | +| Expert composite | 0.7119 | 0.6295–0.7865 | −0.0329 | **Per-component AUROC:** diff --git a/docs/METRICS_CURRENT.md b/docs/METRICS_CURRENT.md index a8d25a49..4f2a8b23 100644 --- a/docs/METRICS_CURRENT.md +++ b/docs/METRICS_CURRENT.md @@ -1,5 +1,7 @@ # Current Pipeline Metrics — Single Source of Truth +Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `make metrics-snapshot`. + > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees > with this file, this file wins. Updated whenever benchmark/benchmark config changes. > @@ -178,7 +180,7 @@ Machine-readable: `outputs/wave0_5_external_predict_results.csv`, `outputs/wave0 |--------|:-----------------------:|:-------------------:| | Ensemble AUROC | 0.7832 | 0.7448 | | Ensemble CI₉₅ | 0.717–0.8423 | 0.6741–0.8118 | -| Expert composite AUROC | 0.7360 | 0.7360 | +| Expert composite AUROC | 0.7119 | 0.7119 | | Expert CI₉₅ | 0.6604–0.8037 | 0.6604–0.8037 | | **Delta (expert − ensemble)** | **−0.0472** | **−0.0088** | | Verdict | Expert LOWER | Within ±0.02 | @@ -471,7 +473,7 @@ The MODERATE class (HC50 25-100, n=68) is excluded from the binary task. | 2026-06-29 | Wave 0.5b: 23-candidate safety-optimized shortlist (SEED-020–024, no aromatics) | OpenAMP Wave 0.5b | | 2026-06-29 | External predictor results filled from wave05_combined_consensus.csv; all 7 gates PASS | OpenAMP Wave 0.5 | | 2026-06-29 | Wave 0.5 scaffold diversification — 24-candidate Wave 1 panel across 14 families | OpenAMP Wave 0.5 | -| 2026-07-01 | Expert ablation benchmark added: expert composite AUROC 0.736 vs ensemble 0.7832 (delta −0.0472); 3 components anti-signal; ensemble remains primary gate | OpenAMP loop | +| 2026-07-01 | Expert ablation benchmark added: expert composite AUROC 0.7119 vs ensemble 0.7832 (delta −0.0713); anti-signal components documented; ensemble remains primary gate | OpenAMP loop | | 2026-07-01 | **Hemolysis benchmark expanded:** 42 -> 238 peptides using DBAASP human erythrocyte data (54 hemolytic vs 125 selective, n=179 binary). Hemolysis risk scorer detection AUROC drops 0.9218 -> 0.5650 (CI 0.47-0.66) — original performance was small-sample inflation. Direction correct, not significant. Safety scorer detection improves 0.3844 -> 0.5116 (still not significant). 196 new peptides from DBAASP v3. | OpenAMP loop | | 2026-07-01 | Dedicated hemolysis risk scorer: 4-component score (synth+aromatic+face+cys) achieves detection AUROC=0.9218 (CI: 0.82-0.99); integrated into expert composite (detection 0.5119→0.6429); safety scorer unchanged; 1471 tests | OpenAMP loop | | 2026-07-01 | Within-AMP selectivity benchmark added: safety scorer FAILS hemolysis detection (AUROC=0.3844); synthesis is only significant risk detector (AUROC=0.8027); expert composite better than ensemble but not significant (0.5119 vs 0.3486) | OpenAMP loop | diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 13c696ca..34407c09 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -141,7 +141,7 @@ Implemented during the pre-wet-lab improvement loop (PRs #31–#54): - `run_expert_ablation_benchmark()` in `benchmark/retrospective.py` — scores all 95 AMPs + 96 decoys with both the simple ensemble and the 7-component expert composite - `bench expert-ablation` CLI command + `make bench-expert-ablation` target - Per-component AUROC attribution: identifies signal-bearing (activity 0.814, selectivity 0.773), near-zero (hinge, novelty, boman, motif), and anti-signal (safety 0.349, serum_stability 0.223, synthesis 0.423) components -- **Key finding:** Expert composite AUROC 0.736 < ensemble AUROC 0.783 (delta −0.047). The added complexity does NOT improve AMP-vs-decoy discrimination. Three components are anti-signal because real AMPs have extreme biophysical properties that safety/stability/synthesis scorers penalise. +- **Key finding:** Expert composite AUROC 0.7119 < ensemble AUROC 0.7832 (delta −0.0713). The added complexity does NOT improve AMP-vs-decoy discrimination. Several added components are anti-signal because real AMPs have extreme biophysical properties that safety, stability, and synthesis scorers penalise. - Ensemble remains primary synthesis gate. Expert composite may add value for within-AMP ranking (tested in v0.5.9 — see selectivity benchmark; expert composite detection AUROC 0.5119 vs ensemble 0.3486, but not significant at n=14 vs n=21). - `make bench-cluster-split` Makefile target added (was missing despite CLI command existing) - 14 new tests in `test_expert_ablation.py`; 1435 total tests diff --git a/src/openamp_foundry/benchmark/metrics_snapshot.py b/src/openamp_foundry/benchmark/metrics_snapshot.py new file mode 100644 index 00000000..bbd431dc --- /dev/null +++ b/src/openamp_foundry/benchmark/metrics_snapshot.py @@ -0,0 +1,153 @@ +"""Build a machine-readable snapshot of the repo's current benchmark truth.""" + +from __future__ import annotations + +from pathlib import Path + +from openamp_foundry.benchmark.retrospective import ( + run_cluster_split_benchmark, + run_expert_ablation_benchmark, + run_retrospective_benchmark, + run_selectivity_benchmark, +) +from openamp_foundry.benchmark.triage import run_triage_benchmark + + +def build_metrics_snapshot( + *, + amp_csv: str | Path = "examples/validation/known_amps.csv", + decoy_csv: str | Path = "examples/validation/random_background.csv", + hemolysis_csv: str | Path = "examples/validation/hemolysis_reference.csv", + standard_config: str | Path = "configs/pipeline.yaml", + phase3_config: str | Path = "configs/phase3.yaml", + cluster_threshold: float = 0.70, + n_bootstrap: int = 2000, +) -> dict[str, object]: + """Return a compact, auditable snapshot of current benchmark metrics.""" + + standard = run_retrospective_benchmark( + amp_csv=amp_csv, + decoy_csv=decoy_csv, + config_path=standard_config, + benchmark_type="standard", + ) + phase3 = run_retrospective_benchmark( + amp_csv=amp_csv, + decoy_csv=decoy_csv, + config_path=phase3_config, + benchmark_type="standard", + ) + cluster_split = run_cluster_split_benchmark( + amp_csv=amp_csv, + decoy_csv=decoy_csv, + config_path=standard_config, + similarity_threshold=cluster_threshold, + n_bootstrap=n_bootstrap, + ) + expert_ablation = run_expert_ablation_benchmark( + amp_csv=amp_csv, + decoy_csv=decoy_csv, + config_path=standard_config, + n_bootstrap=n_bootstrap, + ) + selectivity = run_selectivity_benchmark( + hemolysis_csv=hemolysis_csv, + config_path=standard_config, + n_bootstrap=n_bootstrap, + ) + triage = run_triage_benchmark( + hemolysis_csv=hemolysis_csv, + decoy_csv=decoy_csv, + config_path=standard_config, + n_bootstrap=n_bootstrap, + ) + + return { + "schema_version": 1, + "benchmark_inputs": { + "amp_csv": str(amp_csv), + "decoy_csv": str(decoy_csv), + "hemolysis_csv": str(hemolysis_csv), + "standard_config": str(standard_config), + "phase3_config": str(phase3_config), + "cluster_threshold": cluster_threshold, + "n_bootstrap": n_bootstrap, + }, + "standard": { + "auroc": standard["auroc"], + "auprc": standard.get("auprc"), + "recall_at_10": standard.get("recall_at_10"), + "recall_at_20": standard.get("recall_at_20"), + "recall_at_43": standard.get("recall_at_43"), + "n_positives": standard["n_positives"], + "n_negatives": standard["n_negatives"], + "n_total": standard["n_positives"] + standard["n_negatives"], + "interpretation": standard["interpretation"], + }, + "phase3": { + "auroc": phase3["auroc"], + "auprc": phase3.get("auprc"), + "n_positives": phase3["n_positives"], + "n_negatives": phase3["n_negatives"], + "n_total": phase3["n_positives"] + phase3["n_negatives"], + "interpretation": phase3["interpretation"], + }, + "cluster_split": { + "full_auroc": cluster_split["full_auroc"], + "standard_ci95": [ + cluster_split["standard_ci95_lo"], + cluster_split["standard_ci95_hi"], + ], + "cluster_aware_ci95": [ + cluster_split["cluster_aware_ci95_lo"], + cluster_split["cluster_aware_ci95_hi"], + ], + "representative_auroc": cluster_split["representative_auroc"], + "representative_ci95": [ + cluster_split["representative_ci95_lo"], + cluster_split["representative_ci95_hi"], + ], + "held_out_auroc": cluster_split["held_out_auroc"], + "n_clusters": cluster_split["n_clusters"], + "n_multi_member_clusters": cluster_split["n_multi_member_clusters"], + "n_amps_in_multi_member_clusters": cluster_split[ + "n_amps_in_multi_member_clusters" + ], + "interpretation": cluster_split["interpretation"], + }, + "expert_ablation": { + "ensemble_auroc": expert_ablation["ensemble_auroc"], + "expert_auroc": expert_ablation["expert_auroc"], + "delta_auroc": expert_ablation["delta_auroc"], + "signal_bearing_components": expert_ablation[ + "signal_bearing_components" + ], + "near_zero_components": expert_ablation["near_zero_components"], + "anti_signal_components": expert_ablation["anti_signal_components"], + "verdict": expert_ablation["verdict"], + }, + "selectivity": { + "n_hemolytic": selectivity["n_hemolytic"], + "n_selective": selectivity["n_selective"], + "n_border": selectivity["n_border"], + "safety_verdict": selectivity["safety_verdict"], + "selectivity_proxy_verdict": selectivity["selectivity_proxy_verdict"], + "expert_composite_verdict": selectivity["expert_composite_verdict"], + "risk_detectors": selectivity["risk_detectors"], + "risk_indicators": selectivity["risk_indicators"], + }, + "triage": { + "best_scorer": triage["best_scorer"], + "top_20_by_ensemble": triage["top_20_by_ensemble"], + "top_20_by_triage_score": triage["top_20_by_triage_score"], + "per_scorer": { + scorer: { + "selective_vs_decoy": info["selective_vs_decoy"]["auroc"], + "hemolytic_vs_decoy": info["hemolytic_vs_decoy"]["auroc"], + "selective_vs_hemolytic": info["selective_vs_hemolytic"]["auroc"], + "triages_correctly": info["triages_correctly"], + } + for scorer, info in triage["per_scorer"].items() + }, + }, + } diff --git a/src/openamp_foundry/cli/commands/benchmark.py b/src/openamp_foundry/cli/commands/benchmark.py index e7445a3d..e4021465 100644 --- a/src/openamp_foundry/cli/commands/benchmark.py +++ b/src/openamp_foundry/cli/commands/benchmark.py @@ -219,3 +219,24 @@ def _run_triage(args: argparse.Namespace) -> int: } print(json.dumps(summary, indent=2)) return 0 + + +def _run_metrics_snapshot(args: argparse.Namespace) -> int: + import json + + from openamp_foundry.benchmark.metrics_snapshot import build_metrics_snapshot + from openamp_foundry.utils.io import write_json + + result = build_metrics_snapshot( + amp_csv=args.amp_csv, + decoy_csv=args.decoy_csv, + hemolysis_csv=args.hemolysis_csv, + standard_config=args.config, + phase3_config=args.phase3_config, + cluster_threshold=args.threshold, + n_bootstrap=args.n_bootstrap, + ) + if args.out: + write_json(args.out, result) + print(json.dumps(result, indent=2)) + return 0 diff --git a/src/openamp_foundry/cli/main.py b/src/openamp_foundry/cli/main.py index 9a6056db..c666c7c2 100644 --- a/src/openamp_foundry/cli/main.py +++ b/src/openamp_foundry/cli/main.py @@ -1,7 +1,7 @@ from __future__ import annotations from openamp_foundry.cli.commands.core import _run_generate_batch -from openamp_foundry.cli.commands.benchmark import _run_bench, _run_validate_scoring, _run_cluster_split_bench, _run_expert_ablation_bench, _run_selectivity_bench, _run_triage +from openamp_foundry.cli.commands.benchmark import _run_bench, _run_validate_scoring, _run_cluster_split_bench, _run_expert_ablation_bench, _run_selectivity_bench, _run_triage, _run_metrics_snapshot from openamp_foundry.cli.commands.selection import _run_pilot_panel, _run_pilot_confident, _run_diversity_check from openamp_foundry.cli.commands.external import _run_external_predict, _run_external_consensus from openamp_foundry.cli.commands.qc import _run_synthesis_order, _run_presynth_qc @@ -207,6 +207,35 @@ def build_parser() -> argparse.ArgumentParser: help="Optional JSON output path.", ) + metrics_snapshot = bench_sub.add_parser( + "metrics-snapshot", + help=( + "Build a machine-readable snapshot of the repo's current benchmark truth. " + "Use this to refresh docs/METRICS_CURRENT.md evidence and catch number drift." + ), + ) + metrics_snapshot.add_argument( + "--amp-csv", + default="examples/validation/known_amps.csv", + ) + metrics_snapshot.add_argument( + "--decoy-csv", + default="examples/validation/random_background.csv", + ) + metrics_snapshot.add_argument( + "--hemolysis-csv", + default="examples/validation/hemolysis_reference.csv", + ) + metrics_snapshot.add_argument("--config", default="configs/pipeline.yaml") + metrics_snapshot.add_argument("--phase3-config", default="configs/phase3.yaml") + metrics_snapshot.add_argument("--threshold", type=float, default=0.70) + metrics_snapshot.add_argument("--n-bootstrap", type=int, default=2000) + metrics_snapshot.add_argument( + "--out", + required=False, + help="Optional JSON output path.", + ) + generate = sub.add_parser( "generate-batch", help=( @@ -659,6 +688,8 @@ def main(argv: list[str] | None = None) -> int: return _run_selectivity_bench(args) if args.bench_command == "triage": return _run_triage(args) + if args.bench_command == "metrics-snapshot": + return _run_metrics_snapshot(args) return _run_bench(args) if args.command == "generate-batch": @@ -713,4 +744,3 @@ def main(argv: list[str] | None = None) -> int: if __name__ == "__main__": raise SystemExit(main()) - diff --git a/tests/test_docs_consistent.py b/tests/test_docs_consistent.py index ae8a8fc9..11057c14 100644 --- a/tests/test_docs_consistent.py +++ b/tests/test_docs_consistent.py @@ -3,6 +3,7 @@ from __future__ import annotations import re +import json from pathlib import Path @@ -17,6 +18,7 @@ CURRENT_N_TOTAL = 191 EXPANDED_PR_REF = "PR #110" ORIGINAL_DEMO_AUROC = "0.8420" +SNAPSHOT_PATH = REPO_ROOT / "outputs" / "metrics_snapshot.json" # Patterns that indicate stale "current" usage (not historical context) _STALE_CURRENT_PATTERNS: list[str] = [ @@ -91,6 +93,16 @@ def test_doc_metrics_current_md_is_authoritative(self): f"METRICS_CURRENT.md missing n={CURRENT_N_TOTAL}" ) + def test_doc_metrics_snapshot_matches_authoritative_values(self): + """Machine-readable metrics snapshot must agree with docs' current truth.""" + assert SNAPSHOT_PATH.exists(), "outputs/metrics_snapshot.json not found" + payload = json.loads(SNAPSHOT_PATH.read_text(encoding="utf-8")) + assert payload["standard"]["auroc"] == float(CURRENT_AUROC) + assert payload["phase3"]["auroc"] == float(CURRENT_PHASE3_AUROC) + assert payload["standard"]["n_positives"] == CURRENT_N_POSITIVES + assert payload["standard"]["n_negatives"] == CURRENT_N_NEGATIVES + assert payload["standard"]["n_total"] == CURRENT_N_TOTAL + def test_doc_no_breaking_news_terminology(self): """'breaking news' must be replaced with 'high-impact scenario' in all docs.""" breaking = re.compile(r"breaking[-\s]news", re.IGNORECASE) diff --git a/tests/test_metrics_snapshot.py b/tests/test_metrics_snapshot.py new file mode 100644 index 00000000..cf3a5c80 --- /dev/null +++ b/tests/test_metrics_snapshot.py @@ -0,0 +1,25 @@ +from __future__ import annotations + +import json + +from openamp_foundry.benchmark.metrics_snapshot import build_metrics_snapshot + + +def test_metrics_snapshot_matches_current_benchmark_truth(tmp_path): + snapshot = build_metrics_snapshot(n_bootstrap=200) + + assert snapshot["schema_version"] == 1 + assert snapshot["standard"]["n_total"] == 191 + assert snapshot["standard"]["n_positives"] == 95 + assert snapshot["standard"]["n_negatives"] == 96 + assert snapshot["standard"]["auroc"] == 0.7832 + assert snapshot["phase3"]["auroc"] == 0.7448 + assert snapshot["cluster_split"]["full_auroc"] == 0.7832 + assert snapshot["expert_ablation"]["expert_auroc"] == 0.7119 + assert snapshot["selectivity"]["n_hemolytic"] == 54 + assert snapshot["triage"]["best_scorer"] == "selectivity_proxy" + + out = tmp_path / "metrics_snapshot.json" + out.write_text(json.dumps(snapshot, indent=2, sort_keys=True), encoding="utf-8") + reloaded = json.loads(out.read_text(encoding="utf-8")) + assert reloaded["standard"]["auroc"] == 0.7832