Repository navigation
feat: selectivity benchmark — safety_score AUROC=0.54 (FAILED) reveals critical gap - #117
Closed
cschanhniem wants to merge 1 commit into
Closed
cschanhniem wants to merge 1 commit into
cschanhniem wants to merge 1 commit into
Conversation
…rophobic_fraction AUROC=0.81 Add the first selectivity benchmark to the OpenAMP Foundry pipeline. This directly addresses the v1.1 roadmap requirement: 'benchmark candidate triage against a reference panel that includes selective AMPs, hemolytic positives, inactive peptides, and random controls' and Phase 2.5 DoD: 'Membrane proxy benchmark — Distinguishes bacterial-selective vs clearly hemolytic reference peptides better than naive heuristics.' What was built: - examples/validation/selectivity_panel.csv: 10 selective + 8 hemolytic AMPs with literature hemolysis classifications (Zasloff, Habermann, Skerlavaj, etc.) - src/openamp_foundry/benchmark/selectivity.py: SelectivityResult dataclass + run_selectivity_benchmark() computing AUROC for safety_score, selectivity_proxy, hydrophobic_fraction, hydrophobic_moment, charge_density, and GRAVY - tests/test_selectivity_benchmark.py: 21 tests covering panel integrity, AUROC computation, blind spot detection, CLI integration - CLI: 'validate-selectivity' command + 'make validate-selectivity' target - Docs updated: METRICS_CURRENT, BENCHMARK_CARD, ROADMAP, ARCHITECTURE Key finding (honest, not flattering): safety_score AUROC = 0.54 (near random) — the safety filter does NOT discriminate selective from hemolytic AMPs. 5/8 hemolytic AMPs score safety=1.0, including melittin (HC50 ~ 5 ug/mL). The naive hydrophobic_fraction baseline (AUROC=0.81) significantly outperforms the designed safety_score. This is a documented failure, not a hidden one. The benchmark reveals the gap that the melittin blind spot hint in BENCHMARK_CARD.md previously only described qualitatively. The next loop should target safety_score improvement: lowering the hydrophobic_fraction threshold from 0.65 to ~0.50, and adding multi-residue hydrophobic face coverage analysis. 1417 tests pass, 1 skipped (pre-existing). Ruff clean on all new/modified files.
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the first selectivity benchmark to the OpenAMP Foundry pipeline, directly addressing:
What was built
Key finding (honest, not flattering)
safety_score does NOT discriminate selective from hemolytic AMPs. 5 of 8 hemolytic AMPs receive safety_score=1.0, including melittin (HC50 ~ 5 ug/mL). The naive hydrophobic_fraction baseline (AUROC=0.81) significantly outperforms the designed safety_score.
Why this matters
This is a documented failure, not a hidden one. The melittin blind spot was previously noted qualitatively but never measured. Now it is quantified. The next improvement loop should target safety_score redesign.
Verification