Repository navigation
feat(scoring): add rich selectivity scorer from evidence-identified features - #130
Merged
Merged
Conversation
…eatures The feature decomposition benchmark (v0.5.15) identified 8 statistically significant features for selective_vs_hemolytic discrimination, but the old selectivity_proxy (charge + GRAVY) used only 2. This adds a composite selectivity scorer combining all 8 significant features, weighted by detection AUROC. Key result: rich selectivity detection AUROC=0.7138 (CI 0.6266-0.7951) on n=179 (54 hemolytic vs 125 selective) — first pipeline score with CI excluding 0.5. Old selectivity_proxy=0.5744 (CI 0.4954-0.6558). Honest limitation: does NOT triage AMP-vs-decoy (selective_vs_decoy=0.19); designed for within-AMP selectivity only. Must be combined with an activity gate for candidate selection. Changes: - scoring/selectivity_rich.py: composite scorer with full breakdown - benchmark/retrospective.py: integrated into selectivity benchmark - benchmark/triage.py: integrated into standard and strict triage - benchmark/metrics_snapshot.py: expanded selectivity snapshot - cli/commands/benchmark.py: selectivity verdict in CLI summary - tests/test_selectivity_rich.py: 18 tests - docs updated (METRICS_CURRENT, ROADMAP, ARCHITECTURE)
cschanhniem
added a commit
that referenced
this pull request
Jul 3, 2026
… composite, and pilot selection (#131) The rich selectivity scorer (v0.5.16, PR #130) was benchmarked and proven significant (detection AUROC=0.7138, CI 0.63-0.80) but was not connected to any production path where candidate selection actually happens. This PR wires it into pipeline scoring, expert composite, pilot priority, pilot panel report, and evidence certificates. Changes: - pipeline.py: rich_selectivity computed for every candidate in raw_scores - scoring/expert.py: rich_selectivity replaces hemolysis_safety (not significant) as expert hemolysis-risk component (weight 0.10); old proxy stays as activity-ranking selectivity component (weight 0.20) - selection/pilot.py: pilot_priority uses rich_selectivity with fallback - reports/pilot_panel.py: rich_selectivity column added to CSV and markdown - benchmark/retrospective.py: expert ablation uses rich_selectivity component - tests/test_rich_selectivity_integration.py: 14 integration tests Expert AUROC drops 0.7119 -> 0.7097 (-0.0022) on AMP-vs-decoy. Acceptable tradeoff: expert now includes a significant hemolysis detector instead of a non-significant one. Constraint: wet-lab hemolysis assay remains mandatory; computational selectivity scores are triage signals only, not biological proof. Confidence: high - rich selectivity detection AUROC=0.7138 (CI 0.63-0.80, significant); integration verified by 1571 passing tests. Tested: uv run pytest -q (1571 passed, 7 skipped), ruff check (clean), git diff --check (clean), make metrics-snapshot. Co-authored-by: OmX <omx@oh-my-codex.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The feature decomposition benchmark (v0.5.15, PR #129) identified 8 statistically significant features for selective_vs_hemolytic discrimination, but the old
selectivity_proxy(charge + GRAVY) used only 2. This PR builds a composite selectivity scorer from the evidence — combining all 8 significant features, weighted by detection AUROC.Key Result
The rich selectivity scorer is the first pipeline score with statistically significant hemolysis detection on the expanded n=179 benchmark (CI lower bound 0.6266 > 0.5).
Honest Limitations
Changes
scoring/selectivity_rich.py: composite scorer with full component breakdownbenchmark/retrospective.py: integrated into selectivity benchmark withrich_selectivity_verdictbenchmark/triage.py: integrated into standard and strict triage benchmarksbenchmark/metrics_snapshot.py: selectivity snapshot expanded withper_score_aurocand verdictcli/commands/benchmark.py: selectivity bench summary includes rich selectivity verdicttests/test_selectivity_rich.py: 18 tests covering scoring, breakdown, and benchmark integrationoutputs/metrics_snapshot.json: regeneratedVerification
uv run pytest -q: 1557 passed, 7 skipped (up from 1539)uv run ruff check src tests: cleangit diff --check: cleanmake metrics-snapshot: regenerated with rich selectivity results