Repository navigation
feat(scoring): wire rich selectivity into production pipeline, expert composite, and pilot selection - #131
Merged
Conversation
… composite, and pilot selection The rich selectivity scorer (v0.5.16, PR #130) was benchmarked and proven significant (detection AUROC=0.7138, CI 0.63-0.80) but was not connected to any production path where candidate selection actually happens. This PR wires it into pipeline scoring, expert composite, pilot priority, pilot panel report, and evidence certificates. Changes: - pipeline.py: rich_selectivity computed for every candidate in raw_scores - scoring/expert.py: rich_selectivity replaces hemolysis_safety (not significant) as expert hemolysis-risk component (weight 0.10); old proxy stays as activity-ranking selectivity component (weight 0.20) - selection/pilot.py: pilot_priority uses rich_selectivity with fallback - reports/pilot_panel.py: rich_selectivity column added to CSV and markdown - benchmark/retrospective.py: expert ablation uses rich_selectivity component - tests/test_rich_selectivity_integration.py: 14 integration tests Expert AUROC drops 0.7119 -> 0.7097 (-0.0022) on AMP-vs-decoy. Acceptable tradeoff: expert now includes a significant hemolysis detector instead of a non-significant one. Constraint: wet-lab hemolysis assay remains mandatory; computational selectivity scores are triage signals only, not biological proof. Confidence: high - rich selectivity detection AUROC=0.7138 (CI 0.63-0.80, significant); integration verified by 1571 passing tests. Tested: uv run pytest -q (1571 passed, 7 skipped), ruff check (clean), git diff --check (clean), make metrics-snapshot. Co-authored-by: OmX <omx@oh-my-codex.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The rich selectivity scorer (v0.5.16, PR #130) was benchmarked and proven significant (detection AUROC=0.7138, CI 0.63-0.80) but was not connected to any production path where candidate selection actually happens. Every candidate was still scored with the old
selectivity_proxy(detection AUROC=0.5744, CI 0.50-0.66, NOT significant) for hemolysis risk in:score_candidates()inpipeline.pyhemolysis_safetycomponent, AUROC=0.565, not significant)pilot_priorityformula inselection/pilot.pyreports/pilot_panel.pyraw_scoresin pipeline output)This PR wires
rich_selectivityinto all five paths.Changes
pipeline.py:rich_selectivitynow computed for every candidate and included inraw_scoresalongsideselectivity_proxy(kept for backward comparability).scoring/expert.py:rich_selectivityreplaceshemolysis_safety(AUROC=0.565, not significant) as the expert composite hemolysis-risk component (weight 0.10). The oldselectivity_proxyremains as the expert's activity-ranking selectivity component (weight 0.20) because it carries AMP-vs-decoy signal (AUROC=0.7729). Oldhemolysis_safetypreserved ashemolysis_safety_legacyin expert extras.selection/pilot.py:_pilot_priority()usesrich_selectivityinstead ofselectivity_proxyfor the safety tiebreaker, with backward-compat fallback toselectivity_proxywhenrich_selectivityis absent.reports/pilot_panel.py: pilot CSV and markdown include arich_selectivitycolumn alongsideselectivity_proxyfor expert comparison.benchmark/retrospective.py: expert ablation benchmark referencesrich_selectivityinstead ofhemolysis_safetyin per-component AUROC.tests/test_rich_selectivity_integration.py: 14 new integration tests verifyingrich_selectivityflows through pipeline scores, evidence certificates, expert composite weights/components, pilot priority formula (with fallback), and pilot panel report output.Honest Results
Expert composite AUROC drops slightly: 0.7119 → 0.7097 (−0.0022) on AMP-vs-decoy. This is expected and acceptable:
rich_selectivityis anti-AMP by design (AUROC=0.1973 for AMP-vs-decoy) because it penalises high hydrophobicity and charge that define AMPs.selectivity_proxy(weight 0.20 in expert) compensates for the anti-AMP bias ofrich_selectivity(weight 0.10), keeping the net AUROC drop minimal.Verification
uv run pytest -q— 1571 passed, 7 skippedruff check src tests— cleangit diff --check— cleanmake metrics-snapshot— regenerated with updated expert AUROCpytest tests/test_docs_consistent.py— 12 passedConstraint: wet-lab hemolysis assay remains mandatory; computational selectivity scores are triage signals only, not biological proof.
Confidence: high — rich selectivity detection AUROC=0.7138 (CI 0.63-0.80, significant); integration verified by 1571 passing tests.
Tested: uv run pytest -q (1571 passed, 7 skipped), ruff check (clean), git diff --check (clean), make metrics-snapshot.
Co-authored-by: OmX omx@oh-my-codex.dev