Skip to content

feat(scoring): wire rich selectivity into production pipeline, expert composite, and pilot selection - #131

Merged
cschanhniem merged 1 commit into
mainfrom
agent/oamp-loop-rich-sel-integration
Jul 3, 2026
Merged

cschanhniem merged 1 commit into
mainfrom
agent/oamp-loop-rich-sel-integration

Conversation

@cschanhniem

Copy link
Copy Markdown
Collaborator

Summary

The rich selectivity scorer (v0.5.16, PR #130) was benchmarked and proven significant (detection AUROC=0.7138, CI 0.63-0.80) but was not connected to any production path where candidate selection actually happens. Every candidate was still scored with the old selectivity_proxy (detection AUROC=0.5744, CI 0.50-0.66, NOT significant) for hemolysis risk in:

  • score_candidates() in pipeline.py
  • expert composite scorer (as hemolysis_safety component, AUROC=0.565, not significant)
  • pilot_priority formula in selection/pilot.py
  • pilot panel report in reports/pilot_panel.py
  • evidence certificates (via raw_scores in pipeline output)

This PR wires rich_selectivity into all five paths.

Changes

  1. pipeline.py: rich_selectivity now computed for every candidate and included in raw_scores alongside selectivity_proxy (kept for backward comparability).

  2. scoring/expert.py: rich_selectivity replaces hemolysis_safety (AUROC=0.565, not significant) as the expert composite hemolysis-risk component (weight 0.10). The old selectivity_proxy remains as the expert's activity-ranking selectivity component (weight 0.20) because it carries AMP-vs-decoy signal (AUROC=0.7729). Old hemolysis_safety preserved as hemolysis_safety_legacy in expert extras.

  3. selection/pilot.py: _pilot_priority() uses rich_selectivity instead of selectivity_proxy for the safety tiebreaker, with backward-compat fallback to selectivity_proxy when rich_selectivity is absent.

  4. reports/pilot_panel.py: pilot CSV and markdown include a rich_selectivity column alongside selectivity_proxy for expert comparison.

  5. benchmark/retrospective.py: expert ablation benchmark references rich_selectivity instead of hemolysis_safety in per-component AUROC.

  6. tests/test_rich_selectivity_integration.py: 14 new integration tests verifying rich_selectivity flows through pipeline scores, evidence certificates, expert composite weights/components, pilot priority formula (with fallback), and pilot panel report output.

Honest Results

Expert composite AUROC drops slightly: 0.7119 → 0.7097 (−0.0022) on AMP-vs-decoy. This is expected and acceptable:

  • rich_selectivity is anti-AMP by design (AUROC=0.1973 for AMP-vs-decoy) because it penalises high hydrophobicity and charge that define AMPs.
  • The tradeoff is correct: the expert now includes a significant hemolysis detector (CI excludes 0.5) instead of the old non-significant one (hemolysis_safety, AUROC=0.565).
  • The old selectivity_proxy (weight 0.20 in expert) compensates for the anti-AMP bias of rich_selectivity (weight 0.10), keeping the net AUROC drop minimal.

Verification

  • uv run pytest -q — 1571 passed, 7 skipped
  • ruff check src tests — clean
  • git diff --check — clean
  • make metrics-snapshot — regenerated with updated expert AUROC
  • pytest tests/test_docs_consistent.py — 12 passed

Constraint: wet-lab hemolysis assay remains mandatory; computational selectivity scores are triage signals only, not biological proof.
Confidence: high — rich selectivity detection AUROC=0.7138 (CI 0.63-0.80, significant); integration verified by 1571 passing tests.
Tested: uv run pytest -q (1571 passed, 7 skipped), ruff check (clean), git diff --check (clean), make metrics-snapshot.

Co-authored-by: OmX omx@oh-my-codex.dev

… composite, and pilot selection

The rich selectivity scorer (v0.5.16, PR #130) was benchmarked and proven
significant (detection AUROC=0.7138, CI 0.63-0.80) but was not connected to
any production path where candidate selection actually happens. This PR wires
it into pipeline scoring, expert composite, pilot priority, pilot panel
report, and evidence certificates.

Changes:
- pipeline.py: rich_selectivity computed for every candidate in raw_scores
- scoring/expert.py: rich_selectivity replaces hemolysis_safety (not significant)
  as expert hemolysis-risk component (weight 0.10); old proxy stays as
  activity-ranking selectivity component (weight 0.20)
- selection/pilot.py: pilot_priority uses rich_selectivity with fallback
- reports/pilot_panel.py: rich_selectivity column added to CSV and markdown
- benchmark/retrospective.py: expert ablation uses rich_selectivity component
- tests/test_rich_selectivity_integration.py: 14 integration tests

Expert AUROC drops 0.7119 -> 0.7097 (-0.0022) on AMP-vs-decoy. Acceptable
tradeoff: expert now includes a significant hemolysis detector instead of
a non-significant one.

Constraint: wet-lab hemolysis assay remains mandatory; computational selectivity scores are triage signals only, not biological proof.
Confidence: high - rich selectivity detection AUROC=0.7138 (CI 0.63-0.80, significant); integration verified by 1571 passing tests.
Tested: uv run pytest -q (1571 passed, 7 skipped), ruff check (clean), git diff --check (clean), make metrics-snapshot.

Co-authored-by: OmX <omx@oh-my-codex.dev>
@cschanhniem
cschanhniem merged commit 5773dd8 into main Jul 3, 2026
1 check passed
@cschanhniem
cschanhniem deleted the agent/oamp-loop-rich-sel-integration branch July 3, 2026 13:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant