[cschanhniem]

Chosen bottleneck

Benchmark honesty — the strict triage benchmark (v0.5.14, PR #128) proved no composite scorer passes selective_vs_hemolytic discrimination (AUROC 0.43-0.54). But it did not explain why. The aggregate failure was opaque: which individual features have signal that the composite scorers are cancelling out? No per-feature diagnostic existed.

Why it matters

This is the single deepest scientific credibility gap in the pipeline. The ensemble actively ranks hemolytic AMPs above selective AMPs (AUROC 0.4661 — worse than random). Without knowing why, every future scoring improvement is guessing in the dark.

Changes made

• New benchmark module benchmark/feature_decomp.py: run_feature_decomposition_benchmark() — tests all 30 scalar physicochemical features from compute_features() individually for selective_vs_hemolytic AUROC with bootstrap CIs, direction (risk vs protective), significance flags, and selectivity-proxy usage flags
• Metrics snapshot integration: feature_decomposition section added to build_metrics_snapshot()
• CLI: bench feature-decomp command
• Makefile: bench-feature-decomp target
• 20 new tests in tests/test_feature_decomp.py covering structure, findings, determinism, and snapshot integration
• 3 docs consistency guards in tests/test_docs_consistent.py
• Docs updated: METRICS_CURRENT.md, ROADMAP.md (v0.5.15), ARCHITECTURE.md
• Snapshot regenerated: outputs/metrics_snapshot.json

Key finding

The selectivity proxy ignores the strongest discriminative features.

Feature	Detection AUROC	CI 95%	Used by proxy?
hydrophobic_fraction	0.6745	0.58-0.77	NO
helix_propensity	0.6489	0.54-0.75	NO
net_charge_proxy	0.6394	0.54-0.73	NO
net_charge_ph74	0.6332	0.54-0.73	YES
selectivity_proxy	0.6095	0.52-0.70	YES
interior_trypsin_sites	0.6089	0.51-0.70	NO
longest_repeat_run	0.5946	0.52-0.68	NO
length	0.5785	0.51-0.66	NO

8/30 features have statistically significant signal (CI excludes 0.5). 6 of those 8 are NOT used by the current selectivity model. The best single feature (hydrophobic_fraction) outperforms the selectivity proxy itself (0.6745 vs 0.6095).

This converts the v0.5.14 aggregate failure into actionable diagnostic information: the next loop knows exactly which feature axes to combine into a richer selectivity scorer.

Verification run

• uv run pytest -q: 1539 passed, 7 skipped (was 1516 + 20 new + 3 docs guards)
• uv run ruff check on all changed files: clean
• make bench-feature-decomp: runs end-to-end
• make metrics-snapshot: snapshots include feature_decomposition section
• git diff --check: clean

Risks / uncertainties

• The best single feature (0.6745) is still modest with wide CIs. Combining features may help, but 22/30 features have NO significant signal — this confirms the strict triage finding that 1D physicochemical descriptors alone cannot fully solve selective_vs_hemolytic discrimination.
• Bootstrap CIs use 2000 resamples. The n=45 hemolytic vs n=125 selective comparison is adequate but not large.
• The feature list is comprehensive but not exhaustive — residue-pair or k-mer features are not tested.

Recommended next loop

Build a richer selectivity scorer that combines the top unused significant features (hydrophobic_fraction, helix_propensity, net_charge_proxy, interior_trypsin_sites) and benchmark it against the current proxy on the same selective_vs_hemolytic task. If it clears 0.65 AUROC with significant CIs, integrate it into the expert composite and re-run the full triage benchmark.

Future loop benefit

The next agent/human can now:

1. See exactly which features have signal (instead of a flat aggregate failure)
2. Know which features are unused by the current proxy
3. Have a clear next step: combine the top features into a richer selectivity scorer
4. Run make bench-feature-decomp to re-check after any scoring change
5. Have the results automatically included in make metrics-snapshot