Repository navigation
v0.5.54 — Simulation Ablation Benchmark (Loop 33) - #148
Merged
Merged
Conversation
First honest test of Phase 3 simulation modules. Original Loop 33 (selectivity ratio) superseded by Loop 31 membrane proxy — replaced with ablation benchmark. Changes: - scripts/benchmark_simulation_ablation.py — standalone benchmark testing whether MembraneProxy + StructureProxy improve AMP-vs-decoy discrimination. make bench-simulation-ablation target. Outputs per-module AUROC and combined composite verdict. - tests/test_benchmark_simulation_ablation.py — 17 tests. - docs/50_LOOP_PLAN.md — Loop 33 complete (re-interpreted). Next: Loop 34. - docs/ROADMAP.md — v0.5.54 entry with honest finding. - docs/METRICS_CURRENT.md — Pipeline v0.5.54, 1922 tests. - docs/ARCHITECTURE.md — simulation package map. Honest finding: composite degrades AMP-vs-decoy (delta=-0.1153) but bacterial_binding alone achieves AUROC 0.7512 — genuine non-charge signal from Wimley-White + hydrophobic moment. 1922 tests passing.
cschanhniem
commented
Jul 6, 2026
cschanhniem
left a comment
cschanhniem
left a comment
Collaborator
Author
There was a problem hiding this comment.
Pre-push full suite: 1922 passed, 0 failed. Honest finding documented: simulation composite degrades AMP-vs-decoy discrimination but bacterial_binding alone achieves 0.7512 AUROC (non-charge signal). All 17 new tests pass. Loop 33 re-interpreted from redundant selectivity ratio module to useful ablation benchmark.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First honest test of Phase 3 simulation modules. Original Loop 33 (selectivity ratio) was superseded by Loop 31 membrane proxy -- replaced with a more useful ablation benchmark.
Primary:
scripts/benchmark_simulation_ablation.py— tests MembraneProxy + StructureProxy on 500-AMP AMP-vs-decoy benchmark.make bench-simulation-ablation. Output tooutputs/simulation_ablation.json.Honest finding:
bacterial_bindingalone achieves AUROC 0.7512 -- genuine non-charge signal from Wimley-White + hydrophobic momentTests: 17 tests covering AUROC, composite score, full benchmark on 500-set, CLI exit codes, output writing.
Doc updates: 50_LOOP_PLAN (Loop 33 complete, reinterpreted), ROADMAP, METRICS_CURRENT v0.5.54, ARCHITECTURE.
1922 tests passing (pre-push full suite).