You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 75d2361
Browse filesBrowse the repository at this point in the historyBrowse files
Build the multi-resolution virtual assay layer: structure proxies, membrane interaction models, uncertainty-aware surrogates. Every module must justify itself against cheap heuristics.
98
99
@@ -108,6 +109,7 @@ Build the multi-resolution virtual assay layer: structure proxies, membrane inte
108
109
| 37 ✅ | No `--simulation-mode` CLI flag. Simulation modules exist but users cannot access them | Added `--simulation-mode` to `rank` CLI: `off` (default) and `info` (runs MembraneProxy + StructureProxy, adds `sim_*` keys to scores, includes in JSONL/report). Weighted blocked by gate (Loop 35). v0.5.57 | 6 tests, 1940 total. Demo pipeline with `--simulation-mode info` produces sim scores for all 10 candidates. Report includes sim columns |
109
110
| 38 ✅ | No API contract for third-party simulation modules (e.g. Martini MD, AlphaFold) |`simulation/interfaces.py` extended with `ExternalSimulationAdapter` protocol. ARCHITECTURE.md extension points documented. v0.5.59 | 13 tests, 1966 total. Adapter wraps any callable, checks availability, handles errors, propagates metadata |
110
111
| 39 ✅ | Virtual-assay benchmark report comparing simulation-augmented vs simulation-free ranking on the strict triage set |`docs/SIMULATION_BENCHMARK.md`: head-to-head report for simulation ablation, cheap-baseline comparison, and weighted-mode gate. v0.5.60. | Honest result: simulation did not improve ranking; weighted mode remains blocked. |
112
+
| 40 ✅ | Uncertainty existed inside simulation results but did not reach selected-candidate evidence certificates |`rank --simulation-mode info` now carries `sim_membrane_uncertainty`, `sim_structure_uncertainty`, and `sim_max_uncertainty` into ranked JSONL rows, Markdown reports, and evidence certificates. v0.5.61. | Evidence certificates validate against schema with uncertainty fields; high uncertainty adds an explicit no-ranking-impact failure mode. |
111
113
112
114
**Phase 3 exit criteria:**
113
115
- At least 2 simulation modules exist and are benchmarked
@@ -118,22 +120,22 @@ Build the multi-resolution virtual assay layer: structure proxies, membrane inte
|40| No assay-partner onboarding pack. A new CRO gets no README for how to run the panel |`docs/LAB_PARTNER_ONBOARDING.md`: panel manifest, synthesis instructions, assay protocol references, data return schema, positive/negative control sequences | A new CRO can read this and know exactly what to do |
128
-
|41| No pre-registered pass/fail criteria for the Wave 1 wet-lab batch. ASSAY_PREREGISTRATION exists but criteria are not machine-checkable |`configs/wave1_pass_fail.yaml`: machine-readable pass/fail gates for Wave 1 (MIC ≤ 32 for ≥ N candidates, hemolysis ≤ 10% for ≥ M candidates, at least 1 control passes) | CI validates criteria against lab result schema |
129
-
|42| No expert-review GitHub template. Review process is ad-hoc |`.github/ISSUE_TEMPLATE/expert_review.yml`: structured form for expert reviewers covering novelty, safety, mechanism plausibility, experiment design | Template generates a review issue with all required fields |
130
-
|43| No decision-log format for human review. AGENTS.md requires human review for 6 decision types but no machine-readable log format exists |`schemas/decision_log.schema.json`: JSON Schema for human review decisions. Fields: date, reviewer, decision_type, evidence_refs, reasoning_notes, dissent_flag | Schema validates against real decision examples |
131
-
|44| No candidate batch pack that a lab partner can download as a single artifact |`scripts/build_lab_batch_pack.py`: generates a zip with candidate CSV, evidence certificates, synthesis order, assay protocol refs, controls manifest, and data return template |`make lab-batch-pack` produces a single zip with all required files |
132
-
|45| No chain-of-custody hashing for lab samples. Lab receives peptides but can't verify identity | Add SHA-256 hash of each candidate sequence and synthesis order to the batch pack. Lab can verify received peptides match | Hash verification script works end-to-end on synthetic data |
133
-
|46| No pre-registered analysis plan for Wave 1 results. What analysis runs when data comes back? |`docs/WAVE1_ANALYSIS_PLAN.md`: pre-registered statistical analysis (primary: MIC ≤ 32 hit rate; secondary: selectivity index; exploratory: per-family correlation). No post-hoc changes allowed | Plan is timestamped and locked before data return |
134
-
|47| No data-return validation tool. Lab sends CSV/JSON files; pipeline needs to validate them against schema |`scripts/validate_lab_data_return.py`: checks returned data against `schemas/lab_result.schema.json`, reports missing fields, out-of-range values, control failures | Returns clear error messages for each validation failure mode |
135
-
|48| No public-data release package. If results are publishable, what goes in the supplement? |`docs/PUBLICATION_PACK.md`: checklist for public release — benchmark data, evidence certificates, candidate scores, negative results, analysis scripts, README for reproduction | Pack is complete enough for a reviewer to reproduce all findings |
136
-
|49|**Culmination loop**: run the full dry-lab pipeline end-to-end, calibrate against any available wet-lab data, produce the final report. Pre-register the discovery claim | Full reproducibility report: all outputs regenerated from versioned inputs, benchmark card finalized, calibration report generated, discovery probability estimated with honest uncertainty |`make full-reproducibility-report` succeeds; all 49 gates pass; report is publishable as a preprint |
129
+
|41| No assay-partner onboarding pack. A new CRO gets no README for how to run the panel |`docs/LAB_PARTNER_ONBOARDING.md`: panel manifest, synthesis instructions, assay protocol references, data return schema, positive/negative control sequences | A new CRO can read this and know exactly what to do |
130
+
|42| No pre-registered pass/fail criteria for the Wave 1 wet-lab batch. ASSAY_PREREGISTRATION exists but criteria are not machine-checkable |`configs/wave1_pass_fail.yaml`: machine-readable pass/fail gates for Wave 1 (MIC ≤ 32 for ≥ N candidates, hemolysis ≤ 10% for ≥ M candidates, at least 1 control passes) | CI validates criteria against lab result schema |
131
+
|43| No expert-review GitHub template. Review process is ad-hoc |`.github/ISSUE_TEMPLATE/expert_review.yml`: structured form for expert reviewers covering novelty, safety, mechanism plausibility, experiment design | Template generates a review issue with all required fields |
132
+
|44| No decision-log format for human review. AGENTS.md requires human review for 6 decision types but no machine-readable log format exists |`schemas/decision_log.schema.json`: JSON Schema for human review decisions. Fields: date, reviewer, decision_type, evidence_refs, reasoning_notes, dissent_flag | Schema validates against real decision examples |
133
+
|45| No candidate batch pack that a lab partner can download as a single artifact |`scripts/build_lab_batch_pack.py`: generates a zip with candidate CSV, evidence certificates, synthesis order, assay protocol refs, controls manifest, and data return template |`make lab-batch-pack` produces a single zip with all required files |
134
+
|46| No chain-of-custody hashing for lab samples. Lab receives peptides but can't verify identity | Add SHA-256 hash of each candidate sequence and synthesis order to the batch pack. Lab can verify received peptides match | Hash verification script works end-to-end on synthetic data |
135
+
|47| No pre-registered analysis plan for Wave 1 results. What analysis runs when data comes back? |`docs/WAVE1_ANALYSIS_PLAN.md`: pre-registered statistical analysis (primary: MIC ≤ 32 hit rate; secondary: selectivity index; exploratory: per-family correlation). No post-hoc changes allowed | Plan is timestamped and locked before data return |
136
+
|48| No data-return validation tool. Lab sends CSV/JSON files; pipeline needs to validate them against schema |`scripts/validate_lab_data_return.py`: checks returned data against `schemas/lab_result.schema.json`, reports missing fields, out-of-range values, control failures | Returns clear error messages for each validation failure mode |
137
+
|49| No public-data release package. If results are publishable, what goes in the supplement? |`docs/PUBLICATION_PACK.md`: checklist for public release — benchmark data, evidence certificates, candidate scores, negative results, analysis scripts, README for reproduction | Pack is complete enough for a reviewer to reproduce all findings |
138
+
|50|**Culmination loop**: run the full dry-lab pipeline end-to-end, calibrate against any available wet-lab data, produce the final report. Pre-register the discovery claim | Full reproducibility report: all outputs regenerated from versioned inputs, benchmark card finalized, calibration report generated, discovery probability estimated with honest uncertainty |`make full-reproducibility-report` succeeds; all 50 gates pass; report is publishable as a preprint |
Phase 4 (Loops 40–49): Wet-Lab Readiness ←── both tracks
164
+
Phase 4 (Loops 41–50): Wet-Lab Readiness ←── both tracks
163
165
```
164
166
165
167
**Parallel tracks:** Loops 30–39 (virtual assay) and Loops 18–29 (calibration engine) are
@@ -180,8 +182,8 @@ If no data arrives, virtual assay scaffolding continues independently.
180
182
Phase 0: ✅ Complete (Loops 1–8)
181
183
Phase 1: ✅ Complete (Loops 9–17)
182
184
Phase 2: ✅ Complete (Loops 18–29)
183
-
Phase 3: In progress — Loop 38 ✅ (Loops 30–39)
184
-
Phase 4: Not started (Loops 40–49)
185
+
Phase 3: ✅ Complete (Loops 30–40)
186
+
Phase 4: Not started (Loops 41–50)
185
187
```
186
188
187
189
### Completed
@@ -241,7 +243,7 @@ Phase 4: Not started (Loops 40–49)
241
243
| 28 ✅ | Policy version bump workflow for when real data arrives |`scripts/bump_recalibration_policy.py`: standalone script with `--dry-run`, decision-log guard, auto-increment + write. CI guard in `ci.yml` validates policy version changes against base branch. v0.5.49. 9 tests. | CI rejects policy PRs without valid decision log; `make bump-policy-version` and `make bump-policy-version-dry-run` available |
242
244
| 29 ✅ | No public negative-result archive format. If Wave 1 yields all negatives, where do they go? |`docs/NEGATIVE_RESULT_ARCHIVE.md`: full template with entry schema, procedures, automation notes, and limitations. Covers pre-selection rejects, selected-untested, lab inactives, lab toxic, control failures. v0.5.50. | Template complete enough for a lab partner to fill; schema defines 18 fields with required/conditional markers |
> **Last updated:** 2026-07-06 (simulation uncertainty in evidence — v0.5.61)
9
+
> **New in v0.5.61:**`rank --simulation-mode info` now propagates `sim_membrane_uncertainty`, `sim_structure_uncertainty`, and `sim_max_uncertainty` into ranked outputs and selected-candidate evidence certificates. This is reviewability only; `weighted` remains blocked. 1 focused regression file updated. 1961 total.
9
10
> **New in v0.5.60:**`docs/SIMULATION_BENCHMARK.md` consolidates simulation ablation, cheap-baseline comparison, and weighted-mode gate results. Current conclusion: simulation does not improve ranking; `weighted` remains blocked.
> **New in v0.5.58:** Per-signal cheap-baseline comparison: 0/4 simulation signals beat their cheapest heuristic. All simulation modules remain permanently experimental. `make bench-simulation-baselines`. 13 tests. 1953 total.
@@ -45,7 +46,7 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
45
46
> **New in v0.5.31:** Added dipeptide-order features for sequence-order awareness. `dipeptide_order_score` achieves AUROC 0.7861 on AMP-vs-scrambled discrimination — the strongest order-dependent feature in the pipeline. Only 7/31 features survive scrambling (amphipathicity/helix-wheel + dipeptide). All composition features are purely position-independent (exactly 0.5000 AUROC on scrambled test).
46
47
> **New in v0.5.30:** Easy baseline benchmark added — charge density alone (AUROC 0.8166) outperforms the full pipeline ensemble (0.7792) on AMP-vs-Swiss-Prot-decoy discrimination. Honest finding documented: expected because pipeline optimizes for safety, not raw discrimination.
47
48
> **New in v0.5.29:** Expanded benchmark to 500 AMPs + 500 composition-matched decoys (n=1000). AUROC 0.7792 (CI₉₅: 0.7505–0.8065) confirms signal generalizes. Cluster-aware CI: 0.746–0.8102. Representative AUROC: 0.778. Standard benchmark (n=191) retained for backward comparison.
48
-
> **Pipeline version:** v0.5.60
49
+
> **Pipeline version:** v0.5.61
49
50
> **Branch:** main
50
51
51
52
---
@@ -1115,7 +1116,7 @@ Decoys score low on activity. Selective AMPs score moderately on both.
0 commit comments