Skip to content

Commit 75d2361

Browse files
author
OpenCode
committed
feat: expose simulation uncertainty in evidence
1 parent c651002 commit 75d2361

9 files changed

Lines changed: 138 additions & 26 deletions

File tree

‎docs/50_LOOP_PLAN.md‎

Lines changed: 21 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
# 50-Loop Execution Plan
22

3-
> **Status:** Strategic roadmap. Updated v0.5.56.
4-
> **Current state:** 1927 passing tests, pipeline AUROC 0.7792, calibration pipeline complete (intake + gate + engine + dry-run + policy-version + synthetic generator + recalibration report + batch-2 selector + recovery benchmark + end-to-end integration + pytest golden-path test + policy bump workflow + negative result archive),
3+
> **Status:** Strategic roadmap. Updated v0.5.61.
4+
> **Current state:** 1961 passing tests, pipeline AUROC 0.7792, calibration pipeline complete (intake + gate + engine + dry-run + policy-version + synthetic generator + recalibration report + batch-2 selector + recovery benchmark + end-to-end integration + pytest golden-path test + policy bump workflow + negative result archive),
55
> > Phase 0 complete (Loops 1–8), Phase 1 complete (Loops 9–17), Phase 2 complete (Loops 18–29),
6+
> Phase 3 complete (Loops 30–40),
67
> cluster-split/selectivity/triage now gated in CI, Wave 0.5 panel ready (24 candidates, 15 families), no wet-lab data yet.
78
>
89
> Each loop = one focused PR: bottleneck identified → implemented → verified → merged.
@@ -92,7 +93,7 @@ Build the recalibration engine gated by the policy. Implement active-learning ba
9293

9394
---
9495

95-
## Phase 3 — Virtual Assay Scaffolding (Loops 30–39)
96+
## Phase 3 — Virtual Assay Scaffolding (Loops 30–40) ✅
9697

9798
Build the multi-resolution virtual assay layer: structure proxies, membrane interaction models, uncertainty-aware surrogates. Every module must justify itself against cheap heuristics.
9899

@@ -108,6 +109,7 @@ Build the multi-resolution virtual assay layer: structure proxies, membrane inte
108109
| 37 ✅ | No `--simulation-mode` CLI flag. Simulation modules exist but users cannot access them | Added `--simulation-mode` to `rank` CLI: `off` (default) and `info` (runs MembraneProxy + StructureProxy, adds `sim_*` keys to scores, includes in JSONL/report). Weighted blocked by gate (Loop 35). v0.5.57 | 6 tests, 1940 total. Demo pipeline with `--simulation-mode info` produces sim scores for all 10 candidates. Report includes sim columns |
109110
| 38 ✅ | No API contract for third-party simulation modules (e.g. Martini MD, AlphaFold) | `simulation/interfaces.py` extended with `ExternalSimulationAdapter` protocol. ARCHITECTURE.md extension points documented. v0.5.59 | 13 tests, 1966 total. Adapter wraps any callable, checks availability, handles errors, propagates metadata |
110111
| 39 ✅ | Virtual-assay benchmark report comparing simulation-augmented vs simulation-free ranking on the strict triage set | `docs/SIMULATION_BENCHMARK.md`: head-to-head report for simulation ablation, cheap-baseline comparison, and weighted-mode gate. v0.5.60. | Honest result: simulation did not improve ranking; weighted mode remains blocked. |
112+
| 40 ✅ | Uncertainty existed inside simulation results but did not reach selected-candidate evidence certificates | `rank --simulation-mode info` now carries `sim_membrane_uncertainty`, `sim_structure_uncertainty`, and `sim_max_uncertainty` into ranked JSONL rows, Markdown reports, and evidence certificates. v0.5.61. | Evidence certificates validate against schema with uncertainty fields; high uncertainty adds an explicit no-ranking-impact failure mode. |
111113

112114
**Phase 3 exit criteria:**
113115
- At least 2 simulation modules exist and are benchmarked
@@ -118,22 +120,22 @@ Build the multi-resolution virtual assay layer: structure proxies, membrane inte
118120

119121
---
120122

121-
## Phase 4 — Wet-Lab Readiness & External Validation (Loops 40–49)
123+
## Phase 4 — Wet-Lab Readiness & External Validation (Loops 41–50)
122124

123125
Prepare for real wet-lab partners. Make the pack reviewable, the evidence traceable, and the process pre-registered.
124126

125127
| Loop | Bottleneck | Deliverable | Verification |
126128
|------|-----------|-------------|-------------|
127-
| 40 | No assay-partner onboarding pack. A new CRO gets no README for how to run the panel | `docs/LAB_PARTNER_ONBOARDING.md`: panel manifest, synthesis instructions, assay protocol references, data return schema, positive/negative control sequences | A new CRO can read this and know exactly what to do |
128-
| 41 | No pre-registered pass/fail criteria for the Wave 1 wet-lab batch. ASSAY_PREREGISTRATION exists but criteria are not machine-checkable | `configs/wave1_pass_fail.yaml`: machine-readable pass/fail gates for Wave 1 (MIC ≤ 32 for ≥ N candidates, hemolysis ≤ 10% for ≥ M candidates, at least 1 control passes) | CI validates criteria against lab result schema |
129-
| 42 | No expert-review GitHub template. Review process is ad-hoc | `.github/ISSUE_TEMPLATE/expert_review.yml`: structured form for expert reviewers covering novelty, safety, mechanism plausibility, experiment design | Template generates a review issue with all required fields |
130-
| 43 | No decision-log format for human review. AGENTS.md requires human review for 6 decision types but no machine-readable log format exists | `schemas/decision_log.schema.json`: JSON Schema for human review decisions. Fields: date, reviewer, decision_type, evidence_refs, reasoning_notes, dissent_flag | Schema validates against real decision examples |
131-
| 44 | No candidate batch pack that a lab partner can download as a single artifact | `scripts/build_lab_batch_pack.py`: generates a zip with candidate CSV, evidence certificates, synthesis order, assay protocol refs, controls manifest, and data return template | `make lab-batch-pack` produces a single zip with all required files |
132-
| 45 | No chain-of-custody hashing for lab samples. Lab receives peptides but can't verify identity | Add SHA-256 hash of each candidate sequence and synthesis order to the batch pack. Lab can verify received peptides match | Hash verification script works end-to-end on synthetic data |
133-
| 46 | No pre-registered analysis plan for Wave 1 results. What analysis runs when data comes back? | `docs/WAVE1_ANALYSIS_PLAN.md`: pre-registered statistical analysis (primary: MIC ≤ 32 hit rate; secondary: selectivity index; exploratory: per-family correlation). No post-hoc changes allowed | Plan is timestamped and locked before data return |
134-
| 47 | No data-return validation tool. Lab sends CSV/JSON files; pipeline needs to validate them against schema | `scripts/validate_lab_data_return.py`: checks returned data against `schemas/lab_result.schema.json`, reports missing fields, out-of-range values, control failures | Returns clear error messages for each validation failure mode |
135-
| 48 | No public-data release package. If results are publishable, what goes in the supplement? | `docs/PUBLICATION_PACK.md`: checklist for public release — benchmark data, evidence certificates, candidate scores, negative results, analysis scripts, README for reproduction | Pack is complete enough for a reviewer to reproduce all findings |
136-
| 49 | **Culmination loop**: run the full dry-lab pipeline end-to-end, calibrate against any available wet-lab data, produce the final report. Pre-register the discovery claim | Full reproducibility report: all outputs regenerated from versioned inputs, benchmark card finalized, calibration report generated, discovery probability estimated with honest uncertainty | `make full-reproducibility-report` succeeds; all 49 gates pass; report is publishable as a preprint |
129+
| 41 | No assay-partner onboarding pack. A new CRO gets no README for how to run the panel | `docs/LAB_PARTNER_ONBOARDING.md`: panel manifest, synthesis instructions, assay protocol references, data return schema, positive/negative control sequences | A new CRO can read this and know exactly what to do |
130+
| 42 | No pre-registered pass/fail criteria for the Wave 1 wet-lab batch. ASSAY_PREREGISTRATION exists but criteria are not machine-checkable | `configs/wave1_pass_fail.yaml`: machine-readable pass/fail gates for Wave 1 (MIC ≤ 32 for ≥ N candidates, hemolysis ≤ 10% for ≥ M candidates, at least 1 control passes) | CI validates criteria against lab result schema |
131+
| 43 | No expert-review GitHub template. Review process is ad-hoc | `.github/ISSUE_TEMPLATE/expert_review.yml`: structured form for expert reviewers covering novelty, safety, mechanism plausibility, experiment design | Template generates a review issue with all required fields |
132+
| 44 | No decision-log format for human review. AGENTS.md requires human review for 6 decision types but no machine-readable log format exists | `schemas/decision_log.schema.json`: JSON Schema for human review decisions. Fields: date, reviewer, decision_type, evidence_refs, reasoning_notes, dissent_flag | Schema validates against real decision examples |
133+
| 45 | No candidate batch pack that a lab partner can download as a single artifact | `scripts/build_lab_batch_pack.py`: generates a zip with candidate CSV, evidence certificates, synthesis order, assay protocol refs, controls manifest, and data return template | `make lab-batch-pack` produces a single zip with all required files |
134+
| 46 | No chain-of-custody hashing for lab samples. Lab receives peptides but can't verify identity | Add SHA-256 hash of each candidate sequence and synthesis order to the batch pack. Lab can verify received peptides match | Hash verification script works end-to-end on synthetic data |
135+
| 47 | No pre-registered analysis plan for Wave 1 results. What analysis runs when data comes back? | `docs/WAVE1_ANALYSIS_PLAN.md`: pre-registered statistical analysis (primary: MIC ≤ 32 hit rate; secondary: selectivity index; exploratory: per-family correlation). No post-hoc changes allowed | Plan is timestamped and locked before data return |
136+
| 48 | No data-return validation tool. Lab sends CSV/JSON files; pipeline needs to validate them against schema | `scripts/validate_lab_data_return.py`: checks returned data against `schemas/lab_result.schema.json`, reports missing fields, out-of-range values, control failures | Returns clear error messages for each validation failure mode |
137+
| 49 | No public-data release package. If results are publishable, what goes in the supplement? | `docs/PUBLICATION_PACK.md`: checklist for public release — benchmark data, evidence certificates, candidate scores, negative results, analysis scripts, README for reproduction | Pack is complete enough for a reviewer to reproduce all findings |
138+
| 50 | **Culmination loop**: run the full dry-lab pipeline end-to-end, calibrate against any available wet-lab data, produce the final report. Pre-register the discovery claim | Full reproducibility report: all outputs regenerated from versioned inputs, benchmark card finalized, calibration report generated, discovery probability estimated with honest uncertainty | `make full-reproducibility-report` succeeds; all 50 gates pass; report is publishable as a preprint |
137139

138140
**Phase 4 exit criteria:**
139141
- Lab partner onboarding pack exists
@@ -157,9 +159,9 @@ Phase 1 (Loops 9–17): Benchmark Honesty
157159
└── justifies →
158160
Phase 2 (Loops 18–29): Calibration Engine
159161
└── requires real data ──┐
160-
Phase 3 (Loops 30–39): Virtual Assay │ (parallel tracks)
162+
Phase 3 (Loops 30–40): Virtual Assay │ (parallel tracks)
161163
└── feeds into ──────────┘
162-
Phase 4 (Loops 40–49): Wet-Lab Readiness ←── both tracks
164+
Phase 4 (Loops 41–50): Wet-Lab Readiness ←── both tracks
163165
```
164166

165167
**Parallel tracks:** Loops 30–39 (virtual assay) and Loops 18–29 (calibration engine) are
@@ -180,8 +182,8 @@ If no data arrives, virtual assay scaffolding continues independently.
180182
Phase 0: ✅ Complete (Loops 1–8)
181183
Phase 1: ✅ Complete (Loops 9–17)
182184
Phase 2: ✅ Complete (Loops 18–29)
183-
Phase 3: In progress — Loop 38 ✅ (Loops 30–39)
184-
Phase 4: Not started (Loops 40–49)
185+
Phase 3: ✅ Complete (Loops 30–40)
186+
Phase 4: Not started (Loops 41–50)
185187
```
186188

187189
### Completed
@@ -241,7 +243,7 @@ Phase 4: Not started (Loops 40–49)
241243
| 28 ✅ | Policy version bump workflow for when real data arrives | `scripts/bump_recalibration_policy.py`: standalone script with `--dry-run`, decision-log guard, auto-increment + write. CI guard in `ci.yml` validates policy version changes against base branch. v0.5.49. 9 tests. | CI rejects policy PRs without valid decision log; `make bump-policy-version` and `make bump-policy-version-dry-run` available |
242244
| 29 ✅ | No public negative-result archive format. If Wave 1 yields all negatives, where do they go? | `docs/NEGATIVE_RESULT_ARCHIVE.md`: full template with entry schema, procedures, automation notes, and limitations. Covers pre-selection rejects, selected-untested, lab inactives, lab toxic, control failures. v0.5.50. | Template complete enough for a lab partner to fill; schema defines 18 fields with required/conditional markers |
243245

244-
**Next loop:** Loop 40 — lab partner onboarding pack.
246+
**Next loop:** Loop 41 — lab partner onboarding pack.
245247

246248
**Phase 2 exit criteria (all 5 met ✅):**
247249
- ✅ `make calibration-loop` runs from clean checkout, produces batch-2 manifest

‎docs/METRICS_CURRENT.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
55
> **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
66
> with this file, this file wins. Updated whenever benchmark/benchmark config changes.
77
>
8-
> **Last updated:** 2026-07-06 (simulation benchmark report — v0.5.60)
8+
> **Last updated:** 2026-07-06 (simulation uncertainty in evidence — v0.5.61)
9+
> **New in v0.5.61:** `rank --simulation-mode info` now propagates `sim_membrane_uncertainty`, `sim_structure_uncertainty`, and `sim_max_uncertainty` into ranked outputs and selected-candidate evidence certificates. This is reviewability only; `weighted` remains blocked. 1 focused regression file updated. 1961 total.
910
> **New in v0.5.60:** `docs/SIMULATION_BENCHMARK.md` consolidates simulation ablation, cheap-baseline comparison, and weighted-mode gate results. Current conclusion: simulation does not improve ranking; `weighted` remains blocked.
1011
> **New in v0.5.59:** `ExternalSimulationAdapter` protocol — wraps third-party callables into `VirtualAssayProxy`. Availability check, graceful error handling, metadata override. ARCHITECTURE.md docs updated. 12 tests. 1965 total.
1112
> **New in v0.5.58:** Per-signal cheap-baseline comparison: 0/4 simulation signals beat their cheapest heuristic. All simulation modules remain permanently experimental. `make bench-simulation-baselines`. 13 tests. 1953 total.
@@ -45,7 +46,7 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
4546
> **New in v0.5.31:** Added dipeptide-order features for sequence-order awareness. `dipeptide_order_score` achieves AUROC 0.7861 on AMP-vs-scrambled discrimination — the strongest order-dependent feature in the pipeline. Only 7/31 features survive scrambling (amphipathicity/helix-wheel + dipeptide). All composition features are purely position-independent (exactly 0.5000 AUROC on scrambled test).
4647
> **New in v0.5.30:** Easy baseline benchmark added — charge density alone (AUROC 0.8166) outperforms the full pipeline ensemble (0.7792) on AMP-vs-Swiss-Prot-decoy discrimination. Honest finding documented: expected because pipeline optimizes for safety, not raw discrimination.
4748
> **New in v0.5.29:** Expanded benchmark to 500 AMPs + 500 composition-matched decoys (n=1000). AUROC 0.7792 (CI₉₅: 0.7505–0.8065) confirms signal generalizes. Cluster-aware CI: 0.746–0.8102. Representative AUROC: 0.778. Standard benchmark (n=191) retained for backward comparison.
48-
> **Pipeline version:** v0.5.60
49+
> **Pipeline version:** v0.5.61
4950
> **Branch:** main
5051
5152
---
@@ -1115,7 +1116,7 @@ Decoys score low on activity. Selective AMPs score moderately on both.
11151116

11161117
| Metric | Value |
11171118
|--------|-------|
1118-
| Total tests | 1965 |
1119+
| Total tests | 1961 |
11191120
| Coverage (branch) | 99% (6 CLI guard lines only) |
11201121
| Source modules at 100% | All pipeline, QC, scoring modules |
11211122

‎docs/ROADMAP.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,32 @@
11
# Roadmap
22

3+
## v0.5.61 — Simulation Uncertainty in Evidence (Loop 40) ✓ (2026-07-06)
4+
5+
Closed the last concrete Phase 3 exit-criterion gap: simulation uncertainty now
6+
travels with selected candidates instead of living only inside proxy return
7+
objects.
8+
9+
Changes:
10+
- ``src/openamp_foundry/pipeline.py`` — `rank --simulation-mode info` now
11+
writes three informational uncertainty fields into candidate scores:
12+
``sim_membrane_uncertainty``, ``sim_structure_uncertainty``, and
13+
``sim_max_uncertainty``. Candidates above the `0.5` uncertainty bar gain an
14+
explicit failure-mode note stating that simulation outputs must not affect
15+
selection.
16+
- ``schemas/candidate.schema.json`` — documents the three uncertainty keys so
17+
evidence certificates can carry them without schema drift.
18+
- ``tests/test_simulation_mode_cli.py`` — covers ranked output, report columns,
19+
evidence-certificate inclusion, and schema validation for the new fields.
20+
- ``docs/VIRTUAL_ASSAY_SCOPE.md`` and ``docs/SIMULATION_BENCHMARK.md`` — synced
21+
to current behavior.
22+
- ``docs/50_LOOP_PLAN.md`` and ``docs/METRICS_CURRENT.md`` — current-position
23+
drift fixed; Loop 40 recorded complete.
24+
25+
Honest finding:
26+
- This does not improve ranking.
27+
- It improves reviewability: virtual-assay outputs now carry their own
28+
uncertainty wherever selected-candidate evidence is consumed.
29+
330
## v0.5.60 — Simulation Benchmark Report (Loop 39) ✓ (2026-07-06)
431

532
Closed the benchmark-report gap with `docs/SIMULATION_BENCHMARK.md`.

‎docs/SIMULATION_BENCHMARK.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,4 +70,7 @@ Until this report changes with stronger evidence:
7070
- `weighted` simulation remains blocked.
7171
- Evidence certificates may include simulation values only as experimental
7272
context, not as activity, safety, or selectivity proof.
73+
- Evidence certificates now include `sim_membrane_uncertainty`,
74+
`sim_structure_uncertainty`, and `sim_max_uncertainty` when simulation
75+
info mode is used.
7376
- No public claim may imply simulation-validated antimicrobial activity.

‎docs/VIRTUAL_ASSAY_SCOPE.md‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,9 @@ have explicit ablation evidence. Current state:
1818
`rich_selectivity` on hemolysis detection.
1919
- `bench simulation-gate` therefore blocks `weighted` mode and downgrades to
2020
`info` until the evidence changes.
21+
- `rank --simulation-mode info` propagates membrane, structure, and maximum
22+
simulation uncertainty into ranked JSONL rows, Markdown reports, and evidence
23+
certificates. These values are informational only.
2124

2225
## Core Principle
2326

@@ -70,6 +73,12 @@ surface that directly. A module with `uncertainty > 0.5` (on a 0–1 scale
7073
where 0 = fully constrained, 1 = unconstrained) is experimental and must
7174
not affect selection.
7275

76+
When `rank --simulation-mode info` is used, selected-candidate evidence
77+
certificates include `sim_membrane_uncertainty`,
78+
`sim_structure_uncertainty`, and `sim_max_uncertainty`. These fields expose
79+
proxy uncertainty for reviewers; they do not change ranking, filtering, or
80+
selection.
81+
7382
## Ablation Requirement
7483

7584
Each module must pass an ablation benchmark before it can affect
@@ -149,6 +158,6 @@ Phase 3 is complete when:
149158
`weighted` mode.
150159
3. Modules that fail the ablation are either removed or permanently
151160
marked experimental.
152-
4. Uncertainty is propagated through to the evidence certificate.
161+
4. Uncertainty is propagated through to the evidence certificate. ✅
153162
5. The external adapter protocol (`ExternalSimulationAdapter`) is
154163
documented in ARCHITECTURE.md.

‎schemas/candidate.schema.json‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,18 @@
3838
"disagreement": {
3939
"type": "number", "minimum": 0, "maximum": 1,
4040
"description": "Absolute difference between activity_likeness and boman_activity. High values indicate scoring uncertainty."
41+
},
42+
"sim_membrane_uncertainty": {
43+
"type": "number", "minimum": 0, "maximum": 1,
44+
"description": "Experimental membrane-proxy uncertainty. Informational only unless weighted simulation is explicitly allowed."
45+
},
46+
"sim_structure_uncertainty": {
47+
"type": "number", "minimum": 0, "maximum": 1,
48+
"description": "Experimental structure-proxy uncertainty. Informational only unless weighted simulation is explicitly allowed."
49+
},
50+
"sim_max_uncertainty": {
51+
"type": "number", "minimum": 0, "maximum": 1,
52+
"description": "Maximum virtual-assay proxy uncertainty for this candidate. Values above 0.5 must not affect selection."
4153
}
4254
}
4355
},

0 commit comments

Comments
 (0)