v0.5.46 — Active-learning recovery benchmark (Loop 25)

docs/50_LOOP_PLAN.md

@@ -1,8 +1,8 @@
 # 50-Loop Execution Plan
 
-> **Status:** Strategic roadmap. Updated v0.5.45.
-> **Current state:** 1824 tests, pipeline AUROC 0.7792, calibration intake + gate + engine + dry-run + policy-version + synthetic generator + recalibration report + batch-2 selector shipped,
-> Phase 0 complete (Loops 1–8), Phase 1 complete (Loops 9–17), Phase 2 Loops 18, 19, 20, 21, 22, 23, 24 complete,
+> **Status:** Strategic roadmap. Updated v0.5.46.
+> **Current state:** 1832 tests, pipeline AUROC 0.7792, calibration intake + gate + engine + dry-run + policy-version + synthetic generator + recalibration report + batch-2 selector + recovery benchmark shipped,
+> Phase 0 complete (Loops 1–8), Phase 1 complete (Loops 9–17), Phase 2 Loops 18, 19, 20, 21, 22, 23, 24, 25 complete,
 > cluster-split/selectivity/triage now gated in CI, Wave 0.5 panel ready (24 candidates, 15 families), no wet-lab data yet.
 >
 > Each loop = one focused PR: bottleneck identified → implemented → verified → merged.
@@ -75,7 +75,7 @@ Build the recalibration engine gated by the policy. Implement active-learning ba
 | 22 ✅ | No "dry run" mode for recalibration engine. User can't preview what weights would change | `--dry-run` flag: compute and log proposed weight changes without applying them. Outputs comparison table (current vs proposed). v0.5.43. 1 test, Makefile target. | Dry run produces diff without side effects; verified with `make recalibration-engine-dry-run` + manual inspection |
 | 23 ✅ | No weight-change report that a human reviewer can inspect | `reports/recalibration_report.py`: generates human-readable report with before/after weights, rationale, policy-rule results, gate-verdict context, and explicit "human review required" banner. Schema validation via `schemas/recalibration_report.schema.json`. v0.5.44. 9 tests. | Report validates against schema; gate verdict details preserved; CLI uses combined report for JSON/MD output |
 | 24 ✅ | No second-batch selection logic. The gate allows recalibration, but how do we pick the next 8–12 candidates? | `active_learning/selector.py`: implements uncertainty sampling + diversity + safety gate. CLI `select-batch`. v0.5.45. 11 tests. | Selection respects safety constraints; top-5 include at least 1 high-uncertainty probe; all blocked candidates rejected; CLI produces valid JSON manifest |
-| 25 | No active-learning benchmark on synthetic data. Can the selector recover known "hidden active" candidates better than random? | Synthetic benchmark: hide 3 known active AMPs from the panel, run selection, measure how many rounds to recover them | Recovery within N rounds (N pre-registered) |
+| 25 ✅ | No active-learning benchmark on synthetic data. Can the selector recover known "hidden active" candidates better than random? | `active_learning/benchmark.py`: multi-round recovery benchmark with pre-registered thresholds (max_rounds_to_first=3, min_recall=0.33). CLI `bench active-learning`. v0.5.46. 8 tests. | Recovery verified: selector recovers hidden actives within pre-registered thresholds; compared against random baseline (20-trial average) |
 | 26 | No integration between active-learning selector and the calibration pipeline | End-to-end flow: intake → gate (pass) → recalibration engine (dry-run) → selector → batch 2 manifest | Full chain runs on synthetic data, produces a batch-2 manifest |
 | 27 | No end-to-end regression test for the full calibration loop (the "golden path") | `tests/test_calibration_full_loop.py`: generate synthetic results → intake → gate → dry-run recalibration → selector → batch manifest. Assert all exit codes 0 | Full golden path tested on every PR |
 | 28 | The recalibration policy (v0.5.20) has version 1. No bump workflow exists for when real data arrives | `scripts/bump_recalibration_policy.py`: bumps `policy_version`, requires a non-empty decision-log entry dated within 30 days. CI guard enforces this | CI rejects policy PRs without valid decision log |
@@ -177,7 +177,7 @@ If no data arrives, virtual assay scaffolding continues independently.
 ```
 Phase 0: ✅ Complete (Loops 1–8)
 Phase 1: ✅ Complete (Loops 9–17)
-Phase 2: Loop 24 of 29 (batch-2 selector shipped — Loop 25 next)
+Phase 2: Loop 25 of 29 (recovery benchmark shipped — Loop 26 next)
 Phase 3: Not started (Loops 30–39)
 Phase 4: Not started (Loops 40–49)
 ```
@@ -232,4 +232,4 @@ Phase 4: Not started (Loops 40–49)
 | 21 ✅ | Recalibration engine: the gate existed but no engine. Real weight-update code needs to be safe, auditable, and gated | `calibration/engine.py`: `compute_weight_update()` returns `WeightUpdateProposal`. Control theory: delta = learning_rate × (observed_accuracy − target_accuracy). Conservative LR 0.05. L1 budget enforced. v0.5.36 | 1735 tests; CLI recalibration-engine exits 0 on proposal, 3 on violations, 2 on missing files; 12 engine tests |
 | 22 ✅ | No dry-run mode for recalibration engine. User can't preview what weights would change | `--dry-run` flag: compute and log proposed weight changes without applying them. Outputs comparison table (current vs proposed). v0.5.43. 1 test, Makefile target. | Dry run produces diff without side effects; verified with `make recalibration-engine-dry-run` + manual inspection |
 
-**Next loop:** Loop 24 — Phase 2 (Second-batch selection logic).
+**Next loop:** Loop 26 — Phase 2 (Integration between active-learning selector and calibration pipeline).

docs/ROADMAP.md

@@ -1,5 +1,29 @@
 # Roadmap
 
+## v0.5.46 — Active-Learning Recovery Benchmark ✓ (2026-07-06)
+
+The batch-2 selector could pick candidates but there was no benchmark
+validating that it recovers known active sequences faster than random
+baseline. This release adds a multi-round recovery benchmark with
+pre-registered thresholds.
+
+Changes:
+- Added ``active_learning/benchmark.py`` — ``run_active_learning_benchmark()``
+  simulates a multi-round active-learning loop: hides N active candidates,
+  runs ``select_batch_2`` each round on the remaining pool, tracks rounds-to-
+  first-recovery and cumulative recall. Compares against a random-baseline
+  averaged over 20 trials.
+- ``generate_benchmark_pool()`` creates synthetic candidates where actives
+  have higher ensemble scores and lower disagreement than inactives.
+- Pre-registered thresholds: ``PREREGISTERED_MAX_ROUNDS_TO_FIRST_RECOVERY=3``,
+  ``PREREGISTERED_MIN_RECALL=0.33``. The ``passed`` field encodes both.
+- Added CLI ``bench active-learning`` — generates a synthetic pool inline if
+  no ``--pool-csv`` is provided.
+- Added 8 tests: pool generation, result shape, recovery assertion,
+  comparison to random, empty/absent pool rejection, threshold constants,
+  CLI integration.
+- 1832 total tests passing.
+
 ## v0.5.45 — Active-Learning Batch-2 Selector ✓ (2026-07-06)
 
 The recalibration pipeline could compute weight updates and produce a reviewable

src/openamp_foundry/active_learning/__init__.py

@@ -9,14 +9,29 @@
 - **Diversity**: candidates that are structurally distinct from the first
   batch, to explore new sequence space.
 - **Safety**: candidates that pass a minimum safety/toxicity threshold.
+
+The benchmark module validates whether the selector can recover known active
+candidates faster than random baseline — a code-path integrity check.
 """
 
+from openamp_foundry.active_learning.benchmark import (
+    ActiveLearningBenchmarkResult,
+    RecoveryRound,
+    generate_benchmark_pool,
+    run_active_learning_benchmark,
+    write_benchmark_pool,
+)
 from openamp_foundry.active_learning.selector import (
     BatchSelection,
     select_batch_2,
 )
 
 __all__ = [
+    "ActiveLearningBenchmarkResult",
     "BatchSelection",
+    "RecoveryRound",
+    "generate_benchmark_pool",
+    "run_active_learning_benchmark",
     "select_batch_2",
+    "write_benchmark_pool",
 ]