v0.5.45 — Active-learning batch-2 selector (Loop 24)

docs/50_LOOP_PLAN.md

@@ -1,8 +1,8 @@
 # 50-Loop Execution Plan
 
-> **Status:** Strategic roadmap. Updated v0.5.44.
-> **Current state:** 1813 tests, pipeline AUROC 0.7792, calibration intake + gate + engine + dry-run + policy-version + synthetic generator + recalibration report shipped,
-> Phase 0 complete (Loops 1–8), Phase 1 complete (Loops 9–17), Phase 2 Loops 18, 19, 20, 21, 22, 23 complete,
+> **Status:** Strategic roadmap. Updated v0.5.45.
+> **Current state:** 1824 tests, pipeline AUROC 0.7792, calibration intake + gate + engine + dry-run + policy-version + synthetic generator + recalibration report + batch-2 selector shipped,
+> Phase 0 complete (Loops 1–8), Phase 1 complete (Loops 9–17), Phase 2 Loops 18, 19, 20, 21, 22, 23, 24 complete,
 > cluster-split/selectivity/triage now gated in CI, Wave 0.5 panel ready (24 candidates, 15 families), no wet-lab data yet.
 >
 > Each loop = one focused PR: bottleneck identified → implemented → verified → merged.
@@ -74,7 +74,7 @@ Build the recalibration engine gated by the policy. Implement active-learning ba
 | 21 ✅ | **Recalibration engine**: the gate existed but no engine. Real weight-update code needs to be safe, auditable, and gated | `calibration/engine.py`: `compute_weight_update(intake_report, gate_verdict, current_weights, l1_budget)` returns `WeightUpdateProposal`. Control theory: delta = learning_rate × (observed_accuracy − target_accuracy). Conservative learning rate 0.05. L1 budget enforced. v0.5.36 | 1735 tests; CLI `recalibration-engine` exits 0 on proposal, 3 on violations, 2 on missing files; 12 engine tests |
 | 22 ✅ | No "dry run" mode for recalibration engine. User can't preview what weights would change | `--dry-run` flag: compute and log proposed weight changes without applying them. Outputs comparison table (current vs proposed). v0.5.43. 1 test, Makefile target. | Dry run produces diff without side effects; verified with `make recalibration-engine-dry-run` + manual inspection |
 | 23 ✅ | No weight-change report that a human reviewer can inspect | `reports/recalibration_report.py`: generates human-readable report with before/after weights, rationale, policy-rule results, gate-verdict context, and explicit "human review required" banner. Schema validation via `schemas/recalibration_report.schema.json`. v0.5.44. 9 tests. | Report validates against schema; gate verdict details preserved; CLI uses combined report for JSON/MD output |
-| 24 | No second-batch selection logic. The gate allows recalibration, but how do we pick the next 8–12 candidates? | `active_learning/selector.py`: implements uncertainty sampling + diversity + safety gate. Returns ranked candidates for batch 2 | Selection respects safety constraints; top-5 include at least 1 high-uncertainty probe |
+| 24 ✅ | No second-batch selection logic. The gate allows recalibration, but how do we pick the next 8–12 candidates? | `active_learning/selector.py`: implements uncertainty sampling + diversity + safety gate. CLI `select-batch`. v0.5.45. 11 tests. | Selection respects safety constraints; top-5 include at least 1 high-uncertainty probe; all blocked candidates rejected; CLI produces valid JSON manifest |
 | 25 | No active-learning benchmark on synthetic data. Can the selector recover known "hidden active" candidates better than random? | Synthetic benchmark: hide 3 known active AMPs from the panel, run selection, measure how many rounds to recover them | Recovery within N rounds (N pre-registered) |
 | 26 | No integration between active-learning selector and the calibration pipeline | End-to-end flow: intake → gate (pass) → recalibration engine (dry-run) → selector → batch 2 manifest | Full chain runs on synthetic data, produces a batch-2 manifest |
 | 27 | No end-to-end regression test for the full calibration loop (the "golden path") | `tests/test_calibration_full_loop.py`: generate synthetic results → intake → gate → dry-run recalibration → selector → batch manifest. Assert all exit codes 0 | Full golden path tested on every PR |
@@ -177,7 +177,7 @@ If no data arrives, virtual assay scaffolding continues independently.
 ```
 Phase 0: ✅ Complete (Loops 1–8)
 Phase 1: ✅ Complete (Loops 9–17)
-Phase 2: Loop 23 of 29 (recalibration report shipped — Loop 24 next)
+Phase 2: Loop 24 of 29 (batch-2 selector shipped — Loop 25 next)
 Phase 3: Not started (Loops 30–39)
 Phase 4: Not started (Loops 40–49)
 ```

docs/ROADMAP.md

@@ -1,5 +1,29 @@
 # Roadmap
 
+## v0.5.45 — Active-Learning Batch-2 Selector ✓ (2026-07-06)
+
+The recalibration pipeline could compute weight updates and produce a reviewable
+report, but had no way to select the next set of candidates for lab testing. This
+release adds a standalone batch-2 selector that uses uncertainty sampling,
+diversity ranking, and a safety gate to pick informative candidates.
+
+Changes:
+- Added ``active_learning/selector.py`` — ``select_batch_2()`` combines
+  per-candidate ensemble score, uncertainty (from model disagreement and
+  ensemble proximity to 0.5), and diversity vs batch 1 (via normalized sequence
+  similarity). The safety gate rejects candidates below configurable safety and
+  selectivity thresholds. At least 1 high-uncertainty probe is guaranteed in
+  the top-N selection.
+- Added ``active_learning/__init__.py`` — exports ``BatchSelection`` and
+  ``select_batch_2``.
+- Added CLI ``select-batch`` — reads a candidate pool CSV and batch-1 IDs,
+  writes a JSON manifest with selected candidates and metadata (version,
+  probes_in_top_n, n_after_safety_gate, notes).
+- Added 11 tests: basic selection, safety gate, selectivity gate, uncertainty
+  probe guarantee, diversity vs batch-1, empty pool, all-gated-out,
+  selection_reason field, metadata shape, CLI success, CLI error path.
+- Updated CLI parser (``main.py``) and handler (``selection.py``).
+
 ## v0.5.44 — Recalibration Report with Schema Validation ✓ (2026-07-06)
 
 The existing weight-update proposal markdown writer was embedded in engine.py

src/openamp_foundry/active_learning/__init__.py

@@ -0,0 +1,22 @@
+"""Active learning: second-batch candidate selection for assay panels.
+
+After an initial batch is tested in the lab and the recalibration pipeline
+computes weight updates, the active-learning selector picks the next batch
+of candidates. It balances:
+
+- **Uncertainty**: candidates where the model is most unsure (high
+  disagreement, or ensemble score near 0.5).
+- **Diversity**: candidates that are structurally distinct from the first
+  batch, to explore new sequence space.
+- **Safety**: candidates that pass a minimum safety/toxicity threshold.
+"""
+
+from openamp_foundry.active_learning.selector import (
+    BatchSelection,
+    select_batch_2,
+)
+
+__all__ = [
+    "BatchSelection",
+    "select_batch_2",
+]

src/openamp_foundry/active_learning/selector.py

@@ -0,0 +1,312 @@
+"""Batch-2 candidate selector: uncertainty, diversity, safety gating.
+
+Picks the next N candidates for lab testing after an initial batch has been
+assayed and the recalibration pipeline has run. This is a **selection triage**
+module, not a scorer or predictor. All scores are pipeline-produced.
+"""
+
+from __future__ import annotations
+
+import csv
+import logging
+from dataclasses import dataclass, field
+from pathlib import Path
+
+_log = logging.getLogger(__name__)
+
+_SELECTOR_VERSION = "0.1.0"
+
+
+@dataclass(frozen=True)
+class BatchSelection:
+    """Result of batch-2 selection."""
+
+    version: str
+    n_requested: int
+    n_remaining: int
+    n_after_safety_gate: int
+    selected: tuple[dict, ...]
+    probes_in_top_n: int
+    notes: tuple[str, ...]
+
+    def to_dict(self) -> dict:
+        return {
+            "version": self.version,
+            "n_requested": self.n_requested,
+            "n_remaining": self.n_remaining,
+            "n_after_safety_gate": self.n_after_safety_gate,
+            "selected": list(self.selected),
+            "probes_in_top_n": self.probes_in_top_n,
+            "notes": list(self.notes),
+        }
+
+
+def _load_candidates(csv_path: Path) -> list[dict]:
+    """Load candidates from a panel CSV."""
+    rows: list[dict] = []
+    with open(csv_path, newline="") as f:
+        reader = csv.DictReader(f)
+        for row in reader:
+            parsed: dict[str, float | int | str] = {}
+            for k, v in row.items():
+                k = k.strip()
+                if v is None:
+                    parsed[k] = None
+                else:
+                    v = v.strip()
+                # Try numeric conversion
+                try:
+                    parsed[k] = float(v)
+                except (ValueError, TypeError):
+                    try:
+                        parsed[k] = int(v)
+                    except (ValueError, TypeError):
+                        parsed[k] = v
+            rows.append(parsed)
+    return rows
+
+
+def _safe_float(val: object, default: float = 0.0) -> float:
+    if isinstance(val, (int, float)):
+        return float(val)
+    return default
+
+
+def _compute_uncertainty(row: dict) -> float:
+    """Compute an uncertainty score in [0, 1] for a candidate.
+
+    Uses the pipeline's built-in ``disagreement`` field (when available)
+    and complements it with the distance of the ``ensemble`` score from 0.5
+    (the most uncertain region for a binary classifier).
+
+    A value of 1.0 means maximum uncertainty; 0.0 means complete certainty.
+    """
+    disagreement = _safe_float(row.get("disagreement"), default=0.0)
+
+    ensemble = _safe_float(row.get("ensemble"), default=0.5)
+    ensemble_distance = 1.0 - 2.0 * abs(ensemble - 0.5)
+    ensemble_distance = max(0.0, min(1.0, ensemble_distance))
+
+    return max(disagreement, ensemble_distance)
+
+
+def _compute_diversity_vs_batch(
+    sequence: str,
+    batch_1_sequences: list[str],
+) -> float:
+    """Compute diversity score for a candidate relative to a set of sequences.
+
+    Uses ``normalized_similarity`` from the novelty module. A score of 1.0
+    means the candidate is completely novel relative to batch 1 (maximally
+    diverse); 0.0 means it is identical to at least one batch-1 candidate.
+    """
+    if not batch_1_sequences or not sequence:
+        return 1.0
+    from openamp_foundry.scoring.novelty import normalized_similarity
+
+    max_sim = max(
+        normalized_similarity(sequence, other)
+        for other in batch_1_sequences
+    )
+    return 1.0 - max_sim
+
+
+def _passes_safety_gate(
+    row: dict,
+    safety_threshold: float = 0.5,
+    selectivity_threshold: float = 0.5,
+) -> tuple[bool, str]:
+    """Check whether a candidate passes the safety gate.
+
+    Returns (passes, reason).
+    """
+    reasons: list[str] = []
+
+    safety = _safe_float(row.get("safety"), default=1.0)
+    if safety < safety_threshold:
+        reasons.append(
+            f"safety={safety:.3f} < {safety_threshold:.3f}"
+        )
+
+    selectivity = _safe_float(row.get("rich_selectivity"))
+    if not selectivity:
+        selectivity = _safe_float(row.get("selectivity_proxy"), default=1.0)
+    if selectivity < selectivity_threshold:
+        reasons.append(
+            f"selectivity={selectivity:.3f} < {selectivity_threshold:.3f}"
+        )
+
+    if reasons:
+        return False, "; ".join(reasons)
+    return True, "passes gate"
+
+
+def select_batch_2(
+    candidates_csv: str | Path,
+    batch_1_ids: list[str],
+    *,
+    n: int = 10,
+    safety_threshold: float = 0.5,
+    selectivity_threshold: float = 0.5,
+    ensemble_weight: float = 0.40,
+    uncertainty_weight: float = 0.30,
+    diversity_weight: float = 0.30,
+    min_uncertainty_probes: int = 1,
+) -> BatchSelection:
+    """Select the next batch of candidates for lab testing.
+
+    Args:
+        candidates_csv: Path to the full candidate pool CSV (same format as
+            the pilot panel CSV with columns: candidate_id, sequence,
+            ensemble, activity, disagreement, safety, novelty, etc.).
+        batch_1_ids: List of candidate IDs already tested in batch 1.
+        n: Desired batch-2 size.
+        safety_threshold: Minimum ``safety`` score (0-1) for a candidate to
+            be eligible. Candidates below this are blocked.
+        selectivity_threshold: Minimum ``rich_selectivity`` (or
+            ``selectivity_proxy`` as fallback) score.
+        ensemble_weight: Weight for the ensemble score in the combined rank.
+        uncertainty_weight: Weight for the uncertainty score.
+        diversity_weight: Weight for the diversity vs batch-1 score.
+        min_uncertainty_probes: At least this many high-uncertainty
+            candidates must appear in the final selection (to actively probe
+            model blind spots).
+
+    Returns:
+        ``BatchSelection`` with the selected candidates (enriched with
+        uncertainty, diversity, and selection reason metadata).
+    """
+    csv_path = Path(candidates_csv)
+    if not csv_path.exists():
+        raise FileNotFoundError(f"Candidates CSV not found: {csv_path}")
+
+    all_candidates = _load_candidates(csv_path)
+
+    batch_1_set = set(batch_1_ids)
+    batch_1_rows = [c for c in all_candidates if c.get("candidate_id") in batch_1_set]
+    batch_1_sequences = [str(c.get("sequence", "")) for c in batch_1_rows]
+
+    remaining = [
+        c for c in all_candidates
+        if c.get("candidate_id") not in batch_1_set
+    ]
+
+    if not remaining:
+        return BatchSelection(
+            version=_SELECTOR_VERSION,
+            n_requested=n,
+            n_remaining=0,
+            n_after_safety_gate=0,
+            selected=(),
+            probes_in_top_n=0,
+            notes=("No remaining candidates to select from.",),
+        )
+
+    # Compute scores and gate
+    scored: list[tuple[float, dict, float, float, float, str]] = []
+    n_gated_out = 0
+    for row in remaining:
+        passes, reason = _passes_safety_gate(
+            row,
+            safety_threshold=safety_threshold,
+            selectivity_threshold=selectivity_threshold,
+        )
+        if not passes:
+            n_gated_out += 1
+            _log.debug(
+                "Candidate %s gated out: %s",
+                row.get("candidate_id", "?"), reason,
+            )
+            continue
+
+        ensemble_score = _safe_float(row.get("ensemble"), default=0.0)
+        uncertainty = _compute_uncertainty(row)
+        diversity = _compute_diversity_vs_batch(
+            str(row.get("sequence", "")),
+            batch_1_sequences,
+        )
+
+        combined = (
+            ensemble_weight * ensemble_score
+            + uncertainty_weight * uncertainty
+            + diversity_weight * diversity
+        )
+
+        scored.append((combined, row, ensemble_score, uncertainty, diversity, reason))
+
+    if not scored:
+        return BatchSelection(
+            version=_SELECTOR_VERSION,
+            n_requested=n,
+            n_remaining=len(remaining),
+            n_after_safety_gate=0,
+            selected=(),
+            probes_in_top_n=0,
+            notes=(f"All {len(remaining)} remaining candidates failed the safety gate.",),
+        )
+
+    # Sort by combined score descending
+    scored.sort(key=lambda x: x[0], reverse=True)
+
+    # Build selection with uncertainty probe guarantee
+    selected_list: list[dict] = []
+    notes: list[str] = []
+
+    all_uncertainty_ranked = sorted(
+        [s for s in scored if s[3] >= 0.6],
+        key=lambda x: x[3], reverse=True,
+    )
+
+    probes_placed = 0
+    taken_ids: set[str] = set()
+
+    # Phase 1: place high-uncertainty probes if needed
+    for _, row, _, _, _, _reason in all_uncertainty_ranked:
+        if probes_placed >= min_uncertainty_probes:
+            break
+        cid = str(row.get("candidate_id", ""))
+        if cid not in taken_ids:
+            taken_ids.add(cid)
+            row_meta = dict(row)
+            row_meta["selection_reason"] = "high_uncertainty_probe"
+            selected_list.append(row_meta)
+            probes_placed += 1
+
+    # Phase 2: fill remaining slots from top-ranked pool (skip taken)
+    for _, row, ens, unc, div, _reason in scored:
+        if len(selected_list) >= n:
+            break
+        cid = str(row.get("candidate_id", ""))
+        if cid in taken_ids:
+            continue
+        taken_ids.add(cid)
+        row_meta = dict(row)
+        row_meta["selection_reason"] = (
+            f"combined={ens * ensemble_weight + unc * uncertainty_weight + div * diversity_weight:.4f}; "
+            f"ensemble={ens:.3f}, uncertainty={unc:.3f}, diversity={div:.3f}"
+        )
+        selected_list.append(row_meta)
+
+    if probes_placed < min_uncertainty_probes:
+        notes.append(
+            f"Only {probes_placed}/{min_uncertainty_probes} high-uncertainty probes "
+            f"found in eligible pool; fewer probes placed than requested."
+        )
+
+    if len(selected_list) < n:
+        notes.append(
+            f"Requested {n} but only {len(selected_list)} candidates "
+            f"available after safety gating and deduplication."
+        )
+
+    n_after_gate = len(scored)
+
+    return BatchSelection(
+        version=_SELECTOR_VERSION,
+        n_requested=n,
+        n_remaining=len(remaining),
+        n_after_safety_gate=n_after_gate,
+        selected=tuple(selected_list),
+        probes_in_top_n=probes_placed,
+        notes=tuple(notes),
+    )