diff --git a/Makefile b/Makefile index b273db10..b1f4f321 100644 --- a/Makefile +++ b/Makefile @@ -590,6 +590,12 @@ bench-strategy-compare: --out-json outputs/strategy_comparison.json \ --out-md outputs/strategy_comparison.md +batch-rationale: + PYTHONPATH=src $(PYTHON) -m openamp_foundry.cli batch-rationale \ + --n-total 50 --n-active 10 --batch-size 10 \ + --out-json outputs/batch_rationale_report.json \ + --out-md outputs/batch_rationale_report.md + bench-cross-dataset: PYTHONPATH=src $(PYTHON) scripts/benchmarks/benchmark_cross_dataset.py \ --out outputs/cross_dataset_benchmark.json diff --git a/docs/evidence/METRICS_CURRENT.md b/docs/evidence/METRICS_CURRENT.md index 881725d0..2b0424fb 100644 --- a/docs/evidence/METRICS_CURRENT.md +++ b/docs/evidence/METRICS_CURRENT.md @@ -5,7 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees > with this file, this file wins. Updated whenever benchmark/benchmark config changes. > -> **Last updated:** 2026-07-09 (Phase G G4 — active-learning strategy comparison report — v0.5.82) +> **Last updated:** 2026-07-09 (Phase G G5 — batch-2 selection rationale report — v0.5.83) +> **New in v0.5.83:** Batch-2 selection rationale report (G5) — CLI (`openamp-foundry batch-rationale`) that generates a synthetic candidate pool, runs the batch-2 selector with configurable weights, and produces a per-candidate rationale report classifying each selected candidate into exploit / explore / diversity / combined roles. Reports weight configuration, role breakdown summary, per-candidate contributions (ensemble×weight, uncertainty×weight, diversity×weight), safety gate impact, and caveats. Schema (`schemas/batch_rationale_report.schema.json`). `make batch-rationale` target. 19 tests. 2974 total. > **New in v0.5.82:** Active-learning strategy comparison report (G4) — CLI (`openamp-foundry bench strategy-compare`) that compares 5 selection strategies (exploitation, exploration, diversity, combined, random) on the same synthetic pool with identical hidden active candidates. Each strategy runs multi-round recovery of hidden actives; the report ranks strategies by recall, compares the production selector vs pure strategies and random baseline, and produces structured JSON + Markdown output with caveats. Prevents one-selector bias by making strategy performance transparent. Schema (`schemas/active_learning_strategy_comparison.schema.json`). `make bench-strategy-compare` target. 18 tests. 2955 total. > **New in v0.5.81:** Calibration pipeline consistency audit (G3) — CLI (`openamp-foundry calibration-audit`) that checks consistency across calibration pipeline artifacts (intake report, gate verdict, engine proposal, recalibration report). Checks intake↔gate count matching, engine↔gate verdict agreement, engine L1 budget compliance, report↔gate verdict consistency, report↔engine proposal consistency, timestamp sanity, and cohort-metrics warnings. Schema (`schemas/calibration_audit.schema.json`). `make calibration-audit` and `make calibration-audit-example` targets. 18 tests. 2937 total. > **New in v0.5.80:** Negative-result archive completeness checker (F10) — CLI (`scripts/check_negative_archive_completeness.py`) that reads a JSON archive of negative-result entries and checks each entry against completeness criteria: all required fields present, no duplicate candidate_ids, each entry has at least one content field (assay_result, score_safety, reviewer_notes, or reason_detail), date format valid YYYY-MM-DD, and optional intake_report_id references are well-formed. Outputs a structured completeness report as JSON and Markdown with summary, per-check results, and per-entry pass/fail. Schema (`schemas/negative_result_archive_completeness.schema.json`). Example file (`examples/negative_result_archive_example.json`) with 4 toy entries. 35 tests. 2919 total. @@ -74,7 +75,7 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak > **New in v0.5.31:** Added dipeptide-order features for sequence-order awareness. `dipeptide_order_score` achieves AUROC 0.7861 on AMP-vs-scrambled discrimination — the strongest order-dependent feature in the pipeline. Only 7/31 features survive scrambling (amphipathicity/helix-wheel + dipeptide). All composition features are purely position-independent (exactly 0.5000 AUROC on scrambled test). > **New in v0.5.30:** Easy baseline benchmark added — charge density alone (AUROC 0.8166) outperforms the full pipeline ensemble (0.7792) on AMP-vs-Swiss-Prot-decoy discrimination. Honest finding documented: expected because pipeline optimizes for safety, not raw discrimination. > **New in v0.5.29:** Expanded benchmark to 500 AMPs + 500 composition-matched decoys (n=1000). AUROC 0.7792 (CI₉₅: 0.7505–0.8065) confirms signal generalizes. Cluster-aware CI: 0.746–0.8102. Representative AUROC: 0.778. Standard benchmark (n=191) retained for backward comparison. -> **Pipeline version:** v0.5.82 (G4) +> **Pipeline version:** v0.5.83 (G5) > **Branch:** main --- diff --git a/docs/research/ROADMAP.md b/docs/research/ROADMAP.md index fd5513bf..d2ae59f0 100644 --- a/docs/research/ROADMAP.md +++ b/docs/research/ROADMAP.md @@ -1,5 +1,68 @@ # Roadmap +## v0.5.83 — Loop 83: Phase G G5 — Batch-2 Selection Rationale Report ✓ (2026-07-09) + +CLI (`openamp-foundry batch-rationale`) that generates a synthetic candidate pool, +runs the batch-2 selector with configurable weights, and produces a per-candidate +rationale report classifying each selected candidate into exploit / explore / +diversity / combined roles. Report includes weight configuration, role breakdown +summary, per-candidate contribution detail (ensemble×weight, uncertainty×weight, +diversity×weight), safety gate impact, and caveats. Enables reviewers to +understand why each candidate was selected in terms of the three active-learning +roles. + +Changes: +- `src/openamp_foundry/active_learning/batch_rationale.py` (G5) — + `build_batch_rationale_report()` generates a synthetic pool, runs + `select_batch_2` with configurable weights, classifies each selected candidate + into exploit/explore/diversity/combined roles based on which weight + contribution dominates (threshold: > 0.05 above second-place), and produces a + `BatchRationaleReport` with per-candidate rationales, role summary, and + selector metadata. `PerCandidateRationale` dataclass tracks scores, + contributions, safety gate status, and human-readable explanation. + `write_rationale_json()` and `write_rationale_markdown()` produce structured + output. +- `schemas/batch_rationale_report.schema.json` (G5) — JSON Schema Draft 2020-12 + for the batch-2 rationale report. Validates all required fields including + per-candidate rationales, role summary, role descriptions, selected IDs, + and notes. +- `src/openamp_foundry/active_learning/__init__.py` — Exports + `BatchRationaleReport`, `PerCandidateRationale`, `build_batch_rationale_report`, + `write_rationale_json`, `write_rationale_markdown`. +- `src/openamp_foundry/cli/main.py` — Registered `batch-rationale` subcommand + with all argument flags and dispatch to `_run_batch_rationale`. +- `src/openamp_foundry/cli/commands/selection.py` — Added + `_run_batch_rationale()` CLI handler with `--n-total`, `--n-active`, + `--batch-size`, `--safety-threshold`, `--selectivity-threshold`, + `--ensemble-weight`, `--uncertainty-weight`, `--diversity-weight`, + `--min-uncertainty-probes`, `--rng-seed`, `--out-json`, `--out-md` flags. +- `Makefile` — Added `batch-rationale` target with default params writing to + `outputs/batch_rationale_report.json` and `outputs/batch_rationale_report.md`. +- `tests/active_learning/test_batch_rationale.py` — 19 tests covering: all + required top-level fields, candidates selected, per-candidate required fields, + valid role (exploit/explore/diversity/combined), scores in [0,1], role summary + counts match candidates, selected IDs match candidate IDs, probes non-negative, + notes present, weight config matches input, JSON and Markdown output writing, + CLI exit 0, CLI writes files, JSON Schema conformance, high exploitation weight + produces more exploit roles, all candidates have explanations with role mention, + role descriptions present for all roles, empty roles not in summary. +- `docs/evidence/METRICS_CURRENT.md` — v0.5.83 G5 changelog. Test count: 2974. +- `tests/test_test_count_regression.py` — baseline updated to 2974. + +Honest boundaries: +- This report uses **synthetic data** with known labels. Results reflect + code-path integrity, not biological performance. +- Role classification is based solely on weight contributions. A candidate + classified as "exploit" may also have meaningful uncertainty or diversity + signal — the role label is the dominant contribution, not an exclusive + category. +- The threshold (> 0.05 above second-place) is an arbitrary cutoff; candidates + near the boundary are classified as "combined" to avoid false precision. +- The production selector optimises for multiple objectives (activity, safety, + diversity) that a single-role label does not fully capture. +- This report is informational and requires qualified human review before + influencing selection decisions. + ## v0.5.82 — Loop 82: Phase G G4 — Active-Learning Strategy Comparison Report ✓ (2026-07-09) CLI (`openamp-foundry bench strategy-compare`) that compares 5 selection strategies diff --git a/schemas/batch_rationale_report.schema.json b/schemas/batch_rationale_report.schema.json new file mode 100644 index 00000000..8d1b1202 --- /dev/null +++ b/schemas/batch_rationale_report.schema.json @@ -0,0 +1,137 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://openamp.ai/schemas/batch_rationale_report.schema.json", + "title": "Batch-2 Selection Rationale Report", + "description": "Report explaining why each candidate was selected in batch-2, decomposing selection into exploit, explore, and diversity roles. All fields are computational results — not biological evidence.", + "type": "object", + "required": [ + "version", + "pool_size", + "n_requested", + "n_selected", + "n_after_safety_gate", + "n_gated_out", + "ensemble_weight", + "uncertainty_weight", + "diversity_weight", + "candidates", + "role_summary", + "role_descriptions", + "selected_ids", + "probes_in_top_n", + "notes" + ], + "properties": { + "version": { + "type": "string", + "description": "Schema/report version." + }, + "pool_size": { + "type": "integer", + "minimum": 1, + "description": "Total number of candidates in the pool." + }, + "n_requested": { + "type": "integer", + "minimum": 0, + "description": "Number of candidates requested for batch-2." + }, + "n_selected": { + "type": "integer", + "minimum": 0, + "description": "Number of candidates actually selected." + }, + "n_after_safety_gate": { + "type": "integer", + "minimum": 0, + "description": "Number of candidates remaining after safety gating." + }, + "n_gated_out": { + "type": "integer", + "minimum": 0, + "description": "Number of candidates blocked by safety/selectivity gate." + }, + "ensemble_weight": { + "type": "number", + "minimum": 0, + "maximum": 1, + "description": "Weight for ensemble score in selection." + }, + "uncertainty_weight": { + "type": "number", + "minimum": 0, + "maximum": 1, + "description": "Weight for uncertainty score in selection." + }, + "diversity_weight": { + "type": "number", + "minimum": 0, + "maximum": 1, + "description": "Weight for diversity score in selection." + }, + "candidates": { + "type": "array", + "description": "Per-candidate selection rationales.", + "items": { + "type": "object", + "required": [ + "candidate_id", + "role", + "ensemble_score", + "uncertainty_score", + "diversity_score", + "ensemble_contribution", + "uncertainty_contribution", + "diversity_contribution", + "combined_score", + "passed_safety_gate", + "safety_gate_reason", + "explanation" + ], + "properties": { + "candidate_id": { "type": "string" }, + "role": { + "type": "string", + "enum": ["exploit", "explore", "diversity", "combined"] + }, + "ensemble_score": { "type": "number", "minimum": 0, "maximum": 1 }, + "uncertainty_score": { "type": "number", "minimum": 0, "maximum": 1 }, + "diversity_score": { "type": "number", "minimum": 0, "maximum": 1 }, + "ensemble_contribution": { "type": "number", "minimum": 0, "maximum": 1 }, + "uncertainty_contribution": { "type": "number", "minimum": 0, "maximum": 1 }, + "diversity_contribution": { "type": "number", "minimum": 0, "maximum": 1 }, + "combined_score": { "type": "number", "minimum": 0, "maximum": 1 }, + "passed_safety_gate": { "type": "boolean" }, + "safety_gate_reason": { "type": "string" }, + "explanation": { "type": "string" } + } + } + }, + "role_summary": { + "type": "object", + "description": "Count of candidates per role.", + "additionalProperties": { "type": "integer", "minimum": 0 } + }, + "role_descriptions": { + "type": "object", + "description": "Descriptions for each role.", + "additionalProperties": { "type": "string" } + }, + "selected_ids": { + "type": "array", + "description": "IDs of selected candidates.", + "items": { "type": "string" } + }, + "probes_in_top_n": { + "type": "integer", + "minimum": 0, + "description": "Number of uncertainty probes placed in the top-n." + }, + "notes": { + "type": "array", + "description": "Human-readable notes and caveats.", + "items": { "type": "string" } + } + }, + "_caveat": "This report is a computational decomposition of selection rationale on synthetic data. It is not biological evidence of sampling efficiency, candidate quality, or real-world selection performance." +} diff --git a/src/openamp_foundry/active_learning/__init__.py b/src/openamp_foundry/active_learning/__init__.py index de047862..cf8613ac 100644 --- a/src/openamp_foundry/active_learning/__init__.py +++ b/src/openamp_foundry/active_learning/__init__.py @@ -29,6 +29,13 @@ BatchSelection, select_batch_2, ) +from openamp_foundry.active_learning.batch_rationale import ( + BatchRationaleReport, + PerCandidateRationale, + build_batch_rationale_report, + write_rationale_json, + write_rationale_markdown, +) from openamp_foundry.active_learning.strategy_comparison import ( STRATEGY_WEIGHTS, StrategyComparisonReport, @@ -40,11 +47,14 @@ __all__ = [ "ActiveLearningBenchmarkResult", + "BatchRationaleReport", "BatchSelection", + "PerCandidateRationale", "RecoveryRound", "STRATEGY_WEIGHTS", "StrategyComparisonReport", "StrategyResult", + "build_batch_rationale_report", "generate_benchmark_pool", "run_active_learning_benchmark", "run_strategy_comparison", @@ -52,4 +62,6 @@ "write_benchmark_pool", "write_comparison_json", "write_comparison_markdown", + "write_rationale_json", + "write_rationale_markdown", ] diff --git a/src/openamp_foundry/active_learning/batch_rationale.py b/src/openamp_foundry/active_learning/batch_rationale.py new file mode 100644 index 00000000..0df248dc --- /dev/null +++ b/src/openamp_foundry/active_learning/batch_rationale.py @@ -0,0 +1,533 @@ +"""Batch-2 rationale report: explains why each candidate was selected. + +Decomposes the batch-2 selection into exploit, explore, and diversity roles +so that reviewers can understand the selection logic at the per-candidate +level. Every role explanation is based on the same ``select_batch_2`` +weights and scores — no additional modelling. + +Role definitions: + +- **exploit**: candidate selected primarily for high ensemble score + (ensemble_weight contribution dominates). +- **explore**: candidate selected primarily for high uncertainty + (uncertainty_weight contribution dominates). +- **diversity**: candidate selected primarily for structural novelty + vs batch-1 (diversity_weight contribution dominates). +- **combined**: balanced contributions from all three weights — + the production selector default. +- **safety_gated**: candidate was blocked by the safety or + selectivity gate. + +This report is informational and requires qualified human review +before influencing selection decisions. +""" + +from __future__ import annotations + +import csv +import logging +import random +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + +_log = logging.getLogger(__name__) + +_VERSION = "0.1.0" + +# Default selector weights +_DEFAULT_ENSEMBLE_WEIGHT = 0.40 +_DEFAULT_UNCERTAINTY_WEIGHT = 0.30 +_DEFAULT_DIVERSITY_WEIGHT = 0.30 + + +def _classify_role( + ensemble_contrib: float, + uncertainty_contrib: float, + diversity_contrib: float, +) -> str: + """Classify a candidate's selection role based on which weight + contribution dominates. + + Returns one of 'exploit', 'explore', 'diversity', or 'combined'. + """ + contributions = [ + ("exploit", ensemble_contrib), + ("explore", uncertainty_contrib), + ("diversity", diversity_contrib), + ] + contributions.sort(key=lambda x: x[1], reverse=True) + top_role, top_val = contributions[0] + second_val = contributions[1][1] + + if top_val > second_val + 0.05: + return top_role + return "combined" + + +@dataclass(frozen=True) +class PerCandidateRationale: + """Rationale for a single candidate's selection.""" + + candidate_id: str + role: str + ensemble_score: float + uncertainty_score: float + diversity_score: float + ensemble_contribution: float + uncertainty_contribution: float + diversity_contribution: float + combined_score: float + passed_safety_gate: bool + safety_gate_reason: str + explanation: str + + def to_dict(self) -> dict[str, Any]: + return { + "candidate_id": self.candidate_id, + "role": self.role, + "ensemble_score": self.ensemble_score, + "uncertainty_score": self.uncertainty_score, + "diversity_score": self.diversity_score, + "ensemble_contribution": self.ensemble_contribution, + "uncertainty_contribution": self.uncertainty_contribution, + "diversity_contribution": self.diversity_contribution, + "combined_score": self.combined_score, + "passed_safety_gate": self.passed_safety_gate, + "safety_gate_reason": self.safety_gate_reason, + "explanation": self.explanation, + } + + +@dataclass(frozen=True) +class BatchRationaleReport: + """Report explaining the batch-2 selection rationale.""" + + version: str + pool_size: int + n_requested: int + n_selected: int + n_after_safety_gate: int + n_gated_out: int + ensemble_weight: float + uncertainty_weight: float + diversity_weight: float + candidates: tuple[PerCandidateRationale, ...] + role_summary: dict[str, int] + role_descriptions: dict[str, str] + selected_ids: tuple[str, ...] + probes_in_top_n: int + notes: tuple[str, ...] + + def to_dict(self) -> dict[str, Any]: + return { + "version": self.version, + "pool_size": self.pool_size, + "n_requested": self.n_requested, + "n_selected": self.n_selected, + "n_after_safety_gate": self.n_after_safety_gate, + "n_gated_out": self.n_gated_out, + "ensemble_weight": self.ensemble_weight, + "uncertainty_weight": self.uncertainty_weight, + "diversity_weight": self.diversity_weight, + "candidates": [c.to_dict() for c in self.candidates], + "role_summary": self.role_summary, + "role_descriptions": self.role_descriptions, + "selected_ids": list(self.selected_ids), + "probes_in_top_n": self.probes_in_top_n, + "notes": list(self.notes), + } + + +def _generate_pool( + *, + n_total: int = 50, + n_active: int = 10, + rng_seed: int = 42, +) -> list[dict]: + """Generate a synthetic candidate pool for the rationale report. + + Active candidates get higher ensemble scores and lower disagreement + to create a realistic mix of exploit/explore/diversity candidates. + """ + rng = random.Random(rng_seed) + pool: list[dict] = [] + for i in range(n_total): + is_active = i < n_active + cid = f"RATIONAL-CAND-{i:04d}" + if is_active: + ensemble = rng.uniform(0.75, 0.95) + disagreement = rng.uniform(0.05, 0.20) + else: + ensemble = rng.uniform(0.30, 0.55) + disagreement = rng.uniform(0.20, 0.50) + pool.append({ + "pilot_rank": str(i + 1), + "candidate_id": cid, + "sequence": f"PEPTIDE{i:04d}AKLF", + "length": 12, + "seed": "RATIONAL-SEED", + "ensemble": round(ensemble, 4), + "activity": round(ensemble, 4), + "boman_activity": "0.50", + "disagreement": round(disagreement, 4), + "safety": "0.90", + "synthesis": "0.85", + "novelty": "0.70", + "serum_stability": "0.65", + "selectivity_proxy": "0.60", + "rich_selectivity": "0.75", + "pilot_priority": "0.75", + "amphipathic_score": "1.5", + "charge_ph74": "4.0", + "label": 1 if is_active else 0, + }) + return pool + + +def _write_pool(pool: list[dict], path: Path) -> Path: + fieldnames = list(pool[0].keys()) if pool else [] + with open(path, "w", newline="") as f: + writer = csv.DictWriter(f, fieldnames=fieldnames) + writer.writeheader() + for row in pool: + writer.writerow(row) + return path + + +def build_batch_rationale_report( + *, + n_total: int = 50, + n_active: int = 10, + batch_size: int = 10, + safety_threshold: float = 0.5, + selectivity_threshold: float = 0.5, + ensemble_weight: float = _DEFAULT_ENSEMBLE_WEIGHT, + uncertainty_weight: float = _DEFAULT_UNCERTAINTY_WEIGHT, + diversity_weight: float = _DEFAULT_DIVERSITY_WEIGHT, + min_uncertainty_probes: int = 1, + rng_seed: int = 42, +) -> BatchRationaleReport: + """Build a batch-2 rationale report from a synthetic pool. + + Generates a synthetic candidate pool with known active/inactive labels, + runs ``select_batch_2`` with the specified weights, and produces a + per-candidate rationale report classifying each selected candidate + into exploit / explore / diversity / combined roles. + + All data is synthetic. The report is a code-path integrity check, + not biological evidence. + + Returns: + ``BatchRationaleReport`` with per-candidate rationales and + role summary. + """ + from openamp_foundry.active_learning.selector import ( + _compute_diversity_vs_batch, + _compute_uncertainty, + _passes_safety_gate, + _safe_float, + select_batch_2, + ) + + import tempfile + + rng = random.Random(rng_seed) + pool = _generate_pool( + n_total=n_total, + n_active=n_active, + rng_seed=rng_seed, + ) + + all_ids = [str(c["candidate_id"]) for c in pool] + batch_1_ids = rng.sample(all_ids, max(1, n_total // 5)) + batch_1_rows = [c for c in pool if str(c.get("candidate_id", "")) in batch_1_ids] + batch_1_sequences = [str(c.get("sequence", "")) for c in batch_1_rows] + + pool_csv = Path(tempfile.mktemp(suffix=".csv")) + try: + _write_pool(pool, pool_csv) + + selection = select_batch_2( + candidates_csv=pool_csv, + batch_1_ids=batch_1_ids, + n=batch_size, + safety_threshold=safety_threshold, + selectivity_threshold=selectivity_threshold, + ensemble_weight=ensemble_weight, + uncertainty_weight=uncertainty_weight, + diversity_weight=diversity_weight, + min_uncertainty_probes=min_uncertainty_probes, + ) + finally: + pool_csv.unlink(missing_ok=True) + + # Count gated-out candidates + n_gated_out = selection.n_remaining - selection.n_after_safety_gate + + # Build per-candidate rationales + candidates_list: list[PerCandidateRationale] = [] + for sel in selection.selected: + cid = str(sel.get("candidate_id", "?")) + ens = _safe_float(sel.get("ensemble"), default=0.0) + unc = _compute_uncertainty(sel) + div = _compute_diversity_vs_batch( + str(sel.get("sequence", "")), + batch_1_sequences, + ) + + ens_contrib = ensemble_weight * ens + unc_contrib = uncertainty_weight * unc + div_contrib = diversity_weight * div + combined = ens_contrib + unc_contrib + div_contrib + + passes, reason = _passes_safety_gate( + sel, + safety_threshold=safety_threshold, + selectivity_threshold=selectivity_threshold, + ) + + role = _classify_role(ens_contrib, unc_contrib, div_contrib) + + if role == "exploit": + explanation = ( + f"Candidate {cid} selected primarily for high ensemble score " + f"(ensemble contribution {ens_contrib:.3f} dominates). " + f"Exploitation targets known high-scoring regions." + ) + elif role == "explore": + explanation = ( + f"Candidate {cid} selected primarily for high uncertainty " + f"(uncertainty contribution {unc_contrib:.3f} dominates). " + f"Exploration probes model blind spots." + ) + elif role == "diversity": + explanation = ( + f"Candidate {cid} selected primarily for structural novelty " + f"vs batch-1 (diversity contribution {div_contrib:.3f} dominates). " + f"Diversity expands sequence space coverage." + ) + else: + explanation = ( + f"Candidate {cid} selected with balanced contributions " + f"(ensemble {ens_contrib:.3f}, uncertainty {unc_contrib:.3f}, " + f"diversity {div_contrib:.3f}). Combined production tradeoff." + ) + + candidates_list.append(PerCandidateRationale( + candidate_id=cid, + role=role, + ensemble_score=round(ens, 4), + uncertainty_score=round(unc, 4), + diversity_score=round(div, 4), + ensemble_contribution=round(ens_contrib, 4), + uncertainty_contribution=round(unc_contrib, 4), + diversity_contribution=round(div_contrib, 4), + combined_score=round(combined, 4), + passed_safety_gate=passes, + safety_gate_reason=reason, + explanation=explanation, + )) + + # Build role summary + role_summary: dict[str, int] = {} + for c in candidates_list: + role_summary[c.role] = role_summary.get(c.role, 0) + 1 + + role_descriptions = { + "exploit": ( + "Candidates selected primarily for high ensemble score — exploitation " + "targets known high-scoring regions of sequence space." + ), + "explore": ( + "Candidates selected primarily for high uncertainty — exploration probes " + "model blind spots where the ensemble is most uncertain." + ), + "diversity": ( + "Candidates selected primarily for structural novelty vs batch-1 — " + "diversity expands sequence space coverage." + ), + "combined": ( + "Candidates selected with balanced contributions from all three weights — " + "production tradeoff between exploitation, exploration, and diversity." + ), + } + + selected_ids = tuple( + str(c.get("candidate_id", "")) for c in selection.selected + ) + + notes: list[str] = [] + notes.append( + f"Pool: {len(pool)} total candidates across {n_active} active " + f"(label=1) and {len(pool) - n_active} inactive (label=0)." + ) + notes.append( + f"Batch-2 requested: {batch_size}, selected: {len(candidates_list)}, " + f"gated-out: {n_gated_out}." + ) + notes.append( + "Role classification is based on which weight contribution dominates " + "(threshold: > 0.05 above second-place contribution)." + ) + if selection.notes: + notes.extend(selection.notes) + + return BatchRationaleReport( + version=_VERSION, + pool_size=len(pool), + n_requested=batch_size, + n_selected=len(candidates_list), + n_after_safety_gate=selection.n_after_safety_gate, + n_gated_out=n_gated_out, + ensemble_weight=ensemble_weight, + uncertainty_weight=uncertainty_weight, + diversity_weight=diversity_weight, + candidates=tuple(candidates_list), + role_summary=role_summary, + role_descriptions=role_descriptions, + selected_ids=selected_ids, + probes_in_top_n=selection.probes_in_top_n, + notes=tuple(notes), + ) + + +def write_rationale_json( + report: BatchRationaleReport, + path: Path, +) -> Path: + """Write the rationale report to a JSON file.""" + import json + path.write_text(json.dumps(report.to_dict(), indent=2) + "\n") + return path + + +def write_rationale_markdown( + report: BatchRationaleReport, + path: Path, +) -> Path: + """Write a human-readable Markdown rationale report.""" + lines: list[str] = [] + lines.append("# Batch-2 Selection Rationale Report") + lines.append("") + lines.append(f"> **Version:** {report.version}") + lines.append(f"> **Generated:** auto") + lines.append("") + lines.append("## Overview") + lines.append("") + lines.append( + f"Generated rationale report for batch-2 selection from a synthetic pool of " + f"**{report.pool_size}** candidates." + ) + lines.append( + f"Requested: **{report.n_requested}**, Selected: **{report.n_selected}**, " + f"Gated out: **{report.n_gated_out}**." + ) + lines.append("") + + # Weight configuration + lines.append("## Selection Weights") + lines.append("") + lines.append("| Role | Weight | Description |") + lines.append("|------|:-----:|-------------|") + wmap = [ + ("Exploitation (ensemble)", report.ensemble_weight, + "Rewards high ensemble score"), + ("Exploration (uncertainty)", report.uncertainty_weight, + "Rewards high model uncertainty"), + ("Diversity (diversity)", report.diversity_weight, + "Rewards structural novelty vs batch-1"), + ] + for label, weight, desc in wmap: + lines.append(f"| {label} | {weight} | {desc} |") + lines.append("") + + # Role summary + lines.append("## Selection Breakdown by Role") + lines.append("") + lines.append("| Role | Count | Description |") + lines.append("|------|:-----:|-------------|") + for role in ("exploit", "explore", "diversity", "combined"): + count = report.role_summary.get(role, 0) + desc = report.role_descriptions.get(role, "") + lines.append(f"| {role} | {count} | {desc} |") + lines.append("") + + # Per-candidate rationales + lines.append("## Per-Candidate Rationale") + lines.append("") + lines.append( + "| Candidate ID | Role | Ensemble | Uncertainty | Diversity | " + "Combined | Explanation |" + ) + lines.append( + "|:-------------|:----:|:--------:|:-----------:|:---------:|:--------:|:------------|" + ) + for c in report.candidates: + lines.append( + f"| {c.candidate_id} | {c.role} | {c.ensemble_score} | " + f"{c.uncertainty_score} | {c.diversity_score} | " + f"{c.combined_score} | {c.explanation} |" + ) + lines.append("") + + # Role contribution detail + lines.append("## Contribution Detail") + lines.append("") + lines.append( + "| Candidate ID | Role | Ensemble×W | Uncert×W | Diversity×W | " + "Classification basis |" + ) + lines.append( + "|:-------------|:----:|:----------:|:---------:|:------------:|:---------------------|" + ) + for c in report.candidates: + basis = ( + f"ensemble dominates" if c.role == "exploit" + else f"uncertainty dominates" if c.role == "explore" + else f"diversity dominates" if c.role == "diversity" + else f"balanced contributions" + ) + lines.append( + f"| {c.candidate_id} | {c.role} | {c.ensemble_contribution} | " + f"{c.uncertainty_contribution} | {c.diversity_contribution} | {basis} |" + ) + lines.append("") + + # Safety gate info + if report.n_gated_out > 0: + lines.append("## Safety Gate") + lines.append("") + lines.append( + f"**{report.n_gated_out}** candidates were gated out by safety/" + f"selectivity thresholds (safety >= {0.5}, " + f"selectivity >= {0.5})." + ) + lines.append( + "Gated-out candidates are not available for selection." + ) + lines.append("") + + # Caveat + lines.append("## Caveats") + lines.append("") + lines.append( + "- This report uses **synthetic data** with known labels. " + "Results reflect code-path integrity, not biological performance." + ) + lines.append( + "- Role classification threshold: a role is assigned when its " + "weighted contribution exceeds the second-best by > 0.05. " + "Candidates near the boundary are classified as 'combined'." + ) + lines.append( + "- The production selector optimises for multiple objectives " + "(activity, safety, diversity) that a single-role label does not capture." + ) + lines.append( + "- This report is informational and requires qualified human review " + "before influencing selection decisions." + ) + + path.write_text("\n".join(lines) + "\n") + return path diff --git a/src/openamp_foundry/cli/commands/selection.py b/src/openamp_foundry/cli/commands/selection.py index c87042d9..f033be62 100644 --- a/src/openamp_foundry/cli/commands/selection.py +++ b/src/openamp_foundry/cli/commands/selection.py @@ -345,6 +345,45 @@ def _run_diversity_check(args: argparse.Namespace) -> int: return 0 +def _run_batch_rationale(args: argparse.Namespace) -> int: + """Generate a batch-2 selection rationale report. + + Creates a synthetic candidate pool, runs the batch-2 selector with + configurable weights, and produces a per-candidate rationale report + classifying each selection into exploit / explore / diversity / + combined roles. + """ + from openamp_foundry.active_learning.batch_rationale import ( + build_batch_rationale_report, + write_rationale_json, + write_rationale_markdown, + ) + + report = build_batch_rationale_report( + n_total=args.n_total, + n_active=args.n_active, + batch_size=args.batch_size, + safety_threshold=args.safety_threshold, + selectivity_threshold=args.selectivity_threshold, + ensemble_weight=args.ensemble_weight, + uncertainty_weight=args.uncertainty_weight, + diversity_weight=args.diversity_weight, + min_uncertainty_probes=args.min_uncertainty_probes, + rng_seed=args.rng_seed, + ) + + out_dict = report.to_dict() + if args.out_json: + write_rationale_json(report, Path(args.out_json)) + print(f"JSON rationale report written to {args.out_json}") + if args.out_md: + write_rationale_markdown(report, Path(args.out_md)) + print(f"Markdown rationale report written to {args.out_md}") + if not args.out_json and not args.out_md: + print(json.dumps(out_dict, indent=2)) + return 0 + + def _run_select_batch(args: argparse.Namespace) -> int: """Select a second batch of candidates for lab testing. diff --git a/src/openamp_foundry/cli/main.py b/src/openamp_foundry/cli/main.py index 3be2c834..5cceecd6 100644 --- a/src/openamp_foundry/cli/main.py +++ b/src/openamp_foundry/cli/main.py @@ -2,7 +2,7 @@ from openamp_foundry.cli.commands.core import _run_generate_batch from openamp_foundry.cli.commands.benchmark import _run_bench, _run_validate_scoring, _run_cluster_split_bench, _run_expert_ablation_bench, _run_selectivity_bench, _run_triage, _run_metrics_snapshot, _run_feature_decomp, _run_active_learning_bench, _run_active_learning_strategy_compare, _run_simulation_gate -from openamp_foundry.cli.commands.selection import _run_pilot_panel, _run_pilot_confident, _run_diversity_check, _run_select_batch +from openamp_foundry.cli.commands.selection import _run_pilot_panel, _run_pilot_confident, _run_diversity_check, _run_select_batch, _run_batch_rationale from openamp_foundry.cli.commands.external import _run_external_predict, _run_external_consensus from openamp_foundry.cli.commands.qc import _run_synthesis_order, _run_presynth_qc from openamp_foundry.cli.commands.reports import _run_reviewer_questionnaire, _run_ip_report, _run_batch_pack, _run_gold_standard, _run_novelty_check_broad, _run_lab_result_report, _run_calibration_intake, _run_recalibration_gate, _run_recalibration_engine, _run_validate_policy_version, _run_calibration_audit @@ -1082,6 +1082,67 @@ def build_parser() -> argparse.ArgumentParser: help="ISO date (YYYY-MM-DD) for 'today'. Defaults to actual today.", ) + # ── Batch-2 rationale report ───────────────────────────────────── + batch_rationale = sub.add_parser( + "batch-rationale", + help=( + "Generate a batch-2 selection rationale report. Creates a " + "synthetic candidate pool, runs the batch-2 selector, and " + "explains each selected candidate in terms of exploit / " + "explore / diversity roles." + ), + ) + batch_rationale.add_argument( + "--n-total", type=int, default=50, + help="Total synthetic pool size (default: 50).", + ) + batch_rationale.add_argument( + "--n-active", type=int, default=10, + help="Number of active (label=1) candidates (default: 10).", + ) + batch_rationale.add_argument( + "--batch-size", type=int, default=10, + help="Desired batch-2 size (default: 10).", + ) + batch_rationale.add_argument( + "--safety-threshold", type=float, default=0.5, + help="Minimum safety score (default: 0.5).", + ) + batch_rationale.add_argument( + "--selectivity-threshold", type=float, default=0.5, + help="Minimum selectivity score (default: 0.5).", + ) + batch_rationale.add_argument( + "--ensemble-weight", type=float, default=0.40, + help="Ensemble weight (default: 0.40).", + ) + batch_rationale.add_argument( + "--uncertainty-weight", type=float, default=0.30, + help="Uncertainty weight (default: 0.30).", + ) + batch_rationale.add_argument( + "--diversity-weight", type=float, default=0.30, + help="Diversity weight (default: 0.30).", + ) + batch_rationale.add_argument( + "--min-uncertainty-probes", type=int, default=1, + help="Minimum uncertainty probes in selection (default: 1).", + ) + batch_rationale.add_argument( + "--rng-seed", type=int, default=42, + help="RNG seed for reproducibility (default: 42).", + ) + batch_rationale.add_argument( + "--out-json", + default=None, + help="Optional JSON output path.", + ) + batch_rationale.add_argument( + "--out-md", + default=None, + help="Optional Markdown output path.", + ) + # ── Batch-2 selector ────────────────────────────────────────────── select_batch = sub.add_parser( "select-batch", @@ -1284,6 +1345,9 @@ def main(argv: list[str] | None = None) -> int: if args.command == "calibration-audit": return _run_calibration_audit(args) + if args.command == "batch-rationale": + return _run_batch_rationale(args) + if args.command == "select-batch": return _run_select_batch(args) diff --git a/tests/active_learning/test_batch_rationale.py b/tests/active_learning/test_batch_rationale.py new file mode 100644 index 00000000..c210cc88 --- /dev/null +++ b/tests/active_learning/test_batch_rationale.py @@ -0,0 +1,350 @@ +"""Tests for the batch-2 selection rationale report. + +All data is synthetic. Tests verify code-path integrity, structure, and +honest role classification mechanics — not biological selection quality. +""" + +from __future__ import annotations + +import json +import subprocess +import sys +from pathlib import Path + +import pytest + + +# ── Helpers ─────────────────────────────────────────────────────────── + + +def _check_report_shape(report: dict) -> None: + """Assert all required top-level fields exist.""" + required = [ + "version", "pool_size", "n_requested", "n_selected", + "n_after_safety_gate", "n_gated_out", + "ensemble_weight", "uncertainty_weight", "diversity_weight", + "candidates", "role_summary", "role_descriptions", + "selected_ids", "probes_in_top_n", "notes", + ] + for field in required: + assert field in report, f"Missing field: {field}" + + +def _check_candidate_shape(candidate: dict) -> None: + """Assert all required per-candidate fields exist.""" + required = [ + "candidate_id", "role", "ensemble_score", "uncertainty_score", + "diversity_score", "ensemble_contribution", "uncertainty_contribution", + "diversity_contribution", "combined_score", + "passed_safety_gate", "safety_gate_reason", "explanation", + ] + for field in required: + assert field in candidate, f"Missing candidate field: {field}" + + +# ── Tests ───────────────────────────────────────────────────────────── + + +class TestBatchRationaleReport: + """Tests for the batch rationale report generation.""" + + VALID_ROLES = {"exploit", "explore", "diversity", "combined"} + + def test_report_has_all_required_fields(self): + """Report has all required top-level fields.""" + from openamp_foundry.active_learning.batch_rationale import ( + build_batch_rationale_report, + ) + + report = build_batch_rationale_report( + n_total=30, n_active=5, batch_size=5, + ) + _check_report_shape(report.to_dict()) + + def test_candidates_are_selected(self): + """At least one candidate is selected with a small pool.""" + from openamp_foundry.active_learning.batch_rationale import ( + build_batch_rationale_report, + ) + + report = build_batch_rationale_report( + n_total=30, n_active=5, batch_size=10, + ) + assert report.n_selected > 0, "No candidates selected" + assert len(report.candidates) == report.n_selected + + def test_per_candidate_required_fields(self): + """Each candidate has all required fields.""" + from openamp_foundry.active_learning.batch_rationale import ( + build_batch_rationale_report, + ) + + report = build_batch_rationale_report( + n_total=30, n_active=5, batch_size=10, + ) + for c in report.to_dict()["candidates"]: + _check_candidate_shape(c) + + def test_role_is_valid(self): + """Each candidate has a valid role.""" + from openamp_foundry.active_learning.batch_rationale import ( + build_batch_rationale_report, + ) + + report = build_batch_rationale_report( + n_total=30, n_active=5, batch_size=10, + ) + for c in report.candidates: + assert c.role in self.VALID_ROLES, ( + f"Invalid role: {c.role}" + ) + + def test_scores_in_valid_range(self): + """Scores are between 0 and 1.""" + from openamp_foundry.active_learning.batch_rationale import ( + build_batch_rationale_report, + ) + + report = build_batch_rationale_report( + n_total=30, n_active=5, batch_size=10, + ) + for c in report.candidates: + assert 0.0 <= c.ensemble_score <= 1.0 + assert 0.0 <= c.uncertainty_score <= 1.0 + assert 0.0 <= c.diversity_score <= 1.0 + assert 0.0 <= c.combined_score <= 1.0 + + def test_role_summary_counts_match(self): + """Role summary counts match the number of candidates per role.""" + from openamp_foundry.active_learning.batch_rationale import ( + build_batch_rationale_report, + ) + + report = build_batch_rationale_report( + n_total=30, n_active=5, batch_size=10, + ) + actual_counts: dict[str, int] = {} + for c in report.candidates: + actual_counts[c.role] = actual_counts.get(c.role, 0) + 1 + assert actual_counts == report.role_summary, ( + f"Role summary mismatch: {actual_counts} != {report.role_summary}" + ) + + def test_selected_ids_match_candidates(self): + """Selected IDs list matches candidate IDs.""" + from openamp_foundry.active_learning.batch_rationale import ( + build_batch_rationale_report, + ) + + report = build_batch_rationale_report( + n_total=30, n_active=5, batch_size=10, + ) + candidate_ids = {c.candidate_id for c in report.candidates} + assert set(report.selected_ids) == candidate_ids + + def test_probes_in_top_n_non_negative(self): + """Probes count is non-negative.""" + from openamp_foundry.active_learning.batch_rationale import ( + build_batch_rationale_report, + ) + + report = build_batch_rationale_report( + n_total=30, n_active=5, batch_size=10, + ) + assert report.probes_in_top_n >= 0 + + def test_notes_are_present(self): + """Report has human-readable notes.""" + from openamp_foundry.active_learning.batch_rationale import ( + build_batch_rationale_report, + ) + + report = build_batch_rationale_report( + n_total=30, n_active=5, batch_size=10, + ) + assert len(report.notes) > 0 + assert any("Pool" in n for n in report.notes) + + def test_weight_config_matches_input(self): + """Weights in report match the input weights.""" + from openamp_foundry.active_learning.batch_rationale import ( + build_batch_rationale_report, + ) + + report = build_batch_rationale_report( + n_total=30, n_active=5, batch_size=10, + ensemble_weight=0.50, + uncertainty_weight=0.25, + diversity_weight=0.25, + ) + assert report.ensemble_weight == 0.50 + assert report.uncertainty_weight == 0.25 + assert report.diversity_weight == 0.25 + + def test_write_json(self, tmp_path): + """Writing to JSON produces valid file with expected fields.""" + from openamp_foundry.active_learning.batch_rationale import ( + build_batch_rationale_report, + write_rationale_json, + ) + + report = build_batch_rationale_report( + n_total=30, n_active=5, batch_size=10, + ) + out = tmp_path / "report.json" + write_rationale_json(report, out) + assert out.exists() + data = json.loads(out.read_text()) + _check_report_shape(data) + + def test_write_markdown(self, tmp_path): + """Writing to Markdown produces expected sections.""" + from openamp_foundry.active_learning.batch_rationale import ( + build_batch_rationale_report, + write_rationale_markdown, + ) + + report = build_batch_rationale_report( + n_total=30, n_active=5, batch_size=10, + ) + out = tmp_path / "report.md" + write_rationale_markdown(report, out) + assert out.exists() + content = out.read_text() + assert "# Batch-2 Selection Rationale Report" in content + assert "## Overview" in content + assert "## Selection Weights" in content + assert "## Selection Breakdown by Role" in content + assert "## Per-Candidate Rationale" in content + assert "## Contribution Detail" in content + assert "## Caveats" in content + + def test_cli_exits_zero(self): + """CLI batch-rationale exits 0 and produces valid output.""" + result = subprocess.run( + [ + sys.executable, "-m", "openamp_foundry.cli", + "batch-rationale", + "--n-total", "20", + "--n-active", "4", + "--batch-size", "5", + "--rng-seed", "42", + ], + capture_output=True, text=True, timeout=60, + ) + assert result.returncode == 0, f"CLI failed: {result.stderr[:500]}" + data = json.loads(result.stdout) + assert "version" in data + assert "candidates" in data + assert "role_summary" in data + assert "selected_ids" in data + + def test_cli_writes_json_and_md(self, tmp_path): + """CLI with --out-json and --out-md writes both files.""" + out_json = tmp_path / "report.json" + out_md = tmp_path / "report.md" + result = subprocess.run( + [ + sys.executable, "-m", "openamp_foundry.cli", + "batch-rationale", + "--n-total", "20", + "--n-active", "4", + "--batch-size", "5", + "--out-json", str(out_json), + "--out-md", str(out_md), + ], + capture_output=True, text=True, timeout=60, + ) + assert result.returncode == 0, f"CLI failed: {result.stderr[:500]}" + assert out_json.exists() + assert out_md.exists() + data = json.loads(out_json.read_text()) + _check_report_shape(data) + md_content = out_md.read_text() + assert "# Batch-2 Selection Rationale Report" in md_content + + def test_schema_conformance(self, tmp_path): + """JSON output validates against JSON Schema.""" + import jsonschema + + schema_path = ( + Path(__file__).resolve().parents[2] + / "schemas" + / "batch_rationale_report.schema.json" + ) + schema = json.loads(schema_path.read_text()) + + from openamp_foundry.active_learning.batch_rationale import ( + build_batch_rationale_report, + write_rationale_json, + ) + + report = build_batch_rationale_report( + n_total=30, n_active=5, batch_size=10, + ) + out = tmp_path / "report.json" + write_rationale_json(report, out) + data = json.loads(out.read_text()) + + jsonschema.validate(instance=data, schema=schema) + + def test_high_exploitation_weight_produces_exploit_roles(self): + """High ensemble weight biases roles toward exploit.""" + from openamp_foundry.active_learning.batch_rationale import ( + build_batch_rationale_report, + ) + + report = build_batch_rationale_report( + n_total=30, n_active=5, batch_size=10, + ensemble_weight=0.90, + uncertainty_weight=0.05, + diversity_weight=0.05, + ) + exploit_count = report.role_summary.get("exploit", 0) + explore_count = report.role_summary.get("explore", 0) + assert exploit_count > explore_count, ( + f"Expected more exploit ({exploit_count}) than explore ({explore_count}) " + f"with high ensemble weight" + ) + + def test_all_selected_candidates_have_explanation(self): + """Every selected candidate has a non-empty explanation.""" + from openamp_foundry.active_learning.batch_rationale import ( + build_batch_rationale_report, + ) + + report = build_batch_rationale_report( + n_total=30, n_active=5, batch_size=10, + ) + for c in report.candidates: + assert len(c.explanation) > 0, ( + f"Empty explanation for {c.candidate_id}" + ) + assert c.role in c.explanation.lower() or "balanced" in c.explanation.lower() or "exploration" in c.explanation.lower() + + def test_role_descriptions_are_present_for_all_roles(self): + """Descriptions exist for all expected roles.""" + from openamp_foundry.active_learning.batch_rationale import ( + build_batch_rationale_report, + ) + + report = build_batch_rationale_report( + n_total=30, n_active=5, batch_size=10, + ) + for role in self.VALID_ROLES: + assert role in report.role_descriptions, ( + f"Missing description for role: {role}" + ) + assert len(report.role_descriptions[role]) > 0 + + def test_empty_role_not_in_summary(self): + """Roles with zero candidates should not appear in role_summary.""" + from openamp_foundry.active_learning.batch_rationale import ( + build_batch_rationale_report, + ) + + report = build_batch_rationale_report( + n_total=30, n_active=5, batch_size=10, + ) + for role, count in report.role_summary.items(): + assert count > 0, f"Role {role} has zero count but is in summary" diff --git a/tests/test_test_count_regression.py b/tests/test_test_count_regression.py index ece981fb..e09b992f 100644 --- a/tests/test_test_count_regression.py +++ b/tests/test_test_count_regression.py @@ -2,9 +2,9 @@ import subprocess import sys -# Baseline: 2955 tests as of Loop 82 (2026-07-09) -- Phase G G4 (active-learning strategy comparison report) +# Baseline: 2974 tests as of Loop 83 (2026-07-09) -- Phase G G5 (batch-2 selection rationale report) # Raise threshold if legitimate tests are removed; lower threshold is okay. -BASELINE = 2955 +BASELINE = 2974 TOLERANCE = 0.05 # allow 5% fluctuation