Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 73 additions & 0 deletions docs/evidence/BATCH_SELECTION_PROPOSAL_GUIDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# Batch Selection Proposal Guide

## Purpose

A Batch Selection Proposal (BSP) documents the next-batch candidate selection
plan immediately after a Calibration Readiness Gate (CRG) passes.

The BSP enforces two things:

1. A batch can only be proposed when `gate_passed=True` — the pipeline
cannot nominate candidates when calibration quality is insufficient.
2. The selection strategy (exploitation vs exploration fraction) is documented
before selection, not after, preventing post-hoc rationalization.

## Key constraint

`gate_passed` in a BSP must always be `True`. A failed calibration gate
cannot generate a valid proposal. This links Phase O (Calibration Quality
Assurance) directly to Phase P (Batch Selection).

## Schema fields

| Field | Type | Description |
|---|---|---|
| `bsp_id` | str | Must start with `BSP-` |
| `pipeline_version` | str | Pipeline version at selection time |
| `gate_id` | str | Must start with `CRG-` — the passed gate |
| `gate_passed` | bool | Must be `True` |
| `candidate_ids` | List[str] | At least 1 candidate |
| `selection_strategy` | str | One of 5 valid strategies |
| `exploitation_fraction` | float | [0.0, 1.0] |
| `exploration_fraction` | float | [0.0, 1.0] |
| `max_brier_score_allowed` | float | [0.0, 1.0] |
| `proposal_notes` | str | Max 400 chars |
| `reviewer` | str | Who approved the proposal |
| `dry_lab_only` | bool | Default True |

## Selection strategies

| Strategy | Description |
|---|---|
| `exploitation` | Prefer high-scoring candidates with low uncertainty |
| `exploration` | Prefer candidates in under-sampled regions |
| `diversity` | Prefer structurally distinct candidates |
| `uncertainty_sampling` | Prefer candidates with highest prediction uncertainty |
| `hybrid` | Combination; exploitation + exploration fractions document the mix |

## Fraction constraint

`exploitation_fraction + exploration_fraction` must sum to 1.0 (within 0.001
tolerance). These fractions document the intended mix; the actual selection
algorithm may use any mechanism so long as the mix is documented here.

## Honest boundaries

- A BSP is a plan document. It does not guarantee the selected candidates
will show activity — it documents the rationale for selection.
- The BSP does not validate that `gate_id` references a real CRG entry in the
database; referential integrity requires a separate audit step.
- `dry_lab_only=True` by default. A BSP that results in wet-lab synthesis
should carry `dry_lab_only=False` and must pass safety review before
candidates are shared with a lab.
- Calibration gate passage at the time of the BSP does not guarantee the
pipeline remains calibrated after batch execution.

## Warnings

The validator emits warnings (not errors) for:

- Pure exploitation (`exploitation_fraction ≥ 0.99`) — exploration helps
avoid local optima.
- Single-candidate batch — diversity and calibration coverage may be reduced.
- `max_brier_score_allowed ≥ 0.25` — the gate passed with marginal calibration.
3 changes: 2 additions & 1 deletion docs/evidence/METRICS_CURRENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
> with this file, this file wins. Updated whenever benchmark/benchmark config changes.
>
> **Last updated:** 2026-07-10 (Phase O O5 — calibration readiness gate — v0.10.3)
> **Last updated:** 2026-07-10 (Phase P P1 — batch selection proposal — v0.10.4)
> **New in v0.10.4 (Phase P P1):** BatchSelectionProposalEntry schema (12 fields: bsp_id, pipeline_version, gate_id, gate_passed, candidate_ids, selection_strategy, exploitation_fraction, exploration_fraction, max_brier_score_allowed, proposal_notes, reviewer, dry_lab_only). gate_passed must be True — a batch can only be proposed when the calibration readiness gate passes. 9 validation rules (BSP- prefix, CRG- prefix, gate_passed=True enforced, candidate_ids >=1, strategy in 5 valid values, fraction bounds, fraction sum=1.0 to 0.001 tolerance, Brier bounds [0.0,1.0], notes <=400 chars). 3 warnings (pure exploitation >=0.99, single candidate, marginal Brier >=0.25). Added validate_batch_selection_proposal() and validate_batch_selection_proposal_dict(). Added 68 tests. CLI: openamp-foundry batch-selection-proposal-check with --entry-json and --format text|json. Closes the calibration-to-selection loop from Phase O.
> **New in v0.10.3 (Phase O O5):** CalibrationReadinessEntry schema (11 fields, 8 validation rules, 2 warnings). Added validate_calibration_readiness() and validate_calibration_readiness_dict(). Added 68 tests. CLI: openamp-foundry calibration-readiness-check. Phase O (Calibration Quality Assurance) COMPLETE.
> **New in v0.10.2 (Phase O O4):** CrossBatchAggregatorEntry schema (12 fields, 9 validation rules, 3 warnings). Added validate_cross_batch_aggregator() and validate_cross_batch_aggregator_dict(). Added 68 tests. CLI: openamp-foundry cross-batch-aggregator-check.
> **New in v0.10.0 (Phase O O2):** PredictionDriftEntry schema (15 fields: monitor_id, pipeline_version, reference_batch_id, evaluation_batch_id, reference_mean_score, reference_std_score, evaluation_mean_score, evaluation_std_score, mean_shift_magnitude, population_size_reference, population_size_evaluation, drift_flag, drift_notes, reviewer, dry_lab_only). 8 validation rules (DRM- prefix, batch id must differ, scores 0.0-1.0, std >= 0.0, mean shift tolerance ±0.001, population >=1, notes <=400 chars, drift_flag=True requires non-empty notes). 4 warning conditions (unreported large shift >=0.1, small population <10, variance explosion >2x). 5 constants (DRIFT_NOTES_MAX_LENGTH=400, MEAN_SHIFT_TOLERANCE=0.001, SIGNIFICANT_DRIFT_THRESHOLD=0.1, MIN_POPULATION_FOR_RELIABLE_DRIFT=10, VARIANCE_EXPLOSION_RATIO=2.0). PredictionDriftEntry (15 fields), PredictionDriftResult (8 fields: monitor_id, pipeline_version, mean_shift_magnitude, drift_flag, passed, errors, warnings, dry_lab_only). validate_prediction_drift() (12+ error checks, 4 warning conditions). validate_prediction_drift_dict() (14 required fields guard). CLI (openamp-foundry prediction-drift-check) with --entry-json, --format text|json. make prediction-drift-check target. 68 tests. Phase O O2: prediction drift monitor is the earliest warning signal for pipeline score distribution shifts before wet-lab validation data arrives.
Expand Down
10 changes: 10 additions & 0 deletions docs/research/NEXT_100_PR_MAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -249,6 +249,16 @@ Track how well predictions match experimental outcomes over time. Detect drift,
| O4 | Add cross-batch performance aggregator schema (complete — Loop 140). | Aggregates results across batches for trend analysis. | B/D |
| [x] O5 | Add calibration readiness gate schema. | Validates calibration quality is sufficient before releasing the next batch. | B/D |

## Phase P — Calibration-to-Selection Bridge

Close the loop from Phase O calibration quality assurance back to candidate
selection. A batch can only be proposed when the calibration gate passes,
preventing the pipeline from selecting candidates under poor calibration.

| PR | Task | Why it matters | Review class |
|---:|---|---|---|
| P1 | Add batch selection proposal schema (complete). | Enforces calibration gate must pass before proposing a next batch; documents exploitation/exploration strategy. | B/D |

## Prioritization rule

When choosing between tasks, prefer the one that:
Expand Down
64 changes: 64 additions & 0 deletions src/openamp_foundry/cli/commands/reports.py
Original file line number Diff line number Diff line change
Expand Up @@ -3543,3 +3543,67 @@ def _run_cross_batch_aggregator_check(args) -> int:
for warn in result.warnings:
print(f" WARN: {warn}")
return 0 if result.passed else 1


def _run_batch_selection_proposal_check(args: argparse.Namespace) -> int:
from openamp_foundry.evidence.batch_selection_proposal import (
validate_batch_selection_proposal_dict,
)

if args.entry_json:
import json

try:
d = json.loads(args.entry_json)
except json.JSONDecodeError as exc:
print(f"Invalid JSON: {exc}", file=__import__("sys").stderr)
return 2
result = validate_batch_selection_proposal_dict(d)
else:
result = validate_batch_selection_proposal_dict(
{
"bsp_id": "BSP-DEMO",
"pipeline_version": "v1.0.0",
"gate_id": "CRG-DEMO",
"gate_passed": True,
"candidate_ids": ["CAND-001", "CAND-002"],
"selection_strategy": "hybrid",
"exploitation_fraction": 0.6,
"exploration_fraction": 0.4,
"max_brier_score_allowed": 0.20,
"proposal_notes": "Demo proposal.",
"reviewer": "demo@example.com",
"dry_lab_only": True,
}
)

if args.format == "json":
import json

print(
json.dumps(
{
"bsp_id": result.bsp_id,
"gate_id": result.gate_id,
"gate_passed": result.gate_passed,
"candidate_count": result.candidate_count,
"selection_strategy": result.selection_strategy,
"passed": result.passed,
"errors": result.errors,
"warnings": result.warnings,
},
indent=2,
)
)
else:
status = "PASS" if result.passed else "FAIL"
print(f"[{status}] Batch Selection Proposal: {result.bsp_id}")
print(f" Gate: {result.gate_id} (passed={result.gate_passed})")
print(f" Strategy: {result.selection_strategy}")
print(f" Candidates: {result.candidate_count}")
for e in result.errors:
print(f" ERROR: {e}")
for w in result.warnings:
print(f" WARN: {w}")

return 0 if result.passed else 1
12 changes: 12 additions & 0 deletions src/openamp_foundry/cli/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
_run_calibration_improvement_check,
_run_cross_batch_aggregator_check,
_run_calibration_readiness_check,
_run_batch_selection_proposal_check,
)
from openamp_foundry.cli.commands.gates import _run_gate_check, _run_release_gate_check
from openamp_foundry.cli.commands.reports import _run_contribution_check
Expand Down Expand Up @@ -2356,6 +2357,14 @@ def build_parser() -> argparse.ArgumentParser:
p_crg.add_argument("--format", choices=["text", "json"], default="text")
p_crg.set_defaults(func=_run_calibration_readiness_check)

p_bsp = sub.add_parser(
"batch-selection-proposal-check",
help="Validate batch selection proposal entry",
)
p_bsp.add_argument("--entry-json", default=None)
p_bsp.add_argument("--format", choices=["text", "json"], default="text")
p_bsp.set_defaults(func=_run_batch_selection_proposal_check)

# ── Selection rationale check (Phase K K1) ───────────────────────
src2 = sub.add_parser(
"selection-rationale-check",
Expand Down Expand Up @@ -2731,6 +2740,9 @@ def main(argv: list[str] | None = None) -> int:
if args.command == "calibration-readiness-check":
return _run_calibration_readiness_check(args)

if args.command == "batch-selection-proposal-check":
return _run_batch_selection_proposal_check(args)

parser.error("unknown command")
return 2

Expand Down
Loading
Loading