Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 83 additions & 0 deletions docs/evidence/BATCH_EXPERIMENT_PRIORITY_RANKER_GUIDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# Batch Experiment Priority Ranker Guide (BPR-)

## Purpose

A `BatchExperimentPriorityRanker` (BPR-) records the priority ordering of candidates
for the next synthesis wave. While a `CandidateSelectionRationale` (CSR-) documents
why candidates were selected, a BPR- records which should be synthesized first given
resource constraints and scientific priorities.

BPR- makes synthesis wave ordering explicit, auditable, and defensible to external
reviewers and lab partners.

## Fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `bpr_id` | str | Yes | Unique ID, must start with `BPR-` |
| `pipeline_version` | str | Yes | Version of the pipeline that produced this ranking |
| `batch_id` | str | Yes | Batch being prioritized |
| `csr_id` | str | Yes | Candidate selection rationale (must start with `CSR-`) |
| `ranking_date` | str | Yes | ISO date (YYYY-MM-DD) of ranking decision |
| `priority_method` | str | Yes | Controlled vocabulary: how candidates were ranked |
| `top_priority_candidates` | List[str] | Yes | Ordered list of candidate IDs (at least 1) |
| `priority_rationale` | str | Yes | Why this ordering was chosen (≤400 chars) |
| `synthesis_wave` | int | Yes | Which synthesis wave this is (≥1) |
| `resource_constraint_considered` | bool | Yes | Whether resource constraints were factored in |
| `notes` | str | No | Additional context (≤300 chars) |

## Valid Vocabularies

### `priority_method`

| Value | Description |
|-------|-------------|
| `predicted_activity` | Ranked by predicted antimicrobial activity |
| `structural_diversity` | Ranked to maximize structural diversity in the wave |
| `cost_efficiency` | Ranked by synthesis cost per predicted benefit |
| `risk_stratified` | Ranked to hedge risk across confidence levels |
| `novelty_first` | Most novel candidates synthesized first |
| `expert_ranked` | Final ordering decided by domain expert |

## Validation Rules

1. `bpr_id` must start with `BPR-`
2. `pipeline_version` must be non-empty
3. `batch_id` must be non-empty
4. `csr_id` must start with `CSR-`
5. `ranking_date` must match `YYYY-MM-DD`
6. `priority_method` must be in valid set (6 values)
7. `top_priority_candidates` must contain at least one candidate
8. `priority_rationale` must be non-empty and ≤400 chars
9. `synthesis_wave` must be ≥1
10. `notes` must be ≤300 chars

## Warnings

| Condition | Warning |
|-----------|---------|
| `resource_constraint_considered=False` | Synthesis priority without resource constraints may be suboptimal |
| `priority_method='expert_ranked'` and `notes=''` | Expert ranking decisions should be documented |
| `synthesis_wave > 5` | Unusually high synthesis wave; verify intent |

## Boundaries

- **BPR- records priority ordering, not selection.** Use CSR- for selection rationale; use BPR- for ordering within a selected batch.
- **BPR- is computational input, not a wet-lab instruction.** The lab team decides actual synthesis scheduling; BPR- provides the recommended order.
- **`synthesis_wave` reflects the experiment sequence.** Wave 1 is the first synthesis; wave N is the N-th iteration.

## CLI

```bash
openamp-foundry batch-experiment-priority-ranker-check '{"bpr_id": "BPR-001", ...}'
```

Returns `VALID` or `INVALID` with any errors and warnings.

## Relationship to Other Schemas

```
CSR- ──→ BPR- (selection rationale is the basis for priority ranking)
BPR- ──→ PPC- (synthesis ordering is a pilot package component)
BPR- ──→ PRE- (pre-registration locks priority order before experiments)
```
3 changes: 2 additions & 1 deletion docs/evidence/METRICS_CURRENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
> with this file, this file wins. Updated whenever benchmark/benchmark config changes.
>
> **Last updated:** 2026-07-10 (Phase K K3 — pilot package completeness report — v0.10.15)
> **Last updated:** 2026-07-10 (Phase K K2 — batch experiment priority ranker — v0.10.17)
> **New in v0.10.17 (Phase K K2 — Batch Experiment Priority Ranker):** BatchExperimentPriorityRanker — 11 fields, synthesis wave priority ordering. 10 validation rules — BPR- prefix, pipeline_version+batch_id non-empty, CSR- prefix, ISO date, priority_method controlled vocab (6 values), top_priority_candidates non-empty, priority_rationale non-empty and ≤400 chars, synthesis_wave ≥1, notes ≤300 chars. 3 warnings — resource_constraint_considered=False, expert_ranked without notes, synthesis_wave >5. 63 tests across 9 test classes; BASELINE 5666→5729. CLI: openamp-foundry batch-experiment-priority-ranker-check. BPR- records which candidates to synthesize first; CSR- records why candidates were selected.
> **New in v0.10.16 (Phase K K1 — Candidate Selection Rationale):** CandidateSelectionRationale — 12 fields, auditable candidate selection decisions. 12 validation rules: CSR- prefix, pipeline_version+batch_id non-empty, BSP- prefix, ISO date, strategy controlled vocab (4 values), candidate_count ≥1, count/ids consistency, ranking_method controlled vocab (6 values), calibration_gate_passed=True enforced, rationale non-empty and ≤400 chars, notes ≤300 chars. 3 warnings: large batch (>20), random_balanced without notes, expert_review without notes. 63 tests; BASELINE 5603→5666. CLI: openamp-foundry candidate-selection-rationale-check.
> **New in v0.10.15 (Phase K K3):** PilotPackageCompletenessReport — PPC- completeness gate: confirms CCS-+BSP-+PSC-+PRE-+BCM- all present and ESC- cleared before external sharing (12 fields, 12 validation rules, 3 warnings, 63 tests, BASELINE 5540→5603)
> **New in v0.10.14 (Phase F F1):** NegativeResultEntry — atomic NRR- failure record with stage/reason/confidence vocabulary, foundation for NAS- and FCR- chain (12 fields, 10 validation rules, 4 warnings, 63 tests, BASELINE 5477→5540)
Expand Down
2 changes: 1 addition & 1 deletion docs/research/NEXT_100_PR_MAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ Build the evidence and accountability artifacts needed for credible external val
| PR | Task | Why it matters | Review class |
|---:|---|---|---|
| K1 | Add candidate selection rationale schema (complete). | CSR- records why specific candidates were selected: strategy (4 values), ranking method (6 values), calibration gate enforced, candidate_count/ids consistency check. | B |
| K2 | Add batch experiment priority ranker. | Rank candidates for next synthesis wave. | B |
| K2 | Add batch experiment priority ranker (complete). | BPR- records synthesis wave priority ordering: 6 priority methods, synthesis_wave ≥1, resource_constraint_considered enforced, CSR- reference. | B |
| K3 | Add pilot package completeness checker (complete). | PPC- completeness gate: confirms CCS-+BSP-+PSC-+PRE-+BCM- all present and ESC- cleared before external sharing. | B |
| K4 | Add post-experiment calibration intake schema. | Structured result-to-prediction comparison. | B/D |
| K5 | Add uncertainty quantification report schema (complete — Loop 123). | Honest prediction intervals for reviewers. | B |
Expand Down
22 changes: 22 additions & 0 deletions src/openamp_foundry/cli/commands/reports.py
Original file line number Diff line number Diff line change
Expand Up @@ -2926,3 +2926,25 @@ def _run_candidate_selection_rationale_check(args):
for w in result.warnings:
print(f"WARNING: {w}")
sys.exit(1)


def _run_batch_experiment_priority_ranker_check(args):
import json, sys
from openamp_foundry.evidence.batch_experiment_priority_ranker import validate_dict
try:
data = json.loads(args.json_input)
except json.JSONDecodeError as exc:
print(f"INVALID: JSON parse error: {exc}", file=sys.stderr)
sys.exit(1)
result = validate_dict(data)
if result.valid:
print("VALID")
for w in result.warnings:
print(f"WARNING: {w}")
else:
print("INVALID")
for e in result.errors:
print(f"ERROR: {e}")
for w in result.warnings:
print(f"WARNING: {w}")
sys.exit(1)
11 changes: 11 additions & 0 deletions src/openamp_foundry/cli/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@
import json
from pathlib import Path

from openamp_foundry.evidence.batch_experiment_priority_ranker import validate_dict as validate_batch_experiment_priority_ranker_dict
from openamp_foundry.evidence.candidate_selection_rationale import validate_dict as validate_candidate_selection_rationale_dict
from openamp_foundry.evidence.negative_result_entry import validate_dict as validate_negative_result_entry_dict
from openamp_foundry.evidence.pilot_package_completeness_report import validate_dict as validate_pilot_package_completeness_dict
Expand Down Expand Up @@ -2500,6 +2501,12 @@ def build_parser() -> argparse.ArgumentParser:
)
csr_parser.add_argument("json_input", help="JSON string of the CSR- record")

bpr_parser = sub.add_parser(
"batch-experiment-priority-ranker-check",
help="Validate a BatchExperimentPriorityRanker (BPR-) record",
)
bpr_parser.add_argument("json_input", help="JSON string of the BPR- record")

# ── Selection rationale check (Phase K K1) ───────────────────────
src2 = sub.add_parser(
"selection-rationale-check",
Expand Down Expand Up @@ -2926,6 +2933,10 @@ def main(argv: list[str] | None = None) -> int:
from openamp_foundry.cli.commands.reports import _run_candidate_selection_rationale_check
_run_candidate_selection_rationale_check(args)

elif args.command == "batch-experiment-priority-ranker-check":
from openamp_foundry.cli.commands.reports import _run_batch_experiment_priority_ranker_check
_run_batch_experiment_priority_ranker_check(args)

else:
parser.error("unknown command")
return 2
Expand Down
149 changes: 149 additions & 0 deletions src/openamp_foundry/evidence/batch_experiment_priority_ranker.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
"""Batch experiment priority ranker schema — Phase K K2.

Records the priority ordering of candidates for the next synthesis wave.
Documents priority method, resource constraints, and synthesis wave number,
making synthesis ordering auditable and defensible to external reviewers.
"""

from __future__ import annotations

import re
from dataclasses import dataclass, field
from typing import List

BPR_PREFIX = "BPR-"
CSR_PREFIX = "CSR-"

VALID_PRIORITY_METHODS = frozenset({
"predicted_activity",
"structural_diversity",
"cost_efficiency",
"risk_stratified",
"novelty_first",
"expert_ranked",
})

PRIORITY_RATIONALE_MAX_LENGTH = 400
NOTES_MAX_LENGTH = 300
MAX_SYNTHESIS_WAVE_WARNING = 5
_ISO_DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$")


@dataclass
class BatchExperimentPriorityRanker:
bpr_id: str
pipeline_version: str
batch_id: str
csr_id: str
ranking_date: str
priority_method: str
top_priority_candidates: List[str]
priority_rationale: str
synthesis_wave: int
resource_constraint_considered: bool
notes: str = ""


@dataclass
class BatchExperimentPriorityRankerResult:
valid: bool
errors: List[str] = field(default_factory=list)
warnings: List[str] = field(default_factory=list)


def validate(entry: BatchExperimentPriorityRanker) -> BatchExperimentPriorityRankerResult:
errors: List[str] = []
warnings: List[str] = []

if not entry.bpr_id.startswith(BPR_PREFIX):
errors.append(f"bpr_id must start with '{BPR_PREFIX}', got: {entry.bpr_id!r}")

if not entry.pipeline_version.strip():
errors.append("pipeline_version must be a non-empty string")

if not entry.batch_id.strip():
errors.append("batch_id must be a non-empty string")

if not entry.csr_id.startswith(CSR_PREFIX):
errors.append(f"csr_id must start with '{CSR_PREFIX}', got: {entry.csr_id!r}")

if not _ISO_DATE_RE.match(entry.ranking_date):
errors.append(
f"ranking_date must be ISO format YYYY-MM-DD, got: {entry.ranking_date!r}"
)

if entry.priority_method not in VALID_PRIORITY_METHODS:
errors.append(
f"priority_method {entry.priority_method!r} not in valid set: "
f"{sorted(VALID_PRIORITY_METHODS)}"
)

if len(entry.top_priority_candidates) == 0:
errors.append(
"top_priority_candidates must contain at least one candidate"
)

if not entry.priority_rationale.strip():
errors.append("priority_rationale must be a non-empty string")
elif len(entry.priority_rationale) > PRIORITY_RATIONALE_MAX_LENGTH:
errors.append(
f"priority_rationale exceeds {PRIORITY_RATIONALE_MAX_LENGTH} chars "
f"(got {len(entry.priority_rationale)})"
)

if entry.synthesis_wave < 1:
errors.append(
f"synthesis_wave must be at least 1, got: {entry.synthesis_wave}"
)

if len(entry.notes) > NOTES_MAX_LENGTH:
errors.append(
f"notes exceeds {NOTES_MAX_LENGTH} chars (got {len(entry.notes)})"
)

# Warnings
if not entry.resource_constraint_considered:
warnings.append(
"resource_constraint_considered=False; "
"synthesis priority without resource constraints may be suboptimal"
)

if entry.priority_method == "expert_ranked" and not entry.notes.strip():
warnings.append(
"priority_method='expert_ranked' without notes; "
"expert ranking decisions should be documented"
)

if entry.synthesis_wave > MAX_SYNTHESIS_WAVE_WARNING:
warnings.append(
f"synthesis_wave={entry.synthesis_wave} is unusually high "
f"(>{MAX_SYNTHESIS_WAVE_WARNING}); verify this is correct"
)

return BatchExperimentPriorityRankerResult(
valid=len(errors) == 0, errors=errors, warnings=warnings
)


def validate_dict(data: dict) -> BatchExperimentPriorityRankerResult:
try:
entry = BatchExperimentPriorityRanker(
bpr_id=data.get("bpr_id", ""),
pipeline_version=data.get("pipeline_version", ""),
batch_id=data.get("batch_id", ""),
csr_id=data.get("csr_id", ""),
ranking_date=data.get("ranking_date", ""),
priority_method=data.get("priority_method", ""),
top_priority_candidates=list(data.get("top_priority_candidates", [])),
priority_rationale=data.get("priority_rationale", ""),
synthesis_wave=int(data.get("synthesis_wave", 0)),
resource_constraint_considered=bool(
data.get("resource_constraint_considered", False)
),
notes=data.get("notes", ""),
)
except Exception as exc:
return BatchExperimentPriorityRankerResult(
valid=False, errors=[f"Construction error: {exc}"]
)
return validate(entry)
Loading
Loading