Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 83 additions & 0 deletions docs/evidence/DOMAIN_REVIEW_OUTCOME_GUIDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# Domain Review Outcome Guide

## Purpose

A `DomainReviewOutcome` (DRO-) records the formal verdict of a domain expert after
reviewing a `PilotEvidencePackage`. It uses a controlled taxonomy of expert domains
and outcome verdicts, making review decisions structured, auditable, and comparable
across multiple reviewers. It links to the preceding `ReviewerQuestionnaire` (RVQ-).

## Chain in Phase E

```
ESC- (external sharing) → RVQ- (reviewer questionnaire) → DRO- (expert verdict)
```

## Fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `dro_id` | str | Yes | Unique ID, must start with `DRO-` |
| `pipeline_version` | str | Yes | Pipeline version being reviewed |
| `pep_id` | str | Yes | Which PEP was reviewed (must start with `PEP-`) |
| `rvq_id` | str | Yes | Preceding questionnaire (must start with `RVQ-`) |
| `reviewer_token` | str | Yes | Anonymized reviewer ID — no PII |
| `review_domain` | str | Yes | Expert domain (controlled vocabulary) |
| `review_date` | str | Yes | ISO YYYY-MM-DD date of verdict |
| `outcome_verdict` | str | Yes | Expert verdict (controlled vocabulary) |
| `outcome_confidence` | str | Yes | `high`, `medium`, or `low` |
| `outcome_rationale` | str | Yes | Rationale (max 400 chars) |
| `dry_lab_only` | bool | No | Default `True` |

## Expert domains

| Domain | When to use |
|--------|-------------|
| `antimicrobial_activity` | Reviewer assessing predicted AMP activity |
| `toxicology` | Reviewer assessing cytotoxicity/hemolysis claims |
| `structural_biology` | Reviewer assessing structural predictions |
| `clinical_microbiology` | Reviewer assessing clinical relevance |
| `computational_chemistry` | Reviewer assessing computational methods |
| `general_biomedical` | General biomedical domain expertise |

## Outcome verdicts

| Verdict | Meaning |
|---------|---------|
| `approve` | Package is sufficient for proceeding |
| `reject` | Package is not sufficient in current form |
| `conditional_approve` | Approve if specific conditions are met (document in rationale) |
| `request_revision` | Package needs specific revisions before re-review |
| `insufficient_data` | Cannot reach a verdict — more data required |

## Validation rules

1. `dro_id` must start with `DRO-`
2. `pep_id` must start with `PEP-`
3. `rvq_id` must start with `RVQ-`
4. `pipeline_version`, `reviewer_token` must not be empty
5. `review_domain` must be in the controlled vocabulary
6. `review_date` must be ISO format `YYYY-MM-DD`
7. `outcome_verdict` must be in the controlled vocabulary
8. `outcome_confidence` must be `high`, `medium`, or `low`
9. `outcome_rationale` at most 400 characters

## Warnings

- `conditional_approve` with no rationale — document the conditions
- `reject` with `low` confidence — may need additional reviewer
- `insufficient_data` with no rationale — document what is missing

## Honest boundaries

- A DRO- record reflects one expert's judgment; it does not authorize synthesis.
- `approve` does not imply biological validation — it is a dry-lab package assessment only.
- `reviewer_token` must never contain PII.

## CLI usage

```bash
openamp-foundry domain-review-outcome-check \
--entry-json '{"dro_id":"DRO-001","pipeline_version":"v0.10.13","pep_id":"PEP-001","rvq_id":"RVQ-001","reviewer_token":"REV-A","review_domain":"antimicrobial_activity","review_date":"2026-07-10","outcome_verdict":"approve","outcome_confidence":"high","outcome_rationale":"Evidence base is solid."}' \
--format text
```
3 changes: 2 additions & 1 deletion docs/evidence/METRICS_CURRENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
> with this file, this file wins. Updated whenever benchmark/benchmark config changes.
>
> **Last updated:** 2026-07-10 (Phase E E3 — reviewer questionnaire — v0.10.13)
> **Last updated:** 2026-07-10 (Phase E E9 — domain review outcome — v0.10.14)
> **New in v0.10.14 (Phase E E9):** DomainReviewOutcome — controlled taxonomy expert verdict on a PEP (11 fields, 9 validation rules, 3 warnings, 63 tests, BASELINE 5414→5477); closes ESC→RVQ→DRO review chain
> **New in v0.10.13 (Phase E E3):** ReviewerQuestionnaire — structured external review feedback with Likert clarity ratings, synthesis recommendation, and comments (13 fields, 10 validation rules, 4 warnings, 63 tests, BASELINE 5351→5414)
> **New in v0.10.12 (Phase F F4):** FailedCandidateBatchReport — batch-level failure summary linking RJR- and NAS- with failure rate consistency check (13 fields, 11 validation rules, 4 warnings, 63 tests, BASELINE 5288→5351)
> **New in v0.10.11 (Phase F F2):** NegativeResultArchiveSummary — validates batch of NRR- entries is complete before archiving (11 fields, 9 validation rules, 3 warnings, 63 tests, BASELINE 5225→5288)
Expand Down
2 changes: 1 addition & 1 deletion docs/research/NEXT_100_PR_MAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ Make qualified external review easier and safer.
| E6 | Add packet generator CLI. | Reduces manual packaging errors. | C/D |
| E7 | Add packet validator CLI. | Review readiness becomes testable. | C/D |
| E8 | Add release-summary generator that strips restricted fields. | Safer public summaries. | D |
| E9 | Add domain-review outcome taxonomy. | Rejection becomes structured. | B/C |
| E9 | Add domain review outcome schema (complete). | Structured expert verdict on a PEP with controlled taxonomy of domains and outcomes; closes ESC→RVQ→DRO review chain. | B/C |
| E10 | Add expert-review example with mock/toy candidates only. | Safe demonstration. | B/C |

## Phase F — Negative-result infrastructure
Expand Down
45 changes: 45 additions & 0 deletions src/openamp_foundry/cli/commands/reports.py
Original file line number Diff line number Diff line change
Expand Up @@ -1285,6 +1285,51 @@ def _run_validate_simulation_result(args: argparse.Namespace) -> int:
return 3 if batch_result["any_invalid"] else 0


def _run_domain_review_outcome_check(args):
from openamp_foundry.evidence.domain_review_outcome import (
validate_domain_review_outcome_dict,
)
import json
import sys

data = json.loads(args.entry_json)
result = validate_domain_review_outcome_dict(data)

if args.format == "json":
out = {
"dro_id": result.dro_id,
"pep_id": result.pep_id,
"rvq_id": result.rvq_id,
"review_domain": result.review_domain,
"outcome_verdict": result.outcome_verdict,
"outcome_confidence": result.outcome_confidence,
"passed": result.passed,
"errors": result.errors,
"warnings": result.warnings,
"dry_lab_only": result.dry_lab_only,
}
print(json.dumps(out, indent=2))
else:
status = "PASS" if result.passed else "FAIL"
print(f"Domain Review Outcome: {status}")
print(f" DRO ID: {result.dro_id}")
print(f" PEP ID: {result.pep_id}")
print(f" RVQ ID: {result.rvq_id}")
print(f" Domain: {result.review_domain}")
print(f" Verdict: {result.outcome_verdict}")
print(f" Confidence: {result.outcome_confidence}")
if result.errors:
print(" Errors:")
for e in result.errors:
print(f" - {e}")
if result.warnings:
print(" Warnings:")
for w in result.warnings:
print(f" - {w}")

sys.exit(0 if result.passed else 1)


def _run_simulation_registry(args: argparse.Namespace) -> int:
"""Display simulation module registry information."""
from openamp_foundry.simulation.module_registry import (
Expand Down
13 changes: 13 additions & 0 deletions src/openamp_foundry/cli/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
_run_negative_result_archive_check,
_run_failed_candidate_batch_report_check,
_run_reviewer_questionnaire_check,
_run_domain_review_outcome_check,
_run_hypothesis_outcome_check, _run_baseline_comparison_check, _run_negative_result_check,
_run_experiment_priority_check, _run_calibration_performance_check, _run_prediction_drift_check,
_run_calibration_improvement_check,
Expand Down Expand Up @@ -2328,6 +2329,15 @@ def build_parser() -> argparse.ArgumentParser:
"--format", choices=["text", "json"], default="text"
)

domain_review_outcome_parser = sub.add_parser(
"domain-review-outcome-check",
help="Validate a DomainReviewOutcome (DRO-).",
)
domain_review_outcome_parser.add_argument("--entry-json", required=True)
domain_review_outcome_parser.add_argument(
"--format", choices=["text", "json"], default="text"
)

# ── Hypothesis outcome check (Phase N N2) ────────────────────────
hypothesis_outcome_parser = sub.add_parser(
"hypothesis-outcome-check",
Expand Down Expand Up @@ -2837,6 +2847,9 @@ def main(argv: list[str] | None = None) -> int:
if args.command == "reviewer-questionnaire-check":
_run_reviewer_questionnaire_check(args)

if args.command == "domain-review-outcome-check":
_run_domain_review_outcome_check(args)

if args.command == "hypothesis-outcome-check":
return _run_hypothesis_outcome_check(args)

Expand Down
200 changes: 200 additions & 0 deletions src/openamp_foundry/evidence/domain_review_outcome.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,200 @@
"""Domain review outcome schema — Phase E E9.

Records the final verdict of a domain expert reviewer on a PilotEvidencePackage.
Uses a controlled taxonomy of expert domains and outcome verdicts, making expert
review decisions structured, auditable, and comparable across reviewers.
"""

from __future__ import annotations

import re
from dataclasses import dataclass, field
from typing import List

DRO_PREFIX = "DRO-"
RVQ_PREFIX = "RVQ-"
PEP_PREFIX = "PEP-"

VALID_REVIEW_DOMAINS = frozenset({
"antimicrobial_activity",
"toxicology",
"structural_biology",
"clinical_microbiology",
"computational_chemistry",
"general_biomedical",
})

VALID_OUTCOME_VERDICTS = frozenset({
"approve",
"reject",
"conditional_approve",
"request_revision",
"insufficient_data",
})

VALID_OUTCOME_CONFIDENCES = frozenset({
"high",
"medium",
"low",
})

RATIONALE_MAX_LENGTH = 400


@dataclass
class DomainReviewOutcome:
"""Formal expert verdict on a PilotEvidencePackage."""

dro_id: str
pipeline_version: str
pep_id: str # must start with "PEP-"
rvq_id: str # must start with "RVQ-" — the questionnaire that preceded this verdict
reviewer_token: str # anonymized reviewer ID, no PII
review_domain: str # controlled vocabulary
review_date: str # ISO YYYY-MM-DD
outcome_verdict: str # controlled vocabulary
outcome_confidence: str # {high, medium, low}
outcome_rationale: str # max 400 chars
dry_lab_only: bool = True


@dataclass
class DomainReviewOutcomeResult:
dro_id: str
pep_id: str
rvq_id: str
review_domain: str
outcome_verdict: str
outcome_confidence: str
passed: bool
errors: List[str] = field(default_factory=list)
warnings: List[str] = field(default_factory=list)
dry_lab_only: bool = True


def validate_domain_review_outcome(
entry: DomainReviewOutcome,
) -> DomainReviewOutcomeResult:
errors: List[str] = []
warnings: List[str] = []

# Rule 1: DRO- prefix
if not entry.dro_id.startswith(DRO_PREFIX):
errors.append(
f"dro_id must start with '{DRO_PREFIX}', got '{entry.dro_id}'"
)

# Rule 2: PEP- prefix
if not entry.pep_id.startswith(PEP_PREFIX):
errors.append(
f"pep_id must start with '{PEP_PREFIX}', got '{entry.pep_id}'"
)

# Rule 3: RVQ- prefix
if not entry.rvq_id.startswith(RVQ_PREFIX):
errors.append(
f"rvq_id must start with '{RVQ_PREFIX}', got '{entry.rvq_id}'"
)

# Rule 4: non-empty required strings
for fname, val in [
("pipeline_version", entry.pipeline_version),
("reviewer_token", entry.reviewer_token),
]:
if not val or not val.strip():
errors.append(f"{fname} must not be empty")

# Rule 5: valid review domain
if entry.review_domain not in VALID_REVIEW_DOMAINS:
errors.append(
f"review_domain must be one of {sorted(VALID_REVIEW_DOMAINS)}, "
f"got '{entry.review_domain}'"
)

# Rule 6: ISO date
if not re.match(r"^\d{4}-\d{2}-\d{2}$", entry.review_date):
errors.append(
f"review_date must be ISO format YYYY-MM-DD, got '{entry.review_date}'"
)

# Rule 7: valid outcome verdict
if entry.outcome_verdict not in VALID_OUTCOME_VERDICTS:
errors.append(
f"outcome_verdict must be one of {sorted(VALID_OUTCOME_VERDICTS)}, "
f"got '{entry.outcome_verdict}'"
)

# Rule 8: valid outcome confidence
if entry.outcome_confidence not in VALID_OUTCOME_CONFIDENCES:
errors.append(
f"outcome_confidence must be one of {sorted(VALID_OUTCOME_CONFIDENCES)}, "
f"got '{entry.outcome_confidence}'"
)

# Rule 9: rationale length
if len(entry.outcome_rationale) > RATIONALE_MAX_LENGTH:
errors.append(
f"outcome_rationale must be at most {RATIONALE_MAX_LENGTH} characters, "
f"got {len(entry.outcome_rationale)}"
)

# Warning 1: conditional_approve without rationale
if (
entry.outcome_verdict == "conditional_approve"
and not entry.outcome_rationale.strip()
):
warnings.append(
"outcome_verdict is 'conditional_approve' but outcome_rationale is empty"
" — document the conditions required for approval"
)

# Warning 2: reject with low confidence
if (
entry.outcome_verdict == "reject"
and entry.outcome_confidence == "low"
):
warnings.append(
"outcome_verdict is 'reject' with outcome_confidence='low'"
" — a low-confidence rejection may need additional expert review"
)

# Warning 3: insufficient_data without rationale
if (
entry.outcome_verdict == "insufficient_data"
and not entry.outcome_rationale.strip()
):
warnings.append(
"outcome_verdict is 'insufficient_data' but outcome_rationale is empty"
" — document what data is missing"
)

passed = len(errors) == 0
return DomainReviewOutcomeResult(
dro_id=entry.dro_id,
pep_id=entry.pep_id,
rvq_id=entry.rvq_id,
review_domain=entry.review_domain,
outcome_verdict=entry.outcome_verdict,
outcome_confidence=entry.outcome_confidence,
passed=passed,
errors=errors,
warnings=warnings,
dry_lab_only=entry.dry_lab_only,
)


def validate_domain_review_outcome_dict(data: dict) -> DomainReviewOutcomeResult:
entry = DomainReviewOutcome(
dro_id=data.get("dro_id", ""),
pipeline_version=data.get("pipeline_version", ""),
pep_id=data.get("pep_id", ""),
rvq_id=data.get("rvq_id", ""),
reviewer_token=data.get("reviewer_token", ""),
review_domain=data.get("review_domain", ""),
review_date=data.get("review_date", ""),
outcome_verdict=data.get("outcome_verdict", ""),
outcome_confidence=data.get("outcome_confidence", ""),
outcome_rationale=data.get("outcome_rationale", ""),
dry_lab_only=bool(data.get("dry_lab_only", True)),
)
return validate_domain_review_outcome(entry)
Loading
Loading