Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/evidence/METRICS_CURRENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
> with this file, this file wins. Updated whenever benchmark/benchmark config changes.
>
> **Last updated:** 2026-07-10 (Phase E E9 — domain review outcome — v0.10.14)
> **Last updated:** 2026-07-10 (Phase F F1 — negative result entry — v0.10.14)
> **New in v0.10.14 (Phase F F1):** NegativeResultEntry — atomic NRR- failure record with stage/reason/confidence vocabulary, foundation for NAS- and FCR- chain (12 fields, 10 validation rules, 4 warnings, 63 tests, BASELINE 5477→5540)
> **New in v0.10.14 (Phase E E9):** DomainReviewOutcome — controlled taxonomy expert verdict on a PEP (11 fields, 9 validation rules, 3 warnings, 63 tests, BASELINE 5414→5477); closes ESC→RVQ→DRO review chain
> **New in v0.10.13 (Phase E E3):** ReviewerQuestionnaire — structured external review feedback with Likert clarity ratings, synthesis recommendation, and comments (13 fields, 10 validation rules, 4 warnings, 63 tests, BASELINE 5351→5414)
> **New in v0.10.12 (Phase F F4):** FailedCandidateBatchReport — batch-level failure summary linking RJR- and NAS- with failure rate consistency check (13 fields, 11 validation rules, 4 warnings, 63 tests, BASELINE 5288→5351)
Expand Down
112 changes: 112 additions & 0 deletions docs/evidence/NEGATIVE_RESULT_ENTRY_GUIDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
# Negative Result Entry Guide (NRR-)

## Purpose

A `NegativeResultEntry` (NRR-) is the atomic failure record in the OpenAMP negative-result
infrastructure. It captures why a specific candidate was rejected at a specific pipeline
stage, using controlled vocabularies that enable structured failure analysis.

NRR- records are indexed by `NegativeResultArchiveSummary` (NAS-) and aggregated by
`FailedCandidateBatchReport` (FCR-). Without NRR-, the negative-result chain has no
foundation.

## Fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `nrr_id` | str | Yes | Unique ID, must start with `NRR-` |
| `pipeline_version` | str | Yes | Version string of the pipeline run |
| `candidate_id` | str | Yes | Identifier for the rejected candidate |
| `batch_id` | str | Yes | Batch this candidate belonged to |
| `experiment_date` | str | Yes | ISO date (YYYY-MM-DD) of rejection decision |
| `stage_at_rejection` | str | Yes | Pipeline stage where rejection occurred |
| `rejection_reason` | str | Yes | Controlled vocabulary reason for rejection |
| `rejection_confidence` | str | Yes | Confidence in the rejection decision |
| `outcome_summary` | str | Yes | Human-readable summary (≤400 chars) |
| `rejection_is_final` | bool | No | Whether this rejection is finalized (default: True) |
| `notes` | str | No | Additional context (≤300 chars) |
| `dry_lab_only` | bool | No | Always True; signals computational origin (default: True) |

## Valid Vocabularies

### `stage_at_rejection`

| Value | Description |
|-------|-------------|
| `sequence_quality` | Rejected at raw sequence quality check |
| `candidate_selection` | Rejected during selection/ranking |
| `toxicity_screen` | Rejected by toxicity predictor |
| `hemolysis_screen` | Rejected by hemolysis predictor |
| `novelty_check` | Rejected for insufficient novelty |
| `simulation` | Rejected by simulation module |
| `manual_review` | Excluded by human reviewer |

### `rejection_reason`

| Value | Description |
|-------|-------------|
| `low_score` | Score below selection threshold |
| `failed_toxicity` | Failed toxicity prediction gate |
| `failed_hemolysis` | Failed hemolysis prediction gate |
| `insufficient_novelty` | Too similar to known sequences |
| `simulation_failure` | Simulation returned invalid result |
| `duplicate_sequence` | Exact or near-duplicate already in pipeline |
| `low_confidence` | Model confidence below required minimum |
| `borderline_unsafe` | Near safety threshold, conservatively excluded |
| `manual_exclusion` | Human expert excluded for documented reason |

### `rejection_confidence`

| Value | Description |
|-------|-------------|
| `high` | Clear rejection with strong supporting signal |
| `medium` | Rejection supported but not certain |
| `low` | Marginal rejection; record may be revisited |
| `uncertain` | Reason for rejection unclear; do not treat as final |

## Validation Rules

1. `nrr_id` must start with `NRR-`
2. `pipeline_version` must be non-empty
3. `candidate_id` must be non-empty
4. `batch_id` must be non-empty
5. `experiment_date` must match `YYYY-MM-DD`
6. `stage_at_rejection` must be in valid set (7 values)
7. `rejection_reason` must be in valid set (9 values)
8. `rejection_confidence` must be in valid set (4 values)
9. `outcome_summary` must be non-empty and ≤400 chars
10. `notes` must be ≤300 chars

## Warnings

| Condition | Warning |
|-----------|---------|
| `rejection_is_final=False` and `rejection_confidence='high'` | Unusual: provisional record with high confidence |
| `stage='manual_review'` and `reason!='manual_exclusion'` | Stage/reason mismatch |
| `rejection_confidence='uncertain'` and `rejection_is_final=True` | Low-confidence final record |
| `rejection_reason='manual_exclusion'` and `notes=''` | Missing justification for human exclusion |

## Boundaries

- **NRR- is dry-lab only.** `dry_lab_only` is always True.
- **NRR- does not prove biological failure.** It records a computational or human filter decision.
- **Negative results are valuable.** Every NRR- is a calibration signal; do not discard or omit them.
- **`rejection_is_final=False` records may be revised.** They should not be indexed in NAS- until finalized.

## CLI

```bash
openamp-foundry negative-result-entry-check '{"nrr_id": "NRR-001", ...}'
```

Returns `VALID` or `INVALID` with any errors and warnings.

## Relationship to Other Schemas

```
NRR- ←── NAS- (NegativeResultArchiveSummary indexes NRR- ids)
NRR- ←── FCR- (FailedCandidateBatchReport references RJR- + NAS-, NAS- references NRR-)
RJR- ──→ FCR- (batch-level failure summary)
```

NRR- is the foundation. NAS- is the batch index. FCR- is the aggregated report.
2 changes: 1 addition & 1 deletion docs/research/NEXT_100_PR_MAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ Make failure useful.

| PR | Task | Why it matters | Review class |
|---:|---|---|---|
| F1 | Add negative-result entry schema hardening. | Failure records become useful. | B/C |
| F1 | Add negative-result entry schema hardening (complete). | Atomic NRR- failure record: controlled vocabulary for stage/reason/confidence, foundation for NAS- and FCR- negative-result chain. | B/C |
| F2 | Add negative-result archive summary schema (complete). | Prevents incomplete archives: indexes NRR- entries, enforces completeness_confirmed and all_results_have_reason before batch is marked archived. | B |
| F3 | Add rejection reason entry schema (complete). | Enables analysis of failed candidates with controlled vocabulary of stages and reasons; feeds calibration loop with structured failure signal. | B |
| F4 | Add failed candidate batch report schema (complete). | Makes failed batches reviewable: links RJR- rejection reasons and NAS- archive into a validated batch-level report with failure rate consistency check. | B/C |
Expand Down
22 changes: 22 additions & 0 deletions src/openamp_foundry/cli/commands/reports.py
Original file line number Diff line number Diff line change
Expand Up @@ -2860,3 +2860,25 @@ def _run_domain_review_outcome_check(args):
print(f" - {w}")

sys.exit(0 if result.passed else 1)


def _run_negative_result_entry_check(args):
import json, sys
from openamp_foundry.evidence.negative_result_entry import validate_dict
try:
data = json.loads(args.json_input)
except json.JSONDecodeError as exc:
print(f"INVALID: JSON parse error: {exc}", file=sys.stderr)
sys.exit(1)
result = validate_dict(data)
if result.valid:
print("VALID")
for w in result.warnings:
print(f"WARNING: {w}")
else:
print("INVALID")
for e in result.errors:
print(f"ERROR: {e}")
for w in result.warnings:
print(f"WARNING: {w}")
sys.exit(1)
11 changes: 11 additions & 0 deletions src/openamp_foundry/cli/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@
import json
from pathlib import Path

from openamp_foundry.evidence.negative_result_entry import validate_dict as validate_negative_result_entry_dict
from openamp_foundry.evidence.quality import assess_certificate_quality
from openamp_foundry.evidence.schemas import validate_json_schema
from openamp_foundry.pipeline import run_ranking_pipeline
Expand Down Expand Up @@ -2479,6 +2480,12 @@ def build_parser() -> argparse.ArgumentParser:
p_pre.add_argument("--format", choices=["text", "json"], default="text")
p_pre.set_defaults(func=_run_pre_registration_entry_check)

negative_result_entry_parser = sub.add_parser(
"negative-result-entry-check",
help="Validate a NegativeResultEntry (NRR-) record",
)
negative_result_entry_parser.add_argument("json_input", help="JSON string of the NRR- record")

# ── Selection rationale check (Phase K K1) ───────────────────────
src2 = sub.add_parser(
"selection-rationale-check",
Expand Down Expand Up @@ -2893,6 +2900,10 @@ def main(argv: list[str] | None = None) -> int:
if args.command == "pre-registration-check":
return _run_pre_registration_entry_check(args)

if args.command == "negative-result-entry-check":
from openamp_foundry.cli.commands.reports import _run_negative_result_entry_check
_run_negative_result_entry_check(args)

parser.error("unknown command")
return 2

Expand Down
174 changes: 174 additions & 0 deletions src/openamp_foundry/evidence/negative_result_entry.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,174 @@
"""Negative result entry schema — Phase F F1.

Records the outcome of a failed candidate at a specific pipeline stage.
Atomic failure record indexed by NAS- and aggregated by FCR-.
Uses controlled vocabulary for rejection stage, reason, and confidence.
"""

from __future__ import annotations

import re
from dataclasses import dataclass, field
from typing import List

NRR_PREFIX = "NRR-"

VALID_REJECTION_STAGES = frozenset({
"sequence_quality",
"candidate_selection",
"toxicity_screen",
"hemolysis_screen",
"novelty_check",
"simulation",
"manual_review",
})

VALID_REJECTION_REASONS = frozenset({
"low_score",
"failed_toxicity",
"failed_hemolysis",
"insufficient_novelty",
"simulation_failure",
"duplicate_sequence",
"low_confidence",
"borderline_unsafe",
"manual_exclusion",
})

VALID_REJECTION_CONFIDENCES = frozenset({
"high",
"medium",
"low",
"uncertain",
})

OUTCOME_SUMMARY_MAX_LENGTH = 400
NOTES_MAX_LENGTH = 300
_ISO_DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$")


@dataclass
class NegativeResultEntry:
nrr_id: str
pipeline_version: str
candidate_id: str
batch_id: str
experiment_date: str
stage_at_rejection: str
rejection_reason: str
rejection_confidence: str
outcome_summary: str
rejection_is_final: bool = True
notes: str = ""
dry_lab_only: bool = True


@dataclass
class NegativeResultEntryResult:
valid: bool
errors: List[str] = field(default_factory=list)
warnings: List[str] = field(default_factory=list)


def validate(entry: NegativeResultEntry) -> NegativeResultEntryResult:
errors: List[str] = []
warnings: List[str] = []

if not entry.nrr_id.startswith(NRR_PREFIX):
errors.append(f"nrr_id must start with '{NRR_PREFIX}', got: {entry.nrr_id!r}")

if not entry.pipeline_version.strip():
errors.append("pipeline_version must be a non-empty string")

if not entry.candidate_id.strip():
errors.append("candidate_id must be a non-empty string")

if not entry.batch_id.strip():
errors.append("batch_id must be a non-empty string")

if not _ISO_DATE_RE.match(entry.experiment_date):
errors.append(
f"experiment_date must be ISO format YYYY-MM-DD, got: {entry.experiment_date!r}"
)

if entry.stage_at_rejection not in VALID_REJECTION_STAGES:
errors.append(
f"stage_at_rejection {entry.stage_at_rejection!r} not in valid set: "
f"{sorted(VALID_REJECTION_STAGES)}"
)

if entry.rejection_reason not in VALID_REJECTION_REASONS:
errors.append(
f"rejection_reason {entry.rejection_reason!r} not in valid set: "
f"{sorted(VALID_REJECTION_REASONS)}"
)

if entry.rejection_confidence not in VALID_REJECTION_CONFIDENCES:
errors.append(
f"rejection_confidence {entry.rejection_confidence!r} not in valid set: "
f"{sorted(VALID_REJECTION_CONFIDENCES)}"
)

if not entry.outcome_summary.strip():
errors.append("outcome_summary must be a non-empty string")
elif len(entry.outcome_summary) > OUTCOME_SUMMARY_MAX_LENGTH:
errors.append(
f"outcome_summary exceeds {OUTCOME_SUMMARY_MAX_LENGTH} chars "
f"(got {len(entry.outcome_summary)})"
)

if len(entry.notes) > NOTES_MAX_LENGTH:
errors.append(
f"notes exceeds {NOTES_MAX_LENGTH} chars (got {len(entry.notes)})"
)

# Warnings
if not entry.rejection_is_final and entry.rejection_confidence == "high":
warnings.append(
"rejection_is_final=False with rejection_confidence='high' is unusual; "
"verify this record is intentionally provisional"
)

if (
entry.stage_at_rejection == "manual_review"
and entry.rejection_reason != "manual_exclusion"
):
warnings.append(
"stage_at_rejection='manual_review' typically uses rejection_reason='manual_exclusion'; "
f"got {entry.rejection_reason!r}"
)

if entry.rejection_confidence == "uncertain" and entry.rejection_is_final:
warnings.append(
"rejection_confidence='uncertain' with rejection_is_final=True; "
"consider marking as non-final until confidence improves"
)

if entry.rejection_reason == "manual_exclusion" and not entry.notes.strip():
warnings.append(
"rejection_reason='manual_exclusion' without notes; "
"human exclusion decisions should be explained"
)

return NegativeResultEntryResult(valid=len(errors) == 0, errors=errors, warnings=warnings)


def validate_dict(data: dict) -> NegativeResultEntryResult:
try:
entry = NegativeResultEntry(
nrr_id=data.get("nrr_id", ""),
pipeline_version=data.get("pipeline_version", ""),
candidate_id=data.get("candidate_id", ""),
batch_id=data.get("batch_id", ""),
experiment_date=data.get("experiment_date", ""),
stage_at_rejection=data.get("stage_at_rejection", ""),
rejection_reason=data.get("rejection_reason", ""),
rejection_confidence=data.get("rejection_confidence", ""),
outcome_summary=data.get("outcome_summary", ""),
rejection_is_final=bool(data.get("rejection_is_final", True)),
notes=data.get("notes", ""),
dry_lab_only=bool(data.get("dry_lab_only", True)),
)
except Exception as exc:
return NegativeResultEntryResult(valid=False, errors=[f"Construction error: {exc}"])
return validate(entry)
Loading
Loading