Skip to content

Loop 73: Rejection reason taxonomy schema (Phase F F3) #758

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cschanhniem merged 1 commit into from
Jul 9, 2026
Merged

Loop 73: Rejection reason taxonomy schema (Phase F F3) #758

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions docs/evidence/METRICS_CURRENT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
> with this file, this file wins. Updated whenever benchmark/benchmark config changes.
>
> **Last updated:** 2026-07-09 (Phase E safety release, preregistration, packet generator — v0.5.73)
> **Last updated:** 2026-07-09 (Phase F rejection taxonomy — v0.5.74)
> **New in v0.5.74:** Rejection reason taxonomy schema (F3) — machine-readable taxonomy with severity, evidence_impact, and applies_at_stage fields. Comprehensive REJECTION_TAXONOMY.md doc. 19 tests. 2698 total.
>
> **New in v0.5.73:** Safety-release decision schema (E4), pilot pre-registration schema (E5), review packet generator CLI (E6). 40 tests. 2679 total.
>
> **New in v0.5.72:** External review packet schema (E1), example packet (E2), reviewer questionnaire schema (E3). 27 tests. 2666 total.
Expand Down Expand Up @@ -56,7 +58,7 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **New in v0.5.31:** Added dipeptide-order features for sequence-order awareness. `dipeptide_order_score` achieves AUROC 0.7861 on AMP-vs-scrambled discrimination — the strongest order-dependent feature in the pipeline. Only 7/31 features survive scrambling (amphipathicity/helix-wheel + dipeptide). All composition features are purely position-independent (exactly 0.5000 AUROC on scrambled test).
> **New in v0.5.30:** Easy baseline benchmark added — charge density alone (AUROC 0.8166) outperforms the full pipeline ensemble (0.7792) on AMP-vs-Swiss-Prot-decoy discrimination. Honest finding documented: expected because pipeline optimizes for safety, not raw discrimination.
> **New in v0.5.29:** Expanded benchmark to 500 AMPs + 500 composition-matched decoys (n=1000). AUROC 0.7792 (CI₉₅: 0.7505–0.8065) confirms signal generalizes. Cluster-aware CI: 0.746–0.8102. Representative AUROC: 0.778. Standard benchmark (n=191) retained for backward comparison.
> **Pipeline version:** v0.5.73
> **Pipeline version:** v0.5.74
> **Branch:** main

---
Expand Down
128 changes: 107 additions & 21 deletions docs/evidence/REJECTION_TAXONOMY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,32 +2,118 @@

## Purpose

Standardized categories for why candidates are rejected by the pipeline. Enables analysis of failure patterns across batches.
Standardized, machine-readable categories for why candidates are rejected by
the OpenAMP pipeline. Enables analysis of failure patterns across batches,
consistent archive entries, and evidence-aware claim boundaries for rejected
candidates.

## Categories
## Schema

| Category | Description | Example |
|----------|-------------|---------|
| `sequence_invalid` | Sequence contains non-canonical amino acids or unparseable symbols. | "Sequence contains 'X' at position 5." |
| `length_out_of_range` | Sequence length outside configured min/max filter bounds. | "Length 6 outside filter range [8, 35]." |
| `novelty_below_threshold` | Similarity to a known reference exceeds the novelty cutoff. | "Nearest reference similarity 0.85 exceeds max 0.70." |
| `safety_risk_threshold` | Safety score below minimum (or risk score above maximum). | "Safety score 0.31 below minimum 0.50." |
| `synthesis_feasibility_low` | Synthesis feasibility score below viability threshold. | "Synthesis score 0.20 — multiple Trp residues with oxidation risk." |
| `disagreement_too_high` | Scorer disagreement exceeds max tolerance. | "Disagreement 0.85 exceeds max 0.50." |
| `diversity_filter_prune` | Removed by diversity selection to avoid near-duplicates. | "Similarity 0.82 to already-selected candidate." |
| `simulation_uncertainty_high` | Simulation proxy uncertainty exceeds 0.5 threshold. | "Max simulation uncertainty 0.92 exceeds 0.5." |
| `toxicity_risk_flagged` | Hemolysis or cytotoxicity risk above acceptable level. | "Hemolysis risk 0.89 flagged as HIGH." |
A machine-readable JSON Schema defines the taxonomy structure:

## Usage
- **Source:** `schemas/rejection_taxonomy.schema.json`
- **Example:** `examples/rejection_taxonomy_example.json`

Rejection categories are recorded in:
- Pipeline output JSONL (`known_failure_modes` field)
- Failed-candidate report output (if enabled)
- Negative-result archive entries
Every taxonomy entry includes: `code`, `category`, `label`, `description`,
`severity`, `evidence_impact`, `applies_at_stage`, and `related_reason_category`
(mapping to `schemas/negative_result_entry.schema.json`).

## Taxonomy hierarchy

Rejection reasons are grouped by pipeline stage. Each entry has a severity
and an evidence impact on the proof ladder.

### Pre-selection (before scoring)

| Code | Severity | Evidence impact | Description |
|------|:--------:|:---------------:|-------------|
| `PRE_SEQ_INVALID` | hard | claim_unsupported | Non-canonical amino acids or unparseable symbols |
| `PRE_LENGTH_OUT` | hard | claim_unsupported | Length outside configured range |
| `PRE_PATENT_NEAR` | soft | downgrade_by_1 | Similarity to a known reference exceeds novelty cutoff |

### Pipeline scoring

| Code | Severity | Evidence impact | Description |
|------|:--------:|:---------------:|-------------|
| `PIPE_ACTIVITY_LOW` | soft | downgrade_by_1 | Activity score below minimum gate threshold |
| `PIPE_SAFETY_LOW` | soft | downgrade_by_1 | Safety score below minimum threshold |
| `PIPE_NOVELTY_LOW` | soft | downgrade_by_1 | Novelty score below minimum threshold |
| `PIPE_SYNTHESIS_LOW` | soft | downgrade_by_1 | Synthesis feasibility below viability threshold |
| `PIPE_DISAGREEMENT_HI` | informational | no_impact | Scorer disagreement exceeds maximum tolerance |
| `PIPE_ENSEMBLE_LOW` | soft | downgrade_by_1 | Composite ensemble score below synthesis gate threshold |
| `PIPE_SIM_UNCERTAINTY` | informational | no_impact | Simulation proxy uncertainty exceeds 0.5 threshold |

### Diversity and selection

| Code | Severity | Evidence impact | Description |
|------|:--------:|:---------------:|-------------|
| `DIV_TOO_SIMILAR` | soft | no_impact | Near-duplicate of already-selected candidate |
| `DIV_FAMILY_CAP` | soft | no_impact | Per-family panel cap reached |
| `DIV_BUDGET` | informational | no_impact | Budget or scope limitation |

### Reviewer rejection

| Code | Severity | Evidence impact | Description |
|------|:--------:|:---------------:|-------------|
| `REV_EXPERT_REJECT` | hard | downgrade_by_2 | Qualified domain expert recommended against progression |
| `REV_SAFETY_CONCERN` | hard | downgrade_by_2 | Human safety reviewer flagged structural or dual-use concern |
| `REV_INSUFFICIENT` | soft | downgrade_by_1 | Evidence package does not meet minimum reviewer standards |

### Lab result

| Code | Severity | Evidence impact | Description |
|------|:--------:|:---------------:|-------------|
| `LAB_INACTIVE` | hard | downgrade_by_2 | Lab-confirmed inactive (MIC above cutoff) |
| `LAB_TOXIC` | hard | downgrade_by_2 | Lab-confirmed hemolytic or cytotoxic |
| `LAB_CONTROL_FAIL` | hard | claim_unsupported | Assay controls failed — batch invalidated |
| `LAB_SYNTHESIS_FAIL` | hard | downgrade_by_2 | Could not be synthesized at sufficient purity |

### Lifecycle

| Code | Severity | Evidence impact | Description |
|------|:--------:|:---------------:|-------------|
| `LIFECYCLE_SUPERSEDED` | informational | no_impact | Replaced by a higher-scoring or better-characterised candidate |
| `LIFECYCLE_WITHDRAWN` | informational | no_impact | Withdrawn for non-technical reasons (scope, budget, partner feedback) |

## Severity categories

| Severity | Meaning |
|----------|---------|
| `hard` | Final rejection. The candidate cannot proceed regardless of threshold adjustment. Requires new evidence to overturn. |
| `soft` | Conditional on current thresholds. A policy or threshold change could convert this to a pass. The rejection record should document the specific threshold. |
| `informational` | Context-dependent. The candidate was rejected for reasons external to its own merits (budget, diversity, uncertainty). May still be valid in a different context. |

## Evidence impact

| Impact | Meaning |
|--------|---------|
| `claim_unsupported` | No claim about this candidate can be made at any proof-ladder level. The input itself is invalid or the assay failed. |
| `downgrade_by_2` | The highest supportable proof-ladder level drops by 2. For example, from Level 4 (multi-signal candidate evidence) to Level 2 (baseline-triaged). |
| `downgrade_by_1` | The highest supportable proof-ladder level drops by 1. |
| `no_impact` | The rejection does not affect the candidate's claim level. The candidate remains valid at its current evidence level (e.g., diversity pruning). |

## Usage in tools

| Artifact | How the taxonomy is used |
|----------|--------------------------|
| `schemas/negative_result_entry.schema.json` | `reason_category` field maps to the taxonomy's top-level categories |
| Pipeline output JSONL | `known_failure_modes` field records the applying rejection codes |
| Failed-candidate report generator | Reads taxonomy to produce structured rejection summaries |
| Negative-result archive validator | Checks `reason_category` against allowed values |
| External review packet | Rejection codes and rationales are included for reviewed candidates |

## Cross-references

- `docs/evidence/TAXONOMY_EXCLUSION_REASONS.md` — Lightweight exclusion-reason table for selection workflows
- `docs/evidence/NEGATIVE_RESULT_ARCHIVE.md` — Archive format that stores rejection records
- `schemas/negative_result_entry.schema.json` — Rejection entry schema with reason_category enum
- `schemas/rejection_taxonomy.schema.json` — Machine-readable taxonomy schema

## Non-goals

This taxonomy does not cover:
- Wet-lab assay failures (e.g., synthesis failure, control failure) — see NEGATIVE_RESULT_ARCHIVE.md
- Human review rejections — see EXTERNAL_REVIEW_PACKET.md
- Safety or dual-use review rejections — see DUAL_USE_REVIEW.md

- Wet-lab assay design decisions — see `docs/review/WET_LAB_HANDOFF.md`
- Human review deliberation format — see `docs/review/EXTERNAL_REVIEW_PACKET.md`
- Safety or dual-use review policy — see `docs/trust/SAFETY_DOC_AUDIT.md`
- Recalibration policy decisions — see `docs/evidence/CALIBRATION_POLICY.md`
40 changes: 40 additions & 0 deletions docs/research/ROADMAP.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,45 @@
# Roadmap

## v0.5.74 — Loop 73: Phase F F3 — Rejection Reason Taxonomy Schema ✓ (2026-07-09)

Negative-result infrastructure begins: machine-readable rejection reason taxonomy
with severity, evidence impact, and stage classification. Foundation for F4 (report
generator) and F5 (publication filter).

Changes:
- `schemas/rejection_taxonomy.schema.json` (F3) — JSON Schema Draft 2020-12 defining
rejection taxonomy entries. Required fields: code (pattern `^[A-Z]+_[A-Za-z_]+$`),
category (pre_selection/pipeline/diversity/reviewer/lab/lifecycle), label,
description (min 10 chars), severity (hard/soft/informational), evidence_impact
(claim_unsupported/downgrade_by_1/downgrade_by_2/no_impact), applies_at_stage
(pre_selection/after_scoring/after_diversity/after_review/after_lab/any),
related_reason_category mapping to negative_result_entry schema. Optional: example.
- `examples/rejection_taxonomy_example.json` (F3) — Complete taxonomy with 21 entries
covering all 6 categories: PRE_SEQ_INVALID, PRE_LENGTH_OUT, PRE_PATENT_NEAR,
PIPE_ACTIVITY_LOW, PIPE_SAFETY_LOW, PIPE_NOVELTY_LOW, PIPE_SYNTHESIS_LOW,
PIPE_DISAGREEMENT_HI, PIPE_ENSEMBLE_LOW, PIPE_SIM_UNCERTAINTY, DIV_TOO_SIMILAR,
DIV_FAMILY_CAP, DIV_BUDGET, REV_EXPERT_REJECT, REV_SAFETY_CONCERN, REV_INSUFFICIENT,
LAB_INACTIVE, LAB_TOXIC, LAB_CONTROL_FAIL, LAB_SYNTHESIS_FAIL, LIFECYCLE_SUPERSEDED,
LIFECYCLE_WITHDRAWN.
- `docs/evidence/REJECTION_TAXONOMY.md` — Comprehensive reference doc with hierarchy
tables, severity definitions, evidence impact definitions, usage in tools, and
cross-references to related schemas/docs.
- `tests/evidence/test_rejection_taxonomy_schema.py` — 19 tests covering valid entry,
missing required fields, invalid enums (category, severity, evidence_impact,
applies_at_stage, related_reason_category), code pattern validation, description
length, multiple entries, optional fields, example file validation, code uniqueness,
and stage coverage completeness.

Honest boundaries:
- The taxonomy defines valid rejection reasons but does not enforce that tools
actually use them — adoption requires downstream tool updates (F4, F5).
- Hard rejections require new evidence to overturn, but the taxonomy does not
define what counts as sufficient new evidence.
- The `evidence_impact` field describes claim-level effects; it does not guarantee
that reviewers will apply the specified downgrade in practice.
- Severity is a guideline; final rejection authority rests with human reviewers
for class C/D decisions.

## v0.5.73 — Loop 72: Phase E E4-E6 — Safety Release, Preregistration, Packet CLI ✓ (2026-07-09)

External review infrastructure: human-gated release decisions, pre-registered pilot
Expand Down
Loading
Loading