Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions docs/evidence/FAILED_CANDIDATE_BATCH_REPORT_GUIDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Failed Candidate Batch Report Guide

## Purpose

A `FailedCandidateBatchReport` (FCR-) summarizes all failed candidates from a
screening batch. It links `RejectionReasonEntry` (RJR-) records and the
`NegativeResultArchiveSummary` (NAS-) into a single auditable report, making
failed batches reviewable and enabling calibration loop learning.

## Fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `fcr_id` | str | Yes | Unique ID, must start with `FCR-` |
| `pipeline_version` | str | Yes | Pipeline version at report time |
| `batch_id` | str | Yes | Which batch this report covers |
| `report_date` | str | Yes | ISO YYYY-MM-DD report date |
| `total_candidates_screened` | int | Yes | Must be >= 1 |
| `failed_candidate_count` | int | Yes | Must be >= 0 and <= total |
| `failure_rate` | float | Yes | Must be in [0.0, 1.0], consistent with counts |
| `top_rejection_reasons` | List[str] | Yes | From controlled vocabulary |
| `rejection_reason_ids` | List[str] | Yes | Each must start with `RJR-` |
| `negative_result_archive_id` | str | Yes | Must start with `NAS-` if provided |
| `report_notes` | str | Yes | Context notes (max 400 chars) |
| `reviewer` | str | Yes | Human who reviewed the batch report |
| `dry_lab_only` | bool | No | Default `True` |

## Validation rules

1. `fcr_id` must start with `FCR-`
2. `pipeline_version`, `batch_id`, `reviewer` must not be empty
3. `report_date` must be ISO format `YYYY-MM-DD`
4. `total_candidates_screened` >= 1
5. `failed_candidate_count` in [0, `total_candidates_screened`]
6. `failure_rate` in [0.0, 1.0]
7. `failure_rate` consistent with counts (tolerance 0.01)
8. `top_rejection_reasons` all from controlled vocabulary
9. All `rejection_reason_ids` start with `RJR-`
10. `negative_result_archive_id` starts with `NAS-` if non-empty
11. `report_notes` at most 400 characters

## Warnings

- Empty `rejection_reason_ids` — link RJR- records for each failed candidate
- Empty `negative_result_archive_id` — link the NAS- archive for this batch
- `failure_rate = 1.0` — all candidates failed; review screening thresholds
- Empty `report_notes` — document the batch context

## Rejection reason vocabulary

See `RejectionReasonEntry` (RJR-) guide for the full taxonomy of valid stages and reasons.

## Honest boundaries

- This report summarizes dry-lab screening rejections only.
- `failure_rate` is computed from predicted scores, not wet-lab results.
- A high failure rate may indicate threshold miscalibration — use RJR records to investigate.

## CLI usage

```bash
openamp-foundry failed-candidate-batch-report-check \
--entry-json '{"fcr_id":"FCR-001","pipeline_version":"v0.10.11","batch_id":"BATCH-001","report_date":"2026-07-10","total_candidates_screened":10,"failed_candidate_count":4,"failure_rate":0.4,"top_rejection_reasons":["toxicity_risk"],"rejection_reason_ids":["RJR-001"],"negative_result_archive_id":"NAS-001","report_notes":"Batch summary.","reviewer":"Dr. Smith"}' \
--format text
```
3 changes: 2 additions & 1 deletion docs/evidence/METRICS_CURRENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
> with this file, this file wins. Updated whenever benchmark/benchmark config changes.
>
> **Last updated:** 2026-07-10 (Phase F F2 — negative result archive summary — v0.10.11)
> **Last updated:** 2026-07-10 (Phase F F4 — failed candidate batch report — v0.10.12)
> **New in v0.10.12 (Phase F F4):** FailedCandidateBatchReport — batch-level failure summary linking RJR- and NAS- with failure rate consistency check (13 fields, 11 validation rules, 4 warnings, 63 tests, BASELINE 5288→5351)
> **New in v0.10.11 (Phase F F2):** NegativeResultArchiveSummary — validates batch of NRR- entries is complete before archiving (11 fields, 9 validation rules, 3 warnings, 63 tests, BASELINE 5225→5288)
> **New in v0.10.10 (Phase F F3):** RejectionReasonEntry — controlled vocabulary for pipeline rejections (11 fields, 9 validation rules, 3 warnings, 63 tests, BASELINE 5162→5225)
> **New in v0.10.9 (Phase Q Q3):** ExternalSharingClearance — auditable release gate before external PEP sharing (12 fields, 9 validation rules, 3 warnings, 63 tests, BASELINE 5099→5162)
Expand Down
2 changes: 1 addition & 1 deletion docs/research/NEXT_100_PR_MAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ Make failure useful.
| F1 | Add negative-result entry schema hardening. | Failure records become useful. | B/C |
| F2 | Add negative-result archive summary schema (complete). | Prevents incomplete archives: indexes NRR- entries, enforces completeness_confirmed and all_results_have_reason before batch is marked archived. | B |
| F3 | Add rejection reason entry schema (complete). | Enables analysis of failed candidates with controlled vocabulary of stages and reasons; feeds calibration loop with structured failure signal. | B |
| F4 | Add failed-candidate report generator. | Makes failed batches reviewable. | B/C |
| F4 | Add failed candidate batch report schema (complete). | Makes failed batches reviewable: links RJR- rejection reasons and NAS- archive into a validated batch-level report with failure rate consistency check. | B/C |
| F5 | Add safe-publication filter for negative results. | Supports openness without unsafe release. | D |
| F6 | Add examples of non-informative vs informative negative results. | Better interpretation. | A |
| F7 | Add calibration link from negative-result entries to intake reports. | Closes learning loop. | C |
Expand Down
Loading
Loading