Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions docs/evidence/EXPERT_REVIEW_EXAMPLE_PACKAGE_GUIDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# ExpertReviewExamplePackage (ERP-) Guide

## Purpose

`ExpertReviewExamplePackage` is a self-contained example of a complete expert-review
submission, using only toy/mock candidates. It exists so external reviewers and
partners know exactly what to expect before receiving real data.

**This schema is for examples only.** Real review submissions use
`ReviewerQuestionnaire` (RVQ-) and `DomainReviewOutcome` (DRO-) schemas.

## Schema Fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `erp_id` | str | yes | Unique identifier, must start with `ERP-` |
| `pipeline_version` | str | yes | Pipeline version this example targets |
| `example_version` | str | yes | Version of this example (e.g. `v1.0`) |
| `creation_date` | str | yes | ISO date (YYYY-MM-DD) |
| `review_domain` | str | yes | Domain focus (controlled vocabulary) |
| `mock_candidates` | list | yes | 1-10 toy candidate summaries |
| `overall_clarity_rating` | int | yes | Likert 1-5 overall package clarity |
| `synthesis_recommendation` | str | yes | Example recommendation (controlled vocabulary) |
| `reviewer_comments` | str | yes | Example reviewer comments (≤500 chars) |
| `dry_lab_only` | bool | yes | Must be True |
| `is_example_data` | bool | yes | Must be True |
| `example_use_case` | str | yes | What this example demonstrates |
| `summary` | str | yes | Summary of the example (≤400 chars) |
| `notes` | str | no | Additional context (≤300 chars) |

## Mock Candidate Fields

Each entry in `mock_candidates` must have:
- `candidate_id`: must start with `MOCK-`, `TOY-`, `EXAMPLE-`, `DEMO-`, or `TEST-`
- `sequence_length`: positive integer
- `predicted_mic`: positive float (µg/mL)
- `predicted_toxicity`: `low`, `moderate`, or `high`
- `novelty_score`: float in [0.0, 1.0]
- `include_in_example`: bool (at least one must be True)

## Controlled Vocabularies

**review_domain (6 values):**
`antimicrobial_activity`, `toxicity_safety`, `novelty_assessment`,
`simulation_quality`, `evidence_completeness`, `experimental_design`

**synthesis_recommendation (4 values):**
`proceed`, `proceed_with_conditions`, `defer`, `reject`

## Validation Rules

1. `erp_id` starts with `ERP-`
2. `pipeline_version` non-empty
3. `example_version` non-empty
4. `creation_date` is ISO date (YYYY-MM-DD)
5. `review_domain` in controlled vocabulary (6 values)
6. `mock_candidates` has 1-10 entries
7. Each candidate: ID prefix, sequence_length ≥1, mic >0, toxicity vocab, novelty [0,1]
8. At least one candidate has `include_in_example=True`
9. `overall_clarity_rating` in {1,2,3,4,5}
10. `synthesis_recommendation` in controlled vocabulary (4 values)
11. `reviewer_comments` non-empty and ≤500 chars
12. `dry_lab_only` must be True
13. `is_example_data` must be True
14. `example_use_case` non-empty
15. `summary` non-empty and ≤400 chars
16. `notes` ≤300 chars

## Warnings

- `overall_clarity_rating ≤ 2`: verify the example package is complete
- Excluded candidates: verify `include_in_example=False` is intentional
- `notes` empty: consider documenting example context

## Integration

- Real review submissions: use `ReviewerQuestionnaire` (RVQ-)
- Expert verdicts: use `DomainReviewOutcome` (DRO-)
- This schema: onboarding, CI validation, partner documentation
8 changes: 8 additions & 0 deletions docs/evidence/METRICS_CURRENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,14 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak

## Changelog

### v0.10.24 — Phase E E10: ExpertReviewExamplePackage schema
- Added `ExpertReviewExamplePackage` (ERP-) schema
- 14 fields: erp_id, pipeline_version, example_version, creation_date, review_domain, mock_candidates (1-10), overall_clarity_rating, synthesis_recommendation, reviewer_comments, dry_lab_only, is_example_data, example_use_case, summary, notes
- 16 validation rules; 3 warnings; 63 tests
- BASELINE 6107→6170
- CLI: openamp-foundry expert-review-example-package-check
- Closes Phase E E10 — CI-checkable example package that cannot leak real candidates

### v0.10.23 — Phase G G8: SyntheticBoundaryAuditRecord schema
- Added `SyntheticBoundaryAuditRecord` (SBR-) schema
- 14 fields: sbr_id, pipeline_version, batch_id, audit_date, evidence_source, total_candidates_checked, total_violations, violation_rate, blocked_upgrades, max_proposed_ladder_level, policy_enforced, enforcement_outcome, summary, notes
Expand Down
2 changes: 1 addition & 1 deletion docs/research/NEXT_100_PR_MAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ Make qualified external review easier and safer.
| E7 | Add packet validator CLI. | Review readiness becomes testable. | C/D |
| E8 | Add release-summary generator that strips restricted fields. | Safer public summaries. | D |
| E9 | Add domain review outcome schema (complete). | Structured expert verdict on a PEP with controlled taxonomy of domains and outcomes; closes ESC→RVQ→DRO review chain. | B/C |
| E10 | Add expert-review example with mock/toy candidates only. | Safe demonstration. | B/C |
| E10 | Add expert-review example with mock/toy candidates only (complete). | ERP- schema: 14 fields, 16 validation rules, mock candidate ID prefix enforcement (MOCK-/TOY-/EXAMPLE-/DEMO-/TEST-), is_example_data=True and dry_lab_only=True enforced; CI-checkable template cannot accidentally leak real candidates. | B/C |

## Phase F — Negative-result infrastructure

Expand Down
17 changes: 17 additions & 0 deletions src/openamp_foundry/cli/commands/reports.py
Original file line number Diff line number Diff line change
Expand Up @@ -3065,3 +3065,20 @@ def _run_synthetic_boundary_audit_record_check(args):
print(f"ERROR: {e}", file=sys.stderr)
sys.exit(1)
print("OK: SyntheticBoundaryAuditRecord is valid.")


def _run_expert_review_example_package_check(args):
"""CLI handler for expert-review-example-package-check."""
import json, sys
from openamp_foundry.evidence.expert_review_example_package import validate_dict
data = json.load(open(args.input))
issues = validate_dict(data)
errors = [i for i in issues if not i.startswith("WARNING:")]
warnings = [i for i in issues if i.startswith("WARNING:")]
for w in warnings:
print(w)
if errors:
for e in errors:
print(f"ERROR: {e}", file=sys.stderr)
sys.exit(1)
print("OK: ExpertReviewExamplePackage is valid.")
8 changes: 8 additions & 0 deletions src/openamp_foundry/cli/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@
_run_recalibration_decision_log_check,
_run_recalibration_rejection_summary_check,
_run_synthetic_boundary_audit_record_check,
_run_expert_review_example_package_check,
)
from openamp_foundry.cli.commands.gates import _run_gate_check, _run_release_gate_check
from openamp_foundry.cli.commands.reports import _run_contribution_check
Expand Down Expand Up @@ -2543,6 +2544,10 @@ def build_parser() -> argparse.ArgumentParser:
p_sbr.add_argument("input", help="Path to JSON file")
p_sbr.set_defaults(func=_run_synthetic_boundary_audit_record_check)

p_erp = subparsers.add_parser("expert-review-example-package-check", help="Validate an ExpertReviewExamplePackage JSON")
p_erp.add_argument("input", help="Path to JSON file")
p_erp.set_defaults(func=_run_expert_review_example_package_check)

# ── Selection rationale check (Phase K K1) ───────────────────────
src2 = sub.add_parser(
"selection-rationale-check",
Expand Down Expand Up @@ -2988,6 +2993,9 @@ def main(argv: list[str] | None = None) -> int:
elif args.command == "recalibration-decision-log-check":
_run_recalibration_decision_log_check(args)

elif args.command == "expert-review-example-package-check":
_run_expert_review_example_package_check(args)

else:
parser.error("unknown command")
return 2
Expand Down
Loading
Loading