feat: Phase E E3 reviewer questionnaire schema -- Likert-scale external review feedback (RVQ-) makes review machine-readable

docs/evidence/METRICS_CURRENT.md

@@ -5,7 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
 > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
 > with this file, this file wins. Updated whenever benchmark/benchmark config changes.
 >
-> **Last updated:** 2026-07-10 (Phase F F4 — failed candidate batch report — v0.10.12)
+> **Last updated:** 2026-07-10 (Phase E E3 — reviewer questionnaire — v0.10.13)
+> **New in v0.10.13 (Phase E E3):** ReviewerQuestionnaire — structured external review feedback with Likert clarity ratings, synthesis recommendation, and comments (13 fields, 10 validation rules, 4 warnings, 63 tests, BASELINE 5351→5414)
 > **New in v0.10.12 (Phase F F4):** FailedCandidateBatchReport — batch-level failure summary linking RJR- and NAS- with failure rate consistency check (13 fields, 11 validation rules, 4 warnings, 63 tests, BASELINE 5288→5351)
 > **New in v0.10.11 (Phase F F2):** NegativeResultArchiveSummary — validates batch of NRR- entries is complete before archiving (11 fields, 9 validation rules, 3 warnings, 63 tests, BASELINE 5225→5288)
 > **New in v0.10.10 (Phase F F3):** RejectionReasonEntry — controlled vocabulary for pipeline rejections (11 fields, 9 validation rules, 3 warnings, 63 tests, BASELINE 5162→5225)

docs/evidence/REVIEWER_QUESTIONNAIRE_GUIDE.md

@@ -0,0 +1,69 @@
+# Reviewer Questionnaire Guide
+
+## Purpose
+
+A `ReviewerQuestionnaire` (RVQ-) captures structured feedback from an external reviewer
+on a `PilotEvidencePackage` (PEP). It uses Likert-scale ratings for claim clarity, a
+synthesis recommendation, and structured comments to make review feedback machine-readable
+and comparable across reviewers.
+
+## Fields
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `rvq_id` | str | Yes | Unique ID, must start with `RVQ-` |
+| `pipeline_version` | str | Yes | Pipeline version being reviewed |
+| `pep_id` | str | Yes | Which PEP is being reviewed (must start with `PEP-`) |
+| `reviewer_token` | str | Yes | Anonymized reviewer ID — no PII |
+| `review_date` | str | Yes | ISO YYYY-MM-DD date of review |
+| `activity_prediction_clarity` | int | Yes | Likert 1-5: how clear is the activity prediction? |
+| `safety_claim_clarity` | int | Yes | Likert 1-5: how clear are the safety claims? |
+| `novelty_claim_clarity` | int | Yes | Likert 1-5: how clear is the novelty claim? |
+| `overall_package_quality` | int | Yes | Likert 1-5: overall quality of the evidence package |
+| `would_recommend_for_synthesis` | str | Yes | `yes`, `no`, `conditional`, or `insufficient_information` |
+| `missing_information` | List[str] | Yes | Specific gaps identified (empty if none) |
+| `reviewer_comments` | str | Yes | Free-text comments (max 600 chars) |
+| `dry_lab_only` | bool | No | Default `True` |
+
+## Validation rules
+
+1. `rvq_id` must start with `RVQ-`
+2. `pep_id` must start with `PEP-`
+3. `pipeline_version`, `reviewer_token` must not be empty
+4. `review_date` must be ISO format `YYYY-MM-DD`
+5. `activity_prediction_clarity` in [1, 5]
+6. `safety_claim_clarity` in [1, 5]
+7. `novelty_claim_clarity` in [1, 5]
+8. `overall_package_quality` in [1, 5]
+9. `would_recommend_for_synthesis` in `{yes, no, conditional, insufficient_information}`
+10. `reviewer_comments` at most 600 characters
+
+## Warnings
+
+- `overall_package_quality < 3` — package needs significant improvement
+- `conditional` recommendation with no comments — document the conditions
+- `no` recommendation with no `missing_information` or comments — document why
+- `insufficient_information` with empty `missing_information` — list the specific gaps
+
+## Synthesis recommendations
+
+| Value | Meaning |
+|-------|---------|
+| `yes` | Package is sufficient for proceeding to wet-lab synthesis |
+| `no` | Package is not sufficient in current form |
+| `conditional` | Sufficient if specific gaps are addressed (document in comments) |
+| `insufficient_information` | Cannot assess — more information required |
+
+## Honest boundaries
+
+- A `yes` recommendation from a single reviewer does not authorize wet-lab synthesis.
+- `reviewer_token` must never contain PII (name, email, institutional ID).
+- Likert scores reflect the reviewer's perception of the dry-lab package clarity, not biological validity.
+
+## CLI usage
+
+```bash
+openamp-foundry reviewer-questionnaire-check \
+  --entry-json '{"rvq_id":"RVQ-001","pipeline_version":"v0.10.12","pep_id":"PEP-001","reviewer_token":"REV-A","review_date":"2026-07-10","activity_prediction_clarity":4,"safety_claim_clarity":4,"novelty_claim_clarity":3,"overall_package_quality":4,"would_recommend_for_synthesis":"yes","missing_information":[],"reviewer_comments":"Clear and complete."}' \
+  --format text
+```

docs/research/NEXT_100_PR_MAP.md

@@ -94,7 +94,7 @@ Make qualified external review easier and safer.
 |---:|---|---|---|
 | E1 | Add external review packet schema. | Packets become machine-checkable. | C/D |
 | E2 | Add example external review packet using toy data. | Partners know what to expect. | C/D |
-| E3 | Add reviewer questionnaire JSON schema. | Review feedback becomes structured. | B/C |
+| E3 | Add reviewer questionnaire schema (complete). | Makes external review feedback machine-readable: Likert clarity ratings for activity/safety/novelty claims, synthesis recommendation, structured comments. | B/C |
 | E4 | Add safety-release decision schema. | Release review becomes auditable. | D |
 | E5 | Add non-protocol pilot pre-registration schema. | Freezes selection logic. | C/D |
 | E6 | Add packet generator CLI. | Reduces manual packaging errors. | C/D |

src/openamp_foundry/cli/commands/reports.py

@@ -2729,3 +2729,44 @@ def _run_advisory_review_check(args: argparse.Namespace) -> int:
 
 def _run_roadmap_sync_check(args: argparse.Namespace) -> int:
     """Validate a roadmap sync entry."""
+
+
+def _run_reviewer_questionnaire_check(args):
+    from openamp_foundry.evidence.reviewer_questionnaire import (
+        validate_reviewer_questionnaire_dict,
+    )
+    import json
+    import sys
+
+    data = json.loads(args.entry_json)
+    result = validate_reviewer_questionnaire_dict(data)
+
+    if args.format == "json":
+        out = {
+            "rvq_id": result.rvq_id,
+            "pep_id": result.pep_id,
+            "overall_package_quality": result.overall_package_quality,
+            "would_recommend_for_synthesis": result.would_recommend_for_synthesis,
+            "passed": result.passed,
+            "errors": result.errors,
+            "warnings": result.warnings,
+            "dry_lab_only": result.dry_lab_only,
+        }
+        print(json.dumps(out, indent=2))
+    else:
+        status = "PASS" if result.passed else "FAIL"
+        print(f"Reviewer Questionnaire: {status}")
+        print(f"  RVQ ID: {result.rvq_id}")
+        print(f"  PEP ID: {result.pep_id}")
+        print(f"  Overall Quality: {result.overall_package_quality}/5")
+        print(f"  Recommendation: {result.would_recommend_for_synthesis}")
+        if result.errors:
+            print("  Errors:")
+            for e in result.errors:
+                print(f"    - {e}")
+        if result.warnings:
+            print("  Warnings:")
+            for w in result.warnings:
+                print(f"    - {w}")
+
+    sys.exit(0 if result.passed else 1)

src/openamp_foundry/cli/main.py

@@ -22,6 +22,7 @@
     _run_rejection_reason_check,
     _run_negative_result_archive_check,
     _run_failed_candidate_batch_report_check,
+    _run_reviewer_questionnaire_check,
     _run_hypothesis_outcome_check, _run_baseline_comparison_check, _run_negative_result_check,
     _run_experiment_priority_check, _run_calibration_performance_check, _run_prediction_drift_check,
     _run_calibration_improvement_check,
@@ -2318,6 +2319,15 @@ def build_parser() -> argparse.ArgumentParser:
         "--format", choices=["text", "json"], default="text"
     )
 
+    reviewer_questionnaire_parser = sub.add_parser(
+        "reviewer-questionnaire-check",
+        help="Validate a ReviewerQuestionnaire (RVQ-).",
+    )
+    reviewer_questionnaire_parser.add_argument("--entry-json", required=True)
+    reviewer_questionnaire_parser.add_argument(
+        "--format", choices=["text", "json"], default="text"
+    )
+
     # ── Hypothesis outcome check (Phase N N2) ────────────────────────
     hypothesis_outcome_parser = sub.add_parser(
         "hypothesis-outcome-check",
@@ -2824,6 +2834,9 @@ def main(argv: list[str] | None = None) -> int:
     if args.command == "failed-candidate-batch-report-check":
         return _run_failed_candidate_batch_report_check(args)
 
+    if args.command == "reviewer-questionnaire-check":
+        _run_reviewer_questionnaire_check(args)
+
     if args.command == "hypothesis-outcome-check":
         return _run_hypothesis_outcome_check(args)
 

src/openamp_foundry/evidence/reviewer_questionnaire.py

@@ -0,0 +1,206 @@
+"""Reviewer questionnaire schema — Phase E E3.
+
+Structured schema for external reviewer feedback on a PilotEvidencePackage
+(PEP). Captures Likert-scale clarity ratings, a synthesis recommendation,
+and structured comments. Makes external review machine-readable and enables
+systematic improvement of the evidence package quality.
+"""
+
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass, field
+from typing import List
+
+RVQ_PREFIX = "RVQ-"
+PEP_PREFIX = "PEP-"
+
+VALID_RECOMMENDATIONS = frozenset({
+    "yes",
+    "no",
+    "conditional",
+    "insufficient_information",
+})
+
+LIKERT_MIN = 1
+LIKERT_MAX = 5
+COMMENTS_MAX_LENGTH = 600
+LOW_QUALITY_THRESHOLD = 3
+
+
+@dataclass
+class ReviewerQuestionnaire:
+    """Structured external reviewer feedback on a PilotEvidencePackage."""
+
+    rvq_id: str
+    pipeline_version: str
+    pep_id: str  # must start with "PEP-"
+    reviewer_token: str  # anonymized reviewer ID, no PII
+    review_date: str  # ISO YYYY-MM-DD
+    activity_prediction_clarity: int  # Likert 1-5
+    safety_claim_clarity: int  # Likert 1-5
+    novelty_claim_clarity: int  # Likert 1-5
+    overall_package_quality: int  # Likert 1-5
+    would_recommend_for_synthesis: str  # {yes, no, conditional, insufficient_information}
+    missing_information: List[str]  # free-form list of gaps identified
+    reviewer_comments: str  # max 600 chars
+    dry_lab_only: bool = True
+
+
+@dataclass
+class ReviewerQuestionnaireResult:
+    rvq_id: str
+    pep_id: str
+    overall_package_quality: int
+    would_recommend_for_synthesis: str
+    passed: bool
+    errors: List[str] = field(default_factory=list)
+    warnings: List[str] = field(default_factory=list)
+    dry_lab_only: bool = True
+
+
+def validate_reviewer_questionnaire(
+    entry: ReviewerQuestionnaire,
+) -> ReviewerQuestionnaireResult:
+    errors: List[str] = []
+    warnings: List[str] = []
+
+    # Rule 1: RVQ- prefix
+    if not entry.rvq_id.startswith(RVQ_PREFIX):
+        errors.append(
+            f"rvq_id must start with '{RVQ_PREFIX}', got '{entry.rvq_id}'"
+        )
+
+    # Rule 2: PEP- prefix
+    if not entry.pep_id.startswith(PEP_PREFIX):
+        errors.append(
+            f"pep_id must start with '{PEP_PREFIX}', got '{entry.pep_id}'"
+        )
+
+    # Rule 3: non-empty required strings
+    for fname, val in [
+        ("pipeline_version", entry.pipeline_version),
+        ("reviewer_token", entry.reviewer_token),
+    ]:
+        if not val or not val.strip():
+            errors.append(f"{fname} must not be empty")
+
+    # Rule 4: ISO date
+    if not re.match(r"^\d{4}-\d{2}-\d{2}$", entry.review_date):
+        errors.append(
+            f"review_date must be ISO format YYYY-MM-DD, got '{entry.review_date}'"
+        )
+
+    # Rule 5: activity_prediction_clarity in [1, 5]
+    if not (LIKERT_MIN <= entry.activity_prediction_clarity <= LIKERT_MAX):
+        errors.append(
+            f"activity_prediction_clarity must be in [{LIKERT_MIN}, {LIKERT_MAX}], "
+            f"got {entry.activity_prediction_clarity}"
+        )
+
+    # Rule 6: safety_claim_clarity in [1, 5]
+    if not (LIKERT_MIN <= entry.safety_claim_clarity <= LIKERT_MAX):
+        errors.append(
+            f"safety_claim_clarity must be in [{LIKERT_MIN}, {LIKERT_MAX}], "
+            f"got {entry.safety_claim_clarity}"
+        )
+
+    # Rule 7: novelty_claim_clarity in [1, 5]
+    if not (LIKERT_MIN <= entry.novelty_claim_clarity <= LIKERT_MAX):
+        errors.append(
+            f"novelty_claim_clarity must be in [{LIKERT_MIN}, {LIKERT_MAX}], "
+            f"got {entry.novelty_claim_clarity}"
+        )
+
+    # Rule 8: overall_package_quality in [1, 5]
+    if not (LIKERT_MIN <= entry.overall_package_quality <= LIKERT_MAX):
+        errors.append(
+            f"overall_package_quality must be in [{LIKERT_MIN}, {LIKERT_MAX}], "
+            f"got {entry.overall_package_quality}"
+        )
+
+    # Rule 9: valid recommendation
+    if entry.would_recommend_for_synthesis not in VALID_RECOMMENDATIONS:
+        errors.append(
+            f"would_recommend_for_synthesis must be one of "
+            f"{sorted(VALID_RECOMMENDATIONS)}, got '{entry.would_recommend_for_synthesis}'"
+        )
+
+    # Rule 10: reviewer_comments length
+    if len(entry.reviewer_comments) > COMMENTS_MAX_LENGTH:
+        errors.append(
+            f"reviewer_comments must be at most {COMMENTS_MAX_LENGTH} characters, "
+            f"got {len(entry.reviewer_comments)}"
+        )
+
+    # Warning 1: low overall quality score
+    if (
+        LIKERT_MIN <= entry.overall_package_quality < LOW_QUALITY_THRESHOLD
+        and len(errors) == 0
+    ):
+        warnings.append(
+            f"overall_package_quality is {entry.overall_package_quality}/5 — "
+            "package needs significant improvement before external sharing"
+        )
+
+    # Warning 2: conditional recommendation without comments
+    if (
+        entry.would_recommend_for_synthesis == "conditional"
+        and not entry.reviewer_comments.strip()
+    ):
+        warnings.append(
+            "would_recommend_for_synthesis is 'conditional' but reviewer_comments is empty"
+            " — document the conditions required"
+        )
+
+    # Warning 3: negative recommendation without missing_information
+    if (
+        entry.would_recommend_for_synthesis == "no"
+        and not entry.missing_information
+        and not entry.reviewer_comments.strip()
+    ):
+        warnings.append(
+            "would_recommend_for_synthesis is 'no' with no missing_information or comments"
+            " — document why synthesis is not recommended"
+        )
+
+    # Warning 4: insufficient_information without missing_information list
+    if (
+        entry.would_recommend_for_synthesis == "insufficient_information"
+        and not entry.missing_information
+    ):
+        warnings.append(
+            "would_recommend_for_synthesis is 'insufficient_information' but "
+            "missing_information is empty — list the specific gaps"
+        )
+
+    passed = len(errors) == 0
+    return ReviewerQuestionnaireResult(
+        rvq_id=entry.rvq_id,
+        pep_id=entry.pep_id,
+        overall_package_quality=entry.overall_package_quality,
+        would_recommend_for_synthesis=entry.would_recommend_for_synthesis,
+        passed=passed,
+        errors=errors,
+        warnings=warnings,
+        dry_lab_only=entry.dry_lab_only,
+    )
+
+
+def validate_reviewer_questionnaire_dict(data: dict) -> ReviewerQuestionnaireResult:
+    entry = ReviewerQuestionnaire(
+        rvq_id=data.get("rvq_id", ""),
+        pipeline_version=data.get("pipeline_version", ""),
+        pep_id=data.get("pep_id", ""),
+        reviewer_token=data.get("reviewer_token", ""),
+        review_date=data.get("review_date", ""),
+        activity_prediction_clarity=int(data.get("activity_prediction_clarity", 0)),
+        safety_claim_clarity=int(data.get("safety_claim_clarity", 0)),
+        novelty_claim_clarity=int(data.get("novelty_claim_clarity", 0)),
+        overall_package_quality=int(data.get("overall_package_quality", 0)),
+        would_recommend_for_synthesis=data.get("would_recommend_for_synthesis", ""),
+        missing_information=list(data.get("missing_information", [])),
+        reviewer_comments=data.get("reviewer_comments", ""),
+        dry_lab_only=bool(data.get("dry_lab_only", True)),
+    )
+    return validate_reviewer_questionnaire(entry)