diff --git a/Makefile b/Makefile index 66412671..84274797 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: help demo test lint ci clean bench-leakage bench-multi-negatives bench-baseline bench-hidden-active bench-cluster-split bench-expert-ablation bench-expert-ablation-500 bench-selectivity bench-feature-decomp bench-gate bench-easy-baseline bench-charge-matched bench-order-dependent bench-precision-at-k bench-per-family bench-simulation-ablation bench-simulation-ablation-within-amp simulation-gate bench-strategy-compare regenerate-all generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-pilot pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter recalibration-engine recalibration-engine-dry-run validate-policy-version generate-synthetic-lab-results bump-policy-version generate-review-packet failed-candidate-report safe-publication-filter classify-negative-informativeness validate-rejection-events negative-result-dashboard calibration-audit calibration-audit-example calibration-overfit-check result-quality-filter +.PHONY: help demo test lint ci clean bench-leakage bench-multi-negatives bench-baseline bench-hidden-active bench-cluster-split bench-expert-ablation bench-expert-ablation-500 bench-selectivity bench-feature-decomp bench-gate bench-easy-baseline bench-charge-matched bench-order-dependent bench-precision-at-k bench-per-family bench-simulation-ablation bench-simulation-ablation-within-amp simulation-gate bench-strategy-compare regenerate-all generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-pilot pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter recalibration-engine recalibration-engine-dry-run validate-policy-version generate-synthetic-lab-results bump-policy-version generate-review-packet failed-candidate-report safe-publication-filter classify-negative-informativeness validate-rejection-events negative-result-dashboard calibration-audit calibration-audit-example calibration-overfit-check result-quality-filter synthetic-result-policy-check PYTHON := $(shell [ -f .venv/bin/python ] && echo .venv/bin/python || echo python3) PYTEST := $(shell [ -f .venv/bin/pytest ] && echo .venv/bin/pytest || echo pytest) @@ -685,3 +685,10 @@ result-quality-filter: --results-json /tmp/rq_input.json \ --out-json /tmp/rq_output.json --out-md /tmp/rq_output.md @echo "Result quality filter done. See /tmp/rq_output.json" + +synthetic-result-policy-check: + echo '[{"candidate_id":"AMP001","current_level":2,"proposed_level":3,"evidence_source":"synthetic"},{"candidate_id":"AMP002","current_level":2,"proposed_level":2,"evidence_source":"synthetic"},{"candidate_id":"AMP003","current_level":2,"proposed_level":3,"evidence_source":"lab"}]' > /tmp/srp_input.json + .venv/bin/openamp-foundry synthetic-result-policy-check \ + --proposals-json /tmp/srp_input.json \ + --out-json /tmp/srp_output.json --out-md /tmp/srp_output.md + @echo "Synthetic result policy check done. See /tmp/srp_output.json" diff --git a/docs/evidence/METRICS_CURRENT.md b/docs/evidence/METRICS_CURRENT.md index 7b2c0385..e2141c29 100644 --- a/docs/evidence/METRICS_CURRENT.md +++ b/docs/evidence/METRICS_CURRENT.md @@ -5,7 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees > with this file, this file wins. Updated whenever benchmark/benchmark config changes. > -> **Last updated:** 2026-07-09 (Phase G G7 — result-quality flag propagation — v0.5.85) +> **Last updated:** 2026-07-09 (Phase G G8 — synthetic-result policy — v0.5.86) +> **New in v0.5.86:** Synthetic-result policy enforcement (G8) — `check_synthetic_result_policy()` and `run_policy_batch()` enforce that synthetic/simulation outputs cannot raise the proof-ladder level of a candidate. Levels 4+ require wet-lab evidence; synthetic/unknown sources are rejected for such proposals. CLI (`openamp-foundry synthetic-result-policy-check`) accepts `--proposals-json`, `--out-json`, `--out-md`. Schema (`schemas/synthetic_result_policy_check.schema.json`). `make synthetic-result-policy-check` target. 27 tests. 3049 total. Anti-overclaim safeguard. > **New in v0.5.85:** Result-quality flag propagation (G7) — `assess_result_quality()` and `filter_results_for_calibration()` propagate result-quality flags into the calibration engine. Candidates with contamination or assay interference are excluded; candidates with 2+ quality flags are excluded; candidates with 1 flag are included with caution; clean candidates are fully included. `QUALITY_FLAGS` dictionary (8 flags). `ResultQualityReport` dataclass. CLI (`openamp-foundry result-quality-filter`) accepts `--results-json`, `--out-json`, `--out-md`. JSON + Markdown output. Schema (`schemas/result_quality_report.schema.json`). `make result-quality-filter` target. 27 tests. 3022 total. > **New in v0.5.84:** Calibration-overfit warning (G6) — `check_cohort_overfit_risk()` and `run_overfit_check()` flag when a calibration cohort is too small relative to model parameters. Warns at three severity levels (critical/warning/caution/none). CLI (`openamp-foundry calibration-overfit-check`) accepts comma-separated cohort sizes, model params, n features. JSON + Markdown output. Schema (`schemas/calibration_overfit_check.schema.json`). `make calibration-overfit-check` target. 21 tests. 2995 total. > **New in v0.5.83:** Batch-2 selection rationale report (G5) — CLI (`openamp-foundry batch-rationale`) that generates a synthetic candidate pool, runs the batch-2 selector with configurable weights, and produces a per-candidate rationale report classifying each selected candidate into exploit / explore / diversity / combined roles. Reports weight configuration, role breakdown summary, per-candidate contributions (ensemble×weight, uncertainty×weight, diversity×weight), safety gate impact, and caveats. Schema (`schemas/batch_rationale_report.schema.json`). `make batch-rationale` target. 19 tests. 2974 total. @@ -77,7 +78,7 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak > **New in v0.5.31:** Added dipeptide-order features for sequence-order awareness. `dipeptide_order_score` achieves AUROC 0.7861 on AMP-vs-scrambled discrimination — the strongest order-dependent feature in the pipeline. Only 7/31 features survive scrambling (amphipathicity/helix-wheel + dipeptide). All composition features are purely position-independent (exactly 0.5000 AUROC on scrambled test). > **New in v0.5.30:** Easy baseline benchmark added — charge density alone (AUROC 0.8166) outperforms the full pipeline ensemble (0.7792) on AMP-vs-Swiss-Prot-decoy discrimination. Honest finding documented: expected because pipeline optimizes for safety, not raw discrimination. > **New in v0.5.29:** Expanded benchmark to 500 AMPs + 500 composition-matched decoys (n=1000). AUROC 0.7792 (CI₉₅: 0.7505–0.8065) confirms signal generalizes. Cluster-aware CI: 0.746–0.8102. Representative AUROC: 0.778. Standard benchmark (n=191) retained for backward comparison. -> **Pipeline version:** v0.5.85 (G7) +> **Pipeline version:** v0.5.86 (G8) > **Branch:** main --- diff --git a/docs/research/ROADMAP.md b/docs/research/ROADMAP.md index b35503df..d0ef98d4 100644 --- a/docs/research/ROADMAP.md +++ b/docs/research/ROADMAP.md @@ -1,5 +1,67 @@ # Roadmap +## v0.5.86 — Loop 86: Phase G G8 — Synthetic-Result Policy (Anti-Overclaim) ✓ (2026-07-09) + +`check_synthetic_result_policy()` and `run_policy_batch()` enforce that synthetic/ +simulation results cannot raise the proof-ladder level of a candidate. Levels 4+ +require wet-lab evidence; synthetic or unknown sources are blocked for such proposals. +CLI (`openamp-foundry synthetic-result-policy-check`). Schema +(`schemas/synthetic_result_policy_check.schema.json`). `make synthetic-result-policy-check` +target. 27 tests. 3049 total. Anti-overclaim safeguard. + +## v0.5.86 — Loop 86: Phase G G8 — Synthetic Result Policy — Anti-Overclaim ✓ (2026-07-09) + +`check_synthetic_result_policy()` enforces that synthetic/simulation outputs cannot +raise a candidate's proof-ladder level. Simulation outputs are anti-overclaim — +they must not be used as evidence to move a candidate up the proof ladder. + +Changes: +- `src/openamp_foundry/evidence/synthetic_result_policy.py` (G8) — Core module with + `PROOF_LADDER_LEVELS` dictionary (1–6 mapping to descriptions), + `SyntheticResultPolicyCheck` dataclass (8 fields: candidate_id, current_level, + proposed_level, evidence_source, policy_pass, violation, recommendation, + dry_lab_only), `check_synthetic_result_policy()` applying multi-tier rules + (synthetic cannot raise, synthetic cannot lower, levels 4+ require wet-lab + evidence, invalid level raises ValueError), `run_policy_batch()` aggregating + results with summary counts and any_violation flag, + `write_policy_check_json()` and `write_policy_check_markdown()` for output. +- `schemas/synthetic_result_policy_check.schema.json` (G8) — JSON Schema Draft 07 + for single or batch policy check results. Validates all 8 required fields + including evidence_source enum constraint and dry_lab_only const=true. +- `src/openamp_foundry/evidence/__init__.py` — Exports all synthetic result policy + symbols. +- `src/openamp_foundry/cli/commands/reports.py` — Added + `_run_synthetic_result_policy_check()` CLI handler with `--proposals-json`, + `--out-json`, `--out-md` flags. +- `src/openamp_foundry/cli/main.py` — Registered `synthetic-result-policy-check` + subcommand with all argument flags and dispatch to handler. +- `Makefile` — Added `synthetic-result-policy-check` target with default example + data writing to `/tmp/srp_output.json` and `/tmp/srp_output.md`. Added to + `.PHONY`. +- `tests/evidence/test_synthetic_result_policy.py` — 27 tests covering: synthetic + raising level, synthetic maintaining level, synthetic lowering level, lab raising + level, literature raising level, proposed_level > 3 with synthetic/unknown source + violation, proposed_level > 3 with lab/literature pass, invalid current_level, + invalid proposed_level, unknown source normalization, unknown source + level 4 + violation, dry_lab_only always True, run_policy_batch summary counts, all-pass + batch, any_violation flag, to_dict output, PROOF_LADDER_LEVELS completeness, + JSON writer (single + batch), Markdown writer (single + batch), recommendation + non-empty for violation, recommendation for pass, empty batch list. +- `docs/evidence/METRICS_CURRENT.md` — v0.5.86 G8 changelog. Test count: 3049. +- `tests/test_test_count_regression.py` — baseline updated to 3049. + +Honest boundaries: +- This policy check validates evidence-source discipline, not biological truth. + A PASS does not confirm biological activity or safety. +- Synthetic evidence can still be useful for negative-result documentation and + exploratory research — the policy restricts proof-ladder movement, not usage. +- Level 4+ wet-lab evidence requirement is a policy rule, not a biological + guarantee. Wet-lab evidence can be wrong, inconclusive, or non-reproducible. +- The evidence_source classification relies on the submitter's honest labeling. + A "lab" source may still be noisy or erroneous. +- All proof-ladder determinations require qualified human review regardless of + policy check results. + ## v0.5.85 — Loop 85: Phase G G7 — Result-Quality Flag Propagation into Calibration Engine ✓ (2026-07-09) `assess_result_quality()` and `filter_results_for_calibration()` propagate diff --git a/schemas/synthetic_result_policy_check.schema.json b/schemas/synthetic_result_policy_check.schema.json new file mode 100644 index 00000000..306b6258 --- /dev/null +++ b/schemas/synthetic_result_policy_check.schema.json @@ -0,0 +1,58 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "$id": "https://openamp.ai/schemas/synthetic_result_policy_check.schema.json", + "title": "Synthetic Result Policy Check", + "description": "Schema for a synthetic result policy check result. Dry-lab only — does not measure biological activity.", + "type": "object", + "properties": { + "candidate_id": { + "type": "string", + "description": "Identifier for the candidate peptide." + }, + "current_level": { + "type": "integer", + "minimum": 1, + "maximum": 6, + "description": "Current proof-ladder level of the candidate." + }, + "proposed_level": { + "type": "integer", + "minimum": 1, + "maximum": 6, + "description": "Proposed proof-ladder level for the candidate." + }, + "evidence_source": { + "type": "string", + "enum": ["synthetic", "lab", "literature", "unknown"], + "description": "Source of the evidence for the proposed level." + }, + "policy_pass": { + "type": "boolean", + "description": "Whether the proposal passes the synthetic result policy." + }, + "violation": { + "type": "string", + "description": "Description of the policy violation, if any." + }, + "recommendation": { + "type": "string", + "description": "Human-readable recommendation." + }, + "dry_lab_only": { + "type": "boolean", + "description": "Always true. This is a computational assessment, not biological proof.", + "const": true + } + }, + "required": [ + "candidate_id", + "current_level", + "proposed_level", + "evidence_source", + "policy_pass", + "violation", + "recommendation", + "dry_lab_only" + ], + "additionalProperties": false +} diff --git a/src/openamp_foundry/cli/commands/reports.py b/src/openamp_foundry/cli/commands/reports.py index 75e7eea3..da981a91 100644 --- a/src/openamp_foundry/cli/commands/reports.py +++ b/src/openamp_foundry/cli/commands/reports.py @@ -1032,6 +1032,54 @@ def _run_calibration_overfit_check(args: argparse.Namespace) -> int: return 0 +def _run_synthetic_result_policy_check(args: argparse.Namespace) -> int: + """Check whether synthetic/simulation results raise proof-ladder level.""" + from pathlib import Path + from openamp_foundry.evidence.synthetic_result_policy import ( + check_synthetic_result_policy, + run_policy_batch, + write_policy_check_json, + write_policy_check_markdown, + ) + + proposals_path = Path(args.proposals_json) + if not proposals_path.exists(): + print(json.dumps({ + "status": "error", + "error": f"proposals JSON not found: {proposals_path}", + })) + return 2 + + with proposals_path.open() as f: + proposals = json.load(f) + + if not isinstance(proposals, list): + print(json.dumps({ + "status": "error", + "error": "proposals JSON must be a list of {candidate_id, current_level, proposed_level, evidence_source} objects", + })) + return 2 + + result = run_policy_batch(proposals) + + if args.out_json: + write_policy_check_json(result, args.out_json) + if args.out_md: + write_policy_check_markdown(result, args.out_md) + + print(json.dumps({ + "status": "ok", + "total": result["summary"]["total"], + "passed": result["summary"]["passed"], + "failed": result["summary"]["failed"], + "any_violation": result["any_violation"], + "dry_lab_only": result["dry_lab_only"], + "out_json": args.out_json, + "out_md": args.out_md, + }, indent=2)) + return 3 if result["any_violation"] else 0 + + def _run_result_quality_filter(args: argparse.Namespace) -> int: """Filter lab results by quality flags for calibration eligibility.""" import json diff --git a/src/openamp_foundry/cli/main.py b/src/openamp_foundry/cli/main.py index 29ed10e4..ae47c745 100644 --- a/src/openamp_foundry/cli/main.py +++ b/src/openamp_foundry/cli/main.py @@ -5,7 +5,7 @@ from openamp_foundry.cli.commands.selection import _run_pilot_panel, _run_pilot_confident, _run_diversity_check, _run_select_batch, _run_batch_rationale from openamp_foundry.cli.commands.external import _run_external_predict, _run_external_consensus from openamp_foundry.cli.commands.qc import _run_synthesis_order, _run_presynth_qc -from openamp_foundry.cli.commands.reports import _run_reviewer_questionnaire, _run_ip_report, _run_batch_pack, _run_gold_standard, _run_novelty_check_broad, _run_lab_result_report, _run_calibration_intake, _run_recalibration_gate, _run_recalibration_engine, _run_validate_policy_version, _run_calibration_audit, _run_calibration_overfit_check, _run_result_quality_filter +from openamp_foundry.cli.commands.reports import _run_reviewer_questionnaire, _run_ip_report, _run_batch_pack, _run_gold_standard, _run_novelty_check_broad, _run_lab_result_report, _run_calibration_intake, _run_recalibration_gate, _run_recalibration_engine, _run_validate_policy_version, _run_calibration_audit, _run_calibration_overfit_check, _run_result_quality_filter, _run_synthetic_result_policy_check from openamp_foundry.cli.commands.gates import _run_gate_check import argparse @@ -1280,6 +1280,34 @@ def build_parser() -> argparse.ArgumentParser: help="Optional output path for markdown report.", ) + # ── Synthetic result policy check ────────────────────────────── + srp = sub.add_parser( + "synthetic-result-policy-check", + help=( + "Check whether synthetic/simulation results are used to raise " + "a candidate's proof-ladder level. Anti-overclaim safeguard. " + "Dry-lab only." + ), + ) + srp.add_argument( + "--proposals-json", + required=True, + help=( + "Path to JSON file with list of {candidate_id, current_level, " + "proposed_level, evidence_source} objects." + ), + ) + srp.add_argument( + "--out-json", + default=None, + help="Optional output path for policy check JSON.", + ) + srp.add_argument( + "--out-md", + default=None, + help="Optional output path for policy check markdown report.", + ) + return parser @@ -1426,6 +1454,9 @@ def main(argv: list[str] | None = None) -> int: if args.command == "result-quality-filter": return _run_result_quality_filter(args) + if args.command == "synthetic-result-policy-check": + return _run_synthetic_result_policy_check(args) + parser.error("unknown command") return 2 diff --git a/src/openamp_foundry/evidence/__init__.py b/src/openamp_foundry/evidence/__init__.py index fb25e631..dba7bd88 100644 --- a/src/openamp_foundry/evidence/__init__.py +++ b/src/openamp_foundry/evidence/__init__.py @@ -2,12 +2,30 @@ Builds machine-readable candidate evidence certificates and validates them against `schemas/candidate.schema.json`. + +Includes synthetic result policy (Phase G G8) which enforces that +synthetic/simulation outputs cannot raise the proof-ladder level +of a candidate — anti-overclaim safeguard. """ from openamp_foundry.evidence.certificate import build_certificate from openamp_foundry.evidence.schemas import validate_json_schema +from openamp_foundry.evidence.synthetic_result_policy import ( + PROOF_LADDER_LEVELS, + SyntheticResultPolicyCheck, + check_synthetic_result_policy, + run_policy_batch, + write_policy_check_json, + write_policy_check_markdown, +) __all__ = [ "build_certificate", "validate_json_schema", + "PROOF_LADDER_LEVELS", + "SyntheticResultPolicyCheck", + "check_synthetic_result_policy", + "run_policy_batch", + "write_policy_check_json", + "write_policy_check_markdown", ] diff --git a/src/openamp_foundry/evidence/synthetic_result_policy.py b/src/openamp_foundry/evidence/synthetic_result_policy.py new file mode 100644 index 00000000..276f43c9 --- /dev/null +++ b/src/openamp_foundry/evidence/synthetic_result_policy.py @@ -0,0 +1,243 @@ +"""Synthetic result policy — enforces that synthetic/simulation outputs +cannot raise the proof-ladder level of a candidate. + +Dry-lab only. This is a computational policy check, not biological validation. +""" + +from __future__ import annotations + +import json +from dataclasses import dataclass, field, asdict +from pathlib import Path +from typing import Any + +PROOF_LADDER_LEVELS: dict[int, str] = { + 1: "computational nomination", + 2: "virtual-assay support", + 3: "in-silico ensemble agreement", + 4: "ex-vivo preliminary", + 5: "in-vivo preliminary", + 6: "clinical evidence", +} + + +@dataclass +class SyntheticResultPolicyCheck: + candidate_id: str + current_level: int + proposed_level: int + evidence_source: str + policy_pass: bool + violation: str = "" + recommendation: str = "" + dry_lab_only: bool = True + + def to_dict(self) -> dict[str, Any]: + return asdict(self) + + +def check_synthetic_result_policy( + candidate_id: str, + current_level: int, + proposed_level: int, + evidence_source: str, +) -> SyntheticResultPolicyCheck: + if current_level not in PROOF_LADDER_LEVELS: + raise ValueError( + f"current_level must be 1..6, got {current_level}" + ) + if proposed_level not in PROOF_LADDER_LEVELS: + raise ValueError( + f"proposed_level must be 1..6, got {proposed_level}" + ) + + valid_sources = {"synthetic", "lab", "literature", "unknown"} + if evidence_source not in valid_sources: + evidence_source = "unknown" + + violation = "" + policy_pass = True + recommendation = "" + + if evidence_source == "synthetic": + if proposed_level > current_level: + policy_pass = False + violation = "Synthetic evidence cannot raise proof-ladder level" + recommendation = ( + f"Candidate {candidate_id} proposed at proof-ladder level " + f"{proposed_level} ({PROOF_LADDER_LEVELS[proposed_level]}) " + f"supported only by synthetic evidence. " + f"Level {current_level} ({PROOF_LADDER_LEVELS[current_level]}) " + f"is the maximum allowed without wet-lab evidence." + ) + elif proposed_level < current_level: + policy_pass = False + violation = ( + "Synthetic evidence cannot lower proof-ladder level either; " + "use lab evidence to reassess" + ) + recommendation = ( + f"Candidate {candidate_id} synthetic evidence suggests level " + f"{proposed_level} but non-synthetic evidence supports level " + f"{current_level}. Review lab and literature evidence to " + f"determine the correct level." + ) + else: + recommendation = ( + f"Candidate {candidate_id} maintained at proof-ladder level " + f"{current_level} ({PROOF_LADDER_LEVELS[current_level]}) " + f"with synthetic evidence. This is acceptable." + ) + else: + recommendation = ( + f"Candidate {candidate_id} evidence source '{evidence_source}' " + f"is not restricted by synthetic result policy." + ) + + if proposed_level > 3 and evidence_source in ("synthetic", "unknown"): + policy_pass = False + violation = ( + "Levels 4+ require wet-lab evidence; " + "synthetic/unknown source is insufficient" + ) + recommendation = ( + f"Candidate {candidate_id} proposed at level {proposed_level} " + f"({PROOF_LADDER_LEVELS[proposed_level]}) which requires " + f"wet-lab evidence. Source '{evidence_source}' is insufficient. " + f"Provide lab or literature evidence." + ) + + return SyntheticResultPolicyCheck( + candidate_id=candidate_id, + current_level=current_level, + proposed_level=proposed_level, + evidence_source=evidence_source, + policy_pass=policy_pass, + violation=violation, + recommendation=recommendation, + dry_lab_only=True, + ) + + +def run_policy_batch(proposals: list[dict]) -> dict: + checks: list[SyntheticResultPolicyCheck] = [] + for prop in proposals: + check = check_synthetic_result_policy( + candidate_id=prop["candidate_id"], + current_level=prop["current_level"], + proposed_level=prop["proposed_level"], + evidence_source=prop["evidence_source"], + ) + checks.append(check) + + passed = sum(1 for c in checks if c.policy_pass) + failed = sum(1 for c in checks if not c.policy_pass) + + return { + "checks": [c.to_dict() for c in checks], + "summary": { + "total": len(checks), + "passed": passed, + "failed": failed, + }, + "any_violation": failed > 0, + "dry_lab_only": True, + } + + +def write_policy_check_json( + result: dict | SyntheticResultPolicyCheck, + path: str | Path, +) -> None: + path = Path(path) + if isinstance(result, SyntheticResultPolicyCheck): + data = result.to_dict() + else: + data = result + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(data, indent=2) + "\n", encoding="utf-8") + + +def write_policy_check_markdown( + result: dict | SyntheticResultPolicyCheck, + path: str | Path, +) -> None: + path = Path(path) + path.parent.mkdir(parents=True, exist_ok=True) + + if isinstance(result, SyntheticResultPolicyCheck): + data = result.to_dict() + lines = _single_check_to_markdown(data) + else: + lines = _batch_report_to_markdown(result) + + path.write_text("\n".join(lines) + "\n", encoding="utf-8") + + +def _single_check_to_markdown(data: dict) -> list[str]: + status = "✅ PASS" if data["policy_pass"] else "❌ FAIL" + lines = [ + "# Synthetic Result Policy Check", + "", + f"**Candidate:** {data['candidate_id']}", + f"**Status:** {status}", + f"**Current level:** {data['current_level']}", + f"**Proposed level:** {data['proposed_level']}", + f"**Evidence source:** {data['evidence_source']}", + "", + ] + if data["violation"]: + lines.append(f"**Violation:** {data['violation']}") + lines.append("") + if data["recommendation"]: + lines.append(f"**Recommendation:** {data['recommendation']}") + lines.append("") + lines.append("---") + lines.append( + "_This is a computational policy check and does not measure " + "biological activity._" + ) + return lines + + +def _batch_report_to_markdown(data: dict) -> list[str]: + summary = data.get("summary", {}) + n_total = summary.get("total", 0) + n_passed = summary.get("passed", 0) + n_failed = summary.get("failed", 0) + + lines = [ + "# Synthetic Result Policy Check — Batch Report", + "", + "## Summary", + "", + f"| Metric | Value |", + f"|--------|-------|", + f"| Total checks | {n_total} |", + f"| Passed | {n_passed} |", + f"| Failed | {n_failed} |", + f"| Any violation | {data.get('any_violation', False)} |", + f"| Dry-lab only | {data.get('dry_lab_only', True)} |", + "", + "## Per-Candidate Results", + "", + "| Candidate | Current | Proposed | Source | Pass | Violation |", + "|-----------|:-------:|:--------:|:------:|:---:|-----------|", + ] + + for c in data.get("checks", []): + status_icon = "✅" if c["policy_pass"] else "❌" + violation = c.get("violation", "") or "—" + lines.append( + f"| {c['candidate_id']} | {c['current_level']} " + f"| {c['proposed_level']} | {c['evidence_source']} " + f"| {status_icon} | {violation} |" + ) + + lines.extend([ + "", + "---", + "_This is a computational policy check and does not measure " + "biological activity._", + ]) + return lines diff --git a/tests/evidence/test_synthetic_result_policy.py b/tests/evidence/test_synthetic_result_policy.py new file mode 100644 index 00000000..73cb2dda --- /dev/null +++ b/tests/evidence/test_synthetic_result_policy.py @@ -0,0 +1,311 @@ +"""Tests for synthetic result policy — anti-overclaim safeguard.""" + +from __future__ import annotations + +import json +import tempfile +from pathlib import Path + +import pytest + +from openamp_foundry.evidence.synthetic_result_policy import ( + PROOF_LADDER_LEVELS, + SyntheticResultPolicyCheck, + check_synthetic_result_policy, + run_policy_batch, + write_policy_check_json, + write_policy_check_markdown, +) + + +class TestSyntheticResultPolicy: + def test_synthetic_raising_level_violation(self): + result = check_synthetic_result_policy( + candidate_id="AMP001", + current_level=2, + proposed_level=3, + evidence_source="synthetic", + ) + assert result.policy_pass is False + assert result.violation == "Synthetic evidence cannot raise proof-ladder level" + assert result.dry_lab_only is True + + def test_synthetic_maintaining_level_pass(self): + result = check_synthetic_result_policy( + candidate_id="AMP002", + current_level=2, + proposed_level=2, + evidence_source="synthetic", + ) + assert result.policy_pass is True + assert result.violation == "" + assert result.dry_lab_only is True + + def test_synthetic_lowering_level_violation(self): + result = check_synthetic_result_policy( + candidate_id="AMP003", + current_level=3, + proposed_level=2, + evidence_source="synthetic", + ) + assert result.policy_pass is False + assert "cannot lower" in result.violation + assert result.dry_lab_only is True + + def test_lab_raising_level_pass(self): + result = check_synthetic_result_policy( + candidate_id="AMP004", + current_level=2, + proposed_level=3, + evidence_source="lab", + ) + assert result.policy_pass is True + assert result.violation == "" + assert result.dry_lab_only is True + + def test_literature_raising_level_pass(self): + result = check_synthetic_result_policy( + candidate_id="AMP005", + current_level=1, + proposed_level=2, + evidence_source="literature", + ) + assert result.policy_pass is True + assert result.violation == "" + + def test_proposed_level_4_with_synthetic_violation(self): + result = check_synthetic_result_policy( + candidate_id="AMP006", + current_level=3, + proposed_level=4, + evidence_source="synthetic", + ) + assert result.policy_pass is False + assert "Levels 4+ require wet-lab evidence" in result.violation + + def test_proposed_level_4_with_unknown_violation(self): + result = check_synthetic_result_policy( + candidate_id="AMP007", + current_level=3, + proposed_level=4, + evidence_source="unknown", + ) + assert result.policy_pass is False + assert "Levels 4+ require wet-lab evidence" in result.violation + + def test_proposed_level_4_with_lab_pass(self): + result = check_synthetic_result_policy( + candidate_id="AMP008", + current_level=3, + proposed_level=4, + evidence_source="lab", + ) + assert result.policy_pass is True + assert result.violation == "" + + def test_proposed_level_5_with_lab_pass(self): + result = check_synthetic_result_policy( + candidate_id="AMP009", + current_level=4, + proposed_level=5, + evidence_source="lab", + ) + assert result.policy_pass is True + assert result.violation == "" + + def test_proposed_level_6_with_literature_pass(self): + result = check_synthetic_result_policy( + candidate_id="AMP010", + current_level=5, + proposed_level=6, + evidence_source="literature", + ) + assert result.policy_pass is True + assert result.violation == "" + + def test_invalid_current_level_raises(self): + with pytest.raises(ValueError, match="current_level must be 1..6"): + check_synthetic_result_policy( + candidate_id="AMP011", + current_level=0, + proposed_level=3, + evidence_source="synthetic", + ) + + def test_invalid_proposed_level_raises(self): + with pytest.raises(ValueError, match="proposed_level must be 1..6"): + check_synthetic_result_policy( + candidate_id="AMP012", + current_level=2, + proposed_level=7, + evidence_source="synthetic", + ) + + def test_unknown_source_treated_as_unknown(self): + result = check_synthetic_result_policy( + candidate_id="AMP013", + current_level=2, + proposed_level=3, + evidence_source="invalid_source", + ) + # invalid source becomes "unknown", and unknown + proposing > 3 + # is a violation, but here proposed=3 so no level-4 violation + assert result.evidence_source == "unknown" + assert result.policy_pass is True # unknown + proposed=3 is fine + assert result.violation == "" + + def test_unknown_source_proposing_level_4_violation(self): + result = check_synthetic_result_policy( + candidate_id="AMP014", + current_level=3, + proposed_level=4, + evidence_source="invalid_source", + ) + assert result.evidence_source == "unknown" + assert result.policy_pass is False + assert "Levels 4+ require wet-lab evidence" in result.violation + + def test_dry_lab_only_always_true(self): + sources = ["synthetic", "lab", "literature", "unknown"] + for src in sources: + result = check_synthetic_result_policy( + candidate_id="T001", current_level=1, proposed_level=2, + evidence_source=src, + ) + assert result.dry_lab_only is True + + def test_run_policy_batch_summary_counts(self): + proposals = [ + {"candidate_id": "A", "current_level": 2, "proposed_level": 3, "evidence_source": "synthetic"}, + {"candidate_id": "B", "current_level": 2, "proposed_level": 2, "evidence_source": "synthetic"}, + {"candidate_id": "C", "current_level": 2, "proposed_level": 3, "evidence_source": "lab"}, + ] + result = run_policy_batch(proposals) + assert result["summary"]["total"] == 3 + assert result["summary"]["passed"] == 2 + assert result["summary"]["failed"] == 1 + assert result["any_violation"] is True + assert result["dry_lab_only"] is True + + def test_run_policy_batch_all_pass(self): + proposals = [ + {"candidate_id": "A", "current_level": 2, "proposed_level": 2, "evidence_source": "lab"}, + {"candidate_id": "B", "current_level": 3, "proposed_level": 4, "evidence_source": "lab"}, + ] + result = run_policy_batch(proposals) + assert result["summary"]["total"] == 2 + assert result["summary"]["passed"] == 2 + assert result["summary"]["failed"] == 0 + assert result["any_violation"] is False + + def test_run_policy_batch_any_violation_flag(self): + proposals = [ + {"candidate_id": "A", "current_level": 2, "proposed_level": 3, "evidence_source": "synthetic"}, + ] + result = run_policy_batch(proposals) + assert result["any_violation"] is True + + def test_to_dict_method(self): + check = SyntheticResultPolicyCheck( + candidate_id="T001", + current_level=2, + proposed_level=3, + evidence_source="synthetic", + policy_pass=False, + violation="test violation", + recommendation="test", + ) + d = check.to_dict() + assert d["candidate_id"] == "T001" + assert d["policy_pass"] is False + assert d["dry_lab_only"] is True + + def test_proof_ladder_levels_dict(self): + assert PROOF_LADDER_LEVELS[1] == "computational nomination" + assert PROOF_LADDER_LEVELS[3] == "in-silico ensemble agreement" + assert PROOF_LADDER_LEVELS[6] == "clinical evidence" + assert len(PROOF_LADDER_LEVELS) == 6 + + def test_write_policy_check_json_single(self): + check = check_synthetic_result_policy( + candidate_id="AMP001", + current_level=2, + proposed_level=3, + evidence_source="synthetic", + ) + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / "check.json" + write_policy_check_json(check, path) + assert path.exists() + data = json.loads(path.read_text()) + assert data["candidate_id"] == "AMP001" + assert data["policy_pass"] is False + + def test_write_policy_check_json_batch(self): + proposals = [ + {"candidate_id": "A", "current_level": 2, "proposed_level": 3, "evidence_source": "synthetic"}, + ] + result = run_policy_batch(proposals) + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / "batch.json" + write_policy_check_json(result, path) + assert path.exists() + data = json.loads(path.read_text()) + assert data["summary"]["total"] == 1 + + def test_write_policy_check_markdown_single(self): + check = check_synthetic_result_policy( + candidate_id="AMP001", + current_level=2, + proposed_level=3, + evidence_source="synthetic", + ) + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / "check.md" + write_policy_check_markdown(check, path) + assert path.exists() + content = path.read_text() + assert "Synthetic Result Policy Check" in content + assert "AMP001" in content + assert "FAIL" in content + + def test_write_policy_check_markdown_batch(self): + proposals = [ + {"candidate_id": "A", "current_level": 2, "proposed_level": 3, "evidence_source": "synthetic"}, + {"candidate_id": "B", "current_level": 2, "proposed_level": 2, "evidence_source": "lab"}, + ] + result = run_policy_batch(proposals) + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / "batch.md" + write_policy_check_markdown(result, path) + assert path.exists() + content = path.read_text() + assert "Batch Report" in content + assert "A" in content + assert "B" in content + + def test_recommendation_non_empty_for_violation(self): + result = check_synthetic_result_policy( + candidate_id="AMP001", + current_level=2, + proposed_level=3, + evidence_source="synthetic", + ) + assert result.recommendation != "" + + def test_recommendation_for_pass(self): + result = check_synthetic_result_policy( + candidate_id="AMP001", + current_level=2, + proposed_level=2, + evidence_source="synthetic", + ) + assert result.recommendation != "" + assert "maintained" in result.recommendation + + def test_run_policy_batch_empty_list(self): + result = run_policy_batch([]) + assert result["summary"]["total"] == 0 + assert result["summary"]["passed"] == 0 + assert result["summary"]["failed"] == 0 + assert result["any_violation"] is False diff --git a/tests/test_test_count_regression.py b/tests/test_test_count_regression.py index 80c07da4..11ecfb3b 100644 --- a/tests/test_test_count_regression.py +++ b/tests/test_test_count_regression.py @@ -2,9 +2,9 @@ import subprocess import sys -# Baseline: 3022 tests as of Loop 85 (2026-07-09) -- Phase G G7 (result-quality flag propagation) +# Baseline: 3049 tests as of Loop 86 (2026-07-09) -- Phase G G8 (synthetic result policy) # Raise threshold if legitimate tests are removed; lower threshold is okay. -BASELINE = 3022 +BASELINE = 3049 TOLERANCE = 0.05 # allow 5% fluctuation