Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: help demo test lint ci clean bench-leakage bench-multi-negatives bench-baseline bench-hidden-active bench-cluster-split bench-expert-ablation bench-expert-ablation-500 bench-selectivity bench-feature-decomp bench-gate bench-easy-baseline bench-charge-matched bench-order-dependent bench-precision-at-k bench-per-family bench-simulation-ablation bench-simulation-ablation-within-amp simulation-gate regenerate-all generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-pilot pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter recalibration-engine recalibration-engine-dry-run validate-policy-version generate-synthetic-lab-results bump-policy-version generate-review-packet failed-candidate-report
.PHONY: help demo test lint ci clean bench-leakage bench-multi-negatives bench-baseline bench-hidden-active bench-cluster-split bench-expert-ablation bench-expert-ablation-500 bench-selectivity bench-feature-decomp bench-gate bench-easy-baseline bench-charge-matched bench-order-dependent bench-precision-at-k bench-per-family bench-simulation-ablation bench-simulation-ablation-within-amp simulation-gate regenerate-all generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-pilot pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter recalibration-engine recalibration-engine-dry-run validate-policy-version generate-synthetic-lab-results bump-policy-version generate-review-packet failed-candidate-report safe-publication-filter

PYTHON := $(shell [ -f .venv/bin/python ] && echo .venv/bin/python || echo python3)
PYTEST := $(shell [ -f .venv/bin/pytest ] && echo .venv/bin/pytest || echo pytest)
Expand Down Expand Up @@ -602,3 +602,9 @@ failed-candidate-report:
--out-json outputs/failed_candidate_report.json \
--out-md outputs/failed_candidate_report.md \
--validate-rejection-codes

safe-publication-filter:
PYTHONPATH=src $(PYTHON) scripts/safe_publication_filter.py \
--input examples/safe_publication_filter_example_input.json \
--out-json outputs/safe_publication_filter_result.json \
--out-md outputs/safe_publication_filter_result.md
6 changes: 4 additions & 2 deletions docs/evidence/METRICS_CURRENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
> with this file, this file wins. Updated whenever benchmark/benchmark config changes.
>
> **Last updated:** 2026-07-09 (Phase F report generator — v0.5.75)
> **Last updated:** 2026-07-09 (Phase F safe-publication filter — v0.5.76)
> **New in v0.5.76:** Safe-publication filter (F5) — CLI that reads candidate panel data with safety metadata and checks each candidate against publication safety constraints before external release. FAIL-CLOSED: defaults to reject unless all checks pass. Checks: dry_lab_only (must be true), proof_ladder_level (≤4), toxicity_screened, hemolysis_screened, dual_use_reviewed (all must be true). Produces structured JSON filter result and human-readable Markdown summary. 33 tests. 2758 total.
>
> **New in v0.5.75:** Failed-candidate report generator (F4) — CLI that reads candidate rejection data + taxonomy, produces structured JSON report with summary aggregation (by category, severity, stage, evidence impact) and human-readable Markdown summary. 26 tests. 2724 total.
>
> **New in v0.5.74:** Rejection reason taxonomy schema (F3) — machine-readable taxonomy with severity, evidence_impact, and applies_at_stage fields. Comprehensive REJECTION_TAXONOMY.md doc. 19 tests. 2698 total.
Expand Down Expand Up @@ -60,7 +62,7 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **New in v0.5.31:** Added dipeptide-order features for sequence-order awareness. `dipeptide_order_score` achieves AUROC 0.7861 on AMP-vs-scrambled discrimination — the strongest order-dependent feature in the pipeline. Only 7/31 features survive scrambling (amphipathicity/helix-wheel + dipeptide). All composition features are purely position-independent (exactly 0.5000 AUROC on scrambled test).
> **New in v0.5.30:** Easy baseline benchmark added — charge density alone (AUROC 0.8166) outperforms the full pipeline ensemble (0.7792) on AMP-vs-Swiss-Prot-decoy discrimination. Honest finding documented: expected because pipeline optimizes for safety, not raw discrimination.
> **New in v0.5.29:** Expanded benchmark to 500 AMPs + 500 composition-matched decoys (n=1000). AUROC 0.7792 (CI₉₅: 0.7505–0.8065) confirms signal generalizes. Cluster-aware CI: 0.746–0.8102. Representative AUROC: 0.778. Standard benchmark (n=191) retained for backward comparison.
> **Pipeline version:** v0.5.75
> **Pipeline version:** v0.5.76
> **Branch:** main

---
Expand Down
44 changes: 44 additions & 0 deletions docs/research/ROADMAP.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,49 @@
# Roadmap

## v0.5.76 — Loop 75: Phase F F5 — Safe-Publication Filter ✓ (2026-07-09)

Safe-publication filter: CLI that reads a candidate panel with safety metadata
and checks each candidate against publication safety constraints before external
release. FAIL-CLOSED: defaults to reject unless all checks pass.

Changes:
- `scripts/safe_publication_filter.py` (F5) — Standalone CLI that loads candidate
panel data with publication safety metadata and applies global-level checks
(dry_lab_only must be true, proof_ladder_level ≤ 4, toxicity_screened,
hemolysis_screened, dual_use_reviewed all true) plus per-candidate checks
(proof_ladder_level ≤ 4). Produces structured JSON with filter metadata,
global checks, per-candidate results, and summary. Exit code 0 if all
checks pass, 1 if any fail (FAIL-CLOSED), 2 for input errors.
- `schemas/safe_publication_filter.schema.json` (F5) — JSON Schema Draft 2020-12
for the filter result output. Validates filter_metadata, global_checks
(3 required sub-check groups), per-candidate results, summary counts, and
caveats.
- `examples/safe_publication_filter_example_input.json` (F5) — Toy input
with 4 candidates spanning PASS (pll=2, pll=1), WARN (pll=4 at boundary),
and FAIL (pll=5 too high). Clearly marked EXAMPLE.
- `Makefile` — Added `safe-publication-filter` target.
- `tests/evidence/test_safe_publication_filter.py` — 33 tests covering:
global check success/failure paths (dry_lab_only, proof_ladder_level,
all safety checks individually), per-candidate checks (valid, missing,
too-high, boundary/warn), mixed candidate results, filter result summary
counts, overall PASS/FAIL logic, markdown output sections, input loading
(valid/missing/invalid JSON), CLI exit codes (0, 1, 2), CLI output
(JSON + Markdown), caveat generation, metadata structure, and example
input round-trip validation.

Honest boundaries:
- The filter checks structural publication safety constraints, not scientific
validity. A PASS result does not mean the candidate is biologically active
or safe.
- dry_lab_only attestation is a safety-constraint field; it relies on the
generator's honesty.
- Proof ladder levels are self-reported by the pipeline; independent
verification is recommended.
- Safety checks (toxicity, hemolysis, dual-use) are computational assessments
and require qualified human review.
- A FAIL result does not mean the candidate is biologically inactive — it
means publication safety checks are not satisfied.

## v0.5.75 — Loop 74: Phase F F4 — Failed-Candidate Report Generator ✓ (2026-07-09)

Failed-candidate report generator: CLI that reads candidate rejection data validated
Expand Down
39 changes: 39 additions & 0 deletions examples/safe_publication_filter_example_input.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
{
"source": "wave0.5-example",
"pipeline_version": "v0.5.75",
"report_type": "safe_publication_input",
"description": "EXAMPLE safe-publication filter input. NOT a real candidate panel — illustrative only.",
"dry_lab_only": true,
"proof_ladder_level": 2,
"safety_checks": {
"toxicity_screened": true,
"hemolysis_screened": true,
"dual_use_reviewed": true
},
"candidates": [
{
"candidate_id": "EXAMPLE-CAND-001",
"sequence": "ACDEFGHIKLMNPQRSTVWY",
"ensemble_score": 0.85,
"proof_ladder_level": 2
},
{
"candidate_id": "EXAMPLE-CAND-002",
"sequence": "KLAKLAKKLAKLAK",
"ensemble_score": 0.80,
"proof_ladder_level": 4
},
{
"candidate_id": "EXAMPLE-CAND-003",
"sequence": "GIGKFLHSAKKWGKAFVGEIMNS",
"ensemble_score": 0.78,
"proof_ladder_level": 5
},
{
"candidate_id": "EXAMPLE-CAND-004",
"sequence": "RRWQWRMKKLG",
"ensemble_score": 0.72,
"proof_ladder_level": 1
}
]
}
163 changes: 163 additions & 0 deletions schemas/safe_publication_filter.schema.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://openamp-foundry.org/schemas/safe_publication_filter.schema.json",
"title": "OpenAMP Safe Publication Filter Result",
"description": "JSON Schema for the safe-publication filter result. Validates the output of scripts/safe_publication_filter.py before any external release. FAIL-CLOSED: the filter defaults to reject unless all publication safety checks pass.",
"type": "object",
"required": [
"filter_metadata",
"global_checks",
"global_pass",
"candidates",
"summary",
"caveats"
],
"properties": {
"filter_metadata": {
"type": "object",
"required": [
"generated_at",
"filter_type",
"filter_version",
"input_source",
"pipeline_version"
],
"properties": {
"generated_at": {
"type": "string",
"format": "date-time",
"description": "ISO 8601 timestamp when the filter was run"
},
"filter_type": {
"type": "string",
"enum": ["safe_publication_filter"],
"description": "Filter type identifier"
},
"filter_version": {
"type": "string",
"pattern": "^\\d+\\.\\d+\\.\\d+$",
"description": "Filter schema version (semver)"
},
"input_source": {
"type": "string",
"description": "Source identifier of the input data"
},
"pipeline_version": {
"type": "string",
"description": "OpenAMP pipeline version"
}
}
},
"global_checks": {
"type": "object",
"description": "Global-level publication safety checks applied to the entire batch",
"required": ["dry_lab_only", "global_proof_ladder_level", "safety_checks"],
"properties": {
"dry_lab_only": {
"type": "object",
"required": ["required", "actual", "passed", "description"],
"properties": {
"required": { "type": "boolean", "enum": [true] },
"actual": { "type": "boolean" },
"passed": { "type": "boolean" },
"description": { "type": "string" },
"reason": { "type": "string" }
}
},
"global_proof_ladder_level": {
"type": "object",
"required": ["max_allowed", "actual", "passed", "description"],
"properties": {
"required": { "type": "string" },
"max_allowed": { "type": "integer", "minimum": 1, "maximum": 8 },
"actual": { "type": "integer", "minimum": 1, "maximum": 8 },
"passed": { "type": "boolean" },
"description": { "type": "string" },
"reason": { "type": "string" }
}
},
"safety_checks": {
"type": "object",
"required": ["all_passed", "checks"],
"properties": {
"all_passed": { "type": "boolean" },
"checks": {
"type": "object",
"minProperties": 3,
"additionalProperties": {
"type": "object",
"required": ["required", "actual", "passed"],
"properties": {
"required": { "type": "boolean", "enum": [true] },
"actual": { "type": "boolean" },
"passed": { "type": "boolean" },
"reason": { "type": "string" }
}
}
}
}
}
}
},
"global_pass": {
"type": "boolean",
"description": "Whether all global-level checks passed. If false, the filter FAILS regardless of per-candidate results."
},
"candidates": {
"type": "array",
"items": {
"type": "object",
"required": ["candidate_id", "overall", "checks"],
"properties": {
"candidate_id": { "type": "string" },
"overall": {
"type": "string",
"enum": ["PASS", "FAIL", "WARN"],
"description": "PASS: all checks pass, FAIL: any check fails, WARN: passes but at boundary"
},
"checks": {
"type": "object",
"minProperties": 1,
"additionalProperties": {
"type": "object",
"required": ["value", "max_allowed", "passed"],
"properties": {
"value": { "type": ["integer", "null"] },
"max_allowed": { "type": "integer" },
"passed": { "type": "boolean" },
"reason": { "type": "string" }
}
}
},
"issues": {
"type": "array",
"items": { "type": "string" }
}
}
}
},
"summary": {
"type": "object",
"required": ["total_candidates", "passed", "failed", "overall_filter_pass"],
"properties": {
"total_candidates": { "type": "integer", "minimum": 0 },
"passed": { "type": "integer", "minimum": 0 },
"failed": { "type": "integer", "minimum": 0 },
"warned": { "type": "integer", "minimum": 0 },
"overall_filter_pass": {
"type": "boolean",
"description": "True only if ALL global checks pass AND NO candidates FAIL. FAIL-CLOSED."
}
}
},
"caveats": {
"type": "array",
"minItems": 1,
"items": { "type": "string" }
},
"dry_lab_only": {
"type": "boolean",
"description": "Copied from input — all outputs are dry-lab computational hypotheses, not biological proof."
}
}
}
Loading
Loading