Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: help demo test lint ci clean bench-leakage bench-multi-negatives bench-baseline bench-hidden-active bench-cluster-split bench-expert-ablation bench-expert-ablation-500 bench-selectivity bench-feature-decomp bench-gate bench-easy-baseline bench-charge-matched bench-order-dependent bench-precision-at-k bench-per-family bench-simulation-ablation bench-simulation-ablation-within-amp simulation-gate bench-strategy-compare regenerate-all generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-pilot pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter recalibration-engine recalibration-engine-dry-run validate-policy-version generate-synthetic-lab-results bump-policy-version generate-review-packet failed-candidate-report safe-publication-filter classify-negative-informativeness validate-rejection-events negative-result-dashboard calibration-audit calibration-audit-example calibration-overfit-check result-quality-filter synthetic-result-policy-check calibration-decision-checklist calibration-rollback-plan simulation-registry validate-simulation-result-schema simulation-baseline-check adapter-gate-check simulation-provenance simulation-ensemble-check simulation-ci-report simulation-deprecation-check simulation-scope-check simulation-evidence-packet artifact-version
.PHONY: help demo test lint ci clean bench-leakage bench-multi-negatives bench-baseline bench-hidden-active bench-cluster-split bench-expert-ablation bench-expert-ablation-500 bench-selectivity bench-feature-decomp bench-gate bench-easy-baseline bench-charge-matched bench-order-dependent bench-precision-at-k bench-per-family bench-simulation-ablation bench-simulation-ablation-within-amp simulation-gate bench-strategy-compare regenerate-all generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-pilot pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter recalibration-engine recalibration-engine-dry-run validate-policy-version generate-synthetic-lab-results bump-policy-version generate-review-packet failed-candidate-report safe-publication-filter classify-negative-informativeness validate-rejection-events negative-result-dashboard calibration-audit calibration-audit-example calibration-overfit-check result-quality-filter synthetic-result-policy-check calibration-decision-checklist calibration-rollback-plan simulation-registry validate-simulation-result-schema simulation-baseline-check adapter-gate-check simulation-provenance simulation-ensemble-check simulation-ci-report simulation-deprecation-check simulation-scope-check simulation-evidence-packet artifact-version candidate-manifest

PYTHON := $(shell [ -f .venv/bin/python ] && echo .venv/bin/python || echo python3)
PYTEST := $(shell [ -f .venv/bin/pytest ] && echo .venv/bin/pytest || echo pytest)
Expand Down Expand Up @@ -780,3 +780,15 @@ simulation-evidence-packet:
artifact-version:
.venv/bin/openamp-foundry artifact-version --list
@echo "Artifact version list complete."

candidate-manifest:
.venv/bin/openamp-foundry candidate-manifest \
--candidate-id AMP-001 \
--sequence AKLWKR \
--evidence-level 2 \
--scopes bacterial_binding \
--scores-json '{"binding_energy": 0.75}' \
--uncertainty 0.1 \
--source-modules membrane_proxy \
--validate
@echo "Candidate manifest complete."
5 changes: 3 additions & 2 deletions docs/evidence/METRICS_CURRENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
> with this file, this file wins. Updated whenever benchmark/benchmark config changes.
>
> **Last updated:** 2026-07-09 (Phase I I1 — artifact versioning policy — v0.5.99)
> **Last updated:** 2026-07-09 (Phase I I2 — candidate manifest schema — v0.6.0)
> **New in v0.6.0:** Candidate manifest schema (I2 — Loop 100) — `schemas/candidate_manifest.schema.json` (Draft 2020-12, 14 required fields: candidate_id, sequence, evidence_level, scopes, scores, uncertainty, source_modules, calibration_set, safety_flags, provenance_run_id, dry_lab_only, version, created_at, notes). `$schema`, `$id`, `title`, `description`, `additionalProperties: false`. `src/openamp_foundry/manifests/` module with `CandidateManifest` dataclass (14 fields), `make_candidate_manifest()`, `validate_candidate_manifest()` (8 checks: non-empty candidate_id, non-empty sequence, evidence_level 1-6, non-empty scopes, uncertainty 0.0-1.0, non-empty source_modules, dry_lab_only must be True, version MAJOR.MINOR.PATCH), `manifest_to_dict()`, `manifest_summary()` (total, by_evidence_level, with_safety_flags, dry_lab_only). CLI (`openamp-foundry candidate-manifest`) with `--candidate-id`, `--sequence`, `--evidence-level`, `--scopes`, `--scores-json`, `--uncertainty`, `--source-modules`, `--validate`, `--format text|json`. `make candidate-manifest` target. 19 tests. **3310 total.** A candidate manifest is the core interoperable artifact — it describes a dry-lab candidate (sequence, scores, evidence level, scopes, safety flags, provenance) in a machine-readable format that external tools can consume without the full OpenAMP stack.
> **New in v0.5.99:** Artifact versioning policy (I1 — starts Phase I) — `docs/engineering/ARTIFACT_VERSIONING_POLICY.md` with structured versioning policy covering scope, SemVer format, breaking/non-breaking change definitions, deprecation timeline (at least one minor version warning), schema `$id` policy, changelog requirement, and three stability tiers (stable/experimental/internal). `src/openamp_foundry/versioning/` module with `ArtifactVersionInfo` dataclass (7 fields), `STABILITY_TIERS` dict, `VERSIONED_ARTIFACTS` list (6 entries covering candidate, lab_result, run_manifest, external_review_packet, safety_release_decision, simulation_result at v1.0.0), `get_artifact_version()`, `list_versioned_artifacts()`, `validate_version_format()`, `artifact_version_summary()`. CLI (`openamp-foundry artifact-version`) with `--list`, `--show`, `--tier`, `--format text|json`. `make artifact-version` target. 19 tests. **3291 total.** Starts Phase I (interoperability) — external users now have stability guarantees for schemas, evidence certificates, and candidate manifests.
> **New in v0.5.97:** Simulation-scope coverage checker (H9) — `ScopeCoverageResult` dataclass (8 fields: module_id, requested_scopes, module_scopes, covered, uncovered, coverage_fraction, is_fully_covered, dry_lab_only). `check_scope_coverage()` looks up module in registry, computes covered (intersection) and uncovered (requested scopes not in module scopes), coverage_fraction = len(covered)/len(requested_scopes) if requested else 1.0. `check_result_scope()` uses conservative intersection of registry scopes and result.scope as effective module_scopes. `scope_coverage_report()` returns full dict. CLI (`openamp-foundry simulation-scope-check`) with `--module-id`, `--requested-scopes`, `--format text|json`. `make simulation-scope-check` target. 17 tests. **3255 total.** A simulation module may cover only some biological scopes. If a candidate is evaluated for a scope the module doesn't cover, that result must be flagged as out-of-scope rather than silently trusted.
> **New in v0.5.96:** Simulation-module deprecation enforcer (H8) — `DeprecationCheckResult` dataclass (5 fields: module_id, status, is_blocked, block_reason, dry_lab_only). `BLOCKED_STATUSES = {"deprecated", "unavailable"}`. `check_module_deprecation()` looks up module in registry: not-found returns is_blocked=True status="unknown"; deprecated/unavailable returns is_blocked=True with reason; active/experimental returns is_blocked=False. `enforce_deprecation()` filters list[SimulationResult] to only non-blocked modules, returns total_input/passed/blocked/blocked_modules/passed_results/checks/dry_lab_only. `run_deprecation_check_batch()` bulk-checks module_ids with total/blocked/allowed/any_blocked/results/dry_lab_only. CLI (`openamp-foundry simulation-deprecation-check`) with `--module-ids`, `--format text|json`. `make simulation-deprecation-check` target. 21 tests. **3238 total.** Deprecated simulation modules must not be used in production scoring — the enforcer prevents stale or unreliable modules from tainting evidence packets.
Expand Down Expand Up @@ -89,7 +90,7 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **New in v0.5.31:** Added dipeptide-order features for sequence-order awareness. `dipeptide_order_score` achieves AUROC 0.7861 on AMP-vs-scrambled discrimination — the strongest order-dependent feature in the pipeline. Only 7/31 features survive scrambling (amphipathicity/helix-wheel + dipeptide). All composition features are purely position-independent (exactly 0.5000 AUROC on scrambled test).
> **New in v0.5.30:** Easy baseline benchmark added — charge density alone (AUROC 0.8166) outperforms the full pipeline ensemble (0.7792) on AMP-vs-Swiss-Prot-decoy discrimination. Honest finding documented: expected because pipeline optimizes for safety, not raw discrimination.
> **New in v0.5.29:** Expanded benchmark to 500 AMPs + 500 composition-matched decoys (n=1000). AUROC 0.7792 (CI₉₅: 0.7505–0.8065) confirms signal generalizes. Cluster-aware CI: 0.746–0.8102. Representative AUROC: 0.778. Standard benchmark (n=191) retained for backward comparison.
> **Pipeline version:** v0.5.99 (I1 — artifact versioning policy)
> **Pipeline version:** v0.6.0 (I2 — candidate manifest schema)
> **Branch:** main

---
Expand Down
67 changes: 67 additions & 0 deletions docs/research/ROADMAP.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,72 @@
# Roadmap

## v0.6.0 — Loop 100: Phase I I2 — Candidate Manifest Schema ✓ (2026-07-09)

`schemas/candidate_manifest.schema.json` (Draft 2020-12, 14 required fields:
candidate_id, sequence, evidence_level, scopes, scores, uncertainty,
source_modules, calibration_set, safety_flags, provenance_run_id, dry_lab_only,
version, created_at, notes). `$schema`, `$id`, `title`, `description`,
`additionalProperties: false`.

`src/openamp_foundry/manifests/` module with `CandidateManifest` dataclass
(14 fields), `make_candidate_manifest()`, `validate_candidate_manifest()`
(8 checks: non-empty candidate_id, non-empty sequence, evidence_level 1-6,
non-empty scopes, uncertainty 0.0-1.0, non-empty source_modules,
dry_lab_only must be True, version MAJOR.MINOR.PATCH), `manifest_to_dict()`,
`manifest_summary()` (total, by_evidence_level, with_safety_flags, dry_lab_only).

CLI (`openamp-foundry candidate-manifest`) with `--candidate-id`, `--sequence`,
`--evidence-level`, `--scopes`, `--scores-json`, `--uncertainty`,
`--source-modules`, `--validate`, `--format text|json`.
`make candidate-manifest` target.

19 tests. **3310 total.** A candidate manifest is the core interoperable
artifact — it describes a dry-lab candidate (sequence, scores, evidence level,
scopes, safety flags, provenance) in a machine-readable format that external
tools can consume without the full OpenAMP stack. This is Loop 100 — the
100th PR in the NEXT_100_PR_MAP series.

Changes:
- `schemas/candidate_manifest.schema.json` (I2) — Draft 2020-12 schema with
14 required fields, `$id`, `title`, `description`, `additionalProperties: false`.
- `src/openamp_foundry/manifests/__init__.py` (I2) — Package init, exports
`CandidateManifest`, `make_candidate_manifest`, `validate_candidate_manifest`,
`manifest_to_dict`, `manifest_summary`.
- `src/openamp_foundry/manifests/candidate_manifest.py` (I2) — Core module
with `CandidateManifest` dataclass (14 fields), `make_candidate_manifest()`,
`validate_candidate_manifest()`, `manifest_to_dict()`, `manifest_summary()`.
- `src/openamp_foundry/cli/main.py` (I2) — Registered `candidate-manifest`
subcommand with `--candidate-id`, `--sequence`, `--evidence-level`,
`--scopes`, `--scores-json`, `--uncertainty`, `--source-modules`,
`--validate`, `--format` flags. Added import and dispatch.
- `src/openamp_foundry/cli/commands/reports.py` (I2) — Added
`_run_candidate_manifest()` CLI handler with JSON and text output,
manifest creation, optional validation, and error handling.
- `Makefile` (I2) — Added `candidate-manifest` target with demo invocation
using AMP-001. Added to `.PHONY`.
- `tests/manifests/__init__.py` (I2) — Empty package init.
- `tests/manifests/test_candidate_manifest.py` (I2) — 19 tests covering:
make_candidate_manifest returns CandidateManifest, dry_lab_only always True,
valid manifest passes validation, empty candidate_id fails, empty sequence
fails, evidence_level 0 or 7 fails, empty scopes fails, uncertainty out of
range fails, empty source_modules fails, dry_lab_only=False fails,
invalid version fails, manifest_to_dict returns all fields, manifest_summary
total correct, with_safety_flags correct, dry_lab_only=True, by_evidence_level
correct, schema file exists.
- `docs/evidence/METRICS_CURRENT.md` — v0.6.0 I2 changelog. Test count: 3310.
- `tests/test_test_count_regression.py` — baseline updated to 3310.

Honest boundaries:
- Candidate manifests describe dry-lab candidates only — they are computational
artifacts, not biological proof.
- Scores and uncertainty are pipeline outputs, not measured biological properties.
- The manifest schema is versioned (currently 1.0.0) and should be updated when
fields are added or changed.
- Validation checks structural correctness only — it does not verify biological
plausibility, safety, or efficacy.
- `dry_lab_only: true` is a const field — all manifests are inherently dry-lab
and must never be presented as validated wet-lab candidates.

## v0.5.95 — Loop 95: Phase H H7 — Simulation-Result Confidence Interval Reporter ✓ (2026-07-09)

`ScoreCI` dataclass (9 fields: module_id, score_key, point_estimate, uncertainty,
Expand Down
100 changes: 100 additions & 0 deletions schemas/candidate_manifest.schema.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://openamp-foundry.org/schemas/candidate_manifest.schema.json",
"title": "CandidateManifest",
"description": "Schema for a dry-lab candidate manifest — a machine-readable artifact describing a candidate AMP sequence, its scores, evidence level, scopes, safety flags, and provenance. Dry-lab only — does not measure biological activity.",
"type": "object",
"properties": {
"candidate_id": {
"description": "Unique identifier for the candidate",
"type": "string",
"minLength": 1
},
"sequence": {
"description": "Amino acid sequence of the candidate AMP",
"type": "string",
"minLength": 1
},
"evidence_level": {
"description": "Proof-ladder evidence level (1=computational nomination, 2=virtual-assay support, 3=in-silico ensemble agreement, 4=ex-vivo preliminary, 5=in-vivo preliminary, 6=clinical evidence)",
"type": "integer",
"minimum": 1,
"maximum": 6
},
"scopes": {
"description": "List of biological scopes this candidate is assessed for (e.g. bacterial_binding, anti_fungal)",
"type": "array",
"items": { "type": "string" },
"minItems": 1
},
"scores": {
"description": "Dictionary of named score values from pipeline modules",
"type": "object",
"patternProperties": {
"^.*$": { "type": "number" }
}
},
"uncertainty": {
"description": "Overall uncertainty estimate (0.0 = fully certain, 1.0 = fully uncertain)",
"type": "number",
"minimum": 0.0,
"maximum": 1.0
},
"source_modules": {
"description": "List of pipeline modules or scorers that produced this manifest",
"type": "array",
"items": { "type": "string" },
"minItems": 1
},
"calibration_set": {
"description": "Optional identifier for the calibration dataset used",
"type": ["string", "null"]
},
"safety_flags": {
"description": "List of safety flags (empty array = no flags raised)",
"type": "array",
"items": { "type": "string" }
},
"provenance_run_id": {
"description": "Optional run ID linking this manifest to a specific pipeline execution",
"type": ["string", "null"]
},
"dry_lab_only": {
"description": "Must be true — this is a dry-lab computational artifact",
"type": "boolean",
"const": true
},
"version": {
"description": "Manifest schema version (MAJOR.MINOR.PATCH)",
"type": "string",
"pattern": "^\\d+\\.\\d+\\.\\d+$"
},
"created_at": {
"description": "ISO 8601 UTC timestamp of manifest creation",
"type": "string",
"format": "date-time"
},
"notes": {
"description": "Free-text notes about this candidate",
"type": "array",
"items": { "type": "string" }
}
},
"required": [
"candidate_id",
"sequence",
"evidence_level",
"scopes",
"scores",
"uncertainty",
"source_modules",
"calibration_set",
"safety_flags",
"provenance_run_id",
"dry_lab_only",
"version",
"created_at",
"notes"
],
"additionalProperties": false
}
Loading
Loading