Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: help demo test lint ci clean bench-leakage bench-multi-negatives bench-baseline bench-hidden-active bench-cluster-split bench-expert-ablation bench-expert-ablation-500 bench-selectivity bench-feature-decomp bench-gate bench-easy-baseline bench-charge-matched bench-order-dependent bench-precision-at-k bench-per-family bench-simulation-ablation bench-simulation-ablation-within-amp simulation-gate bench-strategy-compare regenerate-all generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-pilot pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter recalibration-engine recalibration-engine-dry-run validate-policy-version generate-synthetic-lab-results bump-policy-version generate-review-packet failed-candidate-report safe-publication-filter classify-negative-informativeness validate-rejection-events negative-result-dashboard calibration-audit calibration-audit-example calibration-overfit-check result-quality-filter synthetic-result-policy-check calibration-decision-checklist calibration-rollback-plan simulation-registry validate-simulation-result-schema simulation-baseline-check adapter-gate-check simulation-provenance simulation-ensemble-check simulation-ci-report simulation-deprecation-check simulation-scope-check simulation-evidence-packet artifact-version candidate-manifest
.PHONY: help demo test lint ci clean bench-leakage bench-multi-negatives bench-baseline bench-hidden-active bench-cluster-split bench-expert-ablation bench-expert-ablation-500 bench-selectivity bench-feature-decomp bench-gate bench-easy-baseline bench-charge-matched bench-order-dependent bench-precision-at-k bench-per-family bench-simulation-ablation bench-simulation-ablation-within-amp simulation-gate bench-strategy-compare regenerate-all generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-pilot pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter recalibration-engine recalibration-engine-dry-run validate-policy-version generate-synthetic-lab-results bump-policy-version generate-review-packet failed-candidate-report safe-publication-filter classify-negative-informativeness validate-rejection-events negative-result-dashboard calibration-audit calibration-audit-example calibration-overfit-check result-quality-filter synthetic-result-policy-check calibration-decision-checklist calibration-rollback-plan simulation-registry validate-simulation-result-schema simulation-baseline-check adapter-gate-check simulation-provenance simulation-ensemble-check simulation-ci-report simulation-deprecation-check simulation-scope-check simulation-evidence-packet artifact-version candidate-manifest artifact-changelog

PYTHON := $(shell [ -f .venv/bin/python ] && echo .venv/bin/python || echo python3)
PYTEST := $(shell [ -f .venv/bin/pytest ] && echo .venv/bin/pytest || echo pytest)
Expand Down Expand Up @@ -801,3 +801,7 @@ candidate-manifest:
--source-modules membrane_proxy \
--validate
@echo "Candidate manifest complete."

artifact-changelog:
.venv/bin/openamp-foundry artifact-changelog
@echo "Artifact changelog complete."
38 changes: 38 additions & 0 deletions docs/engineering/ARTIFACT_CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Artifact Changelog

> Machine-readable changelog for evidence certificates and computational
> artifacts. External tools can parse this document to detect breaking
> changes and adapt consumers.
>
> All entries are for dry-lab computational artifacts only — they describe
> schema and format changes, not biological findings.

## Unreleased

- (no pending changes)

## [1.0.0] - 2026-07-09

### Added

- **candidate_manifest** — initial schema (14 required fields: candidate_id,
sequence, evidence_level, scopes, scores, uncertainty, source_modules,
calibration_set, safety_flags, provenance_run_id, dry_lab_only, version,
created_at, notes). Phase I I2. Entry type: added. Breaking: false.

- **benchmark_card** — initial schema (15 required fields: benchmark_id,
benchmark_name, version, date, metric, metric_value, baseline_name,
baseline_value, delta, beats_baseline, dataset, dataset_size, scope,
caveats, dry_lab_only). Phase I I3. Entry type: added. Breaking: false.

- **simulation_result** — initial schema (8 required fields: module, version,
scope, scores, uncertainty, calibration_set, validated_against, notes).
Phase H H2. Entry type: added. Breaking: false.

- **simulation_module_registry** — initial schema with module metadata
(module_id, name, description, status, evidence_level, baseline_comparison,
scope, maintainer, notes). Phase H H1. Entry type: added. Breaking: false.

- **artifact_versioning_policy** — initial versioning policy document
covering stability tiers, SemVer format, breaking change rules, and
deprecation timeline. Phase I I1. Entry type: added. Breaking: false.
7 changes: 4 additions & 3 deletions docs/evidence/METRICS_CURRENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
> with this file, this file wins. Updated whenever benchmark/benchmark config changes.
>
> **Last updated:** 2026-07-09 (Phase I I3 — benchmark card schema — v0.6.1)
> **Last updated:** 2026-07-09 (Phase I I4 — evidence-certificate changelog — v0.6.2)
> **New in v0.6.2:** Evidence-certificate changelog (I4 — Loop 102) — `docs/engineering/ARTIFACT_CHANGELOG.md` with structured changelog format (version, date, artifact_name, change_type, description, breaking flag). Unreleased section at top. `src/openamp_foundry/versioning/artifact_changelog.py` with `CHANGE_TYPES` set (6 values), `ChangelogEntry` dataclass (7 fields), `ARTIFACT_CHANGELOG` list (6 entries: candidate_manifest, benchmark_card, simulation_result, simulation_module_registry, artifact_versioning_policy, artifact_changelog — all "added", non-breaking, v1.0.0), `get_changelog_entries()` (filters by artifact_name, version, change_type, breaking_only), `validate_changelog()` (5 checks: version MAJOR.MINOR.PATCH, date YYYY-MM-DD with dashes, artifact_name non-empty, change_type in CHANGE_TYPES, description non-empty), `changelog_summary()` (total, by_change_type, breaking_changes, artifacts_covered sorted, dry_lab_only). CLI (`openamp-foundry artifact-changelog`) with `--artifact`, `--version`, `--change-type`, `--breaking-only`, `--format text|json`. `make artifact-changelog` target. 13 tests. **3342 total.** External tools that consume OpenAMP artifacts now have a machine-readable changelog to detect breaking changes and adapt consumers.
> **New in v0.6.1:** Benchmark card schema (I3 — Loop 101) — `schemas/benchmark_card.schema.json` (Draft 2020-12, 15 required fields: benchmark_id, benchmark_name, version, date, metric, metric_value, baseline_name, baseline_value, delta, beats_baseline, dataset, dataset_size, scope, caveats, dry_lab_only). `$schema`, `$id`, `title`, `additionalProperties: false`. `src/openamp_foundry/benchmarks/` module with `BenchmarkCard` dataclass (15 fields), `make_benchmark_card()` (auto-computes delta, beats_baseline), `validate_benchmark_card()` (10 checks: non-empty benchmark_id, non-empty benchmark_name, non-empty metric, non-empty dataset, non-empty baseline_name, dataset_size >= 1, delta matches metric_value - baseline_value within 1e-9, beats_baseline matches delta > 0, dry_lab_only must be True), `benchmark_card_summary()` (total, beats_baseline_count, fails_baseline_count, dry_lab_only). CLI (`openamp-foundry benchmark-card`) with `--benchmark-id`, `--benchmark-name`, `--metric`, `--metric-value`, `--baseline-name`, `--baseline-value`, `--dataset`, `--dataset-size`, `--validate`, `--format text|json`. `make benchmark-card` target. 19 tests. **3329 total.** A benchmark card is the standard format for describing external benchmark results — what was benchmarked, what the baseline was, what the result was, and what claims are supported.
> **New in v0.6.0:** Candidate manifest schema (I2 — Loop 100) — `schemas/candidate_manifest.schema.json` (Draft 2020-12, 14 required fields: candidate_id, sequence, evidence_level, scopes, scores, uncertainty, source_modules, calibration_set, safety_flags, provenance_run_id, dry_lab_only, version, created_at, notes). `$schema`, `$id`, `title`, `description`, `additionalProperties: false`. `src/openamp_foundry/manifests/` module with `CandidateManifest` dataclass (14 fields), `make_candidate_manifest()`, `validate_candidate_manifest()` (8 checks: non-empty candidate_id, non-empty sequence, evidence_level 1-6, non-empty scopes, uncertainty 0.0-1.0, non-empty source_modules, dry_lab_only must be True, version MAJOR.MINOR.PATCH), `manifest_to_dict()`, `manifest_summary()` (total, by_evidence_level, with_safety_flags, dry_lab_only). CLI (`openamp-foundry candidate-manifest`) with `--candidate-id`, `--sequence`, `--evidence-level`, `--scopes`, `--scores-json`, `--uncertainty`, `--source-modules`, `--validate`, `--format text|json`. `make candidate-manifest` target. 19 tests. **3310 total.** A candidate manifest is the core interoperable artifact — it describes a dry-lab candidate (sequence, scores, evidence level, scopes, safety flags, provenance) in a machine-readable format that external tools can consume without the full OpenAMP stack.
> **New in v0.5.99:** Artifact versioning policy (I1 — starts Phase I) — `docs/engineering/ARTIFACT_VERSIONING_POLICY.md` with structured versioning policy covering scope, SemVer format, breaking/non-breaking change definitions, deprecation timeline (at least one minor version warning), schema `$id` policy, changelog requirement, and three stability tiers (stable/experimental/internal). `src/openamp_foundry/versioning/` module with `ArtifactVersionInfo` dataclass (7 fields), `STABILITY_TIERS` dict, `VERSIONED_ARTIFACTS` list (6 entries covering candidate, lab_result, run_manifest, external_review_packet, safety_release_decision, simulation_result at v1.0.0), `get_artifact_version()`, `list_versioned_artifacts()`, `validate_version_format()`, `artifact_version_summary()`. CLI (`openamp-foundry artifact-version`) with `--list`, `--show`, `--tier`, `--format text|json`. `make artifact-version` target. 19 tests. **3291 total.** Starts Phase I (interoperability) — external users now have stability guarantees for schemas, evidence certificates, and candidate manifests.
Expand Down Expand Up @@ -91,7 +92,7 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **New in v0.5.31:** Added dipeptide-order features for sequence-order awareness. `dipeptide_order_score` achieves AUROC 0.7861 on AMP-vs-scrambled discrimination — the strongest order-dependent feature in the pipeline. Only 7/31 features survive scrambling (amphipathicity/helix-wheel + dipeptide). All composition features are purely position-independent (exactly 0.5000 AUROC on scrambled test).
> **New in v0.5.30:** Easy baseline benchmark added — charge density alone (AUROC 0.8166) outperforms the full pipeline ensemble (0.7792) on AMP-vs-Swiss-Prot-decoy discrimination. Honest finding documented: expected because pipeline optimizes for safety, not raw discrimination.
> **New in v0.5.29:** Expanded benchmark to 500 AMPs + 500 composition-matched decoys (n=1000). AUROC 0.7792 (CI₉₅: 0.7505–0.8065) confirms signal generalizes. Cluster-aware CI: 0.746–0.8102. Representative AUROC: 0.778. Standard benchmark (n=191) retained for backward comparison.
> **Pipeline version:** v0.6.1 (I3 — benchmark card schema)
> **Pipeline version:** v0.6.2 (I4 — evidence-certificate changelog)
> **Branch:** main

---
Expand Down Expand Up @@ -1161,7 +1162,7 @@ Decoys score low on activity. Selective AMPs score moderately on both.

| Metric | Value |
|--------|-------|
| Total tests | 3291 |
| Total tests | 3342 |
| Coverage (branch) | 99% (6 CLI guard lines only) |
| Source modules at 100% | All pipeline, QC, scoring modules |

Expand Down
64 changes: 64 additions & 0 deletions docs/research/ROADMAP.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,69 @@
# Roadmap

## v0.6.2 — Loop 102: Phase I I4 — Evidence-Certificate Changelog ✓ (2026-07-09)

`docs/engineering/ARTIFACT_CHANGELOG.md` with structured changelog format
(version, date, artifact_name, change_type, description, breaking flag).
Unreleased section at top for pending changes. 5 entries covering recent
Phase H and I additions (candidate_manifest, benchmark_card,
simulation_result, simulation_module_registry, artifact_versioning_policy)
plus the changelog itself — all as "added", non-breaking, v1.0.0.

`src/openamp_foundry/versioning/artifact_changelog.py` with `CHANGE_TYPES`
set (6 values: added, changed, deprecated, removed, fixed, security),
`ChangelogEntry` dataclass (7 fields: version, date, artifact_name,
change_type, description, breaking, notes), `ARTIFACT_CHANGELOG` list
(6 entries), `get_changelog_entries()` (filters by artifact_name, version,
change_type, breaking_only), `validate_changelog()` (5 checks: version
MAJOR.MINOR.PATCH, date YYYY-MM-DD with dashes, artifact_name non-empty,
change_type in CHANGE_TYPES, description non-empty), `changelog_summary()`
(total, by_change_type, breaking_changes, artifacts_covered sorted,
dry_lab_only).

CLI (`openamp-foundry artifact-changelog`) with `--artifact`, `--version`,
`--change-type`, `--breaking-only`, `--format text|json`.
`make artifact-changelog` target.

13 tests. **3342 total.** External tools that consume OpenAMP artifacts
now have a machine-readable changelog to detect breaking changes and adapt
consumers. This is Loop 102 — the 102nd PR in the NEXT_100_PR_MAP series.

Changes:
- `docs/engineering/ARTIFACT_CHANGELOG.md` (I4) — Structured changelog
document with versioned entries, Unreleased section, and dry-lab-only
disclaimer.
- `src/openamp_foundry/versioning/artifact_changelog.py` (I4) — Core
module with `CHANGE_TYPES`, `ChangelogEntry` dataclass,
`ARTIFACT_CHANGELOG`, `get_changelog_entries()`, `validate_changelog()`,
`changelog_summary()`.
- `src/openamp_foundry/versioning/__init__.py` (I4) — Exported
`CHANGE_TYPES`, `ChangelogEntry`, `ARTIFACT_CHANGELOG`,
`get_changelog_entries`, `validate_changelog`, `changelog_summary`.
- `src/openamp_foundry/cli/main.py` (I4) — Registered `artifact-changelog`
subcommand with `--artifact`, `--version`, `--change-type`,
`--breaking-only`, `--format` flags. Added import and dispatch.
- `src/openamp_foundry/cli/commands/reports.py` (I4) — Added
`_run_artifact_changelog()` CLI handler with JSON and text output,
filtering, validation display, and summary.
- `Makefile` (I4) — Added `artifact-changelog` target with `.PHONY`.
- `tests/versioning/test_artifact_changelog.py` (I4) — 13 tests covering:
at least 5 entries, all pass validation, no filter returns all, filter
by artifact_name, breaking_only, change_type, catches empty artifact_name,
invalid change_type, invalid version format, summary total, dry_lab_only,
breaking_changes count, artifacts_covered sorted.
- `docs/evidence/METRICS_CURRENT.md` — v0.6.2 I4 changelog. Test count: 3342.
- `tests/test_test_count_regression.py` — baseline updated to 3342.

Honest boundaries:
- Changelog entries describe schema and format changes only — they do not
measure biological activity, safety, or clinical value.
- All current entries are "added" and non-breaking; breaking changes are
expected in future major versions.
- The changelog is a computational artifact — it records version history,
not biological findings.
- `dry_lab_only: true` applies to all entries — changelogs are inherently
dry-lab and must never be presented as validated biological findings.

## v0.6.1 — Loop 101: Phase I I3 — Benchmark Card Schema ✓ (2026-07-09)

`schemas/benchmark_card.schema.json` (Draft 2020-12, 15 required fields:
Expand Down
76 changes: 76 additions & 0 deletions src/openamp_foundry/cli/commands/reports.py
Original file line number Diff line number Diff line change
Expand Up @@ -2087,3 +2087,79 @@ def _run_benchmark_card(args: argparse.Namespace) -> int:
if validation_errors:
return 3
return 0


def _run_artifact_changelog(args: argparse.Namespace) -> int:
"""Display evidence-certificate changelog entries."""
from openamp_foundry.versioning.artifact_changelog import (
ARTIFACT_CHANGELOG,
get_changelog_entries,
validate_changelog,
changelog_summary,
)

artifact = getattr(args, "artifact", None)
version = getattr(args, "version", None)
change_type = getattr(args, "change_type", None)
breaking_only = getattr(args, "breaking_only", False)
output_format = getattr(args, "format", "text")

entries = get_changelog_entries(
artifact_name=artifact,
version=version,
change_type=change_type,
breaking_only=breaking_only,
)

summary = changelog_summary()
validation_errors = validate_changelog()

if output_format == "json":
result = {
"entries": [
{
"version": e.version,
"date": e.date,
"artifact_name": e.artifact_name,
"change_type": e.change_type,
"description": e.description,
"breaking": e.breaking,
"notes": e.notes,
}
for e in entries
],
"summary": summary,
"valid": len(validation_errors) == 0,
}
if validation_errors:
result["validation_errors"] = validation_errors
print(json.dumps(result, indent=2))
else:
print("Artifact Changelog")
print("=" * 60)
print(f"Total entries: {summary['total']}")
print(f"By change type: {dict(sorted(summary['by_change_type'].items()))}")
print(f"Breaking: {summary['breaking_changes']}")
print(f"Artifacts: {', '.join(summary['artifacts_covered'])}")
print(f"Dry-lab only: {summary['dry_lab_only']}")
if validation_errors:
print(f"Validation: {len(validation_errors)} error(s)")
else:
print(f"Validation: passed")
print()
for entry in entries:
breaking_label = " [BREAKING]" if entry.breaking else ""
print(f" v{entry.version:8s} {entry.date} "
f"[{entry.change_type:10s}]{breaking_label}")
print(f" {'':18s} {entry.artifact_name}")
print(f" {'':18s} {entry.description[:80]}")
if entry.notes:
print(f" {'':18s} Notes: {entry.notes[:80]}")
print()

print("Dry-lab only. Changelog entries describe schema and format")
print("changes, not biological validity, safety, or efficacy.")

if validation_errors:
return 3
return 0
Loading
Loading