Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: help demo test lint ci clean bench-leakage bench-multi-negatives bench-baseline bench-hidden-active bench-cluster-split bench-expert-ablation bench-expert-ablation-500 bench-selectivity bench-feature-decomp bench-gate bench-easy-baseline bench-charge-matched bench-order-dependent bench-precision-at-k bench-per-family bench-simulation-ablation bench-simulation-ablation-within-amp simulation-gate bench-strategy-compare regenerate-all generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-pilot pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter recalibration-engine recalibration-engine-dry-run validate-policy-version generate-synthetic-lab-results bump-policy-version generate-review-packet failed-candidate-report safe-publication-filter classify-negative-informativeness validate-rejection-events negative-result-dashboard calibration-audit calibration-audit-example calibration-overfit-check result-quality-filter synthetic-result-policy-check calibration-decision-checklist calibration-rollback-plan simulation-registry validate-simulation-result-schema simulation-baseline-check adapter-gate-check simulation-provenance simulation-ensemble-check simulation-ci-report simulation-deprecation-check simulation-scope-check simulation-evidence-packet artifact-version candidate-manifest artifact-changelog integration-check adapter-author-check license-check artifact-compat-check contribution-check adoption-scorecard release-gate-check decision-log
.PHONY: help demo test lint ci clean bench-leakage bench-multi-negatives bench-baseline bench-hidden-active bench-cluster-split bench-expert-ablation bench-expert-ablation-500 bench-selectivity bench-feature-decomp bench-gate bench-easy-baseline bench-charge-matched bench-order-dependent bench-precision-at-k bench-per-family bench-simulation-ablation bench-simulation-ablation-within-amp simulation-gate bench-strategy-compare regenerate-all generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-pilot pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter recalibration-engine recalibration-engine-dry-run validate-policy-version generate-synthetic-lab-results bump-policy-version generate-review-packet failed-candidate-report safe-publication-filter classify-negative-informativeness validate-rejection-events negative-result-dashboard calibration-audit calibration-audit-example calibration-overfit-check result-quality-filter synthetic-result-policy-check calibration-decision-checklist calibration-rollback-plan simulation-registry validate-simulation-result-schema simulation-baseline-check adapter-gate-check simulation-provenance simulation-ensemble-check simulation-ci-report simulation-deprecation-check simulation-scope-check simulation-evidence-packet artifact-version candidate-manifest artifact-changelog integration-check adapter-author-check license-check artifact-compat-check contribution-check adoption-scorecard release-gate-check decision-log release-request-check

PYTHON := $(shell [ -f .venv/bin/python ] && echo .venv/bin/python || echo python3)
PYTEST := $(shell [ -f .venv/bin/pytest ] && echo .venv/bin/pytest || echo pytest)
Expand Down Expand Up @@ -845,3 +845,8 @@ release-gate-check:
decision-log:
.venv/bin/openamp-foundry decision-log --validate
@echo "Decision log check complete."

release-request-check:
.venv/bin/openamp-foundry release-request-check \
--request-json '{"release_id":"REL-2026-001","release_type":"schema","artifact_id":"candidate_manifest","artifact_version":"1.0.0","requestor_name":"OpenAMP Team","requestor_institution":"Open Problem Lab","request_date":"2026-07-09","evidence_level":2,"dry_lab_only":true,"safety_review_status":"approved","benchmark_summary":"Passes all schema compatibility checks","known_limitations":"Dry-lab only, not wet-lab validated","intended_use":"public","data_license":"CC-BY-4.0","human_reviewer":"maintainer","review_class":"B","approval_status":"pending"}'
@echo "Release request check complete."
7 changes: 4 additions & 3 deletions docs/evidence/METRICS_CURRENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
> with this file, this file wins. Updated whenever benchmark/benchmark config changes.
>
> **Last updated:** 2026-07-09 (Phase J J2 — governance decision log — v0.7.0)
> **Last updated:** 2026-07-09 (Phase J J3 — release request template — v0.7.1)
> **New in v0.7.1:** Release request template (J3 — Loop 111) — `docs/governance/RELEASE_REQUEST_TEMPLATE.md` with structured release request template (purpose, fill-in-the-blank template with 17 fields: Release ID, release type, artifact ID/version, requestor name/institution, request date, evidence level 1-6, dry_lab_only, safety_review_status, benchmark_summary, known_limitations, intended_use, data_license, human_reviewer, review_class, approval_status; review criteria with 8 checks; process with classes A-D timelines and escalation path). `src/openamp_foundry/governance/release_request.py` with `ReleaseRequest` dataclass (17 fields), `ReleaseRequestValidationResult` dataclass (6 fields, dry_lab_only=True), `VALID_RELEASE_TYPES` (5), `VALID_SAFETY_STATUSES` (3), `VALID_INTENDED_USES` (4), `VALID_APPROVAL_STATUSES` (4), `VALID_REVIEW_CLASSES` (4), `validate_release_request()` (17 checks including dry_lab_only+evidence_level>4 error, public+safety_pending error, model+review_class warning), `validate_request_dict()` (dict input with missing-fields guard). CLI (`openamp-foundry release-request-check`) with `--request-json`, `--format text|json`. `make release-request-check` target. 25 tests. **3516 total.** Formal release requests now have a validated structure before entering human review.
> **New in v0.7.0:** Governance decision log index (J2 — Loop 110, Phase J milestone) — `docs/governance/DECISION_LOG.md` with structured decision log (purpose, decision index with 8 entries GOV-001 through GOV-008 covering safety/benchmark/release/evidence/data/adapter/contribution/docs scopes, how to add entries, linked policies). `src/openamp_foundry/governance/decision_log.py` with `VALID_SCOPES` (8 entries: safety, benchmark, release, evidence, data, adapter, contribution, docs), `VALID_STATUSES` (4 entries: active, superseded, under_review, proposed), `VALID_REVIEW_CLASSES` (4 entries: A, B, C, D), `GovernanceDecision` dataclass (8 fields: decision_id, date, scope, decision, status, rationale, review_class, dry_lab_only=True), `DecisionValidationResult` dataclass (5 fields: decision_id, passed, errors, warnings, dry_lab_only=True), `GOVERNANCE_DECISIONS` list (8 entries: GOV-001 through GOV-008), `validate_governance_decision()` (9 checks: decision_id format, date format, valid scope, non-empty decision, valid status, non-empty rationale, valid review_class, dry_lab_only must be True, superseded warning), `validate_all_decisions()` (aggregates total/passed/failed/all_passed/results/dry_lab_only), `get_decisions_by_scope()` (filters by scope), `get_decisions_by_status()` (filters by status). CLI (`openamp-foundry decision-log`) with `--validate`, `--scope`, `--format text|json`. `make decision-log` target. 27 tests. **3505 total.** **Phase J milestone: v0.7.0** — governance decisions are now discoverable and machine-validated.
> **New in v0.6.9:** Release checklist and gate validator (J1 — Loop 109, starts Phase J) — `docs/governance/RELEASE_CHECKLIST.md` with structured checklist (pre-release gates, release-type gates for 5 types, post-release checklist), cross-referencing `docs/trust/RELEASE_CHECKLIST.md`. `src/openamp_foundry/governance/release_gate.py` with `RELEASE_TYPES` (5: candidate, model, dataset, evidence_packet, schema), `UNIVERSAL_GATES` (7: ci_tests_pass, agent_check_passes, no_critical_issues, dry_lab_only_confirmed, safety_flags_reviewed, data_license_verified, no_hardcoded_secrets), `EXTRA_GATES_BY_TYPE` (per-type additional gates), `ReleaseGateResult` dataclass (8 fields, dry_lab_only=True), `validate_release_gate()` (validates all required gates, treats missing gates as failed, raises CRITICAL error on dry_lab_only_confirmed failure). CLI (`openamp-foundry release-gate-check`) with `--release-type`, `--artifact-id`, `--gates-json`, `--format text|json`. `make release-gate-check` target. 18 tests. **3478 total.** **Starts Phase J (Governance and release maturity)** — releases now require all gates to pass before external release, preventing accidental bypass of required checks.
> **New in v0.6.8:** Adoption scorecard dashboard (I10 — Loop 108) — `src/openamp_foundry/adoption/scorecard.py` with `SCORECARD_DIMENSIONS` (5 weighted dimensions summing to 1.0: integration_check 0.25, license_compliance 0.20, adapter_validation 0.20, schema_compatibility 0.20, contribution_readiness 0.15), `ADOPTION_TIERS` (4 tiers: not_ready 0.0-0.40, emerging 0.40-0.65, established 0.65-0.85, mature 0.85-1.01), `DimensionScore` dataclass (8 fields, dry_lab_only=True), `AdoptionScorecard` dataclass (6 fields, dry_lab_only=True), `build_scorecard()` (aggregates dimension inputs into weighted total score with tier and actionable recommendations), `compute_adoption_tier()` (maps score to tier). CLI (`openamp-foundry adoption-scorecard`) with `--scores-json`, `--format text|json`. `make adoption-scorecard` target. 17 tests. **3446 total.** **Phase I (Interoperability and Adoption) is now complete** — all 10 items I1–I10 implemented (artifact versioning, candidate manifest, benchmark card, artifact changelog, integration checker, adapter validator, data license checker, schema compatibility, contribution intake, adoption scorecard).
Expand Down Expand Up @@ -98,7 +99,7 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **New in v0.5.31:** Added dipeptide-order features for sequence-order awareness. `dipeptide_order_score` achieves AUROC 0.7861 on AMP-vs-scrambled discrimination — the strongest order-dependent feature in the pipeline. Only 7/31 features survive scrambling (amphipathicity/helix-wheel + dipeptide). All composition features are purely position-independent (exactly 0.5000 AUROC on scrambled test).
> **New in v0.5.30:** Easy baseline benchmark added — charge density alone (AUROC 0.8166) outperforms the full pipeline ensemble (0.7792) on AMP-vs-Swiss-Prot-decoy discrimination. Honest finding documented: expected because pipeline optimizes for safety, not raw discrimination.
> **New in v0.5.29:** Expanded benchmark to 500 AMPs + 500 composition-matched decoys (n=1000). AUROC 0.7792 (CI₉₅: 0.7505–0.8065) confirms signal generalizes. Cluster-aware CI: 0.746–0.8102. Representative AUROC: 0.778. Standard benchmark (n=191) retained for backward comparison.
> **Pipeline version:** v0.7.0 (J2 — governance decision log)
> **Pipeline version:** v0.7.1 (J3 — release request template)
> **Branch:** main

---
Expand Down Expand Up @@ -1168,7 +1169,7 @@ Decoys score low on activity. Selective AMPs score moderately on both.

| Metric | Value |
|--------|-------|
| Total tests | 3478 |
| Total tests | 3516 |
| Coverage (branch) | 99% (6 CLI guard lines only) |
| Source modules at 100% | All pipeline, QC, scoring, adapter modules |

Expand Down
72 changes: 72 additions & 0 deletions docs/governance/RELEASE_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Release Request Template

## Purpose

A release request is the formal artifact filed when a model, candidate, dataset, evidence packet, or schema is proposed for external release. File one whenever an artifact exits the internal pipeline and enters external distribution. The OpenAMP governance team reviews all requests before publication.

**Who reviews:** human_reviewer (GitHub handle) specified in the request. Review class determines scrutiny level.

---

## Template

Fill in every field. Fields marked REQUIRED must not be empty.

```yaml
release_id: "" # Auto-assigned: REL-YYYY-NNN (REQUIRED)
release_type: "" # candidate | model | dataset | evidence_packet | schema (REQUIRED)
artifact_id: "" # Unique identifier for the artifact (REQUIRED)
artifact_version: "" # Semantic version, e.g. 1.0.0 (REQUIRED)
requestor_name: "" # Full name (REQUIRED)
requestor_institution: "" # Institution or organization (REQUIRED)
request_date: "" # YYYY-MM-DD (REQUIRED)
evidence_level: 0 # Integer 1–6; dry-lab-only max is 4 (REQUIRED)
dry_lab_only: true # Must be true — all releases are dry-lab only (REQUIRED)
safety_review_status: "" # pending | approved | not_required (REQUIRED)
benchmark_summary: "" # What baseline was beaten; must not be empty (REQUIRED)
known_limitations: "" # Must not be empty (REQUIRED)
intended_use: "" # research | internal | external_partner | public (REQUIRED)
data_license: "" # e.g. CC-BY-4.0, Apache-2.0 (REQUIRED)
human_reviewer: "" # GitHub handle of designated reviewer (REQUIRED)
review_class: "" # A | B | C | D (REQUIRED)
approval_status: "" # pending | approved | rejected | deferred (REQUIRED)
```

---

## Review Criteria

Reviewers check the following before approving:

1. **Completeness** — all required fields present and non-empty
2. **Release ID format** — starts with `REL-`
3. **Valid release type** — one of the five recognized types
4. **Evidence level** — within 1–6; dry-lab-only artifacts must not exceed 4
5. **Safety review** — public releases must not have `safety_review_status: pending`
6. **Benchmark summary** — non-trivial; describes what baseline was beaten
7. **Known limitations** — honestly stated; not a placeholder
8. **Data license** — explicitly named; no "TBD"
9. **Review class** — model releases typically require class C or D
10. **Dry-lab boundary** — `dry_lab_only` must be `true`; computational outputs are not biological proof

---

## Process

1. **File the request** — copy the template above, fill in all fields, open a GitHub issue titled `[Release Request] REL-YYYY-NNN — <artifact_id>`.
2. **Machine validation** — run `openamp-foundry release-request-check --request-json '{...}'` to pre-validate before human review.
3. **Human review** — the designated `human_reviewer` reviews within 5 business days.
4. **Decision** — reviewer sets `approval_status` to `approved`, `rejected`, or `deferred` with written rationale.
5. **Escalation** — if no decision within 10 business days, escalate to project maintainers via GitHub discussion.

**Expected timeline:** 5 business days for standard review; 10 business days for class D (model/safety-critical).

---

## Linked Policies

- [GOVERNANCE.md](../../GOVERNANCE.md)
- [SAFETY.md](../../SAFETY.md)
- [RESPONSIBLE_USE.md](../../RESPONSIBLE_USE.md)
- [RELEASE_CHECKLIST.md](RELEASE_CHECKLIST.md)
- [DECISION_LOG.md](DECISION_LOG.md)
Loading
Loading