Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: help demo test lint ci clean bench-leakage bench-multi-negatives bench-baseline bench-hidden-active bench-cluster-split bench-expert-ablation bench-expert-ablation-500 bench-selectivity bench-feature-decomp bench-gate bench-easy-baseline bench-charge-matched bench-order-dependent bench-precision-at-k bench-per-family bench-simulation-ablation bench-simulation-ablation-within-amp simulation-gate bench-strategy-compare regenerate-all generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-pilot pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter recalibration-engine recalibration-engine-dry-run validate-policy-version generate-synthetic-lab-results bump-policy-version generate-review-packet failed-candidate-report safe-publication-filter classify-negative-informativeness validate-rejection-events negative-result-dashboard calibration-audit calibration-audit-example calibration-overfit-check result-quality-filter synthetic-result-policy-check calibration-decision-checklist calibration-rollback-plan simulation-registry validate-simulation-result-schema simulation-baseline-check adapter-gate-check simulation-provenance simulation-ensemble-check simulation-ci-report simulation-deprecation-check simulation-scope-check simulation-evidence-packet artifact-version candidate-manifest artifact-changelog integration-check adapter-author-check license-check artifact-compat-check contribution-check adoption-scorecard release-gate-check
.PHONY: help demo test lint ci clean bench-leakage bench-multi-negatives bench-baseline bench-hidden-active bench-cluster-split bench-expert-ablation bench-expert-ablation-500 bench-selectivity bench-feature-decomp bench-gate bench-easy-baseline bench-charge-matched bench-order-dependent bench-precision-at-k bench-per-family bench-simulation-ablation bench-simulation-ablation-within-amp simulation-gate bench-strategy-compare regenerate-all generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-pilot pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter recalibration-engine recalibration-engine-dry-run validate-policy-version generate-synthetic-lab-results bump-policy-version generate-review-packet failed-candidate-report safe-publication-filter classify-negative-informativeness validate-rejection-events negative-result-dashboard calibration-audit calibration-audit-example calibration-overfit-check result-quality-filter synthetic-result-policy-check calibration-decision-checklist calibration-rollback-plan simulation-registry validate-simulation-result-schema simulation-baseline-check adapter-gate-check simulation-provenance simulation-ensemble-check simulation-ci-report simulation-deprecation-check simulation-scope-check simulation-evidence-packet artifact-version candidate-manifest artifact-changelog integration-check adapter-author-check license-check artifact-compat-check contribution-check adoption-scorecard release-gate-check decision-log

PYTHON := $(shell [ -f .venv/bin/python ] && echo .venv/bin/python || echo python3)
PYTEST := $(shell [ -f .venv/bin/pytest ] && echo .venv/bin/pytest || echo pytest)
Expand Down Expand Up @@ -841,3 +841,7 @@ release-gate-check:
--artifact-id candidate_manifest_v1 \
--gates-json '{"ci_tests_pass":true,"agent_check_passes":true,"no_critical_issues":true,"dry_lab_only_confirmed":true,"safety_flags_reviewed":true,"data_license_verified":true,"no_hardcoded_secrets":true,"schema_compatibility_passes":true,"version_bumped":true,"changelog_updated":true}'
@echo "Release gate check complete."

decision-log:
.venv/bin/openamp-foundry decision-log --validate
@echo "Decision log check complete."
5 changes: 3 additions & 2 deletions docs/evidence/METRICS_CURRENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
> with this file, this file wins. Updated whenever benchmark/benchmark config changes.
>
> **Last updated:** 2026-07-09 (Phase J J1 — release checklist — v0.6.9)
> **Last updated:** 2026-07-09 (Phase J J2 — governance decision log — v0.7.0)
> **New in v0.7.0:** Governance decision log index (J2 — Loop 110, Phase J milestone) — `docs/governance/DECISION_LOG.md` with structured decision log (purpose, decision index with 8 entries GOV-001 through GOV-008 covering safety/benchmark/release/evidence/data/adapter/contribution/docs scopes, how to add entries, linked policies). `src/openamp_foundry/governance/decision_log.py` with `VALID_SCOPES` (8 entries: safety, benchmark, release, evidence, data, adapter, contribution, docs), `VALID_STATUSES` (4 entries: active, superseded, under_review, proposed), `VALID_REVIEW_CLASSES` (4 entries: A, B, C, D), `GovernanceDecision` dataclass (8 fields: decision_id, date, scope, decision, status, rationale, review_class, dry_lab_only=True), `DecisionValidationResult` dataclass (5 fields: decision_id, passed, errors, warnings, dry_lab_only=True), `GOVERNANCE_DECISIONS` list (8 entries: GOV-001 through GOV-008), `validate_governance_decision()` (9 checks: decision_id format, date format, valid scope, non-empty decision, valid status, non-empty rationale, valid review_class, dry_lab_only must be True, superseded warning), `validate_all_decisions()` (aggregates total/passed/failed/all_passed/results/dry_lab_only), `get_decisions_by_scope()` (filters by scope), `get_decisions_by_status()` (filters by status). CLI (`openamp-foundry decision-log`) with `--validate`, `--scope`, `--format text|json`. `make decision-log` target. 27 tests. **3505 total.** **Phase J milestone: v0.7.0** — governance decisions are now discoverable and machine-validated.
> **New in v0.6.9:** Release checklist and gate validator (J1 — Loop 109, starts Phase J) — `docs/governance/RELEASE_CHECKLIST.md` with structured checklist (pre-release gates, release-type gates for 5 types, post-release checklist), cross-referencing `docs/trust/RELEASE_CHECKLIST.md`. `src/openamp_foundry/governance/release_gate.py` with `RELEASE_TYPES` (5: candidate, model, dataset, evidence_packet, schema), `UNIVERSAL_GATES` (7: ci_tests_pass, agent_check_passes, no_critical_issues, dry_lab_only_confirmed, safety_flags_reviewed, data_license_verified, no_hardcoded_secrets), `EXTRA_GATES_BY_TYPE` (per-type additional gates), `ReleaseGateResult` dataclass (8 fields, dry_lab_only=True), `validate_release_gate()` (validates all required gates, treats missing gates as failed, raises CRITICAL error on dry_lab_only_confirmed failure). CLI (`openamp-foundry release-gate-check`) with `--release-type`, `--artifact-id`, `--gates-json`, `--format text|json`. `make release-gate-check` target. 18 tests. **3478 total.** **Starts Phase J (Governance and release maturity)** — releases now require all gates to pass before external release, preventing accidental bypass of required checks.
> **New in v0.6.8:** Adoption scorecard dashboard (I10 — Loop 108) — `src/openamp_foundry/adoption/scorecard.py` with `SCORECARD_DIMENSIONS` (5 weighted dimensions summing to 1.0: integration_check 0.25, license_compliance 0.20, adapter_validation 0.20, schema_compatibility 0.20, contribution_readiness 0.15), `ADOPTION_TIERS` (4 tiers: not_ready 0.0-0.40, emerging 0.40-0.65, established 0.65-0.85, mature 0.85-1.01), `DimensionScore` dataclass (8 fields, dry_lab_only=True), `AdoptionScorecard` dataclass (6 fields, dry_lab_only=True), `build_scorecard()` (aggregates dimension inputs into weighted total score with tier and actionable recommendations), `compute_adoption_tier()` (maps score to tier). CLI (`openamp-foundry adoption-scorecard`) with `--scores-json`, `--format text|json`. `make adoption-scorecard` target. 17 tests. **3446 total.** **Phase I (Interoperability and Adoption) is now complete** — all 10 items I1–I10 implemented (artifact versioning, candidate manifest, benchmark card, artifact changelog, integration checker, adapter validator, data license checker, schema compatibility, contribution intake, adoption scorecard).
> **New in v0.6.7:** Public-good contribution guide (I9 — Loop 107) — `docs/community/PUBLIC_GOOD_CONTRIBUTION_GUIDE.md` with 6 contribution types (wet_lab_validation, dataset_donation, compute_sponsorship, expert_review, governance_participation, algorithm_contribution), review classes A-D, minimum requirements table, initiation process, data governance, and safety constraints. `src/openamp_foundry/community/contribution_intake.py` with `ContributionIntake` dataclass (7 fields), `IntakeValidationResult` dataclass (7 fields), `VALID_CONTRIBUTION_TYPES` (6 entries), `VALID_REVIEW_CLASSES` (4 entries), `REQUIRED_FIELDS_BY_TYPE` (6 entries), `validate_contribution_intake()` (checks all top-level fields, type-specific required fields, wet_lab_validation human_review_required=True enforcement), `validate_intake_dict()` (dict input with missing-fields guard). CLI (`openamp-foundry contribution-check`) with `--intake-json`, `--format text|json`. `make contribution-check` target. 16 tests. Dry-lab only. Funders and institutions now have a clear, validated pathway for contribution.
Expand Down Expand Up @@ -97,7 +98,7 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **New in v0.5.31:** Added dipeptide-order features for sequence-order awareness. `dipeptide_order_score` achieves AUROC 0.7861 on AMP-vs-scrambled discrimination — the strongest order-dependent feature in the pipeline. Only 7/31 features survive scrambling (amphipathicity/helix-wheel + dipeptide). All composition features are purely position-independent (exactly 0.5000 AUROC on scrambled test).
> **New in v0.5.30:** Easy baseline benchmark added — charge density alone (AUROC 0.8166) outperforms the full pipeline ensemble (0.7792) on AMP-vs-Swiss-Prot-decoy discrimination. Honest finding documented: expected because pipeline optimizes for safety, not raw discrimination.
> **New in v0.5.29:** Expanded benchmark to 500 AMPs + 500 composition-matched decoys (n=1000). AUROC 0.7792 (CI₉₅: 0.7505–0.8065) confirms signal generalizes. Cluster-aware CI: 0.746–0.8102. Representative AUROC: 0.778. Standard benchmark (n=191) retained for backward comparison.
> **Pipeline version:** v0.6.9 (J1 — release checklist, starts Phase J)
> **Pipeline version:** v0.7.0 (J2 — governance decision log)
> **Branch:** main

---
Expand Down
56 changes: 56 additions & 0 deletions docs/governance/DECISION_LOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Governance Decision Log

## Purpose

This document records all significant governance decisions made in the OpenAMP
Foundry project. It exists to make decisions discoverable, auditable, and
reviewable by both humans and automated tooling.

**Maintainer:** Project maintainers (via PR review).

**How to add entries:**

1. File a GitHub issue describing the proposed governance decision.
2. Obtain review from at least one project maintainer.
3. Add a new row to the decision index with a unique GOV-NNN ID.
4. Update `src/openamp_foundry/governance/decision_log.py` with the new
`GovernanceDecision` entry.
5. Run `make decision-log` to validate the entry.

## Decision Index

| ID | Date | Scope | Decision | Status | Rationale | Review class |
|----|------|-------|----------|--------|-----------|-------------|
| GOV-001 | 2026-07-01 | safety | All artifacts must have `dry_lab_only=True`; computational outputs are not biological proof. | active | Prevents overclaiming computational results as wet-lab evidence. | D |
| GOV-002 | 2026-07-01 | evidence | Dry-lab-only artifacts are capped at evidence level 4; levels 5-6 require wet-lab evidence. | active | Maintains honest proof-ladder levels that cannot be inflated by simulation alone. | D |
| GOV-003 | 2026-07-04 | benchmark | Any module claiming ranking authority must beat a cheap baseline benchmark first. | active | Prevents simulation theater — modules that don't beat trivial baselines must not rank candidates. | B |
| GOV-004 | 2026-07-09 | contribution | All `wet_lab_validation` contributions require `human_review_required=True`. | active | Experimental data entering the pipeline must be reviewed by a human before use. | D |
| GOV-005 | 2026-07-09 | release | All external releases must pass the release gate validator before publication. | active | Prevents skipping required checks during release; gates are documented and enforced. | B |
| GOV-006 | 2026-07-09 | docs | Documentation files must remain under 200KB each. | active | Keeps docs readable and prevents runaway history accumulation. | B |
| GOV-007 | 2026-07-01 | evidence | Proof-ladder levels above 4 require wet-lab evidence and human review class D. | active | Calibrates claim strength to actual experimental evidence. | D |
| GOV-008 | 2026-07-04 | adapter | Adapter default mode is off or info; gated mode requires benchmark comparison. | active | Prevents external adapters from silently influencing rankings without evidence. | B |

## How to Add a New Entry

1. **Open an issue** on GitHub describing the proposed decision, its scope,
rationale, and intended review class.
2. **Receive review** from at least one project maintainer. Review class D
decisions require documented human review.
3. **Assign a new GOV-NNN ID** (increment the highest existing number).
4. **Add the entry** to the table above and to the
`GOVERNANCE_DECISIONS` list in
`src/openamp_foundry/governance/decision_log.py`.
5. **Validate** with `make decision-log` (runs `openamp-foundry decision-log --validate`).
6. **Submit a PR** referencing the original issue.

## Linked Policies

- `docs/governance/GOVERNANCE.md` — Governance overview and role definitions.
- `SAFETY.md` — Safety policy and scope restrictions.
- `RESPONSIBLE_USE.md` — Allowed and disallowed uses.
- `docs/governance/RELEASE_CHECKLIST.md` — Release gate checklist (J1).

---

*This decision log is a dry-lab governance artifact. It documents policy
decisions, not biological findings. All entries carry `dry_lab_only: True`.*
76 changes: 76 additions & 0 deletions docs/research/ROADMAP.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,81 @@
# Roadmap

## v0.7.0 — Loop 110: Phase J J2 — Governance Decision Log ✓ (2026-07-09)

`docs/governance/DECISION_LOG.md` with structured governance decision log
(purpose, decision index with 8 entries GOV-001 through GOV-008 covering
safety/benchmark/release/evidence/data/adapter/contribution/docs scopes,
how to add entries, linked policies).

`src/openamp_foundry/governance/decision_log.py` with `VALID_SCOPES` (8:
safety, benchmark, release, evidence, data, adapter, contribution, docs),
`VALID_STATUSES` (4: active, superseded, under_review, proposed),
`VALID_REVIEW_CLASSES` (4: A, B, C, D), `GovernanceDecision` dataclass
(8 fields: decision_id, date, scope, decision, status, rationale,
review_class, dry_lab_only=True), `DecisionValidationResult` dataclass
(5 fields: decision_id, passed, errors, warnings, dry_lab_only=True),
`GOVERNANCE_DECISIONS` list (8 entries: GOV-001 through GOV-008),
`validate_governance_decision()` (9 checks: decision_id format, date
format, valid scope, non-empty decision, valid status, non-empty rationale,
valid review_class, dry_lab_only must be True, superseded warning),
`validate_all_decisions()` (aggregates total/passed/failed/all_passed/
results/dry_lab_only), `get_decisions_by_scope()` (filters by scope),
`get_decisions_by_status()` (filters by status).

CLI (`openamp-foundry decision-log`) with `--validate`, `--scope`,
`--format text|json`. Handler `_run_decision_log` in reports.py.

`make decision-log` target. 27 tests. **3491 total.**

**Phase J milestone: v0.7.0** — governance decisions are now discoverable
and machine-validated.

Changes:
- `docs/governance/DECISION_LOG.md` (J2) — Structured governance decision
log with purpose, decision index (8 entries GOV-001 through GOV-008),
how to add entries, linked policies.
- `src/openamp_foundry/governance/decision_log.py` (J2) — Core module with
`VALID_SCOPES` (8), `VALID_STATUSES` (4), `VALID_REVIEW_CLASSES` (4),
`GovernanceDecision` (8 fields, dry_lab_only=True default),
`DecisionValidationResult` (5 fields, dry_lab_only=True default),
`GOVERNANCE_DECISIONS` (8 entries: GOV-001 through GOV-008),
`validate_governance_decision()` (9 checks),
`validate_all_decisions()` (aggregates total/passed/failed/all_passed),
`get_decisions_by_scope()`, `get_decisions_by_status()`.
- `tests/governance/test_decision_log.py` (J2) — 27 tests covering all
8 GOV entries pass validation, empty/invalid decision_id, invalid date
format, invalid scope (parametrized), empty decision text, invalid
status (parametrized), empty rationale, invalid review_class
(parametrized), dry_lab_only=False failure, superseded warning,
validate_all_decisions passes, get_decisions_by_scope safety → GOV-001,
get_decisions_by_status active → all 8, 8 entries constant check,
all results dry_lab_only=True, valid set counts, DecisionValidationResult
dataclass fields.
- `src/openamp_foundry/cli/main.py` (J2) — Registered `decision-log`
subcommand with `--validate`, `--scope`, `--format` flags. Added import
and dispatch.
- `src/openamp_foundry/cli/commands/reports.py` (J2) — Added
`_run_decision_log()` CLI handler with validate/scope filtering/list all
modes, text and JSON output, exit code 3 on validation failure.
- `Makefile` (J2) — Added `decision-log` target. Added to `.PHONY`.
- `docs/evidence/METRICS_CURRENT.md` (J2) — v0.7.0 J2 changelog. Pipeline
version: v0.7.0. Test count: 3491.
- `tests/test_test_count_regression.py` — baseline updated to 3491.

Honest boundaries:
- Decision log tracks governance decisions only — it does not measure
biological activity, safety, or clinical value.
- `dry_lab_only: true` is a const field on all dataclasses — the decision
log is a computational governance artifact, not a biological finding.
- The list of valid scopes and statuses is policy-defined and may need
expansion as governance matures.
- Validation checks structural and policy requirements only — it does
not verify that the decision was correctly implemented or enforced.
- Review classes are declarations stored on each decision; the validator
does not verify that the declared review class was actually applied.
- The decision log is a documentation and validation tool — it does not
replace human judgment about whether a decision is appropriate.

## v0.6.9 — Loop 109: Phase J J1 — Release Checklist ✓ (2026-07-09)

`docs/governance/RELEASE_CHECKLIST.md` with structured release checklist
Expand Down
Loading
Loading