Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: help demo test lint ci clean bench-leakage bench-multi-negatives bench-baseline bench-hidden-active bench-cluster-split bench-expert-ablation bench-expert-ablation-500 bench-selectivity bench-feature-decomp bench-gate bench-easy-baseline bench-charge-matched bench-order-dependent bench-precision-at-k bench-per-family bench-simulation-ablation bench-simulation-ablation-within-amp simulation-gate bench-strategy-compare regenerate-all generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-pilot pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter recalibration-engine recalibration-engine-dry-run validate-policy-version generate-synthetic-lab-results bump-policy-version generate-review-packet failed-candidate-report safe-publication-filter classify-negative-informativeness validate-rejection-events negative-result-dashboard calibration-audit calibration-audit-example calibration-overfit-check result-quality-filter synthetic-result-policy-check calibration-decision-checklist calibration-rollback-plan simulation-registry validate-simulation-result-schema simulation-baseline-check adapter-gate-check simulation-provenance simulation-ensemble-check simulation-ci-report simulation-deprecation-check simulation-scope-check simulation-evidence-packet artifact-version candidate-manifest artifact-changelog integration-check adapter-author-check license-check artifact-compat-check contribution-check adoption-scorecard release-gate-check decision-log release-request-check
.PHONY: help demo test lint ci clean bench-leakage bench-multi-negatives bench-baseline bench-hidden-active bench-cluster-split bench-expert-ablation bench-expert-ablation-500 bench-selectivity bench-feature-decomp bench-gate bench-easy-baseline bench-charge-matched bench-order-dependent bench-precision-at-k bench-per-family bench-simulation-ablation bench-simulation-ablation-within-amp simulation-gate bench-strategy-compare regenerate-all generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-pilot pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter recalibration-engine recalibration-engine-dry-run validate-policy-version generate-synthetic-lab-results bump-policy-version generate-review-packet failed-candidate-report safe-publication-filter classify-negative-informativeness validate-rejection-events negative-result-dashboard calibration-audit calibration-audit-example calibration-overfit-check result-quality-filter synthetic-result-policy-check calibration-decision-checklist calibration-rollback-plan simulation-registry validate-simulation-result-schema simulation-baseline-check adapter-gate-check simulation-provenance simulation-ensemble-check simulation-ci-report simulation-deprecation-check simulation-scope-check simulation-evidence-packet artifact-version candidate-manifest artifact-changelog integration-check adapter-author-check license-check artifact-compat-check contribution-check adoption-scorecard release-gate-check decision-log release-request-check coi-check

PYTHON := $(shell [ -f .venv/bin/python ] && echo .venv/bin/python || echo python3)
PYTEST := $(shell [ -f .venv/bin/pytest ] && echo .venv/bin/pytest || echo pytest)
Expand Down Expand Up @@ -850,3 +850,8 @@ release-request-check:
.venv/bin/openamp-foundry release-request-check \
--request-json '{"release_id":"REL-2026-001","release_type":"schema","artifact_id":"candidate_manifest","artifact_version":"1.0.0","requestor_name":"OpenAMP Team","requestor_institution":"Open Problem Lab","request_date":"2026-07-09","evidence_level":2,"dry_lab_only":true,"safety_review_status":"approved","benchmark_summary":"Passes all schema compatibility checks","known_limitations":"Dry-lab only, not wet-lab validated","intended_use":"public","data_license":"CC-BY-4.0","human_reviewer":"maintainer","review_class":"B","approval_status":"pending"}'
@echo "Release request check complete."

coi-check:
.venv/bin/openamp-foundry coi-check \
--disclosure-json '{"disclosure_id":"COI-2026-001","disclosure_type":"reviewer","subject":"maintainer","related_artifact":"REL-2026-001","relationship_type":"none","description":"","disclosure_date":"2026-07-09","recusal_required":false,"reviewer":"lead-maintainer","review_status":"acknowledged"}'
@echo "COI check complete."
5 changes: 3 additions & 2 deletions docs/evidence/METRICS_CURRENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
> with this file, this file wins. Updated whenever benchmark/benchmark config changes.
>
> **Last updated:** 2026-07-09 (Phase J J3 — release request template — v0.7.1)
> **Last updated:** 2026-07-09 (Phase J J4 — COI disclosure template — v0.7.2)
> **New in v0.7.2:** COI disclosure template (J4 — Loop 112) — `docs/governance/COI_DISCLOSURE_TEMPLATE.md` with structured COI disclosure template (purpose, fill-in-the-blank template with 10 fields: Disclosure ID, disclosure type (reviewer|contributor|maintainer|external_advisor), subject, related artifact, relationship type (financial|institutional|competitive|personal|none), description (conditional), date, recusal required, reviewer, review status (pending|acknowledged|resolved); when to disclose section covering financial/institutional/competitive/personal relationships; process with machine validation and escalation; linked policies). `src/openamp_foundry/governance/coi_disclosure.py` with `COIDisclosure` dataclass (10 fields), `COIValidationResult` dataclass (6 fields, dry_lab_only=True), `VALID_DISCLOSURE_TYPES` (4: contributor, external_advisor, maintainer, reviewer), `VALID_RELATIONSHIP_TYPES` (5: competitive, financial, institutional, none, personal), `VALID_REVIEW_STATUSES` (3: acknowledged, pending, resolved), `validate_coi_disclosure()` (10 checks including disclosure_id COI- prefix, valid disclosure_type, non-empty subject/related_artifact, valid relationship_type, description required unless none, YYYY-MM-DD date, non-empty reviewer, valid review_status, dry_lab_only must be True; financial without recusal warning), `validate_coi_dict()` (dict input with 10 required fields guard, missing fields returns passed=False early). CLI (`openamp-foundry coi-check`) with `--disclosure-json`, `--format text|json`. `make coi-check` target. 20 tests. Dry-lab only. COI disclosures now have a validated structure that builds institutional trust.
> **New in v0.7.1:** Release request template (J3 — Loop 111) — `docs/governance/RELEASE_REQUEST_TEMPLATE.md` with structured release request template (purpose, fill-in-the-blank template with 17 fields: Release ID, release type, artifact ID/version, requestor name/institution, request date, evidence level 1-6, dry_lab_only, safety_review_status, benchmark_summary, known_limitations, intended_use, data_license, human_reviewer, review_class, approval_status; review criteria with 8 checks; process with classes A-D timelines and escalation path). `src/openamp_foundry/governance/release_request.py` with `ReleaseRequest` dataclass (17 fields), `ReleaseRequestValidationResult` dataclass (6 fields, dry_lab_only=True), `VALID_RELEASE_TYPES` (5), `VALID_SAFETY_STATUSES` (3), `VALID_INTENDED_USES` (4), `VALID_APPROVAL_STATUSES` (4), `VALID_REVIEW_CLASSES` (4), `validate_release_request()` (17 checks including dry_lab_only+evidence_level>4 error, public+safety_pending error, model+review_class warning), `validate_request_dict()` (dict input with missing-fields guard). CLI (`openamp-foundry release-request-check`) with `--request-json`, `--format text|json`. `make release-request-check` target. 25 tests. **3516 total.** Formal release requests now have a validated structure before entering human review.
> **New in v0.7.0:** Governance decision log index (J2 — Loop 110, Phase J milestone) — `docs/governance/DECISION_LOG.md` with structured decision log (purpose, decision index with 8 entries GOV-001 through GOV-008 covering safety/benchmark/release/evidence/data/adapter/contribution/docs scopes, how to add entries, linked policies). `src/openamp_foundry/governance/decision_log.py` with `VALID_SCOPES` (8 entries: safety, benchmark, release, evidence, data, adapter, contribution, docs), `VALID_STATUSES` (4 entries: active, superseded, under_review, proposed), `VALID_REVIEW_CLASSES` (4 entries: A, B, C, D), `GovernanceDecision` dataclass (8 fields: decision_id, date, scope, decision, status, rationale, review_class, dry_lab_only=True), `DecisionValidationResult` dataclass (5 fields: decision_id, passed, errors, warnings, dry_lab_only=True), `GOVERNANCE_DECISIONS` list (8 entries: GOV-001 through GOV-008), `validate_governance_decision()` (9 checks: decision_id format, date format, valid scope, non-empty decision, valid status, non-empty rationale, valid review_class, dry_lab_only must be True, superseded warning), `validate_all_decisions()` (aggregates total/passed/failed/all_passed/results/dry_lab_only), `get_decisions_by_scope()` (filters by scope), `get_decisions_by_status()` (filters by status). CLI (`openamp-foundry decision-log`) with `--validate`, `--scope`, `--format text|json`. `make decision-log` target. 27 tests. **3505 total.** **Phase J milestone: v0.7.0** — governance decisions are now discoverable and machine-validated.
> **New in v0.6.9:** Release checklist and gate validator (J1 — Loop 109, starts Phase J) — `docs/governance/RELEASE_CHECKLIST.md` with structured checklist (pre-release gates, release-type gates for 5 types, post-release checklist), cross-referencing `docs/trust/RELEASE_CHECKLIST.md`. `src/openamp_foundry/governance/release_gate.py` with `RELEASE_TYPES` (5: candidate, model, dataset, evidence_packet, schema), `UNIVERSAL_GATES` (7: ci_tests_pass, agent_check_passes, no_critical_issues, dry_lab_only_confirmed, safety_flags_reviewed, data_license_verified, no_hardcoded_secrets), `EXTRA_GATES_BY_TYPE` (per-type additional gates), `ReleaseGateResult` dataclass (8 fields, dry_lab_only=True), `validate_release_gate()` (validates all required gates, treats missing gates as failed, raises CRITICAL error on dry_lab_only_confirmed failure). CLI (`openamp-foundry release-gate-check`) with `--release-type`, `--artifact-id`, `--gates-json`, `--format text|json`. `make release-gate-check` target. 18 tests. **3478 total.** **Starts Phase J (Governance and release maturity)** — releases now require all gates to pass before external release, preventing accidental bypass of required checks.
Expand Down Expand Up @@ -99,7 +100,7 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **New in v0.5.31:** Added dipeptide-order features for sequence-order awareness. `dipeptide_order_score` achieves AUROC 0.7861 on AMP-vs-scrambled discrimination — the strongest order-dependent feature in the pipeline. Only 7/31 features survive scrambling (amphipathicity/helix-wheel + dipeptide). All composition features are purely position-independent (exactly 0.5000 AUROC on scrambled test).
> **New in v0.5.30:** Easy baseline benchmark added — charge density alone (AUROC 0.8166) outperforms the full pipeline ensemble (0.7792) on AMP-vs-Swiss-Prot-decoy discrimination. Honest finding documented: expected because pipeline optimizes for safety, not raw discrimination.
> **New in v0.5.29:** Expanded benchmark to 500 AMPs + 500 composition-matched decoys (n=1000). AUROC 0.7792 (CI₉₅: 0.7505–0.8065) confirms signal generalizes. Cluster-aware CI: 0.746–0.8102. Representative AUROC: 0.778. Standard benchmark (n=191) retained for backward comparison.
> **Pipeline version:** v0.7.1 (J3 — release request template)
> **Pipeline version:** v0.7.2 (J4 — COI disclosure template)
> **Branch:** main

---
Expand Down
63 changes: 63 additions & 0 deletions docs/governance/COI_DISCLOSURE_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Conflict-of-Interest Disclosure Template

## Purpose

A conflict-of-interest (COI) disclosure is filed when a contributor, reviewer, maintainer, or external advisor has a potential conflict with an artifact, decision, or review they are involved with. Filing a disclosure enables transparent governance and builds institutional trust.

**File a disclosure when you have:**
- A financial interest in an artifact or organization related to the work
- An institutional relationship (current or former employer, funder) with a stakeholder
- A competitive relationship with parties being evaluated
- A personal relationship that could influence objectivity

**Who reviews:** the designated `reviewer` (GitHub handle) noted in the disclosure.

---

## Template

Fill in every field. Fields marked REQUIRED must not be empty.

```yaml
disclosure_id: "" # e.g. COI-2026-001 (REQUIRED)
disclosure_type: "" # reviewer | contributor | maintainer | external_advisor (REQUIRED)
subject: "" # GitHub handle of person filing disclosure (REQUIRED)
related_artifact: "" # artifact ID or PR number (REQUIRED)
relationship_type: "" # financial | institutional | competitive | personal | none (REQUIRED)
description: "" # Required when relationship_type != "none"; leave empty only if "none"
disclosure_date: "" # YYYY-MM-DD (REQUIRED)
recusal_required: false # true | false (REQUIRED)
reviewer: "" # GitHub handle of designated reviewer (REQUIRED)
review_status: "" # pending | acknowledged | resolved (REQUIRED)
```

---

## When to Disclose

| Relationship type | Examples | Typical action |
|---|---|---|
| financial | equity, grants, consulting fees from a related org | Disclose; recusal often required |
| institutional | current/former employer, department, funder | Disclose; recusal case-by-case |
| competitive | org whose product competes with the artifact | Disclose; recusal recommended |
| personal | close family, romantic partner, close friend | Disclose; recusal case-by-case |
| none | no conflict | File with description="" to document clean slate |

---

## Process

1. **File the disclosure** — copy the template above, fill in all fields, open a GitHub issue titled `[COI Disclosure] COI-YYYY-NNN — <subject>`.
2. **Machine validation** — run `openamp-foundry coi-check --disclosure-json '{...}'` to pre-validate.
3. **Human review** — the designated `reviewer` acknowledges within 3 business days.
4. **Decision** — reviewer sets `review_status` to `acknowledged` or `resolved` with written rationale on recusal.
5. **Escalation** — if no acknowledgment within 5 business days, escalate to project maintainers via GitHub discussion.

---

## Linked Policies

- [GOVERNANCE.md](../../GOVERNANCE.md)
- [RESPONSIBLE_USE.md](../../RESPONSIBLE_USE.md)
- [RELEASE_CHECKLIST.md](RELEASE_CHECKLIST.md)
- [DECISION_LOG.md](DECISION_LOG.md)
71 changes: 71 additions & 0 deletions docs/research/ROADMAP.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,76 @@
# Roadmap

## v0.7.2 — Loop 112: Phase J J4 — COI Disclosure Template ✓ (2026-07-09)

`docs/governance/COI_DISCLOSURE_TEMPLATE.md` with structured COI disclosure
template (purpose, fill-in-the-blank format with 10 fields: Disclosure ID
COI-YYYY-NNN, disclosure type reviewer|contributor|maintainer|external_advisor,
subject GitHub handle, related artifact or PR, relationship type
financial|institutional|competitive|personal|none, description (required unless
none), date YYYY-MM-DD, recusal_required true|false, reviewer GitHub handle,
review_status pending|acknowledged|resolved).

`src/openamp_foundry/governance/coi_disclosure.py` with `COIDisclosure`
dataclass (10 fields), `COIValidationResult` dataclass (6 fields,
dry_lab_only=True), `VALID_DISCLOSURE_TYPES` (4: contributor, external_advisor,
maintainer, reviewer), `VALID_RELATIONSHIP_TYPES` (5: competitive, financial,
institutional, none, personal), `VALID_REVIEW_STATUSES` (3: acknowledged,
pending, resolved), `validate_coi_disclosure()` (10 checks: disclosure_id
starts with COI-, valid disclosure_type, non-empty subject/related_artifact,
valid relationship_type, description required unless none, YYYY-MM-DD date,
non-empty reviewer, valid review_status, dry_lab_only must be True; financial
without recusal yields warning not error), `validate_coi_dict()` (dict input
with 10 required fields guard).

CLI (`openamp-foundry coi-check`) with `--disclosure-json` (required),
`--format text|json`. Handler `_run_coi_check` in reports.py.

`make coi-check` target. 20 tests. **3536 total.**

COI disclosures now have a validated structure that builds institutional trust.

Changes:
- `docs/governance/COI_DISCLOSURE_TEMPLATE.md` (J4) — Structured COI disclosure
template with purpose, template (10 fields), when to disclose (financial,
institutional, competitive, personal), process (5 steps with escalation).
- `src/openamp_foundry/governance/coi_disclosure.py` (J4) — Core module with
`COIDisclosure` (10 fields), `COIValidationResult` (6 fields,
dry_lab_only=True), `VALID_DISCLOSURE_TYPES` (4), `VALID_RELATIONSHIP_TYPES`
(5), `VALID_REVIEW_STATUSES` (3), `validate_coi_disclosure()` (10 checks),
`validate_coi_dict()` (dict input with 10 required fields guard).
- `tests/governance/test_coi_disclosure.py` (J4) — 20 tests covering: valid
reviewer none relationship passes, valid contributor financial passes,
disclosure_id not starting with COI- fails, empty disclosure_id fails, invalid
disclosure_type fails, empty subject fails, empty related_artifact fails,
invalid relationship_type fails, relationship not none with empty description
fails, relationship none with empty description passes, invalid date format
fails, empty reviewer fails, invalid review_status fails, dry_lab_only=False
fails, financial without recusal warns, validate_coi_dict passes, validate_
coi_dict with missing fields fails, all results dry_lab_only=True,
VALID_DISCLOSURE_TYPES has 4, VALID_RELATIONSHIP_TYPES has 5.
- `src/openamp_foundry/cli/main.py` (J4) — Registered `coi-check` subcommand
with `--disclosure-json`, `--format` flags. Added import and dispatch.
- `src/openamp_foundry/cli/commands/reports.py` (J4) — Added `_run_coi_check()`
CLI handler with JSON parsing, validate_coi_dict call, text and JSON output,
exit code 3 on validation failure.
- `Makefile` (J4) — Added `coi-check` target. Added to `.PHONY`.
- `docs/evidence/METRICS_CURRENT.md` (J4) — v0.7.2 J4 changelog. Pipeline
version: v0.7.2. Test count: 3536.
- `tests/test_test_count_regression.py` — baseline updated to 3536.

Honest boundaries:
- COI disclosure validation checks structural and policy requirements only.
It does not verify that the disclosed information is true, complete, or
accurate.
- Financial relationship without recusal produces a warning, not an error —
the reviewer retains discretion to determine whether recusal is necessary.
- The validator cannot detect undisclosed conflicts — it only checks that
disclosed conflicts are well-formed.
- `dry_lab_only: true` is a const field on all dataclasses — COI disclosures
are governance artifacts, not legal determinations.
- The COI template is a transparency and governance tool — it does not replace
the judgment of the human reviewer or governance team.

## v0.7.1 — Loop 111: Phase J J3 — Release Request Template ✓ (2026-07-09)

`docs/governance/RELEASE_REQUEST_TEMPLATE.md` with structured release request
Expand Down
Loading
Loading