Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: help demo test lint ci clean bench-leakage bench-multi-negatives bench-baseline bench-hidden-active bench-cluster-split bench-expert-ablation bench-expert-ablation-500 bench-selectivity bench-feature-decomp bench-gate bench-easy-baseline bench-charge-matched bench-order-dependent bench-precision-at-k bench-per-family bench-simulation-ablation bench-simulation-ablation-within-amp simulation-gate bench-strategy-compare regenerate-all generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-pilot pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter recalibration-engine recalibration-engine-dry-run validate-policy-version generate-synthetic-lab-results bump-policy-version generate-review-packet failed-candidate-report safe-publication-filter classify-negative-informativeness validate-rejection-events negative-result-dashboard calibration-audit calibration-audit-example calibration-overfit-check result-quality-filter synthetic-result-policy-check calibration-decision-checklist calibration-rollback-plan simulation-registry validate-simulation-result-schema simulation-baseline-check adapter-gate-check simulation-provenance simulation-ensemble-check simulation-ci-report simulation-deprecation-check simulation-scope-check simulation-evidence-packet artifact-version candidate-manifest artifact-changelog integration-check adapter-author-check license-check artifact-compat-check contribution-check
.PHONY: help demo test lint ci clean bench-leakage bench-multi-negatives bench-baseline bench-hidden-active bench-cluster-split bench-expert-ablation bench-expert-ablation-500 bench-selectivity bench-feature-decomp bench-gate bench-easy-baseline bench-charge-matched bench-order-dependent bench-precision-at-k bench-per-family bench-simulation-ablation bench-simulation-ablation-within-amp simulation-gate bench-strategy-compare regenerate-all generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-pilot pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter recalibration-engine recalibration-engine-dry-run validate-policy-version generate-synthetic-lab-results bump-policy-version generate-review-packet failed-candidate-report safe-publication-filter classify-negative-informativeness validate-rejection-events negative-result-dashboard calibration-audit calibration-audit-example calibration-overfit-check result-quality-filter synthetic-result-policy-check calibration-decision-checklist calibration-rollback-plan simulation-registry validate-simulation-result-schema simulation-baseline-check adapter-gate-check simulation-provenance simulation-ensemble-check simulation-ci-report simulation-deprecation-check simulation-scope-check simulation-evidence-packet artifact-version candidate-manifest artifact-changelog integration-check adapter-author-check license-check artifact-compat-check contribution-check adoption-scorecard

PYTHON := $(shell [ -f .venv/bin/python ] && echo .venv/bin/python || echo python3)
PYTEST := $(shell [ -f .venv/bin/pytest ] && echo .venv/bin/pytest || echo pytest)
Expand Down Expand Up @@ -829,3 +829,8 @@ contribution-check:
.venv/bin/openamp-foundry contribution-check \
--intake-json '{"institution_name":"Example University","contact_email":"research@example.edu","contribution_type":"dataset_donation","proposed_scope":"Curated AMP dataset 500 sequences","human_review_required":false,"dry_lab_only":true,"data_license":"CC-BY-4.0","dataset_description":"AMP sequences with MIC data","record_count":500}'
@echo "Contribution intake check complete."

adoption-scorecard:
.venv/bin/openamp-foundry adoption-scorecard \
--scores-json '{"integration_check":{"passed_checks":5,"total_checks":5,"notes":""},"license_compliance":{"passed_checks":3,"total_checks":3,"notes":""},"adapter_validation":{"passed_checks":2,"total_checks":2,"notes":""},"schema_compatibility":{"passed_checks":4,"total_checks":4,"notes":""},"contribution_readiness":{"passed_checks":1,"total_checks":1,"notes":""}}'
@echo "Adoption scorecard complete."
8 changes: 5 additions & 3 deletions docs/evidence/METRICS_CURRENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
> with this file, this file wins. Updated whenever benchmark/benchmark config changes.
>
> **Last updated:** 2026-07-09 (Phase I I9 — public-good contribution guide — v0.6.7)
> **Last updated:** 2026-07-09 (Phase I I10 — adoption scorecard dashboard — v0.6.8)
> **New in v0.6.8:** Adoption scorecard dashboard (I10 — Loop 108) — `src/openamp_foundry/adoption/scorecard.py` with `SCORECARD_DIMENSIONS` (5 weighted dimensions summing to 1.0: integration_check 0.25, license_compliance 0.20, adapter_validation 0.20, schema_compatibility 0.20, contribution_readiness 0.15), `ADOPTION_TIERS` (4 tiers: not_ready 0.0-0.40, emerging 0.40-0.65, established 0.65-0.85, mature 0.85-1.01), `DimensionScore` dataclass (8 fields, dry_lab_only=True), `AdoptionScorecard` dataclass (6 fields, dry_lab_only=True), `build_scorecard()` (aggregates dimension inputs into weighted total score with tier and actionable recommendations), `compute_adoption_tier()` (maps score to tier). CLI (`openamp-foundry adoption-scorecard`) with `--scores-json`, `--format text|json`. `make adoption-scorecard` target. 17 tests. **3446 total.** **Phase I (Interoperability and Adoption) is now complete** — all 10 items I1–I10 implemented (artifact versioning, candidate manifest, benchmark card, artifact changelog, integration checker, adapter validator, data license checker, schema compatibility, contribution intake, adoption scorecard).
> **New in v0.6.7:** Public-good contribution guide (I9 — Loop 107) — `docs/community/PUBLIC_GOOD_CONTRIBUTION_GUIDE.md` with 6 contribution types (wet_lab_validation, dataset_donation, compute_sponsorship, expert_review, governance_participation, algorithm_contribution), review classes A-D, minimum requirements table, initiation process, data governance, and safety constraints. `src/openamp_foundry/community/contribution_intake.py` with `ContributionIntake` dataclass (7 fields), `IntakeValidationResult` dataclass (7 fields), `VALID_CONTRIBUTION_TYPES` (6 entries), `VALID_REVIEW_CLASSES` (4 entries), `REQUIRED_FIELDS_BY_TYPE` (6 entries), `validate_contribution_intake()` (checks all top-level fields, type-specific required fields, wet_lab_validation human_review_required=True enforcement), `validate_intake_dict()` (dict input with missing-fields guard). CLI (`openamp-foundry contribution-check`) with `--intake-json`, `--format text|json`. `make contribution-check` target. 16 tests. Dry-lab only. Funders and institutions now have a clear, validated pathway for contribution.
> **New in v0.6.5:** Data license checker (I7 — Loop 105) — `src/openamp_foundry/licensing/license_checker.py` with `DataLicenseDeclaration` dataclass (11 fields: source_id, source_name, license_id, use_context, attribution_required, commercial_use_allowed, redistribution_allowed, modifications_allowed, human_review_required, notes, dry_lab_only), `LicenseCheckResult` dataclass (8 fields: source_id, license_id, use_context, passed, status, errors, warnings, dry_lab_only), `check_data_license()` (validates declarations against `APPROVED_LICENSES` (11 entries: CC0-1.0, CC-BY-4.0, CC-BY-SA-4.0, MIT, Apache-2.0, GPL-3.0, LGPL-2.1, BSD-2-Clause, BSD-3-Clause, ODbL-1.0, PDDL-1.0), `RESTRICTED_LICENSES` (4 entries: CC-BY-NC-4.0, CC-BY-NC-SA-4.0, custom, proprietary), `BLOCKED_LICENSES` (3 entries: unknown, unlicensed, all-rights-reserved), `VALID_USE_CONTEXTS` (6 entries: training, scoring, benchmarking, reporting, publication, internal). Blocked licenses fail immediately; restricted require human_review_required=True; unknown licenses require governance review before use. `check_license_batch()` summarizes total/passed/failed/blocked/any_blocked/all_passed with dry_lab_only=True. All results carry dry_lab_only=True. CLI (`openamp-foundry license-check`) with `--source-json`, `--format text|json`. `make license-check` target. 16 tests. **3429 total.** External data sources used in pipeline outputs now require explicit license declarations, preventing hidden legal risk.
> **New in v0.6.3:** Downstream project template (I5 — Loop 103) — `docs/adoption/DOWNSTREAM_PROJECT_TEMPLATE.md` with overview of OpenAMP artifacts, minimum viable integration (consume candidate manifest, validate against schema, use Python library), schema validation step, evidence level interpretation table (L1-L6), safety flag conventions table, benchmark card consumption guide, explicit dry-lab limitations section, and contact/contribution section. `src/openamp_foundry/adoption/integration_checker.py` with `REQUIRED_INTEGRATION_CHECKS` (5 checks: manifest_schema_valid, evidence_level_in_range, dry_lab_only_acknowledged, safety_flags_reviewed, baseline_comparison_present), `IntegrationCheckResult` dataclass (4 fields: check_name, passed, detail, dry_lab_only), `run_integration_checks()` (returns dict with checks, passed_count, failed_count, all_passed, dry_lab_only). Exported from `src/openamp_foundry/adoption/__init__.py`. CLI (`openamp-foundry integration-check`) with `--manifest-json`, `--format text|json`. `make integration-check` target. 14 tests. **3356 total.** External researchers who want to use OpenAMP artifacts now have a guide and checker to validate their integration.
Expand Down Expand Up @@ -95,7 +96,7 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **New in v0.5.31:** Added dipeptide-order features for sequence-order awareness. `dipeptide_order_score` achieves AUROC 0.7861 on AMP-vs-scrambled discrimination — the strongest order-dependent feature in the pipeline. Only 7/31 features survive scrambling (amphipathicity/helix-wheel + dipeptide). All composition features are purely position-independent (exactly 0.5000 AUROC on scrambled test).
> **New in v0.5.30:** Easy baseline benchmark added — charge density alone (AUROC 0.8166) outperforms the full pipeline ensemble (0.7792) on AMP-vs-Swiss-Prot-decoy discrimination. Honest finding documented: expected because pipeline optimizes for safety, not raw discrimination.
> **New in v0.5.29:** Expanded benchmark to 500 AMPs + 500 composition-matched decoys (n=1000). AUROC 0.7792 (CI₉₅: 0.7505–0.8065) confirms signal generalizes. Cluster-aware CI: 0.746–0.8102. Representative AUROC: 0.778. Standard benchmark (n=191) retained for backward comparison.
> **Pipeline version:** v0.6.7 (I9 — public-good contribution guide)
> **Pipeline version:** v0.6.8 (I10 — adoption scorecard dashboard, Phase I complete)
> **Branch:** main

---
Expand Down Expand Up @@ -1165,7 +1166,7 @@ Decoys score low on activity. Selective AMPs score moderately on both.

| Metric | Value |
|--------|-------|
| Total tests | 3413 |
| Total tests | 3446 |
| Coverage (branch) | 99% (6 CLI guard lines only) |
| Source modules at 100% | All pipeline, QC, scoring, adapter modules |

Expand All @@ -1189,6 +1190,7 @@ Decoys score low on activity. Selective AMPs score moderately on both.

| Date | Change | Author |
|------|--------|--------|
| 2026-07-09 | **Adoption scorecard dashboard added (I10):** `src/openamp_foundry/adoption/scorecard.py` with `SCORECARD_DIMENSIONS` (5, weights sum 1.0), `ADOPTION_TIERS` (4), `DimensionScore`, `AdoptionScorecard`, `build_scorecard()`, `compute_adoption_tier()`. CLI `adoption-scorecard` with `--scores-json` and `--format`. `make adoption-scorecard`. 17 tests. **Phase I (Interoperability and Adoption) is now complete** — all 10 items I1–I10 implemented. **3446 total.** | OpenAMP Loop 108 |
| 2026-07-09 | **Adapter author validator added (I6):** `src/openamp_foundry/adapters/adapter_validator.py` with `AdapterDeclaration` (14 fields), `AdapterValidationResult` (5 fields), `validate_adapter_declaration()` (10 checks enforcing ADAPTER_AUTHOR_GUIDE contract), `validate_adapter_dict()` (dict input with missing-fields guard). 4 valid-value sets (VALID_ADAPTER_MODES, VALID_OUTPUT_STATUSES, VALID_RANKING_EFFECTS, VALID_RELEASE_STATUSES). CLI (`openamp-foundry adapter-check`) with `--adapter-json`, `--format text|json`. `make adapter-author-check` target. 31 tests. **3387 total.** | OpenAMP Loop 104 |
| 2026-07-08 | **Calibration benchmark added:** Brier score decomposition, reliability diagram, and calibration slope for pipeline ensemble scores. Brier=0.3178 (>0.25=uninformative), skill=-0.27 (worse than base rate), slope=0.43 (ideal=1.0). Honest finding: pipeline ranks well (AUROC~0.78) but scores are not meaningful probabilities. Expanded 500-AMP set confirms same pattern (Brier=0.2772, slope=2.31 — dataset-dependent). Integrated into `make bench-500` and `make bench-calibration`. `scripts/benchmarks/benchmark_calibration.py`, JSON output to `outputs/bench_calibration*.json`. | OpenAMP loop 18 |
| 2026-06-29 | Novelty audit v2: BioPython BLOSUM62 local alignment vs 27,234 AMPs (APD6+DRAMP+UniProt); panel updated (15 families, 4 SAR_CONTROL); all 7 gates PASS | OpenAMP Wave 0.5 |
Expand Down
68 changes: 68 additions & 0 deletions docs/research/ROADMAP.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,73 @@
# Roadmap

## v0.6.8 — Loop 108: Phase I I10 — Adoption Scorecard Dashboard ✓ (2026-07-09)

`src/openamp_foundry/adoption/scorecard.py` with `SCORECARD_DIMENSIONS`
(5 weighted dimensions summing to 1.0: integration_check 0.25,
license_compliance 0.20, adapter_validation 0.20, schema_compatibility 0.20,
contribution_readiness 0.15), `ADOPTION_TIERS` (4 tiers: not_ready 0.0-0.40,
emerging 0.40-0.65, established 0.65-0.85, mature 0.85-1.01),
`DimensionScore` dataclass (8 fields: dimension, weight, raw_score,
weighted_score, passed_checks, total_checks, notes, dry_lab_only=True),
`AdoptionScorecard` dataclass (6 fields: total_score, adoption_tier,
dimensions, summary, recommendations, dry_lab_only=True),
`build_scorecard()` (aggregates dimension inputs into weighted total score
with tier classification and actionable recommendations),
`compute_adoption_tier()` (maps float score 0.0-1.0 to tier string).

CLI (`openamp-foundry adoption-scorecard`) with `--scores-json` and
`--format text|json`. Handler `_run_adoption_scorecard` in reports.py.

`make adoption-scorecard` target. 17 tests. **3446 total.**

**Phase I (Interoperability and Adoption) is now fully complete** — all 10
items I1–I10 implemented (artifact versioning, candidate manifest, benchmark
card, artifact changelog, integration checker, adapter validator, data license
checker, schema compatibility, contribution intake, adoption scorecard).

Changes:
- `src/openamp_foundry/adoption/scorecard.py` (I10) — Core module with
`SCORECARD_DIMENSIONS` (5 weighted dimensions summing to 1.0),
`ADOPTION_TIERS` (4 tiers), `DimensionScore` (8 fields, dry_lab_only=True
default), `AdoptionScorecard` (6 fields, dry_lab_only=True default),
`build_scorecard()` (aggregates dimension inputs into weighted total score,
tier, and per-dimension scores with recommendations),
`compute_adoption_tier()` (maps float 0.0-1.0 to tier string).
- `tests/adoption/__init__.py` (I10) — Empty package init.
- `tests/adoption/test_scorecard.py` (I10) — 17 tests covering: perfect scores
→ mature, all zeros → not_ready, score 0.50 → emerging, score 0.75 →
established, weights sum to 1.0, SCORECARD_DIMENSIONS has 5 entries,
ADOPTION_TIERS has 4 entries, DimensionScore dry_lab_only=True default,
AdoptionScorecard dry_lab_only=True default, recommendations empty when
all pass, recommendations populated on failures, weighted sum correctness,
missing dimension defaults to 0/0, compute_adoption_tier boundaries,
build_scorecard returns correct fields.
- `src/openamp_foundry/cli/main.py` (I10) — Registered `adoption-scorecard`
subcommand with `--scores-json` (required), `--format` flags. Added import
and dispatch.
- `src/openamp_foundry/cli/commands/reports.py` (I10) — Added
`_run_adoption_scorecard()` CLI handler with JSON parsing, `build_scorecard`
call, text and JSON output.
- `Makefile` (I10) — Added `adoption-scorecard` target with demo invocation
using all 5 dimensions at perfect scores. Added to `.PHONY`.
- `docs/evidence/METRICS_CURRENT.md` (I10) — v0.6.8 I10 changelog. Pipeline
version: v0.6.8. Test count: 3446. Phase I complete note.
- `tests/test_test_count_regression.py` — baseline updated to 3446.

Honest boundaries:
- Scorecard measures adoption readiness only — whether the checks pass or fail.
It does not measure actual downstream adoption, user satisfaction, or
biological validity.
- Dimension weights are heuristic and may need adjustment as adoption data
accumulates. The current weights (integration 0.25, license 0.20, adapter
0.20, schema 0.20, contribution 0.15) are the initial guess.
- `dry_lab_only: true` is a const field on all dataclasses — the scorecard is
a computational governance tool, not a biological assessment.
- The scorecard aggregates structural signals from other Phase I modules. If
any of those modules have blind spots, the aggregation inherits them.
- Perfect scores mean all structural checks pass, not that real-world adoption
is happening. Adoption is a social outcome, not a pipeline metric.

## v0.6.7 — Loop 107: Phase I I9 — Public-Good Contribution Guide ✓ (2026-07-09)

`docs/community/PUBLIC_GOOD_CONTRIBUTION_GUIDE.md` with 6 contribution types
Expand Down
Loading
Loading