Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: help demo test lint ci clean bench-leakage bench-multi-negatives bench-baseline bench-hidden-active bench-cluster-split bench-expert-ablation bench-expert-ablation-500 bench-selectivity bench-feature-decomp bench-gate bench-easy-baseline bench-charge-matched bench-order-dependent bench-precision-at-k bench-per-family bench-simulation-ablation bench-simulation-ablation-within-amp simulation-gate bench-strategy-compare regenerate-all generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-pilot pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter recalibration-engine recalibration-engine-dry-run validate-policy-version generate-synthetic-lab-results bump-policy-version generate-review-packet failed-candidate-report safe-publication-filter classify-negative-informativeness validate-rejection-events negative-result-dashboard calibration-audit calibration-audit-example calibration-overfit-check result-quality-filter synthetic-result-policy-check calibration-decision-checklist calibration-rollback-plan simulation-registry validate-simulation-result-schema simulation-baseline-check adapter-gate-check simulation-provenance simulation-ensemble-check simulation-ci-report simulation-deprecation-check simulation-scope-check simulation-evidence-packet artifact-version candidate-manifest artifact-changelog integration-check adapter-author-check license-check artifact-compat-check contribution-check adoption-scorecard release-gate-check decision-log release-request-check coi-check rotation-plan-check security-report-check citation-check roadmap-sync-check advisory-review-check annual-review-check selection-rationale-check batch-priority-check pilot-package-check calibration-intake-check uncertainty-report-check preprint-bundle-check reproducibility-manifest-check candidate-summary-card-check multi-candidate-comparison-check pipeline-decision-audit-check claim-to-evidence-check score-decomposition-check reviewer-briefing-check dataset-release-check
.PHONY: help demo test lint ci clean bench-leakage bench-multi-negatives bench-baseline bench-hidden-active bench-cluster-split bench-expert-ablation bench-expert-ablation-500 bench-selectivity bench-feature-decomp bench-gate bench-easy-baseline bench-charge-matched bench-order-dependent bench-precision-at-k bench-per-family bench-simulation-ablation bench-simulation-ablation-within-amp simulation-gate bench-strategy-compare regenerate-all generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-pilot pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter recalibration-engine recalibration-engine-dry-run validate-policy-version generate-synthetic-lab-results bump-policy-version generate-review-packet failed-candidate-report safe-publication-filter classify-negative-informativeness validate-rejection-events negative-result-dashboard calibration-audit calibration-audit-example calibration-overfit-check result-quality-filter synthetic-result-policy-check calibration-decision-checklist calibration-rollback-plan simulation-registry validate-simulation-result-schema simulation-baseline-check adapter-gate-check simulation-provenance simulation-ensemble-check simulation-ci-report simulation-deprecation-check simulation-scope-check simulation-evidence-packet artifact-version candidate-manifest artifact-changelog integration-check adapter-author-check license-check artifact-compat-check contribution-check adoption-scorecard release-gate-check decision-log release-request-check coi-check rotation-plan-check security-report-check citation-check roadmap-sync-check advisory-review-check annual-review-check selection-rationale-check batch-priority-check pilot-package-check calibration-intake-check uncertainty-report-check preprint-bundle-check reproducibility-manifest-check candidate-summary-card-check multi-candidate-comparison-check pipeline-decision-audit-check claim-to-evidence-check score-decomposition-check reviewer-briefing-check audit-chain-check dataset-release-check

PYTHON := $(shell [ -f .venv/bin/python ] && echo .venv/bin/python || echo python3)
PYTEST := $(shell [ -f .venv/bin/pytest ] && echo .venv/bin/pytest || echo pytest)
Expand Down Expand Up @@ -945,6 +945,9 @@ score-decomposition-check:
reviewer-briefing-check:
.venv/bin/openamp-foundry reviewer-briefing-check --entry-json '{"briefing_id":"RBP-001","batch_id":"BATCH-001","pipeline_version":"0.9.2","prepared_date":"2026-07-10","reviewer_name":"Dr. Test Reviewer","candidate_count":10,"artifact_ids":["CERT-001","SEL-001","SDR-001"],"open_questions":["Is selection rationale adequate?"],"scope_description":"Review candidate evidence packages for BATCH-001.","conflict_of_interest_declared":true,"dry_lab_only":true}' --format text

audit-chain-check:
.venv/bin/openamp-foundry audit-chain-check --entry-json '{"chain_id":"ACH-001","batch_id":"BATCH-001","pipeline_version":"0.9.3","audit_date":"2026-07-10","has_sequence_input":true,"has_benchmark_results":true,"has_filter_log":true,"has_score_decomposition":true,"has_selection_rationale":true,"has_evidence_certificate":true,"has_claim_mappings":true,"has_pipeline_decision_audit":true,"has_reviewer_briefing":true,"missing_links":[],"auditor":"test@example.com","dry_lab_only":true}' --format text

dataset-release-check:
.venv/bin/openamp-foundry dataset-release-check \
--entry-json '{"release_id":"DSR-001","dataset_name":"OpenAMP Wave 0.5 Candidates","dataset_version":"1.0.0","release_date":"2026-07-09","license_identifier":"CC-BY-4.0","data_sources":["UniProt AMP database","APD3","in-house pipeline"],"contains_sequences":true,"contains_activity_data":false,"dual_use_assessed":true,"usage_policy_url":"https://openamp.example.org/data-policy","contact_email":"data@openamp.example.org","release_approved":true,"dry_lab_only":true}' \
Expand Down
61 changes: 61 additions & 0 deletions docs/evidence/AUDIT_CHAIN_GUIDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# Audit Chain Completeness Guide

## Purpose

The audit chain completeness checker validates that an evidence package for a
batch contains all required chain links — from raw sequence input through
pipeline scoring, filtering, candidate selection, and submission readiness.
A gap anywhere in the chain blocks external reproducibility.

## Schema Fields

| Field | Type | Description |
|---|---|---|
| chain_id | str | Unique identifier (prefix "ACH-") |
| batch_id | str | Batch under audit |
| pipeline_version | str | Pipeline version string |
| audit_date | str | ISO 8601 date of this audit |
| has_sequence_input | bool | Raw sequences are documented |
| has_benchmark_results | bool | Benchmark comparison results exist |
| has_filter_log | bool | Candidate filter decisions are logged |
| has_score_decomposition | bool | Score decomposition reports exist |
| has_selection_rationale | bool | Selection rationale documents exist |
| has_evidence_certificate | bool | Evidence certificates are present |
| has_claim_mappings | bool | Claim-to-evidence mappings exist |
| has_pipeline_decision_audit | bool | Pipeline decision audit entries exist |
| has_reviewer_briefing | bool | Reviewer briefing package exists |
| missing_links | list[str] | Auto-populated: names of chain links that are False |
| auditor | str | Person or system performing this audit |
| dry_lab_only | bool | Always True |

## Chain Links (all required)

1. `has_sequence_input` — without this, the pipeline has no traceable starting point
2. `has_benchmark_results` — performance claims need benchmark backing
3. `has_filter_log` — what was excluded must be recorded alongside what was included
4. `has_score_decomposition` — composite scores must be decomposable
5. `has_selection_rationale` — why *these* candidates and not others
6. `has_evidence_certificate` — certificate of computational evidence
7. `has_claim_mappings` — every claim traces to an artifact
8. `has_pipeline_decision_audit` — filter/threshold decisions are traceable
9. `has_reviewer_briefing` — external auditor has a complete briefing

## Validation Rules

1. chain_id must start with "ACH-"
2. auditor must not be empty
3. dry_lab_only must be True
4. All 9 chain links must be True (any False is an error)
5. missing_links must match the set of chain links that are False

## Warning Conditions

- Single auditor: auditor contains no "@" character → warn that a named human auditor is recommended
- Future audit date: audit_date appears to be after the current year (year > 2030) → warn about implausible date

## Honest Use Boundary

Audit chain completeness is a meta-check: it verifies that all *other* evidence
artifacts exist, not that they are scientifically correct. A passing audit chain
means the evidence package is structurally complete, not that every artifact
contains accurate science. Human expert review is still required.
4 changes: 2 additions & 2 deletions docs/evidence/METRICS_CURRENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
> with this file, this file wins. Updated whenever benchmark/benchmark config changes.
>
> **Last updated:** 2026-07-10 (Phase M M4 — reviewer briefing package — v0.9.2)
> **New in v0.9.2:** Reviewer briefing package schema (M4 — Loop 131) — docs/evidence/REVIEWER_BRIEFING_GUIDE.md with purpose, schema fields table (11 fields), validation rules (7), warning conditions (4), honest-use boundary. src/openamp_foundry/evidence/reviewer_briefing_package.py with ReviewerBriefingEntry dataclass (11 fields: briefing_id, batch_id, pipeline_version, prepared_date, reviewer_name, candidate_count, artifact_ids, open_questions, scope_description, conflict_of_interest_declared, dry_lab_only=True), ReviewerBriefingResult dataclass (7 fields: briefing_id, batch_id, reviewer_name, passed, errors, warnings, dry_lab_only=True), MINIMUM_ARTIFACT_IDS (3), MAX_SCOPE_LENGTH (500), LONG_SCOPE_THRESHOLD (300), LARGE_BATCH_THRESHOLD (50), MAX_OPEN_QUESTIONS_WARNING (5), MINIMAL_ARTIFACT_WARNING_COUNT (3), validate_reviewer_briefing() (7 error checks: RBP- prefix, non-empty reviewer_name, candidate_count>=1, artifact_ids>=3, non-empty scope_description<=500 chars, conflict_of_interest_declared must be True, dry_lab_only must be True; 4 warning conditions: large batch, many open questions, long scope, minimal artifacts), validate_reviewer_briefing_dict() (10 required fields guard). CLI (openamp-foundry reviewer-briefing-check) with --entry-json, --format text|json. make reviewer-briefing-check target. 52 tests. Phase M M4: reviewer briefing packages now have a machine-validated schema for external auditor handoff.
> **Last updated:** 2026-07-10 (Phase M M5 — audit chain completeness — v0.9.3)
> **New in v0.9.3:** Audit chain completeness checker (M5 — Loop 132) — docs/evidence/AUDIT_CHAIN_GUIDE.md with purpose, schema fields table (16 fields: chain_id, batch_id, pipeline_version, audit_date, 9 has_* bools, missing_links, auditor, dry_lab_only), 9 required chain links, validation rules (5), warning conditions (2: single auditor, future date), honest-use boundary. src/openamp_foundry/evidence/audit_chain_completeness.py with AuditChainEntry dataclass (16 fields: chain_id, batch_id, pipeline_version, audit_date, 9 has_* bools, missing_links list[str], auditor, dry_lab_only=True), AuditChainResult dataclass (7 fields: chain_id, batch_id, missing_link_count, passed, errors, warnings, dry_lab_only=True), CHAIN_LINK_FIELDS (9 entries), CHAIN_LINK_COUNT (9), AUDITOR_EMAIL_HINT ("@"), IMPLAUSIBLE_YEAR_THRESHOLD (2030), validate_audit_chain() (5+ error checks: ACH- prefix, non-empty auditor, dry_lab_only must be True, each false chain link errors, missing_links consistency; 2 warnings: no email, future date), validate_audit_chain_dict() (15 required fields guard). CLI (openamp-foundry audit-chain-check) with --entry-json, --format text|json. make audit-chain-check target. 39 tests. Phase M M5: the evidence chain is now machine-checkable end to end. Phase M (Audit Trail Infrastructure) is fully complete with M1-M5.
> **New in v0.9.1:** Score decomposition report schema (M3 — Loop 130) — docs/evidence/SCORE_DECOMPOSITION_GUIDE.md with purpose, schema fields table (12 fields), valid scoring methods (6), validation rules (9), warning conditions (4), honest-use boundary. src/openamp_foundry/evidence/score_decomposition_report.py with ScoreDecompositionEntry dataclass (12 fields: report_id, batch_id, candidate_id, pipeline_version, composite_score, component_scores, component_weights, scoring_method, score_range_min, score_range_max, reviewer, dry_lab_only=True), ScoreDecompositionResult dataclass (7 fields: report_id, batch_id, candidate_id, scoring_method, passed, errors, warnings, dry_lab_only=True), VALID_SCORING_METHODS (6: additive_weighted, geometric_mean, harmonic_mean, max_component, min_component, rank_aggregation), MINIMUM_COMPONENTS (2), WEIGHT_SUM_TOLERANCE (0.01), DOMINANT_WEIGHT_THRESHOLD (0.6), UNBALANCED_RATIO_THRESHOLD (5.0), MAX_COMPONENTS_WARNING (8), LOW_SCORE_FRACTION (0.2), validate_score_decomposition() (9 error checks: SDR- prefix, score_range_min<score_range_max, composite_score in range, >=2 components, weight keys match score keys, weights sum to ~1.0, valid scoring_method, non-empty reviewer, dry_lab_only must be True; 4 warning conditions: dominant component, unbalanced weights, many components, low composite score), validate_score_decomposition_dict() (11 required fields guard). CLI (openamp-foundry score-decomposition-check) with --entry-json, --format text|json. make score-decomposition-check target. Phase M M3: every composite score is now machine-decomposable into its named components for external audit.
> **New in v0.9.0:** Claim-to-evidence mapper schema (M2 — Loop 129) — docs/evidence/CLAIM_TO_EVIDENCE_GUIDE.md with purpose, schema fields table (10 fields), valid claim types (7), validation rules (7), warning conditions (4), honest-use boundary, CLI usage. src/openamp_foundry/evidence/claim_to_evidence_mapper.py with ClaimToEvidenceEntry dataclass (10 fields: mapping_id, batch_id, pipeline_version, claim_text, claim_type, supporting_artifact_ids, evidence_level, pre_specified, reviewer, dry_lab_only=True), ClaimToEvidenceResult dataclass (7 fields: mapping_id, batch_id, claim_type, passed, errors, warnings, dry_lab_only=True), VALID_CLAIM_TYPES (7: activity_prediction, calibration_statement, novelty_claim, performance_comparison, reproducibility_claim, safety_assessment, selection_rationale), VALID_EVIDENCE_LEVELS (6: 1-6), MAX_CLAIM_TEXT_LENGTH (500), LONG_CLAIM_TEXT_THRESHOLD (300), WEAK_EVIDENCE_THRESHOLD (2), validate_claim_to_evidence() (7 error checks: CEM- prefix, non-empty claim_text<=500 chars, valid claim_type, non-empty supporting_artifact_ids, valid evidence_level 1-6, non-empty reviewer, dry_lab_only must be True; 4 warning conditions: post-hoc, weak evidence, single artifact, long claim text), validate_claim_to_evidence_dict() (9 required fields guard). CLI (openamp-foundry claim-to-evidence-check) with --entry-json, --format text|json. make claim-to-evidence-check target. Phase M M2: every scientific claim is now machine-mapped to its supporting artifacts for external audit.
> **New in v0.8.9:** Pipeline decision audit entry schema (M1 — Loop 128) — docs/evidence/PIPELINE_DECISION_AUDIT_GUIDE.md with purpose, required field table (13 fields), valid decision types (7), warnings, validation workflow, honest-use boundary. src/openamp_foundry/evidence/pipeline_decision_audit.py with PipelineDecisionAuditEntry dataclass (13 fields: audit_id, batch_id, pipeline_version, decision_date, decision_type, decision_description, rationale, alternatives_considered, affected_candidate_count, evidence_level, pre_specified, reviewer, dry_lab_only), PipelineDecisionAuditResult dataclass (5 fields: audit_id, batch_id, decision_type, passed, errors, warnings, dry_lab_only=True), VALID_DECISION_TYPES (7: benchmark_updated, calibration_adjusted, candidate_ranked, candidate_rejected, filter_applied, safety_flag_applied, threshold_chosen), VALID_EVIDENCE_LEVELS (6: 1-6), MAX_DESCRIPTION_LENGTH (500), MAX_RATIONALE_LENGTH (1000), validate_pipeline_decision_audit() (12 checks: AUD- prefix, non-empty batch_id/pipeline_version/reviewer, YYYY-MM-DD decision_date, valid decision_type, non-empty description<=500 chars, non-empty rationale<=1000 chars, affected_candidate_count>=0, valid evidence_level, dry_lab_only must be True; post-hoc warns, empty alternatives warns, low evidence warns, zero affected warns), validate_pipeline_decision_audit_dict() (12 required fields guard). CLI (openamp-foundry pipeline-decision-audit-check) with --entry-json, --format text|json. make pipeline-decision-audit-check target. Phase M M1: every pipeline decision is now machine-validated and traceable for external audit.
Expand Down
2 changes: 1 addition & 1 deletion docs/research/NEXT_100_PR_MAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,7 @@ Make every pipeline decision traceable, independently verifiable, and resistant
| M2 | Add claim-to-evidence mapper schema (complete — Loop 129). | Maps each scientific claim to the artifact that supports it. | B |
| M3 | Add score decomposition report schema (complete — Loop 130). | Documents how composite scores are computed from components. | B |
| M4 | Add reviewer briefing package checker (complete — Loop 131). | One-stop summary that external reviewers need before auditing. | B |
| M5 | Add audit chain completeness checker. | Validates the evidence chain has no gaps from sequences to submission. | B/D |
| M5 | Add audit chain completeness checker (complete — Loop 132). | Validates the evidence chain has no gaps from sequences to submission. Completes Phase M. | B/D |

## Prioritization rule

Expand Down
8 changes: 8 additions & 0 deletions docs/research/ROADMAP.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,13 @@
# Roadmap

### v0.9.3 — Phase M M5: Audit Chain Completeness Checker (Completes Phase M)
- Added `AuditChainEntry` schema validating all 9 evidence chain links exist for a batch
- Detects gaps from sequence input through benchmark, filter, scoring, selection, certificate, claims, decision audit, and reviewer briefing
- missing_links consistency check catches declaration errors
- CLI: `openamp-foundry audit-chain-check`
- 39 tests; all passing
- Phase M (Audit Trail Infrastructure) complete

### v0.9.2 — Phase M M4: Reviewer Briefing Package
- Added `ReviewerBriefingEntry` schema for one-stop external auditor handoff packages
- Validates CoI declaration, minimum artifact count, scope, candidate count
Expand Down
19 changes: 19 additions & 0 deletions src/openamp_foundry/cli/commands/reports.py
Original file line number Diff line number Diff line change
Expand Up @@ -3290,3 +3290,22 @@ def _run_reviewer_briefing_check(args: argparse.Namespace) -> int:
print(" Reviewer briefing package validated. External auditor handoff is documented.")

return 0 if result.passed else 3


def _run_audit_chain_check(args: argparse.Namespace) -> None:
import json
from openamp_foundry.evidence.audit_chain_completeness import validate_audit_chain_dict

entry_dict = json.loads(args.entry_json)
result = validate_audit_chain_dict(entry_dict)

if args.format == "json":
import dataclasses
print(json.dumps(dataclasses.asdict(result), indent=2))
else:
status = "PASS" if result.passed else "FAIL"
print(f"[{status}] Audit Chain Check: {result.chain_id} ({result.missing_link_count} gaps)")
for e in result.errors:
print(f" ERROR: {e}")
for w in result.warnings:
print(f" WARN: {w}")
Loading
Loading