cschanhniem · Jul 9, 2026 · Jul 9, 2026
diff --git a/Makefile b/Makefile
@@ -1,4 +1,4 @@
-.PHONY: help demo test lint ci clean bench-leakage bench-multi-negatives bench-baseline bench-hidden-active bench-cluster-split bench-expert-ablation bench-expert-ablation-500 bench-selectivity bench-feature-decomp bench-gate bench-easy-baseline bench-charge-matched bench-order-dependent bench-precision-at-k bench-per-family bench-simulation-ablation bench-simulation-ablation-within-amp simulation-gate bench-strategy-compare regenerate-all generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-pilot pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter recalibration-engine recalibration-engine-dry-run validate-policy-version generate-synthetic-lab-results bump-policy-version generate-review-packet failed-candidate-report safe-publication-filter classify-negative-informativeness validate-rejection-events negative-result-dashboard calibration-audit calibration-audit-example calibration-overfit-check result-quality-filter synthetic-result-policy-check calibration-decision-checklist calibration-rollback-plan simulation-registry validate-simulation-result-schema simulation-baseline-check adapter-gate-check simulation-provenance simulation-ensemble-check simulation-ci-report simulation-deprecation-check simulation-scope-check simulation-evidence-packet artifact-version candidate-manifest artifact-changelog integration-check adapter-author-check license-check artifact-compat-check contribution-check adoption-scorecard release-gate-check decision-log release-request-check coi-check rotation-plan-check security-report-check citation-check roadmap-sync-check advisory-review-check annual-review-check selection-rationale-check batch-priority-check pilot-package-check calibration-intake-check uncertainty-report-check preprint-bundle-check reproducibility-manifest-check candidate-summary-card-check multi-candidate-comparison-check pipeline-decision-audit-check claim-to-evidence-check score-decomposition-check reviewer-briefing-check audit-chain-check pre-registration-check hypothesis-outcome-check dataset-release-check
+.PHONY: help demo test lint ci clean bench-leakage bench-multi-negatives bench-baseline bench-hidden-active bench-cluster-split bench-expert-ablation bench-expert-ablation-500 bench-selectivity bench-feature-decomp bench-gate bench-easy-baseline bench-charge-matched bench-order-dependent bench-precision-at-k bench-per-family bench-simulation-ablation bench-simulation-ablation-within-amp simulation-gate bench-strategy-compare regenerate-all generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-pilot pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter recalibration-engine recalibration-engine-dry-run validate-policy-version generate-synthetic-lab-results bump-policy-version generate-review-packet failed-candidate-report safe-publication-filter classify-negative-informativeness validate-rejection-events negative-result-dashboard calibration-audit calibration-audit-example calibration-overfit-check result-quality-filter synthetic-result-policy-check calibration-decision-checklist calibration-rollback-plan simulation-registry validate-simulation-result-schema simulation-baseline-check adapter-gate-check simulation-provenance simulation-ensemble-check simulation-ci-report simulation-deprecation-check simulation-scope-check simulation-evidence-packet artifact-version candidate-manifest artifact-changelog integration-check adapter-author-check license-check artifact-compat-check contribution-check adoption-scorecard release-gate-check decision-log release-request-check coi-check rotation-plan-check security-report-check citation-check roadmap-sync-check advisory-review-check annual-review-check selection-rationale-check batch-priority-check pilot-package-check calibration-intake-check uncertainty-report-check preprint-bundle-check reproducibility-manifest-check candidate-summary-card-check multi-candidate-comparison-check pipeline-decision-audit-check claim-to-evidence-check score-decomposition-check reviewer-briefing-check audit-chain-check pre-registration-check hypothesis-outcome-check baseline-comparison-check dataset-release-check
 
 PYTHON := $(shell [ -f .venv/bin/python ] && echo .venv/bin/python || echo python3)
 PYTEST  := $(shell [ -f .venv/bin/pytest ] && echo .venv/bin/pytest || echo pytest)
@@ -954,6 +954,9 @@ pre-registration-check:
 hypothesis-outcome-check:
 	openamp-foundry hypothesis-outcome-check --entry-json '{"outcome_id":"HOR-001","registration_id":"PRE-001","batch_id":"BATCH-001","pipeline_version":"0.9.5","outcome_date":"2026-07-15","outcome_verdict":"confirmed","observed_metric_value":2.5,"success_threshold_met":true,"interpretation":"Selected candidates showed median MIC of 2.5 ug/mL, meeting the threshold. Random baseline showed 32 ug/mL. Hypothesis confirmed with strong signal vs random.","deviation_from_plan":"","recorded_by":"test@example.com","dry_lab_only":true}' --format text
 
+baseline-comparison-check:
+	openamp-foundry baseline-comparison-check --entry-json '{"manifest_id":"BCM-001","batch_id":"BATCH-001","pipeline_version":"0.9.6","comparison_date":"2026-07-10","metric_name":"hit_rate","pipeline_score":0.7,"baseline_scores":{"random_selection":0.1,"charge_matched":0.15},"pipeline_beats_all_baselines":true,"effect_size":0.55,"p_value":0.02,"comparison_direction":"higher_is_better","notes":"Pipeline beats random and charge-matched baselines.","reviewer":"test@example.com","dry_lab_only":true}' --format text
+
 dataset-release-check:
 	.venv/bin/openamp-foundry dataset-release-check \
 		--entry-json '{"release_id":"DSR-001","dataset_name":"OpenAMP Wave 0.5 Candidates","dataset_version":"1.0.0","release_date":"2026-07-09","license_identifier":"CC-BY-4.0","data_sources":["UniProt AMP database","APD3","in-house pipeline"],"contains_sequences":true,"contains_activity_data":false,"dual_use_assessed":true,"usage_policy_url":"https://openamp.example.org/data-policy","contact_email":"data@openamp.example.org","release_approved":true,"dry_lab_only":true}' \

diff --git a/docs/evidence/BASELINE_COMPARISON_GUIDE.md b/docs/evidence/BASELINE_COMPARISON_GUIDE.md
@@ -0,0 +1,72 @@
+# Baseline Comparison Manifest Guide
+
+## Purpose
+
+The baseline comparison manifest records whether the pipeline's predictions
+actually beat cheap baselines on a specific metric. Every performance claim
+must be grounded in a documented comparison. Without this, it is impossible
+to know whether the pipeline adds value over random selection, charge matching,
+or length matching.
+
+## Schema Fields
+
+| Field | Type | Description |
+|---|---|---|
+| manifest_id | str | Unique identifier (prefix "BCM-") |
+| batch_id | str | Batch being evaluated |
+| pipeline_version | str | Pipeline version at time of comparison |
+| comparison_date | str | ISO 8601 date of comparison |
+| metric_name | str | The metric being compared (see valid metrics) |
+| pipeline_score | float | Pipeline's score on this metric |
+| baseline_scores | dict[str, float] | Score of each baseline (at least 1 required) |
+| pipeline_beats_all_baselines | bool | Whether pipeline outperforms every baseline |
+| effect_size | float | Quantified difference between pipeline and best baseline |
+| p_value | float | Statistical significance (0.0–1.0, or -1.0 if not computed) |
+| comparison_direction | str | Whether higher or lower is better (see valid directions) |
+| notes | str | Optional context (max 300 chars, may be empty) |
+| reviewer | str | Person or system that validated this comparison |
+| dry_lab_only | bool | Always True (computational comparison) |
+
+## Valid Metric Names
+
+- `fold_change_mic` — fold change in minimum inhibitory concentration
+- `hemolysis_fraction` — fraction of hemolysis
+- `hit_rate` — fraction of candidates meeting activity threshold
+- `mic_value` — raw MIC value
+- `novelty_score` — sequence novelty vs. known AMPs
+- `selectivity_index` — cytotoxicity IC50 / MIC ratio
+
+## Valid Comparison Directions
+
+- `higher_is_better` — e.g. hit rate, selectivity index
+- `lower_is_better` — e.g. MIC value, hemolysis fraction
+
+## Validation Rules
+
+1. manifest_id must start with "BCM-"
+2. metric_name must be one of the valid metrics
+3. pipeline_score must be finite (not NaN or Inf)
+4. baseline_scores must contain at least 1 baseline
+5. all baseline_scores values must be finite
+6. effect_size must be finite
+7. p_value must be in [0.0, 1.0] or exactly -1.0 (sentinel for "not computed")
+8. comparison_direction must be one of the valid directions
+9. notes must not exceed 300 characters
+10. reviewer must not be empty
+11. dry_lab_only must be True
+
+## Warning Conditions
+
+- Pipeline does not beat all baselines: pipeline_beats_all_baselines=False → warn that pipeline underperforms at least one baseline
+- Inconsistent verdict: pipeline_beats_all_baselines=True but pipeline score is actually worse than some baseline (accounting for direction) → warn about inconsistency
+- No p-value: p_value == -1.0 → warn that statistical significance was not computed
+- Large effect unchecked: abs(effect_size) > 10.0 and p_value == -1.0 → warn that a large claimed effect needs statistical validation
+
+## Honest Use Boundary
+
+A baseline comparison manifest records computational comparisons only. The
+baselines must actually be evaluated — not assumed. A claim that the pipeline
+beats random selection is only meaningful if random selection was actually
+run and measured on the same data split. Cherry-picking the comparison metric
+after seeing results defeats the purpose; comparisons should be pre-specified
+(link to a PRE- registration where possible).
diff --git a/docs/evidence/METRICS_CURRENT.md b/docs/evidence/METRICS_CURRENT.md
@@ -5,9 +5,9 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
 > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
 > with this file, this file wins. Updated whenever benchmark/benchmark config changes.
 >
-> **Last updated:** 2026-07-10 (Phase N N2 — hypothesis outcome record — v0.9.5)
-> **New in v0.9.5:** Hypothesis outcome record schema (N2 — Loop 134) — docs/evidence/HYPOTHESIS_OUTCOME_GUIDE.md with purpose, schema fields table (12 fields: outcome_id, registration_id, batch_id, pipeline_version, outcome_date, outcome_verdict, observed_metric_value, success_threshold_met, interpretation, deviation_from_plan, recorded_by, dry_lab_only), 4 valid outcome verdicts (confirmed, refuted, inconclusive, partially_confirmed), 6 validation rules, 4 warning conditions (threshold/verdict inconsistency, inconclusive no deviation, short interpretation), honest-use boundary. HypothesisOutcomeEntry dataclass (12 fields), HypothesisOutcomeResult dataclass (7 fields), validate_hypothesis_outcome() (6+ error checks: HOR- prefix, PRE- prefix, valid verdict, finite metric, non-empty interpretation<=500, non-empty recorded_by; 4 warning conditions: threshold/verdict mismatch, inconclusive no deviation, short interpretation), validate_hypothesis_outcome_dict() (11 required fields guard). CLI (openamp-foundry hypothesis-outcome-check) with --entry-json, --format text|json. make hypothesis-outcome-check target. 48 tests. Phase N N2: the pre-registration loop is now closed — outcomes are recorded and matched to their commitments.
-> > **New in v0.9.4:** Pre-registration form schema (N1 — Loop 133) — docs/evidence/PRE_REGISTRATION_GUIDE.md with purpose, schema fields table (13 fields: registration_id, batch_id, pipeline_version, registration_date, primary_hypothesis, primary_outcome_metric, success_threshold, baseline_comparators, candidate_ids, assay_type, statistical_test, registered_by, dry_lab_only), 6 valid primary outcome metrics, 5 valid assay types, 10 validation rules, 4 warning conditions, honest-use boundary. PreRegistrationEntry dataclass (13 fields), PreRegistrationResult dataclass (6 fields), validate_pre_registration() (10+ error checks: PRE- prefix, non-empty hypothesis<=500 chars, valid outcome metric, finite success_threshold, >=1 baseline_comparators, >=1 candidate_ids, valid assay_type, non-empty statistical_test<=200 chars, non-empty registered_by, dry_lab_only must be True; 4 warning conditions: no random baseline, large candidate set >20, short hypothesis <50 chars, placeholder statistical test TBD/N/A/NA/none), validate_pre_registration_dict() (12 required fields guard). CLI (openamp-foundry pre-registration-check) with --entry-json, --format text|json. make pre-registration-check target. 62 tests. Phase N (Pre-registration & Baseline Honesty) started with N1.
+> **Last updated:** 2026-07-10 (Phase N N3 — baseline comparison manifest — v0.9.6)
+> **New in v0.9.6:** Baseline comparison manifest schema (N3 — Loop 135) — docs/evidence/BASELINE_COMPARISON_GUIDE.md with purpose, schema fields table (14 fields: manifest_id, batch_id, pipeline_version, comparison_date, metric_name, pipeline_score, baseline_scores, pipeline_beats_all_baselines, effect_size, p_value, comparison_direction, notes, reviewer, dry_lab_only), 6 valid metric names (fold_change_mic, hemolysis_fraction, hit_rate, mic_value, novelty_score, selectivity_index), 2 valid comparison directions (higher_is_better, lower_is_better), 11 validation rules, 4 warning conditions (pipeline loses, inconsistent verdict, no p-value, large unchecked effect), honest-use boundary. BaselineComparisonEntry dataclass (14 fields), BaselineComparisonResult dataclass (7 fields), validate_baseline_comparison() (11+ error checks: BCM- prefix, valid metric_name, finite pipeline_score, >=1 baseline, finite baseline_scores, finite effect_size, p_value in [0.0,1.0] or -1.0, valid comparison_direction, notes<=300, non-empty reviewer, dry_lab_only must be True; 4 warning conditions: underperforms, inconsistent verdict, no p-value, large+unchecked), validate_baseline_comparison_dict() (13 required fields guard). CLI (openamp-foundry baseline-comparison-check) with --entry-json, --format text|json. make baseline-comparison-check target. 68 tests. Phase N N3: every performance claim against a baseline is now machine-verifiable.
+> **New in v0.9.5:** Pre-registration form schema (N1 — Loop 133) — docs/evidence/PRE_REGISTRATION_GUIDE.md with purpose, schema fields table (13 fields: registration_id, batch_id, pipeline_version, registration_date, primary_hypothesis, primary_outcome_metric, success_threshold, baseline_comparators, candidate_ids, assay_type, statistical_test, registered_by, dry_lab_only), 6 valid primary outcome metrics, 5 valid assay types, 10 validation rules, 4 warning conditions, honest-use boundary. PreRegistrationEntry dataclass (13 fields), PreRegistrationResult dataclass (6 fields), validate_pre_registration() (10+ error checks: PRE- prefix, non-empty hypothesis<=500 chars, valid outcome metric, finite success_threshold, >=1 baseline_comparators, >=1 candidate_ids, valid assay_type, non-empty statistical_test<=200 chars, non-empty registered_by, dry_lab_only must be True; 4 warning conditions: no random baseline, large candidate set >20, short hypothesis <50 chars, placeholder statistical test TBD/N/A/NA/none), validate_pre_registration_dict() (12 required fields guard). CLI (openamp-foundry pre-registration-check) with --entry-json, --format text|json. make pre-registration-check target. 62 tests. Phase N (Pre-registration & Baseline Honesty) started with N1.
 > > **New in v0.9.3:** Audit chain completeness checker (M5 — Loop 132) — docs/evidence/AUDIT_CHAIN_GUIDE.md with purpose, schema fields table (16 fields: chain_id, batch_id, pipeline_version, audit_date, 9 has_* bools, missing_links, auditor, dry_lab_only), 9 required chain links, validation rules (5), warning conditions (2: single auditor, future date), honest-use boundary. src/openamp_foundry/evidence/audit_chain_completeness.py with AuditChainEntry dataclass (16 fields: chain_id, batch_id, pipeline_version, audit_date, 9 has_* bools, missing_links list[str], auditor, dry_lab_only=True), AuditChainResult dataclass (7 fields: chain_id, batch_id, missing_link_count, passed, errors, warnings, dry_lab_only=True), CHAIN_LINK_FIELDS (9 entries), CHAIN_LINK_COUNT (9), AUDITOR_EMAIL_HINT ("@"), IMPLAUSIBLE_YEAR_THRESHOLD (2030), validate_audit_chain() (5+ error checks: ACH- prefix, non-empty auditor, dry_lab_only must be True, each false chain link errors, missing_links consistency; 2 warnings: no email, future date), validate_audit_chain_dict() (15 required fields guard). CLI (openamp-foundry audit-chain-check) with --entry-json, --format text|json. make audit-chain-check target. 39 tests. Phase M M5: the evidence chain is now machine-checkable end to end. Phase M (Audit Trail Infrastructure) is fully complete with M1-M5.
 > **New in v0.9.1:** Score decomposition report schema (M3 — Loop 130) — docs/evidence/SCORE_DECOMPOSITION_GUIDE.md with purpose, schema fields table (12 fields), valid scoring methods (6), validation rules (9), warning conditions (4), honest-use boundary. src/openamp_foundry/evidence/score_decomposition_report.py with ScoreDecompositionEntry dataclass (12 fields: report_id, batch_id, candidate_id, pipeline_version, composite_score, component_scores, component_weights, scoring_method, score_range_min, score_range_max, reviewer, dry_lab_only=True), ScoreDecompositionResult dataclass (7 fields: report_id, batch_id, candidate_id, scoring_method, passed, errors, warnings, dry_lab_only=True), VALID_SCORING_METHODS (6: additive_weighted, geometric_mean, harmonic_mean, max_component, min_component, rank_aggregation), MINIMUM_COMPONENTS (2), WEIGHT_SUM_TOLERANCE (0.01), DOMINANT_WEIGHT_THRESHOLD (0.6), UNBALANCED_RATIO_THRESHOLD (5.0), MAX_COMPONENTS_WARNING (8), LOW_SCORE_FRACTION (0.2), validate_score_decomposition() (9 error checks: SDR- prefix, score_range_min<score_range_max, composite_score in range, >=2 components, weight keys match score keys, weights sum to ~1.0, valid scoring_method, non-empty reviewer, dry_lab_only must be True; 4 warning conditions: dominant component, unbalanced weights, many components, low composite score), validate_score_decomposition_dict() (11 required fields guard). CLI (openamp-foundry score-decomposition-check) with --entry-json, --format text|json. make score-decomposition-check target. Phase M M3: every composite score is now machine-decomposable into its named components for external audit.
 > **New in v0.9.0:** Claim-to-evidence mapper schema (M2 — Loop 129) — docs/evidence/CLAIM_TO_EVIDENCE_GUIDE.md with purpose, schema fields table (10 fields), valid claim types (7), validation rules (7), warning conditions (4), honest-use boundary, CLI usage. src/openamp_foundry/evidence/claim_to_evidence_mapper.py with ClaimToEvidenceEntry dataclass (10 fields: mapping_id, batch_id, pipeline_version, claim_text, claim_type, supporting_artifact_ids, evidence_level, pre_specified, reviewer, dry_lab_only=True), ClaimToEvidenceResult dataclass (7 fields: mapping_id, batch_id, claim_type, passed, errors, warnings, dry_lab_only=True), VALID_CLAIM_TYPES (7: activity_prediction, calibration_statement, novelty_claim, performance_comparison, reproducibility_claim, safety_assessment, selection_rationale), VALID_EVIDENCE_LEVELS (6: 1-6), MAX_CLAIM_TEXT_LENGTH (500), LONG_CLAIM_TEXT_THRESHOLD (300), WEAK_EVIDENCE_THRESHOLD (2), validate_claim_to_evidence() (7 error checks: CEM- prefix, non-empty claim_text<=500 chars, valid claim_type, non-empty supporting_artifact_ids, valid evidence_level 1-6, non-empty reviewer, dry_lab_only must be True; 4 warning conditions: post-hoc, weak evidence, single artifact, long claim text), validate_claim_to_evidence_dict() (9 required fields guard). CLI (openamp-foundry claim-to-evidence-check) with --entry-json, --format text|json. make claim-to-evidence-check target. Phase M M2: every scientific claim is now machine-mapped to its supporting artifacts for external audit.