diff --git a/docs/evidence/DIAGRAM_ARTIFACT_DEPENDENCY.md b/docs/evidence/DIAGRAM_ARTIFACT_DEPENDENCY.md new file mode 100644 index 00000000..90c39162 --- /dev/null +++ b/docs/evidence/DIAGRAM_ARTIFACT_DEPENDENCY.md @@ -0,0 +1,23 @@ +# Artifact Dependency Graph + +Shows how artifacts depend on each other. + +```mermaid +flowchart TD + A[Candidate CSV] --> B[Ranked JSONL] + A --> C[Evidence certs] + B --> D[Batch report] + B --> E[Run manifest] + C --> F[Lab batch pack] + F --> G[Lab results] + G --> H[Calibration intake] + H --> I[Gate verdict] + I -->|pass| J[Weight proposal] + I -->|fail| K[Blocked] +``` + +## Rules +- Artifacts higher in the graph are inputs. +- Arrows show the direction of data flow. +- Dotted lines indicate optional dependencies. +- The graph should be updated when new artifact types are added. diff --git a/docs/evidence/GUIDE_ISSUE_ARTIFACT_TRACE.md b/docs/evidence/GUIDE_ISSUE_ARTIFACT_TRACE.md new file mode 100644 index 00000000..88c84b2d --- /dev/null +++ b/docs/evidence/GUIDE_ISSUE_ARTIFACT_TRACE.md @@ -0,0 +1,19 @@ +# Issue-to-Artifact Trace Field + +How issues trace to their resulting artifacts. + +## Format +Issues should include a `## Artifacts` section when resolved: +```markdown +## Artifacts +- PR: #NNN +- Doc: docs/evidence/EXAMPLE.md +- Script: scripts/example.py +- Test: tests/test_example.py +``` + +## Rules +- Every closed issue should trace to at least one artifact. +- Artifacts can be PRs, docs, scripts, tests, or configs. +- The trace helps future contributors understand why something exists. +- If an issue doesn't produce artifacts (investigation only), note that. diff --git a/docs/evidence/GUIDE_LINEAGE_RENDERER.md b/docs/evidence/GUIDE_LINEAGE_RENDERER.md new file mode 100644 index 00000000..12e8b0f3 --- /dev/null +++ b/docs/evidence/GUIDE_LINEAGE_RENDERER.md @@ -0,0 +1,22 @@ +# Artifact Lineage Markdown Renderer + +How artifact lineage is documented in markdown. + +## Format +```markdown +## Artifact Lineage + +```mermaid +flowchart LR + A[Input CSV] --> B[score_candidates] + B --> C[Ranked JSONL] + B --> D[Evidence certs] +``` +``` + +## Rules +- Lineage diagrams should use Mermaid flowchart format. +- Show inputs → process → outputs. +- Label each node with the artifact name and format. +- Keep lineage diagrams focused on one artifact type at a time. +- Include the lineage in the artifact's documentation. diff --git a/docs/evidence/GUIDE_STALE_ARTIFACT_DETECTOR.md b/docs/evidence/GUIDE_STALE_ARTIFACT_DETECTOR.md new file mode 100644 index 00000000..7d9f025a --- /dev/null +++ b/docs/evidence/GUIDE_STALE_ARTIFACT_DETECTOR.md @@ -0,0 +1,21 @@ +# Stale Artifact Detector + +Detects artifacts that may be stale (out of sync with current code). + +## What It Checks +1. Schema files — compare schema version against code. +2. Benchmark outputs — check if they match current pipeline. +3. Generated docs — check if they reference current features. +4. Config files — check if all keys are still valid. + +## Usage +```bash +python scripts/check_doc_links.py # Check docs for broken references +make bench-gate # Check benchmark regression gates +python scripts/check_pyproject.py # Check pyproject.toml consistency +``` + +## Manual Checks +- Evidence certificates should be regenerated after scoring changes. +- Benchmarks should be re-run after config changes. +- Docs should be reviewed after major feature changes. diff --git a/docs/evidence/MATRIX_CLAIM_EVIDENCE.md b/docs/evidence/MATRIX_CLAIM_EVIDENCE.md new file mode 100644 index 00000000..1e32e6f1 --- /dev/null +++ b/docs/evidence/MATRIX_CLAIM_EVIDENCE.md @@ -0,0 +1,16 @@ +# Claim-to-Evidence Matrix Artifact + +Maps claims to their supporting evidence artifacts. + +| Claim | Evidence Level | Supporting Artifact | Status | +|-------|:--------------:|--------------------|:------:| +| Pipeline ranks AMPs | 2 (baseline) | METRICS_CURRENT.md | ✅ | +| Pipeline is reproducible | 2 (baseline) | RUN_MANIFEST_STANDARD.md | ✅ | +| Candidates are novel | 2 (baseline) | NOVELTY_CHECKLIST.md | ✅ | +| Candidates are safe | 2 (baseline) | LIMITATIONS_OVERVIEW.md | ⚠️ Limited | + +## Rules +- Every claim should be paired with its evidence. +- The evidence level should match the claim strength. +- If evidence is missing, mark the claim as unsupported. +- This matrix should be reviewed before any public release. diff --git a/docs/evidence/POLICY_ARTIFACT_RETENTION.md b/docs/evidence/POLICY_ARTIFACT_RETENTION.md new file mode 100644 index 00000000..00b98e9a --- /dev/null +++ b/docs/evidence/POLICY_ARTIFACT_RETENTION.md @@ -0,0 +1,18 @@ +# Artifact Retention Policy + +How long different artifact types are retained. + +| Artifact Type | Retention | Location | Cleanup | +|:-------------|:---------:|:---------|:--------| +| Evidence certificates | Permanent | outputs/evidence/ | Manual | +| Run manifests | 2 years | outputs/ | Automated | +| Benchmark results | Permanent | outputs/ | Manual | +| Lab results | Permanent | outputs/lab_results/ | Manual | +| Temporary outputs | 90 days | outputs/ | Automated | +| Logs | 30 days | outputs/logs/ | Automated | + +## Rules +- Permanent artifacts must never be deleted without a decision log entry. +- Temporary outputs can be cleaned up with `make clean`. +- Lab results must be preserved for the duration of the study. +- Retention periods start from the date of creation. diff --git a/docs/evidence/POLICY_FLAKY_TEST.md b/docs/evidence/POLICY_FLAKY_TEST.md new file mode 100644 index 00000000..6dbc9dc6 --- /dev/null +++ b/docs/evidence/POLICY_FLAKY_TEST.md @@ -0,0 +1,19 @@ +# Flaky Test Quarantine Policy + +How flaky tests are handled. + +## Definition +A flaky test is one that passes and fails without code changes. + +## Process +1. Identify the flaky test and document the failure pattern in an issue. +2. If the test is low-value, remove it. +3. If the test is high-value, quarantine it by marking with `pytest.mark.flaky`. +4. Investigate the root cause within one sprint. +5. Fix or permanently remove the test. + +## Quarantine Rules +- Flaky tests must not block CI. +- Quarantined tests should be tracked in a GitHub issue. +- Quarantined tests should be reviewed weekly. +- If a test remains flaky for more than 2 sprints, it should be removed. diff --git a/docs/evidence/SCHEMA_LIMITATIONS_SECTION.md b/docs/evidence/SCHEMA_LIMITATIONS_SECTION.md new file mode 100644 index 00000000..16c03687 --- /dev/null +++ b/docs/evidence/SCHEMA_LIMITATIONS_SECTION.md @@ -0,0 +1,26 @@ +# Limitations Section Schema + +Standard structure for limitations sections in documentation. + +## Required Sections +- What the limitation is +- Why it exists +- What it does NOT affect +- Planned mitigation or status + +## Example +```markdown +### Charge-Dominated Scoring +The pipeline's primary discriminative signal is charge density, +not sophisticated multi-feature scoring. The ensemble AUROC of +0.7792 is charge-inflated (collapses to 0.5103 under exact charge +control). This does NOT affect the safety or novelty scoring. +**Mitigation:** `charge_bias` score available per candidate; +charge-matched benchmarks track this gap. +``` + +## Rules +- Every limitations section should follow this schema. +- Limitations should be specific, not general. +- Include both impact and mitigation. +- Link to related docs where available. diff --git a/tests/test_artifact_bundle_validator.py b/tests/test_artifact_bundle_validator.py new file mode 100644 index 00000000..ca8e5a59 --- /dev/null +++ b/tests/test_artifact_bundle_validator.py @@ -0,0 +1,20 @@ +"""Test artifact bundle validation.""" +from pathlib import Path + + +def test_batch_pack_script_exists(): + assert Path("scripts/build_lab_batch_pack.py").exists() + + +def test_evidence_directory_exists(): + ev = Path("outputs/evidence") + if ev.exists(): + certs = list(ev.glob("*.json")) + assert len(certs) > 0, "Evidence directory exists but is empty" + + +def test_phase3_evidence_exists(): + ev = Path("outputs/evidence_wave0_5") + if ev.exists(): + certs = list(ev.glob("*.json")) + assert len(certs) > 0