Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions docs/evidence/DIAGRAM_ARTIFACT_DEPENDENCY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Artifact Dependency Graph

Shows how artifacts depend on each other.

```mermaid
flowchart TD
A[Candidate CSV] --> B[Ranked JSONL]
A --> C[Evidence certs]
B --> D[Batch report]
B --> E[Run manifest]
C --> F[Lab batch pack]
F --> G[Lab results]
G --> H[Calibration intake]
H --> I[Gate verdict]
I -->|pass| J[Weight proposal]
I -->|fail| K[Blocked]
```

## Rules
- Artifacts higher in the graph are inputs.
- Arrows show the direction of data flow.
- Dotted lines indicate optional dependencies.
- The graph should be updated when new artifact types are added.
19 changes: 19 additions & 0 deletions docs/evidence/GUIDE_ISSUE_ARTIFACT_TRACE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Issue-to-Artifact Trace Field

How issues trace to their resulting artifacts.

## Format
Issues should include a `## Artifacts` section when resolved:
```markdown
## Artifacts
- PR: #NNN
- Doc: docs/evidence/EXAMPLE.md
- Script: scripts/example.py
- Test: tests/test_example.py
```

## Rules
- Every closed issue should trace to at least one artifact.
- Artifacts can be PRs, docs, scripts, tests, or configs.
- The trace helps future contributors understand why something exists.
- If an issue doesn't produce artifacts (investigation only), note that.
22 changes: 22 additions & 0 deletions docs/evidence/GUIDE_LINEAGE_RENDERER.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Artifact Lineage Markdown Renderer

How artifact lineage is documented in markdown.

## Format
```markdown
## Artifact Lineage

```mermaid
flowchart LR
A[Input CSV] --> B[score_candidates]
B --> C[Ranked JSONL]
B --> D[Evidence certs]
```
```

## Rules
- Lineage diagrams should use Mermaid flowchart format.
- Show inputs → process → outputs.
- Label each node with the artifact name and format.
- Keep lineage diagrams focused on one artifact type at a time.
- Include the lineage in the artifact's documentation.
21 changes: 21 additions & 0 deletions docs/evidence/GUIDE_STALE_ARTIFACT_DETECTOR.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Stale Artifact Detector

Detects artifacts that may be stale (out of sync with current code).

## What It Checks
1. Schema files — compare schema version against code.
2. Benchmark outputs — check if they match current pipeline.
3. Generated docs — check if they reference current features.
4. Config files — check if all keys are still valid.

## Usage
```bash
python scripts/check_doc_links.py # Check docs for broken references
make bench-gate # Check benchmark regression gates
python scripts/check_pyproject.py # Check pyproject.toml consistency
```

## Manual Checks
- Evidence certificates should be regenerated after scoring changes.
- Benchmarks should be re-run after config changes.
- Docs should be reviewed after major feature changes.
16 changes: 16 additions & 0 deletions docs/evidence/MATRIX_CLAIM_EVIDENCE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Claim-to-Evidence Matrix Artifact

Maps claims to their supporting evidence artifacts.

| Claim | Evidence Level | Supporting Artifact | Status |
|-------|:--------------:|--------------------|:------:|
| Pipeline ranks AMPs | 2 (baseline) | METRICS_CURRENT.md | ✅ |
| Pipeline is reproducible | 2 (baseline) | RUN_MANIFEST_STANDARD.md | ✅ |
| Candidates are novel | 2 (baseline) | NOVELTY_CHECKLIST.md | ✅ |
| Candidates are safe | 2 (baseline) | LIMITATIONS_OVERVIEW.md | ⚠️ Limited |

## Rules
- Every claim should be paired with its evidence.
- The evidence level should match the claim strength.
- If evidence is missing, mark the claim as unsupported.
- This matrix should be reviewed before any public release.
18 changes: 18 additions & 0 deletions docs/evidence/POLICY_ARTIFACT_RETENTION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Artifact Retention Policy

How long different artifact types are retained.

| Artifact Type | Retention | Location | Cleanup |
|:-------------|:---------:|:---------|:--------|
| Evidence certificates | Permanent | outputs/evidence/ | Manual |
| Run manifests | 2 years | outputs/ | Automated |
| Benchmark results | Permanent | outputs/ | Manual |
| Lab results | Permanent | outputs/lab_results/ | Manual |
| Temporary outputs | 90 days | outputs/ | Automated |
| Logs | 30 days | outputs/logs/ | Automated |

## Rules
- Permanent artifacts must never be deleted without a decision log entry.
- Temporary outputs can be cleaned up with `make clean`.
- Lab results must be preserved for the duration of the study.
- Retention periods start from the date of creation.
19 changes: 19 additions & 0 deletions docs/evidence/POLICY_FLAKY_TEST.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Flaky Test Quarantine Policy

How flaky tests are handled.

## Definition
A flaky test is one that passes and fails without code changes.

## Process
1. Identify the flaky test and document the failure pattern in an issue.
2. If the test is low-value, remove it.
3. If the test is high-value, quarantine it by marking with `pytest.mark.flaky`.
4. Investigate the root cause within one sprint.
5. Fix or permanently remove the test.

## Quarantine Rules
- Flaky tests must not block CI.
- Quarantined tests should be tracked in a GitHub issue.
- Quarantined tests should be reviewed weekly.
- If a test remains flaky for more than 2 sprints, it should be removed.
26 changes: 26 additions & 0 deletions docs/evidence/SCHEMA_LIMITATIONS_SECTION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Limitations Section Schema

Standard structure for limitations sections in documentation.

## Required Sections
- What the limitation is
- Why it exists
- What it does NOT affect
- Planned mitigation or status

## Example
```markdown
### Charge-Dominated Scoring
The pipeline's primary discriminative signal is charge density,
not sophisticated multi-feature scoring. The ensemble AUROC of
0.7792 is charge-inflated (collapses to 0.5103 under exact charge
control). This does NOT affect the safety or novelty scoring.
**Mitigation:** `charge_bias` score available per candidate;
charge-matched benchmarks track this gap.
```

## Rules
- Every limitations section should follow this schema.
- Limitations should be specific, not general.
- Include both impact and mitigation.
- Link to related docs where available.
20 changes: 20 additions & 0 deletions tests/test_artifact_bundle_validator.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
"""Test artifact bundle validation."""
from pathlib import Path


def test_batch_pack_script_exists():
assert Path("scripts/build_lab_batch_pack.py").exists()


def test_evidence_directory_exists():
ev = Path("outputs/evidence")
if ev.exists():
certs = list(ev.glob("*.json"))
assert len(certs) > 0, "Evidence directory exists but is empty"


def test_phase3_evidence_exists():
ev = Path("outputs/evidence_wave0_5")
if ev.exists():
certs = list(ev.glob("*.json"))
assert len(certs) > 0
Loading