diff --git a/docs/evidence/GUIDE_CALIBRATION_AUDIT_READING.md b/docs/evidence/GUIDE_CALIBRATION_AUDIT_READING.md new file mode 100644 index 00000000..42fad26b --- /dev/null +++ b/docs/evidence/GUIDE_CALIBRATION_AUDIT_READING.md @@ -0,0 +1,21 @@ +# Calibration Audit Reading Guide + +How to read a calibration audit report. + +## Key Sections +| Section | What to Look For | +|---------|------------------| +| Intake report | How many results matched? Any orphans? | +| Gate verdict | Did the gate pass? If not, why? | +| Weight proposal | What weights would change? By how much? | +| Before/after comparison | Would the change improve or degrade benchmarks? | + +## Red Flags +- Gate rejected (exit code 3) — recalibration is not allowed +- Weight change > 10% — may indicate a fundamental model issue +- No calibration data — synthetic data only, no real results +- Proposal applied without human review — safety violation + +## Related +- `configs/recalibration_policy.yaml` — the gate policy +- `docs/evidence/CALIBRATION_POLICY.md` — full policy documentation diff --git a/docs/evidence/GUIDE_CALIBRATION_GATE_EXPLAINER.md b/docs/evidence/GUIDE_CALIBRATION_GATE_EXPLAINER.md new file mode 100644 index 00000000..6478e795 --- /dev/null +++ b/docs/evidence/GUIDE_CALIBRATION_GATE_EXPLAINER.md @@ -0,0 +1,16 @@ +# Calibration Gate Explainer + +The calibration gate is a safety mechanism that prevents weight updates +unless pre-registered conditions are met. + +## How It Works +1. Lab results feed into `calibration-intake`. +2. The intake report is evaluated against `configs/recalibration_policy.yaml`. +3. The gate emits a binary verdict: `may_recalibrate` or not. +4. If blocked, no weight updates are allowed. + +## Key Rules +- Exit code 0 = gate allows recalibration. +- Exit code 3 = gate rejects recalibration. +- The gate cannot be bypassed. +- Human review is required before any weight update, even if the gate passes. diff --git a/docs/evidence/GUIDE_CALIBRATION_PHILOSOPHY.md b/docs/evidence/GUIDE_CALIBRATION_PHILOSOPHY.md new file mode 100644 index 00000000..83666afd --- /dev/null +++ b/docs/evidence/GUIDE_CALIBRATION_PHILOSOPHY.md @@ -0,0 +1,17 @@ +# Calibration Philosophy Page + +Calibration adjusts model outputs based on real data. It is not a fix for +fundamental model flaws — it adjusts confidence, not correctness. + +## Principles +1. Calibration requires real data. Synthetic data cannot calibrate a model. +2. Calibration is gated by policy (see `configs/recalibration_policy.yaml`). +3. Calibration updates are proposed, not applied — human review is required. +4. Calibration cannot change the definition of success after the fact. +5. Calibration cannot override safety constraints. + +## What Calibration Is NOT +- A way to make a bad model good +- A substitute for wet-lab validation +- A license to change scoring rules after seeing results +- An automated process — human review is always required diff --git a/docs/evidence/GUIDE_CALIBRATION_ROLLBACK.md b/docs/evidence/GUIDE_CALIBRATION_ROLLBACK.md new file mode 100644 index 00000000..437757c4 --- /dev/null +++ b/docs/evidence/GUIDE_CALIBRATION_ROLLBACK.md @@ -0,0 +1,16 @@ +# Calibration Rollback Guide + +How to roll back a calibration update if it causes issues. + +## When to Roll Back +- A calibration update causes benchmark regression > 0.02. +- A calibration update introduces unexpected behavior. +- A calibration update was applied without proper review. + +## Process +1. Identify the version before the calibration update. +2. Revert the weight changes in `configs/pipeline.yaml`. +3. Run `make bench-gate` to verify the regression is resolved. +4. Run `make full-reproducibility-report` to confirm all checks pass. +5. Log the rollback in the decision log with the reason. +6. Investigate what went wrong before attempting recalibration again. diff --git a/docs/evidence/GUIDE_DOCS_DUPLICATION_REVIEW.md b/docs/evidence/GUIDE_DOCS_DUPLICATION_REVIEW.md new file mode 100644 index 00000000..824723c1 --- /dev/null +++ b/docs/evidence/GUIDE_DOCS_DUPLICATION_REVIEW.md @@ -0,0 +1,21 @@ +# Docs Duplication Review Guide + +How to find and resolve documentation duplication. + +## Signs of Duplication +- Same concept described in multiple documents. +- Same policy stated in different words in different places. +- Same procedure documented in multiple runbooks. +- Cross-references that point to each other (circular). + +## Resolution +1. Identify which document is the authoritative source. +2. Keep the content in the authoritative source. +3. Replace duplicate content with a cross-reference. +4. If the duplicate contains unique information, merge it into the authoritative source. +5. Update all cross-references to point to the single source. + +## Prevention +- Before creating a new document, check if the content already exists. +- Use cross-references instead of copying content. +- Maintain a source-of-truth map (see DOC_MAP_SOURCE_OF_TRUTH.md). diff --git a/docs/evidence/GUIDE_DOCS_ISSUE_TEMPLATES.md b/docs/evidence/GUIDE_DOCS_ISSUE_TEMPLATES.md new file mode 100644 index 00000000..fb96de1b --- /dev/null +++ b/docs/evidence/GUIDE_DOCS_ISSUE_TEMPLATES.md @@ -0,0 +1,18 @@ +# Docs Issue Template Guide + +How to create and use issue templates for documentation tasks. + +## Available Templates +| Template | Location | When to Use | +|----------|----------|-------------| +| Bug report | `.github/ISSUE_TEMPLATE/bug_report.md` | Something is broken | +| Documentation improvement | `.github/ISSUE_TEMPLATE/documentation_improvement.md` | Suggesting a doc change | +| Expert review | `.github/ISSUE_TEMPLATE/expert_review.yml` | Requesting expert review | +| Safety review | `.github/ISSUE_TEMPLATE/safety_review.md` | Safety concern | +| Model release review | `.github/ISSUE_TEMPLATE/model_release_review.md` | Model release request | +| Data contribution | `.github/ISSUE_TEMPLATE/data_contribution.md` | Contributing data | + +## When to Use a Blank Issue +- The task doesn't fit any existing template. +- The task is exploratory and doesn't have a clear template yet. +- The task spans multiple template categories. diff --git a/docs/evidence/GUIDE_DOCS_NAVIGATION_AUDIT.md b/docs/evidence/GUIDE_DOCS_NAVIGATION_AUDIT.md new file mode 100644 index 00000000..9b78a1a4 --- /dev/null +++ b/docs/evidence/GUIDE_DOCS_NAVIGATION_AUDIT.md @@ -0,0 +1,18 @@ +# Docs Navigation Audit Guide + +How to audit documentation navigation. + +## Checklist +- [ ] PROJECT_INDEX.md lists all active documents. +- [ ] Each document category has a hub page. +- [ ] Every document has at least one incoming link from an index. +- [ ] No document is orphaned (no incoming links). +- [ ] Navigation paths are logical (start -> next -> next). +- [ ] Related documents are cross-linked. + +## Process +1. Run `python scripts/check_doc_links.py` to find broken links. +2. Check PROJECT_INDEX.md against `find docs/ -name "*.md"`. +3. For each document not in PROJECT_INDEX.md, add it or archive it. +4. For each document with no incoming links, add cross-references. +5. Update navigation paths if the document structure has changed. diff --git a/docs/evidence/GUIDE_DOCS_PR_DESCRIPTION.md b/docs/evidence/GUIDE_DOCS_PR_DESCRIPTION.md new file mode 100644 index 00000000..1166f7cb --- /dev/null +++ b/docs/evidence/GUIDE_DOCS_PR_DESCRIPTION.md @@ -0,0 +1,25 @@ +# Docs PR Description Template Guide + +How to write good PR descriptions for documentation changes. + +## Recommended Format +```markdown +## Change +Brief description of what changed. + +## Why +Why the change was needed. + +## Related Issues +Closes #NNN + +## Verification +- [ ] `python scripts/check_doc_links.py` — 0 broken links +- [ ] `python scripts/safety/check_claims.py` — 0 overclaiming +- [ ] `python3 -m pytest -q` — all tests pass +``` + +## Rules +- Link to the related issue using "Closes #NNN". +- List what verification you performed. +- Keep the description focused on what changed, not how. diff --git a/docs/evidence/GUIDE_DOC_EXAMPLES_STYLE.md b/docs/evidence/GUIDE_DOC_EXAMPLES_STYLE.md new file mode 100644 index 00000000..fe58a825 --- /dev/null +++ b/docs/evidence/GUIDE_DOC_EXAMPLES_STYLE.md @@ -0,0 +1,24 @@ +# Doc Examples Style Guide + +How to write examples in documentation. + +## Command Examples +- Use `bash` code blocks with `$` prefix for commands. +- Show the expected output format, not the exact output. +- Keep examples runnable from a clean checkout. + +## Code Examples +- Use `python` code blocks for Python examples. +- Show only the relevant parts, not the entire file. +- Use comments to explain non-obvious behavior. + +## Data Examples +- Use `csv` or `json` code blocks for data examples. +- Show only a few rows for CSV examples. +- Use realistic but clearly synthetic data. +- Mark synthetic data with a comment or note. + +## Rules +- All examples must work from a clean checkout. +- Examples must not contain real candidate data. +- Examples must include a disclaimer if showing pipeline output. diff --git a/docs/evidence/GUIDE_NEGATIVE_RESULT_LEARNING.md b/docs/evidence/GUIDE_NEGATIVE_RESULT_LEARNING.md new file mode 100644 index 00000000..f62dd078 --- /dev/null +++ b/docs/evidence/GUIDE_NEGATIVE_RESULT_LEARNING.md @@ -0,0 +1,21 @@ +# Negative-Result Learning Guide + +How to learn from negative results. + +## What Negative Results Teach Us +- Which candidate classes are overpredicted by the pipeline +- Which safety filters are insufficient +- Which benchmark assumptions are wrong +- Where the pipeline needs improvement + +## Process +1. Record the negative result in the negative result archive. +2. Analyze why the result was negative (model error, data error, genuine negative). +3. If the model was wrong, identify what feature or weight contributed to the error. +4. Document the finding in the decision log. +5. If multiple negative results point to the same issue, prioritize fixing it. + +## Rules +- Negative results are not failures — they are calibration data. +- Negative results must be preserved (see NEGATIVE_RESULT_ARCHIVE.md). +- Pipeline weights should not be changed based on a single negative result. diff --git a/docs/evidence/GUIDE_RESULT_QUALITY_FLAGS.md b/docs/evidence/GUIDE_RESULT_QUALITY_FLAGS.md new file mode 100644 index 00000000..75a5ab80 --- /dev/null +++ b/docs/evidence/GUIDE_RESULT_QUALITY_FLAGS.md @@ -0,0 +1,16 @@ +# Result-Quality Flag Guide + +Lab results should be flagged by quality before being used for calibration. + +## Quality Levels +| Flag | Meaning | Use in Calibration? | +|------|---------|:-------------------:| +| `high` | Controls passed, triplicate, clear readout | Yes | +| `medium` | Controls passed, single replicate | Yes, with caution | +| `low` | One control failed or unclear readout | No | +| `invalid` | Controls failed or data is missing | No | + +## Rules +- Results flagged `low` or `invalid` must not be used for calibration. +- Results flagged `medium` should be noted in the calibration report. +- Quality flags must be set by the lab partner, not the pipeline. diff --git a/tests/test_calibration_templates_batch.py b/tests/test_calibration_templates_batch.py new file mode 100644 index 00000000..79f634cf --- /dev/null +++ b/tests/test_calibration_templates_batch.py @@ -0,0 +1,21 @@ +"""Tests for calibration and docs templates batch.""" +from pathlib import Path + +FILES = [ + "GUIDE_CALIBRATION_PHILOSOPHY.md", + "GUIDE_CALIBRATION_GATE_EXPLAINER.md", + "GUIDE_RESULT_QUALITY_FLAGS.md", + "GUIDE_CALIBRATION_ROLLBACK.md", + "GUIDE_CALIBRATION_AUDIT_READING.md", + "GUIDE_NEGATIVE_RESULT_LEARNING.md", + "GUIDE_DOCS_ISSUE_TEMPLATES.md", + "GUIDE_DOCS_PR_DESCRIPTION.md", + "GUIDE_DOC_EXAMPLES_STYLE.md", + "GUIDE_DOCS_NAVIGATION_AUDIT.md", + "GUIDE_DOCS_DUPLICATION_REVIEW.md", +] + + +def test_all_exist(): + for f in FILES: + assert Path("docs/evidence", f).exists(), f"Missing: {f}"