Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions docs/evidence/GUIDE_CALIBRATION_AUDIT_READING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Calibration Audit Reading Guide

How to read a calibration audit report.

## Key Sections
| Section | What to Look For |
|---------|------------------|
| Intake report | How many results matched? Any orphans? |
| Gate verdict | Did the gate pass? If not, why? |
| Weight proposal | What weights would change? By how much? |
| Before/after comparison | Would the change improve or degrade benchmarks? |

## Red Flags
- Gate rejected (exit code 3) — recalibration is not allowed
- Weight change > 10% — may indicate a fundamental model issue
- No calibration data — synthetic data only, no real results
- Proposal applied without human review — safety violation

## Related
- `configs/recalibration_policy.yaml` — the gate policy
- `docs/evidence/CALIBRATION_POLICY.md` — full policy documentation
16 changes: 16 additions & 0 deletions docs/evidence/GUIDE_CALIBRATION_GATE_EXPLAINER.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Calibration Gate Explainer

The calibration gate is a safety mechanism that prevents weight updates
unless pre-registered conditions are met.

## How It Works
1. Lab results feed into `calibration-intake`.
2. The intake report is evaluated against `configs/recalibration_policy.yaml`.
3. The gate emits a binary verdict: `may_recalibrate` or not.
4. If blocked, no weight updates are allowed.

## Key Rules
- Exit code 0 = gate allows recalibration.
- Exit code 3 = gate rejects recalibration.
- The gate cannot be bypassed.
- Human review is required before any weight update, even if the gate passes.
17 changes: 17 additions & 0 deletions docs/evidence/GUIDE_CALIBRATION_PHILOSOPHY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Calibration Philosophy Page

Calibration adjusts model outputs based on real data. It is not a fix for
fundamental model flaws — it adjusts confidence, not correctness.

## Principles
1. Calibration requires real data. Synthetic data cannot calibrate a model.
2. Calibration is gated by policy (see `configs/recalibration_policy.yaml`).
3. Calibration updates are proposed, not applied — human review is required.
4. Calibration cannot change the definition of success after the fact.
5. Calibration cannot override safety constraints.

## What Calibration Is NOT
- A way to make a bad model good
- A substitute for wet-lab validation
- A license to change scoring rules after seeing results
- An automated process — human review is always required
16 changes: 16 additions & 0 deletions docs/evidence/GUIDE_CALIBRATION_ROLLBACK.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Calibration Rollback Guide

How to roll back a calibration update if it causes issues.

## When to Roll Back
- A calibration update causes benchmark regression > 0.02.
- A calibration update introduces unexpected behavior.
- A calibration update was applied without proper review.

## Process
1. Identify the version before the calibration update.
2. Revert the weight changes in `configs/pipeline.yaml`.
3. Run `make bench-gate` to verify the regression is resolved.
4. Run `make full-reproducibility-report` to confirm all checks pass.
5. Log the rollback in the decision log with the reason.
6. Investigate what went wrong before attempting recalibration again.
21 changes: 21 additions & 0 deletions docs/evidence/GUIDE_DOCS_DUPLICATION_REVIEW.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Docs Duplication Review Guide

How to find and resolve documentation duplication.

## Signs of Duplication
- Same concept described in multiple documents.
- Same policy stated in different words in different places.
- Same procedure documented in multiple runbooks.
- Cross-references that point to each other (circular).

## Resolution
1. Identify which document is the authoritative source.
2. Keep the content in the authoritative source.
3. Replace duplicate content with a cross-reference.
4. If the duplicate contains unique information, merge it into the authoritative source.
5. Update all cross-references to point to the single source.

## Prevention
- Before creating a new document, check if the content already exists.
- Use cross-references instead of copying content.
- Maintain a source-of-truth map (see DOC_MAP_SOURCE_OF_TRUTH.md).
18 changes: 18 additions & 0 deletions docs/evidence/GUIDE_DOCS_ISSUE_TEMPLATES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Docs Issue Template Guide

How to create and use issue templates for documentation tasks.

## Available Templates
| Template | Location | When to Use |
|----------|----------|-------------|
| Bug report | `.github/ISSUE_TEMPLATE/bug_report.md` | Something is broken |
| Documentation improvement | `.github/ISSUE_TEMPLATE/documentation_improvement.md` | Suggesting a doc change |
| Expert review | `.github/ISSUE_TEMPLATE/expert_review.yml` | Requesting expert review |
| Safety review | `.github/ISSUE_TEMPLATE/safety_review.md` | Safety concern |
| Model release review | `.github/ISSUE_TEMPLATE/model_release_review.md` | Model release request |
| Data contribution | `.github/ISSUE_TEMPLATE/data_contribution.md` | Contributing data |

## When to Use a Blank Issue
- The task doesn't fit any existing template.
- The task is exploratory and doesn't have a clear template yet.
- The task spans multiple template categories.
18 changes: 18 additions & 0 deletions docs/evidence/GUIDE_DOCS_NAVIGATION_AUDIT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Docs Navigation Audit Guide

How to audit documentation navigation.

## Checklist
- [ ] PROJECT_INDEX.md lists all active documents.
- [ ] Each document category has a hub page.
- [ ] Every document has at least one incoming link from an index.
- [ ] No document is orphaned (no incoming links).
- [ ] Navigation paths are logical (start -> next -> next).
- [ ] Related documents are cross-linked.

## Process
1. Run `python scripts/check_doc_links.py` to find broken links.
2. Check PROJECT_INDEX.md against `find docs/ -name "*.md"`.
3. For each document not in PROJECT_INDEX.md, add it or archive it.
4. For each document with no incoming links, add cross-references.
5. Update navigation paths if the document structure has changed.
25 changes: 25 additions & 0 deletions docs/evidence/GUIDE_DOCS_PR_DESCRIPTION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Docs PR Description Template Guide

How to write good PR descriptions for documentation changes.

## Recommended Format
```markdown
## Change
Brief description of what changed.

## Why
Why the change was needed.

## Related Issues
Closes #NNN

## Verification
- [ ] `python scripts/check_doc_links.py` — 0 broken links
- [ ] `python scripts/safety/check_claims.py` — 0 overclaiming
- [ ] `python3 -m pytest -q` — all tests pass
```

## Rules
- Link to the related issue using "Closes #NNN".
- List what verification you performed.
- Keep the description focused on what changed, not how.
24 changes: 24 additions & 0 deletions docs/evidence/GUIDE_DOC_EXAMPLES_STYLE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Doc Examples Style Guide

How to write examples in documentation.

## Command Examples
- Use `bash` code blocks with `$` prefix for commands.
- Show the expected output format, not the exact output.
- Keep examples runnable from a clean checkout.

## Code Examples
- Use `python` code blocks for Python examples.
- Show only the relevant parts, not the entire file.
- Use comments to explain non-obvious behavior.

## Data Examples
- Use `csv` or `json` code blocks for data examples.
- Show only a few rows for CSV examples.
- Use realistic but clearly synthetic data.
- Mark synthetic data with a comment or note.

## Rules
- All examples must work from a clean checkout.
- Examples must not contain real candidate data.
- Examples must include a disclaimer if showing pipeline output.
21 changes: 21 additions & 0 deletions docs/evidence/GUIDE_NEGATIVE_RESULT_LEARNING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Negative-Result Learning Guide

How to learn from negative results.

## What Negative Results Teach Us
- Which candidate classes are overpredicted by the pipeline
- Which safety filters are insufficient
- Which benchmark assumptions are wrong
- Where the pipeline needs improvement

## Process
1. Record the negative result in the negative result archive.
2. Analyze why the result was negative (model error, data error, genuine negative).
3. If the model was wrong, identify what feature or weight contributed to the error.
4. Document the finding in the decision log.
5. If multiple negative results point to the same issue, prioritize fixing it.

## Rules
- Negative results are not failures — they are calibration data.
- Negative results must be preserved (see NEGATIVE_RESULT_ARCHIVE.md).
- Pipeline weights should not be changed based on a single negative result.
16 changes: 16 additions & 0 deletions docs/evidence/GUIDE_RESULT_QUALITY_FLAGS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Result-Quality Flag Guide

Lab results should be flagged by quality before being used for calibration.

## Quality Levels
| Flag | Meaning | Use in Calibration? |
|------|---------|:-------------------:|
| `high` | Controls passed, triplicate, clear readout | Yes |
| `medium` | Controls passed, single replicate | Yes, with caution |
| `low` | One control failed or unclear readout | No |
| `invalid` | Controls failed or data is missing | No |

## Rules
- Results flagged `low` or `invalid` must not be used for calibration.
- Results flagged `medium` should be noted in the calibration report.
- Quality flags must be set by the lab partner, not the pipeline.
21 changes: 21 additions & 0 deletions tests/test_calibration_templates_batch.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
"""Tests for calibration and docs templates batch."""
from pathlib import Path

FILES = [
"GUIDE_CALIBRATION_PHILOSOPHY.md",
"GUIDE_CALIBRATION_GATE_EXPLAINER.md",
"GUIDE_RESULT_QUALITY_FLAGS.md",
"GUIDE_CALIBRATION_ROLLBACK.md",
"GUIDE_CALIBRATION_AUDIT_READING.md",
"GUIDE_NEGATIVE_RESULT_LEARNING.md",
"GUIDE_DOCS_ISSUE_TEMPLATES.md",
"GUIDE_DOCS_PR_DESCRIPTION.md",
"GUIDE_DOC_EXAMPLES_STYLE.md",
"GUIDE_DOCS_NAVIGATION_AUDIT.md",
"GUIDE_DOCS_DUPLICATION_REVIEW.md",
]


def test_all_exist():
for f in FILES:
assert Path("docs/evidence", f).exists(), f"Missing: {f}"
Loading