diff --git a/docs/evidence/PLAYBOOK_BENCHMARK_REGRESSION.md b/docs/evidence/PLAYBOOK_BENCHMARK_REGRESSION.md new file mode 100644 index 00000000..04ecb9bd --- /dev/null +++ b/docs/evidence/PLAYBOOK_BENCHMARK_REGRESSION.md @@ -0,0 +1,13 @@ +# Scenario Playbook: Benchmark Regression + +**Scenario:** A benchmark AUROC drops by more than 0.02. + +## Steps + +1. Verify the regression is real (not a fluke) by running the benchmark twice. +2. Check what changed since the last passing run (`git log --oneline`). +3. Isolate the change that caused the regression using `git bisect` if needed. +4. If the regression is caused by a deliberate tradeoff (e.g., safety penalty increase), document the tradeoff. +5. If the regression is unintended, fix or revert the change. +6. Update `docs/evidence/METRICS_CURRENT.md` with the new value. +7. Run `make bench-gate` to verify the fix. diff --git a/docs/evidence/PLAYBOOK_CHANGED_OUTPUT.md b/docs/evidence/PLAYBOOK_CHANGED_OUTPUT.md new file mode 100644 index 00000000..91b61c5e --- /dev/null +++ b/docs/evidence/PLAYBOOK_CHANGED_OUTPUT.md @@ -0,0 +1,16 @@ +# Scenario Playbook: Changed Command Output + +**Scenario:** A CLI command output format changes, potentially breaking downstream consumers. + +## Steps + +1. Determine if the change is intentional or accidental. +2. If intentional: + - Update all documentation that references the old output format. + - Update any tests that check the output format. + - Add a changelog entry describing the change. + - Consider backward compatibility or a migration path. +3. If accidental: + - Fix the code to restore the expected output format. + - Add a test to prevent future regressions. +4. Run the full test suite to verify no other code depends on the old format. diff --git a/docs/evidence/PLAYBOOK_OVERCLAIM_REVIEW.md b/docs/evidence/PLAYBOOK_OVERCLAIM_REVIEW.md new file mode 100644 index 00000000..14dcb74f --- /dev/null +++ b/docs/evidence/PLAYBOOK_OVERCLAIM_REVIEW.md @@ -0,0 +1,14 @@ +# Scenario Playbook: Overclaim Review + +**Scenario:** A reviewer flags overclaiming language in a PR. + +## Steps + +1. Run `python scripts/safety/check_claims.py` to see all flagged phrases. +2. For each flagged phrase, determine if it's a genuine overclaim or an allowed exception. +3. Rewrite overclaiming phrases using the project's allowed terms: + - "computationally nominated candidate" (not "drug candidate") + - "predicted antimicrobial peptide" (not "proven antimicrobial") + - "selected by reproducible pipeline" (not "validated by computation") +4. Re-run the claim scanner to verify the fix. +5. If the original author disagrees with a flag, document the reasoning. diff --git a/docs/evidence/PLAYBOOK_PROOF_CONFUSION.md b/docs/evidence/PLAYBOOK_PROOF_CONFUSION.md new file mode 100644 index 00000000..4d92d90f --- /dev/null +++ b/docs/evidence/PLAYBOOK_PROOF_CONFUSION.md @@ -0,0 +1,11 @@ +# Scenario Playbook: Proof Confusion + +**Scenario:** Someone cites a computational score as biological proof. + +## Steps + +1. Refer them to `docs/evidence/PROOF_LADDER.md` — levels 0-4 are computational. +2. Explain that only levels 5 (lab assay) and 6 (independent replication) constitute biological evidence. +3. If the confusion is in project documentation, update the document to clarify the proof level. +4. If the confusion is in an external communication, provide a correction. +5. Log the incident in the decision log as a documentation clarification. diff --git a/docs/evidence/PLAYBOOK_STALE_ARTIFACT.md b/docs/evidence/PLAYBOOK_STALE_ARTIFACT.md new file mode 100644 index 00000000..7df8230d --- /dev/null +++ b/docs/evidence/PLAYBOOK_STALE_ARTIFACT.md @@ -0,0 +1,17 @@ +# Scenario Playbook: Stale Artifact Finding + +**Scenario:** An artifact (schema, benchmark output, config) is stale and doesn't match the current code. + +## Steps + +1. Identify what's stale: check the artifact's schema version vs current code. +2. If the artifact is a benchmark output, regenerate it: `make bench-500`. +3. If the artifact is a schema, update it to match the current code. +4. If the artifact is a config file, update it or document why the discrepancy is acceptable. +5. Update any documentation that references the stale artifact. +6. Add a test to catch similar staleness in the future. + +## Prevention + +- Run `make regenerate-all` before major releases. +- CI should validate schema versions against code. diff --git a/tests/test_playbooks.py b/tests/test_playbooks.py new file mode 100644 index 00000000..668f8a73 --- /dev/null +++ b/tests/test_playbooks.py @@ -0,0 +1,21 @@ +"""Tests for playbooks.""" +from pathlib import Path + +PLAYBOOKS = [ + "PLAYBOOK_OVERCLAIM_REVIEW.md", + "PLAYBOOK_PROOF_CONFUSION.md", + "PLAYBOOK_BENCHMARK_REGRESSION.md", + "PLAYBOOK_STALE_ARTIFACT.md", + "PLAYBOOK_CHANGED_OUTPUT.md", +] + + +def test_all_exist(): + for p in PLAYBOOKS: + assert Path(f"docs/evidence/{p}").exists(), f"Missing: {p}" + + +def test_each_has_steps(): + for p in PLAYBOOKS: + text = Path(f"docs/evidence/{p}").read_text() + assert "Steps" in text or "steps" in text