diff --git a/docs/evidence/CHECKLIST_REVIEW_REQUEST.md b/docs/evidence/CHECKLIST_REVIEW_REQUEST.md new file mode 100644 index 00000000..5f5b2d56 --- /dev/null +++ b/docs/evidence/CHECKLIST_REVIEW_REQUEST.md @@ -0,0 +1,10 @@ +# Review Request Checklist + +Before requesting a review, ensure: + +- [ ] All tests pass (`python3 -m pytest -q`) +- [ ] No broken doc links (`python scripts/check_doc_links.py`) +- [ ] No overclaiming language (`python scripts/safety/check_claims.py`) +- [ ] PR description explains what changed and why +- [ ] PR references the related issue +- [ ] Changes are limited to the scope described in the PR diff --git a/docs/evidence/GUIDE_DATASET_SPLIT_DOCS.md b/docs/evidence/GUIDE_DATASET_SPLIT_DOCS.md new file mode 100644 index 00000000..6e4066e1 --- /dev/null +++ b/docs/evidence/GUIDE_DATASET_SPLIT_DOCS.md @@ -0,0 +1,19 @@ +# Dataset Split Documentation Guide + +How to document dataset splits used in benchmarks. + +## Required Information +- Split method (random, cluster, time-based) +- Split ratio (e.g., 80/20) +- Number of folds (for cross-validation) +- Whether stratification was used +- Random seed used for reproducibility + +## Example +```markdown +## Split +- Method: Cluster split at 70% sequence similarity +- Result: 374 independent clusters +- Train: 305 clusters, Test: 69 clusters +- Random seed: 20260705 +``` diff --git a/docs/evidence/GUIDE_DISSENT_UNCERTAINTY.md b/docs/evidence/GUIDE_DISSENT_UNCERTAINTY.md new file mode 100644 index 00000000..734b6969 --- /dev/null +++ b/docs/evidence/GUIDE_DISSENT_UNCERTAINTY.md @@ -0,0 +1,17 @@ +# Dissent and Uncertainty Guide + +How to handle disagreement and uncertainty in reviews. + +## Dissent +- A reviewer may dissent from the majority opinion. +- Dissent must be documented in the review comments. +- Dissent does not block the PR unless the reviewer has blocking authority. +- Dissent should include the reasoning and, if possible, an alternative proposal. + +## Uncertainty +- If a reviewer is uncertain about a change, they should ask for clarification. +- If uncertainty cannot be resolved, escalate to a maintainer. +- Uncertainty should not block a change indefinitely — set a deadline for resolution. + +## Recording +All dissents and significant uncertainties should be logged in the decision log. diff --git a/docs/evidence/GUIDE_EXAMPLE_DATA_LICENSING.md b/docs/evidence/GUIDE_EXAMPLE_DATA_LICENSING.md new file mode 100644 index 00000000..8c48b371 --- /dev/null +++ b/docs/evidence/GUIDE_EXAMPLE_DATA_LICENSING.md @@ -0,0 +1,17 @@ +# Example Data Licensing Note + +Example and demo data in the repository must have clear licensing. + +## Current Example Data +| File | License | Source | +|------|---------|--------| +| examples/sequences/demo_candidates.csv | CC0 | Generated for demo | +| examples/known_reference/demo_known_amps.csv | CC BY 4.0 | Derived from public AMP databases | +| examples/validation/known_amps_500.csv | CC BY 4.0 | UniProt reviewed + APD6 | +| examples/validation/hemolysis_reference.csv | CC BY 4.0 | Literature-derived | + +## Rules +- All example data must have a license header or accompanying LICENSE file. +- Data with unknown or incompatible licenses must not be included. +- Synthetic data must be labeled as synthetic. +- Users are responsible for compliance with data licenses. diff --git a/docs/evidence/GUIDE_EXTERNAL_REVIEWER_ONBOARDING.md b/docs/evidence/GUIDE_EXTERNAL_REVIEWER_ONBOARDING.md new file mode 100644 index 00000000..2e375ddf --- /dev/null +++ b/docs/evidence/GUIDE_EXTERNAL_REVIEWER_ONBOARDING.md @@ -0,0 +1,18 @@ +# External Reviewer Onboarding Note + +When onboarding an external reviewer, provide: + +## Required +- Project README.md for context +- LIMITATIONS_OVERVIEW.md for honest caveats +- The specific artifact or claim being reviewed +- The review questionnaire + +## Optional +- METRICS_CURRENT.md for benchmark context +- ARCHITECTURE.md for system design +- Relevant concept cards + +## What NOT to Share +- Unpublished candidate lists before review is complete +- Sensitive or dual-use information without clear need diff --git a/docs/evidence/GUIDE_FIXTURE_DESIGN.md b/docs/evidence/GUIDE_FIXTURE_DESIGN.md new file mode 100644 index 00000000..57ab3525 --- /dev/null +++ b/docs/evidence/GUIDE_FIXTURE_DESIGN.md @@ -0,0 +1,19 @@ +# Fixture Design Guide + +How to design test fixtures for the pipeline. + +## Principles +- Fixtures should be minimal — only include what's needed for the test. +- Fixtures should be deterministic — same input always produces same output. +- Fixtures should be labeled as synthetic or demo, not real data. +- Fixtures should be small enough to be reviewed manually. + +## Location +- Test fixtures go in `tests/fixtures/`. +- Demo fixtures go in `examples/`. +- Never use real candidate data as test fixtures. + +## Required Fields +- `candidate_id` — clearly identifies the fixture +- `sequence` — valid amino acid sequence +- `source` — indicates it's a fixture (e.g., `test_fixture`, `demo`) diff --git a/docs/evidence/GUIDE_REVIEWER_QUESTIONNAIRE.md b/docs/evidence/GUIDE_REVIEWER_QUESTIONNAIRE.md new file mode 100644 index 00000000..e097759f --- /dev/null +++ b/docs/evidence/GUIDE_REVIEWER_QUESTIONNAIRE.md @@ -0,0 +1,18 @@ +# Reviewer Questionnaire Guide + +Questions a reviewer should be able to answer after reviewing a PR. + +## For the Change +- Does the change do what it claims? +- Are there any side effects or regressions? +- Is the change backward-compatible? + +## For Safety +- Does the change introduce overclaiming language? +- Does the change affect safety policy? +- Does the change require a safety review? + +## For Quality +- Are there tests for the change? +- Do the tests pass? +- Is the documentation updated? diff --git a/docs/evidence/GUIDE_REVIEW_DECISION_VOCAB.md b/docs/evidence/GUIDE_REVIEW_DECISION_VOCAB.md new file mode 100644 index 00000000..36db7978 --- /dev/null +++ b/docs/evidence/GUIDE_REVIEW_DECISION_VOCAB.md @@ -0,0 +1,16 @@ +# Review Decision Vocabulary + +Standard terms for review decisions. + +| Decision | Meaning | Action Required | +|----------|---------|----------------| +| Approved | Ready to merge | Merge the PR | +| Approved with suggestions | Ready but has optional improvements | Consider suggestions | +| Changes requested | Must address before merge | Make changes, re-request review | +| Blocked | Cannot proceed | Resolve the blocking issue first | +| Deferred | Postponed to a later milestone | Close without merging | + +## Rules +- "Approved" with no changes means merge immediately. +- "Changes requested" means re-review is required. +- "Blocked" requires escalation to maintainers. diff --git a/docs/evidence/GUIDE_TOY_DATA_MARKERS.md b/docs/evidence/GUIDE_TOY_DATA_MARKERS.md new file mode 100644 index 00000000..dd36594a --- /dev/null +++ b/docs/evidence/GUIDE_TOY_DATA_MARKERS.md @@ -0,0 +1,23 @@ +# Toy Data Marker Guide + +Toy and demo data must be clearly marked to avoid confusion with real data. + +## Required Markers +- File header or disclaimer: `SYNTHETIC` or `DEMO` +- Source field: `toy`, `demo`, or `synthetic` +- No toy data should have a source that implies real biological data + +## Where to Apply +- Demo candidate CSVs +- Demo reference CSVs +- Synthetic lab results +- Test fixtures + +## Examples +```csv +candidate_id,sequence,source +DEMO-001,ACDEFGHIKLMNPQRSTVWY,demo +``` +```json +{"disclaimer": "SYNTHETIC — replace with real data"} +``` diff --git a/docs/evidence/STANDARD_ARTIFACT_NAMING.md b/docs/evidence/STANDARD_ARTIFACT_NAMING.md new file mode 100644 index 00000000..355ab7d2 --- /dev/null +++ b/docs/evidence/STANDARD_ARTIFACT_NAMING.md @@ -0,0 +1,19 @@ +# Example Artifact Naming Convention + +Standard naming for example artifacts. + +## Patterns +| Type | Pattern | Example | +|------|---------|---------| +| Demo candidates | `demo_{description}.csv` | `demo_candidates.csv` | +| Demo references | `demo_{description}.csv` | `demo_known_amps.csv` | +| Demo output | `demo_{description}.jsonl` | `demo_ranked.jsonl` | +| Lab results | `{candidate_id}_{assay_type}.json` | `SEED-001_VAR_064_MIC.json` | +| Evidence certs | `{candidate_id}.json` | `AMPF-000001.json` | +| Benchmarks | `{description}_benchmark.json` | `validate_scoring_500.json` | + +## Rules +- Use snake_case for file names. +- Use lowercase for all file names. +- Group related files in subdirectories. +- Avoid spaces and special characters. diff --git a/tests/test_review_data_batch.py b/tests/test_review_data_batch.py new file mode 100644 index 00000000..580b4970 --- /dev/null +++ b/tests/test_review_data_batch.py @@ -0,0 +1,20 @@ +"""Tests for review and data docs batch.""" +from pathlib import Path + +FILES = [ + "CHECKLIST_REVIEW_REQUEST.md", + "GUIDE_REVIEWER_QUESTIONNAIRE.md", + "GUIDE_EXTERNAL_REVIEWER_ONBOARDING.md", + "GUIDE_REVIEW_DECISION_VOCAB.md", + "GUIDE_DISSENT_UNCERTAINTY.md", + "GUIDE_DATASET_SPLIT_DOCS.md", + "GUIDE_FIXTURE_DESIGN.md", + "STANDARD_ARTIFACT_NAMING.md", + "GUIDE_EXAMPLE_DATA_LICENSING.md", + "GUIDE_TOY_DATA_MARKERS.md", +] + + +def test_all_exist(): + for f in FILES: + assert Path("docs/evidence", f).exists(), f"Missing: {f}"