diff --git a/docs/evidence/CHECKLIST_MAINTAINER_HANDOFF.md b/docs/evidence/CHECKLIST_MAINTAINER_HANDOFF.md new file mode 100644 index 00000000..2c38e173 --- /dev/null +++ b/docs/evidence/CHECKLIST_MAINTAINER_HANDOFF.md @@ -0,0 +1,12 @@ +# Maintainer Handoff Checklist + +When a maintainer is unavailable or leaving, use this checklist. + +- [ ] Identify all PRs the maintainer is reviewing. +- [ ] Reassign PRs to other maintainers. +- [ ] Identify all issues the maintainer is assigned to. +- [ ] Reassign or unassign issues. +- [ ] Document any in-progress work the maintainer was doing. +- [ ] Update the folder ownership map if needed. +- [ ] Notify the team of the change. +- [ ] Update the decision log. diff --git a/docs/evidence/CHECKLIST_PR_RISK.md b/docs/evidence/CHECKLIST_PR_RISK.md new file mode 100644 index 00000000..43bb2250 --- /dev/null +++ b/docs/evidence/CHECKLIST_PR_RISK.md @@ -0,0 +1,15 @@ +# Pull Request Risk Checklist + +Use this to assess PR risk before review. + +## Questions +- [ ] Does the PR change scoring weights? (High risk) +- [ ] Does the PR change calibration pipeline? (High risk) +- [ ] Does the PR change safety policy? (High risk) +- [ ] Does the PR add new dependencies? (Medium risk) +- [ ] Does the PR change CLI output format? (Medium risk) +- [ ] Does the PR add new functionality? (Medium risk) +- [ ] Does the PR fix a bug? (Low risk) +- [ ] Does the PR improve documentation? (Low risk) + +Based on answers, assign the appropriate review depth per the PR review rubric. diff --git a/docs/evidence/GUIDE_REVIEW_LATENCY.md b/docs/evidence/GUIDE_REVIEW_LATENCY.md new file mode 100644 index 00000000..c842b40e --- /dev/null +++ b/docs/evidence/GUIDE_REVIEW_LATENCY.md @@ -0,0 +1,21 @@ +# Review Latency Tracking Ledger + +Track review latency to identify bottlenecks. + +## Metrics +| Metric | Target | How to Measure | +|--------|:------:|----------------| +| Time to first review | < 24h | Time from PR open to first review comment | +| Time to merge | < 72h | Time from PR open to merge | +| Review depth | All issues addressed | Count of review comments resolved | + +## Ledger Format +```csv +pr_number,author,reviewer,opened_at,first_review_at,merged_at,comments +#NNN,name,name,datetime,datetime,datetime,N +``` + +## Rules +- Review latency is tracked monthly. +- If targets are consistently missed, investigate bottlenecks. +- Review latency data is used for process improvement, not blame. diff --git a/docs/evidence/GUIDE_TASK_SIZE.md b/docs/evidence/GUIDE_TASK_SIZE.md new file mode 100644 index 00000000..bb6dd761 --- /dev/null +++ b/docs/evidence/GUIDE_TASK_SIZE.md @@ -0,0 +1,15 @@ +# Contributor Task Size Guide + +How to estimate task size for issues. + +| Size | Lines of Change | Review Time | Example | +|:----:|:---------------:|:-----------:|---------| +| Small | < 50 | < 30 min | Typo fix, single doc update | +| Medium | 50-200 | 1-2 hours | New doc, small feature, test addition | +| Large | 200-500 | 2-4 hours | New script, policy document | +| XLarge | > 500 | > 4 hours | Major feature, architecture change | + +## Rules +- Large or XLarge tasks should be broken into smaller issues. +- If an issue is XLarge, create a design doc first. +- Task size should be estimated before starting work. diff --git a/docs/evidence/GUIDE_WARNING_TAXONOMY.md b/docs/evidence/GUIDE_WARNING_TAXONOMY.md index e7c94b06..abe4b479 100644 --- a/docs/evidence/GUIDE_WARNING_TAXONOMY.md +++ b/docs/evidence/GUIDE_WARNING_TAXONOMY.md @@ -1,16 +1,16 @@ # Warning Taxonomy Guide -Types of warnings and how to handle them. +Types of warnings produced by the pipeline. -| Warning Type | Meaning | Example | Action | -|-------------|---------|---------|--------| -| `stale_config` | Config option is deprecated | `deprecated key 'foo'` | Update config | -| `missing_field` | Optional field is absent | `field 'bar' not found` | Check source | -| `low_confidence` | Score uncertainty is high | `uncertainty > 0.5` | Treat as experimental | -| `regression` | Metric dropped | `AUROC dropped 0.02` | Investigate | -| `overclaim` | Risky wording detected | `found 'proven'` | Rewrite | +| Type | Meaning | Example | +|------|---------|---------| +| stale_config | Deprecated config key | "key 'foo' is deprecated" | +| missing_field | Optional field missing | "field 'bar' not found" | +| low_confidence | High uncertainty | "uncertainty > 0.5" | +| regression | Metric declined | "AUROC dropped 0.02" | +| overclaim | Risky wording | "found 'proven'" | ## Rules -- Warnings should not change behavior. +- Warnings are logged to stderr, not stdout. - Warnings should include actionable information. -- Warnings should be logged to stderr, not stdout. +- Warnings must not change pipeline behavior. diff --git a/docs/evidence/POLICY_QUIET_MODE.md b/docs/evidence/POLICY_QUIET_MODE.md new file mode 100644 index 00000000..6f21e5a0 --- /dev/null +++ b/docs/evidence/POLICY_QUIET_MODE.md @@ -0,0 +1,16 @@ +# Quiet Mode Policy for CLI Commands + +CLI commands should support a quiet mode for scripting. + +## Behavior +- `--quiet` flag suppresses all non-error output. +- Errors and the final result summary are still printed. +- Progress bars and informational messages are suppressed. + +## Implementation +- Parse `--quiet` in the CLI handler. +- Set log level to WARNING when quiet is enabled. +- Suppress progress bars when quiet is enabled. + +## Related +- `docs/evidence/GUIDE_MINIMAL_OUTPUT.md` diff --git a/docs/evidence/POLICY_SAFE_FILENAMES.md b/docs/evidence/POLICY_SAFE_FILENAMES.md new file mode 100644 index 00000000..57281083 --- /dev/null +++ b/docs/evidence/POLICY_SAFE_FILENAMES.md @@ -0,0 +1,23 @@ +# Safe Filename Policy + +Filenames in the repository must follow these rules. + +## Allowed Characters +- Lowercase letters: `a-z` +- Digits: `0-9` +- Hyphens: `-` +- Underscores: `_` +- Dots: `.` + +## Rules +- No spaces in filenames. +- No special characters except `-` and `_`. +- Use `.md` for documentation, `.py` for Python, `.json` for JSON. +- Use snake_case for Python files and data files. +- Use lowercase for all filenames. + +## Examples +- ✅ `check_doc_links.py` +- ✅ `METRICS_CURRENT.md` +- ❌ `My File.py` +- ❌ `data-file(v2).csv` diff --git a/docs/evidence/POLICY_SECRET_SCAN.md b/docs/evidence/POLICY_SECRET_SCAN.md new file mode 100644 index 00000000..82b807ec --- /dev/null +++ b/docs/evidence/POLICY_SECRET_SCAN.md @@ -0,0 +1,20 @@ +# Secret-Scan Policy for Fixtures and Docs + +Fixtures and documentation must not contain secrets. + +## What to Scan For +- API keys (`sk-...`, `api_key=...`) +- Passwords (`password=...`) +- Tokens (`token=...`, `auth=...`) +- Private keys (`-----BEGIN...`) +- Connection strings (`mongodb://...`, `postgres://...`) + +## Rules +- All fixtures and examples are scanned before commit. +- If a secret is found, remove it and rotate if it was ever committed. +- Use environment variables for any required credentials. +- The `.env` file is git-ignored. + +## Enforcement +- Add secret scanning to CI (e.g., `trufflehog` or `git-secrets`). +- PRs that add new fixtures should be reviewed for secrets. diff --git a/docs/evidence/REGISTRY_TUTORIAL_FIXTURES.md b/docs/evidence/REGISTRY_TUTORIAL_FIXTURES.md new file mode 100644 index 00000000..4481a838 --- /dev/null +++ b/docs/evidence/REGISTRY_TUTORIAL_FIXTURES.md @@ -0,0 +1,15 @@ +# Tutorial Fixture Registry + +All tutorial fixtures must be registered here. + +| Fixture | Location | Used By | +|---------|----------|---------| +| Demo candidates | examples/sequences/demo_candidates.csv | QUICKSTART.md | +| Demo references | examples/known_reference/demo_known_amps.csv | QUICKSTART.md | +| Lab results (synthetic) | examples/lab_results/ | calibration walkthrough | +| Hemolysis reference | examples/validation/hemolysis_reference.csv | selectivity benchmark | + +## Rules +- Add new fixtures to this registry when created. +- Mark fixtures as `synthetic` or `demo` in their metadata. +- Never use real candidate data as tutorial fixtures. diff --git a/docs/evidence/STANDARD_WALKTHROUGH_SNAPSHOTS.md b/docs/evidence/STANDARD_WALKTHROUGH_SNAPSHOTS.md new file mode 100644 index 00000000..f45e1bec --- /dev/null +++ b/docs/evidence/STANDARD_WALKTHROUGH_SNAPSHOTS.md @@ -0,0 +1,21 @@ +# Walkthrough Expected-Output Snapshots + +Walkthroughs should include expected output snapshots so users can verify +they got the right result. + +## Format +```markdown +## Expected Output + +After running `openamp-foundry rank ...`, the first line of the output file +should contain a JSON object with these keys: +- `candidate_id` — string +- `sequence` — string +- `scores` — dict with at least `ensemble`, `activity`, `safety` +- `selected` — boolean +``` + +## Rules +- Show the structure, not exact values. +- If exact values are shown, mark them as examples. +- Update snapshots when output format changes. diff --git a/tests/test_batch_latest.py b/tests/test_batch_latest.py new file mode 100644 index 00000000..23115da3 --- /dev/null +++ b/tests/test_batch_latest.py @@ -0,0 +1,20 @@ +"""Tests for batch of 10 docs.""" +from pathlib import Path + +FILES = [ + "REGISTRY_TUTORIAL_FIXTURES.md", + "STANDARD_WALKTHROUGH_SNAPSHOTS.md", + "GUIDE_TASK_SIZE.md", + "CHECKLIST_PR_RISK.md", + "GUIDE_REVIEW_LATENCY.md", + "CHECKLIST_MAINTAINER_HANDOFF.md", + "POLICY_SECRET_SCAN.md", + "POLICY_SAFE_FILENAMES.md", + "GUIDE_WARNING_TAXONOMY.md", + "POLICY_QUIET_MODE.md", +] + + +def test_all_exist(): + for f in FILES: + assert Path("docs/evidence", f).exists(), f"Missing: {f}"