Skip to content

feat(selfhost): scheduled docs-drift audit sweep for claims the mechanical lint can't check #3048

Description

@JSONbored

Context

The mechanical docs-drift lint (Phase 1 sibling issue) and schema-anchored examples (Phase 2) catch presence/shape drift -- a flag, command, or field that exists in code but not in docs, or an example that no longer matches a schema. Neither catches BEHAVIORAL prose drift: a claim like "blocking is always confirmed-contributor-gated" that was true when written and quietly stopped being true when the gate's confirmed-contributor handling changed (this exact claim was wrong, duplicated verbatim, across three docs pages -- docs.github-app.tsx, docs.how-reviews-work.tsx, docs.tuning.tsx -- until a 2026-07-04 manual audit caught it).

Goal

A recurring (e.g. monthly), low-noise automated sweep that re-runs a scoped version of the multi-agent docs-audit approach used for the 2026-07-04 pass -- read a docs page's behavioral claims, re-verify each against the current source it describes, cite file:line for any real drift found -- and opens ONE consolidated issue when it finds concrete, cited drift. Must not be noisy: no finding without a real file:line citation, no speculative "this might be stale" without verification against the actual current code.

Requirements

  • Design where this runs: a scheduled GitHub Actions workflow calling into the same self-host/ORB review-agent harness, or a dedicated cron job -- pick whichever reuses the most existing infrastructure rather than building a new agent-orchestration path from scratch.
  • Bound scope explicitly per run (e.g. N pages per sweep, rotating) so a single run's cost/time stays predictable -- don't re-audit all 30 pages every time from day one.
  • One consolidated issue per finding-run, not one issue per finding -- avoid flooding the issue tracker.
  • Cross-reference the Phase 1-3 sibling issues in the design doc: this is explicitly the catch-all for claims those can't mechanically verify, not a replacement for them.

Deliverables

A design doc/RFC as the first step (this issue can stay open through that phase), then the actual scheduled job once the design is agreed.

Expected outcomes

Behavioral/prose drift -- the hardest category to catch mechanically -- gets a periodic backstop instead of relying entirely on the next person who happens to read a page closely enough to notice something is wrong.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

Status
Todo

Relationships

None yet

Development

No branches or pull requests

Issue actions