Skip to content

tech-debt: no gate covers .pair/llms.txt drift — the generated KB index goes stale silently #416

Description

@rucka

Story Statement

As a maintainer of the pair knowledge base
I want a quality gate that fails when .pair/llms.txt no longer matches what the generator would emit
So that the machine-readable KB index CLAUDE.md points agents at cannot silently go stale when a guideline is added or removed

Where: a new check next to docs:staleness / skills:conformance, reusing generateLlmsTxt (apps/pair-cli/src/registry/llms-generation.ts), wired into .github/workflows/ci.yml and the root quality-gate script.

Epic Context

Parent Epic: none — cross-cutting quality-gate coverage, sibling of #352 / #384 (mirror integrity) and #259 (tier gate matrix).
Status: Todo
Priority: P2 (Could-Have)

Status Workflow

  • Todo: Story is created but not yet refined
  • Refined: Story is detailed, estimated, and ready for development
  • In Progress: Story is actively being developed
  • Done: Story delivered and accepted

User Value

User Benefit: an agent resolving the KB through .pair/llms.txt (the "AI-friendly index" named in CLAUDE.md / llms.txt convention) can discover every guideline that exists, not only the ones someone remembered to index by hand.
Business Impact: closes an uncovered drift class in a repo whose whole quality story is "gates, not vigilance" — two independent misses are already on record.
Visible UI Value: n/a (CI gate) — demonstrable as a red check on a PR that adds a guideline without regenerating the index.

Rough Sizing

Story Points: S(2)
Confidence: Medium
Reasoning: the generator already exists and is deterministic; the work is a thin check script (generate → compare against the tracked file → diff-on-failure), plus CI/script wiring and a test. Uncertainty is only in where the check lives (pair-cli tool vs. dev-tools quality-gates, alongside code-hygiene-check).

Initial Scope

Likely In Scope

  • A check that runs generateLlmsTxt over the real KB tree and fails when the output differs from the tracked .pair/llms.txt, printing the missing/extra lines.
  • Wiring into .github/workflows/ci.yml (next to docs:staleness, skills:conformance) and the root quality-gate script.
  • A unit test pinning "add a guideline file ⇒ the check fails", so the gate cannot silently stop covering the class.

Likely Out of Scope

  • Auto-regenerating the file in CI (fix-mode belongs to the local quality-gate, not to the pre-merge check).
  • Extending docs:staleness itself (it scans apps/website; different scope, different owner).
  • Any change to the generator's output format.

Open Questions

  • Which package owns the check — @pair/dev-tools quality-gates (where code-hygiene-check lives) or apps/pair-cli (where the generator lives)?
  • Should the same check cover the dataset side, or is .pair/llms.txt the only generated index tracked in git?
  • Fix-mode (llms:fix) in quality-gate alongside prettier:fix / mdlint:fix, or check-only?

Definition of Done Expectations

Standard DoD Requirements (to be detailed during refinement):

  • Functionality implemented and working (check fails on a seeded drift, passes on a clean tree)
  • Automated tests written and passing
  • Code reviewed and merged
  • Documentation updated (way-of-working / quality-gates reference lists the new gate)
  • Wired into CI and the root quality-gate script

Dependencies

Story Dependencies: none blocking. Related: #259 (tier gate matrix), #352 / #384 (mirror-equality guards).
Epic Dependencies: none.

Notes

Raised in the independent review of PR #411 (story #407), round 4 — recorded there as an uncovered drift class rather than fixed in that PR.

Evidence of recurrence (two independent misses, both caught by a human/AI reviewer rather than a gate):

  1. .pair/knowledge/guidelines/architecture/design-patterns/coupling-balance.md — added earlier (chore: post-merge fixups — assess-coupling no longer 'pending #263' + prettier format #385 lineage), never indexed.
  2. .pair/knowledge/guidelines/technical-standards/ai-development/skill-conventions/nested-sub-documents.md — added by PR [US-407] fix: a skill's nested references/ installs inside it — bounded flatten + link rebase + config wiring #411, indexed only after the review round-4 finding.

Why no existing gate covers it:

Verification recipe for the gate (the one used during review): collect every .md under .pair/knowledge/guidelines, collect every indexed path from .pair/llms.txt, assert the two sets are equal.

Metadata

Metadata

Assignees

No one assigned

    Labels

    tech-debtTracked technical debt (living backlog, R7.2 — never blocks a PR)user storyWork item representing a user story

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions