Review agent should flag accidentally included internal scaffolding files

[fullsend-ai-retro]

What happened

On PR #1814, the author accidentally committed an internal spec file at docs/superpowers/specs/2026-06-02-host-side-api-server-design.md alongside the ADR it was used to generate (ADR 0046). The review bot ran on the initial commit (2026-06-02) and approved with a single [low] finding about Tier 3 scope in architecture.md. It did not flag the spec file. Human reviewer rh-hemartin caught the issue the next day (2026-06-03), leaving 10 comments on the spec file and noting it should be renamed or removed. The author removed it entirely in the next revision (2026-06-09), explaining it was the internal document used by the superpowers skill to create the ADR and was never intended for the PR.

What could go better

The review agent should recognize when a PR includes files from paths that are typically internal scaffolding or tooling artifacts — paths like docs/superpowers/specs/, internal/scaffold/, or similar patterns that indicate generated/source material not meant for the final changeset. This is a high-confidence finding: the file was clearly an input artifact (spec used to generate the ADR) that leaked into the output PR. The bot's current review dimensions (technical accuracy, correctness, security) don't include a 'scope hygiene' check for unexpected file paths. This type of issue is distinct from existing proposals about docs review quality (#1480) or ADR-specific review behavior (#1659) — those address the depth of feedback on legitimate files, not the detection of files that shouldn't be present at all.

Proposed change

Add a scope-hygiene check to the review agent (or as a review sub-agent dimension) that flags files from known internal/scaffolding paths. The check should: (1) maintain a configurable list of path patterns considered internal (e.g., docs/superpowers/specs/**, internal/scaffold/**, .fullsend/** when reviewing the source repo), (2) flag any files matching these patterns with a [medium] or higher finding when they appear in a PR diff, and (3) include the rationale that these paths typically contain tooling artifacts not intended for the final changeset. This could be implemented in the review orchestrator's pre-flight checks (before dispatching sub-agents) or as a lightweight sub-agent focused on diff scope validation.

Validation criteria

On the next 5 PRs to fullsend-ai/fullsend that include files from docs/superpowers/specs/ or similar internal paths, the review agent should produce a finding flagging those files as potentially unintended inclusions. Verify by checking review bot comments on PRs that touch those paths. A false-positive rate below 20% (i.e., at most 1 in 5 flagged files was actually intended) would indicate the check is well-calibrated.

---

Generated by retro agent from #1814

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 12:12 PM UTC · Completed 12:14 PM UTC
Commit: 3fa02b3 · View workflow run →

[fullsend-ai-triage]

Triage Summary

Category: Feature — scope-hygiene check for the review agent
Severity: Medium
Related issues: #1480 (docs review depth), #1659 (ADR review behavior), #1908 (repo-specific references in scaffolded skills) — all are distinct from this request.

Problem

The review agent approved PR #1814 without noticing an internal spec file (docs/superpowers/specs/...) that was accidentally included alongside its generated output (ADR 0046). The current review dimensions evaluate file content but do not check whether files belong in the changeset.

Recommended Approach

Add a scope-hygiene check — either as a pre-flight step in the review orchestrator or a lightweight sub-agent — that flags files matching configurable internal path patterns (e.g., docs/superpowers/specs/**, internal/scaffold/**, .fullsend/**). Matching files should receive a [medium] or higher finding explaining they are likely tooling artifacts not intended for the final changeset.

Proposed Test Case

1. PR diff includes `docs/superpowers/specs/some-spec.md`
   -> Expect: finding with severity >= medium flagging it as a potential tooling artifact

2. PR diff includes only `docs/adr/0046-some-decision.md`
   -> Expect: no scope-hygiene findings

3. Custom path patterns in config (e.g., `tmp/**`)
   -> Expect: files under tmp/ are flagged accordingly

4. Edge case: file matches internal pattern but is explicitly allowed
   -> Expect: no finding (if override mechanism exists)

The validation criteria in the issue (checking next 5 PRs touching internal paths, <20% false-positive rate) provide a solid acceptance test strategy.

---

Labels: Review agent feature request for a new scope-hygiene dimension, touching harness/skills configuration for path patterns.

[fullsend-ai-prioritize]

RICE Priority Score: 1.2

Score breakdown

Dimension	Score	Reasoning
Reach	1.5	The review agent runs on all PRs to fullsend-ai/fullsend, so any user opening a PR benefits from improved review coverage. However, the specific scenario — accidentally including internal scaffolding files — is an intermittent occurrence rather than a daily pain point, so this scores at 1.5 (multiple users across the org are affected over time).
Impact	1	When this gap manifests, a human reviewer must manually catch the leaked file (as rh-hemartin did with 10 comments on PR #1814). The workaround exists (human review) but adds friction and delays. This is a noticeable workflow improvement (score 1) — it prevents wasted reviewer effort and reduces the chance of internal artifacts shipping unnoticed.
Confidence	0.8	High confidence (0.8). The issue is exceptionally well-described: it includes the exact PR, the specific file path, a clear proposed solution with configurable path patterns, and concrete validation criteria (<20% false-positive rate across 5 PRs). The triage agent also confirmed it is distinct from related issues #1480, #1659, and #1908.
Effort	1	Medium effort (1). The implementation requires adding a scope-hygiene check — either as a pre-flight step in the review orchestrator or a lightweight sub-agent — with a configurable list of internal path patterns. This touches the harness/review orchestrator and configuration, but the logic itself (glob pattern matching against diff file paths) is straightforward. Testing against the proposed validation criteria adds some work.

Formula: (1.5 x 1 x 0.8) / 1 = 1.2