Across today's 7-dispatch SDD run, reviewer subagents fabricated plausible-sounding internal context three separate times: a nonexistent "Brief 05 / Hallmark project", a "roll-up #22", and a "journal-layer seq filter" from a sibling repo. Each time the verdicts were sound and the findings verifiable against the diff — but the fabricated references could easily send a controller chasing phantom artifacts.
Pattern: hallucinated content always references process artifacts (briefs, roll-ups, ledger entries, other repos' fixes) — never invents code findings. Likely the model pattern-matching "reviewer reports reference their inputs" without having those inputs.
Suggestions:
- Docs (SDD recipe / reviewer template guidance): add a known-issues note — controllers should treat reviewer references to artifacts outside the provided brief/report/diff as suspect and verify findings independently (we now do this, but it cost a whole session to learn).
- Optional enhancement: reviewer dispatches could end with a structured
references block (which input files the review actually relied on) — makes out-of-input references detectable mechanically.
Happy to share the three verbatim hallucination samples if useful for the docs.
Across today's 7-dispatch SDD run, reviewer subagents fabricated plausible-sounding internal context three separate times: a nonexistent "Brief 05 / Hallmark project", a "roll-up #22", and a "journal-layer seq filter" from a sibling repo. Each time the verdicts were sound and the findings verifiable against the diff — but the fabricated references could easily send a controller chasing phantom artifacts.
Pattern: hallucinated content always references process artifacts (briefs, roll-ups, ledger entries, other repos' fixes) — never invents code findings. Likely the model pattern-matching "reviewer reports reference their inputs" without having those inputs.
Suggestions:
referencesblock (which input files the review actually relied on) — makes out-of-input references detectable mechanically.Happy to share the three verbatim hallucination samples if useful for the docs.