Skip to content

docs/enh: reviewer hallucination of cross-domain references — recurring on glm-5.3-flash (3x in one session) #89

Description

@rz1989s

Across today's 7-dispatch SDD run, reviewer subagents fabricated plausible-sounding internal context three separate times: a nonexistent "Brief 05 / Hallmark project", a "roll-up #22", and a "journal-layer seq filter" from a sibling repo. Each time the verdicts were sound and the findings verifiable against the diff — but the fabricated references could easily send a controller chasing phantom artifacts.

Pattern: hallucinated content always references process artifacts (briefs, roll-ups, ledger entries, other repos' fixes) — never invents code findings. Likely the model pattern-matching "reviewer reports reference their inputs" without having those inputs.

Suggestions:

  1. Docs (SDD recipe / reviewer template guidance): add a known-issues note — controllers should treat reviewer references to artifacts outside the provided brief/report/diff as suspect and verify findings independently (we now do this, but it cost a whole session to learn).
  2. Optional enhancement: reviewer dispatches could end with a structured references block (which input files the review actually relied on) — makes out-of-input references detectable mechanically.

Happy to share the three verbatim hallucination samples if useful for the docs.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions