From aa87bfa3c2e0ffb8fc3be195c3162597ceed3dd9 Mon Sep 17 00:00:00 2001 From: fullsend-code <278716306+fullsend-ai-coder[bot]@users.noreply.github.com> Date: Thu, 18 Jun 2026 19:51:06 +0000 Subject: [PATCH 1/2] fix(#1835): require file reads before asserting contents in findings The review agent hallucinated file contents on PR konflux-ci/konflux-test#833, claiming a Dockerfile contained --nogpgcheck when it never did. The root cause was that the code-review skill and correctness sub-agent had no explicit requirement to read files outside the PR diff before asserting what they contain. Changes: - code-review SKILL.md step 2: added cross-file verification bullet requiring the agent to read any file it references in a finding, even if not in the diff. - code-review SKILL.md step 4: added cross-file finding self-check requiring verification that referenced files were read before finalizing findings. - correctness sub-agent: added cross-file verification section with the same read-before-assert requirement. Note: make lint could not run due to sandbox network restrictions preventing shellcheck installation. Closes #1835 --- .../fullsend-repo/skills/code-review/SKILL.md | 13 +++++++++++++ .../skills/pr-review/sub-agents/correctness.md | 12 ++++++++++++ 2 files changed, 25 insertions(+) diff --git a/internal/scaffold/fullsend-repo/skills/code-review/SKILL.md b/internal/scaffold/fullsend-repo/skills/code-review/SKILL.md index f67c35a17..5b4b47ac5 100644 --- a/internal/scaffold/fullsend-repo/skills/code-review/SKILL.md +++ b/internal/scaffold/fullsend-repo/skills/code-review/SKILL.md @@ -52,6 +52,12 @@ git log --oneline -10 -- - Read any security-sensitive files related to the change (auth middleware, RBAC configuration, sandboxing code) even if they are not directly modified. +- **Cross-file verification:** If you intend to reference a file's + contents in a finding — even a file not in the diff — you MUST read + that file first. Never claim a file contains specific text without + having read it in this session. If you cannot read the file (e.g., it + is in another repository or inaccessible), state that you were unable + to verify the contents rather than assuming what they contain. ### 3. Evaluate each dimension @@ -215,6 +221,13 @@ For each issue identified, record: observations, praise, broad suggestions, and anything already handled by the PR. +**Cross-file finding self-check:** Before recording any finding that +asserts what a specific file contains, verify that you read that file +during step 2. If you did not read it, read it now before finalizing +the finding. If the file is unreadable, reframe the finding to state +that the contents could not be verified — do not assert unverified +contents as fact. + #### Severity anchoring (re-reviews) When prior review context is available (passed from the `pr-review` diff --git a/internal/scaffold/fullsend-repo/skills/pr-review/sub-agents/correctness.md b/internal/scaffold/fullsend-repo/skills/pr-review/sub-agents/correctness.md index cb56b9e03..f8658f7d8 100644 --- a/internal/scaffold/fullsend-repo/skills/pr-review/sub-agents/correctness.md +++ b/internal/scaffold/fullsend-repo/skills/pr-review/sub-agents/correctness.md @@ -73,3 +73,15 @@ When reviewing technical documentation, verify: - **Edge case correctness** — Are described edge cases (depth/breadth limits, zero values, error conditions) handled correctly in the described logic? + +### Cross-file verification + +When a finding depends on the contents of a file not in the PR diff +(e.g., claiming a Dockerfile contains a specific flag, or a config file +uses a particular setting), you MUST read that file before asserting +what it contains. Do not reason about what a file "probably" contains +based on common patterns — read it. + +If the file cannot be read (e.g., it is in another repository or +inaccessible), state that you were unable to verify the contents. +Never present unverified file contents as fact in a finding. From f610c77d026d099180fa40b4d8b1bcd3408d9358 Mon Sep 17 00:00:00 2001 From: fullsend-fix <278716306+fullsend-ai-coder[bot]@users.noreply.github.com> Date: Mon, 22 Jun 2026 20:19:04 +0000 Subject: [PATCH 2/2] fix(#1835): add cross-file verification to security sub-agent Add the same cross-file verification section to the security sub-agent that was added to correctness, with domain-appropriate examples (workflow permissions, IAM policies). Addresses review feedback that the security sub-agent reads external files but lacked the read-before- assert checkpoint. Addresses review feedback on #2443 --- .../skills/pr-review/sub-agents/security.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/internal/scaffold/fullsend-repo/skills/pr-review/sub-agents/security.md b/internal/scaffold/fullsend-repo/skills/pr-review/sub-agents/security.md index 3380a91e3..9191126ec 100644 --- a/internal/scaffold/fullsend-repo/skills/pr-review/sub-agents/security.md +++ b/internal/scaffold/fullsend-repo/skills/pr-review/sub-agents/security.md @@ -83,6 +83,18 @@ Calibrate investigation to the diff size and security surface area. to verify permission scope. - Trace call sites of changed functions to check for fail-open paths. +### Cross-file verification + +When a finding depends on the contents of a file not in the PR diff +(e.g., claiming a workflow file contains a specific permission scope, or +an IAM policy grants a particular role), you MUST read that file before +asserting what it contains. Do not reason about what a file "probably" +contains based on common patterns — read it. + +If the file cannot be read (e.g., it is in another repository or +inaccessible), state that you were unable to verify the contents. +Never present unverified file contents as fact in a finding. + ## Fail-open / fail-closed evaluation **Category:** Use `fail-open` for all findings in this section.