Review agent intermittently completes without writing review-metrics.yaml

[nsheaps]

Summary

The AI review pipeline's "review" job (running the reviewer agent via claude-code-action) sometimes completes with is_error: false — the agent itself reports success — without ever writing the required review-metrics.yaml/.json file at REVIEW_METRICS_PATH. The downstream script agent-repo/.github/scripts/review-receiver/read-metrics-and-compute-conclusion.sh then fails with:

##[error]No metrics file at ... — review agent did not complete.

This cascades to a LouisBrunner/checks-action step reporting conclusion: failure, title: "The review agent failed to run" on the original check run in the consumer repo, plus a generic ::error::Receiver pipeline failed. line. The accompanying troubleshooting hint about REVIEW_GITHUB_APP_ID/REVIEW_GITHUB_APP_PRIVATE_KEY does not apply — auth is confirmed present in all three occurrences below.

Evidence

Occurrence 1 — nsheaps/agents#333 (check run 93983045453, job 93983057990, run 31554184270)

• Agent JSON result: is_error: false, num_turns: 49, total_cost_usd: 2.0171772499999996, permission_denials_count: 16
• No metrics file was ever written despite the agent reporting success.

Occurrence 2 — nsheaps/.ai-agent-jack#483 (check run 93984292623, HeadSHA f40ef82e86547f0b67b93c6d7b8820e26f7f4b3f, run 31554600489)

• Same symptom, on the SAME HeadSHA that an EARLIER run successfully completed a full review (Henry's COMMENT-verdict review is posted on this PR). This confirms the failure is intermittent, not something that blocks review outright — a re-run on identical input can succeed or fail.

Occurrence 3 — nsheaps/ai-mktpl#743 (check run 93983718763, HeadSHA 68e8e530a50fec9478c2d3ce699d38bb809a8c71, run 31554405041)

• Same symptom, different repo entirely.

Three independent occurrences across three different repos, with successful runs interspersed on at least one of the same commits — this points to a systemic/intermittent reliability bug in the shared review pipeline (claude-code-action and/or the metrics-handling script), not something caused by the content of any specific PR.

Candidate root cause (not fully confirmed)

The review prompt template embedded in the job log (step 11, "Emit review metrics") explicitly instructs the agent to use the Write tool at /home/runner/work/_temp/review-metrics.yaml, and explicitly warns:

[do NOT use] Bash(gh:*)/Bash(git:*) — allowlist and will be silently denied, leaving this step incomplete even though the rest of the review su[cceeded]...

This suggests that on failing runs, the agent may attempt to write/commit the metrics file via a gh/git-based path instead of the required Write tool, gets silently denied (a plausible source of some of the 16 permission_denials_count seen on occurrence 1), completes the rest of the review successfully, but never actually creates the gating file. This is grounded in log evidence but NOT confirmed via a full turn-by-turn transcript — the specific denied tool call hasn't been isolated.

Suggested investigation checklist

• Pull the full turn-by-turn transcript (not just the final JSON result) for at least one failing run to confirm which tool call was denied around step 11
• Check whether the prompt template's warning about gh/git being denied is itself a signal the template should be hardened (e.g. retry logic, or a repo-side fallback check for the metrics file via a non-agent step)
• Determine whether this correlates with agent turn count / cost (occurrence 1 had a relatively high num_turns: 49 — does the agent "forget" the metrics step on longer reviews?)
• Consider adding a fallback: if the agent's own JSON result reports is_error: false but no metrics file exists, treat it as a distinct, more specific failure mode (e.g. "review completed but metrics step skipped") rather than the generic "review agent did not complete" — this would make the failure mode easier to diagnose from the check-run title alone
• Consider whether claude-code-action's permission allowlist should explicitly grant Write to the metrics path (it should already, per the prompt template, but confirm the actual allowlist config) to rule out a race/config drift

Impact

Blocks the CI-green gate for PRs relying on request-review label re-triggering — three in-flight PRs (nsheaps/agents#333, nsheaps/.ai-agent-jack#483, nsheaps/ai-mktpl#743) hit this and required re-triggering the review.

[nsheaps]

🤖 New recurrence, same signature, after both PRs' latest pushes were re-reviewed:

• nsheaps/agents#333 — check run 93989363281 (run 31556319500), conclusion: failure, title: "The review agent failed to run". Henry's review content (id 4912391578, APPROVED) was posted successfully at 2026-08-12T02:19:09Z — same pattern as the original report: review succeeds, metrics file apparently missing, check fails closed.
• nsheaps/.ai-agent-jack#483 — check run 93989391368 (run 31556329515), same conclusion: failure / title: "The review agent failed to run". Henry's review content (id 4912387352, round 2, APPROVED) posted at 2026-08-12T02:18:02Z.

By contrast, nsheaps/ai-mktpl#743's re-review this round completed clean — no AI Code Review/claude-review failure in its check-run list, review APPROVED (id 4912403995). So the intermittency continues to track with this issue's existing hypothesis; no new root-cause info beyond what's already documented here. Re-applying request-review on both affected PRs to retry, per the existing playbook for this failure mode.

---

Generated by Claude Code

[nsheaps]

Second recurrence on the same two PRs after re-applying request-review to retry: the retry itself failed with the identical signature.

• nsheaps/agents#333 — check run 93989363281, started_at: 2026-08-12T02:15:55Z, completed_at: 2026-08-12T02:21:44Z, conclusion: failure, title: "The review agent failed to run"
• nsheaps/.ai-agent-jack#483 — check run 93989391368, started_at: 2026-08-12T02:16:06Z, completed_at: 2026-08-12T02:20:12Z, conclusion: failure, title: "The review agent failed to run"

Both are second failures in a row for the same PR/commit (first occurrences were ~01:5x UTC per the original report above). Re-applying request-review once more to retry a third time. Flagging in case the retry rate itself is useful signal for root-causing — this doesn't look purely random if it's now hit the same two PRs twice consecutively.

---

Generated by Claude Code

[nsheaps]

🤖 Third consecutive occurrence, same signature, both PRs — this time the review agent's actual review text landed successfully (Henry posted a real "round 3" APPROVE review on both PRs, re-affirming his round-2 findings), but the AI Code Review check run still resolved to failure/"The review agent failed to run" on both:

• nsheaps/agents#333: check run 93992534897, head 644de9268c6069e1cb497a309b5f7fcd1a788c06, started_at 02:36:32Z / completed_at 02:40:55Z, workflow run
• nsheaps/.ai-agent-jack#483: check run 93992539420, head 8b0391b48774736432f46fbbcbd259b7da86ef9b, started_at 02:36:33Z / completed_at 02:40:07Z, workflow run

This strengthens the diagnosis from the retry-#2 comment above: the review text and the review-metrics.yaml artifact write are two independent steps in the reviewer workflow, and the artifact write is failing intermittently even when the review itself completes and posts correctly. Triggering retry #4 on both PRs now via the request-review label re-apply loop.

---

Generated by Claude Code

[nsheaps]

Fourth consecutive occurrence — escalating severity

nsheaps/agents#333 retry #4's AI Code Review check just failed with the same signature:

• Check run: 93993308561, conclusion failure, title "The review agent failed to run"
• Workflow run: https://github.com/nsheaps/.ai-agent-henry/actions/runs/31557688751
• Head SHA: 644de9268c6069e1cb497a309b5f7fcd1a788c06 (unchanged since round 1)
• Henry's review text for this same round: a genuine, substantive third-re-verification APPROVE, posted successfully — see PR #333 review activity

This is the 4th consecutive occurrence of this bug on agents#333, and the pattern has now repeated in lockstep on nsheaps/.ai-agent-jack#483 for 3 of the same 4 rounds (jack#483's round-4 check was still in_progress as of this comment — will confirm separately once it resolves).

Why I'm escalating instead of retrying again

Per this repo's own workflow-self-improvement.md rule ("fix the system, not just retry the symptom indefinitely") and verify-before-blaming.md: four consecutive failures with an identical signature, on the same unchanged head SHA, where the review-posting step visibly succeeds each time (Henry's review text lands, is substantive, and is a real re-verification) but the review-metrics.yaml-artifact-write step fails every single time — this is no longer plausibly transient. Re-triggering via label remove/re-add is treating the symptom; the artifact-write step itself needs investigation (e.g. does it run in a separate job/step from the review-post step? Is there a race, a silent exception, a missing working-directory context on retry-via-relabel specifically vs. a fresh PR open?).

Suggested next step

Rather than a retry #5, it may be worth:

1. Pulling the full job logs for one of these failed runs (e.g. 31557688751) to see what the artifact-write step actually reports (exception, missing file, wrong path, etc.)
2. Checking whether this failure mode is specific to the relabel-retrigger path (as opposed to a fresh PR-open or synchronize event) — both agents#333 and .ai-agent-jack#483 have now failed 100% of their relabel-triggered rounds (4/4 and 3/3 respectively) despite Henry's review text succeeding every time, which is a strong correlation worth checking directly if any prior successful run in nsheaps/ai-mktpl (or elsewhere) was NOT relabel-triggered.

Not retrying again on my own initiative until this is root-caused — will hold both PRs at their current state pending investigation, per the "no workarounds without approval" / "fix the system" guidance.

---

Generated by Claude Code

[nsheaps]

Confirming: jack#483 round 4 also failed — 4/4 lockstep with agents#333

nsheaps/.ai-agent-jack#483's retry #4 has now resolved too, same signature:

• Check run: 93993340320, conclusion failure, title "The review agent failed to run"
• Workflow run: https://github.com/nsheaps/.ai-agent-henry/actions/runs/31557700533
• Head SHA: 8b0391b48774736432f46fbbcbd259b7da86ef9b (unchanged since round 1)

Both agents#333 and .ai-agent-jack#483 are now 4/4 on relabel-triggered rounds failing with the identical review-metrics.yaml-missing signature, despite Henry's review text posting successfully and being a genuine substantive approval each time. Strengthens the "relabel-retrigger path" correlation noted above — holding at current state on both PRs, not retrying again, pending root-cause investigation of the artifact-write step.

---

Generated by Claude Code

[nsheaps]

Update: merging origin/main into agents#333's branch just now pulled in the review-API fallback for a missing review-metrics.yaml (already on main via the metrics-gate rework referenced in this issue). agents#333 had fallen behind main and was conflicted on plugins/claude-code/review-utils/specs/review-dispatch.md; resolved the conflict by keeping both sides (our label-removal step + main's fallback/Stop-hook enforcement), pushed as cc29050, and re-applied request-review to trigger a fresh round on the now-current code.

Given the fallback is now live on agents#333's own branch, the next review round is a natural test of whether it actually prevents the 4/4 failure pattern reported above. Will report back once that round completes — if it still fails the same way, the fallback itself needs investigation; if it passes, that's likely the fix for .ai-agent-jack#483 too once it picks up the same main changes (jack#483 doesn't touch agents, so it's unaffected by this merge and will need its own retry).

---

Generated by Claude Code

[nsheaps]

Update: agents#333's round on cc29050 (the merge that pulled the review-API fallback onto this branch) came back clean — Henry approved, no review-metrics.yaml-missing failure this time. First non-failing round in 5 attempts (rounds 1–4 all failed that way; documented above). Consistent with the fallback being the actual fix, though one clean round isn't full confirmation on its own — will keep watching for recurrence.

lint also failed on this run but for an unrelated reason: a transient TLS handshake error against api.github.com during the checkout-as-app action's App-token step (x509: certificate is not valid for any names, but wanted to match api.github.com), before any lint tooling ran. Re-ran the failed job and it passed on retry — infra flake, not a code issue.

CI on agents#333 is now green except AI Code Review still finishing (expected to flip to success given the approval already posted).

---

Generated by Claude Code