Skip to content

afk-pipeline: Claude session limit kills runs at arbitrary phases — detect, surface honestly, and degrade write-pr instead of failing #7

Description

@JorgeMenaDev

Note

Triage status 2026-07-12 (verified vs origin/main, afk-pipeline v2.13.1)
Fixed since filing: the hidden-crash mode (no failure_reason.txt) is closed — the v2.13.0 attempt loop writes a typed reason on every phase failure (failTyped/hoistFailure/writeFailure in attempt-loop.ts) and the blocked comment renders it plus a salvage note; salvage push (v2.4.0) means a write-pr-stage death no longer loses the branch.
Remaining scope of this issue: (1) Claude session-limit string detection + honest "resets at {time}" surfacing (only the codex quota runbook exists today); (2) write-pr degrade — write-pr.ts has no stub-PR fallback and "Open draft PR" only runs on success(), so a session-limit death at write-pr still fails the run; (3) pre-flight 1-token probe.

Two andyChat runs died on You've hit your session limit · resets 5:50pm (UTC) within 2 minutes (2026-07-07):

  • 28868630706 (slice 5, #287): implement → codex review → disposition → convex gate → verify → push all succeeded; the run FAILED at write-pr — a cosmetic prose step — after 100% of the substantive work was on the pushed branch. Matias recovered by opening the draft PR manually (andyChat#293).
  • 28873581302 (slice 4 retry, #286): died 40s into implement; the AgentError fiber crash meant no failure_reason.txt — the issue comment said '(no reason file written — check workflow logs)', hiding a failure mode we've now seen ≥3 times (also bcr seo-growth-workspace 4.0.2: domain-scope unmigrated_legacy_site in plan approval #99's first two runs, v2.3.1 note).

Proposed fixes (fold into the parallel v2.5.2 lane-exec work, or ship separately):

  1. Honest reason: phase wrappers (runtime.ts / run-with-retry.ts) catch AgentError output matching session limit and write failure_reason.txt = 'Claude session limit — resets {time}. Re-add the label after that time.' The failure comment then tells the operator exactly when to retry instead of pointing at raw logs.
  2. write-pr degrades, never fails the run: by the time write-pr runs, the branch is pushed and evidence committed. On claude-code failure, fall back to a stub PR (title = issue title, body = issue link + evidence pointer + 'PR text authoring failed: {reason}') so the pipeline still ends in a draft PR. A worse-prose PR beats a failed run every time.
  3. (Optional) Pre-flight probe: a 1-token claude-code ping before the implement phase; on session-limit, fail fast with the honest reason BEFORE burning setup — or sleep-until-reset if within N minutes.

Severity: this failure class costs a full run (or its tail) and mislabels itself as an opaque crash. Frequency: 3+ occurrences in 3 days.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions