retry: 🐛 Classify transient errors from the CLI error line, not echoed prompt - #4
Merged
Merged
Conversation
…d prompt The retry policy matched a bare 5xx, overloaded, or rate limit anywhere in the error message, and codex echoes the prompt into stderr, so a prompt mentioning "512 MiB" turned a permanent 400 into ~137s of backoff per call. is_transient now reads only the last non-empty line (codex ERROR:, claude API Error:) with a 5xx in a status position, plus the HTTP backend's exited 5xx: header. A 4xx is never transient. Claude-Session: https://claude.ai/code/session_015yr73szXYEmbjtNVLxnetK Claude-Session-Id: 6da31262-6ad0-4073-8a1f-91de79650dd7
This was referenced Sep 14, 2026
yasyf
added a commit
that referenced
this pull request
Sep 14, 2026
…by version (#5) SidecarError's doc comment claimed a host marks a response transient by pattern-matching the composed message and a bare three-digit number leaking through would read as a retryable 5xx. That stopped being true in v0.13.1 (#4): is_transient now reads only the CLI's last non-empty error line, and counts a 5xx only after a `status` or `API Error` marker, never a bare number, so it now names the keywords (`overloaded`, `rate limit`, `status`/`API Error` 5xx) that interpolated text could actually trip. CHANGELOG.md's Unreleased section carried the 0.12.0, 0.13.0, and 0.13.1 entries with no version headings, because the commits landing the 0.13.0 and 0.13.1 changes never renamed the section the way 4faadf0 did for 0.12.0. Split them under dated `## [0.12.0]`, `## [0.13.0]`, and `## [0.13.1]` headings (attributed by cross-referencing each tag's CHANGELOG.md and commit dates) and added the matching compare links, leaving Unreleased empty. Claude-Session: https://claude.ai/code/session_015yr73szXYEmbjtNVLxnetK
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The core retry policy classified a failed run as transient with
(?i)\b529\b|overloaded|rate.?limit|\b5\d\d\bover the whole error message. For a CLI backend that message is<provider> exited <rc>: <stderr tail>, andcodexechoes the prompt into stderr ahead of its error line. A captain-hook prompt mentioning "512 MiB" turned this permanent failure into a transient one:spawnllm then slept 5+15+45+60s before giving up, roughly 137s per call and about 7 minutes per captain-hook nudge.
Fix
is_transientinrust/spawnllm-core/src/retry.rsnow reads two structured places and nothing else:codexprintsERROR: ...andclaudeprintsAPI Error: .... A 5xx counts there only in a status position ("status":503,last status: 502 Bad Gateway,unexpected status 503,API Error: 500);overloadedandrate limitkeep matching on that line.^<provider> exited 5xx:header, which only the HTTP backend produces, since its return code is the HTTP status.A 4xx is never transient. The status forms come from the strings in the installed
codex0.154.0 andclaude2.1.270 binaries and the real 400 above from captain-hook's logs.Tests
New golden vectors, regenerated with
conformance-gen --writeand replayed by the Python, Go, and Rust hosts:retry_decision/non-transient-codex-400-echoed-promptandresolve/codex-exit-400-echoed-prompt: a 400 whose echoed prompt says "512 MiB" and "503" is not retried. Generated against the old core, both recordedtrue.retry_decision/transient-codex-retry-limit-502:ERROR: exceeded retry limit, last status: 502 Bad Gatewayis retried.retry_decision/non-transient-rate-limit-in-echoed-prompt: "rate limit" in the echoed prompt, over an unrelated error line, is not retried.transient-503-attempt-2andtransient-500-attempt-3-caps-at-60switch to the realclaudeandcodexerror forms; their expected decisions and backoffs are unchanged. The other retry and resolve vectors regenerate byte-identical, rate-limit andoverloadedcases included.https://claude.ai/code/session_015yr73szXYEmbjtNVLxnetK