feat(grug): a dead LLM backend pages an operator instead of failing a review - #821
Merged
Merged
Conversation
"Why are we using OpenRouter" turned out to have an uncomfortable answer:
we are, and it has not worked for a week.
Measured 2026-07-27..08-03 from grug's own logs:
llm_backend_http_failed openrouter http_402 x4 (out of credits)
llm_backend_http_failed poolside http_404 x3 (dead endpoint)
OpenRouter and Poolside are Elder's overload valve - the deliberate
2026-07-14 call so a Cave outage does not leave a review `all_failed`.
BOTH halves were down at once. So the valve was not degraded, it was
absent: any Cave blip went straight to "Grug could not review this", and
the only person told was the PR author, who can neither pay an invoice
nor rotate a key.
Nothing alerted because the only signal was `llm_backend_http_failed`,
which also fires for ordinary 429/5xx overload. Alerting on that token
would page for the system working as designed, so it never was.
So: split the classes.
- `is_terminal_backend_failure` - 401/403 (key wrong or revoked), 402
(unpaid), 404 (endpoint or model gone). None of these are fixed by
retrying or by falling back.
- Terminal failures log `llm_backend_unusable` at ERROR with
`failure_class:config_or_billing`. 429/5xx keep the existing token and
stay a WARNING.
- A digest-tier monitor keys on the new token only. Digest, not page: the
Cave still reviews, so this is not a user-facing outage - it is the
safety net going missing, which you want to see before the next outage
rather than during it.
- Runbook section, including that the two backends are NOT
interchangeable (different models), and that judge/teller/ask use them
as their PRIMARY backend, so a dead one degrades those directly rather
than only removing a fallback.
This is the first slice of #818. It does not yet stop an author ever
seeing "skipped" - it makes the most common CAUSE of it visible to
someone who can fix it.
Suites: 1886 webhook, 288 api, 25 infra.
Refs #818
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
While answering "why are we using OpenRouter", I checked whether it works. It has not, for a week.
From grug's own logs, 2026-07-27 to 08-03:
OpenRouter and Poolside are Elder's overload valve - the deliberate 2026-07-14 call so a Cave outage does not leave a review
all_failed. Both halves were down at the same time. The valve was not degraded, it was absent: any Cave blip went straight to "Grug could not review this", and the only person told was the PR author, who can neither pay an invoice nor rotate a key.Nothing alerted, because the only signal was
llm_backend_http_failed- which also fires for ordinary 429/5xx overload. Alerting on that token would page for the system working as designed, so it never was. The signal that mattered was hidden inside the signal that didn't.Summary
Split the two classes that were sharing one token.
is_terminal_backend_failure- 401/403 (key wrong or revoked), 402 (unpaid), 404 (endpoint or model gone). None are fixed by retrying or falling back.llm_backend_unusableat ERROR withfailure_class:config_or_billing. 429/5xx keep the existing token at WARNING.judge_findings/summarize_pr/answer_pr_questionuse them as their primary backend, so a dead one degrades those directly rather than only removing a fallback.Acceptance criteria
test_billing_and_auth_failures_are_classified_terminaltest_overload_and_transport_failures_are_not_terminaltest_terminal_backend_failure_emits_its_own_log_tokentest_backend_unusable_query_targets_the_terminal_token_onlySize
Size: M
Out of scope
This is the FIRST slice of #818, not the whole issue. It does not yet stop an author ever seeing "skipped" - it makes the most common cause visible to someone who can fix it. Still open there:
Also out of scope: paying the OpenRouter bill or fixing the Poolside 404. This PR makes them visible; it cannot fix them from code.
Refs #818