Skip to content

feat(grug): a dead LLM backend pages an operator instead of failing a review - #821

Merged
quadseven merged 1 commit into
mainfrom
feat/818-dead-backend-pages-an-operator
Aug 3, 2026
Merged

feat(grug): a dead LLM backend pages an operator instead of failing a review#821
quadseven merged 1 commit into
mainfrom
feat/818-dead-backend-pages-an-operator

Conversation

@quadseven

Copy link
Copy Markdown
Owner

Why

While answering "why are we using OpenRouter", I checked whether it works. It has not, for a week.

From grug's own logs, 2026-07-27 to 08-03:

llm_backend_http_failed  openrouter  http_402   x4   (out of credits)
llm_backend_http_failed  poolside    http_404   x3   (dead endpoint)

OpenRouter and Poolside are Elder's overload valve - the deliberate 2026-07-14 call so a Cave outage does not leave a review all_failed. Both halves were down at the same time. The valve was not degraded, it was absent: any Cave blip went straight to "Grug could not review this", and the only person told was the PR author, who can neither pay an invoice nor rotate a key.

Nothing alerted, because the only signal was llm_backend_http_failed - which also fires for ordinary 429/5xx overload. Alerting on that token would page for the system working as designed, so it never was. The signal that mattered was hidden inside the signal that didn't.

Summary

Split the two classes that were sharing one token.

  • is_terminal_backend_failure - 401/403 (key wrong or revoked), 402 (unpaid), 404 (endpoint or model gone). None are fixed by retrying or falling back.
  • Terminal failures log llm_backend_unusable at ERROR with failure_class:config_or_billing. 429/5xx keep the existing token at WARNING.
  • A digest-tier monitor keys on the new token only. Digest rather than page is deliberate: the Cave still reviews, so this is not a user-facing outage - it is the safety net going missing, which you want to see before the next outage rather than during it.
  • Runbook section covering all four statuses, plus two things that are easy to get wrong: the backends are not interchangeable (OpenRouter serves Claude, Poolside serves Laguna, so losing one halves the fallback), and judge_findings / summarize_pr / answer_pr_question use them as their primary backend, so a dead one degrades those directly rather than only removing a fallback.

Acceptance criteria

  • 401/402/403/404 classified terminal - test_billing_and_auth_failures_are_classified_terminal
  • 429/5xx explicitly NOT terminal, so overload never pages - test_overload_and_transport_failures_are_not_terminal
  • Terminal failures get their own alertable token - test_terminal_backend_failure_emits_its_own_log_token
  • The monitor keys on the new token and never on the overload one - test_backend_unusable_query_targets_the_terminal_token_only
  • Both HTTP-failure call sites route through the new classifier
  • Suites: 1886 webhook, 288 api, 25 infra

Size

Size: M

Out of scope

This is the FIRST slice of #818, not the whole issue. It does not yet stop an author ever seeing "skipped" - it makes the most common cause visible to someone who can fix it. Still open there:

  • Retrying transient degradations to exhaustion before any author-facing surface says Grug did not review
  • Reducing the four separate "I could not review" surfaces to one voice
  • Measuring the author-visible skip rate before and after

Also out of scope: paying the OpenRouter bill or fixing the Poolside 404. This PR makes them visible; it cannot fix them from code.

Refs #818

"Why are we using OpenRouter" turned out to have an uncomfortable answer:
we are, and it has not worked for a week.

Measured 2026-07-27..08-03 from grug's own logs:

    llm_backend_http_failed  openrouter  http_402   x4   (out of credits)
    llm_backend_http_failed  poolside    http_404   x3   (dead endpoint)

OpenRouter and Poolside are Elder's overload valve - the deliberate
2026-07-14 call so a Cave outage does not leave a review `all_failed`.
BOTH halves were down at once. So the valve was not degraded, it was
absent: any Cave blip went straight to "Grug could not review this", and
the only person told was the PR author, who can neither pay an invoice
nor rotate a key.

Nothing alerted because the only signal was `llm_backend_http_failed`,
which also fires for ordinary 429/5xx overload. Alerting on that token
would page for the system working as designed, so it never was.

So: split the classes.

- `is_terminal_backend_failure` - 401/403 (key wrong or revoked), 402
  (unpaid), 404 (endpoint or model gone). None of these are fixed by
  retrying or by falling back.
- Terminal failures log `llm_backend_unusable` at ERROR with
  `failure_class:config_or_billing`. 429/5xx keep the existing token and
  stay a WARNING.
- A digest-tier monitor keys on the new token only. Digest, not page: the
  Cave still reviews, so this is not a user-facing outage - it is the
  safety net going missing, which you want to see before the next outage
  rather than during it.
- Runbook section, including that the two backends are NOT
  interchangeable (different models), and that judge/teller/ask use them
  as their PRIMARY backend, so a dead one degrades those directly rather
  than only removing a fallback.

This is the first slice of #818. It does not yet stop an author ever
seeing "skipped" - it makes the most common CAUSE of it visible to
someone who can fix it.

Suites: 1886 webhook, 288 api, 25 infra.

Refs #818
@quadseven
quadseven merged commit ebe1320 into main Aug 3, 2026
18 checks passed
@quadseven
quadseven deleted the feat/818-dead-backend-pages-an-operator branch August 3, 2026 04:26
@quadseven
quadseven deployed to k8s-preview August 3, 2026 04:26 — with GitHub Actions Active
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant