Release 26.05.068: auth-failure alerts + structured detection (SDK 0.1.77) - #406
Merged
Conversation
…ls break (#400) * feat(auth-alerts): notify operator + surface auth_status when Claude auth breaks When the Claude SDK returns error='authentication_failed' the streaming session has always suppressed the response (so raw API errors never reach end users). The downside: the operator who can re-auth has no signal until someone complains. This adds two channels for that signal. 1) AuthFailureTracker (auth_alerts.py): - Per-host sliding window of auth failures across all agents. - Fires an alert at threshold (3 fails in 5 min, OR 3 distinct agents failing at once = host-wide outage). - 30 min cooldown between alerts to prevent spam. - record_success() clears state on the next clean turn. 2) Streaming-session hooks (streaming_session.py): - New auth_alert_callback / auth_success_callback ctor params, called from the existing assistant-error and ResultMessage paths. - _is_auth_error() pattern-matches authentication_failed, invalid_api_key, unauthorized, auth_error, permission_error so future SDK renames keep alerting. 3) Operator delivery (api.py): - _on_auth_failure resolves operator chat (system_settings.operator_chat_id first; otherwise the chat_id appearing across the most approved_users rows) and DMs via the affected agent's own bot through _broker_send. - Codex sessions skip the hook — different auth lifecycle. 4) Health surface: - /admin/watchdog gains auth_status with status=ok|degraded|broken, window/threshold/cooldown config, outage age, alerts sent, and a per-agent failures list. External monitors can alert on broken. 20 new tests covering thresholds, cooldown, window eviction, host-wide detection, success-clears-state, operator resolution fallbacks, and a TestClient integration check on /admin/watchdog. Full suite: 1620 passed. Why now: hit a real outage on the Pi today — all 5 agents auth_failed for 12+ hours and nobody knew because the symptom was "agents stop responding." This makes the next one a 30-second telegram ping instead of a manual log grep. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(auth-alerts): preserve cooldown on broker delivery failure + plain-text alert PR #400 review (Pushok) caught the regression this PR was meant to prevent: record_failure() advanced _last_alert_at *before* awaiting _broker_send, so a network blip on the very first send-attempt silenced the alert system for 30 minutes — exactly the Sasha-down-9-12h shape we're designing against. Two-phase alerting: - record_failure() now decides only; never mutates cooldown. - New commit_alert() is invoked by _on_auth_failure ONLY after _broker_send succeeds. If delivery raises, cooldown is preserved so the next failure retries instead of being lost. Also fix the markdown parse_mode mismatch Pushok flagged: format_alert_message used `*agent*` and backticks, but _broker_send defaults to no parse_mode so operators would see literal `*sasha*`. Switched to plain text — safer than escaping for MarkdownV2 with arbitrary error strings. Tracker docstring: clarified concurrency contract (single event loop, no internal Lock needed) per Pushok's note that "not async-safe" overstated it. New tests: - test_failed_delivery_preserves_cooldown_and_retries_next_failure (regression) - test_commit_alert_advances_cooldown_and_counter - test_format_alert_uses_no_markdown_so_plain_text_renders_clean - two existing assertions extended to verify the decide/commit split Full suite: 1770 passed, 1 skipped. Ruff clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Oleg <oleg@Olegs-Mac-mini.local> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Picks up SDK 0.1.74-0.1.76 fixes on the floor pin: hook event streaming + deferred decisions, strict MCP config validation, eager session-store flushing, permission-suggestions deserialization fix, and bundled CC 2.1.132 (MCP reconnect-flood fix, subprocess-cleanup-on-parent-exit, CLAUDE_CODE_SESSION_ID env in subprocesses). Closes the floor-bump action item from #401. The native API-error-status investigation, subprocess-cleanup verification, and PR #400 auth-alert re-validation remain as separate follow-ups in #401. Verification: - pip install -e . resolves to claude-agent-sdk 0.1.76 (bundled CC 2.1.132) - pytest: 1770 passed, 1 skipped (parity with pre-bump) Co-authored-by: Oleg <oleg@Olegs-Mac-mini.local> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
…age.api_error_status (#403) (#404) * refactor(auth-detect): structured AssistantMessage.error + ResultMessage.api_error_status (#403) Replaces the legacy `_AUTH_ERROR_TOKENS` substring tuple in streaming_session.py with structured detection on both message paths, and bumps `claude-agent-sdk` floor 0.1.76 → 0.1.77. ## What changed **streaming_session.py** - Drop `_AUTH_ERROR_TOKENS` substring tuple and `_is_auth_error(str)`. - Add `_is_auth_error_assistant(msg)`: exact match against the `AssistantMessageError` Literal (`"authentication_failed"`). Other Literal values (billing_error, rate_limit, invalid_request, server_error, unknown) are NOT credential failures and must NOT trip the operator alert. - Add `_is_auth_error_result(result)`: detects `api_error_status` in {401, 403} on `ResultMessage` (added in SDK 0.1.76, emitted by CLI >= 2.1.110). Closes the gap where errors that only land at turn completion (not mid-turn on AssistantMessage) were silently swallowed by the existing error-result path. - Update both reader-loop callsites: AssistantMessage path now uses the assistant variant; ResultMessage `is_error` path now also fires the auth alert callback when status is 401/403. **api.py** - Update the `_on_auth_failure` docstring: the comment used to point at `auth_alerts._is_auth_error` (which never existed there); now references both the assistant and result detectors in `streaming_session`. **tests/test_auth_alerts.py** - Replace `_is_auth_error(str)` tests with structured tests for both detectors using `SimpleNamespace` stand-ins for the SDK message types. Coverage: - assistant: only `"authentication_failed"` is true; all 5 other Literal values are false; `None` and missing-attr are false. - result: 401/403 are true; 429 + 5xx are false; `None`/200/missing are false. **pyproject.toml** - `claude-agent-sdk` floor 0.1.76 → 0.1.77. Required for the `api_error_status` field on `ResultMessage` and gets the bundled Claude CLI 2.1.133 (subagent skill discovery fix, parallel-session refresh-token race fix, hook effort context). ## Why this matters Pre-refactor, the substring-tuple matcher carried entries (`invalid_api_key`, `unauthorized`, `auth_error`, `permission_error`) that the SDK Literal cannot produce — pure cargo from before the type became strict. Worse, the ResultMessage error path had no auth detection at all, so credential failures that landed only at turn completion were silently swallowed. After this refactor, both paths detect credential failures with exact-match semantics over typed values, the operator-alert wiring fires from either path, and we can no longer be silently broken by an AssistantMessageError variant rename — a mismatch becomes a test failure instead of a silent regression. ## Test plan - [x] `pytest tests/test_auth_alerts.py -x` — 27 passed - [x] Full suite: `pytest tests/ -x` — 1774 passed, 1 skipped - [x] `ruff check` on modified files — clean Refs: #401, #402, #403 🤖 Opened by Barsik * fix(auth-detect): per-turn dedupe + SDK Literal invariant guards Address Murzik's PR #404 review (P1-ish double-count) and Pushok's forward-looking concern about silent regression on SDK rename. ## Per-turn auth dedupe (Murzik) A single failed turn can surface auth errors on BOTH paths the reader_loop watches: - AssistantMessage with error="authentication_failed" (mid-turn) - ResultMessage with api_error_status=401/403 (turn completion) Without dedupe, AuthFailureTracker.record_failure() increments twice for one real failure — tripping the operator-alert threshold early (after ~2 actual failed turns instead of 3) and skewing the host-wide multi-agent baseline. Fix: a per-turn `auth_reported_this_turn` flag in the reader_loop. The AssistantMessage auth path sets it before invoking the callback (so a callback exception still dedupes). The ResultMessage auth path gates its callback on `not auth_reported_this_turn`. The flag resets at the end of each ResultMessage handling — turn boundary. ## SDK Literal invariant (Pushok) `_is_auth_error_assistant` does exact-match against `AssistantMessageError`. If a future SDK release renames the Literal value, exact-match silently stops detecting credential failures — re-creating the exact regression #400 was built to catch. Two-layer guard: - Import-time assertion in reader_loop checks `_AUTH_ASSISTANT_ERROR in AssistantMessageError.__args__`. Fails loud at session start on any local-dev SDK upgrade that drops the literal. - Unit test in test_auth_alerts.py asserts the same invariant. Fails loud at CI on any PR that bumps the SDK across a rename. ## Other changes - Enrich the ResultMessage auth-callback string with `msg.errors` when present — operators see actionable triage context in their DM, not just the raw status code (SDK 0.1.77 started returning meaningful error messages here). - Bump uv.lock claude-agent-sdk 0.1.73 → 0.1.77 to match pyproject.toml's floor (lock file was stale from PR #402). ## Tests - 4 new reader-loop regression tests in test_streaming_session.py: - both paths emit for one turn → callback fires once - result-only path still fires (dedupe doesn't break the gap-fill) - dedupe resets between turns (two real failures = two callbacks) - non-auth AssistantMessage error doesn't poison next-turn dedupe - 1 new SDK-Literal-invariant unit test in test_auth_alerts.py. - test_api.py fake_types stub gains AssistantMessageError so the import-time invariant check passes under that test's monkey-patch. Full suite: 1779 passed / 1 skipped, ruff clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Oleg <oleg@Olegs-Mac-mini.local> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| from dataclasses import dataclass, field | ||
| from typing import Iterable | ||
|
|
||
| _log = logging.getLogger("pinky.auth_alerts").info |
| if self._auth_success_callback: | ||
| try: | ||
| self._auth_success_callback(self.agent_name) | ||
| except Exception: |
olegbrok
added a commit
that referenced
this pull request
May 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First main-channel release of May 2026. Promotes the auth-alerts stack — the rest of beta's content (#356–#392) is already on main via the recent release branches.
This PR is 3 cherry-picks from beta, applied cleanly against current
main:c4a3319← feat(auth-alerts): operator alert + auth_status when Claude credentials break #400 —feat(auth-alerts): operator alert + auth_status when Claude credentials break8596b92← chore(deps): bump claude-agent-sdk floor 0.1.73 → 0.1.76 #402 —chore(deps): bump claude-agent-sdk floor 0.1.73 → 0.1.7695c89e7← refactor(auth-detect): structured AssistantMessage.error + ResultMessage.api_error_status (#403) #404 —refactor(auth-detect): structured AssistantMessage.error + ResultMessage.api_error_status🚨 What ships in this release
Sasha-style credential outages are no longer silent. The daemon now detects credential failures across both SDK paths and DMs the operator with cooldown + dedupe.
Auth failure detection (#400)
AuthFailureTrackerwith per-agent threshold + host-wide multi-agent threshold/admin/auth-statusendpoint exposing tracker stateStructured detection (#404, requires #402)
invalid_api_key,unauthorized,permission_error) the SDK Literal cannot produce — pure cargo from before the type became strict._is_auth_error_assistant(msg)— exact match against theAssistantMessageErrorLiteral"authentication_failed". Other Literal members (billing_error, rate_limit, invalid_request, server_error, unknown) are explicitly NOT auth and don't trip the alert._is_auth_error_result(result)—api_error_status ∈ {401, 403}onResultMessage(added in SDK 0.1.76, emitted by CLI ≥ 2.1.110). Closes a real gap where errors that only land at turn completion (not mid-turn on AssistantMessage) were silently swallowed.msg.errorsfor actionable triage context, not just the status code.SDK bump (#402, #404)
claude-agent-sdkfloor: 0.1.73 → 0.1.77ResultMessage.api_error_statusVerification
mainauto-merged cleanly across the post-Refactor api.py: extract Pydantic models + KB/Apps routes #362-refactor api.py surface — no manual conflict editspytest tests/test_auth_alerts.py tests/test_streaming_session.py tests/test_api.py -q→ 316 passedf6177f9since ~16:04 UTC today — green so farDeploy note (per Murzik's release-perspective check)
Because this release bumps
claude-agent-sdkto 0.1.77, fresh deploys off main must rebuild deps. The/admin/updateendpoint on system-python deployments needsforce_deps=true(escape hatch added in #392, already on main). Existing channel=beta deploys are unaffected — they already have 0.1.77.Tag plan
After merge: tag
26.05.068and cut a GH release with these notes.🤖 Opened by Barsik