Skip to content

Release 26.05.068: auth-alerts, structured auth detection, Discord poller, Codex runtime - #405

Closed
olegbrok wants to merge 16 commits into
mainfrom
beta
Closed

Release 26.05.068: auth-alerts, structured auth detection, Discord poller, Codex runtime#405
olegbrok wants to merge 16 commits into
mainfrom
beta

Conversation

@olegbrok

@olegbrok olegbrok commented May 8, 2026

Copy link
Copy Markdown
Collaborator

First main-channel release of May 2026. 15 non-merge commits since 26.04.067 (2026-04-14), spanning two themes: auth-alerts hardening (the headline) and the Codex runtime + Discord poller work that landed late April.

🚨 Auth-failure alerts (the headline)

Sasha-style credential outages are no longer silent. The daemon now detects credential failures across both SDK paths and DMs the operator with cooldown + dedupe.

🤖 Codex runtime + agent runtime column

💬 Discord per-agent bots

🔧 Admin, broker, ops

  • fix(admin): rebuild deps for system-python (rebase of #323 + integration tests) #392feat(admin): force=true on /admin/update to bypass dirty tree
    • Plus force_deps escape hatch for system-python deployments
    • Plus endpoint-level test coverage for the force × force_deps interaction
  • fix(broker): stop typing indicator immediately after outreach send — no more ghost typing after the agent sends and the next turn hasn't started yet

Versions in this release

  • claude-agent-sdk: 0.1.72 → 0.1.77
  • anthropic: 0.96 → 0.98.1

Verification

  • All component PRs reviewed individually (Murzik gate + Pushok contributions)
  • Beta has been running on the Mac mini at f6177f9 since ~16:04 UTC today
  • Full test suite: 1779 passed, 1 skipped
  • ruff check clean

Tag plan

After merge: tag 26.05.068 and cut a GH release with these notes.

🤖 Opened by Barsik

bradbrok and others added 16 commits May 2, 2026 12:54
Co-authored-by: Oleg <oleg@Olegs-Mac-mini.local>
Co-authored-by: Oleg <oleg@Olegs-Mac-mini.local>
Co-authored-by: Oleg <oleg@Olegs-Mac-mini.local>
Restore _resolve_agent_provider() for all runtimes so provider_ref key/model resolution works for Codex agents. Override only resolved_provider_url to codex_cli sentinel after resolution. Extend backfill tests for WHERE-clause defense-in-depth.
…rsistence (closes #361)

Adds a single-register-call test asserting that provider_url, provider_key,
provider_model, provider_ref, thinking_effort, and runtime are persisted through
both the returned Agent object and a subsequent get() DB round-trip.
Covers the INSERT path that silently dropped these fields before PR #358.

https://claude.ai/code/session_01EGnozrh9Fbrp6FMcjDZEnb
Picks up the upstream fixes summarized in #378:

claude-agent-sdk 0.1.73 (claude-code v2.1.122–v2.1.128):
- Sub-agent prompt cache fix (~3× cache_creation reduction)
- MCP reconnection no longer floods conversation with full tool list
- Parallel shell tool sibling-cancellation fix
- MCP image-dropping fix
- EnterWorktree branch-base fix
- OAuth 401 retry-loop fix
- Skill MCP-tools-on-fork fix

anthropic 0.98.1:
- Streaming stop_details propagation fix (#1725)
- Managed Agents API improvements
- Multipart file array field-name fix

Verified: 1700 passed, 1 skipped. No code changes required — pure dep bump.

Heads-up: claude-code v2.1.128 reserves "workspace" as an MCP server name. Relevant to #73 (Pinky Workspace API Server) — pick a different server name.

Co-authored-by: Oleg <oleg@Olegs-Mac-mini.local>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
…#383)

* feat(discord): REST-polling inbound poller for per-agent Discord bots

Adds a `BrokerDiscordPoller` analogous to `BrokerTelegramPoller` so agents
can receive Discord messages, not just send them. The Discord REST API has
no equivalent to Telegram's getUpdates long-poll, so this is a periodic
sweep over watched channels (default 1s). A future v0.2 may add a
Gateway/WebSocket variant for true push delivery.

Behavior:
- Auto-discovers text + announcement channels across the bot's guilds at
  startup; can be overridden per-token via settings.watched_channels.
- Primes last_message_id from the most recent message on first sight so
  history is not replayed.
- Filters out the bot's own messages (no self-reply loop).
- Honors 429 rate limits via DiscordRateLimitError.retry_after with a
  bounded retry, then bubbles up so the poll loop can sleep.
- Re-discovers channels every 60s so newly-joined guilds become reachable
  without a daemon restart.

Adapter changes:
- DiscordAdapter now sends a Discord-style User-Agent (default urllib /
  httpx UA gets Cloudflare-403'd on /users/@me).
- _request() catches 429 and retries up to max_429_retries (default 1)
  before raising DiscordRateLimitError. (DiscordRateLimited kept as alias.)
- get_my_guilds() and discover_text_channels() helpers for the poller.

Wiring:
- api.py PUT /agents/{name}/tokens/discord starts/restarts the poller
  the same way the telegram path does, including reading
  poll_interval_sec and watched_channels from token settings.
- DELETE /agents/{name}/tokens/discord stops the poller.
- on_startup auto-spawns Discord pollers for every enabled agent that
  has a saved discord token.

Tests: 11 new (35 total in test_discord*) — UA header, 429 retry/exhaustion,
guild listing, channel-type filtering, broker delivery, bot-self skip,
chronological ordering, empty-channel sentinel handling, discovery
failure tolerance, rate-limit propagation.

Live smoke-tested against bot user Barsik#2361 (id 1501305263351660674):
auth works, poller boots, discovers 0 channels (bot not yet invited to
any guilds), 2 poll sweeps complete cleanly, stop is graceful.

🤖 Opened by Barsik

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* review(discord): address Pushok's PR #383 feedback

- Q2 (BLOCKING for promotion): hoist get_channel out of the per-message
  loop and add a poller-lifetime cache invalidated on each
  _refresh_channels (60s default). Eliminates the burst pathology where
  a chatty channel would fan out to N round-trips per poll. New tests
  cover single-channel-call-per-burst, cache reuse across polls, and
  cache invalidation on rediscovery.
- Q1 (naming): rename `_refresh_channels(prime_last_ids=...)` to
  `_refresh_channels(verbose=...)` — the kwarg only gates logging, not
  priming, and the old name was misleading.
- Q3 (docstring): document that DiscordAdapter is synchronous and that
  async callers must use run_in_executor; mention 429-induced
  time.sleep behavior.
- Drop the DiscordRateLimited/DiscordRateLimitError alias — both were
  introduced in this same PR, no legacy callers, ossifying it now would
  be premature. Keeps DiscordRateLimited as canonical (poller + tests
  already use it) with a noqa N818 since the name reads naturally at
  call sites without the Error suffix.
- Add `add_done_callback` to the broker-delivery `create_task` so
  unhandled exceptions surface in poller logs instead of silently
  warning at the event loop layer. New test asserts the failure log
  line is emitted.
- `asyncio.get_event_loop()` → `asyncio.get_running_loop()` in
  coroutines (modern equivalent; surfaces non-running-loop bugs).

Tests: 39 in test_discord*.py (4 new), still pass; full suite green.

🤖 Reviewed by Pushok, addressed by Barsik

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Oleg <oleg@Olegs-Mac-mini.local>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Telegram clients clear the typing indicator when a bot message arrives,
but the broker's 4s sendChatAction loop kept running until the agent's
turn formally completed via route_response. The next loop iteration would
fire mid-await — making "typing..." reappear *after* the message had
already landed and lingering for the indicator's ~5s TTL.

Cancel the typing loop the moment any outreach send succeeds: text,
voice, photo, document, animation, gif. Reactions intentionally don't
trigger the cancel (the agent may still be drafting a real reply).

Also silence the no-op log line in _stop_typing so defensive calls after
every send don't spam the logs.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
When the working tree has local mods to tracked files (e.g. stale
frontend-dist build artifacts from a prior build), git pull aborts.
Recovery previously required out-of-band `git checkout -- <file>` access.

force=True now runs `git checkout -- .` before the pull to discard
local mods to TRACKED files only. Untracked files (.env, local notes)
are preserved — no `git clean`. dry_run is unaffected.

Plumbed through pinky_self update_and_restart MCP tool with a force
kwarg; response surfaces forced_reset + forced_files for visibility.

Closes pinkybot task #77.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…r-notify

Two follow-ups from Brad's first Discord live-verify (PR #386 deploy):

1. Fresh first-test message swallowed by replay-prevention floor.
   When a bot joined a brand-new server and a user sent a test message
   right before the next discovery sweep, that message became the floor
   and was silently skipped — bad first-contact UX.
   Fix in BrokerDiscordPoller._refresh_channels: if the most-recent
   message at prime time is <30s old AND from a non-bot, back the floor
   off by 1 snowflake (Discord IDs are monotonic) so the next poll picks
   it up via `?after=<id-1>`. Bot messages and stale messages keep the
   replay-prevention behavior. Always log the priming with timestamp +
   age so users see exactly what happened.

2. Broker tried to DM owner about new Discord users by sending to the
   user_id as a channel_id → "Discord API error 404: Unknown Channel".
   Discord requires opening a DM channel via POST /users/@me/channels
   first. Fix in DiscordAdapter: new open_dm_channel(user_id) helper,
   plus send_message catches 404 Unknown Channel and transparently
   resolves user_id → DM channel + retries. Resolution is cached per
   recipient so subsequent sends skip the lookup.

7 new tests (4 in test_discord.py, 3 in test_discord_poller.py).
Full suite: 1740 pass, 0 fail. Ruff clean.

Closes pinkybot task #75.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…scape hatch

The admin_update endpoint only ran `pip install` if `.venv/bin/pip` existed,
which silently skipped dependency rebuilds on system-python deployments
(Homebrew/Debian). Result: pyproject pin bumps merged to main but the live
daemon kept running stale package versions.

Fix:
- Fall back to `sys.executable -m pip install --break-system-packages` when
  no in-tree venv is present (PEP 668 externally-managed envs).
- Trigger rebuild when pyproject.toml or uv.lock changed in the pulled
  diff, OR when the new `force_deps=true` query param is set.
- Surface `deps_error` in the response so silent failures stop being silent.
- Wire `force_deps` through the `update_and_restart` MCP tool.

Tests: added 3 cases covering force_deps URL wiring, branch+force_deps
combo, and deps_error surfacing. All 172 pinky_self tool tests pass.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Adds 4 tests in TestAdminUpdateForceDepsIntegration verifying:
- force_deps=True triggers pip install on a clean (no-op) pull
- force=True + force_deps=True combine correctly (reset + reinstall)
- default behavior skips pip when pyproject.toml is unchanged
- pip install failures surface as deps_error in the response

Mirrors the matching MCP-tool-level coverage in test_pinky_self_tools.py.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
fix(admin): rebuild deps for system-python (rebase of #323 + integration tests)
…ls break (#400)

* feat(auth-alerts): notify operator + surface auth_status when Claude auth breaks

When the Claude SDK returns error='authentication_failed' the streaming
session has always suppressed the response (so raw API errors never reach
end users). The downside: the operator who can re-auth has no signal until
someone complains. This adds two channels for that signal.

1) AuthFailureTracker (auth_alerts.py):
   - Per-host sliding window of auth failures across all agents.
   - Fires an alert at threshold (3 fails in 5 min, OR 3 distinct agents
     failing at once = host-wide outage).
   - 30 min cooldown between alerts to prevent spam.
   - record_success() clears state on the next clean turn.

2) Streaming-session hooks (streaming_session.py):
   - New auth_alert_callback / auth_success_callback ctor params, called
     from the existing assistant-error and ResultMessage paths.
   - _is_auth_error() pattern-matches authentication_failed, invalid_api_key,
     unauthorized, auth_error, permission_error so future SDK renames keep
     alerting.

3) Operator delivery (api.py):
   - _on_auth_failure resolves operator chat (system_settings.operator_chat_id
     first; otherwise the chat_id appearing across the most approved_users
     rows) and DMs via the affected agent's own bot through _broker_send.
   - Codex sessions skip the hook — different auth lifecycle.

4) Health surface:
   - /admin/watchdog gains auth_status with status=ok|degraded|broken,
     window/threshold/cooldown config, outage age, alerts sent, and a
     per-agent failures list. External monitors can alert on broken.

20 new tests covering thresholds, cooldown, window eviction, host-wide
detection, success-clears-state, operator resolution fallbacks, and a
TestClient integration check on /admin/watchdog. Full suite: 1620 passed.

Why now: hit a real outage on the Pi today — all 5 agents auth_failed for
12+ hours and nobody knew because the symptom was "agents stop responding."
This makes the next one a 30-second telegram ping instead of a manual log
grep.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(auth-alerts): preserve cooldown on broker delivery failure + plain-text alert

PR #400 review (Pushok) caught the regression this PR was meant to prevent:
record_failure() advanced _last_alert_at *before* awaiting _broker_send, so
a network blip on the very first send-attempt silenced the alert system for
30 minutes — exactly the Sasha-down-9-12h shape we're designing against.

Two-phase alerting:
- record_failure() now decides only; never mutates cooldown.
- New commit_alert() is invoked by _on_auth_failure ONLY after _broker_send
  succeeds. If delivery raises, cooldown is preserved so the next failure
  retries instead of being lost.

Also fix the markdown parse_mode mismatch Pushok flagged: format_alert_message
used `*agent*` and backticks, but _broker_send defaults to no parse_mode so
operators would see literal `*sasha*`. Switched to plain text — safer than
escaping for MarkdownV2 with arbitrary error strings.

Tracker docstring: clarified concurrency contract (single event loop, no
internal Lock needed) per Pushok's note that "not async-safe" overstated it.

New tests:
- test_failed_delivery_preserves_cooldown_and_retries_next_failure (regression)
- test_commit_alert_advances_cooldown_and_counter
- test_format_alert_uses_no_markdown_so_plain_text_renders_clean
- two existing assertions extended to verify the decide/commit split

Full suite: 1770 passed, 1 skipped. Ruff clean.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Oleg <oleg@Olegs-Mac-mini.local>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Picks up SDK 0.1.74-0.1.76 fixes on the floor pin: hook event streaming
+ deferred decisions, strict MCP config validation, eager session-store
flushing, permission-suggestions deserialization fix, and bundled CC
2.1.132 (MCP reconnect-flood fix, subprocess-cleanup-on-parent-exit,
CLAUDE_CODE_SESSION_ID env in subprocesses).

Closes the floor-bump action item from #401. The native API-error-status
investigation, subprocess-cleanup verification, and PR #400 auth-alert
re-validation remain as separate follow-ups in #401.

Verification:
- pip install -e . resolves to claude-agent-sdk 0.1.76 (bundled CC 2.1.132)
- pytest: 1770 passed, 1 skipped (parity with pre-bump)

Co-authored-by: Oleg <oleg@Olegs-Mac-mini.local>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
…age.api_error_status (#403) (#404)

* refactor(auth-detect): structured AssistantMessage.error + ResultMessage.api_error_status (#403)

Replaces the legacy `_AUTH_ERROR_TOKENS` substring tuple in
streaming_session.py with structured detection on both message paths,
and bumps `claude-agent-sdk` floor 0.1.76 → 0.1.77.

## What changed

**streaming_session.py**
- Drop `_AUTH_ERROR_TOKENS` substring tuple and `_is_auth_error(str)`.
- Add `_is_auth_error_assistant(msg)`: exact match against the
  `AssistantMessageError` Literal (`"authentication_failed"`). Other
  Literal values (billing_error, rate_limit, invalid_request,
  server_error, unknown) are NOT credential failures and must NOT trip
  the operator alert.
- Add `_is_auth_error_result(result)`: detects `api_error_status` in
  {401, 403} on `ResultMessage` (added in SDK 0.1.76, emitted by CLI
  >= 2.1.110). Closes the gap where errors that only land at turn
  completion (not mid-turn on AssistantMessage) were silently
  swallowed by the existing error-result path.
- Update both reader-loop callsites: AssistantMessage path now uses
  the assistant variant; ResultMessage `is_error` path now also fires
  the auth alert callback when status is 401/403.

**api.py**
- Update the `_on_auth_failure` docstring: the comment used to point
  at `auth_alerts._is_auth_error` (which never existed there); now
  references both the assistant and result detectors in
  `streaming_session`.

**tests/test_auth_alerts.py**
- Replace `_is_auth_error(str)` tests with structured tests for both
  detectors using `SimpleNamespace` stand-ins for the SDK message
  types. Coverage:
  - assistant: only `"authentication_failed"` is true; all 5 other
    Literal values are false; `None` and missing-attr are false.
  - result: 401/403 are true; 429 + 5xx are false; `None`/200/missing
    are false.

**pyproject.toml**
- `claude-agent-sdk` floor 0.1.76 → 0.1.77. Required for the
  `api_error_status` field on `ResultMessage` and gets the bundled
  Claude CLI 2.1.133 (subagent skill discovery fix, parallel-session
  refresh-token race fix, hook effort context).

## Why this matters

Pre-refactor, the substring-tuple matcher carried entries
(`invalid_api_key`, `unauthorized`, `auth_error`, `permission_error`)
that the SDK Literal cannot produce — pure cargo from before the
type became strict. Worse, the ResultMessage error path had no auth
detection at all, so credential failures that landed only at turn
completion were silently swallowed.

After this refactor, both paths detect credential failures with
exact-match semantics over typed values, the operator-alert wiring
fires from either path, and we can no longer be silently broken by
an AssistantMessageError variant rename — a mismatch becomes a
test failure instead of a silent regression.

## Test plan

- [x] `pytest tests/test_auth_alerts.py -x` — 27 passed
- [x] Full suite: `pytest tests/ -x` — 1774 passed, 1 skipped
- [x] `ruff check` on modified files — clean

Refs: #401, #402, #403
🤖 Opened by Barsik

* fix(auth-detect): per-turn dedupe + SDK Literal invariant guards

Address Murzik's PR #404 review (P1-ish double-count) and Pushok's
forward-looking concern about silent regression on SDK rename.

## Per-turn auth dedupe (Murzik)

A single failed turn can surface auth errors on BOTH paths the
reader_loop watches:
- AssistantMessage with error="authentication_failed" (mid-turn)
- ResultMessage with api_error_status=401/403 (turn completion)

Without dedupe, AuthFailureTracker.record_failure() increments twice
for one real failure — tripping the operator-alert threshold early
(after ~2 actual failed turns instead of 3) and skewing the host-wide
multi-agent baseline.

Fix: a per-turn `auth_reported_this_turn` flag in the reader_loop.
The AssistantMessage auth path sets it before invoking the callback
(so a callback exception still dedupes). The ResultMessage auth path
gates its callback on `not auth_reported_this_turn`. The flag resets
at the end of each ResultMessage handling — turn boundary.

## SDK Literal invariant (Pushok)

`_is_auth_error_assistant` does exact-match against
`AssistantMessageError`. If a future SDK release renames the Literal
value, exact-match silently stops detecting credential failures —
re-creating the exact regression #400 was built to catch.

Two-layer guard:
- Import-time assertion in reader_loop checks
  `_AUTH_ASSISTANT_ERROR in AssistantMessageError.__args__`. Fails
  loud at session start on any local-dev SDK upgrade that drops the
  literal.
- Unit test in test_auth_alerts.py asserts the same invariant. Fails
  loud at CI on any PR that bumps the SDK across a rename.

## Other changes

- Enrich the ResultMessage auth-callback string with `msg.errors`
  when present — operators see actionable triage context in their
  DM, not just the raw status code (SDK 0.1.77 started returning
  meaningful error messages here).
- Bump uv.lock claude-agent-sdk 0.1.73 → 0.1.77 to match
  pyproject.toml's floor (lock file was stale from PR #402).

## Tests

- 4 new reader-loop regression tests in test_streaming_session.py:
  - both paths emit for one turn → callback fires once
  - result-only path still fires (dedupe doesn't break the gap-fill)
  - dedupe resets between turns (two real failures = two callbacks)
  - non-auth AssistantMessage error doesn't poison next-turn dedupe
- 1 new SDK-Literal-invariant unit test in test_auth_alerts.py.
- test_api.py fake_types stub gains AssistantMessageError so the
  import-time invariant check passes under that test's monkey-patch.

Full suite: 1779 passed / 1 skipped, ruff clean.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Oleg <oleg@Olegs-Mac-mini.local>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Comment thread tests/test_api.py
def test_wake_uses_codex_session_for_codex_runtime(self):
async def fake_connect(self):
self._connected = True
self.session_id = self.session_id or f"{self.agent_name}-codex"
from dataclasses import dataclass, field
from typing import Iterable

_log = logging.getLogger("pinky.auth_alerts").info
if self._auth_success_callback:
try:
self._auth_success_callback(self.agent_name)
except Exception:
if header:
try:
retry_after = float(header)
except ValueError:
Comment thread tests/test_broker.py
# Yield until cancellation has propagated to the task.
try:
await task
except asyncio.CancelledError:
Comment thread tests/test_broker.py
task_b.cancel()
try:
await task_b
except asyncio.CancelledError:
@olegbrok

olegbrok commented May 8, 2026

Copy link
Copy Markdown
Collaborator Author

Closing — main has diverged from beta (CodeQL hotfixes #395/#396 landed direct-to-main). Cutting a focused release branch from main with just the auth-alerts work (#400, #402, #404) per the recent #390/#393/#394 pattern.

@olegbrok olegbrok closed this May 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants