feat(#663): auto-recover MCP-orphaned sessions (watchdog force-fresh) - #664
Merged
Conversation
Primitive for detecting an identity-unbound resumed session (CC #60949/#9608:
CC won't re-init an MCP transport after a 404, so a daemon restart that tears
down the :8890 gateway leaves live agents' tools dead until a fresh relaunch).
- shared_mcp: per-gateway-generation epoch (bumps on every app build) + an
in-memory per-agent bind ledger written only from inside the tool handler
(a tool executing proves the agent's real MCP client traversed the current
gateway; request arrival does not, since a stale client can still 404).
Epoch bump clears the ledger so a pre-restart bind never counts as current.
- pinky-self: mcp_probe(nonce, launch_id) core tool — records a probe success
and returns {agent, nonce, launch_id, gateway_epoch, observed_at}.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- shared_mcp.record_mcp_success(agent): generic (nonce-less) bind record;
refreshes the current-epoch entry, preserving a prior probe's nonce.
- pinky-self send_heartbeat now records a bind success on every call — a
heartbeat tool executing proves the agent's MCP client reached the current
gateway, giving the watchdog a deterministic periodic liveness signal
(independent of prompt injection) for heartbeat-enabled agents.
- api: GET /agents/{name}/mcp-bind-status — read-only bind ledger view
(bound for current gateway generation? last nonce/launch_id/observed_at).
Does not touch the agent's MCP transport, so it works when that is dead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The recovery net for the CC #60949 outage class: a daemon restart tears down
the :8890 gateway, the resumed CC client never re-inits its MCP transport, so
all pinky tools 404 until a force-fresh relaunch. The watchdog now detects and
heals this — using the agent's own heartbeat as the bind probe.
session_watchdog:
- WatchdogConfig.mcp_recover flag (default OFF — soak on one agent first).
- _evaluate_mcp_bind: for a CONNECTED, heartbeat-enabled agent with no MCP
success recorded for the CURRENT gateway epoch sustained past a deadline
(max(3x heartbeat_interval, 240s)) → force-fresh recover. Safety rails:
heartbeat-disabled/quiet agents never trip; deadline measures a sustained
outage (so a rebuilt gateway gives every agent the full window to re-bind
via its next heartbeat — no deploy storm); per-agent post-recover grace +
a global min-interval backstop (120s) against fleet restart storms; status
errors are swallowed (never kill a healthy session on a diagnostic hiccup).
api: wire _watchdog_mcp_bind_status ({checkable, bound, heartbeat_interval})
and _watchdog_mcp_recover (force-fresh restart mirroring admin_force_restart_
agent — sets force_fresh_context_once so the relaunch drops --continue; audits
as mcp_epoch_unbound; informational owner alert on recovery).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| ) | ||
| try: | ||
| await ss.disconnect() | ||
| except Exception: |
| # so any context-staleness gate is satisfied; preserve saved state. | ||
| try: | ||
| agents.bump_context_updated_at(agent_name) | ||
| except Exception: |
Comment on lines
+40
to
+45
| from pinky_daemon.shared_mcp import ( | ||
| LazyAgentName, | ||
| record_mcp_success, | ||
| record_probe_success, | ||
| resolve_lazy, | ||
| ) |
- test_shared_mcp: epoch bump changes id + clears ledger; probe/heartbeat record + status; nonce preserved within epoch, fresh across epoch; blank agent no-op; unknown agent unbound. - test_session_watchdog: 11 cases — recover only after a sustained deadline; bound clears clock; heartbeat-disabled never trips; flag-off no-op; disconnected skip; post-recover grace; global rate-limit defers-then-allows; status-error swallowed; recover-failure doesn't burn the rate-limit; end-to-end routing through _evaluate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mcp_probe is a new core pinky-self tool, so core goes to 24 and (atop main's #145 register_agent at 69) all-gates goes to 70. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
olegbrok
force-pushed
the
fix/663-mcp-orphan-identity-verify
branch
from
June 2, 2026 21:18
506b698 to
284a46c
Compare
olegbrok
changed the title
olegbrok
added a commit
that referenced
this pull request
Jun 2, 2026
…te + R2 checkable) (#665) * feat(#663): activate MCP-recover for low-cadence agents (R1 enable-gate + R2 checkable) Two refinements that let `mcp_recover` be enabled on an agent whose progress/transition watchdog is OFF (barsik's shape) — the activation blockers found in the #664 post-merge review. R1 — watchdog enable-gate (session_watchdog._evaluate): `mcp_recover` is its own opt-in, independent of the master `cfg.enabled` switch. The MCP-bind branch now runs before the `enabled` gate (return early only when BOTH `enabled` and `mcp_recover` are off); the progress/transition watchdog stays gated on `enabled`. During a BOOTING/RECONNECTING transition the branch is a cheap no-op (connected is False), so the transition branch still handles that wedge. R2 — checkable signal (api._watchdog_mcp_bind_status): `checkable` was `heartbeat_interval > 0`, which wrongly excludes wake-driven sidekicks that carry interval==0 yet heartbeat in practice (barsik; Murzik Finding #2). Now also true when the agent emitted a recent AGENT-ORIGIN heartbeat (get_latest_agent_heartbeat — excludes synthetic server_presence rows, so a wedged agent can't look checkable-and-bound). Extracted as the pure, unit-tested helper `compute_mcp_checkable`. Still flag-gated (mcp_recover default OFF). Tests: R1 enable-gate routing + progress-gated-off, R2 checkable policy matrix. 52 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(#663): add transition-snap + mcp_recover guard test (Murzik nit) Proves the R1 _evaluate reorder doesn't steal lifecycle transitions: with enabled=True + mcp_recover=True and a RECONNECTING (connected=False) snap, the MCP-bind branch no-ops and the transition branch still owns the wedge. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Oleg <oleg@Olegs-Mac-mini.local> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Auto-recovery for the MCP-orphan / identity-unbound blackout (#663): when the daemon restarts it tears down the in-process
:8890MCP gateway, so every live agent'sMcp-Session-Idgoes stale. Per the upstream CC bug anthropics/claude-code#60949 + #9608 (both closed as not planned), Claude Code never re-initializes an HTTP/SSE MCP transport after a404 session-not-found— so the agent's pinky tools 404 for the rest of the session, and a plain--continuerestart re-resumes the same dead transport. Only a force-fresh relaunch re-binds MCP. CC won't self-heal, so the recovery is ours.3rd sibling of #612 (REPL dies) and #613 (poisoned transcript) — but touches none of the same code (
tmux_session.pyuntouched), so it's standalone, not stacked.Designed + reviewed with @murzik (full thread on #663; review on this PR).
How
Detection — the agent's own heartbeat is the probe.
send_heartbeatis an MCP tool call, so a heartbeat executing proves the agent's real MCP client reached the current gateway generation. No prompt-injection / control-turn machinery needed.shared_mcp.py): a per-gateway-generationepoch(bumps on every app build — daemon restart and gateway crash-restart; the bump clears the ledger so a pre-restart bind never counts) + a per-agent last-success ledger written only from inside a tool handler (a stale client can reach the gateway and still 404, so arrival ≠ success).mcp_probe(nonce, launch_id)core pinky-self tool — explicit bind proof (for the phase-2 launch-time fast-path); records into the ledger.send_heartbeatnow records a generic bind success on every call.GET /agents/{name}/mcp-bind-status— read-only ledger view; never touches the agent's MCP transport, so it works precisely when that transport is dead.session_watchdog.py, flag-gatedmcp_recover, default OFF): a CONNECTED + heartbeat-enabled agent with no current-epoch MCP success sustained past a deadline (max(3× heartbeat_interval, 240s)) → force-fresh restart (api._watchdog_mcp_recover, mirrorsadmin_force_restart_agent; auditedmcp_epoch_unbound; informational owner alert). This auto-restart bypasses the stale-save guard by callingss.connect()directly, so the dead-MCP deadlock that needed a manual restart is handled automatically.Review responses
tools_unreachablebypass.!boundis absence-of-proof, not proof-of-unreachable (a healthy session that hasn't emitted its first post-epoch heartbeat also reads!bound), and_get_streaming_restart_guardis shared by many paths — so it could wrongly relax a save guard. The deadlock is already covered by the watchdog auto-recovery (above) +admin_force_restart_agent(manual escape hatch). A sustained-signal guard bypass is deferred to phase-2.send_heartbeatformax(3× hb, 240s)after an epoch bump could be force-restarted. Acceptable for the barsik-only flag-gated soak; before fleet enablement, record heartbeat/proberequested_atand recover on missed requested probes (or include real prompt cadence/skip state in_watchdog_mcp_bind_status).Safety rails (Brad OOO; deploying under Trust-L2 rails)
Testing
tests/test_shared_mcp.py— ledger/epoch semantics.tests/test_session_watchdog.py— 11 cases incl. sustained-deadline, heartbeat-disabled never trips, post-recover grace, global rate-limit, status-error swallowed, failed-recover-doesn't-burn-limit, end-to-end routing.tests/test_pinky_self_tools.py— core 24 / all-gates 70 withmcp_probe.naclcollection errors are a pre-existing local-env gap; CI has PyNaCl).Deploy plan
Merge → cut release →
update_and_restart→ enablewatchdog_config.mcp_recoveron barsik only → soak (watch the bind ledger / no false recoveries) → fleet once clean.Follow-up (phase 2)
Addresses #663. Siblings: #612, #613. Context: #606, #623.
🤖 Opened by Barsik