fix(mcp): bound tools/list drains and recover from wedged transports

[luijoc]

What does this PR do?

An MCP server whose transport hangs (rather than errors) could wedge every
tool call on that server permanently. We hit this in production: an SSE server
behind a gateway answered initialize on a keepalive-triggered reconnect, then
never delivered a response to tools/list. Tool calls on that server were dead
for 40 hours until the process was restarted.

The mechanism: _discover_tools() awaits tools/list while holding the
server's _rpc_lock, and it runs after self.session = session but before
_ready.set() and _wait_for_lifecycle_event(). So the keepalive watchdog
hadn't started, the handler's dead-session recovery path never ran (session
was non-None), the TimeoutError seen by callers matched no
_SESSION_EXPIRED_MARKERS, and the circuit breaker's half-open probe simply
re-blocked on the same lock. Nothing in the process could break the cycle,
because nothing bounded the original await: Hermes never passes
read_timeout_seconds, so the SDK's send_request waits on
anyio.fail_after(None).

This PR bounds the awaits and gives the "transport hangs" case a recovery path.
No behavior change on a healthy server; no new config keys.

Related Issue

No existing issue; the incident analysis is inline above. Happy to file one if maintainers prefer.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

• tools/mcp_tool.py — pass read_timeout_seconds (the per-server timeout,
  default 300s) when constructing ClientSession on all three transports
  (stdio, SSE, streamable HTTP). Without it any request whose response never
  arrives parks forever at the protocol layer. This is the structural floor:
  it bounds the whole request class, not just today's call sites.
• tools/mcp_tool.py — bound the tools/list drain in _discover_tools()
  with connect_timeout rather than letting it inherit the per-tool timeout.
  Session setup shouldn't spend a deliberately generous tool budget, and a
  per-request floor still permits up to _MCP_LIST_MAX_PAGES (50) sequential
  pages under one lock. On timeout the exception propagates into run()'s
  existing except Exception, which tears the transport down and retries with
  backoff.
• tools/mcp_tool.py — same bound on _refresh_tools() (driven by
  notifications/tools/list_changed via a fire-and-forget task). On timeout it
  logs and sets _reconnect_event: an unanswered tools/list means response
  delivery is dead even if the stream still looks alive, and the keepalive
  probe uses a different request (ping), so under selective request loss it
  can keep reporting healthy forever.
• tools/mcp_tool.py — in _make_tool_handler, when consecutive tool calls
  time out at the transport level and the count reaches
  _CIRCUIT_BREAKER_THRESHOLD, ask the server task to rebuild via the existing
  _signal_reconnect(). Signalled only at the threshold crossing, never on
  every call. Without this the breaker opens, cools down, re-probes into the
  same wedged transport, and re-arms — indefinitely.
• tests/tools/test_mcp_list_drain_timeout.py — new regression tests.

How to Test

1. scripts/run_tests.sh tests/tools/test_mcp_list_drain_timeout.py — 7 pass.
2. Check them out against the parent commit: 5 of the 7 fail there (discovery,
   refresh, handler reconnect, HTTP read timeout, stdio read timeout). The 2
   that pass are deliberate controls — a server that answers with an error is
   alive and must NOT trigger a rebuild, and a guard that the pinned SDK
   (mcp==1.26.0) still accepts read_timeout_seconds.
3. Full MCP suite: scripts/run_tests.sh tests/tools/test_mcp*.py tests/test_mcp_serve.py tests/hermes_cli/test_mcp_*.py — 987 pass, 0 fail.
4. Manual: point an mcp_servers entry at an endpoint that completes the
   handshake and then black-holes tools/list. Before: tool calls hang until
   the 300s caller timeout, forever. After: discovery fails at connect_timeout
   and the reconnect loop retries with backoff.

Risks / notes for reviewers

• The discovery bound is whole-drain, not per-page. A server that needs more
  than connect_timeout (default 60s) to return up to 50 pages will now fail
  discovery instead of succeeding slowly. The escape hatch is the existing
  per-server connect_timeout setting. Per-page bounding was the alternative,
  but it reinstates a 50 × 60s ceiling under one lock, which defeats the point.
• Cancellation safety. wait_for cancels list_tools() in the server
  task's own frame (not a child of the session's task group), so the SDK's
  send_request finally pops _response_streams[request_id] and closes both
  memory streams — no half-open request state. A late response then finds no
  stream and is routed to _handle_incoming as a RuntimeError, which Hermes'
  message handler already absorbs via its isinstance(message, Exception)
  branch. This matters for the refresh path, where the session deliberately
  stays alive after the cancel.
• The reconnect signal fires once per breaker cycle (strict threshold
  equality). If a rebuild doesn't clear the fault, later half-open probe
  timeouts won't re-signal until something resets the count — a successful
  reconnect calls _reset_server_error, which re-arms it.

Kept out of scope, deliberately

• No new config keys, no changes to connect_timeout / timeout defaults.
• Nothing touched in the release/changelog automation.
• The list_resources / list_prompts drains were left alone: they run from
  handlers that already carry a caller-side deadline, and read_timeout_seconds
  now bounds each of their pages. Adding drain-level bounds there would be
  unrelated churn.
• An earlier draft also treated the SDK's own McpError(408) as a
  reconnect trigger and factored the timeouts into two helpers. Both were
  dropped: 408 is unreachable here (the caller-side deadline starts earlier and
  always fires first), and the helpers cost more lines than the two expressions
  they hid.

Checklist

• I've read the Contributing Guide
• Commit messages follow Conventional Commits
• I searched for existing PRs
• My PR contains only changes related to this fix
• I've run the test suite (scripts/run_tests.sh) and MCP tests pass
• I've added tests for my changes
• I've tested on my platform: macOS 26.4.1, Python 3.12.12
• Documentation — N/A (no config keys or user-facing behavior added)
• cli-config.yaml.example — N/A (no new/changed config keys)
• CONTRIBUTING.md / AGENTS.md — N/A
• Cross-platform impact considered — the stdio hunk is the only
  platform-adjacent one and it's a pure kwarg on the SDK session
• Tool descriptions/schemas — N/A

🤖 Generated with Claude Code

[alt-glitch]

This was generated by AI during triage.

Related: #63823 bounded an earlier reconnect-setup wedge and #61556 repairs breaker recovery. This PR covers the distinct post-session tools/list drain and protocol read-timeout path.

[teknium1]

Thanks for the focused MCP recovery fix. The premise remains present on current main: tools/mcp_tool.py:3017-3020 and tools/mcp_tool.py:2098-2106 await the full tools/list drain while holding _rpc_lock, without a drain-level deadline. Current ClientSession construction likewise has no read_timeout_seconds at the stdio, SSE, or Streamable HTTP call sites (tools/mcp_tool.py:2507-2509, 2860-2862, 2928).

The proposed bounds target the two setup/refresh paths that lack a normal caller deadline. Resource and prompt list drains already execute through handlers whose _run_on_mcp_loop(..., timeout=tool_timeout) gives them a caller-side deadline (tools/mcp_tool.py:4950-4971, 5067-5093). The patch introduces no config key, model-tool surface, cache mutation, or snapshot-style test.

The PR base predates substantial MCP work, but the target functions and surrounding control flow remain materially compatible on current main; salvage should be mostly mechanical.

Automated hermes-sweeper review.