fix: multi-replica HA — converge cross-pod subscriptions (#1053) + confirm death before reaping (#1052)

[oeway]

Summary

Two tightly-coupled multi-replica (N≥2 pods) HA fixes. #1052 is a direct consequence of the #1053 window, so they ship together as one coherent fix (0.21.136).

#1053 — cross-pod targeted subscriptions never converge (root cause)

RedisEventBus.subscribe_to_client_events optimistically recorded a targeted psubscribe pattern (targeted:<ws>/<client_id>:*) in _subscribed_patterns before confirming the psubscribe. On a timeout/error the pattern stayed recorded-but-unwired, and the if pattern not in self._subscribed_patterns guard turned every future attempt into a no-op → a permanent cross-pod black hole for that client's targeted messages. Masked for same-pod callers by the local short-circuit in emit.

Fix: separate the DESIRED set (_subscribed_patterns) from the CONFIRMED set (_confirmed_patterns, actually wired to the live pubsub); guard on CONFIRMED. Three convergence paths:

1. a subsequent subscribe_to_client_events call retries the unconfirmed pattern;
2. reconnect re-wires every desired pattern (_rewire_desired_patterns, never discards on failure) — and clears the circuit breaker before re-wiring so a stale-open breaker can't refuse to wire on a fresh healthy pubsub;
3. a background reconciler loop (_reconcile_loop, HYPHA_SUBSCRIPTION_RECONCILE_INTERVAL, default 5s) retries desired − confirmed — required because subscribe_to_client_events is called only once per client at register.

Tests: tests/test_cross_pod_subscription_convergence.py (reproduce-before-fix: a dropped targeted psubscribe is later reconciled; the reconcile loop re-wires without a second subscribe call; reconnect re-wire preserves desired and never discards).

#1052 — orphan reaper deletes live clients after a single dropped ping (defense-in-depth)

The #15 continuous reaper deleted all of a client's services:* keys after a single failed cross-pod ping. The ping is best-effort (Redis pub/sub has no buffering), so one dropped message during the #1053 convergence window made the reaper delete a live client's registration — a silent, permanent outage until manual restart.

Fix: track per-client consecutive probe failures across reaper passes and only reap after HYPHA_ORPHAN_REAP_MIN_FAILURES (default 3) consecutive failures. Any successful probe resets the counter; a client that drops out of the candidate set is pruned. At the default 300s interval, 3 failures span ~10 min ≫ the convergence/reconcile window, so only genuinely-dead clients are reaped.

Tests: tests/test_orphan_reaper_confirmation.py (single dropped ping does not reap; N consecutive failures do reap; a recovered probe resets the counter; a disappeared candidate is pruned). The two existing single-pass reaper tests set MIN_FAILURES=1 to preserve their concurrency/startup-cleanup focus.

New env knobs

• HYPHA_SUBSCRIPTION_RECONCILE_INTERVAL (default 5.0s)
• HYPHA_ORPHAN_REAP_MIN_FAILURES (default 3)

Blast radius / review note

Core event-bus subscription path — affects every deployment (single- and multi-pod). Docker-free tests pass locally; multi-replica integration tests run in CI. Do not merge without maintainer (@oeway) sign-off despite green CI, given the all-deployment blast radius.

🤖 Generated with Claude Code