fix(engine): push node context only to reachable nodes, with bounded concurrency

[kjgbot]

Problem

Production workspace rw_7ccfea89 kept returning 429 workspace_busy on channel writes, agent events and spawn channel joins.

Admission allows two in-flight write leases per lane per workspace (RATE_LIMIT_DO, key write-admission-v1, 120s TTL). The worker releases each lease at the end of the request. A production tail showed that every POST /v1/channels/{general,engineering}/{join,mute} in this workspace lost its release:

POST /v1/channels/engineering/join  acquire:acquired
  warn  "workspace write admission release failed; scheduling bounded recovery" {failure_class: timeout}
  release_recovery attempt 1..3 -> retry_failed / exhausted (timeout)
POST /v1/channels/general/join ... Error: Worker exceeded memory limit.

The write-admission Durable Object answered every other call within 0–20ms and never received these releases. Once two leases leaked, the write lane stayed at active_leases: 2 and rejected every write for up to 120s (oldest_lease_age_ms climbing from 59s to 117s). Over 14 minutes: 121 write-lane and 39 control-lane rejections, plus 11 leaked leases, all from join/mute or a follow-up PATCH.

Cause

sendNodeContextForChannel, sendNodePresenceContext and the agent-scoped variants pick targets on agent_node_bindings.status = 'active' alone. They never check whether the node is live. They then push to every target at once with Promise.allSettled.

In this workspace:

channel	fan-out targets (node, provider)	distinct nodes	online nodes
general	4542	3074	28
engineering	1744	909	3

So one join sent thousands of NODE_DO fetches in the background of the request. An offline node has no socket, so every one of those fetches fails. They filled the invocation's concurrent-connection budget, and the lease release waited behind them until its 2s timeout. For general they also pushed the isolate past its memory limit.

Fix

• Context fan-out now targets only nodes that can hold a socket: WebSocket nodes whose status is not offline, plus http_push nodes, which have no liveness status. draining nodes keep their socket until they disconnect, so they stay eligible. context.update is ephemeral and never replayed, and a push to an offline node already failed, so this drops no event that was delivered before.
• sendContextToRows runs at most NODE_CONTEXT_SEND_CONCURRENCY (4) pushes at once, so even a large live audience leaves connections free for the request's own subrequests. Failures are still collected and reported as one AggregateError.
• Updated the bound-parameter comments (+1 binding; still well under D1's limit of 100).

This ships to hosted after an engine release and a relaycast-cloud engine bump. Both are human-gated.

Tests

• New in eventDispatch.test.ts: channel, presence, and single and batched agent-scoped fan-out skip offline ws nodes and keep draining ones. Pushes stay bounded (at most 4 in flight) and still reach all 13 live nodes. Failure reporting is unchanged under the bound.
• Positive control: the 4 new behaviour tests fail against origin/main's nodeContext.ts and pass with this change.
• packages/engine: vitest run 99 files / 1119 tests pass, tsc --noEmit clean, eslint clean.
• npm run test:engine:regression: 814 PASS, 0 FAIL.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes real-time fan-out semantics for channel/presence/agent context on hosted workspaces; behavior is intentional (skip offline WS) but could surprise operators who relied on failed pushes to offline nodes.

Overview
Node context.update fan-out no longer targets every active binding and is capped so hosted requests can finish their own subrequests (e.g. write-admission lease release).

Channel, presence, and agent-scoped context pushes now resolve targets with a reachability predicate: WebSocket nodes must not be offline (draining stays eligible); http_push nodes remain eligible. Offline sockets were never deliverable for ephemeral context.update, so this removes wasted Durable Object round-trips in workspaces with thousands of stale offline nodes.

Delivery uses bounded concurrency (NODE_CONTEXT_SEND_CONCURRENCY = 4) via lazy task pooling instead of unbounded Promise.allSettled. WebSocket pushes are scheduled before http_push POSTs so slow HTTP targets do not occupy slots ahead of live sockets. AggregateError batch failure reporting is unchanged.

Tests in eventDispatch.test.ts cover offline skipping, draining inclusion, concurrency bounds, WS-before-HTTP ordering, and failures under the cap. Changelog documents the production workspace_busy motivation.

^{Reviewed by Cursor Bugbot for commit 7352d8e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 013b0f4c-7473-44dd-8a43-1982c9bf3138

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[devin-ai-integration]

Devin Review found 1 potential issue.

3 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)