Repository navigation
feat(relay): route relay handlers through work admission; producer publication drain - #24181
Conversation
|
There was a problem hiding this comment.
ℹ️ No critical issues — two non-blocking observations below.
Reviewed changes
Both commits reviewed (70772a9587, 0f2363d76a; 13 files).
- Handler work admission —
onRequest/onNotificationnow route every dispatcher handler throughRelayWorkAdmission.run/runNotification;beginWorkDrain(exclude?)andassertActiveWorkContext(context)are exposed. runstops beingasync— returns the handler's own promise and tracks completion on the side, dropping an extra microtask from every relay response while keeping rejection-on-refusal.transportGeneration— stamped fromclient.generationon request and notificationRequestContexts.pty.cancelDelivery— added to the drain request allowlist so cancellation stays executable mid-drain.- Producer settlement —
publishProducerNotificationgainsonSettled/settledTransportGeneration/isStillAdmittedand settles exactly once on every refusal path;assertSettledProducerTransportrejects replaced/closed generations and callback-less sinks. RelayProducerPublicationDrain— counts each publish until local write completion, re-checks admission at dequeue, and fails sticky on authority/transport change.- Tests — dispatcher-level drain/admission and producer settlement/publication drain.
ℹ️ Drain admission never reopens — confirm the reset path
beginDrain latches draining = true and has no reset, so once this PR makes beginWorkDrain consumable, any caller that does not tear the dispatcher down afterward leaves every later request rejected (relay_work_admission_closed) and every later notification silently dropped. That is clearly correct for the terminal shutdown path, but the PR body advertises "shutdown or reset", and a reset that reuses the live dispatcher would break it. Worth confirming which path T3 takes.
Technical details
# One-way drain admission
## Affected sites
- `src/relay/relay-work-admission.ts:59` — `this.draining = true` is set but never cleared.
- `src/relay/dispatcher-client-state.ts:68` — `beginWorkDrain` exposes it on the dispatcher.
## Required outcome
- Confirm the reset caller either rebuilds the dispatcher/admission, or add a reopen path; otherwise post-reset work is permanently refused.
## Open questions for the human
- Is `beginWorkDrain` intended only for the shutdown path (`RelayGraceLifecycle.shutdown` eventually calls `process.exit(0)`), with reset constructing a fresh admission?ℹ️ Nitpicks
- The abstract
enqueueLeasedFrameindispatcher-client-state.tsstill declares five parameters while the concretedispatcher-producer-transport.tsimplementation adds the new optionalisStillAdmittedsixth. Since the base is the only contract for the mixin chain, syncing the declaration keeps the abstract honest.
DeepSeek Flash (free via Pullfrog for OSS) | 𝕏
db0ab2c to
c0395a8
Compare
04c9331 to
42e1b12
Compare
…processes Wires the stream registries into relay shutdown. The dispatcher's RelayWorkAdmission drain (#24181) ends once fs.readFileStream answers its metadata or a handler answers a git response-stream sentinel; the pumps and file descriptors behind those answers are detached producers it does not see. RelayRuntimeServices.disposeOwnedProcesses now also awaits GitResponseStreamRegistry.disposeAllAndWait and FsHandler.disposeFileStreams, and rejects with relay_owned_process_shutdown_incomplete if either fails, so the grace lifecycle defers shutdown (its existing catch path) instead of exiting with an unclosed fd; the next attempt retries only failed handles. fs.readFileStream now holds a registry operation across metadata preparation (including the binary-prefix probe handle) and across each pump, closes pre-registration handles through releaseUnregisteredHandle, releases its terminal-frame slot even when the handle close fails, and publishes no terminal frame for a stream that was aborted or went stale while its last read was in flight. Behavior today: the grace/SIGTERM shutdown path already calls disposeOwnedProcesses after PTY disposal, so a shutdown now also waits for in-flight stream pumps and queued terminal frames (which settle on client close or dispatcher disposal). After disposal, stream requests are refused with a fenced error; if shutdown is deferred by a failed close, the relay stays up with streams fenced, matching #16741's one-way drain. No wire change. Porting note (source: #16741 head a68b6f3, merge-base 277c289; these files are T2 in the track manifest, taken here only as far as the stream drain needs them): - Taken verbatim: fs-handler-file-read delta, fs-handler-utils isBinaryFilePrefix releaseHandle parameter, fs-handler disposeFileStreams. - Adapted: relay-runtime-services takes only the responseStreams field and the response/file-stream drains in disposeOwnedProcesses. Skill-upload and AI Vault cleanup stay log-and-continue rather than joining the aggregate failure. Added a fs-handler-stream test proving disposeFileStreams waits for an undelivered terminal frame and fences new reads. - Dropped (left for T2): agentExecHandler disposal, disposeWatchers, the ownership-transfer adapter/capture wiring, and the relay-grace-lifecycle beginWorkDrain -> disposeOwnedProcesses -> beginWorkDrain sequence.
First consumer of the T1 work-drain seam: every request and notification handler registered on the relay dispatcher now runs under RelayWorkAdmission, and the dispatcher exposes beginWorkDrain(exclude?) and assertActiveWorkContext(context). RequestContext gains an optional transportGeneration, stamped from client.generation for both requests and notifications, so later producers can pin publication to one transport. Behavior today is unchanged: nothing in production calls beginWorkDrain yet (the shutdown/reset caller is T3), so admission never closes. Decision: pty.cancelDelivery joins the drain request allowlist. #16741 omits it. It only retires an existing source delivery (closes the ledger record, releases retained spans, wakes the publication so it drops its record), it cannot open a delivery or produce output, and a retry replays the remembered proof instead of mutating again. It is the request-shaped sibling of pty.ackData / pty.setDeliveryPaused, which are already allowlisted. Refusing it during a drain would make the client's cancellation proof fail (ssh_source_cancellation_proof_invalid) and leave the delivery retained until detach, turning a provable cancel into an unverifiable one. Revisit only if live ownership transfer (T7, deferred) needs a frozen source ledger. Porting note (source: #16741 head a68b6f3, merge-base 277c289; T1 per the track manifest): - Taken verbatim: dispatcher-client-state onRequest/onNotification wrapping, beginWorkDrain, assertActiveWorkContext; dispatcher-contract transportGeneration; dispatcher-rpc-routing generation stamping. - Adapted: RelayWorkAdmission.run is no longer an async function. It returns the handler's own promise and tracks completion on the side, because an async wrapper adds microtask turns to every relay response and broke existing tests on this base that await one turn after feed() (workspace-session-handler, pty-handler ownership/retired-surface). Refusal and sync throws still surface as rejections. - dispatcher-work-drain.test: kept the four dispatcher-only cases; replaced the two RelayGraceLifecycle.prepareShutdown cases with dispatcher-level equivalents (initiator exclusion, copied/settled context rejection); dropped the real-shutdown case (prepareShutdown/finishShutdown is T3). Added cases for transportGeneration stamping and pty.cancelDelivery during drain. Wire compatibility: no new RPC fields, methods, or opcodes on the wire. transportGeneration is relay-internal. Once a later track calls beginWorkDrain, a refused request gets an ordinary JSON-RPC error (-32000 relay_work_admission_closed) and a refused notification is dropped, which old and new clients already handle as a failed call.
…cation drain Second T1 consumer slice: relay producer publication can now be proven, not just enqueued. publishProducerNotification takes optional onSettled, settledTransportGeneration and isStillAdmitted, and settles exactly once on every path (sink completion, disposal, missing client, PTY admission refusal, capacity refusal, serialization failure) while keeping its existing boolean return and throw contract. assertSettledProducerTransport refuses a replaced or closed client generation and a sink without write callbacks. RelayProducerPublicationDrain builds on the landed TransportPublicationDrain: each publish is counted until its local write completes, and the drain fails sticky once the owner's authority or the transport generation changes, even for frames already queued behind backpressure (isStillAdmitted is rechecked at dequeue through the frame codec and producer transport). Existing producers (watcher events, agent hook envelopes, workspace snapshots) pass none of the new options and behave as before. The drain has no production caller yet; the browser network tunnel (T4) is its first. Porting note (source: #16741 head a68b6f3, merge-base 277c289; T1 per the track manifest): - Taken verbatim: dispatcher-client-writer supportsWriteCallback, dispatcher-frame-codec publicationAdmission, dispatcher-producer-transport isStillAdmitted plumbing, dispatcher-notification-publication settlement, relay-producer-publication-drain, and both tests (relay-producer-publication-drain.test, dispatcher-producer-settlement.test). These files were unchanged on this base since the merge-base, so the upstream hunks applied cleanly. - Adapted: nothing needed; the new code uses no Promise.withResolvers or Bun APIs, so it stays within the relay bundle's Node 18 floor. - Not in this slice: fs/git stream shutdown, ws-transport / node-websocket-lifecycle / websocket-transport-limits, renderer flow-controller deltas (later T1 slices). Wire compatibility: no new RPC fields, methods, or opcodes. Settlement, generation checks and admission are relay-local; a client of any version sees the same notification frames it does today, and a frame refused by the new checks is simply not sent, as a capacity refusal already is.
42e1b12 to
808078a
Compare
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (13)
Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review. 📝 WalkthroughWalkthroughRequest handlers and notifications now use work admission, and request contexts include a transport generation. The dispatcher adds drain checks and permits Priority: ⬇️ Low Merge Risk: ⚪ Minimal · up to The relay drain and publication-settlement changes have no identified merge-blocking issue and are ready to merge after normal checks. Security Architecture ReviewSecurity architecture risk: 🔵 Low · up to The reviewed paths preserve ownership and connection-identity checks without demonstrating a new authorization bypass. Production shutdown/reset activation is deferred. Full rollout and recovery behavior remains unverified, so the assessment is low rather than minimal risk. Retained concerns Security review detailsSecurity Blast Radius
Trust Boundaries and Controls
Resilience and Maintainability Implications
Hardening Proposals
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…processes Wires the stream registries into relay shutdown. The dispatcher's RelayWorkAdmission drain (#24181) ends once fs.readFileStream answers its metadata or a handler answers a git response-stream sentinel; the pumps and file descriptors behind those answers are detached producers it does not see. RelayRuntimeServices.disposeOwnedProcesses now also awaits GitResponseStreamRegistry.disposeAllAndWait and FsHandler.disposeFileStreams, and rejects with relay_owned_process_shutdown_incomplete if either fails, so the grace lifecycle defers shutdown (its existing catch path) instead of exiting with an unclosed fd; the next attempt retries only failed handles. fs.readFileStream now holds a registry operation across metadata preparation (including the binary-prefix probe handle) and across each pump, closes pre-registration handles through releaseUnregisteredHandle, releases its terminal-frame slot even when the handle close fails, and publishes no terminal frame for a stream that was aborted or went stale while its last read was in flight. Behavior today: the grace/SIGTERM shutdown path already calls disposeOwnedProcesses after PTY disposal, so a shutdown now also waits for in-flight stream pumps and queued terminal frames (which settle on client close or dispatcher disposal). After disposal, stream requests are refused with a fenced error; if shutdown is deferred by a failed close, the relay stays up with streams fenced, matching #16741's one-way drain. No wire change. Porting note (source: #16741 head a68b6f3, merge-base 277c289; these files are T2 in the track manifest, taken here only as far as the stream drain needs them): - Taken verbatim: fs-handler-file-read delta, fs-handler-utils isBinaryFilePrefix releaseHandle parameter, fs-handler disposeFileStreams. - Adapted: relay-runtime-services takes only the responseStreams field and the response/file-stream drains in disposeOwnedProcesses. Skill-upload and AI Vault cleanup stay log-and-continue rather than joining the aggregate failure. Added a fs-handler-stream test proving disposeFileStreams waits for an undelivered terminal frame and fences new reads. - Dropped (left for T2): agentExecHandler disposal, disposeWatchers, the ownership-transfer adapter/capture wiring, and the relay-grace-lifecycle beginWorkDrain -> disposeOwnedProcesses -> beginWorkDrain sequence.
#24185) * feat(relay): fence and drain file and git response streams on shutdown Third T1 slice of the #16741 port: the relay's two detached stream producers can now be fenced and awaited, not just flagged. RelayStreamRegistry (fs.readFileStream): - disposeAll permanently fences register/beginOperation (relay_file_stream_shutdown_fenced), waits for every close and for every admitted operation, and rejects with relay_file_stream_shutdown_incomplete while any handle is still unclosed, so a later disposeAll retries only the failed handles. - release coalesces concurrent callers onto one close, tolerates only EBADF as already-closed, and keeps a failed-close entry (aborted) for retry. - beginOperation / releaseUnregisteredHandle let work that opens a handle before registration be drained too (wired in the next commit). GitResponseStreamRegistry: disposeAll fences startStream (relay_response_stream_shutdown_fenced); disposeAllAndWait also awaits every scheduled or in-flight pump, and an aborted pump no longer publishes git.responseEnd after its final chunk write settles. reserveTerminalFrameSlot moves unchanged into fs-stream-terminal-frame-slots and now holds a registry operation per slot, so shutdown also waits for undelivered fs.streamEnd/fs.streamError frames. No wire change: no new methods, fields, or opcodes. Mixed versions are unaffected; a fenced relay answers with an ordinary request error. Porting note (source: #16741 head a68b6f3, merge-base 277c289; T1 per the track manifest): - Taken verbatim: fs-stream-registry delta, git-response-stream delta, fs-stream-registry-shutdown.test, git-response-stream-shutdown.test. - Adapted: Promise.withResolvers replaced with plain promises (the relay bundle targets node18). fs-stream-terminal-frame-slots is extracted from the reserveTerminalFrameSlot that main already has inline in fs-handler-file-read (keeping its rationale comment) rather than added as a second copy. The EBADF check uses a type guard instead of a cast, and the tests' stub casts carry SAFETY rationales. - Dropped: nothing. * feat(relay): drain detached file and git response streams with owned processes Wires the stream registries into relay shutdown. The dispatcher's RelayWorkAdmission drain (#24181) ends once fs.readFileStream answers its metadata or a handler answers a git response-stream sentinel; the pumps and file descriptors behind those answers are detached producers it does not see. RelayRuntimeServices.disposeOwnedProcesses now also awaits GitResponseStreamRegistry.disposeAllAndWait and FsHandler.disposeFileStreams, and rejects with relay_owned_process_shutdown_incomplete if either fails, so the grace lifecycle defers shutdown (its existing catch path) instead of exiting with an unclosed fd; the next attempt retries only failed handles. fs.readFileStream now holds a registry operation across metadata preparation (including the binary-prefix probe handle) and across each pump, closes pre-registration handles through releaseUnregisteredHandle, releases its terminal-frame slot even when the handle close fails, and publishes no terminal frame for a stream that was aborted or went stale while its last read was in flight. Behavior today: the grace/SIGTERM shutdown path already calls disposeOwnedProcesses after PTY disposal, so a shutdown now also waits for in-flight stream pumps and queued terminal frames (which settle on client close or dispatcher disposal). After disposal, stream requests are refused with a fenced error; if shutdown is deferred by a failed close, the relay stays up with streams fenced, matching #16741's one-way drain. No wire change. Porting note (source: #16741 head a68b6f3, merge-base 277c289; these files are T2 in the track manifest, taken here only as far as the stream drain needs them): - Taken verbatim: fs-handler-file-read delta, fs-handler-utils isBinaryFilePrefix releaseHandle parameter, fs-handler disposeFileStreams. - Adapted: relay-runtime-services takes only the responseStreams field and the response/file-stream drains in disposeOwnedProcesses. Skill-upload and AI Vault cleanup stay log-and-continue rather than joining the aggregate failure. Added a fs-handler-stream test proving disposeFileStreams waits for an undelivered terminal frame and fences new reads. - Dropped (left for T2): agentExecHandler disposal, disposeWatchers, the ownership-transfer adapter/capture wiring, and the relay-grace-lifecycle beginWorkDrain -> disposeOwnedProcesses -> beginWorkDrain sequence. * test(relay): cover the stream drains in disposeOwnedProcesses disposeOwnedProcesses now waits for the git response-stream and file-stream drains and rejects with relay_owned_process_shutdown_incomplete when either fails, but nothing exercised that path: relay-daemon tests stub the whole runtime. These tests pin that shutdown is not acknowledged before either drain settles, that both drain failures are aggregated after every owner was attempted (and a retry succeeds), and that skill/AI Vault failures stay log-and-continue in this slice. Porting note (source: #16741 head a68b6f3, src/relay/relay-runtime-owned-process-shutdown.test.ts, T2 in the track manifest): - Taken: the prototype-stub fixture, the "does not acknowledge cleanup before %s have settled" case (responses, fileStreams), and the no-vault-service case. - Adapted: Promise.withResolvers replaced with a plain promise; the fixture cast carries a SAFETY rationale; the retry case targets the stream drains instead of skill/vault, and a new case pins skill/vault as log-and-continue to match this slice's adaptation. - Dropped (left for T2): agentExecHandler and disposeWatchers cases, and skill/vault aggregate-failure cases. --------- Co-authored-by: m4air <m4air@m4airs-Air.localdomain>
* Revert "feat(orcad): source-side dormant export of a relay-hosted SSH target (#16741 T6-8) (#24519)" This reverts commit 783101b. * Revert "feat(ssh): update, roll back, recover and stop a managed orcad server (#16741 T6-5 follow-up) (#24463)" This reverts commit 38c2d1d. * Revert "feat(ssh): deploy and pair an empty managed orcad server over SSH (#16741 T6-5) (#24453)" This reverts commit 8b76683. * Revert "fix(ssh): orcad GC honors the activation journal; readiness requires proven daemon coverage (#16741 T6 follow-up) (#24451)" This reverts commit d3f8c50. * Revert "feat(ssh): remote orcad stop by request file and journaled decommission (#16741 T6-4) (#24449)" This reverts commit 43d9b43. * Revert "feat(orcad): supervisable server: stop requests, managed stop receipts and a lifetime that keeps its lock on failed teardown (#16741 T6-3) (#24433)" This reverts commit b093d3a. * Revert "feat(ssh): crash-safe orcad activation, rollback and recovery (#16741 T6-2) (#24423)" This reverts commit 1a9ac0e. * Revert "feat(runtime): SSH access links for paired servers in a downgrade-safe sidecar (#16741 T5-1+T5-2) (#24420)" This reverts commit 99db2bf. * Revert "feat(relay): capability-gated owner reset with a durable preparation journal (#16741 T3 R1) (#24418)" This reverts commit 34a582b. * Revert "feat(ssh): track connection-manager drains, test probes and provider continuations (#16741 T2 P3+P8a) (#24407)" This reverts commit d53063d. * Revert "feat(daemon): idle retirement, session census and recovery-only provider (#16741 T2 P4b) (#24409)" This reverts commit ff212db. * Revert "feat(ssh): add pty.resumeClient and split SSH PTY process listing (#16741 T2 P5+P6) (#24414)" This reverts commit 92cb717. * Revert "feat(relay): await owned watcher and agent children on shutdown (#16741 T2 P1) (#24400)" This reverts commit 6b36e4f. * Revert "feat(session): retry failed renderer session writes and verify local folder PTYs (#16741 T2 P9) (#24406)" This reverts commit d23ecef. * Revert "feat(ssh): remote orcad primitives on the pinned Node runtime (#16741 T6-1) (#24419)" This reverts commit dd87ae5. * Revert "fix(runtime): fence runtime-environment subscriptions and status probes by identity (#16741 T5-3) (#24421)" This reverts commit ece9e4d. * Revert "feat(orcad): migration manifest and dormant-state contracts (#16741 T6-7) (#24422)" This reverts commit 3fbdaba. * Revert "feat(ssh): wire SshConnection through the work and transport close ledgers (#16741 T2 P2) (#24401)" This reverts commit 4e8edc8. * Revert "feat(profiles): carry markdown frontmatter visibility in project transfers (#16741 T2 P7) (#24405)" This reverts commit 60c9326. * Revert "fix(runtime): project the PTY incarnation onto mobile session tabs (#24413)" This reverts commit 99e0303. * Revert "feat(daemon): tag daemon stream data with the PTY incarnation id (#16741 T2 P4a) (#24402)" This reverts commit 817af76. * Revert "feat(ssh): port the SSH connection work ledger and transport close ledger (#16741 T2) (#24210)" This reverts commit c991893. * Revert "feat(relay): fence and drain file and git response streams on shutdown (#24185)" This reverts commit dc08ffe. * Revert "refactor(runtime-rpc): extract the Node WebSocket lifecycle; opt-in pinned port (#24186)" This reverts commit a789233. * Revert "feat(relay): route relay handlers through work admission; producer publication drain (#24181)" This reverts commit 0b812bd. * Revert "feat(relay): land the #16741 T1 seam (work drain, publication drain, release gate) (#24156)" This reverts commit 3aa2d3a. --------- Co-authored-by: m4air <m4air@Mac.localdomain>

ELI5
This is the next piece of the #16741 port. Relay request handlers now run through the work-admission gate from #24156, so a later shutdown or reset can drain in-flight work safely. Producer notifications also settle through the publication drain. Nothing in production starts a drain yet, so behavior is unchanged.
What Changed
RelayWorkAdmission.beginWorkDrain(exclude?)andassertActiveWorkContext.RequestContextgains an optionaltransportGeneration.runreturns the handler's own promise, so no extra microtask turn is added to each response.RelayProducerPublicationDrainis added.Why
Design D9: these are the first consumers of the T1 seam. T3 (shutdown and reset) and T4 (tunnels) will call the drains.
Linked Issue
Part of the Node runtime plan (Phase 3). Stacked on #24156.
Visual Proof
N/A. There is no behavior change.
Testing
The ported tests pass, including dispatcher drain and admission.
Existing relay handler tests that depend on response timing still pass.
Next: the remaining T1 slices are the fs and git stream shutdown and the WebSocket transport lifecycle.
I manually tested these changes locally
Automated tests added/updated, or explained why not below
Agent skill upstream boundary
docs/reference/agent-skill-sharing-upstream-boundary.mdand copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.