Skip to content

feat(runtime): node:http/https streaming response bodies end-to-end + client/server parity fixes#5057

Merged
proggeramlug merged 4 commits into
mainfrom
node-http-local-parity
Jun 13, 2026
Merged

feat(runtime): node:http/https streaming response bodies end-to-end + client/server parity fixes#5057
proggeramlug merged 4 commits into
mainfrom
node-http-local-parity

Conversation

@proggeramlug

Copy link
Copy Markdown
Contributor

What this is

Working the node:http/http2/https parity mandate. The in-repo vectors turned out to be already green on main (node-suite http 18/18, http2 9/9, https 5/5; test_parity_http* inventory tests diff-clean vs node v26) — the mandate's "5/18, 13 crashes" numbers were stale. The real gap is against Node's own test/parallel corpus (scripts/node_core_subset.py radar): http 30%, https 20%, http2 7% at baseline, with 186 of 403 runtime failures being silent hangs.

Root cause of the silent-hang class: nothing streamed

  • Server: the entire response was buffered into a HyperResponseShape and only handed to hyper via a oneshot at res.end(). res.flushHeaders() set a flag and sent nothing; res.write() buffered. Any client waiting on headers/chunks before response completion waited forever.
  • Client: dispatch_request did response.bytes().await — the 'response' callback only fired after the complete body arrived.

Changes

Server streaming (perry-ext-http-server/response.rs, server.rs):

  • HyperResponseShape.body is now ShapeBody::Full | ShapeBody::Stream (channel-backed hyper Body impl, Frame::data + Frame::trailers).
  • res.flushHeaders() / first res.write() flush the head to the wire immediately (chunked unless Content-Length set — Node's wire behavior); res.end() sends the final chunk + trailers and closes the channel. Single-shot res.end(body) keeps the buffered Content-Length path byte-for-byte.
  • Real backpressure: in-flight byte counter; write() returns false past the 16 KiB HWM; the pump emits 'drain' when it sinks below. Reaper treats a dead stream receiver as peer-gone and never synthesizes over a streaming response.
  • Custom res.statusMessage now reaches the HTTP/1 status line via hyper's ReasonPhrase extension.
  • req.httpVersion reflects the wire version (1.0/1.1/2.0); new httpVersionMajor/httpVersionMinor (HIR member lowering + codegen static table + dynamic dispatch tower — the dot-access path previously hit the pre-tagged-receiver 0.0 sentinel).

Client streaming (perry-ext-http/client_dispatch.rs, client_events.rs, lib.rs):

  • New ResponseHead / ResponseChunk / ResponseEnd pump events: the response callback fires when headers arrive; body chunks deliver as 'data' events as they come off the socket (buffered until listeners attach, flushed at 'end').
  • req.flushHeaders() dispatches body-less requests immediately (new js_http_client_request_flush_headers static route + stdlib twin); a later end() drains the write/finish/end callback ordering exactly once.
  • req.abort() before end() now tears down properly: emits 'abort' + once-only 'close', suppresses dispatch (server never sees the request), no spurious 'error'.
  • Dropped the perry/<version> default User-Agent (Node sends none) and send Node's default Connection: keep-alive — aligns server-side header observations with Node.

Results

Node-core radar (Node v22 own tests, --auto-optimize):

api pass before → after
http 85 → 89 (of 283 judged)
https 10 → 9¹
http2 12 → 12

Corpus tests fixed: test-http-flush-headers, test-http-flush-response-headers, test-http-client-response-timeout, test-http-abort-before-end, test-http-client-agent-abort-close-event.
¹ the two "new" failures are verified flakes, not regressions: test-https-agent-sni hangs identically on the pre-change binary (TLS cert-chain resolution, env-dependent); test-http-dns-error passes on both builds when run idle (DNS/load flake during the sweep).

The modest pass-count delta understates the change: most remaining corpus failures sit behind additional blockers (client 'readable'/read() pull API, Expect: 100-continue, a real client-socket surface, Agent pool internals, keep-alive wire enforcement). The streaming foundation is what SSE / long-poll / chunked responses — i.e. real Express/Fastify/Hono workloads — need, and those follow-ups now have somewhere to land.

Regression guard

  • in-repo node-suite http 18/18, http2 9/9, https 5/5 (apparent compile failures in early sweeps were cold ext-crate rebuild timeouts in the harness, re-verified with proper timeouts).
  • neighbor modules net/events/stream: 827/883; all 49 failures re-run against a pristine origin/main build — every one fails there too (0 regressions).
  • cargo test -p perry-ext-http -p perry-ext-http-server green; cargo fmt --all clean; no Cargo.toml-version / CLAUDE.md / CHANGELOG edits (maintainer folds metadata at merge).

Files

perry-ext-http-server/{response,server,request,handle_dispatch,http2_server}.rs, perry-ext-http/{lib,client_dispatch,client_events,client_request_surface,tls_client,agent,tests}.rs, perry-stdlib/src/{common/dispatch.rs,http/client_request_surface.rs}, perry-codegen/src/{ext_registry.rs,runtime_decls/stdlib_ffi.rs,lower_call/native_table/{http_client,http_server}.rs}, perry-hir/src/lower/expr_member.rs

Ralph Küpper added 4 commits June 13, 2026 06:47
…/write→wire, client head/chunk/end events)

- server: HyperResponseShape body becomes Full|Stream; flushHeaders/first
  write flush the head with a channel-backed hyper body (chunked unless
  Content-Length set); end() sends final chunk + trailers and closes;
  single-shot end(body) keeps the buffered Content-Length path
- backpressure: in-flight byte counter, write() returns false past 16KiB,
  pump emits 'drain' when it sinks below the HWM
- reaper: streaming responses aren't synthesized over; dead body receiver
  counts as peer-gone
- client: reqwest reads stream via chunk(); new ResponseHead/Chunk/End
  events deliver headers before the body completes; data buffered until
  listeners attach, flushed at end
- client req.flushHeaders() dispatches body-less requests immediately
  (new js_http_client_request_flush_headers static route + twins)
- req.httpVersion reflects the wire version (1.0/1.1/2.0) + new
  httpVersionMajor/Minor dispatch properties
…essage reason-phrase, Node-default client headers

- req.abort() emits 'abort'+'close' once and suppresses any later end()
  dispatch (server must never see the request)
- httpVersionMajor/Minor wired through HIR member lowering, codegen
  static table, and the dynamic dispatch tower (dot + computed access)
- custom res.statusMessage reaches the HTTP/1 status line via hyper's
  ReasonPhrase extension
- client: drop the perry/<version> default User-Agent (Node sends none)
  and send Node's default 'Connection: keep-alive'
…FFI decls to stdlib_ffi_part2 (2000-line lint gate)
@proggeramlug proggeramlug force-pushed the node-http-local-parity branch from 6e09f48 to 7b270e8 Compare June 13, 2026 04:53
@proggeramlug proggeramlug merged commit 3dc1c94 into main Jun 13, 2026
13 checks passed
@proggeramlug proggeramlug deleted the node-http-local-parity branch June 13, 2026 06:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant