Shared WebSocket RPC sessions in workerd seem to fail under large payload volume

[edmundhung]

While working on a new feature that uses remote bindings to populate OpenNext cache entries into R2, we noticed that a shared WebSocket RPC session could fall over in workerd under large payload volume. I'm not confident whether the right fix belongs in capnweb or in workerd, but it seems to point more at the shared WebSocket session path than anything higher-level.

What seems to be happening is:

• a long-lived shared WebSocket RPC session works fine in Node
• the same pattern does not hold up in workerd once the payloads get large enough
• if we avoid reusing one long-lived session and instead use fresh sessions in smaller batches, the problem goes away

I put together a repro in #157. In the failing test, workerd eventually OOMs and the client side ends with fetch failed / ECONNREFUSED:

workerd/jsg/setup.c++:38: fatal: V8 fatal error; location = Ineffective mark-compacts near heap limit; message = : allocation failed: JavaScript heap out of memory

[kentonv]

Have you tried asking an AI to analyze the issue? What did it say?

[edmundhung]

It suggests the issue to be most likely in the shared WebSocket session path, specifically around backpressure / in-flight data handling. As it doesn’t see any obvious transport-level flow control in the WebSocketTransport:

capnweb/src/websocket.ts

Lines 41 to 57 in cfa1b95

	class WebSocketTransport implements RpcTransport {
	constructor (webSocket: WebSocket) {
	this.#webSocket = webSocket;
	if (webSocket.readyState === WebSocket.CONNECTING) {
	this.#sendQueue = [];
	webSocket.addEventListener("open", event => {
	try {
	for (let message of this.#sendQueue!) {
	webSocket.send(message);
	}
	} catch (err) {
	this.#receivedError(err);
	}
	this.#sendQueue = undefined;
	});
	}

I did notice there's already flow-control / backpressure logic around streams, so I wonder if the WebSocket transport might need something similar?