Local server can crash with unhandled EPIPE during live heartbeat/dashboard activity

[aronprins]

Summary

The embedded/local Paperclip server can exit unexpectedly with an unhandled EPIPE error while the desktop app is actively viewing dashboard and heartbeat run data.

Observed user-facing message:

Exit code: 1, signal: null
...
node:events:496
throw er; // Unhandled 'error' event
^
Error: write EPIPE
...
code: 'EPIPE',
syscall: 'write'

Evidence

The captured server log showed normal request traffic immediately before the crash, including dashboard, agents, live-runs, heartbeat-runs, heartbeat run events, and workspace operations requests. The crash occurred after a burst of successful 200 responses and one 404 for a heartbeat run log endpoint.

The 404 does not currently look like the primary trigger. The stronger candidate is a raw socket/pipe write after the peer had already disconnected.

Initial hypotheses

1. Live events WebSocket upgrade path can write to a closed socket

The server's live events upgrade handler writes directly to the raw socket when rejecting an upgrade:

• node_modules/@paperclipai/server/dist/realtime/live-events-ws.js
• rejectUpgrade(socket, statusLine, message)
• socket.write(...)

If the client disconnects during or just before that rejection path, this can produce an unhandled EPIPE on a Socket.

2. Electron/server stdio piping may also be vulnerable during teardown races

The Electron app spawns the server with piped stdio and forwards/logs its output:

• src/main.ts
• startServer() pipes child stdout/stderr
• boot flow also writes server output to server.log

If the parent/consumer side closes unexpectedly, a later server-side write to stdout/stderr could also surface as EPIPE.

Why this looks like a product bug, not just a local one-off

• Request traffic was otherwise healthy right before the crash.
• The error is a transport/write failure, not a domain or database error.
• The failure mode currently terminates the server process instead of degrading gracefully.

Suggested direction

• Audit raw socket writes in the live events WebSocket upgrade/rejection path.
• Add defensive error handling around upgrade rejection and other direct socket writes.
• Review whether server stdout/stderr writes are safe during Electron app teardown.
• Add a regression test or repro harness for client disconnects during WebSocket upgrade/reconnect races.

Source log excerpt

[09:58:24] INFO: GET /companies/.../dashboard 200
[09:58:24] WARN: GET /api/heartbeat-runs/558ebfbc-95cf-4f41-8958-b6b276e02add/log?offset=0&limitBytes=256000 404
[09:58:24] INFO: GET /heartbeat-runs/558ebfbc-95cf-4f41-8958-b6b276e02add/workspace-operations 200
[09:58:24] INFO: GET /heartbeat-runs/558ebfbc-95cf-4f41-8958-b6b276e02add/events?afterSeq=0&limit=200 200
...
Error: write EPIPE
...
code: 'EPIPE',
syscall: 'write'

Follow-up

I am doing a deeper investigation now and will comment back with the confirmed root cause, fix, and validation notes.

[aronprins]

Deep-dive findings:

This looks like an upstream @paperclipai/server bug, not a desktop-only data issue.

What seems most likely:

• The strongest candidate is the server live-events WebSocket upgrade path.
• In the bundled server, rejectUpgrade(socket, ...) writes directly to the raw HTTP upgrade socket before ws takes ownership.
• That upgrade/auth path is async, so there is a real race where the UI can disconnect or reconnect before the server finishes deciding whether to reject or upgrade the connection.
• If the peer is already gone when that raw socket.write(...) runs, Node can emit EPIPE or ECONNRESET on the underlying Socket.
• There is no raw socket error listener on that pre-upgrade path, so the error can become process-fatal with the exact Unhandled 'error' event / write EPIPE signature we saw.

Why this is a better fit than the heartbeat 404:

• The requests immediately before the crash were otherwise healthy 200s.
• The 404 on the heartbeat log endpoint appears reproducible without crashing the server and is likely incidental.
• The failure signature is transport-level, not domain/data-level.

Secondary finding:

• Electron stdio/lifecycle is a plausible secondary EPIPE source on quit/restart races because the desktop app pipes server stdout/stderr and does not fully harden those teardown paths.
• But for a crash while the UI was actively open and polling, the WebSocket upgrade path is the stronger match.

Proposed upstream solution:

1. Harden the server's pre-ws upgrade socket handling in live-events-ws.
2. Do not write to sockets that are already closed/destroyed/non-writable.
3. Attach a temporary raw socket error listener during the upgrade/auth window, before ws.handleUpgrade(...) takes ownership.
4. Guard the success path too, so the server does not call handleUpgrade(...) after early disconnect.
5. Prefer socket.end(...) / graceful shutdown semantics over raw write(...) + immediate destroy() where possible.
6. Add a regression test in the upstream server source that simulates a client disconnect during async upgrade authorization and asserts the process does not crash.

Suggested regression test shape upstream:

• Start a real http.Server with setupLiveEventsWebSocketServer(...).
• Use a raw net.Socket, not a WebSocket client, because the failure is pre-upgrade.
• Make auth resolve asynchronously to widen the race.
• Close the client before auth resolves, then let the server take the reject path.
• Assert no uncaught exception / no fatal socket error.

I’m not patching this repo locally. The correct fix should land upstream in @paperclipai/server, and this issue is tagged upstream accordingly.