A proactive relay allocation owns two coupled resources: a MASQUE tunnel
to the relay server, and a Quinn endpoint that uses that tunnel as its
socket. `ProactiveRelay::teardown` destroyed the tunnel while the
endpoint was still running on it, and two things followed.
The endpoint driver's next `poll_recv` returned `BrokenPipe`, so a
teardown we asked for was reported as a transport failure and logged at
ERROR. Fleet-wide this was 2,338 occurrences of `relay recv stream
closed` in 48 h across 73 hosts after the 0.36.0 rollout, about 100x the
previous rate.
Ending the driver that way also discarded the CONNECTION_CLOSE frames
`close()` had just queued. `EndpointDriver::drop` clears
`connections.senders`, so the connection driver handled the queued close,
hit channel EOF, and exited before `drive_transmit`. Peers reached
through the relay were told nothing and waited out their own idle
timeout. The connection's local close reason was overwritten by the
internal transport error for the same reason.
Teardown now closes the endpoint, drains it for up to one second, and
only then shuts the tunnel down, so the close frames leave through a
tunnel that is still alive and `wait_idle` measures a real drain rather
than the sender map the dying driver had just cleared.
A `TunnelState`, shared strongly by the control, the socket and the
tunnel tasks, records why the tunnel ended: `Live`, `Failed` or
`ShutdownRequested`, settled by compare-exchange so the first cause wins.
`poll_recv` parks for a teardown we requested and still reports
`BrokenPipe` for a tunnel that broke, and the tunnel-death watcher stays
quiet for the former. First cause matters because cleanup routinely
arrives after a tunnel has already broken; last writer wins would let it
silence the fault that triggered it. For the same reason the relay health
monitor gets its own abort path, which marks the tunnel failed before
tearing it down and names the exact allocation its verdict was about.
Because the driver no longer crashes, shutdown has to release what the
crash used to release. A `WriterExit` guard held by the writer future
records the writer's exit on every path that ends it, including an abort
that lands before the future's first poll, and wakes anything parked on
send capacity. Whenever the poller answers "writable", `enqueue_outbound`
must not answer `WouldBlock`, which Quinn retries immediately without
yielding, so a full queue whose writer has stopped drops the datagram.
Parking `poll_recv` does not strand the driver: nothing can arrive on a
torn-down tunnel, and it keeps its other wakers, so it still retires with
`Ok(())` once the endpoint is dropped.
Documented in ADR-013.
grumbach
changed the title
dirvine
left a comment
Linear issue
https://linear.app/autonominetwork/issue/V2-986/relay-recv-stream-closed-transport-errors-up-100x-fleet-wide-after-the
Risk tier
Compatibility
server can observe differently:
CONNECTION_CLOSE that
close()queues, because the tunnel is no longerdestroyed out from under them first. This is best-effort, not guaranteed. The
unmodified 0.36.0 already decodes and surfaces it as
ApplicationClosed, sono peer version is surprised by it.
capacity slot, for up to the one-second drain budget longer.
NatTraversalEndpoint::shutdowninherits the same bound before it closesordinary connections.
marks the tunnel failed first and closes with the reason
relay tunnel unhealthyinstead ofrelay allocation aborted. That string travels in theapplication CONNECTION_CLOSE to relayed peers and to the relay server, under
the same error code as before. Nothing in tree parses it.
Semver impact
Summary
A node that uses a relay holds two coupled things: a MASQUE tunnel to the relay
server, and a second QUIC endpoint that sends and receives through that tunnel.
When the node gave up a relay allocation it tore them down in the wrong order,
destroying the tunnel first and closing the endpoint second. Two things followed.
We logged a failure for something we did on purpose. Destroying the tunnel
drops the channel the endpoint reads from, so the endpoint driver's next
poll_recvreturnedBrokenPipe,EndpointDriverresolved toErr, and thespawn wrapper logged
ERROR I/O error: relay recv stream closed. Fleet-wide thatwas 2,338 identical lines in 48 h across 73 hosts after 0.36.0 rolled out on
2026-08-12, about 100x the previous rate.
Peers using that relay were usually not told it was gone.
Endpoint::closeonly queues a
ConnectionEvent::Closeper connection. The frames still have totravel out through the tunnel. When the driver died it cleared
connections.senders, so the connection driver handled the queued close, hitchannel EOF straight away, and exited before
drive_transmit. The close frame wasbuilt and never sent, and the peer waited out its own idle timeout. It is a
scheduling race rather than a certainty, so the claim is "not reliably told", not
"never told". The connection's local close reason was overwritten by the internal
transport error for the same reason.
That second one is why this is worth fixing. The log line is the symptom that
found it.
Three changes:
Order.
close(), then a bounded 1 swait_idle(), thentunnel.shutdown(). A pure statement reorder. The close frames now leavethrough a tunnel that is still alive, and
wait_idlemeasures a real draininstead of the empty sender map the dying driver had just cleared.
Classification.
RelayTunnelControlcarries aTunnelCauseofLive,FailedorShutdownRequestedin anAtomicU8shared with the socket,settled by compare-exchange so the first cause wins.
poll_recvparks insteadof returning
BrokenPipewhen the teardown was requested, and still reportsthe error when the tunnel broke. The tunnel-death watcher stays quiet for a
teardown we asked for.
First-cause matters because cleanup routinely arrives after a tunnel has
already broken. For the same reason the relay health monitor now has its own
abort path that marks the tunnel failed before tearing it down. Without it,
is_relay_healthycan condemn a relay from the state of the outer relaysession while the tunnel's own cause is still
Live, and the teardown wouldfile that genuine failure as an intentional one.
That verdict now also names the relay it was reached about. Checking health
and then fetching the published handle are two separate awaits, so a
replacement could publish in between and the monitor would tear down the
healthy replacement on the strength of a verdict about its predecessor. That
one is pre-existing, but marking the tunnel failed would have made it worse,
so it is fixed here rather than inherited.
Releasing what the crash used to release. The driver crashing was what
freed a connection parked waiting for send capacity. It no longer crashes, so
shutdown wakes
send_capacity_freeditself and the poller consults therecorded writer state rather than the send channel, which only closes after the
asynchronous abort lands. Whenever the poller answers "writable",
enqueue_outboundmust not answerWouldBlock, because Quinn retries thatimmediately without yielding. So a full queue whose writer has stopped drops
the datagram.
Parking
poll_recvdoes not strand the driver. Nothing can arrive on a torn-downtunnel, and it keeps its other wakers: the endpoint-event channel, a connection's
Drainedevent, and the explicit wakeEndpointRef::dropissues at refcountzero. So it still retires with
Ok(()).The cause, the writer state and both
Notifys live in oneTunnelStateheldstrongly by the control, the socket and the tunnel tasks. That matters for the
dial-through path in
p2p_endpoint, which drops its control as soon as the dialcompletes while the socket and its tasks live on. A writer that recorded its exit
through a
Weakto the control would be a no-op there and leave a parked pollerwaiting forever.
What this does not claim
The fix silences only teardowns we requested. Of the 3,150 errors in the 48 h
window, 1,773 (56%) are matched by a completed "Proactive relay torn down". The
rest are unattributed, and some are genuine tunnel failures that this deliberately
keeps loud. ≥56% reduction is the defensible claim, not 100%.
The spike itself has already subsided. It collapsed at 2026-08-13 21:00 UTC when
every relay-lifecycle counter and the restart count fell together, most likely the
rolling upgrade finishing. This has not shipped, so it should not be credited with
that. What it fixes is a defect that recurs whenever relays churn.
CONNECTION_CLOSE delivery is improved, not guaranteed.
Endpoint::closeusestry_sendand drops the event if a connection mailbox is full,wait_idlewaitsfor the connection map to empty rather than for the MASQUE send queue to flush,
and 1 s need not cover Quinn's
3 × PTOdrain.Test evidence
cargo fmt --all -- --check: clean.cargo clippy --all-targets --all-features -- -D warnings: clean.cargo test --lib: 1,502 passed / 0 failed / 3 ignored.masque::152 passed,nat_traversal_api::32 passed.Six tests carry the change, and each was ablated to confirm it fails when the
part it guards is reverted:
real_relay_teardown_closes_the_peer_and_releases_capacityproactive_relay_teardown_reports_no_transport_errorunexpected_tunnel_loss_still_reports_a_transport_errorshutdown_releases_a_poller_parked_on_a_full_send_queuethe_health_monitors_abort_is_recorded_as_a_failuremark_failedon the health-monitor aborta_writer_exit_is_recorded_even_after_its_control_is_droppedWeakreal_relay_teardown_closes_the_peer_and_releases_capacityis the one that closesthe network-facing claim. It uses a real relay service, a real CONNECT-UDP
session, the real
run_stream_forwarding_loopdata plane, the realprepare_proactive_relay/abort_proactive_relaylifecycle, and a real peerdialling the relay-allocated address. It asserts the peer observes
ApplicationClosed(RELAY_TUNNEL_LOST_CODE), that no transport error is logged,and that the relay server gets its capacity slot back. With the ordering reverted
the peer is told nothing and it fails on a five-second timeout. With the
classification reverted it captures the fleet's exact line.
Mixed-version interop was exercised on an earlier revision of this branch by
running the three roles as separate processes linked against either unpatched
a68f1c8cor this head, across all six role combinations. Every patched-clientcase delivered the close regardless of whether the relay server and peer were
patched, and every unpatched-client case timed out exactly as it does today. The
same harness over an address-translating lossy path delivered the close 20/20 on a
clean path and 15/20 at 30% loss, against 0/20 unpatched. That harness was a
one-off experiment and is not part of this PR.
Still open, and stated rather than glossed:
cargo test.boxes, real conntrack expiry and bursty loss are not reachable in simulation.
info!lifecycle logsthat are already in prod telemetry, but nothing is wired to watch them. After
deploy,
relay recv stream closedshould fall to the residual and decouple fromProactive relay torn down, where today the two track 1:1. The relay prepared,torn down and canary-published rates should be unchanged.
New dependency
none. One lockfile-only bump rides along:
h20.4.15 to 0.4.16 forRUSTSEC-2026-0258, which landed in the RustSec database on 2026-08-19 and fails
cargo auditonmaintoo, so it blocks every open PR until the lockfile moves.Only the h2 stanza's version and checksum change, so no other dependency edge is
re-resolved, and
cargo auditthen exits clean with the three warnings alreadyallowed on
main. Happy to split it into its own PR if preferred.ADR
https://github.com/WithAutonomi/saorsa-transport/blob/fix/relay-teardown-transport-error/docs/adr/ADR-013-relay-tunnel-teardown-ordering.md
Mitigation / rollback
Revert and deploy through the normal rolling restart. No wire, stored format or
public API changes, so no coordinated downgrade or simultaneous fleet restart is
needed. Reverting restores the previous behaviour exactly, including the ERROR
line for requested teardowns and the lost CONNECTION_CLOSE.