fix(mesh): frame loss when recv_validated is cancelled mid-validation (#2458)

[brocoppler]

Summary

Root-causes the #2458 mesh_demo flake to a cancel-safety bug in production code and fixes it, plus one piece of endpoint hygiene the investigation surfaced.

The bug: run_demo_echo races recv_validated against its 100 ms drain tick in a select!. recv_validated reads a frame off the stream, then awaits Redis fence validation. When the tick fires inside that await, the future is dropped with the frame it already consumed — the stream then looks idle forever, the forwarding peer waits for an echo that can no longer happen, and the probe times out (the familiar 504). Loss probability per exchange ≈ validation latency ÷ 100 ms, which is why it worsens exactly when Redis is slow — i.e. under full-suite parallelism, matching #2458's "fails under cargo test, passes standalone" signature. Reproduced at 40–80 % locally, on an in-memory stream pair and live iroh endpoints alike, which rules the transport out as the cause.

Changes

• tunnel/reliable.rs — split validate_received out of recv_validated so callers can extract frames cancel-safely and validate outside any select!/timeout region; the cancel-safety contract is documented on both.
• mesh_boot.rs — run_demo_echo now selects only over frame extraction; decoding + fence validation run outside the cancellable region. Drain semantics unchanged.
• buzz-relay-mesh/peer.rs — IrohRecvHalf::recv_frame is now resumable: partial-read state lives in the half rather than the future (each underlying RecvStream::read await is itself cancel-safe), so racing recv_frame cannot desync the stream either.
• buzz-relay-mesh/endpoint.rs — loopback binds disable the iroh portmapper. A loopback-bound socket can never transmit to a WAN mapping, but the gateway probe (UPnP/PCP/NAT-PMP) advertised one as a Portmapped candidate and path attempts to it fail (sendmsg → EADDRNOTAVAIL) — observed live during the investigation on a NAT-PMP-capable network. Production binds (0.0.0.0) are unaffected. Regression test included (trivially green on hosts without a portmap-capable gateway).
• api/mesh_demo.rs — the forwarded-arm round trip now runs over an in-memory MeshStream pair by default: deterministic, fast, and it reproduced the frame loss identically, making it the regression test for the fix. The live-iroh variant is kept as demo_join_forwarded_arm_round_trips_echo_live_mesh under #[ignore] for explicit transport evidence runs.

Verification

build	variant	runs	result
pre-fix	in-memory pair	10	6 fail (504)
pre-fix	live iroh	5	3 fail (504)
post-fix	in-memory pair	15	15 pass
post-fix	live iroh (--ignored)	15	15 pass
post-fix	full cargo test -p buzz-relay --lib	3	744 passed ×3

cargo clippy -p buzz-relay -p buzz-relay-mesh --lib --tests: clean. buzz-relay-mesh suite: 33 passed.

Relation to existing work

• Mitigates/closes the remaining mechanism behind test(relay): mesh_demo echo test flaky under parallel suite — passes standalone, fails full-suite on clean main #2458. The lease-TTL mechanism proven earlier in that thread is real but distinct; fix: project agent mentions, mesh_demo flake, Builderlab sign-in timeout #2498's TTL raise addresses it for the live variant and composes cleanly with this change (this PR deliberately does not touch the TTL line).
• Supersedes my earlier "≤1 s threshold / pre-existing transport mechanism" interpretation posted on test(relay): mesh_demo echo test flaky under parallel suite — passes standalone, fails full-suite on clean main #2458/fix: project agent mentions, mesh_demo flake, Builderlab sign-in timeout #2498 — the observations there were real, but the threshold framing was wrong: single runs were sampling a tick-alignment race. Corrections posted on those threads.

Fixes #2458

🤖 Generated with Claude Code