Skip to content

fix(derp): retry-forever backoff + RX-liveness watchdog for the DERP task - #78

Merged
fudio101 merged 1 commit into
mainfrom
fix/issue-37-derp-backoff-watchdog
Aug 19, 2026
Merged

fix(derp): retry-forever backoff + RX-liveness watchdog for the DERP task#78
fudio101 merged 1 commit into
mainfrom
fix/issue-37-derp-backoff-watchdog

Conversation

@fudio101

Copy link
Copy Markdown
Collaborator

Summary

Ports 2 of the remaining pieces of issue #37 (FORK_PRS.md row 18) from Csontikka/microlink's 46e34917 — the retry-forever DERP backoff and the RX-liveness watchdog. (Piece 4, the H2-reassembly fix for poll_map_update(), remains deferred — see below.)

  • Retry-forever backoff: ml_derp_tx_task()'s disconnected branch only ran the bounded 3-attempt/2s retries inside the CONNECT_REQ/RECONNECT event handlers. Once those were exhausted, nothing external re-armed a connect while coord sat in COORD_LONG_POLL — the task just looped on a bare vTaskDelay(100ms) forever. New derp_wanted/derp_next_retry_ms/derp_backoff_ms state keeps retrying with exponential backoff (ML_DERP_RETRY_MIN_MS 5s → ML_DERP_RETRY_MAX_MS 60s, doubling on failure, reset on success).
  • RX-liveness watchdog: ml->derp.last_recv_ms was already being written on every received frame and on connect (existing comment: "For keepalive watchdog") but nothing ever read it — half-wired from the start. Added the read side: if the socket still reports connected but nothing has arrived in ML_DERP_STALE_MS (90s), force a reconnect via ML_EVT_DERP_RECONNECT. Mirrors the already-landed control-plane watchdog pattern (ctrl_stream_rx_ms/ML_CTRL_STREAM_STALE_MS) exactly.

Not ported: the source commit's bundled TLS-state-cleanup half (a fail_tls goto + free on every ml_derp_connect() failure path). Checked ml_derp.c first — issue #14's landed fix already has an equivalent fail_tls/derp_free_tls_state() path on every failure branch, including the mbedtls_ssl_setup() check the source commit added. Porting it again would be a no-op.

Credit to Csontikka/microlink for both fixes.

Test plan

  • Compile-checked via a from-clean pio run against zen-clock with override_path — clean build, -Wall -Werror passes.
  • Flashed to real hardware (LilyGo T-Display-S3, same board/tailnet as DISCO trust-expiry gate, NAT-rebind handshake-skip, SPIRAM pbuf headroom fix (mined from Csontikka/microlink) #33/Dynamic H2 RX window sizing for constrained RAM (mined from djorr5/microlink) #38). ~150s of steady-state boot log: 15/15 DERP status lines showed connected=1, zero DERP retry-forever/DERP RX silent false-positives, zero crashes — confirms the new code doesn't misfire during normal operation.
  • Could not induce a real DERP-side outage from this sandbox (no router or relay-side access) to directly observe the backoff/watchdog recovery path actually firing. Noted as an open verification gap rather than silently assumed correct — the logic was reviewed carefully (mirrors the already-proven ctrl_stream_rx_ms pattern) but hasn't been proven end-to-end under a real failure.

Part of #37 — not closing it, since piece 4 (H2-reassembly for poll_map_update()) is still deferred. Investigated this session too: confirmed it targets a genuinely different, still-unmodified function from do_fetch_peers() (which #38 already rewrote and which already reassembles safely) — no overlap with #38's changes. Reproducing the bug needs a long-poll delta large enough to span a noise_recv() read boundary; no evidence this triggers on the current 5-peer test tailnet, so still deferred pending a larger tailnet or a way to force a large incremental delta.

🤖 Generated with Claude Code

…task

ml_derp_tx_task()'s disconnected branch only ever ran the bounded
3-attempt/2s retries inside the CONNECT_REQ/RECONNECT event handlers --
once those were exhausted, nothing external re-armed a connect while
coord sat in COORD_LONG_POLL, so the task just looped on a bare
vTaskDelay(100ms) forever. New derp_wanted/derp_next_retry_ms/
derp_backoff_ms state keeps retrying with exponential backoff
(ML_DERP_RETRY_MIN_MS 5s -> ML_DERP_RETRY_MAX_MS 60s, doubling on
failure, reset on success) instead.

Separately, ml->derp.last_recv_ms was already being written on every
received frame and on connect (comment: "For keepalive watchdog") but
nothing ever read it. Added the read side: if the socket still reports
connected but nothing has arrived in ML_DERP_STALE_MS (90s), force a
reconnect via ML_EVT_DERP_RECONNECT. Mirrors the already-landed
control-plane watchdog (ctrl_stream_rx_ms/ML_CTRL_STREAM_STALE_MS).

Adapted from Csontikka/microlink's `46e34917`, but the commit's bundled
TLS-state-cleanup half wasn't ported -- issue #14's landed fix already
has an equivalent fail_tls/derp_free_tls_state() path on every
ml_derp_connect() failure branch, porting it again would be a no-op.

Hardware-verified on a real tailnet: ~150s steady-state boot log, 15/15
DERP status lines connected=1, zero false-positive retry/watchdog
triggers, zero crashes. Could not induce a real DERP-side outage from
this sandbox to directly observe the recovery path firing -- noted as
an open gap in FORK_PRS.md rather than silently assumed.

Part of #37 (piece 4, the H2-reassembly fix for poll_map_update(),
remains deferred -- separate, unmodified function from #38's already-
landed do_fetch_peers() rewrite, needs a larger tailnet or a way to
force a large incremental delta to verify).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@fudio101
fudio101 merged commit 065940f into main Aug 19, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants