Skip to content

fix: stop dropping WireGuard packets silently on the receive path - #32

Open
kedimuzafer wants to merge 1 commit into
CamM2325:mainfrom
kedimuzafer:fix/wg-rx-queue-drops
Open

fix: stop dropping WireGuard packets silently on the receive path#32
kedimuzafer wants to merge 1 commit into
CamM2325:mainfrom
kedimuzafer:fix/wg-rx-queue-drops

Conversation

@kedimuzafer

Copy link
Copy Markdown

Problem

Three things on the receive path combine to lose packets under sustained TCP load, and none of them logs anything.

1. The queue is 8 deep and the enqueue cannot block. ML_WG_RX_QUEUE_DEPTH is 8, and route_udp_packet() in ml_net_io.c uses xQueueSend(..., 0):

case PKT_WIREGUARD:
    if (xQueueSend(ml->wg_rx_queue, &pkt, 0) != pdTRUE) {
        free(data);   /* silently */
    }

Anything that arrives while ml_wg_mgr is busy is freed and forgotten. It is busy for a while: disco_periodic_probes() takes ~113 ms once a second on a 15-peer tailnet (see #31 for why), and a handshake is longer still. At line rate that is far more than 8 packets.

2. The drain loop then sleeps a fixed 10 ms. Every packet waits up to a full tick before it is looked at, and the queue keeps filling in the meantime.

3. ml_net_io_task takes one datagram per select() round on the socket that carries all tunnelled traffic, so each packet costs a select() and a scheduler round trip.

Symptom on an ESP32-S3 Tailscale SOCKS5 proxy: transfers under ~10 KB worked, anything larger stalled after a few KB and never recovered. tcpdump on the peer showed the node simply stopping — no retransmissions from it, no response to FIN. Nothing in the device log, because the drop path does not log.

Change

  • ML_WG_RX_QUEUE_DEPTH 8 → 64, ML_DISCO_RX_QUEUE_DEPTH 8 → 16. Each slot is an ml_rx_packet_t — a pointer and a few fields — not the packet, so this is a few hundred bytes.
  • The manager loop's poll-then-vTaskDelay(10) becomes a blocking xQueueReceive with the same 10 ms timeout, followed by a non-blocking drain. The task now wakes the moment a packet lands. Idle cost is unchanged: with an empty queue it still blocks for 10 ms, exactly as the trailing delay did.
  • ml_net_io_task drains the DISCO socket up to 32 datagrams per round with MSG_DONTWAIT, bounded so one busy socket cannot starve the STUN sockets below it.

Measured

ESP32-S3, peer on the same LAN, direct path, 1000-byte ICMP through the tunnel:

before after
50 pps 18% loss, RTT avg 190 ms 0% loss
100 pps 0% loss, RTT avg 6.2 ms

Independent of #30 and #31, though all three were measured on the same board and each helps the same bottleneck.

Three things on the receive path combine to lose packets under sustained TCP
load, with nothing logged:

1. ML_WG_RX_QUEUE_DEPTH is 8 and route_udp_packet() enqueues with a zero
   timeout, so anything arriving while the manager task is busy is freed and
   forgotten. The task is busy for ~113 ms once a second doing DISCO probes
   on a 15-peer tailnet, which is many packets at line rate.

2. The manager loop drains that queue and then sleeps a fixed 10 ms, so every
   packet waits up to a full tick before it is even looked at, and the queue
   keeps filling in between.

3. ml_net_io_task takes exactly one datagram per select() round on the socket
   that carries all tunnelled traffic, putting a select() and a scheduler
   round trip in front of each packet.

Raises the queue to 64 (each slot is a pointer and a few fields, not the
packet), replaces the poll-then-sleep with a blocking receive that has the
same 10 ms timeout, and drains the socket up to 32 datagrams per round.

Idle behaviour is unchanged: with an empty queue the task still blocks for
10 ms, exactly as the trailing vTaskDelay did.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016bqLmUG7wJ7p9ya2Vp4A9J
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant