fix: stop dropping WireGuard packets silently on the receive path - #32
Open
kedimuzafer wants to merge 1 commit into
Open
fix: stop dropping WireGuard packets silently on the receive path#32kedimuzafer wants to merge 1 commit into
kedimuzafer wants to merge 1 commit into
Conversation
Three things on the receive path combine to lose packets under sustained TCP load, with nothing logged: 1. ML_WG_RX_QUEUE_DEPTH is 8 and route_udp_packet() enqueues with a zero timeout, so anything arriving while the manager task is busy is freed and forgotten. The task is busy for ~113 ms once a second doing DISCO probes on a 15-peer tailnet, which is many packets at line rate. 2. The manager loop drains that queue and then sleeps a fixed 10 ms, so every packet waits up to a full tick before it is even looked at, and the queue keeps filling in between. 3. ml_net_io_task takes exactly one datagram per select() round on the socket that carries all tunnelled traffic, putting a select() and a scheduler round trip in front of each packet. Raises the queue to 64 (each slot is a pointer and a few fields, not the packet), replaces the poll-then-sleep with a blocking receive that has the same 10 ms timeout, and drains the socket up to 32 datagrams per round. Idle behaviour is unchanged: with an empty queue the task still blocks for 10 ms, exactly as the trailing vTaskDelay did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016bqLmUG7wJ7p9ya2Vp4A9J
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Three things on the receive path combine to lose packets under sustained TCP load, and none of them logs anything.
1. The queue is 8 deep and the enqueue cannot block.
ML_WG_RX_QUEUE_DEPTHis 8, androute_udp_packet()inml_net_io.cusesxQueueSend(..., 0):Anything that arrives while
ml_wg_mgris busy is freed and forgotten. It is busy for a while:disco_periodic_probes()takes ~113 ms once a second on a 15-peer tailnet (see #31 for why), and a handshake is longer still. At line rate that is far more than 8 packets.2. The drain loop then sleeps a fixed 10 ms. Every packet waits up to a full tick before it is looked at, and the queue keeps filling in the meantime.
3.
ml_net_io_tasktakes one datagram perselect()round on the socket that carries all tunnelled traffic, so each packet costs aselect()and a scheduler round trip.Symptom on an ESP32-S3 Tailscale SOCKS5 proxy: transfers under ~10 KB worked, anything larger stalled after a few KB and never recovered.
tcpdumpon the peer showed the node simply stopping — no retransmissions from it, no response to FIN. Nothing in the device log, because the drop path does not log.Change
ML_WG_RX_QUEUE_DEPTH8 → 64,ML_DISCO_RX_QUEUE_DEPTH8 → 16. Each slot is anml_rx_packet_t— a pointer and a few fields — not the packet, so this is a few hundred bytes.vTaskDelay(10)becomes a blockingxQueueReceivewith the same 10 ms timeout, followed by a non-blocking drain. The task now wakes the moment a packet lands. Idle cost is unchanged: with an empty queue it still blocks for 10 ms, exactly as the trailing delay did.ml_net_io_taskdrains the DISCO socket up to 32 datagrams per round withMSG_DONTWAIT, bounded so one busy socket cannot starve the STUN sockets below it.Measured
ESP32-S3, peer on the same LAN, direct path, 1000-byte ICMP through the tunnel:
Independent of #30 and #31, though all three were measured on the same board and each helps the same bottleneck.