wireguardif.c RX path calls ip_input() off the tcpip_thread — pbuf double-free crash under TCP load

[snowpaper]

Summary

The decrypted-RX path in wireguardif.c calls ip_input() directly from the caller's task (the task that runs wireguardif_network_rx / the wg manager task), not from lwIP's tcpip_thread. lwIP's core is not thread-safe: entering it from two threads concurrently corrupts pbuf refcounts. Under sustained TCP traffic through the tunnel this reliably triggers Assertion "pbuf_free: p->ref > 0" and reboots the device mid-transfer.

Environment

• ESP32-S3 (N16R8), ESP-IDF v5.3
• microlink @ 216da33 (also present on current main)
• Reproduced with a TCP proxy on the tailnet IP relaying a ~600 KB stream (LPR print job)

Root cause

components/microlink/components/wireguard_lwip/src/wireguardif.c line 542: after decrypt, the pbuf is handed to ip_input(pbuf, device->netif) on the calling task. Meanwhile tcpip_thread is processing TCP for the same netif (acks for the outbound leg of the proxy). The race double-frees a sent-segment pbuf:

Assertion "pbuf_free: p->ref > 0" failed at line 753 in .../lwip/src/core/pbuf.c

Note: ml_wg_mgr.c already sets netif->input = tcpip_input, i.e. the integration intends all input to be serialized through the tcpip mailbox — the direct ip_input() call in wireguardif.c bypasses that.

Fix

Route the decrypted pbuf through netif->input (== tcpip_input) instead of calling ip_input() directly, so the lwIP core is only ever entered from tcpip_thread. ~8-line change; PR follows.

Verification

On hardware (ESP32-S3): before the fix, a real print job crashed the board mid-job every time (truncated output, reboot). After the fix, multi-hundred-KB jobs relay to completion; the node also survived overnight idle + WAN modem restart and printed first try afterwards.

[kedimuzafer]

Independent confirmation of this on real hardware, with a different crash signature — adding it because it may help pin down how widely it bites.

Setup: ESP32-S3 (8 MB PSRAM), ESP-IDF v5.3.2, MicroLink at 216da33 running a SOCKS5 proxy bound to the node's tailnet address. Peer on the same LAN, direct path, no DERP.

Symptom: every TCP transfer through the tunnel died a few tens of KB in. 10 KB downloads worked; 50 KB did not. A 1 MB request delivered ~6.5 KB and then the node went completely silent — no retransmissions, no response to the client's FIN, and no ICMP replies either, for 40+ seconds, before recovering on its own.

tcpdump on the peer, tunnel side:

16:53:51.160890 IP 100.110.215.4.1080 > peer: Flags [.], seq ...:...  length 1240
16:53:51.163923 IP 100.110.215.4.1080 > peer: Flags [P.], seq ...:... length 808
16:53:51.223642 IP 100.110.215.4.1080 > peer: Flags [.], seq ...:...  length 1240
16:54:35.695556 IP peer > 100.110.215.4.1080: Flags [F.], ...        <- 44 s later, client gives up

Client window wide open the whole time, everything ACKed, nothing lost. The node just stopped producing data.

What it actually was: not a pbuf double-free here, but TCP's own send queue getting corrupted —

assert failed: tcp_free_acked_segments /IDF/components/lwip/lwip/src/core/tcp_in.c:1138
  (tcp_receive: valid queue length)
Backtrace: 0x40375db6 0x4037ddf1 0x403862b9 0x4202d423 ...
Rebooting...

So the device was panicking and rebooting mid-transfer; the "silence" was the reboot plus the ~15-70 s to rejoin WiFi and re-establish the tunnel. The 40 s recovery is what made it look like a stall rather than a crash from the outside.

Same root cause as reported here — wireguardif_process_data_message() calls ip_input() directly, but it runs on ml_wg_mgr's task, not tcpip_thread, so it races with the TCPIP thread's own TCP processing. It only shows up once there is enough concurrent TCP activity for the two to overlap, which is exactly why small transfers survive and large ones do not.

Routing it through netif->input (as #20 and #28 both do) fixed it here: 30 MB now transfers end to end with no assertions, where 50 KB previously could not.

Worth noting the failure is not deterministic in which assert fires — the corruption lands wherever the racing thread happens to be — so anyone hitting a different lwIP assertion under tunnelled TCP load is likely seeing this same bug.