net: never block the main thread on udp sends - #3293
Conversation
A full socket send buffer wedges a synchronous send_to until it drains, which on a stalling link means the whole client freezes: no rendering, no input, watchdog stack traces, and on macOS behind a VPN tunnel it froze at connect and mid game. Put the socket in non-blocking mode and, when the buffer is full, wait up to 5ms for it to drain and retry once before giving up on the packet. A drop there is what a lost packet on the wire looks like, the chunk layer resends unacked chunks anyway, and try_again was already treated as harmless in CheckErrorCode. Dropped sends are counted and printed in the connection statistics.
Tooling that parses the existing lines keeps working.
Co-authored-by: sprunk <spr.ng@o2.pl>
sprunk
left a comment
sprunk
left a comment
There was a problem hiding this comment.
LGTM but it would be good if somebody more competent at low-level stuff looked too.
|
Added a changelog snippet, please check if it's accurate (from a PoV of a game developer using the engine and treating the networking bits as a blackbox). |
|
Fair enough, is there someone in particular I should ping for a review, or does that usually happen on its own? |
|
@lostsquirrel1 I guess. |
Looks accurate, that captures it well |
Co-authored-by: bruno-dasilva <8520801+bruno-dasilva@users.noreply.github.com>
|
Now I'm curious about the underlying mechanism of UDP sockets. Under what circumstances do they drain/not drain under different network issues, and do they behave similarly or differently to TCP sockets in those circumstances? My original intuition was because UDP sockets are fire and forget, I don't quite understand how the buffer can fill unless you're sending a metric ton of packets edit: maybe the VPN, acting as the fake NIC, is holding onto packets when the VPN has a network issue? I'd think normal NICs would just put them on the wire and if they were dropped by the router then oh well |
|
sendto doesn’t put the packet on the wire, it hands it to the interface’s output queue, and that queue has a limit. With a real NIC it drains at line rate and you never notice. A VPN is just a process, when that process lags, the queue stops draining and the socket buffer backs up On top of that, macOS returns ENOBUFS or blocks in that situation, whereas Linux, if I remember right, tends to drop silently. That’s why the same code freezes here and looks fine there |
|
neat, thanks! |
bruno-dasilva
left a comment
bruno-dasilva
left a comment
There was a problem hiding this comment.
approving, with some minor comment feedback, but still should have TK (or marek?) look at this as he probably has a better intuition of recoil netcode.
| /* Balance between too many resends (lower) | ||
| * and main thread stalls (higher) */ |
There was a problem hiding this comment.
Curious @lostsquirrel1 what you think re: if we actually have to care about spinlock duration here (and in general)
| /* Balance between too many resends (lower) | |
| * and main thread stalls (higher) */ | |
| // Balance between too many resends (lower) and main thread stalls (higher). | |
| // Note that parent code holds holds a spinlock over this, so we definitely don't | |
| // want to block for too long. |
There was a problem hiding this comment.
maybe wait for feedback from TK before applything this one
Co-authored-by: bruno-dasilva <8520801+bruno-dasilva@users.noreply.github.com>
3 participants
A synchronous send_to blocks until the send buffer drains, which freezes the whole client when the link chokes. Hit this on macOS mid game, with the VPN on
The socket is non-blocking now. A full buffer surfaces as try_again, which CheckErrorCode already treats as harmless, so the packet gets 5ms to drain and is dropped after that, same as a packet lost on the wire. Drops are counted in the connection statistics