Ephemeral port exhaustion causing "bind: address already in use"

[jmunson]

What version of shadowsocks-libev are you using?

Debian package shadowsocks-libev 3.3.5+ds-4

What operating system are you using?

Debian 11 (bullseye)

What did you do?

Run one instance of ss-server, sending in a high volume of traffic with many concurrent connections.

What did you expect to see?

Traffic succeeding

What did you see instead?

This error message on stderr "ERROR: bind_to_addr: Address already in use"
And strace showed:
{code}
socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 11259
setsockopt(11259, SOL_TCP, TCP_NODELAY, [1], 4) = 0
setsockopt(11259, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
fcntl(11259, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(11259, F_SETFL, O_RDWR|O_NONBLOCK) = 0
bind(11259, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("EDITED")}, 16) = -1 EADDRINUSE (Address already in use)
{code}

What is your config in detail (with all sensitive info masked)?

{
"server": ["[IPV6ADDR]", "IPV4ADDR"],
"server_port": "1088",
"local_port": "1088",
"password": "PASSWORD",
"timeout": "300",
"method": "aes-256-cfb",
"nameserver": "1.1.1.1",
"local_ipv4_address": "IPV4ADDR",
"local_ipv6_address": "IPV6ADDR",
"acl": "/custom.acl"
}
#########

This appears to be using "bind before connect", where the socket is created, bound to the IP specified as local_ipv4_address, and then connect() is called on it. Since the linux kernel does not know if this socket will have connect() called or listen() called, it assumes it needs a unique local port.

You can see this behavior in the output of ss -npeiO, grepping for the local_ipv4_address, and then excluding the ones to local_port.

So for example ss -npeiO |grep "$local_ipv4_address" |grep -v "${local_ipv4_address}:${local_port}" |awk '{print $5}' |sort should show you all outgoing connections from ss-server to the various websites. Note that no local ports are being used more than once, regardless of the destination. This means the number of outgoing connections can not exceed the amount of local ports as defined in the sysctl net.ipv4.ip_local_port_range

This can be fixed by using the sockopt IP_BIND_ADDRESS_NO_PORT

[madeye]

Triaged — confirmed. bind_to_addr() doesn't set IP_BIND_ADDRESS_NO_PORT (src/netutils.c ~line 117), so the kernel can't defer source-port selection and the ephemeral range is exhausted under load. Labeling bug.