OSError: Too many open files

[inimrod]

After around a couple days of a node continuously running, it keeps getting these out of system resource errors:

2022-10-11 04:19:36,168 [ERROR]: asyncio: socket.accept() out of system resource
socket: <asyncio.TransportSocket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 8421)>
Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 159, in _accept_connection
    conn, addr = sock.accept()
  File "/usr/lib/python3.10/socket.py", line 293, in accept
    fd, addr = self._accept()
OSError: [Errno 24] Too many open files

Consequently, the node appears offline to the rest of the network, even though it is in fact, still running in the host machine. This is especially true if the node app is being run as a systemd service like the setup in the AWS image.

Quick and dirty fix for now is to set the ulimit -n to a really high value, before launching the node app. Example:

ulimit -n 100000

Or in the systemd unit file, the following line is added under the [Service] section:

LimitNOFILE=100000

But ideally, this should be handled by the node app. From the error message, it appears that the number of opened network connections are the ones that get past the ulimit for the no. of open files, since these opened sockets are also counted as open files in Linux systems.

So, this can be controlled from within the node app itself. The process should keep track of the number of open sockets and regularly close (and/or remove the socket files for) those that are no longer used. This way, the node can effectively avoid these Too many open files errors.