Repository navigation
pyln-testing: reserve ports across pytest workers - #43
Merged
Merged
Conversation
…pproach For some reason ports go back into the available pool sooner than what we would expect. I'm not sure why, but it clearly happens. ephemeral_port_reserve puts them in TIME_WAIT, we can bind to them immediately because of SO_REUSEADDR set, but then, probably because of node restarts, during a test the port is somehow returned to a state where ephemeral_port_reserve will return it again immediately for another test. So we need an additional mechanism for the ports to stay consistent during a test with node restarts. Our current mechanism was flawed in combination with pytest-xdist since it was tracking per worker, not per machine. This new filesystem approach should work across pytest workers Changelog-None (cherry picked from commit c43420d)
A node's connectd is a separate process holding the listen socket, and on shutdown it exits on its own schedule after lightningd itself is gone -- under valgrind, its teardown can lag by tens of seconds. If a test restarts the node in that window, the new connectd fails with 'Address already in use', lightningd exits, and the test times out waiting for 'Server started with public key'. Seen in CI in test_emergencyrecoverpenaltytxn, where the port was still held 37 seconds after the old connectd began shutting down. The filesystem port locks don't cover this: they keep other workers from reserving the port, but the restarting node owns its reservation both times; nothing waits for the old connectd to actually release the socket. Before launching lightningd, bind-probe the node's port (with SO_REUSEADDR, matching connectd, so TIME_WAIT sockets don't count as in use) and only proceed once the bind succeeds. Fixes: ElementsProject#9354 Changelog-None (cherry picked from commit 1136b21)
btcproxy was using port 0 by default without going through our port reservation logic, so it may have caused "Address already in use" issues. It was also binding to 0.0.0.0, so on all addresses. For the port reservations to work we use 127.0.0.1 here too. We choose from outside the source-port range of ports in hopes of avoiding more port bind race conditions. Changelog-None (cherry picked from commit f80895b)
(cherry picked from commit 1331ff8)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backports four upstream pyln-testing commits that stop test nodes from colliding on ports, the most common CI failure here: a node exits at startup with "Address already in use" from connectd or cln-grpc, and the test times out waiting for "Server started with public key".
BitcoinD.kill()releases the ports it holds.The bitcoind fixture teardown is resolved to upstream's
bitcoind.kill(), which keeps the not-started guard this tree already had. A duplicateimport tempfileleft by the cherry-pick is dropped.The eight tests that failed this way in CI pass 27/27 locally, run three times each across 8 workers, and a 12-worker run of the connection, misc, plugin, pay, closing, wallet and splicing suites logs no "Address already in use" and leaves no stale port locks.
Changelog-None