ci: hunt the wedge with make test, the only thing that reproduces it (#564) - #569
Merged
Merged
Conversation
#564) Third design of this workflow, and the first grounded in evidence rather than in my assumptions about what "load" means. The first two synthesised it -- one test binary behind four CPU spinners, then four concurrent copies of that binary -- and took 83 CI samples without a single reproduction, against an observed rate of about one per arm CI run. Expected yield was roughly five. Worse, the concurrent-copies design manufactured its own failures. The fixture's ProbeFreePort() binds port 0, reads the assigned port and closes the socket before the caller binds it for real. That is fine for a test that runs alone and wrong for two copies racing, which is how two runs produced a bogus "could not connect to server; Connection refused" that I nearly filed as a second arm bug. The only demonstrated reproducer is `make test`: 203 binaries, four at a time, on a machine that has already been churning for ten minutes. That is what the arm CI job runs, and jhm caches nothing between test runs, so looping it gives genuine repeated samples of the real condition instead of an imitation. With #565 and #568 merged a catch is self-describing, so the workflow greps for all three signals: the join naming a stalled replication loop, the fixture's thread dump from a master that would not die, and the fixture failing at all.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ci: hunt the wedge with
make test, the only thing that reproduces it (#564)Third design of this workflow, and the first grounded in evidence rather than
in my assumptions about what "load" means.
The first two synthesised it -- one test binary behind four CPU spinners, then
four concurrent copies of that binary -- and took 83 CI samples without a
single reproduction, against an observed rate of about one per arm CI run.
Expected yield was roughly five.
Worse, the concurrent-copies design manufactured its own failures. The
fixture's ProbeFreePort() binds port 0, reads the assigned port and closes the
socket before the caller binds it for real. That is fine for a test that runs
alone and wrong for two copies racing, which is how two runs produced a bogus
"could not connect to server; Connection refused" that I nearly filed as a
second arm bug.
The only demonstrated reproducer is
make test: 203 binaries, four at a time,on a machine that has already been churning for ten minutes. That is what the
arm CI job runs, and jhm caches nothing between test runs, so looping it gives
genuine repeated samples of the real condition instead of an imitation.
With #565 and #568 merged a catch is self-describing, so the workflow greps for
all three signals: the join naming a stalled replication loop, the fixture's
thread dump from a master that would not die, and the fixture failing at all.