Skip to content

ci: hunt the wedge with make test, the only thing that reproduces it (#564) - #569

Merged
ohohoreilly merged 1 commit into
masterfrom
ohohoreilly/564-hunt-with-make-test
Sep 18, 2026
Merged

ohohoreilly merged 1 commit into
masterfrom
ohohoreilly/564-hunt-with-make-test

Conversation

@ohohoreilly

Copy link
Copy Markdown
Member

ci: hunt the wedge with make test, the only thing that reproduces it (#564)

Third design of this workflow, and the first grounded in evidence rather than
in my assumptions about what "load" means.

The first two synthesised it -- one test binary behind four CPU spinners, then
four concurrent copies of that binary -- and took 83 CI samples without a
single reproduction, against an observed rate of about one per arm CI run.
Expected yield was roughly five.

Worse, the concurrent-copies design manufactured its own failures. The
fixture's ProbeFreePort() binds port 0, reads the assigned port and closes the
socket before the caller binds it for real. That is fine for a test that runs
alone and wrong for two copies racing, which is how two runs produced a bogus
"could not connect to server; Connection refused" that I nearly filed as a
second arm bug.

The only demonstrated reproducer is make test: 203 binaries, four at a time,
on a machine that has already been churning for ten minutes. That is what the
arm CI job runs, and jhm caches nothing between test runs, so looping it gives
genuine repeated samples of the real condition instead of an imitation.

With #565 and #568 merged a catch is self-describing, so the workflow greps for
all three signals: the join naming a stalled replication loop, the fixture's
thread dump from a master that would not die, and the fixture failing at all.

#564)

Third design of this workflow, and the first grounded in evidence rather than
in my assumptions about what "load" means.

The first two synthesised it -- one test binary behind four CPU spinners, then
four concurrent copies of that binary -- and took 83 CI samples without a
single reproduction, against an observed rate of about one per arm CI run.
Expected yield was roughly five.

Worse, the concurrent-copies design manufactured its own failures. The
fixture's ProbeFreePort() binds port 0, reads the assigned port and closes the
socket before the caller binds it for real. That is fine for a test that runs
alone and wrong for two copies racing, which is how two runs produced a bogus
"could not connect to server; Connection refused" that I nearly filed as a
second arm bug.

The only demonstrated reproducer is `make test`: 203 binaries, four at a time,
on a machine that has already been churning for ten minutes. That is what the
arm CI job runs, and jhm caches nothing between test runs, so looping it gives
genuine repeated samples of the real condition instead of an imitation.

With #565 and #568 merged a catch is self-describing, so the workflow greps for
all three signals: the join naming a stalled replication loop, the fixture's
thread dump from a master that would not die, and the fixture failing at all.
@ohohoreilly ohohoreilly self-assigned this Sep 18, 2026
@ohohoreilly
ohohoreilly merged commit bad63d0 into master Sep 18, 2026
10 checks passed
@ohohoreilly
ohohoreilly deleted the ohohoreilly/564-hunt-with-make-test branch September 18, 2026 03:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant