Skip to content

ci: make the wedge hunt keep what it catches (#564) - #572

Merged
ohohoreilly merged 1 commit into
masterfrom
ohohoreilly/564-hunt-keeps-evidence
Sep 18, 2026
Merged

ohohoreilly merged 1 commit into
masterfrom
ohohoreilly/564-hunt-keeps-evidence

Conversation

@ohohoreilly

Copy link
Copy Markdown
Member

ci: make the wedge hunt keep what it catches (#564)

The hunt reproduced #564 on its second dispatch and we learned nothing from
it. Three reasons, three fixes.

The round hung for 110 minutes and the job's own 120-minute cap cut it off
mid-evidence, so the artifact step never ran on the data that mattered. Bound
each round at 25 minutes instead: a round we kill is a round whose logs we
still have, and the loop moves on to the next sample rather than the whole
batch dying on one.

jhm's output was block-buffered, so whatever it wrote after the last flush
went down with the cancelled job. Run it under stdbuf -oL.

And #568's join warning -- the line that names the stalled replication loop --
is orlyi syslog landing in the CHILD's log file under /tmp/import_repl_test_*.
The fixture only surfaces those tails when it fails its own assertions, so a
round killed from outside discards them. Archive the scratch dirs
unconditionally.

Note a hung round is now itself recorded as a catch. With #571 in, jhm no
longer hangs reaping a test that left grandchildren holding stdout, so a round
that still overruns means something genuinely stuck -- worth stopping on.

The hunt reproduced #564 on its second dispatch and we learned nothing from
it. Three reasons, three fixes.

The round hung for 110 minutes and the job's own 120-minute cap cut it off
mid-evidence, so the artifact step never ran on the data that mattered. Bound
each round at 25 minutes instead: a round we kill is a round whose logs we
still have, and the loop moves on to the next sample rather than the whole
batch dying on one.

jhm's output was block-buffered, so whatever it wrote after the last flush
went down with the cancelled job. Run it under stdbuf -oL.

And #568's join warning -- the line that names the stalled replication loop --
is orlyi syslog landing in the CHILD's log file under /tmp/import_repl_test_*.
The fixture only surfaces those tails when it fails its own assertions, so a
round killed from outside discards them. Archive the scratch dirs
unconditionally.

Note a hung round is now itself recorded as a catch. With #571 in, jhm no
longer hangs reaping a test that left grandchildren holding stdout, so a round
that still overruns means something genuinely stuck -- worth stopping on.
@ohohoreilly ohohoreilly self-assigned this Sep 18, 2026
@ohohoreilly
ohohoreilly merged commit aeb9a51 into master Sep 18, 2026
10 checks passed
@ohohoreilly
ohohoreilly deleted the ohohoreilly/564-hunt-keeps-evidence branch September 18, 2026 06:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant