From 1cab0829cf0003fe3745f21b4be60698f0ea1851 Mon Sep 17 00:00:00 2001
From: Patrick O'Reilly <p@ovpl.co>
Date: Thu, 17 Sep 2026 23:08:20 -0700
Subject: [PATCH] ci: make the wedge hunt keep what it catches (#564)

The hunt reproduced #564 on its second dispatch and we learned nothing from
it. Three reasons, three fixes.

The round hung for 110 minutes and the job's own 120-minute cap cut it off
mid-evidence, so the artifact step never ran on the data that mattered. Bound
each round at 25 minutes instead: a round we kill is a round whose logs we
still have, and the loop moves on to the next sample rather than the whole
batch dying on one.

jhm's output was block-buffered, so whatever it wrote after the last flush
went down with the cancelled job. Run it under stdbuf -oL.

And #568's join warning -- the line that names the stalled replication loop --
is orlyi syslog landing in the CHILD's log file under /tmp/import_repl_test_*.
The fixture only surfaces those tails when it fails its own assertions, so a
round killed from outside discards them. Archive the scratch dirs
unconditionally.

Note a hung round is now itself recorded as a catch. With #571 in, jhm no
longer hangs reaping a test that left grandchildren holding stdout, so a round
that still overruns means something genuinely stuck -- worth stopping on.
---
 .github/workflows/arm-wedge-hunt.yml | 28 ++++++++++++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)

diff --git a/.github/workflows/arm-wedge-hunt.yml b/.github/workflows/arm-wedge-hunt.yml
index 73f62138..824974e0 100644
--- a/.github/workflows/arm-wedge-hunt.yml
+++ b/.github/workflows/arm-wedge-hunt.yml
@@ -59,8 +59,30 @@ jobs:
           caught=""
           for r in $(seq 1 "${{ github.event.inputs.rounds || '5' }}"); do
             echo "==================== make test, round $r ===================="
-            PATH="$PWD/tools:$PATH" jhm --test > "/tmp/suite.$r.log" 2>&1
+            # Bound the round OURSELVES, well under the job cap. The first
+            # reproduction of #564 hung a round for 110 minutes and the job's
+            # own 120-minute timeout cut it off mid-evidence -- the artifacts
+            # that mattered were never written. A round we kill is a round we
+            # still have the logs for, and the loop moves on to the next
+            # sample instead of the batch dying.
+            #
+            # stdbuf so a killed round still yields everything jhm wrote up to
+            # the kill, rather than losing the last block-buffered chunk.
+            timeout -s KILL 1500 stdbuf -oL -eL \
+              env PATH="$PWD/tools:$PATH" jhm --test > "/tmp/suite.$r.log" 2>&1
             rc=$?
+            if [ $rc -eq 137 ]; then
+              echo "round $r was killed at its 25-minute bound (a hung suite is itself a catch)"
+              caught="round $r (suite hung)"
+            fi
+            # The orlyi child logs are where #568's join warning actually
+            # lands. The fixture only surfaces them when IT fails its own
+            # assertions, so anything killed from outside loses them -- which
+            # is how the first reproduction escaped. Keep them unconditionally.
+            for d in /tmp/import_repl_test_*; do
+              [ -d "$d" ] || continue
+              tar -czf "/tmp/scratch.$r.$(basename "$d").tgz" "$d" 2>/dev/null
+            done
             echo "round $r rc=$rc"
             # #568: the join names the loop that never returned.
             if grep -q "JoinReplicationServices() still waiting" "/tmp/suite.$r.log"; then
@@ -89,7 +111,9 @@ jobs:
         uses: actions/upload-artifact@v4
         with:
           name: arm-wedge-hunt-suites
-          path: /tmp/suite.*.log
+          path: |
+            /tmp/suite.*.log
+            /tmp/scratch.*.tgz
           if-no-files-found: warn
 
       - name: Verdict