ci: bench stall watchdog (SIGQUIT stack dumps) - #22358
Conversation
|
Root-cause analysis for the stalls this watchdog targets: #22361. The watchdog in this PR is already live-validated — in the A/B experiment described there, it fired at exactly 60min in both stalled runs, produced the goroutine dumps that pinned the root cause, and failed the jobs fast instead of holding the queue for hours. |
There was a problem hiding this comment.
Pull request overview
Adds guardrails and diagnostics to prevent long-running/stalled benchmark CI jobs from blocking the merge queue, while capturing actionable stack dumps when a stall occurs.
Changes:
- Adds a 90-minute job-level
timeout-minutesbackstop for the benchmarks workflow. - Wraps
make test-benchwith a 60-minute watchdog that sendsSIGQUIT(to trigger Go runtime goroutine dumps) and thenSIGKILLif needed, and writes a short note to the step summary.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # before the bench phase), so without a cap a stalled benchmark runs until | ||
| # GitHub's 6h limit and blocks the merge queue. The in-step watchdog below | ||
| # fails faster (60min) with diagnostics; this is the backstop. | ||
| timeout-minutes: 90 |
There was a problem hiding this comment.
fyi, I set it to 1 hour in #22359
think 60 mins is good
taratorio
left a comment
taratorio
left a comment
There was a problem hiding this comment.
lgtm, maybe let's remove the timeout change since it is done consistently at 60mins across all ci gate jobs in https://github.com/erigontech/erigon/pull/22358/changes#r3551589860
…-watchdog # Conflicts: # .github/workflows/test-bench.yml
yperbasis
changed the title
Problem
Since July 7 the
benchmarks (serial|parallel)jobs occasionally stall for 1.5–5.5 hours (12 events on July 7–9; baseline is ~33min). Examples: 339min parallel, 201min serial, 303min serial, cancelled.Log forensics across six affected runs show one silent gap every time, inside
db/testbetween theBenchmarkSharedDomains_ComputeCommitmentresult and theBenchmarkPruneSmallBatchesresult — i.e. in the latter's untimed setup (generation loop + inline commitments + Flush/Commit + BuildFiles). Meanwhile the other ~238 packages complete at normal speed on the same VM, both matrix shards and multiple runner images/CPU types are affected, and no other CI job type shows any distribution change — so this is a workload-level stall, not an infra-wide slowdown.The job's 60min
timeout-minutescap (#22359) bounds the wall-clock damage, but a timed-out job is killed by GitHub with SIGTERM/SIGKILL — no goroutine dump — so the stall leaves no trace of where it hung.go test -timeoutis no help either:testing.M.Runstops the timeout alarm before the benchmark phase runs, so it never fires on a hung benchmark.Changes
A watchdog around
make test-bench: after 45min — comfortably inside the 60min job cap, and ~2x the normal ~18–21min bench step so it only trips on a real hang — it SIGQUITs the test binaries. The Go runtime (GOTRACEBACK=1) then dumps all goroutine stacks into the step log, so the next stall self-diagnoses with the exact stuck stack, and the job fails fast (inmerge_groupthe existing fail-fast cancel step evicts the run as usual). A SIGKILL follows 2min later as a backstop if SIGQUIT doesn't bring the process down.Deliberately not done here
Per CI-GUIDELINES the durable fix for heavy benchmarks is
testing.Short()guards on the setup. That is deferred on purpose: trimming the workload now would hide the stall before we capture a stack from it. Once the root cause is identified, a follow-up trims thedb/testbenchmark setups.Validation
actionlintclean. This PR touches the workflow file itself, so the benchmarks workflow auto-runs on the PR as a live validation.