Skip to content

ci: bench stall watchdog (SIGQUIT stack dumps) #22358

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yperbasis merged 2 commits into from
Jul 9, 2026
Merged

ci: bench stall watchdog (SIGQUIT stack dumps) #22358

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
66283ba
ci: add stall watchdog and 90min timeout to benchmarks workflow
yperbasis Jul 9, 2026
2ffbc71
Merge remote-tracking branch 'origin/main' into yperbasis/bench-stall…
yperbasis Jul 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion .github/workflows/test-bench.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,29 @@ jobs:

- name: Run benchmarks
if: '!inputs.cache-warming-only'
run: make test-bench
# `go test -timeout` doesn't cover benchmarks (the testing alarm stops
# before the bench phase), so a hung benchmark would only die at the
# 60min job cap — which kills it with no diagnostics. This watchdog
# fires first at 45min: SIGQUIT makes the Go runtime dump all goroutine
# stacks into this step's log, pinpointing the stall, then fails the job.
run: |
(
sleep 2700
{
echo "## bench stall watchdog"
echo "test-bench exceeded 45min: sent SIGQUIT to the test binaries; goroutine dumps are in the 'Run benchmarks' step log"
} >> "$GITHUB_STEP_SUMMARY"
pkill -QUIT -f 'go-build.*\.test' || true
sleep 120
pkill -KILL -f 'go-build.*\.test' || true
) >/dev/null 2>&1 &
watchdog_pid=$!
cleanup() {
pkill -P "$watchdog_pid" 2>/dev/null || true
kill "$watchdog_pid" 2>/dev/null || true
}
trap cleanup EXIT
make test-bench

- name: Build test binaries (cache warming)
if: inputs.cache-warming-only
Expand Down
Loading