ci: enforce 1h job timeout across all merge-queue workflows - #22359
Merged
Conversation
lystopad
approved these changes
Jul 9, 2026
lystopad
left a comment
lystopad
left a comment
Member
There was a problem hiding this comment.
Please, check warnings from "artipacked".
+1 approval.
taratorio
enabled auto-merge
July 9, 2026 12:10
pull Bot
pushed a commit
to Dustin4444/erigon
that referenced
this pull request
Jul 10, 2026
## Problem Since July 7 the `benchmarks (serial|parallel)` jobs occasionally stall for 1.5–5.5 hours (12 events on July 7–9; baseline is ~33min). Examples: [339min parallel](https://github.com/erigontech/erigon/actions/runs/28990116579), [201min serial](https://github.com/erigontech/erigon/actions/runs/28971435753), [303min serial, cancelled](https://github.com/erigontech/erigon/actions/runs/28945231648). Log forensics across six affected runs show one silent gap every time, inside `db/test` between the `BenchmarkSharedDomains_ComputeCommitment` result and the `BenchmarkPruneSmallBatches` result — i.e. in the latter's untimed setup (generation loop + inline commitments + Flush/Commit + BuildFiles). Meanwhile the other ~238 packages complete at normal speed on the same VM, both matrix shards and multiple runner images/CPU types are affected, and no other CI job type shows any distribution change — so this is a workload-level stall, not an infra-wide slowdown. The job's 60min `timeout-minutes` cap (erigontech#22359) bounds the wall-clock damage, but a timed-out job is killed by GitHub with SIGTERM/SIGKILL — no goroutine dump — so the stall leaves no trace of *where* it hung. `go test -timeout` is no help either: `testing.M.Run` stops the timeout alarm before the benchmark phase runs, so it never fires on a hung benchmark. ## Changes A watchdog around `make test-bench`: after 45min — comfortably inside the 60min job cap, and ~2x the normal ~18–21min bench step so it only trips on a real hang — it SIGQUITs the test binaries. The Go runtime (`GOTRACEBACK=1`) then dumps all goroutine stacks into the step log, so the next stall self-diagnoses with the exact stuck stack, and the job fails fast (in `merge_group` the existing fail-fast cancel step evicts the run as usual). A SIGKILL follows 2min later as a backstop if SIGQUIT doesn't bring the process down. ## Deliberately not done here Per CI-GUIDELINES the durable fix for heavy benchmarks is `testing.Short()` guards on the setup. That is deferred on purpose: trimming the workload now would hide the stall before we capture a stack from it. Once the root cause is identified, a follow-up trims the `db/test` benchmark setups. ## Validation `actionlint` clean. This PR touches the workflow file itself, so the benchmarks workflow auto-runs on the PR as a live validation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A merge-queue bench job recently ran for 3h17m before being manually cancelled (https://github.com/erigontech/erigon/actions/runs/28995664276/job/86044573195), where it normally finishes in ~33 min. GitHub's default job timeout is 6 hours, so a single hung job can stall the merge queue for hours. The root cause of that hang will be addressed separately; this PR caps the damage.
Every job in
ci-gate.ymland the reusable workflows it calls now carriestimeout-minutes: 60(timeout-minutescannot be set on auses:job, so it has to live on each job inside the reusable workflows).Duration data from the last 15 successful merge-queue CI Gate runs confirms 60 min is safe: the slowest job apart from bench outliers peaked at 33 min, so every job has ~2x headroom. Bench itself normally runs 30–40 min; the 114–194 min cases were sporadic hangs of the same kind as the cancelled run, which this timeout is meant to kill. This also matches CI-GUIDELINES.md, which targets 15–30 min for merge-queue jobs.
Two pre-existing timeouts were touched or deliberately kept:
test-all-erigon-race.ymltests-linux: reduced 90 → 60. The 90 predates the test-group sharding speedup (Speed up 'all-tests (with -race)' runs #19138); shards now peak at 25 min.test-eest-spec.yml:eest-spec-testsalready had 60 andeest-shard-coveragekeeps its tighter 15.The kurtosis
assertoor_teststep-level caps (matrix.test_timeout_minutes, max 50) remain the effective limit there; the job-level 60 is a backstop around setup/teardown.check-large-files.ymlonly runs on PRs, not in the queue, but gets the same cap for consistency since it is part of ci-gate.Verified with
actionlintandzizmorusing the same flags/config as the lint workflow.zizmor artipacked cleanup
The lint job's zizmor scan annotates PRs with pre-existing
artipackedwarnings (checkout steps that leave the git token persisted in the workspace). This PR also setspersist-credentials: falseon every flagged checkout in the files it touches, plus the handful of other read-only workflows GitHub was surfacing in the annotation list (manifest, docker-tags, ci-cd-main-branch-docker-images, qa-clean-exit-block-downloading, cache-warming-eest-fixtures). All of these only build/test from the checkout; none push via git.Deliberately left alone:
claude.yml(the action pushes commits to branches it creates) andbackups-dashboards.yml(production backup flow with its own git credential setup), plus the wider QA/release sweep — those stay tracked in #21132. Repo-wide emitted findings drop from 59 low to 33 low; the files in this PR are clean.