You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ci: retry Go module downloads so a proxy stream blip does not fail the jobΒ #8053
On 2026-10-28, four dependency-update PRs failed CI on a single dropped download from the Go module proxy, not on their own changes. Each failure is an HTTP/2 stream error: β¦ INTERNAL_ERROR from proxy.golang.org:
A dependency PR changes go.sum, so the module cache key misses and every job downloads modules fresh. One dropped stream then fails the job. Scheduled agent runs cannot rerun failed jobs (monorepo#4428), so each blip costs a Dependabot rebase and a full ~70-minute CI cycle.
Problem
A short proxy outage turns a correct PR red, and recovering it needs a manual rerun.
Hypothesis and success signal
If CI retries the module download step a few times with backoff before building or testing (for example go mod download in a bounded retry), a transient proxy stream error no longer fails the job. Success: over the next two weeks, no ksail PR job fails with proxy.golang.org β¦ stream error as its only error. Guardrail: a genuinely missing module or checksum mismatch still fails, on the first attempt or after the bounded retries, with Go's own error.
Acceptance criteria
Jobs that resolve modules download them through one bounded retry step first.
A checksum mismatch or missing version still fails the job.
The retry does not hide the error: the last attempt's output is kept in the log.
Rough size: small (one shared step, used by the Go jobs).
Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.
Create Plan
π§ͺ Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your .coderabbit.yaml:
issue_enrichment:
auto_enrich:
enabled: false
π¬ Have feedback or questions? Drop into our discord!
One more ordinary failure confirms the same download pattern: the vulnerability-scan job in main CI 40758011111, at 7c6f7c0aa1d72fe1f89c268d0bd0a031c1450c19, failed on Go proxy HTTP/2 stream errors while fetching modules. This is a download failure, not a completed vulnerability verdict. No additional retry was requested in this investigation.
PR #8047 has now merged the shared Go workflow v14.0.5. I traced its scanner checkout to the workflow's exact source and confirmed that it includes the bounded three-attempt dependency pre-download added in .github v13.2.2. The scan itself still runs and enforces findings. The PR's exact-head vulnerability scan passed before merge; post-merge validation is still pending.
This delivers the shared scanner adoption only. Keep this issue open for the other module-resolving jobs and the ordinary-run observation window; neither their recovery nor the two-week outcome is established by this merge.
Next slice is in draft #8349: the shared module-restore step now downloads any missing modules through the existing three-attempt retry (the one the release builds and the shipped-graph audit already use). That covers every job that restores the shared module cache, including π₯ Warm Helm Cache from the evidence above. π§Ύ Verify Workflow Contracts, which downloaded modules with no cache and no retry, now restores the cache first. The desktop Build + test and Build + cask jobs were already covered by their setup action.
The shared validator's own jobs (ποΈ Build, π§ͺ Test, lint and the rest) live in devantler-tech/.github; filed as devantler-tech/.github#911.
KSail jobs that set up Go without the shared restore (for example the dependency-contract, Windows notice-staging and cask-rendering checks) still download without a retry.
#8349 merged on 2026-11-12 at 22:39Z (52396f9). In its own CI the restore found no cached module archive, and jobs such as π₯ Warm Helm Cache and π§Ύ Verify Workflow Contracts downloaded every module through the retry and passed. The two-week outcome window starts now: Measure on: 2026-11-26.
Still open here: the KSail jobs that set up Go without the shared restore, and the shared validator half in devantler-tech/.github#911.
Next slice is in draft #8354: the four Linux checks that set up Go without any cache and resolve packages β π§Ή Validate EKS Smoke Teardown, π Verify Local Source Vulnerability Identities, π Verify Dependency Test Graph and π Verify Dependency Contracts β now go through the shared module restore and its bounded retry, and the module-restore binding test pins them.
Still open after #8354, deliberately left out of it:
The Windows checks (πͺ Validate Native Home Paths, πͺ Stage Go Notices on Windows) and the macOS π§ͺ Verify Bash 3.2 Workflow Contract. The shared module archive is keyed per runner OS and only Linux has one, so the restore would turn their narrow, per-package fetch into a full module download on every run. They need a cheaper retry, not this one.
The single-module provenance fetches in π Verify go-archive Provenance.
The shared validator half is delivered: devantler-tech/.github#920 merged on 2026-11-14 at 11:03Z (closes devantler-tech/.github#911). Every Go job in the shared validator, and its read-only projection, now downloads the modules its restored cache did not supply through the shared three-attempt retry before it builds, tests or lints; only the download is retried, a checksum mismatch or missing version still fails, and a vendoring module is left off the network. In that PR's own CI the validator's test call ran the step on a real runner. KSail's ποΈ Build, π§ͺ Test, lint and the other shared jobs get it once KSail moves to the next release of the shared Go workflow (a normal Dependabot bump).
Still open here, unchanged: the Windows and macOS checks and the single-module provenance fetches listed above. The outcome window is unchanged: Measure on: 2026-11-26.
The Windows and macOS slice is in draft #8390.πͺ Validate Native Home Paths, πͺ Stage Go Notices on Windows and π§ͺ Verify Bash 3.2 Workflow Contract now fetch only the modules their own files compile, with the same three-attempt backoff, before they run. That is the cheaper retry these runners needed: they have no shared module archive, so the full restore would have downloaded the whole graph every time. Locally, from an empty module cache, each check's narrow download was enough for the check to build with the network switched off.
Still open here after #8390: the single-module provenance fetches listed earlier, and the outcome measurement.
One correction to the note above: KSail calls the shared Go validator only on pushes to main and on pull requests that change the vulnerability allow-list, and it still pins v20.1.18. So the retry from devantler-tech/.github#920 reaches KSail's main runs once a bump to v21.3.5 or later lands. Dependabot has not proposed v21 for this workflow yet, although it moved KSail's other shared workflows to v21.3.4 on 13 November. If tonight's run does not propose it either, the next run looks into why.
#8390 merged on 2026-11-14 at 12:48Z after a green CodeRabbit review at its final head. In that PR's own CI, all three checks ran the new step on real runners: πͺ Validate Native Home Paths, πͺ Stage Go Notices on Windows and π§ͺ Verify Bash 3.2 Workflow Contract each fetched only their own modules in 1β10 seconds, then passed.
Still open here: the single-module provenance fetches and the outcome measurement. The shared-validator bump noted above is unchanged.
The last slice, the single-module provenance fetches, is in draft #8391.π Verify go-archive Provenance now downloads the module verifier and the module graph through the narrow package retry, and each provenance check fetches its own pinned upstream release through the shared three-attempt retry before reading it. Locally, from an empty module cache, those downloads were enough for every step of the job to pass with the Go module network switched off; with the network off and nothing cached, each check failed after three attempts with Go's own error.
After #8391 merges, what is left here is the outcome measurement (Measure on: 2026-11-26) and KSail's move to the shared validator release that carries devantler-tech/.github#920.
Activity
coderabbitai commentedon Oct 28, 2026
π Related PRs
#7993 - ci: retry module downloads before shipped graph audit [merged]
π Issue Planner
Check the box below or use the
@coderabbitai plancommand to generate an implementation plan and prompts that you can use with your favorite coding assistant.π§ͺ Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:π¬ Have feedback or questions? Drop into our discord!
devantler commentedon Oct 28, 2026
One more ordinary failure confirms the same download pattern: the vulnerability-scan job in main CI 40758011111, at
7c6f7c0aa1d72fe1f89c268d0bd0a031c1450c19, failed on Go proxy HTTP/2 stream errors while fetching modules. This is a download failure, not a completed vulnerability verdict. No additional retry was requested in this investigation.PR #8047 has now merged the shared Go workflow v14.0.5. I traced its scanner checkout to the workflow's exact source and confirmed that it includes the bounded three-attempt dependency pre-download added in .github v13.2.2. The scan itself still runs and enforces findings. The PR's exact-head vulnerability scan passed before merge; post-merge validation is still pending.
This delivers the shared scanner adoption only. Keep this issue open for the other module-resolving jobs and the ordinary-run observation window; neither their recovery nor the two-week outcome is established by this merge.
chore: agent-claim #8053 nonce=92b811578aaa1c2ec0fb8c3ae66b6a9c
devantler commentedon Nov 12, 2026
Next slice is in draft #8349: the shared module-restore step now downloads any missing modules through the existing three-attempt retry (the one the release builds and the shipped-graph audit already use). That covers every job that restores the shared module cache, including
π₯ Warm Helm Cachefrom the evidence above.π§Ύ Verify Workflow Contracts, which downloaded modules with no cache and no retry, now restores the cache first. The desktopBuild + testandBuild + caskjobs were already covered by their setup action.Still open here after #8349:
ποΈ Build,π§ͺ Test, lint and the rest) live in devantler-tech/.github; filed as devantler-tech/.github#911.ci: retry Go module downloads after every module cache restore (#8349)
devantler commentedon Nov 12, 2026
#8349 merged on 2026-11-12 at 22:39Z (
52396f9). In its own CI the restore found no cached module archive, and jobs such asπ₯ Warm Helm Cacheandπ§Ύ Verify Workflow Contractsdownloaded every module through the retry and passed. The two-week outcome window starts now: Measure on: 2026-11-26.Still open here: the KSail jobs that set up Go without the shared restore, and the shared validator half in devantler-tech/.github#911.
chore: agent-claim #8053 nonce=c1f7fd392e04fb7fff8ef21703705af1
chore: agent-claim #8053 nonce=926585e75551b787f49a875a6b03cf63
devantler commentedon Nov 13, 2026
Next slice is in draft #8354: the four Linux checks that set up Go without any cache and resolve packages β
π§Ή Validate EKS Smoke Teardown,π Verify Local Source Vulnerability Identities,π Verify Dependency Test Graphandπ Verify Dependency Contractsβ now go through the shared module restore and its bounded retry, and the module-restore binding test pins them.Still open after #8354, deliberately left out of it:
πͺ Validate Native Home Paths,πͺ Stage Go Notices on Windows) and the macOSπ§ͺ Verify Bash 3.2 Workflow Contract. The shared module archive is keyed per runner OS and only Linux has one, so the restore would turn their narrow, per-package fetch into a full module download on every run. They need a cheaper retry, not this one.π Verify go-archive Provenance.The outcome window from #8349 is unchanged: Measure on: 2026-11-26.
ci: retry the pinned govulncheck install
ci: restore Go modules with a retry in four module-resolving checks (#β¦
devantler commentedon Nov 14, 2026
The shared validator half is delivered: devantler-tech/.github#920 merged on 2026-11-14 at 11:03Z (closes devantler-tech/.github#911). Every Go job in the shared validator, and its read-only projection, now downloads the modules its restored cache did not supply through the shared three-attempt retry before it builds, tests or lints; only the download is retried, a checksum mismatch or missing version still fails, and a vendoring module is left off the network. In that PR's own CI the validator's test call ran the step on a real runner. KSail's
ποΈ Build,π§ͺ Test, lint and the other shared jobs get it once KSail moves to the next release of the shared Go workflow (a normal Dependabot bump).Still open here, unchanged: the Windows and macOS checks and the single-module provenance fetches listed above. The outcome window is unchanged: Measure on: 2026-11-26.
chore: agent-claim #8053 nonce=8531481cc4bdc827d4c0692640cd3bd7
devantler commentedon Nov 14, 2026
The Windows and macOS slice is in draft #8390.
πͺ Validate Native Home Paths,πͺ Stage Go Notices on Windowsandπ§ͺ Verify Bash 3.2 Workflow Contractnow fetch only the modules their own files compile, with the same three-attempt backoff, before they run. That is the cheaper retry these runners needed: they have no shared module archive, so the full restore would have downloaded the whole graph every time. Locally, from an empty module cache, each check's narrow download was enough for the check to build with the network switched off.Still open here after #8390: the single-module provenance fetches listed earlier, and the outcome measurement.
One correction to the note above: KSail calls the shared Go validator only on pushes to
mainand on pull requests that change the vulnerability allow-list, and it still pins v20.1.18. So the retry from devantler-tech/.github#920 reaches KSail'smainruns once a bump to v21.3.5 or later lands. Dependabot has not proposed v21 for this workflow yet, although it moved KSail's other shared workflows to v21.3.4 on 13 November. If tonight's run does not propose it either, the next run looks into why.test(ci): split the narrow download retry scenarios into their own test
ci: retry module downloads in the narrow Windows and macOS checks (#8390
devantler commentedon Nov 14, 2026
#8390 merged on 2026-11-14 at 12:48Z after a green CodeRabbit review at its final head. In that PR's own CI, all three checks ran the new step on real runners:
πͺ Validate Native Home Paths,πͺ Stage Go Notices on Windowsandπ§ͺ Verify Bash 3.2 Workflow Contracteach fetched only their own modules in 1β10 seconds, then passed.Still open here: the single-module provenance fetches and the outcome measurement. The shared-validator bump noted above is unchanged.
chore: agent-claim #8053 nonce=3dfd7a8134183fc28c8e7f785371a205
devantler commentedon Nov 14, 2026
The last slice, the single-module provenance fetches, is in draft #8391.
π Verify go-archive Provenancenow downloads the module verifier and the module graph through the narrow package retry, and each provenance check fetches its own pinned upstream release through the shared three-attempt retry before reading it. Locally, from an empty module cache, those downloads were enough for every step of the job to pass with the Go module network switched off; with the network off and nothing cached, each check failed after three attempts with Go's own error.After #8391 merges, what is left here is the outcome measurement (Measure on: 2026-11-26) and KSail's move to the shared validator release that carries devantler-tech/.github#920.
ci: keep the audited release download script unchanged