Skip to content

ci: retry Go module downloads so a proxy stream blip does not fail the jobΒ #8053

Open

Description

@devantler

πŸ€– Generated by the Agentic Engineer

Evidence

On 2026-10-28, four dependency-update PRs failed CI on a single dropped download from the Go module proxy, not on their own changes. Each failure is an HTTP/2 stream error: … INTERNAL_ERROR from proxy.golang.org:

A dependency PR changes go.sum, so the module cache key misses and every job downloads modules fresh. One dropped stream then fails the job. Scheduled agent runs cannot rerun failed jobs (monorepo#4428), so each blip costs a Dependabot rebase and a full ~70-minute CI cycle.

Problem

A short proxy outage turns a correct PR red, and recovering it needs a manual rerun.

Hypothesis and success signal

If CI retries the module download step a few times with backoff before building or testing (for example go mod download in a bounded retry), a transient proxy stream error no longer fails the job. Success: over the next two weeks, no ksail PR job fails with proxy.golang.org … stream error as its only error. Guardrail: a genuinely missing module or checksum mismatch still fails, on the first attempt or after the bounded retries, with Go's own error.

Acceptance criteria

  • Jobs that resolve modules download them through one bounded retry step first.
  • A checksum mismatch or missing version still fails the job.
  • The retry does not hide the error: the last attempt's output is kept in the log.

Rough size: small (one shared step, used by the Go jobs).

Activity

  1. added theissue type on Oct 28, 2026
  2. moved this to πŸ“₯ Backlog in 🌊 Project Boardon Oct 28, 2026
  3. coderabbitai commented on Oct 28, 2026

    @coderabbitai
    πŸ”— Related PRs

    #7993 - ci: retry module downloads before shipped graph audit [merged]


    πŸ“ Issue Planner

    Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.

    • Create Plan

    πŸ§ͺ Issue enrichment is currently in open beta.

    You can configure auto-planning by selecting labels in the issue_enrichment configuration.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
      auto_enrich:
        enabled: false

    πŸ’¬ Have feedback or questions? Drop into our discord!

  4. devantler commented on Oct 28, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Agentic Engineer

    One more ordinary failure confirms the same download pattern: the vulnerability-scan job in main CI 40758011111, at 7c6f7c0aa1d72fe1f89c268d0bd0a031c1450c19, failed on Go proxy HTTP/2 stream errors while fetching modules. This is a download failure, not a completed vulnerability verdict. No additional retry was requested in this investigation.

    PR #8047 has now merged the shared Go workflow v14.0.5. I traced its scanner checkout to the workflow's exact source and confirmed that it includes the bounded three-attempt dependency pre-download added in .github v13.2.2. The scan itself still runs and enforces findings. The PR's exact-head vulnerability scan passed before merge; post-merge validation is still pending.

    This delivers the shared scanner adoption only. Keep this issue open for the other module-resolving jobs and the ordinary-run observation window; neither their recovery nor the two-week outcome is established by this merge.

  5. added a commit that references this issue on Nov 12, 2026
  6. moved this from πŸ“₯ Backlog to πŸƒπŸ»β€β™‚οΈ In Progress in 🌊 Project Boardon Nov 12, 2026
  7. devantler commented on Nov 12, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Agentic Engineer

    Next slice is in draft #8349: the shared module-restore step now downloads any missing modules through the existing three-attempt retry (the one the release builds and the shipped-graph audit already use). That covers every job that restores the shared module cache, including πŸ”₯ Warm Helm Cache from the evidence above. 🧾 Verify Workflow Contracts, which downloaded modules with no cache and no retry, now restores the cache first. The desktop Build + test and Build + cask jobs were already covered by their setup action.

    Still open here after #8349:

  8. devantler commented on Nov 12, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Agentic Engineer

    #8349 merged on 2026-11-12 at 22:39Z (52396f9). In its own CI the restore found no cached module archive, and jobs such as πŸ”₯ Warm Helm Cache and 🧾 Verify Workflow Contracts downloaded every module through the retry and passed. The two-week outcome window starts now: Measure on: 2026-11-26.

    Still open here: the KSail jobs that set up Go without the shared restore, and the shared validator half in devantler-tech/.github#911.

  9. self-assigned this
    on Nov 13, 2026
  10. added 2 commits that reference this issue on Nov 13, 2026
  11. devantler commented on Nov 13, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Agentic Engineer

    Next slice is in draft #8354: the four Linux checks that set up Go without any cache and resolve packages β€” 🧹 Validate EKS Smoke Teardown, πŸ” Verify Local Source Vulnerability Identities, πŸ”— Verify Dependency Test Graph and πŸ”— Verify Dependency Contracts β€” now go through the shared module restore and its bounded retry, and the module-restore binding test pins them.

    Still open after #8354, deliberately left out of it:

    • The Windows checks (πŸͺŸ Validate Native Home Paths, πŸͺŸ Stage Go Notices on Windows) and the macOS πŸ§ͺ Verify Bash 3.2 Workflow Contract. The shared module archive is keyed per runner OS and only Linux has one, so the restore would turn their narrow, per-package fetch into a full module download on every run. They need a cheaper retry, not this one.
    • The single-module provenance fetches in πŸ” Verify go-archive Provenance.
    • The shared validator half in Retry Go module downloads in every shared validator jobΒ .github#911.

    The outcome window from #8349 is unchanged: Measure on: 2026-11-26.

  12. devantler commented on Nov 14, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Agentic Engineer

    The shared validator half is delivered: devantler-tech/.github#920 merged on 2026-11-14 at 11:03Z (closes devantler-tech/.github#911). Every Go job in the shared validator, and its read-only projection, now downloads the modules its restored cache did not supply through the shared three-attempt retry before it builds, tests or lints; only the download is retried, a checksum mismatch or missing version still fails, and a vendoring module is left off the network. In that PR's own CI the validator's test call ran the step on a real runner. KSail's πŸ—οΈ Build, πŸ§ͺ Test, lint and the other shared jobs get it once KSail moves to the next release of the shared Go workflow (a normal Dependabot bump).

    Still open here, unchanged: the Windows and macOS checks and the single-module provenance fetches listed above. The outcome window is unchanged: Measure on: 2026-11-26.

  13. added a commit that references this issue on Nov 14, 2026
  14. devantler commented on Nov 14, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Agentic Engineer

    The Windows and macOS slice is in draft #8390. πŸͺŸ Validate Native Home Paths, πŸͺŸ Stage Go Notices on Windows and πŸ§ͺ Verify Bash 3.2 Workflow Contract now fetch only the modules their own files compile, with the same three-attempt backoff, before they run. That is the cheaper retry these runners needed: they have no shared module archive, so the full restore would have downloaded the whole graph every time. Locally, from an empty module cache, each check's narrow download was enough for the check to build with the network switched off.

    Still open here after #8390: the single-module provenance fetches listed earlier, and the outcome measurement.

    One correction to the note above: KSail calls the shared Go validator only on pushes to main and on pull requests that change the vulnerability allow-list, and it still pins v20.1.18. So the retry from devantler-tech/.github#920 reaches KSail's main runs once a bump to v21.3.5 or later lands. Dependabot has not proposed v21 for this workflow yet, although it moved KSail's other shared workflows to v21.3.4 on 13 November. If tonight's run does not propose it either, the next run looks into why.

  15. devantler commented on Nov 14, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Agentic Engineer

    #8390 merged on 2026-11-14 at 12:48Z after a green CodeRabbit review at its final head. In that PR's own CI, all three checks ran the new step on real runners: πŸͺŸ Validate Native Home Paths, πŸͺŸ Stage Go Notices on Windows and πŸ§ͺ Verify Bash 3.2 Workflow Contract each fetched only their own modules in 1–10 seconds, then passed.

    Still open here: the single-module provenance fetches and the outcome measurement. The shared-validator bump noted above is unchanged.

  16. added a commit that references this issue on Nov 14, 2026
  17. devantler commented on Nov 14, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Agentic Engineer

    The last slice, the single-module provenance fetches, is in draft #8391. πŸ” Verify go-archive Provenance now downloads the module verifier and the module graph through the narrow package retry, and each provenance check fetches its own pinned upstream release through the shared three-attempt retry before reading it. Locally, from an empty module cache, those downloads were enough for every step of the job to pass with the Go module network switched off; with the network off and nothing cached, each check failed after three attempts with Go's own error.

    After #8391 merges, what is left here is the outcome measurement (Measure on: 2026-11-26) and KSail's move to the shared validator release that carries devantler-tech/.github#920.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

Priority

None yet

Effort

None yet

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions