Skip to content

CBG-5862: retry Go module downloads in CI - #8792

Merged
torcolvin merged 4 commits into
mainfrom
CBG-5862
Sep 14, 2026
Merged

torcolvin merged 4 commits into
mainfrom
CBG-5862

Conversation

@bbrks

@bbrks bbrks commented Sep 14, 2026 •

Copy link
Copy Markdown
Member

CBG-5862

Add .ci/retry.sh and give every CI job an explicit, retried download step before it builds or tests.

  • GitHub Actions warms the cache with a package-scoped go list rather than go mod download, since go-fleecedelta is a direct requirement but private, and go mod download ignores build tags, so it fails on runners without the EE credentials.
  • The tool meta-pattern covers gotestsum and addlicense
  • Jenkins has credentials to build EE, so it uses regular go mod download, which covers CE, EE and tool modules at once.

golang/go#51323

Integration Tests

The module proxy has been returning transient errors for weeks, failing
builds in a way that says nothing about the code under test.

Add .ci/retry.sh and give every CI job an explicit, retried download step
before it builds or tests. Builds and tests stay unwrapped: the cache is
warm by then, and retrying a test run would mask genuine flakes.

GitHub Actions warms the cache with a package-scoped go list rather than
go mod download. go-fleecedelta is a direct requirement but private, and
go mod download ignores build tags, so it fails on runners without the EE
credentials. The tool meta-pattern covers gotestsum and addlicense, which
nothing in the tree imports. Jenkins has the credentials, so it uses go mod
download, which needs no tags and covers the CE leg, the EE leg and the
tool modules at once.

Jenkins also warms integration-test/tools, the separate module pinning
cbdinocluster, which start_cbs.py reaches for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@bbrks
bbrks requested review from a team and a lite review from Copilot September 14, 2026 11:42
@factory-droid

factory-droid Bot commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Droid finished @bbrks's task —— View job


The retry wrapper and pre-download steps should make CI more resilient to flaky module proxies. One edge case is retry.sh can end with an unbound status when RETRY_ATTEMPTS is 0 or invalid, so it should validate inputs (and missing args) up front.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The E2E script has unresolved compatibility and retry-coverage issues.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds retryable Go dependency downloads across GitHub Actions, Jenkins, and integration tooling to reduce transient CI failures.

Changes:

  • Adds .ci/retry.sh with exponential backoff.
  • Adds retried dependency prefetching to CI, Jenkins, and integration scripts.
  • Updates shell workflow triggers for .ci changes.
File summaries
File Summary
Jenkinsfile Retries toolchain and module downloads.
jenkins-integration-build.sh Warms root and integration-tool modules.
integration-test/e2e/run_e2e_tests.sh Prefetches CBS tooling; moderate issue: root-module downloads remain unwrapped, and critical issue: older revisions may lack .ci/retry.sh when the script is fetched from main.
.github/workflows/shell.yml Tracks .ci shell-script changes.
.github/workflows/service.yml Adds dependency warming before service builds.
.github/workflows/ci.yml Adds module warming to Go jobs.
.ci/retry.sh Provides the shared retry helper.
Review details
  • Files reviewed: 7/7 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread integration-test/e2e/run_e2e_tests.sh
Comment thread integration-test/e2e/run_e2e_tests.sh
Comment thread .ci/retry.sh
retry.sh exited 0 without running anything when called with no arguments,
and died on an unbound status when RETRY_ATTEMPTS was zero or non-numeric.
Reject both up front, along with a non-numeric RETRY_DELAY.

The e2e pipeline overwrites run_e2e_tests.sh with main's copy even when it
checks out an arbitrary SG_COMMIT, so the helper has to come from main too
or every CBS-backed run against an older commit dies on a missing file.
curl drops the exec bit, hence the chmod.

Warm the root module cache in run_e2e_tests.sh as well: start_local.py
builds Sync Gateway out of REPO_DIR, which a cold worker would otherwise
download unretried. It goes after the private module rewrite so
go-fleecedelta still resolves.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Comment thread .github/workflows/ci.yml Outdated
# go mod download would try to fetch the private EE module, which public runners can't reach
- name: Download Go modules
shell: bash
run: .ci/retry.sh go list -deps -test -tags cb_sg_devmode ./... tool > /dev/null

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
run: .ci/retry.sh go list -deps -test -tags cb_sg_devmode ./... tool > /dev/null
run: .ci/retry.sh go list -deps -test -tags cb_sg_devmode,ruleguard ./... tool > /dev/null

Otherwise ruleguard gets skipped

bbrks and others added 2 commits September 14, 2026 15:40
# Conflicts:
#	integration-test/e2e/Jenkinsfile
ruleguard/ only builds under the ruleguard tag, so the dependency warm-up
skipped it and golangci-lint had to fetch go-ruleguard/dsl mid-lint, outside
the retry wrapper.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@torcolvin
torcolvin enabled auto-merge (squash) September 14, 2026 14:52
@torcolvin
torcolvin merged commit 20787df into main Sep 14, 2026
70 of 72 checks passed
@torcolvin
torcolvin deleted the CBG-5862 branch September 14, 2026 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants