build: upgrade Go 1.26 -> 1.27.1

[Behzad Mirkhanzadeh (behzad-mir)]

Upgrade Go 1.26 → 1.27.1

Bumps the Microsoft Go toolchain to 1.27.1 across every build path, module file, and container image.

What changed (45 files, +79/−98)

Area	Change
Builder images	golang:1.27-azurelinux3.0@sha256:78388dc1… (6 generated Dockerfiles) and golang:1.27.1@sha256:edd9a9ae… (npm + bpf-prog)
Modules	toolchain go1.27.1 in all 13 module files. go directive floors intentionally unchanged to preserve module compatibility
Crypto	Removed all GOEXPERIMENT crypto selectors — systemcrypto, nosystemcrypto, ms_nocgo_opensslcrypto
Vet	Copylock fixes in cns/deviceplugin/server.go and cns/restserver/util_test.go
Misc	.devcontainer VARIANT → 1.27; dropped the GOEXPERIMENT=boringcrypto Dockerfile patching in the Cilium nightly release test

Why the GOEXPERIMENT lines had to go

The systemcrypto / nosystemcrypto / ms_nocgo_opensslcrypto experiments no longer exist in Go 1.27 — setting them is a hard build error. Go 1.27 selects the cgo-less OpenSSL backend automatically on supported Linux architectures, which is exactly what ms_nocgo_opensslcrypto provided in 1.26.

This does not weaken FIPS posture. Since Go 1.25 systemcrypto is enabled by default, so removing the explicit selectors leaves the same backend in place.

npm deliberately keeps MS_GO_NOSYSTEMCRYPTO=1

MS_GO_NOSYSTEMCRYPTO is an environment variable, not a GOEXPERIMENT, and it remains supported in Go 1.27. npm ships on an Ubuntu base that does not carry the Microsoft FIPS OpenSSL build, so it must stay on the standard Go crypto backend. Removing it would make the binary require OpenSSL at startup and panic during initialization.

The go-version-check workflow previously instructed the opposite. That guidance is corrected here, for both the 1.26 and 1.27+ branches.

Registry prefix fix

go-version-check also told the agent to set the bpf-prog/ipv6-hp-bpf runtime base to a bare azurelinux/distroless/base:3.0, which resolves to Docker Hub and returns 401 — the image is not pullable. Both the Dockerfile and the workflow guidance now use the full mcr.microsoft.com/azurelinux/distroless/base:3.0.

Validation

• Both image digests re-resolved independently against MCR ✅
• mcr.microsoft.com/azurelinux/distroless/base:3.0 → HTTP 200 ✅
• All 13 module files parse (go mod edit) ✅
• go vet ./cns/deviceplugin/... ./cns/restserver/... clean ✅
• Generated Dockerfiles produced by make dockerfiles, not hand-edited; no base-image drift beyond the Go bump ✅
• No GOEXPERIMENT references remain in any script, Dockerfile, Makefile, or template ✅

Notes for reviewers

• Conflict risk: ci: fix false-positive issue dedup in go-version-check #4834 also modifies .github/workflows/go-version-check.yaml.
• The tools.go.mod → tools-go/go.mod module restructure that the skill prescribes was deliberately excluded from this PR. It is orthogonal to a toolchain bump, and the generated tools-go/go.mod requires github.com/Azure/azure-container-networking at a published version with no replace directive pointing at the working tree. Worth doing as its own change.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟡 Changes recommended

The go-version-check workflow still contains an earlier 1.27+ GOEXPERIMENT mapping that conflicts with the updated 1.27+ behavior and can drive incorrect automation output.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Upgrades the repository’s preferred Microsoft Go toolchain from 1.26.x to Go 1.27.1 across modules, build scripts, and container build paths, and updates FIPS/crypto build guidance to reflect Go 1.27 removing the relevant GOEXPERIMENT selectors.

Changes:

• Bump toolchain go1.27.1 across the root module and all submodules, and update builder image references to Go 1.27 / 1.27.1.
• Remove GOEXPERIMENT crypto selectors from Dockerfiles/templates and pipeline build scripts (per Go 1.27 behavior).
• Address Go vet/copylock warnings in CNS code and update go-version-check workflow’s 1.27+ guidance.

File summaries

File	Description
zapai/go.mod	Update preferred toolchain to Go 1.27.1.
tools/release/go.mod	Update preferred toolchain to Go 1.27.1.
tools/failure-agent/go.mod	Update preferred toolchain to Go 1.27.1.
tools/azure-npm-to-cilium-validator/go.mod	Update preferred toolchain to Go 1.27.1.
tools.go.mod	Update preferred toolchain to Go 1.27.1 for the tools modfile.
pkgerrlint/go.mod	Update preferred toolchain to Go 1.27.1.
npm/windows.Dockerfile	Bump Go builder image tag to 1.27.1.
npm/linux.Dockerfile	Bump Go builder image tag to 1.27.1.
go.mod	Update root module preferred toolchain to Go 1.27.1.
dropgz/go.mod	Update preferred toolchain to Go 1.27.1.
cns/restserver/util_test.go	Adjust test construction to avoid copylock patterns.
cns/Dockerfile.tmpl	Remove GOEXPERIMENT selector from template build stage.
cns/Dockerfile	Update Go builder pin to 1.27-azurelinux3.0 and remove GOEXPERIMENT.
cns/deviceplugin/server.go	Fix copylock-vet logging pattern by avoiding deref in zap.Any.
cni/Dockerfile.tmpl	Remove GOEXPERIMENT selector from template stages.
cni/Dockerfile	Update Go builder pin to 1.27-azurelinux3.0 and remove GOEXPERIMENT.
cilium-log-collector/Makefile	Remove GOEXPERIMENT from CGO build invocation (Go 1.27 change).
cilium-log-collector/go.mod	Update preferred toolchain to Go 1.27.1.
cilium-log-collector/Dockerfile.tmpl	Remove GOEXPERIMENT from plugin build stage.
cilium-log-collector/Dockerfile	Update Go builder pin to 1.27-azurelinux3.0 and remove GOEXPERIMENT.
build/images.mk	Bump default AzureLinux Go builder tag to 1.27-azurelinux3.0.
bpf-prog/ipv6-hp-bpf/linux.Dockerfile	Bump Go builder image to 1.27.1 and remove GOEXPERIMENT.
bpf-prog/ipv6-hp-bpf/go.mod	Update preferred toolchain to Go 1.27.1.
azure-iptables-monitor/go.mod	Update preferred toolchain to Go 1.27.1.
azure-iptables-monitor/Dockerfile.tmpl	Remove GOEXPERIMENT from build stages.
azure-iptables-monitor/Dockerfile	Update Go builder pin to 1.27-azurelinux3.0 and remove GOEXPERIMENT.
azure-ipam/go.mod	Update preferred toolchain to Go 1.27.1.
azure-ipam/Dockerfile.tmpl	Remove GOEXPERIMENT from build stages.
azure-ipam/Dockerfile	Update Go builder pin to 1.27-azurelinux3.0 and remove GOEXPERIMENT.
azure-ip-masq-merger/go.mod	Update preferred toolchain to Go 1.27.1.
azure-ip-masq-merger/Dockerfile.tmpl	Remove GOEXPERIMENT from build stage.
azure-ip-masq-merger/Dockerfile	Update Go builder pin to 1.27-azurelinux3.0 and remove GOEXPERIMENT.
.pipelines/cni/cilium/nightly-release-test.yml	Drop Go 1.26-era Dockerfile patching for removed GOEXPERIMENTs.
.pipelines/build/scripts/npm.sh	Clarify/retain MS_GO_NOSYSTEMCRYPTO=1 and remove GOEXPERIMENT usage.
.pipelines/build/scripts/ipv6-hp-bpf.sh	Remove GOEXPERIMENT export for Go 1.27+.
.pipelines/build/scripts/install-go.sh	Update fallback Go image pin to 1.27-azurelinux3.0 digest.
.pipelines/build/scripts/dropgz.sh	Remove GOEXPERIMENT export for Go 1.27+.
.pipelines/build/scripts/cns.sh	Remove GOEXPERIMENT export for Go 1.27+.
.pipelines/build/scripts/cni.sh	Remove GOEXPERIMENT export for Go 1.27+.
.pipelines/build/scripts/cilium-log-collector.sh	Remove GOEXPERIMENT export for Go 1.27+.
.pipelines/build/scripts/azure-iptables-monitor.sh	Remove GOEXPERIMENT export for Go 1.27+.
.pipelines/build/scripts/azure-ipam.sh	Remove GOEXPERIMENT export for Go 1.27+.
.pipelines/build/scripts/azure-ip-masq-merger.sh	Remove GOEXPERIMENT export for Go 1.27+.
.github/workflows/go-version-check.yaml	Update 1.27+ crypto guidance and expected GOEXPERIMENT outputs.
.devcontainer/Dockerfile	Bump devcontainer Go variant to 1.27.

Review details

• Files reviewed: 45/45 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Behzad Mirkhanzadeh (behzad-mir)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 4 pipeline(s).

[Copilot]

🟡 Changes recommended

CI/workflow and build reproducibility issues remain (go-version-check prereq logic inconsistency, install-go digest mismatch, and a test assertion argument-order bug).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

cns/restserver/util_test.go:62

• assert.Equal argument order is reversed (expected vs actual), which makes failures harder to interpret and is inconsistent with the rest of the file’s assertions.
  .github/workflows/go-version-check.yaml:599
• This step correctly sets goexp_cgo* outputs to empty for Go 1.27+, but the workflow still has an earlier “Check prerequisites on release branch” block (around lines ~403–410) that sets GOEXP_CGO{0,1} to "systemcrypto" for >=27 and then greps for GOEXPERIMENT in scripts. With Go 1.27 (systemcrypto no longer a GOEXPERIMENT), that earlier block will incorrectly mark release branches as not FIPS-ready and create false prerequisite issues.

          # Determine GOEXPERIMENT rules per version
          if [ "$TARGET_MIN_NUM" -ge 27 ]; then
            echo "goexp_cgo1=" >> "$GITHUB_OUTPUT"
            echo "goexp_cgo0=" >> "$GITHUB_OUTPUT"
          elif [ "$TARGET_MIN_NUM" -ge 26 ]; then
            echo "goexp_cgo1=systemcrypto" >> "$GITHUB_OUTPUT"
            echo "goexp_cgo0=ms_nocgo_opensslcrypto" >> "$GITHUB_OUTPUT"

• Files reviewed: 45/45 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

The updated go-version-check.yaml still contains fragile issue-dedup logic using GitHub full-text search (and a related test refactor still copies a mutex-containing struct), which can cause incorrect workflow behavior and should be corrected before approval.

Review details

Suppressed comments (3)

Previously missed (2) — in code that hasn't changed since the last review.

.github/workflows/go-version-check.yaml:467

• The prerequisite-issue dedup uses gh issue list --search "$MARKER", which is full-text search and can match unrelated issues due to tokenization. That can suppress creation of the prerequisite issue even when no issue contains the exact marker. Use an exact substring match against the issue body instead (also consider updating the similar go-backport: dedup later in this workflow).

This issue also appears on line 595 of the same file.
cns/restserver/util_test.go:20

• newService returns HTTPRestService by value, which still copies the embedded sync.RWMutex in HTTPRestService (mutexes must not be copied after first use, and copylocks was the motivation for this refactor). Returning *HTTPRestService here avoids copying and aligns with the areNCsPresent pointer receiver.

.github/workflows/go-version-check.yaml:599

• The "Check for existing issue" step uses GitHub full-text search (gh issue list --search "in:body ..."), which tokenizes marker strings (e.g. go-minor-update:1.27.1) and can match unrelated issues. That can cause the workflow to incorrectly skip issue creation while still reporting success. Prefer fetching candidate issues and doing an exact substring match against the body.

          # Determine GOEXPERIMENT rules per version
          if [ "$TARGET_MIN_NUM" -ge 27 ]; then
            echo "goexp_cgo1=" >> "$GITHUB_OUTPUT"
            echo "goexp_cgo0=" >> "$GITHUB_OUTPUT"
          elif [ "$TARGET_MIN_NUM" -ge 26 ]; then

• Files reviewed: 45/45 changed files
• Comments generated: 0 new
• Review effort level: Lite