Track kfp-launcher and kfp-driver vulnerability fixes on master before 2.16.2

[jeffspahr]

Summary

Track resolving vulnerabilities reported by Trivy in the Kubeflow Pipelines 2.16.1 runtime images:

• ghcr.io/kubeflow/kfp-launcher:2.16.1
• ghcr.io/kubeflow/kfp-driver:2.16.1

Both images have the same vulnerability profile. The findings are in the Go binaries, not the Alpine runtime layer.

The immediate focus is to get the relevant fixes merged to main/master first. This repo currently uses master. Once those changes have merged and we have clean scan signal from the updated images, we can decide whether to cut a 2.16.2 patch release with the corresponding release-branch changes.

Scan Details

Scanned on 2026-05-29 with Trivy 0.70.0:

trivy image --scanners vuln --format json ghcr.io/kubeflow/kfp-launcher:2.16.1
trivy image --scanners vuln --format json ghcr.io/kubeflow/kfp-driver:2.16.1

Image digests scanned:

• kfp-launcher: sha256:546a1bb6040da53c894ae469799af195fe438ae045fd9c4003730a92b01dd315
• kfp-driver: sha256:31655c58406f6289c8100a546e59c75dd165dabc79bcef425ee39c9e81d89ebc

Results for each image:

• Runtime OS: Alpine 3.21.7
• OS package vulnerabilities: 0
• Go binary vulnerabilities: 30 total
• Severity breakdown: 1 critical, 21 high, 7 medium, 1 low

Build Correlation

The 2.16.1 release tag builds both images from:

• backend/Dockerfile.launcher
• backend/Dockerfile.driver

Both Dockerfiles use:

FROM golang:1.24-alpine AS builder
FROM alpine:3.21

Trivy detects the binaries as built with Go stdlib v1.24.13.

Vulnerability Correlation

Component	Detected version	Trivy fixed version	Count / severity	Notes
Go stdlib	v1.24.13	1.25.8+, 1.25.9+, 1.25.10+, 1.26.1+, 1.26.2+, 1.26.3+ depending on CVE	17 findings	Requires rebuilding both binaries with a fixed Go toolchain. Some findings require 1.26.3, so golang:1.26.3-alpine or newer is needed.
google.golang.org/grpc	v1.73.0	1.79.3	1 critical	Master already has 1.79.3; PR #13308 currently moves master to 1.80.0 transitively.
github.com/argoproj/argo-workflows/v3	v3.7.8	3.7.11, 3.7.14	5 high	Open Dependabot PR #13308 updates master from 3.7.11 to 3.7.14.
github.com/aws/aws-sdk-go-v2/service/s3	v1.58.3	1.97.3	1 medium	Open Dependabot PR #13235 updates master from 1.96.4 to 1.97.3.
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream	v1.6.3	1.7.8	1 medium	Open Dependabot PR #13234 updates master from 1.7.6 to 1.7.8; PR #13235 also carries this update transitively.
github.com/go-jose/go-jose/v4	v4.1.0	4.1.4	1 high	Open Dependabot PR #13200 updates master from 4.1.3 to 4.1.4.
github.com/moby/spdystream	v0.5.0	0.5.1	1 high	Open Dependabot PR #13285 updates master from 0.5.0 to 0.5.1.
OpenTelemetry modules	go.opentelemetry.io/otel v1.40.0, go.opentelemetry.io/otel/sdk v1.40.0, go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.36.0	1.41.0, 1.43.0	3 findings	Open Dependabot PRs #13241 and #13240 update master to 1.43.0.

Current Master PRs

Relevant open Dependabot PRs currently target master:

• chore(deps): bump github.com/argoproj/argo-workflows/v3 from 3.7.11 to 3.7.14 #13308
• chore(deps): bump github.com/moby/spdystream from 0.5.0 to 0.5.1 #13285
• chore(deps): bump go.opentelemetry.io/otel/sdk from 1.41.0 to 1.43.0 #13241
• chore(deps): bump go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp from 1.36.0 to 1.43.0 #13240
• chore(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.96.4 to 1.97.3 #13235
• chore(deps): bump github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream from 1.7.6 to 1.7.8 #13234
• chore(deps): bump github.com/go-jose/go-jose/v4 from 4.1.3 to 4.1.4 #13200

Review status as of 2026-05-29:

• All seven PRs are dependency-only go.mod / go.sum changes.
• All seven have /ok-to-test and /lgtm.
• chore(deps): bump go.opentelemetry.io/otel/sdk from 1.41.0 to 1.43.0 #13241, chore(deps): bump go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp from 1.36.0 to 1.43.0 #13240, chore(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.96.4 to 1.97.3 #13235, and chore(deps): bump github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream from 1.7.6 to 1.7.8 #13234 have ci-passed.
• chore(deps): bump github.com/argoproj/argo-workflows/v3 from 3.7.11 to 3.7.14 #13308 and chore(deps): bump github.com/moby/spdystream from 0.5.0 to 0.5.1 #13285 have stale red E2E jobs whose logs are no longer available from GitHub.
• chore(deps): bump github.com/go-jose/go-jose/v4 from 4.1.3 to 4.1.4 #13200 has stale red E2E jobs; accessible logs point to pod startup / PodInitializing / init-container 137 handling rather than module resolution, compile, or image build failures.

There were no open Dependabot PRs targeting release-2.16 during triage. The only open PR against release-2.16 was #13069 for multi-arch build workflow changes.

Proposed Remediation

Master / Main First

• Merge the relevant master Dependabot PRs listed above after CI is current enough for maintainers to be comfortable.
• Re-run or refresh CI for PRs with stale red E2E signal: chore(deps): bump github.com/argoproj/argo-workflows/v3 from 3.7.11 to 3.7.14 #13308, chore(deps): bump github.com/moby/spdystream from 0.5.0 to 0.5.1 #13285, and chore(deps): bump github.com/go-jose/go-jose/v4 from 4.1.3 to 4.1.4 #13200.
• Add and merge a master change for the kfp-launcher and kfp-driver Go builder image so both are rebuilt with a fixed Go toolchain, at least golang:1.26.3-alpine or newer.
• Rebuild the updated master images and run Trivy against them to confirm the Go binary findings are addressed.
• Investigate whether kfp-launcher and kfp-driver should move from Alpine runtime images to distroless/static runtime images to reduce future OS package exposure and image surface area.
• Improve dependency bump automation for these images so vulnerable Go modules and Go toolchain updates are detected and proposed consistently.

Consider 2.16.2 After Master Is Fixed

• Once the master fixes are merged and verified, decide whether the same changes should be released as 2.16.2.
• If 2.16.2 is warranted, backport or recreate the necessary dependency and Go toolchain changes on release-2.16.
• Rebuild and publish patched kfp-launcher and kfp-driver images for the 2.16 line.
• Re-run Trivy on the patched 2.16 images before release and attach the results to this issue.

Expected Outcome

Master should first have updated dependency and Go toolchain fixes for kfp-launcher and kfp-driver, with Trivy confirming the Go binary vulnerabilities are addressed. After that, maintainers can decide whether to publish the same remediation in a 2.16.2 patch release.