chore(ci): support argo workflows 4.x

[jeffspahr]

Summary

• Promote the bundled/default Argo Workflows dependency to v4.0.5 across manifests, backend defaults, marketplace templates, install scripts, and docs.
• Migrate backend Argo imports to github.com/argoproj/argo-workflows/v4 and update v4 API call sites for workflow validation and compression handling.
• Keep Argo 4 CI coverage current by moving v4 lanes to v4.0.5 and widening the support-matrix freshness script to read both E2E and API server workflow matrices.
• Drop EOL Argo v3.5/v3.6 compatibility lanes from the declared support matrix and keep the legacy line on the latest supported v3.7 patch, v3.7.14.

Testing

• go test ./backend/src/common/util ./backend/src/apiserver/client ./backend/src/apiserver/template ./backend/src/v2/compiler/argocompiler ./backend/src/crd/controller/scheduledworkflow/... ./backend/src/agent/persistence/...
• go test $(go list ./backend/... | grep -v backend/test/v2/api | grep -v backend/test/integration | grep -v backend/test/v2/integration | grep -v backend/test/initialization | grep -v backend/test/v2/initialization | grep -v backend/test/compiler | grep -v backend/test/end2end) passed on rerun after one transient backend/src/v2/component log-artifact read failure.
• ./bin/ginkgo -v ./backend/test/compiler passed: 299 passed, 4 skipped, 0 failed. The locally installed CLI emitted a version mismatch warning because make ginkgo installed v2.29.0 while the repo imports v2.27.2.
• PYTHONPATH=/tmp/kfp-argo-matrix-pkgs python3 .github/scripts/verify-argo-matrix.py
• ruby -e "require 'yaml'; YAML.load_file('.github/workflows/e2e-test.yml'); YAML.load_file('.github/workflows/api-server-tests.yml'); puts 'workflow YAML parsed'"
• kubectl kustomize for manifests/kustomize/third-party/argo/installs/{cluster,namespace,namespace/cluster-scoped}
• git diff --check
• stale-reference scan for v4.0.4, Argo v3 imports/images, v3 ARGO_VERSION references, and removed v3.5/v3.6 matrix references

[google-oss-prow]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign droctothorpe for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jeffspahr]

/ok-to-test

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Updates Kubeflow Pipelines’ Argo Workflows dependency and all related references (manifests, images, backend Go modules, and CI matrices) to Argo Workflows v4.0.5.

Changes:

• Bump Argo Workflows to v4.0.5 and update Go module imports from argo-workflows/v3 to argo-workflows/v4.
• Update Kubernetes manifests/helm templates and hosted image tagging scripts to use Argo v4.0.5 controller/executor images.
• Refresh CI workflow Argo version matrices and enhance the verification script to parse multiple workflow files.

Reviewed changes

Copilot reviewed 71 out of 72 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
third_party/argo/VERSION	Bumps tracked Argo version to v4.0.5.
third_party/argo/UPGRADE.md	Updates upgrade instructions and workflow references for the new version/matrix structure.
third_party/argo/Makefile	Updates backend version-bump automation to handle Argo module major versions.
test/tag_for_hosted.sh	Updates hosted image tagging to Argo v4.0.5 images.
test/install-argo-cli.sh	Updates the pinned/commented Argo CLI version reference.
manifests/kustomize/third-party/argo/installs/namespace/kustomization.yaml	Updates remote Argo manifest ref to v4.0.5.
manifests/kustomize/third-party/argo/installs/namespace/cluster-scoped/kustomization.yaml	Updates CRD ref to v4.0.5.
manifests/kustomize/third-party/argo/installs/cluster/kustomization.yaml	Updates cluster install refs to v4.0.5.
manifests/kustomize/third-party/argo/base/workflow-controller-deployment-patch.yaml	Updates controller and executor images to v4.0.5.
manifests/kustomize/third-party/argo/base/workflow-controller-configmap-patch.yaml	Updates reference links to v4.0.5 docs/source.
manifests/kustomize/third-party/argo/base/kustomization.yaml	Updates base workflow-controller manifest ref to v4.0.5.
manifests/gcp_marketplace/chart/kubeflow-pipelines/templates/argo.yaml	Updates reference links to Argo v4.0.5 docs/source.
go.sum	Updates dependency checksums to reflect Argo v4 and related transitive updates.
go.mod	Bumps argo-workflows to v4.0.5 and refreshes related dependencies.
backend/test/end2end/utils/e2e_utils.go	Updates Argo Workflow API import path to v4.
backend/test/compiler/utils/workflow_utils.go	Updates Argo Workflow API import path to v4.
backend/test/compiler/matchers/workflow_matcher.go	Updates Argo Workflow API import path to v4.
backend/test/compiler/argo_ginkgo_test.go	Updates Argo Workflow API import path to v4.
backend/src/v2/compiler/argocompiler/spec_patch_test.go	Updates Argo Workflow API import path to v4.
backend/src/v2/compiler/argocompiler/spec_patch.go	Updates Argo Workflow API import path to v4.
backend/src/v2/compiler/argocompiler/security_context_test.go	Updates Argo Workflow API import path to v4.
backend/src/v2/compiler/argocompiler/security_context.go	Updates Argo Workflow API import path to v4.
backend/src/v2/compiler/argocompiler/importer.go	Updates Argo Workflow API import path to v4.
backend/src/v2/compiler/argocompiler/dag_test.go	Updates Argo Workflow API import path to v4.
backend/src/v2/compiler/argocompiler/dag.go	Updates Argo Workflow API import path to v4.
backend/src/v2/compiler/argocompiler/container_test.go	Updates Argo Workflow API import path to v4.
backend/src/v2/compiler/argocompiler/container.go	Updates Argo Workflow API import path to v4.
backend/src/v2/compiler/argocompiler/common.go	Updates Argo Workflow API import path to v4.
backend/src/v2/compiler/argocompiler/argo.go	Updates Argo Workflow API import path to v4.
backend/src/crd/controller/scheduledworkflow/util/scheduled_workflow_test.go	Updates Argo Workflow API import path to v4.
backend/src/crd/controller/scheduledworkflow/util/label.go	Updates Argo workflow common import path to v4.
backend/src/crd/controller/scheduledworkflow/controller.go	Updates Argo Workflow API import path to v4.
backend/src/crd/controller/scheduledworkflow/client/workflow_client_test.go	Updates Argo APIs/common imports to v4.
backend/src/common/util/workflow_test.go	Updates Argo APIs/client/informer/lister imports to v4.
backend/src/common/util/workflow.go	Updates Argo imports to v4 and adapts to new context-aware APIs.
backend/src/common/util/scheduled_workflow_test.go	Updates Argo Workflow API import path to v4.
backend/src/common/util/pointer_test.go	Updates Argo Workflow API import path to v4.
backend/src/common/util/pointer.go	Updates Argo Workflow API import path to v4.
backend/src/common/util/formatter_test.go	Updates Argo Workflow API import path to v4.
backend/src/common/util/formatter.go	Updates Argo Workflow API import path to v4.
backend/src/common/util/execution_spec_test.go	Updates Argo Workflow API import path to v4.
backend/src/common/util/execution_spec.go	Updates Argo Workflow API import path to v4.
backend/src/common/util/execution_client.go	Updates Argo client/informer imports to v4.
backend/src/common/types.go	Updates Argo pkg.go.dev reference URL to v4.0.5.
backend/src/common/client/api_server/v2/pipeline_client_fake.go	Updates Argo Workflow API import path to v4.
backend/src/common/client/api_server/v1/run_client_fake.go	Updates Argo Workflow API import path to v4.
backend/src/common/client/api_server/v1/run_client.go	Updates Argo Workflow API import path to v4.
backend/src/common/client/api_server/v1/pipeline_client_fake.go	Updates Argo Workflow API import path to v4.
backend/src/apiserver/template/template_test.go	Updates Argo Workflow API import path to v4.
backend/src/apiserver/template/argo_template.go	Updates Argo imports to v4 and adapts validation to context-aware API.
backend/src/apiserver/server/run_server_test.go	Updates Argo Workflow API import path to v4.
backend/src/apiserver/server/run_artifact_server_test.go	Updates Argo Workflow API import path to v4.
backend/src/apiserver/server/report_server_test.go	Updates Argo Workflow API import path to v4.
backend/src/apiserver/server/fakes_test.go	Updates Argo Workflow API import path to v4.
backend/src/apiserver/server/api_converter_test.go	Updates Argo Workflow API import path to v4.
backend/src/apiserver/resource/resource_manager_test.go	Updates Argo imports and adapts compression helper to context-aware API.
backend/src/apiserver/client/workflow_fake_test.go	Updates Argo Workflow API import path to v4.
backend/src/apiserver/client/workflow_fake.go	Updates Argo Workflow API import path to v4.
backend/src/apiserver/client/argo_fake_test.go	Updates Argo Workflow API import path to v4.
backend/src/apiserver/client/argo.go	Updates Argo clientset imports to v4.
backend/src/apiserver/archive/log_test.go	Updates Argo Workflow API import path to v4.
backend/src/agent/persistence/worker/workflow_saver_test.go	Updates Argo Workflow API import path to v4.
backend/src/agent/persistence/worker/persistence_worker_test.go	Updates Argo Workflow API import path to v4.
backend/src/agent/persistence/worker/metrics_reporter_test.go	Updates Argo Workflow API import path to v4.
backend/src/agent/persistence/persistence_agent.go	Updates workflow API registration import to v4.
backend/api/Dockerfile	Updates generator tool versions (grpc-gateway/grpc).
backend/Dockerfile	Updates ARGO_VERSION to v4.0.5 for CLI download.
README.md	Updates Argo compatibility matrix minors to match supported set.
AGENTS.md	Updates “Last updated” and Argo compatibility matrix text.
.github/workflows/e2e-test.yml	Updates CI Argo matrix versions (drops v3.5, updates v3.7/v4.0 patch levels).
.github/workflows/api-server-tests.yml	Updates CI Argo matrix versions (drops v3.5, updates v3.7/v4.0 patch levels).
.github/scripts/verify-argo-matrix.py	Extends verification to parse multiple workflows and robustly find argo_version.

Comments suppressed due to low confidence (2)

third_party/argo/UPGRADE.md:1

• The suggested grep invocation is ordered incorrectly: -R placed after the file glob is interpreted as a filename, not an option, so it won’t perform a recursive search. Update the example to put options before the pattern and use a directory target (e.g., repo root) so the command behaves as intended.
  backend/src/common/util/workflow.go:1
• argoContext() allocates a new logger and context on every call (used by Validate/Decompress). If these utilities are called frequently, this adds avoidable overhead. Consider caching a single context (package-level var and/or sync.Once) and reusing it, or accepting an existing context as a parameter so callers can control reuse.

[jeffspahr]

/retest

[jeffspahr]

/retest