chore(testing): decide Kubernetes version support policy for CI

[jeffspahr]

Chore description

We should make an explicit maintainer decision on how kubeflow/pipelines chooses Kubernetes versions for CI and implied support.

Today the repo mostly follows a low/high spot-check policy in CI, but that policy is not documented clearly and recent version-rotation work showed that the decision criteria are ambiguous:

• upstream Kubernetes support has moved
• major cloud providers still extend support for older minors
• Kind's latest release does not cover both ends of that space

This issue is meant to capture the policy decision for future work. It is not proposing that we broaden the CI matrix right now.

Related prior threads:

• Closed PR: chore(ci): rotate supported Kubernetes versions #12978
• Narrower open issue: Drop K8s v1.29.2 from CI test matrix (EOL since Feb 2025) #13096

Current pipelines state

kubeflow/pipelines currently tests a low/high pair rather than a contiguous rolling range. The workflows still have many references to v1.29.2 and v1.34.0 in files such as:

• .github/workflows/api-server-tests.yml
• .github/workflows/e2e-test.yml
• .github/workflows/e2e-test-frontend.yml
• .github/workflows/integration-tests-v1.yml
• .github/workflows/kfp-kubernetes-native-migration-tests.yaml
• .github/workflows/kfp-sdk-client-tests.yml
• .github/workflows/kfp-webhooks.yml

There is also an internal CI note in AGENTS.md, but it is stale and not a public support statement.

How other Kubeflow repos are operating

Other Kubeflow component repos are not following the exact same pattern:

• kubeflow/katib currently runs e2e CI on a rolling multi-version range: v1.31.3, v1.32.2, v1.33.1, v1.34.0.
• kubeflow/trainer currently runs CPU e2e CI on 1.32.3, 1.33.1, 1.34.0, 1.35.0.
• kubeflow/manifests publishes a platform-level validated floor per release rather than a low/high CI spot-check policy. The Kubeflow AI reference platform 26.03 release documents Kubernetes 1.34+ as validated.

That does not mean pipelines should adopt a broader matrix right now, but it does mean the current pipelines low/high policy is a repo-specific choice and should be explicit.

External support context as of 2026-05-31

Upstream Kubernetes

Upstream supported minors are currently 1.36, 1.35, and 1.34.

• 1.36.1 latest release
• 1.35.5 latest release
• 1.34.8 latest release
• 1.33 reaches EOL on 2026-06-28

Source:

• https://kubernetes.io/releases/

Major cloud providers

The oldest version still covered by major cloud-provider extended support (or the closest equivalent) is effectively 1.30 right now:

• Amazon EKS extended support currently includes 1.32, 1.31, and 1.30.
  • https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html
• AKS LTS currently includes 1.30 through 2026-07, 1.31 through 2026-11, and newer minors after that. 1.29 LTS ended in 2026-04.
  • https://learn.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli
• GKE Extended still covers 1.30 until 2026-07-30 and 1.31 until 2026-10-22.
  • https://cloud.google.com/kubernetes-engine/docs/release-schedule

So if we want the low end to reflect the oldest version still covered by major managed Kubernetes providers, the answer is currently 1.30, not 1.31+.

Kind limitation as of 2026-05-31

Our CI cluster creation path still uses helm/kind-action@v1.13.0, which maps to Kind v0.31.0.

The latest Kind release (v0.31.0) currently ships prebuilt node images for:

• v1.35.0
• v1.34.3
• v1.33.7
• v1.32.11
• v1.31.14

Source:

• https://github.com/kubernetes-sigs/kind/releases/tag/v0.31.0

This creates two important constraints:

1. Latest Kind does not yet cover the latest upstream Kubernetes release.
   • Upstream latest is 1.36.x, but latest Kind prebuilt images stop at 1.35.0.
2. Latest Kind also does not cover the low end of major cloud-provider extended support.
   • Cloud-provider low end is effectively 1.30, but latest Kind prebuilt images start at 1.31.14.

So even if we know what the policy should be, the current Kind release prevents the repo from validating both ends with release-matched prebuilt images.

Questions for maintainers

1. Should kubeflow/pipelines continue the current low/high spot-check CI policy, or should it intentionally move to a broader rolling range later?
2. What should define the low end for pipelines?
   • Upstream Kubernetes support only?
   • Oldest version still covered by major cloud-provider extended support?
   • Kubeflow manifests validated floor?
   • Something else?
3. What should define the high end?
   • Latest upstream supported minor?
   • Latest Kubernetes version with a release-matched Kind prebuilt image?
   • Latest version we can validate another way in CI?
4. When Kind lags both the upstream latest and the cloud-provider low end, what should the repo do?
   • Stay within latest Kind prebuilt images
   • Bump Kind more aggressively when available
   • Build custom node images or use non-prebuilt images
   • Keep support policy separate from CI validation policy and document the difference explicitly

Suggested outcome

The main goal here is to make future version-rotation PRs mechanical instead of policy debates.

A useful resolution would be a short documented rule in the repo that says:

• whether pipelines intentionally uses low/high spot checks or a broader rolling range
• what source of truth determines the low and high Kubernetes versions
• how Kind release limitations should be handled when they conflict with that policy

[jeffspahr]

One follow-up thought on implementation mechanics once the policy is decided:

It would be useful to mirror the existing Argo maintenance pattern in this repo so future Kubernetes-version rotations are less manual.

Concretely, after maintainers decide the policy in this issue, a follow-up could:

1. Add a small Kubernetes version rotation / upgrade doc, analogous to third_party/argo/UPGRADE.md.

   • This would document how to choose the low/high CI-tested pair, which workflow files to update, what Kind constraints to check, and what verification to run.

2. Introduce a single source of truth for the current CI-tested Kubernetes pair.

   • For example, a small YAML/JSON file under .github/ that records low, high, and the pinned Kind release.
   • Workflow/doc updates could then be generated or validated from that file rather than edited ad hoc.

3. Add a freshness check similar to .github/scripts/verify-argo-matrix.py + .github/workflows/docs-freshness.yml.

   • That check could verify that workflow matrices and docs stay aligned with the chosen source of truth.

4. Decide whether the repo should eventually expose a public README statement about Kubernetes compatibility.

   • Right now the repo has a public Argo compatibility matrix in README.md, but Kubernetes support is mostly implied by CI.
   • It may make sense to add a public statement only after the policy here is settled.

I would treat these as follow-on implementation tasks after the policy decision, not as blockers for the current version-rotation PR.