Bump Prow to v20260225, fix Spyglass S3 URI error, add smart deploy restarts#60
Merged
Conversation
m-chau
force-pushed
the
bump-prow-images-v20260328
branch
from
March 31, 2026 03:11
6e48f7d to
9446fca
Compare
m-chau
changed the title
m-chau
force-pushed
the
bump-prow-images-v20260328
branch
from
March 31, 2026 03:14
9446fca to
51d1abc
Compare
m-chau
changed the title
m-chau
force-pushed
the
bump-prow-images-v20260328
branch
from
March 31, 2026 03:28
51d1abc to
dcd0b75
Compare
m-chau
changed the title
Match upstream prow.k8s.io production versions: - Update K8S_PROW_IMAGE_TAG from v20240802-66b115076 to v20260225-8287579aa - Fix ghproxy image registry from gcr.io/k8s-prow (legacy) to us-docker.pkg.dev/k8s-infra-prow/images (current) This applies to all Prow components (deck, hook, tide, crier, horologium, sinker, prow-controller-manager, status-reconciler, ghproxy) and utility images (clonerefs, entrypoint, initupload, sidecar). Upstream production version confirmed via deck's static asset version tag: curl -s https://prow.k8s.io/ | grep -o 'v=[a-z0-9-]*' | head -1 # v=v20260225-8287579aa Note: upstream's git still shows v20240802 in cluster deployment YAMLs, but production was manually deployed with a newer version.
m-chau
force-pushed
the
bump-prow-images-v20260328
branch
from
March 31, 2026 03:37
dcd0b75 to
d244a27
Compare
Newer Prow (v20260225) uses an updated AWS SDK that requires a full URI with scheme for custom endpoints. Without it, Spyglass fails with: 'Custom endpoint minio.prow.svc.cluster.local was not a valid URI'
Both s3-credentials secrets (prow and test-pods namespaces) need the http:// scheme. Previous commit only fixed the prow namespace one.
added 2 commits
March 31, 2026 04:12
Prow components (deck, crier) read s3-credentials at startup and don't watch for changes. After applying updated secrets (e.g. endpoint with http:// scheme), pods must be restarted to pick up the new values. This fixes the 'Custom endpoint was not a valid URI' Spyglass error seen after the Prow image bump.
Snapshot s3-credentials checksums before and after applying manifests. Only trigger rollout restart if the secret data changed, avoiding unnecessary pod restarts on idempotent deploys.
The before/after checksum approach missed cases where the secret was updated in a prior deploy but pods were never restarted. Instead, store the secret hash as a pod template annotation — if the current secret hash doesn't match the annotation, patch the deployment (which triggers a rolling restart). If they match, no restart occurs. This handles both fresh changes and stale pods from prior deploys.
added 2 commits
March 31, 2026 04:46
Hook was missing --plugin-config=/etc/plugins/plugins.yaml in its args. The volume and mount were already present, but without the flag the new Prow image doesn't load plugins (including trigger, which handles /test commands). The old image had this as a default path.
This reverts commit f363c26.
m-chau
changed the title
m-chau
changed the title
wenhug
approved these changes
Apr 1, 2026
Collaborator
|
/lgtm |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Prow image bump
v20260225-8287579aa(matching upstream production)S3 endpoint fix
http://prefix to the MinIO endpoint ins3-credentialssecrets (bothprowandtest-podsnamespaces)Custom endpoint 'minio.prow.svc.cluster.local' was not a valid URIDeploy workflow: smart secret-aware restarts
s3-credentialsat startup and don't watch for changess3-credentialssecret and compares it to an annotation on the deployment's pod templateTesting
/test alltriggers Prow jobs on Test v1.32 PROW update kubernetes#83