Skip to content

add helm.sh/hook annotation to resolve race condition#1281

Open
peirulu wants to merge 3 commits into
aws:mainfrom
peirulu:fix-pbc-race
Open

add helm.sh/hook annotation to resolve race condition#1281
peirulu wants to merge 3 commits into
aws:mainfrom
peirulu:fix-pbc-race

Conversation

@peirulu
Copy link
Copy Markdown
Member

@peirulu peirulu commented Apr 1, 2026

Problem Statement:
After bumping to helm version v4+, the package workflow with registry mirror pull-through-cache configuration and pure helm install will encounter issues.

Helm v4.1.0

helm version
version.BuildInfo{Version:"v4.1.0", GitCommit:"4553a0a96e5205595079b6757236cc6f969ed1b9", GitTreeState:"clean", GoVersion:"go1.25.6", KubeClientVersion:"v1.35"}

helm upgrade --install eks-anywhere-packages oci://975050071200.dkr.ecr.us-west-2.amazonaws.com:443/ecr-public/w9m0f3l5/eks-anywhere-packages --version 0.4.15-eks-a-118 --set sourceRegistry=975050071200.dkr.ecr.us-west-2.amazonaws.com:443/ecr-public/w9m0f3l5 --set defaultRegistry=public.ecr.aws/w9m0f3l5 --set defaultImageRegistry=067575901363.dkr.ecr.us-west-2.amazonaws.com --set clusterName=pull-through-cache --kubeconfig pull-through-cache/pull-through-cache-eks-a-cluster.kubeconfig --create-namespace --namespace eksa-packages -f pull-through-cache/generated/values.yaml --insecure-skip-tls-verify
Release "eks-anywhere-packages" does not exist. Installing it now.
Pulled: 975050071200.dkr.ecr.us-west-2.amazonaws.com:443/ecr-public/w9m0f3l5/eks-anywhere-packages:0.4.15-eks-a-118
Digest: sha256:86c30115075655ca1d2c909317528b21a72f7e33bfc870afdde1dc5848592ecb
Error: Internal error occurred: failed calling webhook "vpackage.kb.io": failed to call webhook: Post "[https://eks-anywhere-packages-eks-anywhere-packages-webhook-service.eksa-packages.svc:443/validate-packages-eks-amazonaws-com-v1alpha1-package?timeout=10s](https://eks-anywhere-packages-eks-anywhere-packages-webhook-service.eksa-packages.svc/validate-packages-eks-amazonaws-com-v1alpha1-package?timeout=10s)": dial tcp 10.105.47.116:443: connect: connection refused
Internal error occurred: failed calling webhook "vpackagebundlecontroller.kb.io": failed to call webhook: Post "[https://eks-anywhere-packages-eks-anywhere-packages-webhook-service.eksa-packages.svc:443/validate-packages-eks-amazonaws-com-v1alpha1-packagebundlecontroller?timeout=10s](https://eks-anywhere-packages-eks-anywhere-packages-webhook-service.eksa-packages.svc/validate-packages-eks-amazonaws-com-v1alpha1-packagebundlecontroller?timeout=10s)": dial tcp 10.105.47.116:443: connect: connection refused

Helm v3

helm upgrade --install eks-anywhere-packages oci://975050071200.dkr.ecr.us-west-2.amazonaws.com:443/ecr-public/w9m0f3l5/eks-anywhere-packages --version 0.4.15-eks-a-118 --set sourceRegistry=975050071200.dkr.ecr.us-west-2.amazonaws.com:443/ecr-public/w9m0f3l5 --set defaultRegistry=public.ecr.aws/w9m0f3l5 --set defaultImageRegistry=067575901363.dkr.ecr.us-west-2.amazonaws.com --set clusterName=pull-through-cache --kubeconfig pull-through-cache/pull-through-cache-eks-a-cluster.kubeconfig --create-namespace --namespace eksa-packages -f pull-through-cache/generated/values.yaml --insecure-skip-tls-verify
Release "eks-anywhere-packages" does not exist. Installing it now.
Pulled: 975050071200.dkr.ecr.us-west-2.amazonaws.com:443/ecr-public/w9m0f3l5/eks-anywhere-packages:0.4.15-eks-a-118
Digest: sha256:86c30115075655ca1d2c909317528b21a72f7e33bfc870afdde1dc5848592ecb
NAME: eks-anywhere-packages
LAST DEPLOYED: Thu Apr  2 19:08:11 2026
NAMESPACE: eksa-packages
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thanks for installing, for any bugs or issues please file them at https://github.com/aws/eks-anywhere-packages/issues

For more information on running eks-anywhere-packages, visit:
https://github.com/aws/eks-anywhere-packages

Description of changes:

  • Add "helm.sh/hook" to make sure package and packagebundlecontroller install after crds

Testing:

  1. Install helm version v4.1.0
  2. Create a tag v0.4.17 on my own branch containing the fix and upload the eks-anywhere-packages image to my public ecr
  3. Helm install eks-anywhere-packages
helm upgrade --install eks-anywhere-packages oci://public.ecr.aws/p2x5x2t2/eks-anywhere-packages \
--version 0.4.17-1ce58938ecd21ce10f38dffc0f014f2c3bd9f149 --set sourceRegistry=public.ecr.aws/p2x5x2t2 \
--set defaultRegistry=public.ecr.aws/p2x5x2t2 --set defaultImageRegistry=public.ecr.aws/p2x5x2t2 \
--set clusterName=redhat-134-ptc --kubeconfig redhat-134-ptc/redhat-134-ptc-eks-a-cluster.kubeconfig \
--create-namespace --namespace eksa-packages -f values.yaml --insecure-skip-tls-verify --wait --timeout 5m
Release "eks-anywhere-packages" does not exist. Installing it now.
Pulled: public.ecr.aws/p2x5x2t2/eks-anywhere-packages:0.4.17-1ce58938ecd21ce10f38dffc0f014f2c3bd9f149
Digest: sha256:2ac91b9faa475599ad67c194c0c8f40fc0d78fa52b9f0f0717be0f3ab5d56513
NAME: eks-anywhere-packages
LAST DEPLOYED: Wed Apr  8 11:55:26 2026
NAMESPACE: eksa-packages
STATUS: deployed
REVISION: 1
DESCRIPTION: Install complete
TEST SUITE: None
NOTES:
Thanks for installing, for any bugs or issues please file them at https://github.com/aws/eks-anywhere-packages/issues

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@eks-distro-bot
Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: peirulu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@eks-distro-bot eks-distro-bot added approved size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Apr 1, 2026
@eks-distro-bot eks-distro-bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Apr 6, 2026
@snarkychef
Copy link
Copy Markdown
Member

snarkychef commented Apr 9, 2026

Hook resources are not managed as part of helm release. So, this will block subsequent upgrades or reinstalls, as the hook resources will be existing causing resource already exists errors.

One workaround that might work is to use "helm.sh/hook-delete-policy": before-hook-creation annotation to delete and recreate hook resources. but this may have unintended consequences as package controller tries to reconcile deleted resources.

@eks-distro-bot eks-distro-bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Apr 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants