Breaking changes in latest package bundles for credential-provider-package

[czomo]

What happened:
Any version of package bundle above v1-27-128 are not usable because of multiple issues. Should we keep those faulty packages in registry? Is there any end2end test that could detect that in future?

v1-27-137	other	7 days ago	 public.ecr.aws/eks-anywhere/...es-bundles:v1-27-137	60.8 KB > doesn't work, image of anywhere-package controller works fine, see log_1
v1-27-134	other	14 days ago	 public.ecr.aws/eks-anywhere/...es-bundles:v1-27-134	60.7 KB > faulty secret, propably because of helm chart
v1-27-130	other	17 days ago	 public.ecr.aws/eks-anywhere/...es-bundles:v1-27-130	60.7 KB > wrong helm app version which causing imagepullbackoff for package controler and refresher, see log_2
v1-27-129	other	17 days ago	 public.ecr.aws/eks-anywhere/...es-bundles:v1-27-129	60.7 KB > wrong helm app version which causing imagepullbackoff for package controler and refresher, see log_2
v1-27-128	other	2 months ago	 public.ecr.aws/eks-anywhere/...es-bundles:v1-27-128 > works fine

log_1

2023-11-02T10:52:24.780Z    ECRCredInjector    Failed to inject ECR credential to docker config    {"error": "operation error ECR: GetAuthorizationToken, failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, static credentials are empty"}
github.com/aws/eks-anywhere-packages/pkg/registry.(*ECRCredInjector).Run
    github.com/aws/eks-anywhere-packages/pkg/registry/ecr_cred_injector.go:56

log_2

eksa-packages  eks-anywhere-packages             8        failed    eks-anywhere-packages-0.0.0-8862036270224f2a6b8d6ecd455b6b1fa1084619              v0.0.0-8862036270224f2a6b8d6ecd455b6b1fa1084619    

What you expected to happen:
eks-anywhere-packages shouldn't be published with such breaking changes
How to reproduce it (as minimally and precisely as possible):

1. Using 0.17.4 eks-anywhere install k8s 1.27 using tinkerbell provider
2. Create eks-anywhere role along with anchor, follow https://anywhere.eks.amazonaws.com/docs/packages/credential-provider-package/iam_roles_anywhere/#prerequisites
3. Create aws-config secret in eks-packages ns

  [default]
  region = eu-west-1
  credential_process = aws_signing_helper credential-process --certificate /var/lib/kubelet/pki/kubelet-client-current.pem --private-key /var/lib/kubelet/pki/kubelet-client-current.pem --profile-arn $PROFILE_ARN --role-arn $ROLE_ARN --trust-anchor-arn $TRUST_ANCHOR_ARN

4. Add package to download from private ECR registry

apiVersion: packages.eks.amazonaws.com/v1alpha1
kind: Package
metadata:
  name: my-credential-provider-package
  namespace: eksa-packages-eksa
  annotations:
    "helm.sh/resource-policy": keep
    "anywhere.eks.aws.com/internal": "true"
spec:
  packageName: credential-provider-package
  targetNamespace: eksa-packages
  config: |-
    tolerations:
      - key: "node-role.kubernetes.io/master"
        operator: "Exists"
        effect: "NoSchedule"
      - key: "node-role.kubernetes.io/control-plane"
        operator: "Exists"
        effect: "NoSchedule"
    sourceRegistry: public.ecr.aws/eks-anywhere
    credential:
      - matchImages:
        - 000000000000.dkr.ecr.eu-west-2.amazonaws.com
        profile: "default"
        secretName: aws-config
        defaultCacheDuration: "12h"

5. Verify you have latest version of packagebundle in packagebundlecontroller
6. Create pod with image from 000000000000.dkr.ecr.eu-west-2.amazonaws.com registry
7. ImagePullBackOff should be logged from kubelet

Anything else we need to know?:
We also checked latest v1-28 with k8s 1.28 and it also experience issues the same as v1-27-137

Environment: k8s 1.27, tinker provisioner with bare bone nodes, ubuntu 22.04 ami

• EKS Anywhere Release: 0.17.4
• EKS Distro Release: -

[chrisdoherty4]

Thanks for the report @czomo. We're aware of the problem and will fix asap.

[czomo]

@chrisdoherty4 Any update from your side? It's not working on v1-27-142 neither of any 1.28-*.

[joeto0]

Is that fixed? We are hitting same behavior with 1.28.7.

[mitalipaygude]

Can you confirm the OS you are using @joeto0 ? Is it Bottlerocket?

Also, @czomo can you confirm the OS you are using as well? Its Ubuntu right?

[joeto0]

yes, bottlerocket.

…

On Wed, Apr 24, 2024 at 6:38 PM Mitali Paygude ***@***.***> wrote: Can you confirm the OS you are using @joeto0 <https://github.com/joeto0> ? Is it Bottlerocket? Also, @czomo <https://github.com/czomo> can you confirm the OS you are using as well? Its Ubuntu right? — Reply to this email directly, view it on GitHub <#1024 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC4HXO5BOEOOISNGQTGLBETY7AX6PAVCNFSM6AAAAAA64K3QD2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZVHE3DGOBXGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[czomo]

Also, @czomo can you confirm the OS you are using as well? Its Ubuntu right?

yes, Ubuntu
we find out that in our case faulty was the host Path, as for now we stopped using packages in favour of predefined daemonset
'''
- hostPath:
path: /etc/sysconfig/kubelet
type: FileOrCreate
'''