feat!: switch http node port to 81 to support pod identity on port 80

[josmo]

We'll need to move away from nginx in general - however for now we need to start switching the node ports away from 80 and to 81 (or another port?) so we can add pod identity which runs on port 80 - aws/eks-pod-identity-agent#10

manual steps for existing alb nginx ingress - configs

1. delete the apiVersion: elbv2.k8s.aws/v1beta1 kind: TargetGroupBinding for the existing port (should validate this is actually needed? might just need the other two steps and will self heal)
2. delete the listener from the lb balancer for port 80
3. delete the target group for port 80

everything should self heal after a few minutes

[iveelsm]

Do we need the advertised proxy port? It seems there is a recent release that allows for removing that binding (aws/eks-pod-identity-agent#107)

[josmo]

Do we need the advertised proxy port? It seems there is a recent release that allows for removing that binding (aws/eks-pod-identity-agent#107)

hum, that's relatively new, but I just tried to find that and didn't initially - good find, we'll have to see what that looks like to deploy (we might be in environments where changing the pod-identity config is harder than the ingress, but I do like turning it off for ours at least)

[josmo]

ok tested that out and it doesn't work without the cni chagnes - which starts add more differences - it only removed the port from the container to "hide" the scheduling from eks. However since nginx binds on 0.0.0.0:80 and the identity agent binds on the 169.x.x.x:80 it still conflicts and then one crashes without the cni changes - for now I think the ingress changes are easier to track and reason about

[josmo]

This will have manual steps required to do the switch. I'll write them down and we'll need to do a major version switch