Repository navigation
feat(eks): run the runtime-plugin sandbox pool on EKS - #2532
Merged
Merged
Conversation
… EKS Runtime-plugin worker Jobs select node-restriction.kubernetes.io/shifter-pool= runtime-plugin under gVisor. NodeRestriction forbids a kubelet from setting that label, and EKS managed node groups apply labels through the kubelet, so the EKS pool could not join and stayed disabled; range launch for any adapter-bound pack then timed out waiting for plugin plans. - The pool gets its own node role and no kubelet-set labels. - shared.cloud.aws.node_pool_labeler labels a node only when its EC2 instance is running, its private DNS name is the node's authenticated name, and its instance profile carries the pool's node role; it removes the label from any node that fails the check and leaves nodes alone on provider errors. - The chart runs it (AWS only) with an IRSA identity limited to ec2:DescribeInstances and iam:GetInstanceProfile on EKS-created profiles, a nodes-only ClusterRole, Kubernetes-API-only egress, and an admission policy that lets its identity toggle the pool label and nothing else. - The AWS bootstrap renders it from the nodePoolLabeler identity and the pool's node role, failing when only one is present; aws-dev enables the pool. Refs #2526
The labeler is a standalone EKS controller, not a cloud adapter with a GCP counterpart (ADR-005-R1 cloud-factory seam): it lives beside the other standalone cloud programs as shared.cloud.eks_node_pool_labeler. Refs #2526
…lients SonarCloud flagged the labeler's Any-typed client parameters and a five-return lookup. Small Protocols and TypedDicts describe exactly the EC2, IAM and Node fields it reads, and the instance lookup is split so each function has a single decision. Refs #2526
…ts boto3 protocol Refs #2526
The labeler was OOMKilled at its 128Mi limit on aws-dev: the platform package, boto3 and the Kubernetes client take about 180 MiB resident before the first reconcile. Request 192Mi and allow 320Mi. Refs #2526
The node-pool labeler resolved each node's instance profile to its role, but the CI permissions boundary denies iam:GetInstanceProfile, so every check deferred and no node was ever labeled. Pool membership is now the AWS-owned aws:autoscaling:groupName tag matching the runtime-plugin node group's ASG, together with a running state and a private DNS name equal to the node name. The labeler's IAM grant shrinks to ec2:DescribeInstances.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runtime-plugin worker Jobs select
node-restriction.kubernetes.io/shifter-pool=runtime-plugin. NodeRestriction stops a kubelet from setting that label, which is what makes it a trustworthy pool identity. GKE applies it through its control plane. EKS managed node groups apply labels through the kubelet, so the pool could never be populated on EKS.shared.cloud.eks_node_pool_labeler) labels a node only when its EC2 instance is running, its private DNS name equals the node name, and its AWS-ownedaws:autoscaling:groupNametag names the pool's Auto Scaling group. A node that fails the check loses the label; a provider error leaves the node unchanged. Its IRSA role can only callec2:DescribeInstances. (An earlier instance-profile check was dropped because the CI permissions boundary deniesiam:GetInstanceProfile.)Closes #2526
Verification
terraform test -filter=tests/security.tftest.hcl(3).ip-10-80-20-57joined Ready and is the only node withshifter-pool=runtime-plugin; the labeler loggednode pool label applied.ip-10-80-20-57and ran sandboxed (dmesg:Starting gVisor..., kernel4.19.0-gvisor).