K8TRE - base infrastructure options: Networking

[manics]

All Kubernetes distributions include some sort of default Container Network Interface (CNI) provider, but they may not support all features required by K8TRE. For example, some have only partial support for Network Policies. We should define the capabilities that a CNI must provide for K8TRE, and examples of how that can be configured on our official supported platforms.

[manics]

AWS:

• The default CNI is the Amazon VPC CNI plugin which assigns IPs from the VPC. It doesn't have full support for the K8S Network Policy specification, e.g. see https://github.com/jupyterhub/mybinder.org-deploy/pull/2832/files
• Calico is a better option which does have full support for standard Network Policies. It also has some vendor specific CRDs, though I think we should avoid those if possible.

[vvcb]

Networking, network policies and workload isolation are all intricately linked.

Having looked at Calico and Cilium in some detail, Cilium has at least one significant advantage over Calico. Layer 7 policies in Cilium are part of the free open source version while you need an enterprise license for Calico.

This allows us to control ingress/egress at a pod level which will be critical for JupyterHub workspaces.

We currently use Kubenet at Lancs but can switch to Azure CNI. The main difference is that the VNET where the cluster is hosted should have a sufficiently large address space when using Azure CNI or AWS VPC CNI as every pod (and service and node) draws an IP address from this VNET. In peered networks, this could potentially result in IP address exhaustion (a problem another project ran into in the region).

[manics]

We should be clear on whether the K8TRE CIDR is solely for in-cluster use only so all external access is via the ingress/load-balancer (this is what I think we should do), or whether applications/services outside the cluster also have access to the CIDR/VPC/VNET.

[t-young31]

+1 for Cilium. ARC TRE and FRIDGE are using it