K8TRE - GitOps using ArgoCD

[vvcb]

K8TRE will follow GitOps principles and will use ArgoCD for managing nearly all resources on the cluster(s) it manages. The alternative to ArgoCD is FluxCD which LSC-SDE currently uses.

With ArgoCD, there are two broad approaches. ArgoCD can deploy and manage applications to the same cluster it is installed on (in-cluster) or deploy applications to an arbitrary number of clusters with fine-grained control over what applications are deployed to which cluster(s) and with what config.

When this is combined with ArgoCD generators and Kustomize overlays, we can achieve all of the following:

• Deployment to dev/test/prod clusters from a single ArgoCD installation rather than a separate ArgoCD installation for each environment each requiring configuration.
• Deployment to different cloud / on-prem clusters where we can control deployment of vendor-specific resources to specific clusters by using label selectors for instance. E.g., EKS-specific storage classes get deployed only to clusters with the EKS label set.
• This also allows organisations to extend K8TRE whereby they can either deploy one K8TRE per project or deploy ephemeral K8TRE development environments per developer.

Dev/Stg/Prod Clusters

graph LR
    subgraph Management Cluster
        A[ArgoCD Management Instance]
    end

    subgraph Dev Cluster
        B[Dev Workloads]
    end

    subgraph Staging Cluster
        C[Staging Workloads]
    end

    subgraph Production Cluster
        D[Production Workloads]
    end


    A --> B & C & D
   
    subgraph Git Repository
        E[Git Repository]
    end

    A -- "Syncs with" --> E

    class ManagementCluster cluster;

Loading

A K8TRE per project (The Turing Model)

graph LR
    subgraph Management Cluster
       A[ArgoCD Management Instance]
   end

   subgraph Production Cluster 1
       B[K8TRE-Project-01 Workloads]
   end

   subgraph Production Cluster 2
       C[K8TRE-Project-02 Workloads]
   end

   subgraph Production Cluster 3
       D[K8TRE-Project-03 Workloads]
   end

   A --> B & C & D

   subgraph Git Repository
       E[Git Repository]
   end

   A -- "Syncs with" --> E

Loading

Pros and Cons of ArgoCD on same cluster vs on separate management cluster

1. ArgoCD on the Same Cluster as Applications (Co-located)

• Pros:

  • Simplicity: Easier to set up initially. The network setup is straightforward – no need to configure cross-cluster communication.
  • Lower Latency: Communication between ArgoCD and your applications is faster due to being on the same network. This can be beneficial for real-time monitoring and synchronization.
  • Simplified Troubleshooting: Debugging issues becomes easier as you're working within the same environment.
  • Resource Efficiency (potentially): Can be more efficient in smaller environments where you're already using the cluster resources.

• Cons:

  • Single Point of Failure: If the cluster where ArgoCD runs experiences issues (outage, corruption), it impacts your application deployments. This is the biggest risk.
  • Resource Contention: ArgoCD consumes resources (CPU, memory). Sharing these resources with your applications can lead to contention and performance issues, especially as your applications scale.
  • Security Considerations: ArgoCD needs sufficient permissions to manage your applications. Granting these permissions creates a broader attack surface. If ArgoCD is compromised, so are your applications.
  • Scaling Challenges: As your application deployments grow, the cluster where ArgoCD resides will need to scale. This can become a bottleneck if you're already pushing the limits of the cluster.

2. ArgoCD on a Separate Cluster (Federated/Multi-Cluster)

• Pros:

  • Increased Resilience: Isolates ArgoCD from application failures. If a cluster goes down, ArgoCD can continue managing deployments to other clusters. This is a significant benefit for production environments.
  • Enhanced Security: Limits the blast radius if ArgoCD is compromised. An attacker gaining control of ArgoCD will only have access to deployments in that cluster, not others.
  • Improved Resource Management: Allows you to dedicate resources specifically for ArgoCD, preventing resource contention with your applications.
  • Multi-Cluster Deployments: This is the primary reason to do this. You can deploy to multiple Kubernetes clusters (e.g., on-prem, cloud, edge) from a central ArgoCD instance. This is powerful for geo-distribution, disaster recovery, and hybrid cloud strategies.
  • Easier Scaling: You can scale the ArgoCD cluster independently of your application clusters.

• Cons:

  • Increased Complexity: Setting up and managing a federated ArgoCD setup is more complex. It requires configuring cross-cluster communication between the management cluster and the deployment clusters. However there are standard patterns for achieving this using hub-spoke networking.
  • Higher Cost (potentially): Running a separate cluster for ArgoCD incurs additional costs. ___We can potentially mitigate this by using vCluster but the cost of running a small management cluster should be trivial in comparison with the cost of the overall infrastructure - both k8s and non-k8s.
  • Latency: Cross-cluster communication introduces some latency, which could affect real-time synchronization, although this is often negligible.

Recommendations Based on Scenario:

• Development/Testing: For development and testing environments, it's often fine to run ArgoCD on the same cluster as your applications. The simplicity outweighs the risks.
• Small Production Environments: If you have a small production environment with limited resources and a low tolerance for complexity, you might consider co-locating. However,
  carefully monitor resource usage and plan for redundancy.
• Production Environments (Generally Recommended - Separate Cluster): For production environments, especially those with critical applications, running ArgoCD on a separate cluster
  is strongly recommended. The increased resilience, security, and scalability are worth the added complexity. Consider using a service mesh (like Istio) to simplify cross-cluster
  communication.
• Hybrid or Multi-Cloud: If you're deploying to multiple Kubernetes clusters (e.g., on-prem, cloud providers), running ArgoCD on a central cluster is essential.

[manics]

That sounds fine to me- both models of deployment are supported, and we provide the above recommendations to admins.

Can we go deeper into what "managing nearly all resources" means? The vision I originally had for K8TRE is that Argo CD manages most infrastructure apart from workspaces. e.g. in the JupyterHub control plane model JupyterHub is responsible for creating/destroying workspaces (whether they're containers, VMs, or something else). Is this still what you're thinking, or are you also thinking that ArgoCD manages individual workspaces/VMs?

[vvcb]

I had initially thought that it may be tricky to bootstrap a cluster with some of the cloud agnostic stuff and that we may have to do this before ArgoCD can deploy apps that depend on these resources - eg. cluster administrators will be responsible for creating the storage classes specific to their cloud.

But, the combination of ApplicationSets and cluster generator with labels may allow us to get ArgoCD to deploy everything including the cloud-specific/environment-specific stuff.

In the JupyterHub model, JHUB will still be responsible for managing workspaces. ArgoCD will complain that JupyterHub is out-of-sync because of the new resources but there are ways of addressing this.