diff --git a/argo-setup/README.md b/argo-setup/README.md index 9397c47..f171eca 100644 --- a/argo-setup/README.md +++ b/argo-setup/README.md @@ -1,25 +1,19 @@ -# Argo CD v3.3.9 – Small‑Team HA Install +# Argo CD – Small‑Team HA Install -This readme bootstraps a **production‑ready, Highly Available** Argo CD -instance for a small DevOps / Platform team (≈ < 100 Applications to start) -running on a single Kubernetes cluster. +This overlay bootstraps a **production-ready, Highly Available** Argo CD instance for a small DevOps / Platform team (≈ < 100 Applications to start) running on a single Kubernetes cluster. -> **External access:** this overlay ships an example `Ingress` in -> [`patches/argocd-server-ingress.yaml`](patches/argocd-server-ingress.yaml) with -> placeholder hostnames. Update the host/TLS secret annotations to match your -> environment before applying. Alternatively, you can expose the UI via -> `kubectl -n argocd-mip-team port-forward svc/argocd-server 8080:443` or a -> `Service` of `type: LoadBalancer` configured out of band. -> The server still runs in `--insecure` mode (HTTP) so any external front end is -> responsible for TLS termination. CLI usage: `argocd login --grpc-web`. +> **External access:** an `Ingress` is shipped in [`patches/argocd-server-ingress.yaml`](patches/argocd-server-ingress.yaml). It assumes a Traefik ingress class with a `letsencrypt` cluster issuer and uses `argocd.example.com` as a placeholder — rewrite that to your real DNS record before applying (the [Bootstrap](#bootstrap) section below does this with a one-liner `sed`). The server runs in `--insecure` mode so Traefik terminates TLS; CLI usage: `argocd login --grpc-web`. If you don't have an ingress controller yet, port-forward instead: `kubectl -n argocd-mip-team port-forward svc/argocd-server 8080:443`. ## Repository layout ``` argo-setup/ -├── README.md # what you’re reading now -└── patches/ # overlay manifests +├── README.md # what you're reading now +├── upstream-snapshot/ # vendored upstream RBAC for drift-check +└── patches/ ├── kustomization.yaml + ├── argocd-server-ingress.yaml + ├── poddisruptionbudgets.yaml ├── patch-argocd-application-controller-resources.yaml ├── patch-argocd-repo-server-resources.yaml ├── patch-argocd-server-resources.yaml @@ -27,148 +21,147 @@ argo-setup/ ├── patch-argocd-redis-ha-statefulset.yaml ├── patch-argocd-cmd-params-cm.yaml ├── patch-argocd-cm.yaml - ├── patch-argocd-rbac-cm.yaml # UI/API authorization (Layer 3) - ├── patch-argocd-application-controller-clusterrole.yaml # full-replace of upstream rules - ├── patch-argocd-server-clusterrole.yaml # full-replace of upstream rules - └── patch-argocd-applicationset-controller-clusterrole.yaml # empties out cluster-wide grants + ├── patch-argocd-rbac-cm.yaml # UI/API authorization (Layer 3) + ├── patch-argocd-application-controller-clusterrole.yaml # full-replace of upstream rules + ├── patch-argocd-server-clusterrole.yaml # full-replace of upstream rules + ├── patch-argocd-applicationset-controller-clusterrole.yaml # empties out cluster-wide grants + ├── patch-argocd-application-controller-clusterrolebinding.yaml + ├── patch-argocd-server-clusterrolebinding.yaml + └── patch-argocd-applicationset-controller-clusterrolebinding.yaml ``` -> ClusterRoleBindings are inherited from the upstream HA base, but we still patch -> them explicitly: kustomize’s `namespace:` directive does **not** reliably -> rewrite `subjects[].namespace` on cluster-scoped bindings coming from a remote -> base, so we point them at `argocd-mip-team` via the `patch-argocd-*-clusterrolebinding.yaml` patches. -> -> See [`docs/rbac-layers.md`](../docs/rbac-layers.md) for the full RBAC & -> privilege map across all four layers, and the open hardening TODOs. +> ClusterRoleBindings are inherited from the upstream HA base, but we still patch them explicitly: kustomize's `namespace:` directive does **not** reliably rewrite `subjects[].namespace` on cluster-scoped bindings coming from a remote base, so we point them at `argocd-mip-team` via the `patch-argocd-*-clusterrolebinding.yaml` patches. +> +> See [`docs/rbac-layers.md`](../docs/rbac-layers.md) for the full RBAC and privilege map across all four layers, and the open hardening TODOs. ## Versioning strategy -We track the **latest released patch in the Argo CD 3.0 line**. In automation and docs we refer to this as `**v3.1.latest**`. At deploy time we *resolve* that alias to the actual tag (e.g., `v3.1.11`, `v3.1.12`, etc.) and commit the resolved tag into Git. +We pin a **specific Argo CD release** in [`patches/kustomization.yaml`](patches/kustomization.yaml). Renovate (or a manual PR) bumps the pinned tag. -Pin that value in your overlays (below) when promoting to prod. +See [Bumping Argo CD](#bumping-argo-cd) below for the upgrade procedure. ## Prerequisites -- Kubernetes cluster with ≥ 3 worker nodes (for HA anti‑affinity) -- `kubectl` v1.27+ and `kustomize` (stand‑alone or `kubectl kustomize`) -- Argo CD CLI installed locally -- A way to reach the cluster API (port-forward, bastion, VPN) — no Ingress - is shipped here; see the note at the top of this file. +- Kubernetes cluster with ≥ 3 schedulable worker nodes (HA anti-affinity for the 3-replica redis statefulset). +- `kubectl` v1.27+ and `kustomize` (standalone or `kubectl kustomize`). +- Argo CD CLI installed locally. +- A way to reach the cluster API (Ingress, port-forward, bastion, VPN). -## Quick start +## Bootstrap > Use this for a **manual bootstrap**; in GitOps you'd just sync the overlay. -``` -# 0. Vars +```bash ARGOCD_NS=argocd-mip-team - -# Resolve latest 3.0 version (or pin to specific version) -export ARGOCD_SERIES=v3.0 -export ARGOCD_VER=$(curl -s https://api.github.com/repos/argoproj/argo-cd/releases \ - | grep -E '"tag_name": *"'${ARGOCD_SERIES}'[0-9.]*"' \ - | cut -d '"' -f4 \ - | sort -Vr \ - | head -n1) -echo "Resolved latest 3.0 tag: $ARGOCD_VER" - -# Update kustomization.yaml with resolved version -# BSD-Style -sed -i '' "s|/v[0-9.]*/manifests/ha/install.yaml|/${ARGOCD_VER}/manifests/ha/install.yaml|g" patches/kustomization.yaml -# GNU-Style -sed -i "s|/v[0-9.]*/manifests/ha/install.yaml|/${ARGOCD_VER}/manifests/ha/install.yaml|g" patches/kustomization.yaml - -# 1. Namespace -kubectl create namespace $ARGOCD_NS - -# 2. Upstream HA base (pinned tag) -kubectl apply -n $ARGOCD_NS -f \ - https://raw.githubusercontent.com/argoproj/argo-cd/${ARGOCD_VER}/manifests/ha/install.yaml - -# 3. Apply our overlay (resource limits, RBAC tightening, CM/RBAC-CM settings) -kustomize build patches | kubectl apply -f - - -# 4. Wait for pods -kubectl -n $ARGOCD_NS get pods -w - -# 5. Initial admin password & rotate -ARGO_TEMP_PW=$(kubectl -n $ARGOCD_NS get secret argocd-initial-admin-secret \ +ARGOCD_HOST=argocd.your-domain.tld # your real DNS record + +# 1. Rewrite the placeholder hostname in the overlay +# BSD-style (macOS): +LC_ALL=C find . -type f -not -path '*/.git/*' \ + -exec sed -i '' "s/argocd.example.com/${ARGOCD_HOST}/g" {} + +# GNU-style (Linux): +LC_ALL=C find . -type f -not -path '*/.git/*' \ + -exec sed -i "s/argocd.example.com/${ARGOCD_HOST}/g" {} + + +# 2. Namespace +kubectl create namespace "$ARGOCD_NS" + +# 3. Apply the overlay (upstream HA base + our patches in one shot). +# Server-side apply is REQUIRED: the `applicationsets.argoproj.io` CRD +# exceeds the 256 KiB limit for the client-side `last-applied-configuration` +# annotation, and a plain `kubectl apply` fails with +# metadata.annotations: Too long: may not be more than 262144 bytes +kustomize build patches \ + | kubectl apply -n "$ARGOCD_NS" --server-side -f - + +# 4. Wait for the server rollout +kubectl -n "$ARGOCD_NS" rollout status deploy/argocd-server + +# 5. Read the bootstrap admin password and rotate +ARGO_TEMP_PW=$(kubectl -n "$ARGOCD_NS" get secret argocd-initial-admin-secret \ -o jsonpath="{.data.password}" | base64 -d) -echo "Initial "admin" password: $ARGO_TEMP_PW" -argocd login $ARGOCD_HOST --grpc-web +echo "Initial 'admin' password: $ARGO_TEMP_PW" +argocd login "$ARGOCD_HOST" --grpc-web --username admin argocd account update-password -kubectl -n $ARGOCD_NS delete secret argocd-initial-admin-secret +kubectl -n "$ARGOCD_NS" delete secret argocd-initial-admin-secret --ignore-not-found ``` - ## Resource sizing -**All patch files below double that baseline** to give safe headroom for a small but growing team. Adjust downward if cluster‑constrained; upward as you add apps. - -### Current sizing baked into patches - +The baked-in requests/limits roughly double the upstream baseline to give safe headroom for a small but growing team. Adjust down if cluster-constrained, up as you add apps. Monitor with `kubectl top pods` or Prometheus. | Component | Replicas | CPU Request | CPU Limit | Mem Request | Mem Limit | -| ------------------------------- | ---------- | ------------- | ----------- | ------------- | ----------- | +| ----------------------------- | -------- | ----------- | --------- | ----------- | --------- | | argocd-application-controller | 2 | 500m | 2 | 1Gi | 2Gi | | argocd-repo-server | 2 | 500m | 2 | 1Gi | 2Gi | | argocd-server (API/UI) | 2 | 200m | 1 | 512Mi | 1Gi | | argocd-dex-server | 2 | 100m | 400m | 256Mi | 512Mi | | argocd-redis-ha (per pod) | 3 | 200m | 1 | 512Mi | 1Gi | -> To monitor pod usage and adjust: `kubectl top pods` or Prometheus - -## Kustomize overlay install - -The overlay adjusts **only what we need**: resources limitations, HA replica counts, TLS‑terminating ingress, `server.insecure`, and our Argo CD config overrides (resourceTrackingMethod, instanceLabelKey, installationID). - -### patches/kustomization.yaml - -See the live file in [`patches/kustomization.yaml`](patches/kustomization.yaml) -— the canonical list of patches lives there, not duplicated here. -The overlay uses the modern `patches:` field (not the deprecated -`patchesStrategicMerge`). +## What the overlay changes -**Apply once:** +The canonical list of patches lives in [`patches/kustomization.yaml`](patches/kustomization.yaml) — it uses the modern `patches:` field (not the deprecated `patchesStrategicMerge`). The overlay's job is to adjust **only what we need** on top of the upstream HA manifest: -``` -# Resolve and update version first -export ARGOCD_VER=$(curl -s https://api.github.com/repos/argoproj/argo-cd/releases \ - | grep -E '"tag_name": *"v3.0[0-9.]*"' \ - | cut -d '"' -f4 \ - | sort -Vr \ - | head -n1) -sed -i "s|/v[0-9.]*/manifests/ha/install.yaml|/${ARGOCD_VER}/manifests/ha/install.yaml|g" patches/kustomization.yaml - -# Then apply -kustomize build patches | kubectl apply -f - -``` +- Resource requests/limits (table above) and HA replica counts. +- Traefik `Ingress` + `--insecure` on `argocd-server` (TLS terminated at ingress). +- PodDisruptionBudgets for every workload. +- `seccompProfile: RuntimeDefault` on every pod (PSS-restricted). +- ClusterRole tightening (full-replace of upstream rules — see the note in `kustomization.yaml`). +- ClusterRoleBinding namespace rewrites. +- `argocd-cm` / `argocd-rbac-cm` / `argocd-cmd-params-cm` config (below). -## Config: argocd-cm customizations +## Config: `argocd-cm` customizations -We set several global behaviors in `patch-argocd-cm.yaml`: +Set in [`patches/patch-argocd-cm.yaml`](patches/patch-argocd-cm.yaml). +| Key | Value | Why | +| ------------------------------------ | -------------------------------------- | ------------------------------------------------------------------------------ | +| `url` | `https://argocd.example.com` | Public URL — rewritten by the `sed` step in [Bootstrap](#bootstrap). | +| `application.resourceTrackingMethod` | `label` | Track managed resources by label; helps with pruning and external tooling. | +| `application.instanceLabelKey` | `argocd.argoproj.io/instanceTracking` | Avoids collision with the standard `app.kubernetes.io/instance` Helm stamps. | +| `installationID` | `mip-team-argo-cd` | Unique install identifier (multi-cluster analytics, logs). | +| `resource.respectRBAC` | `normal` | Respect destination-cluster RBAC when listing resources. | +| `kustomize.buildOptions` | `--enable-helm` | Allow kustomize overlays to inflate Helm charts (used by our deployments). | +| `timeout.reconciliation` | `180s` | Upstream default; revisit if controller CPU is high or apps lag. | +| `resource.exclusions` | `EndpointSlice`, `Endpoints`, `Lease` | Skip noisy / high-churn kinds Argo never reconciles as desired state. | -| Key | Value | Why | -| -------------------------------------- | ------------------------------- | -------------------------------------------------------------------------- | -| `application.resourceTrackingMethod` | `label` | Track managed resources by label; helps with pruning & external tooling. | -| `application.instanceLabelKey` | `argocd.argoproj.io/instance` | Override default instance label key; matches our team label convention. | -| `installationID` | `mip-team-argo-cd` | Unique install identifier (multi‑cluster analytics, logs). | +`argocd-rbac-cm` (Layer 3 authz) is documented separately — see [`docs/rbac-layers.md`](../docs/rbac-layers.md). ## Bootstrap secrets -The upstream HA install ships an `argocd-secret` skeleton that needs to be -populated with at minimum the server signing key, and (when SSO is wired up) -the Dex / OIDC client secret. Use [`scripts/gen_secrets.sh`](../scripts/gen_secrets.sh) -to generate the values, then apply via your secret-management workflow -(SOPS / sealed-secrets / external-secrets) — do **not** check the populated -`argocd-secret` into git. +The upstream HA install ships an `argocd-secret` skeleton that needs to be populated with at minimum the server signing key, and (when SSO is wired up) the Dex / OIDC client secret. Use [`scripts/gen_secrets.sh`](../scripts/gen_secrets.sh) to generate the values, then apply via your secret-management workflow (SOPS / sealed-secrets / external-secrets) — do **not** check the populated `argocd-secret` into git. ## Post-install checks -``` -kubectl -n argocd-mip-team get ingress -kubectl -n argocd-mip-team get certificate argocd-tls -o yaml | grep -E 'Ready|Not Ready' -argocd login argocd.example.com --grpc-web +```bash +ARGOCD_NS=argocd-mip-team +kubectl -n "$ARGOCD_NS" get pods +kubectl -n "$ARGOCD_NS" get ingress +kubectl -n "$ARGOCD_NS" get certificate argocd-server-tls \ + -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{"\n"}' +argocd login "$ARGOCD_HOST" --grpc-web argocd version ``` + +## Bumping Argo CD + +1. Pick the new tag (latest patch in the current minor line, or a deliberate minor bump after reviewing the upstream changelog). +2. Bump the pinned tag in [`patches/kustomization.yaml`](patches/kustomization.yaml) (the `resources:` entry pointing at `raw.githubusercontent.com/argoproj/argo-cd//manifests/ha/install.yaml`). +3. Run the RBAC drift check - it reads the upstream URL from `kustomization.yaml`, so do this *after* step 2: + ```bash + bash scripts/check-upstream-argo-clusterroles.sh + ``` + It covers all three ClusterRoles we patch (`argocd-application-controller`, `argocd-server`, `argocd-applicationset-controller`) plus the seven namespaced `Role`s tracked in [`upstream-snapshot/roles.yaml`](upstream-snapshot/roles.yaml). If it reports drift, read the diff and decide for each new upstream rule whether it belongs in our tightened patches (`patch-argocd-{application-controller,server,applicationset-controller}-clusterrole.yaml`). Then refresh both snapshots in one shot and commit them in the same PR as the version bump: + ```bash + bash scripts/check-upstream-argo-clusterroles.sh --update + ``` +4. Re-render the overlay locally: `kubectl kustomize patches >/dev/null`. +5. Open a PR; CI runs the kustomize render and the drift check. +6. After merge, re-apply against the live cluster — `argo-setup/` is bootstrapped out-of-band, not self-managed by Argo CD: + ```bash + kustomize build argo-setup/patches \ + | kubectl apply -n argocd-mip-team --server-side --force-conflicts -f - + kubectl -n argocd-mip-team rollout status deploy/argocd-server + kubectl -n argocd-mip-team rollout status statefulset/argocd-application-controller + ``` + > **Why `--force-conflicts`:** if any object on the cluster was ever applied client-side (the old `kubectl apply` without `--server-side`), its fields are owned by the manager `kubectl-client-side-apply`. A plain server-side apply will refuse to overwrite those fields with `Apply failed with N conflict(s)`. `--force-conflicts` transfers ownership to the `kubectl` server-side manager — safe here because the rendered overlay is the source of truth. After the first run, ownership is consistent and the flag is a no-op on subsequent bumps. diff --git a/argo-setup/patches/kustomization.yaml b/argo-setup/patches/kustomization.yaml index c658cbb..6e24d71 100644 --- a/argo-setup/patches/kustomization.yaml +++ b/argo-setup/patches/kustomization.yaml @@ -5,7 +5,7 @@ namespace: argocd-mip-team # Base: upstream HA manifest (resolved tag). # Bump policy: pin to a specific tag, manage upgrades via Renovate / manual PR. resources: - - https://raw.githubusercontent.com/argoproj/argo-cd/v3.3.9/manifests/ha/install.yaml + - https://raw.githubusercontent.com/argoproj/argo-cd/v3.3.11/manifests/ha/install.yaml - poddisruptionbudgets.yaml - argocd-server-ingress.yaml # Strategic-merge / JSON patches on top of the upstream base. diff --git a/docs/getting-started.md b/docs/getting-started.md index 376d9d5..f2ecb1a 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -24,67 +24,17 @@ git checkout main ## 1. Install Argo CD -The HA overlay lives in [`argo-setup/`](../argo-setup/). It pins a specific -Argo CD release and applies the repo-specific resource, ingress, and config -patches. See [argo-setup/README.md](../argo-setup/README.md) for the overlay -details and the full bootstrap flow. +The HA overlay lives in [`argo-setup/`](../argo-setup/). Run the steps in +[`argo-setup/README.md` § Bootstrap](../argo-setup/README.md#bootstrap) - +they cover hostname rewrite, namespace creation, `kustomize build … | +kubectl apply --server-side`, rollout wait, and bootstrap admin-password +rotation. From off-cluster you'll need a tunnel + `/etc/hosts` mapping for +the final `argocd login` step — see [remote-access.md](remote-access.md). -```bash -ARGOCD_HOST=example.com #YOUR DOMAIN HERE -ARGOCD_NS=argocd-mip-team - -cd argo-setup - -# BSD-Style -LC_ALL=C find . -type f -not -path '*/.git/*' -exec sed -i '' "s/example.com/$ARGOCD_HOST/g" {} + -# GNU-Style -LC_ALL=C find . -type f -not -path '*/.git/*' -exec sed -i "s/example.com/$ARGOCD_HOST/g" {} + - -kubectl create namespace "$ARGOCD_NS" - -# Server-side apply is required: the `applicationsets.argoproj.io` CRD -# exceeds the 256 KiB limit for the client-side `last-applied-configuration` -# annotation and a plain `kubectl apply` fails with -# metadata.annotations: Too long: may not be more than 262144 bytes -kustomize build patches \ - | kubectl apply -n "$ARGOCD_NS" --server-side -f - - -kubectl -n "$ARGOCD_NS" rollout status deploy/argocd-server -``` - -Update the placeholder hostname in -[`argo-setup/patches/patch-argocd-ingress.yaml`](../argo-setup/patches/patch-argocd-ingress.yaml) -to your real DNS record before applying in production. - -Quick sanity checks: - -```bash -kubectl -n "$ARGOCD_NS" get pods -``` - -## 2. Rotate the bootstrap admin password - -Use the Argo CD CLI. From off-cluster you'll need a tunnel + `/etc/hosts` -mapping first — see [remote-access.md](remote-access.md). - -```bash -ARGOCD_NS=argocd-mip-team - -# Read the bootstrap password -ARGO_TEMP_PW=$(kubectl -n "$ARGOCD_NS" get secret argocd-initial-admin-secret \ - -o jsonpath="{.data.password}" | base64 -d) -echo "Initial 'admin' password: $ARGO_TEMP_PW" - -# Log in (use the host you reach Argo CD on; add :8443 if you're using the -# SSH -L tunnel from remote-access.md) -argocd login --grpc-web --username admin -argocd account update-password - -# Optionally Drop the bootstrap secret -# kubectl -n "$ARGOCD_NS" delete secret argocd-initial-admin-secret --ignore-not-found -``` +When `kubectl -n argocd-mip-team rollout status deploy/argocd-server` +returns successfully and you've rotated the admin password, come back here. -## 2b. Register repositories with Argo CD +## 2. Register repositories with Argo CD Register this repository in Argo CD. If your deployment values live in the private `mip-deployments` repository, register that too. @@ -126,7 +76,7 @@ Two-step bootstrap. The order matters: nothing else can sync until kubectl apply -f projects/mip-infrastructure.yaml # ApplicationSet that creates every other AppProject -kubectl apply -f base/argo-projects.yaml +kubectl apply -f base/argo-projects/argo-projects.yaml # Verify — should show all static projects + per-fed projects kubectl get appprojects -n argocd-mip-team diff --git a/docs/rbac-layers.md b/docs/rbac-layers.md index 9bb8598..4159898 100644 --- a/docs/rbac-layers.md +++ b/docs/rbac-layers.md @@ -24,7 +24,7 @@ Defense-in-depth fails when Layer 1 is broader than Layer 2. Two SAs, both cluster-scoped. Sources: [`patch-argocd-application-controller-clusterrole.yaml`](../argo-setup/patches/patch-argocd-application-controller-clusterrole.yaml), -[`patch-argocd-server-extra-read.yaml`](../argo-setup/patches/patch-argocd-server-extra-read.yaml). +[`patch-argocd-server-clusterrole.yaml`](../argo-setup/patches/patch-argocd-server-clusterrole.yaml). ### `argocd-application-controller` — does the actual sync writes | API group | Resources | Verbs | Notes | @@ -151,7 +151,7 @@ External charts that *do* render their own RBAC (datacatalog, exareme2, mip plat --- -Pinned at Argo CD [v3.3.9](../argo-setup/patches/kustomization.yaml). To bump: +Pinned at Argo CD (../argo-setup/patches/kustomization.yaml). To bump: edit the tag, run `bash scripts/check-upstream-argo-clusterroles.sh --update`, review the diff, commit snapshot + tag together. diff --git a/projects/mip-infrastructure.yaml b/projects/mip-infrastructure.yaml index 23b0d97..93f70b6 100644 --- a/projects/mip-infrastructure.yaml +++ b/projects/mip-infrastructure.yaml @@ -8,7 +8,6 @@ spec: description: Core infrastructure components and ApplicationSets sourceRepos: - https://github.com/Medical-Informatics-Platform/mip-infra.git - - git@github.com:NeuroTech-Platform/mip-deployments.git destinations: - namespace: argocd-mip-team server: https://kubernetes.default.svc