Skip to content

fix: deployment fixes and doc update #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jdaln merged 2 commits into from
Jun 17, 2026
Merged

fix: deployment fixes and doc update #21

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f007d43
fix: deployment fixes and doc update
jdaln Jun 17, 2026
3e1599a
style: auto-fix YAML linting issues ✨
github-actions[bot] Jun 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
227 changes: 110 additions & 117 deletions argo-setup/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,174 +1,167 @@
# Argo CD v3.3.9 – Small‑Team HA Install
# Argo CD – Small‑Team HA Install

This readme bootstraps a **production‑ready, Highly Available** Argo CD
instance for a small DevOps / Platform team (≈ < 100 Applications to start)
running on a single Kubernetes cluster.
This overlay bootstraps a **production-ready, Highly Available** Argo CD instance for a small DevOps / Platform team (≈ < 100 Applications to start) running on a single Kubernetes cluster.

> **External access:** this overlay ships an example `Ingress` in
> [`patches/argocd-server-ingress.yaml`](patches/argocd-server-ingress.yaml) with
> placeholder hostnames. Update the host/TLS secret annotations to match your
> environment before applying. Alternatively, you can expose the UI via
> `kubectl -n argocd-mip-team port-forward svc/argocd-server 8080:443` or a
> `Service` of `type: LoadBalancer` configured out of band.
> The server still runs in `--insecure` mode (HTTP) so any external front end is
> responsible for TLS termination. CLI usage: `argocd login <host> --grpc-web`.
> **External access:** an `Ingress` is shipped in [`patches/argocd-server-ingress.yaml`](patches/argocd-server-ingress.yaml). It assumes a Traefik ingress class with a `letsencrypt` cluster issuer and uses `argocd.example.com` as a placeholder — rewrite that to your real DNS record before applying (the [Bootstrap](#bootstrap) section below does this with a one-liner `sed`). The server runs in `--insecure` mode so Traefik terminates TLS; CLI usage: `argocd login <host> --grpc-web`. If you don't have an ingress controller yet, port-forward instead: `kubectl -n argocd-mip-team port-forward svc/argocd-server 8080:443`.

## Repository layout

```
argo-setup/
├── README.md # what you’re reading now
└── patches/ # overlay manifests
├── README.md # what you're reading now
├── upstream-snapshot/ # vendored upstream RBAC for drift-check
└── patches/
├── kustomization.yaml
├── argocd-server-ingress.yaml
├── poddisruptionbudgets.yaml
├── patch-argocd-application-controller-resources.yaml
├── patch-argocd-repo-server-resources.yaml
├── patch-argocd-server-resources.yaml
├── patch-argocd-dex-server-resources.yaml
├── patch-argocd-redis-ha-statefulset.yaml
├── patch-argocd-cmd-params-cm.yaml
├── patch-argocd-cm.yaml
├── patch-argocd-rbac-cm.yaml # UI/API authorization (Layer 3)
├── patch-argocd-application-controller-clusterrole.yaml # full-replace of upstream rules
├── patch-argocd-server-clusterrole.yaml # full-replace of upstream rules
└── patch-argocd-applicationset-controller-clusterrole.yaml # empties out cluster-wide grants
├── patch-argocd-rbac-cm.yaml # UI/API authorization (Layer 3)
├── patch-argocd-application-controller-clusterrole.yaml # full-replace of upstream rules
├── patch-argocd-server-clusterrole.yaml # full-replace of upstream rules
├── patch-argocd-applicationset-controller-clusterrole.yaml # empties out cluster-wide grants
├── patch-argocd-application-controller-clusterrolebinding.yaml
├── patch-argocd-server-clusterrolebinding.yaml
└── patch-argocd-applicationset-controller-clusterrolebinding.yaml
```

> ClusterRoleBindings are inherited from the upstream HA base, but we still patch
> them explicitly: kustomize’s `namespace:` directive does **not** reliably
> rewrite `subjects[].namespace` on cluster-scoped bindings coming from a remote
> base, so we point them at `argocd-mip-team` via the `patch-argocd-*-clusterrolebinding.yaml` patches.
>
> See [`docs/rbac-layers.md`](../docs/rbac-layers.md) for the full RBAC &
> privilege map across all four layers, and the open hardening TODOs.
> ClusterRoleBindings are inherited from the upstream HA base, but we still patch them explicitly: kustomize's `namespace:` directive does **not** reliably rewrite `subjects[].namespace` on cluster-scoped bindings coming from a remote base, so we point them at `argocd-mip-team` via the `patch-argocd-*-clusterrolebinding.yaml` patches.
>
> See [`docs/rbac-layers.md`](../docs/rbac-layers.md) for the full RBAC and privilege map across all four layers, and the open hardening TODOs.

## Versioning strategy

We track the **latest released patch in the Argo CD 3.0 line**. In automation and docs we refer to this as `**v3.1.latest**`. At deploy time we *resolve* that alias to the actual tag (e.g., `v3.1.11`, `v3.1.12`, etc.) and commit the resolved tag into Git.
We pin a **specific Argo CD release** in [`patches/kustomization.yaml`](patches/kustomization.yaml). Renovate (or a manual PR) bumps the pinned tag.

Pin that value in your overlays (below) when promoting to prod.
See [Bumping Argo CD](#bumping-argo-cd) below for the upgrade procedure.

## Prerequisites

- Kubernetes cluster with ≥ 3 worker nodes (for HA anti‑affinity)
- `kubectl` v1.27+ and `kustomize` (stand‑alone or `kubectl kustomize`)
- Argo CD CLI installed locally
- A way to reach the cluster API (port-forward, bastion, VPN) — no Ingress
is shipped here; see the note at the top of this file.
- Kubernetes cluster with ≥ 3 schedulable worker nodes (HA anti-affinity for the 3-replica redis statefulset).
- `kubectl` v1.27+ and `kustomize` (standalone or `kubectl kustomize`).
- Argo CD CLI installed locally.
- A way to reach the cluster API (Ingress, port-forward, bastion, VPN).

## Quick start
## Bootstrap

> Use this for a **manual bootstrap**; in GitOps you'd just sync the overlay.

```
# 0. Vars
```bash
ARGOCD_NS=argocd-mip-team

# Resolve latest 3.0 version (or pin to specific version)
export ARGOCD_SERIES=v3.0
export ARGOCD_VER=$(curl -s https://api.github.com/repos/argoproj/argo-cd/releases \
| grep -E '"tag_name": *"'${ARGOCD_SERIES}'[0-9.]*"' \
| cut -d '"' -f4 \
| sort -Vr \
| head -n1)
echo "Resolved latest 3.0 tag: $ARGOCD_VER"

# Update kustomization.yaml with resolved version
# BSD-Style
sed -i '' "s|/v[0-9.]*/manifests/ha/install.yaml|/${ARGOCD_VER}/manifests/ha/install.yaml|g" patches/kustomization.yaml
# GNU-Style
sed -i "s|/v[0-9.]*/manifests/ha/install.yaml|/${ARGOCD_VER}/manifests/ha/install.yaml|g" patches/kustomization.yaml

# 1. Namespace
kubectl create namespace $ARGOCD_NS

# 2. Upstream HA base (pinned tag)
kubectl apply -n $ARGOCD_NS -f \
https://raw.githubusercontent.com/argoproj/argo-cd/${ARGOCD_VER}/manifests/ha/install.yaml

# 3. Apply our overlay (resource limits, RBAC tightening, CM/RBAC-CM settings)
kustomize build patches | kubectl apply -f -

# 4. Wait for pods
kubectl -n $ARGOCD_NS get pods -w

# 5. Initial admin password & rotate
ARGO_TEMP_PW=$(kubectl -n $ARGOCD_NS get secret argocd-initial-admin-secret \
ARGOCD_HOST=argocd.your-domain.tld # your real DNS record

# 1. Rewrite the placeholder hostname in the overlay
# BSD-style (macOS):
LC_ALL=C find . -type f -not -path '*/.git/*' \
-exec sed -i '' "s/argocd.example.com/${ARGOCD_HOST}/g" {} +
# GNU-style (Linux):
LC_ALL=C find . -type f -not -path '*/.git/*' \
-exec sed -i "s/argocd.example.com/${ARGOCD_HOST}/g" {} +

# 2. Namespace
kubectl create namespace "$ARGOCD_NS"

# 3. Apply the overlay (upstream HA base + our patches in one shot).
# Server-side apply is REQUIRED: the `applicationsets.argoproj.io` CRD
# exceeds the 256 KiB limit for the client-side `last-applied-configuration`
# annotation, and a plain `kubectl apply` fails with
# metadata.annotations: Too long: may not be more than 262144 bytes
kustomize build patches \
| kubectl apply -n "$ARGOCD_NS" --server-side -f -

# 4. Wait for the server rollout
kubectl -n "$ARGOCD_NS" rollout status deploy/argocd-server

# 5. Read the bootstrap admin password and rotate
ARGO_TEMP_PW=$(kubectl -n "$ARGOCD_NS" get secret argocd-initial-admin-secret \
-o jsonpath="{.data.password}" | base64 -d)
echo "Initial "admin" password: $ARGO_TEMP_PW"
argocd login $ARGOCD_HOST --grpc-web
echo "Initial 'admin' password: $ARGO_TEMP_PW"
argocd login "$ARGOCD_HOST" --grpc-web --username admin
argocd account update-password
kubectl -n $ARGOCD_NS delete secret argocd-initial-admin-secret
kubectl -n "$ARGOCD_NS" delete secret argocd-initial-admin-secret --ignore-not-found
```


## Resource sizing

**All patch files below double that baseline** to give safe headroom for a small but growing team. Adjust downward if cluster‑constrained; upward as you add apps.

### Current sizing baked into patches

The baked-in requests/limits roughly double the upstream baseline to give safe headroom for a small but growing team. Adjust down if cluster-constrained, up as you add apps. Monitor with `kubectl top pods` or Prometheus.

| Component | Replicas | CPU Request | CPU Limit | Mem Request | Mem Limit |
| ------------------------------- | ---------- | ------------- | ----------- | ------------- | ----------- |
| ----------------------------- | -------- | ----------- | --------- | ----------- | --------- |
| argocd-application-controller | 2 | 500m | 2 | 1Gi | 2Gi |
| argocd-repo-server | 2 | 500m | 2 | 1Gi | 2Gi |
| argocd-server (API/UI) | 2 | 200m | 1 | 512Mi | 1Gi |
| argocd-dex-server | 2 | 100m | 400m | 256Mi | 512Mi |
| argocd-redis-ha (per pod) | 3 | 200m | 1 | 512Mi | 1Gi |

> To monitor pod usage and adjust: `kubectl top pods` or Prometheus

## Kustomize overlay install

The overlay adjusts **only what we need**: resources limitations, HA replica counts, TLS‑terminating ingress, `server.insecure`, and our Argo CD config overrides (resourceTrackingMethod, instanceLabelKey, installationID).

### patches/kustomization.yaml

See the live file in [`patches/kustomization.yaml`](patches/kustomization.yaml)
— the canonical list of patches lives there, not duplicated here.
The overlay uses the modern `patches:` field (not the deprecated
`patchesStrategicMerge`).
## What the overlay changes

**Apply once:**
The canonical list of patches lives in [`patches/kustomization.yaml`](patches/kustomization.yaml) — it uses the modern `patches:` field (not the deprecated `patchesStrategicMerge`). The overlay's job is to adjust **only what we need** on top of the upstream HA manifest:

```
# Resolve and update version first
export ARGOCD_VER=$(curl -s https://api.github.com/repos/argoproj/argo-cd/releases \
| grep -E '"tag_name": *"v3.0[0-9.]*"' \
| cut -d '"' -f4 \
| sort -Vr \
| head -n1)
sed -i "s|/v[0-9.]*/manifests/ha/install.yaml|/${ARGOCD_VER}/manifests/ha/install.yaml|g" patches/kustomization.yaml

# Then apply
kustomize build patches | kubectl apply -f -
```
- Resource requests/limits (table above) and HA replica counts.
- Traefik `Ingress` + `--insecure` on `argocd-server` (TLS terminated at ingress).
- PodDisruptionBudgets for every workload.
- `seccompProfile: RuntimeDefault` on every pod (PSS-restricted).
- ClusterRole tightening (full-replace of upstream rules — see the note in `kustomization.yaml`).
- ClusterRoleBinding namespace rewrites.
- `argocd-cm` / `argocd-rbac-cm` / `argocd-cmd-params-cm` config (below).

## Config: argocd-cm customizations
## Config: `argocd-cm` customizations

We set several global behaviors in `patch-argocd-cm.yaml`:
Set in [`patches/patch-argocd-cm.yaml`](patches/patch-argocd-cm.yaml).

| Key | Value | Why |
| ------------------------------------ | -------------------------------------- | ------------------------------------------------------------------------------ |
| `url` | `https://argocd.example.com` | Public URL — rewritten by the `sed` step in [Bootstrap](#bootstrap). |
| `application.resourceTrackingMethod` | `label` | Track managed resources by label; helps with pruning and external tooling. |
| `application.instanceLabelKey` | `argocd.argoproj.io/instanceTracking` | Avoids collision with the standard `app.kubernetes.io/instance` Helm stamps. |
| `installationID` | `mip-team-argo-cd` | Unique install identifier (multi-cluster analytics, logs). |
| `resource.respectRBAC` | `normal` | Respect destination-cluster RBAC when listing resources. |
| `kustomize.buildOptions` | `--enable-helm` | Allow kustomize overlays to inflate Helm charts (used by our deployments). |
| `timeout.reconciliation` | `180s` | Upstream default; revisit if controller CPU is high or apps lag. |
| `resource.exclusions` | `EndpointSlice`, `Endpoints`, `Lease` | Skip noisy / high-churn kinds Argo never reconciles as desired state. |

| Key | Value | Why |
| -------------------------------------- | ------------------------------- | -------------------------------------------------------------------------- |
| `application.resourceTrackingMethod` | `label` | Track managed resources by label; helps with pruning & external tooling. |
| `application.instanceLabelKey` | `argocd.argoproj.io/instance` | Override default instance label key; matches our team label convention. |
| `installationID` | `mip-team-argo-cd` | Unique install identifier (multi‑cluster analytics, logs). |
`argocd-rbac-cm` (Layer 3 authz) is documented separately — see [`docs/rbac-layers.md`](../docs/rbac-layers.md).

## Bootstrap secrets

The upstream HA install ships an `argocd-secret` skeleton that needs to be
populated with at minimum the server signing key, and (when SSO is wired up)
the Dex / OIDC client secret. Use [`scripts/gen_secrets.sh`](../scripts/gen_secrets.sh)
to generate the values, then apply via your secret-management workflow
(SOPS / sealed-secrets / external-secrets) — do **not** check the populated
`argocd-secret` into git.
The upstream HA install ships an `argocd-secret` skeleton that needs to be populated with at minimum the server signing key, and (when SSO is wired up) the Dex / OIDC client secret. Use [`scripts/gen_secrets.sh`](../scripts/gen_secrets.sh) to generate the values, then apply via your secret-management workflow (SOPS / sealed-secrets / external-secrets) — do **not** check the populated `argocd-secret` into git.

## Post-install checks

```
kubectl -n argocd-mip-team get ingress
kubectl -n argocd-mip-team get certificate argocd-tls -o yaml | grep -E 'Ready|Not Ready'
argocd login argocd.example.com --grpc-web
```bash
ARGOCD_NS=argocd-mip-team
kubectl -n "$ARGOCD_NS" get pods
kubectl -n "$ARGOCD_NS" get ingress
kubectl -n "$ARGOCD_NS" get certificate argocd-server-tls \
-o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{"\n"}'
argocd login "$ARGOCD_HOST" --grpc-web
argocd version
```

## Bumping Argo CD

1. Pick the new tag (latest patch in the current minor line, or a deliberate minor bump after reviewing the upstream changelog).
2. Bump the pinned tag in [`patches/kustomization.yaml`](patches/kustomization.yaml) (the `resources:` entry pointing at `raw.githubusercontent.com/argoproj/argo-cd/<TAG>/manifests/ha/install.yaml`).
3. Run the RBAC drift check - it reads the upstream URL from `kustomization.yaml`, so do this *after* step 2:
```bash
bash scripts/check-upstream-argo-clusterroles.sh
```
It covers all three ClusterRoles we patch (`argocd-application-controller`, `argocd-server`, `argocd-applicationset-controller`) plus the seven namespaced `Role`s tracked in [`upstream-snapshot/roles.yaml`](upstream-snapshot/roles.yaml). If it reports drift, read the diff and decide for each new upstream rule whether it belongs in our tightened patches (`patch-argocd-{application-controller,server,applicationset-controller}-clusterrole.yaml`). Then refresh both snapshots in one shot and commit them in the same PR as the version bump:
```bash
bash scripts/check-upstream-argo-clusterroles.sh --update
```
4. Re-render the overlay locally: `kubectl kustomize patches >/dev/null`.
5. Open a PR; CI runs the kustomize render and the drift check.
6. After merge, re-apply against the live cluster — `argo-setup/` is bootstrapped out-of-band, not self-managed by Argo CD:
```bash
kustomize build argo-setup/patches \
| kubectl apply -n argocd-mip-team --server-side --force-conflicts -f -
kubectl -n argocd-mip-team rollout status deploy/argocd-server
kubectl -n argocd-mip-team rollout status statefulset/argocd-application-controller
```
> **Why `--force-conflicts`:** if any object on the cluster was ever applied client-side (the old `kubectl apply` without `--server-side`), its fields are owned by the manager `kubectl-client-side-apply`. A plain server-side apply will refuse to overwrite those fields with `Apply failed with N conflict(s)`. `--force-conflicts` transfers ownership to the `kubectl` server-side manager — safe here because the rendered overlay is the source of truth. After the first run, ownership is consistent and the flag is a no-op on subsequent bumps.
2 changes: 1 addition & 1 deletion argo-setup/patches/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ namespace: argocd-mip-team
# Base: upstream HA manifest (resolved tag).
# Bump policy: pin to a specific tag, manage upgrades via Renovate / manual PR.
resources:
- https://raw.githubusercontent.com/argoproj/argo-cd/v3.3.9/manifests/ha/install.yaml
- https://raw.githubusercontent.com/argoproj/argo-cd/v3.3.11/manifests/ha/install.yaml
- poddisruptionbudgets.yaml
- argocd-server-ingress.yaml
# Strategic-merge / JSON patches on top of the upstream base.
Expand Down
Loading