diff --git a/flux/README.md b/flux/README.md new file mode 100644 index 0000000..51f4c88 --- /dev/null +++ b/flux/README.md @@ -0,0 +1,185 @@ +# Flux, alongside ArgoCD + +A Flux translation of one app — `fider` — living next to the ArgoCD config it +will eventually replace. `fider` is first because it is the only app in the +cluster where a mistake costs nothing (see [Why fider](#why-fider)). + +## Status: inert + +Nothing here does anything until someone runs `flux bootstrap`. ArgoCD cannot +see this directory: + +- the `applications` ApplicationSet generator globs `applications/**/config.json` +- the `bootstrap-cluster` Application syncs `path: applications` + +Both are scoped to `applications/`, so `flux/` is invisible to ArgoCD, and this +PR is safe to merge at any time without changing cluster behaviour. It does not +remove `applications/fider/config.json` or `applications/fider-db/config.json` — +that is the cutover, and it is a separate change. + +## Why fider + +It is publicly served at `feedback.prod.equal.vote` but **not reachable from the +product** — `grep -rn 'fider\|feedback\.prod' packages/frontend/src packages/backend/src` +in the bettervoting repo returns nothing. So it is the one app where the blast +radius of getting a migration wrong is approximately zero. + +That makes it worth using to rehearse the *hard* path rather than to dodge it. +`fider` happens to exercise three of the four blockers in the full migration: + +| Blocker | Why fider hits it | Also needed for | +| --- | --- | --- | +| Helm adoption | real PVC + CNPG cluster to adopt in place | `postgresql`, `keycloak` | +| SSA field handover | ArgoCD owns fields as `argocd-controller` | every app | + +The third blocker, the `devopscoop` chart, is gone from this path entirely: +`flux/base/app` reproduces it as plain manifests we own (see below). + +## flux/base/app + +The Flux path does not use `devopscoop/charts/app`. That chart renders four +objects — ServiceAccount, Service, Deployment, Ingress — from a values file, and +`flux/base/app` is those four objects as a kustomize base, with +`flux/apps/fider` as the overlay. + +Verified with `kubectl kustomize flux/apps/fider` against the live cluster: the +Deployment selector, container name, image, `serviceAccountName`, `env`, +`resources` and `ports`, the Service selector and ports, and the entire Ingress +spec all render **identical** to what is running. + +Two things worth knowing about the design: + +- The label `app.kubernetes.io/name: app` is kept deliberately. It is half of + the live Deployment's selector, and selectors are immutable — changing it + would force a delete/recreate. The overlay adds + `app.kubernetes.io/instance: fider` via `labels` with `includeSelectors: true`, + reproducing the live selector exactly. +- Adoption will cause **one pod rollout**. The rendered pod template drops three + Helm bookkeeping labels (`helm.sh/chart`, `app.kubernetes.io/managed-by`, + `app.kubernetes.io/version`), which changes the pod-template hash. Harmless + here, and worth expecting rather than debugging. + +`fider-db` stays a HelmRelease on the upstream CloudNativePG `cluster` chart (pinned to `0.8.1`, matching `applications/fider-db/config.json`) — +that is a real upstream, not an in-house wrapper, and keeping it preserves the +Helm-adoption rehearsal that `postgresql` and `keycloak` will need. + +## Prerequisites + +1. **`flux bootstrap`**, e.g. + `flux bootstrap github --owner=Equal-Vote --repository=argocd --path=./flux/clusters/equalvote`. + Adds four controllers in `flux-system`. Note the cluster is already firing + `KubeMemoryOvercommit`, so budget for the extra requests. +2. **Make removal non-destructive.** See below. + +No registry credentials are needed. The only chart pulled is CloudNativePG's, +from a public repository. + +## ⚠️ Removing an app from the ApplicationSet currently destroys its data + +This is true today, for every app, independent of Flux: + +``` +ApplicationSet syncPolicy: (none) <- no preserveResourcesOnDeletion +fider / fider-db finalizers: resources-finalizer.argocd.argoproj.io +PV pvc-df9a66c8 (Bound, live): Delete +PV pvc-00a3f621 (Released): Retain <- the OLD volume from the Aug incident +``` + +Deleting a `config.json` deletes the Application, the finalizer cascades, and +ArgoCD deletes the workloads — including the CNPG Cluster and its PVC, whose PV +then reclaims. The `Retain` policy applied after the August CNPG incident +protects the *previous* volume, not the one in use since the 08-29 rebuild. + +Before any cutover, do one of: + +- set `preserveResourcesOnDeletion: true` on the ApplicationSet `syncPolicy` + (preferred — fixes this for every app at once), or +- `kubectl patch pv pvc-df9a66c8-... -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'`, or +- delete the Application with `--cascade=false` before removing its `config.json` + +Verify the *bound* PV is the protected one, not an old `Released` entry. + +## Cutover + +Ordering is the whole game: **detach from ArgoCD, then let Flux adopt.** Never +both live — two controllers server-side-applying the same objects will fight +over field ownership indefinitely. That is why `flux/clusters/equalvote/apps.yaml` +ships with `suspend: true`: bootstrapping Flux is safe on its own, and resuming +is the deliberate cutover. + +The whole chain, in order — three PRs and four commands: + +| Step | Change | Kind | +| --- | --- | --- | +| 1 | `preserveResourcesOnDeletion: true` on the ApplicationSet | PR | +| 2 | this scaffold | PR | +| 3 | `flux bootstrap` | command | +| 4 | Helm-adoption stamping (step 3 below) | commands | +| 5 | remove `applications/fider{,-db}/config.json` | PR | +| 6 | `flux resume kustomization apps` | command | + +1 must precede 5, or removing the config.json destroys the database. 2 must +precede 3, because bootstrap reconciles from a branch in this repo. 5 must +precede 6, so the two controllers never overlap. + +1. Confirm prerequisites. +2. Apply the deletion-safety fix above. +3. Stamp the existing resources so Helm adopts rather than reinstalls. ArgoCD + renders charts and applies the manifests, so **no Helm release exists in the + cluster** and `helm upgrade --install` would otherwise try a fresh install and + collide. On every object the two charts own: + ``` + kubectl -n fider annotate / \ + meta.helm.sh/release-name=fider meta.helm.sh/release-namespace=fider + kubectl -n fider label / app.kubernetes.io/managed-by=Helm + ``` + (`release-name=fider-db` for the CNPG `Cluster` and its objects.) +4. Detach ArgoCD: remove `applications/fider/config.json` and + `applications/fider-db/config.json` in a separate PR. +5. Let Flux reconcile. Because chart versions are pinned to what ArgoCD runs, + the resulting `helm upgrade` should be a no-op. +6. Watch for SSA conflicts against the leftover `argocd-controller` field + manager. Once ArgoCD is no longer reconciling these objects the stale + `managedFields` entries are harmless, but the first apply may need forcing. +7. `flux resume kustomization apps`. +8. Only after one clean reconcile, flip `driftDetection.mode` from `warn` to + `enabled` in the HelmRelease. + +## Rollback + +Restore the two `config.json` files. ArgoCD recreates the Applications and +re-adopts the resources. Suspend Flux first so they do not fight: + +``` +flux suspend helmrelease fider fider-db -n fider +``` + +Because fider carries no data anyone depends on, `kubectl delete ns fider` +followed by a fresh install from either system is also a legitimate recovery. + +## Deliberately not done here + +- **Secrets stay with ArgoCD.** `fider` (JWT_SECRET, EMAIL_SMTP_PASSWORD) and + `fider-db-creds` come from the standalone `bootstrap-secrets` app, which is + *not* part of the ApplicationSet and is unaffected by the cutover. Moving them + to Flux SOPS needs Azure Key Vault workload identity for `kustomize-controller`, + which cannot be done in git alone. Separate step. +- **`cnpg-operator` stays with ArgoCD** (phase `core`). Flux cannot express + `dependsOn` across to an ArgoCD Application. The operator is already running, + so this is a documented gap, not an ordering bug. +- **No other app is touched.** + +## Known gaps + +- `star-server`, `alaska-rcv` and `discord-bot` still use `devopscoop/charts/app` + under ArgoCD. They are untouched here, but `flux/base/app` is written to be + reusable: a second overlay supplying its own name, image, env and ingress + should be all another app needs. The registry is public (anonymous pull + verified 2026-09-08), so this is a continuity question rather than an access + one — `be16440` notes the chart has already been relocated once, "migrated + from the now-deprecated dedevsecops org". +- `flux/clusters/equalvote/flux-system/` is a placeholder; `flux bootstrap` + writes the real controller manifests there. +- Nothing here has been validated against a live cluster — no `flux` or + `kustomize` CLI is available in the authoring environment. Dry-run with + `flux build kustomization apps --path ./flux/apps` before trusting it. diff --git a/flux/apps/fider/helmrelease-fider-db.yaml b/flux/apps/fider/helmrelease-fider-db.yaml new file mode 100644 index 0000000..4e085c2 --- /dev/null +++ b/flux/apps/fider/helmrelease-fider-db.yaml @@ -0,0 +1,66 @@ +# Translated 1:1 from applications/fider-db/{config.json,values.yaml}. +# Chart version pinned to the revision ArgoCD currently runs, so the first +# `helm upgrade` after adoption is a no-op. +apiVersion: helm.toolkit.fluxcd.io/v2 +kind: HelmRelease +metadata: + name: fider-db + namespace: fider +spec: + interval: 10m + releaseName: fider-db + chart: + spec: + chart: cluster + version: "0.8.1" + sourceRef: + kind: HelmRepository + name: cloudnative-pg + namespace: flux-system + install: + remediation: + retries: 3 + upgrade: + remediation: + retries: 3 + # Start in `warn` while ArgoCD's field ownership is still present on these + # objects. Flip to `enabled` only once ArgoCD has been detached and one clean + # reconcile has been observed, or Flux and ArgoCD will fight over the same + # fields. See flux/README.md, "Cutover". + driftDetection: + mode: warn + # NOTE: cnpg-operator stays under ArgoCD for now (phase `core`). Flux cannot + # express a dependsOn across to an ArgoCD Application; the operator is already + # running, so this is a documented gap rather than an ordering bug. + values: + fullnameOverride: fider-db + type: postgresql + mode: standalone + cluster: + instances: 1 + # With one instance the operator's default PDB (minAvailable: 1) can never be + # satisfied: it sits at 0 allowed disruptions permanently, so node drains hang + # and AKS cluster upgrades stall. Disabling it lets the pod be evicted and + # rescheduled -- a brief outage, but unavoidable for a single-instance cluster. + # https://cloudnative-pg.io/documentation/current/kubernetes_upgrade/#pod-disruption-budgets + enablePDB: false + storage: + size: 1Gi + resources: + requests: + cpu: 10m + memory: 256Mi + limits: + memory: 256Mi + initdb: + database: fider + owner: fider + # Password lives in git (SOPS) rather than being generated by the + # operator. With this set, CNPG does NOT create a fider-db-app secret, + # so anything needing the credentials must read fider-db-creds. + # Only consulted at bootstrap: changing it later will not rotate the + # password on an already-initialized cluster. + secret: + name: fider-db-creds + backups: + enabled: false diff --git a/flux/apps/fider/kustomization.yaml b/flux/apps/fider/kustomization.yaml new file mode 100644 index 0000000..995dba3 --- /dev/null +++ b/flux/apps/fider/kustomization.yaml @@ -0,0 +1,49 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: fider + +resources: + - ../../base/app + - ./helmrelease-fider-db.yaml + +# Stamps app.kubernetes.io/instance everywhere INCLUDING selectors, which is +# what reproduces the live Deployment's immutable selector +# {app.kubernetes.io/name: app, app.kubernetes.io/instance: fider}. +labels: + - pairs: + app.kubernetes.io/instance: fider + includeSelectors: true + +images: + - name: app + newName: getfider/fider + newTag: stable + +# The base names every object `app`. Rename them, and fix the two +# cross-references by hand (a JSON6902 name change does not propagate). +patches: + - target: { kind: ServiceAccount, name: app } + patch: |- + - op: replace + path: /metadata/name + value: fider + - target: { kind: Service, name: app } + patch: |- + - op: replace + path: /metadata/name + value: fider + - target: { kind: Ingress, name: app } + patch: |- + - op: replace + path: /metadata/name + value: fider + - target: { kind: Deployment, name: app } + patch: |- + - op: replace + path: /metadata/name + value: fider + - op: replace + path: /spec/template/spec/serviceAccountName + value: fider + - path: ./patch-deployment.yaml + - path: ./patch-ingress.yaml diff --git a/flux/apps/fider/patch-deployment.yaml b/flux/apps/fider/patch-deployment.yaml new file mode 100644 index 0000000..43efa25 --- /dev/null +++ b/flux/apps/fider/patch-deployment.yaml @@ -0,0 +1,72 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: app +spec: + template: + spec: + containers: + - name: app + env: + - name: BASE_URL + value: "https://feedback.prod.equal.vote" + - name: JWT_SECRET + valueFrom: + secretKeyRef: + name: fider + key: JWT_SECRET + # The password is declared in git (secrets/fider-db-secrets.enc.yaml) and + # handed to CNPG via cluster.initdb.secret, so the operator no longer + # publishes a fider-db-app secret with a ready-made `uri`. Assemble the URL + # here instead, injecting the one stored copy of the password via Kubernetes + # dependent-env expansion -- FIDER_DB_PASSWORD must stay above DATABASE_URL + # for $() to resolve. The password is deliberately alphanumeric: this is + # string interpolation, not URL encoding (see #43). + # No sslmode, so lib/pq defaults to `require`; CNPG always serves TLS. + - name: FIDER_DB_PASSWORD + valueFrom: + secretKeyRef: + name: fider-db-creds + key: password + - name: DATABASE_URL + value: "postgresql://fider:$(FIDER_DB_PASSWORD)@fider-db-rw.fider:5432/fider" + - name: EMAIL_NOREPLY + value: "noreply@bettervoting.com" + - name: EMAIL_SMTP_HOST + value: "smtp.sendgrid.net" + - name: EMAIL_SMTP_PORT + value: "587" + - name: EMAIL_SMTP_USERNAME + value: "apikey" + - name: EMAIL_SMTP_PASSWORD + valueFrom: + secretKeyRef: + name: fider + key: EMAIL_SMTP_PASSWORD + - name: LOG_LEVEL + value: "info" + - name: BLOB_STORAGE + value: "sql" + # Fider calls lingua-go for language detection on every post insert + # (addNewPost -> detectPostLanguage), which lazily loads n-gram models for all + # 18 locales in enum.AllLocales. Measured on getfider/fider:stable + # (dev-67c5282): idle 14MB, first insert 3.1s and +338MB, and the models stay + # resident -- so ~356MB is the real steady state, not a transient spike, and it + # is reached the moment anyone submits a suggestion. At the old 200Mi limit the + # pod was OOMKilled mid-request on every first post, which reads to the user as + # the submit hanging forever. The request matters as much as the limit: at + # 100Mi the scheduler was sizing this pod at under a third of what it holds. + resources: + requests: + cpu: 10m + memory: 384Mi + limits: + memory: 640Mi + livenessProbe: + httpGet: + path: / + port: 3000 + readinessProbe: + httpGet: + path: / + port: 3000 diff --git a/flux/apps/fider/patch-ingress.yaml b/flux/apps/fider/patch-ingress.yaml new file mode 100644 index 0000000..f1166e1 --- /dev/null +++ b/flux/apps/fider/patch-ingress.yaml @@ -0,0 +1,22 @@ +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: app + annotations: + cert-manager.io/cluster-issuer: letsencrypt-prod +spec: + rules: + - host: feedback.prod.equal.vote + http: + paths: + - path: / + pathType: ImplementationSpecific + backend: + service: + name: fider + port: + number: 3000 + tls: + - secretName: fider-tls + hosts: + - feedback.prod.equal.vote diff --git a/flux/apps/kustomization.yaml b/flux/apps/kustomization.yaml new file mode 100644 index 0000000..52b800e --- /dev/null +++ b/flux/apps/kustomization.yaml @@ -0,0 +1,4 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - ./fider diff --git a/flux/base/app/deployment.yaml b/flux/base/app/deployment.yaml new file mode 100644 index 0000000..65f2560 --- /dev/null +++ b/flux/base/app/deployment.yaml @@ -0,0 +1,27 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: app + labels: + app.kubernetes.io/name: app +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: app + template: + metadata: + labels: + app.kubernetes.io/name: app + spec: + serviceAccountName: app + containers: + # Named `app`, matching what the chart emitted. Renaming a container + # is a pod-template change, so leaving it alone keeps adoption quiet. + - name: app + image: app + imagePullPolicy: IfNotPresent + ports: + - name: http + containerPort: 3000 + protocol: TCP diff --git a/flux/base/app/ingress.yaml b/flux/base/app/ingress.yaml new file mode 100644 index 0000000..0baf429 --- /dev/null +++ b/flux/base/app/ingress.yaml @@ -0,0 +1,9 @@ +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: app + labels: + app.kubernetes.io/name: app +spec: + ingressClassName: nginx + rules: [] diff --git a/flux/base/app/kustomization.yaml b/flux/base/app/kustomization.yaml new file mode 100644 index 0000000..953afdf --- /dev/null +++ b/flux/base/app/kustomization.yaml @@ -0,0 +1,14 @@ +# A self-contained replacement for the generic `devopscoop/charts/app` Helm +# chart: the same four objects that chart renders, as plain manifests we own. +# +# The base is deliberately un-named -- every object is `app`, and an overlay +# renames them and stamps `app.kubernetes.io/instance`. Keeping the shipped +# label `app.kubernetes.io/name: app` is not cosmetic: it is half of the live +# Deployment's selector, and selectors are immutable. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - serviceaccount.yaml + - service.yaml + - deployment.yaml + - ingress.yaml diff --git a/flux/base/app/service.yaml b/flux/base/app/service.yaml new file mode 100644 index 0000000..51c5cbc --- /dev/null +++ b/flux/base/app/service.yaml @@ -0,0 +1,15 @@ +apiVersion: v1 +kind: Service +metadata: + name: app + labels: + app.kubernetes.io/name: app +spec: + type: ClusterIP + ports: + - name: http + port: 3000 + targetPort: http + protocol: TCP + selector: + app.kubernetes.io/name: app diff --git a/flux/base/app/serviceaccount.yaml b/flux/base/app/serviceaccount.yaml new file mode 100644 index 0000000..3d3de20 --- /dev/null +++ b/flux/base/app/serviceaccount.yaml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: app + labels: + app.kubernetes.io/name: app diff --git a/flux/clusters/equalvote/apps.yaml b/flux/clusters/equalvote/apps.yaml new file mode 100644 index 0000000..6d7477c --- /dev/null +++ b/flux/clusters/equalvote/apps.yaml @@ -0,0 +1,30 @@ +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: apps + namespace: flux-system +spec: + # Suspended on purpose. `flux bootstrap` must not start reconciling fider + # while ArgoCD still owns those objects -- two controllers server-side-applying + # the same resources fight over field ownership indefinitely. Resuming this is + # the cutover, and it happens only after the ArgoCD Applications are gone: + # + # flux resume kustomization apps + # + # See flux/README.md, "Cutover". + suspend: true + interval: 10m + retryInterval: 1m + timeout: 10m + path: ./flux/apps + prune: true + wait: true + # Charts must resolve before any HelmRelease can install. This one dependency + # is the whole ordering model -- if sources isn't Ready, apps wait, and Flux + # re-evaluates every interval rather than pinning an FSM the way RollingSync + # does (see the step-2 wedge in AGENTS.md / PR #51). + dependsOn: + - name: sources + sourceRef: + kind: GitRepository + name: flux-system diff --git a/flux/clusters/equalvote/flux-system/.gitkeep b/flux/clusters/equalvote/flux-system/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/flux/clusters/equalvote/sources.yaml b/flux/clusters/equalvote/sources.yaml new file mode 100644 index 0000000..3b64a20 --- /dev/null +++ b/flux/clusters/equalvote/sources.yaml @@ -0,0 +1,15 @@ +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: sources + namespace: flux-system +spec: + interval: 10m + retryInterval: 1m + timeout: 5m + path: ./flux/sources + prune: true + wait: true + sourceRef: + kind: GitRepository + name: flux-system diff --git a/flux/sources/helmrepositories.yaml b/flux/sources/helmrepositories.yaml new file mode 100644 index 0000000..4872fa7 --- /dev/null +++ b/flux/sources/helmrepositories.yaml @@ -0,0 +1,15 @@ +# Chart sources for the Flux-managed apps. +# +# Only CloudNativePG is here. The generic `devopscoop/charts/app` wrapper that +# ArgoCD uses for fider is replaced by flux/base/app, so the Flux path pulls no +# third-party application chart at all. +--- +apiVersion: source.toolkit.fluxcd.io/v1 +kind: HelmRepository +metadata: + name: cloudnative-pg + namespace: flux-system +spec: + interval: 1h + url: https://cloudnative-pg.github.io/charts +---