Put fider under Flux, alongside ArgoCD

flux/README.md

@@ -0,0 +1,185 @@
+# Flux, alongside ArgoCD
+
+A Flux translation of one app — `fider` — living next to the ArgoCD config it
+will eventually replace. `fider` is first because it is the only app in the
+cluster where a mistake costs nothing (see [Why fider](#why-fider)).
+
+## Status: inert
+
+Nothing here does anything until someone runs `flux bootstrap`. ArgoCD cannot
+see this directory:
+
+- the `applications` ApplicationSet generator globs `applications/**/config.json`
+- the `bootstrap-cluster` Application syncs `path: applications`
+
+Both are scoped to `applications/`, so `flux/` is invisible to ArgoCD, and this
+PR is safe to merge at any time without changing cluster behaviour. It does not
+remove `applications/fider/config.json` or `applications/fider-db/config.json` —
+that is the cutover, and it is a separate change.
+
+## Why fider
+
+It is publicly served at `feedback.prod.equal.vote` but **not reachable from the
+product** — `grep -rn 'fider\|feedback\.prod' packages/frontend/src packages/backend/src`
+in the bettervoting repo returns nothing. So it is the one app where the blast
+radius of getting a migration wrong is approximately zero.
+
+That makes it worth using to rehearse the *hard* path rather than to dodge it.
+`fider` happens to exercise three of the four blockers in the full migration:
+
+| Blocker | Why fider hits it | Also needed for |
+| --- | --- | --- |
+| Helm adoption | real PVC + CNPG cluster to adopt in place | `postgresql`, `keycloak` |
+| SSA field handover | ArgoCD owns fields as `argocd-controller` | every app |
+
+The third blocker, the `devopscoop` chart, is gone from this path entirely:
+`flux/base/app` reproduces it as plain manifests we own (see below).
+
+## flux/base/app
+
+The Flux path does not use `devopscoop/charts/app`. That chart renders four
+objects — ServiceAccount, Service, Deployment, Ingress — from a values file, and
+`flux/base/app` is those four objects as a kustomize base, with
+`flux/apps/fider` as the overlay.
+
+Verified with `kubectl kustomize flux/apps/fider` against the live cluster: the
+Deployment selector, container name, image, `serviceAccountName`, `env`,
+`resources` and `ports`, the Service selector and ports, and the entire Ingress
+spec all render **identical** to what is running.
+
+Two things worth knowing about the design:
+
+- The label `app.kubernetes.io/name: app` is kept deliberately. It is half of
+  the live Deployment's selector, and selectors are immutable — changing it
+  would force a delete/recreate. The overlay adds
+  `app.kubernetes.io/instance: fider` via `labels` with `includeSelectors: true`,
+  reproducing the live selector exactly.
+- Adoption will cause **one pod rollout**. The rendered pod template drops three
+  Helm bookkeeping labels (`helm.sh/chart`, `app.kubernetes.io/managed-by`,
+  `app.kubernetes.io/version`), which changes the pod-template hash. Harmless
+  here, and worth expecting rather than debugging.
+
+`fider-db` stays a HelmRelease on the upstream CloudNativePG `cluster` chart (pinned to `0.8.1`, matching `applications/fider-db/config.json`) —
+that is a real upstream, not an in-house wrapper, and keeping it preserves the
+Helm-adoption rehearsal that `postgresql` and `keycloak` will need.
+
+## Prerequisites
+
+1. **`flux bootstrap`**, e.g.
+   `flux bootstrap github --owner=Equal-Vote --repository=argocd --path=./flux/clusters/equalvote`.
+   Adds four controllers in `flux-system`. Note the cluster is already firing
+   `KubeMemoryOvercommit`, so budget for the extra requests.
+2. **Make removal non-destructive.** See below.
+
+No registry credentials are needed. The only chart pulled is CloudNativePG's,
+from a public repository.
+
+## ⚠️ Removing an app from the ApplicationSet currently destroys its data
+
+This is true today, for every app, independent of Flux:
+
+```
+ApplicationSet syncPolicy:      (none)   <- no preserveResourcesOnDeletion
+fider / fider-db finalizers:    resources-finalizer.argocd.argoproj.io
+PV pvc-df9a66c8 (Bound, live):  Delete
+PV pvc-00a3f621 (Released):     Retain   <- the OLD volume from the Aug incident
+```
+
+Deleting a `config.json` deletes the Application, the finalizer cascades, and
+ArgoCD deletes the workloads — including the CNPG Cluster and its PVC, whose PV
+then reclaims. The `Retain` policy applied after the August CNPG incident
+protects the *previous* volume, not the one in use since the 08-29 rebuild.
+
+Before any cutover, do one of:
+
+- set `preserveResourcesOnDeletion: true` on the ApplicationSet `syncPolicy`
+  (preferred — fixes this for every app at once), or
+- `kubectl patch pv pvc-df9a66c8-... -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'`, or
+- delete the Application with `--cascade=false` before removing its `config.json`
+
+Verify the *bound* PV is the protected one, not an old `Released` entry.
+
+## Cutover
+
+Ordering is the whole game: **detach from ArgoCD, then let Flux adopt.** Never
+both live — two controllers server-side-applying the same objects will fight
+over field ownership indefinitely. That is why `flux/clusters/equalvote/apps.yaml`
+ships with `suspend: true`: bootstrapping Flux is safe on its own, and resuming
+is the deliberate cutover.
+
+The whole chain, in order — three PRs and four commands:
+
+| Step | Change | Kind |
+| --- | --- | --- |
+| 1 | `preserveResourcesOnDeletion: true` on the ApplicationSet | PR |
+| 2 | this scaffold | PR |
+| 3 | `flux bootstrap` | command |
+| 4 | Helm-adoption stamping (step 3 below) | commands |
+| 5 | remove `applications/fider{,-db}/config.json` | PR |
+| 6 | `flux resume kustomization apps` | command |
+
+1 must precede 5, or removing the config.json destroys the database. 2 must
+precede 3, because bootstrap reconciles from a branch in this repo. 5 must
+precede 6, so the two controllers never overlap.
+
+1. Confirm prerequisites.
+2. Apply the deletion-safety fix above.
+3. Stamp the existing resources so Helm adopts rather than reinstalls. ArgoCD
+   renders charts and applies the manifests, so **no Helm release exists in the
+   cluster** and `helm upgrade --install` would otherwise try a fresh install and
+   collide. On every object the two charts own:
+   ```
+   kubectl -n fider annotate <kind>/<name> \
+     meta.helm.sh/release-name=fider meta.helm.sh/release-namespace=fider
+   kubectl -n fider label <kind>/<name> app.kubernetes.io/managed-by=Helm
+   ```
+   (`release-name=fider-db` for the CNPG `Cluster` and its objects.)
+4. Detach ArgoCD: remove `applications/fider/config.json` and
+   `applications/fider-db/config.json` in a separate PR.
+5. Let Flux reconcile. Because chart versions are pinned to what ArgoCD runs,
+   the resulting `helm upgrade` should be a no-op.
+6. Watch for SSA conflicts against the leftover `argocd-controller` field
+   manager. Once ArgoCD is no longer reconciling these objects the stale
+   `managedFields` entries are harmless, but the first apply may need forcing.
+7. `flux resume kustomization apps`.
+8. Only after one clean reconcile, flip `driftDetection.mode` from `warn` to
+   `enabled` in the HelmRelease.
+
+## Rollback
+
+Restore the two `config.json` files. ArgoCD recreates the Applications and
+re-adopts the resources. Suspend Flux first so they do not fight:
+
+```
+flux suspend helmrelease fider fider-db -n fider
+```
+
+Because fider carries no data anyone depends on, `kubectl delete ns fider`
+followed by a fresh install from either system is also a legitimate recovery.
+
+## Deliberately not done here
+
+- **Secrets stay with ArgoCD.** `fider` (JWT_SECRET, EMAIL_SMTP_PASSWORD) and
+  `fider-db-creds` come from the standalone `bootstrap-secrets` app, which is
+  *not* part of the ApplicationSet and is unaffected by the cutover. Moving them
+  to Flux SOPS needs Azure Key Vault workload identity for `kustomize-controller`,
+  which cannot be done in git alone. Separate step.
+- **`cnpg-operator` stays with ArgoCD** (phase `core`). Flux cannot express
+  `dependsOn` across to an ArgoCD Application. The operator is already running,
+  so this is a documented gap, not an ordering bug.
+- **No other app is touched.**
+
+## Known gaps
+
+- `star-server`, `alaska-rcv` and `discord-bot` still use `devopscoop/charts/app`
+  under ArgoCD. They are untouched here, but `flux/base/app` is written to be
+  reusable: a second overlay supplying its own name, image, env and ingress
+  should be all another app needs. The registry is public (anonymous pull
+  verified 2026-09-08), so this is a continuity question rather than an access
+  one — `be16440` notes the chart has already been relocated once, "migrated
+  from the now-deprecated dedevsecops org".
+- `flux/clusters/equalvote/flux-system/` is a placeholder; `flux bootstrap`
+  writes the real controller manifests there.
+- Nothing here has been validated against a live cluster — no `flux` or
+  `kustomize` CLI is available in the authoring environment. Dry-run with
+  `flux build kustomization apps --path ./flux/apps` before trusting it.

flux/apps/fider/helmrelease-fider-db.yaml

@@ -0,0 +1,66 @@
+# Translated 1:1 from applications/fider-db/{config.json,values.yaml}.
+# Chart version pinned to the revision ArgoCD currently runs, so the first
+# `helm upgrade` after adoption is a no-op.
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  name: fider-db
+  namespace: fider
+spec:
+  interval: 10m
+  releaseName: fider-db
+  chart:
+    spec:
+      chart: cluster
+      version: "0.8.1"
+      sourceRef:
+        kind: HelmRepository
+        name: cloudnative-pg
+        namespace: flux-system
+  install:
+    remediation:
+      retries: 3
+  upgrade:
+    remediation:
+      retries: 3
+  # Start in `warn` while ArgoCD's field ownership is still present on these
+  # objects. Flip to `enabled` only once ArgoCD has been detached and one clean
+  # reconcile has been observed, or Flux and ArgoCD will fight over the same
+  # fields. See flux/README.md, "Cutover".
+  driftDetection:
+    mode: warn
+  # NOTE: cnpg-operator stays under ArgoCD for now (phase `core`). Flux cannot
+  # express a dependsOn across to an ArgoCD Application; the operator is already
+  # running, so this is a documented gap rather than an ordering bug.
+  values:
+    fullnameOverride: fider-db
+    type: postgresql
+    mode: standalone
+    cluster:
+      instances: 1
+      # With one instance the operator's default PDB (minAvailable: 1) can never be
+      # satisfied: it sits at 0 allowed disruptions permanently, so node drains hang
+      # and AKS cluster upgrades stall. Disabling it lets the pod be evicted and
+      # rescheduled -- a brief outage, but unavoidable for a single-instance cluster.
+      # https://cloudnative-pg.io/documentation/current/kubernetes_upgrade/#pod-disruption-budgets
+      enablePDB: false
+      storage:
+        size: 1Gi
+      resources:
+        requests:
+          cpu: 10m
+          memory: 256Mi
+        limits:
+          memory: 256Mi
+      initdb:
+        database: fider
+        owner: fider
+        # Password lives in git (SOPS) rather than being generated by the
+        # operator. With this set, CNPG does NOT create a fider-db-app secret,
+        # so anything needing the credentials must read fider-db-creds.
+        # Only consulted at bootstrap: changing it later will not rotate the
+        # password on an already-initialized cluster.
+        secret:
+          name: fider-db-creds
+    backups:
+      enabled: false

flux/apps/fider/kustomization.yaml

@@ -0,0 +1,49 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: fider
+
+resources:
+  - ../../base/app
+  - ./helmrelease-fider-db.yaml
+
+# Stamps app.kubernetes.io/instance everywhere INCLUDING selectors, which is
+# what reproduces the live Deployment's immutable selector
+# {app.kubernetes.io/name: app, app.kubernetes.io/instance: fider}.
+labels:
+  - pairs:
+      app.kubernetes.io/instance: fider
+    includeSelectors: true
+
+images:
+  - name: app
+    newName: getfider/fider
+    newTag: stable
+
+# The base names every object `app`. Rename them, and fix the two
+# cross-references by hand (a JSON6902 name change does not propagate).
+patches:
+  - target: { kind: ServiceAccount, name: app }
+    patch: |-
+      - op: replace
+        path: /metadata/name
+        value: fider
+  - target: { kind: Service, name: app }
+    patch: |-
+      - op: replace
+        path: /metadata/name
+        value: fider
+  - target: { kind: Ingress, name: app }
+    patch: |-
+      - op: replace
+        path: /metadata/name
+        value: fider
+  - target: { kind: Deployment, name: app }
+    patch: |-
+      - op: replace
+        path: /metadata/name
+        value: fider
+      - op: replace
+        path: /spec/template/spec/serviceAccountName
+        value: fider
+  - path: ./patch-deployment.yaml
+  - path: ./patch-ingress.yaml

flux/apps/fider/patch-deployment.yaml

@@ -0,0 +1,72 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: app
+spec:
+  template:
+    spec:
+      containers:
+        - name: app
+          env:
+            - name: BASE_URL
+              value: "https://feedback.prod.equal.vote"
+            - name: JWT_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: fider
+                  key: JWT_SECRET
+            # The password is declared in git (secrets/fider-db-secrets.enc.yaml) and
+            # handed to CNPG via cluster.initdb.secret, so the operator no longer
+            # publishes a fider-db-app secret with a ready-made `uri`. Assemble the URL
+            # here instead, injecting the one stored copy of the password via Kubernetes
+            # dependent-env expansion -- FIDER_DB_PASSWORD must stay above DATABASE_URL
+            # for $() to resolve. The password is deliberately alphanumeric: this is
+            # string interpolation, not URL encoding (see #43).
+            # No sslmode, so lib/pq defaults to `require`; CNPG always serves TLS.
+            - name: FIDER_DB_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: fider-db-creds
+                  key: password
+            - name: DATABASE_URL
+              value: "postgresql://fider:$(FIDER_DB_PASSWORD)@fider-db-rw.fider:5432/fider"
+            - name: EMAIL_NOREPLY
+              value: "noreply@bettervoting.com"
+            - name: EMAIL_SMTP_HOST
+              value: "smtp.sendgrid.net"
+            - name: EMAIL_SMTP_PORT
+              value: "587"
+            - name: EMAIL_SMTP_USERNAME
+              value: "apikey"
+            - name: EMAIL_SMTP_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: fider
+                  key: EMAIL_SMTP_PASSWORD
+            - name: LOG_LEVEL
+              value: "info"
+            - name: BLOB_STORAGE
+              value: "sql"
+          # Fider calls lingua-go for language detection on every post insert
+          # (addNewPost -> detectPostLanguage), which lazily loads n-gram models for all
+          # 18 locales in enum.AllLocales. Measured on getfider/fider:stable
+          # (dev-67c5282): idle 14MB, first insert 3.1s and +338MB, and the models stay
+          # resident -- so ~356MB is the real steady state, not a transient spike, and it
+          # is reached the moment anyone submits a suggestion. At the old 200Mi limit the
+          # pod was OOMKilled mid-request on every first post, which reads to the user as
+          # the submit hanging forever. The request matters as much as the limit: at
+          # 100Mi the scheduler was sizing this pod at under a third of what it holds.
+          resources:
+            requests:
+              cpu: 10m
+              memory: 384Mi
+            limits:
+              memory: 640Mi
+          livenessProbe:
+            httpGet:
+              path: /
+              port: 3000
+          readinessProbe:
+            httpGet:
+              path: /
+              port: 3000

flux/apps/fider/patch-ingress.yaml

@@ -0,0 +1,22 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: app
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+spec:
+  rules:
+    - host: feedback.prod.equal.vote
+      http:
+        paths:
+          - path: /
+            pathType: ImplementationSpecific
+            backend:
+              service:
+                name: fider
+                port:
+                  number: 3000
+  tls:
+    - secretName: fider-tls
+      hosts:
+        - feedback.prod.equal.vote