From 16d876660aea84f304e0b5709b86b9658915b0ba Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Sun, 15 Nov 2026 02:23:16 +0100 Subject: [PATCH 01/12] feat(customer-identity): run ZITADEL for the business-site portal's customer sign-in Deploy self-hosted ZITADEL in a new production-only `customer-identity` namespace, separate from the infrastructure Dex, for invited business-site customers (business-site#302, ADR 0021). The portal stays switched off. - Upstream chart 10.5.0 with the unmodified upstream image v4.20.3, pinned by digest (business-site ADR 0031). Login V1 only; the Login V2 service, the chart's machine administrator and its secret-writing Role are off. - CNPG `customer-identity-db` (2 instances) creates the database and its owner; `zitadel init zitadel` and `zitadel setup` run as that owner, so nothing needs a superuser. Verified database TLS with only the CA certificate mounted. Coroot Postgres integration (annotations, pg_monitor login, network policy) and daily Barman backups to R2 with 30-day retention. - The master key is generated once into OpenBao by a create-only PushSecret and read back, so a rebuilt cluster keeps the key its restored database needs. New read policy and exact-key seed paths for apps/customer-identity. - One human administrator (the maintainer) with a generated one-time password, a forced password change and forced MFA. Instance policies come from the business-site preparer: no self-registration or public organisation creation, no external or passwordless login, 14-character passwords, short token lifetimes. - accounts.devantler.tech on its own Gateway listener and certificate (the certificate also covers login.devantler.tech for the customer Dex next). - Egress to the business site's SES SMTP endpoint is open for ZITADEL's invitation mail; the SMTP provider is added in the console once #5400 provides the credential. Refs devantler-tech/business-site#302 Co-Authored-By: Claude Opus 5.5 --- docs/dr/crypto-custody.md | 16 ++ .../infrastructure/vault-config/job.yaml | 20 +- ...d-customer-identity-zitadel-masterkey.yaml | 17 ++ .../vault-seed/kustomization.yaml | 3 + ...d-customer-identity-zitadel-masterkey.yaml | 13 ++ ...h-customer-identity-zitadel-masterkey.yaml | 25 ++ .../apps/customer-identity/certificate.yaml | 18 ++ .../cilium-network-policy-coroot.yaml | 24 ++ .../cilium-network-policy.yaml | 62 +++++ .../apps/customer-identity/cluster.yaml | 90 +++++++ .../external-secret-db-backup.yaml | 31 +++ .../external-secret-db-coroot-monitor.yaml | 26 +++ .../external-secret-db-zitadel.yaml | 30 +++ .../external-secret-first-admin.yaml | 21 ++ .../external-secret-masterkey.yaml | 22 ++ .../apps/customer-identity/helm-release.yaml | 220 ++++++++++++++++++ .../customer-identity/helm-repository.yaml | 9 + .../apps/customer-identity/http-route.yaml | 28 +++ .../apps/customer-identity/kustomization.yaml | 26 +++ .../apps/customer-identity/namespace.yaml | 11 + .../apps/customer-identity/object-store.yaml | 27 +++ .../password-db-coroot-monitor.yaml | 12 + .../password-db-zitadel.yaml | 12 + .../password-first-admin.yaml | 15 ++ .../customer-identity/reference-grant.yaml | 16 ++ .../customer-identity/scheduled-backup.yaml | 17 ++ k8s/providers/hetzner/apps/kustomization.yaml | 2 + .../patches/add-devantler-tech-listeners.yaml | 23 ++ 28 files changed, 835 insertions(+), 1 deletion(-) create mode 100644 k8s/bases/infrastructure/vault-seed/external-secret-generated-customer-identity-zitadel-masterkey.yaml create mode 100644 k8s/bases/infrastructure/vault-seed/password-customer-identity-zitadel-masterkey.yaml create mode 100644 k8s/bases/infrastructure/vault-seed/push-secret-push-customer-identity-zitadel-masterkey.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/certificate.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/cilium-network-policy-coroot.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/cilium-network-policy.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/cluster.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/external-secret-db-backup.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/external-secret-db-coroot-monitor.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/external-secret-db-zitadel.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/external-secret-first-admin.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/external-secret-masterkey.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/helm-release.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/helm-repository.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/http-route.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/kustomization.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/namespace.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/object-store.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/password-db-coroot-monitor.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/password-db-zitadel.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/password-first-admin.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/reference-grant.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/scheduled-backup.yaml diff --git a/docs/dr/crypto-custody.md b/docs/dr/crypto-custody.md index 17a833985b..1587985fc3 100644 --- a/docs/dr/crypto-custody.md +++ b/docs/dr/crypto-custody.md @@ -25,6 +25,7 @@ and [openbao.md](openbao.md) (OpenBao-specific DR). | **Hetzner Cloud token** | provision / destroy nodes; reconfigure Cloud LB and firewall | destroy infrastructure; pivot via the LB | mint a new one in the Hetzner console; rotate `HCLOUD_TOKEN` | | **Cloudflare API token** | manage DNS for the platform domain | redirect prod traffic; mint Origin CA certs | mint a new one in the Cloudflare dashboard | | **Actual Budget E2EE password** | client-side key to encrypt/decrypt an Actual budget file | decrypt that budget's financial data | the budget is **permanently unrecoverable** — the server never held the key | +| **ZITADEL master key** | encrypt the customer identity provider's signing keys and stored secrets (`customer-identity` namespace) | with a database copy, decrypt those keys and secrets — rotate them in ZITADEL | restored `customer-identity-db` backups cannot be used; customer accounts are recreated | The rest of this document walks each row in detail. @@ -316,6 +317,21 @@ strongest reason to keep the OpenBao entry backed up. --- +## ZITADEL master key + +ZITADEL (customer sign-in for the business-site portal) encrypts its signing keys and stored +secrets with a 32-character master key. The key is generated once in-cluster and written to +OpenBao `apps/customer-identity/zitadel` (property `masterkey`) by a create-only PushSecret +(`k8s/bases/infrastructure/vault-seed/push-secret-push-customer-identity-zitadel-masterkey.yaml`). +OpenBao is its custody of record: an existing value is never replaced, and +`customer-identity/zitadel-masterkey` is read back from it. + +- **Restore order:** restore OpenBao first ([openbao.md](openbao.md)), then `customer-identity-db` + from its R2 backups. A fresh OpenBao would generate a new key that cannot read the restored database. +- **Rotation:** not supported by ZITADEL for an existing database. Treat the key as permanent. + +--- + ## Cross-references - [`runbook.md`](runbook.md) — operational scenarios (rotation steps, full diff --git a/k8s/bases/infrastructure/vault-config/job.yaml b/k8s/bases/infrastructure/vault-config/job.yaml index dfb403d15d..6dc05d78e7 100644 --- a/k8s/bases/infrastructure/vault-config/job.yaml +++ b/k8s/bases/infrastructure/vault-config/job.yaml @@ -906,6 +906,16 @@ spec: } POLICY + # Customer sign-in (apps/customer-identity, business-site#302) + # reads ZITADEL's master key, and later the customer Dex's + # ZITADEL client, through the shared `external-secrets` + # ClusterSecretStore role. + bao policy write app-customer-identity-readonly - <<'POLICY' + path "secret/data/apps/customer-identity/*" { + capabilities = ["read"] + } + POLICY + # actual-budget reads its Enable Banking bank-sync credential # (application-id + secret-key) from OpenBao via the shared # `external-secrets` ClusterSecretStore role (apps/actual-budget/ @@ -1067,6 +1077,11 @@ spec: path "secret/data/apps/umami/*" { capabilities = ["create", "update", "read"] } + # ZITADEL's master key, generated once by its create-only + # PushSecret (vault-seed). Exact key only. + path "secret/data/apps/customer-identity/zitadel" { + capabilities = ["create", "update", "read"] + } path "secret/data/apps/actual-budget/*" { capabilities = ["create", "update", "read"] } @@ -1128,6 +1143,9 @@ spec: path "secret/metadata/apps/umami/*" { capabilities = ["create", "update", "read"] } + path "secret/metadata/apps/customer-identity/zitadel" { + capabilities = ["create", "update", "read"] + } path "secret/metadata/apps/actual-budget/*" { capabilities = ["create", "update", "read"] } @@ -1163,7 +1181,7 @@ spec: bao write auth/kubernetes/role/external-secrets \ bound_service_account_names=external-secrets \ bound_service_account_namespaces=external-secrets \ - policies=infra-backup-readonly,infra-dns-readonly,infra-monitoring-readonly,infra-hcloud-readonly,infra-tls-readonly,infra-ghcr-readonly,app-fleetdm-readonly,app-wedding-readonly,app-umami-readonly,app-kagent-readonly,app-actual-budget-readonly,app-ascoachingogvaner-readonly,infra-oidc-readonly,vault-seed-write \ + policies=infra-backup-readonly,infra-dns-readonly,infra-monitoring-readonly,infra-hcloud-readonly,infra-tls-readonly,infra-ghcr-readonly,app-fleetdm-readonly,app-wedding-readonly,app-umami-readonly,app-kagent-readonly,app-customer-identity-readonly,app-actual-budget-readonly,app-ascoachingogvaner-readonly,infra-oidc-readonly,vault-seed-write \ ttl=1h # Dedicated role for the fleetdm VaultDynamicSecret generator. diff --git a/k8s/bases/infrastructure/vault-seed/external-secret-generated-customer-identity-zitadel-masterkey.yaml b/k8s/bases/infrastructure/vault-seed/external-secret-generated-customer-identity-zitadel-masterkey.yaml new file mode 100644 index 0000000000..e88ed4b151 --- /dev/null +++ b/k8s/bases/infrastructure/vault-seed/external-secret-generated-customer-identity-zitadel-masterkey.yaml @@ -0,0 +1,17 @@ +--- +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: generated-customer-identity-zitadel-masterkey + namespace: openbao +spec: + refreshInterval: "0" + target: + name: generated-customer-identity-zitadel-masterkey + creationPolicy: Owner + dataFrom: + - sourceRef: + generatorRef: + apiVersion: generators.external-secrets.io/v1alpha1 + kind: Password + name: customer-identity-zitadel-masterkey diff --git a/k8s/bases/infrastructure/vault-seed/kustomization.yaml b/k8s/bases/infrastructure/vault-seed/kustomization.yaml index e924834de9..2b03611eed 100644 --- a/k8s/bases/infrastructure/vault-seed/kustomization.yaml +++ b/k8s/bases/infrastructure/vault-seed/kustomization.yaml @@ -26,6 +26,9 @@ resources: - push-secret-push-umami-admin-password.yaml - external-secret-generated-umami-ascoachingogvaner-password.yaml - push-secret-push-umami-ascoachingogvaner-password.yaml + - password-customer-identity-zitadel-masterkey.yaml + - external-secret-generated-customer-identity-zitadel-masterkey.yaml + - push-secret-push-customer-identity-zitadel-masterkey.yaml - push-secret-seed-r2-credentials.yaml - push-secret-seed-wedding-db-backup-r2.yaml - push-secret-seed-github-app.yaml diff --git a/k8s/bases/infrastructure/vault-seed/password-customer-identity-zitadel-masterkey.yaml b/k8s/bases/infrastructure/vault-seed/password-customer-identity-zitadel-masterkey.yaml new file mode 100644 index 0000000000..57bb59f9ff --- /dev/null +++ b/k8s/bases/infrastructure/vault-seed/password-customer-identity-zitadel-masterkey.yaml @@ -0,0 +1,13 @@ +--- +# ZITADEL requires exactly 32 characters. +apiVersion: generators.external-secrets.io/v1alpha1 +kind: Password +metadata: + name: customer-identity-zitadel-masterkey + namespace: openbao +spec: + length: 32 + digits: 8 + symbols: 0 + noUpper: false + allowRepeat: true diff --git a/k8s/bases/infrastructure/vault-seed/push-secret-push-customer-identity-zitadel-masterkey.yaml b/k8s/bases/infrastructure/vault-seed/push-secret-push-customer-identity-zitadel-masterkey.yaml new file mode 100644 index 0000000000..e01abdc28e --- /dev/null +++ b/k8s/bases/infrastructure/vault-seed/push-secret-push-customer-identity-zitadel-masterkey.yaml @@ -0,0 +1,25 @@ +--- +# ZITADEL's master key (apps/customer-identity/external-secret-masterkey.yaml). +# Create-only: once OpenBao holds a key, a regenerated in-cluster value never +# replaces it, so a rebuilt cluster that restores OpenBao keeps the key its +# restored database was encrypted with (docs/dr/crypto-custody.md). +apiVersion: external-secrets.io/v1alpha1 +kind: PushSecret +metadata: + name: push-customer-identity-zitadel-masterkey + namespace: openbao +spec: + refreshInterval: 1h + updatePolicy: IfNotExists + secretStoreRefs: + - name: openbao + kind: ClusterSecretStore + selector: + secret: + name: generated-customer-identity-zitadel-masterkey + data: + - match: + secretKey: password + remoteRef: + remoteKey: apps/customer-identity/zitadel + property: masterkey diff --git a/k8s/providers/hetzner/apps/customer-identity/certificate.yaml b/k8s/providers/hetzner/apps/customer-identity/certificate.yaml new file mode 100644 index 0000000000..b6a61884ab --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/certificate.yaml @@ -0,0 +1,18 @@ +# DNS-01 certificate for the customer identity names: ZITADEL +# (accounts.devantler.tech) and the portal's customer Dex +# (login.devantler.tech, next change). The Gateway listeners read it through +# reference-grant.yaml. +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: customer-identity + namespace: customer-identity +spec: + secretName: customer-identity-tls + dnsNames: + - accounts.devantler.tech + - login.devantler.tech + issuerRef: + group: cert-manager.io + kind: ClusterIssuer + name: letsencrypt-prod diff --git a/k8s/providers/hetzner/apps/customer-identity/cilium-network-policy-coroot.yaml b/k8s/providers/hetzner/apps/customer-identity/cilium-network-policy-coroot.yaml new file mode 100644 index 0000000000..e080ec653e --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/cilium-network-policy-coroot.yaml @@ -0,0 +1,24 @@ +# Let Coroot's cluster-agent (observability namespace) reach the +# customer-identity-db instances on 5432 for its Postgres integration +# (cluster.yaml). Additive: it admits only the cluster-agent, and only to the +# database pods. +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: allow-coroot-postgres-scrape + namespace: customer-identity +spec: + endpointSelector: + matchLabels: + cnpg.io/cluster: customer-identity-db + ingress: + - fromEndpoints: + - matchLabels: + k8s:io.kubernetes.pod.namespace: observability + app.kubernetes.io/managed-by: coroot-operator + app.kubernetes.io/part-of: coroot + app.kubernetes.io/component: coroot-cluster-agent + toPorts: + - ports: + - port: "5432" + protocol: TCP diff --git a/k8s/providers/hetzner/apps/customer-identity/cilium-network-policy.yaml b/k8s/providers/hetzner/apps/customer-identity/cilium-network-policy.yaml new file mode 100644 index 0000000000..6c97224538 --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/cilium-network-policy.yaml @@ -0,0 +1,62 @@ +# ZITADEL and its database. Only the Gateway reaches ZITADEL (8080); only +# ZITADEL, the other instance and the CNPG operator reach the database. +# Egress: the namespace itself, the API server (CNPG instance manager), R2 +# for backups, and the business site's SES SMTP endpoint for ZITADEL's +# invitation and recovery mail (platform#5400). +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: allow-customer-identity + namespace: customer-identity +spec: + endpointSelector: {} + ingress: + - fromEntities: + - ingress + toPorts: + - ports: + - port: "8080" + protocol: TCP + - fromEndpoints: + - matchLabels: + k8s:io.kubernetes.pod.namespace: customer-identity + - fromEndpoints: + - matchLabels: + k8s:io.kubernetes.pod.namespace: cnpg-system + toPorts: + - ports: + - port: "8000" + protocol: TCP + - port: "5432" + protocol: TCP + egress: + - toEndpoints: + - matchLabels: + k8s:io.kubernetes.pod.namespace: customer-identity + - toEntities: + - kube-apiserver + - toFQDNs: + - matchPattern: "*.r2.cloudflarestorage.com" + toPorts: + - ports: + - port: "443" + protocol: TCP + - toFQDNs: + - matchName: email-smtp.eu-central-1.amazonaws.com + toPorts: + - ports: + - port: "587" + protocol: TCP + - toEndpoints: + - matchLabels: + k8s:io.kubernetes.pod.namespace: kube-system + k8s-app: kube-dns + toPorts: + - ports: + - port: "53" + protocol: UDP + - port: "53" + protocol: TCP + rules: + dns: + - matchPattern: "*" diff --git a/k8s/providers/hetzner/apps/customer-identity/cluster.yaml b/k8s/providers/hetzner/apps/customer-identity/cluster.yaml new file mode 100644 index 0000000000..84ba520db9 --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/cluster.yaml @@ -0,0 +1,90 @@ +# ZITADEL's PostgreSQL store: every customer account, credential, MFA factor +# and session. Not rebuildable, so it is backed up off-cluster to R2 +# (object-store.yaml, scheduled-backup.yaml) and Flux never prunes it. +# +# CNPG creates the `zitadel` database and its owner login with the generated +# password (external-secret-db-zitadel.yaml). ZITADEL connects as that owner +# and creates its own schemas (`zitadel init zitadel`, then `zitadel setup`), +# so no ZITADEL step needs a superuser (helm-release.yaml). +apiVersion: postgresql.cnpg.io/v1 +kind: Cluster +metadata: + name: customer-identity-db + namespace: customer-identity + annotations: + kustomize.toolkit.fluxcd.io/prune: disabled +spec: + # renovate: datasource=docker depName=ghcr.io/cloudnative-pg/postgresql + imageName: ghcr.io/cloudnative-pg/postgresql:18.7-system-trixie + instances: 2 + # The platform's storage tier for database pods (as every production + # database declares). CNPG ignores a priorityClassName change, so the + # matching env var is what rolls the instances when it changes. + priorityClassName: hcloud-volume-workload + env: + - name: PLATFORM_DATABASE_PRIORITY_CLASS + value: hcloud-volume-workload + storage: + size: 5Gi + storageClass: longhorn-wffc + # Coroot's Postgres integration, as every platform database has. CNPG + # copies these annotations onto the instance pods, where Coroot's + # cluster-agent discovers them; it logs in as the coroot_monitor role below, + # and only it may reach the instances for this (cilium-network-policy-coroot.yaml). + inheritedMetadata: + annotations: + coroot.com/postgres-scrape: "true" + coroot.com/postgres-scrape-port: "5432" + coroot.com/postgres-scrape-credentials-secret-name: customer-identity-db-coroot-monitor + coroot.com/postgres-scrape-credentials-secret-username-key: username + coroot.com/postgres-scrape-credentials-secret-password-key: password + coroot.com/postgres-scrape-param-sslmode: require + # Same quorum replication as the platform's other databases: a promoted + # standby always holds every committed transaction, and writes continue if + # the standby is down. + postgresql: + synchronous: + method: any + number: 1 + dataDurability: preferred + # Any pg_stat_statements.* parameter makes CNPG preload the extension and + # create it in every database, which feeds Coroot's per-query view. + parameters: + pg_stat_statements.max: "10000" + pg_stat_statements.track: all + bootstrap: + initdb: + database: zitadel + owner: zitadel + secret: + name: customer-identity-db-zitadel + postInitApplicationSQL: + - REVOKE CREATE ON SCHEMA public FROM PUBLIC + managed: + roles: + # Keeps the owner's password equal to its Secret after bootstrap. + - name: zitadel + ensure: present + login: true + passwordSecret: + name: customer-identity-db-zitadel + # Coroot's read-only monitoring login: pg_monitor only, no access to + # ZITADEL's data. + - name: coroot_monitor + ensure: present + login: true + inRoles: + - pg_monitor + passwordSecret: + name: customer-identity-db-coroot-monitor + resources: + requests: + cpu: 50m + memory: 256Mi + limits: + memory: 512Mi + plugins: + - name: barman-cloud.cloudnative-pg.io + isWALArchiver: true + parameters: + barmanObjectName: customer-identity-db diff --git a/k8s/providers/hetzner/apps/customer-identity/external-secret-db-backup.yaml b/k8s/providers/hetzner/apps/customer-identity/external-secret-db-backup.yaml new file mode 100644 index 0000000000..55ec1111f2 --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/external-secret-db-backup.yaml @@ -0,0 +1,31 @@ +# R2 credentials for customer-identity-db's Barman Cloud backups +# (object-store.yaml). The shared backup key from OpenBao +# infrastructure/backup/r2, the same one Velero and umami-db use; CNPG reads +# s3Credentials only from the Cluster's own namespace. +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: customer-identity-db-backup-r2 + namespace: customer-identity +spec: + refreshInterval: 1h + secretStoreRef: + name: openbao + kind: ClusterSecretStore + target: + name: customer-identity-db-backup-r2 + creationPolicy: Owner + template: + data: + ACCESS_KEY_ID: "{{ .ACCESS_KEY_ID }}" + SECRET_ACCESS_KEY: "{{ .SECRET_ACCESS_KEY }}" + REGION: "auto" + data: + - secretKey: ACCESS_KEY_ID + remoteRef: + key: infrastructure/backup/r2 + property: access_key_id + - secretKey: SECRET_ACCESS_KEY + remoteRef: + key: infrastructure/backup/r2 + property: secret_access_key diff --git a/k8s/providers/hetzner/apps/customer-identity/external-secret-db-coroot-monitor.yaml b/k8s/providers/hetzner/apps/customer-identity/external-secret-db-coroot-monitor.yaml new file mode 100644 index 0000000000..2912543f01 --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/external-secret-db-coroot-monitor.yaml @@ -0,0 +1,26 @@ +# Coroot's read-only monitoring login for customer-identity-db (pg_monitor only, +# no access to ZITADEL's data). Read only by the scrape annotations in cluster.yaml. +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: customer-identity-db-coroot-monitor + namespace: customer-identity +spec: + refreshInterval: "0" + target: + name: customer-identity-db-coroot-monitor + creationPolicy: Owner + template: + type: kubernetes.io/basic-auth + metadata: + labels: + cnpg.io/reload: "true" + data: + username: coroot_monitor + password: "{{ .password }}" + dataFrom: + - sourceRef: + generatorRef: + apiVersion: generators.external-secrets.io/v1alpha1 + kind: Password + name: customer-identity-db-coroot-monitor diff --git a/k8s/providers/hetzner/apps/customer-identity/external-secret-db-zitadel.yaml b/k8s/providers/hetzner/apps/customer-identity/external-secret-db-zitadel.yaml new file mode 100644 index 0000000000..bfdd640e41 --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/external-secret-db-zitadel.yaml @@ -0,0 +1,30 @@ +# The `zitadel` database owner login. Generated once in-cluster: CNPG creates +# the role and the database with it (cluster.yaml), and ZITADEL reads the same +# password (helm-release.yaml). ZITADEL's state lives in this database and its +# off-cluster backup, so a regenerated password is only a rotation. +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: customer-identity-db-zitadel + namespace: customer-identity +spec: + refreshInterval: "0" + target: + name: customer-identity-db-zitadel + creationPolicy: Owner + template: + engineVersion: v2 + # CNPG reads a role's password from a basic-auth Secret. + type: kubernetes.io/basic-auth + metadata: + labels: + cnpg.io/reload: "true" + data: + username: zitadel + password: "{{ .password }}" + dataFrom: + - sourceRef: + generatorRef: + apiVersion: generators.external-secrets.io/v1alpha1 + kind: Password + name: customer-identity-db-zitadel diff --git a/k8s/providers/hetzner/apps/customer-identity/external-secret-first-admin.yaml b/k8s/providers/hetzner/apps/customer-identity/external-secret-first-admin.yaml new file mode 100644 index 0000000000..9a4a186c77 --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/external-secret-first-admin.yaml @@ -0,0 +1,21 @@ +# One-time initial password for the instance's first administrator, the +# maintainer (helm-release.yaml FirstInstance). ZITADEL uses it only when it +# creates the instance and forces a change at the first sign-in, where MFA +# enrolment is also required. Read it once with an administrator kubeconfig: +# kubectl -n customer-identity get secret zitadel-first-admin -o jsonpath='{.data.password}' | base64 -d +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: zitadel-first-admin + namespace: customer-identity +spec: + refreshInterval: "0" + target: + name: zitadel-first-admin + creationPolicy: Owner + dataFrom: + - sourceRef: + generatorRef: + apiVersion: generators.external-secrets.io/v1alpha1 + kind: Password + name: zitadel-first-admin diff --git a/k8s/providers/hetzner/apps/customer-identity/external-secret-masterkey.yaml b/k8s/providers/hetzner/apps/customer-identity/external-secret-masterkey.yaml new file mode 100644 index 0000000000..c49370695b --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/external-secret-masterkey.yaml @@ -0,0 +1,22 @@ +# ZITADEL's master key: it encrypts the signing keys and secrets stored in the +# database, so the database is unusable without it. Generated once into OpenBao +# by infrastructure/vault-seed (create-only), which makes OpenBao its custody +# of record: a rebuilt cluster restores OpenBao first and gets the same key. +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: zitadel-masterkey + namespace: customer-identity +spec: + refreshInterval: 1h + secretStoreRef: + name: openbao + kind: ClusterSecretStore + target: + name: zitadel-masterkey + creationPolicy: Owner + data: + - secretKey: masterkey + remoteRef: + key: apps/customer-identity/zitadel + property: masterkey diff --git a/k8s/providers/hetzner/apps/customer-identity/helm-release.yaml b/k8s/providers/hetzner/apps/customer-identity/helm-release.yaml new file mode 100644 index 0000000000..3884fc7f5b --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/helm-release.yaml @@ -0,0 +1,220 @@ +# Self-hosted ZITADEL for invited business-site customers (business-site#302, +# ADR 0021). The official upstream image, unmodified and pinned by digest +# (business-site ADR 0031): upstream does not sign it, so admission treats it +# as a third-party image. +# +# Shape: +# - Login V1, embedded in the API server (ADR 0021). The chart's separate +# Login V2 service and its API credential stay off. +# - No superuser: CNPG creates the database and its owner (cluster.yaml), and +# `zitadel init zitadel` and `zitadel setup` run as that owner. +# - No machine administrator: the chart's default machine user and the +# kubectl containers and secret-writing Role that export its key are +# dropped. The only bootstrap identity is one human administrator (the +# maintainer), with a generated one-time password that must be changed at +# the first sign-in, where the login policy also forces MFA. +# - TLS ends at the platform Gateway, as for every platform app; the Gateway +# reaches ZITADEL over h2c inside the WireGuard-encrypted cluster network. +# +# The instance policies come from the business-site preparer +# (`go run ./cmd/customer-identity-config --prepare`, server/CUSTOMER-IDENTITY.md): +# no self-registration or public organisation creation, no external or +# passwordless login, MFA always, 14-character passwords, short-lived tokens. +# ZITADEL applies DefaultInstance and FirstInstance only when it creates the +# instance; later changes go through the console or its API. +# +# Mail: ZITADEL keeps its SMTP provider in its database, so it is added once in +# the console when the business site's SES sender exists (platform#5400): +# email-smtp.eu-central-1.amazonaws.com:587 with STARTTLS, the credential in +# OpenBao apps/business-site/smtp, sender no-reply@devantler.tech. Egress to +# that host is already open (cilium-network-policy.yaml). +apiVersion: helm.toolkit.fluxcd.io/v2 +kind: HelmRelease +metadata: + name: zitadel + namespace: customer-identity + labels: + helm.toolkit.fluxcd.io/remediation: enabled +spec: + interval: 10m0s + timeout: 10m + chart: + spec: + chart: zitadel + version: 10.5.0 + sourceRef: + kind: HelmRepository + name: zitadel + values: + replicaCount: 2 + image: + repository: ghcr.io/zitadel/zitadel + tag: v4.20.3@sha256:1b30fea7ed88826dc0a98dacee4e2f2c8b05201fd26f5e4cb5386615b29f7638 + login: + enabled: false + # Only the `zitadel` phase of `zitadel init` (its schemas and tables), + # which runs as the database owner. The database, role and grant phases + # need a superuser and are CNPG's job here (cluster.yaml). + initJob: + command: zitadel + activeDeadlineSeconds: 600 + resources: + requests: + cpu: 100m + memory: 128Mi + limits: + cpu: "1" + memory: 256Mi + # Its post-delete hook only removes the machine-key Secrets this release + # never creates. + cleanupJob: + enabled: false + setupJob: + activeDeadlineSeconds: 600 + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: "1" + memory: 512Mi + resources: + requests: + cpu: 50m + memory: 192Mi + limits: + cpu: "1" + memory: 512Mi + pdb: + enabled: true + maxUnavailable: 1 + podSecurityContext: + runAsNonRoot: true + runAsUser: 65532 + runAsGroup: 65532 + fsGroup: 65532 + seccompProfile: + type: RuntimeDefault + securityContext: + runAsNonRoot: true + runAsUser: 65532 + runAsGroup: 65532 + readOnlyRootFilesystem: true + privileged: false + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + seccompProfile: + type: RuntimeDefault + # Only the CA certificate: the CNPG CA Secret also holds the CA's private + # key, which the chart's dbSslCaCrtSecret option would mount as well. + extraVolumes: + - name: db-ca + secret: + secretName: customer-identity-db-ca + defaultMode: 0440 + items: + - key: ca.crt + path: ca.crt + extraVolumeMounts: + - name: db-ca + mountPath: /db-ca + readOnly: true + # Rendered into the setup job and the server alike. + env: + - name: ZITADEL_DATABASE_POSTGRES_USER_PASSWORD + valueFrom: + secretKeyRef: + name: customer-identity-db-zitadel + key: password + - name: ZITADEL_DATABASE_POSTGRES_USER_SSL_ROOTCERT + value: /db-ca/ca.crt + - name: ZITADEL_FIRSTINSTANCE_ORG_HUMAN_PASSWORD + valueFrom: + secretKeyRef: + name: zitadel-first-admin + key: password + zitadel: + masterkeySecretName: zitadel-masterkey + configmapConfig: + ExternalDomain: accounts.devantler.tech + ExternalPort: 443 + ExternalSecure: true + TLS: + Enabled: false + Database: + Postgres: + Host: customer-identity-db-rw.customer-identity.svc + Port: 5432 + Database: zitadel + MaxOpenConns: 10 + MaxIdleConns: 5 + User: + Username: zitadel + SSL: + Mode: verify-full + FirstInstance: + InstanceName: Devantler Tech + DefaultLanguage: en + Org: + Name: Devantler Tech + # Drops the chart's default machine administrator (see above). + Machine: null + Human: + UserName: ned + FirstName: Nikolai + LastName: Damm + # The maintainer's own published company address. Marked + # verified so the first sign-in does not wait for mail, which + # cannot be sent until the SMTP provider is added. Customers are + # never pre-verified (ADR 0021). + Email: + Address: ned@devantler.tech + Verified: true + PreferredLanguage: en + PasswordChangeRequired: true + DefaultInstance: + Features: + LoginV2: + Required: false + LockoutPolicy: + MaxOTPAttempts: 5 + MaxPasswordAttempts: 5 + ShouldShowLockoutFailure: false + LoginPolicy: + AllowDomainDiscovery: false + AllowExternalIDP: false + AllowRegister: false + AllowUsernamePassword: true + DefaultRedirectURI: https://portal.devantler.tech/portal + DisableLoginWithPhone: true + ExternalLoginCheckLifetime: 15m + ForceMFA: true + ForceMFALocalOnly: false + HidePasswordReset: false + IgnoreUnknownUsernames: true + MfaInitSkipLifetime: 0s + MultiFactorCheckLifetime: 15m + PasswordCheckLifetime: 15m + PasswordlessType: 0 + SecondFactorCheckLifetime: 15m + OIDCSettings: + AccessTokenLifetime: 5m + IdTokenLifetime: 5m + RefreshTokenExpiration: 8h + RefreshTokenIdleExpiration: 15m + PasswordComplexityPolicy: + MinLength: 14 + Restrictions: + DisallowPublicOrgRegistration: true + # The Devantler Tech look (near-black with #39ff14). The logo, font + # and watermark are set in the console (Branding). + LabelPolicy: + PrimaryColor: "#1f7a0f" + BackgroundColor: "#fafafa" + FontColor: "#111111" + PrimaryColorDark: "#39ff14" + BackgroundColorDark: "#0b0b0b" + FontColorDark: "#ededed" + HideLoginNameSuffix: true diff --git a/k8s/providers/hetzner/apps/customer-identity/helm-repository.yaml b/k8s/providers/hetzner/apps/customer-identity/helm-repository.yaml new file mode 100644 index 0000000000..050427d2ed --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/helm-repository.yaml @@ -0,0 +1,9 @@ +apiVersion: source.toolkit.fluxcd.io/v1 +kind: HelmRepository +metadata: + name: zitadel + namespace: customer-identity +spec: + # ZITADEL's official chart repository. + url: https://charts.zitadel.com + interval: 60m diff --git a/k8s/providers/hetzner/apps/customer-identity/http-route.yaml b/k8s/providers/hetzner/apps/customer-identity/http-route.yaml new file mode 100644 index 0000000000..b0b6775643 --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/http-route.yaml @@ -0,0 +1,28 @@ +# accounts.devantler.tech: ZITADEL's sign-in, invitation, recovery and MFA pages +# and its console. A customer-facing name on the company domain, because +# customers see it whenever they sign in; once the instance exists it is +# ZITADEL's fixed external domain. The listener and certificate are +# infrastructure/patches/add-devantler-tech-listeners.yaml and certificate.yaml; +# external-dns creates the record from this route. +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: zitadel + namespace: customer-identity +spec: + parentRefs: + - name: platform + namespace: kube-system + sectionName: https-accounts-devantler-tech + hostnames: + - accounts.devantler.tech + rules: + - filters: + - type: ResponseHeaderModifier + responseHeaderModifier: + set: + - name: Strict-Transport-Security + value: max-age=63072000; includeSubDomains; preload + backendRefs: + - name: zitadel + port: 8080 diff --git a/k8s/providers/hetzner/apps/customer-identity/kustomization.yaml b/k8s/providers/hetzner/apps/customer-identity/kustomization.yaml new file mode 100644 index 0000000000..127a67cfc1 --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/kustomization.yaml @@ -0,0 +1,26 @@ +# Customer sign-in for the business-site portal (business-site#302): ZITADEL +# with its own CNPG database. Production-only, like the business-site tenant +# it serves. The portal stays switched off until #302's checks pass. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - namespace.yaml + - password-db-zitadel.yaml + - external-secret-db-zitadel.yaml + # Coroot's Postgres integration for customer-identity-db. + - password-db-coroot-monitor.yaml + - external-secret-db-coroot-monitor.yaml + - cilium-network-policy-coroot.yaml + - external-secret-db-backup.yaml + - object-store.yaml + - cluster.yaml + - scheduled-backup.yaml + - external-secret-masterkey.yaml + - password-first-admin.yaml + - external-secret-first-admin.yaml + - helm-repository.yaml + - helm-release.yaml + - cilium-network-policy.yaml + - certificate.yaml + - reference-grant.yaml + - http-route.yaml diff --git a/k8s/providers/hetzner/apps/customer-identity/namespace.yaml b/k8s/providers/hetzner/apps/customer-identity/namespace.yaml new file mode 100644 index 0000000000..4f2ca5872a --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/namespace.yaml @@ -0,0 +1,11 @@ +# Customer sign-in for the business-site portal (devantler-tech/business-site#302, +# ADR 0021): self-hosted ZITADEL now, and the portal's own customer Dex next. +# Kept apart from the platform's infrastructure identity (the `dex` namespace): +# separate issuers, clients, secrets, administration and stores. +apiVersion: v1 +kind: Namespace +metadata: + name: customer-identity + labels: + pod-security.kubernetes.io/enforce: restricted + pod-security.kubernetes.io/enforce-version: latest diff --git a/k8s/providers/hetzner/apps/customer-identity/object-store.yaml b/k8s/providers/hetzner/apps/customer-identity/object-store.yaml new file mode 100644 index 0000000000..6b201bdbe5 --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/object-store.yaml @@ -0,0 +1,27 @@ +# Off-cluster backups for customer-identity-db: continuous WAL archiving plus the +# daily base backup in scheduled-backup.yaml, so customer accounts can be +# restored to any point in the last 30 days. Same R2 bucket and layout as umami-db. +apiVersion: barmancloud.cnpg.io/v1 +kind: ObjectStore +metadata: + name: customer-identity-db + namespace: customer-identity +spec: + retentionPolicy: 30d + configuration: + destinationPath: s3://${r2_bucket}/cnpg/customer-identity-db + endpointURL: ${r2_endpoint} + s3Credentials: + accessKeyId: + name: customer-identity-db-backup-r2 + key: ACCESS_KEY_ID + secretAccessKey: + name: customer-identity-db-backup-r2 + key: SECRET_ACCESS_KEY + region: + name: customer-identity-db-backup-r2 + key: REGION + wal: + compression: gzip + data: + compression: gzip diff --git a/k8s/providers/hetzner/apps/customer-identity/password-db-coroot-monitor.yaml b/k8s/providers/hetzner/apps/customer-identity/password-db-coroot-monitor.yaml new file mode 100644 index 0000000000..b8efe0ceca --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/password-db-coroot-monitor.yaml @@ -0,0 +1,12 @@ +--- +apiVersion: generators.external-secrets.io/v1alpha1 +kind: Password +metadata: + name: customer-identity-db-coroot-monitor + namespace: customer-identity +spec: + length: 32 + digits: 8 + symbols: 0 + noUpper: false + allowRepeat: true diff --git a/k8s/providers/hetzner/apps/customer-identity/password-db-zitadel.yaml b/k8s/providers/hetzner/apps/customer-identity/password-db-zitadel.yaml new file mode 100644 index 0000000000..0ffc0c244a --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/password-db-zitadel.yaml @@ -0,0 +1,12 @@ +--- +apiVersion: generators.external-secrets.io/v1alpha1 +kind: Password +metadata: + name: customer-identity-db-zitadel + namespace: customer-identity +spec: + length: 32 + digits: 8 + symbols: 0 + noUpper: false + allowRepeat: true diff --git a/k8s/providers/hetzner/apps/customer-identity/password-first-admin.yaml b/k8s/providers/hetzner/apps/customer-identity/password-first-admin.yaml new file mode 100644 index 0000000000..c9704884c1 --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/password-first-admin.yaml @@ -0,0 +1,15 @@ +--- +# Meets ZITADEL's password policy (helm-release.yaml: at least 14 characters +# with upper and lower case, a digit and a symbol). +apiVersion: generators.external-secrets.io/v1alpha1 +kind: Password +metadata: + name: zitadel-first-admin + namespace: customer-identity +spec: + length: 32 + digits: 6 + symbols: 4 + symbolCharacters: "-_.!" + noUpper: false + allowRepeat: true diff --git a/k8s/providers/hetzner/apps/customer-identity/reference-grant.yaml b/k8s/providers/hetzner/apps/customer-identity/reference-grant.yaml new file mode 100644 index 0000000000..738a25f3da --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/reference-grant.yaml @@ -0,0 +1,16 @@ +# Lets the shared platform Gateway (kube-system) terminate TLS for the customer +# identity listeners with certificate.yaml's Secret, and nothing else. +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: ReferenceGrant +metadata: + name: platform-gateway-customer-identity-tls + namespace: customer-identity +spec: + from: + - group: gateway.networking.k8s.io + kind: Gateway + namespace: kube-system + to: + - group: "" + kind: Secret + name: customer-identity-tls diff --git a/k8s/providers/hetzner/apps/customer-identity/scheduled-backup.yaml b/k8s/providers/hetzner/apps/customer-identity/scheduled-backup.yaml new file mode 100644 index 0000000000..f6f06e5c07 --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/scheduled-backup.yaml @@ -0,0 +1,17 @@ +# Daily base backup of customer-identity-db to R2 (object-store.yaml). The first +# runs as soon as the cluster exists. +apiVersion: postgresql.cnpg.io/v1 +kind: ScheduledBackup +metadata: + name: customer-identity-db-daily + namespace: customer-identity +spec: + # CNPG cron is 6-field (seconds first): 03:30 UTC daily, after umami-db. + schedule: "0 30 3 * * *" + immediate: true + backupOwnerReference: self + method: plugin + pluginConfiguration: + name: barman-cloud.cloudnative-pg.io + cluster: + name: customer-identity-db diff --git a/k8s/providers/hetzner/apps/kustomization.yaml b/k8s/providers/hetzner/apps/kustomization.yaml index b3f71d2a0c..4ad405de11 100644 --- a/k8s/providers/hetzner/apps/kustomization.yaml +++ b/k8s/providers/hetzner/apps/kustomization.yaml @@ -10,6 +10,8 @@ resources: - khoist-documentation/ # Fetch/verify the exact reviewed publication only; no tenant application. - business-site/ + # Customer sign-in for the business-site portal (ZITADEL, business-site#302). + - customer-identity/ # Prod-only tenant: the UniFi network managed declaratively through Crossplane # Managed Resources (providers/hetzner/apps/unifi/). Lives in the hetzner apps # overlay because its provider and the controller API it reaches are diff --git a/k8s/providers/hetzner/infrastructure/patches/add-devantler-tech-listeners.yaml b/k8s/providers/hetzner/infrastructure/patches/add-devantler-tech-listeners.yaml index 0982301794..e3f470cf01 100644 --- a/k8s/providers/hetzner/infrastructure/patches/add-devantler-tech-listeners.yaml +++ b/k8s/providers/hetzner/infrastructure/patches/add-devantler-tech-listeners.yaml @@ -81,3 +81,26 @@ kinds: - group: gateway.networking.k8s.io kind: HTTPRoute +# ZITADEL's customer sign-in name (apps/customer-identity/http-route.yaml), with +# the customer-identity namespace's own certificate +# (apps/customer-identity/certificate.yaml, read through its reference-grant.yaml). +- op: add + path: /spec/listeners/- + value: + name: https-accounts-devantler-tech + hostname: accounts.devantler.tech + port: 443 + protocol: HTTPS + tls: + mode: Terminate + certificateRefs: + - kind: Secret + group: "" + namespace: customer-identity + name: customer-identity-tls + allowedRoutes: + namespaces: + from: All + kinds: + - group: gateway.networking.k8s.io + kind: HTTPRoute From 2f9b9234e13c869604829a47012278dd67c38f4f Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Sun, 15 Nov 2026 02:33:02 +0100 Subject: [PATCH 02/12] test(authorization): approve the ZITADEL HelmRelease and HelmRepository The two new documents in apps/customer-identity enter the rendered authorization surface. Both are namespace-scoped to customer-identity and grant no RBAC: the HelmRelease drops the chart's machine-key Role, and the HelmRepository is the upstream chart index. Co-Authored-By: Claude Opus 5.5 --- scripts/validate-eks-ci-role-policy/approved-surface.txt | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/scripts/validate-eks-ci-role-policy/approved-surface.txt b/scripts/validate-eks-ci-role-policy/approved-surface.txt index 81fe16a525..431d3ec3a3 100644 --- a/scripts/validate-eks-ci-role-policy/approved-surface.txt +++ b/scripts/validate-eks-ci-role-policy/approved-surface.txt @@ -60,6 +60,8 @@ helm.toolkit.fluxcd.io/v2|HelmRelease|cnpg-system|plugin-barman-cloud 74682cb45a helm.toolkit.fluxcd.io/v2|HelmRelease|crossplane-system|crossplane fb717f7c9adb21b0f772e6a84554900e568a2675c91d375d1bb6ee76c2a0b1f3 +helm.toolkit.fluxcd.io/v2|HelmRelease|customer-identity|zitadel 9c0c9f431282b3053bbebee8798367240c186e545a5d55e7b4770ad8cdd44495 + helm.toolkit.fluxcd.io/v2|HelmRelease|dex|dex e2cbe6e6b5ace19ebded8d0e75f508277dd450c079ed51062d13649e148904b9 helm.toolkit.fluxcd.io/v2|HelmRelease|external-dns|external-dns 107eaf8ad3d74abff5c6b1c6d7ca1b49ac75829fe425672f81cdc1e799457511 @@ -418,6 +420,8 @@ source.toolkit.fluxcd.io/v1|HelmRepository|cnpg-system|cloudnative-pg 2ca8952aaa source.toolkit.fluxcd.io/v1|HelmRepository|crossplane-system|crossplane f47ce87b7611344b0d38ff450e6f789a60711bd657a2ff063253c5fcbd30e754 +source.toolkit.fluxcd.io/v1|HelmRepository|customer-identity|zitadel f156b723dae5eac81d6b84453a416739de755378d12a665970576b0ce199b219 + source.toolkit.fluxcd.io/v1|HelmRepository|dex|dex d251b34144fb4d376b47526e24b3520a1cc696159aa72f1a8c06fa67aadbda7f source.toolkit.fluxcd.io/v1|HelmRepository|external-dns|external-dns 81022de05d792a296ac5ce122fd92258c7cd7ac85ce62aa703aa12fdb0ac1326 From d85cd0dfe1b12f376b4a9c5792268ed12dba9d2c Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Sun, 15 Nov 2026 02:38:26 +0100 Subject: [PATCH 03/12] fix(customer-identity): alert on failed HelmReleases in customer-identity The reconciliation Alert must watch every namespace holding a HelmRelease, or a failed ZITADEL upgrade would report nothing. Co-Authored-By: Claude Opus 5.5 --- .../hetzner/infrastructure/flux-notifications/alert.yaml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/k8s/providers/hetzner/infrastructure/flux-notifications/alert.yaml b/k8s/providers/hetzner/infrastructure/flux-notifications/alert.yaml index a16fb412e3..59f54422a2 100644 --- a/k8s/providers/hetzner/infrastructure/flux-notifications/alert.yaml +++ b/k8s/providers/hetzner/infrastructure/flux-notifications/alert.yaml @@ -118,6 +118,9 @@ spec: - kind: HelmRelease name: "*" namespace: crossplane-system + - kind: HelmRelease + name: "*" + namespace: customer-identity - kind: HelmRelease name: "*" namespace: data-product-controller From fe5d6c8e0bc3604eedd9dcf9cb6315dbbe01ecc0 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Sun, 15 Nov 2026 02:41:51 +0100 Subject: [PATCH 04/12] fix(customer-identity): restart ZITADEL when the database CA is renewed ZITADEL verifies the database certificate against CNPG's CA, reads it at startup, and CNPG renews that CA before its 90-day expiry. Reloader restarts ZITADEL when any Secret it reads changes, so verified TLS keeps working. The HelmRelease's approved authorization fingerprint moves with it. Co-Authored-By: Claude Opus 5.5 --- .../hetzner/apps/customer-identity/helm-release.yaml | 5 +++++ scripts/validate-eks-ci-role-policy/approved-surface.txt | 2 +- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/k8s/providers/hetzner/apps/customer-identity/helm-release.yaml b/k8s/providers/hetzner/apps/customer-identity/helm-release.yaml index 3884fc7f5b..678958ff58 100644 --- a/k8s/providers/hetzner/apps/customer-identity/helm-release.yaml +++ b/k8s/providers/hetzner/apps/customer-identity/helm-release.yaml @@ -47,6 +47,11 @@ spec: name: zitadel values: replicaCount: 2 + # ZITADEL reads its Secrets at startup. CNPG renews the database CA + # before it expires (every 90 days); restarting on that change keeps the + # verified database TLS working. + annotations: + reloader.stakater.com/auto: "true" image: repository: ghcr.io/zitadel/zitadel tag: v4.20.3@sha256:1b30fea7ed88826dc0a98dacee4e2f2c8b05201fd26f5e4cb5386615b29f7638 diff --git a/scripts/validate-eks-ci-role-policy/approved-surface.txt b/scripts/validate-eks-ci-role-policy/approved-surface.txt index 431d3ec3a3..6b473e8b9a 100644 --- a/scripts/validate-eks-ci-role-policy/approved-surface.txt +++ b/scripts/validate-eks-ci-role-policy/approved-surface.txt @@ -60,7 +60,7 @@ helm.toolkit.fluxcd.io/v2|HelmRelease|cnpg-system|plugin-barman-cloud 74682cb45a helm.toolkit.fluxcd.io/v2|HelmRelease|crossplane-system|crossplane fb717f7c9adb21b0f772e6a84554900e568a2675c91d375d1bb6ee76c2a0b1f3 -helm.toolkit.fluxcd.io/v2|HelmRelease|customer-identity|zitadel 9c0c9f431282b3053bbebee8798367240c186e545a5d55e7b4770ad8cdd44495 +helm.toolkit.fluxcd.io/v2|HelmRelease|customer-identity|zitadel 00a75eff3c318ae5ce20c54994a2eef0f0856596b7a1a5c9972f69370aa54918 helm.toolkit.fluxcd.io/v2|HelmRelease|dex|dex e2cbe6e6b5ace19ebded8d0e75f508277dd450c079ed51062d13649e148904b9 From 733ee7fb99013cadfae157d608cc91a4c8beca19 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Sun, 15 Nov 2026 03:41:49 +0100 Subject: [PATCH 05/12] docs(customer-identity): qualify the database's replication guarantee With dataDurability: preferred, commits continue while the only standby is down, so a promoted standby holds every committed transaction only while synchronous replication is active. Say so instead of implying RPO=0. Co-Authored-By: Claude Opus 5.5 --- k8s/providers/hetzner/apps/customer-identity/cluster.yaml | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/k8s/providers/hetzner/apps/customer-identity/cluster.yaml b/k8s/providers/hetzner/apps/customer-identity/cluster.yaml index 84ba520db9..c2c5013e37 100644 --- a/k8s/providers/hetzner/apps/customer-identity/cluster.yaml +++ b/k8s/providers/hetzner/apps/customer-identity/cluster.yaml @@ -39,9 +39,11 @@ spec: coroot.com/postgres-scrape-credentials-secret-username-key: username coroot.com/postgres-scrape-credentials-secret-password-key: password coroot.com/postgres-scrape-param-sslmode: require - # Same quorum replication as the platform's other databases: a promoted - # standby always holds every committed transaction, and writes continue if - # the standby is down. + # Same quorum replication as the platform's other databases. While the + # standby is up, every commit waits for it, so a promoted standby holds every + # committed transaction. With dataDurability: preferred, writes continue + # when the standby is down, and commits made then exist only on the primary + # until the standby or the WAL archive receives them. This is not RPO=0. postgresql: synchronous: method: any From 5103b921e9df99175ea17c97707074458ec9da03 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Sun, 15 Nov 2026 04:02:59 +0100 Subject: [PATCH 06/12] docs(customer-identity): tie the replication guarantee to active sync replication A running standby can still be excluded from synchronous replication when it is unhealthy, so "while the standby is up" was too broad. The guarantee holds while synchronous replication is active; writes continue when the standby is unavailable. Co-Authored-By: Claude Opus 5.5 --- .../hetzner/apps/customer-identity/cluster.yaml | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/k8s/providers/hetzner/apps/customer-identity/cluster.yaml b/k8s/providers/hetzner/apps/customer-identity/cluster.yaml index c2c5013e37..1e8c59b17e 100644 --- a/k8s/providers/hetzner/apps/customer-identity/cluster.yaml +++ b/k8s/providers/hetzner/apps/customer-identity/cluster.yaml @@ -39,11 +39,13 @@ spec: coroot.com/postgres-scrape-credentials-secret-username-key: username coroot.com/postgres-scrape-credentials-secret-password-key: password coroot.com/postgres-scrape-param-sslmode: require - # Same quorum replication as the platform's other databases. While the - # standby is up, every commit waits for it, so a promoted standby holds every - # committed transaction. With dataDurability: preferred, writes continue - # when the standby is down, and commits made then exist only on the primary - # until the standby or the WAL archive receives them. This is not RPO=0. + # Same quorum replication as the platform's other databases. While + # synchronous replication is active, every commit waits for the standby, so + # a promoted standby holds every committed transaction. With + # dataDurability: preferred, CNPG counts only healthy standbys, so writes + # continue when the standby is unavailable (down or unhealthy), and commits + # made then exist only on the primary until the standby or the WAL archive + # receives them. This is not RPO=0. postgresql: synchronous: method: any From 95665d285ae71c14aba022ed2efb391c0a9f7092 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Sun, 15 Nov 2026 05:24:54 +0100 Subject: [PATCH 07/12] fix(customer-identity): keep the ZITADEL database on Hetzner Cloud volumes The first rollout's database volume never attached: Longhorn refused its replicas for insufficient storage, so CNPG could not initialize and ZITADEL's init job waited on it. CNPG already keeps a second copy, so Longhorn's three replicas per instance would triple storage the shared pool cannot spare. Use Hetzner Cloud volumes, as OpenBao and Coroot do, and keep the instances (and CNPG's jobs) on the static baseline workers, which hcloud volumes require because they are not force-detached when a node leaves. Co-Authored-By: Claude Opus 5.5 --- .../hetzner/apps/customer-identity/cluster.yaml | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/k8s/providers/hetzner/apps/customer-identity/cluster.yaml b/k8s/providers/hetzner/apps/customer-identity/cluster.yaml index 1e8c59b17e..184391e907 100644 --- a/k8s/providers/hetzner/apps/customer-identity/cluster.yaml +++ b/k8s/providers/hetzner/apps/customer-identity/cluster.yaml @@ -24,9 +24,20 @@ spec: env: - name: PLATFORM_DATABASE_PRIORITY_CLASS value: hcloud-volume-workload + # Hetzner Cloud volumes, as OpenBao and Coroot use, not Longhorn: CNPG + # already keeps a second copy, so Longhorn's three replicas per instance + # would triple storage the shared pool cannot spare (the first rollout's + # volume was refused for insufficient Longhorn storage). An hcloud volume is + # not force-detached when its node leaves, so the instances stay on the + # static baseline workers (require-baseline-workers-for-hcloud-volumes). + # Temporary: every platform volume finishes on Longhorn V2, and this + # cluster's two claims move there with the database migration (#5842). storage: - size: 5Gi - storageClass: longhorn-wffc + size: 10Gi + storageClass: hcloud + affinity: + nodeSelector: + platform.devantler.tech/baseline-worker: "true" # Coroot's Postgres integration, as every platform database has. CNPG # copies these annotations onto the instance pods, where Coroot's # cluster-agent discovers them; it logs in as the coroot_monitor role below, From b28941d20c1fa64a7419bf1c0093a4d04c2112cd Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Sun, 15 Nov 2026 06:14:21 +0100 Subject: [PATCH 08/12] feat(customer-identity): declare ZITADEL's mail through GitOps ZITADEL now creates its instance with one active SMTP provider declared in DefaultInstance.SMTPConfiguration: the business site's send-only Amazon SES sender (platform#5400), STARTTLS to email-smtp.eu-central-1.amazonaws.com:587 as no-reply@devantler.tech. The credential comes from OpenBao apps/business-site/smtp through an ExternalSecret, so it is never typed into the console. The shared store already reads that exact key. Verified locally from the rendered config with the pinned image: setup adds and activates the provider with the SMTP user and an encrypted password. ZITADEL reads an environment variable only for a key its configuration already has, so the empty PlainAuth keys are declared, and its sample SMTP.User/Password keys are not read by v4. Like every DefaultInstance setting, the provider is applied when the instance is created, not re-applied to an existing one; a rotated SES credential must also be updated through ZITADEL's admin API. Co-Authored-By: Claude Opus 5.5 --- .../external-secret-smtp.yaml | 26 +++++++++++++ .../apps/customer-identity/helm-release.yaml | 38 ++++++++++++++++--- .../apps/customer-identity/kustomization.yaml | 1 + .../approved-surface.txt | 2 +- 4 files changed, 61 insertions(+), 6 deletions(-) create mode 100644 k8s/providers/hetzner/apps/customer-identity/external-secret-smtp.yaml diff --git a/k8s/providers/hetzner/apps/customer-identity/external-secret-smtp.yaml b/k8s/providers/hetzner/apps/customer-identity/external-secret-smtp.yaml new file mode 100644 index 0000000000..27ae97d258 --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/external-secret-smtp.yaml @@ -0,0 +1,26 @@ +# ZITADEL's mail login: the business site's send-only Amazon SES credential +# (platform#5400), read from OpenBao apps/business-site/smtp through the shared +# store, so no one types it into ZITADEL's console. ZITADEL copies it into its +# own SMTP provider when it creates the instance (helm-release.yaml). +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: zitadel-smtp + namespace: customer-identity +spec: + refreshInterval: 1h + secretStoreRef: + name: openbao + kind: ClusterSecretStore + target: + name: zitadel-smtp + creationPolicy: Owner + data: + - secretKey: username + remoteRef: + key: apps/business-site/smtp + property: username + - secretKey: password + remoteRef: + key: apps/business-site/smtp + property: password diff --git a/k8s/providers/hetzner/apps/customer-identity/helm-release.yaml b/k8s/providers/hetzner/apps/customer-identity/helm-release.yaml index 678958ff58..aa226c0c39 100644 --- a/k8s/providers/hetzner/apps/customer-identity/helm-release.yaml +++ b/k8s/providers/hetzner/apps/customer-identity/helm-release.yaml @@ -23,11 +23,14 @@ # ZITADEL applies DefaultInstance and FirstInstance only when it creates the # instance; later changes go through the console or its API. # -# Mail: ZITADEL keeps its SMTP provider in its database, so it is added once in -# the console when the business site's SES sender exists (platform#5400): -# email-smtp.eu-central-1.amazonaws.com:587 with STARTTLS, the credential in -# OpenBao apps/business-site/smtp, sender no-reply@devantler.tech. Egress to -# that host is already open (cilium-network-policy.yaml). +# Mail is declared: when ZITADEL creates the instance it adds and activates +# one SMTP provider from DefaultInstance.SMTPConfiguration below, the business +# site's send-only Amazon SES sender (platform#5400): STARTTLS to +# email-smtp.eu-central-1.amazonaws.com:587 as no-reply@devantler.tech, with +# the credential from OpenBao apps/business-site/smtp (external-secret-smtp.yaml). +# ZITADEL then keeps the provider in its database: like every DefaultInstance +# setting it is not re-applied to an existing instance, so a rotated SES +# credential must also be updated through ZITADEL's admin API. apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: @@ -140,6 +143,16 @@ spec: secretKeyRef: name: zitadel-first-admin key: password + - name: ZITADEL_DEFAULTINSTANCE_SMTPCONFIGURATION_SMTP_PLAINAUTH_USER + valueFrom: + secretKeyRef: + name: zitadel-smtp + key: username + - name: ZITADEL_DEFAULTINSTANCE_SMTPCONFIGURATION_SMTP_PLAINAUTH_PASSWORD + valueFrom: + secretKeyRef: + name: zitadel-smtp + key: password zitadel: masterkeySecretName: zitadel-masterkey configmapConfig: @@ -213,6 +226,21 @@ spec: MinLength: 14 Restrictions: DisallowPublicOrgRegistration: true + # TLS: true makes ZITADEL upgrade the port-587 session with STARTTLS + # (certificate checked for the host) and never send in plain text. + # The empty PlainAuth keys must stay: ZITADEL reads an environment + # variable only for a key its configuration already has, and the + # SMTP.User/Password keys in its sample defaults are not read by v4. + SMTPConfiguration: + Description: Business site sender (Amazon SES) + SMTP: + Host: email-smtp.eu-central-1.amazonaws.com:587 + PlainAuth: + User: "" + Password: "" + TLS: true + From: no-reply@devantler.tech + FromName: Devantler Tech # The Devantler Tech look (near-black with #39ff14). The logo, font # and watermark are set in the console (Branding). LabelPolicy: diff --git a/k8s/providers/hetzner/apps/customer-identity/kustomization.yaml b/k8s/providers/hetzner/apps/customer-identity/kustomization.yaml index 127a67cfc1..6d8a8ef5fd 100644 --- a/k8s/providers/hetzner/apps/customer-identity/kustomization.yaml +++ b/k8s/providers/hetzner/apps/customer-identity/kustomization.yaml @@ -16,6 +16,7 @@ resources: - cluster.yaml - scheduled-backup.yaml - external-secret-masterkey.yaml + - external-secret-smtp.yaml - password-first-admin.yaml - external-secret-first-admin.yaml - helm-repository.yaml diff --git a/scripts/validate-eks-ci-role-policy/approved-surface.txt b/scripts/validate-eks-ci-role-policy/approved-surface.txt index 6b473e8b9a..3a2036e2b9 100644 --- a/scripts/validate-eks-ci-role-policy/approved-surface.txt +++ b/scripts/validate-eks-ci-role-policy/approved-surface.txt @@ -60,7 +60,7 @@ helm.toolkit.fluxcd.io/v2|HelmRelease|cnpg-system|plugin-barman-cloud 74682cb45a helm.toolkit.fluxcd.io/v2|HelmRelease|crossplane-system|crossplane fb717f7c9adb21b0f772e6a84554900e568a2675c91d375d1bb6ee76c2a0b1f3 -helm.toolkit.fluxcd.io/v2|HelmRelease|customer-identity|zitadel 00a75eff3c318ae5ce20c54994a2eef0f0856596b7a1a5c9972f69370aa54918 +helm.toolkit.fluxcd.io/v2|HelmRelease|customer-identity|zitadel a2f82a6a6fe6d55240c66c5a084925c4007115a35c80c9aecb97eacbb3f862cb helm.toolkit.fluxcd.io/v2|HelmRelease|dex|dex e2cbe6e6b5ace19ebded8d0e75f508277dd450c079ed51062d13649e148904b9 From 670b0e02ecc580e27eea94d55b97cd623b7fab0b Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Sun, 15 Nov 2026 11:09:59 +0100 Subject: [PATCH 09/12] feat(customer-identity): set up ZITADEL declaratively with a setup CronJob The maintainer chose a reviewed setup job over console steps. ZITADEL creates its default machine administrator only with a new instance, so the setup uses a system API user declared in ZITADEL's own config instead: - ZITADEL trusts one cert-manager certificate (zitadel-setup-key) for the system user `zitadel-setup`, whose only role is IAM_OWNER. ZITADEL mounts only the certificate. - A CronJob runs every 15 minutes. Its signer init container alone mounts the private key and writes a 10-minute token to memory. Its setup container alone gets a Kubernetes token, which may only read and update one Secret. - Each run reconciles the "Customer Dex" client from the business-site preparer's template and stores its secret (with a copy in OpenBao). It applies a rotated SES SMTP login, and keeps the administrator's one-time password equal to zitadel-first-admin until his first sign-in. A run with nothing new writes nothing; unexpected state stops it. Proven locally against ZITADEL v4.20.3, using the manifest's own scripts. Refs devantler-tech/business-site#302 Co-Authored-By: Claude Opus 5.5 --- docs/dr/crypto-custody.md | 14 + .../infrastructure/vault-config/job.yaml | 10 +- .../certificate-setup-key.yaml | 29 ++ .../customer-identity/cron-job-setup.yaml | 425 ++++++++++++++++++ .../apps/customer-identity/helm-release.yaml | 33 +- .../apps/customer-identity/issuer.yaml | 10 + .../apps/customer-identity/kustomization.yaml | 11 + .../push-secret-dex-zitadel-client.yaml | 31 ++ .../apps/customer-identity/role-binding.yaml | 13 + .../hetzner/apps/customer-identity/role.yaml | 19 + .../secret-dex-zitadel-client.yaml | 23 + .../customer-identity/service-account.yaml | 8 + .../approved-surface.txt | 6 +- 13 files changed, 628 insertions(+), 4 deletions(-) create mode 100644 k8s/providers/hetzner/apps/customer-identity/certificate-setup-key.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/cron-job-setup.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/issuer.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/push-secret-dex-zitadel-client.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/role-binding.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/role.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/secret-dex-zitadel-client.yaml create mode 100644 k8s/providers/hetzner/apps/customer-identity/service-account.yaml diff --git a/docs/dr/crypto-custody.md b/docs/dr/crypto-custody.md index 1587985fc3..18b021db63 100644 --- a/docs/dr/crypto-custody.md +++ b/docs/dr/crypto-custody.md @@ -26,6 +26,7 @@ and [openbao.md](openbao.md) (OpenBao-specific DR). | **Cloudflare API token** | manage DNS for the platform domain | redirect prod traffic; mint Origin CA certs | mint a new one in the Cloudflare dashboard | | **Actual Budget E2EE password** | client-side key to encrypt/decrypt an Actual budget file | decrypt that budget's financial data | the budget is **permanently unrecoverable** — the server never held the key | | **ZITADEL master key** | encrypt the customer identity provider's signing keys and stored secrets (`customer-identity` namespace) | with a database copy, decrypt those keys and secrets — rotate them in ZITADEL | restored `customer-identity-db` backups cannot be used; customer accounts are recreated | +| **ZITADEL setup user key** | sign the setup CronJob's tokens as ZITADEL's `zitadel-setup` system API user (instance owner) | administer the customer identity provider — delete `customer-identity/zitadel-setup-key` so cert-manager issues a new key | nothing to recover: cert-manager issues a new key and ZITADEL trusts the new certificate | The rest of this document walks each row in detail. @@ -330,6 +331,19 @@ OpenBao is its custody of record: an existing value is never replaced, and from its R2 backups. A fresh OpenBao would generate a new key that cannot read the restored database. - **Rotation:** not supported by ZITADEL for an existing database. Treat the key as permanent. +## ZITADEL setup user key + +The setup CronJob (`k8s/providers/hetzner/apps/customer-identity/cron-job-setup.yaml`) reaches +ZITADEL's APIs as the system API user `zitadel-setup`, whose only role is `IAM_OWNER` on the instance. +ZITADEL trusts the public key in cert-manager's certificate `customer-identity/zitadel-setup-key`. +It mounts only `tls.crt`; only the CronJob's signer container mounts `tls.key`, and no Role grants +reading the Secret. + +- **Custody:** the cluster only. The key is not backed up and needs no backup: a rebuild issues a new + one, and ZITADEL trusts whatever certificate it mounts. +- **Rotation:** automatic. cert-manager issues a new key 30 days before the 90-day certificate expires, + and Reloader restarts ZITADEL to read it. To rotate at once, delete the Secret. + --- ## Cross-references diff --git a/k8s/bases/infrastructure/vault-config/job.yaml b/k8s/bases/infrastructure/vault-config/job.yaml index 6dc05d78e7..ab7119eed6 100644 --- a/k8s/bases/infrastructure/vault-config/job.yaml +++ b/k8s/bases/infrastructure/vault-config/job.yaml @@ -1078,10 +1078,15 @@ spec: capabilities = ["create", "update", "read"] } # ZITADEL's master key, generated once by its create-only - # PushSecret (vault-seed). Exact key only. + # PushSecret (vault-seed), and the customer Dex's ZITADEL + # client, which the customer-identity setup CronJob issues and + # a PushSecret copies here. Exact keys only. path "secret/data/apps/customer-identity/zitadel" { capabilities = ["create", "update", "read"] } + path "secret/data/apps/customer-identity/dex-zitadel-client" { + capabilities = ["create", "update", "read"] + } path "secret/data/apps/actual-budget/*" { capabilities = ["create", "update", "read"] } @@ -1146,6 +1151,9 @@ spec: path "secret/metadata/apps/customer-identity/zitadel" { capabilities = ["create", "update", "read"] } + path "secret/metadata/apps/customer-identity/dex-zitadel-client" { + capabilities = ["create", "update", "read"] + } path "secret/metadata/apps/actual-budget/*" { capabilities = ["create", "update", "read"] } diff --git a/k8s/providers/hetzner/apps/customer-identity/certificate-setup-key.yaml b/k8s/providers/hetzner/apps/customer-identity/certificate-setup-key.yaml new file mode 100644 index 0000000000..724b98878a --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/certificate-setup-key.yaml @@ -0,0 +1,29 @@ +# The key pair of ZITADEL's `zitadel-setup` system API user (helm-release.yaml +# SystemAPIUsers). ZITADEL mounts only tls.crt; only the setup CronJob's +# signer mounts tls.key (cron-job-setup.yaml), and no Role lets anyone read +# the Secret through the API. +# +# cert-manager renews it with a new key 30 days before it expires. Reloader +# then restarts ZITADEL so it reads the new certificate; a setup run during +# that restart fails and the next one succeeds. +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: zitadel-setup-key + namespace: customer-identity +spec: + secretName: zitadel-setup-key + commonName: zitadel-setup + duration: 2160h + renewBefore: 720h + isCA: false + usages: + - digital signature + privateKey: + algorithm: RSA + size: 2048 + encoding: PKCS8 + rotationPolicy: Always + issuerRef: + name: customer-identity-setup + kind: Issuer diff --git a/k8s/providers/hetzner/apps/customer-identity/cron-job-setup.yaml b/k8s/providers/hetzner/apps/customer-identity/cron-job-setup.yaml new file mode 100644 index 0000000000..8f7a09c41a --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/cron-job-setup.yaml @@ -0,0 +1,425 @@ +# Declarative ZITADEL setup for the customer sign-in (business-site#302). +# Every 15 minutes it reconciles, through ZITADEL's own APIs: +# - the "Customer sign-in" project and its "Customer Dex" client, exactly as the +# business-site preparer's CreateApplication template declares it (code grant +# only, the exact Dex callback, Login V1). It restores drifted settings and, +# when the stored secret is missing or is the placeholder, issues a new one +# into secret-dex-zitadel-client.yaml; +# - the SES SMTP login: when the business site's credential (OpenBao +# apps/business-site/smtp) changes, it updates ZITADEL's provider; +# - the administrator's one-time password: until the maintainer's first +# sign-in replaces it, it stays equal to zitadel-first-admin, so the +# commissioning step works even after that Secret is regenerated. Once he has +# chosen his own password it is never touched again. +# A run that finds nothing to change writes nothing. A duplicate, a deactivated +# client, a missing Secret, an API error or a refused write stops the run +# before it changes anything further; the next run starts again. +# +# Credentials stay apart: +# - The signer (init container) alone mounts the private key of ZITADEL's +# `zitadel-setup` system API user (certificate-setup-key.yaml). It writes a +# 10-minute token to a memory volume and exits. +# - The setup container alone gets a Kubernetes token, which may only read and +# update the client Secret (role.yaml). It reads the SMTP login (zitadel-smtp) +# and the one-time password (zitadel-first-admin) through its environment. +# - Neither prints a secret. +# +# The maintainer's own first sign-in (choosing his password, enrolling MFA) is +# not part of this. +apiVersion: batch/v1 +kind: CronJob +metadata: + name: customer-identity-setup + namespace: customer-identity + annotations: + # The scripts' own ${...} expansions are bash, not Flux variables. + kustomize.toolkit.fluxcd.io/substitute: disabled +spec: + schedule: "*/15 * * * *" + concurrencyPolicy: Forbid + startingDeadlineSeconds: 300 + successfulJobsHistoryLimit: 1 + failedJobsHistoryLimit: 3 + jobTemplate: + spec: + backoffLimit: 0 + activeDeadlineSeconds: 300 + ttlSecondsAfterFinished: 86400 + template: + metadata: + labels: + app.kubernetes.io/name: customer-identity-setup + spec: + serviceAccountName: customer-identity-setup + automountServiceAccountToken: false + restartPolicy: Never + securityContext: + fsGroupChangePolicy: OnRootMismatch + runAsNonRoot: true + runAsUser: 65532 + runAsGroup: 65532 + fsGroup: 65532 + seccompProfile: + type: RuntimeDefault + initContainers: + - name: sign-token + image: docker.io/alpine/openssl:3.5.8@sha256:3f25da71f70eba788067daac3f3df03bd1de7a7c52ed89fa93b94ad2c92d986b + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 65532 + runAsGroup: 65532 + capabilities: + drop: + - ALL + env: + - name: ZITADEL_AUDIENCE + value: https://accounts.devantler.tech + - name: ZITADEL_SYSTEM_USER + value: zitadel-setup + volumeMounts: + - name: setup-key + mountPath: /run/setup-key + readOnly: true + - name: setup-token + mountPath: /run/setup-token + resources: + requests: + cpu: 10m + memory: 16Mi + limits: + cpu: 100m + memory: 32Mi + command: + - /bin/sh + - -c + - |- + #!/bin/sh + # Signs a short-lived JWT for ZITADEL's `zitadel-setup` system API user with + # the key cert-manager keeps in zitadel-setup-key. Only this init container + # mounts the private key; it writes the token to a memory-backed volume that + # the setup container reads. The token is never printed. + set -eu + key=/run/setup-key/tls.key + out=/run/setup-token/token + audience="${ZITADEL_AUDIENCE:?}" + user="${ZITADEL_SYSTEM_USER:?}" + lifetime="${TOKEN_LIFETIME_SECONDS:-600}" + + b64url() { openssl base64 -A | tr '+/' '-_' | tr -d '='; } + + [ -s "$key" ] || { echo "sign-token: missing signing key" >&2; exit 1; } + now=$(date +%s) + header=$(printf '{"alg":"RS256","typ":"JWT"}' | b64url) + payload=$(printf '{"iss":"%s","sub":"%s","aud":"%s","iat":%d,"exp":%d}' \ + "$user" "$user" "$audience" "$now" "$((now + lifetime))" | b64url) + signature=$(printf '%s.%s' "$header" "$payload" | openssl dgst -sha256 -sign "$key" -binary | b64url) + umask 077 + printf '%s.%s.%s' "$header" "$payload" "$signature" > "$out" + echo "sign-token: wrote a ${lifetime}s token for ${user}" + containers: + - name: setup + image: docker.io/alpine/k8s:1.37.2@sha256:dcb7185e112a453aa72f426754849a97357727ef8f40168fb208c8cf59952042 + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 65532 + runAsGroup: 65532 + capabilities: + drop: + - ALL + env: + - name: HOME + value: /tmp + - name: ZITADEL_URL + value: http://zitadel.customer-identity.svc:8080 + - name: ZITADEL_HOST + value: accounts.devantler.tech + - name: TOKEN_FILE + value: /run/setup-token/token + - name: SMTP_USER + valueFrom: + secretKeyRef: + name: zitadel-smtp + key: username + - name: SMTP_PASSWORD + valueFrom: + secretKeyRef: + name: zitadel-smtp + key: password + - name: ADMIN_EMAIL + value: ned@devantler.tech + - name: ADMIN_PASSWORD + valueFrom: + secretKeyRef: + name: zitadel-first-admin + key: password + volumeMounts: + - name: setup-token + mountPath: /run/setup-token + readOnly: true + - name: tmp + mountPath: /tmp + - name: kube-api-access + mountPath: /var/run/secrets/kubernetes.io/serviceaccount + readOnly: true + resources: + requests: + cpu: 10m + memory: 32Mi + limits: + cpu: 200m + memory: 128Mi + command: + - /bin/bash + - -c + - |- + #!/usr/bin/env bash + # Reconciles ZITADEL's customer Dex client and its SES SMTP login from the + # state declared here. Every run is idempotent: it changes nothing when ZITADEL + # and the client Secret already match. Any unexpected state (a duplicate, a + # deactivated client, a missing Secret, an API error) stops the run before it + # writes anything further. No secret value is ever printed. + set -euo pipefail + umask 077 + + : "${ZITADEL_URL:?}" "${ZITADEL_HOST:?}" "${TOKEN_FILE:?}" "${SMTP_USER:?}" "${SMTP_PASSWORD:?}" + : "${ADMIN_EMAIL:?}" "${ADMIN_PASSWORD:?}" + namespace=customer-identity + client_secret_name=customer-dex-zitadel-client + # Fingerprints of what this script last applied, kept as annotations on the + # one Secret it may write, so a run with nothing new changes nothing. + fingerprint_annotation=platform.devantler.tech/smtp-credential-sha256 + admin_annotation=platform.devantler.tech/first-admin-password-sha256 + project_name="Customer sign-in" + app_name="Customer Dex" + dex_callback=https://login.devantler.tech/callback + smtp_host=email-smtp.eu-central-1.amazonaws.com:587 + smtp_sender=no-reply@devantler.tech + smtp_sender_name="Devantler Tech" + smtp_description="Business site sender (Amazon SES)" + placeholder=PLACEHOLDER + + # The business-site preparer's CreateApplication template (server/internal/ + # customeridentity/config.go): one confidential web client, code grant only, + # the exact Dex callback, Login V1. + desired_oidc=$(jq -nc --arg cb "$dex_callback" '{ + redirectUris: [$cb], responseTypes: ["OIDC_RESPONSE_TYPE_CODE"], + grantTypes: ["OIDC_GRANT_TYPE_AUTHORIZATION_CODE"], applicationType: "OIDC_APP_TYPE_WEB", + authMethodType: "OIDC_AUTH_METHOD_TYPE_BASIC", version: "OIDC_VERSION_1_0", developmentMode: false, + accessTokenRoleAssertion: false, idTokenRoleAssertion: false, idTokenUserinfoAssertion: false, + additionalOrigins: [], postLogoutRedirectUris: [], loginVersion: {loginV1: {}}}') + # ZITADEL omits fields that hold their default value, so compare normalised. + normalise_oidc='{ + redirectUris: (.redirectUris // []), responseTypes: (.responseTypes // ["OIDC_RESPONSE_TYPE_CODE"]), + grantTypes: (.grantTypes // ["OIDC_GRANT_TYPE_AUTHORIZATION_CODE"]), + applicationType: (.applicationType // "OIDC_APP_TYPE_WEB"), + authMethodType: (.authMethodType // "OIDC_AUTH_METHOD_TYPE_BASIC"), + version: (.version // "OIDC_VERSION_1_0"), developmentMode: (.developmentMode // false), + accessTokenRoleAssertion: (.accessTokenRoleAssertion // false), + idTokenRoleAssertion: (.idTokenRoleAssertion // false), + idTokenUserinfoAssertion: (.idTokenUserinfoAssertion // false), + additionalOrigins: (.additionalOrigins // []), postLogoutRedirectUris: (.postLogoutRedirectUris // []), + loginVersion: (.loginVersion // {loginV1: {}})}' + + log() { printf 'setup: %s\n' "$*"; } + fail() { printf 'setup: FAILED: %s\n' "$*" >&2; exit 1; } + + work=$(mktemp -d) + trap 'rm -rf "$work"' EXIT + [[ -s $TOKEN_FILE ]] || fail "no signed token at $TOKEN_FILE" + printf 'Authorization: Bearer %s\n' "$(<"$TOKEN_FILE")" > "$work/auth" + + # api METHOD PATH [BODY]: prints the response body on a 2xx answer. Any other + # answer fails the run with ZITADEL's error code and message only. + api() { + local method=$1 path=$2 body=${3:-'{}'} status + local -a data=() + [[ $method == GET ]] || data=(--data-binary "$body") + status=$(curl -sS -m 20 -o "$work/response" -w '%{http_code}' -X "$method" \ + -H "@$work/auth" -H "Host: $ZITADEL_HOST" -H 'Content-Type: application/json' \ + -H 'Connect-Protocol-Version: 1' "${data[@]}" "$ZITADEL_URL$path") || + fail "$method $path: no answer from ZITADEL" + if [[ $status != 2?? ]]; then + fail "$method $path: HTTP $status $(jq -rc '[.code, .message] | map(tostring) | join(" ")' "$work/response" 2>/dev/null || echo 'unreadable error')" + fi + cat "$work/response" + } + connect() { api POST "/$1" "${2-}"; } + + # The client Secret is created empty by Flux and only ever patched here. + secret_json() { + kubectl -n "$namespace" get secret "$client_secret_name" -o json || + fail "cannot read Secret $client_secret_name" + } + patch_secret() { + printf '%s' "$1" > "$work/patch" + kubectl -n "$namespace" patch secret "$client_secret_name" --type merge --patch-file "$work/patch" >/dev/null || + fail "cannot update Secret $client_secret_name" + } + store_client() { + local id=$1 + [[ -n $id && -n $CLIENT_SECRET ]] || fail "ZITADEL returned an empty client" + patch_secret "$(jq -nc --arg id "$id" \ + '{data: {"client-id": ($id | @base64), "client-secret": (env.CLIENT_SECRET | @base64)}}')" + [[ $(secret_json | jq -r '.data["client-id"] // "" | @base64d') == "$id" ]] || + fail "Secret $client_secret_name did not keep the new client" + } + + for attempt in $(seq 1 20); do + curl -sf -m 5 -o /dev/null "$ZITADEL_URL/debug/ready" && break + [[ $attempt == 20 ]] && fail "ZITADEL is not ready" + sleep 6 + done + + # Stop before ZITADEL issues anything if there is nowhere to keep it. + secret_json >/dev/null + [[ $(kubectl -n "$namespace" auth can-i patch "secrets/$client_secret_name" 2>/dev/null) == yes ]] || + fail "may not update Secret $client_secret_name" + + changes=() + org_id=$(api GET /admin/v1/orgs/default | jq -er '.org.id') || fail "no default organization" + + # --- Project --------------------------------------------------------------- + projects=$(connect zitadel.project.v2.ProjectService/ListProjects '{"pagination":{"limit":1000}}') + [[ $(jq -r '.pagination.totalResult // "0"' <<<"$projects") -le $(jq '.projects // [] | length' <<<"$projects") ]] || + fail "project list is incomplete" + project_ids=$(jq -r --arg n "$project_name" --arg o "$org_id" \ + '.projects // [] | map(select(.name == $n and .organizationId == $o)) | .[].projectId' <<<"$projects") + case $(grep -c . <<<"$project_ids" || true) in + 0) + project_id=$(connect zitadel.project.v2.ProjectService/CreateProject \ + "$(jq -nc --arg o "$org_id" --arg n "$project_name" '{organizationId: $o, name: $n}')" | jq -er '.projectId') + changes+=("created project") + ;; + 1) project_id=$project_ids ;; + *) fail "more than one project named '$project_name'" ;; + esac + + # --- Application ------------------------------------------------------------- + apps=$(connect zitadel.application.v2.ApplicationService/ListApplications \ + "$(jq -nc --arg p "$project_id" '{pagination: {limit: 1000}, filters: [{projectIdFilter: {projectId: $p}}]}')") + [[ $(jq -r '.pagination.totalResult // "0"' <<<"$apps") -le $(jq '.applications // [] | length' <<<"$apps") ]] || + fail "application list is incomplete" + matches=$(jq -c --arg n "$app_name" '[.applications // [] | .[] | select(.name == $n)]' <<<"$apps") + case $(jq length <<<"$matches") in + 0) + connect zitadel.application.v2.ApplicationService/CreateApplication \ + "$(jq -nc --arg p "$project_id" --arg n "$app_name" --argjson c "$desired_oidc" \ + '{projectId: $p, name: $n, oidcConfiguration: $c}')" > "$work/created" + app_id=$(jq -er '.applicationId' "$work/created") + client_id=$(jq -er '.oidcConfiguration.clientId' "$work/created") + CLIENT_SECRET=$(jq -er '.oidcConfiguration.clientSecret' "$work/created") store_client "$client_id" + changes+=("created client") + ;; + 1) + app=$(jq -c '.[0]' <<<"$matches") + app_id=$(jq -er '.applicationId' <<<"$app") + client_id=$(jq -er '.oidcConfiguration.clientId' <<<"$app") + [[ $(jq -r '.state' <<<"$app") == APPLICATION_STATE_ACTIVE ]] || + fail "client '$app_name' is not active; reactivate or remove it on purpose" + if [[ $(jq -cS ".oidcConfiguration | $normalise_oidc" <<<"$app") != "$(jq -cS "$normalise_oidc" <<<"$desired_oidc")" ]]; then + connect zitadel.application.v2.ApplicationService/UpdateApplication \ + "$(jq -nc --arg p "$project_id" --arg a "$app_id" --argjson c "$desired_oidc" \ + '{projectId: $p, applicationId: $a, oidcConfiguration: $c}')" >/dev/null + changes+=("restored client settings") + fi + stored=$(secret_json) + if [[ $(jq -r '.data["client-id"] // "" | @base64d' <<<"$stored") != "$client_id" ]] || + [[ $(jq -r '.data["client-secret"] // "" | @base64d' <<<"$stored") =~ ^($placeholder)?$ ]]; then + connect zitadel.application.v2.ApplicationService/GenerateClientSecret \ + "$(jq -nc --arg p "$project_id" --arg a "$app_id" '{projectId: $p, applicationId: $a}')" > "$work/generated" + CLIENT_SECRET=$(jq -er '.clientSecret' "$work/generated") store_client "$client_id" + changes+=("issued client secret") + fi + ;; + *) fail "more than one client named '$app_name'" ;; + esac + + # --- SMTP -------------------------------------------------------------------- + fingerprint=$(printf '%s\n%s' "$SMTP_USER" "$SMTP_PASSWORD" | sha256sum | cut -d' ' -f1) + providers=$(api POST /admin/v1/smtp/_search) + provider=$(jq -c --arg h "$smtp_host" --arg s "$smtp_sender" \ + '[.result // [] | .[] | select(.host == $h and .senderAddress == $s)]' <<<"$providers") + [[ $(jq length <<<"$provider") == 1 ]] || fail "expected exactly one SES provider for $smtp_sender" + provider=$(jq -c '.[0]' <<<"$provider") + smtp_id=$(jq -er '.id' <<<"$provider") + recorded=$(secret_json | jq -r --arg a "$fingerprint_annotation" '.metadata.annotations[$a] // ""') + if [[ $recorded != "$fingerprint" || $(jq -r '.user' <<<"$provider") != "$SMTP_USER" || + $(jq -r '.tls // false' <<<"$provider") != true || $(jq -r '.senderName // ""' <<<"$provider") != "$smtp_sender_name" ]]; then + api PUT "/admin/v1/smtp/$smtp_id" "$(jq -nc --arg s "$smtp_sender" --arg n "$smtp_sender_name" \ + --arg h "$smtp_host" --arg u "$SMTP_USER" --arg d "$smtp_description" \ + '{senderAddress: $s, senderName: $n, tls: true, host: $h, user: $u, description: $d}')" >/dev/null + api PUT "/admin/v1/smtp/$smtp_id/password" "$(jq -nc '{password: env.SMTP_PASSWORD}')" >/dev/null + patch_secret "$(jq -nc --arg a "$fingerprint_annotation" --arg f "$fingerprint" '{metadata: {annotations: {($a): $f}}}')" + changes+=("updated SMTP login") + fi + if [[ $(api POST /admin/v1/smtp/_search | jq -r --arg i "$smtp_id" '.result[] | select(.id == $i) | .state') != SMTP_CONFIG_ACTIVE ]]; then + api POST "/admin/v1/smtp/$smtp_id/_activate" >/dev/null + changes+=("activated SMTP provider") + fi + + # --- The administrator's one-time password ------------------------------------- + # ZITADEL set it once, when it created the instance, from zitadel-first-admin. + # Until the maintainer's first sign-in replaces it, keep it equal to that Secret + # (which a rebuild regenerates). Once he has chosen his own password, ZITADEL + # no longer requires a change and this never touches it again. + admins=$(connect zitadel.user.v2.UserService/ListUsers "$(jq -nc --arg e "$ADMIN_EMAIL" \ + '{queries: [{emailQuery: {emailAddress: $e, method: "TEXT_QUERY_METHOD_EQUALS"}}, {typeQuery: {type: "TYPE_HUMAN"}}]}')") + [[ $(jq '.result // [] | length' <<<"$admins") == 1 ]] || fail "expected exactly one administrator with $ADMIN_EMAIL" + admin_id=$(jq -er '.result[0].userId' <<<"$admins") + admin_state=own + if [[ $(jq -r '.result[0].human.passwordChangeRequired // false' <<<"$admins") == true ]]; then + admin_state=one-time + admin_fingerprint=$(printf '%s' "$ADMIN_PASSWORD" | sha256sum | cut -d' ' -f1) + if [[ $(secret_json | jq -r --arg a "$admin_annotation" '.metadata.annotations[$a] // ""') != "$admin_fingerprint" ]]; then + connect zitadel.user.v2.UserService/SetPassword "$(jq -nc --arg u "$admin_id" \ + '{userId: $u, newPassword: {password: env.ADMIN_PASSWORD, changeRequired: true}}')" >/dev/null + patch_secret "$(jq -nc --arg a "$admin_annotation" --arg f "$admin_fingerprint" '{metadata: {annotations: {($a): $f}}}')" + changes+=("set the administrator's one-time password") + fi + fi + + summary="project=$project_id client=$client_id application=$app_id smtp=$smtp_id administrator=$admin_state" + if ((${#changes[@]})); then + log "OK $summary changes: $(IFS=,; echo "${changes[*]}")" + else + log "OK $summary no changes" + fi + volumes: + - name: setup-key + secret: + secretName: zitadel-setup-key + defaultMode: 0440 + items: + - key: tls.key + path: tls.key + - name: setup-token + emptyDir: + medium: Memory + sizeLimit: 1Mi + # Holds ZITADEL's answers, which can carry the new client secret, + # so it never touches a disk. + - name: tmp + emptyDir: + medium: Memory + sizeLimit: 16Mi + - name: kube-api-access + projected: + defaultMode: 0440 + sources: + - serviceAccountToken: + path: token + expirationSeconds: 3600 + - configMap: + name: kube-root-ca.crt + items: + - key: ca.crt + path: ca.crt + - downwardAPI: + items: + - path: namespace + fieldRef: + fieldPath: metadata.namespace diff --git a/k8s/providers/hetzner/apps/customer-identity/helm-release.yaml b/k8s/providers/hetzner/apps/customer-identity/helm-release.yaml index aa226c0c39..103f037790 100644 --- a/k8s/providers/hetzner/apps/customer-identity/helm-release.yaml +++ b/k8s/providers/hetzner/apps/customer-identity/helm-release.yaml @@ -13,6 +13,11 @@ # dropped. The only bootstrap identity is one human administrator (the # maintainer), with a generated one-time password that must be changed at # the first sign-in, where the login policy also forces MFA. +# - One system API user, `zitadel-setup`, declared in SystemAPIUsers below, +# for the setup CronJob (cron-job-setup.yaml). ZITADEL trusts the public key +# in cert-manager's certificate (certificate-setup-key.yaml); only that +# CronJob's signer mounts the private key. Its only role is IAM_OWNER, so it +# can manage this instance but not ZITADEL's system API (no SYSTEM_OWNER). # - TLS ends at the platform Gateway, as for every platform app; the Gateway # reaches ZITADEL over h2c inside the WireGuard-encrypted cluster network. # @@ -29,8 +34,8 @@ # email-smtp.eu-central-1.amazonaws.com:587 as no-reply@devantler.tech, with # the credential from OpenBao apps/business-site/smtp (external-secret-smtp.yaml). # ZITADEL then keeps the provider in its database: like every DefaultInstance -# setting it is not re-applied to an existing instance, so a rotated SES -# credential must also be updated through ZITADEL's admin API. +# setting it is not re-applied to an existing instance, so the setup CronJob +# applies a rotated SES credential through ZITADEL's admin API. apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: @@ -125,10 +130,24 @@ spec: items: - key: ca.crt path: ca.crt + # The setup user's certificate only, never its private key. Optional, + # so ZITADEL starts before cert-manager has issued it; ZITADEL reads it + # when the setup CronJob first signs in. + - name: setup-user + secret: + secretName: zitadel-setup-key + defaultMode: 0440 + optional: true + items: + - key: tls.crt + path: zitadel-setup.crt extraVolumeMounts: - name: db-ca mountPath: /db-ca readOnly: true + - name: setup-user + mountPath: /system-users + readOnly: true # Rendered into the setup job and the server alike. env: - name: ZITADEL_DATABASE_POSTGRES_USER_PASSWORD @@ -161,6 +180,16 @@ spec: ExternalSecure: true TLS: Enabled: false + # The setup CronJob's identity (see above). Tokens must be signed by + # the certificate's key, name this user as issuer and subject, and be + # addressed to https://accounts.devantler.tech. + SystemAPIUsers: + zitadel-setup: + Path: /system-users/zitadel-setup.crt + Memberships: + - MemberType: System + Roles: + - IAM_OWNER Database: Postgres: Host: customer-identity-db-rw.customer-identity.svc diff --git a/k8s/providers/hetzner/apps/customer-identity/issuer.yaml b/k8s/providers/hetzner/apps/customer-identity/issuer.yaml new file mode 100644 index 0000000000..96c1d142f1 --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/issuer.yaml @@ -0,0 +1,10 @@ +# Signs the setup user's certificate (certificate-setup-key.yaml) with its +# own key. ZITADEL trusts that one certificate's public key directly, so no CA +# is involved. +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: customer-identity-setup + namespace: customer-identity +spec: + selfSigned: {} diff --git a/k8s/providers/hetzner/apps/customer-identity/kustomization.yaml b/k8s/providers/hetzner/apps/customer-identity/kustomization.yaml index 6d8a8ef5fd..9e9a3dac58 100644 --- a/k8s/providers/hetzner/apps/customer-identity/kustomization.yaml +++ b/k8s/providers/hetzner/apps/customer-identity/kustomization.yaml @@ -25,3 +25,14 @@ resources: - certificate.yaml - reference-grant.yaml - http-route.yaml + # The declarative setup: ZITADEL's system API user's key, the CronJob that + # reconciles the customer Dex client and the SES login, and the one Secret + # it writes. + - issuer.yaml + - certificate-setup-key.yaml + - service-account.yaml + - role.yaml + - role-binding.yaml + - secret-dex-zitadel-client.yaml + - push-secret-dex-zitadel-client.yaml + - cron-job-setup.yaml diff --git a/k8s/providers/hetzner/apps/customer-identity/push-secret-dex-zitadel-client.yaml b/k8s/providers/hetzner/apps/customer-identity/push-secret-dex-zitadel-client.yaml new file mode 100644 index 0000000000..5ed3b5872c --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/push-secret-dex-zitadel-client.yaml @@ -0,0 +1,31 @@ +# Keeps OpenBao apps/customer-identity/dex-zitadel-client equal to the client +# the setup CronJob wrote (secret-dex-zitadel-client.yaml), as `client-id` and +# `client-secret`. Platform-applied, so it may use the shared store, which +# holds an exact-key grant for this one record (vault-config). Nothing reads +# the copy back: after a rebuild the CronJob issues a new secret. +apiVersion: external-secrets.io/v1alpha1 +kind: PushSecret +metadata: + name: customer-dex-zitadel-client + namespace: customer-identity +spec: + updatePolicy: Replace + deletionPolicy: None + refreshInterval: 10m + secretStoreRefs: + - name: openbao + kind: ClusterSecretStore + selector: + secret: + name: customer-dex-zitadel-client + data: + - match: + secretKey: client-id + remoteRef: + remoteKey: apps/customer-identity/dex-zitadel-client + property: client-id + - match: + secretKey: client-secret + remoteRef: + remoteKey: apps/customer-identity/dex-zitadel-client + property: client-secret diff --git a/k8s/providers/hetzner/apps/customer-identity/role-binding.yaml b/k8s/providers/hetzner/apps/customer-identity/role-binding.yaml new file mode 100644 index 0000000000..7128e77322 --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/role-binding.yaml @@ -0,0 +1,13 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: customer-identity-setup + namespace: customer-identity +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: customer-identity-setup +subjects: + - kind: ServiceAccount + name: customer-identity-setup + namespace: customer-identity diff --git a/k8s/providers/hetzner/apps/customer-identity/role.yaml b/k8s/providers/hetzner/apps/customer-identity/role.yaml new file mode 100644 index 0000000000..ce42e8a708 --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/role.yaml @@ -0,0 +1,19 @@ +# The setup CronJob may read and update one Secret: the customer Dex's ZITADEL +# client (secret-dex-zitadel-client.yaml). It cannot create, list or delete +# Secrets, and it has no access to its own signing key, which reaches it only +# as a mounted volume. +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: customer-identity-setup + namespace: customer-identity +rules: + - apiGroups: + - "" + resources: + - secrets + resourceNames: + - customer-dex-zitadel-client + verbs: + - get + - patch diff --git a/k8s/providers/hetzner/apps/customer-identity/secret-dex-zitadel-client.yaml b/k8s/providers/hetzner/apps/customer-identity/secret-dex-zitadel-client.yaml new file mode 100644 index 0000000000..2434f667ff --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/secret-dex-zitadel-client.yaml @@ -0,0 +1,23 @@ +# The customer Dex's confidential client in ZITADEL (connector +# `customer-zitadel`, business-site#302). The setup CronJob +# (cron-job-setup.yaml) creates the client in ZITADEL and writes its ID and +# secret here, for the portal's customer Dex to read. A PushSecret keeps a copy +# in OpenBao (push-secret-dex-zitadel-client.yaml). +# +# Flux only creates this placeholder (ssa: IfNotPresent) and never overwrites +# the values the CronJob writes. While it holds the placeholder, nothing can +# sign in through the client. +# +# Not a real secret (literally "PLACEHOLDER"), so it is committed in plaintext, +# like the Velero placeholder. +apiVersion: v1 +kind: Secret +metadata: + name: customer-dex-zitadel-client + namespace: customer-identity + annotations: + kustomize.toolkit.fluxcd.io/ssa: IfNotPresent +type: Opaque +stringData: + client-id: PLACEHOLDER + client-secret: PLACEHOLDER # checkov:skip=CKV_SECRET_6:literal placeholder; the setup CronJob writes the real value diff --git a/k8s/providers/hetzner/apps/customer-identity/service-account.yaml b/k8s/providers/hetzner/apps/customer-identity/service-account.yaml new file mode 100644 index 0000000000..cd54d0e9c4 --- /dev/null +++ b/k8s/providers/hetzner/apps/customer-identity/service-account.yaml @@ -0,0 +1,8 @@ +# The setup CronJob's identity in Kubernetes (cron-job-setup.yaml). No token is +# mounted by default; the CronJob projects one into its setup container only. +apiVersion: v1 +kind: ServiceAccount +metadata: + name: customer-identity-setup + namespace: customer-identity +automountServiceAccountToken: false diff --git a/scripts/validate-eks-ci-role-policy/approved-surface.txt b/scripts/validate-eks-ci-role-policy/approved-surface.txt index 3a2036e2b9..6abc29f6dc 100644 --- a/scripts/validate-eks-ci-role-policy/approved-surface.txt +++ b/scripts/validate-eks-ci-role-policy/approved-surface.txt @@ -60,7 +60,7 @@ helm.toolkit.fluxcd.io/v2|HelmRelease|cnpg-system|plugin-barman-cloud 74682cb45a helm.toolkit.fluxcd.io/v2|HelmRelease|crossplane-system|crossplane fb717f7c9adb21b0f772e6a84554900e568a2675c91d375d1bb6ee76c2a0b1f3 -helm.toolkit.fluxcd.io/v2|HelmRelease|customer-identity|zitadel a2f82a6a6fe6d55240c66c5a084925c4007115a35c80c9aecb97eacbb3f862cb +helm.toolkit.fluxcd.io/v2|HelmRelease|customer-identity|zitadel f6fb7a8b2d084beb33aeb0a08b1b82eeeac495cbbe498b65154a57c8c353a905 helm.toolkit.fluxcd.io/v2|HelmRelease|dex|dex e2cbe6e6b5ace19ebded8d0e75f508277dd450c079ed51062d13649e148904b9 @@ -310,6 +310,8 @@ rbac.authorization.k8s.io/v1|RoleBinding|ate-system|substrate-ca-reader b8b1991b rbac.authorization.k8s.io/v1|RoleBinding|aws|aws-managed-resources 876e05322100f78b490b870ba8733b26daa0f93b65b8fdd77d559e9de94588c1 +rbac.authorization.k8s.io/v1|RoleBinding|customer-identity|customer-identity-setup fab367a0961fa37ea2870ac2bd77f7840cedeadc9f589ce7adc72ab758c71377 + rbac.authorization.k8s.io/v1|RoleBinding|data-product-controller|data-product-readiness-observer 00bcf69a4402d05bf5af23e273f9d3c5d7e7ebeed8c1778017a9f702d261360e rbac.authorization.k8s.io/v1|RoleBinding|data-product-controller|khoist-documentation 2690828917afd76647b314b9009a0401578780b33809e8d3a1f04a1b16244565 @@ -366,6 +368,8 @@ rbac.authorization.k8s.io/v1|Role|ate-system|substrate-ca-reader 153cc02bc210a1f rbac.authorization.k8s.io/v1|Role|aws|aws-managed-resources 9207b057fb6d4f1492b13663277658e3bd31849a9aa11dd2185ceb0a0afcbfab +rbac.authorization.k8s.io/v1|Role|customer-identity|customer-identity-setup 7f700c5af8b692b4e51b9e5150cf90b7d7e70f0f4b7d215e5b169a147fadfdcd + rbac.authorization.k8s.io/v1|Role|data-product-controller|data-product-readiness-observer 10df039485be13b0344199a50eacc4fbd942f2bb9b831a84b11c024918a3422a rbac.authorization.k8s.io/v1|Role|data-product-controller|khoist-documentation 35e63ab8f23f3de55976023d5c6144ac5a3cd7ed5c0947f1c451d74cf4167bef From ddc0c913da97f60cb350fcdce15ac1b4e754b89d Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Sun, 15 Nov 2026 11:24:35 +0100 Subject: [PATCH 10/12] fix(customer-identity): mount the setup CronJob's secrets as files MegaLinter found two things on the setup CronJob: - checkov CKV_K8S_35: the SMTP login and the one-time password arrived as environment variables, which every child process (curl, jq, kubectl) inherits. They are now mounted files the script reads into unexported variables, handed only to the one jq call that needs each. - trivy KSV-0036: a container mounted a token at the default service account path. The projected token now sits in /run/kube-api, and the script gives kubectl a kubeconfig that reads it from there. trivy KSV-0113 also fires on the setup Role, as on the other resourceNames-pinned Secret roles: it is dispositioned in .trivyignore.yaml, and its premises (one rule, secrets only, the one Secret, get and patch) are asserted by test-trivyignore-first-party-rbac-boundary.sh. Refs devantler-tech/business-site#302 Co-Authored-By: Claude Opus 5.5 --- .trivyignore.yaml | 10 ++ .../customer-identity/cron-job-setup.yaml | 99 +++++++++++++------ ...t-trivyignore-first-party-rbac-boundary.sh | 3 + 3 files changed, 83 insertions(+), 29 deletions(-) diff --git a/.trivyignore.yaml b/.trivyignore.yaml index 5b26958b3a..773252a408 100644 --- a/.trivyignore.yaml +++ b/.trivyignore.yaml @@ -541,6 +541,16 @@ misconfigurations: grants get alone, scoped by resourceNames to the six substrate-* CA Secrets cert-manager issues, so the identity cannot list or watch the namespace's Secrets, cannot read any other Secret, and cannot create, change or delete one. + - id: KSV-0113 + paths: + - k8s/providers/hetzner/apps/customer-identity/role.yaml + statement: >- + This is a namespaced Role in the customer-identity namespace, bound to the + customer-identity/customer-identity-setup ServiceAccount of the ZITADEL setup CronJob. Its one + rule grants get and patch alone, scoped by resourceNames to the single + customer-dex-zitadel-client Secret the CronJob fills with the client ZITADEL issues, so the + identity cannot list or watch the namespace's Secrets, cannot read any other Secret, and cannot + create or delete one. # KSV-0020 "runs with UID <= 10000" on the vault-config Job — 2 findings, both LOW. # # UID 100 is the identity the openbao image itself DEFINES — measured against the pinned diff --git a/k8s/providers/hetzner/apps/customer-identity/cron-job-setup.yaml b/k8s/providers/hetzner/apps/customer-identity/cron-job-setup.yaml index 8f7a09c41a..daf2e73afc 100644 --- a/k8s/providers/hetzner/apps/customer-identity/cron-job-setup.yaml +++ b/k8s/providers/hetzner/apps/customer-identity/cron-job-setup.yaml @@ -19,9 +19,10 @@ # - The signer (init container) alone mounts the private key of ZITADEL's # `zitadel-setup` system API user (certificate-setup-key.yaml). It writes a # 10-minute token to a memory volume and exits. -# - The setup container alone gets a Kubernetes token, which may only read and -# update the client Secret (role.yaml). It reads the SMTP login (zitadel-smtp) -# and the one-time password (zitadel-first-admin) through its environment. +# - The setup container alone gets a Kubernetes token, projected to +# /run/kube-api, which may only read and update the client Secret (role.yaml). +# It reads the SMTP login (zitadel-smtp) and the one-time password +# (zitadel-first-admin) as mounted files, never as environment variables. # - Neither prints a secret. # # The maintainer's own first sign-in (choosing his password, enrolling MFA) is @@ -139,31 +140,22 @@ spec: value: accounts.devantler.tech - name: TOKEN_FILE value: /run/setup-token/token - - name: SMTP_USER - valueFrom: - secretKeyRef: - name: zitadel-smtp - key: username - - name: SMTP_PASSWORD - valueFrom: - secretKeyRef: - name: zitadel-smtp - key: password - name: ADMIN_EMAIL value: ned@devantler.tech - - name: ADMIN_PASSWORD - valueFrom: - secretKeyRef: - name: zitadel-first-admin - key: password volumeMounts: - name: setup-token mountPath: /run/setup-token readOnly: true - name: tmp mountPath: /tmp - - name: kube-api-access - mountPath: /var/run/secrets/kubernetes.io/serviceaccount + - name: smtp + mountPath: /run/smtp + readOnly: true + - name: first-admin + mountPath: /run/first-admin + readOnly: true + - name: kube-api + mountPath: /run/kube-api readOnly: true resources: requests: @@ -185,8 +177,11 @@ spec: set -euo pipefail umask 077 - : "${ZITADEL_URL:?}" "${ZITADEL_HOST:?}" "${TOKEN_FILE:?}" "${SMTP_USER:?}" "${SMTP_PASSWORD:?}" - : "${ADMIN_EMAIL:?}" "${ADMIN_PASSWORD:?}" + : "${ZITADEL_URL:?}" "${ZITADEL_HOST:?}" "${TOKEN_FILE:?}" "${ADMIN_EMAIL:?}" + # Secrets arrive as files, never as environment variables. + smtp_dir=${SMTP_DIR:-/run/smtp} + admin_password_file=${ADMIN_PASSWORD_FILE:-/run/first-admin/password} + kube_api_dir=${KUBE_API_DIR:-/run/kube-api} namespace=customer-identity client_secret_name=customer-dex-zitadel-client # Fingerprints of what this script last applied, kept as annotations on the @@ -229,8 +224,38 @@ spec: work=$(mktemp -d) trap 'rm -rf "$work"' EXIT + read_secret() { [[ -s $1 ]] || fail "missing $2"; printf '%s' "$(<"$1")"; } [[ -s $TOKEN_FILE ]] || fail "no signed token at $TOKEN_FILE" printf 'Authorization: Bearer %s\n' "$(<"$TOKEN_FILE")" > "$work/auth" + smtp_user=$(read_secret "$smtp_dir/username" "SMTP username") + smtp_password=$(read_secret "$smtp_dir/password" "SMTP password") + admin_password=$(read_secret "$admin_password_file" "administrator one-time password") + + # The Kubernetes token is projected outside the default path, so kubectl gets + # a kubeconfig that reads it from there. + if [[ -z ${KUBECONFIG:-} ]]; then + cat > "$work/kubeconfig" </dev/null - api PUT "/admin/v1/smtp/$smtp_id/password" "$(jq -nc '{password: env.SMTP_PASSWORD}')" >/dev/null + api PUT "/admin/v1/smtp/$smtp_id/password" "$(SMTP_PASSWORD=$smtp_password jq -nc '{password: env.SMTP_PASSWORD}')" >/dev/null patch_secret "$(jq -nc --arg a "$fingerprint_annotation" --arg f "$fingerprint" '{metadata: {annotations: {($a): $f}}}')" changes+=("updated SMTP login") fi @@ -373,9 +398,9 @@ spec: admin_state=own if [[ $(jq -r '.result[0].human.passwordChangeRequired // false' <<<"$admins") == true ]]; then admin_state=one-time - admin_fingerprint=$(printf '%s' "$ADMIN_PASSWORD" | sha256sum | cut -d' ' -f1) + admin_fingerprint=$(printf '%s' "$admin_password" | sha256sum | cut -d' ' -f1) if [[ $(secret_json | jq -r --arg a "$admin_annotation" '.metadata.annotations[$a] // ""') != "$admin_fingerprint" ]]; then - connect zitadel.user.v2.UserService/SetPassword "$(jq -nc --arg u "$admin_id" \ + connect zitadel.user.v2.UserService/SetPassword "$(ADMIN_PASSWORD=$admin_password jq -nc --arg u "$admin_id" \ '{userId: $u, newPassword: {password: env.ADMIN_PASSWORD, changeRequired: true}}')" >/dev/null patch_secret "$(jq -nc --arg a "$admin_annotation" --arg f "$admin_fingerprint" '{metadata: {annotations: {($a): $f}}}')" changes+=("set the administrator's one-time password") @@ -406,7 +431,7 @@ spec: emptyDir: medium: Memory sizeLimit: 16Mi - - name: kube-api-access + - name: kube-api projected: defaultMode: 0440 sources: @@ -423,3 +448,19 @@ spec: - path: namespace fieldRef: fieldPath: metadata.namespace + - name: smtp + secret: + secretName: zitadel-smtp + defaultMode: 0440 + items: + - key: username + path: username + - key: password + path: password + - name: first-admin + secret: + secretName: zitadel-first-admin + defaultMode: 0440 + items: + - key: password + path: password diff --git a/scripts/tests/test-trivyignore-first-party-rbac-boundary.sh b/scripts/tests/test-trivyignore-first-party-rbac-boundary.sh index 347cf7aa76..d7a44a2b71 100755 --- a/scripts/tests/test-trivyignore-first-party-rbac-boundary.sh +++ b/scripts/tests/test-trivyignore-first-party-rbac-boundary.sh @@ -102,6 +102,7 @@ readonly reviewed_pairs=( "KSV-0113:$vault_config_role" # Premises asserted in section 6 below. "KSV-0113:k8s/bases/infrastructure/substrate/role.yaml" + "KSV-0113:k8s/providers/hetzner/apps/customer-identity/role.yaml" "KSV-0048:k8s/providers/hetzner/infrastructure/controllers/longhorn/role-kyverno-mutate-csi-sidecars.yaml" "KSV-0048:k8s/providers/hetzner/infrastructure/controllers/longhorn/role-kyverno-mutate-longhorn-csi-plugin.yaml" "KSV-0048:k8s/providers/hetzner/infrastructure/controllers/longhorn/role-kyverno-mutate-longhorn-engine-image.yaml" @@ -384,6 +385,8 @@ assert_single_rule_role KSV-0049 'k8s/bases/infrastructure/controllers/prune-pro '/configmaps' 'prune-protected-orphan-alert-state' 'get patch' assert_single_rule_role KSV-0113 'k8s/bases/infrastructure/substrate/role.yaml' ate-system '/secrets' \ 'substrate-actor-id-ca,substrate-actor-id-jwt,substrate-egress-mitm-ca,substrate-service-dns-ca,substrate-pod-identity-ca,substrate-postgres-ca' 'get' +assert_single_rule_role KSV-0113 'k8s/providers/hetzner/apps/customer-identity/role.yaml' customer-identity '/secrets' \ + 'customer-dex-zitadel-client' 'get patch' readonly war_role='k8s/providers/hetzner/apps/world-at-ruin/role-operator.yaml' assert_single_rule_role KSV-0053 "$war_role" world-at-ruin '/pods/portforward,pods/exec' '' 'create' From fff116cc8e7767bf81562a8fc749631c0c43912a Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Sun, 15 Nov 2026 11:27:24 +0100 Subject: [PATCH 11/12] fix(customer-identity): prove the Dex client secret and stop managing unused lists Two CodeRabbit findings on the setup CronJob: - ZITADEL reads an empty list in an update as "unchanged", so drift in the client's additional origins or post-logout redirect URIs made every update answer "No changes" and stopped every run. The client uses neither list: both are now set only at creation and left out of the drift check, and after any update the client is read back and must match. - After a database-only restore, ZITADEL can hold an older client secret than the stored one, which the old guard could not see. Every run now proves the stored secret against ZITADEL's token endpoint with a code that cannot exist: invalid_client means the secret is wrong, and a new one is issued. A newly issued secret must pass the same check before the run succeeds. Refs devantler-tech/business-site#302 Co-Authored-By: Claude Opus 5.5 --- docs/dr/crypto-custody.md | 4 ++ .../customer-identity/cron-job-setup.yaml | 66 +++++++++++++++---- 2 files changed, 59 insertions(+), 11 deletions(-) diff --git a/docs/dr/crypto-custody.md b/docs/dr/crypto-custody.md index 18b021db63..8cc54bd4c9 100644 --- a/docs/dr/crypto-custody.md +++ b/docs/dr/crypto-custody.md @@ -329,6 +329,10 @@ OpenBao is its custody of record: an existing value is never replaced, and - **Restore order:** restore OpenBao first ([openbao.md](openbao.md)), then `customer-identity-db` from its R2 backups. A fresh OpenBao would generate a new key that cannot read the restored database. +- **After a database restore:** the restored database can hold an older secret for the customer + Dex's client than `customer-identity/customer-dex-zitadel-client`. Nothing has to be done by + hand: the setup CronJob proves the stored secret against ZITADEL on every run and issues a new one + within 15 minutes when ZITADEL refuses it. - **Rotation:** not supported by ZITADEL for an existing database. Treat the key as permanent. ## ZITADEL setup user key diff --git a/k8s/providers/hetzner/apps/customer-identity/cron-job-setup.yaml b/k8s/providers/hetzner/apps/customer-identity/cron-job-setup.yaml index daf2e73afc..d6ce4565a5 100644 --- a/k8s/providers/hetzner/apps/customer-identity/cron-job-setup.yaml +++ b/k8s/providers/hetzner/apps/customer-identity/cron-job-setup.yaml @@ -2,9 +2,12 @@ # Every 15 minutes it reconciles, through ZITADEL's own APIs: # - the "Customer sign-in" project and its "Customer Dex" client, exactly as the # business-site preparer's CreateApplication template declares it (code grant -# only, the exact Dex callback, Login V1). It restores drifted settings and, -# when the stored secret is missing or is the placeholder, issues a new one -# into secret-dex-zitadel-client.yaml; +# only, the exact Dex callback, Login V1). It restores drifted settings; the +# two lists the client does not use (additional origins, post-logout URIs) +# are set only at creation, because ZITADEL cannot clear a list by update. +# Every run proves the stored secret against ZITADEL's token endpoint, and +# when it is missing, the placeholder or refused (for example after a +# database restore), it issues a new one into secret-dex-zitadel-client.yaml; # - the SES SMTP login: when the business site's credential (OpenBao # apps/business-site/smtp) changes, it updates ZITADEL's provider; # - the administrator's one-time password: until the maintainer's first @@ -200,12 +203,17 @@ spec: # The business-site preparer's CreateApplication template (server/internal/ # customeridentity/config.go): one confidential web client, code grant only, # the exact Dex callback, Login V1. - desired_oidc=$(jq -nc --arg cb "$dex_callback" '{ + create_oidc=$(jq -nc --arg cb "$dex_callback" '{ redirectUris: [$cb], responseTypes: ["OIDC_RESPONSE_TYPE_CODE"], grantTypes: ["OIDC_GRANT_TYPE_AUTHORIZATION_CODE"], applicationType: "OIDC_APP_TYPE_WEB", authMethodType: "OIDC_AUTH_METHOD_TYPE_BASIC", version: "OIDC_VERSION_1_0", developmentMode: false, accessTokenRoleAssertion: false, idTokenRoleAssertion: false, idTokenUserinfoAssertion: false, additionalOrigins: [], postLogoutRedirectUris: [], loginVersion: {loginV1: {}}}') + # The settings kept equal to that template on every run. ZITADEL cannot clear a + # list through an update (it reads an empty list as "unchanged"), so the two + # lists the client does not use, additional origins and post-logout redirect + # URIs, are set only at creation and not managed afterwards. + desired_oidc=$(jq -c 'del(.additionalOrigins, .postLogoutRedirectUris)' <<<"$create_oidc") # ZITADEL omits fields that hold their default value, so compare normalised. normalise_oidc='{ redirectUris: (.redirectUris // []), responseTypes: (.responseTypes // ["OIDC_RESPONSE_TYPE_CODE"]), @@ -216,7 +224,6 @@ spec: accessTokenRoleAssertion: (.accessTokenRoleAssertion // false), idTokenRoleAssertion: (.idTokenRoleAssertion // false), idTokenUserinfoAssertion: (.idTokenUserinfoAssertion // false), - additionalOrigins: (.additionalOrigins // []), postLogoutRedirectUris: (.postLogoutRedirectUris // []), loginVersion: (.loginVersion // {loginV1: {}})}' log() { printf 'setup: %s\n' "$*"; } @@ -284,13 +291,38 @@ spec: kubectl -n "$namespace" patch secret "$client_secret_name" --type merge --patch-file "$work/patch" >/dev/null || fail "cannot update Secret $client_secret_name" } + # client_authenticates ID SECRET: succeeds when ZITADEL accepts this client + # secret. A token request with a code that cannot exist fails either at client + # authentication (invalid_client: the secret is wrong, for example after a + # database restore) or after it, at the code (the secret is right). Any other + # answer stops the run. + client_authenticates() { + local status error + CLIENT_SECRET=$2 jq -rn --arg id "$1" '"Authorization: Basic " + ("\($id | @uri):\(env.CLIENT_SECRET | @uri)" | @base64)' \ + > "$work/client-auth" + status=$(curl -sS -m 20 -o "$work/token" -w '%{http_code}' -X POST -H "@$work/client-auth" \ + -H "Host: $ZITADEL_HOST" --data-urlencode grant_type=authorization_code \ + --data-urlencode code=customer-identity-setup-probe --data-urlencode "redirect_uri=$dex_callback" \ + "$ZITADEL_URL/oauth/v2/token") || fail "token endpoint: no answer from ZITADEL" + error=$(jq -r '.error // ""' "$work/token" 2>/dev/null || true) + case $error in + invalid_client) return 1 ;; + invalid_grant | invalid_request) return 0 ;; + *) fail "token endpoint: unexpected answer (HTTP $status${error:+, $error})" ;; + esac + } store_client() { - local id=$1 + local id=$1 attempt [[ -n $id && -n $CLIENT_SECRET ]] || fail "ZITADEL returned an empty client" patch_secret "$(jq -nc --arg id "$id" \ '{data: {"client-id": ($id | @base64), "client-secret": (env.CLIENT_SECRET | @base64)}}')" [[ $(secret_json | jq -r '.data["client-id"] // "" | @base64d') == "$id" ]] || fail "Secret $client_secret_name did not keep the new client" + for attempt in 1 2 3 4 5; do + client_authenticates "$id" "$CLIENT_SECRET" && return 0 + sleep 2 + done + fail "ZITADEL does not accept the client secret it issued" } for attempt in $(seq 1 20); do @@ -332,7 +364,7 @@ spec: case $(jq length <<<"$matches") in 0) connect zitadel.application.v2.ApplicationService/CreateApplication \ - "$(jq -nc --arg p "$project_id" --arg n "$app_name" --argjson c "$desired_oidc" \ + "$(jq -nc --arg p "$project_id" --arg n "$app_name" --argjson c "$create_oidc" \ '{projectId: $p, name: $n, oidcConfiguration: $c}')" > "$work/created" app_id=$(jq -er '.applicationId' "$work/created") client_id=$(jq -er '.oidcConfiguration.clientId' "$work/created") @@ -345,19 +377,31 @@ spec: client_id=$(jq -er '.oidcConfiguration.clientId' <<<"$app") [[ $(jq -r '.state' <<<"$app") == APPLICATION_STATE_ACTIVE ]] || fail "client '$app_name' is not active; reactivate or remove it on purpose" - if [[ $(jq -cS ".oidcConfiguration | $normalise_oidc" <<<"$app") != "$(jq -cS "$normalise_oidc" <<<"$desired_oidc")" ]]; then + want=$(jq -cS "$normalise_oidc" <<<"$desired_oidc") + if [[ $(jq -cS ".oidcConfiguration | $normalise_oidc" <<<"$app") != "$want" ]]; then connect zitadel.application.v2.ApplicationService/UpdateApplication \ "$(jq -nc --arg p "$project_id" --arg a "$app_id" --argjson c "$desired_oidc" \ '{projectId: $p, applicationId: $a, oidcConfiguration: $c}')" >/dev/null + [[ $(connect zitadel.application.v2.ApplicationService/ListApplications \ + "$(jq -nc --arg p "$project_id" '{pagination: {limit: 1000}, filters: [{projectIdFilter: {projectId: $p}}]}')" | + jq -cS --arg a "$app_id" ".applications[] | select(.applicationId == \$a) | .oidcConfiguration | $normalise_oidc") == "$want" ]] || + fail "client '$app_name' still differs from its declared settings after the update" changes+=("restored client settings") fi stored=$(secret_json) - if [[ $(jq -r '.data["client-id"] // "" | @base64d' <<<"$stored") != "$client_id" ]] || - [[ $(jq -r '.data["client-secret"] // "" | @base64d' <<<"$stored") =~ ^($placeholder)?$ ]]; then + stored_id=$(jq -r '.data["client-id"] // "" | @base64d' <<<"$stored") + stored_secret=$(jq -r '.data["client-secret"] // "" | @base64d' <<<"$stored") + reissue= + if [[ $stored_id != "$client_id" || $stored_secret =~ ^($placeholder)?$ ]]; then + reissue="none stored" + elif ! client_authenticates "$client_id" "$stored_secret"; then + reissue="stored one refused" + fi + if [[ -n $reissue ]]; then connect zitadel.application.v2.ApplicationService/GenerateClientSecret \ "$(jq -nc --arg p "$project_id" --arg a "$app_id" '{projectId: $p, applicationId: $a}')" > "$work/generated" CLIENT_SECRET=$(jq -er '.clientSecret' "$work/generated") store_client "$client_id" - changes+=("issued client secret") + changes+=("issued client secret ($reissue)") fi ;; *) fail "more than one client named '$app_name'" ;; From d63c6b4c768afef8e494e6394f582df6c9290a51 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Sun, 15 Nov 2026 11:30:36 +0100 Subject: [PATCH 12/12] test(customer-identity): list the setup Role's KSV-0113 disposition as reviewed The vendored-operator boundary test rejects any KSV-0113 disposition it does not know, so the reviewed one on the setup Role is added to its first-party pairs, beside the matching entry in the first-party RBAC premises test. Refs devantler-tech/business-site#302 Co-Authored-By: Claude Opus 5.5 --- scripts/tests/test-trivyignore-vendored-operator-boundary.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/tests/test-trivyignore-vendored-operator-boundary.sh b/scripts/tests/test-trivyignore-vendored-operator-boundary.sh index 429ebdbbb2..f027a28ef1 100755 --- a/scripts/tests/test-trivyignore-vendored-operator-boundary.sh +++ b/scripts/tests/test-trivyignore-vendored-operator-boundary.sh @@ -62,6 +62,7 @@ readonly first_party_pairs=( 'KSV-0048:k8s/bases/infrastructure/resource-graph-definitions/webapp/cluster-role.yaml' 'KSV-0113:k8s/bases/infrastructure/vault-config/role.yaml' 'KSV-0113:k8s/bases/infrastructure/substrate/role.yaml' + 'KSV-0113:k8s/providers/hetzner/apps/customer-identity/role.yaml' 'KSV-0048:k8s/providers/hetzner/infrastructure/controllers/longhorn/role-kyverno-mutate-csi-sidecars.yaml' 'KSV-0048:k8s/providers/hetzner/infrastructure/controllers/longhorn/role-kyverno-mutate-longhorn-csi-plugin.yaml' 'KSV-0048:k8s/providers/hetzner/infrastructure/controllers/longhorn/role-kyverno-mutate-longhorn-engine-image.yaml'