Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .trivyignore.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -541,6 +541,16 @@ misconfigurations:
grants get alone, scoped by resourceNames to the six substrate-* CA Secrets cert-manager
issues, so the identity cannot list or watch the namespace's Secrets, cannot read any other
Secret, and cannot create, change or delete one.
- id: KSV-0113
paths:
- k8s/providers/hetzner/apps/customer-identity/role.yaml
statement: >-
This is a namespaced Role in the customer-identity namespace, bound to the
customer-identity/customer-identity-setup ServiceAccount of the ZITADEL setup CronJob. Its one
rule grants get and patch alone, scoped by resourceNames to the single
customer-dex-zitadel-client Secret the CronJob fills with the client ZITADEL issues, so the
identity cannot list or watch the namespace's Secrets, cannot read any other Secret, and cannot
create or delete one.
# KSV-0020 "runs with UID <= 10000" on the vault-config Job — 2 findings, both LOW.
#
# UID 100 is the identity the openbao image itself DEFINES — measured against the pinned
Expand Down
34 changes: 34 additions & 0 deletions docs/dr/crypto-custody.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@ and [openbao.md](openbao.md) (OpenBao-specific DR).
| **Hetzner Cloud token** | provision / destroy nodes; reconfigure Cloud LB and firewall | destroy infrastructure; pivot via the LB | mint a new one in the Hetzner console; rotate `HCLOUD_TOKEN` |
| **Cloudflare API token** | manage DNS for the platform domain | redirect prod traffic; mint Origin CA certs | mint a new one in the Cloudflare dashboard |
| **Actual Budget E2EE password** | client-side key to encrypt/decrypt an Actual budget file | decrypt that budget's financial data | the budget is **permanently unrecoverable** — the server never held the key |
| **ZITADEL master key** | encrypt the customer identity provider's signing keys and stored secrets (`customer-identity` namespace) | with a database copy, decrypt those keys and secrets — rotate them in ZITADEL | restored `customer-identity-db` backups cannot be used; customer accounts are recreated |
| **ZITADEL setup user key** | sign the setup CronJob's tokens as ZITADEL's `zitadel-setup` system API user (instance owner) | administer the customer identity provider — delete `customer-identity/zitadel-setup-key` so cert-manager issues a new key | nothing to recover: cert-manager issues a new key and ZITADEL trusts the new certificate |

The rest of this document walks each row in detail.

Expand Down Expand Up @@ -316,6 +318,38 @@ strongest reason to keep the OpenBao entry backed up.

---

## ZITADEL master key

ZITADEL (customer sign-in for the business-site portal) encrypts its signing keys and stored
secrets with a 32-character master key. The key is generated once in-cluster and written to
OpenBao `apps/customer-identity/zitadel` (property `masterkey`) by a create-only PushSecret
(`k8s/bases/infrastructure/vault-seed/push-secret-push-customer-identity-zitadel-masterkey.yaml`).
OpenBao is its custody of record: an existing value is never replaced, and
`customer-identity/zitadel-masterkey` is read back from it.

- **Restore order:** restore OpenBao first ([openbao.md](openbao.md)), then `customer-identity-db`
from its R2 backups. A fresh OpenBao would generate a new key that cannot read the restored database.
- **After a database restore:** the restored database can hold an older secret for the customer
Dex's client than `customer-identity/customer-dex-zitadel-client`. Nothing has to be done by
hand: the setup CronJob proves the stored secret against ZITADEL on every run and issues a new one
within 15 minutes when ZITADEL refuses it.
- **Rotation:** not supported by ZITADEL for an existing database. Treat the key as permanent.

## ZITADEL setup user key

The setup CronJob (`k8s/providers/hetzner/apps/customer-identity/cron-job-setup.yaml`) reaches
ZITADEL's APIs as the system API user `zitadel-setup`, whose only role is `IAM_OWNER` on the instance.
ZITADEL trusts the public key in cert-manager's certificate `customer-identity/zitadel-setup-key`.
It mounts only `tls.crt`; only the CronJob's signer container mounts `tls.key`, and no Role grants
reading the Secret.

- **Custody:** the cluster only. The key is not backed up and needs no backup: a rebuild issues a new
one, and ZITADEL trusts whatever certificate it mounts.
- **Rotation:** automatic. cert-manager issues a new key 30 days before the 90-day certificate expires,
and Reloader restarts ZITADEL to read it. To rotate at once, delete the Secret.

---

## Cross-references

- [`runbook.md`](runbook.md) — operational scenarios (rotation steps, full
Expand Down
28 changes: 27 additions & 1 deletion k8s/bases/infrastructure/vault-config/job.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -906,6 +906,16 @@ spec:
}
POLICY

# Customer sign-in (apps/customer-identity, business-site#302)
# reads ZITADEL's master key, and later the customer Dex's
# ZITADEL client, through the shared `external-secrets`
# ClusterSecretStore role.
bao policy write app-customer-identity-readonly - <<'POLICY'
path "secret/data/apps/customer-identity/*" {
capabilities = ["read"]
}
POLICY

# actual-budget reads its Enable Banking bank-sync credential
# (application-id + secret-key) from OpenBao via the shared
# `external-secrets` ClusterSecretStore role (apps/actual-budget/
Expand Down Expand Up @@ -1067,6 +1077,16 @@ spec:
path "secret/data/apps/umami/*" {
capabilities = ["create", "update", "read"]
}
# ZITADEL's master key, generated once by its create-only
# PushSecret (vault-seed), and the customer Dex's ZITADEL
# client, which the customer-identity setup CronJob issues and
# a PushSecret copies here. Exact keys only.
path "secret/data/apps/customer-identity/zitadel" {
capabilities = ["create", "update", "read"]
}
path "secret/data/apps/customer-identity/dex-zitadel-client" {
capabilities = ["create", "update", "read"]
}
path "secret/data/apps/actual-budget/*" {
capabilities = ["create", "update", "read"]
}
Expand Down Expand Up @@ -1128,6 +1148,12 @@ spec:
path "secret/metadata/apps/umami/*" {
capabilities = ["create", "update", "read"]
}
path "secret/metadata/apps/customer-identity/zitadel" {
capabilities = ["create", "update", "read"]
}
path "secret/metadata/apps/customer-identity/dex-zitadel-client" {
capabilities = ["create", "update", "read"]
}
path "secret/metadata/apps/actual-budget/*" {
capabilities = ["create", "update", "read"]
}
Expand Down Expand Up @@ -1163,7 +1189,7 @@ spec:
bao write auth/kubernetes/role/external-secrets \
bound_service_account_names=external-secrets \
bound_service_account_namespaces=external-secrets \
policies=infra-backup-readonly,infra-dns-readonly,infra-monitoring-readonly,infra-hcloud-readonly,infra-tls-readonly,infra-ghcr-readonly,app-fleetdm-readonly,app-wedding-readonly,app-umami-readonly,app-kagent-readonly,app-actual-budget-readonly,app-ascoachingogvaner-readonly,infra-oidc-readonly,vault-seed-write \
policies=infra-backup-readonly,infra-dns-readonly,infra-monitoring-readonly,infra-hcloud-readonly,infra-tls-readonly,infra-ghcr-readonly,app-fleetdm-readonly,app-wedding-readonly,app-umami-readonly,app-kagent-readonly,app-customer-identity-readonly,app-actual-budget-readonly,app-ascoachingogvaner-readonly,infra-oidc-readonly,vault-seed-write \
ttl=1h

# Dedicated role for the fleetdm VaultDynamicSecret generator.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: generated-customer-identity-zitadel-masterkey
namespace: openbao
spec:
refreshInterval: "0"
target:
name: generated-customer-identity-zitadel-masterkey
creationPolicy: Owner
dataFrom:
- sourceRef:
generatorRef:
apiVersion: generators.external-secrets.io/v1alpha1
kind: Password
name: customer-identity-zitadel-masterkey
3 changes: 3 additions & 0 deletions k8s/bases/infrastructure/vault-seed/kustomization.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@ resources:
- push-secret-push-umami-admin-password.yaml
- external-secret-generated-umami-ascoachingogvaner-password.yaml
- push-secret-push-umami-ascoachingogvaner-password.yaml
- password-customer-identity-zitadel-masterkey.yaml
- external-secret-generated-customer-identity-zitadel-masterkey.yaml
- push-secret-push-customer-identity-zitadel-masterkey.yaml
- push-secret-seed-r2-credentials.yaml
- push-secret-seed-wedding-db-backup-r2.yaml
- push-secret-seed-github-app.yaml
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
# ZITADEL requires exactly 32 characters.
apiVersion: generators.external-secrets.io/v1alpha1
kind: Password
metadata:
name: customer-identity-zitadel-masterkey
namespace: openbao
spec:
length: 32
digits: 8
symbols: 0
noUpper: false
allowRepeat: true
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
---
# ZITADEL's master key (apps/customer-identity/external-secret-masterkey.yaml).
# Create-only: once OpenBao holds a key, a regenerated in-cluster value never
# replaces it, so a rebuilt cluster that restores OpenBao keeps the key its
# restored database was encrypted with (docs/dr/crypto-custody.md).
apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
name: push-customer-identity-zitadel-masterkey
namespace: openbao
spec:
refreshInterval: 1h
updatePolicy: IfNotExists
secretStoreRefs:
- name: openbao
kind: ClusterSecretStore
selector:
secret:
name: generated-customer-identity-zitadel-masterkey
data:
- match:
secretKey: password
remoteRef:
remoteKey: apps/customer-identity/zitadel
property: masterkey
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# The key pair of ZITADEL's `zitadel-setup` system API user (helm-release.yaml
# SystemAPIUsers). ZITADEL mounts only tls.crt; only the setup CronJob's
# signer mounts tls.key (cron-job-setup.yaml), and no Role lets anyone read
# the Secret through the API.
#
# cert-manager renews it with a new key 30 days before it expires. Reloader
# then restarts ZITADEL so it reads the new certificate; a setup run during
# that restart fails and the next one succeeds.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: zitadel-setup-key
namespace: customer-identity
spec:
secretName: zitadel-setup-key
commonName: zitadel-setup
duration: 2160h
renewBefore: 720h
isCA: false
usages:
- digital signature
privateKey:
algorithm: RSA
size: 2048
encoding: PKCS8
rotationPolicy: Always
issuerRef:
name: customer-identity-setup
kind: Issuer
18 changes: 18 additions & 0 deletions k8s/providers/hetzner/apps/customer-identity/certificate.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# DNS-01 certificate for the customer identity names: ZITADEL
# (accounts.devantler.tech) and the portal's customer Dex
# (login.devantler.tech, next change). The Gateway listeners read it through
# reference-grant.yaml.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: customer-identity
namespace: customer-identity
spec:
secretName: customer-identity-tls
dnsNames:
- accounts.devantler.tech
- login.devantler.tech
issuerRef:
group: cert-manager.io
kind: ClusterIssuer
name: letsencrypt-prod
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Let Coroot's cluster-agent (observability namespace) reach the
# customer-identity-db instances on 5432 for its Postgres integration
# (cluster.yaml). Additive: it admits only the cluster-agent, and only to the
# database pods.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: allow-coroot-postgres-scrape
namespace: customer-identity
spec:
endpointSelector:
matchLabels:
cnpg.io/cluster: customer-identity-db
ingress:
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: observability
app.kubernetes.io/managed-by: coroot-operator
app.kubernetes.io/part-of: coroot
app.kubernetes.io/component: coroot-cluster-agent
toPorts:
- ports:
- port: "5432"
protocol: TCP
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# ZITADEL and its database. Only the Gateway reaches ZITADEL (8080); only
# ZITADEL, the other instance and the CNPG operator reach the database.
# Egress: the namespace itself, the API server (CNPG instance manager), R2
# for backups, and the business site's SES SMTP endpoint for ZITADEL's
# invitation and recovery mail (platform#5400).
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: allow-customer-identity
namespace: customer-identity
spec:
endpointSelector: {}
ingress:
- fromEntities:
- ingress
toPorts:
- ports:
- port: "8080"
protocol: TCP
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: customer-identity
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: cnpg-system
toPorts:
- ports:
- port: "8000"
protocol: TCP
- port: "5432"
protocol: TCP
egress:
- toEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: customer-identity
- toEntities:
- kube-apiserver
- toFQDNs:
- matchPattern: "*.r2.cloudflarestorage.com"
toPorts:
- ports:
- port: "443"
protocol: TCP
- toFQDNs:
- matchName: email-smtp.eu-central-1.amazonaws.com
toPorts:
- ports:
- port: "587"
protocol: TCP
- toEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: kube-system
k8s-app: kube-dns
toPorts:
- ports:
- port: "53"
protocol: UDP
- port: "53"
protocol: TCP
rules:
dns:
- matchPattern: "*"
Loading
Loading