This document describes how secrets are managed in the homelab using SOPS encryption with age keys.
Secrets are encrypted using SOPS with age encryption. This allows secrets to be stored securely in Git while remaining readable only by those with the decryption key.
Flux automatically decrypts secrets during deployment using the cluster's age key stored in the flux-system/sops-age secret.
Each environment has its own SOPS configuration and age encryption key for security isolation:
| Environment | Public Key | Backup |
|---|---|---|
| Staging | age1uq9nturwsx36q045qtrm85lkg8qmzpgk9srduqesxs2ahjurw53sp9rhm6 |
Proton Pass |
| Production | age1hh6cdyljk2ks5mkmxqx6g65c7a8rgndy5p2s2d7w2gvqx4h53ggqtwr7rh |
Proton Pass |
Both private keys live in SOPS's default key file ~/.config/sops/age/keys.txt
(one key per line, preceded by a # staging / # production comment) — never
inside the repository working tree, where a git add -f, a careless
.gitignore edit, or a directory tarball could leak them. SOPS reads this file
automatically, so no SOPS_AGE_KEY_FILE export is needed and decryption works
for both environments transparently.
keys.txt is provisioned by the dotfiles repo
(github.com/ronaldlokers/dotfiles): it is stored there age-encrypted with a
dedicated dotfiles key and decrypted on chezmoi apply, which prompts once
per machine for the dotfiles passphrase. Container rebuilds therefore need no
manual key handling; Proton Pass holds the passphrase and both raw keys as
the disaster-recovery backup.
Important:
- Private keys are stored in Proton Pass and, age-encrypted, in the dotfiles repo — never in plaintext in any repository
- Public keys are used in
.sops.yamlconfiguration files - Each environment can only decrypt its own secrets
Each cluster has its own .sops.yaml configuration file:
creation_rules:
- path_regex: .*.yaml
encrypted_regex: ^(data|stringData)$
age: age1uq9nturwsx36q045qtrm85lkg8qmzpgk9srduqesxs2ahjurw53sp9rhm6creation_rules:
- path_regex: .*.yaml
encrypted_regex: ^(data|stringData)$
age: age1hh6cdyljk2ks5mkmxqx6g65c7a8rgndy5p2s2d7w2gvqx4h53ggqtwr7rhThese configurations:
- Apply to all
.yamlfiles in the directory - Only encrypt the
dataandstringDatafields (Secret content) - Leave metadata unencrypted for readability
- Use environment-specific age public keys
These secrets must be manually created in each cluster before deployment.
Created automatically during Flux bootstrap:
export GITHUB_USER=ronaldlokers
export GITHUB_TOKEN=<personal-access-token>
# For staging
flux bootstrap github \
--context=staging \
--owner=$GITHUB_USER \
--repository=homelab \
--branch=main \
--path=./clusters/staging \
--personal
# For production
flux bootstrap github \
--context=production \
--owner=$GITHUB_USER \
--repository=homelab \
--branch=main \
--path=./clusters/production \
--personalThis secret contains:
- GitHub deploy key for repository access
- Known hosts for Git over SSH
Age private key for decrypting SOPS-encrypted secrets.
Each cluster's secret must contain only its own key (extracted from
~/.config/sops/age/keys.txt by its comment marker):
Create for staging:
grep -A1 "^# staging" ~/.config/sops/age/keys.txt | grep AGE-SECRET-KEY | \
kubectl --context=staging create secret generic sops-age \
--namespace=flux-system \
--from-file=age.agekey=/dev/stdinCreate for production:
grep -A1 "^# production" ~/.config/sops/age/keys.txt | grep AGE-SECRET-KEY | \
kubectl --context=production create secret generic sops-age \
--namespace=flux-system \
--from-file=age.agekey=/dev/stdinThe private keys are retrieved from Proton Pass.
These secrets are encrypted with SOPS and stored in this repository. Flux decrypts them automatically during deployment.
Cloudflare DNS API token for Let's Encrypt DNS-01 challenges.
Location:
- Staging:
infrastructure/configs/staging/cert-manager/cloudflare-api-token-secret.yaml - Production:
infrastructure/configs/production/cert-manager/cloudflare-api-token-secret.yaml
Contents:
api-token: Cloudflare API token with DNS edit permissions
Linkding application environment variables.
Location:
- Staging:
apps/staging/linkding/linkding-container-env-secret.yaml - Production:
apps/production/linkding/linkding-container-env-secret.yaml
Contents:
LD_SUPERUSER_NAME: Admin usernameLD_SUPERUSER_PASSWORD: Admin password
PostgreSQL database credentials for Linkding.
Location:
- Staging:
apps/staging/linkding/linkding-db-secret.yaml - Production:
apps/production/linkding/linkding-db-secret.yaml
Contents:
username: PostgreSQL user (app)password: PostgreSQL password (from postgres-cluster-app secret)
Backblaze B2 object storage credentials for PostgreSQL backups.
Location:
- Staging:
infrastructure/configs/staging/cloudnative-pg/b2-credentials-secret.yaml - Production:
infrastructure/configs/production/cloudnative-pg/b2-credentials-secret.yaml
Contents:
ACCESS_KEY_ID: Backblaze B2 application key IDACCESS_SECRET_KEY: Backblaze B2 application key
Purpose:
- Automated PostgreSQL backups to Backblaze B2
- WAL archiving for point-in-time recovery
- Off-cluster disaster recovery
Backup Details:
- Bucket:
homelab-postgres-backups - Paths:
staging/andproduction/subdirectories - Retention: 14 days (staging), 7 days (production)
- Schedule: Daily automated backups per cluster (
ScheduledBackupresources for both postgres-cluster and immich-cluster)
Security Notes:
- Credentials encrypted with SOPS
- Application key scoped to backup bucket only
- Read and write access required for backup/restore operations
- Store key ID and secret separately from this repository
Nightscout application environment variables.
Location:
- Staging:
apps/staging/nightscout/nightscout-env-secret.yaml - Production:
apps/production/nightscout/nightscout-env-secret.yaml
Contents:
API_SECRET: Nightscout API authentication secret
Purpose:
- Authentication for Nightscout web interface
- API access control for uploading CGM data
- Required for all data modifications
PostgreSQL database credentials for FerretDB (MongoDB API compatibility layer).
Location:
- Staging:
apps/staging/nightscout/ferretdb-postgres-secret.yaml - Production:
apps/production/nightscout/ferretdb-postgres-secret.yaml
Contents:
FERRETDB_POSTGRESQL_PASSWORD: PostgreSQL password (from postgres-cluster-app secret)
Purpose:
- FerretDB connects to PostgreSQL using these credentials
- Provides MongoDB API compatibility for Nightscout
- Uses same PostgreSQL cluster as other applications
Renovate GitHub token for automated dependency updates.
Location:
- Staging:
infrastructure/controllers/staging/renovate/renovate-container-env.yaml - Production:
infrastructure/controllers/production/renovate/renovate-container-env.yaml
Contents:
RENOVATE_TOKEN: GitHub personal access token with repository write permissions
Homepage widget credentials.
Location:
- Staging:
apps/staging/homepage/homepage-env-secret.yaml - Production:
apps/production/homepage/homepage-env-secret.yaml
Contents:
HOMEPAGE_VAR_GRAFANA_USER: Grafana admin usernameHOMEPAGE_VAR_GRAFANA_PASSWORD: Grafana admin password
These secrets are automatically created by controllers and should not be manually managed.
Created by cert-manager:
*-tls: TLS certificates for ingressesletsencrypt-production: Let's Encrypt ACME account keyletsencrypt-staging: Let's Encrypt staging ACME account key
Created by kube-prometheus-stack:
kube-prometheus-stack-grafana: Grafana admin username and password
Retrieve Grafana credentials:
# For staging
kubectl --context=staging get secret -n monitoring kube-prometheus-stack-grafana \
-o jsonpath='{.data.admin-password}' | base64 -d
# For production
kubectl --context=production get secret -n monitoring kube-prometheus-stack-grafana \
-o jsonpath='{.data.admin-password}' | base64 -dCreated by kube-prometheus-stack:
- Configuration secrets for Prometheus
- Configuration secrets for Alertmanager
Prerequisites: The private age keys must be present in ~/.config/sops/age/keys.txt (see Encryption Keys). SOPS finds them automatically — no environment variables required.
- Create the secret YAML file:
apiVersion: v1
kind: Secret
metadata:
name: my-secret
namespace: my-namespace
type: Opaque
stringData:
key: my-value # Use stringData for plain text- Encrypt with SOPS
export AGE_PUBLIC=<public age key>
sops --age=$AGE_PUBLIC --encrypt --encrypted-regex '^(data|stringData)$' --in-place test-secret.yaml# Works for both environments — SOPS picks the right key from
# ~/.config/sops/age/keys.txt automatically
sops path/to/secret.yamlSOPS will:
- Decrypt the file using your age key
- Open it in your default editor
- Re-encrypt it when you save and close
# For staging
export SOPS_AGE_KEY_FILE=/workspaces/homelab/staging-age.key
sops --decrypt path/to/secret.yaml
# For production
export SOPS_AGE_KEY_FILE=/workspaces/homelab/production-age.key
sops --decrypt path/to/secret.yaml- Update the
sops-agesecret in the cluster with the new private key
- Never commit private keys: Age private keys should never be committed to Git
- Use separate keys per environment: Staging and production have separate keys
- Store private keys securely: Use Proton Pass or another password manager
- Rotate keys periodically: Re-encrypt secrets with new age keys regularly
- Limit key access: Only share private keys with trusted team members
- Verify encryption: Always check that secrets are encrypted before committing
- Use strong tokens: Generate strong passwords and API tokens
- Rotate credentials: Periodically rotate API tokens and passwords
If Flux reports decryption errors:
- Verify the
sops-agesecret exists:
kubectl --context=production get secret -n flux-system sops-age- Check the secret contains the correct key:
kubectl --context=production get secret -n flux-system sops-age \
-o jsonpath='{.data.age\.agekey}' | base64 -d- Verify the secret was encrypted with the correct public key:
sops --decrypt path/to/secret.yamlIf decryption fails locally, the secret was encrypted with a different key.
If a secret exists but isn't being decrypted by Flux:
- Check Kustomization for SOPS configuration:
kubectl --context=production get kustomization -n flux-system infrastructure-controllers -o yamlLook for:
spec:
decryption:
provider: sops
secretRef:
name: sops-age- Check Flux logs:
kubectl --context=production logs -n flux-system -l app=kustomize-controllerIf you accidentally encrypted a staging secret with the production key (or vice versa):
- Decrypt with the correct key (the one it was encrypted with):
export SOPS_AGE_KEY_FILE=/workspaces/homelab/production-age.key # or staging-age.key
sops --decrypt path/to/secret.yaml > /tmp/secret-decrypted.yaml- Re-encrypt with the correct key (the one it should use):
export SOPS_AGE_KEY_FILE=/workspaces/homelab/staging-age.key # or production-age.key
sops --encrypt /tmp/secret-decrypted.yaml > path/to/secret.yaml
rm /tmp/secret-decrypted.yamlvalidate.sh decrypts every SOPS file when an age key is present, and fails
when one no longer round-trips. Four had rotted unnoticed before that check
existed (#497), in two ways — both of them "the file was edited after it was
encrypted":
| Symptom | Cause |
|---|---|
MAC mismatch |
The MAC covers every value in the tree, encrypted or not. Adding an annotation by hand after encryption invalidates it while every value still decrypts. |
cipher: message authentication failed |
A multi-document file whose later documents were encrypted separately and appended, each with its own data key. Decrypting the file as a whole tries one key against another's values. |
The cluster keeps working throughout: it holds the decrypted Secret and never reads the file again. What breaks is editing or rotating it.
Recovering one:
- Decrypt what is there.
sops --decrypt --ignore-macfor a MAC mismatch; for the multi-document case, split on---and decrypt each document separately, then join the plaintext back together. - Check the recovered values against the live Secret before trusting them:
Each value's own GCM tag is verified during decryption, so a value that decrypts is authentic — but the comparison also proves the file is the one the cluster is running.
kubectl --context=production get secret -n <ns> <name> \ -o jsonpath='{.data.<key>}' | base64 -d | sha256sum
- Re-encrypt the whole file with the environment's key, then confirm a plain
sops --decryptsucceeds with no flags.