Skip to content

Security: ronaldlokers/homelab

Security

docs/security.md

Security & Secrets Management

This document describes how secrets are managed in the homelab using SOPS encryption with age keys.

Overview

Secrets are encrypted using SOPS with age encryption. This allows secrets to be stored securely in Git while remaining readable only by those with the decryption key.

Flux automatically decrypts secrets during deployment using the cluster's age key stored in the flux-system/sops-age secret.

Encryption Keys

Each environment has its own SOPS configuration and age encryption key for security isolation:

Environment Public Key Backup
Staging age1uq9nturwsx36q045qtrm85lkg8qmzpgk9srduqesxs2ahjurw53sp9rhm6 Proton Pass
Production age1hh6cdyljk2ks5mkmxqx6g65c7a8rgndy5p2s2d7w2gvqx4h53ggqtwr7rh Proton Pass

Both private keys live in SOPS's default key file ~/.config/sops/age/keys.txt (one key per line, preceded by a # staging / # production comment) — never inside the repository working tree, where a git add -f, a careless .gitignore edit, or a directory tarball could leak them. SOPS reads this file automatically, so no SOPS_AGE_KEY_FILE export is needed and decryption works for both environments transparently.

keys.txt is provisioned by the dotfiles repo (github.com/ronaldlokers/dotfiles): it is stored there age-encrypted with a dedicated dotfiles key and decrypted on chezmoi apply, which prompts once per machine for the dotfiles passphrase. Container rebuilds therefore need no manual key handling; Proton Pass holds the passphrase and both raw keys as the disaster-recovery backup.

Important:

  • Private keys are stored in Proton Pass and, age-encrypted, in the dotfiles repo — never in plaintext in any repository
  • Public keys are used in .sops.yaml configuration files
  • Each environment can only decrypt its own secrets

SOPS Configuration

Each cluster has its own .sops.yaml configuration file:

Staging: clusters/staging/.sops.yaml

creation_rules:
  - path_regex: .*.yaml
    encrypted_regex: ^(data|stringData)$
    age: age1uq9nturwsx36q045qtrm85lkg8qmzpgk9srduqesxs2ahjurw53sp9rhm6

Production: clusters/production/.sops.yaml

creation_rules:
  - path_regex: .*.yaml
    encrypted_regex: ^(data|stringData)$
    age: age1hh6cdyljk2ks5mkmxqx6g65c7a8rgndy5p2s2d7w2gvqx4h53ggqtwr7rh

These configurations:

  • Apply to all .yaml files in the directory
  • Only encrypt the data and stringData fields (Secret content)
  • Leave metadata unencrypted for readability
  • Use environment-specific age public keys

Required Secrets

These secrets must be manually created in each cluster before deployment.

flux-system/flux-system

Created automatically during Flux bootstrap:

export GITHUB_USER=ronaldlokers
export GITHUB_TOKEN=<personal-access-token>

# For staging
flux bootstrap github \
  --context=staging \
  --owner=$GITHUB_USER \
  --repository=homelab \
  --branch=main \
  --path=./clusters/staging \
  --personal

# For production
flux bootstrap github \
  --context=production \
  --owner=$GITHUB_USER \
  --repository=homelab \
  --branch=main \
  --path=./clusters/production \
  --personal

This secret contains:

  • GitHub deploy key for repository access
  • Known hosts for Git over SSH

flux-system/sops-age

Age private key for decrypting SOPS-encrypted secrets.

Each cluster's secret must contain only its own key (extracted from ~/.config/sops/age/keys.txt by its comment marker):

Create for staging:

grep -A1 "^# staging" ~/.config/sops/age/keys.txt | grep AGE-SECRET-KEY | \
  kubectl --context=staging create secret generic sops-age \
  --namespace=flux-system \
  --from-file=age.agekey=/dev/stdin

Create for production:

grep -A1 "^# production" ~/.config/sops/age/keys.txt | grep AGE-SECRET-KEY | \
  kubectl --context=production create secret generic sops-age \
  --namespace=flux-system \
  --from-file=age.agekey=/dev/stdin

The private keys are retrieved from Proton Pass.

Managed Secrets

These secrets are encrypted with SOPS and stored in this repository. Flux decrypts them automatically during deployment.

cert-manager/cloudflare-api-token

Cloudflare DNS API token for Let's Encrypt DNS-01 challenges.

Location:

  • Staging: infrastructure/configs/staging/cert-manager/cloudflare-api-token-secret.yaml
  • Production: infrastructure/configs/production/cert-manager/cloudflare-api-token-secret.yaml

Contents:

  • api-token: Cloudflare API token with DNS edit permissions

linkding/linkding-container-env

Linkding application environment variables.

Location:

  • Staging: apps/staging/linkding/linkding-container-env-secret.yaml
  • Production: apps/production/linkding/linkding-container-env-secret.yaml

Contents:

  • LD_SUPERUSER_NAME: Admin username
  • LD_SUPERUSER_PASSWORD: Admin password

linkding/linkding-db-user

PostgreSQL database credentials for Linkding.

Location:

  • Staging: apps/staging/linkding/linkding-db-secret.yaml
  • Production: apps/production/linkding/linkding-db-secret.yaml

Contents:

  • username: PostgreSQL user (app)
  • password: PostgreSQL password (from postgres-cluster-app secret)

database/b2-credentials

Backblaze B2 object storage credentials for PostgreSQL backups.

Location:

  • Staging: infrastructure/configs/staging/cloudnative-pg/b2-credentials-secret.yaml
  • Production: infrastructure/configs/production/cloudnative-pg/b2-credentials-secret.yaml

Contents:

  • ACCESS_KEY_ID: Backblaze B2 application key ID
  • ACCESS_SECRET_KEY: Backblaze B2 application key

Purpose:

  • Automated PostgreSQL backups to Backblaze B2
  • WAL archiving for point-in-time recovery
  • Off-cluster disaster recovery

Backup Details:

  • Bucket: homelab-postgres-backups
  • Paths: staging/ and production/ subdirectories
  • Retention: 14 days (staging), 7 days (production)
  • Schedule: Daily automated backups per cluster (ScheduledBackup resources for both postgres-cluster and immich-cluster)

Security Notes:

  • Credentials encrypted with SOPS
  • Application key scoped to backup bucket only
  • Read and write access required for backup/restore operations
  • Store key ID and secret separately from this repository

nightscout/nightscout-env

Nightscout application environment variables.

Location:

  • Staging: apps/staging/nightscout/nightscout-env-secret.yaml
  • Production: apps/production/nightscout/nightscout-env-secret.yaml

Contents:

  • API_SECRET: Nightscout API authentication secret

Purpose:

  • Authentication for Nightscout web interface
  • API access control for uploading CGM data
  • Required for all data modifications

nightscout/ferretdb-postgres-secret

PostgreSQL database credentials for FerretDB (MongoDB API compatibility layer).

Location:

  • Staging: apps/staging/nightscout/ferretdb-postgres-secret.yaml
  • Production: apps/production/nightscout/ferretdb-postgres-secret.yaml

Contents:

  • FERRETDB_POSTGRESQL_PASSWORD: PostgreSQL password (from postgres-cluster-app secret)

Purpose:

  • FerretDB connects to PostgreSQL using these credentials
  • Provides MongoDB API compatibility for Nightscout
  • Uses same PostgreSQL cluster as other applications

renovate/renovate-container-env

Renovate GitHub token for automated dependency updates.

Location:

  • Staging: infrastructure/controllers/staging/renovate/renovate-container-env.yaml
  • Production: infrastructure/controllers/production/renovate/renovate-container-env.yaml

Contents:

  • RENOVATE_TOKEN: GitHub personal access token with repository write permissions

homepage/homepage-env

Homepage widget credentials.

Location:

  • Staging: apps/staging/homepage/homepage-env-secret.yaml
  • Production: apps/production/homepage/homepage-env-secret.yaml

Contents:

  • HOMEPAGE_VAR_GRAFANA_USER: Grafana admin username
  • HOMEPAGE_VAR_GRAFANA_PASSWORD: Grafana admin password

Auto-Generated Secrets

These secrets are automatically created by controllers and should not be manually managed.

TLS Certificates

Created by cert-manager:

  • *-tls: TLS certificates for ingresses
  • letsencrypt-production: Let's Encrypt ACME account key
  • letsencrypt-staging: Let's Encrypt staging ACME account key

Grafana Credentials

Created by kube-prometheus-stack:

  • kube-prometheus-stack-grafana: Grafana admin username and password

Retrieve Grafana credentials:

# For staging
kubectl --context=staging get secret -n monitoring kube-prometheus-stack-grafana \
  -o jsonpath='{.data.admin-password}' | base64 -d

# For production
kubectl --context=production get secret -n monitoring kube-prometheus-stack-grafana \
  -o jsonpath='{.data.admin-password}' | base64 -d

Prometheus and Alertmanager

Created by kube-prometheus-stack:

  • Configuration secrets for Prometheus
  • Configuration secrets for Alertmanager

Working with SOPS Secrets

Prerequisites: The private age keys must be present in ~/.config/sops/age/keys.txt (see Encryption Keys). SOPS finds them automatically — no environment variables required.

Encrypting a New Secret

  1. Create the secret YAML file:
apiVersion: v1
kind: Secret
metadata:
  name: my-secret
  namespace: my-namespace
type: Opaque
stringData:
  key: my-value  # Use stringData for plain text
  1. Encrypt with SOPS
export AGE_PUBLIC=<public age key>

sops --age=$AGE_PUBLIC --encrypt --encrypted-regex '^(data|stringData)$' --in-place test-secret.yaml

Editing an Encrypted Secret

# Works for both environments — SOPS picks the right key from
# ~/.config/sops/age/keys.txt automatically
sops path/to/secret.yaml

SOPS will:

  1. Decrypt the file using your age key
  2. Open it in your default editor
  3. Re-encrypt it when you save and close

Viewing an Encrypted Secret

# For staging
export SOPS_AGE_KEY_FILE=/workspaces/homelab/staging-age.key
sops --decrypt path/to/secret.yaml

# For production
export SOPS_AGE_KEY_FILE=/workspaces/homelab/production-age.key
sops --decrypt path/to/secret.yaml
  1. Update the sops-age secret in the cluster with the new private key

Security Best Practices

  1. Never commit private keys: Age private keys should never be committed to Git
  2. Use separate keys per environment: Staging and production have separate keys
  3. Store private keys securely: Use Proton Pass or another password manager
  4. Rotate keys periodically: Re-encrypt secrets with new age keys regularly
  5. Limit key access: Only share private keys with trusted team members
  6. Verify encryption: Always check that secrets are encrypted before committing
  7. Use strong tokens: Generate strong passwords and API tokens
  8. Rotate credentials: Periodically rotate API tokens and passwords

Troubleshooting

"failed to decrypt" Error

If Flux reports decryption errors:

  1. Verify the sops-age secret exists:
kubectl --context=production get secret -n flux-system sops-age
  1. Check the secret contains the correct key:
kubectl --context=production get secret -n flux-system sops-age \
  -o jsonpath='{.data.age\.agekey}' | base64 -d
  1. Verify the secret was encrypted with the correct public key:
sops --decrypt path/to/secret.yaml

If decryption fails locally, the secret was encrypted with a different key.

Secret Not Decrypted

If a secret exists but isn't being decrypted by Flux:

  1. Check Kustomization for SOPS configuration:
kubectl --context=production get kustomization -n flux-system infrastructure-controllers -o yaml

Look for:

spec:
  decryption:
    provider: sops
    secretRef:
      name: sops-age
  1. Check Flux logs:
kubectl --context=production logs -n flux-system -l app=kustomize-controller

Wrong Environment Key Used

If you accidentally encrypted a staging secret with the production key (or vice versa):

  1. Decrypt with the correct key (the one it was encrypted with):
export SOPS_AGE_KEY_FILE=/workspaces/homelab/production-age.key  # or staging-age.key
sops --decrypt path/to/secret.yaml > /tmp/secret-decrypted.yaml
  1. Re-encrypt with the correct key (the one it should use):
export SOPS_AGE_KEY_FILE=/workspaces/homelab/staging-age.key  # or production-age.key
sops --encrypt /tmp/secret-decrypted.yaml > path/to/secret.yaml
rm /tmp/secret-decrypted.yaml

A File That No Longer Decrypts

validate.sh decrypts every SOPS file when an age key is present, and fails when one no longer round-trips. Four had rotted unnoticed before that check existed (#497), in two ways — both of them "the file was edited after it was encrypted":

Symptom Cause
MAC mismatch The MAC covers every value in the tree, encrypted or not. Adding an annotation by hand after encryption invalidates it while every value still decrypts.
cipher: message authentication failed A multi-document file whose later documents were encrypted separately and appended, each with its own data key. Decrypting the file as a whole tries one key against another's values.

The cluster keeps working throughout: it holds the decrypted Secret and never reads the file again. What breaks is editing or rotating it.

Recovering one:

  1. Decrypt what is there. sops --decrypt --ignore-mac for a MAC mismatch; for the multi-document case, split on --- and decrypt each document separately, then join the plaintext back together.
  2. Check the recovered values against the live Secret before trusting them:
    kubectl --context=production get secret -n <ns> <name> \
      -o jsonpath='{.data.<key>}' | base64 -d | sha256sum
    Each value's own GCM tag is verified during decryption, so a value that decrypts is authentic — but the comparison also proves the file is the one the cluster is running.
  3. Re-encrypt the whole file with the environment's key, then confirm a plain sops --decrypt succeeds with no flags.

There aren't any published security advisories