Terraform (OpenTofu) provisioner for Kubernetes homelab

External dependencies

Create service tokens

Cloudflare API token permissions:

Account	Workers R2 Storage	Edit
Account	Account Rulesets	Edit
Account	Cloudflare Tunnel	Edit
Account	Zero Trust	Edit
User	API Tokens	Edit
Zone	Config Rules	Edit
Zone	Zone Settings	Edit
Zone	SSL and Certificates	Edit
Zone	DNS	Edit

curl https://api.cloudflare.com/client/v4/user/tokens/permission_groups --header "Authorization: Bearer $CLOUDFLARE_API_TOKEN" | jq

GitHub PAT permissions:

• repo
• workflow

Tailscale auth scopes:

• dns
• policy_file
• oauth_keys
• devices:core
• auth_keys
• services

It should include dns, policy_file, oauth_keys, and scopes needed by the Tailscale operator.

Generate secrets files

[ -f $HOME/.ssh/id_ecdsa ] || ssh-keygen -t ecdsa -f $HOME/.ssh/id_ecdsa -N ""

cat > credentials.env <<EOF
TF_VAR_cloudflare_api_token=$CLOUDFLARE_API_TOKEN
TF_VAR_letsencrypt_username=$LETSENCRYPT_USER
TF_VAR_tailscale_oauth_client_id=$TS_OAUTH_CLIENT_ID
TF_VAR_tailscale_oauth_client_secret=$TS_OAUTH_CLIENT_SECRET
TF_VAR_github_username=$(whoami)
TF_VAR_github_renovate_token=$GITHUB_TOKEN
TF_VAR_github_runner_token=$GITHUB_TOKEN
TF_VAR_github_mcp_token=$GITHUB_MCP_TOKEN
TF_VAR_smtp_host=smtp.gmail.com
TF_VAR_smtp_username=$GMAIL_USER
TF_VAR_smtp_password=$GMAIL_PASSWORD
TF_VAR_scrape_proxy_server=$THORDATA_SERVER
TF_VAR_scrape_proxy_username=$THORDATA_USERNAME
TF_VAR_scrape_proxy_password=$THORDATA_PASSWORD
TF_VAR_ssh_client_key_id=$(whoami)
TF_VAR_ssh_client_public_key_openssh=$(cat $HOME/.ssh/id_ecdsa.pub)
TF_VAR_slack_bot_token=$SLACK_BOT_TOKEN
TF_VAR_slack_app_token=$SLACK_APP_TOKEN
TF_VAR_slack_allowed_users=$SLACK_ALLOWED_USERS
TF_VAR_slack_home_channel=$SLACK_HOME_CHANNEL
TF_VAR_slack_alert_webhook=$SLACK_ALERT_WEBHOOK
TF_VAR_alpaca_api_key=$ALPACA_API_KEY
TF_VAR_alpaca_secret_key=$ALPACA_SECRET_KEY
AWS_ENDPOINT_URL_S3=https://$(curl https://api.cloudflare.com/client/v4/accounts --header "Authorization: Bearer $CLOUDFLARE_API_TOKEN" | jq -r '.result.[0].id').r2.cloudflarestorage.com
AWS_ACCESS_KEY_ID=$(curl https://api.cloudflare.com/client/v4/user/tokens/verify --header "Authorization: Bearer $CLOUDFLARE_API_TOKEN" | jq -r '.result.id')
AWS_SECRET_ACCESS_KEY=$(echo -n $CLOUDFLARE_API_TOKEN | sha256sum --quiet)
EOF

Run Terraform in container (optional):

tofu() {
  set -x
  podman run -it --rm --security-opt label=disable \
    -v $(pwd):$(pwd) \
    -w $(pwd) \
    --env-file=credentials.env \
    --net=host \
    ghcr.io/opentofu/opentofu:latest "$@"
  rc=$?; set +x; return $rc
}

Run external configuration

Configure external services:

tofu -chdir=cloud_resources init -upgrade && \
tofu -chdir=cloud_resources apply

Internal resources

Build OS images

See fedora-coreos-config-custom

If the internal cluster is up, call image-build workflow in the repo above to generate a new image. Run renovate workflow in this repo to update to using the new image.

Create host configuration and secrets

Create host ignition and secrets:

tofu -chdir=host_provisioning init -upgrade && \
tofu -chdir=host_provisioning apply

Deploy services to Kubernetes

Bootstrap low level Kubernetes services needed for FluxCD including MinIO for bucket ops.

tofu -chdir=cluster_bootstrap init -upgrade && \
tofu -chdir=cluster_bootstrap apply

Create MinIO objects and FluxCD kustomizations. This includes iPXE and ignition for network booting. This may trigger Kured to reboot nodes.

tofu -chdir=bucket_ops init -upgrade && \
tofu -chdir=bucket_ops apply

Service management

Generate local credentials to local state:

tofu -chdir=local_credentials init -upgrade && \
tofu -chdir=local_credentials apply

SSH CA client:

SSH_KEY=$HOME/.ssh/id_ecdsa
tofu -chdir=local_credentials output -raw ssh_user_cert_authorized_key > $SSH_KEY-cert.pub

Admin kubeconfig:

mkdir -p $HOME/.kube
tofu -chdir=local_credentials output -raw kubeconfig > $HOME/.kube/config

Internal S3:

mkdir -p $HOME/.mc/certs/CAs
tofu -chdir=host_provisioning output -json internal_ca | jq -r '.cert_pem' > $HOME/.mc/certs/CAs/ca.crt
cat > $HOME/.mc/config.json <<EOF
$(tofu -chdir=cluster_bootstrap output -json minio | jq -r '
  {
    aliases: {
      m: {
        accessKey: .access_key_id,
        api: "S3v4",
        path: "auto",
        secretKey: .secret_access_key,
        url: "https://\(.endpoint)"
      }
    },
    version: "10"
  }
')
EOF

mkdir -p $HOME/.config/rclone
cat > $HOME/.config/rclone/rclone.conf <<EOF
$(tofu -chdir=cluster_bootstrap output -json minio | jq -r '
  "[m]",
  "type = s3",
  "provider = Minio",
  "access_key_id = \(.access_key_id)",
  "secret_access_key = \(.secret_access_key)",
  "region = auto",
  "endpoint = https://\(.endpoint)"
')
override.ca_cert = $HOME/.mc/certs/CAs/ca.crt

$(tofu -chdir=cloud_resources output -json r2_bucket | jq -r '
  to_entries |
  map(
    [
      "[cf-\(.value.bucket)]",
      "type = s3",
      "provider = Cloudflare",
      "access_key_id = \(.value.access_key_id)",
      "secret_access_key = \(.value.secret_access_key)",
      "region = auto",
      "endpoint = https://\(.value.url)"
    ] | join("\n")
  ) |
  join("\n\n")
')
EOF

LDAP admin:

tofu -chdir=bucket_ops output lldap

llama.cpp OpenAI compatible endpoint:

tofu -chdir=bucket_ops output llama-cpp

Hermes Agent OpenAI compatible endpoint:

tofu -chdir=bucket_ops output hermes-agent

Internal registry:

regctl registry set zot.cluster.internal \
  --tls enabled \
  --cacert "$(tofu -chdir=host_provisioning output -json internal_ca | jq -r '.cert_pem')" \
  --client-cert "$(tofu -chdir=local_credentials output -json zot_client | jq -r '.cert_pem')" \
  --client-key "$(tofu -chdir=local_credentials output -json zot_client | jq -r '.private_key_pem')"

regctl repo ls zot.cluster.internal
regctl tag ls zot.cluster.internal/${REPO}
regctl tag delete zot.cluster.internal/${REPO}:${TAG}

Notes

Renovate local test

GITHUB_TOKEN=<token>

podman run -it --rm \
  -v $(pwd):$(pwd) \
  -w $(pwd) \
  -e RENOVATE_TOKEN=$GITHUB_TOKEN \
  -e GITHUB_COM_TOKEN=$GITHUB_TOKEN \
  -e LOG_LEVEL=debug \
  ghcr.io/renovatebot/renovate \
  bash

renovate --platform=local --dry-run