[abtreece]

Description

Fixes the expired TLS certificates on apt.fullstaqruby.org and yum.fullstaqruby.org (fullstaq-ruby/server-edition#199).

Root cause. The Entra ID client secret that Caddy uses for the ACME DNS-01 challenge expired on 2026-07-26. azuread_service_principal_password.caddy sets no end_date, so Entra applied its default 2-year lifetime (created 2024-07-26). Caddy's journal on the VM shows every renewal attempt since 2026-09-11 failing with:

AADSTS7000222: The provided client secret keys for app 'c9ed2a95-812e-4007-8ae6-c8525bca82aa' are expired.

Last successful renewals were 2026-07-13 (yum) and 2026-07-14 (apt). The certs expired 2026-10-11 19:02 UTC (yum) and 2026-10-12 02:22 UTC (apt). Caddy itself is healthy and still serving redirects; only renewal is broken.

Change. Tie the service principal password to a time_rotating clock with rotation_days = 365 via rotate_when_changed, so terraform apply replaces the secret yearly, well inside the 2-year lifetime. Adds the hashicorp/time provider (~> 0.12.1) and its lock-file hashes for linux_amd64, darwin_arm64 and darwin_amd64.

Caveat / who must apply. I cannot apply this. hongli-infra-maintainer@fullstaqruby.org is the only owner of the Caddy app and service principal, and my account holds no Entra directory role, so both the password replacement and the owner update fail with 403 Authorization_RequestDenied. The plan also adds the current Infra Maintainers members as owners of the app and SP (pre-existing drift), which would let other maintainers handle this next time.

terraform plan output (verified with Terraform 1.5.7 against the prod backend):

Plan: 2 to add, 3 to change, 1 to destroy.
  + time_rotating.caddy_password
 -/+ azuread_service_principal_password.caddy          (replaced; old one is already expired)
  ~ azurerm_key_vault_secret.caddy-client-secret       (value updated in place)
  ~ azuread_application.caddy / azuread_service_principal.caddy  (owners added)

Manual steps after apply. Run the Ansible playbook so /etc/caddy/env picks up the new secret from Key Vault, then let Caddy restart (the playbook's handler does this). A --check --diff run of the playbook against the VM today shows only 4 unrelated, non-destructive changes. Caddy will renew both certs on its next maintenance tick (10 minutes).

Follow-up, not in this PR: there is no certificate-expiry or renewal-failure alerting, which is why this went unnoticed for a month.

Testing instructions

After terraform apply and the Ansible run:

journalctl -u caddy --since "10 min ago" | grep "certificate renewed successfully"
for h in apt yum; do echo | openssl s_client -connect $h.fullstaqruby.org:443 -servername $h.fullstaqruby.org 2>/dev/null | openssl x509 -noout -dates; done
az ad sp credential list --id c9ed2a95-812e-4007-8ae6-c8525bca82aa -o table

Expect two "renewed successfully" lines, notAfter roughly 90 days out, and a credential with a 2-year end date starting today.

Checklist for the submitter

• Documentation updated
• Infrastructure overview diagram updated
• The following content/files are kept in sync with each other:
  • The Terraform version specified in **/providers.tf, .github/workflows/code-reviews.yml and docs/required-devtools.md (unchanged)
  • The list of responsibilities specified in .github/ISSUE_TEMPLATE/apply_join_team.md and docs/responsibilities-expectations.md (unchanged)

Checklist for reviewers

• If terraform-hisec/ was updated: someone with the "Infra Owner" role has checked the output of terraform plan

Definition of done (for team members)

• If terraform-hisec/ was updated: someone with the "Infra Owner" role has run terraform apply
• Tested in production

🤖 Generated with Claude Code

[FooBarWidget]

Thanks for looking into this. I've applied Terraform and you should be unblocked from fixing now.

As for this pull request: the intention is that you (infra maintainers) should be able to change the Caddy password. But it looks like the rights for this weren't set properly. All Enterprise Applications are owned by myself, so only I or some other organization owner can update them.

It looks like the Terraform code for syncing the owners is already there. But the onboarding and offboarding instructions should add an additional checklist: re-apply terraform/ (which syncs the owners), not just terraform-hisec/. You should fix this too.

Anyway, the owners lists themselves are fixed now that I've applied Terraform. You still need to run Ansible yourself.

[FooBarWidget]

I've run Ansible now so the certificate has been renewed.