Repository navigation
TLS certificate for apt.fullstaqruby.org expired on Oct 12, 2026 #199
Description
Describe the bug
The Let's Encrypt certificate served by apt.fullstaqruby.org expired on Oct 12, 2026 at 02:22 UTC, so apt update fails to fetch the repository index.
To Reproduce
Linux distribution: Ubuntu 24.04
Run apt update with the Fullstaq Ruby repository configured:
Err:3 https://apt.fullstaqruby.org ubuntu-24.04 InRelease
Certificate verification failed: The certificate is NOT trusted. The certificate chain uses expired certificate. Could not handshake: Error in the certificate verification.
W: Failed to fetch https://apt.fullstaqruby.org/dists/ubuntu-24.04/InRelease Certificate verification failed: The certificate is NOT trusted. The certificate chain uses expired certificate. Could not handshake: Error in the certificate verification.
Certificate details:
$ openssl s_client -connect apt.fullstaqruby.org:443 -servername apt.fullstaqruby.org </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer -dates
subject=CN = apt.fullstaqruby.org
issuer=C = US, O = Let's Encrypt, CN = YE1
notBefore=Jul 14 02:22:07 2026 GMT
notAfter=Oct 12 02:22:06 2026 GMT
Expected behavior
apt update succeeds with a valid certificate. It looks like the automatic Let's Encrypt renewal didn't run or failed.
Additional context
System clock is NTP-synchronized and the CA store is up to date, so this appears to be server-side.
Reacted by Matthew Butch and Brendan Dudley
Reacted by Carlos Kvasir
Activity
trumbaut commentedon Oct 12, 2026
Similar issue for https://yum.fullstaqruby.org, used by RPM based distributions:
abtreece commentedon Oct 12, 2026
Status update: root cause found, fix pending on someone with the right Azure access.
Root cause. Caddy on the repo server obtains these certificates via ACME DNS-01 against Azure DNS, authenticating with an Entra ID service principal secret. That secret was created 2024-07-26 with Entra's default 2-year lifetime and expired on 2026-07-26. Caddy's journal shows every renewal attempt since 2026-09-11 failing with:
The last successful renewals were 2026-07-13 (yum) and 2026-07-14 (apt), so the certs ran out their full 90 days and expired on Oct 11 (yum) and Oct 12 (apt). Caddy itself is up and still serving the repo redirects; only certificate renewal is broken. Nothing is wrong on the client side.
Fix. fullstaq-ruby/infra#64 replaces the expired secret and adds yearly rotation so this cannot recur silently. The Terraform plan is reviewed and CI passes, but applying it requires owner rights on the Entra app that only one maintainer currently holds. I've asked them to apply it. Once applied, the server needs one Ansible run and Caddy will renew both certs within about 10 minutes.
I'll update here when the certs are renewed. Separately, there's no cert-expiry alerting on this host, which is why a month of failed renewals went unnoticed; that will get its own issue.
trumbaut commentedon Oct 14, 2026
@abtreece , certificates still not renewed?
jperville commentedon Oct 14, 2026
As a short term workaround, we can use https://storage.googleapis.com/fsruby-server-edition-apt-repo/versions/50/public in the source.list file, e.g.:
FooBarWidget commentedon Oct 15, 2026
Certificates have been renewed now.
fix(terraform): rotate Caddy service principal secret yearly (#64)