While exploring the TLS Certificates library recently (charmlibs.interfaces.tls_certificates), I noticed that reconciler-style usage of the API seems like it could periodically wipe workload certificates during renewal, leading to intermittend TLS outages for workload charms. Agent write-up below (lightly edited).
Summary
During certificate renewal there is a window in which the library's pull APIs (get_assigned_certificate() / get_assigned_certificates()) report no certificates, even though the previously issued certificate is still valid and still present in the provider's relation data. A reconciler-style charm that derives desired workload state from these APIs on every hook will interpret the empty result as "no certificates desired" and delete the workload's certificate files — causing a brief TLS outage at ~90% of every certificate's lifetime.
How the gap arises
Renewal is driven by Juju secret expiry. The certificate secret is created with expire set at renewal_relative_time (default 0.9) of the certificate's validity. When Juju fires secret_expired, _on_secret_expired calls _renew_certificate_request, which — in a single hook — does all of the following:
- Removes the old CSR from the requirer relation databag (
_remove_requirer_csr_from_relation_data).
- Sends a fresh CSR (
_send_certificate_requests).
- Removes all revisions of the old certificate secret (
event.secret.remove_all_revisions()).
From this moment until the provider signs the new CSR and the requirer processes the resulting relation_changed (at least one provider hook round-trip, longer if the provider is busy, rate-limited, or temporarily down):
- The old certificate no longer matches any CSR in the requirer databag, so
_find_certificate_in_relation_data filters it out — the pull APIs return ([], private_key).
- The certificate secret is gone, so there is no local copy either.
- No
certificate_available event fires (it is only emitted for provider certificates matching a current CSR).
- The provider will additionally delete the now-orphaned certificate entry from its own databag on its next reconcile (
_remove_certificates_for_which_no_csr_exists).
The workload is unaffected by this if its charm doesn't rewrite its certificates during this time.
Example
A natural reconciler pattern:
def _reconcile(self, event: ops.EventBase) -> None:
certs, key = self.tls_certificates.get_assigned_certificates()
if not certs:
self._remove_cert_files() # "desired state: no certs"
return
self._push_cert_files(certs, key)observed on update_status, other relation events, config_changed, scaling events, etc. update-status fires periodically, so across a fleet and months of renewals, some unit will reconcile inside the renewal gap. The symptom is a brief, periodic, self-healing TLS outage (cert files disappear, then reappear seconds later when certificate_available fires).
Cause
The pull API conflates three semantically distinct states into one empty
result:
- Pending — CSR outstanding, certificate not yet issued (initial issuance or renewal in flight). Desired action: keep serving current certs.
- Absent — no relation, or no outstanding CSRs. Desired action: tear down.
- Revoked — provider revoked the certificate mid-lifecycle. Desired action: remove promptly.
The information needed to distinguish these exists in the library (get_csrs_from_requirer_relation_data(), get_provider_certificates() with revoked flags, get_request_errors()), but nothing in the API surface guides charm authors toward using it.
Possible remediation options
Agentic analysis suggested some remediation options, but the library maintainers may well have better alternatives.
Library changes
- Option A — keep returning the previous certificate during renewal.
_find_certificate_in_relation_data could also match provider certificates whose CSR was recently withdrawn but whose certificate still matches the current private key and is not expired/revoked. The old cert is still in the provider databag and still cryptographically valid; filtering it out purely because the CSR was withdrawn is what creates the gap. This makes the naive reconciler safe with no charm-side changes, but is a behaviour change whose potential to cause breakages needs to be considered carefully.
- Option B — expose state explicitly. Add a public method or enum, e.g.
get_certificate_state(request) -> PENDING | ASSIGNED | REVOKED | ABSENT, so reconcilers can branch correctly without hand-rolling the CSR-list and revoked-flag checks.
Documentation changes
Document the pending/absent/revoked distinction prominently in the requirer usage docs, with a canonical safe reconciler snippet like:
certs, key = self.tls_certificates.get_assigned_certificates()
if certs:
self._push_cert_files(certs, key)
elif not self.model.get_relation(self._relation_name):
self._remove_cert_files() # only on actual departure
# else: issuance/renewal pending — keep serving current certsBut this is incomplete, because a reconciler guarded as "empty + relation exists = keep current certs" will keep serving a revoked certificate, because revocation is likewise invisible to the pull APIs (the secret is deleted and the provider entry is marked revoked: true, so the cert simply vanishes from get_assigned_certificates()).
Perhaps a how-to would be warranted, or maybe it's better to document good/bad usage patters in the library docstrings so users and agents have a better chance to see them.
While exploring the TLS Certificates library recently (
charmlibs.interfaces.tls_certificates), I noticed that reconciler-style usage of the API seems like it could periodically wipe workload certificates during renewal, leading to intermittend TLS outages for workload charms. Agent write-up below (lightly edited).Summary
During certificate renewal there is a window in which the library's pull APIs (
get_assigned_certificate()/get_assigned_certificates()) report no certificates, even though the previously issued certificate is still valid and still present in the provider's relation data. A reconciler-style charm that derives desired workload state from these APIs on every hook will interpret the empty result as "no certificates desired" and delete the workload's certificate files — causing a brief TLS outage at ~90% of every certificate's lifetime.How the gap arises
Renewal is driven by Juju secret expiry. The certificate secret is created with
expireset atrenewal_relative_time(default 0.9) of the certificate's validity. When Juju firessecret_expired,_on_secret_expiredcalls_renew_certificate_request, which — in a single hook — does all of the following:_remove_requirer_csr_from_relation_data)._send_certificate_requests).event.secret.remove_all_revisions()).From this moment until the provider signs the new CSR and the requirer processes the resulting
relation_changed(at least one provider hook round-trip, longer if the provider is busy, rate-limited, or temporarily down):_find_certificate_in_relation_datafilters it out — the pull APIs return([], private_key).certificate_availableevent fires (it is only emitted for provider certificates matching a current CSR)._remove_certificates_for_which_no_csr_exists).The workload is unaffected by this if its charm doesn't rewrite its certificates during this time.
Example
A natural reconciler pattern:
observed on
update_status, other relation events,config_changed, scaling events, etc.update-statusfires periodically, so across a fleet and months of renewals, some unit will reconcile inside the renewal gap. The symptom is a brief, periodic, self-healing TLS outage (cert files disappear, then reappear seconds later whencertificate_availablefires).Cause
The pull API conflates three semantically distinct states into one empty
result:
The information needed to distinguish these exists in the library (
get_csrs_from_requirer_relation_data(),get_provider_certificates()withrevokedflags,get_request_errors()), but nothing in the API surface guides charm authors toward using it.Possible remediation options
Agentic analysis suggested some remediation options, but the library maintainers may well have better alternatives.
Library changes
_find_certificate_in_relation_datacould also match provider certificates whose CSR was recently withdrawn but whose certificate still matches the current private key and is not expired/revoked. The old cert is still in the provider databag and still cryptographically valid; filtering it out purely because the CSR was withdrawn is what creates the gap. This makes the naive reconciler safe with no charm-side changes, but is a behaviour change whose potential to cause breakages needs to be considered carefully.get_certificate_state(request) -> PENDING | ASSIGNED | REVOKED | ABSENT, so reconcilers can branch correctly without hand-rolling the CSR-list and revoked-flag checks.Documentation changes
Document the pending/absent/revoked distinction prominently in the requirer usage docs, with a canonical safe reconciler snippet like:
But this is incomplete, because a reconciler guarded as "empty + relation exists = keep current certs" will keep serving a revoked certificate, because revocation is likewise invisible to the pull APIs (the secret is deleted and the provider entry is marked
revoked: true, so the cert simply vanishes fromget_assigned_certificates()).Perhaps a how-to would be warranted, or maybe it's better to document good/bad usage patters in the library docstrings so users and agents have a better chance to see them.