pebble-ready hook fails on microk8s + ubuntu@26.04 with SSL key usage error

[gfouillet]

Bug Description

Deploying prometheus-k8s from 3.11/stable (rev 309) on microk8s with baseubuntu@26.04/stable fails to reach active. Units stay in maintenance and the charm's prometheus-pebble-ready hook (and every subsequent config-changed)
crash with:

 httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED]
      certificate verify failed: CA cert does not include key usage extension (_ssl.c:1081)

The failure originates inside the charm's Python 3.14 venv at:

 charm.py:785 _on_pebble_ready
      -> charm.py:677 _configure (self.resources_patch.is_ready())
        -> charms/observability_libs/v0/kubernetes_compute_resources_patch.py:715 is_ready -> :534 is_patched -> get_templated
            -> :401 self.client.get(StatefulSet, ...) (lightkube call to k8s API server)
              -> httpx -> httpcore -> _ssl.c:1081

Reproduces on Juju 3.6.27 and 4.0.13, with a single unit.
The error is identical in both juju versions, which is consistent with the
root cause being the Python 3.14 runtime brought in by the ubuntu@26.04 base,
not juju itself. CPython 3.14 enforces a keyUsage extension on every CA in the
trust chain; the microk8s self-signed cluster CA does not include one.

No workaround is currently available: the only base published for 3.11 is ubuntu@26.04.

To Reproduce

1. Bootstrap microk8s with juju (tested with juju 3.6.27 and 4.0.13):
   microk8s version # please paste output in the report
   juju version # please paste output juju bootstrap microk8s myctrl
   juju add-model m
2. Deploy prometheus-k8s on the 26.04 base (default 1 unit is sufficient):
   juju deploy prom prom --base ubuntu@26.04/stable
3. Watch the log:
   juju debug-log -m m --replay
4. Observe the unit stuck in maintenance with:
   pebble poll failed for container "prometheus":
   failed to send pebble-ready event: hook failed
   and the httpx.ConnectError: ... CA cert does not include key usage extension
   traceback on every retry.

Environment

• Platform: microk8s
• Track/Channel: prometheus-k8s 3.11/stable, revision 309
• Base: ubuntu@26.04/stable
• Juju versions: 3.6.27 and 4.0.13 (both reproduce)
• Charm venv: Python 3.14 (cpython from the 26.04 base)
• Cloud CA: microk8s self-signed cluster CA, missing the keyUsage extension
  on the CA certificate```

Relevant log output

unit-prom-0: 11:42:28 WARNING unit.prom/0.juju-log No Loki endpoints available
unit-prom-0: 11:42:28 ERROR unit.prom/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
    yield
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/httpx/_transports/default.py", line 250, in handle_request
    resp = self.pool.handle_request(req)
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/httpcore/sync/connection_pool.py", line 256, in handle_request
    raise exc from None
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
    response = connection.handle_request(
        pool_request.request
    )
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/httpcore/sync/connection.py", line 101, in handle_request
    raise exc
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/httpcore/sync/connection.py", line 78, in handle_request
    stream = self.connect(request)
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/httpcore/sync/connection.py", line 156, in connect
    stream = stream.start_tls(**kwargs)
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/httpcore/backends/sync.py", line 154, in start_tls
    with map_exceptions(exc_map):
 File "/usr/lib/python3.14/contextlib.py", line 162, in exit__
    self.gen.throw(value)
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/httpcore/exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: SSL: CERTIFICATE_VERIFY_FAILED certificate verify failed: CA cert does not include key usage extension (ssl.c:1081)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-prom-0/charm/src/charm.py", line 1300, in <module>
    main(PrometheusCharm)
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/ops/main.py", line 39, in main
    return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/ops/_main.py", line 507, in main
    manager.run()
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/ops/main.py", line 491, in run
    self.emit()
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/ops/_main.py", line 426, in emit
    self.emit_charm_event(self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/ops/main.py", line 470, in emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/ops/framework.py", line 337, in emit
    framework.emit(event)
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/ops/framework.py", line 910, in emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/ops/framework.py", line 1028, in reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-prom-0/charm/src/charm.py", line 785, in on_pebble_ready
    self._configure(event)
  File "/var/lib/juju/agents/unit-prom-0/charm/src/charm.py", line 677, in configure
    if self.resources_patch.is_ready():
  File "/var/lib/juju/agents/unit-prom-0/charm/lib/charms/observability_libs/v0/kubernetes_compute_resources_patch.py", line 715, in is_ready
    return self.patcher.is_ready(self.pod, resource_reqs)
  File "/var/lib/juju/agents/unit-prom-0/charm/lib/charms/observability_libs/v0/kubernetes_compute_resources_patch.py", line 534, in is_ready
    return self.is_patched(resource_reqs) and equals_canonically(
  File "/var/lib/juju/agents/unit-prom-0/charm/lib/charms/observability_libs/v0/kubernetes_compute_resources_patch.py", line 397, in is_patched
    return equals_canonically(self.get_templated(), resource_reqs)
  File "/var/lib/juju/agents/unit-prom-0/charm/lib/charms/observability_libs/v0/kubernetes_compute_resources_patch.py", line 401, in get_templated
    statefulset = self.client.get(
        StatefulSet, name=self.statefulset_name, namespace=self.namespace
    )
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/lightkube/core/client.py", line 230, in get
    return self.client.request("get", res=res, name=name, namespace=namespace)
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/lightkube/core/generic_client.py", line 382, in request
    resp = self.send(req)
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/lightkube/core/generic_client.py", line 343, in send
    return self.client.send(req, stream=stream)
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/httpx/_client.py", line 914, in send
    response = self.send_handling_auth(
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/httpx/client.py", line 942, in send_handling_auth
    response = self.send_handling_redirects(
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/httpx/_client.py", line 979, in send_handling_redirects
    response = self.send_single_request(request)
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/httpx/client.py", line 1014, in send_single_request
    response = transport.handle_request(request)
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/httpx/transports/default.py", line 249, in handle_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.14/contextlib.py", line 162, in exit_
    self.gen.throw(value)
  File "/var/lib/juju/agents/unit-prom-0/charm/venv/lib/python3.14/site-packages/httpx/transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: SSL: CERTIFICATE_VERIFY_FAILED certificate verify failed: CA cert does not include key usage extension (ssl.c:1081)
unit-prom-0: 11:42:28 ERROR juju.worker.uniter.operation hook "prometheus-pebble-ready" (via hook dispatching script: dispatch) failed: exit status 1
unit-prom-0: 11:42:28 ERROR juju.worker.uniter pebble poll failed for container "prometheus": failed to send pebble-ready event: hook failed

Additional context

• Reproduced on two separate deployments in the same model (prom and prom2),
  single unit each, identical traceback.
• Same charm code on managed k8s clouds or older bases (22.04/24.04,
  Python 3.10/3.12) does not show the failure, because either the cloud-managed
  cluster CA includes the keyUsage extension or the strict check is not
  enabled yet.
• The call path uses charms.observability_libs.v0.kubernetes_compute_resources_patch,
  so other COS k8s charms using the same lib are likely to hit the same failure
  on microk8s + ubuntu@26.04 (e.g. alertmanager-k8s, grafana-k8s, loki-k8s).