`update-status` hook crashes with uncaught GenericCharmRuntimeError when workload container startup is delayed

[borathexplorer]

Bug Description

There is a race condition in the charm's update-status handling that locks the unit in maintenance state ("Pod startup is not complete") if the workload container takes longer than 5 minutes to start.
When juju firesupdate-status hook, _refresh_status method attempts to get the status for admission-webhook-up check. Because the container is not started yet, it raises ModelError.
The _refresh_status method catches this and raises a GenericCharmRuntimeError. However, _on_update_status handler is only catching ErrorWithStatus and therefore the exception goes uncaught and the hook crashes.

To Reproduce

1. Deploy the admission-webhook charm to a k8s cluster.
2. Somehow delay the workload container startup so it takes longer than 5 minutes.
3. Wait for 5 minutes so juju fires update-status hook.
4. Observe the hook crash and when the container finally starts, it remains stuck in maintenance state showing "Pod startup is not complete"

Environment

Juju: 3.6.21
Charm: admission-webhook
Channel / Revision: 1.10/candidate (Revision 587)
Platform: Azure / AKS
Deployment: Terraform charmed-kubeflow-solutions module track/1.11

Relevant Log Output

nit-admission-webhook-0: 15:03:00 ERROR unit.admission-webhook/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/src/charm.py", line 263, in _refresh_status
    check = self._get_check_status()
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/src/charm.py", line 257, in _get_check_status
    return self.container.get_check("admission-webhook-up").status
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/lib/python3.12/site-packages/ops/model.py", line 2756, in get_check
    raise ModelError(f'check {check_name!r} not found')
ops.model.ModelError: check 'admission-webhook-up' not found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/src/charm.py", line 350, in <module>
    main(AdmissionWebhookCharm)
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/lib/python3.12/site-packages/ops/main.py", line 39, in main
    return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 502, in main
    manager.run()
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 486, in run
    self._emit()
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 421, in _emit
    self._emit_charm_event(self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 465, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 351, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 924, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 1030, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/src/charm.py", line 310, in _on_update_status
    self._refresh_status()
  File "/var/lib/juju/agents/unit-admission-webhook-0/charm/src/charm.py", line 265, in _refresh_status
    raise GenericCharmRuntimeError(
<unknown>.GenericCharmRuntimeError: Failed to run health check on workload container

Additional Context

No response

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-8609.

This message was autogenerated