jupyter-controller: upgrade-charm hook fails after pod deletion (rev 1146, 1.9/edge)

[canonical-vultaire]

Note: This issue was generated with AI assistance (GitHub Copilot) based on automated log analysis and triage.
Filed by @canonical/solutions-qa

---

Summary

jupyter-controller (rev 1146, 1.9/edge) enters an error state after its pod is deleted and Kubernetes restarts it. When Juju attempts to reconcile the unit post-restart it runs the upgrade-charm hook, which fails, leaving the unit in a permanent error state.

Failing Test

• Test case: test_pod_deletion
• Template: charm_integration_testing.test_suite.test_pod_deletion

Environment

• Charm: jupyter-controller rev 1146, track 1.9/edge
• Juju version: 3.6.23
• Cloud: Kubernetes (CMR: two-model setup)
• Test plans (same failure observed across all three):
  • grafana-agent-k8s:logging-provider / loki_push_api / jupyter-controller:logging (neighbors: grafana-agent-k8s rev 211, loki-k8s rev 217)
  • grafana-agent-k8s:logging-provider / loki_push_api / jupyter-controller:grafana-dashboard (neighbors: grafana-k8s rev 180, grafana-agent-k8s rev 211)
  • jupyter-controller:metrics-endpoint / prometheus_scrape / prometheus-k8s:metrics-endpoint (neighbor: prometheus-k8s rev 287)

Observed Error

After pod deletion, the target application enters error state with:

application:error:hook failed: "upgrade-charm"
unit:error:hook failed: "upgrade-charm"

This causes the test's wait-for-active to time out:

juju.backend.JujuWaitTimeoutError: Timed out while waiting for applications: [active],
units: [active] (applications: ['target'], units: ['target/0'])

Steps to Reproduce

1. Deploy jupyter-controller rev 1146 from 1.9/edge on a Kubernetes cloud with Juju 3.6.23
2. Wait for the unit to reach active/idle
3. Delete the jupyter-controller-0 pod (e.g. kubectl delete pod jupyter-controller-0 -n <model-namespace>)
4. Wait for the pod to be recreated
5. Observe: the unit enters error state with hook failed: "upgrade-charm" rather than returning to active

Test Observer Links

• https://test-observer.canonical.com/#/charms/137037?testExecutionId=540285&testResultId=11349857
• https://test-observer.canonical.com/#/charms/137037?testExecutionId=540289
• https://test-observer.canonical.com/#/charms/137037?testExecutionId=540313