Cannot upgrade from `1.9/stable` to `1.9/candidate`

[dariofaccin]

Bug Description

Issue coming from labels change in #295.
Upgrade from rev704 (current 1.9/stable) to rev732 (current 1.9/candidate) fails.
Current stable revision adds the following labels:

labels:
  app.kubernetes.io/name: training-operator-manager
  control-plane: kubeflow-training-operator

while current candidate only has control-plane one. In an already existing deployment, editing labels is not allowed¹.

To Reproduce

1. Deploy training-operator from 1.9/stable
2. Upgrade to 1.9/candidate

Environment

.

Relevant Log Output

2026-09-01T07:43:56.807Z [container-agent] 2026-09-01 07:43:56 INFO juju-log HTTP Request: PATCH https://10.152.183.1/apis/apps/v1/namespaces/kubeflow/deployments/training-operator?force=true&fieldManager=lightkube "HTTP/1.1 422 Unprocessable Entity"
2026-09-01T07:43:56.825Z [container-agent] 2026-09-01 07:43:56 ERROR juju-log Uncaught exception while in charm code:
2026-09-01T07:43:56.825Z [container-agent] Traceback (most recent call last):
2026-09-01T07:43:56.825Z [container-agent]   File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/lightkube/core/generic_client.py", line 235, in raise_for_status
2026-09-01T07:43:56.825Z [container-agent]     resp.raise_for_status()
2026-09-01T07:43:56.825Z [container-agent]   File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/httpx/_models.py", line 763, in raise_for_status
2026-09-01T07:43:56.825Z [container-agent]     raise HTTPStatusError(message, request=request, response=self)
2026-09-01T07:43:56.825Z [container-agent] httpx.HTTPStatusError: Client error '422 Unprocessable Entity' for url 'https://10.152.183.1/apis/apps/v1/namespaces/kubeflow/deployments/training-operator?force=true&fieldManager=lightkube'
2026-09-01T07:43:56.825Z [container-agent] For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/422
2026-09-01T07:43:56.825Z [container-agent] 
2026-09-01T07:43:56.825Z [container-agent] During handling of the above exception, another exception occurred:
2026-09-01T07:43:56.825Z [container-agent] 
2026-09-01T07:43:56.825Z [container-agent] Traceback (most recent call last):
2026-09-01T07:43:56.825Z [container-agent]   File "/var/lib/juju/agents/unit-training-operator-0/charm/src/charm.py", line 252, in _apply_k8s_resources
2026-09-01T07:43:56.825Z [container-agent]     self.k8s_resource_handler.apply()
2026-09-01T07:43:56.825Z [container-agent]   File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/kubernetes/_kubernetes_resource_handler.py", line 363, in apply
2026-09-01T07:43:56.825Z [container-agent]     raise e
2026-09-01T07:43:56.825Z [container-agent]   File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/kubernetes/_kubernetes_resource_handler.py", line 336, in apply
2026-09-01T07:43:56.825Z [container-agent]     apply_many(
2026-09-01T07:43:56.825Z [container-agent]   File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/lightkube/batch/_many.py", line 76, in apply_many
2026-09-01T07:43:56.825Z [container-agent]     returns[i] = client.apply(
2026-09-01T07:43:56.825Z [container-agent]                  ^^^^^^^^^^^^^
2026-09-01T07:43:56.825Z [container-agent]   File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/lightkube/core/client.py", line 757, in apply
2026-09-01T07:43:56.825Z [container-agent]     return self.patch(
2026-09-01T07:43:56.825Z [container-agent]            ^^^^^^^^^^^
2026-09-01T07:43:56.825Z [container-agent]   File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/lightkube/core/client.py", line 492, in patch
2026-09-01T07:43:56.825Z [container-agent]     return self._client.request(
2026-09-01T07:43:56.825Z [container-agent]            ^^^^^^^^^^^^^^^^^^^^^
2026-09-01T07:43:56.825Z [container-agent]   File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/lightkube/core/generic_client.py", line 311, in request
2026-09-01T07:43:56.825Z [container-agent]     return self.handle_response(method, resp, br)
2026-09-01T07:43:56.825Z [container-agent]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-09-01T07:43:56.825Z [container-agent]   File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/lightkube/core/generic_client.py", line 251, in handle_response
2026-09-01T07:43:56.825Z [container-agent]     self.raise_for_status(resp)
2026-09-01T07:43:56.825Z [container-agent]   File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/lightkube/core/generic_client.py", line 237, in raise_for_status
2026-09-01T07:43:56.825Z [container-agent]     raise transform_exception(e)
2026-09-01T07:43:56.825Z [container-agent] lightkube.core.exceptions.ApiError: Deployment.apps "training-operator" is invalid: spec.selector: Invalid value: {"matchLabels":{"control-plane":"kubeflow-training-operator"}}: field is immutable
2026-09-01T07:43:56.825Z [container-agent] 
2026-09-01T07:43:56.825Z [container-agent] The above exception was the direct cause of the following exception:
2026-09-01T07:43:56.825Z [container-agent] 
2026-09-01T07:43:56.825Z [container-agent] Traceback (most recent call last):
2026-09-01T07:43:56.825Z [container-agent]   File "/var/lib/juju/agents/unit-training-operator-0/charm/src/charm.py", line 320, in <module>
2026-09-01T07:43:56.825Z [container-agent]     main(TrainingOperatorCharm)
2026-09-01T07:43:56.825Z [container-agent]   File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/ops/__init__.py", line 356, in __call__
2026-09-01T07:43:56.825Z [container-agent]     return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
2026-09-01T07:43:56.825Z [container-agent]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-09-01T07:43:56.825Z [container-agent]   File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 502, in main
2026-09-01T07:43:56.825Z [container-agent]     manager.run()
2026-09-01T07:43:56.825Z [container-agent]   File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 486, in run
2026-09-01T07:43:56.825Z [container-agent]     self._emit()
2026-09-01T07:43:56.825Z [container-agent]   File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 421, in _emit
2026-09-01T07:43:56.825Z [container-agent]     self._emit_charm_event(self.dispatcher.event_name)
2026-09-01T07:43:56.825Z [container-agent]   File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 465, in _emit_charm_event
2026-09-01T07:43:56.825Z [container-agent]     event_to_emit.emit(*args, **kwargs)
2026-09-01T07:43:56.825Z [container-agent]   File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 351, in emit
2026-09-01T07:43:56.825Z [container-agent]     framework._emit(event)
2026-09-01T07:43:56.825Z [container-agent]   File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 924, in _emit
2026-09-01T07:43:56.825Z [container-agent]     self._reemit(event_path)
2026-09-01T07:43:56.825Z [container-agent]   File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 1030, in _reemit
2026-09-01T07:43:56.825Z [container-agent]     custom_handler(event)
2026-09-01T07:43:56.825Z [container-agent]   File "/var/lib/juju/agents/unit-training-operator-0/charm/src/charm.py", line 300, in _on_upgrade
2026-09-01T07:43:56.825Z [container-agent]     self._on_event(_, force_conflicts=True)
2026-09-01T07:43:56.825Z [container-agent]   File "/var/lib/juju/agents/unit-training-operator-0/charm/src/charm.py", line 278, in _on_event
2026-09-01T07:43:56.825Z [container-agent]     self._apply_k8s_resources(force_conflicts=force_conflicts)
2026-09-01T07:43:56.825Z [container-agent]   File "/var/lib/juju/agents/unit-training-operator-0/charm/src/charm.py", line 261, in _apply_k8s_resources
2026-09-01T07:43:56.825Z [container-agent]     raise GenericCharmRuntimeError("K8S resources creation failed") from error
2026-09-01T07:43:56.825Z [container-agent] <unknown>.GenericCharmRuntimeError: K8S resources creation failed
2026-09-01T07:43:57.075Z [container-agent] 2026-09-01 07:43:57 ERROR juju.worker.uniter.operation runhook.go:180 hook "upgrade-charm" (via hook dispatching script: dispatch) failed: exit status 1

Additional Context

No response

Footnotes

1. https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#label-selector-updates ↩

[dariofaccin]

The issue can be mitigated by removing the old Deployment after the upgrade:

kubectl delete deployment -n kubeflow training-operator