Upgrade lib v0 uncaught exception on scale-up after `pre-upgrade-check`

[paulomach]

Steps to reproduce

juju deploy mysql-k8s --channel 8.0/stable --trust
juju run mysql-k8s/0 pre-upgrade-check
juju scale-application mysql-k8s 3

Expected behavior

App scales

Actual behavior

Uncaught exception on upgrade-relation-changed lib hook for new units only.

Log output

unit-mysql-k8s-2: 01:35:57 ERROR unit.mysql-k8s/2.juju-log upgrade:5: Uncaught exception while in charm code:
Traceback (most recent call last):                                                         
  File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/./src/charm.py", line 888, in <module>
    main(MySQLOperatorCharm)                                                               
  File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/venv/ops/main.py", line 551, in main
    manager.run()                                                                          
  File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/venv/ops/main.py", line 530, in run
    self._emit()                                                                           
  File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/venv/ops/main.py", line 519, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/venv/ops/main.py", line 147, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)                                                    
  File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/venv/ops/framework.py", line 348, in emit
    framework._emit(event)                                                                 
  File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/venv/ops/framework.py", line 860, in _emit
    self._reemit(event_path)                                                               
  File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/venv/ops/framework.py", line 950, in _reemit
    custom_handler(event)                                                                  
  File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 724, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/lib/charms/data_platform_libs/v0/upgrade.py", line 987, in on_upgrade_changed
    top_state = self.peer_relation.data[top_unit].get("state")
  File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/venv/ops/model.py", line 1671, in __getitem__
    return self._data[key]                                                                 
KeyError: <ops.model.Unit mysql-k8s/0>

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/DPE-5287.

This message was autogenerated

[nsakkos]

Hi @paulomach , I got this issue as well on postgresql 14.15 14/stable 553, juju 3.6.12. Have you found any workaround?

[paulomach]

Hi @nsakkos no, it still the case

[nsakkos]

As discussed on the Matrix support channel, this worked to make the unit healthy again, thanks team!

juju show-unit postgresql/ANY-UNIT-NUMBER --endpoint upgrade | grep relation-id

# Use the relation ID outputted in the above command.
juju exec --unit postgresql/leader 'relation-set --app -r RELATION-ID-FROM-ABOVE-COMMAND upgrade-stack=""'

juju resolve postgresql/ERROR-UNIT-NUMBER