When upgrading using "mas upgrade" there is no way to re-launch the command if the environment took more time than expected to complete some task.

[guillemmayolramis1]

MAS CLI version

ibmmas/cli:17.2.0

CLI function used

upgrade

What happened?

Environment:

• OCP 4.16
• SNO
• MAS Manage, IOT and Monitor.

1. I ran mas upgrade
   `
   [ibmmas/cli:17.2.0]mascli$ mas upgrade

IBM Maximo Application Suite Admin CLI v17.2.0
Powered by https://github.com/ibm-mas/ansible-devops/ and https://tekton.dev/

1. Set Target OpenShift Cluster
   Already connected to OCP Cluster:
   https://console-openshift-console.apps.ibm.maximo

Proceed with this cluster? [y/n] y

2. Instance Selection
   Select a MAS instance to upgrade from the list below:

• maxdemo1 v9.0.16

Enter MAS instance ID: maxdemo1

3. License Terms
   To continue with the upgrade, you must accept the license terms:

• https://ibm.biz/MAS91-License
• https://ibm.biz/MAXIT91-License
• https://ibm.biz/MAXESRI91-License
  Do you accept the license terms? [y/n] y

4. Review Settings
   Instance ID ..................... maxdemo1
   Current MAS Channel ............. 9.0.x
   Next MAS Channel ................ 9.1.x
   Skip Pre-Upgrade Checks ......... False

Proceed with these settings? [y/n] y

5. Launch Upgrade
   ✅️ OpenShift Pipelines Operator is installed and ready to use
   ✅️ Namespace is ready (mas-maxdemo1-pipelines)
   ✅️ Latest Tekton definitions are installed (v17.2.0)
   ✅️ PipelineRun for maxdemo1 upgrade submitted

View progress:
https://XXXXXXXXX/k8s/ns/mas-maxdemo1-pipelines/tekton.dev~v1beta1~PipelineRun/maxdemo1-upgrade-251201-0917

[ibmmas/cli:17.2.0]mascli$
`
Pipeline Tasks

1. waitfor/approval. PASS
2. pre-upgrade-check. PASS
3. core/upgrade. PASS
4. core-verify.PASS
5. app-manage-upgrade. FAIL
   due to
   FAILED - RETRYING: [localhost]: Check that all workspace CRs are healthy (manage only) (1 retries left).
6. sync-upgrade. PASS

So, it failed due to the long time that two manage builds took. But finally Manage came up fine on 9.1.5.

The bug is that there is no way to re-lauch the "mas upgrade" again to fix and continue upgrading the rest of the components.

Either if I run "mas upgrade" from CLI or re/run the existing pipeline in OpenShift console, the process attempts to upgrade to the next level. In this case from 9.1.x to 9.2.x-feature. Lucky me having IOT still on 9.0.x did not allow the upgrade proceed.

core-upgrade tasks returned

`
TASK [ibm.mas_devops.suite_upgrade : iot : Check that upgrade is supported from current application] ***
fatal: [localhost]: FAILED! => {
"assertion": "check_app.id in compatibility_matrix[mas_channel]",
"changed": false,
"evaluated_to": false,
"msg": "Unable to upgrade to 9.2.x-feature because ibm-mas-iot is installed and is not compatible with this release."
}

`
I attempted to set the MAS_CHANNEL env in cli container and run "mas upgrade" but still attempts to upgrade to 9.2.x-feature

export MAS_CHANNEL=9.1.x.

So, there MUST be a way to select the MAS CHANNEL for "mas Upgrade" to allow partial upgrades be able to get completed.

Relevant log output

core-upgrade tasks returned

`
TASK [ibm.mas_devops.suite_upgrade : iot : Check that upgrade is supported from current application] ***
fatal: [localhost]: FAILED! => {
    "assertion": "check_app.id in compatibility_matrix[mas_channel]",
    "changed": false,
    "evaluated_to": false,
    "msg": "Unable to upgrade to 9.2.x-feature because ibm-mas-iot is installed and is not compatible with this release."
}

[durera]

Thanks for the details ...

For the team; we should look to support this resume/retry scenario ... in our test envs we should be able to simulate this even without a timeout just by trying to start an upgrade while one is already in progress. We'll need to ensure we gate multiple upgrade pipelines running in parallel too ...

this also highlights another problem that using "auto determine upgrade version" poses ... It would make sense now to force the upgrade pipeline to take a specific target rather than auto select it based on current release found ... this way we can do something like this:

• upgrade to 9.1
• upgrade to 9.1

if we don't make this change on the second upgrade run, it will see that 9.1 is the current version and think the ask is to upgrade to 9.2 from 9.1

We can than make the "upgrade to 9.1 (when already running on 9.1)" scenario be idemopotent in a healthy mas instance, but on a failed upgrade it can resume/fix the previously failed upgrade