There appears to be an error condition that can occur where a large set of hosts can trigger a premature release notification, Initial in the assignment notification table.
In this case, cloud04 skipped the fail update and went straight to releasing the released notification to tenants.
This may likely be related to #634
-=>>quads --ls-notifications --cloud cloud04
cloud ticket fail success initial pre_initial pre one_day three_days five_days seven_days
========================================================================================================
cloud04 4660 False True True True False False False False False
Yet it's still grinding M&R phase (see wiki screenshot)
See M&R log:
=== Validating Started == @ Sun Oct 5 11:30:00 PM UTC 2025
Validating cloud04
The following hosts are marked for build and will now be rebooted:
e26-h07-000-r640.example.com
Host e26-h07-000-r640.example.com didn't pass the health check. Potential provisioning in process. SKIPPING.
e29-h06-000-r640.example.com
Host e29-h06-000-r640.example.com didn't pass the health check. Potential provisioning in process. SKIPPING.
e29-h07-000-r640.example.com
Host e29-h07-000-r640.example.com didn't pass the health check. Potential provisioning in process. SKIPPING.
e29-h09-000-r640.example.com
Host e29-h09-000-r640.example.com didn't pass the health check. Potential provisioning in process. SKIPPING.
e29-h10-000-r640.example.com
- POLLING: [------------------->] 100% - Host state: Off
e29-h11-000-r640.example.com
- POLLING: [------------------->] 100% - Host state: Off
e29-h17-000-r640.example.com
- POLLING: [------------------->] 100% - Host state: Off
e29-h21-000-r640.example.com
- POLLING: [------------------->] 100% - Host state: On
This release notification went out anyway:
On another similarly sized assignment we see correct flagging of Fail to be True and it did not issue a premature release notification.
There appears to be an error condition that can occur where a large set of hosts can trigger a premature release notification,
Initialin the assignment notification table.In this case,
cloud04skipped thefailupdate and went straight to releasing the released notification to tenants.This may likely be related to #634
Yet it's still grinding M&R phase (see wiki screenshot)
See M&R log:
This release notification went out anyway:
On another similarly sized assignment we see correct flagging of
Failto beTrueand it did not issue a premature release notification.