Skip to content

[BUG] occasional premature release notification can happen with lots of hosts #635

@sadsfae

Description

@sadsfae

There appears to be an error condition that can occur where a large set of hosts can trigger a premature release notification, Initial in the assignment notification table.

In this case, cloud04 skipped the fail update and went straight to releasing the released notification to tenants.

This may likely be related to #634

-=>>quads --ls-notifications --cloud cloud04
cloud    ticket  fail   success  initial  pre_initial  pre    one_day  three_days  five_days  seven_days
========================================================================================================
cloud04  4660    False  True     True     True         False  False    False       False      False

Yet it's still grinding M&R phase (see wiki screenshot)

See M&R log:

=== Validating Started == @  Sun Oct 5 11:30:00 PM UTC 2025
Validating cloud04
The following hosts are marked for build and will now be rebooted:
e26-h07-000-r640.example.com
Host e26-h07-000-r640.example.com didn't pass the health check. Potential provisioning in process. SKIPPING.
e29-h06-000-r640.example.com
Host e29-h06-000-r640.example.com didn't pass the health check. Potential provisioning in process. SKIPPING.
e29-h07-000-r640.example.com
Host e29-h07-000-r640.example.com didn't pass the health check. Potential provisioning in process. SKIPPING.
e29-h09-000-r640.example.com
Host e29-h09-000-r640.example.com didn't pass the health check. Potential provisioning in process. SKIPPING.
e29-h10-000-r640.example.com
- POLLING: [------------------->] 100% - Host state: Off
e29-h11-000-r640.example.com
- POLLING: [------------------->] 100% - Host state: Off
e29-h17-000-r640.example.com
- POLLING: [------------------->] 100% - Host state: Off
e29-h21-000-r640.example.com
- POLLING: [------------------->] 100% - Host state: On

This release notification went out anyway:

Image

On another similarly sized assignment we see correct flagging of Fail to be True and it did not issue a premature release notification.

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions