Recover first, roll back only a trial channel after consecutive failures #60

Summary

On unhealthy checks, recover in place and roll back only after repeated failures, and only a channel in trial.

Problem / intent

Rolling back on a single bad check, or rolling back a version that has already been confirmed, turns transient faults into churn.

Proposed solution

On an unhealthy check, reload the model when outputs are implausible, otherwise restart the application, then restart warm-up. After a configured number of consecutive failures, roll back the most recently switched channel still in trial. When every channel is safe, keep recovering and report without rolling back.

• Depends on: Periodic health checks with warm-up and application-supplied probes #59, Atomic directory switch and rollback for directory-based channels #57
• Can proceed in parallel with: Wait for the application's upload backlog before downloading #61, Rescue CLI: show channel status and force a rollback #62

Contract

Setting, read from configuration: update.rollback_after_failures, the number of consecutive failures before an automatic rollback.

Steps on an unhealthy check:

1. Recover: when the inference outputs are implausible (condition 3 of Periodic health checks with warm-up and application-supplied probes #59), reload the model; in every other case restart the application. Warm-up restarts after the recovery.
2. Roll back after consecutive failures: roll back only after update.rollback_after_failures consecutive failures. Every unhealthy check and every unexpected exit counts as one failure; one healthy check resets the count to zero.
3. Report: log the rollback and record the result rolled_back_health in the state store.

Which channel is rolled back:

• Only a channel in trial. When every channel runs its safe version, consecutive failures only repeat step 1 and are reported; nothing is rolled back.
• With two channels in trial, the most recently switched one is rolled back first; if consecutive failures continue, the other one is rolled back.
• A rolled-back channel returns to its most recent safe version: a directory-based channel through Atomic directory switch and rollback for directory-based channels #57, the image channel by booting the original slot (Image channel through RAUC: verify, install, reboot, mark good, roll back #58).

While the application is deliberately stopped because no retained version is compatible (#57), step 1 is not done: the application is neither restarted nor asked to reload its model until a compatible version has been switched in.

• Open: what is written to the state store while recovery repeats with every channel safe

Acceptance criteria

• Unit test: implausible outputs lead to a model reload; any other unhealthy check leads to an application restart; warm-up restarts after each
• Unit test: no rollback before update.rollback_after_failures consecutive failures, and a rollback when the count is reached
• Unit test: one healthy check resets the failure count to zero
• Unit test: an unexpected exit counts as one failure
• Unit test: only the channel in trial is rolled back, and rolled_back_health is recorded
• Unit test: with every channel safe, repeated failures cause recovery and a report but no rollback
• Unit test: with two channels in trial, the most recently switched one is rolled back first, then the other if failures continue
• Unit test: while the application is stopped because no retained version is compatible, no restart or reload is attempted
• Each Open item above is defined in the pull request and documented in the repository
• No deployment-specific content (organisation, site, equipment model, point table, register map or data sample) in code, tests or docs
• make verify passes

Area

• area:health — Health checks, watchdog and failure recovery