Periodic health checks with warm-up and application-supplied probes #59

Summary

Run health checks on a configured interval using data the application exposes.

Problem / intent

The supervisor must judge application health without depending on its internals, and must not judge during warm-up.

Proposed solution

Define the probe data format with at least: last reading, feature and inference times; recent inference outputs; the result of the last queue write. Warm-up lasts the longer of the configured warm-up and the active model's window length; restart it after every boot, application restart, model reload and switch, and only once sampling has started; before that, check only that the process runs. Conditions and thresholds come from configuration; when no model is active, check readings only. Every unexpected process exit since the previous check counts as one failure, read from the service manager's restart count. Network problems never count as unhealthy. Overall health also requires the recorded current versions to match the versions actually running (application and model from the probe, image from the system). Write overall health and unmet conditions to the state store. When the first check after warm-up passes on a freshly switched version, ask the image channel to mark a trial slot good where applicable, then record an applied result.

• Depends on: Define the interfaces for the three update channels #1, Persist per-channel update state atomically and resume after reboot #50
• Can proceed in parallel with: Poll target versions and download in stages with resume and backoff #53–Image channel through RAUC: verify, install, reboot, mark good, roll back #58

Contract

Settings, read from configuration:

Warm-up:

• Restarts after every boot, application restart, model reload, switch to a new version and input-mapping change.
• Lasts the longer of update.warmup_s and the active model's window length.
• Counts only from the moment the clock is synced and sampling has started. Until then only condition 1 below is checked. A clock that cannot sync is a network problem and never unhealthy.
• No condition is judged during the warm-up itself.

Conditions, all of which must hold for the application to be healthy:

1. Process runs: the service manager reports the application running, with no unexpected exit since the previous check. Each unexpected exit since the previous check counts as one failure, read from the service manager's restart count.
2. Readings, features and inference: within the check interval there is a reading, a computed feature and a completed inference. Inference is looked for over the longer of the check interval and the active model's window stride. With no active model, only readings are checked.
3. Plausible outputs: no NaN or infinity, and not update.health_zero_run consecutive outputs that are all zero.
4. Queue writes: no queue write failed within the interval. A write refused because the queue is full is not a failure: it is logged and treated as a network problem. Only a write that itself fails counts.

• While the application has deliberately stopped sampling, conditions 2 and 3 are not checked; conditions 1 and 4 still are.
• Network problems never count as unhealthy: an unreachable server or data that cannot be sent is only logged, and repeated send failures are logged as a network or congestion problem.

Overall health holds only when every channel is consistent and healthy: the recorded current version equals the version actually running (application and model from the probe data, image from the system), and the running combination passes the compatibility check (#56). While a channel is downloading or verifying, its current version is still the one running and is the one judged; from switching onward, the new version is judged.

Probe data, written by the application and read by the supervisor:

• At least: the times of the last reading, feature and inference; the most recent inference outputs; the result of the last queue write; whether the clock is synced.
• Carries format_version. Fields are only ever added and never change meaning; a reader that sees a newer version reads the fields it knows; incompatible changes are not supported.

Results:

• Overall health and the unmet conditions are written to the state store.
• On the first passing check after warm-up on a freshly switched version, the image channel is asked to mark its trial slot good where applicable (Image channel through RAUC: verify, install, reboot, mark good, roll back #58), then applied is recorded.
• Open: how the supervisor learns the heartbeat interval
• Open: how the probe data is passed (file, socket or other), its encoding and its field names
• Open: the names of the unmet conditions written to the state store
• Open: whether a check counts as unhealthy when the application skips inference because a required sensor has failed

Acceptance criteria

• The probe data format, including format_version and its compatibility rule, is documented in the repository
• Unit test: a check interval that is neither the heartbeat interval nor an integer multiple of it stops start-up with a log entry; an equal value and a multiple both start
• Unit test: warm-up lasts the longer of update.warmup_s and the window length
• Unit test: warm-up restarts after each of boot, application restart, model reload, switch and input-mapping change
• Unit test: before the clock is synced and sampling has started, only condition 1 is checked
• Unit test: no condition is judged during warm-up
• Unit tests with a fake probe and a fake service manager: each of conditions 1–4 failing on its own makes the check unhealthy
• Unit test: each unexpected exit in the service manager's restart count counts as one failure
• Unit test: with no active model, only readings are checked
• Unit test: while sampling is deliberately stopped, missing readings, features and inference do not make the check unhealthy
• Unit test: a queue write refused because the queue is full is logged and is not unhealthy; a failed write is unhealthy
• Unit test: an unreachable server and unsent data are not unhealthy
• Unit test: overall health is false when a recorded current version differs from the running version, or when the running combination fails the compatibility check
• Unit test: probe data with a newer format_version and unknown fields is read using the fields the reader knows
• Unit test: changing update.health_zero_run in configuration changes the outcome; no threshold is built in
• Unit test: on the first passing check after warm-up on a switched version, mark-good is requested before applied is recorded
• Each Open item above is defined in the pull request and documented in the repository
• No deployment-specific content (organisation, site, equipment model, point table, register map or data sample) in code, tests or docs
• make verify passes

Pending design decisions

Not settled yet. They are not part of the acceptance criteria until decided.

• Further probe fields: whether a model is active and its window length and stride, whether any queue write failed between reads, sampling start, whether sampling is deliberately stopped, the last input-mapping change time, the versions actually running, the current zero-run count

Area

• area:health — Health checks, watchdog and failure recovery