Skip to content

boot recovery reverts a release because robotd had not ticked yet #196

Description

@Nixxx19

boot recovery can revert a release the robot is about to be healthy on, because it asks robotd once and takes "still starting" as the answer.

robotd answers "control loop has not completed a cycle yet" from the moment its socket opens until its first tick, as healthy: false, degraded: false. the comment on that line in robotd/src/main.rs says why that is fine: "Starting" is not "started". The gate polls, so it will see the transition. the apply gate does poll, for the configured health timeout at half second intervals, and updater/src/engine.rs makes the same three way decision in recover_on_start "in the same words, because the reasoning is identical". but it asks once, with ROBOT_QUERY_TIMEOUT of two seconds, and anything that is not healthy or degraded falls through to the revert. so at boot a robot that is merely late is a robot that failed.

it is reachable. robotd.service is After=local-fs.target and updaterd.service is After=network-online.target, with nothing ordering one against the other, and robotd loads its policy networks before the loop ticks, which is seconds on the board. on the boot that exhausts a trial, if the question lands in that window, the release goes back to the previous one with "never reported healthy across 2 boots" in the log, on a robot that would have answered healthy a poll later. the comment above that branch says exactly why that outcome is the one to avoid: an apply killed before its gate ran is the usual reason a trial is never confirmed, it does not mean the release is bad, and reverting one that is working replaces the code under whoever is using the robot, having told them nothing.

it reproduces in the engine's own tests with a robot that answers "starting" once and healthy after. on main the second recover_on_start returns RolledBack { attempted: 1.1.0, reverted_to: 1.0.0, reason: "never reported healthy across 2 boots" }. the test is on the branch i am opening against this.

on main at 9f7eaad.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions