Skip to content

updater: three correctness fixes in rollback journaling and apply - #198

Open
hadelan wants to merge 5 commits into
pollen-robotics:mainfrom
hadelan:updater-rollback-journal-and-apply-action
Open

hadelan wants to merge 5 commits into
pollen-robotics:mainfrom
hadelan:updater-rollback-journal-and-apply-action

Conversation

@hadelan

@hadelan hadelan commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Three independent fixes in the updater, one commit each, every one with a regression test.

  • Journal the boot-exhaustion revert and the rescue against the version that failed. recover_on_start wrote its entries by hand instead of through record(), and record_rescue journalled the rescue against crumb.to — the version that booted — rather than crumb.from, the one whose failure caused it. A later reader of the journal would pair the rescue with the wrong release.
  • Cut hook output on a char boundary. A bare truncate(MAX) on captured hook output panics when the limit lands mid-character. Now it backs off to the boundary.
  • Roll back and journal a select whose apply action fails after the swap. transition_to propagated a post-swap apply-action failure with ?, leaving the new slot swapped in with the gate unmet — the next start would boot a version whose apply never completed. The failure now joins the same rollback-and-journal path as a failed swap, with a fail_apply_action fault-injection switch covering it in tests.

cargo test --workspace, clippy -D warnings and fmt --check are all green; each commit also builds and passes on its own, so the branch bisects cleanly.

hadelan and others added 5 commits September 2, 2026 14:14
…e version that failed

Both hand-built RolledBack entries put the version landed *on* in `to`:
recover_on_start's boot-counter revert named the release just reverted to,
and record_rescue named the golden the rescue moved to. The documented
invariant is the opposite — a RolledBack entry's `to` is the version that
failed, because Journal::known_bad reads it to keep rollbacks from landing
on a bad release.

The two entries therefore blacklisted the healthy release now running and
never the one that failed: a later rollback would skip the good previous
(escalating to golden or Stuck), and the scheduler's brick-loop guard never
stopped the failed release being retried every check interval.

recover_on_start now writes through Engine::record like every other
outcome, so the entry cannot disagree with journal_outcome again;
record_rescue names crumb.from. The rescue test had pinned the wrong
semantics — fixed, and both it and the boot-exhaustion test now assert on
known_bad directly.

Assisted-by: Kimi:kimi-code
…mid-character

The captured output passes through from_utf8_lossy, which pads invalid
bytes into 3-byte U+FFFD runs, so a chatty hook can easily place a
multi-byte character straddling the 8 KiB cap. String::truncate panics on
that, inside Engine::apply: for a socket-triggered apply the connection
died mid-update with no journal entry; for apply_unattended it killed the
periodic-check task, silently disabling unattended updates until a
restart.

transcript.rs already cuts on a char boundary for the same reason; this
brings the hook path in line. The regression test drives a hook whose
output puts a two-byte é across byte 8192.

Assisted-by: Kimi:kimi-code
…r the swap

transition_to (select / rollback / reset-to-golden) answered a failed
apply action by disarming the trial and returning early: no health gate,
no rollback, no log entry. The board stayed on the unverified release
with no boot-counter protection, and support saw nothing — while apply
rolls back on the identical failure, and the function's own comment
claims every class of outcome is journalled.

A unit refusing to restart reaches this in production, which is what the
new fail_apply_action injection stands in for; the forward direction
only, since the revert's own apply action is already covered by
fail_rollback_apply.

Assisted-by: Kimi:kimi-code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant