Settle the release verify past a process's boot window - #3165
Open
valorengels wants to merge 1 commit into
Open
valorengels wants to merge 1 commit into
valorengels wants to merge 1 commit into
Conversation
A bridge or worker that restarts while /update runs execs tens of seconds before it writes its boot beacon. A verify landing inside that window reads a missing or orphaned beacon (beacon_ts <= process_start_ts) and reports `unknown`, which masks both a genuine `stale` and a clean `matches`. The existing mitigations only cover restarts the update run itself initiates (--skip-bridge, the update-restart-in-progress marker, the worker's --since poll). The bridge restart observed on 2026-09-05 came from outside that set, so there was no skip signal to consult. verify_running_release_settled re-polls any process whose `unknown` is mid-boot -- running, start ts readable, exec'd inside the settle window, beacon absent or orphaned -- until it resolves or the window elapses. Terminal unknowns return immediately, so a broken beacon writer still costs nothing. _classify_process now records process_start_ts even when the beacon is missing, which is what lets the poll tell a booting process from a long-running one. Both knobs are TimeoutSettings fields read at call time (TIMEOUTS__BEACON_SETTLE_TIMEOUT_S / _INTERVAL_S), provisional. Refs #1898
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Diagnosis
The 09:52-09:53 (local, 2026-09-05)
/updaterun emitted:Evidence, from
data/update.txt,logs/bridge.logandlogs/update.log:Connected to Telegramboots, each preceded byReceived SIGTERM, shutting down gracefully). The verify at ~02:53Z landed inside one of those boot windows.data/bridge_boot_shacarried beacon ts02:54:36Z, written about a minute after the verify read it. So at read time the beacon belonged to the previous image:beacon_ts <= process_start_ts→ the orphaned-beacon branch of_classify_process→unknown.Bridge-relevant changes detected — restarting bridgenor a--skip-bridgescoping (hadNEED_BRIDGE_RESTARTbeen true, the summary would have omitted the bridge entirely). Nodata/update-restart-in-progressmarker existed, correctly — no planned restart was in flight from this run.So this is neither "verify read a stale beacon during a planned restart" nor "the beacon writer is broken". The beacon writer is healthy (it wrote at every boot). It is: the verify has no defence against a bridge restarted by an actor other than itself, and every existing mitigation (
--skip-bridge, the restart marker, the worker's--sincepoll) is keyed to a restart the update run initiates.Fix
scripts/update/service.py::verify_running_release_settled— re-poll any process whoseunknownis mid-boot: running, start ts readable, exec'd inside the settle window, beacon absent or orphaned. It resolves to a realmatches/staleverdict or gives up at the window. This is the generalization of the worker's--sincepoll and is restarter-agnostic, which is the property the reported failure needed._classify_processnow recordsprocess_start_tseven when the beacon is missing — that is what distinguishes a booting process from a long-running one with no beacon.boot_sha) return immediately, so a genuinely broken beacon writer still costs zero wait.verify_release.pypassessettle_skip=("bridge",)under--skip-bridge: a verdict that gets discarded is never waited on.TimeoutSettingsfields read at call time, not import time:TIMEOUTS__BEACON_SETTLE_TIMEOUT_S(90s) andTIMEOUTS__BEACON_SETTLE_INTERVAL_S(3s), both marked provisional. 90s = the bridge's measured ~30s exec→beacon gap plus headroom, and it doubles as the cap on how long a never-arriving beacon can stall a run.Tests
Five new cases in
tests/unit/test_update_release_verify.py: mid-boot settles tomatches; an orphaned beacon settles tostale(the masked verdict is surfaced, not swallowed); the poll is bounded by its timeout; a long-running process with no beacon never sleeps;settle_skipnever waits.Out of scope, needs follow-up
The ~111s bridge restart loop itself is a separate defect and is not fixed here — I could not attribute the SIGTERM source. It was not the watchdog (
logs/watchdog.logrecords nothing after 2026-09-04 18:10, and it logs every recovery action), not the update run, and noupdate-release-failedsentinel exists. The loop also destroyed a session worktree: each boot's teardown pass wiped.worktrees/dev-a4e15370, and the auto-preserve hook committed the wipe (ba3ea72e9, 3914 files changed, 902k deletions). This branch is rebuilt clean on top ofmain.Refs #1898