Skip to content

Settle the release verify past a process's boot window - #3165

Open
valorengels wants to merge 1 commit into
mainfrom
session/dev-a4e15370
Open

valorengels wants to merge 1 commit into
mainfrom
session/dev-a4e15370

Conversation

@valorengels

Copy link
Copy Markdown
Collaborator

Diagnosis

The 09:52-09:53 (local, 2026-09-05) /update run emitted:

WARNING: bridge release could not be confirmed (unknown) — not failing the run
release verify OK @ 45d5d42d4 (bridge unknown, worker matches)

Evidence, from data/update.txt, logs/bridge.log and logs/update.log:

  • The bridge was restarting on a ~111s cadence from 02:52:21Z through 03:08:58Z (nine Connected to Telegram boots, each preceded by Received SIGTERM, shutting down gracefully). The verify at ~02:53Z landed inside one of those boot windows.
  • data/bridge_boot_sha carried beacon ts 02:54:36Z, written about a minute after the verify read it. So at read time the beacon belonged to the previous image: beacon_ts <= process_start_ts → the orphaned-beacon branch of _classify_processunknown.
  • The update run did not restart the bridge: the log has neither Bridge-relevant changes detected — restarting bridge nor a --skip-bridge scoping (had NEED_BRIDGE_RESTART been true, the summary would have omitted the bridge entirely). No data/update-restart-in-progress marker existed, correctly — no planned restart was in flight from this run.

So this is neither "verify read a stale beacon during a planned restart" nor "the beacon writer is broken". The beacon writer is healthy (it wrote at every boot). It is: the verify has no defence against a bridge restarted by an actor other than itself, and every existing mitigation (--skip-bridge, the restart marker, the worker's --since poll) is keyed to a restart the update run initiates.

Fix

scripts/update/service.py::verify_running_release_settled — re-poll any process whose unknown is mid-boot: running, start ts readable, exec'd inside the settle window, beacon absent or orphaned. It resolves to a real matches/stale verdict or gives up at the window. This is the generalization of the worker's --since poll and is restarter-agnostic, which is the property the reported failure needed.

  • _classify_process now records process_start_ts even when the beacon is missing — that is what distinguishes a booting process from a long-running one with no beacon.
  • Terminal unknowns (no PID, unreadable start ts, an old process with no beacon, an unresolvable boot_sha) return immediately, so a genuinely broken beacon writer still costs zero wait.
  • verify_release.py passes settle_skip=("bridge",) under --skip-bridge: a verdict that gets discarded is never waited on.
  • Knobs are TimeoutSettings fields read at call time, not import time: TIMEOUTS__BEACON_SETTLE_TIMEOUT_S (90s) and TIMEOUTS__BEACON_SETTLE_INTERVAL_S (3s), both marked provisional. 90s = the bridge's measured ~30s exec→beacon gap plus headroom, and it doubles as the cap on how long a never-arriving beacon can stall a run.

Tests

Five new cases in tests/unit/test_update_release_verify.py: mid-boot settles to matches; an orphaned beacon settles to stale (the masked verdict is surfaced, not swallowed); the poll is bounded by its timeout; a long-running process with no beacon never sleeps; settle_skip never waits.

scripts/pytest-clean.sh tests/unit/test_update_release_verify.py tests/unit/test_settings.py
83 passed

Out of scope, needs follow-up

The ~111s bridge restart loop itself is a separate defect and is not fixed here — I could not attribute the SIGTERM source. It was not the watchdog (logs/watchdog.log records nothing after 2026-09-04 18:10, and it logs every recovery action), not the update run, and no update-release-failed sentinel exists. The loop also destroyed a session worktree: each boot's teardown pass wiped .worktrees/dev-a4e15370, and the auto-preserve hook committed the wipe (ba3ea72e9, 3914 files changed, 902k deletions). This branch is rebuilt clean on top of main.

Refs #1898

A bridge or worker that restarts while /update runs execs tens of seconds
before it writes its boot beacon. A verify landing inside that window reads
a missing or orphaned beacon (beacon_ts <= process_start_ts) and reports
`unknown`, which masks both a genuine `stale` and a clean `matches`.

The existing mitigations only cover restarts the update run itself
initiates (--skip-bridge, the update-restart-in-progress marker, the
worker's --since poll). The bridge restart observed on 2026-09-05 came from
outside that set, so there was no skip signal to consult.

verify_running_release_settled re-polls any process whose `unknown` is
mid-boot -- running, start ts readable, exec'd inside the settle window,
beacon absent or orphaned -- until it resolves or the window elapses.
Terminal unknowns return immediately, so a broken beacon writer still costs
nothing. _classify_process now records process_start_ts even when the
beacon is missing, which is what lets the poll tell a booting process from
a long-running one. Both knobs are TimeoutSettings fields read at call time
(TIMEOUTS__BEACON_SETTLE_TIMEOUT_S / _INTERVAL_S), provisional.

Refs #1898
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant