Repository navigation
Safe update: make every aborting path loud (wrapper v8) - #77
Merged
Merged
Conversation
set -e aborts the safe-update wrapper on any unguarded failing command,
but only the update dispatch carries a revert guard: every other phase
died silently, leaving the stack half-maintained with no trace of why
(a 2026-10-03 run ended this way in an OCI exec error and rewrote
nothing). The wrapper now traps those deaths: it logs the failing
command and wrapper line, writes aborted-{ps,logs}.txt diagnostics
beside the last-good snapshot, and ends the phase correctly - no revert
before deployment because the running stack is untouched, revert after
it. The rollback database snapshot retries its container exec three
times first, since an exec can transiently lose a race against a
container being stopped or recreated; a stack whose rollback database
cannot be created is never updated. The interactive --update path
installs the same abort naming so a bare failure names the command and
script line that killed the run.
Safe-update wrapper v8; installer v1.68.
A release script older than the installed one rewrites the safe-update wrapper with its own before the wrapper's candidate guard can run, then pins older image channels over a database already migrated by newer code. The --update path now compares its SCRIPT_VERSION against the installed copy and refuses before any tooling or wrapper refresh runs; NPMPLUS_FORCE_DOWNGRADE=true overrides, and a fresh install without an installed copy proceeds untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacks on #76 (merge that first — the diff below includes its commits until then).
What
Today's field incident (2026-10-03): a safe update died printing only
OCI runtime exec failed … /proc/2724619/ns/ipcand rewrote nothing — no snapshot completed (override.yamluntouched), no pins, no revert, no diagnostics.set -ehad killed the run on some unguarded failing command, and only the update dispatch line carries a|| revertguard — every other phase aborts silently, leaving the stack half-maintained with no trace of why. This PR closes that class, not the spot:1. Wrapper ERR trap (wrapper v8)
trap on_wrapper_error ERRcovers every unguarded failing command between the flock andupdate healthy. On fire it:update ABORTED: <cmd> failed with exit <rc> (wrapper line <n>)),aborted-{ps,logs}.txtdiagnostics beside the last-good snapshot,Because bash exempts
||-lists,if/whileconditions, and pipelines' non-final commands from bothset -eand the ERR trap, this trap cannot change existing control flow — it only converts silent deaths into loud ones. The dispatch stays|| revert; the hub block stays|| true; the health check stays condition-guarded.2. Rollback-database snapshot retries its exec
The snapshot DB backup is the first bare, unsilenced
docker execin the wrapper — and an exec can transiently lose a race against a container being stopped or recreated by another actor (the incident class). It now retries three times (5 s apart) before aborting loudly; a stack whose rollback database cannot be created is never updated. Explicitly not|| revert: reverting mid-snapshot would restore from a partial backup.3. The interactive
--updatepath names its aborts tooThe outer script's
--updatebranch (which also re-runs as the wrapper's child) installs the same ERR trap, so a bare failure always printsupdate ABORTED: <cmd> failed with exit <rc> (script line <n>). As the wrapper's child, its nonzero exit still flows into the wrapper's|| revertandfailed-*.txtdiagnostics.4. The
--updatepath refuses an older script than the installed one#76's candidate guard lives in the wrapper — but a release script older than the installed one replaces that wrapper with its own during its pre-update tooling refresh before the wrapper ever runs, so the guard never fires for exactly the incident that motivated it (release rc.9, installer v1.60, over an installed v1.66). The outer
--updatebranch now compares its ownSCRIPT_VERSIONagainst$DATA_DIR/setup-npmplus.shfirst and refuses before any tooling, wrapper, or image refresh runs:this setup script v1.60 is older than the installed v1.66 - refusing the downgrade, stack untouched,NPMPLUS_FORCE_DOWNGRADE=trueoverrides, and a fresh install (no installed copy) proceeds. The script's existing GitHub self-check is per-channel (SELF_URLis baked into each release, so a release script always looks current on its own tag) and only hard-blocks non-interactive--updateruns — it cannot catch this case.Verification
Local harness (extracts the shipped code from
setup-npmplus.shand drives it with stubs) — 14/14:bash -non the script and the generated wrapper v8; marker assertedx=$(false), thepin_imagecall shape) fire it;||/if/&&-guarded forms and the dispatch's|| revertshape never dosmoke-test.ymlupdated in lockstep (marker corruption now 8→3; post-update assertion expects 8).