Skip to content

Safe update: make every aborting path loud (wrapper v8) - #77

Merged
mangyan1 merged 3 commits into
developfrom
fix/safe-update-silent-abort
Oct 3, 2026
Merged

mangyan1 merged 3 commits into
developfrom
fix/safe-update-silent-abort

Conversation

@mangyan1

@mangyan1 mangyan1 commented Oct 3, 2026

Copy link
Copy Markdown
Owner

Stacks on #76 (merge that first — the diff below includes its commits until then).

What

Today's field incident (2026-10-03): a safe update died printing only OCI runtime exec failed … /proc/2724619/ns/ipc and rewrote nothing — no snapshot completed (override.yaml untouched), no pins, no revert, no diagnostics. set -e had killed the run on some unguarded failing command, and only the update dispatch line carries a || revert guard — every other phase aborts silently, leaving the stack half-maintained with no trace of why. This PR closes that class, not the spot:

1. Wrapper ERR trap (wrapper v8)

trap on_wrapper_error ERR covers every unguarded failing command between the flock and update healthy. On fire it:

  • logs the failing command, exit code, and wrapper line (update ABORTED: <cmd> failed with exit <rc> (wrapper line <n>)),
  • writes aborted-{ps,logs}.txt diagnostics beside the last-good snapshot,
  • ends the phase correctly: before deployment — no revert, the running stack is untouched (a partial snapshot must never be restored); after deployment — revert.

Because bash exempts ||-lists, if/while conditions, and pipelines' non-final commands from both set -e and the ERR trap, this trap cannot change existing control flow — it only converts silent deaths into loud ones. The dispatch stays || revert; the hub block stays || true; the health check stays condition-guarded.

2. Rollback-database snapshot retries its exec

The snapshot DB backup is the first bare, unsilenced docker exec in the wrapper — and an exec can transiently lose a race against a container being stopped or recreated by another actor (the incident class). It now retries three times (5 s apart) before aborting loudly; a stack whose rollback database cannot be created is never updated. Explicitly not || revert: reverting mid-snapshot would restore from a partial backup.

3. The interactive --update path names its aborts too

The outer script's --update branch (which also re-runs as the wrapper's child) installs the same ERR trap, so a bare failure always prints update ABORTED: <cmd> failed with exit <rc> (script line <n>). As the wrapper's child, its nonzero exit still flows into the wrapper's || revert and failed-*.txt diagnostics.

4. The --update path refuses an older script than the installed one

#76's candidate guard lives in the wrapper — but a release script older than the installed one replaces that wrapper with its own during its pre-update tooling refresh before the wrapper ever runs, so the guard never fires for exactly the incident that motivated it (release rc.9, installer v1.60, over an installed v1.66). The outer --update branch now compares its own SCRIPT_VERSION against $DATA_DIR/setup-npmplus.sh first and refuses before any tooling, wrapper, or image refresh runs: this setup script v1.60 is older than the installed v1.66 - refusing the downgrade, stack untouched, NPMPLUS_FORCE_DOWNGRADE=true overrides, and a fresh install (no installed copy) proceeds. The script's existing GitHub self-check is per-channel (SELF_URL is baked into each release, so a release script always looks current on its own tag) and only hard-blocks non-interactive --update runs — it cannot catch this case.

Verification

Local harness (extracts the shipped code from setup-npmplus.sh and drives it with stubs) — 14/14:

  • bash -n on the script and the generated wrapper v8; marker asserted
  • ERR-trap mechanics: function handler reports the faulting command, exit code, and line; assignment failures (x=$(false), the pin_image call shape) fire it; ||/if/&&-guarded forms and the dispatch's || revert shape never do
  • Real extracted handler: pre-deploy abort writes both diagnostics files and does not revert; post-deploy abort reverts
  • Real extracted retry block: transient exec failure recovers on the 3rd attempt, healthy exec takes one attempt, persistent failure aborts loudly after exactly 3
  • Real extracted outer handler: abort names command and line

smoke-test.yml updated in lockstep (marker corruption now 8→3; post-update assertion expects 8).

set -e aborts the safe-update wrapper on any unguarded failing command,
but only the update dispatch carries a revert guard: every other phase
died silently, leaving the stack half-maintained with no trace of why
(a 2026-10-03 run ended this way in an OCI exec error and rewrote
nothing). The wrapper now traps those deaths: it logs the failing
command and wrapper line, writes aborted-{ps,logs}.txt diagnostics
beside the last-good snapshot, and ends the phase correctly - no revert
before deployment because the running stack is untouched, revert after
it. The rollback database snapshot retries its container exec three
times first, since an exec can transiently lose a race against a
container being stopped or recreated; a stack whose rollback database
cannot be created is never updated. The interactive --update path
installs the same abort naming so a bare failure names the command and
script line that killed the run.

Safe-update wrapper v8; installer v1.68.
A release script older than the installed one rewrites the safe-update
wrapper with its own before the wrapper's candidate guard can run, then
pins older image channels over a database already migrated by newer
code. The --update path now compares its SCRIPT_VERSION against the
installed copy and refuses before any tooling or wrapper refresh runs;
NPMPLUS_FORCE_DOWNGRADE=true overrides, and a fresh install without an
installed copy proceeds untouched.
@mangyan1
mangyan1 merged commit 56243a5 into develop Oct 3, 2026
17 checks passed
@mangyan1
mangyan1 deleted the fix/safe-update-silent-abort branch October 3, 2026 19:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant