Skip to content

Safe update: refuse setup-script downgrades and failed image pulls - #76

Merged
mangyan1 merged 1 commit into
developfrom
fix/safe-update-downgrade-and-pull-guards
Oct 3, 2026
Merged

mangyan1 merged 1 commit into
developfrom
fix/safe-update-downgrade-and-pull-guards

Conversation

@mangyan1

@mangyan1 mangyan1 commented Oct 3, 2026

Copy link
Copy Markdown
Owner

What

Two safe-update robustness fixes, both driven by the 2026-10-03 rig incident:

1. Wrapper refuses to deploy an older setup script (wrapper v7)

The wrapper copied whatever candidate script invoked it into /opt/npmplus/setup-npmplus.sh and ran its --update, with no version comparison. Running an older release over a newer install (field case: installer v1.60 v2.15.1-mangyan1.rc.9 over an installed v1.66) pins older image channels over a database already migrated by newer code. The app then crash-loops at startup — Startup Error: The migration directory is corrupt, the following files are missing: … — never creates /run/npmplus.sock, nginx serves 403s with a dead upstream, the health gate fails, and the wrapper reverts. The safety net worked, but the failure never had to reach the containers: an older script is a proxy for older image channels and older migration files.

The wrapper now extracts SCRIPT_VERSION from both the candidate and the installed script and refuses the copy while the stack is still healthy:

candidate script v1.60 is older than the installed v1.66 - refusing the downgrade
the running stack is untouched; set NPMPLUS_FORCE_DOWNGRADE=true to override

No revert is triggered — nothing has been changed yet. NPMPLUS_FORCE_DOWNGRADE=true restores the old behavior for deliberate downgrades. Comparison uses the same sort -V idiom the script already uses for its self-update check; a candidate without a version marker stays unguarded (unchanged semantics), and a missing/unreadable candidate still fails at the existing cp -a … || revert.

Wrapper marker bumped 6 → 7, so existing installs re-install the wrapper on their next update.

2. pin_image no longer swallows docker pull failures

pin_image ran docker pull "$channel" >/dev/null without checking the exit status, then fell back to the local image's RepoDigests. On a transient registry/resolver failure that silently re-pinned yesterday's digest and the update reported success while deploying nothing new; with no local image carrying the channel tag it failed with a confusing digest-resolution error instead. A failed pull now prints could not pull <channel> and returns 1, so the bare call sites under set -e abort --update and the wrapper reverts honestly.

Verification

Local harness (extracts the shipped code from setup-npmplus.sh, not a retyped copy):

  • bash -n on the whole script and on the generated wrapper v7
  • pin_image: failed pull → refuses loudly, no stale fallback; successful pull → pins the fresh digest
  • Guard: older candidate refused (exit 1, installed script untouched, correct message) · NPMPLUS_FORCE_DOWNGRADE=true deploys · newer candidate deploys · versionless candidate unguarded · missing candidate file still hits the existing revert path
  • All 10 checks pass

smoke-test.yml updated in lockstep: the wrapper-marker corruption step (6 → 3) now starts from marker 7, and the post-update assertion expects marker 7.

The safe-update wrapper deployed whatever candidate script invoked it, so
running an older release (installer v1.60 over an installed v1.66) pinned
older image channels over a database already migrated by newer code and
crash-looped the app at startup until the health check reverted. The
wrapper now refuses a candidate whose SCRIPT_VERSION is older than the
installed script while the stack is still healthy; override with
NPMPLUS_FORCE_DOWNGRADE=true.

pin_image ignored docker pull failures and re-pinned the local image's
stale RepoDigest, so a transient registry error deployed yesterday's
image while the update reported success. A failed pull now fails the pin
so the wrapper reverts instead.

Safe-update wrapper v7; installer v1.67.
@mangyan1
mangyan1 merged commit d64b9ed into develop Oct 3, 2026
17 checks passed
@mangyan1
mangyan1 deleted the fix/safe-update-downgrade-and-pull-guards branch October 3, 2026 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant