Repository navigation
Safe update: refuse setup-script downgrades and failed image pulls - #76
Merged
Merged
Conversation
The safe-update wrapper deployed whatever candidate script invoked it, so running an older release (installer v1.60 over an installed v1.66) pinned older image channels over a database already migrated by newer code and crash-looped the app at startup until the health check reverted. The wrapper now refuses a candidate whose SCRIPT_VERSION is older than the installed script while the stack is still healthy; override with NPMPLUS_FORCE_DOWNGRADE=true. pin_image ignored docker pull failures and re-pinned the local image's stale RepoDigest, so a transient registry error deployed yesterday's image while the update reported success. A failed pull now fails the pin so the wrapper reverts instead. Safe-update wrapper v7; installer v1.67.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two safe-update robustness fixes, both driven by the 2026-10-03 rig incident:
1. Wrapper refuses to deploy an older setup script (wrapper v7)
The wrapper copied whatever candidate script invoked it into
/opt/npmplus/setup-npmplus.shand ran its--update, with no version comparison. Running an older release over a newer install (field case: installer v1.60v2.15.1-mangyan1.rc.9over an installed v1.66) pins older image channels over a database already migrated by newer code. The app then crash-loops at startup —Startup Error: The migration directory is corrupt, the following files are missing: …— never creates/run/npmplus.sock, nginx serves 403s with a dead upstream, the health gate fails, and the wrapper reverts. The safety net worked, but the failure never had to reach the containers: an older script is a proxy for older image channels and older migration files.The wrapper now extracts
SCRIPT_VERSIONfrom both the candidate and the installed script and refuses the copy while the stack is still healthy:No revert is triggered — nothing has been changed yet.
NPMPLUS_FORCE_DOWNGRADE=truerestores the old behavior for deliberate downgrades. Comparison uses the samesort -Vidiom the script already uses for its self-update check; a candidate without a version marker stays unguarded (unchanged semantics), and a missing/unreadable candidate still fails at the existingcp -a … || revert.Wrapper marker bumped 6 → 7, so existing installs re-install the wrapper on their next update.
2.
pin_imageno longer swallowsdocker pullfailurespin_imagerandocker pull "$channel" >/dev/nullwithout checking the exit status, then fell back to the local image'sRepoDigests. On a transient registry/resolver failure that silently re-pinned yesterday's digest and the update reported success while deploying nothing new; with no local image carrying the channel tag it failed with a confusing digest-resolution error instead. A failed pull now printscould not pull <channel>and returns 1, so the bare call sites underset -eabort--updateand the wrapper reverts honestly.Verification
Local harness (extracts the shipped code from
setup-npmplus.sh, not a retyped copy):bash -non the whole script and on the generated wrapper v7pin_image: failed pull → refuses loudly, no stale fallback; successful pull → pins the fresh digestNPMPLUS_FORCE_DOWNGRADE=truedeploys · newer candidate deploys · versionless candidate unguarded · missing candidate file still hits the existingrevertpathsmoke-test.ymlupdated in lockstep: the wrapper-marker corruption step (6 → 3) now starts from marker 7, and the post-update assertion expects marker 7.