Repository navigation
fix(deploy): prune unused tagged images before pull — ENOSPC on EC2 - #206
Merged
Merged
Conversation
Every release is an immutable :<sha> tag, and the post-deploy 'docker image prune -f' removes only dangling images — tagged release images were never cleaned up, one accumulating per deploy until the instance disk filled and today's deploy died mid-pull with 'no space left on device' (run 30553109392; the old container kept serving, so prod stayed up on the previous release). Prune with -a before the pull in both the deploy and rollback scripts: at that point the running last-good container pins its image (the recorded rollback target), so only genuinely unused older releases are removed, and the disk steadies at roughly two release images.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Today's backend deploy (run 30553109392, after #199 merged) failed on the EC2 side with
no space left on devicewhile extracting the new image layer. Prod stayed up — the pull died before the container restart, so the previous release kept serving — but no backend release can land until the disk is cleared.Root cause: releases are immutable
:<sha>tags, and the pipeline's only cleanup isdocker image prune -fafterup -d, which removes dangling images only. Tagged release images are never dangling, so one full image per deploy accumulated since the 2026-07-01 cutover until the disk filled.Fix:
docker image prune -afimmediately before the pull, in both the deploy and rollback scripts. At that point the running last-good container pins its own image (the recorded rollback target), so-aremoves only genuinely unused older releases. Disk usage steadies at ~2 release images, and the prune doubles as self-healing for the current full-disk state — the first deploy after this merges frees the space it needs.Testing
docker image prunesemantics (-f= dangling-only vs-a= unused): per Docker docs.