From bcf243d53cd03f2b51337e7bfcf40af93d962030 Mon Sep 17 00:00:00 2001 From: Anuraj Jit Saikia Date: Thu, 30 Jul 2026 20:41:45 +0530 Subject: [PATCH] =?UTF-8?q?fix(deploy):=20prune=20unused=20tagged=20images?= =?UTF-8?q?=20before=20pull=20=E2=80=94=20ENOSPC=20on=20EC2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every release is an immutable : tag, and the post-deploy 'docker image prune -f' removes only dangling images — tagged release images were never cleaned up, one accumulating per deploy until the instance disk filled and today's deploy died mid-pull with 'no space left on device' (run 30553109392; the old container kept serving, so prod stayed up on the previous release). Prune with -a before the pull in both the deploy and rollback scripts: at that point the running last-good container pins its image (the recorded rollback target), so only genuinely unused older releases are removed, and the disk steadies at roughly two release images. --- .github/workflows/deploy-backend.yml | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/.github/workflows/deploy-backend.yml b/.github/workflows/deploy-backend.yml index 63197c1..2d3d0f4 100644 --- a/.github/workflows/deploy-backend.yml +++ b/.github/workflows/deploy-backend.yml @@ -118,6 +118,14 @@ jobs: git fetch --quiet origin main git checkout --quiet --force --detach "$DEPLOY_SHA" export SERVER_IMAGE + # Free disk BEFORE pulling. Releases are immutable TAGGED : + # images, and dangling-only pruning (`prune -f`) never removes a + # tagged image — so one image per deploy accumulates until the disk + # fills and the pull itself dies with ENOSPC. `-a` removes every + # image not used by a running container; the currently-running + # last-good image (just recorded in .rollback-image) is in use, so + # the rollback target always survives this. + docker image prune -af # Pull the exact published tag and (re)start. `set -e` makes the deploy # fail fast if the pull or container start fails. docker compose -f docker-compose.prod.yml pull @@ -180,6 +188,10 @@ jobs: git fetch --quiet origin main git checkout --quiet --force --detach "$PREV_SHA" export SERVER_IMAGE="$PREV_IMAGE" + # Same ENOSPC guard as the deploy step: drop unused tagged images + # before pulling. The bad-but-running new image stays (in use) and + # gets cleaned up by the next successful deploy's pre-pull prune. + docker image prune -af docker compose -f docker-compose.prod.yml pull docker compose -f docker-compose.prod.yml up -d docker image prune -f