Skip to content

fix: release.yml promotes a pinned per-commit tag, not floating dev-* tags - #66

Merged
dvystrcil merged 1 commit into
mainfrom
fix/release-promote-race-condition
Aug 2, 2026
Merged

fix: release.yml promotes a pinned per-commit tag, not floating dev-* tags#66
dvystrcil merged 1 commit into
mainfrom
fix/release-promote-race-condition

Conversation

@dvystrcil

@dvystrcil dvystrcil commented Aug 1, 2026

Copy link
Copy Markdown
Owner

User description

What

release.yml's promote job now resolves the release tag's actual commit SHA and promotes sha-<sha>-{amd64,arm64} (new, pushed by docker.yml's build step) instead of the floating dev-amd64/dev-arm64 tags.

Why

dev-amd64/dev-arm64 are overwritten by every push to main. release.yml's promote job read whichever images they pointed to at release-creation time, with no check against the commit actually being released. If a second commit's build job won the race and overwrote those tags before the first commit's release had been promoted, the release silently shipped the wrong commit's code -- every workflow step still reported success.

Evidence this actually happened

Investigating why homelab#720's fix (process-log retention sweep) wasn't running live despite being merged, tagged v0.20.41, and documented in CHANGELOG days ago: the deployed image tagged 0.20.44 was missing _sweep_expired_log_files entirely, even though it's present in every commit since the fix landed, including v0.20.44's own git tag's commit (69ad9de). Two Renovate-bot commits landed 10 minutes apart that day; v0.20.44's promote job started 5 seconds before its own triggering commit's docker.yml build had even finished.

Fix

  • docker.yml: build step also pushes sha-<commit>-{amd64,arm64} per platform (immutable, pinned to the exact commit).
  • release.yml: new first step checks out github.event.release.tag_name, resolves git rev-parse HEAD, and the promote step retags from sha-<that-sha>-{amd64,arm64} instead of the floating tags. No race possible -- the source tag is pinned to the exact commit being released.

Immediate remediation (already done, separate from this PR)

Manually re-triggered docker.yml via workflow_dispatch from current main to get a correct build out (v0.20.45) and get the actual homelab#720 fix live, since no other commits were racing at that moment.

Test plan

  • YAML syntax valid (python3 -c "import yaml; yaml.safe_load(...)" on both files).
  • Ran dvystrcil/yaml-validate-action's fold-swallow checker against both files: clean.
  • Will confirm end-to-end on the next real release this workflow produces.

Fixes dvystrcil/homelab#822


PR Type

Bug fix


Description

  • Pinned immutable commit tags in docker workflow

  • Resolved exact release commit SHA in promote job

  • Updated release promotion to use pinned tags

  • Documented race condition fix in changelog


Diagram Walkthrough

flowchart LR
  docker_build["docker.yml build"] -- "pushes sha-<commit> tags" --> release_promote["release.yml promote"]
  release_promote -- "resolves exact commit SHA" --> final_release["pinned release tags"]
Loading

File Walkthrough

Relevant files
Bug fix
docker.yml
Add pinned commit tags to docker build step                           

.github/workflows/docker.yml

  • Added immutable sha--amd64/arm64 tags alongside floating dev tags
  • Documented race condition mitigation in workflow comments
+12/-1   
release.yml
Resolve and use exact commit SHA for release promotion     

.github/workflows/release.yml

  • Added steps to checkout release tag and resolve exact SHA
  • Updated promote job to retag from pinned commit images
+25/-4   
Documentation
CHANGELOG.md
Document release promotion race condition fix in changelog

CHANGELOG.md

  • Added unreleased fix entry documenting the race condition issue
  • Linked to tracking issue homelab#822 for visibility
+4/-0     

@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🔒 No security concerns identified
⚡ No major issues detected

@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown

PR Code Suggestions ✨

Explore these optional code suggestions:

CategorySuggestion                                                                                                                                    Impact
Possible issue
Add --amend flag to manifest creation

Add the --amend flag to docker manifest create. Without it, the command will fail if
the target tag (e.g., latest or a semver version) already contains an existing
manifest list from a previous workflow run.

.github/workflows/release.yml [80-82]

-docker manifest create "$TAG" \
+docker manifest create --amend "$TAG" \
   ${{ env.INTERNAL_REGISTRY }}/${{ env.PROJECT }}/${{ env.IMAGE_NAME }}:sha-${RELEASE_SHA}-amd64 \
   ${{ env.INTERNAL_REGISTRY }}/${{ env.PROJECT }}/${{ env.IMAGE_NAME }}:sha-${RELEASE_SHA}-arm64
Suggestion importance[1-10]: 5

__

Why: Adding --amend improves CI workflow robustness by preventing failures on re-runs or when updating existing tags, though it does not address the primary race condition fix introduced in this PR.

Low

… tags

release.yml's promote job retagged whichever images dev-amd64/dev-arm64
(floating tags, overwritten by every push) happened to point to at
release-creation time, with no verification against the commit
actually being released. If a second commit's build job overwrote
those tags before the first commit's release had been promoted, the
release silently shipped the wrong commit's code -- every step still
reported success.

Confirmed this happened live: v0.20.44's deployed image was missing
the homelab#720 security fix present in the exact commit its own git
tag points to. Manually re-triggered docker.yml (workflow_dispatch)
to get a correct build out as v0.20.45 while this fix was in flight.

docker.yml's build step now also pushes sha-<commit>-{amd64,arm64}
per platform. release.yml resolves the release tag's actual commit
(checks out the tag, reads git rev-parse HEAD) and promotes from that
pinned tag instead of the floating dev-* ones -- eliminates the race
entirely.

Fixes dvystrcil/homelab#822
@dvystrcil
dvystrcil force-pushed the fix/release-promote-race-condition branch from b918f5a to e0fc9d3 Compare August 1, 2026 23:57
@dvystrcil
dvystrcil merged commit 5179555 into main Aug 2, 2026
3 checks passed
@dvystrcil
dvystrcil deleted the fix/release-promote-race-condition branch August 2, 2026 00:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant