Skip to content

fix(ci): deploy with a durable GHCR token so swarm can self-heal - #47

Closed
vreshch wants to merge 1 commit into
masterfrom
feature/ghcr-durable-auth
Closed

fix(ci): deploy with a durable GHCR token so swarm can self-heal#47
vreshch wants to merge 1 commit into
masterfrom
feature/ghcr-durable-auth

Conversation

@vreshch

@vreshch vreshch commented Aug 8, 2026

Copy link
Copy Markdown
Member

Problem

The deploy step logs in with secrets.GITHUB_TOKEN, then runs stack deploy --with-registry-auth, which freezes that credential into every service spec. GITHUB_TOKEN expires when the workflow run ends - so from that moment the stack cannot pull from GHCR.

Deploys still work (each run logs in fresh). What breaks is self-healing: a rebooted host, a rescheduled task, or a pruned image leaves the service down until someone manually re-runs a deploy.

Verified, not assumed

On a host, with the node itself logged in and able to docker pull the image by hand:

  1. docker service scale <svc>=0
  2. docker rmi the local image
  3. docker service scale <svc>=1

Result: four consecutive Rejected tasks, No such image. Swarm uses the spec credential, not the node's config.json. Recovered with a manual pull plus docker service update --with-registry-auth.

Fix

Use GHCR_READ_TOKEN for the login that precedes --with-registry-auth, so the credential baked into the spec is valid for a year rather than minutes.

GHCR_READ_TOKEN is a classic PAT scoped read:packages only - it cannot read source, push, or delete. Set as an org secret on agentage (visibility: all).

The build/push login is unchanged and still uses GITHUB_TOKEN; that one needs write access and is correctly ephemeral.

Risk

Low. Same command shape, different secret. A missing secret would fail the login loudly at deploy time rather than silently later.

The deploy logs in with secrets.GITHUB_TOKEN and then runs
"stack deploy --with-registry-auth", which freezes that credential into every
service spec. GITHUB_TOKEN expires when the run ends, so from that moment
Swarm cannot pull.

Verified on a host: with the node itself logged in and able to pull manually,
deleting a service image and scaling up still produced repeated "No such
image" rejections. Swarm uses the spec credential, not config.json.

GHCR_READ_TOKEN is a classic PAT scoped read:packages only, set as an
agentage org secret. The build/push login keeps GITHUB_TOKEN, which needs
write and is correctly ephemeral.
@vreshch

vreshch commented Aug 9, 2026

Copy link
Copy Markdown
Member Author

Closing: not worth the trade.

All three swarms are single-node, so a rescheduled task lands on the same node where the image is already present. docker image prune -af does not remove images used by running containers, and images survive reboots. The failure needs the image to be genuinely absent - a host rebuild or an emergency disk-full prune - which is rare and already recoverable in ~2 minutes with a docker pull on the host plus docker service update --with-registry-auth.

Against that, this change would bake a durable credential into every service spec on three hosts, add a secret dependency to 8 pipelines, and cost 8 production deploys to land.

The finding stands and is worth remembering: docker login on a node does not make Swarm self-heal - Swarm uses the credential frozen in the service spec at deploy time, not the node's config.json. Verified by deleting a service's image with the node logged in and able to pull manually: four consecutive Rejected / No such image tasks.

The hosts are now logged in to GHCR with a read-only (read:packages) credential, which is what actually helps in a rebuild and needed no pipeline change. If this is ever worth codifying, it belongs in the ansible provisioning role, not in every deploy workflow.

@vreshch vreshch closed this Aug 9, 2026
@vreshch
vreshch deleted the feature/ghcr-durable-auth branch August 9, 2026 00:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant