Skip to content

Fix GHCR version probe stalling on first release - #49

Merged
danielbrownjr merged 1 commit into
mainfrom
fix/promotion-head-probe
Sep 24, 2026
Merged

danielbrownjr merged 1 commit into
mainfrom
fix/promotion-head-probe

Conversation

@danielbrownjr

Copy link
Copy Markdown
Collaborator

Summary

Release run 35992778776 failed in Promote immutable image to version with curl exit 18 after about 8 minutes. The GHCR version-tag existence probe in promote-container.yml used:

curl ... --output /dev/null --request HEAD ...

--request HEAD only changes the method string. curl still waits for the body advertised by Content-Length. On a first release, GHCR answers the probe with 404 and a 70-byte MANIFEST_UNKNOWN body length but, correctly for HEAD, sends no body. curl stalls until the connection drops and exits 18. The --retry 3 --retry-all-errors added in #41 just repeats the stall four times. Reproduced locally: --request HEAD waits on "0 out of 70 bytes received", while --head returns 404 immediately with exit 0.

Changes

  • promote-container.yml: --request HEAD -> --head (one line)
  • tests/test_container_promotion_probe.py:
    • Behavioral: runs the workflow's own probe command, extracted verbatim, against a local registry stub. Covers the 404 first-release path and the 200 existing-tag path, including the dumped Docker-Content-Digest header, and checks the bearer token is sent.
    • Control: the pre-fix --request HEAD form must stall on the same 404.
    • Contract: the promotion workflow must not use --request HEAD / -X HEAD.
    • With the bug reintroduced, 4 of the 5 tests fail.
  • CHANGELOG.md: corrects the v0.5.0 bullet that credited ci: harden GHCR promotion curl calls against transfer failures #41's retries with fixing a "token-fetch transfer failure"; the real cause was this probe.

No version change; this still releases as 0.5.0.

Validation

  • 258 Python tests with warnings as errors, 1 expected skip
  • 30 JS tests, compileall, git diff --check
  • derive 0.5.0 -> v0.5.0, prerelease: false

Not in scope

The release sequence can still strand an annotated tag if a later job fails. Reordering alone would instead strand an immutable version image, which is worse. That's tracked separately as a follow-up rather than redesigned here.

The version-tag existence probe used --request HEAD, which changes only
the method string. curl still waited for the Content-Length body of the
registry's 404 reply, which a HEAD response never carries, so every first
release stalled until the connection dropped and failed promotion with
exit 18. Retries only repeated the stall.

Use --head. A behavioral test runs the workflow's own probe command
against a local registry stub for the 404 and 200 paths, with a control
proving the old construct stalls, and a contract test rejects
--request HEAD in the promotion workflow.
@danielbrownjr
danielbrownjr merged commit f030eeb into main Sep 24, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant