ci: add nightly job to verify published image signatures (#279)#280
Open
fabiocicerchia wants to merge 2 commits into
Open
ci: add nightly job to verify published image signatures (#279)#280fabiocicerchia wants to merge 2 commits into
fabiocicerchia wants to merge 2 commits into
Conversation
Docker Hub tags are the only evidence consumers have that an image is signed, and the CI job status for the signing step turns out not to be trustworthy: on the build that produced the current alpine/amazonlinux images, "Docker Bundle" (the job running sign-manifest.sh) never posted a GitHub status at all for those two distros, while sibling distros in the very same run signed correctly. Nothing previously re-checked a published tag's signature after the fact, so the drift sat undetected for weeks. Add a "Verify Published Signatures" job to the nightly workflow that runs bin/verify-image.sh against :latest and each distro's alias tag, failing loudly if any of them aren't actually signed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix: stop mainline/stable Docker tags colliding on shared major-version aliases generate_tags() gave every alpine build (and, for the major-only OS tags, every distro) tags like "latest", "alpine", "1-alpine", and "1-alpine<os_version>" regardless of whether it was building nginx mainline or stable. Since mainline and stable always share the same major version (e.g. 1.31.x and 1.30.x are both "1"), both builds fought over the same tag names — whichever was built/pushed last silently won, so tags like :latest ended up pointing at content from a different build than the digest cosign actually signed. That's why "no signatures found" was reproducible against fabiocicerchia/nginx-lua:latest and other bare/major-only alpine tags: the signature exists, just against a digest those tags no longer resolve to. Scope the ambiguous major-only tags (bare major, "latest", bare distro, "<major>-<distro>", "<major>-<distro><os_version>") to the mainline build only. Minor/patch-qualified tags are untouched since mainline and stable never share those. * fix: derive BUILD_DATE from commit timestamp instead of wall-clock time Every rebuild of common.py baked in datetime.now() as the image's BUILD_DATE, so rebuilding the exact same commit (a CI rerun of a single job, a retry, anything short of a full clean pipeline run) produced a brand new image digest even though nothing actually changed. That new digest gets pushed under the same tag but isn't signed yet, and any already-signed manifest list / index that referenced the old digest is now orphaned from what the tag actually resolves to -- silently, since the rerun itself reports success. Pin BUILD_DATE to the current commit's timestamp (git log -1 --format=%ct) instead. Rebuilding the same commit now always produces the same build-arg and therefore the same digest, so a rerun is a genuine no-op: the existing push and signature are already correct and nothing can go stale relative to them. --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #279 by adding a detective control, since the underlying flake in the signing pipeline isn't reliably reproducible/preventable from what's visible in CI history (see corrected findings below).
What I found digging further
The scope in #279 is a bit off — it's not alpine-only, and it's not "signatures never persist." Re-checked directly against Rekor + Docker Hub for the commit that produced the current images (
f4b56bd6, 2026-07-12):Docker AMD fedora/Docker ARM fedorafailed outright on that commit — a separate, visible build bug, unrelated to signing. Its old (properly signed) 2026-06-28 image is still live as a result.Docker Bundle <distro>status at all for this commit — no success, no failure, nothing. Compare to the 2026-06-28 commit, where all sixDocker Bundlestatuses posted normally.So the CI green checkmarks people would look at aren't reliable evidence either way here — the job that signs the multi-arch manifest list can apparently complete (or not) without ever reporting a status back to GitHub. That's a CircleCI/GitHub-integration reliability gap I can't fix from this repo, and I don't have access to the actual CircleCI job logs for the un-reported runs to pin down why signing silently didn't happen for alpine/amazonlinux specifically.
The fix
Given a preventive fix isn't cleanly diagnosable right now, this adds a detective one: a
Verify Published Signaturesjob in the nightly workflow (same cron asNightly Update) that runs the existingbin/verify-image.shagainst:latestand each distro's alias tag, and fails the build if any of them aren't actually signed. That turns "silently unsigned for weeks" into "red build within 24h."Test plan
.circleci/config.ymlparses cleanly (yaml.safe_load)cosignv3.0.5 (the version pinned in this config) — correctly PASSes for debian/fedora/ubuntu and FAILs for latest/almalinux/alpine/amazonlinux, matching the current known-bad state