alpine images (incl. :latest) are unsigned since 1.31.2/1.30.3 despite CI reporting sign-image success

[fabiocicerchia]

Summary

The alpine-based images — and therefore :latest, which tracks alpine — are not actually signed with cosign, despite the CI jobs that run bin/sign-image.sh / bin/sign-manifest.sh reporting success. Every other distro (almalinux, amazonlinux, debian, fedora, ubuntu) signs correctly for the same versions. This is a regression scoped to alpine specifically, starting with the bump to 1.31.2 (mainline) / 1.30.3 (stable), alpine base 3.24.1.

Evidence (independently reproduced, cosign v3.0.5 — same version pinned in .circleci/config.yml)

fabiocicerchia/nginx-lua:latest (alpine, current mainline):

$ cosign verify --certificate-oidc-issuer "https://oidc.circleci.com/org/139dac99-01a5-4da1-9a56-a8699c0a2c7c" \
    --certificate-identity-regexp "https://circleci\.com/api/v2/projects/1d8730b7-9e11-4e2e-a1e3-3e21afa59503/pipeline-definitions/.*" \
    fabiocicerchia/nginx-lua:latest
Error: no signatures found

• Resolved digest: sha256:98027badef498002394020bcb2735076ceea147d37e9eed75a205b23cc6c448b
• Rekor index/retrieve by this hash: [] (no transparency-log entry — was never actually signed, not just "unattached")
• No .sig tag and no OCI 1.1 referrer for this digest either

Same result for the per-arch alpine image 1.31.2-alpine3.24.1-amd64 (digest sha256:1d9a3cdb...): Rekor returns [].

Control check — same versions, different distros, verify successfully:

• 1.31.2-fedora44 → digest sha256:92716b95... → cosign verify succeeds, Rekor has 3 entries (signature + 2 SBOM attestations), signatures stored via the OCI 1.1 Referrers API (/v2/.../referrers/<digest>), which Docker Hub + cosign v3 use by default now.
• 1.30.3-ubuntu26.04 → digest sha256:ce3ad843... → cosign verify succeeds, 9 Rekor entries.

So the pipeline and signing infra work correctly in general — this is not an OIDC/Fulcio/CircleCI-wide outage, and it is not a "Docker Hub doesn't support signature storage" problem. It's isolated to the alpine build/sign path.

Narrowing to the regression point

Checked historical Rekor coverage across tags: the last alpine build that has a Rekor entry is 1.31.1-alpine / 1.31.0-alpine3.23.3 / 3.23.4. Every alpine tag at 1.31.2/1.30.3 (base 3.24.1) has none. Every other distro kept signing correctly straight through that same version bump.

What this isn't

• Not stale docs/scripts — SUPPLY_CHAIN_SECURITY.md, bin/sign-image.sh, bin/sign-manifest.sh, bin/verify-image.sh all match what was verified (same OIDC issuer, same identity regexp).
• Not a "tag got overwritten after signing" situation — Rekor has no record the current alpine digests were ever signed by anyone, at any time. There's no orphaned signature to have lost.
• Not "signatures never persist on Docker Hub for any tag" — that broader theory doesn't hold up; fedora/ubuntu verify fine using the same cosign version against Docker Hub right now.

Likely cause

No alpine-specific conditional exists in .circleci/config.yml or Makefile (all six distros go through the identical Docker AMD/Docker ARM/Docker Bundle steps via << matrix.parameters.distro >>), so this looks like a silent failure inside the alpine job path itself rather than a deliberate skip — e.g. the "Sign per-arch images" or "Sign manifest list index" step for alpine erroring in a way whose exit code doesn't fail the CircleCI job (both sign-image.sh and sign-manifest.sh set -euo pipefail, so worth checking whether the loop that calls make sign-image/make sign-manifest per nginx version is somehow short-circuiting, or whether something about the alpine tag/digest resolution around the 3.24.1 bump is off). Recommend checking the actual CircleCI build logs for the alpine jobs on the commit that produced 1.31.2-alpine3.24.1 for the sign step's real output, since the job-level status alone is misleading here.

Impact

Consumers pulling fabiocicerchia/nginx-lua:latest (or any current alpine tag) get no supply-chain verification even though SUPPLY_CHAIN_SECURITY.md documents that these images are signed and bin/verify-image.sh is provided for consumers to check this.

[fabiocicerchia]

Update after digging further with real CircleCI build data (my local clone had been stale at the June 28 commit — the actual current tip is f4b56bd6, 2026-07-12).

Corrected scope:

• Not alpine-only. alpine and amazonlinux at the current mainline version (1.31.2) are both unsigned (zero Rekor entries for their live digests).
• fedora is a separate, unrelated bug: Docker AMD fedora / Docker ARM fedora failed outright on the f4b56bd6 build, so fedora's :latest-tracking tags still serve the older (properly signed) 2026-06-28 image. Worth its own issue if it's still failing.
• almalinux / debian / ubuntu from the same f4b56bd6 build verified fine and are properly recorded in Rekor.

The real gap: for every distro touched by that build — including the three that signed correctly — GitHub never received a Docker Bundle <distro> status at all. Not failure, just absent. Compare to the June 28 commit, where all six posted normally. So "CI is green" was never reliable evidence either way for this job; the actual signing outcome has to be checked independently. I don't have access to the underlying CircleCI job logs for the un-reported runs, so I can't pin down why alpine/amazonlinux specifically didn't sign while their siblings in the same run did.

Given a preventive fix isn't cleanly diagnosable from what's visible here, I opened #280 with a detective control instead: a nightly job that independently verifies :latest + each distro's alias tag are actually signed, so this class of drift shows up as a red build within a day instead of sitting unnoticed for weeks.