Repository navigation
Description
🤖 Generated by the Agentic Engineer
Problem
After #6188 deployed (2026-11-13 18:47Z), the business-site staging app passes admission, but every node refuses to pull its image. Talos's own image check rejects it, so both pods are in ImagePullBackOff and live health reports them failing. The staging hold is still in place, so no traffic is affected, but the app cannot start and the hold cannot be lifted (#6214 is out of the queue).
The image is ghcr.io/devantler-tech/business-site@sha256:d91f543e… (v3.2.2). It carries a valid signature from publish-app v19.0.5 (af042c54…), which the Talos rule for ghcr.io/devantler-tech/* accepts. cosign verify with that rule's exact regex succeeds.
Cause
GHCR has no referrers API, so nodes read the fallback sha256-<digest> tag index. In that index:
| Bundle | Signed by | Index artifactType |
|---|---|---|
6fe31e81… |
business-site CI (ci.yaml@refs/heads/main) |
application/vnd.dev.sigstore.bundle.v0.3+json |
ce4d0aab… |
business-site CI (SLSA provenance) | application/vnd.dev.sigstore.bundle.v0.3+json |
56d5b8e6… |
publish-app v19.0.5 | application/vnd.oci.empty.v1+json (wrong) |
The publish-app bundle's own manifest declares the bundle type. Only its entry in the index is mislabelled. publish-app v19.0.5 and v19.0.12 install cosign v3.0.6 (cosign-installer v4.1.2), which writes that entry. business-site's CI signs with cosign v3.1.4, which labels correctly.
Talos (internal/pkg/containers/image/verify/internal/cosign/cosign.go) first verifies the referrers whose index entry has a sigstore bundle type. It falls back to the bundle tag only when there are none. Here it sees only the two CI bundles, which fail the identity rule, and never tries the publish-app bundle. The error names the CI identity.
Other apps are unaffected because their images carry only the publish-app bundle. With no correctly labelled entry, Talos takes the bundle-tag path and verifies it there. business-site is the first app whose image is built and signed in its own CI and then re-signed by publish-app.
Options
- Talos rule for this image. Add a rule for
ghcr.io/devantler-tech/business-siteahead of the catch-all. It accepts the same publisher identities plus business-site's own main-branch CI identity, which built the image. Admission still requires the publish-app signature. This is a node-level trust change, and it needs a Talos config rollout. - Move to a publish-app revision on cosign v3.1.x. It would write a correctly labelled entry. But it is a third revision for the generic image rules, the Talos rule and the tenant source, and those hold at most two.
- Report the Talos fallback upstream. Talos should also try the bundle tag when every labelled referrer fails. This is right in the long run, but slow.
Done when
The business-site staging pods pull and run on every node, live health is green, and #6214 can lift the hold.
Part of #6177
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Effort
Projects
- StatusShow more project fields✅ Done
Activity
devantler commentedon Nov 13, 2026
Status (updated 2026-11-13 22:50Z). This is unclaimed and handed to the scheduled lanes. Live health is red only because the two business-site staging pods are in ImagePullBackOff. That is this issue, and the chain below fixes it.
Decision. No node-level trust exception: no Talos rule trusts business-site's CI identity, and the generic image rules keep at most two publish-app revisions. The fix is at the source: a publish-app revision whose cosign labels its bundle correctly.
Evidence that cosign v3.1.x fixes the label. These are observed on GHCR, not taken from release notes:
sha256:46295dd3…): published by publish-app v21.3.1 (f3362270…, cosign-installer v4.1.3, i.e. cosign v3.1.4). Its index entry is typedapplication/vnd.dev.sigstore.bundle.v0.3+json.af042c54…, cosign v3.0.6): their entry is typedapplication/vnd.oci.empty.v1+json.Chain (sub-issues, in order):
467f0758…) with a v21.3.x revision across the generic rules. v21.3.4 isa76d5df9371cf712a8f60307fde060825a34cf35; a newer revision on cosign-installer v4.1.3 or later also works after a range review. Use fix(supply-chain): replace the retired v7.23.0 publisher with v19.0.5 #5422 as the template:publishappregex inverify-app-images.yaml; the catch-allghcr.io/devantler-tech/*rule intalos/cluster/verify-first-party-images.yaml; the tenant RGD default subject; andscripts/tenant-source-constructors.lib.sh.467f0758from the ascoachingogvaner, wedding-app and data-product-controller rows ofscripts/publish-workflow-canonical-approvals.tsv. Leave business-site for Move business-site to the v21.3.x publisher and lift its staging hold #6229.scripts/write-publish-workflow-matchers.sh. Take the ledger lines and RGD scan-baseline hashes from their validators.publish-runtime.yamlto the approved v21.3.x revision and update its Dependabot note. Release it throughrelease-runtime.yamlin release mode, switchingRUNTIME_PUBLICATION_ENABLEDon only for the admit step.bootstrap-tenant-source.sh --prepare-canonical-upgrade. The v3.2.2 published-upgrade must be settled to a binding first, by the daily regeneration.fix(release): sign releases with the reviewed v19.0.5 publisher (#614)
fix(publish): sign the runtime with the v21.3.4 publisher (#574)
devantler commentedon Nov 14, 2026
Verified fixed, 2026-11-14 06:25Z. Closing.
Runningand ready (1/1), up for about 3 h 50 m, so the image that Talos refused is now admitted.ImagePullBackOfforErrImagePull, so no running image is refused by the updated rules.#6230 (move wedding-app and ascoachingogvaner to v21.3.x, then retire v19.0.5) is a separate follow-up and stays open in the Backlog.