Skip to content

Talos refuses the business-site staging image because its publisher signature is mislabelled in the registry index #6226

Closed
3 of 4 issues completed

Description

@devantler

🤖 Generated by the Agentic Engineer

Problem

After #6188 deployed (2026-11-13 18:47Z), the business-site staging app passes admission, but every node refuses to pull its image. Talos's own image check rejects it, so both pods are in ImagePullBackOff and live health reports them failing. The staging hold is still in place, so no traffic is affected, but the app cannot start and the hold cannot be lifted (#6214 is out of the queue).

The image is ghcr.io/devantler-tech/business-site@sha256:d91f543e… (v3.2.2). It carries a valid signature from publish-app v19.0.5 (af042c54…), which the Talos rule for ghcr.io/devantler-tech/* accepts. cosign verify with that rule's exact regex succeeds.

Cause

GHCR has no referrers API, so nodes read the fallback sha256-<digest> tag index. In that index:

Bundle Signed by Index artifactType
6fe31e81… business-site CI (ci.yaml@refs/heads/main) application/vnd.dev.sigstore.bundle.v0.3+json
ce4d0aab… business-site CI (SLSA provenance) application/vnd.dev.sigstore.bundle.v0.3+json
56d5b8e6… publish-app v19.0.5 application/vnd.oci.empty.v1+json (wrong)

The publish-app bundle's own manifest declares the bundle type. Only its entry in the index is mislabelled. publish-app v19.0.5 and v19.0.12 install cosign v3.0.6 (cosign-installer v4.1.2), which writes that entry. business-site's CI signs with cosign v3.1.4, which labels correctly.

Talos (internal/pkg/containers/image/verify/internal/cosign/cosign.go) first verifies the referrers whose index entry has a sigstore bundle type. It falls back to the bundle tag only when there are none. Here it sees only the two CI bundles, which fail the identity rule, and never tries the publish-app bundle. The error names the CI identity.

Other apps are unaffected because their images carry only the publish-app bundle. With no correctly labelled entry, Talos takes the bundle-tag path and verifies it there. business-site is the first app whose image is built and signed in its own CI and then re-signed by publish-app.

Options

  1. Talos rule for this image. Add a rule for ghcr.io/devantler-tech/business-site ahead of the catch-all. It accepts the same publisher identities plus business-site's own main-branch CI identity, which built the image. Admission still requires the publish-app signature. This is a node-level trust change, and it needs a Talos config rollout.
  2. Move to a publish-app revision on cosign v3.1.x. It would write a correctly labelled entry. But it is a third revision for the generic image rules, the Talos rule and the tenant source, and those hold at most two.
  3. Report the Talos fallback upstream. Talos should also try the bundle tag when every labelled referrer fails. This is right in the long run, but slow.

Done when

The business-site staging pods pull and run on every node, live health is green, and #6214 can lift the hold.

Part of #6177

Activity

  1. added theissue type on Nov 13, 2026
  2. devantler commented on Nov 13, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Status (updated 2026-11-13 22:50Z). This is unclaimed and handed to the scheduled lanes. Live health is red only because the two business-site staging pods are in ImagePullBackOff. That is this issue, and the chain below fixes it.

    Decision. No node-level trust exception: no Talos rule trusts business-site's CI identity, and the generic image rules keep at most two publish-app revisions. The fix is at the source: a publish-app revision whose cosign labels its bundle correctly.

    Evidence that cosign v3.1.x fixes the label. These are observed on GHCR, not taken from release notes:

    • v3.2.0's image (sha256:46295dd3…): published by publish-app v21.3.1 (f3362270…, cosign-installer v4.1.3, i.e. cosign v3.1.4). Its index entry is typed application/vnd.dev.sigstore.bundle.v0.3+json.
    • Images from v19.0.5 (af042c54…, cosign v3.0.6): their entry is typed application/vnd.oci.empty.v1+json.
    • No change to publish-app itself is needed. v21.3.1 and later already contain the fix, and v21.3.1 to v21.3.4 adds only dependency bumps.

    Chain (sub-issues, in order):

    1. ✅ Move data-product-controller to a KHoist release signed by the v19.0.5 publisher #6227 (done 22:40Z). data-product-controller runs KHoist 9.0.12 (khoist#614), signed by v19.0.5, via feat(data-products): move the controller to KHoist 9.0.12 #6239. Its data products verified live. No running first-party app image is signed by v8.0.0 any more, checked at 22:47Z, so v8.0.0's slot is free.
    2. 🫴 Replace the v8.0.0 app publisher with v21.3.x in the generic image rules #6228 (Ready, next). Replace v8.0.0 (467f0758…) with a v21.3.x revision across the generic rules. v21.3.4 is a76d5df9371cf712a8f60307fde060825a34cf35; a newer revision on cosign-installer v4.1.3 or later also works after a range review. Use fix(supply-chain): replace the retired v7.23.0 publisher with v19.0.5 #5422 as the template:
      • Rule files: the publishapp regex in verify-app-images.yaml; the catch-all ghcr.io/devantler-tech/* rule in talos/cluster/verify-first-party-images.yaml; the tenant RGD default subject; and scripts/tenant-source-constructors.lib.sh.
      • Approvals: drop 467f0758 from the ascoachingogvaner, wedding-app and data-product-controller rows of scripts/publish-workflow-canonical-approvals.tsv. Leave business-site for Move business-site to the v21.3.x publisher and lift its staging hold #6229.
      • Generated files: regenerate the matchers with scripts/write-publish-workflow-matchers.sh. Take the ledger lines and RGD scan-baseline hashes from their validators.
      • Tests: make them refuse v8.0.0 at every boundary.
      • Rollout: the Talos rule goes out with the normal pipeline deploy.
      • Done when: the deployed admission policy and Tenant graph, and the Talos rule loaded on every node, accept exactly v19.0.5 and the chosen v21.3.x, and no running image is refused.
    3. Move business-site to the v21.3.x publisher and lift its staging hold #6229. Move business-site's publish-runtime.yaml to the approved v21.3.x revision and update its Dependabot note. Release it through release-runtime.yaml in release mode, switching RUNTIME_PUBLICATION_ENABLED on only for the admit step.
      • Re-pin: use bootstrap-tenant-source.sh --prepare-canonical-upgrade. The v3.2.2 published-upgrade must be settled to a binding first, by the daily regeneration.
      • Hold: put feat(business-site): lift the staging ingress hold #6214 back in the queue once the pods are Ready, then record the live check on business-site#554.
    4. Move wedding-app and ascoachingogvaner to v21.3.x and retire v19.0.5 #6230. A follow-up: move wedding-app and ascoachingogvaner to v21.3.x, then retire v19.0.5.
  3. devantler commented on Nov 14, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Verified fixed, 2026-11-14 06:25Z. Closing.

    #6230 (move wedding-app and ascoachingogvaner to v21.3.x, then retire v19.0.5) is a separate follow-up and stays open in the Backlog.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    None yet

    Effort

    None yet

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions