From 88f163768f00b222bb2e5f35a06a48057a7328e5 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Sun, 15 Nov 2026 14:18:15 +0100 Subject: [PATCH 1/2] fix(ci): follow Platform's retirement of the GHCR pull credential Platform#6318 deleted the tenant ghcr-auth ExternalSecret, the ServiceAccount pull secret and the OCIRepository secretRef, because every tenant artifact is now public. The envelope test still required them, so Validate Scaffold went red on main. Assert the credential-free shape instead, invert the matching mutations so a returning credential is caught, and drop the scaffold's imagePullSecrets reference to the secret that no longer exists. Co-Authored-By: Claude Opus 5.5 --- deploy/deployment.yaml | 2 - docs/REFERENCE.md | 4 +- .../platform-tenant-envelope-contract.test.sh | 4 +- scripts/platform-tenant-envelope.test.sh | 85 +++++-------------- 4 files changed, 22 insertions(+), 73 deletions(-) diff --git a/deploy/deployment.yaml b/deploy/deployment.yaml index aa29736..e1dad00 100644 --- a/deploy/deployment.yaml +++ b/deploy/deployment.yaml @@ -84,8 +84,6 @@ spec: labelSelector: matchLabels: app.kubernetes.io/name: app - imagePullSecrets: - - name: ghcr-auth containers: # ⚠️ The container name MUST equal the repository name: cd.yaml → # publish-app pins the freshly built image digest into the container diff --git a/docs/REFERENCE.md b/docs/REFERENCE.md index 1431646..9723fab 100644 --- a/docs/REFERENCE.md +++ b/docs/REFERENCE.md @@ -104,14 +104,14 @@ never re-synced, so a tenant still carrying these from an older sync can delete | `stacks/` | The Go and .NET stack sources the scaffold copies once; the copies are then yours | | `scripts/scaffold.sh` (+ its test) | One-shot stack and tenancy scaffold; its test scaffolds every combination and checks what each leaves | | `.github/workflows/validate-stacks.yaml` | Scaffolds each stack as a tenant, then builds, tests, lints, renders `deploy/`, builds the image and checks it serves `GET /` under the restricted container settings | -| `.github/workflows/validate-scaffold.yaml` | Renders `deploy/`, schema-validates every resource, applies the live Platform/shared Kyverno policies, and exercises a pinned Kubernetes API. The live checks prove the template publishes through the pinned signing workflow; Platform still wires the private GHCR pull credential to the tenant identity and cosign-verifying OCI source; and Flux consumes that source while impersonating and targeting the managed `restricted` namespace for both KRO and manual registrations. They then prove the Deployment passes Pod Security `restricted`, the tenant identity can reconcile every rendered kind, and cluster-scoped or interactive privileges remain denied. It gates template PRs, re-validates `main` after every merge, and rechecks upstream drift every day at 06:17 UTC (or on manual dispatch), keeping one tracking issue open while that scheduled check fails on `main`; structural mutation tests keep every layer fail-closed. The workflow no-ops in tenants and is scaffold-time only | +| `.github/workflows/validate-scaffold.yaml` | Renders `deploy/`, schema-validates every resource, applies the live Platform/shared Kyverno policies, and exercises a pinned Kubernetes API. The live checks prove the template publishes through the pinned signing workflow; Platform pulls the public tenant artifact without a credential through a cosign-verifying OCI source; and Flux consumes that source while impersonating and targeting the managed `restricted` namespace for both KRO and manual registrations. They then prove the Deployment passes Pod Security `restricted`, the tenant identity can reconcile every rendered kind, and cluster-scoped or interactive privileges remain denied. It gates template PRs, re-validates `main` after every merge, and rechecks upstream drift every day at 06:17 UTC (or on manual dispatch), keeping one tracking issue open while that scheduled check fails on `main`; structural mutation tests keep every layer fail-closed. The workflow no-ops in tenants and is scaffold-time only | | `scripts/rename-placeholders.sh` (+ its test) | One-shot rename of the placeholder app to your tenant name | | `scripts/agent-instructions.test.sh` | Fails closed if the one-time agent scaffold loses its ownership, dependency-bot, external-code, exact-head review, pinned-merge, or user-path evaluation boundaries, or regains a retired rule | | `scripts/workflow-caller-contract.test.sh` | Keeps the portable caller-pin contract wired into tenant CI while validating template-only publisher, release, ownership, and scaffold invariants | | `scripts/tenant-ci-contract.test.sh` | Keeps default tenant PR CI least-privilege and fail-closed over both the image build and rendered manifests | | `scripts/pod-security-admission*.test.sh` | Proves the rendered Deployment is accepted at Pod Security `restricted` while unsafe mutations are denied, and pins that live gate against structural bypasses | | `scripts/tenant-rbac*.test.sh` | Proves the Platform tenant reconciliation identity can manage every rendered scaffold resource while cluster-scoped and interactive privileges stay denied | -| `scripts/platform-tenant-envelope*.test.sh` | Binds the template's signed publisher and those workload-level models to Platform's live KRO and manual tenant registrations: private GHCR pull identity, cosign-verifying OCI source, managed Pod Security namespace, `tenant-edit` ServiceAccount binding, Flux source/impersonation/target namespace, and the exact OpenBao policy and Kubernetes-auth role that let the renamed tenant read and seed only its own app secrets | +| `scripts/platform-tenant-envelope*.test.sh` | Binds the template's signed publisher and those workload-level models to Platform's live KRO and manual tenant registrations: credential-free GHCR pull, cosign-verifying OCI source, managed Pod Security namespace, `tenant-edit` ServiceAccount binding, Flux source/impersonation/target namespace, and the exact OpenBao policy and Kubernetes-auth role that let the renamed tenant read and seed only its own app secrets | | `scripts/platform-network-floor*.test.sh` | Binds Platform's generated default-deny, DNS, and standard NetworkPolicy floor to the scaffold's required Gateway, namespace, CNPG, Kubernetes API, and DNS paths, reporting a check that cannot read its input separately from a policy verdict | | `scripts/platform-vpa-floor*.test.sh` | Uses Kyverno to evaluate Platform's Deployment GeneratingPolicy and keeps the scaffold's initial CPU request at or above the generated VPA floor, with mutation coverage | diff --git a/scripts/platform-tenant-envelope-contract.test.sh b/scripts/platform-tenant-envelope-contract.test.sh index 4311cb5..cf808ef 100644 --- a/scripts/platform-tenant-envelope-contract.test.sh +++ b/scripts/platform-tenant-envelope-contract.test.sh @@ -90,8 +90,6 @@ validate_contract() { 'pod-security.kubernetes.io/enforce-version' \ 'tenant-edit' \ 'imagePullSecrets' \ - 'infrastructure/ghcr/auth' \ - 'kubernetes.io/dockerconfigjson' \ 'matchOIDCIdentity' \ 'publish-app.yaml' \ 'serviceAccountName' \ @@ -231,7 +229,7 @@ run_mutation "SecretStore managed namespace context removed" \ '(.jobs.admissibility.steps[] | select(.id == "apply-admission-policies").run) |= sub("app.kubernetes.io/managed-by"; "removed-label")' '' run_mutation "SOPS variant validation removed" '' '/kustomizationSops/d' run_mutation "target namespace validation removed" '' '/targetNamespace/d' -run_mutation "private artifact credential validation removed" '' '/ghcrAuth/d' +run_mutation "credential-free artifact validation removed" '' '/ghcrAuth/d' run_mutation "signed publisher validation removed" '' '/publish-app\.yaml/d' run_mutation "OpenBao authorization validation removed" '' \ '/validate_openbao_authorization/d' diff --git a/scripts/platform-tenant-envelope.test.sh b/scripts/platform-tenant-envelope.test.sh index 00f4beb..bd25db1 100644 --- a/scripts/platform-tenant-envelope.test.sh +++ b/scripts/platform-tenant-envelope.test.sh @@ -349,7 +349,6 @@ validate_platform() { manual_root=$platform_root/k8s/bases/apps/ascoachingogvaner manual_namespace=$manual_root/namespace.yaml manual_service_account=$manual_root/service-account.yaml - manual_ghcr_auth=$manual_root/external-secret.yaml manual_oci_repository=$manual_root/oci-repository.yaml manual_role_binding=$manual_root/role-binding-ascoachingogvaner.yaml manual_kustomization=$manual_root/flux-kustomization.yaml @@ -360,7 +359,6 @@ validate_platform() { "$vault_config" \ "$manual_namespace" \ "$manual_service_account" \ - "$manual_ghcr_auth" \ "$manual_oci_repository" \ "$manual_role_binding" \ "$manual_kustomization" \ @@ -375,7 +373,6 @@ validate_platform() { yq eval -e ' (.spec.resources[] | select(.id == "tenantNamespace").template) as $namespace | (.spec.resources[] | select(.id == "serviceAccount").template) as $serviceAccount - | (.spec.resources[] | select(.id == "ghcrAuth").template) as $ghcrAuth | (.spec.resources[] | select(.id == "ociRepository").template) as $ociRepository | (.spec.resources[] | select(.id == "roleBinding").template) as $roleBinding | (.spec.resources[] | select(.id == "kustomization")) as $kustomizationResource @@ -384,7 +381,7 @@ validate_platform() { | $kustomizationSopsResource.template as $kustomizationSops | (.spec.schema.types.TenantSource.digest == "string | default=\"\" maxLength=71 pattern=\"^$|^sha256:[0-9a-f]{64}$\"" and .spec.schema.spec.source == "TenantSource | default={} validation=\"((!has(self.digest) || self.digest == '\'''\'') == (!has(self.publisherRevision) || self.publisherRevision == '\'''\'')) && (!has(self.reconcileApplication) || self.reconcileApplication || (has(self.digest) && self.digest != '\'''\''))\" validation=\"!has(oldSelf.digest) || oldSelf.digest == '\'''\'' || (has(self.digest) && self.digest != '\'''\'')\"") as $sourceSchemaValid - | (([.spec.resources[] | select(.id == "ghcrAuth")] | length) == 1 + | (([.spec.resources[] | select(.id == "ghcrAuth")] | length) == 0 and ([.spec.resources[] | select(.id == "ociRepository")] | length) == 1 and $namespace.kind == "Namespace" and $namespace.metadata.name == "${schema.spec.name}" @@ -396,21 +393,7 @@ validate_platform() { and $serviceAccount.metadata.namespace == "${tenantNamespace.metadata.name}" and $serviceAccount.metadata.labels."app.kubernetes.io/managed-by" == "kro" and $serviceAccount.automountServiceAccountToken == false - and ($serviceAccount.imagePullSecrets | length) == 1 - and $serviceAccount.imagePullSecrets[0].name == "ghcr-auth" - and $ghcrAuth.kind == "ExternalSecret" - and $ghcrAuth.metadata.name == "ghcr-auth" - and $ghcrAuth.metadata.namespace == "${tenantNamespace.metadata.name}" - and $ghcrAuth.spec.secretStoreRef.name == "openbao" - and $ghcrAuth.spec.secretStoreRef.kind == "ClusterSecretStore" - and $ghcrAuth.spec.target.name == "ghcr-auth" - and $ghcrAuth.spec.target.creationPolicy == "Owner" - and $ghcrAuth.spec.target.template.type == "kubernetes.io/dockerconfigjson" - and $ghcrAuth.spec.target.template.data.".dockerconfigjson" == "{{ .dockerconfigjson }}" - and ($ghcrAuth.spec.data | length) == 1 - and $ghcrAuth.spec.data[0].secretKey == "dockerconfigjson" - and $ghcrAuth.spec.data[0].remoteRef.key == "infrastructure/ghcr/auth" - and $ghcrAuth.spec.data[0].remoteRef.property == "dockerconfigjson" + and ($serviceAccount | has("imagePullSecrets") | not) and $ociRepository.kind == "OCIRepository" and $ociRepository.metadata.name == "${schema.spec.name}" and $ociRepository.metadata.namespace == "${tenantNamespace.metadata.name}" @@ -420,7 +403,7 @@ validate_platform() { or ($ociRepository.spec.ref == "${schema.spec.source.digest == \"\" ? {\"semver\": \">=1.0.0\"} : {\"digest\": schema.spec.source.digest}}" and $sourceSchemaValid)) and $ociRepository.spec.url == "oci://ghcr.io/devantler-tech/${schema.spec.name}/manifests" - and $ociRepository.spec.secretRef.name == "ghcr-auth" + and ($ociRepository.spec | has("secretRef") | not) and (($ociRepository.spec.suspend // false) == false) and $ociRepository.spec.verify.provider == "cosign" and ($ociRepository.spec.verify | has("secretRef") | not) @@ -490,27 +473,9 @@ validate_platform() { and .metadata.namespace == strenv(tenant_name) and .metadata.labels."app.kubernetes.io/managed-by" == "ksail" and .automountServiceAccountToken == false - and (.imagePullSecrets | length) == 1 - and .imagePullSecrets[0].name == "ghcr-auth" + and (has("imagePullSecrets") | not) ' "$manual_service_account" >/dev/null || fail "manual Platform tenant ServiceAccount no longer matches its namespace" - # shellcheck disable=SC2016 - yq eval -e ' - .kind == "ExternalSecret" - and .metadata.name == "ghcr-auth" - and .metadata.namespace == strenv(tenant_name) - and .spec.secretStoreRef.name == "openbao" - and .spec.secretStoreRef.kind == "ClusterSecretStore" - and .spec.target.name == "ghcr-auth" - and .spec.target.creationPolicy == "Owner" - and .spec.target.template.type == "kubernetes.io/dockerconfigjson" - and .spec.target.template.data.".dockerconfigjson" == "{{ .dockerconfigjson }}" - and (.spec.data | length) == 1 - and .spec.data[0].secretKey == "dockerconfigjson" - and .spec.data[0].remoteRef.key == "infrastructure/ghcr/auth" - and .spec.data[0].remoteRef.property == "dockerconfigjson" - ' "$manual_ghcr_auth" >/dev/null || fail "manual Platform tenant no longer produces the private GHCR pull secret" - # shellcheck disable=SC2016 yq eval -e ' .kind == "OCIRepository" @@ -520,7 +485,7 @@ validate_platform() { and (.spec.ref | has("digest") | not) and (.spec.ref | has("tag") | not) and .spec.url == strenv(expected_manual_oci_url) - and .spec.secretRef.name == "ghcr-auth" + and (.spec | has("secretRef") | not) and ((.spec.suspend // false) == false) and .spec.verify.provider == "cosign" and (.spec.verify | has("secretRef") | not) @@ -528,7 +493,7 @@ validate_platform() { and .spec.verify.matchOIDCIdentity[0].issuer == strenv(expected_oidc_issuer) and (.spec.verify.matchOIDCIdentity[0].subject | tag == "!!str") and (.spec.verify.matchOIDCIdentity[0].subject | test("[[:space:][:cntrl:]]") | not) - ' "$manual_oci_repository" >/dev/null || fail "manual Platform tenant OCI source no longer requires the signed private artifact" + ' "$manual_oci_repository" >/dev/null || fail "manual Platform tenant OCI source no longer requires the signed artifact" manual_subject=$(yq eval -r '.spec.verify.matchOIDCIdentity[0].subject' "$manual_oci_repository") # The manual source may add only its own repository's release workflow, and # that repository is the tenant whose artifact path was bound above. @@ -566,7 +531,7 @@ validate_platform() { .kind == "Kustomization" and ([.resources[] | select(. == "namespace.yaml")] | length) == 1 and ([.resources[] | select(. == "service-account.yaml")] | length) == 1 - and ([.resources[] | select(. == "external-secret.yaml")] | length) == 1 + and ([.resources[] | select(. == "external-secret.yaml")] | length) == 0 and ([.resources[] | select(. == "oci-repository.yaml")] | length) == 1 and ([.resources[] | select(. == "role-binding-ascoachingogvaner.yaml")] | length) == 1 and ([.resources[] | select(. == "flux-kustomization.yaml")] | length) == 1 @@ -1115,22 +1080,16 @@ run_mutation "KRO reconciler kind changed" "$rgd_path" \ # shellcheck disable=SC2016 run_mutation "KRO activation predicates overlap" "$rgd_path" \ '(.spec.resources[] | select(.id == "kustomization").includeWhen) = ["${schema.spec.sops}"]' -run_mutation "KRO GHCR credential resource removed" "$rgd_path" \ - 'del(.spec.resources[] | select(.id == "ghcrAuth"))' +run_mutation "KRO GHCR credential resource re-added" "$rgd_path" \ + '.spec.resources += [{"id": "ghcrAuth", "template": {"kind": "ExternalSecret"}}]' run_mutation "KRO OCI source resource removed" "$rgd_path" \ 'del(.spec.resources[] | select(.id == "ociRepository"))' -run_mutation "KRO image-pull credential disconnected" "$rgd_path" \ - 'del(.spec.resources[] | select(.id == "serviceAccount").template.imagePullSecrets)' -run_mutation "KRO GHCR source widened" "$rgd_path" \ - '(.spec.resources[] | select(.id == "ghcrAuth").template.spec.data[0].remoteRef.key) = "apps/shared"' -run_mutation "KRO GHCR target type changed" "$rgd_path" \ - '(.spec.resources[] | select(.id == "ghcrAuth").template.spec.target.template.type) = "Opaque"' -run_mutation "KRO GHCR docker config mapping removed" "$rgd_path" \ - 'del(.spec.resources[] | select(.id == "ghcrAuth").template.spec.target.template.data.".dockerconfigjson")' +run_mutation "KRO image-pull credential re-attached" "$rgd_path" \ + '(.spec.resources[] | select(.id == "serviceAccount").template.imagePullSecrets) = [{"name": "ghcr-auth"}]' run_mutation "KRO OCI tenant path widened" "$rgd_path" \ '(.spec.resources[] | select(.id == "ociRepository").template.spec.url) = "oci://ghcr.io/devantler-tech/manifests"' -run_mutation "KRO OCI pull credential disconnected" "$rgd_path" \ - 'del(.spec.resources[] | select(.id == "ociRepository").template.spec.secretRef)' +run_mutation "KRO OCI pull credential re-attached" "$rgd_path" \ + '(.spec.resources[] | select(.id == "ociRepository").template.spec.secretRef.name) = "ghcr-auth"' run_mutation "KRO OCI verification removed" "$rgd_path" \ 'del(.spec.resources[] | select(.id == "ociRepository").template.spec.verify)' run_mutation "KRO OCI issuer widened" "$rgd_path" \ @@ -1161,18 +1120,12 @@ run_mutation "manual reconciler identity removed" "$manual_path/flux-kustomizati 'del(.spec.serviceAccountName)' run_mutation "manual target namespace removed" "$manual_path/flux-kustomization.yaml" \ 'del(.spec.targetNamespace)' -run_mutation "manual GHCR source property changed" "$manual_path/external-secret.yaml" \ - '(.spec.data[0].remoteRef.property) = "token"' -run_mutation "manual GHCR target type changed" "$manual_path/external-secret.yaml" \ - '(.spec.target.template.type) = "Opaque"' -run_mutation "manual GHCR docker config mapping removed" "$manual_path/external-secret.yaml" \ - 'del(.spec.target.template.data.".dockerconfigjson")' -run_mutation "manual image-pull credential disconnected" "$manual_path/service-account.yaml" \ - 'del(.imagePullSecrets)' +run_mutation "manual image-pull credential re-attached" "$manual_path/service-account.yaml" \ + '.imagePullSecrets = [{"name": "ghcr-auth"}]' run_mutation "manual OCI tenant path changed" "$manual_path/oci-repository.yaml" \ '(.spec.url) = "oci://ghcr.io/devantler-tech/another-tenant/manifests"' -run_mutation "manual OCI pull credential disconnected" "$manual_path/oci-repository.yaml" \ - 'del(.spec.secretRef)' +run_mutation "manual OCI pull credential re-attached" "$manual_path/oci-repository.yaml" \ + '.spec.secretRef.name = "ghcr-auth"' run_mutation "manual OCI verification removed" "$manual_path/oci-repository.yaml" \ 'del(.spec.verify)' run_mutation "manual OCI subject widened" "$manual_path/oci-repository.yaml" \ @@ -1218,8 +1171,8 @@ run_mutation "manual Flux artifact source crosses namespace" "$manual_path/flux- '.spec.sourceRef.namespace = "another-tenant"' run_mutation "manual Namespace removed from inventory" "$manual_path/kustomization.yaml" \ '.resources -= ["namespace.yaml"]' -run_mutation "manual GHCR credential removed from inventory" "$manual_path/kustomization.yaml" \ - '.resources -= ["external-secret.yaml"]' +run_mutation "manual GHCR credential returned to inventory" "$manual_path/kustomization.yaml" \ + '.resources += ["external-secret.yaml"]' run_mutation "manual OCI source removed from inventory" "$manual_path/kustomization.yaml" \ '.resources -= ["oci-repository.yaml"]' From 67eab07a646a5bf46c876e7303c7858f5f8b67e0 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Sun, 15 Nov 2026 14:27:09 +0100 Subject: [PATCH 2/2] fix(ci): also reject a pull identity on the tenant OCI source Flux can authenticate an OCIRepository through spec.serviceAccountName as well as spec.secretRef. Refuse both, with a mutation for each path. Co-Authored-By: Claude Opus 5.5 --- scripts/platform-tenant-envelope.test.sh | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/scripts/platform-tenant-envelope.test.sh b/scripts/platform-tenant-envelope.test.sh index bd25db1..285d15c 100644 --- a/scripts/platform-tenant-envelope.test.sh +++ b/scripts/platform-tenant-envelope.test.sh @@ -404,6 +404,7 @@ validate_platform() { and $sourceSchemaValid)) and $ociRepository.spec.url == "oci://ghcr.io/devantler-tech/${schema.spec.name}/manifests" and ($ociRepository.spec | has("secretRef") | not) + and ($ociRepository.spec | has("serviceAccountName") | not) and (($ociRepository.spec.suspend // false) == false) and $ociRepository.spec.verify.provider == "cosign" and ($ociRepository.spec.verify | has("secretRef") | not) @@ -486,6 +487,7 @@ validate_platform() { and (.spec.ref | has("tag") | not) and .spec.url == strenv(expected_manual_oci_url) and (.spec | has("secretRef") | not) + and (.spec | has("serviceAccountName") | not) and ((.spec.suspend // false) == false) and .spec.verify.provider == "cosign" and (.spec.verify | has("secretRef") | not) @@ -1090,6 +1092,8 @@ run_mutation "KRO OCI tenant path widened" "$rgd_path" \ '(.spec.resources[] | select(.id == "ociRepository").template.spec.url) = "oci://ghcr.io/devantler-tech/manifests"' run_mutation "KRO OCI pull credential re-attached" "$rgd_path" \ '(.spec.resources[] | select(.id == "ociRepository").template.spec.secretRef.name) = "ghcr-auth"' +run_mutation "KRO OCI pull identity attached" "$rgd_path" \ + '(.spec.resources[] | select(.id == "ociRepository").template.spec.serviceAccountName) = "ghcr-puller"' run_mutation "KRO OCI verification removed" "$rgd_path" \ 'del(.spec.resources[] | select(.id == "ociRepository").template.spec.verify)' run_mutation "KRO OCI issuer widened" "$rgd_path" \ @@ -1126,6 +1130,8 @@ run_mutation "manual OCI tenant path changed" "$manual_path/oci-repository.yaml" '(.spec.url) = "oci://ghcr.io/devantler-tech/another-tenant/manifests"' run_mutation "manual OCI pull credential re-attached" "$manual_path/oci-repository.yaml" \ '.spec.secretRef.name = "ghcr-auth"' +run_mutation "manual OCI pull identity attached" "$manual_path/oci-repository.yaml" \ + '.spec.serviceAccountName = "ghcr-puller"' run_mutation "manual OCI verification removed" "$manual_path/oci-repository.yaml" \ 'del(.spec.verify)' run_mutation "manual OCI subject widened" "$manual_path/oci-repository.yaml" \