Repository navigation
refactor(deploy): split manifests into base, dev and prod overlays #478
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
devantler
wants to merge
2
commits into
main
Choose a base branch
from
claude/deploy-base-dev-prod-467
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| apiVersion: kustomize.config.k8s.io/v1beta1 | ||
| kind: Kustomization | ||
| # The app and its database that production (../prod) and dev (../dev) both | ||
| # run. Put a manifest here only when both environments run it; production-only | ||
| # objects and settings go in ../prod, and dev changes these with patches in | ||
| # ../dev. | ||
| resources: | ||
| - cluster.yaml | ||
| - deployment.yaml | ||
| - service.yaml | ||
| # Tenant-owned CiliumNetworkPolicy allow-list (layered on the platform's | ||
| # default-deny floor): Gateway ingress (3000), app⇄DB, CNPG + DNS egress. | ||
| # Production adds the R2 backup egress. | ||
| - networkpolicy.yaml | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| apiVersion: kustomize.config.k8s.io/v1beta1 | ||
| kind: Kustomization | ||
| # Dev: the shared app and database (../base), changed only by the settings and | ||
| # patches below. The platform reconciles it at `path: ./dev` into its own | ||
| # namespace and owns its ingress behind sign-on, so dev carries none of | ||
| # production's identity: no route, disruption budget, backup, or OpenBao path. | ||
| # Its database is a separate, empty CloudNativePG cluster in the dev namespace. | ||
| # Change base objects with patches here; never copy a manifest. | ||
| # | ||
| # Left to the dev deployment work (#467): the dev source of the | ||
| # `wedding-app-admin-code` Secret the Deployment reads (dev secrets stay | ||
| # separate from production's), seeded data, the mail sink and the dev bucket. | ||
| # | ||
| # The base hard-codes the production namespace on some objects; pin dev's own | ||
| # (the platform's Flux targetNamespace sets the same) so dev never renders | ||
| # into production's namespace. | ||
| namespace: wedding-app-dev | ||
| resources: | ||
| - ../base | ||
| # One app replica; production runs two. | ||
| replicas: | ||
| - name: wedding-app | ||
| count: 1 | ||
| patches: | ||
| # One database instance. A single instance cannot keep CNPG's own | ||
| # PodDisruptionBudget without deadlocking node drains, so turn it off. | ||
| - target: | ||
| group: postgresql.cnpg.io | ||
| kind: Cluster | ||
| name: wedding-db | ||
| patch: |- | ||
| - op: replace | ||
| path: /spec/instances | ||
| value: 1 | ||
| - op: add | ||
| path: /spec/enablePDB | ||
| value: false | ||
| # The base policy selects its own pods through the production namespace name, | ||
| # and names the cnpg-system, observability and kube-dns namespaces. A new | ||
| # tenant policy may name no namespace (the platform's | ||
| # restrict-tenant-network-policies), so dev selects its app and database | ||
| # pods by label instead and relies on the allow-cnpg-operator and allow-dns | ||
| # policies the platform generates in every tenant namespace. Dev's database | ||
| # is not scraped by Coroot. | ||
| - target: | ||
| kind: CiliumNetworkPolicy | ||
| name: app | ||
| patch: |- | ||
| - op: remove | ||
| path: /spec/ingress/3 | ||
| - op: remove | ||
| path: /spec/ingress/2 | ||
| - op: replace | ||
| path: /spec/ingress/1/fromEndpoints | ||
| value: | ||
| - matchLabels: | ||
| k8s:app.kubernetes.io/name: wedding-app | ||
| - matchLabels: | ||
| k8s:cnpg.io/cluster: wedding-db | ||
| - op: remove | ||
| path: /spec/egress/2 | ||
| - op: replace | ||
| path: /spec/egress/0/toEndpoints | ||
| value: | ||
| - matchLabels: | ||
| k8s:app.kubernetes.io/name: wedding-app | ||
| - matchLabels: | ||
| k8s:cnpg.io/cluster: wedding-db |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,16 +1,9 @@ | ||
| apiVersion: kustomize.config.k8s.io/v1beta1 | ||
| kind: Kustomization | ||
| # Production entry point. The platform reconciles the published manifests | ||
| # artifact at `path: .`, which is this file, so it must only include ./prod. | ||
| # Shared manifests live in ./base, production-only ones in ./prod, and the dev | ||
| # overlay in ./dev (reconciled at `path: ./dev` into the `wedding-app-dev` | ||
| # namespace). | ||
| resources: | ||
| # Namespaced SecretStore backing the admin-code secret below. | ||
| - secretstore.yaml | ||
| - admin-code-externalsecret.yaml | ||
| - admin-code-generator.yaml | ||
| - cluster.yaml | ||
| - db-scheduled-backup.yaml | ||
| - deployment.yaml | ||
| - httproute.yaml | ||
| - pod-disruption-budget.yaml | ||
| - service.yaml | ||
| # Tenant-owned CiliumNetworkPolicy allow-list (layered on the platform's | ||
| # default-deny floor): Gateway ingress (3000), app⇄DB, CNPG, R2 + DNS egress. | ||
| - networkpolicy.yaml | ||
| - prod |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
File renamed without changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| # Production-only: adds continuous backup to the shared wedding-db Cluster | ||
| # (../base/cluster.yaml). Dev runs its own database and never writes to the | ||
| # production archive. | ||
| # | ||
| # Continuous backup through the tenant's dedicated ObjectStore, supplied by | ||
| # the platform together with its dedicated credential. The platform pins the | ||
| # production archive identity separately. No shared backup credential belongs | ||
| # in this tenant. Daily base backups are configured in db-scheduled-backup.yaml. | ||
| apiVersion: postgresql.cnpg.io/v1 | ||
| kind: Cluster | ||
| metadata: | ||
| name: wedding-db | ||
| spec: | ||
| plugins: | ||
| - name: barman-cloud.cloudnative-pg.io | ||
| isWALArchiver: true | ||
| parameters: | ||
| barmanObjectName: wedding-db-dedicated |
File renamed without changes.
File renamed without changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| apiVersion: kustomize.config.k8s.io/v1beta1 | ||
| kind: Kustomization | ||
| # Production: the shared app and database (../base) plus everything that | ||
| # carries the production identity — the public route, the disruption budget, | ||
| # the admin code in OpenBao, and the database backups. The platform reconciles | ||
| # this through ../kustomization.yaml. RESTORE-RUNBOOK.md documents the backups. | ||
| resources: | ||
| - ../base | ||
| # Namespaced SecretStore backing the admin-code secret below. | ||
| - secretstore.yaml | ||
| - admin-code-externalsecret.yaml | ||
| - admin-code-generator.yaml | ||
| - db-scheduled-backup.yaml | ||
| - httproute.yaml | ||
| - pod-disruption-budget.yaml | ||
| patches: | ||
| # Continuous WAL archiving + base backups to the dedicated ObjectStore. | ||
| - path: cluster-backup.yaml | ||
| # CNPG → Cloudflare R2 (S3) for barman-cloud WAL archiving + base backups | ||
| # (the wedding-db CloudNativePG cluster). Pinned by FQDN instead of | ||
| # world:443 so a compromised pod cannot exfiltrate to arbitrary external | ||
| # services; the R2 pattern matches any account host without committing the | ||
| # account id. | ||
| - target: | ||
| kind: CiliumNetworkPolicy | ||
| name: app | ||
| patch: |- | ||
| - op: add | ||
| path: /spec/egress/1 | ||
| value: | ||
| toFQDNs: | ||
| - matchPattern: "*.r2.cloudflarestorage.com" | ||
| toPorts: | ||
| - ports: | ||
| - port: "443" | ||
| protocol: TCP |
File renamed without changes.
File renamed without changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| #!/bin/sh | ||
| # Render deploy/ (the production entry), deploy/prod and deploy/dev, and keep | ||
| # production's identity out of dev. Needs kubectl (with its built-in Kustomize) | ||
| # and yq, both present on the runner. | ||
| set -eu | ||
|
|
||
| repo_root=$(CDPATH='' cd -- "$(dirname -- "$0")/.." && pwd) | ||
| work=$(mktemp -d) | ||
| trap 'rm -rf "$work"' EXIT | ||
|
|
||
| fail() { | ||
| printf 'FAIL: %s\n' "$*" >&2 | ||
| exit 1 | ||
| } | ||
|
|
||
| kubectl kustomize "$repo_root/deploy" > "$work/root.yaml" || fail "deploy/ does not render" | ||
| kubectl kustomize "$repo_root/deploy/prod" > "$work/prod.yaml" || fail "deploy/prod does not render" | ||
| kubectl kustomize "$repo_root/deploy/dev" > "$work/dev.yaml" || fail "deploy/dev does not render" | ||
|
|
||
| # The platform reconciles deploy/ for production, so it must be exactly deploy/prod. | ||
| cmp -s "$work/root.yaml" "$work/prod.yaml" || | ||
| fail "deploy/kustomization.yaml must include only ./prod" | ||
|
|
||
| count_kind() { | ||
| KIND=$1 yq eval-all '[.] | map(select(.kind == strenv(KIND))) | length' "$work/dev.yaml" | ||
| } | ||
| dev_field() { | ||
| KIND=$1 yq eval-all "select(.kind == strenv(KIND)) | $2" "$work/dev.yaml" | ||
| } | ||
|
|
||
| # Dev runs one app replica on one database instance of its own, without backup. | ||
| [ "$(count_kind Deployment)" -eq 1 ] || fail "deploy/dev must render only the app Deployment" | ||
| [ "$(dev_field Deployment .metadata.name)" = wedding-app ] || fail "deploy/dev must render the wedding-app Deployment" | ||
| [ "$(dev_field Deployment .spec.replicas)" = 1 ] || fail "deploy/dev must run one app replica" | ||
| [ "$(count_kind Cluster)" -eq 1 ] || fail "deploy/dev must render one database Cluster" | ||
| [ "$(dev_field Cluster .spec.instances)" = 1 ] || fail "deploy/dev must run one database instance" | ||
| [ "$(dev_field Cluster .spec.enablePDB)" = false ] || fail "deploy/dev's single database instance must not get a CNPG PodDisruptionBudget" | ||
| [ "$(dev_field Cluster '.spec | has("plugins")')" = false ] || fail "deploy/dev's database must not archive to the production backup store" | ||
|
|
||
| # Everything dev renders lives in the wedding-app-dev namespace, and nowhere else. | ||
| [ "$(yq eval-all '[.] | map(select(.metadata.namespace != "wedding-app-dev")) | length' "$work/dev.yaml")" -eq 0 ] || | ||
| fail "every deploy/dev resource must be in the wedding-app-dev namespace" | ||
|
|
||
| # Production identity stays in deploy/prod: the route, the disruption budget, | ||
| # the backups and the admin code in OpenBao. | ||
| for kind in HTTPRoute Gateway Certificate Issuer PodDisruptionBudget ScheduledBackup Backup ObjectStore \ | ||
| SecretStore ExternalSecret PushSecret Password; do | ||
| [ "$(count_kind "$kind")" -eq 0 ] || fail "deploy/dev renders a $kind; it belongs in deploy/prod" | ||
| done | ||
| for needle in wedding.platform.devantler.tech apps/wedding-app/ wedding-db-dedicated r2.cloudflarestorage.com 'namespace: wedding-app$'; do | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
| if grep -Eq -- "$needle" "$work/dev.yaml"; then | ||
| fail "deploy/dev references production ($needle)" | ||
| fi | ||
| done | ||
| # Dev's network policy is created fresh in its own namespace, where the | ||
| # platform refuses a tenant policy that names any namespace. | ||
| if grep -Fq 'io.kubernetes.pod.namespace' "$work/dev.yaml"; then | ||
| fail "deploy/dev network policy names a namespace; select pods by label instead" | ||
| fi | ||
|
|
||
| printf '%s\n' 'PASS: deploy/ renders deploy/prod; deploy/dev runs one replica on its own database without production identity' | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Script executed:
Repository: devantler-tech/wedding-app
Length of output: 33889
Script executed:
Repository: devantler-tech/wedding-app
Length of output: 1918
Merge this file move only with the release workflow change.
At this head,
cd.yamlcallspublish-app.yamlwith the defaultdeploy-path: ./deploy. The publisher's manifest validation requires./deploy/deployment.yaml, and that file no longer exists after the move todeploy/base/deployment.yaml. Every version-tag release therefore fails at validation, before any image or manifest is pushed. Nothing is published unpinned, and no registry state changes. The failure stays blocked until the workflow change that the PR description calls for lands with this move.Do not add a root
deploy/deployment.yamlcopy to satisfy the check. That would duplicate the shared Deployment outside the base. Keep the pinned publisher as is, and land the#479workflow change in the same merge window.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Confirmed: this is valid, and it is the merge-order item this PR's description already names. The release workflow has to move to the shared
actions/publish-appaction with the base manifest (or the verifier has to readdeploy/base) in this PR before it merges. That action shipped in devantler-tech/.github#935, but the platform does not yet trust this repository's own release tags (devantler-tech/platform#6265), so the PR is parked on that and this thread stays open until the switch is pushed here.