feat(scripts): cnpg-demo — a scripted CloudNativePG fixture cluster - #1396
feat(scripts): cnpg-demo — a scripted CloudNativePG fixture cluster#1396nadaverell wants to merge 2 commits into
Conversation
CNPG had the problem `make gitops-demo` exists to solve and no answer: anyone testing CNPG UI changes was testing against whatever cluster was in their current context, usually one healthy Postgres with none of the variety that matters. `make cnpg-demo` builds a kind cluster with four clusters that all reach 2/2 Ready and render four different badges — Healthy, WAL Archiving Failing, and Unrecoverable twice. The uniform instance count is the point: every difference is invisible to counts, which is the class of bug the fixture exists to catch. Plus a Pooler, a plugin-method ScheduledBackup, three Backups, and Velero and KubeBlocks CRs for the shared `backups` and `clusters` plurals — with one engine installed, a guard that matches everything and a guard that matches the right group behave identically. The payload is the ordering, and every constraint below was found by running the script rather than by reasoning about it. Each one fails in a way that looks like success: Freezing the operator does not hold a status. Each Postgres pod runs an instance manager that writes cluster status independently, so a patched ContinuousArchiving=False reverts within ~10 minutes with the operator scaled to zero. `status.phase` survives; conditions and readyInstances do not. So the WAL failure is caused, not asserted — but only after the cluster is up. CNPG verifies the object store during bootstrap, so a cluster created with an unroutable endpoint never becomes Ready at all. That still yields ContinuousArchiving=False, which is the convincing wrong answer: the condition looks right on a cluster that is simply broken. The check therefore requires both the condition AND 2/2 Ready, because asserting the condition alone passes on the broken version — as it did, and reported success. Backups must be created after the freeze, not merely patched after it. The operator attempts a Backup, and one pointed at a cluster with no backup configuration fails and stamps LastBackupSucceeded=False on it. The pristine baseline silently rendered "Backup Failed" — via a resource attached to it rather than anything wrong with it. A Pooler must not share a name with one of its cluster's Services. CNPG creates <cluster>-rw/-ro/-r, and a Pooler creates a Service of its own name, so pg-healthy-rw refused to reconcile forever with "invalid ownership for managed resources" — no status, no condition, no event, just a CR that looks slow. Thawing is not scaling back up. The operator builds its PKI from its own webhook configurations at startup, so with them deleted it crash-loops on ensurePKI. `thaw` re-applies the manifest first. Timestamps are computed relative to now. A hardcoded expiry rendered "Expired" rather than "24d remaining" within months of being written. `live` skips the freeze for v2/v3 work that needs a reconciling controller — real failovers and backup runs, at the cost of hand-written phases. Verified end-to-end from an empty machine, then against Radar: four badges as documented, audit evaluated 4 / passed 1 with only the plugin-method cluster passing, four CNPG issues, and both backup engines rendering their own column sets on one cluster.
Same slot as the GitOps and Crossplane entries, with the instance-manager gotcha called out — anyone editing the fixtures needs to know it before they patch a condition and watch it heal itself ten minutes later.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 0eda339. Configure here.
| date -u -v"${spec}" +%Y-%m-%dT%H:%M:%SZ 2>/dev/null \ | ||
| || date -u -d "${spec/#+/} ${spec:0:1}" +%Y-%m-%dT%H:%M:%SZ 2>/dev/null \ | ||
| || date -u +%Y-%m-%dT%H:%M:%SZ | ||
| } |
There was a problem hiding this comment.
GNU date fallback produces now
Medium Severity
ts_offset builds invalid GNU date -d strings like -4H - and 25d +, so on Linux both relative forms fail and the function silently returns the current time. That stamps Velero expiration as now, so the collision row shows Expired instead of remaining time — the exact failure this helper was added to avoid.
Reviewed by Cursor Bugbot for commit 0eda339. Configure here.
|
|
||
| if cluster_exists; then | ||
| step "Cluster '${CLUSTER_NAME}' already exists — reusing" | ||
| thaw_operator_quiet |
There was a problem hiding this comment.
Re-run thaws over existing Backups
High Severity
Reusing a cluster thaws the operator while CNPG Backup CRs from a prior freeze are still present. The script’s own comments note that a live operator will attempt those Backups and stamp LastBackupSucceeded=False on pg-healthy, turning the baseline badge into Backup Failed. Later re-freeze only rewrites Backup phases, not the cluster condition, so the Healthy fixture stays broken across the documented idempotent up / live / refreeze flows.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 0eda339. Configure here.
|
Superseded by #1402, which carries the CNPG demo plus hardened freeze/live lifecycle handling, truthful invariants, review fixes, and end-to-end verification. Keeping this branch intact for history. |
1 participant
CNPG had the problem
make gitops-demoexists to solve, and no answer. RAD-318's fixtures lived in a hand-built cluster and a chat transcript; when the cluster went, so did they.make cnpg-demobuilds akindcluster in ~6 minutes with enough state to verify badge, drawer, Issues, audit and filter in one pass.What's in it
Four clusters, all 2/2 Ready, four different badges —
Healthy,WAL Archiving Failing,Unrecoverable×2. The uniform instance count is the design: every difference is invisible to counts, which is the class of bug this exists to catch. A reviewer should see four identical2/2rows and four different colours.Plus a Pooler, a
method: pluginScheduledBackup, three Backups (completed/walArchivingFailing/ an unmapped phase), and Velero + KubeBlocks CRs for the sharedbackupsandclustersplurals. Those last two are load-bearing: with one engine installed, a guard that matches everything and a guard that matches the right API group behave identically, so the collision handling can't be verified at all.The payload is the ordering
Anyone can
helm install cloudnative-pg. These are the constraints that cost hours, and every one was found by running the script rather than reasoning about it. Each fails in a way that looks like success.Freezing the operator does not hold a status. Every Postgres pod runs an instance manager that writes cluster status independently of the controller.
status.phasesurvives a scaled-to-zero operator;status.conditionsandstatus.readyInstancesare rewritten from the pods within ~10 minutes. A hand-patchedContinuousArchiving=Falsedemos fine and then heals itself.So the WAL failure is caused, not asserted — but only after the cluster is up. CNPG verifies the object store during bootstrap, so a cluster created pointing at an unroutable endpoint never becomes Ready. That still produces
ContinuousArchiving=False, which is the convincing wrong answer: the condition looks right on a cluster that is simply broken. The success check therefore requires both the condition and 2/2 Ready — because asserting the condition alone passed on the broken version, and reported success.Backups must be created after the freeze, not merely patched after it. The operator attempts a Backup, and one pointed at a cluster with no backup configuration fails and stamps
LastBackupSucceeded=Falseon it. The pristine baseline renderedBackup Failed— because of a resource attached to it, not anything wrong with it.A Pooler must not share a name with one of its cluster's Services. CNPG creates
<cluster>-rw/-ro/-r; a Pooler creates a Service of its own name.pg-healthy-rwfound a Service it didn't own and refused to reconcile forever withinvalid ownership for managed resources— no status, no condition, no event, just a CR that looks slow.Thawing is not scaling back up. The operator builds its PKI from its own webhook configurations at startup, so with them deleted it crash-loops on
ensurePKI.thawre-applies the manifest first.Timestamps are computed relative to now. A hardcoded Velero expiry rendered
Expiredinstead of24d remainingwithin months of being written.All of it is in
scripts/cnpg-demo/README.md, which is the actual deliverable — the script is just its executable form.Verified
Run end-to-end from an empty machine (
downthenup), then Radar pointed at the result:cnpgNoDeclarativeBackup: evaluated 4 / passed 1 — only the plugin-method cluster passes, which is what makes the check demonstrably plugin-aware rather than asserted to beCNPGWALArchivingFailingcritical on the healthy-looking clusterclustersrendering the generic drawermake tscandgo build ./...clean. No production code touched.Also
make cnpg-demo-liveleaves the operator running — real failovers, switchovers and backup runs, at the cost of hand-written terminal phases. The frozen path is optimised for rendering verification and is the one that's proven; v2/v3 work wants the opposite.CLAUDE.md gets a pointer in the same slot as the GitOps and Crossplane entries.
Note
Low Risk
Docs and dev/demo scripts only; no application, API, or auth changes.
Overview
Adds a
make cnpg-demoworkflow (alongsidedown,status, andcnpg-demo-live) that boots akindcluster with pinned CloudNativePG and curated fixtures so Radar’s CNPG UI can be exercised without relying on a random kube context.scripts/cnpg-demo.shorchestrates install, waits, then applies ordering-sensitive steps: induce a real WAL archiving failure on a 2/2 Ready cluster (not a status patch), scale the operator to zero and remove webhooks so terminal phases and backup statuses stick, defer CNPG Backup creation until after the freeze, and use relative timestamps for Velero status. Subcommands cover frozen vs live operator, refreeze/thaw, and inventory.YAML fixtures under
scripts/cnpg-demo/define four Postgres clusters (same Ready counts, different badges), Pooler, plugin-method ScheduledBackup, backup phase variants, and minimal Velero/KubeBlocks CRDs plus instances to exercise sharedbackups/clustersplural handling.scripts/cnpg-demo/README.mddocuments the coverage matrix and the non-obvious constraints; CLAUDE.md points visual testers at this cluster the same way as GitOps/Crossplane demos.Reviewed by Cursor Bugbot for commit 0eda339. Bugbot is set up for automated code reviews on this repo. Configure here.