Skip to content

chore(velero): script the demo cluster so the fixtures outlive the session - #1394

Closed
nadaverell wants to merge 4 commits into
mainfrom
nadav/velero-demo-fixture
Closed

chore(velero): script the demo cluster so the fixtures outlive the session#1394
nadaverell wants to merge 4 commits into
mainfrom
nadav/velero-demo-fixture

Conversation

@nadaverell

@nadaverell nadaverell commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

The Velero fixtures from RAD-314 existed only in one throwaway kind cluster. gitops-demo and crossplane-demo already solve this for their integrations, and CLAUDE.md gives the reason: without a demo cluster you test against whatever context happens to be current, which is usually a real cluster missing exactly the variety that matters. Velero had the same problem and no answer.

make velero-demo          # ~2 min, offline after the chart pull
make velero-demo-status   # inventory
make velero-demo-verify   # re-assert without reapplying
make velero-demo-down

The payload is the states you can't get by installing Velero

Anyone can helm install velero. What took the time was everything below.

All 13 Backup phases, including the eight that must stay silent in Issues. A phase that starts raising is the regression this file exists to catch — silence here is coverage, not its absence.

The supersession series — a later success clearing an earlier failure within one schedule, plus an ad-hoc failure that must never be cleared. Three separate files applied with a pause, because ordering is by creationTimestamp and the API server assigns it: same-second creation decides ordering by the name tie-break instead of by time, and the test then passes for a reason unrelated to what it checks. verify asserts the timestamps are distinct and correctly ordered.

The paused-AND-invalid Schedule, and a schedule rejected for a missing storage location while its cron is valid — the counter-example that stops anyone deriving "invalid cron" from the status and reddening the wrong cell.

A restic BackupRepository, and two distinct repository statuses, since a column holding one distinct value renders no filter control at all and can't exercise the filter.

The real rancher/backup-restore-operator CRD, vendored rather than mocked, because the guard under test is "select on the API group, not the plural" and a mock proves much less.

Two gotchas stated up front, both found the slow way

  • The controller is scaled to 0 on purpose. The failure phases need real object storage; a live controller with no bucket produces exactly one outcome — everything Failed on credentials, the least interesting row in the table.
  • A plain kubectl apply writes status here. None of the six Velero CRDs declare a status subresource, so the familiar "apply drops status" rule doesn't apply, and kubectl patch --subresource=status fails against them with a confusing NotFound. I built the patching layer first and deleted it.

Fixtures store @now±Nm tokens rather than absolute dates, so the demo doesn't age into "expired 8 months ago" and leave the Expires / Last Backup / Age columns demoing nothing.

Deliberately not covered

Anything needing a live controller — data mover (DataUpload/DataDownload, wanted by RAD-316), real progress counters, and per-item error detail, which lives in an object-storage artifact behind a DownloadRequest. RAD-316 would need MinIO and a snapshot-capable CSI driver and the node agent and a workload with a bound PVC: a different and much more fragile fixture, not a flag on this one. Hand-faking DataUpload CRs would test our renderer against our own guess at the shape, and the shape is the thing in doubt. The README says so rather than pretending.

Testing

velero-demo.sh reset from nothing, then Radar pointed at the result: 14 issues, matching the hand-built cluster exactly — 12 backup_failed, 2 backup_target_unavailable. The superseded backup is absent, the ad-hoc failure present, and all eight silent phases silent. The five in-script assertions pass. Also bash -n clean, and the script is idempotent (re-run against an existing cluster reuses it).


Note

Low Risk
Dev-only scripts and fixtures; no production app, auth, or runtime behavior changes.

Overview
Adds a Velero demo bootstrap parallel to crossplane-demo / gitops-demo: make velero-demo (plus down, status, verify) drives scripts/velero-demo.sh, which creates a kind cluster, installs pinned Velero 1.18.1 via Helm, scales the controller to 0, and applies YAML fixtures with @now±Nm timestamps expanded at apply time.

The fixtures encode states that are hard to get on kind without real object storage: all 13 Backup phases (including phases that must stay silent in Issues), supersession (failed then completed nightly backups plus a non-superseded ad-hoc failure), varied Schedules (paused+invalid, errors with no phase), Unavailable BSL, restic and multi-status BackupRepositories, restores across phases, and the Rancher restores.resources.cattle.io plural collision via a vendored CRD + sample CR.

verify re-asserts controller absence, phase coverage, supersession creationTimestamp ordering, collision CR, and restic repo without reapplying. Documentation in scripts/velero-demo/README.md explains the scaled-to-zero trick, status-on-apply behavior, coverage matrix, and explicit limits (no live controller / data mover / real buckets).

Reviewed by Cursor Bugbot for commit 60f24ee. Bugbot is set up for automated code reviews on this repo. Configure here.

…ssion

The Velero fixtures existed only in one throwaway kind cluster. gitops-demo
and crossplane-demo already solve this for their integrations, and CLAUDE.md
gives the reason: without a demo cluster you test against whatever context
happens to be current, which is usually a real cluster missing the variety
that matters. Velero had the same problem and no answer.

Anyone can install Velero. The payload here is the states you cannot get by
installing it:

All 13 Backup phases from the v1.18 enum, including the eight that must stay
SILENT in Issues. A phase that starts raising is the regression the file
exists to catch, so silence is coverage rather than the absence of it.

The supersession series - a later success clearing an earlier failure, per
schedule, with an ad-hoc failure alongside that must never be cleared. These
are three files applied with a pause because ordering is by creationTimestamp
and the API server assigns it. Same-second creation would decide ordering by
the name tie-break instead of by time, and the test would pass for a reason
unrelated to what it checks; verify asserts the timestamps are distinct.

The paused-AND-invalid Schedule, and a schedule rejected for a missing
storage location while its cron is valid - the counter-example that stops
anyone deriving 'invalid cron' from the status and reddening the wrong cell.

A restic BackupRepository, and two distinct repository statuses, since a
column holding one distinct value renders no filter control at all.

The real rancher/backup-restore-operator CRD rather than a mock, because the
guard under test is 'select on the API group, not the plural' and a mock
proves much less.

Two things the header and README state up front, both found the slow way.
The controller is scaled to 0 because the failure phases need real object
storage; a live controller with no bucket produces one outcome, everything
Failed on credentials. And none of the six Velero CRDs declare a status
subresource, so a plain apply writes status - reaching for
'kubectl patch --subresource=status' fails with a confusing NotFound.

Fixtures store @now±Nm tokens rather than absolute dates so the demo does not
age into 'expired 8 months ago'.

Verified by running reset from nothing and pointing Radar at the result: 14
issues, matching the hand-built cluster exactly, with the superseded backup
absent and the ad-hoc failure present.
@nadaverell
nadaverell requested a review from hisco as a code owner August 9, 2026 08:28
Comment thread scripts/velero-demo.sh Outdated
The wait used a pod label the chart never applies — the Velero chart labels
pods name=velero, not deploy=velero — so 'kubectl wait --for=delete' matched
nothing, returned instantly, and reported success. With '|| true' on the end
it could not fail. The fixtures then raced a controller that was still
running and would reconcile every hand-written phase away.

Every run passed anyway, because scale-down finishes in a second or two and
the first two fixture files buy more time than that. Passing on timing is not
the same as being correct, and the failure mode it hides - mysteriously empty
phases - is exactly the one the header warns about.

Now polls the Deployment's own replica count, which needs no label guess and
asserts the condition the whole fixture depends on, and fails loudly instead
of continuing. verify checks the same thing independently, so a controller
scaled back up is caught rather than silently invalidating the run.

Verified with a full run from an empty cluster.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 650fac3. Configure here.

Comment thread scripts/velero-demo.sh Outdated
…eplica count

A Deployment stops counting a pod in .status.replicas the moment it starts
terminating, while the container keeps running for its grace period — and
Velero sets terminationGracePeriodSeconds: 3600. So a replica count of zero
says nothing about whether a controller is still reconciling, and the
fixtures could still be overwritten by a pod that had an hour to live.

Now waits on pod presence, with the selector read from the Deployment rather
than hardcoded, so it cannot silently match nothing the way the first
attempt did. Nothing is in flight worth draining, so the leftover pod is
force-deleted rather than waited out for up to an hour.

verify checks pod presence for the same reason.

Verified with a full run from an empty cluster.
The repo is public and an outside reader cannot resolve a Linear ID. Naming
the capability also makes the paragraph self-contained, which is the point of
the section: it lists what data-mover support would actually require.
@nadaverell

Copy link
Copy Markdown
Contributor Author

Closing in favor of the consolidated scope in #1402. After review, the synthetic Velero harness is intentionally not being added to the shared codebase; this branch remains intact as an archived fixture harness if it is useful later.

@nadaverell nadaverell closed this Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant