chore(velero): script the demo cluster so the fixtures outlive the session - #1394
Closed
nadaverell wants to merge 4 commits into
Closed
chore(velero): script the demo cluster so the fixtures outlive the session#1394nadaverell wants to merge 4 commits into
nadaverell wants to merge 4 commits into
Conversation
…ssion The Velero fixtures existed only in one throwaway kind cluster. gitops-demo and crossplane-demo already solve this for their integrations, and CLAUDE.md gives the reason: without a demo cluster you test against whatever context happens to be current, which is usually a real cluster missing the variety that matters. Velero had the same problem and no answer. Anyone can install Velero. The payload here is the states you cannot get by installing it: All 13 Backup phases from the v1.18 enum, including the eight that must stay SILENT in Issues. A phase that starts raising is the regression the file exists to catch, so silence is coverage rather than the absence of it. The supersession series - a later success clearing an earlier failure, per schedule, with an ad-hoc failure alongside that must never be cleared. These are three files applied with a pause because ordering is by creationTimestamp and the API server assigns it. Same-second creation would decide ordering by the name tie-break instead of by time, and the test would pass for a reason unrelated to what it checks; verify asserts the timestamps are distinct. The paused-AND-invalid Schedule, and a schedule rejected for a missing storage location while its cron is valid - the counter-example that stops anyone deriving 'invalid cron' from the status and reddening the wrong cell. A restic BackupRepository, and two distinct repository statuses, since a column holding one distinct value renders no filter control at all. The real rancher/backup-restore-operator CRD rather than a mock, because the guard under test is 'select on the API group, not the plural' and a mock proves much less. Two things the header and README state up front, both found the slow way. The controller is scaled to 0 because the failure phases need real object storage; a live controller with no bucket produces one outcome, everything Failed on credentials. And none of the six Velero CRDs declare a status subresource, so a plain apply writes status - reaching for 'kubectl patch --subresource=status' fails with a confusing NotFound. Fixtures store @now±Nm tokens rather than absolute dates so the demo does not age into 'expired 8 months ago'. Verified by running reset from nothing and pointing Radar at the result: 14 issues, matching the hand-built cluster exactly, with the superseded backup absent and the ad-hoc failure present.
The wait used a pod label the chart never applies — the Velero chart labels pods name=velero, not deploy=velero — so 'kubectl wait --for=delete' matched nothing, returned instantly, and reported success. With '|| true' on the end it could not fail. The fixtures then raced a controller that was still running and would reconcile every hand-written phase away. Every run passed anyway, because scale-down finishes in a second or two and the first two fixture files buy more time than that. Passing on timing is not the same as being correct, and the failure mode it hides - mysteriously empty phases - is exactly the one the header warns about. Now polls the Deployment's own replica count, which needs no label guess and asserts the condition the whole fixture depends on, and fails loudly instead of continuing. verify checks the same thing independently, so a controller scaled back up is caught rather than silently invalidating the run. Verified with a full run from an empty cluster.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 650fac3. Configure here.
…eplica count A Deployment stops counting a pod in .status.replicas the moment it starts terminating, while the container keeps running for its grace period — and Velero sets terminationGracePeriodSeconds: 3600. So a replica count of zero says nothing about whether a controller is still reconciling, and the fixtures could still be overwritten by a pod that had an hour to live. Now waits on pod presence, with the selector read from the Deployment rather than hardcoded, so it cannot silently match nothing the way the first attempt did. Nothing is in flight worth draining, so the leftover pod is force-deleted rather than waited out for up to an hour. verify checks pod presence for the same reason. Verified with a full run from an empty cluster.
The repo is public and an outside reader cannot resolve a Linear ID. Naming the capability also makes the paragraph self-contained, which is the point of the section: it lists what data-mover support would actually require.
Contributor
Author
|
Closing in favor of the consolidated scope in #1402. After review, the synthetic Velero harness is intentionally not being added to the shared codebase; this branch remains intact as an archived fixture harness if it is useful later. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

The Velero fixtures from RAD-314 existed only in one throwaway kind cluster.
gitops-demoandcrossplane-demoalready solve this for their integrations, and CLAUDE.md gives the reason: without a demo cluster you test against whatever context happens to be current, which is usually a real cluster missing exactly the variety that matters. Velero had the same problem and no answer.The payload is the states you can't get by installing Velero
Anyone can
helm install velero. What took the time was everything below.All 13 Backup phases, including the eight that must stay silent in Issues. A phase that starts raising is the regression this file exists to catch — silence here is coverage, not its absence.
The supersession series — a later success clearing an earlier failure within one schedule, plus an ad-hoc failure that must never be cleared. Three separate files applied with a pause, because ordering is by
creationTimestampand the API server assigns it: same-second creation decides ordering by the name tie-break instead of by time, and the test then passes for a reason unrelated to what it checks.verifyasserts the timestamps are distinct and correctly ordered.The paused-AND-invalid Schedule, and a schedule rejected for a missing storage location while its cron is valid — the counter-example that stops anyone deriving "invalid cron" from the status and reddening the wrong cell.
A
resticBackupRepository, and two distinct repository statuses, since a column holding one distinct value renders no filter control at all and can't exercise the filter.The real
rancher/backup-restore-operatorCRD, vendored rather than mocked, because the guard under test is "select on the API group, not the plural" and a mock proves much less.Two gotchas stated up front, both found the slow way
Failedon credentials, the least interesting row in the table.kubectl applywritesstatushere. None of the six Velero CRDs declare a status subresource, so the familiar "apply drops status" rule doesn't apply, andkubectl patch --subresource=statusfails against them with a confusingNotFound. I built the patching layer first and deleted it.Fixtures store
@now±Nmtokens rather than absolute dates, so the demo doesn't age into "expired 8 months ago" and leave the Expires / Last Backup / Age columns demoing nothing.Deliberately not covered
Anything needing a live controller — data mover (
DataUpload/DataDownload, wanted by RAD-316), real progress counters, and per-item error detail, which lives in an object-storage artifact behind aDownloadRequest. RAD-316 would need MinIO and a snapshot-capable CSI driver and the node agent and a workload with a bound PVC: a different and much more fragile fixture, not a flag on this one. Hand-fakingDataUploadCRs would test our renderer against our own guess at the shape, and the shape is the thing in doubt. The README says so rather than pretending.Testing
velero-demo.sh resetfrom nothing, then Radar pointed at the result: 14 issues, matching the hand-built cluster exactly — 12backup_failed, 2backup_target_unavailable. The superseded backup is absent, the ad-hoc failure present, and all eight silent phases silent. The five in-script assertions pass. Alsobash -nclean, and the script is idempotent (re-run against an existing cluster reuses it).Note
Low Risk
Dev-only scripts and fixtures; no production app, auth, or runtime behavior changes.
Overview
Adds a Velero demo bootstrap parallel to
crossplane-demo/gitops-demo:make velero-demo(plusdown,status,verify) drivesscripts/velero-demo.sh, which creates a kind cluster, installs pinned Velero 1.18.1 via Helm, scales the controller to 0, and applies YAML fixtures with@now±Nmtimestamps expanded at apply time.The fixtures encode states that are hard to get on kind without real object storage: all 13 Backup phases (including phases that must stay silent in Issues), supersession (failed then completed nightly backups plus a non-superseded ad-hoc failure), varied Schedules (paused+invalid, errors with no phase), Unavailable BSL, restic and multi-status BackupRepositories, restores across phases, and the Rancher
restores.resources.cattle.ioplural collision via a vendored CRD + sample CR.verifyre-asserts controller absence, phase coverage, supersessioncreationTimestampordering, collision CR, and restic repo without reapplying. Documentation inscripts/velero-demo/README.mdexplains the scaled-to-zero trick, status-on-apply behavior, coverage matrix, and explicit limits (no live controller / data mover / real buckets).Reviewed by Cursor Bugbot for commit 60f24ee. Bugbot is set up for automated code reviews on this repo. Configure here.