Repository navigation
feat(azure): add azure_velero_backups fetcher - #376
Conversation
Wiz Scan Summary
To detect these findings earlier in the dev lifecycle, try the Wiz Code extension for VS Code, JetBrains, or Visual Studio. |
6ad3a8f to
cd2568c
Compare
cd2568c to
f6b92e1
Compare
Controgon
left a comment
There was a problem hiding this comment.
This is a careful fetcher. The blob read boundary is tight. The summaries stay null when a read is incomplete, and the real Velero fixtures are a good base for the tests. I ran the tests (23 passed), the contract tests and paramify list offline, and all pass. Two problems in the missed-run count need a fix before merge, because a validator will assert on that number.
1. Must fix: backups that Velero deleted on schedule count as missed runs
fetchers/azure/velero_backups/fetcher.py:90-94
Problem. schedule_check looks for a Completed backup in every period of missed_run_lookback_days. Velero deletes each backup when its TTL ends. If the TTL is shorter than the lookback, the older periods have no backup, and each one counts as a missed run.
Why this is a problem. I called schedule_check with synthetic backups where every run succeeded:
- an hourly schedule with a 24-hour TTL gives
missed_runs=144of 168 periods; - a daily schedule with a 72-hour TTL gives
missed_runs=4.
A validator that checks missed_scheduled_runs == 0 fails on every run for these schedules.
Fix. Do not count periods that are older than the schedule's retention. Each backup has expiration, so the TTL is expiration minus the backup's start or completion time. Check min(lookback, TTL) worth of periods.
2. Must fix: small changes in completion time cause false missed runs
fetchers/azure/velero_backups/fetcher.py:93-94
Problem. The periods are fixed windows that count back from the current time. If the current time is between two days' completion times, one window contains two backups and the next window contains none.
Why this is a problem. Take a daily schedule that ran every day for 8 days, where yesterday's backup finished 5 minutes later than usual (02:05 instead of 02:00). A fetcher run at 02:03 reports missed_runs=1. Backup duration changes every day, so the count changes between 0 and 1 from one run to the next.
Fix. Count from the gaps between consecutive Completed backups, and from the newest one to now. For each gap, add max(0, floor((gap - grace) / interval)), with a small grace such as 10% of the interval. Please add a test for this case and a test for item 1.
Smaller items
fetcher.py:177. Onesuccess_window_hoursvalue applies to all schedules. With the default of 26 hours, a weekly schedule is always outside the window. Whenexpected_schedulesgives an interval, please use that interval plus some slack for that schedule.fetcher.py:9,fetcher.py:296. About the Wiz SAST comments onxml.etree: the risk is low, because the host is fixed to blob endpoints from Resource Manager and redirects are refused. Please either usedefusedxmland add it to the Azure dependencies, or close the comments with an ignore reason.tests/test_contracts.py. The instructions pass the plain-text format check. Butazure/velero_backupsis not inTEXT_FORMAT_FETCHERS, so CI does not check them. Please add it.fetcher.py:279-284.BlobReader._getretries on HTTP status only. The shared_sendinazure_rest.pyalso retries a dropped connection and a GET timeout. Without this, one dropped connection fails the run.fetcher.yaml:123. A storage account firewall also blocks the blob reads, and the result looks the same as a missing data role. Please mention network access in the Access line.fetcher.py:268,fetcher.py:278. Importtimeat the top of the file instead of__import__("time")and a local import.
…-schedule windows, unresolved failures, defusedxml, blob retries
|
Thanks for the careful review. All items are fixed in the latest commit. 1. Deleted backups counted as missed runs. Missed runs are now checked only over 2. Small changes in completion time. Missed runs now come from the gaps between consecutive Completed backups and from the newest one to now: each gap adds Smaller items
Also added (from DU's review of the validators). Failures are now told apart without losing history. Tests: 31 passed (8 new). Contract tests: 884 passed. Ruff is clean on the changed files. |
Controgon
left a comment
There was a problem hiding this comment.
Changes still needed: two small fixes. CI fails on an undeclared package, and the new missed-run count can include runs from before the lookback.
Fixed
-
- Backups that Velero deleted on schedule: fixed. Missed runs are checked over
min(lookback, retention), and both cases from the review are tests.
- Backups that Velero deleted on schedule: fixed. Missed runs are checked over
-
- Completion-time drift: fixed. Missed runs now come from the gaps between backups, with 10% grace. The 02:05 case is a test, and so is a real gap.
- Success window for each schedule: fixed. A weekly schedule gets 184 hours.
defusedxml: used in the code and added to theazureextra, with an entity test. See item 1 below.TEXT_FORMAT_FETCHERS, retries on a dropped connection and on a timeout, the Access line, and thetimeimport: fixed.
Still to fix
1. Must fix: CI fails because defusedxml is not declared in the Azure category
fetchers/_categories/azure.yaml:77
Problem. tests/test_category_requires.py::test_python_imports_are_declared_or_core[azure] fails on this head. The CI test job fails on every Python version for this reason: 1 failed, 9924 passed.
Why this is a problem. On a runner without defusedxml, paramify doctor reports the Azure category as ready, and then the Velero run fails with an ImportError.
Fix. Add - defusedxml to requires.python_packages.
2. Must fix: a gap that starts before the lookback adds runs that were due before the lookback
fetchers/azure/velero_backups/fetcher.py:126-128
Problem. The first gap starts at the newest backup before the window start. That is correct, because otherwise a run due just inside the window would be lost. But every missed run in that gap is counted, including the runs due before the window started.
Why this is a problem. Take a daily schedule with the default 30-day TTL and a 7-day lookback. Backups stopped from day -20 to day -7 and have succeeded every day since day -6. The result is periods_checked=7 and missed_runs=13, although no run due in the window was missed. For an outage from day -10 to day -3, the result is 7 instead of 4. A validator that asserts missed_scheduled_runs == 0 keeps failing for as long as the gap lasted, after the lookback has already passed it.
Fix. In the first gap, subtract the runs that were due before the window start: max(0, floor((gap - grace) / step) - floor((window_start - first) / step)). I checked this change. It gives 0 and 4 for the two cases above, and the existing tests still pass. Please add a test with a gap that crosses the window start.
Nit
fetcher.py:248,fetcher.py:254. A backup without a schedule label goes under the keyNone. So a failed manual backup counts as resolved after any later manual backup, even one of other namespaces, andbackups_failed_unresolveddrops it. Please either never mark an unscheduled failure as resolved, or say in thebackups_failed_unresolvedline that any later manual backup resolves it.
Before merge, please merge main. The branch now conflicts in CHANGELOG.md and tests/test_contracts.py.
…kups # Conflicts: # CHANGELOG.md # tests/test_contracts.py
…side the lookback; never resolve unscheduled failures
|
Thanks. Both must-fix items and the nit are in the latest commits, and 1. 2. A gap crossing the window start counted runs due before the lookback. Applied your formula: in the first gap, the runs due before the window start are subtracted. Your two cases are a test: backups stopped from day -20 to day -7 now give 0 missed runs, and an outage from day -10 to day -3 gives 4. Both cases fail on the previous code. Nit: unscheduled failures. A backup with no schedule label is never marked resolved, since a later manual backup may cover other namespaces. It stays in Merging Tests: 33 passed (3 new). Contract, category and all Azure and Entra tests: 6,502 passed. The Velero tests also pass with no Azure SDK installed. |
Summary
Adds
azure_velero_backups, an Azure evidence fetcher for Velero. It reads Velero's backup storage locations in Azure Blob Storage using Azure APIs only (no Kubernetes access), and reports whether every expected backup is happening (including schedules that never produced one, missed daily runs and the recovery point objective), whether backups succeed, whether expired backups are removed, whether restores complete without errors, and whether the backup storage is geo-replicated and encrypted with a customer-managed key.Evidence set:
EVD-AZURE-VELERO-BACKUPS(Azure Velero Backups and Restores). KSIs: KSI-RPL-ABO, KSI-RPL-TRC, KSI-RPL-ARP, KSI-SVC-SIN, KSI-SVC-RUD.How it collects
veleroby default), or the explicitaccount/container[/prefix]locations inAZURE_VELERO_BACKUP_LOCATIONS.backups/andrestores/, then onlybackups/<name>/velero-backup.jsonandrestores/<name>/restore-<name>-results.gz. Tarballs, logs and other Velero files are never downloaded.Access: Reader on the subscription, plus Storage Blob Data Reader on each Velero container.
Blob read boundary
This is the first fetcher to read blob contents. Strict subscription mode's
validate_read_urlrequires the request host to equal the token resource host, which a blob endpoint never does, so blob reads have their own boundary:https://storage.azure.com(valid in every cloud), with the shared retry statuses andretry_wait.Failure handling
Uses
run_subscription, so a failed read is recorded inapi_failures, the run exits non-zero andreport_failurecarries the reason. Counts arenull, not0, when discovery did not finish or any location could not be read: a missing data role reads as unknown, never as "no backups".Testing
tests/test_azure_velero_backups.py: location parsing, backup and restore judgement, null-not-zero summaries, the blob read boundary (untrusted host and plain HTTP refused), a fullmain()run, a missing data role (403), an explicit location outside the subscription, strict subscription mode, andassert_summary_documented.tests/fixtures/velero/, captured from a sandbox AKS cluster: two backups (one with a 5-minute TTL, expired but not yet garbage-collected) and one restore, plus the full 29-blob listing. The fetcher reads only the three records it needs and judges them correctly, including a clean restore's empty"errors": {}and absent volume-snapshot counters. Withdaily:24expected, the same real data shows the sandbox's daily schedule had produced no backup yet: 7 missed runs and past the recovery point objective. No identifying data in the fixtures.paramify listdiscovers it; the fake-credential smoke test exits 1 with a valid JSON file,auth_failedstatus and null counts.test_contracts.py,test_category_requires.pyand all Azure and Entra tests pass (5,421). No new dependencies.Not yet run end to end against a live tenant with the data role: the sandbox credential could not be granted Storage Blob Data Reader, so the real objects were captured with the account key and replayed through the fetcher's collection.
Settings
container_names(defaultvelero),backup_locations,success_window_hours(default 26),expiry_grace_hours(default 24),expected_schedules,rpo_hours,missed_run_lookback_days(default 7),restore_test_interval_daysandresource_exclusions. Expected schedules, the recovery point objective and the restore-test interval have no defaults: until an offering configures them, their counts are null rather than judged against values nobody adopted.Expected schedules, missed runs and the recovery point objective
Velero does not write schedules to backup storage, so a schedule that has never produced a backup cannot be seen there.
expected_schedules(name:hours, such asdaily:24) lists the schedules that must be producing backups. For each one the fetcher reports whether it has any backup, how many of its periods overmissed_run_lookback_dayshad no Completed backup (missed runs), and whether its newest Completed backup is older thanrpo_hours; a schedule with none counts as exceeding it. A malformed list fails the run and leaves the counts null. A restore whose results file has no readable date counts inrestores_date_unknownand never as recent.Limits
Velero does not write schedules or restore status to object storage, so schedules come from each backup's
velero.io/schedule-namelabel, and a restore is judged by its results file.