Skip to content

feat(azure): add azure_velero_backups fetcher - #376

Merged
Controgon merged 4 commits into
mainfrom
feat/azure-velero-backups
Nov 17, 2026
Merged

Controgon merged 4 commits into
mainfrom
feat/azure-velero-backups

Conversation

@bruceweygandt

@bruceweygandt bruceweygandt commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds azure_velero_backups, an Azure evidence fetcher for Velero. It reads Velero's backup storage locations in Azure Blob Storage using Azure APIs only (no Kubernetes access), and reports whether every expected backup is happening (including schedules that never produced one, missed daily runs and the recovery point objective), whether backups succeed, whether expired backups are removed, whether restores complete without errors, and whether the backup storage is geo-replicated and encrypted with a customer-managed key.

Evidence set: EVD-AZURE-VELERO-BACKUPS (Azure Velero Backups and Restores). KSIs: KSI-RPL-ABO, KSI-RPL-TRC, KSI-RPL-ARP, KSI-SVC-SIN, KSI-SVC-RUD.

How it collects

  • Resource Manager: storage accounts in the subscription (blob endpoint, replication SKU, encryption key source) and their blob containers, to find Velero containers by name (velero by default), or the explicit account/container[/prefix] locations in AZURE_VELERO_BACKUP_LOCATIONS.
  • Blob data plane: List Blobs on backups/ and restores/, then only backups/<name>/velero-backup.json and restores/<name>/restore-<name>-results.gz. Tarballs, logs and other Velero files are never downloaded.

Access: Reader on the subscription, plus Storage Blob Data Reader on each Velero container.

Blob read boundary

This is the first fetcher to read blob contents. Strict subscription mode's validate_read_url requires the request host to equal the token resource host, which a blob endpoint never does, so blob reads have their own boundary:

  • only blob endpoints Resource Manager returned for in-scope storage accounts in the target subscription;
  • HTTPS only, no credentials or fragment in the URL, path limited to the configured container;
  • redirects never followed;
  • an Entra token for https://storage.azure.com (valid in every cloud), with the shared retry statuses and retry_wait.

Failure handling

Uses run_subscription, so a failed read is recorded in api_failures, the run exits non-zero and report_failure carries the reason. Counts are null, not 0, when discovery did not finish or any location could not be read: a missing data role reads as unknown, never as "no backups".

Testing

  • 23 tests in tests/test_azure_velero_backups.py: location parsing, backup and restore judgement, null-not-zero summaries, the blob read boundary (untrusted host and plain HTTP refused), a full main() run, a missing data role (403), an explicit location outside the subscription, strict subscription mode, and assert_summary_documented.
  • Real Velero 1.16 objects in tests/fixtures/velero/, captured from a sandbox AKS cluster: two backups (one with a 5-minute TTL, expired but not yet garbage-collected) and one restore, plus the full 29-blob listing. The fetcher reads only the three records it needs and judges them correctly, including a clean restore's empty "errors": {} and absent volume-snapshot counters. With daily:24 expected, the same real data shows the sandbox's daily schedule had produced no backup yet: 7 missed runs and past the recovery point objective. No identifying data in the fixtures.
  • paramify list discovers it; the fake-credential smoke test exits 1 with a valid JSON file, auth_failed status and null counts.
  • test_contracts.py, test_category_requires.py and all Azure and Entra tests pass (5,421). No new dependencies.

Not yet run end to end against a live tenant with the data role: the sandbox credential could not be granted Storage Blob Data Reader, so the real objects were captured with the account key and replayed through the fetcher's collection.

Settings

container_names (default velero), backup_locations, success_window_hours (default 26), expiry_grace_hours (default 24), expected_schedules, rpo_hours, missed_run_lookback_days (default 7), restore_test_interval_days and resource_exclusions. Expected schedules, the recovery point objective and the restore-test interval have no defaults: until an offering configures them, their counts are null rather than judged against values nobody adopted.

Expected schedules, missed runs and the recovery point objective

Velero does not write schedules to backup storage, so a schedule that has never produced a backup cannot be seen there. expected_schedules (name:hours, such as daily:24) lists the schedules that must be producing backups. For each one the fetcher reports whether it has any backup, how many of its periods over missed_run_lookback_days had no Completed backup (missed runs), and whether its newest Completed backup is older than rpo_hours; a schedule with none counts as exceeding it. A malformed list fails the run and leaves the counts null. A restore whose results file has no readable date counts in restores_date_unknown and never as recent.

Limits

Velero does not write schedules or restore status to object storage, so schedules come from each backup's velero.io/schedule-name label, and a restore is judged by its results file.

@wiz-connection

wiz-connection Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Wiz Scan Summary

Scanner Findings
Vulnerability Finding Vulnerabilities -
Data Finding Sensitive Data -
Secret Finding Secrets -
IaC Misconfiguration IaC Misconfigurations -
SAST Finding SAST Findings -
Software Management Finding Software Management Findings -
Total -

View scan details in Wiz

To detect these findings earlier in the dev lifecycle, try the Wiz Code extension for VS Code, JetBrains, or Visual Studio.

@bruceweygandt
bruceweygandt force-pushed the feat/azure-velero-backups branch from 6ad3a8f to cd2568c Compare November 17, 2026 03:58
@bruceweygandt
bruceweygandt force-pushed the feat/azure-velero-backups branch from cd2568c to f6b92e1 Compare November 17, 2026 17:37

@Controgon Controgon left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a careful fetcher. The blob read boundary is tight. The summaries stay null when a read is incomplete, and the real Velero fixtures are a good base for the tests. I ran the tests (23 passed), the contract tests and paramify list offline, and all pass. Two problems in the missed-run count need a fix before merge, because a validator will assert on that number.

1. Must fix: backups that Velero deleted on schedule count as missed runs

fetchers/azure/velero_backups/fetcher.py:90-94

Problem. schedule_check looks for a Completed backup in every period of missed_run_lookback_days. Velero deletes each backup when its TTL ends. If the TTL is shorter than the lookback, the older periods have no backup, and each one counts as a missed run.

Why this is a problem. I called schedule_check with synthetic backups where every run succeeded:

  • an hourly schedule with a 24-hour TTL gives missed_runs=144 of 168 periods;
  • a daily schedule with a 72-hour TTL gives missed_runs=4.

A validator that checks missed_scheduled_runs == 0 fails on every run for these schedules.

Fix. Do not count periods that are older than the schedule's retention. Each backup has expiration, so the TTL is expiration minus the backup's start or completion time. Check min(lookback, TTL) worth of periods.

2. Must fix: small changes in completion time cause false missed runs

fetchers/azure/velero_backups/fetcher.py:93-94

Problem. The periods are fixed windows that count back from the current time. If the current time is between two days' completion times, one window contains two backups and the next window contains none.

Why this is a problem. Take a daily schedule that ran every day for 8 days, where yesterday's backup finished 5 minutes later than usual (02:05 instead of 02:00). A fetcher run at 02:03 reports missed_runs=1. Backup duration changes every day, so the count changes between 0 and 1 from one run to the next.

Fix. Count from the gaps between consecutive Completed backups, and from the newest one to now. For each gap, add max(0, floor((gap - grace) / interval)), with a small grace such as 10% of the interval. Please add a test for this case and a test for item 1.

Smaller items

  • fetcher.py:177. One success_window_hours value applies to all schedules. With the default of 26 hours, a weekly schedule is always outside the window. When expected_schedules gives an interval, please use that interval plus some slack for that schedule.
  • fetcher.py:9, fetcher.py:296. About the Wiz SAST comments on xml.etree: the risk is low, because the host is fixed to blob endpoints from Resource Manager and redirects are refused. Please either use defusedxml and add it to the Azure dependencies, or close the comments with an ignore reason.
  • tests/test_contracts.py. The instructions pass the plain-text format check. But azure/velero_backups is not in TEXT_FORMAT_FETCHERS, so CI does not check them. Please add it.
  • fetcher.py:279-284. BlobReader._get retries on HTTP status only. The shared _send in azure_rest.py also retries a dropped connection and a GET timeout. Without this, one dropped connection fails the run.
  • fetcher.yaml:123. A storage account firewall also blocks the blob reads, and the result looks the same as a missing data role. Please mention network access in the Access line.
  • fetcher.py:268, fetcher.py:278. Import time at the top of the file instead of __import__("time") and a local import.

…-schedule windows, unresolved failures, defusedxml, blob retries
@bruceweygandt

Copy link
Copy Markdown
Contributor Author

Thanks for the careful review. All items are fixed in the latest commit.

1. Deleted backups counted as missed runs. Missed runs are now checked only over min(missed_run_lookback_days, retention), where retention is the shortest expiration minus completion among the schedule's backups. Each schedule check reports retention_hours and hours_checked_for_missed_runs. Your two cases (hourly with a 24-hour TTL, daily with a 72-hour TTL) are tests and return 0.

2. Small changes in completion time. Missed runs now come from the gaps between consecutive Completed backups and from the newest one to now: each gap adds max(0, floor((gap - grace) / interval)), with grace at 10% of the interval. The first gap starts at the newest Completed backup at or before the window start. Your 02:05 vs 02:00 case is a test and returns 0, and a test confirms a real gap still counts every missed interval.

Smaller items

  • Success window per schedule: a schedule listed in expected_schedules is judged against its interval plus 10% (at least 2 hours), so daily:24 keeps 26 hours and weekly:168 gets 184. success_window_hours applies only to unlisted schedules.
  • Wiz XXE: Blob list XML is parsed with defusedxml, added to the azure extra. A test confirms an entity declaration is refused.
  • TEXT_FORMAT_FETCHERS: azure/velero_backups added; the format checks pass.
  • Retries: BlobReader._get retries a dropped connection and a GET timeout, bounded as in azure_rest._send. Tested.
  • Access line: now names the storage account firewall alongside the data role.
  • Imports: time is imported at the top.

Also added (from DU's review of the validators). Failures are now told apart without losing history. backups_failed, backups_partially_failed, backups_volume_snapshots_incomplete and restores_with_errors still count every record. New counts give the current ones: backups_failed_unresolved (no later Completed backup of the same schedule), backups_volume_snapshots_incomplete_unresolved (no later Completed backup with every snapshot complete) and restores_with_errors_unresolved (no later clean restore). A new optional deliberate_failure_tests setting takes name patterns for planned failure tests; matching records are counted in deliberate_failure_tests and never as current failures. A failure without a readable time is never treated as resolved.

Tests: 31 passed (8 new). Contract tests: 884 passed. Ruff is clean on the changed files.

@Controgon Controgon left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes still needed: two small fixes. CI fails on an undeclared package, and the new missed-run count can include runs from before the lookback.

Fixed

    1. Backups that Velero deleted on schedule: fixed. Missed runs are checked over min(lookback, retention), and both cases from the review are tests.
    1. Completion-time drift: fixed. Missed runs now come from the gaps between backups, with 10% grace. The 02:05 case is a test, and so is a real gap.
  • Success window for each schedule: fixed. A weekly schedule gets 184 hours.
  • defusedxml: used in the code and added to the azure extra, with an entity test. See item 1 below.
  • TEXT_FORMAT_FETCHERS, retries on a dropped connection and on a timeout, the Access line, and the time import: fixed.

Still to fix

1. Must fix: CI fails because defusedxml is not declared in the Azure category

fetchers/_categories/azure.yaml:77

Problem. tests/test_category_requires.py::test_python_imports_are_declared_or_core[azure] fails on this head. The CI test job fails on every Python version for this reason: 1 failed, 9924 passed.

Why this is a problem. On a runner without defusedxml, paramify doctor reports the Azure category as ready, and then the Velero run fails with an ImportError.

Fix. Add - defusedxml to requires.python_packages.

2. Must fix: a gap that starts before the lookback adds runs that were due before the lookback

fetchers/azure/velero_backups/fetcher.py:126-128

Problem. The first gap starts at the newest backup before the window start. That is correct, because otherwise a run due just inside the window would be lost. But every missed run in that gap is counted, including the runs due before the window started.

Why this is a problem. Take a daily schedule with the default 30-day TTL and a 7-day lookback. Backups stopped from day -20 to day -7 and have succeeded every day since day -6. The result is periods_checked=7 and missed_runs=13, although no run due in the window was missed. For an outage from day -10 to day -3, the result is 7 instead of 4. A validator that asserts missed_scheduled_runs == 0 keeps failing for as long as the gap lasted, after the lookback has already passed it.

Fix. In the first gap, subtract the runs that were due before the window start: max(0, floor((gap - grace) / step) - floor((window_start - first) / step)). I checked this change. It gives 0 and 4 for the two cases above, and the existing tests still pass. Please add a test with a gap that crosses the window start.

Nit

  • fetcher.py:248, fetcher.py:254. A backup without a schedule label goes under the key None. So a failed manual backup counts as resolved after any later manual backup, even one of other namespaces, and backups_failed_unresolved drops it. Please either never mark an unscheduled failure as resolved, or say in the backups_failed_unresolved line that any later manual backup resolves it.

Before merge, please merge main. The branch now conflicts in CHANGELOG.md and tests/test_contracts.py.

…kups

# Conflicts:
#	CHANGELOG.md
#	tests/test_contracts.py
…side the lookback; never resolve unscheduled failures
@bruceweygandt

Copy link
Copy Markdown
Contributor Author

Thanks. Both must-fix items and the nit are in the latest commits, and main is merged.

1. defusedxml not declared in the Azure category. Added to requires.python_packages in fetchers/_categories/azure.yaml. test_category_requires.py passes.

2. A gap crossing the window start counted runs due before the lookback. Applied your formula: in the first gap, the runs due before the window start are subtracted. Your two cases are a test: backups stopped from day -20 to day -7 now give 0 missed runs, and an outage from day -10 to day -3 gives 4. Both cases fail on the previous code.

Nit: unscheduled failures. A backup with no schedule label is never marked resolved, since a later manual backup may cover other namespaces. It stays in backups_failed_unresolved (and the snapshot count) until it leaves storage, and both field descriptions say so. Tested, along with a scheduled failure that a later Completed backup does resolve.

Merging main. CHANGELOG.md keeps both entries. For tests/test_contracts.py I took main's version: TEXT_FORMAT_FETCHERS became the shrink-only TEXT_FORMAT_PENDING, so the fetcher is now checked automatically and its old allow-list line is gone.

Tests: 33 passed (3 new). Contract, category and all Azure and Entra tests: 6,502 passed. The Velero tests also pass with no Azure SDK installed.

@Controgon
Controgon merged commit 81fabf8 into main Nov 17, 2026
20 checks passed
@Controgon
Controgon deleted the feat/azure-velero-backups branch November 17, 2026 22:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants