Skip to content

fix(azure_velero_backups): resolve failures only by a later success or their own retry - #391

Open
bruceweygandt wants to merge 2 commits into
mainfrom
fde-1056
Open

bruceweygandt wants to merge 2 commits into
mainfrom
fde-1056

Conversation

@bruceweygandt

Copy link
Copy Markdown
Contributor

Summary

Changes how azure_velero_backups decides that a failure is resolved, so that nothing is hidden by name and no unrelated success clears a failure. Follows review feedback after #376 merged.

  • Removed deliberate_failure_tests. Matching a name pattern excluded failed backups and restores from the unresolved counts. Validation tests often carry a label or name prefix that marks a test, not an expected failure, so excluding by name or label could hide a real failure.
  • New retry_suffix (unset by default). When set, a failed backup or a restore with errors is resolved by its own retry: <name><suffix> finishing later without failing (every attempted snapshot, for a snapshot shortfall). A failed retry, or none, leaves the failure current. failures_resolved_by_retry counts these.
  • A failed restore resolves only through its own retry. Before, any later clean restore resolved every earlier failed restore. Backup storage doesn't record which backup a restore used, so an unrelated clean restore can't vouch for a failed one.
  • New validation_label (key=value, unset by default). Matching backups are counted in validation_backups and validation_backups_failed and are still judged like every other backup.
  • Backup records keep their labels, so the evidence shows them.

A failed scheduled backup still resolves through a later Completed backup of the same schedule; an unscheduled one only through its own retry.

Testing

  • 36 tests in tests/test_azure_velero_backups.py, including a failed restore followed by a clean restore of something else (still current, resolved only by its retry) and validation tests whose retries pass and fail.
  • test_contracts.py, test_category_requires.py and all Azure and Entra tests pass (6,574). The Velero tests need no Azure SDK.

Fixes FDE-1056

@togofishicorn

togofishicorn commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Found one P2 issue in fetchers/azure/velero_backups/fetcher.py:272 and the restore retry index at lines 287-288: retry matching crosses storage locations.

summarize() combines records before building name-only retry indexes. A successful drill-retry in one account/container/prefix can therefore mark an unrelated failed drill elsewhere as resolved. I reproduced backups_failed_unresolved dropping from 1 to 0 after adding another location's successful retry; restore failures and snapshot shortfalls are affected too. This can hide a failure that still needs attention.

Proposed fix (not pushed): classify each location's history separately before aggregating counts. This keeps both retry and scheduled recovery within the location. The patch includes nine regression cases covering backups, snapshot shortfalls, and restores across accounts, containers, and prefixes. Each case also verifies that a successful retry in the original location resolves the failure.

Validation: all nine new cases failed before the fix and passed afterward. The focused Velero and contract checks passed (1,020 passed, 17 skipped). The main CI test suite on Python 3.14 produced 10,266 passes and 89 skips; its two remaining color tests passed after clearing the inherited NO_COLOR setting. The first restricted run could not start local mock servers; the rerun with local servers allowed cleared those errors. TUI and separate SDK CI jobs were not run.

Patch:

diff --git a/fetchers/azure/velero_backups/fetcher.py b/fetchers/azure/velero_backups/fetcher.py
index 508052ea..b8c742d8 100644
--- a/fetchers/azure/velero_backups/fetcher.py
+++ b/fetchers/azure/velero_backups/fetcher.py
@@ -239,7 +239,7 @@ def parse_label(value: Optional[str]) -> Optional[Tuple[str, str]]:
 
 
 def classify_history(backups: List[Dict[str, Any]], restores: List[Dict[str, Any]], retry_suffix: Optional[str]) -> None:
-    """Mark each failure as resolved by a later success, or current; records are updated in place.
+    """Classify one location's failures as resolved or current; records are updated in place.
 
     No failure is excluded by its name or labels. When retry_suffix is set, a failure is also resolved by its own retry:
     a backup named <name><retry_suffix> that completed later (every attempted snapshot, for a snapshot shortfall), or a
@@ -301,7 +301,10 @@ def summarize(locations: List[Dict[str, Any]], settings: Dict[str, Any], now: da
     backups = [b for loc in read for b in loc["backups"]]
     restores = [r for loc in locations if loc["restores_read"] for r in loc["restores"]]
     restores_complete = discovered and all(loc["restores_read"] for loc in locations)
-    classify_history(backups, restores, settings.get("retry_suffix"))
+    # Names and schedules are local to an account/container/prefix, not globally unique.
+    for loc in locations:
+        classify_history(loc["backups"] if loc["backups_read"] else [],
+                         loc["restores"] if loc["restores_read"] else [], settings.get("retry_suffix"))
     label = settings.get("validation_label")
     validation = [b for b in backups if label and b.get("labels", {}).get(label[0]) == label[1]]
     latest: Dict[str, datetime] = {}
diff --git a/tests/test_azure_velero_backups.py b/tests/test_azure_velero_backups.py
index a44ce305..f4aa98f4 100644
--- a/tests/test_azure_velero_backups.py
+++ b/tests/test_azure_velero_backups.py
@@ -472,6 +472,45 @@ def test_a_failure_is_resolved_only_by_its_own_successful_retry():
     assert plain["restores_with_errors_unresolved"] == 1
 
 
+@pytest.mark.parametrize("different_field", ["storage_account", "container", "prefix"])
+@pytest.mark.parametrize("failure_kind", ["backup", "snapshots", "restore"])
+def test_only_a_retry_in_the_same_location_resolves_a_failure(different_field, failure_kind):
+    identity = {"storage_account": "backupsa", "container": "velero", "prefix": "cluster-a"}
+    original, unrelated = location(), location()
+    original.update(identity)
+    unrelated.update({**identity, different_field: "other"})
+    if failure_kind == "restore":
+        failed = vb.restore_record("drill", results(errors=1), iso(NOW - timedelta(hours=10)))
+        retry = vb.restore_record("drill-retry", results(), iso(NOW - timedelta(hours=2)))
+        records, unresolved = "restores", "restores_with_errors_unresolved"
+        resolved_field = "error_resolved"
+    else:
+        raw = backup("drill", schedule=None, done_hours_ago=10,
+                     phase="Failed" if failure_kind == "backup" else "Completed",
+                     attempted=2, completed=1)
+        failed = vb.backup_record("drill", raw, NOW, 24)
+        retry = vb.backup_record("drill-retry", backup("drill-retry", schedule=None, done_hours_ago=2), NOW, 24)
+        records = "backups"
+        unresolved = "backups_failed_unresolved" if failure_kind == "backup" else "backups_volume_snapshots_incomplete_unresolved"
+        resolved_field = "failure_resolved" if failure_kind == "backup" else "snapshots_resolved"
+    original[records].append(failed)
+    unrelated[records].append(retry)
+    settings = {**SETTINGS, "retry_suffix": "-retry"}
+
+    summary = vb.summarize([original, unrelated], settings, NOW)
+    assert summary[unresolved] == 1
+    assert summary["failures_resolved_by_retry"] == 0
+    assert failed[resolved_field] is False
+    assert failed["resolved_by_retry"] is False
+
+    original[records].append(dict(retry))
+    recovered = vb.summarize([original, unrelated], settings, NOW)
+    assert recovered[unresolved] == 0
+    assert recovered["failures_resolved_by_retry"] == 1
+    assert failed[resolved_field] is True
+    assert failed["resolved_by_retry"] is True
+
+
 def test_a_validation_label_not_in_key_value_form_is_bad_config():
     assert vb.parse_label("validation-test=true") == ("validation-test", "true")
     assert vb.parse_label("validation-test") is None and vb.parse_label("=true") is None and vb.parse_label(None) is None

@bruceweygandt

Copy link
Copy Markdown
Contributor Author

Thanks, good catch. Applied your patch as posted: each location's history is classified on its own before counts are aggregated, so a retry or later success in one account, container or prefix no longer resolves a failure in another. Your nine cases fail on the previous code and pass now; the full suite passes (6,583).

One related question before I change anything else. The schedule checks (missed runs, the RPO and the success window) still group backups by schedule name across every location. With two clusters that both run daily, one cluster's successful backups could hide the other's missed runs, which is the same problem one level up. Fixing it means choosing what an expected schedule means with more than one location:

  1. Every location must run it. An expected schedule is checked in each location separately, and a location without it counts as missing. Strict, but it fails a location that legitimately doesn't run that schedule.
  2. Checked where it runs. Missed runs and the RPO are judged per location, wherever the schedule has backups, and expected_schedules_without_backups lists only schedules missing everywhere.

I'd lean to 2, with the per-location results in the evidence so a location that stopped running a schedule is still visible. Which fits your setup?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants