Repository navigation
fix(azure_velero_backups): resolve failures only by a later success or their own retry - #391
bruceweygandt wants to merge 2 commits into
Conversation
…r their own retry
|
Found one P2 issue in
Proposed fix (not pushed): classify each location's history separately before aggregating counts. This keeps both retry and scheduled recovery within the location. The patch includes nine regression cases covering backups, snapshot shortfalls, and restores across accounts, containers, and prefixes. Each case also verifies that a successful retry in the original location resolves the failure. Validation: all nine new cases failed before the fix and passed afterward. The focused Velero and contract checks passed (1,020 passed, 17 skipped). The main CI test suite on Python 3.14 produced 10,266 passes and 89 skips; its two remaining color tests passed after clearing the inherited Patch: diff --git a/fetchers/azure/velero_backups/fetcher.py b/fetchers/azure/velero_backups/fetcher.py
index 508052ea..b8c742d8 100644
--- a/fetchers/azure/velero_backups/fetcher.py
+++ b/fetchers/azure/velero_backups/fetcher.py
@@ -239,7 +239,7 @@ def parse_label(value: Optional[str]) -> Optional[Tuple[str, str]]:
def classify_history(backups: List[Dict[str, Any]], restores: List[Dict[str, Any]], retry_suffix: Optional[str]) -> None:
- """Mark each failure as resolved by a later success, or current; records are updated in place.
+ """Classify one location's failures as resolved or current; records are updated in place.
No failure is excluded by its name or labels. When retry_suffix is set, a failure is also resolved by its own retry:
a backup named <name><retry_suffix> that completed later (every attempted snapshot, for a snapshot shortfall), or a
@@ -301,7 +301,10 @@ def summarize(locations: List[Dict[str, Any]], settings: Dict[str, Any], now: da
backups = [b for loc in read for b in loc["backups"]]
restores = [r for loc in locations if loc["restores_read"] for r in loc["restores"]]
restores_complete = discovered and all(loc["restores_read"] for loc in locations)
- classify_history(backups, restores, settings.get("retry_suffix"))
+ # Names and schedules are local to an account/container/prefix, not globally unique.
+ for loc in locations:
+ classify_history(loc["backups"] if loc["backups_read"] else [],
+ loc["restores"] if loc["restores_read"] else [], settings.get("retry_suffix"))
label = settings.get("validation_label")
validation = [b for b in backups if label and b.get("labels", {}).get(label[0]) == label[1]]
latest: Dict[str, datetime] = {}
diff --git a/tests/test_azure_velero_backups.py b/tests/test_azure_velero_backups.py
index a44ce305..f4aa98f4 100644
--- a/tests/test_azure_velero_backups.py
+++ b/tests/test_azure_velero_backups.py
@@ -472,6 +472,45 @@ def test_a_failure_is_resolved_only_by_its_own_successful_retry():
assert plain["restores_with_errors_unresolved"] == 1
+@pytest.mark.parametrize("different_field", ["storage_account", "container", "prefix"])
+@pytest.mark.parametrize("failure_kind", ["backup", "snapshots", "restore"])
+def test_only_a_retry_in_the_same_location_resolves_a_failure(different_field, failure_kind):
+ identity = {"storage_account": "backupsa", "container": "velero", "prefix": "cluster-a"}
+ original, unrelated = location(), location()
+ original.update(identity)
+ unrelated.update({**identity, different_field: "other"})
+ if failure_kind == "restore":
+ failed = vb.restore_record("drill", results(errors=1), iso(NOW - timedelta(hours=10)))
+ retry = vb.restore_record("drill-retry", results(), iso(NOW - timedelta(hours=2)))
+ records, unresolved = "restores", "restores_with_errors_unresolved"
+ resolved_field = "error_resolved"
+ else:
+ raw = backup("drill", schedule=None, done_hours_ago=10,
+ phase="Failed" if failure_kind == "backup" else "Completed",
+ attempted=2, completed=1)
+ failed = vb.backup_record("drill", raw, NOW, 24)
+ retry = vb.backup_record("drill-retry", backup("drill-retry", schedule=None, done_hours_ago=2), NOW, 24)
+ records = "backups"
+ unresolved = "backups_failed_unresolved" if failure_kind == "backup" else "backups_volume_snapshots_incomplete_unresolved"
+ resolved_field = "failure_resolved" if failure_kind == "backup" else "snapshots_resolved"
+ original[records].append(failed)
+ unrelated[records].append(retry)
+ settings = {**SETTINGS, "retry_suffix": "-retry"}
+
+ summary = vb.summarize([original, unrelated], settings, NOW)
+ assert summary[unresolved] == 1
+ assert summary["failures_resolved_by_retry"] == 0
+ assert failed[resolved_field] is False
+ assert failed["resolved_by_retry"] is False
+
+ original[records].append(dict(retry))
+ recovered = vb.summarize([original, unrelated], settings, NOW)
+ assert recovered[unresolved] == 0
+ assert recovered["failures_resolved_by_retry"] == 1
+ assert failed[resolved_field] is True
+ assert failed["resolved_by_retry"] is True
+
+
def test_a_validation_label_not_in_key_value_form_is_bad_config():
assert vb.parse_label("validation-test=true") == ("validation-test", "true")
assert vb.parse_label("validation-test") is None and vb.parse_label("=true") is None and vb.parse_label(None) is None |
Patch from review.
|
Thanks, good catch. Applied your patch as posted: each location's history is classified on its own before counts are aggregated, so a retry or later success in one account, container or prefix no longer resolves a failure in another. Your nine cases fail on the previous code and pass now; the full suite passes (6,583). One related question before I change anything else. The schedule checks (missed runs, the RPO and the success window) still group backups by schedule name across every location. With two clusters that both run
I'd lean to 2, with the per-location results in the evidence so a location that stopped running a schedule is still visible. Which fits your setup? |
Summary
Changes how
azure_velero_backupsdecides that a failure is resolved, so that nothing is hidden by name and no unrelated success clears a failure. Follows review feedback after #376 merged.deliberate_failure_tests. Matching a name pattern excluded failed backups and restores from the unresolved counts. Validation tests often carry a label or name prefix that marks a test, not an expected failure, so excluding by name or label could hide a real failure.retry_suffix(unset by default). When set, a failed backup or a restore with errors is resolved by its own retry:<name><suffix>finishing later without failing (every attempted snapshot, for a snapshot shortfall). A failed retry, or none, leaves the failure current.failures_resolved_by_retrycounts these.validation_label(key=value, unset by default). Matching backups are counted invalidation_backupsandvalidation_backups_failedand are still judged like every other backup.A failed scheduled backup still resolves through a later Completed backup of the same schedule; an unscheduled one only through its own retry.
Testing
tests/test_azure_velero_backups.py, including a failed restore followed by a clean restore of something else (still current, resolved only by its retry) and validation tests whose retries pass and fail.test_contracts.py,test_category_requires.pyand all Azure and Entra tests pass (6,574). The Velero tests need no Azure SDK.Fixes FDE-1056