Skip to content

NetApp: release SnapMirror relationships orphaned by a DR promote - #369

Open
Carthaca wants to merge 1 commit into
stable/2025.1-m3from
netapp-orphan-snapmirror-release
Open

Carthaca wants to merge 1 commit into
stable/2025.1-m3from
netapp-orphan-snapmirror-release

Conversation

@Carthaca

Copy link
Copy Markdown
Collaborator

Problem

After a replica promote (DR failover), the previously active replica is
demoted and its old outgoing SnapMirror relationships are supposed to be
released by cleanup_previous_snapmirror_relationships. That release is
best-effort and is skipped when the demoted replica's host is unreachable
at promote time
delete_snapmirror sets src_client = None and returns
without releasing (and ESOURCE_IS_DIFFERENT is likewise swallowed).

Once the demoted replica converges to in_sync as a destination of the new
source, update_replica_state no longer attempts any cleanup ("Replicas
already in-sync won't try another cleanup attempt"), so the stale
relationship lingers on the backend as a broken-off orphan indefinitely.

Observed symptom: manila reports the live replica in_sync (correct — the
forward-filtered query never matches the stale reverse orphan), while
snapmirror show on the backend lists a Broken-off, Healthy=false
relationship left over from before the promote. This is not a
mis-classification of the live replica — it is an un-reconciled orphan.

Fix

  • Add DataMotionSession.has_leftover_source_snapmirrors(replica, replica_list):
    lists the source-side destinations for the replica and returns True only if
    one targets another known replica volume (known-pairings only). Returns
    False when the source is unreachable (self-healing — retried next cycle) or
    on API error.
  • update_replica_state now runs the existing
    cleanup_previous_snapmirror_relationships once for an in_sync replica when
    such a leftover is detected. The reported replica_state is unchanged.

This is the relationship-cleanup sibling of 7771edc ("Delete SnapMirror
snapshots left behind after a DR replica promotion"), which addressed only the
orphaned snapshots, not the orphaned relationship.

Testing

  • New unit tests:
    • test_update_replica_state_in_sync_with_leftover_snapmirror
    • test_has_leftover_source_snapmirrors_found / _no_known_destination /
      _source_unreachable / _list_error
    • Updated the two existing in_sync cases to pin the new check.
  • tox -e py3 for test_lib_base and test_data_motion: green.
  • flake (hacking) on the changed files: clean.

AI assistance

AI generated the has_leftover_source_snapmirrors helper, the
update_replica_state gating branch, and the unit tests; human reviewed and
verified the SnapMirror direction semantics and the known-pairings safety
constraint. See the Assisted-By trailer on the commit.

After a replica promote (failover), the previously active replica is
demoted and its old outgoing SnapMirror relationships are supposed to be
released by cleanup_previous_snapmirror_relationships. That release is
best-effort and is skipped when the demoted replica's host is unreachable
at promote time (delete_snapmirror sets src_client=None and returns
without releasing; ESOURCE_IS_DIFFERENT is likewise swallowed).

Once the demoted replica converges to 'in_sync' as a destination of the
new source, update_replica_state no longer attempts any cleanup ("Replicas
already in-sync won't try another cleanup attempt"), so the stale
relationship lingers on the backend as a broken-off orphan indefinitely.

Add DataMotionSession.has_leftover_source_snapmirrors(), which lists the
source-side destinations for the replica and reports whether any point at
another known replica volume (known pairings only). update_replica_state
now runs the existing cleanup once for an in_sync replica when such a
leftover is detected, without altering the reported replica_state. When
the source is still unreachable the check is a no-op and retries next
periodic cycle (self-healing).

AI generated the has_leftover_source_snapmirrors helper, the
update_replica_state gating branch, and the accompanying unit tests;
human reviewed and verified the SnapMirror direction semantics and the
known-pairings safety constraint.

Change-Id: I2741f8394f8e603d9d72c454b6da879216aeca27
Assisted-By: Claude (Anthropic, claude-opus-4-1-20250805)
Signed-off-by: Maurice Escher <maurice.escher@sap.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant