Skip to content

adapt: snap_level_boundaries guards its allreduce on a rank-local mask (np>1 hang) #627

Description

@lmoresi

snap_level_boundaries reaches its allreduce only on ranks that own a snapped vertex, so adapt with registered bounding surfaces can hang at np>1.

In src/underworld3/discretisation/discretisation_mesh.py the inversion check
after a boundary snap is guarded by a rank-local mask:

snapped_any = numpy.zeros(arr.shape[0], dtype=bool)
for s in _snap_surfs:
    mask = sync_mask(_pinned_mask(dm, (s.label,)))
    if mask.any():
        arr[mask] = s.restore(arr[mask])
        snapped_any |= mask
dm.setCoordinatesLocal(vec)
if snapped_any.any():              # <-- rank-local
    ...
    if uw.mpi.size > 1:
        flipped = uw.mpi.comm.allreduce(flipped, op=_MPI.SUM)

snapped_any is a mask over this rank's local vertices. sync_mask makes it
agree between ranks that share a vertex, but a rank whose partition holds no
vertex on any registered surface has an all-False mask, skips the block, and
never arrives at the reduction. Its peers stay in allreduce until the job is
killed.

This is the shape in #612 and in the migrate working set: a collective behind a
predicate on local data. It is partition-dependent, so it will present as an
intermittent hang in adapt rather than a reproducible one, and it needs both
np>1 and a partition that leaves a rank clear of the snap surfaces — which is
why it has not shown up in the np=2 suite.

Fix: initialise flipped = 0 before the guard, keep the cell-orientation
loop inside it (the local work is genuinely conditional), and reach the
reduction unconditionally.

Found by a static scan of every comm.<op> call site asking whether it is
reached on all ranks; it flags 2 of 210 sites, this one and #612.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions