Skip to content

Passive swarm save/advection hangs or crashes on empty MPI ranks #679

Description

@bknight1

Problem

Passive swarm operations are not safe when one or more MPI ranks hold zero particles. Observed symptoms:

  • hangs during write_timestep / swarm save — the core reported bug
  • hangs in evaluate / global_evaluate when evaluating swarm particle data
  • estimate_dt() crashing on an empty rank with ValueError: cannot reshape array of size 0 into shape (0,newaxis)

What we found (after rebasing onto development)

The evaluate / global_evaluate / advection empty-rank deadlock was already fixed upstream by #611 / PR #656 (forward the cell-location policy on the empty-rank path). With that merged we verified empirically (np=4) that the default non-evalf path now works on empty ranks, so the earlier evalf-threading workaround was dropped.

The remaining, actual cause of the write_timestep / save hang is a parallel-HDF5 (MPIO) collective-close deadlock triggered by a per-rank dtype mismatch:

  • An int swarm variable (e.g. swarm.add_variable("uid", size=1, dtype=int)) holds particles on some ranks but not others.
  • On an empty rank, SwarmVariable.unpack_raw_data_from_petsc() returned np.zeros((0, n)) — a float64 array — losing the field dtype, so an empty rank reported float64 while a non-empty rank reported int32.
  • SwarmVariable.save() passes the per-rank local_data.dtype to the collective parallel-HDF5 create_dataset(...). Divergent dtypes leave HDF5 metadata inconsistent, so the collective close never synchronises and the save deadlocks (same silent-hang class as [BUG] - swarm.write_timestep fails in parallel (np > 1) #151).

Fix

  1. estimate_dt() — empty-rank guard: zero-particle velocity data is reshaped to an explicit (0, dim) array instead of crashing on reshape(n, -1).
  2. Empty-rank dtype preservation — store the field PETSc dtype (_petsc_dtype) and use it in every zero-length fallback array, so empty-rank data agrees with the field true type across ranks. Fixes the MPIO save deadlock.

Tests (passing at np=2 and np=4)

Both files: 5 passed at np=2 and 5 passed at np=4.

Related empty-rank / global-evaluate work: #611.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions