You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Passive swarm operations are not safe when one or more MPI ranks hold zero particles. Observed symptoms:
hangs during write_timestep / swarm save — the core reported bug
hangs in evaluate / global_evaluate when evaluating swarm particle data
estimate_dt() crashing on an empty rank with ValueError: cannot reshape array of size 0 into shape (0,newaxis)
What we found (after rebasing onto development)
The evaluate / global_evaluate / advection empty-rank deadlock was already fixed upstream by #611 / PR #656 (forward the cell-location policy on the empty-rank path). With that merged we verified empirically (np=4) that the default non-evalf path now works on empty ranks, so the earlier evalf-threading workaround was dropped.
The remaining, actual cause of the write_timestep / save hang is a parallel-HDF5 (MPIO) collective-close deadlock triggered by a per-rank dtype mismatch:
An int swarm variable (e.g. swarm.add_variable("uid", size=1, dtype=int)) holds particles on some ranks but not others.
On an empty rank, SwarmVariable.unpack_raw_data_from_petsc() returned np.zeros((0, n)) — a float64 array — losing the field dtype, so an empty rank reported float64 while a non-empty rank reported int32.
SwarmVariable.save() passes the per-rank local_data.dtype to the collective parallel-HDF5 create_dataset(...). Divergent dtypes leave HDF5 metadata inconsistent, so the collective close never synchronises and the save deadlocks (same silent-hang class as [BUG] - swarm.write_timestep fails in parallel (np > 1) #151).
Fix
estimate_dt() — empty-rank guard: zero-particle velocity data is reshaped to an explicit (0, dim) array instead of crashing on reshape(n, -1).
Empty-rank dtype preservation — store the field PETSc dtype (_petsc_dtype) and use it in every zero-length fallback array, so empty-rank data agrees with the field true type across ranks. Fixes the MPIO save deadlock.
Tests (passing at np=2 and np=4)
tests/parallel/test_0795_swarm_empty_rank_evaluate_save.py — evaluate + write_timestep on empty ranks. test_passive_swarm_save_empty_ranks reproduces the user pattern and hung at np=4 before the dtype fix.
Problem
Passive swarm operations are not safe when one or more MPI ranks hold zero particles. Observed symptoms:
write_timestep/ swarmsave— the core reported bugevaluate/global_evaluatewhen evaluating swarm particle dataestimate_dt()crashing on an empty rank withValueError: cannot reshape array of size 0 into shape (0,newaxis)What we found (after rebasing onto
development)The
evaluate/global_evaluate/advectionempty-rank deadlock was already fixed upstream by #611 / PR #656 (forward the cell-location policy on the empty-rank path). With that merged we verified empirically (np=4) that the default non-evalfpath now works on empty ranks, so the earlierevalf-threading workaround was dropped.The remaining, actual cause of the
write_timestep/savehang is a parallel-HDF5 (MPIO) collective-close deadlock triggered by a per-rank dtype mismatch:intswarm variable (e.g.swarm.add_variable("uid", size=1, dtype=int)) holds particles on some ranks but not others.SwarmVariable.unpack_raw_data_from_petsc()returnednp.zeros((0, n))— afloat64array — losing the field dtype, so an empty rank reportedfloat64while a non-empty rank reportedint32.SwarmVariable.save()passes the per-ranklocal_data.dtypeto the collective parallel-HDF5create_dataset(...). Divergent dtypes leave HDF5 metadata inconsistent, so the collectiveclosenever synchronises and the save deadlocks (same silent-hang class as [BUG] - swarm.write_timestep fails in parallel (np > 1) #151).Fix
estimate_dt()— empty-rank guard: zero-particle velocity data is reshaped to an explicit(0, dim)array instead of crashing onreshape(n, -1)._petsc_dtype) and use it in every zero-length fallback array, so empty-rank data agrees with the field true type across ranks. Fixes the MPIO save deadlock.Tests (passing at np=2 and np=4)
tests/parallel/test_0795_swarm_empty_rank_evaluate_save.py— evaluate +write_timestepon empty ranks.test_passive_swarm_save_empty_ranksreproduces the user pattern and hung at np=4 before the dtype fix.tests/parallel/test_0796_swarm_advection_empty_rank.py— advection on the default path with empty ranks (guards the test_global_evaluate_after_migration hangs at np=4, and its file is in neither parallel-test glob #611 regression).Both files: 5 passed at np=2 and 5 passed at np=4.
Related empty-rank / global-evaluate work: #611.