Skip to content

Latent MPI deadlock: spring smoother returns early on rank-local n_free==0 before collectives #346

Description

@lmoresi

Found (marker-only, Charter §9) during the Wave D smoothing readability pass, 2026-07-08. A TODO(BUG) marker lands with PR #345 in src/underworld3/meshing/smoothing/spring.py.

The spring-equilibrium mover takes a rank-local early return when a rank has no free (unpinned) nodes, BEFORE reaching the collective reduction calls later in the routine. On a partition where some rank owns only pinned/boundary nodes (small partitions, heavily graded meshes at high rank counts), that rank exits while the others enter the allreduce — a classic collective mismatch deadlock.

Not observed in the campaign's np2 gates (both ranks own free nodes in the test meshes); the hazard is latent and partition-dependent.

Suggested fix: replace the early return with a zero-work path that still participates in every collective (empty contribution), or hoist the collectives above the return. Needs an np>2 test with a deliberately starved rank.

Related: the Wave D smoothing PR #345 (marker placed, behaviour untouched per readability-wave scope discipline).

Underworld development team with AI support from Claude Code

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions