Supervise every parallel batch: bound a collective hang and name the rank that caused it

[lmoresi]

Summary

A rank blocked inside a collective produces no output and never returns. An
unsupervised batch therefore spends the whole job budget in silence and is
cancelled from outside with no diagnosis — measured at 76 minutes on one run
(#675), which is also why test_0855 and test_0873 managed to fail there
without leaving a traceback behind.

scripts/mpi_supervisor.py bounds that and says what went wrong.

Why not the mechanisms we already had

Every one of them asks the stuck process to cooperate, which is the one thing it
has stopped being able to do.

mechanism	why it fails
threading.Timer + report	needs the GIL. Measured at np=4 against a 4 s block in comm.allreduce: fired zero times on the blocked ranks.
faulthandler.dump_traceback_later	a C thread walking other threads' live frames; can wedge or crash the process it is diagnosing (#661).
MPI_Abort from a watchdog thread	needs THREAD_MULTIPLE, not safe from inside a collective.
mpirun --timeout	Open MPI --timeout vs MPICH -timeout, different semantics, and it tells you nothing when it fires.

The supervisor asks nothing of the job: a parent process holding a clock and a
signal. Identical behaviour under either MPI and on either platform, because it
is POSIX process management and knows nothing about MPI.

It triggers on silence, not elapsed time

A batch that is still printing is alive. Judging on total runtime means
inventing a wall-clock budget for a matrix nobody has measured, and that number
is wrong the first time someone adds a slow test. The #675 failure was 76
minutes of nothing, not of slow progress.

What it reports

=== MPI SUPERVISOR: no output for 300 s ===
    no rank moved in 20 s: this is stuck, not slow
    rank(s) 0 are somewhere the other 3 are not — that is where to look first
      rank 0 is in _rank_zero_misses_the_collective()
      the other 3 are in main()
    rank 0: 0% CPU, idle — blocked rather than polling
    rank 1: 100% CPU, spinning — a busy-wait, which is what an MPI progress engine does

Three independent pieces of evidence: whether anything moved between two samples
(stuck vs slow), which rank is somewhere the others are not, and CPU per rank.

Ranks are armed through a usercustomize module that registers a SIGUSR1
handler. That is a signal handler, not a concurrent walker thread — it runs on
the blocked thread itself, only when asked, so arming it costs a healthy run
nothing and it cannot reproduce #661.

The control

tests/test_0063_mpi_hang_supervisor.py plants a hang whose answer is known:
rank 0 sits in a function no other rank can be inside while the others wait in a
barrier. The supervisor must end the job, call it stuck, and name rank 0.

It also asserts the other direction — a healthy job passes through untouched
with its own exit status. Without that, every other assertion could be satisfied
by a supervisor that kills everything it is given.

3 passed in 109.62s. A harness that catches hangs is worth nothing until it
has caught one.

Two things worth flagging

The spinning ranks are the innocent ones. An MPI barrier busy-waits, so the
ranks burning 100% CPU are the ones waiting and the guilty rank is the one
asleep. An earlier version of this guessed the rank-to-CPU mapping by sorting
and reported it exactly backwards; the handler now records its own pid so the
attribution is read rather than inferred. A confident wrong label is worse than
no label.

At np=2 there is no majority. The report says so and declines to nominate a
rank on a one-all split, rather than manufacturing a culprit.

Scope

scripts/test.sh routes its two existing parallel batches through it
(PARALLEL_SILENCE overrides the budget). This does not touch #615, which
rewrites those loops — that PR can rebase onto this and inherit the protection.

Does not fix #675 or #661; it makes #675 diagnosable and it stops CI arming the
mechanism behind #661.

Underworld development team with AI support from Claude Code

🤖 Generated with Claude Code

https://claude.ai/code/session_01E87Q7KrpapxeQiLD1RiNXv

[Copilot]

🟡 Changes recommended

The supervisor currently signals all descendant processes with SIGUSR1 (including non-Python MPI helper processes) and the test runner wrapper has unsafe shell word-splitting, both of which can break the intended diagnosis/robustness.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR introduces an MPI “silence supervisor” wrapper to prevent parallel test batches from hanging indefinitely when a rank blocks inside a collective, and adds documentation + a control test to ensure the mechanism both catches a planted hang and leaves healthy jobs untouched.

Changes:

• Add scripts/mpi_supervisor.py to run mpirun ... under a parent process that monitors output silence, requests per-rank stack dumps via SIGUSR1, diagnoses divergence, and kills the job tree if needed.
• Route existing parallel batches in scripts/test.sh through the supervisor (with PARALLEL_SILENCE override).
• Add an end-to-end pytest control (tests/test_0063_mpi_hang_supervisor.py) and a planted hang payload (tests/parallel/hang_controls/...) plus developer docs describing usage and rationale.

File summaries

File	Description
tests/test_0063_mpi_hang_supervisor.py	Adds end-to-end tests that validate the supervisor kills a known hang, reports correctly, and does not interfere with healthy runs.
tests/parallel/hang_controls/rank_zero_misses_the_collective.py	Adds a planted “rank 0 misses collective” hang payload used by the supervisor tests.
scripts/test.sh	Wraps parallel pytest batches with the new supervisor to bound silent hangs in CI.
scripts/mpi_supervisor.py	Implements the supervisor: output-silence monitoring, stack-dump requests, diagnosis heuristics, and job-tree kill/verification.
docs/developer/index.md	Adds the new guide to the developer docs toctree.
docs/developer/guides/mpi-hang-supervision.md	Documents the motivation, usage, reporting semantics, and implementation details of the supervisor.

Review details

• Files reviewed: 6/6 changed files
• Comments generated: 4
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[lmoresi]

First real use, and it found the bug it was built for: #675 is root-caused in
#681. Seventy-six minutes of silence became a named diagnosis in six.

tests/parallel/test_0855…::test_parallel_matches_serial_bit_identical FAILED
tests/parallel/test_0873…::test_premise_the_metric_splits_the_ranks   FAILED

=== MPI SUPERVISOR: no output for 121 s ===
    no rank moved in 20 s: this is stuck, not slow
    the ranks are in different places, and with no majority none can be
    called the odd one out:
      rank 0 is in _from_plexh5()
      rank 1 is in barrier()

Rank 0 inside a collective HDF5 read, rank 1 waiting at a barrier it never
reached — a rank-divergent collective, arriving immediately after a failure, and
correctly reported as a tie rather than a nominated culprit.

One real limitation, worth a follow-up rather than a change now. Killing on
silence costs pytest's end-of-run failure report: the supervisor says where the
ranks are stuck but not what failed just before, because pytest prints its
summary at the end of a run and there is no end. I got #675's assertion by
isolating each suspect test in its own mpirun, which worked, but that was a
manual step.

Two ways to close it, neither in this PR:

• ask pytest to report failures as they happen (-x, or an --instafail-style
  reporter) in the supervised batches, so the traceback precedes any divergence;
• have the supervisor send SIGINT before SIGTERM, which pytest handles by
  printing its summary — worth measuring, since a rank inside a collective may
  not act on it either.

Either belongs with #615, which rewrites these loops and decides how the batches
are split.