test_0054_hang_report leaks a spinning child process whenever pytest is killed (7 orphans, 7 cores)

[lmoresi]

tests/test_0054_hang_report.py spawns child processes that are deliberately designed
never to exit — one blocks in an MPI collective, one is an armed watchdog sitting after
import underworld3. Both are launched with start_new_session=True, which detaches
them from pytest's process group, and both are cleaned up only by an os.killpg(...) in
a finally:

• :200 — _run_until_the_evidence_exists, the four-rank divergent.py job
• :336 — test_the_environment_variable_arms_the_watchdog_at_import, sleepy.py

If the pytest process is killed rather than allowed to complete, the finally never
runs. start_new_session=True then means nothing else reaches the child either, so it
survives indefinitely, re-parented to init. Neither script terminates on its own, and
both busy-poll, so each leak costs a full core forever.

That is not hypothetical. Found running on a 16-core machine today, all ppid=1:

PID	fixture	pytest tmpdir	age	%CPU
72806	divergent.py	pytest-328	2d 19h	97
8235	divergent.py	pytest-331	1d 18h	97
31697	divergent.py	pytest-332	1d 16h	95
13735	divergent.py	pytest-333	1d 4h	98
14240	divergent.py	pytest-334	1d 4h	98
74914	sleepy.py	(fault-clip-boundary worktree)	2h 51m	98
75989	sleepy.py	(fault-clip-boundary worktree)	2h 48m	98

Seven of sixteen cores, from five separate pytest sessions across two worktrees. Load
average was 55+ and an ordinary -m "tier_a or tier_b" run was getting a fraction of a
core — it ran five hours without finishing. After killing them the run behaves normally.

This bites hardest exactly where the file is being worked on: a hang investigation is
the situation in which you interrupt pytest, and interrupting pytest is what leaks.

Suggested fix — the child must be recoverable after its parent dies, which a finally
cannot guarantee:

1. Write each spawned child's pid to a known file (e.g. under the pytest tmp root) as
   soon as it is spawned, and add a session-scoped fixture that reaps any pid still
   alive at session end — including from a previous, killed session.
2. Have the fixture scripts arm their own dead-man switch (UW_HANG_WATCHDOG already
   knows how to time out; a bare alarm()/SIGALRM or a parent-death check would do),
   so a child that loses its parent exits on its own rather than spinning forever.
3. Belt and braces: SIGTERM the group before SIGKILL so mpirun gets the chance to tear
   its ranks down properly, rather than being removed instantly from under them.

(2) is the one that actually closes it, since it does not depend on any cleanup code
running in the parent at all.

[lmoresi]

Still reproduces after #638 merged. #638 fixed the hang (the file now passes in ~21 s where it previously wedged a suite for hours); it did not fix the leak on kill, which is a separate failure mode.

Retested on current development: start test_end_to_end_names_the_divergent_rank, let the four ranks come up, SIGKILL the pytest process. Result:

orphans (ppid=1) before=0 after=1
82987   1   prterun --oversubscribe -n 4 ...
82989  82987  ... (rank)
82990  82987  ... (rank)
82991  82987  ... (rank)
82992  82987  ... (rank)

The orphan is prterun itself, not a bare rank — so the whole MPI job survives, and start_new_session=True means nothing subsequently reaches it. That is consistent with the diagnosis in the issue body: the only cleanup is os.killpg in a finally, and a finally cannot run after its own process is SIGKILLed.

This is worth keeping open. The hang was the loud symptom, but the leak is the one that quietly costs a core per occurrence and is invisible until someone looks at ps.

Underworld development team with AI support from Claude Code