Observation
During lane 4's REVIEW (#3216, PR #3309), tests/unit/test_improvement_eval_runner.py landed alone on one --dist loadfile xdist worker: 30 tests at roughly 25s each (each spawns two redis-server arm subprocesses), about 12.6 minutes for the module. scripts/pytest-clean.sh's stall guard (PYTEST_STALL_LIMIT_S, default 600, cumulative controller CPU sampled every 30s with a 1s floor) reported a wedge and the run was treated as stalled. The controller was alive: the worker was still delivering one result every ~25s, but a single slow worker gives the controller almost nothing to do, so its CPU delta over ten minutes stayed under the floor.
The lane's fix was to split the module in two so no single loadfile unit exceeds the window. That works per module and does nothing for the next arm-spawning test file over roughly 20 tests, which will hit the same wall.
Why it matters
The guard's stated separation ("0.03s wedged versus tens of seconds live") assumes a controller busy collecting from several workers. A long I/O-bound module on one worker sits between those cases: live, but with a controller CPU profile near the wedged one. Every false positive costs an operator a rerun and erodes trust in the guard, which exists precisely so a real wedge is believed.
Fix shape
Make progress the signal rather than controller CPU alone: a live run either accrues controller CPU or emits test outcomes. With -v (which the wrapper can add) the stdout line count advances per test; a window with no new outcome lines and no meaningful CPU is a wedge, either alone is not. Keep the CPU-only path as the fallback when output cannot be observed. Add a test that runs a synthetic module of a few 30s+ sleeps on one worker under a short PYTEST_STALL_LIMIT_S and asserts the guard does not fire, alongside the existing wedge-detection test.
Refs #3177 (surfaced in lane 4, #3216).
Triage 2026-09-15
Status: confirmed by code read on main 205344717.
Single-line hotfix: Raising PYTEST_STALL_LIMIT_S's default (scripts/pytest-clean.sh:263) is a one-line change and would paper over this specific incident, but it isn't the minimum correct fix — the next arm-spawning module with ~20+ tests on a lone loadfile worker hits the same wall at whatever the new ceiling is. The lane's actual fix (splitting the module) confirms the limit itself isn't the real bug.
Reduce-complexity option: Yes — cumulative controller CPU is the wrong signal for this shape. A live run either accrues controller CPU (many workers reporting in) or emits test outcomes (-v line count advancing) even with a single slow worker; today's guard only checks the first. Replacing CPU-delta with an outcome-count-or-CPU-delta check (either advancing means live) fixes the false positive at its root rather than by raising a threshold, per the issue's own fix shape.
Observation
During lane 4's REVIEW (#3216, PR #3309),
tests/unit/test_improvement_eval_runner.pylanded alone on one--dist loadfilexdist worker: 30 tests at roughly 25s each (each spawns tworedis-serverarm subprocesses), about 12.6 minutes for the module.scripts/pytest-clean.sh's stall guard (PYTEST_STALL_LIMIT_S, default 600, cumulative controller CPU sampled every 30s with a 1s floor) reported a wedge and the run was treated as stalled. The controller was alive: the worker was still delivering one result every ~25s, but a single slow worker gives the controller almost nothing to do, so its CPU delta over ten minutes stayed under the floor.The lane's fix was to split the module in two so no single
loadfileunit exceeds the window. That works per module and does nothing for the next arm-spawning test file over roughly 20 tests, which will hit the same wall.Why it matters
The guard's stated separation ("0.03s wedged versus tens of seconds live") assumes a controller busy collecting from several workers. A long I/O-bound module on one worker sits between those cases: live, but with a controller CPU profile near the wedged one. Every false positive costs an operator a rerun and erodes trust in the guard, which exists precisely so a real wedge is believed.
Fix shape
Make progress the signal rather than controller CPU alone: a live run either accrues controller CPU or emits test outcomes. With
-v(which the wrapper can add) the stdout line count advances per test; a window with no new outcome lines and no meaningful CPU is a wedge, either alone is not. Keep the CPU-only path as the fallback when output cannot be observed. Add a test that runs a synthetic module of a few 30s+ sleeps on one worker under a shortPYTEST_STALL_LIMIT_Sand asserts the guard does not fire, alongside the existing wedge-detection test.Refs #3177 (surfaced in lane 4, #3216).
Triage 2026-09-15
Status: confirmed by code read on main
205344717.Single-line hotfix: Raising
PYTEST_STALL_LIMIT_S's default (scripts/pytest-clean.sh:263) is a one-line change and would paper over this specific incident, but it isn't the minimum correct fix — the next arm-spawning module with ~20+ tests on a loneloadfileworker hits the same wall at whatever the new ceiling is. The lane's actual fix (splitting the module) confirms the limit itself isn't the real bug.Reduce-complexity option: Yes — cumulative controller CPU is the wrong signal for this shape. A live run either accrues controller CPU (many workers reporting in) or emits test outcomes (
-vline count advancing) even with a single slow worker; today's guard only checks the first. Replacing CPU-delta with an outcome-count-or-CPU-delta check (either advancing means live) fixes the false positive at its root rather than by raising a threshold, per the issue's own fix shape.