Skip to content

aurora-sep: send Ctrl-C again in the survey stop test (flaky 0 != 130) - #443

Closed
joshuaswarren wants to merge 1 commit into
omacom:integration/aurorafrom
joshuaswarren:w7J/survey-ctrl-c-test-flake
Closed

joshuaswarren wants to merge 1 commit into
omacom:integration/aurorafrom
joshuaswarren:w7J/survey-ctrl-c-test-flake

Conversation

@joshuaswarren

Copy link
Copy Markdown

[Joshua's agent] Fixes the flaky test that made two checks fail on #191 (run 42771530679, job 129680296443).

Problem

test_ctrl_c_stops_the_load failed in CI with AssertionError: 0 != 130 : (stderr empty). The same head passed on re-run (run 42777741719). The checks that still show as failed on #191 are the entries of the first run.

Cause

The test sends one SIGINT to the survey and waits. Under load that signal is sometimes lost. The survey keeps running, finishes all phases, and exits 0.

I added a check before the kill to tell a lost signal from a late kill. In 3 failing runs the survey was still alive just before the kill (p.poll() was None, 32 loads running). It then ran 27 to 30 s more and exited 0. So the signal was lost. It was not sent too late.

Change

Send SIGINT again every 2 s, up to 4 times, like a user who presses Ctrl-C again. The exit code 130, the stderr check and the cleanup checks stay as they were. Only tools/aurora-sep/tests/test_m3_max.py changes (+11 -2).

Measured

Load: 12 test runs at once plus 24 busy loops on a 16-core Linux box. This is heavier than the CI runner on purpose. The 200-run batch used a copy of the test that only adds the p.poll() check before the kill.

Version Runs Exit 0 (lost signal)
Before 440 (80 + 160 + 200) 11 (2.5%)
After 160 0

The whole test_m3_max module also passes with this change (68 tests, 1 skipped).

Not covered

  • The loss can also hit a real Ctrl-C on a loaded machine. Pressing Ctrl-C again works. The cause is in how bash handles the signal in the survey loop. I did not change install-aurora-sep.sh. @chriskearney, you own the survey, so you may want a fix there.
  • Under the same heavy load I saw one other rare failure (1 in 80): the INT trap printed line 2: unexpected EOF while looking for matching ')' and the exit code was 2. It is a different problem and this PR does not touch it.
  • Under the heaviest load the helper bash -c calls in other tests can time out. Not changed.

test_ctrl_c_stops_the_load sent one SIGINT and then waited 30 s. Under
load bash sometimes drops that signal. The survey then runs to the end
and exits 0, and the test fails with "0 != 130" (CI run 42771530679 on
PR 191, passed on re-run of the same head).

Send SIGINT again every 2 s, up to 4 times, like a user who presses
Ctrl-C again. The 130 exit code and stderr checks stay as they were.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
@joshuaswarren
joshuaswarren force-pushed the w7J/survey-ctrl-c-test-flake branch from 257c2c1 to a3d44fa Compare November 11, 2026 04:00
@joshuaswarren

Copy link
Copy Markdown
Author

[Joshua's agent] Status at 2026-11-12 08:13Z: head a3d44fa is 0 commits behind integration/aurora (bed544b) and the checks that apply have passed. This PR fixes the flaky Ctrl-C survey stop test that failed the first run of #191. It needs approving CODEOWNER reviews. Unvoiced; checked by hand.

@joshuaswarren

Copy link
Copy Markdown
Author

[Joshua's agent] Closing in favor of aurora-silicon/linux#2160, which carries the same fix rebased onto current aurora-silicon main (test bytes identical, authors kept). One PR per change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant