bug: seekdb migration runner tests intermittently hang in CI #1924

Describe the bug

The "Test unified migrations with embedded seekdb" step of the code-seekdb job intermittently hangs. A tests/builtin/persistence/test_mysql_migration_runner.py node stops making progress, hits faulthandler_timeout=120, and the per-node timeout 180s wrapper ends the step with exit code 124. This blocks unrelated pull requests and has also failed a master push build. A rerun of the same commit usually passes.

The test and the workflow step were added in #1772.

Steps to reproduce

Not reproducible on demand. Observed occurrences:

The same step also timed out once on #1857; its rerun passed.

Expected behavior

Each seekdb runner node either passes or fails with an assertion. Passing nodes in the same logs finish in 14–36 seconds, so a two-minute stall is not ordinary slowness.

Actual behavior

Every timeout shows the same stack: the test thread is idle in the asyncio selector inside the migration connection wrapper, waiting on embedded seekdb I/O. No assertion or exception is raised first.

Timeout (0:02:00)!
Thread [pytest] (most recent call first):
  asyncio/base_events.py, in _run_once  (selectors.select)
  asyncio/runners.py, in run
  src/powercontext/builtin/persistence/migrations/connections.py, line 115 in run
  src/powercontext/builtin/persistence/migrations/mysql.py, line 271 in apply
  tests/builtin/persistence/test_mysql_migration_runner.py, line 181 in _apply
  tests/builtin/persistence/test_mysql_migration_runner.py, line 417 in test_mysql_baseline_adoption_recovers_an_empty_committed_version_table
##[error]Process completed with exit code 124.

The hanging node differs between runs, which points to the embedded seekdb connection or its lifecycle under asyncio.run(self._seekdb(...)) rather than to one test's logic. The seekdb-native.log and SQL trace artifacts uploaded by the job should show where the server stopped responding.

Suggested handling until the cause is known: make this step non-blocking (continue-on-error) or retry a timed-out node once, and keep collecting the native diagnostics.

Environment

• GitHub-hosted ubuntu-latest runners, Python 3.11 and 3.14
• uv run --locked --extra seekdb, embedded seekdb