x86_64 SMP: gale thread-lifecycle subtest failures + mutex_api in-test hang (exposed after boot-hang fix #46)

[avrabe]

Follow-up to #46, which fixed the early-boot hang in the gale x86_64 SMP path. With the hang gone, the three SMP suites now boot and run, exposing two distinct gale-SMP-correctness issues (previously masked):

1. smp_threads (tests/kernel/threads/thread_apis) — 9 subtest failures
Stock (no gale): PROJECT EXECUTION SUCCESSFUL, 86/0. Gale overlay: 77 PASS / 9 FAIL, threads_lifecycle suite 86.67%. Failing subtests:

• test_resume_unsuspend_thread
• test_thread_name_user_get_set
• test_threads_abort_others
• test_threads_spawn_forever

Points at gale's thread_lifecycle / sched primitives diverging from stock under SMP.

2. smp_mutex (tests/kernel/mutex/mutex_api) — hangs inside the test
Boots and prints Running TESTSUITE mutex_api, then hangs (no further output, >40s). A gale mutex operation appears to deadlock/hang under SMP (distinct from the spinlock-validate boot hang).

Repro (local, qemu_x86_64 SMP, Zephyr SDK x86_64-zephyr-elf):

west build -b qemu_x86_64 -s zephyr/tests/kernel/threads/thread_apis \
  -- -DOVERLAY_CONFIG="<gale>/zephyr/gale_overlay.conf;<gale>/zephyr/gale_smp_overlay.conf" \
     -DZEPHYR_EXTRA_MODULES=<gale>
west build -t run

(smp_semaphore fully passes after #46 — 61/0 — so the harness/overlay is sound; these are primitive-specific.)

[avrabe]

Localized both (local qemu_x86_64 SMP repro, with #46's boot-hang fix applied so we get past boot):

smp_mutex — priority-inheritance / ownership bug, not a print issue. It hangs at:

Running TESTSUITE mutex_api
START - test_complex_inversion
Assertion failed at .../mutex/mutex_api/src/complex_inversion.c:279:
  mutex_api_test_complex_inversion: (thread_06_tid not equal to mutex_1.owner)

The zassert message prints cleanly (so printk/spinlock recursion is NOT involved here — #46 handled that). gale's mutex sets/tracks the wrong owner for mutex_1 under SMP, so test_complex_inversion's priority-inheritance chain doesn't resolve and the threads deadlock → hang. This is a genuine gale-mutex SMP correctness bug (ownership/PI), distinct from the boot hang.

smp_threads — 9 subtest failures (vs stock 86/0), threads_lifecycle suite: test_resume_unsuspend_thread, test_thread_name_user_get_set, test_threads_abort_others, test_threads_spawn_forever. These print clean FAILs (no hang), so they're functional divergences in gale's thread_lifecycle/sched primitives under SMP.

Next: read gale's mutex owner/PI path (FFI + the SMP ownership tracking) to find why mutex_1.owner diverges in the complex-inversion scenario; and diff the 4 thread-lifecycle behaviors vs stock. Both are gale-primitive SMP-correctness issues, not harness/boot problems.

[avrabe]

smp_mutex root-caused: lost-wakeup deadlock in gale's mutex handoff under SMP (local qemu_x86_64 SMP + gdb).

Decisive control: stock mutex_api passes on qemu_x86_64 SMP (20/0, test_complex_inversion PASS) — so this is a genuine gale-mutex bug, not test flakiness.

State at the failure (gdb, both via print mutex_1 and thread backtraces):

• mutex_1 = {owner = 0x0, lock_count = 0, wait_q = empty} — the mutex is fully free at the point test_complex_inversion expects thread_06 to own it.
• Both CPUs are halted in the idle thread — the system is quiescent. The test thread is blocked in WAIT_FOR_THREAD(06) waiting for thread_06 to signal "I took mutex_1", but thread_06 never signals and the mutex is free → deadlock.

Mechanism: thread_06 pended on mutex_1 while thread_07 held it; thread_07's k_mutex_unlock did not hand the mutex to thread_06 (z_unpend_first_thread returned NULL → owner = NULL, lock_count = 0), leaving 06 permanently blocked while the mutex sits free. A classic lost wakeup — a pend/unpend race in gale's decide→apply mutex path under SMP that stock's monolithic lock/unlock doesn't have. (Secondary: the NULL owner is also why the test appears to hang mid-assert — zassert's "...not %s", mutex_1.owner->name dereferences NULL.)

gale_mutex.c's lock/unlock owner logic mirrors stock structurally, so the divergence is in the atomicity of the pend/unpend handoff across the single global lock + the decide/apply split (and/or the PI adjust_owner_prio window). Next: make the decide+apply+pend/unpend sequence atomic to match stock's exact ordering, then re-run on qemu_x86_64 SMP.

[avrabe]

Correction — retracting my previous "lost-wakeup in gale mutex handoff" root-cause. It was inferred from owner=NULL + both-CPUs-idle, but an isolation matrix proves the gale mutex is not the cause. Measuring each config on qemu_x86_64 SMP (mutex_api):

config	test_complex_inversion	other
stock (no gale)	PASS	all PASS (20/0)
gale mutex + SMP only	PASS (0.24s)	FAIL test_mutex_apis.c:53 — k_mutex_lock(.., K_MSEC(t)) returned 0 instead of timing out
gale + SMP + SPINLOCK_VALIDATE only	PASS (0.23s)	FAIL ztest.c:251 cpu_hold (dt < CPU_HOLD_TIME_MS false — timing)
gale + SMP + SPINLOCK_VALIDATE+CPU_MASK+IPI (the CI gale_smp_overlay)	DEADLOCK (mutex_1 free, both CPUs idle)

So:

• The complex_inversion deadlock only appears with the full SMP overlay → it's a multi-primitive interaction involving CPU_MASK/IPI (gale mutex + gale spinlock_validate each pass complex_inversion on their own). NOT a mutex-handoff bug. Next: bisect CPU_MASK vs IPI (+spinlock_validate) to pin which combination loses the wakeup.
• Separately, gale mutex's timeout path is a real candidate bug: test_mutex_apis.c:53 expects a timed k_mutex_lock to fail, but gale returns success (0). Reproduces with gale mutex + SMP, no other SMP extras — so it's the mutex, independent of the deadlock.
• The cpu_hold timing failure is likely spinlock_validate overhead (timing-sensitive), lower priority.

Apologies for the earlier wrong call — the gdb state was real but the inference wasn't; the isolation matrix is the ground truth.

[avrabe]

Single-CPU control closes the diagnosis: gale mutex_api on qemu_cortex_m3 (no SMP) passes 20/0 — including all the timeout subtests. So gale's mutex core logic is correct; every failure in this issue is SMP-integration-specific (and stock passes on SMP, so it's gale's SMP path, not the test):

• test_mutex_apis:53 timeout — gale's timed-lock-vs-handoff has an SMP race (single-CPU: correct; SMP: returns success instead of timing out). Cleanest standalone target.
• complex_inversion deadlock — only with the full overlay → cpu_mask/ipi multi-primitive interaction.
• cpu_hold timing — spinlock_validate overhead.

Net: not core-primitive logic bugs (verified Rust + single-CPU both fine) — they're in the SMP integration layer (cpu_mask/ipi/spinlock_validate interactions + the timeout/pend SMP window). These are deeper SMP-race fixes; flagging for prioritization vs. the wasm-cross-LTO optimization track.