Watchdog: report a stalled event loop instead of freezing silently

[kdroidFilter]

🚀 Feature request

Detect an event loop that has stopped pumping messages, and report it instead of freezing silently.

Motivation

#640 was a self-deadlock in the Tao Windows event loop: a non-reentrant mutex held across a PeekMessageW that re-enters the window procedure. The app froze permanently, and nothing in Nucleus said a word about it.

Two reasons, and neither is a bug in the existing fatal-error dialog:

1. There is nothing to report. The fatal path (TaoApplication.rethrowPendingFatal) starts from a recorded Throwable. A deadlock produces no exception, no panic, no error code. As far as the JVM is concerned the thread is perfectly healthy — the reporter's own thread dump shows main as RUNNABLE / _thread_in_native, with no anomaly flagged anywhere. Only the OS notices, after ~5 s without pumping, and its only way of saying so is to ghost the window as (Not Responding).
2. The reporting point sits downstream of the blockage. rethrowPendingFatal() runs after nativeRunBlocking returns — deliberately, so every tao callback frame is unwound before a modal pump can re-enter tao's non-reentrant handler mutex (Tao: native error dialog for fatal errors (no-AWT replacement for the Swing default) #622). When the loop deadlocks, nativeRunBlocking never returns and that line is unreachable.

The practical cost: diagnosing #640 took a full round trip of thread dumps and a native stack walk from the reporter, for what a watchdog could have named on the spot.

Proposed solution

A daemon watchdog thread that polls IsHungAppWindow(hwnd) every few seconds.

That Win32 call is a pure query of state the OS already maintains — it is what the shell itself uses to decide whether to ghost a window. It sends nothing to the target thread, so the cost on the event-loop thread is zero, and the total cost is one syscall every 2–5 s on an otherwise sleeping thread. Verified empirically while investigating #640: it returned True exactly when the window ghosted and False in every healthy run.

On detection (after a short grace period):

• log SEVERE with Thread.getAllStackTraces() — that alone would have pointed straight at main sitting in nativeRunBlocking;
• optionally show the native error dialog, which must be opened from a fresh thread here: the event-loop thread is precisely the one that is stuck (same constraint as the Tao: native error dialog for fatal errors (no-AWT replacement for the Swing default) #622 fix);
• keep it opt-in or at least disable-able, alongside nucleus.tao.fatalErrorDialog.

Two caveats worth stating: the threshold is Windows' own (~5 s) and is not configurable, and the signal means "not responding", so it also fires for a live loop stuck in a long synchronous operation — which is arguably worth reporting anyway.

This would not surface the native frames (the PeekMessageW re-entry in #640), so a native stack from the user stays useful for root-causing. What it replaces is the silent freeze: "the UI thread stopped responding, here are the Java stacks" instead of nothing at all.

Alternatives considered

• SendMessageTimeout(hwnd, WM_NULL, …, SMTO_ABORTIFHUNG) as the probe — this was the original proposal here, and it is the wrong design on two counts. It wakes the loop whenever it is idle in GetMessage, so polling once a second wakes the UI thread once a second when it could have slept for minutes (relevant to a project that ships an energy-manager). Worse, a cross-thread sent message is exactly the ingredient Application Stops Responding When Moving Between Virtual Desktops on Windows 11 #640 needed: it sits pending and gets delivered inline from inside any PeekMessageW the loop makes, which is the re-entrancy that deadlocked it. A watchdog built this way would raise the odds of triggering the very bug class it is meant to observe. IsHungAppWindow avoids both problems because it touches only the OS, never the target thread.
• Heartbeat counter bumped by each loop iteration — cannot distinguish idle from deadlocked: a loop legitimately parked in GetMessage stops bumping it too.
• Rely on the OS ghost window — the status quo. It tells the user something is wrong but gives the developer nothing, and it is what made Application Stops Responding When Moving Between Virtual Desktops on Windows 11 #640 opaque.
• Debug-only re-entrancy assertions on the locks — worth doing too, but a different, narrower tool: it catches this specific bug class at development time, whereas the watchdog catches any cause of a stalled loop in the field.

Platforms

IsHungAppWindow is Windows-specific. macOS and Linux need their own liveness probe — with the same requirement that it must not perturb the loop it observes — or the watchdog stays Windows-only at first. Worth scoping as part of the work.