SIGTERM can be silently dropped, leaving the daemon running until SIGKILL

[ndizazzo]

What happens

The runtime can silently drop a SIGTERM and keep running. Nothing is logged, no shutdown event is emitted, and the process stays up until something SIGKILLs it. It is intermittent, roughly 1 in 5 in CI.

Caught it on the Linux products / Linux product (vulkan) job of run 34620977716, failing scripts/ci-client-readiness-smoke.sh:

client readiness observed on port 49031
scripts/ci-client-readiness-smoke.sh: line 66: 875 Killed  ... client --mesh-discovery-mode mdns
client did not stop cleanly after SIGTERM within 15s

The captured daemon log from that run ends at the readiness lines. No shutdown_requested, no shutdown warnings, nothing after SIGTERM. The daemon never even started shutting down, so this is not the shutdown sequence being slow. The signal was lost.

The same job passed on rerun at the same SHA, and cpu, cuda, and rocm all passed in the original run on the same binary, so it is not backend specific. Nothing on the PR that surfaced it (#1765) touches the runtime.

Root cause

crates/mesh-llm-host-runtime/src/runtime/control_loop.rs:798 constructs a brand new signal future on every iteration of the main control loop:

loop {
    tokio::select! {
        // ...
        signal = wait_shutdown_signal() => { /* shutdown */ }
        // ...
    }
}

wait_shutdown_signal calls tokio::signal::unix::signal(SignalKind::terminate()), which registers a fresh watch::Receiver each time (tokio 1.53.1, signal/registry.rs:70):

fn register_listener(&self, event_id: EventId) -> watch::Receiver<()> {
    self.storage.event_info(event_id).unwrap().tx.subscribe()
}

and delivery is a plain changed() await (signal/mod.rs:67):

async fn make_future(mut rx: Receiver<()>) -> Receiver<()> {
    rx.changed().await.expect("signal sender went away");
    rx
}

The OS handler is global and installed on first use, so the process no longer dies by default disposition. But the pending flag is consumed by the broadcast (registry.rs:89, pending.swap(false) then tx.send(())). A receiver subscribed after that send starts at the current version and never sees it. If SIGTERM is delivered and broadcast while no receiver is alive, it is gone for good.

Every branch of that select drops the receiver and the next iteration subscribes a new one. The window is however long the branch body takes, and the dashboard tick is DASHBOARD_CONTEXT_USAGE_REFRESH_INTERVAL = 250ms (runtime/dashboard.rs:15), so the loop churns the receiver 4 times a second while awake. That is the 1 in 5.

local_model_only.rs:321 and serving_surface.rs:1638 have the same shape. The serving_surface one is behind #[expect(dead_code)], local_model_only is not.

Why it matters outside CI

systemctl stop and docker stop both send SIGTERM. A dropped one means the unit hangs until TimeoutStopSec and then gets SIGKILLed, which skips broadcast_leaving, the Nostr unpublish, and Endpoint::close. Peers then time the node out and report it failed instead of seeing it depart cleanly. So the ungraceful-shutdown path we deliberately built close_endpoint to avoid is exactly what a dropped SIGTERM produces.

Suggested fix

Create the Signal instances once, before the loop, and poll them from the select rather than constructing a new future per iteration. Something like:

pub(super) struct ShutdownSignals { /* held Signal instances */ }
impl ShutdownSignals {
    fn install() -> Self { /* register once */ }
    async fn recv(&mut self) -> &'static str { /* select over the held instances */ }
}

then signal = shutdown_signals.recv() => { ... } in the loop. Holding one receiver for the process lifetime closes the window. A watch or broadcast channel fed by a single dedicated signal task works too, and has the nice property that everything that needs to observe shutdown can subscribe.

Worth fixing all three call sites, and worth a regression test that sends SIGTERM in a loop while the control loop is ticking.