ingestor: 'send on closed channel' panic on SIGTERM when a watchdog force-reconnect is in flight

[dborup]

Problem

The ingestor can panic with send on closed channel on SIGTERM if an MQTT watchdog force-reconnect is in flight during shutdown.

Where

cmd/ingestor/mqtt_watchdog.go, in the force-reconnect path of the stall watchdog. The watchdog runs ForceReconnectFn in a goroutine and then calls emit(...) ("WATCHDOG reconnect attempt issued"). If shutdown closes the channel behind emit while that goroutine is still inside ForceReconnectFn, the later send panics.

Background

• Found in the independent review of port(upstream#1897): stop watchdog force-reconnect racing paho's retry loop #29 and described there as pre-existing. port(upstream#1897): stop watchdog force-reconnect racing paho's retry loop #29 did not touch it and makes it less likely, not more.
• It only matters at shutdown, but a panic on SIGTERM can hide the real shutdown path in logs and make restarts look like crashes.

Suggested fix

• Make the emit path shutdown-safe. For example:
  • use a context or done channel and select on it instead of sending to a possibly closed channel;
  • or stop and wait for in-flight force-reconnect goroutines before closing the channel.
• Add a test that triggers a force-reconnect whose ForceReconnectFn blocks, shuts the watchdog down, then unblocks it. The test should assert no panic, and run under -race.

No behaviour change for normal operation is expected.

[dborup]

Fixed by #210 (merged as a0086bdd).

• newAsyncEmit drops an emit after stop under a mutex, so a force-reconnect still in flight at SIGTERM can no longer panic with send on closed channel.
• Emit-after-stop is not counted as a queue-full drop.
• Covered by tests with the real runLivenessWatchdog under -race.