feat(decdn_node): enable the systemd watchdog on the node unit

[alpergundogdu]

Problem

When every tokio worker in decdn-node is held by work that never yields, the node serves nothing, logs nothing and answers no /metrics scrape. The process keeps running, though, so the unit stays active with Restart=always never firing.

Fix

decdn/decdn#2145 makes the node send WATCHDOG=1 from a task on its runtime when the unit sets WatchdogSec=. This PR turns that on in the role:

• WatchdogSec={{ decdn_node_watchdog_sec }} and NotifyAccess=main, rendered when decdn_node_watchdog_sec > 0. systemd kills a node that misses the deadline with SIGABRT, and Restart=always brings it back.
• AF_UNIX is added to RestrictAddressFamilies, because the heartbeat goes over NOTIFY_SOCKET.
• decdn_node_watchdog_sec defaults to 0, which omits all three directives. The role cannot tell whether the binary it installs sends the heartbeat, so the watchdog is opt-in. Set it (e.g. 60) once the deployed build includes feat(node): heartbeat the systemd watchdog from the runtime and test serve-loop starvation decdn#2145.

Enabling it: set decdn_node_watchdog_sec only for hosts whose decdn-node build includes decdn/decdn#2145. An older binary sends no heartbeat, so systemd would kill it every period.

I checked the rendered unit with trim_blocks=True (Ansible's default) for both 60 and 0.

🤖 Generated with Claude Code

[Copilot]

Copilot review overview

🟡 Changes recommended

The new watchdog is enabled by default (60s) despite being incompatible with older decdn-node binaries, which can cause immediate restart loops unless users proactively override it.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

• Default watchdog enablement can restart unsupported binaries · New

What changed in this PR

This PR enables systemd watchdog support for the decdn-node systemd unit rendered by the decdn_node Ansible role, so systemd can restart the node when it becomes runtime-stalled (e.g., tokio workers stop yielding) but the process remains alive.

Changes:

• Add conditional WatchdogSec= + NotifyAccess=main to the node unit when decdn_node_watchdog_sec > 0.
• Extend RestrictAddressFamilies= to conditionally allow AF_UNIX when watchdog is enabled (for NOTIFY_SOCKET).
• Introduce decdn_node_watchdog_sec role default (currently set to 60) with documentation about disabling it for older binaries.

File	Description
ansible/​roles/​decdn_node/​templates/​decdn-node.service.j2	Conditionally enables systemd watchdog directives and allows AF_UNIX only when needed.
ansible/​roles/​decdn_node/​defaults/​main.yml	Adds a new default variable controlling watchdog behavior and documents the version dependency.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.