Skip to content

feat(decdn_node): enable the systemd watchdog on the node unit #78

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alpergundogdu merged 2 commits into from
Sep 24, 2026
Merged

feat(decdn_node): enable the systemd watchdog on the node unit #78

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8efa265
feat(decdn_node): enable the systemd watchdog on the node unit
alpergundogdu Sep 24, 2026
be0447e
fix(decdn_node): make the systemd watchdog opt-in
alpergundogdu Sep 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions ansible/roles/decdn_node/defaults/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -406,6 +406,15 @@ decdn_cli_bin: /usr/local/bin/decdn
# templates/decdn-node.service.j2.
decdn_node_stop_timeout_sec: 300

# systemd WatchdogSec for the node unit (seconds); 0 omits the watchdog. The node
# heartbeats from its tokio runtime every half period, so a runtime whose workers
# never yield is killed and restarted instead of staying `active` while it serves
# nothing. It needs a decdn-node build that sends the heartbeat (decdn/decdn#2145):
# an older binary sends none and is killed every period. The role cannot tell which
# build it installs, so the watchdog is opt-in: set e.g. 60 once the deployed binary
# heartbeats. See templates/decdn-node.service.j2.
decdn_node_watchdog_sec: 0

# --- Lifecycle: backup + decommission (tasks_from: backup / decommission) -------
# Only read by those entry points (playbooks/backup.yml, playbooks/decommission.yml);
# a normal deploy ignores them. See docs/lifecycle.md.
Expand Down
18 changes: 16 additions & 2 deletions ansible/roles/decdn_node/templates/decdn-node.service.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,18 @@ TimeoutStopSec={{ decdn_node_stop_timeout_sec }}
Restart=always
RestartSec=5

{% if decdn_node_watchdog_sec | int > 0 %}
# Runtime-liveness watchdog. The node sends WATCHDOG=1 from a task on its tokio
# runtime every WatchdogSec/2, so the heartbeat stops when every runtime worker
# is held by work that never yields. The unit then stays `active` while the node
# serves nothing, logs nothing and answers no /metrics scrape. systemd kills it
# with SIGABRT at the deadline and Restart=always brings it back. The heartbeat
# arrives over NOTIFY_SOCKET, so the unit allows the main PID to notify and
# allows AF_UNIX below.
WatchdogSec={{ decdn_node_watchdog_sec }}
NotifyAccess=main
{% endif %}

# --- Hardening -------------------------------------------------------------
# StateDirectory owns {{ decdn_home }} (the only writable path under
# ProtectSystem=strict): node.secret, keystore.json, cache, redb state.
Expand All @@ -61,8 +73,10 @@ ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
ProtectClock=true
# AF_INET/AF_INET6 for QUIC; AF_NETLINK for interface/address discovery (STUN).
RestrictAddressFamilies=AF_INET AF_INET6 AF_NETLINK
# AF_INET/AF_INET6 for QUIC; AF_NETLINK for interface/address discovery (STUN);
# AF_UNIX for the watchdog heartbeat on NOTIFY_SOCKET.
RestrictAddressFamilies=AF_INET AF_INET6 AF_NETLINK{% if decdn_node_watchdog_sec | int > 0 %} AF_UNIX{% endif %}

RestrictNamespaces=true
RestrictSUIDSGID=true
LockPersonality=true
Expand Down
Loading