fix(roles): daemon-reload inside the restart handlers

[alpergundogdu]

Problem

Install decdn-node systemd unit notifies Reload systemd then Restart decdn-node, assuming the reload runs first. On site.yml it doesn't. baseline loads devsec.hardening.os_hardening via include_role, which appends that role's own Reload systemd handler to the play at run time. Handler lookup takes the last loaded handler of a name, so devsec's shadows the role's, and it runs at the end of the handler list, after the restart.

The internal fleet hit this (internal-devops #24):

11:12:57  unit written (WatchdogSec=60, from #78)
11:12:58  RUNNING HANDLER [decdn_node : Restart decdn-node]
11:13:00  RUNNING HANDLER [devsec.hardening.os_hardening : Reload systemd]

systemd restarted the node from its cached unit. The new unit was on disk and NeedDaemonReload=no, but the running node had WatchdogUSec=0 until a manual restart. Any unit change is silently not applied until the next restart.

Fix

Restart decdn-node and Restart alloy now set daemon_reload: true, so handler order no longer matters. This matches the internal fix (146163e).

Regression test (default scenario, also covers os-matrix)

• prepare.yml stages a stale decdn-node.service (/bin/sleep infinity, no WatchdogSec), loaded and running, so converge changes a loaded unit. A brand-new unit is read from disk on first use and would not reproduce the bug.
• converge.yml loads a test-only role (molecule/default/roles/shadow_reload_systemd) with include_role, giving the same handler shadowing and order as devsec. It also sets decdn_node_watchdog_sec: 60.
• The stub now heartbeats WATCHDOG=1 to NOTIFY_SOCKET every WATCHDOG_USEC/2, like the real daemon, so it survives the watchdog.
• verify.yml reads /proc/<MainPID> and asserts the running process is the stub and that it was started with WATCHDOG_USEC=60000000. Only WATCHDOG_* variables are read, never the RPC URL.

Checked locally both ways:

• Without the fix: handler order was the same as on the fleet (restart, then the shadowing reload). Verify failed with Command line: /bin/sleep infinity.
• With the fix: molecule test -s default passes (converge, idempotence and verify).

Checks

• make lint-ansible passes (production profile)
• pre-commit run --all-files: every hook except markdownlint-cli2 passes. That hook couldn't install locally (node 24.14 < 24.15 required), so CI covers the CHANGELOG entry.
• molecule test -s default passes locally; os-matrix shares this prepare/converge and is left to CI.
• ansible/galaxy/CHANGELOG.md [Unreleased] / Fixed updated

🤖 Generated with Claude Code

[Copilot]

Copilot review overview

🟢 Approval recommended

The fix is narrowly scoped, aligns with Ansible/systemd behavior, and is backed by a targeted Molecule regression that reproduces the handler-shadowing failure mode.

Review effort: Lite
Findings: 1

Open (1)

• Guard against empty MainPID before reading /proc · New

What changed in this PR

This PR hardens Ansible systemd restart behavior so updated unit files take effect immediately, even when a later-loaded role (e.g., devsec.hardening.os_hardening via include_role) shadows the Reload systemd handler name and changes handler ordering at runtime.

Changes:

• Make Restart decdn-node and Restart alloy handlers perform daemon_reload: true to decouple correctness from handler ordering.
• Add/extend Molecule (default + os-matrix) regression coverage by staging a stale loaded unit and intentionally shadowing Reload systemd, then asserting the running process reflects the updated unit/watchdog.
• Document the fix in the Galaxy changelog.

File	Description
ansible/​roles/​grafana_alloy/​handlers/​main.yml	Ensures Alloy restart reloads systemd so unit rewrites apply immediately.
ansible/​roles/​decdn_node/​handlers/​main.yml	Ensures decdn-node restart reloads systemd so unit rewrites apply immediately.
ansible/​molecule/​default/​molecule.yml	Extends roles path so the test-only shadow role can be included.
ansible/​molecule/​os-matrix/​molecule.yml	Mirrors the default scenario roles-path change for matrix reuse.
ansible/​molecule/​default/​prepare.yml	Stages and starts a stale, loaded decdn-node unit to reproduce the cached-unit restart behavior.
ansible/​molecule/​default/​converge.yml	Enables watchdog and loads a shadow Reload systemd handler to reproduce the ordering issue.
ansible/​molecule/​default/​verify.yml	Verifies the running process reflects the updated unit by checking cmdline and WATCHDOG_USEC.
ansible/​molecule/​default/​roles/​shadow_reload_systemd/​handlers/​main.yml	Adds a test-only role that shadows Reload systemd to simulate devsec’s behavior.
ansible/​molecule/​default/​files/​decdn-node-stub	Adds watchdog heartbeating so the stub survives watchdog-enabled units.
ansible/​galaxy/​CHANGELOG.md	Records the behavioral fix for unit updates taking effect on the converge that writes them.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.