feat(ansible): monitor the machine, not just the daemon, in grafana_alloy

[thiras]

Why

roles/grafana_alloy shipped exactly two signals: a scrape of the deCDN daemon's own
/metrics and the OTLP the daemon pushes at the loopback receivers. Nothing observed the
machine. A full disk, a saturated CPU, a lost NIC, or a stopped decdn-node.service
showed up as blank dashboards with nothing to say why — and a wedged agent was
indistinguishable from a healthy, silent node.

Second complaint: first-run setup made operators hand-write four Grafana Cloud values into a
root-owned 0600 file on every host, even though only one of them is a secret.

What

Three new signals, each behind its own sub-knob under the existing
decdn_grafana_cloud_enabled flag (itself still false by default):

Signal	How	Job label
Machine metrics	Alloy's in-process prometheus.exporter.unix — 13 curated collectors + systemd scoped to (decdn-node|alloy|ssh|sshd)\.service	integrations/node_exporter
Host + service logs	journald → Grafana Cloud Loki, debug + noisy units dropped, max_age bounding the post-outage replay	integrations/node_exporter
Agent health	prometheus.exporter.self	integrations/alloy

The integrations/node_exporter label is the point: Grafana Cloud's prebuilt Linux
Server dashboards and alert rules key off it, so they light up with no dashboard work.

Still loopback-only. The exporters run inside the agent — no new listener, no new port,
no firewall hole. Two systemd directives are relaxed only when the signal that needs them
is enabled, each with the rationale inline in the unit:

Directive	Change	Why
ProtectHome	true → read-only (host metrics on)	otherwise /home and /root usage is missing, so a disk-full alert on them can never fire
SupplementaryGroups	+ systemd-journal adm (logs on)	without both, the journal component starts cleanly and collects nothing

Everything else (ProtectSystem=strict, NoNewPrivileges, SystemCallFilter=@system-service,
PrivateDevices, …) stays; the curated collector set deliberately excludes the ones that
would need capabilities or a widened syscall filter.

One secret per host

The five non-secret connection settings (prom URL/username, OTLP endpoint, Loki
URL/username) are now inventory variables that fall back to their GC_… env key when left
empty, so a new host provisions one value:

umask 077
printf 'GC_API_TOKEN=glc_…\n' > grafana-alloy.env
sudo install -m 600 -o root -g root grafana-alloy.env /etc/grafana-alloy.env

GC_API_TOKEN has no inventory variable by design — the rendered config.alloy is
world-readable on the host — and preflight refuses anything token-shaped (glc_, glsa_,
eyJ, or a URL with embedded credentials) in the other five. No secret transits the control
machine: the role only greps key shapes on the target, never reading values back.

Guardrails the binary cannot provide

alloy validate accepts all three of these as perfectly valid config, so they are gated on
the control machine instead:

• An unknown collector name is silently ignored, not an error. Alloy resolves the set by
  map lookup, so diskstat (missing the final s) starts clean and ships no disk metrics
  forever. Names are checked against a committed inventory in the role's vars/.
• An empty set_collectors reads as unset, re-enabling the full ~45-collector default
  set — the opposite of what an operator who disabled everything wanted. A set that nothing
  survives is refused.
• disable_collectors does not win upstream: it is applied before enable_collectors,
  so vetoing systemd would leave it enabled while dropping its unit_include scoping —
  per-unit series for every unit on the box. The veto is subtracted from both lists at render
  time instead.

⚠️ Upgrade notes

1. The daemon's own series change identity. They now carry job="decdn-node" instead of
   the implicit job="prometheus.scrape.decdn_node" (Alloy's component ID — an
   implementation detail no dashboard should hard-code). Set grafana_alloy_node_job: "" to
   keep the old value.
2. Logs need Loki credentials. Journald shipping is on by default and the pre-existing
   four-key env file has no Loki endpoint or instance ID, so preflight fails until you either
   set grafana_alloy_loki_url / _loki_username in inventory (recommended — nothing to do
   per host), append GC_LOKI_URL / GC_LOKI_USERNAME to the env file, or set
   grafana_alloy_logs_enabled: false. The failure names both the missing key and the
   variable that would satisfy it. The other four keys are untouched.

Testing

• make lint-alloy — the gate that matters. Renders the templates in eight variable
  combinations and feeds all of them to the real pinned Alloy v1.19.2: alloy validate
  (component graph, not just syntax), an ExecStart-flag check against alloy run --help,
  and greps proving each Jinja branch actually switched — including that "every sub-knob off"
  reproduces the pre-machine-monitoring pipeline and that the token is never rendered as a
  literal. The molecule stub exits 0 for everything and cannot prove any of this.
• molecule grafana-cloud — plumbing + idempotence: the three pipelines render, the job
  labels are right, both credential halves work, the unit carries the relaxations, and the
  service account really joins the journal groups.
• molecule validation — eight new negative cases: unknown collector, empty effective
  collector set, every collector vetoed, cleartext Loki URL, access token in inventory,
  service-account token in inventory, bad max_age, logs on with no Loki credentials.
• make lint-ansible (production profile), make lint, make security-ansible (KICS: 0
  critical, 0 high), molecule default (teardown still removes everything).

Not in scope

The Helm chart (host monitoring on Kubernetes is the cluster's job), dashboards-as-code
(Grafana Cloud's integration ships its own), and per-process metrics for the daemon (the
processes collector conflicts with ProtectProc and the daemon already exports /metrics).

[Copilot]

🟡 Changes recommended

Unresolved moderate findings affect journal access, credential validation, and numeric ID handling.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Extends opt-in Grafana Alloy monitoring with host metrics, journald logs, and Alloy health telemetry.

Changes:

• Adds machine, self, and journal telemetry pipelines.
• Adds credential fallbacks, collector guardrails, and conditional systemd hardening.
• Expands real-binary, Molecule, and documentation coverage.

File summaries

File	Summary	Findings
ansible/tests/alloy-config/validate.sh	Validates rendered monitoring combinations and secret hygiene.	Moderate (1): Secret detection omits glsa_ and JWT-like eyJ tokens.
ansible/tests/alloy-config/render.yml	Renders eight configuration scenarios.	Nit (1): Legacy pipeline fixture inherits the new decdn-node job label.
ansible/roles/grafana_alloy/vars/main.yml	Adds the collector inventory.	—
ansible/roles/grafana_alloy/templates/config.alloy.j2	Adds host, self, journal, and credential pipelines.	Moderate (2): Numeric instance IDs are not normalized before length checks and JSON encoding.
ansible/roles/grafana_alloy/templates/alloy.service.j2	Adds conditional systemd hardening.	Moderate (1): A partial journal-group set is treated as sufficient and can silently prevent log collection.
ansible/roles/grafana_alloy/tasks/preflight.yml	Adds collector, endpoint, label, and credential validation.	Moderate (1): Numeric Prometheus IDs can fail raw length handling. Moderate (1): Numeric Loki IDs have the same issue. Moderate (3): Instance-ID validation accepts nonnumeric values.
ansible/roles/grafana_alloy/tasks/main.yml	Manages journal-reader group membership.	Moderate (2): Previously added journal memberships persist when logs are disabled. Moderate (2): Partial journal-group availability is allowed despite requiring both groups.
ansible/roles/grafana_alloy/README.md	Documents monitoring and credential behavior.	—
ansible/roles/grafana_alloy/files/grafana-alloy.env.example	Documents token-only provisioning.	—
ansible/roles/grafana_alloy/defaults/main.yml	Adds monitoring and endpoint defaults.	—
ansible/README.md	Updates observability documentation.	Nit (1): Signal inventory says four while Alloy health makes five.
ansible/molecule/validation/converge.yml	Adds negative validation cases.	—
ansible/molecule/grafana-cloud/verify.yml	Verifies pipelines, groups, and hardening.	—
ansible/molecule/grafana-cloud/prepare.yml	Updates credential fixtures.	—
ansible/molecule/grafana-cloud/converge.yml	Exercises Loki inventory configuration.	—
AGENTS.md	Documents the expanded Alloy role.	—

Review details

Suppressed comments (6)

ansible/README.md:194

• This section says the agent ships four signals but lists the node metrics, machine metrics, journald, and OTLP; Alloy self-health is also enabled by default and described immediately below. The signal inventory is inconsistent—count it as five and include Alloy health in this list, or remove it from the later claim.

[Grafana Alloy](https://grafana.com/docs/alloy/) agent shipping four signals to your
Grafana Cloud org — the node's `/metrics`, **the machine itself** (CPU/memory/disk/
filesystem/network/load plus per-unit state, via Alloy's in-process `node_exporter`),
**journald**, and the daemon's OTLP spans (`127.0.0.1:4317`) — off by default, driven
by ONE mirrored inventory flag:

ansible/roles/grafana_alloy/tasks/preflight.yml:365

• The ID validation above coerces grafana_alloy_prom_username to a string, so an unquoted YAML numeric ID such as 1234567 passes, but this conditional applies length to the raw integer. That makes a documented/valid numeric inventory value fail with a Jinja type error before the credential check; coerce it to a string here (and in the template).

      required: "{{ grafana_alloy_prom_username | length == 0 }}"

ansible/roles/grafana_alloy/tasks/preflight.yml:377

• The same raw-value length problem exists for the Loki instance ID: the preceding validator accepts an unquoted numeric grafana_alloy_loki_username, but this expression then tries to take length of the integer. Normalize it before this conditional so valid numeric inventory values do not fail preflight.

      required: "{{ grafana_alloy_logs_enabled | bool and grafana_alloy_loki_username | length == 0 }}"

ansible/roles/grafana_alloy/templates/alloy.service.j2:38

• This condition treats one existing group as sufficient even though the role's comments and README require both systemd-journal and adm. On a host missing the other group, the unit gets a partial SupplementaryGroups setting, no warning is emitted, and journald can start while collecting nothing—the silent failure this guard is meant to prevent. Handle partial availability explicitly (or fail when logs are enabled) rather than using length > 0.

{% if grafana_alloy_logs_enabled | default(false) | bool and (_ga_journal_groups | default([]) | length > 0) %}

ansible/tests/alloy-config/render.yml:179

• This fixture is described as reproducing the pre-machine-monitoring pipeline exactly, but it inherits the new default grafana_alloy_node_job: decdn-node, so the daemon series still change identity from the old component-derived job. Set grafana_alloy_node_job: "" for this legacy case or narrow the claim/assertions to only the added component groups.

        grafana_alloy_logs_enabled: false

ansible/tests/alloy-config/validate.sh:112

• This hygiene check only recognizes glc_, while the role's token gate explicitly treats glsa_ and JWT-like eyJ values as credentials too. A literal service-account/JWT token could therefore be rendered into a matrix config without failing make lint-alloy; keep this render gate aligned with the preflight patterns.

  if grep -qF 'glc_' "$config"; then

• Files reviewed: 16/16 changed files
• Comments generated: 4
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[thiras]

Reviewed the Copilot findings; fixed the valid ones in 57f6839.

Fixed

Finding	Resolution
Numeric instance IDs break length / tojson (macro + 5 required: flags)	Coerced with | string in both places. An unquoted 1234567 now flows through the inventory render case and the grafana-cloud scenario, asserting it lands as username = "1234567" — quoted, which is what Alloy's basic_auth types it as.
Journal membership survives turning logs off	Real leak: dropping SupplementaryGroups is not enough, systemd falls back to the account's groups. The role now revokes systemd-journal/adm on that transition while preserving any other membership, and a new verify play exercises exactly that flip (the earlier play already asserts the grant exists, so the revocation assert cannot pass vacuously).
Warning only fires when both groups are missing	Now fires whenever either is, naming which.
validate.sh literal-token scan only knows glc_	Aligned with the preflight gate: glc_|glsa_|eyJ.
legacy render case inherits the new decdn-node job	Sets grafana_alloy_node_job: "", plus an assert_lacks legacy.alloy '"job"' so "reproduces the old pipeline" is literally true.
README says four signals, lists five	Fixed.

Adjusted rather than applied

• "Restrict the instance ID to digits" — the value is the endpoint's basic-auth username. It is numeric on Grafana Cloud, but a tenant name on a self-hosted Mimir/Loki, and the endpoints are operator-configurable. Tightening to ^[0-9]+$ would reject a legitimate setup, so the contract in the message/docs was corrected instead of the charset.
• "Partial journal groups should fail" — both groups are not actually required: systemd-journal owns the journal files and systemd-tmpfiles puts an ACL for adm on /var/log/journal, so either grants read. The "requires BOTH" claim was wrong in three comments and is corrected; a partial set is now reported rather than treated as fine or as fatal (failing would break container runs that legitimately have neither).

Re-validated: make lint-alloy (8 combinations, real Alloy 1.19.2), molecule grafana-cloud incl. idempotence and the new logs-off play, molecule validation, make lint-ansible, make lint.