thiras · Sep 17, 2026 · Sep 17, 2026 · Sep 17, 2026
diff --git a/AGENTS.md b/AGENTS.md
@@ -50,7 +50,7 @@ economic claims — those trace to the deCDN ADRs. If something here states a pr
 ```
 ansible/                # the deployment project (DevSec-hardened, lean roles)
   playbooks/            # site.yml (decdn node)
-  roles/                # baseline, decdn_node
+  roles/                # baseline, decdn_node, grafana_alloy
   inventory/ galaxy/ molecule/    # see ansible/README.md
 charts/
   decdn-node/           # Helm chart for the node on Kubernetes (see its README.md)
@@ -79,6 +79,24 @@ charts/
   `molecule/schema` scenario checks the rendered key set against a committed inventory of
   upstream field names. Re-sync both when bumping the pinned decdn version.
 
+- **`ansible/roles/grafana_alloy`** — opt-in Grafana Cloud observability for that node,
+  one mirrored flag (`decdn_grafana_cloud_enabled`, `false` by default) driving a
+  loopback-only Grafana Alloy agent: the node's `/metrics`, the **machine** itself
+  (Alloy's in-process `node_exporter`, curated collector set, `systemd` collector scoped
+  to the units that matter), **journald** → Grafana Cloud Loki, Alloy's own health, and
+  the daemon's OTLP spans. Host metrics and logs carry `job="integrations/node_exporter"`
+  so Grafana Cloud's prebuilt Linux Server dashboards work unmodified. Only the API token
+  is host-provisioned (`0600 /etc/grafana-alloy.env`, read via `sys.env` — Alloy has no
+  `--config.expand-env`); the non-secret endpoints/instance IDs are inventory variables
+  that fall back to their `GC_…` env key when empty, and preflight refuses a token in any
+  of them. Two hardening relaxations are conditional on the signals being on
+  (`ProtectHome=read-only` for correct filesystem metrics, `SupplementaryGroups=
+  systemd-journal adm` for journal access — without which collection is silently empty);
+  teardown is gated on the managed-by marker in the unit, so a foreign Alloy is never
+  touched. **`make lint-alloy` is the gate that matters** — the molecule stub exits 0 for
+  everything, so only the real pinned binary proves the rendered config loads. See
+  `ansible/roles/grafana_alloy/README.md`.
+
 - **`charts/decdn-node/`** — the same node on Kubernetes: a one-replica StatefulSet (one
   release = one identity) on the upstream daemon-only image (`ghcr.io/decdn/decdn-node`;
   unpublished, so `image.tag`/`image.digest` is required), PVC data dir, a `prepare` init

diff --git a/ansible/README.md b/ansible/README.md
@@ -187,26 +187,48 @@ primitives underneath it. All take `--dry-run`. See
 ### Grafana Cloud observability (opt-in)
 
 The play ships a third role, `grafana_alloy`, that installs a loopback-only
-[Grafana Alloy](https://grafana.com/docs/alloy/) agent scraping the node's `/metrics`
-and receiving its OTLP span exports (`127.0.0.1:4317`), shipping both to your Grafana
-Cloud org — off by default, driven by ONE mirrored inventory flag:
+[Grafana Alloy](https://grafana.com/docs/alloy/) agent shipping five signals to your
+Grafana Cloud org — the node's `/metrics`, **the machine itself** (CPU/memory/disk/
+filesystem/network/load plus per-unit state, via Alloy's in-process `node_exporter`),
+**journald**, the agent's own health, and the daemon's OTLP spans (`127.0.0.1:4317`) —
+off by default, driven by ONE mirrored inventory flag:
 
 ```yaml
 # group_vars/host_vars — drives BOTH roles from one knob
 decdn_grafana_cloud_enabled: true
 ```
 
-Provision the credentials **on the target host** (they never transit this repo or the
-control machine):
+Machine metrics, journald and Alloy's own health come with it (each has a
+`grafana_alloy_*_enabled` sub-knob, all defaulting to `true`). Host metrics and logs
+carry `job="integrations/node_exporter"`, so Grafana Cloud's prebuilt **Linux Server**
+dashboards and alert rules work unmodified.
+
+Only the API token is a secret. Put the non-secret connection settings in inventory
+once for the fleet (`grafana_alloy_prom_url`, `_prom_username`, `_otlp_endpoint`,
+`_loki_url`, `_loki_username` — note the Loki instance ID differs from the Prometheus
+one), and provision just the token **on each target host** (it never transits this repo
+or the control machine):
 
 ```bash
 umask 077
+printf 'GC_API_TOKEN=glc_…\n' > grafana-alloy.env
 sudo install -m 600 -o root -g root grafana-alloy.env /etc/grafana-alloy.env
 ```
 
-with `roles/grafana_alloy/files/grafana-alloy.env.example` as the template (remote-write
-URL, OTLP endpoint, instance id, API token — all four keys required, https only). Setup,
-rotation, cost/cardinality guardrails and rollback: see
+Each of those variables is optional: left empty, the value is read from the matching
+`GC_…` key in that file instead (`roles/grafana_alloy/files/grafana-alloy.env.example`
+lists them all; URLs must be https).
+
+**Upgrading a host deployed before machine monitoring existed:** journald shipping is on
+by default and needs a Loki endpoint + instance ID the old four-key file does not carry,
+so preflight fails until you either set `grafana_alloy_loki_url` / `_loki_username` in
+inventory, add `GC_LOKI_URL` / `GC_LOKI_USERNAME` to the env file, or set
+`grafana_alloy_logs_enabled: false`. The daemon's own metrics also gain an explicit
+`job="decdn-node"` label (previously the implicit `prometheus.scrape.decdn_node`) —
+`grafana_alloy_node_job: ""` restores the old identity.
+
+Setup, rotation, cost/cardinality guardrails, the two systemd-hardening relaxations
+machine monitoring requires, and rollback: see
 [`roles/grafana_alloy/README.md`](roles/grafana_alloy/README.md). The Helm chart path is
 separate and deliberately untouched.
 
@@ -239,7 +261,9 @@ interleaving gets in the way of reading a failure.
 `grafana-cloud` runs against a *stub* Alloy that exits 0 for every subcommand, so it
 proves the role's plumbing but cannot prove the rendered `config.alloy` is loadable.
 That gap is closed by `make lint-alloy` (repo root; CI job `alloy-config`), which renders
-the templates in several variable combinations and runs the **real**, digest-pinned Alloy
+the templates in eight variable combinations — including one with every observability
+sub-knob off, which must reproduce the pre-machine-monitoring pipeline exactly — and runs
+the **real**, digest-pinned Alloy
 binary's `alloy validate` over them plus a check that every `ExecStart` flag actually
 exists in `alloy run --help`. Re-run it when bumping `grafana_alloy_version`. See
 [`tests/alloy-config/`](tests/alloy-config/).
@@ -273,7 +297,7 @@ the RPC URL when it is not provisioned on the host instead). Highlights:
 | `decdn_rpc_url` + 3 contract addresses | `""` | **required** per node — `rpc_url` from a host-provisioned `0600 /etc/decdn/decdn.env` (preferred) *or* `host_vars/<node>/secret.yml`, addresses in `main.yml`; sourced from an ADR/deployment. |
 | `decdn_region` / `decdn_bind_port` / `decdn_rate_per_mb` | `""` / `4433` / `10` | node identity, QUIC port, USDC base units/MB. |
 | `decdn_env_checksum_file` / `decdn_env_overwrite_host_file` | `/etc/decdn/.decdn.env.sha256` / `false` | Provenance record for the secret env file (`0600 root`), and the opt-in that lets an inventory `decdn_rpc_url` overwrite a host-edited one. |
-| `decdn_grafana_cloud_enabled` | `false` | ONE mirrored knob (identical default in both roles) wiring on Grafana Cloud observability: installs + configures `grafana_alloy` AND injects `otlp_endpoint` into `node.toml`. Credentials are provisioned per role README; label/cost guardrails there too. |
+| `decdn_grafana_cloud_enabled` | `false` | ONE mirrored knob (identical default in both roles) wiring on Grafana Cloud observability: installs + configures `grafana_alloy` — node metrics, machine metrics, journald, agent health — AND injects `otlp_endpoint` into `node.toml`. Only the API token is provisioned per host; the rest are inventory variables. Label/cost guardrails in the role README. |
 
 ---
 

diff --git a/ansible/molecule/grafana-cloud/converge.yml b/ansible/molecule/grafana-cloud/converge.yml
@@ -39,6 +39,20 @@
     # Sanity-coupling override: verify.yml greps the RENDERED config for this value,
     # proving template plumbing passes through rather than constants being checked.
     grafana_alloy_batch_send_size: 512
+
+    # Machine monitoring stays at its defaults (host metrics + agent self-metrics
+    # + journald all ON) so the positive path covers the shape a real deploy
+    # gets. The systemd collector and the journal itself collect nothing useful
+    # in a container — irrelevant here, since this scenario proves plumbing and
+    # `make lint-alloy` proves the config loads.
+    #
+    # The Loki URL comes from inventory while its username stays in the host env
+    # file (see prepare.yml): one converge exercises both credential paths.
+    grafana_alloy_loki_url: https://logs-prod-xx.molecule.invalid/loki/api/v1/push
+    # DELIBERATELY UNQUOTED: a numeric instance ID arrives from YAML as an int,
+    # so every gate and the template must coerce it instead of taking its length
+    # or emitting a bare number into a string field.
+    grafana_alloy_prom_username: 1234567
   roles:
     - role: grafana_alloy
       tags: [observability]

diff --git a/ansible/molecule/grafana-cloud/prepare.yml b/ansible/molecule/grafana-cloud/prepare.yml
@@ -40,6 +40,12 @@
 
     # Values are throwaway; only the SHAPES matter to the role's greps (https://,
     # non-empty). Written directly at 0600 because preflight refuses anything else.
+    #
+    # Deliberately does NOT carry GC_LOKI_URL: converge supplies that one from
+    # inventory instead, so this fixture exercises BOTH halves of the credential
+    # split at once — the env-file path (token, prom, OTLP, Loki username) and
+    # the inventory path (Loki URL), including preflight's rule that a key is
+    # only required when its variable is empty.
     - name: Stage the Grafana Cloud credential fixture on the host
       ansible.builtin.copy:
         dest: /etc/grafana-alloy.env
@@ -50,4 +56,5 @@
           GC_PROM_REMOTE_WRITE_URL=https://prometheus-prod-xx.molecule.invalid/api/prom/push
           GC_OTLP_ENDPOINT=https://otlp-gateway-prod-xx.molecule.invalid/otlp
           GC_PROM_USERNAME=999888777
+          GC_LOKI_USERNAME=999888778
           GC_API_TOKEN=molecule-test-token
diff --git a/ansible/molecule/grafana-cloud/verify.yml b/ansible/molecule/grafana-cloud/verify.yml
@@ -98,6 +98,70 @@
           credential indirection) or carries syntax Alloy rejects. Got:
           {{ ga_config }}
 
+    # --- Machine monitoring ---------------------------------------------------------
+    - name: Assert the host, agent and journald pipelines rendered
+      ansible.builtin.assert:
+        that:
+          # Machine metrics: exporter -> target relabel -> scrape -> remote_write.
+          - >-
+            'prometheus.exporter.unix "host"' in ga_config
+          - >-
+            'discovery.relabel "host"' in ga_config
+          - >-
+            'prometheus.scrape "host"' in ga_config
+          # The job label Grafana Cloud's Linux Server integration dashboards key
+          # off. Changing it silently unwires every prebuilt dashboard, so it is
+          # asserted rather than left to review.
+          - >-
+            '"integrations/node_exporter"' in ga_config
+          # Agent self-monitoring: tells "node went quiet" from "agent wedged".
+          - >-
+            'prometheus.exporter.self "alloy"' in ga_config
+          - >-
+            '"integrations/alloy"' in ga_config
+          # journald -> Loki, with the field mapping that must live in the
+          # rules-only ruleset (those labels are stripped downstream).
+          - >-
+            'loki.source.journal "host"' in ga_config
+          - >-
+            'loki.relabel "journal_rules"' in ga_config
+          - >-
+            '__journal__systemd_unit' in ga_config
+          - >-
+            'loki.write "cloud"' in ga_config
+          # The node's own series carry an explicit job rather than inheriting
+          # the scrape component's ID.
+          - >-
+            '"job"         = "decdn-node"' in ga_config
+        fail_msg: >-
+          config.alloy is missing a machine-monitoring pipeline (host metrics,
+          agent self-metrics or journald). Got: {{ ga_config }}
+
+    # The credential split: whatever inventory supplies becomes a literal, and its
+    # sys.env lookup MUST disappear; everything else keeps reading the env file.
+    - name: Assert the inventory/env credential split rendered both ways
+      ansible.builtin.assert:
+        that:
+          - >-
+            '"https://logs-prod-xx.molecule.invalid/loki/api/v1/push"' in ga_config
+          - >-
+            'sys.env("GC_LOKI_URL")' not in ga_config
+          - >-
+            'sys.env("GC_LOKI_USERNAME")' in ga_config
+          # The unquoted numeric instance ID in converge must reach the config as
+          # a QUOTED string (Alloy's basic_auth username is a string) and must
+          # have survived every preflight gate that takes its length.
+          - >-
+            'username = "1234567"' in ga_config
+          - >-
+            'sys.env("GC_PROM_USERNAME")' not in ga_config
+        fail_msg: >-
+          The inventory-vs-env credential split did not render as expected:
+          grafana_alloy_loki_url and the numeric grafana_alloy_prom_username were
+          set in inventory (so their sys.env lookups must be gone and the ID must
+          appear quoted) while GC_LOKI_USERNAME was left to the host env file (so
+          its lookup must remain). Got: {{ ga_config }}
+
     # --- Secret hygiene -------------------------------------------------------------
     - name: Read the credential fixture back
       ansible.builtin.slurp:
@@ -115,6 +179,12 @@
             'GC_PROM_REMOTE_WRITE_URL=https://' in ga_env_body
           - >-
             'GC_OTLP_ENDPOINT=https://' in ga_env_body
+          - >-
+            'GC_LOKI_USERNAME=999888778' in ga_env_body
+          # Deliberately absent: converge supplies the Loki URL from inventory,
+          # which is what makes preflight's conditional requirement meaningful.
+          - >-
+            'GC_LOKI_URL=' not in ga_env_body
         fail_msg: prepare.yml fixture diverged from what the greps require
       vars:
         ga_env_body: "{{ ga_env_b64.content | b64decode }}"
@@ -195,6 +265,17 @@
           # upstream default.
           - >-
             '--server.http.listen-addr=127.0.0.1:' in ga_unit
+          # Host metrics are on, so ProtectHome must be relaxed to read-only:
+          # with ProtectHome=true the filesystem collector measures an empty
+          # tmpfs for /home and /root, and a full disk there never alerts. The
+          # privilege boundary is unchanged (still no write access).
+          - >-
+            'ProtectHome=read-only' in ga_unit
+          # Journald collection is on and this image has both groups, so the unit
+          # must grant them — without the membership loki.source.journal starts
+          # cleanly and ships nothing.
+          - >-
+            'SupplementaryGroups=systemd-journal adm' in ga_unit
         fail_msg: hardened Alloy unit is missing an expected directive
       vars:
         ga_unit: "{{ ga_unit_b64.content | b64decode }}"
@@ -204,6 +285,24 @@
     - name: Gather service facts
       ansible.builtin.service_facts:
 
+    - name: Read the group database
+      ansible.builtin.getent:
+        database: group
+
+    # The unit's SupplementaryGroups and the account's own membership must agree:
+    # systemd would refuse to start a unit naming a group that does not exist, and
+    # a missing membership makes journal collection silently empty.
+    - name: Assert the alloy account can read the journal
+      ansible.builtin.assert:
+        that:
+          - "'systemd-journal' in ansible_facts.getent_group"
+          - "'alloy' in ansible_facts.getent_group['systemd-journal'][2].split(',')"
+          - "'adm' in ansible_facts.getent_group"
+          - "'alloy' in ansible_facts.getent_group['adm'][2].split(',')"
+        fail_msg: >-
+          The alloy account is not a member of systemd-journal and adm, so
+          loki.source.journal would start without error and collect nothing.
+
     - name: Assert both services are running
       ansible.builtin.assert:
         that:
@@ -215,6 +314,56 @@
             and ansible_facts.services['decdn-node.service'].state == 'running'
         fail_msg: "alloy or decdn-node is not running — journalctl -u alloy -e / -u decdn-node -e"
 
+# Also mutates the converged host, so it runs after every assertion above and
+# before the teardown play below.
+- name: Verify turning logs off takes the journal grant back
+  hosts: all
+  become: true
+  tasks:
+    # Dropping SupplementaryGroups from the unit is not enough on its own:
+    # systemd then falls back to the ACCOUNT's supplementary groups, so a
+    # membership left over from the logs-enabled run would silently keep journal
+    # read access alive after the operator turned logs off.
+    - name: Re-run the role with only the logs signal disabled
+      ansible.builtin.include_role:
+        name: grafana_alloy
+      vars:
+        decdn_grafana_cloud_enabled: true
+        grafana_alloy_install_method: manual
+        grafana_alloy_manual_bin_src: >-
+          {{ lookup('ansible.builtin.env', 'MOLECULE_SCENARIO_DIRECTORY') }}/files/alloy-stub
+        grafana_alloy_deployment_environment: production
+        grafana_alloy_region: US
+        grafana_alloy_logs_enabled: false
+
+    - name: Read the group database back
+      ansible.builtin.getent:
+        database: group
+
+    - name: Read the unit and config back
+      ansible.builtin.slurp:
+        src: "{{ item }}"
+      register: ga_logs_off
+      loop:
+        - /etc/systemd/system/alloy.service
+        - /etc/alloy/config.alloy
+
+    - name: Assert the journal grant is gone from both the unit and the account
+      ansible.builtin.assert:
+        that:
+          - "'alloy' not in (ansible_facts.getent_group['systemd-journal'][2] | default('')).split(',')"
+          - "'alloy' not in (ansible_facts.getent_group['adm'][2] | default('')).split(',')"
+          - "'SupplementaryGroups' not in ga_unit_off"
+          - "'loki.' not in ga_config_off"
+        fail_msg: >-
+          Turning grafana_alloy_logs_enabled off left journal access behind:
+          SupplementaryGroups in the unit or a stale systemd-journal/adm
+          membership on the alloy account would keep the agent able to read the
+          journal after the signal it was granted for was switched off.
+      vars:
+        ga_unit_off: "{{ ga_logs_off.results[0].content | b64decode }}"
+        ga_config_off: "{{ ga_logs_off.results[1].content | b64decode }}"
+
 # Runs LAST, after the idempotence phase and the assertions above, because it
 # deliberately mutates the converged host: it exercises the disable path, whose
 # whole job is to remove things.