From 122c15ff1f478c02edcb3b4442419f163f3d894a Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Thu, 17 Sep 2026 04:36:48 +0300 Subject: [PATCH 1/2] feat(ansible): monitor the machine, not just the daemon, in grafana_alloy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The role shipped two signals: a scrape of the node's own /metrics and the daemon's OTLP push. Nothing observed the host, so a full disk, a saturated CPU, a dead NIC or a stopped decdn-node.service showed up as blank dashboards with no explanation, and a wedged agent was indistinguishable from a healthy, silent node. Add three signals, each behind a sub-knob under the existing decdn_grafana_cloud_enabled flag: - host metrics via Alloy's in-process prometheus.exporter.unix — a curated 13-collector set plus systemd scoped to the units that matter, labelled job="integrations/node_exporter" so Grafana Cloud's prebuilt Linux Server dashboards and alerts work unmodified; - journald -> Grafana Cloud Loki, dropping debug priority and a noisy-unit set while the __journal* fields still exist, with max_age bounding the post-outage replay; - prometheus.exporter.self, so agent health is visible. Everything stays loopback-only: the exporters run inside the agent, so no new listener, port or firewall hole. Two hardening directives are relaxed only when the signal that needs them is on — ProtectHome=read-only (else /home and /root usage is missing and a disk-full alert there can never fire) and SupplementaryGroups=systemd-journal adm (without both, journal collection silently yields nothing). Also make first-run setup cheaper: the five non-secret connection settings become inventory variables that fall back to their GC_* env key when empty, so a new host provisions one secret instead of five values. The API token deliberately has no variable — config.alloy is world-readable on the host — and preflight refuses anything token-shaped in the other five. Guardrails the real binary cannot provide, because Alloy accepts all of these as valid config: - an unknown collector name is silently ignored, not an error, so a typo would ship fewer metrics forever; names are checked against a committed inventory in the role's vars/; - an empty set_collectors reads as unset and re-enables ~45 default collectors, so a set that nothing survives is refused; - upstream applies disable_collectors BEFORE enable_collectors, so a vetoed collector would stay enabled and lose its scoping block; the veto is subtracted from both lists at render time instead. Tested by rendering eight variable combinations through the real pinned Alloy 1.19.2 (make lint-alloy), the grafana-cloud molecule scenario for plumbing and idempotence, and eight new negative cases in the validation scenario. BREAKING CHANGE: the daemon's own series now carry job="decdn-node" instead of the implicit job="prometheus.scrape.decdn_node"; set grafana_alloy_node_job: "" to keep the old identity. Upgrading a host also needs Loki credentials (inventory or env keys) unless logs are turned off; preflight names both the missing key and the variable that satisfies it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- AGENTS.md | 20 +- ansible/README.md | 44 ++- ansible/molecule/grafana-cloud/converge.yml | 10 + ansible/molecule/grafana-cloud/prepare.yml | 7 + ansible/molecule/grafana-cloud/verify.yml | 91 ++++++ ansible/molecule/validation/converge.yml | 151 +++++++++- ansible/roles/grafana_alloy/README.md | 182 ++++++++++-- ansible/roles/grafana_alloy/defaults/main.yml | 123 ++++++++ .../files/grafana-alloy.env.example | 47 ++- ansible/roles/grafana_alloy/tasks/main.yml | 49 +++- .../roles/grafana_alloy/tasks/preflight.yml | 207 ++++++++++++- .../grafana_alloy/templates/alloy.service.j2 | 25 +- .../grafana_alloy/templates/config.alloy.j2 | 276 +++++++++++++++++- ansible/roles/grafana_alloy/vars/main.yml | 89 ++++++ ansible/tests/alloy-config/render.yml | 261 ++++++++++++----- ansible/tests/alloy-config/validate.sh | 75 +++++ 16 files changed, 1510 insertions(+), 147 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index a3a30e0..0392523 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -50,7 +50,7 @@ economic claims — those trace to the deCDN ADRs. If something here states a pr ``` ansible/ # the deployment project (DevSec-hardened, lean roles) playbooks/ # site.yml (decdn node) - roles/ # baseline, decdn_node + roles/ # baseline, decdn_node, grafana_alloy inventory/ galaxy/ molecule/ # see ansible/README.md charts/ decdn-node/ # Helm chart for the node on Kubernetes (see its README.md) @@ -79,6 +79,24 @@ charts/ `molecule/schema` scenario checks the rendered key set against a committed inventory of upstream field names. Re-sync both when bumping the pinned decdn version. +- **`ansible/roles/grafana_alloy`** — opt-in Grafana Cloud observability for that node, + one mirrored flag (`decdn_grafana_cloud_enabled`, `false` by default) driving a + loopback-only Grafana Alloy agent: the node's `/metrics`, the **machine** itself + (Alloy's in-process `node_exporter`, curated collector set, `systemd` collector scoped + to the units that matter), **journald** → Grafana Cloud Loki, Alloy's own health, and + the daemon's OTLP spans. Host metrics and logs carry `job="integrations/node_exporter"` + so Grafana Cloud's prebuilt Linux Server dashboards work unmodified. Only the API token + is host-provisioned (`0600 /etc/grafana-alloy.env`, read via `sys.env` — Alloy has no + `--config.expand-env`); the non-secret endpoints/instance IDs are inventory variables + that fall back to their `GC_…` env key when empty, and preflight refuses a token in any + of them. Two hardening relaxations are conditional on the signals being on + (`ProtectHome=read-only` for correct filesystem metrics, `SupplementaryGroups= + systemd-journal adm` for journal access — without which collection is silently empty); + teardown is gated on the managed-by marker in the unit, so a foreign Alloy is never + touched. **`make lint-alloy` is the gate that matters** — the molecule stub exits 0 for + everything, so only the real pinned binary proves the rendered config loads. See + `ansible/roles/grafana_alloy/README.md`. + - **`charts/decdn-node/`** — the same node on Kubernetes: a one-replica StatefulSet (one release = one identity) on the upstream daemon-only image (`ghcr.io/decdn/decdn-node`; unpublished, so `image.tag`/`image.digest` is required), PVC data dir, a `prepare` init diff --git a/ansible/README.md b/ansible/README.md index ed8c99a..a9ae0b5 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -187,26 +187,48 @@ primitives underneath it. All take `--dry-run`. See ### Grafana Cloud observability (opt-in) The play ships a third role, `grafana_alloy`, that installs a loopback-only -[Grafana Alloy](https://grafana.com/docs/alloy/) agent scraping the node's `/metrics` -and receiving its OTLP span exports (`127.0.0.1:4317`), shipping both to your Grafana -Cloud org — off by default, driven by ONE mirrored inventory flag: +[Grafana Alloy](https://grafana.com/docs/alloy/) agent shipping four signals to your +Grafana Cloud org — the node's `/metrics`, **the machine itself** (CPU/memory/disk/ +filesystem/network/load plus per-unit state, via Alloy's in-process `node_exporter`), +**journald**, and the daemon's OTLP spans (`127.0.0.1:4317`) — off by default, driven +by ONE mirrored inventory flag: ```yaml # group_vars/host_vars — drives BOTH roles from one knob decdn_grafana_cloud_enabled: true ``` -Provision the credentials **on the target host** (they never transit this repo or the -control machine): +Machine metrics, journald and Alloy's own health come with it (each has a +`grafana_alloy_*_enabled` sub-knob, all defaulting to `true`). Host metrics and logs +carry `job="integrations/node_exporter"`, so Grafana Cloud's prebuilt **Linux Server** +dashboards and alert rules work unmodified. + +Only the API token is a secret. Put the non-secret connection settings in inventory +once for the fleet (`grafana_alloy_prom_url`, `_prom_username`, `_otlp_endpoint`, +`_loki_url`, `_loki_username` — note the Loki instance ID differs from the Prometheus +one), and provision just the token **on each target host** (it never transits this repo +or the control machine): ```bash umask 077 +printf 'GC_API_TOKEN=glc_…\n' > grafana-alloy.env sudo install -m 600 -o root -g root grafana-alloy.env /etc/grafana-alloy.env ``` -with `roles/grafana_alloy/files/grafana-alloy.env.example` as the template (remote-write -URL, OTLP endpoint, instance id, API token — all four keys required, https only). Setup, -rotation, cost/cardinality guardrails and rollback: see +Each of those variables is optional: left empty, the value is read from the matching +`GC_…` key in that file instead (`roles/grafana_alloy/files/grafana-alloy.env.example` +lists them all; URLs must be https). + +**Upgrading a host deployed before machine monitoring existed:** journald shipping is on +by default and needs a Loki endpoint + instance ID the old four-key file does not carry, +so preflight fails until you either set `grafana_alloy_loki_url` / `_loki_username` in +inventory, add `GC_LOKI_URL` / `GC_LOKI_USERNAME` to the env file, or set +`grafana_alloy_logs_enabled: false`. The daemon's own metrics also gain an explicit +`job="decdn-node"` label (previously the implicit `prometheus.scrape.decdn_node`) — +`grafana_alloy_node_job: ""` restores the old identity. + +Setup, rotation, cost/cardinality guardrails, the two systemd-hardening relaxations +machine monitoring requires, and rollback: see [`roles/grafana_alloy/README.md`](roles/grafana_alloy/README.md). The Helm chart path is separate and deliberately untouched. @@ -239,7 +261,9 @@ interleaving gets in the way of reading a failure. `grafana-cloud` runs against a *stub* Alloy that exits 0 for every subcommand, so it proves the role's plumbing but cannot prove the rendered `config.alloy` is loadable. That gap is closed by `make lint-alloy` (repo root; CI job `alloy-config`), which renders -the templates in several variable combinations and runs the **real**, digest-pinned Alloy +the templates in eight variable combinations — including one with every observability +sub-knob off, which must reproduce the pre-machine-monitoring pipeline exactly — and runs +the **real**, digest-pinned Alloy binary's `alloy validate` over them plus a check that every `ExecStart` flag actually exists in `alloy run --help`. Re-run it when bumping `grafana_alloy_version`. See [`tests/alloy-config/`](tests/alloy-config/). @@ -273,7 +297,7 @@ the RPC URL when it is not provisioned on the host instead). Highlights: | `decdn_rpc_url` + 3 contract addresses | `""` | **required** per node — `rpc_url` from a host-provisioned `0600 /etc/decdn/decdn.env` (preferred) *or* `host_vars//secret.yml`, addresses in `main.yml`; sourced from an ADR/deployment. | | `decdn_region` / `decdn_bind_port` / `decdn_rate_per_mb` | `""` / `4433` / `10` | node identity, QUIC port, USDC base units/MB. | | `decdn_env_checksum_file` / `decdn_env_overwrite_host_file` | `/etc/decdn/.decdn.env.sha256` / `false` | Provenance record for the secret env file (`0600 root`), and the opt-in that lets an inventory `decdn_rpc_url` overwrite a host-edited one. | -| `decdn_grafana_cloud_enabled` | `false` | ONE mirrored knob (identical default in both roles) wiring on Grafana Cloud observability: installs + configures `grafana_alloy` AND injects `otlp_endpoint` into `node.toml`. Credentials are provisioned per role README; label/cost guardrails there too. | +| `decdn_grafana_cloud_enabled` | `false` | ONE mirrored knob (identical default in both roles) wiring on Grafana Cloud observability: installs + configures `grafana_alloy` — node metrics, machine metrics, journald, agent health — AND injects `otlp_endpoint` into `node.toml`. Only the API token is provisioned per host; the rest are inventory variables. Label/cost guardrails in the role README. | --- diff --git a/ansible/molecule/grafana-cloud/converge.yml b/ansible/molecule/grafana-cloud/converge.yml index 7cf302e..21c5b02 100644 --- a/ansible/molecule/grafana-cloud/converge.yml +++ b/ansible/molecule/grafana-cloud/converge.yml @@ -39,6 +39,16 @@ # Sanity-coupling override: verify.yml greps the RENDERED config for this value, # proving template plumbing passes through rather than constants being checked. grafana_alloy_batch_send_size: 512 + + # Machine monitoring stays at its defaults (host metrics + agent self-metrics + # + journald all ON) so the positive path covers the shape a real deploy + # gets. The systemd collector and the journal itself collect nothing useful + # in a container — irrelevant here, since this scenario proves plumbing and + # `make lint-alloy` proves the config loads. + # + # The Loki URL comes from inventory while its username stays in the host env + # file (see prepare.yml): one converge exercises both credential paths. + grafana_alloy_loki_url: https://logs-prod-xx.molecule.invalid/loki/api/v1/push roles: - role: grafana_alloy tags: [observability] diff --git a/ansible/molecule/grafana-cloud/prepare.yml b/ansible/molecule/grafana-cloud/prepare.yml index 9baf8c5..6d99a81 100644 --- a/ansible/molecule/grafana-cloud/prepare.yml +++ b/ansible/molecule/grafana-cloud/prepare.yml @@ -40,6 +40,12 @@ # Values are throwaway; only the SHAPES matter to the role's greps (https://, # non-empty). Written directly at 0600 because preflight refuses anything else. + # + # Deliberately does NOT carry GC_LOKI_URL: converge supplies that one from + # inventory instead, so this fixture exercises BOTH halves of the credential + # split at once — the env-file path (token, prom, OTLP, Loki username) and + # the inventory path (Loki URL), including preflight's rule that a key is + # only required when its variable is empty. - name: Stage the Grafana Cloud credential fixture on the host ansible.builtin.copy: dest: /etc/grafana-alloy.env @@ -50,4 +56,5 @@ GC_PROM_REMOTE_WRITE_URL=https://prometheus-prod-xx.molecule.invalid/api/prom/push GC_OTLP_ENDPOINT=https://otlp-gateway-prod-xx.molecule.invalid/otlp GC_PROM_USERNAME=999888777 + GC_LOKI_USERNAME=999888778 GC_API_TOKEN=molecule-test-token diff --git a/ansible/molecule/grafana-cloud/verify.yml b/ansible/molecule/grafana-cloud/verify.yml index a26e0df..2a99270 100644 --- a/ansible/molecule/grafana-cloud/verify.yml +++ b/ansible/molecule/grafana-cloud/verify.yml @@ -98,6 +98,62 @@ credential indirection) or carries syntax Alloy rejects. Got: {{ ga_config }} + # --- Machine monitoring --------------------------------------------------------- + - name: Assert the host, agent and journald pipelines rendered + ansible.builtin.assert: + that: + # Machine metrics: exporter -> target relabel -> scrape -> remote_write. + - >- + 'prometheus.exporter.unix "host"' in ga_config + - >- + 'discovery.relabel "host"' in ga_config + - >- + 'prometheus.scrape "host"' in ga_config + # The job label Grafana Cloud's Linux Server integration dashboards key + # off. Changing it silently unwires every prebuilt dashboard, so it is + # asserted rather than left to review. + - >- + '"integrations/node_exporter"' in ga_config + # Agent self-monitoring: tells "node went quiet" from "agent wedged". + - >- + 'prometheus.exporter.self "alloy"' in ga_config + - >- + '"integrations/alloy"' in ga_config + # journald -> Loki, with the field mapping that must live in the + # rules-only ruleset (those labels are stripped downstream). + - >- + 'loki.source.journal "host"' in ga_config + - >- + 'loki.relabel "journal_rules"' in ga_config + - >- + '__journal__systemd_unit' in ga_config + - >- + 'loki.write "cloud"' in ga_config + # The node's own series carry an explicit job rather than inheriting + # the scrape component's ID. + - >- + '"job" = "decdn-node"' in ga_config + fail_msg: >- + config.alloy is missing a machine-monitoring pipeline (host metrics, + agent self-metrics or journald). Got: {{ ga_config }} + + # The credential split: whatever inventory supplies becomes a literal, and its + # sys.env lookup MUST disappear; everything else keeps reading the env file. + - name: Assert the inventory/env credential split rendered both ways + ansible.builtin.assert: + that: + - >- + '"https://logs-prod-xx.molecule.invalid/loki/api/v1/push"' in ga_config + - >- + 'sys.env("GC_LOKI_URL")' not in ga_config + - >- + 'sys.env("GC_LOKI_USERNAME")' in ga_config + fail_msg: >- + The inventory-vs-env credential split did not render as expected: + grafana_alloy_loki_url was set in inventory (so its sys.env lookup must + be gone) while GC_LOKI_USERNAME was left to the host env file (so its + lookup must remain). Got: {{ ga_config }} + # --- Secret hygiene ------------------------------------------------------------- - name: Read the credential fixture back ansible.builtin.slurp: @@ -115,6 +171,12 @@ 'GC_PROM_REMOTE_WRITE_URL=https://' in ga_env_body - >- 'GC_OTLP_ENDPOINT=https://' in ga_env_body + - >- + 'GC_LOKI_USERNAME=999888778' in ga_env_body + # Deliberately absent: converge supplies the Loki URL from inventory, + # which is what makes preflight's conditional requirement meaningful. + - >- + 'GC_LOKI_URL=' not in ga_env_body fail_msg: prepare.yml fixture diverged from what the greps require vars: ga_env_body: "{{ ga_env_b64.content | b64decode }}" @@ -195,6 +257,17 @@ # upstream default. - >- '--server.http.listen-addr=127.0.0.1:' in ga_unit + # Host metrics are on, so ProtectHome must be relaxed to read-only: + # with ProtectHome=true the filesystem collector measures an empty + # tmpfs for /home and /root, and a full disk there never alerts. The + # privilege boundary is unchanged (still no write access). + - >- + 'ProtectHome=read-only' in ga_unit + # Journald collection is on and this image has both groups, so the unit + # must grant them — without the membership loki.source.journal starts + # cleanly and ships nothing. + - >- + 'SupplementaryGroups=systemd-journal adm' in ga_unit fail_msg: hardened Alloy unit is missing an expected directive vars: ga_unit: "{{ ga_unit_b64.content | b64decode }}" @@ -204,6 +277,24 @@ - name: Gather service facts ansible.builtin.service_facts: + - name: Read the group database + ansible.builtin.getent: + database: group + + # The unit's SupplementaryGroups and the account's own membership must agree: + # systemd would refuse to start a unit naming a group that does not exist, and + # a missing membership makes journal collection silently empty. + - name: Assert the alloy account can read the journal + ansible.builtin.assert: + that: + - "'systemd-journal' in ansible_facts.getent_group" + - "'alloy' in ansible_facts.getent_group['systemd-journal'][2].split(',')" + - "'adm' in ansible_facts.getent_group" + - "'alloy' in ansible_facts.getent_group['adm'][2].split(',')" + fail_msg: >- + The alloy account is not a member of systemd-journal and adm, so + loki.source.journal would start without error and collect nothing. + - name: Assert both services are running ansible.builtin.assert: that: diff --git a/ansible/molecule/validation/converge.yml b/ansible/molecule/validation/converge.yml index 265fe5e..b615e31 100644 --- a/ansible/molecule/validation/converge.yml +++ b/ansible/molecule/validation/converge.yml @@ -927,6 +927,152 @@ decdn_rejected: "{{ decdn_rejected + ['managed-paths'] }}" when: ansible_failed_task.name is match('^Require the rendered config to live inside') + - name: "Case ga-collector — unknown node_exporter collector name" + block: + - name: Run grafana_alloy with a bogus host collector + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + # An unknown collector is NOT a startup crash — Alloy resolves the + # set by map lookup and silently ignores what it does not know, so a + # typo ships fewer metrics forever. Only the committed name + # inventory in the role's vars/ can catch it, and only here. + grafana_alloy_host_collectors: [cpu, diskstat] + rescue: + - name: Record ga-collector rejection (only if the collector validator failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['host-collectors'] }}" + when: ansible_failed_task.name is match('^Validate the node_exporter collector names') + + - name: "Case ga-no-collectors — host metrics on with every collector list empty" + block: + - name: Run grafana_alloy with an empty collector set + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_host_collectors: [] + grafana_alloy_host_extra_collectors: [] + rescue: + - name: Record ga-no-collectors rejection (only if the non-empty gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['host-collectors'] }}" + when: ansible_failed_task.name is match('^Require a non-empty effective host collector set') + + # An empty set_collectors does NOT mean "measure nothing" to Alloy: it reads + # as unset and re-enables the full ~45-collector default set. Vetoing every + # collector must therefore be refused, not rendered. + - name: "Case ga-disable-all — every collector vetoed by the disable list" + block: + - name: Run grafana_alloy with the whole collector set disabled + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_host_collectors: [cpu] + grafana_alloy_host_extra_collectors: [systemd] + grafana_alloy_host_disable_collectors: [cpu, systemd] + rescue: + - name: Record ga-disable-all rejection (only if the non-empty gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['host-collectors'] }}" + when: ansible_failed_task.name is match('^Require a non-empty effective host collector set') + + - name: "Case ga-loki-http — cleartext Loki endpoint from inventory" + block: + - name: Run grafana_alloy with a non-https Loki URL + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_loki_url: http://logs-prod-xx.molecule.invalid/loki/api/v1/push + rescue: + - name: Record ga-loki-http rejection (only if the endpoint validator failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['endpoint-scheme'] }}" + when: ansible_failed_task.name is match('^Validate any inventory-provided Grafana Cloud endpoints') + + - name: "Case ga-inventory-token — access token smuggled into inventory" + block: + # config.alloy is 0644 on the host, so a token reaching an inventory + # variable would be rendered world-readable — the one thing the whole + # host-provisioned credential design exists to prevent. + - name: Run grafana_alloy with a token in a non-secret variable + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_prom_username: glc_eyJvIjoiMTIzNDU2NyJ9 + rescue: + - name: Record ga-inventory-token rejection (only if the token gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['inventory-token'] }}" + when: ansible_failed_task.name is match('^Refuse a Grafana Cloud token smuggled into inventory') + + - name: "Case ga-inventory-sa-token — service-account token smuggled into inventory" + block: + # glsa_ tokens carry no '@' and pass the instance-ID charset, so the + # gate must key off the prefix rather than shape alone. + - name: Run grafana_alloy with a service-account token in a non-secret variable + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_loki_username: glsa_AbCdEf0123456789_0a1b2c3d + rescue: + - name: Record ga-inventory-sa-token rejection (only if the token gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['inventory-token'] }}" + when: ansible_failed_task.name is match('^Refuse a Grafana Cloud token smuggled into inventory') + + - name: "Case ga-max-age — malformed journal catch-up window" + block: + - name: Run grafana_alloy with a bad max_age suffix + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_logs_max_age: 12hours + rescue: + - name: Record ga-max-age rejection (only if the knob validator failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['pipeline-knobs'] }}" + when: ansible_failed_task.name is match('^Validate the pipeline knobs') + + - name: "Case ga-loki-key — logs enabled with neither Loki key nor variable" + block: + # The Loki pair is required ONLY when logs are on and inventory supplies + # nothing: this proves the conditional gate still fires when it should. + - name: Stage a credential file with no Loki keys + ansible.builtin.copy: + dest: /etc/grafana-alloy.env + owner: root + group: root + mode: "0600" + content: | + GC_PROM_REMOTE_WRITE_URL=https://prometheus-prod-xx.molecule.invalid/api/prom/push + GC_OTLP_ENDPOINT=https://otlp-gateway-prod-xx.molecule.invalid/otlp + GC_PROM_USERNAME=999888777 + GC_API_TOKEN=molecule-test-token + + - name: Run grafana_alloy with logs enabled and no Loki credentials + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_logs_enabled: true + rescue: + - name: Record ga-loki-key rejection (only if the grep report gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['secret-content'] }}" + when: ansible_failed_task.name is match('^Report which credential key') + always: + - name: Remove the Loki-less credential fixture + ansible.builtin.file: + path: /etc/grafana-alloy.env + state: absent + - name: Confirm every bad value was rejected by the role's own validation ansible.builtin.assert: that: @@ -947,6 +1093,9 @@ "otlp-newline", "env-control-rpc", "env-control-extra", "env-gate", "env-extra", "env-content", "config-gate", "secret-file", "secret-file", "secret-path-nested", "secret-content", - "secret-content", "pipeline-knobs", "alloy-root-user", "alloy-root-group", + "secret-content", "secret-content", "pipeline-knobs", "pipeline-knobs", + "alloy-root-user", "alloy-root-group", "alloy-root-user-alias", "alloy-root-group-alias", + "host-collectors", "host-collectors", "host-collectors", + "endpoint-scheme", "inventory-token", "inventory-token", "managed-paths", "managed-paths", "managed-paths"] diff --git a/ansible/roles/grafana_alloy/README.md b/ansible/roles/grafana_alloy/README.md index 37858de..4878de5 100644 --- a/ansible/roles/grafana_alloy/README.md +++ b/ansible/roles/grafana_alloy/README.md @@ -1,17 +1,31 @@ # grafana_alloy -Opt-in Grafana Cloud Application Observability for an Ansible-deployed deCDN -node ([decdn/devops#55](https://github.com/decdn/devops/issues/55)): a dedicated, +Opt-in Grafana Cloud observability for an Ansible-deployed deCDN node +([decdn/devops#55](https://github.com/decdn/devops/issues/55)): a dedicated, loopback-only [Grafana Alloy](https://grafana.com/docs/alloy/) agent that - scrapes the node's `/metrics` (Prometheus exposition on loopback port 9090) and pushes it to your Grafana Cloud Prometheus, stamped with low-cardinality - identity labels, and + identity labels, +- scrapes **the machine itself** — CPU, memory, disk, filesystem, network, load, + PSI saturation and the state of the units that matter — through Alloy's + in-process `node_exporter`, labelled so Grafana Cloud's prebuilt **Linux + Server** dashboards and alert rules work with no dashboard wiring, +- ships **journald** (host and services) to your Grafana Cloud Loki with the same + `job`/`instance` pair, so logs and metrics correlate in those dashboards, +- reports **its own** health (`prometheus.exporter.self`), which is what + distinguishes "the node went quiet" from "the agent reporting on the node is + wedged", and - receives the daemon's OTLP span exports (`otlp_endpoint`) on gRPC `127.0.0.1:4317` and HTTP `127.0.0.1:4318`, probabilistically samples traces (default keep-ratio 0.25), batches them, and exports via OTLP/HTTP to your Grafana Cloud org. -The Helm-chart path is separate and deliberately untouched by this role. +Everything except the node's public QUIC port stays on loopback: the exporters +run *inside* the agent, so machine monitoring adds no listener and no firewall +hole. + +The Helm-chart path is separate and deliberately untouched by this role — on +Kubernetes, host-level telemetry is the cluster's job. ## Opt in @@ -23,24 +37,67 @@ host level only: decdn_grafana_cloud_enabled: true ``` -Then provision credentials **on the target host** (they never transit the control -machine): +Machine metrics, agent self-metrics and journald come with it (each has its own +`grafana_alloy_*_enabled` sub-knob, all defaulting to `true`). + +### Credentials: one secret per host, the rest in inventory + +Only the **API token** is a secret. The endpoint URLs and the two numeric +instance IDs are not, so they belong in inventory — written once for the fleet +rather than typed on every host: + +```yaml +# group_vars/.yml — non-secret, safe to commit +grafana_alloy_prom_url: https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom/push +grafana_alloy_prom_username: "1234567" # Prometheus instance ID +grafana_alloy_otlp_endpoint: https://otlp-gateway-prod-us-east-0.grafana.net/otlp +grafana_alloy_loki_url: https://logs-prod-006.grafana.net/loki/api/v1/push +grafana_alloy_loki_username: "7654321" # Loki instance ID — a DIFFERENT number +``` + +Then the only thing to provision **on the target host** is the token (it never +transits the control machine): ```bash umask 077 +printf 'GC_API_TOKEN=glc_…\n' > grafana-alloy.env sudo install -m 600 -o root -g root grafana-alloy.env /etc/grafana-alloy.env ``` -using `roles/grafana_alloy/files/grafana-alloy.env.example` as the template. The -credential path is intentionally restricted to a **direct child of `/etc`**. A +Every one of those variables is optional: left empty, its value is read from the +matching `GC_…` key in the env file exactly as before, and preflight requires the +key only then. Any mix of the two halves is valid. +`roles/grafana_alloy/files/grafana-alloy.env.example` shows the full key set. + +> **Upgrading an existing host.** Logs are on by default, and Loki needs an +> endpoint + instance ID that the pre-existing four-key env file does not have. +> Preflight fails loudly, naming both the missing key and the inventory variable +> that satisfies it. Pick one: set `grafana_alloy_loki_url` / +> `grafana_alloy_loki_username` in inventory (recommended — nothing to do per +> host), append `GC_LOKI_URL` / `GC_LOKI_USERNAME` to `/etc/grafana-alloy.env`, +> or set `grafana_alloy_logs_enabled: false` to keep shipping exactly what you +> shipped before. The other four keys are untouched by this change. +> +> Host metrics need no new credential (they ride the existing Prometheus +> remote-write), but note that the daemon's own series now carry +> `job="decdn-node"` instead of the implicit `job="prometheus.scrape.decdn_node"`. +> Set `grafana_alloy_node_job: ""` to keep the old value if dashboards or alert +> rules already hard-code it. + +`GC_API_TOKEN` has **no** inventory variable by design: the rendered +`/etc/alloy/config.alloy` is world-readable, and preflight rejects a value that +looks like a token (or a URL with embedded credentials) in any of the five +variables above. + +The credential path is intentionally restricted to a **direct child of `/etc`**. A nested override (including the old `/etc/decdn/grafana-alloy.env`) is rejected: -write access to any ancestor is enough to replace a root-owned `0600` file. -four required keys are `GC_PROM_REMOTE_WRITE_URL`, `GC_OTLP_ENDPOINT`, -`GC_PROM_USERNAME` and `GC_API_TOKEN`; both URLs must be `https://`. -`config.alloy` reads each of them at load time with `sys.env("GC_…")` — Alloy has -**no** `--config.expand-env` flag, so a `${GC_…}` placeholder would be a literal -string, not an expansion. The file on disk stays free of secret literals, and the -role only ever greps shapes on the host rather than reading values back. +write access to any ancestor is enough to replace a root-owned `0600` file. Both +cloud URLs must be `https://`, whichever half they come from. +`config.alloy` reads each env-provided value at load time with `sys.env("GC_…")` +— Alloy has **no** `--config.expand-env` flag, so a `${GC_…}` placeholder would be +a literal string, not an expansion. The file on disk stays free of secret +literals, and the role only ever greps shapes on the host rather than reading +values back. With the flag off, the role removes the unit, rendered config, state directory and service account **it installed** — identified by the managed-by marker in @@ -49,6 +106,64 @@ upstream apt repo, another role, by hand) carries no marker and is left strictly alone, which matters because the flag is `false` by default on every host. Binary/package removal is manual. +## What lands in Grafana Cloud + +| Signal | Job label | Where it shows up | +| --- | --- | --- | +| Node `/metrics` | `decdn-node` | your own dashboards/queries | +| Machine metrics | `integrations/node_exporter` | Grafana Cloud's **Linux Server** integration dashboards + alerts, unmodified | +| journald | `integrations/node_exporter` | the same integration's logs dashboards (correlated by `instance`) | +| Alloy self-metrics | `integrations/alloy` | agent health | +| Traces | — | Application Observability | + +Changing `grafana_alloy_host_job` / `grafana_alloy_logs_job` away from +`integrations/node_exporter` silently un-wires the prebuilt dashboards; that is +why they are knobs with a warning rather than hard-coded literals. + +## Machine monitoring: what it costs and how to bound it + +Grafana Cloud bills on active series and log volume, so the defaults are +deliberately lean and every lever is a variable: + +- `grafana_alloy_host_collectors` is a **hand-picked replacement** for + node_exporter's ~45-collector default set (Alloy's `set_collectors`: anything + not listed is off). Widen with `grafana_alloy_host_extra_collectors`, veto with + `grafana_alloy_host_disable_collectors` — the veto is applied to both lists at + render time, because upstream applies its own `disable_collectors` *before* + `enable_collectors` and would otherwise leave a vetoed collector running. Names + are checked against a committed inventory (`vars/main.yml`): an unknown + collector is not a startup error, it is silently ignored, so a typo would ship + fewer metrics forever with nothing to show for it. +- `systemd` is the one non-default collector enabled, and it is scoped by + `grafana_alloy_host_systemd_unit_include` to the units that matter rather than + every unit on the box. It answers "is `decdn-node` actually running" — which + the daemon's own `/metrics` cannot, because a dead daemon serves nothing. +- Host metrics scrape at `60s`, not the node's `30s`. +- The `filesystem` / `netdev` / `disk` exclude regexes drop virtual filesystems + and container/virtual interfaces. +- journald drops `debug` priority and a noisy-unit list + (`grafana_alloy_logs_drop_priority_regex` / `_drop_unit_regex`, `""` disables + either), and `grafana_alloy_logs_max_age` bounds the catch-up burst after an + outage — without it a restart can replay days of journal in one go. + +## Hardening: the two relaxations machine monitoring requires + +The unit is stricter than Grafana's own shipped unit (which has no `Protect*` +directives at all), but two directives are incompatible with host telemetry and +are relaxed **only** when the matching signal is enabled: + +| Directive | Condition | Why | +| --- | --- | --- | +| `ProtectHome=read-only` (instead of `true`) | host metrics on | `ProtectHome=true` replaces `/home`, `/root` and `/run/user` with empty tmpfs mounts inside the unit's namespace, so the filesystem collector measures the overlay instead of the disk — a full `/home` would never raise an alert. `read-only` keeps the privilege boundary: the agent still cannot write there. | +| `SupplementaryGroups=systemd-journal adm` | logs on, and the groups exist | Reading the journal as a non-root user requires **both**. Without them `loki.source.journal` starts without error and collects **nothing** — the worst failure mode there is. The role joins the `alloy` account to the same groups, and lists only groups that exist, because systemd refuses to start a unit naming one that does not. | + +Everything else stands: `NoNewPrivileges`, `ProtectSystem=strict`, +`SystemCallFilter=@system-service`, `PrivateDevices`, `RestrictNamespaces`, the +dedicated unprivileged account. The default collector set deliberately excludes +the collectors that would need capabilities or a widened syscall filter (`perf`, +`processes`, the netlink-based ones). `AF_UNIX` — already allowed for journald — +is also what the `systemd` collector uses to reach the D-Bus system bus. + ## Variables All prefixed `grafana_alloy_*` (see `defaults/main.yml` for comments and pinned @@ -62,12 +177,22 @@ upstream version/sha256). Highlights: | `grafana_alloy_trace_sampling_ratio` | `0.25` | Trace keep-ratio, validated to `[0,1]` | | `grafana_alloy_otlp_grpc_port` | `4317` | Hard-coupled to the literal emitted into node.toml | | `grafana_alloy_region` | `decdn_region` | Required identity attribute; derived from the node region | +| `grafana_alloy_host_metrics_enabled` | `true` | Machine metrics (Alloy's in-process `node_exporter`) | +| `grafana_alloy_host_collectors` | 13 collectors | Replacement set; `_host_extra_collectors` adds, `_host_disable_collectors` vetoes | +| `grafana_alloy_host_scrape_interval` | `60s` | Host metrics only; the node keeps `30s` | +| `grafana_alloy_host_systemd_unit_include` | `(decdn-node\|alloy\|ssh\|sshd)\.service` | Scope of the per-unit `systemd` collector | +| `grafana_alloy_logs_enabled` | `true` | journald → Grafana Cloud Loki | +| `grafana_alloy_logs_max_age` | `12h` | Bounds the catch-up burst after an outage | +| `grafana_alloy_self_metrics_enabled` | `true` | Alloy's own health | +| `grafana_alloy_prom_url` / `_prom_username` / `_otlp_endpoint` / `_loki_url` / `_loki_username` | `""` | Non-secret connection settings; empty ⇒ read the matching `GC_…` env key | +| `grafana_alloy_node_job` / `_host_job` / `_self_job` / `_logs_job` | see table above | Job labels; the `integrations/…` ones are what Grafana Cloud's dashboards match | Label variables (`service_name`, `service_namespace`, `instance_id`, `deployment_environment`, `region`) ship on EVERY series/span/log line — keep them low-cardinality; unique label sets are what your Grafana Cloud bill scales -with. Guardrails: one scrape interval per fleet (`30s`), sampling above zero for -spans, and no per-request/per-hash label values. +with. Guardrails: one scrape interval per fleet (`30s` node / `60s` host), +sampling above zero for spans, the curated collector set, the journald drops, and +no per-request/per-hash label values. ## Couplings & guards @@ -86,18 +211,27 @@ If you move either default, update BOTH sides and the molecule guard in Three layers, because the first one cannot prove correctness on its own: 1. **`grafana-cloud` molecule scenario** — the role end-to-end against a *stub* - binary (plumbing, rendered content, hardened unit, teardown scope), plus - negative preflight cases in `validation` and disabled-path assertions in - `default`. The stub exits 0 for every subcommand, so a green run says nothing - about whether Alloy can load the config. + binary (plumbing, rendered content, hardened unit, group membership, teardown + scope), plus negative preflight cases in `validation` and disabled-path + assertions in `default`. The stub exits 0 for every subcommand, so a green run + says nothing about whether Alloy can load the config. It also runs in a + container with no D-Bus and no real journal, so the `systemd` collector and + journald collect nothing there — expected, and not what this layer proves. 2. **`make lint-alloy`** (CI job `alloy-config`) — renders these templates in - several variable combinations and feeds them to the REAL pinned Alloy binary: - `alloy validate` (component graph, not just syntax) and a check that every - `ExecStart` flag exists in `alloy run --help`. See + eight variable combinations (defaults, minimal identity, fully overridden, + host-metrics-only, logs-only, every sub-knob off, inventory-supplied + endpoints, vetoed collector + blanked node job) and feeds them to the REAL pinned Alloy binary: `alloy validate` + (component graph, not just syntax), a check that every `ExecStart` flag exists + in `alloy run --help`, and greps proving each Jinja branch actually switched — + including that "every sub-knob off" reproduces the pre-machine-monitoring + pipeline and that no token is ever rendered as a literal. See `ansible/tests/alloy-config/`. 3. **Deploy time** — the role runs `alloy validate` against the just-rendered file with the installed binary before any restart. +Only a real host proves *collection*: check `journalctl -u alloy -e` and Alloy's +loopback UI (`127.0.0.1:12345`) after the first deploy. + Re-run `make lint-alloy` whenever `grafana_alloy_version` is bumped: the harness downloads the same digest-pinned `.deb` the role installs, so a version bumped without its sha256 fails there rather than on a host. diff --git a/ansible/roles/grafana_alloy/defaults/main.yml b/ansible/roles/grafana_alloy/defaults/main.yml index 0923647..69861dc 100644 --- a/ansible/roles/grafana_alloy/defaults/main.yml +++ b/ansible/roles/grafana_alloy/defaults/main.yml @@ -51,8 +51,131 @@ grafana_alloy_otlp_http_port: 4318 # gRPC value staying put (see node.t # --server.grpc.listen-addr flag, and passing one is an immediate crash-loop. grafana_alloy_server_http_addr: 127.0.0.1:12345 +# --- Grafana Cloud endpoints (NON-SECRET; optional inventory overrides) ------- +# Only the API TOKEN is a secret. The endpoint URLs and the two numeric instance +# IDs are not, so they may be set here (group_vars/host_vars) instead of being +# hand-typed into the root-owned env file on every host — which leaves +# GC_API_TOKEN as the single value an operator must provision per host. +# +# Each of these is optional and falls back, when left EMPTY, to the matching +# sys.env("GC_…") lookup the role has always used, so an existing deployment that +# sets none of them renders byte-identical configuration. Setting one makes its +# env key unnecessary, and preflight stops requiring that key. +# +# NEVER put the token here: config.alloy is world-readable 0644 on the host, and +# inventory is committed. GC_API_TOKEN has no variable by design. +grafana_alloy_prom_url: "" # else sys.env("GC_PROM_REMOTE_WRITE_URL") +grafana_alloy_prom_username: "" # else sys.env("GC_PROM_USERNAME") +grafana_alloy_otlp_endpoint: "" # else sys.env("GC_OTLP_ENDPOINT") +# Grafana Cloud's Loki instance ID is a DIFFERENT number from the Prometheus one +# (the same access-policy token authenticates both). +grafana_alloy_loki_url: "" # else sys.env("GC_LOKI_URL") +grafana_alloy_loki_username: "" # else sys.env("GC_LOKI_USERNAME") + # --- Metrics scrape (node /metrics -> Grafana Cloud Prometheus) --------------- grafana_alloy_scrape_interval: 30s +# Explicit `job` label for the daemon's own series. Without it Prometheus derives +# `job` from the scrape component's ID ("prometheus.scrape.decdn_node"), which is +# an implementation detail no dashboard should hard-code. +# UPGRADE NOTE: setting this changes the identity of already-existing series. +# Hosts deployed before this label existed report job="prometheus.scrape.decdn_node"; +# set grafana_alloy_node_job: "" to keep that value instead of re-pointing +# dashboards, recording rules and alerts. +grafana_alloy_node_job: decdn-node + +# --- Host (machine) metrics --------------------------------------------------- +# Alloy's built-in node_exporter. Runs IN-PROCESS: no extra listener, no extra +# port, no firewall hole (AGENTS.md hard rule 2 is unaffected). Stamped with the +# job label Grafana Cloud's prebuilt "Linux Server / node_exporter" integration +# dashboards and alert rules key off, so they work with zero dashboard wiring. +grafana_alloy_host_metrics_enabled: true +# Host metrics do not need the node's resolution; 60s halves the series cost. +grafana_alloy_host_scrape_interval: 60s +# Changing this away from integrations/node_exporter silently un-wires the +# Grafana Cloud integration dashboards. "integrations/unix" is also accepted. +grafana_alloy_host_job: integrations/node_exporter +# Hand-picked replacement for node_exporter's ~45-collector default set (Alloy's +# `set_collectors`: anything absent here is off). The remote bill scales with +# series count, so this covers the questions an operator actually asks — CPU, +# memory, disk, filesystem, network, load, saturation — and nothing else. +grafana_alloy_host_collectors: + - cpu + - diskstats + - filefd + - filesystem + - loadavg + - meminfo + - netdev + - os + - pressure # PSI saturation; needs kernel 4.20+/CONFIG_PSI, absent = no series + - stat + - time + - uname + - vmstat +# Added ON TOP of the list above (Alloy's `enable_collectors`). `systemd` is +# registered defaultDisabled upstream and must be named here; it answers "is +# decdn-node actually running", which the daemon's own /metrics cannot when the +# daemon is down. It talks to the D-Bus system bus, so it collects nothing in a +# container without one (molecule) — that is expected, not a failure. +grafana_alloy_host_extra_collectors: + - systemd +# Subtracted from BOTH lists above at render time; an escape hatch for a +# collector that misbehaves on a specific fleet. (Doing it in the template is +# deliberate: upstream applies disable_collectors BEFORE enable_collectors, so a +# name in both would end up enabled — the opposite of a veto.) +grafana_alloy_host_disable_collectors: [] +# Alloy's own exporter-internal metrics (scrape durations per collector). Off: +# grafana_alloy_self_metrics_enabled covers agent health more cheaply. +grafana_alloy_host_include_exporter_metrics: false +# Filter regexes. Single-quoted YAML so backslashes stay literal; each is +# JSON-encoded into the rendered config, so quotes/backslashes cannot break out +# of the string. Defaults mirror upstream's, plus tmpfs (Grafana Cloud's own +# snippet adds it) and the usual container/virtual interface noise. +# NOTE the escaped line breaks: YAML double quotes join these back into ONE regex +# with no whitespace. Single quotes cannot be used here (the line is too long for +# yamllint) and a folded scalar would inject spaces into the pattern. +grafana_alloy_host_fs_types_exclude: "^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|\ + fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|\ + selinuxfs|squashfs|sysfs|tmpfs|tracefs)$" +grafana_alloy_host_mount_points_exclude: '^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/)' +grafana_alloy_host_mount_timeout: 5s +grafana_alloy_host_netdev_device_exclude: '^(veth.*|cali.*|docker.*|br-.*|lo|[a-f0-9]{15})$' +grafana_alloy_host_disk_device_exclude: '^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$' +# Per-unit state, deliberately scoped: one series set per unit on the box is a +# real cost, and only these units matter. Units must match include AND NOT match +# exclude. +grafana_alloy_host_systemd_unit_include: '(decdn-node|alloy|ssh|sshd)\.service' +grafana_alloy_host_systemd_unit_exclude: '.+\.(automount|device|mount|scope|slice)' + +# --- Agent self-monitoring ---------------------------------------------------- +# Alloy's own health (component evaluation, WAL, remote-write queue/failures). +# Cheap, and the only way to tell "the node is silent" apart from "the agent that +# reports on the node is wedged". +grafana_alloy_self_metrics_enabled: true +grafana_alloy_self_job: integrations/alloy + +# --- Host + service logs (journald -> Grafana Cloud Loki) --------------------- +# Reading the journal as a non-root user requires membership in systemd-journal +# AND adm — without both the component starts cleanly and collects NOTHING, so +# the role joins the alloy user to whichever of them exist on the host and the +# unit lists them under SupplementaryGroups. +grafana_alloy_logs_enabled: true +# Grafana Cloud's Linux Server integration uses the SAME job value for logs as +# for host metrics; its logs dashboards key off job + instance. +grafana_alloy_logs_job: integrations/node_exporter +# Bounds the catch-up burst after an outage or a fresh install: without it a +# restart can replay days of journal into your bill in one go. +grafana_alloy_logs_max_age: 12h +# "" => Alloy reads both /var/log/journal and /run/log/journal (persistent and +# volatile). Set a single path only if you know the host keeps just one. +grafana_alloy_logs_journal_path: "" +# Cost guardrails, applied while the __journal* fields still exist. Both are +# anchored regexes matched against the whole value; "" disables that rule. +# Priority keywords are emerg/alert/crit/error/warning/notice/info/debug +# (journald's numeric priority 3 surfaces as "error", NOT "err"). +grafana_alloy_logs_drop_priority_regex: 'debug' +grafana_alloy_logs_drop_unit_regex: + '(session-\d+\.scope|user@\d+\.service|systemd-timesyncd\.service|cron\.service|systemd-logind\.service)' # --- Resource identity labels (low-cardinality only!) -------------------------- # These become metric labels and OTLP resource attributes shipped to your Grafana diff --git a/ansible/roles/grafana_alloy/files/grafana-alloy.env.example b/ansible/roles/grafana_alloy/files/grafana-alloy.env.example index 7bea71f..cf20bc8 100644 --- a/ansible/roles/grafana_alloy/files/grafana-alloy.env.example +++ b/ansible/roles/grafana_alloy/files/grafana-alloy.env.example @@ -1,18 +1,45 @@ # Template for /etc/grafana-alloy.env (root-owned 0600). # -# Provision it ON THE TARGET HOST — these values are your Grafana Cloud -# credentials and must never be committed or carried through inventory: +# Provision it ON THE TARGET HOST — this file holds your Grafana Cloud API token +# and must never be committed or carried through inventory: # # umask 077 # sudo install -m 600 -o root -g root grafana-alloy.env /etc/grafana-alloy.env # -# Values come from your Grafana Cloud org: observe -> Application Observability -> -# "Instructions"/details page shows the exact remote-write URL, OTLP endpoint, -# instance id ("user") and API token. The role greps shapes only; it never reads -# values back onto the control machine. +# MINIMUM CONTENT for a NEW host: the token, and nothing else. Every other value +# below is NON-secret and can be set once in inventory (group_vars) instead of +# being re-typed on every host — see the grafana_alloy_* variables named in each +# comment. A key is only required here when its inventory variable is empty. +# +# UPGRADING an existing host: journald shipping is on by default, so GC_LOKI_URL +# and GC_LOKI_USERNAME become required unless you set grafana_alloy_loki_url / +# grafana_alloy_loki_username in inventory, or turn logs off with +# grafana_alloy_logs_enabled: false. Preflight names the missing key and the +# variable that would satisfy it instead. +# +# Values come from your Grafana Cloud org: Home -> Stacks -> your stack, then the +# "Details/Send Metrics|Logs|Traces" pages show each endpoint plus its numeric +# instance ID. The role greps shapes only; it never reads values back onto the +# control machine. -GC_PROM_REMOTE_WRITE_URL=https://prometheus-prod-XX.grafana.net/api/prom/push -GC_OTLP_ENDPOINT=https://otlp-gateway-prod-XX.grafana.net/otlp -GC_PROM_USERNAME=1234567 -# glc_... application token with write scope for this stack +# REQUIRED, always. An access-policy token with metrics:write + logs:write + +# traces:write scope for this stack. This is the one value with no inventory +# variable, deliberately: the rendered /etc/alloy/config.alloy is world-readable. GC_API_TOKEN=glc_example_token_replace_me + +# --- Optional: only needed if the matching inventory variable is left empty ---- + +# grafana_alloy_prom_url +#GC_PROM_REMOTE_WRITE_URL=https://prometheus-prod-XX-REGION.grafana.net/api/prom/push +# grafana_alloy_prom_username — the numeric Prometheus instance ID +#GC_PROM_USERNAME=1234567 + +# grafana_alloy_otlp_endpoint +#GC_OTLP_ENDPOINT=https://otlp-gateway-prod-XX.grafana.net/otlp + +# Logs (grafana_alloy_logs_enabled, on by default). NOTE the Loki instance ID is +# a DIFFERENT number from the Prometheus one above; the token is shared. +# grafana_alloy_loki_url +#GC_LOKI_URL=https://logs-prod-XX.grafana.net/loki/api/v1/push +# grafana_alloy_loki_username — the numeric Loki instance ID +#GC_LOKI_USERNAME=7654321 diff --git a/ansible/roles/grafana_alloy/tasks/main.yml b/ansible/roles/grafana_alloy/tasks/main.yml index a350962..1ea256d 100644 --- a/ansible/roles/grafana_alloy/tasks/main.yml +++ b/ansible/roles/grafana_alloy/tasks/main.yml @@ -1,7 +1,8 @@ --- # Opt-in Grafana Cloud Application Observability for an Ansible-deployed deCDN -# node: a loopback-only Grafana Alloy agent that scrapes the node's /metrics and -# receives the daemon's OTLP traces, then ships both to Grafana Cloud. +# node: a loopback-only Grafana Alloy agent that scrapes the node's /metrics, the +# MACHINE itself (Alloy's in-process node_exporter) and Alloy's own health, +# receives the daemon's OTLP traces, and ships journald to Grafana Cloud Loki. # # Gated by decdn_grafana_cloud_enabled (mirrored in roles/decdn_node/defaults — # ONE inventory knob drives both roles). When false this role does nothing except @@ -144,6 +145,42 @@ - name: Fail loud on preflight misconfiguration ansible.builtin.import_tasks: preflight.yml + # --- Journal access --------------------------------------------------------- + # Reading the systemd journal as a non-root user requires membership in + # systemd-journal AND adm. Without them loki.source.journal starts WITHOUT + # error and collects nothing, so this is a silent-failure guard, not a + # convenience. Detected rather than assumed: a container or a non-Debian host + # may ship neither, and systemd refuses to start a unit whose + # SupplementaryGroups names a group that does not exist. + - name: Detect which journal reader groups exist on this host + ansible.builtin.command: + cmd: getent group {{ item | quote }} + loop: "{{ ['systemd-journal', 'adm'] if grafana_alloy_logs_enabled | bool else [] }}" + changed_when: false + failed_when: false + check_mode: false # read-only lookup; must also run under make check + register: _ga_journal_group_lookup + + - name: Resolve the journal reader group list + ansible.builtin.set_fact: + _ga_journal_groups: >- + {{ _ga_journal_group_lookup.results | default([]) + | selectattr('rc', 'defined') + | selectattr('rc', 'equalto', 0) + | map(attribute='item') | list }} + + - name: Report that journald collection will be silently empty + ansible.builtin.debug: + msg: >- + Neither systemd-journal nor adm exists on this host, so + {{ grafana_alloy_user }} cannot read the journal: loki.source.journal + will run without error and ship no log lines. Expected inside + containers; on a real host, install systemd properly or set + grafana_alloy_logs_enabled=false. + when: + - grafana_alloy_logs_enabled | bool + - _ga_journal_groups | length == 0 + # --- User + directories ---------------------------------------------------- - name: Create the alloy system group ansible.builtin.group: @@ -154,10 +191,18 @@ ansible.builtin.user: name: "{{ grafana_alloy_user }}" group: "{{ grafana_alloy_group }}" + # Journal readers only, and only those that exist. append keeps any + # membership an operator added by hand; the disable path removes the + # account outright, so nothing is left behind either way. + groups: "{{ _ga_journal_groups }}" + append: true system: true home: "{{ grafana_alloy_state_dir }}" create_home: false shell: /usr/sbin/nologin + # Group membership is applied at process start, so a membership change that + # does not also change the unit or the config still needs a restart. + notify: Restart alloy - name: Ensure config + state directories exist ansible.builtin.file: diff --git a/ansible/roles/grafana_alloy/tasks/preflight.yml b/ansible/roles/grafana_alloy/tasks/preflight.yml index a03ad88..49c0ec0 100644 --- a/ansible/roles/grafana_alloy/tasks/preflight.yml +++ b/ansible/roles/grafana_alloy/tasks/preflight.yml @@ -1,8 +1,10 @@ --- # Fail-loud preflight, run BEFORE any host mutation. Mirrors decdn_node's -# secret-file posture: this role never carries secrets in inventory and never -# reads the env file's values onto the control machine — it only greps shapes on -# the host itself. +# secret-file posture: this role never carries the API TOKEN in inventory and +# never reads the env file's values onto the control machine — it only greps +# shapes on the host itself. The non-secret half of the connection settings +# (endpoint URLs, instance IDs) MAY come from inventory; those are validated here +# too, and each one that is set removes its key from the host-state gates below. # # Ordering matters for testability (molecule/validation): PURE-INVENTORY gates # run first so a bad knob rejects deterministically regardless of what fixtures @@ -35,9 +37,93 @@ - "{{ grafana_alloy_retry_initial_interval }}" - "{{ grafana_alloy_retry_max_elapsed_time }}" - "{{ grafana_alloy_scrape_interval }}" + - "{{ grafana_alloy_host_scrape_interval }}" + - "{{ grafana_alloy_host_mount_timeout }}" + - "{{ grafana_alloy_logs_max_age }}" loop_control: label: "{{ item }}" +- name: Validate the observability sub-switches + ansible.builtin.assert: + that: + - grafana_alloy_host_metrics_enabled is boolean + - grafana_alloy_self_metrics_enabled is boolean + - grafana_alloy_logs_enabled is boolean + - grafana_alloy_host_include_exporter_metrics is boolean + fail_msg: >- + grafana_alloy_host_metrics_enabled, _self_metrics_enabled, _logs_enabled + and _host_include_exporter_metrics must be booleans (true/false), not + quoted strings — each gates whole component groups in config.alloy. + +# An unknown collector name is NOT a startup crash: Alloy resolves collectors by +# map lookup and silently ignores names it does not know, so a typo ships an +# exporter that quietly measures less than the operator thinks. A charset check +# cannot catch `diskstat`; only the committed inventory in vars/main.yml can. +- name: Validate the node_exporter collector names + ansible.builtin.assert: + that: + - item in _ga_known_collectors + quiet: true + fail_msg: >- + "{{ item }}" is not a node_exporter collector known to Alloy + {{ grafana_alloy_version }}. Check grafana_alloy_host_collectors / + _host_extra_collectors / _host_disable_collectors against the collector + table in the prometheus.exporter.unix reference. An unknown name is + silently ignored at runtime, so this gate is the only thing standing + between a typo and metrics that never arrive. (If upstream added a + collector, extend _ga_known_collectors in the role's vars/main.yml.) + loop: >- + {{ (grafana_alloy_host_collectors + + grafana_alloy_host_extra_collectors + + grafana_alloy_host_disable_collectors) | map('string') | list }} + loop_control: + label: "{{ item }}" + when: grafana_alloy_host_metrics_enabled | bool + +# NOTE the EFFECTIVE set, i.e. after the template subtracts +# _host_disable_collectors from the other two lists. An empty set_collectors is +# not "measure nothing" to Alloy — it reads as unset and re-enables the whole +# ~45-collector default set, the exact opposite of what an operator who disabled +# everything intended. +- name: Require a non-empty effective host collector set + ansible.builtin.assert: + that: + - >- + ((grafana_alloy_host_collectors + grafana_alloy_host_extra_collectors) + | reject('in', grafana_alloy_host_disable_collectors) | list | length) > 0 + fail_msg: >- + grafana_alloy_host_metrics_enabled is true but no collector survives + grafana_alloy_host_disable_collectors, which would silently fall back to + node_exporter's full default set instead of measuring nothing. Either + populate grafana_alloy_host_collectors or turn host metrics off with + grafana_alloy_host_metrics_enabled: false. + when: grafana_alloy_host_metrics_enabled | bool + +# job/instance are what Grafana Cloud's integration dashboards filter on, and +# like the identity labels they ride on EVERY series and log line. +- name: Validate the job labels + ansible.builtin.assert: + that: + - (item.allow_empty | bool and item.value | length == 0) + or item.value is match('^[A-Za-z0-9._/-]{1,63}$') + quiet: true + fail_msg: >- + Job labels are restricted to the same low-cardinality charset as the + identity labels (letters, digits, dot, underscore, slash, dash, ≤ 63 + chars): "{{ item.value }}" does not fit {{ item.name }}. + {{ item.allow_empty | bool | ternary('It may also be "" (fall back to the + component-ID job Alloy used before this label existed).', '') }} + loop: + # Only the node job may be blanked: that restores the pre-existing + # job="prometheus.scrape.decdn_node" identity. The other three feed Grafana + # Cloud integration dashboards that filter on an exact job value. + - {name: grafana_alloy_node_job, value: "{{ grafana_alloy_node_job | string }}", allow_empty: true} + - {name: grafana_alloy_host_job, value: "{{ grafana_alloy_host_job | string }}", allow_empty: false} + - {name: grafana_alloy_self_job, value: "{{ grafana_alloy_self_job | string }}", allow_empty: false} + - {name: grafana_alloy_logs_job, value: "{{ grafana_alloy_logs_job | string }}", allow_empty: false} + loop_control: + label: "{{ item.name }}" + - name: Validate integer pipeline knobs ansible.builtin.assert: that: @@ -87,7 +173,70 @@ grafana_alloy_deployment_environment and grafana_alloy_region are required identity attributes when Grafana Cloud observability is enabled. -- name: Validate loopback listener addresses +# --- Grafana Cloud connection settings (inventory half) ------------------------- +# These five are NON-secret and therefore inventory-settable; each one that is set +# replaces its sys.env("GC_…") lookup in the rendered config and removes the +# corresponding requirement from the host-state gates at the bottom of this file. +# The API token has no variable by design: config.alloy is 0644 on the host. + +- name: Validate any inventory-provided Grafana Cloud endpoints + ansible.builtin.assert: + that: + # Empty == "fall back to the env file", which is the default and always OK. + - item.value | length == 0 or item.value is match('^https://[^\s]+$') + quiet: true + fail_msg: >- + {{ item.name }} must be an https:// URL (got "{{ item.value }}"). Plain + http would ship your Grafana Cloud API token in cleartext. Leave it empty + to keep reading the value from {{ item.key }} in + {{ grafana_alloy_secret_file }} instead. + loop: + - {name: grafana_alloy_prom_url, key: GC_PROM_REMOTE_WRITE_URL, value: "{{ grafana_alloy_prom_url | string }}"} + - {name: grafana_alloy_otlp_endpoint, key: GC_OTLP_ENDPOINT, value: "{{ grafana_alloy_otlp_endpoint | string }}"} + - {name: grafana_alloy_loki_url, key: GC_LOKI_URL, value: "{{ grafana_alloy_loki_url | string }}"} + loop_control: + label: "{{ item.name }}" + +- name: Validate any inventory-provided Grafana Cloud instance IDs + ansible.builtin.assert: + that: + - item.value | length == 0 or item.value is match('^[A-Za-z0-9._-]{1,64}$') + quiet: true + fail_msg: >- + {{ item.name }} must be the numeric Grafana Cloud instance ID (letters, + digits, dot, underscore, dash), not a token and not a quoted string with + whitespace — got "{{ item.value }}". NOTE the Loki instance ID differs from + the Prometheus one; the access-policy token is shared and stays in + {{ grafana_alloy_secret_file }} as GC_API_TOKEN. + loop: + - {name: grafana_alloy_prom_username, value: "{{ grafana_alloy_prom_username | string }}"} + - {name: grafana_alloy_loki_username, value: "{{ grafana_alloy_loki_username | string }}"} + loop_control: + label: "{{ item.name }}" + +- name: Refuse a Grafana Cloud token smuggled into inventory + ansible.builtin.assert: + that: + # Access-policy tokens (glc_), service-account tokens (glsa_), API keys + # (eyJ… — base64 JSON), and any URL carrying user:pass@host. + - not (item is search('(glc_|glsa_|eyJ)')) + - "'@' not in item" + quiet: true + fail_msg: >- + A Grafana Cloud access token (or a URL with embedded credentials) appears + in an inventory variable. config.alloy is world-readable on the host and + inventory is committed: the token belongs ONLY in + {{ grafana_alloy_secret_file }} as GC_API_TOKEN. + loop: + - "{{ grafana_alloy_prom_url | string }}" + - "{{ grafana_alloy_prom_username | string }}" + - "{{ grafana_alloy_otlp_endpoint | string }}" + - "{{ grafana_alloy_loki_url | string }}" + - "{{ grafana_alloy_loki_username | string }}" + loop_control: + label: "{{ item | truncate(24, true) }}" + +- name: Validate the loopback listener addresses ansible.builtin.assert: that: - grafana_alloy_otlp_grpc_bind == '127.0.0.1' @@ -185,6 +334,11 @@ # key is grepped for presence + value shape directly on the host: grep keeps the # values off stdout AND off the control machine (slurping the file back would put # the API token there, defeating the point of the host-provisioned path). +# +# "Required" is now conditional: a key is only needed when nothing in inventory +# supplies that value, and the Loki pair is only needed when logs are enabled. +# GC_API_TOKEN is ALWAYS required — it is the one value that must never live in +# inventory. - name: Confirm every required credential key is present and shaped correctly ansible.builtin.command: cmd: grep -qE -- '{{ item.regex }}' {{ grafana_alloy_secret_file | quote }} @@ -192,34 +346,57 @@ check_mode: false # read-only gate: must also run under make check failed_when: false loop: - # The two cloud URLs must be https:// (plain http would ship the API token in + # The cloud URLs must be https:// (plain http would ship the API token in # cleartext); the credential values must be non-empty, non-comment text. # POSIX bracket classes only — grep -E does NOT interpret \r/\n escapes (they # become literal 'r'/'n', silently poisoning the class), and each grep line is # already newline-free by construction. + - key: GC_API_TOKEN + regex: ^GC_API_TOKEN=[^[:space:]#][^[:space:]]*$ + var: "(none — the token must never be set in inventory)" + required: true - key: GC_PROM_REMOTE_WRITE_URL regex: ^GC_PROM_REMOTE_WRITE_URL=https://[^[:space:]]+$ + var: grafana_alloy_prom_url + required: "{{ grafana_alloy_prom_url | length == 0 }}" - key: GC_PROM_USERNAME regex: ^GC_PROM_USERNAME=[^[:space:]#][^[:space:]]*$ - - key: GC_API_TOKEN - regex: ^GC_API_TOKEN=[^[:space:]#][^[:space:]]*$ + var: grafana_alloy_prom_username + required: "{{ grafana_alloy_prom_username | length == 0 }}" - key: GC_OTLP_ENDPOINT regex: ^GC_OTLP_ENDPOINT=https://[^[:space:]]+$ + var: grafana_alloy_otlp_endpoint + required: "{{ grafana_alloy_otlp_endpoint | length == 0 }}" + - key: GC_LOKI_URL + regex: ^GC_LOKI_URL=https://[^[:space:]]+$ + var: grafana_alloy_loki_url + required: "{{ grafana_alloy_logs_enabled | bool and grafana_alloy_loki_url | length == 0 }}" + - key: GC_LOKI_USERNAME + regex: ^GC_LOKI_USERNAME=[^[:space:]#][^[:space:]]*$ + var: grafana_alloy_loki_username + required: "{{ grafana_alloy_logs_enabled | bool and grafana_alloy_loki_username | length == 0 }}" loop_control: label: "{{ item.key }}" + when: item.required | bool register: _ga_secret_grep - name: Report which credential key was missing or malformed ansible.builtin.fail: msg: >- {{ grafana_alloy_secret_file }} does not carry a usable line for - {{ _ga_missing_keys | join(', ') }}. The file must contain exactly the four - KEY=value pairs shown in roles/grafana_alloy/files/grafana-alloy.env.example; - both cloud URLs must be https:// and each value non-empty (systemd silently - drops unparseable lines, so a quoting typo looks identical to a missing key). + {{ _ga_missing | map(attribute='key') | join(', ') }}. Either add the + KEY=value pair(s) on the host (see + roles/grafana_alloy/files/grafana-alloy.env.example) — URLs must be + https:// and every value non-empty, since systemd silently drops + unparseable lines so a quoting typo looks identical to a missing key — or, + for everything except the token, set the matching inventory variable + instead: {{ _ga_missing | map(attribute='var') | join(', ') }}. vars: - _ga_missing_keys: >- - {{ _ga_secret_grep.results + # Skipped loop items (key not required) carry no rc, so filter on rc being + # defined first: rejectattr alone would count every skipped key as missing. + _ga_missing: >- + {{ _ga_secret_grep.results | default([]) + | selectattr('rc', 'defined') | rejectattr('rc', 'equalto', 0) - | map(attribute='item.key') | list }} - when: _ga_missing_keys | length > 0 + | map(attribute='item') | list }} + when: _ga_missing | length > 0 diff --git a/ansible/roles/grafana_alloy/templates/alloy.service.j2 b/ansible/roles/grafana_alloy/templates/alloy.service.j2 index 6aabc34..9caa2ff 100644 --- a/ansible/roles/grafana_alloy/templates/alloy.service.j2 +++ b/ansible/roles/grafana_alloy/templates/alloy.service.j2 @@ -35,23 +35,44 @@ TimeoutStopSec={{ grafana_alloy_stop_timeout_sec }} Restart=on-failure RestartSec=5 +{% if grafana_alloy_logs_enabled | default(false) | bool and (_ga_journal_groups | default([]) | length > 0) %} + +# Journal access. Reading the systemd journal as a non-root user requires +# membership in systemd-journal AND adm — without them loki.source.journal starts +# without error and collects NOTHING, which is the worst possible failure mode. +# The role joins {{ grafana_alloy_user }} to these groups too; only groups that +# actually exist on this host are listed, because systemd refuses to start a unit +# naming a group that does not exist. +SupplementaryGroups={{ _ga_journal_groups | default([]) | join(' ') }} +{% endif %} # --- Hardening --------------------------------------------------------------- # Mirrors decdn-node.service: strict filesystem, privilege boundary, syscall # filter. StateDirectory owns {{ grafana_alloy_state_dir }} (the only writable -# path under ProtectSystem=strict): WAL/queue state lives there. +# path under ProtectSystem=strict): WAL/queue state lives there, including +# loki.source.journal's positions file. StateDirectory={{ grafana_alloy_state_dir | regex_replace('^/var/lib/', '') }} StateDirectoryMode=0750 NoNewPrivileges=true ProtectSystem=strict +{% if grafana_alloy_host_metrics_enabled | default(false) | bool %} +# read-only, NOT true: ProtectHome=true replaces /home, /root and /run/user with +# empty tmpfs mounts inside the unit's namespace, so node_exporter's filesystem +# collector reports those mounts as the empty overlay instead of the real disk — +# a full /home would never raise an alert. read-only keeps the privilege boundary +# (the agent still cannot write there) while reporting true usage. +ProtectHome=read-only +{% else %} ProtectHome=true +{% endif %} PrivateTmp=true PrivateDevices=true ProtectKernelTunables=true ProtectKernelModules=true ProtectControlGroups=true ProtectClock=true -# AF_INET/AF_INET6 for the loopback OTLP/scrape/remotes; AF_UNIX for journald. +# AF_INET/AF_INET6 for the loopback OTLP/scrape/remotes; AF_UNIX for journald and +# for the D-Bus system bus the node_exporter `systemd` collector queries. RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX RestrictNamespaces=true RestrictSUIDSGID=true diff --git a/ansible/roles/grafana_alloy/templates/config.alloy.j2 b/ansible/roles/grafana_alloy/templates/config.alloy.j2 index 561458f..a61a37d 100644 --- a/ansible/roles/grafana_alloy/templates/config.alloy.j2 +++ b/ansible/roles/grafana_alloy/templates/config.alloy.j2 @@ -5,9 +5,11 @@ // file (and everything the Jinja layer emits) must stay `//`. // // Pipeline: scrape the node's /metrics and push it to Grafana Cloud Prometheus; -// receive OTLP traces (plus any logs/metrics pushed over localhost) on gRPC 4317 / -// HTTP 4318 bound to loopback only; stamp resource identity on everything, -// probabilistically sample traces, batch, then export via OTLP/HTTP. +// scrape the MACHINE (Alloy's in-process node_exporter) and Alloy itself; ship +// journald to Grafana Cloud Loki; receive OTLP traces (plus any logs/metrics +// pushed over localhost) on gRPC 4317 / HTTP 4318 bound to loopback only; stamp +// resource identity on everything, probabilistically sample traces, batch, then +// export via OTLP/HTTP. // // Secrets are NOT literals here: every credential is read at config-load time // with the `sys.env` stdlib function from the root-owned 0600 environment file @@ -15,11 +17,30 @@ // roles/grafana_alloy/files/grafana-alloy.env.example). Alloy has no // `--config.expand-env` flag — `${VAR}` inside a string is a literal, so // `sys.env("VAR")` is the only working indirection. +// +// The NON-secret half of the credentials (endpoint URLs, the two numeric +// instance IDs) may instead come from inventory; see the ga_endpoint macro. + +{# Renders a Grafana Cloud connection value as either an inline string literal + (when inventory set it) or the sys.env lookup this role has always used (when + inventory left it empty). Keeping both shapes means a host provisioned the old + way — all keys in /etc/grafana-alloy.env — renders exactly the previous + configuration, while a host driven from inventory needs only GC_API_TOKEN on + disk. tojson does the quoting/escaping in both branches. + NEVER call this for the API token: config.alloy is 0644 on the host. #} +{% macro ga_endpoint(value, env_key) -%} +{% if value | default('', true) | length > 0 %}{{ value | tojson }}{% else %}sys.env({{ env_key | tojson }}){% endif %} +{%- endmacro %} prometheus.scrape "decdn_node" { targets = [ { "__address__" = "127.0.0.1:{{ decdn_metrics_port | default(9090) }}", - "instance" = "{{ _ga_instance_id }}" }, + "instance" = "{{ _ga_instance_id }}"{% if grafana_alloy_node_job | length > 0 %}, + {# BREAKING-CHANGE ESCAPE HATCH: with no explicit job label Alloy falls back + to the component ID, so these series used to be job="prometheus.scrape.decdn_node". + Set grafana_alloy_node_job: "" to keep the old identity for dashboards + and alerts that already hard-code it. #} + "job" = {{ grafana_alloy_node_job | tojson }}{% endif %} }, ] forward_to = [prometheus.relabel.decdn_identity.receiver] scrape_interval = {{ grafana_alloy_scrape_interval | to_json }} @@ -48,14 +69,253 @@ prometheus.relabel "decdn_identity" { prometheus.remote_write "cloud" { endpoint { - url = sys.env("GC_PROM_REMOTE_WRITE_URL") + url = {{ ga_endpoint(grafana_alloy_prom_url, 'GC_PROM_REMOTE_WRITE_URL') }} + + basic_auth { + username = {{ ga_endpoint(grafana_alloy_prom_username, 'GC_PROM_USERNAME') }} + password = sys.env("GC_API_TOKEN") + } + } +} + +{# --------------------------------------------------------------------------- + HOST (machine) METRICS — Alloy's in-process node_exporter. + + Runs inside the agent: no extra listener and no extra port, so the + loopback-only posture is unchanged. Identity is applied with + discovery.relabel (TARGET labels — evaluated once per target, then attached to + every sample at scrape time) rather than prometheus.relabel (per SAMPLE, with + a cardinality-sized cache). The scrape forwards STRAIGHT to remote_write, + deliberately bypassing prometheus.relabel.decdn_identity: that component + stamps service_name=decdn-node, which is false for machine series. + + `job` must stay integrations/node_exporter for Grafana Cloud's prebuilt Linux + Server dashboards and alert rules to find these series. + Every regex below is JSON-encoded, so no value can break out of its string. #} +{% if grafana_alloy_host_metrics_enabled | bool %} +{# Upstream applies disable_collectors FIRST and enable_collectors second, so a + name in both ends up ENABLED — the opposite of what an operator vetoing a + collector expects. Subtract the veto list here instead, so it really wins and + so the derived `systemd` block below agrees with what actually runs. #} +{% set ga_host_set_collectors = grafana_alloy_host_collectors + | reject('in', grafana_alloy_host_disable_collectors) | list %} +{% set ga_host_extra_collectors = grafana_alloy_host_extra_collectors + | reject('in', grafana_alloy_host_disable_collectors) | list %} +{% set ga_host_enabled_collectors = ga_host_set_collectors + ga_host_extra_collectors %} + +prometheus.exporter.unix "host" { + // set_collectors REPLACES the default set (~45 collectors): anything not + // listed is off. enable_collectors adds on top (it is the only way to turn on + // an upstream-defaultDisabled collector such as systemd). Both lists already + // have grafana_alloy_host_disable_collectors subtracted at render time. + set_collectors = {{ ga_host_set_collectors | tojson }} +{% if ga_host_extra_collectors | length > 0 %} + enable_collectors = {{ ga_host_extra_collectors | tojson }} +{% endif %} +{% if grafana_alloy_host_disable_collectors | length > 0 %} + disable_collectors = {{ grafana_alloy_host_disable_collectors | tojson }} +{% endif %} + include_exporter_metrics = {{ grafana_alloy_host_include_exporter_metrics | bool | tojson }} + + filesystem { + fs_types_exclude = {{ grafana_alloy_host_fs_types_exclude | tojson }} + mount_points_exclude = {{ grafana_alloy_host_mount_points_exclude | tojson }} + mount_timeout = {{ grafana_alloy_host_mount_timeout | to_json }} + } + + netdev { + device_exclude = {{ grafana_alloy_host_netdev_device_exclude | tojson }} + } + + // `disk` is the block name for the diskstats collector. + disk { + device_exclude = {{ grafana_alloy_host_disk_device_exclude | tojson }} + } +{% if 'systemd' in ga_host_enabled_collectors %} + + // Per-unit state. Reaches systemd over the D-Bus SYSTEM bus (Alloy does not + // expose node_exporter's root-only private-socket mode), so it needs AF_UNIX + // and /run/dbus/system_bus_socket — both available to the hardened unit, and + // both absent in a bare container, where it simply reports nothing. + systemd { + unit_include = {{ grafana_alloy_host_systemd_unit_include | tojson }} + unit_exclude = {{ grafana_alloy_host_systemd_unit_exclude | tojson }} + } +{% endif %} +} + +{# Host series carry job/instance (what the Cloud dashboards filter on) plus the + fleet-identity labels, but NOT service_name: these describe the machine, not + the daemon. Empty values drop their rule entirely. #} +{% set ga_host_labels = [ + ["job", grafana_alloy_host_job], + ["instance", _ga_instance_id], + ["service_namespace", grafana_alloy_service_namespace], + ["deployment_environment", grafana_alloy_deployment_environment], + ["region", grafana_alloy_region], + ] %} +discovery.relabel "host" { + targets = prometheus.exporter.unix.host.targets +{% for name, value in ga_host_labels if value | length > 0 %} + + rule { + target_label = {{ name | tojson }} + replacement = {{ value | tojson }} + } +{% endfor %} +} + +prometheus.scrape "host" { + targets = discovery.relabel.host.output + forward_to = [prometheus.remote_write.cloud.receiver] + scrape_interval = {{ grafana_alloy_host_scrape_interval | to_json }} +} +{% endif %} +{% if grafana_alloy_self_metrics_enabled | bool %} + +{# Alloy's own health: component evaluation, WAL, remote-write queue and + failures. This is what distinguishes "the node went quiet" from "the agent + that reports on the node is wedged", so it is cheap insurance. #} +prometheus.exporter.self "alloy" {} + +{% set ga_self_labels = [ + ["job", grafana_alloy_self_job], + ["instance", _ga_instance_id], + ["service_namespace", grafana_alloy_service_namespace], + ["deployment_environment", grafana_alloy_deployment_environment], + ["region", grafana_alloy_region], + ] %} +discovery.relabel "alloy_self" { + targets = prometheus.exporter.self.alloy.targets +{% for name, value in ga_self_labels if value | length > 0 %} + + rule { + target_label = {{ name | tojson }} + replacement = {{ value | tojson }} + } +{% endfor %} +} + +prometheus.scrape "alloy_self" { + targets = discovery.relabel.alloy_self.output + forward_to = [prometheus.remote_write.cloud.receiver] + scrape_interval = {{ grafana_alloy_host_scrape_interval | to_json }} +} +{% endif %} +{% if grafana_alloy_logs_enabled | bool %} + +{# --------------------------------------------------------------------------- + HOST + SERVICE LOGS — journald to Grafana Cloud Loki. + + Two relabel stages, and the split is not cosmetic: journald fields arrive as + __journal*-prefixed labels that are STRIPPED before entries reach any + downstream receiver, so anything reading them (the field mapping AND the + cost-guardrail drops) MUST live in the rules-only ruleset consumed by + loki.source.journal. Identity labels are applied afterwards, on the entries. + + Reading the journal needs group membership (systemd-journal + adm), granted by + the role and pinned in the unit's SupplementaryGroups — without it the + component starts happily and collects nothing at all. The positions file lands + under --storage.path, i.e. inside the existing StateDirectory. #} +loki.relabel "journal_rules" { + // Rules-only usage: the ruleset is consumed via .rules below, so nothing is + // forwarded from here. The argument is still required. + forward_to = [] + + rule { + source_labels = ["__journal__systemd_unit"] + target_label = "unit" + } + + rule { + source_labels = ["__journal__hostname"] + target_label = "hostname" + } + + rule { + source_labels = ["__journal_priority_keyword"] + target_label = "level" + } + + rule { + source_labels = ["__journal__transport"] + target_label = "transport" + } + + rule { + source_labels = ["__journal__boot_id"] + target_label = "boot_id" + } +{% if grafana_alloy_logs_drop_priority_regex | length > 0 %} + + // Cost guardrail: the regex is anchored to the WHOLE value by Prometheus + // relabel semantics, so "debug" drops debug and nothing else. + rule { + source_labels = ["__journal_priority_keyword"] + regex = {{ grafana_alloy_logs_drop_priority_regex | tojson }} + action = "drop" + } +{% endif %} +{% if grafana_alloy_logs_drop_unit_regex | length > 0 %} + + // Cost guardrail: chatty units nobody debugs from Loki. Entries with no unit + // (kernel, for example) have an empty value here and are kept. + rule { + source_labels = ["__journal__systemd_unit"] + regex = {{ grafana_alloy_logs_drop_unit_regex | tojson }} + action = "drop" + } +{% endif %} +} + +loki.source.journal "host" { +{% if grafana_alloy_logs_journal_path | length > 0 %} + path = {{ grafana_alloy_logs_journal_path | tojson }} +{% else %} + // No path: Alloy reads both /var/log/journal and /run/log/journal. +{% endif %} + // Bounds the catch-up burst after an outage or a fresh install. + max_age = {{ grafana_alloy_logs_max_age | to_json }} + format_as_json = false + relabel_rules = loki.relabel.journal_rules.rules + forward_to = [loki.relabel.journal_identity.receiver] +} + +{# Same job/instance pair as the host metrics above — Grafana Cloud's Linux + Server integration dashboards correlate logs and metrics on exactly these + two labels. Set here (not via the source's `labels` argument) so they + override the job label loki.source.journal adds on its own. #} +{% set ga_log_labels = [ + ["job", grafana_alloy_logs_job], + ["instance", _ga_instance_id], + ["service_namespace", grafana_alloy_service_namespace], + ["deployment_environment", grafana_alloy_deployment_environment], + ["region", grafana_alloy_region], + ] %} +loki.relabel "journal_identity" { + forward_to = [loki.write.cloud.receiver] +{% for name, value in ga_log_labels if value | length > 0 %} + + rule { + target_label = {{ name | tojson }} + replacement = {{ value | tojson }} + } +{% endfor %} +} + +loki.write "cloud" { + endpoint { + url = {{ ga_endpoint(grafana_alloy_loki_url, 'GC_LOKI_URL') }} + // Grafana Cloud's Loki instance ID is NOT the Prometheus one; the same + // access-policy token authenticates both. basic_auth { - username = sys.env("GC_PROM_USERNAME") + username = {{ ga_endpoint(grafana_alloy_loki_username, 'GC_LOKI_USERNAME') }} password = sys.env("GC_API_TOKEN") } } } +{% endif %} otelcol.receiver.otlp "local" { grpc { @@ -141,7 +401,7 @@ otelcol.processor.batch "decdn" { `client` — nesting them inside `client` is rejected at config load. #} otelcol.exporter.otlphttp "cloud" { client { - endpoint = sys.env("GC_OTLP_ENDPOINT") + endpoint = {{ ga_endpoint(grafana_alloy_otlp_endpoint, 'GC_OTLP_ENDPOINT') }} auth = otelcol.auth.basic.cloud.handler } @@ -156,6 +416,6 @@ otelcol.exporter.otlphttp "cloud" { } otelcol.auth.basic "cloud" { - username = sys.env("GC_PROM_USERNAME") + username = {{ ga_endpoint(grafana_alloy_prom_username, 'GC_PROM_USERNAME') }} password = sys.env("GC_API_TOKEN") } diff --git a/ansible/roles/grafana_alloy/vars/main.yml b/ansible/roles/grafana_alloy/vars/main.yml index a2652d8..b7633b4 100644 --- a/ansible/roles/grafana_alloy/vars/main.yml +++ b/ansible/roles/grafana_alloy/vars/main.yml @@ -8,3 +8,92 @@ # destructive operation against an unrelated Alloy install. _ga_unit_path: /etc/systemd/system/alloy.service _ga_managed_marker: "# MANAGED BY the grafana_alloy role — do not edit by hand." + +# Every collector name prometheus.exporter.unix knows, from the Alloy 1.19 +# component reference (the "Collectors list" table), plus four that ship in +# upstream node_exporter but are missing from that table (cgroups, selinux, +# slabinfo, watchdog). Non-Linux names are kept so the gate never punishes a +# copy-pasted example. +# +# WHY A COMMITTED INVENTORY: an unknown collector name is NOT a startup error. +# Alloy builds the set by map lookup and silently drops anything it does not +# recognise, so `diskstat` (missing the final s) validates clean, starts clean, +# and ships no disk metrics forever. A charset check cannot catch that; only a +# name list can. Re-sync when bumping grafana_alloy_version. +_ga_known_collectors: + - arp + - bcache + - bonding + - boottime + - btrfs + - buddyinfo + - cgroups + - conntrack + - cpu + - cpufreq + - devstat + - diskstats + - dmi + - drbd + - drm + - edac + - entropy + - ethtool + - exec + - fibrechannel + - filefd + - filesystem + - hwmon + - infiniband + - interrupts + - ipvs + - ksmd + - lnstat + - loadavg + - logind + - mdadm + - meminfo + - meminfo_numa + - mountstats + - netclass + - netdev + - netisr + - netstat + - network_route + - nfs + - nfsd + - ntp + - nvme + - os + - perf + - powersupplyclass + - pressure + - processes + - qdisc + - rapl + - runit + - schedstat + - selinux + - slabinfo + - sockstat + - softirqs + - softnet + - stat + - supervisord + - sysctl + - systemd + - tapestats + - tcpstat + - textfile + - thermal + - thermal_zone + - time + - timex + - udp_queues + - uname + - vmstat + - watchdog + - wifi + - xfs + - zfs + - zoneinfo diff --git a/ansible/tests/alloy-config/render.yml b/ansible/tests/alloy-config/render.yml index 05a990d..ab8c003 100644 --- a/ansible/tests/alloy-config/render.yml +++ b/ansible/tests/alloy-config/render.yml @@ -10,13 +10,21 @@ # shipped green by the stub. This harness closes that gap. # # Renders several variable combinations so the template's conditional branches -# (omitted identity attributes, 0 and 1 sampling ratios, overridden ports) are -# all parsed by the real binary, not just the default one. +# (omitted identity attributes, 0 and 1 sampling ratios, overridden ports, each +# machine-monitoring sub-knob, inventory-vs-env credentials) are all parsed by +# the real binary, not just the default one. +# +# PRECEDENCE, and why every override lives on the TASK rather than the play: +# `vars_files` (precedence 14) outranks play `vars` (12), so a play-level +# override of anything defined in the role's defaults file is silently ignored — +# it renders the default and the case proves nothing. Task `vars` (19) win, so +# each combination below sets its knobs on the template task itself. Do not move +# them up to the play. # # Not a deploy playbook: localhost only, writes only into $ALLOY_RENDER_DIR, and # `become: false` overrides ansible.cfg's project-wide privilege escalation. -- name: Render the default-values Alloy configuration +- name: Render the grafana_alloy templates in every interesting combination hosts: localhost connection: local gather_facts: false @@ -25,97 +33,202 @@ - ../../roles/grafana_alloy/defaults/main.yml vars: _ga_render_dir: "{{ lookup('ansible.builtin.env', 'ALLOY_RENDER_DIR') }}" - _ga_instance_id: decdn-node-1 - grafana_alloy_bin_effective: /usr/bin/alloy - grafana_alloy_deployment_environment: production - grafana_alloy_region: US tasks: - - name: Render config.alloy and the systemd unit with default knobs + # Everything at its default: machine metrics, agent self-metrics and journald + # are all ON by default, so this is also the shape a fresh deploy gets. + - name: Render the default-values configuration ansible.builtin.template: src: "{{ playbook_dir }}/../../roles/grafana_alloy/templates/{{ item.src }}" dest: "{{ _ga_render_dir }}/defaults.{{ item.ext }}" mode: "0644" - loop: + loop: &ga_render_pair - {src: config.alloy.j2, ext: alloy} - {src: alloy.service.j2, ext: service} loop_control: label: "defaults.{{ item.ext }}" + vars: + _ga_instance_id: decdn-node-1 + _ga_journal_groups: [systemd-journal, adm] + grafana_alloy_bin_effective: /usr/bin/alloy + grafana_alloy_deployment_environment: production + grafana_alloy_region: US - # Hands validate.sh the release pin without making it parse YAML itself: the - # values come from the role's own defaults, so the download it verifies is - # byte-identical to the one the role installs. - - name: Export the pinned release for the validation script - ansible.builtin.copy: - dest: "{{ _ga_render_dir }}/pin.env" - mode: "0644" - content: | - GRAFANA_ALLOY_VERSION={{ grafana_alloy_version }} - GRAFANA_ALLOY_SHA256_AMD64={{ grafana_alloy_sha256.amd64 | default('') }} - GRAFANA_ALLOY_SHA256_ARM64={{ grafana_alloy_sha256.arm64 | default('') }} - -# Exercises the omit branches: an empty namespace/region must drop its relabel -# rule and its OTTL `set()` statement entirely rather than emit an empty value, -# and a 0 ratio must still render a valid float. -- name: Render the minimal-identity Alloy configuration - hosts: localhost - connection: local - gather_facts: false - become: false - vars_files: - - ../../roles/grafana_alloy/defaults/main.yml - vars: - _ga_render_dir: "{{ lookup('ansible.builtin.env', 'ALLOY_RENDER_DIR') }}" - _ga_instance_id: decdn-node-2 - grafana_alloy_bin_effective: /usr/local/bin/alloy - grafana_alloy_service_namespace: "" - grafana_alloy_deployment_environment: "" - grafana_alloy_region: "" - grafana_alloy_trace_sampling_ratio: 0 - tasks: - - name: Render config.alloy and the systemd unit with minimal identity + # Exercises the omit branches: an empty namespace/region must drop its relabel + # rule and its OTTL `set()` statement entirely rather than emit an empty value, + # and a 0 ratio must still render a valid float. + - name: Render the minimal-identity configuration ansible.builtin.template: src: "{{ playbook_dir }}/../../roles/grafana_alloy/templates/{{ item.src }}" dest: "{{ _ga_render_dir }}/minimal.{{ item.ext }}" mode: "0644" - loop: - - {src: config.alloy.j2, ext: alloy} - - {src: alloy.service.j2, ext: service} + loop: *ga_render_pair loop_control: label: "minimal.{{ item.ext }}" + vars: + _ga_instance_id: decdn-node-2 + grafana_alloy_bin_effective: /usr/local/bin/alloy + grafana_alloy_service_namespace: "" + grafana_alloy_deployment_environment: "" + grafana_alloy_region: "" + grafana_alloy_trace_sampling_ratio: 0 -# Exercises the overridden branches: non-default ports, a keep-everything ratio, -# and pipeline knobs well away from their defaults. -- name: Render the fully-overridden Alloy configuration - hosts: localhost - connection: local - gather_facts: false - become: false - vars_files: - - ../../roles/grafana_alloy/defaults/main.yml - vars: - _ga_render_dir: "{{ lookup('ansible.builtin.env', 'ALLOY_RENDER_DIR') }}" - _ga_instance_id: decdn-node-3 - grafana_alloy_bin_effective: /usr/bin/alloy - grafana_alloy_deployment_environment: staging - grafana_alloy_region: EU - grafana_alloy_trace_sampling_ratio: 1 - grafana_alloy_scrape_interval: 15s - grafana_alloy_batch_timeout: 500ms - grafana_alloy_batch_send_size: 512 - grafana_alloy_batch_max_size: 4096 - grafana_alloy_queue_size: 100 - grafana_alloy_retry_initial_interval: 1s - grafana_alloy_retry_max_elapsed_time: 1h - grafana_alloy_otlp_http_port: 14318 - decdn_metrics_port: 19090 - tasks: - - name: Render config.alloy and the systemd unit with overridden knobs + # Exercises the overridden branches: non-default ports, a keep-everything + # ratio, and pipeline knobs well away from their defaults. + - name: Render the fully-overridden configuration ansible.builtin.template: src: "{{ playbook_dir }}/../../roles/grafana_alloy/templates/{{ item.src }}" dest: "{{ _ga_render_dir }}/overridden.{{ item.ext }}" mode: "0644" - loop: - - {src: config.alloy.j2, ext: alloy} - - {src: alloy.service.j2, ext: service} + loop: *ga_render_pair loop_control: label: "overridden.{{ item.ext }}" + vars: + _ga_instance_id: decdn-node-3 + _ga_journal_groups: [systemd-journal] + grafana_alloy_bin_effective: /usr/bin/alloy + grafana_alloy_deployment_environment: staging + grafana_alloy_region: EU + grafana_alloy_trace_sampling_ratio: 1 + grafana_alloy_scrape_interval: 15s + grafana_alloy_batch_timeout: 500ms + grafana_alloy_batch_send_size: 512 + grafana_alloy_batch_max_size: 4096 + grafana_alloy_queue_size: 100 + grafana_alloy_retry_initial_interval: 1s + grafana_alloy_retry_max_elapsed_time: 1h + grafana_alloy_otlp_http_port: 14318 + grafana_alloy_node_job: decdn-node-custom + decdn_metrics_port: 19090 + + # --- Machine-monitoring matrix ------------------------------------------ + # The default case above already renders host metrics, self metrics and + # journald together; these four isolate the branches that would otherwise + # never be parsed by the real binary. + + # Host metrics ONLY, hand-picked collectors, no journal groups on the host: + # proves the exporter/discovery.relabel/scrape trio stands alone, that the + # systemd sub-block disappears with its collector, and that ProtectHome + # relaxes to read-only. + - name: Render the host-metrics-only configuration + ansible.builtin.template: + src: "{{ playbook_dir }}/../../roles/grafana_alloy/templates/{{ item.src }}" + dest: "{{ _ga_render_dir }}/hostonly.{{ item.ext }}" + mode: "0644" + loop: *ga_render_pair + loop_control: + label: "hostonly.{{ item.ext }}" + vars: + _ga_instance_id: decdn-node-4 + _ga_journal_groups: [] + grafana_alloy_bin_effective: /usr/bin/alloy + grafana_alloy_deployment_environment: production + grafana_alloy_region: US + grafana_alloy_host_metrics_enabled: true + grafana_alloy_self_metrics_enabled: false + grafana_alloy_logs_enabled: false + grafana_alloy_host_scrape_interval: 120s + grafana_alloy_host_collectors: [cpu, meminfo] + grafana_alloy_host_extra_collectors: [] + grafana_alloy_host_disable_collectors: [hwmon] + grafana_alloy_host_include_exporter_metrics: true + + # Logs ONLY, explicit journal path, both cost-guardrail drops cleared (the + # omit branches of the rules), journal groups present — the only combination + # that renders SupplementaryGroups. + - name: Render the logs-only configuration + ansible.builtin.template: + src: "{{ playbook_dir }}/../../roles/grafana_alloy/templates/{{ item.src }}" + dest: "{{ _ga_render_dir }}/logsonly.{{ item.ext }}" + mode: "0644" + loop: *ga_render_pair + loop_control: + label: "logsonly.{{ item.ext }}" + vars: + _ga_instance_id: decdn-node-5 + _ga_journal_groups: [systemd-journal, adm] + grafana_alloy_bin_effective: /usr/bin/alloy + grafana_alloy_deployment_environment: production + grafana_alloy_region: US + grafana_alloy_host_metrics_enabled: false + grafana_alloy_self_metrics_enabled: false + grafana_alloy_logs_enabled: true + grafana_alloy_logs_journal_path: /var/log/journal + grafana_alloy_logs_max_age: 30m + grafana_alloy_logs_drop_priority_regex: "" + grafana_alloy_logs_drop_unit_regex: "" + + # BACKWARDS COMPATIBILITY: every sub-knob off must reproduce the + # pre-machine-monitoring pipeline exactly — node scrape + OTLP, all + # credentials via sys.env. validate.sh asserts the absence of every new + # component here. + - name: Render the legacy (sub-knobs off) configuration + ansible.builtin.template: + src: "{{ playbook_dir }}/../../roles/grafana_alloy/templates/{{ item.src }}" + dest: "{{ _ga_render_dir }}/legacy.{{ item.ext }}" + mode: "0644" + loop: *ga_render_pair + loop_control: + label: "legacy.{{ item.ext }}" + vars: + _ga_instance_id: decdn-node-6 + grafana_alloy_bin_effective: /usr/bin/alloy + grafana_alloy_deployment_environment: production + grafana_alloy_region: US + grafana_alloy_host_metrics_enabled: false + grafana_alloy_self_metrics_enabled: false + grafana_alloy_logs_enabled: false + + # The inventory-driven connection settings: all five non-secret values set, + # so every sys.env lookup EXCEPT the token must disappear. + - name: Render the inventory-endpoint configuration + ansible.builtin.template: + src: "{{ playbook_dir }}/../../roles/grafana_alloy/templates/{{ item.src }}" + dest: "{{ _ga_render_dir }}/inventory.{{ item.ext }}" + mode: "0644" + loop: *ga_render_pair + loop_control: + label: "inventory.{{ item.ext }}" + vars: + _ga_instance_id: decdn-node-7 + _ga_journal_groups: [systemd-journal, adm] + grafana_alloy_bin_effective: /usr/bin/alloy + grafana_alloy_deployment_environment: production + grafana_alloy_region: US + grafana_alloy_prom_url: https://prometheus-prod-99.render.invalid/api/prom/push + grafana_alloy_prom_username: "1234567" + grafana_alloy_otlp_endpoint: https://otlp-gateway-prod-99.render.invalid/otlp + grafana_alloy_loki_url: https://logs-prod-99.render.invalid/loki/api/v1/push + grafana_alloy_loki_username: "7654321" + + # The two escape hatches an upgrading operator reaches for: vetoing a + # collector must actually remove it (and its scoping block) rather than + # leaving it enabled-but-unscoped, and blanking the node job must restore the + # pre-existing component-ID identity for dashboards that hard-code it. + - name: Render the vetoed-collector configuration + ansible.builtin.template: + src: "{{ playbook_dir }}/../../roles/grafana_alloy/templates/{{ item.src }}" + dest: "{{ _ga_render_dir }}/vetoed.{{ item.ext }}" + mode: "0644" + loop: *ga_render_pair + loop_control: + label: "vetoed.{{ item.ext }}" + vars: + _ga_instance_id: decdn-node-8 + _ga_journal_groups: [systemd-journal, adm] + grafana_alloy_bin_effective: /usr/bin/alloy + grafana_alloy_deployment_environment: production + grafana_alloy_region: US + grafana_alloy_node_job: "" + grafana_alloy_host_disable_collectors: [systemd, netdev] + + # Hands validate.sh the release pin without making it parse YAML itself: the + # values come from the role's own defaults, so the download it verifies is + # byte-identical to the one the role installs. + - name: Export the pinned release for the validation script + ansible.builtin.copy: + dest: "{{ _ga_render_dir }}/pin.env" + mode: "0644" + content: | + GRAFANA_ALLOY_VERSION={{ grafana_alloy_version }} + GRAFANA_ALLOY_SHA256_AMD64={{ grafana_alloy_sha256.amd64 | default('') }} + GRAFANA_ALLOY_SHA256_ARM64={{ grafana_alloy_sha256.arm64 | default('') }} diff --git a/ansible/tests/alloy-config/validate.sh b/ansible/tests/alloy-config/validate.sh index 10f462b..22bcfd6 100755 --- a/ansible/tests/alloy-config/validate.sh +++ b/ansible/tests/alloy-config/validate.sh @@ -106,9 +106,84 @@ for config in "${configs[@]}"; do if ! grep -qF 'sys.env("GC_API_TOKEN")' "$config"; then fail "$name does not read the API token via sys.env — credentials must never be literals" fi + # The non-secret endpoints may be inventory literals; the token may NOT. This + # catches a token pasted into a group_vars file and rendered into the 0644 + # config on the host. + if grep -qF 'glc_' "$config"; then + fail "$name contains a literal Grafana Cloud token — it belongs only in the 0600 env file" + fi echo "ok: $name" done +# --- Gate 1b: the machine-monitoring matrix renders what it claims ------------ +# `alloy validate` proves each file loads; these greps prove the Jinja branches +# actually switched. Without them a knob could silently render nothing and still +# pass every other gate. +assert_has() { # file, needle, why + grep -qF "$2" "$render_dir/$1" || fail "$1 is missing $3 ($2)" +} +assert_lacks() { # file, needle, why + ! grep -qF "$2" "$render_dir/$1" || fail "$1 unexpectedly contains $3 ($2)" +} + +# Defaults: machine metrics, agent self-metrics and journald are all ON, and the +# job labels Grafana Cloud's Linux Server integration dashboards key off. +assert_has defaults.alloy 'prometheus.exporter.unix "host"' "the host metrics exporter" +assert_has defaults.alloy 'prometheus.exporter.self "alloy"' "agent self-monitoring" +assert_has defaults.alloy 'loki.source.journal "host"' "the journald source" +assert_has defaults.alloy 'loki.write "cloud"' "the Loki writer" +assert_has defaults.alloy '"job"' "an explicit job label" +assert_has defaults.alloy 'integrations/node_exporter' "the Grafana Cloud integration job label" +assert_has defaults.alloy '__journal__systemd_unit' "the journal field mapping" +assert_has defaults.alloy 'systemd {' "the per-unit systemd collector" + +# Sub-knob isolation. +assert_has hostonly.alloy 'prometheus.exporter.unix "host"' "the host metrics exporter" +assert_lacks hostonly.alloy 'loki.' "log components while logs are disabled" +assert_lacks hostonly.alloy 'prometheus.exporter.self' "self metrics while they are disabled" +assert_lacks hostonly.alloy 'systemd {' "the systemd collector block with no systemd collector enabled" +assert_has logsonly.alloy 'loki.source.journal "host"' "the journald source" +assert_has logsonly.alloy 'path = "/var/log/journal"' "the explicit journal path" +assert_lacks logsonly.alloy 'prometheus.exporter' "any exporter while only logs are enabled" +assert_lacks logsonly.alloy 'action = "drop"' "a drop rule after both guardrail regexes were cleared" + +# BACKWARDS COMPATIBILITY: sub-knobs off == the pre-machine-monitoring pipeline. +assert_lacks legacy.alloy 'prometheus.exporter' "any exporter" +assert_lacks legacy.alloy 'discovery.relabel' "any target relabeller" +assert_lacks legacy.alloy 'loki.' "any log component" +assert_has legacy.alloy 'sys.env("GC_PROM_REMOTE_WRITE_URL")' "the env fallback for remote-write" +assert_has legacy.alloy 'sys.env("GC_PROM_USERNAME")' "the env fallback for the instance ID" +assert_has legacy.alloy 'sys.env("GC_OTLP_ENDPOINT")' "the env fallback for OTLP" +assert_lacks legacy.alloy 'GC_LOKI' "a Loki credential lookup while logs are disabled" + +# Inventory-driven connection settings: every non-secret sys.env lookup is gone, +# the token's is not. +for key in GC_PROM_REMOTE_WRITE_URL GC_PROM_USERNAME GC_OTLP_ENDPOINT GC_LOKI_URL GC_LOKI_USERNAME; do + assert_lacks inventory.alloy "sys.env(\"$key\")" "an env lookup that inventory already supplies" +done +assert_has inventory.alloy 'https://prometheus-prod-99.render.invalid/api/prom/push' "the inventory remote-write URL" +assert_has inventory.alloy 'https://logs-prod-99.render.invalid/loki/api/v1/push' "the inventory Loki URL" +assert_has inventory.alloy 'sys.env("GC_API_TOKEN")' "the token env lookup, which must never move to inventory" + +# Escape hatches: the collector veto must remove the collector AND its scoping +# block from the rendered lists (upstream would otherwise re-enable it), and a +# blank node job must restore the implicit component-ID identity. +assert_has vetoed.alloy 'prometheus.exporter.unix "host"' "the host metrics exporter" +assert_lacks vetoed.alloy 'systemd {' "the systemd block for a vetoed collector" +assert_lacks vetoed.alloy 'enable_collectors =' "an enable list whose only entry was vetoed" +assert_has vetoed.alloy 'disable_collectors =' "the veto list itself" +assert_has defaults.alloy 'enable_collectors = ["systemd"]' "the systemd collector on the enable list" +assert_lacks vetoed.alloy '"job" = "decdn-node"' "a node job label that was explicitly blanked" +assert_has vetoed.alloy 'replacement = "integrations/node_exporter"' "the host job label" + +# Unit hardening follows the enabled signals. +assert_has hostonly.service 'ProtectHome=read-only' "the ProtectHome relaxation host metrics require" +assert_has legacy.service 'ProtectHome=true' "the strict ProtectHome used when host metrics are off" +assert_has logsonly.service 'SupplementaryGroups=systemd-journal adm' "the journal reader groups" +assert_lacks hostonly.service 'SupplementaryGroups' "journal groups while logs are disabled" +assert_lacks legacy.service 'SupplementaryGroups' "journal groups while logs are disabled" +echo "ok: machine-monitoring render matrix" + # --- Gate 2: every ExecStart flag exists ------------------------------------- # `alloy run` ignores nothing: an unknown flag exits non-zero, i.e. a systemd # crash-loop on the target host the moment the unit starts. From 57f6839e9721cf88051a1d653e7533191dbfb66c Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Thu, 17 Sep 2026 05:00:01 +0300 Subject: [PATCH 2/2] fix(ansible): address grafana_alloy review findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Numeric instance IDs are the common case (Grafana Cloud hands you a number), and YAML hands an unquoted one over as an int. Both the ga_endpoint macro and the credential-requirement flags took its length, and tojson would have emitted a bare number into a field Alloy types as a string. Coerce once in each place, and pin the behaviour by feeding an unquoted ID through both the render matrix and the grafana-cloud scenario. Turning the logs signal off dropped SupplementaryGroups from the unit but left the account in systemd-journal/adm from the earlier run — and systemd falls back to the account's groups, so the agent kept journal read access after the signal it was granted for was switched off. Take the grant back, preserving any other membership, and assert the transition in the scenario that created it. Also: warn when only one journal group exists instead of only when both are missing; correct the "requires both groups" claim (systemd-journal owns the files, adm gets an ACL — either grants read); keep validate.sh's literal-token scan in lockstep with the preflight gate (glsa_, eyJ); blank the node job in the legacy render case so "reproduces the old pipeline" is literally true; and describe the instance-ID contract as the endpoint's basic-auth username, which is what the charset actually allows. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ansible/README.md | 6 +- ansible/molecule/grafana-cloud/converge.yml | 4 ++ ansible/molecule/grafana-cloud/verify.yml | 64 +++++++++++++++++- ansible/roles/grafana_alloy/defaults/main.yml | 9 +-- ansible/roles/grafana_alloy/tasks/main.yml | 67 +++++++++++++++---- .../roles/grafana_alloy/tasks/preflight.yml | 20 +++--- .../grafana_alloy/templates/alloy.service.j2 | 13 ++-- .../grafana_alloy/templates/config.alloy.j2 | 9 ++- ansible/tests/alloy-config/render.yml | 11 ++- ansible/tests/alloy-config/validate.sh | 10 ++- 10 files changed, 168 insertions(+), 45 deletions(-) diff --git a/ansible/README.md b/ansible/README.md index a9ae0b5..834e41b 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -187,11 +187,11 @@ primitives underneath it. All take `--dry-run`. See ### Grafana Cloud observability (opt-in) The play ships a third role, `grafana_alloy`, that installs a loopback-only -[Grafana Alloy](https://grafana.com/docs/alloy/) agent shipping four signals to your +[Grafana Alloy](https://grafana.com/docs/alloy/) agent shipping five signals to your Grafana Cloud org — the node's `/metrics`, **the machine itself** (CPU/memory/disk/ filesystem/network/load plus per-unit state, via Alloy's in-process `node_exporter`), -**journald**, and the daemon's OTLP spans (`127.0.0.1:4317`) — off by default, driven -by ONE mirrored inventory flag: +**journald**, the agent's own health, and the daemon's OTLP spans (`127.0.0.1:4317`) — +off by default, driven by ONE mirrored inventory flag: ```yaml # group_vars/host_vars — drives BOTH roles from one knob diff --git a/ansible/molecule/grafana-cloud/converge.yml b/ansible/molecule/grafana-cloud/converge.yml index 21c5b02..6b8d0a6 100644 --- a/ansible/molecule/grafana-cloud/converge.yml +++ b/ansible/molecule/grafana-cloud/converge.yml @@ -49,6 +49,10 @@ # The Loki URL comes from inventory while its username stays in the host env # file (see prepare.yml): one converge exercises both credential paths. grafana_alloy_loki_url: https://logs-prod-xx.molecule.invalid/loki/api/v1/push + # DELIBERATELY UNQUOTED: a numeric instance ID arrives from YAML as an int, + # so every gate and the template must coerce it instead of taking its length + # or emitting a bare number into a string field. + grafana_alloy_prom_username: 1234567 roles: - role: grafana_alloy tags: [observability] diff --git a/ansible/molecule/grafana-cloud/verify.yml b/ansible/molecule/grafana-cloud/verify.yml index 2a99270..caae897 100644 --- a/ansible/molecule/grafana-cloud/verify.yml +++ b/ansible/molecule/grafana-cloud/verify.yml @@ -148,11 +148,19 @@ 'sys.env("GC_LOKI_URL")' not in ga_config - >- 'sys.env("GC_LOKI_USERNAME")' in ga_config + # The unquoted numeric instance ID in converge must reach the config as + # a QUOTED string (Alloy's basic_auth username is a string) and must + # have survived every preflight gate that takes its length. + - >- + 'username = "1234567"' in ga_config + - >- + 'sys.env("GC_PROM_USERNAME")' not in ga_config fail_msg: >- The inventory-vs-env credential split did not render as expected: - grafana_alloy_loki_url was set in inventory (so its sys.env lookup must - be gone) while GC_LOKI_USERNAME was left to the host env file (so its - lookup must remain). Got: {{ ga_config }} + grafana_alloy_loki_url and the numeric grafana_alloy_prom_username were + set in inventory (so their sys.env lookups must be gone and the ID must + appear quoted) while GC_LOKI_USERNAME was left to the host env file (so + its lookup must remain). Got: {{ ga_config }} # --- Secret hygiene ------------------------------------------------------------- - name: Read the credential fixture back @@ -306,6 +314,56 @@ and ansible_facts.services['decdn-node.service'].state == 'running' fail_msg: "alloy or decdn-node is not running — journalctl -u alloy -e / -u decdn-node -e" +# Also mutates the converged host, so it runs after every assertion above and +# before the teardown play below. +- name: Verify turning logs off takes the journal grant back + hosts: all + become: true + tasks: + # Dropping SupplementaryGroups from the unit is not enough on its own: + # systemd then falls back to the ACCOUNT's supplementary groups, so a + # membership left over from the logs-enabled run would silently keep journal + # read access alive after the operator turned logs off. + - name: Re-run the role with only the logs signal disabled + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_install_method: manual + grafana_alloy_manual_bin_src: >- + {{ lookup('ansible.builtin.env', 'MOLECULE_SCENARIO_DIRECTORY') }}/files/alloy-stub + grafana_alloy_deployment_environment: production + grafana_alloy_region: US + grafana_alloy_logs_enabled: false + + - name: Read the group database back + ansible.builtin.getent: + database: group + + - name: Read the unit and config back + ansible.builtin.slurp: + src: "{{ item }}" + register: ga_logs_off + loop: + - /etc/systemd/system/alloy.service + - /etc/alloy/config.alloy + + - name: Assert the journal grant is gone from both the unit and the account + ansible.builtin.assert: + that: + - "'alloy' not in (ansible_facts.getent_group['systemd-journal'][2] | default('')).split(',')" + - "'alloy' not in (ansible_facts.getent_group['adm'][2] | default('')).split(',')" + - "'SupplementaryGroups' not in ga_unit_off" + - "'loki.' not in ga_config_off" + fail_msg: >- + Turning grafana_alloy_logs_enabled off left journal access behind: + SupplementaryGroups in the unit or a stale systemd-journal/adm + membership on the alloy account would keep the agent able to read the + journal after the signal it was granted for was switched off. + vars: + ga_unit_off: "{{ ga_logs_off.results[0].content | b64decode }}" + ga_config_off: "{{ ga_logs_off.results[1].content | b64decode }}" + # Runs LAST, after the idempotence phase and the assertions above, because it # deliberately mutates the converged host: it exercises the disable path, whose # whole job is to remove things. diff --git a/ansible/roles/grafana_alloy/defaults/main.yml b/ansible/roles/grafana_alloy/defaults/main.yml index 69861dc..ae6db09 100644 --- a/ansible/roles/grafana_alloy/defaults/main.yml +++ b/ansible/roles/grafana_alloy/defaults/main.yml @@ -155,10 +155,11 @@ grafana_alloy_self_metrics_enabled: true grafana_alloy_self_job: integrations/alloy # --- Host + service logs (journald -> Grafana Cloud Loki) --------------------- -# Reading the journal as a non-root user requires membership in systemd-journal -# AND adm — without both the component starts cleanly and collects NOTHING, so -# the role joins the alloy user to whichever of them exist on the host and the -# unit lists them under SupplementaryGroups. +# Reading the journal as a non-root user needs a group journald grants read to: +# systemd-journal owns the journal files, adm gets an ACL on /var/log/journal. +# With neither, the component starts cleanly and collects NOTHING, so the role +# joins the alloy user to both where they exist, lists them under +# SupplementaryGroups, and warns when either is missing. grafana_alloy_logs_enabled: true # Grafana Cloud's Linux Server integration uses the SAME job value for logs as # for host metrics; its logs dashboards key off job + instance. diff --git a/ansible/roles/grafana_alloy/tasks/main.yml b/ansible/roles/grafana_alloy/tasks/main.yml index 1ea256d..c9b1a47 100644 --- a/ansible/roles/grafana_alloy/tasks/main.yml +++ b/ansible/roles/grafana_alloy/tasks/main.yml @@ -146,12 +146,14 @@ ansible.builtin.import_tasks: preflight.yml # --- Journal access --------------------------------------------------------- - # Reading the systemd journal as a non-root user requires membership in - # systemd-journal AND adm. Without them loki.source.journal starts WITHOUT - # error and collects nothing, so this is a silent-failure guard, not a - # convenience. Detected rather than assumed: a container or a non-Debian host - # may ship neither, and systemd refuses to start a unit whose - # SupplementaryGroups names a group that does not exist. + # A non-root journal reader needs a group journald grants read to: + # systemd-journal owns the journal files, adm gets an ACL on /var/log/journal. + # With neither, loki.source.journal starts WITHOUT error and collects nothing, + # so this is a silent-failure guard, not a convenience. Detected rather than + # assumed: a container or a non-Debian host may ship neither, and systemd + # refuses to start a unit whose SupplementaryGroups names a group that does + # not exist. A partial set still works but is reported, since it means one of + # the two access paths is unavailable. - name: Detect which journal reader groups exist on this host ansible.builtin.command: cmd: getent group {{ item | quote }} @@ -169,17 +171,23 @@ | selectattr('rc', 'equalto', 0) | map(attribute='item') | list }} - - name: Report that journald collection will be silently empty + - name: Report that journald collection will be degraded or empty ansible.builtin.debug: msg: >- - Neither systemd-journal nor adm exists on this host, so - {{ grafana_alloy_user }} cannot read the journal: loki.source.journal - will run without error and ship no log lines. Expected inside - containers; on a real host, install systemd properly or set + {{ _ga_missing_journal_groups | join(' and ') }} missing on this host, + so {{ grafana_alloy_user }} gets + {{ (_ga_journal_groups | length == 0) | ternary('no journal access at + all: loki.source.journal will run without error and ship no log + lines', 'only partial journal access, which covers the common case but + not journal files whose ACL names only the missing group') }}. Expected + inside containers; on a real host, install systemd properly or set grafana_alloy_logs_enabled=false. + vars: + _ga_missing_journal_groups: >- + {{ ['systemd-journal', 'adm'] | difference(_ga_journal_groups) }} when: - grafana_alloy_logs_enabled | bool - - _ga_journal_groups | length == 0 + - _ga_journal_groups | length < 2 # --- User + directories ---------------------------------------------------- - name: Create the alloy system group @@ -192,8 +200,9 @@ name: "{{ grafana_alloy_user }}" group: "{{ grafana_alloy_group }}" # Journal readers only, and only those that exist. append keeps any - # membership an operator added by hand; the disable path removes the - # account outright, so nothing is left behind either way. + # membership an operator added by hand; the task below revokes the + # journal grant again if logs are later turned off, and the full disable + # path removes the account outright. groups: "{{ _ga_journal_groups }}" append: true system: true @@ -204,6 +213,36 @@ # does not also change the unit or the config still needs a restart. notify: Restart alloy + # Turning logs off drops SupplementaryGroups from the unit, but systemd then + # falls back to the ACCOUNT's supplementary groups — so a membership granted + # by an earlier logs-enabled run would silently keep journal read access. Take + # it back, preserving anything else the account belongs to. + - name: Read the alloy account's current supplementary groups + ansible.builtin.command: + cmd: id -nG {{ grafana_alloy_user | quote }} + changed_when: false + failed_when: false + check_mode: false # read-only lookup; must also run under make check + register: _ga_current_groups + when: not grafana_alloy_logs_enabled | bool + + - name: Revoke journal reader groups granted by an earlier logs-enabled run + ansible.builtin.user: + name: "{{ grafana_alloy_user }}" + groups: "{{ _ga_keep_groups }}" + append: false + vars: + # The primary group is set by `group:` above and must not be listed here. + _ga_keep_groups: >- + {{ (_ga_current_groups.stdout | default('')).split() + | difference(['systemd-journal', 'adm', grafana_alloy_group]) }} + when: + - not grafana_alloy_logs_enabled | bool + - _ga_current_groups.rc | default(1) == 0 + - (_ga_current_groups.stdout | default('')).split() + | intersect(['systemd-journal', 'adm']) | length > 0 + notify: Restart alloy + - name: Ensure config + state directories exist ansible.builtin.file: path: "{{ item.path }}" diff --git a/ansible/roles/grafana_alloy/tasks/preflight.yml b/ansible/roles/grafana_alloy/tasks/preflight.yml index 49c0ec0..c459b4f 100644 --- a/ansible/roles/grafana_alloy/tasks/preflight.yml +++ b/ansible/roles/grafana_alloy/tasks/preflight.yml @@ -203,10 +203,12 @@ - item.value | length == 0 or item.value is match('^[A-Za-z0-9._-]{1,64}$') quiet: true fail_msg: >- - {{ item.name }} must be the numeric Grafana Cloud instance ID (letters, - digits, dot, underscore, dash), not a token and not a quoted string with - whitespace — got "{{ item.value }}". NOTE the Loki instance ID differs from - the Prometheus one; the access-policy token is shared and stays in + {{ item.name }} is the basic-auth username for the endpoint — the numeric + instance ID on Grafana Cloud, a tenant name on a self-hosted + Mimir/Loki — so it is restricted to letters, digits, dot, underscore and + dash, and must not be a token or carry whitespace. Got "{{ item.value }}". + NOTE the Loki instance ID differs from the Prometheus one; the + access-policy token is shared and stays in {{ grafana_alloy_secret_file }} as GC_API_TOKEN. loop: - {name: grafana_alloy_prom_username, value: "{{ grafana_alloy_prom_username | string }}"} @@ -358,23 +360,23 @@ - key: GC_PROM_REMOTE_WRITE_URL regex: ^GC_PROM_REMOTE_WRITE_URL=https://[^[:space:]]+$ var: grafana_alloy_prom_url - required: "{{ grafana_alloy_prom_url | length == 0 }}" + required: "{{ grafana_alloy_prom_url | string | length == 0 }}" - key: GC_PROM_USERNAME regex: ^GC_PROM_USERNAME=[^[:space:]#][^[:space:]]*$ var: grafana_alloy_prom_username - required: "{{ grafana_alloy_prom_username | length == 0 }}" + required: "{{ grafana_alloy_prom_username | string | length == 0 }}" - key: GC_OTLP_ENDPOINT regex: ^GC_OTLP_ENDPOINT=https://[^[:space:]]+$ var: grafana_alloy_otlp_endpoint - required: "{{ grafana_alloy_otlp_endpoint | length == 0 }}" + required: "{{ grafana_alloy_otlp_endpoint | string | length == 0 }}" - key: GC_LOKI_URL regex: ^GC_LOKI_URL=https://[^[:space:]]+$ var: grafana_alloy_loki_url - required: "{{ grafana_alloy_logs_enabled | bool and grafana_alloy_loki_url | length == 0 }}" + required: "{{ grafana_alloy_logs_enabled | bool and grafana_alloy_loki_url | string | length == 0 }}" - key: GC_LOKI_USERNAME regex: ^GC_LOKI_USERNAME=[^[:space:]#][^[:space:]]*$ var: grafana_alloy_loki_username - required: "{{ grafana_alloy_logs_enabled | bool and grafana_alloy_loki_username | length == 0 }}" + required: "{{ grafana_alloy_logs_enabled | bool and grafana_alloy_loki_username | string | length == 0 }}" loop_control: label: "{{ item.key }}" when: item.required | bool diff --git a/ansible/roles/grafana_alloy/templates/alloy.service.j2 b/ansible/roles/grafana_alloy/templates/alloy.service.j2 index 9caa2ff..0e1a56a 100644 --- a/ansible/roles/grafana_alloy/templates/alloy.service.j2 +++ b/ansible/roles/grafana_alloy/templates/alloy.service.j2 @@ -37,12 +37,13 @@ Restart=on-failure RestartSec=5 {% if grafana_alloy_logs_enabled | default(false) | bool and (_ga_journal_groups | default([]) | length > 0) %} -# Journal access. Reading the systemd journal as a non-root user requires -# membership in systemd-journal AND adm — without them loki.source.journal starts -# without error and collects NOTHING, which is the worst possible failure mode. -# The role joins {{ grafana_alloy_user }} to these groups too; only groups that -# actually exist on this host are listed, because systemd refuses to start a unit -# naming a group that does not exist. +# Journal access. A non-root reader needs a group that journald grants read to: +# systemd-journal owns the journal files, and systemd-tmpfiles puts an ACL for +# adm on /var/log/journal. Without at least one of them loki.source.journal +# starts without error and collects NOTHING, which is the worst possible failure +# mode, so the role joins {{ grafana_alloy_user }} to both and warns when either +# is absent. Only groups that actually exist on this host are listed, because +# systemd refuses to start a unit naming a group that does not exist. SupplementaryGroups={{ _ga_journal_groups | default([]) | join(' ') }} {% endif %} diff --git a/ansible/roles/grafana_alloy/templates/config.alloy.j2 b/ansible/roles/grafana_alloy/templates/config.alloy.j2 index a61a37d..8083103 100644 --- a/ansible/roles/grafana_alloy/templates/config.alloy.j2 +++ b/ansible/roles/grafana_alloy/templates/config.alloy.j2 @@ -27,9 +27,14 @@ way — all keys in /etc/grafana-alloy.env — renders exactly the previous configuration, while a host driven from inventory needs only GC_API_TOKEN on disk. tojson does the quoting/escaping in both branches. - NEVER call this for the API token: config.alloy is 0644 on the host. #} + NEVER call this for the API token: config.alloy is 0644 on the host. + The `| string` is load-bearing: instance IDs are numeric, so an unquoted + `grafana_alloy_prom_username: 1234567` in inventory arrives as an int, which + has no length and which tojson would emit as a bare number — and Alloy's + basic_auth username is a string, so that is a config-load failure. #} {% macro ga_endpoint(value, env_key) -%} -{% if value | default('', true) | length > 0 %}{{ value | tojson }}{% else %}sys.env({{ env_key | tojson }}){% endif %} +{% set ga_value = value | default('', true) | string %} +{% if ga_value | length > 0 %}{{ ga_value | tojson }}{% else %}sys.env({{ env_key | tojson }}){% endif %} {%- endmacro %} prometheus.scrape "decdn_node" { diff --git a/ansible/tests/alloy-config/render.yml b/ansible/tests/alloy-config/render.yml index ab8c003..4005fb8 100644 --- a/ansible/tests/alloy-config/render.yml +++ b/ansible/tests/alloy-config/render.yml @@ -174,6 +174,10 @@ grafana_alloy_bin_effective: /usr/bin/alloy grafana_alloy_deployment_environment: production grafana_alloy_region: US + # Blanked too: the explicit node job label is itself new, so leaving it + # at its default would change the daemon series' identity and this case + # would no longer be the old pipeline. + grafana_alloy_node_job: "" grafana_alloy_host_metrics_enabled: false grafana_alloy_self_metrics_enabled: false grafana_alloy_logs_enabled: false @@ -195,10 +199,13 @@ grafana_alloy_deployment_environment: production grafana_alloy_region: US grafana_alloy_prom_url: https://prometheus-prod-99.render.invalid/api/prom/push - grafana_alloy_prom_username: "1234567" + # DELIBERATELY UNQUOTED: instance IDs are numeric, so YAML hands these + # over as ints. Alloy's basic_auth username is a string, so the template + # must coerce rather than emit a bare number. + grafana_alloy_prom_username: 1234567 grafana_alloy_otlp_endpoint: https://otlp-gateway-prod-99.render.invalid/otlp grafana_alloy_loki_url: https://logs-prod-99.render.invalid/loki/api/v1/push - grafana_alloy_loki_username: "7654321" + grafana_alloy_loki_username: 7654321 # The two escape hatches an upgrading operator reaches for: vetoing a # collector must actually remove it (and its scoping block) rather than diff --git a/ansible/tests/alloy-config/validate.sh b/ansible/tests/alloy-config/validate.sh index 22bcfd6..a88a189 100755 --- a/ansible/tests/alloy-config/validate.sh +++ b/ansible/tests/alloy-config/validate.sh @@ -108,8 +108,10 @@ for config in "${configs[@]}"; do fi # The non-secret endpoints may be inventory literals; the token may NOT. This # catches a token pasted into a group_vars file and rendered into the 0644 - # config on the host. - if grep -qF 'glc_' "$config"; then + # config on the host. Patterns stay in lockstep with the preflight gate + # ("Refuse a Grafana Cloud token smuggled into inventory"): access-policy + # tokens, service-account tokens, and base64 JSON API keys. + if grep -qE 'glc_|glsa_|eyJ' "$config"; then fail "$name contains a literal Grafana Cloud token — it belongs only in the 0600 env file" fi echo "ok: $name" @@ -155,6 +157,7 @@ assert_has legacy.alloy 'sys.env("GC_PROM_REMOTE_WRITE_URL")' "the env fallback assert_has legacy.alloy 'sys.env("GC_PROM_USERNAME")' "the env fallback for the instance ID" assert_has legacy.alloy 'sys.env("GC_OTLP_ENDPOINT")' "the env fallback for OTLP" assert_lacks legacy.alloy 'GC_LOKI' "a Loki credential lookup while logs are disabled" +assert_lacks legacy.alloy '"job"' "an explicit job label, which did not exist before this change" # Inventory-driven connection settings: every non-secret sys.env lookup is gone, # the token's is not. @@ -164,6 +167,9 @@ done assert_has inventory.alloy 'https://prometheus-prod-99.render.invalid/api/prom/push' "the inventory remote-write URL" assert_has inventory.alloy 'https://logs-prod-99.render.invalid/loki/api/v1/push' "the inventory Loki URL" assert_has inventory.alloy 'sys.env("GC_API_TOKEN")' "the token env lookup, which must never move to inventory" +# Numeric instance IDs arrive from YAML as ints; Alloy needs quoted strings. +assert_has inventory.alloy 'username = "1234567"' "the Prometheus instance ID as a quoted string" +assert_has inventory.alloy 'username = "7654321"' "the Loki instance ID as a quoted string" # Escape hatches: the collector veto must remove the collector AND its scoping # block from the rendered lists (upstream would otherwise re-enable it), and a