diff --git a/AGENTS.md b/AGENTS.md index a3a30e0..0392523 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -50,7 +50,7 @@ economic claims — those trace to the deCDN ADRs. If something here states a pr ``` ansible/ # the deployment project (DevSec-hardened, lean roles) playbooks/ # site.yml (decdn node) - roles/ # baseline, decdn_node + roles/ # baseline, decdn_node, grafana_alloy inventory/ galaxy/ molecule/ # see ansible/README.md charts/ decdn-node/ # Helm chart for the node on Kubernetes (see its README.md) @@ -79,6 +79,24 @@ charts/ `molecule/schema` scenario checks the rendered key set against a committed inventory of upstream field names. Re-sync both when bumping the pinned decdn version. +- **`ansible/roles/grafana_alloy`** — opt-in Grafana Cloud observability for that node, + one mirrored flag (`decdn_grafana_cloud_enabled`, `false` by default) driving a + loopback-only Grafana Alloy agent: the node's `/metrics`, the **machine** itself + (Alloy's in-process `node_exporter`, curated collector set, `systemd` collector scoped + to the units that matter), **journald** → Grafana Cloud Loki, Alloy's own health, and + the daemon's OTLP spans. Host metrics and logs carry `job="integrations/node_exporter"` + so Grafana Cloud's prebuilt Linux Server dashboards work unmodified. Only the API token + is host-provisioned (`0600 /etc/grafana-alloy.env`, read via `sys.env` — Alloy has no + `--config.expand-env`); the non-secret endpoints/instance IDs are inventory variables + that fall back to their `GC_…` env key when empty, and preflight refuses a token in any + of them. Two hardening relaxations are conditional on the signals being on + (`ProtectHome=read-only` for correct filesystem metrics, `SupplementaryGroups= + systemd-journal adm` for journal access — without which collection is silently empty); + teardown is gated on the managed-by marker in the unit, so a foreign Alloy is never + touched. **`make lint-alloy` is the gate that matters** — the molecule stub exits 0 for + everything, so only the real pinned binary proves the rendered config loads. See + `ansible/roles/grafana_alloy/README.md`. + - **`charts/decdn-node/`** — the same node on Kubernetes: a one-replica StatefulSet (one release = one identity) on the upstream daemon-only image (`ghcr.io/decdn/decdn-node`; unpublished, so `image.tag`/`image.digest` is required), PVC data dir, a `prepare` init diff --git a/ansible/README.md b/ansible/README.md index ed8c99a..834e41b 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -187,26 +187,48 @@ primitives underneath it. All take `--dry-run`. See ### Grafana Cloud observability (opt-in) The play ships a third role, `grafana_alloy`, that installs a loopback-only -[Grafana Alloy](https://grafana.com/docs/alloy/) agent scraping the node's `/metrics` -and receiving its OTLP span exports (`127.0.0.1:4317`), shipping both to your Grafana -Cloud org — off by default, driven by ONE mirrored inventory flag: +[Grafana Alloy](https://grafana.com/docs/alloy/) agent shipping five signals to your +Grafana Cloud org — the node's `/metrics`, **the machine itself** (CPU/memory/disk/ +filesystem/network/load plus per-unit state, via Alloy's in-process `node_exporter`), +**journald**, the agent's own health, and the daemon's OTLP spans (`127.0.0.1:4317`) — +off by default, driven by ONE mirrored inventory flag: ```yaml # group_vars/host_vars — drives BOTH roles from one knob decdn_grafana_cloud_enabled: true ``` -Provision the credentials **on the target host** (they never transit this repo or the -control machine): +Machine metrics, journald and Alloy's own health come with it (each has a +`grafana_alloy_*_enabled` sub-knob, all defaulting to `true`). Host metrics and logs +carry `job="integrations/node_exporter"`, so Grafana Cloud's prebuilt **Linux Server** +dashboards and alert rules work unmodified. + +Only the API token is a secret. Put the non-secret connection settings in inventory +once for the fleet (`grafana_alloy_prom_url`, `_prom_username`, `_otlp_endpoint`, +`_loki_url`, `_loki_username` — note the Loki instance ID differs from the Prometheus +one), and provision just the token **on each target host** (it never transits this repo +or the control machine): ```bash umask 077 +printf 'GC_API_TOKEN=glc_…\n' > grafana-alloy.env sudo install -m 600 -o root -g root grafana-alloy.env /etc/grafana-alloy.env ``` -with `roles/grafana_alloy/files/grafana-alloy.env.example` as the template (remote-write -URL, OTLP endpoint, instance id, API token — all four keys required, https only). Setup, -rotation, cost/cardinality guardrails and rollback: see +Each of those variables is optional: left empty, the value is read from the matching +`GC_…` key in that file instead (`roles/grafana_alloy/files/grafana-alloy.env.example` +lists them all; URLs must be https). + +**Upgrading a host deployed before machine monitoring existed:** journald shipping is on +by default and needs a Loki endpoint + instance ID the old four-key file does not carry, +so preflight fails until you either set `grafana_alloy_loki_url` / `_loki_username` in +inventory, add `GC_LOKI_URL` / `GC_LOKI_USERNAME` to the env file, or set +`grafana_alloy_logs_enabled: false`. The daemon's own metrics also gain an explicit +`job="decdn-node"` label (previously the implicit `prometheus.scrape.decdn_node`) — +`grafana_alloy_node_job: ""` restores the old identity. + +Setup, rotation, cost/cardinality guardrails, the two systemd-hardening relaxations +machine monitoring requires, and rollback: see [`roles/grafana_alloy/README.md`](roles/grafana_alloy/README.md). The Helm chart path is separate and deliberately untouched. @@ -239,7 +261,9 @@ interleaving gets in the way of reading a failure. `grafana-cloud` runs against a *stub* Alloy that exits 0 for every subcommand, so it proves the role's plumbing but cannot prove the rendered `config.alloy` is loadable. That gap is closed by `make lint-alloy` (repo root; CI job `alloy-config`), which renders -the templates in several variable combinations and runs the **real**, digest-pinned Alloy +the templates in eight variable combinations — including one with every observability +sub-knob off, which must reproduce the pre-machine-monitoring pipeline exactly — and runs +the **real**, digest-pinned Alloy binary's `alloy validate` over them plus a check that every `ExecStart` flag actually exists in `alloy run --help`. Re-run it when bumping `grafana_alloy_version`. See [`tests/alloy-config/`](tests/alloy-config/). @@ -273,7 +297,7 @@ the RPC URL when it is not provisioned on the host instead). Highlights: | `decdn_rpc_url` + 3 contract addresses | `""` | **required** per node — `rpc_url` from a host-provisioned `0600 /etc/decdn/decdn.env` (preferred) *or* `host_vars//secret.yml`, addresses in `main.yml`; sourced from an ADR/deployment. | | `decdn_region` / `decdn_bind_port` / `decdn_rate_per_mb` | `""` / `4433` / `10` | node identity, QUIC port, USDC base units/MB. | | `decdn_env_checksum_file` / `decdn_env_overwrite_host_file` | `/etc/decdn/.decdn.env.sha256` / `false` | Provenance record for the secret env file (`0600 root`), and the opt-in that lets an inventory `decdn_rpc_url` overwrite a host-edited one. | -| `decdn_grafana_cloud_enabled` | `false` | ONE mirrored knob (identical default in both roles) wiring on Grafana Cloud observability: installs + configures `grafana_alloy` AND injects `otlp_endpoint` into `node.toml`. Credentials are provisioned per role README; label/cost guardrails there too. | +| `decdn_grafana_cloud_enabled` | `false` | ONE mirrored knob (identical default in both roles) wiring on Grafana Cloud observability: installs + configures `grafana_alloy` — node metrics, machine metrics, journald, agent health — AND injects `otlp_endpoint` into `node.toml`. Only the API token is provisioned per host; the rest are inventory variables. Label/cost guardrails in the role README. | --- diff --git a/ansible/molecule/grafana-cloud/converge.yml b/ansible/molecule/grafana-cloud/converge.yml index 7cf302e..6b8d0a6 100644 --- a/ansible/molecule/grafana-cloud/converge.yml +++ b/ansible/molecule/grafana-cloud/converge.yml @@ -39,6 +39,20 @@ # Sanity-coupling override: verify.yml greps the RENDERED config for this value, # proving template plumbing passes through rather than constants being checked. grafana_alloy_batch_send_size: 512 + + # Machine monitoring stays at its defaults (host metrics + agent self-metrics + # + journald all ON) so the positive path covers the shape a real deploy + # gets. The systemd collector and the journal itself collect nothing useful + # in a container — irrelevant here, since this scenario proves plumbing and + # `make lint-alloy` proves the config loads. + # + # The Loki URL comes from inventory while its username stays in the host env + # file (see prepare.yml): one converge exercises both credential paths. + grafana_alloy_loki_url: https://logs-prod-xx.molecule.invalid/loki/api/v1/push + # DELIBERATELY UNQUOTED: a numeric instance ID arrives from YAML as an int, + # so every gate and the template must coerce it instead of taking its length + # or emitting a bare number into a string field. + grafana_alloy_prom_username: 1234567 roles: - role: grafana_alloy tags: [observability] diff --git a/ansible/molecule/grafana-cloud/prepare.yml b/ansible/molecule/grafana-cloud/prepare.yml index 9baf8c5..6d99a81 100644 --- a/ansible/molecule/grafana-cloud/prepare.yml +++ b/ansible/molecule/grafana-cloud/prepare.yml @@ -40,6 +40,12 @@ # Values are throwaway; only the SHAPES matter to the role's greps (https://, # non-empty). Written directly at 0600 because preflight refuses anything else. + # + # Deliberately does NOT carry GC_LOKI_URL: converge supplies that one from + # inventory instead, so this fixture exercises BOTH halves of the credential + # split at once — the env-file path (token, prom, OTLP, Loki username) and + # the inventory path (Loki URL), including preflight's rule that a key is + # only required when its variable is empty. - name: Stage the Grafana Cloud credential fixture on the host ansible.builtin.copy: dest: /etc/grafana-alloy.env @@ -50,4 +56,5 @@ GC_PROM_REMOTE_WRITE_URL=https://prometheus-prod-xx.molecule.invalid/api/prom/push GC_OTLP_ENDPOINT=https://otlp-gateway-prod-xx.molecule.invalid/otlp GC_PROM_USERNAME=999888777 + GC_LOKI_USERNAME=999888778 GC_API_TOKEN=molecule-test-token diff --git a/ansible/molecule/grafana-cloud/verify.yml b/ansible/molecule/grafana-cloud/verify.yml index a26e0df..caae897 100644 --- a/ansible/molecule/grafana-cloud/verify.yml +++ b/ansible/molecule/grafana-cloud/verify.yml @@ -98,6 +98,70 @@ credential indirection) or carries syntax Alloy rejects. Got: {{ ga_config }} + # --- Machine monitoring --------------------------------------------------------- + - name: Assert the host, agent and journald pipelines rendered + ansible.builtin.assert: + that: + # Machine metrics: exporter -> target relabel -> scrape -> remote_write. + - >- + 'prometheus.exporter.unix "host"' in ga_config + - >- + 'discovery.relabel "host"' in ga_config + - >- + 'prometheus.scrape "host"' in ga_config + # The job label Grafana Cloud's Linux Server integration dashboards key + # off. Changing it silently unwires every prebuilt dashboard, so it is + # asserted rather than left to review. + - >- + '"integrations/node_exporter"' in ga_config + # Agent self-monitoring: tells "node went quiet" from "agent wedged". + - >- + 'prometheus.exporter.self "alloy"' in ga_config + - >- + '"integrations/alloy"' in ga_config + # journald -> Loki, with the field mapping that must live in the + # rules-only ruleset (those labels are stripped downstream). + - >- + 'loki.source.journal "host"' in ga_config + - >- + 'loki.relabel "journal_rules"' in ga_config + - >- + '__journal__systemd_unit' in ga_config + - >- + 'loki.write "cloud"' in ga_config + # The node's own series carry an explicit job rather than inheriting + # the scrape component's ID. + - >- + '"job" = "decdn-node"' in ga_config + fail_msg: >- + config.alloy is missing a machine-monitoring pipeline (host metrics, + agent self-metrics or journald). Got: {{ ga_config }} + + # The credential split: whatever inventory supplies becomes a literal, and its + # sys.env lookup MUST disappear; everything else keeps reading the env file. + - name: Assert the inventory/env credential split rendered both ways + ansible.builtin.assert: + that: + - >- + '"https://logs-prod-xx.molecule.invalid/loki/api/v1/push"' in ga_config + - >- + 'sys.env("GC_LOKI_URL")' not in ga_config + - >- + 'sys.env("GC_LOKI_USERNAME")' in ga_config + # The unquoted numeric instance ID in converge must reach the config as + # a QUOTED string (Alloy's basic_auth username is a string) and must + # have survived every preflight gate that takes its length. + - >- + 'username = "1234567"' in ga_config + - >- + 'sys.env("GC_PROM_USERNAME")' not in ga_config + fail_msg: >- + The inventory-vs-env credential split did not render as expected: + grafana_alloy_loki_url and the numeric grafana_alloy_prom_username were + set in inventory (so their sys.env lookups must be gone and the ID must + appear quoted) while GC_LOKI_USERNAME was left to the host env file (so + its lookup must remain). Got: {{ ga_config }} + # --- Secret hygiene ------------------------------------------------------------- - name: Read the credential fixture back ansible.builtin.slurp: @@ -115,6 +179,12 @@ 'GC_PROM_REMOTE_WRITE_URL=https://' in ga_env_body - >- 'GC_OTLP_ENDPOINT=https://' in ga_env_body + - >- + 'GC_LOKI_USERNAME=999888778' in ga_env_body + # Deliberately absent: converge supplies the Loki URL from inventory, + # which is what makes preflight's conditional requirement meaningful. + - >- + 'GC_LOKI_URL=' not in ga_env_body fail_msg: prepare.yml fixture diverged from what the greps require vars: ga_env_body: "{{ ga_env_b64.content | b64decode }}" @@ -195,6 +265,17 @@ # upstream default. - >- '--server.http.listen-addr=127.0.0.1:' in ga_unit + # Host metrics are on, so ProtectHome must be relaxed to read-only: + # with ProtectHome=true the filesystem collector measures an empty + # tmpfs for /home and /root, and a full disk there never alerts. The + # privilege boundary is unchanged (still no write access). + - >- + 'ProtectHome=read-only' in ga_unit + # Journald collection is on and this image has both groups, so the unit + # must grant them — without the membership loki.source.journal starts + # cleanly and ships nothing. + - >- + 'SupplementaryGroups=systemd-journal adm' in ga_unit fail_msg: hardened Alloy unit is missing an expected directive vars: ga_unit: "{{ ga_unit_b64.content | b64decode }}" @@ -204,6 +285,24 @@ - name: Gather service facts ansible.builtin.service_facts: + - name: Read the group database + ansible.builtin.getent: + database: group + + # The unit's SupplementaryGroups and the account's own membership must agree: + # systemd would refuse to start a unit naming a group that does not exist, and + # a missing membership makes journal collection silently empty. + - name: Assert the alloy account can read the journal + ansible.builtin.assert: + that: + - "'systemd-journal' in ansible_facts.getent_group" + - "'alloy' in ansible_facts.getent_group['systemd-journal'][2].split(',')" + - "'adm' in ansible_facts.getent_group" + - "'alloy' in ansible_facts.getent_group['adm'][2].split(',')" + fail_msg: >- + The alloy account is not a member of systemd-journal and adm, so + loki.source.journal would start without error and collect nothing. + - name: Assert both services are running ansible.builtin.assert: that: @@ -215,6 +314,56 @@ and ansible_facts.services['decdn-node.service'].state == 'running' fail_msg: "alloy or decdn-node is not running — journalctl -u alloy -e / -u decdn-node -e" +# Also mutates the converged host, so it runs after every assertion above and +# before the teardown play below. +- name: Verify turning logs off takes the journal grant back + hosts: all + become: true + tasks: + # Dropping SupplementaryGroups from the unit is not enough on its own: + # systemd then falls back to the ACCOUNT's supplementary groups, so a + # membership left over from the logs-enabled run would silently keep journal + # read access alive after the operator turned logs off. + - name: Re-run the role with only the logs signal disabled + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_install_method: manual + grafana_alloy_manual_bin_src: >- + {{ lookup('ansible.builtin.env', 'MOLECULE_SCENARIO_DIRECTORY') }}/files/alloy-stub + grafana_alloy_deployment_environment: production + grafana_alloy_region: US + grafana_alloy_logs_enabled: false + + - name: Read the group database back + ansible.builtin.getent: + database: group + + - name: Read the unit and config back + ansible.builtin.slurp: + src: "{{ item }}" + register: ga_logs_off + loop: + - /etc/systemd/system/alloy.service + - /etc/alloy/config.alloy + + - name: Assert the journal grant is gone from both the unit and the account + ansible.builtin.assert: + that: + - "'alloy' not in (ansible_facts.getent_group['systemd-journal'][2] | default('')).split(',')" + - "'alloy' not in (ansible_facts.getent_group['adm'][2] | default('')).split(',')" + - "'SupplementaryGroups' not in ga_unit_off" + - "'loki.' not in ga_config_off" + fail_msg: >- + Turning grafana_alloy_logs_enabled off left journal access behind: + SupplementaryGroups in the unit or a stale systemd-journal/adm + membership on the alloy account would keep the agent able to read the + journal after the signal it was granted for was switched off. + vars: + ga_unit_off: "{{ ga_logs_off.results[0].content | b64decode }}" + ga_config_off: "{{ ga_logs_off.results[1].content | b64decode }}" + # Runs LAST, after the idempotence phase and the assertions above, because it # deliberately mutates the converged host: it exercises the disable path, whose # whole job is to remove things. diff --git a/ansible/molecule/validation/converge.yml b/ansible/molecule/validation/converge.yml index 265fe5e..b615e31 100644 --- a/ansible/molecule/validation/converge.yml +++ b/ansible/molecule/validation/converge.yml @@ -927,6 +927,152 @@ decdn_rejected: "{{ decdn_rejected + ['managed-paths'] }}" when: ansible_failed_task.name is match('^Require the rendered config to live inside') + - name: "Case ga-collector — unknown node_exporter collector name" + block: + - name: Run grafana_alloy with a bogus host collector + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + # An unknown collector is NOT a startup crash — Alloy resolves the + # set by map lookup and silently ignores what it does not know, so a + # typo ships fewer metrics forever. Only the committed name + # inventory in the role's vars/ can catch it, and only here. + grafana_alloy_host_collectors: [cpu, diskstat] + rescue: + - name: Record ga-collector rejection (only if the collector validator failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['host-collectors'] }}" + when: ansible_failed_task.name is match('^Validate the node_exporter collector names') + + - name: "Case ga-no-collectors — host metrics on with every collector list empty" + block: + - name: Run grafana_alloy with an empty collector set + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_host_collectors: [] + grafana_alloy_host_extra_collectors: [] + rescue: + - name: Record ga-no-collectors rejection (only if the non-empty gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['host-collectors'] }}" + when: ansible_failed_task.name is match('^Require a non-empty effective host collector set') + + # An empty set_collectors does NOT mean "measure nothing" to Alloy: it reads + # as unset and re-enables the full ~45-collector default set. Vetoing every + # collector must therefore be refused, not rendered. + - name: "Case ga-disable-all — every collector vetoed by the disable list" + block: + - name: Run grafana_alloy with the whole collector set disabled + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_host_collectors: [cpu] + grafana_alloy_host_extra_collectors: [systemd] + grafana_alloy_host_disable_collectors: [cpu, systemd] + rescue: + - name: Record ga-disable-all rejection (only if the non-empty gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['host-collectors'] }}" + when: ansible_failed_task.name is match('^Require a non-empty effective host collector set') + + - name: "Case ga-loki-http — cleartext Loki endpoint from inventory" + block: + - name: Run grafana_alloy with a non-https Loki URL + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_loki_url: http://logs-prod-xx.molecule.invalid/loki/api/v1/push + rescue: + - name: Record ga-loki-http rejection (only if the endpoint validator failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['endpoint-scheme'] }}" + when: ansible_failed_task.name is match('^Validate any inventory-provided Grafana Cloud endpoints') + + - name: "Case ga-inventory-token — access token smuggled into inventory" + block: + # config.alloy is 0644 on the host, so a token reaching an inventory + # variable would be rendered world-readable — the one thing the whole + # host-provisioned credential design exists to prevent. + - name: Run grafana_alloy with a token in a non-secret variable + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_prom_username: glc_eyJvIjoiMTIzNDU2NyJ9 + rescue: + - name: Record ga-inventory-token rejection (only if the token gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['inventory-token'] }}" + when: ansible_failed_task.name is match('^Refuse a Grafana Cloud token smuggled into inventory') + + - name: "Case ga-inventory-sa-token — service-account token smuggled into inventory" + block: + # glsa_ tokens carry no '@' and pass the instance-ID charset, so the + # gate must key off the prefix rather than shape alone. + - name: Run grafana_alloy with a service-account token in a non-secret variable + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_loki_username: glsa_AbCdEf0123456789_0a1b2c3d + rescue: + - name: Record ga-inventory-sa-token rejection (only if the token gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['inventory-token'] }}" + when: ansible_failed_task.name is match('^Refuse a Grafana Cloud token smuggled into inventory') + + - name: "Case ga-max-age — malformed journal catch-up window" + block: + - name: Run grafana_alloy with a bad max_age suffix + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_logs_max_age: 12hours + rescue: + - name: Record ga-max-age rejection (only if the knob validator failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['pipeline-knobs'] }}" + when: ansible_failed_task.name is match('^Validate the pipeline knobs') + + - name: "Case ga-loki-key — logs enabled with neither Loki key nor variable" + block: + # The Loki pair is required ONLY when logs are on and inventory supplies + # nothing: this proves the conditional gate still fires when it should. + - name: Stage a credential file with no Loki keys + ansible.builtin.copy: + dest: /etc/grafana-alloy.env + owner: root + group: root + mode: "0600" + content: | + GC_PROM_REMOTE_WRITE_URL=https://prometheus-prod-xx.molecule.invalid/api/prom/push + GC_OTLP_ENDPOINT=https://otlp-gateway-prod-xx.molecule.invalid/otlp + GC_PROM_USERNAME=999888777 + GC_API_TOKEN=molecule-test-token + + - name: Run grafana_alloy with logs enabled and no Loki credentials + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_logs_enabled: true + rescue: + - name: Record ga-loki-key rejection (only if the grep report gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['secret-content'] }}" + when: ansible_failed_task.name is match('^Report which credential key') + always: + - name: Remove the Loki-less credential fixture + ansible.builtin.file: + path: /etc/grafana-alloy.env + state: absent + - name: Confirm every bad value was rejected by the role's own validation ansible.builtin.assert: that: @@ -947,6 +1093,9 @@ "otlp-newline", "env-control-rpc", "env-control-extra", "env-gate", "env-extra", "env-content", "config-gate", "secret-file", "secret-file", "secret-path-nested", "secret-content", - "secret-content", "pipeline-knobs", "alloy-root-user", "alloy-root-group", + "secret-content", "secret-content", "pipeline-knobs", "pipeline-knobs", + "alloy-root-user", "alloy-root-group", "alloy-root-user-alias", "alloy-root-group-alias", + "host-collectors", "host-collectors", "host-collectors", + "endpoint-scheme", "inventory-token", "inventory-token", "managed-paths", "managed-paths", "managed-paths"] diff --git a/ansible/roles/grafana_alloy/README.md b/ansible/roles/grafana_alloy/README.md index 37858de..4878de5 100644 --- a/ansible/roles/grafana_alloy/README.md +++ b/ansible/roles/grafana_alloy/README.md @@ -1,17 +1,31 @@ # grafana_alloy -Opt-in Grafana Cloud Application Observability for an Ansible-deployed deCDN -node ([decdn/devops#55](https://github.com/decdn/devops/issues/55)): a dedicated, +Opt-in Grafana Cloud observability for an Ansible-deployed deCDN node +([decdn/devops#55](https://github.com/decdn/devops/issues/55)): a dedicated, loopback-only [Grafana Alloy](https://grafana.com/docs/alloy/) agent that - scrapes the node's `/metrics` (Prometheus exposition on loopback port 9090) and pushes it to your Grafana Cloud Prometheus, stamped with low-cardinality - identity labels, and + identity labels, +- scrapes **the machine itself** — CPU, memory, disk, filesystem, network, load, + PSI saturation and the state of the units that matter — through Alloy's + in-process `node_exporter`, labelled so Grafana Cloud's prebuilt **Linux + Server** dashboards and alert rules work with no dashboard wiring, +- ships **journald** (host and services) to your Grafana Cloud Loki with the same + `job`/`instance` pair, so logs and metrics correlate in those dashboards, +- reports **its own** health (`prometheus.exporter.self`), which is what + distinguishes "the node went quiet" from "the agent reporting on the node is + wedged", and - receives the daemon's OTLP span exports (`otlp_endpoint`) on gRPC `127.0.0.1:4317` and HTTP `127.0.0.1:4318`, probabilistically samples traces (default keep-ratio 0.25), batches them, and exports via OTLP/HTTP to your Grafana Cloud org. -The Helm-chart path is separate and deliberately untouched by this role. +Everything except the node's public QUIC port stays on loopback: the exporters +run *inside* the agent, so machine monitoring adds no listener and no firewall +hole. + +The Helm-chart path is separate and deliberately untouched by this role — on +Kubernetes, host-level telemetry is the cluster's job. ## Opt in @@ -23,24 +37,67 @@ host level only: decdn_grafana_cloud_enabled: true ``` -Then provision credentials **on the target host** (they never transit the control -machine): +Machine metrics, agent self-metrics and journald come with it (each has its own +`grafana_alloy_*_enabled` sub-knob, all defaulting to `true`). + +### Credentials: one secret per host, the rest in inventory + +Only the **API token** is a secret. The endpoint URLs and the two numeric +instance IDs are not, so they belong in inventory — written once for the fleet +rather than typed on every host: + +```yaml +# group_vars/.yml — non-secret, safe to commit +grafana_alloy_prom_url: https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom/push +grafana_alloy_prom_username: "1234567" # Prometheus instance ID +grafana_alloy_otlp_endpoint: https://otlp-gateway-prod-us-east-0.grafana.net/otlp +grafana_alloy_loki_url: https://logs-prod-006.grafana.net/loki/api/v1/push +grafana_alloy_loki_username: "7654321" # Loki instance ID — a DIFFERENT number +``` + +Then the only thing to provision **on the target host** is the token (it never +transits the control machine): ```bash umask 077 +printf 'GC_API_TOKEN=glc_…\n' > grafana-alloy.env sudo install -m 600 -o root -g root grafana-alloy.env /etc/grafana-alloy.env ``` -using `roles/grafana_alloy/files/grafana-alloy.env.example` as the template. The -credential path is intentionally restricted to a **direct child of `/etc`**. A +Every one of those variables is optional: left empty, its value is read from the +matching `GC_…` key in the env file exactly as before, and preflight requires the +key only then. Any mix of the two halves is valid. +`roles/grafana_alloy/files/grafana-alloy.env.example` shows the full key set. + +> **Upgrading an existing host.** Logs are on by default, and Loki needs an +> endpoint + instance ID that the pre-existing four-key env file does not have. +> Preflight fails loudly, naming both the missing key and the inventory variable +> that satisfies it. Pick one: set `grafana_alloy_loki_url` / +> `grafana_alloy_loki_username` in inventory (recommended — nothing to do per +> host), append `GC_LOKI_URL` / `GC_LOKI_USERNAME` to `/etc/grafana-alloy.env`, +> or set `grafana_alloy_logs_enabled: false` to keep shipping exactly what you +> shipped before. The other four keys are untouched by this change. +> +> Host metrics need no new credential (they ride the existing Prometheus +> remote-write), but note that the daemon's own series now carry +> `job="decdn-node"` instead of the implicit `job="prometheus.scrape.decdn_node"`. +> Set `grafana_alloy_node_job: ""` to keep the old value if dashboards or alert +> rules already hard-code it. + +`GC_API_TOKEN` has **no** inventory variable by design: the rendered +`/etc/alloy/config.alloy` is world-readable, and preflight rejects a value that +looks like a token (or a URL with embedded credentials) in any of the five +variables above. + +The credential path is intentionally restricted to a **direct child of `/etc`**. A nested override (including the old `/etc/decdn/grafana-alloy.env`) is rejected: -write access to any ancestor is enough to replace a root-owned `0600` file. -four required keys are `GC_PROM_REMOTE_WRITE_URL`, `GC_OTLP_ENDPOINT`, -`GC_PROM_USERNAME` and `GC_API_TOKEN`; both URLs must be `https://`. -`config.alloy` reads each of them at load time with `sys.env("GC_…")` — Alloy has -**no** `--config.expand-env` flag, so a `${GC_…}` placeholder would be a literal -string, not an expansion. The file on disk stays free of secret literals, and the -role only ever greps shapes on the host rather than reading values back. +write access to any ancestor is enough to replace a root-owned `0600` file. Both +cloud URLs must be `https://`, whichever half they come from. +`config.alloy` reads each env-provided value at load time with `sys.env("GC_…")` +— Alloy has **no** `--config.expand-env` flag, so a `${GC_…}` placeholder would be +a literal string, not an expansion. The file on disk stays free of secret +literals, and the role only ever greps shapes on the host rather than reading +values back. With the flag off, the role removes the unit, rendered config, state directory and service account **it installed** — identified by the managed-by marker in @@ -49,6 +106,64 @@ upstream apt repo, another role, by hand) carries no marker and is left strictly alone, which matters because the flag is `false` by default on every host. Binary/package removal is manual. +## What lands in Grafana Cloud + +| Signal | Job label | Where it shows up | +| --- | --- | --- | +| Node `/metrics` | `decdn-node` | your own dashboards/queries | +| Machine metrics | `integrations/node_exporter` | Grafana Cloud's **Linux Server** integration dashboards + alerts, unmodified | +| journald | `integrations/node_exporter` | the same integration's logs dashboards (correlated by `instance`) | +| Alloy self-metrics | `integrations/alloy` | agent health | +| Traces | — | Application Observability | + +Changing `grafana_alloy_host_job` / `grafana_alloy_logs_job` away from +`integrations/node_exporter` silently un-wires the prebuilt dashboards; that is +why they are knobs with a warning rather than hard-coded literals. + +## Machine monitoring: what it costs and how to bound it + +Grafana Cloud bills on active series and log volume, so the defaults are +deliberately lean and every lever is a variable: + +- `grafana_alloy_host_collectors` is a **hand-picked replacement** for + node_exporter's ~45-collector default set (Alloy's `set_collectors`: anything + not listed is off). Widen with `grafana_alloy_host_extra_collectors`, veto with + `grafana_alloy_host_disable_collectors` — the veto is applied to both lists at + render time, because upstream applies its own `disable_collectors` *before* + `enable_collectors` and would otherwise leave a vetoed collector running. Names + are checked against a committed inventory (`vars/main.yml`): an unknown + collector is not a startup error, it is silently ignored, so a typo would ship + fewer metrics forever with nothing to show for it. +- `systemd` is the one non-default collector enabled, and it is scoped by + `grafana_alloy_host_systemd_unit_include` to the units that matter rather than + every unit on the box. It answers "is `decdn-node` actually running" — which + the daemon's own `/metrics` cannot, because a dead daemon serves nothing. +- Host metrics scrape at `60s`, not the node's `30s`. +- The `filesystem` / `netdev` / `disk` exclude regexes drop virtual filesystems + and container/virtual interfaces. +- journald drops `debug` priority and a noisy-unit list + (`grafana_alloy_logs_drop_priority_regex` / `_drop_unit_regex`, `""` disables + either), and `grafana_alloy_logs_max_age` bounds the catch-up burst after an + outage — without it a restart can replay days of journal in one go. + +## Hardening: the two relaxations machine monitoring requires + +The unit is stricter than Grafana's own shipped unit (which has no `Protect*` +directives at all), but two directives are incompatible with host telemetry and +are relaxed **only** when the matching signal is enabled: + +| Directive | Condition | Why | +| --- | --- | --- | +| `ProtectHome=read-only` (instead of `true`) | host metrics on | `ProtectHome=true` replaces `/home`, `/root` and `/run/user` with empty tmpfs mounts inside the unit's namespace, so the filesystem collector measures the overlay instead of the disk — a full `/home` would never raise an alert. `read-only` keeps the privilege boundary: the agent still cannot write there. | +| `SupplementaryGroups=systemd-journal adm` | logs on, and the groups exist | Reading the journal as a non-root user requires **both**. Without them `loki.source.journal` starts without error and collects **nothing** — the worst failure mode there is. The role joins the `alloy` account to the same groups, and lists only groups that exist, because systemd refuses to start a unit naming one that does not. | + +Everything else stands: `NoNewPrivileges`, `ProtectSystem=strict`, +`SystemCallFilter=@system-service`, `PrivateDevices`, `RestrictNamespaces`, the +dedicated unprivileged account. The default collector set deliberately excludes +the collectors that would need capabilities or a widened syscall filter (`perf`, +`processes`, the netlink-based ones). `AF_UNIX` — already allowed for journald — +is also what the `systemd` collector uses to reach the D-Bus system bus. + ## Variables All prefixed `grafana_alloy_*` (see `defaults/main.yml` for comments and pinned @@ -62,12 +177,22 @@ upstream version/sha256). Highlights: | `grafana_alloy_trace_sampling_ratio` | `0.25` | Trace keep-ratio, validated to `[0,1]` | | `grafana_alloy_otlp_grpc_port` | `4317` | Hard-coupled to the literal emitted into node.toml | | `grafana_alloy_region` | `decdn_region` | Required identity attribute; derived from the node region | +| `grafana_alloy_host_metrics_enabled` | `true` | Machine metrics (Alloy's in-process `node_exporter`) | +| `grafana_alloy_host_collectors` | 13 collectors | Replacement set; `_host_extra_collectors` adds, `_host_disable_collectors` vetoes | +| `grafana_alloy_host_scrape_interval` | `60s` | Host metrics only; the node keeps `30s` | +| `grafana_alloy_host_systemd_unit_include` | `(decdn-node\|alloy\|ssh\|sshd)\.service` | Scope of the per-unit `systemd` collector | +| `grafana_alloy_logs_enabled` | `true` | journald → Grafana Cloud Loki | +| `grafana_alloy_logs_max_age` | `12h` | Bounds the catch-up burst after an outage | +| `grafana_alloy_self_metrics_enabled` | `true` | Alloy's own health | +| `grafana_alloy_prom_url` / `_prom_username` / `_otlp_endpoint` / `_loki_url` / `_loki_username` | `""` | Non-secret connection settings; empty ⇒ read the matching `GC_…` env key | +| `grafana_alloy_node_job` / `_host_job` / `_self_job` / `_logs_job` | see table above | Job labels; the `integrations/…` ones are what Grafana Cloud's dashboards match | Label variables (`service_name`, `service_namespace`, `instance_id`, `deployment_environment`, `region`) ship on EVERY series/span/log line — keep them low-cardinality; unique label sets are what your Grafana Cloud bill scales -with. Guardrails: one scrape interval per fleet (`30s`), sampling above zero for -spans, and no per-request/per-hash label values. +with. Guardrails: one scrape interval per fleet (`30s` node / `60s` host), +sampling above zero for spans, the curated collector set, the journald drops, and +no per-request/per-hash label values. ## Couplings & guards @@ -86,18 +211,27 @@ If you move either default, update BOTH sides and the molecule guard in Three layers, because the first one cannot prove correctness on its own: 1. **`grafana-cloud` molecule scenario** — the role end-to-end against a *stub* - binary (plumbing, rendered content, hardened unit, teardown scope), plus - negative preflight cases in `validation` and disabled-path assertions in - `default`. The stub exits 0 for every subcommand, so a green run says nothing - about whether Alloy can load the config. + binary (plumbing, rendered content, hardened unit, group membership, teardown + scope), plus negative preflight cases in `validation` and disabled-path + assertions in `default`. The stub exits 0 for every subcommand, so a green run + says nothing about whether Alloy can load the config. It also runs in a + container with no D-Bus and no real journal, so the `systemd` collector and + journald collect nothing there — expected, and not what this layer proves. 2. **`make lint-alloy`** (CI job `alloy-config`) — renders these templates in - several variable combinations and feeds them to the REAL pinned Alloy binary: - `alloy validate` (component graph, not just syntax) and a check that every - `ExecStart` flag exists in `alloy run --help`. See + eight variable combinations (defaults, minimal identity, fully overridden, + host-metrics-only, logs-only, every sub-knob off, inventory-supplied + endpoints, vetoed collector + blanked node job) and feeds them to the REAL pinned Alloy binary: `alloy validate` + (component graph, not just syntax), a check that every `ExecStart` flag exists + in `alloy run --help`, and greps proving each Jinja branch actually switched — + including that "every sub-knob off" reproduces the pre-machine-monitoring + pipeline and that no token is ever rendered as a literal. See `ansible/tests/alloy-config/`. 3. **Deploy time** — the role runs `alloy validate` against the just-rendered file with the installed binary before any restart. +Only a real host proves *collection*: check `journalctl -u alloy -e` and Alloy's +loopback UI (`127.0.0.1:12345`) after the first deploy. + Re-run `make lint-alloy` whenever `grafana_alloy_version` is bumped: the harness downloads the same digest-pinned `.deb` the role installs, so a version bumped without its sha256 fails there rather than on a host. diff --git a/ansible/roles/grafana_alloy/defaults/main.yml b/ansible/roles/grafana_alloy/defaults/main.yml index 0923647..ae6db09 100644 --- a/ansible/roles/grafana_alloy/defaults/main.yml +++ b/ansible/roles/grafana_alloy/defaults/main.yml @@ -51,8 +51,132 @@ grafana_alloy_otlp_http_port: 4318 # gRPC value staying put (see node.t # --server.grpc.listen-addr flag, and passing one is an immediate crash-loop. grafana_alloy_server_http_addr: 127.0.0.1:12345 +# --- Grafana Cloud endpoints (NON-SECRET; optional inventory overrides) ------- +# Only the API TOKEN is a secret. The endpoint URLs and the two numeric instance +# IDs are not, so they may be set here (group_vars/host_vars) instead of being +# hand-typed into the root-owned env file on every host — which leaves +# GC_API_TOKEN as the single value an operator must provision per host. +# +# Each of these is optional and falls back, when left EMPTY, to the matching +# sys.env("GC_…") lookup the role has always used, so an existing deployment that +# sets none of them renders byte-identical configuration. Setting one makes its +# env key unnecessary, and preflight stops requiring that key. +# +# NEVER put the token here: config.alloy is world-readable 0644 on the host, and +# inventory is committed. GC_API_TOKEN has no variable by design. +grafana_alloy_prom_url: "" # else sys.env("GC_PROM_REMOTE_WRITE_URL") +grafana_alloy_prom_username: "" # else sys.env("GC_PROM_USERNAME") +grafana_alloy_otlp_endpoint: "" # else sys.env("GC_OTLP_ENDPOINT") +# Grafana Cloud's Loki instance ID is a DIFFERENT number from the Prometheus one +# (the same access-policy token authenticates both). +grafana_alloy_loki_url: "" # else sys.env("GC_LOKI_URL") +grafana_alloy_loki_username: "" # else sys.env("GC_LOKI_USERNAME") + # --- Metrics scrape (node /metrics -> Grafana Cloud Prometheus) --------------- grafana_alloy_scrape_interval: 30s +# Explicit `job` label for the daemon's own series. Without it Prometheus derives +# `job` from the scrape component's ID ("prometheus.scrape.decdn_node"), which is +# an implementation detail no dashboard should hard-code. +# UPGRADE NOTE: setting this changes the identity of already-existing series. +# Hosts deployed before this label existed report job="prometheus.scrape.decdn_node"; +# set grafana_alloy_node_job: "" to keep that value instead of re-pointing +# dashboards, recording rules and alerts. +grafana_alloy_node_job: decdn-node + +# --- Host (machine) metrics --------------------------------------------------- +# Alloy's built-in node_exporter. Runs IN-PROCESS: no extra listener, no extra +# port, no firewall hole (AGENTS.md hard rule 2 is unaffected). Stamped with the +# job label Grafana Cloud's prebuilt "Linux Server / node_exporter" integration +# dashboards and alert rules key off, so they work with zero dashboard wiring. +grafana_alloy_host_metrics_enabled: true +# Host metrics do not need the node's resolution; 60s halves the series cost. +grafana_alloy_host_scrape_interval: 60s +# Changing this away from integrations/node_exporter silently un-wires the +# Grafana Cloud integration dashboards. "integrations/unix" is also accepted. +grafana_alloy_host_job: integrations/node_exporter +# Hand-picked replacement for node_exporter's ~45-collector default set (Alloy's +# `set_collectors`: anything absent here is off). The remote bill scales with +# series count, so this covers the questions an operator actually asks — CPU, +# memory, disk, filesystem, network, load, saturation — and nothing else. +grafana_alloy_host_collectors: + - cpu + - diskstats + - filefd + - filesystem + - loadavg + - meminfo + - netdev + - os + - pressure # PSI saturation; needs kernel 4.20+/CONFIG_PSI, absent = no series + - stat + - time + - uname + - vmstat +# Added ON TOP of the list above (Alloy's `enable_collectors`). `systemd` is +# registered defaultDisabled upstream and must be named here; it answers "is +# decdn-node actually running", which the daemon's own /metrics cannot when the +# daemon is down. It talks to the D-Bus system bus, so it collects nothing in a +# container without one (molecule) — that is expected, not a failure. +grafana_alloy_host_extra_collectors: + - systemd +# Subtracted from BOTH lists above at render time; an escape hatch for a +# collector that misbehaves on a specific fleet. (Doing it in the template is +# deliberate: upstream applies disable_collectors BEFORE enable_collectors, so a +# name in both would end up enabled — the opposite of a veto.) +grafana_alloy_host_disable_collectors: [] +# Alloy's own exporter-internal metrics (scrape durations per collector). Off: +# grafana_alloy_self_metrics_enabled covers agent health more cheaply. +grafana_alloy_host_include_exporter_metrics: false +# Filter regexes. Single-quoted YAML so backslashes stay literal; each is +# JSON-encoded into the rendered config, so quotes/backslashes cannot break out +# of the string. Defaults mirror upstream's, plus tmpfs (Grafana Cloud's own +# snippet adds it) and the usual container/virtual interface noise. +# NOTE the escaped line breaks: YAML double quotes join these back into ONE regex +# with no whitespace. Single quotes cannot be used here (the line is too long for +# yamllint) and a folded scalar would inject spaces into the pattern. +grafana_alloy_host_fs_types_exclude: "^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|\ + fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|\ + selinuxfs|squashfs|sysfs|tmpfs|tracefs)$" +grafana_alloy_host_mount_points_exclude: '^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/)' +grafana_alloy_host_mount_timeout: 5s +grafana_alloy_host_netdev_device_exclude: '^(veth.*|cali.*|docker.*|br-.*|lo|[a-f0-9]{15})$' +grafana_alloy_host_disk_device_exclude: '^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$' +# Per-unit state, deliberately scoped: one series set per unit on the box is a +# real cost, and only these units matter. Units must match include AND NOT match +# exclude. +grafana_alloy_host_systemd_unit_include: '(decdn-node|alloy|ssh|sshd)\.service' +grafana_alloy_host_systemd_unit_exclude: '.+\.(automount|device|mount|scope|slice)' + +# --- Agent self-monitoring ---------------------------------------------------- +# Alloy's own health (component evaluation, WAL, remote-write queue/failures). +# Cheap, and the only way to tell "the node is silent" apart from "the agent that +# reports on the node is wedged". +grafana_alloy_self_metrics_enabled: true +grafana_alloy_self_job: integrations/alloy + +# --- Host + service logs (journald -> Grafana Cloud Loki) --------------------- +# Reading the journal as a non-root user needs a group journald grants read to: +# systemd-journal owns the journal files, adm gets an ACL on /var/log/journal. +# With neither, the component starts cleanly and collects NOTHING, so the role +# joins the alloy user to both where they exist, lists them under +# SupplementaryGroups, and warns when either is missing. +grafana_alloy_logs_enabled: true +# Grafana Cloud's Linux Server integration uses the SAME job value for logs as +# for host metrics; its logs dashboards key off job + instance. +grafana_alloy_logs_job: integrations/node_exporter +# Bounds the catch-up burst after an outage or a fresh install: without it a +# restart can replay days of journal into your bill in one go. +grafana_alloy_logs_max_age: 12h +# "" => Alloy reads both /var/log/journal and /run/log/journal (persistent and +# volatile). Set a single path only if you know the host keeps just one. +grafana_alloy_logs_journal_path: "" +# Cost guardrails, applied while the __journal* fields still exist. Both are +# anchored regexes matched against the whole value; "" disables that rule. +# Priority keywords are emerg/alert/crit/error/warning/notice/info/debug +# (journald's numeric priority 3 surfaces as "error", NOT "err"). +grafana_alloy_logs_drop_priority_regex: 'debug' +grafana_alloy_logs_drop_unit_regex: + '(session-\d+\.scope|user@\d+\.service|systemd-timesyncd\.service|cron\.service|systemd-logind\.service)' # --- Resource identity labels (low-cardinality only!) -------------------------- # These become metric labels and OTLP resource attributes shipped to your Grafana diff --git a/ansible/roles/grafana_alloy/files/grafana-alloy.env.example b/ansible/roles/grafana_alloy/files/grafana-alloy.env.example index 7bea71f..cf20bc8 100644 --- a/ansible/roles/grafana_alloy/files/grafana-alloy.env.example +++ b/ansible/roles/grafana_alloy/files/grafana-alloy.env.example @@ -1,18 +1,45 @@ # Template for /etc/grafana-alloy.env (root-owned 0600). # -# Provision it ON THE TARGET HOST — these values are your Grafana Cloud -# credentials and must never be committed or carried through inventory: +# Provision it ON THE TARGET HOST — this file holds your Grafana Cloud API token +# and must never be committed or carried through inventory: # # umask 077 # sudo install -m 600 -o root -g root grafana-alloy.env /etc/grafana-alloy.env # -# Values come from your Grafana Cloud org: observe -> Application Observability -> -# "Instructions"/details page shows the exact remote-write URL, OTLP endpoint, -# instance id ("user") and API token. The role greps shapes only; it never reads -# values back onto the control machine. +# MINIMUM CONTENT for a NEW host: the token, and nothing else. Every other value +# below is NON-secret and can be set once in inventory (group_vars) instead of +# being re-typed on every host — see the grafana_alloy_* variables named in each +# comment. A key is only required here when its inventory variable is empty. +# +# UPGRADING an existing host: journald shipping is on by default, so GC_LOKI_URL +# and GC_LOKI_USERNAME become required unless you set grafana_alloy_loki_url / +# grafana_alloy_loki_username in inventory, or turn logs off with +# grafana_alloy_logs_enabled: false. Preflight names the missing key and the +# variable that would satisfy it instead. +# +# Values come from your Grafana Cloud org: Home -> Stacks -> your stack, then the +# "Details/Send Metrics|Logs|Traces" pages show each endpoint plus its numeric +# instance ID. The role greps shapes only; it never reads values back onto the +# control machine. -GC_PROM_REMOTE_WRITE_URL=https://prometheus-prod-XX.grafana.net/api/prom/push -GC_OTLP_ENDPOINT=https://otlp-gateway-prod-XX.grafana.net/otlp -GC_PROM_USERNAME=1234567 -# glc_... application token with write scope for this stack +# REQUIRED, always. An access-policy token with metrics:write + logs:write + +# traces:write scope for this stack. This is the one value with no inventory +# variable, deliberately: the rendered /etc/alloy/config.alloy is world-readable. GC_API_TOKEN=glc_example_token_replace_me + +# --- Optional: only needed if the matching inventory variable is left empty ---- + +# grafana_alloy_prom_url +#GC_PROM_REMOTE_WRITE_URL=https://prometheus-prod-XX-REGION.grafana.net/api/prom/push +# grafana_alloy_prom_username — the numeric Prometheus instance ID +#GC_PROM_USERNAME=1234567 + +# grafana_alloy_otlp_endpoint +#GC_OTLP_ENDPOINT=https://otlp-gateway-prod-XX.grafana.net/otlp + +# Logs (grafana_alloy_logs_enabled, on by default). NOTE the Loki instance ID is +# a DIFFERENT number from the Prometheus one above; the token is shared. +# grafana_alloy_loki_url +#GC_LOKI_URL=https://logs-prod-XX.grafana.net/loki/api/v1/push +# grafana_alloy_loki_username — the numeric Loki instance ID +#GC_LOKI_USERNAME=7654321 diff --git a/ansible/roles/grafana_alloy/tasks/main.yml b/ansible/roles/grafana_alloy/tasks/main.yml index a350962..c9b1a47 100644 --- a/ansible/roles/grafana_alloy/tasks/main.yml +++ b/ansible/roles/grafana_alloy/tasks/main.yml @@ -1,7 +1,8 @@ --- # Opt-in Grafana Cloud Application Observability for an Ansible-deployed deCDN -# node: a loopback-only Grafana Alloy agent that scrapes the node's /metrics and -# receives the daemon's OTLP traces, then ships both to Grafana Cloud. +# node: a loopback-only Grafana Alloy agent that scrapes the node's /metrics, the +# MACHINE itself (Alloy's in-process node_exporter) and Alloy's own health, +# receives the daemon's OTLP traces, and ships journald to Grafana Cloud Loki. # # Gated by decdn_grafana_cloud_enabled (mirrored in roles/decdn_node/defaults — # ONE inventory knob drives both roles). When false this role does nothing except @@ -144,6 +145,50 @@ - name: Fail loud on preflight misconfiguration ansible.builtin.import_tasks: preflight.yml + # --- Journal access --------------------------------------------------------- + # A non-root journal reader needs a group journald grants read to: + # systemd-journal owns the journal files, adm gets an ACL on /var/log/journal. + # With neither, loki.source.journal starts WITHOUT error and collects nothing, + # so this is a silent-failure guard, not a convenience. Detected rather than + # assumed: a container or a non-Debian host may ship neither, and systemd + # refuses to start a unit whose SupplementaryGroups names a group that does + # not exist. A partial set still works but is reported, since it means one of + # the two access paths is unavailable. + - name: Detect which journal reader groups exist on this host + ansible.builtin.command: + cmd: getent group {{ item | quote }} + loop: "{{ ['systemd-journal', 'adm'] if grafana_alloy_logs_enabled | bool else [] }}" + changed_when: false + failed_when: false + check_mode: false # read-only lookup; must also run under make check + register: _ga_journal_group_lookup + + - name: Resolve the journal reader group list + ansible.builtin.set_fact: + _ga_journal_groups: >- + {{ _ga_journal_group_lookup.results | default([]) + | selectattr('rc', 'defined') + | selectattr('rc', 'equalto', 0) + | map(attribute='item') | list }} + + - name: Report that journald collection will be degraded or empty + ansible.builtin.debug: + msg: >- + {{ _ga_missing_journal_groups | join(' and ') }} missing on this host, + so {{ grafana_alloy_user }} gets + {{ (_ga_journal_groups | length == 0) | ternary('no journal access at + all: loki.source.journal will run without error and ship no log + lines', 'only partial journal access, which covers the common case but + not journal files whose ACL names only the missing group') }}. Expected + inside containers; on a real host, install systemd properly or set + grafana_alloy_logs_enabled=false. + vars: + _ga_missing_journal_groups: >- + {{ ['systemd-journal', 'adm'] | difference(_ga_journal_groups) }} + when: + - grafana_alloy_logs_enabled | bool + - _ga_journal_groups | length < 2 + # --- User + directories ---------------------------------------------------- - name: Create the alloy system group ansible.builtin.group: @@ -154,10 +199,49 @@ ansible.builtin.user: name: "{{ grafana_alloy_user }}" group: "{{ grafana_alloy_group }}" + # Journal readers only, and only those that exist. append keeps any + # membership an operator added by hand; the task below revokes the + # journal grant again if logs are later turned off, and the full disable + # path removes the account outright. + groups: "{{ _ga_journal_groups }}" + append: true system: true home: "{{ grafana_alloy_state_dir }}" create_home: false shell: /usr/sbin/nologin + # Group membership is applied at process start, so a membership change that + # does not also change the unit or the config still needs a restart. + notify: Restart alloy + + # Turning logs off drops SupplementaryGroups from the unit, but systemd then + # falls back to the ACCOUNT's supplementary groups — so a membership granted + # by an earlier logs-enabled run would silently keep journal read access. Take + # it back, preserving anything else the account belongs to. + - name: Read the alloy account's current supplementary groups + ansible.builtin.command: + cmd: id -nG {{ grafana_alloy_user | quote }} + changed_when: false + failed_when: false + check_mode: false # read-only lookup; must also run under make check + register: _ga_current_groups + when: not grafana_alloy_logs_enabled | bool + + - name: Revoke journal reader groups granted by an earlier logs-enabled run + ansible.builtin.user: + name: "{{ grafana_alloy_user }}" + groups: "{{ _ga_keep_groups }}" + append: false + vars: + # The primary group is set by `group:` above and must not be listed here. + _ga_keep_groups: >- + {{ (_ga_current_groups.stdout | default('')).split() + | difference(['systemd-journal', 'adm', grafana_alloy_group]) }} + when: + - not grafana_alloy_logs_enabled | bool + - _ga_current_groups.rc | default(1) == 0 + - (_ga_current_groups.stdout | default('')).split() + | intersect(['systemd-journal', 'adm']) | length > 0 + notify: Restart alloy - name: Ensure config + state directories exist ansible.builtin.file: diff --git a/ansible/roles/grafana_alloy/tasks/preflight.yml b/ansible/roles/grafana_alloy/tasks/preflight.yml index a03ad88..c459b4f 100644 --- a/ansible/roles/grafana_alloy/tasks/preflight.yml +++ b/ansible/roles/grafana_alloy/tasks/preflight.yml @@ -1,8 +1,10 @@ --- # Fail-loud preflight, run BEFORE any host mutation. Mirrors decdn_node's -# secret-file posture: this role never carries secrets in inventory and never -# reads the env file's values onto the control machine — it only greps shapes on -# the host itself. +# secret-file posture: this role never carries the API TOKEN in inventory and +# never reads the env file's values onto the control machine — it only greps +# shapes on the host itself. The non-secret half of the connection settings +# (endpoint URLs, instance IDs) MAY come from inventory; those are validated here +# too, and each one that is set removes its key from the host-state gates below. # # Ordering matters for testability (molecule/validation): PURE-INVENTORY gates # run first so a bad knob rejects deterministically regardless of what fixtures @@ -35,9 +37,93 @@ - "{{ grafana_alloy_retry_initial_interval }}" - "{{ grafana_alloy_retry_max_elapsed_time }}" - "{{ grafana_alloy_scrape_interval }}" + - "{{ grafana_alloy_host_scrape_interval }}" + - "{{ grafana_alloy_host_mount_timeout }}" + - "{{ grafana_alloy_logs_max_age }}" loop_control: label: "{{ item }}" +- name: Validate the observability sub-switches + ansible.builtin.assert: + that: + - grafana_alloy_host_metrics_enabled is boolean + - grafana_alloy_self_metrics_enabled is boolean + - grafana_alloy_logs_enabled is boolean + - grafana_alloy_host_include_exporter_metrics is boolean + fail_msg: >- + grafana_alloy_host_metrics_enabled, _self_metrics_enabled, _logs_enabled + and _host_include_exporter_metrics must be booleans (true/false), not + quoted strings — each gates whole component groups in config.alloy. + +# An unknown collector name is NOT a startup crash: Alloy resolves collectors by +# map lookup and silently ignores names it does not know, so a typo ships an +# exporter that quietly measures less than the operator thinks. A charset check +# cannot catch `diskstat`; only the committed inventory in vars/main.yml can. +- name: Validate the node_exporter collector names + ansible.builtin.assert: + that: + - item in _ga_known_collectors + quiet: true + fail_msg: >- + "{{ item }}" is not a node_exporter collector known to Alloy + {{ grafana_alloy_version }}. Check grafana_alloy_host_collectors / + _host_extra_collectors / _host_disable_collectors against the collector + table in the prometheus.exporter.unix reference. An unknown name is + silently ignored at runtime, so this gate is the only thing standing + between a typo and metrics that never arrive. (If upstream added a + collector, extend _ga_known_collectors in the role's vars/main.yml.) + loop: >- + {{ (grafana_alloy_host_collectors + + grafana_alloy_host_extra_collectors + + grafana_alloy_host_disable_collectors) | map('string') | list }} + loop_control: + label: "{{ item }}" + when: grafana_alloy_host_metrics_enabled | bool + +# NOTE the EFFECTIVE set, i.e. after the template subtracts +# _host_disable_collectors from the other two lists. An empty set_collectors is +# not "measure nothing" to Alloy — it reads as unset and re-enables the whole +# ~45-collector default set, the exact opposite of what an operator who disabled +# everything intended. +- name: Require a non-empty effective host collector set + ansible.builtin.assert: + that: + - >- + ((grafana_alloy_host_collectors + grafana_alloy_host_extra_collectors) + | reject('in', grafana_alloy_host_disable_collectors) | list | length) > 0 + fail_msg: >- + grafana_alloy_host_metrics_enabled is true but no collector survives + grafana_alloy_host_disable_collectors, which would silently fall back to + node_exporter's full default set instead of measuring nothing. Either + populate grafana_alloy_host_collectors or turn host metrics off with + grafana_alloy_host_metrics_enabled: false. + when: grafana_alloy_host_metrics_enabled | bool + +# job/instance are what Grafana Cloud's integration dashboards filter on, and +# like the identity labels they ride on EVERY series and log line. +- name: Validate the job labels + ansible.builtin.assert: + that: + - (item.allow_empty | bool and item.value | length == 0) + or item.value is match('^[A-Za-z0-9._/-]{1,63}$') + quiet: true + fail_msg: >- + Job labels are restricted to the same low-cardinality charset as the + identity labels (letters, digits, dot, underscore, slash, dash, ≤ 63 + chars): "{{ item.value }}" does not fit {{ item.name }}. + {{ item.allow_empty | bool | ternary('It may also be "" (fall back to the + component-ID job Alloy used before this label existed).', '') }} + loop: + # Only the node job may be blanked: that restores the pre-existing + # job="prometheus.scrape.decdn_node" identity. The other three feed Grafana + # Cloud integration dashboards that filter on an exact job value. + - {name: grafana_alloy_node_job, value: "{{ grafana_alloy_node_job | string }}", allow_empty: true} + - {name: grafana_alloy_host_job, value: "{{ grafana_alloy_host_job | string }}", allow_empty: false} + - {name: grafana_alloy_self_job, value: "{{ grafana_alloy_self_job | string }}", allow_empty: false} + - {name: grafana_alloy_logs_job, value: "{{ grafana_alloy_logs_job | string }}", allow_empty: false} + loop_control: + label: "{{ item.name }}" + - name: Validate integer pipeline knobs ansible.builtin.assert: that: @@ -87,7 +173,72 @@ grafana_alloy_deployment_environment and grafana_alloy_region are required identity attributes when Grafana Cloud observability is enabled. -- name: Validate loopback listener addresses +# --- Grafana Cloud connection settings (inventory half) ------------------------- +# These five are NON-secret and therefore inventory-settable; each one that is set +# replaces its sys.env("GC_…") lookup in the rendered config and removes the +# corresponding requirement from the host-state gates at the bottom of this file. +# The API token has no variable by design: config.alloy is 0644 on the host. + +- name: Validate any inventory-provided Grafana Cloud endpoints + ansible.builtin.assert: + that: + # Empty == "fall back to the env file", which is the default and always OK. + - item.value | length == 0 or item.value is match('^https://[^\s]+$') + quiet: true + fail_msg: >- + {{ item.name }} must be an https:// URL (got "{{ item.value }}"). Plain + http would ship your Grafana Cloud API token in cleartext. Leave it empty + to keep reading the value from {{ item.key }} in + {{ grafana_alloy_secret_file }} instead. + loop: + - {name: grafana_alloy_prom_url, key: GC_PROM_REMOTE_WRITE_URL, value: "{{ grafana_alloy_prom_url | string }}"} + - {name: grafana_alloy_otlp_endpoint, key: GC_OTLP_ENDPOINT, value: "{{ grafana_alloy_otlp_endpoint | string }}"} + - {name: grafana_alloy_loki_url, key: GC_LOKI_URL, value: "{{ grafana_alloy_loki_url | string }}"} + loop_control: + label: "{{ item.name }}" + +- name: Validate any inventory-provided Grafana Cloud instance IDs + ansible.builtin.assert: + that: + - item.value | length == 0 or item.value is match('^[A-Za-z0-9._-]{1,64}$') + quiet: true + fail_msg: >- + {{ item.name }} is the basic-auth username for the endpoint — the numeric + instance ID on Grafana Cloud, a tenant name on a self-hosted + Mimir/Loki — so it is restricted to letters, digits, dot, underscore and + dash, and must not be a token or carry whitespace. Got "{{ item.value }}". + NOTE the Loki instance ID differs from the Prometheus one; the + access-policy token is shared and stays in + {{ grafana_alloy_secret_file }} as GC_API_TOKEN. + loop: + - {name: grafana_alloy_prom_username, value: "{{ grafana_alloy_prom_username | string }}"} + - {name: grafana_alloy_loki_username, value: "{{ grafana_alloy_loki_username | string }}"} + loop_control: + label: "{{ item.name }}" + +- name: Refuse a Grafana Cloud token smuggled into inventory + ansible.builtin.assert: + that: + # Access-policy tokens (glc_), service-account tokens (glsa_), API keys + # (eyJ… — base64 JSON), and any URL carrying user:pass@host. + - not (item is search('(glc_|glsa_|eyJ)')) + - "'@' not in item" + quiet: true + fail_msg: >- + A Grafana Cloud access token (or a URL with embedded credentials) appears + in an inventory variable. config.alloy is world-readable on the host and + inventory is committed: the token belongs ONLY in + {{ grafana_alloy_secret_file }} as GC_API_TOKEN. + loop: + - "{{ grafana_alloy_prom_url | string }}" + - "{{ grafana_alloy_prom_username | string }}" + - "{{ grafana_alloy_otlp_endpoint | string }}" + - "{{ grafana_alloy_loki_url | string }}" + - "{{ grafana_alloy_loki_username | string }}" + loop_control: + label: "{{ item | truncate(24, true) }}" + +- name: Validate the loopback listener addresses ansible.builtin.assert: that: - grafana_alloy_otlp_grpc_bind == '127.0.0.1' @@ -185,6 +336,11 @@ # key is grepped for presence + value shape directly on the host: grep keeps the # values off stdout AND off the control machine (slurping the file back would put # the API token there, defeating the point of the host-provisioned path). +# +# "Required" is now conditional: a key is only needed when nothing in inventory +# supplies that value, and the Loki pair is only needed when logs are enabled. +# GC_API_TOKEN is ALWAYS required — it is the one value that must never live in +# inventory. - name: Confirm every required credential key is present and shaped correctly ansible.builtin.command: cmd: grep -qE -- '{{ item.regex }}' {{ grafana_alloy_secret_file | quote }} @@ -192,34 +348,57 @@ check_mode: false # read-only gate: must also run under make check failed_when: false loop: - # The two cloud URLs must be https:// (plain http would ship the API token in + # The cloud URLs must be https:// (plain http would ship the API token in # cleartext); the credential values must be non-empty, non-comment text. # POSIX bracket classes only — grep -E does NOT interpret \r/\n escapes (they # become literal 'r'/'n', silently poisoning the class), and each grep line is # already newline-free by construction. + - key: GC_API_TOKEN + regex: ^GC_API_TOKEN=[^[:space:]#][^[:space:]]*$ + var: "(none — the token must never be set in inventory)" + required: true - key: GC_PROM_REMOTE_WRITE_URL regex: ^GC_PROM_REMOTE_WRITE_URL=https://[^[:space:]]+$ + var: grafana_alloy_prom_url + required: "{{ grafana_alloy_prom_url | string | length == 0 }}" - key: GC_PROM_USERNAME regex: ^GC_PROM_USERNAME=[^[:space:]#][^[:space:]]*$ - - key: GC_API_TOKEN - regex: ^GC_API_TOKEN=[^[:space:]#][^[:space:]]*$ + var: grafana_alloy_prom_username + required: "{{ grafana_alloy_prom_username | string | length == 0 }}" - key: GC_OTLP_ENDPOINT regex: ^GC_OTLP_ENDPOINT=https://[^[:space:]]+$ + var: grafana_alloy_otlp_endpoint + required: "{{ grafana_alloy_otlp_endpoint | string | length == 0 }}" + - key: GC_LOKI_URL + regex: ^GC_LOKI_URL=https://[^[:space:]]+$ + var: grafana_alloy_loki_url + required: "{{ grafana_alloy_logs_enabled | bool and grafana_alloy_loki_url | string | length == 0 }}" + - key: GC_LOKI_USERNAME + regex: ^GC_LOKI_USERNAME=[^[:space:]#][^[:space:]]*$ + var: grafana_alloy_loki_username + required: "{{ grafana_alloy_logs_enabled | bool and grafana_alloy_loki_username | string | length == 0 }}" loop_control: label: "{{ item.key }}" + when: item.required | bool register: _ga_secret_grep - name: Report which credential key was missing or malformed ansible.builtin.fail: msg: >- {{ grafana_alloy_secret_file }} does not carry a usable line for - {{ _ga_missing_keys | join(', ') }}. The file must contain exactly the four - KEY=value pairs shown in roles/grafana_alloy/files/grafana-alloy.env.example; - both cloud URLs must be https:// and each value non-empty (systemd silently - drops unparseable lines, so a quoting typo looks identical to a missing key). + {{ _ga_missing | map(attribute='key') | join(', ') }}. Either add the + KEY=value pair(s) on the host (see + roles/grafana_alloy/files/grafana-alloy.env.example) — URLs must be + https:// and every value non-empty, since systemd silently drops + unparseable lines so a quoting typo looks identical to a missing key — or, + for everything except the token, set the matching inventory variable + instead: {{ _ga_missing | map(attribute='var') | join(', ') }}. vars: - _ga_missing_keys: >- - {{ _ga_secret_grep.results + # Skipped loop items (key not required) carry no rc, so filter on rc being + # defined first: rejectattr alone would count every skipped key as missing. + _ga_missing: >- + {{ _ga_secret_grep.results | default([]) + | selectattr('rc', 'defined') | rejectattr('rc', 'equalto', 0) - | map(attribute='item.key') | list }} - when: _ga_missing_keys | length > 0 + | map(attribute='item') | list }} + when: _ga_missing | length > 0 diff --git a/ansible/roles/grafana_alloy/templates/alloy.service.j2 b/ansible/roles/grafana_alloy/templates/alloy.service.j2 index 6aabc34..0e1a56a 100644 --- a/ansible/roles/grafana_alloy/templates/alloy.service.j2 +++ b/ansible/roles/grafana_alloy/templates/alloy.service.j2 @@ -35,23 +35,45 @@ TimeoutStopSec={{ grafana_alloy_stop_timeout_sec }} Restart=on-failure RestartSec=5 +{% if grafana_alloy_logs_enabled | default(false) | bool and (_ga_journal_groups | default([]) | length > 0) %} + +# Journal access. A non-root reader needs a group that journald grants read to: +# systemd-journal owns the journal files, and systemd-tmpfiles puts an ACL for +# adm on /var/log/journal. Without at least one of them loki.source.journal +# starts without error and collects NOTHING, which is the worst possible failure +# mode, so the role joins {{ grafana_alloy_user }} to both and warns when either +# is absent. Only groups that actually exist on this host are listed, because +# systemd refuses to start a unit naming a group that does not exist. +SupplementaryGroups={{ _ga_journal_groups | default([]) | join(' ') }} +{% endif %} # --- Hardening --------------------------------------------------------------- # Mirrors decdn-node.service: strict filesystem, privilege boundary, syscall # filter. StateDirectory owns {{ grafana_alloy_state_dir }} (the only writable -# path under ProtectSystem=strict): WAL/queue state lives there. +# path under ProtectSystem=strict): WAL/queue state lives there, including +# loki.source.journal's positions file. StateDirectory={{ grafana_alloy_state_dir | regex_replace('^/var/lib/', '') }} StateDirectoryMode=0750 NoNewPrivileges=true ProtectSystem=strict +{% if grafana_alloy_host_metrics_enabled | default(false) | bool %} +# read-only, NOT true: ProtectHome=true replaces /home, /root and /run/user with +# empty tmpfs mounts inside the unit's namespace, so node_exporter's filesystem +# collector reports those mounts as the empty overlay instead of the real disk — +# a full /home would never raise an alert. read-only keeps the privilege boundary +# (the agent still cannot write there) while reporting true usage. +ProtectHome=read-only +{% else %} ProtectHome=true +{% endif %} PrivateTmp=true PrivateDevices=true ProtectKernelTunables=true ProtectKernelModules=true ProtectControlGroups=true ProtectClock=true -# AF_INET/AF_INET6 for the loopback OTLP/scrape/remotes; AF_UNIX for journald. +# AF_INET/AF_INET6 for the loopback OTLP/scrape/remotes; AF_UNIX for journald and +# for the D-Bus system bus the node_exporter `systemd` collector queries. RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX RestrictNamespaces=true RestrictSUIDSGID=true diff --git a/ansible/roles/grafana_alloy/templates/config.alloy.j2 b/ansible/roles/grafana_alloy/templates/config.alloy.j2 index 561458f..8083103 100644 --- a/ansible/roles/grafana_alloy/templates/config.alloy.j2 +++ b/ansible/roles/grafana_alloy/templates/config.alloy.j2 @@ -5,9 +5,11 @@ // file (and everything the Jinja layer emits) must stay `//`. // // Pipeline: scrape the node's /metrics and push it to Grafana Cloud Prometheus; -// receive OTLP traces (plus any logs/metrics pushed over localhost) on gRPC 4317 / -// HTTP 4318 bound to loopback only; stamp resource identity on everything, -// probabilistically sample traces, batch, then export via OTLP/HTTP. +// scrape the MACHINE (Alloy's in-process node_exporter) and Alloy itself; ship +// journald to Grafana Cloud Loki; receive OTLP traces (plus any logs/metrics +// pushed over localhost) on gRPC 4317 / HTTP 4318 bound to loopback only; stamp +// resource identity on everything, probabilistically sample traces, batch, then +// export via OTLP/HTTP. // // Secrets are NOT literals here: every credential is read at config-load time // with the `sys.env` stdlib function from the root-owned 0600 environment file @@ -15,11 +17,35 @@ // roles/grafana_alloy/files/grafana-alloy.env.example). Alloy has no // `--config.expand-env` flag — `${VAR}` inside a string is a literal, so // `sys.env("VAR")` is the only working indirection. +// +// The NON-secret half of the credentials (endpoint URLs, the two numeric +// instance IDs) may instead come from inventory; see the ga_endpoint macro. + +{# Renders a Grafana Cloud connection value as either an inline string literal + (when inventory set it) or the sys.env lookup this role has always used (when + inventory left it empty). Keeping both shapes means a host provisioned the old + way — all keys in /etc/grafana-alloy.env — renders exactly the previous + configuration, while a host driven from inventory needs only GC_API_TOKEN on + disk. tojson does the quoting/escaping in both branches. + NEVER call this for the API token: config.alloy is 0644 on the host. + The `| string` is load-bearing: instance IDs are numeric, so an unquoted + `grafana_alloy_prom_username: 1234567` in inventory arrives as an int, which + has no length and which tojson would emit as a bare number — and Alloy's + basic_auth username is a string, so that is a config-load failure. #} +{% macro ga_endpoint(value, env_key) -%} +{% set ga_value = value | default('', true) | string %} +{% if ga_value | length > 0 %}{{ ga_value | tojson }}{% else %}sys.env({{ env_key | tojson }}){% endif %} +{%- endmacro %} prometheus.scrape "decdn_node" { targets = [ { "__address__" = "127.0.0.1:{{ decdn_metrics_port | default(9090) }}", - "instance" = "{{ _ga_instance_id }}" }, + "instance" = "{{ _ga_instance_id }}"{% if grafana_alloy_node_job | length > 0 %}, + {# BREAKING-CHANGE ESCAPE HATCH: with no explicit job label Alloy falls back + to the component ID, so these series used to be job="prometheus.scrape.decdn_node". + Set grafana_alloy_node_job: "" to keep the old identity for dashboards + and alerts that already hard-code it. #} + "job" = {{ grafana_alloy_node_job | tojson }}{% endif %} }, ] forward_to = [prometheus.relabel.decdn_identity.receiver] scrape_interval = {{ grafana_alloy_scrape_interval | to_json }} @@ -48,14 +74,253 @@ prometheus.relabel "decdn_identity" { prometheus.remote_write "cloud" { endpoint { - url = sys.env("GC_PROM_REMOTE_WRITE_URL") + url = {{ ga_endpoint(grafana_alloy_prom_url, 'GC_PROM_REMOTE_WRITE_URL') }} + + basic_auth { + username = {{ ga_endpoint(grafana_alloy_prom_username, 'GC_PROM_USERNAME') }} + password = sys.env("GC_API_TOKEN") + } + } +} + +{# --------------------------------------------------------------------------- + HOST (machine) METRICS — Alloy's in-process node_exporter. + + Runs inside the agent: no extra listener and no extra port, so the + loopback-only posture is unchanged. Identity is applied with + discovery.relabel (TARGET labels — evaluated once per target, then attached to + every sample at scrape time) rather than prometheus.relabel (per SAMPLE, with + a cardinality-sized cache). The scrape forwards STRAIGHT to remote_write, + deliberately bypassing prometheus.relabel.decdn_identity: that component + stamps service_name=decdn-node, which is false for machine series. + + `job` must stay integrations/node_exporter for Grafana Cloud's prebuilt Linux + Server dashboards and alert rules to find these series. + Every regex below is JSON-encoded, so no value can break out of its string. #} +{% if grafana_alloy_host_metrics_enabled | bool %} +{# Upstream applies disable_collectors FIRST and enable_collectors second, so a + name in both ends up ENABLED — the opposite of what an operator vetoing a + collector expects. Subtract the veto list here instead, so it really wins and + so the derived `systemd` block below agrees with what actually runs. #} +{% set ga_host_set_collectors = grafana_alloy_host_collectors + | reject('in', grafana_alloy_host_disable_collectors) | list %} +{% set ga_host_extra_collectors = grafana_alloy_host_extra_collectors + | reject('in', grafana_alloy_host_disable_collectors) | list %} +{% set ga_host_enabled_collectors = ga_host_set_collectors + ga_host_extra_collectors %} + +prometheus.exporter.unix "host" { + // set_collectors REPLACES the default set (~45 collectors): anything not + // listed is off. enable_collectors adds on top (it is the only way to turn on + // an upstream-defaultDisabled collector such as systemd). Both lists already + // have grafana_alloy_host_disable_collectors subtracted at render time. + set_collectors = {{ ga_host_set_collectors | tojson }} +{% if ga_host_extra_collectors | length > 0 %} + enable_collectors = {{ ga_host_extra_collectors | tojson }} +{% endif %} +{% if grafana_alloy_host_disable_collectors | length > 0 %} + disable_collectors = {{ grafana_alloy_host_disable_collectors | tojson }} +{% endif %} + include_exporter_metrics = {{ grafana_alloy_host_include_exporter_metrics | bool | tojson }} + + filesystem { + fs_types_exclude = {{ grafana_alloy_host_fs_types_exclude | tojson }} + mount_points_exclude = {{ grafana_alloy_host_mount_points_exclude | tojson }} + mount_timeout = {{ grafana_alloy_host_mount_timeout | to_json }} + } + + netdev { + device_exclude = {{ grafana_alloy_host_netdev_device_exclude | tojson }} + } + + // `disk` is the block name for the diskstats collector. + disk { + device_exclude = {{ grafana_alloy_host_disk_device_exclude | tojson }} + } +{% if 'systemd' in ga_host_enabled_collectors %} + + // Per-unit state. Reaches systemd over the D-Bus SYSTEM bus (Alloy does not + // expose node_exporter's root-only private-socket mode), so it needs AF_UNIX + // and /run/dbus/system_bus_socket — both available to the hardened unit, and + // both absent in a bare container, where it simply reports nothing. + systemd { + unit_include = {{ grafana_alloy_host_systemd_unit_include | tojson }} + unit_exclude = {{ grafana_alloy_host_systemd_unit_exclude | tojson }} + } +{% endif %} +} + +{# Host series carry job/instance (what the Cloud dashboards filter on) plus the + fleet-identity labels, but NOT service_name: these describe the machine, not + the daemon. Empty values drop their rule entirely. #} +{% set ga_host_labels = [ + ["job", grafana_alloy_host_job], + ["instance", _ga_instance_id], + ["service_namespace", grafana_alloy_service_namespace], + ["deployment_environment", grafana_alloy_deployment_environment], + ["region", grafana_alloy_region], + ] %} +discovery.relabel "host" { + targets = prometheus.exporter.unix.host.targets +{% for name, value in ga_host_labels if value | length > 0 %} + + rule { + target_label = {{ name | tojson }} + replacement = {{ value | tojson }} + } +{% endfor %} +} + +prometheus.scrape "host" { + targets = discovery.relabel.host.output + forward_to = [prometheus.remote_write.cloud.receiver] + scrape_interval = {{ grafana_alloy_host_scrape_interval | to_json }} +} +{% endif %} +{% if grafana_alloy_self_metrics_enabled | bool %} + +{# Alloy's own health: component evaluation, WAL, remote-write queue and + failures. This is what distinguishes "the node went quiet" from "the agent + that reports on the node is wedged", so it is cheap insurance. #} +prometheus.exporter.self "alloy" {} + +{% set ga_self_labels = [ + ["job", grafana_alloy_self_job], + ["instance", _ga_instance_id], + ["service_namespace", grafana_alloy_service_namespace], + ["deployment_environment", grafana_alloy_deployment_environment], + ["region", grafana_alloy_region], + ] %} +discovery.relabel "alloy_self" { + targets = prometheus.exporter.self.alloy.targets +{% for name, value in ga_self_labels if value | length > 0 %} + + rule { + target_label = {{ name | tojson }} + replacement = {{ value | tojson }} + } +{% endfor %} +} + +prometheus.scrape "alloy_self" { + targets = discovery.relabel.alloy_self.output + forward_to = [prometheus.remote_write.cloud.receiver] + scrape_interval = {{ grafana_alloy_host_scrape_interval | to_json }} +} +{% endif %} +{% if grafana_alloy_logs_enabled | bool %} + +{# --------------------------------------------------------------------------- + HOST + SERVICE LOGS — journald to Grafana Cloud Loki. + + Two relabel stages, and the split is not cosmetic: journald fields arrive as + __journal*-prefixed labels that are STRIPPED before entries reach any + downstream receiver, so anything reading them (the field mapping AND the + cost-guardrail drops) MUST live in the rules-only ruleset consumed by + loki.source.journal. Identity labels are applied afterwards, on the entries. + + Reading the journal needs group membership (systemd-journal + adm), granted by + the role and pinned in the unit's SupplementaryGroups — without it the + component starts happily and collects nothing at all. The positions file lands + under --storage.path, i.e. inside the existing StateDirectory. #} +loki.relabel "journal_rules" { + // Rules-only usage: the ruleset is consumed via .rules below, so nothing is + // forwarded from here. The argument is still required. + forward_to = [] + + rule { + source_labels = ["__journal__systemd_unit"] + target_label = "unit" + } + + rule { + source_labels = ["__journal__hostname"] + target_label = "hostname" + } + + rule { + source_labels = ["__journal_priority_keyword"] + target_label = "level" + } + + rule { + source_labels = ["__journal__transport"] + target_label = "transport" + } + + rule { + source_labels = ["__journal__boot_id"] + target_label = "boot_id" + } +{% if grafana_alloy_logs_drop_priority_regex | length > 0 %} + + // Cost guardrail: the regex is anchored to the WHOLE value by Prometheus + // relabel semantics, so "debug" drops debug and nothing else. + rule { + source_labels = ["__journal_priority_keyword"] + regex = {{ grafana_alloy_logs_drop_priority_regex | tojson }} + action = "drop" + } +{% endif %} +{% if grafana_alloy_logs_drop_unit_regex | length > 0 %} + + // Cost guardrail: chatty units nobody debugs from Loki. Entries with no unit + // (kernel, for example) have an empty value here and are kept. + rule { + source_labels = ["__journal__systemd_unit"] + regex = {{ grafana_alloy_logs_drop_unit_regex | tojson }} + action = "drop" + } +{% endif %} +} + +loki.source.journal "host" { +{% if grafana_alloy_logs_journal_path | length > 0 %} + path = {{ grafana_alloy_logs_journal_path | tojson }} +{% else %} + // No path: Alloy reads both /var/log/journal and /run/log/journal. +{% endif %} + // Bounds the catch-up burst after an outage or a fresh install. + max_age = {{ grafana_alloy_logs_max_age | to_json }} + format_as_json = false + relabel_rules = loki.relabel.journal_rules.rules + forward_to = [loki.relabel.journal_identity.receiver] +} + +{# Same job/instance pair as the host metrics above — Grafana Cloud's Linux + Server integration dashboards correlate logs and metrics on exactly these + two labels. Set here (not via the source's `labels` argument) so they + override the job label loki.source.journal adds on its own. #} +{% set ga_log_labels = [ + ["job", grafana_alloy_logs_job], + ["instance", _ga_instance_id], + ["service_namespace", grafana_alloy_service_namespace], + ["deployment_environment", grafana_alloy_deployment_environment], + ["region", grafana_alloy_region], + ] %} +loki.relabel "journal_identity" { + forward_to = [loki.write.cloud.receiver] +{% for name, value in ga_log_labels if value | length > 0 %} + + rule { + target_label = {{ name | tojson }} + replacement = {{ value | tojson }} + } +{% endfor %} +} + +loki.write "cloud" { + endpoint { + url = {{ ga_endpoint(grafana_alloy_loki_url, 'GC_LOKI_URL') }} + // Grafana Cloud's Loki instance ID is NOT the Prometheus one; the same + // access-policy token authenticates both. basic_auth { - username = sys.env("GC_PROM_USERNAME") + username = {{ ga_endpoint(grafana_alloy_loki_username, 'GC_LOKI_USERNAME') }} password = sys.env("GC_API_TOKEN") } } } +{% endif %} otelcol.receiver.otlp "local" { grpc { @@ -141,7 +406,7 @@ otelcol.processor.batch "decdn" { `client` — nesting them inside `client` is rejected at config load. #} otelcol.exporter.otlphttp "cloud" { client { - endpoint = sys.env("GC_OTLP_ENDPOINT") + endpoint = {{ ga_endpoint(grafana_alloy_otlp_endpoint, 'GC_OTLP_ENDPOINT') }} auth = otelcol.auth.basic.cloud.handler } @@ -156,6 +421,6 @@ otelcol.exporter.otlphttp "cloud" { } otelcol.auth.basic "cloud" { - username = sys.env("GC_PROM_USERNAME") + username = {{ ga_endpoint(grafana_alloy_prom_username, 'GC_PROM_USERNAME') }} password = sys.env("GC_API_TOKEN") } diff --git a/ansible/roles/grafana_alloy/vars/main.yml b/ansible/roles/grafana_alloy/vars/main.yml index a2652d8..b7633b4 100644 --- a/ansible/roles/grafana_alloy/vars/main.yml +++ b/ansible/roles/grafana_alloy/vars/main.yml @@ -8,3 +8,92 @@ # destructive operation against an unrelated Alloy install. _ga_unit_path: /etc/systemd/system/alloy.service _ga_managed_marker: "# MANAGED BY the grafana_alloy role — do not edit by hand." + +# Every collector name prometheus.exporter.unix knows, from the Alloy 1.19 +# component reference (the "Collectors list" table), plus four that ship in +# upstream node_exporter but are missing from that table (cgroups, selinux, +# slabinfo, watchdog). Non-Linux names are kept so the gate never punishes a +# copy-pasted example. +# +# WHY A COMMITTED INVENTORY: an unknown collector name is NOT a startup error. +# Alloy builds the set by map lookup and silently drops anything it does not +# recognise, so `diskstat` (missing the final s) validates clean, starts clean, +# and ships no disk metrics forever. A charset check cannot catch that; only a +# name list can. Re-sync when bumping grafana_alloy_version. +_ga_known_collectors: + - arp + - bcache + - bonding + - boottime + - btrfs + - buddyinfo + - cgroups + - conntrack + - cpu + - cpufreq + - devstat + - diskstats + - dmi + - drbd + - drm + - edac + - entropy + - ethtool + - exec + - fibrechannel + - filefd + - filesystem + - hwmon + - infiniband + - interrupts + - ipvs + - ksmd + - lnstat + - loadavg + - logind + - mdadm + - meminfo + - meminfo_numa + - mountstats + - netclass + - netdev + - netisr + - netstat + - network_route + - nfs + - nfsd + - ntp + - nvme + - os + - perf + - powersupplyclass + - pressure + - processes + - qdisc + - rapl + - runit + - schedstat + - selinux + - slabinfo + - sockstat + - softirqs + - softnet + - stat + - supervisord + - sysctl + - systemd + - tapestats + - tcpstat + - textfile + - thermal + - thermal_zone + - time + - timex + - udp_queues + - uname + - vmstat + - watchdog + - wifi + - xfs + - zfs + - zoneinfo diff --git a/ansible/tests/alloy-config/render.yml b/ansible/tests/alloy-config/render.yml index 05a990d..4005fb8 100644 --- a/ansible/tests/alloy-config/render.yml +++ b/ansible/tests/alloy-config/render.yml @@ -10,13 +10,21 @@ # shipped green by the stub. This harness closes that gap. # # Renders several variable combinations so the template's conditional branches -# (omitted identity attributes, 0 and 1 sampling ratios, overridden ports) are -# all parsed by the real binary, not just the default one. +# (omitted identity attributes, 0 and 1 sampling ratios, overridden ports, each +# machine-monitoring sub-knob, inventory-vs-env credentials) are all parsed by +# the real binary, not just the default one. +# +# PRECEDENCE, and why every override lives on the TASK rather than the play: +# `vars_files` (precedence 14) outranks play `vars` (12), so a play-level +# override of anything defined in the role's defaults file is silently ignored — +# it renders the default and the case proves nothing. Task `vars` (19) win, so +# each combination below sets its knobs on the template task itself. Do not move +# them up to the play. # # Not a deploy playbook: localhost only, writes only into $ALLOY_RENDER_DIR, and # `become: false` overrides ansible.cfg's project-wide privilege escalation. -- name: Render the default-values Alloy configuration +- name: Render the grafana_alloy templates in every interesting combination hosts: localhost connection: local gather_facts: false @@ -25,97 +33,209 @@ - ../../roles/grafana_alloy/defaults/main.yml vars: _ga_render_dir: "{{ lookup('ansible.builtin.env', 'ALLOY_RENDER_DIR') }}" - _ga_instance_id: decdn-node-1 - grafana_alloy_bin_effective: /usr/bin/alloy - grafana_alloy_deployment_environment: production - grafana_alloy_region: US tasks: - - name: Render config.alloy and the systemd unit with default knobs + # Everything at its default: machine metrics, agent self-metrics and journald + # are all ON by default, so this is also the shape a fresh deploy gets. + - name: Render the default-values configuration ansible.builtin.template: src: "{{ playbook_dir }}/../../roles/grafana_alloy/templates/{{ item.src }}" dest: "{{ _ga_render_dir }}/defaults.{{ item.ext }}" mode: "0644" - loop: + loop: &ga_render_pair - {src: config.alloy.j2, ext: alloy} - {src: alloy.service.j2, ext: service} loop_control: label: "defaults.{{ item.ext }}" + vars: + _ga_instance_id: decdn-node-1 + _ga_journal_groups: [systemd-journal, adm] + grafana_alloy_bin_effective: /usr/bin/alloy + grafana_alloy_deployment_environment: production + grafana_alloy_region: US - # Hands validate.sh the release pin without making it parse YAML itself: the - # values come from the role's own defaults, so the download it verifies is - # byte-identical to the one the role installs. - - name: Export the pinned release for the validation script - ansible.builtin.copy: - dest: "{{ _ga_render_dir }}/pin.env" - mode: "0644" - content: | - GRAFANA_ALLOY_VERSION={{ grafana_alloy_version }} - GRAFANA_ALLOY_SHA256_AMD64={{ grafana_alloy_sha256.amd64 | default('') }} - GRAFANA_ALLOY_SHA256_ARM64={{ grafana_alloy_sha256.arm64 | default('') }} - -# Exercises the omit branches: an empty namespace/region must drop its relabel -# rule and its OTTL `set()` statement entirely rather than emit an empty value, -# and a 0 ratio must still render a valid float. -- name: Render the minimal-identity Alloy configuration - hosts: localhost - connection: local - gather_facts: false - become: false - vars_files: - - ../../roles/grafana_alloy/defaults/main.yml - vars: - _ga_render_dir: "{{ lookup('ansible.builtin.env', 'ALLOY_RENDER_DIR') }}" - _ga_instance_id: decdn-node-2 - grafana_alloy_bin_effective: /usr/local/bin/alloy - grafana_alloy_service_namespace: "" - grafana_alloy_deployment_environment: "" - grafana_alloy_region: "" - grafana_alloy_trace_sampling_ratio: 0 - tasks: - - name: Render config.alloy and the systemd unit with minimal identity + # Exercises the omit branches: an empty namespace/region must drop its relabel + # rule and its OTTL `set()` statement entirely rather than emit an empty value, + # and a 0 ratio must still render a valid float. + - name: Render the minimal-identity configuration ansible.builtin.template: src: "{{ playbook_dir }}/../../roles/grafana_alloy/templates/{{ item.src }}" dest: "{{ _ga_render_dir }}/minimal.{{ item.ext }}" mode: "0644" - loop: - - {src: config.alloy.j2, ext: alloy} - - {src: alloy.service.j2, ext: service} + loop: *ga_render_pair loop_control: label: "minimal.{{ item.ext }}" + vars: + _ga_instance_id: decdn-node-2 + grafana_alloy_bin_effective: /usr/local/bin/alloy + grafana_alloy_service_namespace: "" + grafana_alloy_deployment_environment: "" + grafana_alloy_region: "" + grafana_alloy_trace_sampling_ratio: 0 -# Exercises the overridden branches: non-default ports, a keep-everything ratio, -# and pipeline knobs well away from their defaults. -- name: Render the fully-overridden Alloy configuration - hosts: localhost - connection: local - gather_facts: false - become: false - vars_files: - - ../../roles/grafana_alloy/defaults/main.yml - vars: - _ga_render_dir: "{{ lookup('ansible.builtin.env', 'ALLOY_RENDER_DIR') }}" - _ga_instance_id: decdn-node-3 - grafana_alloy_bin_effective: /usr/bin/alloy - grafana_alloy_deployment_environment: staging - grafana_alloy_region: EU - grafana_alloy_trace_sampling_ratio: 1 - grafana_alloy_scrape_interval: 15s - grafana_alloy_batch_timeout: 500ms - grafana_alloy_batch_send_size: 512 - grafana_alloy_batch_max_size: 4096 - grafana_alloy_queue_size: 100 - grafana_alloy_retry_initial_interval: 1s - grafana_alloy_retry_max_elapsed_time: 1h - grafana_alloy_otlp_http_port: 14318 - decdn_metrics_port: 19090 - tasks: - - name: Render config.alloy and the systemd unit with overridden knobs + # Exercises the overridden branches: non-default ports, a keep-everything + # ratio, and pipeline knobs well away from their defaults. + - name: Render the fully-overridden configuration ansible.builtin.template: src: "{{ playbook_dir }}/../../roles/grafana_alloy/templates/{{ item.src }}" dest: "{{ _ga_render_dir }}/overridden.{{ item.ext }}" mode: "0644" - loop: - - {src: config.alloy.j2, ext: alloy} - - {src: alloy.service.j2, ext: service} + loop: *ga_render_pair loop_control: label: "overridden.{{ item.ext }}" + vars: + _ga_instance_id: decdn-node-3 + _ga_journal_groups: [systemd-journal] + grafana_alloy_bin_effective: /usr/bin/alloy + grafana_alloy_deployment_environment: staging + grafana_alloy_region: EU + grafana_alloy_trace_sampling_ratio: 1 + grafana_alloy_scrape_interval: 15s + grafana_alloy_batch_timeout: 500ms + grafana_alloy_batch_send_size: 512 + grafana_alloy_batch_max_size: 4096 + grafana_alloy_queue_size: 100 + grafana_alloy_retry_initial_interval: 1s + grafana_alloy_retry_max_elapsed_time: 1h + grafana_alloy_otlp_http_port: 14318 + grafana_alloy_node_job: decdn-node-custom + decdn_metrics_port: 19090 + + # --- Machine-monitoring matrix ------------------------------------------ + # The default case above already renders host metrics, self metrics and + # journald together; these four isolate the branches that would otherwise + # never be parsed by the real binary. + + # Host metrics ONLY, hand-picked collectors, no journal groups on the host: + # proves the exporter/discovery.relabel/scrape trio stands alone, that the + # systemd sub-block disappears with its collector, and that ProtectHome + # relaxes to read-only. + - name: Render the host-metrics-only configuration + ansible.builtin.template: + src: "{{ playbook_dir }}/../../roles/grafana_alloy/templates/{{ item.src }}" + dest: "{{ _ga_render_dir }}/hostonly.{{ item.ext }}" + mode: "0644" + loop: *ga_render_pair + loop_control: + label: "hostonly.{{ item.ext }}" + vars: + _ga_instance_id: decdn-node-4 + _ga_journal_groups: [] + grafana_alloy_bin_effective: /usr/bin/alloy + grafana_alloy_deployment_environment: production + grafana_alloy_region: US + grafana_alloy_host_metrics_enabled: true + grafana_alloy_self_metrics_enabled: false + grafana_alloy_logs_enabled: false + grafana_alloy_host_scrape_interval: 120s + grafana_alloy_host_collectors: [cpu, meminfo] + grafana_alloy_host_extra_collectors: [] + grafana_alloy_host_disable_collectors: [hwmon] + grafana_alloy_host_include_exporter_metrics: true + + # Logs ONLY, explicit journal path, both cost-guardrail drops cleared (the + # omit branches of the rules), journal groups present — the only combination + # that renders SupplementaryGroups. + - name: Render the logs-only configuration + ansible.builtin.template: + src: "{{ playbook_dir }}/../../roles/grafana_alloy/templates/{{ item.src }}" + dest: "{{ _ga_render_dir }}/logsonly.{{ item.ext }}" + mode: "0644" + loop: *ga_render_pair + loop_control: + label: "logsonly.{{ item.ext }}" + vars: + _ga_instance_id: decdn-node-5 + _ga_journal_groups: [systemd-journal, adm] + grafana_alloy_bin_effective: /usr/bin/alloy + grafana_alloy_deployment_environment: production + grafana_alloy_region: US + grafana_alloy_host_metrics_enabled: false + grafana_alloy_self_metrics_enabled: false + grafana_alloy_logs_enabled: true + grafana_alloy_logs_journal_path: /var/log/journal + grafana_alloy_logs_max_age: 30m + grafana_alloy_logs_drop_priority_regex: "" + grafana_alloy_logs_drop_unit_regex: "" + + # BACKWARDS COMPATIBILITY: every sub-knob off must reproduce the + # pre-machine-monitoring pipeline exactly — node scrape + OTLP, all + # credentials via sys.env. validate.sh asserts the absence of every new + # component here. + - name: Render the legacy (sub-knobs off) configuration + ansible.builtin.template: + src: "{{ playbook_dir }}/../../roles/grafana_alloy/templates/{{ item.src }}" + dest: "{{ _ga_render_dir }}/legacy.{{ item.ext }}" + mode: "0644" + loop: *ga_render_pair + loop_control: + label: "legacy.{{ item.ext }}" + vars: + _ga_instance_id: decdn-node-6 + grafana_alloy_bin_effective: /usr/bin/alloy + grafana_alloy_deployment_environment: production + grafana_alloy_region: US + # Blanked too: the explicit node job label is itself new, so leaving it + # at its default would change the daemon series' identity and this case + # would no longer be the old pipeline. + grafana_alloy_node_job: "" + grafana_alloy_host_metrics_enabled: false + grafana_alloy_self_metrics_enabled: false + grafana_alloy_logs_enabled: false + + # The inventory-driven connection settings: all five non-secret values set, + # so every sys.env lookup EXCEPT the token must disappear. + - name: Render the inventory-endpoint configuration + ansible.builtin.template: + src: "{{ playbook_dir }}/../../roles/grafana_alloy/templates/{{ item.src }}" + dest: "{{ _ga_render_dir }}/inventory.{{ item.ext }}" + mode: "0644" + loop: *ga_render_pair + loop_control: + label: "inventory.{{ item.ext }}" + vars: + _ga_instance_id: decdn-node-7 + _ga_journal_groups: [systemd-journal, adm] + grafana_alloy_bin_effective: /usr/bin/alloy + grafana_alloy_deployment_environment: production + grafana_alloy_region: US + grafana_alloy_prom_url: https://prometheus-prod-99.render.invalid/api/prom/push + # DELIBERATELY UNQUOTED: instance IDs are numeric, so YAML hands these + # over as ints. Alloy's basic_auth username is a string, so the template + # must coerce rather than emit a bare number. + grafana_alloy_prom_username: 1234567 + grafana_alloy_otlp_endpoint: https://otlp-gateway-prod-99.render.invalid/otlp + grafana_alloy_loki_url: https://logs-prod-99.render.invalid/loki/api/v1/push + grafana_alloy_loki_username: 7654321 + + # The two escape hatches an upgrading operator reaches for: vetoing a + # collector must actually remove it (and its scoping block) rather than + # leaving it enabled-but-unscoped, and blanking the node job must restore the + # pre-existing component-ID identity for dashboards that hard-code it. + - name: Render the vetoed-collector configuration + ansible.builtin.template: + src: "{{ playbook_dir }}/../../roles/grafana_alloy/templates/{{ item.src }}" + dest: "{{ _ga_render_dir }}/vetoed.{{ item.ext }}" + mode: "0644" + loop: *ga_render_pair + loop_control: + label: "vetoed.{{ item.ext }}" + vars: + _ga_instance_id: decdn-node-8 + _ga_journal_groups: [systemd-journal, adm] + grafana_alloy_bin_effective: /usr/bin/alloy + grafana_alloy_deployment_environment: production + grafana_alloy_region: US + grafana_alloy_node_job: "" + grafana_alloy_host_disable_collectors: [systemd, netdev] + + # Hands validate.sh the release pin without making it parse YAML itself: the + # values come from the role's own defaults, so the download it verifies is + # byte-identical to the one the role installs. + - name: Export the pinned release for the validation script + ansible.builtin.copy: + dest: "{{ _ga_render_dir }}/pin.env" + mode: "0644" + content: | + GRAFANA_ALLOY_VERSION={{ grafana_alloy_version }} + GRAFANA_ALLOY_SHA256_AMD64={{ grafana_alloy_sha256.amd64 | default('') }} + GRAFANA_ALLOY_SHA256_ARM64={{ grafana_alloy_sha256.arm64 | default('') }} diff --git a/ansible/tests/alloy-config/validate.sh b/ansible/tests/alloy-config/validate.sh index 10f462b..a88a189 100755 --- a/ansible/tests/alloy-config/validate.sh +++ b/ansible/tests/alloy-config/validate.sh @@ -106,9 +106,90 @@ for config in "${configs[@]}"; do if ! grep -qF 'sys.env("GC_API_TOKEN")' "$config"; then fail "$name does not read the API token via sys.env — credentials must never be literals" fi + # The non-secret endpoints may be inventory literals; the token may NOT. This + # catches a token pasted into a group_vars file and rendered into the 0644 + # config on the host. Patterns stay in lockstep with the preflight gate + # ("Refuse a Grafana Cloud token smuggled into inventory"): access-policy + # tokens, service-account tokens, and base64 JSON API keys. + if grep -qE 'glc_|glsa_|eyJ' "$config"; then + fail "$name contains a literal Grafana Cloud token — it belongs only in the 0600 env file" + fi echo "ok: $name" done +# --- Gate 1b: the machine-monitoring matrix renders what it claims ------------ +# `alloy validate` proves each file loads; these greps prove the Jinja branches +# actually switched. Without them a knob could silently render nothing and still +# pass every other gate. +assert_has() { # file, needle, why + grep -qF "$2" "$render_dir/$1" || fail "$1 is missing $3 ($2)" +} +assert_lacks() { # file, needle, why + ! grep -qF "$2" "$render_dir/$1" || fail "$1 unexpectedly contains $3 ($2)" +} + +# Defaults: machine metrics, agent self-metrics and journald are all ON, and the +# job labels Grafana Cloud's Linux Server integration dashboards key off. +assert_has defaults.alloy 'prometheus.exporter.unix "host"' "the host metrics exporter" +assert_has defaults.alloy 'prometheus.exporter.self "alloy"' "agent self-monitoring" +assert_has defaults.alloy 'loki.source.journal "host"' "the journald source" +assert_has defaults.alloy 'loki.write "cloud"' "the Loki writer" +assert_has defaults.alloy '"job"' "an explicit job label" +assert_has defaults.alloy 'integrations/node_exporter' "the Grafana Cloud integration job label" +assert_has defaults.alloy '__journal__systemd_unit' "the journal field mapping" +assert_has defaults.alloy 'systemd {' "the per-unit systemd collector" + +# Sub-knob isolation. +assert_has hostonly.alloy 'prometheus.exporter.unix "host"' "the host metrics exporter" +assert_lacks hostonly.alloy 'loki.' "log components while logs are disabled" +assert_lacks hostonly.alloy 'prometheus.exporter.self' "self metrics while they are disabled" +assert_lacks hostonly.alloy 'systemd {' "the systemd collector block with no systemd collector enabled" +assert_has logsonly.alloy 'loki.source.journal "host"' "the journald source" +assert_has logsonly.alloy 'path = "/var/log/journal"' "the explicit journal path" +assert_lacks logsonly.alloy 'prometheus.exporter' "any exporter while only logs are enabled" +assert_lacks logsonly.alloy 'action = "drop"' "a drop rule after both guardrail regexes were cleared" + +# BACKWARDS COMPATIBILITY: sub-knobs off == the pre-machine-monitoring pipeline. +assert_lacks legacy.alloy 'prometheus.exporter' "any exporter" +assert_lacks legacy.alloy 'discovery.relabel' "any target relabeller" +assert_lacks legacy.alloy 'loki.' "any log component" +assert_has legacy.alloy 'sys.env("GC_PROM_REMOTE_WRITE_URL")' "the env fallback for remote-write" +assert_has legacy.alloy 'sys.env("GC_PROM_USERNAME")' "the env fallback for the instance ID" +assert_has legacy.alloy 'sys.env("GC_OTLP_ENDPOINT")' "the env fallback for OTLP" +assert_lacks legacy.alloy 'GC_LOKI' "a Loki credential lookup while logs are disabled" +assert_lacks legacy.alloy '"job"' "an explicit job label, which did not exist before this change" + +# Inventory-driven connection settings: every non-secret sys.env lookup is gone, +# the token's is not. +for key in GC_PROM_REMOTE_WRITE_URL GC_PROM_USERNAME GC_OTLP_ENDPOINT GC_LOKI_URL GC_LOKI_USERNAME; do + assert_lacks inventory.alloy "sys.env(\"$key\")" "an env lookup that inventory already supplies" +done +assert_has inventory.alloy 'https://prometheus-prod-99.render.invalid/api/prom/push' "the inventory remote-write URL" +assert_has inventory.alloy 'https://logs-prod-99.render.invalid/loki/api/v1/push' "the inventory Loki URL" +assert_has inventory.alloy 'sys.env("GC_API_TOKEN")' "the token env lookup, which must never move to inventory" +# Numeric instance IDs arrive from YAML as ints; Alloy needs quoted strings. +assert_has inventory.alloy 'username = "1234567"' "the Prometheus instance ID as a quoted string" +assert_has inventory.alloy 'username = "7654321"' "the Loki instance ID as a quoted string" + +# Escape hatches: the collector veto must remove the collector AND its scoping +# block from the rendered lists (upstream would otherwise re-enable it), and a +# blank node job must restore the implicit component-ID identity. +assert_has vetoed.alloy 'prometheus.exporter.unix "host"' "the host metrics exporter" +assert_lacks vetoed.alloy 'systemd {' "the systemd block for a vetoed collector" +assert_lacks vetoed.alloy 'enable_collectors =' "an enable list whose only entry was vetoed" +assert_has vetoed.alloy 'disable_collectors =' "the veto list itself" +assert_has defaults.alloy 'enable_collectors = ["systemd"]' "the systemd collector on the enable list" +assert_lacks vetoed.alloy '"job" = "decdn-node"' "a node job label that was explicitly blanked" +assert_has vetoed.alloy 'replacement = "integrations/node_exporter"' "the host job label" + +# Unit hardening follows the enabled signals. +assert_has hostonly.service 'ProtectHome=read-only' "the ProtectHome relaxation host metrics require" +assert_has legacy.service 'ProtectHome=true' "the strict ProtectHome used when host metrics are off" +assert_has logsonly.service 'SupplementaryGroups=systemd-journal adm' "the journal reader groups" +assert_lacks hostonly.service 'SupplementaryGroups' "journal groups while logs are disabled" +assert_lacks legacy.service 'SupplementaryGroups' "journal groups while logs are disabled" +echo "ok: machine-monitoring render matrix" + # --- Gate 2: every ExecStart flag exists ------------------------------------- # `alloy run` ignores nothing: an unknown flag exits non-zero, i.e. a systemd # crash-loop on the target host the moment the unit starts.