From 459d76a822ead7394c20a0ff96ebe210c379ba23 Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Thu, 17 Sep 2026 17:43:46 +0300 Subject: [PATCH 1/8] feat(ansible): allow the grafana_alloy API token via git-ignored inventory MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Give GC_API_TOKEN the same dual-home treatment decdn_node gives rpc_url (#39 parity): set grafana_alloy_api_token in host_vars secret.yml and the role authors /etc/grafana-alloy.env itself — token-only, root 0600, JSON-encoded for systemd; leave it empty to keep the operator-provisioned host path. A provenance record next to the file drives two fail-loud guards (missing-secret adoption after an inventory deploy; hand-edited file meeting an inventory token without the overwrite opt-in), the disable path removes its own record behind the managed-by marker, and an out-of-band host edit restarts the agent before the record lands. --- ansible/README.md | 16 +- .../host_vars/decdn-node-1/secret.yml.example | 10 + .../molecule/grafana-cloud-token/converge.yml | 35 ++++ .../grafana-cloud-token/files/alloy-stub | 31 +++ .../molecule/grafana-cloud-token/molecule.yml | 43 +++++ .../molecule/grafana-cloud-token/verify.yml | 179 ++++++++++++++++++ ansible/molecule/validation/converge.yml | 68 ++++++- ansible/roles/grafana_alloy/README.md | 45 ++++- ansible/roles/grafana_alloy/defaults/main.yml | 43 ++++- .../files/grafana-alloy.env.example | 6 +- ansible/roles/grafana_alloy/tasks/main.yml | 113 +++++++++++ .../roles/grafana_alloy/tasks/preflight.yml | 160 ++++++++++++++-- .../grafana_alloy/tasks/validate-paths.yml | 7 + 13 files changed, 722 insertions(+), 34 deletions(-) create mode 100644 ansible/molecule/grafana-cloud-token/converge.yml create mode 100755 ansible/molecule/grafana-cloud-token/files/alloy-stub create mode 100644 ansible/molecule/grafana-cloud-token/molecule.yml create mode 100644 ansible/molecule/grafana-cloud-token/verify.yml diff --git a/ansible/README.md b/ansible/README.md index 278c905..56ca915 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -220,6 +220,13 @@ Each of those variables is optional: left empty, the value is read from the matc `GC_…` key in that file instead (`roles/grafana_alloy/files/grafana-alloy.env.example` lists them all; URLs must be https). +Alternatively the token itself can ride git-ignored inventory — set +`grafana_alloy_api_token: "glc_…"` in `host_vars//secret.yml`, same channel as +`decdn_rpc_url`, and the role authors `/etc/grafana-alloy.env` for you (token-only, +root 0600). The authoring rewrites the file wholesale, so any hand-added `GC_*` keys +must move to their inventory variables first; provenance guards fail loud on silent +adoption or clobbering. See "The API token has two homes now" in the role README. + **Upgrading a host deployed before machine monitoring existed:** journald shipping is on by default and needs a Loki endpoint + instance ID the old four-key file does not carry, so preflight fails until you either set `grafana_alloy_loki_url` / `_loki_username` in @@ -254,9 +261,11 @@ make lint-alloy # render grafana_alloy's templates, then `alloy validate` th `make molecule` runs every scenario under `molecule/`: **`default`** (described below), `schema` (config key-set drift against the upstream field list), `validation` (bad knobs, for both roles, must be rejected by their own asserts), `generate-keystore` (opt-in -host-side wallet), `host-env` (host-provisioned `/etc/decdn/decdn.env`) and +host-side wallet), `host-env` (host-provisioned `/etc/decdn/decdn.env`), `slow-readiness` (advisory `/metrics` probe timeout), `grafana-cloud` (the opt-in -observability wiring — see [Grafana Cloud observability](#grafana-cloud-observability-opt-in)). +observability wiring — see [Grafana Cloud observability](#grafana-cloud-observability-opt-in)) +and `grafana-cloud-token` (the same wiring with the API token carried through +git-ignored inventory instead: role-authored env file + provenance record). They are independent, so they run concurrently — ~151s instead of ~595s — and each line of output is prefixed with its scenario name because the runs interleave. `make molecule-serial` is the escape hatch when that @@ -301,7 +310,8 @@ the RPC URL when it is not provisioned on the host instead). Highlights: | `decdn_rpc_url` + 3 contract addresses | `""` | **required** per node — `rpc_url` from a host-provisioned `0600 /etc/decdn/decdn.env` (preferred) *or* `host_vars//secret.yml`, addresses in `main.yml`; sourced from an ADR/deployment. | | `decdn_region` / `decdn_bind_port` / `decdn_rate_per_mb` | `""` / `4433` / `10` | node identity, QUIC port, USDC base units/MB. | | `decdn_env_checksum_file` / `decdn_env_overwrite_host_file` | `/etc/decdn/.decdn.env.sha256` / `false` | Provenance record for the secret env file (`0600 root`), and the opt-in that lets an inventory `decdn_rpc_url` overwrite a host-edited one. | -| `decdn_grafana_cloud_enabled` | `false` | ONE mirrored knob (identical default in both roles) wiring on Grafana Cloud observability: installs + configures `grafana_alloy` — node metrics, machine metrics, journald, agent health — AND injects `otlp_endpoint` into `node.toml`. Only the API token is provisioned per host; the rest are inventory variables. Label/cost guardrails in the role README. | +| `decdn_grafana_cloud_enabled` | `false` | ONE mirrored knob (identical default in both roles) wiring on Grafana Cloud observability: installs + configures `grafana_alloy` — node metrics, machine metrics, journald, agent health — AND injects `otlp_endpoint` into `node.toml`. Only the API token is provisioned per host *or* carried by `grafana_alloy_api_token` in git-ignored inventory; the rest are inventory variables. Label/cost guardrails in the role README. | +| `grafana_alloy_api_token` / `_env_checksum_file` / `_overwrite_host_file` | `""` / `/etc/grafana-alloy.env.sha256` / `false` | The dual-homed Grafana Cloud token and its provenance machinery (`#39` parity with the row above); see [`roles/grafana_alloy/README.md`](roles/grafana_alloy/README.md). | --- diff --git a/ansible/inventory/host_vars/decdn-node-1/secret.yml.example b/ansible/inventory/host_vars/decdn-node-1/secret.yml.example index edf08b0..5593255 100644 --- a/ansible/inventory/host_vars/decdn-node-1/secret.yml.example +++ b/ansible/inventory/host_vars/decdn-node-1/secret.yml.example @@ -30,3 +30,13 @@ decdn_rpc_url: "https://sepolia-rollup.arbitrum.io/rpc" # AWS_SESSION_TOKEN: "..." # only for assume-role / SSO flows # # Pair that with decdn_cache_origin_s3_use_default_chain: true in main.yml. + +# --- Grafana Cloud (only when decdn_grafana_cloud_enabled is on) --------------- +# +# The opt-in observability agent accepts its API token through the same channel: +# set grafana_alloy_api_token here and the role authors /etc/grafana-alloy.env on +# the host for you (token-only, root 0600), skipping the hand-provisioning shown +# in roles/grafana_alloy/files/grafana-alloy.env.example. SENSITIVE — leave it +# commented out to keep the operator-provisioned host path instead. +# +# grafana_alloy_api_token: "glc_example_token_replace_me" diff --git a/ansible/molecule/grafana-cloud-token/converge.yml b/ansible/molecule/grafana-cloud-token/converge.yml new file mode 100644 index 0000000..5fe0595 --- /dev/null +++ b/ansible/molecule/grafana-cloud-token/converge.yml @@ -0,0 +1,35 @@ +--- +# Token-via-inventory converge: every non-secret connection setting comes from +# this playbook's vars (the normal committed inventory half) and the token rides +# grafana_alloy_api_token — in a real deployment it would live in git-ignored +# inventory/host_vars//secret.yml, exactly like decdn_rpc_url. Nothing +# stages /etc/grafana-alloy.env; the role must author it itself. +# +# Logs stay enabled with BOTH Loki knobs supplied from inventory so the authored +# file needs zero GC_* keys beyond the token — the all-or-nothing contract has +# nothing to clobber on these throwaway containers. +- name: Converge + hosts: all + become: true + vars: + stub_bin: "{{ lookup('ansible.builtin.env', 'MOLECULE_SCENARIO_DIRECTORY') }}/files/alloy-stub" + decdn_grafana_cloud_enabled: true + grafana_alloy_install_method: manual + grafana_alloy_manual_bin_src: "{{ stub_bin }}" + grafana_alloy_deployment_environment: production + grafana_alloy_region: US + + # --- The point of this scenario ------------------------------------------------- + grafana_alloy_api_token: molecule-test-token + + # All six non-secret settings satisfied by inventory, so preflight requires no + # env-file key but GC_API_TOKEN... which the variable now covers too. + grafana_alloy_prom_url: https://prometheus-prod-xx.molecule.invalid/api/prom/push + grafana_alloy_prom_username: "1234567" + grafana_alloy_otlp_endpoint: https://otlp-gateway-prod-xx.molecule.invalid/otlp + grafana_alloy_loki_url: https://logs-prod-xx.molecule.invalid/loki/api/v1/push + grafana_alloy_loki_username: "7654321" + + roles: + - role: grafana_alloy + tags: [observability] diff --git a/ansible/molecule/grafana-cloud-token/files/alloy-stub b/ansible/molecule/grafana-cloud-token/files/alloy-stub new file mode 100755 index 0000000..58629ec --- /dev/null +++ b/ansible/molecule/grafana-cloud-token/files/alloy-stub @@ -0,0 +1,31 @@ +#!/bin/sh +# Molecule stub standing in for the Grafana Alloy binary in CI. +# +# The grafana-cloud scenario runs the real role logic (user/group/dirs, secret +# gate, template rendering, hardened unit, systemd start) against this shim so no +# ~160 MB release download is needed per CI run. It intentionally does NOT listen +# on any socket: runtime socket assertions are out of scope for containerised CI +# (see ../verify.yml comments) and remain covered by real-host deploys + content- +# level checks here. +set -u + +case "${1-}" in + # Version backstop in tasks/install.yml (manual mode degrades to rc==0 anyway). + --version) + echo "alloy, version v${MOLECULE_STUB_VERSION:-0.0.0}-molecule-stub" + ;; + # validate: the real binary builds the component graph and rejects an + # unloadable config. A stub CANNOT do that, so this exits 0 and proves the + # role's gate is wired up — nothing more. Content correctness is asserted + # textually in verify.yml and, against the REAL pinned binary, by + # ../../tests/alloy-config/validate.sh (`make lint-alloy`, run in CI). Do not + # read a green run here as "the configuration loads". + validate | fmt) + : + ;; + # Long-running foreground process keeps the Type=simple unit 'running'. + run) + exec sleep infinity + ;; +esac +exit 0 diff --git a/ansible/molecule/grafana-cloud-token/molecule.yml b/ansible/molecule/grafana-cloud-token/molecule.yml new file mode 100644 index 0000000..2cb78f1 --- /dev/null +++ b/ansible/molecule/grafana-cloud-token/molecule.yml @@ -0,0 +1,43 @@ +--- +# Inventory-path exercise of the #39 relaxation: grafana_alloy_api_token set in +# (git-ignored) inventory instead of a host-provisioned /etc/grafana-alloy.env. +# Positive path only — the broken-input matrix lives in ../validation; the +# host-provisioned positive path lives in ../grafana-cloud. No prepare-stage env +# file AT ALL here by design: the absence is what proves preflight's existence +# gate honours the token variable, and that the role authors the file itself. +# +# Same two boundaries as ../grafana-cloud: the stub binds no sockets and exits 0 +# for every subcommand, so nothing here proves runtime loadability (`make +# lint-alloy` does that against the real pinned binary). +driver: + name: docker +platforms: + - name: decdn-node-grafana-cloud-token + # Same digest-pinned image as the other scenarios (repo convention). Re-resolve + # all of them together to bump. + image: geerlingguy/docker-debian12-ansible@sha256:4553092be2c00b1ffe580927b9ff03f3c3a0df32b7dd693a3eb02efb6c2b77b7 + pre_build_image: true + command: /usr/lib/systemd/systemd + privileged: true + cgroupns_mode: host + volumes: + - /sys/fs/cgroup:/sys/fs/cgroup:rw +provisioner: + name: ansible + env: + ANSIBLE_ROLES_PATH: "${MOLECULE_PROJECT_DIRECTORY}/roles" + ANSIBLE_COLLECTIONS_PATH: "${MOLECULE_PROJECT_DIRECTORY}/collections" + ANSIBLE_PIPELINING: "true" +verifier: + name: ansible +scenario: + test_sequence: + # Leading `destroy` so an orphaned container is never reused — see + # ../default/molecule.yml. No prepare playbook: unlike ../grafana-cloud there + # are NO host-provisioned fixtures to stage — their absence is the point. + - destroy + - create + - converge + - idempotence + - verify + - destroy diff --git a/ansible/molecule/grafana-cloud-token/verify.yml b/ansible/molecule/grafana-cloud-token/verify.yml new file mode 100644 index 0000000..f0075f3 --- /dev/null +++ b/ansible/molecule/grafana-cloud-token/verify.yml @@ -0,0 +1,179 @@ +--- +# Verify the inventory-path authoring: a role-authored token-only 0600 env file, +# its provenance record, no secret literals outside it, and a running agent. +# Content assertions are grep-shape only against values this repo already knew +# (the molecule token) — mirroring how the role itself treats the host path. +- name: Verify + hosts: all + become: true + tasks: + - name: Stat the artifacts the run must have produced + ansible.builtin.stat: + path: "{{ item.path }}" + # Must match the checksum algorithm the role records (stat defaults to sha1). + get_checksum: true + checksum_algorithm: sha256 + register: ga_files + # config.alloy is root:alloy by design (the agent user reads its own config); + # the credential file and its provenance record are strictly root:root. + loop: + - {path: /etc/grafana-alloy.env, mode: "0600", grp: root} + - {path: /etc/grafana-alloy.env.sha256, mode: "0600", grp: root} + - {path: /etc/alloy/config.alloy, mode: "0644", grp: alloy} + + - name: Assert ownership/modes of the artifacts + ansible.builtin.assert: + that: + # exists-and-mode first so a missing file short-circuits to fail_msg + # instead of raising on absent stat fields. + - item.stat.exists and item.stat.mode == item.item.mode + - item.stat.pw_name == 'root' + - item.stat.gr_name == item.item.grp + # Regular file: the authored credential must not be a symlink, same as + # preflight demands from operator-provisioned ones. + - item.stat.isreg | default(false) + loop: "{{ ga_files.results }}" + loop_control: + label: "{{ item.item.path }}" + + - name: Read the authored credential file back + ansible.builtin.slurp: + src: /etc/grafana-alloy.env + register: ga_env_b64 + + - name: Assert the env file is exactly one JSON-encoded token line + ansible.builtin.assert: + that: + # to_json always double-quotes a plain string; systemd strips the outer + # quotes when expanding EnvironmentFile=. + - >- + ga_lines | length == 1 + - >- + ga_lines[0] == 'GC_API_TOKEN="' ~ (ga_expected_token) ~ '"' + - >- + ga_body.count('GC_') == 1 + fail_msg: >- + The authored env file must be a single GC_API_TOKEN="" line. + Got: {{ ga_body }} + vars: + ga_expected_token: molecule-test-token + ga_body: "{{ ga_env_b64.content | b64decode }}" + ga_lines: "{{ (ga_env_b64.content | b64decode).splitlines() }}" + + - name: Read the provenance record back + ansible.builtin.slurp: + src: /etc/grafana-alloy.env.sha256 + register: ga_record_b64 + + - name: Resolve the record's two fields + ansible.builtin.set_fact: + ga_record_fields: "{{ (ga_record_b64.content | b64decode | trim).split() }}" + + - name: Assert the record says inventory-authored and matches the live file + ansible.builtin.assert: + that: + - ga_record_fields | length == 2 + - ga_record_fields[0] == 'inventory' + - ga_record_fields[1] == (ga_files.results[0].stat.checksum) + fail_msg: >- + The provenance record must read " ". + Got: {{ ga_record_fields }} + + # --- Secret hygiene -------------------------------------------------------------- + - name: Register the secret-token absence across non-secret files + ansible.builtin.command: + cmd: grep -Rq molecule-test-token {{ item }} + changed_when: false + failed_when: false + loop: + - /etc/systemd/system/alloy.service + - /etc/alloy/config.alloy + register: ga_leak_probe + + - name: Assert no secret literal leaked into non-secret artifacts + ansible.builtin.assert: + that: + - ga_leak_probe.results | selectattr('rc', 'equalto', 0) | list | length == 0 + fail_msg: >- + The GC_API_TOKEN literal appears outside the 0600 env file — config or + unit rendering stopped using sys.env() indirection. + + - name: Gather service facts + ansible.builtin.service_facts: + + - name: Assert alloy picked up the authored credentials (running, not crash-looping) + ansible.builtin.assert: + that: + - "'alloy.service' in ansible_facts.services" + - ansible_facts.services['alloy.service'].state == 'running' + +# Mutates the converged host deliberately (like ../grafana-cloud's disable checks), +# so it runs AFTER every assertion above: rotating the token must update both the +# authored file and the provenance record atomically-ish, and leave the agent up. +- name: Rotate + hosts: all + become: true + tasks: + - name: Snapshot the pre-rotation record + ansible.builtin.slurp: + src: /etc/grafana-alloy.env.sha256 + register: ga_record_pre + + - name: Re-run the role with a rotated token + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_install_method: manual + grafana_alloy_manual_bin_src: >- + {{ lookup('ansible.builtin.env', 'MOLECULE_SCENARIO_DIRECTORY') }}/files/alloy-stub + grafana_alloy_deployment_environment: production + grafana_alloy_region: US + grafana_alloy_api_token: molecule-rotated-token + grafana_alloy_prom_url: https://prometheus-prod-xx.molecule.invalid/api/prom/push + grafana_alloy_prom_username: "1234567" + grafana_alloy_otlp_endpoint: https://otlp-gateway-prod-xx.molecule.invalid/otlp + grafana_alloy_loki_url: https://logs-prod-xx.molecule.invalid/loki/api/v1/push + grafana_alloy_loki_username: "7654321" + + - name: Read the rotated artifacts + ansible.builtin.slurp: + src: "{{ item }}" + register: ga_rotated + loop: + - /etc/grafana-alloy.env + - /etc/grafana-alloy.env.sha256 + + - name: Stat the rotated credential file for its fresh checksum + ansible.builtin.stat: + path: /etc/grafana-alloy.env + get_checksum: true + checksum_algorithm: sha256 + register: ga_rotated_stat + + - name: Assert rotation landed in both the file and the record + ansible.builtin.assert: + that: + # Untracked-before-warn semantics do not apply here: the record EXISTS + # (seeded by the converge), source='inventory' and checksum matched, so + # no collision assert can fire. Verify content + provenance both moved. + - (ga_rotated.results[0].content | b64decode) is search('molecule-rotated-token') + - (ga_rotated.results[0].content | b64decode) is not search('molecule-test-token') + - ((ga_rotated.results[1].content | b64decode | trim).split()[0]) == 'inventory' + - ((ga_rotated.results[1].content | b64decode | trim).split()[1]) + == (ga_rotated_stat.stat.checksum) + - ((ga_rotated.results[1].content | b64decode | trim).split()[1]) + != (ga_record_pre.content | b64decode | trim).split()[1] + fail_msg: >- + Token rotation did not propagate: either the env file kept the old + value, or the provenance record does not match the freshly authored + file (a stale record would trip the adoption guard next converge). + + - name: Gather service facts after rotation + ansible.builtin.service_facts: + + - name: Assert alloy restarted onto the rotated credential + ansible.builtin.assert: + that: + - "'alloy.service' in ansible_facts.services" + - ansible_facts.services['alloy.service'].state == 'running' diff --git a/ansible/molecule/validation/converge.yml b/ansible/molecule/validation/converge.yml index 495c691..3c31c3a 100644 --- a/ansible/molecule/validation/converge.yml +++ b/ansible/molecule/validation/converge.yml @@ -646,7 +646,7 @@ - name: Record ga-no-secret rejection (only if the secret-file assert failed) ansible.builtin.set_fact: decdn_rejected: "{{ decdn_rejected + ['secret-file'] }}" - when: ansible_failed_task.name is match('^Assert the secret file') + when: ansible_failed_task.name is match('^Require the Grafana Cloud secret environment file') - name: "Case ga-symlink-secret — symlinked credential file" block: @@ -1127,6 +1127,70 @@ path: /etc/grafana-alloy.env state: absent + # The #39 relaxation lets the token ride inventory, but only as a bare, + # whitespace-free value — it becomes a single systemd EnvironmentFile line. + - name: "Case ga-token-var-shape — whitespace inside grafana_alloy_api_token" + block: + - name: Run grafana_alloy with a whitespace-carrying API token variable + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_api_token: "glc_molecule spaced" + rescue: + - name: Record ga-token-var-shape rejection (only if the token-shape gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['token-shape'] }}" + when: ansible_failed_task.name is match('^Validate the shape of an inventory-provided API token') + + # A tracked host-authored env file must not be silently clobbered by an + # inventory token; overwriting requires the explicit opt-in knob. + - name: "Case ga-collision — inventory token aimed at an untracked host env file" + block: + - name: Stage a host-style credential fixture + ansible.builtin.copy: + dest: /etc/grafana-alloy.env + owner: root + group: root + mode: "0600" + content: | + GC_PROM_REMOTE_WRITE_URL=https://prometheus-prod-xx.molecule.invalid/api/prom/push + GC_OTLP_ENDPOINT=https://otlp-gateway-prod-xx.molecule.invalid/otlp + GC_PROM_USERNAME=999888777 + GC_API_TOKEN=molecule-test-token + + # Seed a KNOWN-foreign provenance record: source says 'inventory' but the + # checksum no longer matches the staged file, which is exactly the + # role-didn't-last-write-this state the overwrite knob exists for. + - name: Seed a stale provenance record for the fixture + ansible.builtin.copy: + dest: /etc/grafana-alloy.env.sha256 + owner: root + group: root + mode: "0600" + content: >- + inventory 0000000000000000000000000000000000000000000000000000000000000000 + + - name: Run grafana_alloy with an inventory token against the foreign fixture + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_api_token: molecule-test-token + rescue: + - name: Record ga-collision rejection (only if the overwrite gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['token-collision'] }}" + when: ansible_failed_task.name is match('^Require confirmation to overwrite an env file') + always: + - name: Remove the collision fixtures + ansible.builtin.file: + path: "{{ item }}" + state: absent + loop: + - /etc/grafana-alloy.env + - /etc/grafana-alloy.env.sha256 + - name: Confirm every bad value was rejected by the role's own validation ansible.builtin.assert: that: @@ -1152,5 +1216,5 @@ "alloy-root-user-alias", "alloy-root-group-alias", "host-collectors", "host-collectors", "host-collectors", "endpoint-scheme", "inventory-token", "inventory-token", "instance-id", - "otlp-username-env", + "otlp-username-env", "token-shape", "token-collision", "managed-paths", "managed-paths", "managed-paths"] diff --git a/ansible/roles/grafana_alloy/README.md b/ansible/roles/grafana_alloy/README.md index 479d6c5..bb5e9f3 100644 --- a/ansible/roles/grafana_alloy/README.md +++ b/ansible/roles/grafana_alloy/README.md @@ -76,6 +76,43 @@ matching `GC_…` key in the env file exactly as before, and preflight requires key only then. Any mix of the two halves is valid. `roles/grafana_alloy/files/grafana-alloy.env.example` shows the full key set. +### The API token has two homes now + +The token may also ride **git-ignored inventory**, mirroring how `decdn_rpc_url` +works: + +```yaml +# host_vars//secret.yml — git-ignored; see secret.yml.example +grafana_alloy_api_token: "glc_…" +``` + +With it set, the role authors `/etc/grafana-alloy.env` itself — as a **token-only** +file (`GC_API_TOKEN="…"` at `root:root 0600`, JSON-encoded so quotes/backslashes +survive), rewritten wholesale on every converge. Runtime behaviour is identical: +`config.alloy` still reads `sys.env("GC_API_TOKEN")`; only who fills the file +changes. Rotation is an inventory edit + deploy instead of per-host shell work, +and provenance tracking (a ` ` record next to the file) catches +the classic failure modes loud: + +- **Token vanished from the control machine** while a role-authored file exists + → the run refuses to adopt its own file as operator-provisioned ("restore the + variable or discard the record explicitly"). +- **Untracked hand-edited file** meeting an inventory token → refused unless + `grafana_alloy_overwrite_host_file: true`. + +The authoring is all-or-nothing: any hand-added `GC_*` lines beyond the token are +DISCARDED by a rewrite — migrate them to their inventory variables first +(preflight's per-key gates then verify nothing was lost). An out-of-band edit on +the host-provisioned path restarts the agent with a note, same as `decdn_node`. +Returning a host to hand-provisioned mode: clear the variable, then discard the +provenance record (`sudo rm /etc/grafana-alloy.env.sha256`). The Helm chart is +unaffected (it was Secret-oriented from the start). + +`GC_API_TOKEN` values in plain committed inventory remain rejected: the rendered +`/etc/alloy/config.alloy` is world-readable, and preflight rejects a value that +looks like a token (or a URL with embedded credentials) in any of the six +variables above. + > **Upgrading an existing host.** Logs are on by default, and Loki needs an > endpoint + instance ID that the pre-existing four-key env file does not have. > Preflight fails loudly, naming both the missing key and the inventory variable @@ -99,11 +136,6 @@ key only then. Any mix of the two halves is valid. > present, preflight checks its shape, because a malformed value outranks the > Prometheus fallback at runtime and 401s traces just the same. -`GC_API_TOKEN` has **no** inventory variable by design: the rendered -`/etc/alloy/config.alloy` is world-readable, and preflight rejects a value that -looks like a token (or a URL with embedded credentials) in any of the six -variables above. - The credential path is intentionally restricted to a **direct child of `/etc`**. A nested override (including the old `/etc/decdn/grafana-alloy.env`) is rejected: write access to any ancestor is enough to replace a root-owned `0600` file. Both @@ -200,6 +232,9 @@ upstream version/sha256). Highlights: | `grafana_alloy_logs_max_age` | `12h` | Bounds the catch-up burst after an outage | | `grafana_alloy_self_metrics_enabled` | `true` | Alloy's own health | | `grafana_alloy_prom_url` / `_prom_username` / `_otlp_endpoint` / `_otlp_username` / `_loki_url` / `_loki_username` | `""` | Non-secret connection settings; empty ⇒ read the matching `GC_…` env key (`_otlp_username` then falls back to the Prometheus ID) | +| `grafana_alloy_api_token` | `""` | **SENSITIVE** — the API token (git-ignored `secret.yml`). Empty ⇒ operator-provisioned `/etc/grafana-alloy.env`; set ⇒ the role authors that file token-only, root 0600. See "The API token has two homes now". | +| `grafana_alloy_env_checksum_file` | `/etc/grafana-alloy.env.sha256` | Provenance record (` `) enabling the adoption/overwrite guards below | +| `grafana_alloy_overwrite_host_file` | `false` | Explicit opt-in letting an inventory token rewrite a hand-edited/untracked env file (all-or-nothing: extra `GC_*` lines are discarded) | | `grafana_alloy_node_job` / `_host_job` / `_self_job` / `_logs_job` | see table above | Job labels; the `integrations/…` ones are what Grafana Cloud's dashboards match | Label variables (`service_name`, `service_namespace`, `instance_id`, diff --git a/ansible/roles/grafana_alloy/defaults/main.yml b/ansible/roles/grafana_alloy/defaults/main.yml index 51b2ecd..39bbaac 100644 --- a/ansible/roles/grafana_alloy/defaults/main.yml +++ b/ansible/roles/grafana_alloy/defaults/main.yml @@ -37,6 +37,36 @@ grafana_alloy_state_dir: /var/lib/alloy # WAL/storage state, owned by the a # swap the root-owned EnvironmentFile before systemd reads it. grafana_alloy_secret_file: /etc/grafana-alloy.env +# SENSITIVE — the Grafana Cloud access-policy/service-account token. EMPTY (the +# default) keeps the host-provisioned posture: preflight requires a 0600 env file +# on the target host and the value never transits the control machine. SETTING it +# opts into the inventory path (#39 parity with decdn_rpc_url): this role then +# authors {{ grafana_alloy_secret_file }} as a TOKEN-ONLY EnvironmentFile that is +# rewritten wholesale on every converge, so any hand-added GC_* keys must migrate +# to their inventory variables first (all-or-nothing authoring). Whether the value +# arrives via this variable or was provisioned by hand, config.alloy reads it the +# same way at runtime: sys.env("GC_API_TOKEN"). +grafana_alloy_api_token: "" + +# Provenance tracking for grafana_alloy_secret_file, mirroring roles/decdn_node's +# #39 machinery. After every enabled converge the role records " " +# of the env file the service is now running on ("inventory" when +# grafana_alloy_api_token authored it, "host" otherwise) into a root-owned 0600 +# record NEXT TO the secret file — deliberately NOT inside /etc/alloy or +# /var/lib/alloy, because the disable path removes those directories wholesale. +# Preflight uses the pair to fail loud on (a) adopting a role-authored file after +# the secret went missing from the control machine and (b) clobbering an operator- +# edited file without grafana_alloy_overwrite_host_file. +grafana_alloy_env_checksum_file: /etc/grafana-alloy.env.sha256 + +# Explicit opt-in letting grafana_alloy_api_token rewrite an env file whose +# provenance record is missing or mismatched — i.e. something this role did not +# last write. Identical contract to decdn_env_overwrite_host_file: because the +# rewrite is token-only, any lines added outside Ansible are DISCARDED; migrate +# them to their inventory variables BEFORE turning this on (preflight then stops +# demanding them from the file, which also makes the migration verifiable). +grafana_alloy_overwrite_host_file: false + # --- Network (loopback-only; AGENTS.md hard rule 2) --------------------------- # OTLP receivers bound to loopback ONLY — the daemon accepts telemetry pushed # over localhost, never from the network; no firewall hole exists or is needed. @@ -54,16 +84,21 @@ grafana_alloy_server_http_addr: 127.0.0.1:12345 # --- Grafana Cloud endpoints (NON-SECRET; optional inventory overrides) ------- # Only the API TOKEN is a secret. The endpoint URLs and the two numeric instance # IDs are not, so they may be set here (group_vars/host_vars) instead of being -# hand-typed into the root-owned env file on every host — which leaves -# GC_API_TOKEN as the single value an operator must provision per host. +# hand-typed into the root-owned env file on every host — leaving GC_API_TOKEN +# as the only credential, itself dually homeable: operator-provisioned per host, +# or carried by grafana_alloy_api_token from git-ignored inventory (see below). # # Each of these is optional and falls back, when left EMPTY, to the matching # sys.env("GC_…") lookup the role has always used, so an existing deployment that # sets none of them renders byte-identical configuration. Setting one makes its # env key unnecessary, and preflight stops requiring that key. # -# NEVER put the token here: config.alloy is world-readable 0644 on the host, and -# inventory is committed. GC_API_TOKEN has no variable by design. +# NEVER put a credential in THIS template: config.alloy is world-readable 0644 on +# the host, and inventory is committed. The API token is the ONE value allowed to +# ride through inventory — via grafana_alloy_api_token below, which is SENSITIVE +# and belongs in a git-ignored secret.yml exactly like decdn_rpc_url (#39). Setting +# it makes this role author /etc/grafana-alloy.env itself; leaving it "" keeps the +# original operator-provisioned posture. See README.md §Credentials. grafana_alloy_prom_url: "" # else sys.env("GC_PROM_REMOTE_WRITE_URL") grafana_alloy_prom_username: "" # else sys.env("GC_PROM_USERNAME") grafana_alloy_otlp_endpoint: "" # else sys.env("GC_OTLP_ENDPOINT") diff --git a/ansible/roles/grafana_alloy/files/grafana-alloy.env.example b/ansible/roles/grafana_alloy/files/grafana-alloy.env.example index 24b0d81..9515a2d 100644 --- a/ansible/roles/grafana_alloy/files/grafana-alloy.env.example +++ b/ansible/roles/grafana_alloy/files/grafana-alloy.env.example @@ -24,8 +24,10 @@ # reads values back onto the control machine. # REQUIRED, always. An access-policy token with metrics:write + logs:write + -# traces:write scope for this stack. This is the one value with no inventory -# variable, deliberately: the rendered /etc/alloy/config.alloy is world-readable. +# traces:write scope for this stack. The INVENTORY path can author this file for +# you instead (grafana_alloy_api_token in git-ignored host_vars secret.yml) — in +# that case DO NOT create it by hand; the role writes a token-only version of it, +# and lines added here would be discarded on the next converge. GC_API_TOKEN=glc_example_token_replace_me # --- Optional: only needed if the matching inventory variable is left empty ---- diff --git a/ansible/roles/grafana_alloy/tasks/main.yml b/ansible/roles/grafana_alloy/tasks/main.yml index c9b1a47..d3e0b86 100644 --- a/ansible/roles/grafana_alloy/tasks/main.yml +++ b/ansible/roles/grafana_alloy/tasks/main.yml @@ -114,6 +114,10 @@ - "{{ _ga_unit_path }}" - "{{ grafana_alloy_config_dir }}" - "{{ grafana_alloy_state_dir }}" + # Provenance metadata is ours whenever the unit is (the marker gates + # this whole block); dropping it makes a disable→enable cycle re-seed + # cleanly instead of carrying a stale checksum. + - "{{ grafana_alloy_env_checksum_file }}" notify: Reload systemd - name: Apply pending teardown notifications @@ -263,6 +267,55 @@ - name: Install the Alloy binary ansible.builtin.import_tasks: install.yml + # --- Secret environment file -------------------------------------------------- + # Two authoring paths for {{ grafana_alloy_secret_file }}, decided by the + # provenance gates in preflight exactly like decdn_node's #39: + # * grafana_alloy_api_token set -> the copy below REWRITES the file from + # control-machine values, token-only. + # * empty -> the operator wrote the file on the host. The role never reads its + # content back (that would put the secret on the control machine, which is + # the whole point of the host path); it only enforces root 0600 and greps + # shapes in preflight. + - name: Write the token-only secret environment file (inventory-supplied) + ansible.builtin.copy: + dest: "{{ grafana_alloy_secret_file }}" + owner: root + group: root + mode: "0600" + # to_json rather than hand-rolled escaping: it emits exactly the double- + # quoted, C-escaped form systemd's EnvironmentFile parser expects (and + # supplies the quotes itself), so a value containing quotes/backslashes + # survives intact. ensure_ascii=false keeps UTF-8 literal. Whitespace was + # rejected by preflight's shape gate; no_log because the token is secret. + content: | + GC_API_TOKEN={{ grafana_alloy_api_token | string | to_json(ensure_ascii=false) }} + no_log: true + register: _ga_env_copy + notify: Restart alloy + when: grafana_alloy_api_token | length > 0 + + # The copy above is no_log, so a run that replaced an existing file shows a bare + # "changed" and nothing else. Say what was lost — but only for the UNKNOWABLE + # case: a file whose known-foreign provenance already failed preflight loudly, + # so what remains here is the first converge after a pre-provenance deploy. + # Gated on the task result too, so it speaks up once (when contents moved), + # not on every subsequent converge. + - name: Warn that an inventory grafana_alloy_api_token overwrote an untracked env file + ansible.builtin.debug: + msg: >- + grafana_alloy_api_token is set in inventory, so + {{ grafana_alloy_secret_file }} was REWRITTEN as a token-only file, and + this host carried no provenance record for it — any KEY=value lines added + on the host that are not covered by inventory variables are gone. From + this converge on, that file is tracked: an edit made outside Ansible fails + the next deploy instead of being silently discarded. Clear + grafana_alloy_api_token to hand the file back to the host. + when: + - grafana_alloy_api_token | length > 0 + - _ga_secret_stat.stat.exists + - _ga_recorded_sum | length == 0 + - _ga_env_copy.changed | default(false) | bool + # --- Configuration + hardened unit ------------------------------------------ - name: Render the Alloy pipeline configuration ansible.builtin.template: @@ -313,6 +366,39 @@ - not ansible_check_mode - _ga_config_check.rc | default(1) != 0 + # Restart on an OUT-OF-BAND edit to a HOST-provisioned env file. The copy above + # already notifies when the role itself authors a changed file, so this covers + # only the host path, where nothing else would notice that the operator edited + # /etc/grafana-alloy.env between converges — the agent would keep running on + # the previous credentials behind a fully green deploy. + # + # An ABSENT record is deliberately not a change: it means this host has never + # run a provenance-tracked converge, and treating "not tracked yet" as "edited" + # would bounce every node in the fleet on the upgrade run for no functional + # reason. Handlers dedupe, and the flush below executes this BEFORE start and + # before the record, so the record always trails the restart it owes. + - name: Restat the secret environment file before recording its checksum + ansible.builtin.stat: + path: "{{ grafana_alloy_secret_file }}" + follow: false + # sha256 so the .sha256 record cross-checks against `sha256sum`; stat's + # default is sha1 and would make every documented verification lie. + get_checksum: true + checksum_algorithm: sha256 + register: _ga_env_post_stat + + - name: Note an out-of-band change to the host-provisioned env file + ansible.builtin.debug: + msg: >- + {{ grafana_alloy_secret_file }} has changed since the last converge + recorded it — restarting alloy so the agent picks the new credentials up. + changed_when: true + notify: Restart alloy + when: + - grafana_alloy_api_token | length == 0 + - _ga_recorded_sum | length > 0 + - (_ga_env_post_stat.stat.checksum | default('')) != _ga_recorded_sum + - name: Apply pending Alloy changes ansible.builtin.meta: flush_handlers @@ -336,3 +422,30 @@ - ansible_facts.services['alloy.service'].state == 'running' fail_msg: "alloy is not running — check: journalctl -u alloy -e" when: not ansible_check_mode + + # Record the env file's hash LAST — deliberately after the handler flush and + # start above, so the record means "the agent is running on this content", not + # "Ansible saw this content". Writing it earlier looks equivalent and is not: + # the handlers acting on detected changes do not flush until after the config + # template and the `alloy validate` gate, both of which can abort the play; a + # record written before that abort would say "already handled" on the NEXT + # converge, silently burning the one signal a secret rotation gives. + # + # No notify: by the time this runs, any restart this converge owed — including + # the out-of-band-edit one queued just above — has already been flushed. + - name: Record the env-file checksum (the agent is now running on it) + ansible.builtin.copy: + dest: "{{ grafana_alloy_env_checksum_file }}" + owner: root + group: root + mode: "0600" + # " " — the source token is what a later converge uses to + # tell "the operator owns this file" from "I rendered this from inventory". + # Parenthesised: filters bind tighter than comparisons, so an unbracketed + # ternary parses wrong. + content: > + {{ (grafana_alloy_api_token | length > 0) | ternary('inventory', 'host') }} + {{ _ga_env_post_stat.stat.checksum }} + # Under `--check` on an unprovisioned host the env-file copy only SIMULATES + # the write, so there is nothing to hash and stat.checksum is undefined. + when: (_ga_env_post_stat.stat.checksum | default('')) | length > 0 diff --git a/ansible/roles/grafana_alloy/tasks/preflight.yml b/ansible/roles/grafana_alloy/tasks/preflight.yml index f852de5..ef71e96 100644 --- a/ansible/roles/grafana_alloy/tasks/preflight.yml +++ b/ansible/roles/grafana_alloy/tasks/preflight.yml @@ -1,9 +1,11 @@ --- # Fail-loud preflight, run BEFORE any host mutation. Mirrors decdn_node's -# secret-file posture: this role never carries the API TOKEN in inventory and -# never reads the env file's values onto the control machine — it only greps -# shapes on the host itself. The non-secret half of the connection settings -# (endpoint URLs, instance IDs) MAY come from inventory; those are validated here +# secret-file posture: this role keeps the API token OFF inventory-committed, +# world-readable surfaces and never reads the env file's values onto the control +# machine — it only greps shapes on the host itself. The token MAY now ride a +# git-ignored secret.yml via grafana_alloy_api_token (the #39 relaxation); the +# non-secret half of the connection settings +# (endpoint URLs, instance IDs) MAY come from plain committed inventory; those are validated here # too, and each one that is set removes its key from the host-state gates below. # # Ordering matters for testability (molecule/validation): PURE-INVENTORY gates @@ -177,7 +179,7 @@ # These five are NON-secret and therefore inventory-settable; each one that is set # replaces its sys.env("GC_…") lookup in the rendered config and removes the # corresponding requirement from the host-state gates at the bottom of this file. -# The API token has no variable by design: config.alloy is 0644 on the host. +# The API token is now dually homeable too (grafana_alloy_api_token below). - name: Validate any inventory-provided Grafana Cloud endpoints ansible.builtin.assert: @@ -240,6 +242,26 @@ loop_control: label: "{{ item | truncate(24, true) }}" +# The mirror image of the guard above (and the #39 relaxation): when +# grafana_alloy_api_token is SET it may be whitespace/newline-free, because the +# role JSON-encodes it into a single systemd EnvironmentFile line — quotes and +# backslashes are deliberately NOT rejected here, to_json emits exactly the +# double-quoted C-escaped form systemd's parser expects. A value with embedded +# whitespace would arrive on the daemon side truncated at the first blank. +- name: Validate the shape of an inventory-provided API token + ansible.builtin.assert: + that: + - grafana_alloy_api_token | string is match('^[^\s]+$') + quiet: true + fail_msg: >- + grafana_alloy_api_token must be the bare token value (non-empty, no + whitespace, no newline) — it is written to {{ grafana_alloy_secret_file }} + as GC_API_TOKEN="{{ grafana_alloy_api_token | truncate(24, true) }}" in a + single line systemd parses before dropping privileges. Quotes/backslashes are + fine (JSON-encoded); spaces are not. + # Empty keeps the host-provisioned posture; the gate only guards the inventory path. + when: grafana_alloy_api_token | length > 0 + - name: Validate the loopback listener addresses ansible.builtin.assert: that: @@ -309,8 +331,34 @@ ansible.builtin.stat: path: "{{ grafana_alloy_secret_file }}" follow: false + # sha256, matching what the record task writes into .sha256 — with stat's + # default (sha1) an operator cross-checking `sha256sum` would see a mismatch. + get_checksum: true + checksum_algorithm: sha256 register: _ga_secret_stat +- name: Require the Grafana Cloud secret environment file to exist + ansible.builtin.assert: + that: + - _ga_secret_stat.stat.exists + fail_msg: >- + {{ grafana_alloy_secret_file }} is missing, and neither credential path is + taken: grafana_alloy_api_token is empty, so this role expects an + operator-provisioned env file. Provision it on the target host per + roles/grafana_alloy/files/grafana-alloy.env.example: + umask 077 + sudo install -m 600 -o root -g root grafana-alloy.env {{ grafana_alloy_secret_file }} + or set grafana_alloy_api_token in a git-ignored secret.yml and let this + role author the file (token-only) for you. + # The inventory path authors the file itself, so absence is fine there; only + # the host-provisioned posture demands an operator-created file up front. + when: grafana_alloy_api_token | length == 0 + +# Kept as its own gate even though absence was just handled above: with +# grafana_alloy_api_token set an ABSENT file is legitimate (the role authors it), +# but a PRESENT one must still be a real root-owned 0600 regular file — systemd +# reads EnvironmentFile= as root before dropping privileges. +# # stat.exists alone cannot see the difference between a regular file and a # symlink planted at the credential path that later passes a root chown/chmod # would harden whatever TARGET it points at — exactly what the secret-file gate @@ -318,21 +366,98 @@ - name: Assert the secret file is a real root-owned 0600 file ansible.builtin.assert: that: - - _ga_secret_stat.stat.exists - _ga_secret_stat.stat.isreg | default(false) - (_ga_secret_stat.stat.pw_name | default('')) == 'root' - (_ga_secret_stat.stat.gr_name | default('')) == 'root' - _ga_secret_stat.stat.mode | string == '0600' fail_msg: >- - {{ grafana_alloy_secret_file }} is missing, not a regular file (symlink? - directory?), or not owned root:root 0600 (found owner + {{ grafana_alloy_secret_file }} is not a regular file (symlink? directory?) + or not owned root:root 0600 (found owner {{ _ga_secret_stat.stat.pw_name | default('?') }}, mode - {{ _ga_secret_stat.stat.mode | default('?') }}). Provision it on the target - host per roles/grafana_alloy/files/grafana-alloy.env.example: - umask 077 - sudo install -m 600 -o root -g root grafana-alloy.env {{ grafana_alloy_secret_file }} + {{ _ga_secret_stat.stat.mode | default('?') }}). Fix it on the target host; systemd expands its KEY=value pairs into the unit before dropping privileges, so root ownership is sufficient — do not relax it. + when: _ga_secret_stat.stat.exists + +# --- Secret-file provenance ------------------------------------------------------- +# Mirrors decdn_node's #39 machinery (roles/decdn_node/tasks/main.yml): the role +# records " " of the env file after every enabled converge, so the +# next run can tell three states apart. Only this non-sensitive record is ever +# read back — never the secret file itself. +# +# An absent/truncated record means "no post-relaxation converge yet" and is the +# normal state, which is why both asserts below stay quiet for it. +- name: Read the env-file checksum this role last recorded + ansible.builtin.slurp: + src: "{{ grafana_alloy_env_checksum_file }}" + register: _ga_recorded + failed_when: false + changed_when: false + +# Both fields default to "" — treated exactly like "no record", which also covers +# a truncated or older single-field record. +- name: Resolve the secret file's provenance + ansible.builtin.set_fact: + _ga_record_source: >- + {{ ((_ga_recorded.content | default('') | b64decode | trim).split() | length == 2) + | ternary((_ga_recorded.content | default('') | b64decode | trim).split()[0], '') }} + _ga_recorded_sum: >- + {{ ((_ga_recorded.content | default('') | b64decode | trim).split() | length == 2) + | ternary((_ga_recorded.content | default('') | b64decode | trim).split()[1], '') }} + +# The dangerous half of inferring operator intent. With grafana_alloy_api_token +# empty the operator-supplied file is adopted as-is — but that ALSO happens when +# the git-ignored secret.yml carrying the token went missing (fresh clone, new +# control machine, undecrypted vault) AFTER an earlier inventory-path deploy: the +# role would adopt its OWN previously-written file as host-provisioned and a +# ROTATED token in the missing secret.yml would silently never land. Fail loud +# instead, exactly like the decdn_node adoption guard. +- name: Refuse to adopt a role-authored env file as host-provisioned + ansible.builtin.assert: + that: + - _ga_record_source != 'inventory' + or _ga_recorded_sum != (_ga_secret_stat.stat.checksum | default('')) + fail_msg: >- + grafana_alloy_api_token is empty, but {{ grafana_alloy_secret_file }} is + byte-for-byte the file THIS ROLE last wrote from inventory — so the token + has gone missing from the control machine (an absent + host_vars//secret.yml, a fresh clone, an undecrypted vault) rather + than being deliberately handed to the host. Treating it as + host-provisioned would deploy green while silently keeping the old token. + Restore grafana_alloy_api_token, or — if you really are migrating this node + to the host-provisioned path — hand the file over explicitly by discarding + the role's provenance record: + sudo rm {{ grafana_alloy_env_checksum_file }} + when: + - grafana_alloy_api_token | length == 0 + - _ga_secret_stat.stat.exists + +# The mirror direction: an inventory token REWRITES the file wholesale (and +# token-only), losing any GC_* lines added outside Ansible. Scoped to a KNOWN +# foreign provenance — with no record this cannot distinguish anything, so it +# stays quiet and the run seeds the record for next time (the warn-once task in +# tasks/main.yml covers the unknowable case). +- name: Require confirmation to overwrite an env file this role did not write + ansible.builtin.assert: + that: + - grafana_alloy_overwrite_host_file | bool + fail_msg: >- + grafana_alloy_api_token is set, but {{ grafana_alloy_secret_file }} on this + host is not a file this role authored — the provenance record reads + "{{ _ga_record_source }}" and/or its checksum no longer matches, so it was + written or edited by someone else. Authoring it token-only from inventory + would DISCARD those lines, and because this role never reads the file it + cannot tell you what would be lost. Fold any extra GC_* keys into their + matching inventory variables first, then either clear + grafana_alloy_api_token to keep the host's copy or set + grafana_alloy_overwrite_host_file: true to confirm the rewrite. + when: + - grafana_alloy_api_token | length > 0 + - _ga_secret_stat.stat.exists + - _ga_recorded_sum | length > 0 + - >- + _ga_record_source != 'inventory' + or _ga_recorded_sum != (_ga_secret_stat.stat.checksum | default('')) # systemd skips an EnvironmentFile line it cannot parse SILENTLY, so each required # key is grepped for presence + value shape directly on the host: grep keeps the @@ -341,8 +466,8 @@ # # "Required" is now conditional: a key is only needed when nothing in inventory # supplies that value, and the Loki pair is only needed when logs are enabled. -# GC_API_TOKEN is ALWAYS required — it is the one value that must never live in -# inventory. +# That includes GC_API_TOKEN itself since the grafana_alloy_api_token relaxation: +# the grep only fires when the variable is empty. - name: Confirm every required credential key is present and shaped correctly ansible.builtin.command: cmd: grep -qE -- '{{ item.regex }}' {{ grafana_alloy_secret_file | quote }} @@ -357,8 +482,8 @@ # already newline-free by construction. - key: GC_API_TOKEN regex: ^GC_API_TOKEN=[^[:space:]#][^[:space:]]*$ - var: "(none — the token must never be set in inventory)" - required: true + var: grafana_alloy_api_token + required: "{{ grafana_alloy_api_token | string | length == 0 }}" - key: GC_PROM_REMOTE_WRITE_URL regex: ^GC_PROM_REMOTE_WRITE_URL=https://[^[:space:]]+$ var: grafana_alloy_prom_url @@ -393,8 +518,7 @@ roles/grafana_alloy/files/grafana-alloy.env.example) — URLs must be https:// and every value non-empty, since systemd silently drops unparseable lines so a quoting typo looks identical to a missing key — or, - for everything except the token, set the matching inventory variable - instead: {{ _ga_missing | map(attribute='var') | join(', ') }}. + set the matching inventory variable instead: {{ _ga_missing | map(attribute='var') | join(', ') }}. vars: # Skipped loop items (key not required) carry no rc, so filter on rc being # defined first: rejectattr alone would count every skipped key as missing. diff --git a/ansible/roles/grafana_alloy/tasks/validate-paths.yml b/ansible/roles/grafana_alloy/tasks/validate-paths.yml index 0a324ee..9d5b133 100644 --- a/ansible/roles/grafana_alloy/tasks/validate-paths.yml +++ b/ansible/roles/grafana_alloy/tasks/validate-paths.yml @@ -42,6 +42,13 @@ value: "{{ grafana_alloy_secret_file }}" under: /etc regex: '^/etc/[A-Za-z0-9._-]+$' + # Not removed on disable UNLESS this role authored the unit (see teardown), + # but it is both written and deleted by root-owned `copy`/`file` tasks, so it + # gets the same anchored single-segment treatment as the secret file. + - name: grafana_alloy_env_checksum_file + value: "{{ grafana_alloy_env_checksum_file }}" + under: /etc + regex: '^/etc/[A-Za-z0-9._-]+$' loop_control: label: "{{ item.name }}" From 4d3da01a7c4bdc10cbc9b7039118434bc4db1a89 Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Thu, 17 Sep 2026 18:35:36 +0300 Subject: [PATCH 2/8] fix(ansible): close grafana_alloy token-path review findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three issues from automated review of the inventory-token path: * reject grafana_alloy_env_checksum_file == grafana_alloy_secret_file in validate-paths: equal paths would make preflight slurp the operator's credential file as if it were the provenance record, and let the post- converge record task overwrite the credentials with " ". * scope the env-file content greps to the host-provisioned posture: with grafana_alloy_api_token set the rewrite is token-only, so keys found on the current file prove nothing — demand every non-token connection setting from inventory before any mutation instead. * stop interpolating part of the API token into a preflight assert failure message; assert output lands in plain Ansible/CI logs. --- ansible/roles/grafana_alloy/README.md | 6 ++- .../roles/grafana_alloy/tasks/preflight.yml | 50 ++++++++++++++++--- .../grafana_alloy/tasks/validate-paths.yml | 18 +++++++ 3 files changed, 64 insertions(+), 10 deletions(-) diff --git a/ansible/roles/grafana_alloy/README.md b/ansible/roles/grafana_alloy/README.md index bb5e9f3..8e7b7f5 100644 --- a/ansible/roles/grafana_alloy/README.md +++ b/ansible/roles/grafana_alloy/README.md @@ -101,8 +101,10 @@ the classic failure modes loud: `grafana_alloy_overwrite_host_file: true`. The authoring is all-or-nothing: any hand-added `GC_*` lines beyond the token are -DISCARDED by a rewrite — migrate them to their inventory variables first -(preflight's per-key gates then verify nothing was lost). An out-of-band edit on +DISCARDED by a rewrite — migrate them to their inventory variables first, and +with the token set preflight demands every remaining connection setting from +inventory (the host file's current keys prove nothing once the rewrite lands). +An out-of-band edit on the host-provisioned path restarts the agent with a note, same as `decdn_node`. Returning a host to hand-provisioned mode: clear the variable, then discard the provenance record (`sudo rm /etc/grafana-alloy.env.sha256`). The Helm chart is diff --git a/ansible/roles/grafana_alloy/tasks/preflight.yml b/ansible/roles/grafana_alloy/tasks/preflight.yml index ef71e96..5a49a78 100644 --- a/ansible/roles/grafana_alloy/tasks/preflight.yml +++ b/ansible/roles/grafana_alloy/tasks/preflight.yml @@ -248,6 +248,10 @@ # backslashes are deliberately NOT rejected here, to_json emits exactly the # double-quoted C-escaped form systemd's parser expects. A value with embedded # whitespace would arrive on the daemon side truncated at the first blank. +# +# no_log does not exist for asserts: their output lands in plain Ansible/CI logs, +# so the failure message must carry NO part of the configured value — report only +# which variable is wrong and why, never token-derived text. - name: Validate the shape of an inventory-provided API token ansible.builtin.assert: that: @@ -255,10 +259,9 @@ quiet: true fail_msg: >- grafana_alloy_api_token must be the bare token value (non-empty, no - whitespace, no newline) — it is written to {{ grafana_alloy_secret_file }} - as GC_API_TOKEN="{{ grafana_alloy_api_token | truncate(24, true) }}" in a - single line systemd parses before dropping privileges. Quotes/backslashes are - fine (JSON-encoded); spaces are not. + whitespace, no newline) — it becomes the single GC_API_TOKEN="…" line of + {{ grafana_alloy_secret_file }}, which systemd parses before dropping + privileges. Quotes/backslashes are fine (JSON-encoded); spaces are not. # Empty keeps the host-provisioned posture; the gate only guards the inventory path. when: grafana_alloy_api_token | length > 0 @@ -459,6 +462,30 @@ _ga_record_source != 'inventory' or _ga_recorded_sum != (_ga_secret_stat.stat.checksum | default('')) +# Inventory-authoring mode makes the final env file TOKEN-ONLY, so everything +# else MUST come from inventory — checked HERE (before any mutation) because the +# env-file greps below can only vouch for keys that this run is about to discard. +- name: Demand every non-token connection setting from inventory for a token-authored env file + ansible.builtin.assert: + that: + - grafana_alloy_prom_url | string | length > 0 + - grafana_alloy_prom_username | string | length > 0 + - grafana_alloy_otlp_endpoint | string | length > 0 + - not (grafana_alloy_logs_enabled | bool) or (grafana_alloy_loki_url | string | length > 0) + - not (grafana_alloy_logs_enabled | bool) or (grafana_alloy_loki_username | string | length > 0) + quiet: true + fail_msg: >- + With grafana_alloy_api_token set, {{ grafana_alloy_secret_file }} is + REWRITTEN as a token-only file, so a present GC_PROM_REMOTE_WRITE_URL / + GC_PROM_USERNAME / GC_OTLP_ENDPOINT / GC_LOKI_* line on the CURRENT copy of + the file cannot satisfy preflight — it would be discarded. Set the matching + inventory variables instead (grafana_alloy_prom_url, _prom_username, + _otlp_endpoint, _loki_url, _loki_username; the Loki pair only matters while + logs are enabled). grafana_alloy_otlp_username stays optional — without it + traces fall back to the Prometheus instance ID. + # Empty keeps the host-provisioned posture, where the greps below judge the file. + when: grafana_alloy_api_token | length > 0 + # systemd skips an EnvironmentFile line it cannot parse SILENTLY, so each required # key is grepped for presence + value shape directly on the host: grep keeps the # values off stdout AND off the control machine (slurping the file back would put @@ -467,7 +494,9 @@ # "Required" is now conditional: a key is only needed when nothing in inventory # supplies that value, and the Loki pair is only needed when logs are enabled. # That includes GC_API_TOKEN itself since the grafana_alloy_api_token relaxation: -# the grep only fires when the variable is empty. +# the grep only fires when the variable is empty. The whole loop is also scoped to +# the HOST-PROVISIONED posture: with an inventory token the file below is replaced +# wholesale moments after this runs, so its current contents prove nothing. - name: Confirm every required credential key is present and shaped correctly ansible.builtin.command: cmd: grep -qE -- '{{ item.regex }}' {{ grafana_alloy_secret_file | quote }} @@ -506,7 +535,7 @@ required: "{{ grafana_alloy_logs_enabled | bool and grafana_alloy_loki_username | string | length == 0 }}" loop_control: label: "{{ item.key }}" - when: item.required | bool + when: (grafana_alloy_api_token | length == 0) and (item.required | bool) register: _ga_secret_grep - name: Report which credential key was missing or malformed @@ -538,7 +567,9 @@ # Prometheus fallback and 401s the entire trace pipeline while metrics and logs # stay healthy. The required-key gate above cannot cover it (it only greps keys it # demands), hence this present-then-shaped pair. Skipped when inventory supplies -# the ID, because the template then inlines it and emits no sys.env lookup at all. +# the ID, because the template then inlines it and emits no sys.env lookup at all +# — and likewise on the inventory-authoring path, where the rewrite guarantees no +# GC_OTLP_USERNAME survives into the final file. - name: Detect a host-provided OTLP instance ID ansible.builtin.command: cmd: grep -qE -- '^GC_OTLP_USERNAME=.*[^[:space:]]' {{ grafana_alloy_secret_file | quote }} @@ -546,7 +577,9 @@ check_mode: false # read-only gate: must also run under make check failed_when: false register: _ga_otlp_user_present - when: grafana_alloy_otlp_username | string | length == 0 + when: + - grafana_alloy_otlp_username | string | length == 0 + - grafana_alloy_api_token | length == 0 # Same charset as the inventory instance-ID gate, plus the optional double quotes # systemd strips off an EnvironmentFile value. @@ -577,4 +610,5 @@ grafana_alloy_otlp_username in inventory instead. when: - grafana_alloy_otlp_username | string | length == 0 + - grafana_alloy_api_token | length == 0 - (_ga_otlp_user_present.rc | default(1)) == 0 diff --git a/ansible/roles/grafana_alloy/tasks/validate-paths.yml b/ansible/roles/grafana_alloy/tasks/validate-paths.yml index 9d5b133..36cf8db 100644 --- a/ansible/roles/grafana_alloy/tasks/validate-paths.yml +++ b/ansible/roles/grafana_alloy/tasks/validate-paths.yml @@ -52,6 +52,24 @@ loop_control: label: "{{ item.name }}" +# Equal paths corrupt BOTH files' jobs: the provenance slurp would read the +# operator's credential file (token content!) back onto the control machine as +# if it were the record, and the post-converge record task would OVERWRITE the +# credential file itself with plain " " text — silently breaking +# the agent's auth on the next systemd restart. Both are direct children of /etc +# by the gates above, so equality is possible with mis-typed overrides; reject it +# before preflight reads either file. +- name: Require the provenance record and the credential file to differ + ansible.builtin.assert: + that: + - grafana_alloy_env_checksum_file != grafana_alloy_secret_file + quiet: true + fail_msg: >- + grafana_alloy_env_checksum_file ({{ grafana_alloy_env_checksum_file }}) and + grafana_alloy_secret_file ({{ grafana_alloy_secret_file }}) must be different + paths: the record is rewritten from inventory after every converge, while the + secret file holds the host's credentials — one path cannot serve both. + # The role templates the config INTO the directory it creates and chowns; a file # outside it would be written to an unmanaged (possibly unsafe) location and # would survive the disable path's directory removal. From 01776744283b2aefb9c8d742b64dfedff5207a1c Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Thu, 17 Sep 2026 18:54:28 +0300 Subject: [PATCH 3/8] ci(ansible): silence KICS hardcoded-secret FPs on molecule fixture tokens The new token-via-inventory scenarios carry deliberately fake API tokens (molecule-test-token & co.) as plain YAML literals named grafana_alloy_api_token, which trips KICS' "Passwords And Secrets - Generic Token" HIGH query and fails the fail-on-high gate with 5 findings. Annotate exactly those five fixture lines with # kics-scan ignore-line so molecule stays scanned and any future real secret in it still lights up. --- ansible/molecule/grafana-cloud-token/converge.yml | 3 +++ ansible/molecule/grafana-cloud-token/verify.yml | 2 ++ ansible/molecule/validation/converge.yml | 2 ++ 3 files changed, 7 insertions(+) diff --git a/ansible/molecule/grafana-cloud-token/converge.yml b/ansible/molecule/grafana-cloud-token/converge.yml index 5fe0595..8414e9d 100644 --- a/ansible/molecule/grafana-cloud-token/converge.yml +++ b/ansible/molecule/grafana-cloud-token/converge.yml @@ -20,6 +20,9 @@ grafana_alloy_region: US # --- The point of this scenario ------------------------------------------------- + # Deliberately fake token literal below, not a committed secret — keep only + # KICS' hardcoded-credential query quiet about exactly that one line. + # kics-scan ignore-line grafana_alloy_api_token: molecule-test-token # All six non-secret settings satisfied by inventory, so preflight requires no diff --git a/ansible/molecule/grafana-cloud-token/verify.yml b/ansible/molecule/grafana-cloud-token/verify.yml index f0075f3..6e70aed 100644 --- a/ansible/molecule/grafana-cloud-token/verify.yml +++ b/ansible/molecule/grafana-cloud-token/verify.yml @@ -56,6 +56,7 @@ The authored env file must be a single GC_API_TOKEN="" line. Got: {{ ga_body }} vars: + # kics-scan ignore-line (fake fixture token, not a committed secret) ga_expected_token: molecule-test-token ga_body: "{{ ga_env_b64.content | b64decode }}" ga_lines: "{{ (ga_env_b64.content | b64decode).splitlines() }}" @@ -129,6 +130,7 @@ {{ lookup('ansible.builtin.env', 'MOLECULE_SCENARIO_DIRECTORY') }}/files/alloy-stub grafana_alloy_deployment_environment: production grafana_alloy_region: US + # kics-scan ignore-line (fake fixture token, not a committed secret) grafana_alloy_api_token: molecule-rotated-token grafana_alloy_prom_url: https://prometheus-prod-xx.molecule.invalid/api/prom/push grafana_alloy_prom_username: "1234567" diff --git a/ansible/molecule/validation/converge.yml b/ansible/molecule/validation/converge.yml index 3c31c3a..8f904eb 100644 --- a/ansible/molecule/validation/converge.yml +++ b/ansible/molecule/validation/converge.yml @@ -1136,6 +1136,7 @@ name: grafana_alloy vars: decdn_grafana_cloud_enabled: true + # kics-scan ignore-line (fake fixture token, not a committed secret) grafana_alloy_api_token: "glc_molecule spaced" rescue: - name: Record ga-token-var-shape rejection (only if the token-shape gate failed) @@ -1176,6 +1177,7 @@ name: grafana_alloy vars: decdn_grafana_cloud_enabled: true + # kics-scan ignore-line (fake fixture token, not a committed secret) grafana_alloy_api_token: molecule-test-token rescue: - name: Record ga-collision rejection (only if the overwrite gate failed) From f2762d2da033c1efa5ca0fd4b3d284928b2abf63 Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Thu, 17 Sep 2026 16:40:34 +0000 Subject: [PATCH 4/8] fix(ansible): harden Alloy token provenance Refuse unknown credential overwrites, constrain and validate provenance records, preserve provenance across disable, recover interrupted rotations with a restart, and add behavioral Molecule coverage for each state transition. --- ansible/README.md | 2 +- .../molecule/grafana-cloud-token/verify.yml | 262 +++++++++++++++--- ansible/molecule/grafana-cloud/verify.yml | 51 ++++ ansible/molecule/validation/converge.yml | 85 +++++- ansible/roles/grafana_alloy/README.md | 13 +- ansible/roles/grafana_alloy/defaults/main.yml | 5 +- ansible/roles/grafana_alloy/tasks/main.yml | 47 +++- .../roles/grafana_alloy/tasks/preflight.yml | 58 ++-- .../tasks/validate-env-record.yml | 55 ++++ .../grafana_alloy/tasks/validate-paths.yml | 53 ++-- 10 files changed, 533 insertions(+), 98 deletions(-) create mode 100644 ansible/roles/grafana_alloy/tasks/validate-env-record.yml diff --git a/ansible/README.md b/ansible/README.md index 56ca915..2378475 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -311,7 +311,7 @@ the RPC URL when it is not provisioned on the host instead). Highlights: | `decdn_region` / `decdn_bind_port` / `decdn_rate_per_mb` | `""` / `4433` / `10` | node identity, QUIC port, USDC base units/MB. | | `decdn_env_checksum_file` / `decdn_env_overwrite_host_file` | `/etc/decdn/.decdn.env.sha256` / `false` | Provenance record for the secret env file (`0600 root`), and the opt-in that lets an inventory `decdn_rpc_url` overwrite a host-edited one. | | `decdn_grafana_cloud_enabled` | `false` | ONE mirrored knob (identical default in both roles) wiring on Grafana Cloud observability: installs + configures `grafana_alloy` — node metrics, machine metrics, journald, agent health — AND injects `otlp_endpoint` into `node.toml`. Only the API token is provisioned per host *or* carried by `grafana_alloy_api_token` in git-ignored inventory; the rest are inventory variables. Label/cost guardrails in the role README. | -| `grafana_alloy_api_token` / `_env_checksum_file` / `_overwrite_host_file` | `""` / `/etc/grafana-alloy.env.sha256` / `false` | The dual-homed Grafana Cloud token and its provenance machinery (`#39` parity with the row above); see [`roles/grafana_alloy/README.md`](roles/grafana_alloy/README.md). | +| `grafana_alloy_api_token` / `_env_checksum_file` / `_overwrite_host_file` | `""` / `/etc/grafana-alloy.env.sha256` / `false` | The dual-homed Grafana Cloud token and its provenance machinery (`#39` parity with the row above); the record path is fixed to `.sha256` and survives disable with the secret. See [`roles/grafana_alloy/README.md`](roles/grafana_alloy/README.md). | --- diff --git a/ansible/molecule/grafana-cloud-token/verify.yml b/ansible/molecule/grafana-cloud-token/verify.yml index 6e70aed..591d5aa 100644 --- a/ansible/molecule/grafana-cloud-token/verify.yml +++ b/ansible/molecule/grafana-cloud-token/verify.yml @@ -108,37 +108,135 @@ - "'alloy.service' in ansible_facts.services" - ansible_facts.services['alloy.service'].state == 'running' -# Mutates the converged host deliberately (like ../grafana-cloud's disable checks), -# so it runs AFTER every assertion above: rotating the token must update both the -# authored file and the provenance record atomically-ish, and leave the agent up. -- name: Rotate +# Mutates the converged host deliberately, after all baseline assertions above. +# This is the provenance state-machine regression suite: unknown-file refusal, +# explicit takeover, interrupted-rotation retry, process restart, and the +# disable→enable missing-secret guard all need real host-side behavior. +- name: Exercise token provenance recovery and teardown hosts: all become: true + vars: + ga_stub_bin: >- + {{ lookup('ansible.builtin.env', 'MOLECULE_SCENARIO_DIRECTORY') }}/files/alloy-stub + # kics-scan ignore-line (fake fixture token, not a committed secret) + ga_overwrite_token: molecule-overwrite-token + # kics-scan ignore-line (fake fixture token, not a committed secret) + ga_rotated_token: molecule-rotated-token + ga_role_vars: &ga_role_vars + decdn_grafana_cloud_enabled: true + grafana_alloy_install_method: manual + grafana_alloy_manual_bin_src: "{{ ga_stub_bin }}" + grafana_alloy_deployment_environment: production + grafana_alloy_region: US + grafana_alloy_prom_url: https://prometheus-prod-xx.molecule.invalid/api/prom/push + grafana_alloy_prom_username: "1234567" + grafana_alloy_otlp_endpoint: https://otlp-gateway-prod-xx.molecule.invalid/otlp + grafana_alloy_loki_url: https://logs-prod-xx.molecule.invalid/loki/api/v1/push + grafana_alloy_loki_username: "7654321" tasks: - - name: Snapshot the pre-rotation record + # --- Unknown provenance: fail before destroying host-managed keys ---------- + - name: Remove the provenance record to model a pre-record deployment + ansible.builtin.file: + path: /etc/grafana-alloy.env.sha256 + state: absent + + - name: Stage an untracked host-managed credential file + ansible.builtin.copy: + dest: /etc/grafana-alloy.env + owner: root + group: root + mode: "0600" + content: | + GC_API_TOKEN=molecule-untracked-token + GC_OPERATOR_ONLY=must-survive-refusal + + - name: Default to detecting no untracked-file rejection + ansible.builtin.set_fact: + ga_untracked_rejected: false + + - name: Try an inventory token against the untracked host file + block: + - name: Include grafana_alloy without the overwrite opt-in + ansible.builtin.include_role: + name: grafana_alloy + vars: + <<: *ga_role_vars + grafana_alloy_api_token: "{{ ga_overwrite_token }}" + rescue: + - name: Record the expected untracked-file rejection + ansible.builtin.set_fact: + ga_untracked_rejected: true + when: ansible_failed_task.name is match('^Require confirmation to overwrite an env file') + + - name: Read the refused credential file back ansible.builtin.slurp: - src: /etc/grafana-alloy.env.sha256 - register: ga_record_pre + src: /etc/grafana-alloy.env + register: ga_refused_env - - name: Re-run the role with a rotated token + - name: Assert refusal happened before any host-managed content was lost + ansible.builtin.assert: + that: + - ga_untracked_rejected | bool + - (ga_refused_env.content | b64decode) is search('GC_OPERATOR_ONLY=must-survive-refusal') + - (ga_refused_env.content | b64decode) is search('molecule-untracked-token') + - (ga_refused_env.content | b64decode) is not search('molecule-overwrite-token') + + - name: Re-run with explicit permission to take the untracked file over ansible.builtin.include_role: name: grafana_alloy vars: - decdn_grafana_cloud_enabled: true - grafana_alloy_install_method: manual - grafana_alloy_manual_bin_src: >- - {{ lookup('ansible.builtin.env', 'MOLECULE_SCENARIO_DIRECTORY') }}/files/alloy-stub - grafana_alloy_deployment_environment: production - grafana_alloy_region: US - # kics-scan ignore-line (fake fixture token, not a committed secret) - grafana_alloy_api_token: molecule-rotated-token - grafana_alloy_prom_url: https://prometheus-prod-xx.molecule.invalid/api/prom/push - grafana_alloy_prom_username: "1234567" - grafana_alloy_otlp_endpoint: https://otlp-gateway-prod-xx.molecule.invalid/otlp - grafana_alloy_loki_url: https://logs-prod-xx.molecule.invalid/loki/api/v1/push - grafana_alloy_loki_username: "7654321" - - - name: Read the rotated artifacts + <<: *ga_role_vars + grafana_alloy_api_token: "{{ ga_overwrite_token }}" + grafana_alloy_overwrite_host_file: true + + - name: Read the explicitly adopted credential file + ansible.builtin.slurp: + src: /etc/grafana-alloy.env + register: ga_adopted_env + + - name: Assert explicit takeover produced the promised token-only file + ansible.builtin.assert: + that: + - >- + (ga_adopted_env.content | b64decode).splitlines() + == ['GC_API_TOKEN=' ~ (ga_overwrite_token | to_json)] + + # --- Interrupted rotation: same desired bytes must be resumable ------------ + - name: Read Alloy's PID before the interrupted rotation + ansible.builtin.command: + cmd: pgrep -f '^sleep infinity$' + changed_when: false + register: ga_pid_before_retry + + - name: Require exactly one Alloy stub process before rotation + ansible.builtin.assert: + that: + - ga_pid_before_retry.stdout_lines | length == 1 + + # Model the precise failure window: the role wrote the new bytes, then a later + # validation/start task aborted before the old provenance record was replaced. + - name: Stage the token bytes left by an interrupted rotation + ansible.builtin.copy: + dest: /etc/grafana-alloy.env + owner: root + group: root + mode: "0600" + content: "GC_API_TOKEN={{ ga_rotated_token | to_json }}\n" + + - name: Retry the interrupted rotation without the destructive overwrite knob + ansible.builtin.include_role: + name: grafana_alloy + vars: + <<: *ga_role_vars + grafana_alloy_api_token: "{{ ga_rotated_token }}" + + - name: Read Alloy's PID after the recovered rotation + ansible.builtin.command: + cmd: pgrep -f '^sleep infinity$' + changed_when: false + register: ga_pid_after_retry + + - name: Read the recovered rotation artifacts ansible.builtin.slurp: src: "{{ item }}" register: ga_rotated @@ -146,36 +244,116 @@ - /etc/grafana-alloy.env - /etc/grafana-alloy.env.sha256 - - name: Stat the rotated credential file for its fresh checksum + - name: Stat the recovered credential file ansible.builtin.stat: path: /etc/grafana-alloy.env get_checksum: true checksum_algorithm: sha256 register: ga_rotated_stat - - name: Assert rotation landed in both the file and the record + - name: Assert retry completed provenance and restarted onto the new token ansible.builtin.assert: that: - # Untracked-before-warn semantics do not apply here: the record EXISTS - # (seeded by the converge), source='inventory' and checksum matched, so - # no collision assert can fire. Verify content + provenance both moved. - - (ga_rotated.results[0].content | b64decode) is search('molecule-rotated-token') - - (ga_rotated.results[0].content | b64decode) is not search('molecule-test-token') + - ga_pid_after_retry.stdout_lines | length == 1 + - ga_pid_after_retry.stdout | trim != ga_pid_before_retry.stdout | trim + - >- + (ga_rotated.results[0].content | b64decode).splitlines() + == ['GC_API_TOKEN=' ~ (ga_rotated_token | to_json)] - ((ga_rotated.results[1].content | b64decode | trim).split()[0]) == 'inventory' - ((ga_rotated.results[1].content | b64decode | trim).split()[1]) - == (ga_rotated_stat.stat.checksum) - - ((ga_rotated.results[1].content | b64decode | trim).split()[1]) - != (ga_record_pre.content | b64decode | trim).split()[1] - fail_msg: >- - Token rotation did not propagate: either the env file kept the old - value, or the provenance record does not match the freshly authored - file (a stale record would trip the adoption guard next converge). + == ga_rotated_stat.stat.checksum - - name: Gather service facts after rotation - ansible.builtin.service_facts: + # Exercise the separate "would overwrite" wording and prove dry-run leaves + # both secret and provenance bytes alone. The record is temporarily removed + # outside check mode solely to recreate the untracked-upgrade branch. + - name: Save the recovered provenance record for the dry-run fixture + ansible.builtin.set_fact: + ga_record_before_check: "{{ ga_rotated.results[1].content | b64decode }}" - - name: Assert alloy restarted onto the rotated credential + - name: Remove provenance to model an untracked file during dry-run + ansible.builtin.file: + path: /etc/grafana-alloy.env.sha256 + state: absent + + - name: Dry-run an explicitly approved takeover of the untracked file + check_mode: true + block: + - name: Include grafana_alloy under check mode + ansible.builtin.include_role: + name: grafana_alloy + vars: + <<: *ga_role_vars + # kics-scan ignore-line (fake fixture token, not a committed secret) + grafana_alloy_api_token: molecule-check-mode-token + grafana_alloy_overwrite_host_file: true + + - name: Read the credential file after dry-run + ansible.builtin.slurp: + src: /etc/grafana-alloy.env + register: ga_after_check + + - name: Stat provenance after dry-run + ansible.builtin.stat: + path: /etc/grafana-alloy.env.sha256 + register: ga_record_after_check + + - name: Assert dry-run reported intent without claiming or changing bytes ansible.builtin.assert: that: - - "'alloy.service' in ansible_facts.services" - - ansible_facts.services['alloy.service'].state == 'running' + - ga_after_check.content == ga_rotated.results[0].content + - not ga_record_after_check.stat.exists + - _ga_untracked_overwrite_applied_report.skipped | default(false) | bool + - not (_ga_untracked_overwrite_check_report.skipped | default(false) | bool) + - (_ga_untracked_overwrite_check_report.msg | default('')) is search('WOULD rewrite') + - (_ga_untracked_overwrite_check_report.msg | default('')) is search('check mode made no change') + + - name: Restore the saved provenance record after the dry-run fixture + ansible.builtin.copy: + dest: /etc/grafana-alloy.env.sha256 + owner: root + group: root + mode: "0600" + content: "{{ ga_record_before_check }}" + + # --- Disable/re-enable: keep evidence that the stale file was role-authored -- + - name: Disable the role after an inventory-authored deployment + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: false + + - name: Stat the secret and provenance retained across disable + ansible.builtin.stat: + path: "{{ item }}" + register: ga_retained + loop: + - /etc/grafana-alloy.env + - /etc/grafana-alloy.env.sha256 + + - name: Assert disable retained both halves of the provenance evidence + ansible.builtin.assert: + that: + - ga_retained.results | rejectattr('stat.exists') | list | length == 0 + + - name: Default to detecting no missing-token rejection + ansible.builtin.set_fact: + ga_missing_token_rejected: false + + - name: Try to re-enable after the inventory token disappeared + block: + - name: Include grafana_alloy with the API token missing + ansible.builtin.include_role: + name: grafana_alloy + vars: + <<: *ga_role_vars + grafana_alloy_api_token: "" + rescue: + - name: Record the expected missing-token rejection + ansible.builtin.set_fact: + ga_missing_token_rejected: true + when: ansible_failed_task.name is match('^Refuse to adopt a role-authored env file') + + - name: Assert re-enable did not adopt the stale role-authored token + ansible.builtin.assert: + that: + - ga_missing_token_rejected | bool diff --git a/ansible/molecule/grafana-cloud/verify.yml b/ansible/molecule/grafana-cloud/verify.yml index 982ee2a..5ae5e61 100644 --- a/ansible/molecule/grafana-cloud/verify.yml +++ b/ansible/molecule/grafana-cloud/verify.yml @@ -324,6 +324,57 @@ and ansible_facts.services['decdn-node.service'].state == 'running' fail_msg: "alloy or decdn-node is not running — journalctl -u alloy -e / -u decdn-node -e" +# A host-managed env file is never rendered by the role, so only the provenance +# checksum can notice an out-of-band rotation. Prove that signal restarts the +# actual process; file/checksum assertions alone would stay green if the notify +# disappeared and Alloy kept the old environment forever. +- name: Verify a host-provisioned token edit restarts Alloy + hosts: all + become: true + tasks: + - name: Read Alloy's PID before the host-side credential edit + ansible.builtin.command: + cmd: pgrep -f '^sleep infinity$' + changed_when: false + register: ga_host_pid_before + + - name: Require exactly one Alloy stub process before the edit + ansible.builtin.assert: + that: + - ga_host_pid_before.stdout_lines | length == 1 + + - name: Rotate the host-provisioned token out of band + ansible.builtin.replace: + path: /etc/grafana-alloy.env + regexp: '^GC_API_TOKEN=.*$' + # kics-scan ignore-line (fake fixture token, not a committed secret) + replace: GC_API_TOKEN=molecule-host-rotated-token + + - name: Re-run grafana_alloy after the host-side edit + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_install_method: manual + grafana_alloy_manual_bin_src: >- + {{ lookup('ansible.builtin.env', 'MOLECULE_SCENARIO_DIRECTORY') }}/files/alloy-stub + grafana_alloy_deployment_environment: production + grafana_alloy_region: US + grafana_alloy_loki_url: https://logs-prod-xx.molecule.invalid/loki/api/v1/push + grafana_alloy_prom_username: 1234567 + + - name: Read Alloy's PID after the host-side credential edit + ansible.builtin.command: + cmd: pgrep -f '^sleep infinity$' + changed_when: false + register: ga_host_pid_after + + - name: Assert Alloy restarted onto the edited host environment + ansible.builtin.assert: + that: + - ga_host_pid_after.stdout_lines | length == 1 + - ga_host_pid_after.stdout | trim != ga_host_pid_before.stdout | trim + # Also mutates the converged host, so it runs after every assertion above and # before the teardown play below. - name: Verify turning logs off takes the journal grant back diff --git a/ansible/molecule/validation/converge.yml b/ansible/molecule/validation/converge.yml index 8f904eb..0f5ffcf 100644 --- a/ansible/molecule/validation/converge.yml +++ b/ansible/molecule/validation/converge.yml @@ -927,6 +927,71 @@ decdn_rejected: "{{ decdn_rejected + ['managed-paths'] }}" when: ansible_failed_task.name is match('^Require the rendered config to live inside') + # The provenance file is root-written, root-read and removed during teardown. + # It must be the fixed companion of the credential file, not an independently + # aimable path that can slurp/clobber/delete another file under /etc. + - name: "Case ga-provenance-path — checksum record is not beside the credential file" + block: + - name: Run grafana_alloy with an independently aimed provenance path + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_env_checksum_file: /etc/molecule-unrelated-root-file + rescue: + - name: Record ga-provenance-path rejection (only if the companion-path gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['managed-paths'] }}" + when: ansible_failed_task.name is match('^Require the provenance record to sit beside') + + - name: "Case ga-provenance-content — unrelated file occupies the record path" + block: + - name: Stage an unrelated root-only file at the provenance path + ansible.builtin.copy: + dest: /etc/grafana-alloy.env.sha256 + owner: root + group: root + mode: "0600" + # A valid first NUL-delimited grep record followed by unrelated bytes: + # `grep -zq` accepts this unless validation full-matches the byte stream. + content: >- + {{ ('aG9zdCAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMD' + ~ 'AwMDAwMDAwMDAwMDAwMDAwMDAwAHVucmVsYXRlZC1zZW5zaXRpdmUtY29udGVudAo=') + | b64decode }} + + - name: Run grafana_alloy with the unrelated provenance-path occupant + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + # kics-scan ignore-line (fake fixture token, not a committed secret) + grafana_alloy_api_token: molecule-test-token + rescue: + - name: Record ga-provenance-content rejection (only if shape validation failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['provenance-record'] }}" + when: ansible_failed_task.name is match('^Require an existing provenance record to have') + always: + - name: Remove the unrelated provenance-path occupant + ansible.builtin.file: + path: /etc/grafana-alloy.env.sha256 + state: absent + + - name: "Case ga-config-record-collision — teardown directory contains retained provenance" + block: + - name: Run disabled grafana_alloy with config_dir aimed at the provenance file + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: false + grafana_alloy_config_dir: /etc/grafana-alloy.env.sha256 + grafana_alloy_config_file: /etc/grafana-alloy.env.sha256/config.alloy + rescue: + - name: Record ga-config-record-collision rejection + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['managed-paths'] }}" + when: ansible_failed_task.name is match('^Keep destructive directories away') + - name: "Case ga-collector — unknown node_exporter collector name" block: - name: Run grafana_alloy with a bogus host collector @@ -1144,8 +1209,9 @@ decdn_rejected: "{{ decdn_rejected + ['token-shape'] }}" when: ansible_failed_task.name is match('^Validate the shape of an inventory-provided API token') - # A tracked host-authored env file must not be silently clobbered by an - # inventory token; overwriting requires the explicit opt-in knob. + # An existing host-authored env file with NO provenance is the upgrade state + # of every pre-record deployment. It must not be silently clobbered by an + # inventory token; unknown provenance requires the explicit opt-in knob. - name: "Case ga-collision — inventory token aimed at an untracked host env file" block: - name: Stage a host-style credential fixture @@ -1160,18 +1226,6 @@ GC_PROM_USERNAME=999888777 GC_API_TOKEN=molecule-test-token - # Seed a KNOWN-foreign provenance record: source says 'inventory' but the - # checksum no longer matches the staged file, which is exactly the - # role-didn't-last-write-this state the overwrite knob exists for. - - name: Seed a stale provenance record for the fixture - ansible.builtin.copy: - dest: /etc/grafana-alloy.env.sha256 - owner: root - group: root - mode: "0600" - content: >- - inventory 0000000000000000000000000000000000000000000000000000000000000000 - - name: Run grafana_alloy with an inventory token against the foreign fixture ansible.builtin.include_role: name: grafana_alloy @@ -1219,4 +1273,5 @@ "host-collectors", "host-collectors", "host-collectors", "endpoint-scheme", "inventory-token", "inventory-token", "instance-id", "otlp-username-env", "token-shape", "token-collision", - "managed-paths", "managed-paths", "managed-paths"] + "provenance-record", + "managed-paths", "managed-paths", "managed-paths", "managed-paths", "managed-paths"] diff --git a/ansible/roles/grafana_alloy/README.md b/ansible/roles/grafana_alloy/README.md index 8e7b7f5..b50a97d 100644 --- a/ansible/roles/grafana_alloy/README.md +++ b/ansible/roles/grafana_alloy/README.md @@ -98,7 +98,13 @@ the classic failure modes loud: → the run refuses to adopt its own file as operator-provisioned ("restore the variable or discard the record explicitly"). - **Untracked hand-edited file** meeting an inventory token → refused unless - `grafana_alloy_overwrite_host_file: true`. + `grafana_alloy_overwrite_host_file: true`; the sole safe exception is a file + already byte-identical to the desired token-only payload, because no host bytes + would be discarded and it may be the residue of an interrupted first converge. +- **Interrupted inventory rotation** (new env bytes landed, later validation or + restart failed before the record moved) → retry recognises the exact desired + token-only checksum, restarts Alloy, and completes the record without asking + for the destructive overwrite opt-in. The authoring is all-or-nothing: any hand-added `GC_*` lines beyond the token are DISCARDED by a rewrite — migrate them to their inventory variables first, and @@ -106,6 +112,9 @@ with the token set preflight demands every remaining connection setting from inventory (the host file's current keys prove nothing once the rewrite lands). An out-of-band edit on the host-provisioned path restarts the agent with a note, same as `decdn_node`. +Disabling the role intentionally retains both the secret env file **and** its +provenance record; retaining only the file would let a later re-enable with a +missing `secret.yml` misclassify the stale role-authored token as host-owned. Returning a host to hand-provisioned mode: clear the variable, then discard the provenance record (`sudo rm /etc/grafana-alloy.env.sha256`). The Helm chart is unaffected (it was Secret-oriented from the start). @@ -235,7 +244,7 @@ upstream version/sha256). Highlights: | `grafana_alloy_self_metrics_enabled` | `true` | Alloy's own health | | `grafana_alloy_prom_url` / `_prom_username` / `_otlp_endpoint` / `_otlp_username` / `_loki_url` / `_loki_username` | `""` | Non-secret connection settings; empty ⇒ read the matching `GC_…` env key (`_otlp_username` then falls back to the Prometheus ID) | | `grafana_alloy_api_token` | `""` | **SENSITIVE** — the API token (git-ignored `secret.yml`). Empty ⇒ operator-provisioned `/etc/grafana-alloy.env`; set ⇒ the role authors that file token-only, root 0600. See "The API token has two homes now". | -| `grafana_alloy_env_checksum_file` | `/etc/grafana-alloy.env.sha256` | Provenance record (` `) enabling the adoption/overwrite guards below | +| `grafana_alloy_env_checksum_file` | `/etc/grafana-alloy.env.sha256` | Provenance record (` `) enabling the adoption/overwrite guards; must remain exactly `.sha256` | | `grafana_alloy_overwrite_host_file` | `false` | Explicit opt-in letting an inventory token rewrite a hand-edited/untracked env file (all-or-nothing: extra `GC_*` lines are discarded) | | `grafana_alloy_node_job` / `_host_job` / `_self_job` / `_logs_job` | see table above | Job labels; the `integrations/…` ones are what Grafana Cloud's dashboards match | diff --git a/ansible/roles/grafana_alloy/defaults/main.yml b/ansible/roles/grafana_alloy/defaults/main.yml index 39bbaac..5112641 100644 --- a/ansible/roles/grafana_alloy/defaults/main.yml +++ b/ansible/roles/grafana_alloy/defaults/main.yml @@ -54,9 +54,12 @@ grafana_alloy_api_token: "" # grafana_alloy_api_token authored it, "host" otherwise) into a root-owned 0600 # record NEXT TO the secret file — deliberately NOT inside /etc/alloy or # /var/lib/alloy, because the disable path removes those directories wholesale. +# Path validation requires this to remain exactly `.sha256`; it is +# not an independently aimable root-write path. # Preflight uses the pair to fail loud on (a) adopting a role-authored file after # the secret went missing from the control machine and (b) clobbering an operator- -# edited file without grafana_alloy_overwrite_host_file. +# edited or untracked file without grafana_alloy_overwrite_host_file. Disable keeps +# both file and record so a later re-enable cannot forget who authored the token. grafana_alloy_env_checksum_file: /etc/grafana-alloy.env.sha256 # Explicit opt-in letting grafana_alloy_api_token rewrite an env file whose diff --git a/ansible/roles/grafana_alloy/tasks/main.yml b/ansible/roles/grafana_alloy/tasks/main.yml index d3e0b86..a30c0a3 100644 --- a/ansible/roles/grafana_alloy/tasks/main.yml +++ b/ansible/roles/grafana_alloy/tasks/main.yml @@ -114,12 +114,13 @@ - "{{ _ga_unit_path }}" - "{{ grafana_alloy_config_dir }}" - "{{ grafana_alloy_state_dir }}" - # Provenance metadata is ours whenever the unit is (the marker gates - # this whole block); dropping it makes a disable→enable cycle re-seed - # cleanly instead of carrying a stale checksum. - - "{{ grafana_alloy_env_checksum_file }}" notify: Reload systemd + # Keep both the credential file and its provenance record. If the install + # was inventory-authored, deleting only the record would make a later + # re-enable with a missing secret.yml adopt the stale token as host-owned. + # The documented explicit handoff removes the record deliberately. + - name: Apply pending teardown notifications ansible.builtin.meta: flush_handlers @@ -294,6 +295,27 @@ notify: Restart alloy when: grafana_alloy_api_token | length > 0 + # If a prior run wrote the desired token bytes and then failed before the + # final provenance write, copy is now idempotent and would not notify. Queue a + # restart explicitly so retry both repairs the record and moves the running + # process onto those bytes. The desired checksum proves this is our exact + # token-only payload, not an arbitrary foreign edit. + - name: Resume an interrupted inventory token rotation + ansible.builtin.debug: + msg: >- + {{ 'Would restart Alloy to resume an interrupted inventory-token rotation.' + if ansible_check_mode else + 'Resuming interrupted inventory-token rotation; restart queued before provenance.' }} + changed_when: true + notify: Restart alloy + when: + - grafana_alloy_api_token | length > 0 + - _ga_secret_stat.stat.exists + - _ga_desired_inventory_sum == (_ga_secret_stat.stat.checksum | default('')) + - >- + _ga_record_source != 'inventory' + or _ga_recorded_sum != (_ga_secret_stat.stat.checksum | default('')) + # The copy above is no_log, so a run that replaced an existing file shows a bare # "changed" and nothing else. Say what was lost — but only for the UNKNOWABLE # case: a file whose known-foreign provenance already failed preflight loudly, @@ -310,7 +332,24 @@ this converge on, that file is tracked: an edit made outside Ansible fails the next deploy instead of being silently discarded. Clear grafana_alloy_api_token to hand the file back to the host. + register: _ga_untracked_overwrite_applied_report + when: + - not ansible_check_mode + - grafana_alloy_api_token | length > 0 + - _ga_secret_stat.stat.exists + - _ga_recorded_sum | length == 0 + - _ga_env_copy.changed | default(false) | bool + + - name: Warn what check mode would discard from an untracked env file + ansible.builtin.debug: + msg: >- + grafana_alloy_api_token is set and explicit overwrite permission was + supplied, so a real run WOULD rewrite {{ grafana_alloy_secret_file }} as + a token-only file. Any host-added KEY=value lines not represented by + inventory variables WOULD be discarded; check mode made no change. + register: _ga_untracked_overwrite_check_report when: + - ansible_check_mode - grafana_alloy_api_token | length > 0 - _ga_secret_stat.stat.exists - _ga_recorded_sum | length == 0 diff --git a/ansible/roles/grafana_alloy/tasks/preflight.yml b/ansible/roles/grafana_alloy/tasks/preflight.yml index 5a49a78..6e9e6e6 100644 --- a/ansible/roles/grafana_alloy/tasks/preflight.yml +++ b/ansible/roles/grafana_alloy/tasks/preflight.yml @@ -265,6 +265,22 @@ # Empty keeps the host-provisioned posture; the gate only guards the inventory path. when: grafana_alloy_api_token | length > 0 +# A retry after the role wrote new token bytes but failed before recording their +# checksum must be distinguishable from a foreign edit. Hash the exact bytes the +# copy task will render; the secret itself stays suppressed and never enters task +# output. An empty token has no desired inventory-authored file. +- name: Resolve the desired inventory-authored env-file checksum + ansible.builtin.set_fact: + _ga_desired_inventory_sum: >- + {{ (_ga_desired_inventory_content | hash('sha256')) + if grafana_alloy_api_token | length > 0 else '' }} + vars: + # Keep this byte-for-byte identical to the copy task's block scalar, including + # its trailing newline. Jinja string literals preserve `\\n` as two bytes. + _ga_desired_inventory_content: | + GC_API_TOKEN={{ grafana_alloy_api_token | string | to_json(ensure_ascii=false) }} + no_log: true + - name: Validate the loopback listener addresses ansible.builtin.assert: that: @@ -388,25 +404,29 @@ # next run can tell three states apart. Only this non-sensitive record is ever # read back — never the secret file itself. # -# An absent/truncated record means "no post-relaxation converge yet" and is the -# normal state, which is why both asserts below stay quiet for it. +# An absent record means "no post-relaxation converge yet". A present malformed, +# unsafe or unrelated file is rejected on the target before slurp, so an inventory +# typo cannot copy arbitrary /etc content into controller memory. +- name: Validate the env-file provenance record before reading it + ansible.builtin.import_tasks: validate-env-record.yml + - name: Read the env-file checksum this role last recorded ansible.builtin.slurp: src: "{{ grafana_alloy_env_checksum_file }}" register: _ga_recorded - failed_when: false - changed_when: false + when: _ga_record_stat.stat.exists -# Both fields default to "" — treated exactly like "no record", which also covers -# a truncated or older single-field record. +# Shape was proven before slurp, so an existing record always has exactly two +# fields. Conditional expressions avoid indexing a skipped slurp result when the +# record is absent. - name: Resolve the secret file's provenance ansible.builtin.set_fact: _ga_record_source: >- - {{ ((_ga_recorded.content | default('') | b64decode | trim).split() | length == 2) - | ternary((_ga_recorded.content | default('') | b64decode | trim).split()[0], '') }} + {{ (_ga_recorded.content | b64decode | trim).split()[0] + if _ga_record_stat.stat.exists else '' }} _ga_recorded_sum: >- - {{ ((_ga_recorded.content | default('') | b64decode | trim).split() | length == 2) - | ternary((_ga_recorded.content | default('') | b64decode | trim).split()[1], '') }} + {{ (_ga_recorded.content | b64decode | trim).split()[1] + if _ga_record_stat.stat.exists else '' }} # The dangerous half of inferring operator intent. With grafana_alloy_api_token # empty the operator-supplied file is adopted as-is — but that ALSO happens when @@ -436,10 +456,11 @@ - _ga_secret_stat.stat.exists # The mirror direction: an inventory token REWRITES the file wholesale (and -# token-only), losing any GC_* lines added outside Ansible. Scoped to a KNOWN -# foreign provenance — with no record this cannot distinguish anything, so it -# stays quiet and the run seeds the record for next time (the warn-once task in -# tasks/main.yml covers the unknowable case). +# token-only), losing any GC_* lines added outside Ansible. Existing bytes are +# safe without an opt-in only when (a) a matching inventory record proves this +# role wrote them, or (b) they already equal the exact desired token-only bytes, +# which is the retry state after a failed run wrote the file but not the record. +# Missing provenance is unknown, not permission to destroy a pre-record host file. - name: Require confirmation to overwrite an env file this role did not write ansible.builtin.assert: that: @@ -457,10 +478,13 @@ when: - grafana_alloy_api_token | length > 0 - _ga_secret_stat.stat.exists - - _ga_recorded_sum | length > 0 - >- - _ga_record_source != 'inventory' - or _ga_recorded_sum != (_ga_secret_stat.stat.checksum | default('')) + not ( + (_ga_record_source == 'inventory' + and _ga_recorded_sum == (_ga_secret_stat.stat.checksum | default(''))) + or + (_ga_desired_inventory_sum == (_ga_secret_stat.stat.checksum | default(''))) + ) # Inventory-authoring mode makes the final env file TOKEN-ONLY, so everything # else MUST come from inventory — checked HERE (before any mutation) because the diff --git a/ansible/roles/grafana_alloy/tasks/validate-env-record.yml b/ansible/roles/grafana_alloy/tasks/validate-env-record.yml new file mode 100644 index 0000000..a9e9849 --- /dev/null +++ b/ansible/roles/grafana_alloy/tasks/validate-env-record.yml @@ -0,0 +1,55 @@ +--- +# Validate the provenance record before any task reads it onto the controller or +# trusts it as authorization to rewrite a root-only credential file. The path is +# already constrained by validate-paths.yml to `.sha256`. + +- name: Inspect the env-file provenance record + ansible.builtin.stat: + path: "{{ grafana_alloy_env_checksum_file }}" + follow: false + get_checksum: false + register: _ga_record_stat + +- name: Require an existing provenance record to be a protected regular file + ansible.builtin.assert: + that: + - _ga_record_stat.stat.isreg | default(false) + - (_ga_record_stat.stat.pw_name | default('')) == 'root' + - (_ga_record_stat.stat.gr_name | default('')) == 'root' + - _ga_record_stat.stat.mode | string == '0600' + fail_msg: >- + {{ grafana_alloy_env_checksum_file }} exists but is not a root:root 0600 + regular file (symlinks are refused). Move the unrelated file away or restore + the role-authored provenance record before deploying. + when: _ga_record_stat.stat.exists + +# Probe the complete byte stream on the target with no stdout before slurp. A +# line-oriented or NUL-record grep can accept one valid prefix and ignore trailing +# unrelated bytes; Python's fullmatch cannot. Ansible already requires Python on +# every managed target, so this adds no runtime dependency. +- name: Check the provenance record shape on the target + ansible.builtin.command: + argv: + - python3 + - -c + - >- + import pathlib, re, sys; + raise SystemExit(0 if re.fullmatch( + rb'(inventory|host) [0-9a-f]{64}\n?', + pathlib.Path(sys.argv[1]).read_bytes()) else 1) + - "{{ grafana_alloy_env_checksum_file }}" + changed_when: false + check_mode: false + failed_when: false + register: _ga_record_shape + when: _ga_record_stat.stat.exists + +- name: Require an existing provenance record to have the role-owned format + ansible.builtin.assert: + that: + - (_ga_record_shape.rc | default(1)) == 0 + fail_msg: >- + {{ grafana_alloy_env_checksum_file }} exists but is not a role-owned + provenance record of the exact form " ". Refusing + to read, overwrite, or trust an unrelated /etc file. + when: _ga_record_stat.stat.exists diff --git a/ansible/roles/grafana_alloy/tasks/validate-paths.yml b/ansible/roles/grafana_alloy/tasks/validate-paths.yml index 36cf8db..9e6bb84 100644 --- a/ansible/roles/grafana_alloy/tasks/validate-paths.yml +++ b/ansible/roles/grafana_alloy/tasks/validate-paths.yml @@ -42,9 +42,9 @@ value: "{{ grafana_alloy_secret_file }}" under: /etc regex: '^/etc/[A-Za-z0-9._-]+$' - # Not removed on disable UNLESS this role authored the unit (see teardown), - # but it is both written and deleted by root-owned `copy`/`file` tasks, so it - # gets the same anchored single-segment treatment as the secret file. + # Written by the enabled path and retained beside the secret across disable. + # It authorizes later adoption/overwrite decisions, so keep the same anchored + # single-segment treatment as the secret file. - name: grafana_alloy_env_checksum_file value: "{{ grafana_alloy_env_checksum_file }}" under: /etc @@ -52,23 +52,44 @@ loop_control: label: "{{ item.name }}" -# Equal paths corrupt BOTH files' jobs: the provenance slurp would read the -# operator's credential file (token content!) back onto the control machine as -# if it were the record, and the post-converge record task would OVERWRITE the -# credential file itself with plain " " text — silently breaking -# the agent's auth on the next systemd restart. Both are direct children of /etc -# by the gates above, so equality is possible with mis-typed overrides; reject it -# before preflight reads either file. -- name: Require the provenance record and the credential file to differ +# This root-written record is read onto the controller and used to authorize a +# later rewrite. Letting inventory aim it independently at any direct child of +# /etc would turn a typo into an arbitrary-file read/overwrite. Keep it coupled to +# the already-constrained credential path as its literal `.sha256` companion. +- name: Require the provenance record to sit beside the credential file ansible.builtin.assert: that: - - grafana_alloy_env_checksum_file != grafana_alloy_secret_file + - grafana_alloy_env_checksum_file == grafana_alloy_secret_file ~ '.sha256' quiet: true fail_msg: >- - grafana_alloy_env_checksum_file ({{ grafana_alloy_env_checksum_file }}) and - grafana_alloy_secret_file ({{ grafana_alloy_secret_file }}) must be different - paths: the record is rewritten from inventory after every converge, while the - secret file holds the host's credentials — one path cannot serve both. + grafana_alloy_env_checksum_file must be exactly grafana_alloy_secret_file + + ".sha256" (expected {{ grafana_alloy_secret_file }}.sha256, got + {{ grafana_alloy_env_checksum_file }}). The record is root-read, rewritten, + and trusted as overwrite authorization, so it cannot be aimed independently + at another file under /etc. + +# The disable path recursively removes config_dir. It must never be equal to, or +# an ancestor of, either retained secret artifact or the systemd unit used as the +# ownership marker; otherwise a valid-looking override can erase the evidence that +# prevents stale-token adoption (or an entire systemd directory). +- name: Keep destructive directories away from retained Alloy files + ansible.builtin.assert: + that: + - grafana_alloy_config_dir != grafana_alloy_secret_file + - grafana_alloy_config_dir != grafana_alloy_env_checksum_file + - not grafana_alloy_secret_file.startswith(grafana_alloy_config_dir ~ '/') + - not grafana_alloy_env_checksum_file.startswith(grafana_alloy_config_dir ~ '/') + - grafana_alloy_config_dir != _ga_unit_path + - not _ga_unit_path.startswith(grafana_alloy_config_dir ~ '/') + - grafana_alloy_config_file not in + [grafana_alloy_secret_file, grafana_alloy_env_checksum_file, _ga_unit_path] + quiet: true + fail_msg: >- + grafana_alloy_config_dir/config_file must not equal or contain the retained + credential ({{ grafana_alloy_secret_file }}), provenance record + ({{ grafana_alloy_env_checksum_file }}), or managed unit ({{ _ga_unit_path }}). + The disable path removes config_dir recursively, so overlapping paths would + delete security evidence or unrelated system files. # The role templates the config INTO the directory it creates and chowns; a file # outside it would be written to an unmanaged (possibly unsafe) location and From 522d3cec32197a0909eed0d909b0982a776116cc Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Thu, 17 Sep 2026 22:42:32 +0300 Subject: [PATCH 5/8] fix(ansible): close the grafana_alloy token-path fail-open gaps MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review of #63 found four ways the inventory token path could deploy green while the agent ran on wrong or stale credentials. Token shape gate anchored on `$`, which in Python also matches just before a trailing newline — so `grafana_alloy_api_token: |` (the natural way to wrap a long token) passed a gate whose own fail_msg promises "no newline", and systemd delivered the value with a literal \n. Anchor on \Z, as roles/decdn_node does at :997/:1072/:1089. stat omits `checksum` when a file is unreadable to the connection, so `stat.checksum | default('')` made the adoption guard — the gate whose whole job is catching a vanished secret.yml — pass vacuously, and skipped the provenance record write in silence. Fail closed in the 0600 gate instead and drop the three defaults it was guarding; the record write now fails loudly rather than leaving the host permanently untracked. validate-env-record hardcoded `python3` (bypassing interpreter_python = auto_silent) and folded every non-zero rc into "your record is corrupt", whose suggested remedy disarms the guard. Use the discovered interpreter and let any rc outside {0,1} fail as itself. With no record there is nothing to compare, so a host-side token edit cannot be detected: `state: started` is a no-op on a running unit and the record is then seeded from the new bytes, hiding that edit forever. The documented hand-back procedure lands operators in exactly that state. Keep the deliberate no-restart behaviour (bouncing the fleet on the upgrade run is what it avoids) but say so, scoped to an already-running agent. Also: reject placeholder tokens carried over from the .example templates (shaped like real tokens, so nothing downstream could tell), require a real boolean for grafana_alloy_overwrite_host_file rather than let "yes"/"on"/"1" grant a credential-destroying authorisation by coercion, normalise the token's `| length` tests so an empty `grafana_alloy_api_token:` key fails in the role's own voice, and include `.msg` in the config-validation failure so a missing binary stops reporting an empty reason next to a wrong diagnosis. Co-Authored-By: Claude Opus 5 (1M context) --- ansible/roles/grafana_alloy/defaults/main.yml | 6 + ansible/roles/grafana_alloy/tasks/main.yml | 145 +++++++++++++++--- .../roles/grafana_alloy/tasks/preflight.yml | 133 ++++++++++++---- .../tasks/validate-env-record.yml | 31 +++- 4 files changed, 255 insertions(+), 60 deletions(-) diff --git a/ansible/roles/grafana_alloy/defaults/main.yml b/ansible/roles/grafana_alloy/defaults/main.yml index 5112641..b4360fb 100644 --- a/ansible/roles/grafana_alloy/defaults/main.yml +++ b/ansible/roles/grafana_alloy/defaults/main.yml @@ -68,6 +68,12 @@ grafana_alloy_env_checksum_file: /etc/grafana-alloy.env.sha256 # rewrite is token-only, any lines added outside Ansible are DISCARDED; migrate # them to their inventory variables BEFORE turning this on (preflight then stops # demanding them from the file, which also makes the migration verifiable). +# +# Turn it back to false after the one converge that needs it. Left true, this host +# has NO clobber guard: the next hand-edit to the env file is silently destroyed +# instead of stopping the deploy, which is the whole point of the record. It must +# be a real boolean — preflight rejects "yes"/"on"/"1" rather than let YAML +# truthiness grant an authorisation nobody typed. grafana_alloy_overwrite_host_file: false # --- Network (loopback-only; AGENTS.md hard rule 2) --------------------------- diff --git a/ansible/roles/grafana_alloy/tasks/main.yml b/ansible/roles/grafana_alloy/tasks/main.yml index a30c0a3..b1eaecf 100644 --- a/ansible/roles/grafana_alloy/tasks/main.yml +++ b/ansible/roles/grafana_alloy/tasks/main.yml @@ -61,6 +61,18 @@ - name: Validate the managed filesystem paths before removing them ansible.builtin.import_tasks: validate-paths.yml + # Deliberately placed a credential + a disabled flag is almost always a + # half-finished enablement, and every other task in this block is silent about + # it: the operator sees a clean run and assumes the token took effect. + - name: Note that a configured API token is inert while the flag is off + ansible.builtin.debug: + msg: >- + grafana_alloy_api_token is set, but decdn_grafana_cloud_enabled is false, + so this role tore its installation down and did NOT write + {{ grafana_alloy_secret_file }} — the token has no effect. Set + decdn_grafana_cloud_enabled: true to deploy the agent. + when: grafana_alloy_api_token | default('', true) | string | length > 0 + - name: Check for a leftover Alloy systemd unit ansible.builtin.stat: path: "{{ _ga_unit_path }}" @@ -283,17 +295,25 @@ owner: root group: root mode: "0600" - # to_json rather than hand-rolled escaping: it emits exactly the double- - # quoted, C-escaped form systemd's EnvironmentFile parser expects (and - # supplies the quotes itself), so a value containing quotes/backslashes - # survives intact. ensure_ascii=false keeps UTF-8 literal. Whitespace was - # rejected by preflight's shape gate; no_log because the token is secret. + # to_json rather than hand-rolled escaping: it supplies the surrounding + # double quotes and escapes " and \ — the only two escapes systemd's + # EnvironmentFile parser unescapes inside quotes, so a token containing + # either survives intact. ensure_ascii=false keeps UTF-8 literal. + # + # systemd's parser is shell-like, NOT a C-escape parser: \n / \t / \uXXXX + # would reach the daemon as literal backslash sequences. That is exactly + # why preflight's shape gate rejects whitespace (and, via \Z, a trailing + # newline) rather than trusting to_json to make anything safe. + # + # Keep this block scalar byte-identical to _ga_desired_inventory_content + # in preflight.yml — the interrupted-rotation branch hashes that one and + # compares it against this file. no_log because the token is secret. content: | GC_API_TOKEN={{ grafana_alloy_api_token | string | to_json(ensure_ascii=false) }} no_log: true register: _ga_env_copy notify: Restart alloy - when: grafana_alloy_api_token | length > 0 + when: grafana_alloy_api_token | default('', true) | string | length > 0 # If a prior run wrote the desired token bytes and then failed before the # final provenance write, copy is now idempotent and would not notify. Queue a @@ -309,7 +329,7 @@ changed_when: true notify: Restart alloy when: - - grafana_alloy_api_token | length > 0 + - grafana_alloy_api_token | default('', true) | string | length > 0 - _ga_secret_stat.stat.exists - _ga_desired_inventory_sum == (_ga_secret_stat.stat.checksum | default('')) - >- @@ -322,20 +342,25 @@ # so what remains here is the first converge after a pre-provenance deploy. # Gated on the task result too, so it speaks up once (when contents moved), # not on every subsequent converge. + # + # Reaching this task at all means grafana_alloy_overwrite_host_file was set: + # preflight's overwrite gate demands it for ANY untracked file, so name it — + # otherwise the message reads as if the rewrite happened unprompted. - name: Warn that an inventory grafana_alloy_api_token overwrote an untracked env file ansible.builtin.debug: msg: >- - grafana_alloy_api_token is set in inventory, so - {{ grafana_alloy_secret_file }} was REWRITTEN as a token-only file, and - this host carried no provenance record for it — any KEY=value lines added - on the host that are not covered by inventory variables are gone. From - this converge on, that file is tracked: an edit made outside Ansible fails - the next deploy instead of being silently discarded. Clear - grafana_alloy_api_token to hand the file back to the host. + grafana_alloy_api_token is set and grafana_alloy_overwrite_host_file + authorised the rewrite, so {{ grafana_alloy_secret_file }} was REWRITTEN + as a token-only file — any KEY=value lines added on the host that are not + covered by inventory variables are gone. From this converge on, that file + is tracked: an edit made outside Ansible fails the next deploy instead of + being silently discarded. Set grafana_alloy_overwrite_host_file back to + false now — leaving it true disables that guard permanently on this host. + Clear grafana_alloy_api_token to hand the file back to the host. register: _ga_untracked_overwrite_applied_report when: - not ansible_check_mode - - grafana_alloy_api_token | length > 0 + - grafana_alloy_api_token | default('', true) | string | length > 0 - _ga_secret_stat.stat.exists - _ga_recorded_sum | length == 0 - _ga_env_copy.changed | default(false) | bool @@ -350,7 +375,7 @@ register: _ga_untracked_overwrite_check_report when: - ansible_check_mode - - grafana_alloy_api_token | length > 0 + - grafana_alloy_api_token | default('', true) | string | length > 0 - _ga_secret_stat.stat.exists - _ga_recorded_sum | length == 0 - _ga_env_copy.changed | default(false) | bool @@ -393,14 +418,21 @@ failed_when: false when: not ansible_check_mode + # `.msg` is included because failed_when: false also swallows MODULE-level + # failures — a missing or non-executable binary, ENOEXEC from a wrong-arch + # build. Those carry no stderr/stdout at all, so without it this message + # renders an empty reason next to a diagnosis ("template regression") that is + # actively wrong for that case. - name: Report the configuration-validation failure ansible.builtin.fail: msg: >- `alloy validate` rejected {{ grafana_alloy_config_file }} (rc={{ _ga_config_check.rc | default('?') }}): {{ (_ga_config_check.stderr | default('', true)) - or (_ga_config_check.stdout | default('', true)) }} — fix the - template regression before deploying. + or (_ga_config_check.stdout | default('', true)) + or (_ga_config_check.msg | default('', true)) + or 'no output' }} — fix the template regression, or the binary at + {{ grafana_alloy_bin_effective }} if the reason above is an exec error. when: - not ansible_check_mode - _ga_config_check.rc | default(1) != 0 @@ -420,8 +452,12 @@ ansible.builtin.stat: path: "{{ grafana_alloy_secret_file }}" follow: false - # sha256 so the .sha256 record cross-checks against `sha256sum`; stat's - # default is sha1 and would make every documented verification lie. + # sha256 to match the algorithm named by the record's own filename; stat's + # default is sha1, which would make the .sha256 extension a lie. NOTE the + # record is " ", so `sha256sum -c` cannot read it — verify + # by hand with: + # sudo sha256sum /etc/grafana-alloy.env + # sudo cut -d' ' -f2 /etc/grafana-alloy.env.sha256 get_checksum: true checksum_algorithm: sha256 register: _ga_env_post_stat @@ -434,10 +470,59 @@ changed_when: true notify: Restart alloy when: - - grafana_alloy_api_token | length == 0 + - grafana_alloy_api_token | default('', true) | string | length == 0 - _ga_recorded_sum | length > 0 - (_ga_env_post_stat.stat.checksum | default('')) != _ga_recorded_sum + # The blind spot the gate above leaves open, made audible. + # + # With no record there is nothing to compare, so a host-side token edit cannot + # be detected: `state: started` below is a no-op on an already-running unit and + # the record is then seeded from the NEW bytes, making that edit undetectable + # on every future converge too. Staying silent about it is what turns a + # credential rotation into a green deploy that ships nothing. + # + # This is not only the fleet-upgrade run. The documented hand-back procedure + # (clear grafana_alloy_api_token, `sudo rm` the record — see the adoption + # guard's fail_msg in preflight.yml) lands an operator in exactly this state, + # and writing a fresh token is the natural next step. + # + # Deliberately a warning, not a restart: bouncing every node in the fleet on + # the upgrade run is the cost the gate above exists to avoid. Scoped to an + # agent that was ALREADY RUNNING before this converge, because a fresh install + # starts on the current file anyway and needs no warning. + # Only gathered on the narrow path that needs it — the unit is already + # templated by now, so on a fresh install this correctly reports not-running + # (the config/unit handlers start it moments later, on the current file). + - name: Check whether Alloy predates its first provenance record + ansible.builtin.service_facts: + when: + - grafana_alloy_api_token | default('', true) | string | length == 0 + - _ga_env_post_stat.stat.exists + - _ga_recorded_sum | length == 0 + + - name: Warn that out-of-band edit detection is not armed on this host yet + ansible.builtin.debug: + msg: >- + {{ grafana_alloy_secret_file }} is host-provisioned and carries no + provenance record yet, so this converge CANNOT tell whether its contents + changed since Alloy started, and will not restart the agent on that + basis. This run seeds the record; from the next converge on, an + out-of-band edit is detected and restarts Alloy automatically. If you + changed GC_API_TOKEN (or any GC_* value) as part of THIS change, the + running agent is still on the old credentials — every push will 401 while + the deploy stays green. Restart it once, now: + sudo systemctl restart alloy + when: + - grafana_alloy_api_token | default('', true) | string | length == 0 + - _ga_env_post_stat.stat.exists + - _ga_recorded_sum | length == 0 + # Total expression: `services` is absent when the gather above was skipped, + # and the unit key is absent on a host that has never run Alloy. + - >- + (ansible_facts.services | default({}, true)).get('alloy.service', {}) + .get('state', '') == 'running' + - name: Apply pending Alloy changes ansible.builtin.meta: flush_handlers @@ -483,8 +568,24 @@ # Parenthesised: filters bind tighter than comparisons, so an unbracketed # ternary parses wrong. content: > - {{ (grafana_alloy_api_token | length > 0) | ternary('inventory', 'host') }} + {{ (grafana_alloy_api_token | default('', true) | string | length > 0) | ternary('inventory', 'host') }} {{ _ga_env_post_stat.stat.checksum }} # Under `--check` on an unprovisioned host the env-file copy only SIMULATES # the write, so there is nothing to hash and stat.checksum is undefined. when: (_ga_env_post_stat.stat.checksum | default('')) | length > 0 + + # …but outside check mode a missing checksum on an existing file means stat + # could not READ it, and skipping in silence would leave the host permanently + # untracked: no adoption guard, no out-of-band detection, nothing to say why. + - name: Fail when the env file exists but could not be hashed for the record + ansible.builtin.fail: + msg: >- + {{ grafana_alloy_secret_file }} exists but could not be hashed, so the + provenance record was NOT written — the next converge would silently lose + both the adoption guard and out-of-band edit detection on this host. This + means the file is unreadable to this connection; run the role with become + (the shipped playbooks do). + when: + - not ansible_check_mode + - _ga_env_post_stat.stat.exists + - (_ga_env_post_stat.stat.checksum | default('')) | length == 0 diff --git a/ansible/roles/grafana_alloy/tasks/preflight.yml b/ansible/roles/grafana_alloy/tasks/preflight.yml index 6e9e6e6..9cb6948 100644 --- a/ansible/roles/grafana_alloy/tasks/preflight.yml +++ b/ansible/roles/grafana_alloy/tasks/preflight.yml @@ -210,8 +210,9 @@ Mimir/Loki — so it is restricted to letters, digits, dot, underscore and dash, and must not be a token or carry whitespace. Got "{{ item.value }}". NOTE the Loki and OTLP instance IDs are each their own number, distinct - from the Prometheus one; the access-policy token is shared and stays in - {{ grafana_alloy_secret_file }} as GC_API_TOKEN. + from the Prometheus one; the access-policy token is shared and lives either + in {{ grafana_alloy_secret_file }} as GC_API_TOKEN or in + grafana_alloy_api_token in a git-ignored secret.yml — never in one of these. loop: - {name: grafana_alloy_prom_username, value: "{{ grafana_alloy_prom_username | string }}"} - {name: grafana_alloy_otlp_username, value: "{{ grafana_alloy_otlp_username | string }}"} @@ -228,10 +229,13 @@ - "'@' not in item" quiet: true fail_msg: >- - A Grafana Cloud access token (or a URL with embedded credentials) appears - in an inventory variable. config.alloy is world-readable on the host and - inventory is committed: the token belongs ONLY in - {{ grafana_alloy_secret_file }} as GC_API_TOKEN. + A Grafana Cloud access token (or a URL with embedded credentials) appears in + one of these six inventory variables. They are non-secret by design and are + rendered verbatim into config.alloy, which is world-readable 0644 on the + host — and unlike host_vars//secret.yml they are normally committed. + The token has exactly two homes: GC_API_TOKEN in + {{ grafana_alloy_secret_file }} on the host, or grafana_alloy_api_token in a + git-ignored secret.yml. Neither is one of these variables. loop: - "{{ grafana_alloy_prom_url | string }}" - "{{ grafana_alloy_prom_username | string }}" @@ -242,12 +246,44 @@ loop_control: label: "{{ item | truncate(24, true) }}" +# Type gates for the two credential-path knobs, in the same spirit as the +# observability sub-switches above. grafana_alloy_overwrite_host_file is the one +# flag in this role that authorises DESTROYING an operator-provisioned credential +# file, so it must not be reachable through YAML truthiness: `| bool` turns "yes", +# "on" and "1" into an authorisation the operator never typed, and silently reads +# any typo ("treu", "Flase") as false — producing a refusal whose message cannot +# explain itself. Require a real boolean, exactly like decdn_grafana_cloud_enabled. +# +# The token is checked for type but never for value here; its shape gate is below +# and its content never reaches an assert message. +- name: Validate the credential-path knobs + ansible.builtin.assert: + that: + - grafana_alloy_overwrite_host_file is boolean + - grafana_alloy_api_token is none or grafana_alloy_api_token is string + fail_msg: >- + grafana_alloy_overwrite_host_file must be a real boolean (true/false), not a + quoted string — it authorises rewriting {{ grafana_alloy_secret_file }} over + whatever the host currently has, so "yes"/"on"/"1" must not be able to grant + it by coercion. grafana_alloy_api_token must be a string (quote it: an + all-digit token reads as an int, and a `|` block scalar appends a newline). + # The mirror image of the guard above (and the #39 relaxation): when -# grafana_alloy_api_token is SET it may be whitespace/newline-free, because the -# role JSON-encodes it into a single systemd EnvironmentFile line — quotes and -# backslashes are deliberately NOT rejected here, to_json emits exactly the -# double-quoted C-escaped form systemd's parser expects. A value with embedded -# whitespace would arrive on the daemon side truncated at the first blank. +# grafana_alloy_api_token is SET it must be whitespace/newline-free, because the +# role JSON-encodes it into a single systemd EnvironmentFile line. Quotes and +# backslashes are deliberately NOT rejected: systemd's EnvironmentFile parser +# strips the surrounding double quotes and unescapes \" and \\ — exactly the two +# escapes JSON emits for the whitespace-free tokens this gate admits. +# +# It is NOT a C-escape parser (env-file.c is shell-like): \n, \t and \uXXXX would +# arrive at the daemon as literal backslash sequences, and an embedded blank would +# truncate the value. That is why rejecting whitespace here is load-bearing rather +# than cosmetic — do not relax it on the assumption that to_json will save you. +# +# \Z, not $: Python's $ also matches just before a trailing newline, so `^[^\s]+$` +# accepts "glc_abc\n" — the exact shape a YAML block scalar (`grafana_alloy_api_token: |`) +# produces — and the fail_msg below would be lying about newlines. roles/decdn_node +# uses \Z throughout for this reason; keep the two in step. # # no_log does not exist for asserts: their output lands in plain Ansible/CI logs, # so the failure message must carry NO part of the configured value — report only @@ -255,15 +291,28 @@ - name: Validate the shape of an inventory-provided API token ansible.builtin.assert: that: - - grafana_alloy_api_token | string is match('^[^\s]+$') + - grafana_alloy_api_token | string is match('^[^\s]+\Z') + # A placeholder copied out of the .example templates is shaped exactly like a + # real token, so nothing downstream can catch it: the file is authored, the + # config validates, the unit starts, and every push 401s behind a green + # deploy. These are the literals this repo itself ships, so matching them is + # free of false positives. Case-insensitive, substring — "glc_example_token_replace_me" + # and a hand-typed "CHANGEME" are the same mistake. + - not (grafana_alloy_api_token | string | lower + is search('replace_me|replaceme|changeme|example_token|your_token|xxxxx')) quiet: true fail_msg: >- grafana_alloy_api_token must be the bare token value (non-empty, no whitespace, no newline) — it becomes the single GC_API_TOKEN="…" line of {{ grafana_alloy_secret_file }}, which systemd parses before dropping - privileges. Quotes/backslashes are fine (JSON-encoded); spaces are not. + privileges. Quotes/backslashes are fine (JSON-encoded); spaces are not, and + neither is a trailing newline — set it as an ordinary quoted scalar rather + than a `|` block scalar, which appends one. A placeholder value carried over + from one of the *.example templates is rejected too: it is shaped like a real + token, so nothing after this gate could tell the difference and every push + would 401 behind a green deploy. # Empty keeps the host-provisioned posture; the gate only guards the inventory path. - when: grafana_alloy_api_token | length > 0 + when: grafana_alloy_api_token | default('', true) | string | length > 0 # A retry after the role wrote new token bytes but failed before recording their # checksum must be distinguishable from a foreign edit. Hash the exact bytes the @@ -273,10 +322,15 @@ ansible.builtin.set_fact: _ga_desired_inventory_sum: >- {{ (_ga_desired_inventory_content | hash('sha256')) - if grafana_alloy_api_token | length > 0 else '' }} + if grafana_alloy_api_token | default('', true) | string | length > 0 else '' }} vars: - # Keep this byte-for-byte identical to the copy task's block scalar, including - # its trailing newline. Jinja string literals preserve `\\n` as two bytes. + # Keep this byte-for-byte identical to the copy task's block scalar in + # main.yml, including its trailing newline — the interrupted-rotation branch + # compares this hash against the file's, so any drift between the two makes + # every converge spuriously demand grafana_alloy_overwrite_host_file. + # A literal-clip block scalar (`|`) is used on BOTH sides rather than a quoted + # one-liner, so both render the same single trailing \n; molecule's + # grafana-cloud-token rotation case is the regression guard. _ga_desired_inventory_content: | GC_API_TOKEN={{ grafana_alloy_api_token | string | to_json(ensure_ascii=false) }} no_log: true @@ -371,7 +425,7 @@ role author the file (token-only) for you. # The inventory path authors the file itself, so absence is fine there; only # the host-provisioned posture demands an operator-created file up front. - when: grafana_alloy_api_token | length == 0 + when: grafana_alloy_api_token | default('', true) | string | length == 0 # Kept as its own gate even though absence was just handled above: with # grafana_alloy_api_token set an ABSENT file is legitimate (the role authors it), @@ -389,13 +443,22 @@ - (_ga_secret_stat.stat.pw_name | default('')) == 'root' - (_ga_secret_stat.stat.gr_name | default('')) == 'root' - _ga_secret_stat.stat.mode | string == '0600' + # stat omits `checksum` entirely when the file is not READABLE to this + # connection (it still reports exists/isreg/owner/mode). Every provenance + # comparison below is ` == stat.checksum`, so a missing checksum + # defaulted to '' would make the adoption guard — the one gate whose whole + # job is catching a vanished secret.yml — pass vacuously. Fail closed here + # instead, so the comparisons downstream can index it unguarded. + - _ga_secret_stat.stat.checksum is defined fail_msg: >- - {{ grafana_alloy_secret_file }} is not a regular file (symlink? directory?) - or not owned root:root 0600 (found owner + {{ grafana_alloy_secret_file }} is not a regular file (symlink? directory?), + not owned root:root 0600 (found owner {{ _ga_secret_stat.stat.pw_name | default('?') }}, mode - {{ _ga_secret_stat.stat.mode | default('?') }}). Fix it on the target host; - systemd expands its KEY=value pairs into the unit before dropping - privileges, so root ownership is sufficient — do not relax it. + {{ _ga_secret_stat.stat.mode | default('?') }}), or could not be hashed — + the last means it is unreadable to this connection, so run the role with + become (the playbooks do). Fix it on the target host; systemd expands its + KEY=value pairs into the unit before dropping privileges, so root ownership + is sufficient — do not relax it. when: _ga_secret_stat.stat.exists # --- Secret-file provenance ------------------------------------------------------- @@ -438,8 +501,10 @@ - name: Refuse to adopt a role-authored env file as host-provisioned ansible.builtin.assert: that: + # stat.checksum is proven defined by the 0600 gate above, which shares this + # task's `_ga_secret_stat.stat.exists` guard — no default() to fail open on. - _ga_record_source != 'inventory' - or _ga_recorded_sum != (_ga_secret_stat.stat.checksum | default('')) + or _ga_recorded_sum != _ga_secret_stat.stat.checksum fail_msg: >- grafana_alloy_api_token is empty, but {{ grafana_alloy_secret_file }} is byte-for-byte the file THIS ROLE last wrote from inventory — so the token @@ -452,7 +517,7 @@ the role's provenance record: sudo rm {{ grafana_alloy_env_checksum_file }} when: - - grafana_alloy_api_token | length == 0 + - grafana_alloy_api_token | default('', true) | string | length == 0 - _ga_secret_stat.stat.exists # The mirror direction: an inventory token REWRITES the file wholesale (and @@ -476,14 +541,16 @@ grafana_alloy_api_token to keep the host's copy or set grafana_alloy_overwrite_host_file: true to confirm the rewrite. when: - - grafana_alloy_api_token | length > 0 + - grafana_alloy_api_token | default('', true) | string | length > 0 - _ga_secret_stat.stat.exists + # Same as the adoption guard: stat.checksum is guaranteed defined here by the + # 0600 gate, which runs under the same exists condition. - >- not ( (_ga_record_source == 'inventory' - and _ga_recorded_sum == (_ga_secret_stat.stat.checksum | default(''))) + and _ga_recorded_sum == _ga_secret_stat.stat.checksum) or - (_ga_desired_inventory_sum == (_ga_secret_stat.stat.checksum | default(''))) + (_ga_desired_inventory_sum == _ga_secret_stat.stat.checksum) ) # Inventory-authoring mode makes the final env file TOKEN-ONLY, so everything @@ -508,7 +575,7 @@ logs are enabled). grafana_alloy_otlp_username stays optional — without it traces fall back to the Prometheus instance ID. # Empty keeps the host-provisioned posture, where the greps below judge the file. - when: grafana_alloy_api_token | length > 0 + when: grafana_alloy_api_token | default('', true) | string | length > 0 # systemd skips an EnvironmentFile line it cannot parse SILENTLY, so each required # key is grepped for presence + value shape directly on the host: grep keeps the @@ -536,7 +603,7 @@ - key: GC_API_TOKEN regex: ^GC_API_TOKEN=[^[:space:]#][^[:space:]]*$ var: grafana_alloy_api_token - required: "{{ grafana_alloy_api_token | string | length == 0 }}" + required: "{{ grafana_alloy_api_token | default('', true) | string | length == 0 }}" - key: GC_PROM_REMOTE_WRITE_URL regex: ^GC_PROM_REMOTE_WRITE_URL=https://[^[:space:]]+$ var: grafana_alloy_prom_url @@ -559,7 +626,7 @@ required: "{{ grafana_alloy_logs_enabled | bool and grafana_alloy_loki_username | string | length == 0 }}" loop_control: label: "{{ item.key }}" - when: (grafana_alloy_api_token | length == 0) and (item.required | bool) + when: (grafana_alloy_api_token | default('', true) | string | length == 0) and (item.required | bool) register: _ga_secret_grep - name: Report which credential key was missing or malformed @@ -603,7 +670,7 @@ register: _ga_otlp_user_present when: - grafana_alloy_otlp_username | string | length == 0 - - grafana_alloy_api_token | length == 0 + - grafana_alloy_api_token | default('', true) | string | length == 0 # Same charset as the inventory instance-ID gate, plus the optional double quotes # systemd strips off an EnvironmentFile value. @@ -634,5 +701,5 @@ grafana_alloy_otlp_username in inventory instead. when: - grafana_alloy_otlp_username | string | length == 0 - - grafana_alloy_api_token | length == 0 + - grafana_alloy_api_token | default('', true) | string | length == 0 - (_ga_otlp_user_present.rc | default(1)) == 0 diff --git a/ansible/roles/grafana_alloy/tasks/validate-env-record.yml b/ansible/roles/grafana_alloy/tasks/validate-env-record.yml index a9e9849..916cecf 100644 --- a/ansible/roles/grafana_alloy/tasks/validate-env-record.yml +++ b/ansible/roles/grafana_alloy/tasks/validate-env-record.yml @@ -25,12 +25,27 @@ # Probe the complete byte stream on the target with no stdout before slurp. A # line-oriented or NUL-record grep can accept one valid prefix and ignore trailing -# unrelated bytes; Python's fullmatch cannot. Ansible already requires Python on -# every managed target, so this adds no runtime dependency. +# unrelated bytes; Python's fullmatch cannot. +# +# Use the interpreter Ansible already discovered for this host rather than a bare +# `python3`: ansible.cfg sets interpreter_python = auto_silent, so a target whose +# interpreter is /usr/bin/python3.11 or a venv has no guarantee of a `python3` on +# the become shell's PATH — and rc=2 from a missing interpreter would otherwise be +# reported below as "your record is corrupt". +# +# The script exits 0 (matches) or 1 (does not). ANY other rc is an execution +# failure, not a verdict about the file, so it is not suppressed. - name: Check the provenance record shape on the target + vars: + # `.get()` on both, so neither an ungathered fact nor an unset inventory + # variable raises; the literal is the last resort, not the first choice. + _ga_python: >- + {{ ansible_facts.get('discovered_interpreter_python') + or hostvars[inventory_hostname].get('ansible_python_interpreter') + or '/usr/bin/python3' }} ansible.builtin.command: argv: - - python3 + - "{{ _ga_python }}" - -c - >- import pathlib, re, sys; @@ -40,7 +55,10 @@ - "{{ grafana_alloy_env_checksum_file }}" changed_when: false check_mode: false - failed_when: false + # Only the two verdict codes are ours to interpret; anything else (interpreter + # missing, EACCES, OOM, rc=126/127) fails the task in its own voice, with the + # module's real error, instead of being laundered into a corruption claim. + failed_when: _ga_record_shape.rc not in [0, 1] register: _ga_record_shape when: _ga_record_stat.stat.exists @@ -51,5 +69,8 @@ fail_msg: >- {{ grafana_alloy_env_checksum_file }} exists but is not a role-owned provenance record of the exact form " ". Refusing - to read, overwrite, or trust an unrelated /etc file. + to read, overwrite, or trust an unrelated /etc file. If this file IS the + role's record, do not delete it blindly — a lost record disables the + adoption guard; check it with: + sudo cat {{ grafana_alloy_env_checksum_file }} when: _ga_record_stat.stat.exists From 6ad9526341bb4a9364661943a1c7715e4f09d4d1 Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Thu, 17 Sep 2026 22:42:51 +0300 Subject: [PATCH 6/8] test(ansible): fix a false-positive test and cover the untested token cells MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The grafana-cloud-token verify play failed in CI on `_ga_untracked_overwrite_applied_report.skipped`. The cause is not flakiness: `ansible_check_mode` reflects ONLY the --check CLI flag, never the `check_mode:` keyword (verified at block and play level on ansible-core 2.21). Inside the dry-run block the role therefore sees False, skips its check-mode branch and takes the real-run one, so both `.skipped` predicates asserted the opposite of what they read. Assert the contract the play can actually prove — a dry run mutates neither the credential file nor the record — and document why the message wording is out of reach here (operators meet it via `make check`). grafana-cloud's out-of-band-edit play omitted grafana_alloy_batch_send_size, which converge.yml sets to 512 against a role default of 1024. The config re-rendered, notified Restart alloy on its own, and the PID assertion passed — so the play stayed green with the provenance signal it exists to test deleted. Mirror converge exactly and assert the config checksum is unchanged across the re-run, so future var drift fails instead of masking. New coverage for the cells that had none: - preflight's all-or-nothing gate, whose failure mode is a token-only env file, sys.env resolving to "", a passing `alloy validate`, a running unit and not one metric shipped — both the unconditional and the logs-gated half - both foreign-provenance cells of the overwrite gate: a record reading "host" (the normal host->inventory migration) and one whose checksum no longer matches, each driven with a token that differs from the file so the `desired == actual` escape hatch cannot mask the regression - a symlinked provenance record, mirroring ga-symlink-secret — the record's stat gate runs before the slurp, and that ordering is the whole defence against pulling an arbitrary root-only file onto the control machine - a newline-terminated token, and a stringly-typed overwrite knob - the rendered config's sys.env("GC_API_TOKEN") indirection and the OTLP username fallback, which the leak probe alone cannot prove Co-Authored-By: Claude Opus 5 (1M context) --- .../molecule/grafana-cloud-token/verify.yml | 163 +++++++++++++++++- ansible/molecule/grafana-cloud/verify.yml | 29 ++++ ansible/molecule/validation/converge.yml | 140 ++++++++++++++- 3 files changed, 325 insertions(+), 7 deletions(-) diff --git a/ansible/molecule/grafana-cloud-token/verify.yml b/ansible/molecule/grafana-cloud-token/verify.yml index 591d5aa..47cfad6 100644 --- a/ansible/molecule/grafana-cloud-token/verify.yml +++ b/ansible/molecule/grafana-cloud-token/verify.yml @@ -99,6 +99,40 @@ The GC_API_TOKEN literal appears outside the 0600 env file — config or unit rendering stopped using sys.env() indirection. + # The leak probe above only proves the token LITERAL is absent, which an empty + # or dropped credential reference satisfies just as well. On this path all five + # non-token settings are inlined from inventory and the token is the ONLY value + # still resolved through sys.env, so assert that indirection positively — + # otherwise a template regression that stopped reading GC_API_TOKEN would leave + # every assertion in this play green while all three pipelines 401. + - name: Read the rendered configuration + ansible.builtin.slurp: + src: /etc/alloy/config.alloy + register: ga_config_b64 + + - name: Assert the config still resolves the token through sys.env + ansible.builtin.assert: + that: + - ga_config is search('sys\.env\("GC_API_TOKEN"\)') + # The five inventory-supplied values are inlined, so their sys.env + # lookups must be gone — this is what makes the token-only file correct. + - ga_config is not search('sys\.env\("GC_PROM_REMOTE_WRITE_URL"\)') + - ga_config is not search('sys\.env\("GC_PROM_USERNAME"\)') + - ga_config is not search('sys\.env\("GC_OTLP_ENDPOINT"\)') + - ga_config is not search('sys\.env\("GC_LOKI_URL"\)') + - ga_config is not search('sys\.env\("GC_LOKI_USERNAME"\)') + # grafana_alloy_otlp_username is deliberately unset in this scenario, so + # traces authenticate via the Prometheus-ID fallback. With a token-only + # env file the sys.env side is guaranteed empty, making that fallback the + # only thing keeping the OTLP pipeline authenticated — pin it. + - ga_config is search('coalesce\(sys\.env\("GC_OTLP_USERNAME"\),\s*"1234567"\)') + fail_msg: >- + The rendered config no longer reads the token via sys.env("GC_API_TOKEN"), + or stopped inlining the inventory-supplied endpoints. Got: + {{ ga_config }} + vars: + ga_config: "{{ ga_config_b64.content | b64decode }}" + - name: Gather service facts ansible.builtin.service_facts: @@ -122,6 +156,8 @@ ga_overwrite_token: molecule-overwrite-token # kics-scan ignore-line (fake fixture token, not a committed secret) ga_rotated_token: molecule-rotated-token + # kics-scan ignore-line (fake fixture token, not a committed secret) + ga_gate_token: molecule-gate-probe-token ga_role_vars: &ga_role_vars decdn_grafana_cloud_enabled: true grafana_alloy_install_method: manual @@ -297,15 +333,28 @@ path: /etc/grafana-alloy.env.sha256 register: ga_record_after_check - - name: Assert dry-run reported intent without claiming or changing bytes + # Assert the CONTRACT — that a dry run mutates NOTHING — not which task ran. + # + # Deliberately no assertions on the role's check-mode WORDING here. The role + # branches its two "would discard"/"did discard" messages on + # `ansible_check_mode`, and that magic variable reflects ONLY the `--check` CLI + # flag: the `check_mode: true` keyword above (block or play level, verified on + # ansible-core 2.21) suppresses module writes but leaves `ansible_check_mode` + # False inside an included role. So from a verify play the role always takes + # its real-run branch, and any predicate on those registers asserts the + # opposite of what it reads. Operators meet that branch through `make check`, + # which does pass --check. + # + # What IS proven below is the part that matters and is reachable: modules ran + # in check mode, so the credential bytes and the absent record are untouched. + - name: Assert dry-run changed neither the credential file nor the record ansible.builtin.assert: that: - ga_after_check.content == ga_rotated.results[0].content - not ga_record_after_check.stat.exists - - _ga_untracked_overwrite_applied_report.skipped | default(false) | bool - - not (_ga_untracked_overwrite_check_report.skipped | default(false) | bool) - - (_ga_untracked_overwrite_check_report.msg | default('')) is search('WOULD rewrite') - - (_ga_untracked_overwrite_check_report.msg | default('')) is search('check mode made no change') + fail_msg: >- + A check-mode converge mutated host state: the credential file or the + provenance record moved. Dry runs must never write either. - name: Restore the saved provenance record after the dry-run fixture ansible.builtin.copy: @@ -315,6 +364,110 @@ mode: "0600" content: "{{ ga_record_before_check }}" + # --- Overwrite gate: both halves of "this role wrote it" must be load-bearing - + # The gate allows an existing file through without the opt-in only when + # (source == 'inventory' AND recorded == actual) OR desired == actual. + # The takeover case above covers an ABSENT record and the rotation case covers + # `desired == actual`. These two cover the remaining cells, each killing one + # half of the first disjunct: drop the source test and the "host" case below is + # silently adopted (clobbering a host-provisioned file that may carry GC_* keys + # the operator still needs); drop the checksum test and the "edited" case is. + # + # Both cases drive ga_gate_token, which is deliberately DIFFERENT from what the + # credential file holds at this point: the gate lets an existing file through + # when `desired == actual`, so a matching token would mask either regression. + - name: Stat the current credential file for the provenance fixtures + ansible.builtin.stat: + path: /etc/grafana-alloy.env + get_checksum: true + checksum_algorithm: sha256 + register: ga_gate_stat + + # Case 1 — record says the OPERATOR owns this file: the normal host->inventory + # migration every existing deployment performs. + - name: Stage a host-owned provenance record for the current file + ansible.builtin.copy: + dest: /etc/grafana-alloy.env.sha256 + owner: root + group: root + mode: "0600" + content: "host {{ ga_gate_stat.stat.checksum }}\n" + + - name: Default to detecting no host-record rejection + ansible.builtin.set_fact: + ga_host_record_rejected: false + + - name: Try an inventory token against a host-owned record + block: + - name: Include grafana_alloy against a host-owned record + ansible.builtin.include_role: + name: grafana_alloy + vars: + <<: *ga_role_vars + grafana_alloy_api_token: "{{ ga_gate_token }}" + rescue: + - name: Record the expected host-record rejection + ansible.builtin.set_fact: + ga_host_record_rejected: true + when: ansible_failed_task.name is match('^Require confirmation to overwrite an env file') + + # Case 2 — record says this role wrote it, but the bytes have moved since: an + # out-of-band edit to a role-authored file. + - name: Stage a role-authored record whose checksum no longer matches + ansible.builtin.copy: + dest: /etc/grafana-alloy.env.sha256 + owner: root + group: root + mode: "0600" + content: "inventory {{ '0' * 64 }}\n" + + - name: Default to detecting no stale-checksum rejection + ansible.builtin.set_fact: + ga_stale_sum_rejected: false + + - name: Try an inventory token against a stale role-authored record + block: + - name: Include grafana_alloy against a stale checksum + ansible.builtin.include_role: + name: grafana_alloy + vars: + <<: *ga_role_vars + grafana_alloy_api_token: "{{ ga_gate_token }}" + rescue: + - name: Record the expected stale-checksum rejection + ansible.builtin.set_fact: + ga_stale_sum_rejected: true + when: ansible_failed_task.name is match('^Require confirmation to overwrite an env file') + + - name: Read the credential file after both refusals + ansible.builtin.slurp: + src: /etc/grafana-alloy.env + register: ga_gate_env + + - name: Assert both foreign-provenance cells refused before touching the file + ansible.builtin.assert: + that: + - ga_host_record_rejected | bool + - ga_stale_sum_rejected | bool + # Refusal happens in preflight, before any mutation — the bytes the + # rotation case left behind must still be there, untouched by either run. + - >- + (ga_gate_env.content | b64decode).splitlines() + == ['GC_API_TOKEN=' ~ (ga_rotated_token | to_json)] + fail_msg: >- + The overwrite gate must demand grafana_alloy_overwrite_host_file for a + record that reads "host", and for one whose checksum no longer matches + the file. host-record refused={{ ga_host_record_rejected }}, + stale-checksum refused={{ ga_stale_sum_rejected }}. + + - name: Restore the inventory provenance record after the gate fixtures + ansible.builtin.copy: + dest: /etc/grafana-alloy.env.sha256 + owner: root + group: root + mode: "0600" + content: "{{ ga_record_before_check }}" + # --- Disable/re-enable: keep evidence that the stale file was role-authored -- - name: Disable the role after an inventory-authored deployment ansible.builtin.include_role: diff --git a/ansible/molecule/grafana-cloud/verify.yml b/ansible/molecule/grafana-cloud/verify.yml index 5ae5e61..b40bbde 100644 --- a/ansible/molecule/grafana-cloud/verify.yml +++ b/ansible/molecule/grafana-cloud/verify.yml @@ -350,6 +350,18 @@ # kics-scan ignore-line (fake fixture token, not a committed secret) replace: GC_API_TOKEN=molecule-host-rotated-token + - name: Checksum the rendered config before the re-run + ansible.builtin.stat: + path: /etc/alloy/config.alloy + get_checksum: true + checksum_algorithm: sha256 + register: ga_host_config_before + + # These vars MUST mirror converge.yml exactly. Any knob that differs makes the + # config template re-render, which notifies Restart alloy on its own and would + # make the PID assertion below pass even with the provenance signal deleted — + # certifying a feature this play does not exercise. grafana_alloy_batch_send_size + # is the live example: converge.yml sets 512 against a role default of 1024. - name: Re-run grafana_alloy after the host-side edit ansible.builtin.include_role: name: grafana_alloy @@ -362,6 +374,7 @@ grafana_alloy_region: US grafana_alloy_loki_url: https://logs-prod-xx.molecule.invalid/loki/api/v1/push grafana_alloy_prom_username: 1234567 + grafana_alloy_batch_send_size: 512 - name: Read Alloy's PID after the host-side credential edit ansible.builtin.command: @@ -369,11 +382,27 @@ changed_when: false register: ga_host_pid_after + - name: Checksum the rendered config after the re-run + ansible.builtin.stat: + path: /etc/alloy/config.alloy + get_checksum: true + checksum_algorithm: sha256 + register: ga_host_config_after + - name: Assert Alloy restarted onto the edited host environment ansible.builtin.assert: that: - ga_host_pid_after.stdout_lines | length == 1 - ga_host_pid_after.stdout | trim != ga_host_pid_before.stdout | trim + # The restart must be attributable to the provenance signal ALONE. If the + # config moved, this play proves nothing about out-of-band detection, so + # fail loudly on var drift rather than silently passing on the wrong cause. + - ga_host_config_after.stat.checksum == ga_host_config_before.stat.checksum + fail_msg: >- + Either Alloy did not restart after the host-side credential edit (the + out-of-band provenance signal regressed), or the rendered config changed + across the re-run — meaning the vars above have drifted from converge.yml + and the restart cannot be attributed to the credential edit. # Also mutates the converged host, so it runs after every assertion above and # before the teardown play below. diff --git a/ansible/molecule/validation/converge.yml b/ansible/molecule/validation/converge.yml index 0f5ffcf..f1ad9a5 100644 --- a/ansible/molecule/validation/converge.yml +++ b/ansible/molecule/validation/converge.yml @@ -1247,6 +1247,141 @@ - /etc/grafana-alloy.env - /etc/grafana-alloy.env.sha256 + # Python's `$` also matches just before a trailing newline, so the shape gate + # must anchor on \Z. A YAML block scalar (`grafana_alloy_api_token: |`) is the + # natural way to wrap a long token and appends exactly that newline; systemd + # would deliver it to the daemon as a literal \n and every push would 401 + # behind a green deploy. + - name: "Case ga-token-newline — trailing newline in grafana_alloy_api_token" + block: + - name: Run grafana_alloy with a newline-terminated API token variable + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + # kics-scan ignore-line (fake fixture token, not a committed secret) + grafana_alloy_api_token: "glc_molecule_trailing\n" + rescue: + - name: Record ga-token-newline rejection (only if the token-shape gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['token-shape'] }}" + when: ansible_failed_task.name is match('^Validate the shape of an inventory-provided API token') + + # The flag that authorises destroying an operator-provisioned credential file + # must not be reachable by YAML truthiness: "yes"/"on"/"1" would coerce to an + # authorisation nobody typed. + - name: "Case ga-overwrite-bool — quoted string for the overwrite opt-in" + block: + - name: Run grafana_alloy with a stringly-typed overwrite knob + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + # kics-scan ignore-line (fake fixture token, not a committed secret) + grafana_alloy_api_token: molecule-test-token + grafana_alloy_overwrite_host_file: "yes" + rescue: + - name: Record ga-overwrite-bool rejection (only if the knob-type gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['credential-knobs'] }}" + when: ansible_failed_task.name is match('^Validate the credential-path knobs') + + # An inventory token makes the env file TOKEN-ONLY, so every other connection + # value must already come from inventory. Without this gate the role authors a + # file with nothing but the token, config.alloy resolves the missing endpoint + # to "" via sys.env, `alloy validate` passes, the unit starts, every assertion + # is green — and not one metric is ever shipped. + - name: "Case ga-token-no-endpoints — inventory token without the Prometheus endpoint" + block: + - name: Run grafana_alloy with a token but no inventory endpoints + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + # kics-scan ignore-line (fake fixture token, not a committed secret) + grafana_alloy_api_token: molecule-test-token + rescue: + - name: Record ga-token-no-endpoints rejection (only if the all-or-nothing gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['token-all-or-nothing'] }}" + when: ansible_failed_task.name is match('^Demand every non-token connection setting from inventory') + + # Same gate, the conditional half: the Loki pair is only required while logs + # are enabled, and `not (…) or (…)` is exactly the shape that silently inverts. + - name: "Case ga-token-no-loki — inventory token with logs on but no Loki pair" + block: + - name: Run grafana_alloy with every endpoint but the Loki pair + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_logs_enabled: true + # kics-scan ignore-line (fake fixture token, not a committed secret) + grafana_alloy_api_token: molecule-test-token + grafana_alloy_prom_url: https://prometheus-prod-xx.molecule.invalid/api/prom/push + grafana_alloy_prom_username: "1234567" + grafana_alloy_otlp_endpoint: https://otlp-gateway-prod-xx.molecule.invalid/otlp + rescue: + - name: Record ga-token-no-loki rejection (only if the all-or-nothing gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['token-all-or-nothing'] }}" + when: ansible_failed_task.name is match('^Demand every non-token connection setting from inventory') + + # The record's stat gate runs BEFORE the slurp that reads it onto the control + # machine. That ordering is the whole defence against a symlink planted at the + # record path pulling an arbitrary root-only file into controller memory — so + # it needs its own case, not just the content validator's. + - name: "Case ga-symlink-record — symlinked provenance record" + block: + - name: Stage a complete host-provisioned credential file + ansible.builtin.copy: + dest: /etc/grafana-alloy.env + owner: root + group: root + mode: "0600" + content: | + GC_PROM_REMOTE_WRITE_URL=https://prometheus-prod-xx.molecule.invalid/api/prom/push + GC_OTLP_ENDPOINT=https://otlp-gateway-prod-xx.molecule.invalid/otlp + GC_PROM_USERNAME=999888777 + GC_LOKI_URL=https://logs-prod-xx.molecule.invalid/loki/api/v1/push + GC_LOKI_USERNAME=7654321 + GC_API_TOKEN=molecule-test-token + + - name: Stage the record symlink target outside /etc + ansible.builtin.copy: + dest: /var/lib/grafana-alloy-record-fixture + owner: root + group: root + mode: "0600" + content: "host {{ '0' * 64 }}\n" + + - name: Point the provenance-record path at the out-of-tree target + ansible.builtin.file: + src: /var/lib/grafana-alloy-record-fixture + dest: /etc/grafana-alloy.env.sha256 + state: link + force: true + + - name: Run grafana_alloy against a symlinked provenance record + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + rescue: + - name: Record ga-symlink-record rejection (only if the record stat gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['provenance-record'] }}" + when: ansible_failed_task.name is match('^Require an existing provenance record to be a protected') + always: + - name: Remove the record symlink fixtures + ansible.builtin.file: + path: "{{ item }}" + state: absent + loop: + - /etc/grafana-alloy.env + - /etc/grafana-alloy.env.sha256 + - /var/lib/grafana-alloy-record-fixture + - name: Confirm every bad value was rejected by the role's own validation ansible.builtin.assert: that: @@ -1272,6 +1407,7 @@ "alloy-root-user-alias", "alloy-root-group-alias", "host-collectors", "host-collectors", "host-collectors", "endpoint-scheme", "inventory-token", "inventory-token", "instance-id", - "otlp-username-env", "token-shape", "token-collision", - "provenance-record", + "otlp-username-env", "token-shape", "token-shape", "token-collision", + "credential-knobs", "token-all-or-nothing", "token-all-or-nothing", + "provenance-record", "provenance-record", "managed-paths", "managed-paths", "managed-paths", "managed-paths", "managed-paths"] From dfdf9abe3b35988ff7f314e2051371a9fc47aa04 Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Thu, 17 Sep 2026 22:43:04 +0300 Subject: [PATCH 7/8] docs(ansible): retire the "token is host-only" invariant this branch relaxed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit AGENTS.md still told every agent reading the repo that the API token is host-provisioned, full stop — the exact statement #63 relaxes, in the file read first. grafana-alloy.env.example contradicted itself: its header forbade carrying the token "through inventory" while line 26 of the same file now permits it. And preflight's smuggle gate justified itself with "inventory is committed", which ansible/.gitignore:9 makes untrue for the secret.yml the token rides in. Document the migration path onto the inventory path, which had none: the ordered procedure, `make check` first, and — the part that was missing everywhere — setting grafana_alloy_overwrite_host_file back to false afterwards. Left true it permanently disables the clobber guard on that host, so a later hand-edit is destroyed silently instead of stopping the deploy. Note the blind spot the hand-back direction leaves, matching the role's new warning. Also correct two counts AGENTS.md had drifted on (the collection ships three roles, not two — galaxy/build.sh:19; the suite is glob-discovered, not "six scenarios"), drop the claim that the Helm chart was "Secret-oriented from the start" when it ships no Alloy at all, and add the three new user-facing variables to the collection changelog. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 24 +++++++++----- ansible/README.md | 10 ++++-- ansible/galaxy/CHANGELOG.md | 18 ++++++++++ ansible/roles/grafana_alloy/README.md | 33 +++++++++++++++++-- .../files/grafana-alloy.env.example | 11 +++++-- 5 files changed, 80 insertions(+), 16 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 6efd051..11c2d92 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -85,11 +85,16 @@ charts/ (Alloy's in-process `node_exporter`, curated collector set, `systemd` collector scoped to the units that matter), **journald** → Grafana Cloud Loki, Alloy's own health, and the daemon's OTLP spans. Host metrics and logs carry `job="integrations/node_exporter"` - so Grafana Cloud's prebuilt Linux Server dashboards work unmodified. Only the API token - is host-provisioned (`0600 /etc/grafana-alloy.env`, read via `sys.env` — Alloy has no - `--config.expand-env`); the non-secret endpoints and the three per-service instance IDs - are inventory variables that fall back to their `GC_…` env key when empty, and preflight - refuses a token in any of them. Two hardening relaxations are conditional on the signals being on + so Grafana Cloud's prebuilt Linux Server dashboards work unmodified. The API token is the + only credential, and it has two homes: operator-provisioned on the host + (`0600 /etc/grafana-alloy.env`) or carried by `grafana_alloy_api_token` from a git-ignored + `host_vars//secret.yml`, in which case the role authors that file itself as a + token-only `EnvironmentFile` and tracks who wrote it in `.sha256` (the same + `decdn_rpc_url` dual-home pattern). Either way `config.alloy` reads it as + `sys.env("GC_API_TOKEN")` — Alloy has no `--config.expand-env`. The non-secret endpoints + and the three per-service instance IDs are inventory variables that fall back to their + `GC_…` env key when empty, and preflight refuses a token in any of them; with an inventory + token it also requires all of them, since the authored file is token-only. Two hardening relaxations are conditional on the signals being on (`ProtectHome=read-only` for correct filesystem metrics, `SupplementaryGroups= systemd-journal adm` for journal access — without which collection is silently empty); teardown is gated on the managed-by marker in the unit, so a foreign Alloy is never @@ -121,8 +126,10 @@ deploys (its targets must run from `ansible/`). `make help` lists root targets. make hooks # one-time: install pre-commit git hook (pip install pre-commit first) make lint # all pre-commit hooks on all files (hygiene, shellcheck, yamllint, markdown) make lint-ansible # vendor collections + full ansible-lint (production profile) -make molecule # containerised converge/verify of the decdn_node role — all six - # scenarios in parallel (needs Docker); cap with JOBS= +make molecule # containerised converge/verify of the decdn_node + grafana_alloy + # roles — every molecule/*/ scenario in parallel (the target + # discovers them by glob, so adding one needs no edit here); + # needs Docker, cap with JOBS= make molecule-serial # the same suite one scenario at a time (readable failure output) make lint-helm # chart: helm lint + render tests + kubeconform + schema keys (needs helm, yq, Docker) make lint-alloy # grafana_alloy: render its templates + `alloy validate` them with the real @@ -137,7 +144,8 @@ make check / deploy # deCDN node (site.yml): dry-run / provision make build / galaxy-check # stage + build the decdn.node collection, then validate it ``` -**Galaxy collection (`decdn.node`).** The two roles (`baseline` + `decdn_node`) ship as a +**Galaxy collection (`decdn.node`).** The three roles (`baseline` + `decdn_node` + +`grafana_alloy`) ship as a distributable collection. The overlay lives in `ansible/galaxy/` and is staged into a clean collection tree by `galaxy/build.sh` — there is **no** `galaxy.yml` at the `ansible/` root (that would make ansible-lint treat the deploy project as a collection). Build/validate with diff --git a/ansible/README.md b/ansible/README.md index 2378475..1be880f 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -207,8 +207,9 @@ Only the API token is a secret. Put the non-secret connection settings in invent once for the fleet (`grafana_alloy_prom_url`, `_prom_username`, `_otlp_endpoint`, `_otlp_username`, `_loki_url`, `_loki_username` — Prometheus, OTLP and Loki each have their **own** instance ID, so copy each from the portal page that names it), and -provision just the token **on each target host** (it never transits this repo or the -control machine): +provision just the token **on each target host** (on this path it never transits this +repo or the control machine; the inventory alternative below trades that for a +git-ignored `secret.yml`): ```bash umask 077 @@ -225,7 +226,10 @@ Alternatively the token itself can ride git-ignored inventory — set `decdn_rpc_url`, and the role authors `/etc/grafana-alloy.env` for you (token-only, root 0600). The authoring rewrites the file wholesale, so any hand-added `GC_*` keys must move to their inventory variables first; provenance guards fail loud on silent -adoption or clobbering. See "The API token has two homes now" in the role README. +adoption or clobbering. Migrating a host that already has a hand-provisioned file +needs `grafana_alloy_overwrite_host_file: true` for exactly one converge — set it +back to `false` afterwards, or the guard stays off on that host. See "The API token +has two homes now" in the role README for the ordered procedure in both directions. **Upgrading a host deployed before machine monitoring existed:** journald shipping is on by default and needs a Loki endpoint + instance ID the old four-key file does not carry, diff --git a/ansible/galaxy/CHANGELOG.md b/ansible/galaxy/CHANGELOG.md index 194bd33..d8344b3 100644 --- a/ansible/galaxy/CHANGELOG.md +++ b/ansible/galaxy/CHANGELOG.md @@ -8,6 +8,24 @@ collection adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) ### Added +- `grafana_alloy_api_token`: the Grafana Cloud API token may now come from a + git-ignored `host_vars//secret.yml` instead of only being operator- + provisioned on the host, the same dual-home pattern `decdn_rpc_url` uses. Set, the + role authors `/etc/grafana-alloy.env` itself as a **token-only** file at + `root:root 0600`; left empty (the default), behaviour is unchanged. Because the + authored file is token-only, every other connection setting must then come from + inventory — preflight demands them before any mutation. +- `grafana_alloy_env_checksum_file` (default `/etc/grafana-alloy.env.sha256`): a + root-owned `0600` provenance record, ` `, written after the agent + is running on that content. It lets a later converge tell an operator-owned file + from a role-authored one, and drives two fail-loud guards — refusing to adopt a + role-authored file as host-provisioned when the token goes missing from the + control machine, and refusing to clobber a file this role did not write. Must + remain exactly `.sha256`. +- `grafana_alloy_overwrite_host_file` (default `false`): explicit opt-in to rewrite + an env file of foreign or unknown provenance. Required for one converge when + migrating an existing host onto the inventory path; set it back to `false` + afterwards or the guard stays disabled on that host. - `baseline_packages` now includes `acl`. `decdn_node` runs two tasks as the unprivileged `decdn` user (`decdn key-gen`, and the `decdn config validate` gate), and on Debian Ansible needs ACL support to hand the temp module file to that user. diff --git a/ansible/roles/grafana_alloy/README.md b/ansible/roles/grafana_alloy/README.md index b50a97d..6c1fa67 100644 --- a/ansible/roles/grafana_alloy/README.md +++ b/ansible/roles/grafana_alloy/README.md @@ -115,9 +115,36 @@ the host-provisioned path restarts the agent with a note, same as `decdn_node`. Disabling the role intentionally retains both the secret env file **and** its provenance record; retaining only the file would let a later re-enable with a missing `secret.yml` misclassify the stale role-authored token as host-owned. -Returning a host to hand-provisioned mode: clear the variable, then discard the -provenance record (`sudo rm /etc/grafana-alloy.env.sha256`). The Helm chart is -unaffected (it was Secret-oriented from the start). +The Helm chart ships no Alloy at all, so nothing changes there. + +#### Migrating an existing host ONTO the inventory path + +The overwrite guard refuses this by design, because the rewrite is token-only and +the role cannot read the file to tell you what it would destroy. Do it in this +order: + +1. Move every non-token `GC_*` key in `/etc/grafana-alloy.env` to its + `grafana_alloy_*` inventory variable. Preflight then stops demanding those keys + from the file, which is what makes the migration verifiable rather than hopeful. +2. Set `grafana_alloy_api_token` in the git-ignored `host_vars//secret.yml`. +3. Run `make check LIMIT=` **first**. The dry run names exactly what a real + converge would discard, and changes nothing. +4. Set `grafana_alloy_overwrite_host_file: true` and deploy once. +5. Set it back to `false`. Leaving it `true` permanently disables the guard on that + host, so a future hand-edit is silently destroyed instead of stopping the deploy. + +#### Returning a host TO hand-provisioned mode + +Clear `grafana_alloy_api_token`, then discard the provenance record +(`sudo rm /etc/grafana-alloy.env.sha256`) **before** the next converge — do it +after and the adoption guard hard-fails the deploy. Any `GC_*` keys you want back +on the host must be re-added by hand; the token-only file carries none of them. + +Note the one blind spot this leaves: with the record gone, the next converge has +nothing to compare against, so a token you edit on the host *in the same change* +is not detected and the running agent keeps the old credentials. The role warns +when it hits that state — restart once (`sudo systemctl restart alloy`) if you +rotated the token as part of the hand-back. `GC_API_TOKEN` values in plain committed inventory remain rejected: the rendered `/etc/alloy/config.alloy` is world-readable, and preflight rejects a value that diff --git a/ansible/roles/grafana_alloy/files/grafana-alloy.env.example b/ansible/roles/grafana_alloy/files/grafana-alloy.env.example index 9515a2d..94d440d 100644 --- a/ansible/roles/grafana_alloy/files/grafana-alloy.env.example +++ b/ansible/roles/grafana_alloy/files/grafana-alloy.env.example @@ -1,11 +1,18 @@ # Template for /etc/grafana-alloy.env (root-owned 0600). # -# Provision it ON THE TARGET HOST — this file holds your Grafana Cloud API token -# and must never be committed or carried through inventory: +# This file holds your Grafana Cloud API token and must NEVER be committed. +# Provision it ON THE TARGET HOST: # # umask 077 # sudo install -m 600 -o root -g root grafana-alloy.env /etc/grafana-alloy.env # +# …or skip this file entirely and let inventory author it: set +# grafana_alloy_api_token in a git-ignored host_vars//secret.yml and the +# role writes a token-only version of /etc/grafana-alloy.env for you. That is the +# ONE place a credential may ride inventory — a git-ignored secret.yml, never a +# committed group_vars file. See the GC_API_TOKEN block below and +# roles/grafana_alloy/README.md §Credentials for which path to pick. +# # MINIMUM CONTENT for a NEW host: the token, and nothing else. Every other value # below is NON-secret and can be set once in inventory (group_vars) instead of # being re-typed on every host — see the grafana_alloy_* variables named in each From 0eb2d5551732c31db36635473e60a9a4497e3596 Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Thu, 17 Sep 2026 22:43:13 +0300 Subject: [PATCH 8/8] ci(ansible): enable pipelining so credentials skip the target tmpdir MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two roles write secrets with `copy: content=…` — the node's decdn.env and, as of this branch, Alloy's grafana-alloy.env. Without pipelining the module arguments, plaintext token included, are staged into ~/.ansible/tmp on the target before the atomic move. `no_log` covers task output, not that file, so AGENTS.md hard rule 1 was leaning on a gap the inventory token path widens. Molecule already sets ANSIBLE_PIPELINING per scenario; production did not. Requires requiretty off in sudoers, which is the default on the Debian/Ubuntu targets this project supports. Co-Authored-By: Claude Opus 5 (1M context) --- ansible/ansible.cfg | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/ansible/ansible.cfg b/ansible/ansible.cfg index 8dfc06f..17a24b2 100644 --- a/ansible/ansible.cfg +++ b/ansible/ansible.cfg @@ -10,6 +10,14 @@ stdout_callback = default result_format = yaml nocows = True interpreter_python = auto_silent +# Run modules over the existing SSH session instead of staging a script into the +# target's tmpdir. Two roles write credentials with `copy: content=…` (the node's +# decdn.env, Alloy's grafana-alloy.env), and without pipelining the module args — +# including the plaintext secret — are written to ~/.ansible/tmp on the target +# before the atomic move. `no_log` covers task output, not that file. Requires +# `requiretty` to be off in sudoers, which is the default on the Debian/Ubuntu +# targets this project supports. +pipelining = True [privilege_escalation] become = True