diff --git a/AGENTS.md b/AGENTS.md index 6efd051..11c2d92 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -85,11 +85,16 @@ charts/ (Alloy's in-process `node_exporter`, curated collector set, `systemd` collector scoped to the units that matter), **journald** → Grafana Cloud Loki, Alloy's own health, and the daemon's OTLP spans. Host metrics and logs carry `job="integrations/node_exporter"` - so Grafana Cloud's prebuilt Linux Server dashboards work unmodified. Only the API token - is host-provisioned (`0600 /etc/grafana-alloy.env`, read via `sys.env` — Alloy has no - `--config.expand-env`); the non-secret endpoints and the three per-service instance IDs - are inventory variables that fall back to their `GC_…` env key when empty, and preflight - refuses a token in any of them. Two hardening relaxations are conditional on the signals being on + so Grafana Cloud's prebuilt Linux Server dashboards work unmodified. The API token is the + only credential, and it has two homes: operator-provisioned on the host + (`0600 /etc/grafana-alloy.env`) or carried by `grafana_alloy_api_token` from a git-ignored + `host_vars//secret.yml`, in which case the role authors that file itself as a + token-only `EnvironmentFile` and tracks who wrote it in `.sha256` (the same + `decdn_rpc_url` dual-home pattern). Either way `config.alloy` reads it as + `sys.env("GC_API_TOKEN")` — Alloy has no `--config.expand-env`. The non-secret endpoints + and the three per-service instance IDs are inventory variables that fall back to their + `GC_…` env key when empty, and preflight refuses a token in any of them; with an inventory + token it also requires all of them, since the authored file is token-only. Two hardening relaxations are conditional on the signals being on (`ProtectHome=read-only` for correct filesystem metrics, `SupplementaryGroups= systemd-journal adm` for journal access — without which collection is silently empty); teardown is gated on the managed-by marker in the unit, so a foreign Alloy is never @@ -121,8 +126,10 @@ deploys (its targets must run from `ansible/`). `make help` lists root targets. make hooks # one-time: install pre-commit git hook (pip install pre-commit first) make lint # all pre-commit hooks on all files (hygiene, shellcheck, yamllint, markdown) make lint-ansible # vendor collections + full ansible-lint (production profile) -make molecule # containerised converge/verify of the decdn_node role — all six - # scenarios in parallel (needs Docker); cap with JOBS= +make molecule # containerised converge/verify of the decdn_node + grafana_alloy + # roles — every molecule/*/ scenario in parallel (the target + # discovers them by glob, so adding one needs no edit here); + # needs Docker, cap with JOBS= make molecule-serial # the same suite one scenario at a time (readable failure output) make lint-helm # chart: helm lint + render tests + kubeconform + schema keys (needs helm, yq, Docker) make lint-alloy # grafana_alloy: render its templates + `alloy validate` them with the real @@ -137,7 +144,8 @@ make check / deploy # deCDN node (site.yml): dry-run / provision make build / galaxy-check # stage + build the decdn.node collection, then validate it ``` -**Galaxy collection (`decdn.node`).** The two roles (`baseline` + `decdn_node`) ship as a +**Galaxy collection (`decdn.node`).** The three roles (`baseline` + `decdn_node` + +`grafana_alloy`) ship as a distributable collection. The overlay lives in `ansible/galaxy/` and is staged into a clean collection tree by `galaxy/build.sh` — there is **no** `galaxy.yml` at the `ansible/` root (that would make ansible-lint treat the deploy project as a collection). Build/validate with diff --git a/ansible/README.md b/ansible/README.md index 278c905..1be880f 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -207,8 +207,9 @@ Only the API token is a secret. Put the non-secret connection settings in invent once for the fleet (`grafana_alloy_prom_url`, `_prom_username`, `_otlp_endpoint`, `_otlp_username`, `_loki_url`, `_loki_username` — Prometheus, OTLP and Loki each have their **own** instance ID, so copy each from the portal page that names it), and -provision just the token **on each target host** (it never transits this repo or the -control machine): +provision just the token **on each target host** (on this path it never transits this +repo or the control machine; the inventory alternative below trades that for a +git-ignored `secret.yml`): ```bash umask 077 @@ -220,6 +221,16 @@ Each of those variables is optional: left empty, the value is read from the matc `GC_…` key in that file instead (`roles/grafana_alloy/files/grafana-alloy.env.example` lists them all; URLs must be https). +Alternatively the token itself can ride git-ignored inventory — set +`grafana_alloy_api_token: "glc_…"` in `host_vars//secret.yml`, same channel as +`decdn_rpc_url`, and the role authors `/etc/grafana-alloy.env` for you (token-only, +root 0600). The authoring rewrites the file wholesale, so any hand-added `GC_*` keys +must move to their inventory variables first; provenance guards fail loud on silent +adoption or clobbering. Migrating a host that already has a hand-provisioned file +needs `grafana_alloy_overwrite_host_file: true` for exactly one converge — set it +back to `false` afterwards, or the guard stays off on that host. See "The API token +has two homes now" in the role README for the ordered procedure in both directions. + **Upgrading a host deployed before machine monitoring existed:** journald shipping is on by default and needs a Loki endpoint + instance ID the old four-key file does not carry, so preflight fails until you either set `grafana_alloy_loki_url` / `_loki_username` in @@ -254,9 +265,11 @@ make lint-alloy # render grafana_alloy's templates, then `alloy validate` th `make molecule` runs every scenario under `molecule/`: **`default`** (described below), `schema` (config key-set drift against the upstream field list), `validation` (bad knobs, for both roles, must be rejected by their own asserts), `generate-keystore` (opt-in -host-side wallet), `host-env` (host-provisioned `/etc/decdn/decdn.env`) and +host-side wallet), `host-env` (host-provisioned `/etc/decdn/decdn.env`), `slow-readiness` (advisory `/metrics` probe timeout), `grafana-cloud` (the opt-in -observability wiring — see [Grafana Cloud observability](#grafana-cloud-observability-opt-in)). +observability wiring — see [Grafana Cloud observability](#grafana-cloud-observability-opt-in)) +and `grafana-cloud-token` (the same wiring with the API token carried through +git-ignored inventory instead: role-authored env file + provenance record). They are independent, so they run concurrently — ~151s instead of ~595s — and each line of output is prefixed with its scenario name because the runs interleave. `make molecule-serial` is the escape hatch when that @@ -301,7 +314,8 @@ the RPC URL when it is not provisioned on the host instead). Highlights: | `decdn_rpc_url` + 3 contract addresses | `""` | **required** per node — `rpc_url` from a host-provisioned `0600 /etc/decdn/decdn.env` (preferred) *or* `host_vars//secret.yml`, addresses in `main.yml`; sourced from an ADR/deployment. | | `decdn_region` / `decdn_bind_port` / `decdn_rate_per_mb` | `""` / `4433` / `10` | node identity, QUIC port, USDC base units/MB. | | `decdn_env_checksum_file` / `decdn_env_overwrite_host_file` | `/etc/decdn/.decdn.env.sha256` / `false` | Provenance record for the secret env file (`0600 root`), and the opt-in that lets an inventory `decdn_rpc_url` overwrite a host-edited one. | -| `decdn_grafana_cloud_enabled` | `false` | ONE mirrored knob (identical default in both roles) wiring on Grafana Cloud observability: installs + configures `grafana_alloy` — node metrics, machine metrics, journald, agent health — AND injects `otlp_endpoint` into `node.toml`. Only the API token is provisioned per host; the rest are inventory variables. Label/cost guardrails in the role README. | +| `decdn_grafana_cloud_enabled` | `false` | ONE mirrored knob (identical default in both roles) wiring on Grafana Cloud observability: installs + configures `grafana_alloy` — node metrics, machine metrics, journald, agent health — AND injects `otlp_endpoint` into `node.toml`. Only the API token is provisioned per host *or* carried by `grafana_alloy_api_token` in git-ignored inventory; the rest are inventory variables. Label/cost guardrails in the role README. | +| `grafana_alloy_api_token` / `_env_checksum_file` / `_overwrite_host_file` | `""` / `/etc/grafana-alloy.env.sha256` / `false` | The dual-homed Grafana Cloud token and its provenance machinery (`#39` parity with the row above); the record path is fixed to `.sha256` and survives disable with the secret. See [`roles/grafana_alloy/README.md`](roles/grafana_alloy/README.md). | --- diff --git a/ansible/ansible.cfg b/ansible/ansible.cfg index 8dfc06f..17a24b2 100644 --- a/ansible/ansible.cfg +++ b/ansible/ansible.cfg @@ -10,6 +10,14 @@ stdout_callback = default result_format = yaml nocows = True interpreter_python = auto_silent +# Run modules over the existing SSH session instead of staging a script into the +# target's tmpdir. Two roles write credentials with `copy: content=…` (the node's +# decdn.env, Alloy's grafana-alloy.env), and without pipelining the module args — +# including the plaintext secret — are written to ~/.ansible/tmp on the target +# before the atomic move. `no_log` covers task output, not that file. Requires +# `requiretty` to be off in sudoers, which is the default on the Debian/Ubuntu +# targets this project supports. +pipelining = True [privilege_escalation] become = True diff --git a/ansible/galaxy/CHANGELOG.md b/ansible/galaxy/CHANGELOG.md index 194bd33..d8344b3 100644 --- a/ansible/galaxy/CHANGELOG.md +++ b/ansible/galaxy/CHANGELOG.md @@ -8,6 +8,24 @@ collection adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) ### Added +- `grafana_alloy_api_token`: the Grafana Cloud API token may now come from a + git-ignored `host_vars//secret.yml` instead of only being operator- + provisioned on the host, the same dual-home pattern `decdn_rpc_url` uses. Set, the + role authors `/etc/grafana-alloy.env` itself as a **token-only** file at + `root:root 0600`; left empty (the default), behaviour is unchanged. Because the + authored file is token-only, every other connection setting must then come from + inventory — preflight demands them before any mutation. +- `grafana_alloy_env_checksum_file` (default `/etc/grafana-alloy.env.sha256`): a + root-owned `0600` provenance record, ` `, written after the agent + is running on that content. It lets a later converge tell an operator-owned file + from a role-authored one, and drives two fail-loud guards — refusing to adopt a + role-authored file as host-provisioned when the token goes missing from the + control machine, and refusing to clobber a file this role did not write. Must + remain exactly `.sha256`. +- `grafana_alloy_overwrite_host_file` (default `false`): explicit opt-in to rewrite + an env file of foreign or unknown provenance. Required for one converge when + migrating an existing host onto the inventory path; set it back to `false` + afterwards or the guard stays disabled on that host. - `baseline_packages` now includes `acl`. `decdn_node` runs two tasks as the unprivileged `decdn` user (`decdn key-gen`, and the `decdn config validate` gate), and on Debian Ansible needs ACL support to hand the temp module file to that user. diff --git a/ansible/inventory/host_vars/decdn-node-1/secret.yml.example b/ansible/inventory/host_vars/decdn-node-1/secret.yml.example index edf08b0..5593255 100644 --- a/ansible/inventory/host_vars/decdn-node-1/secret.yml.example +++ b/ansible/inventory/host_vars/decdn-node-1/secret.yml.example @@ -30,3 +30,13 @@ decdn_rpc_url: "https://sepolia-rollup.arbitrum.io/rpc" # AWS_SESSION_TOKEN: "..." # only for assume-role / SSO flows # # Pair that with decdn_cache_origin_s3_use_default_chain: true in main.yml. + +# --- Grafana Cloud (only when decdn_grafana_cloud_enabled is on) --------------- +# +# The opt-in observability agent accepts its API token through the same channel: +# set grafana_alloy_api_token here and the role authors /etc/grafana-alloy.env on +# the host for you (token-only, root 0600), skipping the hand-provisioning shown +# in roles/grafana_alloy/files/grafana-alloy.env.example. SENSITIVE — leave it +# commented out to keep the operator-provisioned host path instead. +# +# grafana_alloy_api_token: "glc_example_token_replace_me" diff --git a/ansible/molecule/grafana-cloud-token/converge.yml b/ansible/molecule/grafana-cloud-token/converge.yml new file mode 100644 index 0000000..8414e9d --- /dev/null +++ b/ansible/molecule/grafana-cloud-token/converge.yml @@ -0,0 +1,38 @@ +--- +# Token-via-inventory converge: every non-secret connection setting comes from +# this playbook's vars (the normal committed inventory half) and the token rides +# grafana_alloy_api_token — in a real deployment it would live in git-ignored +# inventory/host_vars//secret.yml, exactly like decdn_rpc_url. Nothing +# stages /etc/grafana-alloy.env; the role must author it itself. +# +# Logs stay enabled with BOTH Loki knobs supplied from inventory so the authored +# file needs zero GC_* keys beyond the token — the all-or-nothing contract has +# nothing to clobber on these throwaway containers. +- name: Converge + hosts: all + become: true + vars: + stub_bin: "{{ lookup('ansible.builtin.env', 'MOLECULE_SCENARIO_DIRECTORY') }}/files/alloy-stub" + decdn_grafana_cloud_enabled: true + grafana_alloy_install_method: manual + grafana_alloy_manual_bin_src: "{{ stub_bin }}" + grafana_alloy_deployment_environment: production + grafana_alloy_region: US + + # --- The point of this scenario ------------------------------------------------- + # Deliberately fake token literal below, not a committed secret — keep only + # KICS' hardcoded-credential query quiet about exactly that one line. + # kics-scan ignore-line + grafana_alloy_api_token: molecule-test-token + + # All six non-secret settings satisfied by inventory, so preflight requires no + # env-file key but GC_API_TOKEN... which the variable now covers too. + grafana_alloy_prom_url: https://prometheus-prod-xx.molecule.invalid/api/prom/push + grafana_alloy_prom_username: "1234567" + grafana_alloy_otlp_endpoint: https://otlp-gateway-prod-xx.molecule.invalid/otlp + grafana_alloy_loki_url: https://logs-prod-xx.molecule.invalid/loki/api/v1/push + grafana_alloy_loki_username: "7654321" + + roles: + - role: grafana_alloy + tags: [observability] diff --git a/ansible/molecule/grafana-cloud-token/files/alloy-stub b/ansible/molecule/grafana-cloud-token/files/alloy-stub new file mode 100755 index 0000000..58629ec --- /dev/null +++ b/ansible/molecule/grafana-cloud-token/files/alloy-stub @@ -0,0 +1,31 @@ +#!/bin/sh +# Molecule stub standing in for the Grafana Alloy binary in CI. +# +# The grafana-cloud scenario runs the real role logic (user/group/dirs, secret +# gate, template rendering, hardened unit, systemd start) against this shim so no +# ~160 MB release download is needed per CI run. It intentionally does NOT listen +# on any socket: runtime socket assertions are out of scope for containerised CI +# (see ../verify.yml comments) and remain covered by real-host deploys + content- +# level checks here. +set -u + +case "${1-}" in + # Version backstop in tasks/install.yml (manual mode degrades to rc==0 anyway). + --version) + echo "alloy, version v${MOLECULE_STUB_VERSION:-0.0.0}-molecule-stub" + ;; + # validate: the real binary builds the component graph and rejects an + # unloadable config. A stub CANNOT do that, so this exits 0 and proves the + # role's gate is wired up — nothing more. Content correctness is asserted + # textually in verify.yml and, against the REAL pinned binary, by + # ../../tests/alloy-config/validate.sh (`make lint-alloy`, run in CI). Do not + # read a green run here as "the configuration loads". + validate | fmt) + : + ;; + # Long-running foreground process keeps the Type=simple unit 'running'. + run) + exec sleep infinity + ;; +esac +exit 0 diff --git a/ansible/molecule/grafana-cloud-token/molecule.yml b/ansible/molecule/grafana-cloud-token/molecule.yml new file mode 100644 index 0000000..2cb78f1 --- /dev/null +++ b/ansible/molecule/grafana-cloud-token/molecule.yml @@ -0,0 +1,43 @@ +--- +# Inventory-path exercise of the #39 relaxation: grafana_alloy_api_token set in +# (git-ignored) inventory instead of a host-provisioned /etc/grafana-alloy.env. +# Positive path only — the broken-input matrix lives in ../validation; the +# host-provisioned positive path lives in ../grafana-cloud. No prepare-stage env +# file AT ALL here by design: the absence is what proves preflight's existence +# gate honours the token variable, and that the role authors the file itself. +# +# Same two boundaries as ../grafana-cloud: the stub binds no sockets and exits 0 +# for every subcommand, so nothing here proves runtime loadability (`make +# lint-alloy` does that against the real pinned binary). +driver: + name: docker +platforms: + - name: decdn-node-grafana-cloud-token + # Same digest-pinned image as the other scenarios (repo convention). Re-resolve + # all of them together to bump. + image: geerlingguy/docker-debian12-ansible@sha256:4553092be2c00b1ffe580927b9ff03f3c3a0df32b7dd693a3eb02efb6c2b77b7 + pre_build_image: true + command: /usr/lib/systemd/systemd + privileged: true + cgroupns_mode: host + volumes: + - /sys/fs/cgroup:/sys/fs/cgroup:rw +provisioner: + name: ansible + env: + ANSIBLE_ROLES_PATH: "${MOLECULE_PROJECT_DIRECTORY}/roles" + ANSIBLE_COLLECTIONS_PATH: "${MOLECULE_PROJECT_DIRECTORY}/collections" + ANSIBLE_PIPELINING: "true" +verifier: + name: ansible +scenario: + test_sequence: + # Leading `destroy` so an orphaned container is never reused — see + # ../default/molecule.yml. No prepare playbook: unlike ../grafana-cloud there + # are NO host-provisioned fixtures to stage — their absence is the point. + - destroy + - create + - converge + - idempotence + - verify + - destroy diff --git a/ansible/molecule/grafana-cloud-token/verify.yml b/ansible/molecule/grafana-cloud-token/verify.yml new file mode 100644 index 0000000..47cfad6 --- /dev/null +++ b/ansible/molecule/grafana-cloud-token/verify.yml @@ -0,0 +1,512 @@ +--- +# Verify the inventory-path authoring: a role-authored token-only 0600 env file, +# its provenance record, no secret literals outside it, and a running agent. +# Content assertions are grep-shape only against values this repo already knew +# (the molecule token) — mirroring how the role itself treats the host path. +- name: Verify + hosts: all + become: true + tasks: + - name: Stat the artifacts the run must have produced + ansible.builtin.stat: + path: "{{ item.path }}" + # Must match the checksum algorithm the role records (stat defaults to sha1). + get_checksum: true + checksum_algorithm: sha256 + register: ga_files + # config.alloy is root:alloy by design (the agent user reads its own config); + # the credential file and its provenance record are strictly root:root. + loop: + - {path: /etc/grafana-alloy.env, mode: "0600", grp: root} + - {path: /etc/grafana-alloy.env.sha256, mode: "0600", grp: root} + - {path: /etc/alloy/config.alloy, mode: "0644", grp: alloy} + + - name: Assert ownership/modes of the artifacts + ansible.builtin.assert: + that: + # exists-and-mode first so a missing file short-circuits to fail_msg + # instead of raising on absent stat fields. + - item.stat.exists and item.stat.mode == item.item.mode + - item.stat.pw_name == 'root' + - item.stat.gr_name == item.item.grp + # Regular file: the authored credential must not be a symlink, same as + # preflight demands from operator-provisioned ones. + - item.stat.isreg | default(false) + loop: "{{ ga_files.results }}" + loop_control: + label: "{{ item.item.path }}" + + - name: Read the authored credential file back + ansible.builtin.slurp: + src: /etc/grafana-alloy.env + register: ga_env_b64 + + - name: Assert the env file is exactly one JSON-encoded token line + ansible.builtin.assert: + that: + # to_json always double-quotes a plain string; systemd strips the outer + # quotes when expanding EnvironmentFile=. + - >- + ga_lines | length == 1 + - >- + ga_lines[0] == 'GC_API_TOKEN="' ~ (ga_expected_token) ~ '"' + - >- + ga_body.count('GC_') == 1 + fail_msg: >- + The authored env file must be a single GC_API_TOKEN="" line. + Got: {{ ga_body }} + vars: + # kics-scan ignore-line (fake fixture token, not a committed secret) + ga_expected_token: molecule-test-token + ga_body: "{{ ga_env_b64.content | b64decode }}" + ga_lines: "{{ (ga_env_b64.content | b64decode).splitlines() }}" + + - name: Read the provenance record back + ansible.builtin.slurp: + src: /etc/grafana-alloy.env.sha256 + register: ga_record_b64 + + - name: Resolve the record's two fields + ansible.builtin.set_fact: + ga_record_fields: "{{ (ga_record_b64.content | b64decode | trim).split() }}" + + - name: Assert the record says inventory-authored and matches the live file + ansible.builtin.assert: + that: + - ga_record_fields | length == 2 + - ga_record_fields[0] == 'inventory' + - ga_record_fields[1] == (ga_files.results[0].stat.checksum) + fail_msg: >- + The provenance record must read " ". + Got: {{ ga_record_fields }} + + # --- Secret hygiene -------------------------------------------------------------- + - name: Register the secret-token absence across non-secret files + ansible.builtin.command: + cmd: grep -Rq molecule-test-token {{ item }} + changed_when: false + failed_when: false + loop: + - /etc/systemd/system/alloy.service + - /etc/alloy/config.alloy + register: ga_leak_probe + + - name: Assert no secret literal leaked into non-secret artifacts + ansible.builtin.assert: + that: + - ga_leak_probe.results | selectattr('rc', 'equalto', 0) | list | length == 0 + fail_msg: >- + The GC_API_TOKEN literal appears outside the 0600 env file — config or + unit rendering stopped using sys.env() indirection. + + # The leak probe above only proves the token LITERAL is absent, which an empty + # or dropped credential reference satisfies just as well. On this path all five + # non-token settings are inlined from inventory and the token is the ONLY value + # still resolved through sys.env, so assert that indirection positively — + # otherwise a template regression that stopped reading GC_API_TOKEN would leave + # every assertion in this play green while all three pipelines 401. + - name: Read the rendered configuration + ansible.builtin.slurp: + src: /etc/alloy/config.alloy + register: ga_config_b64 + + - name: Assert the config still resolves the token through sys.env + ansible.builtin.assert: + that: + - ga_config is search('sys\.env\("GC_API_TOKEN"\)') + # The five inventory-supplied values are inlined, so their sys.env + # lookups must be gone — this is what makes the token-only file correct. + - ga_config is not search('sys\.env\("GC_PROM_REMOTE_WRITE_URL"\)') + - ga_config is not search('sys\.env\("GC_PROM_USERNAME"\)') + - ga_config is not search('sys\.env\("GC_OTLP_ENDPOINT"\)') + - ga_config is not search('sys\.env\("GC_LOKI_URL"\)') + - ga_config is not search('sys\.env\("GC_LOKI_USERNAME"\)') + # grafana_alloy_otlp_username is deliberately unset in this scenario, so + # traces authenticate via the Prometheus-ID fallback. With a token-only + # env file the sys.env side is guaranteed empty, making that fallback the + # only thing keeping the OTLP pipeline authenticated — pin it. + - ga_config is search('coalesce\(sys\.env\("GC_OTLP_USERNAME"\),\s*"1234567"\)') + fail_msg: >- + The rendered config no longer reads the token via sys.env("GC_API_TOKEN"), + or stopped inlining the inventory-supplied endpoints. Got: + {{ ga_config }} + vars: + ga_config: "{{ ga_config_b64.content | b64decode }}" + + - name: Gather service facts + ansible.builtin.service_facts: + + - name: Assert alloy picked up the authored credentials (running, not crash-looping) + ansible.builtin.assert: + that: + - "'alloy.service' in ansible_facts.services" + - ansible_facts.services['alloy.service'].state == 'running' + +# Mutates the converged host deliberately, after all baseline assertions above. +# This is the provenance state-machine regression suite: unknown-file refusal, +# explicit takeover, interrupted-rotation retry, process restart, and the +# disable→enable missing-secret guard all need real host-side behavior. +- name: Exercise token provenance recovery and teardown + hosts: all + become: true + vars: + ga_stub_bin: >- + {{ lookup('ansible.builtin.env', 'MOLECULE_SCENARIO_DIRECTORY') }}/files/alloy-stub + # kics-scan ignore-line (fake fixture token, not a committed secret) + ga_overwrite_token: molecule-overwrite-token + # kics-scan ignore-line (fake fixture token, not a committed secret) + ga_rotated_token: molecule-rotated-token + # kics-scan ignore-line (fake fixture token, not a committed secret) + ga_gate_token: molecule-gate-probe-token + ga_role_vars: &ga_role_vars + decdn_grafana_cloud_enabled: true + grafana_alloy_install_method: manual + grafana_alloy_manual_bin_src: "{{ ga_stub_bin }}" + grafana_alloy_deployment_environment: production + grafana_alloy_region: US + grafana_alloy_prom_url: https://prometheus-prod-xx.molecule.invalid/api/prom/push + grafana_alloy_prom_username: "1234567" + grafana_alloy_otlp_endpoint: https://otlp-gateway-prod-xx.molecule.invalid/otlp + grafana_alloy_loki_url: https://logs-prod-xx.molecule.invalid/loki/api/v1/push + grafana_alloy_loki_username: "7654321" + tasks: + # --- Unknown provenance: fail before destroying host-managed keys ---------- + - name: Remove the provenance record to model a pre-record deployment + ansible.builtin.file: + path: /etc/grafana-alloy.env.sha256 + state: absent + + - name: Stage an untracked host-managed credential file + ansible.builtin.copy: + dest: /etc/grafana-alloy.env + owner: root + group: root + mode: "0600" + content: | + GC_API_TOKEN=molecule-untracked-token + GC_OPERATOR_ONLY=must-survive-refusal + + - name: Default to detecting no untracked-file rejection + ansible.builtin.set_fact: + ga_untracked_rejected: false + + - name: Try an inventory token against the untracked host file + block: + - name: Include grafana_alloy without the overwrite opt-in + ansible.builtin.include_role: + name: grafana_alloy + vars: + <<: *ga_role_vars + grafana_alloy_api_token: "{{ ga_overwrite_token }}" + rescue: + - name: Record the expected untracked-file rejection + ansible.builtin.set_fact: + ga_untracked_rejected: true + when: ansible_failed_task.name is match('^Require confirmation to overwrite an env file') + + - name: Read the refused credential file back + ansible.builtin.slurp: + src: /etc/grafana-alloy.env + register: ga_refused_env + + - name: Assert refusal happened before any host-managed content was lost + ansible.builtin.assert: + that: + - ga_untracked_rejected | bool + - (ga_refused_env.content | b64decode) is search('GC_OPERATOR_ONLY=must-survive-refusal') + - (ga_refused_env.content | b64decode) is search('molecule-untracked-token') + - (ga_refused_env.content | b64decode) is not search('molecule-overwrite-token') + + - name: Re-run with explicit permission to take the untracked file over + ansible.builtin.include_role: + name: grafana_alloy + vars: + <<: *ga_role_vars + grafana_alloy_api_token: "{{ ga_overwrite_token }}" + grafana_alloy_overwrite_host_file: true + + - name: Read the explicitly adopted credential file + ansible.builtin.slurp: + src: /etc/grafana-alloy.env + register: ga_adopted_env + + - name: Assert explicit takeover produced the promised token-only file + ansible.builtin.assert: + that: + - >- + (ga_adopted_env.content | b64decode).splitlines() + == ['GC_API_TOKEN=' ~ (ga_overwrite_token | to_json)] + + # --- Interrupted rotation: same desired bytes must be resumable ------------ + - name: Read Alloy's PID before the interrupted rotation + ansible.builtin.command: + cmd: pgrep -f '^sleep infinity$' + changed_when: false + register: ga_pid_before_retry + + - name: Require exactly one Alloy stub process before rotation + ansible.builtin.assert: + that: + - ga_pid_before_retry.stdout_lines | length == 1 + + # Model the precise failure window: the role wrote the new bytes, then a later + # validation/start task aborted before the old provenance record was replaced. + - name: Stage the token bytes left by an interrupted rotation + ansible.builtin.copy: + dest: /etc/grafana-alloy.env + owner: root + group: root + mode: "0600" + content: "GC_API_TOKEN={{ ga_rotated_token | to_json }}\n" + + - name: Retry the interrupted rotation without the destructive overwrite knob + ansible.builtin.include_role: + name: grafana_alloy + vars: + <<: *ga_role_vars + grafana_alloy_api_token: "{{ ga_rotated_token }}" + + - name: Read Alloy's PID after the recovered rotation + ansible.builtin.command: + cmd: pgrep -f '^sleep infinity$' + changed_when: false + register: ga_pid_after_retry + + - name: Read the recovered rotation artifacts + ansible.builtin.slurp: + src: "{{ item }}" + register: ga_rotated + loop: + - /etc/grafana-alloy.env + - /etc/grafana-alloy.env.sha256 + + - name: Stat the recovered credential file + ansible.builtin.stat: + path: /etc/grafana-alloy.env + get_checksum: true + checksum_algorithm: sha256 + register: ga_rotated_stat + + - name: Assert retry completed provenance and restarted onto the new token + ansible.builtin.assert: + that: + - ga_pid_after_retry.stdout_lines | length == 1 + - ga_pid_after_retry.stdout | trim != ga_pid_before_retry.stdout | trim + - >- + (ga_rotated.results[0].content | b64decode).splitlines() + == ['GC_API_TOKEN=' ~ (ga_rotated_token | to_json)] + - ((ga_rotated.results[1].content | b64decode | trim).split()[0]) == 'inventory' + - ((ga_rotated.results[1].content | b64decode | trim).split()[1]) + == ga_rotated_stat.stat.checksum + + # Exercise the separate "would overwrite" wording and prove dry-run leaves + # both secret and provenance bytes alone. The record is temporarily removed + # outside check mode solely to recreate the untracked-upgrade branch. + - name: Save the recovered provenance record for the dry-run fixture + ansible.builtin.set_fact: + ga_record_before_check: "{{ ga_rotated.results[1].content | b64decode }}" + + - name: Remove provenance to model an untracked file during dry-run + ansible.builtin.file: + path: /etc/grafana-alloy.env.sha256 + state: absent + + - name: Dry-run an explicitly approved takeover of the untracked file + check_mode: true + block: + - name: Include grafana_alloy under check mode + ansible.builtin.include_role: + name: grafana_alloy + vars: + <<: *ga_role_vars + # kics-scan ignore-line (fake fixture token, not a committed secret) + grafana_alloy_api_token: molecule-check-mode-token + grafana_alloy_overwrite_host_file: true + + - name: Read the credential file after dry-run + ansible.builtin.slurp: + src: /etc/grafana-alloy.env + register: ga_after_check + + - name: Stat provenance after dry-run + ansible.builtin.stat: + path: /etc/grafana-alloy.env.sha256 + register: ga_record_after_check + + # Assert the CONTRACT — that a dry run mutates NOTHING — not which task ran. + # + # Deliberately no assertions on the role's check-mode WORDING here. The role + # branches its two "would discard"/"did discard" messages on + # `ansible_check_mode`, and that magic variable reflects ONLY the `--check` CLI + # flag: the `check_mode: true` keyword above (block or play level, verified on + # ansible-core 2.21) suppresses module writes but leaves `ansible_check_mode` + # False inside an included role. So from a verify play the role always takes + # its real-run branch, and any predicate on those registers asserts the + # opposite of what it reads. Operators meet that branch through `make check`, + # which does pass --check. + # + # What IS proven below is the part that matters and is reachable: modules ran + # in check mode, so the credential bytes and the absent record are untouched. + - name: Assert dry-run changed neither the credential file nor the record + ansible.builtin.assert: + that: + - ga_after_check.content == ga_rotated.results[0].content + - not ga_record_after_check.stat.exists + fail_msg: >- + A check-mode converge mutated host state: the credential file or the + provenance record moved. Dry runs must never write either. + + - name: Restore the saved provenance record after the dry-run fixture + ansible.builtin.copy: + dest: /etc/grafana-alloy.env.sha256 + owner: root + group: root + mode: "0600" + content: "{{ ga_record_before_check }}" + + # --- Overwrite gate: both halves of "this role wrote it" must be load-bearing - + # The gate allows an existing file through without the opt-in only when + # (source == 'inventory' AND recorded == actual) OR desired == actual. + # The takeover case above covers an ABSENT record and the rotation case covers + # `desired == actual`. These two cover the remaining cells, each killing one + # half of the first disjunct: drop the source test and the "host" case below is + # silently adopted (clobbering a host-provisioned file that may carry GC_* keys + # the operator still needs); drop the checksum test and the "edited" case is. + # + # Both cases drive ga_gate_token, which is deliberately DIFFERENT from what the + # credential file holds at this point: the gate lets an existing file through + # when `desired == actual`, so a matching token would mask either regression. + - name: Stat the current credential file for the provenance fixtures + ansible.builtin.stat: + path: /etc/grafana-alloy.env + get_checksum: true + checksum_algorithm: sha256 + register: ga_gate_stat + + # Case 1 — record says the OPERATOR owns this file: the normal host->inventory + # migration every existing deployment performs. + - name: Stage a host-owned provenance record for the current file + ansible.builtin.copy: + dest: /etc/grafana-alloy.env.sha256 + owner: root + group: root + mode: "0600" + content: "host {{ ga_gate_stat.stat.checksum }}\n" + + - name: Default to detecting no host-record rejection + ansible.builtin.set_fact: + ga_host_record_rejected: false + + - name: Try an inventory token against a host-owned record + block: + - name: Include grafana_alloy against a host-owned record + ansible.builtin.include_role: + name: grafana_alloy + vars: + <<: *ga_role_vars + grafana_alloy_api_token: "{{ ga_gate_token }}" + rescue: + - name: Record the expected host-record rejection + ansible.builtin.set_fact: + ga_host_record_rejected: true + when: ansible_failed_task.name is match('^Require confirmation to overwrite an env file') + + # Case 2 — record says this role wrote it, but the bytes have moved since: an + # out-of-band edit to a role-authored file. + - name: Stage a role-authored record whose checksum no longer matches + ansible.builtin.copy: + dest: /etc/grafana-alloy.env.sha256 + owner: root + group: root + mode: "0600" + content: "inventory {{ '0' * 64 }}\n" + + - name: Default to detecting no stale-checksum rejection + ansible.builtin.set_fact: + ga_stale_sum_rejected: false + + - name: Try an inventory token against a stale role-authored record + block: + - name: Include grafana_alloy against a stale checksum + ansible.builtin.include_role: + name: grafana_alloy + vars: + <<: *ga_role_vars + grafana_alloy_api_token: "{{ ga_gate_token }}" + rescue: + - name: Record the expected stale-checksum rejection + ansible.builtin.set_fact: + ga_stale_sum_rejected: true + when: ansible_failed_task.name is match('^Require confirmation to overwrite an env file') + + - name: Read the credential file after both refusals + ansible.builtin.slurp: + src: /etc/grafana-alloy.env + register: ga_gate_env + + - name: Assert both foreign-provenance cells refused before touching the file + ansible.builtin.assert: + that: + - ga_host_record_rejected | bool + - ga_stale_sum_rejected | bool + # Refusal happens in preflight, before any mutation — the bytes the + # rotation case left behind must still be there, untouched by either run. + - >- + (ga_gate_env.content | b64decode).splitlines() + == ['GC_API_TOKEN=' ~ (ga_rotated_token | to_json)] + fail_msg: >- + The overwrite gate must demand grafana_alloy_overwrite_host_file for a + record that reads "host", and for one whose checksum no longer matches + the file. host-record refused={{ ga_host_record_rejected }}, + stale-checksum refused={{ ga_stale_sum_rejected }}. + + - name: Restore the inventory provenance record after the gate fixtures + ansible.builtin.copy: + dest: /etc/grafana-alloy.env.sha256 + owner: root + group: root + mode: "0600" + content: "{{ ga_record_before_check }}" + + # --- Disable/re-enable: keep evidence that the stale file was role-authored -- + - name: Disable the role after an inventory-authored deployment + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: false + + - name: Stat the secret and provenance retained across disable + ansible.builtin.stat: + path: "{{ item }}" + register: ga_retained + loop: + - /etc/grafana-alloy.env + - /etc/grafana-alloy.env.sha256 + + - name: Assert disable retained both halves of the provenance evidence + ansible.builtin.assert: + that: + - ga_retained.results | rejectattr('stat.exists') | list | length == 0 + + - name: Default to detecting no missing-token rejection + ansible.builtin.set_fact: + ga_missing_token_rejected: false + + - name: Try to re-enable after the inventory token disappeared + block: + - name: Include grafana_alloy with the API token missing + ansible.builtin.include_role: + name: grafana_alloy + vars: + <<: *ga_role_vars + grafana_alloy_api_token: "" + rescue: + - name: Record the expected missing-token rejection + ansible.builtin.set_fact: + ga_missing_token_rejected: true + when: ansible_failed_task.name is match('^Refuse to adopt a role-authored env file') + + - name: Assert re-enable did not adopt the stale role-authored token + ansible.builtin.assert: + that: + - ga_missing_token_rejected | bool diff --git a/ansible/molecule/grafana-cloud/verify.yml b/ansible/molecule/grafana-cloud/verify.yml index 982ee2a..b40bbde 100644 --- a/ansible/molecule/grafana-cloud/verify.yml +++ b/ansible/molecule/grafana-cloud/verify.yml @@ -324,6 +324,86 @@ and ansible_facts.services['decdn-node.service'].state == 'running' fail_msg: "alloy or decdn-node is not running — journalctl -u alloy -e / -u decdn-node -e" +# A host-managed env file is never rendered by the role, so only the provenance +# checksum can notice an out-of-band rotation. Prove that signal restarts the +# actual process; file/checksum assertions alone would stay green if the notify +# disappeared and Alloy kept the old environment forever. +- name: Verify a host-provisioned token edit restarts Alloy + hosts: all + become: true + tasks: + - name: Read Alloy's PID before the host-side credential edit + ansible.builtin.command: + cmd: pgrep -f '^sleep infinity$' + changed_when: false + register: ga_host_pid_before + + - name: Require exactly one Alloy stub process before the edit + ansible.builtin.assert: + that: + - ga_host_pid_before.stdout_lines | length == 1 + + - name: Rotate the host-provisioned token out of band + ansible.builtin.replace: + path: /etc/grafana-alloy.env + regexp: '^GC_API_TOKEN=.*$' + # kics-scan ignore-line (fake fixture token, not a committed secret) + replace: GC_API_TOKEN=molecule-host-rotated-token + + - name: Checksum the rendered config before the re-run + ansible.builtin.stat: + path: /etc/alloy/config.alloy + get_checksum: true + checksum_algorithm: sha256 + register: ga_host_config_before + + # These vars MUST mirror converge.yml exactly. Any knob that differs makes the + # config template re-render, which notifies Restart alloy on its own and would + # make the PID assertion below pass even with the provenance signal deleted — + # certifying a feature this play does not exercise. grafana_alloy_batch_send_size + # is the live example: converge.yml sets 512 against a role default of 1024. + - name: Re-run grafana_alloy after the host-side edit + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_install_method: manual + grafana_alloy_manual_bin_src: >- + {{ lookup('ansible.builtin.env', 'MOLECULE_SCENARIO_DIRECTORY') }}/files/alloy-stub + grafana_alloy_deployment_environment: production + grafana_alloy_region: US + grafana_alloy_loki_url: https://logs-prod-xx.molecule.invalid/loki/api/v1/push + grafana_alloy_prom_username: 1234567 + grafana_alloy_batch_send_size: 512 + + - name: Read Alloy's PID after the host-side credential edit + ansible.builtin.command: + cmd: pgrep -f '^sleep infinity$' + changed_when: false + register: ga_host_pid_after + + - name: Checksum the rendered config after the re-run + ansible.builtin.stat: + path: /etc/alloy/config.alloy + get_checksum: true + checksum_algorithm: sha256 + register: ga_host_config_after + + - name: Assert Alloy restarted onto the edited host environment + ansible.builtin.assert: + that: + - ga_host_pid_after.stdout_lines | length == 1 + - ga_host_pid_after.stdout | trim != ga_host_pid_before.stdout | trim + # The restart must be attributable to the provenance signal ALONE. If the + # config moved, this play proves nothing about out-of-band detection, so + # fail loudly on var drift rather than silently passing on the wrong cause. + - ga_host_config_after.stat.checksum == ga_host_config_before.stat.checksum + fail_msg: >- + Either Alloy did not restart after the host-side credential edit (the + out-of-band provenance signal regressed), or the rendered config changed + across the re-run — meaning the vars above have drifted from converge.yml + and the restart cannot be attributed to the credential edit. + # Also mutates the converged host, so it runs after every assertion above and # before the teardown play below. - name: Verify turning logs off takes the journal grant back diff --git a/ansible/molecule/validation/converge.yml b/ansible/molecule/validation/converge.yml index 495c691..f1ad9a5 100644 --- a/ansible/molecule/validation/converge.yml +++ b/ansible/molecule/validation/converge.yml @@ -646,7 +646,7 @@ - name: Record ga-no-secret rejection (only if the secret-file assert failed) ansible.builtin.set_fact: decdn_rejected: "{{ decdn_rejected + ['secret-file'] }}" - when: ansible_failed_task.name is match('^Assert the secret file') + when: ansible_failed_task.name is match('^Require the Grafana Cloud secret environment file') - name: "Case ga-symlink-secret — symlinked credential file" block: @@ -927,6 +927,71 @@ decdn_rejected: "{{ decdn_rejected + ['managed-paths'] }}" when: ansible_failed_task.name is match('^Require the rendered config to live inside') + # The provenance file is root-written, root-read and removed during teardown. + # It must be the fixed companion of the credential file, not an independently + # aimable path that can slurp/clobber/delete another file under /etc. + - name: "Case ga-provenance-path — checksum record is not beside the credential file" + block: + - name: Run grafana_alloy with an independently aimed provenance path + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_env_checksum_file: /etc/molecule-unrelated-root-file + rescue: + - name: Record ga-provenance-path rejection (only if the companion-path gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['managed-paths'] }}" + when: ansible_failed_task.name is match('^Require the provenance record to sit beside') + + - name: "Case ga-provenance-content — unrelated file occupies the record path" + block: + - name: Stage an unrelated root-only file at the provenance path + ansible.builtin.copy: + dest: /etc/grafana-alloy.env.sha256 + owner: root + group: root + mode: "0600" + # A valid first NUL-delimited grep record followed by unrelated bytes: + # `grep -zq` accepts this unless validation full-matches the byte stream. + content: >- + {{ ('aG9zdCAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMD' + ~ 'AwMDAwMDAwMDAwMDAwMDAwMDAwAHVucmVsYXRlZC1zZW5zaXRpdmUtY29udGVudAo=') + | b64decode }} + + - name: Run grafana_alloy with the unrelated provenance-path occupant + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + # kics-scan ignore-line (fake fixture token, not a committed secret) + grafana_alloy_api_token: molecule-test-token + rescue: + - name: Record ga-provenance-content rejection (only if shape validation failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['provenance-record'] }}" + when: ansible_failed_task.name is match('^Require an existing provenance record to have') + always: + - name: Remove the unrelated provenance-path occupant + ansible.builtin.file: + path: /etc/grafana-alloy.env.sha256 + state: absent + + - name: "Case ga-config-record-collision — teardown directory contains retained provenance" + block: + - name: Run disabled grafana_alloy with config_dir aimed at the provenance file + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: false + grafana_alloy_config_dir: /etc/grafana-alloy.env.sha256 + grafana_alloy_config_file: /etc/grafana-alloy.env.sha256/config.alloy + rescue: + - name: Record ga-config-record-collision rejection + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['managed-paths'] }}" + when: ansible_failed_task.name is match('^Keep destructive directories away') + - name: "Case ga-collector — unknown node_exporter collector name" block: - name: Run grafana_alloy with a bogus host collector @@ -1127,6 +1192,196 @@ path: /etc/grafana-alloy.env state: absent + # The #39 relaxation lets the token ride inventory, but only as a bare, + # whitespace-free value — it becomes a single systemd EnvironmentFile line. + - name: "Case ga-token-var-shape — whitespace inside grafana_alloy_api_token" + block: + - name: Run grafana_alloy with a whitespace-carrying API token variable + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + # kics-scan ignore-line (fake fixture token, not a committed secret) + grafana_alloy_api_token: "glc_molecule spaced" + rescue: + - name: Record ga-token-var-shape rejection (only if the token-shape gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['token-shape'] }}" + when: ansible_failed_task.name is match('^Validate the shape of an inventory-provided API token') + + # An existing host-authored env file with NO provenance is the upgrade state + # of every pre-record deployment. It must not be silently clobbered by an + # inventory token; unknown provenance requires the explicit opt-in knob. + - name: "Case ga-collision — inventory token aimed at an untracked host env file" + block: + - name: Stage a host-style credential fixture + ansible.builtin.copy: + dest: /etc/grafana-alloy.env + owner: root + group: root + mode: "0600" + content: | + GC_PROM_REMOTE_WRITE_URL=https://prometheus-prod-xx.molecule.invalid/api/prom/push + GC_OTLP_ENDPOINT=https://otlp-gateway-prod-xx.molecule.invalid/otlp + GC_PROM_USERNAME=999888777 + GC_API_TOKEN=molecule-test-token + + - name: Run grafana_alloy with an inventory token against the foreign fixture + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + # kics-scan ignore-line (fake fixture token, not a committed secret) + grafana_alloy_api_token: molecule-test-token + rescue: + - name: Record ga-collision rejection (only if the overwrite gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['token-collision'] }}" + when: ansible_failed_task.name is match('^Require confirmation to overwrite an env file') + always: + - name: Remove the collision fixtures + ansible.builtin.file: + path: "{{ item }}" + state: absent + loop: + - /etc/grafana-alloy.env + - /etc/grafana-alloy.env.sha256 + + # Python's `$` also matches just before a trailing newline, so the shape gate + # must anchor on \Z. A YAML block scalar (`grafana_alloy_api_token: |`) is the + # natural way to wrap a long token and appends exactly that newline; systemd + # would deliver it to the daemon as a literal \n and every push would 401 + # behind a green deploy. + - name: "Case ga-token-newline — trailing newline in grafana_alloy_api_token" + block: + - name: Run grafana_alloy with a newline-terminated API token variable + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + # kics-scan ignore-line (fake fixture token, not a committed secret) + grafana_alloy_api_token: "glc_molecule_trailing\n" + rescue: + - name: Record ga-token-newline rejection (only if the token-shape gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['token-shape'] }}" + when: ansible_failed_task.name is match('^Validate the shape of an inventory-provided API token') + + # The flag that authorises destroying an operator-provisioned credential file + # must not be reachable by YAML truthiness: "yes"/"on"/"1" would coerce to an + # authorisation nobody typed. + - name: "Case ga-overwrite-bool — quoted string for the overwrite opt-in" + block: + - name: Run grafana_alloy with a stringly-typed overwrite knob + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + # kics-scan ignore-line (fake fixture token, not a committed secret) + grafana_alloy_api_token: molecule-test-token + grafana_alloy_overwrite_host_file: "yes" + rescue: + - name: Record ga-overwrite-bool rejection (only if the knob-type gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['credential-knobs'] }}" + when: ansible_failed_task.name is match('^Validate the credential-path knobs') + + # An inventory token makes the env file TOKEN-ONLY, so every other connection + # value must already come from inventory. Without this gate the role authors a + # file with nothing but the token, config.alloy resolves the missing endpoint + # to "" via sys.env, `alloy validate` passes, the unit starts, every assertion + # is green — and not one metric is ever shipped. + - name: "Case ga-token-no-endpoints — inventory token without the Prometheus endpoint" + block: + - name: Run grafana_alloy with a token but no inventory endpoints + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + # kics-scan ignore-line (fake fixture token, not a committed secret) + grafana_alloy_api_token: molecule-test-token + rescue: + - name: Record ga-token-no-endpoints rejection (only if the all-or-nothing gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['token-all-or-nothing'] }}" + when: ansible_failed_task.name is match('^Demand every non-token connection setting from inventory') + + # Same gate, the conditional half: the Loki pair is only required while logs + # are enabled, and `not (…) or (…)` is exactly the shape that silently inverts. + - name: "Case ga-token-no-loki — inventory token with logs on but no Loki pair" + block: + - name: Run grafana_alloy with every endpoint but the Loki pair + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + grafana_alloy_logs_enabled: true + # kics-scan ignore-line (fake fixture token, not a committed secret) + grafana_alloy_api_token: molecule-test-token + grafana_alloy_prom_url: https://prometheus-prod-xx.molecule.invalid/api/prom/push + grafana_alloy_prom_username: "1234567" + grafana_alloy_otlp_endpoint: https://otlp-gateway-prod-xx.molecule.invalid/otlp + rescue: + - name: Record ga-token-no-loki rejection (only if the all-or-nothing gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['token-all-or-nothing'] }}" + when: ansible_failed_task.name is match('^Demand every non-token connection setting from inventory') + + # The record's stat gate runs BEFORE the slurp that reads it onto the control + # machine. That ordering is the whole defence against a symlink planted at the + # record path pulling an arbitrary root-only file into controller memory — so + # it needs its own case, not just the content validator's. + - name: "Case ga-symlink-record — symlinked provenance record" + block: + - name: Stage a complete host-provisioned credential file + ansible.builtin.copy: + dest: /etc/grafana-alloy.env + owner: root + group: root + mode: "0600" + content: | + GC_PROM_REMOTE_WRITE_URL=https://prometheus-prod-xx.molecule.invalid/api/prom/push + GC_OTLP_ENDPOINT=https://otlp-gateway-prod-xx.molecule.invalid/otlp + GC_PROM_USERNAME=999888777 + GC_LOKI_URL=https://logs-prod-xx.molecule.invalid/loki/api/v1/push + GC_LOKI_USERNAME=7654321 + GC_API_TOKEN=molecule-test-token + + - name: Stage the record symlink target outside /etc + ansible.builtin.copy: + dest: /var/lib/grafana-alloy-record-fixture + owner: root + group: root + mode: "0600" + content: "host {{ '0' * 64 }}\n" + + - name: Point the provenance-record path at the out-of-tree target + ansible.builtin.file: + src: /var/lib/grafana-alloy-record-fixture + dest: /etc/grafana-alloy.env.sha256 + state: link + force: true + + - name: Run grafana_alloy against a symlinked provenance record + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: true + rescue: + - name: Record ga-symlink-record rejection (only if the record stat gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['provenance-record'] }}" + when: ansible_failed_task.name is match('^Require an existing provenance record to be a protected') + always: + - name: Remove the record symlink fixtures + ansible.builtin.file: + path: "{{ item }}" + state: absent + loop: + - /etc/grafana-alloy.env + - /etc/grafana-alloy.env.sha256 + - /var/lib/grafana-alloy-record-fixture + - name: Confirm every bad value was rejected by the role's own validation ansible.builtin.assert: that: @@ -1152,5 +1407,7 @@ "alloy-root-user-alias", "alloy-root-group-alias", "host-collectors", "host-collectors", "host-collectors", "endpoint-scheme", "inventory-token", "inventory-token", "instance-id", - "otlp-username-env", - "managed-paths", "managed-paths", "managed-paths"] + "otlp-username-env", "token-shape", "token-shape", "token-collision", + "credential-knobs", "token-all-or-nothing", "token-all-or-nothing", + "provenance-record", "provenance-record", + "managed-paths", "managed-paths", "managed-paths", "managed-paths", "managed-paths"] diff --git a/ansible/roles/grafana_alloy/README.md b/ansible/roles/grafana_alloy/README.md index 479d6c5..6c1fa67 100644 --- a/ansible/roles/grafana_alloy/README.md +++ b/ansible/roles/grafana_alloy/README.md @@ -76,6 +76,81 @@ matching `GC_…` key in the env file exactly as before, and preflight requires key only then. Any mix of the two halves is valid. `roles/grafana_alloy/files/grafana-alloy.env.example` shows the full key set. +### The API token has two homes now + +The token may also ride **git-ignored inventory**, mirroring how `decdn_rpc_url` +works: + +```yaml +# host_vars//secret.yml — git-ignored; see secret.yml.example +grafana_alloy_api_token: "glc_…" +``` + +With it set, the role authors `/etc/grafana-alloy.env` itself — as a **token-only** +file (`GC_API_TOKEN="…"` at `root:root 0600`, JSON-encoded so quotes/backslashes +survive), rewritten wholesale on every converge. Runtime behaviour is identical: +`config.alloy` still reads `sys.env("GC_API_TOKEN")`; only who fills the file +changes. Rotation is an inventory edit + deploy instead of per-host shell work, +and provenance tracking (a ` ` record next to the file) catches +the classic failure modes loud: + +- **Token vanished from the control machine** while a role-authored file exists + → the run refuses to adopt its own file as operator-provisioned ("restore the + variable or discard the record explicitly"). +- **Untracked hand-edited file** meeting an inventory token → refused unless + `grafana_alloy_overwrite_host_file: true`; the sole safe exception is a file + already byte-identical to the desired token-only payload, because no host bytes + would be discarded and it may be the residue of an interrupted first converge. +- **Interrupted inventory rotation** (new env bytes landed, later validation or + restart failed before the record moved) → retry recognises the exact desired + token-only checksum, restarts Alloy, and completes the record without asking + for the destructive overwrite opt-in. + +The authoring is all-or-nothing: any hand-added `GC_*` lines beyond the token are +DISCARDED by a rewrite — migrate them to their inventory variables first, and +with the token set preflight demands every remaining connection setting from +inventory (the host file's current keys prove nothing once the rewrite lands). +An out-of-band edit on +the host-provisioned path restarts the agent with a note, same as `decdn_node`. +Disabling the role intentionally retains both the secret env file **and** its +provenance record; retaining only the file would let a later re-enable with a +missing `secret.yml` misclassify the stale role-authored token as host-owned. +The Helm chart ships no Alloy at all, so nothing changes there. + +#### Migrating an existing host ONTO the inventory path + +The overwrite guard refuses this by design, because the rewrite is token-only and +the role cannot read the file to tell you what it would destroy. Do it in this +order: + +1. Move every non-token `GC_*` key in `/etc/grafana-alloy.env` to its + `grafana_alloy_*` inventory variable. Preflight then stops demanding those keys + from the file, which is what makes the migration verifiable rather than hopeful. +2. Set `grafana_alloy_api_token` in the git-ignored `host_vars//secret.yml`. +3. Run `make check LIMIT=` **first**. The dry run names exactly what a real + converge would discard, and changes nothing. +4. Set `grafana_alloy_overwrite_host_file: true` and deploy once. +5. Set it back to `false`. Leaving it `true` permanently disables the guard on that + host, so a future hand-edit is silently destroyed instead of stopping the deploy. + +#### Returning a host TO hand-provisioned mode + +Clear `grafana_alloy_api_token`, then discard the provenance record +(`sudo rm /etc/grafana-alloy.env.sha256`) **before** the next converge — do it +after and the adoption guard hard-fails the deploy. Any `GC_*` keys you want back +on the host must be re-added by hand; the token-only file carries none of them. + +Note the one blind spot this leaves: with the record gone, the next converge has +nothing to compare against, so a token you edit on the host *in the same change* +is not detected and the running agent keeps the old credentials. The role warns +when it hits that state — restart once (`sudo systemctl restart alloy`) if you +rotated the token as part of the hand-back. + +`GC_API_TOKEN` values in plain committed inventory remain rejected: the rendered +`/etc/alloy/config.alloy` is world-readable, and preflight rejects a value that +looks like a token (or a URL with embedded credentials) in any of the six +variables above. + > **Upgrading an existing host.** Logs are on by default, and Loki needs an > endpoint + instance ID that the pre-existing four-key env file does not have. > Preflight fails loudly, naming both the missing key and the inventory variable @@ -99,11 +174,6 @@ key only then. Any mix of the two halves is valid. > present, preflight checks its shape, because a malformed value outranks the > Prometheus fallback at runtime and 401s traces just the same. -`GC_API_TOKEN` has **no** inventory variable by design: the rendered -`/etc/alloy/config.alloy` is world-readable, and preflight rejects a value that -looks like a token (or a URL with embedded credentials) in any of the six -variables above. - The credential path is intentionally restricted to a **direct child of `/etc`**. A nested override (including the old `/etc/decdn/grafana-alloy.env`) is rejected: write access to any ancestor is enough to replace a root-owned `0600` file. Both @@ -200,6 +270,9 @@ upstream version/sha256). Highlights: | `grafana_alloy_logs_max_age` | `12h` | Bounds the catch-up burst after an outage | | `grafana_alloy_self_metrics_enabled` | `true` | Alloy's own health | | `grafana_alloy_prom_url` / `_prom_username` / `_otlp_endpoint` / `_otlp_username` / `_loki_url` / `_loki_username` | `""` | Non-secret connection settings; empty ⇒ read the matching `GC_…` env key (`_otlp_username` then falls back to the Prometheus ID) | +| `grafana_alloy_api_token` | `""` | **SENSITIVE** — the API token (git-ignored `secret.yml`). Empty ⇒ operator-provisioned `/etc/grafana-alloy.env`; set ⇒ the role authors that file token-only, root 0600. See "The API token has two homes now". | +| `grafana_alloy_env_checksum_file` | `/etc/grafana-alloy.env.sha256` | Provenance record (` `) enabling the adoption/overwrite guards; must remain exactly `.sha256` | +| `grafana_alloy_overwrite_host_file` | `false` | Explicit opt-in letting an inventory token rewrite a hand-edited/untracked env file (all-or-nothing: extra `GC_*` lines are discarded) | | `grafana_alloy_node_job` / `_host_job` / `_self_job` / `_logs_job` | see table above | Job labels; the `integrations/…` ones are what Grafana Cloud's dashboards match | Label variables (`service_name`, `service_namespace`, `instance_id`, diff --git a/ansible/roles/grafana_alloy/defaults/main.yml b/ansible/roles/grafana_alloy/defaults/main.yml index 51b2ecd..b4360fb 100644 --- a/ansible/roles/grafana_alloy/defaults/main.yml +++ b/ansible/roles/grafana_alloy/defaults/main.yml @@ -37,6 +37,45 @@ grafana_alloy_state_dir: /var/lib/alloy # WAL/storage state, owned by the a # swap the root-owned EnvironmentFile before systemd reads it. grafana_alloy_secret_file: /etc/grafana-alloy.env +# SENSITIVE — the Grafana Cloud access-policy/service-account token. EMPTY (the +# default) keeps the host-provisioned posture: preflight requires a 0600 env file +# on the target host and the value never transits the control machine. SETTING it +# opts into the inventory path (#39 parity with decdn_rpc_url): this role then +# authors {{ grafana_alloy_secret_file }} as a TOKEN-ONLY EnvironmentFile that is +# rewritten wholesale on every converge, so any hand-added GC_* keys must migrate +# to their inventory variables first (all-or-nothing authoring). Whether the value +# arrives via this variable or was provisioned by hand, config.alloy reads it the +# same way at runtime: sys.env("GC_API_TOKEN"). +grafana_alloy_api_token: "" + +# Provenance tracking for grafana_alloy_secret_file, mirroring roles/decdn_node's +# #39 machinery. After every enabled converge the role records " " +# of the env file the service is now running on ("inventory" when +# grafana_alloy_api_token authored it, "host" otherwise) into a root-owned 0600 +# record NEXT TO the secret file — deliberately NOT inside /etc/alloy or +# /var/lib/alloy, because the disable path removes those directories wholesale. +# Path validation requires this to remain exactly `.sha256`; it is +# not an independently aimable root-write path. +# Preflight uses the pair to fail loud on (a) adopting a role-authored file after +# the secret went missing from the control machine and (b) clobbering an operator- +# edited or untracked file without grafana_alloy_overwrite_host_file. Disable keeps +# both file and record so a later re-enable cannot forget who authored the token. +grafana_alloy_env_checksum_file: /etc/grafana-alloy.env.sha256 + +# Explicit opt-in letting grafana_alloy_api_token rewrite an env file whose +# provenance record is missing or mismatched — i.e. something this role did not +# last write. Identical contract to decdn_env_overwrite_host_file: because the +# rewrite is token-only, any lines added outside Ansible are DISCARDED; migrate +# them to their inventory variables BEFORE turning this on (preflight then stops +# demanding them from the file, which also makes the migration verifiable). +# +# Turn it back to false after the one converge that needs it. Left true, this host +# has NO clobber guard: the next hand-edit to the env file is silently destroyed +# instead of stopping the deploy, which is the whole point of the record. It must +# be a real boolean — preflight rejects "yes"/"on"/"1" rather than let YAML +# truthiness grant an authorisation nobody typed. +grafana_alloy_overwrite_host_file: false + # --- Network (loopback-only; AGENTS.md hard rule 2) --------------------------- # OTLP receivers bound to loopback ONLY — the daemon accepts telemetry pushed # over localhost, never from the network; no firewall hole exists or is needed. @@ -54,16 +93,21 @@ grafana_alloy_server_http_addr: 127.0.0.1:12345 # --- Grafana Cloud endpoints (NON-SECRET; optional inventory overrides) ------- # Only the API TOKEN is a secret. The endpoint URLs and the two numeric instance # IDs are not, so they may be set here (group_vars/host_vars) instead of being -# hand-typed into the root-owned env file on every host — which leaves -# GC_API_TOKEN as the single value an operator must provision per host. +# hand-typed into the root-owned env file on every host — leaving GC_API_TOKEN +# as the only credential, itself dually homeable: operator-provisioned per host, +# or carried by grafana_alloy_api_token from git-ignored inventory (see below). # # Each of these is optional and falls back, when left EMPTY, to the matching # sys.env("GC_…") lookup the role has always used, so an existing deployment that # sets none of them renders byte-identical configuration. Setting one makes its # env key unnecessary, and preflight stops requiring that key. # -# NEVER put the token here: config.alloy is world-readable 0644 on the host, and -# inventory is committed. GC_API_TOKEN has no variable by design. +# NEVER put a credential in THIS template: config.alloy is world-readable 0644 on +# the host, and inventory is committed. The API token is the ONE value allowed to +# ride through inventory — via grafana_alloy_api_token below, which is SENSITIVE +# and belongs in a git-ignored secret.yml exactly like decdn_rpc_url (#39). Setting +# it makes this role author /etc/grafana-alloy.env itself; leaving it "" keeps the +# original operator-provisioned posture. See README.md §Credentials. grafana_alloy_prom_url: "" # else sys.env("GC_PROM_REMOTE_WRITE_URL") grafana_alloy_prom_username: "" # else sys.env("GC_PROM_USERNAME") grafana_alloy_otlp_endpoint: "" # else sys.env("GC_OTLP_ENDPOINT") diff --git a/ansible/roles/grafana_alloy/files/grafana-alloy.env.example b/ansible/roles/grafana_alloy/files/grafana-alloy.env.example index 24b0d81..94d440d 100644 --- a/ansible/roles/grafana_alloy/files/grafana-alloy.env.example +++ b/ansible/roles/grafana_alloy/files/grafana-alloy.env.example @@ -1,11 +1,18 @@ # Template for /etc/grafana-alloy.env (root-owned 0600). # -# Provision it ON THE TARGET HOST — this file holds your Grafana Cloud API token -# and must never be committed or carried through inventory: +# This file holds your Grafana Cloud API token and must NEVER be committed. +# Provision it ON THE TARGET HOST: # # umask 077 # sudo install -m 600 -o root -g root grafana-alloy.env /etc/grafana-alloy.env # +# …or skip this file entirely and let inventory author it: set +# grafana_alloy_api_token in a git-ignored host_vars//secret.yml and the +# role writes a token-only version of /etc/grafana-alloy.env for you. That is the +# ONE place a credential may ride inventory — a git-ignored secret.yml, never a +# committed group_vars file. See the GC_API_TOKEN block below and +# roles/grafana_alloy/README.md §Credentials for which path to pick. +# # MINIMUM CONTENT for a NEW host: the token, and nothing else. Every other value # below is NON-secret and can be set once in inventory (group_vars) instead of # being re-typed on every host — see the grafana_alloy_* variables named in each @@ -24,8 +31,10 @@ # reads values back onto the control machine. # REQUIRED, always. An access-policy token with metrics:write + logs:write + -# traces:write scope for this stack. This is the one value with no inventory -# variable, deliberately: the rendered /etc/alloy/config.alloy is world-readable. +# traces:write scope for this stack. The INVENTORY path can author this file for +# you instead (grafana_alloy_api_token in git-ignored host_vars secret.yml) — in +# that case DO NOT create it by hand; the role writes a token-only version of it, +# and lines added here would be discarded on the next converge. GC_API_TOKEN=glc_example_token_replace_me # --- Optional: only needed if the matching inventory variable is left empty ---- diff --git a/ansible/roles/grafana_alloy/tasks/main.yml b/ansible/roles/grafana_alloy/tasks/main.yml index c9b1a47..b1eaecf 100644 --- a/ansible/roles/grafana_alloy/tasks/main.yml +++ b/ansible/roles/grafana_alloy/tasks/main.yml @@ -61,6 +61,18 @@ - name: Validate the managed filesystem paths before removing them ansible.builtin.import_tasks: validate-paths.yml + # Deliberately placed a credential + a disabled flag is almost always a + # half-finished enablement, and every other task in this block is silent about + # it: the operator sees a clean run and assumes the token took effect. + - name: Note that a configured API token is inert while the flag is off + ansible.builtin.debug: + msg: >- + grafana_alloy_api_token is set, but decdn_grafana_cloud_enabled is false, + so this role tore its installation down and did NOT write + {{ grafana_alloy_secret_file }} — the token has no effect. Set + decdn_grafana_cloud_enabled: true to deploy the agent. + when: grafana_alloy_api_token | default('', true) | string | length > 0 + - name: Check for a leftover Alloy systemd unit ansible.builtin.stat: path: "{{ _ga_unit_path }}" @@ -116,6 +128,11 @@ - "{{ grafana_alloy_state_dir }}" notify: Reload systemd + # Keep both the credential file and its provenance record. If the install + # was inventory-authored, deleting only the record would make a later + # re-enable with a missing secret.yml adopt the stale token as host-owned. + # The documented explicit handoff removes the record deliberately. + - name: Apply pending teardown notifications ansible.builtin.meta: flush_handlers @@ -263,6 +280,106 @@ - name: Install the Alloy binary ansible.builtin.import_tasks: install.yml + # --- Secret environment file -------------------------------------------------- + # Two authoring paths for {{ grafana_alloy_secret_file }}, decided by the + # provenance gates in preflight exactly like decdn_node's #39: + # * grafana_alloy_api_token set -> the copy below REWRITES the file from + # control-machine values, token-only. + # * empty -> the operator wrote the file on the host. The role never reads its + # content back (that would put the secret on the control machine, which is + # the whole point of the host path); it only enforces root 0600 and greps + # shapes in preflight. + - name: Write the token-only secret environment file (inventory-supplied) + ansible.builtin.copy: + dest: "{{ grafana_alloy_secret_file }}" + owner: root + group: root + mode: "0600" + # to_json rather than hand-rolled escaping: it supplies the surrounding + # double quotes and escapes " and \ — the only two escapes systemd's + # EnvironmentFile parser unescapes inside quotes, so a token containing + # either survives intact. ensure_ascii=false keeps UTF-8 literal. + # + # systemd's parser is shell-like, NOT a C-escape parser: \n / \t / \uXXXX + # would reach the daemon as literal backslash sequences. That is exactly + # why preflight's shape gate rejects whitespace (and, via \Z, a trailing + # newline) rather than trusting to_json to make anything safe. + # + # Keep this block scalar byte-identical to _ga_desired_inventory_content + # in preflight.yml — the interrupted-rotation branch hashes that one and + # compares it against this file. no_log because the token is secret. + content: | + GC_API_TOKEN={{ grafana_alloy_api_token | string | to_json(ensure_ascii=false) }} + no_log: true + register: _ga_env_copy + notify: Restart alloy + when: grafana_alloy_api_token | default('', true) | string | length > 0 + + # If a prior run wrote the desired token bytes and then failed before the + # final provenance write, copy is now idempotent and would not notify. Queue a + # restart explicitly so retry both repairs the record and moves the running + # process onto those bytes. The desired checksum proves this is our exact + # token-only payload, not an arbitrary foreign edit. + - name: Resume an interrupted inventory token rotation + ansible.builtin.debug: + msg: >- + {{ 'Would restart Alloy to resume an interrupted inventory-token rotation.' + if ansible_check_mode else + 'Resuming interrupted inventory-token rotation; restart queued before provenance.' }} + changed_when: true + notify: Restart alloy + when: + - grafana_alloy_api_token | default('', true) | string | length > 0 + - _ga_secret_stat.stat.exists + - _ga_desired_inventory_sum == (_ga_secret_stat.stat.checksum | default('')) + - >- + _ga_record_source != 'inventory' + or _ga_recorded_sum != (_ga_secret_stat.stat.checksum | default('')) + + # The copy above is no_log, so a run that replaced an existing file shows a bare + # "changed" and nothing else. Say what was lost — but only for the UNKNOWABLE + # case: a file whose known-foreign provenance already failed preflight loudly, + # so what remains here is the first converge after a pre-provenance deploy. + # Gated on the task result too, so it speaks up once (when contents moved), + # not on every subsequent converge. + # + # Reaching this task at all means grafana_alloy_overwrite_host_file was set: + # preflight's overwrite gate demands it for ANY untracked file, so name it — + # otherwise the message reads as if the rewrite happened unprompted. + - name: Warn that an inventory grafana_alloy_api_token overwrote an untracked env file + ansible.builtin.debug: + msg: >- + grafana_alloy_api_token is set and grafana_alloy_overwrite_host_file + authorised the rewrite, so {{ grafana_alloy_secret_file }} was REWRITTEN + as a token-only file — any KEY=value lines added on the host that are not + covered by inventory variables are gone. From this converge on, that file + is tracked: an edit made outside Ansible fails the next deploy instead of + being silently discarded. Set grafana_alloy_overwrite_host_file back to + false now — leaving it true disables that guard permanently on this host. + Clear grafana_alloy_api_token to hand the file back to the host. + register: _ga_untracked_overwrite_applied_report + when: + - not ansible_check_mode + - grafana_alloy_api_token | default('', true) | string | length > 0 + - _ga_secret_stat.stat.exists + - _ga_recorded_sum | length == 0 + - _ga_env_copy.changed | default(false) | bool + + - name: Warn what check mode would discard from an untracked env file + ansible.builtin.debug: + msg: >- + grafana_alloy_api_token is set and explicit overwrite permission was + supplied, so a real run WOULD rewrite {{ grafana_alloy_secret_file }} as + a token-only file. Any host-added KEY=value lines not represented by + inventory variables WOULD be discarded; check mode made no change. + register: _ga_untracked_overwrite_check_report + when: + - ansible_check_mode + - grafana_alloy_api_token | default('', true) | string | length > 0 + - _ga_secret_stat.stat.exists + - _ga_recorded_sum | length == 0 + - _ga_env_copy.changed | default(false) | bool + # --- Configuration + hardened unit ------------------------------------------ - name: Render the Alloy pipeline configuration ansible.builtin.template: @@ -301,18 +418,111 @@ failed_when: false when: not ansible_check_mode + # `.msg` is included because failed_when: false also swallows MODULE-level + # failures — a missing or non-executable binary, ENOEXEC from a wrong-arch + # build. Those carry no stderr/stdout at all, so without it this message + # renders an empty reason next to a diagnosis ("template regression") that is + # actively wrong for that case. - name: Report the configuration-validation failure ansible.builtin.fail: msg: >- `alloy validate` rejected {{ grafana_alloy_config_file }} (rc={{ _ga_config_check.rc | default('?') }}): {{ (_ga_config_check.stderr | default('', true)) - or (_ga_config_check.stdout | default('', true)) }} — fix the - template regression before deploying. + or (_ga_config_check.stdout | default('', true)) + or (_ga_config_check.msg | default('', true)) + or 'no output' }} — fix the template regression, or the binary at + {{ grafana_alloy_bin_effective }} if the reason above is an exec error. when: - not ansible_check_mode - _ga_config_check.rc | default(1) != 0 + # Restart on an OUT-OF-BAND edit to a HOST-provisioned env file. The copy above + # already notifies when the role itself authors a changed file, so this covers + # only the host path, where nothing else would notice that the operator edited + # /etc/grafana-alloy.env between converges — the agent would keep running on + # the previous credentials behind a fully green deploy. + # + # An ABSENT record is deliberately not a change: it means this host has never + # run a provenance-tracked converge, and treating "not tracked yet" as "edited" + # would bounce every node in the fleet on the upgrade run for no functional + # reason. Handlers dedupe, and the flush below executes this BEFORE start and + # before the record, so the record always trails the restart it owes. + - name: Restat the secret environment file before recording its checksum + ansible.builtin.stat: + path: "{{ grafana_alloy_secret_file }}" + follow: false + # sha256 to match the algorithm named by the record's own filename; stat's + # default is sha1, which would make the .sha256 extension a lie. NOTE the + # record is " ", so `sha256sum -c` cannot read it — verify + # by hand with: + # sudo sha256sum /etc/grafana-alloy.env + # sudo cut -d' ' -f2 /etc/grafana-alloy.env.sha256 + get_checksum: true + checksum_algorithm: sha256 + register: _ga_env_post_stat + + - name: Note an out-of-band change to the host-provisioned env file + ansible.builtin.debug: + msg: >- + {{ grafana_alloy_secret_file }} has changed since the last converge + recorded it — restarting alloy so the agent picks the new credentials up. + changed_when: true + notify: Restart alloy + when: + - grafana_alloy_api_token | default('', true) | string | length == 0 + - _ga_recorded_sum | length > 0 + - (_ga_env_post_stat.stat.checksum | default('')) != _ga_recorded_sum + + # The blind spot the gate above leaves open, made audible. + # + # With no record there is nothing to compare, so a host-side token edit cannot + # be detected: `state: started` below is a no-op on an already-running unit and + # the record is then seeded from the NEW bytes, making that edit undetectable + # on every future converge too. Staying silent about it is what turns a + # credential rotation into a green deploy that ships nothing. + # + # This is not only the fleet-upgrade run. The documented hand-back procedure + # (clear grafana_alloy_api_token, `sudo rm` the record — see the adoption + # guard's fail_msg in preflight.yml) lands an operator in exactly this state, + # and writing a fresh token is the natural next step. + # + # Deliberately a warning, not a restart: bouncing every node in the fleet on + # the upgrade run is the cost the gate above exists to avoid. Scoped to an + # agent that was ALREADY RUNNING before this converge, because a fresh install + # starts on the current file anyway and needs no warning. + # Only gathered on the narrow path that needs it — the unit is already + # templated by now, so on a fresh install this correctly reports not-running + # (the config/unit handlers start it moments later, on the current file). + - name: Check whether Alloy predates its first provenance record + ansible.builtin.service_facts: + when: + - grafana_alloy_api_token | default('', true) | string | length == 0 + - _ga_env_post_stat.stat.exists + - _ga_recorded_sum | length == 0 + + - name: Warn that out-of-band edit detection is not armed on this host yet + ansible.builtin.debug: + msg: >- + {{ grafana_alloy_secret_file }} is host-provisioned and carries no + provenance record yet, so this converge CANNOT tell whether its contents + changed since Alloy started, and will not restart the agent on that + basis. This run seeds the record; from the next converge on, an + out-of-band edit is detected and restarts Alloy automatically. If you + changed GC_API_TOKEN (or any GC_* value) as part of THIS change, the + running agent is still on the old credentials — every push will 401 while + the deploy stays green. Restart it once, now: + sudo systemctl restart alloy + when: + - grafana_alloy_api_token | default('', true) | string | length == 0 + - _ga_env_post_stat.stat.exists + - _ga_recorded_sum | length == 0 + # Total expression: `services` is absent when the gather above was skipped, + # and the unit key is absent on a host that has never run Alloy. + - >- + (ansible_facts.services | default({}, true)).get('alloy.service', {}) + .get('state', '') == 'running' + - name: Apply pending Alloy changes ansible.builtin.meta: flush_handlers @@ -336,3 +546,46 @@ - ansible_facts.services['alloy.service'].state == 'running' fail_msg: "alloy is not running — check: journalctl -u alloy -e" when: not ansible_check_mode + + # Record the env file's hash LAST — deliberately after the handler flush and + # start above, so the record means "the agent is running on this content", not + # "Ansible saw this content". Writing it earlier looks equivalent and is not: + # the handlers acting on detected changes do not flush until after the config + # template and the `alloy validate` gate, both of which can abort the play; a + # record written before that abort would say "already handled" on the NEXT + # converge, silently burning the one signal a secret rotation gives. + # + # No notify: by the time this runs, any restart this converge owed — including + # the out-of-band-edit one queued just above — has already been flushed. + - name: Record the env-file checksum (the agent is now running on it) + ansible.builtin.copy: + dest: "{{ grafana_alloy_env_checksum_file }}" + owner: root + group: root + mode: "0600" + # " " — the source token is what a later converge uses to + # tell "the operator owns this file" from "I rendered this from inventory". + # Parenthesised: filters bind tighter than comparisons, so an unbracketed + # ternary parses wrong. + content: > + {{ (grafana_alloy_api_token | default('', true) | string | length > 0) | ternary('inventory', 'host') }} + {{ _ga_env_post_stat.stat.checksum }} + # Under `--check` on an unprovisioned host the env-file copy only SIMULATES + # the write, so there is nothing to hash and stat.checksum is undefined. + when: (_ga_env_post_stat.stat.checksum | default('')) | length > 0 + + # …but outside check mode a missing checksum on an existing file means stat + # could not READ it, and skipping in silence would leave the host permanently + # untracked: no adoption guard, no out-of-band detection, nothing to say why. + - name: Fail when the env file exists but could not be hashed for the record + ansible.builtin.fail: + msg: >- + {{ grafana_alloy_secret_file }} exists but could not be hashed, so the + provenance record was NOT written — the next converge would silently lose + both the adoption guard and out-of-band edit detection on this host. This + means the file is unreadable to this connection; run the role with become + (the shipped playbooks do). + when: + - not ansible_check_mode + - _ga_env_post_stat.stat.exists + - (_ga_env_post_stat.stat.checksum | default('')) | length == 0 diff --git a/ansible/roles/grafana_alloy/tasks/preflight.yml b/ansible/roles/grafana_alloy/tasks/preflight.yml index f852de5..9cb6948 100644 --- a/ansible/roles/grafana_alloy/tasks/preflight.yml +++ b/ansible/roles/grafana_alloy/tasks/preflight.yml @@ -1,9 +1,11 @@ --- # Fail-loud preflight, run BEFORE any host mutation. Mirrors decdn_node's -# secret-file posture: this role never carries the API TOKEN in inventory and -# never reads the env file's values onto the control machine — it only greps -# shapes on the host itself. The non-secret half of the connection settings -# (endpoint URLs, instance IDs) MAY come from inventory; those are validated here +# secret-file posture: this role keeps the API token OFF inventory-committed, +# world-readable surfaces and never reads the env file's values onto the control +# machine — it only greps shapes on the host itself. The token MAY now ride a +# git-ignored secret.yml via grafana_alloy_api_token (the #39 relaxation); the +# non-secret half of the connection settings +# (endpoint URLs, instance IDs) MAY come from plain committed inventory; those are validated here # too, and each one that is set removes its key from the host-state gates below. # # Ordering matters for testability (molecule/validation): PURE-INVENTORY gates @@ -177,7 +179,7 @@ # These five are NON-secret and therefore inventory-settable; each one that is set # replaces its sys.env("GC_…") lookup in the rendered config and removes the # corresponding requirement from the host-state gates at the bottom of this file. -# The API token has no variable by design: config.alloy is 0644 on the host. +# The API token is now dually homeable too (grafana_alloy_api_token below). - name: Validate any inventory-provided Grafana Cloud endpoints ansible.builtin.assert: @@ -208,8 +210,9 @@ Mimir/Loki — so it is restricted to letters, digits, dot, underscore and dash, and must not be a token or carry whitespace. Got "{{ item.value }}". NOTE the Loki and OTLP instance IDs are each their own number, distinct - from the Prometheus one; the access-policy token is shared and stays in - {{ grafana_alloy_secret_file }} as GC_API_TOKEN. + from the Prometheus one; the access-policy token is shared and lives either + in {{ grafana_alloy_secret_file }} as GC_API_TOKEN or in + grafana_alloy_api_token in a git-ignored secret.yml — never in one of these. loop: - {name: grafana_alloy_prom_username, value: "{{ grafana_alloy_prom_username | string }}"} - {name: grafana_alloy_otlp_username, value: "{{ grafana_alloy_otlp_username | string }}"} @@ -226,10 +229,13 @@ - "'@' not in item" quiet: true fail_msg: >- - A Grafana Cloud access token (or a URL with embedded credentials) appears - in an inventory variable. config.alloy is world-readable on the host and - inventory is committed: the token belongs ONLY in - {{ grafana_alloy_secret_file }} as GC_API_TOKEN. + A Grafana Cloud access token (or a URL with embedded credentials) appears in + one of these six inventory variables. They are non-secret by design and are + rendered verbatim into config.alloy, which is world-readable 0644 on the + host — and unlike host_vars//secret.yml they are normally committed. + The token has exactly two homes: GC_API_TOKEN in + {{ grafana_alloy_secret_file }} on the host, or grafana_alloy_api_token in a + git-ignored secret.yml. Neither is one of these variables. loop: - "{{ grafana_alloy_prom_url | string }}" - "{{ grafana_alloy_prom_username | string }}" @@ -240,6 +246,95 @@ loop_control: label: "{{ item | truncate(24, true) }}" +# Type gates for the two credential-path knobs, in the same spirit as the +# observability sub-switches above. grafana_alloy_overwrite_host_file is the one +# flag in this role that authorises DESTROYING an operator-provisioned credential +# file, so it must not be reachable through YAML truthiness: `| bool` turns "yes", +# "on" and "1" into an authorisation the operator never typed, and silently reads +# any typo ("treu", "Flase") as false — producing a refusal whose message cannot +# explain itself. Require a real boolean, exactly like decdn_grafana_cloud_enabled. +# +# The token is checked for type but never for value here; its shape gate is below +# and its content never reaches an assert message. +- name: Validate the credential-path knobs + ansible.builtin.assert: + that: + - grafana_alloy_overwrite_host_file is boolean + - grafana_alloy_api_token is none or grafana_alloy_api_token is string + fail_msg: >- + grafana_alloy_overwrite_host_file must be a real boolean (true/false), not a + quoted string — it authorises rewriting {{ grafana_alloy_secret_file }} over + whatever the host currently has, so "yes"/"on"/"1" must not be able to grant + it by coercion. grafana_alloy_api_token must be a string (quote it: an + all-digit token reads as an int, and a `|` block scalar appends a newline). + +# The mirror image of the guard above (and the #39 relaxation): when +# grafana_alloy_api_token is SET it must be whitespace/newline-free, because the +# role JSON-encodes it into a single systemd EnvironmentFile line. Quotes and +# backslashes are deliberately NOT rejected: systemd's EnvironmentFile parser +# strips the surrounding double quotes and unescapes \" and \\ — exactly the two +# escapes JSON emits for the whitespace-free tokens this gate admits. +# +# It is NOT a C-escape parser (env-file.c is shell-like): \n, \t and \uXXXX would +# arrive at the daemon as literal backslash sequences, and an embedded blank would +# truncate the value. That is why rejecting whitespace here is load-bearing rather +# than cosmetic — do not relax it on the assumption that to_json will save you. +# +# \Z, not $: Python's $ also matches just before a trailing newline, so `^[^\s]+$` +# accepts "glc_abc\n" — the exact shape a YAML block scalar (`grafana_alloy_api_token: |`) +# produces — and the fail_msg below would be lying about newlines. roles/decdn_node +# uses \Z throughout for this reason; keep the two in step. +# +# no_log does not exist for asserts: their output lands in plain Ansible/CI logs, +# so the failure message must carry NO part of the configured value — report only +# which variable is wrong and why, never token-derived text. +- name: Validate the shape of an inventory-provided API token + ansible.builtin.assert: + that: + - grafana_alloy_api_token | string is match('^[^\s]+\Z') + # A placeholder copied out of the .example templates is shaped exactly like a + # real token, so nothing downstream can catch it: the file is authored, the + # config validates, the unit starts, and every push 401s behind a green + # deploy. These are the literals this repo itself ships, so matching them is + # free of false positives. Case-insensitive, substring — "glc_example_token_replace_me" + # and a hand-typed "CHANGEME" are the same mistake. + - not (grafana_alloy_api_token | string | lower + is search('replace_me|replaceme|changeme|example_token|your_token|xxxxx')) + quiet: true + fail_msg: >- + grafana_alloy_api_token must be the bare token value (non-empty, no + whitespace, no newline) — it becomes the single GC_API_TOKEN="…" line of + {{ grafana_alloy_secret_file }}, which systemd parses before dropping + privileges. Quotes/backslashes are fine (JSON-encoded); spaces are not, and + neither is a trailing newline — set it as an ordinary quoted scalar rather + than a `|` block scalar, which appends one. A placeholder value carried over + from one of the *.example templates is rejected too: it is shaped like a real + token, so nothing after this gate could tell the difference and every push + would 401 behind a green deploy. + # Empty keeps the host-provisioned posture; the gate only guards the inventory path. + when: grafana_alloy_api_token | default('', true) | string | length > 0 + +# A retry after the role wrote new token bytes but failed before recording their +# checksum must be distinguishable from a foreign edit. Hash the exact bytes the +# copy task will render; the secret itself stays suppressed and never enters task +# output. An empty token has no desired inventory-authored file. +- name: Resolve the desired inventory-authored env-file checksum + ansible.builtin.set_fact: + _ga_desired_inventory_sum: >- + {{ (_ga_desired_inventory_content | hash('sha256')) + if grafana_alloy_api_token | default('', true) | string | length > 0 else '' }} + vars: + # Keep this byte-for-byte identical to the copy task's block scalar in + # main.yml, including its trailing newline — the interrupted-rotation branch + # compares this hash against the file's, so any drift between the two makes + # every converge spuriously demand grafana_alloy_overwrite_host_file. + # A literal-clip block scalar (`|`) is used on BOTH sides rather than a quoted + # one-liner, so both render the same single trailing \n; molecule's + # grafana-cloud-token rotation case is the regression guard. + _ga_desired_inventory_content: | + GC_API_TOKEN={{ grafana_alloy_api_token | string | to_json(ensure_ascii=false) }} + no_log: true + - name: Validate the loopback listener addresses ansible.builtin.assert: that: @@ -309,8 +404,34 @@ ansible.builtin.stat: path: "{{ grafana_alloy_secret_file }}" follow: false + # sha256, matching what the record task writes into .sha256 — with stat's + # default (sha1) an operator cross-checking `sha256sum` would see a mismatch. + get_checksum: true + checksum_algorithm: sha256 register: _ga_secret_stat +- name: Require the Grafana Cloud secret environment file to exist + ansible.builtin.assert: + that: + - _ga_secret_stat.stat.exists + fail_msg: >- + {{ grafana_alloy_secret_file }} is missing, and neither credential path is + taken: grafana_alloy_api_token is empty, so this role expects an + operator-provisioned env file. Provision it on the target host per + roles/grafana_alloy/files/grafana-alloy.env.example: + umask 077 + sudo install -m 600 -o root -g root grafana-alloy.env {{ grafana_alloy_secret_file }} + or set grafana_alloy_api_token in a git-ignored secret.yml and let this + role author the file (token-only) for you. + # The inventory path authors the file itself, so absence is fine there; only + # the host-provisioned posture demands an operator-created file up front. + when: grafana_alloy_api_token | default('', true) | string | length == 0 + +# Kept as its own gate even though absence was just handled above: with +# grafana_alloy_api_token set an ABSENT file is legitimate (the role authors it), +# but a PRESENT one must still be a real root-owned 0600 regular file — systemd +# reads EnvironmentFile= as root before dropping privileges. +# # stat.exists alone cannot see the difference between a regular file and a # symlink planted at the credential path that later passes a root chown/chmod # would harden whatever TARGET it points at — exactly what the secret-file gate @@ -318,21 +439,143 @@ - name: Assert the secret file is a real root-owned 0600 file ansible.builtin.assert: that: - - _ga_secret_stat.stat.exists - _ga_secret_stat.stat.isreg | default(false) - (_ga_secret_stat.stat.pw_name | default('')) == 'root' - (_ga_secret_stat.stat.gr_name | default('')) == 'root' - _ga_secret_stat.stat.mode | string == '0600' + # stat omits `checksum` entirely when the file is not READABLE to this + # connection (it still reports exists/isreg/owner/mode). Every provenance + # comparison below is ` == stat.checksum`, so a missing checksum + # defaulted to '' would make the adoption guard — the one gate whose whole + # job is catching a vanished secret.yml — pass vacuously. Fail closed here + # instead, so the comparisons downstream can index it unguarded. + - _ga_secret_stat.stat.checksum is defined fail_msg: >- - {{ grafana_alloy_secret_file }} is missing, not a regular file (symlink? - directory?), or not owned root:root 0600 (found owner + {{ grafana_alloy_secret_file }} is not a regular file (symlink? directory?), + not owned root:root 0600 (found owner {{ _ga_secret_stat.stat.pw_name | default('?') }}, mode - {{ _ga_secret_stat.stat.mode | default('?') }}). Provision it on the target - host per roles/grafana_alloy/files/grafana-alloy.env.example: - umask 077 - sudo install -m 600 -o root -g root grafana-alloy.env {{ grafana_alloy_secret_file }} - systemd expands its KEY=value pairs into the unit before dropping - privileges, so root ownership is sufficient — do not relax it. + {{ _ga_secret_stat.stat.mode | default('?') }}), or could not be hashed — + the last means it is unreadable to this connection, so run the role with + become (the playbooks do). Fix it on the target host; systemd expands its + KEY=value pairs into the unit before dropping privileges, so root ownership + is sufficient — do not relax it. + when: _ga_secret_stat.stat.exists + +# --- Secret-file provenance ------------------------------------------------------- +# Mirrors decdn_node's #39 machinery (roles/decdn_node/tasks/main.yml): the role +# records " " of the env file after every enabled converge, so the +# next run can tell three states apart. Only this non-sensitive record is ever +# read back — never the secret file itself. +# +# An absent record means "no post-relaxation converge yet". A present malformed, +# unsafe or unrelated file is rejected on the target before slurp, so an inventory +# typo cannot copy arbitrary /etc content into controller memory. +- name: Validate the env-file provenance record before reading it + ansible.builtin.import_tasks: validate-env-record.yml + +- name: Read the env-file checksum this role last recorded + ansible.builtin.slurp: + src: "{{ grafana_alloy_env_checksum_file }}" + register: _ga_recorded + when: _ga_record_stat.stat.exists + +# Shape was proven before slurp, so an existing record always has exactly two +# fields. Conditional expressions avoid indexing a skipped slurp result when the +# record is absent. +- name: Resolve the secret file's provenance + ansible.builtin.set_fact: + _ga_record_source: >- + {{ (_ga_recorded.content | b64decode | trim).split()[0] + if _ga_record_stat.stat.exists else '' }} + _ga_recorded_sum: >- + {{ (_ga_recorded.content | b64decode | trim).split()[1] + if _ga_record_stat.stat.exists else '' }} + +# The dangerous half of inferring operator intent. With grafana_alloy_api_token +# empty the operator-supplied file is adopted as-is — but that ALSO happens when +# the git-ignored secret.yml carrying the token went missing (fresh clone, new +# control machine, undecrypted vault) AFTER an earlier inventory-path deploy: the +# role would adopt its OWN previously-written file as host-provisioned and a +# ROTATED token in the missing secret.yml would silently never land. Fail loud +# instead, exactly like the decdn_node adoption guard. +- name: Refuse to adopt a role-authored env file as host-provisioned + ansible.builtin.assert: + that: + # stat.checksum is proven defined by the 0600 gate above, which shares this + # task's `_ga_secret_stat.stat.exists` guard — no default() to fail open on. + - _ga_record_source != 'inventory' + or _ga_recorded_sum != _ga_secret_stat.stat.checksum + fail_msg: >- + grafana_alloy_api_token is empty, but {{ grafana_alloy_secret_file }} is + byte-for-byte the file THIS ROLE last wrote from inventory — so the token + has gone missing from the control machine (an absent + host_vars//secret.yml, a fresh clone, an undecrypted vault) rather + than being deliberately handed to the host. Treating it as + host-provisioned would deploy green while silently keeping the old token. + Restore grafana_alloy_api_token, or — if you really are migrating this node + to the host-provisioned path — hand the file over explicitly by discarding + the role's provenance record: + sudo rm {{ grafana_alloy_env_checksum_file }} + when: + - grafana_alloy_api_token | default('', true) | string | length == 0 + - _ga_secret_stat.stat.exists + +# The mirror direction: an inventory token REWRITES the file wholesale (and +# token-only), losing any GC_* lines added outside Ansible. Existing bytes are +# safe without an opt-in only when (a) a matching inventory record proves this +# role wrote them, or (b) they already equal the exact desired token-only bytes, +# which is the retry state after a failed run wrote the file but not the record. +# Missing provenance is unknown, not permission to destroy a pre-record host file. +- name: Require confirmation to overwrite an env file this role did not write + ansible.builtin.assert: + that: + - grafana_alloy_overwrite_host_file | bool + fail_msg: >- + grafana_alloy_api_token is set, but {{ grafana_alloy_secret_file }} on this + host is not a file this role authored — the provenance record reads + "{{ _ga_record_source }}" and/or its checksum no longer matches, so it was + written or edited by someone else. Authoring it token-only from inventory + would DISCARD those lines, and because this role never reads the file it + cannot tell you what would be lost. Fold any extra GC_* keys into their + matching inventory variables first, then either clear + grafana_alloy_api_token to keep the host's copy or set + grafana_alloy_overwrite_host_file: true to confirm the rewrite. + when: + - grafana_alloy_api_token | default('', true) | string | length > 0 + - _ga_secret_stat.stat.exists + # Same as the adoption guard: stat.checksum is guaranteed defined here by the + # 0600 gate, which runs under the same exists condition. + - >- + not ( + (_ga_record_source == 'inventory' + and _ga_recorded_sum == _ga_secret_stat.stat.checksum) + or + (_ga_desired_inventory_sum == _ga_secret_stat.stat.checksum) + ) + +# Inventory-authoring mode makes the final env file TOKEN-ONLY, so everything +# else MUST come from inventory — checked HERE (before any mutation) because the +# env-file greps below can only vouch for keys that this run is about to discard. +- name: Demand every non-token connection setting from inventory for a token-authored env file + ansible.builtin.assert: + that: + - grafana_alloy_prom_url | string | length > 0 + - grafana_alloy_prom_username | string | length > 0 + - grafana_alloy_otlp_endpoint | string | length > 0 + - not (grafana_alloy_logs_enabled | bool) or (grafana_alloy_loki_url | string | length > 0) + - not (grafana_alloy_logs_enabled | bool) or (grafana_alloy_loki_username | string | length > 0) + quiet: true + fail_msg: >- + With grafana_alloy_api_token set, {{ grafana_alloy_secret_file }} is + REWRITTEN as a token-only file, so a present GC_PROM_REMOTE_WRITE_URL / + GC_PROM_USERNAME / GC_OTLP_ENDPOINT / GC_LOKI_* line on the CURRENT copy of + the file cannot satisfy preflight — it would be discarded. Set the matching + inventory variables instead (grafana_alloy_prom_url, _prom_username, + _otlp_endpoint, _loki_url, _loki_username; the Loki pair only matters while + logs are enabled). grafana_alloy_otlp_username stays optional — without it + traces fall back to the Prometheus instance ID. + # Empty keeps the host-provisioned posture, where the greps below judge the file. + when: grafana_alloy_api_token | default('', true) | string | length > 0 # systemd skips an EnvironmentFile line it cannot parse SILENTLY, so each required # key is grepped for presence + value shape directly on the host: grep keeps the @@ -341,8 +584,10 @@ # # "Required" is now conditional: a key is only needed when nothing in inventory # supplies that value, and the Loki pair is only needed when logs are enabled. -# GC_API_TOKEN is ALWAYS required — it is the one value that must never live in -# inventory. +# That includes GC_API_TOKEN itself since the grafana_alloy_api_token relaxation: +# the grep only fires when the variable is empty. The whole loop is also scoped to +# the HOST-PROVISIONED posture: with an inventory token the file below is replaced +# wholesale moments after this runs, so its current contents prove nothing. - name: Confirm every required credential key is present and shaped correctly ansible.builtin.command: cmd: grep -qE -- '{{ item.regex }}' {{ grafana_alloy_secret_file | quote }} @@ -357,8 +602,8 @@ # already newline-free by construction. - key: GC_API_TOKEN regex: ^GC_API_TOKEN=[^[:space:]#][^[:space:]]*$ - var: "(none — the token must never be set in inventory)" - required: true + var: grafana_alloy_api_token + required: "{{ grafana_alloy_api_token | default('', true) | string | length == 0 }}" - key: GC_PROM_REMOTE_WRITE_URL regex: ^GC_PROM_REMOTE_WRITE_URL=https://[^[:space:]]+$ var: grafana_alloy_prom_url @@ -381,7 +626,7 @@ required: "{{ grafana_alloy_logs_enabled | bool and grafana_alloy_loki_username | string | length == 0 }}" loop_control: label: "{{ item.key }}" - when: item.required | bool + when: (grafana_alloy_api_token | default('', true) | string | length == 0) and (item.required | bool) register: _ga_secret_grep - name: Report which credential key was missing or malformed @@ -393,8 +638,7 @@ roles/grafana_alloy/files/grafana-alloy.env.example) — URLs must be https:// and every value non-empty, since systemd silently drops unparseable lines so a quoting typo looks identical to a missing key — or, - for everything except the token, set the matching inventory variable - instead: {{ _ga_missing | map(attribute='var') | join(', ') }}. + set the matching inventory variable instead: {{ _ga_missing | map(attribute='var') | join(', ') }}. vars: # Skipped loop items (key not required) carry no rc, so filter on rc being # defined first: rejectattr alone would count every skipped key as missing. @@ -414,7 +658,9 @@ # Prometheus fallback and 401s the entire trace pipeline while metrics and logs # stay healthy. The required-key gate above cannot cover it (it only greps keys it # demands), hence this present-then-shaped pair. Skipped when inventory supplies -# the ID, because the template then inlines it and emits no sys.env lookup at all. +# the ID, because the template then inlines it and emits no sys.env lookup at all +# — and likewise on the inventory-authoring path, where the rewrite guarantees no +# GC_OTLP_USERNAME survives into the final file. - name: Detect a host-provided OTLP instance ID ansible.builtin.command: cmd: grep -qE -- '^GC_OTLP_USERNAME=.*[^[:space:]]' {{ grafana_alloy_secret_file | quote }} @@ -422,7 +668,9 @@ check_mode: false # read-only gate: must also run under make check failed_when: false register: _ga_otlp_user_present - when: grafana_alloy_otlp_username | string | length == 0 + when: + - grafana_alloy_otlp_username | string | length == 0 + - grafana_alloy_api_token | default('', true) | string | length == 0 # Same charset as the inventory instance-ID gate, plus the optional double quotes # systemd strips off an EnvironmentFile value. @@ -453,4 +701,5 @@ grafana_alloy_otlp_username in inventory instead. when: - grafana_alloy_otlp_username | string | length == 0 + - grafana_alloy_api_token | default('', true) | string | length == 0 - (_ga_otlp_user_present.rc | default(1)) == 0 diff --git a/ansible/roles/grafana_alloy/tasks/validate-env-record.yml b/ansible/roles/grafana_alloy/tasks/validate-env-record.yml new file mode 100644 index 0000000..916cecf --- /dev/null +++ b/ansible/roles/grafana_alloy/tasks/validate-env-record.yml @@ -0,0 +1,76 @@ +--- +# Validate the provenance record before any task reads it onto the controller or +# trusts it as authorization to rewrite a root-only credential file. The path is +# already constrained by validate-paths.yml to `.sha256`. + +- name: Inspect the env-file provenance record + ansible.builtin.stat: + path: "{{ grafana_alloy_env_checksum_file }}" + follow: false + get_checksum: false + register: _ga_record_stat + +- name: Require an existing provenance record to be a protected regular file + ansible.builtin.assert: + that: + - _ga_record_stat.stat.isreg | default(false) + - (_ga_record_stat.stat.pw_name | default('')) == 'root' + - (_ga_record_stat.stat.gr_name | default('')) == 'root' + - _ga_record_stat.stat.mode | string == '0600' + fail_msg: >- + {{ grafana_alloy_env_checksum_file }} exists but is not a root:root 0600 + regular file (symlinks are refused). Move the unrelated file away or restore + the role-authored provenance record before deploying. + when: _ga_record_stat.stat.exists + +# Probe the complete byte stream on the target with no stdout before slurp. A +# line-oriented or NUL-record grep can accept one valid prefix and ignore trailing +# unrelated bytes; Python's fullmatch cannot. +# +# Use the interpreter Ansible already discovered for this host rather than a bare +# `python3`: ansible.cfg sets interpreter_python = auto_silent, so a target whose +# interpreter is /usr/bin/python3.11 or a venv has no guarantee of a `python3` on +# the become shell's PATH — and rc=2 from a missing interpreter would otherwise be +# reported below as "your record is corrupt". +# +# The script exits 0 (matches) or 1 (does not). ANY other rc is an execution +# failure, not a verdict about the file, so it is not suppressed. +- name: Check the provenance record shape on the target + vars: + # `.get()` on both, so neither an ungathered fact nor an unset inventory + # variable raises; the literal is the last resort, not the first choice. + _ga_python: >- + {{ ansible_facts.get('discovered_interpreter_python') + or hostvars[inventory_hostname].get('ansible_python_interpreter') + or '/usr/bin/python3' }} + ansible.builtin.command: + argv: + - "{{ _ga_python }}" + - -c + - >- + import pathlib, re, sys; + raise SystemExit(0 if re.fullmatch( + rb'(inventory|host) [0-9a-f]{64}\n?', + pathlib.Path(sys.argv[1]).read_bytes()) else 1) + - "{{ grafana_alloy_env_checksum_file }}" + changed_when: false + check_mode: false + # Only the two verdict codes are ours to interpret; anything else (interpreter + # missing, EACCES, OOM, rc=126/127) fails the task in its own voice, with the + # module's real error, instead of being laundered into a corruption claim. + failed_when: _ga_record_shape.rc not in [0, 1] + register: _ga_record_shape + when: _ga_record_stat.stat.exists + +- name: Require an existing provenance record to have the role-owned format + ansible.builtin.assert: + that: + - (_ga_record_shape.rc | default(1)) == 0 + fail_msg: >- + {{ grafana_alloy_env_checksum_file }} exists but is not a role-owned + provenance record of the exact form " ". Refusing + to read, overwrite, or trust an unrelated /etc file. If this file IS the + role's record, do not delete it blindly — a lost record disables the + adoption guard; check it with: + sudo cat {{ grafana_alloy_env_checksum_file }} + when: _ga_record_stat.stat.exists diff --git a/ansible/roles/grafana_alloy/tasks/validate-paths.yml b/ansible/roles/grafana_alloy/tasks/validate-paths.yml index 0a324ee..9e6bb84 100644 --- a/ansible/roles/grafana_alloy/tasks/validate-paths.yml +++ b/ansible/roles/grafana_alloy/tasks/validate-paths.yml @@ -42,9 +42,55 @@ value: "{{ grafana_alloy_secret_file }}" under: /etc regex: '^/etc/[A-Za-z0-9._-]+$' + # Written by the enabled path and retained beside the secret across disable. + # It authorizes later adoption/overwrite decisions, so keep the same anchored + # single-segment treatment as the secret file. + - name: grafana_alloy_env_checksum_file + value: "{{ grafana_alloy_env_checksum_file }}" + under: /etc + regex: '^/etc/[A-Za-z0-9._-]+$' loop_control: label: "{{ item.name }}" +# This root-written record is read onto the controller and used to authorize a +# later rewrite. Letting inventory aim it independently at any direct child of +# /etc would turn a typo into an arbitrary-file read/overwrite. Keep it coupled to +# the already-constrained credential path as its literal `.sha256` companion. +- name: Require the provenance record to sit beside the credential file + ansible.builtin.assert: + that: + - grafana_alloy_env_checksum_file == grafana_alloy_secret_file ~ '.sha256' + quiet: true + fail_msg: >- + grafana_alloy_env_checksum_file must be exactly grafana_alloy_secret_file + + ".sha256" (expected {{ grafana_alloy_secret_file }}.sha256, got + {{ grafana_alloy_env_checksum_file }}). The record is root-read, rewritten, + and trusted as overwrite authorization, so it cannot be aimed independently + at another file under /etc. + +# The disable path recursively removes config_dir. It must never be equal to, or +# an ancestor of, either retained secret artifact or the systemd unit used as the +# ownership marker; otherwise a valid-looking override can erase the evidence that +# prevents stale-token adoption (or an entire systemd directory). +- name: Keep destructive directories away from retained Alloy files + ansible.builtin.assert: + that: + - grafana_alloy_config_dir != grafana_alloy_secret_file + - grafana_alloy_config_dir != grafana_alloy_env_checksum_file + - not grafana_alloy_secret_file.startswith(grafana_alloy_config_dir ~ '/') + - not grafana_alloy_env_checksum_file.startswith(grafana_alloy_config_dir ~ '/') + - grafana_alloy_config_dir != _ga_unit_path + - not _ga_unit_path.startswith(grafana_alloy_config_dir ~ '/') + - grafana_alloy_config_file not in + [grafana_alloy_secret_file, grafana_alloy_env_checksum_file, _ga_unit_path] + quiet: true + fail_msg: >- + grafana_alloy_config_dir/config_file must not equal or contain the retained + credential ({{ grafana_alloy_secret_file }}), provenance record + ({{ grafana_alloy_env_checksum_file }}), or managed unit ({{ _ga_unit_path }}). + The disable path removes config_dir recursively, so overlapping paths would + delete security evidence or unrelated system files. + # The role templates the config INTO the directory it creates and chowns; a file # outside it would be written to an unmanaged (possibly unsafe) location and # would survive the disable path's directory removal.