feat(ansible): allow the grafana_alloy API token via git-ignored inventory

AGENTS.md

@@ -85,11 +85,16 @@ charts/
   (Alloy's in-process `node_exporter`, curated collector set, `systemd` collector scoped
   to the units that matter), **journald** → Grafana Cloud Loki, Alloy's own health, and
   the daemon's OTLP spans. Host metrics and logs carry `job="integrations/node_exporter"`
-  so Grafana Cloud's prebuilt Linux Server dashboards work unmodified. Only the API token
-  is host-provisioned (`0600 /etc/grafana-alloy.env`, read via `sys.env` — Alloy has no
-  `--config.expand-env`); the non-secret endpoints and the three per-service instance IDs
-  are inventory variables that fall back to their `GC_…` env key when empty, and preflight
-  refuses a token in any of them. Two hardening relaxations are conditional on the signals being on
+  so Grafana Cloud's prebuilt Linux Server dashboards work unmodified. The API token is the
+  only credential, and it has two homes: operator-provisioned on the host
+  (`0600 /etc/grafana-alloy.env`) or carried by `grafana_alloy_api_token` from a git-ignored
+  `host_vars/<node>/secret.yml`, in which case the role authors that file itself as a
+  token-only `EnvironmentFile` and tracks who wrote it in `<secret-file>.sha256` (the same
+  `decdn_rpc_url` dual-home pattern). Either way `config.alloy` reads it as
+  `sys.env("GC_API_TOKEN")` — Alloy has no `--config.expand-env`. The non-secret endpoints
+  and the three per-service instance IDs are inventory variables that fall back to their
+  `GC_…` env key when empty, and preflight refuses a token in any of them; with an inventory
+  token it also requires all of them, since the authored file is token-only. Two hardening relaxations are conditional on the signals being on
   (`ProtectHome=read-only` for correct filesystem metrics, `SupplementaryGroups=
   systemd-journal adm` for journal access — without which collection is silently empty);
   teardown is gated on the managed-by marker in the unit, so a foreign Alloy is never
@@ -121,8 +126,10 @@ deploys (its targets must run from `ansible/`). `make help` lists root targets.
 make hooks            # one-time: install pre-commit git hook (pip install pre-commit first)
 make lint             # all pre-commit hooks on all files (hygiene, shellcheck, yamllint, markdown)
 make lint-ansible     # vendor collections + full ansible-lint (production profile)
-make molecule         # containerised converge/verify of the decdn_node role — all six
-                      # scenarios in parallel (needs Docker); cap with JOBS=<n>
+make molecule         # containerised converge/verify of the decdn_node + grafana_alloy
+                      # roles — every molecule/*/ scenario in parallel (the target
+                      # discovers them by glob, so adding one needs no edit here);
+                      # needs Docker, cap with JOBS=<n>
 make molecule-serial  # the same suite one scenario at a time (readable failure output)
 make lint-helm        # chart: helm lint + render tests + kubeconform + schema keys (needs helm, yq, Docker)
 make lint-alloy       # grafana_alloy: render its templates + `alloy validate` them with the real
@@ -137,7 +144,8 @@ make check / deploy   # deCDN node (site.yml): dry-run / provision
 make build / galaxy-check       # stage + build the decdn.node collection, then validate it
 ```
 
-**Galaxy collection (`decdn.node`).** The two roles (`baseline` + `decdn_node`) ship as a
+**Galaxy collection (`decdn.node`).** The three roles (`baseline` + `decdn_node` +
+`grafana_alloy`) ship as a
 distributable collection. The overlay lives in `ansible/galaxy/` and is staged into a clean
 collection tree by `galaxy/build.sh` — there is **no** `galaxy.yml` at the `ansible/` root
 (that would make ansible-lint treat the deploy project as a collection). Build/validate with

ansible/README.md

@@ -207,8 +207,9 @@ Only the API token is a secret. Put the non-secret connection settings in invent
 once for the fleet (`grafana_alloy_prom_url`, `_prom_username`, `_otlp_endpoint`,
 `_otlp_username`, `_loki_url`, `_loki_username` — Prometheus, OTLP and Loki each have
 their **own** instance ID, so copy each from the portal page that names it), and
-provision just the token **on each target host** (it never transits this repo or the
-control machine):
+provision just the token **on each target host** (on this path it never transits this
+repo or the control machine; the inventory alternative below trades that for a
+git-ignored `secret.yml`):
 
 ```bash
 umask 077
@@ -220,6 +221,16 @@ Each of those variables is optional: left empty, the value is read from the matc
 `GC_…` key in that file instead (`roles/grafana_alloy/files/grafana-alloy.env.example`
 lists them all; URLs must be https).
 
+Alternatively the token itself can ride git-ignored inventory — set
+`grafana_alloy_api_token: "glc_…"` in `host_vars/<node>/secret.yml`, same channel as
+`decdn_rpc_url`, and the role authors `/etc/grafana-alloy.env` for you (token-only,
+root 0600). The authoring rewrites the file wholesale, so any hand-added `GC_*` keys
+must move to their inventory variables first; provenance guards fail loud on silent
+adoption or clobbering. Migrating a host that already has a hand-provisioned file
+needs `grafana_alloy_overwrite_host_file: true` for exactly one converge — set it
+back to `false` afterwards, or the guard stays off on that host. See "The API token
+has two homes now" in the role README for the ordered procedure in both directions.
+
 **Upgrading a host deployed before machine monitoring existed:** journald shipping is on
 by default and needs a Loki endpoint + instance ID the old four-key file does not carry,
 so preflight fails until you either set `grafana_alloy_loki_url` / `_loki_username` in
@@ -254,9 +265,11 @@ make lint-alloy     # render grafana_alloy's templates, then `alloy validate` th
 `make molecule` runs every scenario under `molecule/`: **`default`** (described below),
 `schema` (config key-set drift against the upstream field list), `validation` (bad knobs,
 for both roles, must be rejected by their own asserts), `generate-keystore` (opt-in
-host-side wallet), `host-env` (host-provisioned `/etc/decdn/decdn.env`) and
+host-side wallet), `host-env` (host-provisioned `/etc/decdn/decdn.env`),
 `slow-readiness` (advisory `/metrics` probe timeout), `grafana-cloud` (the opt-in
-observability wiring — see [Grafana Cloud observability](#grafana-cloud-observability-opt-in)).
+observability wiring — see [Grafana Cloud observability](#grafana-cloud-observability-opt-in))
+and `grafana-cloud-token` (the same wiring with the API token carried through
+git-ignored inventory instead: role-authored env file + provenance record).
 They are independent, so they run concurrently —
 ~151s instead of ~595s — and each line of output is prefixed with its scenario name
 because the runs interleave. `make molecule-serial` is the escape hatch when that
@@ -301,7 +314,8 @@ the RPC URL when it is not provisioned on the host instead). Highlights:
 | `decdn_rpc_url` + 3 contract addresses | `""` | **required** per node — `rpc_url` from a host-provisioned `0600 /etc/decdn/decdn.env` (preferred) *or* `host_vars/<node>/secret.yml`, addresses in `main.yml`; sourced from an ADR/deployment. |
 | `decdn_region` / `decdn_bind_port` / `decdn_rate_per_mb` | `""` / `4433` / `10` | node identity, QUIC port, USDC base units/MB. |
 | `decdn_env_checksum_file` / `decdn_env_overwrite_host_file` | `/etc/decdn/.decdn.env.sha256` / `false` | Provenance record for the secret env file (`0600 root`), and the opt-in that lets an inventory `decdn_rpc_url` overwrite a host-edited one. |
-| `decdn_grafana_cloud_enabled` | `false` | ONE mirrored knob (identical default in both roles) wiring on Grafana Cloud observability: installs + configures `grafana_alloy` — node metrics, machine metrics, journald, agent health — AND injects `otlp_endpoint` into `node.toml`. Only the API token is provisioned per host; the rest are inventory variables. Label/cost guardrails in the role README. |
+| `decdn_grafana_cloud_enabled` | `false` | ONE mirrored knob (identical default in both roles) wiring on Grafana Cloud observability: installs + configures `grafana_alloy` — node metrics, machine metrics, journald, agent health — AND injects `otlp_endpoint` into `node.toml`. Only the API token is provisioned per host *or* carried by `grafana_alloy_api_token` in git-ignored inventory; the rest are inventory variables. Label/cost guardrails in the role README. |
+| `grafana_alloy_api_token` / `_env_checksum_file` / `_overwrite_host_file` | `""` / `/etc/grafana-alloy.env.sha256` / `false` | The dual-homed Grafana Cloud token and its provenance machinery (`#39` parity with the row above); the record path is fixed to `<secret-file>.sha256` and survives disable with the secret. See [`roles/grafana_alloy/README.md`](roles/grafana_alloy/README.md). |
 
 ---
 

ansible/ansible.cfg

@@ -10,6 +10,14 @@ stdout_callback = default
 result_format = yaml
 nocows = True
 interpreter_python = auto_silent
+# Run modules over the existing SSH session instead of staging a script into the
+# target's tmpdir. Two roles write credentials with `copy: content=…` (the node's
+# decdn.env, Alloy's grafana-alloy.env), and without pipelining the module args —
+# including the plaintext secret — are written to ~/.ansible/tmp on the target
+# before the atomic move. `no_log` covers task output, not that file. Requires
+# `requiretty` to be off in sudoers, which is the default on the Debian/Ubuntu
+# targets this project supports.
+pipelining = True
 
 [privilege_escalation]
 become = True

ansible/galaxy/CHANGELOG.md

@@ -8,6 +8,24 @@ collection adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html)
 
 ### Added
 
+- `grafana_alloy_api_token`: the Grafana Cloud API token may now come from a
+  git-ignored `host_vars/<node>/secret.yml` instead of only being operator-
+  provisioned on the host, the same dual-home pattern `decdn_rpc_url` uses. Set, the
+  role authors `/etc/grafana-alloy.env` itself as a **token-only** file at
+  `root:root 0600`; left empty (the default), behaviour is unchanged. Because the
+  authored file is token-only, every other connection setting must then come from
+  inventory — preflight demands them before any mutation.
+- `grafana_alloy_env_checksum_file` (default `/etc/grafana-alloy.env.sha256`): a
+  root-owned `0600` provenance record, `<source> <sha256>`, written after the agent
+  is running on that content. It lets a later converge tell an operator-owned file
+  from a role-authored one, and drives two fail-loud guards — refusing to adopt a
+  role-authored file as host-provisioned when the token goes missing from the
+  control machine, and refusing to clobber a file this role did not write. Must
+  remain exactly `<grafana_alloy_secret_file>.sha256`.
+- `grafana_alloy_overwrite_host_file` (default `false`): explicit opt-in to rewrite
+  an env file of foreign or unknown provenance. Required for one converge when
+  migrating an existing host onto the inventory path; set it back to `false`
+  afterwards or the guard stays disabled on that host.
 - `baseline_packages` now includes `acl`. `decdn_node` runs two tasks as the
   unprivileged `decdn` user (`decdn key-gen`, and the `decdn config validate` gate),
   and on Debian Ansible needs ACL support to hand the temp module file to that user.

ansible/inventory/host_vars/decdn-node-1/secret.yml.example

@@ -30,3 +30,13 @@ decdn_rpc_url: "https://sepolia-rollup.arbitrum.io/rpc"
 #   AWS_SESSION_TOKEN: "..."        # only for assume-role / SSO flows
 #
 # Pair that with decdn_cache_origin_s3_use_default_chain: true in main.yml.
+
+# --- Grafana Cloud (only when decdn_grafana_cloud_enabled is on) ---------------
+#
+# The opt-in observability agent accepts its API token through the same channel:
+# set grafana_alloy_api_token here and the role authors /etc/grafana-alloy.env on
+# the host for you (token-only, root 0600), skipping the hand-provisioning shown
+# in roles/grafana_alloy/files/grafana-alloy.env.example. SENSITIVE — leave it
+# commented out to keep the operator-provisioned host path instead.
+#
+# grafana_alloy_api_token: "glc_example_token_replace_me"

ansible/molecule/grafana-cloud-token/converge.yml

@@ -0,0 +1,38 @@
+---
+# Token-via-inventory converge: every non-secret connection setting comes from
+# this playbook's vars (the normal committed inventory half) and the token rides
+# grafana_alloy_api_token — in a real deployment it would live in git-ignored
+# inventory/host_vars/<node>/secret.yml, exactly like decdn_rpc_url. Nothing
+# stages /etc/grafana-alloy.env; the role must author it itself.
+#
+# Logs stay enabled with BOTH Loki knobs supplied from inventory so the authored
+# file needs zero GC_* keys beyond the token — the all-or-nothing contract has
+# nothing to clobber on these throwaway containers.
+- name: Converge
+  hosts: all
+  become: true
+  vars:
+    stub_bin: "{{ lookup('ansible.builtin.env', 'MOLECULE_SCENARIO_DIRECTORY') }}/files/alloy-stub"
+    decdn_grafana_cloud_enabled: true
+    grafana_alloy_install_method: manual
+    grafana_alloy_manual_bin_src: "{{ stub_bin }}"
+    grafana_alloy_deployment_environment: production
+    grafana_alloy_region: US
+
+    # --- The point of this scenario -------------------------------------------------
+    # Deliberately fake token literal below, not a committed secret — keep only
+    # KICS' hardcoded-credential query quiet about exactly that one line.
+    # kics-scan ignore-line
+    grafana_alloy_api_token: molecule-test-token
+
+    # All six non-secret settings satisfied by inventory, so preflight requires no
+    # env-file key but GC_API_TOKEN... which the variable now covers too.
+    grafana_alloy_prom_url: https://prometheus-prod-xx.molecule.invalid/api/prom/push
+    grafana_alloy_prom_username: "1234567"
+    grafana_alloy_otlp_endpoint: https://otlp-gateway-prod-xx.molecule.invalid/otlp
+    grafana_alloy_loki_url: https://logs-prod-xx.molecule.invalid/loki/api/v1/push
+    grafana_alloy_loki_username: "7654321"
+
+  roles:
+    - role: grafana_alloy
+      tags: [observability]

ansible/molecule/grafana-cloud-token/files/alloy-stub

@@ -0,0 +1,31 @@
+#!/bin/sh
+# Molecule stub standing in for the Grafana Alloy binary in CI.
+#
+# The grafana-cloud scenario runs the real role logic (user/group/dirs, secret
+# gate, template rendering, hardened unit, systemd start) against this shim so no
+# ~160 MB release download is needed per CI run. It intentionally does NOT listen
+# on any socket: runtime socket assertions are out of scope for containerised CI
+# (see ../verify.yml comments) and remain covered by real-host deploys + content-
+# level checks here.
+set -u
+
+case "${1-}" in
+  # Version backstop in tasks/install.yml (manual mode degrades to rc==0 anyway).
+  --version)
+    echo "alloy, version v${MOLECULE_STUB_VERSION:-0.0.0}-molecule-stub"
+    ;;
+  # validate: the real binary builds the component graph and rejects an
+  # unloadable config. A stub CANNOT do that, so this exits 0 and proves the
+  # role's gate is wired up — nothing more. Content correctness is asserted
+  # textually in verify.yml and, against the REAL pinned binary, by
+  # ../../tests/alloy-config/validate.sh (`make lint-alloy`, run in CI). Do not
+  # read a green run here as "the configuration loads".
+  validate | fmt)
+    :
+    ;;
+  # Long-running foreground process keeps the Type=simple unit 'running'.
+  run)
+    exec sleep infinity
+    ;;
+esac
+exit 0

ansible/molecule/grafana-cloud-token/molecule.yml

@@ -0,0 +1,43 @@
+---
+# Inventory-path exercise of the #39 relaxation: grafana_alloy_api_token set in
+# (git-ignored) inventory instead of a host-provisioned /etc/grafana-alloy.env.
+# Positive path only — the broken-input matrix lives in ../validation; the
+# host-provisioned positive path lives in ../grafana-cloud. No prepare-stage env
+# file AT ALL here by design: the absence is what proves preflight's existence
+# gate honours the token variable, and that the role authors the file itself.
+#
+# Same two boundaries as ../grafana-cloud: the stub binds no sockets and exits 0
+# for every subcommand, so nothing here proves runtime loadability (`make
+# lint-alloy` does that against the real pinned binary).
+driver:
+  name: docker
+platforms:
+  - name: decdn-node-grafana-cloud-token
+    # Same digest-pinned image as the other scenarios (repo convention). Re-resolve
+    # all of them together to bump.
+    image: geerlingguy/docker-debian12-ansible@sha256:4553092be2c00b1ffe580927b9ff03f3c3a0df32b7dd693a3eb02efb6c2b77b7
+    pre_build_image: true
+    command: /usr/lib/systemd/systemd
+    privileged: true
+    cgroupns_mode: host
+    volumes:
+      - /sys/fs/cgroup:/sys/fs/cgroup:rw
+provisioner:
+  name: ansible
+  env:
+    ANSIBLE_ROLES_PATH: "${MOLECULE_PROJECT_DIRECTORY}/roles"
+    ANSIBLE_COLLECTIONS_PATH: "${MOLECULE_PROJECT_DIRECTORY}/collections"
+    ANSIBLE_PIPELINING: "true"
+verifier:
+  name: ansible
+scenario:
+  test_sequence:
+    # Leading `destroy` so an orphaned container is never reused — see
+    # ../default/molecule.yml. No prepare playbook: unlike ../grafana-cloud there
+    # are NO host-provisioned fixtures to stage — their absence is the point.
+    - destroy
+    - create
+    - converge
+    - idempotence
+    - verify
+    - destroy