feat(ansible): opt-in Grafana Cloud observability via new grafana_alloy role

[thiras]

Implements #55 (Ansible path only — the Helm chart is deliberately untouched).

What

Adds an opt-in Grafana Cloud Application Observability integration for Ansible-deployed deCDN nodes: a dedicated, loopback-only Grafana Alloy agent that scrapes the node's /metrics and receives the daemon's OTLP span exports, shipping both to your Grafana Cloud org.

New role ansible/roles/grafana_alloy

• One mirrored flag: decdn_grafana_cloud_enabled is defined identically in this role and decdn_node, so inventory overrides it once and both sides react. Off by default; the off-path removes any prior install (clean rollback).
• Install: pinned upstream release (v1.19.2) as .deb, sha256-verified against hand-pinned per-arch digests; dpkg-query-gated idempotence; manual mode copies a control-machine binary (used by CI).
• Fail-loud preflight before any host mutation (AGENTS.md hard rule 4):
  • secret file must exist as a regular root:root 0600 file (symlinks refused, same reasoning as decdn.env)
  • all four credential keys grepped on-host (GC_PROM_REMOTE_WRITE_URL, GC_OTLP_ENDPOINT, GC_PROM_USERNAME, GC_API_TOKEN) — values never transit the control machine; cloud URLs must be https
  • sampling ratio ∈ [0,1], duration-string shapes, positive integer knobs, low-cardinality label charset, and the fixed 4317 coupling port
• Secrets stay out of config: config.alloy carries ${GC_*} placeholders expanded at load time via --config.expand-env; systemd reads the root-owned env file before dropping privileges.
• Loopback everywhere: OTLP gRPC/HTTP receivers on 127.0.0.1:4317/4318, scrape target 127.0.0.1:9090, and Alloy's own admin server explicitly pinned to loopback (upstream defaults it to all interfaces) — no firewall hole needed or added.
• Pipeline: prometheus.scrape → identity relabel → remote_write; otelcol.receiver.otlp → resource attributes → parent-based probabilistic sampler (default keep-ratio 0.25) → batch → otlphttp exporter with queue + retry/backoff.
• Hardened systemd unit mirroring decdn-node.service.j2's sandboxing block, plus molecule-proven idempotent StateDirectory/task-mode agreement.

Node-side wiring (minimal diff)

When the flag flips on, node.toml.j2 emits otlp_endpoint = "http://127.0.0.1:4317" (unless the operator set one explicitly). No new node.toml keys beyond that emission → no schema-keys churn; the existing otlp_endpoint shape validator gates both paths. The 4317 literal is guarded by a preflight assert + molecule assertions on both sides so the two roles can't drift apart silently.

Playbook & packaging

• site.yml: baseline → grafana_alloy → decdn_node (Alloy verified running before the node starts exporting).
• galaxy/build.sh stages the third role into the decdn.node collection.

Tests

• New grafana-cloud scenario: positive end-to-end converge vs a stub binary (user/dirs, secret gate, rendered pipeline config, hardened unit, services running); verifies loopback-only bindings textually, identity labels/sampling/batch knobs, ${GC_*} expansion (no secret literals outside the 0600 env file), and otlp_endpoint in parsed node.toml.
• default scenario: disabled-path asserts — no alloy user/unit/config/state dirs, no fixture file, node.toml byte-identical to the pre-integration rendering.
• validation scenario: five negative cases (missing secret, symlinked secret, missing key, http:// URL, malformed sampling ratio) proving each bad input is rejected by its specific gate.
• Suite is now seven scenarios; docs + CI comments synced (make molecule JOBS=3 green locally, ansible-lint production profile clean, KICS CRITICAL/HIGH 0).

Docs

ansible/README.md gets a deploy subsection + configuration-table row; group_vars/decdn_nodes.yml carries a commented opt-in block; roles/grafana_alloy/README.md documents setup, token rotation posture, rollback, and cost/cardinality guardrails (low-cardinality labels only — the bill scales with unique label sets).

[Copilot]

🟡 Changes recommended

Unresolved critical and moderate review findings remain and must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds opt-in Grafana Cloud observability for Ansible-deployed nodes through a hardened, loopback-only Grafana Alloy role.

Changes:

• Adds Alloy installation, validation, telemetry pipelines, and systemd hardening.
• Wires the opt-in flag and local OTLP endpoint into decdn_node.
• Adds Molecule coverage, documentation, packaging, and CI updates.

Review findings:

• ansible/galaxy/build.sh — nit, 1 vote: collection README omits the new role.
• ansible/molecule/default/verify.yml — moderate, 2 votes: disabled-path assertions never invoke grafana_alloy.
• ansible/molecule/grafana-cloud/verify.yml — moderate, 1 vote: sampling assertion expects 25 instead of rendered 25.0.
• ansible/playbooks/site.yml — moderate, 2 votes: partial decdn_node deployments can leave Alloy missing or stale.
• ansible/roles/grafana_alloy/defaults/main.yml — moderate, 1 vote: scrape target is not derived from decdn_metrics_port.
• ansible/roles/grafana_alloy/defaults/main.yml — moderate, 1 vote: environment and region default empty despite acceptance requirements.
• ansible/roles/grafana_alloy/defaults/main.yml — critical, 2 votes: overridable addresses can expose telemetry and admin endpoints.
• ansible/roles/grafana_alloy/tasks/install.yml — moderate, 3 votes: check mode executes a binary that may not exist.
• ansible/roles/grafana_alloy/tasks/install.yml — critical, 3 votes: substring version checks can accept the wrong installed version, including line 137.
• ansible/roles/grafana_alloy/tasks/main.yml — critical, 3 votes: release validation uses the wrong binary path.
• ansible/roles/grafana_alloy/tasks/main.yml — moderate, 3 votes: check mode still starts and validates the service.
• ansible/roles/grafana_alloy/tasks/main.yml — moderate, 1 vote: disabled mode retains the Alloy state directory and account.
• ansible/roles/grafana_alloy/tasks/preflight.yml — moderate, 2 votes: scrape interval validation is omitted, including line 43.
• ansible/roles/grafana_alloy/tasks/preflight.yml — critical, 1 vote: listener and admin addresses lack loopback validation, including line 84.
• ansible/roles/grafana_alloy/templates/alloy.service.j2 — moderate, 1 vote: StateDirectory is hard-coded instead of following the configured state path.
• ansible/roles/grafana_alloy/templates/alloy.service.j2 — critical, 1 vote: Alloy storage path is not configured under the writable state directory.
• ansible/roles/grafana_alloy/templates/config.alloy.j2 — moderate, 1 vote: configured sampling is not parent-based.
• ansible/roles/grafana_alloy/templates/config.alloy.j2 — moderate, 1 vote: resource attributes use the attributes processor instead of the resource processor.

File summaries

File	Description
ansible/roles/grafana_alloy/templates/config.alloy.j2	Alloy metrics and OTLP pipelines.
ansible/roles/grafana_alloy/templates/alloy.service.j2	Hardened systemd service definition.
ansible/roles/grafana_alloy/tasks/preflight.yml	Secret, credential, port, and configuration validation.
ansible/roles/grafana_alloy/tasks/main.yml	Role lifecycle and service management.
ansible/roles/grafana_alloy/tasks/install.yml	Release and manual binary installation.
ansible/roles/grafana_alloy/README.md	Setup and operational documentation.
ansible/roles/grafana_alloy/meta/main.yml	Role metadata.
ansible/roles/grafana_alloy/handlers/main.yml	Service reload and restart handlers.
ansible/roles/grafana_alloy/files/grafana-alloy.env.example	Credential environment template.
ansible/roles/grafana_alloy/defaults/main.yml	Role defaults and telemetry settings.
ansible/roles/decdn_node/templates/node.toml.j2	Conditional OTLP endpoint wiring.
ansible/roles/decdn_node/defaults/main.yml	Mirrored feature flag.
ansible/README.md	Deployment and testing documentation.
ansible/playbooks/site.yml	Role ordering and deployment tags.
ansible/molecule/validation/converge.yml	Negative validation scenarios.
ansible/molecule/grafana-cloud/verify.yml	Positive integration assertions.
ansible/molecule/grafana-cloud/prepare.yml	Scenario fixtures.
ansible/molecule/grafana-cloud/molecule.yml	Scenario configuration.
ansible/molecule/grafana-cloud/files/alloy-stub	CI Alloy stub binary.
ansible/molecule/grafana-cloud/converge.yml	Positive-path convergence.
ansible/molecule/default/verify.yml	Disabled-path assertions.
ansible/Makefile	Scenario documentation updates.
ansible/inventory/group_vars/decdn_nodes.yml	Commented opt-in configuration.
ansible/galaxy/galaxy.yml	Collection metadata.
ansible/galaxy/build.sh	Collection role staging.
.github/workflows/molecule.yml	CI scenario-count update.

Review details

Suppressed comments (11)

ansible/galaxy/build.sh:19

• This now stages a third role, but the README copied into the collection still says the artifact contains 'two roles' and lists only baseline and decdn_node. Consumers of the built Galaxy collection will not discover or understand grafana_alloy; update the collection README in the same packaging change.

roles=(baseline decdn_node grafana_alloy)

ansible/molecule/grafana-cloud/verify.yml:63

• The template computes 0.25 | float * 100 as the rendered float 25.0, so this assertion looking for sampling_percentage = 25 does not match the generated line and the positive scenario fails despite a valid value. Assert the float form (or render a deliberately normalized integer/decimal format).

            'sampling_percentage = 25' in ga_config

ansible/roles/grafana_alloy/defaults/main.yml:56

• This is an independent 9090 literal rather than the node's decdn_metrics_port. A supported inventory override such as decdn_metrics_port: 9190 leaves the node serving metrics on 9190 while Alloy still scrapes 127.0.0.1:9090, so enabling the feature silently exports no node metrics. Derive the target from the shared port or fail preflight when the two values differ.

grafana_alloy_scrape_target: 127.0.0.1:9090

ansible/roles/grafana_alloy/defaults/main.yml:68

• With only the documented opt-in flag, both grafana_alloy_deployment_environment and grafana_alloy_region default to empty and the template omits them. The linked issue's acceptance criteria require region and environment in the shipped identity attributes, so the default enabled deployment does not meet that contract unless operators provide additional values; make the required values derivable/mandatory or revise the requirement explicitly.

grafana_alloy_instance_id: ""            # "" => derive from inventory_hostname
grafana_alloy_deployment_environment: ""  # "" => attribute omitted entirely
grafana_alloy_region: ""                  # operator sets it = decdn_region ("" => omitted)

ansible/roles/grafana_alloy/tasks/install.yml:138

• The version backstop repeats the same substring comparison, so a binary reporting v1.19.20 also passes for the pinned 1.19.2. This can mark the wrong binary healthy even after the package gate is corrected; validate the complete reported version rather than substring membership.

    or (grafana_alloy_install_method == "release"
        and grafana_alloy_version not in _ga_installed_version.stdout)

ansible/roles/grafana_alloy/tasks/main.yml:45

• The disabled path removes only the unit and config directory; it leaves the role-created /var/lib/alloy state and alloy account/group on a host that was previously enabled. That does not implement the surrounding 'complete rollback' claim or the new verify comments about no state/user artifacts. Either remove these role-owned resources (with an explicit state-loss warning) or document and test the retained-state rollback contract.

    - name: Remove the leftover unit and rendered configuration
      ansible.builtin.file:
        path: "{{ item }}"
        state: absent
      loop:
        - /etc/systemd/system/alloy.service
        - "{{ grafana_alloy_config_dir }}"
      notify: Reload systemd

ansible/roles/grafana_alloy/tasks/preflight.yml:43

• | int coerces malformed values instead of validating their shape, so values such as 1.5 or 1foo pass this positive check and are then rendered as invalid Alloy configuration. Add a decimal-digit assertion before converting, so the promised preflight rejects bad knobs before creating host state.

      - item.value | int >= 1

ansible/roles/grafana_alloy/tasks/preflight.yml:86

• The coupling check also uses | int, so a value such as 4317x or 4317.0 passes as 4317 and is rendered into the receiver endpoint verbatim. The node still exports to the literal 127.0.0.1:4317, so this fails only later at Alloy config load; validate the port's numeric shape before comparing it.

  ansible.builtin.assert:
    that:
      - grafana_alloy_otlp_grpc_port | int == 4317

ansible/roles/grafana_alloy/templates/alloy.service.j2:43

• grafana_alloy_state_dir is exposed and used for the managed directory/user home, but this unit hard-codes StateDirectory=alloy. Overriding the documented state path leaves systemd's ProtectSystem=strict writable exception at /var/lib/alloy while the configured path is elsewhere, so Alloy state can fail to write or be stored in a different directory. Derive the directive from the variable (with a constrained /var/lib contract) or make the path non-overridable.

# filter. StateDirectory owns {{ grafana_alloy_state_dir }} (the only writable
# path under ProtectSystem=strict): WAL/queue state lives there.
StateDirectory=alloy
StateDirectoryMode=0750

ansible/roles/grafana_alloy/templates/config.alloy.j2:100

• The component configured here is probabilistic trace-ID sampling; naming it parent_based_traces and documenting parent-decision inheritance does not make it parent-based. It will not honor an upstream sampled-parent decision, so the emitted traces can differ from the documented whole-tree parent-based policy. Use a parent-aware sampling design or change the documentation/acceptance behavior to match this processor.

{#- Parent-based sampling: spans without an explicit flag inherit their parent's
    decision, so whole trace subtrees survive or drop together. Renders the
    operator-facing 0..1 keep-ratio as the component's percentage. #}
otelcol.processor.probabilistic_sampler "parent_based_traces" {
  sampling_percentage = {{ grafana_alloy_trace_sampling_ratio | float * 100 }}

ansible/roles/grafana_alloy/templates/config.alloy.j2:85

• The otelcol.processor.attributes component writes span/log/datapoint attributes, not OTLP resource attributes. Routing all telemetry through this block therefore leaves service.name, region, and the other required identity fields off the resource, so Grafana Cloud Application Observability cannot use them for service identity; use the otelcol.processor.resource component for this block.

otelcol.processor.attributes "decdn_identity" {
{% for attr_key in ga_res_attrs.keys() | sort if ga_res_attrs[attr_key] | length > 0 %}
  action {
    key    = {{ attr_key | tojson }}
    action = "upsert"
    value  = {{ ga_res_attrs[attr_key] | tojson }}

• Files reviewed: 26/26 changed files
• Comments generated: 10
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[altanoruc]

Code Review Summary

Verdict: Reviewed 💬 (0 critical, 2 follow-ups, 2 suggestions)

PR: #57 — feat(ansible): opt-in Grafana Cloud observability via new grafana_alloy role
Author: @thiras
Files changed: 38 (+2189 -21)
State: already merged into main (b7ce735). Copilot's earlier findings look addressed in this SHA (loopback preflight, --storage.path, grafana_alloy_bin_effective, check-mode guards, scrape-interval gate, decdn_node tag, default scenario actually invoking the role). CI on the head commit is green (molecule, alloy-config, ansible-lint, kics).

This is follow-up, not a merge blocker.

🔴 Critical

None.

⚠️ Warnings

• ansible/roles/decdn_node/templates/node.toml.j2 — decdn_otlp_endpoint silently wins over the Grafana Cloud injection. Alloy can be installed and healthy while the node keeps shipping somewhere else. Fail-loud if both knobs are set.
• ansible/roles/grafana_alloy/tasks/main.yml / install.yml — unsupported arch is not a preflight gate. User/group/dirs are created first; a failed .deb install then has no unit, so marker-gated teardown will not clean up. Move the amd64/arm64 assert into preflight.yml.

💡 Suggestions

• ansible/galaxy/README.md — table lists grafana_alloy, but the usage playbook still only shows baseline + decdn_node.
• ansible/playbooks/site.yml — --tags observability still runs Alloy without decdn_node, so node.toml can keep/omit otlp_endpoint independently.

✅ Looks Good

• Secrets never leave the host: sys.env() + root 0600 EnvironmentFile, grep-only shape checks, no ${GC_*} placeholders.
• Loopback contract is actually enforced (OTLP binds + admin addr), scrape target follows decdn_metrics_port, systemd hardening + --storage.path under StateDirectory.
• Teardown gated on the managed-by marker — default-off will not nuke a foreign Alloy.
• Test split is honest: stub molecule for plumbing, make lint-alloy against the real pinned binary for config graph / ExecStart flags.

---

Reviewed by Hermes Agent

[altanoruc]

Warning: decdn_otlp_endpoint is checked first, so a non-empty explicit endpoint silently disables this Grafana Cloud injection. Enable the flag with a leftover endpoint and Alloy comes up on 127.0.0.1:4317 while the daemon never talks to it. Fail-loud when both are set.

[altanoruc]

Warning: user/group/dirs are created before install.yml proves this arch is supported. Unknown ansible_architecture becomes 'unsupported' and fails the sha256 pin later, with no unit file, so marker-gated teardown will not clean up. Lift the amd64/arm64 assert into preflight.yml.

[altanoruc]

This is the first arch gate and it is not fail-loud — 'unsupported' becomes a pin miss later. Assert ansible_architecture in ['x86_64', 'aarch64'] in preflight before main.yml creates the alloy account and directories.