feat(observability): integrate Ansible nodes with Grafana Cloud Application Observability

[altanoruc]

Objective

Provide an opt-in Grafana Cloud Application Observability integration for Ansible-deployed deCDN nodes, without exposing telemetry endpoints publicly or committing Grafana credentials.

Repository findings

• The Ansible role already renders [observability] with metrics_bind, metrics_port, and an optional otlp_endpoint; its node metrics (9090) and admin RPC remain loopback-only.
• The node service supports a separate 0600 environment file for secrets, but Grafana credentials must not be added to the node process environment unnecessarily.
• The Helm chart already offers a private metrics ClusterIP service and optional ServiceMonitor; this issue targets the VM/Ansible path first.

Proposed implementation

• Add an opt-in decdn_grafana_cloud_enabled: false integration to the Ansible role or a dedicated observability role.
• Install and manage Grafana Alloy as a hardened systemd service.
• Configure Alloy to:
  • scrape http://127.0.0.1:<decdn_metrics_port>/metrics and remote-write Prometheus metrics to Grafana Cloud;
  • receive the node's OTLP export on loopback and forward it to Grafana Cloud with the required authenticated OTLP exporter;
  • attach stable resource/metric labels, at minimum service.name=decdn-node, node identity, region, and environment;
  • avoid high-cardinality identifiers and define trace sampling before enabling fleet-wide export.
• Set decdn_otlp_endpoint to Alloy's loopback receiver only when the integration is enabled.
• Store Grafana Cloud endpoint, instance/tenant ID, and API token in a dedicated host-provisioned 0600 secret file. Do not commit them, render them into node.toml, or place them in the deCDN node environment.
• Keep all listeners loopback-only and make no nftables changes beyond existing node exposure.
• Document Grafana Cloud stack setup, minimum token scopes, secret provisioning, enablement, rollback, and cost/cardinality controls.

Acceptance criteria

• Default deployment remains unchanged and installs no Alloy/Grafana components.
• With the feature enabled and secrets provisioned, ansible-playbook --check and a normal deploy complete successfully.
• Node Prometheus metrics arrive in Grafana Cloud with the required labels.
• Node OTLP data reaches Grafana Cloud through Alloy; the node itself has no Grafana Cloud credentials.
• No Grafana, OTLP, or metrics port is publicly reachable; metrics scrape and OTLP receiver bind to loopback.
• The deployment fails clearly when the integration is enabled but required secret values are absent or malformed.
• Molecule/template coverage verifies default-off behavior, generated Alloy config, secret-file permissions, loopback binds, and decdn_otlp_endpoint wiring.
• Decide in a follow-up whether the Helm chart needs a separate Alloy/OTel Collector integration; retain its existing ServiceMonitor behavior unchanged in this issue.

References

• https://grafana.com/products/cloud/application-observability/
• https://grafana.com/docs/grafana-cloud/observe-and-act/monitor-applications/application-observability-kg/

[altanoruc]

Implementation plan

1. Lock the integration contract before adding files

• Keep scope to the Ansible VM deployment. The Helm chart already has a private metrics Service + optional ServiceMonitor; do not change it in this issue.
• Add an explicit, default-off switch (decdn_grafana_cloud_enabled: false). Existing runs must neither install Alloy nor alter node.toml.
• Use a dedicated grafana_alloy role, called by site.yml before decdn_node. This avoids turning the node role into a Grafana credential owner and guarantees the local OTLP receiver is started before the node is configured to export.
• Define a small public variable surface: Alloy package/version/source, local OTLP gRPC listener (127.0.0.1:4317), local metrics target (127.0.0.1:{{ decdn_metrics_port }}), deployment environment, and a host-provisioned secret-file path. Keep tenant IDs, API tokens, and cloud endpoints out of normal inventory.

2. Establish Grafana Cloud prerequisites and secret format

• Document the exact Grafana Cloud endpoints and minimum-scoped tokens for Prometheus remote-write and OTLP export; keep them configurable because endpoint/stack values are tenant-specific.
• Define a dedicated root-owned, 0600 secret file (for example /etc/decdn/grafana-alloy.env) holding only the Alloy-facing credentials/endpoints. Do not reuse decdn.env: that file is intentionally readable by the decdn service and is checksum/provenance-managed for node secrets.
• Add fail-loud preflight checks when enabled: regular file only (no symlink), mode/owner, required non-empty keys, valid HTTPS cloud URLs, and no credentials in non-secret variables or rendered node.toml.

3. Add the Alloy role with a hardened systemd service

• Install a pinned Grafana Alloy release using an authenticated package/release path with checksum or signature verification; do not use an unpinned latest download.
• Create a dedicated unprivileged Alloy user, config directory, state/WAL directory, secret-file permissions, and a systemd unit with the least network/filesystem access compatible with DNS, loopback receivers, and outbound Grafana Cloud traffic.
• Validate generated configuration with alloy fmt/alloy run validation before restarting the service. Add handlers so config, binary, or secret-file changes restart Alloy first.

4. Generate the telemetry pipeline

• Prometheus path: Alloy scrapes http://127.0.0.1:{{ decdn_metrics_port }}/metrics and remote-writes to the configured Grafana Cloud Prometheus endpoint using the secret-file credentials.
• OTLP path: Alloy exposes OTLP/gRPC only on 127.0.0.1:4317, receives deCDN spans, applies resource attributes, batch/queue/retry settings, and exports authenticated OTLP to Grafana Cloud.
• Apply only low-cardinality attributes: service.name=decdn-node, service.namespace=decdn, service.instance.id (stable node ID/name), cloud.region/node region, and deployment.environment. Do not label metrics with request IDs, content hashes, peer IDs, client addresses, or URLs.
• Start with a conservative parent-based trace sampling policy, make the ratio configurable, and document its cost/coverage impact. Do not add log export in this issue unless the node’s JSON logs are explicitly needed and reviewed separately.

5. Wire the node without weakening its network posture

• When enabled, render observability.otlp_endpoint = "http://127.0.0.1:4317"; when disabled, preserve today’s decdn_otlp_endpoint behavior and omit it by default.
• Keep decdn_metrics_bind: 127.0.0.1, metrics 9090, admin RPC 9191, and nftables exposure unchanged. The only new listener is Alloy’s loopback OTLP receiver.
• Deploy/verify Alloy before restarting decdn-node; retain retries/queueing so a brief remote Grafana outage cannot block node startup or traffic handling.

6. Tests and validation gates

• Extend the default Molecule scenario with the integration disabled and assert: no Alloy package/user/unit/config, no OTLP endpoint injected, and current node behavior is unchanged.
• Add a dedicated Molecule scenario with fixture credentials and a local mock HTTP/gRPC sink. Assert generated Alloy config parses, secrets are 0600 and not present in node.toml/unit logs, Alloy is enabled/running, the OTLP receiver is loopback-only, and metrics remain loopback-only.
• Add negative cases for enabled-with-missing-secret-file, symlink/non-regular secret file, missing required keys, malformed cloud URL, and invalid sampling ratio. Each must fail before changing services.
• Run make lint, make check, Molecule scenarios, and security scanning. Add a deployment smoke-test checklist that confirms one metrics series and one trace in Grafana Cloud, rather than making CI depend on a paid external tenant.

7. Docs and rollout

• Update the Ansible README, role README, inventory examples, and secret-file example with setup, enablement, rollback, token rotation, and troubleshooting commands.
• Include a cost guardrail section: required labels, sampling default, cardinality anti-patterns, queue/drop telemetry metrics, and an initial per-node telemetry budget.
• Roll out to one non-critical node first; verify service catalog discovery, RED metrics, trace-to-metric correlation, no public listeners, and node CPU/memory/network overhead. Then enable the fleet in batches.

Definition of done

• The default-off path is byte-for-byte operationally equivalent for existing users.
• Enabled nodes deliver authenticated Prometheus metrics and OTLP traces through local Alloy, with no Grafana credential available to decdn-node.
• All local telemetry listeners remain loopback-only, validation/tests pass, and the documented smoke test succeeds against a real Grafana Cloud stack.

References: Grafana Cloud Application Observability requires OpenTelemetry data and uses it to connect metrics, logs, traces, and profiles; the existing deCDN role already supports loopback metrics and an OTLP/gRPC endpoint.

[thiras]

Implemented by #57 (merged 2026-09-16): the new opt-in grafana_alloy role (decdn_grafana_cloud_enabled: false by default). It runs a loopback-only Alloy agent that scrapes the node's /metrics, receives OTLP on 127.0.0.1:4317/4318 and forwards it to Grafana Cloud, and wires decdn_node's otlp_endpoint when enabled. Credentials live in 0600 /etc/grafana-alloy.env, never in the node's environment. #63 followed up with the optional inventory-token path. The chart-side Alloy/OTel integration was out of scope here and can be a new issue if it's wanted.