fix(ansible): authenticate the OTLP gateway with its own instance ID

[thiras]

The bug

roles/grafana_alloy/templates/config.alloy.j2 authenticated the OTLP exporter with the Prometheus instance ID:

otelcol.auth.basic "cloud" {
  username = sys.env("GC_PROM_USERNAME")   // or the inventory value
  password = sys.env("GC_API_TOKEN")
}

Grafana Cloud's OTLP gateway authenticates with the stack instance ID — its own number, shown on the portal's OTLP Endpoint → Instance ID panel. Found while wiring up a real stack for the internal instance, where the two are 3585338 (Prometheus) and 1830557 (OTLP).

Where they differ, every trace 401s while remote-write and Loki stay healthy: a silent half-outage that reads as "tracing is broken" rather than "auth is wrong", and one the operator had no variable to correct. Pre-existing — the original four-key env file conflated the two the same way — but decdn/devops#61 made it worth fixing properly now that the non-secret half lives in inventory.

The fix

New grafana_alloy_otlp_username (+ a GC_OTLP_USERNAME env key for the host-provisioned half). Inventory wins outright; left empty, the decision is deferred to runtime, because whether the host's env file carries the key is not knowable at render time:

username = coalesce(sys.env("GC_OTLP_USERNAME"), "1234567")            // prom ID from inventory
username = coalesce(sys.env("GC_OTLP_USERNAME"), sys.env("GC_PROM_USERNAME"))  // nothing in inventory

sys.env yields "" for an unset variable and coalesce returns the first non-empty argument, so an existing deployment, or an org whose IDs coincide, resolves exactly what it resolved before. No new required key — preflight still demands only what nothing else supplies.

The variable joins both inventory gates (instance-ID charset, token smuggling), and the docs stop calling the IDs "two".

Validation

• make lint-alloy — the real pinned Alloy 1.19.2 accepts coalesce() and all nine render combinations. A new otlpfallback case covers the upgrade shape (Prometheus ID in inventory, nothing for OTLP); inventory now sets a third distinct ID and asserts no coalesce( survives; legacy asserts the full fallback chain.
• molecule test -s grafana-cloud — passes, and now asserts the chain end-to-end on a converged host.
• molecule test -s validation — passes with a new negative case (ga-otlp-username-shape) for a malformed OTLP instance ID.
• make lint, make lint-ansible (production profile, 0 failures).

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[Copilot]

🟡 Changes recommended

A present but malformed GC_OTLP_USERNAME is not validated and can override the valid Prometheus fallback, causing OTLP authentication failures.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Updates Grafana Alloy OTLP authentication to support a distinct stack instance ID while preserving legacy Prometheus-ID fallback behavior.

Changes:

• Adds inventory and GC_OTLP_USERNAME support.
• Extends validation and documentation.
• Adds render and Molecule coverage for fallback and malformed values.

File summaries

File	Description
ansible/roles/grafana_alloy/templates/config.alloy.j2	Renders OTLP-specific authentication.
ansible/roles/grafana_alloy/defaults/main.yml	Adds the OTLP username setting.
ansible/roles/grafana_alloy/tasks/preflight.yml	Validates the inventory value.
ansible/tests/alloy-config/render.yml	Adds fallback render cases.
ansible/tests/alloy-config/validate.sh	Verifies rendered authentication paths.
ansible/molecule/validation/converge.yml	Tests malformed inventory IDs.
ansible/molecule/grafana-cloud/verify.yml	Verifies converged fallback output.
ansible/roles/grafana_alloy/README.md	Documents configuration and upgrades.
ansible/roles/grafana_alloy/files/grafana-alloy.env.example	Documents GC_OTLP_USERNAME.
ansible/README.md	Updates deployment guidance.
ansible/galaxy/CHANGELOG.md	Records the fix.
AGENTS.md	Updates repository guidance.

Review details

• Files reviewed: 12/12 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.