grafana_alloy: journald lines that quote the RPC URL (and its API key) ship to Grafana Cloud Loki unredacted #84

Deploy path

Ansible (this repo's playbooks); the same applies to the decdn.node Galaxy collection and to cloud-init, which run the same grafana_alloy role.

Install method

Any (release and manual).

decdn version

Shown with sponsord built from decdn/sponsord 0e85070 (sponsord 0.1.0). decdn-node is not verified yet (see below).

devops revision

main @ 22697c0 (after #82).

Target OS and architecture

Any. Observed on Ubuntu 24.04 x86_64.

What happened, and what did you expect?

The roles go out of their way to keep the RPC URL secret, because it often embeds a provider API key:

• it is never logged and never in a committed or world-readable file;
• it lives only in 0600 host files (/etc/decdn/decdn.env, /etc/sponsord/secret.env);
• the sponsord readiness gate deliberately does not echo the journal back to the control machine.

But with decdn_grafana_cloud_enabled: true, grafana_alloy ships the whole journal to Grafana Cloud Loki (loki.source.journal), and nothing in the pipeline redacts URLs. config.alloy.j2 has no stage.replace and no line filter. Any daemon error line that quotes the RPC URL therefore lands in Loki with the API key in it.

• sponsord, confirmed: a failed chain call logs the full URL, e.g.

  Error: read PaymentPool.usdc(): error sending request for url (http://127.0.0.1:9/)
  

  With a provider endpoint such as https://<provider>/v2/<api-key>, the key is in that line. The leak does not depend on grafana_alloy_sponsord_enabled: that toggle only adds labels and level parsing, and the journal source collects every unit anyway.

• decdn-node: likely, not verified. It uses the same HTTP transport stack, so its RPC errors probably quote the URL the same way. If so, this already affects deployed nodes that have Grafana Cloud logs on.

The exposure is to the operator's own Grafana Cloud stack, not to the public. Still, the key ends up retained in a third-party log store, readable by anyone with Loki access to that stack. That contradicts the roles' secret model.

Expected: a credential embedded in a URL never leaves the host through the observability pipeline.

How to reproduce

# A sponsord host with Grafana Cloud logs on, the RPC endpoint unreachable or
# returning an error (or any decdn-node RPC failure, once verified):
make deploy-sponsord LIMIT=<host>          # with decdn_grafana_cloud_enabled: true
# then, in Grafana Cloud Explore (Loki):
{unit="sponsord.service"} |= "error sending request for url"
# -> the line carries the full RPC URL, path and query included

Proposed fix

Add a redaction step to the journal pipeline, applied to every unit before anything is forwarded:

• keep the scheme and host;
• replace user-info, path and query with a fixed marker, so https://arb-sepolia.example-provider.io/v2/abc123?x=y becomes https://arb-sepolia.example-provider.io/<redacted>.

That is one or two stage.replace stages in loki.process, ahead of the per-daemon level stages. To prove it:

• extend ansible/tests/alloy-config/validate.sh's level harness, which runs the real pinned Alloy, with sample lines carrying URLs, and assert that they come out redacted;
• check that a line with no URL comes out unchanged.

It changes the rendered config.alloy on every Grafana-enabled host, so expect one Alloy restart on the next deploy. Worth noting in the CHANGELOG.

While at it, confirm whether decdn-node's RPC errors include the URL, and record the answer here.