fix(grafana_alloy): redact URLs in journald lines before they ship to Loki (#84)

AGENTS.md

@@ -109,7 +109,10 @@ addresses) the repo carries, and they carry their upstream commit.
   (Alloy's in-process `node_exporter`, curated collector set, `systemd` collector scoped
   to the units that matter), **journald** → Grafana Cloud Loki, Alloy's own health, and
   the daemon's OTLP spans. Host metrics and logs carry `job="integrations/node_exporter"`
-  so Grafana Cloud's prebuilt Linux Server dashboards work unmodified. The API token is the
+  so Grafana Cloud's prebuilt Linux Server dashboards work unmodified. Every journald line
+  passes `loki.process "redact_urls"` ahead of the level stages (URL userinfo, path, query
+  and fragment → `<redacted>`, scheme and host kept, no knob), so an RPC URL quoted in a
+  daemon error ships without its API key; OTLP data is not redacted (#86). The API token is the
   only credential, and it has two homes: operator-provisioned on the host
   (`0600 /etc/grafana-alloy.env`) or carried by `grafana_alloy_api_token` from a git-ignored
   `host_vars/<node>/secret.yml`, in which case the role authors that file itself as a

ansible/galaxy/CHANGELOG.md

@@ -125,6 +125,18 @@ collection adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html)
 
 ### Fixed
 
+- `grafana_alloy` no longer ships URL credentials in journald lines to Grafana
+  Cloud Loki (#84). Every line was forwarded verbatim, so a daemon error that
+  quoted the RPC URL (sponsord's `error sending request for url (…)`) put its
+  provider API key in Loki. A new `loki.process "redact_urls"` stage now runs on
+  every unit's line ahead of the level stages. It replaces the userinfo with
+  `<redacted>` and everything after the host with `/<redacted>`, keeping the
+  scheme and host. That holds for passwords with unencoded `/ ? # @`, and for
+  URLs inside JSON that escapes them (`https:\/\/…`, `\u0026`, one level of
+  nesting). A URL with an `@` in its path loses its host. A line with no URL, or
+  with a URL that has no userinfo and nothing after its host, is unchanged. OTLP
+  logs and spans are not redacted (#86). The rendered `config.alloy` changes on
+  every host with Grafana Cloud logs on, so the next deploy restarts Alloy once.
 - A `release` install no longer fails intermittently at "Remove the download
   staging directory" with `rmtree failed: [Errno 2] No such file or directory:
   'S.gpg-agent.extra'`. Signature verification auto-started a `gpg-agent`

ansible/molecule/grafana-cloud/verify.yml

@@ -134,6 +134,11 @@
             'loki.relabel "journal_rules"' in ga_config
           - >-
             '__journal__systemd_unit' in ga_config
+          # URL credentials are redacted before any journal line ships (#84).
+          - >-
+            'loki.process "redact_urls"' in ga_config
+          - >-
+            'forward_to     = [loki.process.redact_urls.receiver]' in ga_config
           # The daemon's level from its JSON body, not journald's priority
           # (which is info for every stdout line).
           - >-

ansible/roles/grafana_alloy/README.md

@@ -206,6 +206,32 @@ Changing `grafana_alloy_host_job` / `grafana_alloy_logs_job` away from
 `integrations/node_exporter` silently un-wires the prebuilt dashboards; that is
 why they are knobs with a warning rather than hard-coded literals.
 
+## What is redacted before logs leave the host
+
+An RPC URL usually carries its provider API key, and a daemon error can quote
+it. So every journald line, from every unit, passes `loki.process "redact_urls"`
+before anything ships. It has no knob.
+
+- The userinfo becomes `<redacted>`, and everything from the first `/`, `?` or
+  `#` after the host becomes `/<redacted>`. The scheme and host are kept, so the
+  provider is still identifiable:
+  `https://u:p@rpc.example.io/v2/KEY?x=y` ships as
+  `https://<redacted>@rpc.example.io/<redacted>`.
+- A password with unencoded `/ ? # @` is covered. The price is that a URL with an
+  `@` in its path loses its host: `https://h/users/@x/y` ships as
+  `https://<redacted>@x/<redacted>`.
+- URLs inside JSON that escapes them are covered: `https:\/\/…`, `\u0026` and one
+  level of JSON nested in a JSON string. The JSON stays valid.
+- A line with no URL, or with a URL that has no userinfo and nothing after its
+  host, is unchanged.
+
+Not covered: `\u002f`-escaped separators, fully percent-encoded URLs, deeper JSON
+nesting, a URL journald splits across two records (lines over `LineMax`, 48K by
+default), and OTLP logs and spans ([#86](https://github.com/decdn/devops/issues/86)).
+decdn-node also strips URLs from its own errors; sponsord does not
+([decdn/sponsord#38](https://github.com/decdn/sponsord/issues/38)), so for its
+lines this stage is the only guard.
+
 ## Machine monitoring: what it costs and how to bound it
 
 Grafana Cloud bills on active series and log volume, so the defaults are

ansible/roles/grafana_alloy/templates/config.alloy.j2

@@ -352,17 +352,67 @@ loki.source.journal "host" {
   max_age        = {{ grafana_alloy_logs_max_age | to_json }}
   format_as_json = false
   relabel_rules  = loki.relabel.journal_rules.rules
-  forward_to     = [loki.process.daemon_level.receiver]
+  forward_to     = [loki.process.redact_urls.receiver]
+}
+
+{# Credentials embedded in URLs do not leave the host in journald lines. An RPC
+   URL usually carries its provider API key, and a daemon error can quote it:
+   sponsord's "error sending request for url (...)" does. decdn-node also strips
+   URLs at the source (decdn_common::redact); sponsord does not, so for sponsord
+   this stage is the only guard. It runs on EVERY unit's line, ahead of the level
+   stages, and has no knob: it is the secret model, not a cost guardrail.
+
+   Both stages key on "://" (or JSON's ":\/\/"), not a scheme list, so any
+   scheme in any case is caught, as in upstream's redact_bare_urls. Unlike that
+   function, the scheme and host are kept, so the provider stays identifiable:
+     1. userinfo -> <redacted>. It runs to the LAST @ of the whitespace-free
+        token, so a password holding an unencoded / ? # or @ is covered too.
+        The cost: a URL with an @ in its path loses its host
+        (https://h/users/@x/y ships as https://<redacted>@x/<redacted>).
+     2. everything from the first / ? # (or \/) after the host -> /<redacted>,
+        whichever separator it was.
+   A URL with no userinfo and nothing after its host, and a line with no URL,
+   pass through unchanged.
+
+   The redacted text ends at whitespace, '"' or '\"' and over-consumes anything
+   else, such as a closing ')' (legal in a path; losing a bracket beats leaking
+   a key). JSON escapes are read as whole pairs (\u0026, \/, \\), so Go's
+   encoding/json, PHP's json_encode and one level of JSON nested in a JSON
+   string are all covered, and the line stays valid JSON for the level stage
+   below. stage.replace rewrites every match on the line.
+
+   Not covered: a \u002f-escaped separator, a fully percent-encoded URL, deeper
+   JSON nesting, a URL journald split across two records (lines over LineMax,
+   48K by default) and a Debug-printed url::Url struct. OTLP data does not pass
+   through here (#86). A line no expression matches ships as it was, so keep
+   `replace` a literal: a template that fails to render also passes the line
+   through. validate.sh runs these stages in the real Alloy against sample
+   lines. #}
+loki.process "redact_urls" {
+  forward_to = [loki.process.daemon_level.receiver]
+
+  // 1. userinfo (as upstream's redact_userinfo, which writes ***@ instead).
+  stage.replace {
+    expression = `:\\?/\\?/((?:\\[^\s"]|[^\s"\\])+)@`
+    replace    = "<redacted>"
+  }
+
+  // 2. path, query and fragment; the separator is normalised to /.
+  stage.replace {
+    expression = `:\\?/\\?/(?:\\[^\s"\\/]|[^/?#\s"\\])+((?:\\?/|[?#])(?:\\[^\s"]|[^\s"\\])*)`
+    replace    = "/<redacted>"
+  }
 }
 
 {# The daemon's own severity. decdn-node logs JSON to stdout, and journald gives
    every stdout line priority 6, so the journal_rules mapping above labels all of
    them level="info" — WARN and ERROR included. The real level is the `level`
    field of the tracing-subscriber JSON body, e.g. {"level":"WARN",...}.
 
-   This runs after loki.source.journal, where the __journal* fields are already
-   gone, so it keys off the `unit` label journal_rules produced. The unit name
-   is the one decdn_node installs; it is also hard-coded in the journal_rules
+   This runs after loki.source.journal (via redact_urls, which rewrites only
+   the line), where the __journal* fields are already gone, so it keys off the
+   `unit` label journal_rules produced. The unit name is the one decdn_node
+   installs; it is also hard-coded in the journal_rules
    priority exemption above and the service_name rule below — a rename must be
    mirrored in all three (validate.sh pins them).