diff --git a/AGENTS.md b/AGENTS.md index 7d1ccb4..3c19e15 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -109,7 +109,10 @@ addresses) the repo carries, and they carry their upstream commit. (Alloy's in-process `node_exporter`, curated collector set, `systemd` collector scoped to the units that matter), **journald** → Grafana Cloud Loki, Alloy's own health, and the daemon's OTLP spans. Host metrics and logs carry `job="integrations/node_exporter"` - so Grafana Cloud's prebuilt Linux Server dashboards work unmodified. The API token is the + so Grafana Cloud's prebuilt Linux Server dashboards work unmodified. Every journald line + passes `loki.process "redact_urls"` ahead of the level stages (URL userinfo, path, query + and fragment → ``, scheme and host kept, no knob), so an RPC URL quoted in a + daemon error ships without its API key; OTLP data is not redacted (#86). The API token is the only credential, and it has two homes: operator-provisioned on the host (`0600 /etc/grafana-alloy.env`) or carried by `grafana_alloy_api_token` from a git-ignored `host_vars//secret.yml`, in which case the role authors that file itself as a diff --git a/ansible/galaxy/CHANGELOG.md b/ansible/galaxy/CHANGELOG.md index 7112cbf..cc80314 100644 --- a/ansible/galaxy/CHANGELOG.md +++ b/ansible/galaxy/CHANGELOG.md @@ -125,6 +125,18 @@ collection adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) ### Fixed +- `grafana_alloy` no longer ships URL credentials in journald lines to Grafana + Cloud Loki (#84). Every line was forwarded verbatim, so a daemon error that + quoted the RPC URL (sponsord's `error sending request for url (…)`) put its + provider API key in Loki. A new `loki.process "redact_urls"` stage now runs on + every unit's line ahead of the level stages. It replaces the userinfo with + `` and everything after the host with `/`, keeping the + scheme and host. That holds for passwords with unencoded `/ ? # @`, and for + URLs inside JSON that escapes them (`https:\/\/…`, `\u0026`, one level of + nesting). A URL with an `@` in its path loses its host. A line with no URL, or + with a URL that has no userinfo and nothing after its host, is unchanged. OTLP + logs and spans are not redacted (#86). The rendered `config.alloy` changes on + every host with Grafana Cloud logs on, so the next deploy restarts Alloy once. - A `release` install no longer fails intermittently at "Remove the download staging directory" with `rmtree failed: [Errno 2] No such file or directory: 'S.gpg-agent.extra'`. Signature verification auto-started a `gpg-agent` diff --git a/ansible/molecule/grafana-cloud/verify.yml b/ansible/molecule/grafana-cloud/verify.yml index 9171e8a..2b563fa 100644 --- a/ansible/molecule/grafana-cloud/verify.yml +++ b/ansible/molecule/grafana-cloud/verify.yml @@ -134,6 +134,11 @@ 'loki.relabel "journal_rules"' in ga_config - >- '__journal__systemd_unit' in ga_config + # URL credentials are redacted before any journal line ships (#84). + - >- + 'loki.process "redact_urls"' in ga_config + - >- + 'forward_to = [loki.process.redact_urls.receiver]' in ga_config # The daemon's level from its JSON body, not journald's priority # (which is info for every stdout line). - >- diff --git a/ansible/roles/grafana_alloy/README.md b/ansible/roles/grafana_alloy/README.md index c44b996..f8d02cc 100644 --- a/ansible/roles/grafana_alloy/README.md +++ b/ansible/roles/grafana_alloy/README.md @@ -206,6 +206,32 @@ Changing `grafana_alloy_host_job` / `grafana_alloy_logs_job` away from `integrations/node_exporter` silently un-wires the prebuilt dashboards; that is why they are knobs with a warning rather than hard-coded literals. +## What is redacted before logs leave the host + +An RPC URL usually carries its provider API key, and a daemon error can quote +it. So every journald line, from every unit, passes `loki.process "redact_urls"` +before anything ships. It has no knob. + +- The userinfo becomes ``, and everything from the first `/`, `?` or + `#` after the host becomes `/`. The scheme and host are kept, so the + provider is still identifiable: + `https://u:p@rpc.example.io/v2/KEY?x=y` ships as + `https://@rpc.example.io/`. +- A password with unencoded `/ ? # @` is covered. The price is that a URL with an + `@` in its path loses its host: `https://h/users/@x/y` ships as + `https://@x/`. +- URLs inside JSON that escapes them are covered: `https:\/\/…`, `\u0026` and one + level of JSON nested in a JSON string. The JSON stays valid. +- A line with no URL, or with a URL that has no userinfo and nothing after its + host, is unchanged. + +Not covered: `\u002f`-escaped separators, fully percent-encoded URLs, deeper JSON +nesting, a URL journald splits across two records (lines over `LineMax`, 48K by +default), and OTLP logs and spans ([#86](https://github.com/decdn/devops/issues/86)). +decdn-node also strips URLs from its own errors; sponsord does not +([decdn/sponsord#38](https://github.com/decdn/sponsord/issues/38)), so for its +lines this stage is the only guard. + ## Machine monitoring: what it costs and how to bound it Grafana Cloud bills on active series and log volume, so the defaults are diff --git a/ansible/roles/grafana_alloy/templates/config.alloy.j2 b/ansible/roles/grafana_alloy/templates/config.alloy.j2 index 42cabe9..fc28c52 100644 --- a/ansible/roles/grafana_alloy/templates/config.alloy.j2 +++ b/ansible/roles/grafana_alloy/templates/config.alloy.j2 @@ -352,7 +352,56 @@ loki.source.journal "host" { max_age = {{ grafana_alloy_logs_max_age | to_json }} format_as_json = false relabel_rules = loki.relabel.journal_rules.rules - forward_to = [loki.process.daemon_level.receiver] + forward_to = [loki.process.redact_urls.receiver] +} + +{# Credentials embedded in URLs do not leave the host in journald lines. An RPC + URL usually carries its provider API key, and a daemon error can quote it: + sponsord's "error sending request for url (...)" does. decdn-node also strips + URLs at the source (decdn_common::redact); sponsord does not, so for sponsord + this stage is the only guard. It runs on EVERY unit's line, ahead of the level + stages, and has no knob: it is the secret model, not a cost guardrail. + + Both stages key on "://" (or JSON's ":\/\/"), not a scheme list, so any + scheme in any case is caught, as in upstream's redact_bare_urls. Unlike that + function, the scheme and host are kept, so the provider stays identifiable: + 1. userinfo -> . It runs to the LAST @ of the whitespace-free + token, so a password holding an unencoded / ? # or @ is covered too. + The cost: a URL with an @ in its path loses its host + (https://h/users/@x/y ships as https://@x/). + 2. everything from the first / ? # (or \/) after the host -> /, + whichever separator it was. + A URL with no userinfo and nothing after its host, and a line with no URL, + pass through unchanged. + + The redacted text ends at whitespace, '"' or '\"' and over-consumes anything + else, such as a closing ')' (legal in a path; losing a bracket beats leaking + a key). JSON escapes are read as whole pairs (\u0026, \/, \\), so Go's + encoding/json, PHP's json_encode and one level of JSON nested in a JSON + string are all covered, and the line stays valid JSON for the level stage + below. stage.replace rewrites every match on the line. + + Not covered: a \u002f-escaped separator, a fully percent-encoded URL, deeper + JSON nesting, a URL journald split across two records (lines over LineMax, + 48K by default) and a Debug-printed url::Url struct. OTLP data does not pass + through here (#86). A line no expression matches ships as it was, so keep + `replace` a literal: a template that fails to render also passes the line + through. validate.sh runs these stages in the real Alloy against sample + lines. #} +loki.process "redact_urls" { + forward_to = [loki.process.daemon_level.receiver] + + // 1. userinfo (as upstream's redact_userinfo, which writes ***@ instead). + stage.replace { + expression = `:\\?/\\?/((?:\\[^\s"]|[^\s"\\])+)@` + replace = "" + } + + // 2. path, query and fragment; the separator is normalised to /. + stage.replace { + expression = `:\\?/\\?/(?:\\[^\s"\\/]|[^/?#\s"\\])+((?:\\?/|[?#])(?:\\[^\s"]|[^\s"\\])*)` + replace = "/" + } } {# The daemon's own severity. decdn-node logs JSON to stdout, and journald gives @@ -360,9 +409,10 @@ loki.source.journal "host" { them level="info" — WARN and ERROR included. The real level is the `level` field of the tracing-subscriber JSON body, e.g. {"level":"WARN",...}. - This runs after loki.source.journal, where the __journal* fields are already - gone, so it keys off the `unit` label journal_rules produced. The unit name - is the one decdn_node installs; it is also hard-coded in the journal_rules + This runs after loki.source.journal (via redact_urls, which rewrites only + the line), where the __journal* fields are already gone, so it keys off the + `unit` label journal_rules produced. The unit name is the one decdn_node + installs; it is also hard-coded in the journal_rules priority exemption above and the service_name rule below — a rename must be mirrored in all three (validate.sh pins them). diff --git a/ansible/tests/alloy-config/validate.sh b/ansible/tests/alloy-config/validate.sh index 508c0d1..af8dee9 100755 --- a/ansible/tests/alloy-config/validate.sh +++ b/ansible/tests/alloy-config/validate.sh @@ -151,8 +151,29 @@ assert_has defaults.alloy 'systemd {' "the per-unit systemd collector" # guardrail anchored with its alternation grouped. Each unit-name needle is one # of the three sites a decdn-node.service rename must touch. assert_has defaults.alloy 'loki.process "daemon_level"' "the daemon JSON-level stage" -assert_has defaults.alloy 'forward_to = [loki.process.daemon_level.receiver]' "the journal source feeding the daemon-level stage" +assert_has defaults.alloy 'forward_to = [loki.process.daemon_level.receiver]' "the URL redactor feeding the daemon-level stage" assert_has defaults.alloy 'forward_to = [loki.relabel.journal_identity.receiver]' "the daemon-level stage feeding the identity relabeller" +# Every configuration that ships the journal redacts it first (#84): the +# redactor exists and the journal source feeds it, not the stage after it. +journal_configs=0 +for config in "${configs[@]}"; do + name="$(basename "$config")" + grep -qF 'loki.source.journal "host"' "$config" || continue + assert_has "$name" 'loki.process "redact_urls"' "the journald URL redactor" + assert_has "$name" 'forward_to = [loki.process.redact_urls.receiver]' "the journal source feeding the URL redactor" + # ...and nothing reaches the stages after it another way: every log source + # feeds the redactor. + sources="$(grep -c '^loki\.source\.' "$config")" + [ "$(grep -c 'loki\.process\.redact_urls\.receiver' "$config")" -eq "$sources" ] \ + || fail "$name: a loki.source.* component does not feed the URL redactor" + # Each later hop has exactly one feeder, so nothing can join the chain past it. + for receiver in loki.process.daemon_level loki.relabel.journal_identity loki.write.cloud; do + [ "$(grep -cF "$receiver.receiver" "$config")" -eq 1 ] \ + || fail "$name: $receiver has a feeder other than its predecessor, which bypasses the URL redactor" + done + journal_configs=$((journal_configs + 1)) +done +[ "$journal_configs" -gt 0 ] || fail "no rendered configuration ships the journal; the redactor check examined nothing" # shellcheck disable=SC2016 # literal backticks: Alloy's raw-string syntax assert_has defaults.alloy 'selector = `{unit="decdn-node.service"}`' "the daemon unit scoping of the level stage" assert_has defaults.alloy 'target_label = "__priority_guardrail"' "the daemon's exemption from the journald priority drop" @@ -254,9 +275,9 @@ assert_has colocated.alloy 'prometheus.scrape "sponsord"' "the sponsord scrape o assert_has colocated.alloy '"__address__" = "127.0.0.1:8090"' "the default sponsord listener" echo "ok: per-daemon scrape toggles" -# --- Gate 1c: the daemon's level comes from its JSON body ---------------------- -# Loading proves syntax, not behaviour. Run the RENDERED journal_rules and -# loki.process "daemon_level" blocks in the real Alloy, fed sample lines through +# --- Gate 1c: URLs are redacted, and the daemon's level comes from its line ---- +# Loading proves syntax, not behaviour. Run the RENDERED journal_rules, +# redact_urls and daemon_level blocks in the real Alloy, fed sample lines through # loki.source.api, and read what loki.echo prints. loki.source.api strips # __-prefixed labels on push, so the lifted rules read journal_* instead of # __journal_* (the only edit), and a final labeldrop removes those raw labels @@ -285,11 +306,25 @@ samples_json() { # nanosecond timestamp ["$1","2026-10-05T07:59:10.000005Z ERROR sponsord: s-error"], ["$1","Error: s-raw read PaymentPool.usdc()"], ["$1","Caused by: s-cause connection refused"], - ["$1","s-plain an unlevelled line"]]}, + ["$1","s-plain an unlevelled line"], + ["$1","Error: s-rpcurl read PaymentPool.usdc(): error sending request for url (https://arb-sepolia.example-provider.io/v2/SECRET-KEY?x=SECRET-Q)"]]}, {"stream":{"journal__systemd_unit":"other.service","journal_priority_keyword":"info"},"values":[ ["$1","{\"level\":\"ERROR\",\"fields\":{\"message\":\"m-other-info\"}}"]]}, {"stream":{"journal__systemd_unit":"other.service","journal_priority_keyword":"error"},"values":[ - ["$1","m-other-error"]]}, + ["$1","m-other-error"], + ["$1","o-twourls wss://SECRET-USER@a.example/SECRET-A and HTTP://b.example?SECRET-B=1 end"], + ["$1","o-nopath https://c.example:443 done"], + ["$1","o-escslash {\"url\":\"https:\\\\/\\\\/u:SECRET-ESCPASS@esc.example\\\\/v2\\\\/SECRET-ESCKEY?x=SECRET-ESCQ\"}"], + ["$1","o-escamp {\"url\":\"https://amp.example/v2/k?a=1\\\\u0026b=SECRET-AMP\"}"], + ["$1","o-escquote {\"m\":\"say \\\\\"https://q.example/v2/SECRET-QK\\\\\" ok\"}"], + ["$1","o-hashpw postgres://SECRET-HU:SECRET-pa#ss@db.example/app y"], + ["$1","o-slashpw https://SECRET-SU:SECRET-ab/cd+ef==@rpc.example/v2/SECRET-SK y"], + ["$1","o-atpw https://u:SECRET-P@SS@at.example/v2/SECRET-AK y"], + ["$1","o-frag https://app.example#access_token=SECRET-TOK y"], + ["$1","o-ipv6 http://u:SECRET-V6PW@[2001:db8::1]:8545/SECRET-V6 y"], + ["$1","o-userhost redis://:SECRET-RP@cache.example:6379 y"], + ["$1","o-atpath https://h.example/users/@alice/x y"], + ["$1","o-noturl ops@example.com at 10:30, ref #42 path a/b?c"]]}, {"stream":{"journal__systemd_unit":"decdn-node.service","journal_priority_keyword":"info"},"values":[ ["$1","{\"level\":\"TRACE\",\"fields\":{\"message\":\"m-trace\"}}"], ["$1","{\"level\":\" DEBUG\",\"fields\":{\"message\":\"m-debug\"}}"], @@ -299,7 +334,10 @@ samples_json() { # nanosecond timestamp ["$1","{\"level\":\"ERROR\",\"fields\":{\"message\":\"m-error\"}}"], ["$1","{\"fields\":{\"message\":\"m-nolevel\"}}"], ["$1","{\"level\":5,\"fields\":{\"message\":\"m-weird\"}}"], - ["$1","m-text is not json"]]} + ["$1","m-text is not json"], + ["$1","{\"level\":\"WARN\",\"fields\":{\"message\":\"m-rpcurl via https://u:SECRET-PASS@rpc.example.io:8545/v3/SECRET-KEY#SECRET-FRAG failed\"}}"], + ["$1","{\"level\":\"WARN\",\"fields\":{\"url\":\"https://user:p\\\\u0026SECRET-EU@rpc.example/v2/SECRET-EK\",\"message\":\"m-escuser\"}}"], + ["$1","{\"level\":\"WARN\",\"fields\":{\"message\":\"m-nested body={\\\\\"url\\\\\":\\\\\"https://n.example/v2/k?a=1\\\\\\\\u0026key=SECRET-NK\\\\\"}\"}}"]]} ]} JSON } @@ -323,7 +361,7 @@ loki.source.api "in" { listen_port = $api_port } relabel_rules = loki.relabel.journal_rules.rules - forward_to = [loki.process.daemon_level.receiver] + forward_to = [loki.process.redact_urls.receiver] } loki.echo "out" { } @@ -332,11 +370,14 @@ ALLOY awk '/^loki\.relabel "journal_rules" \{/ { on = 1 } on && /^\}/ { print "\n rule {\n regex = \"journal_.*\"\n action = \"labeldrop\"\n }"; print; exit } on { print }' "$src" | sed 's/"__journal_/"journal_/g' + awk '/^loki\.process "redact_urls" \{/ { on = 1 } on { print } on && /^\}/ { exit }' "$src" awk '/^loki\.process "daemon_level" \{/ { on = 1 } on { print } on && /^\}/ { exit }' "$src" \ | sed 's|forward_to = \[loki\.relabel\.journal_identity\.receiver\]|forward_to = [loki.echo.out.receiver]|' } >"$config" - if ! grep -qF 'action = "labeldrop"' "$config" || ! grep -qF 'forward_to = [loki.echo.out.receiver]' "$config"; then - fail "could not lift journal_rules and daemon_level out of $1" + if ! grep -qF 'action = "labeldrop"' "$config" \ + || ! grep -qF 'loki.process "redact_urls"' "$config" \ + || ! grep -qF 'forward_to = [loki.echo.out.receiver]' "$config"; then + fail "could not lift journal_rules, redact_urls and daemon_level out of $1" fi "$alloy" run "$config" --storage.path="$data" \ @@ -382,10 +423,22 @@ kept() { # marker, level, unit dropped() { # marker ! grep -qF "$1" "$harness_log" || harness_fail "$1 should have been dropped" } +# URL redaction (#84). Every credential in the samples is spelled SECRET-*, so +# no_secrets is the blanket check; redacted pins text the marker's line must +# contain, and entry_is pins a whole line. +redacted() { # marker, the redacted text its line must carry + grep -F "$1" "$harness_log" | grep -qF "$2" || harness_fail "$1 should carry \"$2\"" +} +entry_is() { # the whole line exactly as it must come out (redacted or untouched) + grep -qF "entry=\"$1\" " "$harness_log" || harness_fail "no entry came out as \"$1\"" +} +no_secrets() { + ! grep -qF 'SECRET' "$harness_log" || harness_fail "a URL credential (SECRET-*) left the pipeline" +} # Defaults (guardrail "debug|trace"), sponsord not enabled: its plain-text lines # are not re-levelled and stand at journald's info. -run_level_harness defaults.alloy 17 +run_level_harness defaults.alloy 34 kept s-trace info sponsord.service kept s-error info sponsord.service kept m-warn-uc warning decdn-node.service @@ -402,11 +455,52 @@ dropped m-debug # padded + lower-cased before the drop dropped m-other-debug # the journald-priority drop, other units echo "ok: daemon log level follows the JSON body (defaults)" +# URL redaction, on every unit, ahead of the level stages. +no_secrets +# scheme and host survive; path and query go, and so does reqwest's closing ')' +redacted s-rpcurl 'error sending request for url (https://arb-sepolia.example-provider.io/"' +kept s-rpcurl info sponsord.service # sponsord not enabled: journald's level +# userinfo, path and fragment go; the JSON stays valid, so the level still parses +redacted m-rpcurl 'via https://@rpc.example.io:8545/ failed' +kept m-rpcurl warning decdn-node.service +# every URL on the line, any scheme in any case, a query with no path +redacted o-twourls 'o-twourls wss://@a.example/ and HTTP://b.example/ end' +# JSON another unit may write (needles are as loki.echo prints them, logfmt- +# escaped): PHP's \/ separators, Go's \u0026 for '&' mid-query, and a URL +# quoted inside a JSON string, whose \" must survive so the JSON stays valid +redacted o-escslash 'https:\\/\\/@esc.example/\"}"' +redacted o-escamp 'https://amp.example/\"}"' +redacted o-escquote 'say \\\"https://q.example/\\\" ok\"}"' +entry_is 'o-nopath https://c.example:443 done' # no userinfo, nothing after the host: untouched +entry_is 'm-text is not json' # no URL: untouched +# userinfo runs to the token's LAST @, so unencoded / ? # or @ in a password +# cannot cut it short; an @ in a path costs the host instead (over-redaction) +entry_is 'o-hashpw postgres://@db.example/ y' +entry_is 'o-slashpw https://@rpc.example/ y' +entry_is 'o-atpw https://@at.example/ y' +entry_is 'o-frag https://app.example/ y' +entry_is 'o-ipv6 http://@[2001:db8::1]:8545/ y' +entry_is 'o-userhost redis://@cache.example:6379 y' +entry_is 'o-atpath https://@alice/ y' +# userinfo read through a JSON escape, and one level of JSON nested in a JSON +# string; both stay valid JSON, so the node's level still parses +entry_is '{\"level\":\"WARN\",\"fields\":{\"url\":\"https://@rpc.example/\",\"message\":\"m-escuser\"}}' +entry_is '{\"level\":\"WARN\",\"fields\":{\"message\":\"m-nested body={\\\"url\\\":\\\"https://n.example/\\\"}\"}}' +kept m-escuser warning decdn-node.service +kept m-nested warning decdn-node.service +# none of the characters a broader expression might trip on changes a line +entry_is 'o-noturl ops@example.com at 10:30, ref #42 path a/b?c' +entry_is '{\"level\":\"INFO\",\"fields\":{\"message\":\"m-info\"}}' +echo "ok: URL credentials are redacted before the level stages (defaults)" + # Guardrail "info|debug|trace": the daemon's every line is journald-info, so it # must be judged by its JSON level — errors and warnings survive, and lines with # no JSON level are never dropped for a priority they did not choose. -run_level_harness priorityinfo.alloy 7 +run_level_harness priorityinfo.alloy 23 +no_secrets dropped s-error # sponsord not enabled: not exempt either +dropped s-rpcurl +kept m-rpcurl warning decdn-node.service # redacted JSON is still judged by its level kept m-warn-uc warning decdn-node.service kept m-warn-lc warning decdn-node.service kept m-error error decdn-node.service @@ -422,7 +516,9 @@ dropped m-other-debug echo "ok: daemon log level follows the JSON body (info in the guardrail)" # sponsord enabled: its level is parsed from the plain-text line. -run_level_harness sponsordonly.alloy 15 +run_level_harness sponsordonly.alloy 32 +no_secrets +kept s-rpcurl error sponsord.service # the "Error: " match still sees the redacted line kept s-info info sponsord.service kept s-warn warning sponsord.service kept s-error error sponsord.service @@ -436,7 +532,10 @@ echo "ok: sponsord log level follows its text line (defaults)" # Co-located, guardrail "info|debug|trace": both daemons exempt from the # journald-priority drop and judged by the level they logged. -run_level_harness colocated.alloy 12 +run_level_harness colocated.alloy 29 +no_secrets +kept s-rpcurl error sponsord.service +kept m-rpcurl warning decdn-node.service kept s-warn warning sponsord.service kept s-error error sponsord.service kept s-raw error sponsord.service