fix(ansible): close review findings on the daemon log-level stage

- Exempt decdn-node from the journald-priority drop in journal_rules and
  let daemon_level apply the same regex to its JSON level: every daemon
  line is journald-info, so a guardrail containing `info` dropped its
  warnings and errors before the JSON was ever read.
- Default the priority guardrail to 'debug|trace'. TRACE normalises to
  `trace`, not a journald keyword, so it bypassed the old 'debug' default.
- Accept only the five tracing levels (trimmed, case-folded); anything else
  keeps the journald level instead of becoming a label value.
- `{% endraw +%}`: trim_blocks glued the stage's closing brace onto the
  template line in the rendered config.
- Fix comments: upstream's formats are pretty/json (no "text"), the
  three places the unit name is hard-coded, and the empty-template path.

validate.sh: assert the stage is wired in both directions, run the rendered
journal_rules + daemon_level through the real Alloy for the defaults and a new
info-dropping render variant, cover TRACE/lowercase/no-level/non-string cases,
fail explicitly on readiness/entry-count timeouts and on an Alloy crash, and
dump the Alloy log before cleanup deletes it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: thiras

ansible/roles/grafana_alloy/README.md

@@ -226,14 +226,15 @@ deliberately lean and every lever is a variable:
 - Host metrics scrape at `60s`, not the node's `30s`.
 - The `filesystem` / `netdev` / `disk` exclude regexes drop virtual filesystems
   and container/virtual interfaces.
-- journald drops `debug` priority and a noisy-unit list
+- journald drops `debug` (and the daemon's `trace`) and a noisy-unit list
   (`grafana_alloy_logs_drop_priority_regex` / `_drop_unit_regex`, `""` disables
   either), and `grafana_alloy_logs_max_age` bounds the catch-up burst after an
   outage — without it a restart can replay days of journal in one go.
 - decdn-node's `level` label comes from its JSON `level` field, not the journald
-  priority (journald reports every stdout line as `info`). It uses journald's
-  keywords, so query `level="warning"`, not `warn`. Lines that are not JSON keep
-  the journald level.
+  priority (journald reports every stdout line as `info`), and the priority drop
+  regex is applied to that JSON level for this unit. Values follow journald's
+  keywords, so query `level="warning"`, not `warn`; TRACE becomes `trace`. Lines
+  that are not JSON keep the journald level and are never priority-dropped.
 
 ## Hardening: the two relaxations machine monitoring requires
 

ansible/roles/grafana_alloy/defaults/main.yml

@@ -225,10 +225,13 @@ grafana_alloy_logs_journal_path: ""
 # Cost guardrails, applied while the __journal* fields still exist. Both are
 # anchored regexes matched against the whole value; "" disables that rule.
 # Priority keywords are emerg/alert/crit/error/warning/notice/info/debug
-# (journald's numeric priority 3 surfaces as "error", NOT "err"). The priority
-# regex also drops decdn-node lines by their JSON `level`, normalised to these
-# same keywords (WARN -> warning, TRACE -> trace).
-grafana_alloy_logs_drop_priority_regex: 'debug'
+# (journald's numeric priority 3 surfaces as "error", NOT "err").
+# decdn-node is judged by its JSON `level` instead of its journald priority
+# (always info), anchored the same way (^(?:...)$) and normalised to these
+# keywords: WARN -> warning, DEBUG/INFO/ERROR unchanged, TRACE -> trace. `trace`
+# is not a journald keyword, which is why the default lists it: without it the
+# daemon's most verbose level would bypass the guardrail.
+grafana_alloy_logs_drop_priority_regex: 'debug|trace'
 grafana_alloy_logs_drop_unit_regex:
   '(session-\d+\.scope|user@\d+\.service|systemd-timesyncd\.service|cron\.service|systemd-logind\.service)'
 

ansible/roles/grafana_alloy/templates/config.alloy.j2

@@ -268,8 +268,27 @@ loki.relabel "journal_rules" {
 
   // Cost guardrail: the regex is anchored to the WHOLE value by Prometheus
   // relabel semantics, so "debug" drops debug and nothing else.
+  //
+  // decdn-node is exempt: journald reports every line it writes as info, so
+  // judging it by priority would be judging it by a level it never set (a
+  // regex containing "info" would drop its errors). loki.process
+  // "daemon_level" below applies the same regex to its real, JSON level. The
+  // exemption blanks a scratch copy of the keyword for that one unit; like
+  // every __-prefixed label it never leaves this ruleset.
   rule {
     source_labels = ["__journal_priority_keyword"]
+    target_label  = "__priority_guardrail"
+  }
+
+  rule {
+    source_labels = ["__journal__systemd_unit"]
+    regex         = "decdn-node\\.service"
+    target_label  = "__priority_guardrail"
+    replacement   = ""
+  }
+
+  rule {
+    source_labels = ["__priority_guardrail"]
     regex         = {{ grafana_alloy_logs_drop_priority_regex | tojson }}
     action        = "drop"
   }
@@ -305,12 +324,19 @@ loki.source.journal "host" {
    field of the tracing-subscriber JSON body, e.g. {"level":"WARN",...}.
 
    This runs after loki.source.journal, where the __journal* fields are already
-   gone, so it keys off the `unit` label journal_rules produced (the same unit
-   name the service_name rule below pins — a rename must be mirrored in both).
-   The value is normalised to journald's own keywords (WARN -> warning), so one
-   `level` vocabulary spans every unit. A line that is not JSON (log_format =
-   "text", or a panic written raw to stderr) extracts nothing; the template then
-   renders empty, which removes the key, and the journald level stands. #}
+   gone, so it keys off the `unit` label journal_rules produced. The unit name
+   is the one decdn_node installs; it is also hard-coded in the journal_rules
+   priority exemption above and the service_name rule below — a rename must be
+   mirrored in all three (validate.sh pins them).
+
+   Only the five tracing levels are accepted, trimmed and case-folded, and WARN
+   becomes journald's `warning`; DEBUG/INFO/ERROR already match journald's
+   keywords. TRACE stays `trace`, which journald has no keyword for, so the
+   priority guardrail default lists it explicitly. Anything else — a line that
+   is not JSON (log_format = "pretty", or a panic written raw to stderr), JSON
+   with no level, or an unexpected value — leaves decdn_level unset (the
+   template stage deletes a key it renders empty), so stage.labels and
+   stage.drop skip the line and its journald level stands. #}
 loki.process "daemon_level" {
   forward_to = [loki.relabel.journal_identity.receiver]
 
@@ -323,16 +349,16 @@ loki.process "daemon_level" {
 
     stage.template {
       source   = "decdn_level"
-      template = {% raw %}`{{ with .Value }}{{ if eq (ToUpper .) "WARN" }}warning{{ else }}{{ ToLower . }}{{ end }}{{ end }}`{% endraw %}
+      template = {% raw %}`{{ $l := ToLower (TrimSpace (print .Value)) }}{{ if eq $l "warn" "warning" }}warning{{ else if eq $l "trace" "debug" "info" "error" }}{{ $l }}{{ end }}`{% endraw +%}
     }
 
     stage.labels {
       values = { level = "decdn_level" }
     }
 {% if grafana_alloy_logs_drop_priority_regex | length > 0 %}
 
-    // The priority guardrail again, for the daemon's DEBUG lines: journald
-    // reported them as info, so the journal_rules drop never saw them. Unlike
+    // The priority guardrail for the daemon, which journal_rules exempts: the
+    // same regex, applied to the level the daemon actually logged. Unlike
     // relabel regexes, stage.drop is NOT anchored, hence the explicit ^...$.
     stage.drop {
       source     = "decdn_level"
@@ -372,7 +398,9 @@ loki.relabel "journal_identity" {
    so this catches only lines the daemon process writes: systemd's own
    start/stop/crash messages about it come from PID 1 as unit="init.scope". The
    name is the unit decdn_node installs, /etc/systemd/system/decdn-node.service
-   — a rename there must be mirrored here (validate.sh pins it). #}
+   — a rename there must be mirrored here AND in the journal_rules priority
+   exemption and the loki.process "daemon_level" selector above (validate.sh
+   pins all three), plus grafana_alloy_host_systemd_unit_include. #}
 
   rule {
     source_labels = ["unit"]

ansible/tests/alloy-config/render.yml

@@ -257,6 +257,24 @@
     # Hands validate.sh the release pin without making it parse YAML itself: the
     # values come from the role's own defaults, so the download it verifies is
     # byte-identical to the one the role installs.
+    # A priority guardrail that includes `info`: journald reports every
+    # decdn-node line as info, so this proves the daemon is judged by its JSON
+    # level instead (its warnings and errors must survive). validate.sh runs
+    # this file through the real pipeline.
+    - name: Render the info-dropping priority-guardrail configuration
+      ansible.builtin.template:
+        src: "{{ playbook_dir }}/../../roles/grafana_alloy/templates/{{ item.src }}"
+        dest: "{{ _ga_render_dir }}/priorityinfo.{{ item.ext }}"
+        mode: "0644"
+      loop: *ga_render_pair
+      loop_control:
+        label: "priorityinfo.{{ item.ext }}"
+      vars:
+        _ga_instance_id: decdn-node-1
+        _ga_journal_groups: [systemd-journal, adm]
+        grafana_alloy_bin_effective: /usr/bin/alloy
+        grafana_alloy_logs_drop_priority_regex: 'info|debug|trace'
+
     - name: Export the pinned release for the validation script
       ansible.builtin.copy:
         dest: "{{ _ga_render_dir }}/pin.env"