fix(ansible): take decdn-node's Loki level from its JSON log field - #65
Merged
Merged
Conversation
journald gives every stdout line priority 6, so the journal_rules mapping labelled all daemon lines level="info" — WARN and ERROR included — and level-filtered queries (error counts) returned zero. The debug guardrail never dropped the daemon's DEBUG lines for the same reason. Add loki.process "daemon_level", scoped to unit="decdn-node.service": parse the tracing JSON `level`, normalise it to journald's keywords (WARN -> warning), set the label, and apply the anchored priority-drop regex. Non-JSON lines and other units keep their journald level. make lint-alloy now runs the rendered stage in the real pinned Alloy and asserts the resulting labels, and fails against the old behaviour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Exempt decdn-node from the journald-priority drop in journal_rules and
let daemon_level apply the same regex to its JSON level: every daemon
line is journald-info, so a guardrail containing `info` dropped its
warnings and errors before the JSON was ever read.
- Default the priority guardrail to 'debug|trace'. TRACE normalises to
`trace`, not a journald keyword, so it bypassed the old 'debug' default.
- Accept only the five tracing levels (trimmed, case-folded); anything else
keeps the journald level instead of becoming a label value.
- `{% endraw +%}`: trim_blocks glued the stage's closing brace onto the
template line in the rendered config.
- Fix comments: upstream's formats are pretty/json (no "text"), the
three places the unit name is hard-coded, and the empty-template path.
validate.sh: assert the stage is wired in both directions, run the rendered
journal_rules + daemon_level through the real Alloy for the defaults and a new
info-dropping render variant, cover TRACE/lowercase/no-level/non-string cases,
fail explicitly on readiness/entry-count timeouts and on an Alloy crash, and
dump the Alloy log before cleanup deletes it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The rendered Alloy regex uses unsupported non-capturing group syntax and will fail validation.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Open (1)
What changed in this PR
Updates Grafana Alloy to derive decdn-node log severity from JSON fields, with revised filtering, documentation, and validation coverage.
Changes:
- Adds daemon JSON severity parsing and normalization.
- Updates priority filtering defaults and operator documentation.
- Adds render, pipeline, and Molecule validation.
| File | Summary |
|---|---|
ansible/tests/alloy-config/validate.sh |
Adds rendering and behavior validation. |
ansible/tests/alloy-config/render.yml |
Adds an info-dropping render variant. |
ansible/roles/grafana_alloy/templates/config.alloy.j2 |
Adds daemon severity processing and filtering. |
ansible/roles/grafana_alloy/README.md |
Documents new severity behavior. |
ansible/roles/grafana_alloy/defaults/main.yml |
Updates the default priority-drop regex. |
ansible/molecule/grafana-cloud/verify.yml |
Verifies the new processing stage. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Loki labelled every decdn-node log line
level="info", WARN, ERROR and DEBUG included, so level-filtered queries (e.g. an error count) returned zero.Root cause:
grafana_alloysetlevelonly from journald's priority. The daemon logs JSON to stdout, and journald gives every stdout line priority 6 (info). The real severity is thelevelfield of the tracing-subscriber JSON body.Fix: a new
loki.process "daemon_level"step between the journal source and the identity relabel, scoped tounit="decdn-node.service":leveland accepts only the five tracing levels (trimmed, case-folded).WARN→warning;TRACE→trace) and sets thelevellabel.^(?:…)$.journal_rulesexempts the daemon from the journald-priority drop, so a setting that includesinfono longer deletes daemon warnings and errors.Behaviour changes for operators
grafana_alloy_logs_drop_priority_regexnow defaults to'debug|trace', so the daemon's TRACE lines are dropped (they would otherwise bypass the olddebugdefault).level="warning", notwarn.info; the fix applies from deploy onward.Testing
make lint-alloynow runs the renderedjournal_rules+daemon_levelin the real pinned Alloy (1.19.2), fed sample lines, and checks the exact label set. It covers the default settings and a newinfo|debug|tracerender variant.warn, ERROR, INFO, TRACE, a paddedDEBUG, JSON with no level, a non-string level, a non-JSON line, and other units.debugdefault, a bare{% endraw %}, and an unwired stage each fail the gate.make lint-ansible,molecule test -s grafana-cloudand pre-commit all pass.🤖 Generated with Claude Code