Skip to content

feat(loki.source.kafka): Add rack_id to fetch from the closest replica - #7072

Open
nanwp wants to merge 2 commits into
grafana:mainfrom
nanwp:feat/loki-source-kafka-rack-id
Open

feat(loki.source.kafka): Add rack_id to fetch from the closest replica#7072
nanwp wants to merge 2 commits into
grafana:mainfrom
nanwp:feat/loki-source-kafka-rack-id

Conversation

@nanwp

@nanwp nanwp commented Sep 10, 2026

Copy link
Copy Markdown

Brief description of Pull Request

Expose Sarama's Config.RackID as an optional rack_id argument on loki.source.kafka. When brokers run a rack-aware replica selector, a consumer that sends a rack id fetches from the closest replica instead of the partition leader (KIP-392), which keeps traffic zone-local.

Pull Request Details

The argument is optional and defaults to an empty string, so existing configurations are unaffected. The value flows from Arguments through Convert() into kafkatarget.TargetConfig, and is applied to the Sarama client configuration.

The Sarama client configuration was built inline in NewSyncer, which also opens the broker connection, so it could not be tested. This moves that construction into newSaramaConfig. Tests now cover the rack_id wiring and the consumer group assignor mapping, which previously had none.

otelcol.receiver.kafka already exposes rack_id; this brings loki.source.kafka in line with it, and the documentation wording follows that component's page.

The promtail config converter is unchanged because promtail's Kafka target configuration has no rack setting to convert from.

Issue(s) fixed by this Pull Request

Fixes #6952

Notes to the Reviewer

PR Checklist

  • Documentation added
  • Tests updated
  • Config converters updated
  • This pull request was substantially generated with AI assistance (see the GenAI policy)

Expose Sarama's Config.RackID as an optional rack_id argument on
loki.source.kafka. When brokers run a rack-aware replica selector, a
consumer that sends a rack id fetches from the closest replica instead
of the partition leader (KIP-392), which keeps traffic zone-local.

Extract the Sarama client configuration out of NewSyncer into
newSaramaConfig so the wiring can be tested without connecting to a
broker. This also covers the consumer group assignor mapping, which had
no test before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lvzfvoj6i7axVsHjckrnKU
@nanwp
nanwp requested review from a team and clayton-cornell as code owners September 10, 2026 11:54
@github-actions

github-actions Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Signed commits report

1 of 2 commits between main and feat/loki-source-kafka-rack-id could not be fully verified:

Commit Author Reason Message
fd9df62b Nanda Wijaya Putra unknown_key feat(loki.source.kafka): Add rack_id to fetch from the closest replica

This repository requires all commits to be signed. See GitHub docs on commit signature verification.

@cla-assistant

cla-assistant Bot commented Sep 10, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@cla-assistant

cla-assistant Bot commented Sep 10, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

loki.source.kafka support rack_id

1 participant