feat(loki.source.kafka): Add rack_id to fetch from the closest replica - #7072
Open
nanwp wants to merge 2 commits into
Open
feat(loki.source.kafka): Add rack_id to fetch from the closest replica#7072nanwp wants to merge 2 commits into
nanwp wants to merge 2 commits into
Conversation
Expose Sarama's Config.RackID as an optional rack_id argument on loki.source.kafka. When brokers run a rack-aware replica selector, a consumer that sends a rack id fetches from the closest replica instead of the partition leader (KIP-392), which keeps traffic zone-local. Extract the Sarama client configuration out of NewSyncer into newSaramaConfig so the wiring can be tested without connecting to a broker. This also covers the consumer group assignor mapping, which had no test before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lvzfvoj6i7axVsHjckrnKU
Contributor
Signed commits report1 of 2 commits between
This repository requires all commits to be signed. See GitHub docs on commit signature verification. |
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brief description of Pull Request
Expose Sarama's
Config.RackIDas an optionalrack_idargument onloki.source.kafka. When brokers run a rack-aware replica selector, a consumer that sends a rack id fetches from the closest replica instead of the partition leader (KIP-392), which keeps traffic zone-local.Pull Request Details
The argument is optional and defaults to an empty string, so existing configurations are unaffected. The value flows from
ArgumentsthroughConvert()intokafkatarget.TargetConfig, and is applied to the Sarama client configuration.The Sarama client configuration was built inline in
NewSyncer, which also opens the broker connection, so it could not be tested. This moves that construction intonewSaramaConfig. Tests now cover therack_idwiring and the consumer group assignor mapping, which previously had none.otelcol.receiver.kafkaalready exposesrack_id; this bringsloki.source.kafkain line with it, and the documentation wording follows that component's page.The promtail config converter is unchanged because promtail's Kafka target configuration has no rack setting to convert from.
Issue(s) fixed by this Pull Request
Fixes #6952
Notes to the Reviewer
PR Checklist