fix(loki.source.kubernetes): Forward log lines with out-of-order timestamps - #7080
Open
johnpinto-clarity wants to merge 1 commit into
Open
fix(loki.source.kubernetes): Forward log lines with out-of-order timestamps#7080johnpinto-clarity wants to merge 1 commit into
johnpinto-clarity wants to merge 1 commit into
Conversation
…stamps Signed-off-by: John Pinto <john.pinto@clarity.ai>
Contributor
Signed commits report1 of 1 commit between
This repository requires all commits to be signed. See GitHub docs on commit signature verification. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The tailer compared every line with the previous one and skipped the older ones. This dropped stdout lines that containerd wrote after a
stderrline with a newer timestamp. Now it only compares with the resume point of the stream, and the saved position moves with the highest timestamp seen. Two tests added: an out-of-order line on a live stream is forwarded, a line older than the resume time is still skipped.Pull Request Details:
lastReadTimeis there to not resend lines after a reconnect (SinceTimehas second precision). Moving it on every line made it a filter on the live stream, and Kubernetes does not guarantee order betweenstdoutandstderrin the CRI log file.Same behaviour kept for the resume: lines before the resume point are skipped, equal ones go through (like #4613). Positions and
target.LastEntryuse max() so a re-tail never resumes from an older timestamp.One thing not fixed here: a line out of order exactly at a re-tail boundary can still be lost, a timestamp position cannot tell "already sent" from "written later with an older timestamp". A small slack on the resume point would fix it at the cost of some duplicates per re-tail. Open to do it if you prefer.
Tested in production: 0 of 1435 canary entries lost in 2 hours with this patch, 25 of 717 lost in the same window on a control cluster with the upstream image.
Issue(s) fixed:
Fixes #7076
Notes to the Reviewer
PR Checklist