Skip to content

fix(loki.source.kubernetes): Forward log lines with out-of-order timestamps - #7080

Open
johnpinto-clarity wants to merge 1 commit into
grafana:mainfrom
johnpinto-clarity:fix/kubetail-out-of-order-lines
Open

fix(loki.source.kubernetes): Forward log lines with out-of-order timestamps#7080
johnpinto-clarity wants to merge 1 commit into
grafana:mainfrom
johnpinto-clarity:fix/kubetail-out-of-order-lines

Conversation

@johnpinto-clarity

@johnpinto-clarity johnpinto-clarity commented Sep 10, 2026

Copy link
Copy Markdown

The tailer compared every line with the previous one and skipped the older ones. This dropped stdout lines that containerd wrote after a stderr line with a newer timestamp. Now it only compares with the resume point of the stream, and the saved position moves with the highest timestamp seen. Two tests added: an out-of-order line on a live stream is forwarded, a line older than the resume time is still skipped.

Pull Request Details:
lastReadTime is there to not resend lines after a reconnect (SinceTime has second precision). Moving it on every line made it a filter on the live stream, and Kubernetes does not guarantee order between stdout and stderr in the CRI log file.

Same behaviour kept for the resume: lines before the resume point are skipped, equal ones go through (like #4613). Positions and target.LastEntry use max() so a re-tail never resumes from an older timestamp.

One thing not fixed here: a line out of order exactly at a re-tail boundary can still be lost, a timestamp position cannot tell "already sent" from "written later with an older timestamp". A small slack on the resume point would fix it at the cost of some duplicates per re-tail. Open to do it if you prefer.

Tested in production: 0 of 1435 canary entries lost in 2 hours with this patch, 25 of 717 lost in the same window on a control cluster with the upstream image.

Issue(s) fixed:
Fixes #7076

Notes to the Reviewer

PR Checklist

  • Documentation added
  • Tests updated
  • Config converters updated
  • This pull request was substantially generated with AI assistance

…stamps

Signed-off-by: John Pinto <john.pinto@clarity.ai>
@johnpinto-clarity
johnpinto-clarity requested a review from a team as a code owner September 10, 2026 15:23
@github-actions

Copy link
Copy Markdown
Contributor

Signed commits report

1 of 1 commit between main and fix/kubetail-out-of-order-lines could not be fully verified:

Commit Author Reason Message
775a2ec5 John Pinto unknown_key fix(loki.source.kubernetes): Forward log lines with out-of-order timestamps

This repository requires all commits to be signed. See GitHub docs on commit signature verification.

@cla-assistant

cla-assistant Bot commented Sep 10, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@kalleep kalleep self-assigned this Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

"loki.source.kubernetes" drops log lines whose timestamp is earlier than the previous line (stdout/stderr interleaving)

2 participants