Skip to content

docs: add FAQ for Elasticsearch nanosecond timestamp log ordering (#1…#1253

Open
jinggghui wants to merge 1 commit intojenkinsci:mainfrom
jinggghui:docs/fix-elasticsearch-nanosecond-timestamp-1151
Open

docs: add FAQ for Elasticsearch nanosecond timestamp log ordering (#1…#1253
jinggghui wants to merge 1 commit intojenkinsci:mainfrom
jinggghui:docs/fix-elasticsearch-nanosecond-timestamp-1151

Conversation

@jinggghui
Copy link
Copy Markdown

@jinggghui jinggghui commented Mar 17, 2026

Fixes #1151

Adds a new FAQ entry to docs/build-logs.md explaining why logs appear
out of order in Elasticsearch/Kibana when using the Elastic backend, and
how to fix it.

Testing done

This is a documentation-only change. No code was modified.

The content was verified against the plugin source code:

  • @timestamp field name confirmed from ElasticsearchFields.java
  • logs-apm* index pattern confirmed from plugin Kibana dashboard NDJSON
    files (8.14, 8.17, 9.0.x)
  • logs-apm.app@custom component template confirmed as the official
    Elastic-recommended approach for customizing APM Data Stream mappings
    without breaking existing APM field mappings

Submitter checklist

  • Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
  • Ensure that the pull request title represents the desired changelog entry
  • Please describe what you did
  • Link to relevant issues in GitHub or Jira
  • Link to relevant pull requests, esp. upstream and downstream changes
  • Ensure you have provided tests that demonstrate the feature works or the issue is fixed

@jinggghui jinggghui requested a review from a team as a code owner March 17, 2026 16:50
Comment thread docs/build-logs.md
This clock adjustment is required to display in the right ascending order the log messages.
Note that distributed traces don't require such a clock adjustment because all spans are emitted from the Jenkins Controller.

### Why are logs appearing out of order in Elasticsearch/Kibana?
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could affect only steps that take less than a millisecond to execute, so the title is misleading.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, I never saw this behaviour, it is really complex to have a step that takes less than a millisecond to execute. Can you provide a sample pipeline to replicate the error?

Comment thread docs/build-logs.md
Comment on lines +189 to +194
**Fix 2 — Fix the sort order in Kibana:** Even with nanosecond storage,
Kibana may still show logs out of order because it sorts by milliseconds
internally. To fix this, set `event.sequence` as the tie-breaker field in
your Kibana Data View. The Jenkins OpenTelemetry plugin already includes
`event.sequence` in every log entry, so Kibana can use it to determine
the exact order logs were written.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Weird, IIRC the field to order is timestamp.us, which has nanosecond precision. Which version are you testing?

@ArpanC6
Copy link
Copy Markdown

ArpanC6 commented Mar 18, 2026

Great work on this documentation addition @jinggghui!

I noticed @kuisathaverat raised two important concerns:

  1. Reproducibility — The title says "nanosecond timestamp log ordering"
    but as @kuisathaverat pointed out, this would only affect steps completing
    in under 1ms, which is extremely rare in practice. Could you provide a
    sample pipeline that reliably reproduces this ordering issue? That would
    help validate the FAQ is solving a real observed problem.

  2. Fix 2 accuracy@kuisathaverat questioned whether event.sequence
    or timestamp.us is the correct tie-breaker field in Kibana. Looking at
    the plugin source, ElasticsearchFields.java defines both fields — could
    you clarify which Elasticsearch/Kibana version you tested against? The
    behavior may differ between versions.

Also noticed Java Compiler is showing 1 new issue in the CI checks —
while this is a docs-only change, it might be worth investigating if it's
pre-existing or introduced by this branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Log Order Inconsistency in ElasticSearch and Kibana due to Timestamp Precision Mismatch

3 participants