Read the current Databricks event log last in every time zone

[jtwynne]

Fixes #2141

Problem

A Databricks log-delivery directory holds the current eventlog and rolled
eventlog-YYYY-MM-DD--HH-MM.gz files. EventLogPathProcessor.getDBEventLogFileDate dated the
current file LocalDateTime.now() in the machine's zone and the rolled files by their stamps,
which are UTC wall time: in the Databricks 17.3 log this came from, the file named 18-20
ends at 18:19:55Z and the current file starts at 18:20:00Z. On a machine west of UTC, for up
to the zone offset after a rotation, the current file sorted first and the events were
processed out of order:

• every SQL, job and stage spanning the rotation lost its duration;
• totalCoreSeconds was 0 and removed_executors.csv was missing, because the executor's
  removal was processed before its addition;
• nothing was logged: doSparkListenerSQLExecutionEnd drops an end event whose start it has
  not seen.

Both the profiling and the qualification tool read directories through this path.

Fix

The undated current file is by definition the newest, so it is now dated LocalDateTime.MAX.

• LocalDateTime.now(ZoneOffset.UTC) would fix the zone but still depend on the reader's
  clock and still race with a rotation in the same minute; a sentinel needs no clock.
• The value is only ever compared in the file sort, never logged or added to, so nothing else
  changes.
• Two undated files would tie and keep their listing order; a Databricks directory has one.
• modificationTime and fileSizeForLastIndex (unused today) read the last file of the sorted
  list, which is now always the current file.

Tests

• QualificationNoSparkSuite, "db event log rolling with a rolled file dated after the current
  file": the db_sim_eventlog fixture copied into a temp directory with the rolled file renamed
  to eventlog-2099-12-31--23-59.gz, checked against the same expected files as the existing
  "db event log rolling" test, since the log content is unchanged. It reproduces the misorder
  in any zone at any time of day: on dev SQL 0 has an empty duration and App Duration is 0,
  and the test fails; with the change SQL 0 = 119903 ms and App Duration = 133857 ms.
• QualificationNoSparkSuite, "the undated current db event log is dated after every rolled
  file": asserts getDBEventLogFileDate("eventlog") directly.
• QualificationNoSparkSuite: 48 tests pass with the change, the two new ones fail without it.
  ApplicationInfoSuite passes. Scalastyle clean.

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; the ordering change is narrowly scoped and covered by both direct and end-to-end tests.

Summary

• Replaces the local-time now() sort value with LocalDateTime.MAX.
• Adds an end-to-end regression test using a future-dated rolled filename.
• Adds a focused assertion covering the new ordering sentinel.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    A[Databricks log directory] --> B[List eventlog files]
    B --> C{Timestamped filename?}
    C -->|Yes| D[Parse rolled-file timestamp]
    C -->|No| E[Assign LocalDateTime.MAX]
    D --> F[Sort chronologically]
    E --> F
    F --> G[Replay rolled logs first]
    G --> H[Replay current eventlog last]

Loading

[NVnavkumar]

LGTM