Skip to content

perf: Improve global index performance for commit time ordering#17797

Open
yihua wants to merge 1 commit into
apache:masterfrom
yihua:perf-global-index-impr
Open

perf: Improve global index performance for commit time ordering#17797
yihua wants to merge 1 commit into
apache:masterfrom
yihua:perf-global-index-impr

Conversation

@yihua
Copy link
Copy Markdown
Contributor

@yihua yihua commented Jan 7, 2026

Describe the issue this Pull Request addresses

When using COMMIT_TIME_ORDERING merge mode with global indexes (RLI, Global Simple, Global Bloom) on MOR tables, the current implementation unnecessarily reads and merges with older record versions (by reading the latest file slice) during the tagging phase. This is wasteful because with commit time ordering, the newer commit always wins - there's no need to compare event timestamps.

This PR optimizes the tagGlobalLocationBackToRecords method in HoodieIndexUtils to skip the merge phase for COMMIT_TIME_ORDERING mode on MOR tables, reducing I/O overhead.

Summary and Changelog

Performance optimization for global index with COMMIT_TIME_ORDERING:

  • Modified HoodieIndexUtils.tagGlobalLocationBackToRecords() to skip merging with older record versions when COMMIT_TIME_ORDERING is used on MOR tables
  • The optimization is safe because commit time ordering semantics guarantee the newer commit always overwrites the older record, regardless of event time field values
  • Partition path updates still work correctly as shouldUpdatePartitionPath is checked independently
  • Delete operations work correctly as the delete marker from a later commit will naturally override any older record

Added comprehensive test suite TestGlobalIndexCommitTimeOrdering.java covering:

  • Basic upserts with lower/higher/equal event times
  • Partition path updates with commit time ordering
  • Delete operations
  • Deletes with unknown partition (global index lookup)
  • Mixed operations (inserts, updates, deletes)
  • Compaction behavior on MOR tables

Impact

Reduced I/O during write operations when using global indexes (RLI, Global Simple, Global Bloom) with COMMIT_TIME_ORDERING on MOR tables.

Risk Level

low

Documentation Update

none

Contributor's checklist

  • Read through contributor's guide
  • Enough context is provided in the sections above
  • Adequate tests were added if applicable

@github-actions github-actions Bot added the size:L PR with lines of changes in (300, 1000] label Jan 7, 2026
@hudi-bot
Copy link
Copy Markdown
Collaborator

hudi-bot commented Jan 7, 2026

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

// this phase since the writer path will merge when rewriting the files as part of the upsert operation.
boolean requiresMergingWithOlderRecordVersion = shouldUpdatePartitionPath || table.getMetaClient().getTableConfig().getTableType() == HoodieTableType.MERGE_ON_READ;
boolean requiresMergingWithOlderRecordVersion = shouldUpdatePartitionPath
|| (!isCommitTimeOrdered && table.getMetaClient().getTableConfig().getTableType() == HoodieTableType.MERGE_ON_READ);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice catch~

@yihua yihua added this to the release-1.2.0 milestone May 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L PR with lines of changes in (300, 1000]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants