fix: use KeyBasedFileGroupRecordBuffer for non-Parquet base files in MOR merge

[yihua]

Describe the issue this Pull Request addresses

Closes #18558

Deletes are not merged correctly with Lance format on MOR table type.

Summary and Changelog

DefaultFileGroupRecordBufferLoader selects PositionBasedFileGroupRecordBuffer when useRecordPosition is true, regardless of the base file format. However, HoodieFileGroupReader sets shouldMergeUseRecordPosition = false for non-Parquet files via the isParquetBaseFile() gate. This mismatch causes the buffer to enter a fallback hybrid-strategy mode that fails to merge delete blocks for Lance base files.

Added the isParquetBaseFile() check to the buffer selection so non-Parquet formats (Lance, ORC) use KeyBasedFileGroupRecordBuffer directly, matching the merge strategy set by HoodieFileGroupReader.

Impact

Non-Parquet base file formats in MOR tables now correctly merge deletes. No impact on Parquet tables.

Risk Level

Low — narrows PositionBasedFileGroupRecordBuffer selection to Parquet only, which is the only format that supports row-index-based positional merging today.

Documentation Update

None.

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable

[hudi-agent]

🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for the fix! This narrows PositionBasedFileGroupRecordBuffer selection to Parquet base files only, matching the existing isParquetBaseFile() gate that HoodieFileGroupReader uses for shouldMergeUseRecordPosition. The Lance MOR delete test is un-skipped accordingly. No issues flagged from this automated pass — a Hudi committer or PMC member can take it from here for a final review.

cc @yihua

[hudi-agent]

🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for the fix! This narrows PositionBasedFileGroupRecordBuffer selection to Parquet base files, matching the existing shouldMergeUseRecordPosition gate in HoodieFileGroupReader so non-Parquet formats (Lance, ORC) consistently use KeyBasedFileGroupRecordBuffer. No issues flagged from this automated pass — a Hudi committer or PMC member can take it from here for a final review.

cc @yihua

[hudi-agent]

🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for the fix! This narrows PositionBasedFileGroupRecordBuffer selection to Parquet base files, aligning with the merge strategy HoodieFileGroupReader already enforces via shouldMergeUseRecordPosition, and unblocks the previously-skipped MOR DELETE/INSERT cases in TestLanceDataSource. No issues flagged from this automated pass — a Hudi committer or PMC member can take it from here for a final review.

cc @yihua

[voonhous]

Lgtm

[rahil-c]

@yihua can you look at test failures on ci

[hudi-agent]

🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for the fix! This PR removes the MOR-specific workarounds in the Lance test now that deletes merge correctly via KeyBasedFileGroupRecordBuffer for non-Parquet base files. No issues flagged from this automated pass — a Hudi committer or PMC member can take it from here for a final review.

cc @yihua

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 68.11%. Comparing base (4035f70) to head (2c413fd).

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #18588       +/-   ##
=============================================
+ Coverage     45.00%   68.11%   +23.10%     
- Complexity     8571    29089    +20518     
=============================================
  Files          1202     2518     +1316     
  Lines         62816   141221    +78405     
  Branches       6811    17531    +10720     
=============================================
+ Hits          28271    96188    +67917     
- Misses        31428    37139     +5711     
- Partials       3117     7894     +4777     

Flag	Coverage Δ
common-and-other-modules	44.40% <ø> (?)
hadoop-mr-java-client	45.01% <ø> (+<0.01%)	⬆️
spark-client-hadoop-common	48.33% <ø> (?)
spark-java-tests	48.62% <ø> (?)
spark-scala-tests	44.89% <ø> (?)
utilities	37.61% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 2115 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[hudi-bot]

CI report:

• 3d0299e UNKNOWN
• 9f03e32 UNKNOWN
• e14e313 UNKNOWN
• 7b9ed97 UNKNOWN
• a1bcc94 UNKNOWN
• 9854f52 UNKNOWN
• 2c413fd Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[hudi-agent]

🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for the fix! This PR aligns the buffer selection with HoodieFileGroupReader's merge strategy gating so non-Parquet base files (Lance/ORC) use KeyBasedFileGroupRecordBuffer, and the test changes simply remove the now-unnecessary CoW-only workarounds. No issues flagged from this automated pass — a Hudi committer or PMC member can take it from here for a final review.

cc @yihua