Skip to content

[WIP] Support low shuffle merge with multithreaded Parquet reader - #15934

Draft
liurenjie1024 wants to merge 6 commits into
NVIDIA:mainfrom
liurenjie1024:ray/11095
Draft

[WIP] Support low shuffle merge with multithreaded Parquet reader#15934
liurenjie1024 wants to merge 6 commits into
NVIDIA:mainfrom
liurenjie1024:ray/11095

Conversation

@liurenjie1024

@liurenjie1024 liurenjie1024 commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Fixes #11095.

Description

Low shuffle merge previously required the Parquet reader type to be PERFILE. Other reader
modes fell back to classic merge because the Delta 2.4 reader could not apply its per-file
deletion vectors in the multithreaded path.

This change enables low shuffle merge in every configured Parquet reader mode. It:

  • removes the reader-mode gate from low shuffle merge planning;
  • uses a Delta-aware multithreaded reader when a low-shuffle scan would otherwise select the
    coalescing reader, including explicit COALESCING and applicable AUTO scans;
  • reuses cuDF's Parquet deletion-vector processing for both per-file and multithreaded reads,
    through a shared helper also used by newer Delta shims;
  • keeps deletion vectors and row-group offsets aligned with each file/chunk and preserves cuDF's
    row-index output when synthetic Delta metadata columns are required;
  • supports metadata-only scans where no physical Parquet columns are decoded; and
  • keeps predicate pushdown disabled for low-shuffle target scans.

PERFILE continues to use the per-file reader. Low-shuffle scans configured for combined-file
reading use the multithreaded reader so deletion-vector state remains file-scoped. Each test in
delta_lake_low_shuffle_merge_test.py now explicitly parameterizes PERFILE, MULTITHREADED,
and COALESCING; the redundant focused combined-files test was removed.

Draft status:

  • Run upstream CI.
  • Address current early review feedback.

Validation:

  • mvn -o -pl dist,integration_tests -am -DskipTests -DskipITs -Dbuildver=340 -Dscala2.12 package
  • mvn -o -pl delta-lake/delta-24x -am -DskipTests -DskipITs -Dbuildver=340 -Dscala2.12 verify
  • Spark 3.4.0 with Delta Lake 2.4.0: targeted standard-upsert reader matrix passed for all three
    reader settings (6 passed, 6 expected xfailed).
  • Spark 3.4.0 with Delta Lake 2.4.0: metadata-only multithreaded delete matrix passed with all
    data columns used as partition columns (6 passed).

Checklists

Documentation

  • Updated for new or modified user-facing features or behaviors
  • No user-facing change

Testing

  • Added or modified tests to cover new code paths
  • Covered by existing tests
    (Please provide the names of the existing tests in the PR description.)
  • Not required

Performance

  • Tests ran and results are added in the PR description
  • Issue filed with a link in the PR description
  • Not required

Signed-off-by: Ray Liu <liurenjie2008@gmail.com>
Signed-off-by: Ray Liu <liurenjie2008@gmail.com>
Comment thread integration_tests/src/main/python/delta_lake_low_shuffle_merge_test.py Outdated
Signed-off-by: Ray Liu <liurenjie2008@gmail.com>
Comment thread integration_tests/src/main/python/delta_lake_low_shuffle_merge_test.py Outdated
Signed-off-by: Ray Liu <liurenjie2008@gmail.com>
Signed-off-by: Ray Liu <liurenjie2008@gmail.com>
Signed-off-by: Ray Liu <liurenjie2008@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEA] Force use PERFILE scan in low shuffle merge.

2 participants