[FEA] Add GPU low shuffle merge for Databricks 17.3 [databricks]

[liurenjie1024]

Fixes #11079.

Description

Databricks 17.3 supports low shuffle merge, but the GPU implementation was only available for older Delta/Databricks shims. As a result, merge operations on Databricks 17.3 could not use the GPU low shuffle merge path.

This change ports the Delta 2.4 GPU low shuffle merge algorithm to the Databricks 17.3 shim and adapts it to the Databricks 17.3 command interfaces. It supports both classic and Edge merge commands, including source materialization, schema evolution, identity columns, type widening, row tracking, set transactions, and statistics.

GPU low shuffle merge requires the PERFILE Parquet reader mode so metadata row indices remain file-relative. When another reader mode is configured, the command falls back to the existing classic GPU merge path.

The integration-test gate enables the Databricks path only for exact DBR 17.3; later DBR versions remain skipped until they are explicitly supported.

For Databricks 17.3, touched-file discovery uses Databricks' native metadata-row-index scan preparation. The unmodified-row pass adds temporary inline deletion vectors to the touched AddFile entries and relies on the existing Databricks deletion-vector scan path to filter modified rows. This avoids a custom low-shuffle scan registry or file-format implementation.

CDF-enabled tables remain on the low-shuffle path. The modified-row pass now emits explicit insert, delete, update-preimage, and update-postimage rows alongside rewritten table rows. Merge expressions are tagged at the join-processing conversion boundary; replaceable expressions use RapidsProcessDeltaMergeJoin, while unsupported expression trees use the existing joined-row processor without falling back the whole merge command. Non-deterministic update/insert expressions are materialized under their clause-routing predicates so table and CDF outputs share the same value. The same native CDF support is added to the Delta 2.4 implementation used as the basis for the Databricks shim.

The change also generalizes the Databricks 17.3 table-write overflow wrapper to accept any converted GPU expression while retaining the original source type needed for Delta overflow semantics.

The branch contains upstream main at b50f77447 (merge commit 6b69ead62). The functional tree at that merge commit was validated locally and on a one-node, single-user DBR 17.3 g4dn.xlarge GPU cluster. The current head 45a7bc35a only updates the modified Delta 2.4 file's copyright year for CI.

• Local Spark 3.4/Delta 2.4 low-shuffle-merge suite: 122 passed, 0 failed in 16m21s.
• Databricks 17.3 and default upstream Maven reactors using jenkins/databricks/build.sh: build success; the DBR 17.3 Delta module compiled successfully (job 238183867741769).
• Focused DBR 17.3 CDF partial/full/no-op delete coverage: 12 passed, 0 failed in 7m48s (job 536767695989422).
• Full DBR 17.3 CDF-enabled low-shuffle-merge sweep: 31 passed, 31 deselected, 0 failed in 12m20s (job 280834283468655).
• Local build/style checks, git diff --check, current-year license-header validation, Python syntax validation, and Signed-off-by verification: success.

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

This is a shim port of the established Delta 2.4 low shuffle merge algorithm. The validation above exercises the runtime path on the target DBR release; this change does not retune the algorithm.

AI assistance: Codex assisted with the review fixes, Databricks cluster automation, and validation. The author directed this update; normal maintainer review remains required.

[greptile-apps]

Confidence Score: 4/5

The PR is not yet safe to merge because Delta 2.4 CDF events can disagree with table rows when merge actions contain non-deterministic expressions.

Findings

1.  CDF values can diverge ▶
2.  Performance validation is required. ▶

Summary

• Selects low-shuffle merge for supported Databricks 17.3 commands while preserving classic GPU merge fallback.
• Generates insert, delete, update-preimage, and update-postimage CDF records.
• Generalizes table-write overflow handling to converted GPU expressions.
• Documents the reader-mode requirement and gates integration coverage to the supported runtime.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  Merge[MERGE command] --> Gate{DBR 17.3 and supported configuration?}
  Gate -->|No| Classic[Classic GPU merge]
  Gate -->|Yes| Touched[Discover touched files and row indices]
  Touched --> Modified[Generate modified table rows]
  Touched --> Unmodified[Read unmodified rows through deletion vectors]
  Modified --> CDF{CDF enabled?}
  CDF -->|Yes| Events[Generate insert/delete/preimage/postimage rows]
  CDF -->|No| TableRows[Table rows only]
  Events --> Write[Delta write and commit]
  TableRows --> Write
  Unmodified --> Write

Loading

_{Reviews (2) · Last reviewed commit: "[DOC] Update low shuffle merge copyright"}

[greptile-apps]

Performance validation is required. This change selects a new runtime merge algorithm that alters scanning, broadcasting, shuffling, and writing, but the PR marks performance validation as “Not required.” The performance checklist directive requires measurements or a tracked performance issue unless the change cannot affect runtime performance. This requirement must be satisfied before merging.

Rule Used: Report Performance: Not required as a high-sever... (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[greptile-apps]

CDF values can diverge

For a CDF-enabled Delta 2.4 merge, non-deterministic update and insert expressions are evaluated separately for the table row and its CDF row. Expressions such as rand() or uuid() can therefore produce a CDF postimage or insert event whose values differ from the row written to the table. Materialize each clause-routed non-deterministic expression once before generating both outputs, as the Databricks 17.3 implementation does.

Knowledge Base Used: Delta Lake integration

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[nartal1]

@liurenjie1024 - Could you please update the performance results in the PR description.

[liurenjie1024]

@liurenjie1024 - Could you please update the performance results in the PR description.

Yes, I'm working on this.

[gerashegalov]

DBR 16.0+ only treats multiple source matches as ambiguous when more than one joined pair takes a WHEN MATCHED action. This counts every ON match, so a target matched by one apply=true and one apply=false source row is rejected even though DBR 17.3 and the classic GpuMergeIntoCommand accept it. Could we either port the classic command effective-match counting and joined-pair de-duplication, or fall back to classic merge whenever raw duplicates require that decision? It would also be good to run the existing accepted duplicate-match cases with low shuffle enabled.

[gerashegalov]

Should this preserve the DBR row-tracking columns with UpdateCommandShims.preserveRowTrackingColumns, as GpuMergeIntoCommand.writeAllChanges does? Both the modified and unmodified passes rebuild their scans from metadata.schema, which excludes the hidden materialized row ID and row commit version. Those values are therefore absent at writeFiles, so rewritten copied and updated rows can receive new row IDs instead of preserving them. Please also add a low-shuffle row-tracking regression without a NOT MATCHED BY SOURCE clause, since that clause currently selects the classic fallback.

[gerashegalov]

Could we make the non-CDF helper names collision-safe too? These fixed withColumn names replace same-named source or target columns, and the later string-based drop also removes user columns named row_dropped or incr_metrics. The CDF branch already uses uniqueColumnName and drops the generated control attributes precisely. Enabling low shuffle should not change classic-merge behavior for these schemas, so a low-shuffle regression with colliding user column names would help.