Support native Delta OPTIMIZE writes on GPU for DBR 14.3 and 17.3 [databricks]

[nartal1]

Fixes #11169.

Description

On DBR 14.3 and 17.3, native non-clustered Delta OPTIMIZE uses WriteIntoDeltaCommand outside the atomic write context. cudf-spark therefore rejected the command, causing DataWritingCommandExec and WriteFilesExec to
fall back to CPU.

This PR:

• Supports native non-clustered OPTIMIZE writes on GPU when the input is a TahoeBatchFileIndex with action type Optimize.
• Adds version-appropriate support for the DBR 14.3 and 17.3 shims.
• Preserves existing atomic CTAS/RTAS support and the separate DBR 17.3 clustered/liquid OPTIMIZE path.
• Retains CPU fallback for unsupported query, protocol, partition, and column combinations.
• Tightens the compaction allowlist so the nested write must use GpuDataWritingCommandExec and GpuWriteFilesExec.

The outer Databricks OPTIMIZE command remains on CPU. This change accelerates its nested data writing plan.

Testing

Added test_delta_native_optimize_gpu_write, which isolates the native Databricks path and verifies:

• Both nested write operators run on GPU without CPU fallback.
• CPU and GPU data, Delta logs, and history are equivalent.
• OPTIMIZE reduces the file count.

Performance

Benchmark on DBR 17.3 with an NVIDIA T4 GPU:

• 8 million rows in 128 unpartitioned Delta files.
• One warmup and five measured trials.
• Only the OPTIMIZE statement was timed.
• Every trial retained all rows and compacted 128 files to 3 files.

Plan	Median time
Before: CPU nested write	12.240 s
After: GPU nested write	5.758 s

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Please provide the names of the existing tests in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[greptile-apps]

Greptile Summary

Adds GPU acceleration for the nested Delta write performed by native, non-clustered Databricks OPTIMIZE.

• Registers and implements WriteIntoDeltaCommand GPU support for DBR 14.3 while extending DBR 17.3 eligibility beyond atomic CTAS/RTAS.
• Restricts native support to eligible TahoeBatchFileIndex OPTIMIZE inputs and preserves CPU fallback for clustered, partitioned, schema-incompatible, or otherwise unsupported writes.
• Aligns DBR 17.3 deletion-vector file matching with the expected empty partition-filter representation.
• Adds CPU/GPU data, Delta-log, history, compaction, and same-plan GPU-operator validation.

Confidence Score: 5/5

The PR appears safe to merge; no actionable regression remains in the changes since the previous review.

The latest changes consolidate native OPTIMIZE plan validation into the shared helper without weakening the new test’s same-plan GPU requirements or changing existing callers, and the full PR retains explicit fallback guards for unsupported Delta writes.

Important Files Changed

Filename	Overview
delta-lake/delta-spark350db143/src/main/scala/com/databricks/sql/transaction/tahoe/rapids/GpuWriteIntoDeltaCommand.scala	Implements guarded DBR 14.3 native OPTIMIZE GPU writes, including attribute remapping, tracker conversion, and file writing.
delta-lake/delta-spark350db143/src/main/scala/com/databricks/sql/transaction/tahoe/rapids/GpuWriteIntoDeltaCommandStats.scala	Bridges native Delta statistics collection to the GPU columnar write tracker.
delta-lake/delta-spark350db143/src/main/scala/com/nvidia/spark/rapids/delta/DeltaSpark350DB143Provider.scala	Registers the DBR 14.3 WriteIntoDeltaCommand replacement rule.
delta-lake/delta-spark400db173/src/main/scala/com/databricks/sql/transaction/tahoe/rapids/GpuWriteIntoDeltaCommand.scala	Extends DBR 17.3 eligibility to native non-clustered OPTIMIZE while retaining atomic CTAS/RTAS support.
delta-lake/delta-spark400db173/src/main/scala/com/nvidia/spark/rapids/delta/GpuDeltaParquetFileFormat.scala	Uses an empty partition-filter sequence when enumerating matching files for deletion-vector checks.
delta-lake/delta-spark400db173/src/main/scala/com/nvidia/spark/rapids/delta/RapidsDeletionVectors.scala	Aligns deletion-vector matching-file enumeration with DBR 17.3 file-index semantics.
integration_tests/src/main/python/delta_lake_optimize_table_test.py	Adds native OPTIMIZE parity, compaction, history, log, and nested GPU-write plan validation.
integration_tests/src/main/python/delta_lake_utils.py	Generalizes Delta write plan capture to require co-located GPU operators and prohibit named CPU fallbacks.
integration_tests/src/main/python/delta_lake_write_test.py	Tightens the Databricks compaction allowlist now that nested native OPTIMIZE writes are accelerated.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Databricks OPTIMIZE command on CPU] --> B{Native non-clustered OPTIMIZE?}
  B -->|No| C[Retain CPU write path]
  B -->|Yes| D[GpuDataWritingCommandExec]
  D --> E[GpuWriteFilesExec]
  E --> F[GPU Parquet output]
  F --> G[Delta transaction commit and statistics]

Loading

_{Reviews (2): Last reviewed commit: "Support native Delta OPTIMIZE writes on ..." | Re-trigger Greptile}

[nartal1]

build