Adding support for DML writes to DV enabled tables

[sdrp713]

Fixes #12577.

Description

This PR adds GPU acceleration for Apache Delta Lake DELETE, UPDATE, and MERGE commands on deletion vector enabled tables.

With deletion vectors, existing Parquet rows are logically removed by recording their physical row indexes in a DV. Updated or inserted rows are written to new Parquet files. This PR accelerates the data intensive parts of that workflow on the GPU while retaining Delta’s existing transaction and deletion vector semantics.

Implementation

The implementation accelerates:

• Scanning candidate Parquet files and applying existing deletion vectors.
• Producing physical row indexes for rows in their original source files.
• Evaluating DELETE, UPDATE, and MERGE predicates.
• Performing the joins and aggregations required by MERGE.
• Grouping modified physical row indexes by source file.
• Writing updated and inserted rows to new Parquet files.

Collision free dictionary IDs are used in GPU aggregations instead of carrying full file paths or relying on hashes. This reduces GPU memory and shuffle overhead while preserving an exact mapping back to each Delta AddFile.

Candidate files that already have deletion vectors are also supported. The GPU scan returns physical row indexes after applying the existing DV, and newly modified positions are combined with the existing deleted positions before the updated DV is written.

Sparse RoaringBitmap construction, DV serialization, Delta action construction, and optimistic transaction commit remain on the CPU. These are relatively smaller operations compared to the larger scans, joins, aggregations, and Parquet writes that remain GPU accelerated.

Performance

Performance was measured on the Spark2A A100 cluster using Spark 3.5.7 and Delta Lake 3.3.1.

The benchmark used approximately 365 days of the SF3000 NDS store_sales dataset. Deterministic predicates modified approximately 10% of the target rows. Each command was measured using five independent CPU runs and five independent GPU runs, with zero warmup iterations and cluster reuse.

Command	CPU mean	GPU mean	Speedup
DELETE	16.449 s	13.887 s	1.184×
UPDATE	15.276 s	7.637 s	2.000×
MERGE	72.611 s	42.159 s	1.722×

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Please provide the names of the existing tests in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains within the eligible follow-up review scope.

Summary

• Preserves physical Parquet row indexes while applying existing deletion vectors.
• Generates per-file deletion bitmaps and combines them with Delta’s existing transaction flow.
• Uses collision-free dictionary IDs to associate modified rows with candidate files.
• Extends Delta 3.3–4.2 shims and integration coverage for deletion-vector DML.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Delta DELETE, UPDATE, or MERGE] --> B[Scan candidate Parquet files on GPU]
  B --> C[Apply existing deletion vectors]
  C --> D[Preserve physical row indexes]
  D --> E[Evaluate DML predicates and joins]
  E --> F[Group modified indexes by dictionary file ID]
  F --> G[Build deletion bitmaps on CPU]
  G --> H[Combine with existing deletion vectors]
  H --> I[Write updated or inserted Parquet rows]
  I --> J[Construct Delta actions and commit transaction]

Loading

_{Reviews (6) · Last reviewed commit: "Fix scala style"}

[nartal1]

Could you add performance benchmark result for Delta-4.x as well.