Support Iceberg v3 deletion vector writes on GPU

[liurenjie1024]

Fixes #15442.

Description

GPU UPDATE and MERGE support also depends on the row-lineage work tracked by #15441.

Iceberg v3 uses Puffin deletion vectors instead of Parquet position-delete files. The GPU
position-delta write path previously rejected the Puffin delete format, so merge-on-read
writes could not produce or update v3 deletion vectors.

This change:

• routes deletion-vector writer construction through Iceberg-version shims, preserving
  compatibility with Iceberg 1.6, 1.9, 1.10, and 1.11
• shares the deletion-vector shim implementation and reflective Iceberg access helper in
  an Iceberg 1.9+ source layer that is not compiled by the Iceberg 1.6 module
• converts GPU delete batches to Iceberg position-delete records using scoped GPU/host
  resource management and releases the GPU batch before the row-wise writer handoff
• loads and merges rewritable position deletes so repeated writes produce at most one
  deletion vector per referenced data file
• returns rewritten delete files in the position-delta commit result
• accepts Puffin as the resolved delete format without introducing new configuration

Integration coverage exercises clustered and fanout deletion-vector writes, repeated DV
merges, and conversion of existing v2 Parquet position deletes after upgrading a table to
format v3. DELETE, UPDATE, and MERGE use separate CPU/GPU tables and compare final results
through the CPU reader.

Validation performed:

• Spark 3.5.3 / Iceberg 1.6 package build
• Spark 3.5.8 / Iceberg 1.9 and 1.10 Scala 2.12 package builds
• Spark 4.1.2 / Iceberg 1.11 Scala 2.13 package build
• repository Scalastyle (1,839 files, zero errors/warnings) and GPU resource-nesting lint
• five focused DELETE/UPDATE/MERGE tests on Spark 3.5.8 / Iceberg 1.10.1 using the normal
  package layout: 5 passed
• the same five tests using the matching Iceberg runtime and flattened RAPIDS aggregator
  on the extra classpath: 5 passed
• Apache RAT, Python syntax compilation, and git diff --check

The full Spark 3.5.8 reactor currently stops before the Iceberg modules because the current
JNI snapshot deprecates four existing applyBooleanMask call sites under warnings-as-errors.
The affected Iceberg modules were built directly as listed above.

The per-row Iceberg writer handoff and potential bulk API optimization are tracked by
NVIDIA/cudf-spark-jni#5070.

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Please provide the names of the existing tests in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[greptile-apps]

Greptile Summary

This PR adds GPU support for Iceberg v3 Puffin deletion-vector writes while retaining compatibility across Iceberg 1.6 through 1.11.

• Introduces version-specific shims for constructing deletion-vector writers, loading rewritable deletes, and assembling commit results.
• Converts GPU position-delete batches into Iceberg records with scoped GPU and host resource management.
• Adds DELETE, UPDATE, and MERGE integration coverage for clustered, fanout, repeated-write, and v2-to-v3 conversion scenarios.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains in the eligible follow-up-review scope.

No blocking failure remains.

Important Files Changed

Filename	Overview
iceberg/common/src/main/scala/org/apache/iceberg/spark/source/GpuSparkPositionDeltaWrite.scala	Adds deletion-vector writer construction, GPU-to-host position-delete conversion, rewritable-delete propagation, and version-dispatched commit-result assembly.
iceberg/iceberg-19plus-common/src/main/java/com/nvidia/spark/rapids/iceberg/Iceberg19PlusShimUtils.java	Implements the shared Iceberg 1.9+ bridge for rewritable-delete broadcasts, deletion-vector writers, and rewritten-delete commit metadata.
iceberg/iceberg-19plus-common/src/main/java/org/apache/iceberg/spark/source/GpuSparkPositionDeltaWriteAccess.java	Provides reflective access to Iceberg’s internal rewritable-delete broadcast method for supported 1.9+ runtimes.
iceberg/common/src/main/scala/org/apache/iceberg/spark/source/GpuSparkFileWriterFactory.scala	Allows Puffin as the resolved delete format while retaining Parquet data-file requirements.
integration_tests/src/main/python/iceberg/iceberg_delete_test.py	Extends CPU/GPU parity coverage for Iceberg deletion-vector DELETE workflows.
integration_tests/src/main/python/iceberg/iceberg_update_test.py	Adds Iceberg v3 deletion-vector UPDATE coverage.
integration_tests/src/main/python/iceberg/iceberg_merge_test.py	Adds repeated and conversion-oriented deletion-vector MERGE coverage.

Sequence Diagram

sequenceDiagram
  participant Spark as Spark position-delta write
  participant GPU as GPU delta writer
  participant Shim as Iceberg version shim
  participant DV as Iceberg DV writer
  participant Commit as Iceberg commit
  Spark->>GPU: Create task writer
  GPU->>Shim: Load rewritable deletes
  Shim-->>GPU: Version-specific broadcast handle
  GPU->>GPU: Produce path/position batches
  GPU->>Shim: Construct deletion-vector writer
  Shim->>DV: Create PartitioningDVWriter
  loop Each position delete
    GPU->>DV: Write path and position
  end
  DV-->>GPU: DeleteWriteResult
  GPU->>Shim: Combine data and delete results
  Shim-->>Commit: Data, new deletes, referenced data, rewritten deletes

Loading

_{Reviews (5): Last reviewed commit: "Merge remote-tracking branch 'upstream/m..." | Re-trigger Greptile}

[firestarman]

Better try to ask AI to help reduce duplicate/boilerplate code.

[res-life]

The following steps are on CPU, could we file a follow-up issue to move to GPU?
Iceberg CPU writer:

• Builds a map keyed by data-file path.
• Creates a RoaringPositionBitmap.
• Adds each deleted row position to the bitmap.
• Loads and merges existing deletion vectors.
• Serializes the bitmap and computes its CRC.
• Writes the Puffin file.
• Produces metadata for the new and rewritten DeleteFiles.
  Some of the above steps can move to GPU.

[gerashegalov]

LGTM

[liurenjie1024]

build

[res-life]

Filed NVIDIA/cudf-spark-jni#5082 to track the GPU Roaring64 build, union, and portable serialization primitive.

[liurenjie1024]

We need to hold the merge, since we need to merge #15866 first. Iceberg v3 will always update lineage metadata when doing dml operations.