Support Iceberg v3 row lineage writes on GPU

[res-life]

Fixes #15441.

Description

Adds GPU support for preserving Iceberg v3 row lineage during writes.

This keeps row IDs stable for unchanged rows, assigns lineage to inserted rows, and advances sequence lineage only for modified rows while preserving CPU/GPU interoperability.

Earlier copy-on-write measurements showed 1.33x–1.70x GPU speedups for UPDATE, DELETE and MERGE. These measurements predate the review fixes and were not rerun for the current head.

Before the review fixes, the following local Iceberg v3 copy-on-write microbenchmarks used an NVIDIA L4, Spark 3.5.6, Iceberg 1.10.1, Scala 2.12, Java 17, local[16], 16 shuffle partitions, and AQE disabled. Each result is the median of three measured runs after one warmup. Unless noted otherwise, each input contained 5 million rows. CPU/GPU values greater than 1 indicate a GPU speedup.

Operation	CPU median	GPU median	CPU/GPU	GPU latency change
Read, including row-lineage metadata	0.212 s	0.227 s	0.93x	+7.3%
Append	0.284 s	0.327 s	0.87x	+15.0%
CTAS	0.301 s	0.373 s	0.81x	+23.9%
UPDATE 2.5M rows	1.845 s	1.083 s	1.70x	-41.3%
DELETE 2.5M rows	1.106 s	0.832 s	1.33x	-24.8%
MERGE, 2.5M updates + 1.25M inserts	2.654 s	1.888 s	1.41x	-28.9%
INSERT OVERWRITE	0.278 s	0.304 s	0.92x	+9.2%

The short read and direct-write cases are dominated by fixed planning and commit overhead at this scale. The runtime-heavy copy-on-write DML cases are 1.33x-1.70x faster on GPU. Every GPU benchmark captured and verified its expected GPU execution operator, and every iteration checked row-lineage correctness.

A separate Iceberg v2 GPU MERGE regression control measured the parent at 1.669 s and this branch at 1.633 s, so no regression was observed (the branch was 2.2% lower). Both revisions used the same current JNI snapshot and the same applyRetentionMask compatibility patch from #15848, which is not included in this PR.

The write path propagates Iceberg's hidden lineage columns with their required field IDs and ordering. Scan staging preserves row-position information needed by copy-on-write rows, and merge-row execution distinguishes unchanged, modified, and inserted rows so Iceberg can inherit the correct lineage values.

The integration coverage validates read, append, CTAS, UPDATE, DELETE, MERGE, and insert-overwrite behavior, including snapshot metadata, all supported RAPIDS Iceberg reader types, and CPU reads of GPU-written tables. Operation-specific row-lineage tests live in their corresponding UPDATE, DELETE, and MERGE suites. The CTAS test captures its command plan, requires the expected GPU operator, and rejects the corresponding CPU fallback.

Spark 4 position-delta writes preserve REINSERT metadata while keeping INSERT and REINSERT rows in their original partition order. The writer restores missing lineage by metadata column name, retains existing row IDs for reinserted rows, and leaves inserted-row lineage null for Iceberg inheritance. Mixed MERGE action outputs are padded with typed nulls before concatenation.

The GPU batch contract is an independent Java GpuDeltaBatchWriter in sql-plugin-api, packaged once at the JAR root. GpuIcebergDeltaWriter directly implements Spark's DeltaWriter and the batch contract; the two Spark-shim GpuDeltaWriter traits are removed.

Existing v3 coverage is expanded across 140 test functions while retaining parameter interactions, IDs and marks. Tests with specific delete-file-format contracts retain those contracts. The CTAS plan callback accepts both CPU and GPU plans and continues to assert GPU execution.

Full GPU v3 merge-on-read still requires the deletion-vector writer tracked in #15442 / #15717. The tests assert CPU delta-write fallback where that writer is unavailable, while allowing Spark's GPU append rewrite for insert-only MERGE.

Validation

Review-fix builds and runtime checks ran at 874d19a0b. The published head 71a95da74 differs from that revision only in the appendLineage and writeInserts Scaladoc comments and examples; builds/tests were not repeated for those comment-only amendments.

Check	Result
Maven install: Spark 3.5.6 / Scala 2.12	Passed
Maven install: Spark 3.5.6 / Scala 2.13	Passed
Maven install: Spark 4.1.3 / Scala 2.13	Passed
Multi-shim distribution: Spark 356 + 413 / Scala 2.13	Passed, including strict Iceberg package-access audit
Lineage writer, delta dispatcher and MERGE retry unit suites	8 passed
Spark 3.5.6 / Iceberg 1.10.1, normal loading	23 passed
Spark 3.5.6 / Iceberg 1.10.1, ICEBERG_EXTRA_CLASSPATH	23 passed
Spark 4.1.3 / Iceberg 1.11.0, normal loading	23 passed
Spark 4.1.3 / Iceberg 1.11.0, ICEBERG_EXTRA_CLASSPATH	23 passed

The 92 focused integration cases cover MERGE, partition evolution, UPDATE, DELETE, CTAS and row lineage. Normal and extra-classpath runs use identical selected cases for each Spark/runtime pair. Extra-classpath verification puts both the matching flattened RAPIDS aggregator and Iceberg runtime on the system classpath and removes spark.jars.packages. All JUnit reports show zero failures, errors or skips. Spark 4 pytest summaries and complete JUnit reports confirm 23 passes per run; the outer command exit codes were not preserved after a session interruption.

The Java API bytecode is identical across the three builds and independent compilations against Spark 3.3.0, 3.5.6 and 4.1.3. The five affected duplicate Iceberg writer classes are bitwise-identical between Iceberg 1.9.x and 1.10.x for Spark 356, separately for Scala 2.12 and 2.13. Both distribution JARs contain exactly one new API class at the root and no old GpuDeltaWriter class under either package. Builds used Java 17 and skipped ScalaDoc generation. Diff, copyright and commit identity/DCO checks passed.

The complete Iceberg suites and performance benchmarks were not rerun after these review fixes.

Earlier validation, before the review fixes

• Spark 3.5.6 / Scala 2.12 clean build — passed.
• Spark 4.1.3 / Scala 2.13 clean build — passed.
• Six PR-added datagen row-lineage cases on Spark 3.5.6 / Iceberg 1.10.1 — 6 passed.
• Six PR-added datagen row-lineage cases on Spark 4.1.3 / Iceberg 1.11.0 — 6 passed.
• All Iceberg cases on Spark 3.5.6 / Iceberg 1.10.1 — 433 passed, 23 skipped, 7 xfailed, 3 xpassed.
• All Iceberg cases on Spark 4.1.3 / Iceberg 1.11.0 — 439 passed, 18 skipped, 6 xfailed, 3 xpassed.
• Extra-classpath verification using the per-shim aggregator JAR plus the matching Iceberg runtime JAR on Spark 3.5.6 / Iceberg 1.10.1 — 6 affected cases passed.
• Extra-classpath verification using the per-shim aggregator JAR plus the matching Iceberg runtime JAR on Spark 4.1.3 / Iceberg 1.11.0 — 6 affected cases passed.
• Seven Iceberg v3 operations in CPU and GPU modes, one warmup plus three measured iterations — 14/14 passed with GPU-plan and lineage validation.
• git diff --check and copyright-year checks — passed.

AI assistance: Codex assisted with implementation, review-feedback fixes, documentation and validation.

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Please provide the names of the existing tests in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge.

Summary

• Propagates hidden lineage metadata through GPU writer interfaces and Iceberg writers.
• Distinguishes inserted, modified, and reinserted rows during merge execution.
• Extends Iceberg v3 integration and unit coverage across reads, writes, and row-level operations.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Iceberg v3 scan] --> B[Preserve row position and lineage metadata]
  B --> C[GPU merge-row execution]
  C --> D{Row action}
  D -->|Unchanged or reinserted| E[Preserve existing lineage]
  D -->|Modified| F[Preserve row ID and advance sequence lineage]
  D -->|Inserted| G[Leave lineage null for Iceberg inheritance]
  E --> H[GPU Iceberg writer]
  F --> H
  G --> H
  H --> I[Committed Iceberg data files]

Loading

_{Reviews (8) · Last reviewed commit: "Exercise Iceberg v3 across existing test..."}

[res-life]

The last test result of Iceberg ITs:
All Iceberg cases on Spark 3.5.6 / Iceberg 1.10.1 — 433 passed, 23 skipped, 7 xfailed, 3 xpassed.
All Iceberg cases on Spark 4.1.3 / Iceberg 1.11.0 — 439 passed, 18 skipped, 6 xfailed, 3 xpassed.