Describe the bug
Apache Spark 4.2 adds DSv2 transaction management in SPARK-55855. Spark propagates a transaction to V2 write execution and commits it after the write but before refreshing the relation cache.
GpuV2ExistingTableWriteExec does not implement Spark's TransactionalExec contract. Its run path writes the data and refreshes the cache without invoking the transaction commit callback. A transactional DSv2 write replaced with the GPU implementation can therefore skip the transaction commit.
Steps/Code to reproduce bug
- Use Spark 4.2 with a DSv2 catalog that implements
TransactionalCatalogPlugin.
- Run a supported DML operation that is replaced by a GPU V2 write exec.
- Observe that the CPU V2 write commits the transaction before cache refresh, while the GPU path has no transaction callback or commit step.
Expected behavior
GPU V2 existing-table writes should participate in Spark's transaction lifecycle and commit the transaction after a successful write and before refreshing the cache.
Environment details
- Apache Spark 4.2
- cuDF Spark plugin with Spark 4.2 shims
Describe the bug
Apache Spark 4.2 adds DSv2 transaction management in SPARK-55855. Spark propagates a transaction to V2 write execution and commits it after the write but before refreshing the relation cache.
GpuV2ExistingTableWriteExecdoes not implement Spark'sTransactionalExeccontract. Its run path writes the data and refreshes the cache without invoking the transaction commit callback. A transactional DSv2 write replaced with the GPU implementation can therefore skip the transaction commit.Steps/Code to reproduce bug
TransactionalCatalogPlugin.Expected behavior
GPU V2 existing-table writes should participate in Spark's transaction lifecycle and commit the transaction after a successful write and before refreshing the cache.
Environment details