Skip to content

[AUDIT][BUG][SPARK-55855][SQL] Support DSv2 transaction management in GPU writes #15902

Description

@zpuller

Describe the bug

Apache Spark 4.2 adds DSv2 transaction management in SPARK-55855. Spark propagates a transaction to V2 write execution and commits it after the write but before refreshing the relation cache.

GpuV2ExistingTableWriteExec does not implement Spark's TransactionalExec contract. Its run path writes the data and refreshes the cache without invoking the transaction commit callback. A transactional DSv2 write replaced with the GPU implementation can therefore skip the transaction commit.

Steps/Code to reproduce bug

  1. Use Spark 4.2 with a DSv2 catalog that implements TransactionalCatalogPlugin.
  2. Run a supported DML operation that is replaced by a GPU V2 write exec.
  3. Observe that the CPU V2 write commits the transaction before cache refresh, while the GPU path has no transaction callback or commit step.

Expected behavior

GPU V2 existing-table writes should participate in Spark's transaction lifecycle and commit the transaction after a successful write and before refreshing the cache.

Environment details

  • Apache Spark 4.2
  • cuDF Spark plugin with Spark 4.2 shims

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    audit_4.2.0Audit related tasks for 4.2.0bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions