Fix Delta v1 writer detection for spark 4.0+

[jihoonson]

Fixes #15926.

Description

A Delta replaceWhere overwrite through DataFrameWriter.saveAsTable could lose the existing partition specification when running on GPU with Spark 4.0 or later. When .partitionBy(...) was not restated, rows outside the replacement predicate survived, but their partition-column values were read as NULL.

Spark 4.0 moved the classic writer implementation to org.apache.spark.sql.classic.DataFrameWriter. The existing stack-trace-based V1 writer detection still checked the pre-Spark-4 class, causing the operation to be treated as a V2 table replacement and allowing the existing partition metadata to be replaced. This change should introduce no performance change as the fixed v1 writer detection for Spark 4.0 is identical as it for older versions except for the writer class name. For Spark 4.1, the detection logic should be even cheaper as it checks only the DeltaOptions and SaveMode.

This change moves V1 saveAsTable overwrite detection behind the Delta runtime shim and uses the appropriate implementation for each supported Delta version:

• Older Delta versions retain the existing V1 writer detection.
• Delta 4.0 checks for Spark's classic DataFrameWriter.
• Delta 4.1 uses Delta's CreateDeltaTableLikeShims.isV1WriterSaveAsTableOverwrite helper.

The detected writer type is computed once and used consistently when updating table metadata, selecting replace semantics, and handling domain metadata.

The new integration test creates equivalent CPU and GPU partitioned Delta tables and performs a replaceWhere overwrite through saveAsTable without restating .partitionBy(...). It verifies that:

• The GPU overwrite executes GpuOverwriteByExpressionExecV1.
• GPU and CPU partition counts match.
• Rows outside the replacement predicate retain their partition values.
• The table remains partitioned by region.

The change has been verified locally against Spark 3.5.5, 4.0.1, and 4.1.1 by running the new test added.

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Please provide the names of the existing tests in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[greptile-apps]

Confidence Score: 5/5

The implementation appears safe to merge; the remaining test-helper concern is non-blocking because the test now explicitly verifies the corrected GPU V1 path.

Findings

1.  V1 path is unverified ▶

Summary

• Moves V1 writer classification behind the Delta runtime shim.
• Uses Spark’s classic DataFrameWriter for Delta 4.0 and Delta’s native helper for Delta 4.1.
• Computes the classification once and applies it consistently to metadata replacement, write semantics, domain metadata, and operation metadata.
• Adds a regression test covering partition preservation, CPU/GPU result parity, and the version-specific GPU V1 execution node.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Delta saveAsTable overwrite] --> B[Build DeltaOptions and read SaveMode]
    B --> C{Delta runtime}
    C -->|Older versions| D[Check legacy DataFrameWriter stack frame]
    C -->|Delta 4.0| E[Check classic.DataFrameWriter stack frame]
    C -->|Delta 4.1| F[Call CreateDeltaTableLikeShims helper]
    D --> G{V1 overwrite?}
    E --> G
    F --> G
    G -->|Yes| H[Retain existing table metadata]
    G -->|No| I[Apply V2 replacement semantics]

Loading

_{Reviews (5) · Last reviewed commit: "Merge branch 'main' of https://github.co..."}

[jihoonson]

@greptile-apps I have updated the PR description to include why the performance evaluation is not required for this PR. Please have a look.

[greptile-apps]

V1 path is unverified

The regression test now checks only GpuAtomicReplaceTableAsSelectExec, the V2 replacement node, instead of also verifying GpuOverwriteByExpressionExecV1, the V1 path corrected by this PR. A regression in V1 writer detection could therefore pass while the surrounding V2 command still executes. The test also compares CPU and GPU results manually rather than using assert_gpu_and_cpu_are_equal_collect or assert_gpu_fallback_collect, as required by the repository's GPU integration-test directive. This requirement must be satisfied before merging; please retain an explicit V1-node assertion while using one of the required comparison helpers.

Rule Used: Integration tests must verify GPU execution using ... (source)

[jihoonson]

Fixed

[sdrp713]

LGTM

[jihoonson]

build

[gerashegalov]

LGTM

[jihoonson]

build

[gerashegalov]

Delta42xRuntimeShim extends DeltaRuntimeShimBase directly and does not override this method. That reintroduces the bug for Delta 4.2—its Spark 4.0 artifact checks classic.DataFrameWriter, while its Spark 4.1 artifact reads the explicit V1-overwrite option. Could Delta42xRuntimeShim delegate to CreateDeltaTableLikeShims just like Delta41xRuntimeShim?