Skip to content

[FEA] Recognize noop V2 DataFrame writes and emit correct write-operation metadata #2126

Description

@parthosa

Problem Description

The RAPIDS Accelerator plugin is adding GPU support for Spark's built-in noop DataFrame writer in cudf-spark#13314, which tracks cudf-spark#13074. The plugin change supports the V2 AppendDataExec and OverwriteByExpressionExec paths on non-Databricks Spark 3.3+.

A Spark event log generated by:

spark.range(3).write.format("noop").mode("append").save()
spark.range(3).write.format("noop").mode("overwrite").save()

contains these plan nodes:

AppendData ... org.apache.spark.sql.execution.datasources.noop.NoopWrite$@...
OverwriteByExpression ... org.apache.spark.sql.execution.datasources.noop.NoopWrite$@...

Current tools output is incorrect:

  • both nodes are reported as unsupported Exec operations;
  • the generated DataSourceV2Strategy$$Lambda$... text triggers the misleading reason Is Dataset or RDD and action IgnorePerf;
  • neither node produces a WriteOperationRecord, so write_operations.csv is absent; and
  • the application summary has no Noop write format.

DataWritingCommandExecParser recognizes the V1 executor names but not the V2 plan-node names, so these nodes miss both the write-aware parser and write-record paths. A generated-CSV sync alone will not fix this. AppendDataExec and OverwriteByExpressionExec are shared by noop, Iceberg, and other V2 writers, so support must be determined from the concrete writer metadata rather than the executor class name alone.

Proposed Solution

Add a narrow noop-specific V2 write path:

  • Recognize AppendData and OverwriteByExpression only when the plan identifies Spark's built-in NoopWrite$ writer, leaving other V2 writers unchanged.
  • Report matching nodes as supported WriteExec operations on non-Databricks Spark 3.3+, and emit WriteOperationRecord metadata with format Noop and mode Append or Overwrite.
  • Register Noop as a supported write format, sync the final plugin metadata, and assign explicit speedup factors for both executor names.

A small dedicated parser/extractor is preferred to keep noop handling separate from Iceberg and other V2 writers.

Acceptance Criteria

  • On non-Databricks Spark 3.3+, noop AppendData and OverwriteByExpression plans are reported as supported write operations.
  • write_operations.csv records format Noop with mode Append or Overwrite; noop writes are no longer reported as Dataset/RDD operations or unsupported write formats.
  • Other V2 writers and unsupported Spark/platform combinations keep their current behavior.
  • Operator scores are defined for both executor names.

Test Plan

  • Add parser tests for noop append and overwrite, plus a non-noop V2 negative case.
  • Add Qualification coverage using a real noop event log and cover the Spark version/platform gates.

cc: @amahussein @revans2

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions