Problem Description
The RAPIDS Accelerator plugin is adding GPU support for Spark's built-in noop DataFrame writer in cudf-spark#13314, which tracks cudf-spark#13074. The plugin change supports the V2 AppendDataExec and OverwriteByExpressionExec paths on non-Databricks Spark 3.3+.
A Spark event log generated by:
spark.range(3).write.format("noop").mode("append").save()
spark.range(3).write.format("noop").mode("overwrite").save()
contains these plan nodes:
AppendData ... org.apache.spark.sql.execution.datasources.noop.NoopWrite$@...
OverwriteByExpression ... org.apache.spark.sql.execution.datasources.noop.NoopWrite$@...
Current tools output is incorrect:
- both nodes are reported as unsupported
Exec operations;
- the generated
DataSourceV2Strategy$$Lambda$... text triggers the misleading reason Is Dataset or RDD and action IgnorePerf;
- neither node produces a
WriteOperationRecord, so write_operations.csv is absent; and
- the application summary has no
Noop write format.
DataWritingCommandExecParser recognizes the V1 executor names but not the V2 plan-node names, so these nodes miss both the write-aware parser and write-record paths. A generated-CSV sync alone will not fix this. AppendDataExec and OverwriteByExpressionExec are shared by noop, Iceberg, and other V2 writers, so support must be determined from the concrete writer metadata rather than the executor class name alone.
Proposed Solution
Add a narrow noop-specific V2 write path:
- Recognize
AppendData and OverwriteByExpression only when the plan identifies Spark's built-in NoopWrite$ writer, leaving other V2 writers unchanged.
- Report matching nodes as supported
WriteExec operations on non-Databricks Spark 3.3+, and emit WriteOperationRecord metadata with format Noop and mode Append or Overwrite.
- Register
Noop as a supported write format, sync the final plugin metadata, and assign explicit speedup factors for both executor names.
A small dedicated parser/extractor is preferred to keep noop handling separate from Iceberg and other V2 writers.
Acceptance Criteria
- On non-Databricks Spark 3.3+, noop
AppendData and OverwriteByExpression plans are reported as supported write operations.
write_operations.csv records format Noop with mode Append or Overwrite; noop writes are no longer reported as Dataset/RDD operations or unsupported write formats.
- Other V2 writers and unsupported Spark/platform combinations keep their current behavior.
- Operator scores are defined for both executor names.
Test Plan
- Add parser tests for noop append and overwrite, plus a non-noop V2 negative case.
- Add Qualification coverage using a real noop event log and cover the Spark version/platform gates.
cc: @amahussein @revans2
Problem Description
The RAPIDS Accelerator plugin is adding GPU support for Spark's built-in
noopDataFrame writer in cudf-spark#13314, which tracks cudf-spark#13074. The plugin change supports the V2AppendDataExecandOverwriteByExpressionExecpaths on non-Databricks Spark 3.3+.A Spark event log generated by:
contains these plan nodes:
Current tools output is incorrect:
Execoperations;DataSourceV2Strategy$$Lambda$...text triggers the misleading reasonIs Dataset or RDDand actionIgnorePerf;WriteOperationRecord, sowrite_operations.csvis absent; andNoopwrite format.DataWritingCommandExecParserrecognizes the V1 executor names but not the V2 plan-node names, so these nodes miss both the write-aware parser and write-record paths. A generated-CSV sync alone will not fix this.AppendDataExecandOverwriteByExpressionExecare shared by noop, Iceberg, and other V2 writers, so support must be determined from the concrete writer metadata rather than the executor class name alone.Proposed Solution
Add a narrow noop-specific V2 write path:
AppendDataandOverwriteByExpressiononly when the plan identifies Spark's built-inNoopWrite$writer, leaving other V2 writers unchanged.WriteExecoperations on non-Databricks Spark 3.3+, and emitWriteOperationRecordmetadata with formatNoopand modeAppendorOverwrite.Noopas a supported write format, sync the final plugin metadata, and assign explicit speedup factors for both executor names.A small dedicated parser/extractor is preferred to keep noop handling separate from Iceberg and other V2 writers.
Acceptance Criteria
AppendDataandOverwriteByExpressionplans are reported as supported write operations.write_operations.csvrecords formatNoopwith modeAppendorOverwrite; noop writes are no longer reported as Dataset/RDD operations or unsupported write formats.Test Plan
cc: @amahussein @revans2