Sync supported ops with RAPIDS plugin as of 2026-05-26

[parthosa]

Summary

Syncs supported operator metadata in spark-rapids-tools with RAPIDS plugin generated files.

Plugin source: NVIDIA/cudf-spark@794daac

Skill workflow and sync-helper script changes were split to #2096. The override config remains in this PR because these TNEW gates are tied to the supported-ops data sync.

Changes

• Updated supported CSVs from root plugin generated files:
  • core/src/main/resources/supportedDataSource.csv
  • core/src/main/resources/supportedExecs.csv
  • core/src/main/resources/supportedExprs.csv
• Updated operator score CSVs for new operators.
• Updated scripts/sync_plugin_files/override_supported_configs.json with new TNEW gates for the new operators in this sync.

Sync Report Summary

• supportedDataSource.csv: 19 changed rows, 4 plugin-removed rows preserved.
  • Iceberg write support moved from tools-side CO placeholders toward plugin-generated S/PS/NA/NS values.
  • HiveORC and HiveParquet removed rows are preserved by the sync tool.
• supportedExecs.csv: 1 changed row, 12 plugin-removed rows preserved.
  • ExpandExec BINARY support changed from NS to S.
  • Removed rows such as WriteFilesExec, AppendDataExec, WindowGroupLimitExec, MapInArrowExec, CustomShuffleReaderExec, and related execs remain preserved by the sync tool.
• supportedExprs.csv: 13 added rows, 42 changed rows, 35 plugin-removed rows preserved.
  • New rows were added as TNEW.
  • Datatype support values now match plugin-generated metadata where applicable.

New Operators / TNEW Follow-Ups

These new expressions remain TNEW until tools-side parser behavior and test coverage justify promotion:

• ArrayAggregate
• Hex
• Sha2
• StringDecode

Tests

• git diff --check origin/dev...HEAD
• direct-CSV sync verification from origin/dev resources had 0 SQL Func regressions before the helper split
• mvn -f core/pom.xml test -DwildcardSuites=com.nvidia.spark.rapids.tool.qualification.PluginTypeCheckerSuite
• mvn -f core/pom.xml test -DwildcardSuites=com.nvidia.spark.rapids.tool.planparser.SQLPlanParserSuite

Note: the first SQLPlanParserSuite wildcard run failed inside the sandbox because Spark could not bind a local socket. The same wildcard command passed when rerun outside the sandbox.

Follow-Ups

• Validate parser behavior for ArrayAggregate, Hex, Sha2, and StringDecode before promoting any TNEW rows to S.

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge, with the new operators remaining gated as unsupported until tools-side parser validation is completed.

Summary

• Adds ArrayAggregate, Hex, Sha2, and StringDecode metadata and platform score entries.
• Keeps the newly introduced expressions classified as TNEW pending parser validation.
• Updates expression, execution, and data-source datatype support classifications.
• Adds matching sync overrides so subsequent synchronization retains the TNEW gates.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[RAPIDS plugin metadata] --> B[Supported-files sync]
  C[Override configuration] -->|Apply TNEW gates| B
  B --> D[supportedExprs.csv]
  B --> E[supportedExecs.csv]
  B --> F[supportedDataSource.csv]
  D --> G[PluginTypeChecker]
  E --> G
  F --> G
  H[Platform operator scores] --> I[Qualification scoring]
  G -->|TNEW remains unsupported| I

Loading