You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Photon-to-Spark operator mapping is loaded from a single file, core/src/main/resources/parser/photon/databricks-13_3.json
(PhotonOssOpMapper.DEFAULT_OPS_MAPPING_FILE in DatabricksParseHelper.scala), whatever the
runtime version of the event log. The file's own header says "TODO: Create separate mapping
file for different Photon/Databricks versions". Photon operators that appeared after 13.3 are
not in it, so on Databricks 17.3 (Photon) logs they are parsed by the generic path and land in unsupported_operators.csv as Exec, Unsupported, Triage, which both misstates what the
plugin would do with the SQL and pulls the qualification score down.
Steps/Code to reproduce bug
A Databricks 17.3 Photon event log (spark.databricks.clusterUsageTags.sparkVersion = 17.3.x-photon-scala2.13, runtimeEngine = PHOTON) of a Delta workload with a MERGE and a
write.
spark_rapids qualification --platform databricks-azure --eventlogs <log> --output_folder out
qual_core_output/qual_metrics/<app>/unsupported_operators.csv lists the Photon nodes that
have no entry in databricks-13_3.json as unsupported execs. From the 17.3 log, one MERGE
SQL (the same four rows repeat for each of the others):
and elsewhere PhotonColumnarToRow and PhotonRange as Triage, PhotonJsonScan json as IgnorePerf. 39 of the log's 121 SQLs carry at least one of the seven. In the plan the
write pair sits under Execute WriteIntoDeltaCommand, which the tools do recognise, so the
command is scored and its own writer is not.
Control on Databricks 15.4 Photon (same tool version): PhotonWriteStage and PhotonParquetWriter are absent from the 13.3 file and are reported as Exec,"PhotonWriteStage","Unsupported",...,Triage.
Photon operators map to their Spark equivalents on every supported runtime, so that the plugin's
support table applies to them: write stages to the write command / WriteFiles path
(DataWritingCommandExec), and whatever else 17.3 introduces. Two ways to get there:
A shorter first step: add the missing nodes to the existing file, since the mapping is
additive (a 13.3 plan never contains these names, so nothing changes for older logs).
The second shape is enough for these seven. Each has a Spark equivalent the tools already
score or already set aside (WriteFiles, WholeStageCodegen, ColumnarToRow, Subquery, Range, Scan), none of them means something different on another runtime, and #1384 stays
for the day a mapping does diverge between runtimes.
Environment details
Environment location: Cloud (Databricks on Azure), DBR 17.3 Photon (and 15.4 as control).
spark-rapids-user-tools 26.08.0 with the bundled core jar. The same on a core jar built from dev (2073060), whose mapping file is unchanged.
Additional context
With the seven entries added to the file and the same tool build run on the same log, the
seven names leave unsupported_operators.csv (71 rows over 39 of 121 SQLs, to 0), the
unsupported stage share goes from 49.26% to 28.45% and the estimated speedup from 1.05x to
1.15x. The seven names reappear only as ignored rows (Subquery and ColumnarToRow as
IgnoreNoPerf, the JSON scan as the Delta log read it is). The app stays Not Recommended on
what remains, which is the Delta commands and the rest of the plan rather than these nodes.
Found while running the qualification and profiling
tools on Databricks 17.3 workloads as part of building a GPU-fit evaluation for Databricks
pipelines.
Describe the bug
The Photon-to-Spark operator mapping is loaded from a single file,
core/src/main/resources/parser/photon/databricks-13_3.json(
PhotonOssOpMapper.DEFAULT_OPS_MAPPING_FILEinDatabricksParseHelper.scala), whatever theruntime version of the event log. The file's own header says "TODO: Create separate mapping
file for different Photon/Databricks versions". Photon operators that appeared after 13.3 are
not in it, so on Databricks 17.3 (Photon) logs they are parsed by the generic path and land in
unsupported_operators.csvasExec, Unsupported, Triage, which both misstates what theplugin would do with the SQL and pulls the qualification score down.
Steps/Code to reproduce bug
A Databricks 17.3 Photon event log (
spark.databricks.clusterUsageTags.sparkVersion = 17.3.x-photon-scala2.13,runtimeEngine = PHOTON) of a Delta workload with a MERGE and awrite.
spark_rapids qualification --platform databricks-azure --eventlogs <log> --output_folder outqual_core_output/qual_metrics/<app>/unsupported_operators.csvlists the Photon nodes thathave no entry in
databricks-13_3.jsonas unsupported execs. From the 17.3 log, one MERGESQL (the same four rows repeat for each of the others):
and elsewhere
PhotonColumnarToRowandPhotonRangeasTriage,PhotonJsonScan jsonasIgnorePerf. 39 of the log's 121 SQLs carry at least one of the seven. In the plan thewrite pair sits under
Execute WriteIntoDeltaCommand, which the tools do recognise, so thecommand is scored and its own writer is not.
Control on Databricks 15.4 Photon (same tool version):
PhotonWriteStageandPhotonParquetWriterare absent from the 13.3 file and are reported asExec,"PhotonWriteStage","Unsupported",...,Triage.The 13.3 file has 29 keys: PhotonAdapter, PhotonAgg, PhotonBroadcastExchange,
PhotonBroadcastHashJoin, PhotonBroadcastNestedLoopJoin, PhotonExpand, PhotonFilter,
PhotonGenerate, PhotonGlobalLimit, PhotonGroupingAgg, PhotonGroupingAggWithRollup,
PhotonHashJoin, PhotonLocalLimit, PhotonProject, PhotonResultStage, PhotonRowToColumnar,
PhotonScan, PhotonShuffleExchangeSink, PhotonShuffleExchangeSource, PhotonShuffleHashJoin,
PhotonShuffleMapStage, PhotonShuffledHashJoin, PhotonSort, PhotonSubqueryBroadcast, PhotonTopK,
PhotonUnion, PhotonUnionShuffleExchangeSink, PhotonUnionShuffleMapStage, PhotonWindow.
Expected behavior
Photon operators map to their Spark equivalents on every supported runtime, so that the plugin's
support table applies to them: write stages to the write command /
WriteFilespath(
DataWritingCommandExec), and whatever else 17.3 introduces. Two ways to get there:databricks-15_4.json,databricks-17_3.json) selected fromclusterUsageTags.sparkVersion, as [FEA] Implement Version-Specific Photon to Spark Node Mapping for Photon parsing #1384 proposes, with the 13.3 file as the fallback.additive (a 13.3 plan never contains these names, so nothing changes for older logs).
The second shape is enough for these seven. Each has a Spark equivalent the tools already
score or already set aside (
WriteFiles,WholeStageCodegen,ColumnarToRow,Subquery,Range,Scan), none of them means something different on another runtime, and #1384 staysfor the day a mapping does diverge between runtimes.
Environment details
dev(2073060), whose mapping file is unchanged.Additional context
With the seven entries added to the file and the same tool build run on the same log, the
seven names leave
unsupported_operators.csv(71 rows over 39 of 121 SQLs, to 0), theunsupported stage share goes from 49.26% to 28.45% and the estimated speedup from 1.05x to
1.15x. The seven names reappear only as ignored rows (
SubqueryandColumnarToRowasIgnoreNoPerf, the JSON scan as the Delta log read it is). The app stays Not Recommended on
what remains, which is the Delta commands and the rest of the plan rather than these nodes.
Found while running the qualification and profiling
tools on Databricks 17.3 workloads as part of building a GPU-fit evaluation for Databricks
pipelines.