Skip to content

[BUG] Photon operators from Databricks 17.3 fall outside the 13.3 mapping file and are reported as unsupported Triage execs #2158

Description

@jtwynne

Describe the bug

The Photon-to-Spark operator mapping is loaded from a single file,
core/src/main/resources/parser/photon/databricks-13_3.json
(PhotonOssOpMapper.DEFAULT_OPS_MAPPING_FILE in DatabricksParseHelper.scala), whatever the
runtime version of the event log. The file's own header says "TODO: Create separate mapping
file for different Photon/Databricks versions". Photon operators that appeared after 13.3 are
not in it, so on Databricks 17.3 (Photon) logs they are parsed by the generic path and land in
unsupported_operators.csv as Exec, Unsupported, Triage, which both misstates what the
plugin would do with the SQL and pulls the qualification score down.

Steps/Code to reproduce bug

  1. A Databricks 17.3 Photon event log (spark.databricks.clusterUsageTags.sparkVersion = 17.3.x-photon-scala2.13, runtimeEngine = PHOTON) of a Delta workload with a MERGE and a
    write.

  2. spark_rapids qualification --platform databricks-azure --eventlogs <log> --output_folder out

  3. qual_core_output/qual_metrics/<app>/unsupported_operators.csv lists the Photon nodes that
    have no entry in databricks-13_3.json as unsupported execs. From the 17.3 log, one MERGE
    SQL (the same four rows repeat for each of the others):

    SQL ID,Stage ID,ExecId,Unsupported Type,Unsupported Operator,Details,Stage Duration,App Duration,Action
    10,35,21,Exec,"PhotonWriteStage","Unsupported",2603,312028,Triage
    10,35,22,Exec,"PhotonParquetWriter","Unsupported",2603,312028,Triage
    10,22,24,Exec,"PhotonMetadataSubquery","Unsupported",1324,312028,Triage
    10,22,28,Exec,"PhotonRuntimeFilterSource","Unsupported",1324,312028,Triage
    

    and elsewhere PhotonColumnarToRow and PhotonRange as Triage, PhotonJsonScan json as
    IgnorePerf. 39 of the log's 121 SQLs carry at least one of the seven. In the plan the
    write pair sits under Execute WriteIntoDeltaCommand, which the tools do recognise, so the
    command is scored and its own writer is not.

Control on Databricks 15.4 Photon (same tool version): PhotonWriteStage and
PhotonParquetWriter are absent from the 13.3 file and are reported as
Exec,"PhotonWriteStage","Unsupported",...,Triage.

The 13.3 file has 29 keys: PhotonAdapter, PhotonAgg, PhotonBroadcastExchange,
PhotonBroadcastHashJoin, PhotonBroadcastNestedLoopJoin, PhotonExpand, PhotonFilter,
PhotonGenerate, PhotonGlobalLimit, PhotonGroupingAgg, PhotonGroupingAggWithRollup,
PhotonHashJoin, PhotonLocalLimit, PhotonProject, PhotonResultStage, PhotonRowToColumnar,
PhotonScan, PhotonShuffleExchangeSink, PhotonShuffleExchangeSource, PhotonShuffleHashJoin,
PhotonShuffleMapStage, PhotonShuffledHashJoin, PhotonSort, PhotonSubqueryBroadcast, PhotonTopK,
PhotonUnion, PhotonUnionShuffleExchangeSink, PhotonUnionShuffleMapStage, PhotonWindow.

Expected behavior

Photon operators map to their Spark equivalents on every supported runtime, so that the plugin's
support table applies to them: write stages to the write command / WriteFiles path
(DataWritingCommandExec), and whatever else 17.3 introduces. Two ways to get there:

  1. Per-version mapping files (databricks-15_4.json, databricks-17_3.json) selected from
    clusterUsageTags.sparkVersion, as [FEA] Implement Version-Specific Photon to Spark Node Mapping for Photon parsing #1384 proposes, with the 13.3 file as the fallback.
  2. A shorter first step: add the missing nodes to the existing file, since the mapping is
    additive (a 13.3 plan never contains these names, so nothing changes for older logs).

The second shape is enough for these seven. Each has a Spark equivalent the tools already
score or already set aside (WriteFiles, WholeStageCodegen, ColumnarToRow, Subquery,
Range, Scan), none of them means something different on another runtime, and #1384 stays
for the day a mapping does diverge between runtimes.

Environment details

  • Environment location: Cloud (Databricks on Azure), DBR 17.3 Photon (and 15.4 as control).
  • spark-rapids-user-tools 26.08.0 with the bundled core jar. The same on a core jar built from
    dev (2073060), whose mapping file is unchanged.

Additional context

With the seven entries added to the file and the same tool build run on the same log, the
seven names leave unsupported_operators.csv (71 rows over 39 of 121 SQLs, to 0), the
unsupported stage share goes from 49.26% to 28.45% and the estimated speedup from 1.05x to
1.15x. The seven names reappear only as ignored rows (Subquery and ColumnarToRow as
IgnoreNoPerf, the JSON scan as the Delta log read it is). The app stays Not Recommended on
what remains, which is the Delta commands and the rest of the plan rather than these nodes.
Found while running the qualification and profiling
tools on Databricks 17.3 workloads as part of building a GPU-fit evaluation for Databricks
pipelines.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions