Add Databricks 14.3 and 17.3 to the AutoTuner shuffle manager map

[jtwynne]

Fixes #2144 , Fixes #1678.

Problem

On a Databricks 14.3 or 17.3 event log the profiling AutoTuner commented "Cannot recommend
RAPIDS Shuffle Manager for unsupported Databricks runtime" and advised moving to runtime 13.3,
whose shim the plugin has dropped (NVIDIA/cudf-spark#15276). The Databricks entries of
DatabricksPlatform.supportedShuffleManagerVersionMap stopped at 13.3 while the plugin ships
com.nvidia.spark.rapids.spark350db143.RapidsShuffleManager and
com.nvidia.spark.rapids.spark400db173.RapidsShuffleManager, the two classes the shuffle
documentation page linked from the comment lists for Databricks. Databricks sets
spark.shuffle.manager=SORT explicitly, so the plugin's automatic configuration does not apply
and the tuner's recommendation is the only place in the tools' output where a user is told the
class name.

Fix

The map gains 14.3 -> 350db143 and 17.3 -> 400db173.

• The 12.2 and 13.3 entries stay, so event logs from older runtimes keep their recommendation
  ([BUG] Support for Databricks 14.3? #1678).
• The "e.g." runtime in the unsupported-version comment becomes 17.3 through the existing
  maxBy, which compares version strings and orders 12.2 < 13.3 < 14.3 < 17.3 correctly.
• getShuffleManagerVersion matches by contains; no key is a substring of another key.
• Both Databricks shims are enabled by default in the plugin, the open question on [BUG] Support for Databricks 14.3? #1678: the
  14.3 shim since Enable the [databricks] 14.3 Shim cudf-spark#12485, the 17.3 shim from its first release
  (Add Databricks-17.3 support [databricks] cudf-spark#14360); both providers read getOrElse(true). Databricks 14.3 is Spark
  3.5.0 and 17.3 is Spark 4.0.0.
  buildShuffleManagerClassName needs no change for the NNNdb scheme. [BUG] Automate Generation of Supported Shims for RAPIDS Shuffle Manager #1676 (generate the
  list from the plugin) would subsume this.
• Scope: the profiling AutoTuner on Databricks AWS and Azure, which share DatabricksPlatform.
  Qualification never recommends a shuffle manager ([BUG] Qualification Tools (and Bootstrap) should not recommend the shuffle manager #1813). Event logs from 12.2 and 13.3 get
  the same recommendation as before.

Tests

• The per-entry Databricks test in ProfilingAutoTunerSuite
  (dbPlatform.supportedShuffleManagerVersionMap.foreach) runs the two entries, but it takes
  both the runtime and the expected class from the map, so it pins nothing on its own.
• Three cases in ProfilingAutoTunerSuiteV2 (the deprecated suite's header sends new AutoTuner
  cases there) pin the values with the expected class written out: "test shuffle manager
  version for databricks 14.3 is 350db143", "test shuffle manager version for databricks 17.3 on
  scala 2.13 is 400db173" (the -scala2.13 runtime suffix, on the Azure platform), and "test
  shuffle manager version for databricks 15.4 without a plugin shim", which expects the
  unsupported-version comment and asserts latestSupportedShuffleManagerInfo is the 17.3 pair.
  All three fail on dev without the two entries.
• verifyRecommendedShuffleManagerVersion and verifyUnsupportedSparkVersionForShuffleManager
  move from the deprecated suite into ProfilingAutoTunerSuiteBase, unchanged, so both suites
  share them.
• ProfilingAutoTunerSuite and ProfilingAutoTunerSuiteV2 together: 198 succeeded, 0 failed
  with the change (JDK 17, Scala 2.12); scalastyle clean.
• Profiling a Databricks 17.3 GPU event log with the change recommends
  spark.shuffle.manager=com.nvidia.spark.rapids.spark400db173.RapidsShuffleManager and no
  longer prints the "Cannot recommend RAPIDS Shuffle Manager for unsupported Databricks
  runtime" comment.

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; the new mappings are internally consistent with the generated class names and are covered by fixed-value tests.

Summary

• Maps Databricks 14.3 to the 350db143 shuffle-manager shim.
• Maps Databricks 17.3 to the 400db173 shuffle-manager shim.
• Adds fixed-value coverage for both mappings and verifies that 15.4 remains unsupported while 17.3 is presented as the latest supported runtime.
• Moves shared shuffle-manager test helpers into the common profiling AutoTuner suite base.

_{Reviews (2) · Last reviewed commit: "Pin the Databricks shuffle manager entri..."}

[amahussein]

Request changes for the missing regression oracle identified inline. The production change itself checks out: I verified both generated shuffle-manager classes in the plugin, both providers default enabled, and the current lookup and maximum select the intended results. The focused suite passed 92/92 locally under Scala 2.12 and 2.13, and GitHub Maven run 34313805053 passed 977/977 in both matching jobs.

Note This PR fully resolves #2144 and #1678.

[amahussein]

LGTM