Recommend the RAPIDS cache serializer for cached SQL plans

[parthosa]

Fixes #2116

Summary

This PR adds a shared AutoTuner heuristic that recommends spark.sql.cache.serializer=com.nvidia.spark.ParquetCachedBatchSerializer when an application uses Spark SQL caching.

Approach

• Detect cache evidence in every observed SQL plan version, including AQE updates that are later replaced or cleaned up.
• Recognize InMemoryRelation, InMemoryTableScan, TableCacheQueryStage, GpuInMemoryTableScan, and Spark's named-cache scan form (Scan In-memory table <name>).
• Inspect structured stage RDD scopes when Spark omits cache operators from SQL plan events.
• Keep only a Boolean cache-evidence signal and stop traversing plans/scopes once evidence is found.
• Expose the same signal to Qualification and Profiling AutoTuner providers.
• Read the recommended serializer and Spark default from the existing tuning YAML files.

Configuration behavior

• Missing values and Spark's default serializer are replaced with the configured RAPIDS serializer.
• An already-correct serializer is unchanged.
• A custom serializer is preserved with a compatibility advisory.
• Preserved or enforced target-cluster values remain unchanged and retain the advisory; a preserve directive without a source value does not suppress the recommendation.
• Excluded properties receive no recommendation.
• If spark.rapids.sql.exec.InMemoryTableScanExec is explicitly disabled, AutoTuner suppresses the serializer recommendation and reports that GPU cache scans remain disabled.
• Spark 3.5.0 and 3.5.1 receive an advisory that GPU InMemoryTableScan remains disabled under AQE.

Testing

• 104 focused tests passed across the Profiling AutoTuner, Qualification AutoTuner, provider integration, and SQL plan model suites.
• A Spark-generated named-cache event log asserts the actual Scan In-memory table hot_orders node and verifies both Qualification and Profiling recommendations.
• The Spark-generated integration suite passed with Spark 3.4.4, 3.5.1, and 3.5.7 Maven build profiles.
• mvn clean package -DskipTests completed successfully for the Scala 2.12 JAR.

Known limitation

The core module does not currently provide a Spark 4 Maven build profile. Spark 4 plan-shape coverage should use a Spark-4-produced cache event log in the asserting qualification and profiling golden-set jobs.

[sayedbilalbari]

Thanks @parthosa , had some comments !
Can we also evaluate the memory pressure added with this and figure out if these changes need explicit integration tests to make sure the recommendation works for all spark versions.
The explicit RDD handling, are we doing that for any other AutoTuner rules ? If not, why and could that be done for other rules as well ?

[parthosa]

Thanks @parthosa , had some comments ! Can we also evaluate the memory pressure added with this and figure out if these changes need explicit integration tests to make sure the recommendation works for all spark versions. The explicit RDD handling, are we doing that for any other AutoTuner rules ? If not, why and could that be done for other rules as well ?

• After the refactor to use the single observedSqlCache boolean, plan and RDD traversal short circuit once cache evidence is found so memory use is constant.
• RDD inspection is specific to this feature. Generalizing for other rules could be a follow up investigation.
• Verified the test suit with different Spark version build profiles (3.4.4, 3.5.1 and 3.5.7). Also verified in the original event log that created this requirement (it was in Spark 4).
  • We do not have build profile for Spark 4. Will include event log in our CI testing.

Discussed offline, as part of a long term goal, will brainstorm to setup testing framework for AutoTuner across Spark versions, iceberg etc.

[sayedbilalbari]

THanks @parthosa , LGTM !

[sayedbilalbari]

@parthosa can we file a follow up for the target integration test suite for AutoTuner