Currently, the AutoTuner recommends 10m for spark.rapids.sql.reader.multithreaded.combine.sizeBytes as compared to the plugin default. We should update the heuristic to:
- Introduce a rule for updating the metric
- Disable recommendation (so that plugin default gets used)
Ref: https://nvidia.github.io/cudf-spark/docs/additional-functionality/advanced_configs.html
Currently, the AutoTuner recommends
10mforspark.rapids.sql.reader.multithreaded.combine.sizeBytesas compared to the plugin default. We should update the heuristic to:Ref: https://nvidia.github.io/cudf-spark/docs/additional-functionality/advanced_configs.html