Autotune spark.executor.pyspark.memory from Python OOM telemetry

[parthosa]

Summary

AutoTuner now recommends spark.executor.pyspark.memory after a recognized Python memory-limit failure.

• Uses max of the p95 ProcessTreePythonVMemory target with 1.25x headroom and the current limit grown by 1.5x, rounded up to GiB. An absent or zero current limit produces guidance only.
• Spark versions: Uses process-tree evidence on Spark 3.5.7 and later, except Spark 4.0.0. Versions affected by SPARK-52776 use the retry-growth fallback with a warning.
• Memory budget: Transfers the increase atomically from heap by default, or from overhead when explicitly selected, without increasing the modeled executor allocation. Conflicts produce no partial recommendation.
• Telemetry: Recommendations remain gated by PYSPARK_MEMORY_RECOMMEND_TELEMETRY_CONFIGS, which defaults to false, and do not require target-cluster sizing.

Validation

• Focused PySpark memory and AutoTuner suites: 140 passed
• Spark 3.5.7 package build
• Profiling validation with a telemetry-enabled event log

Fixes #2121

[SYangster]

Addressed fixes (0dc87190):

• No-op retry prevented: target is now max(p95-based, 1.5x growth floor)
• Spark version gate: evidence trusted for 3.5.7–3.5.x and 4.0.1+; others get 1.5x fallback + warning
• Atomic output: both PySpark and source-memory sides validated before emitting; partial recs blocked
• Telemetry guidance emitted independently of cluster sizing when opted in

Acknowledged follow-ups:

• p95 can miss the failure-driving tail; censored allocations and sampling gaps remain unaddressed
• Per-worker VMemory (vs. executor aggregate), daemon exclusion, and skew-aware analysis
• JVM GC / spill / native-memory safety gates before heap/overhead transfers
• Parse allocation size from NumPy/Arrow errors
• Cross-workload calibration of heuristic constants
• Regression tests across historical workloads

Verified:

Tested opted-in telemetry path with a real PySpark OOM eventlog. Expected settings appear correctly in tuning_summary.log.