Deferred from #2129, the downward-only shuffle partition pass. Split out so #2128 can close on its own problem. None of these block turning the pass on: each can only withhold a recommendation or size it larger than needed, never produce a wrong one.
- Widen the OOM gate to failure reasons. It currently sees only GPU OOM in scan stages and YARN container kills during GPU shuffle. Heap OOM, host allocation failures, GPU OOM outside scans and non-YARN kills all miss it, and qualification gets no signal at all. In practice the blanket failed-stage gate catches these, since any OOM that kills a stage attempt trips it — the residual is a task-level OOM fully absorbed by retries. Task, stage and job failure reasons are already parsed and would cover it properly. Touches gates beyond this pass.
The two below are TODO(#2133) in ShuffleStageInputAnalyzer.scala.
- Size a CPU exchange in a GPU run. Today it marks the application incomplete and the pass skips. Blocked on
DOWNWARD_SHUFFLE_INPUT_SIZE_FACTOR being application-level: one run can't apply 1.0 to its GPU exchanges and the CPU factor to its plain ones. Making the factor stage-level is the prerequisite.
- Apportion an AQE-split exchange instead of duplicating it. The full exchange size is attributed to every split stage, overstating each. Safe direction, but loose. Splitting
data size by per-stage shuffle read metrics would tighten it, most on skewed joins.
Target-size reconciliation stays in #2128 — that one does gate turning the pass on.
Deferred from #2129, the downward-only shuffle partition pass. Split out so #2128 can close on its own problem. None of these block turning the pass on: each can only withhold a recommendation or size it larger than needed, never produce a wrong one.
The two below are
TODO(#2133)inShuffleStageInputAnalyzer.scala.DOWNWARD_SHUFFLE_INPUT_SIZE_FACTORbeing application-level: one run can't apply 1.0 to its GPU exchanges and the CPU factor to its plain ones. Making the factor stage-level is the prerequisite.data sizeby per-stage shuffle read metrics would tighten it, most on skewed joins.Target-size reconciliation stays in #2128 — that one does gate turning the pass on.