Skip to content

Follow-ups from the downward shuffle partition pass (#2129) #2133

Description

@parthosa

Deferred from #2129, the downward-only shuffle partition pass. Split out so #2128 can close on its own problem. None of these block turning the pass on: each can only withhold a recommendation or size it larger than needed, never produce a wrong one.

  • Widen the OOM gate to failure reasons. It currently sees only GPU OOM in scan stages and YARN container kills during GPU shuffle. Heap OOM, host allocation failures, GPU OOM outside scans and non-YARN kills all miss it, and qualification gets no signal at all. In practice the blanket failed-stage gate catches these, since any OOM that kills a stage attempt trips it — the residual is a task-level OOM fully absorbed by retries. Task, stage and job failure reasons are already parsed and would cover it properly. Touches gates beyond this pass.

The two below are TODO(#2133) in ShuffleStageInputAnalyzer.scala.

  • Size a CPU exchange in a GPU run. Today it marks the application incomplete and the pass skips. Blocked on DOWNWARD_SHUFFLE_INPUT_SIZE_FACTOR being application-level: one run can't apply 1.0 to its GPU exchanges and the CPU factor to its plain ones. Making the factor stage-level is the prerequisite.
  • Apportion an AQE-split exchange instead of duplicating it. The full exchange size is attributed to every split stage, overstating each. Safe direction, but loose. Splitting data size by per-stage shuffle read metrics would tighten it, most on skewed joins.

Target-size reconciliation stays in #2128 — that one does gate turning the pass on.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions