Skip to content

Fix duration minimum for zero-task stages - #2154

Merged
amahussein merged 1 commit into
NVIDIA:devfrom
amahussein:tools-2151-duration-min-zero-tasks
Sep 10, 2026
Merged

amahussein merged 1 commit into
NVIDIA:devfrom
amahussein:tools-2151-duration-min-zero-tasks

Conversation

@amahussein

Copy link
Copy Markdown
Collaborator

Fixes #2151.

A stage that produced no task records is held in memory with numTasks=0 and zeroed extrema, so TaskMetricsAccumRec.finalizeAggregation leaves durationMin=0. Two later folds read that synthetic zero as an observed task duration: the cached-record fold in TaskMetricsAccumRec.addRecord, which builds the job and SQL aggregates, and StageAggTaskMetricsProfileResult.aggregateStageProfileMetric, which merges stage attempts. On the stored gpu_oom_eventlog fixture this made SQL 24 report duration_min=0 while its task-bearing stages had minima of 3085, 4374, and 4760 ms.

Both folds now key on task count rather than value, because 0 remains a legitimate duration for a task that ran. The record fold skips a zero-task record, and the attempt merge takes the populated operand's minimum when exactly one side is empty.

A standalone zero-task stage row also publishes an empty duration_min instead of 0. QualX pools stage rows by SQL and reduces _min columns with pandas min; it recomputes duration_mean from pooled sums but cannot reconstruct a minimum, so retaining numeric 0 would continue feeding the model a false value. For QualX owners: the shipped XGBoost models use duration_min, so predictions can change for affected applications; an all-empty group now supplies a missing value instead of a synthetic 0.

duration_max and duration_avg deliberately keep their zero values on an empty row. Zero never wins a maximum, and the mean is recomputed downstream from pooled sums and counts, so the minimum is the only sentinel that propagates.

Tests cover the stored fixture through SQL aggregation and cover stage-attempt merging with populated, empty-left, empty-right, and both-empty operands. Full verification passed on Scala 2.12 and 2.13.

Ignore empty stage records when aggregating task duration minima across SQLs, jobs, and stage attempts. Render the undefined minimum as an empty cell on zero-task rows so downstream minimum reductions do not treat the sentinel as a measurement.

Add fixture and record-level coverage for the SQL fold and both stage-attempt operand orders.

Signed-off-by: Ahmed Hussein (amahussein) <a@ahussein.me>
@amahussein amahussein self-assigned this Sep 10, 2026
@amahussein amahussein added bug Something isn't working core_tools Scope the core module (scala) labels Sep 10, 2026
@greptile-apps

greptile-apps Bot commented Sep 10, 2026

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

The PR appears safe to merge; no actionable correctness, security, or repository-rule issues were identified.

Summary

  • Ignores zero-task records when folding cached stage metrics into job and SQL aggregates.
  • Merges stage-attempt minima according to task presence while preserving legitimate zero-duration tasks.
  • Serializes an undefined minimum as an empty value for zero-task stage rows.
  • Adds fixture-level SQL aggregation coverage and stage-attempt merge coverage.

Reviews (1) · Last reviewed commit: "Fix duration minimum for zero-task stage..."

@amahussein
amahussein merged commit 2073060 into NVIDIA:dev Sep 10, 2026
19 checks passed
@amahussein
amahussein deleted the tools-2151-duration-min-zero-tasks branch September 10, 2026 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working core_tools Scope the core module (scala)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] duration_min reports 0 when a stage produced no tasks

1 participant