Skip to content

[BUG] duration_avg averages the averages when a stage runs more than once #2145

Description

@amahussein

Describe the bug

StageAggTaskMetricsProfileResult.aggregateStageProfileMetric merges the per-attempt rows of a stage that ran more than once. Every field merges by addition, maximum or minimum, except one:

durationAvg = (this.durationAvg + other.durationAvg) / 2,

That is an unweighted mean of two means, so it ignores how many tasks each attempt contributed. A stage whose first attempt runs 100 tasks averaging 10 ms and whose retry runs 1 task at 1000 ms has a true average of about 19.8 ms; the merge publishes 505 ms. The error grows with the imbalance between attempts, which is the normal shape of a retry.

Steps/Code to reproduce bug

Profile an event log in which a stage has a Stage Attempt ID above 0, and compare duration_avg in stage_level_aggregated_task_metrics.csv against duration_sum divided by num_tasks for that stage. No event log under core/src/test/resources has a stage attempt above 0, so the merge path is unexercised and this is not reproducible from the repository fixtures alone.

Expected behavior

The pooled duration over the pooled task count, which TaskMetricsAccumRec.finalizeAggregation already computes for a single attempt:

durationAvg = ToolUtils.calculateAverage(durationSum, numTasks, 1)

durationSum and numTasks are merged correctly in the same expression, so the fix is to compute the average from them there.

Additional context

Scope is one column in one file: durationAvg is the only averaged field on this result, and job and SQL levels recompute from pooled sums rather than inheriting the merged value. A retried-stage fixture is worth adding with the fix.

Same defect class as the GPU averages corrected in #2139. Found while auditing stage-attempt handling for #2131, which is unaffected.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingcore_toolsScope the core module (scala)

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions