Describe the bug
StageAggTaskMetricsProfileResult.aggregateStageProfileMetric merges the per-attempt rows of a stage that ran more than once. Every field merges by addition, maximum or minimum, except one:
durationAvg = (this.durationAvg + other.durationAvg) / 2,
That is an unweighted mean of two means, so it ignores how many tasks each attempt contributed. A stage whose first attempt runs 100 tasks averaging 10 ms and whose retry runs 1 task at 1000 ms has a true average of about 19.8 ms; the merge publishes 505 ms. The error grows with the imbalance between attempts, which is the normal shape of a retry.
Steps/Code to reproduce bug
Profile an event log in which a stage has a Stage Attempt ID above 0, and compare duration_avg in stage_level_aggregated_task_metrics.csv against duration_sum divided by num_tasks for that stage. No event log under core/src/test/resources has a stage attempt above 0, so the merge path is unexercised and this is not reproducible from the repository fixtures alone.
Expected behavior
The pooled duration over the pooled task count, which TaskMetricsAccumRec.finalizeAggregation already computes for a single attempt:
durationAvg = ToolUtils.calculateAverage(durationSum, numTasks, 1)
durationSum and numTasks are merged correctly in the same expression, so the fix is to compute the average from them there.
Additional context
Scope is one column in one file: durationAvg is the only averaged field on this result, and job and SQL levels recompute from pooled sums rather than inheriting the merged value. A retried-stage fixture is worth adding with the fix.
Same defect class as the GPU averages corrected in #2139. Found while auditing stage-attempt handling for #2131, which is unaffected.
Describe the bug
StageAggTaskMetricsProfileResult.aggregateStageProfileMetricmerges the per-attempt rows of a stage that ran more than once. Every field merges by addition, maximum or minimum, except one:That is an unweighted mean of two means, so it ignores how many tasks each attempt contributed. A stage whose first attempt runs 100 tasks averaging 10 ms and whose retry runs 1 task at 1000 ms has a true average of about 19.8 ms; the merge publishes 505 ms. The error grows with the imbalance between attempts, which is the normal shape of a retry.
Steps/Code to reproduce bug
Profile an event log in which a stage has a
Stage Attempt IDabove 0, and compareduration_avginstage_level_aggregated_task_metrics.csvagainstduration_sumdivided bynum_tasksfor that stage. No event log undercore/src/test/resourceshas a stage attempt above 0, so the merge path is unexercised and this is not reproducible from the repository fixtures alone.Expected behavior
The pooled duration over the pooled task count, which
TaskMetricsAccumRec.finalizeAggregationalready computes for a single attempt:durationSumandnumTasksare merged correctly in the same expression, so the fix is to compute the average from them there.Additional context
Scope is one column in one file:
durationAvgis the only averaged field on this result, and job and SQL levels recompute from pooled sums rather than inheriting the merged value. A retried-stage fixture is worth adding with the fix.Same defect class as the GPU averages corrected in #2139. Found while auditing stage-attempt handling for #2131, which is unaffected.