Skip to content

Report per-stage distribution statistics for GPU task metrics - #2146

Merged
amahussein merged 2 commits into
NVIDIA:devfrom
amahussein:cudf-tools-2131
Sep 9, 2026
Merged

amahussein merged 2 commits into
NVIDIA:devfrom
amahussein:cudf-tools-2131

Conversation

@amahussein

Copy link
Copy Markdown
Collaborator

fixes #2131

Adds sampleCount, min, stddev, cv and maxOverMean to the GPU task metric reports at stage, SQL and application level. Dispersion comes from a second moment kept in the store rather than from retained per-task values: StatisticsMetrics carries welfordSumSqDev, updated in constant time per sample by Welford's method and merged across stages by Chan's parallel form, so cost is per (accumulable, stage) rather than per task.

The statistics describe the task updates alone. StatisticsMetrics gains sampleTotal, so a value a stage reports at completion moves only the published total while count, sampleTotal and welfordSumSqDev keep describing one population whatever order the events arrive in. min and max are seeded from the first sample instead of folded against the placeholder zeros a stage completion event leaves behind, and the reports read them through sampleMin and sampleMax, so a stage no task reported publishes blank extrema rather than 0.

The three GPU row classes share AggregatedMetricStats, which derives sum, avg, stddev, cv and maxOverMean from the stored fields and divides out the metric's fixed-point scale once. ProfileUtils.optionLongToString is replaced by a generic StringUtils.optionToString, and ToolUtils.formatDoublePrecision is repaired: it used scientific notation and floored instead of rounding.

The avg was integer divided and floored while the report contract declared it Double. Compute and render it as a rounded Double.

The report contract for the three GPU tables is updated, including the sampled population, which is task attempts rather than logical tasks.

StatisticsMetricsSuite covers the store lifecycle and the cross-stage merge against a two-pass reference. GpuMetricReportSuite drives the analyzer over a hand-built event log covering gpuMaxTaskFootprint and a stage whose metric arrives only at completion, neither of which any stored fixture provides.

fixes NVIDIA#2131

Adds sampleCount, min, stddev, cv and maxOverMean to the GPU task metric reports at stage, SQL and application level. Dispersion comes from a second moment kept in the store rather than from retained per-task values: StatisticsMetrics carries welfordSumSqDev, updated in constant time per sample by Welford's method and merged across stages by Chan's parallel form, so cost is per (accumulable, stage) rather than per task.

The statistics describe the task updates alone. StatisticsMetrics gains sampleTotal, so a value a stage reports at completion moves only the published total while count, sampleTotal and welfordSumSqDev keep describing one population whatever order the events arrive in. min and max are seeded from the first sample instead of folded against the placeholder zeros a stage completion event leaves behind, and the reports read them through sampleMin and sampleMax, so a stage no task reported publishes blank extrema rather than 0.

The three GPU row classes share AggregatedMetricStats, which derives sum, avg, stddev, cv and maxOverMean from the stored fields and divides out the metric's fixed-point scale once. ProfileUtils.optionLongToString is replaced by a generic StringUtils.optionToString, and ToolUtils.formatDoublePrecision is repaired: it used scientific notation and floored instead of rounding.

The avg was integer divided and floored while the report contract declared it Double. Compute and render it as a rounded Double.

The report contract for the three GPU tables is updated, including the sampled population, which is task attempts rather than logical tasks.

StatisticsMetricsSuite covers the store lifecycle and the cross-stage merge against a two-pass reference. GpuMetricReportSuite drives the analyzer over a hand-built event log covering gpuMaxTaskFootprint and a stage whose metric arrives only at completion, neither of which any stored fixture provides.

Signed-off-by: Ahmed Hussein (amahussein) <a@ahussein.me>
@amahussein amahussein self-assigned this Sep 8, 2026
@amahussein amahussein added the api_change A change affecting the output (add/remove/rename files, add/remove/rename columns) label Sep 8, 2026
@github-actions github-actions Bot added the core_tools Scope the core module (scala) label Sep 8, 2026
Signed-off-by: Ahmed Hussein (amahussein) <a@ahussein.me>
@greptile-apps

greptile-apps Bot commented Sep 9, 2026

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no actionable correctness, security, or repository-rule violations were identified.

Summary

  • Tracks task-attempt sample totals and Welford deviation sums independently from stage-completion totals.
  • Pools variance across stages using Chan’s parallel merge formula.
  • Adds sample count, minimum, standard deviation, coefficient of variation, and max-over-mean columns.
  • Corrects average calculation and decimal rendering, with expanded store, report, and formatter tests.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  TaskEnd[TaskEnd accumulator update] --> Welford[Per-stage Welford state]
  StageCompleted[StageCompleted accumulator value] --> Published[Published stage total]
  Welford --> StageRow[GPU stage metric row]
  Published --> StageRow
  StageRow --> ChanSQL[Chan merge by SQL]
  StageRow --> ChanApp[Chan merge by application]
  ChanSQL --> SQLRow[GPU SQL metric row]
  ChanApp --> AppRow[GPU application metric row]
  StageRow --> Stats[avg / min / max / stddev / cv / maxOverMean]
  SQLRow --> Stats
  AppRow --> Stats
Loading

@amahussein
amahussein merged commit 36c1c56 into NVIDIA:dev Sep 9, 2026
18 checks passed
@amahussein
amahussein deleted the cudf-tools-2131 branch September 9, 2026 12:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api_change A change affecting the output (add/remove/rename files, add/remove/rename columns) core_tools Scope the core module (scala)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEA] Report per-stage distribution statistics for GPU task metrics

1 participant