Skip to content

[BUG] SQL accumulator minimum can report a synthetic zero #2155

Description

@amahussein

Describe the bug

AccumInfo.addAccumToStage initializes a missing stage from StatisticsMetrics.ZERO_RECORD. When a stage has a completion value but no task-level accumulator updates, only total advances, leaving count = 0 and the placeholder min = 0. AccumInfo.calculateAccStats later combines all stage records with an unconditional Math.min, so that placeholder can become the reported minimum for a multi-stage accumulator. sql_plan_metrics_for_application.csv publishes the minimum without the internal sample count, making a synthetic zero indistinguishable from an observed zero.

Source inspection and an independent review confirm this mechanism. Practical incidence is unmeasured. No event log currently reproduces it: the zero-task stages in the fixture used for issue #2151 have no accumulables, so they never enter this path.

Steps/Code to reproduce bug

No runnable reproduction has been constructed yet. The source-derived record-level case to add is:

  1. Add a completion value for one stage through AccumInfo.addAccumToStage, with no task update for that stage.
  2. Add a positive task update for a second stage through AccumInfo.addAccumToTask.
  3. Call AccumInfo.calculateAccStats.
  4. Observe that the combined min is 0 instead of the sampled stage's positive minimum.

Expected behavior

A stage with count = 0 should not contribute to a minimum over task samples. When any sampled stage exists, the result should use the minimum over sampled stages. The owner should decide whether an all-empty result should render blank, following the precedent for duration_min, and whether the output should expose sample count alongside the minimum.

Environment details (please complete the following information)

  • Environment location: Source-level analysis only; no reproducing environment or event log is available yet.

Additional context

PR #2154 fixed the same sentinel-as-observation pattern for task duration by excluding zero-task records and rendering an all-empty minimum as blank. This is an independent follow-up in a different class, accumulator, output file, and ingest path: accumulables rather than TaskModel. QualX does not consume this file's min column, so no current model-feature impact is known.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingcore_toolsScope the core module (scala)

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions