Skip to content

Keep MAX buckets whose values are all non-positive#116

Open
PDGGK wants to merge 1 commit into
apache:masterfrom
PDGGK:fix/grouped-max-non-positive
Open

Keep MAX buckets whose values are all non-positive#116
PDGGK wants to merge 1 commit into
apache:masterfrom
PDGGK:fix/grouped-max-non-positive

Conversation

@PDGGK

@PDGGK PDGGK commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Problem

The milliseconds aggregation path projects the numeric aggregate over a COALESCE expression:

SELECT date_bin(<interval>ms, time, <startTs>) AS bucket_ts,
       MAX(COALESCE(double_v, CAST(long_v AS DOUBLE))) AS agg_num,
       ...
FROM telemetry
WHERE <identity predicates> AND time >= <start> AND time < <end>
GROUP BY 1

IoTDB's grouped max accumulator seeds FLOAT/DOUBLE state with Float.MIN_VALUE / Double.MIN_VALUE — the smallest positive values, not the most negative ones — and only marks a group initialized when value >= state. A bucket whose numeric maximum is zero or negative therefore never sets the flag and comes back as NULL. This is apache/iotdb#18300, fixed on master but not yet in any released version (the 2.0.11 cherry-pick is under discussion on dev@), so every release the module supports is affected.

aggregatedEntry reads a NULL numeric aggregate as "this bucket has no value" and returns null, and readMillisecondsAggregatedQuery skips such buckets. So a MAX downsampling query over telemetry that is always zero or negative — a sub-zero temperature sensor, a gauge that sits at 0 — silently returned fewer points, with no error raised anywhere.

Verified against the released apache/iotdb:2.0.8 and apache/iotdb:2.0.10 standalone images: a bucket of -5.0, -3.0 and a bucket of 0.0, 0.0 both return agg_num = NULL, while a positive bucket is correct.

Scope of the problem, also verified on both images:

  • MIN, AVG, SUM and COUNT are unaffected.
  • The calendar path is unaffected — it issues one bounded aggregate per bucket with no GROUP BY, and the non-grouped accumulators track an explicit initialized flag instead of a sentinel.
  • A long-only bucket is unaffected: it reads the direct MAX(long_v) channel, whose LongBigArray seeds with the true Long.MIN_VALUE.
  • A mixed long+double bucket is affected, because it reads the numeric channel.

Fix

Project -MIN(-x) instead of MAX(x) for the numeric channel. The grouped min accumulator seeds with MAX_VALUE and is not affected, and IEEE-754 negation is exact, so this returns the same value as MAX(x) over finite values, on servers with and without the upstream fix, and costs no extra query. The long and string channels are already correct — MAX(long_v) seeds with the true Long.MIN_VALUE, and the binary channel is flag-based — so they keep using MAX.

The projection is shared with the calendar path, so its SQL changes too. That path's non-grouped accumulators track an explicit initialized flag rather than a sentinel, so the substitution is exact there as well and an empty bucket still returns NULL and is still skipped.

Tests

Two integration tests against a real IoTDB container:

  • maxKeepsBucketsWhoseValuesAreAllNonPositiveAgainstRealIoTDB — buckets of all-negative and all-zero doubles keep their MAX (and MIN is unchanged).
  • maxOverNonPositiveLongOnlyAndMixedBucketsKeepsResultTypeAgainstRealIoTDB — covers the long-only channel and the mixed promotion, since they take different paths.

Both fail on the previous projection with bucket count expected: <2> but was: <0> and <1> respectively — the buckets disappear — and pass with the fix. The existing aggregation coverage only ever used positive doubles, which is why this was not caught earlier.

Full module build is green: 190 unit tests and 54 container integration tests.

The milliseconds aggregation path projected MAX over the numeric COALESCE
expression. IoTDB's grouped max accumulator seeds FLOAT/DOUBLE state with
Float/Double.MIN_VALUE -- the smallest positive value, not the most
negative one -- and only marks a group initialized when value >= state, so
a bucket whose numeric maximum is zero or negative comes back NULL. This
DAO reads a NULL aggregate as an empty bucket and skips it, so a MAX
downsampling query over, for example, a sub-zero sensor silently lost
whole buckets instead of failing.

Project -MIN(-x) instead: the grouped min accumulator seeds with MAX_VALUE
and is not affected, and IEEE-754 negation is exact, so the result is
identical over finite values on servers with and without the upstream fix.
The projection is shared with the calendar path, whose non-grouped
accumulators track an explicit initialized flag rather than a sentinel, so
the substitution is exact there too and an empty bucket still yields NULL.
The long and string channels are already correct and keep using MAX.

Verified against apache/iotdb 2.0.8 and 2.0.10 containers. See
apache/iotdb#18300, fixed on master but not yet in a released version.

Signed-off-by: Zihan Dai <99155080+PDGGK@users.noreply.github.com>
@PDGGK
PDGGK force-pushed the fix/grouped-max-non-positive branch from c9c9a72 to b1f761e Compare July 25, 2026 08:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant