[SkipRecovery][FEA] Support count on ANSI interval columns [databricks]

[wjxiz1992]

JaCoCo production line coverage: +9 lines (sql-plugin +9; fix-line coverage, Spark 3.3, vs 3859bcfca)

Fixes #15776.

Description

This is a GPU coverage and performance improvement, not a result-correctness fix. count(interval_column) already matched Spark CPU results, but the partial hash aggregate fell back to CPU because ANSI interval inputs were rejected at the operator boundary. This PR allows top-level day-time and year-month interval inputs for ordinary count, removing that fallback and its CPU/GPU transition. count(DISTINCT interval) intentionally remains on CPU because Spark implements it by grouping on the interval value, and interval grouping and hash partitioning are not supported on the GPU.

What changed

• Admit top-level ANSI interval inputs to GPU hash aggregation while continuing to reject interval grouping expressions.
• Recover the two issue-linked strict xfails and require GpuHashAggregateExec for ordinary interval counts.
• Add focused coverage proving that distinct interval counts retain the intended CPU fallback.
• Update the generated support documentation and all active support matrices to describe both interval families as partially supported.

AI assistance: The change and PR description were prepared with Codex assistance.

Validation

• Focused Python tests for day-time and year-month ordinary and distinct counts: 4 passed, 32409 deselected. Executed plans contained GpuHashAggregateExec for ordinary counts and the expected CPU aggregate for distinct counts.
• Spark 3.3 HashAggregatesSuite: Tests: succeeded 320, failed 0, canceled 0, ignored 0, pending 0; all 18 reactor modules succeeded; BUILD SUCCESS.
• Full hash_aggregate_test.py on the same affected production and test logic, before only comment/support-matrix generation and a non-overlapping rebase: 2154 passed, 78 skipped.
• JaCoCo analyzed 4,050 classes without a class-ID mismatch warning and covered 9 of 15 added sql-plugin production lines.

Performance benchmark

Measured on PR head 6eb6d1e9d with Spark 3.3.0 in local[8] mode on an NVIDIA RTX 5880 Ada Generation (48 GB), with one fixed concurrent GPU task and the RAPIDS memory pool capped at 30%. Each case read 1 million, 50 million, or 500 million ANSI interval values (10% nulls) from the same uncompressed Parquet input across 12 partitions and executed count(a); data generation was excluded from the timings. The PR path was compared with the same exact-head JAR using spark.rapids.sql.exec.HashAggregateExec=false, which reproduces the pre-PR operator boundary: the same GpuFileGpuScan is followed by GpuColumnarToRow and CPU partial/final HashAggregateExec instead of GPU partial/final aggregation.

Each mode received two warm-up runs followed by seven timed collect() runs in alternating A/B order. Values below are warm-cache wall-clock medians, with min-max ranges in parentheses; every run asserted a non-null count equal to 90% of the input rows.

Rows	Interval input	CPU aggregate fallback	PR GPU aggregate	Speedup
1 million	Day-time	0.081 s (0.069-0.098 s)	0.115 s (0.101-0.132 s)	0.70x
1 million	Year-month	0.064 s (0.057-0.073 s)	0.092 s (0.084-0.107 s)	0.69x
50 million	Day-time	0.423 s (0.388-0.455 s)	0.264 s (0.260-0.286 s)	1.60x
50 million	Year-month	0.393 s (0.356-0.467 s)	0.232 s (0.205-0.248 s)	1.69x
500 million	Day-time	3.480 s (3.450-3.512 s)	1.487 s (1.479-1.515 s)	2.34x
500 million	Year-month	2.586 s (2.520-2.682 s)	0.873 s (0.856-0.881 s)	2.96x

The result is scale-dependent: fixed GPU scheduling and exchange overhead dominates at 1 million rows, while removing the row conversion and CPU aggregate yields a 1.60-2.96x speedup at 50-500 million rows. This benchmark therefore supports the large analytic-workload benefit without claiming a small-query latency improvement.

Additional validation and performance detail

• Spark 3.4 and Spark 4.0/Scala 2.13 sql-plugin builds: all 8 reactor modules succeeded in each build; BUILD SUCCESS.
• Spark 3.3 exact-head distribution assembly: all 17 reactor modules succeeded; BUILD SUCCESS; the runtime JAR reported revision 6eb6d1e9d.
• Canonical and all 29 active versioned support matrices were regenerated; the Spark 3.3 tools generation completed with BUILD SUCCESS.
• git diff --check, Python bytecode compilation, shim coverage, and an exact-head four-reviewer local code review completed with 0 must-fix and 0 should-fix findings.
• The added grouping-expression check runs only during driver-side planning; there is no new executor row loop or GPU kernel work for existing types.

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Please provide the names of the existing tests in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[Copilot]

🟢 Approval recommended

The changes are narrowly scoped to HashAggregate planning/tagging, include explicit CPU fallback for unsupported interval grouping, and are backed by targeted Scala/Python validation plus regenerated support documentation.

Pull request overview

Enables GPU execution for count(interval_column) (both ANSI DayTime and YearMonth interval families) by extending HashAggregateExec type support while explicitly keeping ANSI-interval grouping on CPU (to preserve correctness given unsupported interval hash partitioning). Updates integration tests and generated support matrices/docs to reflect the new “partial support” boundary.

Changes:

• Extend HashAggregateExec input type signature to admit ANSI interval inputs, with support notes clarifying grouping limitations.
• Add GpuHashAggregateMeta tagging to force CPU fallback when ANSI interval types appear in grouping expressions (e.g., count(DISTINCT interval) plans).
• Add/adjust Python IT coverage for both the newly supported GPU count path and the intentional distinct-count fallback path; regenerate support docs/matrices.

File summaries

File	Description
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuExecOverrides.scala	Allows ANSI intervals as HashAggregate inputs and annotates them as PS due to grouping limitations.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuAggregateExec.scala	Tags HashAggregate plans to fall back to CPU when grouping by ANSI interval types.
integration_tests/src/main/python/hash_aggregate_test.py	Converts former xfails into strict validations for interval count, and adds distinct-count fallback tests.
docs/supported_ops.md	Updates the support table to mark ANSI intervals as partially supported for HashAggregate with grouping disallowed.
tools/generated_files/supportedExecs.csv	Regenerates the canonical exec support matrix to reflect PS for ANSI intervals on HashAggregate.
tools/generated_files/420/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/413/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/412/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/411/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/404/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/403/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/402/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/401/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/400/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/359/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/358/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/357/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/356/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/355/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/354/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/353/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/352/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/351/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/350/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/344/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/343/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/342/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/341/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/340/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/334/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/333/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/332/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/331/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.
tools/generated_files/330/supportedExecs.csv	Regenerated versioned exec support matrix reflecting PS for ANSI intervals on HashAggregate.

Review details

• Files reviewed: 34/34 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; no actionable correctness, fallback, test, or repository-rule issue remains.

Summary

• Broadens HashAggregateExec input eligibility to day-time and year-month intervals.
• Rejects hash-aggregate stages that group by ANSI interval values.
• Adds GPU-plan verification for ordinary interval counts and fallback verification for distinct counts.
• Regenerates support documentation and versioned support matrices.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[ANSI interval input] --> B{Aggregation shape}
  B -->|Ordinary count| C[GpuHashAggregateExec]
  B -->|Interval grouping or count distinct| D[CPU HashAggregateExec fallback]
  C --> E[Long count result]
  D --> E

Loading

_{Reviews (3) · Last reviewed commit: "Merge current main into interval count s..."}

[wjxiz1992]

build

[wjxiz1992]

build

[wjxiz1992]

build

[wjxiz1992]

build

[thirtiseven]

LGTM