Liquid-clustering boundary sampling

[sdrp713]

Fixes #15903 and fixes #15444

Description

Range-boundary sampling on very wide tables can create excessive GPU memory pressure and spill because the sampler carries payload columns that do not affect boundary calculation.

This PR optimizes GPU range-boundary sampling by reading only the columns required to calculate range keys, instead of materializing the complete input row.

The narrow boundary plan preserves deterministic filters and derived key dependencies. Unsupported plans and nondeterministic range expressions safely fall back to the existing full row path.

The collected boundaries are then applied to the original full row batches, so the final range partitioning and shuffle semantics remain unchanged.

Performance Results

The optimization was evaluated using a Delta Lake liquid-clustering workload consisting of:

1. A plain append with optimized writes and automatic compaction disabled.
2. An OPTIMIZE FULL operation on a table clustered by a column.
3. Range-boundary collection for 1,000 destination ranges.

The input contained:

• 5,887,163 rows
• 585 columns
• 30 input tasks
• Approximately 58.32 GiB read by the original full-row boundary scan

The following configuration was used for both runs:

• Spark 3.5.3
• Two g4dn.8xlarge GPU executors
• 4 executor cores
• 60 GiB executor memory
• spark.sql.files.maxPartitionBytes=2g
• 512 MiB target GPU batch size
• RAPIDS shuffle manager
• Optimized writes disabled
• Automatic compaction disabled

Only the range-boundary collection implementation changed between the two runs.

Stage metrics	Full-row sampling	Key-only sampling	Improvement
Wall-clock duration	363.478 s	3.399 s	106.9× faster
Aggregate executor runtime	2,752.704 s	23.303 s	118.1× lower
Input rows	5,887,163	5,887,163	Unchanged
Input read	58.32 GiB	4.10 MiB	Payload columns removed
Input batches	938	30	96.8% fewer
GPU decode time	272.929 s	0.331 s	824× lower
Host spill	83.78 GiB	0	Eliminated
Disk spill	15.48 GiB	0	Eliminated
Maximum task duration	134.815 s	1.111 s	121.3× lower

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Please provide the names of the existing tests in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge because no new actionable findings remain.

Summary

• Adds a pruned physical plan for collecting range boundaries from required columns.
• Falls back to full-row sampling for unsupported plans and nondeterministic expressions.
• Applies the collected boundaries to the original full-width batches.
• Adds tests for scan pruning, filter dependencies, payload preservation, and nondeterministic fallback.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Original full-width input plan] --> B{Can boundary plan be safely pruned?}
  B -->|Yes| C[Key-only boundary scan]
  B -->|No| D[Original full-row sampling]
  C --> E[Collect range boundaries]
  D --> E
  A --> F[Original full-width batches]
  E --> G[GPU range partitioner]
  F --> G
  G --> H[Range-partitioned shuffle]

Loading