Skip to content

feat(lance): push LIMIT down into Lance scans#5

Open
jja725 wants to merge 1 commit into
lance-type-struct-mappingfrom
lance-limit-pushdown
Open

feat(lance): push LIMIT down into Lance scans#5
jja725 wants to merge 1 commit into
lance-type-struct-mappingfrom
lance-limit-pushdown

Conversation

@jja725

@jja725 jja725 commented Jun 22, 2026

Copy link
Copy Markdown
Owner

Summary

Adds LIMIT pushdown to the Presto Lance connector. PrestoDB has no applyLimit SPI, so this uses a ConnectorPlanOptimizer.

  • Plan optimizer: LanceLimitPushdown rewrites a LimitNode sitting directly over a Lance TableScanNode into a LanceTableHandle carrying the row count (wired via LanceConnectorPlanOptimizerProviderLanceModuleLanceConnector.getConnectorPlanOptimizerProvider). Only fires when LIMIT is directly over the scan (no intervening FILTER), so semantics stay exact.
  • Split coalescing: with a pushed-down limit, the split manager coalesces just enough leading fragments to cover it into a single split (deletion-aware via Fragment.metadata().getNumRows()) instead of one-split-per-fragment.
  • Scan limit: the fragment page source passes ScanOptions.limit(n) to stop each scan early.

Best-effort IO reduction — the engine still enforces the exact row count.

Test plan

  • New unit tests: TestLanceLimitPushdown (pushes into scan, idempotent), TestLanceSplitManager (split-per-fragment vs single coalesced split + coalesce helper edge cases), additions to TestLanceTableHandle (limit field + JSON round-trip).
  • ./mvnw -o test -pl presto-lance37 tests, BUILD SUCCESS.

Stacked on #4 — review/merge that first. The diff here is just the LIMIT-pushdown changes.

🤖 Generated with Claude Code

PrestoDB has no applyLimit SPI, so add a ConnectorPlanOptimizer that
rewrites LIMIT directly above a Lance TableScan into a LanceTableHandle
carrying the row count (LanceLimitPushdown +
LanceConnectorPlanOptimizerProvider, wired through LanceModule and
LanceConnector.getConnectorPlanOptimizerProvider).

With a pushed-down limit the split manager coalesces just enough leading
fragments to cover it into a single split (deletion-aware via
Fragment.metadata().getNumRows()) instead of one-split-per-fragment, and
the fragment page source passes ScanOptions.limit(n) to stop each scan
early. The engine still enforces the exact count, so this is a best-effort
IO reduction. Only fires when LIMIT sits directly over the scan (no
intervening FILTER), keeping semantics exact.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant