DuckDB Query Streaming with Cursor Pagination#287
Draft
etiennechabert wants to merge 20 commits into
Draft
Conversation
Added streaming query support to DuckDB worker: - New isStreamingQueryRequest type guard for 'streaming-query' messages - handleStreamingRequest async function that uses fetchRowsChunked - Sends incremental 'chunk' messages with hasMore flag - Follows same cancellation/error handling pattern as handleRequest - Hooked up in message listener Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Added queryStreaming method to DuckDBClient interface that accepts SQL, onChunk callback, and optional onStarted callback - Extended WorkerResponse type union with 'chunk' message type containing rows, hasMore flag - Updated isWorkerResponse type guard to validate 'chunk' messages - Created PendingStreamingQuery interface for streaming request state management - Modified message handler to process 'chunk' messages and invoke callback incrementally - Final chunk (hasMore=false) resolves the promise, intermediate chunks only invoke callback - Added comprehensive test coverage for streaming queries with multiple chunks - All tests pass (3/3 passing)
…y method Added runStreamingQuery method to AppContext interface and implementation: - Interface: readonly runStreamingQuery(sql, onChunk, onStarted?) => Promise<void> - Implementation: delegates to ctx.db.queryStreaming for chunk-based streaming - Follows same pattern as runQuery and runPreparedQuery methods - Enables handlers to use streaming queries alongside buffered queries Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Add parseCursor() to decode base64-encoded offset cursors - Add encodeCursor() to create base64 cursor strings - Add clampPageSize() to bound page sizes to valid range [1, maxSize] - All utilities follow query-utils.ts patterns with proper error handling - Comprehensive test suite with 16 tests covering happy paths and edge cases Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…or pagination Added cursor-based pagination to explorer:query-rows handler: - Parse cursor from params using parseCursor() to get offset - Clamp page size (default 1000, max 2000) with clampPageSize() - Add OFFSET clause to SQL query - Calculate hasMore based on offset + limit < totalRows - Return next cursor (base64-encoded offset) when hasMore=true - Maintain backward compatibility with rowLimit parameter Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Implemented usePaginatedQuery hook for cursor-based pagination - Hook accumulates data across pages with loadMore function - Follows discriminated union pattern from use-query.ts - Returns PaginatedQueryState with status, data, hasMore, and loadMore - Handles errors, retries on cancellation, and resets on deps change - Comprehensive test coverage (13 tests) for all scenarios - All type checks and tests pass
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…ncement) Implemented infinite scroll using IntersectionObserver with a sentinel element. When the user scrolls within 200px of the bottom, more rows are automatically loaded without requiring a manual button click. Features: - Sentinel element placed before "Load More" button triggers auto-load - 200px rootMargin provides smooth preloading before reaching the end - State transitions naturally prevent duplicate loads - Manual "Load More" button remains as fallback for accessibility Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Add maxRows parameter to WidgetQueryArgs interface - Default to 1000 for daily widget queries (charts) - Default to 500 for cost widget queries (tables) - Pass maxRows through to fetch functions (prepared for backend support) - Include maxRows in dependency arrays for proper re-fetching Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Add truncation warning when table hits ROW_LIMIT (500 rows) - Display warning banner showing "top N of M rows" with suggestion to use Explorer - Clean up unused maxRows parameter from use-widget-query hooks Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…orer queries - Created comprehensive integration test suite for Explorer pagination - Tests cursor encoding/decoding round-trip - Verifies LIMIT/OFFSET behavior for first and subsequent pages - Validates total count matches accumulated paginated results - Tests hasMore flag calculation on first and last pages - Verifies cursor progression across multiple pages - Tests pagination with filters applied - Handles empty result sets correctly - Uses buildSource from query builder for correct Parquet path structure - All 11 tests passing Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Fixes: - E2E tests now configure both config AND data directories - Updated helpers.ts to accept dataDir override in launchApp() - Added preference file copying to both E2E tests - Fixed explorer-preferences.json date range to match fixture data (2026-01-01 to 2026-02-28) - Updated memory profile test to use fixture config in all cases (not just CI) - Fixed Explorer navigation to check for description text instead of heading Verified: - npm run check passes (533/538 tests passing) - npx playwright test e2e/explorer-pagination.test.ts passes (3/3 tests) - npm run e2e:memory runs successfully (2/3 tests pass, memory test shows 0% reduction due to both baseline and paginated using pagination) QA Fix Session: 1 Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…qa-requested) Fixes: - Remove unused streaming infrastructure (pagination alone is sufficient) - Delete fetchRowsChunked, handleStreamingRequest, isStreamingQueryRequest from duckdb-worker.ts - Delete queryStreaming, PendingStreamingQuery, submitStreamingQuery from duckdb-client.ts - Delete runStreamingQuery from AppContext in context.ts - Delete streaming test suite (duckdb-client.test.ts) - Remove 'chunk' message type from worker protocol - Fix memory test to compare same data loads - Remove premature waitForQuerySettle (was measuring idle state) - Make both scenarios load 10 pages (was 10 vs 1) - Start monitoring before page loads to capture peak memory Verified: - All tests pass (530/530) - Type checking passes - grep confirms no streaming code remains QA Fix Session: 2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implement cursor-based pagination for large Explorer queries to reduce peak memory consumption. Add configurable LIMIT/OFFSET clauses to SQL for dashboard widgets with sensible defaults. Stream large result sets from the DuckDB worker thread to the renderer in chunks rather than buffering the entire result set in memory. Show incremental results as they arrive with a 'load more' or infinite scroll pattern.