polars-engine-cutover - #23
Merged
Merged
Conversation
Design for replacing the DuckDB engine with a pure-Rust Polars engine while preserving the existing CLI and SQL interface. Motivated by a real silent-hang failure in the duckdb crate on a malformed CSV field. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
DuckDB downloads httpfs/excel/azure extensions at runtime, which fails in no-binary-download environments — a second concrete argument for the pure-Rust Polars engine. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Bite-sized TDD plan to build and unit-test PolarsEngine (readers, SQL, schema, writers, compression, cloud guard) alongside the DuckDB engine, keeping the build green. Cutover is Phase 2. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add polars (lazy, sql, csv, parquet, json, strings, dtype-full), calamine, flate2, and zstd dependencies; create the empty PolarsEngine skeleton in src/polars_engine.rs; register it in main.rs. Also fix a pre-existing clippy::collapsible_match lint in config.rs that became a -D warnings failure after the new clippy run. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace wrap_compression (which silently swallowed finalize errors via Drop and panicked on zstd init) with write_with_optional_compression, which explicitly calls GzEncoder::finish() / zstd Encoder::finish() and maps errors to DtooError. Add Default impl for PolarsEngine and a test for the header=false CSV path. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the fixture-dependent `scan_excel_reads_existing_fixture` test with two self-contained tests that generate a tiny workbook at test time using `rust_xlsxwriter` (dev-dependency only). Removes the gitignored `testdata/xlsxs/trips.xlsx` dependency so the test suite passes on CI and fresh checkouts without manual fixture copying. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Regression test proving ragged CSV rows produce Err(DtooError::FileProcess) rather than a silent Ok or hang. Also adds .with_ignore_errors(false) to the LazyCsvReader builder to make the strict-parsing contract explicit and durable. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ng-sheet paths Calamine 0.35 pads all rows to the sheet's max width, so an over-wide data row silently extends the header row with Empty cells (empty strings) rather than raising an error. read_excel now detects trailing empty-string header columns — which only appear when data rows are wider than the user-declared headers — and returns DtooError::FileProcess instead of ingesting data under unnamed columns. Also adds four new tests: scan_excel_over_wide_row_errors (Fix 1), scan_excel_missing_named_sheet_errors, write_csv_to_stdout_sink_succeeds, and ndjson_gzip_output_is_decompressible. Suite grows from 16 to 20. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Slice-based plan to rewire query_pipeline + commands onto PolarsEngine, reimplement masking/lineage/schema/profiler/crypto as native Polars operations, then delete DuckDB. Existing behavioral tests are the regression safety net. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…DuckDB engine) Introduces src/types.rs with InputFormat, ExportFormat, CompressionCodec, and SchemaColumn — four engine-agnostic leaf types previously defined in engine.rs. All importers (engine.rs, polars_engine.rs, output_writer.rs, profile_command.rs, query_pipeline.rs, reference_tables.rs, schema.rs) now resolve these from crate::types. DuckDB engine unchanged; all 146 tests pass, clippy clean. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add `mask_dataframe` to apply deterministic sha256 column masking directly on a Polars DataFrame without going through DuckDB SQL. Existing `MaskingEngine` retained for the DuckDB pipeline until teardown. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add `apply_to_dataframe` to `LineageManager` that applies all requested lineage columns directly to a Polars DataFrame without going through DuckDB SQL. Existing `apply_columns` and `tag_rows_with_origin` are preserved for the DuckDB pipeline until teardown. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add duckdb_type_to_polars() mapping DuckDB type strings to Polars DataType, and coerce_to_schema() projecting a LazyFrame to declared columns with casts, NULL fill for missing columns, and dropping undeclared source columns. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add `load_reference_lazyframes` to `src/reference_tables.rs`, which scans each `ReferenceTable` into a `(name, LazyFrame)` pair via `PolarsEngine::scan`. The existing DuckDB `load_reference_tables` is untouched (still used by the DuckDB pipeline until teardown). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add `Profiler::generate(df, options)` that profiles a Polars DataFrame directly: per-column null%, distinct count, top-5 values, numeric quantiles/mean/stddev/median, string char-lengths and patterns, and date min/max. The original DuckDB-based path is preserved as `generate_from_engine` so query_pipeline and profile_command continue to compile until Task 9/12 completes the pipeline cutover. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add discover_wrapped_values_df, decrypt_dataframe, and encrypt_dataframe alongside the existing engine-based functions. The new DataFrame-based functions reuse all per-value crypto routines (CryptoScheme trait impls, detect_wrapped_value, strip_wrapper, wrap_inner) verbatim; only the column-iteration and column-write paths change from DuckDB SQL to Polars StringChunked/df.replace. Engine-based functions and enforce_output_safety are unchanged. 14 new tests cover both schemes, failure modes, and edge cases. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…itive; DECIMAL default (18,3) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add DataFrame-based write/write_and_get_destination methods on OutputWriter that delegate to PolarsEngine::write. Rename the existing DuckDB-backed methods to write_from_engine / write_and_get_destination_from_engine so the query pipeline build stays green until P2-9 switches the pipeline to the DataFrame path; the _from_engine variants will be deleted in P2-12. Port the three engine tests (write_uses_adjusted_output_path, errors_when_output_directory_missing, returns_effective_destination_path) to use PolarsEngine + df![]; keep the legacy DuckDB tests alongside them under _from_engine names. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
joefrost01
added a commit
that referenced
this pull request
Jun 7, 2026
origin/main (PR #23) was an earlier checkpoint of this same conversion that still ran on DuckDB. This branch carries the work to completion (DuckDB fully removed), so it supersedes it. Tree kept identical to the completed branch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem are you trying to solve?
What does this PR change?
Does this change align with DESIGN.md?
What alternatives did you consider?
Does this PR contain multiple unrelated changes?
Existing PRs
Testing
cargo testpassescargo clippypasses with no warningscargo fmthas been runEvaluation
Human review