polars-engine-cutover

Cargo.toml

@@ -22,10 +22,17 @@ aes = "0.8"
 ecb = "0.1"
 sha1 = "0.10"
 base64 = "0.22"
+polars = { version = "0.54.4", features = ["lazy", "sql", "csv", "parquet", "json", "strings", "dtype-full", "diagonal_concat"] }
+calamine = "0.35.0"
+flate2 = "1.1.9"
+zstd = "0.13.3"
 
 [profile.release]
 opt-level = 3
 lto = true
 codegen-units = 1
 strip = true
 panic = "abort"
+
+[dev-dependencies]
+rust_xlsxwriter = "0.95.0"

docs/specs/34-polars-engine.md

@@ -0,0 +1,168 @@
+# Polars Engine (replaces DuckDB Engine)
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:writing-plans to turn this design into an implementation plan, then superpowers:subagent-driven-development or superpowers:executing-plans to implement it.
+
+**Goal:** Replace the in-memory DuckDB engine (spec 03) with a Polars-based engine that powers the same file reading, SQL execution, and data accumulation — while preserving dtoo's existing CLI and SQL interface.
+
+**Supersedes:** `docs/specs/03-duckdb-engine.md`.
+
+**Tech Stack:** Rust, `polars` crate (pure Rust), `polars-sql` (`SQLContext`).
+
+**Tier:** Open Source
+
+---
+
+## Motivation
+
+This is not a speculative rewrite. The driving problem is concrete and was experienced in real use:
+
+> While trialling dtoo at work, the `duckdb` crate **silently hung** on a malformed field in a test CSV and never propagated an error back to Rust. Half a day was lost to suspected linking issues before the DuckDB CLI revealed it was a bad CSV field. A tool that can hang indefinitely with no error is not shippable to users.
+
+The fix is an engine whose failures are ordinary Rust `Result`s. Polars is pure Rust: read errors, malformed-row errors, and unsupported-SQL errors all return `PolarsError` with useful messages — **no opaque C++ hangs**. The secondary benefit is a pure-Rust build with no bundled C++ toolchain.
+
+A second, environment-specific reason: dtoo's target deployment is a locked-down
+corporate setting where **binaries cannot be downloaded** — everything builds from
+source. DuckDB loads its `httpfs`, `excel`, and `azure` extensions by downloading
+them at runtime (`INSTALL …`), so the Excel and cloud paths would fail there
+regardless of the hang. Polars compiles Excel support in (pure-Rust `calamine`) and
+needs no runtime extension fetch.
+
+## Scope
+
+**In scope (this conversion):** the open-source command surface that exists today — `query`, `convert`, `profile`, `inspect`, `fingerprint`, plus crypto profiles, masking, lineage, reference tables, schema handling, output writing, manifests, fingerprinting.
+
+**Deferred (follow-up specs):** cloud storage (S3/GCS/Azure). Polars supports it, but the `aws`/`gcp`/`azure` features pull in `object_store` + `tokio` + a TLS stack — a heavy async tree that works against the lean pure-Rust goal. We ship the local-file core first, then add cloud back once the engine swap is proven.
+
+**Out of scope:** the Pro/Enterprise features that are spec-only and unimplemented (assertions, sinks, partitioning, pipelines, audit log, webhooks, parallel). Their specs are unaffected by this change and will be re-evaluated when built.
+
+## Architectural Shift
+
+Today, **every** stage mutates a hidden `temp_results` table by issuing SQL strings through `DuckDbEngine` — masking, lineage, limit, crypto, profiling, and schema evolution all generate SQL (including DuckDB-only functions like `hmac()`). The data is never a value; it is a side effect inside the connection.
+
+The new model carries the data **as a value** — a Polars `LazyFrame` — through the pipeline. Only the *user-facing* SQL stays SQL.
+
+```
+files ── scan_csv / scan_parquet / scan_ndjson (glob native) ──▶ Vec<LazyFrame>
+   └─ concat_lf_diagonal(..)        // == today's "UNION ALL BY NAME" schema evolution
+        │
+   USER SQL via SQLContext:
+        register("_", lf); register each --ref; execute(--where / --filter-sql / --post-sql)
+        │
+   INTERNAL stages as native Polars expressions (NO generated SQL):
+        masking  → existing HMAC-SHA256 Rust code, applied via with_column
+        lineage  → with_columns(batch_id / record_id / batch_timestamp / batch_hash / origin_file)
+        limit    → .limit(n)
+        crypto   → existing AES/ECB Rust code applied via expressions
+        │
+   .collect() ──▶ DataFrame ──▶ CsvWriter / ParquetWriter / JsonWriter (file or stdout)
+```
+
+This is cleaner than the current design: internal transforms become honest, typed Rust operations instead of generated SQL, and they no longer depend on DuckDB-only SQL functions.
+
+## Engine API
+
+Replace `DuckDbEngine` with `PolarsEngine`. Rather than `execute(sql)` against a mutable table, the engine exposes value-passing operations:
+
+- `scan(path, format) -> Result<LazyFrame>` — readers for Parquet / CSV (custom delimiter) / NDJSON / Excel (sheet by name).
+- `concat_by_name(frames) -> Result<LazyFrame>` — union-by-name with type coercion (`concat_lf_diagonal`).
+- `run_sql(lf, refs, sql) -> Result<LazyFrame>` — register `_` and refs in a `SQLContext`, execute, return the resulting `LazyFrame`. Surfaces unsupported-SQL as a clean error.
+- `schema(lf) -> Result<Schema>` — `collect_schema()` without materializing (for dry-run, schema mode, profiling).
+- `collect(lf) -> Result<DataFrame>` and `row_count(lf) -> Result<usize>`.
+- `write(df, dest, format, header, delimiter, compression) -> Result<()>` — writers, to a file path or stdout, wrapping the writer in `flate2`/`zstd` for CSV/NDJSON compression.
+
+The `_` magic table and reserved names (`_`, `temp_results`) are preserved exactly as today.
+
+## Stage Mapping (DuckDB → Polars)
+
+| Stage | Today (DuckDB SQL) | New (Polars) |
+|-------|--------------------|--------------|
+| Read file as `_` | `CREATE VIEW _ AS read_csv(...)` | `scan_csv(...)`, registered as `_` in `SQLContext` |
+| Reference tables | `CREATE TABLE r AS read_*` | `scan_*` → `ctx.register("r", lf)` |
+| Schema evolution | `INSERT ... BY NAME` / `UNION ALL BY NAME` | `concat_lf_diagonal` |
+| `--where` | `SELECT * FROM _ WHERE ..` | same SQL via `SQLContext` |
+| `--filter-sql` / `--post-sql` | raw SQL on `_` | same SQL via `SQLContext` |
+| Explicit `--schema` | `CREATE TABLE (...)` + coerce | cast columns to declared dtypes; missing → null, extra → dropped |
+| Masking | SQL `hmac(...)` UPDATE | existing Rust HMAC, applied via `with_column` |
+| Lineage | SQL column adds | `with_columns` expressions |
+| Crypto | SQL transforms | existing Rust AES/ECB via expressions |
+| `--limit` | `LIMIT n` rewrite | `.limit(n)` |
+| Profiling | DuckDB aggregates | Polars aggregate expressions (see below) |
+| Export | `COPY ... TO` | `CsvWriter`/`ParquetWriter`/`JsonWriter` |
+
+## SQL Surface: What Users Gain and Lose
+
+`SQLContext` covers the dtoo SQL use cases: `SELECT`, `WHERE`, `GROUP BY` + aggregates, `INNER`/`LEFT`/`RIGHT`/`FULL`/`CROSS JOIN`, `ORDER BY`, `LIMIT`, CTEs, `UNION`/`UNION ALL`, subqueries, `CASE`, and common string/date functions. Unsupported SQL returns a clear `Result` error — never a hang.
+
+**Known limitations to document in the user guide:**
+
+1. **Window functions** (`OVER (PARTITION BY .. ORDER BY ..)`) have known correctness bugs in Polars SQL. Document as unsupported; do not silently emit wrong results.
+2. **Narrower function library** than DuckDB — some exotic date/regex/string functions are absent. They fail with a clear "unsupported function" error.
+3. **CSV/NDJSON output compression** is not native to Polars — implemented by wrapping the writer in `flate2` (gzip) / `zstd`. Parquet codecs remain built in.
+
+## Dependencies
+
+**Add:**
+```toml
+polars = { version = "1", features = ["lazy", "sql", "csv", "parquet", "json", "excel"] }
+# additional dtype-* features (e.g. decimal, datetime) selected during implementation as the
+# porting of schema/profiling reveals which are required
+flate2 = "1"   # gzip output for CSV/NDJSON (Polars has no native text-output compression)
+zstd = "0.13"  # zstd output for CSV/NDJSON
+```
+`excel` pulls in the pure-Rust `calamine` backend (no C/C++). Cloud features are intentionally **not** enabled in this phase.
+
+**Remove:**
+```toml
+duckdb = { version = "1", features = ["bundled"] }   # eliminates bundled C++ toolchain
+glob = "0.3"                                          # Polars scans globs natively
+```
+
+`flate2`/`zstd` are justified by a capability Polars lacks (text-output compression that dtoo already advertises via `--compress`); they are small, pure-Rust, and widely used.
+
+## Cloud Deferral Behaviour
+
+CLI flags (`--s3-region`, `--s3-profile`, `--gcs-project`, `--azure-account`) remain parsed for forward compatibility. A cloud path (`s3://`, `gs://`, `az://`) in any input/ref/output position returns an explicit error:
+
+> `cloud storage (s3://…) is not supported in this build yet`
+
+This keeps the CLI surface stable and avoids the previous silent-failure class. A later spec re-enables cloud behind the Polars `cloud`/`aws`/`gcp`/`azure` features.
+
+## Profiling with Polars
+
+Per-column statistics map to Polars aggregate expressions: `null_count`, `n_unique` (with `approx_n_unique` for large inputs), `min`/`max`/`mean`/`std`/`median`, `quantile(0.25)`/`quantile(0.75)`, string `len_chars` stats, and top-N via `value_counts` + sort + head. There is no single `describe()` on `LazyFrame`; the profiler builds the aggregation list explicitly per dtype. HTML/JSON/CSV report formats are unchanged.
+
+## Migration Strategy
+
+Convert in slices, keeping `cargo test` green at every step. The existing inline pipeline tests and `tests/` integration tests assert *behaviour*, not engine internals, so they are the regression safety net.
+
+1. **Readers + simple commands:** `scan_*` + glob; convert `convert`, `inspect`, `fingerprint`.
+2. **Query core:** scan → `concat_lf_diagonal` → `SQLContext` (`--where`/`--filter-sql`/`--post-sql`/refs) → write. Includes schema mode.
+3. **Internal expression stages:** masking, lineage, limit as native Polars expressions.
+4. **Profiler:** port aggregate queries to Polars expressions.
+5. **Crypto:** port decrypt/encrypt stages to expression-based application.
+6. **Cleanup:** delete DuckDB-specific code (`sql_utils` escaping helpers no longer needed for internal SQL), update `DESIGN.md` and the user guide (SQL limitations, cloud deferral).
+
+## Error Handling
+
+- File read / malformed data → `PolarsError` surfaced with the file path. **Never hangs** (the originating motivation).
+- Unsupported SQL → clear error naming the construct, returned as `Result`.
+- Schema coercion failure under explicit `--schema` → error naming the column and target dtype.
+- Cloud path → explicit "not supported in this build" error (see above).
+
+## Testing
+
+- Keep all existing behavioural tests passing; they define the conversion's success.
+- Add tests for the new honest-error paths: malformed CSV field returns an error (not a hang), unsupported window-function SQL returns a clear error.
+- Add a union-by-name test across files with differing/overlapping columns (`concat_lf_diagonal`).
+- Verify CSV/NDJSON gzip and zstd output round-trip.
+
+---
+
+## Appendix: Deferred Feature Ideas (separate specs, after the conversion)
+
+Captured here so they are not lost; **not** part of this conversion. Each would be its own spec/PR, and several need a "does this belong in core?" check per `CLAUDE.md` before pursuing:
+
+- **`dtoo dbt-tests`** — profile a dataset and emit a proposed dbt `schema.yml` with tests (`not_null`, `unique`, `accepted_values`, range/`between`, `relationships`, freshness) derived from profiling stats. Strong fit: dtoo already profiles. The headline idea from the original request.
+- **`dtoo dbt-model`** — scaffold a dbt staging model (`.sql` + `.yml`) from a source file's schema: snake_case renames, type casts, a stub `sources.yml`.
+- **BigQuery schema emission** — from profiling/schema, emit a BigQuery DDL or `bq load --schema` JSON, with a Polars→BQ type mapping. Emits artifacts/commands rather than adding a network client, keeping the build lean.
+- **Polars code generation** — emit an equivalent Polars (Python or Rust) snippet for a given dtoo query, as a learning/bridge aid. Niche; lowest priority.