Remove duck db

Cargo.toml

@@ -9,7 +9,6 @@ serde = { version = "1", features = ["derive"] }
 serde_json = "1"
 serde_yaml = "0.9"
 thiserror = "2"
-duckdb = { version = "1", features = ["bundled"] }
 glob = "0.3"
 uuid = { version = "1", features = ["v4"] }
 sha2 = "0.10"

docs/DESIGN.md

@@ -4,7 +4,18 @@ A Rust CLI tool for data engineers to query, profile, and transform data across
 
 ## Core Engine
 
-**DuckDB** (via `duckdb` Rust crate) — provides SQL execution, format readers (Parquet, CSV, NDJSON, Excel), glob support, cloud storage (S3/GCS/Azure via httpfs/azure extensions), temp tables, hashing, and UUID generation.
+**Polars** (pure Rust, `polars` crate ~0.54.x) — provides SQL execution via `SQLContext`, format readers (Parquet, CSV/TSV, NDJSON, and Excel via the pure-Rust `calamine` backend), glob resolution, type-safe lazy evaluation, and the `concat_lf_diagonal` union-by-name schema evolution. There is no bundled C++ toolchain and no runtime extension downloads.
+
+**Cloud storage** (S3/GCS/Azure) is **deferred in this build**: cloud paths (`s3://`, `gs://`, `az://`) return a clear `"cloud storage … is not supported in this build yet"` error. The cloud CLI flags (`--s3-region`, `--s3-profile`, `--gcs-project`, `--azure-account`) still parse for forward compatibility.
+
+### SQL Surface Limitations
+
+User-facing SQL (`--where`, `--filter-sql`, `--post-sql`, reference-table JOINs) runs on Polars `SQLContext`. Compared to DuckDB, the SQL surface is narrower:
+
+- **`DELETE` / `UPDATE` statements are treated as transforms**, not mutations. For example, `DELETE FROM _ WHERE x` drops matching rows and returns a result set rather than raising an error. Do not rely on DML semantics in `--post-sql`.
+- **Window functions** (`OVER (PARTITION BY … ORDER BY …)`) have known correctness issues in Polars SQL. Avoid them; dtoo does not attempt to detect or warn about their use.
+- **Narrower function library**: some exotic DuckDB date, regex, and string functions are absent. Unsupported SQL returns a clear error (it never silently hangs — which was the motivation for the migration).
+- Errors are always explicit `Result` values; the engine never hangs on malformed input.
 
 ---
 
@@ -36,9 +47,9 @@ Execution order within `dtoo query`:
 1.  Resolve file list (glob pattern / pipe input / explicit paths)
 2.  Apply --exclude patterns to filter file list
 3.  If --dry-run: display plan and exit
-4.  Init DuckDB in-memory, load extensions
-5.  Load reference tables into named DuckDB tables
-6.  Create temp_results table (schema from first file, or explicit schema)
+4.  Init Polars engine; scan reference table files into named LazyFrames
+5.  Scan each input file as a LazyFrame; register refs and _ in SQLContext
+6.  Create accumulated result (LazyFrame; schema from first file or explicit --schema)
 7.  For each file:
     a. Register file as `_` (the magic table name)
     b. Apply --where clause if specified:  SELECT * FROM _ WHERE {where}
@@ -173,11 +184,11 @@ dtoo fingerprint <PATH>
 
 Files are resolved from one of three sources (mutually exclusive):
 
-1. **--glob**: DuckDB-compatible glob pattern. Supports `**` for recursive matching.
+1. **--glob**: Glob pattern. Supports `**` for recursive matching (resolved by Polars native glob scan).
 2. **--pipe file**: Newline-delimited file paths from stdin.
 3. **--pipe data**: Raw data stream from stdin (requires `--stdin-format`).
 
-Cloud paths (s3://, gs://, az://) are supported in all modes.
+Cloud paths (`s3://`, `gs://`, `az://`) are **not supported in this build** — they return an explicit error. See Core Engine above.
 
 Format is auto-detected from file extension:
 - `.parquet` — Parquet
@@ -190,6 +201,8 @@ Format is auto-detected from file extension:
 2. **`--sheet` flag**: applies to all `.xlsx` files matched by glob or pipe
 3. **Default**: first sheet
 
+**Excel reading behavior (calamine):** Every cell is read as a string; type inference is deferred to downstream SQL casts or explicit `--schema`. A data row wider than the header row is an error (no silent data loss).
+
 Examples:
 ```bash
 # All xlsx files, same sheet
@@ -205,11 +218,12 @@ find . -name "*.xlsx" | dtoo query --pipe file --sheet "Data"
 
 ### Schema Handling
 
-**Default (no --schema):** Union-by-name with type promotion. The temp_results table schema evolves as new columns are encountered. DuckDB handles type promotion (e.g., INT -> BIGINT -> DOUBLE).
+**Default (no --schema):** Union-by-name with type promotion. The accumulated result schema evolves as new columns are encountered. Polars `concat_lf_diagonal` handles type promotion (e.g., Int32 → Int64 → Float64) and fills missing columns with `null`.
 
 **Explicit (--schema):** Schema file defines the target columns and types. Files are coerced to match. Extra columns in source files are ignored; missing columns become NULL.
 
-Schema file format (YAML):
+Schema file format (YAML). Type strings use DuckDB-style names (e.g. `INTEGER`, `VARCHAR`, `DECIMAL(10,2)`, `TIMESTAMP`) which are mapped to Polars dtypes at load time. Bare `DECIMAL` defaults to `DECIMAL(18,3)`.
+
 ```yaml
 columns:
   - name: id
@@ -226,12 +240,11 @@ columns:
 
 When both are specified, `--where` is applied first as a pre-filter, then `--filter-sql` operates on the result:
 
-```sql
+```
 -- Internal execution when both specified:
--- Step 1: Apply --where
-CREATE TEMP VIEW _pre AS SELECT * FROM read_parquet('{file}') WHERE {where_clause};
--- Step 2: Apply --filter-sql (user's SQL, _ now points to _pre)
-INSERT INTO temp_results SELECT ... FROM _pre ...;
+-- Step 1: Scan file → LazyFrame; register as _; execute: SELECT * FROM _ WHERE {where_clause}
+-- Step 2: Apply --filter-sql against the filtered LazyFrame (user's SELECT from _)
+-- Step 3: Concatenate result into the accumulated LazyFrame (union-by-name)
 ```
 
 When only `--where` is specified, it's equivalent to `--filter-sql "SELECT * FROM _ WHERE {clause}"`.
@@ -240,7 +253,7 @@ When only `--filter-sql` is specified, it runs directly against the file.
 
 ### Reference Tables
 
-Loaded once at startup into named DuckDB tables:
+Loaded once at startup into named LazyFrames registered in the `SQLContext`:
 
 ```
 --ref regions=ref/regions.parquet --ref products=lookups/products.csv
@@ -281,7 +294,7 @@ NULL values remain NULL (not masked).
 
 ### Profiling
 
-Uses DuckDB's analytical capabilities to produce per-column statistics:
+Produces per-column statistics using Polars aggregate expressions:
 
 | Metric | Applies To |
 |--------|-----------|
@@ -533,22 +546,23 @@ CLI flags override config file values. Config file can be combined with CLI flag
 
 | Crate | Purpose |
 |-------|---------|
-| `duckdb` | Core engine, SQL execution, file readers |
+| `polars` | Core engine: SQL execution (`SQLContext`), lazy evaluation, format readers (Parquet, CSV, NDJSON) |
+| `calamine` | Excel reader (pure Rust; bundled via Polars `excel` feature) |
+| `flate2` | gzip output wrapping for CSV/NDJSON (Polars has no native text-output compression) |
+| `zstd` | zstd output wrapping for CSV/NDJSON |
 | `clap` | CLI argument parsing (derive API) |
 | `serde` + `serde_yaml` | Config file and schema parsing |
 | `uuid` | Lineage UUID generation |
 | `sha2` + `hmac` | Column masking and file fingerprinting |
 | `chrono` | Timestamp handling |
-| `glob` | File pattern matching (fallback for non-DuckDB resolution) |
 | `indicatif` | Progress bars for file processing |
 | `comfy-table` | Terminal table output for inspect |
-| `tokio` | Async runtime (needed for cloud storage operations) |
 
 ---
 
 ## v1.0 Scope (Open Source)
 
-Everything described above, including cloud storage (S3, GCS, Azure) since DuckDB extensions make this nearly free.
+Everything described above. **Cloud storage (S3, GCS, Azure) is deferred** in the current build — cloud paths return an explicit error. Cloud support will be re-enabled in a follow-up once the Polars engine is proven in local-file deployments.
 
 ---
 
@@ -560,7 +574,7 @@ Everything described above, including cloud storage (S3, GCS, Azure) since DuckD
 | Core query/transform | Yes | Yes | Yes |
 | Profile/inspect/fingerprint | Yes | Yes | Yes |
 | Lineage, masking, pipe mode | Yes | Yes | Yes |
-| Cloud storage (S3/GCS/Azure) | Yes | Yes | Yes |
+| Cloud storage (S3/GCS/Azure) | Deferred | Deferred | Deferred |
 | Config files, manifests | Yes | Yes | Yes |
 | Data quality assertions | - | Yes | Yes |
 | Incremental processing | - | Yes | Yes |
@@ -639,8 +653,7 @@ overall pass/fail status. Exit code 4 on assertion failure.
 | `referential_integrity` | All values exist in a reference table column |
 | `custom_sql` | User-provided SQL returning violation rows |
 
-Implementation: each rule translates to a DuckDB query against the temp_results table. Runs
-after post-sql but before output. `custom_sql` allows arbitrary validation:
+Implementation: each rule translates to a query (via Polars `SQLContext`) against the accumulated result. Runs after post-sql but before output. `custom_sql` allows arbitrary validation:
 
 ```yaml
   - rule: custom_sql
@@ -765,14 +778,14 @@ dtoo query --glob "data/**/*.parquet" \
 ```
 
 **Supported databases:**
-| Database | Connection String | DuckDB Support |
-|----------|------------------|----------------|
-| PostgreSQL | `postgres://...` | Native (postgres_scanner) |
-| MySQL | `mysql://...` | Native (mysql_scanner) |
-| SQLite | `sqlite:///path/to/db` | Native (sqlite_scanner) |
-| Snowflake | `snowflake://account/db/schema?table=t` | Via ADBC driver |
-| BigQuery | `bigquery://project/dataset?table=t` | Via ADBC driver |
-| Redshift | `redshift://...` | Via Postgres wire protocol |
+| Database | Connection String |
+|----------|------------------|
+| PostgreSQL | `postgres://...` |
+| MySQL | `mysql://...` |
+| SQLite | `sqlite:///path/to/db` |
+| Snowflake | `snowflake://account/db/schema?table=t` |
+| BigQuery | `bigquery://project/dataset?table=t` |
+| Redshift | `redshift://...` (Postgres wire protocol) |
 
 **Sink modes:**
 - `append`: INSERT INTO target table. Schema must be compatible.
@@ -802,7 +815,7 @@ output/region=GB/year=2024/part-0.parquet
 ...
 ```
 
-DuckDB's `COPY ... PARTITION_BY` handles this natively for Parquet and CSV.
+Output is split by writing each partition subset to its own path.
 
 Partition columns are removed from the data by default (they're encoded in the path).
 Use `--partition-keep-columns` to retain them in the data as well.
@@ -1071,9 +1084,7 @@ Process files across multiple threads for large file sets:
 dtoo query --glob "data/**/*.parquet" --parallel 8 --where "amount > 100"
 ```
 
-**Implementation:** Spawn N worker threads, each with its own DuckDB connection. Each worker
-processes files from a shared queue and inserts into a thread-local temp table. After all files
-are processed, merge thread-local tables into the final temp_results.
+**Implementation:** Spawn N worker threads, each scanning files from a shared queue into thread-local `LazyFrame`s. After all files are processed, merge per-worker frames into the final accumulated result via `concat_lf_diagonal`.
 
 **Considerations:**
 - Default: sequential (1 thread). Enterprise unlocks `--parallel N`.
@@ -1101,7 +1112,7 @@ are processed, merge thread-local tables into the final temp_results.
 
 - **Watch mode**: Re-run on file changes
 - **Plugin system**: Custom format readers
-- **SQLite input**: Read from SQLite files via sqlite_scanner extension
+- **SQLite input**: Read from SQLite files (would require a pure-Rust SQLite reader crate)
 - **Avro input**: Via Rust avro crate as preprocessor
 - **Web UI**: Browser-based dashboard for audit logs, pipeline status, and profiling reports
 - **dtoo server**: Long-running daemon mode for API-driven pipeline execution

docs/USER_GUIDE.md

@@ -4,14 +4,14 @@ This guide covers day-to-day usage of `dtoo`, from first run through practical w
 
 ## What is dtoo?
 
-`dtoo` is a Rust CLI for querying and profiling data files with DuckDB. It is designed for fast local analytics and reproducible pipelines across file trees.
+`dtoo` is a Rust CLI for querying and profiling data files. It is built on **Polars** (pure Rust) and is designed for fast local analytics and reproducible pipelines across file trees.
 
 Core capabilities:
 - Query many files using SQL
 - Read CSV, Parquet, NDJSON, and Excel (`.xlsx`, `.xls`)
 - Join reference tables into your query
 - Add lineage, masking, profiling, fingerprinting, and manifests
-- Read from local files or cloud paths (S3/GCS/Azure)
+- Read from local files (cloud paths S3/GCS/Azure are deferred — they return a clear error in this build)
 
 ## Install and Build
 
@@ -94,6 +94,12 @@ dtoo query --glob "data/**/*.parquet" \
   --post-sql "SELECT passenger_count, COUNT(*) AS trips FROM _ GROUP BY 1"
 ```
 
+**SQL limitations (Polars `SQLContext`):** `SELECT`, `WHERE`, `GROUP BY`, `JOIN`, `ORDER BY`, `LIMIT`, CTEs, `UNION`/`UNION ALL`, subqueries, and common string/date functions are supported. Known gaps:
+
+- **Window functions** (`OVER (PARTITION BY … ORDER BY …)`) have correctness issues — avoid them.
+- **`DELETE`/`UPDATE`** are treated as row-filtering transforms, not DML mutations. Do not rely on DML semantics in `--post-sql`.
+- Some exotic DuckDB date/regex/string functions are absent. They return a clear error (never a silent hang).
+
 ### Excel Sheet Selection
 
 You can control sheet selection in two ways:
@@ -223,21 +229,13 @@ dtoo fingerprint data/trips.parquet
 
 ## Cloud Paths
 
-`dtoo` supports cloud URIs when credentials are configured for DuckDB extensions.
-
-Common options:
-- `--s3-region`
-- `--s3-profile`
-- `--gcs-project`
-- `--azure-account`
-
-Examples:
+**Cloud storage is deferred in this build.** Paths beginning with `s3://`, `gs://`, or `az://` return an explicit error:
 
-```bash
-dtoo query --glob "s3://my-bucket/data/**/*.parquet" --s3-region us-east-1
-
-dtoo query gs://my-bucket/input.csv --gcs-project my-project
 ```
+cloud storage (s3://…) is not supported in this build yet
+```
+
+The cloud CLI flags (`--s3-region`, `--s3-profile`, `--gcs-project`, `--azure-account`) still parse so that config files written for a future cloud-enabled build remain valid.
 
 ## Practical Recipes
 

docs/specs/03-duckdb-engine.md

@@ -1,5 +1,7 @@
 # DuckDB Engine
 
+> **Superseded by [docs/specs/34-polars-engine.md](34-polars-engine.md) — dtoo migrated from DuckDB to Polars. This spec is retained for historical context.**
+
 > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this spec.
 
 **Goal:** Initialise and manage the in-memory DuckDB instance that powers all SQL execution, file reading, and data accumulation.