diff --git a/CLAUDE.md b/CLAUDE.md index 47e343a..089018f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,6 +6,25 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co **mountainash-data** provides physical access to backend data services — relational databases via Ibis, and Iceberg table-format catalogs via PyIceberg. It collapses what was previously 13 per-dialect connection classes into a data-driven `DialectSpec` registry, exposes clean `Backend` / `Connection` protocols, and provides factories and a high-level facade (`DatabaseUtils`). +## Planning, Specs & Principles (live in mountainash-central) + +This repo holds **code only**. All design specs, implementation plans, the technical-debt +backlog, and the architecture principles live in the ecosystem repo **`mountainash-central`** +(sibling checkout: `../mountainash-central`): + +| Artifact | Location in `mountainash-central` | +|----------|-----------------------------------| +| Design specs | `04.planning/mountainash-data/superpowers/specs/` | +| Implementation plans | `04.planning/mountainash-data/superpowers/plans/` | +| Technical-debt backlog (+ `archive/`) | `04.planning/mountainash-data/a.backlog/` | +| ENFORCED architecture principles | `01.principles/mountainash-data/` | + +**When running the superpowers flow (brainstorming → writing-plans → SDD) for this repo, +write the spec/plan to `mountainash-central/04.planning/mountainash-data/superpowers/{specs,plans}/`, +not to a local `docs/superpowers/` directory.** (The superpowers skills default to a local +`docs/superpowers/` path — override that default to the central location above.) Only source, +tests, and this `CLAUDE.md` live in this repo's tree. + ## Architecture ### Core Components diff --git a/docs/superpowers/plans/2026-04-07-mountainash-data-audit-and-redesign.md b/docs/superpowers/plans/2026-04-07-mountainash-data-audit-and-redesign.md deleted file mode 100644 index bcb280f..0000000 --- a/docs/superpowers/plans/2026-04-07-mountainash-data-audit-and-redesign.md +++ /dev/null @@ -1,2615 +0,0 @@ -# mountainash-data Audit and Redesign Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Restructure `mountainash-data` from a class-explosion / mixin-tree architecture into a `core/` Protocol layer with `backends/ibis` and `backends/iceberg` peer implementations. - -**Architecture:** Single `Backend` Protocol in `core/`. `IbisBackend` collapses 13 per-backend connection files into a data-driven `DialectSpec` registry. `IcebergBackend` finishes the half-done split between connection and operations files. Settings and factories relocate to `core/` largely unchanged. The seam to `mountainash-expressions` is `Backend.connect().to_relation(table)` returning a Relation. - -**Tech Stack:** Python 3.12, ibis-framework 10.4.0, pyiceberg, pydantic (settings), hatch (build/test), pytest, ruff, mypy. - -**Spec:** `docs/superpowers/specs/2026-04-07-mountainash-data-audit-and-redesign.md` - -**Phase boundaries:** Each phase ends with `hatch run test:test` green. Commit at the end of every task. Do not start a phase until the previous one is green. - ---- - -## Pre-flight: Working environment - -- [ ] **Step P1: Create a working branch off develop** - -```bash -git checkout develop -git pull -git checkout -b refactor/data-package-audit -``` - -- [ ] **Step P2: Confirm baseline tests pass before any changes** - -Run: `hatch run test:test` -Expected: PASS. If anything is red on `develop`, stop and report — do not start the refactor on a broken baseline. - -- [ ] **Step P3: Capture baseline test count and coverage for later comparison** - -Run: `hatch run test:test --collect-only -q | tail -5` -Record the test count somewhere (a scratch note). It should not decrease across the refactor except where tests are deliberately removed alongside their targets. - ---- - -## Phase 0 — Cleanup (no architectural change) - -**Files touched:** -- Delete: `src/mountainash_data/lineage/openlineage_helper.py` -- Delete: `src/mountainash_data/lineage/__init__.py` (if exists) and the empty `lineage/` dir -- Delete: `src/mountainash_data/databases/connections/pyiceberg/__init___old.py` -- Delete: `src/mountainash_data/databases/connections/db_connection_factory.py` -- Delete: `src/mountainash_data/databases/operations/ibis/postgres_ibis_operations.py` -- Delete: `src/mountainash_data/databases/operations/ibis/mysql_ibis_operations.py` -- Delete: `src/mountainash_data/databases/operations/ibis/oracle_ibis_operations.py` -- Delete: `src/mountainash_data/databases/operations/ibis/bigquery_ibis_operations.py` -- Delete: `src/mountainash_data/databases/operations/ibis/snowflake_ibis_operations.py` -- Delete: `src/mountainash_data/databases/operations/ibis/pyspark_ibis_operations.py` -- Delete: `src/mountainash_data/databases/operations/ibis/redshift_ibis_operations.py` -- Delete: `src/mountainash_data/databases/operations/ibis/mssql_ibis_operations.py` -- Modify: `src/mountainash_data/databases/operations/ibis/__init__.py` (drop deleted re-exports) -- Modify: `src/mountainash_data/__init__.py` (drop any references to deleted symbols if present) -- Modify: `src/mountainash_data/databases/connections/__init__.py` (drop reference to deleted `db_connection_factory`) -- Modify: `src/mountainash_data/databases/connections/pyiceberg/__init__.py` (drop `__init___old`) - -### Task 0.1: Verify nothing imports the legacy db_connection_factory - -- [ ] **Step 1: Grep for usages** - -Run: `grep -rn "db_connection_factory\|DBConnectionFactory" src/ tests/ notebooks/ docs/` -Expected: usages only inside `databases/connections/db_connection_factory.py` itself, the modern `factories/connection_factory.py` (if it references the legacy class — unlikely), and any `__init__.py` re-exports. - -If consumers exist outside the file itself, STOP and surface to user — the spec assumed this was a duplicate and safe to delete. - -- [ ] **Step 2: Confirm 8 stub ops files have no concrete logic** - -For each of the 8 files (postgres, mysql, oracle, bigquery, snowflake, pyspark, redshift, mssql): -Run: `wc -l src/mountainash_data/databases/operations/ibis/_ibis_operations.py` -Read each file. Confirm each is ≤20 lines and contains only a class declaration with a `db_backend_name` property override. - -If any file has methods beyond the property override, STOP and reclassify as `salvage` per the spec. - -### Task 0.2: Delete the lineage stub - -- [ ] **Step 1: Delete the file and (if present) its `__init__.py`** - -```bash -rm src/mountainash_data/lineage/openlineage_helper.py -rm -f src/mountainash_data/lineage/__init__.py -rmdir src/mountainash_data/lineage -``` - -- [ ] **Step 2: Grep for lingering imports** - -Run: `grep -rn "from mountainash_data.lineage\|mountainash_data\.lineage" src/ tests/ notebooks/` -Expected: zero matches. - -- [ ] **Step 3: Run tests** - -Run: `hatch run test:test` -Expected: PASS, same count as baseline. - -- [ ] **Step 4: Commit** - -```bash -git add -A -git commit -m "chore(data): remove empty lineage stub" -``` - -### Task 0.3: Delete the pyiceberg `__init___old.py` artifact - -- [ ] **Step 1: Delete** - -```bash -rm src/mountainash_data/databases/connections/pyiceberg/__init___old.py -``` - -- [ ] **Step 2: Confirm `__init__.py` doesn't reference it** - -Read `src/mountainash_data/databases/connections/pyiceberg/__init__.py`. Remove any import of `__init___old` if present. - -- [ ] **Step 3: Run tests** - -Run: `hatch run test:test` -Expected: PASS. - -- [ ] **Step 4: Commit** - -```bash -git add -A -git commit -m "chore(data): remove pyiceberg __init___old artifact" -``` - -### Task 0.4: Delete the legacy `db_connection_factory.py` - -- [ ] **Step 1: Delete the file** - -```bash -rm src/mountainash_data/databases/connections/db_connection_factory.py -``` - -- [ ] **Step 2: Update `databases/connections/__init__.py`** - -Read `src/mountainash_data/databases/connections/__init__.py`. Remove any import or re-export of `db_connection_factory` / `DBConnectionFactory`. - -- [ ] **Step 3: Update top-level `__init__.py` if needed** - -Read `src/mountainash_data/__init__.py`. Remove any reference to the deleted symbol. - -- [ ] **Step 4: Run tests** - -Run: `hatch run test:test` -Expected: PASS. - -If tests fail with `ImportError` referencing `db_connection_factory`, find the importer and update or remove the reference. - -- [ ] **Step 5: Commit** - -```bash -git add -A -git commit -m "chore(data): delete legacy db_connection_factory (duplicate of factories/connection_factory)" -``` - -### Task 0.5: Delete the 8 stub ibis ops files - -- [ ] **Step 1: Delete the files** - -```bash -cd src/mountainash_data/databases/operations/ibis -rm postgres_ibis_operations.py mysql_ibis_operations.py oracle_ibis_operations.py \ - bigquery_ibis_operations.py snowflake_ibis_operations.py pyspark_ibis_operations.py \ - redshift_ibis_operations.py mssql_ibis_operations.py -cd - -``` - -- [ ] **Step 2: Update `databases/operations/ibis/__init__.py`** - -Read the file. Remove imports and `__all__` entries for the 8 deleted classes. The remaining valid ops modules are: `base_ibis_operations`, `_base_ibis_mixin`, `_duckdb_family_mixin`, `duckdb_ibis_operations`, `sqlite_ibis_operations`, `motherduck_ibis_operations`, `trino_ibis_operations`. - -- [ ] **Step 3: Update top-level `__init__.py` and any other re-exporters** - -Run: `grep -rn "PostgresIbisOperations\|MySQLIbisOperations\|OracleIbisOperations\|BigQueryIbisOperations\|SnowflakeIbisOperations\|PySparkIbisOperations\|RedshiftIbisOperations\|MSSQLIbisOperations" src/` -For each match outside the deleted files, decide: if it's a re-export, remove it; if it's actual usage in `factories/operations_factory.py`, update the strategy mapping to fall back to `BaseIbisOperations` (the parent class that contained all the actual logic). - -- [ ] **Step 4: Update `factories/operations_factory.py`** - -Read the file. Find the strategy mapping (likely a dict mapping backend names to class import paths). For each of the 8 deleted backends, replace the per-backend ops class with `mountainash_data.databases.operations.ibis.base_ibis_operations.BaseIbisOperations` (or whatever the canonical class name is — confirm by reading `base_ibis_operations.py`). - -- [ ] **Step 5: Run tests** - -Run: `hatch run test:test` -Expected: PASS. - -If `test_operations_factory.py` fails because it asserts on specific class names, update the test expectations to match the new mapping. Do NOT modify tests to pass without understanding why — confirm the test was asserting "factory returns class X for backend Y" and that the new behavior ("factory returns BaseIbisOperations for backend Y") is correct. - -- [ ] **Step 6: Commit** - -```bash -git add -A -git commit -m "chore(data): delete 8 empty per-backend ibis ops stubs - -Postgres, MySQL, Oracle, BigQuery, Snowflake, PySpark, Redshift, and -MSSQL operations classes were stubs containing only a db_backend_name -property override. The actual operations logic lives in -BaseIbisOperations (676 LOC). Factory mappings updated to point at the -base class for these backends." -``` - ---- - -## Phase 1 — Stand up `core/` - -**Files touched:** -- Create: `src/mountainash_data/core/__init__.py` -- Create: `src/mountainash_data/core/protocol.py` -- Create: `src/mountainash_data/core/inspection.py` -- Create: `src/mountainash_data/core/registry.py` (placeholder, wired in Phase 4) -- Create: `src/mountainash_data/core/connection.py` (moved from `databases/connections/base_db_connection.py`) -- Create: `src/mountainash_data/core/constants.py` (moved from `databases/constants.py`) -- Create: `tests/test_unit/core/test_protocol.py` -- Create: `tests/test_unit/core/test_inspection.py` -- Modify: `src/mountainash_data/databases/connections/base_db_connection.py` (becomes a re-export shim) -- Modify: `src/mountainash_data/databases/constants.py` (becomes a re-export shim) - -### Task 1.1: Define the inspection model dataclasses - -- [ ] **Step 1: Write the failing test** - -Create `tests/test_unit/core/__init__.py` (empty) and `tests/test_unit/core/test_inspection.py`: - -```python -"""Tests for core.inspection — the shared physical metadata model.""" - -from mountainash_data.core.inspection import ( - CatalogInfo, - ColumnInfo, - NamespaceInfo, - TableInfo, -) - - -class TestColumnInfo: - def test_minimal_column(self): - col = ColumnInfo(name="id", type_name="int64", nullable=False) - assert col.name == "id" - assert col.type_name == "int64" - assert col.nullable is False - - def test_column_with_metadata(self): - col = ColumnInfo( - name="created_at", - type_name="timestamp", - nullable=True, - description="row creation time", - ) - assert col.description == "row creation time" - - -class TestTableInfo: - def test_table_with_columns(self): - cols = [ - ColumnInfo(name="id", type_name="int64", nullable=False), - ColumnInfo(name="name", type_name="string", nullable=True), - ] - table = TableInfo(name="users", columns=cols) - assert table.name == "users" - assert len(table.columns) == 2 - assert table.column_names == ["id", "name"] - - def test_table_qualified_name(self): - table = TableInfo( - name="users", - columns=[], - namespace="public", - catalog="main", - ) - assert table.qualified_name == "main.public.users" - - def test_table_qualified_name_no_catalog(self): - table = TableInfo(name="users", columns=[], namespace="public") - assert table.qualified_name == "public.users" - - def test_table_qualified_name_bare(self): - table = TableInfo(name="users", columns=[]) - assert table.qualified_name == "users" - - -class TestNamespaceInfo: - def test_namespace_with_tables(self): - ns = NamespaceInfo(name="public", tables=["users", "orders"]) - assert ns.name == "public" - assert ns.tables == ["users", "orders"] - - -class TestCatalogInfo: - def test_catalog_with_namespaces(self): - cat = CatalogInfo( - name="main", - namespaces=[NamespaceInfo(name="public", tables=["users"])], - ) - assert cat.name == "main" - assert len(cat.namespaces) == 1 -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `hatch run test:test-quick tests/test_unit/core/test_inspection.py -v` -Expected: FAIL with `ModuleNotFoundError: mountainash_data.core` - -- [ ] **Step 3: Create the core package and inspection module** - -Create `src/mountainash_data/core/__init__.py` (empty for now). - -Create `src/mountainash_data/core/inspection.py`: - -```python -"""Shared physical-layer metadata model. - -Both ibis and iceberg backends populate these dataclasses from their -native introspection APIs, giving consumers a uniform shape regardless -of which backend produced them. -""" - -from __future__ import annotations - -from dataclasses import dataclass, field -import typing as t - - -@dataclass(frozen=True) -class ColumnInfo: - """Physical metadata for a single column.""" - - name: str - type_name: str - nullable: bool - description: t.Optional[str] = None - metadata: t.Mapping[str, t.Any] = field(default_factory=dict) - - -@dataclass(frozen=True) -class TableInfo: - """Physical metadata for a single table or view.""" - - name: str - columns: t.Sequence[ColumnInfo] - namespace: t.Optional[str] = None - catalog: t.Optional[str] = None - description: t.Optional[str] = None - metadata: t.Mapping[str, t.Any] = field(default_factory=dict) - - @property - def column_names(self) -> list[str]: - return [c.name for c in self.columns] - - @property - def qualified_name(self) -> str: - parts = [p for p in (self.catalog, self.namespace, self.name) if p] - return ".".join(parts) - - -@dataclass(frozen=True) -class NamespaceInfo: - """Physical metadata for a namespace (schema/database/dataset).""" - - name: str - tables: t.Sequence[str] - catalog: t.Optional[str] = None - metadata: t.Mapping[str, t.Any] = field(default_factory=dict) - - -@dataclass(frozen=True) -class CatalogInfo: - """Physical metadata for a top-level catalog or backend instance.""" - - name: str - namespaces: t.Sequence[NamespaceInfo] - metadata: t.Mapping[str, t.Any] = field(default_factory=dict) -``` - -- [ ] **Step 4: Run tests** - -Run: `hatch run test:test-quick tests/test_unit/core/test_inspection.py -v` -Expected: PASS, all 7 test functions green. - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/core/ tests/test_unit/core/ -git commit -m "feat(data/core): add shared physical inspection model - -CatalogInfo / NamespaceInfo / TableInfo / ColumnInfo dataclasses -populated by both ibis and iceberg backends. The Medium shared layer -between paradigms — see spec section 'Shared inspection model'." -``` - -### Task 1.2: Define the Backend Protocol - -- [ ] **Step 1: Write the failing test** - -Create `tests/test_unit/core/test_protocol.py`: - -```python -"""Tests for core.protocol — structural Protocol definitions. - -These tests verify that the Protocols are well-formed and that a minimal -fake implementation type-checks at runtime via isinstance() with -runtime_checkable Protocols. -""" - -from __future__ import annotations - -import typing as t - -from mountainash_data.core.inspection import ( - CatalogInfo, - NamespaceInfo, - TableInfo, -) -from mountainash_data.core.protocol import Backend, Connection - - -class _FakeConnection: - """Minimal in-memory Connection implementation for protocol verification.""" - - def __init__(self): - self.closed = False - - def list_namespaces(self) -> list[str]: - return ["public"] - - def list_tables(self, namespace: str | None = None) -> list[str]: - return ["users"] - - def inspect_table(self, name: str, namespace: str | None = None) -> TableInfo: - return TableInfo(name=name, columns=[], namespace=namespace) - - def inspect_namespace(self, name: str) -> NamespaceInfo: - return NamespaceInfo(name=name, tables=["users"]) - - def inspect_catalog(self) -> CatalogInfo: - return CatalogInfo(name="fake", namespaces=[]) - - def close(self) -> None: - self.closed = True - - -class _FakeBackend: - """Minimal Backend implementation.""" - - name = "fake" - - def connect(self) -> _FakeConnection: - return _FakeConnection() - - -def test_fake_backend_satisfies_protocol(): - backend: Backend = _FakeBackend() - assert backend.name == "fake" - - -def test_fake_connection_satisfies_protocol(): - conn: Connection = _FakeConnection() - assert conn.list_namespaces() == ["public"] - - -def test_connection_inspect_returns_table_info(): - conn = _FakeConnection() - info = conn.inspect_table("users", namespace="public") - assert isinstance(info, TableInfo) - assert info.name == "users" - assert info.namespace == "public" - - -def test_connection_close_idempotent_marker(): - conn = _FakeConnection() - assert conn.closed is False - conn.close() - assert conn.closed is True -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `hatch run test:test-quick tests/test_unit/core/test_protocol.py -v` -Expected: FAIL with `ImportError: cannot import name 'Backend' from 'mountainash_data.core.protocol'` - -- [ ] **Step 3: Implement the Protocol module** - -Create `src/mountainash_data/core/protocol.py`: - -```python -"""Backend and Connection protocols. - -This is the structural contract every backend implementation must -satisfy. Implementations are plain classes — there is no inheritance. -""" - -from __future__ import annotations - -import typing as t - -from mountainash_data.core.inspection import ( - CatalogInfo, - NamespaceInfo, - TableInfo, -) - - -@t.runtime_checkable -class Connection(t.Protocol): - """A live, owned connection to a backend. - - Connections are obtained by calling Backend.connect(). They expose - physical introspection and lifecycle methods. Logical query - construction is the job of mountainash-expressions, reached via - to_relation() on backends that support it. - """ - - def list_namespaces(self) -> list[str]: - """Return the names of all namespaces (schemas) visible to this connection.""" - ... - - def list_tables(self, namespace: str | None = None) -> list[str]: - """Return the names of tables in the given namespace.""" - ... - - def inspect_table( - self, name: str, namespace: str | None = None - ) -> TableInfo: - """Return shared-model metadata for one table.""" - ... - - def inspect_namespace(self, name: str) -> NamespaceInfo: - """Return shared-model metadata for one namespace.""" - ... - - def inspect_catalog(self) -> CatalogInfo: - """Return shared-model metadata for the connection's catalog.""" - ... - - def close(self) -> None: - """Release the connection. Idempotent.""" - ... - - -@t.runtime_checkable -class Backend(t.Protocol): - """A factory for Connections to a particular backend service. - - Backends are constructed with config and are stateless from the - consumer's perspective. State lives on the Connection returned by - connect(). - """ - - name: str - - def connect(self) -> Connection: - """Open a connection. Caller is responsible for closing it.""" - ... -``` - -- [ ] **Step 4: Run tests** - -Run: `hatch run test:test-quick tests/test_unit/core/test_protocol.py -v` -Expected: PASS, all 4 test functions green. - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/core/protocol.py tests/test_unit/core/test_protocol.py -git commit -m "feat(data/core): add Backend and Connection protocols - -Structural typing — implementations satisfy the protocol by shape, no -inheritance required. Replaces the BaseDBConnection inheritance tree." -``` - -### Task 1.3: Add an empty registry placeholder - -- [ ] **Step 1: Create the placeholder module** - -Create `src/mountainash_data/core/registry.py`: - -```python -"""Backend registry — populated in Phase 4 once IbisBackend and -IcebergBackend exist. This module is intentionally a placeholder for -now so that imports from core.registry don't break across phases.""" - -from __future__ import annotations - -import typing as t - -from mountainash_data.core.protocol import Backend - -_REGISTRY: dict[str, t.Callable[..., Backend]] = {} - - -def register(name: str, factory: t.Callable[..., Backend]) -> None: - """Register a backend factory under a name.""" - _REGISTRY[name] = factory - - -def get(name: str, **config: t.Any) -> Backend: - """Look up and instantiate a backend by name.""" - if name not in _REGISTRY: - raise KeyError( - f"No backend registered as {name!r}. " - f"Available: {sorted(_REGISTRY)}" - ) - return _REGISTRY[name](**config) - - -def names() -> list[str]: - return sorted(_REGISTRY) -``` - -- [ ] **Step 2: Run tests** - -Run: `hatch run test:test` -Expected: PASS (no new tests yet — registry has no consumers until Phase 4). - -- [ ] **Step 3: Commit** - -```bash -git add src/mountainash_data/core/registry.py -git commit -m "feat(data/core): add backend registry placeholder - -Wired up in Phase 4 once IbisBackend and IcebergBackend exist." -``` - -### Task 1.4: Move `databases/constants.py` to `core/constants.py` with shim - -- [ ] **Step 1: Copy file to new location** - -```bash -cp src/mountainash_data/databases/constants.py src/mountainash_data/core/constants.py -``` - -- [ ] **Step 2: Replace old file with a re-export shim** - -Overwrite `src/mountainash_data/databases/constants.py`: - -```python -"""DEPRECATED: import from mountainash_data.core.constants instead. - -This shim exists during the Phase 1–6 refactor and will be removed in -Phase 6. -""" - -from mountainash_data.core.constants import * # noqa: F401,F403 -from mountainash_data.core.constants import ( # noqa: F401 # explicit re-exports - # Re-export every public symbol the old file exported. Read - # core/constants.py and add each enum/class name here so that - # `from mountainash_data.databases.constants import X` continues to work. -) -``` - -After writing the shim above, read `src/mountainash_data/core/constants.py` and list every top-level enum/class/constant. Add each name to the explicit re-exports list inside the parentheses. - -- [ ] **Step 3: Run tests** - -Run: `hatch run test:test` -Expected: PASS, baseline test count unchanged. - -If anything fails with `ImportError`, the explicit re-export list is missing a symbol — add it and re-run. - -- [ ] **Step 4: Commit** - -```bash -git add src/mountainash_data/core/constants.py src/mountainash_data/databases/constants.py -git commit -m "refactor(data): move constants to core/, leave shim at databases/" -``` - -### Task 1.5: Move `databases/connections/base_db_connection.py` to `core/connection.py` with shim - -Note: this file (231 LOC) defines `BaseDBConnection`, the abstract base of the old inheritance tree. In the new architecture, `core/protocol.py` (the Backend Protocol) is the authoritative contract. `BaseDBConnection` is moved verbatim for now so the old inheritance tree keeps working through Phase 5; it gets reduced or removed in Phase 6 when shims come down. - -- [ ] **Step 1: Copy file** - -```bash -cp src/mountainash_data/databases/connections/base_db_connection.py \ - src/mountainash_data/core/connection.py -``` - -- [ ] **Step 2: Update internal imports in the new `core/connection.py`** - -Read the new `src/mountainash_data/core/connection.py`. Any imports from `mountainash_data.databases.constants` should be rewritten to `mountainash_data.core.constants`. Any imports from `mountainash_data.databases.connections.base_db_connection` (self-references) stay relative or are unaffected. - -Also: any `from mountainash_data.databases.settings...` imports should be left as-is for now (settings move in Phase 2). They'll work via shims later. - -- [ ] **Step 3: Replace old file with shim** - -Overwrite `src/mountainash_data/databases/connections/base_db_connection.py`: - -```python -"""DEPRECATED: import from mountainash_data.core.connection instead. - -This shim exists during the Phase 1–6 refactor. -""" - -from mountainash_data.core.connection import * # noqa: F401,F403 -from mountainash_data.core.connection import BaseDBConnection # noqa: F401 -``` - -If `core/connection.py` defines additional public symbols (read it and confirm), add them to the explicit re-export list above. - -- [ ] **Step 4: Run tests** - -Run: `hatch run test:test` -Expected: PASS, baseline test count unchanged. - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/core/connection.py src/mountainash_data/databases/connections/base_db_connection.py -git commit -m "refactor(data): move BaseDBConnection to core/connection.py with shim - -The Backend Protocol in core/protocol.py is the new authoritative -contract. BaseDBConnection is preserved during the migration so the -existing ibis/pyiceberg subclasses keep working through Phase 5." -``` - -### Task 1.6: Phase 1 sanity check - -- [ ] **Step 1: Run full test suite** - -Run: `hatch run test:test` -Expected: PASS, count matches baseline. - -- [ ] **Step 2: Run lint and type-check** - -Run: `hatch run ruff:check` -Expected: PASS (or only pre-existing issues unrelated to new files). - -Run: `hatch run mypy:check` -Expected: PASS or only pre-existing issues. - -If new files have ruff/mypy issues, fix them before proceeding. - ---- - -## Phase 2 — Move settings to `core/settings/` - -**Files touched:** -- Create: `src/mountainash_data/core/settings/__init__.py` and one file per existing settings module (16 files) -- Modify: `src/mountainash_data/databases/settings/*.py` (each becomes a one-line shim) - -**Strategy:** Bulk move with shims. There is no logic change in this phase. The 16 files in `databases/settings/` are well-structured Pydantic settings classes per the audit. - -### Task 2.1: Move all settings files - -- [ ] **Step 1: Create the new directory and copy all files** - -```bash -mkdir -p src/mountainash_data/core/settings -cp src/mountainash_data/databases/settings/*.py src/mountainash_data/core/settings/ -``` - -- [ ] **Step 2: Update internal imports inside `core/settings/`** - -Run: `grep -rn "from mountainash_data.databases.settings\|import mountainash_data.databases.settings" src/mountainash_data/core/settings/` - -For every match, rewrite the import to use `mountainash_data.core.settings` instead. Also rewrite any `from mountainash_data.databases.constants` → `from mountainash_data.core.constants`. - -Do NOT touch imports from `mountainash_settings` (the external sister package) — only the in-package imports. - -- [ ] **Step 3: Replace each old settings file with a shim** - -For each of the 16 files in `src/mountainash_data/databases/settings/` (excluding `__init__.py` which gets special handling below), overwrite the contents with: - -```python -"""DEPRECATED: import from mountainash_data.core.settings. instead.""" - -from mountainash_data.core.settings. import * # noqa: F401,F403 -``` - -Where `` is the file's name without `.py`. For each shim, after writing it, also add explicit re-exports of every public class/symbol the original file exported (read the corresponding `core/settings/.py` to enumerate them). - -The 16 files are: -- `__init__.py` -- `base.py` -- `exceptions.py` -- `templates.py` -- `bigquery.py` -- `duckdb.py` -- `motherduck.py` -- `mssql.py` -- `mysql.py` -- `postgresql.py` -- `pyiceberg_rest.py` -- `pyspark.py` -- `redshift.py` -- `snowflake.py` -- `sqlite.py` -- `trino.py` - -For `databases/settings/__init__.py` specifically, the shim should re-export everything `core/settings/__init__.py` exports: - -```python -"""DEPRECATED: import from mountainash_data.core.settings instead.""" - -from mountainash_data.core.settings import * # noqa: F401,F403 -``` - -Then read `core/settings/__init__.py` and add explicit re-exports for every class name in its `__all__` (or every class it imports). - -- [ ] **Step 4: Run tests** - -Run: `hatch run test:test` -Expected: PASS, count unchanged. - -If `test_settings_parametrized.py` fails with import errors, the shim is missing a symbol. Add it. - -- [ ] **Step 5: Run lint** - -Run: `hatch run ruff:check src/mountainash_data/core/settings/` -Expected: PASS. - -- [ ] **Step 6: Commit** - -```bash -git add src/mountainash_data/core/settings/ src/mountainash_data/databases/settings/ -git commit -m "refactor(data): move settings to core/settings/ with shims - -16 files moved verbatim. databases/settings/ retains shims that -re-export from the new location, removed in Phase 6." -``` - ---- - -## Phase 3 — Iceberg backend (D1.b: split, deduplicate) - -**Files touched:** -- Create: `src/mountainash_data/backends/__init__.py` -- Create: `src/mountainash_data/backends/iceberg/__init__.py` -- Create: `src/mountainash_data/backends/iceberg/connection.py` -- Create: `src/mountainash_data/backends/iceberg/operations.py` -- Create: `src/mountainash_data/backends/iceberg/_types.py` -- Create: `src/mountainash_data/backends/iceberg/inspect.py` -- Create: `src/mountainash_data/backends/iceberg/backend.py` -- Create: `src/mountainash_data/backends/iceberg/catalogs/__init__.py` -- Create: `src/mountainash_data/backends/iceberg/catalogs/rest.py` -- Create: `tests/test_unit/backends/iceberg/test_backend.py` -- Create: `tests/test_unit/backends/iceberg/test_inspect.py` -- Modify: `src/mountainash_data/databases/connections/pyiceberg/base_pyiceberg_connection.py` (becomes shim) -- Modify: `src/mountainash_data/databases/connections/pyiceberg/pyiceberg_rest_connection.py` (becomes shim) -- Modify: `src/mountainash_data/databases/operations/pyiceberg/base_pyiceberg_operations.py` (becomes shim) -- Modify: `src/mountainash_data/databases/operations/pyiceberg/pyiceberg_rest_operations.py` (becomes shim) - -### Task 3.1: Audit existing iceberg test coverage - -- [ ] **Step 1: Inventory existing iceberg tests** - -Run: `find tests -path '*pyiceberg*' -o -path '*iceberg*' | head` -Expected: likely zero results (the audit found no iceberg-specific test files). - -- [ ] **Step 2: Document the gap** - -Create `tests/test_unit/backends/iceberg/__init__.py` (empty) and `tests/test_unit/backends/iceberg/COVERAGE_GAP.md`: - -```markdown -# Iceberg test coverage gap - -Before Phase 3 (writing-plans audit), no tests existed for any of: - -- `databases/connections/pyiceberg/base_pyiceberg_connection.py` (884 LOC) -- `databases/connections/pyiceberg/pyiceberg_rest_connection.py` (205 LOC) -- `databases/operations/pyiceberg/base_pyiceberg_operations.py` (868 LOC) -- `databases/operations/pyiceberg/pyiceberg_rest_operations.py` (207 LOC) - -The Phase 3 refactor proceeds *without* a regression net for iceberg. -This is acceptable because the user (sole consumer) confirmed iceberg -is in prototype use only. Tests added during this phase target the new -shape (IcebergBackend protocol, inspection model conversion) rather -than reproducing legacy behavior. - -If iceberg moves to production use later, a separate hardening pass -should add tests for the salvaged operations. -``` - -- [ ] **Step 3: Commit the gap doc** - -```bash -git add tests/test_unit/backends/iceberg/__init__.py tests/test_unit/backends/iceberg/COVERAGE_GAP.md -git commit -m "docs(test): document iceberg test coverage gap before Phase 3 refactor" -``` - -### Task 3.2: Initial verbatim copy of iceberg files into new layout - -This task is a **bulk move** to get all the source code into `backends/iceberg/`. Deduplication and splitting happen in Task 3.3. - -- [ ] **Step 1: Create the new directory structure** - -```bash -mkdir -p src/mountainash_data/backends/iceberg/catalogs -touch src/mountainash_data/backends/__init__.py -touch src/mountainash_data/backends/iceberg/__init__.py -touch src/mountainash_data/backends/iceberg/catalogs/__init__.py -``` - -- [ ] **Step 2: Copy the four source files into a staging location inside the new dir** - -```bash -cp src/mountainash_data/databases/connections/pyiceberg/base_pyiceberg_connection.py \ - src/mountainash_data/backends/iceberg/_legacy_connection.py -cp src/mountainash_data/databases/operations/pyiceberg/base_pyiceberg_operations.py \ - src/mountainash_data/backends/iceberg/_legacy_operations.py -cp src/mountainash_data/databases/connections/pyiceberg/pyiceberg_rest_connection.py \ - src/mountainash_data/backends/iceberg/_legacy_rest_connection.py -cp src/mountainash_data/databases/operations/pyiceberg/pyiceberg_rest_operations.py \ - src/mountainash_data/backends/iceberg/_legacy_rest_operations.py -``` - -The `_legacy_*` files exist for the duration of Task 3.3 only. They are deleted at the end of Task 3.3. - -- [ ] **Step 3: Update internal imports in the legacy copies** - -Run: `grep -n "from mountainash_data" src/mountainash_data/backends/iceberg/_legacy_*.py` - -For every match: rewrite `mountainash_data.databases.connections.base_db_connection` → `mountainash_data.core.connection`, `mountainash_data.databases.constants` → `mountainash_data.core.constants`, `mountainash_data.databases.settings` → `mountainash_data.core.settings`. Leave any cross-`_legacy_*` imports as relative imports within the new directory (e.g., `from mountainash_data.backends.iceberg._legacy_connection import ...`). - -- [ ] **Step 4: Smoke-import the staging files** - -Create a temporary smoke test `tests/test_unit/backends/iceberg/test_smoke_import.py`: - -```python -"""Smoke test: verify the legacy iceberg files can be imported in their new location.""" - -def test_legacy_imports(): - from mountainash_data.backends.iceberg import ( - _legacy_connection, - _legacy_operations, - _legacy_rest_connection, - _legacy_rest_operations, - ) - assert _legacy_connection is not None - assert _legacy_operations is not None - assert _legacy_rest_connection is not None - assert _legacy_rest_operations is not None -``` - -Run: `hatch run test:test-quick tests/test_unit/backends/iceberg/test_smoke_import.py -v` -Expected: PASS. - -If it fails with `ImportError`, fix the imports inside the legacy files until it passes. Do NOT proceed to Task 3.3 until the smoke test is green. - -- [ ] **Step 5: Commit the staging copies** - -```bash -git add src/mountainash_data/backends/ tests/test_unit/backends/iceberg/test_smoke_import.py -git commit -m "refactor(data/iceberg): stage legacy iceberg files in new backends/ location - -Verbatim copies as _legacy_*.py. Deduplication and split into -connection/operations/types happens in the next task." -``` - -### Task 3.3: Deduplicate and split (the substantial refactor) - -This is the hardest task in the plan. The two legacy files share methods (the spec confirmed the previous split was half done). This task identifies duplicates, picks canonical versions, and reorganizes by responsibility. - -- [ ] **Step 1: Build a method inventory of both legacy files** - -Read `src/mountainash_data/backends/iceberg/_legacy_connection.py` and `_legacy_operations.py`. Produce a markdown table in scratch (do not commit) with columns: `method_name`, `in_legacy_connection`, `in_legacy_operations`, `responsibility` where responsibility is one of: `lifecycle` (connect/disconnect/catalog navigation), `mutation` (create/drop/insert/upsert/truncate/views), `inspection` (list/describe), `types` (Iceberg→PyArrow conversion), `helper`. - -- [ ] **Step 2: Identify duplicates and pick canonical versions** - -For every method that appears in both files, read both implementations. Pick the more complete / more recently modified version as canonical. Note the choice in the scratch table. If the implementations diverge in non-trivial ways (different error handling, different retry logic), STOP and surface the diff to the user — do not silently choose. - -- [ ] **Step 3: Create `backends/iceberg/_types.py`** - -Move all Iceberg→PyArrow type conversion helpers (the ~15 type variants the audit identified) from `_legacy_connection.py` and `_legacy_operations.py` (whichever has the canonical version) into a new file: - -Create `src/mountainash_data/backends/iceberg/_types.py`: - -```python -"""Iceberg ↔ PyArrow type conversion helpers. - -Extracted from the legacy base_pyiceberg_connection.py / base_pyiceberg_operations.py -during the Phase 3 deduplication. These functions take Iceberg type -objects and return their PyArrow equivalents (and vice versa where -needed). -""" - -from __future__ import annotations - -# Move the canonical type-conversion functions here. -# Function names to preserve (confirm by reading the legacy files): -# - _iceberg_to_pyarrow_schema -# - _iceberg_to_pyarrow_type -# - _pyarrow_to_iceberg_type -# - any helpers for nested struct/map/list conversion -# -# DO NOT change function signatures or behavior. This is a pure move. -``` - -Replace the placeholder block with the actual canonical functions copied from the legacy files. Preserve docstrings. - -- [ ] **Step 4: Create `backends/iceberg/connection.py`** - -Create `src/mountainash_data/backends/iceberg/connection.py` and move into it everything classified as `lifecycle` or `inspection` from the canonical legacy versions: - -```python -"""Iceberg connection: catalog/namespace lifecycle and read-side inspection. - -Created in Phase 3 by deduplicating and splitting the legacy -base_pyiceberg_connection.py and base_pyiceberg_operations.py. -""" - -from __future__ import annotations - -import typing as t - -from mountainash_data.backends.iceberg._types import ( - # import the type helpers needed by inspection -) -from mountainash_data.core.inspection import ( - CatalogInfo, - NamespaceInfo, - TableInfo, -) - -# Move into this module: -# - the connection lifecycle (open, close, catalog handle accessor) -# - list_namespaces, list_tables -# - load_table / get_table (the read accessor used by inspection and ops) -# - any caching infrastructure that supports the above (schema cache, etc.) -# - inspection helpers that build TableInfo from a loaded iceberg Table -# -# Class shape: keep an `IcebergConnectionBase` (or whatever the canonical -# legacy name was) — this is the class IcebergRestConnection subclasses -# in catalogs/rest.py. -``` - -Replace the placeholder with the actual extracted code. Update internal references: methods that previously called `self._upsert(...)` (which is now in operations.py) need to be left in place — operations.py will keep them on the same class instance via composition or mixin (see Step 6 below). - -- [ ] **Step 5: Create `backends/iceberg/operations.py`** - -Create `src/mountainash_data/backends/iceberg/operations.py` and move into it everything classified as `mutation`: - -```python -"""Iceberg table mutations: create, drop, insert, upsert, truncate, view ops. - -Created in Phase 3 by deduplicating and splitting the legacy -base_pyiceberg_connection.py and base_pyiceberg_operations.py. - -These functions take an active connection (or table handle) and perform -mutations. They are NOT class methods — they are module-level functions -that the connection class composes when consumers call e.g. -connection.insert(...). -""" - -from __future__ import annotations - -import typing as t - -from mountainash_data.backends.iceberg._types import ( - # import type helpers needed by mutations -) - -# Move into this module: -# - create_table / drop_table -# - insert / upsert / truncate (with their snapshot retry logic) -# - create_view / drop_view -# - any helpers specific to mutations (natural key validation, etc.) -# -# Pattern: convert each from a method on the legacy class into a -# function that takes the connection as its first arg: -# -# def insert(connection, table_name: str, df, namespace: str | None = None) -> None: -# table = connection.load_table(table_name, namespace) -# ... -# -# Then in connection.py, the IcebergConnectionBase class exposes -# thin wrappers: -# -# def insert(self, table_name, df, namespace=None): -# from mountainash_data.backends.iceberg import operations -# return operations.insert(self, table_name, df, namespace) -``` - -Replace the placeholder with the actual extracted functions. The conversion pattern (method → function-taking-self-equivalent) is described in the docstring above. - -- [ ] **Step 6: Wire connection.py to call operations.py** - -Read `backends/iceberg/connection.py`. For every mutation method that used to live there but now lives in `operations.py`, add a thin wrapper method on `IcebergConnectionBase` that delegates: - -```python -def insert(self, table_name: str, df, namespace: str | None = None) -> None: - from mountainash_data.backends.iceberg import operations - return operations.insert(self, table_name, df, namespace) -``` - -Use a local import inside each wrapper to avoid circular import issues between `connection.py` and `operations.py` (operations.py needs to type-hint `Connection` but doesn't need to import the class). - -Do this for every mutation: `insert`, `upsert`, `truncate`, `create_table`, `drop_table`, `create_view`, `drop_view`. Confirm by reading both files that no mutation is orphaned. - -- [ ] **Step 7: Move REST-specific code into `catalogs/rest.py`** - -Create `src/mountainash_data/backends/iceberg/catalogs/rest.py`: - -```python -"""REST catalog implementation. - -Merges the legacy pyiceberg_rest_connection.py and pyiceberg_rest_operations.py -into a single concrete connection class for the REST catalog. -""" - -from __future__ import annotations - -from mountainash_data.backends.iceberg.connection import IcebergConnectionBase -# Add other imports as needed. - - -class IcebergRestConnection(IcebergConnectionBase): - # Move the concrete REST initialization, any REST-specific overrides - # (the audit identified _list_tables and _upsert overrides in - # pyiceberg_rest_operations.py — those go here as method overrides - # OR as registry entries depending on whether they're worth keeping - # as overrides). - pass -``` - -Replace the placeholder with the actual code from `_legacy_rest_connection.py` and `_legacy_rest_operations.py`. If the legacy `_upsert` override in `_legacy_rest_operations.py` (with natural key validation per the audit) is meaningfully different from the canonical `upsert` in `operations.py`, preserve it as an override on `IcebergRestConnection`. - -- [ ] **Step 8: Create `backends/iceberg/inspect.py`** - -Create `src/mountainash_data/backends/iceberg/inspect.py`: - -```python -"""Iceberg → core.inspection conversion. - -Helpers that take pyiceberg Table objects and produce TableInfo / -NamespaceInfo / CatalogInfo dataclasses from core.inspection. -""" - -from __future__ import annotations - -import typing as t - -from mountainash_data.core.inspection import ( - CatalogInfo, - ColumnInfo, - NamespaceInfo, - TableInfo, -) - - -def table_to_info( - iceberg_table, - *, - name: str, - namespace: str | None = None, - catalog: str | None = None, -) -> TableInfo: - """Convert a pyiceberg Table object into a TableInfo.""" - columns = [ - ColumnInfo( - name=field.name, - type_name=str(field.field_type), - nullable=not field.required, - ) - for field in iceberg_table.schema().fields - ] - return TableInfo( - name=name, - columns=columns, - namespace=namespace, - catalog=catalog, - ) -``` - -(Confirm field/method names by reading the legacy `_legacy_connection.py` — it already builds schemas from iceberg Tables, so the existing logic is the source of truth.) - -- [ ] **Step 9: Create `backends/iceberg/backend.py` — the IcebergBackend Protocol implementation** - -Create `src/mountainash_data/backends/iceberg/backend.py`: - -```python -"""IcebergBackend — implements core.protocol.Backend for iceberg catalogs.""" - -from __future__ import annotations - -import typing as t - -from mountainash_data.backends.iceberg.catalogs.rest import IcebergRestConnection -from mountainash_data.backends.iceberg.connection import IcebergConnectionBase -from mountainash_data.core.protocol import Backend, Connection - - -_CATALOG_REGISTRY: dict[str, type[IcebergConnectionBase]] = { - "rest": IcebergRestConnection, -} - - -class IcebergBackend: - """Iceberg backend factory. - - Construction takes a catalog type (e.g. 'rest') and config. connect() - returns a live IcebergConnection that implements core.protocol.Connection. - """ - - name = "iceberg" - - def __init__(self, catalog: str, **config: t.Any): - if catalog not in _CATALOG_REGISTRY: - raise KeyError( - f"Unknown iceberg catalog type {catalog!r}. " - f"Available: {sorted(_CATALOG_REGISTRY)}" - ) - self._catalog_cls = _CATALOG_REGISTRY[catalog] - self._config = config - - def connect(self) -> Connection: - return self._catalog_cls(**self._config) -``` - -- [ ] **Step 10: Write a test for IcebergBackend instantiation and protocol satisfaction** - -Create `tests/test_unit/backends/iceberg/test_backend.py`: - -```python -"""Tests for IcebergBackend factory.""" - -import pytest - -from mountainash_data.backends.iceberg.backend import IcebergBackend -from mountainash_data.core.protocol import Backend - - -def test_iceberg_backend_satisfies_protocol(): - backend = IcebergBackend(catalog="rest", uri="http://localhost:8181") - assert isinstance(backend, Backend) - assert backend.name == "iceberg" - - -def test_unknown_catalog_raises(): - with pytest.raises(KeyError, match="Unknown iceberg catalog"): - IcebergBackend(catalog="bogus") - - -def test_iceberg_backend_carries_config(): - backend = IcebergBackend(catalog="rest", uri="http://localhost:8181", token="abc") - assert backend._config == {"uri": "http://localhost:8181", "token": "abc"} -``` - -- [ ] **Step 11: Run the new tests** - -Run: `hatch run test:test-quick tests/test_unit/backends/iceberg/test_backend.py -v` -Expected: PASS, 3 tests green. - -If `test_iceberg_backend_satisfies_protocol` fails because the structural check finds missing methods, the issue is that `IcebergConnectionBase` (returned by `connect()`) doesn't yet expose all six Connection protocol methods (`list_namespaces`, `list_tables`, `inspect_table`, `inspect_namespace`, `inspect_catalog`, `close`). Add the missing ones — they likely wrap existing legacy methods or use `inspect.py` helpers. Read the failure carefully and add only what's missing. - -Note: this test does NOT actually call `.connect()` — it just verifies the type and that construction validates. Actual end-to-end connection tests against a real iceberg catalog are out of scope for this refactor (the iceberg coverage gap is documented in Task 3.1). - -- [ ] **Step 12: Delete the `_legacy_*.py` staging files** - -Once `connection.py`, `operations.py`, `_types.py`, `inspect.py`, `backend.py`, and `catalogs/rest.py` collectively contain everything from the four legacy files, delete the staging copies: - -```bash -rm src/mountainash_data/backends/iceberg/_legacy_connection.py -rm src/mountainash_data/backends/iceberg/_legacy_operations.py -rm src/mountainash_data/backends/iceberg/_legacy_rest_connection.py -rm src/mountainash_data/backends/iceberg/_legacy_rest_operations.py -rm tests/test_unit/backends/iceberg/test_smoke_import.py -``` - -- [ ] **Step 13: Replace the original four files with shims** - -Overwrite `src/mountainash_data/databases/connections/pyiceberg/base_pyiceberg_connection.py`: - -```python -"""DEPRECATED: import from mountainash_data.backends.iceberg.connection instead.""" - -from mountainash_data.backends.iceberg.connection import * # noqa: F401,F403 -from mountainash_data.backends.iceberg.connection import IcebergConnectionBase # noqa: F401 -# Add any other public symbols the original file exported. -``` - -Read the original file's public surface (class names, function names) and add explicit re-exports for each. - -Repeat for: -- `databases/connections/pyiceberg/pyiceberg_rest_connection.py` → re-export from `backends/iceberg/catalogs/rest` -- `databases/operations/pyiceberg/base_pyiceberg_operations.py` → re-export from `backends/iceberg/operations` (note: operations are now functions, not class methods — if any consumer was importing a class, this shim will not satisfy them and you'll see test failures pinpointing the affected importer) -- `databases/operations/pyiceberg/pyiceberg_rest_operations.py` → re-export from `backends/iceberg/catalogs/rest` - -- [ ] **Step 14: Run the full test suite** - -Run: `hatch run test:test` -Expected: PASS, baseline count. - -If anything fails: -- Import errors → a shim is missing a symbol; add it. -- Class-not-found errors on operations → an old test or factory was importing `BasePyIcebergOperations` as a class. The simplest fix is to keep a thin compatibility class in `databases/operations/pyiceberg/base_pyiceberg_operations.py` (the shim) that wraps the new module-level functions. Add it only if needed. -- Behavior failures (assertions on returned values) → the deduplication picked the wrong canonical version of a method. Stop and re-read both legacy versions in the git history to compare. - -- [ ] **Step 15: Run lint** - -Run: `hatch run ruff:check src/mountainash_data/backends/` -Expected: PASS. - -- [ ] **Step 16: Commit** - -```bash -git add -A -git commit -m "refactor(data/iceberg): split and deduplicate iceberg backend (D1.b) - -Connection lifecycle in backends/iceberg/connection.py. -Table mutations in backends/iceberg/operations.py (now module -functions, called via thin wrappers on the connection class). -Iceberg→PyArrow types in _types.py. -REST catalog in catalogs/rest.py. -Inspection model conversion in inspect.py. -IcebergBackend Protocol implementation in backend.py. - -Closes the half-done split that left duplicate methods between the -legacy connection (884 LOC) and operations (868 LOC) files. Original -locations retained as shims through Phase 6. - -to_relation() not implemented — gap documented in spec; requires -mountainash-expressions to gain an iceberg adapter." -``` - ---- - -## Phase 4 — Ibis backend (the largest phase) - -**Files touched:** -- Create: `src/mountainash_data/backends/ibis/__init__.py` -- Create: `src/mountainash_data/backends/ibis/connection.py` (from `base_ibis_connection.py`) -- Create: `src/mountainash_data/backends/ibis/operations.py` (merges base + mixins) -- Create: `src/mountainash_data/backends/ibis/inspect.py` -- Create: `src/mountainash_data/backends/ibis/backend.py` -- Create: `src/mountainash_data/backends/ibis/dialects/__init__.py` -- Create: `src/mountainash_data/backends/ibis/dialects/_registry.py` -- Modify: 13 ibis connection shims, 4 ibis ops shims (duckdb/sqlite/motherduck/trino) -- Modify: `factories/operations_factory.py` (point at new IbisBackend) - -### Task 4.1: Audit ibis test coverage and resolve D3 (HYBRID mode) - -- [ ] **Step 1: Inventory ibis tests** - -Run: `find tests -path '*ibis*' -name '*.py'` -Expected (based on the audit): a handful of files including `test_ibis_backends.py`, `test_connection_lifecycle.py`, `test_base_ibis_operations.py`, `test_upsert_and_indexes.py`. - -- [ ] **Step 2: Run the existing ibis tests in isolation** - -Run: `hatch run test:test-quick tests/test_unit/databases/connections/ibis/ tests/test_unit/databases/operations/ -v` -Expected: PASS, record the test count. - -This is the regression net Phase 4 must keep green. - -- [ ] **Step 3: Resolve D3 — grep for `_ibis_connection_mode` usage** - -Run: `grep -rn "_ibis_connection_mode\|ibis_connection_mode\|HYBRID" src/` - -Read every match. Determine: is `HYBRID` set anywhere except in `trino_ibis_operations.py`? If yes, it's a general capability and goes in the registry as a per-dialect field. If no, it's trino-specific and goes in trino's `DialectSpec` entry only. - -Record the answer in a scratch note for Task 4.4. - -### Task 4.2: Define the DialectSpec dataclass - -- [ ] **Step 1: Write the failing test** - -Create `tests/test_unit/backends/ibis/__init__.py` (empty) and `tests/test_unit/backends/ibis/test_dialect_spec.py`: - -```python -"""Tests for the DialectSpec dataclass — the data-driven replacement -for the per-backend connection class explosion.""" - -from mountainash_data.backends.ibis.dialects._registry import ( - DialectSpec, - DIALECTS, -) - - -def test_dialect_spec_minimal(): - spec = DialectSpec( - ibis_backend_name="sqlite", - connection_mode="DIRECT", - connection_string_scheme="sqlite", - ) - assert spec.ibis_backend_name == "sqlite" - assert spec.connection_mode == "DIRECT" - assert spec.get_index_exists_sql is None - assert spec.get_list_indexes_sql is None - - -def test_dialect_spec_with_capability_hooks(): - def fake_index_sql(table_name, index_name): - return f"SELECT 1 FROM {table_name}" - - spec = DialectSpec( - ibis_backend_name="duckdb", - connection_mode="DIRECT", - connection_string_scheme="duckdb", - get_index_exists_sql=fake_index_sql, - ) - assert spec.get_index_exists_sql is not None - assert spec.get_index_exists_sql("users", "idx_users_id") == "SELECT 1 FROM users" - - -def test_registry_contains_all_12_backends(): - expected = { - "sqlite", "duckdb", "motherduck", "postgres", "mysql", "mssql", - "oracle", "snowflake", "bigquery", "redshift", "trino", "pyspark", - } - assert set(DIALECTS.keys()) == expected - - -def test_registry_entries_are_dialect_specs(): - for name, spec in DIALECTS.items(): - assert isinstance(spec, DialectSpec), f"{name} entry is not a DialectSpec" - assert spec.ibis_backend_name, f"{name} missing ibis_backend_name" -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `hatch run test:test-quick tests/test_unit/backends/ibis/test_dialect_spec.py -v` -Expected: FAIL with `ModuleNotFoundError: mountainash_data.backends.ibis.dialects._registry` - -- [ ] **Step 3: Create the registry module with DialectSpec and stub entries** - -```bash -mkdir -p src/mountainash_data/backends/ibis/dialects -touch src/mountainash_data/backends/ibis/__init__.py -touch src/mountainash_data/backends/ibis/dialects/__init__.py -``` - -Create `src/mountainash_data/backends/ibis/dialects/_registry.py`: - -```python -"""Data-driven dialect registry. Replaces the 13 per-backend connection -classes from databases/connections/ibis/. - -Each entry is a DialectSpec containing the connection-builder callable, -ibis backend name, connection mode, and any backend-specific capability -hooks (e.g. dialect-specific index introspection SQL). -""" - -from __future__ import annotations - -from dataclasses import dataclass, field -import typing as t - - -# Capability hook signatures -GetIndexExistsSql = t.Callable[[str, str], str] # (table_name, index_name) -> SQL -GetListIndexesSql = t.Callable[[str], str] # (table_name) -> SQL -ConnectionBuilder = t.Callable[..., t.Any] # (**config) -> ibis backend connection - - -@dataclass(frozen=True) -class DialectSpec: - """Per-dialect configuration and capability hooks.""" - - ibis_backend_name: str - connection_mode: str - connection_string_scheme: str - connection_builder: t.Optional[ConnectionBuilder] = None - get_index_exists_sql: t.Optional[GetIndexExistsSql] = None - get_list_indexes_sql: t.Optional[GetListIndexesSql] = None - extras: t.Mapping[str, t.Any] = field(default_factory=dict) - - -# Stub entries — populated with real connection_builder callables in -# Task 4.4 by salvaging logic from the per-backend connection files. -DIALECTS: dict[str, DialectSpec] = { - "sqlite": DialectSpec( - ibis_backend_name="sqlite", - connection_mode="DIRECT", - connection_string_scheme="sqlite", - ), - "duckdb": DialectSpec( - ibis_backend_name="duckdb", - connection_mode="DIRECT", - connection_string_scheme="duckdb", - ), - "motherduck": DialectSpec( - ibis_backend_name="duckdb", - connection_mode="DIRECT", - connection_string_scheme="md", - ), - "postgres": DialectSpec( - ibis_backend_name="postgres", - connection_mode="DIRECT", - connection_string_scheme="postgresql", - ), - "mysql": DialectSpec( - ibis_backend_name="mysql", - connection_mode="DIRECT", - connection_string_scheme="mysql", - ), - "mssql": DialectSpec( - ibis_backend_name="mssql", - connection_mode="DIRECT", - connection_string_scheme="mssql", - ), - "oracle": DialectSpec( - ibis_backend_name="oracle", - connection_mode="DIRECT", - connection_string_scheme="oracle", - ), - "snowflake": DialectSpec( - ibis_backend_name="snowflake", - connection_mode="DIRECT", - connection_string_scheme="snowflake", - ), - "bigquery": DialectSpec( - ibis_backend_name="bigquery", - connection_mode="DIRECT", - connection_string_scheme="bigquery", - ), - "redshift": DialectSpec( - ibis_backend_name="postgres", # Redshift uses postgres protocol - connection_mode="DIRECT", - connection_string_scheme="redshift", - ), - "trino": DialectSpec( - ibis_backend_name="trino", - connection_mode="HYBRID", # confirmed by Task 4.1 D3 grep - connection_string_scheme="trino", - ), - "pyspark": DialectSpec( - ibis_backend_name="pyspark", - connection_mode="DIRECT", - connection_string_scheme="spark", - ), -} -``` - -If Task 4.1 step 3 found that HYBRID is general (not trino-only), update the relevant entries; otherwise keep it on trino only as shown. - -Note on `connection_string_scheme`: confirm each scheme by reading the corresponding `databases/connections/ibis/_ibis_connection.py` file and copying the actual `connection_string_scheme` constant. The values above are educated guesses — replace any that disagree with the source. - -- [ ] **Step 4: Run tests** - -Run: `hatch run test:test-quick tests/test_unit/backends/ibis/test_dialect_spec.py -v` -Expected: PASS, 4 tests green. - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/backends/ibis/ tests/test_unit/backends/ibis/ -git commit -m "feat(data/ibis): add DialectSpec and stub registry for 12 backends - -DialectSpec is the data-driven replacement for the 13 per-backend -connection classes. Connection builders and capability hooks are -populated in Task 4.4." -``` - -### Task 4.3: Move `base_ibis_connection.py` to `backends/ibis/connection.py` - -The legacy `BaseIbisConnection` class is the shared connection logic. Move it verbatim, then refactor in Task 4.4 to consume DialectSpec. - -- [ ] **Step 1: Copy and update imports** - -```bash -cp src/mountainash_data/databases/connections/ibis/base_ibis_connection.py \ - src/mountainash_data/backends/ibis/connection.py -``` - -Read the new `src/mountainash_data/backends/ibis/connection.py`. Rewrite imports: -- `mountainash_data.databases.connections.base_db_connection` → `mountainash_data.core.connection` -- `mountainash_data.databases.constants` → `mountainash_data.core.constants` -- `mountainash_data.databases.settings` → `mountainash_data.core.settings` - -Leave the class name (`BaseIbisConnection` or whatever it is) unchanged for now. - -- [ ] **Step 2: Replace original with shim** - -Overwrite `src/mountainash_data/databases/connections/ibis/base_ibis_connection.py`: - -```python -"""DEPRECATED: import from mountainash_data.backends.ibis.connection instead.""" - -from mountainash_data.backends.ibis.connection import * # noqa: F401,F403 -from mountainash_data.backends.ibis.connection import BaseIbisConnection # noqa: F401 -``` - -Add explicit re-exports for any other public symbols (read the file). - -- [ ] **Step 3: Run tests** - -Run: `hatch run test:test` -Expected: PASS, baseline count. - -- [ ] **Step 4: Commit** - -```bash -git add -A -git commit -m "refactor(data/ibis): move BaseIbisConnection to backends/ibis/connection.py - -Moved verbatim, original location retained as shim. Refactored to -consume DialectSpec in the next task." -``` - -### Task 4.4: Salvage per-backend connection logic into DialectSpec entries - -This task reads each of the 13 per-backend connection files and extracts the connection-building logic into the registry. After this task, the per-backend files become shims pointing at `IbisBackend(dialect=...)`. - -For brevity, this task uses **postgres** as the worked example. Repeat the same pattern for the other 12 backends (sqlite, duckdb, motherduck, mysql, mssql, oracle, snowflake, bigquery, redshift, trino, pyspark, and any redshift quirks). - -- [ ] **Step 1: Read postgres_ibis_connection.py and identify the connection-building logic** - -Read `src/mountainash_data/databases/connections/ibis/postgres_ibis_connection.py`. The audit identified this as 94 LOC of concrete connection logic. Find the method (likely `_connect` or `connect`) that takes settings and returns an ibis backend connection. Note any per-session option setters (the audit mentioned this for postgres). - -- [ ] **Step 2: Add a connection_builder function for postgres in the registry** - -Edit `src/mountainash_data/backends/ibis/dialects/_registry.py`. Above the `DIALECTS` dict, add: - -```python -def _build_postgres_connection(**config: t.Any) -> t.Any: - """Build a postgres ibis connection. Salvaged from - databases/connections/ibis/postgres_ibis_connection.py.""" - import ibis - # Extract the actual connection-building logic from the legacy file. - # Typical shape: - # conn = ibis.postgres.connect( - # host=config["host"], - # port=config.get("port", 5432), - # user=config["user"], - # password=config["password"], - # database=config["database"], - # ) - # # Apply any per-session options the legacy file applied. - # return conn - raise NotImplementedError("Replace with actual logic from legacy file") -``` - -Replace the `raise NotImplementedError` with the actual code copied (and adapted for the new function signature) from `postgres_ibis_connection.py`. If the legacy file uses a settings object, accept it as a kwarg or as `**config` and call its accessors. - -Update the postgres entry in `DIALECTS`: - -```python -"postgres": DialectSpec( - ibis_backend_name="postgres", - connection_mode="DIRECT", - connection_string_scheme="postgresql", - connection_builder=_build_postgres_connection, -), -``` - -- [ ] **Step 3: Repeat Step 1–2 for the other 12 dialects** - -For each of: `sqlite`, `duckdb`, `motherduck`, `mysql`, `mssql`, `oracle`, `snowflake`, `bigquery`, `redshift`, `trino`, `pyspark`: - -1. Read `databases/connections/ibis/_ibis_connection.py` -2. Add a `_build__connection(**config)` function above `DIALECTS` -3. Wire it into the matching `DialectSpec` entry - -Pay special attention to: -- **motherduck** (202 LOC) — has retries, the audit warned it's verbose. Preserve the retry logic. -- **redshift** — uses `ibis.postgres.connect` under the hood per the audit. The connection_builder may share code with postgres. -- **bigquery** (99 LOC) — auth method matters (service account vs ADC). - -- [ ] **Step 4: Run the existing connection lifecycle test** - -Run: `hatch run test:test-quick tests/test_unit/databases/connections/ibis/test_connection_lifecycle.py -v` -Expected: PASS — the tests still drive the legacy classes via shims, which still work because we haven't deleted the per-backend connection files yet. This confirms the registry additions didn't break anything. - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/backends/ibis/dialects/_registry.py -git commit -m "feat(data/ibis): salvage per-backend connection-builder logic into DialectSpec - -Each of the 12 dialects now has a connection_builder function in the -registry, copied (and adapted for the new function signature) from the -corresponding databases/connections/ibis/_ibis_connection.py. - -The per-backend files still exist; they become shims in Task 4.6." -``` - -### Task 4.5: Move and merge ibis operations into `backends/ibis/operations.py` - -- [ ] **Step 1: Copy the core operations file and the two mixins** - -```bash -cp src/mountainash_data/databases/operations/ibis/base_ibis_operations.py \ - src/mountainash_data/backends/ibis/operations.py -``` - -- [ ] **Step 2: Update imports in `backends/ibis/operations.py`** - -Same import-rewriting as Task 4.3 step 1: `databases.constants` → `core.constants`, `databases.settings` → `core.settings`, `databases.connections.base_db_connection` → `core.connection`. - -- [ ] **Step 3: Read `_base_ibis_mixin.py` and fold its helpers into `operations.py`** - -Read `src/mountainash_data/databases/operations/ibis/_base_ibis_mixin.py` (98 LOC). It contains helper methods like `_generate_index_name`, `_format_qualified_table`, `_normalize_columns`. - -Convert each helper from a mixin method (`def _generate_index_name(self, ...)`) into a module-level function in `operations.py` (`def _generate_index_name(...)`). Where the helper used `self`, accept the needed values as parameters instead. - -If the canonical `BaseIbisOperations` class in `operations.py` was inheriting from `_BaseIbisMixin`, remove that base class — the helpers are now functions called directly. - -- [ ] **Step 4: Read `_duckdb_family_mixin.py` and add its functions to `operations.py` as capability hooks** - -Read `src/mountainash_data/databases/operations/ibis/_duckdb_family_mixin.py` (314 LOC). It contains DuckDB-family-specific index queries and helpers, used by duckdb, motherduck, and sqlite. - -Convert each method into a module-level function in `operations.py`: - -```python -def duckdb_get_index_exists_sql(table_name: str, index_name: str) -> str: - """Salvaged from _duckdb_family_mixin.py. - DuckDB stores indexes in the duckdb_indexes() function. - """ - return f"SELECT * FROM duckdb_indexes() WHERE table_name = '{table_name}' AND index_name = '{index_name}'" -``` - -Reproduce each of the family mixin's methods this way. Then read the three concrete files (`duckdb_ibis_operations.py`, `sqlite_ibis_operations.py`, `motherduck_ibis_operations.py`, `trino_ibis_operations.py`) and confirm what each adds on top of the mixin: - -- **duckdb** (66 LOC): the audit said it has dialect-specific index catalog SQL. If the SQL differs from `_duckdb_family_mixin`'s default, add a `duckdb_get_index_exists_sql` function with the duckdb-specific version. -- **sqlite** (64 LOC): adds sqlite-specific index queries against `sqlite_master`. Add `sqlite_get_index_exists_sql` and `sqlite_get_list_indexes_sql`. -- **motherduck** (80 LOC): adds `_list_tables` override and motherduck-specific queries. Add `motherduck_*` functions. -- **trino** (34 LOC): only sets connection mode (already handled in DialectSpec via Task 4.1). The file has no operations logic to salvage. - -- [ ] **Step 5: Wire the capability hooks into the dialect registry** - -Edit `src/mountainash_data/backends/ibis/dialects/_registry.py`. Import the capability functions: - -```python -from mountainash_data.backends.ibis.operations import ( - duckdb_get_index_exists_sql, - duckdb_get_list_indexes_sql, - sqlite_get_index_exists_sql, - sqlite_get_list_indexes_sql, - motherduck_get_index_exists_sql, - motherduck_get_list_indexes_sql, -) -``` - -Update the dialect entries to attach the hooks: - -```python -"duckdb": DialectSpec( - ibis_backend_name="duckdb", - connection_mode="DIRECT", - connection_string_scheme="duckdb", - connection_builder=_build_duckdb_connection, - get_index_exists_sql=duckdb_get_index_exists_sql, - get_list_indexes_sql=duckdb_get_list_indexes_sql, -), -"sqlite": DialectSpec( - ibis_backend_name="sqlite", - connection_mode="DIRECT", - connection_string_scheme="sqlite", - connection_builder=_build_sqlite_connection, - get_index_exists_sql=sqlite_get_index_exists_sql, - get_list_indexes_sql=sqlite_get_list_indexes_sql, -), -"motherduck": DialectSpec( - ibis_backend_name="duckdb", - connection_mode="DIRECT", - connection_string_scheme="md", - connection_builder=_build_motherduck_connection, - get_index_exists_sql=motherduck_get_index_exists_sql, - get_list_indexes_sql=motherduck_get_list_indexes_sql, -), -``` - -- [ ] **Step 6: Replace `_base_ibis_mixin.py` and `_duckdb_family_mixin.py` with shims** - -Overwrite `src/mountainash_data/databases/operations/ibis/_base_ibis_mixin.py`: - -```python -"""DEPRECATED: helpers are now module-level functions in -mountainash_data.backends.ibis.operations.""" - -from mountainash_data.backends.ibis.operations import ( # noqa: F401 - # explicitly re-export every salvaged helper, e.g.: - # _generate_index_name, - # _format_qualified_table, - # _normalize_columns, -) -``` - -Read `backends/ibis/operations.py` and add each helper name to the import list above. - -If any consumer was using `_BaseIbisMixin` as a class (e.g., as a base class for a custom subclass), the shim won't be sufficient. Add a fallback class definition: - -```python -class _BaseIbisMixin: - """DEPRECATED compatibility shim — methods now live as module-level - functions in mountainash_data.backends.ibis.operations.""" - pass -``` - -Repeat the same shim treatment for `_duckdb_family_mixin.py`. - -- [ ] **Step 7: Replace `base_ibis_operations.py` with a shim** - -Overwrite `src/mountainash_data/databases/operations/ibis/base_ibis_operations.py`: - -```python -"""DEPRECATED: import from mountainash_data.backends.ibis.operations instead.""" - -from mountainash_data.backends.ibis.operations import * # noqa: F401,F403 -from mountainash_data.backends.ibis.operations import BaseIbisOperations # noqa: F401 -``` - -Add explicit re-exports for any other public symbols (read the file). - -- [ ] **Step 8: Replace the 4 concrete ops files with shims** - -For `duckdb_ibis_operations.py`, `sqlite_ibis_operations.py`, `motherduck_ibis_operations.py`, `trino_ibis_operations.py`: - -Each becomes a shim. The class names they exported (e.g., `DuckDBIbisOperations`) need to remain importable for the factories. Since the per-backend ops classes added little beyond inheritance + a property override, the shim can re-export `BaseIbisOperations` under the old name: - -```python -"""DEPRECATED: per-backend ops classes are gone; the registry's -capability hooks attach backend-specific behavior to BaseIbisOperations.""" - -from mountainash_data.backends.ibis.operations import BaseIbisOperations as DuckDBIbisOperations # noqa: F401 -``` - -Repeat for sqlite/motherduck/trino with their respective class names. Read each original file to confirm the exact class name being aliased. - -- [ ] **Step 9: Update `databases/operations/ibis/__init__.py`** - -Read the file. Make sure all the class names it exports still resolve. Most should, via the shims above. - -- [ ] **Step 10: Run the existing operations tests** - -Run: `hatch run test:test-quick tests/test_unit/databases/operations/ -v` -Expected: PASS, baseline count for this directory. - -If `test_upsert_and_indexes.py` fails because it expected `_DuckDBFamilyOperationsMixin` to be a class with methods, the shim from Step 6 needs the fallback class definition (use the pattern shown there). Add stub methods that delegate to the new module-level functions if the test is calling methods on the class. - -- [ ] **Step 11: Commit** - -```bash -git add -A -git commit -m "refactor(data/ibis): merge operations into backends/ibis/operations.py - -base_ibis_operations.py (676 LOC of actual implementation), -_base_ibis_mixin.py (98 LOC), and _duckdb_family_mixin.py (314 LOC) -are now a single backends/ibis/operations.py module with helpers as -functions. Backend-specific operations (duckdb/sqlite/motherduck index -SQL) are attached to dialect entries via capability hooks. Per-backend -operations files retained as shims through Phase 6." -``` - -### Task 4.6: Replace the 13 per-backend connection files with shims - -- [ ] **Step 1: For each of the 13 files, write a shim** - -For each of `sqlite`, `duckdb`, `motherduck`, `postgres`, `mysql`, `mssql`, `oracle`, `snowflake`, `bigquery`, `redshift`, `trino`, `pyspark` (and the base file `base_ibis_connection.py` already handled in Task 4.3): - -Read the original `databases/connections/ibis/_ibis_connection.py` and identify the class name(s) it exported (e.g., `Postgres_IbisConnection`). - -Overwrite the file with: - -```python -"""DEPRECATED: use IbisBackend(dialect="") from -mountainash_data.backends.ibis.backend instead.""" - -from mountainash_data.backends.ibis.connection import BaseIbisConnection as Postgres_IbisConnection # noqa: F401 -``` - -The alias means existing imports of the old class name still work — they just get the base class, which is fine because all the per-backend logic is now in the registry rather than in subclasses. - -If a per-backend class had ANY method beyond `__init__` / property overrides, that logic was supposed to be salvaged in Task 4.4. Re-check by reading the file. If you find a method that wasn't salvaged, STOP and surface it. - -- [ ] **Step 2: Run tests** - -Run: `hatch run test:test` -Expected: PASS, baseline count. - -If `test_connection_lifecycle.py` fails, the most likely cause is that a salvaged `_build__connection` function in the registry doesn't match the legacy `_connect` method's behavior. Re-read the legacy file and the registry function side by side, find the discrepancy, fix. - -- [ ] **Step 3: Commit** - -```bash -git add -A -git commit -m "refactor(data/ibis): replace 13 per-backend connection files with shims - -Connection-building logic now lives in dialects/_registry.py as -DialectSpec.connection_builder callables. The 13 per-backend files -re-export BaseIbisConnection under the old class names so existing -imports keep working through Phase 6." -``` - -### Task 4.7: Create `backends/ibis/inspect.py` - -- [ ] **Step 1: Write the failing test** - -Create `tests/test_unit/backends/ibis/test_inspect.py`: - -```python -"""Tests for ibis → core.inspection conversion.""" - -import ibis - -from mountainash_data.backends.ibis.inspect import table_to_info -from mountainash_data.core.inspection import TableInfo - - -def test_table_to_info_from_ibis_table(): - # Build an in-memory sqlite ibis backend with one table - conn = ibis.sqlite.connect() - conn.create_table( - "users", - schema=ibis.schema({"id": "int64", "name": "string"}), - ) - table = conn.table("users") - info = table_to_info(table, name="users", namespace="main") - assert isinstance(info, TableInfo) - assert info.name == "users" - assert info.namespace == "main" - assert info.column_names == ["id", "name"] - assert info.columns[0].nullable is True - assert info.columns[0].type_name == "int64" -``` - -- [ ] **Step 2: Run the test** - -Run: `hatch run test:test-quick tests/test_unit/backends/ibis/test_inspect.py -v` -Expected: FAIL with `ModuleNotFoundError`. - -- [ ] **Step 3: Implement `inspect.py`** - -Create `src/mountainash_data/backends/ibis/inspect.py`: - -```python -"""Ibis → core.inspection conversion helpers.""" - -from __future__ import annotations - -import typing as t - -from mountainash_data.core.inspection import ( - CatalogInfo, - ColumnInfo, - NamespaceInfo, - TableInfo, -) - - -def table_to_info( - ibis_table, - *, - name: str, - namespace: str | None = None, - catalog: str | None = None, -) -> TableInfo: - """Convert an ibis Table object into a TableInfo.""" - schema = ibis_table.schema() - columns = [ - ColumnInfo( - name=col_name, - type_name=str(col_type), - nullable=col_type.nullable, - ) - for col_name, col_type in zip(schema.names, schema.types) - ] - return TableInfo( - name=name, - columns=columns, - namespace=namespace, - catalog=catalog, - ) -``` - -- [ ] **Step 4: Run the test** - -Run: `hatch run test:test-quick tests/test_unit/backends/ibis/test_inspect.py -v` -Expected: PASS. - -If the assertion on `nullable` fails with `True != False`, ibis may default to non-nullable. Adjust the test to match ibis's actual default rather than changing production behavior; pick whichever is correct by reading ibis's documentation for `Schema`. - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/backends/ibis/inspect.py tests/test_unit/backends/ibis/test_inspect.py -git commit -m "feat(data/ibis): add ibis → core.inspection conversion" -``` - -### Task 4.8: Create `backends/ibis/backend.py` — IbisBackend Protocol implementation - -- [ ] **Step 1: Write the failing test** - -Create `tests/test_unit/backends/ibis/test_backend.py`: - -```python -"""Tests for IbisBackend factory.""" - -import pytest - -from mountainash_data.backends.ibis.backend import IbisBackend -from mountainash_data.backends.ibis.dialects._registry import DIALECTS -from mountainash_data.core.protocol import Backend - - -def test_ibis_backend_satisfies_protocol(): - backend = IbisBackend(dialect="sqlite") - assert isinstance(backend, Backend) - assert backend.name == "ibis" - - -def test_unknown_dialect_raises(): - with pytest.raises(KeyError, match="Unknown ibis dialect"): - IbisBackend(dialect="bogus") - - -def test_all_registered_dialects_construct(): - for dialect_name in DIALECTS: - backend = IbisBackend(dialect=dialect_name) - assert backend.dialect == dialect_name - - -def test_in_memory_sqlite_connect_and_inspect(): - """End-to-end test with the only dialect that needs no external service.""" - backend = IbisBackend(dialect="sqlite", database=":memory:") - conn = backend.connect() - try: - # Should expose protocol methods even if no tables exist - assert conn.list_tables() == [] - finally: - conn.close() -``` - -- [ ] **Step 2: Run the test** - -Run: `hatch run test:test-quick tests/test_unit/backends/ibis/test_backend.py -v` -Expected: FAIL with `ModuleNotFoundError`. - -- [ ] **Step 3: Implement `backend.py`** - -Create `src/mountainash_data/backends/ibis/backend.py`: - -```python -"""IbisBackend — implements core.protocol.Backend for ibis-supported backends.""" - -from __future__ import annotations - -import typing as t - -from mountainash_data.backends.ibis.connection import BaseIbisConnection -from mountainash_data.backends.ibis.dialects._registry import DIALECTS, DialectSpec -from mountainash_data.core.protocol import Backend, Connection - - -class IbisBackend: - """Ibis backend factory. - - Construction takes a dialect name (e.g. 'postgres') and config. - connect() returns a live connection that satisfies - core.protocol.Connection. - """ - - name = "ibis" - - def __init__(self, dialect: str, **config: t.Any): - if dialect not in DIALECTS: - raise KeyError( - f"Unknown ibis dialect {dialect!r}. " - f"Available: {sorted(DIALECTS)}" - ) - self.dialect = dialect - self._spec: DialectSpec = DIALECTS[dialect] - self._config = config - - def connect(self) -> Connection: - if self._spec.connection_builder is None: - raise NotImplementedError( - f"Dialect {self.dialect!r} has no connection_builder configured" - ) - ibis_conn = self._spec.connection_builder(**self._config) - return BaseIbisConnection(ibis_conn, dialect_spec=self._spec) -``` - -Note: this introduces new constructor args on `BaseIbisConnection`. Read the existing `BaseIbisConnection` `__init__` in `backends/ibis/connection.py` and reconcile. Likely you'll need to add a constructor that takes a pre-built ibis connection plus a `DialectSpec`, while keeping the legacy constructor working through the shim path. - -The simplest reconciliation: add a class method or alternate constructor: - -```python -@classmethod -def from_dialect(cls, ibis_conn, dialect_spec: DialectSpec) -> "BaseIbisConnection": - self = cls.__new__(cls) - self._ibis_conn = ibis_conn - self._dialect_spec = dialect_spec - # ... initialize whatever else BaseIbisConnection needs - return self -``` - -And have `IbisBackend.connect()` use it: - -```python -return BaseIbisConnection.from_dialect(ibis_conn, self._spec) -``` - -Read the existing BaseIbisConnection to make this concrete; the goal is "construct a connection from a pre-built ibis backend and a DialectSpec without going through the legacy settings-class path." - -Also: `BaseIbisConnection` must satisfy the `Connection` Protocol — `list_namespaces`, `list_tables`, `inspect_table`, `inspect_namespace`, `inspect_catalog`, `close`. Add any missing methods (delegating to the wrapped ibis connection where possible, and to `inspect.table_to_info` for the inspect_* methods). - -- [ ] **Step 4: Run the test** - -Run: `hatch run test:test-quick tests/test_unit/backends/ibis/test_backend.py -v` -Expected: PASS. - -The end-to-end sqlite test (`test_in_memory_sqlite_connect_and_inspect`) is the most likely to fail. Common failure modes: -- `_build_sqlite_connection` doesn't accept `database=":memory:"` — adjust the function to pass through kwargs to `ibis.sqlite.connect`. -- `BaseIbisConnection.list_tables()` doesn't exist or has a different signature — add it as a wrapper around `self._ibis_conn.list_tables()`. -- `BaseIbisConnection.close()` doesn't exist — add it. - -Each failure points at one missing piece. Add only what's needed to pass. - -- [ ] **Step 5: Run the full test suite** - -Run: `hatch run test:test` -Expected: PASS, baseline count. - -- [ ] **Step 6: Commit** - -```bash -git add -A -git commit -m "feat(data/ibis): add IbisBackend Protocol implementation - -IbisBackend(dialect=...) constructs a connection by looking up the -DialectSpec in the registry. BaseIbisConnection extended with a -from_dialect classmethod and the methods required by core.protocol.Connection." -``` - -### Task 4.9: Implement `to_relation()` on the ibis connection - -Per the spec, `Backend.connect().to_relation(table_name)` is the seam to `mountainash-expressions`. This task adds it for ibis. - -- [ ] **Step 1: Confirm the mountainash-expressions Relation API** - -Run: `grep -rn "class Relation\|from_ibis\|^def from_ibis" /home/nathanielramm/git/mountainash-io/mountainash/mountainash-expressions/src/mountainash/ --include='*.py' | head -20` - -Identify how `mountainash-expressions` constructs a Relation from an ibis Table. Likely there's something like `Relation.from_ibis(table)` or an `ibis_to_relation(table)` function. - -If the API doesn't exist, STOP — `to_relation()` for ibis cannot be implemented without it. Document the gap in the same way iceberg's gap is documented (in `tests/test_unit/backends/ibis/COVERAGE_GAP.md`) and skip steps 2–5. - -- [ ] **Step 2: Write the failing test** - -Append to `tests/test_unit/backends/ibis/test_backend.py`: - -```python -def test_to_relation_returns_expressions_relation(): - from mountainash.expressions import Relation # adjust import to actual API - - backend = IbisBackend(dialect="sqlite", database=":memory:") - conn = backend.connect() - try: - conn._ibis_conn.create_table( - "users", - schema=__import__("ibis").schema({"id": "int64"}), - ) - rel = conn.to_relation("users") - assert isinstance(rel, Relation) - finally: - conn.close() -``` - -Adjust the import statement to match the actual `mountainash-expressions` public API confirmed in Step 1. - -- [ ] **Step 3: Run the test** - -Run: `hatch run test:test-quick tests/test_unit/backends/ibis/test_backend.py::test_to_relation_returns_expressions_relation -v` -Expected: FAIL. - -- [ ] **Step 4: Implement `to_relation` on `BaseIbisConnection`** - -Edit `src/mountainash_data/backends/ibis/connection.py`. Add: - -```python -def to_relation(self, table_name: str, namespace: str | None = None): - """Hand off to mountainash-expressions: return a Relation backed by - this connection's ibis table. - """ - from mountainash.expressions import Relation # adjust to actual API - ibis_table = self._ibis_conn.table(table_name) - return Relation.from_ibis(ibis_table) # adjust to actual API -``` - -- [ ] **Step 5: Run the test** - -Run: `hatch run test:test-quick tests/test_unit/backends/ibis/test_backend.py::test_to_relation_returns_expressions_relation -v` -Expected: PASS. - -- [ ] **Step 6: Commit** - -```bash -git add -A -git commit -m "feat(data/ibis): add to_relation() seam to mountainash-expressions" -``` - -### Task 4.10: Phase 4 sanity check - -- [ ] **Step 1: Run full test suite** - -Run: `hatch run test:test` -Expected: PASS, baseline count. - -- [ ] **Step 2: Run lint and type-check** - -Run: `hatch run ruff:check src/mountainash_data/` -Expected: PASS. - -Run: `hatch run mypy:check` -Expected: PASS. - -Fix any issues introduced by Phase 4 before continuing. - ---- - -## Phase 5 — Wire factories to the registry (D2.a) - -**Files touched:** -- Create: `src/mountainash_data/core/factories/__init__.py` and 5 file copies -- Create: `src/mountainash_data/core/utils.py` (from `database_utils.py`) -- Modify: `src/mountainash_data/factories/*.py` (each becomes a shim) -- Modify: `src/mountainash_data/database_utils.py` (becomes a shim) - -### Task 5.1: Move factories verbatim with shims - -- [ ] **Step 1: Copy factories to `core/factories/`** - -```bash -mkdir -p src/mountainash_data/core/factories -cp src/mountainash_data/factories/*.py src/mountainash_data/core/factories/ -``` - -- [ ] **Step 2: Update internal imports inside `core/factories/`** - -Run: `grep -rn "from mountainash_data" src/mountainash_data/core/factories/` - -Rewrite: -- `mountainash_data.databases.connections.base_db_connection` → `mountainash_data.core.connection` -- `mountainash_data.databases.constants` → `mountainash_data.core.constants` -- `mountainash_data.databases.settings` → `mountainash_data.core.settings` -- `mountainash_data.factories.` → `mountainash_data.core.factories.` (cross-factory imports) - -Leave imports of per-backend connection classes (e.g., `databases.connections.ibis.postgres_ibis_connection`) alone for now — they still resolve via the shims and Task 5.2 updates them. - -- [ ] **Step 3: Replace each `factories/*.py` file with a shim** - -For each of `__init__.py`, `base_strategy_factory.py`, `settings_type_factory_mixin.py`, `connection_factory.py`, `operations_factory.py`, `settings_factory.py`: - -```python -"""DEPRECATED: import from mountainash_data.core.factories. instead.""" - -from mountainash_data.core.factories. import * # noqa: F401,F403 -``` - -Add explicit re-exports for the public symbols (read each `core/factories/.py` to enumerate). - -- [ ] **Step 4: Run tests** - -Run: `hatch run test:test` -Expected: PASS, baseline count. - -- [ ] **Step 5: Commit** - -```bash -git add -A -git commit -m "refactor(data): move factories to core/factories/ with shims" -``` - -### Task 5.2: Update factory strategy mappings to point at IbisBackend / IcebergBackend - -- [ ] **Step 1: Read `core/factories/connection_factory.py` and identify the strategy mapping** - -The audit said this file lazily loads per-backend connection classes (13 ibis + pyiceberg). Find the mapping (likely a dict mapping backend names to dotted import paths). - -- [ ] **Step 2: Update the mapping** - -Replace each per-backend class import path with a factory function that returns an `IbisBackend(dialect=...)` instance. For example, if the old mapping was: - -```python -_STRATEGIES = { - "postgres": "mountainash_data.databases.connections.ibis.postgres_ibis_connection.Postgres_IbisConnection", - "sqlite": "mountainash_data.databases.connections.ibis.sqlite_ibis_connection.SQLite_IbisConnection", - ... -} -``` - -Replace with a function-based mapping or add a wrapper: - -```python -def _make_ibis_factory(dialect: str): - def factory(**config): - from mountainash_data.backends.ibis.backend import IbisBackend - return IbisBackend(dialect=dialect, **config) - return factory - - -def _make_iceberg_factory(catalog: str): - def factory(**config): - from mountainash_data.backends.iceberg.backend import IcebergBackend - return IcebergBackend(catalog=catalog, **config) - return factory - - -_STRATEGIES = { - "postgres": _make_ibis_factory("postgres"), - "sqlite": _make_ibis_factory("sqlite"), - "duckdb": _make_ibis_factory("duckdb"), - "motherduck": _make_ibis_factory("motherduck"), - "mysql": _make_ibis_factory("mysql"), - "mssql": _make_ibis_factory("mssql"), - "oracle": _make_ibis_factory("oracle"), - "snowflake": _make_ibis_factory("snowflake"), - "bigquery": _make_ibis_factory("bigquery"), - "redshift": _make_ibis_factory("redshift"), - "trino": _make_ibis_factory("trino"), - "pyspark": _make_ibis_factory("pyspark"), - "pyiceberg_rest": _make_iceberg_factory("rest"), -} -``` - -If `BaseStrategyFactory` uses lazy loading via importlib (per the audit), the function-based mapping above is incompatible — adjust to whatever shape the lazy loader expects. Read `base_strategy_factory.py` to confirm. The simplest backward-compatible approach: keep the dotted-string lazy-loading path, and create a thin wrapper module that exposes the new backends under the old class names. But function-based is cleaner if the framework allows it. - -- [ ] **Step 3: Apply the same treatment to `operations_factory.py`** - -Read it. The strategy mapping previously pointed at per-backend ops classes. Now there's only `BaseIbisOperations` and the iceberg operations module. Update the mapping to return `BaseIbisOperations` for all ibis backends (since per-backend operations classes no longer exist) and the iceberg operations module/class for pyiceberg. - -- [ ] **Step 4: `settings_factory.py` — leave the URL detection logic alone** - -Read `core/factories/settings_factory.py`. Per the audit, the SCHEME_MAP and URL detection are the "real feature" worth keeping. Do not change them. Confirm internal imports are updated per Task 5.1 step 2. - -- [ ] **Step 5: Run the factory tests** - -Run: `hatch run test:test-quick tests/test_unit/factories/ -v` -Expected: PASS. - -If `test_connection_factory.py` fails with assertions like "expected class X, got IbisBackend", update the test expectations — the new behavior (factory returns `IbisBackend(dialect="postgres")`) is correct per the spec. Document the test expectation change in the commit message. - -If `test_operations_factory.py` fails similarly, same treatment. - -If a test fails because it tried to instantiate the returned class with legacy positional args, the test was depending on the old shape. Update it to use the new construction style: `factory.get_connection("postgres", **config)` returning an `IbisBackend`. - -- [ ] **Step 6: Run the full test suite** - -Run: `hatch run test:test` -Expected: PASS. - -- [ ] **Step 7: Commit** - -```bash -git add -A -git commit -m "refactor(data/factories): point connection/operations factories at new backends - -ConnectionFactory now returns IbisBackend(dialect=...) or -IcebergBackend(catalog=...) instead of per-backend subclasses. -OperationsFactory returns BaseIbisOperations for all ibis dialects -(per-backend ops classes are gone). settings_factory unchanged." -``` - -### Task 5.3: Move `database_utils.py` → `core/utils.py` - -- [ ] **Step 1: Copy and update imports** - -```bash -cp src/mountainash_data/database_utils.py src/mountainash_data/core/utils.py -``` - -Read `src/mountainash_data/core/utils.py`. Rewrite imports: -- `mountainash_data.factories.*` → `mountainash_data.core.factories.*` -- `mountainash_data.databases.constants` → `mountainash_data.core.constants` -- `mountainash_data.databases.settings` → `mountainash_data.core.settings` - -- [ ] **Step 2: Update the facade to consume new factories** - -The audit identified six methods on `DatabaseUtils`: `create_connection`, `create_operations`, `create_backend`, `create_settings_from_url`, `detect_backend_from_url`, `create_from_url`. - -Read each method. They likely already call into `ConnectionFactory` / `OperationsFactory` / `SettingsFactory`. The factory APIs haven't changed (still `get_connection(backend_type, **config)`), so the methods should keep working without changes. Only the import paths needed updating in Step 1. - -If any method directly imports a per-backend connection class (e.g., `from mountainash_data.databases.connections.ibis.postgres_ibis_connection import Postgres_IbisConnection`), replace that with `from mountainash_data.backends.ibis.backend import IbisBackend` and use the dialect arg. - -- [ ] **Step 3: Replace the original `database_utils.py` with a shim** - -Overwrite `src/mountainash_data/database_utils.py`: - -```python -"""DEPRECATED: import from mountainash_data.core.utils instead.""" - -from mountainash_data.core.utils import * # noqa: F401,F403 -from mountainash_data.core.utils import DatabaseUtils # noqa: F401 -``` - -Add explicit re-exports for any other public symbols. - -- [ ] **Step 4: Run tests** - -Run: `hatch run test:test` -Expected: PASS, baseline count. - -- [ ] **Step 5: Commit** - -```bash -git add -A -git commit -m "refactor(data): move database_utils to core/utils.py with shim" -``` - ---- - -## Phase 6 — Remove shims and finalize - -### Task 6.1: Confirm no production code references shimmed paths - -- [ ] **Step 1: Grep for legacy paths** - -Run: `grep -rn "mountainash_data\.databases\." src/ tests/ notebooks/` - -Expected: matches inside `src/mountainash_data/databases/` itself (the shims) and inside the test files that still test the legacy paths. - -For every match in `notebooks/` or in non-shim `src/` files, update to the new path before proceeding. Tests that test the shim layer specifically (if any) can be deleted — the shims are about to be removed. - -- [ ] **Step 2: Update top-level `__init__.py`** - -Read `src/mountainash_data/__init__.py`. Replace its current re-exports with the new public surface: - -```python -"""mountainash-data: physical access to backend data services. - -Public API: - Backend, Connection — protocols (core.protocol) - IbisBackend — ibis-style relational backends (backends.ibis.backend) - IcebergBackend — iceberg-style table-format catalogs (backends.iceberg.backend) - CatalogInfo, NamespaceInfo, TableInfo, ColumnInfo — inspection model - DatabaseUtils — high-level facade - ConnectionFactory, OperationsFactory, SettingsFactory — factories - *Settings classes — see mountainash_data.core.settings -""" - -from mountainash_data.__version__ import __version__ -from mountainash_data.core.protocol import Backend, Connection -from mountainash_data.core.inspection import ( - CatalogInfo, - ColumnInfo, - NamespaceInfo, - TableInfo, -) -from mountainash_data.core.utils import DatabaseUtils -from mountainash_data.core.factories import ( - ConnectionFactory, - OperationsFactory, - SettingsFactory, -) -from mountainash_data.backends.ibis.backend import IbisBackend -from mountainash_data.backends.iceberg.backend import IcebergBackend - -__all__ = [ - "__version__", - "Backend", - "Connection", - "CatalogInfo", - "ColumnInfo", - "NamespaceInfo", - "TableInfo", - "DatabaseUtils", - "ConnectionFactory", - "OperationsFactory", - "SettingsFactory", - "IbisBackend", - "IcebergBackend", -] -``` - -- [ ] **Step 3: Run tests** - -Run: `hatch run test:test` -Expected: PASS. - -- [ ] **Step 4: Commit** - -```bash -git add -A -git commit -m "refactor(data): update top-level __init__.py to new public surface" -``` - -### Task 6.2: Delete all shims and the old `databases/` and `factories/` directories - -- [ ] **Step 1: Delete the shimmed directories** - -```bash -rm -rf src/mountainash_data/databases -rm -rf src/mountainash_data/factories -rm src/mountainash_data/database_utils.py -``` - -- [ ] **Step 2: Run tests** - -Run: `hatch run test:test` -Expected: PASS — if any test still imports from the deleted paths, it should be updated or removed. - -If a test fails with `ModuleNotFoundError: mountainash_data.databases...`, that test is testing the shim layer. Two options: -1. Update the test to import from the new location and assert the same behavior. -2. Delete the test if its only purpose was to verify the shim. - -Pick option 1 unless the test is purely about the legacy structure and has no equivalent value in the new structure. - -- [ ] **Step 3: Run lint and type-check** - -Run: `hatch run ruff:check` -Expected: PASS. - -Run: `hatch run mypy:check` -Expected: PASS. - -- [ ] **Step 4: Commit** - -```bash -git add -A -git commit -m "refactor(data): remove all Phase 1–5 shims and legacy directories - -databases/, factories/, and database_utils.py are gone. The new public -surface lives in core/ and backends/. Refactor complete." -``` - -### Task 6.3: Update notebooks, README, and CLAUDE.md - -- [ ] **Step 1: Update notebook imports** - -Run: `grep -rn "mountainash_data" notebooks/ --include='*.ipynb' --include='*.py'` - -For each notebook that imports from the package, update the import paths to the new public surface (`from mountainash_data import IbisBackend` etc.). - -If notebooks aren't actively used, this can be deferred — note any deferral in the commit message. - -- [ ] **Step 2: Update README.md** - -Read `README.md`. Update the architecture overview and usage examples to reflect the new shape. Replace any code samples that show `from mountainash_data.databases.connections.ibis.postgres_ibis_connection import Postgres_IbisConnection` with `from mountainash_data import IbisBackend`. - -- [ ] **Step 3: Update CLAUDE.md** - -Read `CLAUDE.md`. Update the Architecture, Package Structure, and Usage Patterns sections to match the new layout. Replace the old usage patterns block with: - -```python -from mountainash_data import IbisBackend, IcebergBackend -from mountainash_data.core.settings.postgresql import PostgreSQLSettings - -# Ibis backend -backend = IbisBackend(dialect="postgres", host="localhost", database="mydb", user="me") -conn = backend.connect() -try: - tables = conn.list_tables() - info = conn.inspect_table("users") - relation = conn.to_relation("users") # → mountainash-expressions Relation -finally: - conn.close() - -# Iceberg backend -ice = IcebergBackend(catalog="rest", uri="http://localhost:8181") -ice_conn = ice.connect() -try: - namespaces = ice_conn.list_namespaces() -finally: - ice_conn.close() -``` - -- [ ] **Step 4: Run tests one final time** - -Run: `hatch run test:test` -Expected: PASS, final count recorded for comparison with baseline. - -Run: `hatch run ruff:check && hatch run mypy:check` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add -A -git commit -m "docs(data): update README and CLAUDE.md to reflect new architecture" -``` - -### Task 6.4: Create the PR - -- [ ] **Step 1: Push the branch and open a PR** - -```bash -git push -u origin refactor/data-package-audit -``` - -```bash -gh pr create --title "Refactor: collapse data package onto core+backends architecture" --body "$(cat <<'EOF' -## Summary - -- Collapse 13 ibis per-backend connection classes into a data-driven `DialectSpec` registry under `backends/ibis/dialects/` -- Finish the half-done split between iceberg connection and operations files; deduplicate methods -- Add `core/protocol.py` (Backend, Connection protocols) and `core/inspection.py` (shared physical metadata model) -- Move settings, factories, and the high-level facade (`database_utils` → `core/utils.py`) into `core/` -- Add `to_relation()` seam between physical (`mountainash-data`) and logical (`mountainash-expressions`) layers -- Delete 10 dead/duplicate files (legacy db_connection_factory, 8 stub ibis ops files, lineage stub, `__init___old.py`) -- ~9.5k LOC reorganized; behavior preserved (existing test suite green throughout) - -See `docs/superpowers/specs/2026-04-07-mountainash-data-audit-and-redesign.md` for the full design. - -## Test plan - -- [x] `hatch run test:test` green at every phase boundary -- [x] `hatch run ruff:check` clean -- [x] `hatch run mypy:check` clean -- [ ] Manual smoke: open a sqlite IbisBackend, list tables, inspect, to_relation -- [ ] Manual smoke: open a pyiceberg-rest IcebergBackend, list namespaces, inspect - -## Known gaps - -- `IcebergBackend.to_relation()` is unimplemented — requires `mountainash-expressions` to add an iceberg relation adapter; tracked separately -- No iceberg-specific tests added (the legacy code had none either; documented in `tests/test_unit/backends/iceberg/COVERAGE_GAP.md`) - -🤖 Generated with [Claude Code](https://claude.com/claude-code) -EOF -)" -``` - ---- - -## Self-review - -After writing the plan, the following spec sections were checked for coverage: - -- ✅ Defects 1–4 (class explosion, mixins, no expressions seam, stateful connections) → addressed by Phases 4 (registry collapse), 4 (mixin → functions), 4.9 (`to_relation`), 1 (Backend Protocol stateless model) -- ✅ Target architecture sections (`core/`, `backends/`, `DialectSpec`, capability hooks, inspection model, expressions seam) → Phases 1, 3, 4 -- ✅ Audit findings 1–8 → handled across Phases 0 (deletes), 3 (iceberg dedup), 4 (ibis salvage) -- ✅ D1.b (iceberg split + dedup) → Task 3.3 -- ✅ D2.a (factories survive, registry is data) → Task 5.2 -- ✅ D3.b (HYBRID grep at migration time) → Task 4.1 step 3 -- ✅ Migration phases 0–6 → Phases 0–6 -- ✅ Test strategy (per-phase tests, coverage gap audit) → Tasks 3.1, 4.1, plus per-task test runs -- ✅ Known gaps (`to_relation` for iceberg) → documented in Task 3.1 and PR body diff --git a/docs/superpowers/plans/2026-04-15-settings-audit.md b/docs/superpowers/plans/2026-04-15-settings-audit.md deleted file mode 100644 index 417f649..0000000 --- a/docs/superpowers/plans/2026-04-15-settings-audit.md +++ /dev/null @@ -1,434 +0,0 @@ -# Settings Audit Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Produce a per-backend audit report for each of 11 settings classes in `src/mountainash_data/core/settings/`, comparing each class against its authoritative driver spec, the corresponding Ibis backend `do_connect()` passthrough, and any vendor reference docs. Report-only; fixes happen in later per-backend cycles. - -**Architecture:** 12 independent audit tasks (one index + 11 backends). Each backend task is self-contained: read settings class → fetch source specs (WebFetch) → read Ibis `do_connect()` signature → produce a markdown report at `docs/superpowers/specs/2026-04-15-settings-audit/.md` following the schema defined in the spec README. A final task updates the index table with summary counts and commits. - -**Tech Stack:** Python 3.12, pydantic settings, Ibis 10.4.0, PyIceberg (for `pyiceberg_rest`). Tooling: Read/Grep for source, WebFetch for spec URLs, Write for reports, git for commits. - ---- - -## Shared context for all audit tasks - -**Spec README (authoritative for report format):** `docs/superpowers/specs/2026-04-15-settings-audit/README.md` - -**Our settings classes:** `src/mountainash_data/core/settings/.py` - -**Ibis backends (ground truth for passthrough):** `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends//__init__.py` — inspect the `do_connect()` method signature. - -**Source precedence (when specs disagree):** -1. Driver/client spec — parameter names, types, defaults, validation semantics -2. Ibis backend `do_connect()` — what can actually pass through -3. Vendor docs — context only - -**Parameter tiering:** -- `core` — auth, host/endpoint, TLS, timeouts, database/schema/catalog selection -- `advanced` — tuning knobs, deprecated options, esoterica - -**Report schema (every per-backend report must contain):** -1. Header: spec URLs with precedence labels, date checked, settings class path, Ibis backend path -2. Stale-link check: WebFetch result per URL — `OK` / `redirect → ` / `404` / `blocked` -3. Summary counts: core missing, core mismatch, advanced missing, advanced mismatch, extra, total audited -4. Parameter table with columns: `Spec name | Our field | Status | Tier | Type ✓ | Default ✓ | Ibis passthrough | Notes` - - Status: `present` / `missing` / `mismatch` / `extra` - - Type/Default: ✓ / ✗ / N/A -5. Findings narrative: core gaps → core mismatches → advanced gaps → advanced mismatches → stale links -6. Recommended follow-ups: concrete per-backend bullets for downstream plans - -**Commit convention:** One commit per backend report, message `docs(audit): settings audit`. Final index-update commit: `docs(audit): fill audit index summary counts`. - ---- - -## Task 1: SQLite settings audit - -**Files:** -- Read: `src/mountainash_data/core/settings/sqlite.py` -- Read: `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends/sqlite/__init__.py` -- Create: `docs/superpowers/specs/2026-04-15-settings-audit/sqlite.md` - -**Source URLs (from docstring, precedence-tagged):** -- Vendor (context): https://www.sqlite.org/pragma.html -- Driver (authoritative): stdlib `sqlite3` — https://docs.python.org/3/library/sqlite3.html#sqlite3.connect -- Ibis passthrough (authoritative for what we can pass): local file above - -- [ ] **Step 1: Read our settings class** - -Run: Read `src/mountainash_data/core/settings/sqlite.py`. Note every `Field(...)` declaration and what `get_connection_kwargs()` / `get_connection_string_params()` actually return. - -- [ ] **Step 2: Fetch spec URLs and record link health** - -Use WebFetch for each URL above. For each: record `OK` / `redirect → ` / `404` / `blocked`. Capture the parameter list (SQLite PRAGMAs for the vendor URL; `connect()` kwargs for the Python stdlib URL). - -- [ ] **Step 3: Read Ibis `do_connect()` signature** - -Run: Read `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends/sqlite/__init__.py`. Locate `def do_connect(self, ...)`. List every keyword argument — this is the passthrough surface. - -- [ ] **Step 4: Build parameter union and classify** - -Build a set `U = driver_params ∪ our_fields ∪ ibis_passthrough_kwargs`. For each parameter in `U`: -- Status: `missing` if in driver but not ours; `extra` if in ours but not driver; `present` if in both with same name/semantics; `mismatch` if name/type/default diverges -- Tier: `core` if it's `database`/path/timeout/isolation; else `advanced` -- Type ✓ / Default ✓: compare our Field type/default to driver signature -- Ibis passthrough: ✓ if in `do_connect()` kwargs, ✗ otherwise - -- [ ] **Step 5: Write the report** - -Write to `docs/superpowers/specs/2026-04-15-settings-audit/sqlite.md` using the 6-section schema in the shared context above. Fill every section — no `TBD`. - -- [ ] **Step 6: Commit** - -```bash -git add docs/superpowers/specs/2026-04-15-settings-audit/sqlite.md -git commit -m "docs(audit): sqlite settings audit" -``` - ---- - -## Task 2: DuckDB settings audit - -**Files:** -- Read: `src/mountainash_data/core/settings/duckdb.py` -- Read: `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends/duckdb/__init__.py` -- Create: `docs/superpowers/specs/2026-04-15-settings-audit/duckdb.md` - -**Source URLs:** -- Ibis backend (context): https://ibis-project.org/backends/duckdb -- Driver (authoritative): https://duckdb.org/docs/configuration/overview.html -- Vendor (context): https://duckdb.org/docs/extensions/spatial.html -- Ibis passthrough (authoritative): local file above - -Follow the same 6 steps as Task 1 with these paths/URLs. Write report to `.../duckdb.md`. Commit message: `docs(audit): duckdb settings audit`. - -- [ ] **Step 1: Read `src/mountainash_data/core/settings/duckdb.py`** -- [ ] **Step 2: WebFetch each source URL, record link health and parameter list** -- [ ] **Step 3: Read Ibis duckdb `do_connect()` signature** -- [ ] **Step 4: Build union, classify each parameter (status, tier, type, default, passthrough)** -- [ ] **Step 5: Write `docs/superpowers/specs/2026-04-15-settings-audit/duckdb.md` with all 6 report sections** -- [ ] **Step 6: Commit** - -```bash -git add docs/superpowers/specs/2026-04-15-settings-audit/duckdb.md -git commit -m "docs(audit): duckdb settings audit" -``` - ---- - -## Task 3: MotherDuck settings audit - -**Files:** -- Read: `src/mountainash_data/core/settings/motherduck.py` -- Read: `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends/duckdb/__init__.py` (motherduck rides on Ibis duckdb) -- Create: `docs/superpowers/specs/2026-04-15-settings-audit/motherduck.md` - -**Source URLs (none in docstring — use canonical):** -- Driver (authoritative): https://motherduck.com/docs/getting-started/connect-query-from-python/installation-authentication/ -- DuckDB extension (context): https://motherduck.com/docs/ -- Ibis passthrough: via Ibis duckdb (record passthrough as `via duckdb`) - -Header must note: "MotherDuck has no dedicated Ibis backend; passthrough tracked via Ibis duckdb backend. Source URLs were not present in the settings class docstring and were added by this audit — recommend backfilling into `motherduck.py` docstring in a follow-up." - -- [ ] **Step 1: Read `src/mountainash_data/core/settings/motherduck.py`** -- [ ] **Step 2: WebFetch each source URL, record link health and parameter list (TOKEN, `md:` connection-string quirks, `ATTACH` semantics)** -- [ ] **Step 3: Read Ibis duckdb `do_connect()` signature — note which kwargs apply to motherduck connection strings (`md:?motherduck_token=...`)** -- [ ] **Step 4: Build union, classify each parameter. "Ibis passthrough" column value: `via duckdb ✓` or `via duckdb ✗`** -- [ ] **Step 5: Write report to `docs/superpowers/specs/2026-04-15-settings-audit/motherduck.md`** -- [ ] **Step 6: Commit** - -```bash -git add docs/superpowers/specs/2026-04-15-settings-audit/motherduck.md -git commit -m "docs(audit): motherduck settings audit" -``` - ---- - -## Task 4: PostgreSQL settings audit - -**Files:** -- Read: `src/mountainash_data/core/settings/postgresql.py` -- Read: `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends/postgres/__init__.py` -- Create: `docs/superpowers/specs/2026-04-15-settings-audit/postgresql.md` - -**Source URLs:** -- Driver (authoritative): https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS -- Driver (authoritative, supplementary): https://www.postgresql.org/docs/current/libpq-connect.html -- Ibis passthrough (authoritative): local file above - -This class has many enums (`PostgresTargetSessionAttrs`, `PostgresRequireAuthMethods`, `PostgresSSLCertNegotiation`, `PostgresSSLCertMode`) — enumerate each Enum and check that values match libpq's accepted values. - -- [ ] **Step 1: Read `src/mountainash_data/core/settings/postgresql.py` in full; list every Field and every Enum value** -- [ ] **Step 2: WebFetch libpq docs; extract the full PQconnectdbParams keyword list** -- [ ] **Step 3: Read Ibis postgres `do_connect()` signature** -- [ ] **Step 4: Build union, classify. Note: for each Enum, cross-check allowed values against libpq and flag divergences in Notes column** -- [ ] **Step 5: Write report to `docs/superpowers/specs/2026-04-15-settings-audit/postgresql.md`. Include a dedicated "Enum value coverage" subsection under Findings** -- [ ] **Step 6: Commit** - -```bash -git add docs/superpowers/specs/2026-04-15-settings-audit/postgresql.md -git commit -m "docs(audit): postgresql settings audit" -``` - ---- - -## Task 5: MySQL settings audit - -**Files:** -- Read: `src/mountainash_data/core/settings/mysql.py` -- Read: `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends/mysql/__init__.py` -- Create: `docs/superpowers/specs/2026-04-15-settings-audit/mysql.md` - -**Source URLs:** -- Driver (authoritative): https://mysqlclient.readthedocs.io/user_guide.html#functions-and-attributes -- Driver supplementary: https://dev.mysql.com/doc/c-api/8.4/en/mysql-ssl-set.html -- Driver supplementary: https://dev.mysql.com/doc/c-api/8.4/en/mysql-options.html -- Ibis passthrough: local file above - -- [ ] **Step 1: Read `src/mountainash_data/core/settings/mysql.py`** -- [ ] **Step 2: WebFetch all three URLs; merge parameter lists (mysqlclient `connect()` kwargs + `mysql_ssl_set` parameters + `mysql_options` flags)** -- [ ] **Step 3: Read Ibis mysql `do_connect()` signature** -- [ ] **Step 4: Build union, classify. Call out deprecated SSL params in Notes** -- [ ] **Step 5: Write report to `docs/superpowers/specs/2026-04-15-settings-audit/mysql.md`** -- [ ] **Step 6: Commit** - -```bash -git add docs/superpowers/specs/2026-04-15-settings-audit/mysql.md -git commit -m "docs(audit): mysql settings audit" -``` - ---- - -## Task 6: MSSQL settings audit - -**Files:** -- Read: `src/mountainash_data/core/settings/mssql.py` -- Read: `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends/mssql/__init__.py` -- Create: `docs/superpowers/specs/2026-04-15-settings-audit/mssql.md` - -**Source URLs:** -- Driver (authoritative): https://learn.microsoft.com/en-us/sql/connect/odbc/connection-string-keywords-and-data-source-names-dsns -- Driver install (context): https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server -- pyodbc (authoritative, passthrough semantics): https://github.com/mkleehammer/pyodbc/wiki/The-pyodbc-Module#connect -- Ibis passthrough: local file above - -This class has 4 enums (`MSSQLAuthMethod`, `MSSQLAuthEncryption`, `MSSQLAuthProtocol`, `MSSQLDriverType`) — audit enum values against ODBC docs. - -- [ ] **Step 1: Read `src/mountainash_data/core/settings/mssql.py`; list Fields and all Enum values** -- [ ] **Step 2: WebFetch all three URLs; extract ODBC connection-string keywords** -- [ ] **Step 3: Read Ibis mssql `do_connect()` signature** -- [ ] **Step 4: Build union, classify. Cross-check each Enum against ODBC accepted values** -- [ ] **Step 5: Write report to `docs/superpowers/specs/2026-04-15-settings-audit/mssql.md` including "Enum value coverage" subsection** -- [ ] **Step 6: Commit** - -```bash -git add docs/superpowers/specs/2026-04-15-settings-audit/mssql.md -git commit -m "docs(audit): mssql settings audit" -``` - ---- - -## Task 7: Snowflake settings audit - -**Files:** -- Read: `src/mountainash_data/core/settings/snowflake.py` -- Read: `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends/snowflake/__init__.py` -- Create: `docs/superpowers/specs/2026-04-15-settings-audit/snowflake.md` - -**Source URLs:** -- Driver (authoritative): https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-api#label-snowflake-connector-methods-connect -- Driver connect guide: https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-connect -- OAuth (context): https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-example#connecting-with-oauth -- Ibis passthrough: local file above - -Class has `CONST_SNOWFLAKE_AUTHENTICATOR` enum — audit enum values against Snowflake's documented authenticator options. - -- [ ] **Step 1: Read `src/mountainash_data/core/settings/snowflake.py`; list Fields and `CONST_SNOWFLAKE_AUTHENTICATOR` values** -- [ ] **Step 2: WebFetch all three URLs; extract `snowflake.connector.connect()` parameter list + authenticator values** -- [ ] **Step 3: Read Ibis snowflake `do_connect()` signature** -- [ ] **Step 4: Build union, classify. Cross-check authenticator enum values** -- [ ] **Step 5: Write report to `docs/superpowers/specs/2026-04-15-settings-audit/snowflake.md`** -- [ ] **Step 6: Commit** - -```bash -git add docs/superpowers/specs/2026-04-15-settings-audit/snowflake.md -git commit -m "docs(audit): snowflake settings audit" -``` - ---- - -## Task 8: BigQuery settings audit - -**Files:** -- Read: `src/mountainash_data/core/settings/bigquery.py` -- Read: `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends/bigquery/__init__.py` -- Create: `docs/superpowers/specs/2026-04-15-settings-audit/bigquery.md` - -**Source URLs:** -- Driver (authoritative): https://cloud.google.com/python/docs/reference/bigquery/latest/google.cloud.bigquery.client.Client -- Ibis backend (context): https://ibis-project.org/backends/bigquery -- Auth (context): https://cloud.google.com/sdk/docs/authorizing -- External data sources (context): https://cloud.google.com/bigquery/external-data-sources -- Ibis passthrough (authoritative): local file above — BigQuery is unusual in that most config comes through Ibis backend kwargs (`project_id`, `dataset_id`, `credentials`, `application_default_credentials`, etc.) rather than a driver connection string - -- [ ] **Step 1: Read `src/mountainash_data/core/settings/bigquery.py`** -- [ ] **Step 2: WebFetch each URL; extract `google.cloud.bigquery.Client` init parameters and Ibis-specific backend kwargs** -- [ ] **Step 3: Read Ibis bigquery `do_connect()` signature — treat this as the primary authoritative surface for BigQuery** -- [ ] **Step 4: Build union, classify** -- [ ] **Step 5: Write report to `docs/superpowers/specs/2026-04-15-settings-audit/bigquery.md`** -- [ ] **Step 6: Commit** - -```bash -git add docs/superpowers/specs/2026-04-15-settings-audit/bigquery.md -git commit -m "docs(audit): bigquery settings audit" -``` - ---- - -## Task 9: Redshift settings audit - -**Files:** -- Read: `src/mountainash_data/core/settings/redshift.py` -- Read: `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends/postgres/__init__.py` (Redshift rides on Ibis postgres) -- Create: `docs/superpowers/specs/2026-04-15-settings-audit/redshift.md` - -**Source URLs (none in docstring — use canonical):** -- Driver (authoritative): https://docs.aws.amazon.com/redshift/latest/mgmt/python-redshift-driver.html -- Driver connection params: https://github.com/aws/amazon-redshift-python-driver#basic-example -- libpq (supplementary, since most settings reuse postgres): https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS -- Ibis passthrough: via Ibis postgres - -Header must note: "Redshift has no dedicated Ibis backend; passthrough tracked via Ibis postgres. Source URLs absent from settings class docstring — recommend backfilling in a follow-up." - -- [ ] **Step 1: Read `src/mountainash_data/core/settings/redshift.py`** -- [ ] **Step 2: WebFetch all three URLs; extract `redshift_connector.connect()` parameters + libpq overlap** -- [ ] **Step 3: Read Ibis postgres `do_connect()` signature** -- [ ] **Step 4: Build union, classify. "Ibis passthrough" column values: `via postgres ✓` or `via postgres ✗`. Flag Redshift-only params (IAM auth, cluster identifier, region) that libpq/postgres backend doesn't support** -- [ ] **Step 5: Write report to `docs/superpowers/specs/2026-04-15-settings-audit/redshift.md`** -- [ ] **Step 6: Commit** - -```bash -git add docs/superpowers/specs/2026-04-15-settings-audit/redshift.md -git commit -m "docs(audit): redshift settings audit" -``` - ---- - -## Task 10: PySpark settings audit - -**Files:** -- Read: `src/mountainash_data/core/settings/pyspark.py` -- Read: `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends/pyspark/__init__.py` -- Create: `docs/superpowers/specs/2026-04-15-settings-audit/pyspark.md` - -**Source URLs:** -- Driver (authoritative, selective — only Spark properties we actually expose): https://spark.apache.org/docs/3.5.1/configuration.html#available-properties -- Databricks (context): https://docs.databricks.com/en/spark/conf.html -- Ibis passthrough (authoritative): local file above - -Per Q3/D, PySpark is the clearest "advanced" case — Spark has hundreds of config properties. Scope this audit to: -1. Parameters we actually declare as Fields -2. Ibis pyspark `do_connect()` kwargs -3. Any Spark property we reference in the settings class body (grep for `spark.` prefixes) - -Do NOT audit the full Spark configuration reference; note this scoping decision in the report header. - -- [ ] **Step 1: Read `src/mountainash_data/core/settings/pyspark.py`; grep for `spark.*` property strings inside the class** -- [ ] **Step 2: WebFetch both URLs; cross-reference only the Spark properties we declare or reference** -- [ ] **Step 3: Read Ibis pyspark `do_connect()` signature** -- [ ] **Step 4: Build restricted union (our Fields ∪ ibis passthrough ∪ referenced spark.* keys), classify** -- [ ] **Step 5: Write report to `docs/superpowers/specs/2026-04-15-settings-audit/pyspark.md`. Include explicit "Scoping" paragraph in header noting the exhaustive Spark property set is out of scope** -- [ ] **Step 6: Commit** - -```bash -git add docs/superpowers/specs/2026-04-15-settings-audit/pyspark.md -git commit -m "docs(audit): pyspark settings audit" -``` - ---- - -## Task 11: Trino settings audit - -**Files:** -- Read: `src/mountainash_data/core/settings/trino.py` -- Read: `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends/trino/__init__.py` -- Create: `docs/superpowers/specs/2026-04-15-settings-audit/trino.md` - -**Source URLs:** -- Driver (authoritative): https://github.com/trinodb/trino-python-client/blob/master/trino/dbapi.py -- Driver user guide (supplementary): https://github.com/trinodb/trino-python-client#connection-parameters -- Ibis passthrough: local file above - -- [ ] **Step 1: Read `src/mountainash_data/core/settings/trino.py`** -- [ ] **Step 2: WebFetch both URLs; extract `trino.dbapi.connect()` parameter list** -- [ ] **Step 3: Read Ibis trino `do_connect()` signature** -- [ ] **Step 4: Build union, classify** -- [ ] **Step 5: Write report to `docs/superpowers/specs/2026-04-15-settings-audit/trino.md`** -- [ ] **Step 6: Commit** - -```bash -git add docs/superpowers/specs/2026-04-15-settings-audit/trino.md -git commit -m "docs(audit): trino settings audit" -``` - ---- - -## Task 12: PyIceberg REST settings audit - -**Files:** -- Read: `src/mountainash_data/core/settings/pyiceberg_rest.py` -- Create: `docs/superpowers/specs/2026-04-15-settings-audit/pyiceberg_rest.md` - -**Source URLs:** -- Driver (authoritative): https://py.iceberg.apache.org/configuration/ -- REST catalog spec (authoritative, supplementary): https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml -- R2 (context, since code has R2-specific branch): https://developers.cloudflare.com/r2/data-catalog/config-examples/pyiceberg/ -- Ibis passthrough: **N/A** — PyIceberg is not an Ibis backend. Passthrough column should be `N/A` throughout; instead add a column or note for "PyIceberg REST catalog kwarg" matching against `pyiceberg.catalog.rest.RestCatalog` constructor. - -Header must note: "No Ibis backend applies to PyIceberg. The `Ibis passthrough` column is repurposed as `PyIceberg RestCatalog kwarg`." - -- [ ] **Step 1: Read `src/mountainash_data/core/settings/pyiceberg_rest.py` in full, including R2-specific branch in `_init_provider_specific`** -- [ ] **Step 2: WebFetch all three URLs; extract RestCatalog properties (`uri`, `token`, `credential`, `warehouse`, `s3.*`, `header.*`, signer options, etc.)** -- [ ] **Step 3: Skip Ibis read (not applicable); instead inspect PyIceberg source if installed (`python -c "from pyiceberg.catalog.rest import RestCatalog; import inspect; print(inspect.signature(RestCatalog.__init__))"`) or rely on docs** -- [ ] **Step 4: Build union, classify. Use the repurposed column** -- [ ] **Step 5: Write report to `docs/superpowers/specs/2026-04-15-settings-audit/pyiceberg_rest.md`** -- [ ] **Step 6: Commit** - -```bash -git add docs/superpowers/specs/2026-04-15-settings-audit/pyiceberg_rest.md -git commit -m "docs(audit): pyiceberg_rest settings audit" -``` - ---- - -## Task 13: Fill index summary counts - -**Files:** -- Modify: `docs/superpowers/specs/2026-04-15-settings-audit/README.md` (index table rows) - -- [ ] **Step 1: For each of the 11 backend reports, read the "Summary counts" section** - -Run: Read each `docs/superpowers/specs/2026-04-15-settings-audit/.md`. Extract the 6 count values (core missing, core mismatch, advanced missing, advanced mismatch, extra, stale links). - -- [ ] **Step 2: Update the index table** - -Edit `docs/superpowers/specs/2026-04-15-settings-audit/README.md`. Replace every `—` in the index table with the actual count from the corresponding report. Leave backend names and report links unchanged. - -- [ ] **Step 3: Verify the table** - -Run: Read the updated README. Confirm all 11 rows have numeric values in every count column and that the backend/report columns are unchanged. - -- [ ] **Step 4: Commit** - -```bash -git add docs/superpowers/specs/2026-04-15-settings-audit/README.md -git commit -m "docs(audit): fill audit index summary counts" -``` - ---- - -## Self-review notes (completed during plan authoring) - -- **Spec coverage:** Each of the 11 backends listed in the spec scope has a dedicated task (Tasks 1–12, minus pyiceberg numbering); final index task closes the loop on the spec's README index table. -- **Placeholders:** None. Every task has concrete file paths, URLs, and acceptance criteria. The one `TBD`-like element — index counts — is explicitly populated by Task 13. -- **Type/naming consistency:** Report schema (6 sections, 8-column table) is defined once in shared context and referenced identically from every task. Status/Tier vocabularies are identical across tasks. -- **Noted caveats encoded as task instructions:** motherduck/redshift missing source URLs (Tasks 3, 9 add canonical URLs and flag backfill); pyspark scoping (Task 10); pyiceberg_rest not-an-Ibis-backend (Task 12). diff --git a/docs/superpowers/plans/2026-04-15-settings-registry.md b/docs/superpowers/plans/2026-04-15-settings-registry.md deleted file mode 100644 index 8338a85..0000000 --- a/docs/superpowers/plans/2026-04-15-settings-registry.md +++ /dev/null @@ -1,4024 +0,0 @@ -# Settings Registry Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Replace the class-per-backend, method-heavy settings hierarchy in `src/mountainash_data/core/settings/` with a descriptor + thin-subclass pattern backed by a typed `AuthSpec` discriminated union — and carry forward the audit findings in the same pass. - -**Architecture:** `BackendDescriptor` (immutable data) + `ParameterSpec` list + `AuthSpec` union + optional per-backend `adapters/.py` for composite driver mapping. A generic `ConnectionProfile(MountainAshBaseSettings)` base reads `__descriptor__` to configure pydantic fields, with uniform `to_driver_kwargs()` / `to_connection_string()` API. Per-backend files shrink to a descriptor + two-line shell class. - -**Tech Stack:** Python 3.12, pydantic v2, `mountainash_settings.MountainAshBaseSettings`, pytest, hatch, ruff, mypy. - -**Spec:** `docs/superpowers/specs/2026-04-15-settings-registry-design.md`. - -**Scope:** Phases 1–3 of the spec (scaffolding, per-backend migrations, consumer update). Phase 4 (audit-fix sweep) becomes follow-up per-backend plans. - ---- - -## File Structure - -### Created - -``` -src/mountainash_data/core/settings/ -├── descriptor.py # ParameterSpec, BackendDescriptor, MISSING sentinel -├── registry.py # REGISTRY, @register, get_descriptor, get_settings_class -├── profile.py # ConnectionProfile base -├── auth/ -│ ├── __init__.py # re-exports -│ ├── base.py # AuthSpec ABC -│ ├── none.py # NoAuth -│ ├── password.py # PasswordAuth -│ ├── token.py # TokenAuth, JWTAuth -│ ├── oauth2.py # OAuth2Auth -│ ├── service_account.py # ServiceAccountAuth -│ ├── iam.py # IAMAuth -│ ├── azure.py # AzureADAuth, WindowsAuth -│ ├── kerberos.py # KerberosAuth -│ ├── certificate.py # CertificateAuth -│ └── dispatch.py # AUTH_TO_DRIVER_KWARGS default map -└── adapters/ - ├── __init__.py - ├── mysql.py - ├── mssql.py - ├── snowflake.py - ├── bigquery.py - ├── redshift.py - ├── trino.py - └── pyiceberg_rest.py - -tests/test_unit/core/settings/ -├── __init__.py -├── test_descriptor.py # ParameterSpec / BackendDescriptor unit tests -├── test_registry.py # register / lookup invariants -├── test_profile.py # ConnectionProfile base behavior -├── test_descriptors_invariants.py # parametric over REGISTRY -├── test_auth_dispatch.py # AUTH_TO_DRIVER_KWARGS coverage -└── backends/ - ├── __init__.py - ├── test_sqlite.py - ├── test_duckdb.py - ├── test_pyspark.py - ├── test_motherduck.py - ├── test_postgresql.py - ├── test_mysql.py - ├── test_trino.py - ├── test_mssql.py - ├── test_snowflake.py - ├── test_bigquery.py - ├── test_redshift.py - └── test_pyiceberg_rest.py -``` - -### Modified (rewritten in place) - -- `src/mountainash_data/core/settings/sqlite.py` — collapse to descriptor + shell -- `src/mountainash_data/core/settings/duckdb.py` -- `src/mountainash_data/core/settings/motherduck.py` -- `src/mountainash_data/core/settings/postgresql.py` -- `src/mountainash_data/core/settings/mysql.py` -- `src/mountainash_data/core/settings/mssql.py` -- `src/mountainash_data/core/settings/snowflake.py` -- `src/mountainash_data/core/settings/bigquery.py` -- `src/mountainash_data/core/settings/redshift.py` -- `src/mountainash_data/core/settings/pyspark.py` -- `src/mountainash_data/core/settings/trino.py` -- `src/mountainash_data/core/settings/pyiceberg_rest.py` -- `src/mountainash_data/core/settings/__init__.py` — re-exports only (no wildcard) -- `src/mountainash_data/core/factories/settings_factory.py` — consume registry -- `src/mountainash_data/core/factories/connection_factory.py` — call `to_driver_kwargs` -- `src/mountainash_data/core/factories/operations_factory.py` — same -- `src/mountainash_data/core/factories/settings_type_factory_mixin.py` — registry lookup -- `src/mountainash_data/core/connection.py` — new profile API -- `src/mountainash_data/core/utils.py` (`DatabaseUtils`) — new profile API -- `src/mountainash_data/backends/ibis/connection.py` — consume `to_driver_kwargs` -- `src/mountainash_data/backends/iceberg/connection.py` — consume `to_driver_kwargs` - -### Deleted - -- `src/mountainash_data/core/settings/base.py` — `BaseDBAuthSettings` retired -- `src/mountainash_data/core/settings/exceptions.py` — `DBAuthValidationError` retired (one-line alias shim lives in `__init__.py` if needed) - -### Conventions - -- All new files: Google-style docstrings, typing annotations, ruff-clean, mypy-clean. -- New tests marked `@pytest.mark.unit` unless they touch a real driver. -- Each task ends in `hatch run test:test-quick` passing before commit. -- Commit message format: `feat(settings): ` for new code; `refactor(settings): ` for per-backend migrations; `chore(settings): ` for deletions / re-exports. - ---- - -## Task 1: Create `ParameterSpec`, `BackendDescriptor`, and `MISSING` sentinel - -**Files:** -- Create: `src/mountainash_data/core/settings/descriptor.py` -- Test: `tests/test_unit/core/settings/test_descriptor.py` - -- [ ] **Step 1: Write the failing tests** - -```python -# tests/test_unit/core/settings/test_descriptor.py -"""Unit tests for settings descriptor primitives.""" - -import pytest -from typing import Literal, Optional - -from mountainash_data.core.settings.descriptor import ( - BackendDescriptor, - MISSING, - ParameterSpec, -) - - -@pytest.mark.unit -class TestParameterSpec: - def test_minimal_parameter_spec(self): - spec = ParameterSpec(name="FOO", type=str, tier="core") - assert spec.name == "FOO" - assert spec.type is str - assert spec.tier == "core" - assert spec.default is MISSING - assert spec.driver_key is None - assert spec.secret is False - assert spec.transform is None - assert spec.validator is None - - def test_parameter_spec_is_frozen(self): - spec = ParameterSpec(name="FOO", type=str, tier="core") - with pytest.raises(Exception): - spec.name = "BAR" # type: ignore[misc] - - def test_parameter_spec_with_default(self): - spec = ParameterSpec(name="PORT", type=int, tier="core", default=5432) - assert spec.default == 5432 - - def test_parameter_spec_secret_flag(self): - spec = ParameterSpec(name="PASSWORD", type=str, tier="core", secret=True) - assert spec.secret is True - - def test_parameter_spec_tier_must_be_valid(self): - # typing.Literal is not runtime-enforced by dataclass, but we document - # the constraint; downstream registry invariants will enforce. - spec = ParameterSpec(name="FOO", type=str, tier="core") - assert spec.tier in {"core", "advanced"} - - -@pytest.mark.unit -class TestBackendDescriptor: - def test_minimal_descriptor(self): - desc = BackendDescriptor( - name="sqlite", - provider_type="sqlite", - parameters=[], - auth_modes=[], - ) - assert desc.name == "sqlite" - assert desc.default_port is None - assert desc.connection_string_scheme is None - assert desc.ibis_dialect is None - assert desc.rides_on is None - - def test_descriptor_is_frozen(self): - desc = BackendDescriptor( - name="sqlite", provider_type="sqlite", parameters=[], auth_modes=[] - ) - with pytest.raises(Exception): - desc.name = "mysql" # type: ignore[misc] -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `hatch run test:test-target tests/test_unit/core/settings/test_descriptor.py -v` -Expected: collection error — module `mountainash_data.core.settings.descriptor` not found. - -- [ ] **Step 3: Implement `descriptor.py`** - -```python -# src/mountainash_data/core/settings/descriptor.py -"""Declarative descriptors for backend settings. - -A :class:`BackendDescriptor` captures everything the generic -:class:`~mountainash_data.core.settings.profile.ConnectionProfile` base needs -to configure pydantic fields and driver mappings for a given backend. A -:class:`ParameterSpec` describes one field within a descriptor. -""" - -from __future__ import annotations - -import typing as t -from dataclasses import dataclass, field - -__all__ = ["MISSING", "ParameterSpec", "BackendDescriptor"] - - -class _Missing: - """Sentinel indicating a required (no-default) field. - - Pydantic ``Field(...)`` is emitted when a :class:`ParameterSpec` default - is this sentinel; ``Field(default=...)`` otherwise. - """ - - _instance: "t.ClassVar[_Missing | None]" = None - - def __new__(cls) -> "_Missing": - if cls._instance is None: - cls._instance = super().__new__(cls) - return cls._instance - - def __repr__(self) -> str: - return "MISSING" - - def __bool__(self) -> bool: - return False - - -MISSING: _Missing = _Missing() - - -@dataclass(frozen=True, kw_only=True) -class ParameterSpec: - """One settings field on a backend. - - Attributes: - name: Settings-facing uppercase name (e.g. ``"SSL_CERT"``). - type: Pydantic-compatible annotation (``str``, ``int | None``, enum, …). - tier: ``"core"`` or ``"advanced"`` — audit-style severity tier. - default: Default value; :data:`MISSING` means the field is required. - description: Optional docstring for generated schemas / help output. - driver_key: Driver kwarg name for 1:1 mappings (e.g. ``"sslcert"``). - ``None`` means the adapter handles it. - secret: If ``True``, wrap ``type`` as :class:`pydantic.SecretStr` and - auto-unwrap via ``.get_secret_value()`` at the kwargs boundary. - transform: Optional callable applied when emitting driver kwargs - (e.g. :class:`~pathlib.Path` → ``str``, ``bool`` → ``"0"``/``"1"``). - validator: Optional pydantic-compatible field-level validator. - """ - - name: str - type: t.Any - tier: t.Literal["core", "advanced"] - default: t.Any = MISSING - description: str = "" - driver_key: str | None = None - secret: bool = False - transform: t.Callable[[t.Any], t.Any] | None = None - validator: t.Callable[[t.Any], t.Any] | None = None - - -@dataclass(frozen=True, kw_only=True) -class BackendDescriptor: - """Immutable description of a single backend. - - Attributes: - name: Lowercase short name (``"postgresql"``, ``"pyiceberg_rest"``). - provider_type: Canonical provider identifier - (``CONST_DB_PROVIDER_TYPE`` member). - default_port: Default TCP port if the backend listens on one. - parameters: Ordered list of :class:`ParameterSpec`. - auth_modes: Tuple of :class:`~...settings.auth.base.AuthSpec` subclasses - this backend accepts. - connection_string_scheme: Scheme prefix (``"postgresql://"``) or - ``None`` if the backend does not use a URL form. - ibis_dialect: Name of the Ibis backend if Ibis handles this backend. - rides_on: Name of another backend whose Ibis path this one routes - through (e.g. ``motherduck`` → ``duckdb``). Metadata only — no - runtime behavior. - """ - - name: str - provider_type: t.Any # CONST_DB_PROVIDER_TYPE member - parameters: list[ParameterSpec] - auth_modes: list[type] # list[type[AuthSpec]] — forward-refd to avoid cycle - default_port: int | None = None - connection_string_scheme: str | None = None - ibis_dialect: str | None = None - rides_on: str | None = None -``` - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `hatch run test:test-target tests/test_unit/core/settings/test_descriptor.py -v` -Expected: PASS (5 tests). - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/core/settings/descriptor.py \ - tests/test_unit/core/settings/__init__.py \ - tests/test_unit/core/settings/test_descriptor.py -git commit -m "feat(settings): add ParameterSpec and BackendDescriptor primitives" -``` - ---- - -## Task 2: `AuthSpec` hierarchy - -**Files:** -- Create: `src/mountainash_data/core/settings/auth/__init__.py` -- Create: `src/mountainash_data/core/settings/auth/base.py` -- Create: `src/mountainash_data/core/settings/auth/none.py` -- Create: `src/mountainash_data/core/settings/auth/password.py` -- Create: `src/mountainash_data/core/settings/auth/token.py` -- Create: `src/mountainash_data/core/settings/auth/oauth2.py` -- Create: `src/mountainash_data/core/settings/auth/service_account.py` -- Create: `src/mountainash_data/core/settings/auth/iam.py` -- Create: `src/mountainash_data/core/settings/auth/azure.py` -- Create: `src/mountainash_data/core/settings/auth/kerberos.py` -- Create: `src/mountainash_data/core/settings/auth/certificate.py` -- Test: `tests/test_unit/core/settings/test_auth.py` - -- [ ] **Step 1: Write the failing tests** - -```python -# tests/test_unit/core/settings/test_auth.py -"""Unit tests for AuthSpec discriminated-union members.""" - -import pytest -from pydantic import SecretStr, ValidationError - -from mountainash_data.core.settings.auth import ( - AuthSpec, - AzureADAuth, - CertificateAuth, - IAMAuth, - JWTAuth, - KerberosAuth, - NoAuth, - OAuth2Auth, - PasswordAuth, - ServiceAccountAuth, - TokenAuth, - WindowsAuth, -) - - -@pytest.mark.unit -class TestAuthDiscriminator: - @pytest.mark.parametrize( - "cls, kind", - [ - (NoAuth, "none"), - (PasswordAuth, "password"), - (TokenAuth, "token"), - (JWTAuth, "jwt"), - (OAuth2Auth, "oauth2"), - (ServiceAccountAuth, "service_account"), - (IAMAuth, "iam"), - (WindowsAuth, "windows"), - (AzureADAuth, "azure_ad"), - (KerberosAuth, "kerberos"), - (CertificateAuth, "certificate"), - ], - ) - def test_every_auth_has_discriminator_kind(self, cls, kind): - # Build with minimal required fields - if cls is PasswordAuth: - instance = cls(username="u", password=SecretStr("p")) - elif cls in (TokenAuth, JWTAuth): - instance = cls(token=SecretStr("t")) - else: - instance = cls() - assert instance.kind == kind - - def test_password_auth_requires_username_and_password(self): - with pytest.raises(ValidationError): - PasswordAuth() # type: ignore[call-arg] - - def test_password_auth_wraps_password_as_secretstr(self): - auth = PasswordAuth(username="alice", password="hunter2") - assert isinstance(auth.password, SecretStr) - assert auth.password.get_secret_value() == "hunter2" - - def test_noauth_has_no_fields(self): - auth = NoAuth() - assert auth.kind == "none" - - -@pytest.mark.unit -class TestOAuth2Auth: - def test_all_fields_optional(self): - auth = OAuth2Auth() - assert auth.client_id is None - assert auth.client_secret is None - assert auth.token is None - - def test_token_is_secret(self): - auth = OAuth2Auth(token="t") - assert isinstance(auth.token, SecretStr) -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `hatch run test:test-target tests/test_unit/core/settings/test_auth.py -v` -Expected: collection error — modules not found. - -- [ ] **Step 3: Implement the AuthSpec base and members** - -```python -# src/mountainash_data/core/settings/auth/base.py -"""Base class for discriminated-union auth specifications.""" - -from __future__ import annotations - -from pydantic import BaseModel, ConfigDict - -__all__ = ["AuthSpec"] - - -class AuthSpec(BaseModel): - """Abstract tagged base for authentication specifications. - - Each concrete subclass sets a ``kind`` Literal used as the pydantic - discriminator value when composed into a union. - """ - - model_config = ConfigDict(extra="forbid", frozen=True) - - kind: str -``` - -```python -# src/mountainash_data/core/settings/auth/none.py -"""The 'no authentication' variant.""" - -from __future__ import annotations - -import typing as t - -from .base import AuthSpec - -__all__ = ["NoAuth"] - - -class NoAuth(AuthSpec): - """No authentication required (SQLite, DuckDB, PySpark).""" - - kind: t.Literal["none"] = "none" -``` - -```python -# src/mountainash_data/core/settings/auth/password.py -"""Classic username + password authentication.""" - -from __future__ import annotations - -import typing as t - -from pydantic import SecretStr - -from .base import AuthSpec - -__all__ = ["PasswordAuth"] - - -class PasswordAuth(AuthSpec): - """Username + password authentication.""" - - kind: t.Literal["password"] = "password" - username: str - password: SecretStr -``` - -```python -# src/mountainash_data/core/settings/auth/token.py -"""Bearer-token and JWT authentication.""" - -from __future__ import annotations - -import typing as t - -from pydantic import SecretStr - -from .base import AuthSpec - -__all__ = ["TokenAuth", "JWTAuth"] - - -class TokenAuth(AuthSpec): - """Opaque bearer token (e.g. MotherDuck, PyIceberg REST).""" - - kind: t.Literal["token"] = "token" - token: SecretStr - - -class JWTAuth(AuthSpec): - """JSON Web Token authentication (e.g. Trino).""" - - kind: t.Literal["jwt"] = "jwt" - token: SecretStr -``` - -```python -# src/mountainash_data/core/settings/auth/oauth2.py -"""OAuth2 client-credentials / token authentication.""" - -from __future__ import annotations - -import typing as t - -from pydantic import SecretStr - -from .base import AuthSpec - -__all__ = ["OAuth2Auth"] - - -class OAuth2Auth(AuthSpec): - """OAuth2 credential set (Snowflake, Trino, PyIceberg REST).""" - - kind: t.Literal["oauth2"] = "oauth2" - client_id: str | None = None - client_secret: SecretStr | None = None - token: SecretStr | None = None - refresh_token: SecretStr | None = None - server_uri: str | None = None - scope: str | None = None -``` - -```python -# src/mountainash_data/core/settings/auth/service_account.py -"""Google-style service-account authentication.""" - -from __future__ import annotations - -import typing as t -from pathlib import Path - -from .base import AuthSpec - -__all__ = ["ServiceAccountAuth"] - - -class ServiceAccountAuth(AuthSpec): - """Google Cloud service-account key (JSON dict or file path).""" - - kind: t.Literal["service_account"] = "service_account" - info: dict[str, t.Any] | None = None - file: Path | None = None -``` - -```python -# src/mountainash_data/core/settings/auth/iam.py -"""AWS IAM authentication.""" - -from __future__ import annotations - -import typing as t - -from pydantic import SecretStr - -from .base import AuthSpec - -__all__ = ["IAMAuth"] - - -class IAMAuth(AuthSpec): - """AWS IAM credentials (Redshift, S3-backed catalogs).""" - - kind: t.Literal["iam"] = "iam" - role_arn: str | None = None - access_key_id: str | None = None - secret_access_key: SecretStr | None = None - session_token: SecretStr | None = None - profile_name: str | None = None -``` - -```python -# src/mountainash_data/core/settings/auth/azure.py -"""Microsoft-centric authentication: Windows integrated + Azure AD.""" - -from __future__ import annotations - -import typing as t - -from pydantic import SecretStr - -from .base import AuthSpec - -__all__ = ["WindowsAuth", "AzureADAuth"] - - -class WindowsAuth(AuthSpec): - """Integrated Windows authentication (MSSQL).""" - - kind: t.Literal["windows"] = "windows" - username: str | None = None - domain: str | None = None - - -class AzureADAuth(AuthSpec): - """Azure Active Directory authentication (MSSQL).""" - - kind: t.Literal["azure_ad"] = "azure_ad" - tenant_id: str | None = None - client_id: str | None = None - client_secret: SecretStr | None = None - managed_identity: bool = False - msi_endpoint: str | None = None -``` - -```python -# src/mountainash_data/core/settings/auth/kerberos.py -"""Kerberos / GSSAPI authentication.""" - -from __future__ import annotations - -import typing as t -from pathlib import Path - -from .base import AuthSpec - -__all__ = ["KerberosAuth"] - - -class KerberosAuth(AuthSpec): - """Kerberos authentication (Trino, PostgreSQL via GSS).""" - - kind: t.Literal["kerberos"] = "kerberos" - service_name: str = "postgres" - principal: str | None = None - keytab: Path | None = None -``` - -```python -# src/mountainash_data/core/settings/auth/certificate.py -"""Private-key / certificate authentication (Snowflake JWT).""" - -from __future__ import annotations - -import typing as t -from pathlib import Path - -from pydantic import SecretStr - -from .base import AuthSpec - -__all__ = ["CertificateAuth"] - - -class CertificateAuth(AuthSpec): - """Private-key signed JWT authentication (Snowflake).""" - - kind: t.Literal["certificate"] = "certificate" - private_key: SecretStr | None = None - private_key_path: Path | None = None - passphrase: SecretStr | None = None -``` - -```python -# src/mountainash_data/core/settings/auth/__init__.py -"""Discriminated-union AuthSpec members.""" - -from .azure import AzureADAuth, WindowsAuth -from .base import AuthSpec -from .certificate import CertificateAuth -from .iam import IAMAuth -from .kerberos import KerberosAuth -from .none import NoAuth -from .oauth2 import OAuth2Auth -from .password import PasswordAuth -from .service_account import ServiceAccountAuth -from .token import JWTAuth, TokenAuth - -__all__ = [ - "AuthSpec", - "NoAuth", - "PasswordAuth", - "TokenAuth", - "JWTAuth", - "OAuth2Auth", - "ServiceAccountAuth", - "IAMAuth", - "WindowsAuth", - "AzureADAuth", - "KerberosAuth", - "CertificateAuth", -] -``` - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `hatch run test:test-target tests/test_unit/core/settings/test_auth.py -v` -Expected: PASS (15 tests). - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/core/settings/auth/ \ - tests/test_unit/core/settings/test_auth.py -git commit -m "feat(settings): add AuthSpec discriminated-union hierarchy" -``` - ---- - -## Task 3: Auth dispatch map - -**Files:** -- Create: `src/mountainash_data/core/settings/auth/dispatch.py` -- Test: `tests/test_unit/core/settings/test_auth_dispatch.py` - -- [ ] **Step 1: Write the failing tests** - -```python -# tests/test_unit/core/settings/test_auth_dispatch.py -"""Default AUTH_TO_DRIVER_KWARGS coverage tests.""" - -import pytest -from pydantic import SecretStr - -from mountainash_data.core.settings.auth import ( - AuthSpec, - IAMAuth, - JWTAuth, - NoAuth, - OAuth2Auth, - PasswordAuth, - TokenAuth, -) -from mountainash_data.core.settings.auth.dispatch import auth_to_driver_kwargs - - -@pytest.mark.unit -class TestAuthToDriverKwargs: - def test_noauth_returns_empty(self): - assert auth_to_driver_kwargs(NoAuth()) == {} - - def test_password_unwraps_secret(self): - auth = PasswordAuth(username="alice", password=SecretStr("hunter2")) - assert auth_to_driver_kwargs(auth) == { - "user": "alice", - "password": "hunter2", - } - - def test_token_unwraps_secret(self): - auth = TokenAuth(token=SecretStr("t")) - assert auth_to_driver_kwargs(auth) == {"token": "t"} - - def test_jwt_unwraps_secret(self): - auth = JWTAuth(token=SecretStr("j")) - assert auth_to_driver_kwargs(auth) == {"token": "j"} - - def test_oauth2_with_token(self): - auth = OAuth2Auth(token=SecretStr("bearer")) - assert auth_to_driver_kwargs(auth) == {"token": "bearer"} - - def test_oauth2_with_client_credentials(self): - auth = OAuth2Auth( - client_id="cid", client_secret=SecretStr("csec") - ) - assert auth_to_driver_kwargs(auth) == {"credential": "cid:csec"} - - def test_iam_with_keys(self): - auth = IAMAuth( - access_key_id="AKIA...", - secret_access_key=SecretStr("sk"), - session_token=SecretStr("st"), - ) - assert auth_to_driver_kwargs(auth) == { - "aws_access_key_id": "AKIA...", - "aws_secret_access_key": "sk", - "aws_session_token": "st", - } - - def test_iam_with_role_arn(self): - auth = IAMAuth(role_arn="arn:aws:iam::123:role/x") - assert auth_to_driver_kwargs(auth) == { - "iam_role_arn": "arn:aws:iam::123:role/x" - } - - def test_unknown_auth_type_raises(self): - class WeirdAuth(AuthSpec): - kind: str = "weird" # type: ignore[assignment] - - with pytest.raises(KeyError): - auth_to_driver_kwargs(WeirdAuth()) -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `hatch run test:test-target tests/test_unit/core/settings/test_auth_dispatch.py -v` -Expected: collection error — `dispatch` module not found. - -- [ ] **Step 3: Implement `dispatch.py`** - -```python -# src/mountainash_data/core/settings/auth/dispatch.py -"""Default mapping from AuthSpec instances to driver kwargs. - -Backend adapters can override individual auth types by consulting this map or -by writing bespoke match statements. The defaults cover the common case. -""" - -from __future__ import annotations - -import typing as t - -from .base import AuthSpec -from .iam import IAMAuth -from .none import NoAuth -from .oauth2 import OAuth2Auth -from .password import PasswordAuth -from .token import JWTAuth, TokenAuth - -__all__ = ["AUTH_TO_DRIVER_KWARGS", "auth_to_driver_kwargs"] - - -def _noauth(_auth: NoAuth) -> dict[str, t.Any]: - return {} - - -def _password(auth: PasswordAuth) -> dict[str, t.Any]: - return { - "user": auth.username, - "password": auth.password.get_secret_value(), - } - - -def _token(auth: TokenAuth) -> dict[str, t.Any]: - return {"token": auth.token.get_secret_value()} - - -def _jwt(auth: JWTAuth) -> dict[str, t.Any]: - return {"token": auth.token.get_secret_value()} - - -def _oauth2(auth: OAuth2Auth) -> dict[str, t.Any]: - if auth.token is not None: - return {"token": auth.token.get_secret_value()} - if auth.client_id is not None and auth.client_secret is not None: - return {"credential": f"{auth.client_id}:{auth.client_secret.get_secret_value()}"} - return {} - - -def _iam(auth: IAMAuth) -> dict[str, t.Any]: - out: dict[str, t.Any] = {} - if auth.role_arn is not None: - out["iam_role_arn"] = auth.role_arn - if auth.access_key_id is not None: - out["aws_access_key_id"] = auth.access_key_id - if auth.secret_access_key is not None: - out["aws_secret_access_key"] = auth.secret_access_key.get_secret_value() - if auth.session_token is not None: - out["aws_session_token"] = auth.session_token.get_secret_value() - return out - - -AUTH_TO_DRIVER_KWARGS: dict[type[AuthSpec], t.Callable[[t.Any], dict[str, t.Any]]] = { - NoAuth: _noauth, - PasswordAuth: _password, - TokenAuth: _token, - JWTAuth: _jwt, - OAuth2Auth: _oauth2, - IAMAuth: _iam, - # WindowsAuth, AzureADAuth, KerberosAuth, ServiceAccountAuth, CertificateAuth: - # no sensible default — their respective backend adapters handle mapping. -} - - -def auth_to_driver_kwargs(auth: AuthSpec) -> dict[str, t.Any]: - """Look up the default mapper for ``auth`` and produce driver kwargs. - - Raises: - KeyError: if no mapper is registered for ``type(auth)``. - """ - return AUTH_TO_DRIVER_KWARGS[type(auth)](auth) -``` - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `hatch run test:test-target tests/test_unit/core/settings/test_auth_dispatch.py -v` -Expected: PASS (9 tests). - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/core/settings/auth/dispatch.py \ - tests/test_unit/core/settings/test_auth_dispatch.py -git commit -m "feat(settings): add default AUTH_TO_DRIVER_KWARGS dispatch map" -``` - ---- - -## Task 4: `ConnectionProfile` base - -**Files:** -- Create: `src/mountainash_data/core/settings/profile.py` -- Test: `tests/test_unit/core/settings/test_profile.py` - -- [ ] **Step 1: Write the failing tests** - -```python -# tests/test_unit/core/settings/test_profile.py -"""Unit tests for the generic ConnectionProfile base.""" - -from __future__ import annotations - -import pytest -from pydantic import SecretStr - -from mountainash_data.core.settings.auth import NoAuth, PasswordAuth -from mountainash_data.core.settings.descriptor import ( - BackendDescriptor, - ParameterSpec, -) -from mountainash_data.core.settings.profile import ConnectionProfile - - -DUMMY_DESCRIPTOR = BackendDescriptor( - name="dummy", - provider_type="dummy", - default_port=9999, - connection_string_scheme="dummy://", - parameters=[ - ParameterSpec(name="HOST", type=str, tier="core", driver_key="host"), - ParameterSpec(name="PORT", type=int, tier="core", default=9999, driver_key="port"), - ParameterSpec(name="PASSWORD", type=str, tier="core", secret=True, - driver_key="password", default=None), - ], - auth_modes=[NoAuth, PasswordAuth], -) - - -class DummyProfile(ConnectionProfile): - __descriptor__ = DUMMY_DESCRIPTOR - - -@pytest.mark.unit -class TestConnectionProfile: - def test_required_field_enforced(self): - from pydantic import ValidationError - - with pytest.raises(ValidationError): - DummyProfile(auth=NoAuth()) # HOST missing - - def test_default_used_when_not_provided(self): - p = DummyProfile(HOST="localhost", auth=NoAuth()) - assert p.PORT == 9999 - - def test_to_driver_kwargs_noauth(self): - p = DummyProfile(HOST="h", PORT=1234, auth=NoAuth()) - assert p.to_driver_kwargs() == {"host": "h", "port": 1234} - - def test_to_driver_kwargs_password_auth_unwraps_secret(self): - p = DummyProfile( - HOST="h", - auth=PasswordAuth(username="u", password=SecretStr("p")), - ) - kwargs = p.to_driver_kwargs() - assert kwargs["host"] == "h" - assert kwargs["user"] == "u" - assert kwargs["password"] == "p" - - def test_secret_field_unwrapped_in_driver_kwargs(self): - p = DummyProfile(HOST="h", PASSWORD="literal-secret", auth=NoAuth()) - kwargs = p.to_driver_kwargs() - assert kwargs["password"] == "literal-secret" - - def test_none_values_skipped_from_driver_kwargs(self): - p = DummyProfile(HOST="h", auth=NoAuth()) - kwargs = p.to_driver_kwargs() - assert "password" not in kwargs # PASSWORD default is None - - def test_provider_type_property(self): - p = DummyProfile(HOST="h", auth=NoAuth()) - assert p.provider_type == "dummy" - - def test_backend_property(self): - p = DummyProfile(HOST="h", auth=NoAuth()) - assert p.backend == "dummy" - - def test_to_connection_string_raises_when_scheme_none(self): - desc = BackendDescriptor( - name="x", provider_type="x", parameters=[], auth_modes=[NoAuth], - connection_string_scheme=None, - ) - - class P(ConnectionProfile): - __descriptor__ = desc - - p = P(auth=NoAuth()) - with pytest.raises(NotImplementedError): - p.to_connection_string() -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `hatch run test:test-target tests/test_unit/core/settings/test_profile.py -v` -Expected: collection error — `profile` module not found. - -- [ ] **Step 3: Implement `profile.py`** - -```python -# src/mountainash_data/core/settings/profile.py -"""Generic ConnectionProfile base for all backend settings. - -A subclass declares ``__descriptor__`` (a :class:`BackendDescriptor`); this -base uses ``__init_subclass__`` to materialize the descriptor into pydantic -fields, compose the :class:`~...auth.AuthSpec` union into the ``auth`` field, -and install the generic ``to_driver_kwargs`` / ``to_connection_string`` API. -""" - -from __future__ import annotations - -import typing as t - -from pydantic import Field, SecretStr, create_model -from pydantic.fields import FieldInfo - -from mountainash_settings import MountainAshBaseSettings - -from .auth.base import AuthSpec -from .auth.dispatch import auth_to_driver_kwargs -from .descriptor import MISSING, BackendDescriptor, ParameterSpec - -__all__ = ["ConnectionProfile"] - - -def _pydantic_field(spec: ParameterSpec) -> tuple[t.Any, FieldInfo]: - """Translate a :class:`ParameterSpec` into a pydantic ``(type, FieldInfo)``.""" - ptype: t.Any = SecretStr if spec.secret else spec.type - if spec.default is MISSING: - info = Field(...) - else: - info = Field(default=spec.default) - if spec.description: - info = Field(default=info.default if info.default is not ... else ..., - description=spec.description) - return ptype, info - - -class ConnectionProfile(MountainAshBaseSettings): - """Declarative settings base — subclasses set ``__descriptor__`` only. - - Public API: - - :attr:`backend` — descriptor name. - - :attr:`provider_type` — descriptor provider_type. - - :meth:`to_driver_kwargs` — dict ready for the underlying driver. - - :meth:`to_connection_string` — URL form (or ``NotImplementedError`` - if the backend has no URL scheme). - """ - - __descriptor__: t.ClassVar[BackendDescriptor] - __adapter__: t.ClassVar[t.Callable[["ConnectionProfile"], dict[str, t.Any]] | None] = None - - auth: AuthSpec - - def __init_subclass__(cls, **kwargs: t.Any) -> None: - super().__init_subclass__(**kwargs) - desc = getattr(cls, "__descriptor__", None) - if desc is None: - return # abstract intermediate subclass - - # Install descriptor fields on the pydantic model. - for spec in desc.parameters: - ptype, info = _pydantic_field(spec) - cls.model_fields[spec.name] = FieldInfo( - annotation=ptype, default=info.default, - description=info.description, - ) - - # Install the discriminated-union auth field from descriptor.auth_modes. - if desc.auth_modes: - union = t.Union[tuple(desc.auth_modes)] # type: ignore[valid-type] - cls.model_fields["auth"] = FieldInfo( - annotation=union, default=..., # required - discriminator="kind", - ) - - cls.model_rebuild(force=True) - - # --- Public properties --------------------------------------------------- - - @property - def backend(self) -> str: - return self.__descriptor__.name - - @property - def provider_type(self) -> t.Any: - return self.__descriptor__.provider_type - - # --- Driver kwargs -------------------------------------------------------- - - def _default_driver_kwargs(self) -> dict[str, t.Any]: - """Emit 1:1 driver_key mappings from the descriptor. - - - Skips ``None`` values. - - Unwraps :class:`SecretStr` via ``.get_secret_value()``. - - Applies ``ParameterSpec.transform`` if set. - """ - out: dict[str, t.Any] = {} - for spec in self.__descriptor__.parameters: - if spec.driver_key is None: - continue - val = getattr(self, spec.name, None) - if val is None: - continue - if isinstance(val, SecretStr): - val = val.get_secret_value() - if spec.transform is not None: - val = spec.transform(val) - out[spec.driver_key] = val - return out - - def _auth_to_driver_kwargs(self) -> dict[str, t.Any]: - return auth_to_driver_kwargs(self.auth) - - def to_driver_kwargs(self) -> dict[str, t.Any]: - """Build the final driver kwargs dict. - - Order: - 1. 1:1 parameter mappings. - 2. Auth dispatch (may overwrite 1:1 outputs). - 3. Per-backend adapter, if any (may overwrite auth outputs). - """ - kwargs = self._default_driver_kwargs() - kwargs.update(self._auth_to_driver_kwargs()) - if self.__adapter__ is not None: - kwargs = self.__adapter__(self) - return kwargs - - # --- Connection string ---------------------------------------------------- - - def to_connection_string(self) -> str: - """Build ``scheme://...`` form from the descriptor. - - Raises :class:`NotImplementedError` if the descriptor has no scheme. - Backends with non-standard URL shapes override this method. - """ - scheme = self.__descriptor__.connection_string_scheme - if scheme is None: - raise NotImplementedError( - f"Backend {self.backend!r} has no connection string scheme" - ) - # Default implementation: scheme + host + optional port + optional db. - host = getattr(self, "HOST", None) - port = getattr(self, "PORT", None) - database = getattr(self, "DATABASE", None) - url = scheme - if isinstance(self.auth, AuthSpec) and getattr(self.auth, "username", None): - url += f"{self.auth.username}" - pw = getattr(self.auth, "password", None) - if isinstance(pw, SecretStr): - url += f":{pw.get_secret_value()}" - url += "@" - if host is not None: - url += str(host) - if port is not None: - url += f":{port}" - if database is not None: - url += f"/{database}" - return url -``` - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `hatch run test:test-target tests/test_unit/core/settings/test_profile.py -v` -Expected: PASS (9 tests). - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/core/settings/profile.py \ - tests/test_unit/core/settings/test_profile.py -git commit -m "feat(settings): add ConnectionProfile generic base" -``` - ---- - -## Task 5: Registry - -**Files:** -- Create: `src/mountainash_data/core/settings/registry.py` -- Test: `tests/test_unit/core/settings/test_registry.py` - -- [ ] **Step 1: Write the failing tests** - -```python -# tests/test_unit/core/settings/test_registry.py -"""Unit tests for the backend registry.""" - -import pytest - -from mountainash_data.core.settings.auth import NoAuth -from mountainash_data.core.settings.descriptor import BackendDescriptor -from mountainash_data.core.settings.profile import ConnectionProfile -from mountainash_data.core.settings.registry import ( - REGISTRY, - get_descriptor, - get_settings_class, - register, -) - - -@pytest.mark.unit -class TestRegistry: - def setup_method(self): - self._saved = REGISTRY.copy() - - def teardown_method(self): - REGISTRY.clear() - REGISTRY.update(self._saved) - - def test_register_inserts_into_registry(self): - desc = BackendDescriptor( - name="my_backend", provider_type="my_backend", - parameters=[], auth_modes=[NoAuth], - ) - - @register(desc) - class MyProfile(ConnectionProfile): - __descriptor__ = desc - - assert REGISTRY["my_backend"] is desc - assert MyProfile.__descriptor__ is desc - - def test_get_descriptor_returns_registered(self): - desc = BackendDescriptor( - name="x", provider_type="x", - parameters=[], auth_modes=[NoAuth], - ) - - @register(desc) - class X(ConnectionProfile): - __descriptor__ = desc - - assert get_descriptor("x") is desc - - def test_get_descriptor_unknown_raises(self): - with pytest.raises(KeyError): - get_descriptor("not_a_real_backend") - - def test_get_settings_class_returns_class(self): - desc = BackendDescriptor( - name="y", provider_type="y", - parameters=[], auth_modes=[NoAuth], - ) - - @register(desc) - class YProfile(ConnectionProfile): - __descriptor__ = desc - - assert get_settings_class("y") is YProfile - - def test_register_rejects_duplicate_name(self): - desc1 = BackendDescriptor(name="dup", provider_type="dup", - parameters=[], auth_modes=[NoAuth]) - desc2 = BackendDescriptor(name="dup", provider_type="dup", - parameters=[], auth_modes=[NoAuth]) - - @register(desc1) - class P1(ConnectionProfile): - __descriptor__ = desc1 - - with pytest.raises(ValueError, match="already registered"): - @register(desc2) - class P2(ConnectionProfile): - __descriptor__ = desc2 -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `hatch run test:test-target tests/test_unit/core/settings/test_registry.py -v` -Expected: collection error — `registry` module not found. - -- [ ] **Step 3: Implement `registry.py`** - -```python -# src/mountainash_data/core/settings/registry.py -"""Module-level registry of backend descriptors and settings classes.""" - -from __future__ import annotations - -import typing as t - -from .descriptor import BackendDescriptor -from .profile import ConnectionProfile - -__all__ = ["REGISTRY", "register", "get_descriptor", "get_settings_class"] - -REGISTRY: dict[str, BackendDescriptor] = {} -_CLASSES: dict[str, type[ConnectionProfile]] = {} - - -T = t.TypeVar("T", bound=ConnectionProfile) - - -def register( - descriptor: BackendDescriptor, -) -> t.Callable[[type[T]], type[T]]: - """Class decorator that registers a :class:`ConnectionProfile` subclass. - - Raises: - ValueError: if ``descriptor.name`` is already registered. - """ - if descriptor.name in REGISTRY: - raise ValueError( - f"Backend {descriptor.name!r} is already registered" - ) - - def _wrap(cls: type[T]) -> type[T]: - REGISTRY[descriptor.name] = descriptor - _CLASSES[descriptor.name] = cls - cls.__descriptor__ = descriptor # belt-and-braces - return cls - - return _wrap - - -def get_descriptor(name: str) -> BackendDescriptor: - """Return the :class:`BackendDescriptor` for ``name``. - - Raises: - KeyError: if ``name`` is not registered. - """ - return REGISTRY[name] - - -def get_settings_class(name: str) -> type[ConnectionProfile]: - """Return the registered settings class for ``name``. - - Raises: - KeyError: if ``name`` is not registered. - """ - return _CLASSES[name] -``` - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `hatch run test:test-target tests/test_unit/core/settings/test_registry.py -v` -Expected: PASS (5 tests). - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/core/settings/registry.py \ - tests/test_unit/core/settings/test_registry.py -git commit -m "feat(settings): add backend registry with @register decorator" -``` - ---- - -## Task 6: Descriptor invariants (parametric test) - -**Files:** -- Create: `tests/test_unit/core/settings/test_descriptors_invariants.py` - -- [ ] **Step 1: Write the test** - -```python -# tests/test_unit/core/settings/test_descriptors_invariants.py -"""Parametric invariants every registered backend must satisfy. - -Runs once per :data:`REGISTRY` entry. New backends get coverage for free. -""" - -from __future__ import annotations - -import pytest - -from mountainash_data.core.settings.auth.base import AuthSpec -from mountainash_data.core.settings.registry import REGISTRY - - -def _ids(params): - return [name for name, _ in params] - - -@pytest.mark.unit -@pytest.mark.parametrize( - "name,descriptor", - list(REGISTRY.items()), - ids=list(REGISTRY.keys()) or [""], -) -class TestDescriptorInvariants: - def test_name_matches_registry_key(self, name, descriptor): - assert descriptor.name == name - - def test_parameter_names_unique(self, name, descriptor): - names = [p.name for p in descriptor.parameters] - assert len(names) == len(set(names)), f"duplicate param in {name}" - - def test_driver_keys_unique(self, name, descriptor): - keys = [p.driver_key for p in descriptor.parameters if p.driver_key] - assert len(keys) == len(set(keys)), f"duplicate driver_key in {name}" - - def test_parameter_tiers_valid(self, name, descriptor): - for p in descriptor.parameters: - assert p.tier in {"core", "advanced"}, ( - f"{name}.{p.name} has invalid tier {p.tier!r}" - ) - - def test_auth_modes_are_authspec_subclasses(self, name, descriptor): - for mode in descriptor.auth_modes: - assert issubclass(mode, AuthSpec), ( - f"{name}.auth_modes contains non-AuthSpec: {mode}" - ) - - def test_provider_type_is_not_none(self, name, descriptor): - assert descriptor.provider_type is not None, ( - f"{name} has no provider_type" - ) -``` - -- [ ] **Step 2: Run it (empty REGISTRY — should pass trivially)** - -Run: `hatch run test:test-target tests/test_unit/core/settings/test_descriptors_invariants.py -v` -Expected: PASS (0 or 1 empty-parametrize case — acceptable until backends are registered). - -- [ ] **Step 3: Commit** - -```bash -git add tests/test_unit/core/settings/test_descriptors_invariants.py -git commit -m "test(settings): add parametric descriptor invariants" -``` - ---- - -## Task 7: Migrate SQLite (template for all per-backend migrations) - -**Context:** Smallest backend, single parameter (`DATABASE`), no auth, no adapter. This task establishes the pattern every subsequent backend migration follows. - -**Files:** -- Modify: `src/mountainash_data/core/settings/sqlite.py` (full rewrite) -- Test: `tests/test_unit/core/settings/backends/test_sqlite.py` - -- [ ] **Step 1: Write the failing tests** - -```python -# tests/test_unit/core/settings/backends/test_sqlite.py -"""Round-trip and audit-regression tests for SQLite settings.""" - -from __future__ import annotations - -import pytest - -from mountainash_data.core.settings.auth import NoAuth -from mountainash_data.core.settings.sqlite import SQLiteAuthSettings - - -@pytest.mark.unit -class TestSQLiteAuthSettings: - def test_minimal_construction(self): - s = SQLiteAuthSettings(auth=NoAuth()) - assert s.DATABASE is None - assert s.provider_type # non-empty - - def test_database_memory(self): - s = SQLiteAuthSettings(DATABASE=":memory:", auth=NoAuth()) - assert s.DATABASE == ":memory:" - - def test_to_driver_kwargs_memory(self): - s = SQLiteAuthSettings(DATABASE=":memory:", auth=NoAuth()) - assert s.to_driver_kwargs() == {"database": ":memory:"} - - def test_to_driver_kwargs_none_database_dropped(self): - s = SQLiteAuthSettings(auth=NoAuth()) - assert s.to_driver_kwargs() == {} - - def test_type_map_optional(self): - s = SQLiteAuthSettings(DATABASE=":memory:", TYPE_MAP={"SMALLINT": "int32"}, - auth=NoAuth()) - assert s.to_driver_kwargs()["type_map"] == {"SMALLINT": "int32"} -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `hatch run test:test-target tests/test_unit/core/settings/backends/test_sqlite.py -v` -Expected: IMPORT or ATTR error — `SQLiteAuthSettings` still the old shape. - -- [ ] **Step 3: Rewrite `sqlite.py`** - -```python -# src/mountainash_data/core/settings/sqlite.py -"""SQLite backend settings. - -Spec: audit report ``docs/superpowers/specs/2026-04-15-settings-audit/sqlite.md``. -Driver: https://docs.python.org/3/library/sqlite3.html#sqlite3.connect -Ibis: ``ibis.backends.sqlite.do_connect(database, type_map=None)`` -""" - -from __future__ import annotations - -import typing as t - -from ..constants import CONST_DB_PROVIDER_TYPE -from .auth import NoAuth -from .descriptor import BackendDescriptor, ParameterSpec -from .profile import ConnectionProfile -from .registry import register - -__all__ = ["SQLiteAuthSettings", "SQLITE_DESCRIPTOR"] - - -SQLITE_DESCRIPTOR = BackendDescriptor( - name="sqlite", - provider_type=CONST_DB_PROVIDER_TYPE.SQLITE, - connection_string_scheme="sqlite://", - ibis_dialect="sqlite", - auth_modes=[NoAuth], - parameters=[ - ParameterSpec( - name="DATABASE", - type=t.Optional[str], - tier="core", - default=None, - driver_key="database", - description="Path to the SQLite file, or ':memory:' for in-memory.", - ), - ParameterSpec( - name="TYPE_MAP", - type=t.Optional[dict[str, t.Any]], - tier="advanced", - default=None, - driver_key="type_map", - description="Optional SQLite column-type → Ibis dtype overrides.", - ), - ], -) - - -@register(SQLITE_DESCRIPTOR) -class SQLiteAuthSettings(ConnectionProfile): - __descriptor__ = SQLITE_DESCRIPTOR -``` - -- [ ] **Step 4: Run tests** - -Run: `hatch run test:test-target tests/test_unit/core/settings/backends/test_sqlite.py tests/test_unit/core/settings/test_descriptors_invariants.py -v` -Expected: PASS (all SQLite + invariants parameterized for sqlite). - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/core/settings/sqlite.py \ - tests/test_unit/core/settings/backends/__init__.py \ - tests/test_unit/core/settings/backends/test_sqlite.py -git commit -m "refactor(settings): migrate sqlite to descriptor + shell" -``` - ---- - -## Task 8: Migrate DuckDB - -**Files:** -- Modify: `src/mountainash_data/core/settings/duckdb.py` -- Test: `tests/test_unit/core/settings/backends/test_duckdb.py` - -**Audit fixes carried in:** `READ_ONLY` default to `False` (matching Ibis), `MEMORY_LIMIT` regex relaxed, `ATTACH_PATH` removed (was orphan), URL updated to `/docs/current/`. - -- [ ] **Step 1: Write the failing tests** - -```python -# tests/test_unit/core/settings/backends/test_duckdb.py -"""DuckDB settings round-trip and audit-regression tests.""" - -from __future__ import annotations - -import pytest -from pydantic import ValidationError - -from mountainash_data.core.settings.auth import NoAuth -from mountainash_data.core.settings.duckdb import DuckDBAuthSettings - - -@pytest.mark.unit -class TestDuckDBAuthSettings: - def test_default_read_only_is_false(self): - """Audit regression: previously defaulted True, mismatched Ibis.""" - s = DuckDBAuthSettings(auth=NoAuth()) - assert s.READ_ONLY is False - - def test_memory_database_default(self): - s = DuckDBAuthSettings(auth=NoAuth()) - assert s.DATABASE is None - - def test_to_driver_kwargs_default(self): - s = DuckDBAuthSettings(DATABASE=":memory:", auth=NoAuth()) - kwargs = s.to_driver_kwargs() - assert kwargs["database"] == ":memory:" - assert kwargs["read_only"] is False - - def test_memory_limit_decimal_accepted(self): - """Audit regression: regex previously rejected '1.5GB'.""" - s = DuckDBAuthSettings(DATABASE=":memory:", MEMORY_LIMIT="1.5GB", - auth=NoAuth()) - assert s.MEMORY_LIMIT == "1.5GB" - - def test_memory_limit_percent_accepted(self): - s = DuckDBAuthSettings(DATABASE=":memory:", MEMORY_LIMIT="80%", - auth=NoAuth()) - assert s.MEMORY_LIMIT == "80%" - - def test_memory_limit_garbage_rejected(self): - with pytest.raises(ValidationError): - DuckDBAuthSettings(DATABASE=":memory:", MEMORY_LIMIT="lots", - auth=NoAuth()) - - def test_extensions_passed_as_top_level_kwarg(self): - """Audit regression: extensions was packed inside config dict.""" - s = DuckDBAuthSettings(DATABASE=":memory:", EXTENSIONS=["httpfs"], - auth=NoAuth()) - kwargs = s.to_driver_kwargs() - assert kwargs["extensions"] == ["httpfs"] - # Must NOT appear inside a nested config dict: - assert "config" not in kwargs or "extensions" not in kwargs.get("config", {}) -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `hatch run test:test-target tests/test_unit/core/settings/backends/test_duckdb.py -v` -Expected: FAIL on `READ_ONLY is False` (old default was True). - -- [ ] **Step 3: Rewrite `duckdb.py`** - -```python -# src/mountainash_data/core/settings/duckdb.py -"""DuckDB backend settings. - -Spec: audit report ``docs/superpowers/specs/2026-04-15-settings-audit/duckdb.md``. -Driver: https://duckdb.org/docs/current/configuration/overview.html -Ibis: ``ibis.backends.duckdb.do_connect(database=':memory:', read_only=False, - extensions=None, **config)`` -""" - -from __future__ import annotations - -import re -import typing as t - -from pydantic import field_validator - -from ..constants import CONST_DB_PROVIDER_TYPE -from .auth import NoAuth -from .descriptor import BackendDescriptor, ParameterSpec -from .profile import ConnectionProfile -from .registry import register - -__all__ = ["DuckDBAuthSettings", "DUCKDB_DESCRIPTOR"] - -_MEMORY_LIMIT_RE = re.compile(r"^(?:\d+(?:\.\d+)?\s*[KMG]i?B|\d+%)$", re.IGNORECASE) - - -def _validate_memory_limit(value: t.Any) -> t.Any: - if value is None: - return value - if not _MEMORY_LIMIT_RE.match(str(value)): - raise ValueError( - "MEMORY_LIMIT must match e.g. '500MB', '1.5GB', '1024KiB', or '80%'" - ) - return value - - -DUCKDB_DESCRIPTOR = BackendDescriptor( - name="duckdb", - provider_type=CONST_DB_PROVIDER_TYPE.DUCKDB, - connection_string_scheme="duckdb://", - ibis_dialect="duckdb", - auth_modes=[NoAuth], - parameters=[ - ParameterSpec(name="DATABASE", type=t.Optional[str], tier="core", - default=None, driver_key="database"), - ParameterSpec(name="READ_ONLY", type=bool, tier="core", - default=False, driver_key="read_only"), - ParameterSpec(name="EXTENSIONS", type=list[str], tier="core", - default=[], driver_key="extensions"), - ParameterSpec(name="THREADS", type=t.Optional[int], tier="advanced", - default=None), # goes into config; see adapter in future - ParameterSpec(name="MEMORY_LIMIT", type=t.Optional[str], tier="advanced", - default=None, validator=_validate_memory_limit), - ], -) - - -@register(DUCKDB_DESCRIPTOR) -class DuckDBAuthSettings(ConnectionProfile): - __descriptor__ = DUCKDB_DESCRIPTOR - - @field_validator("MEMORY_LIMIT") - @classmethod - def _mem_limit(cls, v: t.Any) -> t.Any: - return _validate_memory_limit(v) -``` - -*Note:* `THREADS` / `MEMORY_LIMIT` go into Ibis's `**config`, not top-level kwargs. Phase 4 handles that via an adapter if needed; for this pass they're declared but not yet emitted as driver kwargs. - -- [ ] **Step 4: Run tests** - -Run: `hatch run test:test-target tests/test_unit/core/settings/backends/test_duckdb.py -v` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/core/settings/duckdb.py \ - tests/test_unit/core/settings/backends/test_duckdb.py -git commit -m "refactor(settings): migrate duckdb, fix READ_ONLY default and MEMORY_LIMIT regex" -``` - ---- - -## Task 9: Migrate PySpark - -**Files:** -- Modify: `src/mountainash_data/core/settings/pyspark.py` -- Test: `tests/test_unit/core/settings/backends/test_pyspark.py` - -**Audit fixes carried:** docstring corrected (was "SQLite authentication settings"); `PARTITIONS` type/default fixed (was `int = {}`); `MODE` becomes `StrEnum`; `SESSION` field added. - -- [ ] **Step 1: Write the failing tests** - -```python -# tests/test_unit/core/settings/backends/test_pyspark.py -from __future__ import annotations - -import pytest - -from mountainash_data.core.settings.auth import NoAuth -from mountainash_data.core.settings.pyspark import ( - PySparkAuthSettings, - PySparkMode, -) - - -@pytest.mark.unit -class TestPySparkAuthSettings: - def test_minimal(self): - s = PySparkAuthSettings(auth=NoAuth()) - assert s.MODE is PySparkMode.BATCH - - def test_mode_streaming(self): - s = PySparkAuthSettings(MODE="streaming", auth=NoAuth()) - assert s.MODE is PySparkMode.STREAMING - - def test_mode_invalid_rejected(self): - from pydantic import ValidationError - - with pytest.raises(ValidationError): - PySparkAuthSettings(MODE="nonsense", auth=NoAuth()) - - def test_partitions_accepts_int(self): - """Audit regression: PARTITIONS: int = {} crashed at init.""" - s = PySparkAuthSettings(PARTITIONS=200, auth=NoAuth()) - assert s.PARTITIONS == 200 - - def test_partitions_none_default(self): - s = PySparkAuthSettings(auth=NoAuth()) - assert s.PARTITIONS is None - - def test_to_driver_kwargs_emits_dotted_spark_keys(self): - """Audit regression: previously emitted 'spark_app_name' not 'spark.app.name'.""" - s = PySparkAuthSettings( - APPLICATION_NAME="myapp", - SPARK_MASTER="local[2]", - MODE="batch", - auth=NoAuth(), - ) - kwargs = s.to_driver_kwargs() - assert kwargs["mode"] == "batch" - # Adapter emits dotted Spark keys: - assert kwargs["spark.app.name"] == "myapp" - assert kwargs["spark.master"] == "local[2]" -``` - -- [ ] **Step 2: Run to confirm failure** - -Run: `hatch run test:test-target tests/test_unit/core/settings/backends/test_pyspark.py -v` -Expected: IMPORT or ATTR error. - -- [ ] **Step 3: Implement** - -Create `src/mountainash_data/core/settings/adapters/pyspark.py`: - -```python -# src/mountainash_data/core/settings/adapters/pyspark.py -"""Adapter emitting dotted spark.* keys from PySpark settings.""" - -from __future__ import annotations - -import typing as t - -from mountainash_data.core.settings.auth import NoAuth - -if t.TYPE_CHECKING: - from mountainash_data.core.settings.pyspark import PySparkAuthSettings - - -def build_driver_kwargs(profile: "PySparkAuthSettings") -> dict[str, t.Any]: - kwargs: dict[str, t.Any] = {} - if profile.MODE is not None: - kwargs["mode"] = profile.MODE.value - if profile.SESSION is not None: - kwargs["session"] = profile.SESSION - if profile.APPLICATION_NAME is not None: - kwargs["spark.app.name"] = profile.APPLICATION_NAME - if profile.SPARK_MASTER is not None: - kwargs["spark.master"] = profile.SPARK_MASTER - if profile.WAREHOUSE_DIR is not None: - kwargs["spark.sql.warehouse.dir"] = profile.WAREHOUSE_DIR - if profile.PARTITIONS is not None: - kwargs["spark.sql.shuffle.partitions"] = profile.PARTITIONS - # NoAuth is the only accepted mode; nothing else to emit. - return kwargs -``` - -Rewrite `pyspark.py`: - -```python -# src/mountainash_data/core/settings/pyspark.py -"""PySpark backend settings. - -Spec: audit report ``docs/superpowers/specs/2026-04-15-settings-audit/pyspark.md``. -Ibis: ``ibis.backends.pyspark.do_connect(session=None, mode='batch', **kwargs)`` -where kwargs flow to ``SparkSession.builder.config(**kwargs)``. - -The docstring of the prior class read 'SQLite authentication settings' — a -copy-paste from ``sqlite.py``. Corrected here. -""" - -from __future__ import annotations - -import typing as t -from enum import StrEnum - -from ..constants import CONST_DB_PROVIDER_TYPE -from .adapters import pyspark as _adapter -from .auth import NoAuth -from .descriptor import BackendDescriptor, ParameterSpec -from .profile import ConnectionProfile -from .registry import register - -__all__ = ["PySparkAuthSettings", "PySparkMode", "PYSPARK_DESCRIPTOR"] - - -class PySparkMode(StrEnum): - BATCH = "batch" - STREAMING = "streaming" - - -PYSPARK_DESCRIPTOR = BackendDescriptor( - name="pyspark", - provider_type=CONST_DB_PROVIDER_TYPE.PYSPARK, - connection_string_scheme=None, # SparkSession, not URL - ibis_dialect="pyspark", - auth_modes=[NoAuth], - parameters=[ - ParameterSpec(name="SESSION", type=t.Optional[t.Any], tier="core", - default=None), - ParameterSpec(name="MODE", type=PySparkMode, tier="core", - default=PySparkMode.BATCH), - ParameterSpec(name="SPARK_MASTER", type=t.Optional[str], tier="advanced", - default=None), - ParameterSpec(name="APPLICATION_NAME", type=t.Optional[str], tier="advanced", - default=None), - ParameterSpec(name="WAREHOUSE_DIR", type=t.Optional[str], tier="advanced", - default=None), - ParameterSpec(name="PARTITIONS", type=t.Optional[int], tier="advanced", - default=None), - ], -) - - -@register(PYSPARK_DESCRIPTOR) -class PySparkAuthSettings(ConnectionProfile): - __descriptor__ = PYSPARK_DESCRIPTOR - __adapter__ = staticmethod(_adapter.build_driver_kwargs) -``` - -Add `src/mountainash_data/core/settings/adapters/__init__.py` (empty, marking package). - -- [ ] **Step 4: Run tests** - -Run: `hatch run test:test-target tests/test_unit/core/settings/backends/test_pyspark.py -v` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/core/settings/pyspark.py \ - src/mountainash_data/core/settings/adapters/ \ - tests/test_unit/core/settings/backends/test_pyspark.py -git commit -m "refactor(settings): migrate pyspark, fix PARTITIONS type and spark.* keys" -``` - ---- - -## Task 10: Migrate MotherDuck - -**Files:** -- Modify: `src/mountainash_data/core/settings/motherduck.py` -- Test: `tests/test_unit/core/settings/backends/test_motherduck.py` - -**Audit fixes carried:** source URLs added; "file-based authentication" comment removed; `DATABASE` nullability consistent; `ATTACH_PATH` removed (was orphan); class uses `TokenAuth`. - -- [ ] **Step 1: Write tests** — mirror the sqlite shape with a token auth case: - -```python -# tests/test_unit/core/settings/backends/test_motherduck.py -from __future__ import annotations - -import pytest -from pydantic import SecretStr - -from mountainash_data.core.settings.auth import TokenAuth -from mountainash_data.core.settings.motherduck import MotherDuckAuthSettings - - -@pytest.mark.unit -class TestMotherDuckAuthSettings: - def test_minimal(self): - s = MotherDuckAuthSettings( - DATABASE="mydb", auth=TokenAuth(token=SecretStr("t")) - ) - assert s.DATABASE == "mydb" - - def test_no_database_ok(self): - """Audit regression: previously validator rejected None, field was Optional.""" - s = MotherDuckAuthSettings(auth=TokenAuth(token=SecretStr("t"))) - assert s.DATABASE is None - - def test_to_driver_kwargs_unwraps_token(self): - s = MotherDuckAuthSettings( - DATABASE="mydb", auth=TokenAuth(token=SecretStr("tok")) - ) - kwargs = s.to_driver_kwargs() - assert kwargs["token"] == "tok" - assert isinstance(kwargs["token"], str) # not SecretStr -``` - -- [ ] **Step 2: Run → FAIL** - -- [ ] **Step 3: Rewrite `motherduck.py`** - -```python -# src/mountainash_data/core/settings/motherduck.py -"""MotherDuck backend settings. - -Spec: audit report ``docs/superpowers/specs/2026-04-15-settings-audit/motherduck.md``. -Driver auth docs: - https://motherduck.com/docs/getting-started/connect-query-from-python/installation-authentication/ -Ibis: routes via the duckdb backend (``rides_on="duckdb"``). -""" - -from __future__ import annotations - -import typing as t - -from ..constants import CONST_DB_PROVIDER_TYPE -from .auth import TokenAuth -from .descriptor import BackendDescriptor, ParameterSpec -from .profile import ConnectionProfile -from .registry import register - -__all__ = ["MotherDuckAuthSettings", "MOTHERDUCK_DESCRIPTOR"] - - -MOTHERDUCK_DESCRIPTOR = BackendDescriptor( - name="motherduck", - provider_type=CONST_DB_PROVIDER_TYPE.MOTHERDUCK, - connection_string_scheme="duckdb://md:", # md:?motherduck_token=... - ibis_dialect="duckdb", - rides_on="duckdb", - auth_modes=[TokenAuth], - parameters=[ - ParameterSpec(name="DATABASE", type=t.Optional[str], tier="core", - default=None), - ParameterSpec(name="READ_ONLY", type=bool, tier="core", default=False, - driver_key="read_only"), - ], -) - - -@register(MOTHERDUCK_DESCRIPTOR) -class MotherDuckAuthSettings(ConnectionProfile): - __descriptor__ = MOTHERDUCK_DESCRIPTOR -``` - -- [ ] **Step 4: Run → PASS** - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/core/settings/motherduck.py \ - tests/test_unit/core/settings/backends/test_motherduck.py -git commit -m "refactor(settings): migrate motherduck to TokenAuth + descriptor" -``` - ---- - -## Task 11: Migrate PostgreSQL - -**Files:** -- Modify: `src/mountainash_data/core/settings/postgresql.py` -- Test: `tests/test_unit/core/settings/backends/test_postgresql.py` - -**Audit fixes carried:** `db_provider_type` BIGQUERY bug → POSTGRESQL; four enums promoted to `StrEnum` field types; `SSL_CERT`/`SSL_KEY`/`SSL_ROOTCERT`/`SSL_CRL`/`SSL_CRLDIR` retyped `Optional[Path]`; `SSL_PASSWORD` → `Optional[SecretStr]`; `REQUIRE_AUTH` → `list[PostgresRequireAuthMethods]`; widened surface (`CONNECT_TIMEOUT`, `AUTOCOMMIT`, `HOSTADDR`, `SERVICE`, etc.). All libpq fields wired via descriptor `driver_key`. - -- [ ] **Step 1: Write tests** - -```python -# tests/test_unit/core/settings/backends/test_postgresql.py -from __future__ import annotations - -import pytest -from pydantic import SecretStr, ValidationError - -from mountainash_data.core.settings.auth import PasswordAuth -from mountainash_data.core.settings.postgresql import ( - PostgresRequireAuthMethods, - PostgresSSLMode, - PostgreSQLAuthSettings, -) - - -@pytest.mark.unit -class TestPostgreSQLAuthSettings: - def _minimal(self, **extra): - return PostgreSQLAuthSettings( - HOST="h", DATABASE="d", - auth=PasswordAuth(username="u", password=SecretStr("p")), - **extra, - ) - - def test_provider_type_is_postgresql(self): - """Audit regression: previously returned BIGQUERY.""" - s = self._minimal() - from mountainash_data.core.constants import CONST_DB_PROVIDER_TYPE - assert s.provider_type == CONST_DB_PROVIDER_TYPE.POSTGRESQL - - def test_ssl_mode_enum_enforced(self): - """Audit regression: SSL_MODE was bare str.""" - with pytest.raises(ValidationError): - self._minimal(SSL_MODE="nonsense") - - def test_ssl_cert_is_path(self): - """Audit regression: SSL_CERT was bool.""" - from pathlib import Path - s = self._minimal(SSL_CERT=Path("/etc/ssl/client.crt")) - assert s.SSL_CERT == Path("/etc/ssl/client.crt") - - def test_require_auth_is_list_of_enum(self): - """Audit regression: REQUIRE_AUTH was bool.""" - s = self._minimal( - REQUIRE_AUTH=[PostgresRequireAuthMethods.SCRAM_SHA_256, - PostgresRequireAuthMethods.MD5] - ) - assert len(s.REQUIRE_AUTH) == 2 - - def test_to_driver_kwargs_plumbs_ssl_and_keepalives(self): - """Audit regression: only SCHEMA was being plumbed.""" - s = self._minimal(SSL_MODE=PostgresSSLMode.REQUIRE, KEEPALIVES_IDLE=30) - kwargs = s.to_driver_kwargs() - assert kwargs["sslmode"] == "require" - assert kwargs["keepalives_idle"] == 30 - assert kwargs["user"] == "u" - assert kwargs["password"] == "p" # SecretStr unwrapped -``` - -- [ ] **Step 2: Run → FAIL** - -- [ ] **Step 3: Rewrite `postgresql.py`** - -```python -# src/mountainash_data/core/settings/postgresql.py -"""PostgreSQL backend settings. - -Spec: audit report ``docs/superpowers/specs/2026-04-15-settings-audit/postgresql.md``. -Driver: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS -Ibis: ``ibis.backends.postgres.do_connect(host, user, password, port=5432, - database, schema, autocommit=True, **kwargs)`` (psycopg). -""" - -from __future__ import annotations - -import typing as t -from enum import StrEnum -from pathlib import Path - -from pydantic import SecretStr - -from ..constants import CONST_DB_PROVIDER_TYPE -from .auth import NoAuth, PasswordAuth -from .descriptor import BackendDescriptor, ParameterSpec -from .profile import ConnectionProfile -from .registry import register - - -class PostgresSSLMode(StrEnum): - DISABLE = "disable" - ALLOW = "allow" - PREFER = "prefer" - REQUIRE = "require" - VERIFY_CA = "verify-ca" - VERIFY_FULL = "verify-full" - - -class PostgresTargetSessionAttrs(StrEnum): - ANY = "any" - READ_WRITE = "read-write" - READ_ONLY = "read-only" - PRIMARY = "primary" - STANDBY = "standby" - PREFER_STANDBY = "prefer-standby" - - -class PostgresRequireAuthMethods(StrEnum): - PASSWORD = "password" - MD5 = "md5" - GSS = "gss" - SSPI = "sspi" - SCRAM_SHA_256 = "scram-sha-256" - NONE = "none" - - -class PostgresSSLNegotiation(StrEnum): - POSTGRES = "postgres" - DIRECT = "direct" - - -class PostgresSSLCertMode(StrEnum): - DISABLE = "disable" - ALLOW = "allow" - REQUIRE = "require" - - -def _join_require_auth(v: list[PostgresRequireAuthMethods]) -> str: - return ",".join(m.value for m in v) - - -POSTGRESQL_DESCRIPTOR = BackendDescriptor( - name="postgresql", - provider_type=CONST_DB_PROVIDER_TYPE.POSTGRESQL, - default_port=5432, - connection_string_scheme="postgresql://", - ibis_dialect="postgres", - auth_modes=[PasswordAuth, NoAuth], - parameters=[ - ParameterSpec(name="HOST", type=str, tier="core", driver_key="host"), - ParameterSpec(name="HOSTADDR", type=t.Optional[str], tier="advanced", - default=None, driver_key="hostaddr"), - ParameterSpec(name="PORT", type=int, tier="core", default=5432, - driver_key="port"), - ParameterSpec(name="DATABASE", type=t.Optional[str], tier="core", - default=None, driver_key="database"), - ParameterSpec(name="SCHEMA", type=t.Optional[str], tier="core", - default=None, driver_key="schema"), - ParameterSpec(name="AUTOCOMMIT", type=bool, tier="core", default=True, - driver_key="autocommit"), - ParameterSpec(name="CONNECT_TIMEOUT", type=t.Optional[int], - tier="core", default=None, driver_key="connect_timeout"), - ParameterSpec(name="APPLICATION_NAME", type=t.Optional[str], - tier="advanced", default=None, - driver_key="application_name"), - ParameterSpec(name="PASSFILE", type=t.Optional[Path], tier="advanced", - default=None, driver_key="passfile", - transform=lambda p: str(p)), - ParameterSpec(name="SERVICE", type=t.Optional[str], tier="advanced", - default=None, driver_key="service"), - ParameterSpec(name="OPTIONS", type=t.Optional[str], tier="advanced", - default=None, driver_key="options"), - ParameterSpec(name="CHANNEL_BINDING", type=t.Optional[str], - tier="advanced", default=None, - driver_key="channel_binding"), - ParameterSpec(name="REQUIRE_AUTH", - type=t.Optional[list[PostgresRequireAuthMethods]], - tier="core", default=None, driver_key="require_auth", - transform=_join_require_auth), - # Keepalives - ParameterSpec(name="KEEPALIVES", type=bool, tier="advanced", default=True, - driver_key="keepalives", - transform=lambda v: 1 if v else 0), - ParameterSpec(name="KEEPALIVES_IDLE", type=t.Optional[int], - tier="advanced", default=None, - driver_key="keepalives_idle"), - ParameterSpec(name="KEEPALIVES_INTERVAL", type=t.Optional[int], - tier="advanced", default=None, - driver_key="keepalives_interval"), - ParameterSpec(name="KEEPALIVES_COUNT", type=t.Optional[int], - tier="advanced", default=None, - driver_key="keepalives_count"), - ParameterSpec(name="TCP_USER_TIMEOUT", type=t.Optional[int], - tier="advanced", default=None, - driver_key="tcp_user_timeout"), - # SSL / TLS - ParameterSpec(name="SSL_MODE", type=PostgresSSLMode, tier="core", - default=PostgresSSLMode.PREFER, driver_key="sslmode"), - ParameterSpec(name="SSL_NEGOTIATION", type=t.Optional[PostgresSSLNegotiation], - tier="advanced", default=None, driver_key="sslnegotiation"), - ParameterSpec(name="SSL_COMPRESSION", type=t.Optional[bool], - tier="advanced", default=None, driver_key="sslcompression", - transform=lambda v: 1 if v else 0), - ParameterSpec(name="SSL_CERT", type=t.Optional[Path], tier="advanced", - default=None, driver_key="sslcert", - transform=lambda p: str(p)), - ParameterSpec(name="SSL_KEY", type=t.Optional[Path], tier="advanced", - default=None, driver_key="sslkey", - transform=lambda p: str(p)), - ParameterSpec(name="SSL_PASSWORD", type=t.Optional[SecretStr], - tier="advanced", default=None, driver_key="sslpassword", - secret=True), - ParameterSpec(name="SSL_CERTMODE", type=t.Optional[PostgresSSLCertMode], - tier="advanced", default=None, driver_key="sslcertmode"), - ParameterSpec(name="SSL_ROOTCERT", type=t.Optional[Path], - tier="advanced", default=None, driver_key="sslrootcert", - transform=lambda p: str(p)), - ParameterSpec(name="SSL_CRL", type=t.Optional[Path], tier="advanced", - default=None, driver_key="sslcrl", - transform=lambda p: str(p)), - ParameterSpec(name="SSL_CRLDIR", type=t.Optional[Path], tier="advanced", - default=None, driver_key="sslcrldir", - transform=lambda p: str(p)), - ParameterSpec(name="SSL_SNI", type=t.Optional[bool], tier="advanced", - default=None, driver_key="sslsni", - transform=lambda v: 1 if v else 0), - ParameterSpec(name="TARGET_SESSION_ATTRS", - type=t.Optional[PostgresTargetSessionAttrs], - tier="advanced", default=None, - driver_key="target_session_attrs"), - ], -) - - -@register(POSTGRESQL_DESCRIPTOR) -class PostgreSQLAuthSettings(ConnectionProfile): - __descriptor__ = POSTGRESQL_DESCRIPTOR -``` - -- [ ] **Step 4: Run → PASS** - -Run: `hatch run test:test-target tests/test_unit/core/settings/backends/test_postgresql.py tests/test_unit/core/settings/test_descriptors_invariants.py -v` - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/core/settings/postgresql.py \ - tests/test_unit/core/settings/backends/test_postgresql.py -git commit -m "refactor(settings): migrate postgresql, fix provider_type and widen libpq surface" -``` - ---- - -## Task 12: Migrate MySQL (with `ssl={}` adapter) - -**Files:** -- Modify: `src/mountainash_data/core/settings/mysql.py` -- Create: `src/mountainash_data/core/settings/adapters/mysql.py` -- Test: `tests/test_unit/core/settings/backends/test_mysql.py` - -**Audit fixes carried:** `db_provider_type` BIGQUERY → MYSQL; `SSL_CAPATH` guard typo fixed (previously guarded by `SSL_CA`); `SSL_MODE != DISABLED` branch fires only when set; `SSL_MODE` as `StrEnum`; `CONV` → `Optional[dict[int, t.Any]]`; SSL fields assembled by adapter into `ssl={}` dict. - -- [ ] **Step 1: Write tests** - -```python -# tests/test_unit/core/settings/backends/test_mysql.py -from __future__ import annotations - -import pytest -from pydantic import SecretStr - -from mountainash_data.core.settings.auth import PasswordAuth -from mountainash_data.core.settings.mysql import MySQLAuthSettings, MySQLSSLMode - - -@pytest.mark.unit -class TestMySQLAuthSettings: - def _minimal(self, **extra): - return MySQLAuthSettings( - HOST="h", DATABASE="d", - auth=PasswordAuth(username="u", password=SecretStr("p")), - **extra, - ) - - def test_provider_type_is_mysql(self): - """Audit regression: previously returned BIGQUERY.""" - from mountainash_data.core.constants import CONST_DB_PROVIDER_TYPE - assert self._minimal().provider_type == CONST_DB_PROVIDER_TYPE.MYSQL - - def test_ssl_dict_assembled_when_capath_only_no_ca(self): - """Audit regression: SSL_CAPATH was gated on SSL_CA.""" - s = self._minimal(SSL_CAPATH="/etc/ssl/ca-dir") - kwargs = s.to_driver_kwargs() - assert kwargs["ssl"] == {"ssl-capath": "/etc/ssl/ca-dir"} - - def test_ssl_not_emitted_when_ssl_mode_none(self): - """Audit regression: SSL branch fired when SSL_MODE was None.""" - s = self._minimal() - kwargs = s.to_driver_kwargs() - assert "ssl_mode" not in kwargs - assert "ssl" not in kwargs - - def test_ssl_mode_preferred(self): - s = self._minimal(SSL_MODE=MySQLSSLMode.PREFERRED) - kwargs = s.to_driver_kwargs() - assert kwargs["ssl_mode"] == "PREFERRED" - - def test_autocommit_false_honored(self): - """Audit regression: `if self.AUTOCOMMIT:` dropped explicit False.""" - s = self._minimal(AUTOCOMMIT=False) - assert s.to_driver_kwargs()["autocommit"] is False -``` - -- [ ] **Step 2: Run → FAIL** - -- [ ] **Step 3: Implement adapter + settings** - -```python -# src/mountainash_data/core/settings/adapters/mysql.py -"""Adapter that assembles mysqlclient's ssl={} dict.""" - -from __future__ import annotations - -import typing as t - -if t.TYPE_CHECKING: - from mountainash_data.core.settings.mysql import MySQLAuthSettings - - -def build_driver_kwargs(profile: "MySQLAuthSettings") -> dict[str, t.Any]: - kwargs = profile._default_driver_kwargs() - kwargs.update(profile._auth_to_driver_kwargs()) - - if profile.SSL_MODE is not None: - kwargs["ssl_mode"] = profile.SSL_MODE.value - ssl: dict[str, str] = {} - for key, val in { - "ssl-key": profile.SSL_KEY, - "ssl-cert": profile.SSL_CERT, - "ssl-ca": profile.SSL_CA, - "ssl-capath": profile.SSL_CAPATH, - "ssl-cipher": profile.SSL_CIPHER, - }.items(): - if val is not None: - ssl[key] = str(val) - if ssl: - kwargs["ssl"] = ssl - return kwargs -``` - -```python -# src/mountainash_data/core/settings/mysql.py -"""MySQL backend settings. - -Spec: ``docs/superpowers/specs/2026-04-15-settings-audit/mysql.md``. -Driver: https://mysqlclient.readthedocs.io/user_guide.html#functions-and-attributes -Ibis: ``ibis.backends.mysql.do_connect(host='localhost', user=None, password=None, - port=3306, autocommit=True, **kwargs)`` -""" - -from __future__ import annotations - -import typing as t -from enum import StrEnum -from pathlib import Path - -from ..constants import CONST_DB_PROVIDER_TYPE -from .adapters import mysql as _adapter -from .auth import PasswordAuth -from .descriptor import BackendDescriptor, ParameterSpec -from .profile import ConnectionProfile -from .registry import register - - -class MySQLSSLMode(StrEnum): - DISABLED = "DISABLED" - PREFERRED = "PREFERRED" - REQUIRED = "REQUIRED" - VERIFY_CA = "VERIFY_CA" - VERIFY_IDENTITY = "VERIFY_IDENTITY" - - -MYSQL_DESCRIPTOR = BackendDescriptor( - name="mysql", - provider_type=CONST_DB_PROVIDER_TYPE.MYSQL, - default_port=3306, - connection_string_scheme="mysql://", - ibis_dialect="mysql", - auth_modes=[PasswordAuth], - parameters=[ - ParameterSpec(name="HOST", type=str, tier="core", driver_key="host"), - ParameterSpec(name="PORT", type=int, tier="core", default=3306, - driver_key="port"), - ParameterSpec(name="DATABASE", type=t.Optional[str], tier="core", - default=None, driver_key="database"), - ParameterSpec(name="CHARSET", type=str, tier="advanced", - default="utf8mb4", driver_key="charset"), - ParameterSpec(name="COLLATION", type=str, tier="advanced", - default="utf8mb4_unicode_ci", driver_key="collation"), - ParameterSpec(name="AUTOCOMMIT", type=bool, tier="core", - default=True, driver_key="autocommit"), - ParameterSpec(name="CONNECT_TIMEOUT", type=t.Optional[int], - tier="advanced", default=None, - driver_key="connect_timeout"), - ParameterSpec(name="READ_TIMEOUT", type=t.Optional[int], - tier="advanced", default=None, driver_key="read_timeout"), - ParameterSpec(name="WRITE_TIMEOUT", type=t.Optional[int], - tier="advanced", default=None, - driver_key="write_timeout"), - ParameterSpec(name="UNIX_SOCKET", type=t.Optional[Path], - tier="advanced", default=None, driver_key="unix_socket", - transform=lambda p: str(p)), - ParameterSpec(name="LOCAL_INFILE", type=t.Optional[bool], - tier="advanced", default=None, driver_key="local_infile"), - ParameterSpec(name="INIT_COMMAND", type=t.Optional[str], - tier="advanced", default=None, - driver_key="init_command"), - # SSL parameters — adapter handles assembly into ssl={} dict - ParameterSpec(name="SSL_MODE", type=t.Optional[MySQLSSLMode], - tier="core", default=None), - ParameterSpec(name="SSL_KEY", type=t.Optional[Path], tier="advanced", - default=None), - ParameterSpec(name="SSL_CERT", type=t.Optional[Path], tier="advanced", - default=None), - ParameterSpec(name="SSL_CA", type=t.Optional[Path], tier="advanced", - default=None), - ParameterSpec(name="SSL_CAPATH", type=t.Optional[Path], tier="advanced", - default=None), - ParameterSpec(name="SSL_CIPHER", type=t.Optional[str], tier="advanced", - default=None), - ParameterSpec(name="CONV", type=t.Optional[dict[int, t.Any]], - tier="advanced", default=None), - ], -) - - -@register(MYSQL_DESCRIPTOR) -class MySQLAuthSettings(ConnectionProfile): - __descriptor__ = MYSQL_DESCRIPTOR - __adapter__ = staticmethod(_adapter.build_driver_kwargs) -``` - -- [ ] **Step 4: Run → PASS** - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/core/settings/mysql.py \ - src/mountainash_data/core/settings/adapters/mysql.py \ - tests/test_unit/core/settings/backends/test_mysql.py -git commit -m "refactor(settings): migrate mysql with ssl adapter and audit fixes" -``` - ---- - -## Task 13: Migrate Trino (auth wrapper adapter) - -**Files:** -- Modify: `src/mountainash_data/core/settings/trino.py` -- Create: `src/mountainash_data/core/settings/adapters/trino.py` -- Test: `tests/test_unit/core/settings/backends/test_trino.py` - -**Audit fixes carried:** PASSWORD wiring broken → use `trino.auth.BasicAuthentication`; advanced fields retyped to native types; `VERIFY` widened to `bool | Path`; `ENCODING` field added; `PORT` default 8080. - -- [ ] **Step 1: Write tests** (adapter contract — verify right auth wrapper is used per auth kind) - -```python -# tests/test_unit/core/settings/backends/test_trino.py -from __future__ import annotations - -import pytest -from pydantic import SecretStr - -from mountainash_data.core.settings.auth import ( - JWTAuth, - KerberosAuth, - NoAuth, - PasswordAuth, -) -from mountainash_data.core.settings.trino import TrinoAuthSettings - - -@pytest.mark.unit -class TestTrinoAuthSettings: - def _minimal(self, auth, **extra): - return TrinoAuthSettings(HOST="h", CATALOG="c", auth=auth, **extra) - - def test_port_default_8080(self): - s = self._minimal(auth=NoAuth()) - assert s.PORT == 8080 - - def test_password_wraps_basic_auth(self): - """Audit regression: previously emitted bare `password=` kwarg. - - The driver has NO `password` kwarg — it must be wrapped. - """ - pytest.importorskip("trino") - from trino.auth import BasicAuthentication - - s = self._minimal( - auth=PasswordAuth(username="alice", password=SecretStr("pw")) - ) - kwargs = s.to_driver_kwargs() - assert kwargs["user"] == "alice" - assert isinstance(kwargs["auth"], BasicAuthentication) - assert "password" not in kwargs # must NOT be bare - - def test_jwt_auth_wraps(self): - pytest.importorskip("trino") - from trino.auth import JWTAuthentication - - s = self._minimal(auth=JWTAuth(token=SecretStr("tok"))) - kwargs = s.to_driver_kwargs() - assert isinstance(kwargs["auth"], JWTAuthentication) - - def test_noauth_no_auth_key(self): - s = self._minimal(auth=NoAuth()) - kwargs = s.to_driver_kwargs() - assert "auth" not in kwargs -``` - -- [ ] **Step 2: Run → FAIL** - -- [ ] **Step 3: Implement** - -```python -# src/mountainash_data/core/settings/adapters/trino.py -"""Adapter translating AuthSpec → trino.auth.Authentication wrappers.""" - -from __future__ import annotations - -import typing as t - -from mountainash_data.core.settings.auth import ( - AuthSpec, - JWTAuth, - KerberosAuth, - NoAuth, - PasswordAuth, -) - -if t.TYPE_CHECKING: - from mountainash_data.core.settings.trino import TrinoAuthSettings - - -def build_driver_kwargs(profile: "TrinoAuthSettings") -> dict[str, t.Any]: - kwargs = profile._default_driver_kwargs() - auth = profile.auth - if isinstance(auth, PasswordAuth): - from trino.auth import BasicAuthentication - - kwargs["user"] = auth.username - kwargs["auth"] = BasicAuthentication( - auth.username, auth.password.get_secret_value() - ) - elif isinstance(auth, JWTAuth): - from trino.auth import JWTAuthentication - - kwargs["auth"] = JWTAuthentication(auth.token.get_secret_value()) - elif isinstance(auth, KerberosAuth): - from trino.auth import KerberosAuthentication - - kwargs["auth"] = KerberosAuthentication( - config=None, - service_name=auth.service_name, - principal=auth.principal, - ) - elif isinstance(auth, NoAuth): - pass - else: - raise ValueError(f"trino adapter does not support auth: {type(auth).__name__}") - return kwargs -``` - -```python -# src/mountainash_data/core/settings/trino.py -"""Trino backend settings. - -Spec: ``docs/superpowers/specs/2026-04-15-settings-audit/trino.md``. -Driver: https://github.com/trinodb/trino-python-client/blob/master/trino/dbapi.py -Ibis: ``ibis.backends.trino.do_connect`` -""" - -from __future__ import annotations - -import typing as t -from pathlib import Path - -from ..constants import CONST_DB_PROVIDER_TYPE -from .adapters import trino as _adapter -from .auth import JWTAuth, KerberosAuth, NoAuth, PasswordAuth -from .descriptor import BackendDescriptor, ParameterSpec -from .profile import ConnectionProfile -from .registry import register - - -TRINO_DESCRIPTOR = BackendDescriptor( - name="trino", - provider_type=CONST_DB_PROVIDER_TYPE.TRINO, - default_port=8080, - connection_string_scheme="trino://", - ibis_dialect="trino", - auth_modes=[PasswordAuth, JWTAuth, KerberosAuth, NoAuth], - parameters=[ - ParameterSpec(name="HOST", type=str, tier="core", driver_key="host"), - ParameterSpec(name="PORT", type=int, tier="core", default=8080, - driver_key="port"), - ParameterSpec(name="CATALOG", type=str, tier="core", driver_key="catalog"), - ParameterSpec(name="SCHEMA", type=t.Optional[str], tier="core", - default=None, driver_key="schema"), - ParameterSpec(name="HTTP_SCHEME", type=str, tier="core", - default="https", driver_key="http_scheme"), - ParameterSpec(name="VERIFY", type=t.Optional[t.Union[bool, Path]], - tier="core", default=True, driver_key="verify", - transform=lambda v: str(v) if isinstance(v, Path) else v), - ParameterSpec(name="SOURCE", type=t.Optional[str], tier="advanced", - default=None, driver_key="source"), - ParameterSpec(name="TIMEZONE", type=t.Optional[str], tier="advanced", - default=None, driver_key="timezone"), - ParameterSpec(name="MAX_ATTEMPTS", type=t.Optional[int], - tier="advanced", default=None, - driver_key="max_attempts"), - ParameterSpec(name="REQUEST_TIMEOUT", type=t.Optional[float], - tier="advanced", default=None, - driver_key="request_timeout"), - ParameterSpec(name="SESSION_PROPERTIES", - type=t.Optional[dict[str, str]], tier="advanced", - default=None, driver_key="session_properties"), - ParameterSpec(name="HTTP_HEADERS", type=t.Optional[dict[str, str]], - tier="advanced", default=None, - driver_key="http_headers"), - ParameterSpec(name="EXTRA_CREDENTIAL", - type=t.Optional[list[tuple[str, str]]], tier="advanced", - default=None, driver_key="extra_credential"), - ParameterSpec(name="CLIENT_TAGS", type=t.Optional[list[str]], - tier="advanced", default=None, - driver_key="client_tags"), - ParameterSpec(name="ROLES", - type=t.Optional[t.Union[dict[str, str], str]], - tier="advanced", default=None, driver_key="roles"), - ParameterSpec(name="LEGACY_PRIMITIVE_TYPES", type=t.Optional[bool], - tier="advanced", default=None, - driver_key="legacy_primitive_types"), - ParameterSpec(name="LEGACY_PREPARED_STATEMENTS", - type=t.Optional[bool], tier="advanced", - default=None, - driver_key="legacy_prepared_statements"), - ParameterSpec(name="ENCODING", type=t.Optional[t.Union[str, list[str]]], - tier="advanced", default=None, driver_key="encoding"), - ], -) - - -@register(TRINO_DESCRIPTOR) -class TrinoAuthSettings(ConnectionProfile): - __descriptor__ = TRINO_DESCRIPTOR - __adapter__ = staticmethod(_adapter.build_driver_kwargs) -``` - -- [ ] **Step 4: Run → PASS** - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/core/settings/trino.py \ - src/mountainash_data/core/settings/adapters/trino.py \ - tests/test_unit/core/settings/backends/test_trino.py -git commit -m "refactor(settings): migrate trino, wire BasicAuthentication, fix type drift" -``` - ---- - -## Task 14: Migrate MSSQL (Windows + Azure AD + encryption adapter) - -**Files:** -- Modify: `src/mountainash_data/core/settings/mssql.py` -- Create: `src/mountainash_data/core/settings/adapters/mssql.py` -- Test: `tests/test_unit/core/settings/backends/test_mssql.py` - -**Audit fixes carried:** missing `AZURE_MANAGED_IDENTITY`/`MSI_ENDPOINT` → now part of `AzureADAuth`; `args["server"]` KeyError → adapter uses `host`; ENCRYPTION + TRUST_SERVER_CERTIFICATE added (core); MARS_ENABLED plumbed; orphan `PROTOCOL` dropped; `MSSQLAuthMethod` enum deleted (supplanted by auth union). - -- [ ] **Step 1: Write tests** - -```python -# tests/test_unit/core/settings/backends/test_mssql.py -from __future__ import annotations - -import pytest -from pydantic import SecretStr - -from mountainash_data.core.settings.auth import ( - AzureADAuth, - PasswordAuth, - WindowsAuth, -) -from mountainash_data.core.settings.mssql import ( - MSSQLAuthSettings, - MSSQLEncryption, -) - - -@pytest.mark.unit -class TestMSSQLAuthSettings: - def _minimal(self, auth, **extra): - return MSSQLAuthSettings(HOST="h", DATABASE="d", auth=auth, **extra) - - def test_password_auth(self): - s = self._minimal( - auth=PasswordAuth(username="u", password=SecretStr("p")) - ) - kwargs = s.to_driver_kwargs() - assert kwargs["user"] == "u" - assert kwargs["password"] == "p" - assert kwargs["host"] == "h" - - def test_windows_auth_sets_trusted_connection(self): - s = self._minimal(auth=WindowsAuth(username="u", domain="CORP")) - kwargs = s.to_driver_kwargs() - assert kwargs["trusted_connection"] == "yes" - assert kwargs["user"] == r"CORP\u" - - def test_azure_ad_managed_identity(self): - """Audit regression: AZURE_MANAGED_IDENTITY/MSI_ENDPOINT were - referenced but not declared — now live on AzureADAuth.""" - s = self._minimal( - auth=AzureADAuth( - managed_identity=True, - msi_endpoint="http://169.254.169.254/", - ) - ) - kwargs = s.to_driver_kwargs() - assert kwargs["authentication"] == "ActiveDirectoryMsi" - assert kwargs["msi_endpoint"] == "http://169.254.169.254/" - - def test_instance_name_appended_to_host(self): - """Audit regression: code referenced args['server'] (KeyError).""" - s = self._minimal( - auth=PasswordAuth(username="u", password=SecretStr("p")), - INSTANCE_NAME="SQLEXPRESS", - ) - kwargs = s.to_driver_kwargs() - assert kwargs["host"] == r"h\SQLEXPRESS" - - def test_encryption_default(self): - """Audit regression: ODBC Driver 18 default Encrypt=Yes requires explicit setting.""" - s = self._minimal(auth=PasswordAuth(username="u", password=SecretStr("p"))) - assert s.ENCRYPTION is MSSQLEncryption.MANDATORY -``` - -- [ ] **Step 2: Run → FAIL** - -- [ ] **Step 3: Implement** - -```python -# src/mountainash_data/core/settings/adapters/mssql.py -"""MSSQL adapter: auth dispatch, instance-name folding, encryption keys.""" - -from __future__ import annotations - -import typing as t - -from mountainash_data.core.settings.auth import ( - AzureADAuth, - PasswordAuth, - WindowsAuth, -) - -if t.TYPE_CHECKING: - from mountainash_data.core.settings.mssql import MSSQLAuthSettings - - -def build_driver_kwargs(profile: "MSSQLAuthSettings") -> dict[str, t.Any]: - kwargs = profile._default_driver_kwargs() - - # Instance name → host\instance - if profile.INSTANCE_NAME: - kwargs["host"] = f"{kwargs['host']}\\{profile.INSTANCE_NAME}" - - # Encryption flags - if profile.ENCRYPTION is not None: - kwargs["encrypt"] = profile.ENCRYPTION.value - if profile.TRUST_SERVER_CERTIFICATE: - kwargs["trust_server_certificate"] = "yes" - if profile.MARS_ENABLED: - kwargs["mars_connection"] = "yes" - - # Auth dispatch - auth = profile.auth - if isinstance(auth, PasswordAuth): - kwargs["user"] = auth.username - kwargs["password"] = auth.password.get_secret_value() - elif isinstance(auth, WindowsAuth): - kwargs["trusted_connection"] = "yes" - if auth.domain and auth.username: - kwargs["user"] = f"{auth.domain}\\{auth.username}" - elif auth.username: - kwargs["user"] = auth.username - elif isinstance(auth, AzureADAuth): - if auth.managed_identity: - kwargs["authentication"] = "ActiveDirectoryMsi" - if auth.msi_endpoint: - kwargs["msi_endpoint"] = auth.msi_endpoint - else: - kwargs["authentication"] = "ActiveDirectoryServicePrincipal" - if auth.client_id: - kwargs["user_id"] = auth.client_id - if auth.client_secret: - kwargs["password"] = auth.client_secret.get_secret_value() - if auth.tenant_id: - kwargs["tenant_id"] = auth.tenant_id - return kwargs -``` - -```python -# src/mountainash_data/core/settings/mssql.py -"""MSSQL backend settings. - -Spec: ``docs/superpowers/specs/2026-04-15-settings-audit/mssql.md``. -Driver: PyODBC connect + ODBC Driver 17/18 for SQL Server. -""" - -from __future__ import annotations - -import typing as t -from enum import StrEnum - -from ..constants import CONST_DB_PROVIDER_TYPE -from .adapters import mssql as _adapter -from .auth import AzureADAuth, PasswordAuth, WindowsAuth -from .descriptor import BackendDescriptor, ParameterSpec -from .profile import ConnectionProfile -from .registry import register - - -class MSSQLDriver(StrEnum): - ODBC_18 = "ODBC Driver 18 for SQL Server" - ODBC_17 = "ODBC Driver 17 for SQL Server" - LEGACY = "SQL Server" - - -class MSSQLEncryption(StrEnum): - DISABLED = "no" - MANDATORY = "yes" - STRICT = "strict" - - -MSSQL_DESCRIPTOR = BackendDescriptor( - name="mssql", - provider_type=CONST_DB_PROVIDER_TYPE.MSSQL, - default_port=1433, - connection_string_scheme="mssql://", - ibis_dialect="mssql", - auth_modes=[PasswordAuth, WindowsAuth, AzureADAuth], - parameters=[ - ParameterSpec(name="HOST", type=str, tier="core", driver_key="host"), - ParameterSpec(name="PORT", type=int, tier="core", default=1433, - driver_key="port"), - ParameterSpec(name="DATABASE", type=t.Optional[str], tier="core", - default=None, driver_key="database"), - ParameterSpec(name="DRIVER", type=MSSQLDriver, tier="core", - default=MSSQLDriver.ODBC_18, driver_key="driver"), - ParameterSpec(name="ENCRYPTION", type=t.Optional[MSSQLEncryption], - tier="core", default=MSSQLEncryption.MANDATORY), - ParameterSpec(name="TRUST_SERVER_CERTIFICATE", type=bool, tier="core", - default=False), - ParameterSpec(name="INSTANCE_NAME", type=t.Optional[str], - tier="advanced", default=None), - ParameterSpec(name="APP_NAME", type=str, tier="advanced", - default="MountainAsh", driver_key="application_name"), - ParameterSpec(name="MARS_ENABLED", type=bool, tier="advanced", - default=False), - ParameterSpec(name="LOGIN_TIMEOUT", type=t.Optional[int], - tier="advanced", default=None, driver_key="login_timeout"), - ], -) - - -@register(MSSQL_DESCRIPTOR) -class MSSQLAuthSettings(ConnectionProfile): - __descriptor__ = MSSQL_DESCRIPTOR - __adapter__ = staticmethod(_adapter.build_driver_kwargs) -``` - -- [ ] **Step 4: Run → PASS** - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/core/settings/mssql.py \ - src/mountainash_data/core/settings/adapters/mssql.py \ - tests/test_unit/core/settings/backends/test_mssql.py -git commit -m "refactor(settings): migrate mssql with auth dispatch and encryption fixes" -``` - ---- - -## Task 15: Migrate Snowflake (session_parameters + authenticator + cert) - -**Files:** -- Modify: `src/mountainash_data/core/settings/snowflake.py` -- Create: `src/mountainash_data/core/settings/adapters/snowflake.py` -- Test: `tests/test_unit/core/settings/backends/test_snowflake.py` - -**Audit fixes carried:** enum whitespace bug fixed; `AUTHENTICATOR` as `StrEnum`; ROLE plumbed; `OKTA_ACCOUNT_NAMER` typo → `OKTA_ACCOUNT_NAME`; SecretStr unwrapped; `HOST` dropped (Snowflake uses `account`); `TIMEZONE` routed through `session_parameters`; `CertificateAuth` used for key auth. - -- [ ] **Step 1: Write tests** - -```python -# tests/test_unit/core/settings/backends/test_snowflake.py -from __future__ import annotations - -import pytest -from pydantic import SecretStr - -from mountainash_data.core.settings.auth import ( - CertificateAuth, - OAuth2Auth, - PasswordAuth, - TokenAuth, -) -from mountainash_data.core.settings.snowflake import ( - SnowflakeAuthenticator, - SnowflakeAuthSettings, -) - - -@pytest.mark.unit -class TestSnowflakeAuthSettings: - def _minimal(self, auth, **extra): - return SnowflakeAuthSettings( - ACCOUNT="acc", WAREHOUSE="wh", auth=auth, **extra, - ) - - def test_authenticator_enum_has_no_whitespace(self): - """Audit regression: enum values had trailing spaces.""" - assert SnowflakeAuthenticator.SNOWFLAKE.value == "snowflake" - assert SnowflakeAuthenticator.PASSWORD_MFA.value == "username_password_mfa" - - def test_password_auth(self): - s = self._minimal( - auth=PasswordAuth(username="u", password=SecretStr("p")) - ) - kwargs = s.to_driver_kwargs() - assert kwargs["account"] == "acc" - assert kwargs["warehouse"] == "wh" - assert kwargs["user"] == "u" - assert kwargs["password"] == "p" - - def test_role_is_plumbed(self): - """Audit regression: ROLE was declared but never emitted.""" - s = self._minimal( - auth=PasswordAuth(username="u", password=SecretStr("p")), - ROLE="analyst", - ) - assert s.to_driver_kwargs()["role"] == "analyst" - - def test_timezone_goes_to_session_parameters(self): - """Audit regression: TIMEZONE was top-level, should be in session_parameters.""" - s = self._minimal( - auth=PasswordAuth(username="u", password=SecretStr("p")), - TIMEZONE="UTC", - ) - kwargs = s.to_driver_kwargs() - assert kwargs["session_parameters"] == {"TIMEZONE": "UTC"} - - def test_certificate_auth(self): - s = self._minimal( - auth=CertificateAuth(private_key=SecretStr("KEYCONTENT")) - ) - kwargs = s.to_driver_kwargs() - assert kwargs["private_key"] == "KEYCONTENT" -``` - -- [ ] **Step 2: Run → FAIL** - -- [ ] **Step 3: Implement** - -```python -# src/mountainash_data/core/settings/adapters/snowflake.py -"""Snowflake adapter: session_parameters, authenticator mapping, cert auth.""" - -from __future__ import annotations - -import typing as t - -from mountainash_data.core.settings.auth import ( - CertificateAuth, - OAuth2Auth, - PasswordAuth, - TokenAuth, -) - -if t.TYPE_CHECKING: - from mountainash_data.core.settings.snowflake import SnowflakeAuthSettings - - -def build_driver_kwargs(profile: "SnowflakeAuthSettings") -> dict[str, t.Any]: - kwargs = profile._default_driver_kwargs() - - # Session parameters - session_params: dict[str, t.Any] = {} - if profile.TIMEZONE is not None: - session_params["TIMEZONE"] = profile.TIMEZONE - if profile.QUERY_TAG is not None: - session_params["QUERY_TAG"] = profile.QUERY_TAG - if session_params: - kwargs["session_parameters"] = session_params - - # Auth dispatch - auth = profile.auth - if isinstance(auth, PasswordAuth): - kwargs["user"] = auth.username - kwargs["password"] = auth.password.get_secret_value() - if profile.AUTHENTICATOR is not None: - kwargs["authenticator"] = profile.AUTHENTICATOR.value - elif isinstance(auth, TokenAuth): - kwargs["authenticator"] = "oauth" - kwargs["token"] = auth.token.get_secret_value() - elif isinstance(auth, OAuth2Auth): - kwargs["authenticator"] = "oauth" - if auth.token is not None: - kwargs["token"] = auth.token.get_secret_value() - elif isinstance(auth, CertificateAuth): - if auth.private_key is not None: - kwargs["private_key"] = auth.private_key.get_secret_value() - if auth.private_key_path is not None: - kwargs["private_key_file"] = str(auth.private_key_path) - if auth.passphrase is not None: - kwargs["private_key_file_pwd"] = auth.passphrase.get_secret_value() - return kwargs -``` - -```python -# src/mountainash_data/core/settings/snowflake.py -"""Snowflake backend settings. - -Spec: ``docs/superpowers/specs/2026-04-15-settings-audit/snowflake.md``. -Driver: https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-api -""" - -from __future__ import annotations - -import typing as t -from enum import StrEnum - -from ..constants import CONST_DB_PROVIDER_TYPE -from .adapters import snowflake as _adapter -from .auth import CertificateAuth, OAuth2Auth, PasswordAuth, TokenAuth -from .descriptor import BackendDescriptor, ParameterSpec -from .profile import ConnectionProfile -from .registry import register - - -class SnowflakeAuthenticator(StrEnum): - SNOWFLAKE = "snowflake" - OAUTH = "oauth" - OKTA = "okta" - EXTERNAL_BROWSER = "externalbrowser" - PASSWORD_MFA = "username_password_mfa" - - -SNOWFLAKE_DESCRIPTOR = BackendDescriptor( - name="snowflake", - provider_type=CONST_DB_PROVIDER_TYPE.SNOWFLAKE, - connection_string_scheme="snowflake://", - ibis_dialect="snowflake", - auth_modes=[PasswordAuth, OAuth2Auth, CertificateAuth, TokenAuth], - parameters=[ - ParameterSpec(name="ACCOUNT", type=str, tier="core", - driver_key="account"), - ParameterSpec(name="WAREHOUSE", type=t.Optional[str], tier="core", - default=None, driver_key="warehouse"), - ParameterSpec(name="DATABASE", type=t.Optional[str], tier="core", - default=None, driver_key="database"), - ParameterSpec(name="SCHEMA", type=t.Optional[str], tier="core", - default=None, driver_key="schema"), - ParameterSpec(name="ROLE", type=t.Optional[str], tier="core", - default=None, driver_key="role"), - ParameterSpec(name="AUTHENTICATOR", - type=t.Optional[SnowflakeAuthenticator], tier="core", - default=None), - ParameterSpec(name="CONNECTION_NAME", type=t.Optional[str], - tier="core", default=None, driver_key="connection_name"), - ParameterSpec(name="TIMEZONE", type=t.Optional[str], tier="advanced", - default=None), - ParameterSpec(name="QUERY_TAG", type=t.Optional[str], tier="advanced", - default=None), - ParameterSpec(name="APPLICATION", type=t.Optional[str], - tier="advanced", default=None, - driver_key="application"), - ParameterSpec(name="LOGIN_TIMEOUT", type=t.Optional[int], - tier="advanced", default=None, - driver_key="login_timeout"), - ParameterSpec(name="NETWORK_TIMEOUT", type=t.Optional[int], - tier="advanced", default=None, - driver_key="network_timeout"), - ParameterSpec(name="OKTA_ACCOUNT_NAME", type=t.Optional[str], - tier="advanced", default=None, - driver_key="okta_account_name"), - ], -) - - -@register(SNOWFLAKE_DESCRIPTOR) -class SnowflakeAuthSettings(ConnectionProfile): - __descriptor__ = SNOWFLAKE_DESCRIPTOR - __adapter__ = staticmethod(_adapter.build_driver_kwargs) -``` - -- [ ] **Step 4: Run → PASS** - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/core/settings/snowflake.py \ - src/mountainash_data/core/settings/adapters/snowflake.py \ - tests/test_unit/core/settings/backends/test_snowflake.py -git commit -m "refactor(settings): migrate snowflake, fix enum whitespace and session_parameters" -``` - ---- - -## Task 16: Migrate BigQuery (SA-info → Credentials adapter) - -**Files:** -- Modify: `src/mountainash_data/core/settings/bigquery.py` -- Create: `src/mountainash_data/core/settings/adapters/bigquery.py` -- Test: `tests/test_unit/core/settings/backends/test_bigquery.py` - -**Audit fixes carried:** `credentials` typed correctly — adapter converts SA-info dict (or file path) → `google.oauth2.service_account.Credentials`; `AUTH_LOCAL_WEBSERVER`/`AUTH_EXTERNAL_DATA`/`AUTH_CACHE` added; `PARTITION_COLUMN` default `"PARTITIONTIME"`; `ServiceAccountAuth` is primary auth. - -- [ ] **Step 1: Write tests** - -```python -# tests/test_unit/core/settings/backends/test_bigquery.py -from __future__ import annotations - -import pytest - -from mountainash_data.core.settings.auth import NoAuth, ServiceAccountAuth -from mountainash_data.core.settings.bigquery import BigQueryAuthSettings - - -@pytest.mark.unit -class TestBigQueryAuthSettings: - def test_partition_column_default(self): - """Audit regression: default was None, should be 'PARTITIONTIME'.""" - s = BigQueryAuthSettings(PROJECT_ID="myproj12", auth=NoAuth()) - assert s.PARTITION_COLUMN == "PARTITIONTIME" - - def test_service_account_info_converts_to_credentials(self): - """Audit regression: SA info dict was passed raw; Ibis needs Credentials.""" - pytest.importorskip("google.oauth2") - - # Minimal valid SA info shape - info = { - "type": "service_account", - "project_id": "myproj12", - "private_key_id": "x", - "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n", - "client_email": "sa@myproj12.iam.gserviceaccount.com", - "client_id": "1", - "auth_uri": "https://accounts.google.com/o/oauth2/auth", - "token_uri": "https://oauth2.googleapis.com/token", - } - - s = BigQueryAuthSettings( - PROJECT_ID="myproj12", - auth=ServiceAccountAuth(info=info), - ) - # We can't fully construct credentials without a valid key, so we - # just verify the adapter attempts conversion and emits the key. - try: - kwargs = s.to_driver_kwargs() - assert "credentials" in kwargs - except ValueError: - # google.oauth2 will reject the dummy key — acceptable here, - # the important thing is no raw dict leak. - pass - - def test_auth_local_webserver_plumbed(self): - """Audit regression: field didn't exist.""" - s = BigQueryAuthSettings( - PROJECT_ID="myproj12", AUTH_LOCAL_WEBSERVER=False, auth=NoAuth(), - ) - assert s.to_driver_kwargs()["auth_local_webserver"] is False -``` - -- [ ] **Step 2: Run → FAIL** - -- [ ] **Step 3: Implement** - -```python -# src/mountainash_data/core/settings/adapters/bigquery.py -"""BigQuery adapter: convert ServiceAccountAuth → google Credentials.""" - -from __future__ import annotations - -import typing as t - -from mountainash_data.core.settings.auth import NoAuth, ServiceAccountAuth - -if t.TYPE_CHECKING: - from mountainash_data.core.settings.bigquery import BigQueryAuthSettings - - -def build_driver_kwargs(profile: "BigQueryAuthSettings") -> dict[str, t.Any]: - kwargs = profile._default_driver_kwargs() - - auth = profile.auth - if isinstance(auth, ServiceAccountAuth): - from google.oauth2 import service_account as _sa - - if auth.info is not None: - kwargs["credentials"] = _sa.Credentials.from_service_account_info(auth.info) - elif auth.file is not None: - kwargs["credentials"] = _sa.Credentials.from_service_account_file( - str(auth.file) - ) - elif isinstance(auth, NoAuth): - pass # Application Default Credentials - return kwargs -``` - -```python -# src/mountainash_data/core/settings/bigquery.py -"""BigQuery backend settings. - -Spec: ``docs/superpowers/specs/2026-04-15-settings-audit/bigquery.md``. -Ibis: ``ibis.backends.bigquery.do_connect`` -""" - -from __future__ import annotations - -import re -import typing as t - -from pydantic import field_validator - -from ..constants import CONST_DB_PROVIDER_TYPE -from .adapters import bigquery as _adapter -from .auth import NoAuth, ServiceAccountAuth -from .descriptor import BackendDescriptor, ParameterSpec -from .profile import ConnectionProfile -from .registry import register - -_PROJECT_ID_RE = re.compile(r"^[a-z][a-z0-9-]{4,28}[a-z0-9]$") - - -def _validate_project_id(value: str) -> str: - if not _PROJECT_ID_RE.match(value): - raise ValueError( - "PROJECT_ID must be 6-30 chars, lowercase/digits/hyphens, " - "not starting or ending with a hyphen" - ) - return value - - -BIGQUERY_DESCRIPTOR = BackendDescriptor( - name="bigquery", - provider_type=CONST_DB_PROVIDER_TYPE.BIGQUERY, - connection_string_scheme="bigquery://", - ibis_dialect="bigquery", - auth_modes=[ServiceAccountAuth, NoAuth], - parameters=[ - ParameterSpec(name="PROJECT_ID", type=str, tier="core", - driver_key="project_id"), - ParameterSpec(name="DATASET_ID", type=t.Optional[str], tier="core", - default=None, driver_key="dataset_id"), - ParameterSpec(name="LOCATION", type=t.Optional[str], tier="advanced", - default=None, driver_key="location"), - ParameterSpec(name="APPLICATION_NAME", type=t.Optional[str], - tier="advanced", default=None, - driver_key="application_name"), - ParameterSpec(name="PARTITION_COLUMN", type=str, tier="advanced", - default="PARTITIONTIME", driver_key="partition_column"), - ParameterSpec(name="AUTH_LOCAL_WEBSERVER", type=bool, tier="core", - default=True, driver_key="auth_local_webserver"), - ParameterSpec(name="AUTH_EXTERNAL_DATA", type=bool, tier="core", - default=False, driver_key="auth_external_data"), - ParameterSpec(name="AUTH_CACHE", type=str, tier="core", - default="default", driver_key="auth_cache"), - ], -) - - -@register(BIGQUERY_DESCRIPTOR) -class BigQueryAuthSettings(ConnectionProfile): - __descriptor__ = BIGQUERY_DESCRIPTOR - __adapter__ = staticmethod(_adapter.build_driver_kwargs) - - @field_validator("PROJECT_ID") - @classmethod - def _pid(cls, v: str) -> str: - return _validate_project_id(v) -``` - -- [ ] **Step 4: Run → PASS** - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/core/settings/bigquery.py \ - src/mountainash_data/core/settings/adapters/bigquery.py \ - tests/test_unit/core/settings/backends/test_bigquery.py -git commit -m "refactor(settings): migrate bigquery with SA credentials conversion" -``` - ---- - -## Task 17: Migrate Redshift (endpoint resolver adapter) - -**Files:** -- Modify: `src/mountainash_data/core/settings/redshift.py` -- Create: `src/mountainash_data/core/settings/adapters/redshift.py` -- Test: `tests/test_unit/core/settings/backends/test_redshift.py` - -**Audit fixes carried:** `_init_provider_specific` → `_post_init`/validators run; broken `_get_cluster_endpoint` either implemented (behind a lazy `boto3` import) or removed (with explicit `HOST` required); region regex widened; role ARN regex widened; `SSL` bool → `SSL_MODE` enum; `CLUSTER_READ_ONLY` / `WORKGROUP_NAME` plumbed when used. - -- [ ] **Step 1: Write tests** - -```python -# tests/test_unit/core/settings/backends/test_redshift.py -from __future__ import annotations - -import pytest -from pydantic import SecretStr, ValidationError - -from mountainash_data.core.settings.auth import IAMAuth, PasswordAuth -from mountainash_data.core.settings.redshift import ( - RedshiftAuthSettings, - RedshiftSSLMode, -) - - -@pytest.mark.unit -class TestRedshiftAuthSettings: - def _password(self, **extra): - return RedshiftAuthSettings( - HOST="cluster.abc.us-east-1.redshift.amazonaws.com", - DATABASE="dev", - REGION="us-east-1", - auth=PasswordAuth(username="u", password=SecretStr("p")), - **extra, - ) - - def test_port_default_5439(self): - s = self._password() - assert s.PORT == 5439 - - def test_region_govcloud_accepted(self): - """Audit regression: region regex rejected GovCloud.""" - s = RedshiftAuthSettings( - HOST="h", DATABASE="d", REGION="us-gov-west-1", - auth=PasswordAuth(username="u", password=SecretStr("p")), - ) - assert s.REGION == "us-gov-west-1" - - def test_role_arn_govcloud_accepted(self): - """Audit regression: role-ARN regex rejected non-commercial partitions.""" - s = self._password(IAM_ROLE_ARN="arn:aws-us-gov:iam::123456789012:role/x") - assert s.IAM_ROLE_ARN.startswith("arn:aws-us-gov:") - - def test_iam_auth(self): - s = RedshiftAuthSettings( - HOST="h", DATABASE="d", REGION="us-east-1", - auth=IAMAuth( - access_key_id="AKIA", secret_access_key=SecretStr("sk"), - ), - ) - kwargs = s.to_driver_kwargs() - assert kwargs["aws_access_key_id"] == "AKIA" - assert kwargs["aws_secret_access_key"] == "sk" - - def test_ssl_mode_enum(self): - """Audit regression: SSL was bool, hardcoded verify-full.""" - s = self._password(SSL_MODE=RedshiftSSLMode.REQUIRE) - assert s.to_driver_kwargs()["sslmode"] == "require" -``` - -- [ ] **Step 2: Run → FAIL** - -- [ ] **Step 3: Implement** - -```python -# src/mountainash_data/core/settings/adapters/redshift.py -"""Redshift adapter: endpoint resolution hook, IAM/password routing.""" - -from __future__ import annotations - -import typing as t - -from mountainash_data.core.settings.auth import IAMAuth, PasswordAuth - -if t.TYPE_CHECKING: - from mountainash_data.core.settings.redshift import RedshiftAuthSettings - - -def build_driver_kwargs(profile: "RedshiftAuthSettings") -> dict[str, t.Any]: - kwargs = profile._default_driver_kwargs() - - auth = profile.auth - if isinstance(auth, PasswordAuth): - kwargs["user"] = auth.username - kwargs["password"] = auth.password.get_secret_value() - elif isinstance(auth, IAMAuth): - kwargs["iam"] = True - if auth.role_arn is not None: - kwargs["iam_role_arn"] = auth.role_arn - if auth.access_key_id is not None: - kwargs["aws_access_key_id"] = auth.access_key_id - if auth.secret_access_key is not None: - kwargs["aws_secret_access_key"] = auth.secret_access_key.get_secret_value() - if auth.session_token is not None: - kwargs["aws_session_token"] = auth.session_token.get_secret_value() - if auth.profile_name is not None: - kwargs["profile_name"] = auth.profile_name - return kwargs -``` - -```python -# src/mountainash_data/core/settings/redshift.py -"""Redshift backend settings. - -Spec: ``docs/superpowers/specs/2026-04-15-settings-audit/redshift.md``. -Driver: redshift_connector OR psycopg (via Ibis postgres). Endpoint -resolution via boto3 ``describe_clusters`` is a Phase-4 follow-up. -""" - -from __future__ import annotations - -import re -import typing as t -from enum import StrEnum - -from pydantic import field_validator - -from ..constants import CONST_DB_PROVIDER_TYPE -from .adapters import redshift as _adapter -from .auth import IAMAuth, PasswordAuth -from .descriptor import BackendDescriptor, ParameterSpec -from .profile import ConnectionProfile -from .registry import register - - -class RedshiftSSLMode(StrEnum): - DISABLE = "disable" - ALLOW = "allow" - PREFER = "prefer" - REQUIRE = "require" - VERIFY_CA = "verify-ca" - VERIFY_FULL = "verify-full" - - -_REGION_RE = re.compile(r"^[a-z]{2,4}-[a-z-]+-\d{1,2}$") -_ROLE_ARN_RE = re.compile(r"^arn:aws(?:-us-gov|-cn)?:iam::\d{12}:role/.+$") - - -REDSHIFT_DESCRIPTOR = BackendDescriptor( - name="redshift", - provider_type=CONST_DB_PROVIDER_TYPE.REDSHIFT, - default_port=5439, - connection_string_scheme="redshift://", - ibis_dialect="postgres", - rides_on="postgres", - auth_modes=[PasswordAuth, IAMAuth], - parameters=[ - ParameterSpec(name="HOST", type=str, tier="core", driver_key="host"), - ParameterSpec(name="PORT", type=int, tier="core", default=5439, - driver_key="port"), - ParameterSpec(name="DATABASE", type=str, tier="core", - driver_key="database"), - ParameterSpec(name="SCHEMA", type=t.Optional[str], tier="core", - default=None, driver_key="schema"), - ParameterSpec(name="REGION", type=str, tier="core"), - ParameterSpec(name="CLUSTER_IDENTIFIER", type=t.Optional[str], - tier="core", default=None), - ParameterSpec(name="WORKGROUP_NAME", type=t.Optional[str], - tier="core", default=None), - ParameterSpec(name="SERVERLESS", type=bool, tier="core", default=False), - ParameterSpec(name="IAM_ROLE_ARN", type=t.Optional[str], - tier="advanced", default=None), - ParameterSpec(name="SSL_MODE", type=RedshiftSSLMode, tier="core", - default=RedshiftSSLMode.VERIFY_FULL, - driver_key="sslmode"), - ParameterSpec(name="CLUSTER_READ_ONLY", type=bool, tier="advanced", - default=False, driver_key="readonly"), - ], -) - - -@register(REDSHIFT_DESCRIPTOR) -class RedshiftAuthSettings(ConnectionProfile): - __descriptor__ = REDSHIFT_DESCRIPTOR - __adapter__ = staticmethod(_adapter.build_driver_kwargs) - - @field_validator("REGION") - @classmethod - def _region(cls, v: str) -> str: - if not _REGION_RE.match(v): - raise ValueError(f"Invalid AWS region: {v}") - return v - - @field_validator("IAM_ROLE_ARN") - @classmethod - def _role_arn(cls, v: t.Optional[str]) -> t.Optional[str]: - if v is not None and not _ROLE_ARN_RE.match(v): - raise ValueError(f"Invalid IAM role ARN: {v}") - return v -``` - -- [ ] **Step 4: Run → PASS** - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/core/settings/redshift.py \ - src/mountainash_data/core/settings/adapters/redshift.py \ - tests/test_unit/core/settings/backends/test_redshift.py -git commit -m "refactor(settings): migrate redshift with SSL_MODE enum and widened region regex" -``` - ---- - -## Task 18: Migrate PyIceberg REST (catalog adapter) - -**Files:** -- Modify: `src/mountainash_data/core/settings/pyiceberg_rest.py` -- Create: `src/mountainash_data/core/settings/adapters/pyiceberg_rest.py` -- Test: `tests/test_unit/core/settings/backends/test_pyiceberg_rest.py` - -**Audit fixes carried:** class identity resolved (generic REST, not R2-specific); `WAREHOUSE` Optional; `USE_SSL` dropped / replaced with scheme inference; `VERIFY_SSL` plumbed; OAuth2 via `OAuth2Auth`; `s3.*`, `rest.sigv4-*`, `header.*` families added as dotted-key params handled by adapter. - -- [ ] **Step 1: Write tests** - -```python -# tests/test_unit/core/settings/backends/test_pyiceberg_rest.py -from __future__ import annotations - -import pytest -from pydantic import SecretStr - -from mountainash_data.core.settings.auth import OAuth2Auth, TokenAuth -from mountainash_data.core.settings.pyiceberg_rest import PyIcebergRestAuthSettings - - -@pytest.mark.unit -class TestPyIcebergRestAuthSettings: - def _min(self, auth, **extra): - return PyIcebergRestAuthSettings( - CATALOG_NAME="cat", - CATALOG_URI="https://catalog.example/v1", - auth=auth, **extra, - ) - - def test_warehouse_optional(self): - """Audit regression: WAREHOUSE was over-required.""" - s = self._min(auth=TokenAuth(token=SecretStr("t"))) - assert s.WAREHOUSE is None - - def test_token_auth(self): - s = self._min(auth=TokenAuth(token=SecretStr("tok"))) - kwargs = s.to_driver_kwargs() - assert kwargs["token"] == "tok" - assert kwargs["uri"] == "https://catalog.example/v1" - - def test_oauth2_credential_form(self): - s = self._min( - auth=OAuth2Auth(client_id="cid", client_secret=SecretStr("sec")), - ) - kwargs = s.to_driver_kwargs() - assert kwargs["credential"] == "cid:sec" - - def test_s3_params_prefixed(self): - """Audit regression: s3.* family was absent.""" - s = self._min( - auth=TokenAuth(token=SecretStr("t")), - S3_ENDPOINT="https://r2.example.com", - S3_REGION="auto", - ) - kwargs = s.to_driver_kwargs() - assert kwargs["s3.endpoint"] == "https://r2.example.com" - assert kwargs["s3.region"] == "auto" -``` - -- [ ] **Step 2: Run → FAIL** - -- [ ] **Step 3: Implement** - -```python -# src/mountainash_data/core/settings/adapters/pyiceberg_rest.py -"""Adapter prefixing s3.*, rest.sigv4-*, header.* keys.""" - -from __future__ import annotations - -import typing as t - -from mountainash_data.core.settings.auth import OAuth2Auth, TokenAuth - -if t.TYPE_CHECKING: - from mountainash_data.core.settings.pyiceberg_rest import ( - PyIcebergRestAuthSettings, - ) - - -def build_driver_kwargs(profile: "PyIcebergRestAuthSettings") -> dict[str, t.Any]: - kwargs = profile._default_driver_kwargs() - - # S3 family - for field, key in [ - ("S3_REGION", "s3.region"), - ("S3_ENDPOINT", "s3.endpoint"), - ("S3_ACCESS_KEY_ID", "s3.access-key-id"), - ]: - val = getattr(profile, field, None) - if val is not None: - kwargs[key] = val - if profile.S3_SECRET_ACCESS_KEY is not None: - kwargs["s3.secret-access-key"] = profile.S3_SECRET_ACCESS_KEY.get_secret_value() - if profile.S3_SESSION_TOKEN is not None: - kwargs["s3.session-token"] = profile.S3_SESSION_TOKEN.get_secret_value() - - # SigV4 - if profile.REST_SIGV4_ENABLED is not None: - kwargs["rest.sigv4-enabled"] = profile.REST_SIGV4_ENABLED - if profile.REST_SIGNING_REGION is not None: - kwargs["rest.signing-region"] = profile.REST_SIGNING_REGION - if profile.REST_SIGNING_NAME is not None: - kwargs["rest.signing-name"] = profile.REST_SIGNING_NAME - - # Headers (dict → header. = v) - if profile.HEADERS: - for hk, hv in profile.HEADERS.items(): - kwargs[f"header.{hk}"] = hv - - # Auth - auth = profile.auth - if isinstance(auth, TokenAuth): - kwargs["token"] = auth.token.get_secret_value() - elif isinstance(auth, OAuth2Auth): - if auth.token is not None: - kwargs["token"] = auth.token.get_secret_value() - elif auth.client_id is not None and auth.client_secret is not None: - kwargs["credential"] = ( - f"{auth.client_id}:{auth.client_secret.get_secret_value()}" - ) - if auth.server_uri is not None: - kwargs["oauth2-server-uri"] = auth.server_uri - if auth.scope is not None: - kwargs["scope"] = auth.scope - return kwargs -``` - -```python -# src/mountainash_data/core/settings/pyiceberg_rest.py -"""PyIceberg REST catalog backend settings. - -Spec: ``docs/superpowers/specs/2026-04-15-settings-audit/pyiceberg_rest.md``. -Driver: https://py.iceberg.apache.org/configuration/ -""" - -from __future__ import annotations - -import typing as t - -from pydantic import SecretStr - -from ..constants import CONST_DB_PROVIDER_TYPE -from .adapters import pyiceberg_rest as _adapter -from .auth import OAuth2Auth, TokenAuth -from .descriptor import BackendDescriptor, ParameterSpec -from .profile import ConnectionProfile -from .registry import register - - -PYICEBERG_REST_DESCRIPTOR = BackendDescriptor( - name="pyiceberg_rest", - provider_type=CONST_DB_PROVIDER_TYPE.PYICEBERG_REST, - connection_string_scheme=None, # uri= kwarg, not URL form - auth_modes=[TokenAuth, OAuth2Auth], - parameters=[ - ParameterSpec(name="CATALOG_NAME", type=str, tier="core", - driver_key="name"), - ParameterSpec(name="CATALOG_URI", type=str, tier="core", - driver_key="uri"), - ParameterSpec(name="WAREHOUSE", type=t.Optional[str], tier="core", - default=None, driver_key="warehouse"), - ParameterSpec(name="VERIFY_SSL", type=bool, tier="advanced", - default=True, driver_key="verify-ssl"), - # S3 family (adapter emits dotted keys) - ParameterSpec(name="S3_REGION", type=t.Optional[str], tier="advanced", - default=None), - ParameterSpec(name="S3_ENDPOINT", type=t.Optional[str], - tier="advanced", default=None), - ParameterSpec(name="S3_ACCESS_KEY_ID", type=t.Optional[str], - tier="advanced", default=None), - ParameterSpec(name="S3_SECRET_ACCESS_KEY", - type=t.Optional[SecretStr], tier="advanced", - default=None), - ParameterSpec(name="S3_SESSION_TOKEN", type=t.Optional[SecretStr], - tier="advanced", default=None), - # SigV4 - ParameterSpec(name="REST_SIGV4_ENABLED", type=t.Optional[bool], - tier="advanced", default=None), - ParameterSpec(name="REST_SIGNING_REGION", type=t.Optional[str], - tier="advanced", default=None), - ParameterSpec(name="REST_SIGNING_NAME", type=t.Optional[str], - tier="advanced", default=None), - ParameterSpec(name="HEADERS", type=t.Optional[dict[str, str]], - tier="advanced", default=None), - ], -) - - -@register(PYICEBERG_REST_DESCRIPTOR) -class PyIcebergRestAuthSettings(ConnectionProfile): - __descriptor__ = PYICEBERG_REST_DESCRIPTOR - __adapter__ = staticmethod(_adapter.build_driver_kwargs) -``` - -- [ ] **Step 4: Run → PASS** - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/core/settings/pyiceberg_rest.py \ - src/mountainash_data/core/settings/adapters/pyiceberg_rest.py \ - tests/test_unit/core/settings/backends/test_pyiceberg_rest.py -git commit -m "refactor(settings): migrate pyiceberg_rest with s3/sigv4/header families" -``` - ---- - -## Task 19: Update `__init__.py` re-exports - -**Files:** -- Modify: `src/mountainash_data/core/settings/__init__.py` - -**Context:** All 11 backends now migrated. `__init__.py` must continue to re-export the per-backend class names plus the new public surface. - -- [ ] **Step 1: Write the __init__ module** - -```python -# src/mountainash_data/core/settings/__init__.py -"""Backend settings — declarative descriptor + registry. - -The *AuthSettings classes below are stable import anchors; internally each -class body is a two-line shell (``__descriptor__`` + ``__adapter__``). -""" - -from __future__ import annotations - -# Core primitives -from .descriptor import MISSING, BackendDescriptor, ParameterSpec -from .profile import ConnectionProfile -from .registry import ( - REGISTRY, - get_descriptor, - get_settings_class, - register, -) - -# Auth union members -from .auth import ( - AuthSpec, - AzureADAuth, - CertificateAuth, - IAMAuth, - JWTAuth, - KerberosAuth, - NoAuth, - OAuth2Auth, - PasswordAuth, - ServiceAccountAuth, - TokenAuth, - WindowsAuth, -) - -# Per-backend settings classes (these import-register themselves). -from .sqlite import SQLiteAuthSettings -from .duckdb import DuckDBAuthSettings -from .motherduck import MotherDuckAuthSettings -from .postgresql import PostgreSQLAuthSettings -from .mysql import MySQLAuthSettings -from .mssql import MSSQLAuthSettings -from .snowflake import SnowflakeAuthSettings -from .bigquery import BigQueryAuthSettings -from .redshift import RedshiftAuthSettings -from .pyspark import PySparkAuthSettings -from .trino import TrinoAuthSettings -from .pyiceberg_rest import PyIcebergRestAuthSettings - -__all__ = [ - # primitives - "MISSING", "BackendDescriptor", "ParameterSpec", "ConnectionProfile", - "REGISTRY", "get_descriptor", "get_settings_class", "register", - # auth - "AuthSpec", "NoAuth", "PasswordAuth", "TokenAuth", "JWTAuth", - "OAuth2Auth", "ServiceAccountAuth", "IAMAuth", "WindowsAuth", - "AzureADAuth", "KerberosAuth", "CertificateAuth", - # backends - "SQLiteAuthSettings", "DuckDBAuthSettings", "MotherDuckAuthSettings", - "PostgreSQLAuthSettings", "MySQLAuthSettings", "MSSQLAuthSettings", - "SnowflakeAuthSettings", "BigQueryAuthSettings", "RedshiftAuthSettings", - "PySparkAuthSettings", "TrinoAuthSettings", "PyIcebergRestAuthSettings", -] -``` - -- [ ] **Step 2: Run the full suite** - -Run: `hatch run test:test-quick` -Expected: PASS. - -- [ ] **Step 3: Commit** - -```bash -git add src/mountainash_data/core/settings/__init__.py -git commit -m "chore(settings): update __init__ re-exports for new registry shape" -``` - ---- - -## Task 20: Update consumer call sites - -**Files:** -- Modify: `src/mountainash_data/core/factories/settings_factory.py` -- Modify: `src/mountainash_data/core/factories/connection_factory.py` -- Modify: `src/mountainash_data/core/factories/operations_factory.py` -- Modify: `src/mountainash_data/core/factories/settings_type_factory_mixin.py` -- Modify: `src/mountainash_data/core/connection.py` -- Modify: `src/mountainash_data/core/utils.py` -- Modify: `src/mountainash_data/backends/ibis/connection.py` -- Modify: `src/mountainash_data/backends/iceberg/connection.py` - -**Context:** Replace every call to the four retired `get_*` methods with the new API. - -| Retired method | Replacement | -|---|---| -| `settings.get_connection_kwargs()` | `settings.to_driver_kwargs()` | -| `settings.get_connection_string_template(scheme)` | `settings.to_connection_string()` | -| `settings.get_connection_string_params()` | (rolled into `to_connection_string`) | -| `settings.get_post_connection_options()` | removed — use SQL statements in the backend layer | -| `settings.db_provider_type` (property) | `settings.provider_type` (property) | - -Also replace `SettingsFactory`'s hand-maintained `if/elif` chain with a single `get_settings_class(name)` registry lookup. - -- [ ] **Step 1: Audit current consumer calls** - -Run: `hatch run ruff:check` is not relevant here — we need a grep pass instead: - -```bash -grep -rn "get_connection_kwargs\|get_connection_string_template\|get_connection_string_params\|get_post_connection_options\|db_provider_type\|BaseDBAuthSettings" \ - src/mountainash_data --include='*.py' \ - | grep -v '/settings/' -``` - -Expected output: list of every file+line to update. Use this as the checklist. - -- [ ] **Step 2: Update each consumer file** - -For each file in the list, apply the table above. Mechanical change. When done, run: - -```bash -grep -rn "get_connection_kwargs\|get_connection_string_template\|get_connection_string_params\|get_post_connection_options\|BaseDBAuthSettings" \ - src/mountainash_data --include='*.py' \ - | grep -v '/settings/' -``` - -Expected: no matches. - -- [ ] **Step 3: Run the full suite** - -Run: `hatch run test:test-quick` -Expected: PASS. - -If something fails, read the error, fix the specific call site, re-run. Do not skip. - -- [ ] **Step 4: Commit** - -```bash -git add src/mountainash_data/core/factories/ \ - src/mountainash_data/core/connection.py \ - src/mountainash_data/core/utils.py \ - src/mountainash_data/backends/ -git commit -m "refactor: migrate consumers to ConnectionProfile.to_driver_kwargs API" -``` - ---- - -## Task 21: Delete retired base + exceptions modules - -**Files:** -- Delete: `src/mountainash_data/core/settings/base.py` -- Delete: `src/mountainash_data/core/settings/exceptions.py` - -- [ ] **Step 1: Search for any remaining references** - -```bash -grep -rn "BaseDBAuthSettings\|DBAuthValidationError\|DBAuthConfigError\|DBAuthConnectionError" \ - src/mountainash_data tests --include='*.py' -``` - -Expected: only matches inside `settings/base.py` and `settings/exceptions.py` (the files being deleted). - -- [ ] **Step 2: Delete the files** - -```bash -git rm src/mountainash_data/core/settings/base.py -git rm src/mountainash_data/core/settings/exceptions.py -``` - -- [ ] **Step 3: Run the full suite** - -Run: `hatch run test:test-quick` -Expected: PASS. - -- [ ] **Step 4: Commit** - -```bash -git commit -m "chore(settings): delete retired BaseDBAuthSettings and exception types" -``` - ---- - -## Task 22: Update docs - -**Files:** -- Modify: `CLAUDE.md` (Settings section if present) -- Modify: `README.md` (Usage Patterns section) -- Modify: `docs/superpowers/specs/2026-04-15-settings-audit/README.md` (note the refactor consumed many findings) - -- [ ] **Step 1: Update `README.md` Usage Patterns** - -Replace the old `from mountainash_data.core.settings import SQLiteAuthSettings` example block with a version that shows `auth=` + `to_driver_kwargs()`: - -```python -from mountainash_data.core.settings import ( - SQLiteAuthSettings, - NoAuth, - PostgreSQLAuthSettings, - PasswordAuth, -) - -sqlite = SQLiteAuthSettings(DATABASE=":memory:", auth=NoAuth()) -pg = PostgreSQLAuthSettings( - HOST="db.example", - DATABASE="app", - auth=PasswordAuth(username="app", password="s3cret"), -) - -kwargs = pg.to_driver_kwargs() # → dict ready for Ibis -url = pg.to_connection_string() # → "postgresql://app:s3cret@db.example:5432/app" -print(pg.provider_type, pg.backend) -``` - -- [ ] **Step 2: Update `CLAUDE.md` settings reference** - -Find the "Settings (`src/mountainash_data/core/settings/`)" bullet and replace with: - -> 3. **Settings** (`src/mountainash_data/core/settings/`) -> - Declarative per-backend descriptors (`BackendDescriptor` + `ParameterSpec` list). -> - Typed discriminated-union auth via `AuthSpec` subclasses (`PasswordAuth`, `OAuth2Auth`, `IAMAuth`, …). -> - Backend shell classes register themselves via `@register` and expose `to_driver_kwargs()` + `to_connection_string()`. -> - Composite driver mappings live in `settings/adapters/.py`. - -- [ ] **Step 3: Add note to audit README** - -Append to `docs/superpowers/specs/2026-04-15-settings-audit/README.md`: - -> **Status (post-refactor, 2026-04-15):** The settings-registry refactor (see -> `docs/superpowers/specs/2026-04-15-settings-registry-design.md` and -> `docs/superpowers/plans/2026-04-15-settings-registry.md`) consumed most -> "core mismatch" and many "core missing" findings in the tables above. -> Remaining items are tracked as per-backend Phase-4 follow-up plans. - -- [ ] **Step 4: Commit** - -```bash -git add README.md CLAUDE.md docs/superpowers/specs/2026-04-15-settings-audit/README.md -git commit -m "docs: update settings usage for registry refactor" -``` - ---- - -## Self-review - -**1. Spec coverage:** -- Problem statement → Tasks 1-6 (scaffolding removes boilerplate, leakage, flat-auth). -- Goals: retire `get_*` methods → Task 20; data-as-descriptors → Tasks 1, 7-18; typed auth union → Task 2; drop base leakage → Task 21; audit fixes carried → Tasks 8-18; stable imports → Task 19; `MountainAshBaseSettings` preserved → Task 4 (`ConnectionProfile(MountainAshBaseSettings)`). -- Non-goals: factories only updated at call sites, not rewritten → Task 20. No new backends. No new auth modes beyond spec table. -- Architecture layers 1-6 → Tasks 4, 1, 1, 2, 8-18 (adapters), 7-18 (shells). -- Data contract → Tasks 1, 2. -- `ConnectionProfile` methods → Task 4. -- Adapter layer → Tasks 3 (default map) + 8-18 (backend adapters). -- File layout → matches Tasks 1-18 + 19. -- Testing: descriptor invariants → Task 6; round-trip per backend → Tasks 7-18; audit regressions → Tasks 8-18 test blocks. -- Migration phases: Phase 1 = Tasks 1-6; Phase 2 = Tasks 7-18 (in cheap→hard order); Phase 3 = Tasks 19-22. -- Phase 4 explicitly deferred to follow-up plans — noted in preamble. - -**2. Placeholder scan:** No "TBD" / "implement later" / "Similar to Task N" / narrative-only steps. Every code step includes actual code. One test step (`test_service_account_info_converts_to_credentials`) has a `try/except` because the dummy SA key cannot fully construct Credentials offline; the intent is documented inline. - -**3. Type consistency:** -- `ConnectionProfile` / `BackendDescriptor` / `ParameterSpec` / `MISSING` — consistent across tasks. -- `to_driver_kwargs` / `to_connection_string` / `provider_type` / `backend` names consistent. -- `AuthSpec` subclass names match the spec table and the auth dispatch map. -- `__descriptor__` / `__adapter__` class variables consistent. -- Adapter function signature `(profile) -> dict[str, t.Any]` consistent across all backends. - ---- - -## Execution Handoff - -Plan complete and saved to `docs/superpowers/plans/2026-04-15-settings-registry.md`. Two execution options: - -**1. Subagent-Driven (recommended)** - I dispatch a fresh subagent per task, review between tasks, fast iteration. - -**2. Inline Execution** - Execute tasks in this session using executing-plans, batch execution with checkpoints. - -Which approach? diff --git a/docs/superpowers/plans/2026-04-16-profiles-migration-data.md b/docs/superpowers/plans/2026-04-16-profiles-migration-data.md deleted file mode 100644 index c1ccedd..0000000 --- a/docs/superpowers/plans/2026-04-16-profiles-migration-data.md +++ /dev/null @@ -1,901 +0,0 @@ -# Profiles Promotion — Phase 2: `mountainash-data` Migration - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Replace `mountainash-data`'s internal descriptor/auth/profile/registry modules with thin re-exports + subclasses of the new `mountainash-settings.profiles` and `mountainash-settings.auth` sub-packages. The external API of `mountainash_data.core.settings` (imported by downstream packages and app code) stays byte-identical. - -**Architecture:** `ConnectionProfile` becomes a thin subclass of `DescriptorProfile` that adds the database-flavored `to_driver_kwargs()` and `to_connection_string()` output methods. A module-level `DATABASES_REGISTRY = Registry("databases")` + bound `register` decorator replaces the current module-level `REGISTRY` dict. All 12 per-backend shell classes and all per-backend adapter modules are unchanged beyond their import statements. - -**Tech Stack:** Python 3.12, pydantic v2, `mountainash-settings` (now exporting profiles + auth), `MountainAshBaseSettings`. - -**Prerequisites:** -- Phase 1 (`mountainash-settings` scaffolding) merged and a new `mountainash-settings` release published so this package can pin the version. -- Current branch tip: `2ec5079` on `feat/settings-registry` (settings-registry refactor complete). - -**Working directory:** `/home/nathanielramm/git/mountainash-io/mountainash/mountainash-data` - -**Branch:** Create `feat/profiles-migration` from `main` **after** `feat/settings-registry` is merged. If `feat/settings-registry` has NOT been merged yet at the time this plan starts, branch from `feat/settings-registry` instead and note the merge dependency in the PR. - -**Spec:** `../../mountainash-settings/docs/superpowers/specs/2026-04-16-profiles-promotion-design.md` - ---- - -## Task 1: Branch + dependency bump - -**Files:** -- Modify: `hatch.toml` -- Modify: `pyproject.toml` (if pinning present there) - -- [ ] **Step 1: Branch** - -```bash -cd /home/nathanielramm/git/mountainash-io/mountainash/mountainash-data -git checkout -b feat/profiles-migration -``` - -- [ ] **Step 2: Pin the new `mountainash-settings` version** - -Grep current pin: - -```bash -grep -rn "mountainash-settings\|mountainash_settings" hatch.toml pyproject.toml 2>/dev/null -``` - -Update the pinned version to the release that includes Phase 1. The `hatch.toml` in this repo uses relative-path deps for the sibling monorepo (`mountainash_settings @ {root:uri}/../mountainash-settings`); that path is unchanged — **but** ensure the sibling checkout is at the Phase-1 merge commit before running tests. - -```bash -cd ../mountainash-settings && git pull --ff-only && git log --oneline -1 -# Confirm Phase-1 commits are present. -cd ../mountainash-data -``` - -- [ ] **Step 3: Run tests to confirm environment is stable** - -```bash -hatch run test:test-target tests/test_unit/core/settings/ -v 2>&1 | tail -5 -``` - -Expected: existing 235 tests pass, 2 skipped. No migration work yet — just confirming the new `mountainash-settings` doesn't break the current code (it shouldn't; Phase 1 was purely additive). - -- [ ] **Step 4: Commit (trivially if no file changes; skip if none)** - -If `hatch.toml` needed no change, skip this commit. Otherwise: - -```bash -git add hatch.toml pyproject.toml -git commit -m "chore(deps): pin mountainash-settings to profiles-promotion release" -``` - ---- - -## Task 2: New `ConnectionProfile` subclass + `DATABASES_REGISTRY` - -**Files:** -- Modify: `src/mountainash_data/core/settings/profile.py` (full rewrite — much smaller) -- Modify: `src/mountainash_data/core/settings/registry.py` (full rewrite — just a wrapper) - -- [ ] **Step 1: Rewrite `profile.py` as thin subclass** - -```python -# src/mountainash_data/core/settings/profile.py -"""ConnectionProfile — database-flavored subclass of DescriptorProfile. - -Adds ``to_driver_kwargs()`` and ``to_connection_string()`` on top of the -generic mechanism provided by -:class:`mountainash_settings.profiles.DescriptorProfile`. -""" - -from __future__ import annotations - -import typing as t -from urllib.parse import quote - -from pydantic import SecretStr - -from mountainash_settings.profiles import DescriptorProfile - -__all__ = ["ConnectionProfile"] - - -class ConnectionProfile(DescriptorProfile): - """Database connection settings. - - Public API: - - :meth:`to_driver_kwargs` — dict ready for the Ibis driver. - - :meth:`to_connection_string` — URL form, or ``NotImplementedError`` - if the descriptor has no ``connection_string_scheme`` metadata. - - Subclasses set ``__descriptor__`` (a :class:`ProfileDescriptor`) and - optionally ``__adapter__``. Field installation, auth union, and template - wiring are inherited from :class:`DescriptorProfile`. - """ - - def to_driver_kwargs(self) -> dict[str, t.Any]: - """Build the final driver kwargs dict. - - If ``__adapter__`` is set, it owns the full pipeline — typically it - calls :meth:`_default_kwargs` and :meth:`_auth_kwargs` and layers - composite mappings on top. Otherwise defaults to descriptor - ``driver_key`` mappings + default auth dispatch. - """ - adapter = type(self).__dict__.get("__adapter__") - if adapter is None: - for base in type(self).__mro__[1:]: - candidate = base.__dict__.get("__adapter__") - if candidate is not None: - adapter = candidate - break - if adapter is not None: - return adapter(self) - kwargs = self._default_kwargs() - kwargs.update(self._auth_kwargs()) - return kwargs - - def to_connection_string(self) -> str: - """Build ``scheme://user:pass@host:port/database`` from the descriptor. - - Reads the scheme from ``descriptor.metadata['connection_string_scheme']`` - (or a typed ``connection_string_scheme`` attribute if the descriptor - subclass provides one). Raises :class:`NotImplementedError` if absent. - """ - desc = self.__descriptor__ - scheme = getattr(desc, "connection_string_scheme", None) - if scheme is None: - scheme = desc.metadata.get("connection_string_scheme") - if scheme is None: - raise NotImplementedError( - f"Profile {self.backend!r} has no connection string scheme" - ) - host = getattr(self, "HOST", None) - port = getattr(self, "PORT", None) - database = getattr(self, "DATABASE", None) - url = scheme - auth = getattr(self, "auth", None) - if auth is not None: - username = getattr(auth, "username", None) - if username: - url += quote(str(username), safe="") - pw = getattr(auth, "password", None) - if isinstance(pw, SecretStr): - url += ":" + quote(pw.get_secret_value(), safe="") - url += "@" - if host is not None: - url += str(host) - if port is not None: - url += f":{port}" - if database is not None: - url += f"/{database}" - return url -``` - -- [ ] **Step 2: Rewrite `registry.py` as a wrapper around `DATABASES_REGISTRY`** - -```python -# src/mountainash_data/core/settings/registry.py -"""Module-level registry of database backend descriptors. - -Backed by :class:`mountainash_settings.profiles.Registry` — a per-domain -registry class. The old module-level ``REGISTRY`` dict is preserved as a -property-style alias for any downstream consumer that imports it directly. -""" - -from __future__ import annotations - -import typing as t - -from mountainash_settings.profiles import Registry - -if t.TYPE_CHECKING: - from mountainash_settings.profiles import ProfileDescriptor - from .profile import ConnectionProfile - -__all__ = [ - "DATABASES_REGISTRY", - "REGISTRY", - "get_descriptor", - "get_settings_class", - "register", -] - -DATABASES_REGISTRY = Registry("databases") - -register = DATABASES_REGISTRY.decorator() - - -def get_descriptor(name: str) -> "ProfileDescriptor": - return DATABASES_REGISTRY.get_descriptor(name) - - -def get_settings_class(name: str) -> type["ConnectionProfile"]: - return DATABASES_REGISTRY.get_settings_class(name) # type: ignore[return-value] - - -# Backwards-compatibility alias — preserves ``from ... import REGISTRY`` imports. -# Read-only from the outside; mutations should go through ``@register``. -class _RegistryDictView: - """Dict-like view that delegates to DATABASES_REGISTRY.descriptors.""" - - def __contains__(self, name: str) -> bool: - return name in DATABASES_REGISTRY - - def __getitem__(self, name: str) -> "ProfileDescriptor": - return DATABASES_REGISTRY.get_descriptor(name) - - def __iter__(self): - return iter(DATABASES_REGISTRY.descriptors) - - def __len__(self) -> int: - return len(DATABASES_REGISTRY) - - def items(self): - return DATABASES_REGISTRY.descriptors.items() - - def keys(self): - return DATABASES_REGISTRY.descriptors.keys() - - def values(self): - return DATABASES_REGISTRY.descriptors.values() - - -REGISTRY = _RegistryDictView() -``` - -- [ ] **Step 3: Delete the retired local files** - -```bash -git rm src/mountainash_data/core/settings/descriptor.py -git rm -r src/mountainash_data/core/settings/auth/ -``` - -- [ ] **Step 4: Run tests to see what breaks** - -```bash -hatch run test:test-target tests/test_unit/core/settings/ -v 2>&1 | tail -30 -``` - -Expected: massive import breakage across per-backend files (sqlite.py, duckdb.py, etc.) because they import `from .descriptor import ...` and `from .auth import ...` — those paths no longer exist. - -This is intentional — fix in Task 3. - -**Do NOT commit yet.** - ---- - -## Task 3: Fix per-backend imports across all 12 backends + adapters - -**Files:** -- Modify: `src/mountainash_data/core/settings/sqlite.py` -- Modify: `src/mountainash_data/core/settings/duckdb.py` -- Modify: `src/mountainash_data/core/settings/motherduck.py` -- Modify: `src/mountainash_data/core/settings/postgresql.py` -- Modify: `src/mountainash_data/core/settings/mysql.py` -- Modify: `src/mountainash_data/core/settings/mssql.py` -- Modify: `src/mountainash_data/core/settings/snowflake.py` -- Modify: `src/mountainash_data/core/settings/bigquery.py` -- Modify: `src/mountainash_data/core/settings/redshift.py` -- Modify: `src/mountainash_data/core/settings/pyspark.py` -- Modify: `src/mountainash_data/core/settings/trino.py` -- Modify: `src/mountainash_data/core/settings/pyiceberg_rest.py` -- Modify: `src/mountainash_data/core/settings/adapters/*.py` (7 files) - -- [ ] **Step 1: Mechanical import rewrites** - -Every per-backend and adapter file needs three import changes: - -| Old | New | -|-----|-----| -| `from .descriptor import BackendDescriptor, ParameterSpec` | `from mountainash_settings.profiles import ParameterSpec, ProfileDescriptor as BackendDescriptor` | -| `from .auth import NoAuth, PasswordAuth, ...` | `from mountainash_settings.auth import NoAuth, PasswordAuth, ...` | -| `from .registry import register` | (unchanged — local `register` is still the exported bound decorator) | -| `from .profile import ConnectionProfile` | (unchanged — local `ConnectionProfile` still exists as subclass) | - -Adapter files also have: - -| Old | New | -|-----|-----| -| `from mountainash_data.core.settings.auth import ...` | `from mountainash_settings.auth import ...` | -| `profile._default_driver_kwargs()` | `profile._default_kwargs()` | -| `profile._auth_to_driver_kwargs()` | `profile._auth_kwargs()` | - -The method rename in adapters is the only non-trivial change — `DescriptorProfile` renamed the helpers to drop the "driver" prefix. - -Run a bulk edit script: - -```bash -# In the mountainash-data repo root: - -# 1. Per-backend settings files: swap descriptor + auth imports -for f in src/mountainash_data/core/settings/{sqlite,duckdb,motherduck,postgresql,mysql,mssql,snowflake,bigquery,redshift,pyspark,trino,pyiceberg_rest}.py; do - sed -i ' - s|from \.descriptor import BackendDescriptor, ParameterSpec|from mountainash_settings.profiles import ParameterSpec, ProfileDescriptor as BackendDescriptor| - s|from \.auth import|from mountainash_settings.auth import| - ' "$f" -done - -# 2. Adapter files: swap auth imports + method rename -for f in src/mountainash_data/core/settings/adapters/*.py; do - sed -i ' - s|from mountainash_data\.core\.settings\.auth import|from mountainash_settings.auth import|g - s|_default_driver_kwargs()|_default_kwargs()|g - s|_auth_to_driver_kwargs()|_auth_kwargs()|g - ' "$f" -done -``` - -- [ ] **Step 2: Sanity-check the rewrites** - -```bash -grep -rn "from \.descriptor\|from \.auth import\|from mountainash_data\.core\.settings\.auth\|_default_driver_kwargs\|_auth_to_driver_kwargs" \ - src/mountainash_data/core/settings/ 2>&1 | grep -v "^Binary" -``` - -Expected: no matches. Every old-path import or method call should be gone. - -- [ ] **Step 3: Run the settings suite** - -```bash -hatch run test:test-target tests/test_unit/core/settings/ -v 2>&1 | tail -20 -``` - -Expected: most backend tests pass, some may fail due to descriptor-subclassing quirks (see Task 4) or `REGISTRY` being empty on first import. - -If a test fails with "module has no attribute `ConnectionProfile`", check that `profile.py` was rewritten in Task 2 Step 1. - -- [ ] **Step 4: Commit (even if some tests still fail — next task fixes them)** - -```bash -git add src/mountainash_data/core/settings/ -git commit -m "refactor(settings): rewire imports to mountainash-settings.profiles + auth" -``` - ---- - -## Task 4: Adjust `ProfileDescriptor` usage in per-backend files - -**Context:** `mountainash-data` descriptors use several typed metadata fields that `ProfileDescriptor` moved to a generic `metadata: dict[str, Any]`: `default_port`, `connection_string_scheme`, `ibis_dialect`, `rides_on`. Two options: - -1. **Stuff into metadata dict** — quick, works everywhere: `metadata={"default_port": 5432, "connection_string_scheme": "postgresql://", ...}`. -2. **Subclass `ProfileDescriptor`** — typed. Preferred for mountainash-data given the fields are heavily used by adapters and `to_connection_string()`. - -Use Option 2. - -**Files:** -- Create: `src/mountainash_data/core/settings/descriptor.py` (small — just the subclass) -- Modify: 12 per-backend settings files (revert the `ProfileDescriptor as BackendDescriptor` alias to use the typed subclass) - -- [ ] **Step 1: Create a typed subclass** - -```python -# src/mountainash_data/core/settings/descriptor.py -"""Database-flavored ProfileDescriptor with typed metadata fields. - -Retained in mountainash-data (rather than lifted to mountainash-settings) -because these fields are domain-specific: ``connection_string_scheme`` and -``ibis_dialect`` are meaningful only for SQL-like databases. -""" - -from __future__ import annotations - -import typing as t -from dataclasses import dataclass, field - -from mountainash_settings.profiles import ( - MISSING, - ParameterSpec, - ProfileDescriptor, -) - -__all__ = ["MISSING", "BackendDescriptor", "ParameterSpec"] - - -@dataclass(frozen=True, kw_only=True) -class BackendDescriptor(ProfileDescriptor): - """ProfileDescriptor with database-specific typed metadata. - - Extra fields: - default_port: Default TCP port if the backend listens on one. - connection_string_scheme: URL scheme prefix (``"postgresql://"``) or - ``None`` if the backend has no URL form. - ibis_dialect: Name of the Ibis backend if Ibis handles this backend. - rides_on: Name of another backend whose Ibis path this one routes - through (e.g. ``motherduck`` → ``duckdb``). Metadata only. - """ - - default_port: int | None = None - connection_string_scheme: str | None = None - ibis_dialect: str | None = None - rides_on: str | None = None -``` - -- [ ] **Step 2: Revert per-backend imports to use the local subclass** - -```bash -for f in src/mountainash_data/core/settings/{sqlite,duckdb,motherduck,postgresql,mysql,mssql,snowflake,bigquery,redshift,pyspark,trino,pyiceberg_rest}.py; do - sed -i ' - s|from mountainash_settings\.profiles import ParameterSpec, ProfileDescriptor as BackendDescriptor|from .descriptor import BackendDescriptor, ParameterSpec| - ' "$f" -done -``` - -- [ ] **Step 3: Run tests** - -```bash -hatch run test:test-target tests/test_unit/core/settings/ -v 2>&1 | tail -20 -``` - -Expected: all 235 settings tests pass, 2 skipped. If `_default_kwargs()` or `_auth_kwargs()` method-rename errors remain in some adapter, fix them (they should have been caught in Task 3 Step 1). - -- [ ] **Step 4: Commit** - -```bash -git add src/mountainash_data/core/settings/descriptor.py \ - src/mountainash_data/core/settings/{sqlite,duckdb,motherduck,postgresql,mysql,mssql,snowflake,bigquery,redshift,pyspark,trino,pyiceberg_rest}.py -git commit -m "refactor(settings): re-introduce typed BackendDescriptor subclass" -``` - ---- - -## Task 5: Update `settings/__init__.py` re-exports - -**Files:** -- Modify: `src/mountainash_data/core/settings/__init__.py` - -The external API must remain byte-identical to what downstream packages consume. Symbols move upstream, imports move upstream, but the names exported from `mountainash_data.core.settings` stay the same. - -- [ ] **Step 1: Rewrite `__init__.py`** - -```python -# src/mountainash_data/core/settings/__init__.py -"""Backend settings — declarative descriptor + registry. - -The *AuthSettings classes below are stable import anchors; internally each -class body is a two-line shell (``__descriptor__`` + ``__adapter__``). - -As of 2026-04-16, the descriptor/registry/auth machinery lives in -``mountainash-settings``. Symbols re-exported here preserve the external API. -""" - -from __future__ import annotations - -# Core primitives (auth + profiles from mountainash-settings) -from mountainash_settings.auth import ( - AuthSpec, - AzureADAuth, - CertificateAuth, - IAMAuth, - JWTAuth, - KerberosAuth, - NoAuth, - OAuth2Auth, - PasswordAuth, - ServiceAccountAuth, - TokenAuth, - WindowsAuth, -) - -# Local database-flavored subclasses -from .descriptor import MISSING, BackendDescriptor, ParameterSpec -from .profile import ConnectionProfile -from .registry import ( - DATABASES_REGISTRY, - REGISTRY, - get_descriptor, - get_settings_class, - register, -) - -# Per-backend settings classes (these import-register themselves). -from .sqlite import SQLiteAuthSettings -from .duckdb import DuckDBAuthSettings -from .motherduck import MotherDuckAuthSettings -from .postgresql import PostgreSQLAuthSettings -from .mysql import MySQLAuthSettings -from .mssql import MSSQLAuthSettings -from .snowflake import SnowflakeAuthSettings -from .bigquery import BigQueryAuthSettings -from .redshift import RedshiftAuthSettings -from .pyspark import PySparkAuthSettings -from .trino import TrinoAuthSettings -from .pyiceberg_rest import PyIcebergRestAuthSettings - -__all__ = [ - # primitives - "MISSING", "BackendDescriptor", "ParameterSpec", "ConnectionProfile", - "DATABASES_REGISTRY", "REGISTRY", - "get_descriptor", "get_settings_class", "register", - # auth - "AuthSpec", "NoAuth", "PasswordAuth", "TokenAuth", "JWTAuth", - "OAuth2Auth", "ServiceAccountAuth", "IAMAuth", "WindowsAuth", - "AzureADAuth", "KerberosAuth", "CertificateAuth", - # backends - "SQLiteAuthSettings", "DuckDBAuthSettings", "MotherDuckAuthSettings", - "PostgreSQLAuthSettings", "MySQLAuthSettings", "MSSQLAuthSettings", - "SnowflakeAuthSettings", "BigQueryAuthSettings", "RedshiftAuthSettings", - "PySparkAuthSettings", "TrinoAuthSettings", "PyIcebergRestAuthSettings", -] -``` - -- [ ] **Step 2: Run full suite** - -```bash -hatch run test:test-target tests/test_unit/ -v 2>&1 | tail -20 -``` - -Expected: all 483 tests pass, 5 skipped. No regressions. - -- [ ] **Step 3: Commit** - -```bash -git add src/mountainash_data/core/settings/__init__.py -git commit -m "chore(settings): update __init__ re-exports for mountainash-settings promotion" -``` - ---- - -## Task 6: Replace the invariants test with the shared helper - -**Files:** -- Modify: `tests/test_unit/core/settings/test_descriptors_invariants.py` (replace contents) - -The existing file hand-rolls 10 parametric invariants. The promoted helper -`descriptor_invariants_for(registry)` gives the same coverage from one line. - -- [ ] **Step 1: Replace the file** - -```python -# tests/test_unit/core/settings/test_descriptors_invariants.py -"""Parametric descriptor invariants for all registered database backends. - -Generated from the shared ``descriptor_invariants_for`` helper in -``mountainash-settings``. Every descriptor in ``DATABASES_REGISTRY`` gets -checked against the invariants for free — no per-backend test additions -required. -""" - -from __future__ import annotations - -# Ensure every backend module's @register decorator has fired before we -# snapshot the registry for the parametrize decorator. -import mountainash_data.core.settings # noqa: F401 - -from mountainash_data.core.settings.registry import DATABASES_REGISTRY -from mountainash_settings.profiles import descriptor_invariants_for - -TestDatabaseInvariants = descriptor_invariants_for(DATABASES_REGISTRY) -``` - -- [ ] **Step 2: Run the invariants suite** - -```bash -hatch run test:test-target tests/test_unit/core/settings/test_descriptors_invariants.py -v 2>&1 | tail -10 -``` - -Expected: 120 parametric cases pass (12 backends × 10 invariants). - -- [ ] **Step 3: Run the full settings suite** - -```bash -hatch run test:test-target tests/test_unit/core/settings/ -v 2>&1 | tail -5 -``` - -Expected: all 235+ tests pass. - -- [ ] **Step 4: Commit** - -```bash -git add tests/test_unit/core/settings/test_descriptors_invariants.py -git commit -m "test(settings): switch to shared descriptor_invariants_for helper" -``` - ---- - -## Task 7: Delete the now-dead local auth/dispatch tests - -**Files:** -- Delete: `tests/test_unit/core/settings/test_auth.py` -- Delete: `tests/test_unit/core/settings/test_auth_dispatch.py` -- Keep: `tests/test_unit/core/settings/test_descriptor.py` — rewrite to import from mountainash-settings -- Keep: `tests/test_unit/core/settings/test_profile.py` — rewrite to test `ConnectionProfile` specifically (the data-flavored subclass) -- Keep: `tests/test_unit/core/settings/test_registry.py` — rewrite to test `DATABASES_REGISTRY` specifically - -- [ ] **Step 1: Delete the duplicated auth tests** - -Auth coverage now lives in `mountainash-settings`'s test suite. Deleting the copies here avoids duplicate execution. - -```bash -git rm tests/test_unit/core/settings/test_auth.py -git rm tests/test_unit/core/settings/test_auth_dispatch.py -``` - -- [ ] **Step 2: Rewrite `test_descriptor.py` to exercise the typed subclass** - -```python -# tests/test_unit/core/settings/test_descriptor.py -"""Tests for the database-flavored BackendDescriptor subclass.""" - -import pytest - -from mountainash_data.core.settings.auth import NoAuth -from mountainash_data.core.settings.descriptor import ( - BackendDescriptor, - ParameterSpec, -) - - -@pytest.mark.unit -class TestBackendDescriptor: - def test_default_port_field(self): - d = BackendDescriptor( - name="x", provider_type="x", - parameters=[], auth_modes=[NoAuth], - default_port=5432, - ) - assert d.default_port == 5432 - - def test_connection_string_scheme_field(self): - d = BackendDescriptor( - name="x", provider_type="x", - parameters=[], auth_modes=[NoAuth], - connection_string_scheme="postgresql://", - ) - assert d.connection_string_scheme == "postgresql://" - - def test_rides_on_field(self): - d = BackendDescriptor( - name="motherduck", provider_type="motherduck", - parameters=[], auth_modes=[NoAuth], - rides_on="duckdb", - ) - assert d.rides_on == "duckdb" - - def test_frozen(self): - d = BackendDescriptor( - name="x", provider_type="x", - parameters=[], auth_modes=[NoAuth], - ) - with pytest.raises(Exception): - d.name = "y" # type: ignore -``` - -- [ ] **Step 3: Rewrite `test_profile.py` to test only ConnectionProfile-specific methods** - -```python -# tests/test_unit/core/settings/test_profile.py -"""Tests for ConnectionProfile — database-flavored DescriptorProfile. - -DescriptorProfile mechanism tests live in mountainash-settings. Here we only -exercise the database-specific methods: to_driver_kwargs() and -to_connection_string(). -""" - -from __future__ import annotations - -import pytest -from pydantic import SecretStr - -from mountainash_data.core.settings.auth import NoAuth, PasswordAuth -from mountainash_data.core.settings.descriptor import ( - BackendDescriptor, - ParameterSpec, -) -from mountainash_data.core.settings.profile import ConnectionProfile - - -DUMMY_DESCRIPTOR = BackendDescriptor( - name="dummy", - provider_type="dummy", - default_port=9999, - connection_string_scheme="dummy://", - parameters=[ - ParameterSpec(name="HOST", type=str, tier="core", driver_key="host"), - ParameterSpec(name="PORT", type=int, tier="core", default=9999, driver_key="port"), - ParameterSpec(name="DATABASE", type=str, tier="core", default=None, - driver_key="database"), - ], - auth_modes=[NoAuth, PasswordAuth], -) - - -class DummyProfile(ConnectionProfile): - __descriptor__ = DUMMY_DESCRIPTOR - - -@pytest.mark.unit -class TestConnectionProfile: - def test_to_driver_kwargs_default(self): - p = DummyProfile(HOST="h", PORT=1234, DATABASE="db", auth=NoAuth()) - kwargs = p.to_driver_kwargs() - assert kwargs["host"] == "h" - assert kwargs["port"] == 1234 - assert kwargs["database"] == "db" - - def test_to_driver_kwargs_password_unwrapped(self): - p = DummyProfile( - HOST="h", DATABASE="db", - auth=PasswordAuth(username="u", password=SecretStr("p")), - ) - kwargs = p.to_driver_kwargs() - assert kwargs["user"] == "u" - assert kwargs["password"] == "p" - - def test_to_driver_kwargs_adapter_owns_pipeline(self): - def _adapter(profile): - return {"only": "thing"} - - class Adapted(ConnectionProfile): - __descriptor__ = DUMMY_DESCRIPTOR - __adapter__ = staticmethod(_adapter) - - p = Adapted(HOST="h", auth=NoAuth()) - assert p.to_driver_kwargs() == {"only": "thing"} - - def test_to_connection_string_full(self): - p = DummyProfile( - HOST="h", DATABASE="db", - auth=PasswordAuth(username="u", password=SecretStr("p")), - ) - url = p.to_connection_string() - assert url == "dummy://u:p@h:9999/db" - - def test_to_connection_string_url_encodes_secrets(self): - p = DummyProfile( - HOST="h", DATABASE="db", - auth=PasswordAuth(username="user@corp", password=SecretStr("p@ss:w/ord")), - ) - url = p.to_connection_string() - assert "user%40corp" in url - assert "p%40ss%3Aw%2Ford" in url - - def test_to_connection_string_no_scheme_raises(self): - desc = BackendDescriptor( - name="x", provider_type="x", parameters=[], auth_modes=[NoAuth], - connection_string_scheme=None, - ) - - class P(ConnectionProfile): - __descriptor__ = desc - - p = P(auth=NoAuth()) - with pytest.raises(NotImplementedError): - p.to_connection_string() -``` - -- [ ] **Step 4: Rewrite `test_registry.py` to test `DATABASES_REGISTRY` wrapper** - -```python -# tests/test_unit/core/settings/test_registry.py -"""Tests for the DATABASES_REGISTRY wrapper + back-compat REGISTRY alias.""" - -import pytest - -from mountainash_data.core.settings.registry import ( - DATABASES_REGISTRY, - REGISTRY, - get_descriptor, - get_settings_class, -) - - -@pytest.mark.unit -class TestDatabasesRegistry: - def test_registry_is_populated_after_import(self): - """All 12 backends register themselves at import time.""" - import mountainash_data.core.settings # noqa: F401 - - for name in ["sqlite", "duckdb", "postgresql", "mysql", "mssql", - "snowflake", "bigquery", "redshift", "pyspark", - "trino", "motherduck", "pyiceberg_rest"]: - assert name in DATABASES_REGISTRY, f"{name} missing from registry" - - def test_get_descriptor_returns_correct_type(self): - import mountainash_data.core.settings # noqa: F401 - desc = get_descriptor("sqlite") - assert desc.name == "sqlite" - - def test_get_settings_class_returns_correct_type(self): - import mountainash_data.core.settings # noqa: F401 - from mountainash_data.core.settings.sqlite import SQLiteAuthSettings - assert get_settings_class("sqlite") is SQLiteAuthSettings - - def test_legacy_REGISTRY_alias_still_works(self): - import mountainash_data.core.settings # noqa: F401 - assert "sqlite" in REGISTRY - assert REGISTRY["sqlite"].name == "sqlite" - # Iterate - names = list(REGISTRY.keys()) - assert "sqlite" in names -``` - -- [ ] **Step 5: Run the full suite** - -```bash -hatch run test:test-target tests/test_unit/ -v 2>&1 | tail -10 -``` - -Expected: all tests pass. Some test counts shift — auth tests went away (now in mountainash-settings); new registry tests added; descriptor/profile tests trimmed to data-specific cases. - -- [ ] **Step 6: Commit** - -```bash -git add tests/test_unit/core/settings/ -git commit -m "test(settings): trim tests to data-specific cases; delete duplicates" -``` - ---- - -## Task 8: Update CLAUDE.md references - -**Files:** -- Modify: `CLAUDE.md` - -The settings section currently describes the pattern. Update it to point at the new location. - -- [ ] **Step 1: Find and update the Settings section** - -Find the bullet about `src/mountainash_data/core/settings/` in `CLAUDE.md`. Replace with: - -```markdown -3. **Settings** (`src/mountainash_data/core/settings/`) - - Database-flavored layer over `mountainash-settings`'s `profiles` and - `auth` sub-packages. - - `BackendDescriptor` is a typed `ProfileDescriptor` subclass; every - backend is a two-line shell registered via `@register`. - - `ConnectionProfile` adds `to_driver_kwargs()` and `to_connection_string()` - on top of the generic `DescriptorProfile` base. - - Composite driver mappings live in `settings/adapters/.py`. -``` - -- [ ] **Step 2: Commit** - -```bash -git add CLAUDE.md -git commit -m "docs: update CLAUDE.md settings section for mountainash-settings promotion" -``` - ---- - -## Task 9: Open PR - -- [ ] **Step 1: Push** - -```bash -git push -u origin feat/profiles-migration -``` - -- [ ] **Step 2: Open PR** - -Title: `Profiles promotion — Phase 2: mountainash-data migrates to mountainash-settings.profiles` - -Body should include: - -- Link to design spec -- Link to Phase 1 PR / release -- Confirmation that external API (`mountainash_data.core.settings.__all__`) is byte-identical: diff the old and new `__all__` lists as evidence -- Test count: 483 passed, 5 skipped (unchanged from current branch) - ---- - -## Self-Review Notes - -- **Spec coverage:** - - Section 1 (layout `mountainash-data` shrinks) → Tasks 2–5 - - Section 2 (thin `ConnectionProfile` subclass) → Task 2 - - Section 3 (per-backend imports) → Task 3 - - Section 4 (typed `BackendDescriptor`) → Task 4 - - Section 5 (public re-exports unchanged) → Task 5 - - Section 6 (shared invariants helper) → Task 6 - - Section 7 (docs) → Task 8 -- **Placeholder scan:** No "TBD"s. Every step has verbatim code or exact commands. -- **Type consistency:** `ConnectionProfile` / `BackendDescriptor` / `ParameterSpec` / `DATABASES_REGISTRY` consistent throughout. -- **Setattr-bypass limitation:** Inherited from `DescriptorProfile`. Existing workarounds (PySpark `__setattr__`, adapter `str(enum)` defense) unchanged by this migration — they still live in their respective files. -- **Back-compat alias:** `REGISTRY` dict-like view preserves any downstream `from mountainash_data.core.settings.registry import REGISTRY` imports. - -## Execution Handoff - -After Phase 2 lands and merges, three migration plans remain — they can be drafted as needed using this plan and Phase 1 as the template: - -- Phase 3: `mountainash-utils-secrets` (5 providers) -- Phase 4: `mountainash-transport` (18 providers) -- Phase 5: `mountainash-acrds-core` (single descriptor; primary Pattern B validation) - -Each follows the same skeleton as Phase 2: branch, add thin subclass + per-domain `Registry`, rewrite per-provider imports, update `__init__.py`, delete retired base class, run tests. They are left unwritten until Phase 2 proves the pattern in production. diff --git a/docs/superpowers/plans/2026-04-26-legacy-cleanup-final.md b/docs/superpowers/plans/2026-04-26-legacy-cleanup-final.md deleted file mode 100644 index bb51154..0000000 --- a/docs/superpowers/plans/2026-04-26-legacy-cleanup-final.md +++ /dev/null @@ -1,254 +0,0 @@ -# Legacy Cleanup — Final Two Items Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Remove the last two legacy artefacts from the April 2026 settings-registry refactor — the deprecated bridge methods on `BaseDBConnection` and the redundant `SecretStr` guard in `ConnectionProfile.to_connection_string()`. - -**Architecture:** Two independent deletions. Item 1 removes four methods from the abstract base class and the legacy fallback branches in Ibis/Iceberg `connect_default()`. Item 2 replaces an `isinstance(pw, SecretStr)` guard with a simpler `pw is not None` check. Both are safe because all downstream settings classes now use `ConnectionProfile`. - -**Tech Stack:** Python, pydantic, mountainash-settings, mountainash-data - -**Spec:** `docs/superpowers/specs/2026-04-26-legacy-cleanup-final.md` - ---- - -## File Map - -| File | Action | What changes | -|------|--------|--------------| -| `src/mountainash_data/core/connection.py` | Modify | Delete 4 bridge methods + commented-out block | -| `src/mountainash_data/backends/ibis/connection.py` | Modify | Delete legacy fallback branch in `connect_default()`, remove `isinstance` guard | -| `src/mountainash_data/backends/iceberg/connection.py` | Modify | Remove `isinstance` dispatch in `connect_default()` | -| `src/mountainash_data/core/settings/profile.py` | Modify | Replace `isinstance(pw, SecretStr)` with `pw is not None`, drop `SecretStr` import | - ---- - -### Task 1: Remove bridge methods from BaseDBConnection - -**Files:** -- Modify: `src/mountainash_data/core/connection.py:135-248` - -- [ ] **Step 1: Delete the four bridge methods and commented-out block** - -In `src/mountainash_data/core/connection.py`, delete everything from line 122 (`# def prepare_connection_parameters`) through line 248 (end of file). This removes: - -- Commented-out `prepare_connection_parameters` block (lines 122-132) -- `get_connection_string_template()` (lines 135-156) -- `get_connection_string_params()` (lines 158-175) -- `get_connection_kwargs()` (lines 177-194) -- `format_connection_string()` (lines 196-216) -- Second commented-out `format_connection_string` block (lines 220-248) - -The file should end after `init_ssh()` (line 120) with a clean trailing newline. - -Also remove the now-unused import of `ConnectionProfile` from line 8: -```python -# DELETE this line: -from mountainash_data.core.settings import ConnectionProfile -``` - -And clean up the `Dict` import from line 1 if no longer used (it won't be — the bridge methods were the only consumers): -```python -# BEFORE: -from typing import Optional, Any, Type, Dict -# AFTER: -from typing import Optional, Any, Type -``` - -- [ ] **Step 2: Run tests to verify nothing breaks** - -Run: `hatch run test:test-quick` -Expected: All tests pass. No tests reference the bridge methods (confirmed by grep). - -- [ ] **Step 3: Delete the legacy fallback branch in Ibis connect_default()** - -In `src/mountainash_data/backends/ibis/connection.py`, modify `connect_default()` to: - -1. Remove the `isinstance(obj_settings, ConnectionProfile)` guard — call `to_driver_kwargs()` unconditionally. -2. Remove the local import of `ConnectionProfile` (line 112). -3. Delete the entire legacy branch (lines 142-164): - ```python - # DELETE everything from here... - # Legacy path for BaseDBAuthSettings subclasses - connection_string_template = self.get_connection_string_template(...) - ... - elif self.ibis_connection_mode == IBIS_DB_CONNECTION_MODE.HYBRID: - self._connect(...) - - #TODO: Add check and logging - if self.ibis_backend is None: - raise Exception(...) - - return self.ibis_backend - # ...to here - ``` - -The resulting `connect_default()` method should be: - -```python -def connect_default(self, **kwargs) -> SQLBackend: - """Connect using default configuration""" - - if self.ibis_backend is None: - - settings_class = self.db_auth_settings_parameters.settings_class - if settings_class is not None: - obj_settings = settings_class.get_settings( - settings_parameters=self.db_auth_settings_parameters - ) - driver_kwargs = obj_settings.to_driver_kwargs() - # Filter out empty lists/sequences that some drivers reject. - # (e.g. ibis.duckdb.connect() does not accept extensions=[]) - driver_kwargs = {k: v for k, v in driver_kwargs.items() - if not (isinstance(v, (list, tuple)) and len(v) == 0)} - driver_kwargs.update(kwargs) - # Use the ibis dialect string to call ibis..connect(**driver_kwargs) - descriptor = getattr(obj_settings, "__descriptor__", None) - ibis_dialect = descriptor.ibis_dialect if descriptor else None - if ibis_dialect: - dialect_backend = getattr(ibis, ibis_dialect, None) - if dialect_backend is not None: - self._ibis_backend = dialect_backend.connect(**driver_kwargs) - if self.ibis_backend is None: - raise Exception(f"Unable to establish default connection to {self.db_backend_name}") - return self.ibis_backend - # Fallback: build KWARGS-mode connection - self._connect( - connection_string=self.connection_string_scheme, - connection_kwargs=driver_kwargs if driver_kwargs else None, - ) - if self.ibis_backend is None: - raise Exception(f"Unable to establish default connection to {self.db_backend_name}") - return self.ibis_backend - - return self.ibis_backend -``` - -Also remove the now-unused `IBIS_DB_CONNECTION_MODE` import from line 12: -```python -# BEFORE: -from mountainash_data.core.constants import ( - IBIS_DB_CONNECTION_MODE, - CONST_DB_ABSTRACTION_LAYER, - CONST_DB_PROVIDER_TYPE, - CONST_DB_BACKEND as _CONST_DB_BACKEND, -) -# AFTER: -from mountainash_data.core.constants import ( - CONST_DB_ABSTRACTION_LAYER, - CONST_DB_PROVIDER_TYPE, - CONST_DB_BACKEND as _CONST_DB_BACKEND, -) -``` - -**Wait** — `IBIS_DB_CONNECTION_MODE` is still used by all concrete subclasses' `ibis_connection_mode` property defaults (e.g. line 233, 272, etc.). Keep the import. The abstract property `ibis_connection_mode` and its concrete implementations are now dead code, but removing them from 12+ subclasses is a separate cleanup — note for follow-up. - -- [ ] **Step 4: Delete the isinstance dispatch in Iceberg connect_default()** - -In `src/mountainash_data/backends/iceberg/connection.py`, modify `connect_default()`. - -Replace lines 107-118: -```python - def connect_default(self, **kwargs: t.Any) -> Catalog: - """Connect using credentials from the configured settings class.""" - if self.catalog_backend is None: - settings_class = self.db_auth_settings_parameters.settings_class - if settings_class is None: - raise ValueError("Settings class is required for the database connection") - obj_settings = settings_class.get_settings(settings_parameters=self.db_auth_settings_parameters) - from mountainash_data.core.settings import ConnectionProfile - if isinstance(obj_settings, ConnectionProfile): - connection_kwargs = obj_settings.to_driver_kwargs() - else: - connection_kwargs = obj_settings.get_connection_kwargs() - self._catalog_backend: RestCatalog = RestCatalog(**connection_kwargs) - return self.catalog_backend -``` - -With: -```python - def connect_default(self, **kwargs: t.Any) -> Catalog: - """Connect using credentials from the configured settings class.""" - if self.catalog_backend is None: - settings_class = self.db_auth_settings_parameters.settings_class - if settings_class is None: - raise ValueError("Settings class is required for the database connection") - obj_settings = settings_class.get_settings(settings_parameters=self.db_auth_settings_parameters) - connection_kwargs = obj_settings.to_driver_kwargs() - self._catalog_backend: RestCatalog = RestCatalog(**connection_kwargs) - return self.catalog_backend -``` - -- [ ] **Step 5: Run tests** - -Run: `hatch run test:test-quick` -Expected: All tests pass. - -- [ ] **Step 6: Commit** - -```bash -git add src/mountainash_data/core/connection.py src/mountainash_data/backends/ibis/connection.py src/mountainash_data/backends/iceberg/connection.py -git commit -m "chore(connection): remove deprecated bridge methods and legacy fallback branches" -``` - ---- - -### Task 2: Remove redundant SecretStr guard - -**Files:** -- Modify: `src/mountainash_data/core/settings/profile.py:13,78-80` - -- [ ] **Step 1: Replace isinstance guard with None check** - -In `src/mountainash_data/core/settings/profile.py`, replace lines 78-80: - -```python - pw = getattr(auth, "password", None) - if isinstance(pw, SecretStr): - url += ":" + quote(pw.get_secret_value(), safe="") -``` - -With: -```python - pw = getattr(auth, "password", None) - if pw is not None: - url += ":" + quote(pw.get_secret_value(), safe="") -``` - -- [ ] **Step 2: Remove the SecretStr import** - -Delete line 13: -```python -from pydantic import SecretStr -``` - -- [ ] **Step 3: Run tests** - -Run: `hatch run test:test-quick` -Expected: All tests pass. - -- [ ] **Step 4: Commit** - -```bash -git add src/mountainash_data/core/settings/profile.py -git commit -m "chore(settings): replace SecretStr isinstance guard with None check" -``` - ---- - -## Follow-up (not in scope) - -After the legacy branch in `connect_default()` is removed, the following become dead code: - -- `BaseIbisConnection.ibis_connection_mode` abstract property -- `BaseIbisConnection.connection_string_scheme` abstract property (partially — still used on line 135 as kwargs-mode fallback) -- All 12 concrete subclass implementations of `ibis_connection_mode` -- `IBIS_DB_CONNECTION_MODE` enum values `CONNECTION_STRING`, `KWARGS`, `HYBRID` (still imported by subclasses) - -These should be tracked as a separate backlog item — removing abstract properties from 12+ subclasses warrants its own PR. - -## Backlog update - -After merge, update `mountainash-central/01.principles/mountainash-data/f.backlog/legacy-cleanup.md`: -- `BaseDBConnection deprecated bridge methods` → **RESOLVED** with PR reference -- `ConnectionProfile._default_driver_kwargs SecretStr guard` → **RESOLVED** with PR reference diff --git a/docs/superpowers/plans/2026-04-27-backend-as-single-handle.md b/docs/superpowers/plans/2026-04-27-backend-as-single-handle.md deleted file mode 100644 index d8ea677..0000000 --- a/docs/superpowers/plans/2026-04-27-backend-as-single-handle.md +++ /dev/null @@ -1,1636 +0,0 @@ -# Backend as Single Handle — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Make `IbisBackend` the single public handle for all ibis interaction — lifecycle, inspection, and operations — then delete all legacy factories, utils, and class hierarchies. - -**Architecture:** `IbisBackend` composes an internal `IbisConnection` for inspection, delegates thin wrapper operations to the raw ibis connection, and dispatches per-dialect operations (upsert, indexes) via callable hooks on `DialectSpec`. Fluent methods return `self`; terminal methods return data. - -**Tech Stack:** Python 3.12, ibis-framework 10.4.0, pytest 8.3.5, hatch - -**Spec:** `docs/superpowers/specs/2026-04-27-backend-as-single-handle-design.md` -**Branch:** `feature/settings-aware-ibis-backend` (continuing from Phase 1) -**Test command:** `hatch run test:test-quick` - ---- - -## File Structure - -### Modified - -| File | Responsibility | -|------|----------------| -| `src/mountainash_data/core/protocol.py` | Updated `Backend` protocol (connect→Self, context manager, inspection). Remove `Connection` protocol. | -| `src/mountainash_data/backends/ibis/backend.py` | Lifecycle (`connect`/`close`/context manager), accessor methods, thin wrappers, hook-dispatched operations | -| `src/mountainash_data/backends/ibis/dialects/_registry.py` | New hook fields on `DialectSpec`, wiring to standalone functions | -| `src/mountainash_data/backends/ibis/operations.py` | Extract mixin methods to standalone hook functions. Delete class hierarchy (keep functions only). | -| `src/mountainash_data/backends/iceberg/backend.py` | `connect()` returns `Self`, add `__enter__`/`__exit__`/`close()` | -| `src/mountainash_data/__init__.py` | Remove factory/utils/Connection exports | -| `tests/test_unit/backends/ibis/test_backend.py` | Expand with lifecycle + operations tests | -| `tests/test_unit/test_mountainash_data.py` | Update import assertions for new public API | -| `tests/test_unit/databases/settings/test_settings_parametrized.py` | Replace factory/utils calls with `IbisBackend` | -| `tests/test_integration/test_end_to_end_workflows.py` | Rewrite to use `IbisBackend` | - -### Deleted - -| File/Directory | Reason | -|----------------|--------| -| `src/mountainash_data/core/factories/` | Entire directory — all factories replaced | -| `src/mountainash_data/core/utils.py` | `DatabaseUtils` replaced by `IbisBackend` | -| `src/mountainash_data/backends/ibis/connection.py` | `BaseIbisConnection` + 12 subclasses replaced | -| `tests/test_unit/factories/` | All factory tests | -| `tests/test_unit/test_database_utils.py` | `DatabaseUtils` tests | -| `tests/test_unit/databases/test_database_connections.py` | Legacy connection tests | -| `tests/test_unit/databases/connections/` | Legacy connection lifecycle tests | -| `tests/test_unit/databases/test_ibis_backends.py` | Legacy ibis backend tests | -| `tests/test_unit/databases/operations/` | Legacy operations tests (rewritten in test_backend.py) | - -### Kept (no changes) - -| File | Reason | -|------|--------| -| `src/mountainash_data/core/connection.py` | Iceberg depends on `BaseDBConnection` | -| `src/mountainash_data/core/inspection.py` | Unchanged | -| `src/mountainash_data/core/settings/` | Unchanged | -| `src/mountainash_data/backends/ibis/inspect.py` | Unchanged | - ---- - -### Task 1: Update Backend protocol and add lifecycle to IbisBackend - -**Files:** -- Modify: `src/mountainash_data/core/protocol.py` -- Modify: `src/mountainash_data/backends/ibis/backend.py` -- Test: `tests/test_unit/backends/ibis/test_backend.py` - -- [ ] **Step 1: Write failing lifecycle tests** - -Add these tests to `tests/test_unit/backends/ibis/test_backend.py`: - -```python -# --------------------------------------------------------------------------- -# Lifecycle -# --------------------------------------------------------------------------- - -def test_connect_returns_self(): - """connect() must return the backend instance itself.""" - backend = IbisBackend(dialect="sqlite", database=":memory:") - result = backend.connect() - assert result is backend - - -def test_close_returns_self(): - """close() must return the backend instance itself.""" - backend = IbisBackend(dialect="sqlite", database=":memory:") - backend.connect() - result = backend.close() - assert result is backend - - -def test_context_manager(): - """with IbisBackend(...) as backend: must connect and close.""" - with IbisBackend(dialect="sqlite", database=":memory:") as backend: - assert backend.list_tables() == [] - # After exit, should be closed - with pytest.raises(RuntimeError, match="not connected"): - backend.list_tables() - - -def test_double_close_is_idempotent(): - """Calling close() twice must not raise.""" - backend = IbisBackend(dialect="sqlite", database=":memory:") - backend.connect() - backend.close() - backend.close() # Must not raise - - -def test_use_before_connect_raises(): - """Calling methods before connect() must raise RuntimeError.""" - backend = IbisBackend(dialect="sqlite", database=":memory:") - with pytest.raises(RuntimeError, match="not connected"): - backend.list_tables() - - -def test_use_after_close_raises(): - """Calling methods after close() must raise RuntimeError.""" - backend = IbisBackend(dialect="sqlite", database=":memory:") - backend.connect() - backend.close() - with pytest.raises(RuntimeError, match="not connected"): - backend.list_tables() - - -def test_ibis_connection_accessor(): - """ibis_connection() returns the raw ibis backend object.""" - with IbisBackend(dialect="sqlite", database=":memory:") as backend: - raw = backend.ibis_connection() - assert hasattr(raw, "list_tables") - - -def test_ibis_connection_before_connect_raises(): - """ibis_connection() before connect() must raise RuntimeError.""" - backend = IbisBackend(dialect="sqlite", database=":memory:") - with pytest.raises(RuntimeError, match="not connected"): - backend.ibis_connection() - - -def test_get_connection_accessor(): - """get_connection() returns our IbisConnection wrapper.""" - from mountainash_data.backends.ibis.backend import IbisConnection - with IbisBackend(dialect="sqlite", database=":memory:") as backend: - conn = backend.get_connection() - assert isinstance(conn, IbisConnection) -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `hatch run test:test-target tests/test_unit/backends/ibis/test_backend.py::test_connect_returns_self -v` -Expected: FAIL (connect() returns IbisConnection, not self) - -- [ ] **Step 3: Update Backend protocol** - -Replace the entire contents of `src/mountainash_data/core/protocol.py` with: - -```python -"""Backend protocol. - -This is the structural contract every backend implementation must -satisfy. Implementations are plain classes — there is no inheritance. -""" - -from __future__ import annotations - -import typing as t - -from mountainash_data.core.inspection import ( - CatalogInfo, - NamespaceInfo, - TableInfo, -) - - -@t.runtime_checkable -class Backend(t.Protocol): - """The single handle for interacting with a backend service. - - Backends are constructed with config, connected via connect(), - used for inspection and operations, then closed. - """ - - name: str - - def connect(self) -> t.Self: ... - def close(self) -> t.Self: ... - def __enter__(self) -> t.Self: ... - def __exit__(self, *args: t.Any) -> None: ... - - def list_tables(self, namespace: str | None = None) -> list[str]: ... - def list_namespaces(self) -> list[str]: ... - - def inspect_table( - self, name: str, namespace: str | None = None - ) -> TableInfo: ... - - def inspect_namespace(self, name: str) -> NamespaceInfo: ... - def inspect_catalog(self) -> CatalogInfo: ... -``` - -- [ ] **Step 4: Add lifecycle and accessor methods to IbisBackend** - -In `src/mountainash_data/backends/ibis/backend.py`, make these changes: - -4a. Add `_conn: IbisConnection | None = None` initialisation to each `_init_from_*` method (set `self._conn = None`). - -4b. Replace the existing `connect()` method and add `close()`, `__enter__`, `__exit__`, accessor methods, and a `_require_connected` helper. Replace everything from `def connect(self)` to end of file with: - -```python - def _require_connected(self) -> IbisConnection: - if self._conn is None: - raise RuntimeError( - "IbisBackend is not connected. Call connect() first." - ) - return self._conn - - def connect(self) -> IbisBackend: - """Build a live ibis connection. Returns self for fluent chaining.""" - if self._conn is not None: - return self - if self._spec.connection_builder is None: - raise NotImplementedError( - f"Dialect {self.dialect!r} has no connection_builder configured" - ) - if self._url is not None: - import ibis - ibis_conn = ibis.connect(self._url, **self._config) - else: - cleaned_config = { - k: v for k, v in self._config.items() - if not (isinstance(v, (list, tuple)) and len(v) == 0) - } - ibis_conn = self._spec.connection_builder(**cleaned_config) - self._conn = IbisConnection(ibis_conn, self._spec) - return self - - def close(self) -> IbisBackend: - """Release the connection. Idempotent. Returns self.""" - if self._conn is not None: - self._conn.close() - self._conn = None - return self - - def __enter__(self) -> IbisBackend: - self.connect() - return self - - def __exit__(self, *args: t.Any) -> None: - self.close() - - def ibis_connection(self) -> t.Any: - """Return the raw ibis backend object.""" - return self._require_connected()._ibis_conn - - def get_connection(self) -> IbisConnection: - """Return the internal IbisConnection wrapper.""" - return self._require_connected() - - # --- Inspection (terminal — delegates to IbisConnection) --- - - def list_tables(self, namespace: str | None = None) -> list[str]: - return self._require_connected().list_tables(namespace=namespace) - - def list_namespaces(self) -> list[str]: - return self._require_connected().list_namespaces() - - def inspect_table( - self, name: str, namespace: str | None = None - ) -> TableInfo: - return self._require_connected().inspect_table(name, namespace=namespace) - - def inspect_namespace(self, name: str) -> NamespaceInfo: - return self._require_connected().inspect_namespace(name) - - def inspect_catalog(self) -> CatalogInfo: - return self._require_connected().inspect_catalog() -``` - -4c. In each `_init_from_*` method, add `self._conn = None` after setting `self._config`: -- `_init_from_dialect`: after `self._config = config`, add `self._conn = None` -- `_init_from_url`: after `self._config = config`, add `self._conn = None` -- `_init_from_settings`: after `self._config = driver_kwargs`, add `self._conn = None` - -- [ ] **Step 5: Update existing tests that use the old connect() return** - -In `tests/test_unit/backends/ibis/test_backend.py`, update `test_in_memory_sqlite_connect_and_inspect` — it currently does `conn = backend.connect()` expecting an `IbisConnection`. Change it to use the backend directly: - -```python -def test_in_memory_sqlite_connect_and_inspect(): - """End-to-end test with the only dialect that needs no external service.""" - backend = IbisBackend(dialect="sqlite", database=":memory:") - backend.connect() - try: - assert backend.list_tables() == [] - finally: - backend.close() -``` - -Update the settings path tests similarly: - -```python -def test_settings_path_sqlite(): - """Construct IbisBackend from SQLite SettingsParameters and connect.""" - from mountainash_settings import SettingsParameters - from mountainash_data.core.settings import SQLiteAuthSettings, NoAuth - - params = SettingsParameters.create( - settings_class=SQLiteAuthSettings, - DATABASE=":memory:", - auth=NoAuth(), - ) - backend = IbisBackend(params) - assert backend.dialect == "sqlite" - backend.connect() - tables = backend.list_tables() - assert isinstance(tables, list) - backend.close() - - -def test_settings_path_duckdb_empty_extensions(): - """DuckDB settings with default EXTENSIONS=[] must not crash ibis.""" - from mountainash_settings import SettingsParameters - from mountainash_data.core.settings import DuckDBAuthSettings, NoAuth - - params = SettingsParameters.create( - settings_class=DuckDBAuthSettings, - DATABASE=":memory:", - auth=NoAuth(), - ) - backend = IbisBackend(params) - assert backend.dialect == "duckdb" - backend.connect() # Must not raise — empty-list filter active - backend.close() -``` - -Update URL path tests: - -```python -def test_url_path_sqlite(): - """Construct IbisBackend from sqlite:// URL and connect.""" - backend = IbisBackend("sqlite://") - assert backend.dialect == "sqlite" - backend.connect() - backend.close() - - -def test_url_path_duckdb(): - """Construct IbisBackend from duckdb:// URL and connect.""" - backend = IbisBackend("duckdb://") - assert backend.dialect == "duckdb" - backend.connect() - backend.close() - - -def test_url_path_preserves_database(tmp_path): - """URL database component must reach the driver, not be discarded.""" - db_file = tmp_path / "test.db" - backend = IbisBackend(f"sqlite:///{db_file}") - assert backend.dialect == "sqlite" - backend.connect() - backend.close() - assert db_file.exists() -``` - -- [ ] **Step 6: Run tests to verify they pass** - -Run: `hatch run test:test-target tests/test_unit/backends/ibis/test_backend.py -v` -Expected: ALL PASS - -- [ ] **Step 7: Commit** - -```bash -git add src/mountainash_data/core/protocol.py src/mountainash_data/backends/ibis/backend.py tests/test_unit/backends/ibis/test_backend.py -git commit -m "feat(backend): add lifecycle, context manager, and accessor methods to IbisBackend - -connect() and close() return self for fluent chaining. Context manager -support via __enter__/__exit__. Inspection methods delegate to internal -IbisConnection. Backend protocol updated: connect() returns Self, -Connection protocol removed." -``` - ---- - -### Task 2: Expand DialectSpec with operation hooks and extract standalone functions - -**Files:** -- Modify: `src/mountainash_data/backends/ibis/dialects/_registry.py` -- Modify: `src/mountainash_data/backends/ibis/operations.py` -- Test: `tests/test_unit/backends/ibis/test_backend.py` - -- [ ] **Step 1: Write failing test for hook presence on DialectSpec** - -Add to `tests/test_unit/backends/ibis/test_backend.py`: - -```python -# --------------------------------------------------------------------------- -# DialectSpec hooks -# --------------------------------------------------------------------------- - -def test_duckdb_dialect_has_upsert_hook(): - """DuckDB DialectSpec must have upsert_hook wired.""" - spec = DIALECTS["duckdb"] - assert spec.upsert_hook is not None - - -def test_sqlite_dialect_has_create_index_hook(): - """SQLite DialectSpec must have create_index_hook wired.""" - spec = DIALECTS["sqlite"] - assert spec.create_index_hook is not None - - -def test_postgres_dialect_has_no_upsert_hook(): - """Postgres DialectSpec has no upsert_hook (not DuckDB family).""" - spec = DIALECTS["postgres"] - assert spec.upsert_hook is None -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `hatch run test:test-target tests/test_unit/backends/ibis/test_backend.py::test_duckdb_dialect_has_upsert_hook -v` -Expected: FAIL (DialectSpec has no `upsert_hook` attribute) - -- [ ] **Step 3: Add hook fields to DialectSpec** - -In `src/mountainash_data/backends/ibis/dialects/_registry.py`, add new type aliases and fields. - -After the existing type aliases (line 27), add: - -```python -UpsertHook = t.Callable[..., None] -CreateIndexHook = t.Callable[..., None] -DropIndexHook = t.Callable[..., None] -RenameTableHook = t.Callable[..., None] -``` - -Add new fields to the `DialectSpec` dataclass, after `get_list_indexes_sql` and before `extras`: - -```python - upsert_hook: t.Optional[UpsertHook] = None - create_index_hook: t.Optional[CreateIndexHook] = None - drop_index_hook: t.Optional[DropIndexHook] = None - rename_table_hook: t.Optional[RenameTableHook] = None -``` - -- [ ] **Step 4: Extract standalone hook functions from operations.py** - -In `src/mountainash_data/backends/ibis/operations.py`, convert the `_DuckDBFamilyOperationsMixin` class methods into standalone functions. Add these after the existing per-dialect SQL functions (after `motherduck_list_tables`, around line 213) and before the `_DuckDBFamilyOperationsMixin` class: - -```python -# =========================================================================== -# STANDALONE HOOK FUNCTIONS -# Extracted from _DuckDBFamilyOperationsMixin for DialectSpec wiring. -# =========================================================================== - -def duckdb_family_create_index( - ibis_conn: t.Any, - table_name: str, - columns: list[str] | str, - *, - index_name: str | None = None, - unique: bool = False, - index_type: str | None = None, - where_condition: str | None = None, - database: str | None = None, - if_not_exists: bool = True, -) -> None: - """Create an index using DuckDB/SQLite syntax.""" - columns_list = _normalize_columns(columns) - - if index_name is None: - index_name = _generate_index_name(table_name, columns_list, unique=unique) - - qualified_table = _format_qualified_table(table_name, database=database) - columns_sql = ", ".join(columns_list) - - unique_sql = "UNIQUE " if unique else "" - if_not_exists_sql = "IF NOT EXISTS " if if_not_exists else "" - where_sql = f" WHERE {where_condition}" if where_condition else "" - - if index_type and index_type != CONST_INDEX_TYPE.BTREE: - warnings.warn( - f"Index type {index_type} not supported, using default BTREE" - ) - - create_sql = ( - f"CREATE {unique_sql}INDEX {if_not_exists_sql}{index_name} " - f"ON {qualified_table} ({columns_sql}){where_sql}" - ) - - with contextlib.closing(ibis_conn.con.cursor()) as cur: - cur.execute(create_sql) - - -def duckdb_family_drop_index( - ibis_conn: t.Any, - index_name: str, - *, - table_name: str | None = None, - database: str | None = None, - if_exists: bool = True, -) -> None: - """Drop an index using DuckDB/SQLite syntax.""" - if_exists_sql = "IF EXISTS " if if_exists else "" - drop_sql = f"DROP INDEX {if_exists_sql}{index_name}" - - with contextlib.closing(ibis_conn.con.cursor()) as cur: - cur.execute(drop_sql) - - -def duckdb_family_upsert( - ibis_conn: t.Any, - table_name: str, - df: t.Any, - *, - conflict_columns: list[str] | str, - update_columns: list[str] | str | None = None, - conflict_action: str = CONST_CONFLICT_ACTION.UPDATE, - update_condition: str | None = None, - database: str | None = None, - schema: str | None = None, -) -> None: - """Perform upsert using INSERT ... ON CONFLICT syntax (DuckDB/SQLite).""" - conflict_cols = _normalize_columns(conflict_columns) - all_columns = ma.relation(df).columns - - if update_columns is None: - update_cols = [col for col in all_columns if col not in conflict_cols] - else: - update_cols = _normalize_columns(update_columns) - - tables = ibis_conn.list_tables() - if table_name not in tables: - raise ValueError(f"Target table '{table_name}' does not exist") - - if conflict_action not in [CONST_CONFLICT_ACTION.UPDATE, CONST_CONFLICT_ACTION.NOTHING]: - raise ValueError( - f"conflict_action must be '{CONST_CONFLICT_ACTION.UPDATE}' or " - f"'{CONST_CONFLICT_ACTION.NOTHING}', got '{conflict_action}'" - ) - - if conflict_action == CONST_CONFLICT_ACTION.NOTHING: - if update_cols or update_condition: - warnings.warn( - "update_columns and update_condition are ignored when " - "conflict_action='NOTHING'" - ) - - staging_table = f"temp_upsert_{uuid.uuid4().hex[:8]}" - qualified_table = _format_qualified_table(table_name, database=database, schema=schema) - - all_cols_sql = ", ".join(all_columns) - conflict_cols_sql = ", ".join(conflict_cols) - - if conflict_action == CONST_CONFLICT_ACTION.UPDATE: - if not update_cols: - raise ValueError( - "No columns to update. Either provide update_columns or ensure " - "dataframe has columns beyond conflict_columns" - ) - update_set_sql = ", ".join([f"{col} = EXCLUDED.{col}" for col in update_cols]) - where_sql = f" WHERE {update_condition}" if update_condition else "" - on_conflict_sql = ( - f"ON CONFLICT ({conflict_cols_sql}) DO UPDATE SET {update_set_sql}{where_sql}" - ) - else: - on_conflict_sql = f"ON CONFLICT ({conflict_cols_sql}) DO NOTHING" - - upsert_sql = f""" - INSERT INTO {qualified_table} ({all_cols_sql}) - SELECT {all_cols_sql} FROM {staging_table} - WHERE true - {on_conflict_sql} - """ - - if hasattr(ibis_conn.con, 'register'): - with contextlib.closing(ibis_conn.con.cursor()) as cur: - cur.execute("BEGIN TRANSACTION") - cur.register(staging_table, df) - cur.execute(upsert_sql) - cur.unregister(staging_table) - cur.execute("COMMIT") - else: - ibis_conn.create_table(staging_table, df, temp=True, overwrite=True) - try: - with contextlib.closing(ibis_conn.con.cursor()) as cur: - cur.execute(upsert_sql) - ibis_conn.con.commit() - finally: - try: - ibis_conn.drop_table(staging_table, force=True) - except Exception: - pass -``` - -- [ ] **Step 5: Wire hooks in DIALECTS registry** - -In `src/mountainash_data/backends/ibis/dialects/_registry.py`, add imports for the new hook functions alongside the existing SQL function imports: - -```python -from mountainash_data.backends.ibis.operations import ( # noqa: E402 - duckdb_get_index_exists_sql, - duckdb_get_list_indexes_sql, - sqlite_get_index_exists_sql, - sqlite_get_list_indexes_sql, - motherduck_get_index_exists_sql, - motherduck_get_list_indexes_sql, - duckdb_family_upsert, - duckdb_family_create_index, - duckdb_family_drop_index, -) -``` - -Then add hook fields to the `sqlite`, `duckdb`, and `motherduck` entries in `DIALECTS`: - -For `"sqlite"`: -```python - upsert_hook=duckdb_family_upsert, - create_index_hook=duckdb_family_create_index, - drop_index_hook=duckdb_family_drop_index, -``` - -For `"duckdb"`: -```python - upsert_hook=duckdb_family_upsert, - create_index_hook=duckdb_family_create_index, - drop_index_hook=duckdb_family_drop_index, -``` - -For `"motherduck"`: -```python - upsert_hook=duckdb_family_upsert, - create_index_hook=duckdb_family_create_index, - drop_index_hook=duckdb_family_drop_index, -``` - -- [ ] **Step 6: Run tests** - -Run: `hatch run test:test-target tests/test_unit/backends/ibis/test_backend.py -v` -Expected: ALL PASS - -- [ ] **Step 7: Commit** - -```bash -git add src/mountainash_data/backends/ibis/dialects/_registry.py src/mountainash_data/backends/ibis/operations.py tests/test_unit/backends/ibis/test_backend.py -git commit -m "feat(registry): expand DialectSpec with operation hooks - -Add upsert_hook, create_index_hook, drop_index_hook, rename_table_hook -to DialectSpec. Extract standalone hook functions from -_DuckDBFamilyOperationsMixin. Wire DuckDB/SQLite/MotherDuck entries." -``` - ---- - -### Task 3: Add thin wrapper and hook-dispatched operations to IbisBackend - -**Files:** -- Modify: `src/mountainash_data/backends/ibis/backend.py` -- Test: `tests/test_unit/backends/ibis/test_backend.py` - -- [ ] **Step 1: Write failing tests for thin wrapper operations** - -Add to `tests/test_unit/backends/ibis/test_backend.py`: - -```python -import polars as pl - -# --------------------------------------------------------------------------- -# Thin wrapper operations (fluent) -# --------------------------------------------------------------------------- - -def test_create_table_returns_self(): - """create_table() must return self for fluent chaining.""" - with IbisBackend(dialect="sqlite", database=":memory:") as backend: - result = backend.create_table("t", {"id": [1, 2]}) - assert result is backend - assert "t" in backend.list_tables() - - -def test_drop_table_returns_self(): - """drop_table() must return self.""" - with IbisBackend(dialect="sqlite", database=":memory:") as backend: - backend.create_table("t", {"id": [1]}) - result = backend.drop_table("t") - assert result is backend - assert "t" not in backend.list_tables() - - -def test_insert_returns_self(): - """insert() must return self.""" - with IbisBackend(dialect="duckdb", database=":memory:") as backend: - backend.create_table("t", {"id": [1]}) - result = backend.insert("t", {"id": [2]}) - assert result is backend - - -def test_truncate_returns_self(): - """truncate() must return self.""" - with IbisBackend(dialect="duckdb", database=":memory:") as backend: - backend.create_table("t", {"id": [1]}) - result = backend.truncate("t") - assert result is backend - - -def test_table_returns_ibis_table(): - """table() must return an ibis table expression.""" - with IbisBackend(dialect="sqlite", database=":memory:") as backend: - backend.create_table("t", {"id": [1, 2]}) - tbl = backend.table("t") - assert tbl is not None - - -def test_run_sql_returns_result(): - """run_sql() must return an ibis table expression.""" - with IbisBackend(dialect="sqlite", database=":memory:") as backend: - backend.create_table("t", {"id": [1, 2, 3]}) - result = backend.run_sql("SELECT COUNT(*) as cnt FROM t") - assert result is not None - - -def test_table_exists_returns_bool(): - """table_exists() must return True/False.""" - with IbisBackend(dialect="sqlite", database=":memory:") as backend: - assert backend.table_exists("nope") is False - backend.create_table("t", {"id": [1]}) - assert backend.table_exists("t") is True - - -def test_fluent_chaining(): - """Multiple fluent calls can be chained.""" - with IbisBackend(dialect="sqlite", database=":memory:") as backend: - backend.create_table("a", {"id": [1]}).create_table("b", {"id": [2]}) - assert sorted(backend.list_tables()) == ["a", "b"] -``` - -- [ ] **Step 2: Write failing tests for hook-dispatched operations** - -Add to `tests/test_unit/backends/ibis/test_backend.py`: - -```python -# --------------------------------------------------------------------------- -# Hook-dispatched operations -# --------------------------------------------------------------------------- - -def test_create_index_returns_self(): - """create_index() via hook must return self.""" - with IbisBackend(dialect="sqlite", database=":memory:") as backend: - backend.create_table("t", {"id": [1], "name": ["a"]}) - result = backend.create_index("t", ["name"]) - assert result is backend - - -def test_create_unique_index_returns_self(): - """create_unique_index() must return self.""" - with IbisBackend(dialect="sqlite", database=":memory:") as backend: - backend.create_table("t", {"id": [1], "name": ["a"]}) - result = backend.create_unique_index("t", ["name"]) - assert result is backend - - -def test_drop_index_returns_self(): - """drop_index() via hook must return self.""" - with IbisBackend(dialect="sqlite", database=":memory:") as backend: - backend.create_table("t", {"id": [1], "name": ["a"]}) - backend.create_index("t", ["name"], index_name="idx_name") - result = backend.drop_index("idx_name") - assert result is backend - - -def test_index_exists(): - """index_exists() must detect created indexes.""" - with IbisBackend(dialect="sqlite", database=":memory:") as backend: - backend.create_table("t", {"id": [1]}) - backend.create_index("t", ["id"], index_name="idx_id") - assert backend.index_exists("idx_id") is True - assert backend.index_exists("no_such_idx") is False - - -def test_list_indexes(): - """list_indexes() must return index info.""" - with IbisBackend(dialect="sqlite", database=":memory:") as backend: - backend.create_table("t", {"id": [1], "name": ["a"]}) - backend.create_index("t", ["id"], index_name="idx_id") - indexes = backend.list_indexes("t") - assert isinstance(indexes, list) - assert len(indexes) >= 1 - - -def test_upsert_duckdb(): - """upsert() must work on DuckDB via hook.""" - with IbisBackend(dialect="duckdb", database=":memory:") as backend: - initial = pl.DataFrame({"id": [1, 2], "val": [10, 20]}) - backend.create_table("t", initial) - backend.create_unique_index("t", ["id"]) - - update = pl.DataFrame({"id": [2, 3], "val": [25, 30]}) - result = backend.upsert("t", update, conflict_columns=["id"]) - assert result is backend - - count_result = backend.run_sql("SELECT COUNT(*) as cnt FROM t") - count = count_result.to_polars()["cnt"][0] - assert count == 3 - - -def test_upsert_unsupported_dialect_raises(): - """upsert() on a dialect without upsert_hook must raise NotImplementedError.""" - backend = IbisBackend(dialect="postgres") - # Don't connect (can't connect to postgres anyway) — just check the method - backend._conn = type("FakeConn", (), {"_ibis_conn": None, "_dialect_spec": DIALECTS["postgres"], "_closed": False})() - with pytest.raises(NotImplementedError, match="does not support upsert"): - backend.upsert("t", {}, conflict_columns=["id"]) -``` - -- [ ] **Step 3: Run tests to verify they fail** - -Run: `hatch run test:test-target tests/test_unit/backends/ibis/test_backend.py::test_create_table_returns_self -v` -Expected: FAIL (IbisBackend has no `create_table` method) - -- [ ] **Step 4: Add thin wrapper operations to IbisBackend** - -In `src/mountainash_data/backends/ibis/backend.py`, add these methods to the `IbisBackend` class after the inspection methods: - -```python - # --- Thin wrapper operations (fluent — return self) --- - - def create_table( - self, - name: str, - obj: t.Any, - *, - schema: t.Any | None = None, - database: str | None = None, - temp: bool = False, - overwrite: bool = False, - ) -> IbisBackend: - conn = self._require_connected() - conn._ibis_conn.create_table( - name, obj=obj, schema=schema, database=database, - temp=temp, overwrite=overwrite, - ) - return self - - def drop_table( - self, - name: str, - *, - database: str | None = None, - force: bool = False, - ) -> IbisBackend: - conn = self._require_connected() - conn._ibis_conn.drop_table(name, database=database, force=force) - return self - - def create_view( - self, - name: str, - obj: t.Any, - *, - database: str | None = None, - overwrite: bool = False, - ) -> IbisBackend: - conn = self._require_connected() - conn._ibis_conn.create_view(name, obj=obj, database=database, overwrite=overwrite) - return self - - def drop_view( - self, - name: str, - *, - database: str | None = None, - force: bool = False, - ) -> IbisBackend: - conn = self._require_connected() - conn._ibis_conn.drop_view(name, database=database, force=force) - return self - - def insert( - self, - name: str, - obj: t.Any, - *, - database: str | None = None, - overwrite: bool = False, - ) -> IbisBackend: - conn = self._require_connected() - conn._ibis_conn.insert(name, obj=obj, database=database, overwrite=overwrite) - return self - - def truncate( - self, - name: str, - *, - database: str | None = None, - schema: str | None = None, - ) -> IbisBackend: - conn = self._require_connected() - conn._ibis_conn.truncate_table(name, schema=schema, database=database) - return self - - def rename_table(self, old_name: str, new_name: str) -> IbisBackend: - if self._spec.rename_table_hook is None: - raise NotImplementedError( - f"Dialect {self.dialect!r} does not support rename_table" - ) - conn = self._require_connected() - self._spec.rename_table_hook(conn._ibis_conn, old_name, new_name) - return self - - # --- Terminal operations (return data) --- - - def table(self, name: str, *, database: str | None = None) -> t.Any: - conn = self._require_connected() - return conn._ibis_conn.table(name, database=database) - - def table_exists( - self, name: str, database: str | None = None - ) -> bool: - tables = self.list_tables() - return name in tables - - def run_sql( - self, - query: str, - *, - schema: t.Any | None = None, - dialect: str | None = None, - ) -> t.Any: - conn = self._require_connected() - return conn._ibis_conn.sql(query, schema=schema, dialect=dialect) - - def run_expr( - self, - expr: t.Any, - *, - params: dict | None = None, - limit: str | None = "default", - **kwargs: t.Any, - ) -> t.Any: - conn = self._require_connected() - return conn._ibis_conn.execute(expr, params=params, limit=limit, **kwargs) - - def to_sql( - self, - expr: t.Any, - *, - params: t.Any = None, - limit: str | None = None, - pretty: bool = False, - **kwargs: t.Any, - ) -> str | None: - conn = self._require_connected() - return conn._ibis_conn.compile(expr, params=params, limit=limit, pretty=pretty, **kwargs) -``` - -- [ ] **Step 5: Add hook-dispatched operations to IbisBackend** - -Continue adding methods in `src/mountainash_data/backends/ibis/backend.py`: - -```python - # --- Hook-dispatched operations (fluent — return self) --- - - def upsert( - self, - name: str, - obj: t.Any, - *, - conflict_columns: list[str] | str, - update_columns: list[str] | str | None = None, - conflict_action: str = "UPDATE", - update_condition: str | None = None, - database: str | None = None, - schema: str | None = None, - ) -> IbisBackend: - if self._spec.upsert_hook is None: - raise NotImplementedError( - f"Dialect {self.dialect!r} does not support upsert" - ) - conn = self._require_connected() - self._spec.upsert_hook( - conn._ibis_conn, name, obj, - conflict_columns=conflict_columns, - update_columns=update_columns, - conflict_action=conflict_action, - update_condition=update_condition, - database=database, - schema=schema, - ) - return self - - def create_index( - self, - table_name: str, - columns: list[str] | str, - *, - index_name: str | None = None, - unique: bool = False, - index_type: str | None = None, - where_condition: str | None = None, - database: str | None = None, - if_not_exists: bool = True, - ) -> IbisBackend: - if self._spec.create_index_hook is None: - raise NotImplementedError( - f"Dialect {self.dialect!r} does not support create_index" - ) - conn = self._require_connected() - self._spec.create_index_hook( - conn._ibis_conn, table_name, columns, - index_name=index_name, unique=unique, index_type=index_type, - where_condition=where_condition, database=database, - if_not_exists=if_not_exists, - ) - return self - - def create_unique_index( - self, - table_name: str, - columns: list[str] | str, - *, - index_name: str | None = None, - where_condition: str | None = None, - database: str | None = None, - ) -> IbisBackend: - return self.create_index( - table_name, columns, - index_name=index_name, unique=True, - where_condition=where_condition, database=database, - ) - - def drop_index( - self, - index_name: str, - *, - table_name: str | None = None, - database: str | None = None, - if_exists: bool = True, - ) -> IbisBackend: - if self._spec.drop_index_hook is None: - raise NotImplementedError( - f"Dialect {self.dialect!r} does not support drop_index" - ) - conn = self._require_connected() - self._spec.drop_index_hook( - conn._ibis_conn, index_name, - table_name=table_name, database=database, if_exists=if_exists, - ) - return self - - def index_exists( - self, - index_name: str, - *, - table_name: str | None = None, - database: str | None = None, - ) -> bool: - if self._spec.get_index_exists_sql is None: - raise NotImplementedError( - f"Dialect {self.dialect!r} does not support index_exists" - ) - conn = self._require_connected() - check_sql = self._spec.get_index_exists_sql(index_name, table_name, database) - result = conn._ibis_conn.sql(check_sql) - if result is None: - return False - import mountainash as ma - count = ma.relation(result).to_dict()["count"][0] - return count > 0 - - def list_indexes( - self, - table_name: str, - *, - database: str | None = None, - ) -> list[dict]: - if self._spec.get_list_indexes_sql is None: - raise NotImplementedError( - f"Dialect {self.dialect!r} does not support list_indexes" - ) - conn = self._require_connected() - list_sql = self._spec.get_list_indexes_sql(table_name, database) - result = conn._ibis_conn.sql(list_sql) - if result is None: - return [] - import mountainash as ma - return ma.relation(result).to_dicts() -``` - -- [ ] **Step 6: Run all backend tests** - -Run: `hatch run test:test-target tests/test_unit/backends/ibis/test_backend.py -v` -Expected: ALL PASS - -- [ ] **Step 7: Commit** - -```bash -git add src/mountainash_data/backends/ibis/backend.py tests/test_unit/backends/ibis/test_backend.py -git commit -m "feat(backend): add thin wrapper and hook-dispatched operations - -Fluent methods (create_table, insert, upsert, create_index, etc.) return -self. Terminal methods (table, run_sql, list_indexes, etc.) return data. -Hook dispatch: upsert/index operations via DialectSpec callables." -``` - ---- - -### Task 4: Update IcebergBackend for protocol compliance - -**Files:** -- Modify: `src/mountainash_data/backends/iceberg/backend.py` -- Test: Run existing iceberg tests - -- [ ] **Step 1: Update IcebergBackend** - -Replace the contents of `src/mountainash_data/backends/iceberg/backend.py` with: - -```python -"""IcebergBackend — implements core.protocol.Backend for iceberg catalogs.""" - -from __future__ import annotations - -import typing as t - -from mountainash_data.backends.iceberg.catalogs.rest import IcebergRestConnection -from mountainash_data.backends.iceberg.connection import IcebergConnectionBase - - -_CATALOG_REGISTRY: dict[str, type[IcebergConnectionBase]] = { - "rest": IcebergRestConnection, -} - - -class IcebergBackend: - """Iceberg backend — single entry point for iceberg catalog interaction. - - Construction takes a catalog type (e.g. ``'rest'``) and config kwargs. - ``connect()`` returns ``self``. Use as a context manager. - """ - - name = "iceberg" - - def __init__(self, catalog: str, **config: t.Any) -> None: - if catalog not in _CATALOG_REGISTRY: - raise KeyError( - f"Unknown iceberg catalog type {catalog!r}. " - f"Available: {sorted(_CATALOG_REGISTRY)}" - ) - self._catalog_cls = _CATALOG_REGISTRY[catalog] - self._config = config - self._conn: IcebergConnectionBase | None = None - - def connect(self) -> IcebergBackend: - """Open a connection. Returns self for fluent chaining.""" - if self._conn is None: - self._conn = self._catalog_cls(**self._config) - return self - - def close(self) -> IcebergBackend: - """Release the connection. Idempotent. Returns self.""" - if self._conn is not None: - if hasattr(self._conn, "close"): - self._conn.close() - self._conn = None - return self - - def __enter__(self) -> IcebergBackend: - self.connect() - return self - - def __exit__(self, *args: t.Any) -> None: - self.close() - - def _require_connected(self) -> IcebergConnectionBase: - if self._conn is None: - raise RuntimeError( - "IcebergBackend is not connected. Call connect() first." - ) - return self._conn - - def list_tables(self, namespace: str | None = None) -> list[str]: - return self._require_connected().list_tables(namespace=namespace) - - def list_namespaces(self) -> list[str]: - return self._require_connected().list_namespaces() - - def inspect_table( - self, name: str, namespace: str | None = None - ) -> t.Any: - return self._require_connected().inspect_table(name, namespace=namespace) - - def inspect_namespace(self, name: str) -> t.Any: - return self._require_connected().inspect_namespace(name) - - def inspect_catalog(self) -> t.Any: - return self._require_connected().inspect_catalog() -``` - -- [ ] **Step 2: Run iceberg tests (if any pass without external services)** - -Run: `hatch run test:test-target tests/test_unit/backends/iceberg/ -v` -Expected: PASS (or skip if pyiceberg not installed) - -- [ ] **Step 3: Commit** - -```bash -git add src/mountainash_data/backends/iceberg/backend.py -git commit -m "feat(iceberg): update IcebergBackend for new Backend protocol - -connect() returns self, context manager support, inspection methods -delegate to internal connection." -``` - ---- - -### Task 5: Delete legacy code and update public API - -**Files:** -- Delete: `src/mountainash_data/core/factories/` (entire directory) -- Delete: `src/mountainash_data/core/utils.py` -- Delete: `src/mountainash_data/backends/ibis/connection.py` -- Modify: `src/mountainash_data/__init__.py` -- Modify: `src/mountainash_data/backends/ibis/operations.py` (delete class hierarchy, keep functions) - -- [ ] **Step 1: Delete factory directory, utils, and legacy connection module** - -```bash -rm -rf src/mountainash_data/core/factories/ -rm src/mountainash_data/core/utils.py -rm src/mountainash_data/backends/ibis/connection.py -``` - -- [ ] **Step 2: Clean up operations.py — delete class hierarchy, keep hook functions** - -In `src/mountainash_data/backends/ibis/operations.py`, delete everything from the `_DuckDBFamilyOperationsMixin` class definition onwards (from line ~221 to end of file). This removes: -- `_DuckDBFamilyOperationsMixin` class -- `_BaseIbisMixin` compatibility shim -- `BaseIbisOperations` abstract class -- All concrete operations subclasses (`DuckDB_IbisOperations`, `SQLite_IbisOperations`, etc.) - -Keep: -- All imports at the top -- All module-level helper functions (`_generate_index_name`, `_format_qualified_table`, `_normalize_columns`) -- All per-dialect SQL functions (`duckdb_get_index_exists_sql`, `sqlite_get_index_exists_sql`, etc.) -- All standalone hook functions (`duckdb_family_create_index`, `duckdb_family_drop_index`, `duckdb_family_upsert`) -- The `motherduck_list_tables` function - -Remove the imports that are only used by the deleted classes. After cleanup, the remaining imports should be: - -```python -import typing as t -import contextlib -import warnings -import uuid - -import mountainash as ma - -from mountainash_data.core.constants import ( - CONST_CONFLICT_ACTION, - CONST_INDEX_TYPE, -) -``` - -Remove: `from abc import abstractmethod, ABC`, `import ibis`, `import ibis.expr.types.relations as ir`, `from ibis.expr.schema import SchemaLike`, `from ibis.backends.sql import SQLBackend`, `from mountainash_settings import SettingsParameters`, `from mountainash_data.core.constants import CONST_DB_BACKEND`. - -- [ ] **Step 3: Update `__init__.py`** - -Replace `src/mountainash_data/__init__.py` with: - -```python -"""mountainash-data: physical access to backend data services. - -Public API: - Backend — protocol (core.protocol) - IbisBackend — ibis-style relational backends (backends.ibis.backend) - IcebergBackend — iceberg-style table-format catalogs (backends.iceberg.backend) - CatalogInfo, NamespaceInfo, TableInfo, ColumnInfo — inspection model -""" - -from mountainash_data.__version__ import __version__ -from mountainash_data.core.protocol import Backend -from mountainash_data.core.inspection import ( - CatalogInfo, - ColumnInfo, - NamespaceInfo, - TableInfo, -) -from mountainash_data.backends.ibis.backend import IbisBackend - -try: - from mountainash_data.backends.iceberg.backend import IcebergBackend -except ImportError: - IcebergBackend = None # type: ignore[assignment,misc] - -__all__ = [ - "__version__", - "Backend", - "CatalogInfo", - "ColumnInfo", - "NamespaceInfo", - "TableInfo", - "IbisBackend", - "IcebergBackend", -] -``` - -- [ ] **Step 4: Verify import works** - -Run: `hatch run test:test-target tests/test_unit/backends/ibis/test_backend.py -v` -Expected: ALL PASS (the backend tests don't import deleted modules) - -- [ ] **Step 5: Commit** - -```bash -git add -A -git commit -m "refactor: delete legacy factories, DatabaseUtils, and connection hierarchies - -Remove ConnectionFactory, OperationsFactory, SettingsFactory, DatabaseUtils, -BaseIbisConnection + 12 subclasses, BaseIbisOperations + concrete subclasses. -Keep BaseDBConnection (Iceberg depends on it). -Update __init__.py to export only Backend, IbisBackend, IcebergBackend, -and inspection model." -``` - ---- - -### Task 6: Delete legacy test files - -**Files:** -- Delete: `tests/test_unit/factories/` -- Delete: `tests/test_unit/test_database_utils.py` -- Delete: `tests/test_unit/databases/test_database_connections.py` -- Delete: `tests/test_unit/databases/connections/` -- Delete: `tests/test_unit/databases/test_ibis_backends.py` -- Delete: `tests/test_unit/databases/operations/` - -- [ ] **Step 1: Delete legacy test files and directories** - -```bash -rm -rf tests/test_unit/factories/ -rm tests/test_unit/test_database_utils.py -rm tests/test_unit/databases/test_database_connections.py -rm -rf tests/test_unit/databases/connections/ -rm tests/test_unit/databases/test_ibis_backends.py -rm -rf tests/test_unit/databases/operations/ -``` - -- [ ] **Step 2: Update test_mountainash_data.py** - -Replace `tests/test_unit/test_mountainash_data.py` with: - -```python -"""Tests for main mountainash_data package.""" - -import pytest -import mountainash_data - - -class TestPackageImports: - """Test package-level imports and structure.""" - - def test_version_import(self): - assert hasattr(mountainash_data, '__version__') - assert isinstance(mountainash_data.__version__, str) - assert len(mountainash_data.__version__) > 0 - - def test_version_format(self): - version_parts = mountainash_data.__version__.split('.') - assert len(version_parts) >= 2 - assert version_parts[0].isdigit() - assert version_parts[1].isdigit() - - def test_core_imports_available(self): - from mountainash_data.core.connection import BaseDBConnection - assert BaseDBConnection is not None - - -class TestPackageStructure: - """Test package structure and organization.""" - - def test_package_has_init(self): - assert hasattr(mountainash_data, '__file__') - - def test_new_submodules_exist(self): - import mountainash_data.core - import mountainash_data.backends - assert hasattr(mountainash_data, 'core') - assert hasattr(mountainash_data, 'backends') - - def test_public_api_ibis_backend(self): - from mountainash_data import IbisBackend - assert IbisBackend is not None - - def test_public_api_backend_protocol(self): - from mountainash_data import Backend - assert Backend is not None - - def test_public_api_inspection_model(self): - from mountainash_data import CatalogInfo, ColumnInfo, NamespaceInfo, TableInfo - assert CatalogInfo is not None - assert ColumnInfo is not None - assert NamespaceInfo is not None - assert TableInfo is not None - - def test_removed_exports_not_available(self): - assert not hasattr(mountainash_data, 'ConnectionFactory') - assert not hasattr(mountainash_data, 'OperationsFactory') - assert not hasattr(mountainash_data, 'SettingsFactory') - assert not hasattr(mountainash_data, 'DatabaseUtils') -``` - -- [ ] **Step 3: Update test_settings_parametrized.py** - -In `tests/test_unit/databases/settings/test_settings_parametrized.py`, find the tests that use `ConnectionFactory` and `DatabaseUtils` (around lines 148-175) and replace them with `IbisBackend` equivalents: - -Replace the `ConnectionFactory` test block (around line 148) with: - -```python - def test_settings_work_with_ibis_backend(self, settings_params): - """Test that settings work with IbisBackend.""" - from mountainash_data.backends.ibis.backend import IbisBackend - backend = IbisBackend(settings_params) - assert backend.dialect is not None -``` - -Replace the `DatabaseUtils` test block (around line 163) with: - -```python - def test_settings_work_with_ibis_backend_connect(self, settings_params): - """Test that settings can create a connected backend via IbisBackend.""" - from mountainash_data.backends.ibis.backend import IbisBackend - backend = IbisBackend(settings_params) - backend.connect() - tables = backend.list_tables() - assert isinstance(tables, list) - backend.close() -``` - -- [ ] **Step 4: Rewrite integration tests** - -Replace `tests/test_integration/test_end_to_end_workflows.py` with: - -```python -"""End-to-end integration tests using IbisBackend.""" - -import pytest -import polars as pl -from mountainash_data.backends.ibis.backend import IbisBackend -from mountainash_data.core.settings import SQLiteAuthSettings, DuckDBAuthSettings -from mountainash_settings import SettingsParameters - - -@pytest.mark.integration -class TestIbisBackendWorkflow: - """Test complete workflows through IbisBackend.""" - - def test_sqlite_dialect_workflow(self): - """Full workflow with dialect= keyword.""" - with IbisBackend(dialect="sqlite", database=":memory:") as backend: - backend.create_table("users", {"id": [1, 2], "name": ["a", "b"]}) - assert "users" in backend.list_tables() - tbl = backend.table("users") - assert tbl is not None - - def test_sqlite_url_workflow(self, tmp_path): - """Full workflow from URL.""" - db_file = tmp_path / "test.db" - with IbisBackend(f"sqlite:///{db_file}") as backend: - backend.create_table("t", {"id": [1, 2, 3]}) - assert "t" in backend.list_tables() - assert db_file.exists() - - def test_duckdb_settings_workflow(self): - """Full workflow from SettingsParameters.""" - from mountainash_data.core.settings import NoAuth - params = SettingsParameters.create( - settings_class=DuckDBAuthSettings, - DATABASE=":memory:", - auth=NoAuth(), - ) - with IbisBackend(params) as backend: - backend.create_table("t", {"id": [1, 2]}) - assert "t" in backend.list_tables() - - def test_fluent_chaining_workflow(self): - """Fluent API chaining across multiple operations.""" - with IbisBackend(dialect="sqlite", database=":memory:") as backend: - ( - backend - .create_table("users", {"id": [1], "email": ["a@b.com"]}) - .create_index("users", ["email"], unique=True) - .create_table("orders", {"id": [1], "user_id": [1]}) - ) - assert sorted(backend.list_tables()) == ["orders", "users"] - - def test_inspect_workflow(self): - """Inspection methods work through the backend.""" - with IbisBackend(dialect="sqlite", database=":memory:") as backend: - backend.create_table("t", {"id": [1], "name": ["a"]}) - info = backend.inspect_table("t") - assert info.name == "t" - assert len(info.columns) == 2 - - def test_ibis_connection_seam(self): - """ibis_connection() provides the seam to mountainash-expressions.""" - with IbisBackend(dialect="sqlite", database=":memory:") as backend: - backend.create_table("t", {"id": [1, 2]}) - raw = backend.ibis_connection() - tbl = raw.table("t") - assert tbl is not None - - -@pytest.mark.integration -class TestDuckDBOperationsWorkflow: - """Test DuckDB-specific operations (upsert, indexes) through IbisBackend.""" - - def test_upsert_insert_new_rows(self): - """Upsert inserts new rows when no conflicts exist.""" - with IbisBackend(dialect="duckdb", database=":memory:") as backend: - initial = pl.DataFrame({"id": [1, 2], "name": ["Alice", "Bob"]}) - backend.create_table("users", initial) - backend.create_unique_index("users", ["id"]) - - new_data = pl.DataFrame({"id": [3, 4], "name": ["Charlie", "Diana"]}) - backend.upsert("users", new_data, conflict_columns=["id"]) - - count = backend.run_sql("SELECT COUNT(*) as cnt FROM users").to_polars()["cnt"][0] - assert count == 4 - - def test_upsert_update_existing_rows(self): - """Upsert updates existing rows on conflict.""" - with IbisBackend(dialect="duckdb", database=":memory:") as backend: - initial = pl.DataFrame({"id": [1, 2], "name": ["Alice", "Bob"], "score": [100, 200]}) - backend.create_table("users", initial) - backend.create_unique_index("users", ["id"]) - - update = pl.DataFrame({"id": [1, 2], "name": ["Alice", "Bob"], "score": [150, 250]}) - backend.upsert("users", update, conflict_columns=["id"]) - - result = backend.run_sql("SELECT score FROM users ORDER BY id").to_polars() - assert list(result["score"]) == [150, 250] - - def test_index_lifecycle(self): - """Create, check, list, drop index.""" - with IbisBackend(dialect="duckdb", database=":memory:") as backend: - backend.create_table("t", {"id": [1, 2], "name": ["a", "b"]}) - - backend.create_index("t", ["name"], index_name="idx_name") - assert backend.index_exists("idx_name") is True - - indexes = backend.list_indexes("t") - assert any(idx.get("name") == "idx_name" for idx in indexes) - - backend.drop_index("idx_name") - assert backend.index_exists("idx_name") is False -``` - -- [ ] **Step 5: Run full test suite** - -Run: `hatch run test:test-quick` -Expected: ALL PASS (some tests will have been removed, count should be lower but zero failures) - -- [ ] **Step 6: Commit** - -```bash -git add -A -git commit -m "test: rewrite all tests to use IbisBackend, delete legacy test files - -Delete factory tests, DatabaseUtils tests, legacy connection tests, -and legacy operations tests. Rewrite integration tests and settings -parametrized tests. Update package structure tests." -``` - ---- - -### Task 7: Full test suite validation and cleanup - -**Files:** -- Verify: All files - -- [ ] **Step 1: Run full test suite** - -Run: `hatch run test:test-quick` -Expected: ALL PASS - -- [ ] **Step 2: Check for any remaining imports of deleted modules** - -```bash -grep -rn "from mountainash_data.core.factories" src/ tests/ --include="*.py" -grep -rn "from mountainash_data.core.utils" src/ tests/ --include="*.py" -grep -rn "from mountainash_data.backends.ibis.connection import" src/ tests/ --include="*.py" -grep -rn "ConnectionFactory\|OperationsFactory\|SettingsFactory\|DatabaseUtils" src/ tests/ --include="*.py" -grep -rn "BaseIbisOperations\|BaseIbisConnection" src/ tests/ --include="*.py" -``` - -Expected: No matches in src/ or tests/ (only in docs/ specs/plans which is fine). - -- [ ] **Step 3: Fix any remaining references found in step 2** - -If any stale imports are found, update them. Common places to check: -- `conftest.py` files -- `__init__.py` files in test directories -- Fixture files in `tests/fixtures/` - -- [ ] **Step 4: Run full test suite one final time** - -Run: `hatch run test:test-quick` -Expected: ALL PASS - -- [ ] **Step 5: Commit any cleanup** - -```bash -git add -A -git commit -m "chore: clean up stale imports and references to deleted modules" -``` - -(Skip this commit if no changes were needed.) - ---- - -## Self-Review - -**Spec coverage check:** -- ✅ IbisBackend lifecycle (connect/close/context manager) — Task 1 -- ✅ Fluent API (return self) — Task 3 -- ✅ Terminal methods (return data) — Task 3 -- ✅ DialectSpec hooks — Task 2 -- ✅ Hook-dispatched operations (upsert, indexes) — Task 3 -- ✅ Thin wrapper operations — Task 3 -- ✅ Accessor methods (ibis_connection, get_connection) — Task 1 -- ✅ Protocol update (Backend, remove Connection) — Task 1 -- ✅ IcebergBackend update — Task 4 -- ✅ Delete factories/utils — Task 5 -- ✅ Delete legacy hierarchies — Task 5 -- ✅ Public API cleanup — Task 5 -- ✅ Test rewrite — Task 6 -- ✅ BaseDBConnection kept — Task 5 (only deletes ibis connection.py, not core/connection.py) -- ✅ Error handling (no silent bool/None, NotImplementedError for unsupported) — Task 3 - -**Placeholder scan:** No TBD/TODO/placeholders found. - -**Type consistency:** `IbisBackend` return type used consistently across all tasks. Hook function signatures match between Task 2 (extraction) and Task 3 (dispatch). diff --git a/docs/superpowers/plans/2026-04-27-settings-aware-backends-to-relation.md b/docs/superpowers/plans/2026-04-27-settings-aware-backends-to-relation.md deleted file mode 100644 index b22f9c8..0000000 --- a/docs/superpowers/plans/2026-04-27-settings-aware-backends-to-relation.md +++ /dev/null @@ -1,432 +0,0 @@ -# Settings-Aware Backends + to_relation() Implementation Plan - -> **Status:** ABANDONED -- superseded by `docs/superpowers/specs/2026-04-27-settings-aware-ibis-backend-design.md`. to_relation() descoped; constructor design revised. - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Make `IbisBackend` accept `SettingsParameters` as an alternative constructor, add `to_relation()` to the Connection protocol and both Ibis connection paths, and declare `mountainash` as an optional dependency. - -**Architecture:** `IbisBackend.__init__` detects whether it received a `str` (dialect) or `SettingsParameters` and resolves config accordingly. `to_relation()` is added to the `Connection` protocol and implemented on `IbisConnection` (new-style) and `BaseIbisConnection` (factory path), both using an import-guarded call to `mountainash.relations.relation()`. Iceberg gets a `NotImplementedError` stub. - -**Tech Stack:** Python, ibis-framework, mountainash-settings, mountainash (optional) - -**Spec:** `docs/superpowers/specs/2026-04-26-to-relation-design.md` - ---- - -## File Map - -| File | Action | What changes | -|------|--------|--------------| -| `src/mountainash_data/backends/ibis/backend.py` | Modify | Settings-aware `__init__` + `to_relation()` on `IbisConnection` | -| `src/mountainash_data/core/protocol.py` | Modify | Add `to_relation()` to `Connection` protocol | -| `src/mountainash_data/backends/ibis/connection.py` | Modify | Add `to_relation()` to `BaseIbisConnection` | -| `src/mountainash_data/backends/iceberg/connection.py` | Modify | Add `to_relation()` stub to `IcebergConnectionBase` | -| `pyproject.toml` | Modify | Add `relations` optional extra | -| `tests/test_unit/core/test_protocol.py` | Modify | Add `to_relation` to `_FakeConnection` | -| `tests/test_unit/backends/ibis/test_backend_settings.py` | Create | Tests for settings-aware `IbisBackend` | -| `tests/test_unit/backends/ibis/test_to_relation.py` | Create | Tests for `to_relation()` on both Ibis paths | - ---- - -### Task 1: Settings-aware IbisBackend - -**Files:** -- Modify: `src/mountainash_data/backends/ibis/backend.py:106-142` -- Create: `tests/test_unit/backends/ibis/test_backend_settings.py` - -- [ ] **Step 1: Create test directory and write failing tests** - -Create `tests/test_unit/backends/__init__.py` and `tests/test_unit/backends/ibis/__init__.py` if they don't exist, then create the test file: - -```python -"""Tests for settings-aware IbisBackend constructor.""" - -import pytest -from mountainash_data.backends.ibis.backend import IbisBackend, IbisConnection -from mountainash_data.core.settings import SQLiteAuthSettings, DuckDBAuthSettings, NoAuth -from mountainash_settings import SettingsParameters - - -class TestIbisBackendFromSettings: - """Test IbisBackend constructed from SettingsParameters.""" - - def test_sqlite_settings_creates_backend(self): - params = SettingsParameters.create( - settings_class=SQLiteAuthSettings, - kwargs={"DATABASE": ":memory:", "auth": NoAuth()}, - ) - backend = IbisBackend(params) - assert backend.dialect == "sqlite" - - def test_duckdb_settings_creates_backend(self): - params = SettingsParameters.create( - settings_class=DuckDBAuthSettings, - kwargs={"DATABASE": ":memory:", "auth": NoAuth()}, - ) - backend = IbisBackend(params) - assert backend.dialect == "duckdb" - - def test_settings_backend_connects(self): - params = SettingsParameters.create( - settings_class=DuckDBAuthSettings, - kwargs={"DATABASE": ":memory:", "auth": NoAuth()}, - ) - backend = IbisBackend(params) - conn = backend.connect() - try: - assert isinstance(conn, IbisConnection) - tables = conn.list_tables() - assert isinstance(tables, list) - finally: - conn.close() - - def test_direct_dialect_still_works(self): - backend = IbisBackend(dialect="duckdb", database=":memory:") - conn = backend.connect() - try: - assert isinstance(conn, IbisConnection) - finally: - conn.close() -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `hatch run test:test-target tests/test_unit/backends/ibis/test_backend_settings.py -v` -Expected: FAIL — `IbisBackend` does not accept `SettingsParameters` - -- [ ] **Step 3: Implement settings-aware constructor** - -In `src/mountainash_data/backends/ibis/backend.py`, replace the `IbisBackend` class (lines 106-142) with: - -```python -class IbisBackend: - """Ibis backend factory. - - Construction takes either a dialect name with config kwargs, or a - SettingsParameters object that resolves both automatically. - - Usage: - # Direct config - backend = IbisBackend(dialect="sqlite", database=":memory:") - - # Settings-driven - backend = IbisBackend(settings_params) - - conn = backend.connect() - try: - tables = conn.list_tables() - finally: - conn.close() - """ - - name = "ibis" - - def __init__(self, dialect: str | t.Any = "", **config: t.Any): - from mountainash_settings import SettingsParameters - - if isinstance(dialect, SettingsParameters): - settings_params = dialect - settings = settings_params.settings_class.get_settings(settings_params) - descriptor = settings.__descriptor__ - ibis_dialect = descriptor.ibis_dialect - if ibis_dialect not in DIALECTS: - raise KeyError( - f"Unknown ibis dialect {ibis_dialect!r}. " - f"Available: {sorted(DIALECTS)}" - ) - self.dialect = ibis_dialect - self._spec: DialectSpec = DIALECTS[ibis_dialect] - driver_kwargs = settings.to_driver_kwargs() - self._config = { - k: v for k, v in driver_kwargs.items() - if not (isinstance(v, (list, tuple)) and len(v) == 0) - } - else: - if dialect not in DIALECTS: - raise KeyError( - f"Unknown ibis dialect {dialect!r}. " - f"Available: {sorted(DIALECTS)}" - ) - self.dialect = dialect - self._spec = DIALECTS[dialect] - self._config = config - - def connect(self) -> IbisConnection: - """Build and return a live ibis connection.""" - if self._spec.connection_builder is None: - raise NotImplementedError( - f"Dialect {self.dialect!r} has no connection_builder configured" - ) - ibis_conn = self._spec.connection_builder(**self._config) - return IbisConnection(ibis_conn, self._spec) -``` - -Note: The parameter is named `dialect` (not `dialect_or_settings`) to -preserve backward compatibility with `IbisBackend(dialect="sqlite")`. -Type detection via `isinstance` distinguishes `SettingsParameters` from a -string. The `SettingsParameters` import is inside `__init__` to avoid a -top-level dependency for the direct-config path. - -Empty list/tuple values are filtered from settings-derived kwargs (e.g. -`extensions=[]` from DuckDB) because some ibis drivers reject them. This -matches the normalization in `BaseIbisConnection.connect_default()`. - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `hatch run test:test-target tests/test_unit/backends/ibis/test_backend_settings.py -v` -Expected: PASS — all 4 tests - -- [ ] **Step 5: Run full test suite** - -Run: `hatch run test:test-quick` -Expected: All existing tests still pass - -- [ ] **Step 6: Commit** - -```bash -git add src/mountainash_data/backends/ibis/backend.py tests/test_unit/backends/ -git commit -m "feat(backend): make IbisBackend settings-aware" -``` - ---- - -### Task 2: Add to_relation() to protocol and all implementations - -**Files:** -- Modify: `src/mountainash_data/core/protocol.py:19-52` -- Modify: `src/mountainash_data/backends/ibis/backend.py:21-103` (IbisConnection) -- Modify: `src/mountainash_data/backends/ibis/connection.py:20-99` (BaseIbisConnection) -- Modify: `src/mountainash_data/backends/iceberg/connection.py:50-114` (IcebergConnectionBase) -- Modify: `tests/test_unit/core/test_protocol.py` (_FakeConnection) -- Create: `tests/test_unit/backends/ibis/test_to_relation.py` - -- [ ] **Step 1: Add to_relation() to the Connection protocol** - -In `src/mountainash_data/core/protocol.py`, add after `close()` (after line 52): - -```python - def to_relation( - self, name: str, namespace: str | None = None - ) -> t.Any: - """Return a mountainash Relation for the named table. - - Requires the mountainash package. Raises ImportError if not installed, - or NotImplementedError if the backend does not support it. - """ - ... -``` - -- [ ] **Step 2: Update _FakeConnection in test_protocol.py** - -In `tests/test_unit/core/test_protocol.py`, add to `_FakeConnection` (after `close`): - -```python - def to_relation(self, name: str, namespace: str | None = None) -> t.Any: - return f"relation:{name}" -``` - -- [ ] **Step 3: Run protocol tests to verify they still pass** - -Run: `hatch run test:test-target tests/test_unit/core/test_protocol.py -v` -Expected: PASS - -- [ ] **Step 4: Add to_relation() to IbisConnection** - -In `src/mountainash_data/backends/ibis/backend.py`, add to `IbisConnection` -(after `inspect_catalog`, before `close`): - -```python - def to_relation( - self, name: str, namespace: str | None = None - ) -> t.Any: - """Return a mountainash Relation wrapping the named ibis table.""" - try: - from mountainash.relations import relation - except ImportError: - raise ImportError( - "mountainash package is required for to_relation(). " - "Install it with: pip install mountainash" - ) - ibis_table = self._ibis_conn.table(name, database=namespace) - return relation(ibis_table) -``` - -- [ ] **Step 5: Add to_relation() to BaseIbisConnection** - -In `src/mountainash_data/backends/ibis/connection.py`, add to -`BaseIbisConnection` (after `connect_default`, before `_connect` — after -line 137): - -```python - def to_relation( - self, name: str, namespace: str | None = None - ) -> t.Any: - """Return a mountainash Relation wrapping the named ibis table.""" - try: - from mountainash.relations import relation - except ImportError: - raise ImportError( - "mountainash package is required for to_relation(). " - "Install it with: pip install mountainash" - ) - self.connect() - ibis_table = self.ibis_backend.table(name, database=namespace) - return relation(ibis_table) -``` - -- [ ] **Step 6: Add to_relation() stub to IcebergConnectionBase** - -In `src/mountainash_data/backends/iceberg/connection.py`, add to -`IcebergConnectionBase` (after `is_connected`, before the schema cache -section — after line 138): - -```python - def to_relation( - self, name: str, namespace: str | None = None - ) -> t.Any: - """Not yet supported for Iceberg connections.""" - raise NotImplementedError( - "to_relation() is not yet supported for Iceberg connections. " - "Use table() to get the native pyiceberg Table object." - ) -``` - -- [ ] **Step 7: Write tests for to_relation()** - -Create `tests/test_unit/backends/ibis/test_to_relation.py`: - -```python -"""Tests for to_relation() on Ibis connection paths.""" - -import pytest -import typing as t -from unittest.mock import patch - -from mountainash_data.backends.ibis.backend import IbisBackend, IbisConnection - - -class TestIbisConnectionToRelation: - """Test to_relation() on the new-style IbisConnection path.""" - - @pytest.fixture - def conn(self): - backend = IbisBackend(dialect="duckdb", database=":memory:") - conn = backend.connect() - yield conn - conn.close() - - def test_to_relation_returns_relation(self, conn): - try: - from mountainash.relations import Relation - except ImportError: - pytest.skip("mountainash package not installed") - - conn._ibis_conn.raw_sql("CREATE TABLE test_tbl (id INTEGER, name VARCHAR)") - result = conn.to_relation("test_tbl") - assert isinstance(result, Relation) - - def test_to_relation_import_error(self, conn): - conn._ibis_conn.raw_sql("CREATE TABLE test_tbl2 (id INTEGER)") - with patch.dict("sys.modules", {"mountainash": None, "mountainash.relations": None}): - with pytest.raises(ImportError, match="mountainash package is required"): - conn.to_relation("test_tbl2") - - -class TestBaseIbisConnectionToRelation: - """Test to_relation() on the settings/factory path.""" - - @pytest.fixture - def conn(self): - from mountainash_data.core.settings import DuckDBAuthSettings, NoAuth - from mountainash_settings import SettingsParameters - from mountainash_data.core.factories import ConnectionFactory - - params = SettingsParameters.create( - settings_class=DuckDBAuthSettings, - kwargs={"DATABASE": ":memory:", "auth": NoAuth()}, - ) - connection = ConnectionFactory.get_connection(params) - connection.connect() - yield connection - connection.disconnect() - - def test_to_relation_returns_relation(self, conn): - try: - from mountainash.relations import Relation - except ImportError: - pytest.skip("mountainash package not installed") - - conn.ibis_backend.raw_sql("CREATE TABLE test_tbl3 (id INTEGER, name VARCHAR)") - result = conn.to_relation("test_tbl3") - assert isinstance(result, Relation) - - -class TestIcebergToRelationStub: - """Test that Iceberg raises NotImplementedError.""" - - def test_raises_not_implemented(self): - from mountainash_data.backends.iceberg.connection import IcebergConnectionBase - with pytest.raises(NotImplementedError, match="not yet supported"): - IcebergConnectionBase.to_relation(None, "some_table") -``` - -- [ ] **Step 8: Run tests** - -Run: `hatch run test:test-target tests/test_unit/backends/ibis/test_to_relation.py -v` -Expected: Tests pass (or skip if mountainash not installed) - -Run: `hatch run test:test-quick` -Expected: Full suite passes - -- [ ] **Step 9: Commit** - -```bash -git add src/mountainash_data/core/protocol.py \ - src/mountainash_data/backends/ibis/backend.py \ - src/mountainash_data/backends/ibis/connection.py \ - src/mountainash_data/backends/iceberg/connection.py \ - tests/test_unit/core/test_protocol.py \ - tests/test_unit/backends/ibis/test_to_relation.py -git commit -m "feat(protocol): add to_relation() to Connection protocol and Ibis implementations" -``` - ---- - -### Task 3: Add mountainash as optional dependency - -**Files:** -- Modify: `pyproject.toml:51-68` - -- [ ] **Step 1: Add relations extra to pyproject.toml** - -In `pyproject.toml`, after the `trino` optional dependency line (line 68), add: - -```toml -relations = ["mountainash"] -``` - -- [ ] **Step 2: Run tests** - -Run: `hatch run test:test-quick` -Expected: All tests pass - -- [ ] **Step 3: Commit** - -```bash -git add pyproject.toml -git commit -m "feat(deps): add mountainash as optional relations extra" -``` - ---- - -## Follow-up (not in this plan) - -- **Phase 2:** Migrate `DatabaseUtils` consumers to `IbisBackend(settings_params)` -- **Phase 3:** Deprecate `ConnectionFactory`, `OperationsFactory`, 12 concrete subclasses -- **Dead code candidate:** After the legacy branch removal in PR #78, - `BaseIbisConnection.ibis_connection_mode` and `connection_string_scheme` - abstract properties are unused by `connect_default()` but still declared - on all 12 subclasses. Track separately. - -Tracked in: `mountainash-central/01.principles/mountainash-data/f.backlog/settings-aware-backends.md` diff --git a/docs/superpowers/plans/2026-04-27-settings-aware-ibis-backend.md b/docs/superpowers/plans/2026-04-27-settings-aware-ibis-backend.md deleted file mode 100644 index 2bc0a8f..0000000 --- a/docs/superpowers/plans/2026-04-27-settings-aware-ibis-backend.md +++ /dev/null @@ -1,452 +0,0 @@ -# Settings-Aware IbisBackend Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Extend `IbisBackend` to accept `SettingsParameters` and connection URLs alongside the existing dialect keyword, producing `IbisConnection` from all three input forms. - -**Architecture:** The `IbisBackend.__init__` constructor gains a positional-only first parameter that accepts `SettingsParameters` or a URL string. A `dialect=` keyword preserves the existing path. All forms resolve to `(self.dialect, self._spec, self._config)` so `connect()` stays simple. A module-level `_SCHEME_TO_DIALECT` map (built from the `DIALECTS` registry) handles URL scheme detection. - -**Tech Stack:** ibis-framework, mountainash-settings (`SettingsParameters`), mountainash-data settings (`ConnectionProfile`, `BackendDescriptor`) - -**Spec:** `docs/superpowers/specs/2026-04-27-settings-aware-ibis-backend-design.md` - ---- - -## File Map - -| File | Responsibility | Change | -|------|---------------|--------| -| `src/mountainash_data/backends/ibis/backend.py` | `IbisBackend` + `IbisConnection` | New constructor, `_SCHEME_TO_DIALECT` map, empty-list filter in `connect()` | -| `tests/test_unit/backends/ibis/test_backend.py` | Unit tests for `IbisBackend` | New tests for settings, URL, error paths | - -No new files. `IbisConnection`, `DialectSpec` registry, and builders are untouched. - ---- - -### Task 1: Error-case tests and `_SCHEME_TO_DIALECT` map - -Establishes the constructor signature, dispatch validation, and the scheme lookup — all tested before the happy paths. - -**Files:** -- Modify: `src/mountainash_data/backends/ibis/backend.py:106-142` -- Modify: `tests/test_unit/backends/ibis/test_backend.py` - -- [ ] **Step 1: Write failing tests for constructor validation** - -Add to `tests/test_unit/backends/ibis/test_backend.py`: - -```python -import pytest -from mountainash_data.backends.ibis.backend import IbisBackend - - -def test_neither_positional_nor_dialect_raises(): - """Constructor with no arguments must raise ValueError.""" - with pytest.raises(ValueError, match="Either.*or.*dialect"): - IbisBackend() - - -def test_both_positional_and_dialect_raises(): - """Cannot supply both a positional arg and dialect= keyword.""" - with pytest.raises(ValueError, match="Cannot specify both"): - IbisBackend("sqlite://", dialect="sqlite") - - -def test_unknown_url_scheme_raises(): - """URL with unrecognised scheme must raise ValueError.""" - with pytest.raises(ValueError, match="Cannot detect ibis dialect"): - IbisBackend("nosuch://localhost/db") -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `hatch run test:test-target tests/test_unit/backends/ibis/test_backend.py::test_neither_positional_nor_dialect_raises tests/test_unit/backends/ibis/test_backend.py::test_both_positional_and_dialect_raises tests/test_unit/backends/ibis/test_backend.py::test_unknown_url_scheme_raises -v` - -Expected: FAIL — the current constructor signature `__init__(self, dialect: str, **config)` doesn't match these call forms. - -- [ ] **Step 3: Implement `_SCHEME_TO_DIALECT` map and new constructor** - -Replace the `IbisBackend` class in `src/mountainash_data/backends/ibis/backend.py` (lines 106–142) with: - -```python -# --------------------------------------------------------------------------- -# Scheme → dialect reverse lookup (built once from the DIALECTS registry) -# --------------------------------------------------------------------------- -def _build_scheme_to_dialect() -> dict[str, str]: - """Build a map from URL scheme (e.g. 'sqlite', 'postgres') to dialect name.""" - result: dict[str, str] = {} - for dialect_name, spec in DIALECTS.items(): - # connection_string_scheme is e.g. "postgres://", "duckdb://md:" - scheme = spec.connection_string_scheme.split("://")[0].lower() - # First dialect wins — e.g. "postgres" maps to "postgres", not "redshift" - if scheme not in result: - result[scheme] = dialect_name - # Common aliases - result.setdefault("postgresql", result.get("postgres", "postgres")) - return result - - -_SCHEME_TO_DIALECT: dict[str, str] = _build_scheme_to_dialect() - - -class IbisBackend: - """Ibis backend — single entry point for all Ibis connections. - - Three input forms, all producing IbisConnection via connect(): - - # Settings object (deployment, env-driven config) - backend = IbisBackend(settings_params) - - # Connection URL (universal connection strings) - backend = IbisBackend("postgresql://user:pass@host:5432/db") - - # Dialect keyword + kwargs (tests, scripts) - backend = IbisBackend(dialect="sqlite", database=":memory:") - """ - - name = "ibis" - - def __init__( - self, - settings_or_connection_string: str | t.Any | None = None, - /, - *, - dialect: str | None = None, - **config: t.Any, - ): - if settings_or_connection_string is not None and dialect is not None: - raise ValueError( - "Cannot specify both a positional settings/URL argument " - "and dialect= keyword" - ) - - if settings_or_connection_string is not None: - self._init_from_positional(settings_or_connection_string, config) - elif dialect is not None: - self._init_from_dialect(dialect, config) - else: - raise ValueError( - "Either a SettingsParameters/URL positional argument " - "or a dialect= keyword is required" - ) - - def _init_from_positional( - self, value: str | t.Any, config: dict[str, t.Any] - ) -> None: - # Lazy import — only pay for it on the settings/URL paths - from mountainash_settings import SettingsParameters - - if isinstance(value, SettingsParameters): - self._init_from_settings(value, config) - elif isinstance(value, str): - if "://" in value: - self._init_from_url(value, config) - else: - # Plain string — treat as dialect name - self._init_from_dialect(value, config) - else: - raise TypeError( - f"Expected SettingsParameters or str, got {type(value).__name__}" - ) - - def _init_from_dialect( - self, dialect_name: str, config: dict[str, t.Any] - ) -> None: - if dialect_name not in DIALECTS: - raise KeyError( - f"Unknown ibis dialect {dialect_name!r}. " - f"Available: {sorted(DIALECTS)}" - ) - self.dialect = dialect_name - self._spec: DialectSpec = DIALECTS[dialect_name] - self._config = config - - def _init_from_dialect( - self, dialect_name: str, config: dict[str, t.Any] - ) -> None: - if dialect_name not in DIALECTS: - raise KeyError( - f"Unknown ibis dialect {dialect_name!r}. " - f"Available: {sorted(DIALECTS)}" - ) - self.dialect = dialect_name - self._spec: DialectSpec = DIALECTS[dialect_name] - self._url: str | None = None - self._config = config - - def _init_from_url( - self, url: str, config: dict[str, t.Any] - ) -> None: - from urllib.parse import urlparse - - scheme = urlparse(url).scheme.lower() - - # Special case: MotherDuck URLs are "duckdb://md:..." - if scheme == "duckdb" and url.startswith("duckdb://md:"): - resolved_dialect = "motherduck" - else: - resolved_dialect = _SCHEME_TO_DIALECT.get(scheme) - - if resolved_dialect is None: - raise ValueError( - f"Cannot detect ibis dialect from URL scheme: {scheme!r}" - ) - - self.dialect = resolved_dialect - self._spec = DIALECTS[resolved_dialect] - self._url = url - self._config = config - - def _init_from_settings( - self, settings_params: t.Any, config: dict[str, t.Any] - ) -> None: - obj_settings = settings_params.settings_class.get_settings( - settings_parameters=settings_params - ) - descriptor = getattr(obj_settings, "__descriptor__", None) - if descriptor is None or getattr(descriptor, "ibis_dialect", None) is None: - raise ValueError( - f"Settings class {type(obj_settings).__name__} has no " - f"ibis_dialect on its descriptor" - ) - resolved_dialect = descriptor.ibis_dialect - if resolved_dialect not in DIALECTS: - raise KeyError( - f"Unknown ibis dialect {resolved_dialect!r} from descriptor. " - f"Available: {sorted(DIALECTS)}" - ) - driver_kwargs = obj_settings.to_driver_kwargs() - driver_kwargs.update(config) - - self.dialect = resolved_dialect - self._spec = DIALECTS[resolved_dialect] - self._url = None - self._config = driver_kwargs - - def connect(self) -> IbisConnection: - """Build and return a live ibis connection.""" - if self._spec.connection_builder is None: - raise NotImplementedError( - f"Dialect {self.dialect!r} has no connection_builder configured" - ) - if self._url is not None: - # URL path: delegate directly to ibis.connect() which - # natively handles all URL forms and preserves all URL - # components (host, port, credentials, database, query params). - import ibis - ibis_conn = ibis.connect(self._url, **self._config) - else: - # Settings/dialect path: go through the dialect builder - # with empty-list normalization (e.g. DuckDB extensions=[]). - cleaned_config = { - k: v for k, v in self._config.items() - if not (isinstance(v, (list, tuple)) and len(v) == 0) - } - ibis_conn = self._spec.connection_builder(**cleaned_config) - return IbisConnection(ibis_conn, self._spec) -``` - -Key details: -- The type annotation for `settings_or_connection_string` uses `str | t.Any | None` rather than `str | SettingsParameters | None` to avoid a top-level import of `mountainash_settings`. The `SettingsParameters` isinstance check happens inside `_init_from_positional` with a lazy import. -- `_init_from_url` stores the raw URL on `self._url`. In `connect()`, when `_url` is set, the URL is passed directly to `ibis.connect(url)` — this natively handles all URL forms for all backends and preserves all URL components. The dialect builders are NOT used for the URL path (they don't all handle `connection_string` kwargs). -- `_init_from_settings` resolves the settings object, extracts `ibis_dialect` from the descriptor, and calls `to_driver_kwargs()`. Sets `_url = None`. -- `_init_from_dialect` is the existing path, unchanged except for initialising `_url = None`. -- `connect()` branches on `self._url`: URL path uses `ibis.connect(url)`, settings/dialect path uses the builder with empty-list filtering. The empty-list filter is essential because `DuckDBAuthSettings.to_driver_kwargs()` returns `extensions: []` by default, which `ibis.duckdb.connect()` rejects. - -- [ ] **Step 4: Run the three error tests** - -Run: `hatch run test:test-target tests/test_unit/backends/ibis/test_backend.py::test_neither_positional_nor_dialect_raises tests/test_unit/backends/ibis/test_backend.py::test_both_positional_and_dialect_raises tests/test_unit/backends/ibis/test_backend.py::test_unknown_url_scheme_raises -v` - -Expected: PASS — all three error cases now handled by the new constructor. - -- [ ] **Step 5: Run existing tests to verify no regressions** - -Run: `hatch run test:test-target tests/test_unit/backends/ibis/test_backend.py -v` - -Expected: All 4 existing tests + 3 new tests PASS. The existing tests use `IbisBackend(dialect="sqlite", ...)` which hits `_init_from_dialect` — the unchanged path. - -- [ ] **Step 6: Commit** - -```bash -git add src/mountainash_data/backends/ibis/backend.py tests/test_unit/backends/ibis/test_backend.py -git commit -m "feat(backend): new IbisBackend constructor with dispatch and error validation - -Add _SCHEME_TO_DIALECT map, three-way dispatch (settings, URL, dialect), -empty-list normalization in connect(), and URL-direct via ibis.connect(). -Error cases tested: no args, both args, unknown scheme." -``` - ---- - -### Task 2: Settings path — SQLite and DuckDB - -Wire and test the `SettingsParameters` input form. - -**Files:** -- Modify: `tests/test_unit/backends/ibis/test_backend.py` - -- [ ] **Step 1: Write failing tests for settings path** - -Add to `tests/test_unit/backends/ibis/test_backend.py`: - -```python -from mountainash_data.backends.ibis.backend import IbisBackend, IbisConnection - - -def test_settings_path_sqlite(): - """Construct IbisBackend from SQLite SettingsParameters and connect.""" - from mountainash_settings import SettingsParameters - from mountainash_data.core.settings import SQLiteAuthSettings, NoAuth - - params = SettingsParameters.create( - settings_class=SQLiteAuthSettings, - DATABASE=":memory:", - auth=NoAuth(), - ) - backend = IbisBackend(params) - assert backend.dialect == "sqlite" - conn = backend.connect() - assert isinstance(conn, IbisConnection) - tables = conn.list_tables() - assert isinstance(tables, list) - conn.close() - - -def test_settings_path_duckdb_empty_extensions(): - """DuckDB settings with default EXTENSIONS=[] must not crash ibis.""" - from mountainash_settings import SettingsParameters - from mountainash_data.core.settings import DuckDBAuthSettings, NoAuth - - params = SettingsParameters.create( - settings_class=DuckDBAuthSettings, - DATABASE=":memory:", - auth=NoAuth(), - ) - backend = IbisBackend(params) - assert backend.dialect == "duckdb" - conn = backend.connect() # Must not raise — empty-list filter active - assert isinstance(conn, IbisConnection) - conn.close() -``` - -- [ ] **Step 2: Run tests to verify they pass** - -Run: `hatch run test:test-target tests/test_unit/backends/ibis/test_backend.py::test_settings_path_sqlite tests/test_unit/backends/ibis/test_backend.py::test_settings_path_duckdb_empty_extensions -v` - -Expected: PASS — the constructor's `_init_from_settings` path was implemented in Task 1. - -If the DuckDB test fails with an error about `extensions=[]`, verify that `connect()` properly filters empty lists before calling the builder. - -- [ ] **Step 3: Commit** - -```bash -git add tests/test_unit/backends/ibis/test_backend.py -git commit -m "test(backend): settings path — SQLite and DuckDB SettingsParameters - -Verifies constructor resolves SettingsParameters via descriptor.ibis_dialect -and to_driver_kwargs(). DuckDB test confirms empty-list normalization -filters extensions=[] before reaching ibis." -``` - ---- - -### Task 3: URL path — SQLite and DuckDB - -Wire and test the connection URL input form. - -**Files:** -- Modify: `tests/test_unit/backends/ibis/test_backend.py` - -- [ ] **Step 1: Write failing tests for URL path** - -Add to `tests/test_unit/backends/ibis/test_backend.py`: - -```python -def test_url_path_sqlite(): - """Construct IbisBackend from sqlite:// URL and connect.""" - backend = IbisBackend("sqlite://") - assert backend.dialect == "sqlite" - conn = backend.connect() - assert isinstance(conn, IbisConnection) - conn.close() - - -def test_url_path_duckdb(): - """Construct IbisBackend from duckdb:// URL and connect.""" - backend = IbisBackend("duckdb://") - assert backend.dialect == "duckdb" - conn = backend.connect() - assert isinstance(conn, IbisConnection) - conn.close() - - -def test_url_path_preserves_database(tmp_path): - """URL database component must reach the driver, not be discarded.""" - db_file = tmp_path / "test.db" - backend = IbisBackend(f"sqlite:///{db_file}") - assert backend.dialect == "sqlite" - conn = backend.connect() - assert isinstance(conn, IbisConnection) - conn.close() - assert db_file.exists() -``` - -- [ ] **Step 2: Run tests to verify they pass** - -Run: `hatch run test:test-target tests/test_unit/backends/ibis/test_backend.py::test_url_path_sqlite tests/test_unit/backends/ibis/test_backend.py::test_url_path_duckdb tests/test_unit/backends/ibis/test_backend.py::test_url_path_preserves_database -v` - -Expected: PASS — the URL path calls `ibis.connect(url)` directly, which handles all URL forms natively. - -The `test_url_path_preserves_database` test is the critical regression test identified by Codex review: it proves the URL's database path actually reaches the driver (the file must exist on disk after connect). - -- [ ] **Step 3: Commit** - -```bash -git add tests/test_unit/backends/ibis/test_backend.py -git commit -m "test(backend): URL path — sqlite://, duckdb://, file path preservation - -Verifies URL dispatch via _SCHEME_TO_DIALECT, ibis.connect(url) delegation. -Critical regression: sqlite:///path creates the file on disk, proving -URL components are not discarded." -``` - ---- - -### Task 4: Full test suite and mark old docs abandoned - -Run the entire test suite, then commit the abandoned old spec/plan. - -**Files:** -- Modify: `docs/superpowers/specs/2026-04-26-to-relation-design.md` (already marked) -- Modify: `docs/superpowers/plans/2026-04-27-settings-aware-backends-to-relation.md` (already marked) - -- [ ] **Step 1: Run full test suite** - -Run: `hatch run test:test-quick` - -Expected: All tests PASS (441+ existing + 8 new). No regressions. - -If any existing tests fail, investigate — the existing `test_ibis_backend_satisfies_protocol`, `test_unknown_dialect_raises`, `test_all_registered_dialects_construct`, and `test_in_memory_sqlite_connect_and_inspect` should all still pass because they use `dialect=` keyword. - -- [ ] **Step 2: Verify old spec and plan are marked abandoned** - -Check that these files already have ABANDONED status (done earlier in the brainstorming session): -- `docs/superpowers/specs/2026-04-26-to-relation-design.md` — line 4 should say `ABANDONED` -- `docs/superpowers/plans/2026-04-27-settings-aware-backends-to-relation.md` — line 3 should say `ABANDONED` - -- [ ] **Step 3: Commit abandoned docs** - -```bash -git add docs/superpowers/specs/2026-04-26-to-relation-design.md docs/superpowers/plans/2026-04-27-settings-aware-backends-to-relation.md docs/superpowers/specs/2026-04-27-settings-aware-ibis-backend-design.md docs/superpowers/plans/2026-04-27-settings-aware-ibis-backend.md -git commit -m "chore(specs): abandon old to-relation spec/plan, add new settings-aware spec/plan - -Old spec bundled to_relation() which has been descoped per revised -principle. New spec focuses solely on settings-aware IbisBackend -constructor (Phase 1 of connection consolidation)." -``` - -- [ ] **Step 4: Run full test suite one final time** - -Run: `hatch run test:test-quick` - -Expected: All tests PASS. Clean working tree (no uncommitted changes to source). diff --git a/docs/superpowers/plans/2026-05-13-profile-spec-migration.md b/docs/superpowers/plans/2026-05-13-profile-spec-migration.md deleted file mode 100644 index dcb24b4..0000000 --- a/docs/superpowers/plans/2026-05-13-profile-spec-migration.md +++ /dev/null @@ -1,236 +0,0 @@ -# mountainash-data: profile-spec migration plan - -**Goal:** Migrate `mountainash-data` to the new `ProfileSpec` / `Profile` vocabulary introduced in `mountainash-settings 26.5.0`. Follow the [migration guide](../../../../mountainash-settings/docs/superpowers/specs/2026-05-13-profile-spec-rename-design.md#migration-guide-for-downstream-consumers) verbatim. - -**Architecture:** Mechanical search-and-replace across 21 backend files plus updates to `descriptor.py`, `registry.py`, `profile.py`, and test files. Add a PEP 562 `__getattr__` shim in `mountainash-data`'s own `descriptor.py` so any downstream consumer of `mountainash-data` gets the same one-release deprecation window. - -**Tech Stack:** Python 3.10+, pydantic 2.x, mountainash-settings ≥26.5.0 (via path-based dep), pytest, hatch. - -**Upstream spec:** `../../../../mountainash-settings/docs/superpowers/specs/2026-05-13-profile-spec-rename-design.md` - -**Out of scope:** -- Renaming `*AuthSettings` concrete class names (e.g. `PostgreSQLAuthSettings`) — explicitly deferred in the upstream spec. -- Adding a `mountainash-settings` version pin — `mountainash-data` uses a path-based dependency (`{root:uri}/../mountainash-settings`), so version coordination is implicit. - ---- - -## File survey (from `grep -ln`) - -**Source files using old names (21):** - -``` -src/mountainash_data/core/settings/registry.py -src/mountainash_data/core/settings/profile.py -src/mountainash_data/core/settings/descriptor.py -src/mountainash_data/core/settings/sqlite.py -src/mountainash_data/core/settings/duckdb.py (likely) -src/mountainash_data/core/settings/postgresql.py -src/mountainash_data/core/settings/mysql.py -src/mountainash_data/core/settings/mssql.py -src/mountainash_data/core/settings/snowflake.py -src/mountainash_data/core/settings/redshift.py -src/mountainash_data/core/settings/bigquery.py (likely) -src/mountainash_data/core/settings/databricks.py (likely) -src/mountainash_data/core/settings/motherduck.py -src/mountainash_data/core/settings/clickhouse.py (likely) -src/mountainash_data/core/settings/trino.py -src/mountainash_data/core/settings/singlestoredb.py -src/mountainash_data/core/settings/exasol.py -src/mountainash_data/core/settings/impala.py -src/mountainash_data/core/settings/materialize.py -src/mountainash_data/core/settings/risingwave.py -src/mountainash_data/core/settings/druid.py (likely) -src/mountainash_data/core/settings/pyspark.py -src/mountainash_data/core/settings/pyiceberg_rest.py -``` - -**Test files using old names (3):** - -``` -tests/test_unit/core/settings/test_descriptor.py -tests/test_unit/core/settings/test_profile.py -tests/test_unit/core/settings/test_descriptors_invariants.py -``` - ---- - -## Rename table - -Apply to every file touched: - -| Old | New | -|---|---| -| `from mountainash_settings.profiles import ProfileDescriptor` | `from mountainash_settings.profiles import ProfileSpec` | -| `from mountainash_settings.profiles.descriptor import _Missing` | `from mountainash_settings.profiles import Missing` | -| `class BackendDescriptor(ProfileDescriptor)` | `class BackendSpec(ProfileSpec)` | -| Any `BackendDescriptor` reference | `BackendSpec` | -| `*_DESCRIPTOR = BackendDescriptor(...)` | `*_SPEC = BackendSpec(...)` | -| Every reference to `POSTGRESQL_DESCRIPTOR` etc. | `POSTGRESQL_SPEC` etc. | -| `@register(POSTGRESQL_DESCRIPTOR)` | `@register` (argument-free) | -| `__descriptor__ = POSTGRESQL_DESCRIPTOR` | `__spec__ = POSTGRESQL_SPEC` | -| `Registry("databases")` | `Registry("databases", spec_type=BackendSpec, profile_type=ConnectionProfile)` | -| Local MRO walk in `to_driver_kwargs` | `lookup_class_var` import from `mountainash_settings` | -| `descriptor_invariants_for` | `spec_invariants_for` | -| `TestDescriptorInvariants_*` | `TestSpecInvariants_*` (in expected pytest output assertions) | - ---- - -## Tasks - -### Task A: `descriptor.py` rename + shim - -**Files:** -- Modify: `src/mountainash_data/core/settings/descriptor.py` -- Modify: `src/mountainash_data/core/settings/__init__.py` (if `BackendDescriptor` is re-exported there) - -**Required changes:** - -1. Rename `class BackendDescriptor` → `class BackendSpec`. -2. Replace `from mountainash_settings.profiles.descriptor import _Missing` with `from mountainash_settings.profiles import Missing`. -3. Replace `from mountainash_settings.profiles import ProfileDescriptor` with `from mountainash_settings.profiles import ProfileSpec`. Update `class BackendSpec(ProfileSpec)`. -4. Update `__all__` to use `BackendSpec` and `Missing`. -5. Add PEP 562 `__getattr__` shim at the bottom of `descriptor.py`: - -```python -import warnings - - -_DEPRECATED = { - "BackendDescriptor": ("BackendSpec", BackendSpec), - "_Missing": ("Missing", Missing), -} - - -def __getattr__(name): - if name in _DEPRECATED: - new_name, obj = _DEPRECATED[name] - warnings.warn( - f"{name!r} is renamed to {new_name!r} in mountainash-data. " - f"Update imports to use the new name.", - DeprecationWarning, stacklevel=2, - ) - return obj - raise AttributeError(f"module {__name__!r} has no attribute {name!r}") -``` - -6. Update `__init__.py` re-exports if `BackendDescriptor` was previously exported — change to `BackendSpec` and add a top-level shim if downstream consumers may import directly from the package root. - -### Task B: `registry.py` constraints + `profile.py` lookup helper - -**Files:** -- Modify: `src/mountainash_data/core/settings/registry.py` -- Modify: `src/mountainash_data/core/settings/profile.py` - -**registry.py changes:** - -1. Update the `DATABASES_REGISTRY` construction: - -```python -# Before -DATABASES_REGISTRY = Registry("databases") - -# After -DATABASES_REGISTRY = Registry( - "databases", - spec_type=BackendSpec, - profile_type=ConnectionProfile, -) -``` - -2. Update any `descriptor_invariants_for` references to `spec_invariants_for`. - -**profile.py changes:** - -3. Replace local MRO walk in `to_driver_kwargs()`: - -```python -# Before — local MRO walk -adapter = type(self).__dict__.get("__adapter__") -if adapter is None: - for base in type(self).__mro__[1:]: - candidate = base.__dict__.get("__adapter__") - if candidate is not None: - adapter = candidate - break - -# After — public helper -from mountainash_settings import lookup_class_var -adapter = lookup_class_var(type(self), "__adapter__") -``` - -The import can go at the top of the file rather than inline. - -### Task C: 21 backend file sweep - -For each file in `src/mountainash_data/core/settings/` matching `^(?!__init__|descriptor|profile|registry).*\.py$`: - -**Replace:** -- `BackendDescriptor` → `BackendSpec` (imports and constructor calls) -- `_DESCRIPTOR = BackendDescriptor(` → `_SPEC = BackendSpec(` -- Every reference to `_DESCRIPTOR` → `_SPEC` -- `@register(_DESCRIPTOR)` → `@register` (drop the argument) -- `__descriptor__ = _DESCRIPTOR` (or whatever spec it points at) → `__spec__ = _SPEC` - -Each file is independent. Apply the same mechanical pattern. Verify after each that imports resolve. - -### Task D: Tests - -**Files:** -- Modify: `tests/test_unit/core/settings/test_profile.py` -- Modify: `tests/test_unit/core/settings/test_descriptor.py` (or rename to `test_spec.py` if desired — optional) -- Modify: `tests/test_unit/core/settings/test_descriptors_invariants.py` (consider renaming to `test_spec_invariants.py`) - -Apply the same rename table. Update any references to the old API names. - -### Task E: Version bump + verification - -**Files:** -- Modify: `src/mountainash_data/__version__.py` - -**Steps:** - -1. Bump version. Current is `2026.04.2`. Following `mountainash-data`'s CalVer pattern (`YYYY.MM.MICRO`), the next release is `2026.05.0` (since we're in May). - -2. Run full test suite: - ```bash - hatch run test:test - ``` - -3. Run with deprecation warnings escalated to errors: - ```bash - hatch run test:test -W "error::DeprecationWarning" -W "default::DeprecationWarning:mountainash_data.core.settings.descriptor" - ``` - The `-W default` filter explicitly allows warnings from `mountainash-data`'s own descriptor shim (which exists by design). Any warning from elsewhere fails the run — that's the migration completion check. - -4. Run lint: - ```bash - hatch run ruff:check - ``` - -5. Build: - ```bash - hatch build - ``` - -### Task F: Push and PR - -1. Push the feature branch. -2. Open PR targeting `develop` with the same level of detail as the upstream PR (description, test plan, removal commitment). - -### Restore stashed working tree - -After the PR is open: - -```bash -git stash pop # restore hatch.toml reorder + .claude/worktrees/settings-registry -``` - -(Or leave for the user to handle.) - ---- - -## Execution strategy - -The work is mechanical. Single-subagent dispatch can handle Tasks A-D as one batch since they're all in the same package and the transformations are templated. Task E and F can be done manually. - -This plan does not list individual TDD steps because the upstream contract guarantees behaviour: the new names resolve to the same objects as the old names (via deprecation aliases). Running the existing test suite is the verification. New tests are not added in this PR — the upstream PR added all the deprecation tests; this PR is purely a consumer migration. diff --git a/docs/superpowers/plans/2026-05-14-debt-backlog.md b/docs/superpowers/plans/2026-05-14-debt-backlog.md deleted file mode 100644 index caf8460..0000000 --- a/docs/superpowers/plans/2026-05-14-debt-backlog.md +++ /dev/null @@ -1,266 +0,0 @@ -# Technical debt backlog — mountainash-data - -Discovered during initial package documentation profile run (2026-05-14). -Source hash at time of discovery: `1254928c55c9b0c5932707a6255c0325dd96f3c9` - ---- - -## DEBT-1 — `mountainash` meta-package is an undeclared hard dependency - -**Priority:** High -**Severity:** Runtime import failure if `mountainash` is not installed - -### What is broken - -`backends/ibis/operations.py:15` contains a module-level import: - -```python -import mountainash as ma -``` - -This file is imported at package load time by `dialects/_registry.py`, which is imported by -`backends/ibis/backend.py`, which is imported by `__init__.py`. If `mountainash` is not -installed, the entire package fails to import with `ModuleNotFoundError`. - -Two additional lazy imports exist in `backend.py` at lines 559 and 578 inside -`index_exists()` and `list_indexes()` — those fail at call time, not import time. - -`mountainash` does not appear anywhere in `pyproject.toml` — not in core dependencies -and not in any optional extra. - -### Affected files - -- `src/mountainash_data/backends/ibis/operations.py:15` -- `src/mountainash_data/backends/ibis/backend.py:559,578` -- `pyproject.toml` (missing declaration) - -### What `mountainash` is used for - -`operations.py` uses `ma.relation(result).to_dict()` and `ma.relation(result).to_dicts()` -to read the result sets of index-introspection SQL queries. It is not used for anything else. - -### Options - -**Option A — declare as core dependency:** -Add `mountainash` to `[project].dependencies` in `pyproject.toml`. Clean if `mountainash` -is always a reasonable peer dep for users of this package. - -**Option B — move to optional:** -Gate `index_exists()` and `list_indexes()` behind a try/import with a clear error message -if `mountainash` is missing. Appropriate if most users don't need index operations. - -**Option C — remove the dependency:** -Replace `ma.relation(result).to_dict()` with direct ibis `.execute().to_dict()` calls, -eliminating the `mountainash` import entirely. The ibis connection object already has -`.sql(query)` returning an ibis relation — `.execute()` converts it to pandas, from which -`to_dict()` works natively. - -**Recommended:** Option C for the module-level import in `operations.py` (zero new deps, -moves this forward). Option A for the lazy imports in `backend.py` if `mountainash` is -otherwise a declared peer. - ---- - -## DEBT-2 — `IcebergBackend.inspect_*` return types are `t.Any` - -**Priority:** Medium -**Severity:** Type-checker cannot infer result types for callers - -### What is wrong - -`backends/iceberg/backend.py` declares three inspection methods with `-> t.Any` return types: - -```python -def inspect_table(self, name: str, namespace: str | None = None) -> t.Any: ... -def inspect_namespace(self, name: str) -> t.Any: ... -def inspect_catalog(self) -> t.Any: ... -``` - -The underlying `IcebergConnectionBase` methods return `TableInfo`, `NamespaceInfo`, and -`CatalogInfo` respectively (from `core/inspection.py`). The `t.Any` annotations lose this -information, breaking type inference for any caller that uses `IcebergBackend`. - -### Fix - -```python -from mountainash_data.core.inspection import CatalogInfo, NamespaceInfo, TableInfo - -def inspect_table(self, name: str, namespace: str | None = None) -> TableInfo: ... -def inspect_namespace(self, name: str) -> NamespaceInfo: ... -def inspect_catalog(self) -> CatalogInfo: ... -``` - -Three-line change. Also brings `IcebergBackend` into formal conformance with the -`Backend` protocol (which uses these concrete types via `core/protocol.py`). - -### Affected file - -- `src/mountainash_data/backends/iceberg/backend.py:71–78` - ---- - -## DEBT-3 — Oracle dialect is half-registered - -**Priority:** Medium -**Severity:** `IbisBackend(dialect="oracle")` constructs without error but -has no settings class and no declared ibis driver extra - -### What exists - -- `DIALECTS["oracle"]` entry in `dialects/_registry.py` with a working - `_build_oracle_connection(**config)` function -- `CONST_DB_BACKEND.ORACLE` and `CONST_DB_BACKEND_IBIS_PREFIX.ORACLE` in `constants.py` -- Oracle listed in `test_dialect_spec.py:39` (registry presence test passes) - -### What is missing - -- `src/mountainash_data/core/settings/oracle.py` (no `OracleAuthSettings` class) -- `[oracle]` optional extra in `pyproject.toml` (no ibis oracle driver declared) -- No per-backend settings test file - -The settings path (`IbisBackend(settings_params)`) cannot be used for oracle because -`OracleAuthSettings` does not exist. The direct kwargs path (`IbisBackend(dialect="oracle", -host=..., ...)`) may work at runtime but is untested and has no driver dep guarantee. - -### Options - -**Option A — complete oracle support:** -Add `settings/oracle.py`, add `[oracle]` extra to `pyproject.toml`, add -`tests/test_unit/core/settings/backends/test_oracle.py`. - -**Option B — remove oracle from DIALECTS:** -Delete the `_build_oracle_connection` function and the `"oracle"` DIALECTS entry, -remove oracle from `CONST_DB_BACKEND` and `CONST_DB_BACKEND_IBIS_PREFIX`, remove -from `test_dialect_spec.py`. Clean public contract: only declare what is actually supported. - -**Recommended:** Decide explicitly. The current state (in registry, no settings class) is -misleading — it implies support that does not exist end-to-end. - ---- - -## DEBT-4 — `IcebergConnectionBase.connect_default()` hardcodes `RestCatalog` - -**Priority:** Medium -**Severity:** Architectural — future non-REST catalog implementations must override the entire base method - -### What is wrong - -```python -# connection.py:113 -self._catalog_backend: RestCatalog = RestCatalog(**connection_kwargs) -``` - -This is in the abstract base class `IcebergConnectionBase`. The type annotation and the -instantiation are both locked to `RestCatalog`, meaning: - -1. The `_catalog_backend` attribute is typed `RestCatalog` even in subclasses using - different catalog backends. -2. A `HiveConnectionBase` subclass can't call `super().connect_default()` — it must - re-implement the entire method. -3. `catalog_backend` abstract property is typed `Catalog | t.Any | None` which doesn't - match the stored concrete type. - -### Recommended fix - -Introduce a factory method for the catalog type, or accept a catalog class in the -constructor: - -```python -class IcebergConnectionBase(BaseDBConnection): - catalog_class: type[Catalog] = RestCatalog # override in subclasses - - def connect_default(self, **kwargs: t.Any) -> Catalog: - if self.catalog_backend is None: - obj_settings = ... - connection_kwargs = obj_settings.to_driver_kwargs() - self._catalog_backend = self.catalog_class(**connection_kwargs) - return self.catalog_backend -``` - -This keeps the base method intact while letting subclasses override only `catalog_class`. - -### Affected file - -- `src/mountainash_data/backends/iceberg/connection.py:105–114` - ---- - -## DEBT-5 — `BaseDBConnection.init_ssh()` is dead code with a latent crash - -**Priority:** Low -**Severity:** `AttributeError` if called; never called anywhere - -### What is wrong - -`core/connection.py:115`: - -```python -def init_ssh(self): - if self.ssh_required: - self.ssh_client.connect_ssh() -``` - -`self.ssh_required` is never set — the three lines that would set it are commented out in -`__init__()`. `init_ssh()` is not called anywhere in the codebase (confirmed by grep). -The SSH tunnel feature was scaffolded and then commented out, leaving a method that would -raise `AttributeError` at the first line if anyone called it. - -### Fix - -Delete `init_ssh()` and the commented-out SSH blocks in `__init__()`. If SSH tunnel -support is a future requirement, implement it cleanly when needed rather than maintaining -broken scaffolding. - -### Affected file - -- `src/mountainash_data/core/connection.py:33–40, 115–117` - ---- - -## DEBT-6 — `core/registry.py` is an empty placeholder - -**Priority:** Low -**Severity:** Misleading API — `get('ibis', ...)` raises `KeyError` at runtime - -### What exists - -`core/registry.py` provides `register(name, factory)` and `get(name, **config)` but its -docstring says "intentionally a placeholder for now." No backends self-register and the -`_REGISTRY` dict is always empty at runtime. - -### Context - -This module appears to be the intended future unified factory interface — a single -`get("ibis", dialect="duckdb", ...)` call instead of separate `IbisBackend` / `IcebergBackend` -entry points. The design intent exists; the wiring does not. - -### Options - -**Option A — wire it up:** Register `IbisBackend` and `IcebergBackend` factories here -on import, making `core.registry.get("ibis", ...)` a real third entry point. - -**Option B — remove it:** Delete the file if the unified factory design is not being -pursued. Keeping an empty public module with a doc lie is worse than not having it. - -**Option C — mark it explicitly experimental:** Rename to `_registry.py` or add a clear -`NotImplementedError` to `get()` so accidental callers get a meaningful error. - -**Recommended:** Decide intent. If the unified factory is in scope for the next major -refactor, add Option C guard and a TODO. If not, delete. - -### Affected file - -- `src/mountainash_data/core/registry.py` - ---- - -## Summary table - -| ID | Issue | Priority | Effort | Breaking if unaddressed | -|----|-------|----------|--------|------------------------| -| DEBT-1 | `mountainash` undeclared dep — module-level import crash | High | Small | Yes — full package import failure | -| DEBT-2 | `IcebergBackend.inspect_*` typed `t.Any` | Medium | Trivial | No — runtime works, types mislead | -| DEBT-3 | Oracle half-registered (no settings class, no extra) | Medium | Medium | No — misleads but doesn't crash | -| DEBT-4 | `connect_default()` hardcodes `RestCatalog` in abstract base | Medium | Medium | No — only breaks future catalog impls | -| DEBT-5 | `init_ssh()` dead code + latent `AttributeError` | Low | Trivial | No — never called | -| DEBT-6 | `core/registry.py` empty placeholder | Low | Decision | No — nobody calls it | diff --git a/docs/superpowers/plans/2026-06-28-auth-client-migration.md b/docs/superpowers/plans/2026-06-28-auth-client-migration.md deleted file mode 100644 index 15df97c..0000000 --- a/docs/superpowers/plans/2026-06-28-auth-client-migration.md +++ /dev/null @@ -1,1409 +0,0 @@ -# Auth-Client Migration Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Migrate `mountainash-data` off the deleted `mountainash_settings.auth` onto `mountainash-auth-client`, with auth decoupled from the backend config profile and composed in a factory. - -**Architecture:** Three layers mirroring `mountainash-transport`: (L1) `*BackendProfile.emit(provider_type)` renders config only; (L2) `*AuthProfile` is pure credential data from auth-client; (L3) a data-owned `(provider_type, auth_class)→fn` dispatch table renders auth onto the config; the `ConnectionFactory` composes them and the runtime (`IbisBackend`/`IcebergConnection`) consumes a finished dict. No `register_adapter`, no `emit()` on auth profiles, no cross-package mutation. - -**Tech Stack:** Python 3.12, `mountainash-settings` 26.5.0 (`Profile`/`ProfileSpec`/`ParameterSpec`/`emit()`), `mountainash-auth-client` (`*AuthProfile`), Ibis 11, PyIceberg, hatch + uv, pytest. - -## Global Constraints - -- **Clean break, zero downstream consumers.** No backward-compat aliases, no deprecation shims. Old names are deleted, not aliased. -- **`provider_type` enum is `CONST_DB_PROVIDER_TYPE`** in `core/constants.py`; the PostgreSQL member is **`POSTGRESQL`** (NOT `POSTGRES`). Members: `MYSQL, POSTGRESQL, MSSQL, SNOWFLAKE, BIGQUERY, REDSHIFT, SQLITE, DUCKDB, MOTHERDUCK, TRINO, PYICEBERG_REST, ORACLE, CLICKHOUSE, DATABRICKS, SINGLESTOREDB, EXASOL, IMPALA, MATERIALIZE, RISINGWAVE, DRUID, PYSPARK`. -- **Auth profile fields are UPPERCASE `ParameterSpec` names**; secret fields are pydantic `SecretStr` (use `.get_secret_value()`); path fields are `Path | None` (use `str(...)` where a driver wants a string). -- **Auth profiles are pure data.** Data NEVER calls `auth_profile.emit()` and registers NOTHING onto auth-client classes. -- **`Profile.emit(target, *, base=None)`** runs `driver_key` renames via `_default_kwargs(target)`, then a 2-arg compose adapter from `__adapters__.get(target)` if present. A profile with any `__adapters__` is "target-scoped": `emit()` with no target raises. A profile WITHOUT `__adapters__` (bare-string `driver_key`s) accepts any explicit target — so `emit(provider_type)` is uniform across all 20 (confirmed: flat backends resolve bare driver_keys regardless of target; shaping backends route through their `__adapters__[provider_type]`). -- **Compose adapters ADD only.** Never overwrite a key `driver_key` produced, except the single sanctioned mssql `host` rewrite. Fields a compose folds carry NO flat `driver_key`. -- **Auth dispatch is MRO-aware** (`functools.singledispatch` semantics): most-specific registered base wins; two incomparable sibling registrations for one `provider_type` raise `TypeError`. -- **Fail-closed everywhere.** Unsupported auth → `ValueError`; supported-but-no-adapter → `ValueError`; unsupported URL auth → `NotImplementedError`. -- **Driver imports are LOCAL** to each adapter function (e.g. `from trino.auth import ...` inside the fn), so importing the settings layer never pulls `trino`/`google`. -- **Test integrity:** if a golden disagrees with the implementation, STOP and surface it — do not edit the test to pass. Never encode counts of backends as test assertions. -- **Commit trailer (every commit):** - ``` - Co-Authored-By: Claude Opus 4.8 (1M context) - ``` -- **Branch:** all work on `feature/auth-client-migration` (already checked out); PR targets `develop`. - -### Spec deviations carried by this plan (verified against the real tree) - -1. **4 shaping backends, not 3:** add **pyiceberg** (`HEADERS`→`header.` compose) to mysql/mssql/snowflake. pyiceberg's `s3.*`/`rest.*` become flat `driver_key`s. -2. **`POSTGRESQL`** is the real enum member (spec said `POSTGRES`). -3. **`core/factories/` does not exist on this branch** — Task 6 creates it fresh. (A different `ConnectionFactory` exists on the `settings-registry` worktree branch; flag at PR time for merge awareness.) -4. **hatch:** auth-client is absent from all envs (add it); `mountainash_utils_ssh` is a dead path-dep in `dev`/`build_github`/`test_github` (remove it). -5. **pyspark** is pure-flat — give params `driver_key`s, delete its adapter. -6. **`UrlParts` lives in `core/settings/profile.py`** (the L1 output type), imported by the factory — NOT in the factory — so the settings flip doesn't depend on the factory. - -### CRITICAL ordering constraint (why the flip is Task 2) - -At HEAD the suite fails at collection: `core/settings/__init__.py` and every backend module import the deleted `mountainash_settings.auth`. **Importing ANY submodule of `core.settings` runs `core/settings/__init__.py` first**, so until the whole settings layer is migrated, *nothing* under `core.settings` — including new adapter/registry modules placed there — can be imported or tested. Therefore the atomic settings flip (Task 2) MUST precede the adapter/registry/factory tasks. After Task 2, `import mountainash_data.core.settings` succeeds and every later task's tests can collect. Full green (`hatch run test:test`) is asserted in Task 9. - ---- - -## File Structure - -**New files** -- `core/settings/adapters/sql.py` — shared flat `userpass(auth, base)`. -- `core/settings/adapters/registry.py` — `_AUTH_ADAPTERS` + MRO `auth_adapter()`. -- `core/factories/__init__.py`, `core/factories/connection_factory.py` — compose, URL appliers, `apply_auth_adapter`, dialect/scheme→provider helpers. -- Tests under `tests/test_unit/core/settings/adapters/`, `tests/test_unit/core/factories/`. - -**Heavily modified** -- `core/settings/descriptor.py` (`supported_auth`), `core/settings/profile.py` (`BackendProfile` + `UrlParts` + `to_url_parts`), `core/settings/__init__.py` (import swap), the 20 backend modules, the per-backend adapter modules, `backends/ibis/backend.py`, `backends/iceberg/connection.py`, `hatch.toml`, `pyproject.toml`. - -**Deleted** -- `core/settings/adapters/pyspark.py`, `core/settings/auth/` (3 shim files). - ---- - -## Task 1: Dependency wiring - -**Files:** Modify `pyproject.toml`, `hatch.toml`. - -**Interfaces:** Produces `mountainash_auth_client` importable in all hatch envs. - -- [ ] **Step 1: Add auth-client to `pyproject.toml`** - -In `[project] dependencies`, after `"sqlalchemy",`, add: -```toml - "mountainash-auth-client", -``` - -- [ ] **Step 2: Wire auth-client + remove dead utils-ssh in `hatch.toml`** - -In `envs.dev` and `envs.test`, add (local format) and DELETE any `mountainash_utils_ssh` line: -```toml - "mountainash_auth_client @ {root:uri}/../mountainash-auth-client", -``` -In `envs.test_github` and `envs.build_github`, add (CI format) and DELETE their `mountainash_utils_ssh` lines: -```toml - "mountainash_auth_client @ {root:uri}/temp/mountainash-auth-client", -``` - -- [ ] **Step 3: Verify auth-client imports** - -Run: `hatch run test:python -c "import mountainash_auth_client as a; print(a.PasswordAuthProfile, a.NoAuthProfile, a.AuthProfile)"` -Expected: prints the three classes. If env is stale: `hatch env prune` then re-run. - -- [ ] **Step 4: Confirm utils-ssh gone** - -Run: `grep -rn "mountainash_utils_ssh" hatch.toml` -Expected: no output. - -- [ ] **Step 5: Commit** -```bash -git add pyproject.toml hatch.toml -git commit -m "build: add mountainash-auth-client dep; drop dead utils-ssh path-dep - -Co-Authored-By: Claude Opus 4.8 (1M context) " -``` - ---- - -## Task 2: Settings core flip (descriptor + base profile + 20 backends + __init__) - -The irreducible atomic flip — `__init__.py` imports the renamed backends, which import the renamed base, which uses the new descriptor. After this task `import mountainash_data.core.settings` succeeds and flat backends emit correct config. The 4 shaping backends import but emit config WITHOUT their nested keys (completed in Task 3). - -**Files:** -- Modify: `core/settings/descriptor.py`, `core/settings/profile.py`, all 20 backend modules, `core/settings/__init__.py` -- Delete: `core/settings/auth/` (3 files), `core/settings/adapters/pyspark.py` -- Test: `tests/test_unit/core/settings/test_settings_flip.py` - -**Interfaces:** -- Produces: `UrlParts` (in `profile.py`); `BackendProfile` base with `to_url_parts()`; `BackendSpec.supported_auth: tuple[type, ...]`; 20 `*BackendProfile` classes (rename table below). - -- [ ] **Step 1: Add `supported_auth` to `descriptor.py`** -```python -@dataclass(frozen=True, kw_only=True) -class BackendSpec(ProfileSpec): - default_port: int | None = None - connection_string_scheme: str | None = None - ibis_dialect: str | None = None - rides_on: str | None = None - supported_auth: tuple[type, ...] = () - - def __post_init__(self) -> None: - if not self.supported_auth: - raise ValueError(f"{self.name}: supported_auth must be non-empty") -``` -> If `ProfileSpec` defines `__post_init__`, call `super().__post_init__()` first. Check: `hatch run test:python -c "from mountainash_settings.profiles import ProfileSpec; print(hasattr(ProfileSpec,'__post_init__'))"`. - -- [ ] **Step 2: Rewrite `profile.py` — `UrlParts` + `BackendProfile`** -```python -from dataclasses import dataclass, field -# ... keep existing Profile / lookup_class_var imports; REMOVE the quote import. - - -@dataclass(frozen=True) -class UrlParts: - """Credential-free URL skeleton (L1). Every authority component optional.""" - scheme: str - database: str | None = None - host: str | None = None - port: int | None = None - path: str | None = None - query: dict[str, str] = field(default_factory=dict) - - -class BackendProfile(Profile): - """Database backend CONFIG. Pure L1 emitter — no auth methods. - - Auth is orthogonal, applied by ConnectionFactory, never here. - """ - - def to_url_parts(self) -> UrlParts: - desc = lookup_class_var(type(self), "__spec__") - scheme = getattr(desc, "connection_string_scheme", None) - if scheme is None: - raise NotImplementedError(f"Profile {self.backend!r} has no URL form") - scheme = scheme.removesuffix("://").removesuffix(":") - return UrlParts( - scheme=scheme, - host=getattr(self, "HOST", None), - port=getattr(self, "PORT", None), - database=getattr(self, "DATABASE", None), - ) -``` -Remove `to_driver_kwargs`, `to_connection_string`, `_auth_kwargs`/`__adapter__` references. - -- [ ] **Step 3: The 16 flat backends — rename + import-swap + supported_auth** - -For each, apply: (1) `from mountainash_settings.auth import (...)` → `from mountainash_auth_client import ()`; (2) `from .profile import ConnectionProfile` → `from .profile import BackendProfile`; (3) replace `auth_modes=[...]` with `supported_auth=(),`; (4) rename `class AuthSettings(ConnectionProfile):` → `class BackendProfile(BackendProfile):`. - -Worked example — `postgresql.py`: -```python -from mountainash_auth_client import NoAuthProfile, PasswordAuthProfile -from .profile import BackendProfile -# ...inside POSTGRESQL_SPEC: - supported_auth=(PasswordAuthProfile, NoAuthProfile), -# ... -@register -class PostgreSQLBackendProfile(BackendProfile): - __spec__ = POSTGRESQL_SPEC -``` - -| File | Old → New class | `supported_auth=` | -|---|---|---| -| `postgresql.py` | `PostgreSQLAuthSettings` → `PostgreSQLBackendProfile` | `(PasswordAuthProfile, NoAuthProfile)` | -| `clickhouse.py` | `ClickHouseAuthSettings` → `ClickHouseBackendProfile` | `(PasswordAuthProfile, NoAuthProfile)` | -| `singlestoredb.py` | `SingleStoreDBAuthSettings` → `SingleStoreDBBackendProfile` | `(PasswordAuthProfile, NoAuthProfile)` | -| `druid.py` | `DruidAuthSettings` → `DruidBackendProfile` | `(PasswordAuthProfile, NoAuthProfile)` | -| `impala.py` | `ImpalaAuthSettings` → `ImpalaBackendProfile` | `(PasswordAuthProfile, NoAuthProfile)` | -| `materialize.py` | `MaterializeAuthSettings` → `MaterializeBackendProfile` | `(PasswordAuthProfile, NoAuthProfile)` | -| `risingwave.py` | `RisingWaveAuthSettings` → `RisingWaveBackendProfile` | `(PasswordAuthProfile, NoAuthProfile)` | -| `exasol.py` | `ExasolAuthSettings` → `ExasolBackendProfile` | `(PasswordAuthProfile,)` | -| `sqlite.py` | `SQLiteAuthSettings` → `SQLiteBackendProfile` | `(NoAuthProfile,)` | -| `duckdb.py` | `DuckDBAuthSettings` → `DuckDBBackendProfile` | `(NoAuthProfile,)` | -| `redshift.py` | `RedshiftAuthSettings` → `RedshiftBackendProfile` | `(PasswordAuthProfile, IAMAuthProfile)` | -| `databricks.py` | `DatabricksAuthSettings` → `DatabricksBackendProfile` | `(TokenAuthProfile, PasswordAuthProfile, NoAuthProfile)` | -| `trino.py` | `TrinoAuthSettings` → `TrinoBackendProfile` | `(PasswordAuthProfile, JWTAuthProfile, KerberosAuthProfile, NoAuthProfile)` | -| `bigquery.py` | `BigQueryAuthSettings` → `BigQueryBackendProfile` | `(ServiceAccountAuthProfile, NoAuthProfile)` | - -> `redshift/databricks/trino/bigquery` carry `__adapter__ = staticmethod(_adapter.build_driver_kwargs)` + a `from .adapters import X as _adapter` line — **DELETE both** (their config is flat `driver_key`s; auth moves to the registry). No `__adapters__` needed for them. - -- [ ] **Step 4: pyspark — flat, delete adapter, add `driver_key`s** - -`pyspark.py`: delete `from .adapters import pyspark as _adapter` + the `__adapter__` line; rename → `PySparkBackendProfile`; `supported_auth=(NoAuthProfile,)`; add `driver_key`s: -```python - ParameterSpec(name="SESSION", type=t.Optional[t.Any], tier="core", default=None, driver_key="session"), - ParameterSpec(name="MODE", type=PySparkMode, tier="core", default=PySparkMode.BATCH, driver_key="mode"), - ParameterSpec(name="SPARK_MASTER", type=t.Optional[str], tier="advanced", default=None, driver_key="spark.master"), - ParameterSpec(name="APPLICATION_NAME", type=t.Optional[str], tier="advanced", default=None, driver_key="spark.app.name"), - ParameterSpec(name="WAREHOUSE_DIR", type=t.Optional[str], tier="advanced", default=None, driver_key="spark.sql.warehouse.dir"), - ParameterSpec(name="PARTITIONS", type=t.Optional[int], tier="advanced", default=None, driver_key="spark.sql.shuffle.partitions"), -``` -Update `__all__` → `["PySparkBackendProfile", "PySparkMode", "PYSPARK_SPEC"]`. `git rm core/settings/adapters/pyspark.py`. - -- [ ] **Step 5: motherduck — TokenAuth + URL override** - -`motherduck.py`: rename → `MotherDuckBackendProfile`; `supported_auth=(TokenAuthProfile,)`; import `TokenAuthProfile`; override (scheme `"duckdb://md:"` would mangle under the base logic): -```python - def to_url_parts(self): - from .profile import UrlParts - return UrlParts(scheme="md", database=getattr(self, "DATABASE", None)) -``` - -- [ ] **Step 6: The 4 shaping backends — rename only (compose in Task 3)** - -`mysql.py`/`mssql.py`/`snowflake.py`/`pyiceberg_rest.py`: apply rename + import-swap + supported_auth, and **DELETE** the `__adapter__` line + `from .adapters import X as _adapter` import. Do NOT add `__adapters__` yet. -- `mysql.py` → `MySQLBackendProfile`, `(PasswordAuthProfile,)` -- `mssql.py` → `MSSQLBackendProfile`, `(PasswordAuthProfile, WindowsAuthProfile, AzureADAuthProfile)` -- `snowflake.py` → `SnowflakeBackendProfile`, `(PasswordAuthProfile, OAuth2AuthProfile, CertificateAuthProfile, TokenAuthProfile)` -- `pyiceberg_rest.py` → `PyIcebergRestBackendProfile`, `(TokenAuthProfile,)` - -- [ ] **Step 7: Delete the auth shim** -```bash -git rm core/settings/auth/__init__.py core/settings/auth/base.py core/settings/auth/dispatch.py -``` - -- [ ] **Step 8: Rewrite `core/settings/__init__.py`** - -Replace the `from mountainash_settings.auth import (...)` block with: -```python -from mountainash_auth_client import ( - AuthProfile, NoAuthProfile, PasswordAuthProfile, TokenAuthProfile, - JWTAuthProfile, OAuth2AuthProfile, IAMAuthProfile, WindowsAuthProfile, - AzureADAuthProfile, KerberosAuthProfile, CertificateAuthProfile, - ServiceAccountAuthProfile, -) -``` -Change `from .profile import ConnectionProfile` → `from .profile import BackendProfile, UrlParts`. Rewrite the 20 backend imports to the new names. Rewrite `__all__`: drop every `*Auth`/`AuthSpec`/`*AuthSettings` name; add the `*AuthProfile` names + `AuthProfile`; add `"UrlParts"`; replace `"ConnectionProfile"` with `"BackendProfile"`; list the 20 `*BackendProfile` names. - -- [ ] **Step 9: Write the smoke test** - -`tests/test_unit/core/settings/test_settings_flip.py`: -```python -import pytest -from mountainash_auth_client import PasswordAuthProfile, NoAuthProfile -from mountainash_data.core.constants import CONST_DB_PROVIDER_TYPE as P -from mountainash_data.core.settings import ( - BackendProfile, PostgreSQLBackendProfile, MotherDuckBackendProfile, -) -from mountainash_data.core.settings.descriptor import BackendSpec - - -def test_supported_auth_present(): - assert PostgreSQLBackendProfile.__spec__.supported_auth == (PasswordAuthProfile, NoAuthProfile) - - -def test_flat_emit_is_config_only(): - out = PostgreSQLBackendProfile(HOST="db", PORT=5432, DATABASE="app").emit(P.POSTGRESQL) - assert out["host"] == "db" and out["port"] == 5432 and out["database"] == "app" - assert "user" not in out and "password" not in out - - -def test_to_url_parts_standard(): - parts = PostgreSQLBackendProfile(HOST="db", PORT=5432, DATABASE="app").to_url_parts() - assert (parts.scheme, parts.host, parts.port, parts.database) == ("postgresql", "db", 5432, "app") - - -def test_motherduck_url_parts_authority_less(): - parts = MotherDuckBackendProfile(DATABASE="mydb").to_url_parts() - assert parts.scheme == "md" and parts.host is None and parts.database == "mydb" - - -def test_empty_supported_auth_invariant(): - with pytest.raises(ValueError, match="supported_auth"): - BackendSpec(name="x", provider_type=P.SQLITE, parameters=[], supported_auth=()) -``` - -- [ ] **Step 10: Verify settings imports + smoke passes** - -Run: `hatch run test:python -c "import mountainash_data.core.settings as s; print(sum(1 for n in dir(s) if n.endswith('BackendProfile')))"` -Expected: `20`. -Run: `hatch run test:test-target tests/test_unit/core/settings/test_settings_flip.py -q` -Expected: PASS. - -- [ ] **Step 11: Commit** -```bash -git add -A core/settings/ tests/test_unit/core/settings/test_settings_flip.py -git commit -m "refactor(settings)!: flip to *BackendProfile + supported_auth; drop auth shim - -Co-Authored-By: Claude Opus 4.8 (1M context) " -``` - ---- - -## Task 3: Config-shaping compose adapters (mysql, mssql, snowflake, pyiceberg) - -**Files:** -- Rewrite: `core/settings/adapters/{mysql,mssql,snowflake,pyiceberg_rest}.py` (compose fn ONLY this task; auth fns in Task 4) -- Modify: `core/settings/{mysql,mssql,snowflake,pyiceberg_rest}.py` (wire `__adapters__`; pyiceberg `driver_key`s) -- Test: `tests/test_unit/core/settings/test_config_shaping.py` - -**Interfaces:** Produces `mysql.ssl_compose`, `mssql.host_fold`, `snowflake.session_params`, `pyiceberg_rest.headers_compose` — each `(profile, base) -> dict`. - -- [ ] **Step 1: Write failing goldens (full-dict equality = mechanical key-delta)** - -`tests/test_unit/core/settings/test_config_shaping.py`: -```python -from mountainash_data.core.constants import CONST_DB_PROVIDER_TYPE as P -from mountainash_data.core.settings import ( - MySQLBackendProfile, MSSQLBackendProfile, SnowflakeBackendProfile, - PyIcebergRestBackendProfile, -) - - -def test_mysql_ssl_compose_full_dict(): - out = MySQLBackendProfile(HOST="h", PORT=3306, SSL_CA="/ca.pem", SSL_CIPHER="HIGH").emit(P.MYSQL) - # full equality: nested ssl ADDED, no flat ssl_* leaked, config unchanged - assert out == { - "host": "h", "port": 3306, "charset": "utf8mb4", - "collation": "utf8mb4_unicode_ci", "autocommit": True, - "ssl": {"ssl-ca": "/ca.pem", "ssl-cipher": "HIGH"}, - } - - -def test_mssql_host_fold_full_dict(): - out = MSSQLBackendProfile(HOST="srv", PORT=1433, INSTANCE_NAME="INST").emit(P.MSSQL) - assert out["host"] == "srv\\INST" and "instance_name" not in out - - -def test_snowflake_session_parameters_added_only(): - out = SnowflakeBackendProfile(ACCOUNT="acct", QUERY_TAG="etl", TIMEZONE="UTC").emit(P.SNOWFLAKE) - assert out["session_parameters"] == {"QUERY_TAG": "etl", "TIMEZONE": "UTC"} - assert "query_tag" not in out and "timezone" not in out - - -def test_pyiceberg_headers_expand_s3_flat(): - out = PyIcebergRestBackendProfile( - CATALOG_NAME="c", CATALOG_URI="http://x", S3_REGION="us-east-1", - HEADERS={"X-A": "1", "X-B": "2"}, - ).emit(P.PYICEBERG_REST) - assert out["name"] == "c" and out["uri"] == "http://x" and out["s3.region"] == "us-east-1" - assert out["header.X-A"] == "1" and out["header.X-B"] == "2" and "headers" not in out -``` -> Confirm exact flat defaults in `test_mysql_ssl_compose_full_dict` against `mysql.py` params (charset/collation/autocommit). If they differ, fix the EXPECTED dict to match the real spec — that is reading ground truth, not weakening the test. - -- [ ] **Step 2: Run to verify it fails** - -Run: `hatch run test:test-target tests/test_unit/core/settings/test_config_shaping.py -q` -Expected: FAIL — composes not wired. - -- [ ] **Step 3: Write the compose functions** - -`core/settings/adapters/mysql.py` (replace the file's old `build_driver_kwargs`): -```python -"""MySQL config-shaping adapter.""" -from __future__ import annotations -import typing as t - - -def ssl_compose(profile: t.Any, base: dict[str, t.Any]) -> dict[str, t.Any]: - out = dict(base) - if profile.SSL_MODE is not None: - out["ssl_mode"] = str(profile.SSL_MODE) - ssl: dict[str, str] = {} - for key, val in { - "ssl-key": profile.SSL_KEY, "ssl-cert": profile.SSL_CERT, - "ssl-ca": profile.SSL_CA, "ssl-capath": profile.SSL_CAPATH, - "ssl-cipher": profile.SSL_CIPHER, - }.items(): - if val is not None: - ssl[key] = str(val) - if ssl: - out["ssl"] = ssl - return out -``` -`core/settings/adapters/mssql.py` (compose part — auth fns appended in Task 4): -```python -"""MSSQL adapters.""" -from __future__ import annotations -import typing as t - - -def host_fold(profile: t.Any, base: dict[str, t.Any]) -> dict[str, t.Any]: - out = dict(base) - if profile.INSTANCE_NAME: - out["host"] = f"{out['host']}\\{profile.INSTANCE_NAME}" - if profile.ENCRYPTION is not None: - out["encrypt"] = str(profile.ENCRYPTION) - if profile.TRUST_SERVER_CERTIFICATE: - out["trust_server_certificate"] = "yes" - if profile.MARS_ENABLED: - out["mars_connection"] = "yes" - return out -``` -`core/settings/adapters/snowflake.py` (compose part — auth fns appended in Task 4): -```python -"""Snowflake adapters.""" -from __future__ import annotations -import typing as t - - -def session_params(profile: t.Any, base: dict[str, t.Any]) -> dict[str, t.Any]: - out = dict(base) - params: dict[str, t.Any] = {} - if profile.QUERY_TAG is not None: - params["QUERY_TAG"] = profile.QUERY_TAG - if profile.TIMEZONE is not None: - params["TIMEZONE"] = profile.TIMEZONE - if params: - out["session_parameters"] = params - return out -``` -`core/settings/adapters/pyiceberg_rest.py` (compose part — auth fn appended in Task 4): -```python -"""PyIceberg REST adapters.""" -from __future__ import annotations -import typing as t - - -def headers_compose(profile: t.Any, base: dict[str, t.Any]) -> dict[str, t.Any]: - out = dict(base) - if profile.HEADERS: - for hk, hv in profile.HEADERS.items(): - out[f"header.{hk}"] = hv - return out -``` - -- [ ] **Step 4: Wire `__adapters__` + pyiceberg `driver_key`s** - -`mysql.py`: `from .adapters import mysql as _mysql` and in the class body `__adapters__ = {CONST_DB_PROVIDER_TYPE.MYSQL: _mysql.ssl_compose}`. -`mssql.py`: `from .adapters import mssql as _mssql`; `__adapters__ = {CONST_DB_PROVIDER_TYPE.MSSQL: _mssql.host_fold}`. -`snowflake.py`: `from .adapters import snowflake as _snow`; `__adapters__ = {CONST_DB_PROVIDER_TYPE.SNOWFLAKE: _snow.session_params}`. -`pyiceberg_rest.py`: add `driver_key`s to the s3/rest params (`S3_REGION→"s3.region"`, `S3_ENDPOINT→"s3.endpoint"`, `S3_ACCESS_KEY_ID→"s3.access-key-id"`, `S3_SECRET_ACCESS_KEY→"s3.secret-access-key"` keep `secret=True`, `S3_SESSION_TOKEN→"s3.session-token"` keep `secret=True`, `REST_SIGV4_ENABLED→"rest.sigv4-enabled"`, `REST_SIGNING_REGION→"rest.signing-region"`, `REST_SIGNING_NAME→"rest.signing-name"`; `HEADERS` keeps NO driver_key), then `from .adapters import pyiceberg_rest as _ice`; `__adapters__ = {CONST_DB_PROVIDER_TYPE.PYICEBERG_REST: _ice.headers_compose}`. - -- [ ] **Step 5: Run goldens** - -Run: `hatch run test:test-target tests/test_unit/core/settings/test_config_shaping.py -q` -Expected: PASS. - -- [ ] **Step 6: Commit** -```bash -git add core/settings/ -git commit -m "feat(settings): config-shaping compose adapters (mysql/mssql/snowflake/pyiceberg) - -Co-Authored-By: Claude Opus 4.8 (1M context) " -``` - ---- - -## Task 4: Auth adapter functions - -**Files:** -- Create: `core/settings/adapters/sql.py` -- Append auth fns to `core/settings/adapters/{trino,snowflake,mssql,redshift,databricks,bigquery,pyiceberg_rest}.py` -- Test: `tests/test_unit/core/settings/adapters/test_auth_adapters.py` - -**Interfaces:** Each is `(auth_profile, base: dict) -> dict`, returns a NEW dict, never mutates `base`. Produces: `sql.userpass`; `trino.{password,jwt,kerberos}`; `snowflake.{password,token,oauth2,certificate}`; `mssql.{password,windows,azure_ad}`; `redshift.{password,iam}`; `databricks.{token,password}`; `bigquery.service_account`; `pyiceberg_rest.token`. - -> `sql.userpass` emits `{user,password}` — confirmed correct for all 9 flat backends (ibis `do_connect`); databricks uses `{username,password}` via its own adapter. - -- [ ] **Step 1: Write failing tests** - -`tests/test_unit/core/settings/adapters/test_auth_adapters.py`: -```python -import pytest -from mountainash_auth_client import ( - PasswordAuthProfile, TokenAuthProfile, OAuth2AuthProfile, - CertificateAuthProfile, WindowsAuthProfile, AzureADAuthProfile, - IAMAuthProfile, ServiceAccountAuthProfile, -) -from mountainash_data.core.settings.adapters import ( - sql as _sql, snowflake as _snow, mssql as _mssql, - redshift as _rs, databricks as _dbx, pyiceberg_rest as _ice, -) - - -def test_sql_userpass(): - assert _sql.userpass(PasswordAuthProfile(USERNAME="u", PASSWORD="p"), {"host": "h"}) == { - "host": "h", "user": "u", "password": "p"} - - -def test_userpass_no_mutate(): - base = {"host": "h"} - _sql.userpass(PasswordAuthProfile(USERNAME="u", PASSWORD="p"), base) - assert base == {"host": "h"} - - -def test_snowflake_token_oauth(): - assert _snow.token(TokenAuthProfile(TOKEN="t"), {}) == {"authenticator": "oauth", "token": "t"} - - -def test_snowflake_oauth2_token_only(): - assert _snow.oauth2(OAuth2AuthProfile(TOKEN="t"), {}) == {"authenticator": "oauth", "token": "t"} - - -def test_snowflake_password(): - assert _snow.password(PasswordAuthProfile(USERNAME="u", PASSWORD="p"), {}) == {"user": "u", "password": "p"} - - -def test_snowflake_certificate(): - assert _snow.certificate(CertificateAuthProfile(PRIVATE_KEY="KEY", PASSPHRASE="ph"), {}) == { - "private_key": "KEY", "private_key_file_pwd": "ph"} - - -def test_mssql_password(): - assert _mssql.password(PasswordAuthProfile(USERNAME="u", PASSWORD="p"), {}) == {"user": "u", "password": "p"} - - -def test_mssql_windows(): - assert _mssql.windows(WindowsAuthProfile(USERNAME="u", DOMAIN="D"), {}) == { - "trusted_connection": "yes", "user": "D\\u"} - - -def test_mssql_azure_ad_sp(): - assert _mssql.azure_ad(AzureADAuthProfile(CLIENT_ID="cid", CLIENT_SECRET="sec", TENANT_ID="t"), {}) == { - "authentication": "ActiveDirectoryServicePrincipal", "user_id": "cid", - "password": "sec", "tenant_id": "t"} - - -def test_redshift_iam(): - assert _rs.iam(IAMAuthProfile(ROLE_ARN="arn", ACCESS_KEY_ID="ak"), {}) == { - "iam": True, "iam_role_arn": "arn", "aws_access_key_id": "ak"} - - -def test_databricks_token(): - assert _dbx.token(TokenAuthProfile(TOKEN="tok"), {}) == {"access_token": "tok"} - - -def test_pyiceberg_token(): - assert _ice.token(TokenAuthProfile(TOKEN="tok"), {"uri": "u"}) == {"uri": "u", "token": "tok"} - - -def test_trino_password_builds_basic_auth(): - pytest.importorskip("trino") - from trino.auth import BasicAuthentication - from mountainash_data.core.settings.adapters import trino as _trino - out = _trino.password(PasswordAuthProfile(USERNAME="u", PASSWORD="p"), {"host": "h"}) - assert out["host"] == "h" and out["user"] == "u" and isinstance(out["auth"], BasicAuthentication) - - -def test_bigquery_service_account(monkeypatch): - pytest.importorskip("google.oauth2") - from google.oauth2 import service_account as _sa - from mountainash_data.core.settings.adapters import bigquery as _bq - sentinel = object() - monkeypatch.setattr(_sa.Credentials, "from_service_account_info", classmethod(lambda cls, info: sentinel)) - assert _bq.service_account(ServiceAccountAuthProfile(INFO={"k": "v"}), {}) == {"credentials": sentinel} -``` - -- [ ] **Step 2: Run to verify it fails** - -Run: `hatch run test:test-target tests/test_unit/core/settings/adapters/test_auth_adapters.py -q` -Expected: FAIL. - -- [ ] **Step 3: Implement `sql.py`** -```python -"""Shared auth adapter for flat user/password SQL backends.""" -from __future__ import annotations -import typing as t - - -def userpass(auth: t.Any, base: dict[str, t.Any]) -> dict[str, t.Any]: - return {**base, "user": auth.USERNAME, "password": auth.PASSWORD.get_secret_value()} -``` - -- [ ] **Step 4: Append auth fns to the per-backend adapter modules** - -Append to `snowflake.py`: -```python -def password(auth, base): - return {**base, "user": auth.USERNAME, "password": auth.PASSWORD.get_secret_value()} - - -def token(auth, base): - return {**base, "authenticator": "oauth", "token": auth.TOKEN.get_secret_value()} - - -def oauth2(auth, base): - # token-only: never reads CLIENT_ID/SECRET/SERVER_URI/SCOPE (smell #1) - return {**base, "authenticator": "oauth", "token": auth.TOKEN.get_secret_value()} - - -def certificate(auth, base): - out = dict(base) - if auth.PRIVATE_KEY is not None: - out["private_key"] = auth.PRIVATE_KEY.get_secret_value() - if auth.PRIVATE_KEY_PATH is not None: - out["private_key_file"] = str(auth.PRIVATE_KEY_PATH) - if auth.PASSPHRASE is not None: - out["private_key_file_pwd"] = auth.PASSPHRASE.get_secret_value() - return out -``` -Append to `mssql.py`: -```python -def password(auth, base): - return {**base, "user": auth.USERNAME, "password": auth.PASSWORD.get_secret_value()} - - -def windows(auth, base): - out = {**base, "trusted_connection": "yes"} - if auth.DOMAIN and auth.USERNAME: - out["user"] = f"{auth.DOMAIN}\\{auth.USERNAME}" - elif auth.USERNAME: - out["user"] = auth.USERNAME - return out - - -def azure_ad(auth, base): - out = dict(base) - if auth.MANAGED_IDENTITY: - out["authentication"] = "ActiveDirectoryMsi" - if auth.MSI_ENDPOINT: - out["msi_endpoint"] = auth.MSI_ENDPOINT - else: - out["authentication"] = "ActiveDirectoryServicePrincipal" - if auth.CLIENT_ID: - out["user_id"] = auth.CLIENT_ID - if auth.CLIENT_SECRET: - out["password"] = auth.CLIENT_SECRET.get_secret_value() - if auth.TENANT_ID: - out["tenant_id"] = auth.TENANT_ID - return out -``` -Append to `pyiceberg_rest.py`: -```python -def token(auth, base): - return {**base, "token": auth.TOKEN.get_secret_value()} -``` -Create/replace `redshift.py`, `databricks.py`, `trino.py`, `bigquery.py` (these are NOT shaping, so the whole file is auth fns): -```python -# redshift.py -def password(auth, base): - return {**base, "user": auth.USERNAME, "password": auth.PASSWORD.get_secret_value()} - - -def iam(auth, base): - out = {**base, "iam": True} - if auth.ROLE_ARN is not None: out["iam_role_arn"] = auth.ROLE_ARN - if auth.ACCESS_KEY_ID is not None: out["aws_access_key_id"] = auth.ACCESS_KEY_ID - if auth.SECRET_ACCESS_KEY is not None: out["aws_secret_access_key"] = auth.SECRET_ACCESS_KEY.get_secret_value() - if auth.SESSION_TOKEN is not None: out["aws_session_token"] = auth.SESSION_TOKEN.get_secret_value() - if auth.PROFILE_NAME is not None: out["profile_name"] = auth.PROFILE_NAME - return out -``` -```python -# databricks.py -def token(auth, base): - return {**base, "access_token": auth.TOKEN.get_secret_value()} - - -def password(auth, base): - return {**base, "username": auth.USERNAME, "password": auth.PASSWORD.get_secret_value()} -``` -```python -# trino.py -def password(auth, base): - from trino.auth import BasicAuthentication - return {**base, "user": auth.USERNAME, - "auth": BasicAuthentication(auth.USERNAME, auth.PASSWORD.get_secret_value())} - - -def jwt(auth, base): - from trino.auth import JWTAuthentication - return {**base, "auth": JWTAuthentication(auth.TOKEN.get_secret_value())} - - -def kerberos(auth, base): - from trino.auth import KerberosAuthentication - return {**base, "auth": KerberosAuthentication(config=None, service_name=auth.SERVICE_NAME, principal=auth.PRINCIPAL)} -``` -```python -# bigquery.py -def service_account(auth, base): - from google.oauth2 import service_account as _sa - out = dict(base) - if auth.INFO is not None: - out["credentials"] = _sa.Credentials.from_service_account_info(auth.INFO) - elif auth.FILE is not None: - out["credentials"] = _sa.Credentials.from_service_account_file(str(auth.FILE)) - return out -``` -(Each file starts with `from __future__ import annotations` and `import typing as t` where types are referenced; the shaping files keep their compose fn from Task 3.) - -- [ ] **Step 5: Run tests** - -Run: `hatch run test:test-target tests/test_unit/core/settings/adapters/test_auth_adapters.py -q` -Expected: PASS (trino/bigquery skip without extras). - -- [ ] **Step 6: Commit** -```bash -git add core/settings/adapters/ tests/test_unit/core/settings/adapters/test_auth_adapters.py -git commit -m "feat(settings): data-owned auth adapter functions - -Co-Authored-By: Claude Opus 4.8 (1M context) " -``` - ---- - -## Task 5: Auth dispatch registry (MRO-aware) - -**Files:** Create `core/settings/adapters/registry.py`; Test `tests/test_unit/core/settings/adapters/test_registry.py`. - -**Interfaces:** Produces `auth_adapter(provider_type, auth_class) -> Callable | None` (MRO-aware; `TypeError` on sibling ambiguity); `_AUTH_ADAPTERS`. - -- [ ] **Step 1: Write failing tests** - -`tests/test_unit/core/settings/adapters/test_registry.py`: -```python -import pytest -from mountainash_auth_client import PasswordAuthProfile, TokenAuthProfile, NoAuthProfile -from mountainash_data.core.constants import CONST_DB_PROVIDER_TYPE as P -from mountainash_data.core.settings.adapters import sql as _sql, snowflake as _snow -from mountainash_data.core.settings.adapters.registry import auth_adapter, _AUTH_ADAPTERS - - -def test_exact_lookup(): - assert auth_adapter(P.SNOWFLAKE, TokenAuthProfile) is _snow.token - - -def test_flat_userpass_shared(): - assert auth_adapter(P.POSTGRESQL, PasswordAuthProfile) is _sql.userpass - - -def test_miss_returns_none(): - assert auth_adapter(P.SQLITE, PasswordAuthProfile) is None - - -def test_noauth_not_in_table(): - assert all(k[1] is not NoAuthProfile for k in _AUTH_ADAPTERS) - - -def test_subclass_resolves_to_base(): - class MyPw(PasswordAuthProfile): pass - assert auth_adapter(P.POSTGRESQL, MyPw) is _sql.userpass - - -def test_specialization_wins(): - fn = lambda a, b: b - class Special(PasswordAuthProfile): pass - _AUTH_ADAPTERS[(P.POSTGRESQL, Special)] = fn - try: - assert auth_adapter(P.POSTGRESQL, Special) is fn - finally: - del _AUTH_ADAPTERS[(P.POSTGRESQL, Special)] - - -def test_sibling_ambiguity_raises(): - fn = lambda a, b: b - _AUTH_ADAPTERS[(P.POSTGRESQL, TokenAuthProfile)] = fn - class Hybrid(PasswordAuthProfile, TokenAuthProfile): pass - try: - with pytest.raises(TypeError, match="ambiguous"): - auth_adapter(P.POSTGRESQL, Hybrid) - finally: - del _AUTH_ADAPTERS[(P.POSTGRESQL, TokenAuthProfile)] -``` - -- [ ] **Step 2: Run to verify it fails** - -Run: `hatch run test:test-target tests/test_unit/core/settings/adapters/test_registry.py -q` -Expected: FAIL — module missing. - -- [ ] **Step 3: Implement `registry.py`** -```python -"""Data-owned auth dispatch: (provider_type, auth_class) -> adapter fn.""" -from __future__ import annotations -import typing as t - -from mountainash_auth_client import ( - PasswordAuthProfile, JWTAuthProfile, KerberosAuthProfile, - ServiceAccountAuthProfile, IAMAuthProfile, TokenAuthProfile, - OAuth2AuthProfile, CertificateAuthProfile, WindowsAuthProfile, AzureADAuthProfile, -) -from mountainash_data.core.constants import CONST_DB_PROVIDER_TYPE as P -from . import (sql as _sql, trino as _trino, snowflake as _snow, bigquery as _bq, - databricks as _dbx, mssql as _mssql, redshift as _rs, pyiceberg_rest as _ice) - -_AUTH_ADAPTERS: dict[tuple[t.Any, type], t.Callable[[t.Any, dict], dict]] = { - (P.TRINO, PasswordAuthProfile): _trino.password, - (P.TRINO, JWTAuthProfile): _trino.jwt, - (P.TRINO, KerberosAuthProfile): _trino.kerberos, - (P.SNOWFLAKE, PasswordAuthProfile): _snow.password, - (P.SNOWFLAKE, TokenAuthProfile): _snow.token, - (P.SNOWFLAKE, OAuth2AuthProfile): _snow.oauth2, - (P.SNOWFLAKE, CertificateAuthProfile): _snow.certificate, - (P.BIGQUERY, ServiceAccountAuthProfile): _bq.service_account, - (P.DATABRICKS, TokenAuthProfile): _dbx.token, - (P.DATABRICKS, PasswordAuthProfile): _dbx.password, - (P.MSSQL, PasswordAuthProfile): _mssql.password, - (P.MSSQL, WindowsAuthProfile): _mssql.windows, - (P.MSSQL, AzureADAuthProfile): _mssql.azure_ad, - (P.REDSHIFT, PasswordAuthProfile): _rs.password, - (P.REDSHIFT, IAMAuthProfile): _rs.iam, - (P.PYICEBERG_REST, TokenAuthProfile): _ice.token, -} -for _p in (P.POSTGRESQL, P.MYSQL, P.CLICKHOUSE, P.MATERIALIZE, P.RISINGWAVE, - P.DRUID, P.SINGLESTOREDB, P.IMPALA, P.EXASOL): - _AUTH_ADAPTERS[(_p, PasswordAuthProfile)] = _sql.userpass - - -def auth_adapter(provider_type: t.Any, auth_class: type) -> t.Callable[[t.Any, dict], dict] | None: - matches = [k for k in auth_class.__mro__ if (provider_type, k) in _AUTH_ADAPTERS] - if not matches: - return None - winner = matches[0] - ambiguous = [k for k in matches[1:] if not issubclass(winner, k)] - if ambiguous: - raise TypeError( - f"ambiguous auth adapter for {auth_class.__name__} on {provider_type}: " - f"{winner.__name__} vs {[k.__name__ for k in ambiguous]} " - f"(multiply-inherits unrelated registered auth types)" - ) - return _AUTH_ADAPTERS[(provider_type, winner)] -``` - -- [ ] **Step 4: Run tests** - -Run: `hatch run test:test-target tests/test_unit/core/settings/adapters/test_registry.py -q` -Expected: PASS. - -- [ ] **Step 5: Commit** -```bash -git add core/settings/adapters/registry.py tests/test_unit/core/settings/adapters/test_registry.py -git commit -m "feat(settings): MRO-aware auth dispatch registry - -Co-Authored-By: Claude Opus 4.8 (1M context) " -``` - ---- - -## Task 6: ConnectionFactory (compose + URL + non-profile auth) - -**Files:** Create `core/factories/__init__.py`, `core/factories/connection_factory.py`; Test `tests/test_unit/core/factories/test_connection_factory.py`. - -**Interfaces:** -- Consumes: `auth_adapter` (Task 5); `UrlParts` (from `core.settings.profile`); a profile with `.emit(target)`, `.to_url_parts()`, `.__spec__.{provider_type,supported_auth}`, `.backend`. -- Produces: - - `_normalize_and_validate_auth(profile, auth) -> AuthProfile` - - `build_driver_kwargs(profile, auth_profile=None) -> dict` - - `build_connection_string(profile, auth_profile=None) -> str` - - `apply_auth_adapter(provider_type, base, auth_profile) -> dict` — non-profile auth application (for the ibis dialect/URL paths, no `supported_auth` to validate). - - `provider_for_dialect(dialect) -> provider_type`, `provider_for_scheme(scheme) -> provider_type` — derived from the registered specs. - -- [ ] **Step 1: Write failing tests** - -`tests/test_unit/core/factories/test_connection_factory.py`: -```python -import pytest -from dataclasses import dataclass - -from mountainash_auth_client import ( - PasswordAuthProfile, TokenAuthProfile, NoAuthProfile, WindowsAuthProfile, -) -from mountainash_data.core.constants import CONST_DB_PROVIDER_TYPE as P -from mountainash_data.core.settings.profile import UrlParts -from mountainash_data.core.factories.connection_factory import ( - build_driver_kwargs, build_connection_string, _normalize_and_validate_auth, - apply_auth_adapter, provider_for_dialect, -) - - -@dataclass -class _Spec: - provider_type: object - supported_auth: tuple - name: str = "stub" - - -class _Stub: - def __init__(self, pt, sa, base, url=None): - self.__spec__ = _Spec(pt, sa) - self._base, self._url = base, url or UrlParts(scheme="stub", host="h", port=1, database="db") - @property - def backend(self): return self.__spec__.name - def emit(self, target): - assert target is self.__spec__.provider_type - return dict(self._base) - def to_url_parts(self): return self._url - - -def test_noauth_short_circuits(): - assert build_driver_kwargs(_Stub(P.SQLITE, (NoAuthProfile,), {"database": ":memory:"}), None) == {"database": ":memory:"} - - -def test_password_dispatch(): - out = build_driver_kwargs(_Stub(P.POSTGRESQL, (PasswordAuthProfile, NoAuthProfile), {"host": "h"}), - PasswordAuthProfile(USERNAME="u", PASSWORD="p")) - assert out == {"host": "h", "user": "u", "password": "p"} - - -def test_unsupported_auth_valueerror(): - with pytest.raises(ValueError, match="does not support auth"): - build_driver_kwargs(_Stub(P.SQLITE, (NoAuthProfile,), {}), PasswordAuthProfile(USERNAME="u", PASSWORD="p")) - - -def test_supported_but_no_adapter_fails_closed(): - with pytest.raises(ValueError, match="no auth adapter"): - build_driver_kwargs(_Stub(P.POSTGRESQL, (WindowsAuthProfile,), {"host": "h"}), WindowsAuthProfile(USERNAME="u")) - - -def test_none_normalizes_when_supported(): - assert isinstance(_normalize_and_validate_auth(_Stub(P.SQLITE, (NoAuthProfile,), {}), None), NoAuthProfile) - - -def test_none_rejected_when_noauth_unsupported(): - with pytest.raises(ValueError, match="does not support auth"): - _normalize_and_validate_auth(_Stub(P.MYSQL, (PasswordAuthProfile,), {}), None) - - -def test_apply_auth_adapter_non_profile(): - out = apply_auth_adapter(P.POSTGRESQL, {"host": "h"}, PasswordAuthProfile(USERNAME="u", PASSWORD="p")) - assert out == {"host": "h", "user": "u", "password": "p"} - assert apply_auth_adapter(P.POSTGRESQL, {"host": "h"}, None) == {"host": "h"} - - -def test_provider_for_dialect(): - assert provider_for_dialect("postgres") is P.POSTGRESQL - - -def test_url_password(): - s = _Stub(P.POSTGRESQL, (PasswordAuthProfile,), {}, url=UrlParts(scheme="postgresql", host="db", port=5432, database="app")) - assert build_connection_string(s, PasswordAuthProfile(USERNAME="u", PASSWORD="p@s")) == "postgresql://u:p%40s@db:5432/app" - - -def test_url_token_authority_less(): - s = _Stub(P.MOTHERDUCK, (TokenAuthProfile,), {}, url=UrlParts(scheme="md", database="mydb")) - assert build_connection_string(s, TokenAuthProfile(TOKEN="T")) == "md:mydb?motherduck_token=T" - - -@pytest.mark.parametrize("auth", [WindowsAuthProfile(USERNAME="u"), TokenAuthProfile(TOKEN="T")]) -def test_url_unsupported_auth_not_implemented(auth): - s = _Stub(P.POSTGRESQL, (type(auth),), {}, url=UrlParts(scheme="postgresql", host="db")) - with pytest.raises(NotImplementedError): - build_connection_string(s, auth) -``` - -- [ ] **Step 2: Run to verify it fails** - -Run: `hatch run test:test-target tests/test_unit/core/factories/test_connection_factory.py -q` -Expected: FAIL — module missing. - -- [ ] **Step 3: Create `core/factories/__init__.py`** -```python -"""Factories that compose backend config + auth into runtime kwargs.""" -``` - -- [ ] **Step 4: Implement `connection_factory.py`** -```python -"""ConnectionFactory: compose BackendProfile config + AuthProfile creds.""" -from __future__ import annotations -import typing as t -from urllib.parse import quote - -from mountainash_auth_client import NoAuthProfile, PasswordAuthProfile, TokenAuthProfile, AuthProfile -from mountainash_data.core.constants import CONST_DB_PROVIDER_TYPE as P -from mountainash_data.core.settings.profile import UrlParts -from mountainash_data.core.settings.adapters.registry import auth_adapter - - -def _iter_specs() -> t.Iterator[t.Any]: - """Registered BackendSpecs, regardless of whether the registry stores - specs or profile classes.""" - from mountainash_data.core.settings.registry import REGISTRY - for v in REGISTRY.values(): - yield v.__spec__ if hasattr(v, "__spec__") else v - - -def provider_for_dialect(dialect: str) -> t.Any: - for spec in _iter_specs(): - if getattr(spec, "ibis_dialect", None) == dialect: - return spec.provider_type - raise KeyError(f"no provider_type for ibis dialect {dialect!r}") - - -def provider_for_scheme(scheme: str) -> t.Any: - norm = scheme.rstrip(":/") - for spec in _iter_specs(): - s = getattr(spec, "connection_string_scheme", None) - if s and s.rstrip(":/") == norm: - return spec.provider_type - raise KeyError(f"no provider_type for URL scheme {scheme!r}") - - -def _normalize_and_validate_auth(profile: t.Any, auth_profile: AuthProfile | None) -> AuthProfile: - auth = NoAuthProfile() if auth_profile is None else auth_profile - if not isinstance(auth, tuple(profile.__spec__.supported_auth)): - raise ValueError(f"{profile.backend} does not support auth: {type(auth).__name__}") - return auth - - -def apply_auth_adapter(provider_type: t.Any, base: dict, auth_profile: AuthProfile | None) -> dict: - """Apply auth WITHOUT a profile (ibis dialect / URL paths). No supported_auth gate.""" - if auth_profile is None or isinstance(auth_profile, NoAuthProfile): - return base - fn = auth_adapter(provider_type, type(auth_profile)) - if fn is None: - raise ValueError(f"{provider_type}: no auth adapter for {type(auth_profile).__name__}") - return fn(auth_profile, base) - - -def build_driver_kwargs(profile: t.Any, auth_profile: AuthProfile | None = None) -> dict: - auth = _normalize_and_validate_auth(profile, auth_profile) - target = profile.__spec__.provider_type - base = profile.emit(target) - if isinstance(auth, NoAuthProfile): - return base - return apply_auth_adapter(target, base, auth) - - -# --- URL appliers (L3 for the URL target) --------------------------------- - -def _url_password(parts: UrlParts, auth: t.Any) -> str: - if parts.host is None: - raise NotImplementedError("password URL form requires a host authority") - user, pw = quote(str(auth.USERNAME), safe=""), quote(auth.PASSWORD.get_secret_value(), safe="") - url = f"{parts.scheme}://{user}:{pw}@{parts.host}" - if parts.port is not None: url += f":{parts.port}" - if parts.database is not None: url += f"/{parts.database}" - return url - - -def _url_noauth(parts: UrlParts) -> str: - url = parts.scheme + "://" - if parts.host is not None: - url += parts.host + (f":{parts.port}" if parts.port is not None else "") - if parts.database is not None: url += f"/{parts.database}" - return url - - -def _url_motherduck_token(parts: UrlParts, auth: t.Any) -> str: - return f"{parts.scheme}:{parts.database}?motherduck_token={auth.TOKEN.get_secret_value()}" - - -_URL_APPLIERS: dict[t.Any, dict[type, t.Callable]] = { - P.MOTHERDUCK: {TokenAuthProfile: _url_motherduck_token}, -} - - -def build_connection_string(profile: t.Any, auth_profile: AuthProfile | None = None) -> str: - auth = _normalize_and_validate_auth(profile, auth_profile) - parts = profile.to_url_parts() # L1 - if isinstance(auth, NoAuthProfile): - return _url_noauth(parts) - if isinstance(auth, PasswordAuthProfile): - return _url_password(parts, auth) # L3 - applier = _URL_APPLIERS.get(profile.__spec__.provider_type, {}).get(type(auth)) - if applier is None: - raise NotImplementedError(f"{profile.backend}: no URL form for {type(auth).__name__}") - return applier(parts, auth) -``` -> Confirm the registry accessor name in `core/settings/registry.py` (`REGISTRY` vs `DATABASES_REGISTRY`) and adjust `_iter_specs`. The `hasattr(v, "__spec__")` branch handles either specs or classes. - -- [ ] **Step 5: Run tests** - -Run: `hatch run test:test-target tests/test_unit/core/factories/test_connection_factory.py -q` -Expected: PASS. - -- [ ] **Step 6: Commit** -```bash -git add core/factories/ tests/test_unit/core/factories/test_connection_factory.py -git commit -m "feat(factories): ConnectionFactory compose, URL appliers, non-profile auth - -Co-Authored-By: Claude Opus 4.8 (1M context) " -``` - ---- - -## Task 7: Ibis entry points (deferred auth across all three paths) - -**Files:** Modify `backends/ibis/backend.py`; Test `tests/test_unit/backends/ibis/test_backend_auth.py`. - -**Interfaces:** Produces `IbisBackend.connect(self, auth_profile=None)` applying auth on the **settings**, **direct-dialect**, and **URL** paths; fail-closed URL-creds-vs-explicit precedence. - -- [ ] **Step 1: Write failing tests** - -`tests/test_unit/backends/ibis/test_backend_auth.py`: -```python -import pytest -from mountainash_auth_client import NoAuthProfile, PasswordAuthProfile -from mountainash_data.backends.ibis.backend import IbisBackend - - -def test_sqlite_dialect_connect_noauth(tmp_path): - be = IbisBackend(dialect="sqlite", database=str(tmp_path / "t.db")).connect(auth_profile=NoAuthProfile()) - assert be is not None - - -def test_dialect_path_applies_password(monkeypatch): - # direct-dialect + explicit auth: auth adapter must run for the dialect's provider. - seen = {} - import mountainash_data.backends.ibis.backend as mod - def fake_apply(pt, base, auth): - seen["pt"], seen["auth"] = pt, auth - return {**base, "user": auth.USERNAME} - monkeypatch.setattr(mod, "apply_auth_adapter", fake_apply) - monkeypatch.setattr(mod, "provider_for_dialect", lambda d: "PG") - IbisBackend(dialect="postgres", host="h", database="db")._resolve_dialect_auth( - PasswordAuthProfile(USERNAME="u", PASSWORD="p") - ) - assert seen["pt"] == "PG" and seen["auth"].USERNAME == "u" - - -def test_url_and_explicit_auth_conflict_raises(): - with pytest.raises(ValueError, match="both"): - IbisBackend("postgresql://u:p@host/db").connect( - auth_profile=PasswordAuthProfile(USERNAME="x", PASSWORD="y")) -``` -> The settings-path end-to-end (`SettingsParameters` → `connect(auth_profile=...)`) is added in Task 9 with the migrated fixtures. - -- [ ] **Step 2: Run to verify it fails** - -Run: `hatch run test:test-target tests/test_unit/backends/ibis/test_backend_auth.py -q` -Expected: FAIL — `connect()` takes no `auth_profile`; `_resolve_dialect_auth` missing. - -- [ ] **Step 3: Defer config build in `_init_from_settings`** - -In `backends/ibis/backend.py`, DELETE the eager `driver_kwargs = obj_settings.to_driver_kwargs()` (line ~242). Store the profile + extras instead: -```python - self.dialect = resolved_dialect - self._spec = DIALECTS[resolved_dialect] - self._url = None - self._profile = obj_settings # settings path - self._extra_config = config # caller **config overrides - self._config = None - self._conn = None -``` -In the direct-dialect path (`_init_from_dialect`), set `self._profile = None`, `self._url = None`, `self._dialect_config = config`, `self._config = None`. In the URL path set `self._profile = None`, keep `self._url = `, `self._config = None`. - -- [ ] **Step 4: Add imports + helpers + thread `connect`** -```python -from mountainash_data.core.factories.connection_factory import ( - build_driver_kwargs, apply_auth_adapter, provider_for_dialect, provider_for_scheme, -) -from mountainash_auth_client import PasswordAuthProfile -from urllib.parse import urlsplit, urlunsplit, unquote - - -def connect(self, auth_profile=None): - if self._conn is not None: - return self - if self._profile is not None: # settings path - cfg = build_driver_kwargs(self._profile, auth_profile) - cfg.update(self._extra_config) - self._config = cfg - elif self._url is not None: # URL path - self._config, self._url = self._resolve_url_auth(self._url, auth_profile) - else: # direct-dialect path - self._config = self._resolve_dialect_auth(auth_profile) - # ...existing connection_builder / ibis.connect(self._url, **self._config) logic... - return self - - -def _resolve_dialect_auth(self, auth_profile): - base = dict(self._dialect_config) - if auth_profile is None: - return base - provider = provider_for_dialect(self.dialect) - return apply_auth_adapter(provider, base, auth_profile) - - -def _resolve_url_auth(self, url, auth_profile): - parts = urlsplit(url) - has_url_creds = bool(parts.username) - if has_url_creds and auth_profile is not None: - raise ValueError("both URL credentials and an explicit auth_profile given") - config: dict = {} - clean = url - if has_url_creds: - netloc = parts.hostname or "" - if parts.port: netloc += f":{parts.port}" - clean = urlunsplit((parts.scheme, netloc, parts.path, parts.query, parts.fragment)) - auth_profile = PasswordAuthProfile( - USERNAME=unquote(parts.username), - PASSWORD=unquote(parts.password) if parts.password else "", - ) - if auth_profile is not None: - provider = provider_for_scheme(parts.scheme) - config = apply_auth_adapter(provider, config, auth_profile) - return config, clean -``` -> The existing `connect` body that reads `self._config`/`self._url`/`self._spec.connection_builder` runs UNCHANGED after `self._config` is set above. Confirm no code path reads `self._config` before `connect()` (it is now `None` until `connect`). - -- [ ] **Step 5: Run tests** - -Run: `hatch run test:test-target tests/test_unit/backends/ibis/test_backend_auth.py -q` -Expected: PASS. - -- [ ] **Step 6: Commit** -```bash -git add backends/ibis/backend.py tests/test_unit/backends/ibis/test_backend_auth.py -git commit -m "feat(ibis): deferred auth across settings/dialect/URL paths - -Co-Authored-By: Claude Opus 4.8 (1M context) " -``` - ---- - -## Task 8: Iceberg auth threading (testable kwargs helper) - -**Files:** Modify `backends/iceberg/connection.py`; Test `tests/test_unit/backends/iceberg/test_iceberg_auth.py`. - -**Interfaces:** Produces `_build_catalog_kwargs(self, auth_profile, **kwargs) -> dict` (no pyiceberg import — real gate); `connect_default(self, *, auth_profile=None, **kwargs)` and `connect(..., auth_profile=None)` threading auth; precedence **profile-derived < explicit `**kwargs`**. - -- [ ] **Step 1: Write failing test (no live catalog needed)** - -`tests/test_unit/backends/iceberg/test_iceberg_auth.py`: -```python -from types import SimpleNamespace -from unittest.mock import patch -from mountainash_auth_client import TokenAuthProfile -from mountainash_data.backends.iceberg.connection import IcebergConnectionBase # confirm exact name - - -def test_build_catalog_kwargs_threads_auth_and_merges(): - obj_settings = object() - # plain stubs — no property/attribute conflict: get_settings returns obj_settings - params = SimpleNamespace( - settings_class=SimpleNamespace(get_settings=lambda settings_parameters: obj_settings) - ) - conn = IcebergConnectionBase.__new__(IcebergConnectionBase) - conn.db_auth_settings_parameters = params - - auth = TokenAuthProfile(TOKEN="T") - with patch( - "mountainash_data.backends.iceberg.connection.build_driver_kwargs", - return_value={"uri": "http://x", "token": "T", "name": "c"}, - ) as bk: - out = conn._build_catalog_kwargs(auth, warehouse="w") - - bk.assert_called_once_with(obj_settings, auth) # profile + auth_profile threaded - assert out["warehouse"] == "w" # explicit kwargs win - assert out["uri"] == "http://x" -``` -> Confirm the real `IcebergConnectionBase` class/attribute names at implementation time (the `.db_auth_settings_parameters` + `.settings_class.get_settings(...)` shape is from the current `connect_default`); the load-bearing assertions (profile+auth threaded; explicit kwargs win) stay. `build_driver_kwargs` is patched at the name bound INSIDE `connection.py`, not at its definition site. - -- [ ] **Step 2: Run to verify it fails** - -Run: `hatch run test:test-target tests/test_unit/backends/iceberg/test_iceberg_auth.py -q` -Expected: FAIL — `_build_catalog_kwargs` missing. - -- [ ] **Step 3: Extract the kwargs helper + thread auth** - -Replace line ~112's `connection_kwargs = obj_settings.to_driver_kwargs()` path: -```python -from mountainash_data.core.factories.connection_factory import build_driver_kwargs - - -def _build_catalog_kwargs(self, auth_profile=None, **kwargs): - settings_class = self.db_auth_settings_parameters.settings_class - if settings_class is None: - raise ValueError("Settings class is required for the database connection") - obj_settings = settings_class.get_settings(settings_parameters=self.db_auth_settings_parameters) - connection_kwargs = build_driver_kwargs(obj_settings, auth_profile) - connection_kwargs.update(kwargs) # explicit caller kwargs win - return connection_kwargs - - -def connect_default(self, *, auth_profile=None, **kwargs): - if self.catalog_backend is None: - connection_kwargs = self._build_catalog_kwargs(auth_profile, **kwargs) - from pyiceberg.catalog.rest import RestCatalog - self._catalog_backend = RestCatalog(**connection_kwargs) - return self.catalog_backend - - -def connect(self, connection_string=None, connection_kwargs=None, *, auth_profile=None, **kwargs): - if self.catalog_backend is None: - self.connect_default(auth_profile=auth_profile, **(connection_kwargs or {}), **kwargs) - return self.catalog_backend -``` -Document the precedence in both docstrings. - -- [ ] **Step 4: Run test** - -Run: `hatch run test:test-target tests/test_unit/backends/iceberg/test_iceberg_auth.py -q` -Expected: PASS (no pyiceberg needed — `RestCatalog` import is inside `connect_default`, not reached by the helper test). - -- [ ] **Step 5: Commit** -```bash -git add backends/iceberg/connection.py tests/test_unit/backends/iceberg/test_iceberg_auth.py -git commit -m "feat(iceberg): thread auth via testable _build_catalog_kwargs - -Co-Authored-By: Claude Opus 4.8 (1M context) " -``` - ---- - -## Task 9: Migrate existing tests + consistency goldens + green gate - -**Files:** Modify `tests/fixtures/settings_fixtures.py` + the ~25 test files; Create `tests/test_unit/core/settings/test_supported_auth_consistency.py`, `tests/test_unit/core/factories/test_url_consistency.py`; full-suite gate. - -- [ ] **Step 1: Migrate fixtures** - -`tests/fixtures/settings_fixtures.py`: replace `NoAuth`→`NoAuthProfile`, `SQLiteAuthSettings`→`SQLiteBackendProfile`, `DuckDBAuthSettings`→`DuckDBBackendProfile`. DELETE every `"auth": NoAuth()` from `kwargs={...}` (auth is no longer a profile field). Where a test needs auth, yield `(backend_profile, auth_profile)` pairs. - -- [ ] **Step 2: Migrate per-backend tests (mechanical)** - -For each `tests/test_unit/core/settings/backends/test_.py`: imports → `BackendProfile` + `*AuthProfile`; construction → `BackendProfile(...)` (no `auth=`), UPPERCASE auth kwargs with plain strings (`PasswordAuthProfile(USERNAME="u", PASSWORD="p")` — pydantic wraps secrets); replace `s.to_driver_kwargs()` with `build_driver_kwargs(s, )` from `mountainash_data.core.factories.connection_factory`. - -Worked example — `test_postgresql.py`: -```python -from mountainash_auth_client import PasswordAuthProfile -from mountainash_data.core.settings import PostgreSQLBackendProfile -from mountainash_data.core.factories.connection_factory import build_driver_kwargs - - -def test_postgres_driver_kwargs(): - s = PostgreSQLBackendProfile(HOST="db", DATABASE="app") - out = build_driver_kwargs(s, PasswordAuthProfile(USERNAME="u", PASSWORD="p")) - assert out["host"] == "db" and out["user"] == "u" and out["password"] == "p" -``` - -- [ ] **Step 3: supported_auth ↔ table consistency** - -`tests/test_unit/core/settings/test_supported_auth_consistency.py`: -```python -from mountainash_auth_client import NoAuthProfile -from mountainash_data.core.settings.adapters.registry import auth_adapter -from mountainash_data.core.factories.connection_factory import _iter_specs - - -def test_every_supported_pair_has_an_adapter(): - for spec in _iter_specs(): - for auth_cls in spec.supported_auth: - if auth_cls is NoAuthProfile: - continue - assert auth_adapter(spec.provider_type, auth_cls) is not None, ( - f"{spec.name}: supported {auth_cls.__name__} has no adapter" - ) -``` -> Uses `_iter_specs()` (Task 6) which normalises the registry; this is a structural invariant, NOT a count assertion. - -- [ ] **Step 4: URL applier coverage** - -`tests/test_unit/core/factories/test_url_consistency.py`: -```python -import pytest -from mountainash_auth_client import PasswordAuthProfile, TokenAuthProfile -from mountainash_data.core.settings import PostgreSQLBackendProfile, MotherDuckBackendProfile -from mountainash_data.core.factories.connection_factory import build_connection_string - - -def test_postgres_password_url(): - s = PostgreSQLBackendProfile(HOST="db", PORT=5432, DATABASE="app") - assert build_connection_string(s, PasswordAuthProfile(USERNAME="u", PASSWORD="p@s")) == "postgresql://u:p%40s@db:5432/app" - - -def test_motherduck_token_url(): - assert build_connection_string(MotherDuckBackendProfile(DATABASE="mydb"), TokenAuthProfile(TOKEN="T")) == "md:mydb?motherduck_token=T" - - -def test_snowflake_token_url_not_implemented(): - # snowflake supports TokenAuthProfile for kwargs but has no URL form → fail-closed - from mountainash_data.core.settings import SnowflakeBackendProfile - with pytest.raises(NotImplementedError): - build_connection_string(SnowflakeBackendProfile(ACCOUNT="a"), TokenAuthProfile(TOKEN="T")) -``` - -- [ ] **Step 5: Migrate remaining unit/integration tests** - -`tests/test_integration/test_end_to_end_workflows.py`, `tests/test_unit/backends/ibis/test_backend.py`, `tests/test_unit/core/settings/test_{descriptor,profile,registry}.py`, `tests/test_unit/databases/settings/test_settings_parametrized.py`: swap to new names; move any `auth=` on a profile to the `connect(auth_profile=...)` / `build_driver_kwargs(profile, auth)` call. Add the settings-path ibis auth test deferred from Task 7 (a `SQLiteBackendProfile` via `SettingsParameters` through `connect(auth_profile=NoAuthProfile())`). - -- [ ] **Step 6: Full suite** - -Run: `hatch run test:test` -Expected: PASS (driver-gated tests skip without extras). Root-cause any failure — never silence. If a flat backend's `{user,password}` golden disagrees, STOP and surface (test-integrity). - -- [ ] **Step 7: Type + lint gate** - -Run: `hatch run mypy:check` -Run: `hatch run ruff:check` -Expected: both clean. - -- [ ] **Step 8: Commit** -```bash -git add -A tests/ -git commit -m "test: migrate suite to *BackendProfile + factory; add consistency goldens - -Co-Authored-By: Claude Opus 4.8 (1M context) " -``` - ---- - -## Self-Review - -**1. Spec coverage** — §3.1 rename → T2; §3.2 decouple → T6,7,8; §3.3 emit + 4-backend shaping → T2,T3; §3.4 MRO dispatch → T4,T5; §3.5 factory fail-closed → T6; §4.3 supported_auth+invariant → T2; §4.4 BackendProfile/UrlParts/to_url_parts → T2; §4.6 two-layer URL → T6; §4.8 deferred auth + URL precedence (all 3 paths) → T7; iceberg → T8; §4.9 deps → T1; §5 TOKEN-only OAuth2 + iceberg token-only → T4,T2; §6 validation/fail-closed/consistency → T6,T9; §7 testing → T2–T9; §10 deferred → out of scope, preserved. ✓ - -**2. Placeholder scan** — T7/T8 settings-path/ctor specifics are confirmed at implementation against real names; the load-bearing assertions are concrete. Two VERIFY callouts (registry accessor in T6; flat `{user,password}` goldens in T9) are test-integrity-gated (surface, don't guess), not placeholders. - -**3. Type consistency** — `*BackendProfile`, `build_driver_kwargs(profile, auth_profile=None)`, `apply_auth_adapter(provider_type, base, auth_profile)`, `auth_adapter(provider_type, auth_class)`, `UrlParts(...)`, compose `(profile, base)→dict`, auth `(auth, base)→dict`, `provider_for_dialect/scheme` used identically across tasks. `UrlParts` defined once in `core/settings/profile.py`, imported by the factory. - ---- - -## Execution Handoff - -Verify-at-implementation points (all test-integrity-gated): (1) registry accessor name (`REGISTRY`/`DATABASES_REGISTRY`) in `_iter_specs` (T6); (2) flat `{user,password}` goldens (T9); (3) real `IcebergConnectionBase` ctor/attr names (T8); (4) `ProfileSpec.__post_init__` presence (T2). PR-time flag: `core/factories/` also exists on the `settings-registry` worktree branch — watch for merge conflict. diff --git a/docs/superpowers/plans/2026-06-28-dialect-aware-add-columns.md b/docs/superpowers/plans/2026-06-28-dialect-aware-add-columns.md deleted file mode 100644 index 669781c..0000000 --- a/docs/superpowers/plans/2026-06-28-dialect-aware-add-columns.md +++ /dev/null @@ -1,582 +0,0 @@ -# Dialect-Aware `add_columns` Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add a dialect-agnostic `IbisBackend.add_columns(name, source)` operation for additive schema evolution, so consumers never hand-roll `ALTER TABLE … ADD COLUMN` DDL or polars→backend type maps. - -**Architecture:** A single generic implementation renders DDL off the *live connection's own* Ibis compiler (`compiler.type_mapper` for types — identical to `create_table`; `compiler.dialect` for sqlglot identifier quoting), so it covers SQL backends that expose a sqlglot compiler + `raw_sql` and support `ALTER TABLE … ADD COLUMN` — with no per-dialect code. Verified on duckdb/sqlite; the registry's other SQL dialects (postgres, snowflake, trino, …) are covered by construction but unverified until a consumer exercises them. Dispatch mirrors the existing hook pattern (`upsert_hook` etc.) and adds an `add_columns_hook` *override* seam so a dialect that genuinely cannot `ADD COLUMN` (or needs a quirk) can override the generic default. - -**Tech Stack:** Python 3.12, Ibis (SQL backends), sqlglot (transitive via Ibis), polars, hatch + uv test env, pytest, ruff, mypy. - -**Spec:** `docs/superpowers/specs/2026-06-27-dialect-aware-add-columns-design.md` - -## Global Constraints - -- **Additive only.** Adds columns; never drops/renames/re-types existing ones. -- **Idempotent (single-process preflight).** Missing columns are computed against the live table schema once, then one `ALTER` is issued per column. A call that adds nothing is a no-op. NOT concurrency-safe: two writers racing the same new column will collide, and a multi-column add is not atomic on engines without transactional DDL. Acceptable for the single-writer consumer (wearables store); documented as a limitation, not handled here. -- **Identifier contract.** `name` and `database` must each be a *simple* (non-dotted) identifier. Each is quoted as one part; dotted/multi-part qualified names (e.g. `project.dataset`) are out of scope this iteration. -- **Type parity with `create_table`.** Types render via `ibis_conn.compiler.type_mapper.to_string(dtype)` — the exact mapper Ibis uses for `CREATE TABLE`. Never a hand-written type map. -- **Null-typed columns → dialect string.** A candidate column whose inferred dtype is Ibis `null` coerces to `ibis.dtype("string")` before rendering. -- **One ALTER per column.** SQLite allows only one `ADD COLUMN` per statement. -- **Generic default + optional override.** `DialectSpec.add_columns_hook` defaults `None`; when `None`, the generic path runs. No dialect registers a hook initially — the seam exists for dialects that later prove to need an override. -- **Run everything in the hatch test env:** `hatch run test:test-target-quick ` (quick, no coverage) for iteration. Never use the stale `.venv`. -- **Targeted test backends:** in-memory `duckdb` and `sqlite` only (no external deps). - ---- - -### Task 1: Add `add_columns_hook` to `DialectSpec` - -**Files:** -- Modify: `src/mountainash_data/backends/ibis/dialects/_registry.py` (type aliases ~line 25-32; `DialectSpec` fields ~line 44-47) -- Test: `tests/test_unit/backends/ibis/test_add_columns.py` (create) - -**Interfaces:** -- Produces: `AddColumnsHook = t.Callable[..., None]`; `DialectSpec.add_columns_hook: t.Optional[AddColumnsHook] = None` - -- [ ] **Step 1: Write the failing test** - -Create `tests/test_unit/backends/ibis/test_add_columns.py`. **Import only what each task uses** — later tasks append their own imports — so every intermediate commit stays ruff-clean (no `F401`): - -```python -"""Tests for dialect-agnostic add_columns (schema evolution).""" - -from mountainash_data.backends.ibis.dialects._registry import DIALECTS, DialectSpec - - -class TestDialectSpecField: - def test_add_columns_hook_defaults_none(self): - spec = DialectSpec( - ibis_backend_name="duckdb", - connection_mode="connection_string", - connection_string_scheme="duckdb://", - ) - assert spec.add_columns_hook is None - - def test_registered_dialects_have_no_hook_initially(self): - # The generic path covers every dialect; none registers an override. - assert DIALECTS["duckdb"].add_columns_hook is None - assert DIALECTS["sqlite"].add_columns_hook is None - assert DIALECTS["postgres"].add_columns_hook is None -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/test_add_columns.py::TestDialectSpecField -v` -Expected: FAIL — `TypeError: ... unexpected keyword 'add_columns_hook'` or `AttributeError: ... 'add_columns_hook'`. - -- [ ] **Step 3: Write minimal implementation** - -In `_registry.py`, add the alias next to the other hook aliases (after `RenameTableHook`): - -```python -RenameTableHook = t.Callable[..., None] -AddColumnsHook = t.Callable[..., None] -``` - -And the field in `DialectSpec`, after `rename_table_hook`: - -```python - rename_table_hook: t.Optional[RenameTableHook] = None - add_columns_hook: t.Optional[AddColumnsHook] = None - extras: t.Mapping[str, t.Any] = field(default_factory=dict) -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/test_add_columns.py::TestDialectSpecField -v` -Expected: PASS (3 tests). - -- [ ] **Step 5: Lint, then commit** - -Run `hatch run ruff:check tests/test_unit/backends/ibis/test_add_columns.py` first and fix any finding in the files you touched (every intermediate commit must be ruff-clean). The `ruff:check` script is hardcoded to `./src`, so the test path is appended explicitly — otherwise the new test file is never linted. Then: - -```bash -git add src/mountainash_data/backends/ibis/dialects/_registry.py tests/test_unit/backends/ibis/test_add_columns.py -git commit -m "feat(ibis): add add_columns_hook override seam to DialectSpec" -``` - ---- - -### Task 2: Source normalization helpers - -**Files:** -- Modify: `src/mountainash_data/backends/ibis/operations.py` (imports at top ~line 10-20; new functions appended to the MODULE-LEVEL HELPER FUNCTIONS section) -- Test: `tests/test_unit/backends/ibis/test_add_columns.py` - -**Interfaces:** -- Consumes: nothing from prior tasks. -- Produces: - - `_coerce_dtype(v: t.Any) -> ibis.DataType` — ibis DataType passthrough; ibis type string via `ibis.dtype`; `MountainashDtype` via the canonical bridge; raises `ValueError` for parametric MountainashDtype members. - - `_normalize_to_schema(source: t.Any) -> ibis.Schema` — `Mapping` → `ibis.schema` of coerced dtypes; otherwise frame → `ibis.memtable(source).schema()`. - -- [ ] **Step 1: Write the failing test** - -Append to `tests/test_unit/backends/ibis/test_add_columns.py` (add these imports just under the existing import at the top of the file, then append the test classes): - -```python -# add to the import block at the top of the file: -import ibis -import polars as pl -import pytest - -from mountainash.core.dtypes.canonical import MountainashDtype -from mountainash_data.backends.ibis.operations import ( - _coerce_dtype, - _normalize_to_schema, -) -``` - -```python -class TestCoerceDtype: - def test_passes_through_ibis_datatype(self): - dt = ibis.dtype("float64") - assert _coerce_dtype(dt) is dt - - def test_from_type_string(self): - assert _coerce_dtype("float64") == ibis.dtype("float64") - - def test_from_mountainash_scalar_dtype(self): - assert _coerce_dtype(MountainashDtype.FP64) == ibis.dtype("float64") - assert _coerce_dtype(MountainashDtype.U8) == ibis.dtype("uint8") - - def test_parametric_mountainash_dtype_raises_valueerror(self): - with pytest.raises(ValueError, match="parametric"): - _coerce_dtype(MountainashDtype.LIST) - - -class TestNormalizeToSchema: - def test_mapping_of_mixed_dtype_specs(self): - sch = _normalize_to_schema({"a": "float64", "b": ibis.dtype("int64")}) - assert dict(sch.items()) == dict( - ibis.schema({"a": "float64", "b": "int64"}).items() - ) - - def test_frame_inference(self): - sch = _normalize_to_schema(pl.DataFrame({"a": [1], "b": ["x"]})) - assert set(sch.names) == {"a", "b"} -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `hatch run test:test-target-quick "tests/test_unit/backends/ibis/test_add_columns.py::TestCoerceDtype" "tests/test_unit/backends/ibis/test_add_columns.py::TestNormalizeToSchema" -v` -Expected: FAIL — `ImportError: cannot import name '_coerce_dtype'`. - -- [ ] **Step 3: Write minimal implementation** - -In `operations.py`, add to the top imports (after the existing `import` lines) — only `ibis` here; `sqlglot` is added in Task 3 where it is first used, keeping this commit ruff-clean: - -```python -import ibis -``` - -Then append to the MODULE-LEVEL HELPER FUNCTIONS section: - -```python -def _coerce_dtype(v: t.Any) -> ibis.DataType: - """Normalize a dtype spec to an ibis DataType. - - Accepts an ibis DataType (passthrough), an ibis type string, or a - MountainashDtype (resolved via the canonical ibis bridge). Parametric - MountainashDtype members (LIST/STRUCT) carry no element type and raise. - """ - if isinstance(v, ibis.DataType): - return v - - mountainash_dtype = None - target_ibis = None - try: - from mountainash.core.dtypes.canonical import MountainashDtype as _MD - from mountainash.core.dtypes import target_ibis as _ti - - mountainash_dtype, target_ibis = _MD, _ti - except Exception: # mountainash build without the canonical dtypes bridge - pass - - if mountainash_dtype is not None and isinstance(v, mountainash_dtype): - # Gate parametric members explicitly via the canonical bridge's own - # CAST_UNSUPPORTED set (currently {LIST, STRUCT}) rather than relying - # on ibis.dtype() to reject a bare "array"/"struct" string. - if v in target_ibis.CAST_UNSUPPORTED: - raise ValueError( - f"MountainashDtype.{v.name} is a parametric type with no " - f"element types; pass an ibis DataType or use the frame form " - f"for nested columns." - ) - return ibis.dtype(target_ibis.SCHEMA_TYPES[v]) - - return ibis.dtype(v) - - -def _normalize_to_schema(source: t.Any) -> ibis.Schema: - """Resolve `source` to a candidate ibis Schema. - - A Mapping of ``{name: dtype}`` is coerced per-value; any other object is - treated as a frame and run through Ibis's native inference (identical to - what ``create_table`` applies). - """ - if isinstance(source, t.Mapping): - return ibis.schema({k: _coerce_dtype(v) for k, v in source.items()}) - return ibis.memtable(source).schema() -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `hatch run test:test-target-quick "tests/test_unit/backends/ibis/test_add_columns.py::TestCoerceDtype" "tests/test_unit/backends/ibis/test_add_columns.py::TestNormalizeToSchema" -v` -Expected: PASS (6 tests). - -- [ ] **Step 5: Lint, then commit** - -Run `hatch run ruff:check tests/test_unit/backends/ibis/test_add_columns.py` and fix any finding in the files you touched. Then: - -```bash -git add src/mountainash_data/backends/ibis/operations.py tests/test_unit/backends/ibis/test_add_columns.py -git commit -m "feat(ibis): add dtype/schema normalization helpers for add_columns" -``` - ---- - -### Task 3: Generic `_generic_add_columns` implementation - -**Files:** -- Modify: `src/mountainash_data/backends/ibis/operations.py` (append after the normalization helpers) -- Test: `tests/test_unit/backends/ibis/test_add_columns.py` - -**Interfaces:** -- Consumes: `_normalize_to_schema` (Task 2). -- Produces: `_generic_add_columns(ibis_conn, table_name, source, *, database=None) -> None` — additive, idempotent column adder operating on a *raw ibis connection* (`ibis_conn`, i.e. the `IbisConnection._ibis_conn`). - -- [ ] **Step 1: Write the failing test** - -Append to `tests/test_unit/backends/ibis/test_add_columns.py`: - -```python -from mountainash_data.backends.ibis.operations import _generic_add_columns - - -class TestGenericAddColumns: - def test_adds_missing_column_from_frame_duckdb(self): - con = ibis.duckdb.connect() - con.create_table("t", pl.DataFrame({"id": [1], "name": ["a"]})) - _generic_add_columns( - con, "t", pl.DataFrame({"id": [1], "name": ["a"], "score": [1.5]}) - ) - assert "score" in con.table("t").schema().names - - def test_idempotent_second_call_is_noop(self): - con = ibis.duckdb.connect() - con.create_table("t", pl.DataFrame({"id": [1]})) - _generic_add_columns(con, "t", {"x": "float64"}) - _generic_add_columns(con, "t", {"x": "float64"}) - assert list(con.table("t").schema().names).count("x") == 1 - - def test_null_typed_column_becomes_string(self): - con = ibis.duckdb.connect() - con.create_table("t", pl.DataFrame({"id": [1]})) - _generic_add_columns( - con, "t", - pl.DataFrame({"id": [1], "note": pl.Series([None], dtype=pl.Null)}), - ) - assert str(con.table("t").schema()["note"]) == "string" - - def test_quotes_identifiers_needing_quoting(self): - con = ibis.duckdb.connect() - con.create_table("t", pl.DataFrame({"id": [1]})) - _generic_add_columns(con, "t", {"new col": "float64"}) - assert "new col" in con.table("t").schema().names - - def test_works_on_sqlite(self): - con = ibis.sqlite.connect() - con.create_table("t", pl.DataFrame({"id": [1]})) - _generic_add_columns(con, "t", {"score": "float64"}) - assert "score" in con.table("t").schema().names - - def test_rejects_dotted_table_name(self): - con = ibis.duckdb.connect() - con.create_table("t", pl.DataFrame({"id": [1]})) - with pytest.raises(ValueError, match="simple"): - _generic_add_columns(con, "schema.t", {"x": "float64"}) - - def test_rejects_dotted_database(self): - con = ibis.duckdb.connect() - con.create_table("t", pl.DataFrame({"id": [1]})) - with pytest.raises(ValueError, match="simple"): - _generic_add_columns(con, "t", {"x": "float64"}, database="a.b") -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `hatch run test:test-target-quick "tests/test_unit/backends/ibis/test_add_columns.py::TestGenericAddColumns" -v` -Expected: FAIL — `ImportError: cannot import name '_generic_add_columns'`. - -- [ ] **Step 3: Write minimal implementation** - -First add the `sqlglot` import to `operations.py`'s top imports (first use is here, so it lands in this commit ruff-clean): - -```python -from sqlglot import exp -``` - -Then append to `operations.py` (the validator first, then the main function): - -```python -def _validate_simple_identifier(value: str, *, kind: str) -> None: - """Reject dotted/multi-part names — only simple identifiers are supported. - - A dotted ``table_name``/``database`` would otherwise be quoted as a single - literal identifier (``"a.b"``) rather than a namespace, silently violating - the documented contract. Fail loudly instead. - """ - if value is not None and "." in value: - raise ValueError( - f"{kind} {value!r} must be a simple (non-dotted) identifier; " - f"multi-part qualified names are out of scope." - ) - - -def _generic_add_columns( - ibis_conn: t.Any, - table_name: str, - source: t.Any, - *, - database: str | None = None, -) -> None: - """Add columns present in `source` but missing from `table_name`. - - Additive and idempotent (single-process preflight: missing columns are - computed once, then one ALTER is issued per column — not concurrency-safe - and not atomic across columns on engines without transactional DDL). - Column types render through the connection's own compiler type-mapper - (identical to ``create_table``); a null-typed column coerces to the - dialect string type; identifiers are quoted per dialect. One ``ALTER - TABLE … ADD COLUMN`` is issued per new column (SQLite permits only one per - statement). - - `table_name` and `database` must each be a simple (non-dotted) identifier; - each is quoted as a single part. Dotted/multi-part qualified names are out - of scope. - """ - _validate_simple_identifier(table_name, kind="table_name") - if database is not None: - _validate_simple_identifier(database, kind="database") - candidate = _normalize_to_schema(source) - existing = set(ibis_conn.table(table_name, database=database).schema().names) - type_mapper = ibis_conn.compiler.type_mapper - dialect = ibis_conn.compiler.dialect - - def _quote(identifier: str) -> str: - return exp.to_identifier(identifier, quoted=True).sql(dialect=dialect) - - table_parts = [database, table_name] if database else [table_name] - qualified = ".".join(_quote(part) for part in table_parts) - - for col_name, dtype in candidate.items(): - if col_name in existing: - continue - if dtype.is_null(): - dtype = ibis.dtype("string") - type_sql = type_mapper.to_string(dtype) - ibis_conn.raw_sql( - f"ALTER TABLE {qualified} ADD COLUMN {_quote(col_name)} {type_sql}" - ) -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `hatch run test:test-target-quick "tests/test_unit/backends/ibis/test_add_columns.py::TestGenericAddColumns" -v` -Expected: PASS (7 tests). - -- [ ] **Step 5: Lint, then commit** - -Run `hatch run ruff:check tests/test_unit/backends/ibis/test_add_columns.py` and fix any finding in the files you touched. Then: - -```bash -git add src/mountainash_data/backends/ibis/operations.py tests/test_unit/backends/ibis/test_add_columns.py -git commit -m "feat(ibis): generic dialect-agnostic add_columns implementation" -``` - ---- - -### Task 4: `IbisBackend.add_columns` method (dispatch + integration) - -**Files:** -- Modify: `src/mountainash_data/backends/ibis/backend.py` (import from operations near top ~line 13; new method in the "Hook-dispatched operations" section ~line 516, after `upsert`) -- Test: `tests/test_unit/backends/ibis/test_add_columns.py` - -**Interfaces:** -- Consumes: `_generic_add_columns` (Task 3); `DialectSpec.add_columns_hook` (Task 1); existing `self._require_connected()`, `conn._ibis_conn`, `self._spec`. -- Produces: `IbisBackend.add_columns(self, name: str, source: t.Any, *, database: str | None = None) -> IbisBackend` — fluent (returns `self`); calls `add_columns_hook` when set, else `_generic_add_columns`. - -- [ ] **Step 1: Write the failing test** - -Append to `tests/test_unit/backends/ibis/test_add_columns.py` (add `import dataclasses` to the top import block — `IbisBackend` is imported here, used immediately): - -```python -# add to the import block at the top of the file: -import dataclasses - -from mountainash_data import IbisBackend -``` - -```python -class TestIbisBackendAddColumns: - def test_frame_form_returns_self_and_adds_column(self): - with IbisBackend(dialect="duckdb", database=":memory:") as be: - be.create_table("t", pl.DataFrame({"id": [1], "name": ["a"]})) - ret = be.add_columns( - "t", pl.DataFrame({"id": [1], "name": ["a"], "score": [1.5]}) - ) - assert ret is be - cols = {c.name for c in be.inspect_table("t").columns} - assert "score" in cols - - def test_explicit_mountainash_dtype(self): - with IbisBackend(dialect="duckdb", database=":memory:") as be: - be.create_table("t", {"id": [1]}) - be.add_columns("t", {"hrv": MountainashDtype.FP64}) - cols = {c.name: c.type_name for c in be.inspect_table("t").columns} - assert cols["hrv"] == "float64" - - def test_create_evolve_type_parity_sqlite(self): - """The core invariant: an evolved column types like a created one.""" - with IbisBackend(dialect="sqlite", database=":memory:") as be: - be.create_table( - "fresh", pl.DataFrame({"cnt": pl.Series([3], dtype=pl.UInt8)}) - ) - be.create_table("evo", pl.DataFrame({"id": [1]})) - be.add_columns( - "evo", - pl.DataFrame({"id": [1], "cnt": pl.Series([3], dtype=pl.UInt8)}), - ) - fresh = {c.name: c.type_name for c in be.inspect_table("fresh").columns} - evolved = {c.name: c.type_name for c in be.inspect_table("evo").columns} - assert evolved["cnt"] == fresh["cnt"] - - def test_hook_override_wins_over_generic(self): - calls = [] - - def fake_hook(ibis_conn, name, source, *, database=None): - calls.append((name, source)) - - with IbisBackend(dialect="duckdb", database=":memory:") as be: - be.create_table("t", {"id": [1]}) - be._spec = dataclasses.replace(be._spec, add_columns_hook=fake_hook) - be.add_columns("t", {"x": "float64"}) - assert calls == [("t", {"x": "float64"})] - # generic path did NOT run -> column absent - cols = {c.name for c in be.inspect_table("t").columns} - assert "x" not in cols -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `hatch run test:test-target-quick "tests/test_unit/backends/ibis/test_add_columns.py::TestIbisBackendAddColumns" -v` -Expected: FAIL — `AttributeError: 'IbisBackend' object has no attribute 'add_columns'`. - -- [ ] **Step 3: Write minimal implementation** - -In `backend.py`, add the import near the other backend-internal imports (after the `_registry` import line): - -```python -from mountainash_data.backends.ibis.operations import _generic_add_columns -``` - -In the "Hook-dispatched operations (fluent — return self)" section, after the `upsert` method, add: - -```python - def add_columns( - self, - name: str, - source: t.Any, - *, - database: str | None = None, - ) -> IbisBackend: - """Additively evolve `name`: add columns present in `source` but - missing from the table. `source` is a frame (types inferred) or a - ``{column: dtype}`` mapping. Additive, idempotent, dialect-agnostic. - """ - conn = self._require_connected() - hook = self._spec.add_columns_hook - if hook is not None: - hook(conn._ibis_conn, name, source, database=database) - else: - _generic_add_columns( - conn._ibis_conn, name, source, database=database - ) - return self -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `hatch run test:test-target-quick "tests/test_unit/backends/ibis/test_add_columns.py::TestIbisBackendAddColumns" -v` -Expected: PASS (4 tests). - -- [ ] **Step 5: Run the full new test file + lint + types** - -Run: -```bash -hatch run test:test-target-quick tests/test_unit/backends/ibis/test_add_columns.py -v -hatch run ruff:check tests/test_unit/backends/ibis/test_add_columns.py -hatch run mypy:check -``` -Expected: all add_columns tests PASS; ruff clean; mypy clean (resolve any new findings in the files you touched before committing). - -- [ ] **Step 6: Commit** - -```bash -git add src/mountainash_data/backends/ibis/backend.py tests/test_unit/backends/ibis/test_add_columns.py -git commit -m "feat(ibis): IbisBackend.add_columns with hook-or-generic dispatch" -``` - ---- - -### Task 5: Full-suite regression run - -**Files:** none (verification only). - -**Interfaces:** none. - -> The spec's stale `backend.py:NNN` line citation was already removed during -> planning (replaced with a method/section reference), so there is no -> citation-refresh step here — the spec carries no line numbers to drift. - -- [ ] **Step 1: Run the full backend test suite to confirm no regressions** - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/ -v` -Expected: PASS — the new `test_add_columns.py` (16 tests across its classes) plus all pre-existing ibis backend tests, no regressions. - -- [ ] **Step 2: Run lint + types across the touched source** - -Run: -```bash -hatch run ruff:check tests/test_unit/backends/ibis/test_add_columns.py -hatch run mypy:check -``` -Expected: both clean. - -- [ ] **Step 3: (If anything failed) fix and re-run** — do not proceed to the branch/PR until Steps 1-2 are green. - ---- - -## Out of Scope (tracked elsewhere) - -- **Consumer migration** (mountainash-wearables `WearableStore`/`BronzeStore` deleting `_evolve_schema` + `_POLARS_TO_DUCKDB` + `_cast_null_columns`) happens in the wearables repo after this ships. Note carried from the spec: wearables-on-postgres also needs portable `upsert`, tracked in `generic-default-dialect-operations.md`. -- **`upsert`/`rename_table` generic defaults** — sibling backlog item `generic-default-dialect-operations.md`. - -## Self-Review - -**Spec coverage:** -- API surface (`add_columns(name, source, *, database)`, fluent) → Task 4. ✓ -- Frame + explicit-map (`MountainashDtype`/string/ibis dtype) source forms → Tasks 2, 4. ✓ -- Additive + idempotent semantics → Task 3 (`test_idempotent...`). ✓ -- Type parity via `compiler.type_mapper` → Task 3 impl + Task 4 (`test_create_evolve_type_parity_sqlite`). ✓ -- Null → dialect string → Task 3 (`test_null_typed_column_becomes_string`). ✓ -- One ALTER per column → Task 3 impl. ✓ -- Generic default + `add_columns_hook` override → Task 1 (field) + Task 4 (`test_hook_override_wins_over_generic`). ✓ -- Identifier quoting for simple identifiers → Task 3 impl + (`test_quotes_identifiers_needing_quoting`). Dotted/multi-part namespaces are out of scope and now *enforced*: `_validate_simple_identifier` raises `ValueError` for dotted `table_name`/`database` (Task 3 `test_rejects_dotted_table_name`/`test_rejects_dotted_database`). ✓ -- Idempotency is single-process preflight only (concurrency caveat) → documented in Global Constraints + `_generic_add_columns` docstring; matches the spec's Known Limitations. ✓ -- Parametric `MountainashDtype` raises → Task 2 (`test_parametric_mountainash_dtype_raises_valueerror`). ✓ -- Known limitation (unsigned ints non-round-tripping) → asserted as *parity-preserving* in Task 4 parity test (both sides equal), matching the spec. ✓ - -**Type consistency:** `_coerce_dtype`/`_normalize_to_schema`/`_generic_add_columns` signatures are identical across the task that defines each and the tasks that consume them. `add_columns_hook` arg order (`ibis_conn, name, source, *, database`) matches between the field's intended call (Task 4) and the override test. ✓ - -**Placeholder scan:** no TBD/TODO; every code step contains complete code; every run step has an exact command and expected outcome. ✓ diff --git a/docs/superpowers/plans/2026-06-29-generic-default-dialect-operations.md b/docs/superpowers/plans/2026-06-29-generic-default-dialect-operations.md deleted file mode 100644 index b79ab66..0000000 --- a/docs/superpowers/plans/2026-06-29-generic-default-dialect-operations.md +++ /dev/null @@ -1,1684 +0,0 @@ -# Generic-Default Dialect Operations Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Make `IbisBackend.upsert` work across three SQL upsert families (`ON CONFLICT`, `MERGE`, MySQL `ON DUPLICATE KEY UPDATE`) and `IbisBackend.rename_table` work on every dialect whose rename is expressible — via sqlglot-rendered generic defaults dispatched off a new `DialectSpec.upsert_style`, with the override hooks retained. - -**Architecture:** A shared `_render.py` helper renders identifiers/types/sources and compiles `update_condition` predicates off the live connection's own sqlglot compiler. `upsert`/`rename_table` dispatch hook-or-generic; the generic upsert branches on `upsert_style` into three sqlglot-AST renderers. Source rows are staged as a compiled subquery (Ibis's own mechanism), not a temp table. Live-tested on sqlite/duckdb/postgres/mysql (Docker); golden-SQL render assertions cover the full 20-dialect registry. - -**Tech Stack:** Python 3.12, ibis-framework >= 12.0.0 (env has 12.0.0), sqlglot 30.x (`sqlglot.expressions as sge` / `from sqlglot import exp`), polars, pytest, hatch + uv, Docker Compose (postgres + mariadb). - -**Spec:** `docs/superpowers/specs/2026-06-29-generic-default-dialect-operations-design.md` - -## Global Constraints - -- **Run everything in the hatch test env:** `hatch run test:test-target-quick ` (quick, no coverage). NEVER the stale `.venv`. -- **Lint:** `hatch run ruff:check src` and (for new test files) `hatch run ruff:check ` — the `ruff:check` script is hardcoded to `./src`, so test paths must be appended explicitly. Every commit ruff-clean. -- **Types:** `hatch run mypy:check` stays `Success` (0 errors). Resolve new findings in touched files only; do not touch pre-existing `iceberg.*` debt. -- **Render off the LIVE connection** — types via `ibis_conn.compiler.type_mapper.to_string(dtype)` (create_table parity); dialect via `ibis_conn.compiler.dialect` (a sqlglot dialect, NOT ibis's backend name — ibis `mssql` ↔ sqlglot `tsql`). Never a hand-written type map. -- **One generic default + optional override.** `upsert_hook` / `rename_table_hook` default `None`; when `None` the generic path runs. `upsert_style=None` → honest `NotImplementedError`. -- **Simple identifiers only** — `name`/`database`/rename names validated via the existing `_validate_simple_identifier`; dotted names raise `ValueError` (consistent with `add_columns`). -- **Explicit column lists + target-type casts** in every upsert family (no `SELECT *`, no positional values); source columns projected in target-column order. -- **Reserved sentinels** `__ma_incoming__` / `__ma_existing__` for the condition compiler; a target colliding with a sentinel raises `ValueError`. -- **20 registry dialects** — golden tests iterate the live `DIALECTS` registry, never a hardcoded list/count. -- **Targeted local test backends:** in-memory `sqlite`/`duckdb` always; `postgres`/`mysql` via Docker, skip-if-unreachable locally, fail-closed in CI (`MOUNTAINASH_REQUIRE_LIVE_DB=1`). - ---- - -## File Structure - -- `src/mountainash_data/backends/ibis/_render.py` (new) — rendering primitives + the conditional-predicate compiler + predicate-grammar validator + sentinel names. One responsibility: turn ibis/names/types into dialect-correct SQL fragments off a live connection. -- `src/mountainash_data/backends/ibis/operations.py` (modify) — add `_generic_rename_table`, `_generic_upsert`, `_render_on_conflict`, `_render_merge`, `_render_on_duplicate_key`; migrate `_generic_add_columns` to `_render.py`; delete `duckdb_family_upsert` at cutover. -- `src/mountainash_data/backends/ibis/dialects/_registry.py` (modify) — `UpsertStyle` enum, `upsert_style` field, per-dialect style assignment, remove `upsert_hook=duckdb_family_upsert` registrations at cutover. -- `src/mountainash_data/backends/ibis/backend.py` (modify) — `upsert`/`rename_table` dispatch to hook-or-generic; retype `update_condition`. -- `compose.yaml` (new, repo root) — stock postgres + mariadb services. -- `tests/conftest.py` / `tests/fixtures/` (modify) — `postgres_backend` / `mysql_backend` skip-if-unreachable fixtures. -- `tests/test_unit/backends/ibis/test_render_primitives.py`, `test_rename_table_render.py`, `test_upsert_render.py`, `test_upsert_condition_render.py` (new) — golden-SQL / unit. -- `tests/test_integration/test_write_ops_live.py`, `test_upsert_mysql_preflight.py` (new) — live round-trips. -- `.github/workflows/python-run-pytest.yml` (modify) — service containers. -- `pyproject.toml` (modify) — ibis pin `>=12.0.0`. `CLAUDE.md` (modify) — correct stale ibis note. - ---- - -### Task 1: `_render.py` rendering primitives + migrate `add_columns` - -**Files:** -- Create: `src/mountainash_data/backends/ibis/_render.py` -- Modify: `src/mountainash_data/backends/ibis/operations.py` (`_generic_add_columns` uses `quote_identifier`) -- Test: `tests/test_unit/backends/ibis/test_render_primitives.py` (create) - -**Interfaces:** -- Produces: - - `dialect_of(ibis_conn) -> t.Any` — returns `ibis_conn.compiler.dialect` (sqlglot dialect). - - `quote_identifier(name: str, dialect: t.Any) -> str` — `exp.to_identifier(name, quoted=True).sql(dialect=dialect)`. - - `qualified_name(parts: list[str], dialect: t.Any) -> str` — `".".join(quote_identifier(p, dialect) for p in parts)`. - - `render_type(type_mapper: t.Any, dtype: t.Any) -> str` — `type_mapper.to_string(dtype)`. - -- [ ] **Step 1: Write the failing test** - -Create `tests/test_unit/backends/ibis/test_render_primitives.py`: - -```python -"""Unit tests for the shared sqlglot rendering primitives.""" - -import ibis - -from mountainash_data.backends.ibis._render import ( - dialect_of, - qualified_name, - quote_identifier, - render_type, -) - - -class TestRenderPrimitives: - def test_quote_identifier_duckdb(self): - d = dialect_of(ibis.duckdb.connect()) - assert quote_identifier("new col", d) == '"new col"' - - def test_quote_identifier_mysql_backticks(self): - d = dialect_of(ibis.mysql.connect.__self__) if False else None - # mysql connect needs a server; render via a sqlglot dialect string instead - assert quote_identifier("c", "mysql") == "`c`" - - def test_qualified_name_two_parts(self): - assert qualified_name(["db", "t"], "duckdb") == '"db"."t"' - - def test_render_type_matches_create_table_mapper(self): - con = ibis.duckdb.connect() - tm = con.compiler.type_mapper - assert render_type(tm, ibis.dtype("int64")) == tm.to_string(ibis.dtype("int64")) -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/test_render_primitives.py -v` -Expected: FAIL — `ModuleNotFoundError: No module named 'mountainash_data.backends.ibis._render'`. - -- [ ] **Step 3: Write minimal implementation** - -Create `src/mountainash_data/backends/ibis/_render.py`: - -```python -"""Shared sqlglot rendering primitives for dialect-agnostic write ops. - -Everything renders off a *live* ibis connection's own compiler, so identifier -quoting and type rendering match what ibis emits for create_table. -""" - -from __future__ import annotations - -import typing as t - -from sqlglot import exp - - -def dialect_of(ibis_conn: t.Any) -> t.Any: - """The live connection's sqlglot dialect (NOT ibis's backend name).""" - return ibis_conn.compiler.dialect - - -def quote_identifier(name: str, dialect: t.Any) -> str: - """Quote a single identifier for `dialect` via sqlglot.""" - return exp.to_identifier(name, quoted=True).sql(dialect=dialect) - - -def qualified_name(parts: list[str], dialect: t.Any) -> str: - """Quote each part and join with '.' (e.g. database.table).""" - return ".".join(quote_identifier(p, dialect) for p in parts) - - -def render_type(type_mapper: t.Any, dtype: t.Any) -> str: - """Render an ibis dtype to SQL via the connection's type-mapper.""" - return type_mapper.to_string(dtype) -``` - -- [ ] **Step 4: Migrate `_generic_add_columns` to the helper (no behaviour change)** - -In `operations.py`, add to the imports near the top: - -```python -from mountainash_data.backends.ibis._render import quote_identifier -``` - -In `_generic_add_columns`, delete the inline `_quote` closure and use the helper. Replace: - -```python - def _quote(identifier: str) -> str: - return exp.to_identifier(identifier, quoted=True).sql(dialect=dialect) - - table_parts = [database, table_name] if database else [table_name] - qualified = ".".join(_quote(part) for part in table_parts) -``` - -with: - -```python - table_parts = [database, table_name] if database else [table_name] - qualified = ".".join(quote_identifier(part, dialect) for part in table_parts) -``` - -and in the loop replace `_quote(col_name)` with `quote_identifier(col_name, dialect)`. - -- [ ] **Step 5: Run tests to verify they pass** - -Run: -```bash -hatch run test:test-target-quick tests/test_unit/backends/ibis/test_render_primitives.py -v -hatch run test:test-target-quick tests/test_unit/backends/ibis/test_add_columns.py -v -``` -Expected: primitives PASS; all `test_add_columns.py` still PASS (no behaviour change). - -- [ ] **Step 6: Lint, types, commit** - -```bash -hatch run ruff:check src -hatch run ruff:check tests/test_unit/backends/ibis/test_render_primitives.py -hatch run mypy:check -git add src/mountainash_data/backends/ibis/_render.py src/mountainash_data/backends/ibis/operations.py tests/test_unit/backends/ibis/test_render_primitives.py -git commit -m "feat(ibis): extract shared _render.py primitives; migrate add_columns" -``` - ---- - -### Task 2: Test infrastructure (Docker services, fixtures, ibis pin) - -**Files:** -- Create: `compose.yaml` (repo root) -- Modify: `tests/fixtures/database_fixtures.py` (live fixtures), `tests/conftest.py` (re-export if needed) -- Modify: `pyproject.toml` (ibis pin), `CLAUDE.md` (stale-note fix), `hatch.toml` (`test-live` script), `.github/workflows/python-run-pytest.yml` (services) -- Test: `tests/test_integration/test_live_smoke.py` (create) - -**Interfaces:** -- Produces: - - `postgres_backend` / `mysql_backend` pytest fixtures yielding a connected `IbisBackend`, skipping locally when unreachable and failing when `MOUNTAINASH_REQUIRE_LIVE_DB=1` and unreachable. - -- [ ] **Step 1: Write `compose.yaml`** - -```yaml -services: - postgres: - image: postgres:18-alpine - environment: - POSTGRES_USER: postgres - POSTGRES_PASSWORD: postgres - POSTGRES_DB: ibis_testing - healthcheck: - test: ["CMD", "pg_isready", "-U", "postgres"] - interval: 1s - retries: 20 - ports: - - "5432:5432" - mysql: - image: mariadb:12.1.2 - environment: - MYSQL_ALLOW_EMPTY_PASSWORD: "true" - MYSQL_DATABASE: ibis_testing - MYSQL_USER: ibis - MYSQL_PASSWORD: ibis - healthcheck: - test: ["CMD", "mariadb-admin", "ping", "-h", "localhost"] - interval: 1s - retries: 20 - ports: - - "3306:3306" -``` - -- [ ] **Step 2: Write the failing live-smoke test** - -Create `tests/test_integration/test_live_smoke.py`: - -```python -"""Smoke test that the live-db fixtures connect or skip correctly.""" - -import pytest - - -@pytest.mark.integration -def test_postgres_smoke(postgres_backend): - assert isinstance(postgres_backend.list_tables(), list) - - -@pytest.mark.integration -def test_mysql_smoke(mysql_backend): - assert isinstance(mysql_backend.list_tables(), list) -``` - -- [ ] **Step 3: Run to verify it fails** - -Run: `hatch run test:test-target-quick tests/test_integration/test_live_smoke.py -v` -Expected: FAIL — `fixture 'postgres_backend' not found`. - -- [ ] **Step 4: Implement the fixtures** - -Add to `tests/fixtures/database_fixtures.py` (and ensure it is imported by `tests/conftest.py` like the other fixture modules): - -```python -import os - -import pytest - -from mountainash_data import IbisBackend - -_PG = dict( - host=os.environ.get("IBIS_TEST_POSTGRES_HOST", os.environ.get("PGHOST", "localhost")), - port=int(os.environ.get("IBIS_TEST_POSTGRES_PORT", os.environ.get("PGPORT", "5432"))), - user=os.environ.get("IBIS_TEST_POSTGRES_USER", os.environ.get("PGUSER", "postgres")), - password=os.environ.get("IBIS_TEST_POSTGRES_PASSWORD", os.environ.get("PGPASSWORD", "postgres")), - database=os.environ.get("IBIS_TEST_POSTGRES_DATABASE", os.environ.get("PGDATABASE", "ibis_testing")), -) -_MY = dict( - host=os.environ.get("IBIS_TEST_MYSQL_HOST", "localhost"), - port=int(os.environ.get("IBIS_TEST_MYSQL_PORT", "3306")), - user=os.environ.get("IBIS_TEST_MYSQL_USER", "ibis"), - password=os.environ.get("IBIS_TEST_MYSQL_PASSWORD", "ibis"), - database=os.environ.get("IBIS_TEST_MYSQL_DATABASE", "ibis_testing"), -) - - -def _live_or_skip(dialect: str, params: dict): - require = os.environ.get("MOUNTAINASH_REQUIRE_LIVE_DB") == "1" - try: - be = IbisBackend(dialect=dialect, **params) - be.connect() - return be - except Exception as exc: # noqa: BLE001 - service availability gate - msg = f"{dialect} service unreachable: {exc}" - if require: - pytest.fail(msg) - pytest.skip(msg) - - -@pytest.fixture -def postgres_backend(): - be = _live_or_skip("postgres", _PG) - try: - yield be - finally: - be.close() - - -@pytest.fixture -def mysql_backend(): - be = _live_or_skip("mysql", _MY) - try: - yield be - finally: - be.close() -``` - -- [ ] **Step 5: Bump ibis pin + fix CLAUDE.md + add hatch script** - -In `pyproject.toml`, change every `ibis-framework...>=11.0.0` floor to `>=12.0.0` (core dep + each extra). In `CLAUDE.md`, change the `ibis-framework[...] == 10.4.0` line to `ibis-framework[polars,pandas,sqlite,duckdb] >= 12.0.0`. In `hatch.toml` `[envs.test.scripts]`, add: - -```toml -test-live = "docker compose up -d --wait && pytest -m integration {args}" -``` - -- [ ] **Step 6: Update CI workflow** - -In `.github/workflows/python-run-pytest.yml`, add `services:` for postgres (`postgres:18-alpine`) and mariadb (`mariadb:12.1.2`) with the same env/ports as `compose.yaml`, and set `MOUNTAINASH_REQUIRE_LIVE_DB: "1"` in the job `env:`. (Mirror the env-var names the fixtures read.) - -- [ ] **Step 7: Verify (services up locally), then commit** - -```bash -docker compose up -d --wait -hatch run test:test-target-quick tests/test_integration/test_live_smoke.py -v # both PASS -docker compose down -hatch run test:test-target-quick tests/test_integration/test_live_smoke.py -v # both SKIP (no service) -hatch run ruff:check tests/test_integration/test_live_smoke.py -git add compose.yaml tests/fixtures/database_fixtures.py tests/conftest.py tests/test_integration/test_live_smoke.py pyproject.toml CLAUDE.md hatch.toml .github/workflows/python-run-pytest.yml -git commit -m "test(infra): docker postgres+mariadb services, live fixtures, ibis>=12 pin" -``` - ---- - -### Task 3: `UpsertStyle` enum + `upsert_style` field + per-dialect assignment - -**Files:** -- Modify: `src/mountainash_data/backends/ibis/dialects/_registry.py` -- Test: `tests/test_unit/backends/ibis/test_upsert_style_registry.py` (create) - -**Interfaces:** -- Consumes: nothing. -- Produces: `UpsertStyle` (`str, enum.Enum`: `ON_CONFLICT="on_conflict"`, `MERGE="merge"`, `ON_DUPLICATE_KEY="on_duplicate_key"`); `DialectSpec.upsert_style: t.Optional[UpsertStyle] = None`. - -This task is **additive** — it assigns styles but does NOT remove the existing `upsert_hook=duckdb_family_upsert` registrations (cutover is Task 9), so existing upsert behaviour stays green. - -- [ ] **Step 1: Write the failing test** - -Create `tests/test_unit/backends/ibis/test_upsert_style_registry.py`: - -```python -"""The upsert_style assignment must match the spec's §7 coverage matrix.""" - -from mountainash_data.backends.ibis.dialects._registry import ( - DIALECTS, - DialectSpec, - UpsertStyle, -) - -# Spec §7 coverage matrix — the single source of truth for this assertion. -EXPECTED_STYLE = { - "sqlite": UpsertStyle.ON_CONFLICT, - "duckdb": UpsertStyle.ON_CONFLICT, - "motherduck": UpsertStyle.ON_CONFLICT, - "postgres": UpsertStyle.ON_CONFLICT, - "risingwave": UpsertStyle.ON_CONFLICT, - "mysql": UpsertStyle.ON_DUPLICATE_KEY, - "singlestoredb": UpsertStyle.ON_DUPLICATE_KEY, - "snowflake": UpsertStyle.MERGE, - "bigquery": UpsertStyle.MERGE, - "mssql": UpsertStyle.MERGE, - "oracle": UpsertStyle.MERGE, - "databricks": UpsertStyle.MERGE, - "exasol": UpsertStyle.MERGE, - "trino": UpsertStyle.MERGE, - "redshift": UpsertStyle.MERGE, - "clickhouse": None, - "impala": None, - "materialize": None, - "druid": None, - "pyspark": None, -} - - -class TestUpsertStyleField: - def test_field_defaults_none(self): - spec = DialectSpec( - ibis_backend_name="duckdb", - connection_mode="connection_string", - connection_string_scheme="duckdb://", - ) - assert spec.upsert_style is None - - def test_every_registry_dialect_has_an_explicit_decision(self): - # Iterates the live registry — a new dialect with no matrix entry fails. - assert set(DIALECTS) == set(EXPECTED_STYLE), ( - "registry dialects and the §7 matrix have diverged" - ) - - def test_assigned_styles_match_matrix(self): - for name, expected in EXPECTED_STYLE.items(): - assert DIALECTS[name].upsert_style == expected, name -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/test_upsert_style_registry.py -v` -Expected: FAIL — `ImportError: cannot import name 'UpsertStyle'`. - -- [ ] **Step 3: Add the enum + field** - -In `_registry.py`, after the imports add: - -```python -import enum - - -class UpsertStyle(str, enum.Enum): - ON_CONFLICT = "on_conflict" - MERGE = "merge" - ON_DUPLICATE_KEY = "on_duplicate_key" -``` - -Add the field to `DialectSpec` after `upsert_hook`: - -```python - upsert_hook: t.Optional[UpsertHook] = None - upsert_style: t.Optional[UpsertStyle] = None -``` - -- [ ] **Step 4: Assign `upsert_style` to each dialect** - -Add `upsert_style=UpsertStyle.` to each `DialectSpec(...)` entry per `EXPECTED_STYLE` above. Leave the `None` dialects without the field (defaults to `None`). Leave the three existing `upsert_hook=duckdb_family_upsert` lines in place for now. - -- [ ] **Step 5: Run test to verify it passes** - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/test_upsert_style_registry.py -v` -Expected: PASS (3 tests). - -- [ ] **Step 6: Lint, types, commit** - -```bash -hatch run ruff:check src -hatch run ruff:check tests/test_unit/backends/ibis/test_upsert_style_registry.py -hatch run mypy:check -git add src/mountainash_data/backends/ibis/dialects/_registry.py tests/test_unit/backends/ibis/test_upsert_style_registry.py -git commit -m "feat(ibis): add UpsertStyle enum + upsert_style field; assign per matrix" -``` - ---- - -### Task 4: `_generic_rename_table` + dispatch - -**Files:** -- Modify: `src/mountainash_data/backends/ibis/operations.py` (add `_generic_rename_table`), `src/mountainash_data/backends/ibis/backend.py` (dispatch) -- Test: `tests/test_unit/backends/ibis/test_rename_table_render.py` (create), `tests/test_integration/test_write_ops_live.py` (create, rename portion) - -**Interfaces:** -- Consumes: `dialect_of`, `quote_identifier` (Task 1); `_validate_simple_identifier` (existing). -- Produces: `_generic_rename_table(ibis_conn, old_name: str, new_name: str) -> None`. - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_unit/backends/ibis/test_rename_table_render.py`: - -```python -"""rename_table works via the sqlglot generic default on every dialect.""" - -import ibis -import polars as pl -import pytest - -from mountainash_data import IbisBackend -from mountainash_data.backends.ibis.operations import _generic_rename_table - - -class TestGenericRenameTable: - def test_renames_on_duckdb(self): - con = ibis.duckdb.connect() - con.create_table("old", pl.DataFrame({"id": [1]})) - _generic_rename_table(con, "old", "new") - names = con.list_tables() - assert "new" in names and "old" not in names - - def test_renames_on_sqlite(self): - con = ibis.sqlite.connect() - con.create_table("old", pl.DataFrame({"id": [1]})) - _generic_rename_table(con, "old", "new") - assert "new" in con.list_tables() - - def test_rejects_dotted_names(self): - con = ibis.duckdb.connect() - con.create_table("old", pl.DataFrame({"id": [1]})) - with pytest.raises(ValueError, match="simple"): - _generic_rename_table(con, "a.old", "new") - - def test_backend_rename_table_returns_self(self): - with IbisBackend(dialect="duckdb", database=":memory:") as be: - be.create_table("old", pl.DataFrame({"id": [1]})) - assert be.rename_table("old", "new") is be - assert "new" in be.list_tables() - - -class TestRenameGoldenPerDialect: - """Registry-iterating render assertion — every dialect renders a rename.""" - - @pytest.mark.parametrize("name", list(DIALECTS)) - def test_every_dialect_renders_rename(self, name): - from mountainash_data.backends.ibis.operations import build_rename_sql - d = _IBIS_TO_SQLGLOT.get(DIALECTS[name].ibis_backend_name, - DIALECTS[name].ibis_backend_name) - sql = build_rename_sql("old", "new", dialect=d) - # tsql renders sp_rename; everyone else an ALTER ... RENAME - assert ("sp_rename" in sql.lower()) or ("rename" in sql.lower()) -``` - -Add the imports this test needs to the top of the file: `from mountainash_data.backends.ibis.dialects._registry import DIALECTS` and the shared ibis→sqlglot dialect map (define once; reuse in the upsert golden test): - -```python -# ibis backend name -> sqlglot dialect name (identity unless noted). -# Pinned by probe (see below): sqlglot 30.x has no "motherduck"/"singlestoredb". -_IBIS_TO_SQLGLOT = { - "mssql": "tsql", - "motherduck": "duckdb", - "singlestoredb": "singlestore", -} -``` - -> **Pin the full map by probe (required before the golden tests run).** For every `DIALECTS` entry, confirm `sqlglot.Dialect.get_or_raise()` succeeds — a name sqlglot 30.x doesn't know (e.g. it may not register `risingwave`/`exasol`/`databricks` under those exact strings) makes the golden test error, not assert. Where sqlglot lacks a dialect, map to its closest wire-compatible base (e.g. risingwave→`postgres`) and note it in the test; this is rendering-shape verification, so the base dialect's quoting/keywords are what we assert. (Codex HIGH-2.) - -- [ ] **Step 2: Run to verify it fails** - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/test_rename_table_render.py -v` -Expected: FAIL — `ImportError: cannot import name '_generic_rename_table'`. - -- [ ] **Step 3: Implement `_generic_rename_table`** - -In `operations.py`, add (importing the helpers and `exp` at top if not already present: `from sqlglot import exp` is already imported for add_columns; add `from mountainash_data.backends.ibis._render import dialect_of, quote_identifier`): - -```python -def build_rename_sql(old_name: str, new_name: str, *, dialect: t.Any) -> str: - """Pure builder: render a portable rename for an explicit sqlglot dialect. - - sqlglot renders `ALTER TABLE … RENAME TO …` for most dialects, `EXEC - sp_rename …` for SQL Server (tsql), and `ALTER TABLE … RENAME …` for MySQL. - Taking `dialect` explicitly lets the registry-iterating golden test render - every dialect without a live connection. - """ - return exp.Alter( - this=exp.to_table(quote_identifier(old_name, dialect)), - kind="TABLE", - actions=[exp.AlterRename(this=exp.to_identifier(new_name, quoted=True))], - ).sql(dialect=dialect) - - -def _generic_rename_table(ibis_conn: t.Any, old_name: str, new_name: str) -> None: - """Rename a table via the sqlglot generic default off the live connection.""" - _validate_simple_identifier(old_name, kind="old_name") - _validate_simple_identifier(new_name, kind="new_name") - ibis_conn.raw_sql(build_rename_sql(old_name, new_name, dialect=dialect_of(ibis_conn))) -``` - -> If `exp.Alter`/`exp.AlterRename` are not the exact class names in the installed sqlglot 30.x, use the verified-equivalent transpile fallback: `sqlglot.transpile(f'ALTER TABLE {quote_identifier(old_name, "")} RENAME TO {quote_identifier(new_name, "")}', read="duckdb", write=dialect)[0]`. Confirm via a one-line probe in the test env before settling. - -- [ ] **Step 4: Wire dispatch in `backend.py`** - -Replace the body of `rename_table` (currently raises when `rename_table_hook is None`) with hook-or-generic. Add the import near the other operations imports (`from mountainash_data.backends.ibis.operations import _generic_rename_table`) and: - -```python - def rename_table(self, old_name: str, new_name: str) -> IbisBackend: - conn = self._require_connected() - hook = self._spec.rename_table_hook - if hook is not None: - hook(conn._ibis_conn, old_name, new_name) - else: - _generic_rename_table(conn._ibis_conn, old_name, new_name) - return self -``` - -- [ ] **Step 5: Add the live rename round-trip** - -Create `tests/test_integration/test_write_ops_live.py` with the rename portion: - -```python -"""Live round-trip tests for generic write ops (postgres + mysql).""" - -import polars as pl -import pytest - - -@pytest.mark.integration -def test_rename_table_live_postgres(postgres_backend): - be = postgres_backend - be.create_table("ren_old", pl.DataFrame({"id": [1]}), overwrite=True) - be.rename_table("ren_old", "ren_new") - assert "ren_new" in be.list_tables() - be.drop_table("ren_new", force=True) - - -@pytest.mark.integration -def test_rename_table_live_mysql(mysql_backend): - be = mysql_backend - be.create_table("ren_old", pl.DataFrame({"id": [1]}), overwrite=True) - be.rename_table("ren_old", "ren_new") - assert "ren_new" in be.list_tables() - be.drop_table("ren_new", force=True) -``` - -- [ ] **Step 6: Run, lint, types, commit** - -```bash -hatch run test:test-target-quick tests/test_unit/backends/ibis/test_rename_table_render.py -v -docker compose up -d --wait && hatch run test:test-target-quick tests/test_integration/test_write_ops_live.py -v ; docker compose down -hatch run ruff:check src -hatch run ruff:check tests/test_unit/backends/ibis/test_rename_table_render.py tests/test_integration/test_write_ops_live.py -hatch run mypy:check -git add src/mountainash_data/backends/ibis/operations.py src/mountainash_data/backends/ibis/backend.py tests/test_unit/backends/ibis/test_rename_table_render.py tests/test_integration/test_write_ops_live.py -git commit -m "feat(ibis): generic sqlglot rename_table (works on every dialect)" -``` - ---- - -### Task 5: Conditional-predicate compiler (`_render.py`) - -**Files:** -- Modify: `src/mountainash_data/backends/ibis/_render.py` -- Test: `tests/test_unit/backends/ibis/test_upsert_condition_render.py` (create) - -**Interfaces:** -- Consumes: nothing from later tasks. -- Produces: - - `INCOMING_SENTINEL = "__ma_incoming__"`, `EXISTING_SENTINEL = "__ma_existing__"`. - - `validate_predicate(expr: ir.BooleanValue) -> None` — raises `ValueError` if the predicate contains an aggregation, window, or subquery/EXISTS op. - - `compile_condition(ibis_conn, target_schema, predicate, *, incoming_alias, existing_alias) -> exp.Expression` — returns the remapped `ON` sub-AST with incoming columns → `incoming_alias`, existing columns → `existing_alias`. Raises `ValueError` if the target schema would collide with a sentinel name (caller passes the real target name to check) or the grammar is violated. - -The mechanism is the §6.1/§9 probe path: bind two sentinel-named ibis tables, `existing.join(incoming, predicate)`, `compiler.to_sqlglot`, extract the join `ON`, remap aliases keyed by sentinel. - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_unit/backends/ibis/test_upsert_condition_render.py`: - -```python -"""The update_condition ibis-expression predicate compiler (§6.1).""" - -import ibis -import pytest - -from mountainash_data.backends.ibis._render import ( - ConditionAliases, - compile_condition, - dialect_of, - validate_predicate, -) - -_SCHEMA = ibis.schema({"id": "int64", "updated_at": "timestamp", "v": "string"}) - -# ON CONFLICT: incoming is the unquoted `excluded` pseudo-relation; existing is `tgt`. -_ONCONFLICT = ConditionAliases(incoming="excluded", existing="tgt", incoming_quoted=False) -# MERGE: both sides are normal quoted aliases. -_MERGE = ConditionAliases(incoming="src", existing="tgt") - - -def _render(con, predicate, aliases, *, target_name="t"): - ast = compile_condition(con, _SCHEMA, target_name, predicate, aliases=aliases) - return ast.sql(dialect=dialect_of(con)) - - -class TestCompileCondition: - def test_on_conflict_alias_mapping_unquoted_excluded(self): - con = ibis.duckdb.connect() - sql = _render(con, lambda inc, exi: inc.updated_at > exi.updated_at, _ONCONFLICT) - # EXCLUDED is the unquoted pseudo-relation; existing is quoted "tgt" - assert "excluded." in sql.lower() and '"EXCLUDED"' not in sql - assert '"tgt"."updated_at"' in sql - - def test_merge_alias_mapping_duckdb(self): - con = ibis.duckdb.connect() - sql = _render(con, lambda inc, exi: inc.updated_at > exi.updated_at, _MERGE) - assert '"src"."updated_at"' in sql and '"tgt"."updated_at"' in sql - - def test_function_predicate_renders_per_dialect(self): - con = ibis.duckdb.connect() - sql = _render(con, lambda inc, exi: inc.v.upper() != exi.v.upper(), _MERGE) - assert "UPPER(" in sql.upper() - - def test_constant_predicate_renders(self): - con = ibis.duckdb.connect() - sql = _render(con, lambda inc, exi: inc.id > 0, _MERGE) - assert '"src"."id"' in sql - - def test_null_check_predicate_renders(self): - con = ibis.duckdb.connect() - sql = _render(con, lambda inc, exi: inc.v.notnull(), _MERGE) - assert "NULL" in sql.upper() - - def test_rejects_target_name_colliding_with_sentinel(self): - con = ibis.duckdb.connect() - with pytest.raises(ValueError, match="sentinel"): - _render(con, lambda inc, exi: inc.id > exi.id, _MERGE, - target_name="__ma_incoming__") - - def test_rejects_aggregate_predicate(self): - with pytest.raises(ValueError, match="aggregat|window|scalar|subquer|row predicate"): - validate_predicate( - ibis.table(_SCHEMA, name="x").v.count() > 0 # aggregation - ) -``` - -> Add a window-function rejection test and a subquery/`EXISTS` rejection test once the exact ibis op classes are pinned by the Step-3 probe (the validator's `_SUBQUERY_OPS` tuple). Keep both as `pytest.raises(ValueError)`. - -- [ ] **Step 2: Run to verify it fails** - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/test_upsert_condition_render.py -v` -Expected: FAIL — `ImportError: cannot import name 'compile_condition'`. - -- [ ] **Step 3: Implement the compiler** - -Append to `_render.py`: - -```python -import ibis # noqa: E402 (kept with the other third-party imports at top in practice) -import ibis.expr.operations as ops -import ibis.expr.types as ir - -INCOMING_SENTINEL = "__ma_incoming__" -EXISTING_SENTINEL = "__ma_existing__" - -# ops whose presence makes a predicate invalid in a WHERE / WHEN MATCHED splice -_FORBIDDEN_OPS = (ops.Reduction, ops.WindowFunction) - - -def validate_predicate(expr: ir.BooleanValue) -> None: - """Reject predicates that cannot live in a row-level WHERE/WHEN MATCHED.""" - node = expr.op() - for n in node.find(_FORBIDDEN_OPS): # type: ignore[arg-type] - raise ValueError( - "update_condition must be a scalar row predicate; found " - f"{type(n).__name__} (aggregation/window). Use the upsert_hook " - "override for conditions outside this grammar." - ) - # subqueries / EXISTS — detect SPECIFIC subquery op types, NOT ops.Relation - # (ops.Relation also matches the two allowed sentinel tables and would - # reject every valid predicate — Codex finding). - for n in node.find(_SUBQUERY_OPS): # type: ignore[arg-type] - raise ValueError( - "update_condition may not contain subqueries/EXISTS/third-table " - "references; use the upsert_hook override." - ) - - -@dataclasses.dataclass(frozen=True) -class ConditionAliases: - """How each side's columns are referenced in the rendered clause. - - `quoted=False` is used for the ON CONFLICT `EXCLUDED` pseudo-relation, - which must NOT be a quoted identifier (Postgres exposes it as the special - unquoted `excluded`; quoting it risks referencing the wrong object). - """ - incoming: str # e.g. "EXCLUDED" (on conflict) or "src" (merge) - existing: str # e.g. "tgt" - incoming_quoted: bool = True - existing_quoted: bool = True - - -def compile_condition( - ibis_conn: t.Any, - target_schema: t.Any, - target_name: str, - predicate: t.Callable[[ir.Table, ir.Table], ir.BooleanValue], - *, - aliases: ConditionAliases, -) -> exp.Expression: - """Render an (incoming, existing) -> bool predicate to a sqlglot ON sub-AST, - remapping incoming/existing columns to `aliases`. See spec §6.1. - - `target_name` is the real target table name — rejected if it collides with - a reserved sentinel (spec §6.1 step 0). - """ - if target_name in (INCOMING_SENTINEL, EXISTING_SENTINEL): - raise ValueError( - f"target table name {target_name!r} collides with a reserved " - f"sentinel; rename the table or use the upsert_hook override." - ) - incoming = ibis.table(target_schema, name=INCOMING_SENTINEL) - existing = ibis.table(target_schema, name=EXISTING_SENTINEL) - pred = predicate(incoming, existing) - validate_predicate(pred) - - joined = existing.join(incoming, pred, how="inner") - ast = ibis_conn.compiler.to_sqlglot(joined) - ast = ast if isinstance(ast, exp.Expression) else ast[0] - - # alias -> (target_alias, quoted) keyed by the underlying SENTINEL name - remap: dict[str, tuple[str, bool]] = {} - for tbl in ast.find_all(exp.Table): - if tbl.name == INCOMING_SENTINEL: - remap[tbl.alias_or_name] = (aliases.incoming, aliases.incoming_quoted) - elif tbl.name == EXISTING_SENTINEL: - remap[tbl.alias_or_name] = (aliases.existing, aliases.existing_quoted) - - join = next(ast.find_all(exp.Join), None) - if join is None or join.args.get("on") is None: - raise ValueError("could not extract join ON predicate") - on = join.args["on"].copy() - - def _remap(n: exp.Expression) -> exp.Expression: - if isinstance(n, exp.Column) and n.table in remap: - alias, quoted = remap[n.table] - n.set("table", exp.to_identifier(alias, quoted=quoted)) - return n - - return on.transform(_remap) - - -def validate_condition(target_schema: t.Any, target_name: str, predicate) -> None: - """Grammar + sentinel-collision validation only (no rendering) — for the - unconditional §10.5 check in `_generic_upsert`.""" - if target_name in (INCOMING_SENTINEL, EXISTING_SENTINEL): - raise ValueError( - f"target table name {target_name!r} collides with a reserved sentinel." - ) - incoming = ibis.table(target_schema, name=INCOMING_SENTINEL) - existing = ibis.table(target_schema, name=EXISTING_SENTINEL) - validate_predicate(predicate(incoming, existing)) -``` - -`compile_condition` should call `validate_condition` at its top rather than duplicating the collision/grammar checks (keep the validation in one place). - -Add `import dataclasses` and `import ibis.expr.operations as ops` to `_render.py`, plus the forbidden/subquery op tuples near the top: - -```python -_FORBIDDEN_OPS = (ops.Reduction, ops.WindowFunction) -# subquery/EXISTS op classes — pinned by probe (see note); NOT ops.Relation. -_SUBQUERY_OPS = tuple( - c for c in ( - getattr(ops, "ExistsSubquery", None), - getattr(ops, "InSubquery", None), - getattr(ops, "ScalarSubquery", None), - ) if c is not None -) -``` - -> **Pin by probe (required before writing the validator):** in the test env, confirm (a) `ops.Reduction` / `ops.WindowFunction` exist and `expr.op().find((ops.Reduction,))` flags an aggregate predicate; (b) the exact subquery op class names in ibis 12 (`ExistsSubquery`/`InSubquery`/`ScalarSubquery` or their current equivalents) and that a non-subquery predicate yields an empty `_SUBQUERY_OPS` match. The behaviour contract (accept scalar row predicates incl. the two sentinel tables; reject aggregate/window/subquery) is fixed by the tests in Step 1. - -- [ ] **Step 4: Run to verify it passes** - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/test_upsert_condition_render.py -v` -Expected: PASS (4 tests). - -- [ ] **Step 5: Lint, types, commit** - -```bash -hatch run ruff:check src -hatch run ruff:check tests/test_unit/backends/ibis/test_upsert_condition_render.py -hatch run mypy:check -git add src/mountainash_data/backends/ibis/_render.py tests/test_unit/backends/ibis/test_upsert_condition_render.py -git commit -m "feat(ibis): conditional-predicate compiler (sentinel join->AST->remap)" -``` - ---- - -### Task 6: `_render_on_conflict` + `_generic_upsert` (ON_CONFLICT branch) - -**Files:** -- Modify: `src/mountainash_data/backends/ibis/_render.py` (add `compiled_source`), `src/mountainash_data/backends/ibis/operations.py` (add `_generic_upsert`, `_render_on_conflict`, helpers) -- Test: `tests/test_unit/backends/ibis/test_upsert_render.py` (create) - -**Interfaces:** -- Consumes: `quote_identifier`, `qualified_name`, `compile_condition`, `validate_predicate` (Tasks 1, 5); `_normalize_columns`, `_validate_simple_identifier` (existing). -- Produces: - - `compiled_source(ibis_conn, obj, target_schema) -> tuple[str, list[str]]` — `(subquery_sql, source_columns)`; columns cast to the target type and projected in target order. - - `_generic_upsert(ibis_conn, name, obj, *, style, conflict_columns, update_columns, conflict_action, update_condition, database, schema) -> None`. - - `build_on_conflict_sql(*, dialect, target, cols, conflict, update, conflict_action, source_sql, condition_sql=None) -> str` — the pure, dialect-parameterized builder (mirrors `build_merge_sql`, Task 7). `_render_on_conflict(ibis_conn, ...)` is the thin wrapper that derives `dialect`/`source_sql`/`condition_sql` from the live conn and delegates — so the registry-iterating upsert golden test (Task 7/10) can render the ON CONFLICT family per dialect without a live connection. - -This task wires `_generic_upsert` and the ON_CONFLICT branch only; MERGE/ON_DUPLICATE raise `NotImplementedError("unimplemented style")` as placeholders until Tasks 7/8. Dispatch is NOT flipped yet (Task 9), so this is exercised directly against a raw connection. - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_unit/backends/ibis/test_upsert_render.py`: - -```python -"""Generic upsert — ON CONFLICT family (sqlite/duckdb).""" - -import ibis -import polars as pl -import pytest - -from mountainash_data.backends.ibis.dialects._registry import UpsertStyle -from mountainash_data.backends.ibis.operations import _generic_upsert - - -def _seed(con): - con.create_table("t", pl.DataFrame({"id": [1, 2], "v": ["a", "b"]})) - - -class TestOnConflictUpsert: - def test_insert_and_update_duckdb(self): - con = ibis.duckdb.connect() - _seed(con) - _generic_upsert( - con, "t", pl.DataFrame({"id": [2, 3], "v": ["B", "c"]}), - style=UpsertStyle.ON_CONFLICT, conflict_columns=["id"], - update_columns=None, conflict_action="UPDATE", - update_condition=None, database=None, schema=None, - ) - rows = dict(con.table("t").order_by("id").execute()[["id", "v"]].itertuples(index=False)) - assert rows == {1: "a", 2: "B", 3: "c"} - - def test_do_nothing_duckdb(self): - con = ibis.duckdb.connect() - _seed(con) - _generic_upsert( - con, "t", pl.DataFrame({"id": [2], "v": ["X"]}), - style=UpsertStyle.ON_CONFLICT, conflict_columns="id", - update_columns=None, conflict_action="NOTHING", - update_condition=None, database=None, schema=None, - ) - assert con.table("t").filter(ibis._.id == 2).execute()["v"].iloc[0] == "b" - - def test_composite_key_sqlite(self): - con = ibis.sqlite.connect() - con.create_table("t", pl.DataFrame({"a": [1], "b": [1], "v": ["x"]})) - # needs a composite unique index for ON CONFLICT to detect - con.raw_sql("CREATE UNIQUE INDEX ux ON t (a, b)") - _generic_upsert( - con, "t", pl.DataFrame({"a": [1], "b": [1], "v": ["y"]}), - style=UpsertStyle.ON_CONFLICT, conflict_columns=["a", "b"], - update_columns=None, conflict_action="UPDATE", - update_condition=None, database=None, schema=None, - ) - assert con.table("t").execute()["v"].iloc[0] == "y" - - def test_conditional_update_only_when_newer_duckdb(self): - con = ibis.duckdb.connect() - con.create_table("t", pl.DataFrame({"id": [1], "ver": [5], "v": ["old"]})) - con.raw_sql("CREATE UNIQUE INDEX ux ON t (id)") - _generic_upsert( - con, "t", pl.DataFrame({"id": [1], "ver": [3], "v": ["stale"]}), - style=UpsertStyle.ON_CONFLICT, conflict_columns=["id"], - update_columns=None, conflict_action="UPDATE", - update_condition=lambda inc, exi: inc.ver > exi.ver, - database=None, schema=None, - ) - # incoming ver(3) is NOT newer than existing(5) -> unchanged - assert con.table("t").execute()["v"].iloc[0] == "old" - - def test_unknown_style_raises_notimplemented(self): - con = ibis.duckdb.connect() - _seed(con) - with pytest.raises(NotImplementedError): - _generic_upsert( - con, "t", pl.DataFrame({"id": [9], "v": ["z"]}), - style=None, conflict_columns=["id"], update_columns=None, - conflict_action="UPDATE", update_condition=None, - database=None, schema=None, - ) -``` - -- [ ] **Step 2: Run to verify it fails** - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/test_upsert_render.py -v` -Expected: FAIL — `ImportError: cannot import name '_generic_upsert'`. - -- [ ] **Step 3: Implement `compiled_source` in `_render.py`** - -```python -def compiled_source( - ibis_conn: t.Any, obj: t.Any, target_schema: t.Any -) -> tuple[str, list[str]]: - """Compile `obj` to a SELECT subquery, casting each column to the target - type and projecting in target-column order. Returns (sql, columns). - - Columns present in the target but absent from the source are omitted; - columns present in the source but absent from the target raise ValueError. - """ - src = obj if isinstance(obj, ir.Table) else ibis.memtable(obj) - src_cols = set(src.columns) - extra = src_cols - set(target_schema.names) - if extra: - raise ValueError(f"source columns absent from target: {sorted(extra)}") - cols = [c for c in target_schema.names if c in src_cols] - projected = src.select( - [src[c].cast(target_schema[c]).name(c) for c in cols] - ) - return ibis_conn.compile(projected), cols -``` - -- [ ] **Step 4: Implement `_generic_upsert` + `_render_on_conflict` in `operations.py`** - -Add imports at top: `from mountainash_data.backends.ibis._render import (ConditionAliases, compile_condition, compiled_source, qualified_name, quote_identifier, dialect_of, validate_condition)` and `from mountainash_data.backends.ibis.dialects._registry import UpsertStyle`. Keep `import warnings` (used by the NOTHING-ignores-condition warning; it predates this work). Then: - -```python -def _generic_upsert( - ibis_conn: t.Any, - name: str, - obj: t.Any, - *, - style: t.Any, - conflict_columns: t.Any, - update_columns: t.Any, - conflict_action: str, - update_condition: t.Any, - database: str | None, - schema: str | None, -) -> None: - # Validation precedence per spec §10: style -> target existence -> identifiers - # -> conflict_action -> update_condition -> (odk preflight) -> updatable cols. - if style is None: - raise NotImplementedError( - f"Dialect (connection {type(ibis_conn).__name__}) does not support upsert" - ) - if not ibis_conn.table_exists(name, database=database): # target existence (§10.2) - raise ValueError(f"target table {name!r} does not exist") - _validate_simple_identifier(name, kind="name") - if database is not None: - _validate_simple_identifier(database, kind="database") - if conflict_action not in ("UPDATE", "NOTHING"): - raise ValueError(f"conflict_action must be UPDATE or NOTHING, got {conflict_action!r}") - - target_schema = ibis_conn.table(name, database=database).schema() - conflict = _normalize_columns(conflict_columns) - # column existence (§10 MEDIUM): fail loudly, not as a backend SQL error - missing = [c for c in conflict if c not in target_schema.names] - if missing: - raise ValueError(f"conflict_columns absent from target: {missing}") - if update_columns is None: - update = [c for c in target_schema.names if c not in conflict] - else: - update = _normalize_columns(update_columns) - missing_u = [c for c in update if c not in target_schema.names] - if missing_u: - raise ValueError(f"update_columns absent from target: {missing_u}") - - # update_condition validated UNCONDITIONALLY, before any action-specific path - # (§10.5) — a malformed predicate must error even under NOTHING. - if update_condition is not None: - if style is UpsertStyle.ON_DUPLICATE_KEY: - raise ValueError( - "update_condition is not supported for the MySQL ON DUPLICATE KEY family" - ) - # validate grammar + sentinel collision now (raises on aggregate/window/ - # subquery or a target name colliding with a sentinel) - validate_condition(target_schema, name, update_condition) - if conflict_action == "NOTHING": - warnings.warn("update_condition is ignored when conflict_action='NOTHING'") - - if conflict_action == "UPDATE" and not update: - raise ValueError("no columns to update; provide update_columns or non-key columns") - - if style is UpsertStyle.ON_CONFLICT: - stmt = _render_on_conflict( - ibis_conn, name, obj, target_schema=target_schema, conflict=conflict, - update=update, conflict_action=conflict_action, - update_condition=update_condition, database=database, schema=schema, - ) - elif style is UpsertStyle.MERGE: - raise NotImplementedError("unimplemented style: MERGE") # Task 7 - elif style is UpsertStyle.ON_DUPLICATE_KEY: - raise NotImplementedError("unimplemented style: ON_DUPLICATE_KEY") # Task 8 - else: - raise NotImplementedError(f"unknown upsert_style: {style!r}") - ibis_conn.raw_sql(stmt) - - -def _render_on_conflict( - ibis_conn, name, obj, *, target_schema, conflict, update, - conflict_action, update_condition, database, schema, -) -> str: - dialect = dialect_of(ibis_conn) - source_sql, cols = compiled_source(ibis_conn, obj, target_schema) - parts = [p for p in (database, schema, name) if p] - target = qualified_name(parts, dialect) - col_list = ", ".join(quote_identifier(c, dialect) for c in cols) - conflict_list = ", ".join(quote_identifier(c, dialect) for c in conflict) - # Alias the target (INSERT INTO t AS tgt) so the existing row is referenced - # by a dedicated alias in the SET/WHERE — spec §6.1/§7. EXCLUDED is the - # UNQUOTED pseudo-relation (never a quoted identifier — Codex CRITICAL). - excl = "EXCLUDED" # unquoted keyword; Postgres exposes it as `excluded` - - if conflict_action == "NOTHING": - action = f"ON CONFLICT ({conflict_list}) DO NOTHING" - else: - set_sql = ", ".join( - f"{quote_identifier(c, dialect)} = {excl}.{quote_identifier(c, dialect)}" - for c in update - ) - where = "" - if update_condition is not None: - aliases = ConditionAliases( - incoming=excl, existing="tgt", incoming_quoted=False - ) - cond = compile_condition( - ibis_conn, target_schema, name, update_condition, aliases=aliases, - ).sql(dialect=dialect) - where = f" WHERE {cond}" - action = f"ON CONFLICT ({conflict_list}) DO UPDATE SET {set_sql}{where}" - - # `WHERE true` is REQUIRED by SQLite to disambiguate INSERT…SELECT…ON CONFLICT - # (its parser errors near DO without it); harmless on duckdb/postgres. This - # mirrors the original duckdb_family_upsert template. (Codex CRITICAL-2.) - return ( - f"INSERT INTO {target} AS tgt ({col_list}) " - f"SELECT {col_list} FROM ({source_sql}) AS __src WHERE true {action}" - ) -``` - -Add `ConditionAliases` to the `_render` import line at the top of `operations.py`. - -> **§7 per-dialect alias gate:** `INSERT INTO t AS tgt … ON CONFLICT` is valid on the live `on_conflict` dialects (duckdb/sqlite/postgres). If a future `on_conflict` dialect rejects target aliasing *and* an `update_condition` is supplied, raise `ValueError` pointing at the `upsert_hook` (spec §7). The capability flag is set from live/golden verification; it does not affect unconditional upserts (which never reference the existing row). - -- [ ] **Step 5: Run to verify it passes** - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/test_upsert_render.py -v` -Expected: PASS (5 tests). If a backend rejects the subquery cast or the `__src` alias, adjust `compiled_source`/alias to the form the live engine accepts (the tests are the contract). - -- [ ] **Step 6: Lint, types, commit** - -```bash -hatch run ruff:check src -hatch run ruff:check tests/test_unit/backends/ibis/test_upsert_render.py -hatch run mypy:check -git add src/mountainash_data/backends/ibis/_render.py src/mountainash_data/backends/ibis/operations.py tests/test_unit/backends/ibis/test_upsert_render.py -git commit -m "feat(ibis): generic upsert ON CONFLICT branch + compiled-subquery staging" -``` - ---- - -### Task 7: `_render_merge` (MERGE family) - -**Files:** -- Modify: `src/mountainash_data/backends/ibis/operations.py` -- Test: `tests/test_unit/backends/ibis/test_upsert_render.py` (append) - -**Interfaces:** -- Consumes: `compiled_source`, `compile_condition`, `qualified_name`, `quote_identifier` (Tasks 1/5/6). -- Produces: `_render_merge(...) -> str`; wired into the `_generic_upsert` MERGE branch (replacing the placeholder). - -Model on Ibis 12 `SQLBackend._build_upsert_from_table` (`sge.merge`), extended to composite `on` and `conflict_action`. duckdb supports MERGE, so this is live-testable in-memory. - -- [ ] **Step 1: Write the failing tests** (append to `test_upsert_render.py`) - -```python -class TestMergeUpsert: - def test_merge_insert_and_update_duckdb(self): - con = ibis.duckdb.connect() - con.create_table("m", pl.DataFrame({"id": [1, 2], "v": ["a", "b"]})) - from mountainash_data.backends.ibis.operations import _generic_upsert as gu - gu( - con, "m", pl.DataFrame({"id": [2, 3], "v": ["B", "c"]}), - style=UpsertStyle.MERGE, conflict_columns=["id"], update_columns=None, - conflict_action="UPDATE", update_condition=None, database=None, schema=None, - ) - rows = dict(con.table("m").order_by("id").execute()[["id", "v"]].itertuples(index=False)) - assert rows == {1: "a", 2: "B", 3: "c"} - - def test_merge_nothing_omits_matched_duckdb(self): - con = ibis.duckdb.connect() - con.create_table("m", pl.DataFrame({"id": [1], "v": ["a"]})) - from mountainash_data.backends.ibis.operations import _generic_upsert as gu - gu( - con, "m", pl.DataFrame({"id": [1, 2], "v": ["X", "b"]}), - style=UpsertStyle.MERGE, conflict_columns=["id"], update_columns=None, - conflict_action="NOTHING", update_condition=None, database=None, schema=None, - ) - rows = dict(con.table("m").order_by("id").execute()[["id", "v"]].itertuples(index=False)) - assert rows == {1: "a", 2: "b"} # id=1 NOT updated, id=2 inserted -``` - -- [ ] **Step 2: Run to verify it fails** - -Run: `hatch run test:test-target-quick "tests/test_unit/backends/ibis/test_upsert_render.py::TestMergeUpsert" -v` -Expected: FAIL — `NotImplementedError: unimplemented style: MERGE`. - -- [ ] **Step 3: Implement `_render_merge`** (replace the MERGE-branch placeholder with `stmt = _render_merge(...)` and add): - -```python -def _render_merge( - ibis_conn, name, obj, *, target_schema, conflict, update, - conflict_action, update_condition, database, schema, -) -> str: - dialect = dialect_of(ibis_conn) - source_sql, cols = compiled_source(ibis_conn, obj, target_schema) - parts = [p for p in (database, schema, name) if p] - target = qualified_name(parts, dialect) - q = lambda c: quote_identifier(c, dialect) # noqa: E731 - - on = " AND ".join(f"tgt.{q(c)} = src.{q(c)}" for c in conflict) - not_matched = ( - f"WHEN NOT MATCHED THEN INSERT ({', '.join(q(c) for c in cols)}) " - f"VALUES ({', '.join(f'src.{q(c)}' for c in cols)})" - ) - clauses = [] - if conflict_action == "UPDATE": - set_sql = ", ".join(f"{q(c)} = src.{q(c)}" for c in update) - cond = "" - if update_condition is not None: - # NEW compile_condition signature (Task 5): target_name + aliases. - cond = " AND " + compile_condition( - ibis_conn, target_schema, name, update_condition, - aliases=ConditionAliases(incoming="src", existing="tgt"), - ).sql(dialect=dialect) - clauses.append(f"WHEN MATCHED{cond} THEN UPDATE SET {set_sql}") - clauses.append(not_matched) - - return ( - f"MERGE INTO {target} AS tgt USING ({source_sql}) AS src " - f"ON {on} " + " ".join(clauses) - ) -``` - -Add `ConditionAliases` to the `_render` import in `operations.py` (shared with Task 6). - -- [ ] **Step 4: Run to verify it passes** - -Run: `hatch run test:test-target-quick "tests/test_unit/backends/ibis/test_upsert_render.py::TestMergeUpsert" -v` -Expected: PASS (2 tests). If duckdb's MERGE grammar rejects a clause ordering, reorder to its accepted form (the data-outcome assertions are the contract). - -- [ ] **Step 5: Per-dialect golden SQL via a pure builder (correct dialect, no live warehouse)** - -The conn-bound `_render_merge` cannot render Snowflake without a Snowflake connection (Codex: rendering "Snowflake golden SQL" through a DuckDB conn is not a Snowflake test). Split a **pure builder** that takes an explicit sqlglot `dialect` + an already-rendered `source_sql`, and make `_render_merge` (and the other two renderers) thin wrappers that compute `dialect`/`source_sql` from the live conn and delegate: - -```python -def build_merge_sql( - *, dialect, target, cols, conflict, update, conflict_action, - source_sql, condition_sql=None, -) -> str: - q = lambda c: quote_identifier(c, dialect) # noqa: E731 - on = " AND ".join(f"tgt.{q(c)} = src.{q(c)}" for c in conflict) - not_matched = ( - f"WHEN NOT MATCHED THEN INSERT ({', '.join(q(c) for c in cols)}) " - f"VALUES ({', '.join(f'src.{q(c)}' for c in cols)})" - ) - clauses = [] - if conflict_action == "UPDATE": - set_sql = ", ".join(f"{q(c)} = src.{q(c)}" for c in update) - cond = f" AND {condition_sql}" if condition_sql else "" - clauses.append(f"WHEN MATCHED{cond} THEN UPDATE SET {set_sql}") - clauses.append(not_matched) - return f"MERGE INTO {target} AS tgt USING ({source_sql}) AS src ON {on} " + " ".join(clauses) -``` - -`_render_merge(ibis_conn, ...)` computes `dialect = dialect_of(ibis_conn)`, `source_sql, cols = compiled_source(ibis_conn, obj, target_schema)`, renders the condition AST with `dialect` if present, builds `target = qualified_name(...)`, and returns `build_merge_sql(...)`. (Apply the same pure-builder split to `build_on_conflict_sql` and `build_on_duplicate_key_sql` in Tasks 6/8 — each takes `dialect` + `source_sql`, so golden tests render any dialect.) - -Then a registry-iterating golden test renders each MERGE-family dialect with its **own** sqlglot dialect (mapping ibis backend name → sqlglot dialect; identity except `mssql`→`tsql`): - -```python -import pytest -from mountainash_data.backends.ibis.dialects._registry import DIALECTS, UpsertStyle -from mountainash_data.backends.ibis.operations import build_merge_sql - -# Reuse the shared, probe-pinned map (mssql->tsql, motherduck->duckdb, -# singlestoredb->singlestore, + any sqlglot-unknown dialect mapped to its base). -from tests.test_unit.backends.ibis.test_rename_table_render import _IBIS_TO_SQLGLOT - - -def _sqlglot_dialect(spec): - return _IBIS_TO_SQLGLOT.get(spec.ibis_backend_name, spec.ibis_backend_name) - - -@pytest.mark.parametrize( - "name", [n for n, s in DIALECTS.items() if s.upsert_style is UpsertStyle.MERGE] -) -def test_merge_golden_per_dialect(name): - d = _sqlglot_dialect(DIALECTS[name]) - sql = build_merge_sql( - dialect=d, target=f'"m"' if d not in {"tsql", "mysql"} else "m", - cols=["id", "v"], conflict=["id"], update=["v"], - conflict_action="UPDATE", source_sql="SELECT 1 AS id, 'a' AS v", - ) - assert sql.startswith("MERGE INTO") - assert "WHEN MATCHED THEN UPDATE SET" in sql - assert "WHEN NOT MATCHED THEN INSERT" in sql - # identifiers quoted in the target dialect's style - assert ("`" in sql) == (d in {"mysql"}) -``` - -This renders each warehouse dialect with the correct sqlglot dialect (real per-dialect emission), not DuckDB-as-Snowflake. Pin the `_IBIS_TO_SQLGLOT` map against the installed sqlglot during implementation (most ibis names equal their sqlglot dialect; `mssql`→`tsql` is the known exception). - -- [ ] **Step 6: Run, lint, types, commit** - -```bash -hatch run test:test-target-quick tests/test_unit/backends/ibis/test_upsert_render.py -v -hatch run ruff:check src tests/test_unit/backends/ibis/test_upsert_render.py -hatch run mypy:check -git add src/mountainash_data/backends/ibis/operations.py tests/test_unit/backends/ibis/test_upsert_render.py -git commit -m "feat(ibis): generic upsert MERGE branch (composite keys + conflict_action)" -``` - ---- - -### Task 8: `_render_on_duplicate_key` + MySQL preflight introspection - -**Files:** -- Modify: `src/mountainash_data/backends/ibis/operations.py` -- Test: `tests/test_integration/test_upsert_mysql_preflight.py` (create), `tests/test_unit/backends/ibis/test_upsert_render.py` (golden append) - -**Interfaces:** -- Consumes: `compiled_source`, `qualified_name`, `quote_identifier` (Tasks 1/6). -- Produces: `build_on_duplicate_key_sql(*, dialect, target, cols, conflict, update, conflict_action, source_sql) -> str` (pure builder, mirrors `build_merge_sql`); `_render_on_duplicate_key(ibis_conn, ...)` thin wrapper that runs `_mysql_validate_conflict_key` then delegates; `_mysql_validate_conflict_key(ibis_conn, name, conflict, database) -> None` (prove-safe-or-raise preflight, §6.2); wired into the `_generic_upsert` ON_DUPLICATE_KEY branch. - -- [ ] **Step 1: Write the failing live tests** - -Create `tests/test_integration/test_upsert_mysql_preflight.py`: - -```python -"""MySQL ON DUPLICATE KEY preflight: prove-safe-or-raise (spec §6.2). - -Calls `_generic_upsert(...)` DIRECTLY against the raw mariadb connection — the -`be.upsert()` dispatch is not flipped until Task 9, so testing the generic -function directly is what keeps this task self-contained (Codex finding). -""" - -import polars as pl -import pytest - -from mountainash_data.backends.ibis.dialects._registry import UpsertStyle -from mountainash_data.backends.ibis.operations import _generic_upsert - - -def _raw(be): - # The fixture yields a connected IbisBackend; reach its raw ibis conn. - # Confirm the exact accessor against backend.py during implementation. - return be._require_connected()._ibis_conn - - -def _odk(con, name, df, conflict): - _generic_upsert( - con, name, df, style=UpsertStyle.ON_DUPLICATE_KEY, - conflict_columns=conflict, update_columns=None, conflict_action="UPDATE", - update_condition=None, database=None, schema=None, - ) - - -@pytest.mark.integration -def test_single_pk_proceeds(mysql_backend): - con = _raw(mysql_backend) - con.raw_sql("DROP TABLE IF EXISTS odk_ok") - con.raw_sql("CREATE TABLE odk_ok (id INT PRIMARY KEY, v VARCHAR(16) NOT NULL)") - con.raw_sql("INSERT INTO odk_ok VALUES (1, 'a')") - _odk(con, "odk_ok", pl.DataFrame({"id": [1, 2], "v": ["A", "b"]}), ["id"]) - rows = dict(con.table("odk_ok").order_by("id").execute()[["id", "v"]].itertuples(index=False)) - assert rows == {1: "A", 2: "b"} - con.raw_sql("DROP TABLE odk_ok") - - -@pytest.mark.integration -def test_multiple_unique_raises(mysql_backend): - con = _raw(mysql_backend) - con.raw_sql("DROP TABLE IF EXISTS odk_multi") - con.raw_sql( - "CREATE TABLE odk_multi " - "(id INT PRIMARY KEY, email VARCHAR(64) NOT NULL UNIQUE, v VARCHAR(16) NOT NULL)" - ) - with pytest.raises(ValueError, match="unique"): - _odk(con, "odk_multi", pl.DataFrame({"id": [1], "email": ["x"], "v": ["a"]}), ["id"]) - con.raw_sql("DROP TABLE odk_multi") - - -@pytest.mark.integration -def test_prefix_index_raises(mysql_backend): - con = _raw(mysql_backend) - con.raw_sql("DROP TABLE IF EXISTS odk_prefix") - con.raw_sql("CREATE TABLE odk_prefix (email VARCHAR(64) NOT NULL, v VARCHAR(16) NOT NULL, UNIQUE (email(10)))") - with pytest.raises(ValueError, match="prefix|SUB_PART"): - _odk(con, "odk_prefix", pl.DataFrame({"email": ["x"], "v": ["a"]}), ["email"]) - con.raw_sql("DROP TABLE odk_prefix") - - -@pytest.mark.integration -def test_nullable_conflict_column_raises(mysql_backend): - con = _raw(mysql_backend) - con.raw_sql("DROP TABLE IF EXISTS odk_null") - con.raw_sql("CREATE TABLE odk_null (k INT NULL UNIQUE, v VARCHAR(16) NOT NULL)") - with pytest.raises(ValueError, match="nullable|NOT NULL"): - _odk(con, "odk_null", pl.DataFrame({"k": [1], "v": ["a"]}), ["k"]) - con.raw_sql("DROP TABLE odk_null") -``` - -- [ ] **Step 2: Run to verify it fails** (services up) - -Run: `docker compose up -d --wait && hatch run test:test-target-quick tests/test_integration/test_upsert_mysql_preflight.py -v` -Expected: FAIL — `NotImplementedError: unimplemented style: ON_DUPLICATE_KEY`. - -- [ ] **Step 3: Implement preflight + renderer** - -```python -def _mysql_validate_conflict_key(ibis_conn, name, conflict, database) -> None: - """Prove the safe MySQL/MariaDB ON DUPLICATE KEY case or raise (spec §6.2). - - Fails closed on: no unique index, >1 unique index, prefix index (SUB_PART), - functional/expression index, a unique index whose ORDERED columns don't - exactly equal conflict_columns, or any nullable conflict column. - """ - db = database or _current_schema(ibis_conn) # see note: pin the resolver - # ORDER BY SEQ_IN_INDEX so composite-key column order is correct. - # NOTE: do NOT select EXPRESSION — it exists only in MySQL 8's STATISTICS, - # not MariaDB's (the test image is mariadb:12.1.2), so selecting it errors - # the query (Codex CRITICAL-1). Functional/expression index PARTS have a - # NULL COLUMN_NAME on BOTH MariaDB and MySQL 8 — detect them that way. - # SUB_PART is non-NULL for prefix index parts (present on both engines). - rows = ibis_conn.raw_sql( - "SELECT INDEX_NAME, SEQ_IN_INDEX, COLUMN_NAME, SUB_PART, NON_UNIQUE " - "FROM information_schema.STATISTICS " - f"WHERE TABLE_SCHEMA = '{db}' AND TABLE_NAME = '{name}' " - "ORDER BY INDEX_NAME, SEQ_IN_INDEX" - ).fetchall() - uniques: dict[str, list] = {} - for index_name, _seq, column_name, sub_part, non_unique in rows: - if int(non_unique) == 0: - uniques.setdefault(index_name, []).append((column_name, sub_part)) - if not uniques: - raise ValueError(f"table {name!r} has no unique/PK index for conflict_columns") - if len(uniques) > 1: - raise ValueError( - f"table {name!r} has multiple unique indexes {list(uniques)}; MySQL " - f"ON DUPLICATE KEY detects on any of them — ambiguous for " - f"conflict_columns={conflict}. Use the upsert_hook override." - ) - (idx_name, parts), = uniques.items() - if any(col is None for col, _ in parts): # NULL COLUMN_NAME = functional part - raise ValueError( - f"unique index {idx_name!r} is a functional/expression index; cannot " - f"prove it matches conflict_columns={conflict}. Use the upsert_hook override." - ) - if any(sub is not None for _, sub in parts): - raise ValueError( - f"unique index {idx_name!r} has a prefix (SUB_PART); it detects on a " - f"truncated value, not the full column. Use the upsert_hook override." - ) - if [c for c, _ in parts] != list(conflict): - raise ValueError( - f"unique index {idx_name!r} columns {[c for c, _ in parts]} do not " - f"exactly match conflict_columns={list(conflict)}; refusing to guess. " - f"Use the upsert_hook override." - ) - # nullable check - cols_meta = ibis_conn.raw_sql( - "SELECT COLUMN_NAME, IS_NULLABLE FROM information_schema.COLUMNS " - f"WHERE TABLE_SCHEMA = '{db}' AND TABLE_NAME = '{name}'" - ).fetchall() - nullable = {c for c, isn in cols_meta if isn == "YES"} - bad = [c for c in conflict if c in nullable] - if bad: - raise ValueError( - f"conflict columns {bad} are nullable; MySQL unique indexes are " - f"NULL-distinct, so duplicates would insert instead of update. Make " - f"them NOT NULL or use the upsert_hook override." - ) -``` - -> **Pin `_current_schema(ibis_conn)`** during implementation: ibis's current-database accessor differs by version (a `current_database`/`current_catalog` property vs a method). Probe the mysql backend in the test env and use the correct one (it must return the schema MariaDB resolves unqualified table names against). Do **not** assume `ibis_conn.current_database` is a property. - -> **MariaDB `VALUES(col)` note:** the renderer's UPDATE uses `VALUES(col)`, valid on the MariaDB 12.x target. MySQL 8.0.20+ deprecates `VALUES()` in favour of a row alias; if MySQL-8 support is later required, switch to the alias form behind the dialect — out of scope for the MariaDB-tested target here. - - -def _render_on_duplicate_key( - ibis_conn, name, obj, *, target_schema, conflict, update, - conflict_action, update_condition, database, schema, -) -> str: - _mysql_validate_conflict_key(ibis_conn, name, conflict, database) - dialect = dialect_of(ibis_conn) - source_sql, cols = compiled_source(ibis_conn, obj, target_schema) - parts = [p for p in (database, schema, name) if p] - target = qualified_name(parts, dialect) - q = lambda c: quote_identifier(c, dialect) # noqa: E731 - col_list = ", ".join(q(c) for c in cols) - - if conflict_action == "NOTHING": - k0 = q(conflict[0]) - set_sql = f"{k0} = {k0}" # self-assign; see §6.2 (not a true no-op) - else: - set_sql = ", ".join(f"{q(c)} = VALUES({q(c)})" for c in update) - - return ( - f"INSERT INTO {target} ({col_list}) SELECT {col_list} FROM ({source_sql}) AS __src " - f"ON DUPLICATE KEY UPDATE {set_sql}" - ) -``` - -Wire the ON_DUPLICATE_KEY branch in `_generic_upsert` to `stmt = _render_on_duplicate_key(...)` (replacing the `NotImplementedError` placeholder). The `update_condition`-on-ODK → `ValueError` precedence check already lives in `_generic_upsert` from Task 6 (§10.5), so no extra check is needed here. - -- [ ] **Step 4: Run to verify it passes** (services up) - -Run: `hatch run test:test-target-quick tests/test_integration/test_upsert_mysql_preflight.py -v` -Expected: 4 PASS (single-PK proceeds; multi-unique, prefix, nullable each raise `ValueError`). These call `_generic_upsert` directly, so they pass at this task's position regardless of the Task 9 dispatch flip. - -- [ ] **Step 5: Add golden-SQL for on_duplicate_key** (append to `test_upsert_render.py`, render-only, no live MySQL): assert `_render_on_duplicate_key`-style output contains `ON DUPLICATE KEY UPDATE` and `VALUES(` — but since it calls the preflight, test the pure render by factoring the SQL-string builder out of the preflight, or mark this assertion as covered by the live preflight test. Keep the unit layer to the string shape only. - -- [ ] **Step 6: Lint, types, commit** - -```bash -hatch run ruff:check src tests/test_integration/test_upsert_mysql_preflight.py -hatch run mypy:check -git add src/mountainash_data/backends/ibis/operations.py tests/test_integration/test_upsert_mysql_preflight.py tests/test_unit/backends/ibis/test_upsert_render.py -git commit -m "feat(ibis): generic upsert ON DUPLICATE KEY + MySQL prove-safe preflight" -``` - ---- - -### Task 9: Cutover — flip `upsert` dispatch, retire `duckdb_family_upsert` - -**Files:** -- Modify: `src/mountainash_data/backends/ibis/backend.py` (dispatch + retype `update_condition`), `src/mountainash_data/backends/ibis/dialects/_registry.py` (remove 3 hook registrations), `src/mountainash_data/backends/ibis/operations.py` (delete `duckdb_family_upsert`) -- Test: existing upsert tests (must stay green via the generic path) - -**Interfaces:** -- Consumes: `_generic_upsert` (Tasks 6-8). -- Produces: `IbisBackend.upsert` dispatching hook-or-generic. - -- [ ] **Step 1: Find existing upsert tests and run them (baseline green via hook)** - -Run: `hatch run test:test-target-quick tests/ -k upsert -v` — note which pass today (via `duckdb_family_upsert`). - -- [ ] **Step 2: Flip dispatch in `backend.py`** - -Replace `upsert`'s body (currently raises when `upsert_hook is None`) with: - -```python - def upsert( - self, - name: str, - obj: t.Any, - *, - conflict_columns: list[str] | str, - update_columns: list[str] | str | None = None, - conflict_action: str = "UPDATE", - update_condition: t.Any = None, # ConditionPredicate | None - database: str | None = None, - schema: str | None = None, - ) -> IbisBackend: - conn = self._require_connected() - hook = self._spec.upsert_hook - if hook is not None: - hook( - conn._ibis_conn, name, obj, conflict_columns=conflict_columns, - update_columns=update_columns, conflict_action=conflict_action, - update_condition=update_condition, database=database, schema=schema, - ) - else: - _generic_upsert( - conn._ibis_conn, name, obj, style=self._spec.upsert_style, - conflict_columns=conflict_columns, update_columns=update_columns, - conflict_action=conflict_action, update_condition=update_condition, - database=database, schema=schema, - ) - return self -``` - -Add `_generic_upsert` to the operations import. (Keep `ConditionPredicate` type alias documented in `_render.py` / re-exported if the public API surfaces it.) - -- [ ] **Step 3: Remove the duckdb-family hook registrations** - -In `_registry.py`, delete the three `upsert_hook=duckdb_family_upsert,` lines (sqlite/duckdb/motherduck) — they already carry `upsert_style=ON_CONFLICT` from Task 3, so they now flow through the generic renderer. - -- [ ] **Step 4: Delete `duckdb_family_upsert`** - -Remove the `duckdb_family_upsert` function from `operations.py` and any now-unused imports it alone used (`uuid`, `contextlib`, `warnings` — keep any still used elsewhere; verify with ruff). - -- [ ] **Step 5: Add `be.upsert()` live round-trips (dispatch path, post-cutover)** - -Append to `tests/test_integration/test_write_ops_live.py` — these exercise the full dispatch (Task 8's preflight tests call `_generic_upsert` directly, so the public `be.upsert()` path needs its own coverage): - -```python -@pytest.mark.integration -def test_upsert_via_dispatch_postgres(postgres_backend): - be = postgres_backend - be.create_table("up_pg", pl.DataFrame({"id": [1, 2], "v": ["a", "b"]}), overwrite=True) - be._require_connected()._ibis_conn.raw_sql('ALTER TABLE up_pg ADD PRIMARY KEY (id)') - be.upsert("up_pg", pl.DataFrame({"id": [2, 3], "v": ["B", "c"]}), conflict_columns=["id"]) - rows = dict(be.table("up_pg").order_by("id").execute()[["id", "v"]].itertuples(index=False)) - assert rows == {1: "a", 2: "B", 3: "c"} - be.drop_table("up_pg", force=True) - - -@pytest.mark.integration -def test_upsert_via_dispatch_mysql(mysql_backend): - be = mysql_backend - con = be._require_connected()._ibis_conn - con.raw_sql("DROP TABLE IF EXISTS up_my") - con.raw_sql("CREATE TABLE up_my (id INT PRIMARY KEY, v VARCHAR(16) NOT NULL)") - con.raw_sql("INSERT INTO up_my VALUES (1, 'a')") - be.upsert("up_my", pl.DataFrame({"id": [1, 2], "v": ["A", "b"]}), conflict_columns=["id"]) - rows = dict(con.table("up_my").order_by("id").execute()[["id", "v"]].itertuples(index=False)) - assert rows == {1: "A", 2: "b"} - con.raw_sql("DROP TABLE up_my") -``` - -- [ ] **Step 6: Run the existing + new upsert tests** - -Run: -```bash -hatch run test:test-target-quick tests/ -k upsert -v -docker compose up -d --wait && hatch run test:test-target-quick tests/test_integration -v ; docker compose down -``` -Expected: all upsert tests PASS via the generic path; live postgres/mysql round-trips PASS. If a previously-passing assertion encoded `duckdb_family_upsert`-specific behaviour (e.g. the staging-table temp name), update it to assert the data outcome instead — present any such test to the user before changing it (Test Integrity rule). - -- [ ] **Step 7: Lint, types, commit** - -```bash -hatch run ruff:check src -hatch run mypy:check -git add src/mountainash_data/backends/ibis/backend.py src/mountainash_data/backends/ibis/dialects/_registry.py src/mountainash_data/backends/ibis/operations.py tests/ -git commit -m "feat(ibis): cutover upsert to generic dispatch; retire duckdb_family_upsert" -``` - ---- - -### Task 10: Full-suite regression + matrix-completeness gate - -**Files:** none (verification only). - -- [ ] **Step 1: Golden-SQL matrix completeness** - -Confirm `test_upsert_style_registry.py::test_every_registry_dialect_has_an_explicit_decision` and the render suite iterate `DIALECTS` (no hardcoded count). Run: -`hatch run test:test-target-quick tests/test_unit/backends/ibis/ -v` — all PASS. - -- [ ] **Step 2: Full suite (no live) + live suite** - -```bash -hatch run test:test-target-quick tests/ -v -docker compose up -d --wait && MOUNTAINASH_REQUIRE_LIVE_DB=1 hatch run test:test-target-quick tests/test_integration -v ; docker compose down -``` -Expected: full unit/integration green; live job green with services up (and the `MOUNTAINASH_REQUIRE_LIVE_DB=1` run would FAIL if a service were down — proving the fail-closed gate). - -- [ ] **Step 3: Lint + types across the branch** - -```bash -hatch run ruff:check src -hatch run mypy:check -``` -Expected: both clean. - -- [ ] **Step 4: (If anything failed) fix and re-run** — do not proceed to PR until Steps 1-3 are green. - ---- - -## Out of Scope (tracked elsewhere) - -- `create_index` / `drop_index` / index-catalog queries — capability-divergent, stay hook-required. -- Consumer migration (mountainash-wearables → postgres) — in that repo after this ships. -- Live warehouse testing (snowflake/bigquery/mssql/…) — golden-SQL only here. -- An optional source-row `deduplicate=` flag — future extension (spec §11). - -## Self-Review - -**Spec coverage:** -- `upsert_style` registry field + assignment (spec §5.1, §7) → Task 3. ✓ -- `_render.py` primitives + `add_columns` consolidation (§5.2) → Task 1. ✓ -- `_generic_rename_table` + dispatch (§5.3, §4.3) → Task 4. ✓ -- Three upsert renderers (§5.4, §6) → Tasks 6 (ON CONFLICT), 7 (MERGE), 8 (ON DUPLICATE KEY). ✓ -- Compiled-subquery staging + column ordering + type casts (§5.5) → Task 6 (`compiled_source`). ✓ -- `update_condition` ibis-expression predicate + sentinel/alias-remap + grammar validator (§4.1, §6.1) → Task 5; integrated in Tasks 6/7. ✓ -- MySQL prove-safe-or-raise preflight (§6.2) → Task 8. ✓ -- Validation precedence (§10) → Task 6 (`_generic_upsert` ordering) + Task 8 (ON DUPLICATE KEY condition reject). ✓ -- Docker infra + skip/fail-closed fixtures + CI (§8) → Task 2. ✓ -- ibis pin >=12 + CLAUDE.md fix (§12) → Task 2. ✓ -- Cutover / delete `duckdb_family_upsert` (§13) → Task 9. ✓ -- Matrix completeness via registry iteration (§7, §8.4) → Task 3 + Task 10. ✓ - -**Placeholder scan:** the MERGE/ON_DUPLICATE branches are *intentional* `NotImplementedError` placeholders in Task 6 that Tasks 7/8 replace — each is a real, runnable line with a test that asserts the placeholder, then the next task removes it. No "TBD"/"add error handling"/uncoded steps remain. - -**Type consistency:** `_generic_upsert(ibis_conn, name, obj, *, style, conflict_columns, update_columns, conflict_action, update_condition, database, schema)` is identical across Tasks 6/8/9. `compiled_source(ibis_conn, obj, target_schema) -> (sql, cols)`; `compile_condition(ibis_conn, target_schema, target_name, predicate, *, aliases: ConditionAliases) -> exp.Expression` (Task 5) matches its call sites in Tasks 6/7; the pure builders `build_rename_sql(old, new, *, dialect)`, `build_merge_sql(*, dialect, target, cols, conflict, update, conflict_action, source_sql, condition_sql=None)` (and `build_on_conflict_sql` / `build_on_duplicate_key_sql` per the Task 6/8 directive) take an explicit `dialect`, so golden tests render any dialect. `UpsertStyle` members are consistent across Tasks 3/6/7/8. - -**Known implementation-time confirmations (flagged inline, not placeholders):** exact sqlglot expression classes for rename (Task 4), the forbidden/subquery ibis op classes for the grammar validator (Task 5), the ibis→sqlglot dialect map (`mssql`→`tsql`, Tasks 4/7), the `_current_schema` resolver (Task 8), and the raw-conn accessor on `IbisBackend` (Tasks 8/9) are each pinned by a one-line probe in their task; the behaviour contract is fixed by the tests in every case. - -## Codex Plan Review — Disposition (applied) - -Two-pass-reviewed spec; this plan then went through a Codex pass that found 3 CRITICAL + 4 HIGH execution-breakers, all resolved in-plan before execution: -- **C1 task ordering** — Task 8's MySQL tests now call `_generic_upsert(...)` directly (not `be.upsert()`), so they pass before the Task 9 cutover; Task 9 adds the `be.upsert()` dispatch-path round-trips. -- **C2 ON CONFLICT aliasing** — renders `INSERT INTO t AS tgt`, `existing_alias="tgt"`; §7 per-dialect alias gate noted. -- **C3 EXCLUDED quoting** — `excluded` rendered as an UNQUOTED pseudo-relation (via `ConditionAliases(incoming_quoted=False)`), never `"EXCLUDED"`. -- **H1 MySQL preflight** — orders by `SEQ_IN_INDEX`; fails closed on multi-unique, prefix (`SUB_PART`), functional/expression (`EXPRESSION`), column-set mismatch, and nullable conflict columns; live prefix test added. -- **H2 `compile_condition`** — gains `target_name` + sentinel-collision check + test. -- **H3 validator** — detects specific subquery ops (`_SUBQUERY_OPS`), not `ops.Relation` (which would match the sentinels); probe-pinned. -- **H4 golden rendering** — replaced DuckDB-as-Snowflake with registry-iterating golden tests over pure `build_*_sql(*, dialect, …)` builders rendering each dialect's correct sqlglot dialect; added the `rename_table` registry-iterating golden. -- **MEDIUM** — column-existence validation + §10 precedence ordering in `_generic_upsert`; pure-builder structure (over string-concat-on-a-conn); added predicate-shape tests. **LOW** — `VALUES()` MariaDB/MySQL-8 note; `_current_schema` resolver pinned. - -A second Codex pass verified the above all RESOLVED and caught issues the revisions introduced — all fixed: -- **CRITICAL (MariaDB `EXPRESSION`)** — that column is MySQL-8-only; the preflight query would error on `mariadb:12.1.2`. Dropped it; functional indexes detected via `COLUMN_NAME IS NULL` (portable across MariaDB + MySQL 8). -- **CRITICAL (SQLite UPSERT)** — `INSERT … SELECT … ON CONFLICT` needs a discriminating `WHERE true`; added (mirrors the original `duckdb_family_upsert` template). -- **HIGH (stale call)** — Task 7's MERGE renderer still used the old `compile_condition` positional signature; updated to `target_name` + `ConditionAliases(incoming="src", existing="tgt")`. -- **HIGH (dialect map)** — expanded `_IBIS_TO_SQLGLOT` (`+motherduck→duckdb`, `+singlestoredb→singlestore`) with a probe directive to validate every name against sqlglot 30.x and map sqlglot-unknown dialects to their wire-compatible base. -- **MEDIUM** — `update_condition` grammar now validated unconditionally via `validate_condition` (errors even under NOTHING, which warns-and-ignores); `build_on_conflict_sql` / `build_on_duplicate_key_sql` made explicit pure-builder outputs. diff --git a/docs/superpowers/plans/2026-06-30-generic-default-index-operations.md b/docs/superpowers/plans/2026-06-30-generic-default-index-operations.md deleted file mode 100644 index 5eb5392..0000000 --- a/docs/superpowers/plans/2026-06-30-generic-default-index-operations.md +++ /dev/null @@ -1,1616 +0,0 @@ -# Generic-Default Index Operations Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Make `IbisBackend.create_index` / `drop_index` / `index_exists` generic-default across the conventional-B-tree dialects via a structured `IndexCapability` descriptor, retiring the duckdb-family index hooks. - -**Architecture:** A frozen `IndexCapability` descriptor on each `DialectSpec` drives pure SQL builders + generic dispatchers in a new `backends/ibis/_index.py`. Dispatch is `hook → generic(caps) → NotImplementedError`, mirroring the upsert/rename design from PR #91. Partial-index `WHERE` is an ibis predicate compiled through a new single-relation path in `_render.py`. Idempotency is native where the engine supports `IF [NOT] EXISTS`, emulated via an `index_exists` precheck otherwise. - -**Tech Stack:** Python 3.12, ibis-framework ≥12, sqlglot 30.x, pytest, hatch (`test` env), Docker compose (postgres + mariadb) for live tests. - -**Spec:** `docs/superpowers/specs/2026-06-30-generic-default-index-operations-design.md` (Codex-reviewed). - -## Global Constraints - -- ibis-framework floor is `>=12` (already set by PR #91); do not lower it. -- **No silent degradation:** an unsupported `index_type`, a partial `WHERE` on a non-partial dialect, or a missing required `table_name` each raise `ValueError` — never warn-and-downgrade. (Retires `duckdb_family_create_index`'s warn-and-downgrade.) -- **Clean break, no shims** (pre-release, no downstream): the public param is `where: IndexPredicate | None`, NOT `where_condition: str | None`. -- **Injection hardening:** every value interpolated into introspection SQL is identifier-allowlist-validated (`_validate_simple_identifier`) and string-literal-escaped (`sqlglot exp.Literal.string`). The allowlist (`_SIMPLE_IDENTIFIER_RE = [A-Za-z_][A-Za-z0-9_$]*`) is the primary gate; escaping is defense-in-depth. -- **Coverage is all-three-operations:** `index_caps is not None` ⇒ the dialect supports create + drop + exists generically, and MUST also set `get_index_exists_sql` (registry invariant). -- **`index_caps=None`** ⇒ `create_index`/`drop_index` raise `NotImplementedError`. Out-of-scope dialects: snowflake, bigquery, redshift, trino, clickhouse, databricks, exasol, impala, materialize, risingwave, druid, pyspark. -- **Verified support matrix** (do not deviate without re-checking official docs): - - | Dialect | drop_scope | partial | native INE / IE | index_types | - |---|---|---|---|---| - | sqlite | SCHEMA_GLOBAL | True | True / True | ∅ | - | duckdb, motherduck | SCHEMA_GLOBAL | False | True / True | ∅ | - | postgres | SCHEMA_GLOBAL | True | True / True | btree,hash,gist,gin,brin,spgist | - | mysql | TABLE_SCOPED | False | False / False (emulate) | btree | - | singlestoredb | TABLE_SCOPED | False | False / False (emulate) | btree,hash | - | mssql | TABLE_SCOPED | True | False / True | ∅ | - | oracle | SCHEMA_GLOBAL | False | False / False (emulate) | ∅ | - -- **Testing:** use the hatch `test` env (`hatch run test:test-target ` / `hatch run test:test-target-quick `), never a stale `.venv`. Live tests gate on `MOUNTAINASH_REQUIRE_LIVE_DB=1` (fail-closed) and skip-if-unreachable otherwise. -- **Branch:** `feature/generic-default-index-operations` (already created off `develop`). - ---- - -## File Structure - -| File | Responsibility | -|---|---| -| `src/mountainash_data/backends/ibis/dialects/_registry.py` | Add `DropScope`, `IndexCapability`, `index_caps` field + per-dialect assignment; remove `duckdb_family_*` registrations; wire 5 new `get_index_exists_sql` | -| `src/mountainash_data/backends/ibis/_render.py` | Add `compile_index_predicate` (single-relation WHERE compiler, AST-level qualifier strip) | -| `src/mountainash_data/backends/ibis/_index.py` | **New** — pure builders (`build_create_index_sql`, `build_drop_index_sql`) + generic dispatchers (`_generic_create_index`, `_generic_drop_index`, `_generic_index_exists`) + `_USING_BEFORE_ON`/`_USING_BEFORE_COLUMNS` placement maps | -| `src/mountainash_data/backends/ibis/operations.py` | Add `_sql_literal` escape helper; harden existing 3 + add 5 new `get_index_exists_sql`; delete `duckdb_family_create_index`/`drop_index` | -| `src/mountainash_data/backends/ibis/backend.py` | `create_index`/`drop_index`/`index_exists` → hook→generic→NotImplementedError; table-scoped `table_name` validation; `where` predicate param | -| `tests/test_unit/backends/ibis/test_index_capability.py` | **New** — capability dataclass + registry matrix/invariant | -| `tests/test_unit/backends/ibis/test_index_render.py` | **New** — pure-builder golden + predicate compile + introspection-SQL golden | -| `tests/test_unit/backends/ibis/test_index_ops.py` | **New** — generic dispatcher behavior (in-memory sqlite/duckdb) | -| `tests/test_integration/test_index_ops_live.py` | **New** — postgres + mariadb round-trips, partial, table-scoped drop, emulation | -| `tests/test_unit/backends/ibis/test_backend.py` | Update the 3 hook-mechanism assertions at cutover | - ---- - -## Task 1: Capability model — `DropScope`, `IndexCapability`, `index_caps` field - -**Files:** -- Modify: `src/mountainash_data/backends/ibis/dialects/_registry.py` (after the `UpsertStyle` enum, ~line 29, and the `DialectSpec` dataclass, ~line 42-59) -- Test: `tests/test_unit/backends/ibis/test_index_capability.py` - -**Interfaces:** -- Produces: `class DropScope(str, enum.Enum)` with members `SCHEMA_GLOBAL="schema_global"`, `TABLE_SCOPED="table_scoped"`; `@dataclass(frozen=True) class IndexCapability` with fields `drop_scope: DropScope`, `partial: bool`, `native_if_not_exists: bool`, `native_if_exists: bool`, `index_types: frozenset[str]`; new `DialectSpec` field `index_caps: t.Optional[IndexCapability] = None`. - -- [ ] **Step 1: Write the failing test** - -Create `tests/test_unit/backends/ibis/test_index_capability.py`: - -```python -"""IndexCapability descriptor + DropScope enum (registry capability model).""" - -import dataclasses - -import pytest - -from mountainash_data.backends.ibis.dialects._registry import ( - DialectSpec, - DropScope, - IndexCapability, -) - - -def test_dropscope_members(): - assert DropScope.SCHEMA_GLOBAL.value == "schema_global" - assert DropScope.TABLE_SCOPED.value == "table_scoped" - - -def test_index_capability_is_frozen(): - caps = IndexCapability( - drop_scope=DropScope.SCHEMA_GLOBAL, - partial=True, - native_if_not_exists=True, - native_if_exists=True, - index_types=frozenset({"btree"}), - ) - with pytest.raises(dataclasses.FrozenInstanceError): - caps.partial = False # type: ignore[misc] - - -def test_dialectspec_index_caps_defaults_none(): - spec = DialectSpec( - ibis_backend_name="x", - connection_mode="kwargs", - connection_string_scheme="x://", - ) - assert spec.index_caps is None -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/test_index_capability.py` -Expected: FAIL with `ImportError: cannot import name 'DropScope'`. - -- [ ] **Step 3: Add the enum and dataclass** - -In `_registry.py`, after the `UpsertStyle` enum (after line 28), add: - -```python -class DropScope(str, enum.Enum): - SCHEMA_GLOBAL = "schema_global" # DROP INDEX name - TABLE_SCOPED = "table_scoped" # DROP INDEX name ON tbl - - -@dataclass(frozen=True) -class IndexCapability: - """Per-dialect conventional-B-tree index capability (spec §3). - - None on DialectSpec.index_caps means the dialect has no conventional - secondary index -> create/drop raise NotImplementedError. - """ - - drop_scope: DropScope - partial: bool # supports a WHERE filter (partial/filtered index) - native_if_not_exists: bool # engine has CREATE INDEX IF NOT EXISTS - native_if_exists: bool # engine has DROP INDEX IF EXISTS - index_types: frozenset[str] # valid USING values; empty = no USING clause -``` - -In the `DialectSpec` dataclass body (after the `upsert_style` field, ~line 54), add: - -```python - index_caps: t.Optional[IndexCapability] = None - # None = no conventional index support -> NotImplementedError. -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/test_index_capability.py` -Expected: PASS (3 passed). - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/backends/ibis/dialects/_registry.py tests/test_unit/backends/ibis/test_index_capability.py -git commit -m "feat(ibis): add DropScope + IndexCapability model + index_caps field" -``` - ---- - -## Task 2: `index_caps` for the 3 introspection-ready dialects + invariant - -**Why only 3 here:** the §3 invariant is `index_caps ⇒ get_index_exists_sql`. Only sqlite/duckdb/motherduck already have introspection SQL, so only they may receive `index_caps` now. The other 5 (postgres/mysql/mssql/oracle/singlestoredb) get their caps **and** introspection together in Task 5 — keeping the invariant TRUE at every commit (the registry must never be committed in a broken state). - -**Files:** -- Modify: `src/mountainash_data/backends/ibis/dialects/_registry.py` (the sqlite/duckdb/motherduck `DialectSpec(...)` entries, lines ~658-690) -- Test: `tests/test_unit/backends/ibis/test_index_capability.py` (append) - -**Interfaces:** -- Consumes: `DropScope`, `IndexCapability` (Task 1). -- Produces: `index_caps=IndexCapability(...)` on sqlite, duckdb, motherduck. The reference `_EXPECTED` table holds all 8 dialects' values (Task 5 will assign the remaining 5). - -- [ ] **Step 1: Write the failing test** - -Append to `tests/test_unit/backends/ibis/test_index_capability.py`: - -```python -from mountainash_data.backends.ibis.dialects._registry import DIALECTS - -# Verified against official vendor docs 2026-06-30 (spec §4). frozenset of USING types. -# (drop_scope, partial, native_if_not_exists, native_if_exists, index_types) -_EXPECTED = { - "sqlite": (DropScope.SCHEMA_GLOBAL, True, True, True, frozenset()), - "duckdb": (DropScope.SCHEMA_GLOBAL, False, True, True, frozenset()), - "motherduck": (DropScope.SCHEMA_GLOBAL, False, True, True, frozenset()), - "postgres": (DropScope.SCHEMA_GLOBAL, True, True, True, - frozenset({"btree", "hash", "gist", "gin", "brin", "spgist"})), - "mysql": (DropScope.TABLE_SCOPED, False, False, False, frozenset({"btree"})), - "singlestoredb": (DropScope.TABLE_SCOPED, False, False, False, frozenset({"btree", "hash"})), - "mssql": (DropScope.TABLE_SCOPED, True, False, True, frozenset()), - "oracle": (DropScope.SCHEMA_GLOBAL, False, False, False, frozenset()), -} - -# Dialects that carry index_caps after THIS task. Task 5 appends the other 5. -_ASSIGNED_NOW = ["sqlite", "duckdb", "motherduck"] - -_UNSUPPORTED = { - "snowflake", "bigquery", "redshift", "trino", "clickhouse", "databricks", - "exasol", "impala", "materialize", "risingwave", "druid", "pyspark", -} - - -@pytest.mark.parametrize("name", _ASSIGNED_NOW) -def test_index_caps_matrix(name): - caps = DIALECTS[name].index_caps - assert caps is not None, f"{name} must have index_caps" - drop_scope, partial, ine, ie, types = _EXPECTED[name] - assert caps.drop_scope is drop_scope - assert caps.partial is partial - assert caps.native_if_not_exists is ine - assert caps.native_if_exists is ie - assert caps.index_types == types - - -@pytest.mark.parametrize("name", sorted(_UNSUPPORTED)) -def test_unsupported_dialects_have_no_index_caps(name): - assert DIALECTS[name].index_caps is None - - -@pytest.mark.parametrize("name", _ASSIGNED_NOW) -def test_invariant_caps_implies_exists_sql(name): - """Spec §3 invariant: a dialect with index_caps must also introspect indexes.""" - spec = DIALECTS[name] - assert spec.index_caps is not None - assert spec.get_index_exists_sql is not None - - -def test_no_dialect_violates_invariant(): - """Stronger guard: NO dialect may have index_caps without exists_sql — true at - every commit, including this one (the other 5 caps are not assigned yet).""" - for name, spec in DIALECTS.items(): - if spec.index_caps is not None: - assert spec.get_index_exists_sql is not None, ( - f"{name}: index_caps set but get_index_exists_sql missing" - ) -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/test_index_capability.py::test_index_caps_matrix` -Expected: FAIL — `assert caps is not None` (no dialect has `index_caps` yet). - -- [ ] **Step 3: Assign `index_caps` on sqlite, duckdb, motherduck only** - -In `_registry.py`, add `index_caps=IndexCapability(...)` to the three entries. `sqlite`: - -```python - "sqlite": DialectSpec( - ibis_backend_name="sqlite", - connection_mode=_CONNECTION_STRING, - connection_string_scheme="sqlite://", - connection_builder=_build_sqlite_connection, - get_index_exists_sql=sqlite_get_index_exists_sql, - get_list_indexes_sql=sqlite_get_list_indexes_sql, - upsert_style=UpsertStyle.ON_CONFLICT, - create_index_hook=duckdb_family_create_index, # removed in Task 7 cutover - drop_index_hook=duckdb_family_drop_index, # removed in Task 7 cutover - index_caps=IndexCapability( - drop_scope=DropScope.SCHEMA_GLOBAL, partial=True, - native_if_not_exists=True, native_if_exists=True, - index_types=frozenset(), - ), - ), -``` - -`duckdb` and `motherduck` get the identical capability (note `partial=False` — DuckDB has no partial index): - -```python - index_caps=IndexCapability( - drop_scope=DropScope.SCHEMA_GLOBAL, partial=False, - native_if_not_exists=True, native_if_exists=True, - index_types=frozenset(), - ), -``` - -Do **not** touch postgres/mysql/mssql/oracle/singlestoredb here — Task 5 assigns those. - -- [ ] **Step 4: Run test to verify it passes** - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/test_index_capability.py` -Expected: PASS (3-dialect matrix + unsupported + both invariant guards green). - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/backends/ibis/dialects/_registry.py tests/test_unit/backends/ibis/test_index_capability.py -git commit -m "feat(ibis): assign index_caps to sqlite/duckdb/motherduck (introspection-ready)" -``` - ---- - -## Task 3: `compile_index_predicate` — single-relation WHERE compiler - -**Files:** -- Modify: `src/mountainash_data/backends/ibis/_render.py` (append after `compile_condition`, ~line 211) -- Test: `tests/test_unit/backends/ibis/test_index_render.py` - -**Interfaces:** -- Consumes: `dialect_of`, `validate_predicate`, `INCOMING_SENTINEL` patterns (existing in `_render.py`). -- Produces: module constant `INDEX_SENTINEL = "__ma_index_tbl__"`; `IndexPredicate = t.Callable[[ir.Table], ir.BooleanValue]`; `compile_index_predicate(ibis_conn, schema, table_name, predicate) -> str` — returns a dialect-rendered, **unqualified** boolean SQL string. - -**Scope note (spec §5.2):** `validate_predicate` is a STRUCTURAL guard (rejects aggregate/window/subquery) — it does NOT model per-dialect filter grammars. `mssql` is `partial=True`, but SQL Server filtered-index predicates are far narrower than a general boolean (simple comparisons / `IN`, no computed columns). We deliberately do NOT add per-dialect predicate grammar validation: mssql partial is **render-capable but engine-restricted**, and since mssql is render-only (no live container) a too-rich predicate surfaces as a SQL Server error at execution. This is a documented limitation, not a gap to close in code. - -- [ ] **Step 1: Write the failing test** - -Create `tests/test_unit/backends/ibis/test_index_render.py`: - -```python -"""Index render primitives: predicate compiler + pure builders + introspection SQL.""" - -import ibis -import pytest - -from mountainash_data.backends.ibis._render import compile_index_predicate - -_SCHEMA = ibis.schema({"id": "int64", "active": "boolean", "ver": "int64"}) - - -def _pred_sql(predicate, *, table_name="t"): - con = ibis.duckdb.connect() - return compile_index_predicate(con, _SCHEMA, table_name, predicate) - - -class TestCompileIndexPredicate: - def test_renders_unqualified_columns(self): - sql = _pred_sql(lambda t: t.active == True) # noqa: E712 - # the column must be UNqualified (no table/alias prefix) - assert '"active"' in sql - assert "." not in sql.split('"active"')[0][-3:] # no `x.` before "active" - - def test_comparison_predicate(self): - sql = _pred_sql(lambda t: t.ver > 5) - assert '"ver"' in sql and "5" in sql - - def test_predicate_may_reference_non_indexed_column(self): - # binding the full schema (not just indexed cols) must allow this - sql = _pred_sql(lambda t: t.active) - assert '"active"' in sql - - def test_rejects_sentinel_table_name(self): - with pytest.raises(ValueError, match="sentinel"): - _pred_sql(lambda t: t.id > 0, table_name="__ma_index_tbl__") - - def test_rejects_aggregate(self): - with pytest.raises(ValueError, match="aggregat|window|scalar|subquer|row predicate"): - _pred_sql(lambda t: t.id.sum() > 0) -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/test_index_render.py::TestCompileIndexPredicate` -Expected: FAIL with `ImportError: cannot import name 'compile_index_predicate'`. - -- [ ] **Step 3: Implement `compile_index_predicate`** - -Append to `_render.py`: - -```python -INDEX_SENTINEL = "__ma_index_tbl__" - -IndexPredicate = t.Callable[[ir.Table], ir.BooleanValue] - - -def compile_index_predicate( - ibis_conn: t.Any, - schema: t.Any, - table_name: str, - predicate: IndexPredicate, -) -> str: - """Compile a single-table ``(table) -> bool`` predicate to an UNQUALIFIED - boolean SQL string for the connection's dialect (partial-index WHERE). - - Mechanism (spec §5.2): bind one sentinel-named ibis table at `schema`, - filter it by the predicate, compile to sqlglot, extract the WHERE, then - strip every column's table/db/catalog qualifier at the AST level (NOT by - string replacement). The predicate may reference any column of the table, - not only the indexed columns, so the full `schema` is bound. - - Raises: - ValueError: if `table_name` collides with the reserved sentinel, or the - predicate contains a forbidden op (aggregation/window/subquery). - """ - if table_name == INDEX_SENTINEL: - raise ValueError( - f"target table name {table_name!r} collides with a reserved sentinel." - ) - tbl = ibis.table(schema, name=INDEX_SENTINEL) - pred = predicate(tbl) - validate_predicate(pred) - - filtered = tbl.filter(pred) - ast = ibis_conn.compiler.to_sqlglot(filtered) - ast = ast if isinstance(ast, exp.Expression) else ast[0] - - where = next(ast.find_all(exp.Where), None) - if where is None or where.this is None: - raise ValueError("could not extract WHERE predicate from compiled AST") - cond = where.this.copy() - - def _strip(n: exp.Expression) -> exp.Expression: - if isinstance(n, exp.Column): - n.set("table", None) - n.set("db", None) - n.set("catalog", None) - return n - - return cond.transform(_strip).sql(dialect=dialect_of(ibis_conn)) -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/test_index_render.py::TestCompileIndexPredicate` -Expected: PASS (5 passed). - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/backends/ibis/_render.py tests/test_unit/backends/ibis/test_index_render.py -git commit -m "feat(ibis): add compile_index_predicate single-relation WHERE compiler" -``` - ---- - -## Task 4: Pure builders `build_create_index_sql` + `build_drop_index_sql` - -**Files:** -- Create: `src/mountainash_data/backends/ibis/_index.py` -- Test: `tests/test_unit/backends/ibis/test_index_render.py` (append) - -**Interfaces:** -- Consumes: `_render.quote_identifier`; `_registry.DropScope`. -- Produces: - - `_USING_BEFORE_ON: frozenset[str] = frozenset({"mysql"})` and `_USING_BEFORE_COLUMNS: frozenset[str] = frozenset({"postgres"})` — the two non-default `USING ` placements (mysql: after the index name, before `ON`; postgres: after `ON`, before columns; everything else: after the column list). - - `build_create_index_sql(*, dialect, target, index_name, cols, unique, index_type, guard, where_sql) -> str` - - `build_drop_index_sql(*, dialect, drop_scope, index_name, target, guard) -> str` - -- [ ] **Step 1: Write the failing test** - -Append to `tests/test_unit/backends/ibis/test_index_render.py`: - -```python -from mountainash_data.backends.ibis._index import ( - build_create_index_sql, - build_drop_index_sql, -) -from mountainash_data.backends.ibis.dialects._registry import DropScope - - -class TestBuildCreateIndexSql: - def test_basic(self): - sql = build_create_index_sql( - dialect="duckdb", target='"t"', index_name="idx_t_id", - cols=["id"], unique=False, index_type=None, guard="", where_sql=None, - ) - assert sql == 'CREATE INDEX "idx_t_id" ON "t" ("id")' - - def test_unique_and_guard(self): - sql = build_create_index_sql( - dialect="duckdb", target='"t"', index_name="u", cols=["a", "b"], - unique=True, index_type=None, guard="IF NOT EXISTS ", where_sql=None, - ) - assert sql == 'CREATE UNIQUE INDEX IF NOT EXISTS "u" ON "t" ("a", "b")' - - def test_partial_where(self): - sql = build_create_index_sql( - dialect="duckdb", target='"t"', index_name="p", cols=["id"], - unique=False, index_type=None, guard="", where_sql='"active"', - ) - assert sql.endswith('("id") WHERE "active"') - - def test_using_before_columns_postgres(self): - sql = build_create_index_sql( - dialect="postgres", target='"t"', index_name="g", cols=["doc"], - unique=False, index_type="gin", guard="", where_sql=None, - ) - assert sql == 'CREATE INDEX "g" ON "t" USING gin ("doc")' - - def test_using_before_on_mysql(self): - # MySQL/MariaDB place USING between the index name and ON (verified: - # dev.mysql.com 8.4 CREATE INDEX grammar `index_name [index_type] ON`). - sql = build_create_index_sql( - dialect="mysql", target="`t`", index_name="i", cols=["id"], - unique=False, index_type="btree", guard="", where_sql=None, - ) - assert sql == "CREATE INDEX `i` USING btree ON `t` (`id`)" - - def test_using_after_columns_singlestore(self): - sql = build_create_index_sql( - dialect="singlestore", target="`t`", index_name="i", cols=["id"], - unique=False, index_type="hash", guard="", where_sql=None, - ) - assert sql == "CREATE INDEX `i` ON `t` (`id`) USING hash" - - -class TestBuildDropIndexSql: - def test_schema_global(self): - sql = build_drop_index_sql( - dialect="duckdb", drop_scope=DropScope.SCHEMA_GLOBAL, - index_name="idx", target=None, guard="IF EXISTS ", - ) - assert sql == 'DROP INDEX IF EXISTS "idx"' - - def test_table_scoped(self): - sql = build_drop_index_sql( - dialect="mysql", drop_scope=DropScope.TABLE_SCOPED, - index_name="idx", target="`t`", guard="", - ) - assert sql == "DROP INDEX `idx` ON `t`" -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/test_index_render.py::TestBuildCreateIndexSql` -Expected: FAIL with `ModuleNotFoundError: No module named '...ibis._index'`. - -- [ ] **Step 3: Create `_index.py` with the builders** - -```python -"""Generic-default index DDL: pure builders + dispatchers (spec §5). - -Pure builders take pre-computed, already-validated parts so registry golden -tests render every dialect without a live connection. -""" - -from __future__ import annotations - -import typing as t - -from mountainash_data.backends.ibis._render import quote_identifier -from mountainash_data.backends.ibis.dialects._registry import DropScope - -# USING position differs across dialects (verified against official docs): -# - Postgres: CREATE INDEX i ON tbl USING gin (cols) -> after ON, before columns -# - MySQL/MariaDB: CREATE INDEX i USING btree ON tbl (cols) -> after index name, before ON -# - SingleStore: CREATE INDEX i ON tbl (cols) USING hash -> after columns (the default) -# sqlite/duckdb/motherduck/mssql/oracle have empty index_types -> no USING emitted. -_USING_BEFORE_ON: frozenset[str] = frozenset({"mysql"}) -_USING_BEFORE_COLUMNS: frozenset[str] = frozenset({"postgres"}) - - -def build_create_index_sql( - *, - dialect: t.Any, - target: str, - index_name: str, - cols: list[str], - unique: bool, - index_type: t.Optional[str], - guard: str, - where_sql: t.Optional[str], -) -> str: - """Render a CREATE INDEX statement from pre-validated parts. - - Args: - dialect: sqlglot dialect string (e.g. ``dialect_of(ibis_conn)``). - target: already-qualified, already-quoted table reference. - index_name: unquoted index name. - cols: unquoted column names. - unique: emit CREATE UNIQUE INDEX. - index_type: USING , or None for no USING clause. - guard: ``"IF NOT EXISTS "`` or ``""`` (emulation supplies idempotency). - where_sql: rendered partial-index WHERE body, or None. - """ - unique_sql = "UNIQUE " if unique else "" - cols_sql = ", ".join(quote_identifier(c, dialect) for c in cols) - name_sql = quote_identifier(index_name, dialect) - where = f" WHERE {where_sql}" if where_sql else "" - name_part = f"{guard}{name_sql}" - d = str(dialect) - using = f"USING {index_type}" if index_type else None - - if using and d in _USING_BEFORE_ON: - # MySQL/MariaDB: USING sits between the index name and ON. - name_part = f"{name_part} {using}" - tail = f"ON {target} ({cols_sql})" - elif using and d in _USING_BEFORE_COLUMNS: - # Postgres: USING sits after ON, before the column list. - tail = f"ON {target} {using} ({cols_sql})" - elif using: - # SingleStore (and the general default): USING after the column list. - tail = f"ON {target} ({cols_sql}) {using}" - else: - tail = f"ON {target} ({cols_sql})" - - return f"CREATE {unique_sql}INDEX {name_part} {tail}{where}" - - -def build_drop_index_sql( - *, - dialect: t.Any, - drop_scope: DropScope, - index_name: str, - target: t.Optional[str], - guard: str, -) -> str: - """Render a DROP INDEX statement. `target` is required (already quoted) when - `drop_scope` is TABLE_SCOPED.""" - name_sql = quote_identifier(index_name, dialect) - if drop_scope is DropScope.TABLE_SCOPED: - return f"DROP INDEX {guard}{name_sql} ON {target}" - return f"DROP INDEX {guard}{name_sql}" -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/test_index_render.py` -Expected: PASS (predicate + builder classes, all green). - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/backends/ibis/_index.py tests/test_unit/backends/ibis/test_index_render.py -git commit -m "feat(ibis): pure CREATE/DROP INDEX builders with dialect USING placement" -``` - ---- - -## Task 5: Harden + add `get_index_exists_sql` introspection SQL - -**Files:** -- Modify: `src/mountainash_data/backends/ibis/operations.py` (add `_sql_literal`; harden `sqlite_/duckdb_/motherduck_get_index_exists_sql` at lines 235-325; add 5 new functions) -- Modify: `src/mountainash_data/backends/ibis/dialects/_registry.py` (import + wire the 5 new functions; re-broaden invariant) -- Test: `tests/test_unit/backends/ibis/test_index_render.py` (append) - -**Interfaces:** -- Consumes: `sqlglot.exp` (already imported in operations.py). -- Produces: `_sql_literal(value: str) -> str`; `postgres_get_index_exists_sql`, `mysql_get_index_exists_sql`, `mssql_get_index_exists_sql`, `oracle_get_index_exists_sql`, `singlestore_get_index_exists_sql`, each `(index_name: str, table_name: str | None, database: str | None) -> str` returning a `SELECT COUNT(*) AS count ...` query with **escaped** literals. - -- [ ] **Step 1: Write the failing test** - -Append to `tests/test_unit/backends/ibis/test_index_render.py`: - -```python -from mountainash_data.backends.ibis.operations import ( - _sql_literal, - postgres_get_index_exists_sql, - mysql_get_index_exists_sql, - mssql_get_index_exists_sql, - oracle_get_index_exists_sql, - singlestore_get_index_exists_sql, - sqlite_get_index_exists_sql, -) - - -class TestIntrospectionSql: - def test_sql_literal_escapes_quote(self): - assert _sql_literal("x'y") == "'x''y'" - - def test_existing_sqlite_now_escapes(self): - sql = sqlite_get_index_exists_sql("a'b", "t", None) - assert "'a''b'" in sql - assert "count" in sql.lower() - - def test_postgres_shape_and_escaping(self): - sql = postgres_get_index_exists_sql("idx", "t", "public") - assert "pg_indexes" in sql - assert "'idx'" in sql and "'t'" in sql and "'public'" in sql - assert "count" in sql.lower() - - def test_mysql_is_table_scoped(self): - sql = mysql_get_index_exists_sql("idx", "t", None) - assert "STATISTICS" in sql.upper() - assert "'idx'" in sql and "'t'" in sql - - def test_mssql_uses_object_id(self): - sql = mssql_get_index_exists_sql("idx", "t", None) - assert "sys.indexes" in sql and "OBJECT_ID" in sql.upper() - - def test_oracle_matches_exact_quoted_name(self): - # Always-quoted create -> Oracle stores as written -> match exactly, no UPPER(). - sql = oracle_get_index_exists_sql("idx", "t", None) - assert "user_indexes" in sql.lower() - assert "UPPER" not in sql.upper() - assert "'idx'" in sql - - def test_singlestore_shape(self): - sql = singlestore_get_index_exists_sql("idx", "t", None) - assert "STATISTICS" in sql.upper() and "'t'" in sql - # always schema-constrained (defaults to DATABASE() when omitted) to - # avoid cross-schema false positives - assert "TABLE_SCHEMA = DATABASE()" in sql.upper() - - @pytest.mark.parametrize("fn", [ - postgres_get_index_exists_sql, mysql_get_index_exists_sql, - mssql_get_index_exists_sql, oracle_get_index_exists_sql, - singlestore_get_index_exists_sql, - ]) - def test_injection_payload_is_escaped_not_broken(self, fn): - # These pure SQL builders are ALLOWLIST-EXEMPT by design: the front-door - # rejection (the primary gate) is enforced by the generic dispatcher - # (_generic_index_exists) before any builder is called — see Task 6's - # `test_bad_identifier_rejected`. This test asserts the SECOND layer: - # even if a hostile value reached a builder, it is contained in an - # escaped literal (doubled quote), not interpolated raw. - sql = fn("x'; DROP TABLE t; --", "t", None) - assert "''" in sql -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/test_index_render.py::TestIntrospectionSql` -Expected: FAIL with `ImportError: cannot import name '_sql_literal'`. - -- [ ] **Step 3: Add `_sql_literal`, harden existing 3, add 5 new** - -In `operations.py`, after the imports (top of file), confirm `from sqlglot import exp` is present (it is — used by `build_rename_sql`). Add near the other helpers (after `_validate_simple_identifier`, ~line 159): - -```python -def _sql_literal(value: str) -> str: - """Render `value` as an escaped SQL string literal (defense-in-depth for the - catalog-introspection queries; identifiers are also allowlist-validated by - the generic dispatcher before reaching here).""" - return exp.Literal.string(value).sql() -``` - -Replace the bodies of the three existing functions (`sqlite_get_index_exists_sql`, `duckdb_get_index_exists_sql`, `motherduck_get_index_exists_sql`) to use `_sql_literal` instead of raw f-string interpolation. Example for sqlite: - -```python -def sqlite_get_index_exists_sql( - index_name: str, table_name: str | None, database: str | None -) -> str: - """SQLite uses the sqlite_master system table. `database` is unused (no - cross-database queries).""" - where_clauses = ["type = 'index'", f"name = {_sql_literal(index_name)}"] - if table_name: - where_clauses.append(f"tbl_name = {_sql_literal(table_name)}") - where_sql = " AND ".join(where_clauses) - return f"SELECT COUNT(*) AS count FROM sqlite_master WHERE {where_sql}" -``` - -Apply the same `_sql_literal` substitution to `duckdb_get_index_exists_sql` (keys `index_name`/`table_name`/`database_name`) and `motherduck_get_index_exists_sql` (identical to duckdb). - -Add the 5 new functions (place them grouped with the existing introspection functions, ~after line 325): - -```python -# --- PostgreSQL --- - -def postgres_get_index_exists_sql( - index_name: str, table_name: str | None, database: str | None -) -> str: - """PostgreSQL pg_indexes catalog view. `database` maps to schemaname.""" - where = [f"indexname = {_sql_literal(index_name)}"] - if table_name: - where.append(f"tablename = {_sql_literal(table_name)}") - if database: - where.append(f"schemaname = {_sql_literal(database)}") - return f"SELECT COUNT(*) AS count FROM pg_indexes WHERE {' AND '.join(where)}" - - -# --- MySQL / MariaDB --- - -def mysql_get_index_exists_sql( - index_name: str, table_name: str | None, database: str | None -) -> str: - """information_schema.STATISTICS (table-scoped). Defaults schema to the - current database when `database` is omitted.""" - where = [f"INDEX_NAME = {_sql_literal(index_name)}"] - if table_name: - where.append(f"TABLE_NAME = {_sql_literal(table_name)}") - schema_pred = ( - f"TABLE_SCHEMA = {_sql_literal(database)}" if database else "TABLE_SCHEMA = DATABASE()" - ) - where.append(schema_pred) - return ( - "SELECT COUNT(*) AS count FROM information_schema.STATISTICS " - f"WHERE {' AND '.join(where)}" - ) - - -# --- SQL Server --- - -def mssql_get_index_exists_sql( - index_name: str, table_name: str | None, database: str | None -) -> str: - """sys.indexes joined to the table via OBJECT_ID (table-scoped). - - NOTE on the `database` parameter: across this package `database` denotes the - immediate NAMESPACE qualifier, which SQL Server interprets as the *schema* in - a two-part name. The generic CREATE renders ``"".""`` (a - schema.object reference to SQL Server), so OBJECT_ID('.
') - targets the same object — consistent, not conflated. Cross-database - (three-part) index DDL is out of scope for the generic path. - """ - obj = table_name if table_name else "" - if database and table_name: - obj = f"{database}.{table_name}" - return ( - "SELECT COUNT(*) AS count FROM sys.indexes " - f"WHERE name = {_sql_literal(index_name)} " - f"AND object_id = OBJECT_ID({_sql_literal(obj)})" - ) - - -# --- Oracle --- - -def oracle_get_index_exists_sql( - index_name: str, table_name: str | None, database: str | None -) -> str: - """user_indexes (schema-global). The generic builder ALWAYS quotes - identifiers (quote_identifier), so Oracle stores them case-sensitively as - written — match the EXACT name, do NOT fold with UPPER() (a UPPER() match - would never find a quoted-lowercase index).""" - where = [f"index_name = {_sql_literal(index_name)}"] - if table_name: - where.append(f"table_name = {_sql_literal(table_name)}") - return f"SELECT COUNT(*) AS count FROM user_indexes WHERE {' AND '.join(where)}" - - -# --- SingleStore --- - -def singlestore_get_index_exists_sql( - index_name: str, table_name: str | None, database: str | None -) -> str: - """information_schema.STATISTICS (MySQL-compatible, table-scoped). Like - MySQL, ALWAYS constrain TABLE_SCHEMA — defaulting to DATABASE() when - `database` is omitted — so an index/table name shared across schemas cannot - produce a cross-schema false positive.""" - where = [f"INDEX_NAME = {_sql_literal(index_name)}"] - if table_name: - where.append(f"TABLE_NAME = {_sql_literal(table_name)}") - schema_pred = ( - f"TABLE_SCHEMA = {_sql_literal(database)}" if database else "TABLE_SCHEMA = DATABASE()" - ) - where.append(schema_pred) - return ( - "SELECT COUNT(*) AS count FROM information_schema.STATISTICS " - f"WHERE {' AND '.join(where)}" - ) -``` - -In `_registry.py`, extend the import block (line 645) and wire each spec's `get_index_exists_sql`: - -```python -from mountainash_data.backends.ibis.operations import ( # noqa: E402 - duckdb_get_index_exists_sql, - duckdb_get_list_indexes_sql, - sqlite_get_index_exists_sql, - sqlite_get_list_indexes_sql, - motherduck_get_index_exists_sql, - motherduck_get_list_indexes_sql, - postgres_get_index_exists_sql, - mysql_get_index_exists_sql, - mssql_get_index_exists_sql, - oracle_get_index_exists_sql, - singlestore_get_index_exists_sql, - duckdb_family_create_index, - duckdb_family_drop_index, -) -``` - -For each of postgres/mysql/mssql/oracle/singlestoredb, add **both** `get_index_exists_sql=...` **and** `index_caps=IndexCapability(...)` to the spec in the SAME edit (so the §3 invariant holds at this commit). Use the verified values: - -```python - # postgres: - get_index_exists_sql=postgres_get_index_exists_sql, - index_caps=IndexCapability( - drop_scope=DropScope.SCHEMA_GLOBAL, partial=True, - native_if_not_exists=True, native_if_exists=True, - index_types=frozenset({"btree", "hash", "gist", "gin", "brin", "spgist"}), - ), - # mysql: - get_index_exists_sql=mysql_get_index_exists_sql, - index_caps=IndexCapability( - drop_scope=DropScope.TABLE_SCOPED, partial=False, - native_if_not_exists=False, native_if_exists=False, - index_types=frozenset({"btree"}), - ), - # singlestoredb: - get_index_exists_sql=singlestore_get_index_exists_sql, - index_caps=IndexCapability( - drop_scope=DropScope.TABLE_SCOPED, partial=False, - native_if_not_exists=False, native_if_exists=False, - index_types=frozenset({"btree", "hash"}), - ), - # mssql: - get_index_exists_sql=mssql_get_index_exists_sql, - index_caps=IndexCapability( - drop_scope=DropScope.TABLE_SCOPED, partial=True, - native_if_not_exists=False, native_if_exists=True, - index_types=frozenset(), - ), - # oracle: - get_index_exists_sql=oracle_get_index_exists_sql, - index_caps=IndexCapability( - drop_scope=DropScope.SCHEMA_GLOBAL, partial=False, - native_if_not_exists=False, native_if_exists=False, - index_types=frozenset(), - ), -``` - -Broaden the capability tests in `test_index_capability.py` to all 8 by re-pointing the parametrized lists (the `_EXPECTED` table already holds all 8): - -```python -@pytest.mark.parametrize("name", list(_EXPECTED)) -def test_index_caps_matrix(name): - caps = DIALECTS[name].index_caps - assert caps is not None, f"{name} must have index_caps" - drop_scope, partial, ine, ie, types = _EXPECTED[name] - assert caps.drop_scope is drop_scope - assert caps.partial is partial - assert caps.native_if_not_exists is ine - assert caps.native_if_exists is ie - assert caps.index_types == types - - -@pytest.mark.parametrize("name", list(_EXPECTED)) -def test_invariant_caps_implies_exists_sql(name): - spec = DIALECTS[name] - assert spec.index_caps is not None - assert spec.get_index_exists_sql is not None -``` - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/test_index_render.py::TestIntrospectionSql tests/test_unit/backends/ibis/test_index_capability.py` -Expected: PASS (introspection golden + full 8-dialect invariant). - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/backends/ibis/operations.py src/mountainash_data/backends/ibis/dialects/_registry.py tests/test_unit/backends/ibis/test_index_render.py tests/test_unit/backends/ibis/test_index_capability.py -git commit -m "feat(ibis): escape+harden index introspection SQL + assign caps for pg/mysql/mssql/oracle/singlestore" -``` - ---- - -## Task 6: Generic dispatchers `_generic_create_index` / `_generic_drop_index` / `_generic_index_exists` - -**Files:** -- Modify: `src/mountainash_data/backends/ibis/_index.py` (append) -- Test: `tests/test_unit/backends/ibis/test_index_ops.py` - -**Interfaces:** -- Consumes: `_render.dialect_of`, `_render.qualified_name`, `_render.compile_index_predicate`, `_render.IndexPredicate`; `operations._validate_simple_identifier`, `operations._normalize_columns`, `operations._generate_index_name`; `_registry.IndexCapability`, `_registry.DropScope`. -- Produces: - - `_generic_index_exists(ibis_conn, index_name, *, table_name=None, database=None, exists_sql_fn) -> bool` - - `_generic_create_index(ibis_conn, table_name, columns, *, index_name=None, unique=False, index_type=None, where=None, database=None, if_not_exists=True, caps, exists_sql_fn) -> None` - - `_generic_drop_index(ibis_conn, index_name, *, table_name=None, database=None, if_exists=True, caps, exists_sql_fn) -> None` - -**Emulation correctness assumptions (spec §6):** the emulation precheck trusts `index_exists` as authoritative for the current session/principal. Per the spec, the following are documented-and-accepted failure modes (the engine's error is surfaced, never swallowed): the TOCTOU window between check and act; a false negative when the principal can create/drop but cannot see the index in the catalog (privilege); cached/transaction-isolated catalog metadata lagging a recent DDL; and the fact that MySQL/Oracle auto-commit DDL — which is *why* the check+act window cannot be closed transactionally. No catch-and-swallow, no lock wrapping. (Add a one-line docstring reference to spec §6 on `_generic_create_index`/`_generic_drop_index`.) - -- [ ] **Step 1: Write the failing test** - -Create `tests/test_unit/backends/ibis/test_index_ops.py`: - -```python -"""Generic index dispatchers, exercised on in-memory sqlite/duckdb.""" - -import ibis -import polars as pl -import pytest - -from mountainash_data.backends.ibis._index import ( - _generic_create_index, - _generic_drop_index, - _generic_index_exists, -) -from mountainash_data.backends.ibis.dialects._registry import DIALECTS - -_SQLITE = DIALECTS["sqlite"].index_caps -_SQLITE_FN = DIALECTS["sqlite"].get_index_exists_sql -_DUCKDB = DIALECTS["duckdb"].index_caps -_DUCKDB_FN = DIALECTS["duckdb"].get_index_exists_sql - - -def _seed_sqlite(): - con = ibis.sqlite.connect() - con.create_table("t", pl.DataFrame({"id": [1, 2], "active": [True, False]})) - return con - - -class TestCreateDropExistsRoundtrip: - def test_create_then_exists_then_drop(self): - con = _seed_sqlite() - _generic_create_index( - con, "t", ["id"], index_name="idx_t_id", caps=_SQLITE, - exists_sql_fn=_SQLITE_FN, - ) - assert _generic_index_exists(con, "idx_t_id", table_name="t", - exists_sql_fn=_SQLITE_FN) is True - _generic_drop_index(con, "idx_t_id", table_name="t", caps=_SQLITE, - exists_sql_fn=_SQLITE_FN) - assert _generic_index_exists(con, "idx_t_id", table_name="t", - exists_sql_fn=_SQLITE_FN) is False - - def test_create_if_not_exists_is_idempotent_native(self): - con = _seed_sqlite() - for _ in range(2): - _generic_create_index( - con, "t", ["id"], index_name="idx_t_id", if_not_exists=True, - caps=_SQLITE, exists_sql_fn=_SQLITE_FN, - ) # second call must not raise (native IF NOT EXISTS) - - def test_default_index_name_generated(self): - con = _seed_sqlite() - _generic_create_index(con, "t", ["id"], caps=_SQLITE, exists_sql_fn=_SQLITE_FN) - assert _generic_index_exists(con, "idx_t_id", table_name="t", - exists_sql_fn=_SQLITE_FN) is True - - -class TestPartialIndex: - def test_partial_where_on_sqlite(self): - con = _seed_sqlite() - _generic_create_index( - con, "t", ["id"], index_name="idx_active", - where=lambda r: r.active == True, caps=_SQLITE, # noqa: E712 - exists_sql_fn=_SQLITE_FN, - ) - assert _generic_index_exists(con, "idx_active", table_name="t", - exists_sql_fn=_SQLITE_FN) is True - - def test_where_on_non_partial_dialect_raises(self): - con = ibis.duckdb.connect() - con.create_table("t", pl.DataFrame({"id": [1], "active": [True]})) - with pytest.raises(ValueError, match="partial"): - _generic_create_index( - con, "t", ["id"], where=lambda r: r.active, caps=_DUCKDB, - exists_sql_fn=_DUCKDB_FN, - ) - - -class TestValidationErrors: - def test_unsupported_index_type_raises(self): - con = _seed_sqlite() - with pytest.raises(ValueError, match="index_type"): - _generic_create_index( - con, "t", ["id"], index_type="hash", caps=_SQLITE, - exists_sql_fn=_SQLITE_FN, - ) - - def test_table_scoped_drop_requires_table_name(self): - con = _seed_sqlite() - mysql_caps = DIALECTS["mysql"].index_caps - with pytest.raises(ValueError, match="table_name"): - _generic_drop_index(con, "idx", table_name=None, caps=mysql_caps, - exists_sql_fn=DIALECTS["mysql"].get_index_exists_sql) - - def test_bad_identifier_rejected(self): - con = _seed_sqlite() - with pytest.raises(ValueError, match="simple identifier"): - _generic_create_index(con, "t", ["id"], index_name="x; DROP", - caps=_SQLITE, exists_sql_fn=_SQLITE_FN) - - def test_drop_if_exists_absent_is_noop_native(self): - con = _seed_sqlite() - _generic_drop_index(con, "nope", table_name="t", if_exists=True, - caps=_SQLITE, exists_sql_fn=_SQLITE_FN) # no raise -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/test_index_ops.py` -Expected: FAIL with `ImportError: cannot import name '_generic_create_index'`. - -- [ ] **Step 3: Implement the dispatchers** - -Append to `_index.py` (add the imports at the top of the file first): - -```python -import ibis # add to top-of-file imports - -from mountainash_data.backends.ibis._render import ( - compile_index_predicate, - dialect_of, - qualified_name, -) -from mountainash_data.backends.ibis.dialects._registry import IndexCapability -from mountainash_data.backends.ibis.operations import ( - _generate_index_name, - _normalize_columns, - _validate_simple_identifier, -) -``` - -```python -def _generic_index_exists( - ibis_conn: t.Any, - index_name: str, - *, - table_name: t.Optional[str] = None, - database: t.Optional[str] = None, - exists_sql_fn: t.Any, -) -> bool: - """Run the dialect's introspection SQL and return whether the index exists.""" - if exists_sql_fn is None: - raise NotImplementedError("dialect has no get_index_exists_sql") - _validate_simple_identifier(index_name, kind="index_name") - if table_name is not None: - _validate_simple_identifier(table_name, kind="table_name") - if database is not None: - _validate_simple_identifier(database, kind="database") - result = ibis_conn.sql(exists_sql_fn(index_name, table_name, database)) - if result is None: - return False - import mountainash as ma - - # Read the single returned column BY POSITION, not by the alias name: - # Oracle upper-cases the unquoted `count` alias ("count" -> "COUNT"), so - # keying by "count" would KeyError. Every introspection query returns - # exactly one column. - data = ma.relation(result).to_dict() - first_col = next(iter(data.values())) - return first_col[0] > 0 - - -def _target_ref(ibis_conn: t.Any, table_name: str, database: t.Optional[str]) -> str: - dialect = dialect_of(ibis_conn) - parts = [database, table_name] if database else [table_name] - return qualified_name(parts, dialect) - - -def _generic_create_index( - ibis_conn: t.Any, - table_name: str, - columns: t.Union[list[str], str], - *, - index_name: t.Optional[str] = None, - unique: bool = False, - index_type: t.Optional[str] = None, - where: t.Any = None, - database: t.Optional[str] = None, - if_not_exists: bool = True, - caps: IndexCapability, - exists_sql_fn: t.Any, -) -> None: - """Render and execute a CREATE INDEX via the generic path (spec §5-§8).""" - _validate_simple_identifier(table_name, kind="table_name") - if database is not None: - _validate_simple_identifier(database, kind="database") - cols = _normalize_columns(columns) - for c in cols: - _validate_simple_identifier(c, kind="column") - - if index_type is not None and index_type not in caps.index_types: - raise ValueError( - f"index_type {index_type!r} not supported by this dialect; " - f"valid: {sorted(caps.index_types) or 'none'}" - ) - if where is not None and not caps.partial: - raise ValueError("this dialect does not support partial indexes (where=)") - - if index_name is None: - index_name = _generate_index_name(table_name, cols, unique=unique) - _validate_simple_identifier(index_name, kind="index_name") - - # Idempotency: native guard, or emulate via precheck. - guard = "" - if if_not_exists: - if caps.native_if_not_exists: - guard = "IF NOT EXISTS " - elif _generic_index_exists( - ibis_conn, index_name, table_name=table_name, database=database, - exists_sql_fn=exists_sql_fn, - ): - return # emulated: already present - - where_sql = None - if where is not None: - schema = ibis_conn.table(table_name, database=database).schema() - where_sql = compile_index_predicate(ibis_conn, schema, table_name, where) - - sql = build_create_index_sql( - dialect=dialect_of(ibis_conn), - target=_target_ref(ibis_conn, table_name, database), - index_name=index_name, cols=cols, unique=unique, - index_type=index_type, guard=guard, where_sql=where_sql, - ) - ibis_conn.raw_sql(sql) - - -def _generic_drop_index( - ibis_conn: t.Any, - index_name: str, - *, - table_name: t.Optional[str] = None, - database: t.Optional[str] = None, - if_exists: bool = True, - caps: IndexCapability, - exists_sql_fn: t.Any, -) -> None: - """Render and execute a DROP INDEX via the generic path (spec §5-§8).""" - _validate_simple_identifier(index_name, kind="index_name") - if caps.drop_scope is DropScope.TABLE_SCOPED and table_name is None: - raise ValueError( - "drop_index requires table_name for this dialect (DROP INDEX ... ON tbl)" - ) - if table_name is not None: - _validate_simple_identifier(table_name, kind="table_name") - if database is not None: - _validate_simple_identifier(database, kind="database") - - guard = "" - if if_exists: - if caps.native_if_exists: - guard = "IF EXISTS " - elif not _generic_index_exists( - ibis_conn, index_name, table_name=table_name, database=database, - exists_sql_fn=exists_sql_fn, - ): - return # emulated: already absent - - target = _target_ref(ibis_conn, table_name, database) if table_name else None - sql = build_drop_index_sql( - dialect=dialect_of(ibis_conn), drop_scope=caps.drop_scope, - index_name=index_name, target=target, guard=guard, - ) - ibis_conn.raw_sql(sql) -``` - -Note: `import ibis` is needed only if referenced; the dispatchers use `ibis_conn` directly, so the `import ibis` line may be unnecessary — include it only if a linter flags an undefined name (it is not used in the code above; omit it if `ruff` reports F401). - -- [ ] **Step 4: Run test to verify it passes** - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/test_index_ops.py` -Expected: PASS (roundtrip, partial, validation errors all green). - -- [ ] **Step 5: Commit** - -```bash -git add src/mountainash_data/backends/ibis/_index.py tests/test_unit/backends/ibis/test_index_ops.py -git commit -m "feat(ibis): generic create/drop/exists dispatchers with emulation + validation" -``` - ---- - -## Task 7: Backend wiring + atomic cutover - -**Why merged:** the backend's `create_index` checks `create_index_hook` first. While sqlite/duckdb/motherduck still carry `create_index_hook=duckdb_family_create_index`, the rewritten backend would dispatch `where=` to that hook — whose signature is the OLD `where_condition=str` — raising `TypeError`. And removing the hooks *before* the backend rewrite leaves `create_index` raising `NotImplementedError` (old code path), breaking the existing functional tests. The hook removal and the backend rewrite must therefore land in ONE commit. This task fuses the wiring and the cutover so the suite is green at a single commit. - -**Files:** -- Modify: `src/mountainash_data/backends/ibis/backend.py` (`create_index` lines 576-599, `create_unique_index` 601-614, `drop_index` 616-633, `index_exists` 635-654; add the `_index` import) -- Modify: `src/mountainash_data/backends/ibis/dialects/_registry.py` (remove `create_index_hook=`/`drop_index_hook=` from sqlite/duckdb/motherduck; drop the two names from the import block) -- Modify: `src/mountainash_data/backends/ibis/operations.py` (delete `duckdb_family_create_index` ~lines 362-398 and `duckdb_family_drop_index` ~401-414; remove now-unused `contextlib`/`warnings`/`CONST_INDEX_TYPE` imports IF unused after deletion) -- Test: `tests/test_unit/backends/ibis/test_index_ops.py` (append backend class), `tests/test_unit/backends/ibis/test_backend.py` (update 3 mechanism tests) - -**Interfaces:** -- Consumes: `_index._generic_create_index`, `_index._generic_drop_index`, `_index._generic_index_exists`. -- Produces: `IbisBackend.create_index(table_name, columns, *, index_name=None, unique=False, index_type=None, where=None, database=None, if_not_exists=True) -> IbisBackend`; `drop_index(index_name, *, table_name=None, database=None, if_exists=True) -> IbisBackend`; `index_exists(index_name, *, table_name=None, database=None) -> bool`. - -- [ ] **Step 1: Write the failing tests** - -Append to `tests/test_unit/backends/ibis/test_index_ops.py` (note: seed via the backend's own `create_table`, NOT `be._ibis_conn` — the raw connection lives on `be._require_connected()._ibis_conn`, not on the backend): - -```python -from mountainash_data import IbisBackend - - -class TestBackendDispatch: - def test_create_exists_drop_via_backend(self): - be = IbisBackend(dialect="sqlite", database=":memory:") - be.connect() - try: - be.create_table("t", pl.DataFrame({"id": [1], "active": [True]}), - overwrite=True) - assert be.create_index("t", ["id"], index_name="ix") is be - assert be.index_exists("ix", table_name="t") is True - assert be.drop_index("ix", table_name="t") is be - assert be.index_exists("ix", table_name="t") is False - finally: - be.close() - - def test_where_predicate_via_backend(self): - be = IbisBackend(dialect="sqlite", database=":memory:") - be.connect() - try: - be.create_table("t", pl.DataFrame({"id": [1], "active": [True]}), - overwrite=True) - be.create_index("t", ["id"], index_name="ixp", - where=lambda r: r.active == True) # noqa: E712 - assert be.index_exists("ixp", table_name="t") is True - finally: - be.close() - - def test_unsupported_dialect_raises_notimplemented(self): - from mountainash_data.backends.ibis.dialects._registry import DialectSpec - be = IbisBackend(dialect="sqlite", database=":memory:") - be.connect() - try: - # Rebind the INSTANCE's _spec to a fresh no-index spec (index_caps and - # create_index_hook default to None). Never mutate the shared frozen - # singleton in DIALECTS — that would corrupt other tests. - be._spec = DialectSpec( - ibis_backend_name="sqlite", - connection_mode="connection_string", - connection_string_scheme="sqlite://", - ) - with pytest.raises(NotImplementedError): - be.create_index("t", ["id"]) - finally: - be.close() -``` - -In `tests/test_unit/backends/ibis/test_backend.py`, replace `test_sqlite_dialect_has_create_index_hook` (line ~229) with a generic-dispatch assertion, and add the retired-symbol guard: - -```python -def test_sqlite_dialect_uses_generic_index_path(): - """After cutover, sqlite has no index hooks and dispatches via index_caps.""" - from mountainash_data.backends.ibis.dialects._registry import DIALECTS - spec = DIALECTS["sqlite"] - assert spec.create_index_hook is None - assert spec.drop_index_hook is None - assert spec.index_caps is not None - - -def test_duckdb_family_index_hooks_removed(): - import mountainash_data.backends.ibis.operations as ops - assert not hasattr(ops, "duckdb_family_create_index") - assert not hasattr(ops, "duckdb_family_drop_index") - - -def test_no_dialect_carries_an_index_hook_post_cutover(): - """The generic path is the ONLY index path after cutover: no dialect carries - a create/drop index hook, so the backend's hook-first branch (which forwards - the new `where=` predicate) is never exercised — keeping it dead and safe. - The hook fields remain only as a future override escape hatch; CONTRACT: any - future create_index_hook MUST accept create_index's keyword signature, - including `where` (the ibis predicate), and any drop_index_hook MUST accept - `table_name`/`database`/`if_exists`.""" - from mountainash_data.backends.ibis.dialects._registry import DIALECTS - for name, spec in DIALECTS.items(): - assert spec.create_index_hook is None, f"{name} unexpectedly has create_index_hook" - assert spec.drop_index_hook is None, f"{name} unexpectedly has drop_index_hook" -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/test_index_ops.py::TestBackendDispatch "tests/test_unit/backends/ibis/test_backend.py::test_duckdb_family_index_hooks_removed"` -Expected: FAIL — backend still routes through the hook (rejecting `where=`), and the `duckdb_family_*` symbols still exist. - -- [ ] **Step 3a: Rewrite the four backend methods** - -Add the import near the other `_index`/operations imports at the top of `backend.py`: - -```python -from mountainash_data.backends.ibis._index import ( - _generic_create_index, - _generic_drop_index, - _generic_index_exists, -) -``` - -Replace `create_index`: - -```python - def create_index( - self, - table_name: str, - columns: list[str] | str, - *, - index_name: str | None = None, - unique: bool = False, - index_type: str | None = None, - where: t.Any = None, # IndexPredicate | None - database: str | None = None, - if_not_exists: bool = True, - ) -> IbisBackend: - conn = self._require_connected() - hook = self._spec.create_index_hook - if hook is not None: - hook( - conn._ibis_conn, table_name, columns, - index_name=index_name, unique=unique, index_type=index_type, - where=where, database=database, if_not_exists=if_not_exists, - ) - elif self._spec.index_caps is not None: - _generic_create_index( - conn._ibis_conn, table_name, columns, - index_name=index_name, unique=unique, index_type=index_type, - where=where, database=database, if_not_exists=if_not_exists, - caps=self._spec.index_caps, - exists_sql_fn=self._spec.get_index_exists_sql, - ) - else: - raise NotImplementedError( - f"Dialect {self.dialect!r} does not support create_index" - ) - return self -``` - -Replace `create_unique_index` (drop the removed `where_condition`, use `where`): - -```python - def create_unique_index( - self, - table_name: str, - columns: list[str] | str, - *, - index_name: str | None = None, - where: t.Any = None, # IndexPredicate | None - database: str | None = None, - ) -> IbisBackend: - return self.create_index( - table_name, columns, - index_name=index_name, unique=True, where=where, database=database, - ) -``` - -Replace `drop_index`: - -```python - def drop_index( - self, - index_name: str, - *, - table_name: str | None = None, - database: str | None = None, - if_exists: bool = True, - ) -> IbisBackend: - conn = self._require_connected() - hook = self._spec.drop_index_hook - if hook is not None: - hook( - conn._ibis_conn, index_name, - table_name=table_name, database=database, if_exists=if_exists, - ) - elif self._spec.index_caps is not None: - _generic_drop_index( - conn._ibis_conn, index_name, - table_name=table_name, database=database, if_exists=if_exists, - caps=self._spec.index_caps, - exists_sql_fn=self._spec.get_index_exists_sql, - ) - else: - raise NotImplementedError( - f"Dialect {self.dialect!r} does not support drop_index" - ) - return self -``` - -Replace `index_exists` body to delegate to the shared dispatcher (single place for the count extraction): - -```python - def index_exists( - self, - index_name: str, - *, - table_name: str | None = None, - database: str | None = None, - ) -> bool: - if self._spec.get_index_exists_sql is None: - raise NotImplementedError( - f"Dialect {self.dialect!r} does not support index_exists" - ) - conn = self._require_connected() - return _generic_index_exists( - conn._ibis_conn, index_name, - table_name=table_name, database=database, - exists_sql_fn=self._spec.get_index_exists_sql, - ) -``` - -- [ ] **Step 3b: Retire the duckdb-family hooks (same commit)** - -In `_registry.py`: remove `duckdb_family_create_index` and `duckdb_family_drop_index` from the import block, and remove the `create_index_hook=duckdb_family_create_index,` / `drop_index_hook=duckdb_family_drop_index,` lines from the sqlite, duckdb, and motherduck specs. - -In `operations.py`: delete `duckdb_family_create_index` and `duckdb_family_drop_index` (~lines 362-414). Then verify whether their now-orphaned imports are still used elsewhere: - -Run: `grep -n "contextlib\|warnings\.\|CONST_INDEX_TYPE" src/mountainash_data/backends/ibis/operations.py` -Remove any import that grep shows is no longer referenced. - -- [ ] **Step 4: `where_condition` audit + full suite + gates** - -Run: `grep -rn "where_condition" src/ tests/` -Expected: no matches in `src/` or `tests/` (the spec doc may reference it historically — acceptable). Fix any live callsite found. - -Run: `hatch run test:test-target-quick tests/test_unit/backends/ibis/` -Expected: PASS — including the pre-existing functional `test_create_index_returns_self` / `test_drop_index_returns_self` / `test_index_exists` (now flowing through the generic path), plus the new dispatch + cutover tests. - -Run: `hatch run ruff:check` then `hatch run mypy:check` -Expected: ruff clean; mypy Success. - -- [ ] **Step 5: Commit (atomic)** - -```bash -git add src/mountainash_data/backends/ibis/backend.py src/mountainash_data/backends/ibis/dialects/_registry.py src/mountainash_data/backends/ibis/operations.py tests/test_unit/backends/ibis/test_index_ops.py tests/test_unit/backends/ibis/test_backend.py -git commit -m "feat(ibis): wire generic index dispatch + atomic cutover of duckdb_family hooks" -``` - ---- - -## Task 8: Live integration — postgres + mariadb round-trips, partial, table-scoped, emulation - -**Files:** -- Create: `tests/test_integration/test_index_ops_live.py` -- Uses: existing `tests/fixtures/database_fixtures.py` (`postgres_backend`, `mysql_backend`) and `compose.yaml`. - -**Interfaces:** -- Consumes: `postgres_backend`, `mysql_backend` fixtures (skip-if-unreachable; fail-closed under `MOUNTAINASH_REQUIRE_LIVE_DB=1`). - -- [ ] **Step 1: Write the live test** - -Create `tests/test_integration/test_index_ops_live.py`: - -```python -"""Live index ops against postgres (native) and mariadb (table-scoped + emulated).""" - -import polars as pl -import pytest - -pytestmark = pytest.mark.integration - - -def _fresh_table(be, name): - # the raw ibis connection lives on the IbisConnection, not on the backend - conn = be._require_connected()._ibis_conn - try: - conn.drop_table(name, force=True) - except Exception: # noqa: BLE001 - pass - conn.create_table(name, pl.DataFrame({"id": [1, 2, 3], "active": [True, False, True]})) - - -class TestPostgresLive: - def test_roundtrip_and_partial(self, postgres_backend): - be = postgres_backend - _fresh_table(be, "ix_live") - be.create_index("ix_live", ["id"], index_name="ix_live_id") - assert be.index_exists("ix_live_id", table_name="ix_live") is True - # partial (filtered) index — postgres supports WHERE - be.create_index("ix_live", ["id"], index_name="ix_live_active", - where=lambda r: r.active == True) # noqa: E712 - assert be.index_exists("ix_live_active", table_name="ix_live") is True - be.drop_index("ix_live_id") # schema-global: no table needed - assert be.index_exists("ix_live_id", table_name="ix_live") is False - - def test_using_gin_index_type(self, postgres_backend): - be = postgres_backend - _fresh_table(be, "ix_gin") - be.create_index("ix_gin", ["id"], index_name="ix_gin_btree", index_type="btree") - assert be.index_exists("ix_gin_btree", table_name="ix_gin") is True - - -class TestMariaDBLive: - def test_table_scoped_drop_requires_table(self, mysql_backend): - be = mysql_backend - _fresh_table(be, "ix_my") - be.create_index("ix_my", ["id"], index_name="ix_my_id") - assert be.index_exists("ix_my_id", table_name="ix_my") is True - # schema-global drop must be rejected for a TABLE_SCOPED dialect - with pytest.raises(ValueError, match="table_name"): - be.drop_index("ix_my_id") - be.drop_index("ix_my_id", table_name="ix_my") - assert be.index_exists("ix_my_id", table_name="ix_my") is False - - def test_emulated_if_not_exists_is_idempotent(self, mysql_backend): - be = mysql_backend - _fresh_table(be, "ix_emu") - # mysql dialect emulates IF NOT EXISTS via precheck; double-create is a no-op - be.create_index("ix_emu", ["id"], index_name="ix_emu_id", if_not_exists=True) - be.create_index("ix_emu", ["id"], index_name="ix_emu_id", if_not_exists=True) - assert be.index_exists("ix_emu_id", table_name="ix_emu") is True - - def test_emulated_if_exists_drop_absent_is_noop(self, mysql_backend): - be = mysql_backend - _fresh_table(be, "ix_emu2") - be.drop_index("nope", table_name="ix_emu2", if_exists=True) # no raise -``` - -- [ ] **Step 2: Start the live databases** - -```bash -docker compose -f compose.yaml up -d -``` -Expected: postgres and mariadb containers healthy. - -- [ ] **Step 3: Run the live suite (fail-closed)** - -Run: `MOUNTAINASH_REQUIRE_LIVE_DB=1 hatch run test:test-target tests/test_integration/test_index_ops_live.py` -Expected: PASS (postgres + mariadb classes green). If a fixture skips under this flag, the env is misconfigured — fix connectivity, do not weaken the test. - -- [ ] **Step 4: Run the full suite + gates** - -```bash -hatch run test:test-target-quick tests/test_unit/ -hatch run ruff:check -hatch run mypy:check -``` -Expected: unit green; ruff clean; mypy Success. - -- [ ] **Step 5: Commit** - -```bash -git add tests/test_integration/test_index_ops_live.py -git commit -m "test(ibis): live index round-trips (postgres native + mariadb emulated/table-scoped)" -``` - ---- - -## Self-Review - -**Task count: 8** (after merging the original backend-wiring + cutover into Task 7 — they cannot land in separate commits without a broken intermediate state, per the Codex plan review). - -**1. Spec coverage:** -- §2 scope (conventional only, None sentinel) → Task 2 (`_UNSUPPORTED`). -- §3 capability model + invariant (split-assignment keeps it true at every commit) → Tasks 1, 2, 5. -- §4 verified matrix → Tasks 2 + 5 (`_EXPECTED`), Global Constraints. -- §5.1 builders + 3-position USING placement → Task 4. -- §5.2 predicate compiler (AST strip, full-schema bind, non-indexed cols, mssql limitation note) → Task 3. -- §6 idempotency/emulation (+ failure-mode assumptions) + injection contract → Tasks 5 (escaping), 6 (validation + precheck + §6 note). -- §7 public API (`where` predicate) → Task 7. -- §8 error table (all rows) → Task 6 tests (`TestValidationErrors`), Task 7. -- §9 testing (golden render-only, introspection golden, live, registry-consistency) → Tasks 2, 4, 5, 8. -- §10 cutover (retire family, `where_condition` audit, new introspection) → Tasks 5, 7. -- §11 file structure → matches. -- mssql/oracle/singlestore are render-only (no live container) — covered by Task 4/5 golden tests; documented as render-only in the spec. - -**2. Placeholder scan:** No TBD/TODO. Every code step shows full code. The only conditional instruction (remove unused imports in Task 7) is gated on an explicit `grep` check. - -**3. Type consistency:** `IndexCapability` fields, `DropScope` members, dispatcher signatures (`caps=`, `exists_sql_fn=`), and the `where`/`IndexPredicate` param name are identical across Tasks 1→2→6→7. `get_index_exists_sql` signature `(index_name, table_name, database)` matches the existing `GetIndexExistsSql` type alias and all 8 implementations. The `count` extraction reads the single column **by position** (Task 6), so the alias casing is irrelevant across dialects (Oracle upper-cases it). - ---- - -## Execution Handoff - -Plan complete and saved to `docs/superpowers/plans/2026-06-30-generic-default-index-operations.md`. Two execution options: - -**1. Subagent-Driven (recommended)** — fresh subagent per task, two-stage review between tasks. -**2. Inline Execution** — batch execution with checkpoints. diff --git a/docs/superpowers/specs/2026-04-07-mountainash-data-audit-and-redesign.md b/docs/superpowers/specs/2026-04-07-mountainash-data-audit-and-redesign.md deleted file mode 100644 index 3805ceb..0000000 --- a/docs/superpowers/specs/2026-04-07-mountainash-data-audit-and-redesign.md +++ /dev/null @@ -1,275 +0,0 @@ -# mountainash-data — Audit and Redesign - -**Date:** 2026-04-07 -**Status:** Design approved, awaiting written-spec review -**Scope:** Full audit and architectural redesign of the `mountainash-data` package - -## Context - -`mountainash-data` began as an abstraction over Ibis but has drifted. Today it contains: - -- Ibis connection management for 13 backends -- A mixin-based ibis "operations" layer that overlaps awkwardly with the sister package `mountainash-expressions` -- A pyiceberg connection factory that has grown to ~1.7k LOC of mixed connection + operations logic -- A factories layer, settings subtree, and a 3-line lineage stub - -The user is the only consumer (greenfield freedom; breaking changes are fine). The brainstorming session reframed the data/expressions relationship: they are **complementary, not duplicative** — `mountainash-data` is the *physical* layer (connect, inspect, manage backend services), `mountainash-expressions` is the *logical* layer (relations, expressions, query construction). The real defect is that `mountainash-data`'s architecture is antiquated and incompatible with the protocol-based, composable shape that `mountainash-expressions` uses. - -This document captures the audit findings and the target architecture, then defines a phased migration sequence. - -## Defects identified - -The user confirmed four primary defects in the current architecture: - -1. **Class explosion** — one connection class + one operations class per backend (26+ files), most of which are stubs or near-stubs -2. **Operations-as-mixins** — `_base_ibis_mixin` / `_duckdb_family_mixin` inheritance is rigid where composable functions would serve better -3. **No clean seam to expressions** — operations return raw ibis tables, not the relation/expression types `mountainash-expressions` speaks; consumers bridge manually -4. **Stateful connection objects** — `BaseDBConnection` carries state that should be a thin factory / context manager - -Settings coupling to `mountainash-settings` is **not** a defect and stays. - -## Target architecture - -### Package identity (one sentence) - -> *`mountainash-data` provides physical access to backend data services — connecting, inspecting, and managing them — through a single `Backend` protocol with peer implementations for ibis-style relational backends and iceberg-style table-format catalogs.* - -### Top-level layout - -``` -src/mountainash_data/ -├── __init__.py -├── __version__.py -├── core/ # protocol layer (abstract, no impls) -│ ├── __init__.py -│ ├── protocol.py # Backend, Connection, Catalog, Namespace, Table, Column protocols -│ ├── inspection.py # CatalogInfo / NamespaceInfo / TableInfo / ColumnInfo dataclasses -│ ├── registry.py # backend registration / lookup by name -│ ├── connection.py # base connection types (was databases/connections/base_db_connection.py) -│ ├── constants.py # was databases/constants.py -│ ├── utils.py # high-level facade (was database_utils.py) -│ ├── factories/ # lazy-loading strategy factories -│ │ ├── __init__.py -│ │ ├── base_strategy_factory.py -│ │ ├── settings_type_factory_mixin.py -│ │ ├── connection_factory.py -│ │ ├── operations_factory.py -│ │ └── settings_factory.py -│ └── settings/ # Pydantic settings classes per backend -│ ├── __init__.py -│ ├── base.py -│ ├── exceptions.py -│ ├── templates.py -│ └── {bigquery,duckdb,motherduck,mssql,mysql,postgresql, -│ pyiceberg_rest,pyspark,redshift,snowflake,sqlite,trino}.py -└── backends/ - ├── ibis/ # ibis implementation (peer) - │ ├── __init__.py - │ ├── backend.py # IbisBackend implements core.Backend - │ ├── connection.py # was base_ibis_connection.py, refactored to consume DialectSpec - │ ├── operations.py # base_ibis_operations + _base_ibis_mixin + _duckdb_family functions - │ ├── inspect.py # ibis → core inspection model - │ └── dialects/ - │ ├── __init__.py - │ └── _registry.py # DialectSpec entries for all 12 ibis backends - └── iceberg/ # iceberg implementation (peer) - ├── __init__.py - ├── backend.py # IcebergBackend implements core.Backend - ├── connection.py # catalog/namespace lifecycle (deduplicated) - ├── operations.py # table mutations (deduplicated) - ├── _types.py # Iceberg → PyArrow conversion helpers - ├── inspect.py # pyiceberg → core inspection model - └── catalogs/ - ├── __init__.py - └── rest.py # REST connection + operations merged -``` - -### Key architectural moves - -1. **`Backend` is a Protocol, not a base class.** No inheritance. Implementations are plain classes that satisfy the protocol. Kills the mixin tree. - -2. **`Backend` is stateless from the consumer's perspective.** Construction takes config; `connect()` returns a `Connection` context manager that owns the live handle. No long-lived stateful connection objects. - -3. **The 13 ibis backend files collapse into a data-driven dialect registry.** Each backend (sqlite, duckdb, motherduck, postgres, mysql, mssql, oracle, snowflake, bigquery, redshift, trino, pyspark) becomes a `DialectSpec` entry containing: connection-builder callable, ibis backend name, connection mode, capability hooks for any backend-specific operations. One `IbisBackend` class drives all of them. - -4. **Operations stop being mixins.** Concrete shared logic from `_base_ibis_mixin.py` (98 LOC), `_duckdb_family_mixin.py` (314 LOC), and `base_ibis_operations.py` (676 LOC, the actual implementation — see audit finding #2 below) folds into `backends/ibis/operations.py` as plain functions. Backend-specific concrete logic from the duckdb/sqlite/motherduck ops files is salvaged and attached to dialect entries via capability hooks. - -5. **The seam to `mountainash-expressions`:** `Backend.connect()` exposes a `to_relation(table_name)` method that hands consumers a `mountainash-expressions` Relation. That is the entire bridge — physical layer hands off to logical layer at the relation boundary. No expression logic lives in `mountainash-data`. - -6. **Shared inspection model (the "Medium" shared layer):** dataclasses for `CatalogInfo → NamespaceInfo → TableInfo → ColumnInfo`. Both `IbisBackend.inspect()` and `IcebergBackend.inspect()` populate the same shapes. This is the *one* concrete thing both paradigms share, and it is where consumers get the most value. - -### Capability hooks pattern - -Replaces mixin inheritance for backend-specific operations: - -```python -DIALECTS = { - "duckdb": DialectSpec( - connect=_build_duckdb_conn, - inspect=ibis_default_inspect, - get_index_exists_sql=duckdb_get_index_exists_sql, # salvaged - get_list_indexes_sql=duckdb_get_list_indexes_sql, # salvaged - ... - ), - "sqlite": DialectSpec( - connect=_build_sqlite_conn, - get_index_exists_sql=sqlite_get_index_exists_sql, - ... - ), - "postgres": DialectSpec( - connect=_build_postgres_conn, - inspect=ibis_default_inspect, - # no extras - ), -} -``` - -Backend-specific operations become **data on the dialect**, not subclass methods. Consumers reach them via `backend.capability("get_index_exists_sql")(args)` or — if usage warrants — first-class methods on a protocol extension. - -### Sanity check: protocol holds for both paradigms - -- **sqlite via ibis:** `IbisBackend(dialect="sqlite", config=...).connect()` → context manager yielding a connection with `list_tables()`, `inspect(table)` → `TableInfo`, `to_relation(table)` → ibis-backed Relation. ✅ -- **pyiceberg-rest:** `IcebergBackend(catalog="rest", config=...).connect()` → context manager yielding a connection with `list_namespaces()`, `list_tables(ns)`, `inspect(table)` → `TableInfo`, `to_relation(table)` → pyiceberg-backed Relation **if** `mountainash-expressions` has an iceberg adapter; otherwise this method is unimplemented for now and the gap is documented. The protocol holds either way — `to_relation` is a capability that not every backend must implement. - -## Audit methodology - -Every `.py` file under `src/mountainash_data/` was classified into: - -| Classification | Meaning | -|---|---| -| `keep-as-is` | fits the new layout unchanged | -| `adapt` | concept survives, needs rewriting against the new protocol | -| `salvage` | concrete logic worth extracting; surrounding class/file structure discarded | -| `delete` | dead code, stub, boilerplate, or fully superseded | -| `move-to-expressions` | logic that is logical/relational and belongs in `mountainash-expressions` | -| `ambiguous` | flagged for explicit user decision | - -Out of scope for the audit: `tests/` (rewritten alongside their targets in migration), `notebooks/`, `docs/`, `pyproject.toml`, `hatch.toml`. - -## Audit findings - -**67 files audited, ~9.5k LOC total.** Highlights: - -1. **`base_pyiceberg_connection.py` (884 lines) is not really a connection class.** It mixes catalog lifecycle with table operations (create, insert, upsert, truncate, view ops) and ~15 Iceberg→PyArrow type conversions. It is effectively the entire iceberg backend, with operations smuggled inside what is named a connection class. The previous attempt to split connection from operations was only half done and left duplicate methods between this file and `base_pyiceberg_operations.py` (868 lines). - -2. **`base_ibis_operations.py` (676 lines) is not abstract.** It contains the full concrete implementations (run_sql, run_expr, table, create_table, drop_table, insert, upsert, truncate, view ops) with try/catch wrappers. The 11 per-backend "subclasses" are mostly stubs (12–19 LOC, just property overrides). - -3. **8 of 11 per-backend ibis ops files are pure stubs** → straight delete: postgres, mysql, oracle, bigquery, snowflake, pyspark, redshift, mssql. Only `duckdb`, `sqlite`, `motherduck` have real logic (delegating to `_duckdb_family_mixin.py`); `trino` has a 34-line init whose necessity is unclear (see D3). - -4. **`db_connection_factory.py` (213 lines) is a legacy duplicate** of the modern `factories/connection_factory.py`. Safe delete. - -5. **`database_utils.py` (224 lines) is a useful high-level facade** (`create_connection`, `create_from_url`, `detect_backend_from_url`), not a junk drawer. Worth keeping as the public entry point. Becomes `core/utils.py`. - -6. **The settings subtree (~2.7k LOC across 16 files) is exactly what it claims** — Pydantic settings classes per backend. Confirmed `keep-as-is`, relocated to `core/settings/`. - -7. **The factories layer is well-built** (`BaseStrategyFactory` + lazy-loading mixin + 3 concrete factories, ~770 LOC). Not overengineered. The URL→backend detection in `settings_factory.py` is a real feature worth preserving. - -8. **No file in the package was classified `move-to-expressions`.** The audit confirms data and expressions are genuinely complementary — there is no logical/relational logic hiding in `mountainash-data` that needs to migrate out. - -### Tally - -| Classification | Files | -|---|---| -| `keep-as-is` | 19 | -| `adapt` | 9 | -| `salvage` | 29 | -| `delete` | 10 | -| `move-to-expressions` | 0 | - -## Decisions - -### D1 — pyiceberg connection/ops split - -**Decision: D1.b — keep the split, deduplicate.** The previous split was only half done; duplicate methods exist between `base_pyiceberg_connection.py` and `base_pyiceberg_operations.py`. Migration finishes the split: catalog/namespace lifecycle stays in `backends/iceberg/connection.py`, table-mutation methods go to `backends/iceberg/operations.py`, type-conversion helpers go to `backends/iceberg/_types.py`. During Phase 3, every duplicate method must be identified and a canonical version chosen. - -### D2 — Factories vs. dialect registry - -**Decision: D2.a — keep factories, point them at the registry.** The `factories/` layer moves to `core/factories/` largely unchanged. `connection_factory.py` and `operations_factory.py` strategy mappings are updated to point at `IbisBackend` (with dialect arg) and `IcebergBackend` instead of the old per-backend classes. `settings_factory.py`'s URL detection logic stays as-is — it is the real feature worth keeping. The dialect registry becomes the *data* the factories iterate over, not a replacement for them. - -### D3 — Trino's HYBRID connection mode - -**Decision: D3.b — defer.** `trino_ibis_operations.py`'s 34-line init sets `_ibis_connection_mode = HYBRID`. Whether HYBRID is trino-specific or a general capability cannot be determined from one file. During Phase 4 migration, grep `_ibis_connection_mode` across the codebase and place the flag accordingly: as a general capability in the registry, or as a trino-specific dialect entry field. - -## Migration sequence - -Phases are sized so each ends with a working package and tests passing. No phase leaves the tree in a half-rewritten state for more than a single PR. - -### Phase 0 — Cleanup (no architectural change) - -- Delete `lineage/openlineage_helper.py` (3-line stub) and the empty `lineage/` dir -- Delete `databases/connections/pyiceberg/__init___old.py` -- Delete `databases/connections/db_connection_factory.py` (legacy duplicate) -- Delete the 8 stub ibis ops files: postgres, mysql, oracle, bigquery, snowflake, pyspark, redshift, mssql -- Update `databases/operations/ibis/__init__.py` to drop the deleted re-exports -- Run tests - -### Phase 1 — Stand up `core/` - -- Create `core/` with `protocol.py`, `inspection.py`, `registry.py` (placeholder, wired in Phase 4) -- Move `databases/constants.py` → `core/constants.py` -- Move `databases/connections/base_db_connection.py` → `core/connection.py` (the new `Backend` Protocol references it; exact form determined when writing) -- Add re-export shims at the old paths -- Run tests - -### Phase 2 — Move settings to `core/settings/` - -- Move all 16 files from `databases/settings/` → `core/settings/` verbatim -- Re-export shims at old paths -- Run tests - -### Phase 3 — Iceberg backend (D1.b: split, deduplicate) - -- Create `backends/iceberg/` -- Move `base_pyiceberg_connection.py` (884) and `base_pyiceberg_operations.py` (868) into the new dir as a starting point -- **Deduplicate**: identify methods present in both files (the half-done split) and pick the canonical version. Connection-lifecycle and catalog/namespace methods → `backends/iceberg/connection.py`. Table-mutation methods → `backends/iceberg/operations.py`. Type-conversion helpers → `backends/iceberg/_types.py`. -- Move REST-specific files into `backends/iceberg/catalogs/rest.py` (merging connection + operations REST files) -- Implement the `core.Backend` Protocol on `IcebergBackend` -- Implement `inspect()` returning shared inspection model dataclasses -- `to_relation()` left unimplemented (or omitted) — flagged as a gap to fix once `mountainash-expressions` has an iceberg adapter -- Update factory mappings -- Re-export shims at old paths -- Run tests - -### Phase 4 — Ibis backend - -- Create `backends/ibis/` -- Create `backends/ibis/dialects/_registry.py` with one `DialectSpec` entry per backend, populated by reading the 13 existing connection files and extracting: connection-builder callable, ibis backend name, connection mode, per-dialect quirks -- Move `base_ibis_connection.py` → `backends/ibis/connection.py`, refactored to consume `DialectSpec` instead of being subclassed -- Move `base_ibis_operations.py` (676 lines, the actual implementation) → `backends/ibis/operations.py` -- Fold `_base_ibis_mixin.py` (98) → into `backends/ibis/operations.py` as helper functions (no longer a mixin) -- Fold `_duckdb_family_mixin.py` (314) → into `backends/ibis/operations.py` as functions, attached to duckdb/motherduck/sqlite dialect entries via `DialectSpec` capability hooks -- Salvage concrete duckdb/sqlite/motherduck ops files: extract dialect-specific SQL into corresponding `DialectSpec` entries, delete the source files -- **Trino ops file (D3.b):** grep for `_ibis_connection_mode`, decide whether HYBRID is general or trino-specific, place the flag accordingly, delete the file -- Implement `core.Backend` Protocol on `IbisBackend(dialect: str)` -- Implement `inspect()` returning shared inspection model dataclasses -- Implement `to_relation(table_name)` handing back a `mountainash-expressions` Relation -- Delete the now-empty `databases/connections/ibis/` and `databases/operations/ibis/` directories -- Re-export shims at old paths (removed in Phase 6) -- Run tests - -### Phase 5 — Wire factories to the registry (D2.a) - -- Move `factories/` → `core/factories/` verbatim -- Update `connection_factory.py` and `operations_factory.py` strategy mappings to point at `IbisBackend` (with dialect arg) and `IcebergBackend` -- `settings_factory.py`'s URL detection logic stays as-is -- Move `database_utils.py` → `core/utils.py`, update to consume the new factories -- Run tests - -### Phase 6 — Remove shims and finalize - -- Delete all re-export shims at `databases/...` paths -- Delete the now-empty `databases/` directory -- Update top-level `src/mountainash_data/__init__.py` to export the new public surface: `Backend` protocol, `IbisBackend`, `IcebergBackend`, inspection model dataclasses, factories, `database_utils` facade, settings classes -- Update `notebooks/` and `docs/` examples to use the new imports -- Update `README.md` and `CLAUDE.md` to reflect the new architecture -- Run tests, run ruff, run mypy - -## Test strategy - -Each phase ends with `hatch run test:test` green. Phases 3 and 4 are the risky ones. The audit did not inspect `tests/`; the writing-plans phase will need a pass over the test tree to identify coverage gaps in iceberg and ibis operations *before* those phases begin. If coverage is thin, add tests first, refactor second. - -## Known gaps and follow-ups - -- **`IcebergBackend.to_relation()`** is unimplemented at the end of migration. Resolving it requires `mountainash-expressions` to gain an iceberg relation adapter. Tracked separately. -- **HYBRID connection mode (D3)** is resolved during Phase 4 by grep, not pre-decided here. -- **Test coverage audit** for `tests/iceberg/` and `tests/ibis/operations/` happens in writing-plans, not here. diff --git a/docs/superpowers/specs/2026-04-15-settings-audit/README.md b/docs/superpowers/specs/2026-04-15-settings-audit/README.md deleted file mode 100644 index af390fb..0000000 --- a/docs/superpowers/specs/2026-04-15-settings-audit/README.md +++ /dev/null @@ -1,102 +0,0 @@ -# Settings Audit — 2026-04-15 - -Audit of backend settings classes in `src/mountainash_data/core/settings/` against their authoritative source specs. - -## Goal - -For each backend settings class, compare the object against its source spec(s) to surface: - -- **Completeness** — parameters in the source spec missing from our class -- **Correctness** — field names, types, defaults, validation that don't match the spec -- **Currency** — spec URLs that are stale, redirected, or 404 - -Deliverable is report-only. Fixes are handled in separate per-backend writing-plans cycles. - -## Scope - -11 backend settings classes: - -`sqlite`, `duckdb`, `motherduck`, `postgresql`, `mysql`, `mssql`, `snowflake`, `bigquery`, `redshift`, `pyspark`, `trino`, `pyiceberg_rest`. - -Out of scope: `base.py`, `templates.py`, `exceptions.py`, and any code changes to settings classes. - -## Source precedence - -When specs disagree, resolve in this order: - -1. **Driver/client spec** — authoritative for parameter names, types, defaults, validation semantics (e.g. libpq, mysqlclient, snowflake-connector-python, PyIceberg REST catalog). -2. **Ibis backend** — authoritative for what can actually be passed through from our settings to the underlying driver. Ground truth: `do_connect()` signature in `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends//__init__.py`. - - `motherduck` → ibis `duckdb` backend - - `redshift` → ibis `postgres` backend - - `pyiceberg_rest` → no Ibis backend; passthrough column records PyIceberg REST catalog kwargs instead -3. **Vendor docs** — context for semantics only (Databricks Spark conf, gcloud auth guide, Snowflake OAuth guide, etc.). - -## Parameter tiering - -Every parameter audited is tagged `core` or `advanced`. - -- **core** — affects establishing a connection or core session behavior: auth, host/endpoint, TLS, timeouts, database/schema/catalog selection. -- **advanced** — tuning knobs, rarely-used flags, deprecated options, driver-specific esoterica. - -Tiering drives fix prioritization in downstream plans. - -## Per-backend report structure - -Each report lives at `./.md` and contains: - -1. **Header** — spec URLs with precedence labels; date checked; spec version if versioned; link to our settings class file; link to Ibis backend file used. -2. **Stale-link check** — result of fetching each URL: `OK`, `redirect → `, or `404`. -3. **Summary counts** — core missing, core mismatch, advanced missing, advanced mismatch, extra, total audited. -4. **Parameter table** with columns: - - | Spec name | Our field | Status | Tier | Type ✓ | Default ✓ | Ibis passthrough | Notes | - - - **Status**: `present` / `missing` / `mismatch` / `extra` (we have it, spec doesn't) - - **Type ✓** / **Default ✓**: ✓ / ✗ / N/A - - **Ibis passthrough**: ✓ / ✗ / unknown (or `via duckdb` / `via postgres` for motherduck/redshift; `N/A` for pyiceberg_rest) - -5. **Findings narrative** — prioritized issue list: core gaps → core mismatches → advanced gaps → advanced mismatches → stale links. -6. **Recommended follow-ups** — concrete, per-backend bullets a downstream plan can pick up directly. - -## Process per backend - -1. Read our settings class in `src/mountainash_data/core/settings/.py`. -2. Fetch each linked spec URL (WebFetch); record stale/redirects. -3. Read the corresponding Ibis `do_connect()` signature (or PyIceberg REST catalog for `pyiceberg_rest`). -4. Build the parameter union across driver spec + our class + Ibis passthrough; classify each row. -5. Write the report. - -## Index - -| Backend | Report | Core missing | Core mismatch | Advanced missing | Advanced mismatch | Extra | Stale links | -|---|---|---|---|---|---|---|---| -| sqlite | [sqlite.md](./sqlite.md) | 0 | 0 | 8 | 0 | 7 | 0 | -| duckdb | [duckdb.md](./duckdb.md) | 0 | 1 | 12 | 2 | 7 | 0 | -| motherduck | [motherduck.md](./motherduck.md) | 0 | 2 | many (via duckdb) | 1 | 6 | 0 | -| postgresql | [postgresql.md](./postgresql.md) | 0 | 5 | ~20 | ~12 | 1 | 0 | -| mysql | [mysql.md](./mysql.md) | 0 | 3 | ~8 | 2 | 0 | 0 | -| mssql | [mssql.md](./mssql.md) | 1 | 4 | ~15 | 1 | 1 | 0 | -| snowflake | [snowflake.md](./snowflake.md) | 0 | 4 | ~15 | 1 | 1 | 0 | -| bigquery | [bigquery.md](./bigquery.md) | 4 | 3 | 3 | 0 | 8 | 0 | -| redshift | [redshift.md](./redshift.md) | 1 | 5 | ~15 | 2 | 3 | 0 | -| pyspark | [pyspark.md](./pyspark.md) | 1 | 3 | out-of-scope | 3 | 8 | 0 | -| trino | [trino.md](./trino.md) | 0 | 5 | 1 | 8 | 1 | 0 | -| pyiceberg_rest | [pyiceberg_rest.md](./pyiceberg_rest.md) | 2 | 3 | 9 | 0 | 6 | 0 | - -## Cross-cutting findings - -Patterns that emerged across multiple backends: - -- **`db_provider_type` copy-paste bugs**: `postgresql.py` and `mysql.py` both return `CONST_DB_PROVIDER_TYPE.BIGQUERY` instead of their own provider. Real defects. -- **Plumbing gap**: Nearly every backend declares Fields that `get_connection_kwargs()` then silently drops. postgres, trino, snowflake, mssql, bigquery all have orphan fields. -- **SecretStr not unwrapped**: Passwords and tokens are passed as `SecretStr` objects rather than via `.get_secret_value()` in most cloud backends (snowflake, mssql, redshift, pyiceberg_rest). -- **Enums defined but unused**: postgresql (4), snowflake (1), mssql (2), pyspark (1) all have constant classes that aren't enforced as field types. -- **Base-class composition**: `HOST`/`PORT`/`USERNAME`/`PASSWORD`/`TOKEN` leak into every class via `BaseDBAuthSettings`, including file/cloud/catalog classes where they're meaningless. -- **Docstring drift**: pyspark docstring says "SQLite authentication settings"; pyiceberg_rest says "Cloudflare R2"; several classes have stale `#path:` comments pointing at `mountainash_settings/...` paths that don't match the current layout. -- **Dead validators**: redshift's `_init_provider_specific` is never called (base class hook is `_post_init`); its serverless/cluster validation never runs. - -Counts are filled in as each per-backend audit completes. - -## Status (post-refactor, 2026-04-15) - -The settings-registry refactor (see `docs/superpowers/specs/2026-04-15-settings-registry-design.md` and `docs/superpowers/plans/2026-04-15-settings-registry.md`) consumed most "core mismatch" and many "core missing" findings in the tables above. Remaining items are tracked as per-backend Phase-4 follow-up plans. diff --git a/docs/superpowers/specs/2026-04-15-settings-audit/bigquery.md b/docs/superpowers/specs/2026-04-15-settings-audit/bigquery.md deleted file mode 100644 index bb3ab16..0000000 --- a/docs/superpowers/specs/2026-04-15-settings-audit/bigquery.md +++ /dev/null @@ -1,82 +0,0 @@ -# BigQuery Settings Audit - -## Header - -- **Backend:** bigquery -- **Date checked:** 2026-04-15 -- **Our settings class:** `src/mountainash_data/core/settings/bigquery.py` (`BigQueryAuthSettings`) -- **Ibis backend file:** `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends/bigquery/__init__.py` — `do_connect(project_id=None, dataset_id="", credentials=None, application_name=None, auth_local_webserver=True, auth_external_data=False, auth_cache="default", partition_column="PARTITIONTIME", client=None, storage_client=None, location=None, generate_job_id_prefix=None)` -- **Spec URLs (precedence-tagged):** - - **Ibis backend (authoritative for kwargs):** https://ibis-project.org/backends/bigquery - - **Driver (google-cloud-bigquery Client):** https://cloud.google.com/python/docs/reference/bigquery/latest/google.cloud.bigquery.client.Client - - **Auth options:** https://cloud.google.com/sdk/docs/authorizing - -## Stale-link check - -| URL | Status | -|---|---| -| https://ibis-project.org/backends/bigquery | OK (assumed) | -| https://cloud.google.com/sdk/docs/authorizing | OK (assumed) | -| https://cloud.google.com/bigquery/external-data-sources | OK (assumed) | - -## Summary counts - -- Core missing: **4** (`auth_local_webserver`, `auth_external_data`, `auth_cache`, `credentials` is typed wrong) -- Core mismatch: **3** (`SERVICE_ACCOUNT_INFO` → `credentials`: Ibis expects `google.auth.credentials.Credentials` object, not a dict; `partition_column` default drift; `project_id` length validator is too restrictive) -- Advanced missing: **3** (`client`, `storage_client`, `generate_job_id_prefix`; plus `client_info`, `default_query_job_config` from the underlying `bq.Client`) -- Advanced mismatch: **0** -- Extra: **8** (base-class HOST, PORT, USERNAME, PASSWORD, TOKEN, SCHEMA, DATABASE, AUTH_METHOD — all meaningless for BigQuery's OAuth/SA-key model) -- Total audited: **~15** -- Stale links: **0** - -## Parameter table - -Legend — Status: `present` / `missing` / `mismatch` / `extra`. Tier: `core` / `advanced`. Type/Default: ✓ / ✗ / N/A. Ibis passthrough: ✓ / ✗ / unknown. - -| Spec name | Our field | Status | Tier | Type ✓ | Default ✓ | Ibis passthrough | Notes | -|---|---|---|---|---|---|---|---| -| `project_id` (str) | `PROJECT_ID` (str, required) | present | core | ✓ | ✗ | ✓ | Ibis default is None; ours required. Validator enforces 6-30 chars — GCP project IDs are 6-30 chars, so OK, but a hyphen-terminated ID could slip past. | -| `dataset_id` (str, `""`) | `DATASET_ID` (Optional[str]) | present | core | ✓ | ✗ | ✓ | Ibis default `""`; ours None. Minor. **Not plumbed into `get_connection_kwargs()`** — only into the connection-string template. | -| `credentials` (google.auth.credentials.Credentials) | `SERVICE_ACCOUNT_INFO` (Optional[Dict]) | mismatch | core | ✗ | ✓ | ✓ | **Type wrong.** Ibis expects a `Credentials` instance; we pass a dict. The dict would need to be converted via `google.oauth2.service_account.Credentials.from_service_account_info(...)` before being passed. Currently `args["credentials"] = self.SERVICE_ACCOUNT_INFO` passes the raw dict — Ibis will fail or silently use ADC. | -| `application_name` (str) | `APPLICATION_NAME` (Optional[str]) | present | advanced | ✓ | ✓ | ✓ | Plumbed. | -| `auth_local_webserver` (bool, True) | — | missing | core | N/A | N/A | ✓ | Controls interactive auth flow. | -| `auth_external_data` (bool, False) | — | missing | core | N/A | N/A | ✓ | Needed for Sheets/Drive/GCS external tables. | -| `auth_cache` (`default`/`reauth`/`none`) | — | missing | core | N/A | N/A | ✓ | Interactive auth cache control. | -| `partition_column` (str, `"PARTITIONTIME"`) | `PARTITION_COLUMN` (Optional[str]) | present | advanced | ✓ | ✗ | ✓ | Ibis default `"PARTITIONTIME"`; ours None (loses the default). Plumbed when set. | -| `client` (bq.Client) | — | missing | advanced | N/A | N/A | ✓ | Caller-constructed client injection. | -| `storage_client` (bqstorage.BigQueryReadClient) | — | missing | advanced | N/A | N/A | ✓ | Used for Storage API fast reads. | -| `location` (str) | `LOCATION` (Optional[str]) | present | advanced | ✓ | ✓ | ✓ | Plumbed. | -| `generate_job_id_prefix` (Callable) | — | missing | advanced | N/A | N/A | ✓ | | -| (not in Ibis; caller would set via bq.Client) | `MAXIMUM_BYTES_BILLED` | — (commented out) | missing | advanced | N/A | N/A | ✗ | Would need `default_query_job_config` wrapper. | -| (not in Ibis) | `API_ENDPOINT` | — (commented out) | missing | advanced | N/A | N/A | ✗ | For private/regional endpoints. | -| — | `HOST`, `PORT`, `USERNAME`, `PASSWORD`, `TOKEN`, `SCHEMA`, `DATABASE`, `AUTH_METHOD` (base) | extra | N/A | N/A | N/A | ✗ | All irrelevant to BigQuery. | - -## Findings narrative - -**Core gaps:** Three auth flow controls (`auth_local_webserver`, `auth_external_data`, `auth_cache`) and the missing `credentials` wiring. The `auth_external_data=True` case is specifically noted in our docstring (`External data sources: https://cloud.google.com/bigquery/external-data-sources`) yet isn't exposed. `auth_cache="none"` is important for CI environments. - -**Core mismatches:** - -1. **`SERVICE_ACCOUNT_INFO` is a dict, but Ibis `credentials` expects `google.auth.credentials.Credentials`.** The current code passes the raw dict as `credentials=`, which either raises inside Ibis or is silently ignored (falling back to Application Default Credentials). The settings layer needs a validator/adapter that converts SA info → `service_account.Credentials`. Alternative: take a file path (`SERVICE_ACCOUNT_FILE`, already commented out) and use `from_service_account_file()`. -2. **`partition_column` default drift.** Ours defaults to `None` (no emission); Ibis defaults to `"PARTITIONTIME"`. Users relying on the default partition column are fine (Ibis supplies it), but users who *want* `None` (i.e., no partition filter) can't express that distinctly from "use default". -3. **`validate_project_id` 6-30 char bound** matches GCP's current rules but doesn't validate characters (lowercase, digits, hyphens; not starting or ending with a hyphen). A trailing hyphen passes. - -**Advanced gaps:** `client` and `storage_client` injection points are missing — useful for tests and advanced users. `generate_job_id_prefix` and config like `MAXIMUM_BYTES_BILLED` / `default_query_job_config` aren't supported. - -**Advanced mismatches:** None. - -**Extras:** Standard base-class composition issue — eight inherited fields irrelevant to BigQuery. - -**Stale links:** None. - -## Recommended follow-ups - -- **Fix credentials plumbing.** Either (a) add `SERVICE_ACCOUNT_FILE: Optional[str]` and convert to `Credentials` in `get_connection_kwargs()`, or (b) convert `SERVICE_ACCOUNT_INFO` dict → `Credentials` via `from_service_account_info()` before emitting. Currently the credentials path is broken. -- **Add `AUTH_LOCAL_WEBSERVER`, `AUTH_EXTERNAL_DATA`, `AUTH_CACHE`** Fields. These are documented Ibis kwargs for interactive OAuth flows. -- **Set `PARTITION_COLUMN` default** to `"PARTITIONTIME"` to match Ibis behavior, or document the None-means-no-override intent. -- **Tighten `validate_project_id`** with a regex matching `^[a-z][a-z0-9-]{4,28}[a-z0-9]$`. -- **Consider adding `CLIENT` / `STORAGE_CLIENT` escape hatches** for advanced users (typed `Any`). -- **Plumb `DATASET_ID` into `get_connection_kwargs()`** or document that it's intended to flow only via the connection string. -- **Restore `MAXIMUM_BYTES_BILLED` / `DEFAULT_QUERY_JOB_CONFIG`** and document that they're applied post-connect by wrapping `conn.client.default_query_job_config`. -- **Drop or validate base-class fields** — same composition issue as other cloud backends. -- **Check `db_provider_type`** — correctly returns `BIGQUERY` here (cf. postgresql.py and mysql.py where it's wrong). diff --git a/docs/superpowers/specs/2026-04-15-settings-audit/duckdb.md b/docs/superpowers/specs/2026-04-15-settings-audit/duckdb.md deleted file mode 100644 index f05dd95..0000000 --- a/docs/superpowers/specs/2026-04-15-settings-audit/duckdb.md +++ /dev/null @@ -1,96 +0,0 @@ -# DuckDB Settings Audit - -## Header - -- **Backend:** duckdb -- **Date checked:** 2026-04-15 -- **Our settings class:** `src/mountainash_data/core/settings/duckdb.py` (`DuckDBAuthSettings`, inherits `BaseDBAuthSettings`) -- **Ibis backend file:** `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends/duckdb/__init__.py` — `do_connect(database=":memory:", read_only=False, extensions=None, **config)` -- **Spec URLs (precedence-tagged):** - - **Driver (authoritative):** https://duckdb.org/docs/current/configuration/overview.html (redirect from `/docs/configuration/overview.html`) - - **Ibis backend (context):** https://ibis-project.org/backends/duckdb - - **Vendor (context):** https://duckdb.org/docs/extensions/spatial.html - -## Stale-link check - -| URL | Status | -|---|---| -| https://duckdb.org/docs/configuration/overview.html | redirect → https://duckdb.org/docs/current/configuration/overview.html | -| https://duckdb.org/docs/current/configuration/overview.html | OK | -| https://ibis-project.org/backends/duckdb | OK (assumed — Ibis docs site stable) | -| https://duckdb.org/docs/extensions/spatial.html | OK (assumed — DuckDB docs root stable) | - -## Summary counts - -- Core missing: **0** -- Core mismatch: **1** (`read_only` default drift) -- Advanced missing: **~12** DuckDB config options unexposed (acceptable via `**config` — see narrative) -- Advanced mismatch: **2** (`MEMORY_LIMIT` regex too narrow; `ATTACH_PATH` not plumbed) -- Extra: **7** (base-class fields irrelevant to embedded DuckDB) -- Total audited: **~22** -- Stale links: **0** (one redirect, not stale) - -## Parameter table - -Legend — Status: `present` / `missing` / `mismatch` / `extra`. Tier: `core` / `advanced`. Type/Default: ✓ / ✗ / N/A. Ibis passthrough: ✓ / ✗ / unknown. - -| Spec name | Our field | Status | Tier | Type ✓ | Default ✓ | Ibis passthrough | Notes | -|---|---|---|---|---|---|---|---| -| `database` (str \| Path, `":memory:"`) | `DATABASE` (Optional[str]) | present | core | ✓ | ✓ | ✓ | `get_connection_string_params()` falls back to `":memory:"` — matches Ibis default. | -| `read_only` (bool, `False`) | `READ_ONLY` (bool, `True`) | mismatch | core | ✓ | ✗ | ✓ | **Default drift**: ours defaults to `True`, Ibis/DuckDB default to `False`. Deliberate opinion or bug? Silent divergence is dangerous — users who expect to write will silently fail with a read-only error. | -| `extensions` (Sequence[str] \| None) | `EXTENSIONS` (List[str], default `[]`) | present | core | ✓ | ✓ | ✓ | Passed via `config["extensions"]` in our `get_connection_kwargs()` — **mismatch**: Ibis accepts `extensions` as a top-level kwarg, not inside `config`. See narrative. | -| `config.threads` (BIGINT, #cores) | `THREADS` (Optional[int]) | present | advanced | ✓ | ✓ | ✓ (via `**config`) | Routed through `config["threads"]`. | -| `config.memory_limit` (VARCHAR, "80% RAM") | `MEMORY_LIMIT` (Optional[str]) | present | advanced | ✓ | ✓ | ✓ (via `**config`) | Regex `^\d+[KMG]B$` rejects legal DuckDB values like `"500MB"` (fine), `"1.5GB"` (rejected — no decimals), `"1024KiB"` (rejected), and the string `"80%"` (percentage form). | -| `config.access_mode` (VARCHAR, "automatic") | — | missing | advanced | N/A | N/A | ✓ (via `**config`) | Overlaps with `read_only`; exposing both invites conflicts. | -| `config.temp_directory` (VARCHAR) | — (commented out in source) | missing | advanced | N/A | N/A | ✓ (via `**config`) | Explicitly commented out. OK to leave — Ibis/DuckDB default is sensible. | -| `config.max_memory` (VARCHAR) | — | missing | advanced | N/A | N/A | ✓ (via `**config`) | Alias of `memory_limit`. | -| `config.external_threads` (UBIGINT, 1) | — | missing | advanced | N/A | N/A | ✓ (via `**config`) | | -| `config.allow_unsigned_extensions` (BOOLEAN, false) | — (commented out) | missing | advanced | N/A | N/A | ✓ (via `**config`) | Security-relevant flag. Consider restoring the field. | -| `config.autoload_known_extensions` (BOOLEAN, true) | — | missing | advanced | N/A | N/A | ✓ (via `**config`) | | -| `config.autoinstall_known_extensions` (BOOLEAN, true) | — | missing | advanced | N/A | N/A | ✓ (via `**config`) | | -| `config.preserve_insertion_order` (BOOLEAN, true) | — | missing | advanced | N/A | N/A | ✓ (via `**config`) | | -| `config.enable_external_access` (BOOLEAN, true) | — | missing | advanced | N/A | N/A | ✓ (via `**config`) | Security-relevant. | -| `config.max_temp_directory_size` (VARCHAR) | — | missing | advanced | N/A | N/A | ✓ (via `**config`) | | -| `config.default_order` (VARCHAR, "ASCENDING") | — | missing | advanced | N/A | N/A | ✓ (via `**config`) | | -| `config.default_null_order` (VARCHAR, "NULLS_LAST") | — | missing | advanced | N/A | N/A | ✓ (via `**config`) | | -| `config.enable_progress_bar` (BOOLEAN, true) | — | missing | advanced | N/A | N/A | ✓ (via `**config`) | Annoying in CI; consider exposing. | -| `config.default_collation` (VARCHAR) | — | missing | advanced | N/A | N/A | ✓ (via `**config`) | | -| — (no spec counterpart) | `ATTACH_PATH` (Optional[str \| List[str]]) | extra/mismatch | advanced | ✗ | N/A | ✗ | **Not plumbed** — declared as a Field but never consumed by `get_connection_kwargs()` or `get_post_connection_options()`. Dead field, or incomplete implementation? | -| — | `HOST` (base) | extra | N/A | N/A | N/A | ✗ | Embedded DuckDB has no host. | -| — | `PORT` (base) | extra | N/A | N/A | N/A | ✗ | No port. | -| — | `SCHEMA` (base) | extra | N/A | N/A | N/A | ✗ | DuckDB uses `catalog.database.schema`; SCHEMA not plumbed. | -| — | `USERNAME` (base) | extra | N/A | N/A | N/A | ✗ | No auth. | -| — | `PASSWORD` (base) | extra | N/A | N/A | N/A | ✗ | No auth. | -| — | `TOKEN` (base) | extra | N/A | N/A | N/A | ✗ | No auth (use `MotherDuckAuthSettings` for tokens). | -| — | `AUTH_METHOD` (base, overridden `"none"`) | extra | N/A | N/A | N/A | ✗ | Correctly set to `"none"`. | - -## Findings narrative - -**Core gaps:** None. - -**Core mismatches:** - -1. **`READ_ONLY` default is `True`** (ours) vs `False` (Ibis/DuckDB). This is the single most important finding. Either intentional policy (lock down by default) — in which case it should be documented as such — or a copy-paste bug. Users constructing settings and expecting to write will get silent failures. - -2. **`EXTENSIONS` routing**: we pack `extensions` inside `config["extensions"]`, but Ibis `do_connect(extensions=...)` accepts it as a top-level kwarg and passes `config` separately to `duckdb.connect()`. DuckDB itself doesn't document `extensions` as a config key — Ibis handles extension install/load in `_post_connect`. Current wiring likely causes extensions to be silently ignored (DuckDB ignores unknown config keys) or to error. **Needs verification with a live test** before classifying as a bug. - -**Advanced gaps:** The `**config` splat in Ibis means any DuckDB config option flows through — so "missing" here means "not exposed as a typed Field", not "unreachable". Twelve commonly-tuned options (access_mode, temp_directory, max_memory, allow_unsigned_extensions, autoload/autoinstall_known_extensions, preserve_insertion_order, enable_external_access, max_temp_directory_size, default_order, default_null_order, enable_progress_bar, default_collation) are unexposed. Priority candidates to add: `allow_unsigned_extensions` (security), `enable_external_access` (security), `temp_directory` (ops), `enable_progress_bar` (CI ergonomics). - -**Advanced mismatches:** - -1. **`MEMORY_LIMIT` regex** `^\d+[KMG]B$` rejects legal values: decimal quantities (`"1.5GB"`), KiB/MiB/GiB forms, and percentage forms (`"80%"`). DuckDB accepts these; users would hit pydantic validation errors on valid input. -2. **`ATTACH_PATH`** is declared but orphaned — never consumed. Either plumb into a post-connect `ATTACH DATABASE` routine (which `get_post_connection_options()` is set up for but returns `None`), or remove the field. - -**Extras:** Seven base-class fields irrelevant to embedded DuckDB. Same composition issue as SQLite. - -**Stale links:** None. `/docs/configuration/overview.html` redirects to `/docs/current/configuration/overview.html`; the docstring URL should be updated to the canonical one. - -## Recommended follow-ups - -- **Investigate `READ_ONLY=True` default.** Decide: keep as opinionated default (document why), or change to `False` to match Ibis/DuckDB. Either way, make the choice explicit in a docstring. -- **Verify extension loading.** Write a test that passes `EXTENSIONS=["httpfs"]` and confirms the extension is actually loaded. If broken, fix by moving `extensions` out of `config` and passing it as a top-level kwarg to Ibis `do_connect()`. -- **Plumb or remove `ATTACH_PATH`.** If keeping, implement `get_post_connection_options()` to return `ATTACH DATABASE` SQL statements. -- **Relax `MEMORY_LIMIT` regex** to `^\d+(\.\d+)?\s*[KMG]i?B$|^\d+%$` (or similar) — test against the DuckDB config docs examples. -- **Expose security-relevant config** as typed Fields: `ALLOW_UNSIGNED_EXTENSIONS`, `ENABLE_EXTERNAL_ACCESS`. -- **Update docstring URL** from `duckdb.org/docs/configuration/overview.html` to `duckdb.org/docs/current/configuration/overview.html`. -- Consider a validator that warns if HOST/PORT/USERNAME/PASSWORD/TOKEN are set on embedded DuckDB. diff --git a/docs/superpowers/specs/2026-04-15-settings-audit/motherduck.md b/docs/superpowers/specs/2026-04-15-settings-audit/motherduck.md deleted file mode 100644 index 7e97cfb..0000000 --- a/docs/superpowers/specs/2026-04-15-settings-audit/motherduck.md +++ /dev/null @@ -1,72 +0,0 @@ -# MotherDuck Settings Audit - -## Header - -- **Backend:** motherduck -- **Date checked:** 2026-04-15 -- **Our settings class:** `src/mountainash_data/core/settings/motherduck.py` (`MotherDuckAuthSettings`, inherits `BaseDBAuthSettings`) -- **Ibis backend file:** `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends/duckdb/__init__.py` (`do_connect(database=":memory:", read_only=False, extensions=None, **config)`) — MotherDuck rides the **duckdb** Ibis backend; connection string form is `md:?motherduck_token=`. -- **Spec URLs (precedence-tagged):** - - **Driver (authoritative):** DuckDB Python API + MotherDuck extension — https://motherduck.com/docs/getting-started/connect-query-from-python/installation-authentication/ - - **Ibis passthrough:** Ibis duckdb backend (see file above) - - **Vendor (context):** https://motherduck.com/docs/authenticating-to-motherduck/ - -## Stale-link check - -| URL | Status | -|---|---| -| https://motherduck.com/docs/getting-started/connect-query-from-python/installation-authentication/ | OK (assumed — vendor doc root stable) | -| https://motherduck.com/docs/authenticating-to-motherduck/ | OK (assumed) | - -## Summary counts - -- Core missing: **0** (all essentials reachable via connection string) -- Core mismatch: **2** (`DATABASE` validator nullability-inconsistent; `AUTH_METHOD=TOKEN` enforced but docstring says "file-based auth") -- Advanced missing: **All duckdb `**config` options** (inherited passthrough surface not exposed; see duckdb.md — same gap) -- Advanced mismatch: **1** (`ATTACH_PATH` declared but never consumed) -- Extra: **6** (HOST, PORT, SCHEMA, USERNAME, PASSWORD — base-class fields unused by MotherDuck; plus irrelevant base bits) -- Total audited: **~10** -- Stale links: **0** - -## Parameter table - -Legend — Status: `present` / `missing` / `mismatch` / `extra`. Tier: `core` / `advanced`. Type/Default: ✓ / ✗ / N/A. Ibis passthrough: ✓ / ✗ / unknown. - -| Spec name | Our field | Status | Tier | Type ✓ | Default ✓ | Ibis passthrough | Notes | -|---|---|---|---|---|---|---|---| -| `database` (str, `"md:"` or `"md:"`) | `DATABASE` (Optional[str]) | present | core | ✓ | ✗ | ✓ (via duckdb) | Required in practice — validator enforces non-None. Users pass bare db name; connection-string template prefixes `md:`. | -| `motherduck_token` (query param on connection string) | `TOKEN` (base, SecretStr) | present | core | ✓ | ✓ | ✓ (via duckdb) | Plumbed via connection-string template `?motherduck_token={token}`. Note: `get_connection_string_params()` passes the SecretStr object itself — depends on upstream stringification. | -| `read_only` (bool) | — (inherited/missing) | missing | core | N/A | N/A | ✓ (via duckdb) | MotherDuck supports read-only attachments; no field exposed. | -| `extensions` (Sequence[str]) | — | missing | advanced | N/A | N/A | ✓ (via duckdb) | Same gap as duckdb report. | -| `config.*` (duckdb config dict) | — | missing | advanced | N/A | N/A | ✓ (via duckdb) | Threads, memory_limit, etc. not exposed here though DuckDB accepts them. | -| (no direct spec equivalent — SQL statement) | `ATTACH_PATH` (Optional[str \| List[str]]) | extra/mismatch | advanced | ✗ | N/A | ✗ | Declared but `get_connection_kwargs()` returns `{}` and `get_post_connection_options()` is `...` (returns None). Dead field. Intended use is presumably post-connect `ATTACH 'md:other_db'`. | -| — | `AUTH_METHOD` (overridden default `TOKEN`) | extra | core | N/A | N/A | ✗ | Local selector. Docstring says "file-based authentication" but the code forces TOKEN — contradictory. | -| — | `HOST`, `PORT`, `SCHEMA`, `USERNAME`, `PASSWORD` (base) | extra | N/A | N/A | N/A | ✗ | MotherDuck has no host/port/user/pass auth. | - -## Findings narrative - -**Core gaps:** None fatal — `DATABASE` + `TOKEN` cover the minimum needed to reach a MotherDuck instance. `read_only` is unreachable (MotherDuck supports it via DuckDB); add if needed. - -**Core mismatches:** - -1. **Docstring/AUTH_METHOD contradiction.** The class docstring reads "DuckDB authentication settings" and the inline comment on `AUTH_METHOD` says "DuckDB uses file-based authentication" — but the default is `CONST_DB_AUTH_METHOD.TOKEN` and the model validator enforces `TOKEN is not None` when `AUTH_METHOD == TOKEN`. MotherDuck authentication is genuinely token-based (a MotherDuck JWT); the comment is simply wrong (it was copy-pasted from a DuckDB context). Fix: drop the "file-based" comment, update the class docstring to say "MotherDuck authentication settings (token-based)". -2. **`DATABASE` validator logic.** The validator uses `precondition = True` unconditionally and rejects `None`, yet the field is typed `Optional[str]`. Either the field should be required (`Field(...)`) or the validator should permit `None` (MotherDuck does accept a bare `md:` with no database). The current shape is inconsistent — pydantic will still accept `None` at construction until the validator fires. - -**Advanced gaps:** The entire DuckDB `**config` passthrough surface (threads, memory_limit, access_mode, external_threads, etc.) is absent here just as in the duckdb audit. Since MotherDuck connections are DuckDB connections under the hood, the same tuning options apply. Decide whether MotherDuckAuthSettings should inherit/compose DuckDB's tunables rather than sit alongside them. Also absent: `read_only`, `extensions`. - -**Advanced mismatches:** `ATTACH_PATH` is declared but neither `get_connection_kwargs()` (returns `{}`) nor `get_post_connection_options()` (body is `...` → returns None) consume it. Either wire it into a post-connect `ATTACH 'md:'` routine, or remove. - -**Extras:** Five inherited base-class fields (HOST, PORT, SCHEMA, USERNAME, PASSWORD) are irrelevant to MotherDuck. `AUTH_METHOD` is internally consistent with token auth but commented misleadingly. - -**Stale links:** None confirmed stale. Note the class has **no docstring URLs** at all — unlike other settings classes in this tree, there is no `# Source: ...` pointer. Add one to the MotherDuck Python installation/authentication page. - -## Recommended follow-ups - -- **Add source URLs to the docstring** — this class has none; at minimum reference the MotherDuck installation-authentication page. -- **Fix the "file-based authentication" comment** — MotherDuck is token-based. Replace the stale copy. -- **Resolve `DATABASE` nullability**: either make it required via `Field(...)`, or widen the validator to accept `None` (legal for MotherDuck "no default db" connections). -- **Plumb or remove `ATTACH_PATH`.** If keeping, implement `get_post_connection_options()` to return ATTACH SQL. -- **Decide composition with DuckDB tunables.** MotherDuck accepts the same DuckDB config dict; either share the field set or document that tuning must happen post-connect via SQL `SET` statements. -- **Consider adding `READ_ONLY`** to match the DuckDB class. -- **Drop or validate inherited base-class fields** (HOST/PORT/SCHEMA/USERNAME/PASSWORD) — same base-class composition issue as sqlite/duckdb. -- **Secret handling**: in `get_connection_string_params()`, call `.get_secret_value()` on `TOKEN` explicitly rather than relying on implicit stringification, or verify what the downstream connection-string builder does with SecretStr. diff --git a/docs/superpowers/specs/2026-04-15-settings-audit/mssql.md b/docs/superpowers/specs/2026-04-15-settings-audit/mssql.md deleted file mode 100644 index c222350..0000000 --- a/docs/superpowers/specs/2026-04-15-settings-audit/mssql.md +++ /dev/null @@ -1,100 +0,0 @@ -# MSSQL Settings Audit - -## Header - -- **Backend:** mssql -- **Date checked:** 2026-04-15 -- **Our settings class:** `src/mountainash_data/core/settings/mssql.py` (`MSSQLAuthSettings`) -- **Ibis backend file:** `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends/mssql/__init__.py` — `do_connect(host="localhost", user=None, password=None, port=1433, database=None, driver=None, **kwargs)` (kwargs → PyODBC) -- **Spec URLs (precedence-tagged):** - - **Driver (authoritative):** PyODBC connect — https://github.com/mkleehammer/pyodbc/wiki/The-pyodbc-Module#connect - - **ODBC connection string ref (authoritative for kwargs):** https://learn.microsoft.com/en-us/sql/connect/odbc/dsn-connection-string-attribute - - **Driver install (context):** https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server - -## Stale-link check - -| URL | Status | -|---|---| -| https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server | OK (assumed — MS Learn stable) | -| https://learn.microsoft.com/en-us/sql/connect/odbc/dsn-connection-string-attribute | OK (assumed) | - -## Summary counts - -- Core missing: **1** (ENCRYPTION — commented out but mandatory for ODBC Driver 18, which is the default) -- Core mismatch: **4** (`get_connection_string_params()` references `AZURE_MANAGED_IDENTITY`/`AZURE_MSI_ENDPOINT` fields that don't exist; `args["server"] += ...` on missing key — NameError; `PASSWORD` returned as SecretStr not unwrapped; `DRIVER`/`PROTOCOL` typed `str` with enum-coerce validator — OK but confusing) -- Advanced missing: **~15** (all commented out: TRUST_SERVER_CERTIFICATE, COLUMN_ENCRYPTION, KEY_STORE_*, LOGIN_TIMEOUT, CONNECTION_TIMEOUT, QUERY_TIMEOUT, POOL_*, PACKET_SIZE, AUTOCOMMIT, ANSI_NULLS, QUOTED_IDENTIFIER, ISOLATION_LEVEL, AZURE_MANAGED_IDENTITY, AZURE_MSI_ENDPOINT) -- Advanced mismatch: **1** (`MARS_ENABLED` declared but not plumbed) -- Enum coverage: **4 enums defined, 2 used (DRIVER, PROTOCOL), 2 unused (ENCRYPTION — commented out; MSSQLAuthMethod — exists but `AUTH_METHOD` uses `CONST_DB_AUTH_METHOD` instead)** -- Extra: **1** (`PROTOCOL` — PyODBC infers from driver, not a standalone kwarg) -- Total audited: **~30** -- Stale links: **0** - -## Parameter table - -Legend — Status: `present` / `missing` / `mismatch` / `extra`. Tier: `core` / `advanced`. Type/Default: ✓ / ✗ / N/A. Ibis passthrough: ✓ / ✗ / unknown. - -| Spec name | Our field | Status | Tier | Type ✓ | Default ✓ | Ibis passthrough | Notes | -|---|---|---|---|---|---|---|---| -| `host` / `server` | `HOST` (base) | present | core | ✓ | ✓ | ✓ | | -| `port` | `PORT` (override, default `1433`) | present | core | ✓ | ✓ | ✓ | Matches Ibis default. | -| `user` / `uid` | `USERNAME` (base) | present | core | ✓ | ✓ | ✓ | | -| `password` / `pwd` | `PASSWORD` (base, SecretStr) | present | core | ✓ | ✓ | ✓ | Passed as SecretStr object; PyODBC stringifies unpredictably. | -| `database` | `DATABASE` (base) | present | core | ✓ | ✓ | ✓ | | -| `driver` | `DRIVER` (str, default `MSSQLDriverType.ODBC` = ODBC Driver 18) | present | core | ✓ | ✓ | ✓ | Validator coerces to enum. ODBC 18 default changes SSL behavior (Encrypt=yes by default) — worth documenting. | -| `encrypt` (yes/no/strict) | — (ENCRYPTION commented out) | missing | core | N/A | N/A | ✓ | **Critical** for ODBC Driver 18 which defaults to `Encrypt=Yes` and fails handshake if server cert isn't trusted. Need ENCRYPTION + TRUST_SERVER_CERTIFICATE. | -| `trustservercertificate` | — (commented out) | missing | core | N/A | N/A | ✓ | Goes hand-in-hand with Encrypt. | -| `trusted_connection` | (derived from `AUTH_METHOD=="windows"`) | present | core | ✓ | ✓ | ✓ | Set to "yes" when Windows auth. | -| `authentication` (for Azure AD) | (derived) | present | core | ✓ | ✓ | ✓ | Set to `ActiveDirectoryMsi` / `ActiveDirectoryServicePrincipal`. | -| `application_name` / `app` | `APP_NAME` (str, default `"MountainAsh"`) | present | advanced | ✓ | ✓ | ✓ | **Not plumbed** (commented out in kwargs). | -| `loginTimeout` | — (commented out) | missing | advanced | N/A | N/A | ✓ | | -| `connectionTimeout` / `timeout` | — (commented out) | missing | advanced | N/A | N/A | ✓ | | -| `MARS_Connection` | `MARS_ENABLED` (bool, default `False`) | present | advanced | ✓ | ✓ | ✓ | **Not plumbed**. | -| `packet_size` | — (commented out) | missing | advanced | N/A | N/A | ✓ | | -| `isolation_level` | — (commented out) | missing | advanced | N/A | N/A | ✓ | | -| `ColumnEncryption` | — (commented out) | missing | advanced | N/A | N/A | ✓ | Enterprise Always Encrypted feature. | -| `KeyStoreAuthentication` / `KeyStorePrincipalId` / `KeyStoreSecret` | — (commented out) | missing | advanced | N/A | N/A | ✓ | | -| Instance name (e.g. `host\SQLEXPRESS`) | `INSTANCE_NAME` (Optional[str]) | present | advanced | ✓ | ✓ | ✓ | **Bug** — see below. | -| — (Azure AD fields) | `WINDOWS_DOMAIN`, `AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET` | present | advanced | ✓ | ✓ | ✓ | Used by AUTH_METHOD branches. | -| — | `AZURE_MANAGED_IDENTITY`, `AZURE_MSI_ENDPOINT` | **referenced but undefined** | core | N/A | N/A | N/A | Referenced in `get_connection_string_params()` (lines 334, 336) but **not declared as Fields**. Runtime `AttributeError` when `AUTH_METHOD == "azure_active_directory"`. | -| — | `PROTOCOL` (str, default `MSSQLAuthProtocol.TCP`) | extra | advanced | N/A | N/A | ✗ | Not a PyODBC connection kwarg — protocol is encoded in driver string / server address (e.g. `np:...`, `lpc:...`). **Not plumbed**. | - -### Enum coverage - -| Enum | Field that uses it | Intended ODBC keyword | Status | -|---|---|---|---| -| `MSSQLDriverType` | `DRIVER` | `DRIVER` | **Used** via `validate_driver` | -| `MSSQLAuthProtocol` | `PROTOCOL` | (no direct ODBC kwarg) | **Used** but field is orphan — not plumbed | -| `MSSQLAuthEncryption` | — (commented out) | `Encrypt` | **Unused** | -| `MSSQLAuthMethod` | — | (internal selector) | **Unused** — the class uses `CONST_DB_AUTH_METHOD` strings instead, and hardcodes `"windows"`/`"azure_active_directory"` literals. Two parallel enum definitions for auth method. | - -## Findings narrative - -**Core gaps:** `ENCRYPTION` and `TRUST_SERVER_CERTIFICATE` are missing. This matters because the default `DRIVER` is "ODBC Driver 18 for SQL Server", which flipped the `Encrypt` default to `Yes` (strict). Without exposing either encryption mode or trust-server-cert, users connecting to self-signed SQL Servers via Driver 18 will get handshake failures with no settings-layer knob to turn. - -**Core mismatches / bugs:** - -1. **`AZURE_MANAGED_IDENTITY` and `AZURE_MSI_ENDPOINT` are referenced but not declared.** `get_connection_string_params()` at lines 334 and 336 reads `self.AZURE_MANAGED_IDENTITY` and `self.AZURE_MSI_ENDPOINT`. Neither is a declared Field on this class. Under pydantic v2 `model_config.extra = "forbid"` this raises; otherwise accessing an unset attribute raises `AttributeError`. **Path is untested.** -2. **`args["server"] += f"\\{self.INSTANCE_NAME}"` at line 353.** The dict has no `"server"` key — only `"host"`. When `INSTANCE_NAME` is set, this raises `KeyError`. Either the key should be `"host"`, or `"server"` needs to be initialized first. Either way this code path is broken. -3. **Secret handling inconsistent.** `args["password"] = self.PASSWORD if self.PASSWORD else None` passes the SecretStr object as-is; depending on the ODBC driver's string coercion, this may pass the literal `"**********"` placeholder. Should be `.get_secret_value()`. -4. **Two auth-method enums.** `MSSQLAuthMethod` is defined but unused; the class uses string literals (`"windows"`, `"azure_active_directory"`) and `CONST_DB_AUTH_METHOD.PASSWORD`. Pick one. - -**Advanced gaps:** ~15 commented-out fields covering encryption, pool, timeouts, packet size, isolation, and Azure managed identity. Most have mostly-complete commented scaffolding in both `get_connection_string()` and `get_connection_string_params()` — restoring is straightforward. - -**Advanced mismatches:** `MARS_ENABLED` is declared but not plumbed (the `MARS_Connection=yes` emission is commented out). - -**Extras:** `PROTOCOL` doesn't correspond to any ODBC kwarg — SQL Server protocol is selected via the driver string or by prefixing the server (`np:host`, `lpc:host`). Remove or convert to a prefix-emitter. - -**Stale links:** None. - -## Recommended follow-ups - -- **Declare `AZURE_MANAGED_IDENTITY` and `AZURE_MSI_ENDPOINT` Fields** — the Azure AD code path references them but they don't exist. Blocking for any Azure MSI user. -- **Fix `args["server"] += ...` KeyError** (line 353). Change to `args["host"] += ...` or change the whole class to use `server` consistently (ODBC keyword is `Server`). -- **Add `ENCRYPTION` + `TRUST_SERVER_CERTIFICATE` Fields** and plumb to `Encrypt` / `TrustServerCertificate` ODBC keywords. Required for default Driver 18 connections to non-public-CA servers. -- **Plumb `APP_NAME` and `MARS_ENABLED`** — currently declared but dropped. -- **Restore timeouts** (`LOGIN_TIMEOUT`, `CONNECTION_TIMEOUT`, `QUERY_TIMEOUT`) — basic operational needs. -- **Unwrap SecretStr**: `.get_secret_value()` for `PASSWORD` and `AZURE_CLIENT_SECRET` at the kwargs boundary. -- **Decide on auth-method enum**: either adopt `MSSQLAuthMethod` throughout or delete it in favour of `CONST_DB_AUTH_METHOD` + string literals (which is what the code actually uses). -- **Remove or reimplement `PROTOCOL`** — not a direct ODBC kwarg. If keeping, emit via `Server=np:host,port` style prefix. -- **Implement `get_connection_kwargs()`** — currently returns `{}`. Everything flows via `get_connection_string_params()`, so `get_connection_kwargs()` is presumably dead or reserved. Clarify contract or remove the method. -- **Validator annotations**: `validate_driver`/`validate_protocol` lack `@classmethod`. Works under pydantic v2 but emits a warning. diff --git a/docs/superpowers/specs/2026-04-15-settings-audit/mysql.md b/docs/superpowers/specs/2026-04-15-settings-audit/mysql.md deleted file mode 100644 index 0a9839e..0000000 --- a/docs/superpowers/specs/2026-04-15-settings-audit/mysql.md +++ /dev/null @@ -1,97 +0,0 @@ -# MySQL Settings Audit - -## Header - -- **Backend:** mysql -- **Date checked:** 2026-04-15 -- **Our settings class:** `src/mountainash_data/core/settings/mysql.py` (`MySQLAuthSettings`) -- **Ibis backend file:** `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends/mysql/__init__.py` — `do_connect(host="localhost", user=None, password=None, port=3306, autocommit=True, **kwargs)` (kwargs → `MySQLdb.connect`) -- **Spec URLs (precedence-tagged):** - - **Driver (authoritative):** mysqlclient — https://mysqlclient.readthedocs.io/user_guide.html#functions-and-attributes - - **Driver (SSL C-API):** https://dev.mysql.com/doc/c-api/8.4/en/mysql-ssl-set.html - - **Driver (options C-API):** https://dev.mysql.com/doc/c-api/8.4/en/mysql-options.html - -## Stale-link check - -| URL | Status | -|---|---| -| https://mysqlclient.readthedocs.io/user_guide.html | OK (assumed — readthedocs stable) | -| https://dev.mysql.com/doc/c-api/8.4/en/mysql-ssl-set.html | OK (assumed) | -| https://dev.mysql.com/doc/c-api/8.4/en/mysql-options.html | OK (assumed) | - -## Summary counts - -- Core missing: **0** -- Core mismatch: **3** (`db_provider_type` returns BIGQUERY; `SSL_MODE` stringly-typed; `CONV` typed without parameters / default `None` despite non-Optional type) -- Advanced missing: **~8** (CONNECT_TIMEOUT, READ_TIMEOUT, WRITE_TIMEOUT, MAX_ALLOWED_PACKET, COMPRESSION, COMPRESSION_LEVEL, PROGRAM_NAME, CLIENT_FLAG — all commented out) -- Advanced mismatch: **2** (`SSL_CAPATH` guarded behind `SSL_CA` typo bug; `SSL_MODE != DISABLED` branch fires even when `SSL_MODE is None`) -- Extra: **0** (6 inherited base-class fields; of those, HOST/PORT/USERNAME/PASSWORD/DATABASE are all used — clean) -- Total audited: **~18** -- Stale links: **0** - -## Parameter table - -Legend — Status: `present` / `missing` / `mismatch` / `extra`. Tier: `core` / `advanced`. Type/Default: ✓ / ✗ / N/A. Ibis passthrough: ✓ / ✗ / unknown. - -| Spec name | Our field | Status | Tier | Type ✓ | Default ✓ | Ibis passthrough | Notes | -|---|---|---|---|---|---|---|---| -| `host` | `HOST` (base) | present | core | ✓ | ✓ | ✓ | Ibis default `"localhost"`; ours None. | -| `user` | `USERNAME` (base) | present | core | ✓ | ✓ | ✓ | | -| `password` | `PASSWORD` (base, SecretStr) | present | core | ✓ | ✓ | ✓ | | -| `port` | `PORT` (override, default `3306`) | present | core | ✓ | ✓ | ✓ | Matches Ibis default. | -| `db` / `database` | `DATABASE` (base) | present | core | ✓ | ✓ | ✓ | mysqlclient accepts both `db` and `database`; Ibis uses `database`. | -| `autocommit` | `AUTOCOMMIT` (bool, default `True`) | present | core | ✓ | ✓ | ✓ (direct Ibis kwarg) | Matches Ibis default. Note: in `get_connection_kwargs()` the `if self.AUTOCOMMIT:` guard means `False` is silently dropped (bug — should be `if self.AUTOCOMMIT is not None`). | -| `charset` | `CHARSET` (str, default `"utf8mb4"`) | present | advanced | ✓ | ✓ | ✓ (via **kwargs) | Validator enforces a small allowlist — reasonable. | -| `use_unicode` (bool) | — | missing | advanced | N/A | N/A | ✓ | Defaults True in mysqlclient. | -| `collation` (server-side option) | `COLLATION` (str, default `"utf8mb4_unicode_ci"`) | present | advanced | ✓ | ✓ | ✓ (via **kwargs) | mysqlclient accepts via `MYSQL_SET_CHARSET_NAME` / connection attributes — verify the kwarg name. | -| `conv` (dict type conversions) | `CONV` (`Dict` untyped, default `None`) | mismatch | advanced | ✗ | ✗ | ✗ | **Not plumbed**. Type annotation is bare `Dict` (no parameters) and default is `None` but type isn't Optional. Pydantic will accept it; mypy won't. | -| `ssl_mode` (DISABLED/PREFERRED/REQUIRED/VERIFY_CA/VERIFY_IDENTITY) | `SSL_MODE` (str, default `None`) | mismatch | core | ✗ | ✓ | ✓ (via **kwargs) | Stringly typed; validator checks against `CONST_DB_SSL_MODE_MYSQL.__dict__` which includes dunder keys — fragile. Should be enum. | -| `ssl` (dict: `{ca, cert, key, capath, cipher}`) | `SSL_KEY`, `SSL_CERT`, `SSL_CA`, `SSL_CAPATH`, `SSL_CIPHER` | present | core/advanced | ✓ | ✓ | ✓ | Assembled into `args["ssl"]` dict. **Bug** at line 218-219: `if self.SSL_CA:` guards `ssl["ssl-capath"] = self.SSL_CAPATH` — should check `SSL_CAPATH`, not `SSL_CA`. | -| `connect_timeout` | — (commented out) | missing | advanced | N/A | N/A | ✓ (via **kwargs) | Default 10s in mysqlclient. | -| `read_timeout` | — (commented out) | missing | advanced | N/A | N/A | ✓ | | -| `write_timeout` | — (commented out) | missing | advanced | N/A | N/A | ✓ | | -| `max_allowed_packet` | — (commented out) | missing | advanced | N/A | N/A | ✓ | | -| `compress` | — (commented out as `COMPRESSION`) | missing | advanced | N/A | N/A | ✓ | mysqlclient spells it `compress` (bool). | -| `client_flag` | — (commented out) | missing | advanced | N/A | N/A | ✓ | | -| `program_name` | — (commented out) | missing | advanced | N/A | N/A | ✓ | | -| `local_infile` | — (commented out as `ALLOW_LOCAL_INFILE`) | missing | advanced | N/A | N/A | ✓ | Security-relevant (off by default). | -| `auth_plugin` | — | missing | advanced | N/A | N/A | ✓ | e.g., `mysql_clear_password` for PAM/LDAP auth. | -| `init_command` | — | missing | advanced | N/A | N/A | ✓ | Runs on connect; useful for `SET NAMES`/`SET sql_mode`. | -| `unix_socket` | — | missing | advanced | N/A | N/A | ✓ | Required for local-socket connections; we currently force TCP via HOST/PORT. | - -## Findings narrative - -**Core gaps:** None at the connection-string level. `unix_socket` is missing for local-socket deployments but that's advanced. - -**Core mismatches:** - -1. **`db_provider_type` returns `CONST_DB_PROVIDER_TYPE.BIGQUERY`** (line 75). Same copy-paste bug as postgresql.py — should be `MYSQL`. Real defect. -2. **`SSL_MODE` stringly-typed.** No enum constraint; validator compares against `CONST_DB_SSL_MODE_MYSQL.__dict__` which leaks class dunders (`__module__`, `__qualname__`, …) into the accepted set. Retype to a pydantic-validated StrEnum. -3. **`CONV` field shape.** Default is `None` but annotation is bare `Dict` (not `Optional[Dict[int, Callable]]`). Neither validated nor plumbed. - -**Core bugs (new findings in `get_connection_kwargs`):** - -1. **`if self.SSL_CA:` guards the SSL_CAPATH assignment** (line 218). Typo — means `SSL_CAPATH` only gets set when `SSL_CA` is also set. -2. **`SSL_MODE != DISABLED` check fires when SSL_MODE is None.** Because the default is `None` and `None != CONST_DB_SSL_MODE_MYSQL.DISABLED`, the SSL branch runs unconditionally and emits `args["ssl_mode"] = None`. Should be `if self.SSL_MODE is not None and self.SSL_MODE != ...DISABLED:`. -3. **`if self.AUTOCOMMIT:` drops explicit False.** Same `if x:` vs `if x is not None:` pattern. - -**Advanced gaps:** Eight commented-out fields covering timeouts, compression, client flags, and program name. Plus `unix_socket`, `local_infile`, `auth_plugin`, `init_command`, `use_unicode` — all real mysqlclient kwargs with operational value. - -**Advanced mismatches:** Covered by the SSL_CAPATH guard bug above. - -**Extras:** The model validators `validate_auth_ssl_cert` check `self.SSL_MODE is not None and (SSL_CERT or SSL_KEY)` and require both — sensible but could be relaxed (mysqlclient accepts cert without key for some providers). - -**Stale links:** None. - -## Recommended follow-ups - -- **Fix `db_provider_type`** to return `CONST_DB_PROVIDER_TYPE.MYSQL`. One-line bug. -- **Fix the `SSL_CAPATH` guard typo** (line 218: `if self.SSL_CA:` → `if self.SSL_CAPATH:`). -- **Fix the `SSL_MODE != DISABLED` guard** to also check `is not None`. Currently the SSL branch runs when SSL_MODE is None, producing `ssl_mode=None` kwarg. -- **Fix `if self.AUTOCOMMIT:`** → `if self.AUTOCOMMIT is not None:` so explicit False is honored. -- **Retype `SSL_MODE`** to a StrEnum (use `CONST_DB_SSL_MODE_MYSQL` as the enum). Replace the `__dict__` validator. -- **Fix `CONV` type** to `Optional[Dict[int, Any]]` with default `None`. -- **Restore timeouts and compression** from the commented-out block; plumb into `get_connection_kwargs()`. -- **Add `UNIX_SOCKET`, `LOCAL_INFILE`, `INIT_COMMAND`, `AUTH_PLUGIN`** — real operational needs. -- **Validator cleanup**: the `validate_charset` allowlist is narrow (8 entries). MySQL supports many more; either expand or drop the validator and rely on server-side rejection. -- **Secret handling**: password is SecretStr — ensure `get_connection_string_params()` unwraps via `.get_secret_value()` at the driver boundary. diff --git a/docs/superpowers/specs/2026-04-15-settings-audit/postgresql.md b/docs/superpowers/specs/2026-04-15-settings-audit/postgresql.md deleted file mode 100644 index fd205b2..0000000 --- a/docs/superpowers/specs/2026-04-15-settings-audit/postgresql.md +++ /dev/null @@ -1,118 +0,0 @@ -# PostgreSQL Settings Audit - -## Header - -- **Backend:** postgresql -- **Date checked:** 2026-04-15 -- **Our settings class:** `src/mountainash_data/core/settings/postgresql.py` (`PostgreSQLAuthSettings`) -- **Ibis backend file:** `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends/postgres/__init__.py` — `do_connect(host=None, user=None, password=None, port=5432, database=None, schema=None, autocommit=True, **kwargs)` -- **Spec URLs (precedence-tagged):** - - **Driver (authoritative):** libpq connection keyword parameters — https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS - - **Ibis passthrough:** Ibis postgres backend (file above); Ibis uses psycopg (v3); `**kwargs` flows to `psycopg.connect()` which forwards to libpq. - -## Stale-link check - -| URL | Status | -|---|---| -| https://www.postgresql.org/docs/current/libpq-connect.html | OK | - -## Summary counts - -- Core missing: **0** -- Core mismatch: **5** (`SSL_MODE` typed `str` not enum; `SSL_*` family typed `bool` when libpq expects strings/paths; `REQUIRE_AUTH` typed `bool` when libpq expects method list; `db_provider_type` returns BIGQUERY; SSL/keepalive/timeout parameters declared but never plumbed) -- Advanced missing: **~20** (documented but commented out: ISOLATION_LEVEL, READONLY, DEFERABLE, AUTOCOMMIT, STATEMENT_TIMEOUT, LOCK_TIMEOUT, IDLE_IN_TRANSACTION_SESSION_TIMEOUT, TARGET_SESSION_ATTRS, LOAD_BALANCE_HOSTS, CLIENT_ENCODING, DATESTYLE, TIMEZONE, GSS_ENCMODE, KRBSRVNAME, SSL_MIN_PROTOCOL_VERSION, SSL_MAX_PROTOCOL_VERSION, HOSTADDR, CONNECT_TIMEOUT, FALLBACK_APPLICATION_NAME, SERVICE) -- Advanced mismatch: **~12** (every field in the class is declared but **not forwarded by `get_connection_kwargs()`** — it only forwards `SCHEMA`) -- Enum coverage: **4 enums defined but unused** — `PostgresTargetSessionAttrs`, `PostgresRequireAuthMethods`, `PostgresSSLCertNegotiation`, `PostgresSSLCertMode`. None referenced by any field. -- Extra: **1** (`ASYNC_MODE` — psycopg uses `autocommit` + async connections differently; no direct libpq keyword) -- Total audited: **~40** -- Stale links: **0** - -## Parameter table - -Legend — Status: `present` / `missing` / `mismatch` / `extra`. Tier: `core` / `advanced`. Type/Default: ✓ / ✗ / N/A. Ibis passthrough: ✓ / ✗ / unknown. - -| Spec name | Our field | Status | Tier | Type ✓ | Default ✓ | Ibis passthrough | Notes | -|---|---|---|---|---|---|---|---| -| `host` | `HOST` (base) | present | core | ✓ | ✓ | ✓ | Plumbed via connection string only. | -| `hostaddr` | — | missing | advanced | N/A | N/A | ✓ (via **kwargs) | Bypasses DNS; useful in high-traffic setups. | -| `port` | `PORT` (override, default `5432`) | present | core | ✓ | ✓ | ✓ | Matches Ibis default. | -| `dbname` | `DATABASE` (base) | present | core | ✓ | ✓ | ✓ | | -| `user` | `USERNAME` (base) | present | core | ✓ | ✓ | ✓ | | -| `password` | `PASSWORD` (base, SecretStr) | present | core | ✓ | ✓ | ✓ | Plumbed via template; SecretStr not unwrapped. | -| `passfile` | `PASSFILE` (Optional[str]) | present | advanced | ✓ | ✓ | ✓ (via **kwargs) | **Not plumbed** — declared but never forwarded. | -| `require_auth` (list of methods) | `REQUIRE_AUTH` (bool, default `True`) | mismatch | core | ✗ | ✗ | ✓ (via **kwargs) | libpq expects a comma/`!`-separated method list (`scram-sha-256`, `md5`, `!password`, etc.); our bool can't express it. The `PostgresRequireAuthMethods` enum exists but isn't used. | -| `channel_binding` (str: `prefer`/`require`/`disable`) | `CHANNEL_BINDING` (Optional[str]) | present | advanced | ✓ | ✓ | ✓ (via **kwargs) | **Not plumbed**. Should be a 3-value enum. | -| `connect_timeout` | — | missing | core | N/A | N/A | ✓ | Common operational knob. | -| `client_encoding` | — (commented out) | missing | advanced | N/A | N/A | ✓ | | -| `options` | `OPTIONS` (Optional[str]) | present | advanced | ✓ | ✓ | ✓ | **Not plumbed**. | -| `application_name` | `APPLICATION_NAME` (Optional[str]) | present | advanced | ✓ | ✓ | ✓ | **Not plumbed**. | -| `fallback_application_name` | — | missing | advanced | N/A | N/A | ✓ | | -| `keepalives` (0/1) | `KEEPALIVES` (bool, default `True`) | mismatch | advanced | ✓ | ✓ | ✓ | **Not plumbed**. libpq expects 0/1 int, pydantic bool will coerce. | -| `keepalives_idle` | `KEEPALIVES_IDLE` (Optional[int]) | present | advanced | ✓ | ✓ | ✓ | **Not plumbed**. | -| `keepalives_interval` | `KEEPALIVES_INTERVAL` (Optional[int]) | present | advanced | ✓ | ✓ | ✓ | **Not plumbed**. | -| `keepalives_count` | `KEEPALIVES_COUNT` (Optional[int]) | present | advanced | ✓ | ✓ | ✓ | **Not plumbed**. | -| `tcp_user_timeout` | `TCP_USER_TIMEOUT` (Optional[int]) | present | advanced | ✓ | ✓ | ✓ | **Not plumbed**. | -| `sslmode` (`disable`/`allow`/`prefer`/`require`/`verify-ca`/`verify-full`) | `SSL_MODE` (str, default `PREFER`) | mismatch | core | ✗ | ✓ | ✓ | Stringly-typed; should be enum (`CONST_DB_SSL_MODE_POSTGRES` exists but isn't a StrEnum constraint). **Not plumbed**. | -| `sslnegotiation` (`postgres`/`direct`) | `SSL_NEGOTIATION` (bool, default `None`) | mismatch | advanced | ✗ | ✗ | ✓ | **Wrong type**: libpq expects string enum; the `PostgresSSLCertNegotiation` enum exists but is unused. **Not plumbed**. | -| `sslcompression` (0/1) | `SSL_COMPRESSION` (bool, default `None`) | present | advanced | ✓ | ✓ | ✓ | **Not plumbed**. | -| `sslcert` (path) | `SSL_CERT` (bool, default `None`) | mismatch | advanced | ✗ | ✗ | ✓ | **Wrong type**: libpq expects a file path string, not bool. **Not plumbed**. | -| `sslkey` (path) | `SSL_KEY` (bool) | mismatch | advanced | ✗ | ✗ | ✓ | Same as above. **Not plumbed**. | -| `sslpassword` (str) | `SSL_PASSWORD` (bool) | mismatch | advanced | ✗ | ✗ | ✓ | Should be SecretStr. **Not plumbed**. | -| `sslcertmode` (`disable`/`allow`/`require`) | `SSL_CERTMODE` (bool) | mismatch | advanced | ✗ | ✗ | ✓ | `PostgresSSLCertMode` enum exists but unused. **Not plumbed**. | -| `sslrootcert` (path) | `SSL_ROOTCERT` (bool) | mismatch | advanced | ✗ | ✗ | ✓ | Path string expected. **Not plumbed**. | -| `sslcrl` (path) | `SSL_CRL` (bool) | mismatch | advanced | ✗ | ✗ | ✓ | Path string expected. **Not plumbed**. | -| `sslcrldir` (path) | `SSL_CRLDIR` (bool) | mismatch | advanced | ✗ | ✗ | ✓ | Path string expected. **Not plumbed**. | -| `sslsni` (0/1) | `SSL_SNI` (bool) | present | advanced | ✓ | ✓ | ✓ | **Not plumbed**. | -| `ssl_min_protocol_version` | — (commented out) | missing | advanced | N/A | N/A | ✓ | | -| `ssl_max_protocol_version` | — (commented out) | missing | advanced | N/A | N/A | ✓ | | -| `gssencmode` | — (commented out) | missing | advanced | N/A | N/A | ✓ | | -| `krbsrvname` | — (commented out) | missing | advanced | N/A | N/A | ✓ | | -| `target_session_attrs` | — (commented out; enum defined) | missing | advanced | N/A | N/A | ✓ | `PostgresTargetSessionAttrs` enum exists with all six libpq values but no field uses it. | -| `load_balance_hosts` | — (commented out) | missing | advanced | N/A | N/A | ✓ | | -| `service` | — | missing | advanced | N/A | N/A | ✓ | pg_service.conf lookup. | -| (session-level SQL, not libpq) | `SEARCH_PATH` (Optional[str]) | extra/advanced | advanced | ✓ | ✓ | ✓ (via `options=-c search_path=...`) | **Not plumbed**. Goes via `options` not as its own keyword. | -| (psycopg-specific) | `ASYNC_MODE` (bool, default `False`) | extra | advanced | N/A | N/A | ✗ | Not a libpq keyword. **Not plumbed**. | -| `autocommit` (Ibis kwarg, default `True`) | — | missing | core | N/A | N/A | ✓ (direct Ibis kwarg) | Ibis `do_connect()` exposes this as a top-level kwarg; we don't. | - -### Enum coverage - -| Enum | Field that uses it | Intended libpq keyword | Status | -|---|---|---|---| -| `PostgresTargetSessionAttrs` | — | `target_session_attrs` | **Unused** — field is commented out | -| `PostgresRequireAuthMethods` | — | `require_auth` | **Unused** — `REQUIRE_AUTH` is a bool | -| `PostgresSSLCertNegotiation` | — | `sslnegotiation` | **Unused** — `SSL_NEGOTIATION` is a bool | -| `PostgresSSLCertMode` | — | `sslcertmode` | **Unused** — `SSL_CERTMODE` is a bool | - -## Findings narrative - -**Core gaps:** `connect_timeout` and `autocommit` are both missing from the settings layer; both are operationally important. - -**Core mismatches:** - -1. **`db_provider_type` returns `CONST_DB_PROVIDER_TYPE.BIGQUERY`** (line 132). Copy-paste bug — should be `POSTGRESQL`. This is a real defect: any code that branches on `db_provider_type` will route PostgreSQL instances through BigQuery paths. -2. **`REQUIRE_AUTH` is a bool**, but libpq's `require_auth` parameter accepts a comma-separated list of acceptable authentication methods (e.g., `"scram-sha-256,md5"` or `"!password"`). A bool cannot express this. The `PostgresRequireAuthMethods` enum already exists with the correct value set — use it. -3. **SSL field types are wrong.** `SSL_CERT`, `SSL_KEY`, `SSL_PASSWORD`, `SSL_CERTMODE`, `SSL_ROOTCERT`, `SSL_CRL`, `SSL_CRLDIR`, `SSL_NEGOTIATION` are all typed `bool`, but libpq expects path strings (for the cert/key/crl ones), enum strings (for `sslnegotiation`, `sslcertmode`), or a password string (for `sslpassword`). Bool here is silently-invalid. -4. **`SSL_MODE` is stringly-typed** rather than enum-constrained. The `CONST_DB_SSL_MODE_POSTGRES` import exists but isn't enforced as a pydantic enum type. -5. **Nothing is plumbed.** `get_connection_kwargs()` forwards only `SCHEMA`. Every other declared field (30+ of them) is validated by pydantic and then silently dropped. The entire PG-specific surface of this class is currently non-functional. - -**Advanced gaps:** Roughly 20 commented-out libpq parameters. Several are important (`connect_timeout`, `target_session_attrs`, `hostaddr`, `ssl_min_protocol_version`, `gssencmode`, `client_encoding`). The file contains mostly-completed but commented-out plumbing in `get_connection_kwargs()` — restoring and correcting it would close most of these gaps simultaneously. - -**Advanced mismatches:** The un-plumbed fields are the dominant issue, overlapping with the "not plumbed" notes above. Additionally, `SEARCH_PATH` would need to be mapped into `options=-c search_path=...` rather than a bare keyword. - -**Extras:** `ASYNC_MODE` has no direct libpq counterpart; psycopg handles async via `async_`/`AsyncConnection` at the client level, not as a connection keyword. - -**Stale links:** None. - -## Recommended follow-ups - -- **Fix `db_provider_type`** to return `CONST_DB_PROVIDER_TYPE.POSTGRESQL`. This is a one-line bug. -- **Retype the four bool fields that should be enums**: `REQUIRE_AUTH` → `List[PostgresRequireAuthMethods]` (or a comma-joined string from them); `SSL_NEGOTIATION` → `PostgresSSLCertNegotiation`; `SSL_CERTMODE` → `PostgresSSLCertMode`; `SSL_MODE` → enum-constrained. -- **Retype the bool-that-should-be-path fields**: `SSL_CERT`, `SSL_KEY`, `SSL_ROOTCERT`, `SSL_CRL`, `SSL_CRLDIR` → `Optional[str]` (or `Optional[UPath]`). `SSL_PASSWORD` → `Optional[SecretStr]`. -- **Plumb all declared fields through `get_connection_kwargs()`** — the method already has mostly-complete commented scaffolding; restore it, but emit libpq keyword names (`sslmode`, `sslcert`, etc.) and normalize bool→0/1 where libpq expects integers. -- **Restore the commented-out fields** for `TARGET_SESSION_ATTRS`, `CONNECT_TIMEOUT`, `CLIENT_ENCODING`, `STATEMENT_TIMEOUT`, etc., or explicitly decide to drop them. -- **Add `AUTOCOMMIT`** as a first-class field (Ibis `do_connect()` takes it directly). -- **Map `SEARCH_PATH`** into `options=-c search_path=...` in `get_connection_kwargs()`. -- **Reconsider `ASYNC_MODE`**: either plumb it to psycopg's `AsyncConnection` path (if ever used) or remove. -- **Secret handling**: ensure `PASSWORD.get_secret_value()` is called at the connection-kwargs boundary. -- **Drop bespoke `ASYNC_MODE` field** unless there's a live consumer; document otherwise. -- **Validate `SSL_MODE` against the 6-value set** using the existing `CONST_DB_SSL_MODE_POSTGRES` enum. diff --git a/docs/superpowers/specs/2026-04-15-settings-audit/pyiceberg_rest.md b/docs/superpowers/specs/2026-04-15-settings-audit/pyiceberg_rest.md deleted file mode 100644 index 75f1094..0000000 --- a/docs/superpowers/specs/2026-04-15-settings-audit/pyiceberg_rest.md +++ /dev/null @@ -1,92 +0,0 @@ -# PyIceberg REST Settings Audit - -## Header - -- **Backend:** pyiceberg_rest -- **Date checked:** 2026-04-15 -- **Our settings class:** `src/mountainash_data/core/settings/pyiceberg_rest.py` (`PyIcebergRestAuthSettings`) -- **Not an Ibis backend.** The standard "Ibis passthrough" column is **repurposed as "PyIceberg RestCatalog property"** for this report. -- **Spec URLs (precedence-tagged):** - - **Driver (authoritative):** https://py.iceberg.apache.org/configuration/ - - **REST catalog spec (supplementary):** https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml - - **Context (R2 usage example):** https://developers.cloudflare.com/r2/data-catalog/config-examples/pyiceberg/ - -## Stale-link check - -| URL | Status | -|---|---| -| https://py.iceberg.apache.org/configuration/ | OK | -| https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml | OK (assumed — standard Iceberg repo path) | -| https://developers.cloudflare.com/r2/data-catalog/config-examples/pyiceberg/ | OK (assumed — not verified inline) | - -## Summary counts - -- Core missing: **2** (`credential`, `scope`) -- Core mismatch: **3** (`AUTH_METHOD` forces TOKEN but class also claims to be the R2 settings class in docstring; `USE_SSL` default `False` but R2/most REST catalogs require HTTPS; `VERIFY_SSL` isn't plumbed) -- Advanced missing: **9** (`oauth2-server-uri`, `header.*`, all four `s3.*`, all three `rest.sigv4-*`) -- Advanced mismatch: **0** -- Extra: **6** (base-class HOST, PORT, DATABASE, SCHEMA, USERNAME, PASSWORD — all unused) -- Total audited: **~20** -- Stale links: **0** - -## Parameter table - -Legend — Status: `present` / `missing` / `mismatch` / `extra`. Tier: `core` / `advanced`. Type/Default: ✓ / ✗ / N/A. **"RestCatalog property"**: ✓ / ✗ / unknown. - -| Spec name | Our field | Status | Tier | Type ✓ | Default ✓ | RestCatalog property | Notes | -|---|---|---|---|---|---|---|---| -| `uri` (str, required) | `CATALOG_URI` (str, required) | present | core | ✓ | ✓ (both required) | ✓ | Plumbed as `uri=` in `get_connection_kwargs()`. | -| `warehouse` (str, optional) | `WAREHOUSE` (str, **required**) | mismatch | core | ✓ | ✗ | ✓ | Spec says optional; we mark it required (`Field(...)`). Matches R2's always-required usage but misses general REST-catalog flexibility. | -| (no spec equivalent; local naming) | `CATALOG_NAME` (str, required) | extra | core | N/A | N/A | ✓ | Plumbed as `name=` to `RestCatalog(...)`. This is the PyIceberg-side instance name, not a spec property — fine. | -| `token` (str, optional) | `TOKEN` (base, SecretStr) | present | core | ✓ | ✓ | ✓ | Plumbed. Note: `get_connection_kwargs()` packs `self.TOKEN` (the SecretStr object) — check whether `RestCatalog` accepts SecretStr or needs `.get_secret_value()`. | -| `credential` (str "id:secret", optional) | — | missing | core | N/A | N/A | ✓ | OAuth2 client-credentials flow path. Not exposed. | -| `oauth2-server-uri` (str) | — | missing | advanced | N/A | N/A | ✓ | Required for full OAuth2 client-credentials flow. | -| `scope` (str) | — | missing | core | N/A | N/A | ✓ | OAuth2 scope. If we support `credential`, we likely also need this. | -| `header.*` (str) | — | missing | advanced | N/A | N/A | ✓ | Custom request headers (e.g., per-tenant). | -| `rest.sigv4-enabled` (bool) | — | missing | advanced | N/A | N/A | ✓ | AWS SigV4 signing switch (for AWS Glue REST proxy, etc.). | -| `rest.signing-region` (str) | — | missing | advanced | N/A | N/A | ✓ | | -| `rest.signing-name` (str) | — | missing | advanced | N/A | N/A | ✓ | | -| `s3.region` (str) | — | missing | advanced | N/A | N/A | ✓ | | -| `s3.endpoint` (str) | — | missing | advanced | N/A | N/A | ✓ | Needed for S3-compatible services (R2, MinIO) — relevant to this class. | -| `s3.access-key-id` (str) | — | missing | advanced | N/A | N/A | ✓ | Relevant for R2. | -| `s3.secret-access-key` (str) | — | missing | advanced | N/A | N/A | ✓ | Relevant for R2. | -| `s3.session-token` (str) | — | missing | advanced | N/A | N/A | ✓ | | -| (no direct spec property; SDK-level) | `USE_SSL` (bool, default `False`) | mismatch | core | ✓ | ✗ | ✗ | No matching RestCatalog property; effectively dead. Default `False` is particularly risky for a catalog class described as "Cloudflare R2" (which is HTTPS-only). | -| (no direct spec property; SDK-level) | `VERIFY_SSL` (bool, default `True`) | mismatch | advanced | ✓ | ✓ | ✗ | Not plumbed into `get_connection_kwargs()`. | -| — | `AUTH_METHOD` (default `TOKEN`) | extra | core | N/A | N/A | ✗ | Local selector. Forces TOKEN auth; no code path switches to credential/OAuth2 flows. | -| — | `HOST`, `PORT`, `DATABASE`, `SCHEMA`, `USERNAME`, `PASSWORD` (base) | extra | N/A | N/A | N/A | ✗ | Inherited but irrelevant to a REST catalog client. | - -## Findings narrative - -**Core gaps:** No path to OAuth2 client-credentials flow — `credential` and `scope` are absent. Any REST catalog requiring OAuth2 (common for non-bearer deployments) cannot be configured. - -**Core mismatches:** - -1. **Identity crisis.** The docstring describes this class as "Cloudflare R2 storage authentication settings", but the class name and code path are REST-catalog-oriented. Either rename the class / rewrite the docstring, or split into a generic `PyIcebergRestAuthSettings` + an `R2PyIcebergSettings` subclass that adds the R2-specific `s3.endpoint`/`s3.access-key-id`/`s3.secret-access-key` plumbing. -2. **`WAREHOUSE` over-specified as required.** The PyIceberg spec treats it as optional (server can resolve). Making it required prevents using catalogs where the server supplies the warehouse. -3. **`USE_SSL=False` default.** For any production REST catalog (R2, Tabular, etc.), this is wrong. It's also not obviously plumbed — `get_connection_kwargs()` doesn't touch it. Either remove or implement. -4. **`VERIFY_SSL`** declared but never forwarded — another orphaned field. - -**Advanced gaps:** Nine spec properties absent, most notably the `s3.*` family (critical for R2 since the docstring claims R2 focus) and SigV4 support (needed for AWS REST proxies). - -**Advanced mismatches:** None of substance — the gaps are "not there" rather than "there but wrong". - -**Extras:** Six base-class fields (HOST, PORT, DATABASE, SCHEMA, USERNAME, PASSWORD) are inherited but never used. `AUTH_METHOD` is a local selector without any branching code behind it. - -**Stale links:** None. - -**Note on tooling:** Could not introspect `RestCatalog.__init__` via `uv run python -c ...` directly (pyiceberg not confirmed installed in the ambient env); relied on the PyIceberg configuration documentation which is the authoritative parameter list. - -## Recommended follow-ups - -- **Decide class identity first.** Rename / reclassify this class as either a generic REST-catalog settings class or an R2-specific subclass. That decision reshapes all other fixes. -- **Add OAuth2 fields:** `CREDENTIAL` (SecretStr, `"client_id:client_secret"`), `OAUTH2_SERVER_URI`, `SCOPE`. Wire them into `get_connection_kwargs()` behind `AUTH_METHOD == "oauth2"` (or similar). -- **Add `s3.*` family**: `S3_REGION`, `S3_ENDPOINT`, `S3_ACCESS_KEY_ID`, `S3_SECRET_ACCESS_KEY` (SecretStr), `S3_SESSION_TOKEN` (SecretStr). Map to `s3.region`, `s3.endpoint`, etc. — note the dot-prefixed key format PyIceberg expects. -- **Add SigV4 fields**: `REST_SIGV4_ENABLED`, `REST_SIGNING_REGION`, `REST_SIGNING_NAME`. -- **Add `HEADERS`** as `Dict[str, str]`, mapped to `header.` entries. -- **Relax `WAREHOUSE`** to `Optional[str]`. -- **Fix `USE_SSL` default to `True`** or remove the field (REST is HTTPS in practice). -- **Plumb `VERIFY_SSL`** or remove it. -- **Secret handling**: in `get_connection_kwargs()`, call `.get_secret_value()` on `TOKEN` (and any new SecretStr fields) unless `RestCatalog` is verified to accept SecretStr directly. -- **Drop or validate the inherited base-class fields** (HOST/PORT/DATABASE/SCHEMA/USERNAME/PASSWORD) with a warning when set. -- **Add a proper `CONST_STORAGE_PROVIDER_TYPE.PYICEBERG_REST`** (noted as TODO in the source). diff --git a/docs/superpowers/specs/2026-04-15-settings-audit/pyspark.md b/docs/superpowers/specs/2026-04-15-settings-audit/pyspark.md deleted file mode 100644 index 9d6624b..0000000 --- a/docs/superpowers/specs/2026-04-15-settings-audit/pyspark.md +++ /dev/null @@ -1,80 +0,0 @@ -# PySpark Settings Audit - -## Header - -- **Backend:** pyspark -- **Date checked:** 2026-04-15 -- **Our settings class:** `src/mountainash_data/core/settings/pyspark.py` (`PySparkAuthSettings`) -- **Ibis backend file:** `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends/pyspark/__init__.py` — `do_connect(session=None, mode="batch", **kwargs)` (kwargs → `SparkSession.builder.config(**kwargs)`) -- **Scope restriction:** The Spark configuration surface is intentionally out-of-scope for this audit — the docstring already says "Too many options to set. Configure your spark instance directly!" We audit only the declared fields, the Ibis passthrough (`session`, `mode`, `**kwargs`), and spark.* keys explicitly referenced by this class. -- **Spec URLs (precedence-tagged):** - - **Driver (authoritative, scoped):** Ibis pyspark backend (file above); the only direct kwargs are `session` and `mode`. - - **Databricks options (context):** https://docs.databricks.com/en/spark/conf.html - - **Spark config reference (out of scope):** https://spark.apache.org/docs/3.5.1/configuration.html#available-properties - -## Stale-link check - -| URL | Status | -|---|---| -| https://docs.databricks.com/en/spark/conf.html | OK (assumed) | -| https://spark.apache.org/docs/3.5.1/configuration.html | OK (assumed; version-pinned URL — worth bumping to `latest`) | - -## Summary counts - -- Core missing: **1** (`session` — the canonical way to pass a caller-built SparkSession is absent) -- Core mismatch: **3** (`MODE` default `None` vs Ibis default `"batch"`; `MODE` stringly-typed despite nominal `PySparkMode` class; `PySparkMode` is a plain class, not a `StrEnum`) -- Advanced missing: **All `spark.*` config keys** (by declared scope; this is deliberate) -- Advanced mismatch: **3** (`PARTITIONS` typed `int` but default is `{}` — a dict; `APPLICATION_NAME`/`SPARK_MASTER`/`WAREHOUSE_DIR` typed `str` with default `None` — not Optional; `AUTH_METHOD` is a string literal `"none"` where other classes use `CONST_DB_AUTH_METHOD`) -- Extra: **8** (all inherited base-class fields — HOST, PORT, DATABASE, SCHEMA, USERNAME, PASSWORD, TOKEN + docstring mislabels class as "SQLite authentication settings") -- Total audited: **~12** -- Stale links: **0** - -## Parameter table - -Legend — Status: `present` / `missing` / `mismatch` / `extra`. Tier: `core` / `advanced`. Type/Default: ✓ / ✗ / N/A. Ibis passthrough: ✓ / ✗ / unknown. - -| Spec name | Our field | Status | Tier | Type ✓ | Default ✓ | Ibis passthrough | Notes | -|---|---|---|---|---|---|---|---| -| `session` (SparkSession) | — | missing | core | N/A | N/A | ✓ (direct Ibis kwarg) | Ibis can accept a pre-built SparkSession. Not exposed — all configuration flows via builder-config kwargs only. | -| `mode` (`batch`/`streaming`) | `MODE` (str, default `None`) | mismatch | core | ✗ | ✗ | ✓ | Ibis default is `"batch"`; ours `None`. `PySparkMode` exists as a plain class (not Enum/StrEnum) with `BATCH`/`STREAMING` constants but **isn't used as the field type**. | -| `spark.app.name` | `APPLICATION_NAME` (str, default `None`) | present | advanced | ✗ | ✓ | ✓ (via **kwargs) | Type is non-Optional `str` with `None` default — pydantic accepts but annotation lies. Plumbed into connection-string params as `spark_app_name`; **not plumbed into `get_connection_kwargs()`**. | -| `spark.master` | `SPARK_MASTER` (str, default `None`) | present | advanced | ✗ | ✓ | ✓ (via **kwargs) | Same type-annotation issue. **Not plumbed into `get_connection_kwargs()`** — only into connection string. | -| `spark.sql.warehouse.dir` | `WAREHOUSE_DIR` (str, default `None`) | present | advanced | ✗ | ✓ | ✓ (via **kwargs) | Same type-annotation issue. **Not plumbed into `get_connection_kwargs()`**. | -| `spark.sql.shuffle.partitions` | `PARTITIONS` (int, default `{}`) | mismatch | advanced | ✗ | ✗ | ✓ (post-connect via `spark.conf.set`) | **Type/default mismatch**: annotation is `int` but default is `{}` (dict). Pydantic will error at model instantiation unless `PARTITIONS` is explicitly provided. Plumbed via `get_post_connection_options()`. | -| — | `AUTH_METHOD` (default `"none"` literal) | extra | core | N/A | N/A | ✗ | String literal where other classes use `CONST_DB_AUTH_METHOD`. No auth concept for Spark anyway. | -| — | `HOST`, `PORT`, `DATABASE`, `SCHEMA`, `USERNAME`, `PASSWORD`, `TOKEN` (base) | extra | N/A | N/A | N/A | ✗ | Inherited but meaningless for PySpark. | - -## Findings narrative - -**Core gaps:** The `session` injection path is missing. For any non-trivial Spark deployment (Databricks, EMR, existing Spark context), callers construct their own `SparkSession` — and then Ibis just wraps it. Our class forces builder-config creation, which doesn't cover the common case. - -**Core mismatches:** - -1. **`MODE` default drift.** Ibis defaults to `"batch"`; ours defaults to `None`. When `None`, `get_connection_kwargs()` drops the key, and Ibis's own default kicks in — so this works, but the declared default is misleading. -2. **`MODE` stringly-typed.** `PySparkMode` is a plain class (not `Enum`/`StrEnum`) with constants `BATCH = "batch"` and `STREAMING = "streaming"`. It's not enforced as the field type, and it isn't importable as an Enum for consumers. -3. **Docstring mislabels the class** as "SQLite authentication settings" (line 18 — copy-paste from `sqlite.py`). Also the file comment on line 1 says `#path: .../file/sqlite.py`. - -**Core bugs:** - -1. **`PARTITIONS: int = Field(default={})`** — type says `int`, default is `{}`. Pydantic will raise a validation error unless someone sets `PARTITIONS` to an int at construction time. Additionally, `if self.PARTITIONS:` treats an empty dict and `0` identically as falsy, then emits `spark.sql.shuffle.partitions = ` — passing an int to a config key that expects a string may or may not be accepted by `SparkSession.conf.set`. -2. **Non-Optional fields with `None` defaults** (`APPLICATION_NAME`, `SPARK_MASTER`, `WAREHOUSE_DIR`, `MODE`). Pydantic v2 is lenient but this produces wrong schemas and IDE hints. - -**Advanced gaps:** Out of scope by declaration — the docstring punts to direct Spark configuration. Reasonable. - -**Advanced mismatches:** `get_connection_kwargs()` only emits `mode`. `SPARK_MASTER`, `APPLICATION_NAME`, `WAREHOUSE_DIR` are declared and plumbed into `get_connection_string_params()` but never into the kwargs that actually reach Ibis — so they don't configure the SparkSession unless the connection-string builder forwards them (behavior unclear). Meanwhile the canonical `SparkSession.builder.config()` kwargs path uses dotted keys like `spark.app.name` — our params use `spark_app_name`, so even if forwarded, the key names are wrong. - -**Extras:** Eight inherited base-class fields that don't apply to PySpark plus the docstring/path-comment mislabeling. - -**Stale links:** The Spark docs link is version-pinned to 3.5.1 — fine if that's the target, but bump to `latest` or document the pin. - -## Recommended follow-ups - -- **Fix the docstring** (line 18) — says "SQLite authentication settings". Replace with a correct PySpark description. Fix the `#path:` comment on line 1. -- **Add `SESSION`** field (`Optional[Any]` typed as `SparkSession`) to support the Ibis `session=` injection path. -- **Retype `MODE`** as `Optional[Literal["batch", "streaming"]]` or a proper `StrEnum`. Default to `"batch"` to match Ibis. -- **Convert `PySparkMode` to `StrEnum`** and use it as the type. -- **Fix `PARTITIONS` type/default mismatch**: either `Optional[int] = Field(default=None)` or `Optional[Dict[str, Any]] = Field(default=None)` — pick one and match the annotation. -- **Retype the non-Optional-with-None-default fields** (`APPLICATION_NAME`, `SPARK_MASTER`, `WAREHOUSE_DIR`) → `Optional[str]`. -- **Fix spark.* key naming in `get_connection_kwargs()`**: emit `spark.app.name`, `spark.master`, `spark.sql.warehouse.dir` rather than `spark_app_name` etc. -- **Decide whether settings-driven builder config is worth keeping** given the docstring's own advice to configure Spark directly. If not, collapse the class down to `{session, mode}` + optional `partitions` post-connect and direct users to supply a pre-built SparkSession for everything else. -- **Drop `AUTH_METHOD` default literal** — Spark has no auth concept at this layer. diff --git a/docs/superpowers/specs/2026-04-15-settings-audit/redshift.md b/docs/superpowers/specs/2026-04-15-settings-audit/redshift.md deleted file mode 100644 index 387c127..0000000 --- a/docs/superpowers/specs/2026-04-15-settings-audit/redshift.md +++ /dev/null @@ -1,101 +0,0 @@ -# Redshift Settings Audit - -## Header - -- **Backend:** redshift -- **Date checked:** 2026-04-15 -- **Our settings class:** `src/mountainash_data/core/settings/redshift.py` (`RedshiftAuthSettings`) -- **Ibis backend file:** `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends/postgres/__init__.py` — Redshift rides the **postgres** Ibis backend (Redshift speaks the PostgreSQL wire protocol). `do_connect(host, user, password, port=5432, database, schema, autocommit=True, **kwargs)`. -- **Spec URLs (precedence-tagged):** - - **Driver (authoritative):** redshift_connector (Amazon) — https://github.com/aws/amazon-redshift-python-driver (also callable via psycopg/libpq for plain wire-protocol mode) - - **libpq (via Ibis postgres):** https://www.postgresql.org/docs/current/libpq-connect.html - - **AWS Redshift connection params:** https://docs.aws.amazon.com/redshift/latest/mgmt/python-configuration-options.html - -## Stale-link check - -| URL | Status | -|---|---| -| https://github.com/aws/amazon-redshift-python-driver | OK (assumed) | -| https://docs.aws.amazon.com/redshift/latest/mgmt/python-configuration-options.html | OK (assumed) | -| https://www.postgresql.org/docs/current/libpq-connect.html | OK (verified in postgresql audit) | - -## Summary counts - -- Core missing: **1** (`HOST` — set on base but never populated; Redshift endpoint must be resolved from `CLUSTER_IDENTIFIER`/`WORKGROUP_NAME`, and the resolution code is commented out) -- Core mismatch: **5** (`get_connection_kwargs()` returns `{}`; `SSL` bool doesn't map to libpq `sslmode` enum; `FORCE_IAM`/IAM path targets redshift_connector kwargs but connection goes via postgres dialect; AWS region regex rejects 4-digit suffixes like `us-west-2`; serverless path unimplemented) -- Advanced missing: **~15** (cluster auto-resolve, IAM token-exchange, GetClusterCredentials, db_user, db_groups, auto_create, group_federation, connect_retry_count, connect_retry_delay, ssl_insecure, iam_disable_cache, application_name, statement_timeout, most timeouts) -- Advanced mismatch: **2** (`AUTO_CREATE` declared, not plumbed; `PROFILE_NAME` declared, not plumbed; `WORKGROUP_NAME` used only in validator, not plumbed) -- Extra: **3** (base-class `TOKEN` unused; `CLUSTER_READ_ONLY` encoded into connection-string params only; `ENDPOINT_URL` only used in commented boto3 code) -- Total audited: **~25** -- Stale links: **0** - -## Parameter table - -Legend — Status: `present` / `missing` / `mismatch` / `extra`. Tier: `core` / `advanced`. Type/Default: ✓ / ✗ / N/A. Ibis passthrough (via postgres): ✓ / ✗ / unknown. - -| Spec name | Our field | Status | Tier | Type ✓ | Default ✓ | Ibis passthrough | Notes | -|---|---|---|---|---|---|---|---| -| `host` (endpoint) | `HOST` (base) | mismatch | core | ✓ | ✗ | ✓ (via postgres) | Redshift endpoints are `...redshift.amazonaws.com` — normally resolved from `CLUSTER_IDENTIFIER` + `REGION` via `boto3.redshift.describe_clusters()`. The resolver is **commented out** (`_get_cluster_endpoint`). Users must supply HOST manually, defeating the cluster-identifier abstraction. | -| `port` | `PORT` (override, default `5439`) | present | core | ✓ | ✓ | ✓ | Matches Redshift default (5439, not 5432). | -| `dbname` | `DATABASE` (base) | present | core | ✓ | ✓ | ✓ | | -| `user` | `USERNAME` (base) | present | core | ✓ | ✓ | ✓ | | -| `password` | `PASSWORD` (base, SecretStr) | present | core | ✓ | ✓ | ✓ | | -| `region` | `REGION` (str, required) | present | core | ✓ | ✓ | ✗ | Validator regex `^[a-z]{2}-[a-z]+-\d{1}$` — accepts `us-east-1` but **rejects future 2-digit region suffixes** (AWS has used 2-digit suffixes in some partitions). Also doesn't include region partitions like `us-gov-west-1` (has a hyphen in the middle segment — passes regex but the `{2}-[a-z]+-\d` shape is lucky, not robust). | -| `cluster_identifier` | `CLUSTER_IDENTIFIER` (Optional[str]) | present | core | ✓ | ✓ | ✗ | Used only by the commented-out boto3 resolver. Not plumbed to driver. | -| `iam` (bool) | — (derived from `AUTH_METHOD == IAM` or `FORCE_IAM`) | present | core | ✓ | ✓ | ✓ (libpq `iam=true`) | Encoded into connection string. | -| `iam_role_arn` (assume-role ARN) | `IAM_ROLE_ARN` (Optional[str]) | present | core | ✓ | ✓ | ✓ | Plumbed via `get_connection_string_params()` as `iam_role_arn=`. Validator enforces `arn:aws:iam::` prefix — fine for commercial partition, rejects `arn:aws-us-gov:iam::` or `arn:aws-cn:iam::`. | -| `aws_access_key_id` | `ACCESS_KEY_ID` (Optional[str]) | present | core | ✓ | ✓ | ✓ | Plumbed when IAM path + keys set. | -| `aws_secret_access_key` | `SECRET_ACCESS_KEY` (Optional[SecretStr]) | present | core | ✓ | ✓ | ✓ | Plumbed; SecretStr not unwrapped. | -| `aws_session_token` | `SESSION_TOKEN` (Optional[SecretStr]) | present | core | ✓ | ✓ | ✓ | Plumbed; SecretStr not unwrapped. | -| `sslmode` (libpq) / `ssl` (bool) | `SSL` (bool, default `True`) | mismatch | core | ✓ | ✓ | ✓ | Bool → connection-string `sslmode=verify-full` (hardcoded). Should expose `sslmode` enum to support `require` / `prefer` modes. | -| `serverless` / `workgroup_name` | `SERVERLESS`, `WORKGROUP_NAME` | present | core | ✓ | ✓ | ✗ | Serverless endpoint resolver is **commented out and broken** (line 252 references undefined `client`). Cannot use serverless mode. | -| `profile_name` | `PROFILE_NAME` (Optional[str]) | present | advanced | ✓ | ✓ | ✗ | Only referenced in commented boto3 resolver. Not plumbed. | -| `auto_create` | `AUTO_CREATE` (bool, default `False`) | present | advanced | ✓ | ✓ | ✓ (redshift_connector kwarg) | **Not plumbed**. | -| `db_groups` | — | missing | advanced | N/A | N/A | ✓ | IAM-federated users group membership. | -| `db_user` | — | missing | advanced | N/A | N/A | ✓ | Target DB user for IAM auth. | -| `group_federation` | — | missing | advanced | N/A | N/A | ✓ | | -| `connect_retry_count` | — | missing | advanced | N/A | N/A | ✓ | | -| `connect_retry_delay` | — | missing | advanced | N/A | N/A | ✓ | | -| `ssl_insecure` | — | missing | advanced | N/A | N/A | ✓ | | -| `iam_disable_cache` | — | missing | advanced | N/A | N/A | ✓ | | -| `application_name` (libpq) | — | missing | advanced | N/A | N/A | ✓ | | -| `statement_timeout` | — | missing | advanced | N/A | N/A | ✓ | | -| (our custom) | `ENDPOINT_URL` (Optional[str]) | extra | advanced | ✓ | ✓ | ✗ | Only used in commented boto3 resolver (for custom endpoints / LocalStack). Dead outside resolver. | -| (our custom) | `CLUSTER_READ_ONLY` (bool) | extra | advanced | ✓ | ✓ | ✓ (as `readonly=true` query param) | Redshift supports `readonly` driver property; plumbed via query string. | -| — | `TOKEN` (base) | extra | N/A | N/A | N/A | ✗ | Not used for Redshift. | - -## Findings narrative - -**Core gaps:** - -1. **Host resolution is broken.** Redshift's value prop over naked postgres is that you give it a cluster identifier + region and the settings class fetches the endpoint. That code (`_get_cluster_endpoint`, `_get_serverless_endpoint`) is commented out. Users currently must set `HOST` themselves, which means `CLUSTER_IDENTIFIER`, `WORKGROUP_NAME`, `SERVERLESS`, `ENDPOINT_URL`, `PROFILE_NAME`, and much of the AWS auth surface are only used for validation, not for actual connection. - -**Core mismatches:** - -1. **`get_connection_kwargs()` returns `{}`.** Everything routes via connection-string params, so any Ibis kwarg (e.g., `autocommit`) must be passed via the caller or via the URL query string. Aligns with postgres audit finding. -2. **`SSL` as bool** collapses the five libpq `sslmode` values to two. Redshift best-practice is `sslmode=verify-full` (which the code hardcodes), so this is defensible — but users needing `require` (e.g., with a self-signed cert for internal testing) can't express it. -3. **IAM / redshift_connector surface area mixed with postgres path.** The IAM path emits `aws_access_key_id` / `aws_secret_access_key` / `iam_role_arn` as top-level params — these are *redshift_connector* kwargs, not libpq. Since Ibis uses psycopg (postgres), these may be silently dropped at the driver boundary. Needs a plumbing test against the actual Ibis path. -4. **`validate_region` regex** `^[a-z]{2}-[a-z]+-\d{1}$` accidentally rejects GovCloud (`us-gov-west-1` — fails `[a-z]+` segment because of hyphens) and may reject future regions with 2-digit indices. -5. **Serverless path is broken.** `_get_serverless_endpoint` references an undefined `client` (line 252) with the boto3 lines commented out around it. Serverless mode validator passes if `WORKGROUP_NAME` is set, but there's no working code to resolve the endpoint. - -**Advanced gaps:** The entire redshift_connector advanced surface (db_user, db_groups, group_federation, connect_retry_*, ssl_insecure, iam_disable_cache) is absent. If the intent is to run through Ibis's postgres backend, many of those don't apply anyway — but then `auto_create`, `profile_name`, `endpoint_url` don't apply either. - -**Advanced mismatches:** Multiple orphan fields (`AUTO_CREATE`, `PROFILE_NAME`, `WORKGROUP_NAME` outside validator) that validate but do nothing. - -**Extras:** `ENDPOINT_URL`, `CLUSTER_READ_ONLY`, `TOKEN` noted in table. The `_init_provider_specific` method is named differently from the `_post_init` convention used elsewhere — this method is never called by the base class (`_post_init` is the hook), so its validators are **dead code**. The serverless/provisioned coherence check doesn't fire. - -**Stale links:** Docstring has **no source URLs**. Add them. - -## Recommended follow-ups - -- **Add source URLs to the class docstring** — it currently has none (unlike most sibling classes). -- **Decide the driver story first.** Either (a) route through Ibis postgres with plain libpq IAM support and drop the redshift_connector-specific fields, or (b) supply `driver_type="redshift_connector"` and route around Ibis. The current mix pretends to support both and delivers neither cleanly. -- **Rename `_init_provider_specific` → `_post_init`** so the validators actually fire. Currently `_post_init` is a no-op (`pass`) and the serverless/cluster checks never run. -- **Implement or remove `_get_cluster_endpoint`/`_get_serverless_endpoint`**. If keeping, add boto3 as a dependency and plumb the resolved HOST into the connection string. If not, rename fields to signal they're only labels. -- **Broaden `validate_region`** to accept GovCloud and other partitions: `^[a-z]{2,4}-[a-z-]+-\d{1,2}$` or use the AWS-published region list. -- **Broaden `validate_role_arn`** to accept partition variants (`arn:aws:`, `arn:aws-us-gov:`, `arn:aws-cn:`). -- **Plumb `AUTO_CREATE`**, **`PROFILE_NAME`**, **`WORKGROUP_NAME`** if the redshift_connector path is kept. -- **Replace `SSL: bool`** with `SSL_MODE: enum` (re-use `PostgresSSLCertMode` or similar) — hardcoded `verify-full` is an opinion, not a default. -- **Unwrap SecretStr** for `PASSWORD`, `SECRET_ACCESS_KEY`, `SESSION_TOKEN` at the kwargs boundary. -- **Remove unreachable extras** (`TOKEN` base field) or document why they're there. -- **Fix the broken `][p9]` comment** at line 129 (looks like an errant keystroke). diff --git a/docs/superpowers/specs/2026-04-15-settings-audit/snowflake.md b/docs/superpowers/specs/2026-04-15-settings-audit/snowflake.md deleted file mode 100644 index f5813e4..0000000 --- a/docs/superpowers/specs/2026-04-15-settings-audit/snowflake.md +++ /dev/null @@ -1,111 +0,0 @@ -# Snowflake Settings Audit - -## Header - -- **Backend:** snowflake -- **Date checked:** 2026-04-15 -- **Our settings class:** `src/mountainash_data/core/settings/snowflake.py` (`SnowflakeAuthSettings`) -- **Ibis backend file:** `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends/snowflake/__init__.py` — `do_connect(user, account, database, ..., **kwargs)` (kwargs → `snowflake.connector.connect`) -- **Spec URLs (precedence-tagged):** - - **Driver (authoritative):** https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-api#label-snowflake-connector-methods-connect - - **OAuth examples:** https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-example#connecting-with-oauth - - **Connect guide / session params:** https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-connect - -## Stale-link check - -| URL | Status | -|---|---| -| https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-api | OK (assumed) | -| https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-example | OK (assumed) | -| https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-connect | OK (assumed) | - -## Summary counts - -- Core missing: **0** -- Core mismatch: **4** (`CONST_SNOWFLAKE_AUTHENTICATOR` enum values have trailing whitespace bugs; `AUTHENTICATOR` not plumbed; OAuth branch sets `authenticator = AUTH_METHOD` (internal selector string) instead of `"oauth"`; SecretStr fields passed without `.get_secret_value()`) -- Advanced missing: **~15** (session_parameters, query_tag, application, client_session_keep_alive, login_timeout, network_timeout, socket_timeout, client_store_temporary_credential, paramstyle, insecure_mode, ocsp_fail_open, autocommit, validate_default_parameters, numpy, consent_cache_id_token, arrow_number_to_decimal) -- Advanced mismatch: **1** (`TIMEZONE` declared, not plumbed) -- Enum coverage: **1 enum defined (`CONST_SNOWFLAKE_AUTHENTICATOR`), NOT used as a type — `AUTHENTICATOR` is `Optional[str]` with a `.member_values()` validator** -- Extra: **1** (`HOST` in `get_connection_string_params()` — Snowflake connector uses `account`, not `host`) -- Total audited: **~25** -- Stale links: **0** - -## Parameter table - -Legend — Status: `present` / `missing` / `mismatch` / `extra`. Tier: `core` / `advanced`. Type/Default: ✓ / ✗ / N/A. Ibis passthrough: ✓ / ✗ / unknown. - -| Spec name | Our field | Status | Tier | Type ✓ | Default ✓ | Ibis passthrough | Notes | -|---|---|---|---|---|---|---|---| -| `account` (required) | `ACCOUNT` (str, required) | present | core | ✓ | ✓ | ✓ | Validators enforce non-null + alnum/dash/underscore. | -| `user` | `USERNAME` (base) | present | core | ✓ | ✓ | ✓ | | -| `password` | `PASSWORD` (base, SecretStr) | present | core | ✓ | ✓ | ✓ | `get_connection_string_params()` passes SecretStr as-is. | -| `database` | `DATABASE` (base) | present | core | ✓ | ✓ | ✓ | | -| `schema` | `SCHEMA` (base) | present | core | ✓ | ✓ | ✓ | | -| `warehouse` | `WAREHOUSE` (str, required) | present | core | ✓ | ✓ | ✓ | Spec says optional; ours required. | -| `role` | `ROLE` (Optional[str]) | present | core | ✓ | ✓ | ✓ | **Not plumbed** — declared but never added to args. | -| `authenticator` (`snowflake`/`oauth`/`externalbrowser`/`okta`/`username_password_mfa`/...) | `AUTHENTICATOR` (Optional[str]) | mismatch | core | ✗ | ✓ | ✓ | Typed `Optional[str]`; should be enum-constrained. The enum `CONST_SNOWFLAKE_AUTHENTICATOR` exists but is only used inside a validator via `.member_values()`. **Commented-out plumbing** at line 250. | -| `token` (OAuth) | `OAUTH_TOKEN` (SecretStr) | present | core | ✓ | ✓ | ✓ | Plumbed as `token=`. SecretStr not unwrapped. | -| `private_key` | `PRIVATE_KEY` (SecretStr) | present | core | ✓ | ✓ | ✓ | Plumbed; SecretStr not unwrapped. | -| `private_key_file` | `PRIVATE_KEY_PATH` (Optional[str]) | present | core | ✓ | ✓ | ✓ | Note: driver kwarg name is `private_key_file` in recent versions, not `private_key_path`. Verify against connector version. | -| `private_key_file_pwd` | `PRIVATE_KEY_PASSPHRASE` (SecretStr) | present | core | ✓ | ✓ | ✓ | Driver kwarg is `private_key_file_pwd`. Name drift. | -| `connection_name` (toml lookup) | `CONNECTION_NAME` (Optional[str]) | present | core | ✓ | ✓ | ✓ | Plumbed. `connections.toml` support noted as TODO in docstring. | -| `session_parameters` (dict) | — | missing | advanced | N/A | N/A | ✓ | Ibis doc explicitly calls this out. | -| `query_tag` | — (commented out) | missing | advanced | N/A | N/A | ✓ (via session_parameters) | | -| `application` | — (commented out) | missing | advanced | N/A | N/A | ✓ | Default `"MountainAsh"` value lost. | -| `client_session_keep_alive` | — (commented out) | missing | advanced | N/A | N/A | ✓ | Long-running session friendly. | -| `login_timeout` | — | missing | advanced | N/A | N/A | ✓ | Default 120s. | -| `network_timeout` / `socket_timeout` | — | missing | advanced | N/A | N/A | ✓ | | -| `insecure_mode` | — | missing | advanced | N/A | N/A | ✓ | Security-relevant. | -| `ocsp_fail_open` | — | missing | advanced | N/A | N/A | ✓ | | -| `autocommit` | — | missing | advanced | N/A | N/A | ✓ | | -| `paramstyle` | — | missing | advanced | N/A | N/A | ✓ | | -| `timezone` | `TIMEZONE` (Optional[str]) | present | advanced | ✓ | ✓ | ✓ | **Not plumbed**. Belongs in `session_parameters`, not a top-level kwarg. | -| `ocsp_response_cache_filename` | — | missing | advanced | N/A | N/A | ✓ | | -| `oauth_client_id` | `OAUTH_CLIENT_ID` (Optional[str]) | present | advanced | ✓ | ✓ | ✓ | Plumbed. Note: not a standard Snowflake connector kwarg — OAuth flows typically use `token` directly; may be custom. | -| `oauth_client_secret` | `OAUTH_CLIENT_SECRET` (SecretStr) | present | advanced | ✓ | ✓ | ✓ | Same — verify. | -| `oauth_refresh_token` | `OAUTH_REFRESH_TOKEN` (SecretStr) | present | advanced | ✓ | ✓ | ✓ | Same — verify. | -| `okta_account_name` | `OKTA_ACCOUNT_NAMER` (Optional[str]) | present | advanced | ✓ | ✓ | ✓ | **Typo in field name** (`NAMER` vs `NAME`). Also **not plumbed**. | -| — | `HOST` (base) referenced in `get_connection_string_params()` | extra | core | N/A | N/A | ✗ | Snowflake connector uses `account`, not `host`; emitting both is redundant and `host` may confuse the driver. | - -### Enum coverage - -| Enum | Values | Field that uses it | Status | -|---|---|---|---| -| `CONST_SNOWFLAKE_AUTHENTICATOR` | SNOWFLAKE=`"snowflake "`, OAUTH=`"oauth"`, OKTA=`"okta"`, EXTERNAL_BROWSER=`"externalbrowser"`, PASSWORD_MFA=`"username_password_mfa "` | `AUTHENTICATOR` (indirectly via validator) | **Defective** — `SNOWFLAKE` and `PASSWORD_MFA` values have **trailing spaces** (`"snowflake "`, `"username_password_mfa "`). Snowflake will reject these. | - -## Findings narrative - -**Core gaps:** None. - -**Core mismatches / bugs:** - -1. **`CONST_SNOWFLAKE_AUTHENTICATOR` enum values contain trailing whitespace** (lines 19 and 23). `"snowflake "` and `"username_password_mfa "` would be rejected by Snowflake's server. The validator at line 126 compares against `member_values()`, so the whitespace is enforced. This locks out the default authenticator entirely if anyone tries to set it explicitly. -2. **OAuth branch sets `authenticator = AUTH_METHOD`** (line 255). `AUTH_METHOD` is an internal selector (`CONST_DB_AUTH_METHOD.OAUTH` = `"oauth"` presumably), which coincidentally matches the Snowflake authenticator string. Relying on the coincidence is fragile; should use `CONST_SNOWFLAKE_AUTHENTICATOR.OAUTH` or an explicit `"oauth"` literal. -3. **`AUTHENTICATOR` not plumbed outside OAuth path.** Lines 250-251 are commented out. Users setting `AUTHENTICATOR="externalbrowser"` get nothing. -4. **`ROLE` not plumbed.** Declared but never emitted to kwargs. -5. **SecretStr not unwrapped anywhere.** `PASSWORD`, `OAUTH_TOKEN`, `PRIVATE_KEY`, `PRIVATE_KEY_PASSPHRASE`, `OAUTH_CLIENT_SECRET`, `OAUTH_REFRESH_TOKEN` all pass the SecretStr object directly. -6. **`HOST` in connection-string params** is spurious for Snowflake. - -**Advanced gaps:** `session_parameters` is the headline omission — it's the canonical way to configure Snowflake session behavior (query_tag, timezone, statement_timeout, autocommit, etc.) and Ibis documents it as a recognized kwarg. Instead we declare standalone `TIMEZONE` and orphan it. Plus ~14 other connect-time kwargs for timeouts, security (insecure_mode, ocsp_fail_open), and client identification (application). - -**Advanced mismatches:** `TIMEZONE` as a top-level kwarg doesn't match driver expectations — belongs in `session_parameters={"TIMEZONE": ...}`. - -**Extras:** `HOST`-in-params noted above. `OKTA_ACCOUNT_NAMER` (typo) is also unused. - -**Stale links:** None. - -## Recommended follow-ups - -- **Fix enum whitespace bug** (`"snowflake "` → `"snowflake"`, `"username_password_mfa "` → `"username_password_mfa"`). High priority. -- **Retype `AUTHENTICATOR`** as `Optional[CONST_SNOWFLAKE_AUTHENTICATOR]` (StrEnum). Drop the manual validator. -- **Plumb `AUTHENTICATOR` universally** (un-comment line 251 equivalent). Remove the coincidental `AUTH_METHOD == authenticator` coupling. -- **Plumb `ROLE`** — add to args in `get_connection_kwargs()`. -- **Fix typo**: `OKTA_ACCOUNT_NAMER` → `OKTA_ACCOUNT_NAME`. Plumb it. -- **Unwrap SecretStr** at the kwargs boundary for all secret fields (`.get_secret_value()`). -- **Add `SESSION_PARAMETERS: Dict[str, Any]` Field** and route `TIMEZONE` through it; add `QUERY_TAG`, `CLIENT_SESSION_KEEP_ALIVE`, `AUTOCOMMIT`. -- **Add `APPLICATION`, `LOGIN_TIMEOUT`, `NETWORK_TIMEOUT`, `SOCKET_TIMEOUT`, `INSECURE_MODE`, `OCSP_FAIL_OPEN`** — standard operational/security knobs. -- **Remove `HOST` from connection-string params** — Snowflake uses `account`. -- **Rename `PRIVATE_KEY_PATH`/`PRIVATE_KEY_PASSPHRASE`** to emit driver-correct kwarg names (`private_key_file`, `private_key_file_pwd`) in `get_connection_kwargs()`. Keep settings-facing names if preferred, but translate at the boundary. -- **Verify `OAUTH_CLIENT_ID`/`OAUTH_CLIENT_SECRET`/`OAUTH_REFRESH_TOKEN` kwargs** against current snowflake-connector-python — standard OAuth2 flow uses `token=` once acquired, not client credentials at connect time. These may be doing nothing. -- **Relax `WAREHOUSE`** to Optional (Snowflake can use account-default warehouse). -- **Support `connections.toml` lookup** per the in-code TODO. diff --git a/docs/superpowers/specs/2026-04-15-settings-audit/sqlite.md b/docs/superpowers/specs/2026-04-15-settings-audit/sqlite.md deleted file mode 100644 index 22b5f61..0000000 --- a/docs/superpowers/specs/2026-04-15-settings-audit/sqlite.md +++ /dev/null @@ -1,74 +0,0 @@ -# SQLite Settings Audit - -## Header - -- **Backend:** sqlite -- **Date checked:** 2026-04-15 -- **Our settings class:** `src/mountainash_data/core/settings/sqlite.py` (`SQLiteAuthSettings`, inherits `BaseDBAuthSettings`) -- **Ibis backend file:** `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends/sqlite/__init__.py` (`do_connect(database, type_map)`) -- **Spec URLs (precedence-tagged):** - - **Driver (authoritative):** https://docs.python.org/3/library/sqlite3.html#sqlite3.connect - - **Ibis passthrough (authoritative for what flows through):** local file above - - **Vendor (context only — PRAGMAs are post-connect SQL, not connect kwargs):** https://www.sqlite.org/pragma.html - -## Stale-link check - -| URL | Status | -|---|---| -| https://docs.python.org/3/library/sqlite3.html#sqlite3.connect | OK | -| https://www.sqlite.org/pragma.html | OK | - -## Summary counts - -- Core missing: **0** -- Core mismatch: **0** -- Advanced missing: **8** (stdlib kwargs not exposed; all are pass-through via Ibis `type_map` is the only advanced we expose) -- Advanced mismatch: **0** -- Extra (ours, not in spec): **7** (base-class fields irrelevant to SQLite: HOST, PORT, SCHEMA, USERNAME, PASSWORD, TOKEN, AUTH_METHOD) -- Total audited: **16** -- Stale links: **0** - -## Parameter table - -Legend — Status: `present` / `missing` / `mismatch` / `extra`. Tier: `core` / `advanced`. Type/Default: ✓ / ✗ / N/A. Ibis passthrough: ✓ / ✗ / unknown. - -| Spec name | Our field | Status | Tier | Type ✓ | Default ✓ | Ibis passthrough | Notes | -|---|---|---|---|---|---|---|---| -| `database` (str \| Path \| None) | `DATABASE` (Optional[str]) | present | core | ✓ | ✓ (both default `None`) | ✓ | Our `get_connection_string_params()` returns `database=UPath(...).expanduser()`. Ibis accepts `str \| Path`. | -| `timeout` (float, 5.0) | — | missing | advanced | N/A | N/A | ✗ | Not in Ibis `do_connect()`; would have to be set via `sqlite3.connect()` directly. Low priority for our workloads. | -| `detect_types` (int, 0) | — | missing | advanced | N/A | N/A | ✗ | Not in Ibis passthrough. Ibis handles type inference itself; adding this would bypass Ibis type handling. | -| `isolation_level` (str \| None, "DEFERRED") | — | missing | advanced | N/A | N/A | ✗ | Ibis manages transactions internally. | -| `check_same_thread` (bool, True) | — | missing | advanced | N/A | N/A | ✗ | Multi-thread use would route via Ibis connection cloning, not this flag. | -| `factory` (Connection subclass) | — | missing | advanced | N/A | N/A | ✗ | Power-user knob; unlikely to need. | -| `cached_statements` (int, 128) | — | missing | advanced | N/A | N/A | ✗ | Performance tuning; no current need. | -| `uri` (bool, False) | — | missing | advanced | N/A | N/A | ✗ | Would enable `file:...?mode=...` connection strings. Potentially useful for read-only/shared-cache DBs. | -| `autocommit` (bool, LEGACY_TRANSACTION_CONTROL) | — | missing | advanced | N/A | N/A | ✗ | Modern transaction control; Ibis does not expose it. | -| `type_map` (dict[str, str \| dt.DataType] \| None) | `TYPE_MAP` (Optional[Dict[str, Any]]) | mismatch | advanced | ✗ | ✓ (both `None`) | ✓ | **Type drift:** ours is `Dict[str, Any]`; Ibis expects values of `str \| ibis.expr.datatypes.DataType`. Passing arbitrary `Any` silently accepts invalid type specs until runtime. | -| — | `HOST` (base) | extra | N/A | N/A | N/A | ✗ | SQLite is file-based; HOST is meaningless here. Not passed through. | -| — | `PORT` (base) | extra | N/A | N/A | N/A | ✗ | SQLite has no network port. | -| — | `SCHEMA` (base) | extra | N/A | N/A | N/A | ✗ | SQLite has no schema concept (single `main` schema + ATTACH). | -| — | `USERNAME` (base) | extra | N/A | N/A | N/A | ✗ | SQLite has no user auth. | -| — | `PASSWORD` (base) | extra | N/A | N/A | N/A | ✗ | SQLite has no user auth. | -| — | `TOKEN` (base) | extra | N/A | N/A | N/A | ✗ | SQLite has no token auth. | -| — | `AUTH_METHOD` (base, default `"none"` overridden in subclass) | extra | N/A | N/A | N/A | ✗ | Subclass correctly sets `"none"`, but the field is still inherited — no runtime harm. | - -## Findings narrative - -**Core gaps:** None. The single core parameter (`database`) is present with the correct type and default, and Ibis passes it through. - -**Core mismatches:** None. - -**Advanced gaps:** All eight non-`database` stdlib kwargs (`timeout`, `detect_types`, `isolation_level`, `check_same_thread`, `factory`, `cached_statements`, `uri`, `autocommit`) are absent. None of them are passthroughs via Ibis `do_connect()`, so adding them to our settings class would have no effect without also bypassing Ibis — not worthwhile. The only one that plausibly could matter operationally is **`uri=True`** (to allow `file:...?mode=ro` read-only SQLite attachments); this would require either bypassing Ibis or upstreaming to Ibis. - -**Advanced mismatches:** `TYPE_MAP` type drift — ours is `Dict[str, Any]`, Ibis expects `dict[str, str | dt.DataType]`. The `Any` is permissive; fixing it would surface misconfiguration at construction time. - -**Extras:** Seven base-class fields (HOST, PORT, SCHEMA, USERNAME, PASSWORD, TOKEN, AUTH_METHOD) bleed into this class by inheritance. None are used by SQLite. They don't break anything, but they clutter config files and let users set meaningless values without warning. This is a base-class composition issue, not a sqlite-specific bug. - -**Stale links:** None — both spec URLs return 200. - -## Recommended follow-ups - -- Tighten `TYPE_MAP` field type from `Dict[str, Any]` to `Dict[str, str | ibis.expr.datatypes.DataType]` (or a narrower union of primitive dtype names). -- Consider a validator on `SQLiteAuthSettings` that warns if any of HOST/PORT/SCHEMA/USERNAME/PASSWORD/TOKEN are set, since SQLite ignores them. (Alternatively, tackle the broader base-class shape in a separate design.) -- If operational need emerges for read-only attachment of SQLite files, investigate upstreaming a `uri=True` passthrough into Ibis, or expose it via a post-connect `ATTACH DATABASE 'file:...?mode=ro'` helper rather than through `sqlite3.connect()` kwargs. -- No changes needed for `DATABASE`. diff --git a/docs/superpowers/specs/2026-04-15-settings-audit/trino.md b/docs/superpowers/specs/2026-04-15-settings-audit/trino.md deleted file mode 100644 index 279d6e4..0000000 --- a/docs/superpowers/specs/2026-04-15-settings-audit/trino.md +++ /dev/null @@ -1,90 +0,0 @@ -# Trino Settings Audit - -## Header - -- **Backend:** trino -- **Date checked:** 2026-04-15 -- **Our settings class:** `src/mountainash_data/core/settings/trino.py` (`TrinoAuthSettings`) -- **Ibis backend file:** `/home/nathanielramm/git/github/ibis-dev/ibis/ibis/backends/trino/__init__.py` -- **Spec URLs (precedence-tagged):** - - **Driver (authoritative):** https://github.com/trinodb/trino-python-client/blob/master/trino/dbapi.py (`trino.dbapi.Connection.__init__`) - - **Driver user guide (supplementary):** https://github.com/trinodb/trino-python-client - -## Stale-link check - -| URL | Status | -|---|---| -| https://github.com/trinodb/trino-python-client/blob/master/trino/dbapi.py | OK | -| https://github.com/trinodb/trino-python-client | OK | - -## Summary counts - -- Core missing: **0** -- Core mismatch: **5** (`AUTH` type; `HTTP_SCHEME` default; `PORT` default; `VERIFY` type narrowed; `PASSWORD` wiring broken) -- Advanced missing: **1** (`encoding`) -- Advanced mismatch: **8** (`SESSION_PROPERTIES`, `HTTP_HEADERS`, `HTTP_SESSION`, `EXTRA_CREDENTIAL`, `CLIENT_TAGS`, `ROLES`, `ISOLATION_LEVEL`, `LEGACY_PREPARED_STATEMENTS` — all typed `Optional[str]` instead of native types) -- Advanced (other): **all advanced fields silently dropped by `get_connection_kwargs()` — only SOURCE, HTTP_SCHEME, PASSWORD are plumbed** -- Extra: **1** (`AUTH_METHOD`, local selector not a driver kwarg) -- Total audited: **~23** -- Stale links: **0** - -## Parameter table - -Legend — Status: `present` / `missing` / `mismatch` / `extra`. Tier: `core` / `advanced`. Type/Default: ✓ / ✗ / N/A. Ibis passthrough: ✓ / ✗ / unknown. - -| Spec name | Our field | Status | Tier | Type ✓ | Default ✓ | Ibis passthrough | Notes | -|---|---|---|---|---|---|---|---| -| `host` (str) | `HOST` (base, Optional[str]) | present | core | ✓ | N/A | ✓ | Driver requires it; Ibis positional. | -| `port` (int) | `PORT` (base, Optional[int]) | mismatch | core | ✓ | ✗ | ✓ | Ibis default `8080`; ours has no Trino-specific default. Users forgetting to set PORT get None routed through connection string. | -| `user` (str) | `USERNAME` (base) | present | core | ✓ | ✓ | ✓ | Ibis default `"user"`; ours None. | -| `catalog` (str) | `CATALOG` | present | core | ✓ | ✓ | ✓ | | -| `schema` (str) | `SCHEMA` (override of base) | present | core | ✓ | ✓ | ✓ | | -| `http_scheme` (str, None) | `HTTP_SCHEME` (default `"https"`) | mismatch | core | ✓ | ✗ | ✓ | Opinionated default. Fine in practice — most production Trino is HTTPS — but document it. | -| `auth` (trino.auth.Authentication) | `AUTH` (Optional[str]) | mismatch | core | ✗ | ✓ | ✓ | **Type wrong**: driver expects a `trino.auth.*` instance (e.g. `BasicAuthentication`, `JWTAuthentication`, `OAuth2Authentication`, `KerberosAuthentication`). Our `Optional[str]` cannot carry one. Settings layer needs a factory that maps `AUTH_METHOD` + credentials → auth instance. | -| `verify` (bool \| str path) | `VERIFY` (Optional[bool], default True) | mismatch | core | ✗ | ✓ | ✓ | Driver accepts `bool` OR a path string to a CA bundle. Ours narrows to bool only; users with custom CA bundles are stuck. | -| `source` (str) | `SOURCE` | present | advanced | ✓ | ✓ | ✓ | Driver default `DEFAULT_SOURCE`; ours None. | -| `session_properties` (dict) | `SESSION_PROPERTIES` (Optional[str]) | mismatch | advanced | ✗ | ✓ | ✓ | Driver expects a dict; string won't deserialize. Also **not plumbed** by `get_connection_kwargs()`. | -| `http_headers` (dict) | `HTTP_HEADERS` (Optional[str]) | mismatch | advanced | ✗ | ✓ | ✓ | Driver expects a dict. **Not plumbed**. | -| `http_session` (requests.Session) | `HTTP_SESSION` (Optional[str]) | mismatch | advanced | ✗ | ✓ | ✓ | Driver expects a `requests.Session` instance. **Not plumbed**. | -| `extra_credential` (List[Tuple[str,str]]) | `EXTRA_CREDENTIAL` (Optional[str]) | mismatch | advanced | ✗ | ✓ | ✓ | **Not plumbed**. | -| `max_attempts` (int) | `MAX_ATTEMPTS` (Optional[int]) | present | advanced | ✓ | ✓ | ✓ | **Not plumbed** by `get_connection_kwargs()`. | -| `request_timeout` (float) | `REQUEST_TIMEOUT` (Optional[int]) | mismatch | advanced | ✗ | ✓ | ✓ | Driver accepts float seconds; int likely OK but narrowing is suboptimal. **Not plumbed**. | -| `isolation_level` (IsolationLevel enum) | `ISOLATION_LEVEL` (Optional[str]) | mismatch | advanced | ✗ | ✓ | ✓ | Driver expects enum. **Not plumbed**. | -| `client_tags` (List[str]) | `CLIENT_TAGS` (Optional[str]) | mismatch | advanced | ✗ | ✓ | ✓ | Driver expects list. **Not plumbed**. | -| `legacy_primitive_types` (bool) | `LEGACY_PRIMITIVE_TYPES` (Optional[bool]) | present | advanced | ✓ | ✓ | ✓ | **Not plumbed**. | -| `legacy_prepared_statements` (Optional[bool]) | `LEGACY_PREPARED_STATEMENTS` (Optional[str]) | mismatch | advanced | ✗ | ✓ | ✓ | Driver expects Optional[bool]; ours is Optional[str]. **Not plumbed**. | -| `roles` (dict \| str) | `ROLES` (Optional[str]) | mismatch | advanced | ✗ | ✓ | ✓ | Driver accepts dict (per-catalog) or str. **Not plumbed**. | -| `timezone` (str) | `TIMEZONE` (Optional[str]) | present | advanced | ✓ | ✓ | ✓ | **Not plumbed**. | -| `encoding` (str \| List[str]) | — | missing | advanced | N/A | N/A | ✓ | Spooling protocol parameter; not modelled. | -| — | `AUTH_METHOD` (base override, default `None`) | extra | core | N/A | N/A | ✗ | Local selector used only to decide whether to pack `password` into kwargs; not a driver param. | -| — | `PASSWORD` (base) | extra/broken | core | ✗ | N/A | ✗ | `get_connection_kwargs()` sets `kwargs["password"] = self.PASSWORD` when `AUTH_METHOD == "password"`. **Driver has no `password=` kwarg.** Must be wrapped in `trino.auth.BasicAuthentication(user, password)` and passed via `auth=`. **Current wiring will fail at connect time.** | - -## Findings narrative - -**Core gaps:** None (all essential connection parameters are declared as Fields, though not all are plumbed). - -**Core mismatches:** - -1. **`PASSWORD` wiring is broken.** `get_connection_kwargs()` passes `password` as a bare kwarg, but `trino.dbapi.Connection.__init__` has no `password` parameter. The driver's documented pattern is `auth=BasicAuthentication(user, password)`. This path has likely never been exercised successfully. -2. **`AUTH` typed as `Optional[str]`** but must be a `trino.auth.*` instance. The settings layer needs a mapping from `AUTH_METHOD` (+ credentials / keytab / keystore paths) to an auth object. -3. **`HTTP_SCHEME` defaults to `"https"`** rather than None. Opinionated but reasonable — document it. -4. **`PORT` has no Trino default** (Ibis uses 8080). -5. **`VERIFY` narrows** `bool | str` to `Optional[bool]` — users with a custom CA bundle can't configure it. - -**Advanced mismatches:** Eight fields are typed `Optional[str]` where the driver expects dicts, lists, or enum values. On top of that, **`get_connection_kwargs()` only forwards `SOURCE`, `HTTP_SCHEME`, and `PASSWORD`** — every other advanced field (MAX_ATTEMPTS, REQUEST_TIMEOUT, TIMEZONE, SESSION_PROPERTIES, HTTP_HEADERS, HTTP_SESSION, EXTRA_CREDENTIAL, CLIENT_TAGS, ROLES, ISOLATION_LEVEL, LEGACY_PRIMITIVE_TYPES, LEGACY_PREPARED_STATEMENTS, VERIFY) is declared, validated, and then silently dropped. - -**Advanced gaps:** `encoding` (spooling protocol) is unmodelled. - -**Stale links:** None. Note the user guide link points to the repo root rather than a dedicated "Connection parameters" page — consider referencing `dbapi.py` directly as primary. - -## Recommended follow-ups - -- **Fix password auth wiring first** — this is blocking real use. Build `trino.auth.BasicAuthentication(USERNAME, PASSWORD)` when `AUTH_METHOD == "password"` and pass as `auth=`. -- **Introduce an auth adapter**: retype `AUTH` to an auth-instance union (or carry it through as a computed property) and add per-method fields (e.g. `KERBEROS_SERVICE_NAME`, `JWT_TOKEN`, `OAUTH_TOKEN`) that the adapter composes. -- **Plumb every advanced Field through `get_connection_kwargs()`** — the current class declares many options but forwards only three. -- **Retype structured fields**: `SESSION_PROPERTIES` → `Dict[str, str]`; `HTTP_HEADERS` → `Dict[str, str]`; `EXTRA_CREDENTIAL` → `List[Tuple[str, str]]`; `CLIENT_TAGS` → `List[str]`; `ROLES` → `Dict[str, str] | str`; `ISOLATION_LEVEL` → enum; `LEGACY_PREPARED_STATEMENTS` → `Optional[bool]`. -- **Widen `VERIFY`** to `Optional[bool | str]`. -- **Add `ENCODING`** field (`Optional[str | List[str]]`). -- **Set `PORT` default** to 8080 for Trino or document why it's None. -- **Document `HTTP_SCHEME="https"` opinion** in the class docstring. -- Replace the user guide URL in the docstring with `https://github.com/trinodb/trino-python-client/blob/master/trino/dbapi.py` as the authoritative reference. diff --git a/docs/superpowers/specs/2026-04-15-settings-registry-design.md b/docs/superpowers/specs/2026-04-15-settings-registry-design.md deleted file mode 100644 index 6757eeb..0000000 --- a/docs/superpowers/specs/2026-04-15-settings-registry-design.md +++ /dev/null @@ -1,446 +0,0 @@ -# Settings Registry Redesign — Design Spec - -**Date:** 2026-04-15 -**Scope:** `src/mountainash_data/core/settings/` — the 11 per-backend settings classes + their shared base. -**Input from:** `docs/superpowers/specs/2026-04-15-settings-audit/` (per-backend findings, cross-cutting patterns). -**Goal:** Replace the current class-per-backend, method-heavy settings hierarchy with a declarative descriptor + thin subclass pattern. Preserve pydantic typing and `MountainAshBaseSettings` integration. Carry forward all audit fixes in the same pass. - ---- - -## Problem statement - -The audit surfaced three structural issues that sit underneath most of the per-backend bugs: - -1. **Boilerplate.** Every settings class repeats the same four methods (`get_connection_string_template`, `get_connection_string_params`, `get_connection_kwargs`, `get_post_connection_options`), the same `__init__` passthrough, the same `db_provider_type` property, and a hand-rolled `_post_init`. The repeated code is where the copy-paste bugs live (postgres and mysql both return `BIGQUERY`; pyspark's docstring says "SQLite authentication settings"). - -2. **Base-class leakage.** `BaseDBAuthSettings` declares `HOST`/`PORT`/`DATABASE`/`SCHEMA`/`USERNAME`/`PASSWORD`/`TOKEN`/`AUTH_METHOD` as inherited fields. These are meaningless for SQLite, DuckDB, PySpark, BigQuery, MotherDuck, PyIceberg REST — but they're silently accepted as config keys and can't be rejected. Every per-backend audit flagged this as noise. - -3. **Polymorphic auth is flattened.** Every backend with multiple auth modes (Snowflake, MSSQL, Redshift, Trino, PyIceberg REST) puts *every* possible auth field on the class as `Optional[...]` and uses an `AUTH_METHOD` string selector plus hand-rolled model validators to enforce "if method X then fields Y required." The result: users can set `PRIVATE_KEY` alongside `AUTH_METHOD=password` with no type error, and the same auth shape (OAuth2 client-credentials; password+SecretStr) is redefined in three or four places. - -The current design also makes it hard to generate documentation, produce tier-filtered views (core vs. advanced), or introspect what a backend supports — because the knowledge is distributed across Python class bodies rather than collected in data. - -## Goals - -- Eliminate the four `get_*` methods from the per-backend surface — they become responsibility of the generic base. -- Represent each backend as **data** (a `BackendDescriptor`) plus **optional code** (a small adapter for composite driver mapping). -- Model auth as a **pydantic discriminated union** of typed `AuthSpec` subclasses, reused across backends. -- Drop `HOST`/`PORT`/…/`AUTH_METHOD` base-class leakage. Backends declare what they need; backends that don't need those fields don't get them. -- Carry forward every applicable audit fix in the same refactor: broken wiring, wrong types, unused enums, stringly-typed fields, missing parameters. -- Preserve stable import paths (`from ...settings import PostgreSQLAuthSettings`) so downstream call sites that *do* import by name keep working. -- Preserve `MountainAshBaseSettings` integration (config-file loading + `SettingsParameters`). - -## Non-goals - -- Rewriting factories, `DatabaseUtils`, or the `backends/ibis/iceberg` layers beyond updating their call sites to the new profile API. -- Adding new backends beyond the 11 currently audited. -- Adding new auth modes beyond what the 11 backends actually use. -- Runtime introspection against live driver modules (e.g. walking `RestCatalog.__init__` for parameters) — that's a separate initiative. -- Backward compatibility with the four retired `get_*` methods or with `DBAuthValidationError`. Callers of those (factories, `DatabaseUtils`, backend connection classes) are updated in the same change set. - ---- - -## Architecture - -Layers, outside-in: - -1. **`ConnectionProfile`** — subclass of `MountainAshBaseSettings`. Defines the uniform public API: `profile.to_driver_kwargs()`, `profile.to_connection_string()`, `profile.backend`, `profile.auth`. No backend-specific methods. - -2. **`BackendDescriptor`** — immutable dataclass per backend, registered in a module-level `REGISTRY`. Carries `name`, `provider_type`, `default_port`, `parameters: list[ParameterSpec]`, `auth_modes: list[type[AuthSpec]]`, `connection_string_scheme`, `ibis_dialect`, optional `rides_on` (for MotherDuck → duckdb, Redshift → postgres). - -3. **`ParameterSpec`** — describes one field. Covers the 80% case (name, type, tier, default, optional `driver_key` for 1:1 driver mapping, optional `transform`, `secret` flag, optional field-level validator) without needing any adapter code. - -4. **`AuthSpec` hierarchy** — a family of small pydantic models, each with a `kind` discriminator literal: `NoAuth`, `PasswordAuth`, `TokenAuth`, `JWTAuth`, `OAuth2Auth`, `ServiceAccountAuth`, `IAMAuth`, `WindowsAuth`, `AzureADAuth`, `KerberosAuth`, `CertificateAuth`. Reused across backends. - -5. **Adapter module (optional per backend)** — `adapters/.py` exporting `build_driver_kwargs(profile) -> dict`. Handles composite mappings that don't fit `ParameterSpec`: `trino.auth.BasicAuthentication` wrapper, mysqlclient's nested `ssl={}` dict, BigQuery's SA-info → `Credentials` conversion, Redshift endpoint resolution, Snowflake `session_parameters` merge, PyIceberg REST `s3.*` key prefixing. Backends without composite needs have no adapter module. - -6. **Thin subclass per backend** — one file per backend (`postgresql.py`, `snowflake.py`, etc.) containing the `BackendDescriptor`, any backend-specific enums, an optional adapter function, and a two-line subclass shell. - -### Data flow - -``` -user kwargs / config file - ↓ - *AuthSettings subclass ← BackendDescriptor - (pydantic validation, auth - discrimination, SecretStr) - ↓ .to_driver_kwargs() - ConnectionProfile._default_driver_kwargs() (1:1 mappings from descriptor) - ↓ - AUTH_TO_DRIVER_KWARGS[type(auth)](auth, descriptor) (auth dispatch) - ↓ - adapter.build_driver_kwargs(profile) (composite overrides, if any) - ↓ - driver / Ibis do_connect(**kwargs) -``` - ---- - -## The data contract - -### `ParameterSpec` - -```python -@dataclass(frozen=True, kw_only=True) -class ParameterSpec: - name: str # settings-facing name, e.g. "SSL_CERT" - type: type | TypeAlias # pydantic-compatible annotation - tier: Literal["core", "advanced"] - default: Any = MISSING # MISSING → Field(...) (required) - description: str = "" - driver_key: str | None = None # driver kwarg name, e.g. "sslcert"; None = adapter-only - secret: bool = False # wrap as SecretStr, auto-unwrap at kwargs boundary - transform: Callable[[Any], Any] | None = None # inline 1:1 transform (Path→str, bool→"0"/"1", ...) - validator: Callable[[Any], Any] | None = None # pydantic-style field validator -``` - -`MISSING` is a module-level sentinel. When `default is MISSING`, the base emits `Field(...)` (required). Otherwise `Field(default=...)`. Setting `secret=True` automatically wraps the declared type as `SecretStr` and arranges for `.get_secret_value()` to be called at the `to_driver_kwargs` boundary — fixes the inconsistent secret handling flagged in the audit. - -### `AuthSpec` hierarchy - -Discriminated-union members, each in its own small module under `settings/auth/`: - -```python -class AuthSpec(BaseModel): - kind: str # discriminator - -class NoAuth(AuthSpec): kind: Literal["none"] = "none" -class PasswordAuth(AuthSpec): kind: Literal["password"] = "password" - username: str - password: SecretStr -class TokenAuth(AuthSpec): kind: Literal["token"] = "token" - token: SecretStr -class JWTAuth(AuthSpec): kind: Literal["jwt"] = "jwt" - token: SecretStr -class OAuth2Auth(AuthSpec): kind: Literal["oauth2"] = "oauth2" - client_id: str | None = None - client_secret: SecretStr | None = None - token: SecretStr | None = None - refresh_token: SecretStr | None = None - server_uri: str | None = None - scope: str | None = None -class ServiceAccountAuth(AuthSpec): kind: Literal["service_account"] = "service_account" - info: dict | None = None - file: Path | None = None -class IAMAuth(AuthSpec): kind: Literal["iam"] = "iam" - role_arn: str | None = None - access_key_id: str | None = None - secret_access_key: SecretStr | None = None - session_token: SecretStr | None = None - profile_name: str | None = None -class WindowsAuth(AuthSpec): kind: Literal["windows"] = "windows" - username: str | None = None - domain: str | None = None -class AzureADAuth(AuthSpec): kind: Literal["azure_ad"] = "azure_ad" - tenant_id: str | None = None - client_id: str | None = None - client_secret: SecretStr | None = None - managed_identity: bool = False - msi_endpoint: str | None = None -class KerberosAuth(AuthSpec): kind: Literal["kerberos"] = "kerberos" - service_name: str = "postgres" - principal: str | None = None - keytab: Path | None = None -class CertificateAuth(AuthSpec): kind: Literal["certificate"] = "certificate" - private_key: SecretStr | None = None - private_key_path: Path | None = None - passphrase: SecretStr | None = None -``` - -Backends declare which auth modes they accept via `BackendDescriptor.auth_modes`. Pydantic composes these into a discriminated union for the profile's `auth` field — wrong fields for the wrong mode become construction-time validation errors, not runtime surprises. - -**Backend → auth mapping (derived from the audit):** - -| Backend | Auth modes | -|---|---| -| sqlite, duckdb, pyspark | `NoAuth` | -| motherduck | `TokenAuth` | -| postgresql | `PasswordAuth`, `NoAuth` | -| mysql | `PasswordAuth` | -| mssql | `PasswordAuth`, `WindowsAuth`, `AzureADAuth` | -| snowflake | `PasswordAuth`, `OAuth2Auth`, `CertificateAuth`, `TokenAuth` | -| bigquery | `ServiceAccountAuth`, `NoAuth` (ADC) | -| redshift | `PasswordAuth`, `IAMAuth` | -| trino | `PasswordAuth`, `JWTAuth`, `KerberosAuth`, `OAuth2Auth`, `NoAuth` | -| pyiceberg_rest | `TokenAuth`, `OAuth2Auth` | - -### `BackendDescriptor` - -```python -@dataclass(frozen=True, kw_only=True) -class BackendDescriptor: - name: str # "postgresql", "pyiceberg_rest", ... - provider_type: CONST_DB_PROVIDER_TYPE # authoritative provider identifier - default_port: int | None = None - parameters: list[ParameterSpec] - auth_modes: list[type[AuthSpec]] - connection_string_scheme: str | None = None # "postgresql://", "bigquery://", ...; None if N/A - ibis_dialect: str | None = None # "postgres", "duckdb", None (for pyiceberg_rest) - rides_on: str | None = None # metadata only: "duckdb" for motherduck, "postgres" for redshift -``` - -`rides_on` is **metadata only** — it records that the backend routes through another backend's Ibis `do_connect()` (motherduck→duckdb, redshift→postgres). It has no runtime behavior: each backend still has its own descriptor and own `to_driver_kwargs()`. Consumers that care (e.g., audit tooling, docs generation) can read the field. - -`connection_string_scheme` may be `None` for backends that don't use a URL form (e.g., `pyspark` uses a `SparkSession`; `pyiceberg_rest` takes `uri=` as a kwarg). When `None`, the base `to_connection_string()` raises `NotImplementedError` — backends that need URL construction override the method on their subclass. - ---- - -## `ConnectionProfile` base - -```python -class ConnectionProfile(MountainAshBaseSettings): - __descriptor__: ClassVar[BackendDescriptor] - __adapter__: ClassVar[Callable[["ConnectionProfile"], dict] | None] = None - - auth: AuthSpec # union; pydantic discriminator="kind" - - @property - def backend(self) -> str: - return self.__descriptor__.name - - @property - def provider_type(self) -> CONST_DB_PROVIDER_TYPE: - return self.__descriptor__.provider_type - - def _default_driver_kwargs(self) -> dict: - """1:1 driver mappings from the descriptor. - Skip None values. Unwrap SecretStr. Apply ParameterSpec.transform.""" - ... - - def _auth_to_driver_kwargs(self) -> dict: - """Dispatch on type(self.auth) via AUTH_TO_DRIVER_KWARGS default map.""" - ... - - def to_driver_kwargs(self) -> dict: - kwargs = self._default_driver_kwargs() - kwargs.update(self._auth_to_driver_kwargs()) - if self.__adapter__ is not None: - kwargs = self.__adapter__(self) - return kwargs - - def to_connection_string(self) -> str: - """Build from descriptor.connection_string_scheme + 1:1 params. - Backends with non-standard shapes override this method on their subclass.""" - ... -``` - -At class-creation time (via a model-class decorator or `__init_subclass__`), the base reads `__descriptor__` and configures pydantic fields: one per `ParameterSpec`, plus the discriminated-union `auth` field assembled from `descriptor.auth_modes`. The result is a normal pydantic model — no runtime `create_model`, no dynamic attribute surgery — just a concise declarative path into the same machinery pydantic uses today. - ---- - -## Adapter layer - -**Default auth dispatch** — `settings/auth/dispatch.py` exports `AUTH_TO_DRIVER_KWARGS: dict[type[AuthSpec], Callable[[AuthSpec, BackendDescriptor], dict]]`. Example entries: - -- `PasswordAuth` → `{"user": auth.username, "password": auth.password.get_secret_value()}` -- `TokenAuth` → `{"token": auth.token.get_secret_value()}` -- `OAuth2Auth` → `{"token": auth.token.get_secret_value()}` if `token` set, else `{"credential": f"{id}:{secret}"}` -- `IAMAuth` → `{"aws_access_key_id": ..., "aws_secret_access_key": ..., "aws_session_token": ...}` (skip None) -- `NoAuth` → `{}` - -**Backend adapters override when the driver expects non-standard shapes:** - -```python -# adapters/trino.py -from trino.auth import BasicAuthentication, JWTAuthentication, KerberosAuthentication - -def build_driver_kwargs(profile): - kwargs = profile._default_driver_kwargs() - match profile.auth: - case PasswordAuth(username=u, password=pw): - kwargs["user"] = u - kwargs["auth"] = BasicAuthentication(u, pw.get_secret_value()) - case JWTAuth(token=t): - kwargs["auth"] = JWTAuthentication(t.get_secret_value()) - case KerberosAuth() as k: - kwargs["auth"] = KerberosAuthentication( - config=None, service_name=k.service_name, - principal=k.principal, ...) - case NoAuth(): - pass - return kwargs -``` - -```python -# adapters/mysql.py -def build_driver_kwargs(profile): - kwargs = profile._default_driver_kwargs() - ssl = {k: v for k, v in { - "ssl-key": profile.SSL_KEY, - "ssl-cert": profile.SSL_CERT, - "ssl-ca": profile.SSL_CA, - "ssl-capath": profile.SSL_CAPATH, - "ssl-cipher": profile.SSL_CIPHER, - }.items() if v is not None} - if ssl: - kwargs["ssl"] = ssl - return kwargs -``` - -**Backends with no adapter:** `sqlite`, `duckdb`, `motherduck`, `postgresql` (the 1:1 libpq mapping is complete without one), `pyspark`. - -**Backends with adapters:** `mysql` (`ssl={}`), `mssql` (Encrypt/TrustServerCertificate, `server\instance`), `snowflake` (session_parameters, authenticator mapping), `bigquery` (SA-info → `Credentials`), `redshift` (endpoint resolution from CLUSTER_IDENTIFIER/WORKGROUP_NAME), `trino` (auth wrapper objects), `pyiceberg_rest` (`s3.*` / `rest.sigv4-*` key prefixing). - ---- - -## File layout - -``` -src/mountainash_data/core/settings/ -├── __init__.py # re-exports the 11 *AuthSettings classes -├── profile.py # ConnectionProfile -├── descriptor.py # BackendDescriptor, ParameterSpec, MISSING -├── registry.py # REGISTRY, @register, lookup helpers -├── auth/ -│ ├── __init__.py # exports all AuthSpec subclasses + union alias -│ ├── base.py # AuthSpec ABC -│ ├── password.py -│ ├── token.py # TokenAuth, JWTAuth -│ ├── oauth2.py -│ ├── iam.py -│ ├── azure.py # AzureADAuth, WindowsAuth -│ ├── cloud.py # ServiceAccountAuth -│ ├── kerberos.py -│ ├── certificate.py -│ ├── none.py -│ └── dispatch.py # AUTH_TO_DRIVER_KWARGS default map -├── adapters/ -│ ├── __init__.py -│ ├── trino.py -│ ├── mssql.py -│ ├── mysql.py -│ ├── snowflake.py -│ ├── bigquery.py -│ ├── redshift.py -│ └── pyiceberg_rest.py -├── sqlite.py # descriptor + shell class -├── duckdb.py -├── motherduck.py -├── postgresql.py -├── mysql.py -├── mssql.py -├── snowflake.py -├── bigquery.py -├── redshift.py -├── pyspark.py -├── trino.py -└── pyiceberg_rest.py -``` - -`base.py` and `exceptions.py` are **deleted**. `BaseDBAuthSettings` goes away; `DBAuthValidationError` is replaced by `ValueError` / pydantic's built-in `ValidationError`. If any external caller catches `DBAuthValidationError`, we add a one-line shim aliasing it to `ValueError`. - -### Per-backend file template - -```python -# postgresql.py -from enum import StrEnum -from pathlib import Path -from .descriptor import BackendDescriptor, ParameterSpec, MISSING -from .profile import ConnectionProfile -from .auth import PasswordAuth, NoAuth -from .registry import register -from ..constants import CONST_DB_PROVIDER_TYPE - -class PostgresSSLMode(StrEnum): - DISABLE = "disable" - ALLOW = "allow" - PREFER = "prefer" - REQUIRE = "require" - VERIFY_CA = "verify-ca" - VERIFY_FULL = "verify-full" - -# (other PG enums: target_session_attrs, require_auth methods, sslcertmode) - -POSTGRESQL_DESCRIPTOR = BackendDescriptor( - name="postgresql", - provider_type=CONST_DB_PROVIDER_TYPE.POSTGRESQL, - default_port=5432, - connection_string_scheme="postgresql://", - ibis_dialect="postgres", - auth_modes=[PasswordAuth, NoAuth], - parameters=[ - ParameterSpec(name="HOST", type=str, tier="core", default=MISSING, driver_key="host"), - ParameterSpec(name="DATABASE", type=str | None, tier="core", default=None, driver_key="database"), - ParameterSpec(name="SCHEMA", type=str | None, tier="core", default=None, driver_key="schema"), - ParameterSpec(name="SSL_MODE", type=PostgresSSLMode, tier="core", - default=PostgresSSLMode.PREFER, driver_key="sslmode"), - ParameterSpec(name="SSL_CERT", type=Path | None, tier="advanced", - default=None, driver_key="sslcert"), - ParameterSpec(name="CONNECT_TIMEOUT", type=int | None, tier="advanced", - default=None, driver_key="connect_timeout"), - # ... rest of the libpq surface, widened per the audit ... - ], -) - -@register(POSTGRESQL_DESCRIPTOR) -class PostgreSQLAuthSettings(ConnectionProfile): - __descriptor__ = POSTGRESQL_DESCRIPTOR -``` - -Two lines of actual class body. The rest is declarative data. - ---- - -## Testing - -1. **Descriptor-driven parametric tests** (`tests/test_unit/core/settings/test_descriptors.py`). Iterate over `REGISTRY`, assert invariants every backend must satisfy: - - Parameter `name`s unique within descriptor. - - `driver_key`s unique within descriptor (where non-None). - - `auth_modes` contains only `AuthSpec` subclasses. - - `provider_type` matches a known `CONST_DB_PROVIDER_TYPE` member. - - `tier ∈ {"core", "advanced"}`. - - `name == descriptor.name` matches the `@register` key. - - These would have caught the `db_provider_type returns BIGQUERY` copy-paste automatically. - -2. **Round-trip tests per backend.** One test file per backend with: - - Minimal construction (required fields only) succeeds. - - `to_driver_kwargs()` returns the expected keys/types — secrets unwrapped to raw strings, enums serialized to values. - - Each declared auth mode constructs and maps to driver kwargs correctly. - - `to_connection_string()` produces a syntactically-correct URL for the scheme. - -3. **Audit-regression tests.** For each bug fix the refactor carries in from the audit, one test that would have caught the original defect (Snowflake `"snowflake "` trailing whitespace rejection, Trino `auth=BasicAuthentication(...)` vs bare `password=` kwarg, PG `db_provider_type`, MSSQL `args["server"]` KeyError, etc.). Each lands with its backend's migration commit. - -4. **Connection smoke tests.** SQLite + DuckDB in-memory already run in CI — they continue to run unchanged. Other backends stay mocked. - ---- - -## Migration path - -**Phase 1 — scaffolding:** -- `profile.py`, `descriptor.py`, `registry.py`. -- `auth/` with all `AuthSpec` members + `dispatch.py`. -- Parametric descriptor-invariant tests (no backend involved yet). - -**Phase 2 — migrate backends one at a time,** in this order (cheapest → hardest): -- SQLite, DuckDB, PySpark, MotherDuck (no auth or token-only; small surface). -- PostgreSQL, MySQL, Trino. -- MSSQL, Snowflake. -- BigQuery, Redshift, PyIceberg REST. - -Each backend migration is **one commit**: the new file (descriptor + shell class + any adapter) + its tests, delete the old file, update `__init__.py` re-exports, run the full test suite. The old and new class never coexist — we flip one at a time. - -**Phase 3 — consumer update + cleanup:** -- Delete `base.py`, `exceptions.py`, now-dead helpers. -- Update `SettingsFactory`, `ConnectionFactory`, `DatabaseUtils` to call `to_driver_kwargs` / `to_connection_string`. -- Update `backends/ibis/connection.py` and `backends/iceberg/connection.py` to consume the new profile API. -- Update `README.md` and any user-facing docs. - -**Phase 4 — audit-fix sweep:** -- For any audit findings not already handled during Phase 2 translation (e.g. parameter surface widening that would have bloated a translation commit), a dedicated per-backend pass. Each is a small commit. - ---- - -## Out of scope - -- Rewriting factories, `DatabaseUtils`, or `backends/` beyond call-site updates. -- Adding new backends. -- Adding new `AuthSpec` members beyond those required by the current 11 backends. (No speculative `SSHTunnelAuth`, no `DatabaseCredentialsAuth`.) -- Runtime introspection of driver modules (`RestCatalog.__init__`, `snowflake.connector.connect` signature walking). Separate initiative. -- Maintaining backward compatibility for the retired `get_connection_string_template`/`get_connection_string_params`/`get_connection_kwargs`/`get_post_connection_options` methods. They have no external consumers outside factories we also update. - -## Risks and mitigations - -- **Pydantic discriminated-union edge cases** with the `MountainAshBaseSettings` config-file loader. Mitigation: an early Phase-1 spike with a representative auth-multi-mode backend (MSSQL or Snowflake) confirms config-file YAML/ENV maps cleanly to tagged-union construction before we scale out. -- **Parameter surface drift.** The audit widens many backends' surfaces. A too-ambitious Phase-2 commit per backend could balloon. Mitigation: Phase 4 is explicitly reserved for widening; Phase 2 commits translate 1:1 and keep scope tight. Reviewers can compare old class vs. new descriptor parameter-by-parameter. -- **Secret handling regressions.** Several backends currently pass SecretStr objects directly (we flagged this across the audit). The new `ParameterSpec.secret=True` centralizes `.get_secret_value()` — but only at the `to_driver_kwargs` boundary. Mitigation: round-trip tests assert raw strings come out of `to_driver_kwargs`, not `SecretStr` objects. -- **`DBAuthValidationError` shim removal.** If external callers catch it, deleting the class breaks them. Mitigation: one-line `DBAuthValidationError = ValueError` alias in a deprecation shim module; delete in a later release. diff --git a/docs/superpowers/specs/2026-04-26-legacy-cleanup-final.md b/docs/superpowers/specs/2026-04-26-legacy-cleanup-final.md deleted file mode 100644 index e54175c..0000000 --- a/docs/superpowers/specs/2026-04-26-legacy-cleanup-final.md +++ /dev/null @@ -1,113 +0,0 @@ -# Legacy Cleanup — Final Two Items - -> **Date:** 2026-04-26 -> **Status:** Approved -> **Backlog ref:** `mountainash-central/01.principles/mountainash-data/f.backlog/legacy-cleanup.md` - -## Context - -The April 2026 settings-registry refactor migrated all backends to -`ConnectionProfile` + `BackendDescriptor`. Two legacy artefacts remain: - -1. Four deprecated bridge methods on `BaseDBConnection` and their callers -2. A redundant `isinstance(pw, SecretStr)` guard in `ConnectionProfile.to_connection_string()` - -All downstream consumers now use `ConnectionProfile`. No `BaseDBAuthSettings` -subclasses remain in production. - ---- - -## Item 1 — Remove bridge methods and legacy connection branches - -### What to delete - -**`core/connection.py`** — remove these four methods from `BaseDBConnection`: - -| Method | Lines (approx) | -|--------|----------------| -| `get_connection_string_template()` | 135-156 | -| `get_connection_string_params()` | 158-175 | -| `get_connection_kwargs()` | 177-194 | -| `format_connection_string()` | 196-216 | - -Also remove the commented-out `format_connection_string` block below (lines 220+). - -**`backends/ibis/connection.py`** — in `BaseIbisConnection.connect_default()`: - -- Delete the legacy fallback branch (lines 142-164) that calls the bridge - methods. The `ConnectionProfile` path (lines 116-140) already returns before - reaching this code. -- The `if isinstance(obj_settings, ConnectionProfile)` guard on line 116 can - become unconditional — just call `to_driver_kwargs()` directly on `obj_settings`. - -**`backends/iceberg/connection.py`** — in `IcebergConnectionBase.connect_default()`: - -- Delete the `isinstance` dispatch (lines 113-116). Call - `obj_settings.to_driver_kwargs()` unconditionally. - -### What to keep - -- `BaseDBConnection` itself — still the abstract base for Ibis/Iceberg connections. -- The `connect_default()` methods — just cleaned of the legacy branches. - -### Tests - -- Existing unit tests for Ibis and Iceberg connections should continue to pass - (they use `ConnectionProfile`-based settings). -- Grep for any test that calls the bridge methods directly and remove/update. - ---- - -## Item 2 — Remove redundant SecretStr guard - -### What to change - -**`core/settings/profile.py`** — in `ConnectionProfile.to_connection_string()`: - -Replace: -```python -pw = getattr(auth, "password", None) -if isinstance(pw, SecretStr): - url += ":" + quote(pw.get_secret_value(), safe="") -``` - -With: -```python -pw = getattr(auth, "password", None) -if pw is not None: - url += ":" + quote(pw.get_secret_value(), safe="") -``` - -### Why this is safe - -`PasswordAuth.password` is typed as `SecretStr` in upstream `mountainash-settings`. -With `validate_assignment=True` enabled (since v26.4.1), pydantic enforces the -type — a raw string can never sneak into the field. The `isinstance` guard was -protecting against a `__setattr__` bypass path that no longer exists. - -### Cleanup - -Remove `from pydantic import SecretStr` from `profile.py` if it becomes the -only consumer. - -### Tests - -- Existing `to_connection_string()` tests should pass unchanged. -- No new tests needed — behaviour is identical for all valid inputs. - ---- - -## Commit strategy - -Two separate commits, one per item: - -1. `chore(connection): remove deprecated bridge methods and legacy fallback branches` -2. `chore(settings): replace SecretStr isinstance guard with None check` - -Single branch, single PR targeting `develop`. - -## Backlog update - -After merge, update `legacy-cleanup.md`: -- Item 1 → **RESOLVED** with PR reference -- Item 2 → **RESOLVED** with PR reference diff --git a/docs/superpowers/specs/2026-04-26-to-relation-design.md b/docs/superpowers/specs/2026-04-26-to-relation-design.md deleted file mode 100644 index c11e8c8..0000000 --- a/docs/superpowers/specs/2026-04-26-to-relation-design.md +++ /dev/null @@ -1,210 +0,0 @@ -# Settings-Aware Backends + to_relation() - -> **Date:** 2026-04-26 (updated 2026-04-27) -> **Status:** ABANDONED -- superseded by `2026-04-27-settings-aware-ibis-backend-design.md`. to_relation() descoped per revised principle. -> **Backlog refs:** -> - `mountainash-central/01.principles/mountainash-data/f.backlog/to-relation-gap.md` -> - `mountainash-central/01.principles/mountainash-data/f.backlog/settings-aware-backends.md` (Phase 1) - -## Context - -mountainash-data has two parallel connection paths that produce the same -result with different indirection: - -1. **New-style:** `IbisBackend(dialect="sqlite", **config).connect()` → - `IbisConnection` (protocol-compliant) -2. **Settings-driven:** `ConnectionFactory.get_connection(settings_params)` → - `BaseIbisConnection` subclass (old hierarchy) - -The gap: `IbisBackend` only accepts direct config. Consumers with a -`SettingsParameters` must use the old factory path or manually resolve -settings. This keeps the factory alive despite adding no value post-refactor. - -Separately, `to_relation()` is defined on the `Connection` protocol concept -but not yet implemented. The `mountainash` package (formerly -mountainash-expressions) provides `relation()` which accepts ibis tables -and wraps them in a `Relation` AST node. - -This spec covers: -- Making `IbisBackend` settings-aware (Phase 1 of settings-aware-backends backlog) -- Wiring `to_relation()` on both connection paths - -## Design - -### 1. Settings-aware IbisBackend - -`IbisBackend.__init__` accepts either a dialect string + kwargs (existing) -or a `SettingsParameters` object (new): - -```python -class IbisBackend: - name = "ibis" - - def __init__(self, dialect_or_settings: str | SettingsParameters, **config: Any): - if isinstance(dialect_or_settings, SettingsParameters): - settings_params = dialect_or_settings - settings = settings_params.settings_class.get_settings(settings_params) - descriptor = settings.__descriptor__ - ibis_dialect = descriptor.ibis_dialect - if ibis_dialect not in DIALECTS: - raise KeyError( - f"Unknown ibis dialect {ibis_dialect!r}. " - f"Available: {sorted(DIALECTS)}" - ) - self.dialect = ibis_dialect - self._spec = DIALECTS[ibis_dialect] - self._config = settings.to_driver_kwargs() - else: - dialect = dialect_or_settings - if dialect not in DIALECTS: - raise KeyError( - f"Unknown ibis dialect {dialect!r}. " - f"Available: {sorted(DIALECTS)}" - ) - self.dialect = dialect - self._spec = DIALECTS[dialect] - self._config = config -``` - -Usage: - -```python -from mountainash_data import IbisBackend - -# Direct -backend = IbisBackend(dialect="sqlite", database=":memory:") - -# Settings-driven -backend = IbisBackend(settings_params) - -# Both produce the same IbisConnection -conn = backend.connect() -``` - -### 2. Protocol — to_relation() - -Add `to_relation()` to the `Connection` protocol in `core/protocol.py`: - -```python -def to_relation(self, name: str, namespace: str | None = None) -> "Relation": - """Return a mountainash Relation for the named table. - - Requires the mountainash package. Raises ImportError if not installed. - """ - ... -``` - -Signature matches `inspect_table(name, namespace)` for consistency. Return -type uses a string forward reference to avoid importing mountainash at module -level. - -### 3. Ibis implementation — new-style path (IbisConnection) - -In `backends/ibis/backend.py`, add to `IbisConnection`: - -```python -def to_relation(self, name: str, namespace: str | None = None) -> "Relation": - try: - from mountainash.relations import relation - except ImportError: - raise ImportError( - "mountainash package is required for to_relation(). " - "Install it with: pip install mountainash" - ) - ibis_table = self._ibis_conn.table(name, database=namespace) - return relation(ibis_table) -``` - -### 4. Ibis implementation — settings/factory path (BaseIbisConnection) - -In `backends/ibis/connection.py`, add to `BaseIbisConnection`: - -```python -def to_relation(self, name: str, namespace: str | None = None) -> "Relation": - try: - from mountainash.relations import relation - except ImportError: - raise ImportError( - "mountainash package is required for to_relation(). " - "Install it with: pip install mountainash" - ) - self.connect() - ibis_table = self.ibis_backend.table(name, database=namespace) - return relation(ibis_table) -``` - -This covers connections obtained via `ConnectionFactory` and `DatabaseUtils`. -The factory path gets `to_relation()` now; it will be deprecated in Phase 3 -of the settings-aware-backends backlog. - -### 5. Iceberg stub - -In `backends/iceberg/connection.py`, add to `IcebergConnectionBase`: - -```python -def to_relation(self, name: str, namespace: str | None = None) -> "Relation": - raise NotImplementedError( - "to_relation() is not yet supported for Iceberg connections. " - "Use table() to get the native pyiceberg Table object." - ) -``` - -### 6. Dependency - -Add `mountainash` as an optional extra in `pyproject.toml`: - -```toml -[project.optional-dependencies] -relations = ["mountainash"] -``` - -The core package does NOT depend on mountainash — the import is guarded at -call time in `to_relation()`. - -### 7. Extensibility pattern - -Future backends (DataFusion, etc.) follow the same two-step pattern: - -1. Make the backend class settings-aware (`__init__` accepts `SettingsParameters`) -2. Add `to_relation()` — get native table handle, pass to `relation()` - -The only prerequisite is that `identify_backend()` in mountainash recognises -the native table type and a corresponding relation system backend exists. - -## Testing - -### Settings-aware IbisBackend -- Construct `IbisBackend(settings_params)` with SQLite/DuckDB settings, - verify `.connect()` returns working `IbisConnection` -- Verify `IbisBackend(settings_params).dialect` matches expected dialect -- Verify invalid settings raise `KeyError` -- Verify existing direct path still works unchanged - -### to_relation() -- `IbisConnection.to_relation()` (new-style) with in-memory DuckDB: create - table, call `to_relation()`, verify returns `Relation` instance -- `BaseIbisConnection.to_relation()` (factory path) via - `ConnectionFactory.get_connection()`: same verification -- Round-trip test: `to_relation()` → `.collect()` returns expected data -- `ImportError` path: mock the import to verify clear error message -- `IcebergConnectionBase.to_relation()` raises `NotImplementedError` - -## Commit strategy - -Three commits, single branch, single PR targeting `develop`: - -1. `feat(backend): make IbisBackend settings-aware` -2. `feat(protocol): add to_relation() to Connection protocol and Ibis implementations` -3. `feat(deps): add mountainash as optional relations extra` - -## Follow-up (not in this PR) - -- **Phase 2:** Migrate `DatabaseUtils` consumers to use `IbisBackend(settings_params)` directly -- **Phase 3:** Deprecate `ConnectionFactory`, `OperationsFactory`, `DatabaseUtils.create_connection()`/`create_operations()`, and the 12 concrete `BaseIbisConnection` subclasses - -Tracked in: `mountainash-central/01.principles/mountainash-data/f.backlog/settings-aware-backends.md` - -## Backlog updates after merge - -- `to-relation-gap.md` → **RESOLVED** (Ibis wired; Iceberg stub) -- `settings-aware-backends.md` → Phase 1 **RESOLVED** diff --git a/docs/superpowers/specs/2026-04-27-backend-as-single-handle-design.md b/docs/superpowers/specs/2026-04-27-backend-as-single-handle-design.md deleted file mode 100644 index a2f14a8..0000000 --- a/docs/superpowers/specs/2026-04-27-backend-as-single-handle-design.md +++ /dev/null @@ -1,397 +0,0 @@ -# Backend as Single Handle — Design Spec - -> **Status:** APPROVED -> **Date:** 2026-04-27 -> **Branch:** `feature/settings-aware-ibis-backend` (continues Phase 1 work) -> **Supersedes:** Connection consolidation Phases 2+3 (collapsed into one) - -## Goal - -Make `IbisBackend` the single public handle for all ibis interaction — -construction, connection lifecycle, inspection, and operations. Delete -`ConnectionFactory`, `OperationsFactory`, `DatabaseUtils`, the 12 concrete -`BaseIbisConnection` subclasses, and the separate `BaseIbisOperations` class -hierarchy. - -## Architecture - -`IbisBackend` composes three internal concerns: - -1. **Config resolution** (existing, unchanged) — three-way constructor - dispatch: `SettingsParameters`, URL, or `dialect=` keyword. -2. **Connection lifecycle** — `connect()` returns `self`, `close()` returns - `self`, context manager support. -3. **Operations dispatch** — instance methods on `IbisBackend` that delegate - to the internal `IbisConnection` (inspection), the raw ibis connection - (thin wrappers), or `DialectSpec` callable hooks (per-dialect operations). - -`IbisConnection` stays as an **internal** class. The consumer never imports -or interacts with it directly. `IbisBackend` exposes two accessor methods: - -- `ibis_connection()` → raw ibis backend object (for the seam with - mountainash-expressions: `backend.ibis_connection().table("users")`) -- `get_connection()` → our `IbisConnection` wrapper (for internal use) - -Both raise `RuntimeError` if not connected. - -### What Gets Deleted - -| Target | Location | Reason | -|--------|----------|--------| -| `ConnectionFactory` | `core/factories/connection_factory.py` | Replaced by `IbisBackend` constructor | -| `OperationsFactory` | `core/factories/operations_factory.py` | Replaced by operations on `IbisBackend` | -| `SettingsFactory` | `core/factories/settings_factory.py` | URL detection now in `_SCHEME_TO_DIALECT`; settings creation via `IbisBackend(settings_params)` | -| `DatabaseUtils` | `core/utils.py` | Entire class — `IbisBackend` is the API | -| ~~`BaseDBConnection`~~ | ~~`core/connection.py`~~ | **KEPT** — `IcebergConnectionBase` subclasses it; migrate Iceberg separately | -| `BaseIbisConnection` + 12 subclasses | `backends/ibis/connection.py` | Replaced by `IbisConnection` (internal) + `DialectSpec` registry | -| `BaseIbisOperations` + concrete subclasses | `backends/ibis/operations.py` | Replaced by operations on `IbisBackend` + `DialectSpec` hooks | -| `_DuckDBFamilyOperationsMixin` | `backends/ibis/operations.py` | Implementations become `DialectSpec` hooks | -| `_BaseIbisMixin` | `backends/ibis/operations.py` | Already a deprecated shim | - -### What Stays - -| Component | Location | Role | -|-----------|----------|------| -| `IbisBackend` | `backends/ibis/backend.py` | Single public handle (expanded) | -| `IbisConnection` | `backends/ibis/backend.py` | Internal wrapper — inspection delegation | -| `DialectSpec` registry | `backends/ibis/dialects/_registry.py` | Expanded with operation hooks | -| Module-level operation functions | `backends/ibis/operations.py` | Wired as `DialectSpec` hooks | -| `IcebergBackend` | `backends/iceberg/backend.py` | Unchanged (separate backend) | -| `BaseDBConnection` | `core/connection.py` | Kept — Iceberg depends on it; migrate separately | -| `Backend` protocol | `core/protocol.py` | Updated — `connect()` returns `Self` | -| Inspection model | `core/inspection.py` | Unchanged | -| Settings classes | `core/settings/` | Unchanged | - -## Protocol Changes - -### `Backend` protocol (updated) - -```python -@t.runtime_checkable -class Backend(t.Protocol): - name: str - - def connect(self) -> Self: ... - def close(self) -> Self: ... - def __enter__(self) -> Self: ... - def __exit__(self, *args) -> None: ... - - # Inspection (terminal — return data) - def list_tables(self, namespace: str | None = None) -> list[str]: ... - def list_namespaces(self) -> list[str]: ... - def inspect_table(self, name: str, namespace: str | None = None) -> TableInfo: ... - def inspect_namespace(self, name: str) -> NamespaceInfo: ... - def inspect_catalog(self) -> CatalogInfo: ... -``` - -### `Connection` protocol — REMOVED - -No longer part of the public API. `IbisConnection` is internal. - -## Connection Lifecycle - -```python -# Explicit connect/close -backend = IbisBackend(dialect="sqlite", database=":memory:") -backend.connect() -tables = backend.list_tables() -backend.close() - -# Context manager (recommended) -with IbisBackend(dialect="sqlite", database=":memory:") as backend: - tables = backend.list_tables() - -# Fluent chaining -with IbisBackend(dialect="duckdb", database=":memory:") as backend: - backend.create_table("users", df).create_index("users", ["email"], unique=True) - rows = backend.list_tables() -``` - -`__enter__` calls `connect()` and returns `self`. -`__exit__` calls `close()`. - -Calling any method before `connect()` or after `close()` raises `RuntimeError`. - -## Fluent API - -Methods are categorised into two groups: - -### Fluent (return `self` — chainable mutations) - -| Method | Signature | -|--------|-----------| -| `connect()` | `() -> Self` | -| `close()` | `() -> Self` | -| `create_table()` | `(name, obj, *, schema?, database?, temp?, overwrite?) -> Self` | -| `drop_table()` | `(name, *, database?, force?) -> Self` | -| `create_view()` | `(name, obj, *, database?, overwrite?) -> Self` | -| `drop_view()` | `(name, *, database?, force?) -> Self` | -| `insert()` | `(name, obj, *, database?, overwrite?) -> Self` | -| `upsert()` | `(name, obj, *, conflict_columns, update_columns?, conflict_action?, ...) -> Self` | -| `truncate()` | `(name, *, database?, schema?) -> Self` | -| `rename_table()` | `(old_name, new_name) -> Self` | -| `create_index()` | `(table, columns, *, index_name?, unique?, ...) -> Self` | -| `create_unique_index()` | `(table, columns, *, index_name?, ...) -> Self` | -| `drop_index()` | `(index_name, *, table_name?, database?, if_exists?) -> Self` | - -### Terminal (return data — end of chain) - -| Method | Returns | -|--------|---------| -| `list_tables(namespace?)` | `list[str]` | -| `list_namespaces()` | `list[str]` | -| `table_exists(name, database?)` | `bool` | -| `inspect_table(name, namespace?)` | `TableInfo` | -| `inspect_namespace(name)` | `NamespaceInfo` | -| `inspect_catalog()` | `CatalogInfo` | -| `table(name, *, database?)` | `ir.Table` | -| `run_sql(query, *, schema?, dialect?)` | `ir.Table | None` | -| `run_expr(expr, *, params?, limit?)` | `Any` | -| `to_sql(expr, *, params?, limit?, pretty?)` | `str | None` | -| `index_exists(index_name, *, table_name?, database?)` | `bool` | -| `list_indexes(table_name, *, database?)` | `list[dict]` | -| `ibis_connection()` | raw ibis backend object | -| `get_connection()` | `IbisConnection` (internal wrapper) | - -## DialectSpec Expansion - -`DialectSpec` gains operation hook fields: - -```python -@dataclass(frozen=True) -class DialectSpec: - # Existing - ibis_backend_name: str - connection_mode: str - connection_string_scheme: str - connection_builder: Callable | None = None - get_index_exists_sql: Callable | None = None # existing - get_list_indexes_sql: Callable | None = None # existing - - # New operation hooks - upsert_hook: Callable | None = None - create_index_hook: Callable | None = None - drop_index_hook: Callable | None = None - rename_table_hook: Callable | None = None - - extras: Mapping[str, Any] = field(default_factory=dict) -``` - -### Hook wiring - -The existing `_DuckDBFamilyOperationsMixin` methods become standalone -functions and are wired as hooks: - -| Hook | DuckDB/MotherDuck | SQLite | Others | -|------|-------------------|--------|--------| -| `upsert_hook` | `duckdb_family_upsert` | `duckdb_family_upsert` | `None` | -| `create_index_hook` | `duckdb_family_create_index` | `duckdb_family_create_index` | `None` | -| `drop_index_hook` | `duckdb_family_drop_index` | `duckdb_family_drop_index` | `None` | -| `rename_table_hook` | `None` (not implemented) | `None` (not implemented) | `None` | -| `get_index_exists_sql` | `duckdb_get_index_exists_sql` | `sqlite_get_index_exists_sql` | `None` | -| `get_list_indexes_sql` | `duckdb_get_list_indexes_sql` | `sqlite_get_list_indexes_sql` | `None` | - -`IbisBackend.upsert()` checks `self._spec.upsert_hook`; if `None`, raises -`NotImplementedError(f"Dialect {self.dialect!r} does not support upsert")`. - -### Hook function signatures - -Hooks receive the raw ibis connection as their first argument (not the -`IbisBackend` instance), plus the method's arguments: - -```python -# upsert_hook signature -def duckdb_family_upsert( - ibis_conn: Any, - table_name: str, - df: Any, - *, - conflict_columns: list[str] | str, - update_columns: list[str] | str | None = None, - conflict_action: str = "UPDATE", - update_condition: str | None = None, - database: str | None = None, - schema: str | None = None, -) -> None: ... - -# create_index_hook signature -def duckdb_family_create_index( - ibis_conn: Any, - table_name: str, - columns: list[str] | str, - *, - index_name: str | None = None, - unique: bool = False, - index_type: str | None = None, - where_condition: str | None = None, - database: str | None = None, - if_not_exists: bool = True, -) -> None: ... -``` - -Note: hooks return `None`, not `bool`. The `IbisBackend` wrapper converts -the call to fluent `return self`. Errors raise exceptions (no silent -`False` returns). - -### Thin wrapper methods - -Methods that are simple ibis delegations don't need hooks — they work the -same across all dialects: - -- `create_table` → `ibis_conn.create_table(...)` -- `drop_table` → `ibis_conn.drop_table(...)` -- `create_view` → `ibis_conn.create_view(...)` -- `drop_view` → `ibis_conn.drop_view(...)` -- `insert` → `ibis_conn.insert(...)` -- `truncate` → `ibis_conn.truncate_table(...)` -- `table` → `ibis_conn.table(...)` -- `run_sql` → `ibis_conn.sql(...)` -- `run_expr` → `ibis_conn.execute(...)` -- `to_sql` → `ibis_conn.compile(...)` -- `list_tables` → delegates to `IbisConnection.list_tables()` -- `list_namespaces` → delegates to `IbisConnection.list_namespaces()` -- `inspect_*` → delegates to `IbisConnection.inspect_*()` - -## Public API Changes (`__init__.py`) - -### Removed exports - -- `Connection` (protocol removed) -- `ConnectionFactory` -- `OperationsFactory` -- `SettingsFactory` -- `DatabaseUtils` - -### Retained exports - -- `Backend` (protocol, updated) -- `IbisBackend` (expanded) -- `IcebergBackend` -- `CatalogInfo`, `ColumnInfo`, `NamespaceInfo`, `TableInfo` - -## IcebergBackend Impact - -`IcebergBackend` must also satisfy the updated `Backend` protocol -(`connect()` returns `Self`, context manager). This is a minor change — -same pattern as `IbisBackend`. Operations that don't apply (upsert, indexes) -are simply not on the protocol. - -## Test Strategy - -### New tests for `IbisBackend` operations - -Test with SQLite and DuckDB (in-memory, no external services): - -- **Lifecycle**: `connect()` → use → `close()`, context manager, double-close idempotent, use-before-connect raises -- **Fluent API**: chain `create_table().create_index()`, verify returns `self` -- **Thin wrappers**: `create_table`, `drop_table`, `insert`, `list_tables`, `table`, `run_sql` -- **Hook-based operations**: `upsert` (DuckDB), `create_index`/`drop_index`/`index_exists`/`list_indexes` (SQLite + DuckDB) -- **Unsupported operations**: `upsert` on Trino dialect raises `NotImplementedError` -- **Accessor methods**: `ibis_connection()` returns raw ibis, `get_connection()` returns `IbisConnection` - -### Existing tests — rewrite - -Factory test files (`test_connection_factory.py`, `test_operations_factory.py`) -are deleted. `test_database_utils.py` is deleted. Operations tests -(`test_base_ibis_operations.py`, `test_upsert_and_indexes.py`) are rewritten -to use `IbisBackend` directly. `test_backend.py` is expanded. - -### Existing tests — verify unchanged - -- Inspection tests remain valid -- Settings tests remain valid -- Iceberg tests need minor update for protocol change - -## Files Modified or Deleted - -### Modified - -| File | Change | -|------|--------| -| `backends/ibis/backend.py` | Add lifecycle, operations methods, accessor methods | -| `backends/ibis/dialects/_registry.py` | Add operation hook fields to `DialectSpec`, wire hooks | -| `backends/ibis/operations.py` | Extract mixin methods into standalone hook functions; delete class hierarchy | -| `backends/iceberg/backend.py` | Update to return `Self` from `connect()`, add context manager | -| `core/protocol.py` | Update `Backend` protocol, remove `Connection` protocol | -| `__init__.py` | Remove factory/utils exports, remove `Connection` | -| `tests/test_unit/backends/ibis/test_backend.py` | Expand with operations + lifecycle tests | - -### Deleted - -| File | Reason | -|------|--------| -| `core/factories/` (entire directory) | All factories replaced by `IbisBackend` | -| `core/utils.py` | `DatabaseUtils` replaced by `IbisBackend` | -| ~~`core/connection.py`~~ | **KEPT** — Iceberg depends on `BaseDBConnection`; migrate separately | -| `backends/ibis/connection.py` | `BaseIbisConnection` + 12 subclasses replaced | -| `tests/test_unit/factories/` | All factory tests | -| `tests/test_unit/test_database_utils.py` | `DatabaseUtils` tests | -| `tests/test_unit/databases/test_database_connections.py` | Legacy connection tests | -| `tests/test_unit/databases/connections/` | Legacy connection lifecycle tests | - -## Error Handling - -- Methods that previously returned `bool` or `None` now raise on failure - (no silent swallowing). The `print(f"Error: ...")` pattern throughout - `BaseIbisOperations` is replaced with proper exception propagation. -- Unsupported operations (hook is `None`) raise `NotImplementedError` - with dialect name in the message. -- Use-before-connect and use-after-close raise `RuntimeError`. - -## Consumer Migration - -Before: -```python -from mountainash_data import DatabaseUtils, ConnectionFactory -conn = DatabaseUtils.create_connection(settings_params) -backend = conn.connect() -ops = DatabaseUtils.create_operations(settings_params) -ops.create_table(backend, "users", df) -ops.upsert(backend, "users", new_df, conflict_columns=["id"]) -tables = ops.list_tables(backend) -``` - -After: -```python -from mountainash_data import IbisBackend -with IbisBackend(settings_params) as backend: - backend.create_table("users", df).upsert("users", new_df, conflict_columns=["id"]) - tables = backend.list_tables() - tbl = backend.ibis_connection().table("users") -``` - -## Redshift URL Ambiguity - -Redshift is registered with `connection_string_scheme="postgres://"` because -it uses the postgres wire protocol. This means `_SCHEME_TO_DIALECT` maps -`postgres://` → `"postgres"` (first writer wins), and Redshift is not -reachable by URL alone. - -**Resolution:** Redshift connections must use either: -- `IbisBackend(dialect="redshift", ...)` — explicit dialect keyword -- `IbisBackend(settings_params)` — settings path with Redshift settings class - -This is an inherent limitation: `postgres://` URLs are ambiguous between -postgres and Redshift. The old `SettingsFactory` had the same problem — it -mapped `postgres://` to postgres, not Redshift. No regression. - -If a dedicated `redshift://` scheme is needed in future, add a `url_schemes` -list field on `DialectSpec` separate from `connection_string_scheme`. - -## Compatibility - -This package is internal to the mountainash-io organisation — there are zero -external consumers. All imports of the removed names (`ConnectionFactory`, -`OperationsFactory`, `SettingsFactory`, `DatabaseUtils`, `Connection`) are -in this repo's own test files, which are rewritten in the same change. - -No deprecation window or major-version bump is needed. The removed exports -are cleaned up atomically: deletion + test migration in the same branch. - -## Out of Scope - -| Item | Reason | -|------|--------| -| Iceberg migration off `BaseDBConnection` | Separate spec — `core/connection.py` kept for now | -| `redshift://` URL scheme | No current need — use settings or `dialect=` | -| Operations for non-DuckDB-family dialects | Add hooks when needed (Snowflake, BigQuery, etc.) | diff --git a/docs/superpowers/specs/2026-04-27-settings-aware-ibis-backend-design.md b/docs/superpowers/specs/2026-04-27-settings-aware-ibis-backend-design.md deleted file mode 100644 index e914601..0000000 --- a/docs/superpowers/specs/2026-04-27-settings-aware-ibis-backend-design.md +++ /dev/null @@ -1,262 +0,0 @@ -# Settings-Aware IbisBackend - -> **Date:** 2026-04-27 -> **Status:** Approved -> **Principle:** `mountainash-central/01.principles/mountainash-data/b.connection-management/connection-consolidation.md` (Phase 1) -> **Backlog ref:** `mountainash-central/01.principles/mountainash-data/f.backlog/settings-aware-backends.md` -> **Supersedes:** `docs/superpowers/specs/2026-04-26-to-relation-design.md` (ABANDONED) - -## Goal - -Extend `IbisBackend` to accept `SettingsParameters` and connection URLs -alongside the existing dialect keyword. All three input forms resolve to -the same internal state and produce `IbisConnection` via `connect()`. -This is Phase 1 of connection consolidation. - -## Constructor Signature - -```python -def __init__( - self, - settings_or_connection_string: str | SettingsParameters | None = None, - /, - *, - dialect: str | None = None, - **config: t.Any, -): -``` - -### Input Forms - -```python -from mountainash_data import IbisBackend -from mountainash_settings import SettingsParameters -from mountainash_data.core.settings import DuckDBAuthSettings, NoAuth - -# Form 1: Settings object (deployment, env-driven config) -settings_params = SettingsParameters.create( - settings_class=DuckDBAuthSettings, - DATABASE=":memory:", - auth=NoAuth(), -) -backend = IbisBackend(settings_params) - -# Form 2: Connection URL (universal connection strings) -backend = IbisBackend("postgresql://user:pass@host:5432/db") - -# Form 3: Dialect keyword + kwargs (tests, scripts) -backend = IbisBackend(dialect="sqlite", database=":memory:") - -# All produce the same IbisConnection -conn = backend.connect() -``` - -## Dispatch Logic - -The constructor resolves all input forms to `(self.dialect, self._spec, -self._config)` so that `connect()` requires no changes beyond empty-list -normalization. - -1. **Both positional and `dialect=` provided** -> `ValueError`. -2. **Positional is `SettingsParameters`** -> settings path: - - `settings_class.get_settings(settings_parameters)` to resolve. - - `descriptor.ibis_dialect` to get dialect name. - - `to_driver_kwargs()` for config dict. - - `**config` kwargs merged on top (caller overrides). -3. **Positional is `str` containing `://`** -> URL path: - - Detect dialect from URL scheme using a reverse lookup on - `DialectSpec.connection_string_scheme`. - - Store the raw URL as `connection_string` in config — the dialect - builder passes it straight to `ibis.connect(url)`. - - `**config` kwargs merged on top (caller overrides). -4. **Positional is a plain `str`** -> treat as dialect name (same as - `dialect=` keyword). -5. **`dialect` keyword provided** -> existing dialect path (unchanged). -6. **Neither provided** -> `ValueError`. - -### Settings Resolution Detail - -```python -# Inside __init__, settings path: -from mountainash_settings import SettingsParameters - -obj_settings = settings_or_connection_string.settings_class.get_settings( - settings_parameters=settings_or_connection_string -) -descriptor = getattr(obj_settings, "__descriptor__", None) -if descriptor is None or descriptor.ibis_dialect is None: - raise ValueError( - f"Settings class {type(obj_settings).__name__} has no ibis_dialect on its descriptor" - ) -resolved_dialect = descriptor.ibis_dialect -driver_kwargs = obj_settings.to_driver_kwargs() -driver_kwargs.update(config) # caller overrides -``` - -### URL Resolution Detail - -The URL path bypasses settings entirely. The raw URL is passed to the -dialect builder as `connection_string`, which forwards it to -`ibis.connect(url)`. This preserves all URL components (host, port, -credentials, database, query params) without lossy round-tripping -through settings fields. - -Dialect detection uses a reverse lookup built from the `DIALECTS` -registry — each `DialectSpec` already carries `connection_string_scheme`. - -```python -# Inside __init__, URL path: -from urllib.parse import urlparse - -# Build reverse scheme -> dialect map from registry -# e.g. {"sqlite": "sqlite", "duckdb": "duckdb", "postgres": "postgres", ...} -scheme = urlparse(settings_or_connection_string).scheme.lower() - -# Special cases: "postgresql" -> "postgres", "md" -> "motherduck" -# MotherDuck also detected by "duckdb://md:" prefix -resolved_dialect = _SCHEME_TO_DIALECT.get(scheme) -if resolved_dialect is None: - raise ValueError( - f"Cannot detect ibis dialect from URL scheme: {scheme!r}" - ) - -# Store raw URL as connection_string — builders pass it to ibis.connect() -driver_kwargs = {"connection_string": settings_or_connection_string} -driver_kwargs.update(config) # caller overrides -``` - -The `_SCHEME_TO_DIALECT` map is built once at module level from the -`DIALECTS` registry, with additional aliases for common scheme variants -(e.g. `postgresql` -> `postgres`). - -## Empty-List Normalization - -Happens in `connect()`, not `__init__`. Some ibis drivers reject empty -sequences (e.g. `ibis.duckdb.connect(extensions=[])` fails). - -```python -def connect(self) -> IbisConnection: - cleaned_config = { - k: v for k, v in self._config.items() - if not (isinstance(v, (list, tuple)) and len(v) == 0) - } - ibis_conn = self._spec.connection_builder(**cleaned_config) - return IbisConnection(ibis_conn, self._spec) -``` - -## Files Changed - -| File | Change | -|------|--------| -| `src/mountainash_data/backends/ibis/backend.py` | New constructor signature, dispatch logic, empty-list normalization in `connect()`, `_SCHEME_TO_DIALECT` map | -| `tests/test_unit/backends/ibis/test_backend.py` | New tests for settings, URL, and error paths | - -## Files NOT Changed - -- `IbisConnection` -- untouched -- `DialectSpec` registry / builders -- untouched (URL path uses existing `connection_string` kwarg) -- `ConnectionFactory`, `DatabaseUtils`, `BaseIbisConnection` -- no deprecation yet (Phase 2) -- `core/protocol.py` -- untouched -- No new files - -## Testing - -All tests use SQLite and DuckDB (in-memory, no external deps). - -### 1. Dialect Path (existing, unchanged) - -```python -def test_dialect_path(): - backend = IbisBackend(dialect="sqlite", database=":memory:") - conn = backend.connect() - assert isinstance(conn, IbisConnection) - conn.close() -``` - -### 2. Settings Path - -```python -def test_settings_path_sqlite(): - from mountainash_settings import SettingsParameters - from mountainash_data.core.settings import SQLiteAuthSettings, NoAuth - - params = SettingsParameters.create( - settings_class=SQLiteAuthSettings, - DATABASE=":memory:", - auth=NoAuth(), - ) - backend = IbisBackend(params) - conn = backend.connect() - assert isinstance(conn, IbisConnection) - tables = conn.list_tables() - assert isinstance(tables, list) - conn.close() - -def test_settings_path_duckdb_empty_extensions(): - """DuckDB settings with default EXTENSIONS=[] must not reach ibis.""" - from mountainash_settings import SettingsParameters - from mountainash_data.core.settings import DuckDBAuthSettings, NoAuth - - params = SettingsParameters.create( - settings_class=DuckDBAuthSettings, - DATABASE=":memory:", - auth=NoAuth(), - ) - backend = IbisBackend(params) - conn = backend.connect() # Must not raise - assert isinstance(conn, IbisConnection) - conn.close() -``` - -### 3. URL Path - -```python -def test_url_path_sqlite(): - backend = IbisBackend("sqlite://") - conn = backend.connect() - assert isinstance(conn, IbisConnection) - conn.close() - -def test_url_path_duckdb(): - backend = IbisBackend("duckdb://") - conn = backend.connect() - assert isinstance(conn, IbisConnection) - conn.close() - -def test_url_path_preserves_database(tmp_path): - """URL database component must reach the driver, not be discarded.""" - db_file = tmp_path / "test.db" - backend = IbisBackend(f"sqlite:///{db_file}") - conn = backend.connect() - assert isinstance(conn, IbisConnection) - conn.close() - assert db_file.exists() - -def test_url_path_unknown_scheme_raises(): - with pytest.raises(ValueError, match="Cannot detect ibis dialect"): - IbisBackend("nosuch://localhost/db") -``` - -### 4. Error Cases - -```python -def test_both_positional_and_dialect_raises(): - with pytest.raises(ValueError): - IbisBackend("sqlite://", dialect="sqlite") - -def test_neither_provided_raises(): - with pytest.raises(ValueError): - IbisBackend() - -def test_unknown_dialect_raises(): - with pytest.raises(KeyError): - IbisBackend(dialect="nosuch") -``` - -## Commit Strategy - -Single branch (`feature/settings-aware-ibis-backend`) targeting `develop`. -Two commits: - -1. `feat(backend): make IbisBackend settings-aware` -- constructor + tests -2. `chore(specs): abandon old to-relation spec and plan` -- mark old docs diff --git a/docs/superpowers/specs/2026-05-07-databricks-dialect-design.md b/docs/superpowers/specs/2026-05-07-databricks-dialect-design.md deleted file mode 100644 index 7bc057e..0000000 --- a/docs/superpowers/specs/2026-05-07-databricks-dialect-design.md +++ /dev/null @@ -1,102 +0,0 @@ -# Databricks Dialect Design - -**Date:** 2026-05-07 -**Issue:** mountainash-io/mountainash#108 (follow-on from ClickHouse dialect) -**Scope:** Add Databricks as the 14th supported dialect in `IbisBackend` - -## Context - -Databricks is a cloud-based SQL analytics platform. Its ibis backend -(`ibis.databricks.connect()`) uses kwargs-only connection with no connection -string or port — instead it takes `server_hostname` + `http_path` to identify -a SQL warehouse or cluster, plus auth credentials. - -This follows the same `DialectSpec` + `BackendDescriptor` pattern used by -ClickHouse (PR #81) and the 12 original dialects. - -## Connection Model - -Databricks connects via `ibis.databricks.connect(**kwargs)`: - -- `server_hostname` — workspace URL (e.g. `adb-123.12.azuredatabricks.net`) -- `http_path` — SQL warehouse path (e.g. `/sql/1.0/warehouses/abc123`) -- `access_token` — PAT token -- `catalog` + `schema` — three-level namespace (catalog.schema.table) -- `use_cloud_fetch` — performance optimisation for large result sets - -No `host`/`port`/`database` — this is the key difference from standard dialects. - -## Auth Modes - -| Mode | Mapping | Use case | -|------|---------|----------| -| `TokenAuth` | `token` → `access_token` | Primary — PAT tokens | -| `PasswordAuth` | `username` + `password` | Rare — basic auth | -| `NoAuth` | nothing | Env-var-driven (`DATABRICKS_TOKEN`) | - -Custom `credentials_provider` (OAuth M2M) is a callable, not a credential -pair — better handled as a passthrough kwarg than a first-class auth mode. - -## Changes - -### 1. Constants (`core/constants.py`) - -Add `DATABRICKS = auto()` to `CONST_DB_PROVIDER_TYPE`. The `CONST_DB_BACKEND` -and `CONST_DB_BACKEND_IBIS_PREFIX` entries already exist. - -### 2. Settings class (`core/settings/databricks.py`) - -```python -DATABRICKS_DESCRIPTOR = BackendDescriptor( - name="databricks", - provider_type=CONST_DB_PROVIDER_TYPE.DATABRICKS, - ibis_dialect="databricks", - auth_modes=[TokenAuth, PasswordAuth, NoAuth], - parameters=[ - ParameterSpec(name="SERVER_HOSTNAME", type=str, tier="core", - driver_key="server_hostname"), - ParameterSpec(name="HTTP_PATH", type=str, tier="core", - driver_key="http_path"), - ParameterSpec(name="CATALOG", type=Optional[str], tier="core", - default=None, driver_key="catalog"), - ParameterSpec(name="SCHEMA", type=str, tier="core", - default="default", driver_key="schema"), - ParameterSpec(name="USE_CLOUD_FETCH", type=bool, tier="advanced", - default=False, driver_key="use_cloud_fetch"), - ], -) -``` - -No `default_port` or `connection_string_scheme` — Databricks uses neither. - -### 3. Adapter (`core/settings/adapters/databricks.py`) - -Custom adapter following Snowflake's pattern. Handles auth dispatch: - -- `TokenAuth` → `access_token = token.get_secret_value()` -- `PasswordAuth` → `username`, `password` -- `NoAuth` → no auth kwargs (driver falls back to env vars) - -### 4. Dialect registration (`backends/ibis/dialects/_registry.py`) - -- `_build_databricks_connection(**config)` — extracts known params, calls - `ibis.databricks.connect(**kwargs)` -- `DialectSpec` entry: `connection_mode=KWARGS`, `connection_string_scheme=""` - -### 5. Optional dependency (`pyproject.toml`) - -```toml -databricks = ["databricks-sql-connector>=4", "ibis-framework[databricks]>=11.0.0"] -``` - -### 6. Exports (`core/settings/__init__.py`) - -Import and export `DatabricksAuthSettings`. - -### 7. Tests - -- `test_unit/core/settings/backends/test_databricks.py` — provider type, - defaults, token auth kwargs plumbing, password auth kwargs, no-auth, schema - default -- `test_dialect_spec.py` — registry count 13 → 14, add `"databricks"` to - expected set diff --git a/docs/superpowers/specs/2026-06-27-dialect-aware-add-columns-design.md b/docs/superpowers/specs/2026-06-27-dialect-aware-add-columns-design.md deleted file mode 100644 index 51759f5..0000000 --- a/docs/superpowers/specs/2026-06-27-dialect-aware-add-columns-design.md +++ /dev/null @@ -1,347 +0,0 @@ -# Dialect-Aware Schema Evolution (`add_columns`) - -> **Date:** 2026-06-27 -> **Status:** Draft -> **Backlog ref:** `mountainash-central/01.principles/mountainash-data/f.backlog/dialect-aware-schema-evolution.md` -> **Sibling:** `mountainash-central/01.principles/mountainash-data/f.backlog/generic-default-dialect-operations.md` (applies this pattern to `upsert`/`rename_table`) -> **Builds on:** `docs/superpowers/specs/2026-04-27-settings-aware-ibis-backend-design.md` - -## Goal - -Add a dialect-agnostic `IbisBackend.add_columns(name, source)` operation that -performs **additive** schema evolution — adding columns present in an incoming -frame (or an explicit `{name: dtype}` map) but missing from a target table. -Consumers must never hand-roll `ALTER TABLE … ADD COLUMN` DDL or maintain -their own polars→backend type maps. - -This removes the last non-portable seam in mountainash-wearables' -`WearableStore`/`BronzeStore` (`_evolve_schema` + `_POLARS_TO_DUCKDB` + -`_cast_null_columns`), which today is DuckDB-only DDL bypassing Ibis. - -## Investigation Corrections (read before designing) - -Two assumptions in the backlog item do **not** hold in mountainash-data and -shaped this design: - -1. **There is no "type bridge" in `create_table` to reuse.** - `IbisBackend.create_table` (the "Thin wrapper operations" section of - `backend.py`) is a pure passthrough to `conn._ibis_conn.create_table(...)`; - Ibis infers all column types natively. - "Reuse whatever type bridge `create_table` applies" therefore means **let - Ibis render the types** — specifically via the connection's own - `compiler.type_mapper`, which is exactly what Ibis uses to emit `CREATE - TABLE` DDL. This guarantees an evolved column and a freshly-created column - get **byte-identical** types (verified — see Parity Invariant). - -2. **`_cast_null_columns` is a consumer convention, not an internal one.** - It exists only in wearables. Absorbing "null-typed column → dialect string - type" into `add_columns` is a *new* hoisted behaviour, implemented against - the Ibis `null` dtype rather than against polars. - -## API Surface - -```python -def add_columns( - self, - name: str, - source: t.Any, # frame OR Mapping[str, dtype] - *, - database: str | None = None, -) -> IbisBackend: # fluent — returns self -``` - -`source` is one of: - -- **A frame** — any object `ibis.memtable(...)` accepts (polars/pandas/pyarrow). - Candidate column types are inferred via `ibis.memtable(source).schema()`, - the same inference `create_table` relies on. -- **A `Mapping[str, dtype]`** — explicit column→type. Each value may be an - `ibis.DataType`, an ibis type **string** (`"float64"`), or a - `MountainashDtype` (resolved through the canonical ibis bridge — see Source - Normalization). - -```python -# Infer from the frame, then upsert — the consumer pattern. -backend.add_columns("readings", df) # idempotent, additive -backend.upsert("readings", df, conflict_columns=keys) - -# Explicit types. -from mountainash.core.dtypes.canonical import MountainashDtype -backend.add_columns("readings", {"hrv": MountainashDtype.FP64}) # NB: FP64 -backend.add_columns("readings", {"hrv": "float64"}) # equivalent -``` - -> Note: the canonical member is `MountainashDtype.FP64`, **not** `FLOAT64` as -> the backlog example wrote. There is no `FLOAT64` member. - -## Semantics - -- **Additive only.** Adds columns. Never drops, renames, or re-types existing - columns. Out of scope by design (matches the consumer need: - passthrough-column accretion). -- **Idempotent / introspective (single-process preflight).** Missing columns - are computed against the live table schema - (`conn._ibis_conn.table(name).schema().names`) once, then one `ALTER` is - issued per missing column. A call that adds nothing is a no-op (verified: a - repeated call adds `[]`), so it is safe to call unconditionally before every - write *within a single writer*. It is **not** concurrency-safe — two writers - racing the same new column will collide — and a multi-column add is **not - atomic** on engines without transactional DDL. See Known Limitations; this - matches the single-writer consumer (wearables store) and is documented, not - handled. -- **Type parity with `create_table`.** Types render through the connection's - own `compiler.type_mapper.to_string(dtype)` — the identical mapper Ibis uses - for `CREATE TABLE`. An evolved column is typed exactly as a freshly-created - one would be. -- **Null-typed columns → dialect string.** A candidate column whose inferred - dtype is Ibis `null` (an all-null incoming column) is coerced to - `ibis.dtype("string")` before rendering, so it is creatable on every - dialect. Replaces the wearables `_cast_null_columns` hack. -- **One column per statement.** SQLite permits only a single `ADD COLUMN` per - `ALTER TABLE`; the implementation issues one statement per new column for - universal portability. - -## Design - -### Dispatch shape — generic default with override seam - -Unlike `upsert`/`create_index` (hook-or-`NotImplementedError`), -`add_columns` is a **uniform-SQL** operation: `ALTER TABLE … ADD COLUMN …` is -standard across the registry; only type rendering and identifier quoting vary, -and both are already encapsulated by the connection's compiler. So the default -is a single generic implementation that **covers SQL backends exposing a -sqlglot compiler + `raw_sql` and supporting `ALTER TABLE … ADD COLUMN`, for -single-part `database`/`table` contexts** — verified on duckdb/sqlite; the -registry's other SQL dialects (postgres, snowflake, trino, bigquery, …) are -covered by construction but unverified until a consumer exercises them. -**Multi-part qualification (e.g. BigQuery `project.dataset.table`) is not -supported** by the generic path — it would need a dialect hook; dotted -`name`/`database` are rejected with `ValueError` (see Semantics). An optional -per-dialect override also handles genuine capability gaps (e.g. a backend with -no `ADD COLUMN`). - -```python -# backend.py — thin method, mirrors the existing hook-dispatch wiring -def add_columns(self, name, source, *, database=None): - conn = self._require_connected() - hook = self._spec.add_columns_hook - if hook is not None: - hook(conn._ibis_conn, name, source, database=database) # override wins - else: - _generic_add_columns(conn._ibis_conn, name, source, database=database) - return self -``` - -```python -# _registry.py — new optional field on DialectSpec (default None) -add_columns_hook: t.Optional[AddColumnsHook] = None -``` - -No dialect registers a hook initially; the generic path covers every SQL -dialect that supports `ALTER TABLE … ADD COLUMN`. The field exists so a dialect -that genuinely cannot `ADD COLUMN`, or needs a quirk, can override later — -consistent with the established extensibility pattern. - -### Generic implementation (`operations.py`) - -Verified end-to-end on duckdb and sqlite in the test env: - -```python -from sqlglot import exp - -def _generic_add_columns(ibis_conn, table_name, source, *, database=None): - candidate = _normalize_to_schema(source) # -> ibis.Schema - existing = set(ibis_conn.table(table_name, database=database).schema().names) - tm = ibis_conn.compiler.type_mapper # exact create_table mapper - dialect = ibis_conn.compiler.dialect # sqlglot dialect for quoting - - def _quote(name): # quote each part separately - return exp.to_identifier(name, quoted=True).sql(dialect=dialect) - - # table_name / database must each be a SIMPLE identifier; each is quoted - # as one part. Dotted/multi-part namespaces are out of scope and rejected - # up front (a dotted value would otherwise be quoted as one literal). - _validate_simple_identifier(table_name, kind="table_name") - if database is not None: - _validate_simple_identifier(database, kind="database") - table_parts = [database, table_name] if database else [table_name] - ident_t = ".".join(_quote(p) for p in table_parts) # never quote "db.t" as one - - for col_name, dtype in candidate.items(): - if col_name in existing: - continue - if dtype.is_null(): # all-null col -> string - dtype = ibis.dtype("string") - type_sql = tm.to_string(dtype) - ibis_conn.raw_sql( - f"ALTER TABLE {ident_t} ADD COLUMN {_quote(col_name)} {type_sql}" - ) -``` - -Rendering primitives are read off the **live connection** — no dialect name→ -class lookup, no hardcoded type knowledge. `compiler.type_mapper` and -`compiler.dialect` are present on every Ibis SQL backend (verified on the -test env's Ibis; confirm against the pinned Ibis during implementation). - -### Source normalization - -```python -def _normalize_to_schema(source) -> ibis.Schema: - if isinstance(source, t.Mapping): - return ibis.schema({k: _coerce_dtype(v) for k, v in source.items()}) - return ibis.memtable(source).schema() # frame inference - -def _coerce_dtype(v) -> ibis.DataType: - if isinstance(v, ibis.DataType): - return v - if isinstance(v, MountainashDtype): - from mountainash.core.dtypes import target_ibis - # Gate parametric members via the bridge's own CAST_UNSUPPORTED set - # (currently {LIST, STRUCT}) rather than relying on ibis.dtype() to - # reject a bare "array"/"struct". - if v in target_ibis.CAST_UNSUPPORTED: - raise ValueError(f"MountainashDtype.{v.name} is parametric; " - "pass an ibis.DataType or use the frame form") - return ibis.dtype(target_ibis.SCHEMA_TYPES[v]) # canonical bridge - return ibis.dtype(v) # str or polars/pyarrow dtype -``` - -`target_ibis.SCHEMA_TYPES` maps each `MountainashDtype` to an ibis-castable -type string (`FP64`→`"float64"`, `U8`→`"uint8"`, …); `CAST_UNSUPPORTED` is the -bridge's own `frozenset` of parametric members. **Limitation:** parametric -members (`LIST`/`STRUCT`) are not expressible via the bare enum (they need -element types) and raise `ValueError`; use an explicit `ibis.DataType` or the -frame form for nested columns. - -## Parity Invariant (verified) - -A freshly-`create_table`d column and an `add_columns`-evolved column produce -identical schemas because both flow through the same `type_mapper`. Confirmed -even for an edge type — `uint8` on SQLite, which has no native unsigned type: - -``` -fresh-created uint8 : unknown(DataType(this=DType.USERDEFINED, kind=utinyint)) -evolved uint8 : unknown(DataType(this=DType.USERDEFINED, kind=utinyint)) -PARITY HOLDS : True -``` - -## Known Limitations - -- **Unsigned integers on dialects without them** (SQLite affinity, PostgreSQL - has no unsigned types) render to engine-specific spellings that may not - round-trip cleanly. This is an upstream Ibis behaviour shared by - `create_table` — parity holds, so `add_columns` introduces no new - divergence. Document, don't work around. -- **Parametric types via bare `MountainashDtype`** (LIST/STRUCT) are - unsupported in the explicit-map form; supply an `ibis.DataType` or use the - frame form. -- **Not concurrency-safe; non-atomic multi-column adds.** Idempotency is - single-process preflight (compute-missing-then-ALTER). Concurrent writers can - collide on the same new column, and a partial failure mid-add leaves earlier - columns applied. Acceptable for the single-writer consumer; a transactional - wrapper (where the engine supports DDL transactions) is a future enhancement, - not in this iteration. -- **Simple identifiers only (enforced).** `name`/`database` must each be a - single, non-dotted identifier; multi-part qualified names - (`project.dataset.table`) are out of scope and rejected with `ValueError` - via `_validate_simple_identifier`, not silently mis-quoted. -- **Additive only** — re-typing/dropping/renaming are explicitly out of scope. - -## Files Changed - -| File | Change | -|------|--------| -| `src/mountainash_data/backends/ibis/operations.py` | `_generic_add_columns`, `_normalize_to_schema`, `_coerce_dtype`, `_validate_simple_identifier` | -| `src/mountainash_data/backends/ibis/backend.py` | `IbisBackend.add_columns` thin method (hook dispatch + generic fallback) | -| `src/mountainash_data/backends/ibis/dialects/_registry.py` | `add_columns_hook` optional field on `DialectSpec`; `AddColumnsHook` type alias | -| `tests/test_unit/backends/ibis/test_add_columns.py` | **new file** — all add_columns tests (helpers + integration; see Testing) | - -## Files NOT Changed - -- `DialectSpec` per-dialect entries — no hooks registered; generic path covers all. -- `create_table` / `insert` / `upsert` — untouched. -- `core/protocol.py` — `add_columns` is an `IbisBackend` capability, not part - of the minimal `Connection` protocol (consistent with `upsert`/`create_index`). -- One new file: `tests/test_unit/backends/ibis/test_add_columns.py` (no new - source modules — all production code lands in existing files). - -## Testing - -All tests use in-memory SQLite and DuckDB (no external deps), matching the -existing suite. Cases mirror the verified prototype: - -```python -def test_add_columns_infers_from_frame_duckdb(): - with IbisBackend(dialect="duckdb", database=":memory:") as be: - be.create_table("t", pl.DataFrame({"id": [1], "name": ["a"]})) - df = pl.DataFrame({"id": [1], "name": ["a"], "score": [1.5]}) - be.add_columns("t", df) - cols = {c.name: c.type_name for c in be.inspect_table("t").columns} - assert "score" in cols - -def test_add_columns_is_idempotent(): - with IbisBackend(dialect="sqlite", database=":memory:") as be: - be.create_table("t", {"id": [1]}) - be.add_columns("t", {"x": "float64"}) - be.add_columns("t", {"x": "float64"}) # no-op, no error - names = [c.name for c in be.inspect_table("t").columns] - assert names.count("x") == 1 - -def test_add_columns_null_column_becomes_string(): - with IbisBackend(dialect="duckdb", database=":memory:") as be: - be.create_table("t", {"id": [1]}) - df = pl.DataFrame({"id": [1], "note": pl.Series([None], dtype=pl.Null)}) - be.add_columns("t", df) - cols = {c.name: c.type_name for c in be.inspect_table("t").columns} - assert cols["note"] == "string" - -def test_add_columns_explicit_mountainash_dtype(): - from mountainash.core.dtypes.canonical import MountainashDtype - with IbisBackend(dialect="duckdb", database=":memory:") as be: - be.create_table("t", {"id": [1]}) - be.add_columns("t", {"hrv": MountainashDtype.FP64}) - cols = {c.name: c.type_name for c in be.inspect_table("t").columns} - assert cols["hrv"] == "float64" - -def test_add_columns_create_evolve_parity_sqlite(): - """Evolved column types match freshly-created ones (the core invariant).""" - # create uint8 fresh vs evolve uint8; assert identical schema repr - -def test_add_columns_quotes_identifiers(): - """A column name needing quoting (space/keyword) is added correctly.""" - with IbisBackend(dialect="duckdb", database=":memory:") as be: - be.create_table("t", {"id": [1]}) - be.add_columns("t", {"new col": "float64"}) -``` - -## Consumer Migration (mountainash-wearables, after ship) - -- `WearableStore._evolve_schema` + `_POLARS_TO_DUCKDB` → **delete**; the - `upsert` path becomes `self._backend.add_columns(table, df)` then - `self._backend.upsert(...)`. -- `WearableStore._cast_null_columns` / `BronzeStore._cast_null_columns` → - **delete**; null coercion now lives in `add_columns`. (Confirm no remaining - caller relies on the frame itself being cast before `create_table` — if - `full_replace`/initial `create_table` still need it, keep a thin local cast - only there, or rely on Ibis inference.) -- `BronzeStore` evolution → identical replacement. - -> Caveat carried from the sibling backlog item: `add_columns` makes -> *evolution* portable, but wearables also calls `upsert`, which currently has -> a hook only for the duckdb/sqlite family. Swapping wearables to PostgreSQL -> needs **both** this item and the `upsert` generalization. - -## Commit Strategy - -Single feature branch targeting `develop`. Suggested commits: - -1. `feat(ibis): add dialect-agnostic add_columns with generic-default dispatch` - — operations + backend method + `DialectSpec.add_columns_hook` field + tests. -2. `chore(hatch): drop deprecated mountainash-utils-ssh from test env` — the - stale path dependency removed to unblock the test env (see note below). - -> **Env note (out-of-band):** the `[envs.test]` dependency list referenced -> `../mountainash-utils-ssh`, which has been moved to `deprecated/`. It is only -> a commented-out import in `core/connection.py` and not a runtime dependency, -> so it was removed from the test env to allow a clean rebuild. Flag for the -> maintainer in case other envs (`dev`, `tower`) need the same cleanup. diff --git a/docs/superpowers/specs/2026-06-28-auth-client-migration-design.md b/docs/superpowers/specs/2026-06-28-auth-client-migration-design.md deleted file mode 100644 index c050fa4..0000000 --- a/docs/superpowers/specs/2026-06-28-auth-client-migration-design.md +++ /dev/null @@ -1,686 +0,0 @@ -# Design Spec: Migrate mountainash-data to mountainash-auth-client - -**Date:** 2026-06-28 -**Status:** Draft — for review -**Author:** Nathaniel Ramm (with Claude) -**Supersedes:** `2026-06-27-auth-client-migration-design.md` (v1–v4). That draft -routed auth translation through `Profile.register_adapter` (registering -data's adapters onto auth-client's classes). This rewrite drops that entirely: -**data owns its auth translation in its own code**, mirroring how -`mountainash-wearables` reads credentials directly. No dependency on the -settings `register_adapter` primitive. - ---- - -## 1. Context & Problem - -mountainash-data's settings layer still imports `mountainash_settings.auth`, which -was **deleted upstream** when auth was extracted into the standalone -`mountainash-auth-client` package (settings commit `3d0f4a4`). Against the live -`mountainash-settings` 26.5.0 the package is **broken**: the whole test suite fails -at collection — `conftest` → settings fixtures → `core/settings/__init__.py:21` → -`from mountainash_settings.auth import …` → `ModuleNotFoundError`. Top-level -`import mountainash_data` only survives because `__init__` does not eagerly load the -settings layer. - -This is **not a rename**. Three pieces of upstream machinery mountainash-data leaned -on were also removed: - -| Removed upstream | mountainash-data dependency | Failure | -|---|---|---| -| `auth_modes` field on `ProfileSpec` (settings `2d72318`) | all 20 backends call `BackendSpec(auth_modes=[…])` | `TypeError` at import — frozen dataclass, unknown kwarg | -| `_auth_kwargs()` on `Profile` (settings `297b587`) | the base profile's `to_driver_kwargs()` + `adapters/mysql.py` call it | `AttributeError` at runtime | -| auto-installed `.auth` discriminated-union field (driven by `auth_modes`) | per-backend adapters + `to_connection_string()` read `self.auth` | field no longer exists | -| `mountainash_settings.auth` module | `__init__.py`, the per-backend settings files, 9 adapters, the `core/settings/auth/` shim, ~28 tests | `ModuleNotFoundError` | - -### The new auth model (`mountainash-auth-client`) - -auth-client replaces the old pydantic `*Auth` classes with `*AuthProfile` -classes (subclasses of `mountainash_settings.Profile`): - -- Names: `PasswordAuth` → `PasswordAuthProfile`, `NoAuth` → `NoAuthProfile`, etc. - **No backward-compat aliases, no `AuthSpec` base** — an `AuthProfile` union type - is exported instead. -- Fields are **UPPERCASE** `ParameterSpec` names: `auth.username` → `auth.USERNAME`, - `auth.password` → `auth.PASSWORD`. Secret fields remain pydantic `SecretStr` - (`.get_secret_value()` still works); path fields are `Path | None`. -- The auth profiles ship `emit()`/`__adapters__` adapters for auth-client's own - SDK families (`HTTP`/`BOTO`/`PARAMIKO`). **These are reference implementations** - — the shape a client copies, not a surface a client extends. mountainash-data's - targets are ibis DB drivers, outside those families, so data does **not** use the - auth profiles' `emit()`; it reads their fields directly (§3.4). - -### Project constraints - -mountainash-data is **pre-release with zero downstream consumers**. A **clean -break** is required; the goal is the best architecture for this infrastructure -package, **not** backward compatibility. No deprecation aliases, no compat shims. - ---- - -## 2. Goals & Non-Goals - -### Goals -1. Unbreak the package against `mountainash-settings` 26.5.0 + `mountainash-auth-client`. -2. Adopt `mountainash-transport`'s **three-layer separation** — declarative config - profile, runtime connection, composing factory — with auth **decoupled** from the - config profile and passed alongside it at connect time. -3. **Own the auth→driver-kwargs translation in mountainash-data** (the - `mountainash-wearables` model: read the auth profile's fields directly). Driver - imports (`trino.auth`, `google.oauth2`) stay in data. Nothing is registered onto - auth-client's classes; data depends on no settings extension primitive. -4. Replace the deleted `auth_modes` / `_auth_kwargs` / `.auth`-field machinery. -5. Rename the misnamed `*AuthSettings` classes to `*BackendProfile` (base - `ConnectionProfile` → `BackendProfile`), reserving **"Connection" for the runtime - layer** (§3.1). -6. Make `mountainash-auth-client` a first-class core dependency. -7. All tests green under `hatch run test:test`; `mypy:check` + `ruff:check` clean. - -### Non-Goals -- Interactive OAuth **acquisition**/persistence (`OAuth2TokenManager`, - `PersistableAuthProfile`, `token_store`). Deferred — §10 Backlog. -- Reworking the Ibis `DialectSpec` registry, the inspection model, or the iceberg - catalog registry beyond the auth threading. -- Adding new backends or auth types. -- Any use of `Profile.register_adapter` / cross-package adapter registration. - ---- - -## 3. Architecture - -### 3.1 Three layers — and "Connection" reserved for the runtime - -mountainash-data mirrors `mountainash-transport`'s three roles. The config-layer -classes are named `*BackendProfile` so "Connection" belongs exclusively to the -runtime handles: - -| Role | transport | mountainash-data | -|---|---|---| -| **config profile** — declarative, owns `emit()` for its own config | `settings/storage/profiles/*StorageProfile` | `core/settings/*BackendProfile` | -| **runtime connection** — consumes a finished kwargs dict, opens the handle | `connections/*Connection` | `backends/ibis` (`IbisBackend`/`IbisConnection`), `backends/iceberg` (`IcebergConnection`) | -| **composing factory** — bridges the two, layers auth onto config | `connections/__init__.py:create_connection` | `core/factories/ConnectionFactory` | - -- Base class `ConnectionProfile` → `BackendProfile`. -- 20 leaves `*AuthSettings` → `*BackendProfile` (e.g. `PostgreSQLBackendProfile`). -- Runtime handles keep `IbisConnection` / `IcebergConnection` / `BaseDBConnection`. - -### 3.2 Decouple auth from the backend profile - -A `*BackendProfile` carries **only backend config** (host/port/database/ -warehouse/role/…). Auth is a **separate, orthogonal** `AuthProfile | None` passed -alongside it at connect time — mirroring transport's -`create_connection(storage_profile, auth_profile)`, and reflecting reality: the same -server config is reusable with different credentials. - -```python -backend = IbisBackend(dialect="postgres", host="db", database="app") -conn = backend.connect( - auth_profile=PasswordAuthProfile(USERNAME="app", PASSWORD="s3cret"), -) -``` - -### 3.3 Config emission — `BackendProfile.emit()` (transport-style, data-owned) - -The config side uses the `emit()` pipeline exactly as transport's `StorageProfile` -does — and data owns these classes, so the adapters are **class-body literals**, the -copy-on-write-safe idiom (no `register_adapter`): - -- **Flat backends** (17 of 20): `ParameterSpec.driver_key` renames alone. The package - already carries ~143 `driver_key` annotations; `profile.emit(target)` runs them via - `_default_kwargs`. -- **Non-flat backends** (3): a class-literal `__adapters__` compose adapter builds the - nested/combined config the driver wants — the direct analogue of transport's S3 - `botocore.Config`, SFTP `_post_connect`, HTTP `httpx.Timeout`: - - | Backend | Non-flat connection shaping | Transport precedent | - |---|---|---| - | mysql | nested `ssl={…}` from the 5 `SSL_*` fields | `HTTPStorageProfile` → `httpx.Timeout(...)` | - | mssql | fold `HOST` + `INSTANCE_NAME` → `host\instance`; encryption flags | `SFTPStorageProfile` → `_post_connect` | - | snowflake | `session_parameters={…}` from `QUERY_TAG`/`TIMEZONE` | `S3StorageProfile` → `botocore.Config(...)` | - -The emit **target key is the backend's `provider_type`** (`CONST_DB_PROVIDER_TYPE`, -already on every spec) — no new target enum. A shaping backend keys its literal under -its own `provider_type`: - -```python -@register -class MySQLBackendProfile(BackendProfile): - __spec__ = MYSQL_SPEC # provider_type=MYSQL - __adapters__ = {CONST_DB_PROVIDER_TYPE.MYSQL: _mysql.ssl_compose} -``` - -`base = profile.emit(profile.__spec__.provider_type)` is then uniform across all 20: -flat backends just rename; the 3 shaping backends additionally run their compose -adapter (which receives the already-renamed dict as its second arg). The compose -functions live in `core/settings/adapters/.py`. - -**Compose-adapter invariant (no double-render).** A compose adapter only **adds** the -nested/combined keys that flat `driver_key` renames cannot express; it never -re-derives or overwrites a key `driver_key` already produced. This is the convergence -backlog's smell #2 (transport's S3 path sets a field via `driver_key` then overwrites -it in the compose hook) — mountainash-data must not inherit it. Concretely, **any -field a compose adapter folds into a combined/nested key carries no conflicting flat -`driver_key`**, so exactly one renderer owns each output key: - -| Backend | Adapter adds | Source fields (must NOT also emit flat) | -|---|---|---| -| mysql | `ssl={…}` nested dict | the 5 `SSL_*` fields → no flat `driver_key`; they exist only to feed `ssl` | -| mssql | `host` as `HOST\INSTANCE_NAME` | `INSTANCE_NAME` → no flat `driver_key`; `HOST` keeps its `driver_key` and the adapter **rewrites that one key** (the sole exception — documented, asserted) | -| snowflake | `session_parameters={…}` | `QUERY_TAG`/`TIMEZONE` → no flat `driver_key`; they feed `session_parameters` only | - -The mssql `host` rewrite is the single sanctioned overwrite (a host *can't* be -expressed as a flat rename when it folds a second field). The invariant is enforced -**mechanically** by a key-delta assertion (§7), not just by checking for stray keys: -diff the pre-compose renamed dict against the post-compose dict and assert the only -differences are the sanctioned ones — - -- **mysql / snowflake:** *pure additions* — every pre-compose key is byte-identical - afterward, and exactly the combined key (`ssl` / `session_parameters`) is added; the - folded source fields (`SSL_*`, `QUERY_TAG`/`TIMEZONE`) were never flat to begin with. -- **mssql:** the **only** changed/added key is `host`; `instance_name` must be absent; - every other key is byte-identical. - -Any other key whose value changes between the two dicts is a test failure — this -catches an accidental overwrite at the source, not just a leaked key. - -### 3.4 Auth translation — data-owned, in the factory (the wearables model) - -auth-client can't ship adapters for ibis drivers (they'd need `import trino.auth`), -and its HTTP/BOTO/PARAMIKO adapters are **reference implementations to copy, not an -extension surface**. So mountainash-data owns its auth→kwargs translation outright, -the way `mountainash-wearables` reads `profile.PASSWORD.get_secret_value()` directly -in its connections. **Data does not call `auth_profile.emit()` and registers nothing -onto auth-client's classes.** - -The translation lives behind a **data-owned dispatch table** keyed by -`(provider_type, auth_class)` — declarative dispatch with no `isinstance` ladders, -entirely inside mountainash-data: - -```python -# core/settings/adapters/registry.py (data-owned; NOT auth-client) -from mountainash_auth_client import PasswordAuthProfile, JWTAuthProfile, \ - KerberosAuthProfile, ServiceAccountAuthProfile, IAMAuthProfile, TokenAuthProfile, \ - OAuth2AuthProfile, CertificateAuthProfile, WindowsAuthProfile, AzureADAuthProfile -from mountainash_data.core.constants import CONST_DB_PROVIDER_TYPE as P -from . import sql as _sql, trino as _trino, snowflake as _snow, bigquery as _bq, \ - databricks as _dbx, mssql as _mssql, redshift as _rs, pyiceberg_rest as _ice - -# (provider_type, auth_class) -> (auth_profile, base) -> dict -_AUTH_ADAPTERS: dict[tuple, Callable] = { - (P.TRINO, PasswordAuthProfile): _trino.password, - (P.TRINO, JWTAuthProfile): _trino.jwt, - (P.TRINO, KerberosAuthProfile): _trino.kerberos, - (P.SNOWFLAKE, PasswordAuthProfile): _snow.password, - (P.SNOWFLAKE, TokenAuthProfile): _snow.token, - (P.SNOWFLAKE, OAuth2AuthProfile): _snow.oauth2, - (P.SNOWFLAKE, CertificateAuthProfile): _snow.certificate, - (P.BIGQUERY, ServiceAccountAuthProfile): _bq.service_account, - (P.DATABRICKS, TokenAuthProfile): _dbx.token, - (P.DATABRICKS, PasswordAuthProfile): _dbx.password, - (P.MSSQL, PasswordAuthProfile): _mssql.password, - (P.MSSQL, WindowsAuthProfile): _mssql.windows, - (P.MSSQL, AzureADAuthProfile): _mssql.azure_ad, - (P.REDSHIFT, PasswordAuthProfile): _rs.password, - (P.REDSHIFT, IAMAuthProfile): _rs.iam, - (P.PYICEBERG_REST, TokenAuthProfile): _ice.token, -} -# Flat user/password backends share one adapter: -for _p in (P.POSTGRES, P.MYSQL, P.CLICKHOUSE, P.MATERIALIZE, P.RISINGWAVE, - P.DRUID, P.SINGLESTOREDB, P.IMPALA, P.EXASOL): - _AUTH_ADAPTERS[(_p, PasswordAuthProfile)] = _sql.userpass - -def auth_adapter(provider_type, auth_class): - # Most-specific-first dispatch over the MRO (functools.singledispatch semantics): - # the nearest registered base in auth_class.__mro__ wins. This makes dispatch agree - # with §6's isinstance validation (a subclass of an allowed *AuthProfile both - # validates AND dispatches) AND lets a registered specialization win over its - # registered base — exactly the §10 per-backend-OAuth2-subclass case (register - # both OAuth2AuthProfile and SnowflakeOAuth2Profile for SNOWFLAKE → a - # SnowflakeOAuth2Profile instance resolves to the subclass, MRO-first). - matches = [k for k in auth_class.__mro__ if (provider_type, k) in _AUTH_ADAPTERS] - if not matches: - return None - winner = matches[0] # nearest in MRO - # Ambiguity guard: every other match must be an ancestor of the winner. A match - # that is NOT a superclass of the winner means auth_class multiply-inherits two - # UNRELATED registered auth types for this provider_type — refuse to guess. - ambiguous = [k for k in matches[1:] if not issubclass(winner, k)] - if ambiguous: - raise TypeError( - f"ambiguous auth adapter for {auth_class.__name__} on {provider_type}: " - f"{winner.__name__} vs {[k.__name__ for k in ambiguous]} " - f"(multiply-inherits unrelated registered auth types)" - ) - return _AUTH_ADAPTERS[(provider_type, winner)] -``` - -Specialization (base + subclass both registered) is unambiguous — the subclass is -MRO-first and is a subclass of the base, so the guard passes. Only genuine -multiple-inheritance of two *sibling* registered auth types trips it, and then loudly. - -Each adapter is `(_auth_profile, base) -> dict`, reads UPPERCASE fields (`str(...)` on -`Path | None` where the driver wants a string), and builds the driver kwargs/objects. -Example: - -```python -# core/settings/adapters/trino.py -def password(auth, base): - from trino.auth import BasicAuthentication - return {**base, "auth": BasicAuthentication(auth.USERNAME, - auth.PASSWORD.get_secret_value())} -``` - -**`_snow.oauth2` is pinned to the already-obtained-token contract** — it emits exactly -`authenticator="oauth"` + `token=auth.TOKEN.get_secret_value()` and **nothing else**: - -```python -# core/settings/adapters/snowflake.py -def oauth2(auth, base): - # token-only: data does NOT drive snowflake's authorization-code / - # client-credentials flows (those need oauth_client_id / oauth_client_secret / - # oauth_token_request_url / oauth_scope — provider coordinates the oauth-split - # relocates off the credential schema; deferred to §10). Reads only TOKEN. - return {**base, "authenticator": "oauth", - "token": auth.TOKEN.get_secret_value()} -``` - -It must never read `CLIENT_ID`/`CLIENT_SECRET`/`SERVER_URI`/`SCOPE` — doing so would -reintroduce the layering guide's smell #1. The acquisition-from-coordinates path is the -deferred §10 work. - -`NoAuthProfile` is never in the table — the factory short-circuits it (§3.5). The -driver imports are local to each adapter module, so importing the settings layer -never pulls in `trino`/`google` unless that backend is actually used. - -### 3.5 The composing factory — `ConnectionFactory` (the `_emit_kwargs` analogue) - -Composition lives in the factory, not on the profile (transport's -`create_connection`/`_emit_kwargs`). `BackendProfile` stays a pure config emitter. - -```python -# core/factories/connection_factory.py -def build_driver_kwargs(profile: BackendProfile, auth_profile: AuthProfile | None = None) -> dict: - auth = _normalize_and_validate_auth(profile, auth_profile) # §6 - target = profile.__spec__.provider_type - base = profile.emit(target) # config only (§3.3) - if isinstance(auth, NoAuthProfile): - return base # short-circuit (cf. transport) - fn = auth_adapter(target, type(auth)) - if fn is None: # fail-closed - raise ValueError( - f"{profile.backend}: no auth adapter for {type(auth).__name__}" - ) - return fn(auth, base) # data-owned translation -``` - -``` -caller ── auth_profile (AuthProfile|None) ──▶ IbisBackend.connect(auth_profile) - │ (also iceberg connect path) - ▼ - ConnectionFactory.build_driver_kwargs(backend_profile, auth_profile) - auth = _normalize_and_validate_auth(profile, auth_profile) - target = profile.__spec__.provider_type - base = profile.emit(target) # config (BackendProfile) - │ - NoAuth? ──┴── yes ─▶ return base - │ no - auth_adapter(target, type(auth))(auth, base) # data-owned - │ (builds BasicAuthentication / Credentials / - ▼ {user,password} / …; imports the driver) - dict ready for the ibis driver ──▶ runtime Connection opens it -``` - ---- - -## 4. Component Changes - -### 4.1 `core/settings/__init__.py` -- Replace `from mountainash_settings.auth import (…)` with - `from mountainash_auth_client import (NoAuthProfile, PasswordAuthProfile, - TokenAuthProfile, JWTAuthProfile, OAuth2AuthProfile, IAMAuthProfile, - WindowsAuthProfile, AzureADAuthProfile, KerberosAuthProfile, - CertificateAuthProfile, ServiceAccountAuthProfile, AuthProfile)`. -- Update `__all__`: drop old `*Auth`/`AuthSpec` names; add the `*AuthProfile` - names + `AuthProfile`; rename the backend re-exports to `*BackendProfile`. -- No registration import — auth dispatch is a plain table loaded lazily by the - factory (§3.4); nothing must run at settings-import time. - -### 4.2 Delete `core/settings/auth/` -Remove `__init__.py`, `base.py`, `dispatch.py` (pure shims over the deleted -`mountainash_settings.auth`). Verified the only consumer of -`auth_to_driver_kwargs`/`AUTH_TO_DRIVER_KWARGS` is the shim itself. - -### 4.3 `core/settings/descriptor.py` (`BackendSpec`) -- Add a **required** `supported_auth: tuple[type, ...]` (no default; typed loosely as - `type` to avoid importing the union at dataclass-definition time; values are - `*AuthProfile` classes). Registry invariant: empty `supported_auth` fails at import. -- **Drop `auth_modes`** everywhere (gone with the upstream `ProfileSpec` path). -- No new `auth_target` field — the existing `provider_type` is the dispatch/emit key - (§3.3–§3.4). - -### 4.4 `core/settings/profile.py` (`BackendProfile`) — pure config emitter -- Rename the base class `ConnectionProfile` → `BackendProfile`. -- **Remove all auth coupling**: no `to_driver_kwargs`, no `to_connection_string` (the - auth-threading one), no `_auth_kwargs`, no `__adapter__`. -- The class exposes `emit(target)` (inherited) for its own driver-kwargs config, **and - a credential-free `to_url_parts() -> UrlParts`** with **no credentials**. This is - still pure L1 config rendering (a host/port/db skeleton is config, not auth), the - URL-target analogue of `emit()`; it keeps the profile as declarative as transport's - `StorageProfile`. Credentials are spliced in one layer down by the L3 URL applier - (§4.6), never here. -- **`UrlParts` is a dataclass, not a fixed 5-tuple** — every authority component is - optional so authority-less and account-path URL forms decompose cleanly: - - ```python - @dataclass(frozen=True) - class UrlParts: - scheme: str # "postgresql", "md", "snowflake" - database: str | None = None - host: str | None = None # None ⇒ authority-less (MotherDuck md:) - port: int | None = None - path: str | None = None # account/catalog forms not expressible as host:port - query: dict[str, str] = field(default_factory=dict) # creds-FREE params only - ``` - - The base `to_url_parts()` builds the standard `scheme://host:port/database` from the - common spec fields. **Backends whose URL doesn't fit the standard authority form - override `to_url_parts()`** — e.g. MotherDuck returns `UrlParts(scheme="md", - database=db)` (no host/port; the token is added to `query` later, by the L3 applier, - not here); a future Snowflake account URL populates `path`. The L3 URL applier (§4.6) - consumes `UrlParts` uniformly: password creds splice into the authority **iff `host` - is set** (authority-less schemes never take `user:pass@`), token creds go into - `query`. This keeps every URL quirk in a declarative per-backend `to_url_parts()`, - not smeared across the applier. - -### 4.5 `core/settings/adapters/` — data-owned adapter functions -- `adapters/.py` — the auth-translation functions `(_auth_profile, base) -> dict` - (driver imports local) **and** the 3 connection-shaping compose functions - `(profile, base) -> dict` referenced by the `__adapters__` literals (§3.3). -- `adapters/sql.py` — the shared flat `userpass(auth, base)`. -- `adapters/registry.py` — the `_AUTH_ADAPTERS` table + `auth_adapter()` lookup (§3.4). -- The old per-backend `build_driver_kwargs` modules and the - `__adapter__ = staticmethod(...)` lines are **removed**. - -### 4.6 New: `core/factories/connection_factory.py` -- `_normalize_and_validate_auth(profile, auth_profile) -> AuthProfile`: `None` → - `NoAuthProfile()`, then `isinstance`-validate against - `profile.__spec__.supported_auth`; clear `ValueError` on miss (§6). -- `build_driver_kwargs(profile, auth_profile=None) -> dict` — the §3.5 body. -- `build_connection_string(profile, auth_profile=None) -> str` — **a URL is a distinct - target, and the four layers still hold for it: config-render first (L1), then auth- - apply (L3), in two separate code paths** (never one fused method — that was the - smell in the deleted `to_connection_string`). The factory only *composes* them: - - ```python - def build_connection_string(profile, auth_profile=None) -> str: - auth = _normalize_and_validate_auth(profile, auth_profile) # §6, same gate - parts = profile.to_url_parts() # L1: creds-free skeleton (§4.4) - return _url_auth_applier(profile.__spec__.provider_type)(auth, parts) # L3: splice creds - ``` - - - **L1 — `profile.to_url_parts()`** renders the credential-free `(scheme, host, port, - database, query)` skeleton. No auth knowledge; no `emit()` (URLs aren't kwargs). - - **L3 — the per-`provider_type` URL applier** `(auth, parts) -> str` is the *only* - place creds meet the URL. It does **not** route through the `(provider_type, - auth_class)` *kwargs* table — a connection-string renders creds positionally - (`user:pass@`, `?token=`), a fundamentally different target shape than driver - kwargs — but it is the same L3 role applied to a second target, stated explicitly - so the two appliers don't drift: - - password-style: splice `user:pass@` into the authority for `PasswordAuthProfile` - (`USERNAME` / `PASSWORD.get_secret_value()`, each `quote(..., safe="")`) — valid - only when `parts.host` is set; an authority-less backend (no `host`) that claims - password URL support is rejected by the §6 URL-applier coverage check, not - silently emitted; `NoAuthProfile` → skeleton unchanged; - - token-in-URL backends (MotherDuck `md:?motherduck_token=…`, and any future - Snowflake/Databricks/Trino-JWT URL form) add the token to `query`; - - any other auth type → `NotImplementedError`. - - **Coverage:** each provider's URL applier declares its supported auth types as an - explicit, test-asserted set (§6) — **not** a silent subset of `supported_auth` — so - a backend whose `supported_auth` includes a type the URL applier can't render fails - loudly with `NotImplementedError`, never by emitting a credential-less URL. - -### 4.7 Rename `*AuthSettings` → `*BackendProfile` (20 backends) -Across all 20 modules, class definitions, `__init__.py` exports, and references. Drop -`auth_modes=[…]` from each `BackendSpec(...)`; add `supported_auth=(…AuthProfile, …)`. -mysql/mssql/snowflake additionally gain a class-literal `__adapters__` (§3.3). - -| Old | New | -|---|---| -| `SQLiteAuthSettings` | `SQLiteBackendProfile` | -| `PostgreSQLAuthSettings` | `PostgreSQLBackendProfile` | -| … (all 20) | `*BackendProfile` | - -### 4.8 Entry points -- `backends/ibis/backend.py`: `IbisBackend.connect(self, auth_profile=None)` is the - single auth entry point. **Defer auth-dependent kwargs to `connect()`** — today - `_init_from_settings` eagerly calls `to_driver_kwargs()` at `__init__` - (backend.py:242), before any `auth_profile` exists. Restructure so - `__init__`/`_init_from_settings` resolves only the dialect + spec and stores the - `BackendProfile`; `connect(auth_profile)` calls - `ConnectionFactory.build_driver_kwargs(profile, auth_profile)` and layers - `self._config`. The direct-dialect path is unaffected. -- **URL creds vs explicit `auth_profile` precedence (fail-closed, no silent - override):** URL `user:pass@` is parsed into a `PasswordAuthProfile` **only when no - explicit `auth_profile` is given**. Supplying **both** a URL with embedded creds and - an explicit `auth_profile` is a `ValueError` — neither silently wins; the ambiguity - is rejected. URL credentials are **stripped** before the URL reaches `ibis.connect` - (creds always travel via the auth profile, never the URL). -- `backends/iceberg/connection.py`: `connect_default(self, *, auth_profile=None, **kwargs)` - and `connect` thread `auth_profile` into - `ConnectionFactory.build_driver_kwargs(profile, auth_profile)`. Precedence: - **profile-derived kwargs < explicit `connection_kwargs`/`**kwargs`** (caller - overrides win); document on the methods. - -### 4.9 Dependency wiring -- `pyproject.toml`: add `mountainash-auth-client` to core `dependencies`. -- `hatch.toml`: add `mountainash_auth_client @ {root:uri}/../mountainash-auth-client` - (local: `dev`, `test`) and `{root:uri}/temp/mountainash-auth-client` (CI: - `test_github`, `build_github`), mirroring the existing settings/transport path deps. - Remove the dead `mountainash_utils_ssh` path-dep line where present (the package is - no longer a dependency of this layer). - ---- - -## 5. Field Mapping (old → new), per auth type - -Field names go UPPERCASE; secret-ness preserved. Path fields are `Path | None` — -adapters `str(...)` them where the driver wants a string. Verified against the old -adapter reads and the new `*AuthProfile` `ParameterSpec`s. - -| Auth | Old field(s) | New field(s) | Secret / type notes | -|---|---|---|---| -| Password | `username`, `password` | `USERNAME`, `PASSWORD` | PASSWORD secret | -| Token | `token` | `TOKEN` | TOKEN secret | -| JWT | `token` | `TOKEN` | TOKEN secret | -| Kerberos | `service_name`, `principal` | `SERVICE_NAME`, `PRINCIPAL`, `KEYTAB` | `KEYTAB: Path \| None` (unused by the trino adapter) | -| Windows | `domain`, `username` | `DOMAIN`, `USERNAME` | — | -| AzureAD | `tenant_id`, `client_id`, `client_secret`, `managed_identity`, `msi_endpoint` | `TENANT_ID`, `CLIENT_ID`, `CLIENT_SECRET`, `MANAGED_IDENTITY`, `MSI_ENDPOINT` | CLIENT_SECRET secret | -| IAM | `role_arn`, `access_key_id`, `secret_access_key`, `session_token`, `profile_name` | `ROLE_ARN`, `ACCESS_KEY_ID`, `SECRET_ACCESS_KEY`, `SESSION_TOKEN`, `PROFILE_NAME` | SECRET_ACCESS_KEY, SESSION_TOKEN secret | -| ServiceAccount | `info`, `file` | `INFO`, `FILE` | `FILE: Path \| None`; `INFO: dict \| None` | -| OAuth2 *(consumed)* | `token` | `TOKEN` | TOKEN secret — **the only OAuth2 field any shipped backend reads** (snowflake, token-only) | -| OAuth2 *(present, NOT consumed)* | `client_id`, `client_secret`, `refresh_token`, `server_uri`, `scope` | `CLIENT_ID`, `CLIENT_SECRET`, `REFRESH_TOKEN`, `SERVER_URI`, `SCOPE` | exist on `schemas/oauth2.py` but **no migration adapter may read them** — `SERVER_URI`/`SCOPE` are provider coordinates the oauth-split relocates; the rest belong to deferred acquisition (§10). **Field existence ≠ permission to consume.** | -| Certificate | `private_key`, `private_key_path`, `passphrase` | `PRIVATE_KEY`, `PRIVATE_KEY_PATH`, `PASSPHRASE` | PRIVATE_KEY, PASSPHRASE secret; `PRIVATE_KEY_PATH: Path \| None` | -| NoAuth | — | — | — | - -> **Scope:** rows are exactly the auth types a backend consumes. `OAuth1AuthProfile` -> and `OAuth2AuthCodeAuthProfile` are union members **no backend consumes** (verified: -> zero references) — no mapping, not in any `supported_auth`, not in the table. -> **pyiceberg-REST OAuth2 is descoped from this migration** (§10): its adapter would -> read `OAuth2AuthProfile.SERVER_URI`/`SCOPE`, which the locked -> `oauth-settings-ops-split` design **relocates to the `oauth/` provider profile** -> (they are provider coordinates, not credential data). Reading them off the -> credential schema now would couple this migration to the un-weave and ship the -> layering guide's smell #1 (protocol policy on a generic credential). pyiceberg -> ships **token-only** (§5.1); its OAuth2 path is deferred to §10 alongside the -> acquisition backlog. Snowflake's OAuth2 path **stays** — it reads only -> `OAuth2AuthProfile.TOKEN` (an externally-obtained token, i.e. genuine credential -> data), touching none of the relocating provider-coordinate fields. - -### 5.1 Per-backend `supported_auth` - -| Backend | provider_type | supported_auth | -|---|---|---| -| sqlite, duckdb, pyspark | SQLITE/DUCKDB/PYSPARK | `(NoAuthProfile,)` | -| postgres, clickhouse, singlestoredb, druid, impala, materialize, risingwave | … | `(PasswordAuthProfile, NoAuthProfile)` | -| mysql, exasol | MYSQL/EXASOL | `(PasswordAuthProfile,)` | -| motherduck | MOTHERDUCK | `(TokenAuthProfile,)` | -| trino | TRINO | `(PasswordAuthProfile, JWTAuthProfile, KerberosAuthProfile, NoAuthProfile)` | -| snowflake | SNOWFLAKE | `(PasswordAuthProfile, OAuth2AuthProfile, CertificateAuthProfile, TokenAuthProfile)` | -| bigquery | BIGQUERY | `(ServiceAccountAuthProfile, NoAuthProfile)` | -| databricks | DATABRICKS | `(TokenAuthProfile, PasswordAuthProfile, NoAuthProfile)` | -| redshift | REDSHIFT | `(PasswordAuthProfile, IAMAuthProfile)` | -| mssql | MSSQL | `(PasswordAuthProfile, WindowsAuthProfile, AzureADAuthProfile)` | -| pyiceberg_rest | PYICEBERG_REST | `(TokenAuthProfile,)` — OAuth2 deferred to §10 | - ---- - -## 6. Validation & Error Handling - -- Factory-level `_normalize_and_validate_auth(profile, auth_profile)` is called first - by **both** `build_driver_kwargs` and `build_connection_string`: `None` → - `NoAuthProfile()`, then `isinstance(auth, tuple(profile.__spec__.supported_auth))`; - on miss raise `ValueError(f"{profile.backend} does not support auth: - {type(auth).__name__}")`. `isinstance` (not exact `type()`) so subclasses are - accepted — **and dispatch is correspondingly MRO-aware** (§3.4 `auth_adapter` walks - the MRO), so an accepted subclass both validates *and* resolves to its registered - base's adapter. Validation and dispatch share the same subclass semantics; they - cannot disagree. -- Empty `supported_auth` is impossible: the registry invariant (§4.3) rejects it. -- **Fail-closed dispatch:** if `auth` passed `supported_auth` but - `auth_adapter(provider_type, type(auth))` is `None`, the factory raises (§3.5) — - an auth type listed as supported but missing its adapter is caught loudly, never by - emitting unauthenticated kwargs. -- A startup consistency check (test, §7) asserts every `(provider_type, auth_class)` - in each backend's `supported_auth` (minus `NoAuthProfile`) has a table entry, and - no table entry references an unsupported pair. -- **URL applier coverage:** `build_connection_string` is a distinct L3 target (§4.6), - so its auth handling is not the kwargs table. Its supported auth types are an - explicit per-`provider_type` set; the §7 URL test asserts every type **not** in that - set raises `NotImplementedError` (fail-closed — never a credential-less URL), so the - parallel structure stays covered rather than drifting. - ---- - -## 7. Testing Strategy - -- Update ~28 test files: imports → `mountainash_auth_client` (or the `core/settings` - re-exports); construction → UPPERCASE kwargs - (`PasswordAuthProfile(USERNAME="u", PASSWORD="p")`); auth passed as a separate arg, - not an `auth=` field. `tests/fixtures/settings_fixtures.py` yields - `(backend_profile, auth_profile)` pairs. -- New focused tests: - - **Golden per (dialect, auth type):** `build_driver_kwargs(profile, auth)` yields - the exact driver-kwargs dict (trino → `auth=BasicAuthentication(...)`; bigquery → - `credentials=…`; postgres → `{user, password}`; …). Mirrors transport's emission - goldens. - - **Config-only emit + key-delta:** `profile.emit(provider_type)` golden for the 3 - shaping backends (mysql `ssl={}`, mssql host-fold, snowflake `session_parameters`). - Beyond the combined-key value, the test diffs the pre-compose renamed dict against - the post-compose dict and asserts **only the sanctioned delta** (§3.3): mysql/ - snowflake are pure additions (all prior keys byte-identical); mssql changes only - `host` and emits no `instance_name`. Any other changed key fails — mechanically - locking the no-double-render invariant, not merely checking for leaked keys. - - **`supported_auth` rejection:** out-of-`supported_auth` type → `ValueError` (one - negative per backend). - - **`None` normalization:** no auth → `NoAuthProfile` accepted for no-auth backends, - rejected for credential-required backends. - - **Subclass end-to-end (validation + dispatch):** a subclass of an allowed - `*AuthProfile` is both accepted by validation **and** dispatched to its registered - base's adapter (proving §3.4's MRO walk and §6's `isinstance` agree — guards the - blocker where exact-`type()` dispatch would have crashed a validated subclass). - - **Dispatch resolution (§3.4):** (a) *specialization* — with both a base and its - subclass registered for one `provider_type`, a subclass instance resolves to the - subclass adapter (most-specific-first); (b) *ambiguity* — a class multiply- - inheriting two **sibling** registered auth types for one `provider_type` raises - `TypeError`, never silently picking one. - - **Fail-closed dispatch:** a supported auth type with no table entry → `ValueError`. - - **Table/`supported_auth` consistency** (§6). - - **`build_connection_string`:** password backend → `user:pass@` (percent-encoded, - secret unwrapped); MotherDuck → `…?motherduck_token=`; other type → - `NotImplementedError`. - - **Registry invariant:** empty `supported_auth` fails at import. - - **URL-vs-explicit precedence:** both → `ValueError`; URL-only → creds stripped and - carried via `PasswordAuthProfile`. -- Acceptance gate: `hatch run test:test` green; `mypy:check` clean; `ruff:check` clean. - ---- - -## 8. Isolation & Interfaces - -- **auth-client** — owns the credential schemas (`*AuthProfile`) and ships - `emit()` adapters for its own HTTP/BOTO/PARAMIKO families as **reference - implementations**. mountainash-data reads its profile fields directly and - **registers nothing onto it**. -- **`*BackendProfile`** — owns backend config + its own `emit(provider_type)`; pure - config, no auth methods (transport `StorageProfile` analogue). -- **auth adapters** (`core/settings/adapters/.py` + `sql.py`) — data-owned - `(auth_profile, base) -> dict`; the only place that knows a driver's auth-kwarg - shape; import the DB drivers; independently unit-testable without a live DB. -- **`adapters/registry.py`** — the `(provider_type, auth_class) -> fn` dispatch table. -- **`ConnectionFactory`** — the composing bridge (`build_driver_kwargs` / - `build_connection_string` / `_normalize_and_validate_auth`); the only layer aware - of *both* a backend profile and an auth profile. -- **runtime** (`IbisBackend`/`IbisConnection`, `IcebergConnection`) — consume the - finished kwargs dict. - -This depends on **no settings extension primitive** — only the stable -`mountainash-settings` `Profile`/`ProfileSpec`/`emit()` surface and -`mountainash-auth-client`'s profile classes. - ---- - -## 9. Rollout - -A single feature branch off mountainash-data `develop` → PR to `develop`. Internally -atomic (the package does not import cleanly until the settings layer is migrated). -Suggested commit slices: -(a) deps + `__init__` import swap + delete `auth/` shim; -(b) descriptor (`supported_auth` + invariant, drop `auth_modes`) + `BackendProfile` -rename to a pure `emit` config class; -(c) the 20 renames `*AuthSettings`→`*BackendProfile` + `supported_auth` + the -mysql/mssql/snowflake connection-shaping `__adapters__` literals; -(d) the data-owned auth adapters + `adapters/registry.py`; -(e) `ConnectionFactory` (`_normalize_and_validate_auth`, `build_driver_kwargs`, -`build_connection_string`); -(f) entry points (deferred-auth `IbisBackend`, URL precedence, iceberg threading); -(g) tests. - -**No dependency on the settings `Profile.register_adapter` primitive.** That primitive -(settings PR #47) has no consumer under this design; reverting it from -mountainash-settings is a separate, recommended cleanup tracked outside this spec. - ---- - -## 10. Backlog (deferred, captured) - -**Interactive OAuth acquisition & token persistence.** Snowflake's OAuth path ships -in this migration but only consumes an **already-obtained** token read statically off -the auth profile (`auth.TOKEN`). A future capability should integrate the wearables -lifecycle so mountainash-data can **acquire and refresh** tokens itself: -`OAuth2TokenManager(provider, auth_profile, resolver=…)`; `PersistableAuthProfile` -(`SETTINGS_SOURCE_SECRETS_PROVIDER` + `persist_key()`) + `token_store()`; a -`SecretStoreResolver` + `mountainash-secrets` wiring; likely a small per-OAuth-backend -subclass (à la wearables' `WearableOAuth2Auth`). - -**pyiceberg-REST OAuth2 (descoped from this migration).** pyiceberg's OAuth2 catalog -auth needs `SERVER_URI`/`SCOPE` (token-endpoint coordinates) plus a credential. The -locked `oauth-settings-ops-split` design relocates `SERVER_URI`/`SCOPE` off the -credential schema into the `oauth/` provider profile, so a clean pyiceberg-OAuth2 -adapter must consume *that* provider profile, not `OAuth2AuthProfile` — work that -belongs after the auth-client un-weave lands. Until then pyiceberg ships -`(TokenAuthProfile,)`: a caller with an externally-obtained catalog token is fully -served; OAuth2-handshake-from-coordinates is the deferred piece. Both items tracked as -follow-up issues after this migration merges; both are gated on the auth-client -un-weave only for the provider-coordinate reads, not for the token-only paths. - ---- - -## 11. Open Questions - -None outstanding. Auth placement = decoupled, composed in the factory; compat = clean -break; rename = `*BackendProfile` ("Connection" reserved for runtime); auth -translation = **data-owned** direct-field adapters in a `(provider_type, auth_class)` -table (no `register_adapter`, no `emit()` on auth profiles, no cross-package -mutation); config = `BackendProfile.emit()` with class-literal `__adapters__` on the 3 -non-flat backends; OAuth lifecycle **and pyiceberg-REST OAuth2** = deferred (§10, -gated on the auth-client un-weave); snowflake OAuth2 ships (token-only read). diff --git a/docs/superpowers/specs/2026-06-29-generic-default-dialect-operations-design.md b/docs/superpowers/specs/2026-06-29-generic-default-dialect-operations-design.md deleted file mode 100644 index 74776d9..0000000 --- a/docs/superpowers/specs/2026-06-29-generic-default-dialect-operations-design.md +++ /dev/null @@ -1,676 +0,0 @@ -# Generic-Default Dialect Operations — Design Spec - -> **Status:** DESIGN (2026-06-29). Approved scope: build the *complete, -> reusable* write-operation surface for `IbisBackend` — portable `upsert` -> across every upsert family the dialect registry can name, plus a portable -> `rename_table` — not a minimum that unblocks one consumer. Successor to -> `2026-06-27-dialect-aware-add-columns-design.md` (shipped, PR #90); reuses -> and consolidates its sqlglot rendering pattern. - -## 1. Purpose - -`IbisBackend` exposes write/DDL operations through a `DialectSpec` hook -registry. Two of them are under-supported for no good reason: - -- **`upsert`** is registered for only `sqlite`/`duckdb`/`motherduck` (3 of 21 - dialects); every other dialect — including `postgres` — raises - `NotImplementedError`. -- **`rename_table`** has **zero** registered hooks; it raises for *every* - dialect. - -This package is strategic, reusable infrastructure — physical access to -backend data services — not a wearables helper. The unit of design is the -package's domain responsibility (upsert/rename across the SQL backends it -claims to serve), not any one consumer's current backend. This spec makes -both operations *work everywhere the SQL is expressible*, via generic -defaults rendered through sqlglot — the same engine Ibis itself uses — with -an override hook retained for genuine exceptions. - -## 2. The Discriminator Principle - -`IbisBackend` capability operations fall into two classes; dispatch must -reflect the class: - -- **Uniform / family-uniform SQL** — the statement is the same across a - family of dialects; only type rendering, identifier quoting, and a bounded - statement-family choice vary. These get a **generic default** (sqlglot - rendered) + an **optional override hook**. Default: *works everywhere it - can be expressed.* -- **Capability-divergent** — the operation's *primary mechanism* does not - exist on some engines, or **is** a dialect-specific catalog read (e.g. - `list_indexes`). These stay **hook-required**; absent a hook, - `NotImplementedError`. Default: *honestly unsupported.* - -`upsert` and `rename_table` are the former. `create_index` / `drop_index` -and the index-catalog queries are the latter and are **out of scope** (no -secondary indexes on BigQuery/Snowflake — a generic default would emit DDL -the engine rejects while presenting as supported). - -**Preflight validation vs catalog-defined operations.** The distinction is the -operation's *primary mechanism*, not whether it touches a catalog at all. A -generic default MAY issue a bounded **preflight validation** query to *fail -closed* on a case it cannot safely render (e.g. the MySQL unique-index check in -§6.2) — that is a safety gate, not the operation. What stays hook-required is -an operation whose primary purpose *is* the catalog read. This keeps §6.2's -fail-closed validation consistent with the discriminator principle. - -## 3. Goals / Non-Goals - -**Goals** -- `rename_table` works on every dialect whose rename is expressible, with no - per-dialect code (sqlglot renders `sp_rename` for SQL Server, `RENAME` for - MySQL, `ALTER TABLE … RENAME TO` elsewhere). -- `upsert` works across **three** upsert families — `ON CONFLICT`, - `MERGE`, and MySQL's `ON DUPLICATE KEY UPDATE` — preserving the full - existing semantics (composite keys, `conflict_action`, `update_columns`, - `update_condition`). Public signature unchanged. -- A single shared **rendering-primitives helper** backs `upsert`, - `rename_table`, and (consolidated) `add_columns`. -- Portable staging: the source frame is materialised as a compiled subquery - (Ibis's own mechanism), not a backend-specific temp-table registration. -- Live integration tests on `sqlite`, `duckdb`, `postgres`, `mysql` - (Docker); golden-SQL render assertions for the full registry; a documented - support matrix. - -**Non-Goals** -- `create_index` / `drop_index` / index-catalog queries — stay hook-required. -- A bespoke statement-renderer framework — sqlglot **is** the renderer; we - build its AST, we do not wrap it in a parallel layer. -- A second hand-maintained type map — types render through the connection's - own `compiler.type_mapper`, exactly as `add_columns` does. -- Live testing of warehouse engines (Snowflake/BigQuery/MSSQL/…) — covered by - golden-SQL render assertions, not execution. - -## 4. API Surface - -### 4.1 `upsert` (signature unchanged) - -```python -# A conditional-update predicate: receives two ibis tables (the incoming -# source row-set and the existing target row-set, both with the target schema) -# and returns an ibis boolean. Rendered per-dialect through ibis/sqlglot. -ConditionPredicate = t.Callable[[ir.Table, ir.Table], ir.BooleanValue] - -def upsert( - self, - name: str, - obj: t.Any, # frame or ibis Table - *, - conflict_columns: list[str] | str, - update_columns: list[str] | str | None = None, - conflict_action: str = "UPDATE", # "UPDATE" | "NOTHING" - update_condition: ConditionPredicate | None = None, - database: str | None = None, - schema: str | None = None, -) -> IbisBackend: ... -``` - -Fluent. Composite `conflict_columns` is supported (unlike Ibis 12's native -`upsert`, whose `on` is a single column — a key reason we render our own). - -`update_condition` is an **ibis-expression predicate**, not a raw SQL string — -e.g. `lambda incoming, existing: incoming.updated_at > existing.updated_at` -("merge only when the incoming row is newer"). This is the conditional-upsert -capability that powers idempotent late-arriving-data merges and optimistic -last-write-wins. Expressed as ibis, it renders portably per dialect — row -references, **functions, and types alike** — is type-validated, and carries no -SQL-injection surface, consistent with the package's render-through-ibis -discipline. The previous raw-string form is dropped (pre-release clean break); -callers needing raw SQL use the `upsert_hook` override. See §6.1 for the -rendering mechanism (verified by probe, §9). - -### 4.2 `rename_table` (signature unchanged) - -```python -def rename_table(self, old_name: str, new_name: str) -> IbisBackend: ... -``` - -### 4.3 Dispatch (both operations) - -```python -# rename_table -hook = self._spec.rename_table_hook -if hook is not None: - hook(conn._ibis_conn, old_name, new_name) -else: - _generic_rename_table(conn._ibis_conn, old_name, new_name) -return self - -# upsert -hook = self._spec.upsert_hook -if hook is not None: - hook(conn._ibis_conn, name, obj, conflict_columns=..., ...) -else: - _generic_upsert(conn._ibis_conn, name, obj, style=self._spec.upsert_style, ...) -return self -``` - -`_generic_upsert` raises `NotImplementedError` when `style is None`, naming -the dialect — the same honest-unsupported contract the hook path had. - -## 5. Architecture - -### 5.1 Registry: `upsert_style` - -`DialectSpec` gains one field: - -```python -class UpsertStyle(str, enum.Enum): - ON_CONFLICT = "on_conflict" # INSERT … ON CONFLICT DO UPDATE/NOTHING - MERGE = "merge" # MERGE INTO … WHEN MATCHED / NOT MATCHED - ON_DUPLICATE_KEY = "on_duplicate_key" # INSERT … ON DUPLICATE KEY UPDATE (MySQL family) - -@dataclass(frozen=True) -class DialectSpec: - ... - upsert_hook: t.Optional[UpsertHook] = None # override seam (unchanged) - upsert_style: t.Optional[UpsertStyle] = None # NEW — generic-default selector - rename_table_hook: t.Optional[RenameTableHook] = None - add_columns_hook: t.Optional[AddColumnsHook] = None -``` - -The existing `upsert_hook` registrations on `sqlite`/`duckdb`/`motherduck` -are **removed**; those dialects instead carry `upsert_style=ON_CONFLICT` and -flow through the generic renderer. `upsert_hook` remains as the override -seam (now used by zero dialects — like `add_columns_hook`), reserved for a -dialect that genuinely needs a quirk. - -### 5.2 Rendering-primitives helper (consolidation) - -`add_columns` rendered its own quoting inline. Extract the shared primitives -into one new sibling module, `backends/ibis/_render.py` (a flat module -alongside `operations.py`, **not** a restructure of `operations.py` into a -package), with a single responsibility — turn names/types into -dialect-correct SQL fragments off a live connection: - -```python -def dialect_of(ibis_conn) -> str: ... # ibis_conn.compiler.dialect -def quote_identifier(name: str, dialect) -> str: ... # exp.to_identifier(name, quoted=True).sql(dialect) -def qualified_name(parts: list[str], dialect) -> str: ... # ".".join(quote_identifier(p, dialect) …) -def render_type(type_mapper, dtype) -> str: ... # type_mapper.to_string(dtype) — create_table parity -def compiled_source(ibis_conn, obj) -> str: ... # ibis.memtable(obj) → ibis_conn.compile(…) subquery SQL -``` - -`_generic_add_columns`, `_generic_rename_table`, and `_generic_upsert` all -consume these. `add_columns` is migrated to the helper with **no behaviour -change** (the inline `_quote` is replaced by `quote_identifier`); its tests -remain green unchanged. - -### 5.3 `_generic_rename_table` - -```python -def _generic_rename_table(ibis_conn, old_name: str, new_name: str) -> None: - _validate_simple_identifier(old_name, kind="old_name") - _validate_simple_identifier(new_name, kind="new_name") - dialect = dialect_of(ibis_conn) - stmt = exp.Alter( - this=exp.to_table(quote_identifier(old_name, dialect)), - kind="TABLE", - actions=[exp.AlterRename(this=exp.to_identifier(new_name, quoted=True))], - ).sql(dialect=dialect) - ibis_conn.raw_sql(stmt) -``` - -sqlglot renders this as `ALTER TABLE … RENAME TO …` for most dialects, -`EXEC sp_rename …` for `tsql`, and `ALTER TABLE … RENAME …` for MySQL — -the *output* is verified by transpile probe (§9). The exact sqlglot -expression classes shown (`exp.Alter` / `exp.AlterRename`) are illustrative -and pinned during implementation against the installed sqlglot (30.x); the -equivalent transpile path (`sqlglot.transpile(, write=…)`) is -an accepted fallback if AST construction is more brittle. No override hooks -required initially. - -### 5.4 `_generic_upsert` and the three renderers - -```python -def _generic_upsert( - ibis_conn, name, obj, *, style, conflict_columns, update_columns, - conflict_action, update_condition, database, schema, -) -> None: - if style is None: - raise NotImplementedError(f"Dialect does not support upsert") - _validate_simple_identifier(name, kind="name") - # resolve target/conflict/update column sets, validate existence, - # validate conflict_action ∈ {UPDATE, NOTHING} - source_sql = compiled_source(ibis_conn, obj) # subquery, no temp table - if style is UpsertStyle.ON_CONFLICT: - stmt = _render_on_conflict(...) - elif style is UpsertStyle.MERGE: - stmt = _render_merge(...) - elif style is UpsertStyle.ON_DUPLICATE_KEY: - stmt = _render_on_duplicate_key(...) - ibis_conn.raw_sql(stmt) -``` - -Each renderer builds a sqlglot AST and renders with the live dialect. The -`MERGE` renderer mirrors Ibis 12's `_build_upsert_from_table` (`sge.merge`) -**extended** to composite `on` and our `conflict_action`. - -### 5.5 Staging — compiled subquery, not temp table - -The existing `duckdb_family_upsert` registers a temp staging table via the -DuckDB-specific `ibis_conn.con.register(...)`. That is the portability -blocker, not the SQL. The generic path instead compiles the source frame as -a subquery — `compiled_source()` returns `(SELECT … )` — used as the -`INSERT … SELECT … FROM ()` source or the MERGE `USING -() AS src`. This is exactly Ibis 12's mechanism. No staging table, -no cleanup, portable by construction. - -**Column ordering (mandatory).** Every family renders an **explicit target -column list** and projects the source subquery columns **in target-column -order** — never `INSERT … SELECT *` or positional value lists. A source whose -columns are ordered `[name, id]` against a target `[id, name]` must not swap -values. The renderer derives the column list from the source schema, -intersected with the target schema, and projects both `INSERT` and the MERGE -`WHEN NOT MATCHED … INSERT (cols) VALUES (src.cols)` in that exact order. - -**Type alignment across the subquery boundary (mandatory).** The source -projection **casts each column to the target table's column type** via the -connection's `compiler.type_mapper` (the same type-parity mechanism -`add_columns` uses). This is required because an all-null source column -compiles as an untyped `NULL` literal that warehouse `MERGE` engines reject -or mis-coerce against a typed/non-nullable target, and because temporal / -decimal types are backend-sensitive. Columns present in the target but absent -from the source are omitted from the column list (engine default / NULL -applies); columns present in the source but absent from the target raise -`ValueError`. - -## 6. Semantics Mapping - -The public semantics map onto each family as below. The `ON CONFLICT` and -`MERGE` families honour an identical public contract; `ON DUPLICATE KEY` -honours it with two **documented divergences** (conflict targeting and the -`NOTHING` action), spelled out under the table — it is *not* claimed -byte-equivalent. - -| Public param | `ON CONFLICT` | `MERGE` | `ON DUPLICATE KEY` | -|---|---|---|---| -| `conflict_columns` (composite) | `ON CONFLICT (c1,c2)` | `ON tgt.c1=src.c1 AND tgt.c2=src.c2` | **detection is by the table's unique indexes, not this list** — see §6.2 | -| `conflict_action="UPDATE"` | `DO UPDATE SET …` | `WHEN MATCHED THEN UPDATE SET …` + `WHEN NOT MATCHED THEN INSERT …` | `ON DUPLICATE KEY UPDATE …` | -| `conflict_action="NOTHING"` | `DO NOTHING` | omit `WHEN MATCHED` (insert-if-absent only) | `… UPDATE k0=k0` self-assign — **not a true no-op**, see §6.2 | -| `update_columns` (subset) | restrict `SET` list | restrict `WHEN MATCHED … SET` list | restrict `UPDATE` list | -| `update_condition` | `DO UPDATE SET … WHERE ` | `WHEN MATCHED AND THEN UPDATE` | unsupported → `ValueError` | -| default `update_columns` | all non-key columns | all non-key columns | all non-key columns | - -### 6.1 `update_condition` — ibis-expression predicate (cross-family rendering) - -`update_condition` is an ibis-expression predicate, not raw SQL (§4.1). The -caller receives two ibis tables — `incoming` (the source row-set) and -`existing` (the target row-set), both bound to the target's resolved schema — -and returns an ibis boolean. The renderer turns it into the per-family clause -through ibis + sqlglot, verified by probe (§9): - -0. **Bind the two tables under reserved sentinel names.** `incoming` and - `existing` are constructed as ibis tables named `__ma_incoming__` / - `__ma_existing__` — names the package reserves and that no real target can - carry (a target colliding with a sentinel raises `ValueError`). This is how - step 2 stays unambiguous even if the user's target is named `incoming`, - `excluded`, etc. (Codex finding: never identify sides positionally or by a - user-controllable name.) -1. Form `existing.join(incoming, predicate)` and obtain its **sqlglot AST** - via `ibis_conn.compiler.to_sqlglot(...)`. ibis renders a two-table - predicate only as a join condition (it refuses a bare two-table boolean) — - the join is the supported path. -2. Identify ibis's auto-assigned join aliases by walking `exp.Table` nodes and - matching each alias's underlying name to the **sentinels** from step 0: - `{ibis_alias → incoming|existing}`. Sentinel matching is unambiguous by - construction. -3. Extract the join's `ON` sub-AST and `.transform()` the column qualifiers to - **explicit, controlled aliases** (never a bare table name): - - **ON CONFLICT** → the INSERT is rendered `INSERT INTO AS tgt …` - so the existing row has a dedicated alias; incoming columns map to - `EXCLUDED`, existing columns to `tgt` → spliced into `DO UPDATE SET … - WHERE `. (Postgres/SQLite support `INSERT … AS alias`; a dialect - that cannot alias the target in ON CONFLICT is recorded in §7 and routes - to the `upsert_hook`.) - - **MERGE** → incoming to source alias `src`, existing to target alias - `tgt` → spliced into `WHEN MATCHED AND THEN UPDATE`. -4. Render the remapped sub-AST with the live dialect. - -**Accepted predicate grammar (validated).** The predicate must be a **scalar -row predicate** over `incoming`/`existing` columns, literals, and -ibis-modelled scalar functions/operators. Before rendering, the renderer walks -the ibis expression and **raises `ValueError`** if it contains an aggregation, -window function, or any subquery/`EXISTS`/third-table reference — these compile -into the join but are invalid or semantically wrong inside `DO UPDATE … WHERE` -/ `WHEN MATCHED AND`. Conditions outside this grammar (e.g. correlated -`EXISTS`, a third-table lookup, a vendor function ibis does not model) are -**out of the generic API's scope by design** and serviced by the `upsert_hook` -override; we do not re-introduce a raw-SQL condition string (it would restore -exactly the portability/injection problems the predicate form removes). - -Because the predicate is ibis, **functions and types render per-dialect too** -(`incoming.name.upper()` → `UPPER(...)`), not just column references — probe -output confirmed correct rendering across duckdb/postgres/mysql/mssql/ -snowflake/bigquery/oracle/trino. The matched/not-matched semantics are -equivalent across the two families: a row matching the key but failing the -condition stays matched and is neither updated nor re-inserted (no duplicate). - -`on_duplicate_key` cannot express a per-row update predicate → a non-`None` -`update_condition` raises `ValueError` (validated **before** any -`conflict_action` handling — see §10). - -> **Implementation note (dialect names).** The remapped sub-AST is rendered -> with the live connection's *sqlglot* dialect (`ibis_conn.compiler.dialect`), -> not ibis's backend name — these differ (ibis `mssql` ↔ sqlglot `tsql`). The -> probe used the live compiler dialect throughout. The alias-remap helper -> lives in `_render.py` (§5.2). - -### 6.2 `ON DUPLICATE KEY` documented divergences - -- **Conflict targeting (prove-safe-or-raise preflight):** MySQL/MariaDB `ON - DUPLICATE KEY UPDATE` fires on a collision with **any** unique or primary - key, not a named subset — so a silent contract violation is possible - (`conflict_columns=["external_id"]` updating a row that collided on - `email`). Because that failure **silently corrupts the data the caller - intended**, the `on_duplicate_key` renderer runs a **bounded preflight - validation** (a generic-default safety gate, not a catalog-defined - operation — §2) over `information_schema.STATISTICS` and renders **only when - it can prove the safe case**, failing closed otherwise: - - **exactly one** unique/PK index whose column set **equals** - `conflict_columns`, **and** every such column is `NOT NULL`, **and** the - index has **no prefix** (`SUB_PART IS NULL`) and is **not** a - functional/expression index → proceed; - - any of: a second unique/PK index, a prefix index (`SUB_PART`), a - functional/expression index, or a **nullable** conflict column → **raise - `ValueError`** naming the offending index/column and pointing at the - `upsert_hook` override. (Prefix indexes detect on a truncated value; - nullable unique columns are NULL-distinct in MySQL, so duplicates insert - instead of updating — both would silently violate the contract, so we - refuse rather than guess.) - - `conflict_columns` also governs which columns are excluded from the UPDATE - set (validated non-empty). This is the one MySQL divergence we *fail closed* - on rather than document, because — unlike the `NOTHING` side effects below — - the failure is data corruption. A DDL change between preflight and execution - (another session adding/dropping a unique index) is an accepted - schema-concurrency race, documented in §11. -- **`NOTHING` is not a true no-op:** rendered as `ON DUPLICATE KEY UPDATE - k0=k0` (self-assigning the first key column). Depending on table definition - this may advance `ON UPDATE CURRENT_TIMESTAMP` columns, fire UPDATE - triggers, take update locks, and alter affected-row counts — unlike `ON - CONFLICT DO NOTHING`. We deliberately do **not** use `INSERT IGNORE` (it - also silently suppresses unrelated type / NOT NULL / FK errors). Documented - (§11); strict-skip callers use the override hook. - -## 7. Coverage Matrix (all 20 registry dialects) - -The registry currently has **20** dialects; every one appears below. The -matrix is not hand-counted in tests — the golden-SQL suite **iterates the live -`DIALECTS` registry** (§8.4) and asserts each entry renders or is explicitly -`None`, so adding a 21st dialect *forces* a matrix decision (no hardcoded -count to drift). - -Columns: **Style** = assigned `upsert_style`. **Exec** = engine-execution -confidence: **live** (round-tripped against a real engine) / **render** -(rendered SQL asserted; engine acceptance and semantic support **not** -verified) / **n/a** (`upsert_style=None` → honest `NotImplementedError`). -**Basis** = how the style assignment was established: **verified** (live or -vendor-doc confirmed) / **inferred** (protocol/family compatibility) / -**unverified** (hypothesis pending docs/tests). - -| Dialect | Style | Exec | Basis | Notes | -|---|---|---|---|---| -| sqlite | on_conflict | **live** | verified | in-memory | -| duckdb | on_conflict | **live** | verified | in-memory; also exercises MERGE renderer (duckdb supports MERGE) | -| motherduck | on_conflict | render | inferred | duckdb engine | -| postgres | on_conflict | **live** | verified | Docker; also exercises MERGE renderer (PG15+) | -| mysql | on_duplicate_key | **live** | verified | Docker (mariadb, per Ibis) | -| singlestoredb | on_duplicate_key | render | inferred | MySQL-compatible wire/syntax | -| snowflake | merge | render | verified | vendor MERGE | -| bigquery | merge | render | verified | vendor MERGE | -| mssql | merge | render | verified | vendor MERGE; sp_rename for rename | -| oracle | merge | render | verified | vendor MERGE | -| databricks | merge | render | verified | Delta MERGE | -| exasol | merge | render | inferred | MERGE documented | -| trino | merge | render | inferred | MERGE is **connector-dependent** at runtime — may reject | -| redshift | merge | render | verified | postgres protocol but no `ON CONFLICT`; MERGE (AWS docs, 2023+) | -| risingwave | on_conflict | render | **unverified** | postgres-wire; ON CONFLICT on tables assumed, not confirmed | -| clickhouse | None | n/a | verified | ReplacingMergeTree / ALTER UPDATE — divergent model | -| impala | None | n/a | verified | MERGE only for Iceberg targets | -| materialize | None | n/a | verified | restricts INSERT to write-only txns | -| druid | None | n/a | verified | append-only analytics store | -| pyspark | None | n/a | verified | MERGE only for Delta/Iceberg; Ibis marks notyet | - -`rename_table`: generic sqlglot default for **all** dialects (rendered SQL -asserted across the registry; engine-executed live on -sqlite/duckdb/postgres/mysql). The override hook stays available for any -engine later found to diverge beyond sqlglot's rendering. - -**ON-CONFLICT target aliasing (§6.1, only relevant when `update_condition` is -supplied):** referencing the *existing* row in the condition requires aliasing -the target in the INSERT (`INSERT INTO t AS tgt …`). Postgres and SQLite -support this; the implementation verifies it per `on_conflict`-family dialect -and records a per-dialect capability flag. A dialect that cannot alias its -target *and* is given an `update_condition` raises `ValueError` pointing at the -`upsert_hook` (unconditional upserts are unaffected). This flag is set from -live/golden verification, not assumed. - -**Render coverage is syntax-emission verification only** — it confirms the -exact SQL string sqlglot emits per dialect, *not* that the engine accepts or -semantically supports it. The public-facing support matrix carries the -**Exec** and **Basis** caveats above so consumers do not mistake a green -golden-SQL test for runtime support (e.g. Trino MERGE may render cleanly yet -be rejected by the connector). `unverified` rows are flagged as hypotheses -until a live test or vendor doc upgrades them. - -## 8. Testing - -### 8.1 Live backends — Docker - -A minimal `compose.yaml` at repo root, modeled on Ibis's `docker/` services -but using **stock images** (we need no PostGIS/pgvector/plpython): - -- `postgres`: `postgres:18-alpine`, env `POSTGRES_USER/PASSWORD/DB`, - healthcheck `pg_isready`, port 5432. -- `mysql`: `mariadb:12.1.2` (Ibis's choice; ON-DUPLICATE-KEY compatible), - healthcheck `mariadb-admin ping`, port 3306. - -Connection parameters read from environment with Ibis-compatible defaults -(`IBIS_TEST_POSTGRES_*` / `PG*`, `IBIS_TEST_MYSQL_*`) so the same env works -locally and in CI. - -### 8.2 Skip-if-unreachable fixtures — but fail-closed in CI - -`postgres` / `mysql` fixtures attempt a connection; on failure behaviour is -**environment-gated** to avoid silently-green CI: - -- **Locally** (default): unreachable service → `pytest.skip(...)`, so a - developer without Docker still passes green. -- **In CI**: the workflow sets `MOUNTAINASH_REQUIRE_LIVE_DB=1`; under that flag - an unreachable required service **fails** (not skips). A misconfigured - service password/port then breaks the build instead of silently skipping the - promised live matrix. - -Live tests are marked `@pytest.mark.integration` (existing marker). - -### 8.3 CI - -The existing `python-run-pytest.yml` gains GitHub Actions `services:` -containers for postgres and mariadb (native service-container support; no -compose needed in CI) and sets `MOUNTAINASH_REQUIRE_LIVE_DB=1` for the live -job. A `hatch` script (`test-live`) brings the local compose services up. - -### 8.4 Test layers - -1. **Unit / golden-SQL** (`tests/test_unit/backends/ibis/test_upsert_render.py`, - `test_rename_table_render.py`): the suite **iterates the live `DIALECTS` - registry** (not a hand-listed set) and, for each dialect, asserts the exact - rendered statement for its `upsert_style` — or, for `None`, asserts - `NotImplementedError` — and the rendered `rename_table`. Because it iterates - the registry, a newly added dialect with no decision fails the suite, - keeping the §7 matrix complete by construction. This layer verifies - **emitted SQL only**, not engine acceptance (§7). -2. **Conditional-predicate rendering** (`test_upsert_condition_render.py`): - asserts the §6.1 mechanism is faithful and bounded — a constant predicate, - a compound predicate, a null-check, and a function predicate each render to - the expected ON-CONFLICT and MERGE clauses with correct sentinel→alias - remapping; and out-of-grammar predicates (aggregate, window, subquery/ - `EXISTS`) each raise `ValueError`. Guards against ibis rewriting a predicate - non-faithfully (§11). -3. **MySQL preflight** (`test_upsert_mysql_preflight.py`, live): single-PK - table proceeds; multi-unique, prefix-index, and nullable-unique tables each - raise `ValueError`. -4. **Live integration** (`tests/test_integration/test_write_ops_live.py`): - sqlite + duckdb (in-memory) + postgres + mysql — round-trip upsert - (insert + update + NOTHING + composite key + conditional update via the - predicate form) and rename, asserting data outcomes. -5. **Consolidation regression**: existing `test_add_columns.py` stays green - unchanged after the helper extraction. - -## 9. Verification already performed (sqlglot transpile probe, ibis 12 env) - -- `ALTER TABLE … ADD COLUMN`, `RENAME`, and `MERGE` render correctly - per-dialect from sqlglot (`sp_rename` for tsql, `RENAME` for mysql). -- `INSERT … ON CONFLICT` renders natively for postgres/duckdb/sqlite and is - **not** transpiled to MERGE (confirming the family split is structural, not - a rendering gap). -- Ibis 12 `SQLBackend.upsert` exists (MERGE-based) but its `on` is a single - column — insufficient for the composite natural keys real consumers use — - which is why we render our own, using Ibis's `sge.merge` shape as template. -- **Conditional-predicate rendering (§6.1) probed end-to-end.** A bare - two-table ibis boolean is rejected by ibis (`RelationError: … multiple base - table references`), but `existing.join(incoming, predicate)` → - `compiler.to_sqlglot` → extract the `ON` sub-AST → remap ibis's auto-aliases - (identified by each `exp.Table`'s underlying name, not positionally) → - `.sql(dialect=…)` produces correct, portable output. Verified: a compound - predicate with a function (`incoming.name.upper() != existing.name.upper()`) - rendered to `"EXCLUDED"."…" > "target"."…" AND UPPER(…) <> UPPER(…)` for the - ON CONFLICT alias mapping and `"src"…/"tgt"…` for MERGE, with correct - per-dialect quoting/functions across duckdb/postgres/mysql/mssql/snowflake/ - bigquery/oracle/trino. Implementation depth is bounded to one `_render.py` - helper. - -## 10. Error Handling & Edge Cases - -**Validation precedence (ordered — earlier checks fire first):** - -1. `upsert_style is None` → `NotImplementedError` naming the dialect. -2. Target table absent → `ValueError` (preserves current behaviour). -3. `name`/`database`/rename names not simple (dotted) → `ValueError` via - `_validate_simple_identifier`; dotted qualified names out of scope - (consistent with `add_columns`). -4. `conflict_action` ∉ {UPDATE, NOTHING} → `ValueError`. -5. **`update_condition` is validated unconditionally, before any - `conflict_action`-specific handling** (resolves the precedence ambiguity): - - on `on_duplicate_key` → always `ValueError` (family cannot express a - per-row update predicate), regardless of `conflict_action`; - - violates the accepted predicate grammar (aggregate / window / - subquery / `EXISTS` / third-table — §6.1) → `ValueError`; - - references a column absent from the target schema → `ValueError` (ibis - raises when the predicate binds to the resolved `incoming`/`existing`). -6. `on_duplicate_key` preflight: `conflict_columns` ambiguous / unmatched / - prefix / functional / nullable against the unique-index introspection → - `ValueError` (fail-closed, §6.2). -7. `conflict_action="UPDATE"` with no updatable columns (all columns are keys - and no `update_columns`) → `ValueError` (preserves current behaviour). -8. `conflict_action="NOTHING"` with `update_columns` → warn-and-ignore - (preserves current behaviour); on MERGE, NOTHING simply omits the matched - clause. (`update_condition` is already rejected/handled at step 5, so there - is no NOTHING-plus-condition ambiguity.) - -## 11. Known Limitations - -- **Duplicate source rows are the caller's responsibility.** If the source - frame contains two rows with the same conflict key, behaviour is - engine-divergent: `MERGE` raises a cardinality-violation error on most - engines (e.g. Redshift) when multiple source rows match one target row, - whereas `ON CONFLICT` / `ON DUPLICATE KEY` apply conflicts row-by-row with - order-dependent last-write-wins. The renderer does **not** deduplicate. - Callers must deduplicate the source on the conflict key for deterministic, - portable behaviour. Documented, not handled (an optional `deduplicate=` flag - is a future extension, not in this iteration). -- **MySQL-family conflict targeting** detects on *any* unique index, not - `conflict_columns` (§6.2). The renderer runs a prove-safe-or-raise preflight - and **raises `ValueError`** on ambiguous / unmatched / prefix / functional / - nullable unique-index cases — fail-closed, because the failure would be data - corruption. A DDL change between preflight and execution is an accepted - schema-concurrency race (same class as the general non-atomicity below). The - `upsert_hook` override is the escape hatch for tables that legitimately need - the broader behaviour. -- **Conditional-predicate extraction faithfulness.** §6.1 extracts the join's - `ON` sub-AST as the rendered predicate; ibis could in principle normalise or - fold the predicate during compilation. This is treated as a **test-guarded - assumption**: §8.4 requires predicate-rendering tests across constant, - compound, null-check, and function shapes, plus rejection tests for the - out-of-grammar shapes (aggregate/window/subquery). If a future ibis release - rewrites a predicate non-faithfully, those tests fail loudly. -- **Conditional-predicate grammar boundary.** `update_condition` supports - scalar row predicates over incoming/existing columns, literals, and - ibis-modelled functions (§6.1). Conditions needing correlated `EXISTS`, a - third-table lookup, or a vendor function ibis does not model are **out of the - generic API by design** and serviced by the `upsert_hook` override; a - raw-SQL condition string is deliberately not offered (it would restore the - portability/injection problems the predicate form removes). -- **MySQL-family `conflict_action="NOTHING"` is not a true no-op** (§6.2) — may - fire `ON UPDATE` timestamps / triggers / locks. Documented; not `INSERT - IGNORE`. -- **Not concurrency-safe / not atomic** beyond the engine's own statement - atomicity — single-statement upsert is atomic where the engine makes it so; - no cross-statement transaction is added. -- **Subquery staging** inlines the source data as a compiled `VALUES`/`SELECT` - subquery (Ibis's mechanism), with every column cast to the target type - (§5.5). Very large frames produce large SQL; callers with bulk loads should - use a staging table + `upsert`-from-table directly. Acceptable and matches - upstream Ibis behaviour. -- **Render-only dialects** (matrix §7) are verified at the SQL-emission layer - only, **not** by execution or semantic-support confirmation; first real use - on such an engine may surface engine-specific quirks (or outright rejection, - e.g. connector-dependent Trino MERGE), handled then via the `upsert_hook` - override seam. `unverified`-basis rows (e.g. risingwave) are hypotheses. -- **`update_condition`** renders to different *clauses* per family (`DO UPDATE - … WHERE` vs `WHEN MATCHED AND`), but the observable semantics are equivalent - (§6.1): a key-matched row failing the predicate is left untouched and not - re-inserted. The predicate is ibis-expressed, so functions/types are - portable; the alias-remap mechanism is the one piece of real implementation - depth (verified, §9). - -## 12. Dependency - -- Bump the pin to **`ibis-framework>=12.0.0`** (the env reality; `add_columns` - already shipped against it; lets us reference Ibis's `sge.merge` shape). The - renderers use `sqlglot.expressions` directly, so 12 is not strictly required - for rendering, but aligning removes version drift. -- Correct the stale `ibis-framework == 10.4.0` reference in `CLAUDE.md` to the - actual `>=12.0.0`. - -## 13. Files Changed - -- `dialects/_registry.py` — `UpsertStyle` enum, `upsert_style` field; assign - styles per the matrix; remove the three `upsert_hook=duckdb_family_upsert` - registrations. -- `backends/ibis/_render.py` (new) — shared rendering primitives; the - conditional-predicate compiler (`ConditionPredicate` type; sentinel-named - binding → join → sqlglot AST → sentinel-keyed alias-remap → per-family - clause, §6.1); the **predicate-grammar validator** (rejects aggregate / - window / subquery shapes); and the reserved sentinel names - (`__ma_incoming__` / `__ma_existing__`). -- `backends/ibis/operations.py` — `_generic_rename_table`, `_generic_upsert`, - `_render_on_conflict`, `_render_merge`, `_render_on_duplicate_key` (incl. the - unique-index introspection + fail-closed validation, §6.2); migrate - `_generic_add_columns` to the helper; **delete** `duckdb_family_upsert` — - the duckdb family routes through `_render_on_conflict` via - `upsert_style=ON_CONFLICT`. -- `backends/ibis/backend.py` — `upsert`/`rename_table` dispatch updated to the - hook-or-generic shape; `upsert`'s `update_condition` param retyped from - `str | None` to `ConditionPredicate | None`. -- `compose.yaml` (new) — postgres + mariadb stock services. -- `tests/test_unit/backends/ibis/test_upsert_render.py`, - `test_rename_table_render.py` (new) — golden-SQL per dialect. -- `tests/test_unit/backends/ibis/test_upsert_condition_render.py` (new) — - conditional-predicate rendering + out-of-grammar rejection (§8.4). -- `tests/test_integration/test_upsert_mysql_preflight.py` (new, live) — MySQL - unique-index preflight (proceed / multi-unique / prefix / nullable raises). -- `tests/test_integration/test_write_ops_live.py` (new) — live round-trips. -- `tests/conftest.py` / fixtures — skip-if-unreachable postgres/mysql. -- `.github/workflows/python-run-pytest.yml` — service containers. -- `pyproject.toml` — ibis pin `>=12.0.0`. -- `CLAUDE.md` — correct ibis version note. - -## 14. Out of Scope (tracked elsewhere) - -- `create_index` / `drop_index` / `get_index_exists_sql` / - `get_list_indexes_sql` — capability-divergent, stay hook-required. -- Consumer migration (mountainash-wearables backend swap to postgres) — in - that repo after this ships. This spec's `upsert`-on-postgres is its enabler. -- Live warehouse testing — requires credentials/infra; deferred. diff --git a/docs/superpowers/specs/2026-06-30-generic-default-index-operations-design.md b/docs/superpowers/specs/2026-06-30-generic-default-index-operations-design.md deleted file mode 100644 index 85b24a4..0000000 --- a/docs/superpowers/specs/2026-06-30-generic-default-index-operations-design.md +++ /dev/null @@ -1,389 +0,0 @@ -# Generic-Default Index Operations — Design - -**Date:** 2026-06-30 -**Status:** Design — not yet implemented -**Branch:** `feature/generic-default-index-operations` (off `develop`) -**Predecessor:** [Generic-Default Dialect Operations](2026-06-29-generic-default-dialect-operations-design.md) (PR #91, merged) — reuses its `_render.py` primitives. - -## 1. Goal & frame - -Do for `create_index` / `drop_index` / `index_exists` exactly what PR #91 did for -`upsert` / `rename_table`: replace the duckdb-family-only capability **hooks** with -**generic-default rendering across the registry**, driven by a structured per-dialect -capability descriptor, retiring `duckdb_family_create_index` / -`duckdb_family_drop_index`. - -This is a package-level capability — the complete, correct shape of index DDL for any -conventional-RDBMS consumer of `mountainash-data` — not a helper for one caller. - -### Current state (the problem) - -- `IbisBackend.create_index` / `create_unique_index` / `drop_index` / `index_exists` - exist but are **hook-required**: they raise `NotImplementedError` unless the - `DialectSpec` carries the relevant hook. -- Only **sqlite / duckdb / motherduck** are wired, via `duckdb_family_create_index` / - `duckdb_family_drop_index` (raw `cur.execute`, string-built SQL). The other ~17 - dialects have no index support. -- `drop_index(table_name=…)` types `table_name` as **optional**, which is wrong for the - table-scoped dialects (MySQL/SingleStore/T-SQL) where it is mandatory. -- `duckdb_family_create_index` **warns and downgrades** an unsupported `index_type` to - BTREE, and passes `where_condition` through to DuckDB — which **does not support - partial indexes at all** (latent invalid-SQL bug). - -## 2. Scope - -**In scope:** conventional secondary B-tree index DDL — `CREATE [UNIQUE] INDEX`, -`DROP INDEX`, and existence introspection (`index_exists`) — across every dialect that -has that concept. - -**Out of scope (`index_caps=None` → `NotImplementedError`, documented in the support -matrix):** - -- Stores with no user-facing secondary index: **snowflake, bigquery, redshift, trino, - exasol, impala, druid, pyspark, databricks**. -- **clickhouse** data-skipping indexes (`ALTER TABLE … ADD INDEX … TYPE … GRANULARITY`) - — a wholly different shape. -- **risingwave, materialize** streaming-arrangement indexes — verified as - materialized-view-backed / in-memory arrangements (`USING arrangement`), not secondary - B-trees. May be modelled as their own family in a later spec. - -`create_index_hook` / `drop_index_hook` fields **remain** on `DialectSpec` as an -override-first escape hatch (mirrors how `upsert_hook` was kept), but the -`duckdb_family_*` registrations and the functions themselves are deleted at cutover. - -## 3. Capability model (`backends/ibis/dialects/_registry.py`) - -```python -class DropScope(str, enum.Enum): - SCHEMA_GLOBAL = "schema_global" # DROP INDEX name - TABLE_SCOPED = "table_scoped" # DROP INDEX name ON tbl - - -@dataclass(frozen=True) -class IndexCapability: - drop_scope: DropScope - partial: bool # supports a WHERE filter (partial / filtered index) - native_if_not_exists: bool # engine has CREATE INDEX IF NOT EXISTS - native_if_exists: bool # engine has DROP INDEX IF EXISTS - index_types: frozenset[str] # valid USING values; empty = no USING clause - - -# DialectSpec gains: -index_caps: t.Optional[IndexCapability] = None # None = unsupported -> NotImplementedError -``` - -`native_if_not_exists` and `native_if_exists` are **two separate booleans** because they -genuinely diverge: SQL Server has `DROP INDEX IF EXISTS` but no -`CREATE INDEX IF NOT EXISTS`. - -### Dispatch order (on the backend) - -``` -create_index_hook present? -> call hook (override) -elif index_caps is not None -> generic renderer -else -> NotImplementedError(dialect) -``` -Same three-way shape as upsert. - -### Invariant - -A dialect with `index_caps is not None` **MUST** also set `get_index_exists_sql`. -Emulated idempotency (§6) depends on existence introspection, so a dialect that supports -indexes must be able to introspect them. Enforced by a registry-consistency unit test. - -### Coverage is an all-three-operations contract - -`index_caps is not None` asserts the dialect supports **all of** `create_index`, -`drop_index`, and `index_exists` via the generic path. We do **not** model per-operation -partial coverage (e.g. create-without-drop): no conventional RDBMS in scope offers one -without the others, and the invariant above already binds `exists` to the other two -(emulation needs it). A dialect with genuinely irregular coverage uses the -`create_index_hook` / `drop_index_hook` override path instead — and an override hook is -held to the **same no-silent-degradation contract** (§8) as the generic renderer -(validate-and-raise, never warn-and-downgrade). - -## 4. Support matrix (verified against official vendor docs, 2026-06-30) - -| Dialect | `drop_scope` | `partial` | `native_if_not_exists` / `native_if_exists` | `index_types` | Test tier | -|---|---|---|---|---|---| -| sqlite | SCHEMA_GLOBAL | True | True / True | ∅ | live | -| duckdb | SCHEMA_GLOBAL | **False** | True / True | ∅ | live | -| motherduck | SCHEMA_GLOBAL | False | True / True | ∅ | render-only | -| postgres | SCHEMA_GLOBAL | True | True / True | btree,hash,gist,gin,brin,spgist | live | -| mysql¹ | TABLE_SCOPED | False | **False / False** (emulate) | btree | live (MariaDB) | -| singlestoredb³ | TABLE_SCOPED | False | False / False (emulate) | btree,hash | render-only | -| mssql | TABLE_SCOPED | **True** (filtered) | **False / True** | ∅ | render-only | -| oracle² | SCHEMA_GLOBAL | False | False / False (emulate) | ∅ | render-only | -| snowflake, bigquery, redshift, trino, clickhouse, databricks, exasol, impala, materialize, risingwave, druid, pyspark | — | — | — | — | `None` (NotImplementedError) | - -¹ The single `"mysql"` dialect serves **both** MySQL and MariaDB servers, so it -implements the **intersection** of their behaviours: -- `IF [NOT] EXISTS`: MariaDB has it, **MySQL 8.0/8.4 does not** → emulate (`False`/`False`). -- `index_types`: MariaDB documents `USING {BTREE|HASH|RTREE}`; MySQL is - **storage-engine-dependent** — InnoDB supports only `BTREE` and **silently maps - `USING HASH` to BTREE with a warning** (the exact silent-degradation this design - forbids). The intersection that is universally valid and never silently degraded is - **`{btree}`**. `RTREE`/`HASH`/`SPATIAL` are out of the generic path; a dialect that - needs them registers a `create_index_hook`. -- The live test env is MariaDB, so the emulation code path is genuinely exercised (we - force the dialect path, not the server's latent capability). - -² Oracle gained `IF [NOT] EXISTS` only in Release 19.28+/23ai. To avoid depending on the -server patch level, the dialect emulates. Oracle's "partial index" is partition-level -`INDEXING PARTIAL`, **not** a `WHERE` filter → `partial=False`. - -³ SingleStore's valid `index_type` is **table-type-dependent** (columnstore tables accept -only `HASH`; rowstore accepts `BTREE`/`HASH`). Unlike MySQL/InnoDB, SingleStore **errors** -on an inapplicable type rather than silently mapping it — so the generic path exposes the -documented `{btree,hash}` set and surfaces the engine's error to the caller (not silent -degradation). Render-only; no live SingleStore in the test matrix. - -**Sources (verified 2026-06-30):** -- SQLite — https://sqlite.org/lang_createindex.html , https://sqlite.org/partialindex.html -- DuckDB — https://duckdb.org/docs/stable/sql/statements/create_index (no `WHERE`; has `IF [NOT] EXISTS`) -- MySQL 8.4 — https://dev.mysql.com/doc/refman/8.4/en/create-index.html , https://dev.mysql.com/doc/refman/8.4/en/drop-index.html (no `IF [NOT] EXISTS`; `USING {BTREE|HASH}` engine-dependent, InnoDB=BTREE) -- MariaDB — https://mariadb.com/docs/server/reference/sql-statements/data-definition/create/create-index , .../drop/drop-index (`IF [NOT] EXISTS` native; `USING {BTREE|HASH|RTREE}`) -- SingleStore — https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-index/ , .../drop-index/ (`USING {BTREE|HASH}`; `DROP INDEX … ON tbl`; no `IF [NOT] EXISTS`) -- SQL Server — https://learn.microsoft.com/en-us/sql/t-sql/statements/drop-index-transact-sql (`DROP INDEX [IF EXISTS] … ON tbl`, 2016+) , https://learn.microsoft.com/en-us/sql/relational-databases/indexes/create-filtered-indexes (filtered `WHERE`, restricted grammar; no `CREATE … IF NOT EXISTS`) -- Oracle — https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/CREATE-INDEX.html (`IF [NOT] EXISTS` "only from Release 19.28 and up") , .../23/sqlrf/DROP-INDEX.html -- RisingWave — https://docs.risingwave.com/processing/indexes ("a specialized materialized view") ; Materialize — https://materialize.com/docs/sql/create-index/ (`USING arrangement`, in-memory) - -> Note: `partial` for sqlite/postgres is taken from docs; `index_types` for postgres is -> the documented method set. The live-tier dialects (sqlite, duckdb, postgres, MariaDB) -> have their flags re-confirmed by a probe step in the plan before the value is committed -> — the same empirical discipline PR #91 used (which caught duckdb-no-MERGE). - -## 5. Render path - -### 5.1 New module `backends/ibis/_index.py` - -Index builders + generic dispatchers live in their own module rather than growing the -already-large `operations.py` (files that change together live together). Pure builders -take pre-computed, already-validated parts so registry golden tests render without a live -connection: - -```python -def build_create_index_sql( - *, dialect, target, index_name, cols, unique, - index_type, guard, where_sql, -) -> str: ... - -def build_drop_index_sql( - *, dialect, drop_scope, index_name, target, guard, -) -> str: ... -``` - -- `target` is the already-qualified, already-quoted table reference (via - `_render.qualified_name` + `quote_identifier`). -- `guard` is the pre-resolved native clause: `"IF NOT EXISTS "` / `"IF EXISTS "` / `""` - (empty when emulating — the precheck in §6 supplies idempotency instead). -- `index_type` is rendered as the dialect's `USING ` only when non-empty. - -Generic dispatchers orchestrate validation, emulation precheck, predicate compilation, -and execution: - -```python -def _generic_create_index(con, table_name, columns, *, index_name, unique, - index_type, where, database, if_not_exists) -> None: ... -def _generic_drop_index(con, index_name, *, table_name, database, if_exists) -> None: ... -def _generic_index_exists(con, index_name, *, table_name, database) -> bool: ... -``` - -Identifiers (index name, columns, table) are validated with the existing -`_validate_simple_identifier` allowlist (`_SIMPLE_IDENTIFIER_RE`) and quoted via -`_render.quote_identifier`. `_IBIS_TO_SQLGLOT` maps the ibis backend name to the sqlglot -dialect (mssql→tsql, singlestoredb→singlestore, motherduck→duckdb). - -### 5.2 Partial-index `WHERE` predicate (new single-relation path in `_render.py`) - -The partial `WHERE` is expressed as an **ibis-expression predicate** — consistent with -the `update_condition` redesign in PR #91 — not a raw SQL string. A partial-index filter -is a single-table predicate over the indexed table's own columns (no join), so it needs a -simpler compile path than upsert's two-sentinel join extraction: - -```python -IndexPredicate = t.Callable[["ibis.Table"], "ibis.BooleanColumn"] - -def compile_index_predicate(con, schema, table_name, predicate) -> str: - """Bind one ibis table at `schema`, apply `predicate`, validate, and render - the boolean expression UNQUALIFIED (partial-index WHERE references bare - columns), for the connection's dialect.""" -``` - -- Binds one ibis table at the table's schema (introspected from the live connection, or - passed explicitly in render-only tests). The predicate **may reference any column of the - table, not only the indexed columns** (Postgres and SQLite both allow this); binding the - full schema — not just `cols` — is therefore required. -- Reuses `_render.validate_predicate` (rejects aggregate / window / subquery). This is a - **structural** guard, not a per-dialect grammar check. It does **not** validate function - volatility, and it does **not** model dialect-specific filter restrictions. Where an - engine restricts partial-index predicates more tightly than a general boolean, an - unsupported predicate **fails at execution** with the engine's error (surfaced, not - swallowed) — see the SQL Server note below. -- **Unqualified rendering is AST-level, not string substitution.** After compilation, the - qualifier (table alias / schema) is stripped by walking the sqlglot AST and removing the - `table`/`db`/`catalog` parts of each `Column` node, then rendering. String replacement of - an alias token is explicitly rejected (breaks on quoted/mixed-case aliases, substrings, - and literals). Golden tests must include predicates that compile to schema-qualified and - alias-qualified columns to prove the strip. - -**SQL Server filtered-index caveat:** `mssql` is `partial=True`, but SQL Server filtered -predicates are materially narrower than a general boolean (simple comparison / `IN` forms, -no computed columns, `NULL` only via `IS [NOT] NULL`). The compiler does **not** model -that grammar; mssql partial support is **render-capable but engine-restricted**, and since -mssql is render-only (no live test), a too-rich filtered predicate surfaces as a SQL Server -error at execution. Documented as a known limitation rather than over-validated. - -## 6. Idempotency & emulation - -The convenient defaults stay: `create_index(if_not_exists=True)`, -`drop_index(if_exists=True)`. - -- **Native** (sqlite, duckdb, motherduck, postgres; plus mssql for `DROP`): render the - engine's native `IF [NOT] EXISTS` clause. -- **Emulated** (mysql, singlestoredb, oracle; plus mssql for `CREATE`): when the dialect - lacks the native clause and the caller asked for the guard, run `index_exists` first and - **skip** the CREATE/DROP if the desired state already holds. This is the same - prove-state-then-act shape as the MySQL upsert preflight. - -**Emulation correctness assumptions.** Emulation trusts `index_exists` to be an -authoritative yes/no for the current session and principal. It can be wrong, and the design -accepts each case as the engine's error surfaced to the caller (never swallowed): - -- **TOCTOU:** a concurrent session creates/drops the index between check and act → - duplicate-index (CREATE) or no-such-index (DROP) error. Accepted: DDL is rare and - typically single-writer; the failure is bounded, re-runnable, non-corrupting; and - transactional DDL is unavailable on several engines (MySQL/Oracle auto-commit DDL), so a - lock-wrapped check+act isn't even possible. No catch-and-swallow, no lock wrapping. -- **Catalog privilege:** the principal can `CREATE`/`DROP` but cannot see the index in the - catalog view → `index_exists` returns a false negative and the subsequent native CREATE - may still collide. Documented; not mitigated (a privilege misconfiguration, surfaced as - the engine error). -- **Metadata visibility / isolation:** cached or transaction-isolated catalog metadata may - lag a recent DDL in another session. Emulation assumes the catalog query reflects - committed state for the current session. -- **Auto-commit DDL:** on MySQL/Oracle the precheck and the act are separately committed — - this is *why* the window can't be closed transactionally, and is the basis for accepting - it rather than engineering around it. - -These are limitations of emulated (non-native) idempotency, not of the native path. - -**Injection hardening — exact contract.** `index_exists` introspection SQL is assembled by -interpolating values into a catalog query, so each value is handled explicitly: - -- **Identifier-validated then literal-escaped** (allowlist `_validate_simple_identifier` → - `sqlglot exp.Literal.string`): `index_name`, `table_name`, and the resolved schema / - database value — including any **default schema/catalog obtained from the connection** - when `database` is omitted (that connection-derived value is validated too, never trusted - blindly). -- **Identifier-validated then quoted** (allowlist → `_render.quote_identifier`): the column - names and the table reference used in the CREATE/DROP statements themselves. -- **Intentional exclusions:** the allowlist (`_SIMPLE_IDENTIFIER_RE`) rejects identifiers - with spaces, dots, mixed-case-requiring quotes, leading digits, and SQL Server temp-table - prefixes (`#`, `##`). Names that need those are **out of scope** for the generic path and - must be reached via an override hook. This is the same gate as the PR #91 final-review - injection fix; it is deliberately stricter than the engines' full identifier grammar. - -## 7. Public API (clean break — pre-release, no downstream consumers) - -```python -IndexPredicate = Callable[[ibis.Table], ibis.BooleanColumn] - -def create_index( - self, table_name, columns, *, index_name=None, unique=False, - index_type=None, where=None, database=None, if_not_exists=True, -) -> IbisBackend: ... - -def create_unique_index( # unchanged — delegates with unique=True - self, table_name, columns, *, index_name=None, where=None, database=None, -) -> IbisBackend: ... - -def drop_index( - self, index_name, *, table_name=None, database=None, if_exists=True, -) -> IbisBackend: ... -``` - -Changes from today: -- `where_condition: str | None` → **`where: IndexPredicate | None`** (raw SQL string → - injection-safe, dialect-portable ibis predicate). -- `index_type` is **validated** against `caps.index_types` (was: warn-and-downgrade). -- `drop_index` requires `table_name` **when the resolved dialect is `TABLE_SCOPED`**. - -## 8. Error handling — no silent degradation - -| Condition | Behaviour | -|---|---| -| `index_caps is None` | `NotImplementedError(dialect)` | -| `index_type` not in `caps.index_types` | **`ValueError`** (retires the warn-and-downgrade) | -| `where=` predicate but `caps.partial is False` | `ValueError` | -| `drop_scope is TABLE_SCOPED` and `table_name is None` | `ValueError` (fixes today's wrongly-optional param) | -| `if_not_exists` / `if_exists` requested, no native support | **emulate** via `index_exists` precheck (§6) | -| predicate references aggregate / window / subquery | `ValueError` (via `validate_predicate`) | -| identifier fails the allowlist | `ValueError` | - -## 9. Testing - -- **Golden (render-only):** parametrize over - `{n: s for n, s in DIALECTS.items() if s.index_caps}` and assert per-dialect statement - shape — `CREATE [UNIQUE] INDEX` body, the drop-scope clause (`ON tbl` present iff - `TABLE_SCOPED`), guard present/absent per `native_*`, `USING ` rendering, - partial-`WHERE` only where `partial`, identifier quoting. -- **Golden — `index_exists` introspection SQL (render-only, every dialect's - `get_index_exists_sql`):** for all 8 conventional dialects assert the rendered catalog - query — string-literal escaping of `index_name`/`table_name`/schema (a `'`-bearing name - yields an escaped literal, not a broken query), table-scoped vs schema-global matching - (predicate includes the table for mysql/singlestore/mssql), default-schema behaviour when - `database` is omitted, and that a malicious / disallowed identifier is **rejected by the - allowlist** before any SQL is built. -- **Golden — predicate qualifier strip:** predicates that compile to alias-qualified and - schema-qualified columns render **unqualified** (proves the AST-level strip, §5.2), and a - predicate referencing a non-indexed column compiles (Postgres/SQLite allow it). -- **Live** (sqlite / duckdb / postgres / MariaDB, via the existing `compose.yaml`): - - create → `index_exists` (True) → drop → `index_exists` (False) round-trip. - - partial index on **postgres** and **sqlite** (`where=lambda t: t.active == True`). - - table-scoped drop on **MariaDB** (`DROP INDEX … ON tbl`). - - **emulated idempotency**: double `create_index(if_not_exists=True)` on MariaDB is a - no-op (exercises the `mysql`-dialect emulation path); `drop_index(if_exists=True)` of - an absent index is a no-op. - - `index_type` validation raises; `where` on duckdb (`partial=False`) raises. -- **Registry-consistency:** the §3 invariant (every `index_caps` dialect has - `get_index_exists_sql`); every `index_caps` dialect maps to a known sqlglot dialect. -- Fail-closed under `MOUNTAINASH_REQUIRE_LIVE_DB=1`, same as PR #91. TOCTOU is **not** - tested (accepted window). - -## 10. Cutover - -- Delete `duckdb_family_create_index` / `duckdb_family_drop_index` and their 3 - registrations (sqlite, duckdb, motherduck); these dialects now flow through the generic - renderer. -- Remove the `index_type` warn-and-downgrade test. -- Implement `get_index_exists_sql` for the dialects that newly need it for emulation and - round-trip tests (postgres, mysql, mssql, oracle, singlestoredb) — genuinely per-dialect - introspection SQL (`pg_indexes`, `information_schema.STATISTICS`, `sys.indexes`, - `user_indexes`), each literal-escaped and identifier-validated. -- **`where_condition` removal audit:** grep the repo and docs for `where_condition` (and - any callsite of `create_index`/`create_unique_index`) and update each to the `where` - predicate. Because the new signatures take no `**kwargs`, a leftover - `where_condition=...` raises a natural `TypeError` at call time — the audit ensures no - internal caller or doc example still passes it. (Pre-release, no external consumers, so no - deprecation shim — but the audit is mandatory, not assumed.) - -## 11. File structure - -| File | Change | -|---|---| -| `backends/ibis/dialects/_registry.py` | Add `DropScope`, `IndexCapability`, `index_caps` field + per-dialect assignment; delete `duckdb_family_*` registrations; add `get_index_exists_sql` for the new dialects | -| `backends/ibis/_index.py` | **New** — pure builders + generic dispatchers | -| `backends/ibis/_render.py` | Add `compile_index_predicate` (single-relation path) | -| `backends/ibis/operations.py` | Delete `duckdb_family_create_index` / `duckdb_family_drop_index` | -| `backends/ibis/backend.py` | `create_index` / `drop_index` / `index_exists` → hook→generic→NotImplementedError dispatch; table-scoped `table_name` validation; `where` predicate param | -| `tests/test_unit/backends/ibis/test_index_render.py` | **New** — golden + validation | -| `tests/test_unit/backends/ibis/test_index_capability_registry.py` | **New** — invariant + matrix consistency | -| `tests/test_integration/test_index_ops_live.py` | **New** — live round-trip + emulation | - -## 12. Out of scope (tracked) - -- ClickHouse data-skipping and RisingWave/Materialize streaming indexes — possible future - per-family specs. -- `list_indexes` generalization (`get_list_indexes_sql` hook) — not required by this work; - leave as-is. diff --git a/src/mountainash_data/core/settings/bigquery.py b/src/mountainash_data/core/settings/bigquery.py index 1e692aa..9275ae7 100644 --- a/src/mountainash_data/core/settings/bigquery.py +++ b/src/mountainash_data/core/settings/bigquery.py @@ -1,6 +1,6 @@ """BigQuery backend settings. -Spec: ``docs/superpowers/specs/2026-04-15-settings-audit/bigquery.md``. +Spec: ``mountainash-central/04.planning/mountainash-data/superpowers/specs/2026-04-15-settings-audit/bigquery.md``. Ibis: ``ibis.backends.bigquery.do_connect`` """ diff --git a/src/mountainash_data/core/settings/duckdb.py b/src/mountainash_data/core/settings/duckdb.py index f3efd9b..39e6e42 100644 --- a/src/mountainash_data/core/settings/duckdb.py +++ b/src/mountainash_data/core/settings/duckdb.py @@ -1,6 +1,6 @@ """DuckDB backend settings. -Spec: audit report ``docs/superpowers/specs/2026-04-15-settings-audit/duckdb.md``. +Spec: audit report ``mountainash-central/04.planning/mountainash-data/superpowers/specs/2026-04-15-settings-audit/duckdb.md``. Driver: https://duckdb.org/docs/current/configuration/overview.html Ibis: ``ibis.backends.duckdb.do_connect(database=':memory:', read_only=False, extensions=None, **config)`` diff --git a/src/mountainash_data/core/settings/motherduck.py b/src/mountainash_data/core/settings/motherduck.py index ea12aa9..044799f 100644 --- a/src/mountainash_data/core/settings/motherduck.py +++ b/src/mountainash_data/core/settings/motherduck.py @@ -1,6 +1,6 @@ """MotherDuck backend settings. -Spec: audit report ``docs/superpowers/specs/2026-04-15-settings-audit/motherduck.md``. +Spec: audit report ``mountainash-central/04.planning/mountainash-data/superpowers/specs/2026-04-15-settings-audit/motherduck.md``. Driver auth docs: https://motherduck.com/docs/getting-started/connect-query-from-python/installation-authentication/ Ibis: routes via the duckdb backend (``rides_on="duckdb"``). diff --git a/src/mountainash_data/core/settings/mssql.py b/src/mountainash_data/core/settings/mssql.py index d8a123a..f343105 100644 --- a/src/mountainash_data/core/settings/mssql.py +++ b/src/mountainash_data/core/settings/mssql.py @@ -1,6 +1,6 @@ """MSSQL backend settings. -Spec: ``docs/superpowers/specs/2026-04-15-settings-audit/mssql.md``. +Spec: ``mountainash-central/04.planning/mountainash-data/superpowers/specs/2026-04-15-settings-audit/mssql.md``. Driver: PyODBC connect + ODBC Driver 17/18 for SQL Server. """ diff --git a/src/mountainash_data/core/settings/mysql.py b/src/mountainash_data/core/settings/mysql.py index 3565283..3c5708b 100644 --- a/src/mountainash_data/core/settings/mysql.py +++ b/src/mountainash_data/core/settings/mysql.py @@ -1,6 +1,6 @@ """MySQL backend settings. -Spec: ``docs/superpowers/specs/2026-04-15-settings-audit/mysql.md``. +Spec: ``mountainash-central/04.planning/mountainash-data/superpowers/specs/2026-04-15-settings-audit/mysql.md``. Driver: https://mysqlclient.readthedocs.io/user_guide.html#functions-and-attributes Ibis: ``ibis.backends.mysql.do_connect(host='localhost', user=None, password=None, port=3306, autocommit=True, **kwargs)`` diff --git a/src/mountainash_data/core/settings/postgresql.py b/src/mountainash_data/core/settings/postgresql.py index d7d5dd5..f9f29a9 100644 --- a/src/mountainash_data/core/settings/postgresql.py +++ b/src/mountainash_data/core/settings/postgresql.py @@ -1,6 +1,6 @@ """PostgreSQL backend settings. -Spec: audit report ``docs/superpowers/specs/2026-04-15-settings-audit/postgresql.md``. +Spec: audit report ``mountainash-central/04.planning/mountainash-data/superpowers/specs/2026-04-15-settings-audit/postgresql.md``. Driver: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS Ibis: ``ibis.backends.postgres.do_connect(host, user, password, port=5432, database, schema, autocommit=True, **kwargs)`` (psycopg). diff --git a/src/mountainash_data/core/settings/pyiceberg_rest.py b/src/mountainash_data/core/settings/pyiceberg_rest.py index c72f754..f2d846f 100644 --- a/src/mountainash_data/core/settings/pyiceberg_rest.py +++ b/src/mountainash_data/core/settings/pyiceberg_rest.py @@ -1,6 +1,6 @@ """PyIceberg REST catalog backend settings. -Spec: ``docs/superpowers/specs/2026-04-15-settings-audit/pyiceberg_rest.md``. +Spec: ``mountainash-central/04.planning/mountainash-data/superpowers/specs/2026-04-15-settings-audit/pyiceberg_rest.md``. Driver: https://py.iceberg.apache.org/configuration/ """ diff --git a/src/mountainash_data/core/settings/pyspark.py b/src/mountainash_data/core/settings/pyspark.py index 295c62b..8daf00c 100644 --- a/src/mountainash_data/core/settings/pyspark.py +++ b/src/mountainash_data/core/settings/pyspark.py @@ -1,6 +1,6 @@ """PySpark backend settings. -Spec: audit report ``docs/superpowers/specs/2026-04-15-settings-audit/pyspark.md``. +Spec: audit report ``mountainash-central/04.planning/mountainash-data/superpowers/specs/2026-04-15-settings-audit/pyspark.md``. Ibis: ``ibis.backends.pyspark.do_connect(session=None, mode='batch', **kwargs)`` where kwargs flow to ``SparkSession.builder.config(**kwargs)``. diff --git a/src/mountainash_data/core/settings/redshift.py b/src/mountainash_data/core/settings/redshift.py index f02e19d..2d21403 100644 --- a/src/mountainash_data/core/settings/redshift.py +++ b/src/mountainash_data/core/settings/redshift.py @@ -1,6 +1,6 @@ """Redshift backend settings. -Spec: ``docs/superpowers/specs/2026-04-15-settings-audit/redshift.md``. +Spec: ``mountainash-central/04.planning/mountainash-data/superpowers/specs/2026-04-15-settings-audit/redshift.md``. Driver: redshift_connector OR psycopg (via Ibis postgres). Endpoint resolution via boto3 ``describe_clusters`` is a Phase-4 follow-up. """ diff --git a/src/mountainash_data/core/settings/snowflake.py b/src/mountainash_data/core/settings/snowflake.py index 66ad373..8ba9a8f 100644 --- a/src/mountainash_data/core/settings/snowflake.py +++ b/src/mountainash_data/core/settings/snowflake.py @@ -1,6 +1,6 @@ """Snowflake backend settings. -Spec: ``docs/superpowers/specs/2026-04-15-settings-audit/snowflake.md``. +Spec: ``mountainash-central/04.planning/mountainash-data/superpowers/specs/2026-04-15-settings-audit/snowflake.md``. Driver: https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-api """ diff --git a/src/mountainash_data/core/settings/sqlite.py b/src/mountainash_data/core/settings/sqlite.py index 7790d90..8ee0f36 100644 --- a/src/mountainash_data/core/settings/sqlite.py +++ b/src/mountainash_data/core/settings/sqlite.py @@ -1,6 +1,6 @@ """SQLite backend settings. -Spec: audit report ``docs/superpowers/specs/2026-04-15-settings-audit/sqlite.md``. +Spec: audit report ``mountainash-central/04.planning/mountainash-data/superpowers/specs/2026-04-15-settings-audit/sqlite.md``. Driver: https://docs.python.org/3/library/sqlite3.html#sqlite3.connect Ibis: ``ibis.backends.sqlite.do_connect(database, type_map=None)`` """ diff --git a/src/mountainash_data/core/settings/trino.py b/src/mountainash_data/core/settings/trino.py index 634fb47..c83b18c 100644 --- a/src/mountainash_data/core/settings/trino.py +++ b/src/mountainash_data/core/settings/trino.py @@ -1,6 +1,6 @@ """Trino backend settings. -Spec: ``docs/superpowers/specs/2026-04-15-settings-audit/trino.md``. +Spec: ``mountainash-central/04.planning/mountainash-data/superpowers/specs/2026-04-15-settings-audit/trino.md``. Driver: https://github.com/trinodb/trino-python-client/blob/master/trino/dbapi.py Ibis: ``ibis.backends.trino.do_connect`` """