Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 8 additions & 29 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co

## Project Overview

**mountainash-data** provides physical access to backend data services — relational databases via Ibis, and Iceberg table-format catalogs via PyIceberg. It collapses what was previously 13 per-dialect connection classes into a data-driven `DialectSpec` registry, exposes clean `Backend` / `Connection` protocols, and provides factories and a high-level facade (`DatabaseUtils`).
**mountainash-data** provides physical access to backend data services — relational databases via Ibis behind a backend-agnostic `Backend` protocol ready for additional backends. It collapses what was previously 13 per-dialect connection classes into a data-driven `DialectSpec` registry, exposes clean `Backend` / `Connection` protocols, and provides factories and a high-level facade (`DatabaseUtils`).

## Planning, Specs & Principles (live in mountainash-central)

Expand Down Expand Up @@ -63,11 +63,6 @@ tests, and this `CLAUDE.md` live in this repo's tree.
- `BaseIbisOperations` + per-dialect subclasses in `operations.py`
- `DialectSpec` registry in `dialects/`

7. **Iceberg backend** (`src/mountainash_data/backends/iceberg/`)
- `IcebergBackend` — catalog-type registry (currently: `"rest"`)
- Connection, operations, and inspection classes
- Requires optional `pyiceberg` dependency

### Package Structure

```
Expand All @@ -83,22 +78,15 @@ src/mountainash_data/
│ ├── settings/ # Per-dialect auth settings (pydantic)
│ └── factories/ # ConnectionFactory, OperationsFactory, SettingsFactory
└── backends/
├── ibis/
│ ├── backend.py # IbisBackend + IbisConnection (new-style)
│ ├── connection.py # BaseIbisConnection + dialect subclasses
│ ├── operations.py # BaseIbisOperations + dialect subclasses
│ ├── inspect.py # Ibis-specific inspection helpers
│ └── dialects/ # DialectSpec registry (data-driven)
└── iceberg/
├── backend.py # IcebergBackend + catalog registry
├── connection.py # IcebergConnectionBase
├── operations.py # IcebergOperationsBase
├── inspect.py # Iceberg inspection helpers
└── catalogs/ # Per-catalog implementations
└── ibis/
├── backend.py # IbisBackend + IbisConnection (new-style)
├── connection.py # BaseIbisConnection + dialect subclasses
├── operations.py # BaseIbisOperations + dialect subclasses
├── inspect.py # Ibis-specific inspection helpers
└── dialects/ # DialectSpec registry (data-driven)
```

### Optional Dependencies (extras)
- **pyiceberg**: Required for `IcebergBackend`
- **postgres**: PostgreSQL support (psycopg2-binary, ibis-framework[postgres])
- **mssql**: SQL Server support (pyodbc, ibis-framework[mssql])
- **snowflake**: Snowflake support (snowflake-connector-python, ibis-framework[snowflake])
Expand Down Expand Up @@ -196,7 +184,6 @@ tests/
├── test_unit/
│ ├── core/ # Protocol, inspection tests
│ ├── backends/ibis/ # IbisBackend tests
│ ├── backends/iceberg/ # IcebergBackend tests
│ ├── factories/ # Factory tests
│ ├── databases/ # Legacy-path tests (updated to new paths)
│ ├── test_database_utils.py # DatabaseUtils tests
Expand All @@ -219,7 +206,7 @@ tests/
## Usage Patterns

```python
from mountainash_data import IbisBackend, IcebergBackend
from mountainash_data import IbisBackend
from mountainash_data.core.settings import (
SQLiteAuthSettings,
NoAuth,
Expand Down Expand Up @@ -250,14 +237,6 @@ try:
# rel = ma.relation(ibis_table) # compiles against Ibis automatically
finally:
conn.close()

# Iceberg backend (requires pyiceberg)
ice = IcebergBackend(catalog="rest", uri="http://localhost:8181")
ice_conn = ice.connect()
try:
namespaces = ice_conn.list_namespaces()
finally:
ice_conn.close()
```

## Development Environments
Expand Down
22 changes: 7 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
Mountain Ash - Data

Physical access to backend data services — relational databases via Ibis,
and Iceberg table-format catalogs via PyIceberg.
behind a backend-agnostic `Backend` protocol ready for additional backends.



Expand Down Expand Up @@ -83,18 +83,12 @@ src/mountainash_data/
│ ├── settings/ # Per-dialect auth settings (pydantic)
│ └── factories/ # ConnectionFactory, OperationsFactory, SettingsFactory
└── backends/
├── ibis/ # IbisBackend — 12-dialect registry
│ ├── backend.py # IbisBackend + IbisConnection
│ ├── connection.py # BaseIbisConnection + per-dialect subclasses
│ ├── operations.py # BaseIbisOperations + per-dialect subclasses
│ ├── inspect.py # Ibis-specific inspection helpers
│ └── dialects/ # DialectSpec registry (data-driven)
└── iceberg/ # IcebergBackend — PyIceberg catalogs
├── backend.py # IcebergBackend + catalog registry
├── connection.py # IcebergConnectionBase
├── operations.py # IcebergOperationsBase
├── inspect.py # Iceberg inspection helpers
└── catalogs/ # Per-catalog implementations (rest, …)
└── ibis/ # IbisBackend — 12-dialect registry
├── backend.py # IbisBackend + IbisConnection
├── connection.py # BaseIbisConnection + per-dialect subclasses
├── operations.py # BaseIbisOperations + per-dialect subclasses
├── inspect.py # Ibis-specific inspection helpers
└── dialects/ # DialectSpec registry (data-driven)
```

### Public API
Expand All @@ -104,7 +98,6 @@ from mountainash_data import (
Backend, # Protocol: what every backend must implement
Connection, # Protocol: what every connection must implement
IbisBackend, # Ibis-style relational backends (sqlite, duckdb, postgres, …)
IcebergBackend, # Iceberg-style table-format catalogs (pyiceberg required)
CatalogInfo, # Physical catalog metadata
NamespaceInfo, # Physical namespace/schema metadata
TableInfo, # Physical table metadata
Expand All @@ -118,7 +111,6 @@ from mountainash_data import (

### Optional Dependencies

- **pyiceberg**: Required for `IcebergBackend`. Not installed by default.
- **postgres**: `psycopg2-binary` + `ibis-framework[postgres]`
- **mssql**: `pyodbc` + `ibis-framework[mssql]`
- **snowflake**: `snowflake-connector-python` + `ibis-framework[snowflake]`
Expand Down
43 changes: 18 additions & 25 deletions docs/PROJECT_OVERVIEW.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Provides unified database connections and dataframe abstractions for multiple ba

## Architecture
The package is built on a layered architecture with three main components:
1. **Database Connections Layer** - Abstracts database connections using Ibis framework and PyIceberg
1. **Database Connections Layer** - Abstracts database connections using the Ibis framework
2. **DataFrame Abstraction Layer** - Provides unified dataframe interface across multiple backends
3. **Utilities Layer** - Supporting utilities for data transformation, mapping, and conversion

Expand All @@ -17,31 +17,25 @@ src/mountainash_data/
├── databases/ # Database connection layer
│ ├── __init__.py
│ ├── base_db_connection.py # Abstract base connection
│ ├── ibis/ # Ibis-based connections
│ │ ├── __init__.py
│ │ ├── base_ibis_connection.py
│ │ ├── constants.py
│ │ ├── ibis_connection_factory.py
│ │ └── connections/ # Specific backend implementations
│ │ ├── __init__.py
│ │ ├── bigquery_ibis_connection.py
│ │ ├── duckdb_ibis_connection.py
│ │ ├── motherduck_ibis_connection.py
│ │ ├── mssql_ibis_connection.py
│ │ ├── mysql_ibis_connection.py
│ │ ├── oracle_ibis_connection.py
│ │ ├── postgres_ibis_connection.py
│ │ ├── pyspark_ibis_connection.py
│ │ ├── redshift_ibis_connection.py
│ │ ├── snowflake_ibis_connection.py
│ │ ├── sqlite_ibis_connection.py
│ │ └── trino_ibis_connection.py
│ └── pyiceberg/ # PyIceberg support
│ └── ibis/ # Ibis-based connections
│ ├── __init__.py
│ ├── base_pyiceberg_connection.py
│ └── connections/
│ ├── base_ibis_connection.py
│ ├── constants.py
│ ├── ibis_connection_factory.py
│ └── connections/ # Specific backend implementations
│ ├── __init__.py
│ └── pyiceberg_rest_connection.py
│ ├── bigquery_ibis_connection.py
│ ├── duckdb_ibis_connection.py
│ ├── motherduck_ibis_connection.py
│ ├── mssql_ibis_connection.py
│ ├── mysql_ibis_connection.py
│ ├── oracle_ibis_connection.py
│ ├── postgres_ibis_connection.py
│ ├── pyspark_ibis_connection.py
│ ├── redshift_ibis_connection.py
│ ├── snowflake_ibis_connection.py
│ ├── sqlite_ibis_connection.py
│ └── trino_ibis_connection.py
├── dataframes/ # DataFrame abstraction layer
│ ├── __init__.py
│ ├── base_dataframe.py # Abstract dataframe interface
Expand Down Expand Up @@ -101,7 +95,6 @@ Core package providing unified data access layer with key classes:
Database connection abstraction supporting multiple backends:
- **Base Layer**: `BaseDBConnection` abstract interface
- **Ibis Layer**: Connection implementations for 12+ database backends
- **PyIceberg Layer**: Data lake connectivity via Apache Iceberg

### dataframes/
DataFrame abstraction and utilities:
Expand Down
2 changes: 0 additions & 2 deletions docs/assets/package_overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
1. **Database Connections Layer** (`src/mountainash_data/databases/`)
- Base database connection abstraction (`BaseDBConnection`)
- Ibis-based connections supporting multiple backends (SQLite, DuckDB, PostgreSQL, SQL Server, etc.)
- PyIceberg support for data lake operations
- Connection factory pattern for backend instantiation

2. **DataFrame Abstraction Layer** (`src/mountainash_data/dataframes/`)
Expand Down Expand Up @@ -43,7 +42,6 @@ src/mountainash_data/
│ │ │ ├── postgres_ibis_connection.py
│ │ │ └── [other backends...]
│ │ └── ibis_connection_factory.py
│ └── pyiceberg/ # PyIceberg support
├── dataframes/ # DataFrame abstraction layer
│ ├── base_dataframe.py # Abstract dataframe interface
│ ├── ibis_dataframe.py # Ibis dataframe implementation
Expand Down
4 changes: 1 addition & 3 deletions docs/dbt_integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -526,7 +526,7 @@
| Snapshots | mountainash-data | mountainash-dataframes | Operations + Join for SCD |
| Seeds (Enhanced) | mountainash-dataframes | - | Convert from structured Python data |
| Materializations | mountainash-data | - | Connection, Operations |
| External Tables | mountainash-data | mountainash-dataframes | PyIceberg, External file ops |
| External Tables | mountainash-data | mountainash-dataframes | External file ops |

---
🎯 Integration Architecture Pattern
Expand Down Expand Up @@ -598,7 +598,6 @@

Phase 4: Advanced Features (Innovation)
9. ✅ Custom materializations using mountainash-data connections
10. ✅ PyIceberg integration for data lakes

---
This alignment creates a powerful synergy where:
Expand Down Expand Up @@ -1640,7 +1639,6 @@
- ✅ PostgreSQL, MySQL, Oracle, MS SQL Server
- ✅ Snowflake, BigQuery, Redshift
- ✅ PySpark, Trino
- ✅ PyIceberg (data lakes)

dbt Integration Points:
- ✅ Incremental model upserts
Expand Down
15 changes: 0 additions & 15 deletions docs/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ pip install mountainash-data[snowflake] # Snowflake
pip install mountainash-data[bigquery] # BigQuery
pip install mountainash-data[pyspark] # Apache Spark
pip install mountainash-data[trino] # Trino
pip install mountainash-data[pyiceberg] # Iceberg catalogs
```

SQLite and DuckDB work out of the box — no extra needed.
Expand Down Expand Up @@ -214,17 +213,3 @@ with IbisBackend(dialect="duckdb") as backend:
| `druid` | `druid://` | |
| `pyspark` | `pyspark://` | Requires `[pyspark]` extra |

---

## Iceberg catalogs (optional)

Requires `pip install mountainash-data[pyiceberg]`.

```python
from mountainash_data import IcebergBackend

with IcebergBackend(catalog="rest", uri="http://localhost:8181") as backend:
namespaces = backend.list_namespaces()
tables = backend.list_tables(namespace="analytics")
info = backend.inspect_table("events", namespace="analytics")
```
12 changes: 1 addition & 11 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -102,18 +102,8 @@ exclude_lines = ["no cov", "if __name__ == .__main__:", "if TYPE_CHECKING:"]

[tool.mypy]
# Internal mountainash siblings ship no py.typed marker, and optional drivers
# (trino/google/pyiceberg/mountainash_dataframes) are not installed in the type
# (trino/google/mountainash_dataframes) are not installed in the type
# env — treat all unresolved/untyped third-party imports as Any rather than noise.
ignore_missing_imports = true
disable_error_code = ["import-untyped"]

# Pre-existing type debt in the Iceberg backend (catalog_backend None-narrowing,
# pyiceberg Any-typed surfaces, hook-signature mismatches) — NOT introduced by
# the auth-client migration and in code paths that require the optional pyiceberg
# + mountainash_dataframes stack to exercise. Carved out for a separate Iceberg
# type-hardening pass; the auth-migration code (factories/adapters/registry/ibis)
# is type-checked normally.
[[tool.mypy.overrides]]
module = "mountainash_data.backends.iceberg.*"
ignore_errors = true

7 changes: 0 additions & 7 deletions src/mountainash_data/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
Public API:
Backend — protocol (core.protocol)
IbisBackend — ibis-style relational backends (backends.ibis.backend)
IcebergBackend — iceberg-style table-format catalogs (backends.iceberg.backend)
CatalogInfo, NamespaceInfo, TableInfo, ColumnInfo — inspection model
"""

Expand All @@ -18,11 +17,6 @@
from mountainash_data.core.namespace import Namespace, NamespaceLike
from mountainash_data.backends.ibis.backend import IbisBackend

try:
from mountainash_data.backends.iceberg.backend import IcebergBackend
except ImportError:
IcebergBackend = None # type: ignore[assignment,misc]

__all__ = [
"__version__",
"Backend",
Expand All @@ -33,5 +27,4 @@
"Namespace",
"NamespaceLike",
"IbisBackend",
"IcebergBackend",
]
Empty file.
Loading
Loading