Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 15 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -596,14 +596,14 @@ Comprehensive comparison of pandas/DataFrame typing and validation tools. **type

| Feature | typedframes | Pandera | Great Expectations | strictly_typed_pandas | pandas-stubs | dataenforce | pandas-type-checks | StaticFrame | narwhals | dataframely | patito |
|---------------------------------|------------------------|-------------|--------------------|-----------------------|--------------|-------------|--------------------|------------------|----------|------------------|------------------|
| **Version tested** | 0.2.0 | 0.29.0 | 1.4.3 | 0.3.6 | 3.0.0 | 0.1.2 | 1.1.3 | 3.7.0 | 2.16.0 | | — |
| **Version tested** | 0.2.1 | 0.29.0 | 1.18.0 | 0.3.7 | 3.0.3 | 0.1.2 | 1.1.3 | 4.1.0 | 2.22.1 | 2.10.1 | 0.8.6 |
| **Analysis Type** |
| When errors are caught | **Static (lint-time)** | Runtime | Runtime | Static + Runtime | Static | Runtime | Runtime | Static + Runtime | Runtime | Runtime | Runtime |
| When errors are caught | **Static (lint-time)** | Runtime | Runtime | Runtime | Static | Runtime | Runtime | Runtime | Runtime | Runtime | Runtime |
| **Static Analysis (our focus)** |
| Mypy plugin | ✅ Yes | ⚠️ Limited | ❌ No | ✅ Yes | ✅ Yes | ❌ No | ❌ No | ⚠️ Basic | ❌ No | ❌ No | ❌ No |
| Mypy plugin | ✅ Yes | ⚠️ Limited | ❌ No | ❌ No | ✅ Yes | ❌ No | ❌ No | ⚠️ Basic | ❌ No | ❌ No | ❌ No |
| Standalone checker | ✅ Rust (~1ms) | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
| Column name checking | ✅ Yes | ⚠️ Limited | ❌ No | ✅ Yes | ❌ No | ❌ No | ❌ No | ✅ Yes | ❌ No | ❌ No | ❌ No |
| Column type checking | ✅ Yes | ⚠️ Limited | ❌ No | ✅ Yes | ❌ No | ❌ No | ❌ No | ✅ Yes | ❌ No | ❌ No | ❌ No |
| Column name checking | ✅ Yes | ⚠️ Limited | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
| Column type checking | ✅ Yes | ⚠️ Limited | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
| Typo suggestions | ✅ Yes | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
| **Runtime Validation** |
| Data validation | ❌ No | ✅ Excellent | ✅ Excellent | ✅ typeguard | ❌ No | ✅ Yes | ✅ Yes | ✅ Yes | ❌ No | ✅ Yes | ✅ Yes |
Expand All @@ -626,8 +626,9 @@ Comprehensive comparison of pandas/DataFrame typing and validation tools. **type
but has limitations—column access via `df["column"]` is not validated, and schema mismatches between functions may not
be caught.

- **[strictly_typed_pandas](https://strictly-typed-pandas.readthedocs.io/)** (v0.3.6): Provides `DataSet[Schema]` type
hints with mypy support. No standalone checker. No polars support. Runtime validation via typeguard.
- **[strictly_typed_pandas](https://strictly-typed-pandas.readthedocs.io/)** (v0.3.7): Provides `DataSet[Schema]` type
hints for runtime validation via typeguard. Despite documentation implying mypy support, there is no mypy plugin —
column access errors are not caught statically. No standalone checker. No polars support.

- **[pandas-stubs](https://github.com/pandas-dev/pandas-stubs)** (v3.0.0): Official pandas type stubs. Provides
API-level types but no column-level checking.
Expand All @@ -638,27 +639,28 @@ Comprehensive comparison of pandas/DataFrame typing and validation tools. **type
- **[pandas-type-checks](https://pypi.org/project/pandas-type-checks/)** (v1.1.3): Runtime validation decorator. No
static analysis.

- **[StaticFrame](https://github.com/static-frame/static-frame)** (v3.7.0): Alternative immutable DataFrame library with
built-in static typing. Not compatible with pandas/polars—requires using StaticFrame's own DataFrame implementation.
- **[StaticFrame](https://github.com/static-frame/static-frame)** (v4.1.0): Alternative immutable DataFrame library.
Not compatible with pandas/polars — requires a full rewrite to StaticFrame's own API. Column access is still
string-based; mypy does not catch column name typos. Type safety comes from immutability guarantees, not schema checking.

- **[narwhals](https://narwhals-dev.github.io/narwhals/)** (v2.16.0): Compatibility layer that provides a unified API
- **[narwhals](https://narwhals-dev.github.io/narwhals/)** (v2.22.1): Compatibility layer that provides a unified API
across pandas, polars, DuckDB, cuDF, and more. Solves a different problem—write-once-run-anywhere portability, not
type safety. See [Why Abstraction Layers Don't Solve Type Safety](#why-abstraction-layers-dont-solve-type-safety)
below.

- **[Great Expectations](https://greatexpectations.io/)** (v1.4.3): Comprehensive data quality framework. Defines
- **[Great Expectations](https://greatexpectations.io/)** (v1.18.0): Comprehensive data quality framework. Defines
"expectations" (assertions) about data values, distributions, and schema properties. Excellent for runtime
validation, data documentation, and data quality monitoring. No static analysis or column-level type checking in
code. Supports pandas, Spark, and SQL backends.

- **[dataframely](https://github.com/Quantco/dataframely)**: Polars-only runtime validation library from Quantco.
- **[dataframely](https://github.com/Quantco/dataframely)** (v2.10.1): Polars-only runtime validation library from Quantco.
Schemas are defined as classes inheriting `dy.Schema` with typed descriptor fields (`dy.String()`, `dy.Float64()`)
and `@dy.rule()` decorators for cross-column and group-level constraints. Returns `dy.DataFrame[Schema]` generic
types that give call-site narrowing to type checkers, but does not validate column subscript access inside function
bodies. No lint-time or static analysis capability. Supports nullability, string constraints, numeric bounds,
cross-column rules, soft validation, test data generation, and SQLAlchemy/PyArrow export.

- **[patito](https://github.com/JakobGM/patito)**: Runtime validation library using a Pydantic-style `patito.Model`
- **[patito](https://github.com/JakobGM/patito)** (v0.8.6): Runtime validation library using a Pydantic-style `patito.Model`
class. Polars is the primary backend; pandas is supported but works by converting to Polars via PyArrow (an
undeclared dependency). DuckDB is not supported despite appearing in some documentation—validation crashes immediately
on DuckDB relations. No static analysis or standalone checker.
Expand Down
106 changes: 106 additions & 0 deletions examples/dataenforce/SPEC.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# dataenforce vs typedframes Comparison

## Installation & Compatibility

### dataenforce Status
- **Version**: 0.1.2 (latest available)
- **Package Status**: ❌ BROKEN on Python 3.13+ (and 3.14)
- **Error**: `ImportError: cannot import name '_TypingEmpty' from 'typing'`
- The package relies on internal typing module APIs (`_TypingEmpty`, `_tp_cache`) that were removed/changed in Python 3.13
- No newer versions available on PyPI
- This makes the package unusable on current Python versions

### Installation Report
- **Installed successfully**: Yes (version 0.1.2 added to pyproject.toml)
- **Runs on Python 3.14**: No (import error at runtime)
- **Runs on Python 3.13**: No (same import error)
- **Runs on Python 3.11**: Likely yes (not tested — workspace limited to Python 3.14)

## Design Pattern Comparison

### dataenforce API
- **Type of Validation**: Runtime-only (decorator pattern)
- **Schema Definition**: Inline in function signature as `Dataset["column":type, ...]`
- **Example**:
```python
from dataenforce import Dataset, validate

@validate
def process_orders(df: Dataset["order_id":int, "customer_name":str]):
return df["customer_name"]
```

### typedframes API
- **Type of Validation**: Static analysis at lint-time + optional runtime
- **Schema Definition**: Explicit `BaseSchema` class with `Column` definitions
- **Example**:
```python
from typing import Annotated
import pandas as pd
from typedframes import BaseSchema, Column

class OrderSchema(BaseSchema):
order_id = Column(type=int)
customer_name = Column(type=str)

def process_orders(df: Annotated[pd.DataFrame, OrderSchema]) -> pd.Series:
return df["customer_name"]
```

## Static Analysis Comparison

### dataenforce + mypy
- **Result**: ❌ No column schema validation
- **Test File**: `b_static_analysis.py`
- **Mypy Output**:
```
Success: no issues found in 1 source file
```
- **Findings**:
- Column typos like `df["custmer_name"]` (missing 's') are NOT caught
- Missing columns like `df["quanity"]` (missing 't') are NOT caught
- Function calls with incomplete data are NOT caught
- Errors only surface at **runtime** if the function is actually called with bad data

### typedframes + mypy
- **Result**: ✅ Full column schema validation at lint-time
- **Test File**: `c_typedframes_comparison.py`
- **Mypy Output**:
```
c_typedframes_comparison.py:50: error: Column 'custmer_name' does not exist in OrderSchema (defined at line 38) (did you mean 'customer_name'?) [misc]
c_typedframes_comparison.py:55: error: Column 'unit_cost' does not exist in OrderSchema (defined at line 38) [misc]
Found 2 errors in 1 file (checked 1 source file)
```
- **Findings**:
- Column typos are caught with fuzzy suggestions ("did you mean 'customer_name'?")
- Wrong column names are detected before runtime
- All direct column access errors are caught at lint-time

## Key Differences

| Aspect | dataenforce | typedframes |
|--------|-------------|-------------|
| Validation Timing | Runtime only | Static (lint-time) |
| Schema Definition | Inline in signature | Explicit class |
| Mypy Support | None (SILENT) | Full (mypy plugin) |
| Error Detection | Function must be called with bad data | Caught at lint time |
| IDE Integration | No schema hints | Full IDE schema hints |
| Column Typos | Caught at runtime | Caught at lint time |
| Package Status | Broken on Python 3.13+ | Works across Python 3.11-3.14 |

## Conclusion

### dataenforce
- **Design**: Runtime-only validation via decorator on function signatures
- **Current Status**: Broken on Python 3.13+ due to internal typing module API changes
- **Practical Impact**: Cannot demonstrate the working example on Python 3.13/3.14 environments
- **Limitation**: Only catches errors when functions are actually called with bad data; errors caught at runtime, not development time

### typedframes
- **Design**: Static analysis via mypy plugin + optional runtime checks
- **Current Status**: Works on Python 3.11–3.14
- **Practical Impact**: Catches schema errors during type checking, before code runs
- **Advantage**: Errors caught at lint-time via IDE integration or pre-commit hooks; developers know about bugs immediately during development

### Key Takeaway
typedframes provides **early error detection** (lint-time) while dataenforce relies on **late error detection** (runtime). For data pipelines where columns matter, lint-time checking via typedframes prevents errors from reaching production code.
78 changes: 78 additions & 0 deletions examples/dataenforce/a_working_example.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
"""Working dataenforce example — runtime validation of DataFrame schemas.

dataenforce uses a decorator @validate that enforces column names and types
at runtime. The schema is defined inline in the function signature as a
Dataset annotation with column:type pairs.
"""

import pandas as pd
from dataenforce import Dataset, validate


@validate
def process_orders(df: Dataset["order_id":int, "customer_name":str, "unit_price":float]) -> pd.Series:
"""Extract customer names from validated order DataFrame.

Args:
df: Orders DataFrame with order_id (int), customer_name (str), unit_price (float)

Returns:
Series of customer names
"""
return df["customer_name"]


@validate
def calculate_total_revenue(df: Dataset["quantity":int, "unit_price":float]) -> float:
"""Calculate total revenue from quantity and unit_price columns.

Args:
df: DataFrame with quantity (int) and unit_price (float) columns

Returns:
Total revenue as float
"""
return (df["quantity"] * df["unit_price"]).sum()


if __name__ == "__main__":
# Valid DataFrame — validation passes at runtime
orders = pd.DataFrame(
{
"order_id": [1, 2, 3],
"customer_name": ["Alice", "Bob", "Carol"],
"unit_price": [19.99, 29.50, 15.75],
}
)

print("Valid DataFrame:")
print(orders)
print("\nExtracted customer names:")
result = process_orders(orders)
print(result)
print()

# Calculate revenue
revenue = calculate_total_revenue(
pd.DataFrame(
{
"quantity": [2, 1, 3],
"unit_price": [19.99, 29.50, 15.75],
}
)
)
print(f"Total revenue: ${revenue:.2f}")
print()

# Invalid DataFrame — validation fails at runtime
print("Invalid DataFrame (missing unit_price column):")
try:
invalid_orders = pd.DataFrame(
{
"order_id": [1, 2, 3],
"customer_name": ["Alice", "Bob", "Carol"],
}
)
process_orders(invalid_orders)
except Exception as e:
print(f"Runtime error caught: {type(e).__name__}: {e}")
74 changes: 74 additions & 0 deletions examples/dataenforce/b_static_analysis.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
"""Static analysis of dataenforce code — mypy provides no schema validation.

Despite the Dataset annotation in the function signature, mypy cannot understand
the column schema syntax (Dataset["col":type, ...]). The decorator is purely
runtime validation, so column typos and schema mismatches are not caught by
static analysis.

This file demonstrates:
1. Accessing a non-existent column (typo: "custmer_name" instead of "customer_name")
— mypy is SILENT (no error)
2. Calling the function with mismatched data — mypy is SILENT (error only happens at runtime)
"""

import pandas as pd
from dataenforce import Dataset, validate


@validate
def process_orders(df: Dataset["order_id":int, "customer_name":str, "unit_price":float]) -> pd.Series:
"""Extract customer names from order DataFrame.

Args:
df: Orders DataFrame with order_id (int), customer_name (str), unit_price (float)

Returns:
Series of customer names
"""
# BUG: Typo in column name "custmer_name" (missing 's')
# mypy does NOT catch this — it's runtime-only checking
return df["custmer_name"]


@validate
def calculate_revenue(df: Dataset["quantity":int, "unit_price":float]) -> float:
"""Calculate total revenue.

Args:
df: DataFrame with quantity (int) and unit_price (float)

Returns:
Total revenue as float
"""
# BUG: Accessing "quanity" (typo: missing 't')
# mypy does NOT catch this
return (df["quanity"] * df["unit_price"]).sum()


# This function call will pass mypy even with wrong data
def main() -> None:
"""Call functions with incompatible data."""
# This DataFrame is missing the "unit_price" column,
# but mypy is SILENT — error only occurs at runtime
invalid_df = pd.DataFrame(
{
"order_id": [1, 2, 3],
"customer_name": ["Alice", "Bob", "Carol"],
}
)

# mypy won't complain about calling process_orders with incomplete schema
_ = process_orders(invalid_df)

# This will also pass mypy but fail at runtime
another_invalid = pd.DataFrame(
{
"quantity": [1, 2, 3],
# Missing "unit_price" column
}
)
_ = calculate_revenue(another_invalid)


if __name__ == "__main__":
main()
Loading
Loading