Add schema versioning, upgrade compatibility checking, and auto-migration

Introduces a schema introspection and diff engine that detects breaking
changes between entity versions and rejects upgrades that lack a migrate()
method. Safe changes (new fields with defaults) are auto-migrated without
developer intervention.

Closes #8

Co-authored-by: Cursor <cursoragent@cursor.com>

Author: deucalioncodes

README.md

@@ -12,6 +12,7 @@ A lightweight key-value database with entity relationships and audit logging cap
 
 - **Persistent Storage**: Works with StableBTreeMap stable structure for persistent storage on your canister's stable memory so your data persists automatically across canister upgrades.
 - **Entity-Relational Database**: Create, read and write entities with OneToOne, OneToMany, ManyToOne, and ManyToMany relationships.
+- **Schema Versioning & Upgrade Safety**: Automatic schema introspection, compatibility checking, and auto-migration for safe changes. Breaking changes without a `migrate()` method are rejected, preventing data corruption on canister upgrades.
 - **Entity Hooks**: Intercept and control entity lifecycle events (create, modify, delete) with `on_event` hooks.
 - **Access Control**: Thread-safe context management for user identity tracking and ownership-based permissions.
 - **Namespaces**: Organize entities into namespaces to avoid type conflicts when you have multiple entities with the same class name.
@@ -267,6 +268,90 @@ class User(Entity):
     profile: Optional["Profile"] = OneToOne("Profile", "user")
 ```
 
+## Schema Versioning & Upgrade Safety
+
+ic-python-db includes built-in upgrade compatibility checking. The system introspects your Entity class definitions to detect schema changes and ensure safe upgrades.
+
+### Auto-migration for safe changes
+
+Adding a field with a default value requires no migration code — the system handles it automatically:
+
+```python
+# v1
+class Product(Entity):
+    __version__ = 1
+    name = String()
+
+# v2 — just add the field with a default, no migrate() needed
+class Product(Entity):
+    __version__ = 2
+    name = String()
+    price = Float(default=0.0)    # auto-injected for existing entities
+    active = Boolean(default=True) # auto-injected for existing entities
+```
+
+### Custom migration for breaking changes
+
+For type changes, field renames, or data transformations, override `migrate()`:
+
+```python
+class Product(Entity):
+    __version__ = 2
+    name = String()
+    price_dollars = Float()  # was price_cents: Integer in v1
+
+    @classmethod
+    def migrate(cls, obj, from_version, to_version):
+        if from_version == 1:
+            obj["price_dollars"] = obj.pop("price_cents") / 100.0
+        return obj
+```
+
+### Upgrade compatibility enforcement
+
+Call `check_upgrade_compatibility()` from your canister's `post_upgrade` to reject incompatible upgrades before they corrupt data:
+
+```python
+from basilisk import post_upgrade
+from ic_python_db import Database
+
+@post_upgrade
+def on_post_upgrade():
+    db = Database.get_instance()
+    db.check_upgrade_compatibility()
+    # If a breaking change lacks migrate(), this raises SchemaIncompatibleError
+    # which traps post_upgrade, causing the IC to roll back the upgrade
+```
+
+The system classifies schema changes as:
+
+| Change | Safety | Action |
+|--------|--------|--------|
+| Add field with `default=` | Safe | Auto-migrated |
+| Remove field | Safe | Old data ignored |
+| Add new Entity type | Safe | No migration needed |
+| Change field type | Breaking | Requires `migrate()` |
+| Add field without default | Breaking | Requires `migrate()` |
+| Change relationship type | Breaking | Requires `migrate()` |
+
+### Schema introspection
+
+You can inspect and compare schemas programmatically:
+
+```python
+from ic_python_db import Database, build_schema, diff_schemas
+
+db = Database.get_instance()
+
+# Build current schema from Entity definitions
+schema = db.build_schema_from_entities()
+
+# Compare two schemas
+changes = diff_schemas(old_schema, new_schema)
+for change in changes:
+    print(f"{change.entity_type}.{change.field}: {change.reason}")
+```
+
 ## API Reference
 
 - **Core**: `Database`, `Entity`
@@ -275,6 +360,7 @@ class User(Entity):
 - **Mixins**: `TimestampedMixin` (timestamps and ownership tracking)
 - **Hooks**: `ACTION_CREATE`, `ACTION_MODIFY`, `ACTION_DELETE`
 - **Context**: `get_caller_id()`, `set_caller_id()`, `Database.as_user()`
+- **Schema**: `build_schema`, `diff_schemas`, `schema_hash`, `SchemaIncompatibleError`
 
 ## Development
 

ic_python_db/__init__.py

@@ -16,6 +16,7 @@
     OneToOne,
     String,
 )
+from .schema import SchemaIncompatibleError, build_schema, diff_schemas, schema_hash
 from .storage import MemoryStorage, Storage
 from .system_time import SystemTime
 
@@ -38,4 +39,8 @@
     "ACTION_CREATE",
     "ACTION_MODIFY",
     "ACTION_DELETE",
+    "SchemaIncompatibleError",
+    "build_schema",
+    "diff_schemas",
+    "schema_hash",
 ]

ic_python_db/db_engine.py

@@ -322,6 +322,50 @@ def raw_dump_json(self, pretty: bool = False) -> str:
             return json.dumps(result, indent=2)
         return json.dumps(result)
 
+    def build_schema_from_entities(self) -> Dict[str, Any]:
+        """Build a schema descriptor from all registered entity types.
+
+        Returns:
+            Dict describing every entity type's fields, types, defaults,
+            constraints, and relationships.
+        """
+        from .schema import build_schema
+
+        return build_schema(self._entity_types)
+
+    def get_schema_hash(self) -> Optional[str]:
+        """Return the stored schema hash, or None if no schema has been saved."""
+        return self.load("_system", "_schema_hash")
+
+    def save_schema(self) -> None:
+        """Persist the current schema descriptor and its hash."""
+        from .schema import schema_hash
+
+        current = self.build_schema_from_entities()
+        self.save("_system", "_schema", current)
+        self.save("_system", "_schema_hash", schema_hash(current))
+
+    def check_upgrade_compatibility(self, raise_on_error: bool = True):
+        """Check that current Entity definitions are compatible with stored schema.
+
+        Compares the previously stored schema against the live class definitions.
+        Breaking changes (type changes, new fields without defaults, relationship
+        cardinality changes) require the Entity to define a migrate() override.
+
+        On success, saves the new schema. On failure (with raise_on_error=True),
+        raises SchemaIncompatibleError — designed to be called from post_upgrade
+        so the IC rolls back the upgrade.
+
+        Returns:
+            List of SchemaChange objects.
+
+        Raises:
+            SchemaIncompatibleError: if incompatible and raise_on_error is True.
+        """
+        from .schema import check_upgrade_compatibility
+
+        return check_upgrade_compatibility(self, raise_on_error=raise_on_error)
+
     def get_audit(
         self, id_from: Optional[int] = None, id_to: Optional[int] = None
     ) -> Dict[str, str]:

ic_python_db/entity.py

@@ -334,6 +334,25 @@ def _alias_key(cls: Type[T], field_name: str = None) -> str:
             return cls.get_full_type_name() + "_alias"
         return f"{cls.get_full_type_name()}_{field_name}_alias"
 
+    @classmethod
+    def _auto_migrate_defaults(cls, data: dict) -> dict:
+        """Inject default values for new Property fields not present in stored data.
+
+        Called before migrate() during version-mismatch loads so that developers
+        don't need to write migrate() for the trivial case of adding a field
+        with a default value.
+        """
+        from .properties import Property
+
+        for klass in reversed(cls.__mro__):
+            for attr_name, attr_value in klass.__dict__.items():
+                if attr_name.startswith("_"):
+                    continue
+                if isinstance(attr_value, Property) and attr_name not in data:
+                    if attr_value.default is not None:
+                        data[attr_name] = attr_value.default
+        return data
+
     @classmethod
     def migrate(cls, obj: dict, from_version: int, to_version: int) -> dict:
         """Migrate entity data from one version to another.
@@ -404,8 +423,10 @@ def load(
                 f"Version mismatch for {type_name}@{entity_id}: "
                 f"stored={stored_version}, current={current_version}"
             )
-            # Apply migration
+            # Apply custom migration first (developer logic takes priority)
             data = cls.migrate(data, stored_version, current_version)
+            # Auto-inject defaults for new fields not handled by migrate()
+            data = cls._auto_migrate_defaults(data)
             data["__version__"] = current_version
             logger.debug(
                 f"Migrated {type_name}@{entity_id} to version {current_version}"