Add SQLite + JSON1 backend for V_gamma document storage

[stevevanhooser]

Summary

Implements step 3 of the v2 architecture plan: a SQLite-backed storage engine with JSON1 query compilation. This adds the first concrete database backend for V_gamma documents, complementing the in-memory reference evaluator with a persistent, queryable store.

Key Changes

• did2.database.sqlitedb — New storage backend class that:

  • Opens/creates SQLite databases via mksqlite with a v2-specific schema
  • Manages three core tables: documents (full JSON body + metadata), superclasses (class hierarchy for isa queries), and depends_on (dependency tracking)
  • Provides CRUD operations: add(), remove(), get(), has(), count(), allIds()
  • Implements search(query) and searchIds(query) with SQL pre-filtering + in-memory post-filtering
  • Includes schema validation on open to reject non-v2 databases
  • Supports Validate=false for bulk loads (unsafe_insert escape hatch)

• did2.database.compileQuery — SQL compiler that translates did2.query objects to SQLite WHERE clauses:

  • Scalar operators (exact_string, contains_string, numeric comparisons) → json_extract() expressions
  • Array iteration ([*] paths) → EXISTS subqueries with nested json_each() joins
  • isa operator → indexed lookup on superclasses sidecar table
  • depends_on operator → indexed lookup on depends_on sidecar table with wildcard support
  • hasfield → json_type() checks (treats JSON null as present)
  • hasmember and array-of-structures predicates → json_each() expansion
  • Negation handling with NULL-safe guards for missing paths
  • Permissive pre-filters (1=1) for operators SQLite cannot express natively (regexp, multi-element arrays)

• Test coverage:

  • testCompileQuery.m — 20+ unit tests validating SQL output without requiring mksqlite
  • testSqliteDb.m — 25+ integration tests covering round-trip persistence, schema validation, CRUD, and query semantics across scalar/array/sidecar operators

Implementation Details

• The compiler is conservative by design: it emits over-approximations (e.g., 1=1 for regexp) and relies on the in-memory evaluator (did2.query.matches) as a correctness backstop, ensuring the SQL layer is only required to be a pre-filter.
• Foreign-key cascades automatically clean up sidecar rows when documents are deleted.
• The schema includes a meta table for version tracking; the constructor validates that opened files are v2 databases.
• Path handling supports dot-notation with [*] array-iteration segments; nested arrays compose as cross-products of json_each joins.
• All tests skip gracefully if mksqlite is not available.

https://claude.ai/code/session_01JE5mYZsiyVBJYw55836w8h