V0.4.0/dsl foundation

.github/workflows/build.yml

@@ -17,8 +17,8 @@ jobs:
       fail-fast: false
       matrix:
         include:
-          - elixir: "1.20.0-rc.3"
-            otp: "28.4.1"
+          - elixir: "1.20"
+            otp: "29"
           # Latest stable versions
           - elixir: "1.19.5"
             otp: "28.4.1"

CHANGELOG.md

@@ -5,6 +5,32 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## v0.4.0 (2026-06-20)
+
+### Added
+- `ExDatalog.Schema` — Ecto-inspired DSL macro module for defining Datalog programs
+  - `relation/2` macro declares typed relation schemas
+  - `fact/1` and `facts/2` macros declare ground facts
+  - `rule/2` macro declares rules with lowercase logic variables, `not_` for negation, named constraint predicates (`gt`, `eq`, `add`, etc.)
+  - `query/2` macro declares named post-materialization queries with `find`/`where`
+  - `wildcard/0` helper for explicit wildcards in rule bodies
+  - Generated `program/0`, `materialize/0,1`, `queries/0`, `query/2` functions
+- `ExDatalog.UnsupportedFeature` struct for forward-compatible aggregate syntax parsing
+- `ExDatalog.DSL.CompileError` exception for readable DSL macro errors
+- 33 integration tests for the DSL (relations, facts, rules, negation, constraints, queries, backward compatibility)
+- Livebook tutorial: `livebooks/ex_datalog_dsl.livemd`
+- Educational articles in `docs/articles/`
+
+### Changed
+- Version bumped from 0.3.0 to 0.4.0
+- DSL is the recommended authoring layer; builder API remains the stable lower-level API
+- README updated with DSL quickstart
+
+### Notes
+- Query DSL operates on materialized knowledge only (no query planner yet)
+- Aggregate syntax is parsed but not yet executable — returns `%UnsupportedFeature{feature: :aggregates}`
+- All 718 existing tests continue to pass (now 751 total)
+
 ## [0.3.0] - 2025-06-19
 
 ### Added

README.md

@@ -34,14 +34,16 @@ It continues to influence modern databases, compilers, static analysis tools, kn
 ## Features
 
 - **Builder API** for constructing programs (relations, facts, rules, constraints)
+- **Schema DSL** — Ecto-inspired macros for declaring relations, facts, rules, and queries
 - **Constraint types**: comparisons, arithmetic, type predicates, string predicates, membership
 - **Negation** with stratified evaluation
 - **Recursive rules** with semi-naive fixpoint evaluation
+- **Post-materialization queries** (`query` macro with `find`/`where`)
 - **Pluggable storage backends**: `Storage.Map` (default, on-heap) and `Storage.ETS` (off-heap, concurrent reads)
 - **Provenance / derivation explain** (`explain: true`)
 - **Telemetry** integration (`:telemetry` events for query lifecycle)
 - **Deterministic**: same program + same facts = same result regardless of backend
-- 601 tests, 0 failures, credo clean, dialyzer clean
+- 751 tests, 0 failures, credo clean
 
 ## Installation
 
@@ -50,13 +52,80 @@ Add `ex_datalog` to your dependencies in `mix.exs`:
 ```elixir
 def deps do
   [
-    {:ex_datalog, "~> 0.3.0"}
+    {:ex_datalog, "~> 0.4.0"}
   ]
 end
 ```
 
 ## Quick Start
 
+### DSL (Schema macro)
+
+The recommended way to define Datalog programs in v0.4.0+:
+
+```elixir
+defmodule AncestorRules do
+  use ExDatalog.Schema
+
+  relation :parent do
+    field :parent, :atom
+    field :child, :atom
+  end
+
+  relation :ancestor do
+    field :ancestor, :atom
+    field :descendant, :atom
+  end
+
+  fact parent(:alice, :bob)
+  fact parent(:bob, :carol)
+  fact parent(:carol, :dave)
+
+  rule ancestor(X, Y) do
+    parent(X, Y)
+  end
+
+  rule ancestor(X, Z) do
+    parent(X, Y)
+    ancestor(Y, Z)
+  end
+
+  query :descendants_of_alice do
+    find Y
+    where ancestor(:alice, Y)
+  end
+end
+
+{:ok, knowledge} = AncestorRules.materialize()
+AncestorRules.query(:descendants_of_alice, knowledge)
+#=> [:bob, :carol, :dave]
+```
+
+Lowercase variables in rule bodies are logic variables. Constants use
+atom syntax (`:alice`). Use `_` or `wildcard()` for wildcards. Negation
+uses `not_`:
+
+```elixir
+rule bachelor(P) do
+  male(P)
+  not_ married(P, _)
+end
+```
+
+Constraints use named predicates:
+
+```elixir
+rule high_earner(P) do
+  income(P, S)
+  gt(S, 100_000)
+end
+```
+
+The builder API (`Program.add_rule`, `Program.add_fact`, etc.) remains
+fully supported as the lower-level interface.
+
+### Builder API
+
 ### Transitive closure
 
 The classic Datalog example: compute all ancestors from parent facts.
@@ -280,8 +349,11 @@ reference.
 - [What is Datalog?](docs/what-is-datalog.md) — introduction, history, Prolog comparison, industry use cases, LLM integration
 - [Constraints](docs/constraints.md) — constraint types, evaluation, and the dispatch model
 - [Storage Backends](docs/storage_backends.md) — Map vs ETS, options, capabilities, determinism guarantee
-- [Quickstart Tutorial](livebook/quickstart.livemd) — interactive Livebook walkthrough
-- [Examples](livebook/examples.livemd) — 10 realistic use cases (RBAC, supply chain, fraud detection, and more)
+- [Migration: Builder API → DSL](docs/migration_dsl.md) — migrate existing builder-API code to the Schema DSL
+- [DSL Articles](docs/articles/01_why_datalog_on_the_beam.md) — why Datalog on the BEAM, building the DSL, rules as macros, queries, negation
+- [Quickstart Tutorial](livebooks/quickstart.livemd) — interactive Livebook walkthrough
+- [DSL Tutorial](livebooks/ex_datalog_dsl.livemd) — interactive DSL walkthrough
+- [Examples](livebooks/examples.livemd) — 10 realistic use cases (RBAC, supply chain, fraud detection, and more)
 - [API reference](https://hexdocs.pm/ex_datalog) — full module and function documentation
 
 Generate docs locally:
@@ -349,8 +421,8 @@ The following references are highly recommended for understanding both the theor
 | Version | Description |
 |---|---|
 | v0.3.0 | Tuple shorthand for rules (`add_rule/3`, `add_rule/4`), `Term.from/1`, `ExDatalog.Atom.from_tuple/1`, `Constraint.from_tuple/1`; renamed `Result` → `Knowledge`, `query` → `materialize` |
-| v0.4.0 | Sigil DSL (`~d`), aggregation (`count`, `sum`, `min`, `max`), general predicates as deterministic BEAM callbacks |
-| v0.5.0 | Magic sets / demand-driven evaluation, external solver adapter (experimental Z3/Soufflé) |
+| v0.4.0 | Schema DSL (`use ExDatalog.Schema`), `relation`, `fact`, `rule`, `query` macros, `not_` negation, constraint DSL, post-materialization queries, aggregate syntax preview |
+| v0.5.0 | Magic sets / demand-driven evaluation, general predicates as BEAM callbacks |
 | v1.0.0 | Stable public API, hardened production semantics |
 
 ## License

docs/articles/01_why_datalog_on_the_beam.md

@@ -0,0 +1,102 @@
+# Why Datalog Fits the BEAM VM
+
+Datalog is a declarative logic programming language rooted in first-order logic. It computes derived facts from base facts using recursive rules, converging to a fixed point where no new facts can be produced. ExDatalog brings this model to the BEAM — Erlang's virtual machine — and the fit is more than coincidental. The BEAM's foundational design choices align with Datalog's semantics in ways that make the implementation feel natural rather than forced.
+
+## Shared Foundations: Immutability and Pattern Matching
+
+Datalog operates on immutable sets of facts. Once a fact is asserted, it never changes — the evaluation engine simply accumulates new derivations until reaching a fixpoint. The BEAM's runtime is built around the same principle. Elixir data structures are immutable by default; every "modification" produces a new value, and references to the old version remain valid.
+
+This isn't just a philosophical alignment. ExDatalog's `Knowledge` struct holds each relation as a `MapSet` of tuples. During semi-naive evaluation, each iteration produces a *delta* — the set of newly derived facts. The engine merges the delta into the full fact set using `MapSet.union/2`, which returns a new set without mutating the old one. The old snapshot is preserved as `old`, and the delta is computed as `full \ old`. No defensive copying, no lock-based concurrency concerns, no risk of one iteration corrupting another's view of the database.
+
+Pattern matching is the other shared primitive. In Datalog, a rule body like:
+
+```elixir
+rule ancestor(x, z) do
+  parent(x, y)
+  ancestor(y, z)
+end
+```
+
+means: for every binding of `x`, `y`, `z` where `parent(x, y)` and `ancestor(y, z)` both hold, derive `ancestor(x, z)`. The engine joins relations by matching tuples against term patterns — variables bind, constants must equal, wildcards match anything. This is precisely what the BEAM's pattern matching engine does when it dispatches function clauses. ExDatalog's `Engine.Binding` module extends a binding environment by matching IR values against stored tuple positions, just as Elixir extends a function's local scope by matching a pattern against an argument.
+
+## Recursive Evaluation and the Fixpoint
+
+Datalog's defining computational model is the fixpoint: start with the base facts, apply every rule, collect new derivations, and repeat until nothing new emerges. This maps directly onto the BEAM's strength in managing long-running, message-driven processes — but even without processes, the fixpoint loop is a natural fit for a functional runtime.
+
+ExDatalog's `Engine.Naive` implements the semi-naive algorithm. Each iteration considers only facts that are *new* since the last iteration (the delta), avoiding redundant derivations. The implementation is a straightforward recursive loop:
+
+```elixir
+defp fixpoint(ctx) do
+  if delta_empty?(ctx.delta, ctx.all_rels) do
+    ctx
+  else
+    iterate(ctx)
+  end
+end
+```
+
+The BEAM's tail-call optimization ensures this loop runs in constant stack space, even for programs requiring thousands of iterations. The default iteration limit is 10,000, configurable via `max_iterations`. Timeouts are checked each iteration using monotonic time, avoiding clock drift:
+
+```elixir
+if System.monotonic_time(:millisecond) > ctx.deadline do
+  %{ctx | termination: :timeout}
+```
+
+The termination status (`:fixpoint`, `:iteration_limit`, or `:timeout`) is returned in the `Knowledge` struct's `stats` field, giving callers a clear signal about whether the result is complete.
+
+## Hot Code Reloading and Knowledge Evolution
+
+The BEAM's hot code reloading is one of its most distinctive features — you can upgrade a running system's code without stopping it. ExDatalog's DSL leverages this through Ecto-inspired compile-time macros. When you `use ExDatalog.Schema` and define a module like:
+
+```elixir
+defmodule AncestorRules do
+  use ExDatalog.Schema
+
+  relation :parent do
+    field :parent, :atom
+    field :child, :atom
+  end
+
+  rule ancestor(x, y) do
+    parent(x, y)
+  end
+
+  rule ancestor(x, z) do
+    parent(x, y)
+    ancestor(y, z)
+  end
+end
+```
+
+The module compiles into a `program/0` function that builds the `ExDatalog.Program` struct at runtime. If you modify the rules — say, adding a new relation or changing a constraint — and hot-reload the module, the next call to `AncestorRules.program()` returns the updated program. The knowledge base itself is immutable; you re-materialize to get new results. This separation between the rule definition (code) and the derived knowledge (data) is exactly how Datalog is meant to work, and the BEAM's hot code reloading makes it operationally seamless.
+
+## Stratification and Process Isolation
+
+When Datalog programs include negation, stratification determines the evaluation order: relations appearing under negation must be fully computed before they can be negated. ExDatalog uses Tarjan's strongly connected components algorithm to compute strata at compile time, then evaluates them sequentially.
+
+```elixir
+{state_final, total_iterations, origins, termination} =
+  eval_strata(state, ir.strata, ir.rules, max_iterations, ...)
+```
+
+Each stratum runs to a local fixpoint before the next one begins. This sequential dependency would be awkward in a system that assumes concurrent mutation, but on the BEAM — where processes share no memory and communicate by message passing — the isolation is inherent. The current implementation evaluates strata sequentially within a single process, but the architecture leaves room for parallelizing independent strata across BEAM processes, since each stratum's derived facts can be computed from immutable snapshots.
+
+## Persistent Data Structures and Incremental Computation
+
+The BEAM's immutable data structures have another advantage: they make incremental computation efficient. During semi-naive evaluation, ExDatalog maintains three snapshots per relation per iteration:
+
+- **`full`** — all known facts
+- **`delta`** — facts newly derived in the previous iteration
+- **`old`** — the `full` snapshot before the current derivation step
+
+Because Elixir's Maps and MapSets are persistent data structures, creating `old` from `full` is an O(1) pointer copy — the entire snapshot shares structure with the previous version. Computing `delta = full \ old` requires only the diff, avoiding a full scan of every relation on every iteration.
+
+The `Storage.Map` backend stores facts directly in process heap Maps and MapSets. For workloads exceeding ~100K facts, the `Storage.ETS` backend moves data off-heap into per-relation ETS tables, reducing GC pressure while preserving the same deterministic output guarantee. Both backends produce identical `Knowledge` structs for the same program and facts.
+
+## Why Not Prolog?
+
+Elixir developers sometimes ask: why Datalog rather than Prolog on the BEAM? The answer is convergence. Prolog's top-down evaluation with backtracking can loop infinitely on recursive programs, and implementing a complete Prolog engine (with cut, occur-check, and tabling) is a deep undertaking. Datalog's bottom-up, ground-term-only model guarantees termination for programs without arithmetic constraints on unbounded domains, and its fixpoint semantics are simpler to implement correctly.
+
+ExDatalog's validation pipeline catches non-terminating programs before evaluation begins. The safety checker rejects rules where head variables aren't bound by positive body atoms. The stratification checker rejects programs with unstratifiable negation cycles. These are compile-time guarantees — not runtime gambles.
+
+The BEAM's strengths — immutability, pattern matching, hot code reloading, persistent data structures — are not just compatible with Datalog's semantics. They make Datalog on the BEAM feel like the natural expression of a logic language in a concurrent, fault-tolerant runtime. ExDatalog v0.4.0 is the first version to surface these strengths through a declarative Schema DSL, and the result is a system where writing Datalog programs feels like writing Elixir.