V0.5.0/code review fixes

CHANGELOG.md

@@ -5,6 +5,64 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## v0.5.0 (2026-06-21)
+
+### Added
+
+- **Query Planner** (`ExDatalog.Planner`) — a thin planning layer between the
+  compiled IR and the engine. `plan/2` produces an `ExDatalog.Planner.Plan`
+  (strategy, strata, joins, predicates); `explain_plan/1,2` renders a
+  human-readable plan. Emits `[:ex_datalog, :planner, :start|:stop|:exception]`
+  telemetry.
+- **Aggregates** — `count`, `sum`, `min`, `max` in rule bodies, via the DSL
+  (`count(X, N)`) or the builder API (`Constraint.count/2`, `from_tuple({:count,
+  X, N})`, `add_rule/4`). Aggregates group surviving bindings by the head
+  variables other than the result, reduce each group, and are stratified
+  strictly above their source relations. Integer-only; one aggregate per rule.
+- **BEAM callback predicates** — call deterministic, side-effect-free Elixir
+  functions from rule bodies. DSL: `predicate :name, Module, :fun, [types],
+  :boolean | :value`. Builder: `ExDatalog.Callback` literals in a rule body.
+  Boolean callbacks filter; `:value` callbacks bind a result variable. The
+  engine isolates callbacks with a configurable timeout
+  (`:callback_timeout_ms`, default 100ms) and rescues exceptions — a
+  timeout/raise filters the binding.
+- **Magic sets** (experimental) — demand-driven evaluation via
+  `materialize(program, strategy: :magic_sets, goal: {relation, pattern})`. The
+  IR is rewritten to compute only facts relevant to the goal. Supports positive
+  recursive programs with ground bound positions; unsupported programs fall back
+  to semi-naive evaluation (never producing incorrect results).
+- `ExDatalog.Callback` and `ExDatalog.IR.Callback` for callback predicates.
+- `ExDatalog.Constraints.Aggregate` and `ExDatalog.Constraints.BeamCallback`.
+- `Capabilities` gains `aggregate_constraints` and `beam_callbacks` fields.
+- `Constraint.Context` gains an `opts` field (threads evaluation options such
+  as `callback_timeout_ms`).
+- Educational articles 06–09 (planner, aggregates, callbacks, magic sets) and a
+  v0.4 → v0.5 migration guide.
+- Benchmark harness (`bench/`) and report (`chest/benchmarks/v0.5.0-report.md`).
+- **Runtime facts API** — `Schema.new/0` returns a blank program (relations +
+  rules, no compile-time facts). Pipe facts at runtime via
+  `Program.add_fact/2` (tuple form `{"rel", [values]}`) or
+  `Program.add_facts/2` (bulk). `Program.materialize/1,2` is a pipe-friendly
+  wrapper for `ExDatalog.materialize/2`. Generated relation constructors
+  (`Schema.emp(:alice, :eng)` -> `{"emp", [:alice, :eng]}`) bridge the DSL
+  to runtime data.
+
+### Changed
+
+- Version bumped from 0.4.1 to 0.5.0.
+- `Engine.Naive.evaluate/2` dispatches on a `:strategy` option (`:semi_naive`
+  default, or `:magic_sets`).
+- `Engine.Naive` stratification validation now also rejects aggregate rules
+  whose source relations are not in a strictly lower stratum.
+- The `agg(...)` placeholder now points users to `count/sum/min/max`.
+
+### Notes
+
+- Aggregates are integer-only; `avg` and float support are deferred to v0.6.0.
+- Magic sets is experimental and opt-in; the default strategy is unchanged.
+- All v0.4.1 tests continue to pass. Suite: 871 tests, 10 properties,
+  152 doctests, ~93% coverage.
+
 ## v0.4.1 (2026-06-21)
 
 ### Fixed

README.md

@@ -36,14 +36,18 @@ It continues to influence modern databases, compilers, static analysis tools, kn
 - **Builder API** for constructing programs (relations, facts, rules, constraints)
 - **Schema DSL** — Ecto-inspired macros for declaring relations, facts, rules, and queries
 - **Constraint types**: comparisons, arithmetic, type predicates, string predicates, membership
+- **Aggregates**: `count`, `sum`, `min`, `max` with grouping and stratification
+- **BEAM callback predicates**: call deterministic Elixir functions from rules (timeout-isolated)
 - **Negation** with stratified evaluation
 - **Recursive rules** with semi-naive fixpoint evaluation
+- **Query planner** (`ExDatalog.Planner`) with `explain_plan/1,2`
+- **Magic sets** (experimental): demand-driven evaluation via `strategy: :magic_sets`
 - **Post-materialization queries** (`query` macro with `find`/`where`)
 - **Pluggable storage backends**: `Storage.Map` (default, on-heap) and `Storage.ETS` (off-heap, concurrent reads)
 - **Provenance / derivation explain** (`explain: true`)
-- **Telemetry** integration (`:telemetry` events for query lifecycle)
+- **Telemetry** integration (`:telemetry` events for query/planner lifecycle)
 - **Deterministic**: same program + same facts = same result regardless of backend
-- 792 tests, 0 failures, credo clean
+- 871 tests, 0 failures, credo clean
 
 ## Installation
 
@@ -52,7 +56,7 @@ Add `ex_datalog` to your dependencies in `mix.exs`:
 ```elixir
 def deps do
   [
-    {:ex_datalog, "~> 0.4.1"}
+    {:ex_datalog, "~> 0.5.0"}
   ]
 end
 ```
@@ -229,6 +233,96 @@ Program.add_rule(program, {"hello_word", [:X]},
 )
 ```
 
+### Aggregates
+
+Group and reduce bindings with `count`, `sum`, `min`, `max`:
+
+```elixir
+defmodule DeptStats do
+  use ExDatalog.Schema
+
+  relation :emp do
+    field(:name, :atom)
+    field(:dept, :atom)
+  end
+
+  relation :dept_count do
+    field(:dept, :atom)
+    field(:n, :integer)
+  end
+
+  fact(emp(:alice, :eng))
+  fact(emp(:bob, :eng))
+  fact(emp(:carol, :sales))
+
+  rule dept_count(D, N) do
+    emp(_, D)
+    count(E, N)
+  end
+end
+
+{:ok, k} = DeptStats.materialize()
+Knowledge.get(k, "dept_count")
+#=> MapSet.new([{:eng, 2}, {:sales, 1}])
+```
+
+### BEAM callback predicates
+
+Call deterministic Elixir functions from rules:
+
+```elixir
+defmodule Gated do
+  use ExDatalog.Schema
+
+  relation :user do
+    field(:name, :atom)
+    field(:age, :integer)
+  end
+
+  relation :active_user do
+    field(:name, :atom)
+  end
+
+  predicate(:adult?, AgeChecker, :adult?, [:integer], :boolean)
+
+  fact(user(:alice, 30))
+  fact(user(:bob, 12))
+
+  rule active_user(U) do
+    user(U, Age)
+    adult?(Age)
+  end
+end
+
+defmodule AgeChecker do
+  def adult?(age), do: age >= 18
+end
+
+{:ok, k} = Gated.materialize()
+Knowledge.get(k, "active_user")
+#=> MapSet.new([{:alice}])
+```
+
+Value-returning callbacks bind a result variable:
+
+```elixir
+predicate :length_of, StringLength, :compute, [:atom], :value
+```
+
+### Magic sets (experimental)
+
+Compute only facts relevant to a goal, instead of the full fixpoint:
+
+```elixir
+{:ok, k} = ExDatalog.materialize(program,
+  strategy: :magic_sets,
+  goal: {"ancestor", [:alice, :_]}
+)
+```
+
+Unsupported programs (negation, aggregates) automatically fall back to
+semi-naive evaluation.
+
 ### Negation
 
 Use negative body atoms with stratified evaluation. Find people who are not parents:
@@ -283,24 +377,27 @@ end
 ## Architecture
 
 ```
-ExDatalog.Program  (builder)
-        |
-        v
+ExDatalog.Program  (builder) / ExDatalog.Schema  (DSL)
+         |
+         v
 ExDatalog.Validator  (structural + semantic + stratification)
-        |
-        v
+         |
+         v
 ExDatalog.Compiler  (AST -> IR)
-        |
-        v
+         |
+         v
+ExDatalog.Planner  (strategy, strata, joins)
+         |
+         v
 ExDatalog.Engine  (behaviour)
-        |
-        v
-ExDatalog.Engine.Naive  (semi-naive fixpoint)
-        |
-        v  uses
+         |
+         v
+ExDatalog.Engine.Naive  (semi-naive fixpoint / magic-sets)
+         |
+         v  uses
 ExDatalog.Storage.Map | ExDatalog.Storage.ETS
-        |
-        v
+         |
+         v
 ExDatalog.Knowledge
 ```
 
@@ -343,14 +440,16 @@ reference.
 | Type predicate | `type_integer`, `type_binary`, `type_atom` | No (filter) |
 | String predicate | `starts_with`, `contains` | No (filter) |
 | Membership | `member` | No (filter) |
+| Aggregate | `count`, `sum`, `min`, `max` | Yes (group-and-reduce) |
 
 ## Documentation
 
 - [What is Datalog?](docs/what-is-datalog.md) — introduction, history, Prolog comparison, industry use cases, LLM integration
 - [Constraints](docs/constraints.md) — constraint types, evaluation, and the dispatch model
 - [Storage Backends](docs/storage_backends.md) — Map vs ETS, options, capabilities, determinism guarantee
 - [Migration: Builder API → DSL](docs/migration_dsl.md) — migrate existing builder-API code to the Schema DSL
-- [DSL Articles](docs/articles/01_why_datalog_on_the_beam.md) — why Datalog on the BEAM, building the DSL, rules as macros, queries, negation
+- [Migration: v0.4 → v0.5](docs/migration_v0.5.md) — aggregates, callbacks, magic sets, and planner changes
+- [DSL Articles](docs/articles/01_why_datalog_on_the_beam.md) — why Datalog on the BEAM, building the DSL, rules as macros, queries, negation, planning, aggregates, callbacks, magic sets
 - [Quickstart Tutorial](livebooks/quickstart.livemd) — interactive Livebook walkthrough
 - [DSL Tutorial](livebooks/ex_datalog_dsl.livemd) — interactive DSL walkthrough
 - [Examples](livebooks/examples.livemd) — 10 realistic use cases (RBAC, supply chain, fraud detection, and more)
@@ -423,7 +522,7 @@ The following references are highly recommended for understanding both the theor
 | v0.3.0 | Tuple shorthand for rules (`add_rule/3`, `add_rule/4`), `Term.from/1`, `ExDatalog.Atom.from_tuple/1`, `Constraint.from_tuple/1`; renamed `Result` → `Knowledge`, `query` → `materialize` |
 | v0.4.0 | Schema DSL (`use ExDatalog.Schema`), `relation`, `fact`, `rule`, `query` macros, `not_` negation, constraint DSL, post-materialization queries |
 | v0.4.1 | DSL review fixes: correct uppercase-variable docs, clean aggregate error, query/find validation, unified `DSL.CompileError`, 792 tests |
-| v0.5.0 | Magic sets / demand-driven evaluation, general predicates as BEAM callbacks |
+| v0.5.0 | Aggregates (`count`/`sum`/`min`/`max`), BEAM callback predicates, query planner, magic sets (experimental), runtime facts API, 871 tests |
 | v1.0.0 | Stable public API, hardened production semantics |
 
 ## License

bench/aggregate_bench.exs

@@ -0,0 +1,82 @@
+alias ExDatalog
+alias ExDatalog.{Program, Rule, Atom, Term, Constraint}
+
+{n, m} = {200, 5}
+
+departments = for i <- 0..(m - 1), do: String.to_atom("dept_#{i}")
+
+base_program =
+  Program.new()
+  |> Program.add_relation("emp", [:atom, :atom])
+  |> Program.add_relation("salary", [:atom, :atom, :integer])
+
+base_program =
+  Enum.reduce(1..n, base_program, fn i, acc ->
+    dept = Enum.at(departments, rem(i, m))
+    salary = 50_000 + i * 100
+    name = String.to_atom("emp_#{i}")
+
+    acc
+    |> Program.add_fact("emp", [name, dept])
+    |> Program.add_fact("salary", [name, dept, salary])
+  end)
+
+program_no_aggregates =
+  base_program
+  |> Program.add_relation("emp_salary", [:atom, :atom, :integer])
+  |> Program.add_rule(
+    Rule.new(
+      Atom.new("emp_salary", [Term.var("E"), Term.var("D"), Term.var("S")]),
+      [{:positive, Atom.new("salary", [Term.var("E"), Term.var("D"), Term.var("S")])}]
+    )
+  )
+
+program_with_aggregates =
+  base_program
+  |> Program.add_relation("dept_count", [:atom, :integer])
+  |> Program.add_relation("dept_total", [:atom, :integer])
+  |> Program.add_relation("dept_min_salary", [:atom, :integer])
+  |> Program.add_relation("dept_max_salary", [:atom, :integer])
+  |> Program.add_rule(
+    Rule.new(
+      Atom.new("dept_count", [Term.var("D"), Term.var("N")]),
+      [{:positive, Atom.new("emp", [Term.var("E"), Term.var("D")])}],
+      [Constraint.count(Term.var("E"), Term.var("N"))]
+    )
+  )
+  |> Program.add_rule(
+    Rule.new(
+      Atom.new("dept_total", [Term.var("D"), Term.var("T")]),
+      [{:positive, Atom.new("salary", [Term.var("E"), Term.var("D"), Term.var("A")])}],
+      [Constraint.sum(Term.var("A"), Term.var("T"))]
+    )
+  )
+  |> Program.add_rule(
+    Rule.new(
+      Atom.new("dept_min_salary", [Term.var("D"), Term.var("V")]),
+      [{:positive, Atom.new("salary", [Term.var("E"), Term.var("D"), Term.var("S")])}],
+      [Constraint.min(Term.var("S"), Term.var("V"))]
+    )
+  )
+  |> Program.add_rule(
+    Rule.new(
+      Atom.new("dept_max_salary", [Term.var("D"), Term.var("V")]),
+      [{:positive, Atom.new("salary", [Term.var("E"), Term.var("D"), Term.var("S")])}],
+      [Constraint.max(Term.var("S"), Term.var("V"))]
+    )
+  )
+
+Benchee.run(
+  %{
+    "no_aggregates" => fn ->
+      {:ok, _knowledge} = ExDatalog.materialize(program_no_aggregates)
+    end,
+    "with_aggregates_count_sum_min_max" => fn ->
+      {:ok, _knowledge} = ExDatalog.materialize(program_with_aggregates)
+    end
+  },
+  time: 10,
+  memory_time: 2,
+  formatters: [{Benchee.Formatters.Console, comparison: true}],
+  print: [fast_warning: false]
+)

bench/magic_sets_bench.exs

@@ -0,0 +1,51 @@
+alias ExDatalog
+alias ExDatalog.{Program, Rule, Atom, Term}
+
+n = 50
+
+chain_facts = for i <- 0..(n - 2), do: {String.to_atom("n#{i}"), String.to_atom("n#{i + 1}")}
+
+program =
+  Program.new()
+  |> Program.add_relation("parent", [:atom, :atom])
+  |> Program.add_relation("ancestor", [:atom, :atom])
+
+program =
+  Enum.reduce(chain_facts, program, fn {p, c}, acc ->
+    Program.add_fact(acc, "parent", [p, c])
+  end)
+
+program =
+  program
+  |> Program.add_rule(
+    Rule.new(
+      Atom.new("ancestor", [Term.var("X"), Term.var("Y")]),
+      [{:positive, Atom.new("parent", [Term.var("X"), Term.var("Y")])}]
+    )
+  )
+  |> Program.add_rule(
+    Rule.new(
+      Atom.new("ancestor", [Term.var("X"), Term.var("Z")]),
+      [
+        {:positive, Atom.new("parent", [Term.var("X"), Term.var("Y")])},
+        {:positive, Atom.new("ancestor", [Term.var("Y"), Term.var("Z")])}
+      ]
+    )
+  )
+
+goal = {"ancestor", [String.to_atom("n0"), :_]}
+
+Benchee.run(
+  %{
+    "semi_naive_full" => fn ->
+      {:ok, _knowledge} = ExDatalog.materialize(program)
+    end,
+    "magic_sets_goal_driven" => fn ->
+      {:ok, _knowledge} = ExDatalog.materialize(program, strategy: :magic_sets, goal: goal)
+    end
+  },
+  time: 10,
+  memory_time: 2,
+  formatters: [{Benchee.Formatters.Console, comparison: true}],
+  print: [fast_warning: false]
+)