V0.2.0/constraint storage foundations

[thanos]

ExDatalog v0.2.0 — Implementation Plan

Architectural Overview

ExDatalog v0.1.0 provides a complete Datalog evaluation pipeline: Program → Validator → Compiler → Engine.Naive → Result. The storage layer uses an immutable Storage.Map backend (Maps + MapSets). Constraints are a flat struct (ExDatalog.Constraint) with comparison and arithmetic ops interleaved, evaluated inline by Engine.ConstraintEval.

v0.2.0 introduces three cross-cutting concerns:

1. Pluggable storage backends — an ETS backend alongside the existing Map backend, sharing a formalized behaviour contract.
2. Extensible constraint architecture — a ExDatalog.Constraint behaviour that current arithmetic/comparison constraints implement, enabling future constraint types without engine changes.
3. Capability metadata — backends and constraint sets declare what they support, enabling portable semantics and future engine routing.

The key invariant: no existing API breaks. The ExDatalog top-level API, Program builder, and Engine.Naive evaluation path remain unchanged for all existing users.

Pipeline Change Diagram

v0.1.0:
  Program → Validator → Compiler → Engine.Naive → Result
                                      ↑
                                 Storage.Map (only)

v0.2.0:
  Program → Validator → Compiler → Engine.Naive → Result
                                      ↑
                            Storage.Backend (behaviour)
                            ├── Storage.Map (existing, refactored)
                            └── Storage.ETS (new)

  Constraint (behaviour)
  ├── Constraints.Arithmetic (existing, refactored)
  └── Constraints.{Comparison,Type,String,...} (new)

---

Module-by-Module Changes

Phase 1 — Storage Backend Abstraction

ExDatalog.Storage (modify: lib/ex_datalog/storage.ex)

Current state: A @callback-based behaviour with 10 callbacks including init/1, insert/3, insert_many/3, member?/3, size/2, stream/2, get_indexed/4, build_index/3, update_index/4, relations/1.

Changes:

• Add @callback capabilities(state) :: ExDatalog.Capabilities.t() to the behaviour. This lets engines query what a backend supports.
• Promote get_indexed/4, build_index/3, update_index/4 from @doc false to documented callbacks — they are needed by ETS and any future indexed backend.
• Add @callback teardown(state) :: :ok for ETS table cleanup (no-op for Map).
• All existing callbacks remain unchanged. Storage.Map gains capabilities/1 and teardown/1 implementations.

ExDatalog.Storage.Backend (new: lib/ex_datalog/storage/backend.ex)

A convenience module that:

• Documents the storage backend contract with examples.
• Provides default_capabilities/0 returning the baseline capability map.
• Provides a __using__/1 macro that generates boilerplate for backend modules (delegating to default implementations where appropriate).

This is optional and can be deferred to after Phase 1 if it adds complexity. The behaviour in Storage itself is sufficient. Decision: defer the macro — the behaviour alone is enough. Just add documentation.

ExDatalog.Storage.Map (modify: lib/ex_datalog/storage/map.ex)

Current state: Implements ExDatalog.Storage using Maps + MapSets. 248 lines. Well-structured.

Changes:

• Add @impl true def capabilities(_state) returning %Capabilities{map_storage: true, indexed_lookup: false}.
• Add @impl true def teardown(_state), do: :ok (no-op for immutable data).
• Remove the @doc false from get_indexed, build_index, update_index — they're part of the public contract now.
• Update the @moduledoc to remove "Phase 8" language and reflect that ETS is now a peer implementation.

No semantic changes to any existing function.

ExDatalog.Capabilities (new: lib/ex_datalog/capabilities.ex)

A struct defining what a backend/constraint set supports:

defmodule ExDatalog.Capabilities do
  @moduledoc """
  Capability metadata for storage backends and constraint sets.

  Capabilities enable engines and tooling to reason about what a given
  configuration supports — portable constraints, indexed lookup,
  provenance, etc. — without runtime introspection of implementation details.
  """

  @type t :: %__MODULE__{
    storage_type: :map | :ets | :external,
    indexed_lookup: boolean(),
    concurrent_reads: boolean(),
    arithmetic_constraints: boolean(),
    comparison_constraints: boolean(),
    type_predicates: boolean(),
    string_predicates: boolean(),
    provenance: boolean(),
    external_execution: boolean()
  }

  defstruct [
    storage_type: :map,
    indexed_lookup: false,
    concurrent_reads: false,
    arithmetic_constraints: true,
    comparison_constraints: true,
    type_predicates: false,
    string_predicates: false,
    provenance: true,
    external_execution: false
  ]
end

Design notes:

• storage_type distinguishes Map (immutable, process-local) from ETS (off-heap, concurrent reads) from future external stores.
• arithmetic_constraints and comparison_constraints default to true because v0.1.0 already supports them.
• type_predicates and string_predicates default to false because they're new in Phase 4.
• provenance defaults to true because v0.1.0 already supports it.
• external_execution defaults to false — reserved for Z3, Soufflé, etc.

Phase 2 — ETS Storage Backend

ExDatalog.Storage.ETS (new: lib/ex_datalog/storage/ets.ex)

Design:

• Uses one ETS table per relation. Tables are named {:ex_datalog, relation_name} using :named_table with :ordered_set type for deterministic iteration order.
• Each table stores tuples directly — the tuple key IS the data. :ordered_set gives O(log n) membership test and ordered first/next iteration.
• Deterministic iteration: ETS iteration via :ets.first/1 / :ets.next/2 over :ordered_set tables is deterministic for term ordering. This is critical — the prompt explicitly calls out that ETS iteration order is nondeterministic, and we must normalize. Using :ordered_set gives us term-ordered iteration for free. However, for cross-relation ordering guarantees, we still sort stream/2 output before returning.
• Lifecycle: Owned by the process that calls init/1. teardown/1 deletes all tables. Options may include :access (default :private), passed through to :ets.new/2.

Callbacks:

Callback	Implementation
init/1	Create one ETS table per relation schema. Store schemas in a map, track table refs.
insert/3	:ets.insert(table_ref, tuple) (idempotent for :ordered_set).
insert_many/3	Iterate and :ets.insert each tuple.
member?/3	:ets.member(table_ref, tuple) — O(log n).
size/2	:ets.info(table_ref, :size) — O(1).
stream/2	`:ets.tab2list(table_ref)
build_index/3	Build a secondary index as a separate ETS table keyed on projected columns.
get_indexed/4	Lookup from index table, return sorted results.
update_index/4	Insert new tuples into index table.
relations/1	Return sorted list of relation names from stored schemas.
capabilities/1	%Capabilities{storage_type: :ets, indexed_lookup: true, concurrent_reads: true, ...}
teardown/1	Delete all ETS tables.

Determinism strategy:

• :ordered_set gives term-ordered traversal.
• stream/2 explicitly sorts via Enum.sort/1 for defense-in-depth.
• relations/1 returns Enum.sort/1.
• All test assertions sort output before comparing.

Concurrency:

• The default access mode is :private (owning process only). The :public option allows concurrent reads.
• No locking is needed for the semi-naive algorithm — each stratum is evaluated sequentially within the owning process.
• Concurrent reads are useful for read-only query workloads after evaluation completes. The :concurrent_reads capability indicates this.

ExDatalog.Engine.Naive (modify: lib/ex_datalog/engine/naive.ex)

Current state: Accepts storage option (default ExDatalog.Storage.Map). Uses storage_mod.init/1, storage_mod.insert/3, storage_mod.stream/2, etc.

Changes:

• Call storage_mod.teardown(state_final) after building the result, before returning {:ok, result}. For Map this is a no-op. For ETS this cleans up tables.
• Call storage_mod.capabilities(state_final) and store capabilities in telemetry metadata (optional, informational).
• No changes to the evaluation algorithm.

ExDatalog.Storage.Map — Conformance Test (new: test/ex_datalog/storage/backend_conformance_test.exs)

A shared conformance test suite that both Storage.Map and Storage.ETS must pass. Uses ExUnit's describe pattern:

defmodule ExDatalog.Storage.BackendConformanceTest do
  @moduledoc """
  Shared conformance tests for Storage backends.

  Any module implementing ExDatalog.Storage must pass all tests in this module.
  """
  # Test cases for: init, insert, insert_many, member?, size, stream,
  # build_index, get_indexed, update_index, relations, capabilities, teardown
  # Determinism: stream returns sorted output, relations returns sorted names
  # Idempotency: inserting duplicate tuples doesn't change size
end

Then:

defmodule ExDatalog.Storage.MapConformanceTest do
  use ExDatalog.Storage.BackendConformanceTest, backend: ExDatalog.Storage.Map
end

defmodule ExDatalog.Storage.ETSConformanceTest do
  use ExDatalog.Storage.BackendConformanceTest, backend: ExDatalog.Storage.ETS
end

Phase 3 — Constraint Behaviour

ExDatalog.Constraint (modify: lib/ex_datalog/constraint.ex)

Current state: A single struct %Constraint{op, left, right, result} with comparison and arithmetic constructors. All ops are in one module.

Changes:

The key architectural change. Currently, Constraint is a concrete struct. v0.2.0 introduces a Constraint behaviour that constraint modules implement. The existing struct remains for backward compatibility — it becomes the "built-in IR constraint" format, and the behaviour wraps it.

Strategy — Wrapper approach:

1. Define a ExDatalog.Constraint behaviour with @callback evaluate(term(), bindings :: map(), context :: map()) :: {:ok, boolean() | map()} | {:error, term()}.

Wait — this needs more thought. The prompt suggests:

defmodule ExDatalog.Constraint do
  @callback evaluate(term(), bindings :: map(), context :: map()) ::
      {:ok, boolean()} | {:error, term()}
end

But the existing Constraint module is a struct module, not a behaviour. We need to:

• Keep the existing Constraint struct and its constructors working (no API break).
• Add a behaviour for extensible constraints.
• Make Engine.ConstraintEval aware of the behaviour.

Revised approach:

1. Rename the current module to ExDatalog.Constraint.BuiltIn (or keep it as-is and add the behaviour in a new module).

Actually, the prompt says:

lib/ex_datalog/constraint.ex
lib/ex_datalog/constraints/
  arithmetic.ex
  comparison.ex
  string.ex
  type.ex

This means:

• ExDatalog.Constraint becomes a behaviour module.
• ExDatalog.Constraint.Arithmetic and ExDatalog.Constraint.Comparison are the first implementations.
• The existing Constraint struct moves to an internal representation.

Migration strategy:

The existing ExDatalog.Constraint struct and constructors (gt/2, lt/2, add/3, etc.) are public API. They appear in user-facing Program construction. We cannot break them.

Approach:

1. Keep ExDatalog.Constraint as the public-facing struct module with all existing constructors.
2. Create ExDatalog.Constraint.Behaviour as the behaviour module with the @callback evaluate/3.
3. Create ExDatalog.Constraint.Arithmetic implementing Constraint.Behaviour for arithmetic ops.
4. Create ExDatalog.Constraint.Comparison implementing Constraint.Behaviour for comparison ops.
5. Modify Engine.ConstraintEval to dispatch through the behaviour when a constraint module is specified, falling back to the existing inline logic for built-in IR constraints.

Wait — there's a subtlety. The constraint struct carries op, left, right, result — the IR format used by the compiler. The behaviour's evaluate/3 takes a term() which could be this struct. So the relationship is:

• Compile time / program construction: Users still call Constraint.gt/2, Constraint.add/3 etc. to build constraints.
• IR level: Constraints are compiled to %IR.Constraint{} structs in the IR.
• Evaluation time: Engine.ConstraintEval.apply_one/2 dispatches based on op. This is where the behaviour should plug in.

Revised approach — minimal refactoring:

Rather than a full restructuring, the Constraint behaviour serves as an extension point for future constraint types (string predicates, type checks) that don't fit the {op, left, right, result} struct format.

1. ExDatalog.Constraint — keep as-is with all existing structs and constructors. Add the @callback evaluate/3 behaviour.

2. ExDatalog.Constraint.Context (new) — evaluation context passed to evaluate/3.

3. ExDatalog.Constraints.Arithmetic (new: lib/ex_datalog/constraints/arithmetic.ex) — implements the behaviour for arithmetic ops. evaluate/3 delegates to the arithmetic logic currently in ConstraintEval.

4. ExDatalog.Constraints.Comparison (new: lib/ex_datalog/constraints/comparison.ex) — implements the behaviour for comparison ops. evaluate/3 delegates to the comparison logic currently in ConstraintEval.

5. Engine.ConstraintEval — add a dispatch layer: if a constraint has a :module field (or resolves to a module via op), call the behaviour. If not, use the existing inline logic. Actually, since we're not changing the IR representation, and all built-in ops are already handled inline, the dispatch for built-in constraints stays as-is. New constraint types use the behaviour.

Actually, let me reconsider the simplest path that satisfies the prompt requirements:

Simplest approach:

1. Add @callback evaluate/3 to ExDatalog.Constraint module.
2. The built-in struct (%Constraint{}) already carries all evaluation data. The behaviour's evaluate/3 callback receives this struct.
3. Constraints.Arithmetic implements evaluate/3 — it receives a %Constraint{} struct, dispatches on op, and performs the arithmetic.
4. Constraints.Comparison implements evaluate/3 — similar.
5. ConstraintEval.apply_one/2 is refactored to call Constraint.evaluate(constraint, binding, context) where evaluate/3 dispatches to the appropriate module.
6. Constraint.Context carries evaluation context (currently empty/unused, but reserved for future needs like capabilities, provenance tracking mode).

This way:

• The struct stays.
• The constructors stay.
• A behaviour is introduced.
• Evaluation is delegated to behaviour implementations.
• New constraint types (string, type) in Phase 4 implement the same behaviour.

Constraint evaluation flow (v0.2.0):

Engine.ConstraintEval.apply_one(constraint, binding)
  → ExDatalog.Constraint.evaluate(constraint, binding, %Context{})
    → dispatches to Constraints.Arithmetic or Constraints.Comparison
      based on constraint.op

This adds one indirection layer but is entirely internal. No public API change.

Phase 4 — Built-in Pure Constraints

New constraint types

All implement ExDatalog.Constraint behaviour and are represented as %Constraint{} structs with new ops. This keeps the IR format uniform.

New ops added to ExDatalog.Constraint:

Comparison extensions:

• :is_integer — type check (unary, result: nil)
• :is_binary — type check (unary)
• :is_atom — type check (unary)

String predicates:

• :starts_with — starts_with(left, right) where right is a literal string
• :contains — contains(left, right) where right is a literal string
• :matches — matches(left, right) where right is a regex pattern (serialized as string)

Wait — the current struct only has left and right (binary). Type checks are unary. We need to handle this.

Option A: Add unary constraints as left only, right: nil.
Option B: Keep right: {:const, nil} for unary constraints.

Decision: Option A. For unary constraints, right is nil and result is nil.

New constructors in ExDatalog.Constraint:

# Type predicates
def is_integer(term), do: unary(:is_integer, term)
def is_binary(term), do: unary(:is_binary, term)
def is_atom(term), do: unary(:is_atom, term)

# String predicates
def starts_with(left, right), do: comparison(:starts_with, left, right)
def contains(left, right), do: comparison(:contains, left, right)
def matches(left, right, pattern), ...

Wait — matches/3 doesn't fit the {left, right, result} pattern well, and regex determinism is a concern (noted in the prompt). Let's keep it simple for v0.2.0:

Phase 4 scope:

• Type predicates: is_integer/1, is_binary/1, is_atom/1
• Equality/inequality: already exist as eq/2, neq/2
• Comparisons: already exist as gt/2, lt/2, gte/2, lte/2
• String predicates: starts_with/2, contains/2 (no regex for determinism)
• Membership: in/2 — checks if a value is in a list of constants

Constructor additions:

# Type predicates (unary)
def is_integer(term)
def is_binary(term)
def is_atom(term)

# String predicates (binary, filter)
def starts_with(left, right)
def contains(left, right)

# Membership (binary, filter)
def member(left, right)  # right is {:const, [...]}

Validation updates in ExDatalog.Validator.Safety:

• Unary constraints bind no new variables and only need their operand to be bound.
• member constraint: right is a constant list; left must be bound.

ConstraintEval updates:

• Type predicate evaluation: check Elixir type of bound value.
• String predicate evaluation: String.starts_with?/2, String.contains?/2.
• Membership: check if bound value is in the constant list.

IR representation:

For type predicates, the IR gains a new unary constraint type. The %IR.Constraint{} struct still works — right will be nil for unary ops, and the compiler handles this.

ExDatalog.Constraints.Type (new: lib/ex_datalog/constraints/type.ex)

Implements Constraint behaviour for :is_integer, :is_binary, :is_atom.

ExDatalog.Constraints.String (new: lib/ex_datalog/constraints/string.ex)

Implements Constraint behaviour for :starts_with, :contains.

ExDatalog.Constraints.Membership (new: lib/ex_datalog/constraints/membership.ex)

Implements Constraint behaviour for :member.

Phase 5 — Capability Model

ExDatalog.Capabilities (created in Phase 1, extended in Phase 5)

The struct is already defined. Phase 5 adds:

• Capabilities.merge/2 — merges two capability structs (AND semantics: both must support).
• Capabilities.from_backend/1 — shorthand for querying a backend's capabilities.
• Capabilities.satisfies?/2 — checks if capabilities satisfy a required set.

Integration with Engine.Naive:

• At evaluation start, query storage_mod.capabilities(state) and store in the evaluation context.
• Constraint modules can optionally check capabilities at evaluation time (but for v0.2.0, this is informational only — no enforcement).

Integration with Constraint evaluation:

• Constraint.Context includes the capability map. Constraint implementations can check it if they need specific backend features.
• For v0.2.0, no constraint checks capabilities — they're all pure and backend-agnostic.

---

Storage Design

Storage.Backend Behaviour

The existing ExDatalog.Storage behaviour is already well-defined. The main addition is capabilities/1 and teardown/1.

@callback capabilities(state) :: ExDatalog.Capabilities.t()
@callback teardown(state) :: :ok | {:error, term()}

Storage.ETS Design

Table strategy: One ETS table per relation, type :ordered_set, named {:ex_datalog, relation_name}.

Why :ordered_set:

• Guaranteed term ordering iteration (deterministic).
• O(log n) lookup and membership test.
• Duplicate prevention (by definition of ordered_set with the tuple as key).

State struct:

defstruct tables: %{},        # %{relation_name => table_ref}
          schemas: %{},        # %{relation_name => %{arity, types}}
          indexes: %{},        # %{{relation_name, columns} => index_table_ref}
          options: []           # ETS options passed to init

Thread safety:

• Default: :private (only owning process). Matches current single-process evaluation.
• Option: :public + :read_concurrency for concurrent read workloads.
• The concurrent_reads capability reflects the access mode.

Memory considerations:

• ETS stores data off-heap, reducing GC pressure for large fact sets.
• :ordered_set uses a balanced tree internally. Memory overhead is slightly higher than :set, but the deterministic ordering benefit outweighs this.
• No :compressed option by default (can be added as an option).

Determinism Guarantees

Operation	Map	ETS
stream/2	MapSet → list (hash order, not deterministic)	:ets.tab2list → Enum.sort (deterministic)
relations/1	Map.keys → sorted	Map.keys → sorted
member?/3	MapSet.member?	:ets.member
size/2	MapSet.size	:ets.info(:size)

Important: Storage.Map.stream/2 currently returns MapSet.to_list(set) which is not deterministic across BEAM versions or runs. However, the engine consumes stream/2 output into MapSet.new() in snapshot_facts/3, which doesn't depend on order. The final result's MapSet is also unordered but equality-comparable.

For Phase 1, Storage.Map.stream/2 should also sort its output for consistency:

def stream(%__MODULE__{relations: rels}, relation) do
  case Map.fetch(rels, relation) do
    {:ok, set} -> set |> MapSet.to_list() |> Enum.sort()
    :error -> []
  end
end

This adds a small cost to Map backend but ensures deterministic iteration across both backends. The prompt says "deterministic iteration semantics" are required.

Actually — let me reconsider. The engine currently uses stream in two places:

1. snapshot_facts/3 which wraps the result in MapSet.new() — order doesn't matter.
2. The evaluator uses MapSet.to_list(Map.get(delta, relation, MapSet.new())) directly — these go through Join.join/3 which also doesn't depend on order.

However, deterministic iteration is still required for reproducibility. If two runs produce facts in different orders, the provenance tracking could attribute to different rules. And for ETS, without explicit sorting, :ets.tab2list on :set or :bag tables returns in insertion order which varies.

Decision: Both backends will sort the output of stream/2. For Map, this is Enum.sort(MapSet.to_list(set)). For ETS, this is Enum.sort(:ets.tab2list(table)).

---

Constraint Design

Architecture

ExDatalog.Constraint (public API module)
├── Constructors: gt/2, lt/2, gte/2, lte/2, eq/2, neq/2 (existing)
├──               add/3, sub/3, mul/3, div/3 (existing)
├──               is_integer/1, is_binary/1, is_atom/1 (new)
├──               starts_with/2, contains/2 (new)
├──               member/2 (new)
├── Introspection: comparison?/1, arithmetic?/1, valid?/1, input_variables/1, result_variable/1
└── Behaviour: @callback evaluate(constraint, bindings, context)

ExDatalog.Constraint.Context (evaluation context — new)
└── %Context{capabilities: Capabilities.t(), provenance: boolean()}

ExDatalog.Constraints.Arithmetic (new: behaviour implementation)
└── Implements evaluate/3 for :add, :sub, :mul, :div

ExDatalog.Constraints.Comparison (new: behaviour implementation)
└── Implements evaluate/3 for :gt, :lt, :gte, :lte, :eq, :neq

ExDatalog.Constraints.Type (new: behaviour implementation)
└── Implements evaluate/3 for :is_integer, :is_binary, :is_atom

ExDatalog.Constraints.String (new: behaviour implementation)
└── Implements evaluate/3 for :starts_with, :contains

ExDatalog.Constraints.Membership (new: behaviour implementation)
└── Implements evaluate/3 for :member

Evaluation dispatch

Engine.ConstraintEval.apply_one/2 currently has inline logic for all ops. Refactored:

def apply_one(%Constraint{} = constraint, binding) do
  context = %Constraint.Context{capabilities: %Capabilities{}}
  case Constraint.evaluate(constraint, binding, context) do
    {:ok, true} -> {:ok, binding}          # comparison passed, no binding change
    {:ok, false} -> :filter                 # comparison failed
    {:ok, new_bindings} -> {:ok, new_bindings}  # arithmetic, new binding
    {:error, _} -> :filter                  # type error, etc.
  end
end

But wait — this changes the return type. Currently apply_one returns {:ok, binding()} | :filter. The Constraint.evaluate/3 callback returns {:ok, boolean()} | {:ok, map()} | {:error, term()}.

Let me reconcile:

Option A: evaluate/3 returns:

• {:ok, binding} — constraint passed, binding possibly extended (arithmetic)
• {:ok, :pass} — constraint passed, binding unchanged (comparison, type, string)
• :filter — constraint failed

Option B: evaluate/3 returns:

• {:ok, true} — passed, no binding change
• {:ok, false} — failed → filter
• {:ok, extended_binding} — passed, binding extended
• {:error, reason} — error, treat as filter

Option A is closer to the existing ConstraintEval pattern. But the prompt's original suggestion is {:ok, boolean()} | {:error, term()}. Let me keep it close to the prompt but practical:

Final design:

@callback evaluate(constraint :: term(), bindings :: map(), context :: Constraint.Context.t()) ::
    {:ok, binding()} | :filter

Where binding() is %{String.t() => term()} — the same as Binding.t().

This matches the existing ConstraintEval.apply/2 and apply_one/2 return types exactly. Each constraint module implements this callback. The dispatch:

def apply_one(%Constraint{} = constraint, binding) do
  module = constraint_module(constraint.op)
  module.evaluate(constraint, binding, %Constraint.Context{})
end

defp constraint_module(op) when op in [:gt, :lt, :gte, :lte, :eq, :neq], do: Constraints.Comparison
defp constraint_module(op) when op in [:add, :sub, :mul, :div], do: Constraints.Arithmetic
defp constraint_module(op) when op in [:is_integer, :is_binary, :is_atom], do: Constraints.Type
defp constraint_module(op) when op in [:starts_with, :contains], do: Constraints.String
defp constraint_module(:member), do: Constraints.Membership

This keeps the existing struct, adds a dispatch layer, and makes it easy to add new constraint types by:

1. Adding a constructor to ExDatalog.Constraint.
2. Adding an op → module mapping.
3. Creating a module that implements the behaviour.

Constraint.Context

defmodule ExDatalog.Constraint.Context do
  @type t :: %__MODULE__{
    capabilities: ExDatalog.Capabilities.t(),
    provenance: boolean()
  }

  defstruct capabilities: %ExDatalog.Capabilities{},
            provenance: false
end

For v0.2.0, Context carries capabilities and provenance flag. Constraints don't use it yet — it's reserved for future use (e.g., a Z3 constraint might need to know if the backend supports external execution).

---

Determinism Strategy

Sources of nondeterminism

1. ETS iteration order — Handled by :ordered_set tables + Enum.sort/1 in stream/2.
2. MapSet iteration order — Not guaranteed across runs. Handled by sorting stream/2 output in both Map and ETS backends.
3. Map key ordering — Elixir Maps have deterministic ordering for small maps (<32 keys) but not for larger maps. Our snapshot_facts/3 uses Map.new/1 with a reduce — safe because we construct maps from sorted keys.
4. Provenance — "Last-wins" attribution. If two rules derive the same fact, the last-evaluated rule gets attribution. Rule evaluation order is determined by stratum and rule list order, which is deterministic after compilation.

Guarantees

• Same program, same facts, same result — regardless of backend.
• stream/2 output is deterministically sorted — both backends.
• relations/1 output is deterministically sorted — both backends.
• Provenance tracking is deterministic — rule evaluation order is deterministic.

Testing strategy

• All integration tests sort results before assertion.
• Property tests generate arbitrary programs and assert result equality across backends.
• ETS stress tests verify >100K fact workloads produce identical results to Map backend.

---

Migration Strategy

Backward compatibility

• No existing API breaks. The ExDatalog top-level API (new/0, validate/1, compile/1, evaluate/2, query/2) is unchanged.
• ExDatalog.Constraint.gt/2, lt/2, etc. are unchanged. New constructors (is_integer/1, etc.) are additive.
• ExDatalog.Storage.Map gains capabilities/1 and teardown/1 but all existing callbacks are unchanged.
• Engine.ConstraintEval dispatches through behaviour internally but apply/2 and apply_one/2 signatures are unchanged.
• Program.add_rule/2 and IR compilation are unchanged — new constraint ops flow through the same compiler path.

Versioning

• Bump mix.exs version to 0.2.0.
• Update @source_ref in docs to v0.2.0.

Deprecations

• None in v0.2.0. Storage.Map is the default and remains the default.

---

Telemetry Considerations

Existing events

• [:ex_datalog, :query, :start] — unchanged.
• [:ex_datalog, :query, :stop] — add storage_type to metadata.
• [:ex_datalog, :query, :exception] — unchanged.

New events

• [:ex_datalog, :storage, :init] — measurements: %{count: n} (number of relations). Metadata: %{storage_type: :map | :ets}.
• [:ex_datalog, :storage, :insert] — measurements: %{count: n} (tuples inserted). Metadata: %{storage_type, relation}.

For v0.2.0, backend-level telemetry can be deferred. The most important addition is storage_type in the existing :stop event metadata.

Constraint telemetry

• Not adding separate constraint evaluation telemetry in v0.2.0. Constraints are evaluated inline by the evaluator and any per-constraint telemetry would add micro-overhead with marginal value. If needed, it can be added later via :telemetry.span.

---

Provenance Considerations

Current architecture

• Explain.explain/3 reconstructs derivation trees from Result.provenance.fact_origins and Result.provenance.rules.
• Naive tracks origins as %{relation => %{tuple => rule_id | :base}} — last-wins attribution.
• Constraints do not affect provenance tracking — they filter/extend bindings but don't attribute individual constraint failures.

Changes needed

• None for provenance tracking. The existing system works fine. Constraints filter bindings, and surviving bindings produce derived facts attributed to the rule.
• Constraint.Context carries a provenance: boolean() field for future use. Not used in v0.2.0.

Future considerations

• Constraint provenance (why a binding was filtered) would require tracking failed constraints. This is out of scope for v0.2.0.
• Backend-specific provenance (which storage contributed which facts) doesn't apply — both backends materialize the same facts.

---

Testing Strategy

Backend conformance tests

Shared test module ExDatalog.Storage.BackendConformanceTest covering:

• init/1 — creates storage for given schemas.
• insert/3 — inserts single tuple, idempotent.
• insert_many/3 — bulk insert.
• member?/3 — membership test.
• size/2 — cardinality.
• stream/2 — deterministic iteration (sorted).
• relations/1 — sorted list.
• build_index/3, get_indexed/4, update_index/4 — index operations.
• capabilities/1 — returns capabilities struct.
• teardown/1 — cleanup.
• Cross-backend parity: Same insert/query operations produce identical results.

ETS stress tests

• Load >100K facts into ETS storage.
• Verify membership, size, stream consistency.
• Verify deterministic iteration over large sets.
• Compare results against Map backend for the same program.

Constraint tests

For each new constraint type:

• Constructor tests (like existing constraint_test.exs).
• Evaluation tests (like existing constraint_eval_test.exs).
• Safety validation tests (unbound variables, invalid types).
• Integration tests (rules using new constraints produce correct results).

Portability tests

• Evaluate the same program with Map backend and ETS backend.
• Assert identical Result (after sorting).
• This is the strongest correctness guarantee.

Deterministic tests

• All test assertions sort results before comparison.
• No MapSet order-dependent assertions.
• Property tests assert equivalence across backends.

---

Performance Considerations

ETS vs Map

Operation	Map (MapSet)	ETS (ordered_set)
Insert (single)	O(log n) copy	O(log n)
Member?	O(log n)	O(log n)
Size	O(1) (MapSet tracks size)	O(1) (:ets.info)
Stream (full scan)	O(n log n) sort	O(n log n) sort
Memory	on-heap, copied	off-heap, no copy for reads

Key insight: The Map backend copies the entire MapSet on every insert (immutable data). For >100K facts, this creates significant GC pressure. ETS avoids this for reads.

The ETS backend wins on:

• Concurrent reads (multiple processes can read without copying).
• Large fact sets (off-heap storage, no GC pressure).
• Membership tests (no data copying).

The Map backend wins on:

• Small workloads (<10K facts) — no ETS overhead.
• Simplicity — immutable, inspectable, easy to debug.
• Single-process evaluation where GC isn't a bottleneck.

Benchmarks

Required benchmarks:

1. Insert throughput — time to insert N facts (N = 1K, 10K, 100K, 500K).
2. Query throughput — time to evaluate a transitive closure program with N base facts.
3. Large relation joins — time for a two-relation join with N facts each.
4. Memory — BEAM memory usage (via :erlang.process_info/2) for Map vs ETS backends at various sizes.

Benchmark module: bench/storage_bench.exs using Benchee.

Optimization avoids

• No index-based evaluation in v0.2.0. The engine still uses sequential joins.
• No concurrent evaluation in v0.2.0.
• No query planning or optimization.

---

Risks

1. ETS table ownership — ETS tables are owned by the process that creates them. If the evaluation process dies, tables are automatically deleted. But if a caller expects the storage to survive beyond the evaluation, this would be a problem. Mitigation: ETS storage is scoped to Engine.Naive.evaluate/2 — created on init, torn down on completion. No cross-process sharing in v0.2.0.

2. ETS :ordered_set performance — :ordered_set uses a balanced tree with O(log n) operations. For very large fact sets, :set (hash table) would be faster for lookups but loses deterministic ordering. Mitigation: Sort on stream/2 regardless; use :set and sort. Actually — :set iteration order is not guaranteed, but we're sorting anyway. Use :set for better insert/lookup performance. Decision: Use :set for performance, sort in stream/2 for determinism. This matches the prompt's requirement that ETS iteration must be normalized.

   Wait — :set doesn't prevent duplicates at the ETS level (same tuple can be inserted twice in :bag). But :set replaces on :ets.insert. So :set gives us idempotent insert (same as MapSet) AND O(1) lookup. But iteration order is still not deterministic — we sort in stream/2.

   Actually, for Datalog facts (tuples), :set semantics (one copy per key, :ets.insert replaces) is what we want. Key = the entire tuple.

   Revised decision: Use :set for ETS tables. Sort in stream/2. This gives better performance than :ordered_set while maintaining determinism.

   Wait — but member?/3 needs to check if a tuple exists. With :set, the key IS the tuple. :ets.member(table, tuple) works in O(1). With :ordered_set, it's O(log n). So :set is faster for both member? and insert.

   Final decision: :set with sorted stream/2.

3. Constraint dispatch overhead — Adding a dispatch layer in ConstraintEval adds one function call per constraint. This is negligible for typical Datalog programs with few constraints per rule. Mitigation: Measure in benchmarks. If significant, inline the hot path for built-in constraints.

4. Unary constraints in IR — The current IR.Constraint struct has left, right, result fields. Unary constraints (type predicates) don't use right. The struct still works — right: nil. But the compiler's from_constraint/1 needs to handle unary constraints. Mitigation: Extend from_constraint/1 with a clause for unary ops.

5. Regex determinism — The prompt warns about regex determinism. matches/2 with ~r/foo/ is not serializable across BEAM nodes. Mitigation: Don't implement matches in v0.2.0. Only starts_with and contains, which are deterministic.

---

Rollback Strategy

Phase 1 rollback

If Storage.Backend abstraction causes issues:

• Revert Storage.ex to original (remove capabilities/1 and teardown/1).
• Revert Storage.Map changes.
• Delete Capabilities module.
• All existing tests pass since no semantic changes.

Phase 2 rollback

If ETS backend causes issues:

• Delete Storage.ETS module.
• Remove :ets option from engine.
• Delete ETS conformance tests.
• Map backend remains default and unaffected.

Phase 3 rollback

If Constraint Behaviour causes issues:

• Remove Constraint.Behaviour and Constraint.Context.
• Inline dispatch back into ConstraintEval.
• Delete Constraints.Arithmetic and Constraints.Comparison.
• Existing Constraint struct and constructors unchanged.

Phase 4 rollback

If new constraint types cause issues:

• Remove new ops from Constraint module.
• Remove Constraints.Type, Constraints.String, Constraints.Membership.
• Remove dispatch clauses for new ops.
• Existing constraints unaffected.

Phase 5 rollback

If Capability model causes issues:

• Remove Capabilities struct and merge/satisfies functions.
• Remove capabilities/1 from Storage behaviour.
• Remove capability checks from engine.
• Everything still works — capabilities are informational.

Full rollback

If the entire release causes issues:

• Revert to v0.1.0 tag.
• All changes are additive — no breaking changes to revert.

---

File Listing (New/Modified)

Phase 1 — Storage Backend Abstraction

New:

• lib/ex_datalog/capabilities.ex
• test/ex_datalog/capabilities_test.exs
• test/ex_datalog/storage/backend_conformance_test.exs

Modified:

• lib/ex_datalog/storage.ex — add capabilities/1, teardown/1 callbacks, update docs
• lib/ex_datalog/storage/map.ex — implement capabilities/1, teardown/1, sort stream/2, update docs
• mix.exs — bump version to 0.2.0

Phase 2 — ETS Storage Backend

New:

• lib/ex_datalog/storage/ets.ex
• test/ex_datalog/storage/ets_test.exs
• bench/storage_bench.exs

Modified:

• lib/ex_datalog/engine/naive.ex — add teardown/1 call, add storage_type to telemetry metadata

Phase 3 — Constraint Behaviour

New:

• lib/ex_datalog/constraint/context.ex
• lib/ex_datalog/constraints/arithmetic.ex
• lib/ex_datalog/constraints/comparison.ex
• test/ex_datalog/constraints/arithmetic_test.exs
• test/ex_datalog/constraints/comparison_test.exs

Modified:

• lib/ex_datalog/constraint.ex — add @callback evaluate/3, add @behaviour definition, add dispatch function
• lib/ex_datalog/engine/constraint_eval.ex — refactor to dispatch through Constraint.evaluate/3

Phase 4 — Built-in Pure Constraints

New:

• lib/ex_datalog/constraints/type.ex
• lib/ex_datalog/constraints/string.ex
• lib/ex_datalog/constraints/membership.ex
• test/ex_datalog/constraints/type_test.exs
• test/ex_datalog/constraints/string_test.exs
• test/ex_datalog/constraints/membership_test.exs

Modified:

• lib/ex_datalog/constraint.ex — add new ops, constructors, validation
• lib/ex_datalog/engine/constraint_eval.ex — add dispatch for new ops
• lib/ex_datalog/validator/safety.ex — handle unary constraints and membership
• lib/ex_datalog/ir.ex — handle unary constraints in from_constraint/1

Phase 5 — Capability Model

Modified:

• lib/ex_datalog/capabilities.ex — add merge/2, from_backend/1, satisfies?/2
• test/ex_datalog/capabilities_test.exs — test new functions

Documentation

New:

• docs/constraints.md
• docs/storage_backends.md
• docs/capabilities.md

Modified:

• README.md — add ETS backend section, constraints section
• CHANGELOG.md — v0.2.0 entry

---

Summary of Invariants

1. No breaking changes to public APIs. All existing functions, types, and modules continue to work identically.
2. Deterministic execution. Same program + same facts = same result, regardless of backend.
3. No arbitrary Elixir execution. No fn predicates, no calling Elixir functions from rules.
4. No NIFs, Ports, or Z3 integration. Pure BEAM.
5. No planner/evaluator rewrite. Engine.Naive's algorithm is unchanged.
6. Test coverage ≥ v0.1.0 level. All existing tests pass, new tests added for each phase.
7. Credo, Dialyzer, Formatter clean.
8. Telemetry parity. ETS backend emits same events as Map backend, plus storage_type metadata.