Full Factorized Query Execution Engine

[maxdemarzi]

Full Factorized Query Execution (Complete Rewrite)

This PR implements a complete rewrite of the intermediate join execution layer in frogql to support Full Factorized Execution (compressed tree representation of intermediate relations using products and unions of disjoint factors to avoid combinatorial cross-product materialization).

---

Technical Context & Architectural Gaps

In a query engine using factorized execution, relations are kept in a compressed representation (a tree or forest of independent factors) to avoid the combinatorial explosion of flat cross-products.

Previously, the hybrid approach attempted to merge all product operations under a single FactorNode::Product variant. However, query execution performs two semantically distinct kinds of products:

1. Endpoint-based Path Concatenation:

   • Context: Concatenating path steps (e.g. (a)-[e1]->(b)-[e2]->(c) or repetitions -->{1,2}).
   • Logic: Join on the endpoint node IDs (last_node_id(left) == first_node_id(right)) and extend the path using r1.extend_path(r2.path()).
   • Key: This must NOT treat variables or assignments as the primary join key, but rather the path's structural endpoints.

2. Variable-based Join (Natural/Disjoint):

   • Context: Joining separate query branches (e.g. MATCH (a)-[:knows]->(b), (a)-[:likes]->(c)).
   • Logic: Join on shared variable bindings in the assignment table (or cross-product if disjoint), concatenating independent paths via ResultRow::join(r1, r2).
   • Key: This must NOT unify paths along node endpoints.

By conflating these two under a single FactorNode::Product variant, variable-based natural joins were incorrectly attempting path-endpoint lookups, leading to empty results and test failures (e.g. -->{1,2} returning 12 instead of 23 rows).

---

Proposed Architecture

1. Differentiated FactorNode Variants

We redefined FactorNode in result.rs to represent both join semantics:

• Flat(Vec<ResultRow>) for leaf relations.
• Product(Vec<FactorNode>) for variable-based Cartesian/natural joins.
• PathConcat(Vec<FactorNode>) for path step/endpoint concatenations.
• Union(Vec<FactorNode>) for query branches/alternatives.

2. Recursive Factorized Partitioning & Join Algorithms

• FactorNode::freevars() tracks bound variables in factorized trees.
• FactorNode::partition(self, x: &str) splits factorized representations based on variable values.
• FactorNode::join(self, other: FactorNode, join_var: &str) aligns partitioned left/right factor nodes recursively, producing a factorized union of products without Cartesian materialization.

3. Factorized Join and Concat Operators

• Refactored natural_join in engine.rs to execute fully factorized variable-based joins without early flattening.
• Refactored left_outer_join in engine.rs to partition on the join variable and perform factorized padded joins, propagating Nothing values safely using lazy flat leaf placeholders.
• Simplified run_join in engine.rs to delegate fallback pairwise joins directly to natural_join, completely removing manual flat hash-join loops.
• Defer ensure_flat() in run_concat_pattern to only when optimized adjacency traversal is chosen. Fallback path concatenations construct FactorNode::PathConcat directly from factorized sub-results.
• Updated PathPattern::Union execution in engine.rs to perform lazy factorized unions unless a limit is active, in which case it flattens as needed.

4. Zero-Regression Boundary Flattening

• Retained backward-compatibility for external bindings (Python, Node, WASM) and tests by invoking .ensure_flat() at query boundaries (e.g. run, run_with_limit, run_query, filters, and sorting).

---

Verification Results

Automated Tests

• Ran the entire test suite sequentially: cargo test --workspace -- --test-threads=1
• Result: 193 passed (0 failed, 1 ignored). All tests, including the path repetition test suite (unused_variable_repetition_test), pass successfully.

Clippy Cleanliness

• Compiles and finishes clean under cargo clippy --workspace --all-targets -- -D clippy::all with no warnings or errors.