Implement Cypher 25 allReduce predicate function and dynamic early path pruning

[maxdemarzi]

Add support for the Cypher 25 allReduce() predicate function and dynamic path pruning during variable-length repeating pattern matching.

Key Changes

• Syntax: Parse allReduce(acc = init, x IN list | reduction, predicate) inside lexer.rs and grammar.rs.
• Typechecking: Add validation rules in checker.rs to verify source list types, predicate boolean return types, and local variable scopes.
• Runtime Evaluation: Add evaluation of AllReduce expressions in engine.rs's run_expr.
• Early Path Pruning: Track active filters during repeating pattern traversals in try_concat_with_edge_repetition. Evaluate allReduce step-by-step at each level of the traversal and prune invalid branches early.
• Global Constraints: Support querying variables from the outer MATCH query context inside the predicate of allReduce (e.g., comparing against starting node attributes).
• Tests: Add comprehensive integration tests in tests/all_reduce_test.rs.
• Bug Fix: Resolve subtraction overflow issues when measuring memory differences in tests/bench_test.rs.

[maxdemarzi]

Walkthrough - allReduce Dynamic Path Pruning

Cypher 25 allReduce and Dynamic Path Pruning

• AST Definition: Extended the Expr AST enum with AllReduce containing accumulator name, initial value, step variable, list expression, reduction expression, and predicate.
• Parsing: Updated the lexer (lexer.rs) and parser (grammar.rs) to support the allReduce(acc = init, x IN list | reduction, predicate) syntax.
• Typechecking: Added validation rules in checker.rs to ensure the list expression is a list, the predicate returns a boolean, and variables acc_name and step_var are in scope during reduction and predicate checking.
• Runtime Evaluation: Implemented step-by-step evaluation of the accumulator and predicate inside engine.rs's run_expr.
• Early Path Pruning: Integrated pruning inside the level-by-level loop in try_concat_with_edge_repetition. By tracking active filters, if a path prefix fails the allReduce predicate, the traversal branch is pruned immediately, eliminating dead branches early and preventing exponential path explosion.

---

Verification Plan

Automated Tests

• Added comprehensive integration tests in tests/all_reduce_test.rs validating:
  • Pure expression evaluation of allReduce (passing/failing cases).
  • Early path pruning correctness under different constraint thresholds (dist < 6 and dist < 8).
  • Evaluation of global constraints inside allReduce predicates, allowing the path search to query outer/query-scoped nodes or parameters (resolving User Review Comment 2).
  • Enforcing that any repetition variable referenced inside allReduce is preserved (not stripped or unrolled by the compiler optimization passes), forcing level-by-level fallback traversal where dynamic path pruning runs (resolving User Review Comment 1).
  • Typechecker validation rules (checking list source types and boolean predicate return types).
• All tests (including all_reduce_test.rs, memory allocation benchmarks, and the full frogql test suite) compile and pass successfully.

---

Repository Contribution

The changes have been staged, committed, and submitted as a pull request to the upstream repository:

• Pull Request: #60
• Branch: maxdemarzi:all-reduce-pruning

[maxdemarzi]

Walkthrough - Variable-Length Repeating Pattern Optimization & allReduce Dynamic Path Pruning

We have optimized the execution of variable-length repeating patterns on large graphs to prevent Out-Of-Memory (OOM) and exponential path explosion errors, and added support for Cypher 25's allReduce dynamic path pruning predicate.

What Was Fixed & Added

1. Unused Variable Elimination Pass

• We implemented a compiler optimization pass in src/lib.rs that identifies and strips unused variable names from pattern descriptors (e.g., node and edge patterns in MATCH statements).
• This allows repeating patterns like -[e]-{1,3} (where e is not referenced elsewhere) to be converted to anonymous repeating patterns -[]-{1,3}.
• Anonymous repeating patterns are unrolled by the compiler into flat concat/union structures, enabling worst-case optimal join evaluations and avoiding fallback path evaluations.

2. Adjacency-Driven Repeat Traversal Fallback

• When repetition variables are used elsewhere (precluding unrolling), the query engine now falls back to an adjacency-driven execution.
• Rather than executing repeating patterns globally across the entire database, the engine starts from the active left-side result set and traverses level-by-level, tracking intermediate results step-by-step.
• This controls path evaluation scope and matches intermediate results correctly, binding lists of traversed edges to the respective group variables.

3. Boundary Node Positional Fix in Pushdowns

• Resolving a test regression in pushdown.rs, we corrected boundary_node_vars and collect_top_node_vars to treat anonymous nodes as positional boundaries.
• Previously, when an unused boundary variable was stripped, collect_top_node_vars skipped the anonymous node, causing interior variables to incorrectly shift and be classified as boundary nodes, triggering invalid predicate pushdown checks. Now, anonymous nodes are tracked at their correct positional index.

4. Cypher 25 allReduce and Dynamic Path Pruning

• AST Definition: Extended the Expr AST enum with AllReduce containing accumulator name, initial value, step variable, list expression, reduction expression, and predicate.
• Parsing: Updated the lexer (lexer.rs) and parser (grammar.rs) to support the allReduce(acc = init, x IN list | reduction, predicate) syntax.
• Typechecking: Added validation rules in checker.rs to ensure the list expression is a list, the predicate returns a boolean, and variables acc_name and step_var are in scope during reduction and predicate checking.
• Runtime Evaluation & Incremental Caching: Implemented a highly optimized eval_all_reduce_incremental traversal evaluator. Instead of re-evaluating the entire list of edges from scratch at each step (O(k^2) over length k), the engine caches the accumulator value computed at step k-1 on the prefix path and performs a single-step reduction and predicate validation for step k.
• Early Path Pruning: Integrated pruning inside the level-by-level loop in try_concat_with_edge_repetition. By tracking active filters, if a path prefix fails the allReduce predicate, the traversal branch is pruned immediately, eliminating dead branches early and preventing exponential path explosion.

---

Performance Comparison Results

A benchmark was run on a deterministic 10K-node graph (with 50K directed and 5K undirected edges) for the query:
MATCH (n1:Person)-[e]-{1,2}(n2:Person) WHERE n1.score > 9950 RETURN ...

Scenario / Case	Optimization Strategy	Execution Time (ms)	Rows Produced	Description / Notes
Case 1: Unused Variable	Unrolled Union	285.57 ms	6,080	Variable e is unused and stripped. Pattern unrolls to flat union of joins.
Case 2: Used Variable	Adjacency Fallback	8.55 ms	6,080	Variable e is returned, preventing unrolling. Uses new level-by-level traversal.
Case 3: Simulated Legacy	Global Path Enumeration	2604.69 ms	1,430,812	Old behavior where the repeating pattern evaluates globally before filtering.

Insights

• Adjacency Fallback (Case 2): Starts directly from the filtered set of n1 nodes (nodes where score > 9950) and expands paths locally step-by-step. This drops execution time to 8.55 ms (a 300x speedup compared to legacy global enumeration).
• Legacy Behavior (Case 3): In the original engine, because the pattern could not be unrolled (since variables were named or filters couldn't push past the repetition boundary), the engine evaluated -[e]-{1,2} globally first. On a 10K-node graph, this enumerated 1,430,812 paths of length 1 and 2 globally before any filtering, taking over 2.6 seconds. On a larger 57MB graph database, this would cause the OOM crash.

---

Verification Plan

Automated Tests

• Added comprehensive integration tests in tests/all_reduce_test.rs validating:
  • Pure expression evaluation of allReduce (passing/failing cases).
  • Early path pruning correctness under different constraint thresholds (dist < 6 and dist < 8).
  • Dynamic pruning of deep repetitions (e.g., {1,8} which exceeds unrolling bounds) to verify early branch pruning at scale.
  • Evaluation of global constraints inside allReduce predicates, allowing the path search to query outer/query-scoped nodes or parameters (resolving User Review Comment 2).
  • Enforcing that any repetition variable referenced inside allReduce is preserved (not stripped or unrolled by the compiler optimization passes), forcing level-by-level fallback traversal where dynamic path pruning runs (resolving User Review Comment 1).
  • Typechecker validation rules (checking list source types and boolean predicate return types).
• All tests (including all_reduce_test.rs, memory allocation benchmarks, and the full frogql test suite) compile and pass successfully.

---

Repository Contribution

The changes have been staged, committed, and submitted as a pull request to the upstream repository:

• Pull Request: #60
• Branch: maxdemarzi:feat/all-reduce-pruning

[maxdemarzi]

Walkthrough - Variable-Length Repeating Pattern Optimization & allReduce Dynamic Path Pruning

We have optimized the execution of variable-length repeating patterns on large graphs to prevent Out-Of-Memory (OOM) and exponential path explosion errors, and added support for Cypher 25's allReduce dynamic path pruning predicate.

What Was Fixed & Added

1. Unused Variable Elimination Pass

• We implemented a compiler optimization pass in src/lib.rs that identifies and strips unused variable names from pattern descriptors (e.g., node and edge patterns in MATCH statements).
• This allows repeating patterns like -[e]-{1,3} (where e is not referenced elsewhere) to be converted to anonymous repeating patterns -[]-{1,3}.
• Anonymous repeating patterns are unrolled by the compiler into flat concat/union structures, enabling worst-case optimal join evaluations and avoiding fallback path evaluations.

2. Adjacency-Driven Repeat Traversal Fallback

• When repetition variables are used elsewhere (precluding unrolling), the query engine now falls back to an adjacency-driven execution.
• Rather than executing repeating patterns globally across the entire database, the engine starts from the active left-side result set and traverses level-by-level, tracking intermediate results step-by-step.
• This controls path evaluation scope and matches intermediate results correctly, binding lists of traversed edges to the respective group variables.

3. Boundary Node Positional Fix in Pushdowns

• Resolving a test regression in pushdown.rs, we corrected boundary_node_vars and collect_top_node_vars to treat anonymous nodes as positional boundaries.
• Previously, when an unused boundary variable was stripped, collect_top_node_vars skipped the anonymous node, causing interior variables to incorrectly shift and be classified as boundary nodes, triggering invalid predicate pushdown checks. Now, anonymous nodes are tracked at their correct positional index.

4. Cypher 25 allReduce and Dynamic Path Pruning

• AST Definition: Extended the Expr AST enum with AllReduce containing accumulator name, initial value, step variable, list expression, reduction expression, and predicate.
• Parsing: Updated the lexer (lexer.rs) and parser (grammar.rs) to support the allReduce(acc = init, x IN list | reduction, predicate) syntax.
• Typechecking: Added validation rules in checker.rs to ensure the list expression is a list, the predicate returns a boolean, and variables acc_name and step_var are in scope during reduction and predicate checking.
• Runtime Evaluation & Incremental Caching: Implemented a highly optimized eval_all_reduce_incremental traversal evaluator. Instead of re-evaluating the entire list of edges from scratch at each step (O(k^2) over length k), the engine caches the accumulator value computed at step k-1 on the prefix path and performs a single-step reduction and predicate validation for step k.
• Early Path Pruning: Integrated pruning inside the level-by-level loop in try_concat_with_edge_repetition. By tracking active filters, if a path prefix fails the allReduce predicate, the traversal branch is pruned immediately, eliminating dead branches early and preventing exponential path explosion.

5. Edge Variable Binding in BFS/SHORTEST Path Patterns

• Previously, the query engine would bail out (return None) from the fast-path BFS traversal (e.g. try_shortest_bfs) when the repetition pattern had a named edge variable (e.g. -[e:R]->*).
• We implemented extraction of edge values along the BFS paths.
• The engine now binds the matched edges list as a PathValue::Group to the edge variable, allowing edge-variable retrieval in shortest path patterns without fallback penalties.

---

Performance Comparison Results

A benchmark was run on a deterministic 10K-node graph (with 50K directed and 5K undirected edges) for the query:
MATCH (n1:Person)-[e]-{1,2}(n2:Person) WHERE n1.score > 9950 RETURN ...

Scenario / Case	Optimization Strategy	Execution Time (ms)	Rows Produced	Description / Notes
Case 1: Unused Variable	Unrolled Union	285.57 ms	6,080	Variable e is unused and stripped. Pattern unrolls to flat union of joins.
Case 2: Used Variable	Adjacency Fallback	8.55 ms	6,080	Variable e is returned, preventing unrolling. Uses new level-by-level traversal.
Case 3: Simulated Legacy	Global Path Enumeration	2604.69 ms	1,430,812	Old behavior where the repeating pattern evaluates globally before filtering.

Insights

• Adjacency Fallback (Case 2): Starts directly from the filtered set of n1 nodes (nodes where score > 9950) and expands paths locally step-by-step. This drops execution time to 8.55 ms (a 300x speedup compared to legacy global enumeration).
• Legacy Behavior (Case 3): In the original engine, because the pattern could not be unrolled (since variables were named or filters couldn't push past the repetition boundary), the engine evaluated -[e]-{1,2} globally first. On a 10K-node graph, this enumerated 1,430,812 paths of length 1 and 2 globally before any filtering, taking over 2.6 seconds. On a larger 57MB graph database, this would cause the OOM crash.

---

Verification Plan

Automated Tests

• Added comprehensive integration tests in tests/all_reduce_test.rs validating:
  • Pure expression evaluation of allReduce (passing/failing cases).
  • Early path pruning correctness under different constraint thresholds (dist < 6 and dist < 8).
  • Dynamic pruning of deep repetitions (e.g., {1,8} which exceeds unrolling bounds) to verify early branch pruning at scale.
  • Evaluation of global constraints inside allReduce predicates, allowing the path search to query outer/query-scoped nodes or parameters (resolving User Review Comment 2).
  • Enforcing that any repetition variable referenced inside allReduce is preserved (not stripped or unrolled by the compiler optimization passes), forcing level-by-level fallback traversal where dynamic path pruning runs (resolving User Review Comment 1).
  • Typechecker validation rules (checking list source types and boolean predicate return types).
• Added integration tests in tests/shortest_bfs_test.rs (differential_edge_variables) validating that edge variables in ANY SHORTEST and ALL SHORTEST patterns are correctly bound to lists of traversed edges for both directed and undirected graphs, including self-matches and coincident endpoints.
• All tests (including all_reduce_test.rs, shortest_bfs_test.rs, memory allocation benchmarks, and the full frogql test suite) compile and pass successfully.

---

Repository Contribution

The changes have been staged, committed, and submitted as a pull request to the upstream repository:

• Pull Request: #60
• Branch: maxdemarzi:feat/all-reduce-pruning