Optimize repeating paths with unused variables and level-by-level fallback

[maxdemarzi]

This PR introduces two optimizations for variable-length repeating patterns on large graphs to prevent OOM errors:

1. Unused Variable Elimination: Strips unused variables from patterns, allowing the compiler to unroll repetitions (like -[e]-{1,3} where e is unused) into standard concat joins.
2. Adjacency-Driven Traversal Fallback: Traverses repeating paths level-by-level starting from active left-side filtered nodes instead of doing a legacy global path enumeration.

Also fixes a boundary node positioning bug in the pushdown optimizer that was exposed by removing unused end-of-path variables.

[maxdemarzi]

Walkthrough - Variable-Length Repeating Pattern Optimization

We have optimized the execution of variable-length repeating patterns on large graphs to prevent Out-Of-Memory (OOM) and exponential path explosion errors.

What Was Fixed

1. Unused Variable Elimination Pass

• We implemented a compiler optimization pass in src/lib.rs that identifies and strips unused variable names from pattern descriptors (e.g., node and edge patterns in MATCH statements).
• This allows repeating patterns like -[e]-{1,3} (where e is not referenced elsewhere) to be converted to anonymous repeating patterns -[]-{1,3}.
• Anonymous repeating patterns are unrolled by the compiler into flat concat/union structures, enabling worst-case optimal join evaluations and avoiding fallback path evaluations.

2. Adjacency-Driven Repeat Traversal Fallback

• When repetition variables are used elsewhere (precluding unrolling), the query engine now falls back to an adjacency-driven execution.
• Rather than executing repeating patterns globally across the entire database, the engine starts from the active left-side result set and traverses level-by-level, tracking intermediate results step-by-step.
• This controls path evaluation scope and matches intermediate results correctly, binding lists of traversed edges to the respective group variables.

3. Boundary Node Positional Fix in Pushdowns

• Resolving a test regression in pushdown.rs, we corrected boundary_node_vars and collect_top_node_vars to treat anonymous nodes as positional boundaries.
• Previously, when an unused boundary variable was stripped, collect_top_node_vars skipped the anonymous node, causing interior variables to incorrectly shift and be classified as boundary nodes, triggering invalid predicate pushdown checks. Now, anonymous nodes are tracked at their correct positional index.

---

Performance Comparison Results

A benchmark was run on a deterministic 10K-node graph (with 50K directed and 5K undirected edges) for the query:
MATCH (n1:Person)-[e]-{1,2}(n2:Person) WHERE n1.score > 9950 RETURN ...

Scenario / Case	Optimization Strategy	Execution Time (ms)	Rows Produced	Description / Notes
Case 1: Unused Variable	Unrolled Union	285.57 ms	6,080	Variable e is unused and stripped. Pattern unrolls to flat union of joins.
Case 2: Used Variable	Adjacency Fallback	8.55 ms	6,080	Variable e is returned, preventing unrolling. Uses new level-by-level traversal.
Case 3: Simulated Legacy	Global Path Enumeration	2604.69 ms	1,430,812	Old behavior where the repeating pattern evaluates globally before filtering.

Insights

• Adjacency Fallback (Case 2): Starts directly from the filtered set of n1 nodes (nodes where score > 9950) and expands paths locally step-by-step. This drops execution time to 8.55 ms (a 300x speedup compared to legacy global enumeration).
• Legacy Behavior (Case 3): In the original engine, because the pattern could not be unrolled (since variables were named or filters couldn't push past the repetition boundary), the engine evaluated -[e]-{1,2} globally first. On a 10K-node graph, this enumerated 1,430,812 paths of length 1 and 2 globally before any filtering, taking over 2.6 seconds. On a larger 57MB graph database, this would cause the OOM crash.

---

Repository Contribution

The changes have been staged, committed, and submitted as a pull request to the upstream repository:

• Pull Request: #59
• Branch: maxdemarzi:optimize-repetition-path

[maxdemarzi]

Walkthrough - Variable-Length Repeating Pattern Optimization & allReduce Dynamic Path Pruning

We have optimized the execution of variable-length repeating patterns on large graphs to prevent Out-Of-Memory (OOM) and exponential path explosion errors, and added support for Cypher 25's allReduce dynamic path pruning predicate.

What Was Fixed & Added

1. Unused Variable Elimination Pass

• We implemented a compiler optimization pass in src/lib.rs that identifies and strips unused variable names from pattern descriptors (e.g., node and edge patterns in MATCH statements).
• This allows repeating patterns like -[e]-{1,3} (where e is not referenced elsewhere) to be converted to anonymous repeating patterns -[]-{1,3}.
• Anonymous repeating patterns are unrolled by the compiler into flat concat/union structures, enabling worst-case optimal join evaluations and avoiding fallback path evaluations.

2. Adjacency-Driven Repeat Traversal Fallback

• When repetition variables are used elsewhere (precluding unrolling), the query engine now falls back to an adjacency-driven execution.
• Rather than executing repeating patterns globally across the entire database, the engine starts from the active left-side result set and traverses level-by-level, tracking intermediate results step-by-step.
• This controls path evaluation scope and matches intermediate results correctly, binding lists of traversed edges to the respective group variables.

3. Boundary Node Positional Fix in Pushdowns

• Resolving a test regression in pushdown.rs, we corrected boundary_node_vars and collect_top_node_vars to treat anonymous nodes as positional boundaries.
• Previously, when an unused boundary variable was stripped, collect_top_node_vars skipped the anonymous node, causing interior variables to incorrectly shift and be classified as boundary nodes, triggering invalid predicate pushdown checks. Now, anonymous nodes are tracked at their correct positional index.

4. Cypher 25 allReduce and Dynamic Path Pruning

• AST Definition: Extended the Expr AST enum with AllReduce containing accumulator name, initial value, step variable, list expression, reduction expression, and predicate.
• Parsing: Updated the lexer (lexer.rs) and parser (grammar.rs) to support the allReduce(acc = init, x IN list | reduction, predicate) syntax.
• Typechecking: Added validation rules in checker.rs to ensure the list expression is a list, the predicate returns a boolean, and variables acc_name and step_var are in scope during reduction and predicate checking.
• Runtime Evaluation: Implemented step-by-step evaluation of the accumulator and predicate inside engine.rs's run_expr.
• Early Path Pruning: Integrated pruning inside the level-by-level loop in try_concat_with_edge_repetition. By tracking active filters, if a path prefix fails the allReduce predicate, the traversal branch is pruned immediately, eliminating dead branches early and preventing exponential path explosion.

---

Performance Comparison Results

A benchmark was run on a deterministic 10K-node graph (with 50K directed and 5K undirected edges) for the query:
MATCH (n1:Person)-[e]-{1,2}(n2:Person) WHERE n1.score > 9950 RETURN ...

Scenario / Case	Optimization Strategy	Execution Time (ms)	Rows Produced	Description / Notes
Case 1: Unused Variable	Unrolled Union	285.57 ms	6,080	Variable e is unused and stripped. Pattern unrolls to flat union of joins.
Case 2: Used Variable	Adjacency Fallback	8.55 ms	6,080	Variable e is returned, preventing unrolling. Uses new level-by-level traversal.
Case 3: Simulated Legacy	Global Path Enumeration	2604.69 ms	1,430,812	Old behavior where the repeating pattern evaluates globally before filtering.

Insights

• Adjacency Fallback (Case 2): Starts directly from the filtered set of n1 nodes (nodes where score > 9950) and expands paths locally step-by-step. This drops execution time to 8.55 ms (a 300x speedup compared to legacy global enumeration).
• Legacy Behavior (Case 3): In the original engine, because the pattern could not be unrolled (since variables were named or filters couldn't push past the repetition boundary), the engine evaluated -[e]-{1,2} globally first. On a 10K-node graph, this enumerated 1,430,812 paths of length 1 and 2 globally before any filtering, taking over 2.6 seconds. On a larger 57MB graph database, this would cause the OOM crash.

---

Verification Plan

Automated Tests

• Added comprehensive integration tests in tests/all_reduce_test.rs validating:
  • Pure expression evaluation of allReduce (passing/failing cases).
  • Early path pruning correctness under different constraint thresholds (dist < 6 and dist < 8).
  • Typechecker validation rules (checking list source types and boolean predicate return types).
• All tests (including all_reduce_test.rs, memory allocation benchmarks, and the full frogql test suite) compile and pass successfully.

---

Repository Contribution

The changes have been staged, committed, and submitted as a pull request to the upstream repository:

• Pull Request: #59
• Branch: maxdemarzi:optimize-repetition-path

[maxdemarzi]

Walkthrough - Variable-Length Repeating Pattern Optimization & allReduce Dynamic Path Pruning

We have optimized the execution of variable-length repeating patterns on large graphs to prevent Out-Of-Memory (OOM) and exponential path explosion errors, and added support for Cypher 25's allReduce dynamic path pruning predicate.

What Was Fixed & Added

1. Unused Variable Elimination Pass

• We implemented a compiler optimization pass in src/lib.rs that identifies and strips unused variable names from pattern descriptors (e.g., node and edge patterns in MATCH statements).
• This allows repeating patterns like -[e]-{1,3} (where e is not referenced elsewhere) to be converted to anonymous repeating patterns -[]-{1,3}.
• Anonymous repeating patterns are unrolled by the compiler into flat concat/union structures, enabling worst-case optimal join evaluations and avoiding fallback path evaluations.

2. Adjacency-Driven Repeat Traversal Fallback

• When repetition variables are used elsewhere (precluding unrolling), the query engine now falls back to an adjacency-driven execution.
• Rather than executing repeating patterns globally across the entire database, the engine starts from the active left-side result set and traverses level-by-level, tracking intermediate results step-by-step.
• This controls path evaluation scope and matches intermediate results correctly, binding lists of traversed edges to the respective group variables.

3. Boundary Node Positional Fix in Pushdowns

• Resolving a test regression in pushdown.rs, we corrected boundary_node_vars and collect_top_node_vars to treat anonymous nodes as positional boundaries.
• Previously, when an unused boundary variable was stripped, collect_top_node_vars skipped the anonymous node, causing interior variables to incorrectly shift and be classified as boundary nodes, triggering invalid predicate pushdown checks. Now, anonymous nodes are tracked at their correct positional index.

4. Cypher 25 allReduce and Dynamic Path Pruning

• AST Definition: Extended the Expr AST enum with AllReduce containing accumulator name, initial value, step variable, list expression, reduction expression, and predicate.
• Parsing: Updated the lexer (lexer.rs) and parser (grammar.rs) to support the allReduce(acc = init, x IN list | reduction, predicate) syntax.
• Typechecking: Added validation rules in checker.rs to ensure the list expression is a list, the predicate returns a boolean, and variables acc_name and step_var are in scope during reduction and predicate checking.
• Runtime Evaluation: Implemented step-by-step evaluation of the accumulator and predicate inside engine.rs's run_expr.
• Early Path Pruning: Integrated pruning inside the level-by-level loop in try_concat_with_edge_repetition. By tracking active filters, if a path prefix fails the allReduce predicate, the traversal branch is pruned immediately, eliminating dead branches early and preventing exponential path explosion.

---

Performance Comparison Results

A benchmark was run on a deterministic 10K-node graph (with 50K directed and 5K undirected edges) for the query:
MATCH (n1:Person)-[e]-{1,2}(n2:Person) WHERE n1.score > 9950 RETURN ...

Scenario / Case	Optimization Strategy	Execution Time (ms)	Rows Produced	Description / Notes
Case 1: Unused Variable	Unrolled Union	285.57 ms	6,080	Variable e is unused and stripped. Pattern unrolls to flat union of joins.
Case 2: Used Variable	Adjacency Fallback	8.55 ms	6,080	Variable e is returned, preventing unrolling. Uses new level-by-level traversal.
Case 3: Simulated Legacy	Global Path Enumeration	2604.69 ms	1,430,812	Old behavior where the repeating pattern evaluates globally before filtering.

Insights

• Adjacency Fallback (Case 2): Starts directly from the filtered set of n1 nodes (nodes where score > 9950) and expands paths locally step-by-step. This drops execution time to 8.55 ms (a 300x speedup compared to legacy global enumeration).
• Legacy Behavior (Case 3): In the original engine, because the pattern could not be unrolled (since variables were named or filters couldn't push past the repetition boundary), the engine evaluated -[e]-{1,2} globally first. On a 10K-node graph, this enumerated 1,430,812 paths of length 1 and 2 globally before any filtering, taking over 2.6 seconds. On a larger 57MB graph database, this would cause the OOM crash.

---

Verification Plan

Automated Tests

• Added comprehensive integration tests in tests/all_reduce_test.rs validating:
  • Pure expression evaluation of allReduce (passing/failing cases).
  • Early path pruning correctness under different constraint thresholds (dist < 6 and dist < 8).
  • Evaluation of global constraints inside allReduce predicates, allowing the path search to query outer/query-scoped nodes or parameters (resolving User Review Comment 2).
  • Enforcing that any repetition variable referenced inside allReduce is preserved (not stripped or unrolled by the compiler optimization passes), forcing level-by-level fallback traversal where dynamic path pruning runs (resolving User Review Comment 1).
  • Typechecker validation rules (checking list source types and boolean predicate return types).
• All tests (including all_reduce_test.rs, memory allocation benchmarks, and the full frogql test suite) compile and pass successfully.

---

Repository Contribution

The changes have been staged, committed, and submitted as a pull request to the upstream repository:

• Pull Request: #59
• Branch: maxdemarzi:optimize-repetition-path