[rjb32]

Stage each filter's conjuncts by cost, so a costly conjunct runs only over the rows its cheap and selective conjuncts keep.

[rjb32]

Note

AI-generated summary.

Summary

Goal. Win back the four reactome queries ecf5cf583 (MergeFilters) made 1.7x to 2.5x slower, and keep the 15 it made faster. In MATCH (a:ReactionLikeEvent)-[:precedingEvent]-(b:ReactionLikeEvent) RETURN count(*) the merged mask reads the label set of the far node of every edge. Before the merge, only the far nodes of the precedingEvent edges were read.

Approach. A new last db pass, stage_filter_conjuncts, splits a filter into a chain of filters, its cheap and selective conjuncts first. It ranks the conjuncts by (selectivity - 1) / cost and picks the partition into stages with the least estimated cost, a dynamic program solved in linear time with the convex hull trick. Per-row costs come from a new RowCostOpInterface on 12 ops, selectivities from RowEstimator::getPredicateSelectivity, now public. MergeFilters is unchanged, so the fusion passes still see one mask.

Where to look.

• query/ir/dialect/db/DBPasses.cpp:13154 chooseStageStarts: the cost of a stage and the dynamic program. StageStartEnvelope at line 12941 is the convex hull trick.
• query/ir/dialect/db/DBPasses.cpp:13052 placeConeOps: which ops move to a later stage. An op something outside the mask reads stays before the filter, and so do the ops it reads.
• query/ir/dialect/db/DBPasses.cpp:13300 stageConjuncts: clones each later stage's ops after the filter before it, and carries the columns they read through that filter.
• query/ir/dialect/db/DBPasses.cpp:13349 StageFilterConjuncts::runOnOperation: plans every filter, then rewrites them last to first, each once.
• query/ir/dialect/db/DBInterfaces.td:67 RowCostOpInterface, and the costs at query/ir/dialect/db/DBOps.cpp:38.

Query 139's filter, in the shape StageFilterConjunctsTest.readsTheFarNodeLabelsOfTheTypedEdgesAlone pins:

   %5 = db.check_edge_type_constraint(%3, ["precedingEvent"])
-  %6 = db.get_node_label_set(%4)                        # every edge
-  %7 = db.check_label_constraint(%6, ["ReactionLikeEvent"])
-  %8 = db.and %5, %7
-  %9 = db.filter(%8, {%1})
+  %6:2 = db.filter(%5, {%1, %4})                         # precedingEvent edges, carrying the far node
+  %7 = db.get_node_label_set(%6#1)                       # their far nodes alone
+  %8 = db.check_label_constraint(%7, ["ReactionLikeEvent"])
+  %9 = db.filter(%8, {%6#0})

Evidence

• Before: turing-bench run Add test for GraphWriter nullopt bug: cannot reproduce (!nullopt-writer) #22 (a2eedb83e1 to f58112ab9) reported 4 regressions. An A/B of ecf5cf583 against its parent d33df163d on reactome put all four on that commit, and found 15 other queries faster and none slower.

  After: an A/B on hetz-bench of main (66245d01e) against this branch. Four servers one at a time in ABAB order, pinned, one seed, 3 warmup and 15 measured rounds, judged by turing-bench's gate (10% and 0.5 ms):

  0 regressions · 4 speedups · 216 no change · 0 errors · 0 answers changed
  

  #	Query	Before ecf5cf583	main	This PR
  139	undirected precedingEvent count	17.1	42.5, 42.2	17.3, 17.0
  162	Nature x Science pairs, year and volume tests	21.9	37.7, 37.4	22.9, 22.7
  212	undirected precedingEvent triangle	202	385, 386	128, 120
  215	four-hop diamond with <> and NOT IN	34.1	70.1, 69.8	40.6, 39.1

  Server medians in ms. The first column comes from the separate d33df163d A/B.

• Tests: StageFilterConjunctsTest (5 plan shapes, compiled with the graph's counts) and StageFilterConjunctsCypherTest (3, answers across chunks against answers computed in C++). ctest -R "test_query_ir|V3QueryTestSuite": 1043 of 1043 pass.

• Compile time: the pass takes 167 ms in all over the 50 compiles of test_query_ir_deep_chain_query, and 181 ms on a WHERE of 100,000 conjuncts, 1.8 µs per conjunct.

Merge Danger

Door: two-way

Taking the pass out of the pipeline gives back today's plans. No IR, file or wire format changes.

Blast Radius: plans

• A query whose filter holds two or more conjuncts over the rows of one op gets a new plan wherever the model says a stage pays. The costs at DBOps.cpp:38 are estimates, checked on the 220 reactome queries, not calibrated op by op.
• Selectivities treat conjuncts as independent, so 10,000 copies of one predicate get 10 stages, not 1.
• With no graph in the pass context the pass splits nothing. A test compiling without counts sees the plan it saw before.
• A raising op never moves: 10 / n.v > 2 AND n.v <> 0 raises as before.