Skip to content

Analyze bloom filters for membership / semi-join acceleration #70

Description

@matiastoro

Motivation

Analyze where bloom filters could cheaply short-circuit negative membership tests and reduce probing, for EXISTS/NOT EXISTS anti/semi-joins, hash-join fallback, and possibly page-level skipping.

Current state (feasibility)

  • EXISTS/NOT EXISTS correlated evaluation uses HashSet<Vec<PathValue>> for membership (ExistsCache::Correlated, src/runtime/engine.rs) — semi-join for EXISTS, anti-join for NOT EXISTS.
  • The pairwise hash-join fallback builds a hash index on the first shared variable and filters a cross-product.
  • The LTJ leapfrog intersection seeks across sorted candidate lists.

Where bloom might pay (to study)

  1. Anti/semi-join pre-filter: a bloom over the built side lets negative NOT EXISTS rows skip the HashSet probe. Marginal when the HashSet is already O(1) in RAM; more interesting when the set is large or when it avoids materializing the probe key.
  2. Join build/probe on large sets: bloom pre-check before hashing.
  3. Persisted per-(label, prop) bloom: skip pages/records during scans that cannot contain a value — useful for the disk-backed backends.

Feasibility

Additive and contained. Start with a study/prototype to find where bloom beats an in-RAM HashSet (likely large-set joins and disk-page skipping, not small correlated sets). Then implement the winning case behind a flag with an A/B bench.

Metadata

Metadata

Assignees

No one assigned

    Labels

    performanceSpeed / memory optimizationresearchResearch / evaluation / spike

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions