Skip to content

Faster filtering #16

Description

@kasper9n

After moving to SQLite, we can store 1-3 ngrams in a table.

We store both a normalised version and an exact version. If query_term != normalised(query_term), then we only check exact ngrams for that term.

For example, with 3-grams:

`e.ee` gets stored as
`eee` (normalised)
`e.e`
`.ee`

`ééé` gets stored as
`eee` (normalised)
`ééé`

`one  two` gets stored as
`one`
`two`
`ne `
`e  `
`  t`
` tw`

Even though we store whitespace, we would only get whitespace results if the user does an "exact search"

CREATE TABLE search_ngrams (
    ngram    TEXT NOT NULL,
    field    INTEGER NOT NULL,
    is_normalised BOOL NOT NULL,
    track_id TEXT NOT NULL,
);

For 1-3 character query terms, we know that our results are correct. But for 4+ char query terms, we would have to verify them.

Problem: Would need a fast way to query this, which does not seem to exist

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions