After moving to SQLite, we can store 1-3 ngrams in a table.
We store both a normalised version and an exact version. If query_term != normalised(query_term), then we only check exact ngrams for that term.
For example, with 3-grams:
`e.ee` gets stored as
`eee` (normalised)
`e.e`
`.ee`
`ééé` gets stored as
`eee` (normalised)
`ééé`
`one two` gets stored as
`one`
`two`
`ne `
`e `
` t`
` tw`
Even though we store whitespace, we would only get whitespace results if the user does an "exact search"
CREATE TABLE search_ngrams (
ngram TEXT NOT NULL,
field INTEGER NOT NULL,
is_normalised BOOL NOT NULL,
track_id TEXT NOT NULL,
);
For 1-3 character query terms, we know that our results are correct. But for 4+ char query terms, we would have to verify them.
Problem: Would need a fast way to query this, which does not seem to exist
After moving to SQLite, we can store 1-3 ngrams in a table.
We store both a normalised version and an exact version. If
query_term != normalised(query_term), then we only check exact ngrams for that term.For example, with 3-grams:
Even though we store whitespace, we would only get whitespace results if the user does an
"exact search"For 1-3 character query terms, we know that our results are correct. But for 4+ char query terms, we would have to verify them.
Problem: Would need a fast way to query this, which does not seem to exist