Full Text / Fuzzy Search

[maxfriedmann]

I am looking for a good option to integrate SignalDb with fulltext / fuzzy search. I am kind of happy with my current solution (flexsearch + collection hooks) but want to know if anyone has a better approach or a good working solution since my expertise in that field is "still growing" ...

My requirements are:

• full text search across any data type: string, date, number etc.
• works with a lot of data, lets say ~50MB
• fuzzy search (typo friendly)
• works offline

Please see https://stackblitz.com/~/github.com/maxfriedmann/signaldb-flexsearch?file=src/lib/SignalDb.svelte for my current approach which basically consists of:

• create a flexsearch document index for each SignalDb collection
• add/update/remove documents to/from the index whenever the SignalDb collection emits events

What I like:

• unrelated to the SignalDB code since its based on collection hooks
• flexsearch is fast and also supports various persistences

What I don't like:

• data gets duplicated, at least in a search index optimized way

Alternatives I tried:

• Fuse.js - it gets slow quite soon and is only in memory

[jswhisperer]

I don't have experinece in large data sets like youre facing but would creating multple smaller indexed schemas work and you can persist to filesystem or wherever, they obviously get loaded into memory when loaded but any javascript solution thats not a server or api would need to?

https://www.fusejs.io/api/indexing.html#fuse-parseindex

[maxfriedmann]

signaldb is in-browser so anything it evaluates whether a server, an api, a local filesystem will be in memory

True, but if you use an external library that uses its own persistence that does not keep all data in memory then you would at least not double the used memory.

fuse.io parseindex createIndex can be anywhere, so OPFS filesystem, indexedb, a server call etc.

I did not know that, will look into that, thank you very much!

[jswhisperer]

actually last thought, sorry for all the messages once my brain gets going...

you might solve most your perf issues by using fuse in a web worker to not block the main thread, this repo is deprecated but probably a starting point https://github.com/kamilmielnik/fuse.js-worker

[maxfriedmann]

no worries, good input for me. I was kind of stuck on flexsearch because a) RxDB uses it and b) their performance comparison is so well aligned towards their lib is the fastest, see https://nextapps-de.github.io/flexsearch/

[maxnowack]

basically though any solution is an will be an in memory solution AFAIK because signaldb is in memory

By the way, I plan to change this in v2 🙂 See #1557

[SemAiSoft]

basically though any solution is an will be an in memory solution AFAIK because signaldb is in memory

By the way, I plan to change this in v2 🙂 See #1557

This will be a milestone!

[jswhisperer]

Not say SignalDB isn't amazing but if its a real bottleneck you could explore sqlite.ws or turso plus postgres extension https://docs.turso.tech/features/sqlite-extensions fuzzy search.
I've used levenstein algo on postgresdb, I think everyone uses ts-vector now though

[maxfriedmann]

uh, postgres in browser you mean?

[jswhisperer]

Yeah like turso or https://pglite.dev/docs/filesystems
I've used levenstein on real postgresDB server not in browser

some ideas sqlean
with extension fuzzy

https://duckdb.org/docs/stable/clients/wasm/overview.html
pluse https://duckdb.org/docs/stable/extensions/vss

sqlite.wasm
plus https://alexgarcia.xyz/blog/2024/building-new-vector-search-sqlite/index.html

https://github.com/beowolx/haiku-search (13x faster than fuse.io)

[obedm503]

I wouldn't want SignalDB to have a strict dependency on flexsearch. I've used uFuzzy and want to continue using it.

This could be solved by writing a guide on how to integrate a fuzzy library without making changes to SignalDB.

[maxfriedmann]

I agree, even though it could be done via e.g. @signaldb/flexsearch just like the reactive adapters.

[maxfriedmann]

@maxnowack would contributing guides be a good way for you?

[maxnowack]

@maxnowack would contributing guides be a good way for you?

Absolutely! But I'm also happy with dedicated packages 🙂

[jswhisperer]

Unless I missed the mark and you're looking for a node solution or node to browser, like https://electric-sql.com/blog/2024/05/14/electricsql-postgres-client-support#sync-to-postgres-pglite-or-sqlite

https://docs.turso.tech/sdk/ts/reference#manual-sync

https://neon.tech/

and supabase

[maxfriedmann]

I am looking for something that works local first, so in browser only. I forgot to add that requirement.