Handling TB of data?

[doplu]

Hello,
Thank you for creating file-brain. Do you know if it can easily handle terabytes (TB) of data (pdf etc. Note: most of my data is old and doesn't need to be updated) on an old computer (I currently use another software Dtsearch which is working fine once I have done the indexing)?
Thanks in advance

[Hamza5]

Sorry for the late reply, I just saw this.

I think I have replied to you by email because someone mentioned that they were using dtSearch in an email.

[doplu]

Hello, Thank you for your answer. It was the first time that I have contacted you. ;) Time passes by and I have now new computer win11. So I just tested file-brain. Here are my first impressions (I am not a coder) : It was difficult to install and start (AI helped me on that hopefully). Once started with a PowerShell command, I couldn't click on adding folders (on the bottom part). I have looked at issues and did this "pip install -U file-brain". Now I have found a way to add folder about the top right. ;) I have added 2 folders for testing (nothing fancy just a few thousand pdf files) and indexing was really slow and seems blocked after 2 files.... Well, I understand that it is still early in the development stage. But for me, for now I am sorry as I have spent too much time on it this morning. I am sharing my current feelings : I think I have too much data (many millions of small html files with their subfolders related files for instance (try CTRL+S in a browser)) for file-brain to be working fine (it reminds me of Copernic Desktop search usage : fine for a computer with a limited amount of data to index as it does a lot of indexing things continuously and can do a lot of things for the user : OCR etc...). The way Dtsearch handle this is by creating one index for one or a few folders (yes there can be overlapses in the indexes, but overall this is light on computer resources). You choose when you want to index it. Then you search only in the indexes that you want to. I also do the OCR of pdf and images files by myself once. I plan to also do the automatic transcriptions (txt files) of my video and audio files later. And also add a local AI for some AI things. Anyway, thank you for what you have already done and I wish you the best for the future. ;)

[Hamza5]

Thanks for the detailed feedback! I wish you the best, too!