It might be interesting to consider a feature to "prune" an index of certain items; to reduce the RAM requirements for hosting the dataset.
Perhaps it's useful to have the "full" index for a certain stage of analysis, but at a later stage present only a relevant subset. It should be moderately straightforward to just delete any un-needed items from the index and wildly cut down on the resulting file size.
How to know it works
Simple test case
It might be interesting to consider a feature to "prune" an index of certain items; to reduce the RAM requirements for hosting the dataset.
Perhaps it's useful to have the "full" index for a certain stage of analysis, but at a later stage present only a relevant subset. It should be moderately straightforward to just delete any un-needed items from the index and wildly cut down on the resulting file size.
How to know it works
Simple test case