Handling large buckets using less disk space for the database

[chrowe]

I am working with a bucket that has ~60 million objects.
I am able to get this to import into the inventory table in the db but when it tries to create the inventory_flatten materialized view, I run out of disk space. My rdb container has ~300GB of disk space, but this doesn't seem to be enough because I keep getting ERROR - Failed to ingest inventory files: could not write to file "base/pgsql_tmp/pgsql_tmp1074.81": No space left on device. After my last try I have ~360 millions rows and it is using 194GB of disk space.

I also tried just generating the PREFIXES_MATERIALIZED_VIEW materialized view for the partially completed inventory_flatten table but that also ran out of disk space.

I am able to create the as just basic views, but for this obviously will not perform well on such a large inventory table.

[jpriebe]

I'm impressed by s3dmap, but seeing your comment tells me that it won't work for my use case. We have a bucket with 1B objects, 270TB of data.

Using our own software, we're able to get a top-level treemap. It takes about 4 hours for the python preprocessor to read the inventory files (50+ GB of gzipped CSV). The resulting treemap is too big for a browser to handle in an interactive fashion, but at least we can get a birds-eye view of what's in the bucket.

In order to process a huge inventory without running out of memory or creating a treemap so large that it locks up the browser where it is running, we have implemented a couple of optimizations. These are designed to reduce the size of the python data structure used during processing as well as the javascript data structure used for the interactive treemap.

• prefix depth limiting: roll up all object details to a depth of N
• prefix filtering: only process objects matching a certain prefix

We have considered other options as well:

• sampling of objects: e.g., only record every Nth object and extrapolate; this might speed up processing time, but doesn't really reduce the memory requirements for the processor or the interactive treemap
• filtering objects based on current/noncurrent status: this probably isn't an order-of-magnitude improvement, but if you have a lot of non-current object versions, it might improve the processing time

s3dmap has a lot of promise (and can do way more than our homegrown solution), but for big buckets, it doesn't seem viable. Maybe if postgresql was deployed on a dedicated server it could handle larger buckets.

If there is one thing I have learned from this experience - don't fill a bucket with 100s of millions of small objects. If at all possible, aggregate that data into larger files (instead of ~10KB files, use 1MB files). Analyzing a bucket with object counts in the hundreds of millions is difficult and time consuming, and the costs to transition the objects to glacier is massive.