Compressed container result

[flrgsr]

Hi guys,

I am trying to compute a huge correlation matrix in an out of core or in memory compressed manner (as my data is normalize I only need the dot product). In the example given on the main page here on github, the "result" variable is a normal python variable stored in main memory, which blows everything up on my machine as the result is ~20GB.
My question is, how can I have the corresponding result immediately going to an compressed container (either memory or out of core, where memory would be preferred given that I could get a factor ~5 compression).

Many thanks in advance.

[waylonflinn]

Thanks for the question. If I'm understanding you correctly, you want to write the result of a carray . ndarray multiply directly to disk.

If you put your ndarray into a one row carray, you should be able to use the functionality I added yesterday. It writes the results of a carray . carray multiply directly to disk using the rootdir named parameter to dot (see README updates).

I haven't actually tried this yet. So, please report back with your results!

[waylonflinn]

I tried my own suggestion and it didn't seem to work. I added an extra named parameter to support this.

adapted from the new README

import bdot
import bcolz
import numpy as np

matrix = np.random.random_integers(0, 12000, size=(300000, 100))
bcarray = bdot.carray(matrix, chunklen=2**13, cparams=bcolz.cparams(clevel=2))

v = bcarray[0]

# create correctly sized bdot.carray container 
# (helper method, not required, rootdir parameter is optional)
output = bcarray1.empty_like_dot(v, rootdir='/path/to/bcolz/output')

# generate results directly on disk
bcarray1.dot(bcarray2, out=output)

# make sure the last bits get written
output.flush()

Leaving off the rootdir parameter will allow use of an in-memory store.

You can also see it in action in it's unit test:
https://github.com/pinleague/bdot/blob/master/bdot/tests/test_carray.py#L63

Please do let me know how this works for you.

[waylonflinn]

@flrgsr Can you tell me about the size and shape of your data? How many rows and columns are in your input matrix and vector? What data type are you using (int64, int32, etc)?

I'm trying to optimize performance and it would help to have another concrete use case.

Thanks!

[flrgsr]

Hi @waylonflinn.

Sorry, for the slow reply.

My input data has the following shape:

tsnorm.shape
(81602, 116)

tsnorm.dtype
dtype('float32')

So the full correlation matrix that I want to compute has dimensions 81602.
So far I was experimenting with pytables to solve my problem and haven't get around to set it up properly with bcolz/bdot. Here is a little notebook of what I am currently doing: http://nbviewer.ipython.org/gist/flrgsr/0bff976728d1ae9d13e4
Maybe I should open a separate thread on how to translate this to bcolz.

Best
Florian

[waylonflinn]

Beautiful! Thanks @flrgsr!
I'll have a look at that this evening, and see if I can come up with a representation that works!

[flrgsr]

that would be fantastic! thx @waylonflinn

[waylonflinn]

I haven't forgotten about this. I started testing it and discovered that I needed to implement division (or multiplication) in the library to create a data set.

If you have input data that you can share, that would be a great help.