Refactor key-management

[cowlicks]

Objects which contexts create are now put in dictionaries on the engines. This is still a work in progress, but it currently has all tests passing. I still need to alter decorators.py to use this properly.

[cowlicks]

Also needs some tests. And probably has some dead code left around.

[cowlicks]

Now when we create a context object. It creates an empty dict on the engines with a name like context_<uid>. Then when we create objects with that context, we create a name on the engines it is of the form context_<uid>['key_<uid>'] so when we assign something to it, it is put in the context's dict.

[cowlicks]

Closes issue #307.

[kwmsmith]

@cowlicks this and PR #317 are related -- I'm thinking we should merge #317 first and then rebase this on top of that one.

One change that will have to be made in this PR is that the context dictionaries should use the distarray.context.DISTARRAY_BASE_NAME prefix so the cleanup machinery works.

[cowlicks]

Sounds good, I'll rebase and incorporate those changes after #317 is in.

[kwmsmith]

@cowlicks -- I'm trying to do the following after running the testsuite:

def dump(prefix):
    names = {}
    for name in globals():
        if name.startswith(prefix):
            names[name] = globals()[name]
    return names

print view.apply_async(dump, '__distarray__').get_dict()

But this leads to a PicklingError: Can't pickle <class 'distarray.local.localarray.LocalArray'>: it's not the same object as distarray.local.localarray.LocalArray

Any idea what's going on here? Is this a regression or the same as previous behavior? ISTR that I could do this before this PR...

[cowlicks]

@kwmsmith Are you getting that error on the client or on the engines?

[kwmsmith]

@cowlicks I'm inside an IPython session, with a new Client. The error is a CompositeError from each engine.

[cowlicks]

This works fine for me on macosx...? Here is what I did

• pkill python
• make restart_cluster
• make test
• ipython

from IPython.parallel import Client; c = Client(); v = c[:]

def dumps(prefix):
    names = {}
    for name in globals():
        if name.startswith(prefix):
            names[name] = globals()[name]
    return names

print view.apply_async(dump, '__distarray__').get_dict()

Out[4]: {0: {}, 1: {}, 2: {}, 3:{}}

[kwmsmith]

I'm getting "too many open files" exception with this one -- looking into it...

[cowlicks]

I'll rebase on master so travis should annouche the number of files.

[kwmsmith]

OK, after updating the branch, the "too many open files" issue goes away -- it may have been something else with my setup that was shaken out.

[kwmsmith]

OK, I confirm that this works with PR #331 . I feel like a bigger cleanup of key management is needed, but this PR is certainly an improvement.