Add apply function like IPython's apply

[cowlicks]

No description provided.

[kwmsmith]

apply should take a flag argument that indicates whether it should return the result of func on the engines directly, or if it should store the result on the engines like here. Sometimes we want the result returned to the client, other times we want the result stored on the engines and the name returned.

[cowlicks]

@kwmsmith I added the flag with tests.

[kwmsmith]

@cowlicks Better, but there are usecases where we don't want the result stored on the engines at all, just returned directly to the end user. I think a more straightforward option that would cover this usecase would be something like:

def func_wrapper(func, result_name, args, kwargs):
    # ... args and kwargs processing code as before ...
    result = func(*args, **kwargs)
    if result_name:
        setattr(main, result_name, result)
        result = result_name
    return result

And then the apply() function would have an argument result_name=None, and no return_name=True argument. This way we only store objects on the engines if explicitly asked to do so.

[cowlicks]

I think the default behavior should be returning the name since it is less costly to send than the potentially large set of results.

[cowlicks]

Now when result_name is passed in, A variable of that name is created on the engines, and result_name is returned to act as a proxy to the result on the engines.

if result_name is None then a list of the result on each engine is returned.

[kwmsmith]

Minor question: why use *args and **kwargs rather than just passing in a list for args and a dict for kwargs? It would clean up some of the logic for calling func_wrapper below inside apply.

[cowlicks]

I think I originally did this because _really_apply expands the arguments. However I can work around this.

[cowlicks]

@kwmsmith noted

[kwmsmith]

@cowlicks looks great, merging.