multiprocess requirement?

[folded]

Hi,

I'm curious what is required from multiprocess that is not available from multiprocessing? The change seems to have occurred in a large refactoring CL that doesn't really spell out the reason for the change. Tests still pass if it's changed back to multiprocessing, and I'm slightly concerned that the multiprocess module won't play well with existing code that requires customized multiprocessing wrappers version to fork safely.

[nvictus]

Thanks for asking. The original motivation was related to issues with pickle serialization during multiprocessing. One is just that dill can serialize more things (as can cloudpickle) like lambdas, but you're probably right that we probably don't make use of it at the moment.

The other was limitations of the pickle protocol versions < 4, where the payload size is limited to 2GB and was causing problems with large distributed chunking operations. In hindsight, I'm not entirely sure switching to dill resolved it. As of Python 3.8, the default pickle version for multiprocessing is 4, and Python now lets you directly customize multiprocessing's serialization which wasn't the case before. So we may be able to deprecate multiprocess in favor of multiprocessing again.

I'm curious what kind of customized multiprocessing wrappers you are using. I've also been tempted to switch to loky.