Nested WorkerPool led to Error: runtime/cgo: pthread_create failed: Resource temporarily unavailable

[tle4336]

Hi Mr. Jansen @sybrenjansen, another issue that I encountered when using nested WorkerPool is runtime/cgo: pthread_create failed: Resource temporarily unavailable. I am not sure if this package mpire has some functionality/trick to prevent this from happening, particularly within the Unix-based Docker container? The total number of WorkerPool that I set for the outer and inner pool was 25 and 22, which fits with the number of cores in the environment (47 cores)? Or should I set it with 13 cores for outer pool and 3 cores for inner pool, given the outer pool handles a much more heavy job.

[sybrenjansen]

When you start 25 children in the outer and 22 child processes in the inner, you create 2522 = 550 jobs. 133 is a better choice for your system I would say.

I usually get the pthread_create failed error when the task that I'm running itself uses multithreading on all cores. Then n_jobs * all threads is usually more than the threads limit on my system.

Also, nested pools are very experimental. So use it with caution.