[SciPy-Dev] General discussion on parallelisation

[Gael Varoquaux]

On Mon, Sep 03, 2018 at 07:30:22PM +1000, Andrew Nelson wrote:
> SNIP
> > In particular it is more robust than multiprocessing.Pool (specifically,
> robust to segfault), and the improvements have been contributed upstream to
> Python concurrent.futures's process pool executor (available in Python 3.7).

> SNIP
> > While simpler multiprocessing-based code will sometimes give less overhead
> compared to joblib, it will probably be brittle.

> Is there anything I can read to learn a bit more about this? Is it mainly
> related to how easy things are to pickle?

The major problem are related to state inside multiprocessing.Pool that
can be broken when child processes go astray. As a result,
multiprocessing.Pool often gets into deadlock (any heavy user of
multiprocessing has probably experienced this). See for instance
https://tommoral.github.io/talks/pyparis17/#37

Gaël