[SciPy-Dev] General discussion on parallelisation

[Ralf Gommers]

On Mon, Sep 3, 2018 at 3:34 AM Andrew Nelson <andyfaff at gmail.com> wrote:

> <sheepishly> Ok, I managed to search and AFAICT the main issue is that if
> a child process terminates then the master process can hang waiting for it
> to return (https://bugs.python.org/issue22393).
>
> > This is merged and released since a while. Loky is now used in anger. So
> far, I think that people are happy with it. In particular it is more robust
> than multiprocessing.Pool (specifically, robust to segfault), and the
> improvements have been contributed upstream to Python concurrent.futures's
> process pool executor (available in Python 3.7).
>

Thanks for the details Gael!

So, reading the `concurrent.futures.ProcessPoolExecutor` documentation
> indicates that it is resistant to this issue. concurrent.futures is
> available in Python3, but wasn't ported to 2.7.
>

The PR now uses multiprocessing on Python 2.7 and concurrent.futures on 3.x
- this seems fine to me. We're not supporting 2.7 for that much longer, so
the code can be simplified a bit when we drop 2.7

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180903/32c37d93/attachment.html>