[SciPy-Dev] General discussion on parallelisation

[Ralf Gommers]

reviving an old thread because there's movement on the PR (
https://github.com/scipy/scipy/pull/8259) and would be nice to get that
merged.

On Wed, Jan 10, 2018 at 2:34 AM, Gael Varoquaux <
gael.varoquaux at normalesup.org> wrote:

> > 5. joblib.Parallel doesn't have a map method (desirable to allow 3) so a
> small
>
> joblib has a custom backend framework that can be used for such purpose
> (if I understnad you well):
>
> https://pythonhosted.org/joblib/parallel.html#custom-backend-api-experimental


Updated link (status is still experimental):
https://joblib.readthedocs.io/en/latest/parallel.html#custom-backend-api-experimental


>
> There are currently a Yarn and a dask.distributed backend that are
> getting better and better.
>

There's also this JoblibPool that can be taken over:
https://github.com/adrn/schwimmbad/blob/master/schwimmbad/jl.py#L14
Seems simpler than a backend still tagged experimental.


> > 6. joblib.Parallel creates/destroys a multiprocessing.Pool each time the
> > Parallel object is `__call__`ed. This leads to significant overhead. One
> can
> > use the Parallel object with a context manager, which allows reuse of
> the Pool,
> > but I don't think that's do-able in the context of using the
> > DifferentialEvolutionSolver (DES) object as an iterator:
>
> This is evolving. However, the reason behind this is that Pool get
> corrupted and lead to deadlock. Olivier Grisel and Thomas Moreau are
> working on fixing this in the Python standard library (first PR merged
> recently)!
>

Anyone know the status of this? And can this issue be avoided by the new
loky backend to joblib?

Ralf


> One of the vision of joblib is to provide very light mid-layer that can
> connect to multiprocessing and threading (though we are considering
> switching to concurrent.futures) as well as other backends. Hopefully
> this common language makes it easier to do things like embedding dask in
> numerical algorithms without a hard dependencies (yes we are working with
> the dask team on this).
>
> Gaël
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180902/5d42ec61/attachment.html>