[SciPy-Dev] General discussion on parallelisation

[Ralf Gommers]

On Mon, Sep 3, 2018 at 1:27 PM Andrew Nelson <andyfaff at gmail.com> wrote:

> > Right now we have a single keyword `workers=1`, which behaves like
>> > scikit-learn's n_jobs=1 (# of CPUs), and also accept objects with a
>> map()
>> > method like multiprocessing.Pool
>>
>> I think that we wouldn't mind adding a "map" to a Parallel object. Though
>> it feels quite strange, as the Parallel object is meant to be
>> higher-level abstract than the Pool.
>>
>
I agree that that's strange, and it looks like we don't need it.


>
> The design also allows for things like `mpi4py.futures.MPIPoolExecutor` to
> be supplied.
>

Update for everyone: at the moment the PR is updated to use
concurrent.future.ProcessPoolExecutor on Python 3 by default, and
multiprocessing on Python 2.7. The design is compatible with joblib,
however vendoring joblib and using it for the workers=int case is left to a
future PR. Unless there's more comments/thoughts, I plan to hit the green
button after a final review/benchmark.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180906/ee2fb9c7/attachment.html>