[SciPy-Dev] Using OpenMP in SciPy

[Jeremie du Boisberranger]

Hi everyone,

here's some feedback from the recent use of OpenMP in sklearn.


 > 1. macOS’s apple clang doesn't support openmp and supporting OpenMP 
requires distributing a runtime with your library.

We ship libgomp in the wheels for macOS users.

For users who want to build from source, they need to install libomp and 
set a few environment variables (apple clang can support OpenMP in 
preprocessing).Here 
<https://scikit-learn.org/stable/developers/advanced_installation.html> 
are the instructions to build with OpenMP support on macOS.


>  2. On macOS, the openmp version crashes if mkl numpy is installed. 
This is a general problem because several different OpenMP runtimes exist.

This issue also exists on linux. The crash comes from intel's libiomp 
which errors when it tries to load although there's already another 
OpenMP runtime loaded.

A workaround exists by setting the KMP_DUPLICATE_OK environment 
variable, which is what we do in the init of sklearn. I think it's fine 
as long as we don't do too fancy stuff such as trying to dynamically 
manage the threadpool of one lib while inside a parallel region of 
another lib.


>  3. openmp doesn't play nicely with multiprocessing.Pool

I thought that was fixed in python 3.4 with the forkserver start method. 
In sklearn we now only support 3.5+.


> Some macOS users still needed the option to compile the library without 
OpenMP. Although, this might have also been Accelerate related.

I don't know the reason why some users needed to build without openmp 
support.

In sklearn we only use OpenMP in cython through prange. We added the 
possibility to build without OpenMP support via an environment variable 
(but we don't provide wheels without openmp). It allows to be sure 
there's a way to build sklearn without triggering any OpenMP issues 
because it was added recently and users haven't found all possible ways 
to break it yet :)


Jérémie



On 02/08/2019 21:24, Peter Bell wrote:
> Hello all,
> I’d like to revisit a discussion started a while ago in _gh-10239_ 
> <https://github.com/scipy/scipy/issues/10239>: Can we build and use 
> OpenMP in a portable way?
> _andyfaff_ <https://github.com/scipy/scipy/issues/10239>’s comment on 
> the issue pointed out a number of portability issues that he’s 
> experienced that put him off of OpenMP. Summarising:
>
>  1. macOS’s apple clang doesn't support openmp and supporting OpenMP
>     requires distributing a runtime with your library.
>  2. On macOS, the openmp version crashes if mkl numpy is installed.
>     This is a general problem because several different OpenMP
>     runtimes exist.
>  3. openmp doesn't play nicely with multiprocessing.Pool
>
> To expand on that 3^rd point, some openmp runtimes aren’t fork-safe. 
> Most notably, this includes gcc’s libgomp. Upon entering the first 
> openmp parallel region, the runtime initializes a thread pool which 
> won’t be rebuilt in the child after fork. This means that any parallel 
> regions in the child will deadlock. Single threaded openmp loops seem 
> to be safe though.
> Eric Larson has also recently had a chat with an sklearn maintainer 
> which has been using OpenMP in the wild for ~6 months.  A few of the 
> key take-aways from his discussion were:
>
>   * Packaging OpenMP code has many pitfalls but they think they’ve
>     found a solution that works for them.
>   * They use joblib with the “loky” backend to avoid multiprocessing
>     and the associated forking issues.
>   * They had issues related to incompatibility with macOS Accelerate
>     for BLAS/LAPACK but SciPy has already _dropped support for
>     accelerate_
>     <https://github.com/scipy/scipy/wiki/Dropping-support-for-Accelerate>.
>   * Some macOS users still needed the option to compile the library
>     without OpenMP. Although, this might have also been Accelerate
>     related.
>
> So, given these issues, are we willing to use OpenMP in SciPy?
> Peter
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20190804/10238d07/attachment-0001.html>