[Numpy-discussion] Use OpenBLAS for the binary releases?

[Sturla Molden]

Ok, so using BLAS on each side of a fork requires an exec. That means 
multiprocessing on Mac must behave as it does on Windows, with the 
restrictions there. I.e. no fork, no closures as target kwarg to 
Process, etc. But that is also why multiprocessing feels crippled on 
Windows, and why the GIL sucks 10x more on Windows than Mac or Linux.

Sturla


On 21.11.2012 15:37, David Cournapeau wrote:
> On Wed, Nov 21, 2012 at 2:31 PM, Sturla Molden<sturla at molden.no>  wrote:
>> On 21.11.2012 15:01, David Cournapeau wrote:
>>> On Wed, Nov 21, 2012 at 12:00 PM, Sturla Molden<sturla at molden.no>   wrote:
>>>> But do we need a binary OpenBLAS on Mac? Isn't there an accelerate framework with BLAS and LAPACK on that platform?
>>>
>>> Because of this: https://gist.github.com/2027412
>>>
>>> Numpy + accelerate is essentially unusable with multiprocessing.
>>
>> See this too:
>>
>> http://stackoverflow.com/questions/8748790/memory-leak-in-dgemm
>>
>> If they put a memory leak in DGEMM, basically most of LAPACK is useless
>> as well. That is very sloppy of Apple.
>>
>> And how could they fail to detect that DGEMM crashes on fork()? Amazing...
>>
>> It doesn't seem the Grand Central Dispatch is very robust then...
>
> I don't think it has anything to do with robustness as much as
> multiprocessing doing things not supported by GCD. The answer we got
> from Apple when we looked into this with Olivier Grisel at pycon was:
>
> """
> For API outside of POSIX, including GCD and technologies like
> Accelerate, we do not support usage on both sides of a fork(). For
> this reason among others, use of fork() without exec is discouraged in
> general in processes that use layers above POSIX.
>
> We recommend that you either restrict usage of blas to the parent or
> the child process but not both, or that you switch to using GCD or
> pthreads rather than forking to create parallelism.
>
> Please also see the following in the fork(2) manpage:
>
> CAVEATS
>       There are limits to what you can do in the child process.  To be
> totally safe you should restrict your-
>       self to only executing async-signal safe operations until such
> time as one of the exec functions is
>       called.  All APIs, including global data symbols, in any
> framework or library should be assumed to be
>       unsafe after a fork() unless explicitly documented to be safe or
> async-signal safe.  If you need to use
>       these frameworks in the child process, you must exec.  In this
> situation it is reasonable to exec your-
>       self.
> """
>
> cheers,
> David
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion