[Numpy-discussion] Use OpenBLAS for the binary releases?

[Sturla Molden]

But if this issue is in the GCD, it will probably affect other 
applications that uses the GCD and fork without exec as well. Unless you 
are certain the GCD is not used, a fork would never be safe without an exec.

Sturla

On 21.11.2012 15:45, Sturla Molden wrote:
> Ok, so using BLAS on each side of a fork requires an exec. That means
> multiprocessing on Mac must behave as it does on Windows, with the
> restrictions there. I.e. no fork, no closures as target kwarg to
> Process, etc. But that is also why multiprocessing feels crippled on
> Windows, and why the GIL sucks 10x more on Windows than Mac or Linux.
>
> Sturla
>
>
> On 21.11.2012 15:37, David Cournapeau wrote:
>> On Wed, Nov 21, 2012 at 2:31 PM, Sturla Molden<sturla at molden.no>   wrote:
>>> On 21.11.2012 15:01, David Cournapeau wrote:
>>>> On Wed, Nov 21, 2012 at 12:00 PM, Sturla Molden<sturla at molden.no>    wrote:
>>>>> But do we need a binary OpenBLAS on Mac? Isn't there an accelerate framework with BLAS and LAPACK on that platform?
>>>>
>>>> Because of this: https://gist.github.com/2027412
>>>>
>>>> Numpy + accelerate is essentially unusable with multiprocessing.
>>>
>>> See this too:
>>>
>>> http://stackoverflow.com/questions/8748790/memory-leak-in-dgemm
>>>
>>> If they put a memory leak in DGEMM, basically most of LAPACK is useless
>>> as well. That is very sloppy of Apple.
>>>
>>> And how could they fail to detect that DGEMM crashes on fork()? Amazing...
>>>
>>> It doesn't seem the Grand Central Dispatch is very robust then...
>>
>> I don't think it has anything to do with robustness as much as
>> multiprocessing doing things not supported by GCD. The answer we got
>> from Apple when we looked into this with Olivier Grisel at pycon was:
>>
>> """
>> For API outside of POSIX, including GCD and technologies like
>> Accelerate, we do not support usage on both sides of a fork(). For
>> this reason among others, use of fork() without exec is discouraged in
>> general in processes that use layers above POSIX.
>>
>> We recommend that you either restrict usage of blas to the parent or
>> the child process but not both, or that you switch to using GCD or
>> pthreads rather than forking to create parallelism.
>>
>> Please also see the following in the fork(2) manpage:
>>
>> CAVEATS
>>        There are limits to what you can do in the child process.  To be
>> totally safe you should restrict your-
>>        self to only executing async-signal safe operations until such
>> time as one of the exec functions is
>>        called.  All APIs, including global data symbols, in any
>> framework or library should be assumed to be
>>        unsafe after a fork() unless explicitly documented to be safe or
>> async-signal safe.  If you need to use
>>        these frameworks in the child process, you must exec.  In this
>> situation it is reasonable to exec your-
>>        self.
>> """
>>
>> cheers,
>> David
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion