[Numpy-discussion] The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows)

[Nathaniel Smith]

On Fri, Apr 11, 2014 at 11:26 PM, Sturla Molden <sturla.molden at gmail.com> wrote:
> On 11/04/14 20:47, Nathaniel Smith wrote:
>
>> Also, while Windows is maybe in the worst shape, all platforms would
>> seriously benefit from the existence of a reliable speed-competitive
>> binary-distribution-compatible BLAS that doesn't break fork().
>
> Windows is worst off, yes.
>
> I don't think fork breakage by Accelerate is a big problem on Mac OS X.
> Apple has made clear that only POSIX APIs are fork safe. And actually
> this is now recognized as an error in multiprocessing and fixed in
> Python 3.4:
>
> multiprocessing.set_start_method('spawn')

I don't really care whether it's *documented* that BLAS and fork are
incompatible. I care whether it *works*, because it is useful
functionality :-).

The spawn mode is fine and all, but (a) the presence of something in
3.4 helps only a minority of users, (b) "spawn" is not a full
replacement for fork; with large read-mostly data sets it can be a
*huge* win to load them into the parent process and then let them be
COW-inherited by forked children. ATM the only other way to work with
a data set that's larger than memory-divided-by-numcpus is to
explicitly set up shared memory, and this is *really* hard for
anything more complicated than a single flat array.

> On Linux the distributions will usually ship with prebuilt ATLAS.

And it's generally recommended that everyone rebuild their own ATLAS
anyway. I can do it, but I'd much rather be able to install a BLAS
library that just worked. (Presumably this is a large part of why
scipy-stack distributors prefer MKL over ATLAS.)

If it comes down to it then of course I'd rather have a Windows-only
BLAS than no BLAS at all. I just don't think we should be setting our
sights so low at this point. The marginal cost of portability doesn't
seem high.

Besides, even Windows users will benefit more from having a standard
cross-platform BLAS that everyone uses -- it would mean lots more
people familiar with the library's quirks, better testing, etc.

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org