[Numpy-discussion] NumPy nogil API

[Dag Sverre Seljebotn]

On 10/31/2011 11:48 AM, mark florisson wrote:
> On 31 October 2011 10:03, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no>  wrote:
>> Mark: I'm just wondering what you wanted to do with NumPy from Cython -- a
>> stopgap solution for SIMD, iterator support, or something else?
>>
>> SIMD using NumPy really isn't the best idea long-term because of all the
>> temporaries needed in compound expressions, which is really bad on the
>> memory bus for anything but tiny arrays. For that I'd rather look at finding
>> a nogil core of numexpr or similar.
>
> Yes I'm aware of numexpr and the general problem with array
> expressions in NumPy. It's not just about SIMD or iterators, it's as
> you say below, there's lots of stuff that wouldn't be available even
> if we get SIMD. And if NumPy would get such an API, Cython could
> figure out how many (or if) temporaries are actually needed and call
> into the NumPy API with inplace operations.
>
> The thing is, how much of NumPy (and numexpr or theano) does Cython
> want to reimplement? Will you stop at SIMD with elemental functions?
> And will it run on my GPU?
>
> I suppose from a purity perspective I'd just like this functionality
> to be available in a library and have my language use the library
> efficiently behind my back, instead of implementing everything itself.

I do totally agree, but I'm also afraid that this is a neverending quest 
as long as the GIL is present in CPython. There will always be stuff I'd 
like to call without the GIL. Only NumPy is not sufficient; I'd also 
like to use all the scientific libraries which relies on and extends 
NumPy (all of SciPy for starters), and so on.

I do feel that what we have + SIMD covers just enough situations that it 
is useful for writing "numerical cores", without needing the rest of 
NumPy. If one starts to pull in more conveniences then I feel I might 
equally likely need something in SciPy.

I'm not really against what you try to do; any progress at all on how 
much one can do without the GIL is great, I'm just playing the devil's 
advocate for a bit.

Dag Sverre