[Numpy-discussion] NumPy nogil API

[Dag Sverre Seljebotn]

On 10/31/2011 12:01 PM, Dag Sverre Seljebotn wrote:
> On 10/31/2011 11:48 AM, mark florisson wrote:
>> On 31 October 2011 10:03, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no>   wrote:
>>> Mark: I'm just wondering what you wanted to do with NumPy from Cython -- a
>>> stopgap solution for SIMD, iterator support, or something else?
>>>
>>> SIMD using NumPy really isn't the best idea long-term because of all the
>>> temporaries needed in compound expressions, which is really bad on the
>>> memory bus for anything but tiny arrays. For that I'd rather look at finding
>>> a nogil core of numexpr or similar.
>>
>> Yes I'm aware of numexpr and the general problem with array
>> expressions in NumPy. It's not just about SIMD or iterators, it's as
>> you say below, there's lots of stuff that wouldn't be available even
>> if we get SIMD. And if NumPy would get such an API, Cython could
>> figure out how many (or if) temporaries are actually needed and call
>> into the NumPy API with inplace operations.
>>
>> The thing is, how much of NumPy (and numexpr or theano) does Cython
>> want to reimplement? Will you stop at SIMD with elemental functions?
>> And will it run on my GPU?
>>
>> I suppose from a purity perspective I'd just like this functionality
>> to be available in a library and have my language use the library
>> efficiently behind my back, instead of implementing everything itself.
>
> I do totally agree, but I'm also afraid that this is a neverending quest
> as long as the GIL is present in CPython. There will always be stuff I'd
> like to call without the GIL. Only NumPy is not sufficient; I'd also
> like to use all the scientific libraries which relies on and extends
> NumPy (all of SciPy for starters), and so on.

As an example, it'd be nice to have scipy.ndimage available without the GIL:

http://docs.scipy.org/doc/scipy/reference/ndimage.html

Now, this *can* easily be done as the core is written in C++. I'm just 
pointing out that some people may wish more for calling scipy.ndimage 
inside their prange than for some parts of NumPy.

Although if the proposal is some way of writing Cython code that makes 
skipping the GIL possible to a larger degree than today then this is 
indeed highly relevant. People are already talking loudly about 
reimplementing NumPy in Cython (e.g., Travis Oliphant on his blog)., and 
SciPy is already largely reimplemented in Cython for the .NET port.


Dag Sverre