[Numpy-discussion] core library structure

[Darren Dale]

On Thu, Feb 3, 2011 at 2:07 PM, Mark Wiebe <mwwiebe at gmail.com> wrote:
> Moving this to a new thread.
> On Thu, Feb 3, 2011 at 10:50 AM, Charles R
> Harris <charlesr.harris at gmail.com> wrote:
>>
>> On Thu, Feb 3, 2011 at 11:07 AM, Mark Wiebe <mwwiebe at gmail.com> wrote:
[...]
>>> Yeah, I understand it's the result of organic growth and merging from
>>> many different sources. The core library should probably become layered in a
>>> manner roughly as follows, with each layer depending only on the previous
>>> APIs.  This is what Travis was getting at, I believe, with the generator
>>> array idea affecting mainly the Storage and Iteration APIs, generalizing
>>> them so that new storage and iteration methods can be plugged in.
>>> Data Type API: data type numbers, casting, byte-swapping, etc.
>>> Array Storage API: array creation/manipulation/deletion.
>>> Array Iteration API: array iterators, copying routines.
>>> Array Calculation API: typedefs for various types of calculation
>>> functions, common calculation idioms, ufunc creation API, etc.
>>> Then, the ufunc module would use the Array Calculation API to implement
>>> all the ufuncs and other routines like inner, dot, trace, diag, tensordot,
>>> einsum, etc.
>>
>> I like the lower two levels if, as I assume, they are basically aimed at
>> allocating, deallocating blocks of memory (or equivalent) and doing basic
>> manipulations such as dealing with endianess and casting. Where do you see
>> array methods making an appearance?
>
> That's correct. Currently, for example, the cast functions take array
> objects as parameters, something that would no longer be the case.  The
> array methods vs functions only shows up in the Python exposure, I believe.
>  The above structure only affects the C library, and how its exposed to
> Python could remain as it is now.
>>
>> The original Numeric only had three (IIRC) rather basic methods and
>> everything else was function based, an approach which is probably easier to
>> maintain. The extensive use of methods came from numarray and might be
>> something that could be added at a higher level so that the current ndarrays
>> would be objects combining ow level arrays and ufuncs.

Concerning ufuncs: I wonder if we could consider generic functions as
a replacement for the current __array_prepare__/__array_wrap__
mechanism. For example, if I have an ndarray, a masked array, and
quantity, and I want to multiply the three together, it would be great
to be able to do so with two calls to a single mul ufunc.

Also, a generic function approach might provide a better mechanism to
allow changes to the arrays on their way into the ufunc:

import numpy as np
import quantities as pq
a = [1,2,3]*pq.deg # yields a subclass of ndarray
np.sin(a)

This is not currently possible with __array_prepare__/__array_wrap__,
because __array_prepare__ is called too late by the ufunc to return
process the input array and rescale it to radians. I suggested on this
list that it might be possible to so with the addition of a *third*
method, call it __input_prepare__... at which point Chuck rightly
complained that things were getting way out of hand.

Imagine if we could do something like

@np.sin.register(pq.Quantity)
def my_sin(q):
    return np.sin.default(q.rescale('radians'))

np.sin(a)

With generic functions, you wouldn't have to remember to use the ufunc
provided by masked array for one type, or the default numpy for
another type.

This is something I have been meaning to revisit on the list for a
while (along with the possibility of merging quantities into numpy),
but keep forgetting to do so.

Darren