[Numpy-discussion] Trying out Numeric3

[Travis Oliphant]

Chris Barker wrote:

> Scott Gilbert wrote:
>
>> I don't know what particular meta data you plan to store with the buffer
>> itself, and I'm going to resist the urge to guess.  You probably have 
>> some
>> very good use cases.  What are you planning?
>
>
> I don't know what Travis has in mind, but I thought I'd bring up a use 
> case that I think provides some of the motivation for this.
>
This is exactly the kind of thing I mean.   One of the reasons for 
putting Numeric in the core is so other extension writers can use it 
reliably.  But, really, a better solution is to create ways to deal with 
each others memory reliably.    I think the bytes object Scott proposed 
is really very close to what is needed.   Extension writers will likely 
need some additional information in order to use somebody else's byte 
object well.  What information is needed can vary (but it should be 
standard for different types of objects).   You can already get at the 
memory directly through the buffer protocol.   But,

I think pickling could be handled efficiently with a single new opcode 
very similar to a string but instead creating a bytes object on 
unpickling.   Then, an array could use the memory of that bytes object 
(instead of creating it's own and copying).   This would be very easy to 
handle. 

I really believe we need to push forward the bytes object.  This 
attitude towards the buffer interface (due to an easily fixed problem) 
is really disquieting.   I could spend time pushing the bytes object, 
but it would take away from the work I'm currently doing.  This is 
something that we really need some help with. 

Scott's PEP is quite good and gives an outline for how to do this, so 
except for pickling you don't even need to be an expert to get something 
done.    You could start with the outline of numarray's memory object 
(getting rid of the Int64 stuff in it), and proceeding from there.

It would probably take a week to get it done.

>
> My understanding of this memory object is that an extension like 
> wxPython wouldn't not need to know about Numeric, but could simply get 
> the memory Object, and there would be enough meta-data with it to 
> typecast and loop through the data. I'm a bit skeptical about how this 
> would work. It seems that the metadata required would be the full set 
> of stuff in an array Object already:
>
> type
> dimensions
> strides
>
> This could be made a bit simpler by allowing only contiguous arrays, 
> but then there would need to be a contiguous flag.

I'm thinking just contiguous arrays would be passed.  While Numeric does 
support the multi-segment buffer interface.  I doubt extension writers 
want to try and understand how to deal with it.    I think it would be 
too much of a burden to other extensions if the array they saw was not 
contiguous.    Even internal to Numeric, discontiguous arrays are made 
contiguous all the time (although the new iterator in Numeric3 makes it 
much easier for a programmer to deal with discontiguous arrays).

>
> To make use of this, wxPython would have to know a fair bit about 
> Numeric Arrays anyway, so that it can check to see if the data is 
> appropriate. I guess the advantage is that while the wxPython code 
> would have to know about Numeric arrays, it wouldn't have to include 
> Numeric headers or code.

It only has to know the shape and type (typechar and itemsize) of the 
array if we stick to contiguous arrays (this is where the typechar 
becomes very valuable). 

I still think the bytes object is a really, really good idea.  Python 
needs it very badly.   If every extension module that allocated memory 
went through a bytes object instead, then a lot of unnecessary copying 
could be minimized.

-Travis