[Python-Dev] Extended Buffer Interface/Protocol

[Travis Oliphant]

I'm soliciting feedback on my extended buffer protocol that I am 
proposing for inclusion in Python 3000.  As soon as the Python 3.0 
implementation is complete, I plan to back-port the result to Python 
2.6, therefore, I think there may be some interest on this list.

Basically, the extended buffer protocol seeks to allow memory sharing with

1) information about what is "in" the memory (float, int, C-structure, etc.)
2) information about the "shape" of the memory (if any)

3) information about discontiguous memory segments


Number 3 is where I could use feedback --- especially from PIL users and 
developers.   Strides are a common way to think about a possibly 
discontiguous chunk of memory (which appear in NumPy when you select a 
sub-region from a larger array). The strides vector tells you how many 
bytes to skip in each dimension to get to the next memory location for 
that dimension.

Because NumPy uses this memory model as do several compute libraries 
(like BLAS and LAPACK), it makes sense to allow this memory model to be 
shared between objects in Python.

Currently, the proposed buffer interface eliminates the multi-segment 
option (for Python 3.0) which I think was originally put in place 
because of the PIL.   However, I don't know if it is actually used by 
any extension types.  This is a big reason why Guido wants to drop the 
multi-segment interface option.

The question is should we eliminate the possibility of sharing memory 
for objects that store data basically as "arrays" of arrays (i.e. true 
C-style arrays).  That is what I'm currently proposing, but I could also 
see an argument that states that if we are going to support strided 
memory access, we should also support array of array memory access.

If this is added, then it would be another function-call that gets a 
array-of-array-style memory from the object.  What do others think of 
these ideas?


One possible C-API call that Python could grow with the current buffer 
interface is to allow contiguous-memory mirroring of discontiguous 
memory, or an iterator object that iterates through every element of any 
object that exposes the buffer protocol.


Thanks for any feedback,

-Travis Oliphant