[Python-3000] Pre-PEP: Altering buffer protocol (tp_as_buffer)

[Travis Oliphant]

Hi everybody,

It was great to see so many of you at PyCon --- even if we saw each 
other for too long during the PSF meeting.

After hearing Guido's keynote talk today and realizing that the alpha 
release of Python 3.0 is so soon,  I decided that the right approach I 
should take in pushing the array protocol/interface is to actually 
propose it being used as a replacement/enhancement of the buffer 
protocol for Python 3.0

I will write a PEP and the implementation but I would like to start a 
discussion about what concerns or issues developers have with a proposal 
like this and try to weed most of these out.

I've started a Wiki page where we can document the issues that are 
raised.  Perhaps this Wiki can become the PEP within a few weeks.

The Wiki is at http://wiki.python.org/moin/ArrayInterface

The basic idea is given below (a copy of the web-page).

Thanks for any and all feedback.

Best regards,

-Travis Oliphant



This pre-PEP proposes enhancing the buffer protocol in Python 3000 to 
implement the array interface (protocol).

= Overview =

The buffer protocol allows different Python types to exchange a pointer 
to a sequence of internal buffers.  This functionality is 
'''extremely''' useful for sharing large segments of memory between 
different high-level objects, but it's too limited and has issues.

   1. There is the little used "sequence-of-segments" option.
   2. There is no way for a consumer to tell the protocol-exporting 
object it is "finished" with its view of the memory and therefore no way 
for the object to be sure that it can reallocate the pointer to the 
memory that it owns (the array object reallocating its memory after 
sharing it with the buffer object led to the infamous buffer-object 
problem).
   3. Memory is just a pointer. There is no way to describe what's "in" 
the memory (float, int, C-structure, etc.)
   4. There is no shape information provided for the memory.  But, 
several array-like Python types could make use of a standard way to 
describe the shape of the memory (!wxPython, GTK, CVXOPT, !PyVox, Audio 
and Video Libraries, ctypes, !NumPy)

= Proposal =

   1. Replace the buffer protocol that allows sharing of a single 
pointer to memory
   2. Have the protocol define a way to describe what's in the memory 
location (this should unify what is done now in struct, array, ctypes, 
and NumPy)
   3. Have the protocol be able to share information about shape (and 
striding if any)
   4. Allow exporting objects to define some function that should be 
called when the consumer object is "done" with the view.

= Idea =

All that is needed is to create a Python "memory_view" object that can 
contain all the information needed and be returned when the buffer 
protocol is called --- when it is garbage-collected, the 
"bp_release_view" function is called on the exporting object.

This "memory_view" is essentially the old Numeric C-structure (including 
the fact that the data-format is described by another C-structure).

This object is what the buffer protocol should return.