[Python-Dev] Fw: Behavior of buffer()

[Todd Miller]

Here's the numarray perspective on things.  

Tim Peters wrote:

>[Tim]
>
>>Fredrik pressed for details, but we haven't seen any concrete use cases.
>>In the absence of the latter, it's impossible to guess what would be
>>backward compatible for MAL's purposes.
>>
I updated my CVS copy of Python and tried out MAL's patch with numarray. 
 Nothing broke as far as I can tell.  I guess it probably doesn't matter 
anyway given that both buffer() and MAL's patch are headed to oblivion.

>>
>
>[M.-A. Lemburg]
>
>>For my purposes, the strategy buffer slice returns a buffer
>>would be more appropriate because it would save the buffer type
>>information across the slicing operation... I mean, you don't
>>want to get bananas when you slice an apple in real life either ;-)
>>
>>I use buffers to mean: this is a chunk of binary data. The purpose
>>is to recognize this type of data for pickling via xml-rpc,
>>soap and other rpc mechanisms etc.
>>
>
>How do you use buffers?  
>
We use buffers in numarray to store our array data.   We use readinto to 
load array buffers efficiently from a file.    We operate on the buffer 
data in-place.  Since numarrays are python classe instances, buffers 
provide a place for the data to live.

>Do you stick to their C API?  
>
We use the C-API, and currently use the buffer object too.   Using the 
buffer object has always seemed like a necessary evil, but having 
reviewed numarray usage of buffer(), ditching it sounds good to me.

>Do you use the
>Python-level buffer() function?  
>
Yes.  We go one step further, and expose writeable buffers using our own 
extension function.  I had a feeling I was on thin ice when I did this.

>If the latter, what do you do in Python
>code with a buffer object after you get one?  The only use I've seen made of
>a buffer object in Python code is as a way to trick the interpreter into
>crashing (via recycling the memory the buffer object points to).
>
I'm getting the following things by using the buffer object:

1.  Knowledge that the C-type the buffer refers to meets the buffer C-API.

2.  Mutable string behavior for any object which meets the buffer C-API.

3.  Storage.  At least we used to get storage until we found out that 
there's no guarantee on double alignment.

I plan to work around each of these uses as  follows:

1.  Create an extension function which determines whether an object 
meets the buffer C-API.

2.  Create an extension function which copies from one buffer region to 
another buffer region.  

3. We already have our own memory object which is now typically 
referenced by a buffer object.  With the above extensions, I don't need 
a buffer "wrapper" object around it anymore.

>
>
>And from where do you get a buffer?  There are darned few types in Python
>
We get ours from mmap and our own homegrown memory object.

>
>that buffer() accepts as an argument.  Do your extension types implement
>tp_as_buffer?  I'm blindly casting for a reason why your appreciation of the
>
>
>buffer object seems unique.
>
Numarray uses buffer() too, but dumping it sounds OK.

Todd

-- 
Todd Miller 
Space Telescope Science Institute