[Python-Dev] buffer objects

[Scott Gilbert]

Mark Hammond <mhammond@skippinet.com.au>:
> 
> It turns out that the buffer *interface* has a design flaw, best
> described by using the array object as an example.  The real
> possibility exists that code can use the buffer interfaces to get a
> pointer to writable memory for the array.  The array may then be
> resized, thereby moving the buffer but leaving the buffer pointer
> dangling.
> 

Thank you for the very clear explanation.  Now I see why there is a
problem.  Actually there are a number of implementation problems in
PyBufferObject.

This isn't a problem with the buffer() builtin though, and it's not a
really a problem with making the buffer() builtin return a read-write
object.

Even without my change to the buffer() builtin, the C-API to
PyBufferObjects can currently still cause real damage.  I'm working on a
patch to fix this if anyone wants to champion it.

If no one wants to champion this new patch, then I need to withdraw my
older patch as it is working on incorrect assumptions.  I'll also submit
bug reports about the list of other issues in bufferobject.c.  (Hashing is
broken for buffer objects too.)


Greg Ewing <greg@cosc.canterbury.ac.nz>:
>
> If you do that using the (C-level) buffer interface,
> you're misusing it. You shouldn't be keeping a pointer
> to the internals of an object across any operation which
> could cause it to move, however that pointer is obtained.
> 
> The design flaw is in the Python-level buffer *object*,
> which misuses the buffer interface in exactly that way.
> This is so insane that I can't understand why the
> buffer object hasn't either been fixed or ripped
> out by now.
>

You're right.  Bummer too.  That means an extension module can't release
the GIL while working with the pointer to memory.  (For instance to perform
intensive calculations with the pointer on one processor while letting
another thread run concurrently.)

It also means that other asynchronous operations can't safely work on
pointers returned from PyBufferProcs.  (Unless the GIL is held the whole
time, and that would probably make doing anything asynchronous less
interesting...)

So PyBufferProcs is less useful than I thought it was a day ago.  I naively
assumed the pointer was supposed to be constant for the lifetime of the
object.


> 
> The fix seems obvious to me -- don't cache the pointer
> in the buffer object, but use the buffer interface to
> re-fetch it on each Python-level call which needs it.
> 

Right you are, and that's what I'm doing.  Patch coming soon...


Cheers,
    -Scott


__________________________________________________
Do You Yahoo!?
Yahoo! Health - your guide to health and wellness
http://health.yahoo.com