A case for the buffer interface (was: Improved struct module)

[Robin Boerdijk]

[Tim about the buffer interface]
> >> It's not ready for prime time (e.g., it's easy to crash the interpreter
> >> with the little that's already there, staying within pure Python).
>
[Robin]
> > Can you give me an example ?
>
[Tim]
> Yes, but I won't.  You can create your own by thinking about the
> implementation:  a buffer object contains a pointer to a base object and a
> separate pointer to a chunk of untyped memory.  If the base object happens
> to stop using that memory, there's no way for the buffer object to know
> that.  And if that memory happens to get reused by some other object for
> something else (or even unmapped by the OS), anything at all can happen --
> it's a hole so big even Perl could fit into it <wink>.

Yes, this obviously doesn't work. But what has this got to do with the
buffer *interface* ? I now that the Python built in buffer *objects* work
like
this and that's why I don't use them. The buffer object also *happens* to
have a
buffer interface, which is probably very confusing.

> Your new objects are
> safe because they hold on to their buffer memory until they die (and so
long
> as a buffer object refers to them, they can't die).

Exactly, and every Python object should (either directly or indirectly)
regardless of whether it implements the buffer interface or not.

> Some object types
> aren't so nicely behaved.

I'm assuming you are talking about the Python buffer object here. When I
first
read about the buffer object, I thought, great, that's exactly what I need.
I
had expected this to be nothing more than a "writeable string", which is
very
useful, but it turned out to be the strange thing that it is.

Now, back to the buffer interface. The reason why it is so useful is because
it allows direct but controlled access to the internal data buffers of
objects.
If every Python object that holds an internal memory buffer exposes the
buffer
interface, copying between them, for example, can be very efficient (as
opposed
to through intermediate string objects).

It may be interesting to note that string objects themselves expose a
(read-only)
buffer interface. As a matter of fact, PyArg_ParseTuple(args, "s#", &str,
&len)
doesn't even care if the argument is a string; any object that exposes a
readable
buffer through its buffer interface can be used here.

There are more examples of the proper use of the buffer interface in Python,
such
as the readinto() method of the file object type. The cStringIO and
socket object types have not followed in its footsteps yet, but I'd be happy
to add
this. And then I would also add the writable string object and probably call
it
(suprise): buffer.

I hope this also puts my buffer related arguments for the xstruct module
into perspective.

Robin.