[Python-Dev] buffer object (was: Unicode debate)

[Greg Stein]

[ damn, I wish people would pay more attention to changing the subject
  line to reflect the contents of the email ... I could not figure out if
  there were any further responses to this without opening most of those
  dang "Unicode debate" emails. sheesh... ]

On Tue, 2 May 2000, M.-A. Lemburg wrote:
> Guido van Rossum wrote:
> > 
> > [MAL]
> > > Let's not do the same mistake again: Unicode objects should *not*
> > > be used to hold binary data. Please use buffers instead.
> > 
> > Easier said than done -- Python doesn't really have a buffer data
> > type.

The buffer object. We *do* have the type.

> > Or do you mean the array module?  It's not trivial to read a
> > file into an array (although it's possible, there are even two ways).
> > Fact is, most of Python's standard library and built-in objects use
> > (8-bit) strings as buffers.

For historical reasons only. It would be very easy to change these to use
buffer objects, except for the simple fact that callers might expect a
*string* rather than something with string-like behavior.

>...
> > > BTW, I think that this behaviour should be changed:
> > >
> > > >>> buffer('binary') + 'data'
> > > 'binarydata'

In several places, bufferobject.c uses PyString_FromStringAndSize(). It
wouldn't be hard at all to use PyBuffer_New() to allow the memory, then
copy the data in. A new API could also help out here:

  PyBuffer_CopyMemory(void *ptr, int size)


> > > while:
> > >
> > > >>> 'data' + buffer('binary')
> > > Traceback (most recent call last):
> > >   File "<stdin>", line 1, in ?
> > > TypeError: illegal argument type for built-in operation

The string object can't handle the buffer on the right side. Buffer
objects use the buffer interface, so they can deal with strings on the
right. Therefore: asymmetry :-(

> > > IMHO, buffer objects should never coerce to strings, but instead
> > > return a buffer object holding the combined contents. The
> > > same applies to slicing buffer objects:
> > >
> > > >>> buffer('binary')[2:5]
> > > 'nar'
> > >
> > > should prefereably be buffer('nar').

Sure. Wouldn't be a problem. The FromStringAndSize() thing.

> > Note that a buffer object doesn't hold data!  It's only a pointer to
> > data.  I can't off-hand explain the asymmetry though.
> 
> Dang, you're right...

Untrue. There is an API call which will construct a buffer object with its
own memory:

  PyObject * PyBuffer_New(int size)

The resulting buffer object will be read/write, and you can stuff values
into it using the slice notation.


> > > Hmm, perhaps we need something like a data string object
> > > to get this 100% right ?!

Nope. The buffer object is intended to be exactly this.

>...
> > Not clear.  I'd rather do the equivalent of byte arrays in Java, for
> > which no "string literal" notations exist.
> 
> Anyway, one way or another I think we should make it clear
> to users that they should start using some other type for
> storing binary data.

Buffer objects. There are a couple changes to make this a bit easier for
people:

1) buffer(ob [,offset [,size]]) should be changed to allow buffer(size) to
   create a read/write buffer of a particular size. buffer() should create
   a zero-length read/write buffer.

2) if slice assignment is updated to allow changes to the length (for
   example: buf[1:2] = 'abcdefgh'), then the buffer object definition must
   change. Specifically: when the buffer object owns the memory, it does
   this by appending the memory after the PyObject_HEAD and setting its
   internal pointer to it; when the dealloc() occurs, the target memory
   goes with the object. A flag would need to be added to tell the buffer
   object to do a second free() for the case where a realloc has returned
   a new pointer.
   [ I'm not sure that I would agree with this change, however; but it
     does make them a bit easier to work with; on the other hand, people
     have been working with immutable strings for a long time, so they're
     okay with concatenation, so I'm okay with saying length-altering
     operations must simply be done thru concatenation. ]


IMO, extensions should be using the buffer object for raw bytes. I know
that Mark has been updating some of the Win32 extensions to do this.
Python programs could use the objects if the buffer() builtin is tweaked
to allow a bit more flexibility in the arguments.

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/