[Python-Dev] buffer interface considered harmful

[M.-A. Lemburg]

Greg Stein wrote:
> 
> Fredrik Lundh wrote:
> >
> > > I think the buffer interface was introduced in 1.5 (by Jack?). I added
> > > the 8-bit character buffer slot and buffer objects in 1.5.2.
> > >
> > > > from array import array
> > > >
> > > > a = array("f", [0]*8192)
> > > >
> > > > b = buffer(a)
> > > >
> > > > for i in range(1000):
> > > >     a.append(1234)
> > > >
> > > > print b
> > > >
> > > > in other words, the buffer interface should
> > > > be redesigned, or removed.
> > >
> > > I don't understand what you believe is weird here.
> >
> > did you run that code?
> 
> Yup. It printed nothing.
> 
> > it may work, it may bomb, or it may generate bogus
> > output. all depending on your memory allocator, the
> > phase of the moon, etc. just like back in the C/C++
> > days...
> 
> It probably appeared as an empty string because the construction of the
> array filled it with zeroes (at least the first byte).
> 
> Regardless, I'd be surprised if it crashed the interpreter. The print
> command is supposed to do a str() on the object, which creates a
> PyStringObject from the buffer contents. Shouldn't be a crash there.
> 
> > imo, that's not good enough for a core feature.
> 
> If it crashed, then sure. But I'd say that indicates a bug rather than a
> design problem. Do you have a stack trace from a crash?
> 
> Ah. I just worked through, in my head, what is happening here. The
> buffer object caches the pointer returned by the array object. The
> append on the array does a realloc() somewhere, thereby invalidating the
> pointer inside the buffer object.
> 
> Icky. Gotta think on this one... As an initial thought, it would seem
> that the buffer would have to re-query the pointer for each operation.
> There are performance implications there, of course, but that would
> certainly fix the problem.

I guess that's the way to go. I wouldn't want to think
about those details when using buffer objects and a function call
is still better than a copy... it would do the init/exit
wrapping implicitly: init at the time the getreadbuffer
call is made and exit next time a thread switch is done - 
provided that the functions using the memory pointer also
keep a reference to the buffer object alive (but that should
be natural as this is always done when dealing with references
in a safe way).

-- 
Marc-Andre Lemburg
______________________________________________________________________
Y2000:                                                   138 days left
Business:                                      http://www.lemburg.com/
Python Pages:                           http://www.lemburg.com/python/