[Python-Dev] PEP-298: buffer interface widening too much?

[Todd Miller]

I've been following the PEP-298 discussion a little, because I fear 
eventually IAGNI.   But when I look at the proposed changes,  I can't 
help but get the feeling that the "too wide" buffer interface just got 
wider.  By "too wide",  I am referring to the extraneous segmentation 
and character buffer aspects of the existing interface.  Adding three 
more calls for the locking problem bugs me.  Doesn't anything ever get 
simpler?

Looking at the proposed changes,  the only thing that really seems 
needed for the bufferprocs is the release call and a flag bit.  Assuming 
the release call is not implemented,  the acquired buffer is never 
locked or unlocked, and we're no worse off than we are now:  it's the 
BO's problem.   Assuming the release call is present,  the acquired 
buffer should *always* be locked and later released.   The existing 
wrapper functions (plus one for release) can hide/implement this, making 
it straight forward to provide correct extensions:  just implement a 
locking extension.  It seems to me that the existing (3!) calls for 
acquiring the buffer pointer can do just fine.   Except...

In the case of existing extensions,  there will certainly be no call to 
the release function.   So for an extension which is slow to update to 
the revised protocol,  there are negative consequences:  buffers which 
are never unlocked.   But so what?  These consequences seem less dire 
(if more probable) than the existing consequences of "moving buffers." 
 It would be a bigger problem if PEP-298 wound up fiddling with 
reference counts.  In that case,  un-released buffers would also become 
memory leaks.   But is it really necessary to mess with the reference 
counts?   I'm having a hard time imagining a situation in which the 
reference count for a buffer goes to zero while it is locked which is 
not a bug somewhere else.  What am I missing?  

Another possibility is to implement a "release buffer pointer" wrapper 
function which *does nothing* for a Python release 2.3 and change 
nothing else.   Let extensions which use buffers adopt usage of the 
wrapper and the locking protocol, with no immediate benefit.   Then 
implement locking for real in Python 2.4.   That way,  the number of 
non-conforming extensions can be minimized,  but the buffer API can be 
kept simple.  

So,  in the end,  what I'm wondering is whether there's a simpler middle 
ground between dropping the PEP and implementing it as it stands now.   
 Are there compromises to be found between guaranteeing correctness, 
backwards compatability, and growing complexity?   Is there another 
trade between backwards compatability and complexity which is tolerable 
for the short term with both simplicity and correctness available in the 
longer term?  

Todd