[Python-Dev] PEP-298: buffer interface widening too much?

[Martin v. Löwis]

> I really liked the idea of doing the buffer release behind the scenes, 
>  as suggested by PEP-286 and below.

Alas, this has a fundamental flaw also, that's why I haven't advanced it,
yet. Jack found the flaw while reviewing it, but I did not manage to integrate
that into the PEP, and now I forgot what it was :-( It should be in the
mailing list archives, though.

> But won't *any* systematic solution have a performance cost?  Argtuples 
> don't sound "free".  I'm really playing devil's advocate here,  because 
> I like the concept of PEP-286.

Without any numbers to back up this claim: I believe the largest cost from
Py*Parse comes from the need to process the printf-style string, and the
corresponding va_arg processing, perhap along with the many type checks.

That's why I protest any unnecessary ParseTuple call, and object any
functions in this class. In comparison, the PEP 286 approach should
a) only produce costs if there is actually any buffer management, and
b) have a management processing cost (for building data structures and
   iterating over them) roughly the same size as the number of resources
   managed.

> I disagree here.   I didn't see PEP-298 address PyArg_ParseTuple at all, 
>  so there's no real basis for comparison on this point.    I'll grant 
> you that PyArg_ReleaseBuffers is probably a lame solution to a problem 
> better solved by PEP-286;  that said,  because use of 
> PyArg_ReleaseBuffers is explicit,  there is no problem distributing the 
> performance penalty to places where there is a corresponding benefit.  I 
> was not suggesting dropping the existing release call;   merely adding a 
> layer which could make sense out of PyArg_ParseTuple specs.

I understand that. But please compare

  if (!PyArg_ParseTuple(args,"sss",&string1, &string2, &string3))
     return NULL;
  if(!do_some_stuff(string1)){
     PyArg_Release(args,"sss",string1,string2,string3);
     return NULL;
  }
  if(!do_more_stuff(string2)){
     PyArg_Release(args,"sss",string1,string2,string3);
     return NULL;
  }
  compute_results(string3);

  PyArg_Release(args,"sss",string1,string2,string3);
  return result;

to

  if (!PyArg_ParseTuple(args,"ssO",&string1, &string2, &obj3))
     return NULL;
  if(!do_some_stuff(string1)){
     return NULL;
  }
  if(!do_more_stuff(string2)){
     return NULL;
  }
  PyBuffer_Lock(obj3, &string3);
  compute_results(string3);
  PyBuffer_Unlock(obj3);

  return result;

In general, you won't have to lock everything, so you can avoid
a lot of clutter if you use just the PEP-298 interfaces, together
with an "O" parser.

> If the default behavior for the existing Python objects (array, mmap, 
> ...) is a onetime warning when a lock is abused,  where's the issue? 

It's unclear. I would expect that every application that uses arrays will
trigger the warning. When you get the warning, you will have no clue what
you did wrong, and most likely, the error is in the Python core. Who is
going to fix the hundreds and hundreds of functions that use the "s" parser,
and who is not going to be terribly bored by this, knowing that 99% of them
will never see an array.

> My thinking is that *all* buffer pointers should be "logically" locked. 

That is the intent of the PEP, indeed. For backwards compatibility, it
can't change the existing interface, but needs to create a new one. After
some time, the core could require that objects support it, much more later,
it could remove support for the old interface.

>   Objects that don't need locks don't need to change.   All extensions 
> should be written and/or updated to handle objects which require 
> locking.  Extensions which aren't updated will trigger warnings which 
> could be supressed or converted into exceptions.

The problem is that you are talking about virtually *all* extensions.
I bet you won't find a single extension that doesn't use an "s" parser.

> For me, that's simpler than saying "correct extensions" lock the buffer, 
>  but "obsolete ones" may still segfault w/o warning.

But, with your warning, this would still happen?

> I like this idea,  but wonder if it won't inflict a performance penalty 
> on all function calls.

I hope it won't. There is no implementation of enhanced argument tuples, yet
(and well may not appear for Python 2.3); but if this is done, a 
PyArg_ParseTuple call that doesn't allocate any resources will leave a NULL
pointer in the slot for the resources - on deallocation, all you pay is the
null pointer test.

Regards,
Martin