[Python-Dev] PEP 3118: Extended buffer protocol (new version)

[Carl Banks]

Travis Oliphant wrote:
> Carl Banks wrote:
>>
>>
>> Travis Oliphant wrote:
>> > Py_BUF_READONLY
>> >    The returned buffer must be readonly and the underlying object 
>> should make
>> >    its memory readonly if that is possible.
>>
>> I don't like the "if possible" thing.  If it makes no guarantees, it 
>> pretty much useless over Py_BUF_SIMPLE.
> O.K.  Let's make it raise an error if it can't set it read-only.

The thing that bothers me about this whole flags setup is that different 
flags can do opposite things.

Some of the flags RESTRICT the kind of buffers that can be
exported (Py_BUF_WRITABLE); other flags EXPAND the kind of buffers that
can be exported (Py_BUF_INDIRECT).  That is highly confusing and I'm -1
on any proposal that includes both behaviors.  (Mutually exclusive sets
of flags are a minor exception: they can be thought of as either
RESTICTING or EXPANDING, so they could be mixed with either.)

I originally suggested a small set of flags that expand the set of 
allowed buffers.  Here's a little Venn diagram of buffers to illustrate 
what I was thinking:

http://www.aerojockey.com/temp/venn.png

With no flags, the only buffers allowed to be returned are in the "All"
circle but no others.  Add Py_BUF_WRITABLE and now you can export
writable buffers as well.  Add Py_BUF_STRIDED and the strided circle is
opened to you, and so on.

My recommendation is, any flag should turn on some circle in the Venn
diagram (it could be a circle I didn't draw--shaped arrays, for
example--but it should be *some* circle).


>>> Py_BUF_FORMAT
>>>    The consumer will be using the format string information so make 
>>> sure that    member is filled correctly. 
>>
>> Is the idea to throw an exception if there's some other data format 
>> besides "b", and this flag isn't set?  It seems superfluous otherwise.
> 
> The idea is that a consumer may not care about the format and the 
> exporter may want to know that to simplify the interface.    In other 
> words the flag is a way for the consumer to communicate that it wants 
> format information (or not).

I'm -1 on using the flags for this.  It's completely out of character
compared to the rest of the flags.  All other flags are there for the
benefit of the consumer; this flag is useless to the consumer.

More concretely, all the rest of the flags are there to tell the 
exporter what kind of buffer they're prepared to accept.  This flag, 
alone, does not do that.

Even the benefits to the exporter are dubious.  This flag can't reduce 
code complexity, since all buffer objects have to be prepared to furnish 
type information.  At best, this flag is a rare optimization.  In fact, 
most buffers are going to point format to a constant string, regardless 
of whether this flag was passed or not:

bufinfo->format = "b";


> If the exporter wants to raise an exception if the format is not
> requested is up to the exporter.

That seems like a bad idea.  Suppose I have a contiguous numpy array of
floats and I want to view it as a sequence of bytes.  If the exporter's
allowed to raise an exception for this, any consumer that wanted a
data-neutral view of the data would still have to pass Py_BUF_FORMAT to
guard against this.  Wouldn't that be ironic?


>>> Py_BUF_SHAPE
>>>    The consumer can (and might) make use of using the ndims and shape 
>>> members of the structure
>>>    so make sure they are filled in correctly.    Py_BUF_STRIDES 
>>> (implies SHAPE)
>>>    The consumer can (and might) make use of the strides member of the 
>>> structure (as well
>>>    as ndims and shape)
>>
>> Is there any reasonable benefit for allowing Py_BUF_SHAPE without 
>> Py_BUF_STRIDES?  Would the array be C- or Fortran-like?
> 
> Yes,  I could see a consumer not being able to handle simple striding 
> but could handle shape information.  Many users of NumPy arrays like to 
> think of the array as an N-d array but want to ignore striding.

Ok, but is the indexing row-major or column-major?  That has to be decided.


Carl Banks