[Python-Dev] Understanding the buffer API

[Jeff Allen]

Thanks for a swift reply: you're just the person I hoped would do so.

On 04/08/2012 10:11, Stefan Krah wrote:
> You are right that the PEP does not explicitly state that rule for
> strides. However, NULL always has an implied meaning:
>
>    format=NULL  ->   treat the buffer as unsigned bytes.
>
>    shape=NULL   ->   one-dimensional AND treat the buffer as unsigned bytes.
>
>    strides=NULL ->   C-contiguous
>
> I think relaxing the NULL rule for strides would complicate things,
> since it would introduce yet another special case.
... Ok, I think I see that how the absence of certain arrays is used to 
deduce structural simplicity, over and above their straightforward use 
in navigating the data. So although no shape array is (sort of) 
equivalent to ndim==1, shape[0]==len, it also means I can call simpler 
code instead of using the arrays for navigation.

I still don't see why, if the consumer says "I'm assuming 1-D unsigned 
bytes", and that's what the data is, memoryview_getbuf could not provide 
a shape and strides that agree with the data. Is the catch perhaps that 
there is code (in abstract.c etc.) that does not know what the consumer 
promised not to use/look at? Would it actually break, e.g. not treat it 
as bytes, or just be inefficient?

> Because of all the implied meanings of NULL, I think the safest way is
> to implement memoryview_getbuf() for Jython. After all the PEP describes
> a protocol, so everyone should really be doing the same thing.
I'll look carefully at what you've written (snipped here) because it is 
these "consumer expectations" that are most important. The Jython buffer 
API is necessarily a lot different from the C one: some things are not 
possible in Java (pointer arithmetic) and some are just un-Javan 
activities (allocate a struct and have the library fill it in). I'm only 
going for a logical conformance to the PEP: the same navigational and 
other attributes, that mean the same things for the consumer.

When you say such-and-such is disallowed, but the PEP or the data 
structures seem to provide for it, you mean memoryview_getbuf() 
disallows it, since you've concluded it is not sensible?
> I think the protocol would benefit from changing the getbuffer rules to:
>
>     a) The buffer gets a 'flags' field that can store properties like
>        PyBUF_SIMPLE, PyBUF_C_CONTIGUOUS etc.
>
>     b) The exporter must *always* provide full information.
>
>     c) If a buffer can be exported as unsigned bytes but has a different
>        layout, the exporter must perform a full cast so that the above
>        mentioned invariants are kept.
>
Just like PyManagedBuffer mbuf and its sister view in memoryview? I've 
thought the same things, but the tricky part is to do it compatibly.

a) I think I can achieve this. As I have interfaces and polymorphism on 
my side, and a commitment only to logical equivalence to CPython, I can 
have the preserved flags stashed away inside to affect behaviour. But 
it's not as simple as saving the consumer's request, and I'm still 
trying to work it out what to do, e.g. when the consumer didn't ask for 
C-contiguity, but in this case it happens to be true.

In the same way, functions you have in abstract.c etc. can be methods 
that, rather than work out by inspection of a struct how to navigate the 
data on this call, already know what kind of buffer they are in. So 
SimpleBuffer.isContiguous(char order) can simply return true.

b) What I'm hoping can work, but maybe not.

c) Java will not of course give you raw memory it thinks is one thing, 
to treat as another, so this aspect is immature in my thinking. I got as 
far as accommodating multi-byte items, but have no use for them as yet.

Thanks again for the chance to test my ideas.
Jeff Allen