[Numpy-discussion] PyBUF_SIMPLE/PyBUF_FORMAT: casts to unsigned bytes

[Stefan Krah]

Hello,

PEP-3118 presumably intended that a PyBUF_SIMPLE request should cast the
original buffer's data type to 'B' (unsigned bytes). Here is a one-dimensional
example that currently occurs in Lib/test/test_multiprocessing:

>>> import array, io
>>> a = array.array('i', [1,2,3,4,5])
>>> m = memoryview(a)
>>> m.format
'i'
>>> buf = io.BytesIO(bytearray(5*8))
>>> buf.readinto(m)

buf.readinto() calls PyObject_AsWriteBuffer(), which requests a simple buffer
from the memoryview, thus casting the 'i' data type to the implied type 'B'.

The consumer can see that a cast has occurred because the new buffer's
format field is NULL.


This seems fine for the one-dimensional case. Numpy currently also allows
such casts for multidimensional contiguous and non-contiguous arrays.
See below for the examples; I don't want to distract from the main
point of the post, which is this:



I'm seeking a clear specification for the Python documentation that determines
under what circumstances casts to 'B' should succeed. I'll formulate the points
as statements for clarity, but in fact they are also questions:

1) An exporter of a C-contiguous array with ndim <= 1 MUST honor
   a PyBUF_SIMPLE request, setting format, shape and strides to NULL
   and itemsize to 1.

   As a corner case, an array with ndim = 0, format = "L" (or other)
   would also morph into a buffer of unsigned bytes. test_ctypes
   currently makes use of this.

2) An exporter of a C-contiguous buffer with ndim > 1 MUST honor
   a PyBUF_SIMPLE request, setting format, shape, and strides to NULL
   and itemsize to 1.

3) An exporter of a buffer that is not C-contiguous MUST raise BufferError
   in response to a PyBUF_SIMPLE request.


Why am I looking for such rigid rules? The problem with memoryview is
that it has to act as a re-exporter itself.

For several reasons (performance of chained memoryviews, garbage collection,
early release, etc.) it has been decided that the new memoryview object has
a managed buffer that takes a snapshot of the original exporter's buffer
(See: http://bugs.python.org/issue10181).

Now, since getbuffer requests to the memoryview object cannot be redirected
to the original object, strict rules are needed for memory_getbuf().



Could you agree with these rules? Point 2) isn't clear from the PEP itself.
I assumed it because Numpy currently allows it, and it appears harmless.


Stefan Krah


Examples:
=========

Cast a multidimensional contiguous array:
-----------------------------------------

I think itemsize in the result should be 1.

[_testbuffer.ndarray is from http://hg.python.org/features/pep-3118#memoryview]

>>> from _testbuffer import *
>>> from numpy import *
>>> from _testbuffer import ndarray as pyarray
>>>
>>> exporter = ndarray(shape=[3,4], dtype="L")
# Issue a PyBUF_SIMPLE request to 'exporter' and act as a re-exporter:
>>> x = pyarray(exporter, getbuf=PyBUF_SIMPLE)
>>> x.len
96
>>> x.shape
()
>>> x.strides
()
>>> x.format
''
>>> x.itemsize # I think this should be 1, not 8.
8

Cast a multidimensional non-contiguous array:
---------------------------------------------

This is clearly not right, since y.buf points to a location that the consumer
cannot handle without shape and strides.

>>> nd = ndarray(buffer=bytearray(96), shape=[3,4], dtype="L")
[182658 refs]
>>> exporter = nd[::-1, ::-2]
[182661 refs]
>>> exporter
array([[0, 0],
       [0, 0],
       [0, 0]], dtype=uint64)
[182659 refs]
>>> y = pyarray(exporter, getbuf=PyBUF_SIMPLE)
[182665 refs]
>>> y.len
48
[182666 refs]
>>> y.strides
()
[182666 refs]
>>> y.shape
()
[182666 refs]
>>> y.format
''
[182666 refs]
>>> y.itemsize
8
[182666 refs]