[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

[Stefan Krah]

Stefan Krah <stefan-usenet at bytereef.org> added the comment:

Antoine Pitrou <report at bugs.python.org> wrote:
> > I thought the whole Py_buffer API was only temporarily removed from the
> > limited API until the issues were sorted out:
> > 
> > http://bugs.python.org/issue10181#msg125462
> 
> I'm talking about the memoryview access macros. It's like
> PyList_GET_SIZE, it isn't part of the limited API and you have to use
> PyList_GetSize instead.

You're right: I presumed that the macros were excluded temporarily
when in fact that had already happened in 62b61abd02b8.

> > The flags are primarily for the memoryview itself to avoid repeated calls
> > to PyBuffer_IsContiguous(). So when 3rd-party uses PyMemoryView_GET_BUFFER
> > to get the view and also needs to determine the contiguity type, that
> > code could also use the flags.
> 
> But why would 3rd-party code use PyMemoryView_GET_BUFFER instead of, for
> example, PyObject_GetBuffer? You seldom have APIs which *expect* a
> memoryview, I think. Instead, they would expect a buffer-compatible
> object.

That's a good question. It looks to me that the macro was present as
PyMemoryView() initially. You renamed it in #3560 (with Guido's approval),
and later PyMemoryView_GET_BUFFER appeared in the docs.

I think 3rd-party code uses the macros mainly because they are
present and, in the case of PyMemoryView_GET_BUFFER, documented.
In most situations PyObject_GetBuffer() could be used indeed.

Most use cases I can think of would also involve having access to
the managed buffer API. As an example, here's a technique that is
similar to what goes on in PyMemoryView_GetContiguous():

Suppose you have an initialized bytes object that you want to
wrap as a multi-dimensional exporter. Then:

   - Base the memoryview on the bytes object and keep exactly one
     reference to it.

   - Adjust the shape, strides etc. to get the structure you want.

   - Return the view: You now have a fully compliant exporter that
     only needs a single Py_DECREF() to disappear and do all cleanup.

Of course this could also be exposed as a function, e.g.:

   /* stealing a reference to bytes */
   PyMemoryView_FromBytesAndInfo(PyObject *bytes, Py_buffer *info);

So let's make the flags private. What do you prefer?

  1) Leave them in memoryview.h, but with a leading underscore:

       _Py_MEMORYVIEW_C
       [...]

  2) Move them to memoryobject.c, with a leading underscore.

  3) Move them to memoryobject.c, without a leading underscore (I find
     this more readable).

  4) Move them to memoryobject.c as MV_C, MV_FORTRAN, etc.

Also, I'll add a note to the docs that PyMemoryView_GET_BUFFER can
probably be avoided by using PyObject_GetBuffer() directly.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________