[New-bugs-announce] [issue45459] Limited API support for Py_buffer

[Christian Heimes]

New submission from Christian Heimes <lists at cheimes.de>:

Currently all APIs related to Py_buffer are excluded from the limited API. It's neither possible to use Py_buffer from a C extension with limited API nor is it possible to define heap types with buffer support using the stable ABI.

The lack of Py_buffer support prevents prominent projects like NumPy or Pillow to use the limited API and produce abi3 binary wheel. To be fair it's not the only reason why these projects have not adopted the stable abi3 yet. Still Py_buffer is a necessary but not sufficient condition. Stable abi3 support would enable NumPy stack to build binary wheels that work with any Python version >= 3.11, < 4.0.

The limited API excludes any C API that references Py_buffer:

- 8 PyBuffer_*() functions
- 21 PyBUF_* constants
- PyMemoryView_FromBuffer()
- PyObject_GetBuffer
- Py_bf_getbuffer / Py_bf_releasebuffer type slots for PyBufferProcs

It should not be terribly complicated to add Py_buffer to the stable API. All it takes are an opaque struct definition of Py_buffer, an allocate function, a free function and a bunch of getters and setters. The hard part is to figure out which getters and setters are needed and how many struct members must be exposed by getters and setters. I recommend to get feedback from NumPy, Pillow, and Cython devs first.


Prototype
---------

typedef struct bufferinfo Py_buffer;

/* allocate a new Py_buffer object on the heap and initialize all members to NULL / 0 */
Py_buffer*
PyBuffer_New()
{
    Py_buffer *view = PyMem_Calloc(1, sizeof(Py_buffer));
    if (view == NULL) {
        PyErr_NoMemory();
    }
    return view;
}

/* convenience function */
Py_buffer*
PyBuffer_NewEx(PyObject *obj, void *buf,  Py_ssize_t len, Py_ssize_t itemsize,
               int readonly, int ndim, char *format, Py_ssize_t *shape, Py_ssize_t *strides,
               Py_ssize_t *suboffsets, void *internal)
{
    ...
}

/* release and free buffer */
void
PyBuffer_Free(Py_buffer *view)
{
    if (view != NULL) {
        PyBuffer_Release(view);
        PyMem_Free(view);
    }
}

----------
components: C API
messages: 403813
nosy: christian.heimes, petr.viktorin, vstinner
priority: normal
severity: normal
status: open
title: Limited API support for Py_buffer
type: enhancement
versions: Python 3.11

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue45459>
_______________________________________