[Numpy-discussion] Changing the size of PyArrayObject_fields (the ndarray c-struct)

[Charles R Harris]

On Sat, Nov 21, 2020 at 12:28 PM Matti Picus <matti.picus at gmail.com> wrote:

> PyArrayObject_fields is the c-struct that underlies ndarray. It is
> defined in ndarraytypes.h [0]. Since version 1.7, we have been trying to
> hide it from the public C-API so that we can freely modify it, the
> structure has the comment:
>
>
>   * It has been recommended to use the inline functions defined below
>   * (PyArray_DATA and friends) to access fields here for a number of
>   * releases. Direct access to the members themselves is deprecated.
>   * To ensure that your code does not use deprecated access,
>   * #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
>   * (or NPY_1_8_API_VERSION or higher as required).
>
>
>
> In order to clean up buffer exports, Sebastian suggested (and I pushed
> for supporting) PR 16938 [1] which would add a new field to the struct.
> As Eric pointed out on the pull request, this would change the size of
> the struct, meaning users of the struct (i.e., subclassing it in C)
> would have to be very careful interacting with NumPy-generated objects
> which may have changed sizes.
>
>
> Or we should give up and declare that we cannot change the size of the
> struct until we release a NumPy 2.0?
>
>
> Are there real-world cases that changing the size of the struct would
> break? I admit I have an agenda to further modify the struct in upcoming
> versions to better support things like alternative data memory allocator
> strategies, so in my opinion it would be a shame if we are stuck forever
> with the current struct.
>
>
> Matti
>
>
I think the risk is small and this will probably work, the potential
problem is if people have extended the essentially private structure in C
code. That said, it might be better to put it in after 1.20.x is branched
so that it is out there for others to test against, and perhaps implement
fixes if needed. We do need to make this move at some point and can't stay
fixed on an old structure forever, so the proposed change is something of a
warning shot.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/numpy-discussion/attachments/20201121/588f0f1e/attachment.html>