[Numpy-discussion] Changing the size of PyArrayObject_fields (the ndarray c-struct)

[Matti Picus]

PyArrayObject_fields is the c-struct that underlies ndarray. It is 
defined in ndarraytypes.h [0]. Since version 1.7, we have been trying to 
hide it from the public C-API so that we can freely modify it, the 
structure has the comment:


  * It has been recommended to use the inline functions defined below
  * (PyArray_DATA and friends) to access fields here for a number of
  * releases. Direct access to the members themselves is deprecated.
  * To ensure that your code does not use deprecated access,
  * #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
  * (or NPY_1_8_API_VERSION or higher as required).



In order to clean up buffer exports, Sebastian suggested (and I pushed 
for supporting) PR 16938 [1] which would add a new field to the struct. 
As Eric pointed out on the pull request, this would change the size of 
the struct, meaning users of the struct (i.e., subclassing it in C) 
would have to be very careful interacting with NumPy-generated objects 
which may have changed sizes.


Or we should give up and declare that we cannot change the size of the 
struct until we release a NumPy 2.0?


Are there real-world cases that changing the size of the struct would 
break? I admit I have an agenda to further modify the struct in upcoming 
versions to better support things like alternative data memory allocator 
strategies, so in my opinion it would be a shame if we are stuck forever 
with the current struct.


Matti


[0] 
https://github.com/numpy/numpy/blob/v1.19.4/numpy/core/include/numpy/ndarraytypes.h#L659

[1] https://github.com/numpy/numpy/pull/16938