[Numpy-discussion] nditer as a context manager (reformatted?)

[Allan Haldane]

Given the lack of objections, we are probably going forward with this
change to nditer.

Anyone who uses nditers may have to update their code slightly if they
want to avoid deprecation warnings, but otherwise old nditer code should
work for a long time from now.

Allan

On 03/22/2018 01:43 PM, Matti Picus wrote:
> Hello all, PR #9998 (https://github.com/numpy/numpy/pull/9998/) proposes
> an update to the nditer API, both C and python. The issue
> (https://github.com/numpy/numpy/issues/9714) is that sometimes nditer
> uses temp arrays via the "writeback" mechanism, the data is copied back
> to the original arrays "when finished". However "when finished" was
> implemented using nditer deallocation.
> 
> This mechanism is implicit and unclear, and relies on refcount semantics
> which do not work on non-refcount python implementations like PyPY. It
> also leads to lines of code like "iter=None" to trigger the writeback
> resolution.
> 
> On the c-api level the agreed upon solution is to add a new
> `NpyIter_Close` function in C, this is to be called before
> `NpyIter_Dealloc`.
> 
> The reviewers and I would like to ask the wider NumPy community for
> opinions about the proposed python-level solution: turning the python
> nditer object into a context manager. This way "writeback" occurs at
> context manager exit via a call to `NpyIter_Close`, instead of like
> before when it occurred at `nditer` deallocation (which might not happen
> until much later in Pypy, and could be delayed by GC even in Cpython).
> 
> Another solution that was rejected
> (https://github.com/numpy/numpy/pull/10184) was to add an nditer.close()
> python-level function that would not require a context manager It was
> felt that this is more error-prone since it requires users to add the
> line for each iterator created.
> 
> The back-compat issues are that:
> 
> 1. We are adding a new function to the numpy API, `NpyIter_Close`
> (pretty harmless)
> 
> 2. We want people to update their C code using nditer, to call
> `NpyIter_Close` before  they call `NpyIter_Dealloc` and will start
> raising a deprecation warning if misuse is detected
> 
> 3. We want people to update their Python code to use the nditer object
> as a context manager, and will warn if they do not.
> 
> We tried to minimize back-compat issues, in the sense that old code
> (which didn't work in PyPy anyway) will still work, although it will now
> emit deprecation warnings. In the future we also plan to raise an error
> if an nditer is used in Python without a context manager (when it should
> have been). For C code, we plan to leave the deprecation warning in
> place probably forever, as we can only detect the deprecated behavior in
> the deallocator, where exceptions cannot be raised.
> 
> Anybody who uses nditers should take a look and please reply if it seems
> the change will be too painful.
> 
> For more details, please see the updated docs in that PR
> 
> Matti (and reviewers)
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion