[Numpy-discussion] Add guaranteed no-copy to array creation and reshape?

[Sebastian Berg]

On Wed, 2018-12-26 at 16:40 -0800, Ralf Gommers wrote:
> 
> 
> On Wed, Dec 26, 2018 at 3:29 PM Sebastian Berg <
> sebastian at sipsolutions.net> wrote:
> > Hi all,
> > 
> > In https://github.com/numpy/numpy/pull/11897 I am looking into the
> > addition of a `copy=np.never_copy` argument to:
> >   * np.array
> >   * arr.reshape/np.reshape
> >   * arr.astype
> > 
> > Which would cause an error to be raised when numpy cannot guarantee
> > that the returned array is a view of the input array.
> > The motivation is to easier avoid accidental copies of large data,
> > or
> > ensure that in-place manipulation will be meaningful.
> > 
> > The copy flag API would be:
> >   * `copy=True` forces a copy
> >   * `copy=False` allows numpy to copy if necessary
> >   * `copy=np.never_copy` will error if a copy would be necessary
> >   * (almost) all other input will be deprecated.
> > 
> > Unfortunately using `copy="never"` is tricky, because currently
> > `np.array(..., copy="never")` behaves exactly the same as
> > `np.array(..., copy=bool("never"))`. So that the wrong result would
> > be
> > given on old numpy versions and it would be a behaviour change. 
> 
> I think np.never_copy is really ugly. I'd much rather simply use
> 'never', and clearly document that if users start using this and they
> critically rely on it really being never, then they should ensure
> that their code is only used with numpy >= 1.17.0.
> 
> Note also that this would not be a backwards compatibility break,
> because `copy` is now clearly documented as bool, and not bool_like
> or some such thing. So we do not need to worry about the very
> improbable case that users now are using `copy='never'`.
> 

I agree that it is much nicer to pass "never" and that I am not too
worried about people actually passing in strings. But, I am a bit
worried of new code silently doing the exact opposite when run on an
older version. Although, I guess we typically do not worry too much
about such compatibilities and it is possible to make a note of it in
the docs.

- Sebastian

> If others think `copy='never'` isn't acceptable now, there are two
> other options:
> 1. add code first to catch `copy='never'` in 1.17.x and raise on it,
> then in a later numpy version introduce it.
> 2. just do nothing. I'd prefer that over `np.never_copy`.
> 
> Cheers,
> Ralf
> 
> > Some things that are a not so nice maybe:
> >  * adding/using `np.never_copy` is not very nice
> >  * Scalars need a copy and so will not be allowed
> >  * For rare array-likes numpy may not be able to guarantee no-copy,
> >    although it could happen (but should not).
> > 
> > 
> > The history is that a long while ago I considered adding a copy
> > flag to
> > `reshape` so that it is possible to do `copy=np.never_copy` (or
> > similar) to ensure that no copy is made. In these, you may want
> > something like an assertion:
> > 
> > ```
> > new_arr = arr.reshape(new_shape)
> > assert np.may_share_memory(arr, new_arr)
> > 
> > # Which is sometimes -- but should not be -- written as:
> > arr.shape = new_shape  # unnecessary container modification
> > 
> > # Or:
> > view = np.array(arr, order="F")
> > assert np.may_share_memory(arr, new_arr)
> > ```
> > 
> > but is more readable and will not cause an intermediate copy on
> > error.
> > 
> > 
> > So what do you think? Other variants would be to not expose this
> > for
> > `np.array` and probably limit `copy="never"` to the reshape method.
> > Or
> > just to not do it at all. Or to also accept "never" for `reshape`,
> > although I think I would prefer to keep it in sync and wait for a
> > few
> > years to consider that.
> > 
> > Best,
> > 
> > Sebastian
> > 
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at python.org
> > https://mail.python.org/mailman/listinfo/numpy-discussion
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20181227/143a7da1/attachment-0001.sig>