[Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

[Marten van Kerkwijk]

HI All,

On the backwards compatibility: from an astropy perspective, I would expect
that the introduction of `__array_function__` implies a guarantee that the
*functionality* it provides will remain, i.e., that it will continue to be
possible to override, say, concatenate. It is not a big deal if the actual
implementation changes (say, `__array_concatenate__` is introduced) and we
need to do some version-dependent magic to ensure that our users do not
notice the change; we did the same with `__array_ufunc__` - the
wrap/prepare machinery will be gone only in the next version of astropy,
when our minimum version for numpy reaches 1.13.

One thing perhaps worth remembering that what really matters is not so much
`__array_function__` but the set of functions that actually implement the
override. It would seem unlikely this would be the complete numpy API on
the first go; perhaps it is an idea to be somewhat specific about which
part we start with? My vote would go towards array manipulation routines
like concatenate. I would also suggest to avoid those functions for which
ufunc implementations would seem quite possible (i.e., avoid things like
median, etc.)

All the best,

Marten

On Wed, Aug 29, 2018 at 8:31 AM Matthew Rocklin <mrocklin at gmail.com> wrote:

> >> 1. if we do find ourselves in a situation where changing this would
> break lots of users, will we consider ourselves beholden to them?
>
> I think that it would be useful for Numpy's continued evolution to develop
> the ability to include code on a provisional basis.  Other projects do this
> and they just have big bold "Experimental" notes everywhere that a user
> might go to learn about the functionality (docs, docstrings, blogposts).
> Some users will definitely get burned, yes, but the alternative is to burn
> all other users a little bit by moving too slowly and not trying things out.
>
> This is different from how Numpy has operated in the past, but that might
> be ok.
>
> >> 2. is it plausible that we'll find ourselves in that situation?
>
> > Personally, for as long as this protocol is experimental, I’ll add a
> warning in the docs of sparse as well; saying this might disappear anytime
>
> Yup.  Same for Dask.  We're pretty accustomed to this.
>
> On Wed, Aug 29, 2018 at 7:01 AM Hameer Abbasi <einstein.edison at gmail.com>
> wrote:
>
>> > On 29. Aug 2018, at 11:44, Matti Picus <matti.picus at gmail.com> wrote:
>> >
>> > On 29/08/18 10:37, Nathaniel Smith wrote:
>> >> it's easy to imagine scenarios where the
>> >> people being broken aren't the ones who had a chance to read the docs
>> >> – e.g. if a major package starts relying on __array_function__, then
>> >> it's all*their*  users who we'd be breaking, even though they had
>> >> nothing to do with it.
>> > This is a packaging problem. This proposal is intended for use by other
>> "major packages", not so much for end-users. We would have much more
>> trouble if we were proposing a broad change to something like indexing or
>> the random number module (see those NEPs). If we break one of those major
>> packages, it is on them to pin the version of NumPy they can work with. In
>> my opinion very few end users will be implementing their own ndarray
>> classes with `__array_function__`. While we will get issue reports, we can
>> handle them much as we do the MKL or OpenBLAS ones - pinpoint the problem
>> and urge users to complain to those packages.
>>
>> One thing that might help here is nightly or continuous CI builds of the
>> NumPy wheels. This would be good, as we could test it in CI, and fix it
>> when it comes up. But I guess that’s another discussion.
>>
>> Personally, for as long as this protocol is experimental, I’ll add a
>> warning in the docs of sparse as well; saying this might disappear anytime.
>>
>> >
>> > Other than adding a warning, I am not sure what the concrete proposal
>> is here. To not accept the NEP?
>> > Matti
>> > _______________________________________________
>> > NumPy-Discussion mailing list
>> > NumPy-Discussion at python.org
>> > https://mail.python.org/mailman/listinfo/numpy-discussion
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180829/609fc462/attachment-0001.html>