[Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

[Matthew Rocklin]

> On the backwards compatibility: from an astropy perspective, I would
expect that the introduction of `__array_function__` implies a guarantee
that the *functionality* it provides will remain,

My guess is that you wouldn't have this expectation if Numpy released this
feature with explicit "Experimental" warnings  attached to it.   In that
case astropy might just wait before adopting it until that label was
dropped.  Projects that were more risk-friendly would then take on the
burden of trying it out, working out some kinks, and then hopefully in a
version or two the "Experimental" label would be dropped and astropy would
step in and adopt it more safely.

This would give the ecosystem an opportunity to try something and modify it
after having some experience without promising to support it in a fully
backwards compatible way forever.

On Wed, Aug 29, 2018 at 9:46 AM Marten van Kerkwijk <
m.h.vankerkwijk at gmail.com> wrote:

> HI All,
>
> On the backwards compatibility: from an astropy perspective, I would
> expect that the introduction of `__array_function__` implies a guarantee
> that the *functionality* it provides will remain, i.e., that it will
> continue to be possible to override, say, concatenate. It is not a big deal
> if the actual implementation changes (say, `__array_concatenate__` is
> introduced) and we need to do some version-dependent magic to ensure that
> our users do not notice the change; we did the same with `__array_ufunc__`
> - the wrap/prepare machinery will be gone only in the next version of
> astropy, when our minimum version for numpy reaches 1.13.
>
> One thing perhaps worth remembering that what really matters is not so
> much `__array_function__` but the set of functions that actually implement
> the override. It would seem unlikely this would be the complete numpy API
> on the first go; perhaps it is an idea to be somewhat specific about which
> part we start with? My vote would go towards array manipulation routines
> like concatenate. I would also suggest to avoid those functions for which
> ufunc implementations would seem quite possible (i.e., avoid things like
> median, etc.)
>
> All the best,
>
> Marten
>
> On Wed, Aug 29, 2018 at 8:31 AM Matthew Rocklin <mrocklin at gmail.com>
> wrote:
>
>> >> 1. if we do find ourselves in a situation where changing this would
>> break lots of users, will we consider ourselves beholden to them?
>>
>> I think that it would be useful for Numpy's continued evolution to
>> develop the ability to include code on a provisional basis.  Other projects
>> do this and they just have big bold "Experimental" notes everywhere that a
>> user might go to learn about the functionality (docs, docstrings,
>> blogposts).  Some users will definitely get burned, yes, but the
>> alternative is to burn all other users a little bit by moving too slowly
>> and not trying things out.
>>
>> This is different from how Numpy has operated in the past, but that might
>> be ok.
>>
>> >> 2. is it plausible that we'll find ourselves in that situation?
>>
>> > Personally, for as long as this protocol is experimental, I’ll add a
>> warning in the docs of sparse as well; saying this might disappear anytime
>>
>> Yup.  Same for Dask.  We're pretty accustomed to this.
>>
>> On Wed, Aug 29, 2018 at 7:01 AM Hameer Abbasi <einstein.edison at gmail.com>
>> wrote:
>>
>>> > On 29. Aug 2018, at 11:44, Matti Picus <matti.picus at gmail.com> wrote:
>>> >
>>> > On 29/08/18 10:37, Nathaniel Smith wrote:
>>> >> it's easy to imagine scenarios where the
>>> >> people being broken aren't the ones who had a chance to read the docs
>>> >> – e.g. if a major package starts relying on __array_function__, then
>>> >> it's all*their*  users who we'd be breaking, even though they had
>>> >> nothing to do with it.
>>> > This is a packaging problem. This proposal is intended for use by
>>> other "major packages", not so much for end-users. We would have much more
>>> trouble if we were proposing a broad change to something like indexing or
>>> the random number module (see those NEPs). If we break one of those major
>>> packages, it is on them to pin the version of NumPy they can work with. In
>>> my opinion very few end users will be implementing their own ndarray
>>> classes with `__array_function__`. While we will get issue reports, we can
>>> handle them much as we do the MKL or OpenBLAS ones - pinpoint the problem
>>> and urge users to complain to those packages.
>>>
>>> One thing that might help here is nightly or continuous CI builds of the
>>> NumPy wheels. This would be good, as we could test it in CI, and fix it
>>> when it comes up. But I guess that’s another discussion.
>>>
>>> Personally, for as long as this protocol is experimental, I’ll add a
>>> warning in the docs of sparse as well; saying this might disappear anytime.
>>>
>>> >
>>> > Other than adding a warning, I am not sure what the concrete proposal
>>> is here. To not accept the NEP?
>>> > Matti
>>> > _______________________________________________
>>> > NumPy-Discussion mailing list
>>> > NumPy-Discussion at python.org
>>> > https://mail.python.org/mailman/listinfo/numpy-discussion
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at python.org
>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180829/8725d78c/attachment.html>