[Numpy-discussion] Adding to the non-dispatched implementation of NumPy methods

[Ralf Gommers]

On Sat, Apr 27, 2019 at 7:05 PM Marten van Kerkwijk <
m.h.vankerkwijk at gmail.com> wrote:

> Hi All,
>
> I agree with Ralf that there are two discussions going on, but also with
> Hameer that they are related, in that part of the very purpose of
> __array_function__ was to gain freedom to experiment with implementations.
>

This may have been your motivation, but I don't see that in NEP 18
anywhere. On the contrary - the single sentence under "backwards
compatibility" in the NEP reads: "This proposal does not change existing
semantics, except for those arguments that currently have __array_function__
methods, which should be rare."

If I'm missing something that's actually in NEP 18, can you please point
out the actual text?

By the way, I suspect that Hameer may have meant replacing
__array_function__ completely (not sure though). Technically that may even
be a good idea, but despite the "experimental" label I'm pretty sure that
that ship has sailed given the review process we went through for NEP 18
and it starting to get uptake in other libraries.

And in particular the freedom to *assume* that inputs are arrays so that we
> can stop worrying about breaking subclasses and duck arrays: with
> __array_function__, we can now (well, eventually) tell people to just
> override the function and make it do what they want.
>

This is not in NEP 18, and against our backwards compatibility policy.

In that respect, if having `__numpy_implementation__` means it would not
> change the existing situation with backwards compatibility, then that is
> bad: we do want to change that! The point is to keep the exposed API
> stable, not the implementation.
>

API and semantics. The NEP really is very clear on this.


> Of course, the idea proposed here is not to keep the implementation
> stable, but rather to help people implement __array_function__ who are
> daunted by the many functions that can be overridden, especially for those
> classes for which many numpy functions work out of the box: they can have a
> default of just calling the old implementation.
>

This is not feasible. All the things you talk about are related to NEP 22
(duck typing for NumPy arrays). NEP 18 is about overrides. It's not about
skipping parts of the implementation of functions (like asarray).

For code like:
```
import numpy as np
import mylib

x = mylib.customarray([1, 2, 3])  # duck array or subclass
np.somefunc(x)
```
you simply cannot bypass asarray in the future if somefunc has asarray now.
To get somefunc-without-asarray, you need a new opt-in interface. Telling
our whole userbase to use __array_ufunc__ or __array_function__ to keep
their code working is just not a good idea.


> In the end, the proposed goal for __numpy_implementation__ really seems
> simple: to provide a better name for __wrapped__. But to me the discussion
> has proven that this is in fact not a good idea. It does suggest stability
> where there should be none, and the name itself is contentious. Maybe it is
> best to just stick with __wrapped__? If so, the the only change would be
> that we mention it in the NEP,
>

Sticking with __wrapped__ is fine with me too. It's a reasonable name.

Marten, I understand where you're coming from and why you want to bypass
array coercion, but this insistence on being able to break the world isn't
helpful. I like the goal and would like to see a design for how to achieve
it. It just needs that: a new design. It could be more protocols, or be
based on uarray (what Hameer proposes, probably the better option because
it's a coherent design that can do what you want), or yet something else.

It seems like we all have a different mental model of what NEP 18 actually
does. I'm going to try to put mine on a few slides with diagrams/examples
to see if that helps, since mailing list threads are hard to process.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20190427/ee1cd898/attachment.html>