[Numpy-discussion] Keep __array_function__ unexposed by default for 1.17?

[Stephan Hoyer]

On Wed, May 22, 2019 at 2:36 PM Ralf Gommers <ralf.gommers at gmail.com> wrote:

> I would still like to turn on __array_function__ in NumPy 1.17. At least,
>>> let's try that for the release candidate and see how it goes.
>>>
>>
> I agree. I'd actually suggest flipping the switch asap and see if it
> causes any issues for projects that test against numpy master in their CI,
> and the people that like to live on the bleeding edge by installing master
> into their environment.
>

The switch actually has already been done on master for several months now,
until for a period in the 1.16 release cycle before we added the off
switch. Doing so did turn up a few bugs, e.g.,
https://github.com/numpy/numpy/issues/12263

We will actually need to re-add in the code that does the environment
variable to allow for turning it off, but this isn't a big deal. My main
concern is that this adds some complexity for third-party projects in
detecting whether __array_function__ is enabled or not. They can't just use
the NumPy version and will need to check the environment variable as well,
or actually try using it on an example object.

If we want to keep an "off" switch we might want to add some sort of API
for exposing whether NumPy is using __array_function__ or not. Maybe
numpy.__experimental_array_function_enabled__ = True, so you can just test
`hasattr(numpy, '__experimental_array_function_enabled__')`? This is
assuming that we are OK with adding an underscore attribute to NumPy's
namespace semi-indefinitely.


>
> Cheers,
> Ralf
>
>
> The "all in" nature of __array_function__ without __skip_array_function__
>>> will both limit its use to cases where it is strongly motivated, and also
>>> limits the API implications for NumPy. There is still plenty of room for
>>> expanding the protocol, but it's really hard to see what is necessary (and
>>> prudent!) without actual use.
>>>
>>> [1] e.g., see
>>> https://github.com/google/jax/blob/62473351643cecb6c248a50601af163646ba7be6/jax/numpy/lax_numpy.py#L2440-L2459
>>> [2] https://github.com/numpy/numpy/pull/13305
>>>
>>>
>>>
>>>
>>> On Tue, May 21, 2019 at 11:44 PM Juan Nunez-Iglesias <jni.soma at gmail.com>
>>> wrote:
>>>
>>>> I just want to express my general support for Marten's concerns. As an
>>>> "interested observer", I've been meaning to give `__array_function__` a try
>>>> but haven't had the chance yet. So from my anecdotal experience I expect
>>>> that more people need to play with this before setting the API in stone.
>>>>
>>>> At scikit-image we place a very strong emphasis on code simplicity and
>>>> readability, so I also share Marten's concerns about code getting too
>>>> complex. My impression reading the NEP was "whoa, this is hard, I'm glad
>>>> smarter people than me are working on this, I'm sure it'll get simpler in
>>>> time". But I haven't seen the simplicity materialise...
>>>>
>>>> On Wed, 22 May 2019, at 11:31 AM, Marten van Kerkwijk wrote:
>>>>
>>>> Hi All,
>>>>
>>>> For 1.17, there has been a big effort, especially by Stephan, to make
>>>> __array_function__ sufficiently usable that it can be exposed. I think this
>>>> is great, and still like the idea very much, but its impact on the numpy
>>>> code base has gotten so big in the most recent PR (gh-13585) that I wonder
>>>> if we shouldn't reconsider the approach, and at least for 1.17 stick with
>>>> the status quo. Since that seems to be a bigger question than can be
>>>> usefully addressed in the PR, I thought I would raise it here.
>>>>
>>>> Specifically, now not only does every numpy function have its
>>>> dispatcher function, but also internally all numpy function calls are being
>>>> done via the new `__skip_array_function__` attribute, to avoid further
>>>> overrides. I think both changes make the code significantly less readable,
>>>> thus, e.g., making it even harder than it is already to attract new
>>>> contributors.
>>>>
>>>> I think with this it is probably time to step back and check whether
>>>> the implementation is in fact the right one. For instance, among the
>>>> alternatives we originally considered was one that had the overridable
>>>> versions of functions in the regular `numpy` namespace, and the once that
>>>> would not themselves check in a different one. Alternatively, for some of
>>>> the benefits provided by `__skip_array_function__`, there was a different
>>>> suggestion to have a special return value, of `NotImplementedButCoercible`.
>>>> Might these be better after all?
>>>>
>>>> More generally, I think we're suffering from the fact that several of
>>>> us seem to have rather different final goals in mind  In particular, I'd
>>>> like to move to a state where as much of the code as possible makes use of
>>>> the simplest possible implementation, with only a few true base functions,
>>>> so that all but those simplest functions will generally work on any type of
>>>> array. Others, however, worry much more about making implementations (even
>>>> more) part of the API.
>>>>
>>>> All the best,
>>>>
>>>> Marten
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at python.org
>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>
>>>>
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at python.org
>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at python.org
>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20190522/d3692bea/attachment.html>