[Numpy-discussion] NEP 31 — Context-local and global overrides of the NumPy API

[Ralf Gommers]

On Wed, Oct 9, 2019 at 6:00 PM Juan Nunez-Iglesias <jni at fastmail.com> wrote:

> Hi all, and thank you for all your hard work with this.
>
> I wanted to provide more of an "end user" perspective than I think has
> been present in this discussion so far. Over the past month, I've quickly
> skimmed some emails on this thread and skipped others altogether. I am far
> from a NumPy novice, but essentially *all* of the discussion went over my
> head. For a while my attitude was "Oh well, far smarter people than me are
> dealing with this, I'll let them figure it out." Looking at the
> participants in the thread, I worry that this is the attitude almost
> everyone has taken, and that the solution proposed will not be easy enough
> to deal with for any meaningful adoption. Certainly with
> `__array_function__` I only took interest when our tests broke with 1.17rc1.
>
> Today I was particularly interested because I'm working to improve
> scikit-image support for pyopencl.Array inputs. I went back and read the
> original NEP and the latest iteration. Thank you again for the discussion,
> because the latest is indeed a vast improvement over the original.
>
> I think the very motivation has the wrong focus. I would summarise it as
> "we've been coming up with all kinds of ways to do multiple dispatch for
> array-likes, and we've found that we need more ways, so let's come up with
> the One True Way." I think the focus should be on the users and community.
> Something along the lines of: "New implementations of array computing are
> cropping up left, right, and centre in Python (not to speak of other
> languages!). There are good reasons for this (GPUs, distributed computing,
> sparse data, etc), but it leaves users and library authors in a pickle: how
> can they ensure that their functions, written with NumPy array inputs and
> outputs in mind, work well in this ecosystem?"
>
> With this new motivation in mind, I think that the user story below is (a)
> the best part of the NEP, but (b) underdeveloped. The NEP is all about "if
> I want my array implementation to work with this fancy dispatch system,
> what do I need to do?". But there should be more of "in user situations X,
> Y, and Z, what is the desired behaviour?"
>
> The way we propose the overrides will be used by end users is::
>
>     # On the library side
>
>     import numpy.overridable as unp
>
>     def library_function(array):
>         array = unp.asarray(array)
>         # Code using unumpy as usual
>         return array
>
>     # On the user side:
>
>     import numpy.overridable as unp
>     import uarray as ua
>     import dask.array as da
>
>     ua.register_backend(da)
>
>     library_function(dask_array)  # works and returns dask_array
>
>     with unp.set_backend(da):
>         library_function([1, 2, 3, 4])  # actually returns a Dask array.
>
> Here, ``backend`` can be any compatible object defined either by NumPy or
> an
> external library, such as Dask or CuPy. Ideally, it should be the module
> ``dask.array`` or ``cupy`` itself.
>
>
> Some questions about the above:
>
> - What happens if  I call `library_function(dask_array)` without
> registering `da` as a backend first? Will `unp.asarray` try to instantiate
> a potentially 100GB array? This seems bad.
> - To get `library_function`, I presumably have to do `from
> fancy_array_library import library_function`. Can the code in
> `fancy_array_library` itself register backends, and if so, should/would
> fancy array libraries that want to maximise compatibility pre-register a
> bunch of backends so that users don't have to?
>
> Here are a couple of code snippets that I would *want* to "just work".
> Maybe it's unreasonable, but imho the NEP should provide these as use cases
> (specifically: how library_function should be written so that they work,
> and what dask.array and pytorch would need to do so that they work, OR, why
> the NEP doesn't solve them).
>
> 1.
> from dask import array as da
> from fancy_array_library import library_function  # hopefully skimage one
> day ;)
>
> data = da.from_zarr('myfile.zarr')
> result = library_function(data)  # result should still be dask, all things
> being equal
> result.to_zarr('output.zarr')
>
> 2.
> from dask import array as da
> from magic_library import pytorch_predict
>
> data = da.from_zarr('myfile.zarr')
> result = pytorch_predict(data)  # normally here I would use e.g.
> data.map_overlap, but could this be done magically?
> result.to_zarr('output.zarr')
>
>
> There's probably a whole bunch of other "user stories" one can concoct,
> and no doubt many from the authors of the NEP themselves, but they don't
> come through in the NEP text. My apologies that I haven't read *all* the
> references: I understand that it is frustrating if the above are addressed
> there, but I think it's important to have this kind of context in the NEP
> itself.
>
> Thank you again, and I hope the above is helpful rather than feels like
> more unnecessary churn.
>

Thanks Juan, this feedback is amazing and I couldn't agree more. I think we
have to have this "end user focus" for this NEP, as well as for other
large-scope design efforts: we should do or have done this for
__array_ufunc__, __array_function__, the dtype redesign, etc.

I think in this case, the user stories and a "vision" on the whole topic
don't belong inside this NEP. Rather, it should be a separate one like
https://numpy.org/neps/nep-0022-ndarray-duck-typing-overview.html. If you
read the first paragraph of that NEP, it actually starts out exactly right
in the detailed description. But then it dives straight into design.

In my experience, when you mix user stories or external requirements with
design, it's extremely easy to ignore or be super brief about the former,
and let design considerations/details lead rather than follow from those
external requirements.

Note that this is also why I wanted to update the NEP template. We've done
a tweak by adding the "Motivation and Scope" section, but that doesn't go
nearly far enough.

Back to this NEP: I don't think we should significantly extend it, we
should write a new separate one. Rationale: these user stories apply
equally to __array_function__ et al., and will have to guide how NumPy as a
whole and the usage of the NumPy API evolves over the next couple of years.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20191009/3973b858/attachment.html>