[Numpy-discussion] NEP 37: A dispatch protocol for NumPy-like modules

[Ralf Gommers]

On Thu, Apr 9, 2020 at 6:54 PM Sebastian Berg <sebastian at sipsolutions.net>
wrote:

> On Thu, 2020-04-09 at 13:52 +0200, Ralf Gommers wrote:
> > On Wed, Mar 4, 2020 at 1:22 AM Sebastian Berg <
> > sebastian at sipsolutions.net>
> > wrote:
> >
> > The NumPy version should normally be older than other libraries, and
> > if
> > > NumPy updates the API so do the downstream implementers.
> > > E.g. dask may have to provide multiple version of the same function
> > > depending on the installed NumPy version, but that seems OK to me?
> >
> > That seems unworkable, and I don't think any libraries do this.
> > Coupling
> > the semantics of a single Dask function to the installed numpy
> > version is
> > odd.
>
> Is it all that odd? Libraries (not array providers) already need to
> test for NumPy version occasionally due to API changes, so they also
> have two versions of the same thing around (e.g. a fallback).
>

That's completely different, it's internal to a library and not visible to
end users via different signatures/behavior.

This simply would move the burden to the array-object implementer to
> some degree. Assume that we have a versioned API in some form or
> another, it seems to me we either require:
>
>    module = np.get_array_module(..., api_version=2)
>

Yes, this is the version I was thinking about.


> or define `module.__api_version__`.
>
> Where the latter means that sklearn/SciPy may have to check
> `__api_version__` on every function call, while currently such checks
> usually happen at import time. On the other hand, the former means that
> sklearn/scipy can only opt-in to new API after 3+ years easily?
>

That's anyway the case, has very little to do with API versioning I think -
it's simply determined by minimum NumPy version supported.


> Saying that the NumPy version is what pins the api-version, is not much
> more than assuming/requiring that NumPy will be the least up-to-date
> package?
>
> Of course it is unworkable to get 100% right in practice but are you
> saying that because it seems like an impractical approach,


Yes this, impractical and undesired.

or because
> the API surface is currently so large that, of course, we will never
> get it 100% right (but that is generally true, nobody will be able to
> implement NumPy 100% compatible)?
>

That's true too, we *don't want* anyone to start adding compat features for
outdated or "wish we could deprecate" NumPy features.


> `__array_function__` has same issue? If we change our API, Dask has to
> catch up.


Yes, that's true. The restricted API should be more stable than the whole
NumPy API, otherwise no one will be able to be fully compatible.

If SciPy expects it to be the old version though (based on
> the NumPy import) it will incorrectly assume the old-api will be used.
>

That's not incorrect unless it's a backwards-incompatible change, which
should be rare.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20200410/e3d78eb0/attachment.html>