[SciPy-Dev] Backend support for fftpack - GSoC 2019

[Peter Bell]

I’ve had another look at the uarray docs and was quite surprised to see that the API has changed significantly since I last looked at them. It seems that uarray is now based on a __ua_function__ protocol which is similar to NEP 18 except that it’s held in the backend rather than the array type.  I think I’ve tracked the change to PR #152<https://github.com/Quansight-Labs/uarray/pull/152/> which is dated after the API was claimed to be stable.

Hameer, would you be able to clarify what state uarray development is in: exactly how stable is the API?

Peter


From: SciPy-Dev <scipy-dev-bounces+peterbell10=live.co.uk at python.org> On Behalf Of Hameer Abbasi
Sent: 09 May 2019 13:39
To: SciPy Developers List <scipy-dev at python.org>
Subject: Re: [SciPy-Dev] Backend support for fftpack - GSoC 2019

Hi,

Since uarray came up: I’m the author. Yes, performance is slow (since it’s pure python) but the API is stable at this point (the docs are a bit out of date, I should update them).

We’re welcoming any kind of contributions as I, myself, am not familiar enough with the CPython API to do it. However, I’d love to learn!

It does have a focus on array computing but it is meant to be fairly general, and you shouldn’t have any problems with the interface.

Best Regards,
Hameer Abbasi

On Thursday, May 09, 2019 at 8:25 AM, Peter Bell <PeterBell10 at live.co.uk<mailto:PeterBell10 at live.co.uk>> wrote:
>> Should backends be allowed to add functionality or should they be completely
>> interchangeable. E.g. unlike fftpack, FFTW has support for long doubles;
>> should this be exposed to SciPy users?
>
> If a user writes backend-specific code, then they may just as well bypass the
> backend right, and call the other library directly?

I agree, in fact I quote you in the proposal saying something similar.

> Although for long double things will work out of the box perhaps, because the
> dtype of an input array doesn't come back in the API. So the array can be
> passed straight to the backend.

This was why I picked it as an example. It doesn't add any new parameters or
change the function signature at all, but it does change the result of a
particular call to the API. So it's a very minimal change yet it still breaks
the interchangeability of backends.
> This is probably also a good time to point out http://uarray.readthedocs.io/.
> It can provide a complete backend system without too much work

After reading through the docs it does seem to cover this and more. However, it
also seems to be unstable at this point so I’m not sure it’s wise to rely on it just
yet.

> it was designed exactly for the type of problem you're trying to solve here.

Unless I'm misunderstanding it seems to be more focused on supporting different
types of array, rather than different implementations for NumPy arrays. I can't
see any utility in having a complicated dispatch mechanism when we really only
want to have a single backend at a time for any given function.

I would assume that because of this, uarray would come with a greater
performance overhead. Whereas I think the overhead of an fftpack backend should
just be some argument validation and a single function call. I could always
implement it both ways and benchmark it if there's interest.

Peter

From: SciPy-Dev <scipy-dev-bounces+peterbell10=live.co.uk at python.org<mailto:scipy-dev-bounces+peterbell10=live.co.uk at python.org>> On Behalf Of Ralf Gommers
Sent: 08 May 2019 23:59
To: SciPy Developers List <scipy-dev at python.org<mailto:scipy-dev at python.org>>
Subject: Re: [SciPy-Dev] Backend support for fftpack - GSoC 2019



On Wed, May 8, 2019 at 5:50 PM Peter Bell <PeterBell10 at live.co.uk<mailto:PeterBell10 at live.co.uk>> wrote:
Hello all,

I’m happy to say my GSoC proposal was accepted,

Hey Peter, congratulations! We're happy to have you on board!

so I’ll be working over the summer to add support for multiple fftpack backends. For those interested, I’ve extracted the main discussion from my proposal which you can read here:
https://docs.google.com/document/d/1hGMc6ZrsmSi_LSixI5oFbo5cZkPUo92sfmnVthg1u6Y/edit

Thanks, interesting. Will take a little while to process. I just give some very initial thoughts below (so take with a grain of salt).


There are a several design decisions I raise in the proposal and any comments would be appreciated. Of particular note:
* Should backends be allowed to add functionality or should they be completely interchangeable. E.g. unlike fftpack, FFTW has support for long doubles; should this be exposed to SciPy users?

If a user writes backend-specific code, then they may just as well bypass the backend right, and call the other library directly? Although for long double things will work out of the box perhaps, because the dtype of an input array doesn't come back in the API. So the array can be passed straight to the backend.

* Is there any interest in adding a SciPy config file? For just one option it’s probably not worthwhile but if there is interest in more config options then it starts to make more sense.

My sense is no. Something like a context manager (`with backend('pyfftw'):`) is lightweight enough probably.

This is probably also a good time to point out http://uarray.readthedocs.io/. It can provide a complete backend system without too much work, it was designed exactly for the type of problem you're trying to solve here. Would be good to have your thoughts on that.


Ideally a clear set of design goals can be agreed upon now, before it gets too far into the coding period which begins at the end of the month.

Yes great idea.

Cheers,
Ralf

_______________________________________________
SciPy-Dev mailing list
SciPy-Dev at python.org<mailto:SciPy-Dev at python.org>
https://mail.python.org/mailman/listinfo/scipy-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20190610/a0d1400e/attachment-0001.html>