[SciPy-Dev] GSoC 2019: Project Proposal

Tue Apr 9 01:24:16 EDT 2019

On Mon, Apr 8, 2019 at 7:54 AM Gregory Lee <grlee77 at gmail.com> wrote:

>
>
> On Sat, Apr 6, 2019 at 5:08 PM Anany Shrey Jain <
> f20170920 at goa.bits-pilani.ac.in> wrote:
>
>>
>> Hey ,
>> Thanks Greg for a detailed reply. I will be working on both
>> https://github.com/pyFFTW/pyFFTW/pull/256 and
>> https://github.com/cupy/cupy/issues/1936 as a part of this project. I
>> have constructed my API implementation based on the `set_backend`  function
>> approach, but in comparatively cleaner way. I have written my proposal for
>> this project and here is a link to it:
>> https://docs.google.com/document/d/1TIiH9j3rPAN--53MwjZz7NFO1GGtxmEH3RiGzvqnEBk/edit?usp=sharing.
>> Please go through my proposal and if possible send some feedback on it.
>>
>> -Anany
>>
>>
> Hi Anany,
>
> I appreciate that you have narrowed down some of the possibilities. The
> API will need further refinement to reach consensus, but the general
> concept as presented seems reasonable and I don't think we have to reach
> consensus on the finer implementation details for purposes of the proposal.
> I am not sure it is necessary to have the `set_backend` function return a
> class that allows setting additional arguments. Ralf's made a comment
> earlier that was in favor of just supporting the arguments currently
> present in SciPy's fftpack. I was also thinking something a bit simpler in
> terms of just reading the name of the backend from a configuration file or
> environment variable as opposed to running a full configuration script, but
> others may disagree.
>

The google doc proposal right now seems to say environment variable. I
think that's fine, better than a config file/script (which I'd really like
to avoid). The name should be scipy-specific though, right now it says
"BACKENDPATH". Perhaps something like "SCIPY_FFT_BACKEND".

As for setting it in code, a context manager would be nice in addition to a
regular function. API something like:

1. set_backend(name)
2. get_backend()
3. with fftpack.backend(name):

> I will comment mainly on the timeline section of the proposal as we did
> not discuss that previously:
>
> I think the timeline is a bit overly optimistic and dedicates too much
> time to third party libraries. It is probably not realistically going to be
> possible to complete changes to SciPy as well as pyFFTW, mkl_fft and CuPy.
> I would focus the majority of the timeline on SciPy itself. You can leave
> the text about working on the pyFFTW and CuPy issues as goals for weeks
> 9-12, but I think all work in the early weeks should be in SciPy itself and
> not require any changes to the third-party libraries in order for the
> backend code to function. Each of the third party libraries already has a
> substantial portion of the SciPy API implemented and can usefully be
> integrated as backends as-is.
>
> I think you can keep the plan to implement an initial interface and pyFFTW
> support in weeks 1-2, but I think week 3-4 should be primarily along the
> lines of improving and revising the initial interface based on reviewer
> feedback from your first PR. Realistically, it will take a substantial
> amount of time to reach agreement among all parties and code will not be
> merged right away. I would focus weeks 3-4 on revising based on feedback
> rather than proposing to "make CuPy compatible". CuPy already has
> compatible interfaces for many of the functions (
> https://github.com/cupy/cupy/blob/master/cupyx/scipy/fftpack/fft.py).
>
> Week 5 seems to be missing from the timeline?
>
> I would remove the item about adding changes to mkl_fft (in week 6). Was
> there something specific you had in mind for that? There are already
> existing wrappers (
> https://github.com/IntelPython/mkl_fft/blob/master/mkl_fft/_scipy_fft.py)
> and I'm not sure there is much more to do there.  A suitable replacement
> for the week 5-6 section of the timeline could be working on the SciPy user
> documentation detailing how to use and configure the backends. It could be
> considered to potentially make small PRs to third party library
> documentation as well, mentioning how scipy can be configured to use them
> as its default backend. For instance, the new backend-based approach will
> be preferable to the monkey patching currently discussed in the pyFFTW docs.
>
> I proposed above to remove mention of CuPy from weeks 3-4 to reduce the
> overall proportion of time in the proposal that is related to CuPy for a
> couple of reasons:
>
> 1.) There is no existing precedent for interfacing with GPU code from
> SciPy so that may be a little controversial among the larger group of SciPy
> developers and have a harder time of getting merged. If we assume NumPy
> inputs should give NumPy outputs then the required host/device memory
> transfers will reduce performance, making the GPU-based approach only
> beneficial for large transforms. We could allow GPU-based input/output, but
> other functions in SciPy itself do not support GPU-based arrays.
>

None of the code in SciPy should directly rely on GPUs, however some code
in SciPy may already work if you pass in a CuPy array (due to
__array_ufunc__ and __array_function__). When you think about a CuPy
backend, it should work by passing in CuPy arrays instead of NumPy arrays,
and having things work transparently via those array protocols. It doesn't
really make sense to put in code to convert array types imho, that will be
inefficient and hard to maintain.

http://scipy.github.io/devdocs/roadmap.html#support-for-distributed-arrays-and-gpu-arrays
is
the relevant item on our roadmap here.

Cheers,
Ralf

>
> 2.) Using CuPy requires access to a relatively recent (~last five years or
> so) NVidia GPU that supports CUDA (I think at least CUDA 8.0 is required).
> Do you have access to an appropriate GPU? There is no requirement for
> applicants to have access to such hardware, but it will difficult to
> propose making contributions to CuPy itself without it. We may be able to
> work out remote access to a machine that has appropriate hardware, but I
> have not looked into specifics of how we would arrange that yet and cannot
> guarantee it.
>
>
>
>
> Message: 1
>> Date: Fri, 5 Apr 2019 12:13:15 -0400
>> From: Gregory Lee <grlee77 at gmail.com>
>> To: SciPy Developers List <scipy-dev at python.org>
>> Subject: Re: [SciPy-Dev] GSoC 2019
>> Message-ID:
>>         <
>> CAJR3sXeVPAB+jGB_EdxiDg45jbR4wHzLVjjBk7+RiesT0XMrpw at mail.gmail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> On Fri, Apr 5, 2019 at 4:25 AM Anany Shrey Jain <
>> f20170920 at goa.bits-pilani.ac.in> wrote:
>>
>> > Hey Greg,
>> > The project "Revamp FFTPACK" also aims at contributing to third party
>> > libraries to fill in functionality gaps. For pyFFTW I can see that work
>> on
>> > adding Real-to-Real transforms has been going on for a long time. So do
>> I
>> > need to continue the work going on in
>> > https://github.com/pyFFTW/pyFFTW/pull/256 as a part of this project?
>> >
>>
>> It is not a requirement that you work on the pyFFTW wrappers, but that
>> would be a good complement to this project. Here are some thoughts.
>>
>> 1.) I would first have a mechanism to fall back to the existing
>> scipy.fftpack implementations for any functions not currently supported by
>> a given backend. pyFFTW currently already falls back to scipy's own
>> implementation for DCT and DST transforms (
>>
>> https://github.com/pyFFTW/pyFFTW/blob/e9966e5d215a00f89156bd88e1d42b884f85df78/pyfftw/interfaces/scipy_fftpack.py#L81-L82
>> ).
>> The PR you referenced would allow it to use FFTW to perform these
>> transforms instead and there would be no remaining functions that fall
>> back
>> to scipy.fftpack.
>>
>> 2.) You can work on adding real-to-real support in supporting libraries,
>> but this is a secondary goal to pursue once the SciPy side of the project
>> has been completed.
>>    pyFFTW builds on FFTW which does already have functions providing
>> real-to-real transforms. So what you see in the PR referenced above is
>> mainly adding the wrappers for those to pyFFTW.
>>
>>
>>
>> > What needs to be done for the other two libraries? Do I need to make a
>> > similar PR to other two libraries as well?
>> >
>> >
>> You could potentially try that, but I think that is more of a stretch
>> goal.
>> The approach for CuPy or mkl-fft would have to be different, because for
>> those two cases the underlying libraries DO NOT have dedicated functions
>> implementing the real-to-real transforms. Also, the underlying FFT
>> libraries are proprietary, so we cannot propose to add new transforms on
>> the C/C++ side. However, it is possible in principle to implement
>> real-to-real transforms using only the already existing complex-valued
>> transforms (albeit at some additional computational/memory cost). Some
>> detailed examples of this are given in the following thread on DSP
>> StackExchange:
>> https://dsp.stackexchange.com/questions/2807/fast-cosine-transform-via-fft
>>
>> I think that is probably the only feasible route for real-to-real
>> transforms using those two backends. I think whether or not this type of
>> implementation would give a performance benefit over just reusing the
>> existing scipy.fftpack implementations is unknown.
>>
>> One FFT-related issues that could be improved in CuPy in a more
>> straightforward manner than adding real-to-real support, is improving the
>> real-to-complex and complex-to-real support.  These transforms are
>> supported by the underlying CUFFT library, but support for this within
>> CuPy
>> is incomplete.  One issue related to this was opened at (
>> https://github.com/cupy/cupy/issues/1936), but no PR implementing it has
>> been made.
>>
>>
>> > For all the three libraries there are SciPy-compatable routines already
>> > defined. After getting required information using the newly created
>> > interface I need to use them at the backend, right?
>> >
>>
>> Yes, you should rely on the already existing interfaces when implementing
>> the backends in SciPy. Any additional 3rd party libraries would have to
>> provide a scipy-like interface in order to be compatible for use as a
>> backend.
>>
>>
>>
>> >
>> > -Anany
>> >
>> >> ----------------------------------------------------------------------
>> >>
>> >> Message: 1
>> >> Date: Thu, 4 Apr 2019 10:58:27 -0400
>> >> From: Gregory Lee <grlee77 at gmail.com>
>> >> To: SciPy Developers List <scipy-dev at python.org>
>> >> Subject: Re: [SciPy-Dev] SciPy-Dev Digest, Vol 185, Issue 18
>> >> Message-ID:
>> >>         <
>> >> CAJR3sXcJuqQJrDboN5zH-pd7NG-wD8EWCdTCAtQ+vJpnZjKsRw at mail.gmail.com>
>> >> Content-Type: text/plain; charset="utf-8"
>> >>
>> >> Hi Anany,
>> >>
>> >> It is nice to see that you have updated with multiple potential APIs.
>> Can
>> >> some other core SciPy devs also help give feedback on what SciPy would
>> >> find
>> >> preferable? Out of the four you proposed, I think something along the
>> >> lines
>> >> of the second method is cleanest in terms of setting the backend at
>> >> runtime. I think having some aspect from your 4th variant where the
>> >> default
>> >> backend can be read from a configuration file and/or environment
>> variable
>> >> is also desirable so that the user's desired default can always be
>> >> automatically set at import time.
>> >>
>> >> I disagree with the following specific statement, though:
>> >> "It is better to use all the above mentioned ideas because every idea
>> has
>> >> its own benefits besides it gives user the luxury to choose the method
>> >> implementation of backend."
>> >>
>> >> I do not think we want to provide multiple ways to do the same thing.
>> It
>> >> can be confusing to users and more work for devs to maintain. I think
>> >> there
>> >> should be one method of setting the backend that the user can either
>> call
>> >> at run time or have it be automatically called based on some sort of
>> >> configuration file and/or environment variable(s) at import time.
>> >>
>> >> - Greg
>> >>
>> >>
>> >> On Tue, Apr 2, 2019 at 2:19 PM Anany Shrey Jain <
>> >> f20170920 at goa.bits-pilani.ac.in> wrote:
>> >>
>> >> > Hey,
>> >> > I have already sent the link to the page containing the list of ideas
>> >> for
>> >> > implementation of new API that I propose for this project. Would you
>> >> like
>> >> > to add something to it or give some comments on it? The Gsoc proposal
>> >> > deadline is 9th of April and I am planning to put all of this in my
>> >> > proposal to make it effective . So any more ideas or comments will be
>> >> > really helpful for me . Here is the link to the page:
>> >> > https://github.com/ananyashreyjain/Gsoc-Scipy/wiki/Gsoc-2019:-Scipy
>> .
>> >> >
>> >> > On Sun, Mar 31, 2019 at 5:23 AM <scipy-dev-request at python.org>
>> wrote:
>> >> >
>> >> >>
>> >> >> Message: 1
>> >> >> Date: Sat, 30 Mar 2019 13:31:51 -0400
>> >> >> From: Eric Larson <larson.eric.d at gmail.com>
>> >> >> To: SciPy Developers List <scipy-dev at python.org>
>> >> >> Subject: Re: [SciPy-Dev] GSoC 2019
>> >> >> Message-ID:
>> >> >>         <
>> >> >> CAGu2niVWsKrMukfWcAWAtj-1NBm2utCeh3BOnNku7jNoAcx-kw at mail.gmail.com>
>> >> >> Content-Type: text/plain; charset="utf-8"
>> >> >>
>> >> >> It looks like the main ideas are in these bits of code, but not
>> >> explicitly
>> >> >> stated. In brief it looks like the proposed API is to add a
>> >> `fft_object`
>> >> >> kwarg to every function in `scipy.fftpack`. It would be good to
>> discuss
>> >> >> this choice explicitly -- what advantages and disadvantages it has
>> >> >> relative
>> >> >> to other approaches (e.g., creating new namespaces, or having a
>> >> >> `set_backend` function (and/or context manager) that would change
>> >> things
>> >> >> under the hood).
>> >> >>
>> >> >> If you make changes and need more feedback, let us know. The more
>> >> specific
>> >> >> you can be about what you need feedback on (high level ideas, low
>> level
>> >> >> implementation, etc.), the better.
>> >> >>
>> >> >> Eric
>> >> >>
>> >> >>
>> >> >> On Fri, Mar 29, 2019 at 4:12 AM Anany Shrey Jain <
>> >> >> f20170920 at goa.bits-pilani.ac.in> wrote:
>> >> >>
>> >> >> > Hey,
>> >> >> > I went through the idea list of Scipy for this years's GSoC and I
>> >> intend
>> >> >> > to work towards the idea of revamping the Scipy's fftpack. I have
>> >> >> already
>> >> >> > been contributing to Scipy since the end of last year and I have
>> some
>> >> >> > knowledge about Fourier transforms as it was the part of my
>> >> discipline.
>> >> >> I
>> >> >> > could not take part in this discussion earlier because my mid
>> >> semester
>> >> >> > exams were going on, but since they are now over so I can devote
>> as
>> >> much
>> >> >> > time as required on this project. I have already created a github
>> >> page
>> >> >> with
>> >> >> > some code snippets and functionalities that we can implement,
>> here is
>> >> >> the
>> >> >> > link to it:
>> >> >> >
>> https://github.com/ananyashreyjain/Gsoc-Scipy/wiki/Gsoc-2019:-Scipy
>> >> .
>> >> >> >
>> >> >> > Cheers,
>> >> >> > Anany Jain
>> >> >> >
>> >> >> > _______________________________________________
>> >> >> > SciPy-Dev mailing list
>> >> >> > SciPy-Dev at python.org
>> >> >> > https://mail.python.org/mailman/listinfo/scipy-dev
>> >> >> >
>> >> >> -------------- next part --------------
>> >> >> An HTML attachment was scrubbed...
>> >> >> URL: <
>> >> >>
>> >>
>> http://mail.python.org/pipermail/scipy-dev/attachments/20190330/f1755dfb/attachment-0001.html
>> >> >> >
>> >> >>
>> >> >> ------------------------------
>> >> >>
>> >> >> Message: 2
>> >> >> Date: Sat, 30 Mar 2019 19:38:41 -0400
>> >> >> From: Gregory Lee <grlee77 at gmail.com>
>> >> >> To: SciPy Developers List <scipy-dev at python.org>
>> >> >> Subject: Re: [SciPy-Dev] GSoC 2019
>> >> >> Message-ID:
>> >> >>         <CAJR3sXfnnr-0JO87gLbhh-7b-w=KjaNpac+F+FHTeFt2K=_
>> >> >> QMw at mail.gmail.com>
>> >> >> Content-Type: text/plain; charset="utf-8"
>> >> >>
>> >> >> On Fri, Mar 29, 2019 at 4:12 AM Anany Shrey Jain <
>> >> >> f20170920 at goa.bits-pilani.ac.in> wrote:
>> >> >>
>> >> >> > Hey,
>> >> >> > I went through the idea list of Scipy for this years's GSoC and I
>> >> intend
>> >> >> > to work towards the idea of revamping the Scipy's fftpack. I have
>> >> >> already
>> >> >> > been contributing to Scipy since the end of last year and I have
>> some
>> >> >> > knowledge about Fourier transforms as it was the part of my
>> >> discipline.
>> >> >> I
>> >> >> > could not take part in this discussion earlier because my mid
>> >> semester
>> >> >> > exams were going on, but since they are now over so I can devote
>> as
>> >> much
>> >> >> > time as required on this project. I have already created a github
>> >> page
>> >> >> with
>> >> >> > some code snippets and functionalities that we can implement,
>> here is
>> >> >> the
>> >> >> > link to it:
>> >> >> >
>> https://github.com/ananyashreyjain/Gsoc-Scipy/wiki/Gsoc-2019:-Scipy
>> >> .
>> >> >> >
>> >> >> > Cheers,
>> >> >> > Anany Jain
>> >> >> >
>> >> >> > _______________________________________________
>> >> >> > SciPy-Dev mailing list
>> >> >> > SciPy-Dev at python.org
>> >> >> > https://mail.python.org/mailman/listinfo/scipy-dev
>> >> >>
>> >> >>
>> >> >> Hi Anany,
>> >> >>
>> >> >> Thank you for your interest in GSOC with SciPy!  It's great to see
>> that
>> >> >> you
>> >> >> are already thinking about a potential API and learning about the
>> 3rd
>> >> >> party
>> >> >> FFT libraries. Along the lines of Eric's comment, think about what
>> this
>> >> >> API
>> >> >> choice implies for users of libraries built upon SciPy. Would users
>> of
>> >> 3rd
>> >> >> party libraries that rely on SciPy's FFTs be immediately able to
>> take
>> >> >> advantage of a differentf FFT backend in this scenario? (in other
>> >> words,
>> >> >> the case where the user is not calling scipy.fftpack.fft directly
>> but
>> >> is
>> >> >> using it only indirectly via a function in a downstream library).
>> >> >>
>> >> >> Matplotlib is an example of an existing Python library supporting
>> >> multiple
>> >> >> backends that can be considered for comparison (see:
>> >> >> https://matplotlib.org/faq/usage_faq.html#what-is-a-backend). Think
>> >> about
>> >> >> the relative benefits/drawbacks of some of the methods they use for
>> >> >> setting
>> >> >> a backend there as compared to the approach proposed for FFTs here.
>> >> >>
>> >> >> Another thing that should be considered: pyFFTW, MKL and CuPy
>> already
>> >> have
>> >> >> some functions that match the existing scipy.fftpack API, but they
>> may
>> >> >> have
>> >> >> one or more additional arguments particular to features of that
>> >> library.
>> >> >> How would you propose to deal with supporting additional optional
>> >> >> arguments
>> >> >> on the SciPy end?
>> >> >>
>> >> >> One other minor comment: I know that the blog post you made may be
>> >> serving
>> >> >> partly as personal notes, but please do not directly copy sections
>> of
>> >> text
>> >> >> (e.g. the description of BLAS/LAPACK libraries) from elsewhere
>> without
>> >> >> attribution when writing a final proposal. (Also, the FFT functions
>> are
>> >> >> separate from the BLAS/LAPACK functions in MKL, so you will not
>> need to
>> >> >> discuss BLAS or LAPACK specifically in your proposal)
>> >> >>
>> >> >> - Greg
>> >> >> -------------- next part --------------
>> >> >> An HTML attachment was scrubbed...
>> >> >> URL: <
>> >> >>
>> >>
>> http://mail.python.org/pipermail/scipy-dev/attachments/20190330/c1351e7e/attachment-0001.html
>> >> >> >
>> >> >>
>> >> >> ------------------------------
>> >> >>
>> >> >> Message: 3
>> >> >> Date: Sat, 30 Mar 2019 19:53:02 -0400
>> >> >> From: Gregory Lee <grlee77 at gmail.com>
>> >> >> To: SciPy Developers List <scipy-dev at python.org>
>> >> >> Subject: Re: [SciPy-Dev] GSoC 2019
>> >> >> Message-ID:
>> >> >>         <CAJR3sXdcPMAn_DETF8W-=
>> >> >> MWnGbkJMpQ2hK1ni20zoLbgf6S0Yw at mail.gmail.com>
>> >> >> Content-Type: text/plain; charset="utf-8"
>> >> >>
>> >> >> Hi Gopi,
>> >> >>
>> >> >> I am not sure I follow exactly what you are suggesting here. Are you
>> >> >> proposing that the `fftpackN` variable in your example would be a
>> >> Python
>> >> >> module corresponding to a given backend? If so, why is an array of
>> >> data,
>> >> >> z,
>> >> >> being passed into the newAPI function here? I don't see why a data
>> >> array
>> >> >> should be needed at all to choose a backend.
>> >> >>
>> >> >> Please also see my response to Anany's recent post on this list for
>> a
>> >> >> couple of other suggestions of things to consider in terms of the
>> API.
>> >> >>
>> >> >> - Greg
>> >> >>
>> >> >>
>> >> >> On Fri, Mar 29, 2019 at 5:27 AM Gopi Manohar Tatiraju <
>> >> >> deathcoderx at gmail.com>
>> >> >> wrote:
>> >> >>
>> >> >> > Hey,
>> >> >> > I went through the code and got some ideas.
>> >> >> > So what will happen is:
>> >> >> > First, we will declare FFTW as we use them normally.
>> >> >> >
>> >> >> > z = np.random.rand(2, 128, 64)
>> >> >> > pyfftw.FFTW(z, axes=(-1, ))
>> >> >> > ..
>> >> >> > ..
>> >> >> > Now after declaring our basic work to shift to another FFTpack,
>> what
>> >> we
>> >> >> > can do is:
>> >> >> > fftPackN= newAPI(z, pack = 'PYFFTW');
>> >> >> >
>> >> >> > Pack will take the name of the 3rd party FFT pack which we want to
>> >> use.
>> >> >> > Suggestions needed.
>> >> >> >
>> >> >> > Cheers.
>> >> >> >
>> >> >> >
>> >> >>
>> >> >> >
>> >> >> > On Wed, Mar 27, 2019 at 8:43 PM Eric Larson <
>> larson.eric.d at gmail.com
>> >> >
>> >> >> > wrote:
>> >> >> >
>> >> >> >> After reading the related issues and thinking about the problem,
>> >> >> perhaps
>> >> >> >> the best first step is you come up with some initial API
>> proposal.
>> >> >> >>
>> >> >> >> One way to work through the process is to think about the use
>> cases
>> >> >> that
>> >> >> >> we want to support. How should users actually write code to use
>> the
>> >> new
>> >> >> >> functionality? You can start with short code snippets that do not
>> >> work
>> >> >> >> currently, but should work for users at the end of the project.
>> Then
>> >> >> think
>> >> >> >> about what needs to change inside SciPy for these code snippets
>> to
>> >> >> work.
>> >> >> >> This can become an iterative process as you realize that the
>> SciPy
>> >> >> changes
>> >> >> >> might be better or easier with changes to the user-facing API,
>> etc.
>> >> >> >>
>> >> >> >> Eric
>> >> >> >>
>> >> >> >>
>> >> >> >> On Tue, Mar 26, 2019 at 3:38 PM Gopi Manohar Tatiraju <
>> >> >> >> deathcoderx at gmail.com> wrote:
>> >> >> >>
>> >> >> >>> Hey,
>> >> >> >>> As proposal submission period started I want to discuss a bit
>> anobt
>> >> >> how
>> >> >> >>> can we implement the API.
>> >> >> >>> I would very much appriciate inputs and guidance.
>> >> >> >>>
>> >> >> >>> Cheers.
>> >> >> >>>
>> >> >> >>> On Thu, Mar 7, 2019 at 11:40 PM Eric Larson <
>> >> larson.eric.d at gmail.com>
>> >> >> >>> wrote:
>> >> >> >>>
>> >> >> >>>> The best way to figure that out is probably to take a look at
>> >> some of
>> >> >> >>>> the existing (minor) SciPy GitHub issues and try to work on
>> one or
>> >> >> two that
>> >> >> >>>> you feel like working on. Actually trying to work out how to
>> >> >> contribute
>> >> >> >>>> code changes (as independently as possible/reasonable based on
>> the
>> >> >> existing
>> >> >> >>>> guides/docs already discoverable online) will quickly highlight
>> >> >> things that
>> >> >> >>>> need to be learned. Some of the general things are the SciPy
>> >> >> contributing
>> >> >> >>>> guidelines, standard GitHub PR workflow, communication with
>> >> >> reviewers, and
>> >> >> >>>> so forth.
>> >> >> >>>>
>> >> >> >>>> For this particular project, it's important to understand the
>> >> various
>> >> >> >>>> flavors and properties of FFT in SciPy (real, complex,
>> >> n-dimensional,
>> >> >> >>>> caching plans, etc.) as well as other projects like NumPy,
>> CuPy,
>> >> and
>> >> >> >>>> pyFFTW. Some of this can be learned and improved upon as part
>> of
>> >> >> GSoC, but
>> >> >> >>>> the more you know and can demonstrate / put into practice
>> >> knowledge
>> >> >> of
>> >> >> >>>> before the applications are reviewed, the better.
>> >> >> >>>>
>> >> >> >>>> Eric
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>> On Wed, Mar 6, 2019 at 4:55 AM Gopi Manohar Tatiraju <
>> >> >> >>>> deathcoderx at gmail.com> wrote:
>> >> >> >>>>
>> >> >> >>>>> Thank you for the detailed reply.
>> >> >> >>>>> I will keep these point in mind while preparing my proposal.
>> >> Apart
>> >> >> >>>>> from proposal, what skills shall I improve so that I can
>> >> contribute
>> >> >> more
>> >> >> >>>>> effectively and efficiently.
>> >> >> >>>>>
>> >> >> >>>>> Cheers.
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>> On Wed, Mar 6, 2019, 3:42 AM Eric Larson <
>> >> larson.eric.d at gmail.com>
>> >> >> >>>>> wrote:
>> >> >> >>>>>
>> >> >> >>>>>> There are a number of considerations that would be good to
>> >> address.
>> >> >> >>>>>> I'd start by considering the things that are typically
>> >> considered
>> >> >> >>>>>> when making changes to SciPy
>> >> >> >>>>>> <
>> >> >>
>> >>
>> https://docs.scipy.org/doc/scipy/reference/hacking.html#contributing-to-scipy
>> >> >> >,
>> >> >> >>>>>> but I'd expand it to a longer list since the scope is bigger
>> >> than
>> >> >> a typical
>> >> >> >>>>>> single PR. The list would contain (at least):
>> >> >> >>>>>>
>> >> >> >>>>>> - what will the user-facing API / contract be?
>> >> >> >>>>>> - what in SciPy needs to change to allow it (internals /
>> >> >> >>>>>> implementation outline, perhaps skeleton functions/classes)?
>> >> >> >>>>>> - what examples should we show users (in general this ends up
>> >> being
>> >> >> >>>>>> just simple use cases of the API)?
>> >> >> >>>>>> - how will we test it for correctness?
>> >> >> >>>>>> - how will we benchmark it to prevent performance
>> regressions?
>> >> >> >>>>>> - how can we break up the work for these steps (if possible)
>> >> into
>> >> >> >>>>>> separable, sequential (or -- better but not usually possible
>> --
>> >> >> >>>>>> independent) PRs?
>> >> >> >>>>>>
>> >> >> >>>>>> You might notice that none of these points are actually all
>> that
>> >> >> >>>>>> specific to the FFT project, but can be generally applicable
>> to
>> >> >> GSoC
>> >> >> >>>>>> projects. For the FFT project, I would say "how do we build
>> the
>> >> >> backend" is
>> >> >> >>>>>> indeed implicitly contained / outlined in these bullet
>> points.
>> >> And
>> >> >> as with
>> >> >> >>>>>> most projects, the first two points (and possibly the last)
>> will
>> >> >> probably
>> >> >> >>>>>> be the most challenging to sort out.
>> >> >> >>>>>>
>> >> >> >>>>>> Eric
>> >> >> >>>>>>
>> >> >> >>>>>>
>> >> >> >>>>>> On Mon, Mar 4, 2019 at 10:09 PM Gopi Manohar Tatiraju <
>> >> >> >>>>>> deathcoderx at gmail.com> wrote:
>> >> >> >>>>>>
>> >> >> >>>>>>> Hey Eric,
>> >> >> >>>>>>> So for my proposal, shall I include how can we built a back
>> >> end?
>> >> >> >>>>>>> What all points shall I include to make it effective?
>> >> >> >>>>>>>
>> >> >> >>>>>>> Cheers.
>> >> >> >>>>>>>
>> >> >> >>>>>>> On Tue, Mar 5, 2019, 4:50 AM Eric Larson <
>> >> larson.eric.d at gmail.com
>> >> >> >
>> >> >> >>>>>>> wrote:
>> >> >> >>>>>>>
>> >> >> >>>>>>>> That would be me -- don't worry, I'm on this list, as are
>> >> >> >>>>>>>> (presumably) other developers interested in the
>> discussion, so
>> >> >> no need to
>> >> >> >>>>>>>> document elsewhere for now from our end. It looks like Greg
>> >> has
>> >> >> summarized
>> >> >> >>>>>>>> the current status and thoughts we've had, so I don't have
>> >> much
>> >> >> to add at
>> >> >> >>>>>>>> this point.
>> >> >> >>>>>>>>
>> >> >> >>>>>>>> Eric
>> >> >> >>>>>>>>
>> >> >> >>>>>>>>
>> >> >> >>>>>>>>
>> >> >> >>>>>>>>
>> >> >> >>>>>>>> On Sun, Mar 3, 2019 at 11:59 PM Gopi Manohar Tatiraju <
>> >> >> >>>>>>>> deathcoderx at gmail.com> wrote:
>> >> >> >>>>>>>>
>> >> >> >>>>>>>>> Hey Greg,
>> >> >> >>>>>>>>> I got the basic idea about what we want to implement this
>> >> year.
>> >> >> >>>>>>>>> Our first priority is to create a backend for 3rd party
>> >> >> fftpacks.
>> >> >> >>>>>>>>> We can create API using Python and Flask or any other
>> >> suitable
>> >> >> framework.
>> >> >> >>>>>>>>>
>> >> >> >>>>>>>>> I thought this discussion was going on on SciPy Developer
>> >> List
>> >> >> >>>>>>>>> only. So do I need to document it all and post it
>> somewhere
>> >> so
>> >> >> that other
>> >> >> >>>>>>>>> contributors can see this?
>> >> >> >>>>>>>>> Also, I couldn't find any contact info regarding Eric. It
>> >> would
>> >> >> be
>> >> >> >>>>>>>>> very helpful if we include him in this discussion as this
>> is
>> >> a
>> >> >> roadmap for
>> >> >> >>>>>>>>> GSoC project and I can also get more insight into the
>> project
>> >> >> and expected
>> >> >> >>>>>>>>> outcomes.
>> >> >> >>>>>>>>>
>> >> >> >>>>>>>>> Cheers.
>> >> >> >>>>>>>>>
>> >> >> >>>>>>>>> On Mon, Mar 4, 2019 at 9:31 AM Gregory Lee <
>> >> grlee77 at gmail.com>
>> >> >> >>>>>>>>> wrote:
>> >> >> >>>>>>>>>
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>>> On Sun, Mar 3, 2019 at 12:28 PM Gopi Manohar Tatiraju <
>> >> >> >>>>>>>>>> deathcoderx at gmail.com> wrote:
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>>>> Hey Gregory,
>> >> >> >>>>>>>>>>> I went through the idea and I am very much interested in
>> >> >> working
>> >> >> >>>>>>>>>>> on it. I had Fourier Transforms as a part of my academic
>> >> >> course, so I am
>> >> >> >>>>>>>>>>> well aware with basic concepts.
>> >> >> >>>>>>>>>>> One thing I want to discuss is, its mentioned that we
>> have
>> >> to
>> >> >> >>>>>>>>>>> create a backend system for 3rd party FFTpacks so my
>> dobuts
>> >> >> >>>>>>>>>>>
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>>>> 1. How are we going to create API?
>> >> >> >>>>>>>>>>>
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>>> This will needed to be decided on in consensus with the
>> >> SciPy
>> >> >> >>>>>>>>>> developers and user community. I am mentoring for the
>> >> project
>> >> >> as a
>> >> >> >>>>>>>>>> contributor here and a maintainer of the related pyFFTW
>> >> >> package, but I am
>> >> >> >>>>>>>>>> not a core SciPy dev (the other mentor, Eric, is a core
>> dev
>> >> >> here). I have
>> >> >> >>>>>>>>>> not discussed the API in any detail with the SciPy team
>> at
>> >> >> this stage.
>> >> >> >>>>>>>>>> Working out details of what the API should look like and
>> >> >> working to
>> >> >> >>>>>>>>>> finalize that in collaboration with the community will
>> be a
>> >> >> primary task
>> >> >> >>>>>>>>>> during the early part of the GSOC project.
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>>> 2. After the creation of backend will user at the time of
>> >> >> >>>>>>>>>>> installation of SciPy will decide what all 3rd party
>> >> FFTpacks
>> >> >> he needs or
>> >> >> >>>>>>>>>>> we will include some by default?
>> >> >> >>>>>>>>>>>
>> >> >> >>>>>>>>>>> The idea is to allow users to opt-in to a different FFT
>> >> >> backend
>> >> >> >>>>>>>>>> at run time, but these alternative libraries will not be
>> >> >> distributed with
>> >> >> >>>>>>>>>> SciPy itself due to incompatible software licenses
>> (pyFFTW
>> >> and
>> >> >> mkl-fft) or
>> >> >> >>>>>>>>>> a requirement for specific hardware (CuPy requires an
>> NVIDIA
>> >> >> GPU). SciPy
>> >> >> >>>>>>>>>> will continue to offer its own bundled FFT library
>> >> (currently
>> >> >> fftpack, but
>> >> >> >>>>>>>>>> this could be switched to pocketfft as discussed in the
>> >> ideas
>> >> >> page). The
>> >> >> >>>>>>>>>> purpose of the backend interface is to provide a
>> convenient
>> >> >> way for users
>> >> >> >>>>>>>>>> to have third party libraries perform the FFTs instead,
>> but
>> >> >> under the
>> >> >> >>>>>>>>>> existing scipy.fftpack API. This is motivated by a
>> desire by
>> >> >> end users and
>> >> >> >>>>>>>>>> downstream projects to have features such as
>> multi-threading
>> >> >> (pyFFTW and
>> >> >> >>>>>>>>>> mkl-fft) or GPU support (CuPy) that are not implemented
>> in
>> >> >> scipy.fftpack.
>> >> >> >>>>>>>>>> Ultimately it will be up to users that opt-in to using
>> these
>> >> >> other
>> >> >> >>>>>>>>>> libraries to confirm that the license terms and are
>> >> compatible
>> >> >> with their
>> >> >> >>>>>>>>>> use case.
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>>> p.s. It is not a big deal, but I think the SciPy-Dev
>> mailing
>> >> >> >>>>>>>>>> lists prefers replies **below** the email you are
>> replying
>> >> to
>> >> >> so people can
>> >> >> >>>>>>>>>> more easily follow the thread of discussion. There are
>> some
>> >> >> guidelines at:
>> >> >> >>>>>>>>>> https://www.scipy.org/scipylib/mailing-lists.html
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>>> I will research more about how to implement a backend
>> myself
>> >> >> and
>> >> >> >>>>>>>>>>> will let you know as soon as I come up with something
>> new.
>> >> >> >>>>>>>>>>>
>> >> >> >>>>>>>>>>>
>> >> >> >>>>>>>>>> CHeers,
>> >> >> >>>>>>>>>>> Greg
>> >> >> >>>>>>>>>>>
>> >> >> >>>>>>>>>>> On Sun, Mar 3, 2019 at 10:25 PM Gregory Lee <
>> >> >> grlee77 at gmail.com>
>> >> >> >>>>>>>>>>> wrote:
>> >> >> >>>>>>>>>>>
>> >> >> >>>>>>>>>>>> On Sun, Mar 3, 2019 at 4:19 AM Gopi Manohar Tatiraju <
>> >> >> >>>>>>>>>>>> deathcoderx at gmail.com> wrote:
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>> I want to work on SciPy in GSoC 2019. I started fixing
>> >> some
>> >> >> >>>>>>>>>>>>> bugs, also made some pull request. Can anyone suggest
>> me
>> >> >> how should I
>> >> >> >>>>>>>>>>>>> proceed further?
>> >> >> >>>>>>>>>>>>> _______________________________________________
>> >> >> >>>>>>>>>>>>> SciPy-Dev mailing list
>> >> >> >>>>>>>>>>>>> SciPy-Dev at python.org
>> >> >> >>>>>>>>>>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>> Hi Gopi,
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>> I think you have probably already found it, but the
>> GSOC
>> >> >> >>>>>>>>>>>> project we have already identified mentors for in
>> SciPy is
>> >> >> here:
>> >> >> >>>>>>>>>>>>
>> >> https://github.com/scipy/scipy/wiki/GSoC-2019-project-ideas
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>> If this is a project you are interested in then I would
>> >> start
>> >> >> >>>>>>>>>>>> thinking about what the FFT interface would look like
>> and
>> >> >> reviewing some of
>> >> >> >>>>>>>>>>>> the past/present issues related to fftpack / FFT in
>> SciPy.
>> >> >> Please ask
>> >> >> >>>>>>>>>>>> questions you have related to that project here so that
>> >> you
>> >> >> can clarify
>> >> >> >>>>>>>>>>>> what would be involved in writing a successful
>> >> application.
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>> If you have a different idea in mind you can propose it
>> >> here,
>> >> >> >>>>>>>>>>>> but you should do that soon, because it will be
>> necessary
>> >> to
>> >> >> determine if
>> >> >> >>>>>>>>>>>> there is interest from the SciPy team before you spend
>> >> time
>> >> >> coming up with
>> >> >> >>>>>>>>>>>> a detailed proposal. We have to determine if a project
>> >> >> sounds feasible to
>> >> >> >>>>>>>>>>>> complete for GSOC and if appropriate mentors can be
>> >> >> identified.
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>> Ultimately, you will need to submit an application to
>> >> SciPy
>> >> >> >>>>>>>>>>>> between March 25th - April 9th.
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>> Cheers,
>> >> >> >>>>>>>>>>>> Greg
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>> _______________________________________________
>> >> >> >>>>>>>>>>>> SciPy-Dev mailing list
>> >> >> >>>>>>>>>>>> SciPy-Dev at python.org
>> >> >> >>>>>>>>>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>> _______________________________________________
>> >> >> >>>>>>>>>>> SciPy-Dev mailing list
>> >> >> >>>>>>>>>>> SciPy-Dev at python.org
>> >> >> >>>>>>>>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>> >> >> >>>>>>>>>>>
>> >> >> >>>>>>>>>> _______________________________________________
>> >> >> >>>>>>>>>> SciPy-Dev mailing list
>> >> >> >>>>>>>>>> SciPy-Dev at python.org
>> >> >> >>>>>>>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>> _______________________________________________
>> >> >> >>>>>>>>> SciPy-Dev mailing list
>> >> >> >>>>>>>>> SciPy-Dev at python.org
>> >> >> >>>>>>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>> >> >> >>>>>>>>>
>> >> >> >>>>>>>> _______________________________________________
>> >> >> >>>>>>>> SciPy-Dev mailing list
>> >> >> >>>>>>>> SciPy-Dev at python.org
>> >> >> >>>>>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>> >> >> >>>>>>>>
>> >> >> >>>>>>> _______________________________________________
>> >> >> >>>>>>> SciPy-Dev mailing list
>> >> >> >>>>>>> SciPy-Dev at python.org
>> >> >> >>>>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>> >> >> >>>>>>>
>> >> >> >>>>>> _______________________________________________
>> >> >> >>>>>> SciPy-Dev mailing list
>> >> >> >>>>>> SciPy-Dev at python.org
>> >> >> >>>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>> >> >> >>>>>>
>> >> >> >>>>> _______________________________________________
>> >> >> >>>>> SciPy-Dev mailing list
>> >> >> >>>>> SciPy-Dev at python.org
>> >> >> >>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>> >> >> >>>>>
>> >> >> >>>> _______________________________________________
>> >> >> >>>> SciPy-Dev mailing list
>> >> >> >>>> SciPy-Dev at python.org
>> >> >> >>>> https://mail.python.org/mailman/listinfo/scipy-dev
>> >> >> >>>>
>> >> >> >>> _______________________________________________
>> >> >> >>> SciPy-Dev mailing list
>> >> >> >>> SciPy-Dev at python.org
>> >> >> >>> https://mail.python.org/mailman/listinfo/scipy-dev
>> >> >> >>>
>> >> >> >> _______________________________________________
>> >> >> >> SciPy-Dev mailing list
>> >> >> >> SciPy-Dev at python.org
>> >> >> >> https://mail.python.org/mailman/listinfo/scipy-dev
>> >> >> >>
>> >> >> > _______________________________________________
>> >> >> > SciPy-Dev mailing list
>> >> >> > SciPy-Dev at python.org
>> >> >> > https://mail.python.org/mailman/listinfo/scipy-dev
>> >> >> >
>> >> >> -------------- next part --------------
>> >> >> An HTML attachment was scrubbed...
>> >> >> URL: <
>> >> >>
>> >>
>> http://mail.python.org/pipermail/scipy-dev/attachments/20190330/54f38250/attachment.html
>> >> >> >
>> >> >>
>> >> >> ------------------------------
>> >> >>
>> >> >> Subject: Digest Footer
>> >> >>
>> >> >> _______________________________________________
>> >> >> SciPy-Dev mailing list
>> >> >> SciPy-Dev at python.org
>> >> >> https://mail.python.org/mailman/listinfo/scipy-dev
>> >> >>
>> >> >>
>> >> >> ------------------------------
>> >> >>
>> >> >> End of SciPy-Dev Digest, Vol 185, Issue 18
>> >> >> ******************************************
>> >> >>
>> >> > _______________________________________________
>> >> > SciPy-Dev mailing list
>> >> > SciPy-Dev at python.org
>> >> > https://mail.python.org/mailman/listinfo/scipy-dev
>> >> >
>> >> -------------- next part --------------
>> >> An HTML attachment was scrubbed...
>> >> URL: <
>> >>
>> http://mail.python.org/pipermail/scipy-dev/attachments/20190404/1a670020/attachment.html
>> >> >
>> >>
>> >> ------------------------------
>> >>
>> >> Subject: Digest Footer
>> >>
>> >> _______________________________________________
>> >> SciPy-Dev mailing list
>> >> SciPy-Dev at python.org
>> >> https://mail.python.org/mailman/listinfo/scipy-dev
>> >>
>> >>
>> >> ------------------------------
>> >>
>> >> End of SciPy-Dev Digest, Vol 186, Issue 11
>> >> ******************************************
>> >>
>> > _______________________________________________
>> > SciPy-Dev mailing list
>> > SciPy-Dev at python.org
>> > https://mail.python.org/mailman/listinfo/scipy-dev
>> >
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> http://mail.python.org/pipermail/scipy-dev/attachments/20190405/cc4019f5/attachment.html
>> >
>>
>> ------------------------------
>>
>> Subject: Digest Footer
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
>> ------------------------------
>>
>> End of SciPy-Dev Digest, Vol 186, Issue 15
>> ******************************************
>>
> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20190409/0214c149/attachment-0001.html>