From ralf.gommers at gmail.com Sun Sep 2 23:58:12 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sun, 2 Sep 2018 20:58:12 -0700 Subject: [SciPy-Dev] General discussion on parallelisation In-Reply-To: <20180109133448.GA832752@phare.normalesup.org> References: <20180109133448.GA832752@phare.normalesup.org> Message-ID: reviving an old thread because there's movement on the PR ( https://github.com/scipy/scipy/pull/8259) and would be nice to get that merged. On Wed, Jan 10, 2018 at 2:34 AM, Gael Varoquaux < gael.varoquaux at normalesup.org> wrote: > > 5. joblib.Parallel doesn't have a map method (desirable to allow 3) so a > small > > joblib has a custom backend framework that can be used for such purpose > (if I understnad you well): > > https://pythonhosted.org/joblib/parallel.html#custom-backend-api-experimental Updated link (status is still experimental): https://joblib.readthedocs.io/en/latest/parallel.html#custom-backend-api-experimental > > There are currently a Yarn and a dask.distributed backend that are > getting better and better. > There's also this JoblibPool that can be taken over: https://github.com/adrn/schwimmbad/blob/master/schwimmbad/jl.py#L14 Seems simpler than a backend still tagged experimental. > > 6. joblib.Parallel creates/destroys a multiprocessing.Pool each time the > > Parallel object is `__call__`ed. This leads to significant overhead. One > can > > use the Parallel object with a context manager, which allows reuse of > the Pool, > > but I don't think that's do-able in the context of using the > > DifferentialEvolutionSolver (DES) object as an iterator: > > This is evolving. However, the reason behind this is that Pool get > corrupted and lead to deadlock. Olivier Grisel and Thomas Moreau are > working on fixing this in the Python standard library (first PR merged > recently)! > Anyone know the status of this? And can this issue be avoided by the new loky backend to joblib? Ralf > One of the vision of joblib is to provide very light mid-layer that can > connect to multiprocessing and threading (though we are considering > switching to concurrent.futures) as well as other backends. Hopefully > this common language makes it easier to do things like embedding dask in > numerical algorithms without a hard dependencies (yes we are working with > the dask team on this). > > Ga?l > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Sep 3 02:36:30 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sun, 2 Sep 2018 23:36:30 -0700 Subject: [SciPy-Dev] grants for sparse arrays & statsmodels Message-ID: Hi all, This is really cool, so I'm advertising it a bit more widely: https://numfocus.org/blog/summer-2018-open-source-development-grants Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Mon Sep 3 05:22:17 2018 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 3 Sep 2018 11:22:17 +0200 Subject: [SciPy-Dev] General discussion on parallelisation In-Reply-To: References: <20180109133448.GA832752@phare.normalesup.org> Message-ID: <20180903092217.afvqf26bblmyd67s@phare.normalesup.org> On Sun, Sep 02, 2018 at 08:58:12PM -0700, Ralf Gommers wrote: > joblib has a custom backend framework that can be used for such purpose > (if I understnad you well): > https://pythonhosted.org/joblib/parallel.html# > custom-backend-api-experimental > Updated link (status is still experimental): https://joblib.readthedocs.io/en/latest/parallel.html#custom-backend-api-experimental Still experimental, but less and less :). We are using this in anger these days. > There's also this JoblibPool that can be taken over: https://github.com/adrn/schwimmbad/blob/master/schwimmbad/jl.py#L14 > Seems simpler than a backend still tagged experimental. Well, we have a fairly stringent definition of experimental. This feature is no longer very experimental. > This is evolving. However, the reason behind this is that Pool get > corrupted and lead to deadlock. Olivier Grisel and Thomas Moreau are > working on fixing this in the Python standard library (first PR merged > recently)! > Anyone know the status of this? And can this issue be avoided by the new loky > backend to joblib? This is merged and released since a while. Loky is now used in anger. So far, I think that people are happy with it. In particular it is more robust than multiprocessing.Pool (specifically, robust to segfault), and the improvements have been contributed upstream to Python concurrent.futures's process pool executor (available in Python 3.7). Joblib has lately been getting a lot of improvements to make it more robust and scaleable [1]. It will still have some overhead, due to pickling. Pickling speed should be solved by coordinating upstream changes in Python with implementations in numpy. Olivier Grisel has been coordinating with Python for this. I believe that PEP 574 [2] is related to these efforts. The specific challenges are to enable fast code paths in cloud pickle, which is necessary to pickle arbitrary objects and code. While simpler multiprocessing-based code will sometimes give less overhead compared to joblib, it will probably be brittle. I think that the best way to move forward from here would be to do some prototyping and experimentation. Ga?l [1] Joblib changelog: https://joblib.readthedocs.io/en/latest/developing.html#latest-changes [2] Pickling improvement PEP: https://www.python.org/dev/peps/pep-0574/ From andyfaff at gmail.com Mon Sep 3 05:30:22 2018 From: andyfaff at gmail.com (Andrew Nelson) Date: Mon, 3 Sep 2018 19:30:22 +1000 Subject: [SciPy-Dev] General discussion on parallelisation In-Reply-To: <20180903092217.afvqf26bblmyd67s@phare.normalesup.org> References: <20180109133448.GA832752@phare.normalesup.org> <20180903092217.afvqf26bblmyd67s@phare.normalesup.org> Message-ID: SNIP > In particular it is more robust than multiprocessing.Pool (specifically, robust to segfault), and the improvements have been contributed upstream to Python concurrent.futures's process pool executor (available in Python 3.7). SNIP > While simpler multiprocessing-based code will sometimes give less overhead compared to joblib, it will probably be brittle. Is there anything I can read to learn a bit more about this? Is it mainly related to how easy things are to pickle? -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Mon Sep 3 05:50:21 2018 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 3 Sep 2018 11:50:21 +0200 Subject: [SciPy-Dev] General discussion on parallelisation In-Reply-To: References: <20180109133448.GA832752@phare.normalesup.org> <20180903092217.afvqf26bblmyd67s@phare.normalesup.org> Message-ID: <20180903095021.5if653k3vc2poucz@phare.normalesup.org> On Mon, Sep 03, 2018 at 07:30:22PM +1000, Andrew Nelson wrote: > SNIP > > In particular it is more robust than multiprocessing.Pool (specifically, > robust to segfault), and the improvements have been contributed upstream to > Python concurrent.futures's process pool executor (available in Python 3.7). > SNIP > > While simpler multiprocessing-based code will sometimes give less overhead > compared to joblib, it will probably be brittle. > Is there anything I can read to learn a bit more about this? Is it mainly > related to how easy things are to pickle? The major problem are related to state inside multiprocessing.Pool that can be broken when child processes go astray. As a result, multiprocessing.Pool often gets into deadlock (any heavy user of multiprocessing has probably experienced this). See for instance https://tommoral.github.io/talks/pyparis17/#37 Ga?l From andyfaff at gmail.com Mon Sep 3 06:34:02 2018 From: andyfaff at gmail.com (Andrew Nelson) Date: Mon, 3 Sep 2018 20:34:02 +1000 Subject: [SciPy-Dev] General discussion on parallelisation In-Reply-To: References: <20180109133448.GA832752@phare.normalesup.org> <20180903092217.afvqf26bblmyd67s@phare.normalesup.org> Message-ID: Ok, I managed to search and AFAICT the main issue is that if a child process terminates then the master process can hang waiting for it to return (https://bugs.python.org/issue22393). > This is merged and released since a while. Loky is now used in anger. So far, I think that people are happy with it. In particular it is more robust than multiprocessing.Pool (specifically, robust to segfault), and the improvements have been contributed upstream to Python concurrent.futures's process pool executor (available in Python 3.7). So, reading the `concurrent.futures.ProcessPoolExecutor` documentation indicates that it is resistant to this issue. concurrent.futures is available in Python3, but wasn't ported to 2.7. On Mon, 3 Sep 2018 at 19:30, Andrew Nelson wrote: > SNIP > > In particular it is more robust than multiprocessing.Pool (specifically, > robust to segfault), and the improvements have been contributed upstream to > Python concurrent.futures's process pool executor (available in Python 3.7). > > SNIP > > While simpler multiprocessing-based code will sometimes give less > overhead compared to joblib, it will probably be brittle. > > Is there anything I can read to learn a bit more about this? Is it mainly > related to how easy things are to pickle? > > -- _____________________________________ Dr. Andrew Nelson _____________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From josua.sassen at adscheid.de Mon Sep 3 06:51:52 2018 From: josua.sassen at adscheid.de (Josua Sassen) Date: Mon, 3 Sep 2018 12:51:52 +0200 (CEST) Subject: [SciPy-Dev] Changed Rbf API - PR #9215 Message-ID: <2085325596.423153.1535971912637@email.1und1.de> Hello everybody, in PR #9215 I modified the API of scipy.interpolate.Rbf. I removed the seperate arguments for different input dimensions and rather introduced one array for the data points and one for the values. This matches the APIs of other interpolators such as griddata or LinearNDInterpolator. Is this an acceptable change? Best, Josua From ralf.gommers at gmail.com Mon Sep 3 13:16:51 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 3 Sep 2018 10:16:51 -0700 Subject: [SciPy-Dev] General discussion on parallelisation In-Reply-To: References: <20180109133448.GA832752@phare.normalesup.org> <20180903092217.afvqf26bblmyd67s@phare.normalesup.org> Message-ID: On Mon, Sep 3, 2018 at 3:34 AM Andrew Nelson wrote: > Ok, I managed to search and AFAICT the main issue is that if > a child process terminates then the master process can hang waiting for it > to return (https://bugs.python.org/issue22393). > > > This is merged and released since a while. Loky is now used in anger. So > far, I think that people are happy with it. In particular it is more robust > than multiprocessing.Pool (specifically, robust to segfault), and the > improvements have been contributed upstream to Python concurrent.futures's > process pool executor (available in Python 3.7). > Thanks for the details Gael! So, reading the `concurrent.futures.ProcessPoolExecutor` documentation > indicates that it is resistant to this issue. concurrent.futures is > available in Python3, but wasn't ported to 2.7. > The PR now uses multiprocessing on Python 2.7 and concurrent.futures on 3.x - this seems fine to me. We're not supporting 2.7 for that much longer, so the code can be simplified a bit when we drop 2.7 Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Sep 3 13:31:48 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 3 Sep 2018 10:31:48 -0700 Subject: [SciPy-Dev] Changed Rbf API - PR #9215 In-Reply-To: <2085325596.423153.1535971912637@email.1und1.de> References: <2085325596.423153.1535971912637@email.1und1.de> Message-ID: On Mon, Sep 3, 2018 at 3:52 AM Josua Sassen wrote: > Hello everybody, > > in PR #9215 I modified the API of scipy.interpolate.Rbf. I removed the > seperate arguments for different input dimensions and rather introduced one > array for the data points and one for the values. This matches the APIs of > other interpolators such as griddata or LinearNDInterpolator. > Is this an acceptable change? > Hi Joshua, as I commented on the PR, we have to remain backwards compatible here. Matching signatures would be nice, but they're not worth breaking users' code for. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Mon Sep 3 16:05:36 2018 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 3 Sep 2018 22:05:36 +0200 Subject: [SciPy-Dev] General discussion on parallelisation In-Reply-To: References: <20180109133448.GA832752@phare.normalesup.org> <20180903092217.afvqf26bblmyd67s@phare.normalesup.org> Message-ID: <20180903200536.3tbzvw5ogxitw5rm@phare.normalesup.org> On Mon, Sep 03, 2018 at 10:16:51AM -0700, Ralf Gommers wrote: > So, reading the `concurrent.futures.ProcessPoolExecutor` documentation > indicates that it is resistant to this issue. concurrent.futures is > available in Python3, but wasn't ported to 2.7. > The PR now uses multiprocessing on Python 2.7 and concurrent.futures on 3.x - > this seems fine to me. We're not supporting 2.7 for that much longer, so the > code can be simplified a bit when we drop 2.7 OK. I can think of two quite use features that joblib add: * Support of dask.distributed as a backend, to distribute code across computers. * Fallback to threading in case of nested parallelism, and in case of two levels of nesting, fall back to sequential to avoid over committing. Ga?l From ralf.gommers at gmail.com Mon Sep 3 16:10:50 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 3 Sep 2018 13:10:50 -0700 Subject: [SciPy-Dev] General discussion on parallelisation In-Reply-To: <20180903200536.3tbzvw5ogxitw5rm@phare.normalesup.org> References: <20180109133448.GA832752@phare.normalesup.org> <20180903092217.afvqf26bblmyd67s@phare.normalesup.org> <20180903200536.3tbzvw5ogxitw5rm@phare.normalesup.org> Message-ID: On Mon, Sep 3, 2018 at 1:05 PM Gael Varoquaux wrote: > On Mon, Sep 03, 2018 at 10:16:51AM -0700, Ralf Gommers wrote: > > So, reading the `concurrent.futures.ProcessPoolExecutor` > documentation > > indicates that it is resistant to this issue. concurrent.futures is > > available in Python3, but wasn't ported to 2.7. > > > > The PR now uses multiprocessing on Python 2.7 and concurrent.futures on > 3.x - > > this seems fine to me. We're not supporting 2.7 for that much longer, so > the > > code can be simplified a bit when we drop 2.7 > > OK. I can think of two quite use features that joblib add: > > * Support of dask.distributed as a backend, to distribute code across > computers. > > * Fallback to threading in case of nested parallelism, and in case of two > levels of nesting, fall back to sequential to avoid over committing. > Those are both quite useful. How would you add an API for those to SciPy functions (if that's necessary - I assume threading fallback is automatic)? Right now we have a single keyword `workers=1`, which behaves like scikit-learn's n_jobs=1 (# of CPUs), and also accept objects with a map() method like multiprocessing.Pool Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Mon Sep 3 16:16:57 2018 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 3 Sep 2018 22:16:57 +0200 Subject: [SciPy-Dev] General discussion on parallelisation In-Reply-To: References: <20180109133448.GA832752@phare.normalesup.org> <20180903092217.afvqf26bblmyd67s@phare.normalesup.org> <20180903200536.3tbzvw5ogxitw5rm@phare.normalesup.org> Message-ID: <20180903201657.epfarwwmkhusqapa@phare.normalesup.org> On Mon, Sep 03, 2018 at 01:10:50PM -0700, Ralf Gommers wrote: > Right now we have a single keyword `workers=1`, which behaves like > scikit-learn's n_jobs=1 (# of CPUs), and also accept objects with a map() > method like multiprocessing.Pool I think that we wouldn't mind adding a "map" to a Parallel object. Though it feels quite strange, as the Parallel object is meant to be higher-level abstract than the Pool. Ga?l From andyfaff at gmail.com Mon Sep 3 16:26:51 2018 From: andyfaff at gmail.com (Andrew Nelson) Date: Tue, 4 Sep 2018 06:26:51 +1000 Subject: [SciPy-Dev] General discussion on parallelisation In-Reply-To: <20180903201657.epfarwwmkhusqapa@phare.normalesup.org> References: <20180109133448.GA832752@phare.normalesup.org> <20180903092217.afvqf26bblmyd67s@phare.normalesup.org> <20180903200536.3tbzvw5ogxitw5rm@phare.normalesup.org> <20180903201657.epfarwwmkhusqapa@phare.normalesup.org> Message-ID: > > > Right now we have a single keyword `workers=1`, which behaves like > > scikit-learn's n_jobs=1 (# of CPUs), and also accept objects with a map() > > method like multiprocessing.Pool > > I think that we wouldn't mind adding a "map" to a Parallel object. Though > it feels quite strange, as the Parallel object is meant to be > higher-level abstract than the Pool. > The design also allows for things like `mpi4py.futures.MPIPoolExecutor` to be supplied. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Sep 3 16:44:44 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 3 Sep 2018 13:44:44 -0700 Subject: [SciPy-Dev] General discussion on parallelisation In-Reply-To: References: <20180109133448.GA832752@phare.normalesup.org> <20180903092217.afvqf26bblmyd67s@phare.normalesup.org> <20180903200536.3tbzvw5ogxitw5rm@phare.normalesup.org> Message-ID: On Mon, Sep 3, 2018 at 1:10 PM Ralf Gommers wrote: > > > On Mon, Sep 3, 2018 at 1:05 PM Gael Varoquaux < > gael.varoquaux at normalesup.org> wrote: > >> On Mon, Sep 03, 2018 at 10:16:51AM -0700, Ralf Gommers wrote: >> > So, reading the `concurrent.futures.ProcessPoolExecutor` >> documentation >> > indicates that it is resistant to this issue. concurrent.futures is >> > available in Python3, but wasn't ported to 2.7. >> >> >> > The PR now uses multiprocessing on Python 2.7 and concurrent.futures on >> 3.x - >> > this seems fine to me. We're not supporting 2.7 for that much longer, >> so the >> > code can be simplified a bit when we drop 2.7 >> >> OK. I can think of two quite use features that joblib add: >> >> * Support of dask.distributed as a backend, to distribute code across >> computers. >> >> * Fallback to threading in case of nested parallelism, and in case of two >> levels of nesting, fall back to sequential to avoid over committing. >> > > Those are both quite useful. How would you add an API for those to SciPy > functions (if that's necessary - I assume threading fallback is automatic)? > Okay found the answer I think (from http://matthewrocklin.com/blog/work/2017/02/07/dask-sklearn-simple): from joblib import parallel_backend with parallel_backend('dask.distributed', scheduler_host='scheduler-address:8786'): some_scipy_func(..., workers=N) That would be quite nice. If we vendor joblib then the import can be: from scipy import parallel_backend Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikofski at berkeley.edu Tue Sep 4 12:21:16 2018 From: mikofski at berkeley.edu (Mark Alexander Mikofski) Date: Tue, 4 Sep 2018 09:21:16 -0700 Subject: [SciPy-Dev] grants for sparse arrays & statsmodels In-Reply-To: References: Message-ID: Very cool. I re-posted to Twitter and LinkedIn, and I'll mention it at a talk I'm giving at next week at the Data-Analyst Tools Series (formerly The Hacker Within) at UC Berkeley. BTW: the DATS talk is on using GitHub in OSS, so any pointers are greatly appreciated. Here's a link to last years talk - https://bids.github.io/dats/posts/2017-10-04-github-oss-f17.html - although this year I plan to focus more on nuts and bolts of PR[s], using Git for version control, and helping them setup GitHub static pages. On Sun, Sep 2, 2018 at 11:37 PM Ralf Gommers wrote: > Hi all, > > This is really cool, so I'm advertising it a bit more widely: > https://numfocus.org/blog/summer-2018-open-source-development-grants > > Cheers, > Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -- Mark Mikofski, PhD (2005) *Fiat Lux* -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Tue Sep 4 12:27:05 2018 From: toddrjen at gmail.com (Todd) Date: Tue, 4 Sep 2018 12:27:05 -0400 Subject: [SciPy-Dev] Pull request 9190 - Support specifying axes for fftconvolve Message-ID: Allowing specifying the axes for convolve and fftconvolve was discussed in issue #3525 [1]. I have implemented this feature for fftconvolve [2]. It allows the user the specify one or more axes as the axes over which the convolution will be carried. Other axes will be broadcast together, although both arrays must have the same number of dimensions (it will not automatically add implicit length-1 dimensions to make the number of dimensions match). As I side-effect of this, I also implemented a new pair of helper functions for n-dimensional transforms that calculates the shape and axes arguments. This is shared between fftn, ifftn, dctn, idctn, dstn, idstn, and fftconvolve. It also adds some additional benefits. These functions now support any integer array-like as their shape and axes arguments instead of just tuples. The functions now consistently interpret individual ints as length-1 arrays of ints. They also support negative values in the shape argument, which is interpreted as just using the length of that particular dimension from the original array. I have also implemented unit tests for everything, and updated the relevant function signatures. This feature is a prerequisite for implementing the overlap-add method [2] for convolution, which has enormous performance benefits for two arrays of very different sizes (one is larger by a factor of more than a few times) where both arrays are large (both are more than a few hundred samples long). [1] https://github.com/scipy/scipy/issues/3525 [2] https://github.com/scipy/scipy/pull/9190 [3] https://en.wikipedia.org/wiki/Overlap%E2%80%93add_method -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfw48 at cam.ac.uk Wed Sep 5 08:43:25 2018 From: jfw48 at cam.ac.uk (Joshua Wharton) Date: Wed, 5 Sep 2018 12:43:25 +0000 Subject: [SciPy-Dev] Modifying interpolate.pade function In-Reply-To: References: Message-ID: Hi all, This is my first time contributing to scipy, so apologies if I've messed up somehow. Here is a link to the PR https://github.com/scipy/scipy/pull/9211) Currently interpolate.pade only takes the order of the denominator with the order of the numerator assumed from the order of the inputted array . Added an optional parameter (following standard naming convention of the pade function) which specifies the order of the numerator. This is useful if the order of the inputted array returns a higher order (and computationally more expensive) numerator than needed. This change does not affect any currently written code, I cannot see any obvious performance cost. Do you think this modification is acceptable for scipy? Also I would like to say that the docs for contributing are wonderful. Thank you, Joshua -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Sep 5 22:20:31 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 5 Sep 2018 19:20:31 -0700 Subject: [SciPy-Dev] Pull request 9190 - Support specifying axes for fftconvolve In-Reply-To: References: Message-ID: On Tue, Sep 4, 2018 at 9:27 AM Todd wrote: > Allowing specifying the axes for convolve and fftconvolve was discussed in > issue #3525 [1]. I have implemented this feature for fftconvolve [2]. It > allows the user the specify one or more axes as the axes over which the > convolution will be carried. Other axes will be broadcast together, > although both arrays must have the same number of dimensions (it will not > automatically add implicit length-1 dimensions to make the number of > dimensions match). > > As I side-effect of this, I also implemented a new pair of helper > functions for n-dimensional transforms that calculates the shape and axes > arguments. This is shared between fftn, ifftn, dctn, idctn, dstn, idstn, > and fftconvolve. It also adds some additional benefits. These functions > now support any integer array-like as their shape and axes arguments > instead of just tuples. The functions now consistently interpret > individual ints as length-1 arrays of ints. They also support negative > values in the shape argument, which is interpreted as just using the length > of that particular dimension from the original array. > > I have also implemented unit tests for everything, and updated the > relevant function signatures. > > This feature is a prerequisite for implementing the overlap-add method [2] > for convolution, which has enormous performance benefits for two arrays of > very different sizes (one is larger by a factor of more than a few times) > where both arrays are large (both are more than a few hundred samples long). > That sounds really useful, thanks Todd. Cheers, Ralf > [1] https://github.com/scipy/scipy/issues/3525 > [2] https://github.com/scipy/scipy/pull/9190 > [3] https://en.wikipedia.org/wiki/Overlap%E2%80%93add_method > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Sep 5 22:37:35 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 5 Sep 2018 19:37:35 -0700 Subject: [SciPy-Dev] grants for sparse arrays & statsmodels In-Reply-To: References: Message-ID: On Sun, Sep 2, 2018 at 11:36 PM Ralf Gommers wrote: > Hi all, > > This is really cool, so I'm advertising it a bit more widely: > https://numfocus.org/blog/summer-2018-open-source-development-grants > I'll also add some context: this is one of the ways that money donated to NumFOCUS (as a general donation/sponsorship, not earmarked for a specific project) directly flows to and benefits sponsored/affiliated projects. SciPy is an affliated project. I hope we'll decide to become a sponsored project at some point, but that's a discussion for another day. NumFOCUS just launched a fundraising sprint from now until Sept. 25th: https://numfocus.org/blog/numfocus-sustainer-weeks; thought I'd advertise that too:) Cheers, Ralf > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Sep 5 23:49:56 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 5 Sep 2018 20:49:56 -0700 Subject: [SciPy-Dev] Modifying interpolate.pade function In-Reply-To: References: Message-ID: On Wed, Sep 5, 2018 at 5:43 AM Joshua Wharton wrote: > Hi all, > > > > This is my first time contributing to scipy, so apologies if I?ve messed > up somehow. > > Here is a link to the PR https://github.com/scipy/scipy/pull/9211) > That looks good. I didn't review in detail, but from a process/git/etc perspective it looks like you did everything right. Currently interpolate.pade only takes the order of the denominator with the > order of the numerator assumed from the order of the inputted array . Added > an optional parameter (following standard naming convention of the pade > function) which specifies the order of the numerator. This is useful if the > order of the inputted array returns a higher order (and computationally > more expensive) numerator than needed. This change does not affect any > currently written code, I cannot see any obvious performance cost. Do you > think this modification is acceptable for scipy? > > Also I would like to say that the docs for contributing are wonderful. > That's great to hear:) Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From email.the.odonnells at gmail.com Thu Sep 6 05:28:49 2018 From: email.the.odonnells at gmail.com (Kane & Anna O'Donnell) Date: Thu, 6 Sep 2018 21:28:49 +1200 Subject: [SciPy-Dev] Improving spatial.distance code and performance Message-ID: Hi all, TLDR; I'm wondering about a) porting the spatial.distance code to Cython, and b) adding some performance optimizations, and I'd like the dev community's input/feedback. For context (though I don't really need you to read these to give the feedback I'm looking for), - original issue/proposal: https://github.com/scipy/scipy/issues/9205 - PR: https://github.com/scipy/scipy/pull/9218 Before submitting the PR, I naively thought it was going to be nice Cython or similar. Turns out it's some pretty old code, that I found pretty hard to wrap my head around and understand. I eventually figured it out after spending ages debugging a nastily hidden 'bug', and demonstrated the performance optimizations, but it prompted the discussion about whether it was best to port everything to Cython first. *Existing stuff to Cython* Doing so shouldn't be too hard, and it shouldn't change any functionality, except to replace the distance functions with their Cython ones (instead of the current approach, where the distance functions are actually numpy things, and there's not supported access to the underlying C stuff). A few possible 'bugs' (as above) should hopefully become non-issues too. So, it'd be a win for performance (e.g. all the distance functions will be much faster), and code quality, and future maintainability and development. However, things to think about: - should I just replace like-for-like, or consider some nicer OOP stuff like e.g. sklearn's approach (which is already Cython)? https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/neighbors/dist_metrics.pyx (I'm guessing the reason they rolled their own was because they found scipy lacking, as above.) In fact, we could largely just copy that file over. Not sure about the interplay between scipy and scikit learn though. - what's the best way to structure a PR? One metric at a time, or the whole caboodle? *Adding performance optimizations and future functionality* As per the links, this was initially about providing a nifty performance hack. It should still be pretty easy to implement. Personally, I think it makes sense to implement after the Cythonization - unless the community are against that. However, there are other possibilities: - using 'indices' within calculations. E.g. when doing pdist, it might pay to use a tree of some description. I also proposed another 'index' to optimize the 'bail early' approach further (which would, I think, actually work well with trees too). This would involve more API changes, possibly significant. - using approximate searches (e.g. Faiss). My understanding is that depending on other libraries probably isn't really an option, so I'm not sure what means. - maybe other cool approaches like https://github.com/droyed/eucl_dist - providing a way to 'tune' distance computations to be optimal to your particular dataset and constraints (e.g. my 'bail early' optimization might be a lot faster or a bit slower, depending on your data ... or you might be OK with approximate matching with a 'low' error rate, etc.) I guess what I'd like to see is a single place where users can get access to everything related to distance metrics and their uses, including all sorts of optimizations etc. (possibly for different hardware, and possibly distributed). To do that well is a pretty big undertaking, and I don't know whether it's suited to scipy - e.g. maybe scipy doesn't really care about distance stuff, or only wants to stick with 'simple' distance metric cases (e.g. a few thousand vectors, etc.). So, maybe it'd be better to start a completely new python package - which would probably be a lot easier to develop as I'd have a lot more flexibility (e.g. to depend on other packages, and not have to worry about breaking the scipy API etc.). On the other hand (as discussed in the latest comment on the PR), that might not be best - it might never get used/maintained etc. So, what does the community think is the best approach? I've got too little context of what scipy is and what it's aiming for, and I don't want to head off on the wrong tack. Comments on any of the other implied questions would also be appreciated. Thanks, kodonnell -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Fri Sep 7 01:49:23 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 6 Sep 2018 22:49:23 -0700 Subject: [SciPy-Dev] General discussion on parallelisation In-Reply-To: References: <20180109133448.GA832752@phare.normalesup.org> <20180903092217.afvqf26bblmyd67s@phare.normalesup.org> <20180903200536.3tbzvw5ogxitw5rm@phare.normalesup.org> <20180903201657.epfarwwmkhusqapa@phare.normalesup.org> Message-ID: On Mon, Sep 3, 2018 at 1:27 PM Andrew Nelson wrote: > > Right now we have a single keyword `workers=1`, which behaves like >> > scikit-learn's n_jobs=1 (# of CPUs), and also accept objects with a >> map() >> > method like multiprocessing.Pool >> >> I think that we wouldn't mind adding a "map" to a Parallel object. Though >> it feels quite strange, as the Parallel object is meant to be >> higher-level abstract than the Pool. >> > I agree that that's strange, and it looks like we don't need it. > > The design also allows for things like `mpi4py.futures.MPIPoolExecutor` to > be supplied. > Update for everyone: at the moment the PR is updated to use concurrent.future.ProcessPoolExecutor on Python 3 by default, and multiprocessing on Python 2.7. The design is compatible with joblib, however vendoring joblib and using it for the workers=int case is left to a future PR. Unless there's more comments/thoughts, I plan to hit the green button after a final review/benchmark. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Fri Sep 7 02:11:31 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 6 Sep 2018 23:11:31 -0700 Subject: [SciPy-Dev] 1D parabolic PDE solver In-Reply-To: References: Message-ID: On Mon, Aug 13, 2018 at 2:28 PM Saixoz wrote: > Dear all, > > > > A recent task that I had to perform involved rewriting a large number of > scripts and files from MATLAB to Python. In the process I noticed that > Scipy lacks a PDE solver. When I tried to find alternatives, the options > found were all unusable for various reasons. > > > > I wrote a solver which solves slab symmetric problems with a similar PDE > form to that which MATLAB?s pdepe solves, namely c(x,u,t,du/dx)(du/dt) = > (d/dx)(f(x,u,t,du/dx) + s(x,u,t,du/dx) using the method of lines approach, > finite differences, and the BDF method provided by solve_ivp. > > > > I was wondering whether there would be some interest in adding the solver > under scipy.integrate, to begin building a part of a collection. I feel > that for very simple problems such as the ones that my solver can solve, > users shouldn?t need to turn to overly complex packages. > Hi Nicolas, thanks for the thoughtful email. Sorry about the lack of responses - this reflects not that there's no interest, but that we're short on maintainer power for SciPy. Given this, and especially that we don't have a dedicated maintainer for scipy.integrate, I would suggest to develop your PDE solver as an independent package. If it becomes popular/mature, we may consider integration into SciPy again. Cheers, Ralf > > If there is interest, I would need to clean up the code and add tests and > stability checks to it, help/advice would be appreciated. I believe BDF > already has stability in the time grid, so I would only need to check > stability in space grid? Can someone confirm? I?d also like to move a large > chunk of the code to Cython since I wrote it in native Python and used > Numba to speed it up, which still makes it rather slow. I?ve worked in C > and Python before but never in Cython, could someone provide me with a link > to get started? > > > > As far as legal issues go, there should be no concern ? I explicitly > avoided looking at and using the MATLAB source code as a reference since I > wanted to leave publishing the solver under Scipy an option. I did use > their solver to check some of the solutions that my solver produced though. > The solver was written while working for the ETH Zurich, where my > supervisor responded very enthusiastically to the possibility of providing > the solver to the public under Scipy, so there should be no issues there > either. > > > > All the best, > > Nicolas > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Fri Sep 7 02:28:27 2018 From: andyfaff at gmail.com (Andrew Nelson) Date: Fri, 7 Sep 2018 16:28:27 +1000 Subject: [SciPy-Dev] CI issues Message-ID: Hi all, there's currently some CI issues with various entries in the appveyor/travis test matrix failing. It's the same entries everytime, so I'm wondering if it's related to 4e743d57bd53f22ae5d30a567df3b565badac890 ? The error messages on appveyor and travis are similar, it looks like various imports aren't working properly. For an example see https://travis-ci.org/scipy/scipy/jobs/425548905. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri Sep 7 04:10:45 2018 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 07 Sep 2018 10:10:45 +0200 Subject: [SciPy-Dev] CI issues In-Reply-To: References: Message-ID: <99f2b67d8033d8286110e179df0c9bf174c4bcd8.camel@iki.fi> pe, 2018-09-07 kello 16:28 +1000, Andrew Nelson kirjoitti: > Hi all, > there's currently some CI issues with various entries in the > appveyor/travis test matrix failing. It's the same entries everytime, > so > I'm wondering if it's related to > 4e743d57bd53f22ae5d30a567df3b565badac890 ? > The error messages on appveyor and travis are similar, it looks like > various imports aren't working properly. > For an example see https://travis-ci.org/scipy/scipy/jobs/425548905. It probably has to do with pytest 3.8 release. Numpy also had problems: https://github.com/numpy/numpy/pull/11896 From andyfaff at gmail.com Fri Sep 7 05:13:11 2018 From: andyfaff at gmail.com (Andrew Nelson) Date: Fri, 7 Sep 2018 19:13:11 +1000 Subject: [SciPy-Dev] CI issues In-Reply-To: <99f2b67d8033d8286110e179df0c9bf174c4bcd8.camel@iki.fi> References: <99f2b67d8033d8286110e179df0c9bf174c4bcd8.camel@iki.fi> Message-ID: > > It probably has to do with pytest 3.8 release. Numpy also had problems: > https://github.com/numpy/numpy/pull/11896 This seems to be correct. I pinned the version to 3.7 in .travis.yml and things ran smoothly. I don't see how it's related to numpy/numpy#11896 though, that's talking about warnings, our errors are due to import problems. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri Sep 7 05:25:59 2018 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 07 Sep 2018 11:25:59 +0200 Subject: [SciPy-Dev] CI issues In-Reply-To: References: <99f2b67d8033d8286110e179df0c9bf174c4bcd8.camel@iki.fi> Message-ID: pe, 2018-09-07 kello 19:13 +1000, Andrew Nelson kirjoitti: > > > > It probably has to do with pytest 3.8 release. Numpy also had > > problems: > > https://github.com/numpy/numpy/pull/11896 > > > This seems to be correct. I pinned the version to 3.7 in .travis.yml > and > things ran smoothly. I don't see how it's related to > numpy/numpy#11896 > though, that's talking about warnings, our errors are due to import > problems. The import problem appears also due to a warning converted to error. From tyler.je.reddy at gmail.com Fri Sep 7 09:23:39 2018 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Fri, 7 Sep 2018 06:23:39 -0700 Subject: [SciPy-Dev] CI issues In-Reply-To: References: <99f2b67d8033d8286110e179df0c9bf174c4bcd8.camel@iki.fi> Message-ID: Also, codecov acknowledged receipt of our support email re: recent CI issues ( https://github.com/scipy/scipy/pull/9186#issuecomment-419225782 ) -- I'll relay their response when I get it. On Fri, 7 Sep 2018 at 02:26, Pauli Virtanen wrote: > pe, 2018-09-07 kello 19:13 +1000, Andrew Nelson kirjoitti: > > > > > > It probably has to do with pytest 3.8 release. Numpy also had > > > problems: > > > https://github.com/numpy/numpy/pull/11896 > > > > > > This seems to be correct. I pinned the version to 3.7 in .travis.yml > > and > > things ran smoothly. I don't see how it's related to > > numpy/numpy#11896 > > though, that's talking about warnings, our errors are due to import > > problems. > > The import problem appears also due to a warning converted to error. > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From larson.eric.d at gmail.com Fri Sep 7 10:00:52 2018 From: larson.eric.d at gmail.com (Eric Larson) Date: Fri, 7 Sep 2018 10:00:52 -0400 Subject: [SciPy-Dev] CI issues In-Reply-To: References: <99f2b67d8033d8286110e179df0c9bf174c4bcd8.camel@iki.fi> Message-ID: I am working on it now by getting rid of the warnings. I've gotten this warning about `qhull` before so it would be nice to fix it for good anyway. Eric On Fri, Sep 7, 2018 at 9:24 AM Tyler Reddy wrote: > Also, codecov acknowledged receipt of our support email re: recent CI > issues ( https://github.com/scipy/scipy/pull/9186#issuecomment-419225782 > ) -- I'll relay > their response when I get it. > > On Fri, 7 Sep 2018 at 02:26, Pauli Virtanen wrote: > >> pe, 2018-09-07 kello 19:13 +1000, Andrew Nelson kirjoitti: >> > > >> > > It probably has to do with pytest 3.8 release. Numpy also had >> > > problems: >> > > https://github.com/numpy/numpy/pull/11896 >> > >> > >> > This seems to be correct. I pinned the version to 3.7 in .travis.yml >> > and >> > things ran smoothly. I don't see how it's related to >> > numpy/numpy#11896 >> > though, that's talking about warnings, our errors are due to import >> > problems. >> >> The import problem appears also due to a warning converted to error. >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phillip.m.feldman at gmail.com Sat Sep 8 18:03:54 2018 From: phillip.m.feldman at gmail.com (Phillip Feldman) Date: Sat, 8 Sep 2018 15:03:54 -0700 Subject: [SciPy-Dev] General discussion on parallelisation In-Reply-To: <20180903092217.afvqf26bblmyd67s@phare.normalesup.org> References: <20180109133448.GA832752@phare.normalesup.org> <20180903092217.afvqf26bblmyd67s@phare.normalesup.org> Message-ID: I don't understand what is meant by the phase "used in anger". Phillip On Mon, Sep 3, 2018 at 2:22 AM Gael Varoquaux wrote: > On Sun, Sep 02, 2018 at 08:58:12PM -0700, Ralf Gommers wrote: > > joblib has a custom backend framework that can be used for such > purpose > > (if I understnad you well): > > https://pythonhosted.org/joblib/parallel.html# > > custom-backend-api-experimental > > > > Updated link (status is still experimental): > https://joblib.readthedocs.io/en/latest/parallel.html#custom-backend-api-experimental > > Still experimental, but less and less :). We are using this in anger > these days. > > > There's also this JoblibPool that can be taken over: > https://github.com/adrn/schwimmbad/blob/master/schwimmbad/jl.py#L14 > > Seems simpler than a backend still tagged experimental. > > Well, we have a fairly stringent definition of experimental. This feature > is no longer very experimental. > > > This is evolving. However, the reason behind this is that Pool get > > corrupted and lead to deadlock. Olivier Grisel and Thomas Moreau are > > working on fixing this in the Python standard library (first PR > merged > > recently)! > > > Anyone know the status of this? And can this issue be avoided by the new > loky > > backend to joblib? > > This is merged and released since a while. Loky is now used in anger. So > far, I think that people are happy with it. In particular it is more > robust than multiprocessing.Pool (specifically, robust to segfault), and > the improvements have been contributed upstream to Python > concurrent.futures's process pool executor (available in Python 3.7). > > Joblib has lately been getting a lot of improvements to make it more > robust and scaleable [1]. It will still have some overhead, due to > pickling. > Pickling speed should be solved by coordinating upstream changes in > Python with implementations in numpy. Olivier Grisel has been > coordinating with Python for this. I believe that PEP 574 [2] is related to > these efforts. The specific challenges are to enable fast code paths in > cloud pickle, which is necessary to pickle arbitrary objects and code. > > While simpler multiprocessing-based code will sometimes give less > overhead compared to joblib, it will probably be brittle. > > I think that the best way to move forward from here would be to do some > prototyping and experimentation. > > Ga?l > > [1] Joblib changelog: > https://joblib.readthedocs.io/en/latest/developing.html#latest-changes > > [2] Pickling improvement PEP: https://www.python.org/dev/peps/pep-0574/ > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Sat Sep 8 18:39:03 2018 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 09 Sep 2018 00:39:03 +0200 Subject: [SciPy-Dev] General discussion on parallelisation In-Reply-To: References: <20180109133448.GA832752@phare.normalesup.org> <20180903092217.afvqf26bblmyd67s@phare.normalesup.org> Message-ID: I meant: used heavily. Ga?l ?Sent from my phone. Please forgive typos and briefness.? On Sep 9, 2018, 00:04, at 00:04, Phillip Feldman wrote: >I don't understand what is meant by the phase "used in anger". > >Phillip > >On Mon, Sep 3, 2018 at 2:22 AM Gael Varoquaux > >wrote: > >> On Sun, Sep 02, 2018 at 08:58:12PM -0700, Ralf Gommers wrote: >> > joblib has a custom backend framework that can be used for such >> purpose >> > (if I understnad you well): >> > https://pythonhosted.org/joblib/parallel.html# >> > custom-backend-api-experimental >> >> >> > Updated link (status is still experimental): >> >https://joblib.readthedocs.io/en/latest/parallel.html#custom-backend-api-experimental >> >> Still experimental, but less and less :). We are using this in anger >> these days. >> >> > There's also this JoblibPool that can be taken over: >> https://github.com/adrn/schwimmbad/blob/master/schwimmbad/jl.py#L14 >> > Seems simpler than a backend still tagged experimental. >> >> Well, we have a fairly stringent definition of experimental. This >feature >> is no longer very experimental. >> >> > This is evolving. However, the reason behind this is that Pool >get >> > corrupted and lead to deadlock. Olivier Grisel and Thomas >Moreau are >> > working on fixing this in the Python standard library (first PR >> merged >> > recently)! >> >> > Anyone know the status of this? And can this issue be avoided by >the new >> loky >> > backend to joblib? >> >> This is merged and released since a while. Loky is now used in anger. >So >> far, I think that people are happy with it. In particular it is more >> robust than multiprocessing.Pool (specifically, robust to segfault), >and >> the improvements have been contributed upstream to Python >> concurrent.futures's process pool executor (available in Python 3.7). >> >> Joblib has lately been getting a lot of improvements to make it more >> robust and scaleable [1]. It will still have some overhead, due to >> pickling. >> Pickling speed should be solved by coordinating upstream changes in >> Python with implementations in numpy. Olivier Grisel has been >> coordinating with Python for this. I believe that PEP 574 [2] is >related to >> these efforts. The specific challenges are to enable fast code paths >in >> cloud pickle, which is necessary to pickle arbitrary objects and >code. >> >> While simpler multiprocessing-based code will sometimes give less >> overhead compared to joblib, it will probably be brittle. >> >> I think that the best way to move forward from here would be to do >some >> prototyping and experimentation. >> >> Ga?l >> >> [1] Joblib changelog: >> >https://joblib.readthedocs.io/en/latest/developing.html#latest-changes >> >> [2] Pickling improvement PEP: >https://www.python.org/dev/peps/pep-0574/ >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > > >------------------------------------------------------------------------ > >_______________________________________________ >SciPy-Dev mailing list >SciPy-Dev at python.org >https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyler.je.reddy at gmail.com Sun Sep 9 20:09:37 2018 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Sun, 9 Sep 2018 17:09:37 -0700 Subject: [SciPy-Dev] Improving spatial.distance code and performance In-Reply-To: References: Message-ID: Good to see some activity / interest in spatial. Definitely agree with Ralf's github comments re: using smaller / more tractable PRs -- it really is tough to sit down at night with 30 minutes of free time or whatever and look at a massive diff & not want to give up. I like the idea of making small / targeted / clearly demonstrated performance improvements without overhauling the entire infrastructure first, but maybe that's a controversial view if it just causes too much heterogeneity. Presumably the affected code is all thoroughly covered by unit tests? That's an important pre-requisite to have the confidence to really start making changes, esp. with older low-level code like that. On Thu, 6 Sep 2018 at 02:29, Kane & Anna O'Donnell < email.the.odonnells at gmail.com> wrote: > Hi all, > > TLDR; I'm wondering about a) porting the spatial.distance code to Cython, > and b) adding some performance optimizations, and I'd like the dev > community's input/feedback. > > For context (though I don't really need you to read these to give the > feedback I'm looking for), > > - original issue/proposal: https://github.com/scipy/scipy/issues/9205 > - PR: https://github.com/scipy/scipy/pull/9218 > > Before submitting the PR, I naively thought it was going to be nice Cython > or similar. Turns out it's some pretty old code, that I found pretty hard > to wrap my head around and understand. I eventually figured it out after > spending ages debugging a nastily hidden 'bug', and demonstrated the > performance optimizations, but it prompted the discussion about whether it > was best to port everything to Cython first. > > *Existing stuff to Cython* > > Doing so shouldn't be too hard, and it shouldn't change any functionality, > except to replace the distance functions with their Cython ones (instead of > the current approach, where the distance functions are actually numpy > things, and there's not supported access to the underlying C stuff). A few > possible 'bugs' (as above) should hopefully become non-issues too. So, it'd > be a win for performance (e.g. all the distance functions will be much > faster), and code quality, and future maintainability and development. > However, things to think about: > > - should I just replace like-for-like, or consider some nicer OOP stuff > like e.g. sklearn's approach (which is already Cython)? > https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/neighbors/dist_metrics.pyx > (I'm guessing the reason they rolled their own was because they found scipy > lacking, as above.) In fact, we could largely just copy that file over. Not > sure about the interplay between scipy and scikit learn though. > - what's the best way to structure a PR? One metric at a time, or the > whole caboodle? > > *Adding performance optimizations and future functionality* > > As per the links, this was initially about providing a nifty performance > hack. It should still be pretty easy to implement. Personally, I think it > makes sense to implement after the Cythonization - unless the community are > against that. > > However, there are other possibilities: > > - using 'indices' within calculations. E.g. when doing pdist, it might pay > to use a tree of some description. I also proposed another 'index' to > optimize the 'bail early' approach further (which would, I think, actually > work well with trees too). This would involve more API changes, possibly > significant. > - using approximate searches (e.g. Faiss). My understanding is that > depending on other libraries probably isn't really an option, so I'm not > sure what means. > - maybe other cool approaches like https://github.com/droyed/eucl_dist > - providing a way to 'tune' distance computations to be optimal to your > particular dataset and constraints (e.g. my 'bail early' optimization might > be a lot faster or a bit slower, depending on your data ... or you might be > OK with approximate matching with a 'low' error rate, etc.) > > I guess what I'd like to see is a single place where users can get access > to everything related to distance metrics and their uses, including all > sorts of optimizations etc. (possibly for different hardware, and possibly > distributed). To do that well is a pretty big undertaking, and I don't know > whether it's suited to scipy - e.g. maybe scipy doesn't really care about > distance stuff, or only wants to stick with 'simple' distance metric cases > (e.g. a few thousand vectors, etc.). So, maybe it'd be better to start a > completely new python package - which would probably be a lot easier to > develop as I'd have a lot more flexibility (e.g. to depend on other > packages, and not have to worry about breaking the scipy API etc.). On the > other hand (as discussed in the latest comment on the PR), that might not > be best - it might never get used/maintained etc. > > So, what does the community think is the best approach? I've got too > little context of what scipy is and what it's aiming for, and I don't want > to head off on the wrong tack. Comments on any of the other implied > questions would also be appreciated. > > Thanks, > > kodonnell > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikofski at berkeley.edu Mon Sep 10 13:05:17 2018 From: mikofski at berkeley.edu (Mark Alexander Mikofski) Date: Mon, 10 Sep 2018 10:05:17 -0700 Subject: [SciPy-Dev] Improving spatial.distance code and performance In-Reply-To: References: Message-ID: I'm very interested to see how a successful cython/performance PR progresses from a reviewers standpoint. On Sun, Sep 9, 2018, 5:10 PM Tyler Reddy wrote: > Good to see some activity / interest in spatial. > > Definitely agree with Ralf's github comments re: using smaller / more > tractable PRs -- it really is tough to sit down at night with 30 minutes of > free time or whatever and look at a massive diff & not want to give up. > > I like the idea of making small / targeted / clearly demonstrated > performance improvements without overhauling the entire infrastructure > first, but maybe that's a controversial view if it just causes too much > heterogeneity. > > Presumably the affected code is all thoroughly covered by unit tests? > That's an important pre-requisite to have the confidence to really start > making changes, esp. with older low-level code like that. > > On Thu, 6 Sep 2018 at 02:29, Kane & Anna O'Donnell < > email.the.odonnells at gmail.com> wrote: > >> Hi all, >> >> TLDR; I'm wondering about a) porting the spatial.distance code to Cython, >> and b) adding some performance optimizations, and I'd like the dev >> community's input/feedback. >> >> For context (though I don't really need you to read these to give the >> feedback I'm looking for), >> >> - original issue/proposal: https://github.com/scipy/scipy/issues/9205 >> - PR: https://github.com/scipy/scipy/pull/9218 >> >> Before submitting the PR, I naively thought it was going to be nice >> Cython or similar. Turns out it's some pretty old code, that I found pretty >> hard to wrap my head around and understand. I eventually figured it out >> after spending ages debugging a nastily hidden 'bug', and demonstrated the >> performance optimizations, but it prompted the discussion about whether it >> was best to port everything to Cython first. >> >> *Existing stuff to Cython* >> >> Doing so shouldn't be too hard, and it shouldn't change any >> functionality, except to replace the distance functions with their Cython >> ones (instead of the current approach, where the distance functions are >> actually numpy things, and there's not supported access to the underlying C >> stuff). A few possible 'bugs' (as above) should hopefully become non-issues >> too. So, it'd be a win for performance (e.g. all the distance functions >> will be much faster), and code quality, and future maintainability and >> development. However, things to think about: >> >> - should I just replace like-for-like, or consider some nicer OOP stuff >> like e.g. sklearn's approach (which is already Cython)? >> https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/neighbors/dist_metrics.pyx >> (I'm guessing the reason they rolled their own was because they found scipy >> lacking, as above.) In fact, we could largely just copy that file over. Not >> sure about the interplay between scipy and scikit learn though. >> - what's the best way to structure a PR? One metric at a time, or the >> whole caboodle? >> >> *Adding performance optimizations and future functionality* >> >> As per the links, this was initially about providing a nifty performance >> hack. It should still be pretty easy to implement. Personally, I think it >> makes sense to implement after the Cythonization - unless the community are >> against that. >> >> However, there are other possibilities: >> >> - using 'indices' within calculations. E.g. when doing pdist, it might >> pay to use a tree of some description. I also proposed another 'index' to >> optimize the 'bail early' approach further (which would, I think, actually >> work well with trees too). This would involve more API changes, possibly >> significant. >> - using approximate searches (e.g. Faiss). My understanding is that >> depending on other libraries probably isn't really an option, so I'm not >> sure what means. >> - maybe other cool approaches like https://github.com/droyed/eucl_dist >> - providing a way to 'tune' distance computations to be optimal to your >> particular dataset and constraints (e.g. my 'bail early' optimization might >> be a lot faster or a bit slower, depending on your data ... or you might be >> OK with approximate matching with a 'low' error rate, etc.) >> >> I guess what I'd like to see is a single place where users can get access >> to everything related to distance metrics and their uses, including all >> sorts of optimizations etc. (possibly for different hardware, and possibly >> distributed). To do that well is a pretty big undertaking, and I don't know >> whether it's suited to scipy - e.g. maybe scipy doesn't really care about >> distance stuff, or only wants to stick with 'simple' distance metric cases >> (e.g. a few thousand vectors, etc.). So, maybe it'd be better to start a >> completely new python package - which would probably be a lot easier to >> develop as I'd have a lot more flexibility (e.g. to depend on other >> packages, and not have to worry about breaking the scipy API etc.). On the >> other hand (as discussed in the latest comment on the PR), that might not >> be best - it might never get used/maintained etc. >> >> So, what does the community think is the best approach? I've got too >> little context of what scipy is and what it's aiming for, and I don't want >> to head off on the wrong tack. Comments on any of the other implied >> questions would also be appreciated. >> >> Thanks, >> >> kodonnell >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From einstein.edison at gmail.com Wed Sep 12 13:20:21 2018 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Wed, 12 Sep 2018 19:20:21 +0200 Subject: [SciPy-Dev] ANN: PyData/Sparse 0.4.1 Message-ID: ============================= Announcing PyData/Sparse 0.4.1 ============================= (Apologies for the cross-posting) Hi everyone, This is a performance, bug-fix and feature release. The changelog can be seen at https://sparse.pydata.org/en/latest/changelog.html Highlights include: Faux In-place operations Mixed ndarray-sparse operations Fill-values other than zero Misc support for different functions What?s PyData/Sparse? ??????????? PyData/Sparse is a an N-dimensional sparse array library. It?s compatible with NumPy and follows the ndarray interface as closely as possible. It depends on NumPy, SciPy and Numba. Where can I find PyData/Sparse? ??????????????? The project is available on GitHub at https://github.com/pydata/sparse and is available on PyPI and conda-forge as ?sparse?. Documentation is hosted as https://sparse.pydata.org. Best Regards, Hameer Abbasi -------------- next part -------------- An HTML attachment was scrubbed... URL: From touqir at ualberta.ca Fri Sep 14 18:42:00 2018 From: touqir at ualberta.ca (Touqir Sajed) Date: Fri, 14 Sep 2018 16:42:00 -0600 Subject: [SciPy-Dev] Large Scale Optimization algorithm implementation Message-ID: Hi, So, the most efficient optimization algorithm implemented in Scipy is LBFGS-B in terms of memory consumption and runtime speed. It would be good to see algorithms suited for large scale optimization in scipy for the scenarios when Gradient computation is quite expensive and/or Number of variables is huge. SVRG and (SVRG+LBFGS) can both attain much better performance than LBFGS-B for this scenario. The links: (SVRG) : https://papers.nips.cc/paper/4937-accelerating-stochastic-gradient-descent-using-predictive-variance-reduction.pdf (SVRG + LBFGS) : http://opt-ml.org/papers/OPT2015_paper_41.pdf What do you think? Cheers, Touqir -- Computing Science Master's student at University of Alberta, Canada, specializing in Machine Learning. Website : https://ca.linkedin.com/in/touqir-sajed-6a95b1126 -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Fri Sep 14 19:07:32 2018 From: shoyer at gmail.com (Stephan Hoyer) Date: Fri, 14 Sep 2018 16:07:32 -0700 Subject: [SciPy-Dev] Large Scale Optimization algorithm implementation In-Reply-To: References: Message-ID: These algorithms look specialized to problems where the optimization objective can be written as an average over many examples. This is not necessarily the case for the functions optimized by scipy.optimize.minimize -- we don't have any equivalent of stochastic gradient descent. This might be more suitable for libraries that specialize in these sort of optimization problems, like TensorFlow or pytorch. On Fri, Sep 14, 2018 at 3:42 PM Touqir Sajed wrote: > Hi, > > So, the most efficient optimization algorithm implemented in Scipy is > LBFGS-B in terms of memory consumption and runtime speed. It would be good > to see algorithms suited for large scale optimization in scipy for the > scenarios when Gradient computation is quite expensive and/or Number of > variables is huge. SVRG and (SVRG+LBFGS) can both attain much better > performance than LBFGS-B for this scenario. > The links: > (SVRG) : > https://papers.nips.cc/paper/4937-accelerating-stochastic-gradient-descent-using-predictive-variance-reduction.pdf > > (SVRG + LBFGS) : http://opt-ml.org/papers/OPT2015_paper_41.pdf > > > What do you think? > > Cheers, > Touqir > > -- > Computing Science Master's student at University of Alberta, Canada, > specializing in Machine Learning. Website : > https://ca.linkedin.com/in/touqir-sajed-6a95b1126 > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From touqir at ualberta.ca Fri Sep 14 20:13:02 2018 From: touqir at ualberta.ca (Touqir Sajed) Date: Fri, 14 Sep 2018 18:13:02 -0600 Subject: [SciPy-Dev] On robust mean estimators Message-ID: Talking about robust mean estimators, there is the trimmed mean estimator in scipy which unfortunately can be biased if the distribution is not symmetric. The median of the means estimator can be more robust and at the same time be unbiased. Should the median of the means estimator be added to scipy? The idea is to separate the whole sample set into small sized minibatches of size k and then outputting the median of their(k minibatches) mean values. -- Computing Science Master's student at University of Alberta, Canada, specializing in Machine Learning. Website : https://ca.linkedin.com/in/touqir-sajed-6a95b1126 -------------- next part -------------- An HTML attachment was scrubbed... URL: From email.the.odonnells at gmail.com Sat Sep 15 04:09:20 2018 From: email.the.odonnells at gmail.com (Kane & Anna O'Donnell) Date: Sat, 15 Sep 2018 20:09:20 +1200 Subject: [SciPy-Dev] Improving spatial.distance code and, performance In-Reply-To: References: Message-ID: <84a744f2-cb30-9bc3-0cfe-579fe06a0b5a@gmail.com> Sorry if this doesn't work - I haven't used mailing lists before, and got the digest, which doesn't seem to allow me to reply to a specific thread. So I'm attempting to set the In-Reply-To header manually. Sorry if that's silly. Anyway: Yes, I believe there are good unit tests (Ralf actually ran them I think). From the basis of this, and the feedback in the issue/PR, it sounds like the preference would be for me to upgrade the distance metrics to cython bit-by-bit. However, I'm still unsure about scipy vs a new package, since scipy apparently won't support dependencies. For example, I could use Faiss or flann, but scipy wouldn't add them as a dependency. With that in mind, it seems the only option for offering users full-functionality (aside from rewriting those libraries from scratch etc.) is to offer a package. Unless scipy is willing to specify a run-time dependency (not sure of the correct terminology) - e.g. scipy doesn't depend on Faiss (when you install it), but if the user calls a distance metric with e.g. method='faiss', it will fail if faiss cannot be imported. Is this a possibility (and a good idea)? On 2018/09/12 4:00 am, scipy-dev-request at python.org wrote: > Send SciPy-Dev mailing list submissions to > scipy-dev at python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scipy-dev > or, via email, send a message with subject or body 'help' to > scipy-dev-request at python.org > > You can reach the person managing the list at > scipy-dev-owner at python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of SciPy-Dev digest..." > > > Today's Topics: > > 1. Re: Improving spatial.distance code and performance > (Mark Alexander Mikofski) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 10 Sep 2018 10:05:17 -0700 > From: Mark Alexander Mikofski > To: SciPy Developers List > Subject: Re: [SciPy-Dev] Improving spatial.distance code and > performance > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > I'm very interested to see how a successful cython/performance PR > progresses from a reviewers standpoint. > > On Sun, Sep 9, 2018, 5:10 PM Tyler Reddy wrote: > >> Good to see some activity / interest in spatial. >> >> Definitely agree with Ralf's github comments re: using smaller / more >> tractable PRs -- it really is tough to sit down at night with 30 minutes of >> free time or whatever and look at a massive diff & not want to give up. >> >> I like the idea of making small / targeted / clearly demonstrated >> performance improvements without overhauling the entire infrastructure >> first, but maybe that's a controversial view if it just causes too much >> heterogeneity. >> >> Presumably the affected code is all thoroughly covered by unit tests? >> That's an important pre-requisite to have the confidence to really start >> making changes, esp. with older low-level code like that. >> >> On Thu, 6 Sep 2018 at 02:29, Kane & Anna O'Donnell < >> email.the.odonnells at gmail.com> wrote: >> >>> Hi all, >>> >>> TLDR; I'm wondering about a) porting the spatial.distance code to Cython, >>> and b) adding some performance optimizations, and I'd like the dev >>> community's input/feedback. >>> >>> For context (though I don't really need you to read these to give the >>> feedback I'm looking for), >>> >>> - original issue/proposal: https://github.com/scipy/scipy/issues/9205 >>> - PR: https://github.com/scipy/scipy/pull/9218 >>> >>> Before submitting the PR, I naively thought it was going to be nice >>> Cython or similar. Turns out it's some pretty old code, that I found pretty >>> hard to wrap my head around and understand. I eventually figured it out >>> after spending ages debugging a nastily hidden 'bug', and demonstrated the >>> performance optimizations, but it prompted the discussion about whether it >>> was best to port everything to Cython first. >>> >>> *Existing stuff to Cython* >>> >>> Doing so shouldn't be too hard, and it shouldn't change any >>> functionality, except to replace the distance functions with their Cython >>> ones (instead of the current approach, where the distance functions are >>> actually numpy things, and there's not supported access to the underlying C >>> stuff). A few possible 'bugs' (as above) should hopefully become non-issues >>> too. So, it'd be a win for performance (e.g. all the distance functions >>> will be much faster), and code quality, and future maintainability and >>> development. However, things to think about: >>> >>> - should I just replace like-for-like, or consider some nicer OOP stuff >>> like e.g. sklearn's approach (which is already Cython)? >>> https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/neighbors/dist_metrics.pyx >>> (I'm guessing the reason they rolled their own was because they found scipy >>> lacking, as above.) In fact, we could largely just copy that file over. Not >>> sure about the interplay between scipy and scikit learn though. >>> - what's the best way to structure a PR? One metric at a time, or the >>> whole caboodle? >>> >>> *Adding performance optimizations and future functionality* >>> >>> As per the links, this was initially about providing a nifty performance >>> hack. It should still be pretty easy to implement. Personally, I think it >>> makes sense to implement after the Cythonization - unless the community are >>> against that. >>> >>> However, there are other possibilities: >>> >>> - using 'indices' within calculations. E.g. when doing pdist, it might >>> pay to use a tree of some description. I also proposed another 'index' to >>> optimize the 'bail early' approach further (which would, I think, actually >>> work well with trees too). This would involve more API changes, possibly >>> significant. >>> - using approximate searches (e.g. Faiss). My understanding is that >>> depending on other libraries probably isn't really an option, so I'm not >>> sure what means. >>> - maybe other cool approaches like https://github.com/droyed/eucl_dist >>> - providing a way to 'tune' distance computations to be optimal to your >>> particular dataset and constraints (e.g. my 'bail early' optimization might >>> be a lot faster or a bit slower, depending on your data ... or you might be >>> OK with approximate matching with a 'low' error rate, etc.) >>> >>> I guess what I'd like to see is a single place where users can get access >>> to everything related to distance metrics and their uses, including all >>> sorts of optimizations etc. (possibly for different hardware, and possibly >>> distributed). To do that well is a pretty big undertaking, and I don't know >>> whether it's suited to scipy - e.g. maybe scipy doesn't really care about >>> distance stuff, or only wants to stick with 'simple' distance metric cases >>> (e.g. a few thousand vectors, etc.). So, maybe it'd be better to start a >>> completely new python package - which would probably be a lot easier to >>> develop as I'd have a lot more flexibility (e.g. to depend on other >>> packages, and not have to worry about breaking the scipy API etc.). On the >>> other hand (as discussed in the latest comment on the PR), that might not >>> be best - it might never get used/maintained etc. >>> >>> So, what does the community think is the best approach? I've got too >>> little context of what scipy is and what it's aiming for, and I don't want >>> to head off on the wrong tack. Comments on any of the other implied >>> questions would also be appreciated. >>> >>> Thanks, >>> >>> kodonnell >>> >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > > ------------------------------ > > End of SciPy-Dev Digest, Vol 179, Issue 12 > ****************************************** From touqir at ualberta.ca Sat Sep 15 05:30:27 2018 From: touqir at ualberta.ca (Touqir Sajed) Date: Sat, 15 Sep 2018 03:30:27 -0600 Subject: [SciPy-Dev] Randomized linear algebra functionality Message-ID: There seems to be some interest in implementing randomized linear algebra algorithms such as JL transformation, low-rank approximation, etc as written in https://github.com/scipy/scipy/wiki/GSoC-2018-project-ideas . To the best of my knowledge, they are not implemented yet. If there still exists interests I would like to implement the Fast JL transformation and the randomized SVD for low-rank approximation. The pseudo code of Fast JL transformation is given here: https://pdfs.semanticscholar.org/ccf0/1e0375d11df5335b645161b4833e32380d89.pdf . The randomized SVD is based on https://arxiv.org/pdf/0909.4061.pdf . -- Computing Science Master's student at University of Alberta, Canada, specializing in Machine Learning. Website : https://ca.linkedin.com/in/touqir-sajed-6a95b1126 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sat Sep 15 14:15:36 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 15 Sep 2018 11:15:36 -0700 Subject: [SciPy-Dev] Randomized linear algebra functionality In-Reply-To: References: Message-ID: On Sat, Sep 15, 2018 at 2:30 AM Touqir Sajed wrote: > There seems to be some interest in implementing randomized linear algebra > algorithms such as JL transformation, low-rank approximation, etc as > written in https://github.com/scipy/scipy/wiki/GSoC-2018-project-ideas > > . To the best of my knowledge, they are not implemented yet. If there still > exists interests I would like to implement the Fast JL transformation and > the randomized SVD for low-rank approximation. The pseudo code of Fast JL > transformation is given here: > https://pdfs.semanticscholar.org/ccf0/1e0375d11df5335b645161b4833e32380d89.pdf > > . The randomized SVD is based on https://arxiv.org/pdf/0909.4061.pdf > > . > Hi Touqir. The problem with randomized linear algebra is that there's no core developer with the knowledge and interest to work on this. That GSoC proposal never materialized. So I suspect that while there's interest from users, PRs will be difficult to get merged. A better approach here may be to implement these methods in a separate package, and once that stabilized and has gained users, then revisit whether it can be incorporated into SciPy. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at informa.tiker.net Sat Sep 15 15:04:13 2018 From: lists at informa.tiker.net (Andreas Kloeckner) Date: Sat, 15 Sep 2018 14:04:13 -0500 Subject: [SciPy-Dev] Randomized linear algebra functionality In-Reply-To: References: Message-ID: <87sh2ainv6.fsf@bolt.d.tiker.net> Touqir, Touqir Sajed writes: > There seems to be some interest in implementing randomized linear algebra > algorithms such as JL transformation, low-rank approximation, etc as > written in https://github.com/scipy/scipy/wiki/GSoC-2018-project-ideas > > . To the best of my knowledge, they are not implemented yet. If there still > exists interests I would like to implement the Fast JL transformation and > the randomized SVD for low-rank approximation. The pseudo code of Fast JL > transformation is given here: > https://pdfs.semanticscholar.org/ccf0/1e0375d11df5335b645161b4833e32380d89.pdf > > . The randomized SVD is based on https://arxiv.org/pdf/0909.4061.pdf > scipy.linalg.interpolative is in fact based on randomized linear algebra. In particular, this function: https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.interpolative.svd.html#scipy.linalg.interpolative.svd already does implement a randomized SVD with a number of acceleration bells and whistles. Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From touqir at ualberta.ca Sat Sep 15 15:44:14 2018 From: touqir at ualberta.ca (Touqir Sajed) Date: Sat, 15 Sep 2018 13:44:14 -0600 Subject: [SciPy-Dev] Randomized linear algebra functionality In-Reply-To: <87sh2ainv6.fsf@bolt.d.tiker.net> References: <87sh2ainv6.fsf@bolt.d.tiker.net> Message-ID: Aha, this implements the exact functionality that I had in mind! Thanks for bringing this up Andreas. Regarding fast JL transformation, there is an existing package in python : https://github.com/michaelmathen/FJLT . Are there functionalities that the core developers have in mind and can use some help implementing them? On Sat, Sep 15, 2018 at 1:04 PM Andreas Kloeckner wrote: > Touqir, > > Touqir Sajed writes: > > There seems to be some interest in implementing randomized linear algebra > > algorithms such as JL transformation, low-rank approximation, etc as > > written in https://github.com/scipy/scipy/wiki/GSoC-2018-project-ideas > > < > https://ml-trckr.com/link/https%3A%2F%2Fgithub.com%2Fscipy%2Fscipy%2Fwiki%2FGSoC-2018-project-ideas/cBzmzxqJ6V7Fi58kyipk > > > > . To the best of my knowledge, they are not implemented yet. If there > still > > exists interests I would like to implement the Fast JL transformation and > > the randomized SVD for low-rank approximation. The pseudo code of Fast JL > > transformation is given here: > > > https://pdfs.semanticscholar.org/ccf0/1e0375d11df5335b645161b4833e32380d89.pdf > > < > https://ml-trckr.com/link/https%3A%2F%2Fpdfs.semanticscholar.org%2Fccf0%2F1e0375d11df5335b645161b4833e32380d89.pdf/cBzmzxqJ6V7Fi58kyipk > > > > . The randomized SVD is based on https://arxiv.org/pdf/0909.4061.pdf > > < > https://ml-trckr.com/link/https%3A%2F%2Farxiv.org%2Fpdf%2F0909.4061.pdf/cBzmzxqJ6V7Fi58kyipk > > > > scipy.linalg.interpolative is in fact based on randomized linear > algebra. In particular, this function: > > > https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.interpolative.svd.html#scipy.linalg.interpolative.svd > > already does implement a randomized SVD with a number of acceleration > bells and whistles. > > Andreas > -- Computing Science Master's student at University of Alberta, Canada, specializing in Machine Learning. Website : https://ca.linkedin.com/in/touqir-sajed-6a95b1126 -------------- next part -------------- An HTML attachment was scrubbed... URL: From haberland at ucla.edu Sat Sep 15 16:06:16 2018 From: haberland at ucla.edu (Matt Haberland) Date: Sat, 15 Sep 2018 13:06:16 -0700 Subject: [SciPy-Dev] Randomized linear algebra functionality In-Reply-To: References: <87sh2ainv6.fsf@bolt.d.tiker.net> Message-ID: I'd really love to see some of the techniques in: https://arxiv.org/pdf/1203.6705.pdf implemented! Matt On Sat, Sep 15, 2018 at 12:44 PM, Touqir Sajed wrote: > Aha, this implements the exact functionality that I had in mind! Thanks > for bringing this up Andreas. > Regarding fast JL transformation, there is an existing package in python : > https://github.com/michaelmathen/FJLT > > . > > Are there functionalities that the core developers have in mind and can > use some help implementing them? > > On Sat, Sep 15, 2018 at 1:04 PM Andreas Kloeckner > wrote: > >> Touqir, >> >> Touqir Sajed writes: >> > There seems to be some interest in implementing randomized linear >> algebra >> > algorithms such as JL transformation, low-rank approximation, etc as >> > written in https://github.com/scipy/scipy/wiki/GSoC-2018-project-ideas >> > > 2Fscipy%2Fscipy%2Fwiki%2FGSoC-2018-project-ideas/cBzmzxqJ6V7Fi58kyipk> >> > . To the best of my knowledge, they are not implemented yet. If there >> still >> > exists interests I would like to implement the Fast JL transformation >> and >> > the randomized SVD for low-rank approximation. The pseudo code of Fast >> JL >> > transformation is given here: >> > https://pdfs.semanticscholar.org/ccf0/1e0375d11df5335b645161b4833e32 >> 380d89.pdf >> > > semanticscholar.org%2Fccf0%2F1e0375d11df5335b645161b4833e32380d89.pdf/ >> cBzmzxqJ6V7Fi58kyipk> >> > . The randomized SVD is based on https://arxiv.org/pdf/0909.4061.pdf >> > > 2F0909.4061.pdf/cBzmzxqJ6V7Fi58kyipk> >> >> scipy.linalg.interpolative is in fact based on randomized linear >> algebra. In particular, this function: >> >> https://docs.scipy.org/doc/scipy/reference/generated/ >> scipy.linalg.interpolative.svd.html#scipy.linalg.interpolative.svd >> >> already does implement a randomized SVD with a number of acceleration >> bells and whistles. >> >> Andreas >> > > > -- > Computing Science Master's student at University of Alberta, Canada, > specializing in Machine Learning. Website : https://ca.linkedin.com/in/ > touqir-sajed-6a95b1126 > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > -- Matt Haberland Assistant Adjunct Professor in the Program in Computing Department of Mathematics 6617A Math Sciences Building, UCLA -------------- next part -------------- An HTML attachment was scrubbed... URL: From touqir at ualberta.ca Sat Sep 15 18:47:55 2018 From: touqir at ualberta.ca (Touqir Sajed) Date: Sat, 15 Sep 2018 16:47:55 -0600 Subject: [SciPy-Dev] Randomized linear algebra functionality In-Reply-To: References: <87sh2ainv6.fsf@bolt.d.tiker.net> Message-ID: I assume you are referring to the linear algebra algorithms in section 4.1 that use their basis construction algorithm and not the graph matching and matroid optimization algorithms? It seems their results are still the state of the art. I am fine with implementing them. I am thinking of the following API changes : (1) Implement scipy.linalg.dot_lowrank or scipy.linalg.mult_lowrank for multiplying a low-rank matrix A with another matrix B. Or just implement this additional functionality within numpy's dot function and add a boolean parameter. (2) Add a boolean parameter "rand" to scipy.linalg.null_space to use the more sophisticated randomized algorithm for null space computation. (3) Implement scipy.linalg.rankone_matrix_decomposition for decomposing the matrix into a sum of r rank 1 matrices. All of these algorithms should give performance improvements for low-rank matrices. On Sat, Sep 15, 2018 at 2:07 PM Matt Haberland wrote: > I'd really love to see some of the techniques in: > https://arxiv.org/pdf/1203.6705.pdf > implemented! > Matt > > On Sat, Sep 15, 2018 at 12:44 PM, Touqir Sajed wrote: > >> Aha, this implements the exact functionality that I had in mind! Thanks >> for bringing this up Andreas. >> Regarding fast JL transformation, there is an existing package in python >> : https://github.com/michaelmathen/FJLT >> >> . >> >> Are there functionalities that the core developers have in mind and can >> use some help implementing them? >> >> On Sat, Sep 15, 2018 at 1:04 PM Andreas Kloeckner < >> lists at informa.tiker.net> wrote: >> >>> Touqir, >>> >>> Touqir Sajed writes: >>> > There seems to be some interest in implementing randomized linear >>> algebra >>> > algorithms such as JL transformation, low-rank approximation, etc as >>> > written in >>> https://github.com/scipy/scipy/wiki/GSoC-2018-project-ideas >>> > < >>> https://ml-trckr.com/link/https%3A%2F%2Fgithub.com%2Fscipy%2Fscipy%2Fwiki%2FGSoC-2018-project-ideas/cBzmzxqJ6V7Fi58kyipk >>> > >>> > . To the best of my knowledge, they are not implemented yet. If there >>> still >>> > exists interests I would like to implement the Fast JL transformation >>> and >>> > the randomized SVD for low-rank approximation. The pseudo code of Fast >>> JL >>> > transformation is given here: >>> > >>> https://pdfs.semanticscholar.org/ccf0/1e0375d11df5335b645161b4833e32380d89.pdf >>> > < >>> https://ml-trckr.com/link/https%3A%2F%2Fpdfs.semanticscholar.org%2Fccf0%2F1e0375d11df5335b645161b4833e32380d89.pdf/cBzmzxqJ6V7Fi58kyipk >>> > >>> > . The randomized SVD is based on https://arxiv.org/pdf/0909.4061.pdf >>> > < >>> https://ml-trckr.com/link/https%3A%2F%2Farxiv.org%2Fpdf%2F0909.4061.pdf/cBzmzxqJ6V7Fi58kyipk >>> > >>> >>> scipy.linalg.interpolative is in fact based on randomized linear >>> algebra. In particular, this function: >>> >>> >>> https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.interpolative.svd.html#scipy.linalg.interpolative.svd >>> >>> already does implement a randomized SVD with a number of acceleration >>> bells and whistles. >>> >>> Andreas >>> >> >> >> -- >> Computing Science Master's student at University of Alberta, Canada, >> specializing in Machine Learning. Website : >> https://ca.linkedin.com/in/touqir-sajed-6a95b1126 >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> >> > > > -- > Matt Haberland > Assistant Adjunct Professor in the Program in Computing > Department of Mathematics > 6617A Math Sciences Building, UCLA > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -- Computing Science Master's student at University of Alberta, Canada, specializing in Machine Learning. Website : https://ca.linkedin.com/in/touqir-sajed-6a95b1126 -------------- next part -------------- An HTML attachment was scrubbed... URL: From haberland at ucla.edu Sat Sep 15 19:45:16 2018 From: haberland at ucla.edu (Matt Haberland) Date: Sat, 15 Sep 2018 16:45:16 -0700 Subject: [SciPy-Dev] Randomized linear algebra functionality In-Reply-To: References: <87sh2ainv6.fsf@bolt.d.tiker.net> Message-ID: Hi Touqir, Yes, well mostly (2). I am particularly interested in a calculating matrix rank and especially finding the linearly dependent columns. I don't necessarily need an orthonormal basis for the nullspace. Matt On Sat, Sep 15, 2018 at 3:47 PM, Touqir Sajed wrote: > I assume you are referring to the linear algebra algorithms in section 4.1 > that use their basis construction algorithm and not the graph matching and > matroid optimization algorithms? It seems their results are still the state > of the art. I am fine with implementing them. > I am thinking of the following API changes : > > (1) Implement scipy.linalg.dot_lowrank or scipy.linalg.mult_lowrank > for multiplying a low-rank matrix A with another matrix B. Or just > implement this additional functionality within numpy's dot function and add > a boolean parameter. > > (2) Add a boolean parameter "rand" to scipy.linalg.null_space to use > the more sophisticated randomized algorithm for null space computation. > > (3) Implement scipy.linalg.rankone_matrix_decomposition for > decomposing the matrix into a sum of r rank 1 matrices. > > All of these algorithms should give performance improvements for low-rank > matrices. > > On Sat, Sep 15, 2018 at 2:07 PM Matt Haberland wrote: > >> I'd really love to see some of the techniques in: >> https://arxiv.org/pdf/1203.6705.pdf >> implemented! >> Matt >> >> On Sat, Sep 15, 2018 at 12:44 PM, Touqir Sajed >> wrote: >> >>> Aha, this implements the exact functionality that I had in mind! Thanks >>> for bringing this up Andreas. >>> Regarding fast JL transformation, there is an existing package in python >>> : https://github.com/michaelmathen/FJLT >>> >>> . >>> >>> Are there functionalities that the core developers have in mind and can >>> use some help implementing them? >>> >>> On Sat, Sep 15, 2018 at 1:04 PM Andreas Kloeckner < >>> lists at informa.tiker.net> wrote: >>> >>>> Touqir, >>>> >>>> Touqir Sajed writes: >>>> > There seems to be some interest in implementing randomized linear >>>> algebra >>>> > algorithms such as JL transformation, low-rank approximation, etc as >>>> > written in https://github.com/scipy/scipy/wiki/GSoC-2018-project- >>>> ideas >>>> > >>> 2Fscipy%2Fscipy%2Fwiki%2FGSoC-2018-project-ideas/cBzmzxqJ6V7Fi58kyipk> >>>> > . To the best of my knowledge, they are not implemented yet. If there >>>> still >>>> > exists interests I would like to implement the Fast JL transformation >>>> and >>>> > the randomized SVD for low-rank approximation. The pseudo code of >>>> Fast JL >>>> > transformation is given here: >>>> > https://pdfs.semanticscholar.org/ccf0/1e0375d11df5335b645161b4833e32 >>>> 380d89.pdf >>>> > >>> semanticscholar.org%2Fccf0%2F1e0375d11df5335b645161b4833e32380d89.pdf/ >>>> cBzmzxqJ6V7Fi58kyipk> >>>> > . The randomized SVD is based on https://arxiv.org/pdf/0909.4061.pdf >>>> > >>> 2F0909.4061.pdf/cBzmzxqJ6V7Fi58kyipk> >>>> >>>> scipy.linalg.interpolative is in fact based on randomized linear >>>> algebra. In particular, this function: >>>> >>>> https://docs.scipy.org/doc/scipy/reference/generated/ >>>> scipy.linalg.interpolative.svd.html#scipy.linalg.interpolative.svd >>>> >>>> already does implement a randomized SVD with a number of acceleration >>>> bells and whistles. >>>> >>>> Andreas >>>> >>> >>> >>> -- >>> Computing Science Master's student at University of Alberta, Canada, >>> specializing in Machine Learning. Website : https://ca.linkedin.com/in/ >>> touqir-sajed-6a95b1126 >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >>> >> >> >> -- >> Matt Haberland >> Assistant Adjunct Professor in the Program in Computing >> Department of Mathematics >> 6617A Math Sciences Building, UCLA >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > > > -- > Computing Science Master's student at University of Alberta, Canada, > specializing in Machine Learning. Website : https://ca.linkedin.com/in/ > touqir-sajed-6a95b1126 > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > -- Matt Haberland Assistant Adjunct Professor in the Program in Computing Department of Mathematics 6617A Math Sciences Building, UCLA -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Sat Sep 15 20:38:27 2018 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 16 Sep 2018 02:38:27 +0200 Subject: [SciPy-Dev] Randomized linear algebra functionality In-Reply-To: References: <87sh2ainv6.fsf@bolt.d.tiker.net> Message-ID: <30e25171452280a02fe320f78d112064ddebeb4e.camel@iki.fi> la, 2018-09-15 kello 16:45 -0700, Matt Haberland kirjoitti: > Yes, well mostly (2). I am particularly interested in a calculating > matrix > rank and especially finding the linearly dependent columns. I don't > necessarily need an orthonormal basis for the nullspace. As Andreas notes, the former exists in https://docs.scipy.org/doc/scipy/reference/linalg.interpolative.html The latter probably also can be expressed in terms of this decomposition. Pauli > > Matt > > On Sat, Sep 15, 2018 at 3:47 PM, Touqir Sajed > wrote: > > > I assume you are referring to the linear algebra algorithms in > > section 4.1 > > that use their basis construction algorithm and not the graph > > matching and > > matroid optimization algorithms? It seems their results are still > > the state > > of the art. I am fine with implementing them. > > I am thinking of the following API changes : > > > > (1) Implement scipy.linalg.dot_lowrank or > > scipy.linalg.mult_lowrank > > for multiplying a low-rank matrix A with another matrix B. Or just > > implement this additional functionality within numpy's dot function > > and add > > a boolean parameter. > > > > (2) Add a boolean parameter "rand" > > to scipy.linalg.null_space to use > > the more sophisticated randomized algorithm for null space > > computation. > > > > (3) Implement scipy.linalg.rankone_matrix_decomposition for > > decomposing the matrix into a sum of r rank 1 matrices. > > > > All of these algorithms should give performance improvements for > > low-rank > > matrices. > > > > On Sat, Sep 15, 2018 at 2:07 PM Matt Haberland > > wrote: > > > > > I'd really love to see some of the techniques in: > > > https://arxiv.org/pdf/1203.6705.pdf > > > implemented! > > > Matt > > > > > > On Sat, Sep 15, 2018 at 12:44 PM, Touqir Sajed < > > > touqir at ualberta.ca> > > > wrote: > > > > > > > Aha, this implements the exact functionality that I had in > > > > mind! Thanks > > > > for bringing this up Andreas. > > > > Regarding fast JL transformation, there is an existing package > > > > in python > > > > : https://github.com/michaelmathen/FJLT > > > > < > > > > https://ml-trckr.com/link/https%3A%2F%2Fgithub.com%2Fmichaelmathen%2FFJLT/ItaYhTc7pvlDqpF6qIPH > > > > > > > > > . > > > > > > > > Are there functionalities that the core developers have in mind > > > > and can > > > > use some help implementing them? > > > > > > > > On Sat, Sep 15, 2018 at 1:04 PM Andreas Kloeckner < > > > > lists at informa.tiker.net> wrote: > > > > > > > > > Touqir, > > > > > > > > > > Touqir Sajed writes: > > > > > > There seems to be some interest in implementing randomized > > > > > > linear > > > > > > > > > > algebra > > > > > > algorithms such as JL transformation, low-rank > > > > > > approximation, etc as > > > > > > written in > > > > > > https://github.com/scipy/scipy/wiki/GSoC-2018-project- > > > > > > > > > > ideas > > > > > > > > > > > > > > > 2Fscipy%2Fscipy%2Fwiki%2FGSoC-2018-project- > > > > > ideas/cBzmzxqJ6V7Fi58kyipk> > > > > > > . To the best of my knowledge, they are not implemented > > > > > > yet. If there > > > > > > > > > > still > > > > > > exists interests I would like to implement the Fast JL > > > > > > transformation > > > > > > > > > > and > > > > > > the randomized SVD for low-rank approximation. The pseudo > > > > > > code of > > > > > > > > > > Fast JL > > > > > > transformation is given here: > > > > > > https://pdfs.semanticscholar.org/ccf0/1e0375d11df5335b645161b4833e32 > > > > > > > > > > 380d89.pdf > > > > > > > > > > > > > > > semanticscholar.org%2Fccf0%2F1e0375d11df5335b645161b4833e3238 > > > > > 0d89.pdf/ > > > > > cBzmzxqJ6V7Fi58kyipk> > > > > > > . The randomized SVD is based on > > > > > > https://arxiv.org/pdf/0909.4061.pdf > > > > > > > > > > > > > > > 2F0909.4061.pdf/cBzmzxqJ6V7Fi58kyipk> > > > > > > > > > > scipy.linalg.interpolative is in fact based on randomized > > > > > linear > > > > > algebra. In particular, this function: > > > > > > > > > > https://docs.scipy.org/doc/scipy/reference/generated/ > > > > > scipy.linalg.interpolative.svd.html#scipy.linalg.interpolativ > > > > > e.svd > > > > > > > > > > already does implement a randomized SVD with a number of > > > > > acceleration > > > > > bells and whistles. > > > > > > > > > > Andreas > > > > > > > > > > > > > > > > > -- > > > > Computing Science Master's student at University of Alberta, > > > > Canada, > > > > specializing in Machine Learning. Website : > > > > https://ca.linkedin.com/in/ > > > > touqir-sajed-6a95b1126 > > > > > > > > _______________________________________________ > > > > SciPy-Dev mailing list > > > > SciPy-Dev at python.org > > > > https://mail.python.org/mailman/listinfo/scipy-dev > > > > > > > > > > > > > > > > > -- > > > Matt Haberland > > > Assistant Adjunct Professor in the Program in Computing > > > Department of Mathematics > > > 6617A Math Sciences Building, UCLA > > > _______________________________________________ > > > SciPy-Dev mailing list > > > SciPy-Dev at python.org > > > https://mail.python.org/mailman/listinfo/scipy-dev > > > > > > > > > -- > > Computing Science Master's student at University of Alberta, > > Canada, > > specializing in Machine Learning. Website : > > https://ca.linkedin.com/in/ > > touqir-sajed-6a95b1126 > > > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at python.org > > https://mail.python.org/mailman/listinfo/scipy-dev > > > > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev From tyler.je.reddy at gmail.com Sat Sep 15 22:26:28 2018 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Sat, 15 Sep 2018 19:26:28 -0700 Subject: [SciPy-Dev] CI issues In-Reply-To: References: <99f2b67d8033d8286110e179df0c9bf174c4bcd8.camel@iki.fi> Message-ID: Still haven't heard back from codecov team On Fri, 7 Sep 2018 at 07:01, Eric Larson wrote: > I am working on it now by getting rid of the warnings. I've gotten this > warning about `qhull` before so it would be nice to fix it for good anyway. > > Eric > > > On Fri, Sep 7, 2018 at 9:24 AM Tyler Reddy > wrote: > >> Also, codecov acknowledged receipt of our support email re: recent CI >> issues ( https://github.com/scipy/scipy/pull/9186#issuecomment-419225782 >> ) -- I'll relay >> their response when I get it. >> >> On Fri, 7 Sep 2018 at 02:26, Pauli Virtanen wrote: >> >>> pe, 2018-09-07 kello 19:13 +1000, Andrew Nelson kirjoitti: >>> > > >>> > > It probably has to do with pytest 3.8 release. Numpy also had >>> > > problems: >>> > > https://github.com/numpy/numpy/pull/11896 >>> > >>> > >>> > This seems to be correct. I pinned the version to 3.7 in .travis.yml >>> > and >>> > things ran smoothly. I don't see how it's related to >>> > numpy/numpy#11896 >>> > though, that's talking about warnings, our errors are due to import >>> > problems. >>> >>> The import problem appears also due to a warning converted to error. >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sun Sep 16 03:38:23 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sun, 16 Sep 2018 00:38:23 -0700 Subject: [SciPy-Dev] Improving spatial.distance code and, performance In-Reply-To: <84a744f2-cb30-9bc3-0cfe-579fe06a0b5a@gmail.com> References: <84a744f2-cb30-9bc3-0cfe-579fe06a0b5a@gmail.com> Message-ID: On Sat, Sep 15, 2018 at 1:09 AM Kane & Anna O'Donnell < email.the.odonnells at gmail.com> wrote: > Sorry if this doesn't work - I haven't used mailing lists before, and > got the digest, which doesn't seem to allow me to reply to a specific > thread. So I'm attempting to set the In-Reply-To header manually. Sorry > if that's silly. > Yes I don't think that worked; replying requires getting individual emails rather than the digest. Anyway, not much a problem - I'll copy your reply back to the thread. Ralf > Anyway: > > Yes, I believe there are good unit tests (Ralf actually ran them I > think). From the basis of this, and the feedback in the issue/PR, it > sounds like the preference would be for me to upgrade the distance > metrics to cython bit-by-bit. > > However, I'm still unsure about scipy vs a new package, since scipy > apparently won't support dependencies. For example, I could use Faiss or > flann, but scipy wouldn't add them as a dependency. With that in mind, > it seems the only option for offering users full-functionality (aside > from rewriting those libraries from scratch etc.) is to offer a package. > Unless scipy is willing to specify a run-time dependency (not sure of > the correct terminology) - e.g. scipy doesn't depend on Faiss (when you > install it), but if the user calls a distance metric with e.g. > method='faiss', it will fail if faiss cannot be imported. Is this a > possibility (and a good idea)? > > > On 2018/09/12 4:00 am, scipy-dev-request at python.org wrote: > > Send SciPy-Dev mailing list submissions to > > scipy-dev at python.org > > > > To subscribe or unsubscribe via the World Wide Web, visit > > https://mail.python.org/mailman/listinfo/scipy-dev > > or, via email, send a message with subject or body 'help' to > > scipy-dev-request at python.org > > > > You can reach the person managing the list at > > scipy-dev-owner at python.org > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of SciPy-Dev digest..." > > > > > > Today's Topics: > > > > 1. Re: Improving spatial.distance code and performance > > (Mark Alexander Mikofski) > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Mon, 10 Sep 2018 10:05:17 -0700 > > From: Mark Alexander Mikofski > > To: SciPy Developers List > > Subject: Re: [SciPy-Dev] Improving spatial.distance code and > > performance > > Message-ID: > > E4+S6UtkoM3pFQRv8PzQQBCw+w at mail.gmail.com> > > Content-Type: text/plain; charset="utf-8" > > > > I'm very interested to see how a successful cython/performance PR > > progresses from a reviewers standpoint. > > > > On Sun, Sep 9, 2018, 5:10 PM Tyler Reddy > wrote: > > > >> Good to see some activity / interest in spatial. > >> > >> Definitely agree with Ralf's github comments re: using smaller / more > >> tractable PRs -- it really is tough to sit down at night with 30 > minutes of > >> free time or whatever and look at a massive diff & not want to give up. > >> > >> I like the idea of making small / targeted / clearly demonstrated > >> performance improvements without overhauling the entire infrastructure > >> first, but maybe that's a controversial view if it just causes too much > >> heterogeneity. > >> > >> Presumably the affected code is all thoroughly covered by unit tests? > >> That's an important pre-requisite to have the confidence to really start > >> making changes, esp. with older low-level code like that. > >> > >> On Thu, 6 Sep 2018 at 02:29, Kane & Anna O'Donnell < > >> email.the.odonnells at gmail.com> wrote: > >> > >>> Hi all, > >>> > >>> TLDR; I'm wondering about a) porting the spatial.distance code to > Cython, > >>> and b) adding some performance optimizations, and I'd like the dev > >>> community's input/feedback. > >>> > >>> For context (though I don't really need you to read these to give the > >>> feedback I'm looking for), > >>> > >>> - original issue/proposal: https://github.com/scipy/scipy/issues/9205 > >>> - PR: https://github.com/scipy/scipy/pull/9218 > >>> > >>> Before submitting the PR, I naively thought it was going to be nice > >>> Cython or similar. Turns out it's some pretty old code, that I found > pretty > >>> hard to wrap my head around and understand. I eventually figured it out > >>> after spending ages debugging a nastily hidden 'bug', and demonstrated > the > >>> performance optimizations, but it prompted the discussion about > whether it > >>> was best to port everything to Cython first. > >>> > >>> *Existing stuff to Cython* > >>> > >>> Doing so shouldn't be too hard, and it shouldn't change any > >>> functionality, except to replace the distance functions with their > Cython > >>> ones (instead of the current approach, where the distance functions are > >>> actually numpy things, and there's not supported access to the > underlying C > >>> stuff). A few possible 'bugs' (as above) should hopefully become > non-issues > >>> too. So, it'd be a win for performance (e.g. all the distance functions > >>> will be much faster), and code quality, and future maintainability and > >>> development. However, things to think about: > >>> > >>> - should I just replace like-for-like, or consider some nicer OOP stuff > >>> like e.g. sklearn's approach (which is already Cython)? > >>> > https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/neighbors/dist_metrics.pyx > >>> (I'm guessing the reason they rolled their own was because they found > scipy > >>> lacking, as above.) In fact, we could largely just copy that file > over. Not > >>> sure about the interplay between scipy and scikit learn though. > >>> - what's the best way to structure a PR? One metric at a time, or the > >>> whole caboodle? > >>> > >>> *Adding performance optimizations and future functionality* > >>> > >>> As per the links, this was initially about providing a nifty > performance > >>> hack. It should still be pretty easy to implement. Personally, I think > it > >>> makes sense to implement after the Cythonization - unless the > community are > >>> against that. > >>> > >>> However, there are other possibilities: > >>> > >>> - using 'indices' within calculations. E.g. when doing pdist, it might > >>> pay to use a tree of some description. I also proposed another 'index' > to > >>> optimize the 'bail early' approach further (which would, I think, > actually > >>> work well with trees too). This would involve more API changes, > possibly > >>> significant. > >>> - using approximate searches (e.g. Faiss). My understanding is that > >>> depending on other libraries probably isn't really an option, so I'm > not > >>> sure what means. > >>> - maybe other cool approaches like https://github.com/droyed/eucl_dist > >>> - providing a way to 'tune' distance computations to be optimal to your > >>> particular dataset and constraints (e.g. my 'bail early' optimization > might > >>> be a lot faster or a bit slower, depending on your data ... or you > might be > >>> OK with approximate matching with a 'low' error rate, etc.) > >>> > >>> I guess what I'd like to see is a single place where users can get > access > >>> to everything related to distance metrics and their uses, including all > >>> sorts of optimizations etc. (possibly for different hardware, and > possibly > >>> distributed). To do that well is a pretty big undertaking, and I don't > know > >>> whether it's suited to scipy - e.g. maybe scipy doesn't really care > about > >>> distance stuff, or only wants to stick with 'simple' distance metric > cases > >>> (e.g. a few thousand vectors, etc.). So, maybe it'd be better to start > a > >>> completely new python package - which would probably be a lot easier to > >>> develop as I'd have a lot more flexibility (e.g. to depend on other > >>> packages, and not have to worry about breaking the scipy API etc.). On > the > >>> other hand (as discussed in the latest comment on the PR), that might > not > >>> be best - it might never get used/maintained etc. > >>> > >>> So, what does the community think is the best approach? I've got too > >>> little context of what scipy is and what it's aiming for, and I don't > want > >>> to head off on the wrong tack. Comments on any of the other implied > >>> questions would also be appreciated. > >>> > >>> Thanks, > >>> > >>> kodonnell > >>> > >>> > >>> _______________________________________________ > >>> SciPy-Dev mailing list > >>> SciPy-Dev at python.org > >>> https://mail.python.org/mailman/listinfo/scipy-dev > >>> > >> _______________________________________________ > >> SciPy-Dev mailing list > >> SciPy-Dev at python.org > >> https://mail.python.org/mailman/listinfo/scipy-dev > >> > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: < > http://mail.python.org/pipermail/scipy-dev/attachments/20180910/148af471/attachment-0001.html > > > > > > ------------------------------ > > > > Subject: Digest Footer > > > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at python.org > > https://mail.python.org/mailman/listinfo/scipy-dev > > > > > > ------------------------------ > > > > End of SciPy-Dev Digest, Vol 179, Issue 12 > > ****************************************** > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sun Sep 16 04:04:24 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sun, 16 Sep 2018 01:04:24 -0700 Subject: [SciPy-Dev] Improving spatial.distance code and performance In-Reply-To: References: Message-ID: Reply from Kane to the digest: " Yes, I believe there are good unit tests (Ralf actually ran them I think). From the basis of this, and the feedback in the issue/PR, it sounds like the preference would be for me to upgrade the distance metrics to cython bit-by-bit. However, I'm still unsure about scipy vs a new package, since scipy apparently won't support dependencies. For example, I could use Faiss or flann, but scipy wouldn't add them as a dependency. With that in mind, it seems the only option for offering users full-functionality (aside from rewriting those libraries from scratch etc.) is to offer a package. Unless scipy is willing to specify a run-time dependency (not sure of the correct terminology) - e.g. scipy doesn't depend on Faiss (when you install it), but if the user calls a distance metric with e.g. method='faiss', it will fail if faiss cannot be imported. Is this a possibility (and a good idea)? " This older discussion on Flann may be relevant: https://mail.python.org/pipermail/scipy-dev/2011-May/thread.html. It says Flann only does Euclidean; not sure if that has changed since then. Regarding a dependency: https://github.com/mariusmuja/flann is basically inactivate for the last years; we wouldn't depend on it but could consider vendoring it. However, probably not worth it if it's for one method inside euclidean only. Faiss is still actively developed: https://github.com/facebookresearch/faiss, and looks like a much better option than Flann. However, something fast-moving like that which itself depends on BLAS and has GPU code in it too is not something we'd like to depend on nor want to vendor. Either way, Flann/Faiss is not about a 1:1 Cython translation, but about new features. We've got the distance metrics; approximate distances are probably best left to another package it looks like to me. On Mon, Sep 10, 2018 at 10:05 AM Mark Alexander Mikofski < mikofski at berkeley.edu> wrote: > I'm very interested to see how a successful cython/performance PR > progresses from a reviewers standpoint. > > On Sun, Sep 9, 2018, 5:10 PM Tyler Reddy wrote: > >> Good to see some activity / interest in spatial. >> >> Definitely agree with Ralf's github comments re: using smaller / more >> tractable PRs -- it really is tough to sit down at night with 30 minutes of >> free time or whatever and look at a massive diff & not want to give up. >> >> I like the idea of making small / targeted / clearly demonstrated >> performance improvements without overhauling the entire infrastructure >> first, but maybe that's a controversial view if it just causes too much >> heterogeneity. >> >> Presumably the affected code is all thoroughly covered by unit tests? >> That's an important pre-requisite to have the confidence to really start >> making changes, esp. with older low-level code like that. >> >> On Thu, 6 Sep 2018 at 02:29, Kane & Anna O'Donnell < >> email.the.odonnells at gmail.com> wrote: >> >>> Hi all, >>> >>> TLDR; I'm wondering about a) porting the spatial.distance code to >>> Cython, and b) adding some performance optimizations, and I'd like the dev >>> community's input/feedback. >>> >>> For context (though I don't really need you to read these to give the >>> feedback I'm looking for), >>> >>> - original issue/proposal: https://github.com/scipy/scipy/issues/9205 >>> - PR: https://github.com/scipy/scipy/pull/9218 >>> >>> Before submitting the PR, I naively thought it was going to be nice >>> Cython or similar. Turns out it's some pretty old code, that I found pretty >>> hard to wrap my head around and understand. I eventually figured it out >>> after spending ages debugging a nastily hidden 'bug', and demonstrated the >>> performance optimizations, but it prompted the discussion about whether it >>> was best to port everything to Cython first. >>> >>> *Existing stuff to Cython* >>> >>> Doing so shouldn't be too hard, and it shouldn't change any >>> functionality, except to replace the distance functions with their Cython >>> ones (instead of the current approach, where the distance functions are >>> actually numpy things, and there's not supported access to the underlying C >>> stuff). A few possible 'bugs' (as above) should hopefully become non-issues >>> too. So, it'd be a win for performance (e.g. all the distance functions >>> will be much faster), and code quality, and future maintainability and >>> development. However, things to think about: >>> >>> - should I just replace like-for-like, or consider some nicer OOP stuff >>> like e.g. sklearn's approach (which is already Cython)? >>> https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/neighbors/dist_metrics.pyx >>> (I'm guessing the reason they rolled their own was because they found scipy >>> lacking, as above.) In fact, we could largely just copy that file over. Not >>> sure about the interplay between scipy and scikit learn though. >>> >> It wouldn't be the first time that code is moved from scikit-learn to scipy, that could make sense. Would be good to see if that makes sense from scikit-learn's point of view. > - what's the best way to structure a PR? One metric at a time, or the >>> whole caboodle? >>> >> How about one metric first (easier to review), and then the rest in one go? >>> *Adding performance optimizations and future functionality* >>> >>> As per the links, this was initially about providing a nifty performance >>> hack. It should still be pretty easy to implement. Personally, I think it >>> makes sense to implement after the Cythonization - unless the community are >>> against that. >>> >>> However, there are other possibilities: >>> >>> - using 'indices' within calculations. E.g. when doing pdist, it might >>> pay to use a tree of some description. I also proposed another 'index' to >>> optimize the 'bail early' approach further (which would, I think, actually >>> work well with trees too). This would involve more API changes, possibly >>> significant. >>> >> - using approximate searches (e.g. Faiss). My understanding is that >>> depending on other libraries probably isn't really an option, so I'm not >>> sure what means. >>> - maybe other cool approaches like https://github.com/droyed/eucl_dist >>> - providing a way to 'tune' distance computations to be optimal to your >>> particular dataset and constraints (e.g. my 'bail early' optimization might >>> be a lot faster or a bit slower, depending on your data ... or you might be >>> OK with approximate matching with a 'low' error rate, etc.) >>> >> I think some of these are an option; they'd need to be applicable to all distance metrics though and not just euclidean or a small subset. In the mailing list thread I linked to above there was some discussion as well about using kdtree/balltree. >>> I guess what I'd like to see is a single place where users can get >>> access to everything related to distance metrics and their uses, including >>> all sorts of optimizations etc. (possibly for different hardware, and >>> possibly distributed). To do that well is a pretty big undertaking, and I >>> don't know whether it's suited to scipy - e.g. maybe scipy doesn't really >>> care about distance stuff, or only wants to stick with 'simple' distance >>> metric cases (e.g. a few thousand vectors, etc.). So, maybe it'd be better >>> to start a completely new python package - which would probably be a lot >>> easier to develop as I'd have a lot more flexibility (e.g. to depend on >>> other packages, and not have to worry about breaking the scipy API etc.). >>> On the other hand (as discussed in the latest comment on the PR), that >>> might not be best - it might never get used/maintained etc. >>> >> If you want to get really fancy, I'd lean towards a separate package. The idea of your current PR is in scopy for scipy.spatial though. We'd also be happy to link to a separate package from the scipy docs. Cheers, Ralf >>> So, what does the community think is the best approach? I've got too >>> little context of what scipy is and what it's aiming for, and I don't want >>> to head off on the wrong tack. Comments on any of the other implied >>> questions would also be appreciated. >>> >>> Thanks, >>> >>> kodonnell >>> >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From email.the.odonnells at gmail.com Sun Sep 16 04:26:18 2018 From: email.the.odonnells at gmail.com (Kane & Anna O'Donnell) Date: Sun, 16 Sep 2018 20:26:18 +1200 Subject: [SciPy-Dev] Improving spatial.distance code and performance In-Reply-To: References: Message-ID: <30c6c7dd-fbe6-4cc2-feac-1d5dbbb96b26@gmail.com> Sorry, flann and faiss were just examples (I haven't actually researched different libraries in depth). > ... approximate distances are probably best left to another package it looks like to me ... If you want to get really fancy, I'd lean towards a separate package. OK, I want to try things which it sounds like scipy won't support, so, decision made: I'll aim to create a new package. If I actually do it, and if it's actually 'good', then there'll be a better discussion point for integrating it (if at all). > This older discussion on Flann may be relevant: > https://mail.python.org/pipermail/scipy-dev/2011-May/thread.html. It > says Flann only does Euclidean; not sure if that has changed since > then. Regarding a dependency: https://github.com/mariusmuja/flann is > basically inactivate for the last years; we wouldn't depend on it but > could consider vendoring it. However, probably not worth it if it's > for one method inside euclidean only. > > Faiss is still actively developed: > https://github.com/facebookresearch/faiss, and looks like a much > better option than Flann. However, something fast-moving like that > which itself depends on BLAS and has GPU code in it too is not > something we'd like to depend on nor want to vendor. > > Either way, Flann/Faiss is not about a 1:1 Cython translation, but > about new features. We've got the distance metrics; approximate > distances are probably best left to another package it looks like to me. > > On Mon, Sep 10, 2018 at 10:05 AM Mark Alexander Mikofski > > wrote: > > I'm very interested to see how a successful cython/performance PR > progresses from a reviewers standpoint. > > On Sun, Sep 9, 2018, 5:10 PM Tyler Reddy > wrote: > > Good to see some activity / interest in spatial. > > Definitely agree with Ralf's github comments re: using smaller > / more tractable PRs -- it really is tough to sit down at > night with 30 minutes of free time or whatever and look at a > massive diff & not want to give up. > > I like the idea of making small / targeted / clearly > demonstrated performance improvements without overhauling the > entire infrastructure first, but maybe that's a controversial > view if it just causes too much heterogeneity. > > Presumably the affected code is all thoroughly covered by unit > tests? That's an important pre-requisite to have the > confidence to really start making changes, esp. with older > low-level code like that. > > On Thu, 6 Sep 2018 at 02:29, Kane & Anna O'Donnell > > wrote: > > Hi all, > > TLDR; I'm wondering about a) porting the spatial.distance > code to Cython, and b) adding some performance > optimizations, and I'd like the dev community's > input/feedback. > > For context (though I don't really need you to read these > to give the feedback I'm looking for), > > - original issue/proposal: > https://github.com/scipy/scipy/issues/9205 > - PR: https://github.com/scipy/scipy/pull/9218 > > Before submitting the PR, I naively thought it was going > to be nice Cython or similar. Turns out it's some pretty > old code, that I found pretty hard to wrap my head around > and understand. I eventually figured it out after spending > ages debugging a nastily hidden 'bug', and demonstrated > the performance optimizations, but it prompted the > discussion about whether it was best to port everything to > Cython first. > > *Existing stuff to Cython* > > Doing so shouldn't be too hard, and it shouldn't change > any functionality, except to replace the distance > functions with their Cython ones (instead of the current > approach, where the distance functions are actually numpy > things, and there's not supported access to the underlying > C stuff). A few possible 'bugs' (as above) should > hopefully become non-issues too. So, it'd be a win for > performance (e.g. all the distance functions will be much > faster), and code quality, and future maintainability and > development. However, things to think about: > > - should I just replace like-for-like, or consider some > nicer OOP stuff like e.g. sklearn's approach (which is > already Cython)? > https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/neighbors/dist_metrics.pyx > (I'm guessing the reason they rolled their own was because > they found scipy lacking, as above.) In fact, we could > largely just copy that file over. Not sure about the > interplay between scipy and scikit learn though. > > > It wouldn't be the first time that code is moved from scikit-learn to > scipy, that could make sense. Would be good to see if that makes sense > from scikit-learn's point of view. > > - what's the best way to structure a PR? One metric at a > time, or the whole caboodle? > > > How about one metric first (easier to review), and then the rest in > one go? > > > *Adding performance optimizations and future functionality* > * > * > As per the links, this was initially about providing a > nifty performance hack. It should still be pretty easy to > implement. Personally, I think it makes sense to implement > after the Cythonization - unless the community are against > that. > > However, there are other possibilities: > > - using 'indices' within calculations. E.g. when doing > pdist, it might pay to use a tree of some description. I > also proposed another 'index' to optimize the 'bail early' > approach further (which would, I think, actually work well > with trees too). This would involve more API changes, > possibly significant. > > - using approximate searches (e.g. Faiss). My > understanding is that depending on other libraries > probably isn't really an option, so I'm not sure what means. > - maybe other cool approaches like > https://github.com/droyed/eucl_dist > - providing a way to 'tune' distance computations to be > optimal to your particular dataset and constraints (e.g. > my 'bail early' optimization might be a lot faster or a > bit slower, depending on your data ... or you might be OK > with approximate matching with a 'low' error rate, etc.) > > > I think some of these are an option; they'd need to be applicable to > all distance metrics though and not just euclidean or a small subset. > In the mailing list thread I linked to above there was some discussion > as well about using kdtree/balltree. > > > I guess what I'd like to see is a single place where users > can get access to everything related to distance metrics > and their uses, including all sorts of optimizations etc. > (possibly for different hardware, and possibly > distributed). To do that well is a pretty big undertaking, > and I don't know whether it's suited to scipy - e.g. maybe > scipy doesn't really care about distance stuff, or only > wants to stick with 'simple' distance metric cases (e.g. a > few thousand vectors, etc.). So, maybe it'd be better to > start a completely new python package - which would > probably be a lot easier to develop as I'd have a lot more > flexibility (e.g. to depend on other packages, and not > have to worry about breaking the scipy API etc.). On the > other hand (as discussed in the latest comment on the PR), > that might not be best - it might never get > used/maintained etc. > > > If you want to get really fancy, I'd lean towards a separate package. > The idea of your current PR is in scopy for scipy.spatial though. We'd > also be happy to link to a separate package from the scipy docs. > > Cheers, > Ralf > > > > So, what does the community think is the best approach? > I've got too little context of what scipy is and what it's > aiming for, and I don't want to head off on the wrong > tack. Comments on any of the other implied questions would > also be appreciated. > > Thanks, > > kodonnell > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From serge.guelton at telecom-bretagne.eu Sun Sep 16 08:09:23 2018 From: serge.guelton at telecom-bretagne.eu (Serge Guelton) Date: Sun, 16 Sep 2018 14:09:23 +0200 Subject: [SciPy-Dev] Pythran - 0.8.7 Message-ID: <20180916120923.GA28119@lakota> Hi there, Pythran just released its 0.8.7 version. A quick reminder: Pythran is a compiler for scientific Python, it can be used to turn Python kernels as: #pythran export laplacien(float64[][][3]) import numpy as np def laplacien(image): out_image = np.abs(4*image[1:-1,1:-1] - image[0:-2,1:-1] - image[2:,1:-1] - image[1:-1,0:-2] - image[1:-1,2:]) valmax = np.max(out_image) valmax = max(1.,valmax)+1.E-9 out_image /= valmax return out_image In a native library that runs much faster. Note the ``#pythran export`` line that specifies that the native version only accepts 3D arrays with last dimension set to 3. It's available on pypi: https://pypi.org/project/pythran/ on conda-forge (linux and osx): https://anaconda.org/conda-forge/pythran and on github: https://github.com/serge-sans-paille/pythran Special Thanks for this release to: - Yann Diorcet for all his work on the Windows compatibility - Jean Laroche for the bug reports on OSX - Marteen for his ? red dragon ? gift - h-vetinari, vgroff, DerWeh, ucyo, Chronum94, paugier, gouarin and Dapid for their bug reports. - alexbw for setting up the conda-forge build This release closes the following issues: https://github.com/serge-sans-paille/pythran/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aclosed+updated%3A%3E2018-06-01+ A more detailed changelog is available at https://pythran.readthedocs.io/en/latest/Changelog.html Enjoy, and if not, submit bug report :-) Serge From sylvain.corlay at gmail.com Sun Sep 16 10:42:16 2018 From: sylvain.corlay at gmail.com (Sylvain Corlay) Date: Sun, 16 Sep 2018 16:42:16 +0200 Subject: [SciPy-Dev] Pythran - 0.8.7 In-Reply-To: <20180916120923.GA28119@lakota> References: <20180916120923.GA28119@lakota> Message-ID: Hi Serge, Congrats on the release! Still amazed by the performances of Pythran. Cheers, Sylvain On Sun, Sep 16, 2018 at 2:19 PM Serge Guelton < serge.guelton at telecom-bretagne.eu> wrote: > Hi there, > > Pythran just released its 0.8.7 version. A quick reminder: Pythran is a > compiler for scientific Python, it can be used to turn Python kernels > as: > > #pythran export laplacien(float64[][][3]) > import numpy as np > def laplacien(image): > out_image = np.abs(4*image[1:-1,1:-1] - > image[0:-2,1:-1] - image[2:,1:-1] - > image[1:-1,0:-2] - image[1:-1,2:]) > valmax = np.max(out_image) > valmax = max(1.,valmax)+1.E-9 > out_image /= valmax > return out_image > > In a native library that runs much faster. Note the ``#pythran export`` > line that specifies that the native version only accepts 3D arrays with > last dimension set to 3. > > It's available on pypi: https://pypi.org/project/pythran/ > on conda-forge (linux and osx): https://anaconda.org/conda-forge/pythran > and on github: https://github.com/serge-sans-paille/pythran > > Special Thanks for this release to: > > - Yann Diorcet for all his work on the Windows compatibility > - Jean Laroche for the bug reports on OSX > - Marteen for his ? red dragon ? gift > - h-vetinari, vgroff, DerWeh, ucyo, Chronum94, paugier, gouarin and Dapid > for their bug reports. > - alexbw for setting up the conda-forge build > > This release closes the following issues: > > > https://github.com/serge-sans-paille/pythran/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aclosed+updated%3A%3E2018-06-01+ > > A more detailed changelog is available at > https://pythran.readthedocs.io/en/latest/Changelog.html > > Enjoy, and if not, submit bug report :-) > > Serge > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cimrman3 at ntc.zcu.cz Mon Sep 17 05:45:11 2018 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Mon, 17 Sep 2018 11:45:11 +0200 Subject: [SciPy-Dev] ANN: SfePy 2018.3 Message-ID: <8de51b72-39b8-d407-471a-d8a7b5b2703e@ntc.zcu.cz> I am pleased to announce release 2018.3 of SfePy. Description ----------- SfePy (simple finite elements in Python) is a software for solving systems of coupled partial differential equations by the finite element method or by the isogeometric analysis (limited support). It is distributed under the new BSD license. Home page: http://sfepy.org Mailing list: https://mail.python.org/mm3/mailman3/lists/sfepy.python.org/ Git (source) repository, issue tracker: https://github.com/sfepy/sfepy Highlights of this release -------------------------- - easier setting of values of variables - new script for outline edge extraction - new example: homogenization of a piezoelectric heterogeneous structure For full release notes see http://docs.sfepy.org/doc/release_notes.html#id1 (rather long and technical). Cheers, Robert Cimrman --- Contributors to this release in alphabetical order: Robert Cimrman Vladimir Lukes From juanlu001 at gmail.com Mon Sep 17 06:00:34 2018 From: juanlu001 at gmail.com (Juan Luis Cano) Date: Mon, 17 Sep 2018 12:00:34 +0200 Subject: [SciPy-Dev] Dormand-Prince 8(5,3) in new SciPy IVP integrators Message-ID: Hi all, We wanted to use the new initial value problem solvers present in SciPy but the DOP853 method was not present there, so we implemented it ourselves: https://github.com/poliastro/poliastro/blob/v0.11b1/src/poliastro/integrators.py We have done some soft validation (i.e. it has worked well so far for our specific use case) and we think it would be great to have it in SciPy, given that it was present in the "Old API" and that it's a widely used method in Celestial Mechanics. If I understood correctly, in-depth quantitative validation was not written for the current solvers: https://github.com/scipy/scipy/pull/6326#issuecomment-250830082 Would you be interested in a PR? For testing, would it be enough to add the new method here? https://github.com/scipy/scipy/blob/4d16aea1267898e8532f962f0104d485578ab670/scipy/integrate/tests/test_ivp.py#L147 Regards, -- Juan Luis Cano -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Sep 17 12:07:58 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 17 Sep 2018 09:07:58 -0700 Subject: [SciPy-Dev] welcome Kai to the core team Message-ID: Hi all, On behalf of the SciPy developers I'd like to welcome Kai Striega as a member of the core dev team. Kai has been contributing to scipy.optimize since June, and has so far made a number of improvements to the simplex method of linprog that fixed all of the many known bugs - https://github.com/scipy/scipy/pulls/Kai-Striega I'm looking forward to his continued contributions! Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikofski at berkeley.edu Mon Sep 17 12:16:19 2018 From: mikofski at berkeley.edu (Mark Alexander Mikofski) Date: Mon, 17 Sep 2018 09:16:19 -0700 Subject: [SciPy-Dev] welcome Kai to the core team In-Reply-To: References: Message-ID: Congratulations Kai! Thanks for your work on SciPy optimize! On Mon, Sep 17, 2018 at 9:08 AM Ralf Gommers wrote: > Hi all, > > On behalf of the SciPy developers I'd like to welcome Kai Striega as a > member of the core dev team. > > Kai has been contributing to scipy.optimize since June, and has so far > made a number of improvements to the simplex method of linprog that fixed > all of the many known bugs - > https://github.com/scipy/scipy/pulls/Kai-Striega > > I'm looking forward to his continued contributions! > > Cheers, > Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -- Mark Mikofski, PhD (2005) *Fiat Lux* -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Mon Sep 17 12:31:02 2018 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Mon, 17 Sep 2018 12:31:02 -0400 Subject: [SciPy-Dev] welcome Kai to the core team In-Reply-To: References: Message-ID: On Mon, Sep 17, 2018 at 12:08 PM Ralf Gommers wrote: > Hi all, > > On behalf of the SciPy developers I'd like to welcome Kai Striega as a > member of the core dev team. > > Kai has been contributing to scipy.optimize since June, and has so far > made a number of improvements to the simplex method of linprog that fixed > all of the many known bugs - > https://github.com/scipy/scipy/pulls/Kai-Striega > > I'm looking forward to his continued contributions! > > Welcome, Kai. Keep up the great work! Warren Cheers, > Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Mon Sep 17 12:39:02 2018 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Mon, 17 Sep 2018 09:39:02 -0700 Subject: [SciPy-Dev] welcome Kai to the core team In-Reply-To: References: Message-ID: <165e8664870.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> Congratulations, Kai, and welcome! We are happy to have you on board. Best regards, St?fan On September 17, 2018 09:31:42 Warren Weckesser wrote: > > > On Mon, Sep 17, 2018 at 12:08 PM Ralf Gommers wrote: > Hi all, > > On behalf of the SciPy developers I'd like to welcome Kai Striega as a > member of the core dev team. > > Kai has been contributing to scipy.optimize since June, and has so far made > a number of improvements to the simplex method of linprog that fixed all of > the many known bugs - https://github.com/scipy/scipy/pulls/Kai-Striega > > > I'm looking forward to his continued contributions! > > > > Welcome, Kai. Keep up the great work! > > Warren > > > Cheers, > Ralf > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From phillip.m.feldman at gmail.com Mon Sep 17 14:43:39 2018 From: phillip.m.feldman at gmail.com (Phillip Feldman) Date: Mon, 17 Sep 2018 11:43:39 -0700 Subject: [SciPy-Dev] [GSOC 2018 Project Thread]: Rotation formalism in 3 dimensions In-Reply-To: References: <163213d9b99.e31ddfbf140246.3867213885941953730@zoho.com> Message-ID: I've noticed that the date of the last commit is June 24, which suggests that this activity is moribund. This is unfortunate, because it seemed really promising. On Mon, Jun 4, 2018 at 1:51 PM Aditya Bharti wrote: > We do plan to create a tutorial with examples highlighting the major > functionality of the module and some common use cases, but its exact nature > and extent will depend upon how much time we have left after completing the > implementation (including docstrings and tests). > > Best, > Aditya > > On Mon, 4 Jun 2018 at 09:45, Phillip Feldman > wrote: > >> "Refer to docs for more details." Will there be user documentation >> beyond the doc strings in the various methods? >> >> On Tue, May 29, 2018 at 2:21 PM, Aditya Bharti >> wrote: >> >>> Hi all, >>> Continuing the work so far, the following have been implemented this >>> week: >>> >>> - `from_rotvec`, `as_rotvec`: Representing Euler angles as rotation >>> vectors, with appropriate Taylor series expansions for small angles >>> - `from_euler`: Initialization from Euler angles, along with a >>> string based axis sequence specification. Refer to docs for more details. >>> >>> As always, the project lives here >>> , >>> and my personal experiences can be found on the blog >>> . >>> >>> Thanks, >>> Aditya >>> >>> On Sun, 20 May 2018 at 13:28, Phillip Feldman < >>> phillip.m.feldman at gmail.com> wrote: >>> >>>> When you say "discrete cosine matrix", I think that you mean "direction >>>> cosine matrix" (see >>>> https://en.wikiversity.org/wiki/PlanetPhysics/Direction_Cosine_Matrix). >>>> >>>> Phillip >>>> >>>> On Sat, May 19, 2018 at 12:23 PM, Aditya Bharti >>>> wrote: >>>> >>>>> Hi all, >>>>> So this concludes week 1 of GSoC 2018. I must say it was a great >>>>> learning experience and I invite you all to check out my account of this >>>>> week on the blog . This >>>>> email is more of a technical update. >>>>> >>>>> - So, the main `Rotation` class will live under a new sub module >>>>> `scipy.spatial.transform`. >>>>> - Conversion between quaternions and discrete cosine matrices was >>>>> implemented. >>>>> - The rotation class now supports `from_quaternion`, `from_dcm`, >>>>> `as_quaternion` and `as_dcm`, with support for multiple rotations in one >>>>> call. >>>>> >>>>> The project currently lives in my own fork of scipy here >>>>> . >>>>> Stay tuned for more updates! >>>>> >>>>> Best, >>>>> Aditya >>>>> >>>>> On Wed, 2 May 2018 at 21:03, Aditya Bharti >>>>> wrote: >>>>> >>>>>> Hi Nikolay, >>>>>> >>>>>> I've used Wordpress only once before, so I don't know much about it. >>>>>> From my limited experience, it is extremely customizable. You can customize >>>>>> every thing from the look and feel to SEO characteristics. There are >>>>>> apparently a lot of wordpress plugins for these kind of tasks. For this >>>>>> particular blog, PSF had already setup an account for me with a site on it. >>>>>> All I had to do was click on the 'New' button and open up the new post >>>>>> page. There's a field for a header and body text, with options for adding >>>>>> audio, video and hyperlinks. >>>>>> >>>>>> As regards to the post itself, sure I'll expand it with a brief >>>>>> overview, motivation and an example. Note that the example will only show >>>>>> sample usage, not any internals. I plan to borrow heavily from my proposal >>>>>> for this purpose, I hope that's ok. >>>>>> >>>>>> Regards, >>>>>> Aditya >>>>>> >>>>>> On 2 May 2018 at 19:54, Nikolay Mayorov >>>>>> wrote: >>>>>> >>>>>>> Hi, Aditya! >>>>>>> >>>>>>> Glad that you set up the blog and good job on setting up the >>>>>>> documentation build as well. >>>>>>> >>>>>>> Curious, what is this blogging platform like? How do you create >>>>>>> posts in it? >>>>>>> >>>>>>> As for your first post: while not strictly necessary I think it >>>>>>> would be nice to see a more thorough introductory post with a brief >>>>>>> overview, motivation and/or an example. Do you want to work on it? >>>>>>> >>>>>>> Best, >>>>>>> Nikolay >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> SciPy-Dev mailing list >>>>>>> SciPy-Dev at python.org >>>>>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>>>>> >>>>>>> >>>>>> >>>>> _______________________________________________ >>>>> SciPy-Dev mailing list >>>>> SciPy-Dev at python.org >>>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>>> >>>>> >>>> _______________________________________________ >>>> SciPy-Dev mailing list >>>> SciPy-Dev at python.org >>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>> >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >>> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Sep 17 15:41:06 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 17 Sep 2018 12:41:06 -0700 Subject: [SciPy-Dev] [GSOC 2018 Project Thread]: Rotation formalism in 3 dimensions In-Reply-To: References: <163213d9b99.e31ddfbf140246.3867213885941953730@zoho.com> Message-ID: On Mon, Sep 17, 2018 at 11:44 AM Phillip Feldman < phillip.m.feldman at gmail.com> wrote: > I've noticed that the date of the last commit is June 24, which suggests > that this activity is moribund. This is unfortunate, because it seemed > really promising. > You probably checked the wrong branch. Aditya's PRs are here: https://github.com/scipy/scipy/pulls/adbugger A good part of his work is merged, docs at http://scipy.github.io/devdocs/spatial.transform.html#module-scipy.spatial.transform Cheers, Ralf > On Mon, Jun 4, 2018 at 1:51 PM Aditya Bharti wrote: > >> We do plan to create a tutorial with examples highlighting the major >> functionality of the module and some common use cases, but its exact nature >> and extent will depend upon how much time we have left after completing the >> implementation (including docstrings and tests). >> >> Best, >> Aditya >> >> On Mon, 4 Jun 2018 at 09:45, Phillip Feldman >> wrote: >> >>> "Refer to docs for more details." Will there be user documentation >>> beyond the doc strings in the various methods? >>> >>> On Tue, May 29, 2018 at 2:21 PM, Aditya Bharti >>> wrote: >>> >>>> Hi all, >>>> Continuing the work so far, the following have been implemented this >>>> week: >>>> >>>> - `from_rotvec`, `as_rotvec`: Representing Euler angles as rotation >>>> vectors, with appropriate Taylor series expansions for small angles >>>> - `from_euler`: Initialization from Euler angles, along with a >>>> string based axis sequence specification. Refer to docs for more details. >>>> >>>> As always, the project lives here >>>> , >>>> and my personal experiences can be found on the blog >>>> . >>>> >>>> Thanks, >>>> Aditya >>>> >>>> On Sun, 20 May 2018 at 13:28, Phillip Feldman < >>>> phillip.m.feldman at gmail.com> wrote: >>>> >>>>> When you say "discrete cosine matrix", I think that you mean >>>>> "direction cosine matrix" (see >>>>> https://en.wikiversity.org/wiki/PlanetPhysics/Direction_Cosine_Matrix >>>>> ). >>>>> >>>>> Phillip >>>>> >>>>> On Sat, May 19, 2018 at 12:23 PM, Aditya Bharti >>>>> wrote: >>>>> >>>>>> Hi all, >>>>>> So this concludes week 1 of GSoC 2018. I must say it was a great >>>>>> learning experience and I invite you all to check out my account of this >>>>>> week on the blog . >>>>>> This email is more of a technical update. >>>>>> >>>>>> - So, the main `Rotation` class will live under a new sub module >>>>>> `scipy.spatial.transform`. >>>>>> - Conversion between quaternions and discrete cosine matrices was >>>>>> implemented. >>>>>> - The rotation class now supports `from_quaternion`, `from_dcm`, >>>>>> `as_quaternion` and `as_dcm`, with support for multiple rotations in one >>>>>> call. >>>>>> >>>>>> The project currently lives in my own fork of scipy here >>>>>> . >>>>>> Stay tuned for more updates! >>>>>> >>>>>> Best, >>>>>> Aditya >>>>>> >>>>>> On Wed, 2 May 2018 at 21:03, Aditya Bharti >>>>>> wrote: >>>>>> >>>>>>> Hi Nikolay, >>>>>>> >>>>>>> I've used Wordpress only once before, so I don't know much about it. >>>>>>> From my limited experience, it is extremely customizable. You can customize >>>>>>> every thing from the look and feel to SEO characteristics. There are >>>>>>> apparently a lot of wordpress plugins for these kind of tasks. For this >>>>>>> particular blog, PSF had already setup an account for me with a site on it. >>>>>>> All I had to do was click on the 'New' button and open up the new post >>>>>>> page. There's a field for a header and body text, with options for adding >>>>>>> audio, video and hyperlinks. >>>>>>> >>>>>>> As regards to the post itself, sure I'll expand it with a brief >>>>>>> overview, motivation and an example. Note that the example will only show >>>>>>> sample usage, not any internals. I plan to borrow heavily from my proposal >>>>>>> for this purpose, I hope that's ok. >>>>>>> >>>>>>> Regards, >>>>>>> Aditya >>>>>>> >>>>>>> On 2 May 2018 at 19:54, Nikolay Mayorov >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, Aditya! >>>>>>>> >>>>>>>> Glad that you set up the blog and good job on setting up the >>>>>>>> documentation build as well. >>>>>>>> >>>>>>>> Curious, what is this blogging platform like? How do you create >>>>>>>> posts in it? >>>>>>>> >>>>>>>> As for your first post: while not strictly necessary I think it >>>>>>>> would be nice to see a more thorough introductory post with a brief >>>>>>>> overview, motivation and/or an example. Do you want to work on it? >>>>>>>> >>>>>>>> Best, >>>>>>>> Nikolay >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> SciPy-Dev mailing list >>>>>>>> SciPy-Dev at python.org >>>>>>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>>>>>> >>>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> SciPy-Dev mailing list >>>>>> SciPy-Dev at python.org >>>>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> SciPy-Dev mailing list >>>>> SciPy-Dev at python.org >>>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>>> >>>> >>>> _______________________________________________ >>>> SciPy-Dev mailing list >>>> SciPy-Dev at python.org >>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>> >>>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phillip.m.feldman at gmail.com Mon Sep 17 16:02:46 2018 From: phillip.m.feldman at gmail.com (Phillip Feldman) Date: Mon, 17 Sep 2018 13:02:46 -0700 Subject: [SciPy-Dev] [GSOC 2018 Project Thread]: Rotation formalism in 3 dimensions In-Reply-To: References: <163213d9b99.e31ddfbf140246.3867213885941953730@zoho.com> Message-ID: That's tremendous. My apologies for the false alarm! Phillip On Mon, Sep 17, 2018 at 12:41 PM Ralf Gommers wrote: > > > On Mon, Sep 17, 2018 at 11:44 AM Phillip Feldman < > phillip.m.feldman at gmail.com> wrote: > >> I've noticed that the date of the last commit is June 24, which suggests >> that this activity is moribund. This is unfortunate, because it seemed >> really promising. >> > > You probably checked the wrong branch. Aditya's PRs are here: > https://github.com/scipy/scipy/pulls/adbugger > > A good part of his work is merged, docs at > http://scipy.github.io/devdocs/spatial.transform.html#module-scipy.spatial.transform > > Cheers, > Ralf > > >> On Mon, Jun 4, 2018 at 1:51 PM Aditya Bharti wrote: >> >>> We do plan to create a tutorial with examples highlighting the major >>> functionality of the module and some common use cases, but its exact nature >>> and extent will depend upon how much time we have left after completing the >>> implementation (including docstrings and tests). >>> >>> Best, >>> Aditya >>> >>> On Mon, 4 Jun 2018 at 09:45, Phillip Feldman < >>> phillip.m.feldman at gmail.com> wrote: >>> >>>> "Refer to docs for more details." Will there be user documentation >>>> beyond the doc strings in the various methods? >>>> >>>> On Tue, May 29, 2018 at 2:21 PM, Aditya Bharti >>>> wrote: >>>> >>>>> Hi all, >>>>> Continuing the work so far, the following have been implemented this >>>>> week: >>>>> >>>>> - `from_rotvec`, `as_rotvec`: Representing Euler angles as >>>>> rotation vectors, with appropriate Taylor series expansions for small angles >>>>> - `from_euler`: Initialization from Euler angles, along with a >>>>> string based axis sequence specification. Refer to docs for more details. >>>>> >>>>> As always, the project lives here >>>>> , >>>>> and my personal experiences can be found on the blog >>>>> . >>>>> >>>>> Thanks, >>>>> Aditya >>>>> >>>>> On Sun, 20 May 2018 at 13:28, Phillip Feldman < >>>>> phillip.m.feldman at gmail.com> wrote: >>>>> >>>>>> When you say "discrete cosine matrix", I think that you mean >>>>>> "direction cosine matrix" (see >>>>>> https://en.wikiversity.org/wiki/PlanetPhysics/Direction_Cosine_Matrix >>>>>> ). >>>>>> >>>>>> Phillip >>>>>> >>>>>> On Sat, May 19, 2018 at 12:23 PM, Aditya Bharti >>>>>> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> So this concludes week 1 of GSoC 2018. I must say it was a great >>>>>>> learning experience and I invite you all to check out my account of this >>>>>>> week on the blog . >>>>>>> This email is more of a technical update. >>>>>>> >>>>>>> - So, the main `Rotation` class will live under a new sub module >>>>>>> `scipy.spatial.transform`. >>>>>>> - Conversion between quaternions and discrete cosine matrices >>>>>>> was implemented. >>>>>>> - The rotation class now supports `from_quaternion`, `from_dcm`, >>>>>>> `as_quaternion` and `as_dcm`, with support for multiple rotations in one >>>>>>> call. >>>>>>> >>>>>>> The project currently lives in my own fork of scipy here >>>>>>> . >>>>>>> Stay tuned for more updates! >>>>>>> >>>>>>> Best, >>>>>>> Aditya >>>>>>> >>>>>>> On Wed, 2 May 2018 at 21:03, Aditya Bharti >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Nikolay, >>>>>>>> >>>>>>>> I've used Wordpress only once before, so I don't know much about >>>>>>>> it. From my limited experience, it is extremely customizable. You can >>>>>>>> customize every thing from the look and feel to SEO characteristics. There >>>>>>>> are apparently a lot of wordpress plugins for these kind of tasks. For this >>>>>>>> particular blog, PSF had already setup an account for me with a site on it. >>>>>>>> All I had to do was click on the 'New' button and open up the new post >>>>>>>> page. There's a field for a header and body text, with options for adding >>>>>>>> audio, video and hyperlinks. >>>>>>>> >>>>>>>> As regards to the post itself, sure I'll expand it with a brief >>>>>>>> overview, motivation and an example. Note that the example will only show >>>>>>>> sample usage, not any internals. I plan to borrow heavily from my proposal >>>>>>>> for this purpose, I hope that's ok. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Aditya >>>>>>>> >>>>>>>> On 2 May 2018 at 19:54, Nikolay Mayorov >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi, Aditya! >>>>>>>>> >>>>>>>>> Glad that you set up the blog and good job on setting up the >>>>>>>>> documentation build as well. >>>>>>>>> >>>>>>>>> Curious, what is this blogging platform like? How do you create >>>>>>>>> posts in it? >>>>>>>>> >>>>>>>>> As for your first post: while not strictly necessary I think it >>>>>>>>> would be nice to see a more thorough introductory post with a brief >>>>>>>>> overview, motivation and/or an example. Do you want to work on it? >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Nikolay >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> SciPy-Dev mailing list >>>>>>>>> SciPy-Dev at python.org >>>>>>>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> SciPy-Dev mailing list >>>>>>> SciPy-Dev at python.org >>>>>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> SciPy-Dev mailing list >>>>>> SciPy-Dev at python.org >>>>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>>>> >>>>> >>>>> _______________________________________________ >>>>> SciPy-Dev mailing list >>>>> SciPy-Dev at python.org >>>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>>> >>>>> >>>> _______________________________________________ >>>> SciPy-Dev mailing list >>>> SciPy-Dev at python.org >>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From touqir at ualberta.ca Tue Sep 18 01:31:13 2018 From: touqir at ualberta.ca (Touqir Sajed) Date: Mon, 17 Sep 2018 23:31:13 -0600 Subject: [SciPy-Dev] Regarding presolve methods used in scipy's linear programming implementation Message-ID: This question is regarding how well the scipy's linear programming's presolve implementation method works for sparse matrices using scipy.sparse.linalg.spsolve compared to scipy.linalg.lu_solve. For reference : line 322 and 201 in https://github.com/scipy/scipy/blob/master/scipy/optimize/_remove_redundancy.py . scipy.sparse.linalg.spsolve uses the sparse linear system solver from scikit-umfpack (at the moment I am not interested in the SuperLU library) and have looked into several academic papers regarding the approaches used in scikit-umfpack which are Unsymmetric MultiFrontal methods but none of them has analyzed the runtime complexity. Hence it would be useful to know its empirical performance in comparison to an existing non-sparse linear system solver implementation. I asked Matt and wanted to know if anyone else has experimented with these. Thanks! -- Computing Science Master's student at University of Alberta, Canada, specializing in Machine Learning. Website : https://ca.linkedin.com/in/touqir-sajed-6a95b1126 -------------- next part -------------- An HTML attachment was scrubbed... URL: From haberland at ucla.edu Tue Sep 18 01:55:37 2018 From: haberland at ucla.edu (Matt Haberland) Date: Mon, 17 Sep 2018 22:55:37 -0700 Subject: [SciPy-Dev] Regarding presolve methods used in scipy's linear programming implementation In-Reply-To: References: Message-ID: We can't assume that everyone has UMFPACK, because it's tough to get on Windows. And for the performance of `_remove_redundancy_sparse`, it's not so important whether the user has UMFPACK installed or not / which solver actual runs. The real problem is that I didn't assume users would have UMFPACK when I wrote it; I assumed only SuperLU, and therefore I didn't write important code for `_remove_redundancy_sparse` that would yield a big performance boost due to an issue with SuperLU . Nonetheless, `_remove_redundancy_sparse` works better for large, sparse problems than `_remove_redundancy_dense` and `_remove_redundancy`. But I think it can be a lot better, so I was thinking it would be nice to see what randomized algorithms can do for the problem of redundancy detection. On Mon, Sep 17, 2018 at 10:35 PM Touqir Sajed wrote: > This question is regarding how well the scipy's linear programming's > presolve implementation method works for sparse matrices using > scipy.sparse.linalg.spsolve compared to scipy.linalg.lu_solve. For > reference : line 322 and 201 in > https://github.com/scipy/scipy/blob/master/scipy/optimize/_remove_redundancy.py > > . > > scipy.sparse.linalg.spsolve uses the sparse linear system solver from scikit-umfpack (at > the moment I am not interested in the SuperLU library) and have looked into > several academic papers regarding the approaches used in > scikit-umfpack which are Unsymmetric MultiFrontal methods but none of them > has analyzed the runtime complexity. Hence it would be useful to know its > empirical performance in comparison to an existing non-sparse linear system > solver implementation. I asked Matt and wanted to know if anyone else has > experimented with these. Thanks! > > > -- > Computing Science Master's student at University of Alberta, Canada, > specializing in Machine Learning. Website : > https://ca.linkedin.com/in/touqir-sajed-6a95b1126 > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -- Matt Haberland Assistant Adjunct Professor in the Program in Computing Department of Mathematics 6617A Math Sciences Building, UCLA -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlucente at pipeline.com Mon Sep 17 16:41:27 2018 From: rlucente at pipeline.com (rlucente at pipeline.com) Date: Mon, 17 Sep 2018 16:41:27 -0400 Subject: [SciPy-Dev] Intel's Math Kernel Library (MKL) - MKL DNN Message-ID: <010b01d44ec6$ceeebab0$6ccc3010$@pipeline.com> I think that most people on this mailing list are aware of Intel?s Math Kernel Library (MKL) https://software.intel.com/en-us/mkl Since neural networks are now all the rage, they now have MKL-DNN https://software.intel.com/en-us/articles/intel-mkl-dnn-part-1-library-overview-and-installation They actually did a presentation at O?Reilly?s Strata https://www.oreilly.com/ideas/accelerating-ai-on-xeon-through-sw-optimization Thought that people on this mailing list might be interested in it -------------- next part -------------- An HTML attachment was scrubbed... URL: From nikolay.mayorov at zoho.com Tue Sep 18 02:36:46 2018 From: nikolay.mayorov at zoho.com (Nikolay Mayorov) Date: Tue, 18 Sep 2018 11:36:46 +0500 Subject: [SciPy-Dev] Dormand-Prince 8(5,3) in new SciPy IVP integrators In-Reply-To: References: Message-ID: <165eb65436a.1234286673798.6479979211901456586@zoho.com> Hi, Juan! I know that several people were interested to see this method implemented. I know it has a difficult logic for step size control so I couldn't implement it quickly. Now that you have a working version (say it might contain some small errors, but still the core logic is here)? conforming the new API I think you should definitely put the pull request. Perhaps adding a new simple example from celestial mechanics (say) to demonstrate how it works will be useful and interesting too (say compare lower order RK methods with it). Please create the PR and mention @nmayorov here. Best, Nikolay ---- On Mon, 17 Sep 2018 15:00:34 +0500?Juan Luis Cano wrote ---- Hi all, We wanted to use the new initial value problem solvers present in SciPy but the DOP853 method was not present there, so we implemented it ourselves: https://github.com/poliastro/poliastro/blob/v0.11b1/src/poliastro/integrators.py We have done some soft validation (i.e. it has worked well so far for our specific use case) and we think it would be great to have it in SciPy, given that it was present in the "Old API" and that it's a widely used method in Celestial Mechanics. If I understood correctly, in-depth quantitative validation was not written for the current solvers: https://github.com/scipy/scipy/pull/6326#issuecomment-250830082 Would you be interested in a PR? For testing, would it be enough to add the new method here? https://github.com/scipy/scipy/blob/4d16aea1267898e8532f962f0104d485578ab670/scipy/integrate/tests/test_ivp.py#L147 Regards, -- Juan Luis Cano _______________________________________________ SciPy-Dev mailing list SciPy-Dev at python.org https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From evgeny.burovskiy at gmail.com Tue Sep 18 11:24:36 2018 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Tue, 18 Sep 2018 18:24:36 +0300 Subject: [SciPy-Dev] welcome Kai to the core team In-Reply-To: References: Message-ID: Welcome Kai! On Mon, Sep 17, 2018 at 7:08 PM Ralf Gommers wrote: > > Hi all, > > On behalf of the SciPy developers I'd like to welcome Kai Striega as a member of the core dev team. > > Kai has been contributing to scipy.optimize since June, and has so far made a number of improvements to the simplex method of linprog that fixed all of the many known bugs - https://github.com/scipy/scipy/pulls/Kai-Striega > > I'm looking forward to his continued contributions! > > Cheers, > Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev From einstein.edison at gmail.com Tue Sep 18 11:26:24 2018 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Tue, 18 Sep 2018 17:26:24 +0200 Subject: [SciPy-Dev] welcome Kai to the core team In-Reply-To: References: Message-ID: <11d6cb2f-2f17-4a33-9f45-6400a54b3e63@Canary> Warmest welcome to you, Kai. Your contributions to the ecosystem are highly appreciated. Best Regards, Hameer Abbasi > On Monday, Sep 17, 2018 at 6:08 PM, Ralf Gommers wrote: > Hi all, > > On behalf of the SciPy developers I'd like to welcome Kai Striega as a member of the core dev team. > > Kai has been contributing to scipy.optimize since June, and has so far made a number of improvements to the simplex method of linprog that fixed all of the many known bugs - https://github.com/scipy/scipy/pulls/Kai-Striega > > I'm looking forward to his continued contributions! > > Cheers, > Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilhanpolat at gmail.com Tue Sep 18 11:35:06 2018 From: ilhanpolat at gmail.com (Ilhan Polat) Date: Tue, 18 Sep 2018 17:35:06 +0200 Subject: [SciPy-Dev] welcome Kai to the core team In-Reply-To: <11d6cb2f-2f17-4a33-9f45-6400a54b3e63@Canary> References: <11d6cb2f-2f17-4a33-9f45-6400a54b3e63@Canary> Message-ID: Welcome on board Kai! On Tue, Sep 18, 2018, 17:27 Hameer Abbasi wrote: > Warmest welcome to you, Kai. > > Your contributions to the ecosystem are highly appreciated. > > Best Regards, > Hameer Abbasi > > On Monday, Sep 17, 2018 at 6:08 PM, Ralf Gommers > wrote: > Hi all, > > On behalf of the SciPy developers I'd like to welcome Kai Striega as a > member of the core dev team. > > Kai has been contributing to scipy.optimize since June, and has so far > made a number of improvements to the simplex method of linprog that fixed > all of the many known bugs - > https://github.com/scipy/scipy/pulls/Kai-Striega > > I'm looking forward to his continued contributions! > > Cheers, > Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at drhagen.com Wed Sep 19 07:20:04 2018 From: david at drhagen.com (David Hagen) Date: Wed, 19 Sep 2018 07:20:04 -0400 Subject: [SciPy-Dev] Dormand-Prince 8(5,3) in new SciPy IVP integrators In-Reply-To: <165eb65436a.1234286673798.6479979211901456586@zoho.com> References: <165eb65436a.1234286673798.6479979211901456586@zoho.com> Message-ID: I agree that this would be a good solver to add. Please mention @drhagen on the PR also. For testing, the solver should be added to all the testing sections that make sense. But we can discuss the details of that on the PR. On Tue, Sep 18, 2018 at 2:37 AM Nikolay Mayorov wrote: > Hi, Juan! > > I know that several people were interested to see this method implemented. > I know it has a difficult logic for step size control so I couldn't > implement it quickly. Now that you have a working version (say it might > contain some small errors, but still the core logic is here) conforming > the new API I think you should definitely put the pull request. Perhaps > adding a new simple example from celestial mechanics (say) to demonstrate > how it works will be useful and interesting too (say compare lower order RK > methods with it). > > Please create the PR and mention @nmayorov here. > > Best, > Nikolay > > > > ---- On Mon, 17 Sep 2018 15:00:34 +0500 *Juan Luis Cano > >* wrote ---- > > Hi all, > > We wanted to use the new initial value problem solvers present in SciPy > but the DOP853 method was not present there, so we implemented it ourselves: > > > https://github.com/poliastro/poliastro/blob/v0.11b1/src/poliastro/integrators.py > > We have done some soft validation (i.e. it has worked well so far for our > specific use case) and we think it would be great to have it in SciPy, > given that it was present in the "Old API" and that it's a widely used > method in Celestial Mechanics. > > If I understood correctly, in-depth quantitative validation was not > written for the current solvers: > > https://github.com/scipy/scipy/pull/6326#issuecomment-250830082 > > Would you be interested in a PR? For testing, would it be enough to add > the new method here? > > > https://github.com/scipy/scipy/blob/4d16aea1267898e8532f962f0104d485578ab670/scipy/integrate/tests/test_ivp.py#L147 > > Regards, > > -- > Juan Luis Cano > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From juanlu001 at gmail.com Thu Sep 20 05:50:13 2018 From: juanlu001 at gmail.com (Juan Luis Cano) Date: Thu, 20 Sep 2018 11:50:13 +0200 Subject: [SciPy-Dev] Dormand-Prince 8(5,3) in new SciPy IVP integrators In-Reply-To: References: <165eb65436a.1234286673798.6479979211901456586@zoho.com> Message-ID: For the rest of the subscribers, the pull request is in: https://github.com/scipy/scipy/pull/9290 On Wed, Sep 19, 2018 at 1:44 PM David Hagen wrote: > I agree that this would be a good solver to add. Please mention @drhagen > on the PR also. For testing, the solver should be added to all the testing > sections that make sense. But we can discuss the details of that on the PR. > > On Tue, Sep 18, 2018 at 2:37 AM Nikolay Mayorov > wrote: > >> Hi, Juan! >> >> I know that several people were interested to see this method >> implemented. I know it has a difficult logic for step size control so I >> couldn't implement it quickly. Now that you have a working version (say it >> might contain some small errors, but still the core logic is here) >> conforming the new API I think you should definitely put the pull request. >> Perhaps adding a new simple example from celestial mechanics (say) to >> demonstrate how it works will be useful and interesting too (say compare >> lower order RK methods with it). >> >> Please create the PR and mention @nmayorov here. >> >> Best, >> Nikolay >> >> >> >> ---- On Mon, 17 Sep 2018 15:00:34 +0500 *Juan Luis Cano >> >* wrote ---- >> >> Hi all, >> >> We wanted to use the new initial value problem solvers present in SciPy >> but the DOP853 method was not present there, so we implemented it ourselves: >> >> >> https://github.com/poliastro/poliastro/blob/v0.11b1/src/poliastro/integrators.py >> >> We have done some soft validation (i.e. it has worked well so far for our >> specific use case) and we think it would be great to have it in SciPy, >> given that it was present in the "Old API" and that it's a widely used >> method in Celestial Mechanics. >> >> If I understood correctly, in-depth quantitative validation was not >> written for the current solvers: >> >> https://github.com/scipy/scipy/pull/6326#issuecomment-250830082 >> >> Would you be interested in a PR? For testing, would it be enough to add >> the new method here? >> >> >> https://github.com/scipy/scipy/blob/4d16aea1267898e8532f962f0104d485578ab670/scipy/integrate/tests/test_ivp.py#L147 >> >> Regards, >> >> -- >> Juan Luis Cano >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -- Juan Luis Cano -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Sep 23 08:59:57 2018 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 23 Sep 2018 06:59:57 -0600 Subject: [SciPy-Dev] NumPy 1.15.2 released Message-ID: Hi All, On behalf of the NumPy team, I am pleased to announce the release of NumPy 1.15.2. This is a bug fix release for bugs and regressions reported following the 1.15.2 release. The fixes are: - the matrix PendingDeprecationWarning is now suppressed in pytest 3.8, - the new cached allocations machinery has been fixed to be thread safe, - the boolean indexing of subclasses now works correctly, - a small memory leak in PyArray_AdaptFlexibleDType has been fixed. The Python versions supported by this release are 2.7, 3.4-3.7. The wheels are linked with OpenBLAS v0.3.0, which should fix some of the linalg problems reported for NumPy 1.14. Wheels for this release can be downloaded from PyPI , source archives are available from Github Compatibility Note ================== The NumPy 1.15.x OS X wheels released on PyPI no longer contain 32-bit binaries. That will also be the case in future releases. See `#11625 `__ for the related discussion. Those needing 32-bit support should look elsewhere or build from source. Contributors ============ A total of 4 people contributed to this release. People with a "+" by their names contributed a patch for the first time. * Charles Harris * Julian Taylor * Marten van Kerkwijk * Matti Picus Pull requests merged ==================== A total of 4 pull requests were merged for this release. * `#11902 `__: BUG: Fix matrix PendingDeprecationWarning suppression for pytest... * `#11981 `__: BUG: fix cached allocations without the GIL for 1.15.x * `#11982 `__: BUG: fix refcount leak in PyArray_AdaptFlexibleDType * `#11992 `__: BUG: Ensure boolean indexing of subclasses sets base correctly. Cheers, Charles Harris -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Sep 23 13:36:01 2018 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 23 Sep 2018 11:36:01 -0600 Subject: [SciPy-Dev] Numpy 1.14.6 release Message-ID: Hi All, On behalf of the NumPy team, I am pleased to announce the release of NumPy 1.14.6. This is a bug fix release for bugs and regressions reported following the 1.14.5 release. The significant fixes are: - Fix for behavior change in ``ma.masked_values(shrink=True)`` - Fix the new cached allocations machinery to be thread safe. The Python versions supported by this release are 2.7, 3.4-3.7. The wheels are linked with OpenBLAS v0.3.0, which should fix some of the linalg problems reported for earlier releases of NumPy 1.14. Wheels for this release can be downloaded from PyPI , source archives are available from Github . Compatibility Note ================== The NumPy 1.14.6 OS X wheels released on PyPI no longer contain 32-bit binaries. See #11625 for the related discussion. Those needing 32-bit support should look elsewhere or build from source. Contributors ============ A total of 4 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Charles Harris - Eric Wieser - Julian Taylor - Matti Picus Pull requests merged ==================== A total of 4 pull requests were merged for this release. - #11985: BUG: fix cached allocations without the GIL - #11986: BUG: Undo behavior change in ma.masked_values(shrink=True) - #11987: BUG: fix refcount leak in PyArray_AdaptFlexibleDType - #11995: TST: Add Python 3.7 testing to NumPy 1.14. Cheers, Charles Harris -------------- next part -------------- An HTML attachment was scrubbed... URL: From firman.kasim at gmail.com Sun Sep 23 18:03:48 2018 From: firman.kasim at gmail.com (Muhammad Firmansyah Kasim) Date: Sun, 23 Sep 2018 23:03:48 +0100 Subject: [SciPy-Dev] Proposal for a new scipy.special function: Voigt Message-ID: I am proposing to implement a new special function under `scipy.special` module: Voigt function. Voigt function/profile is a very common line profile in atomic transitions and widely used in the spectroscopy field. It is a convolution between Gaussian and Cauchy distribution, thus it is not trivial to calculate it efficiently. There are several approximations to calculate it efficiently, but I am proposing to use the one in the "Stellar atmosphere" by Mihalas (1978). It is the one used in FLYCHK by NIST and IAEA (I work closely with the author of the code). Please let me know if you have any suggestions or objections to the proposal. Thanks! Muhammad -------------- next part -------------- An HTML attachment was scrubbed... URL: From theodore.goetz at gmail.com Sun Sep 23 18:15:46 2018 From: theodore.goetz at gmail.com (Johann Goetz) Date: Sun, 23 Sep 2018 15:15:46 -0700 Subject: [SciPy-Dev] Proposal for a new scipy.special function: Voigt In-Reply-To: References: Message-ID: This is good as I've always thought scipy.special could use a dedicated Voigtian distribution. I am curious though how it compares (both mathematically and performance) with the code I've been using for this: # The Voigt probability distribution # where the integral from -inf to inf is equal to one # voigt_pdf(mean, sigma, gamma)(x) def voigt_pdf(mean,sigma,gamma): def voigt_pdf_fn(x,mean=mean,sigma=sigma,gamma=gamma): z = ((x-mean) + 1j*gamma) / (sigma * np.sqrt(2)) return sp.special.wofz(z).real / (sigma * np.sqrt(2.*np.pi)) return voigt_pdf_fn # Voigt distribution # with total integral as the first parameter # voigt(x, N, mean, sigma, gamma) def voigt(x, N, mean, sigma, gamma): return N * voigt_pdf(mean,sigma,gamma)(x) -- Johann On Sun, Sep 23, 2018 at 3:04 PM Muhammad Firmansyah Kasim < firman.kasim at gmail.com> wrote: > I am proposing to implement a new special function under `scipy.special` module: Voigt function. > Voigt function/profile is a very common line profile in atomic transitions and widely used in the spectroscopy field. It is a convolution between Gaussian and Cauchy distribution, thus it is not trivial to calculate it efficiently. > There are several approximations to calculate it efficiently, but I am proposing to use the one in the "Stellar atmosphere" by Mihalas (1978). It is the one used in FLYCHK by NIST and IAEA (I work closely with the author of the code). > Please let me know if you have any suggestions or objections to the proposal. > Thanks! > > Muhammad -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Sun Sep 23 18:39:46 2018 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Sun, 23 Sep 2018 18:39:46 -0400 Subject: [SciPy-Dev] Proposal for a new scipy.special function: Voigt In-Reply-To: References: Message-ID: On 9/23/18, Muhammad Firmansyah Kasim wrote: > I am proposing to implement a new special function under `scipy.special` > module: Voigt function. > > Voigt function/profile is a very common line profile in atomic transitions > and widely used in the spectroscopy field. It is a convolution between > Gaussian and Cauchy distribution, thus it is not trivial to calculate it > efficiently. > > There are several approximations to calculate it efficiently, but I am > proposing to use the one in the "Stellar atmosphere" by Mihalas (1978). It > is the one used in FLYCHK by NIST and IAEA (I work closely with the author > of the code). > > Please let me know if you have any suggestions or objections to the > proposal. > Thanks! > > Muhammad > Thanks, Muhammad. I've seen questions about implementing the Voigt function more than once over on stackoverflow, so I think it would be a worthwhile addition to scipy. The Voigt function can be expressed in terms of the complex Faddeeva function w(z), which is in scipy.special. Using w(z) allows the Voigt function to be implemented in just a few lines. I experimented with the Voigt function when working on an answer to a question at stackoverflow: https://stackoverflow.com/questions/40536467/scipy-integrate-quad-not-returning-the-expected-values Below is a modified version of the script in my stackoverflow answer. It generates the attached plot, which reproduces the plot at https://dlmf.nist.gov/7.19. It would be good to know how the accuracy and performance of these implementations compare to the code that you are proposing. Warren ----- from __future__ import division import numpy as np from scipy.special import wofz, erfc from scipy.integrate import quad _SQRTPI = np.sqrt(np.pi) _SQRTPI2 = _SQRTPI/2 def voigtuv(x, t): """ Voigt functions U(x,t) and V(x,t). The return value is U(x,t) + 1j*V(x,t). """ sqrtt = np.sqrt(t) z = (1j + x)/(2*sqrtt) w = wofz(z) * _SQRTPI2 / sqrtt return w def voigth(a, u): """ Voigt function H(a, u). """ x = u/a t = 1/(4*a**2) voigtU = voigtuv(x, t).real h = voigtU/(a*_SQRTPI) return h if __name__ == "__main__": # Create plots of U(x,t) and V(x,t) like those # in http://dlmf.nist.gov/7.19 import matplotlib.pyplot as plt xmax = 15 x = np.linspace(0, xmax, 100) t = np.array([0.1, 2.5, 5, 10]) y = voigtuv(x, t.reshape(-1, 1)) colors = ['g', 'r', 'b', 'y'] plt.subplot(2, 1, 1) for tval, ya, clr in zip(t, y, colors): plt.plot(x, ya.real, clr, label="t=%4.1f" % tval) plt.grid() plt.xlim(0, xmax) plt.ylim(0, 1) plt.legend(loc='best') plt.title('U(x,t)') plt.subplot(2, 1, 2) for tval, ya, clr in zip(t, y, colors): plt.plot(x, ya.imag, clr, label="t=%4.1f" % tval) plt.grid() plt.xlim(0, xmax) plt.ylim(0, 0.5) plt.legend(loc='best') plt.title('V(x,t)') plt.xlabel('x') plt.tight_layout() plt.show() ----- -------------- next part -------------- A non-text attachment was scrubbed... Name: voigtuv.png Type: image/png Size: 114375 bytes Desc: not available URL: From ralf.gommers at gmail.com Tue Sep 25 15:20:32 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 25 Sep 2018 19:20:32 +0000 Subject: [SciPy-Dev] planning 1.2.0 release Message-ID: Hi all, We released 1.1.0 in early May, so aiming for a 1.2.0 in the first half of November sounds about right. A couple of questions: - is anyone interested in being the release manager for this release? (I can do it, but volunteers *very* welcome) - are there any PRs or issues you are aware of that you want for the next release and are not under https://github.com/scipy/scipy/milestone/36? If so, please add them or suggest them here Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Tue Sep 25 15:39:12 2018 From: toddrjen at gmail.com (Todd) Date: Tue, 25 Sep 2018 15:39:12 -0400 Subject: [SciPy-Dev] planning 1.2.0 release In-Reply-To: References: Message-ID: I am working on an overlap-add convolution implementation. I would like to get it in to 1.2.0. When do I need to have it submitted by? On Tue, Sep 25, 2018 at 3:20 PM Ralf Gommers wrote: > Hi all, > > We released 1.1.0 in early May, so aiming for a 1.2.0 in the first half of > November sounds about right. A couple of questions: > - is anyone interested in being the release manager for this release? (I > can do it, but volunteers *very* welcome) > - are there any PRs or issues you are aware of that you want for the next > release and are not under https://github.com/scipy/scipy/milestone/36? If > so, please add them or suggest them here > > Cheers, > Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Sep 25 17:52:56 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 25 Sep 2018 17:52:56 -0400 Subject: [SciPy-Dev] planning 1.2.0 release In-Reply-To: References: Message-ID: On Tue, Sep 25, 2018 at 3:39 PM Todd wrote: > I am working on an overlap-add convolution implementation. I would like > to get it in to 1.2.0. When do I need to have it submitted by? > We don't have an exact timeline yet, but I'm hoping that we can branch by the end of October. So that's when it has to be merged. Submit - review cycle is harder to predict of course. Cheers, Ralf > On Tue, Sep 25, 2018 at 3:20 PM Ralf Gommers > wrote: > >> Hi all, >> >> We released 1.1.0 in early May, so aiming for a 1.2.0 in the first half >> of November sounds about right. A couple of questions: >> - is anyone interested in being the release manager for this release? (I >> can do it, but volunteers *very* welcome) >> - are there any PRs or issues you are aware of that you want for the next >> release and are not under https://github.com/scipy/scipy/milestone/36? >> If so, please add them or suggest them here >> >> Cheers, >> Ralf >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyler.je.reddy at gmail.com Tue Sep 25 18:03:17 2018 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Tue, 25 Sep 2018 15:03:17 -0700 Subject: [SciPy-Dev] planning 1.2.0 release In-Reply-To: References: Message-ID: > > is anyone interested in being the release manager for this release? (I > can do it, but volunteers *very* welcome) I should probably learn to do this at some point -- are the instructions on scipy.org relatively recent / accurate? https://docs.scipy.org/doc/scipy/reference/dev/releasing.html On Tue, 25 Sep 2018 at 14:53, Ralf Gommers wrote: > > > On Tue, Sep 25, 2018 at 3:39 PM Todd wrote: > >> I am working on an overlap-add convolution implementation. I would like >> to get it in to 1.2.0. When do I need to have it submitted by? >> > > We don't have an exact timeline yet, but I'm hoping that we can branch by > the end of October. So that's when it has to be merged. Submit - review > cycle is harder to predict of course. > > Cheers, > Ralf > > >> On Tue, Sep 25, 2018 at 3:20 PM Ralf Gommers >> wrote: >> >>> Hi all, >>> >>> We released 1.1.0 in early May, so aiming for a 1.2.0 in the first half >>> of November sounds about right. A couple of questions: >>> - is anyone interested in being the release manager for this release? (I >>> can do it, but volunteers *very* welcome) >>> - are there any PRs or issues you are aware of that you want for the >>> next release and are not under >>> https://github.com/scipy/scipy/milestone/36? If so, please add them or >>> suggest them here >>> >>> Cheers, >>> Ralf >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Sep 25 18:10:26 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 25 Sep 2018 18:10:26 -0400 Subject: [SciPy-Dev] planning 1.2.0 release In-Reply-To: References: Message-ID: On Tue, Sep 25, 2018 at 6:03 PM Tyler Reddy wrote: > is anyone interested in being the release manager for this release? (I >> can do it, but volunteers *very* welcome) > > > I should probably learn to do this at some point -- are the instructions > on scipy.org relatively recent / accurate? > https://docs.scipy.org/doc/scipy/reference/dev/releasing.html > > Great, thanks for volunteering! Yes, those are pretty close - a quick scan doesn't show any steps that shouldn't be there or are missing. The first time you do this you may run into a few small things, e.g. not having access to PyPI for uploading wheels. We can solve and document those as we go. Cheers, Ralf > > On Tue, 25 Sep 2018 at 14:53, Ralf Gommers wrote: > >> >> >> On Tue, Sep 25, 2018 at 3:39 PM Todd wrote: >> >>> I am working on an overlap-add convolution implementation. I would like >>> to get it in to 1.2.0. When do I need to have it submitted by? >>> >> >> We don't have an exact timeline yet, but I'm hoping that we can branch by >> the end of October. So that's when it has to be merged. Submit - review >> cycle is harder to predict of course. >> >> Cheers, >> Ralf >> >> >>> On Tue, Sep 25, 2018 at 3:20 PM Ralf Gommers >>> wrote: >>> >>>> Hi all, >>>> >>>> We released 1.1.0 in early May, so aiming for a 1.2.0 in the first half >>>> of November sounds about right. A couple of questions: >>>> - is anyone interested in being the release manager for this release? >>>> (I can do it, but volunteers *very* welcome) >>>> - are there any PRs or issues you are aware of that you want for the >>>> next release and are not under >>>> https://github.com/scipy/scipy/milestone/36? If so, please add them or >>>> suggest them here >>>> >>>> Cheers, >>>> Ralf >>>> >>>> _______________________________________________ >>>> SciPy-Dev mailing list >>>> SciPy-Dev at python.org >>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phillip.m.feldman at gmail.com Tue Sep 25 19:08:06 2018 From: phillip.m.feldman at gmail.com (Phillip Feldman) Date: Tue, 25 Sep 2018 16:08:06 -0700 Subject: [SciPy-Dev] planning 1.2.0 release In-Reply-To: References: Message-ID: I would like to see the Aditya Bharti's changes to implement rotation formalism in 3 dimensions in the 1.2.0 release. Phillip -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Sep 25 20:02:32 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 25 Sep 2018 20:02:32 -0400 Subject: [SciPy-Dev] planning 1.2.0 release In-Reply-To: References: Message-ID: On Tue, Sep 25, 2018 at 7:08 PM Phillip Feldman wrote: > I would like to see the Aditya Bharti's changes to implement rotation > formalism in 3 dimensions in the 1.2.0 release. > It seems there are still some issues in the one open PR: https://github.com/scipy/scipy/pull/9024. If those get addressed, it could make it into the next release. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Sep 26 09:54:55 2018 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 26 Sep 2018 07:54:55 -0600 Subject: [SciPy-Dev] planning 1.2.0 release In-Reply-To: References: Message-ID: On Tue, Sep 25, 2018 at 4:03 PM Tyler Reddy wrote: > is anyone interested in being the release manager for this release? (I >> can do it, but volunteers *very* welcome) > > > I should probably learn to do this at some point -- are the instructions > on scipy.org relatively recent / accurate? > https://docs.scipy.org/doc/scipy/reference/dev/releasing.html > > > The NumPy walkthrough is similar and will help you see the steps. Getting yourself set up with the infrastructure will probably be the time consuming part. Might be worth making a SciPy version as there are some differences in the tools. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From evgeny.burovskiy at gmail.com Wed Sep 26 14:18:46 2018 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Wed, 26 Sep 2018 21:18:46 +0300 Subject: [SciPy-Dev] planning 1.2.0 release In-Reply-To: References: Message-ID: On Wed, Sep 26, 2018, 4:55 PM Charles R Harris wrote: > > > On Tue, Sep 25, 2018 at 4:03 PM Tyler Reddy > wrote: > >> is anyone interested in being the release manager for this release? (I >>> can do it, but volunteers *very* welcome) >> >> >> I should probably learn to do this at some point -- are the instructions >> on scipy.org relatively recent / accurate? >> https://docs.scipy.org/doc/scipy/reference/dev/releasing.html >> >> >> > The NumPy walkthrough > > is similar and will help you see the steps. Getting yourself set up with > the infrastructure will probably be the time consuming part. Might be worth > making a SciPy version as there are some differences in the tools. > > > > Chuck > Here's the checklist I used for 0.18.x: https://gist.github.com/ev-br/dd5878cb66aa52e8a548fa7eece55fec Some parts of this are certainly obsolete, though. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.vanmulbregt at comcast.net Wed Sep 26 23:27:15 2018 From: p.vanmulbregt at comcast.net (Paul van Mulbregt) Date: Wed, 26 Sep 2018 23:27:15 -0400 Subject: [SciPy-Dev] planning 1.2.0 release In-Reply-To: References: Message-ID: <716C4D61-3BB4-4866-9840-26904E70DB99@comcast.net> I?d request a decision on gh-8737 "BUG: Improve numerical precision/convergence failures of smirnov/kolmogorov". The prerequisite for it, the dd_real part of the qd package gh-8551, was released in 1.1.0, and its (essentially) only use-case is gh-8737. -Paul > On Sep 25, 2018, at 3:20 PM, Ralf Gommers wrote: > > Hi all, > > We released 1.1.0 in early May, so aiming for a 1.2.0 in the first half of November sounds about right. A couple of questions: > - is anyone interested in being the release manager for this release? (I can do it, but volunteers *very* welcome) > - are there any PRs or issues you are aware of that you want for the next release and are not under https://github.com/scipy/scipy/milestone/36 ? If so, please add them or suggest them here > > Cheers, > Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From phillip.m.feldman at gmail.com Thu Sep 27 00:48:14 2018 From: phillip.m.feldman at gmail.com (Phillip Feldman) Date: Wed, 26 Sep 2018 21:48:14 -0700 Subject: [SciPy-Dev] planning 1.2.0 release In-Reply-To: <716C4D61-3BB4-4866-9840-26904E70DB99@comcast.net> References: <716C4D61-3BB4-4866-9840-26904E70DB99@comcast.net> Message-ID: KS is one of the most heavily-used tests of distribution. I had not known about the false convergence issue; will have to check whether I need to retract any old results that I generated using that code! On Wed, Sep 26, 2018 at 8:32 PM Paul van Mulbregt wrote: > I?d request a decision on gh-8737 "BUG: Improve numerical > precision/convergence failures of smirnov/kolmogorov". > The prerequisite for it, the dd_real part of the qd package gh-8551, was > released in 1.1.0, and its (essentially) only use-case is gh-8737. > > -Paul > > > > On Sep 25, 2018, at 3:20 PM, Ralf Gommers wrote: > > Hi all, > > We released 1.1.0 in early May, so aiming for a 1.2.0 in the first half of > November sounds about right. A couple of questions: > - is anyone interested in being the release manager for this release? (I > can do it, but volunteers *very* welcome) > - are there any PRs or issues you are aware of that you want for the next > release and are not under https://github.com/scipy/scipy/milestone/36? If > so, please add them or suggest them here > > Cheers, > Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: