From hi.hellbee at gmail.com Mon Sep 2 16:44:59 2019 From: hi.hellbee at gmail.com (Filippo Bovo) Date: Mon, 2 Sep 2019 21:44:59 +0100 Subject: [SciPy-Dev] Donating code for robust statistical estimators Message-ID: <612FEDAD-1E68-422B-9B6D-B97416743676@gmail.com> Hi, I wrote a Python package for the high-performance calculation of three robust statistical estimators: weighted median, medcouple and mode. The repository with the code is at this link: https://github.com/FilippoBovo/robustats . The estimators are implemented in C using linear or log-linear algorithms, thereby making the code run pretty fast. I would like to donate the code for the above three estimators to SciPy. I believe that `scipy.stats` is a good location where to add the weighted median, medcouple and mode. Could you please let me know if you are happy with this? Thank you. Best wishes, Filippo -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Sep 3 01:45:43 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 2 Sep 2019 22:45:43 -0700 Subject: [SciPy-Dev] Donating code for robust statistical estimators In-Reply-To: <612FEDAD-1E68-422B-9B6D-B97416743676@gmail.com> References: <612FEDAD-1E68-422B-9B6D-B97416743676@gmail.com> Message-ID: On Mon, Sep 2, 2019 at 1:45 PM Filippo Bovo wrote: > Hi, > > I wrote a Python package for the high-performance calculation of three > robust statistical estimators: weighted median, medcouple and mode. > > The repository with the code is at this link: > https://github.com/FilippoBovo/robustats. > > The estimators are implemented in C using linear or log-linear algorithms, > thereby making the code run pretty fast. > > I would like to donate the code for the above three estimators to SciPy. > Hi Filippo. Thanks for your interest in contributing! I believe that `scipy.stats` is a good location where to add the weighted > median, medcouple and mode. > There is a mode function in scipy.stats. It looks nontrivial to make that work in C in a backwards-compatible fashion, but please have a look at it and see if that can work and would be an improvement. I've looked at your medcouple, but the function doesn't explain what it's for and I'm not familiar with the name. Can you explain? A weighted median doesn't fit well. NumPy has a median function, perhaps better to consider whether to add weights to that, in the same fashion as for numpy.average Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From hi.hellbee at gmail.com Tue Sep 3 05:01:08 2019 From: hi.hellbee at gmail.com (Filippo Bovo) Date: Tue, 3 Sep 2019 10:01:08 +0100 Subject: [SciPy-Dev] Donating code for robust statistical estimators In-Reply-To: References: <612FEDAD-1E68-422B-9B6D-B97416743676@gmail.com> Message-ID: <211CF6BF-32F5-4AA9-B659-A2A7933FEF85@gmail.com> > On 3 Sep 2019, at 06:45, Ralf Gommers wrote: > > > > On Mon, Sep 2, 2019 at 1:45 PM Filippo Bovo > wrote: > Hi, > > I wrote a Python package for the high-performance calculation of three robust statistical estimators: weighted median, medcouple and mode. > > The repository with the code is at this link: https://github.com/FilippoBovo/robustats . > > The estimators are implemented in C using linear or log-linear algorithms, thereby making the code run pretty fast. > > I would like to donate the code for the above three estimators to SciPy. > > Hi Filippo. Thanks for your interest in contributing! > > I believe that `scipy.stats` is a good location where to add the weighted median, medcouple and mode. > > There is a mode function in scipy.stats. It looks nontrivial to make that work in C in a backwards-compatible fashion, but please have a look at it and see if that can work and would be an improvement. > > I've looked at your medcouple, but the function doesn't explain what it's for and I'm not familiar with the name. Can you explain? The medcouple is a quantity used to determine the outliers of a skewed distribution. For example, say that we have a data sample that leans more towards the right than the left; in this case, we would use the medcouple to determine the skewness of the distribution and use the result to determine the left and right outliers. You may find more information in the Wikipedia page: https://en.wikipedia.org/wiki/Medcouple . > > A weighted median doesn't fit well. NumPy has a median function, perhaps better to consider whether to add weights to that, in the same fashion as for numpy.average > Ok, thank you. > Cheers, > Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev Cheers, Filippo -------------- next part -------------- An HTML attachment was scrubbed... URL: From josh.craig.wilson at gmail.com Tue Sep 3 14:33:09 2019 From: josh.craig.wilson at gmail.com (Joshua Wilson) Date: Tue, 3 Sep 2019 13:33:09 -0500 Subject: [SciPy-Dev] Direction of polynomial/rational approximation in SciPy Message-ID: Hey all, The goal of this post is to try and build consensus around where polynomial/rational approximation in SciPy should be headed. I'd like to add whatever we come up with to the roadmap. Over the years we've had many discussions on what polynomial/rational approximations should look like in SciPy; for a sample see e.g. https://github.com/scipy/scipy/pull/6591 https://github.com/scipy/scipy/issues/7181 https://github.com/scipy/scipy/issues/6928 https://github.com/scipy/scipy/issues/6929 https://github.com/scipy/scipy/pull/4674 Some high-level takeaways from those discussions seem to be: (1) We would like to have better support for rational approximations in SciPy (2) `special.orthopoly1d` is not worth trying to fix (3) We want to avoid overlap with `numpy.polynomial` At this point I think (1) is pretty vague; we need to figure out what methods we want to support and how to organize them. We've discussed (2) and (3) a lot, but haven't come to a consensus. One solution would be to do something like (1) Add series implementations that inherit from NumPy's `ABCPolyBase` for the families that are currently in SciPy but not in NumPy (though we can drop the shifted variants since NumPy handles that already). (2) Deprecate `orthopoly1d` I'll note that Chuck had some concerns about families with infinite support: https://github.com/scipy/scipy/issues/7181#issuecomment-288273394 What are people's thoughts on these issues? From juanlu001 at gmail.com Thu Sep 5 12:36:28 2019 From: juanlu001 at gmail.com (Juan Luis Cano) Date: Thu, 5 Sep 2019 18:36:28 +0200 Subject: [SciPy-Dev] Sprints at EuroSciPy In-Reply-To: References: Message-ID: Hi all, This is a reminder that tomorrow at 10:30 UTC+02 the EuroSciPy sprints will start. I will organize the volunteers that want to contribute to SciPy but I am not a core contributor, so if anyone from the projects happens to have some spare time to answer questions on IRC (which is bridged in Matrix/Riot) or keep an eye on the issue tracker that would be awesome. Best! On Thu, 22 Aug 2019, 19:36 Ilhan Polat, wrote: > I was planning to attend for the maintainers summit but didn't hear back > from them on Twitter and I couldn't get into contact with the organizers > which is probably an error on my side. Then I was thinking about presenting > something but then I decided not to, since like last year, this is kind of > a PyData-variant rather than Scipy, or a "Euroscipy.stats" if you will. > Just glancing through, I think I've spotted only 3-4 out of "a lot > 30" > that are not necessarily data/ML related. > > Having said all that, we need all the love there is out there :) > Especially about adding examples to documentation. Since you mention > scipy.signal have a look at https://github.com/scipy/scipy/issues/7168 > > > > > On Thu, Aug 22, 2019 at 4:45 PM Juan Luis Cano > wrote: > >> Hi all, >> >> There are some free slots for the EuroSciPy sprints and the organizers >> encouraged us to propose more: >> >> https://www.euroscipy.org/2019/program.html >> >> I wonder if there are other people attending EuroSciPy interested in >> sprinting, what topics could we choose (general bug triaging? high priority >> defects? a specific sub package needing some love? some work on >> numba-scipy?), and if people not attending (especially core developers) >> would be open to participate remotely, perhaps answering questions on IRC >> (or Matrix, see https://riot.im/app/#/room/#freenode_#scipy:matrix.org). >> >> I have a general interest in scipy.integrate, scipy.optimize and >> scipy.signal but I'm not an expert in anything. What I'd like is to see a >> SciPy sprint at EuroSciPy to, you know, honor the name of the conference :) >> If there's interest, I will propose it to the organizers. >> >> Best, >> >> -- >> Juan Luis Cano >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Sep 6 20:42:04 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 6 Sep 2019 18:42:04 -0600 Subject: [SciPy-Dev] NumPy 1.17.2 released. Message-ID: Hi All, On behalf of the NumPy team I am pleased to announce that NumPy 1.17.2 has been released. This release contains fixes for bugs reported against NumPy 1.17.1 along with some documentation improvements. The most important fix is for lexsort when the keys are of type (u)int8 or (u)int16. If you are currently using 1.17 you should upgrade. The Python versions supported in this release are 3.5-3.7, Python 3.8b4 should work with the released source packages, but there are no future guarantees. Downstream developers should use Cython >= 0.29.13 for Python 3.8 support and OpenBLAS >= 3.7 to avoid wrong results on the Skylake architecture. The NumPy wheels on PyPI are built from the OpenBLAS development branch in order to avoid those problems. Wheels for this release can be downloaded from PyPI , source archives and release notes are available from Github . *Contributors* A total of 7 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - CakeWithSteak + - Charles Harris - Dan Allan - Hameer Abbasi - Lars Grueter - Matti Picus - Sebastian Berg *Pull requests merged* A total of 8 pull requests were merged for this release. - #14418: BUG: Fix aradixsort indirect indexing. - #14420: DOC: Fix a minor typo in dispatch documentation. - #14421: BUG: test, fix regression in converting to ctypes - #14430: BUG: Do not show Override module in private error classes. - #14432: BUG: Fixed maximum relative error reporting in assert_allclose. - #14433: BUG: Fix uint-overflow if padding with linear_ramp and negative... - #14436: BUG: Update 1.17.x with 1.18.0-dev pocketfft.py. - #14446: REL: Prepare for NumPy 1.17.2 release. Cheers, Charles Harris -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sat Sep 7 14:53:09 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 7 Sep 2019 11:53:09 -0700 Subject: [SciPy-Dev] Direction of polynomial/rational approximation in SciPy In-Reply-To: References: Message-ID: On Tue, Sep 3, 2019 at 11:33 AM Joshua Wilson wrote: > Hey all, > > The goal of this post is to try and build consensus around where > polynomial/rational approximation in SciPy should be headed. I'd like > to add whatever we come up with to the roadmap. > > Over the years we've had many discussions on what polynomial/rational > approximations should look like in SciPy; for a sample see e.g. > > https://github.com/scipy/scipy/pull/6591 > https://github.com/scipy/scipy/issues/7181 > https://github.com/scipy/scipy/issues/6928 > https://github.com/scipy/scipy/issues/6929 > https://github.com/scipy/scipy/pull/4674 > > Some high-level takeaways from those discussions seem to be: > > (1) We would like to have better support for rational approximations in > SciPy > (2) `special.orthopoly1d` is not worth trying to fix > (3) We want to avoid overlap with `numpy.polynomial` > > At this point I think (1) is pretty vague; we need to figure out what > methods we want to support and how to organize them. > > We've discussed (2) and (3) a lot, but haven't come to a consensus. > One solution would be to do something like > > (1) Add series implementations that inherit from NumPy's `ABCPolyBase` > for the families that are currently in SciPy but not in NumPy (though > we can drop the shifted variants since NumPy handles that already). > (2) Deprecate `orthopoly1d` > > I'll note that Chuck had some concerns about families with infinite > support: > > https://github.com/scipy/scipy/issues/7181#issuecomment-288273394 > > What are people's thoughts on these issues? > Your proposal makes sense to me. I don't know enough about either the NumPy or SciPy implementations to comment in more detail, but +1 from a maintenance point of view. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Sat Sep 7 23:56:26 2019 From: andyfaff at gmail.com (Andrew Nelson) Date: Sun, 8 Sep 2019 13:56:26 +1000 Subject: [SciPy-Dev] feedback on adding global optimiser methods to `minimize` Message-ID: I'd like to gauge the support for adding the global minimizers (dual_annealing, shgo, differential_evolution, basinhopping) as new `minimize` methods. The problems the the 'local' and 'global' optimizers are trying to solve are very similar and both are specified in the same way, so my thought is that it would be nice to access them via the same unified interface that `minimize` offers (but not deprecating the `shgo` function, etc). It's important that the users are able to understand the distinction between local and global optimisation and how they go about finding a minimum. I'm hoping that this could be made plain in the documentation. The change would allow the following: ``` # global minimizer minimize(func, x0, bounds=bounds, method='differential-evolution', constraints=constraints) # local minimizers minimize(func, x0, bounds=bounds, method='SLSQP', constraints=constraints) minimize(func, x0, bounds=bounds, method='trust-constr', constraints=constraints) minimize(func, x0, bounds=bounds, method='L-BFGS-B') ``` Please chip in with what your thoughts are, is it a bad idea, good idea, etc. -- _____________________________________ Dr. Andrew Nelson _____________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan.c.endres at gmail.com Sun Sep 8 16:15:09 2019 From: stefan.c.endres at gmail.com (Stefan Endres) Date: Sun, 8 Sep 2019 22:15:09 +0200 Subject: [SciPy-Dev] feedback on adding global optimiser methods to `minimize` In-Reply-To: References: Message-ID: I agree that a single unified function for all the routines would be neater since all the algorithms have black-box functions as input anyway. I believe that any extra optional arguments to the global optimisation functions not already defined by `minimize` can be handled with the `options` dictionary object passed to the `minimize` function One additional thought is the possibility of adding a unified `Minimizer` class to scipy.optimize. There are two reasons for this: (i) Manual control of iterations and/or memoization of progress. For example, sometimes the stopping criteria for a global optimisation problem is unknown and the optimisation practitioner would like to continue iterations according to some external criteria not definable in the algorithm itself (this is something that is often requested by `shgo` users to me through e-mail correspondence). Another example is with objective functions that have very long evaluations and the practitioner would like save the progress possibly including all evaluations/current best solution(s) and then continue the routine. (ii) More importantly to skip a few initiation steps in algorithm selection and initiation. This is useful when the function is called thousands to millions of times and only a few arguments/parameters need to be changed while the algorithm selection etc. remain unchanged. I'm not sure if this will have a significant effect for the local routines in `minimize` though. On Sun, Sep 8, 2019 at 5:57 AM Andrew Nelson wrote: > I'd like to gauge the support for adding the global minimizers > (dual_annealing, shgo, differential_evolution, basinhopping) as new > `minimize` methods. > > The problems the the 'local' and 'global' optimizers are trying to solve > are very similar and both are specified in the same way, so my thought is > that it would be nice to access them via the same unified interface that > `minimize` offers (but not deprecating the `shgo` function, etc). > > It's important that the users are able to understand the distinction > between local and global optimisation and how they go about finding a > minimum. I'm hoping that this could be made plain in the documentation. > > The change would allow the following: > > ``` > # global minimizer > minimize(func, x0, bounds=bounds, method='differential-evolution', > constraints=constraints) > # local minimizers > minimize(func, x0, bounds=bounds, method='SLSQP', constraints=constraints) > minimize(func, x0, bounds=bounds, method='trust-constr', > constraints=constraints) > minimize(func, x0, bounds=bounds, method='L-BFGS-B') > ``` > > Please chip in with what your thoughts are, is it a bad idea, good idea, > etc. > > -- > _____________________________________ > Dr. Andrew Nelson > > > _____________________________________ > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -- Stefan Endres (MEng, AMIChemE, BEng (Hons) Chemical Engineering) Postgraduate Student: Institute of Applied Materials Department of Chemical Engineering, University of Pretoria Cell: +27 (0) 82 972 42 89 E-mail: Stefan.C.Endres at gmail.com St. Number: 11004968 -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Sun Sep 8 19:32:01 2019 From: andyfaff at gmail.com (Andrew Nelson) Date: Mon, 9 Sep 2019 09:32:01 +1000 Subject: [SciPy-Dev] feedback on adding global optimiser methods to `minimize` In-Reply-To: References: Message-ID: > > I agree that a single unified function for all the routines would be > neater since all the algorithms have black-box functions as input anyway. I > believe that any extra optional arguments to the global optimisation > functions not already defined by `minimize` can be handled with the > `options` dictionary object passed to the `minimize` function > Thank you for feedback. One additional thought is the possibility of adding a unified `Minimizer` > class to scipy.optimize. There are two reasons for this: > As you may be aware this idea has been brought forward before, for the reasons you suggested and more. The idea stalled unfortunately. Because it would be a large changeset it's necessary to get everyone on board first (via a SciPEP). https://github.com/scipy/scipy/pull/8552 (SciPEP proposing it) https://github.com/scipy/scipy/pull/8414 (PR exploring the idea) Perhaps another way to bring this about is to setup a separate implementation project and show it working before it could be incorporated. That would give time and freedom for any issues to be ironed out. The development of differential_evolution has certainly benefitted from the solver being class based, but also from being a private interface, which has allowed relatively large changes to be made to its functionality. -------------- next part -------------- An HTML attachment was scrubbed... URL: From newville at cars.uchicago.edu Mon Sep 9 08:08:30 2019 From: newville at cars.uchicago.edu (Matt Newville) Date: Mon, 9 Sep 2019 07:08:30 -0500 Subject: [SciPy-Dev] feedback on adding global optimiser methods to `minimize` In-Reply-To: References: Message-ID: On Sat, Sep 7, 2019 at 10:57 PM Andrew Nelson wrote: > I'd like to gauge the support for adding the global minimizers > (dual_annealing, shgo, differential_evolution, basinhopping) as new > `minimize` methods. > > The problems the the 'local' and 'global' optimizers are trying to solve > are very similar and both are specified in the same way, so my thought is > that it would be nice to access them via the same unified interface that > `minimize` offers (but not deprecating the `shgo` function, etc). > > It's important that the users are able to understand the distinction > between local and global optimisation and how they go about finding a > minimum. I'm hoping that this could be made plain in the documentation. > > The change would allow the following: > > ``` > # global minimizer > minimize(func, x0, bounds=bounds, method='differential-evolution', > constraints=constraints) > # local minimizers > minimize(func, x0, bounds=bounds, method='SLSQP', constraints=constraints) > minimize(func, x0, bounds=bounds, method='trust-constr', > constraints=constraints) > minimize(func, x0, bounds=bounds, method='L-BFGS-B') > ``` > > Please chip in with what your thoughts are, is it a bad idea, good idea, > etc. > > -- > Personally, I find `minimize()` to be a bit unwieldy I do understand the desire to make things simple and uniform, but I'm not sure `minimize()` is really doing that. Correct me if any of this is wrong, but the interface for the current `minimize()` has the following features: 1. `minimize` takes 2 required arguments: func, x0. So far, so good. 2. There are 14 named "methods" and a "custom object" method (which, arguably means that the "global" methods are already supported). 3. There are 3 optional arguments used for all methods with sensible defaults: "args" , "tol" , and "callback". 4. The "method" argument is actually optional with the default used being a slightly complicated combination of other optional arguments. 5. There are 6 optional arguments that each depends on which "method" is used. 6. Most but not all of the 14 methods support "jac". 4 of 6 options ("hess", "hessp", "bounds", "constraints") are supported by fewer than half the methods. 7. One of the 6 optional arguments is called "options" and holds keyword options specific to each method and not explicitly listed. 8. Some of the options in "options" (say, "eps", "ftol") are supported by more methods than some of the explicitly named optional arguments (say, "constraints"). 9. "constraints" is supported for 3 methods. It has 2 incompatible forms. 10. "bounds" can either be a tuple of (called "min", "max" in the docs) values or a "Bounds" object that has attributes "lb" and "ub". Are "lower" and "upper" are too verbose? 11. As if to stick a finger in your eye, "callback" requires a different signature for exactly one of the 14 methods. It is all very well documented. But it is also extremely complicated. And, honestly, not very good. It seems to me that there are really 14 (or 15) different minimization methods. The `minimize()` function smashes these all into one function with a jumble of options. I think the idea was to emphasize the commonality of the interfaces, but the result sort of emphasizes what is not common among the methods. Instead of creating a common way to say "Bounds", "Constraints", "Hessian", "Tolerance", "Max Number of Function Calls", or "Step Size" for all methods, it allows multiple variations and then works extra hard to document the differences. How does this help the user (or person writing a wrapping library)? Can the user just change the name of the method argument? In a simple case that might work. But, in general, No, definitely not. Too many of the arguments change validity when the `method` is changed. To the extent that the user could change "minimize(...., method='foo', ...)" to "minimize(...., method='bar', ...)", they could just as easily change "_minimize_foo(....)" to "_minimize_bar(...)". That actually is less typing. I guess in a fatalistic sense, I would say "Sure, why not add more methods? You aren't going to make it any worse". Except that you will want to add more options that are not common to the other methods. Differential evolution itself has a dozen or so "strategy" options ("strategy" being obviously different from "method", I guess). There are "seeds" and "nworkers" and "max number of iterations" (whatever "iteration" means for each method). I do not know if any of these have uniform meaning; I guess not. It looks to me like "callback" would have yet another variation. And, eventually, you will want to deprecate the current working functions, breaking downstream code for the sake of a uniform API that really is not uniform. The minimization methods really are different. They have different options. IMO, it would be better to have one function per method and avoid the a sprawling mess of a dispatch function. It would be nice if the interfaces were as similar as possible, with similar concepts using consistent names and values. I would definitely encourage a class-based Minimizer, but mostly in the sense of being able to use inheritance to make it easier to have a common interface across the different solvers: so not one Minimizer, but a BaseMinizer and then PowellMinizer, BFGSMinimizer, KrylovMinimizer, etc. I would suggest looking at the stats module and scikit-learn for inspiration. And *then* make powell_minimize(), etc functions that create and use these. Oh, and a Parameters class would be nice ;). I have no expectation of being listened too. We'll all be able to work around the added complexity, just like we can sort of handle the similarly infuriating complexity of `least_squares` (by basically treating it as 3 separate functions). Finally, I will say that I am very glad to have `numpy.sin(x)`, `numpy.cos(x)`, etc instead of `numpy.ufunc(x, method='sin')`. Sorry if that seems like a rant. Cheers, --Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan.c.endres at gmail.com Mon Sep 9 14:54:47 2019 From: stefan.c.endres at gmail.com (Stefan Endres) Date: Mon, 9 Sep 2019 20:54:47 +0200 Subject: [SciPy-Dev] feedback on adding global optimiser methods to `minimize` In-Reply-To: References: Message-ID: >As you may be aware this idea has been brought forward before, for the reasons you suggested and more. Yes, thank you I was looking for links to these discussions. >The development of differential_evolution has certainly benefitted from the solver being class based, but also from being a private interface, which has allowed relatively large changes to be made to its functionality. For now I will look through the differential_evolution code for similarities with shgo that can be abstracted to and incorporate this into the `SHGO` class in case this idea picks up again in the future. On Mon, Sep 9, 2019 at 1:33 AM Andrew Nelson wrote: > I agree that a single unified function for all the routines would be >> neater since all the algorithms have black-box functions as input anyway. I >> believe that any extra optional arguments to the global optimisation >> functions not already defined by `minimize` can be handled with the >> `options` dictionary object passed to the `minimize` function >> > > Thank you for feedback. > > One additional thought is the possibility of adding a unified `Minimizer` >> class to scipy.optimize. There are two reasons for this: >> > > As you may be aware this idea has been brought forward before, for the > reasons you suggested and more. The idea stalled unfortunately. Because it > would be a large changeset it's necessary to get everyone on board first > (via a SciPEP). > https://github.com/scipy/scipy/pull/8552 (SciPEP proposing it) > https://github.com/scipy/scipy/pull/8414 (PR exploring the idea) > Perhaps another way to bring this about is to setup a separate > implementation project and show it working before it could be incorporated. > That would give time and freedom for any issues to be ironed out. The > development of differential_evolution has certainly benefitted from the > solver being class based, but also from being a private interface, which > has allowed relatively large changes to be made to its functionality. > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -- Stefan Endres (MEng, AMIChemE, BEng (Hons) Chemical Engineering) Postgraduate Student: Institute of Applied Materials Department of Chemical Engineering, University of Pretoria Cell: +27 (0) 82 972 42 89 E-mail: Stefan.C.Endres at gmail.com St. Number: 11004968 -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlucas7 at vt.edu Tue Sep 10 19:06:37 2019 From: rlucas7 at vt.edu (rlucas7 at vt.edu) Date: Tue, 10 Sep 2019 19:06:37 -0400 Subject: [SciPy-Dev] SciPy-Dev Digest, Vol 191, Issue 6 In-Reply-To: References: Message-ID: I agree, my opinion is different optimization methods should have different functions. I?m not sure if the current minimize method via callable() would handle global but if so all the more reason to not monolith. I?ve answer questions on internal email threads at work about the usage of minimize(), my impression is that although I don?t find it unwieldly, others do and the others seem to be the majority. Also, conceptually global optimizers are usually differently handled that convex optimizers. Maybe a global optimized callback handler function would work? -Lucas Roberts > On Sep 9, 2019, at 8:09 AM, scipy-dev-request at python.org wrote: > > Send SciPy-Dev mailing list submissions to > scipy-dev at python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scipy-dev > or, via email, send a message with subject or body 'help' to > scipy-dev-request at python.org > > You can reach the person managing the list at > scipy-dev-owner at python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of SciPy-Dev digest..." > > > Today's Topics: > > 1. Re: feedback on adding global optimiser methods to `minimize` > (Stefan Endres) > 2. Re: feedback on adding global optimiser methods to `minimize` > (Andrew Nelson) > 3. Re: feedback on adding global optimiser methods to `minimize` > (Matt Newville) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 8 Sep 2019 22:15:09 +0200 > From: Stefan Endres > To: SciPy Developers List > Subject: Re: [SciPy-Dev] feedback on adding global optimiser methods > to `minimize` > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > I agree that a single unified function for all the routines would be neater > since all the algorithms have black-box functions as input anyway. I > believe that any extra optional arguments to the global optimisation > functions not already defined by `minimize` can be handled with the > `options` dictionary object passed to the `minimize` function > > One additional thought is the possibility of adding a unified `Minimizer` > class to scipy.optimize. There are two reasons for this: > (i) Manual control of iterations and/or memoization of progress. For > example, sometimes the stopping criteria for a global optimisation problem > is unknown and the optimisation practitioner would like to continue > iterations according to some external criteria not definable in the > algorithm itself (this is something that is often requested by `shgo` users > to me through e-mail correspondence). Another example is with objective > functions that have very long evaluations and the practitioner would like > save the progress possibly including all evaluations/current best > solution(s) and then continue the routine. > (ii) More importantly to skip a few initiation steps in algorithm selection > and initiation. This is useful when the function is called thousands to > millions of times and only a few arguments/parameters need to be changed > while the algorithm selection etc. remain unchanged. I'm not sure if this > will have a significant effect for the local routines in `minimize` though. > > >> On Sun, Sep 8, 2019 at 5:57 AM Andrew Nelson wrote: >> >> I'd like to gauge the support for adding the global minimizers >> (dual_annealing, shgo, differential_evolution, basinhopping) as new >> `minimize` methods. >> >> The problems the the 'local' and 'global' optimizers are trying to solve >> are very similar and both are specified in the same way, so my thought is >> that it would be nice to access them via the same unified interface that >> `minimize` offers (but not deprecating the `shgo` function, etc). >> >> It's important that the users are able to understand the distinction >> between local and global optimisation and how they go about finding a >> minimum. I'm hoping that this could be made plain in the documentation. >> >> The change would allow the following: >> >> ``` >> # global minimizer >> minimize(func, x0, bounds=bounds, method='differential-evolution', >> constraints=constraints) >> # local minimizers >> minimize(func, x0, bounds=bounds, method='SLSQP', constraints=constraints) >> minimize(func, x0, bounds=bounds, method='trust-constr', >> constraints=constraints) >> minimize(func, x0, bounds=bounds, method='L-BFGS-B') >> ``` >> >> Please chip in with what your thoughts are, is it a bad idea, good idea, >> etc. >> >> -- >> _____________________________________ >> Dr. Andrew Nelson >> >> >> _____________________________________ >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > > > -- > Stefan Endres (MEng, AMIChemE, BEng (Hons) Chemical Engineering) > Postgraduate Student: Institute of Applied Materials > Department of Chemical Engineering, University of Pretoria > Cell: +27 (0) 82 972 42 89 > E-mail: Stefan.C.Endres at gmail.com > St. Number: 11004968 > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > Message: 2 > Date: Mon, 9 Sep 2019 09:32:01 +1000 > From: Andrew Nelson > To: SciPy Developers List > Subject: Re: [SciPy-Dev] feedback on adding global optimiser methods > to `minimize` > Message-ID: > > Content-Type: text/plain; charset="utf-8" > >> >> I agree that a single unified function for all the routines would be >> neater since all the algorithms have black-box functions as input anyway. I >> believe that any extra optional arguments to the global optimisation >> functions not already defined by `minimize` can be handled with the >> `options` dictionary object passed to the `minimize` function >> > > Thank you for feedback. > > One additional thought is the possibility of adding a unified `Minimizer` >> class to scipy.optimize. There are two reasons for this: >> > > As you may be aware this idea has been brought forward before, for the > reasons you suggested and more. The idea stalled unfortunately. Because it > would be a large changeset it's necessary to get everyone on board first > (via a SciPEP). > https://github.com/scipy/scipy/pull/8552 (SciPEP proposing it) > https://github.com/scipy/scipy/pull/8414 (PR exploring the idea) > Perhaps another way to bring this about is to setup a separate > implementation project and show it working before it could be incorporated. > That would give time and freedom for any issues to be ironed out. The > development of differential_evolution has certainly benefitted from the > solver being class based, but also from being a private interface, which > has allowed relatively large changes to be made to its functionality. > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > Message: 3 > Date: Mon, 9 Sep 2019 07:08:30 -0500 > From: Matt Newville > To: SciPy Developers List > Subject: Re: [SciPy-Dev] feedback on adding global optimiser methods > to `minimize` > Message-ID: > > Content-Type: text/plain; charset="utf-8" > >> On Sat, Sep 7, 2019 at 10:57 PM Andrew Nelson wrote: >> >> I'd like to gauge the support for adding the global minimizers >> (dual_annealing, shgo, differential_evolution, basinhopping) as new >> `minimize` methods. >> >> The problems the the 'local' and 'global' optimizers are trying to solve >> are very similar and both are specified in the same way, so my thought is >> that it would be nice to access them via the same unified interface that >> `minimize` offers (but not deprecating the `shgo` function, etc). >> >> It's important that the users are able to understand the distinction >> between local and global optimisation and how they go about finding a >> minimum. I'm hoping that this could be made plain in the documentation. >> >> The change would allow the following: >> >> ``` >> # global minimizer >> minimize(func, x0, bounds=bounds, method='differential-evolution', >> constraints=constraints) >> # local minimizers >> minimize(func, x0, bounds=bounds, method='SLSQP', constraints=constraints) >> minimize(func, x0, bounds=bounds, method='trust-constr', >> constraints=constraints) >> minimize(func, x0, bounds=bounds, method='L-BFGS-B') >> ``` >> >> Please chip in with what your thoughts are, is it a bad idea, good idea, >> etc. >> >> -- >> > > > Personally, I find `minimize()` to be a bit unwieldy I do understand the > desire to make things simple and uniform, but I'm not sure `minimize()` is > really doing that. Correct me if any of this is wrong, but the interface > for the current `minimize()` has the following features: > > 1. `minimize` takes 2 required arguments: func, x0. So far, so good. > 2. There are 14 named "methods" and a "custom object" method (which, > arguably means that the "global" methods are already supported). > 3. There are 3 optional arguments used for all methods with sensible > defaults: "args" , "tol" , and "callback". > 4. The "method" argument is actually optional with the default used being > a slightly complicated combination of other optional arguments. > 5. There are 6 optional arguments that each depends on which "method" is > used. > 6. Most but not all of the 14 methods support "jac". 4 of 6 options > ("hess", "hessp", "bounds", "constraints") are supported by fewer than half > the methods. > 7. One of the 6 optional arguments is called "options" and holds keyword > options specific to each method and not explicitly listed. > 8. Some of the options in "options" (say, "eps", "ftol") are supported by > more methods than some of the explicitly named optional arguments (say, > "constraints"). > 9. "constraints" is supported for 3 methods. It has 2 incompatible forms. > 10. "bounds" can either be a tuple of (called "min", "max" in the docs) > values or a "Bounds" object that has attributes "lb" and "ub". Are "lower" > and "upper" are too verbose? > 11. As if to stick a finger in your eye, "callback" requires a different > signature for exactly one of the 14 methods. > > It is all very well documented. But it is also extremely complicated. > And, honestly, not very good. > > It seems to me that there are really 14 (or 15) different minimization > methods. The `minimize()` function smashes these all into one function > with a jumble of options. I think the idea was to emphasize the > commonality of the interfaces, but the result sort of emphasizes what is > not common among the methods. Instead of creating a common way to say > "Bounds", "Constraints", "Hessian", "Tolerance", "Max Number of Function > Calls", or "Step Size" for all methods, it allows multiple variations and > then works extra hard to document the differences. > > How does this help the user (or person writing a wrapping library)? Can > the user just change the name of the method argument? In a simple case > that might work. But, in general, No, definitely not. Too many of the > arguments change validity when the `method` is changed. To the extent that > the user could change "minimize(...., method='foo', ...)" to > "minimize(...., method='bar', ...)", they could just as easily change > "_minimize_foo(....)" to "_minimize_bar(...)". That actually is less > typing. > > I guess in a fatalistic sense, I would say "Sure, why not add more > methods? You aren't going to make it any worse". Except that you will > want to add more options that are not common to the other methods. > Differential evolution itself has a dozen or so "strategy" options > ("strategy" being obviously different from "method", I guess). There are > "seeds" and "nworkers" and "max number of iterations" (whatever > "iteration" means for each method). I do not know if any of these have > uniform meaning; I guess not. It looks to me like "callback" would have > yet another variation. And, eventually, you will want to deprecate the > current working functions, breaking downstream code for the sake of a > uniform API that really is not uniform. > > The minimization methods really are different. They have different > options. IMO, it would be better to have one function per method and avoid > the a sprawling mess of a dispatch function. It would be nice if the > interfaces were as similar as possible, with similar concepts using > consistent names and values. > > I would definitely encourage a class-based Minimizer, but mostly in the > sense of being able to use inheritance to make it easier to have a common > interface across the different solvers: so not one Minimizer, but a > BaseMinizer and then PowellMinizer, BFGSMinimizer, KrylovMinimizer, etc. I > would suggest looking at the stats module and scikit-learn for > inspiration. And *then* make powell_minimize(), etc functions that create > and use these. Oh, and a Parameters class would be nice ;). > > I have no expectation of being listened too. We'll all be able to work > around the added complexity, just like we can sort of handle the similarly > infuriating complexity of `least_squares` (by basically treating it as 3 > separate functions). > > Finally, I will say that I am very glad to have `numpy.sin(x)`, > `numpy.cos(x)`, etc instead of `numpy.ufunc(x, method='sin')`. > > Sorry if that seems like a rant. > Cheers, > > --Matt > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > > ------------------------------ > > End of SciPy-Dev Digest, Vol 191, Issue 6 > ***************************************** From ikaspriskie at gmail.com Tue Sep 10 20:49:31 2019 From: ikaspriskie at gmail.com (Isabel Kaspriskie) Date: Tue, 10 Sep 2019 19:49:31 -0500 Subject: [SciPy-Dev] Building SciPy from source for development Message-ID: Hello, I am setting up my development environment in order to start contributing to the scipy repo. I have run into an error that I have not been able to diagnose myself. When running `python setup.py install --user`, I find the following error. The output is large, so I have only copied the end of it, but please let me know if it is acceptable to attach a text file of the output if that helps. ... building 'rectangular_lsap' library compiling C++ sources C compiler: g++ -pthread -B /home/isabelgk/miniconda3/envs/scipy-dev/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC creating build/temp.linux-x86_64-3.7/scipy/optimize/rectangular_lsap compile options: '-I/home/isabelgk/miniconda3/envs/scipy-dev/lib/python3.7/site-packages/numpy/core/include -I/home/isabelgk/miniconda3/envs/scipy-dev/include/python3.7m -c' g++: scipy/optimize/rectangular_lsap/rectangular_lsap.cpp error: Command "g++ -pthread -B /home/isabelgk/miniconda3/envs/scipy-dev/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/isabelgk/miniconda3/envs/scipy-dev/lib/python3.7/site-packages/numpy/core/include -I/home/isabelgk/miniconda3/envs/scipy-dev/include/python3.7m -c scipy/optimize/rectangular_lsap/rectangular_lsap.cpp -o build/temp.linux-x86_64-3.7/scipy/optimize/rectangular_lsap/rectangular_lsap.o -MMD -MF build/temp.linux-x86_64-3.7/scipy/optimize/rectangular_lsap/rectangular_lsap.o.d" failed with exit status 127 I have run the numpy diagnose utility, and it seems that there is a warning about being unable to read the registry to find my Fortran compiler setting. >>> python -c 'from numpy.f2py.diagnose import run; run()' ------ os.name='posix' ------ sys.platform='linux' ------ sys.version: 3.7.4 (default, Aug 13 2019, 20:35:49) [GCC 7.3.0] ------ sys.prefix: /home/isabelgk/miniconda3/envs/scipy-dev ------ sys.path=':/home/isabelgk/miniconda3/envs/scipy-dev/lib/python37.zip:/home/isabelgk/miniconda3/envs/scipy-dev/lib/python3.7:/home/isabelgk/miniconda3/envs/scipy-dev/lib/python3.7/lib-dynload:/home/isabelgk/.local/lib/python3.7/site-packages:/home/isabelgk/miniconda3/envs/scipy-dev/lib/python3.7/site-packages' ------ Found new numpy version '1.16.4' in /home/isabelgk/miniconda3/envs/scipy-dev/lib/python3.7/site-packages/numpy/__init__.py Found f2py2e version '2' in /home/isabelgk/miniconda3/envs/scipy-dev/lib/python3.7/site-packages/numpy/f2py/f2py2e.py Found numpy.distutils version '0.4.0' in '/home/isabelgk/miniconda3/envs/scipy-dev/lib/python3.7/site-packages/numpy/distutils/__init__.py' ------ Importing numpy.distutils.fcompiler ... ok ------ Checking availability of supported Fortran compilers: Warning: Can't read registry to find the necessary compiler setting Make sure that Python modules winreg, win32api or win32con are installed. Gnu95FCompiler instance properties: archiver = ['/usr/bin/gfortran', '-cr'] compile_switch = '-c' compiler_f77 = ['/usr/bin/gfortran', '-Wall', '-g', '-ffixed-form', '- fno-second-underscore', '-fPIC', '-O3', '-funroll-loops'] compiler_f90 = ['/usr/bin/gfortran', '-Wall', '-g', '-fno-second- underscore', '-fPIC', '-O3', '-funroll-loops'] compiler_fix = ['/usr/bin/gfortran', '-Wall', '-g', '-ffixed-form', '- fno-second-underscore', '-Wall', '-g', '-fno-second- underscore', '-fPIC', '-O3', '-funroll-loops'] libraries = ['gfortran'] library_dirs = ['/usr/lib/gcc/x86_64-linux-gnu/7', '/usr/lib/gcc/x86_64 -linux-gnu/7'] linker_exe = ['/usr/bin/gfortran', '-Wall', '-Wall'] linker_so = ['/usr/bin/gfortran', '-Wall', '-g', '-Wall', '-g', '- shared'] object_switch = '-o ' ranlib = ['/usr/bin/gfortran'] version = LooseVersion ('7') version_cmd = ['/usr/bin/gfortran', '-dumpversion'] Fortran compilers found: --fcompiler=gnu95 GNU Fortran 95 compiler (7) Compilers available for this platform, but not found: --fcompiler=absoft Absoft Corp Fortran Compiler --fcompiler=compaq Compaq Fortran Compiler --fcompiler=g95 G95 Fortran Compiler --fcompiler=gnu GNU Fortran 77 compiler --fcompiler=intel Intel Fortran Compiler for 32-bit apps --fcompiler=intele Intel Fortran Compiler for Itanium apps --fcompiler=intelem Intel Fortran Compiler for 64-bit apps --fcompiler=lahey Lahey/Fujitsu Fortran 95 Compiler --fcompiler=nag NAGWare Fortran 95 Compiler --fcompiler=nagfor NAG Fortran Compiler --fcompiler=pathf95 PathScale Fortran Compiler --fcompiler=pg Portland Group Fortran Compiler --fcompiler=vast Pacific-Sierra Research Fortran 90 Compiler Compilers not available on this platform: --fcompiler=flang Portland Group Fortran LLVM Compiler --fcompiler=hpux HP Fortran 90 Compiler --fcompiler=ibm IBM XL Fortran Compiler --fcompiler=intelev Intel Visual Fortran Compiler for Itanium apps --fcompiler=intelv Intel Visual Fortran Compiler for 32-bit apps --fcompiler=intelvem Intel Visual Fortran Compiler for 64-bit apps --fcompiler=mips MIPSpro Fortran Compiler --fcompiler=none Fake Fortran compiler --fcompiler=sun Sun or Forte Fortran 95 Compiler For compiler details, run 'config_fc --verbose' setup command. ------ Importing numpy.distutils.cpuinfo ... ok ------ CPU information: CPUInfoBase__get_nbits getNCPUs has_mmx has_sse has_sse2 has_sse3 has_ssse3 is_64bit is_Intel is_i686 ------ Thank you in advance, Isabel -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikofski at berkeley.edu Tue Sep 10 23:02:25 2019 From: mikofski at berkeley.edu (Dr. Mark Alexander Mikofski PhD) Date: Tue, 10 Sep 2019 20:02:25 -0700 Subject: [SciPy-Dev] Building SciPy from source for development In-Reply-To: References: Message-ID: Probably a more experienced contributor will also answer better, but what worked for me was to use the runtest.py script in a virtualenv: https://github.com/scipy/scipy/blob/master/runtests.py there's good info here in the docs, look for "runtest" under "recommended dev setup" https://docs.scipy.org/doc/scipy/reference/hacking.html there's some more info in the numpy docs on runtest.py and recommended dev setup: https://docs.scipy.org/doc/numpy/dev/development_environment.html If you are trying to build scipy to install or package, then you may need to change the settings in site.cfg (or setup.cfg), see the example https://github.com/scipy/scipy/blob/master/site.cfg.example I guess for Linux the INSTALL instructions are similar to the docs, but I don't think they will work with Anaconda/miniconda https://github.com/scipy/scipy/blob/master/INSTALL.rst.txt Here's more info in the documentation: * building from source: https://docs.scipy.org/doc/scipy/reference/building/index.html * dev guide: https://docs.scipy.org/doc/scipy/reference/dev/index.html * hacking: https://docs.scipy.org/doc/scipy/reference/hacking.html >From your diagnostic messages, SciPy seems to think you are on a Windows machine instead of a Linux machine, not sure why: >Warning: Can't read registry to find the necessary compiler setting >Make sure that Python modules winreg, win32api or win32con are installed. Linux doesn't have a "registry" does it? And winreg, win32api, and win32con are python bindings for Windows machines right? As I said, I first created a virtualenv, activated the venv, installed the requirements (Numpy, pytest, cython, etc., see build docs), and then used runtest.py - I did not use anaconda/miniconda Are you on a mac, linux, or windows machine? On mac I used homebrew, and on windows I used WSL. https://brew.sh/ https://docs.microsoft.com/en-us/windows/wsl/install-win10 Good luck! PS: If I have given any incorrect information, I do apologize. On Tue, Sep 10, 2019 at 5:50 PM Isabel Kaspriskie wrote: > Hello, > > I am setting up my development environment in order to start contributing > to the scipy repo. I have run into an error that I have not been able to > diagnose myself. When running `python setup.py install --user`, I find the > following error. The output is large, so I have only copied the end of it, > but please let me know if it is acceptable to attach a text file of the > output if that helps. > > ... > building 'rectangular_lsap' library > compiling C++ sources > C compiler: g++ -pthread -B > /home/isabelgk/miniconda3/envs/scipy-dev/compiler_compat -Wl,--sysroot=/ > -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC > > creating build/temp.linux-x86_64-3.7/scipy/optimize/rectangular_lsap > compile options: > '-I/home/isabelgk/miniconda3/envs/scipy-dev/lib/python3.7/site-packages/numpy/core/include > -I/home/isabelgk/miniconda3/envs/scipy-dev/include/python3.7m -c' > g++: scipy/optimize/rectangular_lsap/rectangular_lsap.cpp > error: Command "g++ -pthread -B > /home/isabelgk/miniconda3/envs/scipy-dev/compiler_compat -Wl,--sysroot=/ > -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC > -I/home/isabelgk/miniconda3/envs/scipy-dev/lib/python3.7/site-packages/numpy/core/include > -I/home/isabelgk/miniconda3/envs/scipy-dev/include/python3.7m -c > scipy/optimize/rectangular_lsap/rectangular_lsap.cpp -o > build/temp.linux-x86_64-3.7/scipy/optimize/rectangular_lsap/rectangular_lsap.o > -MMD -MF > build/temp.linux-x86_64-3.7/scipy/optimize/rectangular_lsap/rectangular_lsap.o.d" > failed with exit status 127 > > > > I have run the numpy diagnose utility, and it seems that there is a > warning about being unable to read the registry to find my Fortran compiler > setting. > > >>> python -c 'from numpy.f2py.diagnose import run; run()' > > ------ > os.name='posix' > ------ > sys.platform='linux' > ------ > sys.version: > 3.7.4 (default, Aug 13 2019, 20:35:49) > [GCC 7.3.0] > ------ > sys.prefix: > /home/isabelgk/miniconda3/envs/scipy-dev > ------ > sys.path=':/home/isabelgk/miniconda3/envs/scipy-dev/lib/python37.zip:/home/isabelgk/miniconda3/envs/scipy-dev/lib/python3.7:/home/isabelgk/miniconda3/envs/scipy-dev/lib/python3.7/lib-dynload:/home/isabelgk/.local/lib/python3.7/site-packages:/home/isabelgk/miniconda3/envs/scipy-dev/lib/python3.7/site-packages' > ------ > Found new numpy version '1.16.4' in /home/isabelgk/miniconda3/envs/scipy-dev/lib/python3.7/site-packages/numpy/__init__.py > Found f2py2e version '2' in /home/isabelgk/miniconda3/envs/scipy-dev/lib/python3.7/site-packages/numpy/f2py/f2py2e.py > Found numpy.distutils version '0.4.0' in '/home/isabelgk/miniconda3/envs/scipy-dev/lib/python3.7/site-packages/numpy/distutils/__init__.py' > ------ > Importing numpy.distutils.fcompiler ... ok > ------ > Checking availability of supported Fortran compilers: > Warning: Can't read registry to find the necessary compiler setting > Make sure that Python modules winreg, win32api or win32con are installed. > Gnu95FCompiler instance properties: > archiver = ['/usr/bin/gfortran', '-cr'] > compile_switch = '-c' > compiler_f77 = ['/usr/bin/gfortran', '-Wall', '-g', '-ffixed-form', '- > fno-second-underscore', '-fPIC', '-O3', '-funroll-loops'] > compiler_f90 = ['/usr/bin/gfortran', '-Wall', '-g', '-fno-second- > underscore', '-fPIC', '-O3', '-funroll-loops'] > compiler_fix = ['/usr/bin/gfortran', '-Wall', '-g', '-ffixed-form', '- > fno-second-underscore', '-Wall', '-g', '-fno-second- > underscore', '-fPIC', '-O3', '-funroll-loops'] > libraries = ['gfortran'] > library_dirs = ['/usr/lib/gcc/x86_64-linux-gnu/7', '/usr/lib/gcc/x86_64 > -linux-gnu/7'] > linker_exe = ['/usr/bin/gfortran', '-Wall', '-Wall'] > linker_so = ['/usr/bin/gfortran', '-Wall', '-g', '-Wall', '-g', '- > shared'] > object_switch = '-o ' > ranlib = ['/usr/bin/gfortran'] > version = LooseVersion ('7') > version_cmd = ['/usr/bin/gfortran', '-dumpversion'] > Fortran compilers found: > --fcompiler=gnu95 GNU Fortran 95 compiler (7) > Compilers available for this platform, but not found: > --fcompiler=absoft Absoft Corp Fortran Compiler > --fcompiler=compaq Compaq Fortran Compiler > --fcompiler=g95 G95 Fortran Compiler > --fcompiler=gnu GNU Fortran 77 compiler > --fcompiler=intel Intel Fortran Compiler for 32-bit apps > --fcompiler=intele Intel Fortran Compiler for Itanium apps > --fcompiler=intelem Intel Fortran Compiler for 64-bit apps > --fcompiler=lahey Lahey/Fujitsu Fortran 95 Compiler > --fcompiler=nag NAGWare Fortran 95 Compiler > --fcompiler=nagfor NAG Fortran Compiler > --fcompiler=pathf95 PathScale Fortran Compiler > --fcompiler=pg Portland Group Fortran Compiler > --fcompiler=vast Pacific-Sierra Research Fortran 90 Compiler > Compilers not available on this platform: > --fcompiler=flang Portland Group Fortran LLVM Compiler > --fcompiler=hpux HP Fortran 90 Compiler > --fcompiler=ibm IBM XL Fortran Compiler > --fcompiler=intelev Intel Visual Fortran Compiler for Itanium apps > --fcompiler=intelv Intel Visual Fortran Compiler for 32-bit apps > --fcompiler=intelvem Intel Visual Fortran Compiler for 64-bit apps > --fcompiler=mips MIPSpro Fortran Compiler > --fcompiler=none Fake Fortran compiler > --fcompiler=sun Sun or Forte Fortran 95 Compiler > For compiler details, run 'config_fc --verbose' setup command. > ------ > Importing numpy.distutils.cpuinfo ... ok > ------ > CPU information: CPUInfoBase__get_nbits getNCPUs has_mmx has_sse has_sse2 has_sse3 has_ssse3 is_64bit is_Intel is_i686 ------ > > > Thank you in advance, > > Isabel > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -- Mark Mikofski, PhD (2005) *Fiat Lux* -------------- next part -------------- An HTML attachment was scrubbed... URL: From haberland at ucla.edu Tue Sep 10 23:51:38 2019 From: haberland at ucla.edu (Matt Haberland) Date: Tue, 10 Sep 2019 20:51:38 -0700 Subject: [SciPy-Dev] Building SciPy from source for development In-Reply-To: References: Message-ID: Hi Isabel, Mark - There has been a lot of work on the developer documentation since 1.3 was released, so in general I would suggest using the development version of the documentation: http://scipy.github.io/devdocs/ Rather than those links. However, there are important updates to the new Linux development environment quickstart guide in progress: https://github.com/scipy/scipy/pull/10461 Isabel, would you try following the very latest Linux instructions: https://15105-1460385-gh.circle-artifacts.com/0/html-scipyorg/dev/contributor/quickstart_ubuntu.html#quickstart-ubuntu And let me know if you run into any issues? They are written for full Anaconda, but you can do it with Miniconda with minor adaptations (e.g. you might be prompted to install `conda-build` if you try to use the `conda develop` command) if you prefer. Matt On Tue, Sep 10, 2019, 8:03 PM Dr. Mark Alexander Mikofski PhD < mikofski at berkeley.edu> wrote: > Probably a more experienced contributor will also answer better, but what > worked for me was to use the runtest.py script in a virtualenv: > https://github.com/scipy/scipy/blob/master/runtests.py > > there's good info here in the docs, look for "runtest" under "recommended > dev setup" > https://docs.scipy.org/doc/scipy/reference/hacking.html > > there's some more info in the numpy docs on runtest.py and recommended dev > setup: > https://docs.scipy.org/doc/numpy/dev/development_environment.html > > If you are trying to build scipy to install or package, then you may need > to change the settings in site.cfg (or setup.cfg), see the example > https://github.com/scipy/scipy/blob/master/site.cfg.example > > I guess for Linux the INSTALL instructions are similar to the docs, but I > don't think they will work with Anaconda/miniconda > https://github.com/scipy/scipy/blob/master/INSTALL.rst.txt > > Here's more info in the documentation: > * building from source: > https://docs.scipy.org/doc/scipy/reference/building/index.html > * dev guide: https://docs.scipy.org/doc/scipy/reference/dev/index.html > * hacking: https://docs.scipy.org/doc/scipy/reference/hacking.html > > From your diagnostic messages, SciPy seems to think you are on a Windows > machine instead of a Linux machine, not sure why: > >Warning: Can't read registry to find the necessary compiler setting > >Make sure that Python modules winreg, win32api or win32con are installed. > > Linux doesn't have a "registry" does it? And winreg, win32api, and > win32con are python bindings for Windows machines right? > > As I said, I first created a virtualenv, activated the venv, installed the > requirements (Numpy, pytest, cython, etc., see build docs), and then used > runtest.py - I did not use anaconda/miniconda > > Are you on a mac, linux, or windows machine? On mac I used homebrew, and > on windows I used WSL. > https://brew.sh/ > https://docs.microsoft.com/en-us/windows/wsl/install-win10 > > Good luck! > > PS: If I have given any incorrect information, I do apologize. > > > On Tue, Sep 10, 2019 at 5:50 PM Isabel Kaspriskie > wrote: > >> Hello, >> >> I am setting up my development environment in order to start contributing >> to the scipy repo. I have run into an error that I have not been able to >> diagnose myself. When running `python setup.py install --user`, I find the >> following error. The output is large, so I have only copied the end of it, >> but please let me know if it is acceptable to attach a text file of the >> output if that helps. >> >> ... >> building 'rectangular_lsap' library >> compiling C++ sources >> C compiler: g++ -pthread -B >> /home/isabelgk/miniconda3/envs/scipy-dev/compiler_compat -Wl,--sysroot=/ >> -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC >> >> creating build/temp.linux-x86_64-3.7/scipy/optimize/rectangular_lsap >> compile options: >> '-I/home/isabelgk/miniconda3/envs/scipy-dev/lib/python3.7/site-packages/numpy/core/include >> -I/home/isabelgk/miniconda3/envs/scipy-dev/include/python3.7m -c' >> g++: scipy/optimize/rectangular_lsap/rectangular_lsap.cpp >> error: Command "g++ -pthread -B >> /home/isabelgk/miniconda3/envs/scipy-dev/compiler_compat -Wl,--sysroot=/ >> -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC >> -I/home/isabelgk/miniconda3/envs/scipy-dev/lib/python3.7/site-packages/numpy/core/include >> -I/home/isabelgk/miniconda3/envs/scipy-dev/include/python3.7m -c >> scipy/optimize/rectangular_lsap/rectangular_lsap.cpp -o >> build/temp.linux-x86_64-3.7/scipy/optimize/rectangular_lsap/rectangular_lsap.o >> -MMD -MF >> build/temp.linux-x86_64-3.7/scipy/optimize/rectangular_lsap/rectangular_lsap.o.d" >> failed with exit status 127 >> >> >> >> I have run the numpy diagnose utility, and it seems that there is a >> warning about being unable to read the registry to find my Fortran compiler >> setting. >> >> >>> python -c 'from numpy.f2py.diagnose import run; run()' >> >> ------ >> os.name='posix' >> ------ >> sys.platform='linux' >> ------ >> sys.version: >> 3.7.4 (default, Aug 13 2019, 20:35:49) >> [GCC 7.3.0] >> ------ >> sys.prefix: >> /home/isabelgk/miniconda3/envs/scipy-dev >> ------ >> sys.path=':/home/isabelgk/miniconda3/envs/scipy-dev/lib/python37.zip:/home/isabelgk/miniconda3/envs/scipy-dev/lib/python3.7:/home/isabelgk/miniconda3/envs/scipy-dev/lib/python3.7/lib-dynload:/home/isabelgk/.local/lib/python3.7/site-packages:/home/isabelgk/miniconda3/envs/scipy-dev/lib/python3.7/site-packages' >> ------ >> Found new numpy version '1.16.4' in /home/isabelgk/miniconda3/envs/scipy-dev/lib/python3.7/site-packages/numpy/__init__.py >> Found f2py2e version '2' in /home/isabelgk/miniconda3/envs/scipy-dev/lib/python3.7/site-packages/numpy/f2py/f2py2e.py >> Found numpy.distutils version '0.4.0' in '/home/isabelgk/miniconda3/envs/scipy-dev/lib/python3.7/site-packages/numpy/distutils/__init__.py' >> ------ >> Importing numpy.distutils.fcompiler ... ok >> ------ >> Checking availability of supported Fortran compilers: >> Warning: Can't read registry to find the necessary compiler setting >> Make sure that Python modules winreg, win32api or win32con are installed. >> Gnu95FCompiler instance properties: >> archiver = ['/usr/bin/gfortran', '-cr'] >> compile_switch = '-c' >> compiler_f77 = ['/usr/bin/gfortran', '-Wall', '-g', '-ffixed-form', '- >> fno-second-underscore', '-fPIC', '-O3', '-funroll-loops'] >> compiler_f90 = ['/usr/bin/gfortran', '-Wall', '-g', '-fno-second- >> underscore', '-fPIC', '-O3', '-funroll-loops'] >> compiler_fix = ['/usr/bin/gfortran', '-Wall', '-g', '-ffixed-form', '- >> fno-second-underscore', '-Wall', '-g', '-fno-second- >> underscore', '-fPIC', '-O3', '-funroll-loops'] >> libraries = ['gfortran'] >> library_dirs = ['/usr/lib/gcc/x86_64-linux-gnu/7', '/usr/lib/gcc/x86_64 >> -linux-gnu/7'] >> linker_exe = ['/usr/bin/gfortran', '-Wall', '-Wall'] >> linker_so = ['/usr/bin/gfortran', '-Wall', '-g', '-Wall', '-g', '- >> shared'] >> object_switch = '-o ' >> ranlib = ['/usr/bin/gfortran'] >> version = LooseVersion ('7') >> version_cmd = ['/usr/bin/gfortran', '-dumpversion'] >> Fortran compilers found: >> --fcompiler=gnu95 GNU Fortran 95 compiler (7) >> Compilers available for this platform, but not found: >> --fcompiler=absoft Absoft Corp Fortran Compiler >> --fcompiler=compaq Compaq Fortran Compiler >> --fcompiler=g95 G95 Fortran Compiler >> --fcompiler=gnu GNU Fortran 77 compiler >> --fcompiler=intel Intel Fortran Compiler for 32-bit apps >> --fcompiler=intele Intel Fortran Compiler for Itanium apps >> --fcompiler=intelem Intel Fortran Compiler for 64-bit apps >> --fcompiler=lahey Lahey/Fujitsu Fortran 95 Compiler >> --fcompiler=nag NAGWare Fortran 95 Compiler >> --fcompiler=nagfor NAG Fortran Compiler >> --fcompiler=pathf95 PathScale Fortran Compiler >> --fcompiler=pg Portland Group Fortran Compiler >> --fcompiler=vast Pacific-Sierra Research Fortran 90 Compiler >> Compilers not available on this platform: >> --fcompiler=flang Portland Group Fortran LLVM Compiler >> --fcompiler=hpux HP Fortran 90 Compiler >> --fcompiler=ibm IBM XL Fortran Compiler >> --fcompiler=intelev Intel Visual Fortran Compiler for Itanium apps >> --fcompiler=intelv Intel Visual Fortran Compiler for 32-bit apps >> --fcompiler=intelvem Intel Visual Fortran Compiler for 64-bit apps >> --fcompiler=mips MIPSpro Fortran Compiler >> --fcompiler=none Fake Fortran compiler >> --fcompiler=sun Sun or Forte Fortran 95 Compiler >> For compiler details, run 'config_fc --verbose' setup command. >> ------ >> Importing numpy.distutils.cpuinfo ... ok >> ------ >> CPU information: CPUInfoBase__get_nbits getNCPUs has_mmx has_sse has_sse2 has_sse3 has_ssse3 is_64bit is_Intel is_i686 ------ >> >> >> Thank you in advance, >> >> Isabel >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > > > -- > Mark Mikofski, PhD (2005) > *Fiat Lux* > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Sep 11 01:14:19 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 10 Sep 2019 22:14:19 -0700 Subject: [SciPy-Dev] Building SciPy from source for development In-Reply-To: References: Message-ID: Hi Isabel, On Tue, Sep 10, 2019 at 5:49 PM Isabel Kaspriskie wrote: > Hello, > > I am setting up my development environment in order to start contributing > to the scipy repo. I have run into an error that I have not been able to > diagnose myself. When running `python setup.py install --user`, I find the > following error. The output is large, so I have only copied the end of it, > but please let me know if it is acceptable to attach a text file of the > output if that helps. > What you can do is put the full build log in a gist ( https://gist.github.com/) and link to it. The actual error isn't visible, it could be many things (and I doubt it's that f2py warning). A full build log usually lets someone spot the issue quickly. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Sep 11 17:57:11 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 11 Sep 2019 14:57:11 -0700 Subject: [SciPy-Dev] Building SciPy from source for development In-Reply-To: References: Message-ID: On Tue, Sep 10, 2019 at 10:14 PM Ralf Gommers wrote: > Hi Isabel, > > > On Tue, Sep 10, 2019 at 5:49 PM Isabel Kaspriskie > wrote: > >> Hello, >> >> I am setting up my development environment in order to start contributing >> to the scipy repo. I have run into an error that I have not been able to >> diagnose myself. When running `python setup.py install --user`, I find the >> following error. The output is large, so I have only copied the end of it, >> but please let me know if it is acceptable to attach a text file of the >> output if that helps. >> > > What you can do is put the full build log in a gist ( > https://gist.github.com/) and link to it. The actual error isn't visible, > it could be many things (and I doubt it's that f2py warning). A full build > log usually lets someone spot the issue quickly. > Somehow Kai's answer bounced from the list, it may help too so here it is: An exit code of 127 occurs when a required command is not found in your $PATH. In this case you are trying to compile SciPy using g++. I suspect this occurs because g++ is either not installed in which case you will need to install it (in Ubuntu you can install these using ``apt install g++``, this may require sudo). If it is already installed it may be that miniconda does not find g++ as expected. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ikaspriskie at gmail.com Wed Sep 11 19:57:45 2019 From: ikaspriskie at gmail.com (Isabel Kaspriskie) Date: Wed, 11 Sep 2019 18:57:45 -0500 Subject: [SciPy-Dev] Building SciPy from source for development In-Reply-To: References: Message-ID: Thank you everyone! The only things I had to do differently: - Renamed miniconda3/lib/libgfortran.so - apt installed g++ In my build output, I saw that I was using both g++ and gcc, and I'm not sure why that might be, but the build was successful and tests ran fine. Cheers, Isabel On Wed, Sep 11, 2019 at 4:57 PM Ralf Gommers wrote: > > > On Tue, Sep 10, 2019 at 10:14 PM Ralf Gommers > wrote: > >> Hi Isabel, >> >> >> On Tue, Sep 10, 2019 at 5:49 PM Isabel Kaspriskie >> wrote: >> >>> Hello, >>> >>> I am setting up my development environment in order to start >>> contributing to the scipy repo. I have run into an error that I have not >>> been able to diagnose myself. When running `python setup.py install >>> --user`, I find the following error. The output is large, so I have only >>> copied the end of it, but please let me know if it is acceptable to attach >>> a text file of the output if that helps. >>> >> >> What you can do is put the full build log in a gist ( >> https://gist.github.com/) and link to it. The actual error isn't >> visible, it could be many things (and I doubt it's that f2py warning). A >> full build log usually lets someone spot the issue quickly. >> > > Somehow Kai's answer bounced from the list, it may help too so here it is: > > An exit code of 127 occurs when a required command is not found in your > $PATH. In this case you are trying to compile SciPy using g++. > > I suspect this occurs because g++ is either not installed in which case > you will need to install it (in Ubuntu you can install these using ``apt > install g++``, this may require sudo). If it is already installed it may be > that miniconda does not find g++ as expected. > > Cheers, > Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Sep 11 20:57:26 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 11 Sep 2019 17:57:26 -0700 Subject: [SciPy-Dev] Building SciPy from source for development In-Reply-To: References: Message-ID: On Wed, Sep 11, 2019 at 4:58 PM Isabel Kaspriskie wrote: > Thank you everyone! The only things I had to do differently: > > - Renamed miniconda3/lib/libgfortran.so > - apt installed g++ > > In my build output, I saw that I was using both g++ and gcc, and I'm not > sure why that might be, > That's because SciPy contained both C and C++ code. but the build was successful and tests ran fine. > Glad that solved it. Cheers, Ralf > Cheers, > Isabel > > On Wed, Sep 11, 2019 at 4:57 PM Ralf Gommers > wrote: > >> >> >> On Tue, Sep 10, 2019 at 10:14 PM Ralf Gommers >> wrote: >> >>> Hi Isabel, >>> >>> >>> On Tue, Sep 10, 2019 at 5:49 PM Isabel Kaspriskie >>> wrote: >>> >>>> Hello, >>>> >>>> I am setting up my development environment in order to start >>>> contributing to the scipy repo. I have run into an error that I have not >>>> been able to diagnose myself. When running `python setup.py install >>>> --user`, I find the following error. The output is large, so I have only >>>> copied the end of it, but please let me know if it is acceptable to attach >>>> a text file of the output if that helps. >>>> >>> >>> What you can do is put the full build log in a gist ( >>> https://gist.github.com/) and link to it. The actual error isn't >>> visible, it could be many things (and I doubt it's that f2py warning). A >>> full build log usually lets someone spot the issue quickly. >>> >> >> Somehow Kai's answer bounced from the list, it may help too so here it is: >> >> An exit code of 127 occurs when a required command is not found in your >> $PATH. In this case you are trying to compile SciPy using g++. >> >> I suspect this occurs because g++ is either not installed in which case >> you will need to install it (in Ubuntu you can install these using ``apt >> install g++``, this may require sudo). If it is already installed it may be >> that miniconda does not find g++ as expected. >> >> Cheers, >> Ralf >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From maja.gwozdz.mkg33 at gmail.com Mon Sep 16 15:13:59 2019 From: maja.gwozdz.mkg33 at gmail.com (Maja Gwozdz) Date: Mon, 16 Sep 2019 21:13:59 +0200 Subject: [SciPy-Dev] Documentation: user survey Message-ID: Hi everyone, I'm currently working on improving the SciPy documentation during the Google Season of Docs. One of my projects is about a user survey. The basic idea is to get (documentation) feedback from SciPy users and try to improve our docs accordingly. It'd be great if you could take a look at the survey form and say if you have any comments that could help me. You can access it here: https://docs.google.com/forms/d/e/1FAIpQLSeBAO0UFKDZyKpg2XzRslsLJVHU61ugjc18-2PVEabTQg2_6g/viewform?usp=sf_link Also, I'm compiling a list of websites where I could deploy the survey to get a fair amount of responses. If you know any such websites, please let me know. Your help is much appreciated! All the best, Maja Gwozdz -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Sep 18 01:20:53 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 17 Sep 2019 22:20:53 -0700 Subject: [SciPy-Dev] Documentation: user survey In-Reply-To: References: Message-ID: HI Maja, On Mon, Sep 16, 2019 at 12:14 PM Maja Gwozdz wrote: > Hi everyone, > > I'm currently working on improving the SciPy documentation during the > Google Season of Docs. One of my projects is about a user survey. The basic > idea is to get (documentation) feedback from SciPy users and try to improve > our docs accordingly. > > It'd be great if you could take a look at the survey form and say if you > have any comments that could help me. You can access it here: > https://docs.google.com/forms/d/e/1FAIpQLSeBAO0UFKDZyKpg2XzRslsLJVHU61ugjc18-2PVEabTQg2_6g/viewform?usp=sf_link > This looks like a very good start. A few thoughts: In the description at the top of the survey, perhaps add a link to http://scipy.github.io/devdocs/? It would be good to get some info on the users answering the questions. E.g. beginner/imtermediate/advanced or years of experience. That way you can analyze the other answers better, for example if only 10% of respondents are beginners and they all say that tutorials need improving, that's 100% of beginners that need this. I'm not a survey design expert, but I think it's more common to have 5 options for the "are you satisfied" or "is this clear" type of questions. With 3, you may get a lot of "neutral" responses. To "which doc features need to be improved/added" you could add "Ease of navigation". A question I would find interesting is "how do you view the docs (check all that apply)". Possible answers: online html docs, pdf, view docstring in terminal, Help feature in an IDE, Other. > Also, I'm compiling a list of websites where I could deploy the survey to > get a fair amount of responses. If you know any such websites, please let > me know. > When you say deploy I'm thinking SurveyMonkey or the like - you need just one of those, the Google Form you have may be perfectly fine. Response rates should be mostly determined by where we publicize the request for people to participate (or maybe that's what you meant). To mind come: mailing lists (SciPy and other projects, PyData, NumFOCUS), Twitter, post it on scipy.org, the scipy IRC and numpy Gitter channels, and a blog post that shows up on Planet SciPy and Planet Python. Cheers, Ralf > Your help is much appreciated! > > All the best, > Maja Gwozdz > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From maja.gwozdz.mkg33 at gmail.com Wed Sep 18 04:57:30 2019 From: maja.gwozdz.mkg33 at gmail.com (Maja Gwozdz) Date: Wed, 18 Sep 2019 10:57:30 +0200 Subject: [SciPy-Dev] Documentation: user survey In-Reply-To: References: Message-ID: Hi Ralf, thanks for your comments, I've adjusted the survey accordingly. I've decided to phrase the 'experience question' in terms of levels, i.e., beginner, intermediate, etc., due to the fact that 'years of experience' is probably a more difficult question to answer (at least for me). As for the linear scale, I now modified it and made it a standard 5-point Likert scale. The reason I initially opted for a 3-point scale was to avoid centre tendency bias that might crop up in its 5-point counterpart. But I don't think it's a big deal. You're right, a 5-point scale is more likely to give us fewer neutral responses, which is good for our purposes. I've also discarded the uppercase response options because they looked ugly, stylistically speaking. In order to facilitate a quicker response, I rearranged the response options in 'what features could be improved/added' alphabetically. I've also included the other questions you proposed and those kindly suggested by one other person on the mailing list. If there are no objections / further comments from the Community, I'll start publishing the user survey in the appropriate places (say, in two days). One more important question: should we limit the number of responses to one? This would require signing in, which is less than optimal. Do we have any legitimate concerns about getting spam? You can see the new version here: https://docs.google.com/forms/d/e/1FAIpQLSeBAO0UFKDZyKpg2XzRslsLJVHU61ugjc18-2PVEabTQg2_6g/viewform?usp=sf_link . I also took the liberty of including a header with a scientific flavour (no copyright infringement here, it's taken from the generic Google headers). All the best, Maja On Wed, Sep 18, 2019 at 7:21 AM Ralf Gommers wrote: > HI Maja, > > > On Mon, Sep 16, 2019 at 12:14 PM Maja Gwozdz > wrote: > >> Hi everyone, >> >> I'm currently working on improving the SciPy documentation during the >> Google Season of Docs. One of my projects is about a user survey. The basic >> idea is to get (documentation) feedback from SciPy users and try to improve >> our docs accordingly. >> >> It'd be great if you could take a look at the survey form and say if you >> have any comments that could help me. You can access it here: >> https://docs.google.com/forms/d/e/1FAIpQLSeBAO0UFKDZyKpg2XzRslsLJVHU61ugjc18-2PVEabTQg2_6g/viewform?usp=sf_link >> > > This looks like a very good start. A few thoughts: > > In the description at the top of the survey, perhaps add a link to > http://scipy.github.io/devdocs/? > > It would be good to get some info on the users answering the questions. > E.g. beginner/imtermediate/advanced or years of experience. That way you > can analyze the other answers better, for example if only 10% of > respondents are beginners and they all say that tutorials need improving, > that's 100% of beginners that need this. > > I'm not a survey design expert, but I think it's more common to have 5 > options for the "are you satisfied" or "is this clear" type of questions. > With 3, you may get a lot of "neutral" responses. > > To "which doc features need to be improved/added" you could add "Ease of > navigation". > > A question I would find interesting is "how do you view the docs (check > all that apply)". Possible answers: online html docs, pdf, view docstring > in terminal, Help feature in an IDE, Other. > > > >> Also, I'm compiling a list of websites where I could deploy the survey to >> get a fair amount of responses. If you know any such websites, please let >> me know. >> > > When you say deploy I'm thinking SurveyMonkey or the like - you need just > one of those, the Google Form you have may be perfectly fine. Response > rates should be mostly determined by where we publicize the request for > people to participate (or maybe that's what you meant). To mind come: > mailing lists (SciPy and other projects, PyData, NumFOCUS), Twitter, post > it on scipy.org, the scipy IRC and numpy Gitter channels, and a blog post > that shows up on Planet SciPy and Planet Python. > > Cheers, > Ralf > > > >> Your help is much appreciated! >> >> All the best, >> Maja Gwozdz >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From egouden at gmail.com Wed Sep 18 05:29:12 2019 From: egouden at gmail.com (Edouard Goudenhoofdt) Date: Wed, 18 Sep 2019 11:29:12 +0200 Subject: [SciPy-Dev] improvement to binned statistic Message-ID: Dear scipy developers, One could use scipy.stats.binned_statistic_dd for the same sample points but for values available at different times. Currently this involves the computation of the bin numbers every time the function is called. Therefore I would like to add an optional argument "binnumbers" to skip this step when calling the function again. Best regards, Edouard Goudenhoofdt -------------- next part -------------- An HTML attachment was scrubbed... URL: From egouden at gmail.com Wed Sep 18 05:50:25 2019 From: egouden at gmail.com (Edouard Goudenhoofdt) Date: Wed, 18 Sep 2019 11:50:25 +0200 Subject: [SciPy-Dev] Improvement to regular grid interpolation Message-ID: Dear scipy developers, One could use *scipy.interpolate.RegularGridInterpolator* at different times with different values but the same source grid and target points. Currently this would recompute the indices each time the function is called. I would like to add the possibility to provide the target points at initialisation and the values when calling the object. Best regards, Edouard Goudenhoofdt -------------- next part -------------- An HTML attachment was scrubbed... URL: From Dieter at Werthmuller.org Wed Sep 18 06:41:27 2019 From: Dieter at Werthmuller.org (=?UTF-8?Q?Dieter_Werthm=c3=bcller?=) Date: Wed, 18 Sep 2019 12:41:27 +0200 Subject: [SciPy-Dev] Improvement to regular grid interpolation In-Reply-To: References: Message-ID: Edouard, I think this thread touched on that a bit: https://mail.python.org/pipermail/scipy-dev/2019-May/023540.html So Simon Clift might have already some functional code for that which could serve as a starting point. I personally would be certainly interested in the functionality. Dieter On 18/09/2019 11:50, Edouard Goudenhoofdt wrote: > Dear scipy developers, > > One could use /scipy.interpolate.RegularGridInterpolator/ at different > times with different values but the same source grid and target points. > Currently this would recompute the indices each time the function is called. > I would like to add the possibility to provide the target points at > initialisation and the values when calling the object. > > Best regards, > > Edouard Goudenhoofdt > // > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > From egouden at gmail.com Wed Sep 18 08:33:33 2019 From: egouden at gmail.com (Edouard Goudenhoofdt) Date: Wed, 18 Sep 2019 14:33:33 +0200 Subject: [SciPy-Dev] Improvement to regular grid interpolation In-Reply-To: References: Message-ID: This is clearly a duplicate of your previous post (How can I search the mailing list?). I think this is a basic task for many geoscientists. I did not make the effort to understand Simon Clift smarter approach. But I already implemented a quick solution using a switch for the two use cases. May I submit a pull request? On Wed, Sep 18, 2019 at 12:47 PM Dieter Werthm?ller wrote: > Edouard, > > I think this thread touched on that a bit: > https://mail.python.org/pipermail/scipy-dev/2019-May/023540.html > > So Simon Clift might have already some functional code for that which > could serve as a starting point. > > I personally would be certainly interested in the functionality. > > Dieter > > > On 18/09/2019 11:50, Edouard Goudenhoofdt wrote: > > Dear scipy developers, > > > > One could use /scipy.interpolate.RegularGridInterpolator/ at different > > times with different values but the same source grid and target points. > > Currently this would recompute the indices each time the function is > called. > > I would like to add the possibility to provide the target points at > > initialisation and the values when calling the object. > > > > Best regards, > > > > Edouard Goudenhoofdt > > // > > > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at python.org > > https://mail.python.org/mailman/listinfo/scipy-dev > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Sep 18 08:49:40 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 18 Sep 2019 14:49:40 +0200 Subject: [SciPy-Dev] Improvement to regular grid interpolation In-Reply-To: References: Message-ID: On Wed, Sep 18, 2019 at 2:33 PM Edouard Goudenhoofdt wrote: > This is clearly a duplicate of your previous post (How can I search the > mailing list?). > Yeah, good question .... that's kind of problematic at the moment. Eventually we'll switch to Mailman 3 which has a sane search interface. Sorry, we don't have a better answer than using Google et al. right now. I think this is a basic task for many geoscientists. > I did not make the effort to understand Simon Clift smarter approach. > But I already implemented a quick solution using a switch for the two use > cases. > May I submit a pull request? > It's now come up a couple of times, so clearly there's interest. Yes, opening a PR would be very helpful. Cheers, Ralf > > > On Wed, Sep 18, 2019 at 12:47 PM Dieter Werthm?ller < > Dieter at werthmuller.org> wrote: > >> Edouard, >> >> I think this thread touched on that a bit: >> https://mail.python.org/pipermail/scipy-dev/2019-May/023540.html >> >> So Simon Clift might have already some functional code for that which >> could serve as a starting point. >> >> I personally would be certainly interested in the functionality. >> >> Dieter >> >> >> On 18/09/2019 11:50, Edouard Goudenhoofdt wrote: >> > Dear scipy developers, >> > >> > One could use /scipy.interpolate.RegularGridInterpolator/ at different >> > times with different values but the same source grid and target points. >> > Currently this would recompute the indices each time the function is >> called. >> > I would like to add the possibility to provide the target points at >> > initialisation and the values when calling the object. >> > >> > Best regards, >> > >> > Edouard Goudenhoofdt >> > // >> > >> > _______________________________________________ >> > SciPy-Dev mailing list >> > SciPy-Dev at python.org >> > https://mail.python.org/mailman/listinfo/scipy-dev >> > >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Sep 18 09:02:17 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 18 Sep 2019 15:02:17 +0200 Subject: [SciPy-Dev] improvement to binned statistic In-Reply-To: References: Message-ID: Hi Edouard, On Wed, Sep 18, 2019 at 11:29 AM Edouard Goudenhoofdt wrote: > Dear scipy developers, > > One could use scipy.stats.binned_statistic_dd for the same sample points > but for values available at different times. > Currently this involves the computation of the bin numbers every time the > function is called. > Therefore I would like to add an optional argument "binnumbers" to skip > this step when calling the function again. > That seems sensible. Could you check that creating the bin numbers really takes the majority of the time? There's also a fair amount of input validation that shouldn't be skipped even when a new `binnumbers` is passed in. If that is the case, sending a PR with a benchmark would be very welcome. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From wham at live.unc.edu Wed Sep 18 09:19:12 2019 From: wham at live.unc.edu (Hamilton, Wesley) Date: Wed, 18 Sep 2019 13:19:12 +0000 Subject: [SciPy-Dev] Adding alpha complexes/filtrations to scipy.spatial? Message-ID: Hi all, I have some code that computes alpha complexes of a point cloud , and I think it would be a good contribution to the scipy.spatial library. I've looked around and there aren't any other implementations of an alpha complex construction (besides a short tutorial from PyPlot). My question is: would this contribution be welcome? I need to formalize the documentation, and add the construction for weighted alpha complexes, but otherwise everything's there. Best, Wesley -------------- next part -------------- An HTML attachment was scrubbed... URL: From ssclift at gmail.com Wed Sep 18 09:44:48 2019 From: ssclift at gmail.com (Simon S. Clift) Date: Wed, 18 Sep 2019 09:44:48 -0400 Subject: [SciPy-Dev] Improvement to regular grid interpolation In-Reply-To: References: Message-ID: Hi Edouard, I'll keep an eye out for your pull request, or perhaps you could send me a note on your use case to my gmail address. My approach just consists of saving the interpolation weights in a sparse matrix then doing a sparse multiply for subsequent interpolations. Weights can be either multi-linear or using barycentric co-ordinates on regular grids; it's an arbitrary choice unless you have a simplicial grid layout you prefer. Cheers -- Simon On Wed, Sep 18, 2019 at 8:50 AM Ralf Gommers wrote: > > > On Wed, Sep 18, 2019 at 2:33 PM Edouard Goudenhoofdt > wrote: > >> This is clearly a duplicate of your previous post (How can I search the >> mailing list?). >> > > Yeah, good question .... that's kind of problematic at the moment. > Eventually we'll switch to Mailman 3 which has a sane search interface. > Sorry, we don't have a better answer than using Google et al. right now. > > I think this is a basic task for many geoscientists. >> I did not make the effort to understand Simon Clift smarter approach. >> But I already implemented a quick solution using a switch for the two use >> cases. >> May I submit a pull request? >> > > It's now come up a couple of times, so clearly there's interest. Yes, > opening a PR would be very helpful. > > Cheers, > Ralf > > > >> >> >> On Wed, Sep 18, 2019 at 12:47 PM Dieter Werthm?ller < >> Dieter at werthmuller.org> wrote: >> >>> Edouard, >>> >>> I think this thread touched on that a bit: >>> https://mail.python.org/pipermail/scipy-dev/2019-May/023540.html >>> >>> So Simon Clift might have already some functional code for that which >>> could serve as a starting point. >>> >>> I personally would be certainly interested in the functionality. >>> >>> Dieter >>> >>> >>> On 18/09/2019 11:50, Edouard Goudenhoofdt wrote: >>> > Dear scipy developers, >>> > >>> > One could use /scipy.interpolate.RegularGridInterpolator/ at different >>> > times with different values but the same source grid and target points. >>> > Currently this would recompute the indices each time the function is >>> called. >>> > I would like to add the possibility to provide the target points at >>> > initialisation and the values when calling the object. >>> > >>> > Best regards, >>> > >>> > Edouard Goudenhoofdt >>> > // >>> > >>> > _______________________________________________ >>> > SciPy-Dev mailing list >>> > SciPy-Dev at python.org >>> > https://mail.python.org/mailman/listinfo/scipy-dev >>> > >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -- 1129 Ibbetson Lane Mississauga, Ontario L5C 1K9 Canada -------------- next part -------------- An HTML attachment was scrubbed... URL: From josh.craig.wilson at gmail.com Wed Sep 18 11:00:28 2019 From: josh.craig.wilson at gmail.com (Joshua Wilson) Date: Wed, 18 Sep 2019 08:00:28 -0700 Subject: [SciPy-Dev] Documentation: user survey In-Reply-To: References: Message-ID: One small thing which you can ignore if desired: in the ?How do you browse the documentation? question it might make sense to split the ?online HTML documentation option? into mobile and non-mobile variants. > On Sep 18, 2019, at 01:57, Maja Gwozdz wrote: > > Hi Ralf, > > thanks for your comments, I've adjusted the survey accordingly. > > I've decided to phrase the 'experience question' in terms of levels, i.e., beginner, intermediate, etc., due to the fact that 'years of experience' is probably a more difficult question to answer (at least for me). > > As for the linear scale, I now modified it and made it a standard 5-point Likert scale. The reason I initially opted for a 3-point scale was to avoid centre tendency bias that might crop up in its 5-point counterpart. But I don't think it's a big deal. You're right, a 5-point scale is more likely to give us fewer neutral responses, which is good for our purposes. > > I've also discarded the uppercase response options because they looked ugly, stylistically speaking. In order to facilitate a quicker response, I rearranged the response options in 'what features could be improved/added' alphabetically. > > I've also included the other questions you proposed and those kindly suggested by one other person on the mailing list. > > If there are no objections / further comments from the Community, I'll start publishing the user survey in the appropriate places (say, in two days). > > One more important question: should we limit the number of responses to one? This would require signing in, which is less than optimal. Do we have any legitimate concerns about getting spam? > > You can see the new version here: https://docs.google.com/forms/d/e/1FAIpQLSeBAO0UFKDZyKpg2XzRslsLJVHU61ugjc18-2PVEabTQg2_6g/viewform?usp=sf_link. > > I also took the liberty of including a header with a scientific flavour (no copyright infringement here, it's taken from the generic Google headers). > > All the best, > Maja > >> On Wed, Sep 18, 2019 at 7:21 AM Ralf Gommers wrote: >> HI Maja, >> >> >>> On Mon, Sep 16, 2019 at 12:14 PM Maja Gwozdz wrote: >>> Hi everyone, >>> >>> I'm currently working on improving the SciPy documentation during the Google Season of Docs. One of my projects is about a user survey. The basic idea is to get (documentation) feedback from SciPy users and try to improve our docs accordingly. >>> >>> It'd be great if you could take a look at the survey form and say if you have any comments that could help me. You can access it here: https://docs.google.com/forms/d/e/1FAIpQLSeBAO0UFKDZyKpg2XzRslsLJVHU61ugjc18-2PVEabTQg2_6g/viewform?usp=sf_link >> >> This looks like a very good start. A few thoughts: >> >> In the description at the top of the survey, perhaps add a link to http://scipy.github.io/devdocs/? >> >> It would be good to get some info on the users answering the questions. E.g. beginner/imtermediate/advanced or years of experience. That way you can analyze the other answers better, for example if only 10% of respondents are beginners and they all say that tutorials need improving, that's 100% of beginners that need this. >> >> I'm not a survey design expert, but I think it's more common to have 5 options for the "are you satisfied" or "is this clear" type of questions. With 3, you may get a lot of "neutral" responses. >> >> To "which doc features need to be improved/added" you could add "Ease of navigation". >> >> A question I would find interesting is "how do you view the docs (check all that apply)". Possible answers: online html docs, pdf, view docstring in terminal, Help feature in an IDE, Other. >> >> >>> >>> Also, I'm compiling a list of websites where I could deploy the survey to get a fair amount of responses. If you know any such websites, please let me know. >> >> When you say deploy I'm thinking SurveyMonkey or the like - you need just one of those, the Google Form you have may be perfectly fine. Response rates should be mostly determined by where we publicize the request for people to participate (or maybe that's what you meant). To mind come: mailing lists (SciPy and other projects, PyData, NumFOCUS), Twitter, post it on scipy.org, the scipy IRC and numpy Gitter channels, and a blog post that shows up on Planet SciPy and Planet Python. >> >> Cheers, >> Ralf >> >> >>> >>> Your help is much appreciated! >>> >>> All the best, >>> Maja Gwozdz >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyler.je.reddy at gmail.com Wed Sep 18 15:53:25 2019 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Wed, 18 Sep 2019 13:53:25 -0600 Subject: [SciPy-Dev] Adding alpha complexes/filtrations to scipy.spatial? In-Reply-To: References: Message-ID: Is this similar to "alpha shape:" https://en.wikipedia.org/wiki/Alpha_shape I know I've been wanting to have that in, but not sure if the community would agree. Probably some design decisions to think about, but a useful alternative to convex hull seems like an interesting idea for some applications. On Wed, 18 Sep 2019 at 07:19, Hamilton, Wesley wrote: > Hi all, > > I have some code that computes alpha complexes of a point cloud , and I > think it would be a good contribution to the scipy.spatial library. I've > looked around and there aren't any other implementations of an alpha > complex construction (besides a short tutorial from PyPlot). > > My question is: would this contribution be welcome? I need to formalize > the documentation, and add the construction for weighted alpha complexes, > but otherwise everything's there. > > Best, > Wesley > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlucas7 at vt.edu Wed Sep 18 14:36:50 2019 From: rlucas7 at vt.edu (rlucas7 at vt.edu) Date: Wed, 18 Sep 2019 14:36:50 -0400 Subject: [SciPy-Dev] SciPy-Dev Digest, Vol 191, Issue 15 In-Reply-To: References: Message-ID: <4F3FDCDF-6339-4157-8A26-645D5479189C@vt.edu> > On Sep 18, 2019, at 9:45 AM, scipy-dev-request at python.org wrote: > > Send SciPy-Dev mailing list submissions to > scipy-dev at python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scipy-dev > or, via email, send a message with subject or body 'help' to > scipy-dev-request at python.org > > You can reach the person managing the list at > scipy-dev-owner at python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of SciPy-Dev digest..." > > > Today's Topics: > > 1. Re: improvement to binned statistic (Ralf Gommers) > 2. Adding alpha complexes/filtrations to scipy.spatial? > (Hamilton, Wesley) > 3. Re: Improvement to regular grid interpolation (Simon S. Clift) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 18 Sep 2019 15:02:17 +0200 > From: Ralf Gommers > To: SciPy Developers List > Subject: Re: [SciPy-Dev] improvement to binned statistic > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > Hi Edouard, > > > On Wed, Sep 18, 2019 at 11:29 AM Edouard Goudenhoofdt > wrote: > >> Dear scipy developers, >> >> One could use scipy.stats.binned_statistic_dd for the same sample points >> but for values available at different times. >> Currently this involves the computation of the bin numbers every time the >> function is called. >> Therefore I would like to add an optional argument "binnumbers" to skip >> this step when calling the function again. >> > > That seems sensible. Could you check that creating the bin numbers really > takes the majority of the time? There's also a fair amount of input > validation that shouldn't be skipped even when a new `binnumbers` is passed > in. If that is the case, sending a PR with a benchmark would be very > welcome. > > Cheers, > Ralf IIUC Edouard what you?d like to do is take input data, run binned_statistic_dd() and then do the same thing with the bin edges calculated from this first call either on a new input dataset or on the same data(perhaps calculating on a new statistic?). AFAIK the binned_statistic_dd() function isn?t able to take binedges as an argument. If you want multiple stats for the same data I think you can achieve that via a custom callable() that returns multiple statistics rather than a single scalar, but I haven?t done this so you should confirm that the approach would work fine. If you want to take that up I?m happy to review the PR. If not, and this is something others agree is useful and should be implemented, it seems reasonable to do. I can implement if you don?t have time or are otherwise unable to open a PR. Let me know either way. -Lucas Roberts From ndbecker2 at gmail.com Wed Sep 18 17:47:54 2019 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 18 Sep 2019 17:47:54 -0400 Subject: [SciPy-Dev] Adding alpha complexes/filtrations to scipy.spatial? In-Reply-To: References: Message-ID: I have wanted this as well, hopefully with some reasonable computational complexity On Wed, Sep 18, 2019, 3:54 PM Tyler Reddy wrote: > Is this similar to "alpha shape:" > https://en.wikipedia.org/wiki/Alpha_shape > > I know I've been wanting to have that in, but not sure if the community > would agree. Probably some design decisions to think about, but a useful > alternative to convex hull seems like an interesting idea for some > applications. > > On Wed, 18 Sep 2019 at 07:19, Hamilton, Wesley wrote: > >> Hi all, >> >> I have some code that computes alpha complexes of a point cloud , and I >> think it would be a good contribution to the scipy.spatial library. I've >> looked around and there aren't any other implementations of an alpha >> complex construction (besides a short tutorial from PyPlot). >> >> My question is: would this contribution be welcome? I need to formalize >> the documentation, and add the construction for weighted alpha complexes, >> but otherwise everything's there. >> >> Best, >> Wesley >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Wed Sep 18 18:37:05 2019 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Wed, 18 Sep 2019 15:37:05 -0700 Subject: [SciPy-Dev] Adding alpha complexes/filtrations to scipy.spatial? In-Reply-To: References: Message-ID: <84b94482-ce82-4014-a018-90c346cc1285@www.fastmail.com> On Wed, Sep 18, 2019, at 12:53, Tyler Reddy wrote: > Is this similar to "alpha shape:" https://en.wikipedia.org/wiki/Alpha_shape > > I know I've been wanting to have that in, but not sure if the community would agree. Probably some design decisions to think about, but a useful alternative to convex hull seems like an interesting idea for some applications. We would find alpha shapes useful for scikit-image too. St?fan -------------- next part -------------- An HTML attachment was scrubbed... URL: From wham at live.unc.edu Wed Sep 18 19:05:28 2019 From: wham at live.unc.edu (Hamilton, Wesley) Date: Wed, 18 Sep 2019 23:05:28 +0000 Subject: [SciPy-Dev] Adding alpha complexes/filtrations to scipy.spatial? In-Reply-To: References: Message-ID: These are the same as alpha shapes: alpha complexes are alpha shapes, and an alpha filtration is a collection of alpha shapes corresponding to varying alpha. The situation I've had to use these for is in studying protein structure (which is one of the original applications for this stuff), though I'm sure there are other uses (like for scikit image?). In practice, these constructions can work in any dimension, but geometrically I think the intuition changes drastically above 3 dimensions. In terms of computational complexity, I think the worst bit is in computing determinants for the in-circle radius and center, but again for small dimensions it isn't terrible. I'd also be happy to look into more optimal (if applicable) implementations for this construction, including reaching out to, say, Edelsbrunner and other experts. I can also share the code somewhere, or if anyone wants to discuss via direct email, then that would also be great. I admit I'm relatively new to software development (and git, etc.), but I'm hoping this code can be implemented and useful to others! Best, Wesley -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sat Sep 21 04:56:03 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 21 Sep 2019 10:56:03 +0200 Subject: [SciPy-Dev] Documentation: user survey In-Reply-To: References: Message-ID: On Wed, Sep 18, 2019 at 10:57 AM Maja Gwozdz wrote: > Hi Ralf, > > thanks for your comments, I've adjusted the survey accordingly. > > I've decided to phrase the 'experience question' in terms of levels, i.e., > beginner, intermediate, etc., due to the fact that 'years of experience' is > probably a more difficult question to answer (at least for me). > > As for the linear scale, I now modified it and made it a standard 5-point > Likert scale. The reason I initially opted for a 3-point scale was to avoid > centre tendency bias that might crop up in its 5-point counterpart. But I > don't think it's a big deal. You're right, a 5-point scale is more likely > to give us fewer neutral responses, which is good for our purposes. > > I've also discarded the uppercase response options because they looked > ugly, stylistically speaking. In order to facilitate a quicker response, I > rearranged the response options in 'what features could be improved/added' > alphabetically. > > I've also included the other questions you proposed and those kindly > suggested by one other person on the mailing list. > > If there are no objections / further comments from the Community, I'll > start publishing the user survey in the appropriate places (say, in two > days). > This sounds great. Sorry for the delayed reply, please go ahead I'd say. > One more important question: should we limit the number of responses to > one? This would require signing in, which is less than optimal. Do we have > any legitimate concerns about getting spam? > I'm not really concerned about that, we've never had a serious issue with that. Not having to sign in seems preferable. > > You can see the new version here: > https://docs.google.com/forms/d/e/1FAIpQLSeBAO0UFKDZyKpg2XzRslsLJVHU61ugjc18-2PVEabTQg2_6g/viewform?usp=sf_link > . > This looks great to me, good to go! > I also took the liberty of including a header with a scientific flavour > (no copyright infringement here, it's taken from the generic Google > headers). > :) Cheers, Ralf > All the best, > Maja > > On Wed, Sep 18, 2019 at 7:21 AM Ralf Gommers > wrote: > >> HI Maja, >> >> >> On Mon, Sep 16, 2019 at 12:14 PM Maja Gwozdz >> wrote: >> >>> Hi everyone, >>> >>> I'm currently working on improving the SciPy documentation during the >>> Google Season of Docs. One of my projects is about a user survey. The basic >>> idea is to get (documentation) feedback from SciPy users and try to improve >>> our docs accordingly. >>> >>> It'd be great if you could take a look at the survey form and say if you >>> have any comments that could help me. You can access it here: >>> https://docs.google.com/forms/d/e/1FAIpQLSeBAO0UFKDZyKpg2XzRslsLJVHU61ugjc18-2PVEabTQg2_6g/viewform?usp=sf_link >>> >> >> This looks like a very good start. A few thoughts: >> >> In the description at the top of the survey, perhaps add a link to >> http://scipy.github.io/devdocs/? >> >> It would be good to get some info on the users answering the questions. >> E.g. beginner/imtermediate/advanced or years of experience. That way you >> can analyze the other answers better, for example if only 10% of >> respondents are beginners and they all say that tutorials need improving, >> that's 100% of beginners that need this. >> >> I'm not a survey design expert, but I think it's more common to have 5 >> options for the "are you satisfied" or "is this clear" type of questions. >> With 3, you may get a lot of "neutral" responses. >> >> To "which doc features need to be improved/added" you could add "Ease of >> navigation". >> >> A question I would find interesting is "how do you view the docs (check >> all that apply)". Possible answers: online html docs, pdf, view docstring >> in terminal, Help feature in an IDE, Other. >> >> >> >>> Also, I'm compiling a list of websites where I could deploy the survey >>> to get a fair amount of responses. If you know any such websites, please >>> let me know. >>> >> >> When you say deploy I'm thinking SurveyMonkey or the like - you need just >> one of those, the Google Form you have may be perfectly fine. Response >> rates should be mostly determined by where we publicize the request for >> people to participate (or maybe that's what you meant). To mind come: >> mailing lists (SciPy and other projects, PyData, NumFOCUS), Twitter, post >> it on scipy.org, the scipy IRC and numpy Gitter channels, and a blog >> post that shows up on Planet SciPy and Planet Python. >> >> Cheers, >> Ralf >> >> >> >>> Your help is much appreciated! >>> >>> All the best, >>> Maja Gwozdz >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From egouden at gmail.com Mon Sep 23 05:12:02 2019 From: egouden at gmail.com (Edouard Goudenhoofdt) Date: Mon, 23 Sep 2019 11:12:02 +0200 Subject: [SciPy-Dev] SciPy-Dev Digest, Vol 191, Issue 15 In-Reply-To: <4F3FDCDF-6339-4157-8A26-645D5479189C@vt.edu> References: <4F3FDCDF-6339-4157-8A26-645D5479189C@vt.edu> Message-ID: Dear Lucas, I want the ability to reuse the bin numbers for a new input dataset. Indeed one should already be able to compute several statistics at once (and also for several datasets available at the same time). I have a PR ready to submit. Thank you for proposing to review it. Best regards, Edouard On Wed, Sep 18, 2019 at 9:59 PM wrote: > > > On Sep 18, 2019, at 9:45 AM, scipy-dev-request at python.org wrote: > > > > Send SciPy-Dev mailing list submissions to > > scipy-dev at python.org > > > > To subscribe or unsubscribe via the World Wide Web, visit > > https://mail.python.org/mailman/listinfo/scipy-dev > > or, via email, send a message with subject or body 'help' to > > scipy-dev-request at python.org > > > > You can reach the person managing the list at > > scipy-dev-owner at python.org > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of SciPy-Dev digest..." > > > > > > Today's Topics: > > > > 1. Re: improvement to binned statistic (Ralf Gommers) > > 2. Adding alpha complexes/filtrations to scipy.spatial? > > (Hamilton, Wesley) > > 3. Re: Improvement to regular grid interpolation (Simon S. Clift) > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Wed, 18 Sep 2019 15:02:17 +0200 > > From: Ralf Gommers > > To: SciPy Developers List > > Subject: Re: [SciPy-Dev] improvement to binned statistic > > Message-ID: > > > > Content-Type: text/plain; charset="utf-8" > > > > Hi Edouard, > > > > > > On Wed, Sep 18, 2019 at 11:29 AM Edouard Goudenhoofdt > > > wrote: > > > >> Dear scipy developers, > >> > >> One could use scipy.stats.binned_statistic_dd for the same sample points > >> but for values available at different times. > >> Currently this involves the computation of the bin numbers every time > the > >> function is called. > >> Therefore I would like to add an optional argument "binnumbers" to skip > >> this step when calling the function again. > >> > > > > That seems sensible. Could you check that creating the bin numbers really > > takes the majority of the time? There's also a fair amount of input > > validation that shouldn't be skipped even when a new `binnumbers` is > passed > > in. If that is the case, sending a PR with a benchmark would be very > > welcome. > > > > Cheers, > > Ralf > > IIUC Edouard what you?d like to do is take input data, run > binned_statistic_dd() and then do the same thing with the bin edges > calculated from this first call either on a new input dataset or on the > same data(perhaps calculating on a new statistic?). > > AFAIK the binned_statistic_dd() function isn?t able to take binedges as an > argument. If you want multiple stats for the same data I think you can > achieve that via a custom callable() that returns multiple statistics > rather than a single scalar, but I haven?t done this so you should confirm > that the approach would work fine. > > If you want to take that up I?m happy to review the PR. > > If not, and this is something others agree is useful and should be > implemented, it seems reasonable to do. I can implement if you don?t have > time or are otherwise unable to open a PR. > > Let me know either way. > > -Lucas Roberts > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From egouden at gmail.com Mon Sep 23 05:29:10 2019 From: egouden at gmail.com (Edouard Goudenhoofdt) Date: Mon, 23 Sep 2019 11:29:10 +0200 Subject: [SciPy-Dev] Improvement to regular grid interpolation In-Reply-To: References: Message-ID: Dear Simon, Thank you for your comment. For the sake of time I just kept the current implementation while saving weights. I guess it could later be used as a reference to benchmark your approach. Best regards, Edouard On Wed, Sep 18, 2019 at 3:45 PM Simon S. Clift wrote: > Hi Edouard, > > I'll keep an eye out for your pull request, or perhaps you could send me a > note on your use case to my gmail address. My approach just consists of > saving the interpolation weights in a sparse matrix then doing a sparse > multiply for subsequent interpolations. Weights can be either multi-linear > or using barycentric co-ordinates on regular grids; it's an arbitrary > choice unless you have a simplicial grid layout you prefer. > > Cheers > -- Simon > > > On Wed, Sep 18, 2019 at 8:50 AM Ralf Gommers > wrote: > >> >> >> On Wed, Sep 18, 2019 at 2:33 PM Edouard Goudenhoofdt >> wrote: >> >>> This is clearly a duplicate of your previous post (How can I search the >>> mailing list?). >>> >> >> Yeah, good question .... that's kind of problematic at the moment. >> Eventually we'll switch to Mailman 3 which has a sane search interface. >> Sorry, we don't have a better answer than using Google et al. right now. >> >> I think this is a basic task for many geoscientists. >>> I did not make the effort to understand Simon Clift smarter approach. >>> But I already implemented a quick solution using a switch for the two >>> use cases. >>> May I submit a pull request? >>> >> >> It's now come up a couple of times, so clearly there's interest. Yes, >> opening a PR would be very helpful. >> >> Cheers, >> Ralf >> >> >> >>> >>> >>> On Wed, Sep 18, 2019 at 12:47 PM Dieter Werthm?ller < >>> Dieter at werthmuller.org> wrote: >>> >>>> Edouard, >>>> >>>> I think this thread touched on that a bit: >>>> https://mail.python.org/pipermail/scipy-dev/2019-May/023540.html >>>> >>>> So Simon Clift might have already some functional code for that which >>>> could serve as a starting point. >>>> >>>> I personally would be certainly interested in the functionality. >>>> >>>> Dieter >>>> >>>> >>>> On 18/09/2019 11:50, Edouard Goudenhoofdt wrote: >>>> > Dear scipy developers, >>>> > >>>> > One could use /scipy.interpolate.RegularGridInterpolator/ at different >>>> > times with different values but the same source grid and target >>>> points. >>>> > Currently this would recompute the indices each time the function is >>>> called. >>>> > I would like to add the possibility to provide the target points at >>>> > initialisation and the values when calling the object. >>>> > >>>> > Best regards, >>>> > >>>> > Edouard Goudenhoofdt >>>> > // >>>> > >>>> > _______________________________________________ >>>> > SciPy-Dev mailing list >>>> > SciPy-Dev at python.org >>>> > https://mail.python.org/mailman/listinfo/scipy-dev >>>> > >>>> _______________________________________________ >>>> SciPy-Dev mailing list >>>> SciPy-Dev at python.org >>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > > > -- > 1129 Ibbetson Lane > Mississauga, Ontario > L5C 1K9 Canada > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From egouden at gmail.com Mon Sep 23 07:41:57 2019 From: egouden at gmail.com (Edouard Goudenhoofdt) Date: Mon, 23 Sep 2019 13:41:57 +0200 Subject: [SciPy-Dev] Improvement to regular grid interpolation In-Reply-To: References: Message-ID: The pull request has been made. On Mon, Sep 23, 2019 at 11:29 AM Edouard Goudenhoofdt wrote: > Dear Simon, > > Thank you for your comment. > For the sake of time I just kept the current implementation while saving > weights. > I guess it could later be used as a reference to benchmark your approach. > > Best regards, > > Edouard > > On Wed, Sep 18, 2019 at 3:45 PM Simon S. Clift wrote: > >> Hi Edouard, >> >> I'll keep an eye out for your pull request, or perhaps you could send me >> a note on your use case to my gmail address. My approach just consists of >> saving the interpolation weights in a sparse matrix then doing a sparse >> multiply for subsequent interpolations. Weights can be either multi-linear >> or using barycentric co-ordinates on regular grids; it's an arbitrary >> choice unless you have a simplicial grid layout you prefer. >> >> Cheers >> -- Simon >> >> >> On Wed, Sep 18, 2019 at 8:50 AM Ralf Gommers >> wrote: >> >>> >>> >>> On Wed, Sep 18, 2019 at 2:33 PM Edouard Goudenhoofdt >>> wrote: >>> >>>> This is clearly a duplicate of your previous post (How can I search the >>>> mailing list?). >>>> >>> >>> Yeah, good question .... that's kind of problematic at the moment. >>> Eventually we'll switch to Mailman 3 which has a sane search interface. >>> Sorry, we don't have a better answer than using Google et al. right now. >>> >>> I think this is a basic task for many geoscientists. >>>> I did not make the effort to understand Simon Clift smarter approach. >>>> But I already implemented a quick solution using a switch for the two >>>> use cases. >>>> May I submit a pull request? >>>> >>> >>> It's now come up a couple of times, so clearly there's interest. Yes, >>> opening a PR would be very helpful. >>> >>> Cheers, >>> Ralf >>> >>> >>> >>>> >>>> >>>> On Wed, Sep 18, 2019 at 12:47 PM Dieter Werthm?ller < >>>> Dieter at werthmuller.org> wrote: >>>> >>>>> Edouard, >>>>> >>>>> I think this thread touched on that a bit: >>>>> https://mail.python.org/pipermail/scipy-dev/2019-May/023540.html >>>>> >>>>> So Simon Clift might have already some functional code for that which >>>>> could serve as a starting point. >>>>> >>>>> I personally would be certainly interested in the functionality. >>>>> >>>>> Dieter >>>>> >>>>> >>>>> On 18/09/2019 11:50, Edouard Goudenhoofdt wrote: >>>>> > Dear scipy developers, >>>>> > >>>>> > One could use /scipy.interpolate.RegularGridInterpolator/ at >>>>> different >>>>> > times with different values but the same source grid and target >>>>> points. >>>>> > Currently this would recompute the indices each time the function is >>>>> called. >>>>> > I would like to add the possibility to provide the target points at >>>>> > initialisation and the values when calling the object. >>>>> > >>>>> > Best regards, >>>>> > >>>>> > Edouard Goudenhoofdt >>>>> > // >>>>> > >>>>> > _______________________________________________ >>>>> > SciPy-Dev mailing list >>>>> > SciPy-Dev at python.org >>>>> > https://mail.python.org/mailman/listinfo/scipy-dev >>>>> > >>>>> _______________________________________________ >>>>> SciPy-Dev mailing list >>>>> SciPy-Dev at python.org >>>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>>> >>>> _______________________________________________ >>>> SciPy-Dev mailing list >>>> SciPy-Dev at python.org >>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> >> >> -- >> 1129 Ibbetson Lane >> Mississauga, Ontario >> L5C 1K9 Canada >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Sep 24 05:59:03 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 24 Sep 2019 11:59:03 +0200 Subject: [SciPy-Dev] changing scipy.org to just be about SciPy the library Message-ID: Hi all, The https://scipy.org/ website has always been about SciPy the ecosystem not SciPy the project/library, and it's always resulted in confusion. I propose we finally change that. From my review of a recent PR [1] to the scipy.org website repo: """ Perhaps the confusing thing here is that https://scipy.org is about "SciPy the ecosystem" rather than "SciPy the library". That's what we would really like to fix, then this becomes a lot more straightforward. So far I had in mind to have a replacement "ecosystem site" before changing scipy.org, however we could go without that. There really isn't that much content on the current site that we'd have to remove, and some of the things that are there now (like SciPy Stack) have become irrelevant. We could easily say that `scipy.org` will be about SciPy the library, while keeping a page like https://scipy.org/about.html that explains the ecosystem context. """ Besides some news about and doc hosting for NumPy releases, there's not much content about other projects, and those other projects don't seem to care. The "ecosystem" thing really has had its time. "SciPy ecosystem" isn't even a preferred term anymore I think, the "scientific" is seen as too narrow by a lot of people. PyData or NumPy would be better qualifiers (or some new umbrella term). We may not be able to do all the work straight away, but I want to get approval for the decision to make scipy.org about the SciPy project now. Thoughts? Cheers, Ralf [1] https://github.com/scipy/scipy.org/pull/309 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jh at physics.ucf.edu Tue Sep 24 09:14:43 2019 From: jh at physics.ucf.edu (Joe Harrington) Date: Tue, 24 Sep 2019 09:14:43 -0400 Subject: [SciPy-Dev] changing scipy.org to just be about SciPy the library In-Reply-To: References: Message-ID: I'm not against it, but the stuff at the top of the page about related libraries, docs, getting started, etc. should not disappear, though it might move to another site, or be at the head of all the related sites.? Since NumPy is the base package, and is now better known than SciPy, perhaps that material can move there.? I think it would be a mistake to have one page per package and not to let new users know what the other packages are, how they fit together, how to get docs that describe using them together, how to connect to the community, etc. Also, there should be no confusion on a well written and well designed site.? The text can concisely communicate that "SciPy" refers both to a specific package and to the community of packages, other resources, and people.? Whether it's "SciPy" or not, we need a term for this ecosystem.? It isn't the worst thing in the world to change the term, especially as use has spread from science to pretty much everything numerical.? We've used "Python Numerical Core" in a recent funding proposal and community white paper.? "Numerical Python" is shorter, though the term has some history from the early days.? I don't (yet!) have a strong opinion on what the term should be, as long as it is relatively short, descriptive, and obvious. It isn't even clear to me that the term itself needs its own web page, though of course that's a possibility.? I think most people will land in our universe either by googling the term "numpy" or something like "python math library" (which might take them to the wrong place at first), or going to numpy.org, following someone's advice or a link.? Wherever users are likely to land should have some intro material about the ecosystem at the top, and not just dive into the specifics of that module, as numpy.org does, currently. --jh-- On 9/24/19 5:59 AM, Ralf Gommers wrote: > Hi all, > > The https://scipy.org/ website has always been about SciPy the > ecosystem not SciPy the project/library, and it's always resulted in > confusion. I propose we finally change that.? From my review of a > recent PR [1] to the scipy.org website repo: > > """ > Perhaps the confusing thing here is that https://scipy.org is about > "SciPy the ecosystem" rather than "SciPy the library". That's what we > would really like to fix, then this becomes a lot more > straightforward. So far I had in mind to have a replacement "ecosystem > site" before changing scipy.org , however we could > go without that. There really isn't that much content on the current > site that we'd have to remove, and some of the things that are there > now (like SciPy Stack) have become irrelevant. We could easily say > that `scipy.org ` will be about SciPy the library, > while keeping a page like https://scipy.org/about.html that explains > the ecosystem context. > """ > > Besides some news about and doc hosting for NumPy releases, there's > not much content about other projects, and those other projects don't > seem to care. The "ecosystem" thing really has had its time. "SciPy > ecosystem" isn't even a preferred term anymore I think, the > "scientific" is seen as too narrow by a lot of people. PyData or NumPy > would be better qualifiers (or some new umbrella term). > > We may not be able to do all the work straight away, but I want to get > approval for the decision to make scipy.org about > the SciPy project now. > > Thoughts? > > Cheers, > Ralf > > [1] https://github.com/scipy/scipy.org/pull/309 > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From cimrman3 at ntc.zcu.cz Tue Sep 24 10:23:07 2019 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Tue, 24 Sep 2019 16:23:07 +0200 Subject: [SciPy-Dev] ANN: SfePy 2019.3 Message-ID: I am pleased to announce release 2019.3 of SfePy. Description ----------- SfePy (simple finite elements in Python) is a software for solving systems of coupled partial differential equations by the finite element method or by the isogeometric analysis (limited support). It is distributed under the new BSD license. Home page: http://sfepy.org Mailing list: https://mail.python.org/mm3/mailman3/lists/sfepy.python.org/ Git (source) repository, issue tracker: https://github.com/sfepy/sfepy Highlights of this release -------------------------- - interface to eigenvalue problem solvers in SLEPc - new Python 3 enabled Timer class and other Python 3 compatibility fixes For full release notes see [1]. Cheers, Robert Cimrman [1] http://docs.sfepy.org/doc/release_notes.html#id1 --- Contributors to this release in alphabetical order: Robert Cimrman Vladimir Lukes From haberland at ucla.edu Tue Sep 24 17:31:11 2019 From: haberland at ucla.edu (Matt Haberland) Date: Tue, 24 Sep 2019 14:31:11 -0700 Subject: [SciPy-Dev] changing scipy.org to just be about SciPy the library In-Reply-To: References: Message-ID: Sounds like a good idea to me. I agree with the main points (not much content about the ecosystem as it is, and we can certainly link to an "ecosystem site" for backwards compatibility). On Tue, Sep 24, 2019 at 2:59 AM Ralf Gommers wrote: > Hi all, > > The https://scipy.org/ website has always been about SciPy the ecosystem > not SciPy the project/library, and it's always resulted in confusion. I > propose we finally change that. From my review of a recent PR [1] to the > scipy.org website repo: > > """ > Perhaps the confusing thing here is that https://scipy.org is about > "SciPy the ecosystem" rather than "SciPy the library". That's what we would > really like to fix, then this becomes a lot more straightforward. So far I > had in mind to have a replacement "ecosystem site" before changing > scipy.org, however we could go without that. There really isn't that much > content on the current site that we'd have to remove, and some of the > things that are there now (like SciPy Stack) have become irrelevant. We > could easily say that `scipy.org` will be about SciPy the library, while > keeping a page like https://scipy.org/about.html that explains the > ecosystem context. > """ > > Besides some news about and doc hosting for NumPy releases, there's not > much content about other projects, and those other projects don't seem to > care. The "ecosystem" thing really has had its time. "SciPy ecosystem" > isn't even a preferred term anymore I think, the "scientific" is seen as > too narrow by a lot of people. PyData or NumPy would be better qualifiers > (or some new umbrella term). > > We may not be able to do all the work straight away, but I want to get > approval for the decision to make scipy.org about the SciPy project now. > > Thoughts? > > Cheers, > Ralf > > [1] https://github.com/scipy/scipy.org/pull/309 > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -- Matt Haberland Assistant Adjunct Professor in the Program in Computing Department of Mathematics 6617A Math Sciences Building, UCLA -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Tue Sep 24 17:41:46 2019 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Tue, 24 Sep 2019 14:41:46 -0700 Subject: [SciPy-Dev] changing scipy.org to just be about SciPy the library In-Reply-To: References: Message-ID: On Tue, Sep 24, 2019, at 14:31, Matt Haberland wrote: > Sounds like a good idea to me. I agree with the main points (not much content about the ecosystem as it is, and we can certainly link to an "ecosystem site" for backwards compatibility). > > On Tue, Sep 24, 2019 at 2:59 AM Ralf Gommers wrote: >> Hi all, >> >> The https://scipy.org/ website has always been about SciPy the ecosystem not SciPy the project/library, and it's always resulted in confusion. I propose we finally change that. From my review of a recent PR [1] to the scipy.org website repo: >> I support making the website clearer. W.r.t. Joe's concern: if we spell out "scientific Python ecosystem" as we've been doing recently, and use "SciPy library", it's fairly easy to disambiguate. It may also be worth clarifying our position in/relative to "PyData", which thus far has stood mostly for industry applications of the same ecosystem. We also own https://scientific-python.org, even though that domain is not currently in use. Best regards, St?fan -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Thu Sep 26 21:50:50 2019 From: toddrjen at gmail.com (Todd) Date: Thu, 26 Sep 2019 21:50:50 -0400 Subject: [SciPy-Dev] PR for overlap-add convolution Message-ID: I have submitted a PR [1] for a new function, "oaconvolve", which is an implementation of the overlap-add algorithm for convolution. This algorithm is much faster than conventional FFT-based convolution when both inputs are large and one is much bigger than the other. I would be interested in getting your feedback. Any improvements on the algorithm would be greatly appreciated, and I am open to any suggestions on any aspect of the PR. My plans were previously discussed on the mailing list [2]. The original request for this feature dates back to 2009 [3]. More details below for those interested: The function supports all arguments fftconvolve supports, including working on an arbitrary subset of axes of multidimensional arrays. It will also automatically fall back on using fftconvolve in situations where using the overlap-add method doesn't apply (such as when the two inputs are the same shape). This implementation works in a vectorized manner. The input arrays are reshaped so the frequency-domain convolution of all overlapping blocks can be calculated in a single step rather than using a loop like most implementations I have seen. This means that it should work efficiently on any FFT backend that supports vectorization, unlike loops that would be slowed down by FFT backends with long plan creation times. Currently only pocketfft is supported by scipy, but there are plans to support additional FFT backends [4]. This approach should just work for any of them. pocketfft has very low plan creation time so this is less of an issue, but it also supports vectorizing multidimensional FFTs so this approach is still highly efficient for pocketfft. Some potential issues: We previously discussed whether this should be its own function or part of fftconvolve. The performance and memory characteristics of this function are significantly different than those of fftconvolve and would really be used in different situations, so the consensus seemed to be that this should be its own function. I would ultimately like "convolve" to be able to use oaconvolve when it is the best choice, but that is an issue for later. There is a "method" argument for choosing whether to automatically fall back on fftconvolve or raise an error. Being able to prevent falling back on fftconvolve is essentially for testing. We need to be able to make sure that oaconvolve is actually being tested and not the fftconvolve fallback. But this may not be the best way to do that. I have abstracted much of the shared functionality between oaconvolve and fftconvolve to several hidden functions, which should make it easier to add shared features to both. But if this is considered too complicated I can make the functions independent. The tests are slow. There are many potential corner cases with particular combinations of sizes so I have a test marked slow to check a large variety of sizes. This means it is disabled in some situations, but it does need to be run at least sometimes. This is only for one dimension though. Checking more this way would likely be prohibitively slow. There is a detailed docstring, but I didn't add anything to the tutorial. Should I? Are there any additional references anyone things I should add? Is the example okay? Is there another one someone thinks would be useful? I included a bugfix that this code depends on. Should I split it into another pull request or is it okay having them together? The bugfix has to be accepted before the new function can be accepted since oaconvolve will not work without it. [1] https://github.com/scipy/scipy/pull/10869 [2] https://mail.python.org/pipermail/scipy-dev/2019-June/023555.html [3] https://github.com/scipy/scipy/issues/1364 [4] https://mail.python.org/pipermail/scipy-dev/2019-May/023506.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Sep 26 22:43:13 2019 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 26 Sep 2019 19:43:13 -0700 Subject: [SciPy-Dev] PR for overlap-add convolution In-Reply-To: References: Message-ID: On Thu, Sep 26, 2019 at 6:51 PM Todd wrote: > I have submitted a PR [1] for a new function, "oaconvolve", which is an implementation of the overlap-add algorithm for convolution. This algorithm is much faster than conventional FFT-based > We previously discussed whether this should be its own function or part of fftconvolve. The performance and memory characteristics of this function are significantly different than those of fftconvolve and would really be used in different situations, so the consensus seemed to be that this should be its own function. I would ultimately like "convolve" to be able to use oaconvolve when it is the best choice, but that is an issue for later. > > There is a "method" argument for choosing whether to automatically fall back on fftconvolve or raise an error. Being able to prevent falling back on fftconvolve is essentially for testing. We need to be able to make sure that oaconvolve is actually being tested and not the fftconvolve fallback. But this may not be the best way to do that. It seems like there's some contradiction between the idea that it needs to be its own function so that users can control which algorithm they're using, but then it sometimes silently falls back to fftconvolve, so they can't ? and then you need an argument to disable the fallback because you do need to control the choice of algorithm. What would be the consequences of removing the fallback entirely? -n -- Nathaniel J. Smith -- https://vorpus.org From toddrjen at gmail.com Thu Sep 26 23:56:08 2019 From: toddrjen at gmail.com (Todd) Date: Thu, 26 Sep 2019 23:56:08 -0400 Subject: [SciPy-Dev] PR for overlap-add convolution In-Reply-To: References: Message-ID: On Thu, Sep 26, 2019 at 10:43 PM Nathaniel Smith wrote: > On Thu, Sep 26, 2019 at 6:51 PM Todd wrote: > > I have submitted a PR [1] for a new function, "oaconvolve", which is an > implementation of the overlap-add algorithm for convolution. This > algorithm is much faster than conventional FFT-based > We previously > discussed whether this should be its own function or part of fftconvolve. > The performance and memory characteristics of this function are > significantly different than those of fftconvolve and would really be used > in different situations, so the consensus seemed to be that this should be > its own function. I would ultimately like "convolve" to be able to use > oaconvolve when it is the best choice, but that is an issue for later. > > > > There is a "method" argument for choosing whether to automatically fall > back on fftconvolve or raise an error. Being able to prevent falling back > on fftconvolve is essentially for testing. We need to be able to make sure > that oaconvolve is actually being tested and not the fftconvolve fallback. > But this may not be the best way to do that. > > It seems like there's some contradiction between the idea that it > needs to be its own function so that users can control which algorithm > they're using, but then it sometimes silently falls back to > fftconvolve, so they can't ? and then you need an argument to disable > the fallback because you do need to control the choice of algorithm. > > What would be the consequences of removing the fallback entirely? > > -n > You would get exceptions in specific, very hard-to-predict combinations of shapes. Unless you have total control over the shapes of the inputs, the function couldn't be used safely on its own. You would need to handle such cases somehow. To be clear, the situations where it falls back on fftconvolve are the situations where oaconvolve reduces to fftconvolve. It would be possible to implement oaconvolve in these situations, and it would ultimately be the same algorithm as fftconvolve, but it would add unnecessary time and memory overhead compared to just using the fftconvolve fallback. I didn't see the point in doing a numerically identical algorithm slower, hence the fallback. Further, handling these situations in the oaconvolve function would slow down the oaconvolve function under all conditions, not just situations where oaconvolve reduces to fftconvolve. I need the argument, or something like it, because I want to make sure the tests don't hit situations where oaconvolve reduces to fftconvolve. Even if these situations where handled in oaconvolve itself rather than an fftconvolve fallback, I would still need to be able to raise an exception for these situations because I need to make sure the tests are actually testing the oaconvolve algorithm, not the fftconvolve one. The reason there are two functions are situations where fftconvolve and oaconvolve are substantially different in their characteristics. There is no fallback in such cases, even if fftconvolve is faster. Picking the fastest algorithm for a given set of inputs should be handled by the "convolve" function, but adding support for oaconvolve to convolve will be a later pull request. -------------- next part -------------- An HTML attachment was scrubbed... URL: From josh.craig.wilson at gmail.com Fri Sep 27 01:37:51 2019 From: josh.craig.wilson at gmail.com (Joshua Wilson) Date: Thu, 26 Sep 2019 22:37:51 -0700 Subject: [SciPy-Dev] PR for overlap-add convolution In-Reply-To: References: Message-ID: <461764E5-2997-45AD-9B94-CF826C649F4E@gmail.com> > On Sep 26, 2019, at 18:50, Todd wrote: > > Being able to prevent falling back on fftconvolve is essentially for testing. We need to be able to make sure that oaconvolve is actually being tested and not the fftconvolve fallback. But this may not be the best way to do that. If preventing the fallback is just for testing, one thing you might consider is mocking fftconvolve in the tests with something that raises as a side effect instead. This can be done with `unittest.mock.Mock` and will ensure that the tests fail if you hit the fallback route without affecting the actual functionality. From njs at pobox.com Fri Sep 27 02:17:01 2019 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 26 Sep 2019 23:17:01 -0700 Subject: [SciPy-Dev] PR for overlap-add convolution In-Reply-To: References: Message-ID: On Thu, Sep 26, 2019 at 8:56 PM Todd wrote: > To be clear, the situations where it falls back on fftconvolve are the situations where oaconvolve reduces to fftconvolve. I definitely wouldn't expose any argument to let users control this then. You could have an internal function that has that argument, write the fine-grained tests against that internal function, and then have a public wrapper that just calls the internal function. -n -- Nathaniel J. Smith -- https://vorpus.org From phillip.m.feldman at gmail.com Fri Sep 27 02:19:45 2019 From: phillip.m.feldman at gmail.com (Phillip Feldman) Date: Thu, 26 Sep 2019 23:19:45 -0700 Subject: [SciPy-Dev] suggestion re. scipy.spatial.transform.Rotation Message-ID: I've noticed that this class supports four of the five common representations of a rotation. The one that's missing is the rotation matrix. I can supply code that converts an arbitrary sequence of Euler rotations into a rotation matrix. It would also be nice to be able to go in the reverse direction, but I haven't yet figured that out. Phillip scipy.spatial.transform.Rotation *class *scipy.spatial.transform.Rotation(*quat*, *normalized=False*, *copy=True*)[source] Rotation in 3 dimensions. This class provides an interface to initialize from and represent rotations with: - Quaternions - Direction Cosine Matrices - Rotation Vectors - Euler angles -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.hilger at gmail.com Fri Sep 27 03:41:00 2019 From: thomas.hilger at gmail.com (Thomas Hilger) Date: Fri, 27 Sep 2019 09:41:00 +0200 Subject: [SciPy-Dev] suggestion: vector spherical harmonics Message-ID: The generalization of spherical harmonics to vector-fields are the vector spherical harmonics [ https://en.wikipedia.org/wiki/Vector_spherical_harmonics ] . I can provide simple code which relies on scipy's (scalar) spherical harmonics implementation and follows the same conventions to give vector spherical harmonics. Could this be a nice-thing-to-have for scipy? A couple of scientific communities (astronomy, geodesy) use them a lot. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ali.cetin at outlook.com Fri Sep 27 07:23:54 2019 From: ali.cetin at outlook.com (Ali Cetin) Date: Fri, 27 Sep 2019 11:23:54 +0000 Subject: [SciPy-Dev] suggestion re. scipy.spatial.transform.Rotation In-Reply-To: References: Message-ID: DCM (directional cosine matrix) is the rotation matrix. In order to go in the reversed direction, just supply the "inverse = True" when calling the apply method. (sorry for not using appropriate quote convention when replying. Answering from mobile from the beach ;)) Cheers, Ali From: Phillip Feldman Sent: Friday, September 27, 09:20 Subject: [SciPy-Dev] suggestion re. scipy.spatial.transform.Rotation To: SciPy Developers List I've noticed that this class supports four of the five common representations of a rotation. The one that's missing is the rotation matrix. I can supply code that converts an arbitrary sequence of Euler rotations into a rotation matrix. It would also be nice to be able to go in the reverse direction, but I haven't yet figured that out. Phillip scipy.spatial.transform.Rotation class scipy.spatial.transform.Rotation(quat, normalized=False, copy=True)[source] Rotation in 3 dimensions. This class provides an interface to initialize from and represent rotations with: Quaternions Direction Cosine Matrices Rotation Vectors Euler angles -------------- next part -------------- An HTML attachment was scrubbed... URL: From larson.eric.d at gmail.com Fri Sep 27 09:12:47 2019 From: larson.eric.d at gmail.com (Eric Larson) Date: Fri, 27 Sep 2019 09:12:47 -0400 Subject: [SciPy-Dev] PR for overlap-add convolution In-Reply-To: References: Message-ID: Agreed. Or you can use the `monkeypatch` pytest fixture, monkeypatching (in the given test) `scipy.signal.fftconvolve` to be a one-liner than throws an error. Eric On Fri, Sep 27, 2019 at 2:17 AM Nathaniel Smith wrote: > On Thu, Sep 26, 2019 at 8:56 PM Todd wrote: > > To be clear, the situations where it falls back on fftconvolve are the > situations where oaconvolve reduces to fftconvolve. > > I definitely wouldn't expose any argument to let users control this then. > > You could have an internal function that has that argument, write the > fine-grained tests against that internal function, and then have a > public wrapper that just calls the internal function. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From larson.eric.d at gmail.com Fri Sep 27 10:13:46 2019 From: larson.eric.d at gmail.com (Eric Larson) Date: Fri, 27 Sep 2019 10:13:46 -0400 Subject: [SciPy-Dev] suggestion: vector spherical harmonics In-Reply-To: References: Message-ID: Sounds like a useful, in-scope addition to me. Eric On Fri, Sep 27, 2019 at 3:41 AM Thomas Hilger wrote: > The generalization of spherical harmonics to vector-fields are the vector > spherical harmonics [ > https://en.wikipedia.org/wiki/Vector_spherical_harmonics ] . > I can provide simple code which relies on scipy's (scalar) spherical > harmonics implementation and follows the same conventions to give vector > spherical harmonics. > > Could this be a nice-thing-to-have for scipy? A couple of scientific > communities (astronomy, geodesy) use them a lot. > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josh.craig.wilson at gmail.com Fri Sep 27 10:42:48 2019 From: josh.craig.wilson at gmail.com (Joshua Wilson) Date: Fri, 27 Sep 2019 07:42:48 -0700 Subject: [SciPy-Dev] suggestion: vector spherical harmonics In-Reply-To: References: Message-ID: > I can provide simple code which relies on scipy's (scalar) spherical harmonics implementation The problem we're going to run into is that SciPy's scalar spherical harmonics need work. (There is at least https://github.com/scipy/scipy/issues/7778.) Before adding anything that uses them I'd want to see their issues addressed. - Josh On Fri, Sep 27, 2019 at 7:14 AM Eric Larson wrote: > > Sounds like a useful, in-scope addition to me. > > Eric > > > On Fri, Sep 27, 2019 at 3:41 AM Thomas Hilger wrote: >> >> The generalization of spherical harmonics to vector-fields are the vector spherical harmonics [ https://en.wikipedia.org/wiki/Vector_spherical_harmonics ] . >> I can provide simple code which relies on scipy's (scalar) spherical harmonics implementation and follows the same conventions to give vector spherical harmonics. >> >> Could this be a nice-thing-to-have for scipy? A couple of scientific communities (astronomy, geodesy) use them a lot. >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev From toddrjen at gmail.com Fri Sep 27 15:08:45 2019 From: toddrjen at gmail.com (Todd) Date: Fri, 27 Sep 2019 15:08:45 -0400 Subject: [SciPy-Dev] PR for overlap-add convolution In-Reply-To: References: Message-ID: Excellent ideas, thank you both. I have implemented this using pytest's monkeypatch and have removed the argument from the function entirely. -Todd On Fri, Sep 27, 2019 at 9:13 AM Eric Larson wrote: > Agreed. Or you can use the `monkeypatch` pytest fixture, monkeypatching > (in the given test) `scipy.signal.fftconvolve` to be a one-liner than > throws an error. > > Eric > > > On Fri, Sep 27, 2019 at 2:17 AM Nathaniel Smith wrote: > >> On Thu, Sep 26, 2019 at 8:56 PM Todd wrote: >> > To be clear, the situations where it falls back on fftconvolve are the >> situations where oaconvolve reduces to fftconvolve. >> >> I definitely wouldn't expose any argument to let users control this then. >> >> You could have an internal function that has that argument, write the >> fine-grained tests against that internal function, and then have a >> public wrapper that just calls the internal function. >> >> -n >> >> -- >> Nathaniel J. Smith -- https://vorpus.org >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phillip.m.feldman at gmail.com Sat Sep 28 15:09:22 2019 From: phillip.m.feldman at gmail.com (Phillip Feldman) Date: Sat, 28 Sep 2019 12:09:22 -0700 Subject: [SciPy-Dev] suggestion re. scipy.spatial.transform.Rotation In-Reply-To: References: Message-ID: >From the article at https://en.wikiversity.org/wiki/PlanetPhysics/Direction_Cosine_Matrix, it seems as though the direction cosine matrix might be more general than a rotation matrix. Is it possible to add the term "rotation matrix" to the documentation? Thanks! Phillip On Fri, Sep 27, 2019 at 4:24 AM Ali Cetin wrote: > DCM (directional cosine matrix) is the rotation matrix. In order to go in > the reversed direction, just supply the "inverse = True" when calling the > apply method. > > (sorry for not using appropriate quote convention when replying. Answering > from mobile from the beach ;)) > > Cheers, > Ali > > > From: Phillip Feldman > Sent: Friday, September 27, 09:20 > Subject: [SciPy-Dev] suggestion re. scipy.spatial.transform.Rotation > To: SciPy Developers List > > > I've noticed that this class supports four of the five common > representations of a rotation. The one that's missing is the rotation > matrix. I can supply code that converts an arbitrary sequence of Euler > rotations into a rotation matrix. It would also be nice to be able to go > in the reverse direction, but I haven't yet figured that out. > > Phillip > *scipy.spatial.transform.Rotation* > *class *scipy.spatial.transform.Rotation(*quat*, *normalized=False*, > *copy=True*)[source] > > Rotation in 3 dimensions. > This class provides an interface to initialize from and represent > rotations with: > Quaternions > Direction Cosine Matrices > Rotation Vectors > Euler angles > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chrissie.c.l at gmail.com Mon Sep 30 19:42:22 2019 From: chrissie.c.l at gmail.com (Christina Lee) Date: Mon, 30 Sep 2019 16:42:22 -0700 Subject: [SciPy-Dev] New Tutorial on Optimize Help Message-ID: Hi, I'm a SciPy technical writer and am currently rewriting the scipy.optimize tutorial, focusing on `minimize` right now. While I've gotten a grasp of the "how", I still want to explain "why". Why choose one option over another? I could use information from those with more experience. A lot of methods are available. Most problems can have BFGS thrown at them, but I want to explain something for those other cases. Other situations could have features, like constraints or non-differentiability, that lend themselves to a specific method. But the module still has a lot of alternatives. Are they there for academic purposes? Are they the best for some problems? How could someone find that out? For derivatives, users can choose to provide a function or three different types of finite-difference schemes. When is providing a function better than finite-difference derivatives? For Hessians, approximations are sometimes more efficient. How can we know in advance if that's true? Is that ever true for gradients? How do we choose which finite-difference scheme? `3-point` and `cs` (if it is the symmetric approximation I think) have higher-order accuracy, but `cs` uses a point not yet computed. Is `3-point` ever not the way to go? Thanks for your expertise, Christina -------------- next part -------------- An HTML attachment was scrubbed... URL: