From mhaberla at calpoly.edu Sat May 1 17:30:08 2021 From: mhaberla at calpoly.edu (Matt Haberland) Date: Sat, 1 May 2021 14:30:08 -0700 Subject: [SciPy-Dev] Proposal to add `bootstrap_ci` function for estimating confidence intervals of n-sample statistics Message-ID: Dear SciPy Developers, One of the items in the detailed SciPy roadmap is: > Where appropriate, include confidence intervals for the statistic in the results of any statistical test. gh-13371 proposes a function, `bootstrap_ci`, that accepts two arguments, `data` and `statistic`, and (by default) estimates a 95% confidence interval for the true value of the statistic. https://github.com/scipy/scipy/pull/13371 Features: - Implements "percentile", "reverse percentile/basic", and "BCa" bootstrap - Works on n-sample statistics (for now, "BCa" is only available for one-sample statistics) - Works along the provided `axis` of n-dimensional data The PR does not add a confidence interval (or a method for calculating the confidence interval) to the information returned by existing statistical tests, but the function `bootstrap_ci` can easily be used with existing statistical tests by providing a lightly-wrapped version of the test as the `statistic` argument (see examples in documentation). Adding this functionality to existing tests (and new tests) for convenience is being discussed for a follow-up PR. Please consider joining the discussion about this proposed feature in gh-13371! Thanks, Matt -- Matt Haberland Assistant Professor BioResource and Agricultural Engineering 08A-3K, Cal Poly -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyler.je.reddy at gmail.com Sun May 2 13:37:36 2021 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Sun, 2 May 2021 11:37:36 -0600 Subject: [SciPy-Dev] Proposed 1.7.0 Release Schedule Message-ID: Hi, SciPy 1.6.0 was released December 31/2020 (a bit late), a little more than 4 months ago, and I think we'd like to keep a roughly biannual release cadence. I'd like to proposed the following schedule for 1.7.0: - May 26/2021: branch 1.7.x - May 29/2021: rc1 - June 11/2021: rc2 (if needed) - June 20/2021: final release As always, it is a good idea to start tagging things that should be in the 1.7.0 release & please do help with reviewing PRs/issues that are tagged--current counts are: - PRs: 24 open with 1.7.0 milestone - issues: 8 open with 1.7.0 milestone Also great to update the release notes wiki as appropriate: https://github.com/scipy/scipy/wiki/Release-note-entries-for-SciPy-1.7.0 Thoughts/objections for the proposed schedule? The new build time dependency/transpiler, Pythran, would be a notable addition to this release, assuming that moves forward. Best wishes, Tyler -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sun May 2 14:07:46 2021 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sun, 2 May 2021 20:07:46 +0200 Subject: [SciPy-Dev] Proposed 1.7.0 Release Schedule In-Reply-To: References: Message-ID: On Sun, May 2, 2021 at 7:38 PM Tyler Reddy wrote: > Hi, > > SciPy 1.6.0 was released December 31/2020 (a bit late), a little more than > 4 months ago, and I think we'd like to keep a roughly biannual release > cadence. > > I'd like to proposed the following schedule for 1.7.0: > - May 26/2021: branch 1.7.x > - May 29/2021: rc1 > - June 11/2021: rc2 (if needed) > - June 20/2021: final release > Sounds good to me. As always, it is a good idea to start tagging things that should be in the > 1.7.0 release & please do help with reviewing PRs/issues that are > tagged--current counts are: > > - PRs: 24 open with 1.7.0 milestone > - issues: 8 open with 1.7.0 milestone > > Also great to update the release notes wiki as appropriate: > https://github.com/scipy/scipy/wiki/Release-note-entries-for-SciPy-1.7.0 > > Thoughts/objections for the proposed schedule? The new build time > dependency/transpiler, Pythran, would be a notable addition to this > release, assuming that moves forward. > There's a couple of nice PRs pending that depend on it with only a pure Python fallback. While I'm pretty happy with how the use of Pythran is shaping up, I think we should leave it as an optional build dependency only for this release, and try to make it non-optional afterwards. There's always the chance we'll find something unexpected in the release, and there's Windows build thing that needs sorting out - it works for all wheels, but Ilhan is having some trouble with a local build setup. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilhanpolat at gmail.com Mon May 3 06:43:12 2021 From: ilhanpolat at gmail.com (Ilhan Polat) Date: Mon, 3 May 2021 13:43:12 +0300 Subject: [SciPy-Dev] Proposed 1.7.0 Release Schedule In-Reply-To: References: Message-ID: I would also appreciate if other win users can give it a go. It might be the case that my system is the exception so we can ignore my issue. On Sun, 2 May 2021, 21:08 Ralf Gommers, wrote: > > > On Sun, May 2, 2021 at 7:38 PM Tyler Reddy > wrote: > >> Hi, >> >> SciPy 1.6.0 was released December 31/2020 (a bit late), a little more >> than 4 months ago, and I think we'd like to keep a roughly biannual release >> cadence. >> >> I'd like to proposed the following schedule for 1.7.0: >> - May 26/2021: branch 1.7.x >> - May 29/2021: rc1 >> - June 11/2021: rc2 (if needed) >> - June 20/2021: final release >> > > Sounds good to me. > > As always, it is a good idea to start tagging things that should be in the >> 1.7.0 release & please do help with reviewing PRs/issues that are >> tagged--current counts are: >> >> - PRs: 24 open with 1.7.0 milestone >> - issues: 8 open with 1.7.0 milestone >> >> Also great to update the release notes wiki as appropriate: >> https://github.com/scipy/scipy/wiki/Release-note-entries-for-SciPy-1.7.0 >> >> Thoughts/objections for the proposed schedule? The new build time >> dependency/transpiler, Pythran, would be a notable addition to this >> release, assuming that moves forward. >> > > There's a couple of nice PRs pending that depend on it with only a pure > Python fallback. While I'm pretty happy with how the use of Pythran is > shaping up, I think we should leave it as an optional build dependency only > for this release, and try to make it non-optional afterwards. There's > always the chance we'll find something unexpected in the release, and > there's Windows build thing that needs sorting out - it works for all > wheels, but Ilhan is having some trouble with a local build setup. > > Cheers, > Ralf > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Albert_Steppi at hms.harvard.edu Mon May 3 15:10:21 2021 From: Albert_Steppi at hms.harvard.edu (Steppi, Albert) Date: Mon, 3 May 2021 19:10:21 +0000 Subject: [SciPy-Dev] Update: Inverse of Log CDF of Normal Distribution Message-ID: Dear SciPy developers, I mentioned last week that I have an implementation of the inverse of the log CDF of the normal distribution that I'd like to contribute to scipy. scipy.special currently contains the functions ndtr, ndtri, and log_ndtr, for the CDF, the inverse CDF, and the log of the CDF of the normal distribution respectively. I propose adding a function ndtri_exp for the inverse of the log CDF. Earlier I submitted the issue gh-13923 regarding this. Today I've submitted the PR gh-13979 with an enhancement that adds ndtri_exp as a ufunc to scipy.special. I think this could be a valuable function to add. It has proven useful in my group's research and was specifically requested in gh-11465 so is likely to be useful to others. I understand that you are all very busy and may not have time to attend to this PR immediately. I appreciate all the work you do; I've been a scipy user for over a decade and have found it an immensely valuable resource. I hope someone else may find this ndtri_exp implementation valuable. Thanks, Albert -- Albert Steppi III, PhD Scientific Software Developer Laboratory of Systems Pharmacology Harvard Medical School -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Tue May 4 05:51:31 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Tue, 4 May 2021 11:51:31 +0200 Subject: [SciPy-Dev] SciPy Community Meetups are back! In-Reply-To: <1605BE4D-E642-4CD7-848A-5FA3BBE56982@gmail.com> References: <1605BE4D-E642-4CD7-848A-5FA3BBE56982@gmail.com> Message-ID: Hi everyone, This is a friendly reminder to complete the doodle to plan the meetup. If not otherwise stated, I am planning on fixing the date by the end of this week (May 7). > https://doodle.com/poll/fd62d6755texie5s Thank you all for your help and engagement. Cheers, Pamphile > On 30.04.2021, at 11:48, Pamphile Roy wrote: > > Hi everyone, > > I would like to propose regular SciPy Community meetings again! > > https://doodle.com/poll/fd62d6755texie5s > > For this first meeting, it would be good if as many people as possible could join as we would decide things like frequency of such meeting. > > Everyone is invited and encouraged to > join in and edit the work-in-progress meeting topics and notes at: > > https://hackmd.io/@tupui/scipy-meetup/edit > > Cheers, > Pamphile > > PS. share the news: https://twitter.com/PamphileRoy/status/1388067651721830401?s=20 -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Tue May 4 05:56:26 2021 From: andyfaff at gmail.com (Andrew Nelson) Date: Tue, 4 May 2021 19:56:26 +1000 Subject: [SciPy-Dev] SciPy Community Meetups are back! In-Reply-To: References: <1605BE4D-E642-4CD7-848A-5FA3BBE56982@gmail.com> Message-ID: All those times are pretty much impossible for those in AUS/NZ. On Tue, 4 May 2021, 19:53 Pamphile Roy, wrote: > Hi everyone, > > This is a friendly reminder to complete the doodle to plan the meetup. > > If not otherwise stated, I am planning on fixing the date by the end of > this week (May 7). > > https://doodle.com/poll/fd62d6755texie5s > > > Thank you all for your help and engagement. > > Cheers, > Pamphile > > > On 30.04.2021, at 11:48, Pamphile Roy wrote: > > Hi everyone, > > I would like to propose regular SciPy Community meetings again! > > https://doodle.com/poll/fd62d6755texie5s > > For this first meeting, it would be good if as many people as possible > could join as we would decide things like frequency of such meeting. > > Everyone is invited and encouraged to > join in and edit the work-in-progress meeting topics and notes at: > > https://hackmd.io/@tupui/scipy-meetup/edit > > Cheers, > Pamphile > > PS. share the news: > https://twitter.com/PamphileRoy/status/1388067651721830401?s=20 > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sayedmazen70 at gmail.com Tue May 4 06:08:51 2021 From: sayedmazen70 at gmail.com (Mazen Sayed) Date: Tue, 4 May 2021 12:08:51 +0200 Subject: [SciPy-Dev] SciPy Community Meetups are back! In-Reply-To: References: <1605BE4D-E642-4CD7-848A-5FA3BBE56982@gmail.com> Message-ID: Hi Pamphile For some reason, the choices starts from May 17 , Do I have a problem from my end? On Tuesday, May 4, 2021, Pamphile Roy wrote: > Hi everyone, > > This is a friendly reminder to complete the doodle to plan the meetup. > > If not otherwise stated, I am planning on fixing the date by the end of > this week (May 7). > > https://doodle.com/poll/fd62d6755texie5s > > > Thank you all for your help and engagement. > > Cheers, > Pamphile > > > On 30.04.2021, at 11:48, Pamphile Roy wrote: > > Hi everyone, > > I would like to propose regular SciPy Community meetings again! > > https://doodle.com/poll/fd62d6755texie5s > > For this first meeting, it would be good if as many people as possible > could join as we would decide things like frequency of such meeting. > > Everyone is invited and encouraged to > join in and edit the work-in-progress meeting topics and notes at: > > https://hackmd.io/@tupui/scipy-meetup/edit > > Cheers, > Pamphile > > PS. share the news: https://twitter.com/PamphileRoy/status/ > 1388067651721830401?s=20 > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Tue May 4 06:08:59 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Tue, 4 May 2021 12:08:59 +0200 Subject: [SciPy-Dev] SciPy Community Meetups are back! In-Reply-To: References: <1605BE4D-E642-4CD7-848A-5FA3BBE56982@gmail.com> Message-ID: <222F4B45-F11E-4142-9BD6-36AEC27D9A29@gmail.com> I am not sure if we can solve this. From the table below, the only solution I could see would be 6 or 7a.m. CET. I can add other times or I could propose that we rotate the time from one meeting to the other (might be the most reasonable). Cheers, Pamphile > On 04.05.2021, at 11:56, Andrew Nelson wrote: > > All those times are pretty much impossible for those in AUS/NZ. > > On Tue, 4 May 2021, 19:53 Pamphile Roy, > wrote: > Hi everyone, > > This is a friendly reminder to complete the doodle to plan the meetup. > > If not otherwise stated, I am planning on fixing the date by the end of this week (May 7). > >> https://doodle.com/poll/fd62d6755texie5s > > > Thank you all for your help and engagement. > > Cheers, > Pamphile > > >> On 30.04.2021, at 11:48, Pamphile Roy > wrote: >> >> Hi everyone, >> >> I would like to propose regular SciPy Community meetings again! >> >> https://doodle.com/poll/fd62d6755texie5s >> >> For this first meeting, it would be good if as many people as possible could join as we would decide things like frequency of such meeting. >> >> Everyone is invited and encouraged to >> join in and edit the work-in-progress meeting topics and notes at: >> >> https://hackmd.io/@tupui/scipy-meetup/edit >> >> Cheers, >> Pamphile >> >> PS. share the news: https://twitter.com/PamphileRoy/status/1388067651721830401?s=20 > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot 2021-05-04 at 12.03.10.png Type: image/png Size: 243552 bytes Desc: not available URL: From roy.pamphile at gmail.com Tue May 4 06:10:58 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Tue, 4 May 2021 12:10:58 +0200 Subject: [SciPy-Dev] SciPy Community Meetups are back! In-Reply-To: References: <1605BE4D-E642-4CD7-848A-5FA3BBE56982@gmail.com> Message-ID: <96FDB9E3-6801-4E08-A4E5-59959C47DE45@gmail.com> Hi Mazen, It?s correct, it goes from May 17 to May 21. But I want to close the pool by May 7. Cheers, Pamphile > On 04.05.2021, at 12:08, Mazen Sayed wrote: > > Hi Pamphile > > For some reason, the choices starts from May 17 , Do I have a problem from my end? > > > On Tuesday, May 4, 2021, Pamphile Roy > wrote: > Hi everyone, > > This is a friendly reminder to complete the doodle to plan the meetup. > > If not otherwise stated, I am planning on fixing the date by the end of this week (May 7). > >> https://doodle.com/poll/fd62d6755texie5s > > > Thank you all for your help and engagement. > > Cheers, > Pamphile > > >> On 30.04.2021, at 11:48, Pamphile Roy > wrote: >> >> Hi everyone, >> >> I would like to propose regular SciPy Community meetings again! >> >> https://doodle.com/poll/fd62d6755texie5s >> >> For this first meeting, it would be good if as many people as possible could join as we would decide things like frequency of such meeting. >> >> Everyone is invited and encouraged to >> join in and edit the work-in-progress meeting topics and notes at: >> >> https://hackmd.io/@tupui/scipy-meetup/edit >> >> Cheers, >> Pamphile >> >> PS. share the news: https://twitter.com/PamphileRoy/status/1388067651721830401?s=20 > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From sayedmazen70 at gmail.com Tue May 4 06:12:18 2021 From: sayedmazen70 at gmail.com (Mazen Sayed) Date: Tue, 4 May 2021 12:12:18 +0200 Subject: [SciPy-Dev] SciPy Community Meetups are back! In-Reply-To: <96FDB9E3-6801-4E08-A4E5-59959C47DE45@gmail.com> References: <1605BE4D-E642-4CD7-848A-5FA3BBE56982@gmail.com> <96FDB9E3-6801-4E08-A4E5-59959C47DE45@gmail.com> Message-ID: Got it. Thanks On Tuesday, May 4, 2021, Pamphile Roy wrote: > Hi Mazen, > > It?s correct, it goes from May 17 to May 21. But I want to close the pool > by May 7. > > Cheers, > Pamphile > > On 04.05.2021, at 12:08, Mazen Sayed wrote: > > Hi Pamphile > > For some reason, the choices starts from May 17 , Do I have a problem from > my end? > > > On Tuesday, May 4, 2021, Pamphile Roy wrote: > >> Hi everyone, >> >> This is a friendly reminder to complete the doodle to plan the meetup. >> >> If not otherwise stated, I am planning on fixing the date by the end of >> this week (May 7). >> >> https://doodle.com/poll/fd62d6755texie5s >> >> >> Thank you all for your help and engagement. >> >> Cheers, >> Pamphile >> >> >> On 30.04.2021, at 11:48, Pamphile Roy wrote: >> >> Hi everyone, >> >> I would like to propose regular SciPy Community meetings again! >> >> https://doodle.com/poll/fd62d6755texie5s >> >> For this first meeting, it would be good if as many people as possible >> could join as we would decide things like frequency of such meeting. >> >> Everyone is invited and encouraged to >> join in and edit the work-in-progress meeting topics and notes at: >> >> https://hackmd.io/@tupui/scipy-meetup/edit >> >> Cheers, >> Pamphile >> >> PS. share the news: https://twitter.com/Pamp >> hileRoy/status/1388067651721830401?s=20 >> >> >> _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Tue May 4 06:21:02 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Tue, 4 May 2021 12:21:02 +0200 Subject: [SciPy-Dev] SciPy Community Meetups are back! In-Reply-To: <222F4B45-F11E-4142-9BD6-36AEC27D9A29@gmail.com> References: <1605BE4D-E642-4CD7-848A-5FA3BBE56982@gmail.com> <222F4B45-F11E-4142-9BD6-36AEC27D9A29@gmail.com> Message-ID: <151F12E8-BC7B-40D9-A27D-462FF3567873@gmail.com> I updated the doodle to propose 7a.m. CET. Cheers, Pamphile > On 04.05.2021, at 12:08, Pamphile Roy wrote: > > I am not sure if we can solve this. From the table below, the only solution I could see would be 6 or 7a.m. CET. > > I can add other times or I could propose that we rotate the time from one meeting to the other (might be the most reasonable). > > Cheers, > Pamphile > > > > > > >> On 04.05.2021, at 11:56, Andrew Nelson > wrote: >> >> All those times are pretty much impossible for those in AUS/NZ. >> >> On Tue, 4 May 2021, 19:53 Pamphile Roy, > wrote: >> Hi everyone, >> >> This is a friendly reminder to complete the doodle to plan the meetup. >> >> If not otherwise stated, I am planning on fixing the date by the end of this week (May 7). >> >>> https://doodle.com/poll/fd62d6755texie5s >> >> >> Thank you all for your help and engagement. >> >> Cheers, >> Pamphile >> >> >>> On 30.04.2021, at 11:48, Pamphile Roy > wrote: >>> >>> Hi everyone, >>> >>> I would like to propose regular SciPy Community meetings again! >>> >>> https://doodle.com/poll/fd62d6755texie5s >>> >>> For this first meeting, it would be good if as many people as possible could join as we would decide things like frequency of such meeting. >>> >>> Everyone is invited and encouraged to >>> join in and edit the work-in-progress meeting topics and notes at: >>> >>> https://hackmd.io/@tupui/scipy-meetup/edit >>> >>> Cheers, >>> Pamphile >>> >>> PS. share the news: https://twitter.com/PamphileRoy/status/1388067651721830401?s=20 >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue May 4 09:41:30 2021 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 4 May 2021 15:41:30 +0200 Subject: [SciPy-Dev] Update: Inverse of Log CDF of Normal Distribution In-Reply-To: References: Message-ID: On Mon, May 3, 2021 at 9:10 PM Steppi, Albert wrote: > Dear SciPy developers, > > I mentioned last week that I have an implementation of the inverse of the > log > CDF of the normal distribution that I'd like to contribute to > scipy. scipy.special currently contains the functions ndtr, ndtri, and > log_ndtr, > for the CDF, the inverse CDF, and the log of the CDF of the normal > distribution > respectively. I propose adding a function ndtri_exp for the inverse of the > log > CDF. Earlier I submitted the issue gh-13923 regarding this. Today I've > submitted > the PR gh-13979 with an enhancement that adds ndtri_exp as a ufunc to > scipy.special. I think this could be a valuable function to add. It has > proven > useful in my group's research and was specifically requested in gh-11465 > so is > likely to be useful to others. > > I understand that you are all very busy and may not have time to attend to > this > PR immediately. I appreciate all the work you do; I've been a scipy user > for > over a decade and have found it an immensely valuable resource. I hope > someone > else may find this ndtri_exp implementation valuable. > Hi Albert, sorry about the slow/no reply. It's not for a lack of interest, your PR does look good and seems like a valuable contribution. Thanks for submitting it! Other than "+1 for adding this function" I don't have much to add here; will try to look at your PR in more detail soon. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From Albert_Steppi at hms.harvard.edu Tue May 4 11:34:07 2021 From: Albert_Steppi at hms.harvard.edu (Steppi, Albert) Date: Tue, 4 May 2021 15:34:07 +0000 Subject: [SciPy-Dev] Update: Inverse of Log CDF of Normal Distribution In-Reply-To: References: , Message-ID: Thanks Ralf, No worries. I know you all have a lot to keep on top of. Happy to have something to contribute! Best, Albert ________________________________ From: SciPy-Dev on behalf of Ralf Gommers Sent: Tuesday, May 4, 2021 9:41 AM To: SciPy Developers List Subject: Re: [SciPy-Dev] Update: Inverse of Log CDF of Normal Distribution On Mon, May 3, 2021 at 9:10 PM Steppi, Albert > wrote: Dear SciPy developers, I mentioned last week that I have an implementation of the inverse of the log CDF of the normal distribution that I'd like to contribute to scipy. scipy.special currently contains the functions ndtr, ndtri, and log_ndtr, for the CDF, the inverse CDF, and the log of the CDF of the normal distribution respectively. I propose adding a function ndtri_exp for the inverse of the log CDF. Earlier I submitted the issue gh-13923 regarding this. Today I've submitted the PR gh-13979 with an enhancement that adds ndtri_exp as a ufunc to scipy.special. I think this could be a valuable function to add. It has proven useful in my group's research and was specifically requested in gh-11465 so is likely to be useful to others. I understand that you are all very busy and may not have time to attend to this PR immediately. I appreciate all the work you do; I've been a scipy user for over a decade and have found it an immensely valuable resource. I hope someone else may find this ndtri_exp implementation valuable. Hi Albert, sorry about the slow/no reply. It's not for a lack of interest, your PR does look good and seems like a valuable contribution. Thanks for submitting it! Other than "+1 for adding this function" I don't have much to add here; will try to look at your PR in more detail soon. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Thu May 6 04:49:40 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Thu, 6 May 2021 10:49:40 +0200 Subject: [SciPy-Dev] Custom __repr__? Message-ID: Hi everyone, I was wondering if we would want to improve our __repr__/__str__ functions. This just came out with the Matplotlib release: https://matplotlib.org/stable/users/whats_new.html#ipython-representations-for-colormap-objects I can imagine something like this to display distributions for instance. There are many possible applications as you can imagine. FYI, this extra capabilities are available thanks to iPython reading extra _repr_xxx functions: https://ipython.readthedocs.io/en/stable/api/generated/IPython.display.html#IPython.display.display Let me know what you think. Cheers, Pamphile p.s.: don?t forget to register for SciPy?s next community meetup https://mail.python.org/pipermail/scipy-dev/2021-May/024763.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu May 6 10:58:09 2021 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 6 May 2021 10:58:09 -0400 Subject: [SciPy-Dev] Custom __repr__? In-Reply-To: References: Message-ID: On Thu, May 6, 2021 at 4:50 AM Pamphile Roy wrote: > Hi everyone, > > I was wondering if we would want to improve our *__repr__/__str__* > functions. > > This just came out with the Matplotlib release: > > https://matplotlib.org/stable/users/whats_new.html#ipython-representations-for-colormap-objects > > > I can imagine something like this to display distributions for instance. > There are many possible applications as you can imagine. > > FYI, this extra capabilities are available thanks to iPython reading extra > *_repr_xxx* functions: > > https://ipython.readthedocs.io/en/stable/api/generated/IPython.display.html#IPython.display.display > > Let me know what you think. > Because these would involve extra dependencies (either matplotlib or some JavaScript), I would recommend starting with implementing these outside of scipy. You can register formatters for objects without modifying the objects themselves. https://nbviewer.jupyter.org/github/ipython/ipython/blob/master/examples/IPython%20Kernel/Custom%20Display%20Logic.ipynb -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Thu May 6 11:10:11 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Thu, 6 May 2021 17:10:11 +0200 Subject: [SciPy-Dev] Custom __repr__? In-Reply-To: References: Message-ID: Thanks for your inputs Robert. > Because these would involve extra dependencies (either matplotlib or some JavaScript), I would recommend starting with implementing these outside of scipy. You can register formatters for objects without modifying the objects themselves. We already have a soft dependency on matplotlib in spatial for instance with a few helper functions (https://scipy.github.io/devdocs/reference/spatial.html#plotting-helpers ). It would be the same here, something completely optional. Plus here the functions are private and don?t conflict with Python?s internals. > https://nbviewer.jupyter.org/github/ipython/ipython/blob/master/examples/IPython%20Kernel/Custom%20Display%20Logic.ipynb Nice examples, this is exactly this I had in mind. Well, this registration outside is quite complicated IMO? Personally, I think I would even prefer to mock the underlying object and add the _repr_xxx I want? Cheers, Pamphile -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu May 6 11:50:12 2021 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 6 May 2021 11:50:12 -0400 Subject: [SciPy-Dev] Custom __repr__? In-Reply-To: References: Message-ID: On Thu, May 6, 2021 at 11:10 AM Pamphile Roy wrote: > Thanks for your inputs Robert. > > Because these would involve extra dependencies (either matplotlib or some > JavaScript), I would recommend starting with implementing these outside of > scipy. You can register formatters for objects without modifying the > objects themselves. > > > We already have a soft dependency on matplotlib in *spatial* for instance > with a few helper functions ( > https://scipy.github.io/devdocs/reference/spatial.html#plotting-helpers). > It would be the same here, something completely optional. Plus here the > functions are private and don?t conflict with Python?s internals. > > > https://nbviewer.jupyter.org/github/ipython/ipython/blob/master/examples/IPython%20Kernel/Custom%20Display%20Logic.ipynb > > > Nice examples, this is exactly this I had in mind. > > Well, this registration outside is quite complicated IMO? Personally, I > think I would even prefer to mock the underlying object and add the > *_repr_xxx* I want? > The user doesn't do this in each notebook. You do it once inside of an IPython extension, and the user just %load_ext's the extension. I think this is the kind of thing I'd like to review where it's opt-in at first. An IPython extension is a good way to do that. Then maybe after we get some experience using it for a while, then decide to make it always-on by modifying the objects themselves. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Thu May 6 11:53:46 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Thu, 6 May 2021 17:53:46 +0200 Subject: [SciPy-Dev] Custom __repr__? In-Reply-To: References: Message-ID: > The user doesn't do this in each notebook. You do it once inside of an IPython extension, and the user just %load_ext's the extension. > > I think this is the kind of thing I'd like to review where it's opt-in at first. An IPython extension is a good way to do that. Then maybe after we get some experience using it for a while, then decide to make it always-on by modifying the objects themselves. Ah so you propose I do an extension instead? Sounds reasonable indeed. I will see if I can get some time for that ;) Cheers, Pamphile -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu May 6 12:03:25 2021 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 6 May 2021 12:03:25 -0400 Subject: [SciPy-Dev] Custom __repr__? In-Reply-To: References: Message-ID: On Thu, May 6, 2021 at 11:54 AM Pamphile Roy wrote: > The user doesn't do this in each notebook. You do it once inside of an > IPython extension, and the user just %load_ext's the extension. > > I think this is the kind of thing I'd like to review where it's opt-in at > first. An IPython extension is a good way to do that. Then maybe after we > get some experience using it for a while, then decide to make it always-on > by modifying the objects themselves. > > Ah so you propose I do an extension instead? Sounds reasonable indeed. I > will see if I can get some time for that ;) > Yeah, I think it gives you more freedom to play, people can use it with older versions of scipy, and we can use it for real work for a while before we commit to supporting it forever inside scipy itself. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Fri May 7 17:17:52 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Fri, 7 May 2021 23:17:52 +0200 Subject: [SciPy-Dev] SciPy Community Meetups are back! In-Reply-To: <151F12E8-BC7B-40D9-A27D-462FF3567873@gmail.com> References: <1605BE4D-E642-4CD7-848A-5FA3BBE56982@gmail.com> <222F4B45-F11E-4142-9BD6-36AEC27D9A29@gmail.com> <151F12E8-BC7B-40D9-A27D-462FF3567873@gmail.com> Message-ID: Hi everyone, First of all, thank you for your participation to the pool! You have been the most to vote for May 17th at 16 CET. https://zoom.us/j/93827717491 Please feel free to contribute to the agenda: https://hackmd.io/@tupui/scipy-meetup/edit In case you cannot join us, I am thinking about recording this event. Of course, before any recording starts, we will discuss about it. I will see you there, and I am looking forward to finally meet you all. In the meantime, don?t hesitate to get in touch with me =) Cheers, Pamphile > On 04.05.2021, at 12:21, Pamphile Roy wrote: > > I updated the doodle to propose 7a.m. CET. > > Cheers, > Pamphile > >> On 04.05.2021, at 12:08, Pamphile Roy > wrote: >> >> I am not sure if we can solve this. From the table below, the only solution I could see would be 6 or 7a.m. CET. >> >> I can add other times or I could propose that we rotate the time from one meeting to the other (might be the most reasonable). >> >> Cheers, >> Pamphile >> >> >> >> >> >> >>> On 04.05.2021, at 11:56, Andrew Nelson > wrote: >>> >>> All those times are pretty much impossible for those in AUS/NZ. >>> >>> On Tue, 4 May 2021, 19:53 Pamphile Roy, > wrote: >>> Hi everyone, >>> >>> This is a friendly reminder to complete the doodle to plan the meetup. >>> >>> If not otherwise stated, I am planning on fixing the date by the end of this week (May 7). >>> >>>> https://doodle.com/poll/fd62d6755texie5s >>> >>> >>> Thank you all for your help and engagement. >>> >>> Cheers, >>> Pamphile >>> >>> >>>> On 30.04.2021, at 11:48, Pamphile Roy > wrote: >>>> >>>> Hi everyone, >>>> >>>> I would like to propose regular SciPy Community meetings again! >>>> >>>> https://doodle.com/poll/fd62d6755texie5s >>>> >>>> For this first meeting, it would be good if as many people as possible could join as we would decide things like frequency of such meeting. >>>> >>>> Everyone is invited and encouraged to >>>> join in and edit the work-in-progress meeting topics and notes at: >>>> >>>> https://hackmd.io/@tupui/scipy-meetup/edit >>>> >>>> Cheers, >>>> Pamphile >>>> >>>> PS. share the news: https://twitter.com/PamphileRoy/status/1388067651721830401?s=20 >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Sat May 8 04:45:45 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Sat, 8 May 2021 10:45:45 +0200 Subject: [SciPy-Dev] SciPy Community Meetups are back! In-Reply-To: References: <1605BE4D-E642-4CD7-848A-5FA3BBE56982@gmail.com> <222F4B45-F11E-4142-9BD6-36AEC27D9A29@gmail.com> <151F12E8-BC7B-40D9-A27D-462FF3567873@gmail.com> Message-ID: Hi everyone, I forgot to include the password, here is the complete link: https://zoom.us/j/93827717491?pwd=UGdkMzlnWEFjbDNQUmxpRkp6L0VLdz09 Meeting ID: 938 2771 7491 Passcode: 602749 Also, thank you to Ralf for his help and providing us with this link! Cheers, Pamphile > On 07.05.2021, at 23:17, Pamphile Roy wrote: > > Hi everyone, > > First of all, thank you for your participation to the pool! > > You have been the most to vote for May 17th at 16 CET. > > https://zoom.us/j/93827717491 > > Please feel free to contribute to the agenda: > > https://hackmd.io/@tupui/scipy-meetup/edit > > In case you cannot join us, I am thinking about recording this event. > Of course, before any recording starts, we will discuss about it. > > I will see you there, and I am looking forward to finally meet you all. > In the meantime, don?t hesitate to get in touch with me =) > > Cheers, > Pamphile > > >> On 04.05.2021, at 12:21, Pamphile Roy > wrote: >> >> I updated the doodle to propose 7a.m. CET. >> >> Cheers, >> Pamphile >> >>> On 04.05.2021, at 12:08, Pamphile Roy > wrote: >>> >>> I am not sure if we can solve this. From the table below, the only solution I could see would be 6 or 7a.m. CET. >>> >>> I can add other times or I could propose that we rotate the time from one meeting to the other (might be the most reasonable). >>> >>> Cheers, >>> Pamphile >>> >>> >>> >>> >>> >>> >>>> On 04.05.2021, at 11:56, Andrew Nelson > wrote: >>>> >>>> All those times are pretty much impossible for those in AUS/NZ. >>>> >>>> On Tue, 4 May 2021, 19:53 Pamphile Roy, > wrote: >>>> Hi everyone, >>>> >>>> This is a friendly reminder to complete the doodle to plan the meetup. >>>> >>>> If not otherwise stated, I am planning on fixing the date by the end of this week (May 7). >>>> >>>>> https://doodle.com/poll/fd62d6755texie5s >>>> >>>> >>>> Thank you all for your help and engagement. >>>> >>>> Cheers, >>>> Pamphile >>>> >>>> >>>>> On 30.04.2021, at 11:48, Pamphile Roy > wrote: >>>>> >>>>> Hi everyone, >>>>> >>>>> I would like to propose regular SciPy Community meetings again! >>>>> >>>>> https://doodle.com/poll/fd62d6755texie5s >>>>> >>>>> For this first meeting, it would be good if as many people as possible could join as we would decide things like frequency of such meeting. >>>>> >>>>> Everyone is invited and encouraged to >>>>> join in and edit the work-in-progress meeting topics and notes at: >>>>> >>>>> https://hackmd.io/@tupui/scipy-meetup/edit >>>>> >>>>> Cheers, >>>>> Pamphile >>>>> >>>>> PS. share the news: https://twitter.com/PamphileRoy/status/1388067651721830401?s=20 >>>> >>>> _______________________________________________ >>>> SciPy-Dev mailing list >>>> SciPy-Dev at python.org >>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>> _______________________________________________ >>>> SciPy-Dev mailing list >>>> SciPy-Dev at python.org >>>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peterbell10 at live.co.uk Sat May 8 15:40:24 2021 From: peterbell10 at live.co.uk (Peter Bell) Date: Sat, 8 May 2021 19:40:24 +0000 Subject: [SciPy-Dev] numpy/npy_math.h vs C/C++ stdlib math.h/ In-Reply-To: References: Message-ID: what is the recommendation going forward for the math functions (log, exp, trig functions etc) in scipy C/C++ code: do we import them from `numpy/npy_math.h` or do we rely on the standard library? My main concern is more exotic platforms/toolchains and related edge cases--- do we rely on numpy or the standard library vendors? I've had a look through the npy_math.h implementation, most of which is in this file: https://github.com/numpy/numpy/blob/main/numpy/core/src/npymath/npy_math_internal.h.src All re-implementations of standard functions are guarded by "#ifndef HAVE_ at KIND@". i.e. they are only compiled on older toolchains without C99 math library functions. SciPy requires C99 and C++11 already, so AFAIK all of the functions should be available and in fact call the same underlying implementation as the npy_math functions. My suggestion would be: prefer the standard library functions but don't go out of your way to avoid npy_math. Cheers, Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Sat May 8 23:21:27 2021 From: matti.picus at gmail.com (Matti Picus) Date: Sun, 9 May 2021 06:21:27 +0300 Subject: [SciPy-Dev] numpy/npy_math.h vs C/C++ stdlib math.h/ In-Reply-To: References: Message-ID: <9613fc5d-c7c3-32b9-3c7b-9a905415acca@gmail.com> On 8/5/21 10:40 pm, Peter Bell wrote: > I've had a look through the npy_math.h implementation, most of which > is in this file: > https://github.com/numpy/numpy/blob/main/numpy/core/src/npymath/npy_math_internal.h.src > > > All re-implementations of standard functions are guarded by > "#ifndef HAVE_ at KIND@". i.e. they are only compiled on older toolchains > without C99 math library functions. SciPy requires C99 and C++11 > already, so AFAIK all of the functions should be available and > in fact call the same underlying implementation as the npy_math functions. > > My suggestion would be: prefer the standard library functions but > don't go out of your way to avoid npy_math. > > Cheers, > Peter It is a little more complicated and not dependent only on C99. Platforms that use glibc<2.18 or MSVC have known buggy implementations of some functions. Including npy_config.h will blocklist the known bad ones. Then the alternate implementations will be used. Today it is impossible to produce a wheel with glibc >= 2.18: even manylinux2014 uses glibc 2.17 with the buggy functions. The problem is worse on IBM platforms where even the NumPy alternatives are buggy (long double is defined differently on Power and S390X, which breaks the NumPy functions). This is the reason I am pushing the manylinux2_24 wheel standard which will use glibc 2.24, and we can finally not use the NumPy alternatives. Matti From serge.guelton at telecom-bretagne.eu Sun May 9 16:12:36 2021 From: serge.guelton at telecom-bretagne.eu (Serge Guelton) Date: Sun, 9 May 2021 22:12:36 +0200 Subject: [SciPy-Dev] =?iso-8859-1?q?pythran_0=2E9=2E10_-_troer-bi=F1so=F9?= Message-ID: <20210509201236.GA1982635@sguelton.remote.csb> Hi pythraners, /pythran is an ahead of time compiler for high-level scientific kernels written in Python./ I did a quick release of the pythran compiler today. It's only a few months since the previous release, but this one fixes an important memory leak with transposed arguments, so here it comes. Changelog: https://github.com/serge-sans-paille/pythran/blob/0.9.10/Changelog On Github: https://github.com/serge-sans-paille/pythran/tree/0.9.10 On Pypi: https://pypi.org/project/pythran/ Release on conda-forge and on fedora-rawhide should come soon. We recieved a lot more third party contributions than usual for this release, so let me thank Angus Gibson David Brochart Konrad Kleine ocaisa Pierre Augier Ralf Gommers Stefan van der Walt Sylvain Corlay From charlesr.harris at gmail.com Mon May 10 11:52:47 2021 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 10 May 2021 09:52:47 -0600 Subject: [SciPy-Dev] NumPy 1.20.3 release Message-ID: Hi All, On behalf of the NumPy team I am pleased to announce the release of NumPy 1.20.3. NumPy 1,20.3 is a bugfix release containing several fixes merged to the main branch after the NumPy 1.20.2 release. The Python versions supported for this release are 3.7-3.9. Wheels can be downloaded from PyPI ; source archives, release notes, and wheel hashes are available on Github . Linux users will need pip >= 0.19.3 in order to install manylinux2010 and manylinux2014 wheels. *Contributors* A total of 7 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Anne Archibald - Bas van Beek - Charles Harris - Dong Keun Oh + - Kamil Choudhury + - Sayed Adel - Sebastian Berg *Pull requests merged* A total of 15 pull requests were merged for this release. - #18763: BUG: Correct ``datetime64`` missing type overload for ``datetime.date``... - #18764: MAINT: Remove ``__all__`` in favor of explicit re-exports - #18768: BLD: Strip extra newline when dumping gfortran version on MacOS - #18769: BUG: fix segfault in object/longdouble operations - #18794: MAINT: Use towncrier build explicitly - #18887: MAINT: Relax certain integer-type constraints - #18915: MAINT: Remove unsafe unions and ABCs from return-annotations - #18921: MAINT: Allow more recursion depth for scalar tests. - #18922: BUG: Initialize the full nditer buffer in case of error - #18923: BLD: remove unnecessary flag ``-faltivec`` on macOS - #18924: MAINT, CI: treats _SIMD module build warnings as errors through... - #18925: BUG: for MINGW, threads.h existence test requires GLIBC > 2.12 - #18941: BUG: Make changelog recognize gh- as a PR number prefix. - #18948: REL, DOC: Prepare for the NumPy 1.20.3 release. - #18953: BUG: Fix failing mypy test in 1.20.x. Cheers, Charles Harris -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue May 11 10:24:36 2021 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 11 May 2021 16:24:36 +0200 Subject: [SciPy-Dev] =?utf-8?q?pythran_0=2E9=2E10_-_troer-bi=C3=B1so?= =?utf-8?b?w7k=?= In-Reply-To: <20210509201236.GA1982635@sguelton.remote.csb> References: <20210509201236.GA1982635@sguelton.remote.csb> Message-ID: On Sun, May 9, 2021 at 10:22 PM Serge Guelton < serge.guelton at telecom-bretagne.eu> wrote: > Hi pythraners, > > /pythran is an ahead of time compiler for high-level scientific kernels > written in Python./ > > I did a quick release of the pythran compiler today. It's only a few > months since the > previous release, but this one fixes an important memory leak with > transposed arguments, > so here it comes. > > Changelog: > https://github.com/serge-sans-paille/pythran/blob/0.9.10/Changelog > > On Github: https://github.com/serge-sans-paille/pythran/tree/0.9.10 > On Pypi: https://pypi.org/project/pythran/ > > Release on conda-forge and on fedora-rawhide should come soon. > Thanks Serge! There's quite a few fixes/improvements here that are useful for SciPy. I see the conda-forge release is done as well now, which means we can soon merge the PR for RBF interpolation ( https://github.com/scipy/scipy/pull/13595/) - that'll be nice to have for the next release. Cheers, Ralf > We recieved a lot more third party contributions than usual for this > release, so > let me thank > > Angus Gibson > David Brochart > Konrad Kleine > ocaisa > Pierre Augier > Ralf Gommers > Stefan van der Walt > Sylvain Corlay > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue May 11 16:04:50 2021 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 11 May 2021 22:04:50 +0200 Subject: [SciPy-Dev] SciPy virtual sprint @ PyCon Message-ID: Hi all, After the NumPy sprint last Sunday, which was fun and successful, I got an invite to participate during the PyCon mentored sprints the coming weekend (see https://us.pycon.org/2021/summits/mentored-sprints/). They are run on the same infrastructure (Discord), it works quite well. There are multiple projects participating, I signed up for one sprint "table" for SciPy - noon to 4pm CEST this Saturday (May 15th). Are there other people who would like to help out and mentor newcomers make their first contribution to SciPy? Note that there are also Americas and Asia-Pacific 4 hour time slots, see the link above. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Tue May 11 16:10:45 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Tue, 11 May 2021 22:10:45 +0200 Subject: [SciPy-Dev] SciPy virtual sprint @ PyCon In-Reply-To: References: Message-ID: Hey Ralf, Sounds great! I can help if you have a role for me :) Cheers, Pamphile > On 11 May 2021, at 22:05, Ralf Gommers wrote: > > ? > Hi all, > > After the NumPy sprint last Sunday, which was fun and successful, I got an invite to participate during the PyCon mentored sprints the coming weekend (see https://us.pycon.org/2021/summits/mentored-sprints/). They are run on the same infrastructure (Discord), it works quite well. > > There are multiple projects participating, I signed up for one sprint "table" for SciPy - noon to 4pm CEST this Saturday (May 15th). Are there other people who would like to help out and mentor newcomers make their first contribution to SciPy? > > Note that there are also Americas and Asia-Pacific 4 hour time slots, see the link above. > > Cheers, > Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue May 11 16:22:00 2021 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 11 May 2021 22:22:00 +0200 Subject: [SciPy-Dev] SciPy virtual sprint @ PyCon In-Reply-To: References: Message-ID: On Tue, May 11, 2021 at 10:11 PM Pamphile Roy wrote: > Hey Ralf, > > Sounds great! I can help if you have a role for me :) > Great, thanks! I'll forward you the registration details. Cheers, Ralf > Cheers, > Pamphile > > On 11 May 2021, at 22:05, Ralf Gommers wrote: > > ? > Hi all, > > After the NumPy sprint last Sunday, which was fun and successful, I got an > invite to participate during the PyCon mentored sprints the coming weekend > (see https://us.pycon.org/2021/summits/mentored-sprints/). They are run > on the same infrastructure (Discord), it works quite well. > > There are multiple projects participating, I signed up for one sprint > "table" for SciPy - noon to 4pm CEST this Saturday (May 15th). Are there > other people who would like to help out and mentor newcomers make their > first contribution to SciPy? > > Note that there are also Americas and Asia-Pacific 4 hour time slots, see > the link above. > > Cheers, > Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mhaberla at calpoly.edu Tue May 11 17:26:50 2021 From: mhaberla at calpoly.edu (Matt Haberland) Date: Tue, 11 May 2021 14:26:50 -0700 Subject: [SciPy-Dev] SciPy virtual sprint @ PyCon In-Reply-To: References: Message-ID: Warren and I can do the 9 a.m. Pacific / 12 p.m. Eastern on Saturday. Should we sign up separately? Do you expect a lot of people will want to use Gitpod, or should we plan on helping them set up a local development environment? On Tue, May 11, 2021 at 1:22 PM Ralf Gommers wrote: > > > On Tue, May 11, 2021 at 10:11 PM Pamphile Roy > wrote: > >> Hey Ralf, >> >> Sounds great! I can help if you have a role for me :) >> > > Great, thanks! I'll forward you the registration details. > > Cheers, > Ralf > > > >> Cheers, >> Pamphile >> >> On 11 May 2021, at 22:05, Ralf Gommers wrote: >> >> ? >> Hi all, >> >> After the NumPy sprint last Sunday, which was fun and successful, I got >> an invite to participate during the PyCon mentored sprints the coming >> weekend (see https://us.pycon.org/2021/summits/mentored-sprints/). They >> are run on the same infrastructure (Discord), it works quite well. >> >> There are multiple projects participating, I signed up for one sprint >> "table" for SciPy - noon to 4pm CEST this Saturday (May 15th). Are there >> other people who would like to help out and mentor newcomers make their >> first contribution to SciPy? >> >> Note that there are also Americas and Asia-Pacific 4 hour time slots, see >> the link above. >> >> Cheers, >> Ralf >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -- Matt Haberland Assistant Professor BioResource and Agricultural Engineering 08A-3K, Cal Poly -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue May 11 18:05:44 2021 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 12 May 2021 00:05:44 +0200 Subject: [SciPy-Dev] SciPy virtual sprint @ PyCon In-Reply-To: References: Message-ID: On Tue, May 11, 2021 at 11:27 PM Matt Haberland wrote: > Warren and I can do the 9 a.m. Pacific / 12 p.m. Eastern on Saturday. > Should we sign up separately? > Great! Yes, please sign up through the "mentor form" at https://us.pycon.org/2021/summits/mentored-sprints/ (you can but both of your names in at once). > Do you expect a lot of people will want to use Gitpod, or should we plan > on helping them set up a local development environment? > For the NumPy sprint we found that the majority of new contributors are interested to learn how to do a local build. However, quite a few get stuck (and that'll be worse for SciPy) and then Gitpod is super useful as a way to get them coding. I think with the easy conda-forge environment setup plus Gitpod, the remote setup went smoother than the average sprint at previous SciPy conferences. Cheers, Ralf > On Tue, May 11, 2021 at 1:22 PM Ralf Gommers > wrote: > >> >> >> On Tue, May 11, 2021 at 10:11 PM Pamphile Roy >> wrote: >> >>> Hey Ralf, >>> >>> Sounds great! I can help if you have a role for me :) >>> >> >> Great, thanks! I'll forward you the registration details. >> >> Cheers, >> Ralf >> >> >> >>> Cheers, >>> Pamphile >>> >>> On 11 May 2021, at 22:05, Ralf Gommers wrote: >>> >>> ? >>> Hi all, >>> >>> After the NumPy sprint last Sunday, which was fun and successful, I got >>> an invite to participate during the PyCon mentored sprints the coming >>> weekend (see https://us.pycon.org/2021/summits/mentored-sprints/). They >>> are run on the same infrastructure (Discord), it works quite well. >>> >>> There are multiple projects participating, I signed up for one sprint >>> "table" for SciPy - noon to 4pm CEST this Saturday (May 15th). Are there >>> other people who would like to help out and mentor newcomers make their >>> first contribution to SciPy? >>> >>> Note that there are also Americas and Asia-Pacific 4 hour time slots, >>> see the link above. >>> >>> Cheers, >>> Ralf >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > > > -- > Matt Haberland > Assistant Professor > BioResource and Agricultural Engineering > 08A-3K, Cal Poly > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Wed May 12 03:49:30 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Wed, 12 May 2021 09:49:30 +0200 Subject: [SciPy-Dev] Custom __repr__? In-Reply-To: References: Message-ID: <2F843165-0299-4866-A626-100F60A3C711@gmail.com> > On 06.05.2021, at 18:03, Robert Kern wrote: > > On Thu, May 6, 2021 at 11:54 AM Pamphile Roy > wrote: >> The user doesn't do this in each notebook. You do it once inside of an IPython extension, and the user just %load_ext's the extension. >> >> I think this is the kind of thing I'd like to review where it's opt-in at first. An IPython extension is a good way to do that. Then maybe after we get some experience using it for a while, then decide to make it always-on by modifying the objects themselves. > > Ah so you propose I do an extension instead? Sounds reasonable indeed. I will see if I can get some time for that ;) > > Yeah, I think it gives you more freedom to play, people can use it with older versions of scipy, and we can use it for real work for a while before we commit to supporting it forever inside scipy itself. I created a repo as suggested by Robert: https://github.com/tupui/scipy-magic For now there are 2 simple extensions to visualize a KDE and any frozen distribution. Any help is always appreciated. Also the name is just an idea and can change ;) Cheers, Pamphile -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Wed May 12 04:00:04 2021 From: andyfaff at gmail.com (Andrew Nelson) Date: Wed, 12 May 2021 18:00:04 +1000 Subject: [SciPy-Dev] Custom __repr__? In-Reply-To: <2F843165-0299-4866-A626-100F60A3C711@gmail.com> References: <2F843165-0299-4866-A626-100F60A3C711@gmail.com> Message-ID: What would be nice would be to have a proper __repr__ that would allow objects to be recreated. I would find both the following useful in serialisation: ``` from scipy.stats import norm # serialise rv_gen a = norm b = eval(repr(norm) np.testing.assert_equal(a.pdf(0.0), b.pdf(0.0)) # serialise rv_frozen c = norm() d = eval(repr(norm())) np.testing.assert_equal(a.pdf(0.0), b.pdf(0.0)) ``` Of course this gets a bit more interesting when there are shape parameters, etc, but it would still be v. useful. -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Wed May 12 04:03:00 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Wed, 12 May 2021 10:03:00 +0200 Subject: [SciPy-Dev] Custom __repr__? In-Reply-To: References: <2F843165-0299-4866-A626-100F60A3C711@gmail.com> Message-ID: <04051E1B-41CC-42B7-8464-8A7B99047FDF@gmail.com> > On 12.05.2021, at 10:00, Andrew Nelson wrote: > > What would be nice would be to have a proper __repr__ that would allow objects to be recreated. I would find both the following useful in serialisation: > > ``` > from scipy.stats import norm > # serialise rv_gen > a = norm > b = eval(repr(norm) > np.testing.assert_equal(a.pdf(0.0), b.pdf(0.0)) > > # serialise rv_frozen > c = norm() > d = eval(repr(norm())) > np.testing.assert_equal(a.pdf(0.0), b.pdf(0.0)) > ``` > > Of course this gets a bit more interesting when there are shape parameters, etc, but it would still be v. useful. Could you create an issue maybe? Better to keep track of everything on the repo directly. From roy.pamphile at gmail.com Wed May 12 09:44:07 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Wed, 12 May 2021 15:44:07 +0200 Subject: [SciPy-Dev] Custom __repr__? In-Reply-To: <04051E1B-41CC-42B7-8464-8A7B99047FDF@gmail.com> References: <2F843165-0299-4866-A626-100F60A3C711@gmail.com> <04051E1B-41CC-42B7-8464-8A7B99047FDF@gmail.com> Message-ID: <1DF7A1FC-662D-4527-B00F-D6FBD8671343@gmail.com> > On 12.05.2021, at 10:03, Pamphile Roy wrote: > >> >> On 12.05.2021, at 10:00, Andrew Nelson wrote: >> >> What would be nice would be to have a proper __repr__ that would allow objects to be recreated. I would find both the following useful in serialisation: >> >> ``` >> from scipy.stats import norm >> # serialise rv_gen >> a = norm >> b = eval(repr(norm) >> np.testing.assert_equal(a.pdf(0.0), b.pdf(0.0)) >> >> # serialise rv_frozen >> c = norm() >> d = eval(repr(norm())) >> np.testing.assert_equal(a.pdf(0.0), b.pdf(0.0)) >> ``` >> >> Of course this gets a bit more interesting when there are shape parameters, etc, but it would still be v. useful. > > Could you create an issue maybe? Better to keep track of everything on the repo directly. In case you need this now, here is a simple way: import numpy.testing as npt from scipy.stats import beta dist = beta(2, 6, loc=4) dist2 = eval(f"{dist.dist.name}(*{dist.args}, **{dist.kwds})") npt.assert_equal(dist.pdf(0.0), dist2.pdf(0.0)) Cheers, Pamphile -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Wed May 12 10:03:37 2021 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Wed, 12 May 2021 10:03:37 -0400 Subject: [SciPy-Dev] Importing numpy as np in examples in docstrings Message-ID: Hey all, The question of including `import numpy as np` in the examples in docstrings has come up in some recent pull requests, and there is an issue for it at https://github.com/scipy/scipy/issues/13049. For many years, the policy has been to not include the import. The NumPy docstring standard (https://numpydoc.readthedocs.io/en/latest/format.html#examples) says "The examples may assume that `import numpy as np` is executed before the example code in numpy." Note that it says "*may* assume"--it does not say that the import must not be included. The question is whether the SciPy policy should (continue to) be "The import *must* not be included." If you would like to provide input, head over to the github issue and let us know what you think. Warren From hans.dembinski at gmail.com Thu May 13 03:29:38 2021 From: hans.dembinski at gmail.com (Hans Dembinski) Date: Thu, 13 May 2021 09:29:38 +0200 Subject: [SciPy-Dev] Custom __repr__? In-Reply-To: References: <2F843165-0299-4866-A626-100F60A3C711@gmail.com> Message-ID: <45BDA906-0CD4-4B97-86AD-7A9940AB6F4D@gmail.com> > On 12. May 2021, at 10:00, Andrew Nelson wrote: > > What would be nice would be to have a proper __repr__ that would allow objects to be recreated. I would find both the following useful in serialisation: [...] Having a proper __repr__ that allows to recreate objects is nice, but serialization seems to be a misuse of __repr__. We have the __getstate__ __setstate__ mechanism in Python for serialization. Could you explain your use case a bit more for my curiosity? From roy.pamphile at gmail.com Thu May 13 08:17:31 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Thu, 13 May 2021 14:17:31 +0200 Subject: [SciPy-Dev] SciPy virtual sprint @ PyCon In-Reply-To: References: Message-ID: Hi everyone, As you?ve read, we will be participating to the virtual sprint @ PyCon. For this experience to be more effective, could you help us identify approachable issues? Ralf created the label `sprint-pycon`. If you cannot add a label yourself, you can mention me on the issue and I will do it. Thank you in advance for your help! Cheers, Pamphile P.S. for the mentors, I will send you another email later today. > On 12.05.2021, at 00:05, Ralf Gommers wrote: > > > > On Tue, May 11, 2021 at 11:27 PM Matt Haberland > wrote: > Warren and I can do the 9 a.m. Pacific / 12 p.m. Eastern on Saturday. Should we sign up separately? > > Great! Yes, please sign up through the "mentor form" at https://us.pycon.org/2021/summits/mentored-sprints/ (you can but both of your names in at once). > > Do you expect a lot of people will want to use Gitpod, or should we plan on helping them set up a local development environment? > > For the NumPy sprint we found that the majority of new contributors are interested to learn how to do a local build. However, quite a few get stuck (and that'll be worse for SciPy) and then Gitpod is super useful as a way to get them coding. > > I think with the easy conda-forge environment setup plus Gitpod, the remote setup went smoother than the average sprint at previous SciPy conferences. > > Cheers, > Ralf > > > > On Tue, May 11, 2021 at 1:22 PM Ralf Gommers > wrote: > > > On Tue, May 11, 2021 at 10:11 PM Pamphile Roy > wrote: > Hey Ralf, > > Sounds great! I can help if you have a role for me :) > > Great, thanks! I'll forward you the registration details. > > Cheers, > Ralf > > > > Cheers, > Pamphile > >> On 11 May 2021, at 22:05, Ralf Gommers > wrote: >> >> ? >> Hi all, >> >> After the NumPy sprint last Sunday, which was fun and successful, I got an invite to participate during the PyCon mentored sprints the coming weekend (see https://us.pycon.org/2021/summits/mentored-sprints/ ). They are run on the same infrastructure (Discord), it works quite well. >> >> There are multiple projects participating, I signed up for one sprint "table" for SciPy - noon to 4pm CEST this Saturday (May 15th). Are there other people who would like to help out and mentor newcomers make their first contribution to SciPy? >> >> Note that there are also Americas and Asia-Pacific 4 hour time slots, see the link above. >> >> Cheers, >> Ralf >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > > -- > Matt Haberland > Assistant Professor > BioResource and Agricultural Engineering > 08A-3K, Cal Poly > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Thu May 13 21:02:06 2021 From: andyfaff at gmail.com (Andrew Nelson) Date: Fri, 14 May 2021 11:02:06 +1000 Subject: [SciPy-Dev] Custom __repr__? In-Reply-To: <45BDA906-0CD4-4B97-86AD-7A9940AB6F4D@gmail.com> References: <2F843165-0299-4866-A626-100F60A3C711@gmail.com> <45BDA906-0CD4-4B97-86AD-7A9940AB6F4D@gmail.com> Message-ID: > > Having a proper __repr__ that allows to recreate objects is nice, but > serialization seems to be a misuse of __repr__. We have the __getstate__ > __setstate__ mechanism in Python for serialization. Could you explain your > use case a bit more for my curiosity? > It's the recreation of objects that I wanted to have. My use case is for a piece of code that's running in a PyQt5 GUI to output a standalone Python script that can then be executed separately (on a cluster). The script has to recreate an Object that has a few layers of complexity (Objective --> Model --> Structure --> Layers --> Parameters --> Parameter --> Bounds). By implementing a __repr__ for all of those it allows me to call `repr(Objective)` to create the script. The Bounds are more than likely a uniform distribution, which I've implemented myself for speed, but they can also be any scipy.stats.rv_continuous distribution. At the moment I can't recreate rv_continuous from their __repr__. I suppose I could create a script and associated pickle file, with the script loading the pickle file. This will be less robust/reproducible, both laterally and vertically (laterally = use at same time, but on different machine, with roughly similar Python environment; vertically = use N years later). The main issue with vertically is that new attributes may get added to classes which means that unpickling issues often occur because the pickle doesn't possess the new attributes. Writing out classes with a __repr__ doesn't have that issue because they're created from scratch. There is a slightly different aspect to this kind of issue which is also important to me, saving the state of a GUI program. At the moment I save the state of the GUI into a pickle, but that's not necessarily readable in later versions of the program because of the forwards compatibility of the pickle against updated classes. I'd be interested in knowing how to serialise these kinds of complex structures into something that's e.g. json/text file based. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu May 13 21:39:52 2021 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 13 May 2021 21:39:52 -0400 Subject: [SciPy-Dev] Custom __repr__? In-Reply-To: References: <2F843165-0299-4866-A626-100F60A3C711@gmail.com> <45BDA906-0CD4-4B97-86AD-7A9940AB6F4D@gmail.com> Message-ID: On Thu, May 13, 2021 at 9:03 PM Andrew Nelson wrote: > Having a proper __repr__ that allows to recreate objects is nice, but >> serialization seems to be a misuse of __repr__. We have the __getstate__ >> __setstate__ mechanism in Python for serialization. Could you explain your >> use case a bit more for my curiosity? >> > > It's the recreation of objects that I wanted to have. My use case is for a > piece of code that's running in a PyQt5 GUI to output a standalone Python > script that can then be executed separately (on a cluster). The script has > to recreate an Object that has a few layers of complexity (Objective --> > Model --> Structure --> Layers --> Parameters --> Parameter --> Bounds). By > implementing a __repr__ for all of those it allows me to call > `repr(Objective)` to create the script. The Bounds are more than likely a > uniform distribution, which I've implemented myself for speed, but they can > also be any scipy.stats.rv_continuous distribution. At the moment I can't > recreate rv_continuous from their __repr__. > You'll run into more such objects. Having a round-trippable `__repr__()` isn't all that common. Even `ndarray` will trip you up with its summarization. And that's not even considering what kind of imports you need to have done in order for even the eval()able representations to actually evaluate. You will probably need to incorporate a subsystem in your code generation to get eval()able string representations of the objects you care about. You can often bound the effort a bit more than what the class author would need to do since you likely only deal with a subset of functionality (do you need eval()able datetime64 arrays? Probably not. Do you need array parameter arguments to the distributions or just scalars? Maybe). You'll need that to make sure the imports are there in any case. I suppose I could create a script and associated pickle file, with the > script loading the pickle file. This will be less robust/reproducible, both > laterally and vertically (laterally = use at same time, but on different > machine, with roughly similar Python environment; vertically = use N years > later). The main issue with vertically is that new attributes may get added > to classes which means that unpickling issues often occur because the > pickle doesn't possess the new attributes. Writing out classes with a > __repr__ doesn't have that issue because they're created from scratch. > > There is a slightly different aspect to this kind of issue which is also > important to me, saving the state of a GUI program. At the moment I save > the state of the GUI into a pickle, but that's not necessarily readable in > later versions of the program because of the forwards compatibility of the > pickle against updated classes. I'd be interested in knowing how to > serialise these kinds of complex structures into something that's e.g. > json/text file based. > Having done this many times in many ways, there's no easy road for this. You can either have a system that can serialize any object and be subject to the forward compatibility problem, or you can have a system that has to know about every object and give it information about where to put the serialized data from outdated classes. Moving from pickle to JSON or other format is irrelevant: those are just file formats. Any serialization system that can just take any Python object is also going to record the same kind of "fragile" information that is going to go out of date as you change the classes. You can build a serialization system that manages the update information with any format, even pickle. Building that off of pickle has a lot of advantages as the __getstate__()/__setstate__() machinery is standard; handling changes of the class itself like the one you describe above just takes the author treating the serialization as part of its API. The only thing you have to handle in your serialization system, per se, is dealing with class renamings/refactorings. The pickle system doesn't necessarily make that *easy*, but it's not terrible. Moving to a JSON file format doesn't really solve the problem; it just makes you rewrite all that machinery from scratch so you have to solve it at the same time as you're rewriting everything else. The JSON (or HDF5 or whatever) file format may have other benefits, like communication with non-Python programs. The "opportunity" for writing everything from scratch will make you take serialization more seriously and not make changes that would muck things up by accident. I have done it all the different ways. They are all painful. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Thu May 13 22:28:55 2021 From: andyfaff at gmail.com (Andrew Nelson) Date: Fri, 14 May 2021 12:28:55 +1000 Subject: [SciPy-Dev] Custom __repr__? In-Reply-To: References: <2F843165-0299-4866-A626-100F60A3C711@gmail.com> <45BDA906-0CD4-4B97-86AD-7A9940AB6F4D@gmail.com> Message-ID: On Fri, 14 May 2021 at 11:40, Robert Kern wrote: > > You'll run into more such objects. Having a round-trippable `__repr__()` > isn't all that common. Even `ndarray` will trip you up > with its summarization. And that's not even considering what kind of > imports you need to have done in order for even the eval()able > representations to actually evaluate. You will probably need to incorporate > a subsystem in your code generation to get eval()able string > representations of the objects you care about. > You can often bound the effort a bit more than what the class author would > need to do since you likely only deal with a subset of functionality (do > you need eval()able datetime64 arrays? Probably not. Do you > need array parameter arguments to the distributions or just scalars? > Maybe). You'll need that to make sure the imports are there in any case. > I have __repr__ for all the major classes I need to care about, and I know which imports I need for the code generation (they're pretty much fixed). I just use the name of the object in the __repr__ (norm), rather than the name of the object with the full import path (scipy.stats.norm). In my case I do need to use 'simple' numpy arrays in the code generation, but they're directly loaded from datafiles rather than having a __repr__. I don't need datetime64 arrays. My requirements would be to have a __repr__ for frozen rv_continuous distributions with scalar shape parameters. I hadn't appreciated before that rv_continuous could be frozen with scale/loc/shape parameters that were array rather than scalar. I agree that it's a major hassle to get a __repr__ of a numpy array. Perhaps Pamphiles suggestion of ``` f"{dist.dist.name}(*{dist.args!r}, **{dist.kwds!r})" ``` would work as a good general __repr__, with more specific implementations as required. This would have the specific proviso that roundtripping would work for scalar cases, but not for array scale/loc/shape cases. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bussonniermatthias at gmail.com Thu May 13 22:39:32 2021 From: bussonniermatthias at gmail.com (Matthias Bussonnier) Date: Thu, 13 May 2021 19:39:32 -0700 Subject: [SciPy-Dev] Custom __repr__? In-Reply-To: References: <2F843165-0299-4866-A626-100F60A3C711@gmail.com> <45BDA906-0CD4-4B97-86AD-7A9940AB6F4D@gmail.com> Message-ID: To be fair, and not taking any sides, the idea of having __repr__ be eval round-tripable is mentioned in CPython docs: from https://docs.python.org/3/reference/datamodel.html object.__repr__(self) Called by the repr() built-in function to compute the ?official? string representation of an object. If at all possible, this should look like a valid Python expression that could be used to recreate an object with the same value (given an appropriate environment). If this is not possible, [...] object.__str__(self) Called by str(object) and the built-in functions format() and print() to compute the ?informal? or nicely printable string representation of an object. [...] And yes IPython/Jupyter uses __repr__ and _repr_*_ for non-text format, maybe we should have use __str__ and _str_*_ ... but that seem too late to change now. -- M On Thu, 13 May 2021 at 19:29, Andrew Nelson wrote: > > On Fri, 14 May 2021 at 11:40, Robert Kern wrote: >> >> >> You'll run into more such objects. Having a round-trippable `__repr__()` isn't all that common. Even `ndarray` will trip you up with its summarization. And that's not even considering what kind of imports you need to have done in order for even the eval()able representations to actually evaluate. You will probably need to incorporate a subsystem in your code generation to get eval()able string representations of the objects you care about. >> >> You can often bound the effort a bit more than what the class author would need to do since you likely only deal with a subset of functionality (do you need eval()able datetime64 arrays? Probably not. Do you need array parameter arguments to the distributions or just scalars? Maybe). You'll need that to make sure the imports are there in any case. > > > I have __repr__ for all the major classes I need to care about, and I know which imports I need for the code generation (they're pretty much fixed). I just use the name of the object in the __repr__ (norm), rather than the name of the object with the full import path (scipy.stats.norm). In my case I do need to use 'simple' numpy arrays in the code generation, but they're directly loaded from datafiles rather than having a __repr__. I don't need datetime64 arrays. > > My requirements would be to have a __repr__ for frozen rv_continuous distributions with scalar shape parameters. I hadn't appreciated before that rv_continuous could be frozen with scale/loc/shape parameters that were array rather than scalar. I agree that it's a major hassle to get a __repr__ of a numpy array. Perhaps Pamphiles suggestion of > ``` > f"{dist.dist.name}(*{dist.args!r}, **{dist.kwds!r})" > ``` > > would work as a good general __repr__, with more specific implementations as required. This would have the specific proviso that roundtripping would work for scalar cases, but not for array scale/loc/shape cases. > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev From robert.kern at gmail.com Thu May 13 22:41:34 2021 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 13 May 2021 22:41:34 -0400 Subject: [SciPy-Dev] Custom __repr__? In-Reply-To: References: <2F843165-0299-4866-A626-100F60A3C711@gmail.com> <45BDA906-0CD4-4B97-86AD-7A9940AB6F4D@gmail.com> Message-ID: On Thu, May 13, 2021 at 10:29 PM Andrew Nelson wrote: > On Fri, 14 May 2021 at 11:40, Robert Kern wrote: > >> >> You'll run into more such objects. Having a round-trippable `__repr__()` >> isn't all that common. Even `ndarray` will trip you up >> with its summarization. And that's not even considering what kind of >> imports you need to have done in order for even the eval()able >> representations to actually evaluate. You will probably need to incorporate >> a subsystem in your code generation to get eval()able string >> representations of the objects you care about. >> > You can often bound the effort a bit more than what the class author would >> need to do since you likely only deal with a subset of functionality (do >> you need eval()able datetime64 arrays? Probably not. Do you >> need array parameter arguments to the distributions or just scalars? >> Maybe). You'll need that to make sure the imports are there in any case. >> > > I have __repr__ for all the major classes I need to care about, and I know > which imports I need for the code generation (they're pretty much fixed). I > just use the name of the object in the __repr__ (norm), rather than the > name of the object with the full import path (scipy.stats.norm). In my case > I do need to use 'simple' numpy arrays in the code generation, but they're > directly loaded from datafiles rather than having a __repr__. I don't need > datetime64 arrays. > The remark about datetime64 arrays was rhetorical. :-) I was just demonstrating that you are in a position to make simplifying assumptions that we often can't make. It's a good idea for you to own your code generation in any case. You can write what you need in there without needing our __repr__ to do anything. > My requirements would be to have a __repr__ for frozen rv_continuous > distributions with scalar shape parameters. I hadn't appreciated before > that rv_continuous could be frozen with scale/loc/shape parameters that > were array rather than scalar. I agree that it's a major hassle to get a > __repr__ of a numpy array. Perhaps Pamphiles suggestion of > ``` > f"{dist.dist.name}(*{dist.args!r}, **{dist.kwds!r})" > ``` > > would work as a good general __repr__, with more specific implementations > as required. This would have the specific proviso that roundtripping would > work for scalar cases, but not for array scale/loc/shape cases. > That would work for something like quick-and-dirty code generation, but it will look awful on the prompt, and that's where I would want our use cases geared towards. To me, the main use case for a (roughly) eval()able __repr__ is to allow users to copy-paste an Out[] cell for use elsewhere, not for mechanical eval()ing. It's not hard to fix to assemble the args and kwds into what users would naturally type; it's just not a one-liner. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Thu May 13 22:57:24 2021 From: andyfaff at gmail.com (Andrew Nelson) Date: Fri, 14 May 2021 12:57:24 +1000 Subject: [SciPy-Dev] Custom __repr__? In-Reply-To: References: <2F843165-0299-4866-A626-100F60A3C711@gmail.com> <45BDA906-0CD4-4B97-86AD-7A9940AB6F4D@gmail.com> Message-ID: On Fri, 14 May 2021 at 12:42, Robert Kern wrote: > > That would work for something like quick-and-dirty code generation, but it > will look awful on the prompt, and that's where I would want our use cases > geared towards. To me, the main use case for a (roughly) > eval()able __repr__ is to allow users to copy-paste an Out[] cell for use > elsewhere, not for mechanical eval()ing. It's not hard to fix to assemble > the args and kwds into what users would naturally type; it's just not a > one-liner. > Implementing __str__ and __repr__ would alleviate some awfulness (so long as one used `print(obj)`). One could argue that the default position is not all that useful anyway: ``` >>> norm(scale=[1,2], loc=[3, 4]) ``` because indicating that it's an rv_frozen, we don't know anything else about it. Rough __repr__ to allow copy-paste may be good enough, in most cases that is also sufficient to mechanical eval(). Perhaps there would have to be a no-promises, best effort, you get what you get approach, to the output. The entire prefix would have to be used to help with the import: ``` >>> from scipy.stats import norm >>> repr(norm(scale=1, loc=2)) scipy.stats.norm(scale=1, loc=2) ``` -------------- next part -------------- An HTML attachment was scrubbed... URL: From hans.dembinski at gmail.com Fri May 14 05:34:22 2021 From: hans.dembinski at gmail.com (Hans Dembinski) Date: Fri, 14 May 2021 11:34:22 +0200 Subject: [SciPy-Dev] Custom __repr__? In-Reply-To: References: <2F843165-0299-4866-A626-100F60A3C711@gmail.com> <45BDA906-0CD4-4B97-86AD-7A9940AB6F4D@gmail.com> Message-ID: <3671D661-567A-41D1-A98A-ECDF47E81442@gmail.com> > On 14. May 2021, at 04:57, Andrew Nelson wrote: > > ``` > >>> from scipy.stats import norm > >>> repr(norm(scale=1, loc=2)) > scipy.stats.norm(scale=1, loc=2) > ``` +1 on this, it is much more user friendly. Moreover the original repr spills implementation details that a user probably should not see. From tyler.je.reddy at gmail.com Sat May 15 20:26:46 2021 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Sat, 15 May 2021 18:26:46 -0600 Subject: [SciPy-Dev] Proposed 1.7.0 Release Schedule In-Reply-To: References: Message-ID: Hi, A few PRs that might benefit from some extra attention: - a stats PR with a tricky debate related to the "nan" policy: https://github.com/scipy/scipy/pull/13572#discussion_r625334934 - a set of alternate PRs that have been described as one of the bigger pain points for optimize: https://github.com/scipy/scipy/pull/13143#issuecomment-841737552 Best wishes, Tyler On Mon, 3 May 2021 at 04:43, Ilhan Polat wrote: > I would also appreciate if other win users can give it a go. It might be > the case that my system is the exception so we can ignore my issue. > > On Sun, 2 May 2021, 21:08 Ralf Gommers, wrote: > >> >> >> On Sun, May 2, 2021 at 7:38 PM Tyler Reddy >> wrote: >> >>> Hi, >>> >>> SciPy 1.6.0 was released December 31/2020 (a bit late), a little more >>> than 4 months ago, and I think we'd like to keep a roughly biannual release >>> cadence. >>> >>> I'd like to proposed the following schedule for 1.7.0: >>> - May 26/2021: branch 1.7.x >>> - May 29/2021: rc1 >>> - June 11/2021: rc2 (if needed) >>> - June 20/2021: final release >>> >> >> Sounds good to me. >> >> As always, it is a good idea to start tagging things that should be in >>> the 1.7.0 release & please do help with reviewing PRs/issues that are >>> tagged--current counts are: >>> >>> - PRs: 24 open with 1.7.0 milestone >>> - issues: 8 open with 1.7.0 milestone >>> >>> Also great to update the release notes wiki as appropriate: >>> https://github.com/scipy/scipy/wiki/Release-note-entries-for-SciPy-1.7.0 >>> >>> Thoughts/objections for the proposed schedule? The new build time >>> dependency/transpiler, Pythran, would be a notable addition to this >>> release, assuming that moves forward. >>> >> >> There's a couple of nice PRs pending that depend on it with only a pure >> Python fallback. While I'm pretty happy with how the use of Pythran is >> shaping up, I think we should leave it as an optional build dependency only >> for this release, and try to make it non-optional afterwards. There's >> always the chance we'll find something unexpected in the release, and >> there's Windows build thing that needs sorting out - it works for all >> wheels, but Ilhan is having some trouble with a local build setup. >> >> Cheers, >> Ralf >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyler.je.reddy at gmail.com Sat May 15 20:47:37 2021 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Sat, 15 May 2021 18:47:37 -0600 Subject: [SciPy-Dev] Proposed 1.7.0 Release Schedule In-Reply-To: References: Message-ID: There's also a request for core devs to "tie break" a decision here: https://github.com/scipy/scipy/pull/11263#issuecomment-840265287 On Sat, 15 May 2021 at 18:26, Tyler Reddy wrote: > Hi, > > A few PRs that might benefit from some extra attention: > > - a stats PR with a tricky debate related to the "nan" policy: > https://github.com/scipy/scipy/pull/13572#discussion_r625334934 > - a set of alternate PRs that have been described as one of the bigger > pain points for optimize: > https://github.com/scipy/scipy/pull/13143#issuecomment-841737552 > > Best wishes, > Tyler > > On Mon, 3 May 2021 at 04:43, Ilhan Polat wrote: > >> I would also appreciate if other win users can give it a go. It might be >> the case that my system is the exception so we can ignore my issue. >> >> On Sun, 2 May 2021, 21:08 Ralf Gommers, wrote: >> >>> >>> >>> On Sun, May 2, 2021 at 7:38 PM Tyler Reddy >>> wrote: >>> >>>> Hi, >>>> >>>> SciPy 1.6.0 was released December 31/2020 (a bit late), a little more >>>> than 4 months ago, and I think we'd like to keep a roughly biannual release >>>> cadence. >>>> >>>> I'd like to proposed the following schedule for 1.7.0: >>>> - May 26/2021: branch 1.7.x >>>> - May 29/2021: rc1 >>>> - June 11/2021: rc2 (if needed) >>>> - June 20/2021: final release >>>> >>> >>> Sounds good to me. >>> >>> As always, it is a good idea to start tagging things that should be in >>>> the 1.7.0 release & please do help with reviewing PRs/issues that are >>>> tagged--current counts are: >>>> >>>> - PRs: 24 open with 1.7.0 milestone >>>> - issues: 8 open with 1.7.0 milestone >>>> >>>> Also great to update the release notes wiki as appropriate: >>>> https://github.com/scipy/scipy/wiki/Release-note-entries-for-SciPy-1.7.0 >>>> >>>> Thoughts/objections for the proposed schedule? The new build time >>>> dependency/transpiler, Pythran, would be a notable addition to this >>>> release, assuming that moves forward. >>>> >>> >>> There's a couple of nice PRs pending that depend on it with only a pure >>> Python fallback. While I'm pretty happy with how the use of Pythran is >>> shaping up, I think we should leave it as an optional build dependency only >>> for this release, and try to make it non-optional afterwards. There's >>> always the chance we'll find something unexpected in the release, and >>> there's Windows build thing that needs sorting out - it works for all >>> wheels, but Ilhan is having some trouble with a local build setup. >>> >>> Cheers, >>> Ralf >>> >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Sun May 16 08:14:15 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Sun, 16 May 2021 14:14:15 +0200 Subject: [SciPy-Dev] Proposed 1.7.0 Release Schedule In-Reply-To: References: Message-ID: <043BDEF9-8DA3-4FB1-B134-11F804776119@gmail.com> Thanks Tyler. Regarding Optimize. It seems that lots of PRs and issues are pointing towards an overhaul of the module. My position would be to solve what can be solved for the release with open PRs and for the next release we should lay out a plan to evolve the module. What do you think Andrew? Could we plan a specific meeting for that maybe? Cheers, Pamphile > On 16 May 2021, at 02:48, Tyler Reddy wrote: > > ? > There's also a request for core devs to "tie break" a decision here: https://github.com/scipy/scipy/pull/11263#issuecomment-840265287 > >> On Sat, 15 May 2021 at 18:26, Tyler Reddy wrote: >> Hi, >> >> A few PRs that might benefit from some extra attention: >> >> - a stats PR with a tricky debate related to the "nan" policy: https://github.com/scipy/scipy/pull/13572#discussion_r625334934 >> - a set of alternate PRs that have been described as one of the bigger pain points for optimize: https://github.com/scipy/scipy/pull/13143#issuecomment-841737552 >> >> Best wishes, >> Tyler >> >>> On Mon, 3 May 2021 at 04:43, Ilhan Polat wrote: >>> I would also appreciate if other win users can give it a go. It might be the case that my system is the exception so we can ignore my issue. >>> >>>> On Sun, 2 May 2021, 21:08 Ralf Gommers, wrote: >>>> >>>> >>>>> On Sun, May 2, 2021 at 7:38 PM Tyler Reddy wrote: >>>>> Hi, >>>>> >>>>> SciPy 1.6.0 was released December 31/2020 (a bit late), a little more than 4 months ago, and I think we'd like to keep a roughly biannual release cadence. >>>>> >>>>> I'd like to proposed the following schedule for 1.7.0: >>>>> - May 26/2021: branch 1.7.x >>>>> - May 29/2021: rc1 >>>>> - June 11/2021: rc2 (if needed) >>>>> - June 20/2021: final release >>>> >>>> Sounds good to me. >>>> >>>>> As always, it is a good idea to start tagging things that should be in the 1.7.0 release & please do help with reviewing PRs/issues that are tagged--current counts are: >>>>> >>>>> - PRs: 24 open with 1.7.0 milestone >>>>> - issues: 8 open with 1.7.0 milestone >>>>> >>>>> Also great to update the release notes wiki as appropriate: https://github.com/scipy/scipy/wiki/Release-note-entries-for-SciPy-1.7.0 >>>>> >>>>> Thoughts/objections for the proposed schedule? The new build time dependency/transpiler, Pythran, would be a notable addition to this release, assuming that moves forward. >>>> >>>> There's a couple of nice PRs pending that depend on it with only a pure Python fallback. While I'm pretty happy with how the use of Pythran is shaping up, I think we should leave it as an optional build dependency only for this release, and try to make it non-optional afterwards. There's always the chance we'll find something unexpected in the release, and there's Windows build thing that needs sorting out - it works for all wheels, but Ilhan is having some trouble with a local build setup. >>>> >>>> Cheers, >>>> Ralf >>>> >>>> >>>> _______________________________________________ >>>> SciPy-Dev mailing list >>>> SciPy-Dev at python.org >>>> https://mail.python.org/mailman/listinfo/scipy-dev >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Sun May 16 21:01:55 2021 From: andyfaff at gmail.com (Andrew Nelson) Date: Mon, 17 May 2021 11:01:55 +1000 Subject: [SciPy-Dev] Proposed 1.7.0 Release Schedule In-Reply-To: <043BDEF9-8DA3-4FB1-B134-11F804776119@gmail.com> References: <043BDEF9-8DA3-4FB1-B134-11F804776119@gmail.com> Message-ID: On Sun, 16 May 2021 at 22:14, Pamphile Roy wrote: > Regarding Optimize. It seems that lots of PRs and issues are pointing > towards an overhaul of the module. My position would be to solve what can > be solved for the release with open PRs and for the next release we should > lay out a plan to evolve the module. > It'd certainly be good for the number of active optimize maintainers to increase, to tackle PRs and issues, etc. optimize is probably one of the most heavily used parts of scipy, that's why it gets a lot of traffic on github. I think the module is probably in not too bad a good shape. A review of outstanding issues would give an idea of what types of issues are commonly experienced, it's not clear that a large scale redesign would necessarily change anything. One thing I'm mainly interested in minimisations of scalar functions. I had a desire to implement a class-based minimization system for those, but I'm fairly convinced it would have to be developed outside scipy in a spin-off project first. > What do you think Andrew? Could we plan a specific meeting for that maybe? > > Cheers, > Pamphile > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amulyabitspilani at gmail.com Mon May 17 13:58:21 2021 From: amulyabitspilani at gmail.com (Amulya Gupta) Date: Mon, 17 May 2021 23:28:21 +0530 Subject: [SciPy-Dev] Regarding contribution in Scipy organization for GSoC 21 In-Reply-To: References: Message-ID: Dear Ralf, I am Amulya Gupta. I had applied for GSOC,21 project on *Add typed annotations of a submodule *project, It has come to my notice that I was not accepted for this project. Can you please take time out of your busy schedule and help me in finding out my shortcomings. Regards Amulya Gupta On Thu, Mar 25, 2021 at 11:48 PM Amulya Gupta wrote: > Please can you ignore the above mail > Regards > AMULYA > > On Thu, Mar 25, 2021 at 11:22 PM Amulya Gupta > wrote: > >> Greetings Ralf, >> In the project idea mentions *Add type annotation of a submodule*. Can >> you please tell which submodule I will be adding the annotation. >> Regards >> AMULYA GUPTA >> >> On Sun, Mar 21, 2021 at 2:10 AM Ralf Gommers >> wrote: >> >>> >>> >>> On Sat, Mar 20, 2021 at 5:35 PM Amulya Gupta >>> wrote: >>> >>>> My intentions were not that. I just inquired that you need any pre-plan >>>> for the proposal. I have started my early proposal and will submit as soon >>>> as possible. >>>> >>> >>> That sounds good, thanks Amulya. >>> >>> Cheers, >>> Ralf >>> >>> >>> Regards >>>> AMULYA GUPTA >>>> >>>> On Sat, Mar 20, 2021 at 8:56 PM Ralf Gommers >>>> wrote: >>>> >>>>> Hi Amulya, >>>>> >>>>> >>>>> On Sat, Mar 20, 2021 at 4:09 PM Amulya Gupta < >>>>> amulyabitspilani at gmail.com> wrote: >>>>> >>>>>> Dear Sir/Ma'am, >>>>>> I am Amulya Gupta, a sophomore from BITS,Pilani. I have been coding >>>>>> for the past 3 years. I am profound in Python(Modules: nose test, poetry, >>>>>> numpy, matplotlib, cmake, pipenv, tkinter, turtle and currently learning >>>>>> mypy etc), C++, Mysql, Java script. I have experience in open source with >>>>>> an internship with an organisation called NumFocus. I want to openly >>>>>> contribute to the project "*Add type annotations to a submodule*" >>>>>> from SciPy. Please can I talk to the mentor about his expectations >>>>>> from the student applying. How many more students have applied for the same >>>>>> project and how does the mentor has pre-planned about the project >>>>>> and his inputs for the proposal buildup. >>>>>> >>>>> >>>>> I have replied to you already. We are happy to help, but do expect >>>>> that you read the information we have put together on the ideas page ( >>>>> https://github.com/scipy/scipy/wiki/GSoC-2021-project-ideas) and try >>>>> to understand what is needed. If you have concrete questions or an early >>>>> draft of a proposal we can give you feedback. But we won't draft the >>>>> proposal for you - there are lots of examples, from the PSF template to the >>>>> guidance in >>>>> https://google.github.io/gsocguides/student/writing-a-proposal.html >>>>> >>>>> Cheers, >>>>> Ralf >>>>> >>>>> _______________________________________________ >>>>> SciPy-Dev mailing list >>>>> SciPy-Dev at python.org >>>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>>> >>>> _______________________________________________ >>>> SciPy-Dev mailing list >>>> SciPy-Dev at python.org >>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Mon May 17 14:06:20 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Mon, 17 May 2021 20:06:20 +0200 Subject: [SciPy-Dev] SciPy Community Meetups are back! In-Reply-To: References: <1605BE4D-E642-4CD7-848A-5FA3BBE56982@gmail.com> <222F4B45-F11E-4142-9BD6-36AEC27D9A29@gmail.com> <151F12E8-BC7B-40D9-A27D-462FF3567873@gmail.com> Message-ID: <600FCAD2-05F9-45A8-8130-8A2C0EBABA9C@gmail.com> Thanks everyone for today?s meetup. It was nice to finally meet you all and I found it quite productive! Thank you Matt for taking notes during the meeting (same link): https://hackmd.io/@tupui/scipy-meetup/edit For those of you who could not join, we also recorded the session (thank you Ralf) and we will see to make it available. We proposed to have regular meetups every 2 weeks and change the time zone every meetup (EU/Africa/Americas and Pacific/Asia/EU). So that anyone could join once a month. The first thing to decide, in these respective meetings, would be to pick a day and time not to have to do a pool all the time. Until then, the proposed new meetups dates would be Pacific/Asia/EU: https://doodle.com/poll/tmnenfc9c9v6c42z EU/Africa/Americas: https://doodle.com/poll/9cse6mt58cp9d6px The next meetup notes are here: https://hackmd.io/@tupui/scipy-meetup-2/edit Thank you for your help and participation. And if there is any question or issue, feel free to drop me a line. Cheers, Pamphile > On 08.05.2021, at 10:45, Pamphile Roy wrote: > > Hi everyone, > > I forgot to include the password, here is the complete link: > > https://zoom.us/j/93827717491?pwd=UGdkMzlnWEFjbDNQUmxpRkp6L0VLdz09 > > Meeting ID: 938 2771 7491 > Passcode: 602749 > > Also, thank you to Ralf for his help and providing us with this link! > > Cheers, > Pamphile > > > >> On 07.05.2021, at 23:17, Pamphile Roy wrote: >> >> Hi everyone, >> >> First of all, thank you for your participation to the pool! >> >> You have been the most to vote for May 17th at 16 CET. >> >> https://zoom.us/j/93827717491 >> >> Please feel free to contribute to the agenda: >> >> https://hackmd.io/@tupui/scipy-meetup/edit >> >> In case you cannot join us, I am thinking about recording this event. >> Of course, before any recording starts, we will discuss about it. >> >> I will see you there, and I am looking forward to finally meet you all. >> In the meantime, don?t hesitate to get in touch with me =) >> >> Cheers, >> Pamphile >> >> >>> On 04.05.2021, at 12:21, Pamphile Roy wrote: >>> >>> I updated the doodle to propose 7a.m. CET. >>> >>> Cheers, >>> Pamphile >>> >>>> On 04.05.2021, at 12:08, Pamphile Roy wrote: >>>> >>>> I am not sure if we can solve this. From the table below, the only solution I could see would be 6 or 7a.m. CET. >>>> >>>> I can add other times or I could propose that we rotate the time from one meeting to the other (might be the most reasonable). >>>> >>>> Cheers, >>>> Pamphile >>>> >>>> >>>> >>>> >>>> >>>> >>>>> On 04.05.2021, at 11:56, Andrew Nelson wrote: >>>>> >>>>> All those times are pretty much impossible for those in AUS/NZ. >>>>> >>>>> On Tue, 4 May 2021, 19:53 Pamphile Roy, wrote: >>>>> Hi everyone, >>>>> >>>>> This is a friendly reminder to complete the doodle to plan the meetup. >>>>> >>>>> If not otherwise stated, I am planning on fixing the date by the end of this week (May 7). >>>>> >>>>>> https://doodle.com/poll/fd62d6755texie5s >>>>> >>>>> Thank you all for your help and engagement. >>>>> >>>>> Cheers, >>>>> Pamphile >>>>> >>>>> >>>>>> On 30.04.2021, at 11:48, Pamphile Roy wrote: >>>>>> >>>>>> Hi everyone, >>>>>> >>>>>> I would like to propose regular SciPy Community meetings again! >>>>>> >>>>>> https://doodle.com/poll/fd62d6755texie5s >>>>>> >>>>>> For this first meeting, it would be good if as many people as possible could join as we would decide things like frequency of such meeting. >>>>>> >>>>>> Everyone is invited and encouraged to >>>>>> join in and edit the work-in-progress meeting topics and notes at: >>>>>> >>>>>> https://hackmd.io/@tupui/scipy-meetup/edit >>>>>> >>>>>> Cheers, >>>>>> Pamphile >>>>>> >>>>>> PS. share the news: https://twitter.com/PamphileRoy/status/1388067651721830401?s=20 >>>>> >>>>> _______________________________________________ >>>>> SciPy-Dev mailing list >>>>> SciPy-Dev at python.org >>>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>>> _______________________________________________ >>>>> SciPy-Dev mailing list >>>>> SciPy-Dev at python.org >>>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon May 17 16:42:59 2021 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 17 May 2021 22:42:59 +0200 Subject: [SciPy-Dev] welcome Xingyu and Tirth as GSoC students! Message-ID: Hi all, Google just announced the accepted GSoC students for this year. Congratulations to Xingyu Liu and Tirth Patel! Xingyu's project is titled "Improve performance through the use of Pythran", her project summary can be found at https://summerofcode.withgoogle.com/projects/#5526713011798016. The mentors are Serge Guelton and myself. Tirth's project is titled "Integrate library UNU.RAN into scipy.stats", his project summary can be found at https://summerofcode.withgoogle.com/projects/#5912428874825728. The mentors are Christoph Baumgarten and Nicholas McKibben. Both projects are exciting, and I'm looking forward to Tirth and Xingyu's work and interaction with the community. >From today till June 7th is "community bonding period". Xingyu and Tirth have already been contributing for a little while, but please do give them a hand or say hi! That we had the first community meeting in a long time today and plan to have regular community calls going forward is also nice, should help with the bonding:) For the full GSoC timeline, please see https://summerofcode.withgoogle.com/how-it-works/#timeline. There are also a number of other students who worked hard on their proposals, and we had more students we wished we could have accepted if we had had more slots. Thanks to those students for all that hard work and creativity! I hope that you are still interested in contributing to SciPy. And for some of you, there is hopefully a chance to participate next year. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilhanpolat at gmail.com Mon May 17 16:50:48 2021 From: ilhanpolat at gmail.com (Ilhan Polat) Date: Mon, 17 May 2021 22:50:48 +0200 Subject: [SciPy-Dev] welcome Xingyu and Tirth as GSoC students! In-Reply-To: References: Message-ID: Yay! Welcome on board both. Please reach out for anything that raises some questions, would be happy to help. On Mon, May 17, 2021 at 10:44 PM Ralf Gommers wrote: > Hi all, > > Google just announced the accepted GSoC students for this year. > Congratulations to Xingyu Liu and Tirth Patel! > > Xingyu's project is titled "Improve performance through the use of > Pythran", her project summary can be found at > https://summerofcode.withgoogle.com/projects/#5526713011798016. The > mentors are Serge Guelton and myself. > > Tirth's project is titled "Integrate library UNU.RAN into scipy.stats", > his project summary can be found at > https://summerofcode.withgoogle.com/projects/#5912428874825728. The > mentors are Christoph Baumgarten and Nicholas McKibben. > > Both projects are exciting, and I'm looking forward to Tirth and Xingyu's > work and interaction with the community. > > From today till June 7th is "community bonding period". Xingyu and Tirth > have already been contributing for a little while, but please do give them > a hand or say hi! That we had the first community meeting in a long time > today and plan to have regular community calls going forward is also nice, > should help with the bonding:) > > For the full GSoC timeline, please see > https://summerofcode.withgoogle.com/how-it-works/#timeline. > > There are also a number of other students who worked hard on their > proposals, and we had more students we wished we could have accepted if we > had had more slots. Thanks to those students for all that hard work and > creativity! I hope that you are still interested in contributing to SciPy. > And for some of you, there is hopefully a chance to participate next year. > > Cheers, > Ralf > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Mon May 17 16:57:13 2021 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Mon, 17 May 2021 13:57:13 -0700 Subject: [SciPy-Dev] welcome Xingyu and Tirth as GSoC students! In-Reply-To: References: Message-ID: On Mon, May 17, 2021, at 13:42, Ralf Gommers wrote: > Xingyu's project is titled "Improve performance through the use of Pythran", her project summary can be found at https://summerofcode.withgoogle.com/projects/#5526713011798016. The mentors are Serge Guelton and myself. > > Tirth's project is titled "Integrate library UNU.RAN into scipy.stats", his project summary can be found at https://summerofcode.withgoogle.com/projects/#5912428874825728. The mentors are Christoph Baumgarten and Nicholas McKibben. Congratulations, Xingyu and Tirth! These sound like useful projects, and it will be fun to watch them evolve. Tirth, I notice that UNU.RAN is GPL, so I'm curious how that is going to work with SciPy. Xingyu, in the recent RBF PR by Trever Hines I was impressed to see how tidy Pythran code can be; so I am happy that we'll have more of that in SciPy. Welcome on board! St?fan -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon May 17 17:28:59 2021 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 17 May 2021 17:28:59 -0400 Subject: [SciPy-Dev] welcome Xingyu and Tirth as GSoC students! In-Reply-To: References: Message-ID: On Mon, May 17, 2021 at 4:58 PM Stefan van der Walt wrote: > Tirth, I notice that UNU.RAN is GPL, so I'm curious how that is going to > work with SciPy. > They have relicensed it just for us. https://mail.python.org/pipermail/scipy-dev/2021-March/024641.html -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Mon May 17 17:32:32 2021 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Mon, 17 May 2021 17:32:32 -0400 Subject: [SciPy-Dev] welcome Xingyu and Tirth as GSoC students! In-Reply-To: References: Message-ID: On 5/17/21, Ralf Gommers wrote: > Hi all, > > Google just announced the accepted GSoC students for this year. > Congratulations to Xingyu Liu and Tirth Patel! > > Xingyu's project is titled "Improve performance through the use of > Pythran", her project summary can be found at > https://summerofcode.withgoogle.com/projects/#5526713011798016. The mentors > are Serge Guelton and myself. > > Tirth's project is titled "Integrate library UNU.RAN into scipy.stats", his > project summary can be found at > https://summerofcode.withgoogle.com/projects/#5912428874825728. The mentors > are Christoph Baumgarten and Nicholas McKibben. > > Both projects are exciting, and I'm looking forward to Tirth and Xingyu's > work and interaction with the community. > > From today till June 7th is "community bonding period". Xingyu and Tirth > have already been contributing for a little while, but please do give them > a hand or say hi! That we had the first community meeting in a long time > today and plan to have regular community calls going forward is also nice, > should help with the bonding:) > > For the full GSoC timeline, please see > https://summerofcode.withgoogle.com/how-it-works/#timeline. > > There are also a number of other students who worked hard on their > proposals, and we had more students we wished we could have accepted if we > had had more slots. Thanks to those students for all that hard work and > creativity! I hope that you are still interested in contributing to SciPy. > And for some of you, there is hopefully a chance to participate next year. > > Cheers, > Ralf > This is great news. Congratulations Xingyu and Tirth! Warren From stefanv at berkeley.edu Mon May 17 17:36:13 2021 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Mon, 17 May 2021 14:36:13 -0700 Subject: [SciPy-Dev] welcome Xingyu and Tirth as GSoC students! In-Reply-To: References: Message-ID: <44f6d152-fc1d-4e40-8b5a-b4dac2a6a7c2@www.fastmail.com> On Mon, May 17, 2021, at 14:28, Robert Kern wrote: > On Mon, May 17, 2021 at 4:58 PM Stefan van der Walt wrote: >> Tirth, I notice that UNU.RAN is GPL, so I'm curious how that is going to work with SciPy. > > They have relicensed it just for us. > > https://mail.python.org/pipermail/scipy-dev/2021-March/024641.html Great, thanks Robert. Christoph, can we replicate that email somewhere public, like on an issue? St?fan -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Mon May 17 17:37:59 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Mon, 17 May 2021 23:37:59 +0200 Subject: [SciPy-Dev] welcome Xingyu and Tirth as GSoC students! In-Reply-To: References: Message-ID: <47A4B85D-93EF-42D3-B2FA-61BB798B5263@gmail.com> Congratulations Xingyu and Tirth! Both projects are interesting and I am really looking forward. Cheers, Pamphile > On 17 May 2021, at 23:33, Warren Weckesser wrote: > > Congratulations Xingyu and Tirth! From roy.pamphile at gmail.com Mon May 17 17:39:44 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Mon, 17 May 2021 23:39:44 +0200 Subject: [SciPy-Dev] welcome Xingyu and Tirth as GSoC students! In-Reply-To: <44f6d152-fc1d-4e40-8b5a-b4dac2a6a7c2@www.fastmail.com> References: <44f6d152-fc1d-4e40-8b5a-b4dac2a6a7c2@www.fastmail.com> Message-ID: <730EEB64-56A8-4F89-A386-EC3867011204@gmail.com> Yes we can extract the raw email and put it on an issue. I did this recently for some StackOverflow code (sieve primes in QMC). > On 17 May 2021, at 23:37, Stefan van der Walt wrote: > > ? >> On Mon, May 17, 2021, at 14:28, Robert Kern wrote: >> On Mon, May 17, 2021 at 4:58 PM Stefan van der Walt wrote: >> Tirth, I notice that UNU.RAN is GPL, so I'm curious how that is going to work with SciPy. >> >> They have relicensed it just for us. >> >> https://mail.python.org/pipermail/scipy-dev/2021-March/024641.html > > Great, thanks Robert. Christoph, can we replicate that email somewhere public, like on an issue? > > St?fan > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominicchm at gmail.com Mon May 17 21:36:58 2021 From: dominicchm at gmail.com (Dominic C.) Date: Mon, 17 May 2021 18:36:58 -0700 Subject: [SciPy-Dev] ENH: (Stats) Studentized Range Distribution Message-ID: Hi all, We've implemented the studentized range distribution to support the upcoming addition of Tukey's HSD test. We used Cython and SciPy's LowLevelCallables to speed up the integration of its CDF, PDF, and moments, and the integrands are logarithmized to make it robust to inputs over several orders of magnitude. Accuracy is tested against tabulated values, R's ptukey, and a slow but accurate implementation in mpmath. If you're interested, please come take a look at https://github.com/scipy/scipy/pull/13732. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicholas.bgp at gmail.com Mon May 17 23:47:08 2021 From: nicholas.bgp at gmail.com (Nicholas McKibben) Date: Mon, 17 May 2021 20:47:08 -0700 Subject: [SciPy-Dev] Marginals/sensitivity analysis for linprog HiGHS Message-ID: Hi all, gh-13893 is under review and provides sensitivity analysis for the SciPy HiGHS LP solvers. For each of the three constraint types -- equality constraints, inequality constraints, and lower/upper bounds -- we want to convey three pieces of information: marginals (also commonly referred to as "Lagrange multipliers", "dual values", or "shadow prices"), ranging information (to be added in a future PR), as well as some measure of the distance between the solution and the constraints. Many LP solvers return this information but unfortunately there is little agreement in the field on how it should be labeled or presented. This comment gives a brief overview of the different conventions other popular LP solvers follow. For SciPy, we have settled on organizing the OptimizeResult object in the following way according to the right-hand diagram in this file . Specifically: - "slack" and "con" fields are doc-deprecated - new nested OptimizeResults objects "ineqlin" (corresponding to inequality constraints), "eqlin" (equality constraints), "lower", and "upper" (bound constraints) are added, containing: - "ineqlin": "slack", "marginals" - "eqlin": "violation", "marginals" - "lower": "freedom", "marginals" - "upper": "freedom", "marginals" Notes: - "marginals" is chosen over e.g. "shadow price" to avoid jargon that is associated with a particular field - "slack" is obvious - "freedom" was suggested by Julian Hall, creator and maintainer of HiGHS - "violation", "eqlin", "ineqlin" are inspired by Matlab's result interface - the new nested OptimizeResult objects will also house future ranging sensitivity information - this functionality is only available for the HiGHS solvers, i.e. method='highs', 'highs-ds', or 'highs-ipm' Please join us in the discussion on the PR or send feedback here. Thanks, Nicholas -------------- next part -------------- An HTML attachment was scrubbed... URL: From tirthasheshpatel at gmail.com Mon May 17 23:55:44 2021 From: tirthasheshpatel at gmail.com (Tirth Patel) Date: Tue, 18 May 2021 09:25:44 +0530 Subject: [SciPy-Dev] welcome Xingyu and Tirth as GSoC students! In-Reply-To: <47A4B85D-93EF-42D3-B2FA-61BB798B5263@gmail.com> References: <47A4B85D-93EF-42D3-B2FA-61BB798B5263@gmail.com> Message-ID: Hi everyone, I received the GSoC acceptance mail a few hours ago! Thanks for giving me this opportunity. And thanks to everyone for the warm welcome. I really hope to deliver something helpful to SciPy this summer! Kind Regards, Tirth Patel On Tue, May 18, 2021 at 3:08 AM Pamphile Roy wrote: > Congratulations Xingyu and Tirth! > > Both projects are interesting and I am really looking forward. > > Cheers, > Pamphile > > > On 17 May 2021, at 23:33, Warren Weckesser > wrote: > > > > Congratulations Xingyu and Tirth! > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xingyuliu at g.harvard.edu Tue May 18 00:40:31 2021 From: xingyuliu at g.harvard.edu (Xingyu Liu) Date: Tue, 18 May 2021 04:40:31 +0000 Subject: [SciPy-Dev] welcome Xingyu and Tirth as GSoC students! In-Reply-To: References: <47A4B85D-93EF-42D3-B2FA-61BB798B5263@gmail.com>, Message-ID: Hi everyone, Thank you so much for giving me such a great opportunity and thanks for the warm welcome!!! I?m looking forward to working with you and contributing to SciPy this summer! Thanks, Xingyu ________________________________ From: SciPy-Dev on behalf of Tirth Patel Sent: Tuesday, May 18, 2021 11:55:44 AM To: SciPy Developers List Subject: Re: [SciPy-Dev] welcome Xingyu and Tirth as GSoC students! Hi everyone, I received the GSoC acceptance mail a few hours ago! Thanks for giving me this opportunity. And thanks to everyone for the warm welcome. I really hope to deliver something helpful to SciPy this summer! Kind Regards, Tirth Patel On Tue, May 18, 2021 at 3:08 AM Pamphile Roy > wrote: Congratulations Xingyu and Tirth! Both projects are interesting and I am really looking forward. Cheers, Pamphile > On 17 May 2021, at 23:33, Warren Weckesser > wrote: > > Congratulations Xingyu and Tirth! _______________________________________________ SciPy-Dev mailing list SciPy-Dev at python.org https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From serge.guelton at telecom-bretagne.eu Tue May 18 01:52:03 2021 From: serge.guelton at telecom-bretagne.eu (Serge Guelton) Date: Tue, 18 May 2021 07:52:03 +0200 Subject: [SciPy-Dev] welcome Xingyu and Tirth as GSoC students! In-Reply-To: References: <47A4B85D-93EF-42D3-B2FA-61BB798B5263@gmail.com> Message-ID: <20210518055203.GA1803042@sguelton.remote.csb> On Tue, May 18, 2021 at 04:40:31AM +0000, Xingyu Liu wrote: > > Hi everyone, > > > Thank you so much for giving me such a great opportunity and thanks for the > warm welcome!!! I?m looking forward to working with you and contributing to > SciPy this summer! The pleasure is mutual, welcome! From ralf.gommers at gmail.com Tue May 18 14:59:15 2021 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 18 May 2021 20:59:15 +0200 Subject: [SciPy-Dev] SciPy Community Meetups are back! In-Reply-To: <600FCAD2-05F9-45A8-8130-8A2C0EBABA9C@gmail.com> References: <1605BE4D-E642-4CD7-848A-5FA3BBE56982@gmail.com> <222F4B45-F11E-4142-9BD6-36AEC27D9A29@gmail.com> <151F12E8-BC7B-40D9-A27D-462FF3567873@gmail.com> <600FCAD2-05F9-45A8-8130-8A2C0EBABA9C@gmail.com> Message-ID: On Mon, May 17, 2021 at 8:06 PM Pamphile Roy wrote: > Thanks everyone for today?s meetup. > It was nice to finally meet you all and I found it quite productive! > +1, great to see so many people > Thank you Matt for taking notes during the meeting (same link): > https://hackmd.io/@tupui/scipy-meetup/edit > For those of you who could not join, we also recorded the session (thank > you Ralf) and we will see to make it available. > One unfortunate thing with the recording is that I had my screen in "gallery view" so we could capture all ~15 participants, but Zoom decided to record only the talking person each time. Which is a little unfortunate, especially since we started recording only after the round of introductions. I'll try to figure out how to improve that for next time. Audio: https://www.dropbox.com/s/fo2dxo8eh3ccwml/SciPy_community_meeting__audio_2021_05_17.m4a?dl=0 Video: https://www.dropbox.com/s/16vd1fq35x186f1/SciPy_community_meeting_video_2021_05_17.mp4?dl=0 Cheers, Ralf > We proposed to have regular meetups every 2 weeks and change the time zone > every meetup (EU/Africa/Americas and Pacific/Asia/EU). > So that anyone could join once a month. The first thing to decide, in > these respective meetings, would be to pick a day and time not to have to > do a pool all the time. > > Until then, the proposed new meetups dates would be > > > 1. Pacific/Asia/EU: https://doodle.com/poll/tmnenfc9c9v6c42z > 2. EU/Africa/Americas: https://doodle.com/poll/9cse6mt58cp9d6px > > > The next meetup notes are here: > https://hackmd.io/@tupui/scipy-meetup-2/edit > > Thank you for your help and participation. And if there is any question or > issue, feel free to drop me a line. > > Cheers, > Pamphile > > > On 08.05.2021, at 10:45, Pamphile Roy wrote: > > Hi everyone, > > I forgot to include the password, here is the complete link: > > https://zoom.us/j/93827717491?pwd=UGdkMzlnWEFjbDNQUmxpRkp6L0VLdz09 > > Meeting ID: 938 2771 7491 > Passcode: 602749 > > Also, thank you to Ralf for his help and providing us with this link! > > Cheers, > Pamphile > > > > On 07.05.2021, at 23:17, Pamphile Roy wrote: > > Hi everyone, > > First of all, thank you for your participation to the pool! > > You have been the most to vote for May 17th at 16 CET. > > https://zoom.us/j/93827717491 > > Please feel free to contribute to the agenda: > > https://hackmd.io/@tupui/scipy-meetup/edit > > In case you cannot join us, I am thinking about recording this event. > Of course, before any recording starts, we will discuss about it. > > I will see you there, and I am looking forward to finally meet you all. > In the meantime, don?t hesitate to get in touch with me =) > > Cheers, > Pamphile > > > On 04.05.2021, at 12:21, Pamphile Roy wrote: > > I updated the doodle to propose 7a.m. CET. > > Cheers, > Pamphile > > On 04.05.2021, at 12:08, Pamphile Roy wrote: > > I am not sure if we can solve this. From the table below, the only > solution I could see would be 6 or 7a.m. CET. > > I can add other times or I could propose that we rotate the time from one > meeting to the other (might be the most reasonable). > > Cheers, > Pamphile > > > > > > > On 04.05.2021, at 11:56, Andrew Nelson wrote: > > All those times are pretty much impossible for those in AUS/NZ. > > On Tue, 4 May 2021, 19:53 Pamphile Roy, wrote: > Hi everyone, > > This is a friendly reminder to complete the doodle to plan the meetup. > > If not otherwise stated, I am planning on fixing the date by the end of > this week (May 7). > > https://doodle.com/poll/fd62d6755texie5s > > > Thank you all for your help and engagement. > > Cheers, > Pamphile > > > On 30.04.2021, at 11:48, Pamphile Roy wrote: > > Hi everyone, > > I would like to propose regular SciPy Community meetings again! > > https://doodle.com/poll/fd62d6755texie5s > > For this first meeting, it would be good if as many people as possible > could join as we would decide things like frequency of such meeting. > > Everyone is invited and encouraged to > join in and edit the work-in-progress meeting topics and notes at: > > https://hackmd.io/@tupui/scipy-meetup/edit > > Cheers, > Pamphile > > PS. share the news: > https://twitter.com/PamphileRoy/status/1388067651721830401?s=20 > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > > > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Thu May 20 05:40:28 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Thu, 20 May 2021 11:40:28 +0200 Subject: [SciPy-Dev] Unpacking the mailing list: Discourse, Slack = Mailing list Message-ID: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> Hi everyone, We had a chat Ralf, St?fan and I about moving away from the mailing list. The idea would be to move over Discourse under the umbrella of https://discuss.scientific-python.org St?fan proposes to do the migration work and I can also provide some help if needed (the complete history would be imported). I also propose to help administrate/moderate this space. As Serge expressed during the meetup, we should be careful when moving to Discourse as some people might be reluctant to come. Which would fragment the community. Hence, we should force the move by sunsetting the mail list. On the same topic, I would discuss the possibility to use Slack (or another platform for real-time chat). This has proven to be helpful for NumPy and other projects and we should also see the benefits. Let us know what you think. Cheers, Pamphile -------------- next part -------------- An HTML attachment was scrubbed... URL: From jacobr at ethz.ch Thu May 20 06:00:26 2021 From: jacobr at ethz.ch (Romain Jacob) Date: Thu, 20 May 2021 12:00:26 +0200 Subject: [SciPy-Dev] Unpacking the mailing list: Discourse, Slack = Mailing list In-Reply-To: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> References: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> Message-ID: <0f8b67c1-160f-220e-c13d-a172762d4e88@ethz.ch> On 20/05/2021 11:40, Pamphile Roy wrote: > Let us know what you think. Personally, I'm all for less emails, in general. Discourse and Slack are both widely-used tools, so I don't think it creates a real hurdle to the engagement of community members. That being said, I'm myself very much inactive, so my opinion shouldn't weight much :-) Cheers, -- Romain Jacob Postdoctoral Researcher ETH Zurich - Networked Systems Group (NSG) www.romainjacob.net @RJacobPartner Gloriastrasse 35, ETZ G78 8092 Zurich +41 7 68 16 88 22 -------------- next part -------------- An HTML attachment was scrubbed... URL: From deak.andris at gmail.com Thu May 20 06:20:02 2021 From: deak.andris at gmail.com (Andras Deak) Date: Thu, 20 May 2021 12:20:02 +0200 Subject: [SciPy-Dev] Unpacking the mailing list: Discourse, Slack = Mailing list In-Reply-To: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> References: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> Message-ID: On Thu, May 20, 2021 at 11:41 AM Pamphile Roy wrote: > > Hi everyone, > > We had a chat Ralf, St?fan and I about moving away from the mailing list. > > The idea would be to move over Discourse under the umbrella of > > https://discuss.scientific-python.org > > St?fan proposes to do the migration work and I can also provide some help if needed (the complete history would be imported). > I also propose to help administrate/moderate this space. > > As Serge expressed during the meetup, we should be careful when moving to Discourse as some people > might be reluctant to come. Which would fragment the community. Hence, we should force the move by sunsetting the mail list. > > On the same topic, I would discuss the possibility to use Slack (or another platform for real-time chat). > This has proven to be helpful for NumPy and other projects and we should also see the benefits. Hi Pamphile, Do you happen to have know if there might be any accessibilty issues with discourse, slack and the like? I have both accessibility in the traditional sense in mind, as well as things like governments that censor internet access. I have no idea about either of these things, and for all I know the alternatives might be perfect replacements, so this is just an aspect that should also be considered for such a switch :) Andr?s > Let us know what you think. > > Cheers, > Pamphile > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev From roy.pamphile at gmail.com Thu May 20 06:37:38 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Thu, 20 May 2021 12:37:38 +0200 Subject: [SciPy-Dev] Unpacking the mailing list: Discourse, Slack = Mailing list In-Reply-To: References: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> Message-ID: <98E2AB36-236B-4318-82DD-3451FC651814@gmail.com> Hi Andr?s, > On 20.05.2021, at 12:20, Andras Deak wrote: > > Do you happen to have know if there might be any accessibilty issues > with discourse, slack and the like? I have both accessibility in the > traditional sense in mind, as well as things like governments that > censor internet access. I have no idea about either of these things, > and for all I know the alternatives might be perfect replacements, so > this is just an aspect that should also be considered for such a > switch :) Interesting question. My understanding is that both platforms have accessibility policy or are looking at it: https://slack.com/intl/en-at/accessibility-plan https://meta.discourse.org/tag/accessibility As for being blocked, Slack is not accessible (or inconsistently) in some countries. Discourse seems ok and depends on your IP and your content. Cheers, Pamphile -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Thu May 20 06:53:51 2021 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 20 May 2021 11:53:51 +0100 Subject: [SciPy-Dev] Unpacking the mailing list: Discourse, Slack = Mailing list In-Reply-To: <98E2AB36-236B-4318-82DD-3451FC651814@gmail.com> References: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> <98E2AB36-236B-4318-82DD-3451FC651814@gmail.com> Message-ID: Hi, I have strong preference for being able to avoid Slack: https://blogs.sap.com/2018/10/26/why-i-dont-like-slack-anymore-for-public-use/ Cheers, Matthew On Thu, May 20, 2021 at 11:38 AM Pamphile Roy wrote: > > Hi Andr?s, > > On 20.05.2021, at 12:20, Andras Deak wrote: > > Do you happen to have know if there might be any accessibilty issues > with discourse, slack and the like? I have both accessibility in the > traditional sense in mind, as well as things like governments that > censor internet access. I have no idea about either of these things, > and for all I know the alternatives might be perfect replacements, so > this is just an aspect that should also be considered for such a > switch :) > > > Interesting question. My understanding is that both platforms have accessibility policy or are looking at it: > > https://slack.com/intl/en-at/accessibility-plan > https://meta.discourse.org/tag/accessibility > > As for being blocked, Slack is not accessible (or inconsistently) in some countries. > Discourse seems ok and depends on your IP and your content. > > Cheers, > Pamphile > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev From larson.eric.d at gmail.com Thu May 20 07:19:31 2021 From: larson.eric.d at gmail.com (Eric Larson) Date: Thu, 20 May 2021 07:19:31 -0400 Subject: [SciPy-Dev] Unpacking the mailing list: Discourse, Slack = Mailing list In-Reply-To: References: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> <98E2AB36-236B-4318-82DD-3451FC651814@gmail.com> Message-ID: About 6 months ago we transitioned MNE-Python from a traditional GNU Mailman mailing list (almost 15 years old) to Discourse , including porting the old mailing list posts so they are directly searchable there . It has been a pretty smooth transition and the reply rate has gone up -- I think the forum setting is much more inviting for most people, and the threading is better than what you get from slack or gitter. My 2c, Eric On Thu, May 20, 2021 at 6:54 AM Matthew Brett wrote: > Hi, > > I have strong preference for being able to avoid Slack: > > > https://blogs.sap.com/2018/10/26/why-i-dont-like-slack-anymore-for-public-use/ > > Cheers, > > Matthew > > On Thu, May 20, 2021 at 11:38 AM Pamphile Roy > wrote: > > > > Hi Andr?s, > > > > On 20.05.2021, at 12:20, Andras Deak wrote: > > > > Do you happen to have know if there might be any accessibilty issues > > with discourse, slack and the like? I have both accessibility in the > > traditional sense in mind, as well as things like governments that > > censor internet access. I have no idea about either of these things, > > and for all I know the alternatives might be perfect replacements, so > > this is just an aspect that should also be considered for such a > > switch :) > > > > > > Interesting question. My understanding is that both platforms have > accessibility policy or are looking at it: > > > > https://slack.com/intl/en-at/accessibility-plan > > https://meta.discourse.org/tag/accessibility > > > > As for being blocked, Slack is not accessible (or inconsistently) in > some countries. > > Discourse seems ok and depends on your IP and your content. > > > > Cheers, > > Pamphile > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at python.org > > https://mail.python.org/mailman/listinfo/scipy-dev > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From einstein.edison at gmail.com Thu May 20 07:26:21 2021 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Thu, 20 May 2021 13:26:21 +0200 Subject: [SciPy-Dev] Unpacking the mailing list: Discourse, Slack = Mailing list In-Reply-To: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> References: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> Message-ID: <692A182D-3AB1-4D00-BFC3-9EC213168EA8@gmail.com> Hello all, One option I haven?t seen mentioned is GitHub discussions. Is that being evaluated at all as a possibility? Best regards, Hameer Abbasi > Am 20.05.2021 um 11:40 schrieb Pamphile Roy : > > Hi everyone, > > We had a chat Ralf, St?fan and I about moving away from the mailing list. > > The idea would be to move over Discourse under the umbrella of > > https://discuss.scientific-python.org > > St?fan proposes to do the migration work and I can also provide some help if needed (the complete history would be imported). > I also propose to help administrate/moderate this space. > > As Serge expressed during the meetup, we should be careful when moving to Discourse as some people > might be reluctant to come. Which would fragment the community. Hence, we should force the move by sunsetting the mail list. > > On the same topic, I would discuss the possibility to use Slack (or another platform for real-time chat). > This has proven to be helpful for NumPy and other projects and we should also see the benefits. > > Let us know what you think. > > Cheers, > Pamphile > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From matti.picus at gmail.com Thu May 20 07:26:25 2021 From: matti.picus at gmail.com (Matti Picus) Date: Thu, 20 May 2021 14:26:25 +0300 Subject: [SciPy-Dev] Unpacking the mailing list: Discourse, Slack = Mailing list In-Reply-To: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> References: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> Message-ID: <20739a41-0d9b-2394-733b-0cb5dbd72b7c@gmail.com> On 20/5/21 12:40 pm, Pamphile Roy wrote: > Hi everyone, > > We had a chat Ralf, St?fan and I about moving away from the mailing list. > > The idea would be to move over Discourse under the umbrella of > > https://discuss.scientific-python.org > > > St?fan proposes to do the migration work and I can also provide some > help if needed (the complete history would be imported). > I also propose to help administrate/moderate this space. > > As Serge expressed during the meetup, we should be careful when moving > to Discourse as some people > might be reluctant to come. Which would fragment the community. Hence, > we should force the move by sunsetting the mail list. > > On the same topic, I would discuss the possibility to use Slack (or > another platform for real-time chat). > This has proven to be helpful for NumPy and other projects and we > should also see the benefits. > > Let us know what you think. > > Cheers, > Pamphile > Among criteria for a community forum might be: - permanence: can we find a discussion in ten years when we went to figure out what led to a particular decision - transparency: will the discussions be archived in a public manner with little need for logins or signups - searchability: is there a good way to search the discussions for relevant fragments (single words are often not enough). So for me slack is not a good option for a community forum, but it could be an additional side channel, as long as it is clear what goes where. NumPy's use of slack is limited to short informal chats between developers or chats with newcomers who might be intimidated by the permanence of the community forum. I am not sure how long discourse promises to preserve discussions. I am a dinosaur and prefer email. Discourse -> email seems to work OK for the python forums I monitor. Matti From evgeny.burovskiy at gmail.com Thu May 20 07:28:37 2021 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Thu, 20 May 2021 14:28:37 +0300 Subject: [SciPy-Dev] Unpacking the mailing list: Discourse, Slack = Mailing list In-Reply-To: References: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> <98E2AB36-236B-4318-82DD-3451FC651814@gmail.com> Message-ID: +1 for discourse -0 for sunsetting the ML -1 for slack. My 2c, Evgeni ??, 20 ??? 2021 ?., 14:19 Eric Larson : > About 6 months ago we transitioned MNE-Python from a traditional GNU > Mailman mailing list > (almost 15 > years old) to Discourse , including porting > the old mailing list posts so they are directly searchable there > . It has > been a pretty smooth transition and the reply rate has gone up -- I think > the forum setting is much more inviting for most people, and the threading > is better than what you get from slack or gitter. > > My 2c, > Eric > > > On Thu, May 20, 2021 at 6:54 AM Matthew Brett > wrote: > >> Hi, >> >> I have strong preference for being able to avoid Slack: >> >> >> https://blogs.sap.com/2018/10/26/why-i-dont-like-slack-anymore-for-public-use/ >> >> Cheers, >> >> Matthew >> >> On Thu, May 20, 2021 at 11:38 AM Pamphile Roy >> wrote: >> > >> > Hi Andr?s, >> > >> > On 20.05.2021, at 12:20, Andras Deak wrote: >> > >> > Do you happen to have know if there might be any accessibilty issues >> > with discourse, slack and the like? I have both accessibility in the >> > traditional sense in mind, as well as things like governments that >> > censor internet access. I have no idea about either of these things, >> > and for all I know the alternatives might be perfect replacements, so >> > this is just an aspect that should also be considered for such a >> > switch :) >> > >> > >> > Interesting question. My understanding is that both platforms have >> accessibility policy or are looking at it: >> > >> > https://slack.com/intl/en-at/accessibility-plan >> > https://meta.discourse.org/tag/accessibility >> > >> > As for being blocked, Slack is not accessible (or inconsistently) in >> some countries. >> > Discourse seems ok and depends on your IP and your content. >> > >> > Cheers, >> > Pamphile >> > _______________________________________________ >> > SciPy-Dev mailing list >> > SciPy-Dev at python.org >> > https://mail.python.org/mailman/listinfo/scipy-dev >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu May 20 08:11:11 2021 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 20 May 2021 14:11:11 +0200 Subject: [SciPy-Dev] Unpacking the mailing list: Discourse, Slack = Mailing list In-Reply-To: <20739a41-0d9b-2394-733b-0cb5dbd72b7c@gmail.com> References: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> <20739a41-0d9b-2394-733b-0cb5dbd72b7c@gmail.com> Message-ID: On Thu, May 20, 2021 at 1:26 PM Matti Picus wrote: > > On 20/5/21 12:40 pm, Pamphile Roy wrote: > > Hi everyone, > > > > We had a chat Ralf, St?fan and I about moving away from the mailing list. > > > > The idea would be to move over Discourse under the umbrella of > > > > https://discuss.scientific-python.org > > > > > > St?fan proposes to do the migration work and I can also provide some > > help if needed (the complete history would be imported). > > I also propose to help administrate/moderate this space. > Thanks Pamphile and St?fan for offering to do the heavy lifting here. > > > As Serge expressed during the meetup, we should be careful when moving > > to Discourse as some people > > might be reluctant to come. Which would fragment the community. Hence, > > we should force the move by sunsetting the mail list. > +1, if the decision is to move - then we should move (= migrate content, leave the mailing list behind). > > On the same topic, I would discuss the possibility to use Slack (or > > another platform for real-time chat). > > This has proven to be helpful for NumPy and other projects and we > > should also see the benefits. > > > > Let us know what you think. > > > > Cheers, > > Pamphile > > > Among criteria for a community forum might be: > > - permanence: can we find a discussion in ten years when we went to > figure out what led to a particular decision > > - transparency: will the discussions be archived in a public manner with > little need for logins or signups > > - searchability: is there a good way to search the discussions for > relevant fragments (single words are often not enough). > > > So for me slack is not a good option for a community forum, but it could > be an additional side channel, as long as it is clear what goes where. > NumPy's use of slack is limited to short informal chats between > developers or chats with newcomers who might be intimidated by the > permanence of the community forum. > +1 to this point. It's probably best to leave Slack for a separate discussion at a later time, since it serves a very different purpose from a mailing list or forum like Discourse. > > I am not sure how long discourse promises to preserve discussions. I think forever? You can always export an archive, in case the hosted Discourse ever shuts up shop. I am a dinosaur and prefer email. Discourse -> email seems to work OK for > the > python forums I monitor. > Same here. The Python packaging Discourse I follow seems to work well though, and is searchable. So while for me personally email / mailing lists works okay, I think it's more important to provide a better experience for newcomers and the generation that didn't grow up with old school mailing lists. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu May 20 09:12:57 2021 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 20 May 2021 15:12:57 +0200 Subject: [SciPy-Dev] Regarding contribution in Scipy organization for GSoC 21 In-Reply-To: References: Message-ID: On Mon, May 17, 2021 at 7:58 PM Amulya Gupta wrote: > Dear Ralf, > I am Amulya Gupta. I had applied for GSOC,21 project on *Add typed > annotations of a submodule *project, It has come to my notice that I was > not accepted for this project. Can you please take time out of your busy > schedule and help me in finding out my shortcomings. > Hi Amulya, thanks for asking. I'll follow up in private. Cheers, Ralf Regards > Amulya Gupta > > On Thu, Mar 25, 2021 at 11:48 PM Amulya Gupta > wrote: > >> Please can you ignore the above mail >> Regards >> AMULYA >> >> On Thu, Mar 25, 2021 at 11:22 PM Amulya Gupta >> wrote: >> >>> Greetings Ralf, >>> In the project idea mentions *Add type annotation of a submodule*. Can >>> you please tell which submodule I will be adding the annotation. >>> Regards >>> AMULYA GUPTA >>> >>> On Sun, Mar 21, 2021 at 2:10 AM Ralf Gommers >>> wrote: >>> >>>> >>>> >>>> On Sat, Mar 20, 2021 at 5:35 PM Amulya Gupta < >>>> amulyabitspilani at gmail.com> wrote: >>>> >>>>> My intentions were not that. I just inquired that you need any >>>>> pre-plan for the proposal. I have started my early proposal and will submit >>>>> as soon as possible. >>>>> >>>> >>>> That sounds good, thanks Amulya. >>>> >>>> Cheers, >>>> Ralf >>>> >>>> >>>> Regards >>>>> AMULYA GUPTA >>>>> >>>>> On Sat, Mar 20, 2021 at 8:56 PM Ralf Gommers >>>>> wrote: >>>>> >>>>>> Hi Amulya, >>>>>> >>>>>> >>>>>> On Sat, Mar 20, 2021 at 4:09 PM Amulya Gupta < >>>>>> amulyabitspilani at gmail.com> wrote: >>>>>> >>>>>>> Dear Sir/Ma'am, >>>>>>> I am Amulya Gupta, a sophomore from BITS,Pilani. I have been coding >>>>>>> for the past 3 years. I am profound in Python(Modules: nose test, poetry, >>>>>>> numpy, matplotlib, cmake, pipenv, tkinter, turtle and currently learning >>>>>>> mypy etc), C++, Mysql, Java script. I have experience in open source with >>>>>>> an internship with an organisation called NumFocus. I want to openly >>>>>>> contribute to the project "*Add type annotations to a submodule*" >>>>>>> from SciPy. Please can I talk to the mentor about his expectations >>>>>>> from the student applying. How many more students have applied for the same >>>>>>> project and how does the mentor has pre-planned about the project >>>>>>> and his inputs for the proposal buildup. >>>>>>> >>>>>> >>>>>> I have replied to you already. We are happy to help, but do expect >>>>>> that you read the information we have put together on the ideas page ( >>>>>> https://github.com/scipy/scipy/wiki/GSoC-2021-project-ideas) and try >>>>>> to understand what is needed. If you have concrete questions or an early >>>>>> draft of a proposal we can give you feedback. But we won't draft the >>>>>> proposal for you - there are lots of examples, from the PSF template to the >>>>>> guidance in >>>>>> https://google.github.io/gsocguides/student/writing-a-proposal.html >>>>>> >>>>>> Cheers, >>>>>> Ralf >>>>>> >>>>>> _______________________________________________ >>>>>> SciPy-Dev mailing list >>>>>> SciPy-Dev at python.org >>>>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>>>> >>>>> _______________________________________________ >>>>> SciPy-Dev mailing list >>>>> SciPy-Dev at python.org >>>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>>> >>>> _______________________________________________ >>>> SciPy-Dev mailing list >>>> SciPy-Dev at python.org >>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>> >>> _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyler.je.reddy at gmail.com Thu May 20 10:36:13 2021 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Thu, 20 May 2021 08:36:13 -0600 Subject: [SciPy-Dev] Unpacking the mailing list: Discourse, Slack = Mailing list In-Reply-To: References: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> <20739a41-0d9b-2394-733b-0cb5dbd72b7c@gmail.com> Message-ID: -1 for sunsetting the mailing list +1 for having any alternative solutions only "informal--" if people want to go there and interact with newcomers that's fine IMO, but I'd strongly prefer not to have more things/technologies/channels to monitor for decision making On Thu, 20 May 2021 at 06:11, Ralf Gommers wrote: > > > On Thu, May 20, 2021 at 1:26 PM Matti Picus wrote: > >> >> On 20/5/21 12:40 pm, Pamphile Roy wrote: >> > Hi everyone, >> > >> > We had a chat Ralf, St?fan and I about moving away from the mailing >> list. >> > >> > The idea would be to move over Discourse under the umbrella of >> > >> > https://discuss.scientific-python.org >> > >> > >> > St?fan proposes to do the migration work and I can also provide some >> > help if needed (the complete history would be imported). >> > I also propose to help administrate/moderate this space. >> > > Thanks Pamphile and St?fan for offering to do the heavy lifting here. > > > >> > As Serge expressed during the meetup, we should be careful when moving >> > to Discourse as some people >> > might be reluctant to come. Which would fragment the community. Hence, >> > we should force the move by sunsetting the mail list. >> > > +1, if the decision is to move - then we should move (= migrate content, > leave the mailing list behind). > > >> > On the same topic, I would discuss the possibility to use Slack (or >> > another platform for real-time chat). >> > This has proven to be helpful for NumPy and other projects and we >> > should also see the benefits. >> > >> > Let us know what you think. >> > >> > Cheers, >> > Pamphile >> > >> Among criteria for a community forum might be: >> >> - permanence: can we find a discussion in ten years when we went to >> figure out what led to a particular decision >> >> - transparency: will the discussions be archived in a public manner with >> little need for logins or signups >> >> - searchability: is there a good way to search the discussions for >> relevant fragments (single words are often not enough). >> >> >> So for me slack is not a good option for a community forum, but it could >> be an additional side channel, as long as it is clear what goes where. >> NumPy's use of slack is limited to short informal chats between >> developers or chats with newcomers who might be intimidated by the >> permanence of the community forum. >> > > +1 to this point. It's probably best to leave Slack for a separate > discussion at a later time, since it serves a very different purpose from a > mailing list or forum like Discourse. > > >> >> I am not sure how long discourse promises to preserve discussions. > > > I think forever? You can always export an archive, in case the hosted > Discourse ever shuts up shop. > > I am a dinosaur and prefer email. Discourse -> email seems to work OK for >> the >> python forums I monitor. >> > > Same here. The Python packaging Discourse I follow seems to work well > though, and is searchable. So while for me personally email / mailing lists > works okay, I think it's more important to provide a better experience for > newcomers and the generation that didn't grow up with old school mailing > lists. > > Cheers, > Ralf > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Thu May 20 10:45:02 2021 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 20 May 2021 15:45:02 +0100 Subject: [SciPy-Dev] Unpacking the mailing list: Discourse, Slack = Mailing list In-Reply-To: References: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> <20739a41-0d9b-2394-733b-0cb5dbd72b7c@gmail.com> Message-ID: Hi, On Thu, May 20, 2021 at 3:37 PM Tyler Reddy wrote: > > -1 for sunsetting the mailing list > +1 for having any alternative solutions only "informal--" if people want to go there and interact with newcomers that's fine IMO, but I'd strongly prefer not to have more things/technologies/channels to monitor for decision making Speaking as a dinosaur - like Matti - I personally prefer email, but I would say that, despite reservations, my experience with Discourse hasn't been too bad. You can ask it to send emails, so you don't have to monitor it separately. I have heard Eric L's comment from others, about greater traffic from Discourse than using email. When I went to check for one project, the traffic looked to be of reasonable quality. Cheers, Matthew From tyler.je.reddy at gmail.com Thu May 20 11:13:37 2021 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Thu, 20 May 2021 09:13:37 -0600 Subject: [SciPy-Dev] Unpacking the mailing list: Discourse, Slack = Mailing list In-Reply-To: References: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> <20739a41-0d9b-2394-733b-0cb5dbd72b7c@gmail.com> Message-ID: Just keep the discourse stuff for onboarding/mentoring/informal stuff and mailing list for "votes"/substantive decisions IMO. On Thu, 20 May 2021 at 08:46, Matthew Brett wrote: > Hi, > > On Thu, May 20, 2021 at 3:37 PM Tyler Reddy > wrote: > > > > -1 for sunsetting the mailing list > > +1 for having any alternative solutions only "informal--" if people want > to go there and interact with newcomers that's fine IMO, but I'd strongly > prefer not to have more things/technologies/channels to monitor for > decision making > > Speaking as a dinosaur - like Matti - I personally prefer email, but I > would say that, despite reservations, my experience with Discourse > hasn't been too bad. You can ask it to send emails, so you don't > have to monitor it separately. I have heard Eric L's comment from > others, about greater traffic from Discourse than using email. When I > went to check for one project, the traffic looked to be of reasonable > quality. > > Cheers, > > Matthew > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Thu May 20 12:11:51 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Thu, 20 May 2021 18:11:51 +0200 Subject: [SciPy-Dev] Unpacking the mailing list: Discourse, Slack = Mailing list Message-ID: <7ED40F24-7840-4B19-8C10-77C642E7F3D3@gmail.com> > On 20 May 2021, at 17:14, Tyler Reddy wrote: > ? > Just keep the discourse stuff for onboarding/mentoring/informal stuff and mailing list for "votes"/substantive decisions IMO. What you describe is not suitable for Discourse but more Slack or Discord (which is working well IMO as the sprint showed). Discourse is more a forum with topics (like a mailing list) than a medium to have active discussions. Having Discourse and the mail list would for sure fragment the community. Cheers, Pamphile From tyler.je.reddy at gmail.com Thu May 20 12:35:21 2021 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Thu, 20 May 2021 10:35:21 -0600 Subject: [SciPy-Dev] Unpacking the mailing list: Discourse, Slack = Mailing list In-Reply-To: <7ED40F24-7840-4B19-8C10-77C642E7F3D3@gmail.com> References: <7ED40F24-7840-4B19-8C10-77C642E7F3D3@gmail.com> Message-ID: Ok, let me rephrase--I think you should use whatever you want (I probably don't understand what it is, and decent chance my employer blocks it) for "other discussions" and we should keep the low traffic decision making on the mailing list. On Thu, 20 May 2021 at 10:11, Pamphile Roy wrote: > > > > On 20 May 2021, at 17:14, Tyler Reddy wrote: > > ? > > Just keep the discourse stuff for onboarding/mentoring/informal stuff > and mailing list for "votes"/substantive decisions IMO. > > What you describe is not suitable for Discourse but more Slack or Discord > (which is working well IMO as the sprint showed). > > Discourse is more a forum with topics (like a mailing list) than a medium > to have active discussions. Having Discourse and the mail list would for > sure fragment the community. > > Cheers, > Pamphile > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From evgeny.burovskiy at gmail.com Thu May 20 13:34:42 2021 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Thu, 20 May 2021 20:34:42 +0300 Subject: [SciPy-Dev] Unpacking the mailing list: Discourse, Slack = Mailing list In-Reply-To: References: <7ED40F24-7840-4B19-8C10-77C642E7F3D3@gmail.com> Message-ID: +1 what Tyler said ??, 20 ??? 2021 ?., 19:35 Tyler Reddy : > Ok, let me rephrase--I think you should use whatever you want (I probably > don't understand what it is, and decent chance my employer blocks it) for > "other discussions" and we should keep the low traffic decision making on > the mailing list. > > > On Thu, 20 May 2021 at 10:11, Pamphile Roy wrote: > >> >> >> > On 20 May 2021, at 17:14, Tyler Reddy wrote: >> > ? >> > Just keep the discourse stuff for onboarding/mentoring/informal stuff >> and mailing list for "votes"/substantive decisions IMO. >> >> What you describe is not suitable for Discourse but more Slack or Discord >> (which is working well IMO as the sprint showed). >> >> Discourse is more a forum with topics (like a mailing list) than a medium >> to have active discussions. Having Discourse and the mail list would for >> sure fragment the community. >> >> Cheers, >> Pamphile >> >> >> _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From perimosocordiae at gmail.com Thu May 20 13:49:20 2021 From: perimosocordiae at gmail.com (CJ Carey) Date: Thu, 20 May 2021 13:49:20 -0400 Subject: [SciPy-Dev] Unpacking the mailing list: Discourse, Slack = Mailing list In-Reply-To: References: <7ED40F24-7840-4B19-8C10-77C642E7F3D3@gmail.com> Message-ID: Is it possible to interact with Discourse entirely via email? If so, it sounds like the existing workflow wouldn't need to change. On Thu, May 20, 2021 at 1:35 PM Evgeni Burovski wrote: > +1 what Tyler said > > ??, 20 ??? 2021 ?., 19:35 Tyler Reddy : > >> Ok, let me rephrase--I think you should use whatever you want (I probably >> don't understand what it is, and decent chance my employer blocks it) for >> "other discussions" and we should keep the low traffic decision making on >> the mailing list. >> >> >> On Thu, 20 May 2021 at 10:11, Pamphile Roy >> wrote: >> >>> >>> >>> > On 20 May 2021, at 17:14, Tyler Reddy >>> wrote: >>> > ? >>> > Just keep the discourse stuff for onboarding/mentoring/informal stuff >>> and mailing list for "votes"/substantive decisions IMO. >>> >>> What you describe is not suitable for Discourse but more Slack or >>> Discord (which is working well IMO as the sprint showed). >>> >>> Discourse is more a forum with topics (like a mailing list) than a >>> medium to have active discussions. Having Discourse and the mail list would >>> for sure fragment the community. >>> >>> Cheers, >>> Pamphile >>> >>> >>> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Thu May 20 14:06:15 2021 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Thu, 20 May 2021 11:06:15 -0700 Subject: [SciPy-Dev] =?utf-8?q?Unpacking_the_mailing_list=3A_Discourse=2C?= =?utf-8?q?_Slack_=3D_Mailing_list?= In-Reply-To: References: <7ED40F24-7840-4B19-8C10-77C642E7F3D3@gmail.com> Message-ID: On Thu, May 20, 2021, at 10:49, CJ Carey wrote: > Is it possible to interact with Discourse entirely via email? If so, it sounds like the existing workflow wouldn't need to change. Yes, it looks that way: https://meta.discourse.org/t/start-a-new-topic-via-email/62977 St?fan From stefanv at berkeley.edu Thu May 20 14:07:16 2021 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Thu, 20 May 2021 11:07:16 -0700 Subject: [SciPy-Dev] =?utf-8?q?Unpacking_the_mailing_list=3A_Discourse=2C?= =?utf-8?q?_Slack_=3D_Mailing_list?= In-Reply-To: References: <7ED40F24-7840-4B19-8C10-77C642E7F3D3@gmail.com> Message-ID: <74bf9fd9-0656-4906-bdd9-7ff9d5956945@www.fastmail.com> On Thu, May 20, 2021, at 09:35, Tyler Reddy wrote: > Ok, let me rephrase--I think you should use whatever you want (I probably don't understand what it is, and decent chance my employer blocks it) for "other discussions" and we should keep the low traffic decision making on the mailing list. I would not suspect so, if you can browse the web. Check whether you can you see: https://discuss.scientific-python.org/ As others mentioned, we can set it up for email interaction too. Best regards, St?fan -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Thu May 20 14:13:51 2021 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Thu, 20 May 2021 11:13:51 -0700 Subject: [SciPy-Dev] =?utf-8?q?Unpacking_the_mailing_list=3A_Discourse=2C?= =?utf-8?q?_Slack_=3D_Mailing_list?= In-Reply-To: <20739a41-0d9b-2394-733b-0cb5dbd72b7c@gmail.com> References: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> <20739a41-0d9b-2394-733b-0cb5dbd72b7c@gmail.com> Message-ID: <27a7f47b-5bad-49f0-9425-e49185315c64@www.fastmail.com> On Thu, May 20, 2021, at 04:26, Matti Picus wrote: > Among criteria for a community forum might be: > > - permanence: can we find a discussion in ten years when we went to > figure out what led to a particular decision That's certainly the idea. The server is hosted by NumFOCUS, which should give it some permanency. We have backups set up to Backblaze, and someone from SciPy will get credentials to that so SciPy can extract their data at any time. > - transparency: will the discussions be archived in a public manner with > little need for logins or signups Yes, the discussions can be browsed (but not added to) without logging in. > - searchability: is there a good way to search the discussions for > relevant fragments (single words are often not enough). It's as searchable as any site; Discourse also has a built-in search. > I am not sure how long discourse promises to preserve discussions. I am > a dinosaur and prefer email. Discourse -> email seems to work OK for the > python forums I monitor. I am also on the email train, but I have heard from enough people about their preference for Discourse that I find it compelling. A good example to check out is: https://discuss.python.org/ (Python moved away from a mailing list to Discourse a while ago.) I mentioned to Pamphile and Ralf that, for SciPy, there is one further consideration. If you use discuss.scientific-python.org, SciPy gets its own category and subcategories. If you launch your own entire server, then you get something more like discuss.python.org (you can set your own categories). Personally, I would like to see our entire community come together in one place for discussions (similar to https://forum.image.sc/) , but this is obviously up to each project to decide for themselves. Best regards, St?fan From tyler.je.reddy at gmail.com Thu May 20 15:10:13 2021 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Thu, 20 May 2021 13:10:13 -0600 Subject: [SciPy-Dev] Unpacking the mailing list: Discourse, Slack = Mailing list In-Reply-To: <27a7f47b-5bad-49f0-9425-e49185315c64@www.fastmail.com> References: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> <20739a41-0d9b-2394-733b-0cb5dbd72b7c@gmail.com> <27a7f47b-5bad-49f0-9425-e49185315c64@www.fastmail.com> Message-ID: - Discord is blocked for me, both web and application; discourse seems "ok" for the site you point to - whitelists/blacklists exist even if you can browse the web - Why don't you just add a read-only mirror for the SciPy mailing list if you really want to use discourse (or whatever) and then add separate channels for the other discussions you want? (looks possible: https://meta.discourse.org/t/creating-a-read-only-mailing-list-mirror/77990 ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From serge.guelton at telecom-bretagne.eu Thu May 20 15:23:49 2021 From: serge.guelton at telecom-bretagne.eu (Serge Guelton) Date: Thu, 20 May 2021 21:23:49 +0200 Subject: [SciPy-Dev] Unpacking the mailing list: Discourse, Slack = Mailing list In-Reply-To: <27a7f47b-5bad-49f0-9425-e49185315c64@www.fastmail.com> References: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> <20739a41-0d9b-2394-733b-0cb5dbd72b7c@gmail.com> <27a7f47b-5bad-49f0-9425-e49185315c64@www.fastmail.com> Message-ID: <20210520192349.GA2599326@sguelton.remote.csb> On Thu, May 20, 2021 at 11:13:51AM -0700, Stefan van der Walt wrote: > On Thu, May 20, 2021, at 04:26, Matti Picus wrote: > > Among criteria for a community forum might be: > > > > - permanence: can we find a discussion in ten years when we went to > > figure out what led to a particular decision > > That's certainly the idea. The server is hosted by NumFOCUS, which should give it some permanency. We have backups set up to Backblaze, and someone from SciPy will get credentials to that so SciPy can extract their data at any time. > > > - transparency: will the discussions be archived in a public manner with > > little need for logins or signups > > Yes, the discussions can be browsed (but not added to) without logging in. > > > - searchability: is there a good way to search the discussions for > > relevant fragments (single words are often not enough). > > It's as searchable as any site; Discourse also has a built-in search. > > > I am not sure how long discourse promises to preserve discussions. I am > > a dinosaur and prefer email. Discourse -> email seems to work OK for the > > python forums I monitor. > > I am also on the email train, but I have heard from enough people about their preference for Discourse that I find it compelling. A good example to check out is: > > https://discuss.python.org/ > > (Python moved away from a mailing list to Discourse a while ago.) > > I mentioned to Pamphile and Ralf that, for SciPy, there is one further consideration. If you use discuss.scientific-python.org, SciPy gets its own category and subcategories. If you launch your own entire server, then you get something more like discuss.python.org (you can set your own categories). > > Personally, I would like to see our entire community come together in one place for discussions (similar to https://forum.image.sc/) , but this is obviously up to each project to decide for themselves. I'm also on the email wagon, but it looks like discourse can import a mailing list archive, and if it can be configured to send a email, it may be considered as a modern view on a mailing list. I'm 100% fine with that. From stefanv at berkeley.edu Thu May 20 16:15:59 2021 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Thu, 20 May 2021 13:15:59 -0700 Subject: [SciPy-Dev] =?utf-8?q?Unpacking_the_mailing_list=3A_Discourse=2C?= =?utf-8?q?_Slack_=3D_Mailing_list?= In-Reply-To: References: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> <20739a41-0d9b-2394-733b-0cb5dbd72b7c@gmail.com> <27a7f47b-5bad-49f0-9425-e49185315c64@www.fastmail.com> Message-ID: On Thu, May 20, 2021, at 12:10, Tyler Reddy wrote: > - Why don't you just add a read-only mirror for the SciPy mailing list if you really want to use discourse (or whatever) and then add separate channels for the other discussions you want? (looks possible: https://meta.discourse.org/t/creating-a-read-only-mailing-list-mirror/77990 ) I think that's the way to do it, yes. St?fan -------------- next part -------------- An HTML attachment was scrubbed... URL: From njgodber at gmail.com Thu May 20 20:04:38 2021 From: njgodber at gmail.com (Neil Godber) Date: Fri, 21 May 2021 10:04:38 +1000 Subject: [SciPy-Dev] Enhancement Request: SuperLU_dist/SuperLU_mt replace current scipy.sparse.superLU Message-ID: Hi dev-list, I've logged an enhancement request on github to bring the parallel versions of SuperLU into scipy. As requested I'm circulating the enhancement here to solicit comment/feedback/engagement. Content of request is as follows (with light editing and links): "Is your feature request related to a problem? Please describe. The direct solution of sparse matrices is a common problem that arises across many domains. Current scipy.sparse.linalg provides spsolve and splu to solve such classes of problems. Neither solution appears to utilise more than one or two threads leaving a lot of the available compute on modern machines unused. Other solutions (MUMPS, Pardiso) exploit multicore processors but are either minimally supported and/or locked behind proprietary libraries. Describe the solution you'd like Fortunately SuperLU has two multithreaded implementations superLU_mt and superLU_dt, both actively maintained (SuperLU: Home Page (nersc.gov) ). Ideally SciPy would replace the current sequential implementation with one of the parallel implementations to yield large performance improvements. Describe alternatives you've considered pyMKL (Pardiso dwfmarchant/pyMKL: Python wrappers to Intel MKL routines (github.com) ) - MKL with issues associated with that for non Intel processors, unmaintained relatively to scipy, PyTrilinos (implements all three superLU types but only available on osx and linux PyTrilinos | Trilinos ), petsc (petsc4py PETSc / petsc ? GitLab ) (implements all three superLU types but only available on osx and linux). All are relatively inaccessible compared to ubiquity and accessibility of scipy. Additional context (e.g. screenshots) Anecdotally large number of projects use various non scipy sparse solvers to achieve high level of performance. This comes with considerable time and maintenance investment. It would be fantastic for scipy to provide a 'performance competitive' sparse direct solver out of the hood as it does for dense matrices." Cheers, Neil -------------- next part -------------- An HTML attachment was scrubbed... URL: From njgodber at gmail.com Thu May 20 20:10:39 2021 From: njgodber at gmail.com (Neil Godber) Date: Fri, 21 May 2021 10:10:39 +1000 Subject: [SciPy-Dev] Enhancement Request: SuperLU_dist/SuperLU_mt replace current scipy.sparse.superLU In-Reply-To: References: Message-ID: Hi dev-list, Further last, realised that I failed to link to the issue itself despite linking to virtually everything else: Replace superLU sequential with superLU dist ? Issue #14096 ? scipy/scipy (github.com) Cheers, Neil On Fri, 21 May 2021 at 10:04, Neil Godber wrote: > Hi dev-list, > > I've logged an enhancement request on github to bring the > parallel versions of SuperLU into scipy. As requested I'm circulating the > enhancement here to solicit comment/feedback/engagement. > > Content of request is as follows (with light editing and links): > > "Is your feature request related to a problem? Please describe. > > The direct solution of sparse matrices is a common problem that arises > across many domains. Current scipy.sparse.linalg provides spsolve and splu > to solve such classes of problems. Neither solution appears to utilise more > than one or two threads leaving a lot of the available compute on modern > machines unused. Other solutions (MUMPS, Pardiso) exploit multicore > processors but are either minimally supported and/or locked behind > proprietary libraries. > > Describe the solution you'd like > Fortunately SuperLU has two multithreaded implementations superLU_mt and > superLU_dt, both actively maintained (SuperLU: Home Page (nersc.gov) > ). Ideally SciPy would > replace the current sequential implementation with one of the parallel > implementations to yield large performance improvements. > > Describe alternatives you've considered > pyMKL (Pardiso dwfmarchant/pyMKL: Python wrappers to Intel MKL routines > (github.com) ) - MKL with issues > associated with that for non Intel processors, unmaintained relatively to > scipy, PyTrilinos (implements all three superLU types but only available on > osx and linux PyTrilinos | Trilinos > ), petsc (petsc4py PETSc / > petsc ? GitLab ) (implements all three > superLU types but only available on osx and linux). All are relatively > inaccessible compared to ubiquity and accessibility of scipy. > > Additional context (e.g. screenshots) > Anecdotally large number of projects use various non scipy sparse solvers > to achieve high level of performance. This comes with considerable time and > maintenance investment. It would be fantastic for scipy to provide a > 'performance competitive' sparse direct solver out of the hood as it does > for dense matrices." > > > Cheers, > > Neil > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicholas.bgp at gmail.com Fri May 21 10:56:17 2021 From: nicholas.bgp at gmail.com (Nicholas McKibben) Date: Fri, 21 May 2021 07:56:17 -0700 Subject: [SciPy-Dev] Unpacking the mailing list: Discourse, Slack = Mailing list In-Reply-To: References: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> <20739a41-0d9b-2394-733b-0cb5dbd72b7c@gmail.com> <27a7f47b-5bad-49f0-9425-e49185315c64@www.fastmail.com> Message-ID: > I'm also on the email wagon, but it looks like discourse can import a mailing list archive, and if it can be configured to send a email, it may be considered as a modern view on a mailing list. I'm 100% fine with that. +1 what Serge said, maintaining the existing email workflow would reduce the barrier of entry for me. On Thu, May 20, 2021 at 1:16 PM Stefan van der Walt wrote: > On Thu, May 20, 2021, at 12:10, Tyler Reddy wrote: > > - Why don't you just add a read-only mirror for the SciPy mailing list if > you really want to use discourse (or whatever) and then add separate > channels for the other discussions you want? (looks possible: > https://meta.discourse.org/t/creating-a-read-only-mailing-list-mirror/77990 > ) > > > I think that's the way to do it, yes. > > St?fan > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From topolskikevin at gmail.com Fri May 21 12:46:33 2021 From: topolskikevin at gmail.com (Kevin Topolski) Date: Fri, 21 May 2021 11:46:33 -0500 Subject: [SciPy-Dev] Introduction! Message-ID: Hi all, I introduced myself at the last Scipy community meeting and recently, made my first pull request to Scipy so I thought this would be another great opportunity to introduce myself to a larger audience! I got my Ph'd in Chemical Engineering and have used Numpy and Scipy for my own projects and studies for about 2 years now. After using optimization and other computational techniques in my work, I started an interest to develop these tools in open source and saw Scipy as the place to do so! I would love to get your feedback all as I am new to developing and open source. I look forward to learning from you all! Link to my first PR: https://github.com/scipy/scipy/pull/14103 Best and cheers! Kevin Topolski -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilhanpolat at gmail.com Fri May 21 12:51:07 2021 From: ilhanpolat at gmail.com (Ilhan Polat) Date: Fri, 21 May 2021 18:51:07 +0200 Subject: [SciPy-Dev] Unpacking the mailing list: Discourse, Slack = Mailing list In-Reply-To: References: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> <20739a41-0d9b-2394-733b-0cb5dbd72b7c@gmail.com> <27a7f47b-5bad-49f0-9425-e49185315c64@www.fastmail.com> Message-ID: Discourse and emails have the same issue for me that I can't track who said what to whom; for example https://discuss.python.org/t/pep-632-deprecate-distutils-module/5134/4 I spent way too much time following who replied to Matti and then who reacted back etc. (discourse has this weird habit showing the reply both in reply window and in the flow later in the page but maybe that's on me) but moving away from email seems the right thing for me. Especially when a discussion doesn't converge to a point it is just a useless pile of text that promotes misunderstanding as can be seen from many discussion in many places (linux kernel being the notorious) and starts to have those >>>>>>>>>>> trains. I don't mind the next tech though I share the same concern for Slack as mentioned. I still think that github discussions would help a lot in not-so-strategic decisions as I'm seeing it used within the GitHub beta repos. Obvious disclosure, this is my personal view on it for what its worth. I get that there is still a huge dependence on email on different sectors and academia apparently. On Fri, May 21, 2021 at 4:57 PM Nicholas McKibben wrote: > > I'm also on the email wagon, but it looks like discourse can import a > mailing list archive, and if it can be configured to send a email, it may > be considered as a modern view on a mailing list. I'm 100% fine with that. > > +1 what Serge said, maintaining the existing email workflow would reduce > the barrier of entry for me. > > On Thu, May 20, 2021 at 1:16 PM Stefan van der Walt > wrote: > >> On Thu, May 20, 2021, at 12:10, Tyler Reddy wrote: >> >> - Why don't you just add a read-only mirror for the SciPy mailing list if >> you really want to use discourse (or whatever) and then add separate >> channels for the other discussions you want? (looks possible: >> https://meta.discourse.org/t/creating-a-read-only-mailing-list-mirror/77990 >> ) >> >> >> I think that's the way to do it, yes. >> >> St?fan >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Fri May 21 13:04:23 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Fri, 21 May 2021 19:04:23 +0200 Subject: [SciPy-Dev] Introduction! In-Reply-To: References: Message-ID: > On 21.05.2021, at 18:46, Kevin Topolski wrote: > > I would love to get your feedback all as I am new to developing and open source. I look forward to learning from you all! > > Link to my first PR: https://github.com/scipy/scipy/pull/14103 Welcome Kevin! Good first PR, examples are always a good addition! I left a few cosmetic comments (please don?t mind my mess with my accounts, sorry). Cheers, Pamphile -------------- next part -------------- An HTML attachment was scrubbed... URL: From scipy-dev at fuglede.dk Sat May 22 07:47:48 2021 From: scipy-dev at fuglede.dk (=?iso-8859-1?Q?S=F8ren_Fuglede_J=F8rgensen?=) Date: Sat, 22 May 2021 13:47:48 +0200 Subject: [SciPy-Dev] Unpacking the mailing list: Discourse, Slack = Mailing list In-Reply-To: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> References: <7A9AC337-1584-45C6-B825-FBC2D20B9905@gmail.com> Message-ID: <20210522114748.GA3888@hackenturet.dk> On Thu, May 20, 2021 at 11:40:28AM +0200, Pamphile Roy wrote: > > Let us know what you think. > As someone who was hit by the DMARC hammer on the mailing list, a good web forum might lower the barrier to entry. One show-stopper I've always found with the various bulletin boards is the inability to get an overview of the reply tree (or DAG) for a given topic: Ensuring that it's easy to follow a particular thread. Mailman does this okay-ish, but from a quick attempt, Discourse seems to offer limited improvement over e.g. phpBB, while e.g. the comment sections on Reddit or Hacker News work better. This is getting off-topic, but I wonder if there's a service that offers bulletin board style topics a la Discourse but with comment sections a la Reddit and modern commenting facilities (Markdown/non-text content/...) a la Slack? Best, S?ren From ralf.gommers at gmail.com Sun May 23 14:32:56 2021 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sun, 23 May 2021 20:32:56 +0200 Subject: [SciPy-Dev] Enhancement Request: SuperLU_dist/SuperLU_mt replace current scipy.sparse.superLU In-Reply-To: References: Message-ID: On Fri, May 21, 2021 at 2:11 AM Neil Godber wrote: > Hi dev-list, > > Further last, realised that I failed to link to the issue itself despite > linking to virtually everything else: > > Replace superLU sequential with superLU dist ? Issue #14096 ? scipy/scipy > (github.com) > > Cheers, > Neil > > On Fri, 21 May 2021 at 10:04, Neil Godber wrote: > >> Hi dev-list, >> > Thanks for the proposal Neil. >> I've logged an enhancement request on github to bring the >> parallel versions of SuperLU into scipy. As requested I'm circulating the >> enhancement here to solicit comment/feedback/engagement. >> >> Content of request is as follows (with light editing and links): >> >> "Is your feature request related to a problem? Please describe. >> >> The direct solution of sparse matrices is a common problem that arises >> across many domains. Current scipy.sparse.linalg provides spsolve and splu >> to solve such classes of problems. Neither solution appears to utilise more >> than one or two threads leaving a lot of the available compute on modern >> machines unused. Other solutions (MUMPS, Pardiso) exploit multicore >> processors but are either minimally supported and/or locked behind >> proprietary libraries. >> >> Describe the solution you'd like >> Fortunately SuperLU has two multithreaded implementations superLU_mt and >> superLU_dt, both actively maintained (SuperLU: Home Page (nersc.gov) >> ). Ideally SciPy would >> replace the current sequential implementation with one of the parallel >> implementations to yield large performance improvements. >> > The relevant one seems to be superLU_mt. The `_dt` flavor is for distributed computing, which is out of scope for SciPy. superLU_mt says it has both pthreads and OpenMP interfaces - we'd prefer pthreads (and actually forbid OpenMP within SciPy). The last release of superLU_mt is v3.1, from March 2016. Our current copy of SuperLU is 5.2.1, from May 2016 - but then there's a bunch of patches applied to it, and then https://github.com/scipy/scipy/pull/12243 did a sync with SuperLU master in May 2020. So it would be necessary to figure out if we can get superLU_mt from upstream master as well, and that we don't lose bug fixes we applied since 2016. Could you look into that Neil? And do you know if the parallel and sequential versions are developed in sync or not? Cheers, Ralf Describe alternatives you've considered >> pyMKL (Pardiso dwfmarchant/pyMKL: Python wrappers to Intel MKL routines >> (github.com) ) - MKL with issues >> associated with that for non Intel processors, unmaintained relatively to >> scipy, PyTrilinos (implements all three superLU types but only available on >> osx and linux PyTrilinos | Trilinos >> ), petsc (petsc4py PETSc / >> petsc ? GitLab ) (implements all three >> superLU types but only available on osx and linux). All are relatively >> inaccessible compared to ubiquity and accessibility of scipy. >> >> Additional context (e.g. screenshots) >> Anecdotally large number of projects use various non scipy sparse solvers >> to achieve high level of performance. This comes with considerable time and >> maintenance investment. It would be fantastic for scipy to provide a >> 'performance competitive' sparse direct solver out of the hood as it does >> for dense matrices." >> >> >> Cheers, >> >> Neil >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyler.je.reddy at gmail.com Sun May 23 20:17:19 2021 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Sun, 23 May 2021 18:17:19 -0600 Subject: [SciPy-Dev] Proposed 1.7.0 Release Schedule In-Reply-To: References: <043BDEF9-8DA3-4FB1-B134-11F804776119@gmail.com> Message-ID: The draft release notes are ready for review/suggestions: https://github.com/scipy/scipy/pull/14118 We have about 20 open PRs with milestone, which is a bit high this close to branching: https://github.com/scipy/scipy/pulls?q=is%3Aopen+is%3Apr+milestone%3A1.7.0 Quite a few are marked "approved." Some are likely to get bumped--I'm probably going to get a bit stricter tomorrow with bumping PRs that aren't on a clear trajectory to merging by Wednesday based on the comments or available expert reviewer bandwidth, CI issues that haven't been fixed, etc. Best wishes, Tyler On Sun, 16 May 2021 at 19:02, Andrew Nelson wrote: > On Sun, 16 May 2021 at 22:14, Pamphile Roy wrote: > >> Regarding Optimize. It seems that lots of PRs and issues are pointing >> towards an overhaul of the module. My position would be to solve what can >> be solved for the release with open PRs and for the next release we should >> lay out a plan to evolve the module. >> > > It'd certainly be good for the number of active optimize maintainers to > increase, to tackle PRs and issues, etc. > > optimize is probably one of the most heavily used parts of scipy, that's > why it gets a lot of traffic on github. I think the module is probably in > not too bad a good shape. A review of outstanding issues would give an idea > of what types of issues are commonly experienced, it's not clear that a > large scale redesign would necessarily change anything. > > One thing I'm mainly interested in minimisations of scalar functions. I > had a desire to implement a class-based minimization system for those, but > I'm fairly convinced it would have to be developed outside scipy in a > spin-off project first. > > > > >> What do you think Andrew? Could we plan a specific meeting for that maybe? >> >> Cheers, >> Pamphile >> > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njgodber at gmail.com Sun May 23 21:09:40 2021 From: njgodber at gmail.com (Neil Godber) Date: Mon, 24 May 2021 11:09:40 +1000 Subject: [SciPy-Dev] Enhancement Request: SuperLU_dist/SuperLU_mt replace current scipy.sparse.superLU In-Reply-To: References: Message-ID: Hi Ralf, Thanks for your email and happy to assist in whatever way I can. I've reached out to Dr Xiaoyei Li of Lawrence Berkley who appears to be the principal maintainer of the code base with a request to host the _mt code on github. Given the other two versions are already hosted I can't see that being an issue. I've also queried whether developments are done in sync and if so, if there is a pending patching for mt in the similar fashion to sequential. I will revert with further information as I receive it. Cheers, Neil On Mon, 24 May 2021 at 04:33, Ralf Gommers wrote: > > > On Fri, May 21, 2021 at 2:11 AM Neil Godber wrote: > >> Hi dev-list, >> >> Further last, realised that I failed to link to the issue itself despite >> linking to virtually everything else: >> >> Replace superLU sequential with superLU dist ? Issue #14096 ? scipy/scipy >> (github.com) >> >> Cheers, >> Neil >> >> On Fri, 21 May 2021 at 10:04, Neil Godber wrote: >> >>> Hi dev-list, >>> >> > Thanks for the proposal Neil. > > >>> I've logged an enhancement request on github to bring the >>> parallel versions of SuperLU into scipy. As requested I'm circulating the >>> enhancement here to solicit comment/feedback/engagement. >>> >>> Content of request is as follows (with light editing and links): >>> >>> "Is your feature request related to a problem? Please describe. >>> >>> The direct solution of sparse matrices is a common problem that arises >>> across many domains. Current scipy.sparse.linalg provides spsolve and splu >>> to solve such classes of problems. Neither solution appears to utilise more >>> than one or two threads leaving a lot of the available compute on modern >>> machines unused. Other solutions (MUMPS, Pardiso) exploit multicore >>> processors but are either minimally supported and/or locked behind >>> proprietary libraries. >>> >>> Describe the solution you'd like >>> Fortunately SuperLU has two multithreaded implementations superLU_mt and >>> superLU_dt, both actively maintained (SuperLU: Home Page (nersc.gov) >>> ). Ideally SciPy >>> would replace the current sequential implementation with one of the >>> parallel implementations to yield large performance improvements. >>> >> The relevant one seems to be superLU_mt. The `_dt` flavor is for > distributed computing, which is out of scope for SciPy. superLU_mt says it > has both pthreads and OpenMP interfaces - we'd prefer pthreads (and > actually forbid OpenMP within SciPy). > > The last release of superLU_mt is v3.1, from March 2016. Our current copy > of SuperLU is 5.2.1, from May 2016 - but then there's a bunch of patches > applied to it, and then https://github.com/scipy/scipy/pull/12243 did a > sync with SuperLU master in May 2020. So it would be necessary to figure > out if we can get superLU_mt from upstream master as well, and that we > don't lose bug fixes we applied since 2016. Could you look into that Neil? > And do you know if the parallel and sequential versions are developed in > sync or not? > > Cheers, > Ralf > > > Describe alternatives you've considered >>> pyMKL (Pardiso dwfmarchant/pyMKL: Python wrappers to Intel MKL routines >>> (github.com) ) - MKL with issues >>> associated with that for non Intel processors, unmaintained relatively to >>> scipy, PyTrilinos (implements all three superLU types but only available on >>> osx and linux PyTrilinos | Trilinos >>> ), petsc (petsc4py PETSc / >>> petsc ? GitLab ) (implements all three >>> superLU types but only available on osx and linux). All are relatively >>> inaccessible compared to ubiquity and accessibility of scipy. >>> >>> Additional context (e.g. screenshots) >>> Anecdotally large number of projects use various non scipy sparse >>> solvers to achieve high level of performance. This comes with considerable >>> time and maintenance investment. It would be fantastic for scipy to provide >>> a 'performance competitive' sparse direct solver out of the hood as it does >>> for dense matrices." >>> >>> >>> Cheers, >>> >>> Neil >>> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From danielschmitzsiegen at googlemail.com Mon May 24 03:22:34 2021 From: danielschmitzsiegen at googlemail.com (Daniel Schmitz) Date: Mon, 24 May 2021 09:22:34 +0200 Subject: [SciPy-Dev] Global Optimization Benchmarks In-Reply-To: References: Message-ID: Hey Ralf, discussion for DIRECT was opened here: https://github.com/scipy/scipy/issues/14121 and the scipydirect maintainer was pinged: https://github.com/andim/scipydirect/issues/9 . About the general discussion regarding global optimization in SciPy: I am working on a SciPy style API for NLopt in my free time (mostly missing documentation at the moment) and would like to benchmark its MLSL and StoGO algorithms using the SciPy benchmarks. Currently, this seems very complicated as any new algorithm would have to be included into SciPy first. Is there a way to circumvent this? Of course, Andrea's benchmark suite would be the mother of all benchmarks if available ;). Cheers, Daniel On Sun, 25 Apr 2021 at 22:32, Ralf Gommers wrote: > Hi Daniel, > > Sorry for the extremely delayed reply. > > > On Thu, Mar 11, 2021 at 10:19 AM Daniel Schmitz < > danielschmitzsiegen at googlemail.com> wrote: > >> Hey all, >> >> after Andrea's suggestions, I did an extensive github search and found >> several global optimization python libraries which mimic the scipy >> API, so that a user only has to change the import statements. Could it >> be useful to add a page in the documentation of these? >> > > This does sound useful. Probably not a whole page, more like a note in the > `minimize` docstring with references I'd think. > > >> Non exhaustive list: >> >> DIRECT: https://github.com/andim/scipydirect >> DE/PSO/CMA-ES: https://github.com/keurfonluu/stochopy >> PSO: https://github.com/jerrytheo/psopy >> Powell's derivative free optimizers: https://www.pdfo.net/index.html >> >> As DIRECT was very competitive on some of Andrea's benchmarks, it >> could be useful to mimic the scipydirect repo for inclusion into scipy >> (MIT license). The code is unfortunately a f2py port of the original >> Fortran implementation which has hard coded bounds on the number of >> function evaluations (90.000) and iterations (6.000). Any opinions on >> this? >> > > This sounds like a good idea. Would you mind opening a GitHub issue with > the feature request, so we keep track of this? Contacting the original > author about this would also be useful; if the author would like to > upstream their code, that'd be a good outcome. > > Re hard coded bounds, I assume those can be removed again without too much > trouble. > > >> >> I personally am very impressed by biteopt's performance, and although >> it ranked very high in other global optimization benchmarks there is >> no formal paper on it yet. I understand the scipy guidelines in a way >> that such a paper is a requisite for inclusion into scipy. >> > > Well, we do have the very extensive benchmarks - if it does indeed perform > better than what we have, then of course we'd be happy to add a new > algorithm even if it doesn't have a paper. We use papers and the citations > it has as an indication of usefulness only; anything that outperforms our > existing code is clearly useful. > > Cheers, > Ralf > > > >> >> Best, >> >> Daniel >> >> Daniel >> >> >> On Sun, 17 Jan 2021 at 14:33, Andrea Gavana >> wrote: >> > >> > Hi Stefan, >> > >> > You?re most welcome :-) . I?m happy the experts in the community >> are commenting and suggesting things, and constructive criticism is also >> always welcome. >> > >> > On Sun, 17 Jan 2021 at 12.11, Stefan Endres >> wrote: >> >> >> >> Dear Andrea, >> >> >> >> Thank you very much for this detailed analysis. I don't think I've >> seen such a large collection of benchmark test suites or collection of DFO >> algorithms since the publication by Rios and Sahinidis in 2013. Some >> questions: >> >> >> >> Many of the commercial algorithms offer free licenses for benchmarking >> problems of less than 10 dimensions. Would you be willing to include some >> of these in your benchmarks at some point? It would be a great reference to >> use. >> > >> > >> > I?m definitely willing to include those commercial algorithms. The test >> suite per se is almost completely automated, so it?s not that complicated >> to add one or more solvers. I?m generally more inclined in testing open >> source algorithms but there?s nothing stopping the inclusion of commercial >> ones. >> > >> > I welcome any suggestions related to commercial solvers, as long as >> they can run on Python 2 / Python 3 and on Windows (I might be able to >> setup a Linux virtual machine if absolutely needed but that would defy part >> of the purpose of the exercise - SHGO, Dual Annealing and the other SciPy >> solvers run on all platforms that support SciPy). >> > >> >> The collection of test suites you've garnered could be immensely >> useful for further algorithm development. Is there a possibility of >> releasing the code publicly (presumably after you've published the results >> in a journal)? >> >> >> >> In this case I would also like to volunteer to run some of the >> commercial solvers on the benchmark suite. >> >> It would also help to have a central repository for fixing bugs and >> adding lower global minima when they are found (of which there are quite >> few ). >> > >> > >> > >> > I?m still sorting out all the implications related to a potential paper >> with my employer, but as far as I can see there shouldn?t be any problem >> with that: assuming everything goes as it should, I will definitely push >> for making the code open source. >> > >> > >> >> >> >> Comments on shgo: >> >> >> >> High RAM use in higher dimensions: >> >> >> >> In the higher dimensions the new simplicial sampling can be used (not >> pushed to scipy yet; I still need to update some documentation before the >> PR). This alleviates, but does not eliminate the memory leak issue. As >> you've said SHGO is best suited to problems below 10 dimensions as any >> higher leaves the realm of DFO problems and starts to enter the domain of >> NLP problems. My personal preference in this case is to use the stochastic >> algorithms (basinhopping and differential evolution) on problems where it >> is known that a gradient based solver won't work. >> >> >> >> An exception to this "rule" is when special grey box information such >> as symmetry of the objective function (something that can be supplied to >> shgo to push the applicability of the algorithm up to ~100 variables) or >> pre-computed bounds on the Lipschitz constants is known. >> >> >> >> In the symmetry case SHGO can solve these by supplying the `symmetry` >> option (which was used in the previous benchmarks done by me for the JOGO >> publication, although I did not specifically check if performance was >> actually improved on those problems, but shgo did converge on all benchmark >> problems in the scipy test suite). >> >> >> >> I have had a few reports of memory leaks from various users. I have >> spoken to a few collaborators about the possibility of finding a Masters >> student to cythonize some of the code or otherwise improve it. Hopefully, >> this will happen in the summer semester of 2021. >> > >> > >> > To be honest I wouldn?t be so concerned in general: SHGO is an >> excellent global optimization algorithm and it consistently ranks at the >> top, no matter what problems you throw at it. Together with Dual Annealing, >> SciPy has gained two phenomenal nonlinear solvers and I?m very happy to see >> that SciPy is now at the cutting edge of the open source global >> optimization universe. >> > >> > Andrea. >> > >> >> Thank you again for compiling this large set of benchmark results. >> >> >> >> Best regards, >> >> Stefan >> >> On Fri, Jan 8, 2021 at 10:21 AM Andrea Gavana >> wrote: >> >>> >> >>> Dear SciPy Developers & Users, >> >>> >> >>> long time no see :-) . I thought to start 2021 with a bit of a >> bang, to try and forget how bad 2020 has been... So I am happy to present >> you with a revamped version of the Global Optimization Benchmarks from my >> previous exercise in 2013. >> >>> >> >>> This new set of benchmarks pretty much superseeds - and greatly >> expands - the previous analysis that you can find at this location: >> http://infinity77.net/global_optimization/ . >> >>> >> >>> The approach I have taken this time is to select as many benchmark >> test suites as possible: most of them are characterized by test function >> generators, from which we can actually create almost an unlimited number of >> unique test problems. Biggest news are: >> >>> >> >>> This whole exercise is made up of 6,825 test problems divided across >> 16 different test suites: most of these problems are of low dimensionality >> (2 to 6 variables) with a few benchmarks extending to 9+ variables. With >> all the sensitivities performed during this exercise on those benchmarks, >> the overall grand total number of functions evaluations stands at >> 3,859,786,025 - close to 4 billion. Not bad. >> >>> A couple of "new" optimization algorithms I have ported to Python: >> >>> >> >>> MCS: Multilevel Coordinate Search, it?s my translation to Python of >> the original Matlab code from A. Neumaier and W. Huyer (giving then for >> free also GLS and MINQ) I have added a few, minor improvements compared to >> the original implementation. >> >>> BiteOpt: BITmask Evolution OPTimization , I have converted the C++ >> code into Python and added a few, minor modifications. >> >>> >> >>> >> >>> Enough chatting for now. The 13 tested algorithms are described here: >> >>> >> >>> http://infinity77.net/go_2021/ >> >>> >> >>> High level description & results of the 16 benchmarks: >> >>> >> >>> http://infinity77.net/go_2021/thebenchmarks.html >> >>> >> >>> Each benchmark test suite has its own dedicated page, with more >> detailed results and sensitivities. >> >>> >> >>> List of tested algorithms: >> >>> >> >>> AMPGO: Adaptive Memory Programming for Global Optimization: this is >> my Python implementation of the algorithm described here: >> >>> >> >>> >> http://leeds-faculty.colorado.edu/glover/fred%20pubs/416%20-%20AMP%20(TS)%20for%20Constrained%20Global%20Opt%20w%20Lasdon%20et%20al%20.pdf >> >>> >> >>> I have added a few improvements here and there based on my Master >> Thesis work on the standard Tunnelling Algorithm of Levy, Montalvo and >> Gomez. After AMPGO was integrated in lmfit, I have improved it even more - >> in my opinion. >> >>> >> >>> BasinHopping: Basin hopping is a random algorithm which attempts to >> find the global minimum of a smooth scalar function of one or more >> variables. The algorithm was originally described by David Wales: >> >>> >> >>> http://www-wales.ch.cam.ac.uk/ >> >>> >> >>> BasinHopping is now part of the standard SciPy distribution. >> >>> >> >>> BiteOpt: BITmask Evolution OPTimization, based on the algorithm >> presented in this GitHub link: >> >>> >> >>> https://github.com/avaneev/biteopt >> >>> >> >>> I have converted the C++ code into Python and added a few, minor >> modifications. >> >>> >> >>> CMA-ES: Covariance Matrix Adaptation Evolution Strategy, based on the >> following algorithm: >> >>> >> >>> http://www.lri.fr/~hansen/cmaesintro.html >> >>> >> >>> http://www.lri.fr/~hansen/cmaes_inmatlab.html#python (Python code >> for the algorithm) >> >>> >> >>> CRS2: Controlled Random Search with Local Mutation, as implemented in >> the NLOpt package: >> >>> >> >>> >> http://ab-initio.mit.edu/wiki/index.php/NLopt_Algorithms#Controlled_Random_Search_.28CRS.29_with_local_mutation >> >>> >> >>> DE: Differential Evolution, described in the following page: >> >>> >> >>> http://www1.icsi.berkeley.edu/~storn/code.html >> >>> >> >>> DE is now part of the standard SciPy distribution, and I have taken >> the implementation as it stands in SciPy: >> >>> >> >>> >> https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.differential_evolution.html#scipy.optimize.differential_evolution >> >>> >> >>> DIRECT: the DIviding RECTangles procedure, described in: >> >>> >> >>> >> https://www.tol-project.org/export/2776/tolp/OfficialTolArchiveNetwork/NonLinGloOpt/doc/DIRECT_Lipschitzian%20optimization%20without%20the%20lipschitz%20constant.pdf >> >>> >> >>> >> http://ab-initio.mit.edu/wiki/index.php/NLopt_Algorithms#DIRECT_and_DIRECT-L >> (Python code for the algorithm) >> >>> >> >>> DualAnnealing: the Dual Annealing algorithm, taken directly from the >> SciPy implementation: >> >>> >> >>> >> https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.dual_annealing.html#scipy.optimize.dual_annealing >> >>> >> >>> LeapFrog: the Leap Frog procedure, which I have been recommended for >> use, taken from: >> >>> >> >>> https://github.com/flythereddflagg/lpfgopt >> >>> >> >>> MCS: Multilevel Coordinate Search, it?s my translation to Python of >> the original Matlab code from A. Neumaier and W. Huyer (giving then for >> free also GLS and MINQ): >> >>> >> >>> https://www.mat.univie.ac.at/~neum/software/mcs/ >> >>> >> >>> I have added a few, minor improvements compared to the original >> implementation. See the MCS section for a quick and dirty comparison >> between the Matlab code and my Python conversion. >> >>> >> >>> PSWARM: Particle Swarm optimization algorithm, it has been described >> in many online papers. I have used a compiled version of the C source code >> from: >> >>> >> >>> http://www.norg.uminho.pt/aivaz/pswarm/ >> >>> >> >>> SCE: Shuffled Complex Evolution, described in: >> >>> >> >>> Duan, Q., S. Sorooshian, and V. Gupta, Effective and efficient global >> optimization for conceptual rainfall-runoff models, Water Resour. Res., 28, >> 1015-1031, 1992. >> >>> >> >>> The version I used was graciously made available by Matthias Cuntz >> via a personal e-mail. >> >>> >> >>> SHGO: Simplicial Homology Global Optimization, taken directly from >> the SciPy implementation: >> >>> >> >>> >> https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.shgo.html#scipy.optimize.shgo >> >>> >> >>> >> >>> List of benchmark test suites: >> >>> >> >>> SciPy Extended: 235 multivariate problems (where the number of >> independent variables ranges from 2 to 17), again with multiple >> local/global minima. >> >>> >> >>> I have added about 40 new functions to the standard SciPy benchmarks >> and fixed a few bugs in the existing benchmark models in the SciPy >> repository. >> >>> >> >>> GKLS: 1,500 test functions, with dimensionality varying from 2 to 6, >> generated with the super famous GKLS Test Functions Generator. I have taken >> the original C code (available at http://netlib.org/toms/) and converted >> it to Python. >> >>> >> >>> GlobOpt: 288 tough problems, with dimensionality varying from 2 to 5, >> created with another test function generator which I arbitrarily named >> ?GlobOpt?: >> https://www.researchgate.net/publication/225566516_A_new_class_of_test_functions_for_global_optimization >> . The original code is in C++ and I have bridged it to Python using Cython. >> >>> >> >>> Many thanks go to Professor Marco Locatelli for providing an updated >> copy of the C++ source code. >> >>> >> >>> MMTFG: sort-of an acronym for ?Multi-Modal Test Function with >> multiple Global minima?, this test suite implements the work of Jani >> Ronkkonen: >> https://www.researchgate.net/publication/220265526_A_Generator_for_Multimodal_Test_Functions_with_Multiple_Global_Optima >> . It contains 981 test problems with dimensionality varying from 2 to 4. >> The original code is in C and I have bridge it to Python using Cython. >> >>> >> >>> GOTPY: a generator of benchmark functions using the Bocharov-Feldbaum >> ?Method-Min?, containing 400 test problems with dimensionality varying from >> 2 to 5. I have taken the Python implementation from >> https://github.com/redb0/gotpy and improved it in terms of runtime. >> >>> >> >>> Original paper from >> http://www.mathnet.ru/php/archive.phtml?wshow=paper&jrnid=at&paperid=11985&option_lang=eng >> . >> >>> >> >>> Huygens: this benchmark suite is very different from the rest, as it >> uses a ?fractal? approach to generate test functions. It is based on the >> work of Cara MacNish on Fractal Functions. The original code is in Java, >> and at the beginning I just converted it to Python: given it was slow as a >> turtle, I have re-implemented it in Fortran and wrapped it using f2py, then >> generating 600 2-dimensional test problems out of it. >> >>> >> >>> LGMVG: not sure about the meaning of the acronym, but the >> implementation follows the ?Max-Set of Gaussians Landscape Generator? >> described in http://boyuan.global-optimization.com/LGMVG/index.htm . >> Source code is given in Matlab, but it?s fairly easy to convert it to >> Python. This test suite contains 304 problems with dimensionality varying >> from 2 to 5. >> >>> >> >>> NgLi: Stemming from the work of Chi-Kong Ng and Duan Li, this is a >> test problem generator for unconstrained optimization, but it?s fairly easy >> to assign bound constraints to it. The methodology is described in >> https://www.sciencedirect.com/science/article/pii/S0305054814001774 , >> while the Matlab source code can be found in >> http://www1.se.cuhk.edu.hk/~ckng/generator/ . I have used the Matlab >> script to generate 240 problems with dimensionality varying from 2 to 5 by >> outputting the generator parameters in text files, then used Python to >> create the objective functions based on those parameters and the benchmark >> methodology. >> >>> >> >>> MPM2: Implementing the ?Multiple Peaks Model 2?, there is a Python >> implementation at >> https://github.com/jakobbossek/smoof/blob/master/inst/mpm2.py . This is >> a test problem generator also used in the smoof library, I have taken the >> code almost as is and generated 480 benchmark functions with dimensionality >> varying from 2 to 5. >> >>> >> >>> RandomFields: as described in >> https://www.researchgate.net/publication/301940420_Global_optimization_test_problems_based_on_random_field_composition >> , it generates benchmark functions by ?smoothing? one or more >> multidimensional discrete random fields and composing them. No source code >> is given, but the implementation is fairly straightforward from the article >> itself. >> >>> >> >>> NIST: not exactly the realm of Global Optimization solvers, but the >> NIST StRD dataset can be used to generate a single objective function as >> ?sum of squares?. I have used the NIST dataset as implemented in lmfit, >> thus creating 27 test problems with dimensionality ranging from 2 to 9. >> >>> >> >>> GlobalLib: Arnold Neumaier maintains a suite of test problems termed >> ?COCONUT Benchmark? and Sahinidis has converted the GlobalLib and >> PricentonLib AMPL/GAMS dataset into C/Fortran code ( >> http://archimedes.cheme.cmu.edu/?q=dfocomp ). I have used a simple C >> parser to convert the benchmarks from C to Python. >> >>> >> >>> The global minima are taken from Sahinidis or from Neumaier or >> refined using the NEOS server when the accuracy of the reported minima is >> too low. The suite contains 181 test functions with dimensionality varying >> between 2 and 9. >> >>> >> >>> CVMG: another ?landscape generator?, I had to dig it out using the >> Wayback Machine at >> http://web.archive.org/web/20100612044104/https://www.cs.uwyo.edu/~wspears/multi.kennedy.html >> , the acronym stands for ?Continuous Valued Multimodality Generator?. >> Source code is in C++ but it?s fairly easy to port it to Python. In >> addition to the original implementation (that uses the Sigmoid as a >> softmax/transformation function) I have added a few others to create varied >> landscapes. 360 test problems have been generated, with dimensionality >> ranging from 2 to 5. >> >>> >> >>> NLSE: again, not really the realm of Global optimization solvers, but >> Nonlinear Systems of Equations can be transformed to single objective >> functions to optimize. I have drawn from many different sources >> (Publications, ALIAS/COPRIN and many others) to create 44 systems of >> nonlinear equations with dimensionality ranging from 2 to 8. >> >>> >> >>> Schoen: based on the early work of Fabio Schoen and his short note on >> a simple but interesting idea on a test function generator, I have taken >> the C code in the note and converted it into Python, thus creating 285 >> benchmark functions with dimensionality ranging from 2 to 6. >> >>> >> >>> Many thanks go to Professor Fabio Schoen for providing an updated >> copy of the source code and for the email communications. >> >>> >> >>> Robust: the last benchmark test suite for this exercise, it is >> actually composed of 5 different kind-of analytical test function >> generators, containing deceptive, multimodal, flat functions depending on >> the settings. Matlab source code is available at >> http://www.alimirjalili.com/RO.html , I simply converted it to Python >> and created 420 benchmark functions with dimensionality ranging from 2 to 6. >> >>> >> >>> >> >>> Enjoy, and Happy 2021 :-) . >> >>> >> >>> >> >>> Andrea. >> >>> >> >>> _______________________________________________ >> >>> >> >>> >> >>> SciPy-Dev mailing list >> >>> SciPy-Dev at python.org >> >>> https://mail.python.org/mailman/listinfo/scipy-dev >> >> >> >> >> >> >> >> -- >> >> Stefan Endres (MEng, AMIChemE, BEng (Hons) Chemical Engineering) >> >> >> >> Wissenchaftlicher Mitarbeiter: Leibniz Institute for Materials >> Engineering IWT, Badgasteiner Stra?e 3, 28359 Bremen, Germany >> >> Work phone (DE): +49 (0) 421 218 51238 >> >> Cellphone (DE): +49 (0) 160 949 86417 >> >> Cellphone (ZA): +27 (0) 82 972 42 89 >> >> E-mail (work): s.endres at iwt.uni-bremen.de >> >> Website: https://stefan-endres.github.io/ >> >> _______________________________________________ >> >> SciPy-Dev mailing list >> >> SciPy-Dev at python.org >> >> https://mail.python.org/mailman/listinfo/scipy-dev >> > >> > _______________________________________________ >> > SciPy-Dev mailing list >> > SciPy-Dev at python.org >> > https://mail.python.org/mailman/listinfo/scipy-dev >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrea.gavana at gmail.com Mon May 24 03:36:20 2021 From: andrea.gavana at gmail.com (Andrea Gavana) Date: Mon, 24 May 2021 09:36:20 +0200 Subject: [SciPy-Dev] Global Optimization Benchmarks In-Reply-To: References: Message-ID: Hi Daniel, On Mon, 24 May 2021 at 09.23, Daniel Schmitz < danielschmitzsiegen at googlemail.com> wrote: > Hey Ralf, > > discussion for DIRECT was opened here: > https://github.com/scipy/scipy/issues/14121 and the scipydirect > maintainer was pinged: https://github.com/andim/scipydirect/issues/9 . > > About the general discussion regarding global optimization in SciPy: I am > working on a SciPy style API for NLopt in my free time (mostly missing > documentation at the moment) and would like to benchmark its MLSL and StoGO > algorithms using the SciPy benchmarks. Currently, this seems very > complicated as any new algorithm would have to be included into SciPy > first. Is there a way to circumvent this? Of course, Andrea's benchmark > suite would be the mother of all benchmarks if available ;). > I think it sounds like an very good idea to design a SciPy stile API for NLOpt. In my personal experience the two algorithms you mention (MLSL and StoGo) are generally less performant compared to the other ones available in NLOpt. That said, assuming enough interest was there and some spare time, I could easily rerun the whole benchmarks with those two methods as well. Andrea. > Cheers, > Daniel > > On Sun, 25 Apr 2021 at 22:32, Ralf Gommers wrote: > >> Hi Daniel, >> >> Sorry for the extremely delayed reply. >> >> >> On Thu, Mar 11, 2021 at 10:19 AM Daniel Schmitz < >> danielschmitzsiegen at googlemail.com> wrote: >> >>> Hey all, >>> >>> after Andrea's suggestions, I did an extensive github search and found >>> several global optimization python libraries which mimic the scipy >>> API, so that a user only has to change the import statements. Could it >>> be useful to add a page in the documentation of these? >>> >> >> This does sound useful. Probably not a whole page, more like a note in >> the `minimize` docstring with references I'd think. >> >> >>> Non exhaustive list: >>> >>> DIRECT: https://github.com/andim/scipydirect >>> DE/PSO/CMA-ES: https://github.com/keurfonluu/stochopy >>> PSO: https://github.com/jerrytheo/psopy >>> Powell's derivative free optimizers: https://www.pdfo.net/index.html >>> >>> As DIRECT was very competitive on some of Andrea's benchmarks, it >>> could be useful to mimic the scipydirect repo for inclusion into scipy >>> (MIT license). The code is unfortunately a f2py port of the original >>> Fortran implementation which has hard coded bounds on the number of >>> function evaluations (90.000) and iterations (6.000). Any opinions on >>> this? >>> >> >> This sounds like a good idea. Would you mind opening a GitHub issue with >> the feature request, so we keep track of this? Contacting the original >> author about this would also be useful; if the author would like to >> upstream their code, that'd be a good outcome. >> >> Re hard coded bounds, I assume those can be removed again without too >> much trouble. >> >> >>> >>> I personally am very impressed by biteopt's performance, and although >>> it ranked very high in other global optimization benchmarks there is >>> no formal paper on it yet. I understand the scipy guidelines in a way >>> that such a paper is a requisite for inclusion into scipy. >>> >> >> Well, we do have the very extensive benchmarks - if it does indeed >> perform better than what we have, then of course we'd be happy to add a new >> algorithm even if it doesn't have a paper. We use papers and the citations >> it has as an indication of usefulness only; anything that outperforms our >> existing code is clearly useful. >> >> Cheers, >> Ralf >> >> >> >>> >>> Best, >>> >>> Daniel >>> >>> Daniel >>> >>> >>> On Sun, 17 Jan 2021 at 14:33, Andrea Gavana >>> wrote: >>> > >>> > Hi Stefan, >>> > >>> > You?re most welcome :-) . I?m happy the experts in the community >>> are commenting and suggesting things, and constructive criticism is also >>> always welcome. >>> > >>> > On Sun, 17 Jan 2021 at 12.11, Stefan Endres >>> wrote: >>> >> >>> >> Dear Andrea, >>> >> >>> >> Thank you very much for this detailed analysis. I don't think I've >>> seen such a large collection of benchmark test suites or collection of DFO >>> algorithms since the publication by Rios and Sahinidis in 2013. Some >>> questions: >>> >> >>> >> Many of the commercial algorithms offer free licenses for >>> benchmarking problems of less than 10 dimensions. Would you be willing to >>> include some of these in your benchmarks at some point? It would be a great >>> reference to use. >>> > >>> > >>> > I?m definitely willing to include those commercial algorithms. The >>> test suite per se is almost completely automated, so it?s not that >>> complicated to add one or more solvers. I?m generally more inclined in >>> testing open source algorithms but there?s nothing stopping the inclusion >>> of commercial ones. >>> > >>> > I welcome any suggestions related to commercial solvers, as long as >>> they can run on Python 2 / Python 3 and on Windows (I might be able to >>> setup a Linux virtual machine if absolutely needed but that would defy part >>> of the purpose of the exercise - SHGO, Dual Annealing and the other SciPy >>> solvers run on all platforms that support SciPy). >>> > >>> >> The collection of test suites you've garnered could be immensely >>> useful for further algorithm development. Is there a possibility of >>> releasing the code publicly (presumably after you've published the results >>> in a journal)? >>> >> >>> >> In this case I would also like to volunteer to run some of the >>> commercial solvers on the benchmark suite. >>> >> It would also help to have a central repository for fixing bugs and >>> adding lower global minima when they are found (of which there are quite >>> few ). >>> > >>> > >>> > >>> > I?m still sorting out all the implications related to a potential >>> paper with my employer, but as far as I can see there shouldn?t be any >>> problem with that: assuming everything goes as it should, I will definitely >>> push for making the code open source. >>> > >>> > >>> >> >>> >> Comments on shgo: >>> >> >>> >> High RAM use in higher dimensions: >>> >> >>> >> In the higher dimensions the new simplicial sampling can be used (not >>> pushed to scipy yet; I still need to update some documentation before the >>> PR). This alleviates, but does not eliminate the memory leak issue. As >>> you've said SHGO is best suited to problems below 10 dimensions as any >>> higher leaves the realm of DFO problems and starts to enter the domain of >>> NLP problems. My personal preference in this case is to use the stochastic >>> algorithms (basinhopping and differential evolution) on problems where it >>> is known that a gradient based solver won't work. >>> >> >>> >> An exception to this "rule" is when special grey box information such >>> as symmetry of the objective function (something that can be supplied to >>> shgo to push the applicability of the algorithm up to ~100 variables) or >>> pre-computed bounds on the Lipschitz constants is known. >>> >> >>> >> In the symmetry case SHGO can solve these by supplying the `symmetry` >>> option (which was used in the previous benchmarks done by me for the JOGO >>> publication, although I did not specifically check if performance was >>> actually improved on those problems, but shgo did converge on all benchmark >>> problems in the scipy test suite). >>> >> >>> >> I have had a few reports of memory leaks from various users. I have >>> spoken to a few collaborators about the possibility of finding a Masters >>> student to cythonize some of the code or otherwise improve it. Hopefully, >>> this will happen in the summer semester of 2021. >>> > >>> > >>> > To be honest I wouldn?t be so concerned in general: SHGO is an >>> excellent global optimization algorithm and it consistently ranks at the >>> top, no matter what problems you throw at it. Together with Dual Annealing, >>> SciPy has gained two phenomenal nonlinear solvers and I?m very happy to see >>> that SciPy is now at the cutting edge of the open source global >>> optimization universe. >>> > >>> > Andrea. >>> > >>> >> Thank you again for compiling this large set of benchmark results. >>> >> >>> >> Best regards, >>> >> Stefan >>> >> On Fri, Jan 8, 2021 at 10:21 AM Andrea Gavana < >>> andrea.gavana at gmail.com> wrote: >>> >>> >>> >>> Dear SciPy Developers & Users, >>> >>> >>> >>> long time no see :-) . I thought to start 2021 with a bit of a >>> bang, to try and forget how bad 2020 has been... So I am happy to present >>> you with a revamped version of the Global Optimization Benchmarks from my >>> previous exercise in 2013. >>> >>> >>> >>> This new set of benchmarks pretty much superseeds - and greatly >>> expands - the previous analysis that you can find at this location: >>> http://infinity77.net/global_optimization/ . >>> >>> >>> >>> The approach I have taken this time is to select as many benchmark >>> test suites as possible: most of them are characterized by test function >>> generators, from which we can actually create almost an unlimited number of >>> unique test problems. Biggest news are: >>> >>> >>> >>> This whole exercise is made up of 6,825 test problems divided across >>> 16 different test suites: most of these problems are of low dimensionality >>> (2 to 6 variables) with a few benchmarks extending to 9+ variables. With >>> all the sensitivities performed during this exercise on those benchmarks, >>> the overall grand total number of functions evaluations stands at >>> 3,859,786,025 - close to 4 billion. Not bad. >>> >>> A couple of "new" optimization algorithms I have ported to Python: >>> >>> >>> >>> MCS: Multilevel Coordinate Search, it?s my translation to Python of >>> the original Matlab code from A. Neumaier and W. Huyer (giving then for >>> free also GLS and MINQ) I have added a few, minor improvements compared to >>> the original implementation. >>> >>> BiteOpt: BITmask Evolution OPTimization , I have converted the C++ >>> code into Python and added a few, minor modifications. >>> >>> >>> >>> >>> >>> Enough chatting for now. The 13 tested algorithms are described here: >>> >>> >>> >>> http://infinity77.net/go_2021/ >>> >>> >>> >>> High level description & results of the 16 benchmarks: >>> >>> >>> >>> http://infinity77.net/go_2021/thebenchmarks.html >>> >>> >>> >>> Each benchmark test suite has its own dedicated page, with more >>> detailed results and sensitivities. >>> >>> >>> >>> List of tested algorithms: >>> >>> >>> >>> AMPGO: Adaptive Memory Programming for Global Optimization: this is >>> my Python implementation of the algorithm described here: >>> >>> >>> >>> >>> http://leeds-faculty.colorado.edu/glover/fred%20pubs/416%20-%20AMP%20(TS)%20for%20Constrained%20Global%20Opt%20w%20Lasdon%20et%20al%20.pdf >>> >>> >>> >>> I have added a few improvements here and there based on my Master >>> Thesis work on the standard Tunnelling Algorithm of Levy, Montalvo and >>> Gomez. After AMPGO was integrated in lmfit, I have improved it even more - >>> in my opinion. >>> >>> >>> >>> BasinHopping: Basin hopping is a random algorithm which attempts to >>> find the global minimum of a smooth scalar function of one or more >>> variables. The algorithm was originally described by David Wales: >>> >>> >>> >>> http://www-wales.ch.cam.ac.uk/ >>> >>> >>> >>> BasinHopping is now part of the standard SciPy distribution. >>> >>> >>> >>> BiteOpt: BITmask Evolution OPTimization, based on the algorithm >>> presented in this GitHub link: >>> >>> >>> >>> https://github.com/avaneev/biteopt >>> >>> >>> >>> I have converted the C++ code into Python and added a few, minor >>> modifications. >>> >>> >>> >>> CMA-ES: Covariance Matrix Adaptation Evolution Strategy, based on >>> the following algorithm: >>> >>> >>> >>> http://www.lri.fr/~hansen/cmaesintro.html >>> >>> >>> >>> http://www.lri.fr/~hansen/cmaes_inmatlab.html#python (Python code >>> for the algorithm) >>> >>> >>> >>> CRS2: Controlled Random Search with Local Mutation, as implemented >>> in the NLOpt package: >>> >>> >>> >>> >>> http://ab-initio.mit.edu/wiki/index.php/NLopt_Algorithms#Controlled_Random_Search_.28CRS.29_with_local_mutation >>> >>> >>> >>> DE: Differential Evolution, described in the following page: >>> >>> >>> >>> http://www1.icsi.berkeley.edu/~storn/code.html >>> >>> >>> >>> DE is now part of the standard SciPy distribution, and I have taken >>> the implementation as it stands in SciPy: >>> >>> >>> >>> >>> https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.differential_evolution.html#scipy.optimize.differential_evolution >>> >>> >>> >>> DIRECT: the DIviding RECTangles procedure, described in: >>> >>> >>> >>> >>> https://www.tol-project.org/export/2776/tolp/OfficialTolArchiveNetwork/NonLinGloOpt/doc/DIRECT_Lipschitzian%20optimization%20without%20the%20lipschitz%20constant.pdf >>> >>> >>> >>> >>> http://ab-initio.mit.edu/wiki/index.php/NLopt_Algorithms#DIRECT_and_DIRECT-L >>> (Python code for the algorithm) >>> >>> >>> >>> DualAnnealing: the Dual Annealing algorithm, taken directly from the >>> SciPy implementation: >>> >>> >>> >>> >>> https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.dual_annealing.html#scipy.optimize.dual_annealing >>> >>> >>> >>> LeapFrog: the Leap Frog procedure, which I have been recommended for >>> use, taken from: >>> >>> >>> >>> https://github.com/flythereddflagg/lpfgopt >>> >>> >>> >>> MCS: Multilevel Coordinate Search, it?s my translation to Python of >>> the original Matlab code from A. Neumaier and W. Huyer (giving then for >>> free also GLS and MINQ): >>> >>> >>> >>> https://www.mat.univie.ac.at/~neum/software/mcs/ >>> >>> >>> >>> I have added a few, minor improvements compared to the original >>> implementation. See the MCS section for a quick and dirty comparison >>> between the Matlab code and my Python conversion. >>> >>> >>> >>> PSWARM: Particle Swarm optimization algorithm, it has been described >>> in many online papers. I have used a compiled version of the C source code >>> from: >>> >>> >>> >>> http://www.norg.uminho.pt/aivaz/pswarm/ >>> >>> >>> >>> SCE: Shuffled Complex Evolution, described in: >>> >>> >>> >>> Duan, Q., S. Sorooshian, and V. Gupta, Effective and efficient >>> global optimization for conceptual rainfall-runoff models, Water Resour. >>> Res., 28, 1015-1031, 1992. >>> >>> >>> >>> The version I used was graciously made available by Matthias Cuntz >>> via a personal e-mail. >>> >>> >>> >>> SHGO: Simplicial Homology Global Optimization, taken directly from >>> the SciPy implementation: >>> >>> >>> >>> >>> https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.shgo.html#scipy.optimize.shgo >>> >>> >>> >>> >>> >>> List of benchmark test suites: >>> >>> >>> >>> SciPy Extended: 235 multivariate problems (where the number of >>> independent variables ranges from 2 to 17), again with multiple >>> local/global minima. >>> >>> >>> >>> I have added about 40 new functions to the standard SciPy benchmarks >>> and fixed a few bugs in the existing benchmark models in the SciPy >>> repository. >>> >>> >>> >>> GKLS: 1,500 test functions, with dimensionality varying from 2 to 6, >>> generated with the super famous GKLS Test Functions Generator. I have taken >>> the original C code (available at http://netlib.org/toms/) and >>> converted it to Python. >>> >>> >>> >>> GlobOpt: 288 tough problems, with dimensionality varying from 2 to >>> 5, created with another test function generator which I arbitrarily named >>> ?GlobOpt?: >>> https://www.researchgate.net/publication/225566516_A_new_class_of_test_functions_for_global_optimization >>> . The original code is in C++ and I have bridged it to Python using Cython. >>> >>> >>> >>> Many thanks go to Professor Marco Locatelli for providing an updated >>> copy of the C++ source code. >>> >>> >>> >>> MMTFG: sort-of an acronym for ?Multi-Modal Test Function with >>> multiple Global minima?, this test suite implements the work of Jani >>> Ronkkonen: >>> https://www.researchgate.net/publication/220265526_A_Generator_for_Multimodal_Test_Functions_with_Multiple_Global_Optima >>> . It contains 981 test problems with dimensionality varying from 2 to 4. >>> The original code is in C and I have bridge it to Python using Cython. >>> >>> >>> >>> GOTPY: a generator of benchmark functions using the >>> Bocharov-Feldbaum ?Method-Min?, containing 400 test problems with >>> dimensionality varying from 2 to 5. I have taken the Python implementation >>> from https://github.com/redb0/gotpy and improved it in terms of runtime. >>> >>> >>> >>> Original paper from >>> http://www.mathnet.ru/php/archive.phtml?wshow=paper&jrnid=at&paperid=11985&option_lang=eng >>> . >>> >>> >>> >>> Huygens: this benchmark suite is very different from the rest, as it >>> uses a ?fractal? approach to generate test functions. It is based on the >>> work of Cara MacNish on Fractal Functions. The original code is in Java, >>> and at the beginning I just converted it to Python: given it was slow as a >>> turtle, I have re-implemented it in Fortran and wrapped it using f2py, then >>> generating 600 2-dimensional test problems out of it. >>> >>> >>> >>> LGMVG: not sure about the meaning of the acronym, but the >>> implementation follows the ?Max-Set of Gaussians Landscape Generator? >>> described in http://boyuan.global-optimization.com/LGMVG/index.htm . >>> Source code is given in Matlab, but it?s fairly easy to convert it to >>> Python. This test suite contains 304 problems with dimensionality varying >>> from 2 to 5. >>> >>> >>> >>> NgLi: Stemming from the work of Chi-Kong Ng and Duan Li, this is a >>> test problem generator for unconstrained optimization, but it?s fairly easy >>> to assign bound constraints to it. The methodology is described in >>> https://www.sciencedirect.com/science/article/pii/S0305054814001774 , >>> while the Matlab source code can be found in >>> http://www1.se.cuhk.edu.hk/~ckng/generator/ . I have used the Matlab >>> script to generate 240 problems with dimensionality varying from 2 to 5 by >>> outputting the generator parameters in text files, then used Python to >>> create the objective functions based on those parameters and the benchmark >>> methodology. >>> >>> >>> >>> MPM2: Implementing the ?Multiple Peaks Model 2?, there is a Python >>> implementation at >>> https://github.com/jakobbossek/smoof/blob/master/inst/mpm2.py . This is >>> a test problem generator also used in the smoof library, I have taken the >>> code almost as is and generated 480 benchmark functions with dimensionality >>> varying from 2 to 5. >>> >>> >>> >>> RandomFields: as described in >>> https://www.researchgate.net/publication/301940420_Global_optimization_test_problems_based_on_random_field_composition >>> , it generates benchmark functions by ?smoothing? one or more >>> multidimensional discrete random fields and composing them. No source code >>> is given, but the implementation is fairly straightforward from the article >>> itself. >>> >>> >>> >>> NIST: not exactly the realm of Global Optimization solvers, but the >>> NIST StRD dataset can be used to generate a single objective function as >>> ?sum of squares?. I have used the NIST dataset as implemented in lmfit, >>> thus creating 27 test problems with dimensionality ranging from 2 to 9. >>> >>> >>> >>> GlobalLib: Arnold Neumaier maintains a suite of test problems termed >>> ?COCONUT Benchmark? and Sahinidis has converted the GlobalLib and >>> PricentonLib AMPL/GAMS dataset into C/Fortran code ( >>> http://archimedes.cheme.cmu.edu/?q=dfocomp ). I have used a simple C >>> parser to convert the benchmarks from C to Python. >>> >>> >>> >>> The global minima are taken from Sahinidis or from Neumaier or >>> refined using the NEOS server when the accuracy of the reported minima is >>> too low. The suite contains 181 test functions with dimensionality varying >>> between 2 and 9. >>> >>> >>> >>> CVMG: another ?landscape generator?, I had to dig it out using the >>> Wayback Machine at >>> http://web.archive.org/web/20100612044104/https://www.cs.uwyo.edu/~wspears/multi.kennedy.html >>> , the acronym stands for ?Continuous Valued Multimodality Generator?. >>> Source code is in C++ but it?s fairly easy to port it to Python. In >>> addition to the original implementation (that uses the Sigmoid as a >>> softmax/transformation function) I have added a few others to create varied >>> landscapes. 360 test problems have been generated, with dimensionality >>> ranging from 2 to 5. >>> >>> >>> >>> NLSE: again, not really the realm of Global optimization solvers, >>> but Nonlinear Systems of Equations can be transformed to single objective >>> functions to optimize. I have drawn from many different sources >>> (Publications, ALIAS/COPRIN and many others) to create 44 systems of >>> nonlinear equations with dimensionality ranging from 2 to 8. >>> >>> >>> >>> Schoen: based on the early work of Fabio Schoen and his short note >>> on a simple but interesting idea on a test function generator, I have taken >>> the C code in the note and converted it into Python, thus creating 285 >>> benchmark functions with dimensionality ranging from 2 to 6. >>> >>> >>> >>> Many thanks go to Professor Fabio Schoen for providing an updated >>> copy of the source code and for the email communications. >>> >>> >>> >>> Robust: the last benchmark test suite for this exercise, it is >>> actually composed of 5 different kind-of analytical test function >>> generators, containing deceptive, multimodal, flat functions depending on >>> the settings. Matlab source code is available at >>> http://www.alimirjalili.com/RO.html , I simply converted it to Python >>> and created 420 benchmark functions with dimensionality ranging from 2 to 6. >>> >>> >>> >>> >>> >>> Enjoy, and Happy 2021 :-) . >>> >>> >>> >>> >>> >>> Andrea. >>> >>> >>> >>> _______________________________________________ >>> >>> >>> >>> >>> >>> SciPy-Dev mailing list >>> >>> SciPy-Dev at python.org >>> >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> >>> >> >>> >> >>> >> -- >>> >> Stefan Endres (MEng, AMIChemE, BEng (Hons) Chemical Engineering) >>> >> >>> >> Wissenchaftlicher Mitarbeiter: Leibniz Institute for Materials >>> Engineering IWT, Badgasteiner Stra?e 3, 28359 Bremen, Germany >>> >>> >> Work phone (DE): +49 (0) 421 218 51238 >>> >> Cellphone (DE): +49 (0) 160 949 86417 >>> >> Cellphone (ZA): +27 (0) 82 972 42 89 >>> >> E-mail (work): s.endres at iwt.uni-bremen.de >>> >> Website: https://stefan-endres.github.io/ >>> >> _______________________________________________ >>> >> SciPy-Dev mailing list >>> >> SciPy-Dev at python.org >>> >> https://mail.python.org/mailman/listinfo/scipy-dev >>> > >>> > _______________________________________________ >>> > SciPy-Dev mailing list >>> > SciPy-Dev at python.org >>> > https://mail.python.org/mailman/listinfo/scipy-dev >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon May 24 13:12:59 2021 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 24 May 2021 11:12:59 -0600 Subject: [SciPy-Dev] NumPy 1.21.0rc1 has been released. Message-ID: Hi All, On behalf of the NumPy team I am pleased to announce the release of NumPy 1.21.0rc1. The highlights are - continued SIMD work covering more functions and platforms, - initial work on the new dtype infrastructure and casting, - improved documentation, - improved annotations, - the new ``PCG64DXSM`` bitgenerator for random numbers. This NumPy release is the result of 561 merged pull requests contributed by 171 people. The Python versions supported for this release are 3.7-3.9, support for Python 3.10 will be added after Python 3.10 is released. Wheels can be downloaded from PyPI ; source archives, release notes, and wheel hashes are available on Github . Linux users will need pip >= 0.19.3 in order to install manylinux2010 and manylinux2014 wheels. *Contributors* A total of 171 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - @8bitmp3 + - @DWesl + - @Endolith - @Illviljan + - @Lbogula + - @Lisa + - @Patrick + - @Scian + - @h-vetinari + - @h6197627 + - @jbCodeHub + - @legoffant + - @sfolje0 + - @tautaus + - @yetanothercheer + - Abhay Raghuvanshi + - Adrian Price-Whelan + - Aerik Pawson + - Agbonze Osazuwa + - Aitik Gupta + - Al-Baraa El-Hag - Alex Henrie - Alexander Hunt + - Aliz? Papp + - Allan Haldane - Amarnath1904 + - Amrit Krishnan + - Andras Deak - AngelGris + - Anne Archibald - Anthony Vo + - Antony Lee - Atharva-Vidwans + - Ayush Verma + - Bas van Beek - Bharat Raghunathan - Bhargav V + - Brian Soto - Carl Michal + - Charles Harris - Charles Stern + - Chiara Marmo + - Chris Barnes + - Chris Vavaliaris - Christina Hedges + - Christoph Gohlke - Christopher Dahlin + - Christos Efstathiou + - Chunlin Fang - Constanza Fierro + - Daniel Evans + - Daniel Montes + - Dario Mory + - David Carlier + - David Stansby - Deepyaman Datta + - Derek Homeier - Dong Keun Oh + - Dylan Cutler + - Eric Larson - Eric Wieser - Eva Jau + - Evgeni Burovski - FX Coudert + - Faris A Chugthai + - Filip Ter + - Filip Trojan + - Fran?ois Le Lay + - Ganesh Kathiresan - Giannis Zapantis + - Giulio Procopio + - Greg Lucas + - Hollow Man + - Holly Corbett + - Inessa Pawson - Isabela Presedo-Floyd - Ismael Jimenez + - Isuru Fernando - Jakob Jakobson - James Gerity + - Jamie Macey + - Jasmin Classen + - Jody Klymak + - Joseph Fox-Rabinovitz - J?rome Eertmans + - Kamil Choudhury + - Kasia Leszek + - Keller Meier + - Kevin Sheppard - Kulin Seth + - Kumud Lakara + - Laura Kopf + - Laura Martens + - Leo Singer + - Leonardus Chen + - Lima Tango + - Lumir Balhar + - Maia Kaplan + - Mainak Debnath + - Marco Aur?lio da Costa + - Marta Lemanczyk + - Marten van Kerkwijk - Mary Conley + - Marysia Winkels + - Mateusz Sok?? + - Matt Haberland - Matt Hall + - Matt Ord + - Matthew Badin + - Matthias Bussonnier - Matthias Geier - Matti Picus - Mat?as R?os + - Maxim Belkin + - Melissa Weber Mendon?a - Meltem Eren Copur + - Michael Dubravski + - Michael Lamparski - Michal W. Tarnowski + - Micha? G?rny + - Mike Boyle + - Mike Toews - Misal Raj + - Mitchell Faas + - Mukulikaa Parhari + - Neil Girdhar + - Nicholas McKibben + - Nico Schl?mer - Nicolas Hug + - Nilo Kruchelski + - Nirjas Jakilim + - Ohad Ravid + - Olivier Grisel - Pamphile ROY + - Panos Mavrogiorgos + - Patrick T. Komiske III + - Pearu Peterson - Raghuveer Devulapalli - Ralf Gommers - Ra?l Mont?n Pinillos + - Rin Arakaki + - Robert Kern - Rohit Sanjay - Roman Yurchak - Ronan Lamy - Ross Barnowski - Ryan C Cooper - Ryan Polley + - Ryan Soklaski - Sabrina Simao + - Sayed Adel - Sebastian Berg - Shen Zhou + - Stefan van der Walt - Sylwester Arabas + - Takanori Hirano - Tania Allard + - Thomas J. Fan + - Thomas Orgis + - Tim Hoffmann - Tomoki, Karatsu + - Tong Zou + - Touqir Sajed + - Tyler Reddy - Wansoo Kim - Warren Weckesser - Weh Andreas + - Yang Hau - Yashasvi Misra + - Zolboo Erdenebaatar + - Zolisa Bleki Cheers, Charles Harris -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue May 25 05:10:36 2021 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 25 May 2021 11:10:36 +0200 Subject: [SciPy-Dev] Global Optimization Benchmarks In-Reply-To: References: Message-ID: On Mon, May 24, 2021 at 9:36 AM Andrea Gavana wrote: > Hi Daniel, > > On Mon, 24 May 2021 at 09.23, Daniel Schmitz < > danielschmitzsiegen at googlemail.com> wrote: > >> Hey Ralf, >> >> discussion for DIRECT was opened here: >> https://github.com/scipy/scipy/issues/14121 and the scipydirect >> maintainer was pinged: https://github.com/andim/scipydirect/issues/9 . >> > Thanks! I commented on the issue about having checked the license info and what I think needs to be done. >> About the general discussion regarding global optimization in SciPy: I am >> working on a SciPy style API for NLopt in my free time (mostly missing >> documentation at the moment) and would like to benchmark its MLSL and StoGO >> algorithms using the SciPy benchmarks. Currently, this seems very >> complicated as any new algorithm would have to be included into SciPy >> first. Is there a way to circumvent this? Of course, Andrea's benchmark >> suite would be the mother of all benchmarks if available ;). >> > > I think it sounds like an very good idea to design a SciPy stile API for > NLOpt. > This seems valuable for LGPL'd algorithms that we cannot include directly. Having an optional dependency also has maintenance and CI costs, so for BSD/MIT-licensed algorithms I'd say we prefer to vendor them rather than have a scikit-umfpack like optional dependency. Unless there's really a ton of code - but if it's one or two algorithms, then vendoring seems like the way to go. Cheers, Ralf > In my personal experience the two algorithms you mention (MLSL and StoGo) > are generally less performant compared to the other ones available in > NLOpt. > > That said, assuming enough interest was there and some spare time, I could > easily rerun the whole benchmarks with those two methods as well. > > Andrea. > > > > >> Cheers, >> Daniel >> >> On Sun, 25 Apr 2021 at 22:32, Ralf Gommers >> wrote: >> >>> Hi Daniel, >>> >>> Sorry for the extremely delayed reply. >>> >>> >>> On Thu, Mar 11, 2021 at 10:19 AM Daniel Schmitz < >>> danielschmitzsiegen at googlemail.com> wrote: >>> >>>> Hey all, >>>> >>>> after Andrea's suggestions, I did an extensive github search and found >>>> several global optimization python libraries which mimic the scipy >>>> API, so that a user only has to change the import statements. Could it >>>> be useful to add a page in the documentation of these? >>>> >>> >>> This does sound useful. Probably not a whole page, more like a note in >>> the `minimize` docstring with references I'd think. >>> >>> >>>> Non exhaustive list: >>>> >>>> DIRECT: https://github.com/andim/scipydirect >>>> DE/PSO/CMA-ES: https://github.com/keurfonluu/stochopy >>>> PSO: https://github.com/jerrytheo/psopy >>>> Powell's derivative free optimizers: https://www.pdfo.net/index.html >>>> >>>> As DIRECT was very competitive on some of Andrea's benchmarks, it >>>> could be useful to mimic the scipydirect repo for inclusion into scipy >>>> (MIT license). The code is unfortunately a f2py port of the original >>>> Fortran implementation which has hard coded bounds on the number of >>>> function evaluations (90.000) and iterations (6.000). Any opinions on >>>> this? >>>> >>> >>> This sounds like a good idea. Would you mind opening a GitHub issue with >>> the feature request, so we keep track of this? Contacting the original >>> author about this would also be useful; if the author would like to >>> upstream their code, that'd be a good outcome. >>> >>> Re hard coded bounds, I assume those can be removed again without too >>> much trouble. >>> >>> >>>> >>>> I personally am very impressed by biteopt's performance, and although >>>> it ranked very high in other global optimization benchmarks there is >>>> no formal paper on it yet. I understand the scipy guidelines in a way >>>> that such a paper is a requisite for inclusion into scipy. >>>> >>> >>> Well, we do have the very extensive benchmarks - if it does indeed >>> perform better than what we have, then of course we'd be happy to add a new >>> algorithm even if it doesn't have a paper. We use papers and the citations >>> it has as an indication of usefulness only; anything that outperforms our >>> existing code is clearly useful. >>> >>> Cheers, >>> Ralf >>> >>> >>> >>>> >>>> Best, >>>> >>>> Daniel >>>> >>>> Daniel >>>> >>>> >>>> On Sun, 17 Jan 2021 at 14:33, Andrea Gavana >>>> wrote: >>>> > >>>> > Hi Stefan, >>>> > >>>> > You?re most welcome :-) . I?m happy the experts in the community >>>> are commenting and suggesting things, and constructive criticism is also >>>> always welcome. >>>> > >>>> > On Sun, 17 Jan 2021 at 12.11, Stefan Endres < >>>> stefan.c.endres at gmail.com> wrote: >>>> >> >>>> >> Dear Andrea, >>>> >> >>>> >> Thank you very much for this detailed analysis. I don't think I've >>>> seen such a large collection of benchmark test suites or collection of DFO >>>> algorithms since the publication by Rios and Sahinidis in 2013. Some >>>> questions: >>>> >> >>>> >> Many of the commercial algorithms offer free licenses for >>>> benchmarking problems of less than 10 dimensions. Would you be willing to >>>> include some of these in your benchmarks at some point? It would be a great >>>> reference to use. >>>> > >>>> > >>>> > I?m definitely willing to include those commercial algorithms. The >>>> test suite per se is almost completely automated, so it?s not that >>>> complicated to add one or more solvers. I?m generally more inclined in >>>> testing open source algorithms but there?s nothing stopping the inclusion >>>> of commercial ones. >>>> > >>>> > I welcome any suggestions related to commercial solvers, as long as >>>> they can run on Python 2 / Python 3 and on Windows (I might be able to >>>> setup a Linux virtual machine if absolutely needed but that would defy part >>>> of the purpose of the exercise - SHGO, Dual Annealing and the other SciPy >>>> solvers run on all platforms that support SciPy). >>>> > >>>> >> The collection of test suites you've garnered could be immensely >>>> useful for further algorithm development. Is there a possibility of >>>> releasing the code publicly (presumably after you've published the results >>>> in a journal)? >>>> >> >>>> >> In this case I would also like to volunteer to run some of the >>>> commercial solvers on the benchmark suite. >>>> >> It would also help to have a central repository for fixing bugs and >>>> adding lower global minima when they are found (of which there are quite >>>> few ). >>>> > >>>> > >>>> > >>>> > I?m still sorting out all the implications related to a potential >>>> paper with my employer, but as far as I can see there shouldn?t be any >>>> problem with that: assuming everything goes as it should, I will definitely >>>> push for making the code open source. >>>> > >>>> > >>>> >> >>>> >> Comments on shgo: >>>> >> >>>> >> High RAM use in higher dimensions: >>>> >> >>>> >> In the higher dimensions the new simplicial sampling can be used >>>> (not pushed to scipy yet; I still need to update some documentation before >>>> the PR). This alleviates, but does not eliminate the memory leak issue. As >>>> you've said SHGO is best suited to problems below 10 dimensions as any >>>> higher leaves the realm of DFO problems and starts to enter the domain of >>>> NLP problems. My personal preference in this case is to use the stochastic >>>> algorithms (basinhopping and differential evolution) on problems where it >>>> is known that a gradient based solver won't work. >>>> >> >>>> >> An exception to this "rule" is when special grey box information >>>> such as symmetry of the objective function (something that can be supplied >>>> to shgo to push the applicability of the algorithm up to ~100 variables) or >>>> pre-computed bounds on the Lipschitz constants is known. >>>> >> >>>> >> In the symmetry case SHGO can solve these by supplying the >>>> `symmetry` option (which was used in the previous benchmarks done by me for >>>> the JOGO publication, although I did not specifically check if performance >>>> was actually improved on those problems, but shgo did converge on all >>>> benchmark problems in the scipy test suite). >>>> >> >>>> >> I have had a few reports of memory leaks from various users. I have >>>> spoken to a few collaborators about the possibility of finding a Masters >>>> student to cythonize some of the code or otherwise improve it. Hopefully, >>>> this will happen in the summer semester of 2021. >>>> > >>>> > >>>> > To be honest I wouldn?t be so concerned in general: SHGO is an >>>> excellent global optimization algorithm and it consistently ranks at the >>>> top, no matter what problems you throw at it. Together with Dual Annealing, >>>> SciPy has gained two phenomenal nonlinear solvers and I?m very happy to see >>>> that SciPy is now at the cutting edge of the open source global >>>> optimization universe. >>>> > >>>> > Andrea. >>>> > >>>> >> Thank you again for compiling this large set of benchmark results. >>>> >> >>>> >> Best regards, >>>> >> Stefan >>>> >> On Fri, Jan 8, 2021 at 10:21 AM Andrea Gavana < >>>> andrea.gavana at gmail.com> wrote: >>>> >>> >>>> >>> Dear SciPy Developers & Users, >>>> >>> >>>> >>> long time no see :-) . I thought to start 2021 with a bit of a >>>> bang, to try and forget how bad 2020 has been... So I am happy to present >>>> you with a revamped version of the Global Optimization Benchmarks from my >>>> previous exercise in 2013. >>>> >>> >>>> >>> This new set of benchmarks pretty much superseeds - and greatly >>>> expands - the previous analysis that you can find at this location: >>>> http://infinity77.net/global_optimization/ . >>>> >>> >>>> >>> The approach I have taken this time is to select as many benchmark >>>> test suites as possible: most of them are characterized by test function >>>> generators, from which we can actually create almost an unlimited number of >>>> unique test problems. Biggest news are: >>>> >>> >>>> >>> This whole exercise is made up of 6,825 test problems divided >>>> across 16 different test suites: most of these problems are of low >>>> dimensionality (2 to 6 variables) with a few benchmarks extending to 9+ >>>> variables. With all the sensitivities performed during this exercise on >>>> those benchmarks, the overall grand total number of functions evaluations >>>> stands at 3,859,786,025 - close to 4 billion. Not bad. >>>> >>> A couple of "new" optimization algorithms I have ported to Python: >>>> >>> >>>> >>> MCS: Multilevel Coordinate Search, it?s my translation to Python of >>>> the original Matlab code from A. Neumaier and W. Huyer (giving then for >>>> free also GLS and MINQ) I have added a few, minor improvements compared to >>>> the original implementation. >>>> >>> BiteOpt: BITmask Evolution OPTimization , I have converted the C++ >>>> code into Python and added a few, minor modifications. >>>> >>> >>>> >>> >>>> >>> Enough chatting for now. The 13 tested algorithms are described >>>> here: >>>> >>> >>>> >>> http://infinity77.net/go_2021/ >>>> >>> >>>> >>> High level description & results of the 16 benchmarks: >>>> >>> >>>> >>> http://infinity77.net/go_2021/thebenchmarks.html >>>> >>> >>>> >>> Each benchmark test suite has its own dedicated page, with more >>>> detailed results and sensitivities. >>>> >>> >>>> >>> List of tested algorithms: >>>> >>> >>>> >>> AMPGO: Adaptive Memory Programming for Global Optimization: this is >>>> my Python implementation of the algorithm described here: >>>> >>> >>>> >>> >>>> http://leeds-faculty.colorado.edu/glover/fred%20pubs/416%20-%20AMP%20(TS)%20for%20Constrained%20Global%20Opt%20w%20Lasdon%20et%20al%20.pdf >>>> >>> >>>> >>> I have added a few improvements here and there based on my Master >>>> Thesis work on the standard Tunnelling Algorithm of Levy, Montalvo and >>>> Gomez. After AMPGO was integrated in lmfit, I have improved it even more - >>>> in my opinion. >>>> >>> >>>> >>> BasinHopping: Basin hopping is a random algorithm which attempts to >>>> find the global minimum of a smooth scalar function of one or more >>>> variables. The algorithm was originally described by David Wales: >>>> >>> >>>> >>> http://www-wales.ch.cam.ac.uk/ >>>> >>> >>>> >>> BasinHopping is now part of the standard SciPy distribution. >>>> >>> >>>> >>> BiteOpt: BITmask Evolution OPTimization, based on the algorithm >>>> presented in this GitHub link: >>>> >>> >>>> >>> https://github.com/avaneev/biteopt >>>> >>> >>>> >>> I have converted the C++ code into Python and added a few, minor >>>> modifications. >>>> >>> >>>> >>> CMA-ES: Covariance Matrix Adaptation Evolution Strategy, based on >>>> the following algorithm: >>>> >>> >>>> >>> http://www.lri.fr/~hansen/cmaesintro.html >>>> >>> >>>> >>> http://www.lri.fr/~hansen/cmaes_inmatlab.html#python (Python code >>>> for the algorithm) >>>> >>> >>>> >>> CRS2: Controlled Random Search with Local Mutation, as implemented >>>> in the NLOpt package: >>>> >>> >>>> >>> >>>> http://ab-initio.mit.edu/wiki/index.php/NLopt_Algorithms#Controlled_Random_Search_.28CRS.29_with_local_mutation >>>> >>> >>>> >>> DE: Differential Evolution, described in the following page: >>>> >>> >>>> >>> http://www1.icsi.berkeley.edu/~storn/code.html >>>> >>> >>>> >>> DE is now part of the standard SciPy distribution, and I have taken >>>> the implementation as it stands in SciPy: >>>> >>> >>>> >>> >>>> https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.differential_evolution.html#scipy.optimize.differential_evolution >>>> >>> >>>> >>> DIRECT: the DIviding RECTangles procedure, described in: >>>> >>> >>>> >>> >>>> https://www.tol-project.org/export/2776/tolp/OfficialTolArchiveNetwork/NonLinGloOpt/doc/DIRECT_Lipschitzian%20optimization%20without%20the%20lipschitz%20constant.pdf >>>> >>> >>>> >>> >>>> http://ab-initio.mit.edu/wiki/index.php/NLopt_Algorithms#DIRECT_and_DIRECT-L >>>> (Python code for the algorithm) >>>> >>> >>>> >>> DualAnnealing: the Dual Annealing algorithm, taken directly from >>>> the SciPy implementation: >>>> >>> >>>> >>> >>>> https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.dual_annealing.html#scipy.optimize.dual_annealing >>>> >>> >>>> >>> LeapFrog: the Leap Frog procedure, which I have been recommended >>>> for use, taken from: >>>> >>> >>>> >>> https://github.com/flythereddflagg/lpfgopt >>>> >>> >>>> >>> MCS: Multilevel Coordinate Search, it?s my translation to Python of >>>> the original Matlab code from A. Neumaier and W. Huyer (giving then for >>>> free also GLS and MINQ): >>>> >>> >>>> >>> https://www.mat.univie.ac.at/~neum/software/mcs/ >>>> >>> >>>> >>> I have added a few, minor improvements compared to the original >>>> implementation. See the MCS section for a quick and dirty comparison >>>> between the Matlab code and my Python conversion. >>>> >>> >>>> >>> PSWARM: Particle Swarm optimization algorithm, it has been >>>> described in many online papers. I have used a compiled version of the C >>>> source code from: >>>> >>> >>>> >>> http://www.norg.uminho.pt/aivaz/pswarm/ >>>> >>> >>>> >>> SCE: Shuffled Complex Evolution, described in: >>>> >>> >>>> >>> Duan, Q., S. Sorooshian, and V. Gupta, Effective and efficient >>>> global optimization for conceptual rainfall-runoff models, Water Resour. >>>> Res., 28, 1015-1031, 1992. >>>> >>> >>>> >>> The version I used was graciously made available by Matthias Cuntz >>>> via a personal e-mail. >>>> >>> >>>> >>> SHGO: Simplicial Homology Global Optimization, taken directly from >>>> the SciPy implementation: >>>> >>> >>>> >>> >>>> https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.shgo.html#scipy.optimize.shgo >>>> >>> >>>> >>> >>>> >>> List of benchmark test suites: >>>> >>> >>>> >>> SciPy Extended: 235 multivariate problems (where the number of >>>> independent variables ranges from 2 to 17), again with multiple >>>> local/global minima. >>>> >>> >>>> >>> I have added about 40 new functions to the standard SciPy >>>> benchmarks and fixed a few bugs in the existing benchmark models in the >>>> SciPy repository. >>>> >>> >>>> >>> GKLS: 1,500 test functions, with dimensionality varying from 2 to >>>> 6, generated with the super famous GKLS Test Functions Generator. I have >>>> taken the original C code (available at http://netlib.org/toms/) and >>>> converted it to Python. >>>> >>> >>>> >>> GlobOpt: 288 tough problems, with dimensionality varying from 2 to >>>> 5, created with another test function generator which I arbitrarily named >>>> ?GlobOpt?: >>>> https://www.researchgate.net/publication/225566516_A_new_class_of_test_functions_for_global_optimization >>>> . The original code is in C++ and I have bridged it to Python using Cython. >>>> >>> >>>> >>> Many thanks go to Professor Marco Locatelli for providing an >>>> updated copy of the C++ source code. >>>> >>> >>>> >>> MMTFG: sort-of an acronym for ?Multi-Modal Test Function with >>>> multiple Global minima?, this test suite implements the work of Jani >>>> Ronkkonen: >>>> https://www.researchgate.net/publication/220265526_A_Generator_for_Multimodal_Test_Functions_with_Multiple_Global_Optima >>>> . It contains 981 test problems with dimensionality varying from 2 to 4. >>>> The original code is in C and I have bridge it to Python using Cython. >>>> >>> >>>> >>> GOTPY: a generator of benchmark functions using the >>>> Bocharov-Feldbaum ?Method-Min?, containing 400 test problems with >>>> dimensionality varying from 2 to 5. I have taken the Python implementation >>>> from https://github.com/redb0/gotpy and improved it in terms of >>>> runtime. >>>> >>> >>>> >>> Original paper from >>>> http://www.mathnet.ru/php/archive.phtml?wshow=paper&jrnid=at&paperid=11985&option_lang=eng >>>> . >>>> >>> >>>> >>> Huygens: this benchmark suite is very different from the rest, as >>>> it uses a ?fractal? approach to generate test functions. It is based on the >>>> work of Cara MacNish on Fractal Functions. The original code is in Java, >>>> and at the beginning I just converted it to Python: given it was slow as a >>>> turtle, I have re-implemented it in Fortran and wrapped it using f2py, then >>>> generating 600 2-dimensional test problems out of it. >>>> >>> >>>> >>> LGMVG: not sure about the meaning of the acronym, but the >>>> implementation follows the ?Max-Set of Gaussians Landscape Generator? >>>> described in http://boyuan.global-optimization.com/LGMVG/index.htm . >>>> Source code is given in Matlab, but it?s fairly easy to convert it to >>>> Python. This test suite contains 304 problems with dimensionality varying >>>> from 2 to 5. >>>> >>> >>>> >>> NgLi: Stemming from the work of Chi-Kong Ng and Duan Li, this is a >>>> test problem generator for unconstrained optimization, but it?s fairly easy >>>> to assign bound constraints to it. The methodology is described in >>>> https://www.sciencedirect.com/science/article/pii/S0305054814001774 , >>>> while the Matlab source code can be found in >>>> http://www1.se.cuhk.edu.hk/~ckng/generator/ . I have used the Matlab >>>> script to generate 240 problems with dimensionality varying from 2 to 5 by >>>> outputting the generator parameters in text files, then used Python to >>>> create the objective functions based on those parameters and the benchmark >>>> methodology. >>>> >>> >>>> >>> MPM2: Implementing the ?Multiple Peaks Model 2?, there is a Python >>>> implementation at >>>> https://github.com/jakobbossek/smoof/blob/master/inst/mpm2.py . This >>>> is a test problem generator also used in the smoof library, I have taken >>>> the code almost as is and generated 480 benchmark functions with >>>> dimensionality varying from 2 to 5. >>>> >>> >>>> >>> RandomFields: as described in >>>> https://www.researchgate.net/publication/301940420_Global_optimization_test_problems_based_on_random_field_composition >>>> , it generates benchmark functions by ?smoothing? one or more >>>> multidimensional discrete random fields and composing them. No source code >>>> is given, but the implementation is fairly straightforward from the article >>>> itself. >>>> >>> >>>> >>> NIST: not exactly the realm of Global Optimization solvers, but the >>>> NIST StRD dataset can be used to generate a single objective function as >>>> ?sum of squares?. I have used the NIST dataset as implemented in lmfit, >>>> thus creating 27 test problems with dimensionality ranging from 2 to 9. >>>> >>> >>>> >>> GlobalLib: Arnold Neumaier maintains a suite of test problems >>>> termed ?COCONUT Benchmark? and Sahinidis has converted the GlobalLib and >>>> PricentonLib AMPL/GAMS dataset into C/Fortran code ( >>>> http://archimedes.cheme.cmu.edu/?q=dfocomp ). I have used a simple C >>>> parser to convert the benchmarks from C to Python. >>>> >>> >>>> >>> The global minima are taken from Sahinidis or from Neumaier or >>>> refined using the NEOS server when the accuracy of the reported minima is >>>> too low. The suite contains 181 test functions with dimensionality varying >>>> between 2 and 9. >>>> >>> >>>> >>> CVMG: another ?landscape generator?, I had to dig it out using the >>>> Wayback Machine at >>>> http://web.archive.org/web/20100612044104/https://www.cs.uwyo.edu/~wspears/multi.kennedy.html >>>> , the acronym stands for ?Continuous Valued Multimodality Generator?. >>>> Source code is in C++ but it?s fairly easy to port it to Python. In >>>> addition to the original implementation (that uses the Sigmoid as a >>>> softmax/transformation function) I have added a few others to create varied >>>> landscapes. 360 test problems have been generated, with dimensionality >>>> ranging from 2 to 5. >>>> >>> >>>> >>> NLSE: again, not really the realm of Global optimization solvers, >>>> but Nonlinear Systems of Equations can be transformed to single objective >>>> functions to optimize. I have drawn from many different sources >>>> (Publications, ALIAS/COPRIN and many others) to create 44 systems of >>>> nonlinear equations with dimensionality ranging from 2 to 8. >>>> >>> >>>> >>> Schoen: based on the early work of Fabio Schoen and his short note >>>> on a simple but interesting idea on a test function generator, I have taken >>>> the C code in the note and converted it into Python, thus creating 285 >>>> benchmark functions with dimensionality ranging from 2 to 6. >>>> >>> >>>> >>> Many thanks go to Professor Fabio Schoen for providing an updated >>>> copy of the source code and for the email communications. >>>> >>> >>>> >>> Robust: the last benchmark test suite for this exercise, it is >>>> actually composed of 5 different kind-of analytical test function >>>> generators, containing deceptive, multimodal, flat functions depending on >>>> the settings. Matlab source code is available at >>>> http://www.alimirjalili.com/RO.html , I simply converted it to Python >>>> and created 420 benchmark functions with dimensionality ranging from 2 to 6. >>>> >>> >>>> >>> >>>> >>> Enjoy, and Happy 2021 :-) . >>>> >>> >>>> >>> >>>> >>> Andrea. >>>> >>> >>>> >>> _______________________________________________ >>>> >>> >>>> >>> >>>> >>> SciPy-Dev mailing list >>>> >>> SciPy-Dev at python.org >>>> >>> https://mail.python.org/mailman/listinfo/scipy-dev >>>> >> >>>> >> >>>> >> >>>> >> -- >>>> >> Stefan Endres (MEng, AMIChemE, BEng (Hons) Chemical Engineering) >>>> >> >>>> >> Wissenchaftlicher Mitarbeiter: Leibniz Institute for Materials >>>> Engineering IWT, Badgasteiner Stra?e 3, 28359 Bremen, Germany >>>> >>>> >> Work phone (DE): +49 (0) 421 218 51238 >>>> >> Cellphone (DE): +49 (0) 160 949 86417 >>>> >> Cellphone (ZA): +27 (0) 82 972 42 89 >>>> >> E-mail (work): s.endres at iwt.uni-bremen.de >>>> >> Website: https://stefan-endres.github.io/ >>>> >> _______________________________________________ >>>> >> SciPy-Dev mailing list >>>> >> SciPy-Dev at python.org >>>> >> https://mail.python.org/mailman/listinfo/scipy-dev >>>> > >>>> > _______________________________________________ >>>> > SciPy-Dev mailing list >>>> > SciPy-Dev at python.org >>>> > https://mail.python.org/mailman/listinfo/scipy-dev >>>> _______________________________________________ >>>> SciPy-Dev mailing list >>>> SciPy-Dev at python.org >>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyler.je.reddy at gmail.com Thu May 27 01:30:07 2021 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Wed, 26 May 2021 23:30:07 -0600 Subject: [SciPy-Dev] master branch open for development of SciPy 1.8.0 Message-ID: I've branched maintenance/1.7.x and master branch is now open for the development of SciPy 1.8.0. (There's a strange Windows-only CI failure after bumping to 1.8.0, but it is getting late) Best wishes, Tyler -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyler.je.reddy at gmail.com Thu May 27 18:22:58 2021 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Thu, 27 May 2021 16:22:58 -0600 Subject: [SciPy-Dev] Proposed 1.7.0 Release Schedule In-Reply-To: References: <043BDEF9-8DA3-4FB1-B134-11F804776119@gmail.com> Message-ID: I'll probably try to bump rc1 forward into early-mid next week because of the long weekend here in the US. Maybe a few more backport/bug fixes will trickle in as well. Tyler On Sun, 23 May 2021 at 18:17, Tyler Reddy wrote: > The draft release notes are ready for review/suggestions: > https://github.com/scipy/scipy/pull/14118 > > We have about 20 open PRs with milestone, which is a bit high this close > to branching: > https://github.com/scipy/scipy/pulls?q=is%3Aopen+is%3Apr+milestone%3A1.7.0 > > Quite a few are marked "approved." Some are likely to get bumped--I'm > probably going to get a bit stricter tomorrow with bumping PRs that aren't > on a clear trajectory to merging by Wednesday based on the comments or > available expert reviewer bandwidth, CI issues that haven't been fixed, etc. > > Best wishes, > Tyler > > On Sun, 16 May 2021 at 19:02, Andrew Nelson wrote: > >> On Sun, 16 May 2021 at 22:14, Pamphile Roy >> wrote: >> >>> Regarding Optimize. It seems that lots of PRs and issues are pointing >>> towards an overhaul of the module. My position would be to solve what can >>> be solved for the release with open PRs and for the next release we should >>> lay out a plan to evolve the module. >>> >> >> It'd certainly be good for the number of active optimize maintainers to >> increase, to tackle PRs and issues, etc. >> >> optimize is probably one of the most heavily used parts of scipy, that's >> why it gets a lot of traffic on github. I think the module is probably in >> not too bad a good shape. A review of outstanding issues would give an idea >> of what types of issues are commonly experienced, it's not clear that a >> large scale redesign would necessarily change anything. >> >> One thing I'm mainly interested in minimisations of scalar functions. I >> had a desire to implement a class-based minimization system for those, but >> I'm fairly convinced it would have to be developed outside scipy in a >> spin-off project first. >> >> >> >> >>> What do you think Andrew? Could we plan a specific meeting for that >>> maybe? >>> >>> Cheers, >>> Pamphile >>> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Fri May 28 16:08:59 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Fri, 28 May 2021 22:08:59 +0200 Subject: [SciPy-Dev] SciPy Community Meetups are back! In-Reply-To: <600FCAD2-05F9-45A8-8130-8A2C0EBABA9C@gmail.com> References: <1605BE4D-E642-4CD7-848A-5FA3BBE56982@gmail.com> <222F4B45-F11E-4142-9BD6-36AEC27D9A29@gmail.com> <151F12E8-BC7B-40D9-A27D-462FF3567873@gmail.com> <600FCAD2-05F9-45A8-8130-8A2C0EBABA9C@gmail.com> Message-ID: Hi everyone, Just a friendly reminder to indicate your preferences for the time and date of the next meetups. Also, feel free to comment and suggest topics to discuss. Cheers, Pamphile > On 17.05.2021, at 20:06, Pamphile Roy wrote: > > Thanks everyone for today?s meetup. > It was nice to finally meet you all and I found it quite productive! > > Thank you Matt for taking notes during the meeting (same link): https://hackmd.io/@tupui/scipy-meetup/edit > For those of you who could not join, we also recorded the session (thank you Ralf) and we will see to make it available. > > We proposed to have regular meetups every 2 weeks and change the time zone every meetup (EU/Africa/Americas and Pacific/Asia/EU). > So that anyone could join once a month. The first thing to decide, in these respective meetings, would be to pick a day and time not to have to > do a pool all the time. > > Until then, the proposed new meetups dates would be > > Pacific/Asia/EU: https://doodle.com/poll/tmnenfc9c9v6c42z > EU/Africa/Americas: https://doodle.com/poll/9cse6mt58cp9d6px > > The next meetup notes are here: https://hackmd.io/@tupui/scipy-meetup-2/edit > > Thank you for your help and participation. And if there is any question or issue, feel free to drop me a line. > > Cheers, > Pamphile > > >> On 08.05.2021, at 10:45, Pamphile Roy > wrote: >> >> Hi everyone, >> >> I forgot to include the password, here is the complete link: >> >> https://zoom.us/j/93827717491?pwd=UGdkMzlnWEFjbDNQUmxpRkp6L0VLdz09 >> >> Meeting ID: 938 2771 7491 >> Passcode: 602749 >> >> Also, thank you to Ralf for his help and providing us with this link! >> >> Cheers, >> Pamphile >> >> >> >>> On 07.05.2021, at 23:17, Pamphile Roy wrote: >>> >>> Hi everyone, >>> >>> First of all, thank you for your participation to the pool! >>> >>> You have been the most to vote for May 17th at 16 CET. >>> >>> https://zoom.us/j/93827717491 >>> >>> Please feel free to contribute to the agenda: >>> >>> https://hackmd.io/@tupui/scipy-meetup/edit >>> >>> In case you cannot join us, I am thinking about recording this event. >>> Of course, before any recording starts, we will discuss about it. >>> >>> I will see you there, and I am looking forward to finally meet you all. >>> In the meantime, don?t hesitate to get in touch with me =) >>> >>> Cheers, >>> Pamphile >>> >>> >>>> On 04.05.2021, at 12:21, Pamphile Roy wrote: >>>> >>>> I updated the doodle to propose 7a.m. CET. >>>> >>>> Cheers, >>>> Pamphile >>>> >>>>> On 04.05.2021, at 12:08, Pamphile Roy wrote: >>>>> >>>>> I am not sure if we can solve this. From the table below, the only solution I could see would be 6 or 7a.m. CET. >>>>> >>>>> I can add other times or I could propose that we rotate the time from one meeting to the other (might be the most reasonable). >>>>> >>>>> Cheers, >>>>> Pamphile >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> On 04.05.2021, at 11:56, Andrew Nelson wrote: >>>>>> >>>>>> All those times are pretty much impossible for those in AUS/NZ. >>>>>> >>>>>> On Tue, 4 May 2021, 19:53 Pamphile Roy, wrote: >>>>>> Hi everyone, >>>>>> >>>>>> This is a friendly reminder to complete the doodle to plan the meetup. >>>>>> >>>>>> If not otherwise stated, I am planning on fixing the date by the end of this week (May 7). >>>>>> >>>>>>> https://doodle.com/poll/fd62d6755texie5s >>>>>> >>>>>> Thank you all for your help and engagement. >>>>>> >>>>>> Cheers, >>>>>> Pamphile >>>>>> >>>>>> >>>>>>> On 30.04.2021, at 11:48, Pamphile Roy wrote: >>>>>>> >>>>>>> Hi everyone, >>>>>>> >>>>>>> I would like to propose regular SciPy Community meetings again! >>>>>>> >>>>>>> https://doodle.com/poll/fd62d6755texie5s >>>>>>> >>>>>>> For this first meeting, it would be good if as many people as possible could join as we would decide things like frequency of such meeting. >>>>>>> >>>>>>> Everyone is invited and encouraged to >>>>>>> join in and edit the work-in-progress meeting topics and notes at: >>>>>>> >>>>>>> https://hackmd.io/@tupui/scipy-meetup/edit >>>>>>> >>>>>>> Cheers, >>>>>>> Pamphile >>>>>>> >>>>>>> PS. share the news: https://twitter.com/PamphileRoy/status/1388067651721830401?s=20 >>>>>> >>>>>> _______________________________________________ >>>>>> SciPy-Dev mailing list >>>>>> SciPy-Dev at python.org >>>>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>>>> _______________________________________________ >>>>>> SciPy-Dev mailing list >>>>>> SciPy-Dev at python.org >>>>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njgodber at gmail.com Sat May 29 03:00:50 2021 From: njgodber at gmail.com (Neil Godber) Date: Sat, 29 May 2021 17:00:50 +1000 Subject: [SciPy-Dev] Enhancement Request: SuperLU_dist/SuperLU_mt replace current scipy.sparse.superLU In-Reply-To: References: Message-ID: Hi Ralf + ScipyDev, Dr Li has helpfully uploaded the SuperLU_mt code base to github https://github.com/xiaoyeli/superlu_mt . Dr Li indicated that most development effort is focused on _dt, but that sequential and mt remain supported and bug reports will be attended to. Cheers, Neil On Mon, 24 May 2021 at 11:09, Neil Godber wrote: > Hi Ralf, > > Thanks for your email and happy to assist in whatever way I can. I've > reached out to Dr Xiaoyei Li of Lawrence Berkley who appears to be the > principal maintainer of the code base with a request to host the _mt code > on github. Given the other two versions are already hosted I can't see that > being an issue. I've also queried whether developments are done in sync and > if so, if there is a pending patching for mt in the similar fashion to > sequential. > > I will revert with further information as I receive it. > > Cheers, > Neil > > On Mon, 24 May 2021 at 04:33, Ralf Gommers wrote: > >> >> >> On Fri, May 21, 2021 at 2:11 AM Neil Godber wrote: >> >>> Hi dev-list, >>> >>> Further last, realised that I failed to link to the issue itself despite >>> linking to virtually everything else: >>> >>> Replace superLU sequential with superLU dist ? Issue #14096 ? >>> scipy/scipy (github.com) >>> >>> Cheers, >>> Neil >>> >>> On Fri, 21 May 2021 at 10:04, Neil Godber wrote: >>> >>>> Hi dev-list, >>>> >>> >> Thanks for the proposal Neil. >> >> >>>> I've logged an enhancement request on github to bring the >>>> parallel versions of SuperLU into scipy. As requested I'm circulating the >>>> enhancement here to solicit comment/feedback/engagement. >>>> >>>> Content of request is as follows (with light editing and links): >>>> >>>> "Is your feature request related to a problem? Please describe. >>>> >>>> The direct solution of sparse matrices is a common problem that arises >>>> across many domains. Current scipy.sparse.linalg provides spsolve and splu >>>> to solve such classes of problems. Neither solution appears to utilise more >>>> than one or two threads leaving a lot of the available compute on modern >>>> machines unused. Other solutions (MUMPS, Pardiso) exploit multicore >>>> processors but are either minimally supported and/or locked behind >>>> proprietary libraries. >>>> >>>> Describe the solution you'd like >>>> Fortunately SuperLU has two multithreaded implementations superLU_mt >>>> and superLU_dt, both actively maintained (SuperLU: Home Page >>>> (nersc.gov) ). >>>> Ideally SciPy would replace the current sequential implementation with one >>>> of the parallel implementations to yield large performance improvements. >>>> >>> The relevant one seems to be superLU_mt. The `_dt` flavor is for >> distributed computing, which is out of scope for SciPy. superLU_mt says it >> has both pthreads and OpenMP interfaces - we'd prefer pthreads (and >> actually forbid OpenMP within SciPy). >> >> The last release of superLU_mt is v3.1, from March 2016. Our current copy >> of SuperLU is 5.2.1, from May 2016 - but then there's a bunch of patches >> applied to it, and then https://github.com/scipy/scipy/pull/12243 did a >> sync with SuperLU master in May 2020. So it would be necessary to figure >> out if we can get superLU_mt from upstream master as well, and that we >> don't lose bug fixes we applied since 2016. Could you look into that Neil? >> And do you know if the parallel and sequential versions are developed in >> sync or not? >> >> Cheers, >> Ralf >> >> >> Describe alternatives you've considered >>>> pyMKL (Pardiso dwfmarchant/pyMKL: Python wrappers to Intel MKL >>>> routines (github.com) ) - MKL >>>> with issues associated with that for non Intel processors, unmaintained >>>> relatively to scipy, PyTrilinos (implements all three superLU types but >>>> only available on osx and linux PyTrilinos | Trilinos >>>> ), petsc (petsc4py PETSc / >>>> petsc ? GitLab ) (implements all three >>>> superLU types but only available on osx and linux). All are relatively >>>> inaccessible compared to ubiquity and accessibility of scipy. >>>> >>>> Additional context (e.g. screenshots) >>>> Anecdotally large number of projects use various non scipy sparse >>>> solvers to achieve high level of performance. This comes with considerable >>>> time and maintenance investment. It would be fantastic for scipy to provide >>>> a 'performance competitive' sparse direct solver out of the hood as it does >>>> for dense matrices." >>>> >>>> >>>> Cheers, >>>> >>>> Neil >>>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njgodber at gmail.com Sat May 29 19:40:22 2021 From: njgodber at gmail.com (Neil Godber) Date: Sun, 30 May 2021 09:40:22 +1000 Subject: [SciPy-Dev] Enhancement Request: SuperLU_dist/SuperLU_mt replace current scipy.sparse.superLU In-Reply-To: References: Message-ID: Hi Ralf, Saw your comment on the issue. Is there anything else I can do to help? Unfortunately my C/C++ skills are lacking so I'm not sure how much help I on the actual integration itself. Cheers, Neil On Sat, 29 May 2021 at 17:00, Neil Godber wrote: > Hi Ralf + ScipyDev, > > Dr Li has helpfully uploaded the SuperLU_mt code base to github > https://github.com/xiaoyeli/superlu_mt . Dr Li indicated that most > development effort is focused on _dt, but that sequential and mt remain > supported and bug reports will be attended to. > > Cheers, > Neil > > On Mon, 24 May 2021 at 11:09, Neil Godber wrote: > >> Hi Ralf, >> >> Thanks for your email and happy to assist in whatever way I can. I've >> reached out to Dr Xiaoyei Li of Lawrence Berkley who appears to be the >> principal maintainer of the code base with a request to host the _mt code >> on github. Given the other two versions are already hosted I can't see that >> being an issue. I've also queried whether developments are done in sync and >> if so, if there is a pending patching for mt in the similar fashion to >> sequential. >> >> I will revert with further information as I receive it. >> >> Cheers, >> Neil >> >> On Mon, 24 May 2021 at 04:33, Ralf Gommers >> wrote: >> >>> >>> >>> On Fri, May 21, 2021 at 2:11 AM Neil Godber wrote: >>> >>>> Hi dev-list, >>>> >>>> Further last, realised that I failed to link to the issue itself >>>> despite linking to virtually everything else: >>>> >>>> Replace superLU sequential with superLU dist ? Issue #14096 ? >>>> scipy/scipy (github.com) >>>> >>>> Cheers, >>>> Neil >>>> >>>> On Fri, 21 May 2021 at 10:04, Neil Godber wrote: >>>> >>>>> Hi dev-list, >>>>> >>>> >>> Thanks for the proposal Neil. >>> >>> >>>>> I've logged an enhancement request on github to bring the >>>>> parallel versions of SuperLU into scipy. As requested I'm circulating the >>>>> enhancement here to solicit comment/feedback/engagement. >>>>> >>>>> Content of request is as follows (with light editing and links): >>>>> >>>>> "Is your feature request related to a problem? Please describe. >>>>> >>>>> The direct solution of sparse matrices is a common problem that arises >>>>> across many domains. Current scipy.sparse.linalg provides spsolve and splu >>>>> to solve such classes of problems. Neither solution appears to utilise more >>>>> than one or two threads leaving a lot of the available compute on modern >>>>> machines unused. Other solutions (MUMPS, Pardiso) exploit multicore >>>>> processors but are either minimally supported and/or locked behind >>>>> proprietary libraries. >>>>> >>>>> Describe the solution you'd like >>>>> Fortunately SuperLU has two multithreaded implementations superLU_mt >>>>> and superLU_dt, both actively maintained (SuperLU: Home Page >>>>> (nersc.gov) ). >>>>> Ideally SciPy would replace the current sequential implementation with one >>>>> of the parallel implementations to yield large performance improvements. >>>>> >>>> The relevant one seems to be superLU_mt. The `_dt` flavor is for >>> distributed computing, which is out of scope for SciPy. superLU_mt says it >>> has both pthreads and OpenMP interfaces - we'd prefer pthreads (and >>> actually forbid OpenMP within SciPy). >>> >>> The last release of superLU_mt is v3.1, from March 2016. Our current >>> copy of SuperLU is 5.2.1, from May 2016 - but then there's a bunch of >>> patches applied to it, and then >>> https://github.com/scipy/scipy/pull/12243 did a sync with SuperLU >>> master in May 2020. So it would be necessary to figure out if we can get >>> superLU_mt from upstream master as well, and that we don't lose bug fixes >>> we applied since 2016. Could you look into that Neil? And do you know if >>> the parallel and sequential versions are developed in sync or not? >>> >>> Cheers, >>> Ralf >>> >>> >>> Describe alternatives you've considered >>>>> pyMKL (Pardiso dwfmarchant/pyMKL: Python wrappers to Intel MKL >>>>> routines (github.com) ) - MKL >>>>> with issues associated with that for non Intel processors, unmaintained >>>>> relatively to scipy, PyTrilinos (implements all three superLU types but >>>>> only available on osx and linux PyTrilinos | Trilinos >>>>> ), petsc (petsc4py PETSc >>>>> / petsc ? GitLab ) (implements all >>>>> three superLU types but only available on osx and linux). All are >>>>> relatively inaccessible compared to ubiquity and accessibility of scipy. >>>>> >>>>> Additional context (e.g. screenshots) >>>>> Anecdotally large number of projects use various non scipy sparse >>>>> solvers to achieve high level of performance. This comes with considerable >>>>> time and maintenance investment. It would be fantastic for scipy to provide >>>>> a 'performance competitive' sparse direct solver out of the hood as it does >>>>> for dense matrices." >>>>> >>>>> >>>>> Cheers, >>>>> >>>>> Neil >>>>> >>>> _______________________________________________ >>>> SciPy-Dev mailing list >>>> SciPy-Dev at python.org >>>> https://mail.python.org/mailman/listinfo/scipy-dev >>>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Sun May 30 17:17:11 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Sun, 30 May 2021 23:17:11 +0200 Subject: [SciPy-Dev] SciPy Community Meetups are back! Message-ID: <629AB633-82BD-4F03-BFD3-EB3F18A09E9E@gmail.com> Hi everyone, The next meeting (PAC/ASIA/EU) will be on June 3 (Thursday) at 9am CEST. Feel free to add some inputs/comments to the meeting notes. https://hackmd.io/@tupui/scipy-meetup-2/edit Cheers, Pamphile ps. I will try to be there but I have some last minute change... > On 28 May 2021, at 22:09, Pamphile Roy wrote: -------------- next part -------------- An HTML attachment was scrubbed... URL: