From zhaog6 at lsec.cc.ac.cn Wed Sep 1 22:31:53 2021 From: zhaog6 at lsec.cc.ac.cn (=?UTF-8?B?6LW15Yia?=) Date: Thu, 2 Sep 2021 10:31:53 +0800 (GMT+08:00) Subject: [SciPy-Dev] ENH: Add Conjugate Residual (CR) method Message-ID: <64f2b9e5.22777.17ba4592dfd.Coremail.zhaog6@lsec.cc.ac.cn> Dear all, I submitted a PR to support Conjugate Residual (CR) method: https://github.com/scipy/scipy/pull/14675 which can be used in symmetric indefinite or Hermitian systems.The algorithm is formally similar to Conjugate Gradient (CG) but mathematically equivalent to Minimal Residual (MINRES). For symmetric positive-definite problems, CR has the same convergence as CG. But for symmetric indefinite problems, CR has almost the same iterative steps and less computation in each iteration than MINRES, which means higher computational efficiency. The arrival of CR algorithm will further enrich the Krylov subspace methods implemented in SciPy and give users more/better choices. The PR is only a part of CR type methods which include CR, Bi-CR, Generalized CR, GCRO and GCRO-DR etc. This large class of algorithms are widely used in some specific problems and play an important role. So we intend to implement CR first, and then implement Bi-CR and GCR methods respectively. The PR used the functional features of NumPy and SciPy to implement the CR method. Hope you like it. Thanks. Best Regards, Gang From wusher at kth.se Thu Sep 2 09:11:43 2021 From: wusher at kth.se (William Usher) Date: Thu, 2 Sep 2021 13:11:43 +0000 Subject: [SciPy-Dev] Sensitivity analysis module proposal Message-ID: Hi; I?m one of the maintainers of SALib, a Python library which offers implementations of popular global sensitivity analysis approaches including Sobol?, Morris, DMIM, DGSM, FAST and others. We?ve been approached by roy.pamphile at gmail.com regarding the possibility of either - contributing the entire SALib library to SciPy, or - contributing key functions from SALib to SciPy, keeping SALib as a user-friendly wrapper around the SciPy implementations + extensions that don?t fit within SciPy?s remit. We believe that this move could overall benefit the scientific and sensitivity analysis communities. We?ve discussed this internally, and would be open to either of the above options. What are the next steps? Kind regards, Will Usher [cid:image001.jpg at 01D51FA4.648A23D0] Dr. Will Usher Assistant Professor in Integrated Assessment for Sustainable Development, KTH Royal Institute of Technology Division of Energy Systems School of Industrial Engineering and Management KTH Royal Institute of Technology Brinellv?gen 68 SE-100 44 Stockholm, Sweden wusher at kth.se +46 (0)8 790 74 61 https://www.energy.kth.se/research/energy-systems https://www.github.com/kth-desa https://www.github.com/osemosys https://www.github.com/salib -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 1976 bytes Desc: image001.jpg URL: From roy.pamphile at gmail.com Thu Sep 2 09:17:26 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Thu, 2 Sep 2021 15:17:26 +0200 Subject: [SciPy-Dev] Sensitivity analysis module proposal In-Reply-To: References: Message-ID: Hi Will, Thank you very much for this very positive reply. I will let other maintainers have a say as I am biased here. For reference, I initially started the discussion on the mailing list here https://mail.python.org/pipermail/scipy-dev/2021-April/024696.html If one or the other is approved. I would be more than happy to schedule a call and discuss next steps. I would be more than happy to lead this on SciPy?s side as I have experience in SA. Cheers, Pamphile > On 02.09.2021, at 15:11, William Usher wrote: > > Hi; > > I?m one of the maintainers of SALib , a Python library which offers implementations of popular global sensitivity analysis approaches including Sobol?, Morris, DMIM, DGSM, FAST and others. > > We?ve been approached by roy.pamphile at gmail.com regarding the possibility of either > > - contributing the entire SALib library to SciPy, or > - contributing key functions from SALib to SciPy, keeping SALib as a user-friendly wrapper around the SciPy implementations + extensions that don?t fit within SciPy?s remit. > > We believe that this move could overall benefit the scientific and sensitivity analysis communities. > > We?ve discussed this internally, and would be open to either of the above options. > > What are the next steps? > > Kind regards, > > Will Usher > > > > Dr. Will Usher > Assistant Professor in Integrated Assessment for Sustainable Development, KTH Royal Institute of Technology > > Division of Energy Systems > School of Industrial Engineering and Management > KTH Royal Institute of Technology > > Brinellv?gen 68 > SE-100 44 Stockholm, Sweden > wusher at kth.se > > +46 (0)8 790 74 61 > > https://www.energy.kth.se/research/energy-systems > https://www.github.com/kth-desa > https://www.github.com/osemosys > https://www.github.com/salib > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Sep 2 10:14:13 2021 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 2 Sep 2021 10:14:13 -0400 Subject: [SciPy-Dev] Sensitivity analysis module proposal In-Reply-To: References: Message-ID: On Thu, Sep 2, 2021 at 9:12 AM William Usher wrote: > Hi; > > I?m one of the maintainers of SALib , a > Python library which offers implementations of popular global sensitivity > analysis approaches including Sobol?, Morris, DMIM, DGSM, FAST and others. > > We?ve been approached by > roy.pamphile at gmail.com regarding the possibility of either > > - contributing the entire SALib library to SciPy, or > - contributing key functions from SALib to SciPy, keeping SALib as a > user-friendly wrapper around the SciPy implementations + extensions that > don?t fit within SciPy?s remit. > > We believe that this move could overall benefit the scientific and > sensitivity analysis communities. > I laid out my general rubric for considering absorbing functionality into scipy the last time this was brought up: https://mail.python.org/pipermail/scipy-dev/2021-April/024715.html Obviously, that you are interested in the proposal assuages some of that, but I'm still unclear on why you are interested in this. What is the benefit that you think everyone will get by absorbing SALib into scipy? It still looks to me like a mostly-lateral move that will merely be disruptive to your dependent projects more than anything else. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From wusher at kth.se Thu Sep 2 15:23:24 2021 From: wusher at kth.se (William Usher) Date: Thu, 2 Sep 2021 19:23:24 +0000 Subject: [SciPy-Dev] Sensitivity analysis module proposal (Robert Kern) In-Reply-To: References: Message-ID: <9BABFF4B-94B7-4E28-B9AA-B3E47DBC68B6@kth.se> Hi Robert, Thanks for the response. You raise good points. Obviously, that you are interested in the proposal assuages some of that, but I'm still unclear on why you are interested in this. What is the benefit that you think everyone will get by absorbing SALib into scipy? It still looks to me like a mostly-lateral move that will merely be disruptive to your dependent projects more than anything else. The real value of SALib is in providing a consistent interface to a (large) suite of sensitivity analysis methods which allows users to easily switch between those methods. We (as maintainers) could benefit from reducing duplication of code and implementations, such has Sobol? sequence generation, LHS, and could contribute some of the more general sample generation implementations where appropriate (many are linked directly to the SA implementations and not useful outside of that). We think the scientific community as a whole would benefit from the greater exposure a SciPy implementation of SA would bring - as a large community-led effort - it could provide a neutral forum for further development of these methods. This would likely come at some ?cost? to the successful ?cottage industry? we?ve established and grown (SALib is getting lots of citations and use). I think a key argument against integration is that it may reduce the agility with which we can add new methods to our SA suite (although this could be mitigated with careful design). I think you raise an important point about our dependent projects, and particularly how we would continue to support legacy releases if developer resources were focussed on a SciPy integration? An attraction is the possibility of funding to support the development of SA within SciPy. Like all open-source projects, we suffer from resourcing issues, and are predominantly volunteer-driven from the academic community. And while we are technically a ?multi-developer? community, we?re only a few bus accidents (or career changes) away from being a lone-maintainer. Will From: Robert Kern > Subject: Re: [SciPy-Dev] Sensitivity analysis module proposal Date: 2 September 2021 at 16:14:13 CEST To: SciPy Developers List > Cc: Jon Herman >, Takuya Iwanaga > On Thu, Sep 2, 2021 at 9:12 AM William Usher > wrote: Hi; I?m one of the maintainers of SALib, a Python library which offers implementations of popular global sensitivity analysis approaches including Sobol?, Morris, DMIM, DGSM, FAST and others. We?ve been approached by roy.pamphile at gmail.com regarding the possibility of either - contributing the entire SALib library to SciPy, or - contributing key functions from SALib to SciPy, keeping SALib as a user-friendly wrapper around the SciPy implementations + extensions that don?t fit within SciPy?s remit. We believe that this move could overall benefit the scientific and sensitivity analysis communities. I laid out my general rubric for considering absorbing functionality into scipy the last time this was brought up: https://mail.python.org/pipermail/scipy-dev/2021-April/024715.html Obviously, that you are interested in the proposal assuages some of that, but I'm still unclear on why you are interested in this. What is the benefit that you think everyone will get by absorbing SALib into scipy? It still looks to me like a mostly-lateral move that will merely be disruptive to your dependent projects more than anything else. -- Robert Kern _______________________________________________ SciPy-Dev mailing list SciPy-Dev at python.org https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilhanpolat at gmail.com Thu Sep 2 18:28:12 2021 From: ilhanpolat at gmail.com (Ilhan Polat) Date: Fri, 3 Sep 2021 00:28:12 +0200 Subject: [SciPy-Dev] Sensitivity analysis module proposal (Robert Kern) In-Reply-To: <9BABFF4B-94B7-4E28-B9AA-B3E47DBC68B6@kth.se> References: <9BABFF4B-94B7-4E28-B9AA-B3E47DBC68B6@kth.se> Message-ID: I am hesitant for exactly the points you raised. One is that once this goes in it's in for a decade. So your bus factor is critical for us too because who's going to maintain it if not you? We have some modules already in need of love and care for which sometimes we feel they don't belong to scipy. Moreover there's a lack of time and effort, in salibthis case would be also expertise. Second who is going to use this? I think it's far from a generic usage topic. And SAlib already made a name for itself anyways so whoever is benefiting from this finds the right place and we don't get requests for it to much. Adding to scipy seems like a compromise for both sides and hence lose-lose to me for these reasons On Thu, 2 Sep 2021, 21:30 William Usher, wrote: > Hi Robert, > > Thanks for the response. You raise good points. > > Obviously, that you are interested in the proposal assuages some of that, > but I'm still unclear on why you are interested in this. What is the > benefit that you think everyone will get by absorbing SALib into scipy? It > still looks to me like a mostly-lateral move that will merely be disruptive > to your dependent projects more than anything else. > > > The real value of SALib is in providing a consistent interface to a > (large) suite of sensitivity analysis methods which allows users to easily > switch between those methods. > > We (as maintainers) could benefit from reducing duplication of code and > implementations, such has Sobol? sequence generation, LHS, and could > contribute some of the more general sample generation implementations where > appropriate (many are linked directly to the SA implementations and not > useful outside of that). We think the scientific community as a whole > would benefit from the greater exposure a SciPy implementation of SA would > bring - as a large community-led effort - it could provide a neutral forum > for further development of these methods. This would likely come at some > ?cost? to the successful ?cottage industry? we?ve established and grown > (SALib is getting lots of citations and use). I think a key argument > against integration is that it may reduce the agility with which we can add > new methods to our SA suite (although this could be mitigated with careful > design). > > I think you raise an important point about our dependent projects, and > particularly how we would continue to support legacy releases if developer > resources were focussed on a SciPy integration? > > An attraction is the possibility of funding to support the development of > SA within SciPy. Like all open-source projects, we suffer from resourcing > issues, and are predominantly volunteer-driven from the academic > community. And while we are technically a ?multi-developer? community, > we?re only a few bus accidents (or career changes) away from being a > lone-maintainer. > > Will > > *From: *Robert Kern > *Subject: **Re: [SciPy-Dev] Sensitivity analysis module proposal* > *Date: *2 September 2021 at 16:14:13 CEST > *To: *SciPy Developers List > *Cc: *Jon Herman , Takuya Iwanaga < > iwanaga.takuya at anu.edu.au> > > > On Thu, Sep 2, 2021 at 9:12 AM William Usher wrote: > >> Hi; >> >> I?m one of the maintainers of SALib , a >> Python library which offers implementations of popular global sensitivity >> analysis approaches including Sobol?, Morris, DMIM, DGSM, FAST and others. >> >> We?ve been approached by >> roy.pamphile at gmail.com regarding the possibility of either >> >> - contributing the entire SALib library to SciPy, or >> - contributing key functions from SALib to SciPy, keeping SALib as a >> user-friendly wrapper around the SciPy implementations + extensions that >> don?t fit within SciPy?s remit. >> >> We believe that this move could overall benefit the scientific and >> sensitivity analysis communities. >> > > I laid out my general rubric for considering absorbing functionality into > scipy the last time this was brought up: > > https://mail.python.org/pipermail/scipy-dev/2021-April/024715.html > > Obviously, that you are interested in the proposal assuages some of that, > but I'm still unclear on why you are interested in this. What is the > benefit that you think everyone will get by absorbing SALib into scipy? It > still looks to me like a mostly-lateral move that will merely be disruptive > to your dependent projects more than anything else. > > -- > Robert Kern > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Sep 2 21:34:40 2021 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 2 Sep 2021 21:34:40 -0400 Subject: [SciPy-Dev] Sensitivity analysis module proposal (Robert Kern) In-Reply-To: <9BABFF4B-94B7-4E28-B9AA-B3E47DBC68B6@kth.se> References: <9BABFF4B-94B7-4E28-B9AA-B3E47DBC68B6@kth.se> Message-ID: On Thu, Sep 2, 2021 at 3:30 PM William Usher wrote: > Hi Robert, > > Thanks for the response. You raise good points. > > Obviously, that you are interested in the proposal assuages some of that, > but I'm still unclear on why you are interested in this. What is the > benefit that you think everyone will get by absorbing SALib into scipy? It > still looks to me like a mostly-lateral move that will merely be disruptive > to your dependent projects more than anything else. > > > The real value of SALib is in providing a consistent interface to a > (large) suite of sensitivity analysis methods which allows users to easily > switch between those methods. > > We (as maintainers) could benefit from reducing duplication of code and > implementations, such has Sobol? sequence generation, LHS, and could > contribute some of the more general sample generation implementations where > appropriate (many are linked directly to the SA implementations and not > useful outside of that). > I definitely think anything that could plausibly fall in the purview of scipy.stats.qmc would be a good target for convergence. If you can't use scipy.stats.qmc due to missing functionality (and not just that you don't want to require that recent version of scipy), then let's see how we can shore it up. I think that where your sampling methods overlap with design of experiments in general might also be a fruitful place for scipy.stats to grow/absorb some community-wide functionality. We think the scientific community as a whole would benefit from the > greater exposure a SciPy implementation of SA would bring - as a large > community-led effort - it could provide a neutral forum for further > development of these methods. This would likely come at some ?cost? to the > successful ?cottage industry? we?ve established and grown (SALib is getting > lots of citations and use). > That is gold. I am having trouble understanding why you would contemplate anything that might place that at risk. If I were dictator here, I would reject you out of hand for your own good. ;-) I think a key argument against integration is that it may reduce the > agility with which we can add new methods to our SA suite (although this > could be mitigated with careful design). > > I think you raise an important point about our dependent projects, and > particularly how we would continue to support legacy releases if developer > resources were focussed on a SciPy integration? > > An attraction is the possibility of funding to support the development of > SA within SciPy. Like all open-source projects, we suffer from resourcing > issues, and are predominantly volunteer-driven from the academic > community. And while we are technically a ?multi-developer? community, > we?re only a few bus accidents (or career changes) away from being a > lone-maintainer. > That's more required bus accidents than many parts of scipy. :-) I'm afraid that contributing into scipy doesn't unlock a pot of funds. I'm out of the grant-writing game, but I suspect that the difference between applying for funds on your own and arguing for allocation from competing needs inside of scipy is mostly a push. Have you considered applying to NumFocus? I occasionally look at their accepted and rejected projects, and I'd lump you in with the former, IMO. You're doing important work, playing well with other projects in the ecosystem, and have at least the seed kernel of sustainable community development so that funds are likely to actually sustain development. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Fri Sep 3 13:15:29 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Fri, 3 Sep 2021 19:15:29 +0200 Subject: [SciPy-Dev] Sensitivity analysis module proposal (Robert Kern) Message-ID: <1A590E8A-9602-49FB-A85F-65698B952111@gmail.com> Hi all, First things first, there are too many things to discuss and email is not appropriate. I propose to make a call with people interested next week or the week after. Let me know if we can do this and what time would work so I can put together a doodle. Some replies on specific points: Let me preface this by saying that I am not gaining anything from working on SciPy. I have no project related to SA at the moment and nothing is planned. I am only giving some personal time and have zero financial support for my time. SALib into SciPy: I proposed this following your input Robert. You said that if the maintainers of this library would be willing to do so, then we could discuss. I am very happy to see that they would actually be willing to do this and help. Funding: I can only disagree. Yes, being part of SciPy does not open directly a line of credit, but it does indirectly help. Art Owen already told me that he is using the fact that he helped us on stats.qmc to get some fundings. And I am certain some of us also use this with their current employers. Bus factor: We are way pass the bus factor. There is the team behind SALib, me, and I already got interest from people like Sergei Kucherenko (one of the pillar of SA). And if we go that way, I am 100% sure Saltelli (another pillar) will be interested (I worked with him on a tooling project). People like them are willing to contribute only because it?s SciPy. We had this discussion at MCM2021 when I presented SciPy?s QMC module to the QMC community. Researchers, at least from the communities I know and talked with, do not want to contribute to something that is uncertain or linked to a particular group. With SciPy they see an opportunity to have a large impact across a wide range of fields. And also it?s the assurance to have a long term impact. Why SA? How is this important? SA is becoming a major field. Not only by itself, but other fields are starting to use its methods. Things like Shapley values in AI, importance factor, etc. In most engineering field, it?s now mandatory to assess uncertainties. Hence there is a real need for tooling. There are mature libraries in the field. OpenTURNS, UQLab and Dakota are the most used among practitioners. And they are all not independent and open as we are (explains partially why people like Sergei or Saltelli are not contributors). I will not go here about why I think these library should not be used. I really don?t get the push back on this one. It?s about adding a few method in stats at most and the benefits would be arguably huge compared to some functions/sub modules we added recently. Behind it, there would be the most renown people of the field and that could give another great exposure for SciPy. We will talk of SciPy at new conferences, etc. My presentation of QMC had a great impact in the QMC community. I had 4 concrete proposal of collaboration just during the conference. We went from not being in one field to be the recommended tool by a community. SciPy is at the foundation of the scientific ecosystem in Python and not having basic tooling a about uncertainty/SA that other higher lever packages could rely on is a puzzle for me. As I said at the beginning, let?s have a talk to decide what?s in the best interest of SciPy. Cheers, Pamphile > On 3 Sep 2021, at 03:35, Robert Kern wrote: > > ? >> On Thu, Sep 2, 2021 at 3:30 PM William Usher wrote: > >> Hi Robert, >> >> Thanks for the response. You raise good points. >> >>>> Obviously, that you are interested in the proposal assuages some of that, but I'm still unclear on why you are interested in this. What is the benefit that you think everyone will get by absorbing SALib into scipy? It still looks to me like a mostly-lateral move that will merely be disruptive to your dependent projects more than anything else. >> >> The real value of SALib is in providing a consistent interface to a (large) suite of sensitivity analysis methods which allows users to easily switch between those methods. >> >> We (as maintainers) could benefit from reducing duplication of code and implementations, such has Sobol? sequence generation, LHS, and could contribute some of the more general sample generation implementations where appropriate (many are linked directly to the SA implementations and not useful outside of that). > > I definitely think anything that could plausibly fall in the purview of scipy.stats.qmc would be a good target for convergence. If you can't use scipy.stats.qmc due to missing functionality (and not just that you don't want to require that recent version of scipy), then let's see how we can shore it up. I think that where your sampling methods overlap with design of experiments in general might also be a fruitful place for scipy.stats to grow/absorb some community-wide functionality. > >> We think the scientific community as a whole would benefit from the greater exposure a SciPy implementation of SA would bring - as a large community-led effort - it could provide a neutral forum for further development of these methods. This would likely come at some ?cost? to the successful ?cottage industry? we?ve established and grown (SALib is getting lots of citations and use). > > That is gold. I am having trouble understanding why you would contemplate anything that might place that at risk. If I were dictator here, I would reject you out of hand for your own good. ;-) > >> I think a key argument against integration is that it may reduce the agility with which we can add new methods to our SA suite (although this could be mitigated with careful design). >> >> I think you raise an important point about our dependent projects, and particularly how we would continue to support legacy releases if developer resources were focussed on a SciPy integration? >> >> An attraction is the possibility of funding to support the development of SA within SciPy. Like all open-source projects, we suffer from resourcing issues, and are predominantly volunteer-driven from the academic community. And while we are technically a ?multi-developer? community, we?re only a few bus accidents (or career changes) away from being a lone-maintainer. > > > That's more required bus accidents than many parts of scipy. :-) > > I'm afraid that contributing into scipy doesn't unlock a pot of funds. I'm out of the grant-writing game, but I suspect that the difference between applying for funds on your own and arguing for allocation from competing needs inside of scipy is mostly a push. Have you considered applying to NumFocus? I occasionally look at their accepted and rejected projects, and I'd lump you in with the former, IMO. You're doing important work, playing well with other projects in the ecosystem, and have at least the seed kernel of sustainable community development so that funds are likely to actually sustain development. > > -- > Robert Kern > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Sep 3 13:31:42 2021 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 3 Sep 2021 13:31:42 -0400 Subject: [SciPy-Dev] Sensitivity analysis module proposal (Robert Kern) In-Reply-To: <1A590E8A-9602-49FB-A85F-65698B952111@gmail.com> References: <1A590E8A-9602-49FB-A85F-65698B952111@gmail.com> Message-ID: [private] On Fri, Sep 3, 2021 at 1:16 PM Pamphile Roy wrote: > > > Hi all, > > > First things first, there are too many things to discuss and email is not > appropriate. I propose to make a call with people interested next week or > the week after. Let me know if we can do this and what time would work so I > can put together a doodle. > > > Some replies on specific points: > This is not the first time that you have declared a forum inappropriate for discussion as a prelude to dropping a lengthy final argument in that same forum. That's not an acceptable way to treat your colleagues. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From teddy at mit.edu Fri Sep 3 19:12:20 2021 From: teddy at mit.edu (Teddy Ort) Date: Fri, 3 Sep 2021 19:12:20 -0400 Subject: [SciPy-Dev] PR's waiting for review Message-ID: Hi Scipy, A couple of weeks ago, I submitted two PRs to improve the performance of scipy.stats.binned_statistic. These are both quite small (fewer than 25 lines combined) and greatly improve the benchmark performance. If anyone has time to review, that would be great! https://github.com/scipy/scipy/pull/14625 https://github.com/scipy/scipy/pull/14629 Best, Teddy -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilhanpolat at gmail.com Wed Sep 8 07:25:01 2021 From: ilhanpolat at gmail.com (Ilhan Polat) Date: Wed, 8 Sep 2021 13:25:01 +0200 Subject: [SciPy-Dev] PR to add get_array_bandwidth, issymmetric, ishermitian functions Message-ID: Dear all, There is now a PR for scipy.linalg module including functions mentioned in the title. The idea is actually to enable a central place to provide these rather occasionally needed functionalities with some basic benchmarking available. If and when these are in then we can work on the context aware eig, solve functions much easier delegating the right structure to the right algorithm. Feedback is most welcome. https://github.com/scipy/scipy/pull/14701 best, ilhan -------------- next part -------------- An HTML attachment was scrubbed... URL: From serge.guelton at telecom-bretagne.eu Tue Sep 14 01:51:03 2021 From: serge.guelton at telecom-bretagne.eu (Serge Guelton) Date: Tue, 14 Sep 2021 07:51:03 +0200 Subject: [SciPy-Dev] Pythran release - 0.10.0 - palig Message-ID: <20210914055102.GA14782@sguelton.remote.csb> Hi Folks, Pythran, a compiler for scientific kernels written in Python, got a new release. It contains a few changes detailed in the changelog[0] but I'd like to highlight the work[1] of Xingyu Liu during her GSoC on improving pythran <-> scipy integration. It's also a release that recieved a lot of external contributions, and that's always a pleasure. Thanks to you all! $ git log 0.9.9..0.10.0 | grep Author | sort -u | wc -l 21 As usual, Pythran is available since today on PyPI and github, and this will propagate to conda, Fedora etc thanks to our beloved packagers! [0] https://github.com/serge-sans-paille/pythran/blob/0.10.0/Changelog [1] https://serge-sans-paille.github.io/pythran-stories/gsoc21-improve-performance-through-the-use-of-pythran.html From ralf.gommers at gmail.com Tue Sep 14 10:45:52 2021 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 14 Sep 2021 16:45:52 +0200 Subject: [SciPy-Dev] Sensitivity analysis module proposal (Robert Kern) In-Reply-To: References: <1A590E8A-9602-49FB-A85F-65698B952111@gmail.com> Message-ID: On Fri, Sep 3, 2021 at 7:32 PM Robert Kern wrote: > [private] > I will reply to the rest of the thread later today. It looks like this message went to the scipy-dev list instead of in private by accident. That happens. I think I understand the tensions and disconnect in this conversation. I have planned a call with Pamphile to talk about how to structure the conversation differently next time, and also about how we are used to discussing and making decisions about SciPy development and community topics here. I think we all have the best of intentions in moving the project forward, so I suggest leaving the public meta conversation here. Cheers, Ralf > On Fri, Sep 3, 2021 at 1:16 PM Pamphile Roy > wrote: > >> >> >> Hi all, >> >> >> First things first, there are too many things to discuss and email is not >> appropriate. I propose to make a call with people interested next week or >> the week after. Let me know if we can do this and what time would work so I >> can put together a doodle. >> >> >> Some replies on specific points: >> > > This is not the first time that you have declared a forum inappropriate > for discussion as a prelude to dropping a lengthy final argument in that > same forum. That's not an acceptable way to treat your colleagues. > > -- > Robert Kern > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Sep 14 10:58:11 2021 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 14 Sep 2021 10:58:11 -0400 Subject: [SciPy-Dev] Sensitivity analysis module proposal (Robert Kern) In-Reply-To: References: <1A590E8A-9602-49FB-A85F-65698B952111@gmail.com> Message-ID: On Tue, Sep 14, 2021 at 10:46 AM Ralf Gommers wrote: > > > On Fri, Sep 3, 2021 at 7:32 PM Robert Kern wrote: > >> [private] >> > > I will reply to the rest of the thread later today. > > It looks like this message went to the scipy-dev list instead of in > private by accident. > Everyone, but especially Pamphile, has my deepest apologies. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Sep 14 11:11:47 2021 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 14 Sep 2021 17:11:47 +0200 Subject: [SciPy-Dev] Sensitivity analysis module proposal (Robert Kern) In-Reply-To: <1A590E8A-9602-49FB-A85F-65698B952111@gmail.com> References: <1A590E8A-9602-49FB-A85F-65698B952111@gmail.com> Message-ID: On Fri, Sep 3, 2021 at 7:15 PM Pamphile Roy wrote: > > > Hi all, > > > First things first, there are too many things to discuss and email is not > appropriate. I propose to make a call with people interested next week or > the week after. Let me know if we can do this and what time would work so I > can put together a doodle. > I think I understand the point you were trying to make - this is a large topic, and email is not the easiest conversation mechanism. So having a call with people who are interested to hash out a few things can be very useful. However let me point out that the purpose of that would be to get to a clearer proposal - final discussions and decisions on new features and directions for SciPy development has always been and will continue to be on this list. We will need a better scoped proposal, like a list of functionality that should go into (e.g.) `scipy.stats`. > Some replies on specific points: > > > Let me preface this by saying that I am not gaining anything from working > on SciPy. I have no project related to SA at the moment and nothing is > planned. I am only giving some personal time and have zero financial > support for my time. > This is true for almost everyone at the moment (good news is that we do have a new grant from CZI starting soon, for ~0.5 FTE). > - SALib into SciPy: > > > I proposed this following your input Robert. You said that if the > maintainers of this library would be willing to do so, then we could > discuss. I am very happy to see that they would actually be willing to do > this and help. > > > > - Funding: > > > I can only disagree. Yes, being part of SciPy does not open directly a > line of credit, but it does indirectly help. Art Owen already told me that > he is using the fact that he helped us on stats.qmc to get some fundings. > And I am certain some of us also use this with their current employers. > I think both points of view are valid here. For dedicated direct funding, like a grant for developing new SA methods, a separate project is probably better. For "secondary benefits", being able to say that something goes into SciPy - and therefore reaches potentially O(10 million) users - can be powerful. > > - Bus factor: > > > We are way pass the bus factor. There is the team behind SALib, me, and I > already got interest from people like Sergei Kucherenko (one of the pillar > of SA). And if we go that way, I am 100% sure Saltelli (another pillar) > will be interested (I worked with him on a tooling project). > It's pure Python code and relatively straightforward technically, so if there's a commitment from more than one person to be maintainers, I think this part doesn't worry me. > People like them are willing to contribute only because it?s SciPy. We had > this discussion at MCM2021 when I presented SciPy?s QMC module to the QMC > community. Researchers, at least from the communities I know and talked > with, do not want to contribute to something that is uncertain or linked to > a particular group. With SciPy they see an opportunity to have a large > impact across a wide range of fields. And also it?s the assurance to have a > long term impact. > This is not the primary factor for deciding whether or not something belongs in SciPy, but it's helpful to know that there will be expert support/reviewers. When adding new functionality, ensuring correctness and fit for purpose of algorithms is quite often a pain point when deciding whether to merge a PR. > - Why SA? How is this important? > > SA is becoming a major field. Not only by itself, but other fields are > starting to use its methods. Things like Shapley values in AI, importance > factor, etc. In most engineering field, it?s now mandatory to assess > uncertainties. Hence there is a real need for tooling. > I think this is where more detail and data is needed. For reference, these are the sub-submodules we added in the past 10 years: `stats.qmc`, `spatial.transform`, `signal.windows`, Cython interfaces for `linalg` and `special`, `sparse.csgraph`. We haven't added any new submodules in over a decade. All of those seem more "general" than SA, except perhaps `stats.qmc` which has a similar audience. So I'd say that a new top-level submodule does not sound like a good fit. A sub-submodule or just a set of functions/classes in `scipy.stats` could make sense on the other hand. The question is what that list should look like to make this effort make sense for both SciPy and SALib. There are mature libraries in the field. OpenTURNS, UQLab and Dakota are > the most used among practitioners. And they are all not independent and > open as we are (explains partially why people like Sergei or Saltelli are > not contributors). I will not go here about why I think these library > should not be used. > > > I really don?t get the push back on this one. It?s about adding a few > method in stats at most and the benefits would be arguably huge compared to > some functions/sub modules we added recently. > That sounds like what I said before - so let's make that list, to make the conversation concrete. Behind it, there would be the most renown people of the field and that > could give another great exposure for SciPy. We will talk of SciPy at new > conferences, etc. > > > My presentation of QMC had a great impact in the QMC community. I had 4 > concrete proposal of collaboration just during the conference. We went from > not being in one field to be the recommended tool by a community. > > > SciPy is at the foundation of the scientific ecosystem in Python and not > having basic tooling a about uncertainty/SA that other higher lever > packages could rely on is a puzzle for me. > That is usually a good reason to put something in SciPy: if other packages with a significant user base need/want it, and those do not want to rely on a smaller package like SALib as a dependency. This is how we got sparse graph algorithms for example - upstreamed from scikit-learn and then expanded. Cheers, Ralf > As I said at the beginning, let?s have a talk to decide what?s in the best > interest of SciPy. > > > Cheers, > > Pamphile > > > On 3 Sep 2021, at 03:35, Robert Kern wrote: > > ? > On Thu, Sep 2, 2021 at 3:30 PM William Usher wrote: > >> Hi Robert, >> >> Thanks for the response. You raise good points. >> >> Obviously, that you are interested in the proposal assuages some of that, >> but I'm still unclear on why you are interested in this. What is the >> benefit that you think everyone will get by absorbing SALib into scipy? It >> still looks to me like a mostly-lateral move that will merely be disruptive >> to your dependent projects more than anything else. >> >> >> The real value of SALib is in providing a consistent interface to a >> (large) suite of sensitivity analysis methods which allows users to easily >> switch between those methods. >> >> We (as maintainers) could benefit from reducing duplication of code and >> implementations, such has Sobol? sequence generation, LHS, and could >> contribute some of the more general sample generation implementations where >> appropriate (many are linked directly to the SA implementations and not >> useful outside of that). >> > > I definitely think anything that could plausibly fall in the purview of > scipy.stats.qmc would be a good target for convergence. If you can't use > scipy.stats.qmc due to missing functionality (and not just that you don't > want to require that recent version of scipy), then let's see how we can > shore it up. I think that where your sampling methods overlap with design > of experiments in general might also be a fruitful place for scipy.stats to > grow/absorb some community-wide functionality. > > We think the scientific community as a whole would benefit from the >> greater exposure a SciPy implementation of SA would bring - as a large >> community-led effort - it could provide a neutral forum for further >> development of these methods. This would likely come at some ?cost? to the >> successful ?cottage industry? we?ve established and grown (SALib is getting >> lots of citations and use). >> > > That is gold. I am having trouble understanding why you would contemplate > anything that might place that at risk. If I were dictator here, I would > reject you out of hand for your own good. ;-) > > I think a key argument against integration is that it may reduce the >> agility with which we can add new methods to our SA suite (although this >> could be mitigated with careful design). >> >> I think you raise an important point about our dependent projects, and >> particularly how we would continue to support legacy releases if developer >> resources were focussed on a SciPy integration? >> >> An attraction is the possibility of funding to support the development of >> SA within SciPy. Like all open-source projects, we suffer from resourcing >> issues, and are predominantly volunteer-driven from the academic >> community. And while we are technically a ?multi-developer? community, >> we?re only a few bus accidents (or career changes) away from being a >> lone-maintainer. >> > > That's more required bus accidents than many parts of scipy. :-) > > I'm afraid that contributing into scipy doesn't unlock a pot of funds. I'm > out of the grant-writing game, but I suspect that the difference between > applying for funds on your own and arguing for allocation from competing > needs inside of scipy is mostly a push. Have you considered applying to > NumFocus? I occasionally look at their accepted and rejected projects, and > I'd lump you in with the former, IMO. You're doing important work, playing > well with other projects in the ecosystem, and have at least the seed > kernel of sustainable community development so that funds are likely to > actually sustain development. > > -- > Robert Kern > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilhanpolat at gmail.com Wed Sep 15 08:35:10 2021 From: ilhanpolat at gmail.com (Ilhan Polat) Date: Wed, 15 Sep 2021 14:35:10 +0200 Subject: [SciPy-Dev] PR to add get_array_bandwidth, issymmetric, ishermitian functions In-Reply-To: References: Message-ID: Update mail for the aforementioned PR. After using the hint from Eric Moore, it got slightly faster but probably there is more room for improvement. Evgeni raised the issue of approximate checks instead of identical checks for zero, mostly the use case of an array contaminated with numerical noise. I am not sure about the business case for it but I'd like to fish for some feedback here too. The main issues I can see is that int types have no room for noise and also the tolerance adjustment would be quite difficult to pull off since it would depend on the magnitudes of the nonzero entries to decide what constitutes as noise. Finally, though I started this in SciPy, I still have some doubts whether this should go into NumPy or not. But we can do that anytime later. If it looks OK then I'd like to put these in well before the 1.8 deadline since I want to attempt to get the generic linalg functions solve, eig, and svd context aware, that is to say using these functions eig will dispatch to eigh or triangular solvers, linalg.solve will dispatch to different solvers depending on the structure etc. Hence any feedback is welcome. Best, ilhan On Wed, Sep 8, 2021 at 1:25 PM Ilhan Polat wrote: > Dear all, > > There is now a PR for scipy.linalg module including functions mentioned in > the title. The idea is actually to enable a central place to provide these > rather occasionally needed functionalities with some basic benchmarking > available. If and when these are in then we can work on the context aware > eig, solve functions much easier delegating the right structure to the > right algorithm. > > Feedback is most welcome. > > https://github.com/scipy/scipy/pull/14701 > > best, > ilhan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From larson.eric.d at gmail.com Wed Sep 15 11:54:24 2021 From: larson.eric.d at gmail.com (Eric Larson) Date: Wed, 15 Sep 2021 11:54:24 -0400 Subject: [SciPy-Dev] PR to add get_array_bandwidth, issymmetric, ishermitian functions In-Reply-To: References: Message-ID: > > I am not sure about the business case for it but I'd like to fish for some > feedback here too. > I hit this sort of thing often working with floating point data, for example with a simple left-and-right multiplies of some matrix (and its transpose, respectively) that should preserve symmetry: >>> import numpy as np >>> rng = np.random.RandomState(0) >>> x = rng.randn(10, 10) >>> y = x @ x.T # symmetric >>> np.array_equal(y, y.T) True >>> p = rng.randn(10, 10) >>> z = p @ y @ p.T # this should still be symmetric >>> np.array_equal(z, z.T) False >>> np.allclose(z, z.T) # it is symmetric to numerical precision True >>> z[0, 1] 7.912658519091485 >>> z[1, 0] 7.9126585190914875 The main issues I can see is that int types have no room for noise and also > the tolerance adjustment would be quite difficult to pull off since it > would depend on the magnitudes of the nonzero entries to decide what > constitutes as noise. > I would vote for / expect `atol=0, rtol=1e-7` keyword arguments (maybe with different defaults) to have an API similar to `np.allclose`. Here I chose the defaults for `assert_allclose` because I have found in practice the `atol=1e-8` to be a bit dangerous / lead to silent bugs in people's code. Finally, though I started this in SciPy, I still have some doubts whether > this should go into NumPy or not. But we can do that anytime later. > To me NumPy would be a more natural fit for this sort of thing, and I would enjoy having `np.testing.*` variants of these things, too, for code development in other downstream libraries. My 2c, Eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilhanpolat at gmail.com Wed Sep 15 16:35:15 2021 From: ilhanpolat at gmail.com (Ilhan Polat) Date: Wed, 15 Sep 2021 22:35:15 +0200 Subject: [SciPy-Dev] PR to add get_array_bandwidth, issymmetric, ishermitian functions In-Reply-To: References: Message-ID: On Wed, Sep 15, 2021 at 5:55 PM Eric Larson wrote: > I am not sure about the business case for it but I'd like to fish for some >> feedback here too. >> > > I hit this sort of thing often working with floating point data, for > example with a simple left-and-right multiplies of some matrix (and its > transpose, respectively) that should preserve symmetry:... > >> Yes I think you convinced me about this (for the array bandwidth still not quite sure though, very difficult to pull it off with noise and still being fast) > The main issues I can see is that int types have no room for noise and >> also the tolerance adjustment would be quite difficult to pull off since it >> would depend on the magnitudes of the nonzero entries to decide what >> constitutes as noise. >> > > I would vote for / expect `atol=0, rtol=1e-7` keyword arguments (maybe > with different defaults) to have an API similar to `np.allclose`. Here I > chose the defaults for `assert_allclose` because I have found in practice > the `atol=1e-8` to be a bit dangerous / lead to silent bugs in people's > code. > This is precisely why I am hesitating. I am a big fan of "People should be allowed to shoot themselves in the foot" but I would suggest the defaults to be zero for both to delegate all responsibility to the user. Because if we define it, it's going to get quite messy in the bug reports. > > Finally, though I started this in SciPy, I still have some doubts whether >> this should go into NumPy or not. But we can do that anytime later. >> > > To me NumPy would be a more natural fit for this sort of thing, and I > would enjoy having `np.testing.*` variants of these things, too, for code > development in other downstream libraries. > Yes, that would be my preference too but I can't crack the C code no matter how hard I try (and I tried quite a few times). So I think they have to chime in. > > My 2c, > Eric > Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: