From serge.guelton at telecom-bretagne.eu  Sat Aug  1 17:04:51 2020
From: serge.guelton at telecom-bretagne.eu (Serge Guelton)
Date: Sat, 1 Aug 2020 23:04:51 +0200
Subject: [SciPy-Dev] Pythran 0.9.6 - talar tro
Message-ID: <20200801210451.GA838183@sguelton.remote.csb>

Hi folks, and sorry for the double posting if any,

It's my pleasure to share with you the 0.9.6 release of Pythran, code-named
talar-tro [0].

Pythran is a compiler for numerical kernels written in Python.
It doesn't require much type information and its inputs are 100% compatible with
the Python interpreter (but the other way around is not true!)

More than 6 months have passed since last revision, so the changelog is a bit
dense [1], but basically, that's the first release that only supports Python3 [2],
I made quite a lot of changes in the expression engine and it comes with more
supported numpy stuff, hopefully less bugs etc.

As usual, it's available on Pypi, conda-forge and github :-)


[0] https://br.wikipedia.org/wiki/Talar-tro
[1] https://pythran.readthedocs.io/en/latest/Changelog.html
[2] Is that even a feature?

From ralf.gommers at gmail.com  Tue Aug  4 06:36:42 2020
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Tue, 4 Aug 2020 11:36:42 +0100
Subject: [SciPy-Dev] ENH: Add an approximate Linear Sum Assignment Solver
In-Reply-To: <BL0PR01MB521711E343314E1BE4AF79A099640@BL0PR01MB5217.prod.exchangelabs.com>
References: <BL0PR01MB5217B6207EA7A0ED1D43E84E99640@BL0PR01MB5217.prod.exchangelabs.com>
 <BL0PR01MB521711E343314E1BE4AF79A099640@BL0PR01MB5217.prod.exchangelabs.com>
Message-ID: <CABL7CQgHWhejkDOG5AfmgJ11x37TPw0H9vz9wcToZR1VH-QSSQ@mail.gmail.com>

On Fri, Jul 10, 2020 at 4:03 PM Ali Saad-Eldin <asaadel1 at jhu.edu> wrote:

> Hi all,
> Hi all,
>
> I would like to add an approximate Linear Assignment Problem
> <http://www.assignmentproblems.com/doc/LSAPIntroduction.pdf> (aLAP)
> solver to scipy.optimize. The specific approximation algorithm I propose
> adding can be found here
> <https://link.springer.com/content/pdf/10.1007%2F978-3-540-68111-3_74.pdf>.
> Scipy.optimize currently houses an exact solver for LAP in
> linear_sum_assignment()
> <https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.linear_sum_assignment.html>,
> and I was thinking this could be added either as a new method within
> linear_sum_assignment(), or as its own function.
>
> The approximation algorithm has time complexity O(n2log(n)), compared to
> the algorithm implemented in linear_sum_assignment() with complexity ~O(n
> 2.3 ), while guaranteeing a score >= 1/2 optimum (though in practice the
> scores appear to be much closer to optimum).   I've attached two plots
> demonstrating runtime and accuracy, running linear_sum_assignment() and
> aLAP on simulated, dense nxn cost_matrices with entries randomly selected
> from a uniform(0,100) distribution, for n sampled from [0, 3000]. Note that
> linear_sum_assignment() is essentially C/C++, while the aLAP implementation
> is native Python, so a cython implementation could make aLAP even faster.
> Additionally, an advantage to this algorithm is that it is parallelizable,
> which could cause a further speed up.
>
> Current version of the implementation can be found here
> <https://github.com/neurodata/graspy/blob/ase-alap/graspy/match/alap.py>,
> and proof of effectiveness here
> <https://nbviewer.jupyter.org/github/asaadeldin11/nbview/blob/master/aLAP_code.ipynb>
> .
>

Hi Ali, thanks for proposing this! Just so this email doesn't go completely
unanswered: this seems interesting. As you noticed, we're (more than a
little) short on reviewer power, so this may be hard to do in parallel with
your Quadratic Assignment Solver in https://github.com/scipy/scip/pull/11862
.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200804/712bc982/attachment.html>

From PeterBell10 at live.co.uk  Tue Aug  4 12:23:26 2020
From: PeterBell10 at live.co.uk (Peter Bell)
Date: Tue, 4 Aug 2020 16:23:26 +0000
Subject: [SciPy-Dev] Changing return types in a public API?
Message-ID: <AM6PR0402MB38620E0561B089218E07689F8E4A0@AM6PR0402MB3862.eurprd04.prod.outlook.com>

Hello all,

I've been working towards unifying KDTree and cKDTree in gh-12382<https://github.com/scipy/scipy/pull/12382#discussion_r451491803>. This has raised some general questions about API compatibility that I'd appreciate any thoughts on.

First, KDTree returns NumPy scalars everywhere because the results come from indexing NumPy arrays. Whereas, cKDTree returns python ints wherever possible. Is it reasonable for an API to change from returning NumPy scalars to python int? The NumPy scalars do mimic the array interface to some extent so there is a small interface incompatibility. However, the documentation usually just says a function returns int or float, not NumPy scalar specifically.

Secondly, KDTree uses `dtype=int` everywhere which results in 64-bit integers on linux but 32-bit integers on windows. Ideally, I'd want to return 64-bit integers (or at least np.intp) on all platforms for consistency and to avoid issues with integer overflow. Again, this is behaviour that isn't documented but code could still be relying on implicitly.

- Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200804/d0372c5b/attachment.html>

From larson.eric.d at gmail.com  Tue Aug  4 13:02:49 2020
From: larson.eric.d at gmail.com (Eric Larson)
Date: Tue, 4 Aug 2020 13:02:49 -0400
Subject: [SciPy-Dev] Changing return types in a public API?
In-Reply-To: <AM6PR0402MB38620E0561B089218E07689F8E4A0@AM6PR0402MB3862.eurprd04.prod.outlook.com>
References: <AM6PR0402MB38620E0561B089218E07689F8E4A0@AM6PR0402MB3862.eurprd04.prod.outlook.com>
Message-ID: <CAGu2niUykiTWmC04=xksnawUzZs0_PY=H5TeftE7RuOhqnCTMQ@mail.gmail.com>

>
> First, KDTree returns NumPy scalars everywhere because the results come
> from indexing NumPy arrays. Whereas, cKDTree returns python ints wherever
> possible. Is it reasonable for an API to change from returning NumPy
> scalars to python int? The NumPy scalars do mimic the array interface to
> some extent so there is a small interface incompatibility. However, the
> documentation usually just says a function returns int or float, not NumPy
> scalar specifically.
>
> Secondly, KDTree uses `dtype=int` everywhere which results in 64-bit
> integers on linux but 32-bit integers on windows. Ideally, I'd want to
> return 64-bit integers (or at least np.intp) on all platforms for
> consistency and to avoid issues with integer overflow. Again, this is
> behaviour that isn't documented but code could still be relying on
> implicitly.
>

Changing to a NumPy int return type seems fine in this case, especially
since it can be considered a bugfix (32-bit int problems seem to come up a
lot across SciPy for Windows users).

My 2c,
Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200804/2099937a/attachment.html>

From ralf.gommers at gmail.com  Tue Aug  4 17:58:27 2020
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Tue, 4 Aug 2020 22:58:27 +0100
Subject: [SciPy-Dev] Changing return types in a public API?
In-Reply-To: <CAGu2niUykiTWmC04=xksnawUzZs0_PY=H5TeftE7RuOhqnCTMQ@mail.gmail.com>
References: <AM6PR0402MB38620E0561B089218E07689F8E4A0@AM6PR0402MB3862.eurprd04.prod.outlook.com>
 <CAGu2niUykiTWmC04=xksnawUzZs0_PY=H5TeftE7RuOhqnCTMQ@mail.gmail.com>
Message-ID: <CABL7CQg2mpcpvnTr+odp7PLXNV52+sSwZaiVpL-qxDuV=Pno4A@mail.gmail.com>

On Tue, Aug 4, 2020 at 6:03 PM Eric Larson <larson.eric.d at gmail.com> wrote:

> First, KDTree returns NumPy scalars everywhere because the results come
>> from indexing NumPy arrays. Whereas, cKDTree returns python ints wherever
>> possible. Is it reasonable for an API to change from returning NumPy
>> scalars to python int? The NumPy scalars do mimic the array interface to
>> some extent so there is a small interface incompatibility. However, the
>> documentation usually just says a function returns int or float, not NumPy
>> scalar specifically.
>>
>> Secondly, KDTree uses `dtype=int` everywhere which results in 64-bit
>> integers on linux but 32-bit integers on windows. Ideally, I'd want to
>> return 64-bit integers (or at least np.intp) on all platforms for
>> consistency and to avoid issues with integer overflow. Again, this is
>> behaviour that isn't documented but code could still be relying on
>> implicitly.
>>
>
> Changing to a NumPy int return type seems fine in this case, especially
> since it can be considered a bugfix (32-bit int problems seem to come up a
> lot across SciPy for Windows users).
>

I agree, seems okay to make this change

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200804/2d6eb65a/attachment.html>

From e.lakkis at me.com  Thu Aug  6 01:13:49 2020
From: e.lakkis at me.com (=?utf-8?B?0K3QvNC40LvRjCDQm9Cw0LrQutC40YE=?=)
Date: Thu, 6 Aug 2020 01:13:49 -0400
Subject: [SciPy-Dev] Implementing fit method for discrete distributions
Message-ID: <FEEBA7EB-79AC-481D-93B7-C4FA0C4BC60D@me.com>

Hi everyone,


I am a newcomer here and after reading through #11948 I thought implementing fit option for discrete distributions could be a good start. In particular, I was thinking of implementing MLE + MM (whenever possible). Do people generally think this is interesting? If so, is it generally expected to provide references for every estimator being used?


Thank you!

Best,
Emil


From peterbell10 at live.co.uk  Thu Aug  6 08:57:57 2020
From: peterbell10 at live.co.uk (Peter Bell)
Date: Thu, 6 Aug 2020 12:57:57 +0000
Subject: [SciPy-Dev] Changing return types in a public API?
In-Reply-To: <CABL7CQg2mpcpvnTr+odp7PLXNV52+sSwZaiVpL-qxDuV=Pno4A@mail.gmail.com>
References: <AM6PR0402MB38620E0561B089218E07689F8E4A0@AM6PR0402MB3862.eurprd04.prod.outlook.com>
 <CAGu2niUykiTWmC04=xksnawUzZs0_PY=H5TeftE7RuOhqnCTMQ@mail.gmail.com>,
 <CABL7CQg2mpcpvnTr+odp7PLXNV52+sSwZaiVpL-qxDuV=Pno4A@mail.gmail.com>
Message-ID: <AM6PR0402MB386297147A976F8A2F0B5BE08E480@AM6PR0402MB3862.eurprd04.prod.outlook.com>

Okay, so I'll keep KDTree returning NumPy scalars for now but change the integer sizes to match cKDTree. Thanks guys.nteger sizes to match cKDTree. Thanks guys.

Cheers,
Peter
________________________________
From: SciPy-Dev <scipy-dev-bounces+peterbell10=live.co.uk at python.org> on behalf of Ralf Gommers <ralf.gommers at gmail.com>
Sent: 04 August 2020 22:58
To: SciPy Developers List <scipy-dev at python.org>
Subject: Re: [SciPy-Dev] Changing return types in a public API?



On Tue, Aug 4, 2020 at 6:03 PM Eric Larson <larson.eric.d at gmail.com<mailto:larson.eric.d at gmail.com>> wrote:
First, KDTree returns NumPy scalars everywhere because the results come from indexing NumPy arrays. Whereas, cKDTree returns python ints wherever possible. Is it reasonable for an API to change from returning NumPy scalars to python int? The NumPy scalars do mimic the array interface to some extent so there is a small interface incompatibility. However, the documentation usually just says a function returns int or float, not NumPy scalar specifically.

Secondly, KDTree uses `dtype=int` everywhere which results in 64-bit integers on linux but 32-bit integers on windows. Ideally, I'd want to return 64-bit integers (or at least np.intp) on all platforms for consistency and to avoid issues with integer overflow. Again, this is behaviour that isn't documented but code could still be relying on implicitly.

Changing to a NumPy int return type seems fine in this case, especially since it can be considered a bugfix (32-bit int problems seem to come up a lot across SciPy for Windows users).

I agree, seems okay to make this change

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200806/56a21c71/attachment.html>

From mhaberla at calpoly.edu  Sun Aug  9 16:25:27 2020
From: mhaberla at calpoly.edu (Matt Haberland)
Date: Sun, 9 Aug 2020 13:25:27 -0700
Subject: [SciPy-Dev] Wallenius hypergeometric distribution
In-Reply-To: <CADuxUizWkhy1nkns+09hAkTXQ7TkyOhaVYwzMhdbp2TyZMOvaQ@mail.gmail.com>
References: <CAJdHkOw5D_sGBJi9qa78Zq-aF6QXX85VR30nLJmd+j-1Aboajw@mail.gmail.com>
 <CADuxUizWkhy1nkns+09hAkTXQ7TkyOhaVYwzMhdbp2TyZMOvaQ@mail.gmail.com>
Message-ID: <CADuxUix7gbTp10XHbN-c36BS1H+idr=D2zFCsmi_4ECJEn79AQ@mail.gmail.com>

Hi Team,

Here's the noncentral hypergeometric distribution code we have permission
to distribute under the BSD license:
https://www.agner.org/random/stocc.zip
Ralf suggested I send it around to see if there are any opinions about
using it.

It doesn't include two files that are needed to build it (mersenne.cpp and
userintf.cpp), but we want to replace those anyway. In the meantime, you
can find those files here <https://www.agner.org/random/randomc.zip>.

Alternatively, you can check out my branch:
https://github.com/mdhaber/scipy/tree/nchg
in which I've started a wrapper for Fisher's noncental hypergeometric
distribution.
Everything we'd be using is in scipy/stats/biasedurn
<https://github.com/mdhaber/scipy/tree/nchg/scipy/stats/biasedurn>. The
Cython files are BiasedUrn.pxd
<https://github.com/mdhaber/scipy/blob/nchg/scipy/stats/BiasedUrn.pxd> and
biasedurn.pyx
<https://github.com/mdhaber/scipy/blob/nchg/scipy/stats/biasedurn.pyx>.

After building, this should work:
from scipy.stats.biasedurn import _PyFishersNCHypergeometric as
PyFishersNCHypergeometric
n, m, N, odds, accuracy = 8, 25, 50, 5, 1e-8
py_fnch = PyFishersNCHypergeometric(n, m, N, odds, accuracy)
print(py_fnch.mode())
print(py_fnch.mean())
print(py_fnch.variance())
print(py_fnch.probability(7))
print(py_fnch.moments())
(Of course, we still need to add it to stats as a proper distribution.)

Thanks for your thoughts!
Matt

On Fri, Jul 24, 2020 at 7:54 AM Matt Haberland <mhaberla at calpoly.edu> wrote:

> Funny that you should mention this now! On Tuesday we got permission from
> Agner Fog to use his C++ implementation (used in the R package BiasedUrn,
> https://cran.r-project.org/web/packages/BiasedUrn/index.html) under
> SciPy's license. Would you be interested in wrapping a C++ implementation?
>
> On Fri, Jul 24, 2020, 2:53 AM Harry Gulliver <gulliver.harry at gmail.com>
> wrote:
>
>> Hi all, new (potential) contributor here!
>>
>> I'm working on a project using Wallenius' non-central hypergeometric
>> distribution atm and noticed it's not yet available in scipy.stats, so
>> thought I'd volunteer to add it. Could possibly also do Fisher's hypergeom
>> while I'm at it.
>>
>> Anything I ought to know before just writing some code and making a pull
>> request? How can I best help?
>>
>> Thanks!
>> Harry
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>

-- 
Matt Haberland
Assistant Professor
BioResource and Agricultural Engineering
08A-3K, Cal Poly
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200809/7a00bf3e/attachment.html>

From evgeny.burovskiy at gmail.com  Mon Aug 10 08:01:42 2020
From: evgeny.burovskiy at gmail.com (Evgeni Burovski)
Date: Mon, 10 Aug 2020 15:01:42 +0300
Subject: [SciPy-Dev] Wallenius hypergeometric distribution
In-Reply-To: <CADuxUix7gbTp10XHbN-c36BS1H+idr=D2zFCsmi_4ECJEn79AQ@mail.gmail.com>
References: <CAJdHkOw5D_sGBJi9qa78Zq-aF6QXX85VR30nLJmd+j-1Aboajw@mail.gmail.com>
 <CADuxUizWkhy1nkns+09hAkTXQ7TkyOhaVYwzMhdbp2TyZMOvaQ@mail.gmail.com>
 <CADuxUix7gbTp10XHbN-c36BS1H+idr=D2zFCsmi_4ECJEn79AQ@mail.gmail.com>
Message-ID: <CAMRo0isypeTkQUHJ875zHoFWtEu7CbJmXgNMFPJ9HGW7swUOiA@mail.gmail.com>

Does it need to carry its own Mersenne Twister implementation? Mixing
different random streams is prone to surprises, can the scipy version rely
on numpy.random instead?

??, 9 ???. 2020 ?., 23:26 Matt Haberland <mhaberla at calpoly.edu>:

> Hi Team,
>
> Here's the noncentral hypergeometric distribution code we have permission
> to distribute under the BSD license:
> https://www.agner.org/random/stocc.zip
> Ralf suggested I send it around to see if there are any opinions about
> using it.
>
> It doesn't include two files that are needed to build it (mersenne.cpp and
> userintf.cpp), but we want to replace those anyway. In the meantime, you
> can find those files here <https://www.agner.org/random/randomc.zip>.
>
> Alternatively, you can check out my branch:
> https://github.com/mdhaber/scipy/tree/nchg
> in which I've started a wrapper for Fisher's noncental hypergeometric
> distribution.
> Everything we'd be using is in scipy/stats/biasedurn
> <https://github.com/mdhaber/scipy/tree/nchg/scipy/stats/biasedurn>. The
> Cython files are BiasedUrn.pxd
> <https://github.com/mdhaber/scipy/blob/nchg/scipy/stats/BiasedUrn.pxd> and
> biasedurn.pyx
> <https://github.com/mdhaber/scipy/blob/nchg/scipy/stats/biasedurn.pyx>.
>
> After building, this should work:
> from scipy.stats.biasedurn import _PyFishersNCHypergeometric as
> PyFishersNCHypergeometric
> n, m, N, odds, accuracy = 8, 25, 50, 5, 1e-8
> py_fnch = PyFishersNCHypergeometric(n, m, N, odds, accuracy)
> print(py_fnch.mode())
> print(py_fnch.mean())
> print(py_fnch.variance())
> print(py_fnch.probability(7))
> print(py_fnch.moments())
> (Of course, we still need to add it to stats as a proper distribution.)
>
> Thanks for your thoughts!
> Matt
>
> On Fri, Jul 24, 2020 at 7:54 AM Matt Haberland <mhaberla at calpoly.edu>
> wrote:
>
>> Funny that you should mention this now! On Tuesday we got permission from
>> Agner Fog to use his C++ implementation (used in the R package BiasedUrn,
>> https://cran.r-project.org/web/packages/BiasedUrn/index.html) under
>> SciPy's license. Would you be interested in wrapping a C++ implementation?
>>
>> On Fri, Jul 24, 2020, 2:53 AM Harry Gulliver <gulliver.harry at gmail.com>
>> wrote:
>>
>>> Hi all, new (potential) contributor here!
>>>
>>> I'm working on a project using Wallenius' non-central hypergeometric
>>> distribution atm and noticed it's not yet available in scipy.stats, so
>>> thought I'd volunteer to add it. Could possibly also do Fisher's hypergeom
>>> while I'm at it.
>>>
>>> Anything I ought to know before just writing some code and making a pull
>>> request? How can I best help?
>>>
>>> Thanks!
>>> Harry
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>
>
> --
> Matt Haberland
> Assistant Professor
> BioResource and Agricultural Engineering
> 08A-3K, Cal Poly
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200810/648ddf70/attachment-0001.html>

From mhaberla at calpoly.edu  Mon Aug 10 10:52:27 2020
From: mhaberla at calpoly.edu (Matt Haberland)
Date: Mon, 10 Aug 2020 07:52:27 -0700
Subject: [SciPy-Dev] Wallenius hypergeometric distribution
In-Reply-To: <CAMRo0isypeTkQUHJ875zHoFWtEu7CbJmXgNMFPJ9HGW7swUOiA@mail.gmail.com>
References: <CAJdHkOw5D_sGBJi9qa78Zq-aF6QXX85VR30nLJmd+j-1Aboajw@mail.gmail.com>
 <CADuxUizWkhy1nkns+09hAkTXQ7TkyOhaVYwzMhdbp2TyZMOvaQ@mail.gmail.com>
 <CADuxUix7gbTp10XHbN-c36BS1H+idr=D2zFCsmi_4ECJEn79AQ@mail.gmail.com>
 <CAMRo0isypeTkQUHJ875zHoFWtEu7CbJmXgNMFPJ9HGW7swUOiA@mail.gmail.com>
Message-ID: <CADuxUixRgv-3O3tqted7BvS1N3C7yaM3f_Xk9-1SUmEzn0YY0A@mail.gmail.com>

No, it can use NumPy's `Generator` and `RandomState` like all other
distributions. I have already eliminated dependency on mersenne.cpp locally.

On Mon, Aug 10, 2020, 5:00 AM Evgeni Burovski <evgeny.burovskiy at gmail.com>
wrote:

> Does it need to carry its own Mersenne Twister implementation? Mixing
> different random streams is prone to surprises, can the scipy version rely
> on numpy.random instead?
>
> ??, 9 ???. 2020 ?., 23:26 Matt Haberland <mhaberla at calpoly.edu>:
>
>> Hi Team,
>>
>> Here's the noncentral hypergeometric distribution code we have permission
>> to distribute under the BSD license:
>> https://www.agner.org/random/stocc.zip
>> Ralf suggested I send it around to see if there are any opinions about
>> using it.
>>
>> It doesn't include two files that are needed to build it (mersenne.cpp
>> and userintf.cpp), but we want to replace those anyway. In the meantime,
>> you can find those files here <https://www.agner.org/random/randomc.zip>.
>>
>> Alternatively, you can check out my branch:
>> https://github.com/mdhaber/scipy/tree/nchg
>> in which I've started a wrapper for Fisher's noncental hypergeometric
>> distribution.
>> Everything we'd be using is in scipy/stats/biasedurn
>> <https://github.com/mdhaber/scipy/tree/nchg/scipy/stats/biasedurn>. The
>> Cython files are BiasedUrn.pxd
>> <https://github.com/mdhaber/scipy/blob/nchg/scipy/stats/BiasedUrn.pxd> and
>> biasedurn.pyx
>> <https://github.com/mdhaber/scipy/blob/nchg/scipy/stats/biasedurn.pyx>.
>>
>> After building, this should work:
>> from scipy.stats.biasedurn import _PyFishersNCHypergeometric as
>> PyFishersNCHypergeometric
>> n, m, N, odds, accuracy = 8, 25, 50, 5, 1e-8
>> py_fnch = PyFishersNCHypergeometric(n, m, N, odds, accuracy)
>> print(py_fnch.mode())
>> print(py_fnch.mean())
>> print(py_fnch.variance())
>> print(py_fnch.probability(7))
>> print(py_fnch.moments())
>> (Of course, we still need to add it to stats as a proper distribution.)
>>
>> Thanks for your thoughts!
>> Matt
>>
>> On Fri, Jul 24, 2020 at 7:54 AM Matt Haberland <mhaberla at calpoly.edu>
>> wrote:
>>
>>> Funny that you should mention this now! On Tuesday we got permission
>>> from Agner Fog to use his C++ implementation (used in the R package
>>> BiasedUrn, https://cran.r-project.org/web/packages/BiasedUrn/index.html)
>>> under SciPy's license. Would you be interested in wrapping a C++
>>> implementation?
>>>
>>> On Fri, Jul 24, 2020, 2:53 AM Harry Gulliver <gulliver.harry at gmail.com>
>>> wrote:
>>>
>>>> Hi all, new (potential) contributor here!
>>>>
>>>> I'm working on a project using Wallenius' non-central hypergeometric
>>>> distribution atm and noticed it's not yet available in scipy.stats, so
>>>> thought I'd volunteer to add it. Could possibly also do Fisher's hypergeom
>>>> while I'm at it.
>>>>
>>>> Anything I ought to know before just writing some code and making a
>>>> pull request? How can I best help?
>>>>
>>>> Thanks!
>>>> Harry
>>>> _______________________________________________
>>>> SciPy-Dev mailing list
>>>> SciPy-Dev at python.org
>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>
>>>
>>
>> --
>> Matt Haberland
>> Assistant Professor
>> BioResource and Agricultural Engineering
>> 08A-3K, Cal Poly
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200810/5696e36d/attachment.html>

From rlucas7 at vt.edu  Sat Aug 15 16:43:36 2020
From: rlucas7 at vt.edu (rlucas7 at vt.edu)
Date: Sat, 15 Aug 2020 16:43:36 -0400
Subject: [SciPy-Dev] Thanks wiki protocol
Message-ID: <1018717B-5410-4B61-A523-3C5BAB800FCD@vt.edu>

Hi scipy-dev, 

For work on the 1.5 release we used a wiki rather than have people add commits to the thanks file in the repo. 
I?m assuming this was to alleviate rebase conflicts to merge. My question is, 

Is using a wikifor tracking thanks to add in each release the way we track thanks entries moving forward? 

To be clear these would be separate wiki pages from the release notes wiki pages. 

-Lucas Roberts 

From ralf.gommers at gmail.com  Sat Aug 15 18:47:57 2020
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sat, 15 Aug 2020 23:47:57 +0100
Subject: [SciPy-Dev] proposal to drop Python 3.6 and NumPy 1.14 / 1.15
Message-ID: <CABL7CQg_MYscpnnnCcEW+rCzoCCRJvj82jWX_4LmSOSqjgixbQ@mail.gmail.com>

Hi all,

Python 3.9 is coming out before the next SciPy release, and NEP 29
recommends to support only Python >=3.7 and NumPy >= 1.16 from last month
onwards [1]. I think supporting 3 Python versions and 5 NumPy versions in
SciPy 1.6.0 is easily enough. That would also bring us back in line with
SciPy on conda-forge, which built against NumPy 1.16 for 1.5.2.

Any objections?

Cheers,
Ralf

[1] https://numpy.org/neps/nep-0029-deprecation_policy.html#support-table
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200815/41d00885/attachment.html>

From ralf.gommers at gmail.com  Sat Aug 15 18:55:24 2020
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sat, 15 Aug 2020 23:55:24 +0100
Subject: [SciPy-Dev] Thanks wiki protocol
In-Reply-To: <1018717B-5410-4B61-A523-3C5BAB800FCD@vt.edu>
References: <1018717B-5410-4B61-A523-3C5BAB800FCD@vt.edu>
Message-ID: <CABL7CQhpEL9nO6AkGSiR2GB78YSzkMF9XPG_qvGXrJLAzQnP7Q@mail.gmail.com>

On Sat, Aug 15, 2020 at 9:43 PM <rlucas7 at vt.edu> wrote:

> Hi scipy-dev,
>
> For work on the 1.5 release we used a wiki rather than have people add
> commits to the thanks file in the repo.
> I?m assuming this was to alleviate rebase conflicts to merge. My question
> is,
>
> Is using a wikifor tracking thanks to add in each release the way we track
> thanks entries moving forward?
>
> To be clear these would be separate wiki pages from the release notes wiki
> pages.
>

That does sound like a good idea.

A perhaps even better one is to retire the THANKS.txt file. It really
doesn't have much value, is very inconsistently updated and old entries are
horribly out of date (e.g. Pauli is thanked for docs work and interpolation
bug fixes - that doesn't quite cover his contributions :)).

I'd prefer to extract the relevant historical info and put that on the
website. Or just have a full list of contributors that's autogenerated. The
thanks to institutions at the bottom of the file already lives at
https://scipy.org/scipylib/donations.html#acknowledgements

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200815/958c3269/attachment.html>

From tyler.je.reddy at gmail.com  Sat Aug 15 19:54:01 2020
From: tyler.je.reddy at gmail.com (Tyler Reddy)
Date: Sat, 15 Aug 2020 17:54:01 -0600
Subject: [SciPy-Dev] proposal to drop Python 3.6 and NumPy 1.14 / 1.15
In-Reply-To: <CABL7CQg_MYscpnnnCcEW+rCzoCCRJvj82jWX_4LmSOSqjgixbQ@mail.gmail.com>
References: <CABL7CQg_MYscpnnnCcEW+rCzoCCRJvj82jWX_4LmSOSqjgixbQ@mail.gmail.com>
Message-ID: <CAHPuU_bAnm9S__hi9bm4k0H8DPBPE2cz0SHwO7MZ3VNEyX4UUw@mail.gmail.com>

+1

On Sat, 15 Aug 2020 at 16:49, Ralf Gommers <ralf.gommers at gmail.com> wrote:

> Hi all,
>
> Python 3.9 is coming out before the next SciPy release, and NEP 29
> recommends to support only Python >=3.7 and NumPy >= 1.16 from last month
> onwards [1]. I think supporting 3 Python versions and 5 NumPy versions in
> SciPy 1.6.0 is easily enough. That would also bring us back in line with
> SciPy on conda-forge, which built against NumPy 1.16 for 1.5.2.
>
> Any objections?
>
> Cheers,
> Ralf
>
> [1] https://numpy.org/neps/nep-0029-deprecation_policy.html#support-table
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200815/d90376f6/attachment.html>

From mhaberla at calpoly.edu  Sat Aug 15 20:14:43 2020
From: mhaberla at calpoly.edu (Matt Haberland)
Date: Sat, 15 Aug 2020 17:14:43 -0700
Subject: [SciPy-Dev] Thanks wiki protocol
In-Reply-To: <CABL7CQhpEL9nO6AkGSiR2GB78YSzkMF9XPG_qvGXrJLAzQnP7Q@mail.gmail.com>
References: <1018717B-5410-4B61-A523-3C5BAB800FCD@vt.edu>
 <CABL7CQhpEL9nO6AkGSiR2GB78YSzkMF9XPG_qvGXrJLAzQnP7Q@mail.gmail.com>
Message-ID: <CADuxUixohC5YFQ2Mtqf4CN9xN4Z3K6NNj1cpuao0JQthjC0rPQ@mail.gmail.com>

+1 for retiring the manually updated file.

On Sat, Aug 15, 2020, 3:55 PM Ralf Gommers <ralf.gommers at gmail.com> wrote:

>
>
> On Sat, Aug 15, 2020 at 9:43 PM <rlucas7 at vt.edu> wrote:
>
>> Hi scipy-dev,
>>
>> For work on the 1.5 release we used a wiki rather than have people add
>> commits to the thanks file in the repo.
>> I?m assuming this was to alleviate rebase conflicts to merge. My question
>> is,
>>
>> Is using a wikifor tracking thanks to add in each release the way we
>> track thanks entries moving forward?
>>
>> To be clear these would be separate wiki pages from the release notes
>> wiki pages.
>>
>
> That does sound like a good idea.
>
> A perhaps even better one is to retire the THANKS.txt file. It really
> doesn't have much value, is very inconsistently updated and old entries are
> horribly out of date (e.g. Pauli is thanked for docs work and interpolation
> bug fixes - that doesn't quite cover his contributions :)).
>
> I'd prefer to extract the relevant historical info and put that on the
> website. Or just have a full list of contributors that's autogenerated. The
> thanks to institutions at the bottom of the file already lives at
> https://scipy.org/scipylib/donations.html#acknowledgements
>
> Cheers,
> Ralf
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200815/dc263d6c/attachment-0001.html>

From bennet at umich.edu  Sun Aug 16 09:23:16 2020
From: bennet at umich.edu (Bennet Fauber)
Date: Sun, 16 Aug 2020 09:23:16 -0400
Subject: [SciPy-Dev] proposal to drop Python 3.6 and NumPy 1.14 / 1.15
In-Reply-To: <CABL7CQg_MYscpnnnCcEW+rCzoCCRJvj82jWX_4LmSOSqjgixbQ@mail.gmail.com>
References: <CABL7CQg_MYscpnnnCcEW+rCzoCCRJvj82jWX_4LmSOSqjgixbQ@mail.gmail.com>
Message-ID: <CAB2ovovq144TYAzdMLwko7uHKJqnzhedKMAcM-hNqzVWXSjWww@mail.gmail.com>

This post is by way of trying to articulate some concerns from the end
user side of things.

It looks like NumPy 19.1 supports Python 3.6.

> This release supports Python 3.6-3.8. Cython >= 0.29.21 needs to be used when building
> with Python 3.9 for testing purposes.
https://numpy.org/devdocs/release/1.19.1-notes.html

The section of the release notes for NumPy 1.20.0 does not contain the
section at the top saying which versions of Python are supported.
(https://numpy.org/devdocs/release/1.20.0-notes.html)

For Python itself,  I find on their releases, the 3.6.12 schedule was

    3.6.12 final: 2020-08-14 (expected)
https://www.python.org/dev/peps/pep-0494/#id9

but does not seem to have made it, and Python 3.6.11 was released June
27, 2020.  The Python plans for 3.6.13 and beyond are Security fixes
only, as needed, until 2021-12.  That seems, from my user perspective,
a better date for deprecation and not too far from the just proposed
one.

Acknowledging the additional burden of 'supporting' older versions of
Python, still it would seem that matching NumPy is not a bad thing.

>From a strictly consumer perspective, where much of the work is
getting all of the non-NumPy and non-SciPy functionality to work and
be stable, upgrading Python can be very disruptive.  Time spent
getting the 'glue' around analytics to work is time we cannot spend on
science.  There are large projects that do keep up, but they tend also
to be funded well.  The many small labs on my campus do not have
funding for software development, are unlikely to ever get any, so any
work required to fix software because of updates comes from their
budget for science and from their scientists who are not as proficient
and therefore not as efficient at adapting to a newer version of
Python and all the reinstallation of libraries that attends it.

If there were a way to provide _only_ actual bug fixes for some
versions longer, a la the LTS releases of some Linux distributions,
that might be of benefit to users and would reduce the support burden
on developers who would not need to worry about fitting new features
into the older framework.

All of that is not to say that you should not continue with current
deprecation plans.  Just to be sure that these concerns were voiced at
least once.  I hope that this perspective is of some use in your
considerations.

Thanks for all the software, regardless!


On Sat, Aug 15, 2020 at 6:48 PM Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
> Hi all,
>
> Python 3.9 is coming out before the next SciPy release, and NEP 29 recommends to support only Python >=3.7 and NumPy >= 1.16 from last month onwards [1]. I think supporting 3 Python versions and 5 NumPy versions in SciPy 1.6.0 is easily enough. That would also bring us back in line with SciPy on conda-forge, which built against NumPy 1.16 for 1.5.2.
>
> Any objections?
>
> Cheers,
> Ralf
>
> [1] https://numpy.org/neps/nep-0029-deprecation_policy.html#support-table
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev

From ralf.gommers at gmail.com  Sun Aug 16 11:02:49 2020
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sun, 16 Aug 2020 16:02:49 +0100
Subject: [SciPy-Dev] proposal to drop Python 3.6 and NumPy 1.14 / 1.15
In-Reply-To: <CAB2ovovq144TYAzdMLwko7uHKJqnzhedKMAcM-hNqzVWXSjWww@mail.gmail.com>
References: <CABL7CQg_MYscpnnnCcEW+rCzoCCRJvj82jWX_4LmSOSqjgixbQ@mail.gmail.com>
 <CAB2ovovq144TYAzdMLwko7uHKJqnzhedKMAcM-hNqzVWXSjWww@mail.gmail.com>
Message-ID: <CABL7CQjF5ZWOqjmaHF-koBM9rYGZbGKGEmfa5T0W5Nx1_XaDVA@mail.gmail.com>

On Sun, Aug 16, 2020 at 2:23 PM Bennet Fauber <bennet at umich.edu> wrote:

> This post is by way of trying to articulate some concerns from the end
> user side of things.
>
> It looks like NumPy 19.1 supports Python 3.6.
>
> > This release supports Python 3.6-3.8. Cython >= 0.29.21 needs to be used
> when building
> > with Python 3.9 for testing purposes.
> https://numpy.org/devdocs/release/1.19.1-notes.html
>
> The section of the release notes for NumPy 1.20.0 does not contain the
> section at the top saying which versions of Python are supported.
> (https://numpy.org/devdocs/release/1.20.0-notes.html)
>
> For Python itself,  I find on their releases, the 3.6.12 schedule was
>
>     3.6.12 final: 2020-08-14 (expected)
> https://www.python.org/dev/peps/pep-0494/#id9
>
> but does not seem to have made it, and Python 3.6.11 was released June
> 27, 2020.  The Python plans for 3.6.13 and beyond are Security fixes
> only, as needed, until 2021-12.  That seems, from my user perspective,
> a better date for deprecation and not too far from the just proposed
> one.
>
> Acknowledging the additional burden of 'supporting' older versions of
> Python, still it would seem that matching NumPy is not a bad thing.
>
> From a strictly consumer perspective, where much of the work is
> getting all of the non-NumPy and non-SciPy functionality to work and
> be stable, upgrading Python can be very disruptive.  Time spent
> getting the 'glue' around analytics to work is time we cannot spend on
> science.  There are large projects that do keep up, but they tend also
> to be funded well.  The many small labs on my campus do not have
> funding for software development, are unlikely to ever get any, so any
> work required to fix software because of updates comes from their
> budget for science and from their scientists who are not as proficient
> and therefore not as efficient at adapting to a newer version of
> Python and all the reinstallation of libraries that attends it.
>
> If there were a way to provide _only_ actual bug fixes for some
> versions longer, a la the LTS releases of some Linux distributions,
> that might be of benefit to users and would reduce the support burden
> on developers who would not need to worry about fitting new features
> into the older framework.
>
> All of that is not to say that you should not continue with current
> deprecation plans.  Just to be sure that these concerns were voiced at
> least once.  I hope that this perspective is of some use in your
> considerations.
>

Thanks for your thoughts Bennet. All of this was brought up in the NEP 29
discussion which recommended timelines for Python and NumPy version
support. I don't think any of it is specific to SciPy. It's a trade-off:
users may want perfect stability and updates for whatever version they're
on for as long as possible, but that comes at the cost of making
maintainers doing tedious work, like more release effort and figuring out
why CI fails on some particular combination of old Python and NumPy
versions. That tedious work then reduces the amount of time for tackling
more relevant bugs and adding new features, and it likely also reduces
total effort spent by maintainers over time (most people do this
voluntarily, so motivation comes partly from enjoying to contribute).

Cheers,
Ralf



> Thanks for all the software, regardless!
>
>
> On Sat, Aug 15, 2020 at 6:48 PM Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
> >
> > Hi all,
> >
> > Python 3.9 is coming out before the next SciPy release, and NEP 29
> recommends to support only Python >=3.7 and NumPy >= 1.16 from last month
> onwards [1]. I think supporting 3 Python versions and 5 NumPy versions in
> SciPy 1.6.0 is easily enough. That would also bring us back in line with
> SciPy on conda-forge, which built against NumPy 1.16 for 1.5.2.
> >
> > Any objections?
> >
> > Cheers,
> > Ralf
> >
> > [1]
> https://numpy.org/neps/nep-0029-deprecation_policy.html#support-table
> > _______________________________________________
> > SciPy-Dev mailing list
> > SciPy-Dev at python.org
> > https://mail.python.org/mailman/listinfo/scipy-dev
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200816/d0e3ce25/attachment.html>

From bennet at umich.edu  Sun Aug 16 11:08:56 2020
From: bennet at umich.edu (Bennet Fauber)
Date: Sun, 16 Aug 2020 11:08:56 -0400
Subject: [SciPy-Dev] proposal to drop Python 3.6 and NumPy 1.14 / 1.15
In-Reply-To: <CABL7CQjF5ZWOqjmaHF-koBM9rYGZbGKGEmfa5T0W5Nx1_XaDVA@mail.gmail.com>
References: <CABL7CQg_MYscpnnnCcEW+rCzoCCRJvj82jWX_4LmSOSqjgixbQ@mail.gmail.com>
 <CAB2ovovq144TYAzdMLwko7uHKJqnzhedKMAcM-hNqzVWXSjWww@mail.gmail.com>
 <CABL7CQjF5ZWOqjmaHF-koBM9rYGZbGKGEmfa5T0W5Nx1_XaDVA@mail.gmail.com>
Message-ID: <CAB2ovovuDm4m8Vfz=PJaLh2djnNic2tazQWCQp8vq56idcKLOg@mail.gmail.com>

Thanks, Ralf.

Yes, I get that there are good reasons for wanting it both ways and
that is not possible.

That's why I asked whether something like an LTS had ever been
considered, which might make it possible for both sides to be happier.

In the end, if development outpaces users's ability to update, they
will keep using old versions and SciPy/NumPy will move on without
those people until they can update.

Whatever you choose, thanks for all the hard work you folks put into
this.  Computing is a better place for it.


On Sun, Aug 16, 2020 at 11:03 AM Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>
>
> On Sun, Aug 16, 2020 at 2:23 PM Bennet Fauber <bennet at umich.edu> wrote:
>>
>> This post is by way of trying to articulate some concerns from the end
>> user side of things.
>>
>> It looks like NumPy 19.1 supports Python 3.6.
>>
>> > This release supports Python 3.6-3.8. Cython >= 0.29.21 needs to be used when building
>> > with Python 3.9 for testing purposes.
>> https://numpy.org/devdocs/release/1.19.1-notes.html
>>
>> The section of the release notes for NumPy 1.20.0 does not contain the
>> section at the top saying which versions of Python are supported.
>> (https://numpy.org/devdocs/release/1.20.0-notes.html)
>>
>> For Python itself,  I find on their releases, the 3.6.12 schedule was
>>
>>     3.6.12 final: 2020-08-14 (expected)
>> https://www.python.org/dev/peps/pep-0494/#id9
>>
>> but does not seem to have made it, and Python 3.6.11 was released June
>> 27, 2020.  The Python plans for 3.6.13 and beyond are Security fixes
>> only, as needed, until 2021-12.  That seems, from my user perspective,
>> a better date for deprecation and not too far from the just proposed
>> one.
>>
>> Acknowledging the additional burden of 'supporting' older versions of
>> Python, still it would seem that matching NumPy is not a bad thing.
>>
>> From a strictly consumer perspective, where much of the work is
>> getting all of the non-NumPy and non-SciPy functionality to work and
>> be stable, upgrading Python can be very disruptive.  Time spent
>> getting the 'glue' around analytics to work is time we cannot spend on
>> science.  There are large projects that do keep up, but they tend also
>> to be funded well.  The many small labs on my campus do not have
>> funding for software development, are unlikely to ever get any, so any
>> work required to fix software because of updates comes from their
>> budget for science and from their scientists who are not as proficient
>> and therefore not as efficient at adapting to a newer version of
>> Python and all the reinstallation of libraries that attends it.
>>
>> If there were a way to provide _only_ actual bug fixes for some
>> versions longer, a la the LTS releases of some Linux distributions,
>> that might be of benefit to users and would reduce the support burden
>> on developers who would not need to worry about fitting new features
>> into the older framework.
>>
>> All of that is not to say that you should not continue with current
>> deprecation plans.  Just to be sure that these concerns were voiced at
>> least once.  I hope that this perspective is of some use in your
>> considerations.
>
>
> Thanks for your thoughts Bennet. All of this was brought up in the NEP 29 discussion which recommended timelines for Python and NumPy version support. I don't think any of it is specific to SciPy. It's a trade-off: users may want perfect stability and updates for whatever version they're on for as long as possible, but that comes at the cost of making maintainers doing tedious work, like more release effort and figuring out why CI fails on some particular combination of old Python and NumPy versions. That tedious work then reduces the amount of time for tackling more relevant bugs and adding new features, and it likely also reduces total effort spent by maintainers over time (most people do this voluntarily, so motivation comes partly from enjoying to contribute).
>
> Cheers,
> Ralf
>
>
>>
>> Thanks for all the software, regardless!
>>
>>
>> On Sat, Aug 15, 2020 at 6:48 PM Ralf Gommers <ralf.gommers at gmail.com> wrote:
>> >
>> > Hi all,
>> >
>> > Python 3.9 is coming out before the next SciPy release, and NEP 29 recommends to support only Python >=3.7 and NumPy >= 1.16 from last month onwards [1]. I think supporting 3 Python versions and 5 NumPy versions in SciPy 1.6.0 is easily enough. That would also bring us back in line with SciPy on conda-forge, which built against NumPy 1.16 for 1.5.2.
>> >
>> > Any objections?
>> >
>> > Cheers,
>> > Ralf
>> >
>> > [1] https://numpy.org/neps/nep-0029-deprecation_policy.html#support-table
>> > _______________________________________________
>> > SciPy-Dev mailing list
>> > SciPy-Dev at python.org
>> > https://mail.python.org/mailman/listinfo/scipy-dev
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev

From matti.picus at gmail.com  Sun Aug 16 11:22:01 2020
From: matti.picus at gmail.com (Matti Picus)
Date: Sun, 16 Aug 2020 18:22:01 +0300
Subject: [SciPy-Dev] proposal to drop Python 3.6 and NumPy 1.14 / 1.15
In-Reply-To: <CAB2ovovq144TYAzdMLwko7uHKJqnzhedKMAcM-hNqzVWXSjWww@mail.gmail.com>
References: <CABL7CQg_MYscpnnnCcEW+rCzoCCRJvj82jWX_4LmSOSqjgixbQ@mail.gmail.com>
 <CAB2ovovq144TYAzdMLwko7uHKJqnzhedKMAcM-hNqzVWXSjWww@mail.gmail.com>
Message-ID: <cef560fb-bee3-ab6c-c035-2e89020820cc@gmail.com>


On 8/16/20 4:23 PM, Bennet Fauber wrote:
> This post is by way of trying to articulate some concerns from the end
> user side of things.
>
> It looks like NumPy 19.1 supports Python 3.6.
>
>> This release supports Python 3.6-3.8. Cython >= 0.29.21 needs to be used when building
>> with Python 3.9 for testing purposes.
> https://numpy.org/devdocs/release/1.19.1-notes.html
>
> The section of the release notes for NumPy 1.20.0 does not contain the
> section at the top saying which versions of Python are supported.
> (https://numpy.org/devdocs/release/1.20.0-notes.html)
>
> For Python itself,  I find on their releases, the 3.6.12 schedule was
>
>      3.6.12 final: 2020-08-14 (expected)
> https://www.python.org/dev/peps/pep-0494/#id9
>
> but does not seem to have made it, and Python 3.6.11 was released June
> 27, 2020.  The Python plans for 3.6.13 and beyond are Security fixes
> only, as needed, until 2021-12.  That seems, from my user perspective,
> a better date for deprecation and not too far from the just proposed
> one.
>
> Acknowledging the additional burden of 'supporting' older versions of
> Python, still it would seem that matching NumPy is not a bad thing.
>
>  From a strictly consumer perspective, where much of the work is
> getting all of the non-NumPy and non-SciPy functionality to work and
> be stable, upgrading Python can be very disruptive.  Time spent
> getting the 'glue' around analytics to work is time we cannot spend on
> science.  There are large projects that do keep up, but they tend also
> to be funded well.  The many small labs on my campus do not have
> funding for software development, are unlikely to ever get any, so any
> work required to fix software because of updates comes from their
> budget for science and from their scientists who are not as proficient
> and therefore not as efficient at adapting to a newer version of
> Python and all the reinstallation of libraries that attends it.
>

Is there a forum where we could explore the larger question around best 
practices in helping projects and teams keep their software up-to-date 
through rolling updates, test suites, and documentation? A few things 
come to mind:

- developing a culture of version control and tests when creating 
software, even if it is grad students racing towards their degrees

- freezing environments in images, which is a whole discipline unto 
itself, and needed for reproducible science

- offloading the task of maintaining computing environments using things 
like Jupyter Hub or other web-based services


FWIW, python is changing their release cadence from once every 18 months 
to once every 12 months[1] so it is likely this problem will get harder.

Matti

[1] https://www.python.org/dev/peps/pep-0602/#annual-release-cadence

From evgeny.burovskiy at gmail.com  Sun Aug 16 18:18:43 2020
From: evgeny.burovskiy at gmail.com (Evgeni Burovski)
Date: Mon, 17 Aug 2020 01:18:43 +0300
Subject: [SciPy-Dev] proposal to drop Python 3.6 and NumPy 1.14 / 1.15
In-Reply-To: <cef560fb-bee3-ab6c-c035-2e89020820cc@gmail.com>
References: <CABL7CQg_MYscpnnnCcEW+rCzoCCRJvj82jWX_4LmSOSqjgixbQ@mail.gmail.com>
 <CAB2ovovq144TYAzdMLwko7uHKJqnzhedKMAcM-hNqzVWXSjWww@mail.gmail.com>
 <cef560fb-bee3-ab6c-c035-2e89020820cc@gmail.com>
Message-ID: <CAMRo0isPvbSX2bspzpxvW795+i2Vs+-3hZr-8f4bcaGHaX+_xA@mail.gmail.com>

No objections for bumping the minimum NumPy version.
For bumping the python version.. not quite an objection, more of a
suggestion to consider holding it back.
I guess there are three kinds of stakeholders here:
- for end users, upgrading the python version is a fairly major hassle.
Three versions of python now means three years back if they switch to a
12-months cadence. Three years of support is definitely on a low side of
things. (with my PI hat on, I can certainly relate to what Bennet was
saying, even if our lab has it way easier than others).
- for the scipy devs, what exactly drives us to drop it? For 3.5 it was of
course the matmul operator, for 3.6 it's f-strings, what's there in 3.7
which makes it not fun to stay on 3.6?
- for the RM and other packagers: how much of a hassle is it to keep an
older version? I realize it's more wheels to build for the scipy RM, but
that's fairly automated now?



Cheers,
Evgeni


On Sun, Aug 16, 2020 at 6:22 PM Matti Picus <matti.picus at gmail.com> wrote:
>
>
> On 8/16/20 4:23 PM, Bennet Fauber wrote:
> > This post is by way of trying to articulate some concerns from the end
> > user side of things.
> >
> > It looks like NumPy 19.1 supports Python 3.6.
> >
> >> This release supports Python 3.6-3.8. Cython >= 0.29.21 needs to be
used when building
> >> with Python 3.9 for testing purposes.
> > https://numpy.org/devdocs/release/1.19.1-notes.html
> >
> > The section of the release notes for NumPy 1.20.0 does not contain the
> > section at the top saying which versions of Python are supported.
> > (https://numpy.org/devdocs/release/1.20.0-notes.html)
> >
> > For Python itself,  I find on their releases, the 3.6.12 schedule was
> >
> >      3.6.12 final: 2020-08-14 (expected)
> > https://www.python.org/dev/peps/pep-0494/#id9
> >
> > but does not seem to have made it, and Python 3.6.11 was released June
> > 27, 2020.  The Python plans for 3.6.13 and beyond are Security fixes
> > only, as needed, until 2021-12.  That seems, from my user perspective,
> > a better date for deprecation and not too far from the just proposed
> > one.
> >
> > Acknowledging the additional burden of 'supporting' older versions of
> > Python, still it would seem that matching NumPy is not a bad thing.
> >
> >  From a strictly consumer perspective, where much of the work is
> > getting all of the non-NumPy and non-SciPy functionality to work and
> > be stable, upgrading Python can be very disruptive.  Time spent
> > getting the 'glue' around analytics to work is time we cannot spend on
> > science.  There are large projects that do keep up, but they tend also
> > to be funded well.  The many small labs on my campus do not have
> > funding for software development, are unlikely to ever get any, so any
> > work required to fix software because of updates comes from their
> > budget for science and from their scientists who are not as proficient
> > and therefore not as efficient at adapting to a newer version of
> > Python and all the reinstallation of libraries that attends it.
> >
>
> Is there a forum where we could explore the larger question around best
> practices in helping projects and teams keep their software up-to-date
> through rolling updates, test suites, and documentation? A few things
> come to mind:
>
> - developing a culture of version control and tests when creating
> software, even if it is grad students racing towards their degrees
>
> - freezing environments in images, which is a whole discipline unto
> itself, and needed for reproducible science
>
> - offloading the task of maintaining computing environments using things
> like Jupyter Hub or other web-based services
>
>
> FWIW, python is changing their release cadence from once every 18 months
> to once every 12 months[1] so it is likely this problem will get harder.
>
> Matti
>
> [1] https://www.python.org/dev/peps/pep-0602/#annual-release-cadence
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200817/52dcfc2a/attachment.html>

From ilhanpolat at gmail.com  Sun Aug 16 19:16:35 2020
From: ilhanpolat at gmail.com (Ilhan Polat)
Date: Mon, 17 Aug 2020 01:16:35 +0200
Subject: [SciPy-Dev] proposal to drop Python 3.6 and NumPy 1.14 / 1.15
In-Reply-To: <CAMRo0isPvbSX2bspzpxvW795+i2Vs+-3hZr-8f4bcaGHaX+_xA@mail.gmail.com>
References: <CABL7CQg_MYscpnnnCcEW+rCzoCCRJvj82jWX_4LmSOSqjgixbQ@mail.gmail.com>
 <CAB2ovovq144TYAzdMLwko7uHKJqnzhedKMAcM-hNqzVWXSjWww@mail.gmail.com>
 <cef560fb-bee3-ab6c-c035-2e89020820cc@gmail.com>
 <CAMRo0isPvbSX2bspzpxvW795+i2Vs+-3hZr-8f4bcaGHaX+_xA@mail.gmail.com>
Message-ID: <CAEBuzr8BT-CPVYcQjdTt6ARKGcQUPfOGe8Nv5qH53By26aFvgw@mail.gmail.com>

I have too many personas in my head arguing about this. And many of them
are in conflict.

1- I think I had enough arguments online to be known as buying none of that
scientific stability views. That doesn't mean I don't get their point, I
really really do, but still. It's in the job description. Software is a big
part of science just like the papers. (What Matti said)
2- Even a two-year old code is not safe to let it collect dust and
constantly requires attention and becomes a liability rather than a tool
(what Bennet said).
3- Matlab breaks things all the time with 0 user consideration. None. But
admittedly they break in style. We shouldn't be like them (not even close)
but the point is when it is a commercial product, we don't hear the uproar
we receive. People silently fix things.
4- Just because we update the python version, a lot of packages stop
working. This is seriously disheartening. Happens to me all the time
(protobuf, influxdb etc). And really annoying if one package has not
released the new version yet.
5- Python is releasing too quick. I know Python is not Fortran (thankfully)
but this is the other end of the spectrum. With every version one or two of
us spent at least 2 weeks of intensive "What did they change on Windows
again?" bug hunting. Hence our platform is not reliable and now they want
12 months.  (What Ralf said)
6- There are always new users, downloading Python for the first time and it
is expected that SciPy stack should work off-the-shelf. Hence we don't have
the luxury to wait and see (see 4th item).
7- If there are limited resources for software support, then why people
take the risk and update their stuff is something that has been discussed
for decades now. And no one seems to converge to a point. (What Matti said)
8-  what Evgeni suggested

I'm not sure what to do about this but I am also more worried about the
Python version than the NumPy version, my limited anectodal evidence around
me suggests that people update NumPy + SciPy together mainly when they use
pip with -U (--upgrade) on some package that has SciPy as a dependency. So
I feel that it is safe to bump.





On Mon, Aug 17, 2020 at 12:19 AM Evgeni Burovski <evgeny.burovskiy at gmail.com>
wrote:

> No objections for bumping the minimum NumPy version.
> For bumping the python version.. not quite an objection, more of a
> suggestion to consider holding it back.
> I guess there are three kinds of stakeholders here:
> - for end users, upgrading the python version is a fairly major hassle.
> Three versions of python now means three years back if they switch to a
> 12-months cadence. Three years of support is definitely on a low side of
> things. (with my PI hat on, I can certainly relate to what Bennet was
> saying, even if our lab has it way easier than others).
> - for the scipy devs, what exactly drives us to drop it? For 3.5 it was of
> course the matmul operator, for 3.6 it's f-strings, what's there in 3.7
> which makes it not fun to stay on 3.6?
> - for the RM and other packagers: how much of a hassle is it to keep an
> older version? I realize it's more wheels to build for the scipy RM, but
> that's fairly automated now?
>
>
>
> Cheers,
> Evgeni
>
>
> On Sun, Aug 16, 2020 at 6:22 PM Matti Picus <matti.picus at gmail.com> wrote:
> >
> >
> > On 8/16/20 4:23 PM, Bennet Fauber wrote:
> > > This post is by way of trying to articulate some concerns from the end
> > > user side of things.
> > >
> > > It looks like NumPy 19.1 supports Python 3.6.
> > >
> > >> This release supports Python 3.6-3.8. Cython >= 0.29.21 needs to be
> used when building
> > >> with Python 3.9 for testing purposes.
> > > https://numpy.org/devdocs/release/1.19.1-notes.html
> > >
> > > The section of the release notes for NumPy 1.20.0 does not contain the
> > > section at the top saying which versions of Python are supported.
> > > (https://numpy.org/devdocs/release/1.20.0-notes.html)
> > >
> > > For Python itself,  I find on their releases, the 3.6.12 schedule was
> > >
> > >      3.6.12 final: 2020-08-14 (expected)
> > > https://www.python.org/dev/peps/pep-0494/#id9
> > >
> > > but does not seem to have made it, and Python 3.6.11 was released June
> > > 27, 2020.  The Python plans for 3.6.13 and beyond are Security fixes
> > > only, as needed, until 2021-12.  That seems, from my user perspective,
> > > a better date for deprecation and not too far from the just proposed
> > > one.
> > >
> > > Acknowledging the additional burden of 'supporting' older versions of
> > > Python, still it would seem that matching NumPy is not a bad thing.
> > >
> > >  From a strictly consumer perspective, where much of the work is
> > > getting all of the non-NumPy and non-SciPy functionality to work and
> > > be stable, upgrading Python can be very disruptive.  Time spent
> > > getting the 'glue' around analytics to work is time we cannot spend on
> > > science.  There are large projects that do keep up, but they tend also
> > > to be funded well.  The many small labs on my campus do not have
> > > funding for software development, are unlikely to ever get any, so any
> > > work required to fix software because of updates comes from their
> > > budget for science and from their scientists who are not as proficient
> > > and therefore not as efficient at adapting to a newer version of
> > > Python and all the reinstallation of libraries that attends it.
> > >
> >
> > Is there a forum where we could explore the larger question around best
> > practices in helping projects and teams keep their software up-to-date
> > through rolling updates, test suites, and documentation? A few things
> > come to mind:
> >
> > - developing a culture of version control and tests when creating
> > software, even if it is grad students racing towards their degrees
> >
> > - freezing environments in images, which is a whole discipline unto
> > itself, and needed for reproducible science
> >
> > - offloading the task of maintaining computing environments using things
> > like Jupyter Hub or other web-based services
> >
> >
> > FWIW, python is changing their release cadence from once every 18 months
> > to once every 12 months[1] so it is likely this problem will get harder.
> >
> > Matti
> >
> > [1] https://www.python.org/dev/peps/pep-0602/#annual-release-cadence
> > _______________________________________________
> > SciPy-Dev mailing list
> > SciPy-Dev at python.org
> > https://mail.python.org/mailman/listinfo/scipy-dev
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200817/e907f295/attachment-0001.html>

From rlucas7 at vt.edu  Sun Aug 16 20:03:27 2020
From: rlucas7 at vt.edu (rlucas7 at vt.edu)
Date: Sun, 16 Aug 2020 20:03:27 -0400
Subject: [SciPy-Dev] Thanks wiki protocol
In-Reply-To: <CADuxUixohC5YFQ2Mtqf4CN9xN4Z3K6NNj1cpuao0JQthjC0rPQ@mail.gmail.com>
References: <CADuxUixohC5YFQ2Mtqf4CN9xN4Z3K6NNj1cpuao0JQthjC0rPQ@mail.gmail.com>
Message-ID: <F04F8FE0-5F79-40DA-A47F-9CF18D9A791C@vt.edu>


> On Aug 15, 2020, at 8:15 PM, Matt Haberland <mhaberla at calpoly.edu> wrote:
> 
> ?
> +1 for retiring the manually updated file. 
> 
>> On Sat, Aug 15, 2020, 3:55 PM Ralf Gommers <ralf.gommers at gmail.com> wrote:
>> 
>> 
>>> On Sat, Aug 15, 2020 at 9:43 PM <rlucas7 at vt.edu> wrote:
>>> Hi scipy-dev, 
>>> 
>>> For work on the 1.5 release we used a wiki rather than have people add commits to the thanks file in the repo. 
>>> I?m assuming this was to alleviate rebase conflicts to merge. My question is, 
>>> 
>>> Is using a wikifor tracking thanks to add in each release the way we track thanks entries moving forward? 
>>> 
>>> To be clear these would be separate wiki pages from the release notes wiki pages. 
>> 
>> That does sound like a good idea.
>> 
>> A perhaps even better one is to retire the THANKS.txt file. It really doesn't have much value, is very inconsistently updated and old entries are horribly out of date (e.g. Pauli is thanked for docs work and interpolation bug fixes - that doesn't quite cover his contributions :)).
>> 

Yeah agree. 

>> I'd prefer to extract the relevant historical info and put that on the website. Or just have a full list of contributors that's autogenerated.

We already do the auto generate for each release IIRC. At least I assume that the RM (Tyler) does that in an auto generated way. 

>> The thanks to institutions at the bottom of the file already lives at https://scipy.org/scipylib/donations.html#acknowledgements

We could move the individual thanks entries (as currently written) to the scipy site. As a separate section on this acknowledgements page.

That would be a one off change but what about moving forward? 

I?ve noticed on the release notes there are indications for first time contributors. I?ve not used those tools myself.

 Is there a way we can get a list of these first time contributors/individuals from the same tool(s)? 

That would probably be the easiest thing to do if we want to periodically add/update a contributors page. 

>> Cheers,
>> Ralf
>> 
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200816/fa84a65c/attachment.html>

From jacobr at ethz.ch  Mon Aug 17 04:38:28 2020
From: jacobr at ethz.ch (Romain Jacob)
Date: Mon, 17 Aug 2020 10:38:28 +0200
Subject: [SciPy-Dev] Adding non-parametric methods to scipy.stats
In-Reply-To: <3893d209-892f-b738-07ba-3a03aa4b502a@tik.ee.ethz.ch>
References: <6a5dee0d-3d55-409d-6bf8-b86e3f465275@tik.ee.ethz.ch>
 <CADuxUiz45SJEZm5Mm6V6cC9Z0vMUxJt9D5DD_Shm=PeKrC=MJg@mail.gmail.com>
 <9ed88efe-5504-3d60-ab73-3ffc7cce1edf@tik.ee.ethz.ch>
 <CADuxUiym12uh4bbg1W_m6Lj6xb2LJ4Eb+wyJok7EyWmrDv8enQ@mail.gmail.com>
 <cedc67bf-6dcf-4267-f7c8-84d23200f45c@ethz.ch>
 <CADuxUiy+GE91fwcBtVEWbvn2MMDkiJO5BPd9w5z2aTbqQz52pw@mail.gmail.com>
 <CAMMTP+A43zKdBNHBmqz4ij=80BZrci6tNk4Adr9FEBMp9uwMNQ@mail.gmail.com>
 <CAGzF1ucda-ucx+9n=H58OrA2Rd7uVwf-UpxeAqZNNwvegvB0Pw@mail.gmail.com>
 <a54e89f1-2a6f-ced2-e2a8-7391a76937e0@tik.ee.ethz.ch>
 <CAMMTP+AMX1qAiw5B9pQrWGnNRqNXLvLHUPaBnD92q0j-M-eR+w@mail.gmail.com>
 <CAMMTP+CFXhwPLO-GFPUftDPYvWUvo2xw1C2f5CJu=3ZEz0WqQg@mail.gmail.com>
 <3893d209-892f-b738-07ba-3a03aa4b502a@tik.ee.ethz.ch>
Message-ID: <64991457-41eb-d2da-b32b-17c35c0726e4@tik.ee.ethz.ch>

Hello everyone,

I've submitted the PR adding support for non-parametric confidence 
intervals for quantiles (https://github.com/scipy/scipy/pull/12680). 
There has been quite some comments made already, which I fixed 
appropriately I believe

Will be happy to get some more feedback or see the PR merged :-)

Note: the last commit has a CI failing apparently due to a file change 
in `scipy/sparse/linalg/` which is completely unrelated. I'm not sure 
how to go about this... ?

Cheers,
-- 
Romain

On 15/06/2020 08:27, Romain Jacob wrote:
> On 13/06/2020 20:54, josef.pktd at gmail.com wrote:
>> On Fri, Jun 12, 2020 at 11:29 AM <josef.pktd at gmail.com 
>> <mailto:josef.pktd at gmail.com>> wrote:
>>
>>     On Fri, Jun 12, 2020 at 1:58 AM Romain Jacob <jacobr at ethz.ch
>>     <mailto:jacobr at ethz.ch>> wrote:
>>
>>         On 11/06/2020 20:54, Warren Weckesser wrote:
>>>         On 6/11/20,josef.pktd at gmail.com  <mailto:josef.pktd at gmail.com>  <josef.pktd at gmail.com>  <mailto:josef.pktd at gmail.com>  wrote:
>>>>         I think it would make a good and useful addition and fit into scipy.stats.
>>>>         There are no pure confint functions yet, AFAIR.
>>>         I agree with Josef and Matt, this looks like it would be a nice
>>>         addition to SciPy.  At the moment, I'm not sure what the API should
>>>         look like.  Romain, is the work that you've already done available
>>>         online somewhere?
>>>
>>>         Warren
>>
>>         Yes, I have some functional implementation available here:
>>         https://github.com/TriScale-Anon/triscale/blob/master/helpers.py#L397
>>
>>
>>     An implementation detail:
>>     binom has cdf and ppf functions
>>     My guess, not verified, is that we can just use binom.interval
>>
>>     (at least I used those for similar cases)
>>
>>
>> I found my version again
>> https://github.com/statsmodels/statsmodels/issues/6562#issuecomment-592769480 
>>
>>
>> I guess that's the same for two sided confint as the references.
>> It doesn't have interpolation if that could be applied in this case.
>>
> I don't entirely follow what you mean here: that the building of the 
> probabilities in these two lines( 
> https://github.com/TriScale-Anon/triscale/blob/master/helpers.py#L438 
> and L439) can be built directly form binom without np.cumsum? That 
> definitely correct (I actually have code also doing that somewhere).
>
> I did not know about the `interval` method. That's sound interesting 
> indeed, but it's not 100% clear to me how the uniqueness problem is 
> handled. I looked for the implementation of the method but couldn't 
> find it in `binom`... I'm looking in the wrong place?
>
> Cheers,
> -- 
> Romain
>
>
>> This will eventually end up in statsmodels, but I don't know yet 
>> where. That's not a reason not to add it to scipy.stats.
>>
>> Josef
>>
>>
>>     Josef
>>
>>         There is quite some work to be done on formatting and
>>         documentation to comply with the SciPy standards, but
>>         functionally it's already there (and as you'll see, the
>>         method is quite simple).
>>
>>         Cheers,
>>         -- 
>>         Romain
>>
>>>>         I recently wrote a function for the confidence interval for the median,
>>>>         mainly because I ran into the formulas that were easy to code.
>>>>         related open issue: how do we get confidence intervals for QQ-plot.
>>>>
>>>>         aside: I don't like "percent", I prefer quantiles in [0, 1]. See discussion
>>>>         a while ago in numpy.
>>>>
>>>>         Josef
>>>>
>>>>
>>>>         On Thu, Jun 11, 2020 at 1:01 PM Matt Haberland<mhaberla at calpoly.edu>  <mailto:mhaberla at calpoly.edu>
>>>>         wrote:
>>>>
>>>>>         OK, we should let our statistics experts weigh in on this. (I'm not
>>>>>         actually one of them.)
>>>>>
>>>>>         On Wed, Jun 10, 2020 at 10:46 PM Romain Jacob<jacobr at ethz.ch>  <mailto:jacobr at ethz.ch>  wrote:
>>>>>
>>>>>>         I think a dedicated function makes more sense. This function takes as
>>>>>>         input an array, a percentile and a confidence level, and returns the
>>>>>>         corresponding one-sided confidence intervals.
>>>>>>
>>>>>>         I quickly looked at the list of existing functions in scipy.stats but
>>>>>>         did
>>>>>>         not see any function in "summary statistics" that does similar things. So
>>>>>>         I
>>>>>>         would go for a new function.
>>>>>>         On 10/06/2020 20:38, Matt Haberland wrote:
>>>>>>
>>>>>>         Where do you envision this living in SciPy? In its own function, or
>>>>>>         added
>>>>>>         functionality to other functions e.g. scipy.stats.percentileofscore
>>>>>>         <https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.percentileofscore.html#scipy.stats.percentileofscore>  <https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.percentileofscore.html#scipy.stats.percentileofscore>
>>>>>>         ?
>>>>>>
>>>>>>         On Tue, Jun 9, 2020 at 11:12 PM Romain Jacob<jacobr at ethz.ch>  <mailto:jacobr at ethz.ch>  wrote:
>>>>>>
>>>>>>>         On 09/06/2020 20:18, Matt Haberland wrote:
>>>>>>>
>>>>>>>         Yes, I think we would be interested in confidence intervals, but I
>>>>>>>         think
>>>>>>>         the algorithm should be very well standard/cited, even if it's not the
>>>>>>>         best/most modern.
>>>>>>>
>>>>>>>         Yes definitely! We did not invented the method I am referring to, it a
>>>>>>>         long-known approach (first proposed by Thompson in 1936 [1], extended
>>>>>>>         later
>>>>>>>         and commonly found in textbooks, eg [2,3]). This method is very simple,
>>>>>>>         quite powerful, yet it has been largely overlooked in many scientific
>>>>>>>         fields. I found no available implementation to facilitate its use (at
>>>>>>>         least
>>>>>>>         not in Python, there may be something in R, I have not looked).
>>>>>>>
>>>>>>>         [1]https://www.jstor.org/stable/2957563
>>>>>>>         [2]doi.org/10.1002/0471722162.ch7  <http://doi.org/10.1002/0471722162.ch7>
>>>>>>>         [3]https://perfeval.epfl.ch/
>>>>>>>
>>>>>>>         @WarrenWeckesser and I had planned to work on confidence intervals for
>>>>>>>         the test statistics returned by our statistical tests
>>>>>>>         <https://docs.scipy.org/doc/scipy/reference/stats.html#statistical-tests>  <https://docs.scipy.org/doc/scipy/reference/stats.html#statistical-tests>.
>>>>>>>
>>>>>>>
>>>>>>>         That is also definitely interesting, although I am not myself an expert
>>>>>>>         in that area. I am glad to see that the complete list contains some
>>>>>>>         non-parametric tests :-)
>>>>>>>
>>>>>>>         Cheers,
>>>>>>>         --
>>>>>>>         Romain
>>>>>>>
>>>>>>>
>>>>>>>         On Mon, Jun 8, 2020 at 2:11 AM Romain Jacob<jacobr at ethz.ch>  <mailto:jacobr at ethz.ch>  wrote:
>>>>>>>
>>>>>>>>         Hello everyone,
>>>>>>>>
>>>>>>>>         I have been working for some time on the implementation of
>>>>>>>>         non-parametric methods to compute confidence intervals for
>>>>>>>>         percentiles.
>>>>>>>>         There are some very interesting results in the literature (see e.g. a
>>>>>>>>         nice
>>>>>>>>         pitch in [1]) which I think it would be great to add to SciPy to make
>>>>>>>>         them
>>>>>>>>         more readily available. It also seems to be rather in line with
>>>>>>>>         "recent"
>>>>>>>>         discussions of the roadmap for scipy.stats [2].
>>>>>>>>
>>>>>>>>         I would be interested in contributing this. What do you think?
>>>>>>>>
>>>>>>>>         Cheers,
>>>>>>>>         --
>>>>>>>>         Romain
>>>>>>>>
>>>>>>>>         [1]https://ieeexplore.ieee.org/document/6841797
>>>>>>>>         [2]https://github.com/scipy/scipy/issues/10577
>>>>>>>>         --
>>>>>>>>         Romain Jacob
>>>>>>>>         Postdoctoral Researcher
>>>>>>>>         ETH Zurich - Computer Engineering and Networks Laboratory
>>>>>>>>         www.romainjacob.net  <http://www.romainjacob.net>
>>>>>>>>         @RJacobPartner<https://twitter.com/RJacobPartner>  <https://twitter.com/RJacobPartner>
>>>>>>>>         Gloriastrasse 35, ETZ G75
>>>>>>>>         8092 Zurich
>>>>>>>>         +41 7 68 16 88 22
>>>>>>>>         _______________________________________________
>>>>>>>>         SciPy-Dev mailing list
>>>>>>>>         SciPy-Dev at python.org  <mailto:SciPy-Dev at python.org>
>>>>>>>>         https://mail.python.org/mailman/listinfo/scipy-dev
>>>>>>>>
>>>>>>>         --
>>>>>>>         Matt Haberland
>>>>>>>         Assistant Professor
>>>>>>>         BioResource and Agricultural Engineering
>>>>>>>         08A-3K, Cal Poly
>>>>>>>
>>>>>>>         _______________________________________________
>>>>>>>         SciPy-Dev mailing
>>>>>>>         listSciPy-Dev at python.orghttps://mail.python.org/mailman/listinfo/scipy-dev  <mailto:listSciPy-Dev at python.orghttps://mail.python.org/mailman/listinfo/scipy-dev>
>>>>>>>
>>>>>>>         --
>>>>>>>         Romain Jacob
>>>>>>>         Postdoctoral Researcher
>>>>>>>         ETH Zurich - Computer Engineering and Networks Laboratory
>>>>>>>         www.romainjacob.net  <http://www.romainjacob.net>
>>>>>>>         @RJacobPartner<https://twitter.com/RJacobPartner>  <https://twitter.com/RJacobPartner>
>>>>>>>         Gloriastrasse 35, ETZ G75
>>>>>>>         8092 Zurich
>>>>>>>         +41 7 68 16 88 22
>>>>>>>         _______________________________________________
>>>>>>>         SciPy-Dev mailing list
>>>>>>>         SciPy-Dev at python.org  <mailto:SciPy-Dev at python.org>
>>>>>>>         https://mail.python.org/mailman/listinfo/scipy-dev
>>>>>>>
>>>>>>         --
>>>>>>         Matt Haberland
>>>>>>         Assistant Professor
>>>>>>         BioResource and Agricultural Engineering
>>>>>>         08A-3K, Cal Poly
>>>>>>
>>>>>>         _______________________________________________
>>>>>>         SciPy-Dev mailing
>>>>>>         listSciPy-Dev at python.orghttps://mail.python.org/mailman/listinfo/scipy-dev  <mailto:listSciPy-Dev at python.orghttps://mail.python.org/mailman/listinfo/scipy-dev>
>>>>>>
>>>>>>         _______________________________________________
>>>>>>         SciPy-Dev mailing list
>>>>>>         SciPy-Dev at python.org  <mailto:SciPy-Dev at python.org>
>>>>>>         https://mail.python.org/mailman/listinfo/scipy-dev
>>>>>>
>>>>>         --
>>>>>         Matt Haberland
>>>>>         Assistant Professor
>>>>>         BioResource and Agricultural Engineering
>>>>>         08A-3K, Cal Poly
>>>>>         _______________________________________________
>>>>>         SciPy-Dev mailing list
>>>>>         SciPy-Dev at python.org  <mailto:SciPy-Dev at python.org>
>>>>>         https://mail.python.org/mailman/listinfo/scipy-dev
>>>>>
>>>         _______________________________________________
>>>         SciPy-Dev mailing list
>>>         SciPy-Dev at python.org  <mailto:SciPy-Dev at python.org>
>>>         https://mail.python.org/mailman/listinfo/scipy-dev
>>         -- 
>>         Romain Jacob
>>         Postdoctoral Researcher
>>         ETH Zurich - Computer Engineering and Networks Laboratory
>>         www.romainjacob.net <https://www.romainjacob.net/>
>>         @RJacobPartner <https://twitter.com/RJacobPartner>
>>         Gloriastrasse 35, ETZ G75
>>         8092 Zurich
>>         +41 7 68 16 88 22
>>         _______________________________________________
>>         SciPy-Dev mailing list
>>         SciPy-Dev at python.org <mailto:SciPy-Dev at python.org>
>>         https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
> -- 
> Romain Jacob
> Postdoctoral Researcher
> ETH Zurich - Computer Engineering and Networks Laboratory
> www.romainjacob.net <https://www.romainjacob.net/>
> @RJacobPartner <https://twitter.com/RJacobPartner>
> Gloriastrasse 35, ETZ G75
> 8092 Zurich
> +41 7 68 16 88 22
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
-- 
Romain Jacob
Postdoctoral Researcher
ETH Zurich - Computer Engineering and Networks Laboratory
www.romainjacob.net <https://www.romainjacob.net/>
@RJacobPartner <https://twitter.com/RJacobPartner>
Gloriastrasse 35, ETZ G75
8092 Zurich
+41 7 68 16 88 22
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200817/98be1711/attachment-0001.html>

From bennet at umich.edu  Mon Aug 17 08:58:54 2020
From: bennet at umich.edu (Bennet Fauber)
Date: Mon, 17 Aug 2020 08:58:54 -0400
Subject: [SciPy-Dev] proposal to drop Python 3.6 and NumPy 1.14 / 1.15
In-Reply-To: <CAEBuzr8BT-CPVYcQjdTt6ARKGcQUPfOGe8Nv5qH53By26aFvgw@mail.gmail.com>
References: <CABL7CQg_MYscpnnnCcEW+rCzoCCRJvj82jWX_4LmSOSqjgixbQ@mail.gmail.com>
 <CAB2ovovq144TYAzdMLwko7uHKJqnzhedKMAcM-hNqzVWXSjWww@mail.gmail.com>
 <cef560fb-bee3-ab6c-c035-2e89020820cc@gmail.com>
 <CAMRo0isPvbSX2bspzpxvW795+i2Vs+-3hZr-8f4bcaGHaX+_xA@mail.gmail.com>
 <CAEBuzr8BT-CPVYcQjdTt6ARKGcQUPfOGe8Nv5qH53By26aFvgw@mail.gmail.com>
Message-ID: <CAB2ovouOuBDXK9oNCHCgarzsS7i=dSES+wbPT-1=5sUDCA7Cfw@mail.gmail.com>

Boy, yes, Ilhan, I agree.  Too many different roles in my head!

Just to be clear, I also am not concerned about deprecating older
NumPy, only the base Python version, and specifically 3.6.  It looks
like the latest fully released NumPy supports Python 3.6.  In my Linux
world, we still have a preponderance of RedHat/CentOS 7 installed, and
it does not easily provide Python 3.7:  python36 is available from
EPEL.  Our most prevalent Ubuntu is 16.04.  We will probably jump to
20.04 next calendar year.

I had not realized that Python itself was going to a 12-month release
cycle.  That seems to me destructive and predicated on the notion that
software is disposable.

I think there is a very good reason why many Linux distributions have
separate distributions for longer term stability and more up-to-date
features.  Red Hat and Fedora, Ubuntu and LTS, Debian Stable and
Unstable.  Is that viable with SciPy, NumPy, where only the bug fixes
get backported, which might reduce the cost of longer maintenance
(except for the wheels, as noted)?

Thanks for the discussion on this, no matter where it ends up.



On Sun, Aug 16, 2020 at 7:17 PM Ilhan Polat <ilhanpolat at gmail.com> wrote:
>
> I have too many personas in my head arguing about this. And many of them are in conflict.
>
> 1- I think I had enough arguments online to be known as buying none of that scientific stability views. That doesn't mean I don't get their point, I really really do, but still. It's in the job description. Software is a big part of science just like the papers. (What Matti said)
> 2- Even a two-year old code is not safe to let it collect dust and constantly requires attention and becomes a liability rather than a tool (what Bennet said).
> 3- Matlab breaks things all the time with 0 user consideration. None. But admittedly they break in style. We shouldn't be like them (not even close) but the point is when it is a commercial product, we don't hear the uproar we receive. People silently fix things.
> 4- Just because we update the python version, a lot of packages stop working. This is seriously disheartening. Happens to me all the time (protobuf, influxdb etc). And really annoying if one package has not released the new version yet.
> 5- Python is releasing too quick. I know Python is not Fortran (thankfully) but this is the other end of the spectrum. With every version one or two of us spent at least 2 weeks of intensive "What did they change on Windows again?" bug hunting. Hence our platform is not reliable and now they want  12 months.  (What Ralf said)
> 6- There are always new users, downloading Python for the first time and it is expected that SciPy stack should work off-the-shelf. Hence we don't have the luxury to wait and see (see 4th item).
> 7- If there are limited resources for software support, then why people take the risk and update their stuff is something that has been discussed for decades now. And no one seems to converge to a point. (What Matti said)
> 8-  what Evgeni suggested
>
> I'm not sure what to do about this but I am also more worried about the Python version than the NumPy version, my limited anectodal evidence around me suggests that people update NumPy + SciPy together mainly when they use pip with -U (--upgrade) on some package that has SciPy as a dependency. So I feel that it is safe to bump.
>
>
>
>
>
> On Mon, Aug 17, 2020 at 12:19 AM Evgeni Burovski <evgeny.burovskiy at gmail.com> wrote:
>>
>> No objections for bumping the minimum NumPy version.
>> For bumping the python version.. not quite an objection, more of a suggestion to consider holding it back.
>> I guess there are three kinds of stakeholders here:
>> - for end users, upgrading the python version is a fairly major hassle. Three versions of python now means three years back if they switch to a 12-months cadence. Three years of support is definitely on a low side of things. (with my PI hat on, I can certainly relate to what Bennet was saying, even if our lab has it way easier than others).
>> - for the scipy devs, what exactly drives us to drop it? For 3.5 it was of course the matmul operator, for 3.6 it's f-strings, what's there in 3.7 which makes it not fun to stay on 3.6?
>> - for the RM and other packagers: how much of a hassle is it to keep an older version? I realize it's more wheels to build for the scipy RM, but that's fairly automated now?
>>
>>
>>
>> Cheers,
>> Evgeni
>>
>>
>> On Sun, Aug 16, 2020 at 6:22 PM Matti Picus <matti.picus at gmail.com> wrote:
>> >
>> >
>> > On 8/16/20 4:23 PM, Bennet Fauber wrote:
>> > > This post is by way of trying to articulate some concerns from the end
>> > > user side of things.
>> > >
>> > > It looks like NumPy 19.1 supports Python 3.6.
>> > >
>> > >> This release supports Python 3.6-3.8. Cython >= 0.29.21 needs to be used when building
>> > >> with Python 3.9 for testing purposes.
>> > > https://numpy.org/devdocs/release/1.19.1-notes.html
>> > >
>> > > The section of the release notes for NumPy 1.20.0 does not contain the
>> > > section at the top saying which versions of Python are supported.
>> > > (https://numpy.org/devdocs/release/1.20.0-notes.html)
>> > >
>> > > For Python itself,  I find on their releases, the 3.6.12 schedule was
>> > >
>> > >      3.6.12 final: 2020-08-14 (expected)
>> > > https://www.python.org/dev/peps/pep-0494/#id9
>> > >
>> > > but does not seem to have made it, and Python 3.6.11 was released June
>> > > 27, 2020.  The Python plans for 3.6.13 and beyond are Security fixes
>> > > only, as needed, until 2021-12.  That seems, from my user perspective,
>> > > a better date for deprecation and not too far from the just proposed
>> > > one.
>> > >
>> > > Acknowledging the additional burden of 'supporting' older versions of
>> > > Python, still it would seem that matching NumPy is not a bad thing.
>> > >
>> > >  From a strictly consumer perspective, where much of the work is
>> > > getting all of the non-NumPy and non-SciPy functionality to work and
>> > > be stable, upgrading Python can be very disruptive.  Time spent
>> > > getting the 'glue' around analytics to work is time we cannot spend on
>> > > science.  There are large projects that do keep up, but they tend also
>> > > to be funded well.  The many small labs on my campus do not have
>> > > funding for software development, are unlikely to ever get any, so any
>> > > work required to fix software because of updates comes from their
>> > > budget for science and from their scientists who are not as proficient
>> > > and therefore not as efficient at adapting to a newer version of
>> > > Python and all the reinstallation of libraries that attends it.
>> > >
>> >
>> > Is there a forum where we could explore the larger question around best
>> > practices in helping projects and teams keep their software up-to-date
>> > through rolling updates, test suites, and documentation? A few things
>> > come to mind:
>> >
>> > - developing a culture of version control and tests when creating
>> > software, even if it is grad students racing towards their degrees
>> >
>> > - freezing environments in images, which is a whole discipline unto
>> > itself, and needed for reproducible science
>> >
>> > - offloading the task of maintaining computing environments using things
>> > like Jupyter Hub or other web-based services
>> >
>> >
>> > FWIW, python is changing their release cadence from once every 18 months
>> > to once every 12 months[1] so it is likely this problem will get harder.
>> >
>> > Matti
>> >
>> > [1] https://www.python.org/dev/peps/pep-0602/#annual-release-cadence
>> > _______________________________________________
>> > SciPy-Dev mailing list
>> > SciPy-Dev at python.org
>> > https://mail.python.org/mailman/listinfo/scipy-dev
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev

From tyler.je.reddy at gmail.com  Mon Aug 17 22:39:26 2020
From: tyler.je.reddy at gmail.com (Tyler Reddy)
Date: Mon, 17 Aug 2020 20:39:26 -0600
Subject: [SciPy-Dev] Thanks wiki protocol
In-Reply-To: <F04F8FE0-5F79-40DA-A47F-9CF18D9A791C@vt.edu>
References: <CADuxUixohC5YFQ2Mtqf4CN9xN4Z3K6NNj1cpuao0JQthjC0rPQ@mail.gmail.com>
 <F04F8FE0-5F79-40DA-A47F-9CF18D9A791C@vt.edu>
Message-ID: <CAHPuU_b+facUkYt3M67W+jBzSCw9eYUQF01Y7azBzF_grapB9Q@mail.gmail.com>

For the author lists (with new contributors highlighted), the command looks
like " python tools/authors.py 042731365..fb51f487c8 "

I think some of those release management tools are dependent on proper
usage of milestones, though perhaps not in that specific case where the
hash range is provided directly.

On Sun, 16 Aug 2020 at 18:04, <rlucas7 at vt.edu> wrote:

>
> On Aug 15, 2020, at 8:15 PM, Matt Haberland <mhaberla at calpoly.edu> wrote:
>
> ?
> +1 for retiring the manually updated file.
>
> On Sat, Aug 15, 2020, 3:55 PM Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>>
>>
>> On Sat, Aug 15, 2020 at 9:43 PM <rlucas7 at vt.edu> wrote:
>>
>>> Hi scipy-dev,
>>>
>>> For work on the 1.5 release we used a wiki rather than have people add
>>> commits to the thanks file in the repo.
>>> I?m assuming this was to alleviate rebase conflicts to merge. My
>>> question is,
>>>
>>> Is using a wikifor tracking thanks to add in each release the way we
>>> track thanks entries moving forward?
>>>
>>> To be clear these would be separate wiki pages from the release notes
>>> wiki pages.
>>>
>>
>> That does sound like a good idea.
>>
>> A perhaps even better one is to retire the THANKS.txt file. It really
>> doesn't have much value, is very inconsistently updated and old entries are
>> horribly out of date (e.g. Pauli is thanked for docs work and interpolation
>> bug fixes - that doesn't quite cover his contributions :)).
>>
>>
> Yeah agree.
>
> I'd prefer to extract the relevant historical info and put that on the
>> website. Or just have a full list of contributors that's autogenerated.
>>
>
> We already do the auto generate for each release IIRC. At least I assume
> that the RM (Tyler) does that in an auto generated way.
>
> The thanks to institutions at the bottom of the file already lives at
>> https://scipy.org/scipylib/donations.html#acknowledgements
>>
>
> We could move the individual thanks entries (as currently written) to the
> scipy site. As a separate section on this acknowledgements page.
>
> That would be a one off change but what about moving forward?
>
> I?ve noticed on the release notes there are indications for first time
> contributors. I?ve not used those tools myself.
>
>  Is there a way we can get a list of these first time
> contributors/individuals from the same tool(s)?
>
> That would probably be the easiest thing to do if we want to periodically
> add/update a contributors page.
>
> Cheers,
>> Ralf
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200817/1b564439/attachment.html>

From tyler.je.reddy at gmail.com  Mon Aug 17 22:58:30 2020
From: tyler.je.reddy at gmail.com (Tyler Reddy)
Date: Mon, 17 Aug 2020 20:58:30 -0600
Subject: [SciPy-Dev] proposal to drop Python 3.6 and NumPy 1.14 / 1.15
In-Reply-To: <CAB2ovouOuBDXK9oNCHCgarzsS7i=dSES+wbPT-1=5sUDCA7Cfw@mail.gmail.com>
References: <CABL7CQg_MYscpnnnCcEW+rCzoCCRJvj82jWX_4LmSOSqjgixbQ@mail.gmail.com>
 <CAB2ovovq144TYAzdMLwko7uHKJqnzhedKMAcM-hNqzVWXSjWww@mail.gmail.com>
 <cef560fb-bee3-ab6c-c035-2e89020820cc@gmail.com>
 <CAMRo0isPvbSX2bspzpxvW795+i2Vs+-3hZr-8f4bcaGHaX+_xA@mail.gmail.com>
 <CAEBuzr8BT-CPVYcQjdTt6ARKGcQUPfOGe8Nv5qH53By26aFvgw@mail.gmail.com>
 <CAB2ovouOuBDXK9oNCHCgarzsS7i=dSES+wbPT-1=5sUDCA7Cfw@mail.gmail.com>
Message-ID: <CAHPuU_aE5xRPnYspa0rBYwq0zTyOVCAqCFO+rkH=KWvqXqtC5Q@mail.gmail.com>

FWIW, the policy for all minor point releases is already that we only
backport bug fixes and not enhancements, so there's not much more we can
trim on that side of things.

We've done LTS releases before with i.e., the Python 2.x support series.

I can pretty much guarantee that we will take on a non-zero maintenance
burden if we support an additional Python version. Devs spend a fair bit of
time just "keeping CI green" so that new contributions/fixes can flow in
without confusion/delays, and that is compounded by the backport CI and
separate wheels repo CI.

If you convince Ralf et al. that we should go against the NEP I don't
really mind, but it is going to cost us time that could be spent on other
things.

On Mon, 17 Aug 2020 at 06:59, Bennet Fauber <bennet at umich.edu> wrote:

> Boy, yes, Ilhan, I agree.  Too many different roles in my head!
>
> Just to be clear, I also am not concerned about deprecating older
> NumPy, only the base Python version, and specifically 3.6.  It looks
> like the latest fully released NumPy supports Python 3.6.  In my Linux
> world, we still have a preponderance of RedHat/CentOS 7 installed, and
> it does not easily provide Python 3.7:  python36 is available from
> EPEL.  Our most prevalent Ubuntu is 16.04.  We will probably jump to
> 20.04 next calendar year.
>
> I had not realized that Python itself was going to a 12-month release
> cycle.  That seems to me destructive and predicated on the notion that
> software is disposable.
>
> I think there is a very good reason why many Linux distributions have
> separate distributions for longer term stability and more up-to-date
> features.  Red Hat and Fedora, Ubuntu and LTS, Debian Stable and
> Unstable.  Is that viable with SciPy, NumPy, where only the bug fixes
> get backported, which might reduce the cost of longer maintenance
> (except for the wheels, as noted)?
>
> Thanks for the discussion on this, no matter where it ends up.
>
>
>
> On Sun, Aug 16, 2020 at 7:17 PM Ilhan Polat <ilhanpolat at gmail.com> wrote:
> >
> > I have too many personas in my head arguing about this. And many of them
> are in conflict.
> >
> > 1- I think I had enough arguments online to be known as buying none of
> that scientific stability views. That doesn't mean I don't get their point,
> I really really do, but still. It's in the job description. Software is a
> big part of science just like the papers. (What Matti said)
> > 2- Even a two-year old code is not safe to let it collect dust and
> constantly requires attention and becomes a liability rather than a tool
> (what Bennet said).
> > 3- Matlab breaks things all the time with 0 user consideration. None.
> But admittedly they break in style. We shouldn't be like them (not even
> close) but the point is when it is a commercial product, we don't hear the
> uproar we receive. People silently fix things.
> > 4- Just because we update the python version, a lot of packages stop
> working. This is seriously disheartening. Happens to me all the time
> (protobuf, influxdb etc). And really annoying if one package has not
> released the new version yet.
> > 5- Python is releasing too quick. I know Python is not Fortran
> (thankfully) but this is the other end of the spectrum. With every version
> one or two of us spent at least 2 weeks of intensive "What did they change
> on Windows again?" bug hunting. Hence our platform is not reliable and now
> they want  12 months.  (What Ralf said)
> > 6- There are always new users, downloading Python for the first time and
> it is expected that SciPy stack should work off-the-shelf. Hence we don't
> have the luxury to wait and see (see 4th item).
> > 7- If there are limited resources for software support, then why people
> take the risk and update their stuff is something that has been discussed
> for decades now. And no one seems to converge to a point. (What Matti said)
> > 8-  what Evgeni suggested
> >
> > I'm not sure what to do about this but I am also more worried about the
> Python version than the NumPy version, my limited anectodal evidence around
> me suggests that people update NumPy + SciPy together mainly when they use
> pip with -U (--upgrade) on some package that has SciPy as a dependency. So
> I feel that it is safe to bump.
> >
> >
> >
> >
> >
> > On Mon, Aug 17, 2020 at 12:19 AM Evgeni Burovski <
> evgeny.burovskiy at gmail.com> wrote:
> >>
> >> No objections for bumping the minimum NumPy version.
> >> For bumping the python version.. not quite an objection, more of a
> suggestion to consider holding it back.
> >> I guess there are three kinds of stakeholders here:
> >> - for end users, upgrading the python version is a fairly major hassle.
> Three versions of python now means three years back if they switch to a
> 12-months cadence. Three years of support is definitely on a low side of
> things. (with my PI hat on, I can certainly relate to what Bennet was
> saying, even if our lab has it way easier than others).
> >> - for the scipy devs, what exactly drives us to drop it? For 3.5 it was
> of course the matmul operator, for 3.6 it's f-strings, what's there in 3.7
> which makes it not fun to stay on 3.6?
> >> - for the RM and other packagers: how much of a hassle is it to keep an
> older version? I realize it's more wheels to build for the scipy RM, but
> that's fairly automated now?
> >>
> >>
> >>
> >> Cheers,
> >> Evgeni
> >>
> >>
> >> On Sun, Aug 16, 2020 at 6:22 PM Matti Picus <matti.picus at gmail.com>
> wrote:
> >> >
> >> >
> >> > On 8/16/20 4:23 PM, Bennet Fauber wrote:
> >> > > This post is by way of trying to articulate some concerns from the
> end
> >> > > user side of things.
> >> > >
> >> > > It looks like NumPy 19.1 supports Python 3.6.
> >> > >
> >> > >> This release supports Python 3.6-3.8. Cython >= 0.29.21 needs to
> be used when building
> >> > >> with Python 3.9 for testing purposes.
> >> > > https://numpy.org/devdocs/release/1.19.1-notes.html
> >> > >
> >> > > The section of the release notes for NumPy 1.20.0 does not contain
> the
> >> > > section at the top saying which versions of Python are supported.
> >> > > (https://numpy.org/devdocs/release/1.20.0-notes.html)
> >> > >
> >> > > For Python itself,  I find on their releases, the 3.6.12 schedule
> was
> >> > >
> >> > >      3.6.12 final: 2020-08-14 (expected)
> >> > > https://www.python.org/dev/peps/pep-0494/#id9
> >> > >
> >> > > but does not seem to have made it, and Python 3.6.11 was released
> June
> >> > > 27, 2020.  The Python plans for 3.6.13 and beyond are Security fixes
> >> > > only, as needed, until 2021-12.  That seems, from my user
> perspective,
> >> > > a better date for deprecation and not too far from the just proposed
> >> > > one.
> >> > >
> >> > > Acknowledging the additional burden of 'supporting' older versions
> of
> >> > > Python, still it would seem that matching NumPy is not a bad thing.
> >> > >
> >> > >  From a strictly consumer perspective, where much of the work is
> >> > > getting all of the non-NumPy and non-SciPy functionality to work and
> >> > > be stable, upgrading Python can be very disruptive.  Time spent
> >> > > getting the 'glue' around analytics to work is time we cannot spend
> on
> >> > > science.  There are large projects that do keep up, but they tend
> also
> >> > > to be funded well.  The many small labs on my campus do not have
> >> > > funding for software development, are unlikely to ever get any, so
> any
> >> > > work required to fix software because of updates comes from their
> >> > > budget for science and from their scientists who are not as
> proficient
> >> > > and therefore not as efficient at adapting to a newer version of
> >> > > Python and all the reinstallation of libraries that attends it.
> >> > >
> >> >
> >> > Is there a forum where we could explore the larger question around
> best
> >> > practices in helping projects and teams keep their software up-to-date
> >> > through rolling updates, test suites, and documentation? A few things
> >> > come to mind:
> >> >
> >> > - developing a culture of version control and tests when creating
> >> > software, even if it is grad students racing towards their degrees
> >> >
> >> > - freezing environments in images, which is a whole discipline unto
> >> > itself, and needed for reproducible science
> >> >
> >> > - offloading the task of maintaining computing environments using
> things
> >> > like Jupyter Hub or other web-based services
> >> >
> >> >
> >> > FWIW, python is changing their release cadence from once every 18
> months
> >> > to once every 12 months[1] so it is likely this problem will get
> harder.
> >> >
> >> > Matti
> >> >
> >> > [1] https://www.python.org/dev/peps/pep-0602/#annual-release-cadence
> >> > _______________________________________________
> >> > SciPy-Dev mailing list
> >> > SciPy-Dev at python.org
> >> > https://mail.python.org/mailman/listinfo/scipy-dev
> >> _______________________________________________
> >> SciPy-Dev mailing list
> >> SciPy-Dev at python.org
> >> https://mail.python.org/mailman/listinfo/scipy-dev
> >
> > _______________________________________________
> > SciPy-Dev mailing list
> > SciPy-Dev at python.org
> > https://mail.python.org/mailman/listinfo/scipy-dev
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200817/59b4f3e6/attachment-0001.html>

From bennet at umich.edu  Tue Aug 18 07:43:45 2020
From: bennet at umich.edu (Bennet Fauber)
Date: Tue, 18 Aug 2020 07:43:45 -0400
Subject: [SciPy-Dev] proposal to drop Python 3.6 and NumPy 1.14 / 1.15
In-Reply-To: <CAHPuU_aE5xRPnYspa0rBYwq0zTyOVCAqCFO+rkH=KWvqXqtC5Q@mail.gmail.com>
References: <CABL7CQg_MYscpnnnCcEW+rCzoCCRJvj82jWX_4LmSOSqjgixbQ@mail.gmail.com>
 <CAB2ovovq144TYAzdMLwko7uHKJqnzhedKMAcM-hNqzVWXSjWww@mail.gmail.com>
 <cef560fb-bee3-ab6c-c035-2e89020820cc@gmail.com>
 <CAMRo0isPvbSX2bspzpxvW795+i2Vs+-3hZr-8f4bcaGHaX+_xA@mail.gmail.com>
 <CAEBuzr8BT-CPVYcQjdTt6ARKGcQUPfOGe8Nv5qH53By26aFvgw@mail.gmail.com>
 <CAB2ovouOuBDXK9oNCHCgarzsS7i=dSES+wbPT-1=5sUDCA7Cfw@mail.gmail.com>
 <CAHPuU_aE5xRPnYspa0rBYwq0zTyOVCAqCFO+rkH=KWvqXqtC5Q@mail.gmail.com>
Message-ID: <CAB2ovotoWDQBWtYM3t9eRS2JA9KpvpZ1UOcMNAPRNRTpkUxk1Q@mail.gmail.com>

Thanks for the report 'from the trenches', Tyler.

It's been thouroughly discussed, which is all I could ask.  I
appreciate the time everyone has taken.

This gives us some food for thought on our side, as well.  Time to
start that with my colleagues here.

Thanks, again, for the great software.




On Mon, Aug 17, 2020 at 10:58 PM Tyler Reddy <tyler.je.reddy at gmail.com> wrote:
>
> FWIW, the policy for all minor point releases is already that we only backport bug fixes and not enhancements, so there's not much more we can trim on that side of things.
>
> We've done LTS releases before with i.e., the Python 2.x support series.
>
> I can pretty much guarantee that we will take on a non-zero maintenance burden if we support an additional Python version. Devs spend a fair bit of time just "keeping CI green" so that new contributions/fixes can flow in without confusion/delays, and that is compounded by the backport CI and separate wheels repo CI.
>
> If you convince Ralf et al. that we should go against the NEP I don't really mind, but it is going to cost us time that could be spent on other things.
>
> On Mon, 17 Aug 2020 at 06:59, Bennet Fauber <bennet at umich.edu> wrote:
>>
>> Boy, yes, Ilhan, I agree.  Too many different roles in my head!
>>
>> Just to be clear, I also am not concerned about deprecating older
>> NumPy, only the base Python version, and specifically 3.6.  It looks
>> like the latest fully released NumPy supports Python 3.6.  In my Linux
>> world, we still have a preponderance of RedHat/CentOS 7 installed, and
>> it does not easily provide Python 3.7:  python36 is available from
>> EPEL.  Our most prevalent Ubuntu is 16.04.  We will probably jump to
>> 20.04 next calendar year.
>>
>> I had not realized that Python itself was going to a 12-month release
>> cycle.  That seems to me destructive and predicated on the notion that
>> software is disposable.
>>
>> I think there is a very good reason why many Linux distributions have
>> separate distributions for longer term stability and more up-to-date
>> features.  Red Hat and Fedora, Ubuntu and LTS, Debian Stable and
>> Unstable.  Is that viable with SciPy, NumPy, where only the bug fixes
>> get backported, which might reduce the cost of longer maintenance
>> (except for the wheels, as noted)?
>>
>> Thanks for the discussion on this, no matter where it ends up.
>>
>>
>>
>> On Sun, Aug 16, 2020 at 7:17 PM Ilhan Polat <ilhanpolat at gmail.com> wrote:
>> >
>> > I have too many personas in my head arguing about this. And many of them are in conflict.
>> >
>> > 1- I think I had enough arguments online to be known as buying none of that scientific stability views. That doesn't mean I don't get their point, I really really do, but still. It's in the job description. Software is a big part of science just like the papers. (What Matti said)
>> > 2- Even a two-year old code is not safe to let it collect dust and constantly requires attention and becomes a liability rather than a tool (what Bennet said).
>> > 3- Matlab breaks things all the time with 0 user consideration. None. But admittedly they break in style. We shouldn't be like them (not even close) but the point is when it is a commercial product, we don't hear the uproar we receive. People silently fix things.
>> > 4- Just because we update the python version, a lot of packages stop working. This is seriously disheartening. Happens to me all the time (protobuf, influxdb etc). And really annoying if one package has not released the new version yet.
>> > 5- Python is releasing too quick. I know Python is not Fortran (thankfully) but this is the other end of the spectrum. With every version one or two of us spent at least 2 weeks of intensive "What did they change on Windows again?" bug hunting. Hence our platform is not reliable and now they want  12 months.  (What Ralf said)
>> > 6- There are always new users, downloading Python for the first time and it is expected that SciPy stack should work off-the-shelf. Hence we don't have the luxury to wait and see (see 4th item).
>> > 7- If there are limited resources for software support, then why people take the risk and update their stuff is something that has been discussed for decades now. And no one seems to converge to a point. (What Matti said)
>> > 8-  what Evgeni suggested
>> >
>> > I'm not sure what to do about this but I am also more worried about the Python version than the NumPy version, my limited anectodal evidence around me suggests that people update NumPy + SciPy together mainly when they use pip with -U (--upgrade) on some package that has SciPy as a dependency. So I feel that it is safe to bump.
>> >
>> >
>> >
>> >
>> >
>> > On Mon, Aug 17, 2020 at 12:19 AM Evgeni Burovski <evgeny.burovskiy at gmail.com> wrote:
>> >>
>> >> No objections for bumping the minimum NumPy version.
>> >> For bumping the python version.. not quite an objection, more of a suggestion to consider holding it back.
>> >> I guess there are three kinds of stakeholders here:
>> >> - for end users, upgrading the python version is a fairly major hassle. Three versions of python now means three years back if they switch to a 12-months cadence. Three years of support is definitely on a low side of things. (with my PI hat on, I can certainly relate to what Bennet was saying, even if our lab has it way easier than others).
>> >> - for the scipy devs, what exactly drives us to drop it? For 3.5 it was of course the matmul operator, for 3.6 it's f-strings, what's there in 3.7 which makes it not fun to stay on 3.6?
>> >> - for the RM and other packagers: how much of a hassle is it to keep an older version? I realize it's more wheels to build for the scipy RM, but that's fairly automated now?
>> >>
>> >>
>> >>
>> >> Cheers,
>> >> Evgeni
>> >>
>> >>
>> >> On Sun, Aug 16, 2020 at 6:22 PM Matti Picus <matti.picus at gmail.com> wrote:
>> >> >
>> >> >
>> >> > On 8/16/20 4:23 PM, Bennet Fauber wrote:
>> >> > > This post is by way of trying to articulate some concerns from the end
>> >> > > user side of things.
>> >> > >
>> >> > > It looks like NumPy 19.1 supports Python 3.6.
>> >> > >
>> >> > >> This release supports Python 3.6-3.8. Cython >= 0.29.21 needs to be used when building
>> >> > >> with Python 3.9 for testing purposes.
>> >> > > https://numpy.org/devdocs/release/1.19.1-notes.html
>> >> > >
>> >> > > The section of the release notes for NumPy 1.20.0 does not contain the
>> >> > > section at the top saying which versions of Python are supported.
>> >> > > (https://numpy.org/devdocs/release/1.20.0-notes.html)
>> >> > >
>> >> > > For Python itself,  I find on their releases, the 3.6.12 schedule was
>> >> > >
>> >> > >      3.6.12 final: 2020-08-14 (expected)
>> >> > > https://www.python.org/dev/peps/pep-0494/#id9
>> >> > >
>> >> > > but does not seem to have made it, and Python 3.6.11 was released June
>> >> > > 27, 2020.  The Python plans for 3.6.13 and beyond are Security fixes
>> >> > > only, as needed, until 2021-12.  That seems, from my user perspective,
>> >> > > a better date for deprecation and not too far from the just proposed
>> >> > > one.
>> >> > >
>> >> > > Acknowledging the additional burden of 'supporting' older versions of
>> >> > > Python, still it would seem that matching NumPy is not a bad thing.
>> >> > >
>> >> > >  From a strictly consumer perspective, where much of the work is
>> >> > > getting all of the non-NumPy and non-SciPy functionality to work and
>> >> > > be stable, upgrading Python can be very disruptive.  Time spent
>> >> > > getting the 'glue' around analytics to work is time we cannot spend on
>> >> > > science.  There are large projects that do keep up, but they tend also
>> >> > > to be funded well.  The many small labs on my campus do not have
>> >> > > funding for software development, are unlikely to ever get any, so any
>> >> > > work required to fix software because of updates comes from their
>> >> > > budget for science and from their scientists who are not as proficient
>> >> > > and therefore not as efficient at adapting to a newer version of
>> >> > > Python and all the reinstallation of libraries that attends it.
>> >> > >
>> >> >
>> >> > Is there a forum where we could explore the larger question around best
>> >> > practices in helping projects and teams keep their software up-to-date
>> >> > through rolling updates, test suites, and documentation? A few things
>> >> > come to mind:
>> >> >
>> >> > - developing a culture of version control and tests when creating
>> >> > software, even if it is grad students racing towards their degrees
>> >> >
>> >> > - freezing environments in images, which is a whole discipline unto
>> >> > itself, and needed for reproducible science
>> >> >
>> >> > - offloading the task of maintaining computing environments using things
>> >> > like Jupyter Hub or other web-based services
>> >> >
>> >> >
>> >> > FWIW, python is changing their release cadence from once every 18 months
>> >> > to once every 12 months[1] so it is likely this problem will get harder.
>> >> >
>> >> > Matti
>> >> >
>> >> > [1] https://www.python.org/dev/peps/pep-0602/#annual-release-cadence
>> >> > _______________________________________________
>> >> > SciPy-Dev mailing list
>> >> > SciPy-Dev at python.org
>> >> > https://mail.python.org/mailman/listinfo/scipy-dev
>> >> _______________________________________________
>> >> SciPy-Dev mailing list
>> >> SciPy-Dev at python.org
>> >> https://mail.python.org/mailman/listinfo/scipy-dev
>> >
>> > _______________________________________________
>> > SciPy-Dev mailing list
>> > SciPy-Dev at python.org
>> > https://mail.python.org/mailman/listinfo/scipy-dev
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev

From ralf.gommers at gmail.com  Tue Aug 18 08:06:07 2020
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Tue, 18 Aug 2020 13:06:07 +0100
Subject: [SciPy-Dev] proposal to drop Python 3.6 and NumPy 1.14 / 1.15
In-Reply-To: <CAHPuU_aE5xRPnYspa0rBYwq0zTyOVCAqCFO+rkH=KWvqXqtC5Q@mail.gmail.com>
References: <CABL7CQg_MYscpnnnCcEW+rCzoCCRJvj82jWX_4LmSOSqjgixbQ@mail.gmail.com>
 <CAB2ovovq144TYAzdMLwko7uHKJqnzhedKMAcM-hNqzVWXSjWww@mail.gmail.com>
 <cef560fb-bee3-ab6c-c035-2e89020820cc@gmail.com>
 <CAMRo0isPvbSX2bspzpxvW795+i2Vs+-3hZr-8f4bcaGHaX+_xA@mail.gmail.com>
 <CAEBuzr8BT-CPVYcQjdTt6ARKGcQUPfOGe8Nv5qH53By26aFvgw@mail.gmail.com>
 <CAB2ovouOuBDXK9oNCHCgarzsS7i=dSES+wbPT-1=5sUDCA7Cfw@mail.gmail.com>
 <CAHPuU_aE5xRPnYspa0rBYwq0zTyOVCAqCFO+rkH=KWvqXqtC5Q@mail.gmail.com>
Message-ID: <CABL7CQjDEdTRpDy1CP2vohBXCb7khtMSy2kVbVbSonvdEfE8DQ@mail.gmail.com>

On Tue, Aug 18, 2020 at 3:58 AM Tyler Reddy <tyler.je.reddy at gmail.com>
wrote:

>
> I can pretty much guarantee that we will take on a non-zero maintenance
> burden if we support an additional Python version. Devs spend a fair bit of
> time just "keeping CI green" so that new contributions/fixes can flow in
> without confusion/delays, and that is compounded by the backport CI and
> separate wheels repo CI.
>

Yep, this really is the main argument.


> If you convince Ralf et al. that we should go against the NEP I don't
> really mind, but it is going to cost us time that could be spent on other
> things.
>
> On Mon, 17 Aug 2020 at 06:59, Bennet Fauber <bennet at umich.edu> wrote:
>
>> Boy, yes, Ilhan, I agree.  Too many different roles in my head!
>>
>> Just to be clear, I also am not concerned about deprecating older
>> NumPy, only the base Python version, and specifically 3.6.  It looks
>> like the latest fully released NumPy supports Python 3.6.  In my Linux
>> world, we still have a preponderance of RedHat/CentOS 7 installed, and
>> it does not easily provide Python 3.7:  python36 is available from
>> EPEL.  Our most prevalent Ubuntu is 16.04.  We will probably jump to
>> 20.04 next calendar year.
>>
>
`pip install scipy --user`, or whatever you do on an old CentOS
machine/cluster, is still going to work. As is `apt-get install scipy`. And
it will get you the latest supported SciPy for that Python version, which
is pretty good. All you're missing out on is a few new features and bug
fixes in 1.6.0. If you discover you need those, it's time to learn Conda so
it's trivial to install a newer Python version.


>> I had not realized that Python itself was going to a 12-month release
>> cycle.  That seems to me destructive and predicated on the notion that
>> software is disposable.
>>
>
I have to admit I'm not a fan, but that's because new Python versions
typically bring us new burdens and few or no new features we need. Having
bug fixes propagate a bit faster is the upside.


>> I think there is a very good reason why many Linux distributions have
>> separate distributions for longer term stability and more up-to-date
>> features.  Red Hat and Fedora, Ubuntu and LTS, Debian Stable and
>> Unstable.  Is that viable with SciPy, NumPy, where only the bug fixes
>> get backported, which might reduce the cost of longer maintenance
>> (except for the wheels, as noted)?
>>
>> Thanks for the discussion on this, no matter where it ends up.
>>
>> >
>> > On Mon, Aug 17, 2020 at 12:19 AM Evgeni Burovski <
>> evgeny.burovskiy at gmail.com> wrote:
>> >>
>> >> No objections for bumping the minimum NumPy version.
>> >> For bumping the python version.. not quite an objection, more of a
>> suggestion to consider holding it back.
>> >> I guess there are three kinds of stakeholders here:
>> >> - for end users, upgrading the python version is a fairly major
>> hassle. Three versions of python now means three years back if they switch
>> to a 12-months cadence. Three years of support is definitely on a low side
>> of things. (with my PI hat on, I can certainly relate to what Bennet was
>> saying, even if our lab has it way easier than others).
>>
>
Python 3.6 was released in Dec 2016, and we're talking about our Dec 2020
release - so 4 years. When it's actually down to 3 years we can review
again, but that's not today's decision.

>> - for the scipy devs, what exactly drives us to drop it? For 3.5 it was
>> of course the matmul operator, for 3.6 it's f-strings, what's there in 3.7
>> which makes it not fun to stay on 3.6?
>>
>
f-strings are/were not very interesting imho. If you spend all day doing
string processing yes, but for numerical code it really doesn't matter all
that much. We had two ways of doing string formatting, now we have three.

As Tyler said, it's to reduce the cost of keeping it working, and CI happy.
We can spend that time writing or reviewing new features and other
improvements instead.

Cheers,
Ralf


>> - for the RM and other packagers: how much of a hassle is it to keep an
>> older version? I realize it's more wheels to build for the scipy RM, but
>> that's fairly automated now?
>>
> >>
>> >>
>> >>
>> >> Cheers,
>> >> Evgeni
>> >>
>> >>
>> >> On Sun, Aug 16, 2020 at 6:22 PM Matti Picus <matti.picus at gmail.com>
>> wrote:
>> >> >
>> >> >
>> >> > On 8/16/20 4:23 PM, Bennet Fauber wrote:
>> >> > > This post is by way of trying to articulate some concerns from the
>> end
>> >> > > user side of things.
>> >> > >
>> >> > > It looks like NumPy 19.1 supports Python 3.6.
>> >> > >
>> >> > >> This release supports Python 3.6-3.8. Cython >= 0.29.21 needs to
>> be used when building
>> >> > >> with Python 3.9 for testing purposes.
>> >> > > https://numpy.org/devdocs/release/1.19.1-notes.html
>> >> > >
>> >> > > The section of the release notes for NumPy 1.20.0 does not contain
>> the
>> >> > > section at the top saying which versions of Python are supported.
>> >> > > (https://numpy.org/devdocs/release/1.20.0-notes.html)
>> >> > >
>> >> > > For Python itself,  I find on their releases, the 3.6.12 schedule
>> was
>> >> > >
>> >> > >      3.6.12 final: 2020-08-14 (expected)
>> >> > > https://www.python.org/dev/peps/pep-0494/#id9
>> >> > >
>> >> > > but does not seem to have made it, and Python 3.6.11 was released
>> June
>> >> > > 27, 2020.  The Python plans for 3.6.13 and beyond are Security
>> fixes
>> >> > > only, as needed, until 2021-12.  That seems, from my user
>> perspective,
>> >> > > a better date for deprecation and not too far from the just
>> proposed
>> >> > > one.
>> >> > >
>> >> > > Acknowledging the additional burden of 'supporting' older versions
>> of
>> >> > > Python, still it would seem that matching NumPy is not a bad thing.
>> >> > >
>> >> > >  From a strictly consumer perspective, where much of the work is
>> >> > > getting all of the non-NumPy and non-SciPy functionality to work
>> and
>> >> > > be stable, upgrading Python can be very disruptive.  Time spent
>> >> > > getting the 'glue' around analytics to work is time we cannot
>> spend on
>> >> > > science.  There are large projects that do keep up, but they tend
>> also
>> >> > > to be funded well.  The many small labs on my campus do not have
>> >> > > funding for software development, are unlikely to ever get any, so
>> any
>> >> > > work required to fix software because of updates comes from their
>> >> > > budget for science and from their scientists who are not as
>> proficient
>> >> > > and therefore not as efficient at adapting to a newer version of
>> >> > > Python and all the reinstallation of libraries that attends it.
>> >> > >
>> >> >
>> >> > Is there a forum where we could explore the larger question around
>> best
>> >> > practices in helping projects and teams keep their software
>> up-to-date
>> >> > through rolling updates, test suites, and documentation? A few things
>> >> > come to mind:
>> >> >
>> >> > - developing a culture of version control and tests when creating
>> >> > software, even if it is grad students racing towards their degrees
>> >> >
>> >> > - freezing environments in images, which is a whole discipline unto
>> >> > itself, and needed for reproducible science
>> >> >
>> >> > - offloading the task of maintaining computing environments using
>> things
>> >> > like Jupyter Hub or other web-based services
>> >> >
>> >> >
>> >> > FWIW, python is changing their release cadence from once every 18
>> months
>> >> > to once every 12 months[1] so it is likely this problem will get
>> harder.
>> >> >
>> >> > Matti
>> >> >
>> >> > [1] https://www.python.org/dev/peps/pep-0602/#annual-release-cadence
>> >> > _______________________________________________
>> >> > SciPy-Dev mailing list
>> >> > SciPy-Dev at python.org
>> >> > https://mail.python.org/mailman/listinfo/scipy-dev
>> >> _______________________________________________
>> >> SciPy-Dev mailing list
>> >> SciPy-Dev at python.org
>> >> https://mail.python.org/mailman/listinfo/scipy-dev
>> >
>> > _______________________________________________
>> > SciPy-Dev mailing list
>> > SciPy-Dev at python.org
>> > https://mail.python.org/mailman/listinfo/scipy-dev
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200818/d8dc14c2/attachment-0001.html>

From ralf.gommers at gmail.com  Wed Aug 19 17:26:31 2020
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Wed, 19 Aug 2020 22:26:31 +0100
Subject: [SciPy-Dev] Adding non-parametric methods to scipy.stats
In-Reply-To: <64991457-41eb-d2da-b32b-17c35c0726e4@tik.ee.ethz.ch>
References: <6a5dee0d-3d55-409d-6bf8-b86e3f465275@tik.ee.ethz.ch>
 <CADuxUiz45SJEZm5Mm6V6cC9Z0vMUxJt9D5DD_Shm=PeKrC=MJg@mail.gmail.com>
 <9ed88efe-5504-3d60-ab73-3ffc7cce1edf@tik.ee.ethz.ch>
 <CADuxUiym12uh4bbg1W_m6Lj6xb2LJ4Eb+wyJok7EyWmrDv8enQ@mail.gmail.com>
 <cedc67bf-6dcf-4267-f7c8-84d23200f45c@ethz.ch>
 <CADuxUiy+GE91fwcBtVEWbvn2MMDkiJO5BPd9w5z2aTbqQz52pw@mail.gmail.com>
 <CAMMTP+A43zKdBNHBmqz4ij=80BZrci6tNk4Adr9FEBMp9uwMNQ@mail.gmail.com>
 <CAGzF1ucda-ucx+9n=H58OrA2Rd7uVwf-UpxeAqZNNwvegvB0Pw@mail.gmail.com>
 <a54e89f1-2a6f-ced2-e2a8-7391a76937e0@tik.ee.ethz.ch>
 <CAMMTP+AMX1qAiw5B9pQrWGnNRqNXLvLHUPaBnD92q0j-M-eR+w@mail.gmail.com>
 <CAMMTP+CFXhwPLO-GFPUftDPYvWUvo2xw1C2f5CJu=3ZEz0WqQg@mail.gmail.com>
 <3893d209-892f-b738-07ba-3a03aa4b502a@tik.ee.ethz.ch>
 <64991457-41eb-d2da-b32b-17c35c0726e4@tik.ee.ethz.ch>
Message-ID: <CABL7CQhzv4h+j_J+oKixxZAKb9J8O4bsXzrQ8LDb=M4x5+GAfw@mail.gmail.com>

On Mon, Aug 17, 2020 at 9:44 AM Romain Jacob <jacobr at ethz.ch> wrote:

> Hello everyone,
>
> I've submitted the PR adding support for non-parametric confidence
> intervals for quantiles (https://github.com/scipy/scipy/pull/12680).
> There has been quite some comments made already, which I fixed
> appropriately I believe
>
> Will be happy to get some more feedback or see the PR merged :-)
>
> Note: the last commit has a CI failing apparently due to a file change in
> `scipy/sparse/linalg/` which is completely unrelated. I'm not sure how to
> go about this... ?
>
If it's clearly unrelated, you can just ignore it. Or add a comment "the
only CI failure is in sparse.linalg and unrelated to this PR". Then the
reviewer can just go ahead and merge if everything else looks good - CI
doesn't have to be green.

Cheers,
Ralf

Cheers,
> --
> Romain
>
> On 15/06/2020 08:27, Romain Jacob wrote:
>
> On 13/06/2020 20:54, josef.pktd at gmail.com wrote:
>
> On Fri, Jun 12, 2020 at 11:29 AM <josef.pktd at gmail.com> wrote:
>
>> On Fri, Jun 12, 2020 at 1:58 AM Romain Jacob <jacobr at ethz.ch> wrote:
>>
>>> On 11/06/2020 20:54, Warren Weckesser wrote:
>>>
>>> On 6/11/20, josef.pktd at gmail.com <josef.pktd at gmail.com> <josef.pktd at gmail.com> wrote:
>>>
>>> I think it would make a good and useful addition and fit into scipy.stats.
>>> There are no pure confint functions yet, AFAIR.
>>>
>>> I agree with Josef and Matt, this looks like it would be a nice
>>> addition to SciPy.  At the moment, I'm not sure what the API should
>>> look like.  Romain, is the work that you've already done available
>>> online somewhere?
>>>
>>> Warren
>>>
>>> Yes, I have some functional implementation available here:
>>> https://github.com/TriScale-Anon/triscale/blob/master/helpers.py#L397
>>>
>>
>> An implementation detail:
>> binom has cdf and ppf functions
>> My guess, not verified, is that we can just use binom.interval
>>
>> (at least I used those for similar cases)
>>
>
> I found my version again
>
> https://github.com/statsmodels/statsmodels/issues/6562#issuecomment-592769480
>
>
> I guess that's the same for two sided confint as the references.
> It doesn't have interpolation if that could be applied in this case.
>
> I don't entirely follow what you mean here: that the building of the
> probabilities in these two lines(
> https://github.com/TriScale-Anon/triscale/blob/master/helpers.py#L438 and
> L439) can be built directly form binom without np.cumsum? That definitely
> correct (I actually have code also doing that somewhere).
>
> I did not know about the `interval` method. That's sound interesting
> indeed, but it's not 100% clear to me how the uniqueness problem is
> handled. I looked for the implementation of the method but couldn't find it
> in `binom`... I'm looking in the wrong place?
>
> Cheers,
> --
> Romain
>
>
> This will eventually end up in statsmodels, but I don't know yet where.
> That's not a reason not to add it to scipy.stats.
>
> Josef
>
>
>> Josef
>>
>>
>>> There is quite some work to be done on formatting and documentation to
>>> comply with the SciPy standards, but functionally it's already there (and
>>> as you'll see, the method is quite simple).
>>>
>>> Cheers,
>>> --
>>> Romain
>>>
>>> I recently wrote a function for the confidence interval for the median,
>>> mainly because I ran into the formulas that were easy to code.
>>> related open issue: how do we get confidence intervals for QQ-plot.
>>>
>>> aside: I don't like "percent", I prefer quantiles in [0, 1]. See discussion
>>> a while ago in numpy.
>>>
>>> Josef
>>>
>>>
>>> On Thu, Jun 11, 2020 at 1:01 PM Matt Haberland <mhaberla at calpoly.edu> <mhaberla at calpoly.edu>
>>> wrote:
>>>
>>>
>>> OK, we should let our statistics experts weigh in on this. (I'm not
>>> actually one of them.)
>>>
>>> On Wed, Jun 10, 2020 at 10:46 PM Romain Jacob <jacobr at ethz.ch> <jacobr at ethz.ch> wrote:
>>>
>>>
>>> I think a dedicated function makes more sense. This function takes as
>>> input an array, a percentile and a confidence level, and returns the
>>> corresponding one-sided confidence intervals.
>>>
>>> I quickly looked at the list of existing functions in scipy.stats but
>>> did
>>> not see any function in "summary statistics" that does similar things. So
>>> I
>>> would go for a new function.
>>> On 10/06/2020 20:38, Matt Haberland wrote:
>>>
>>> Where do you envision this living in SciPy? In its own function, or
>>> added
>>> functionality to other functions e.g. scipy.stats.percentileofscore<https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.percentileofscore.html#scipy.stats.percentileofscore> <https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.percentileofscore.html#scipy.stats.percentileofscore>
>>> ?
>>>
>>> On Tue, Jun 9, 2020 at 11:12 PM Romain Jacob <jacobr at ethz.ch> <jacobr at ethz.ch> wrote:
>>>
>>>
>>> On 09/06/2020 20:18, Matt Haberland wrote:
>>>
>>> Yes, I think we would be interested in confidence intervals, but I
>>> think
>>> the algorithm should be very well standard/cited, even if it's not the
>>> best/most modern.
>>>
>>> Yes definitely! We did not invented the method I am referring to, it a
>>> long-known approach (first proposed by Thompson in 1936 [1], extended
>>> later
>>> and commonly found in textbooks, eg [2,3]). This method is very simple,
>>> quite powerful, yet it has been largely overlooked in many scientific
>>> fields. I found no available implementation to facilitate its use (at
>>> least
>>> not in Python, there may be something in R, I have not looked).
>>>
>>> [1] https://www.jstor.org/stable/2957563
>>> [2] doi.org/10.1002/0471722162.ch7
>>> [3] https://perfeval.epfl.ch/
>>>
>>> @WarrenWeckesser and I had planned to work on confidence intervals for
>>> the test statistics returned by our statistical tests<https://docs.scipy.org/doc/scipy/reference/stats.html#statistical-tests> <https://docs.scipy.org/doc/scipy/reference/stats.html#statistical-tests>.
>>>
>>>
>>> That is also definitely interesting, although I am not myself an expert
>>> in that area. I am glad to see that the complete list contains some
>>> non-parametric tests :-)
>>>
>>> Cheers,
>>> --
>>> Romain
>>>
>>>
>>> On Mon, Jun 8, 2020 at 2:11 AM Romain Jacob <jacobr at ethz.ch> <jacobr at ethz.ch> wrote:
>>>
>>>
>>> Hello everyone,
>>>
>>> I have been working for some time on the implementation of
>>> non-parametric methods to compute confidence intervals for
>>> percentiles.
>>> There are some very interesting results in the literature (see e.g. a
>>> nice
>>> pitch in [1]) which I think it would be great to add to SciPy to make
>>> them
>>> more readily available. It also seems to be rather in line with
>>> "recent"
>>> discussions of the roadmap for scipy.stats [2].
>>>
>>> I would be interested in contributing this. What do you think?
>>>
>>> Cheers,
>>> --
>>> Romain
>>>
>>> [1] https://ieeexplore.ieee.org/document/6841797
>>> [2] https://github.com/scipy/scipy/issues/10577
>>> --
>>> Romain Jacob
>>> Postdoctoral Researcher
>>> ETH Zurich - Computer Engineering and Networks Laboratorywww.romainjacob.net
>>> @RJacobPartner <https://twitter.com/RJacobPartner> <https://twitter.com/RJacobPartner>
>>> Gloriastrasse 35, ETZ G75
>>> 8092 Zurich
>>> +41 7 68 16 88 22
>>> _______________________________________________
>>> SciPy-Dev mailing listSciPy-Dev at python.orghttps://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>> --
>>> Matt Haberland
>>> Assistant Professor
>>> BioResource and Agricultural Engineering
>>> 08A-3K, Cal Poly
>>>
>>> _______________________________________________
>>> SciPy-Dev mailinglistSciPy-Dev at python.orghttps://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>> --
>>> Romain Jacob
>>> Postdoctoral Researcher
>>> ETH Zurich - Computer Engineering and Networks Laboratorywww.romainjacob.net
>>> @RJacobPartner <https://twitter.com/RJacobPartner> <https://twitter.com/RJacobPartner>
>>> Gloriastrasse 35, ETZ G75
>>> 8092 Zurich
>>> +41 7 68 16 88 22
>>> _______________________________________________
>>> SciPy-Dev mailing listSciPy-Dev at python.orghttps://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>> --
>>> Matt Haberland
>>> Assistant Professor
>>> BioResource and Agricultural Engineering
>>> 08A-3K, Cal Poly
>>>
>>> _______________________________________________
>>> SciPy-Dev mailinglistSciPy-Dev at python.orghttps://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>> _______________________________________________
>>> SciPy-Dev mailing listSciPy-Dev at python.orghttps://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>> --
>>> Matt Haberland
>>> Assistant Professor
>>> BioResource and Agricultural Engineering
>>> 08A-3K, Cal Poly
>>> _______________________________________________
>>> SciPy-Dev mailing listSciPy-Dev at python.orghttps://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>> _______________________________________________
>>> SciPy-Dev mailing listSciPy-Dev at python.orghttps://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>> --
>>> Romain Jacob
>>> Postdoctoral Researcher
>>> ETH Zurich - Computer Engineering and Networks Laboratory
>>> www.romainjacob.net
>>> @RJacobPartner <https://twitter.com/RJacobPartner>
>>> Gloriastrasse 35, ETZ G75
>>> 8092 Zurich
>>> +41 7 68 16 88 22
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>
> _______________________________________________
> SciPy-Dev mailing listSciPy-Dev at python.orghttps://mail.python.org/mailman/listinfo/scipy-dev
>
> --
> Romain Jacob
> Postdoctoral Researcher
> ETH Zurich - Computer Engineering and Networks Laboratory
> www.romainjacob.net
> @RJacobPartner <https://twitter.com/RJacobPartner>
> Gloriastrasse 35, ETZ G75
> 8092 Zurich
> +41 7 68 16 88 22
>
> _______________________________________________
> SciPy-Dev mailing listSciPy-Dev at python.orghttps://mail.python.org/mailman/listinfo/scipy-dev
>
> --
> Romain Jacob
> Postdoctoral Researcher
> ETH Zurich - Computer Engineering and Networks Laboratory
> www.romainjacob.net
> @RJacobPartner <https://twitter.com/RJacobPartner>
> Gloriastrasse 35, ETZ G75
> 8092 Zurich
> +41 7 68 16 88 22
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200819/74caece9/attachment-0001.html>

From charlesr.harris at gmail.com  Thu Aug 20 09:30:41 2020
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 20 Aug 2020 07:30:41 -0600
Subject: [SciPy-Dev] proposal to drop Python 3.6 and NumPy 1.14 / 1.15
In-Reply-To: <CAEBuzr8BT-CPVYcQjdTt6ARKGcQUPfOGe8Nv5qH53By26aFvgw@mail.gmail.com>
References: <CABL7CQg_MYscpnnnCcEW+rCzoCCRJvj82jWX_4LmSOSqjgixbQ@mail.gmail.com>
 <CAB2ovovq144TYAzdMLwko7uHKJqnzhedKMAcM-hNqzVWXSjWww@mail.gmail.com>
 <cef560fb-bee3-ab6c-c035-2e89020820cc@gmail.com>
 <CAMRo0isPvbSX2bspzpxvW795+i2Vs+-3hZr-8f4bcaGHaX+_xA@mail.gmail.com>
 <CAEBuzr8BT-CPVYcQjdTt6ARKGcQUPfOGe8Nv5qH53By26aFvgw@mail.gmail.com>
Message-ID: <CAB6mnxJLgP_3YhO_6Ses8omXoNu0z198kRNu5VJJeZwww=6biw@mail.gmail.com>

On Sun, Aug 16, 2020 at 5:17 PM Ilhan Polat <ilhanpolat at gmail.com> wrote:

> I have too many personas in my head arguing about this. And many of them
> are in conflict.
>
> 1- I think I had enough arguments online to be known as buying none of
> that scientific stability views. That doesn't mean I don't get their point,
> I really really do, but still. It's in the job description. Software is a
> big part of science just like the papers. (What Matti said)
> 2- Even a two-year old code is not safe to let it collect dust and
> constantly requires attention and becomes a liability rather than a tool
> (what Bennet said).
> 3- Matlab breaks things all the time with 0 user consideration. None. But
> admittedly they break in style. We shouldn't be like them (not even close)
> but the point is when it is a commercial product, we don't hear the uproar
> we receive. People silently fix things.
> 4- Just because we update the python version, a lot of packages stop
> working. This is seriously disheartening. Happens to me all the time
> (protobuf, influxdb etc). And really annoying if one package has not
> released the new version yet.
> 5- Python is releasing too quick. I know Python is not Fortran
> (thankfully) but this is the other end of the spectrum. With every version
> one or two of us spent at least 2 weeks of intensive "What did they change
> on Windows again?" bug hunting. Hence our platform is not reliable and now
> they want  12 months.  (What Ralf said)
> 6- There are always new users, downloading Python for the first time and
> it is expected that SciPy stack should work off-the-shelf. Hence we don't
> have the luxury to wait and see (see 4th item).
> 7- If there are limited resources for software support, then why people
> take the risk and update their stuff is something that has been discussed
> for decades now. And no one seems to converge to a point. (What Matti said)
> 8-  what Evgeni suggested
>
> I'm not sure what to do about this but I am also more worried about the
> Python version than the NumPy version, my limited anectodal evidence around
> me suggests that people update NumPy + SciPy together mainly when they use
> pip with -U (--upgrade) on some package that has SciPy as a dependency. So
> I feel that it is safe to bump.
>
>
Just for the NumPy perspective, there is nothing in the forthcoming NumPy
1.20 that doesn't work with Python 3.6, it is just a question of what
wheels to provide. OTOH, as Python advances I don't want to worry about
someone using newer features, whatever they are.  I go back and forth, but
feel that 48 months is about the right support window and that argues for
dropping Python 3.6. With the faster pace of Python development we will
probably want to support the last four versions going forward. Note that
SciPy doesn't need to support all the NumPy versions that support a
particular version of Python, just the latest one. The main concern doesn't
seem to be compatibility as much as availability.

<snip>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200820/39ac8b15/attachment.html>

From evgeny.burovskiy at gmail.com  Fri Aug 21 07:12:22 2020
From: evgeny.burovskiy at gmail.com (Evgeni Burovski)
Date: Fri, 21 Aug 2020 14:12:22 +0300
Subject: [SciPy-Dev] (no subject)
In-Reply-To: <cef560fb-bee3-ab6c-c035-2e89020820cc@gmail.com>
References: <CABL7CQg_MYscpnnnCcEW+rCzoCCRJvj82jWX_4LmSOSqjgixbQ@mail.gmail.com>
 <CAB2ovovq144TYAzdMLwko7uHKJqnzhedKMAcM-hNqzVWXSjWww@mail.gmail.com>
 <cef560fb-bee3-ab6c-c035-2e89020820cc@gmail.com>
Message-ID: <CAMRo0isS3S=FrYe9cSUM4GaU8159TW3OYTQtgtddjq9N9hzD6Q@mail.gmail.com>

Hi Matti,

(am changing the subject line to separate a subtopic).


> Is there a forum where we could explore the larger question around best
> practices in helping projects and teams keep their software up-to-date
> through rolling updates, test suites, and documentation? A few things
> come to mind:
>
> - developing a culture of version control and tests when creating
> software, even if it is grad students racing towards their degrees
>
> - freezing environments in images, which is a whole discipline unto
> itself, and needed for reproducible science
>
> - offloading the task of maintaining computing environments using things
> like Jupyter Hub or other web-based services


t's a great point, thanks for starting the conversation!

I've asked around (mostly undergrad science students, not comp sci). A
common theme is that
- it'd be great to have some short best practices how-tos would be
helpful, especially for testing, version control and containerization
- there's lots of info, and it's overwhelming (hence a short how-to to
get started)
- the vast majority of learning material on the internet seems
off-base and is difficult to connect to the R&D / research projects
students are dealing with.

Anecdotally, when I was starting (non comp sci background here), one
of the eye openers for me was Nelle V's git tutorial at EuroScipy.

There are several tutorials from the SciPy community (broadly
defined), but they are scattered and not very easy to find (e.g.
Matthew Brett's https://matthew-brett.github.io/pydagogue, I'm sure
there are others)

Just throwing it out there: I wonder if SciPy lecture notes
(https://scipy-lectures.org/) could be a place for these sorts of
tutorials.

Cheers,

Evgeni

From stefanv at berkeley.edu  Fri Aug 21 13:55:23 2020
From: stefanv at berkeley.edu (Stefan van der Walt)
Date: Fri, 21 Aug 2020 10:55:23 -0700
Subject: [SciPy-Dev] (no subject)
In-Reply-To: <CAMRo0isS3S=FrYe9cSUM4GaU8159TW3OYTQtgtddjq9N9hzD6Q@mail.gmail.com>
References: <CABL7CQg_MYscpnnnCcEW+rCzoCCRJvj82jWX_4LmSOSqjgixbQ@mail.gmail.com>
 <CAB2ovovq144TYAzdMLwko7uHKJqnzhedKMAcM-hNqzVWXSjWww@mail.gmail.com>
 <cef560fb-bee3-ab6c-c035-2e89020820cc@gmail.com>
 <CAMRo0isS3S=FrYe9cSUM4GaU8159TW3OYTQtgtddjq9N9hzD6Q@mail.gmail.com>
Message-ID: <3e6ab146-237b-49cc-be10-a272ea43b364@www.fastmail.com>

Hi Evgeni,

On Fri, Aug 21, 2020, at 04:12, Evgeni Burovski wrote:
> There are several tutorials from the SciPy community (broadly
> defined), but they are scattered and not very easy to find (e.g.
> Matthew Brett's https://matthew-brett.github.io/pydagogue, I'm sure
> there are others)
> 
> Just throwing it out there: I wonder if SciPy lecture notes
> (https://scipy-lectures.org/) could be a place for these sorts of
> tutorials.

It's true, these are spread all over the place, and it would be good to reconcile.

We've also started writing contributor manuals for some projects.  See, e.g.,:

https://scikit-image.org/docs/stable/core_developer.html (for core developers)

and

https://skyportal.io/docs/contributing.html

for contributing developers.  And, of course, SciPy's very extensive https://docs.scipy.org/doc/scipy/reference/dev/contributor/contributor_toc.html

Topics handled includes scope and size of PRs, handling merge conflicts, guidance on how to do a good review, etc.

So, if we could figure out which topics are general and relevant to most contributors, we could add those to a central set of documentation that several projects can refer back to (I think that was also the idea behind Matthew Brett's https://github.com/matthew-brett/gitwash).

Best regards,
St?fan

From warren.weckesser at gmail.com  Mon Aug 24 17:47:08 2020
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Mon, 24 Aug 2020 17:47:08 -0400
Subject: [SciPy-Dev] proposal to drop Python 3.6 and NumPy 1.14 / 1.15
In-Reply-To: <CAB6mnxJLgP_3YhO_6Ses8omXoNu0z198kRNu5VJJeZwww=6biw@mail.gmail.com>
References: <CABL7CQg_MYscpnnnCcEW+rCzoCCRJvj82jWX_4LmSOSqjgixbQ@mail.gmail.com>
 <CAB2ovovq144TYAzdMLwko7uHKJqnzhedKMAcM-hNqzVWXSjWww@mail.gmail.com>
 <cef560fb-bee3-ab6c-c035-2e89020820cc@gmail.com>
 <CAMRo0isPvbSX2bspzpxvW795+i2Vs+-3hZr-8f4bcaGHaX+_xA@mail.gmail.com>
 <CAEBuzr8BT-CPVYcQjdTt6ARKGcQUPfOGe8Nv5qH53By26aFvgw@mail.gmail.com>
 <CAB6mnxJLgP_3YhO_6Ses8omXoNu0z198kRNu5VJJeZwww=6biw@mail.gmail.com>
Message-ID: <CAGzF1ucFRtkg4gQK19CtNe12WBmO45otzxAueOH3fa6O785OTw@mail.gmail.com>

Sorry for the top post, but this question doesn't follow immediately
from any previous comment.

Is the conclusion of this thread that with SciPy 1.6 we will
definitely drop Python 3.6 support?  It might be useful to use
dataclasses (https://docs.python.org/3/library/dataclasses.html), and
they are new in Python 3.7.

Warren


On 8/20/20, Charles R Harris <charlesr.harris at gmail.com> wrote:
> On Sun, Aug 16, 2020 at 5:17 PM Ilhan Polat <ilhanpolat at gmail.com> wrote:
>
>> I have too many personas in my head arguing about this. And many of them
>> are in conflict.
>>
>> 1- I think I had enough arguments online to be known as buying none of
>> that scientific stability views. That doesn't mean I don't get their
>> point,
>> I really really do, but still. It's in the job description. Software is a
>> big part of science just like the papers. (What Matti said)
>> 2- Even a two-year old code is not safe to let it collect dust and
>> constantly requires attention and becomes a liability rather than a tool
>> (what Bennet said).
>> 3- Matlab breaks things all the time with 0 user consideration. None. But
>> admittedly they break in style. We shouldn't be like them (not even
>> close)
>> but the point is when it is a commercial product, we don't hear the
>> uproar
>> we receive. People silently fix things.
>> 4- Just because we update the python version, a lot of packages stop
>> working. This is seriously disheartening. Happens to me all the time
>> (protobuf, influxdb etc). And really annoying if one package has not
>> released the new version yet.
>> 5- Python is releasing too quick. I know Python is not Fortran
>> (thankfully) but this is the other end of the spectrum. With every
>> version
>> one or two of us spent at least 2 weeks of intensive "What did they
>> change
>> on Windows again?" bug hunting. Hence our platform is not reliable and
>> now
>> they want  12 months.  (What Ralf said)
>> 6- There are always new users, downloading Python for the first time and
>> it is expected that SciPy stack should work off-the-shelf. Hence we don't
>> have the luxury to wait and see (see 4th item).
>> 7- If there are limited resources for software support, then why people
>> take the risk and update their stuff is something that has been discussed
>> for decades now. And no one seems to converge to a point. (What Matti
>> said)
>> 8-  what Evgeni suggested
>>
>> I'm not sure what to do about this but I am also more worried about the
>> Python version than the NumPy version, my limited anectodal evidence
>> around
>> me suggests that people update NumPy + SciPy together mainly when they
>> use
>> pip with -U (--upgrade) on some package that has SciPy as a dependency.
>> So
>> I feel that it is safe to bump.
>>
>>
> Just for the NumPy perspective, there is nothing in the forthcoming NumPy
> 1.20 that doesn't work with Python 3.6, it is just a question of what
> wheels to provide. OTOH, as Python advances I don't want to worry about
> someone using newer features, whatever they are.  I go back and forth, but
> feel that 48 months is about the right support window and that argues for
> dropping Python 3.6. With the faster pace of Python development we will
> probably want to support the last four versions going forward. Note that
> SciPy doesn't need to support all the NumPy versions that support a
> particular version of Python, just the latest one. The main concern doesn't
> seem to be compatibility as much as availability.
>
> <snip>
>
> Chuck
>

From laurin.steidle at uni-hamburg.de  Thu Aug 27 05:13:10 2020
From: laurin.steidle at uni-hamburg.de (Laurin Steidle)
Date: Thu, 27 Aug 2020 11:13:10 +0200
Subject: [SciPy-Dev] Proposal for a new ODE Solver
Message-ID: <c75b9032-826d-88a9-6032-dfa3a9b134ff@uni-hamburg.de>

Hi all,

I opened an github issue [1] asking the following and was recommended to 
post it here.

I am *proposing to add a new ODE solver* to the list of solvers 
available in solve_ivp.
The solver I propose is the so called "*modified Patanka-Runge-Kutta" 
(MPRK)* [2] scheme.
MPRK is a integration scheme that satisfies the condition of 
'unconditional positivity',
a requirement for many ODEs especially in the field of biochemical models.
A simple and famous example of such a ODE being the Lotka-Volterra 
equations.

So far there exists no such solver in scipy.
Hence, a ODE solution with the present solvers can become "unphysical" 
and therefore invalid if the solution at a any time-step becomes 
negative due to numerical errors.

If this is something the community agrees on adding I am happy to write 
the implementation.
However, I would like to ask for assistance by a more experienced scipy 
dev as I am new to scipy.

Cheers,
Laurin

[1] https://github.com/scipy/scipy/issues/12766
[2] https://doi.org/10.1016/S0168-9274(03)00101-6

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200827/b3caaa14/attachment.html>

From juanlu001 at gmail.com  Thu Aug 27 05:56:59 2020
From: juanlu001 at gmail.com (Juan Luis Cano)
Date: Thu, 27 Aug 2020 11:56:59 +0200
Subject: [SciPy-Dev] More robust scalar root finding
Message-ID: <CAFmivUWdXbTKpYy2fPBi0zx8j6ce7Csffcpr1E1WrSers8Vq0Q@mail.gmail.com>

Hi all,

A month ago I was trying to migrate some MATLAB code to Python and found a
couple of annoyances in the scalar root finding functions of SciPy. I
opened these issues:

"Make root_scalar work when only x0 is given, similar to newton"
https://github.com/scipy/scipy/issues/12620
"newton chokes when iteration steps outside function domain"
https://github.com/scipy/scipy/issues/12625

and a pull request to address #12625:

"Make scalar, iterative root finding methods robust against steps outside
domain" https://github.com/scipy/scipy/pull/12626

It seems that the conversation has stalled, so I wanted to bring them to
the mailing list.

As I said in #12625, common wisdom dictates that if one chooses a starting
guess closer to the real solution, convergence is achieved. However,
leaving this well known numerical fact aside for a moment, I wonder if some
work could be done on the *usability* side of things to

(a) make root_scalar a drop-in replacement to newton (#12620)
(b) have some opt-out way to increase the robustness of the solution in a
way that "just works", which is that MATLAB and Octave achieve (#12625)

A personal side note: The original authors of the code I'm migrating tried
Python half a decade ago but noticed that "SciPy was not converging, while
MATLAB had no problems", so they ended up using the latter. While we all
scoffed when the infamous "top reasons to choose MATLAB" comparison was
published [1][2][3][4], I do think there's still a lot to understand from a
UX point of view of why people choose one or another. After 7 years of
teaching Python to engineers, they either become data scientists or they go
back to MATLAB, and as a result I have decided to get my hands dirty and
find out the reasons, to the best of my possibilities.

Best,

Juan Luis

[1]
https://web.archive.org/web/20190806100605/https://www.mathworks.com/products/matlab/matlab-vs-python.html
[2] https://twitter.com/moorepants/status/1158248132355874817
[3] https://twitter.com/randal_olson/status/1158402855818018816
[4] https://twitter.com/neuromusic/status/1158556716096618497
[5] https://twitter.com/vboykis/status/1158559458173313024
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200827/2537c691/attachment.html>

From evgeny.burovskiy at gmail.com  Thu Aug 27 06:28:28 2020
From: evgeny.burovskiy at gmail.com (Evgeni Burovski)
Date: Thu, 27 Aug 2020 13:28:28 +0300
Subject: [SciPy-Dev] More robust scalar root finding
In-Reply-To: <CAFmivUWdXbTKpYy2fPBi0zx8j6ce7Csffcpr1E1WrSers8Vq0Q@mail.gmail.com>
References: <CAFmivUWdXbTKpYy2fPBi0zx8j6ce7Csffcpr1E1WrSers8Vq0Q@mail.gmail.com>
Message-ID: <CAMRo0it_XmhmwLmYNPPdAo1E6_iH37ZEXAi=aqUgLg9eea=gJQ@mail.gmail.com>

FWIW I'd always recommend brentq for scalar root-finding.

Trying to make newton-based methods more robust seems to be more or less
reinventing that wheel, which has already been implemented (don't know who
implemented it; hope you read this thread so that you see my thank you!)
and is as robust  as it gets.

??, 27 ???. 2020 ?., 12:57 Juan Luis Cano <juanlu001 at gmail.com>:

> Hi all,
>
> A month ago I was trying to migrate some MATLAB code to Python and found a
> couple of annoyances in the scalar root finding functions of SciPy. I
> opened these issues:
>
> "Make root_scalar work when only x0 is given, similar to newton"
> https://github.com/scipy/scipy/issues/12620
> "newton chokes when iteration steps outside function domain"
> https://github.com/scipy/scipy/issues/12625
>
> and a pull request to address #12625:
>
> "Make scalar, iterative root finding methods robust against steps outside
> domain" https://github.com/scipy/scipy/pull/12626
>
> It seems that the conversation has stalled, so I wanted to bring them to
> the mailing list.
>
> As I said in #12625, common wisdom dictates that if one chooses a starting
> guess closer to the real solution, convergence is achieved. However,
> leaving this well known numerical fact aside for a moment, I wonder if some
> work could be done on the *usability* side of things to
>
> (a) make root_scalar a drop-in replacement to newton (#12620)
> (b) have some opt-out way to increase the robustness of the solution in a
> way that "just works", which is that MATLAB and Octave achieve (#12625)
>
> A personal side note: The original authors of the code I'm migrating tried
> Python half a decade ago but noticed that "SciPy was not converging, while
> MATLAB had no problems", so they ended up using the latter. While we all
> scoffed when the infamous "top reasons to choose MATLAB" comparison was
> published [1][2][3][4], I do think there's still a lot to understand from a
> UX point of view of why people choose one or another. After 7 years of
> teaching Python to engineers, they either become data scientists or they go
> back to MATLAB, and as a result I have decided to get my hands dirty and
> find out the reasons, to the best of my possibilities.
>
> Best,
>
> Juan Luis
>
> [1]
> https://web.archive.org/web/20190806100605/https://www.mathworks.com/products/matlab/matlab-vs-python.html
> [2] https://twitter.com/moorepants/status/1158248132355874817
> [3] https://twitter.com/randal_olson/status/1158402855818018816
> [4] https://twitter.com/neuromusic/status/1158556716096618497
> [5] https://twitter.com/vboykis/status/1158559458173313024
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200827/bc3ba959/attachment-0001.html>

From ewm at redtetrahedron.org  Fri Aug 28 09:54:11 2020
From: ewm at redtetrahedron.org (Eric Moore)
Date: Fri, 28 Aug 2020 09:54:11 -0400
Subject: [SciPy-Dev] More robust scalar root finding
In-Reply-To: <CAMRo0it_XmhmwLmYNPPdAo1E6_iH37ZEXAi=aqUgLg9eea=gJQ@mail.gmail.com>
References: <CAFmivUWdXbTKpYy2fPBi0zx8j6ce7Csffcpr1E1WrSers8Vq0Q@mail.gmail.com>
 <CAMRo0it_XmhmwLmYNPPdAo1E6_iH37ZEXAi=aqUgLg9eea=gJQ@mail.gmail.com>
Message-ID: <CAGeA38mOgg6en9yaB4MyuXeyM4=Uq+VSkhyuEZDm8bRB+CSJLQ@mail.gmail.com>

I suspect that people end up not using brentq since it requires a bracket
and not a single point.  I think adding some kind of heuristic to
root_scalar to find a bracket from an initial guess and then using brentq
or toms748 to find the root would be a good addition, if a good bit of
work.  This really is the missing piece between say Matlab's fzero and
root_scalar and is why I think people are using newton instead.   I think
boost has something for this, but I haven't used it to know if it is any
good.

Eric

On Thu, Aug 27, 2020 at 6:29 AM Evgeni Burovski <evgeny.burovskiy at gmail.com>
wrote:

> FWIW I'd always recommend brentq for scalar root-finding.
>
> Trying to make newton-based methods more robust seems to be more or less
> reinventing that wheel, which has already been implemented (don't know who
> implemented it; hope you read this thread so that you see my thank you!)
> and is as robust  as it gets.
>
> ??, 27 ???. 2020 ?., 12:57 Juan Luis Cano <juanlu001 at gmail.com>:
>
>> Hi all,
>>
>> A month ago I was trying to migrate some MATLAB code to Python and found
>> a couple of annoyances in the scalar root finding functions of SciPy. I
>> opened these issues:
>>
>> "Make root_scalar work when only x0 is given, similar to newton"
>> https://github.com/scipy/scipy/issues/12620
>> "newton chokes when iteration steps outside function domain"
>> https://github.com/scipy/scipy/issues/12625
>>
>> and a pull request to address #12625:
>>
>> "Make scalar, iterative root finding methods robust against steps outside
>> domain" https://github.com/scipy/scipy/pull/12626
>>
>> It seems that the conversation has stalled, so I wanted to bring them to
>> the mailing list.
>>
>> As I said in #12625, common wisdom dictates that if one chooses a
>> starting guess closer to the real solution, convergence is achieved.
>> However, leaving this well known numerical fact aside for a moment, I
>> wonder if some work could be done on the *usability* side of things to
>>
>> (a) make root_scalar a drop-in replacement to newton (#12620)
>> (b) have some opt-out way to increase the robustness of the solution in a
>> way that "just works", which is that MATLAB and Octave achieve (#12625)
>>
>> A personal side note: The original authors of the code I'm migrating
>> tried Python half a decade ago but noticed that "SciPy was not converging,
>> while MATLAB had no problems", so they ended up using the latter. While we
>> all scoffed when the infamous "top reasons to choose MATLAB" comparison was
>> published [1][2][3][4], I do think there's still a lot to understand from a
>> UX point of view of why people choose one or another. After 7 years of
>> teaching Python to engineers, they either become data scientists or they go
>> back to MATLAB, and as a result I have decided to get my hands dirty and
>> find out the reasons, to the best of my possibilities.
>>
>> Best,
>>
>> Juan Luis
>>
>> [1]
>> https://web.archive.org/web/20190806100605/https://www.mathworks.com/products/matlab/matlab-vs-python.html
>> [2] https://twitter.com/moorepants/status/1158248132355874817
>> [3] https://twitter.com/randal_olson/status/1158402855818018816
>> [4] https://twitter.com/neuromusic/status/1158556716096618497
>> [5] https://twitter.com/vboykis/status/1158559458173313024
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200828/f41afcb0/attachment.html>

From evgeny.burovskiy at gmail.com  Fri Aug 28 11:05:54 2020
From: evgeny.burovskiy at gmail.com (Evgeni Burovski)
Date: Fri, 28 Aug 2020 18:05:54 +0300
Subject: [SciPy-Dev] More robust scalar root finding
In-Reply-To: <CAGeA38mOgg6en9yaB4MyuXeyM4=Uq+VSkhyuEZDm8bRB+CSJLQ@mail.gmail.com>
References: <CAFmivUWdXbTKpYy2fPBi0zx8j6ce7Csffcpr1E1WrSers8Vq0Q@mail.gmail.com>
 <CAMRo0it_XmhmwLmYNPPdAo1E6_iH37ZEXAi=aqUgLg9eea=gJQ@mail.gmail.com>
 <CAGeA38mOgg6en9yaB4MyuXeyM4=Uq+VSkhyuEZDm8bRB+CSJLQ@mail.gmail.com>
Message-ID: <CAMRo0itQKH5j_jZRk69f=8WxCrW4BNNqaE56LeDhijCEnKGt=w@mail.gmail.com>

Shrug. IMO bracketing is a feature. Unless the interval is really
inbounded, i.e. the roots may go to some sort of asymptotics. In which case
I suspect a heuristics need to take into account specific details of the
problem, and a general case heuristics breaks down.
(A prototypical case in mind here is something like `x = a ctg(x) + c`,
where depending on a and c, a root may be close to a multiple of \pi/2.)

I once tried using boost root-finding and after some effort moved over to
brentq + fsolve, which worked much better. It was a while ago, maybe things
have improved over there.

This all is not to say that a there is no room for some bracketing
heuristics. What I'm saying is that this should probably be a separate
step, as in

>>> x1, x2 = try_bracketing(f, x0)
>>> root = brentq(f, x1, x2)

Cheers,

Evgeni

??, 28 ???. 2020 ?., 16:54 Eric Moore <ewm at redtetrahedron.org>:

> I suspect that people end up not using brentq since it requires a bracket
> and not a single point.  I think adding some kind of heuristic to
> root_scalar to find a bracket from an initial guess and then using brentq
> or toms748 to find the root would be a good addition, if a good bit of
> work.  This really is the missing piece between say Matlab's fzero and
> root_scalar and is why I think people are using newton instead.   I think
> boost has something for this, but I haven't used it to know if it is any
> good.
>
> Eric
>
> On Thu, Aug 27, 2020 at 6:29 AM Evgeni Burovski <
> evgeny.burovskiy at gmail.com> wrote:
>
>> FWIW I'd always recommend brentq for scalar root-finding.
>>
>> Trying to make newton-based methods more robust seems to be more or less
>> reinventing that wheel, which has already been implemented (don't know who
>> implemented it; hope you read this thread so that you see my thank you!)
>> and is as robust  as it gets.
>>
>> ??, 27 ???. 2020 ?., 12:57 Juan Luis Cano <juanlu001 at gmail.com>:
>>
>>> Hi all,
>>>
>>> A month ago I was trying to migrate some MATLAB code to Python and found
>>> a couple of annoyances in the scalar root finding functions of SciPy. I
>>> opened these issues:
>>>
>>> "Make root_scalar work when only x0 is given, similar to newton"
>>> https://github.com/scipy/scipy/issues/12620
>>> "newton chokes when iteration steps outside function domain"
>>> https://github.com/scipy/scipy/issues/12625
>>>
>>> and a pull request to address #12625:
>>>
>>> "Make scalar, iterative root finding methods robust against steps
>>> outside domain" https://github.com/scipy/scipy/pull/12626
>>>
>>> It seems that the conversation has stalled, so I wanted to bring them to
>>> the mailing list.
>>>
>>> As I said in #12625, common wisdom dictates that if one chooses a
>>> starting guess closer to the real solution, convergence is achieved.
>>> However, leaving this well known numerical fact aside for a moment, I
>>> wonder if some work could be done on the *usability* side of things to
>>>
>>> (a) make root_scalar a drop-in replacement to newton (#12620)
>>> (b) have some opt-out way to increase the robustness of the solution in
>>> a way that "just works", which is that MATLAB and Octave achieve (#12625)
>>>
>>> A personal side note: The original authors of the code I'm migrating
>>> tried Python half a decade ago but noticed that "SciPy was not converging,
>>> while MATLAB had no problems", so they ended up using the latter. While we
>>> all scoffed when the infamous "top reasons to choose MATLAB" comparison was
>>> published [1][2][3][4], I do think there's still a lot to understand from a
>>> UX point of view of why people choose one or another. After 7 years of
>>> teaching Python to engineers, they either become data scientists or they go
>>> back to MATLAB, and as a result I have decided to get my hands dirty and
>>> find out the reasons, to the best of my possibilities.
>>>
>>> Best,
>>>
>>> Juan Luis
>>>
>>> [1]
>>> https://web.archive.org/web/20190806100605/https://www.mathworks.com/products/matlab/matlab-vs-python.html
>>> [2] https://twitter.com/moorepants/status/1158248132355874817
>>> [3] https://twitter.com/randal_olson/status/1158402855818018816
>>> [4] https://twitter.com/neuromusic/status/1158556716096618497
>>> [5] https://twitter.com/vboykis/status/1158559458173313024
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200828/0cae4a24/attachment-0001.html>

From ralf.gommers at gmail.com  Sat Aug 29 18:03:28 2020
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sat, 29 Aug 2020 23:03:28 +0100
Subject: [SciPy-Dev] Proposal for a new ODE Solver
In-Reply-To: <c75b9032-826d-88a9-6032-dfa3a9b134ff@uni-hamburg.de>
References: <c75b9032-826d-88a9-6032-dfa3a9b134ff@uni-hamburg.de>
Message-ID: <CABL7CQhB27TMd7oHYp8KM-VRvo0CHbD323U3_mCfmVsykESbfg@mail.gmail.com>

Hi Laurin,


On Thu, Aug 27, 2020 at 10:21 AM Laurin Steidle <
laurin.steidle at uni-hamburg.de> wrote:

> Hi all,
>
> I opened an github issue [1] asking the following and was recommended to
> post it here.
>
> I am *proposing to add a new ODE solver* to the list of solvers available
> in solve_ivp.
> The solver I propose is the so called "*modified Patanka-Runge-Kutta"
> (MPRK)* [2] scheme.
> MPRK is a integration scheme that satisfies the condition of
> 'unconditional positivity',
> a requirement for many ODEs especially in the field of biochemical models.
> A simple and famous example of such a ODE being the Lotka-Volterra
> equations.
>
> So far there exists no such solver in scipy.
> Hence, a ODE solution with the present solvers can become "unphysical" and
> therefore invalid if the solution at a any time-step becomes negative due
> to numerical errors.
>
Thanks for proposing this. That would indeed be a nice addition to
scipy.integrate it looks like. I'm not familiar with the method you
propose, however the paper seems to have a good amount of citations.

> If this is something the community agrees on adding I am happy to write
> the implementation.
> However, I would like to ask for assistance by a more experienced scipy
> dev as I am new to scipy.
>
We can help you out as needed. Following the code/test/doc pattern of a
newer solver like `solve_ivp` should be a good start.

Cheers,
Ralf

Cheers,
> Laurin
>
> [1] https://github.com/scipy/scipy/issues/12766
> [2] https://doi.org/10.1016/S0168-9274(03)00101-6
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200829/b19d0b9b/attachment.html>

From rlucas7 at vt.edu  Sun Aug 30 21:47:15 2020
From: rlucas7 at vt.edu (rlucas7 at vt.edu)
Date: Sun, 30 Aug 2020 21:47:15 -0400
Subject: [SciPy-Dev] Thanks wiki protocol
In-Reply-To: <CAHPuU_b+facUkYt3M67W+jBzSCw9eYUQF01Y7azBzF_grapB9Q@mail.gmail.com>
References: <CAHPuU_b+facUkYt3M67W+jBzSCw9eYUQF01Y7azBzF_grapB9Q@mail.gmail.com>
Message-ID: <77147890-3279-4870-9B98-D996FB584121@vt.edu>


> On Aug 17, 2020, at 10:40 PM, Tyler Reddy <tyler.je.reddy at gmail.com> wrote:
> 
> ?
> For the author lists (with new contributors highlighted), the command looks like "  python tools/authors.py 042731365..fb51f487c8 "
> 
> I think some of those release management tools are dependent on proper usage of milestones, though perhaps not in that specific case where the hash range is provided directly.
> 

Thanks Tyler. I didn?t check closely on the milestone issue-you mean use of tags on the commit log right?

I was able to add a flag to get the names and nothing else so those could easily be copied into the file on the website. 

Pr is:

https://github.com/scipy/scipy/pull/12793

I also opened a pr on the website repo to move over thanks file:

https://github.com/scipy/scipy.org/pull/364

Finally, I opened a separate pr to delete the existing thanks:

https://github.com/scipy/scipy/pull/12792

I think this is what you had in mind Ralf but if not please let me know on the PR(s). 

>> On Sun, 16 Aug 2020 at 18:04, <rlucas7 at vt.edu> wrote:
>> 
>>>> On Aug 15, 2020, at 8:15 PM, Matt Haberland <mhaberla at calpoly.edu> wrote:
>>>> 
>>> ?
>>> +1 for retiring the manually updated file. 
>>> 
>>>> On Sat, Aug 15, 2020, 3:55 PM Ralf Gommers <ralf.gommers at gmail.com> wrote:
>>>> 
>>>> 
>>>>> On Sat, Aug 15, 2020 at 9:43 PM <rlucas7 at vt.edu> wrote:
>>>>> Hi scipy-dev, 
>>>>> 
>>>>> For work on the 1.5 release we used a wiki rather than have people add commits to the thanks file in the repo. 
>>>>> I?m assuming this was to alleviate rebase conflicts to merge. My question is, 
>>>>> 
>>>>> Is using a wikifor tracking thanks to add in each release the way we track thanks entries moving forward? 
>>>>> 
>>>>> To be clear these would be separate wiki pages from the release notes wiki pages. 
>>>> 
>>>> That does sound like a good idea.
>>>> 
>>>> A perhaps even better one is to retire the THANKS.txt file. It really doesn't have much value, is very inconsistently updated and old entries are horribly out of date (e.g. Pauli is thanked for docs work and interpolation bug fixes - that doesn't quite cover his contributions :)).
>>>> 
>> 
>> Yeah agree. 
>> 
>>>> I'd prefer to extract the relevant historical info and put that on the website. Or just have a full list of contributors that's autogenerated.
>> 
>> We already do the auto generate for each release IIRC. At least I assume that the RM (Tyler) does that in an auto generated way. 
>> 
>>>> The thanks to institutions at the bottom of the file already lives at https://scipy.org/scipylib/donations.html#acknowledgements
>> 
>> We could move the individual thanks entries (as currently written) to the scipy site. As a separate section on this acknowledgements page.
>> 
>> That would be a one off change but what about moving forward? 
>> 
>> I?ve noticed on the release notes there are indications for first time contributors. I?ve not used those tools myself.
>> 
>>  Is there a way we can get a list of these first time contributors/individuals from the same tool(s)? 
>> 
>> That would probably be the easiest thing to do if we want to periodically add/update a contributors page. 
>> 
>>>> Cheers,
>>>> Ralf
>>>> 
>>>> _______________________________________________
>>>> SciPy-Dev mailing list
>>>> SciPy-Dev at python.org
>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200830/4450ae72/attachment.html>

From ralf.gommers at gmail.com  Mon Aug 31 05:29:12 2020
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Mon, 31 Aug 2020 10:29:12 +0100
Subject: [SciPy-Dev] Thanks wiki protocol
In-Reply-To: <77147890-3279-4870-9B98-D996FB584121@vt.edu>
References: <CAHPuU_b+facUkYt3M67W+jBzSCw9eYUQF01Y7azBzF_grapB9Q@mail.gmail.com>
 <77147890-3279-4870-9B98-D996FB584121@vt.edu>
Message-ID: <CABL7CQjBhzP85akGDstD_rDGeVF2uRpttY0fXF1fpETcKqqToA@mail.gmail.com>

On Mon, Aug 31, 2020 at 2:47 AM <rlucas7 at vt.edu> wrote:

>
> On Aug 17, 2020, at 10:40 PM, Tyler Reddy <tyler.je.reddy at gmail.com>
> wrote:
>
> ?
> For the author lists (with new contributors highlighted), the command
> looks like " python tools/authors.py 042731365..fb51f487c8 "
>
> I think some of those release management tools are dependent on proper
> usage of milestones, though perhaps not in that specific case where the
> hash range is provided directly.
>
>
> Thanks Tyler. I didn?t check closely on the milestone issue-you mean use
> of tags on the commit log right?
>
> I was able to add a flag to get the names and nothing else so those could
> easily be copied into the file on the website.
>
> Pr is:
>
> https://github.com/scipy/scipy/pull/12793
>
> I also opened a pr on the website repo to move over thanks file:
>
> https://github.com/scipy/scipy.org/pull/364
>
> Finally, I opened a separate pr to delete the existing thanks:
>
> https://github.com/scipy/scipy/pull/12792
>
> I think this is what you had in mind Ralf but if not please let me know on
> the PR(s).
>

Thanks Lucas. That set of PRs looks like what I had in mind. Looking at the
moving of content - should we decide right now to drop all the
outdated/incomplete stuff, and simply have a list of contributor names?
Either organized by date of first contribution, or alphabetically?

Cheers,
Ralf



> On Sun, 16 Aug 2020 at 18:04, <rlucas7 at vt.edu> wrote:
>
>>
>> On Aug 15, 2020, at 8:15 PM, Matt Haberland <mhaberla at calpoly.edu> wrote:
>>
>> ?
>> +1 for retiring the manually updated file.
>>
>> On Sat, Aug 15, 2020, 3:55 PM Ralf Gommers <ralf.gommers at gmail.com>
>> wrote:
>>
>>>
>>>
>>> On Sat, Aug 15, 2020 at 9:43 PM <rlucas7 at vt.edu> wrote:
>>>
>>>> Hi scipy-dev,
>>>>
>>>> For work on the 1.5 release we used a wiki rather than have people add
>>>> commits to the thanks file in the repo.
>>>> I?m assuming this was to alleviate rebase conflicts to merge. My
>>>> question is,
>>>>
>>>> Is using a wikifor tracking thanks to add in each release the way we
>>>> track thanks entries moving forward?
>>>>
>>>> To be clear these would be separate wiki pages from the release notes
>>>> wiki pages.
>>>>
>>>
>>> That does sound like a good idea.
>>>
>>> A perhaps even better one is to retire the THANKS.txt file. It really
>>> doesn't have much value, is very inconsistently updated and old entries are
>>> horribly out of date (e.g. Pauli is thanked for docs work and interpolation
>>> bug fixes - that doesn't quite cover his contributions :)).
>>>
>>>
>> Yeah agree.
>>
>> I'd prefer to extract the relevant historical info and put that on the
>>> website. Or just have a full list of contributors that's autogenerated.
>>>
>>
>> We already do the auto generate for each release IIRC. At least I assume
>> that the RM (Tyler) does that in an auto generated way.
>>
>> The thanks to institutions at the bottom of the file already lives at
>>> https://scipy.org/scipylib/donations.html#acknowledgements
>>>
>>
>> We could move the individual thanks entries (as currently written) to the
>> scipy site. As a separate section on this acknowledgements page.
>>
>> That would be a one off change but what about moving forward?
>>
>> I?ve noticed on the release notes there are indications for first time
>> contributors. I?ve not used those tools myself.
>>
>>  Is there a way we can get a list of these first time
>> contributors/individuals from the same tool(s)?
>>
>> That would probably be the easiest thing to do if we want to periodically
>> add/update a contributors page.
>>
>> Cheers,
>>> Ralf
>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200831/652899eb/attachment-0001.html>

From rlucas7 at vt.edu  Mon Aug 31 18:19:50 2020
From: rlucas7 at vt.edu (rlucas7 at vt.edu)
Date: Mon, 31 Aug 2020 18:19:50 -0400
Subject: [SciPy-Dev] Thanks wiki protocol
In-Reply-To: <CABL7CQjBhzP85akGDstD_rDGeVF2uRpttY0fXF1fpETcKqqToA@mail.gmail.com>
References: <CABL7CQjBhzP85akGDstD_rDGeVF2uRpttY0fXF1fpETcKqqToA@mail.gmail.com>
Message-ID: <281A1467-9EA2-4751-8AB7-79B57D61E441@vt.edu>



> On Aug 31, 2020, at 5:29 AM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
> 
> ?
> 
> 
> =
> On Mon, Aug 31, 2020 at 2:47 AM <<= a href=3D"mailto:rlucas7 at vt.edu">rlucas7 at vt.edu> wrote:
> 
> On= Aug 17, 2020, at 10:40 PM, Tyler Reddy <tyler.je.reddy at gmail.com> wrote:
> <= br>
> =EF=BB=BF<= div dir=3D"ltr">For the author lists (with new contributors highlighted), t= he command looks like " python tools/authors.py 042731365..fb51f487c= 8 "
> I = think some of those release management tools are dependent on proper usage = of milestones, though perhaps not in that specific case where the hash rang= e is provided directly.
> 
> 
> <= /div>Thanks Tyler. I didn=E2=80=99t check closely on the milestone issue-yo= u mean use of tags on the commit log right?
> 
> I was able t= o add a flag to get the names and nothing else so those could easily be cop= ied into the file on the website.=C2=A0
> 
> Pr is:
> 
> https://github.com/scipy/scipy/pull/12793
> = 
> I also opened a pr on the website repo to move over thanks f= ile:
> 
> https://github.com/scipy/scipy.org/pull/364
> 
> Finally, I opened a separate pr to delete the ex= isting thanks:
> 
> https://github.com/scipy/scipy/pull/12= 792
> 
> I think this is what you had in mind Ralf = but if not please let me know on the PR(s).=C2=A0
> <= div>
> Thanks Lucas. That set of PRs looks like what I had in m= ind. Looking at the moving of content - should we decide right now to drop = all the outdated/incomplete stuff, and simply have a list of contributor na= mes? Either organized by date of first contribution, or alphabetically?

 yes unless anyone objects let?s keep the names only. I?ll update the PR for that one.

Hmm, For the authors tool it?s not giving the date of the commit, unless I missed that in the source, dates way would be easier to track because names can just be appended. Lexicographic would be difficult to insert new unless we overwrite the *full* list of contributors (on the website) on each append.

If the lexicogrqphic case it would need to be by the first characters displayed. 
Some of the contributors only have their github handle and not a first/last name.

> 
> Cheers,
> Ralf
> 
> = 
> 
> On Sun, 16 Aug 2020 at 18:04, = <rlucas7 at vt.edu&= gt; wrote:
> 
> On Aug 15, 2020= , at 8:15 PM, Matt Haberland <mhaberla at calpoly.edu> wrote:
> 
> =EF=BB=BF
> +1 for retiring the manually updated file.=C2=A0
> 
> On Sat, Aug 15, 2020, 3:55= PM Ralf Gommers <ralf.gommers at gmail.com> wrote:
> 
> On Sat, A= ug 15, 2020 at 9:43 PM <rlucas7 at vt.edu> wrote:
> Hi scipy-dev, 
> 
> For work on the 1.5 release we used a wiki rather than have people add comm= its to the thanks file in the repo. 
> I=E2=80=99m assuming this was to alleviate rebase conflicts to merge. My qu= estion is, 
> 
> Is using a wikifor tracking thanks to add in each release the way we track = thanks entries moving forward? 
> 
> To be clear these would be separate wiki pages from the release notes wiki = pages.
> 
> That does sound like a good ide= a.
> 
> A perhaps even better one is to retire the THAN= KS.txt file. It really doesn't have much value, is very inconsistently = updated and old entries are horribly out of date (e.g. Pauli is thanked for= docs work and interpolation bug fixes - that doesn't quite cover his c= ontributions :)).
> 
> =
> 
> Yeah agree.=C2=A0
> 
> I'd prefer to extract the relevant historic= al info and put that on the website. Or just have a full list of contributo= rs that's autogenerated.
> 
> We already do the auto generate for each release I= IRC. At least I assume that the RM (Tyler) does that in an auto generated w= ay.=C2=A0
> 
> The than= ks to institutions at the bottom of the file already lives at https://scipy.org/scipylib/donations.html#acknowledgeme= nts
> = 
> We could move the individual thanks entries (as currently written) to t= he scipy site. As a separate section on this acknowledgements page.
> 
> That would be a one off change but what about moving forw= ard?=C2=A0
> 
> I=E2=80=99ve noticed on the release not= es there are indications for first time contributors. I=E2=80=99ve not used= those tools myself.
> 
> =C2=A0Is there a way we can g= et a list of these first time contributors/individuals from the same tool(s= )?=C2=A0
> 
> That would probably be the easiest thing = to do if we want to periodically add/update a contributors page.=C2=A0
> 
> Cheers,
> R= alf
> 
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinf= o/scipy-dev
> _______________________________________________
> SciPy= -Dev mailing list
> SciPy-Dev at python.org
> https://mai= l.python.org/mailman/listinfo/scipy-dev
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.= org
> https://mail.python.org/mailman/listinfo/scipy-dev= 
> _______________________________________________
> SciPy= -Dev mailing list
> SciPy-Dev at python.org
> https://mai= l.python.org/mailman/listinfo/scipy-dev
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.= org
> https://mail.python.org/mailman/listinfo/scipy-dev= 
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200831/a3054a8d/attachment.html>