From i at antonakhmerov.org Sun Aug 1 09:47:52 2021 From: i at antonakhmerov.org (Anton Akhmerov) Date: Sun, 1 Aug 2021 15:47:52 +0200 Subject: [SciPy-Dev] Proposal: loosen the coupling between scipy.sparse and np.matrix Message-ID: Dear scipy developers, TL;DR: instead of big changes to the scipy.sparse interface to become more array-like, I propose to reduce the exposure of users specifically to np.matrix. Firstly, I'm new to the list, and therefore I apologize in advance if I overlooked relevant previous discussions?I checked the issue tracker, the roadmap, and the last few months of the mailing list. Having found nothing that contradicts my take, I hope my proposal will improve the state of scipy.sparse. --- # Current state of the art For historical reasons, scipy.sparse was designed to mimic the np.matrix interface. However, due to various interface warts, numpy matrices became unpopular over time, and hence less known to users. They are slowly removed from various codebases, and numpy itself does not recommend using matrices (although it doesn't quite issue a deprecation warning yet). I assume that there's an overall community preference to get rid of numpy matrices over time, also from the scipy side. There are two main ways in which scipy.sparse.spmatrix relates to np.matrix: 1. The interface of spmatrix itself is similar to np.matrix. Some examples of this behavior are: - slicing a spmatrix returns another spmatrix even in cases when numpy would return a vector - __mul__ is __matmul__, and not element-wise multiplication - properties .H, .A, etc mimic np.matrix 2. spmatrix produces numpy matrices. Some examples of this behavior are the spmatrix.todense method and additions of ndarray and spmatrix. While changing the first aspect is hard both for reasons of backwards compatibility, and the amount of required changes in the codebase, the second takes less work but it is likely to improve the usability of spmatrix. A natural step would be to deprecate spmatrix.todense (or potentially switch it to return an array). Shortly after I opened an issue [1] proposing to do so, I learned that spmatrix.todense continues confusing new users [2], and even some contributors [3]. I think the current usage pattern capturing these cases is as follows (I certainly went through these steps a bunch). 1. Do something with sparse.spmatrix that produces an np.matrix without realizing it 2. See a bug, exception, or deprecation warning. 3. Figure out what's going wrong, play whack-a-mole and modify the code to produce an array instead. The other part of the interface (replicating np.matrix behavior) is also not without its problems. For example, carbon copying the semantics of matrix.A implements silent conversion of an spmatrix to array on attribute access, and therefore hides computational complexity. # Proposal I recognize that scipy.sparse is a mature and widely used codebase, and backwards compatibility is extremely important. At the same time, I believe that exposing users and developers to np.matrix also has a continued user and maintainer code. Therefore I propose to deprecate and then remove the ability to produce np.matrix from scipy.sparse codebase. I also propose to revisit the parts of the np.matrix interface that were copied, and that can cause troubles, spmatrix.A being a prime example [4]. What do you all think? Best, Anton [1]: https://github.com/scipy/scipy/issues/14494 [2]: https://github.com/scipy/scipy/issues/14131 [3]: https://github.com/scipy/scipy/pull/14488#discussion_r678451919 [4]: https://github.com/scipy/scipy/issues/14503 From tyler.je.reddy at gmail.com Sun Aug 1 22:30:31 2021 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Sun, 1 Aug 2021 20:30:31 -0600 Subject: [SciPy-Dev] ANN: SciPy 1.7.1 Message-ID: Hi all, On behalf of the SciPy development team I'm pleased to announce the release of SciPy 1.7.1, which is a bug fix release. Sources and binary wheels can be found at: https://pypi.org/project/scipy/ and at: https://github.com/scipy/scipy/releases/tag/v1.7.1 One of a few ways to install this release with pip: pip install scipy==1.7.1 ===================== SciPy 1.7.1 Release Notes ===================== SciPy 1.7.1 is a bug-fix release with no new features compared to 1.7.0. Authors ======= * Peter Bell * Evgeni Burovski * Justin Charlong + * Ralf Gommers * Matti Picus * Tyler Reddy * Pamphile Roy * Sebastian Wallk?tter * Arthur Volant A total of 9 people contributed to this release. People with a "+" by their names contributed a patch for the first time. This list of names is automatically generated, and may not be fully complete. Issues closed for 1.7.1 ------------------------------ * `#14074 `__: Segmentation fault when building cKDTree with Scipy 1.6.3. * `#14271 `__: scipy.io.loadmat failure in 1.7.0 * `#14273 `__: \`scipy.signal.{medfilt,medfilt2d}\` hit "Windows fatal exception:... * `#14282 `__: DOC, CI: stats skewtest refguide failure * `#14363 `__: Huge stack allocation in _sobol.pyx may cause stack overvflow * `#14382 `__: Memory leak in \`scipy.spatial.distance\` for \`cdist\` * `#14396 `__: BUG: Sphinx 4.1 breaks the banner's logo * `#14444 `__: DOC/FEAT Rotation.from_rotvec documents a degrees argument which... Pull requests for 1.7.1 ------------------------------ * `#14178 `__: DEV: Update Boschloo Exact test * `#14264 `__: REL: prepare for SciPy 1.7.1 * `#14283 `__: BUG: fix refguide-check namedtuple handling * `#14303 `__: FIX: Check for None before calling str methods * `#14327 `__: BUG: medfilt can access beyond the end of an array * `#14355 `__: BUG: KDTree balanced_tree is unbalanced for degenerate data * `#14368 `__: BUG: avoid large cython global variable in function * `#14384 `__: BUG: Reference count leak in distance_pybind * `#14397 `__: DOC/CI: do not allow sphinx 4.1. * `#14417 `__: DOC/CI: pin sphinx to !=4.1.0 * `#14460 `__: DOC: add required scipy version to kwarg * `#14466 `__: MAINT: 1.7.1 backports (round 1) * `#14508 `__: MAINT: bump scipy-mathjax * `#14509 `__: MAINT: 1.7.1 backports (round 2) Checksums ========= MD5 ~~~ ef8b44a175818183de28f2c3dacf4b74 scipy-1.7.1-cp37-cp37m-macosx_10_9_x86_64.whl 4f717b62946a6306bba88696b4992c73 scipy-1.7.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl decf6837d0a28bdeb911e6e2d18b777c scipy-1.7.1-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.whl 6449932605e3284f731744eb207e5612 scipy-1.7.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl fb94deaf9b43bf18890b0bd12fc26fda scipy-1.7.1-cp37-cp37m-win32.whl c5894e5811278243d7f4abeb1a5f230f scipy-1.7.1-cp37-cp37m-win_amd64.whl 3aec592a699f835319cbb4649f30df71 scipy-1.7.1-cp38-cp38-macosx_10_9_x86_64.whl 774a74a6c81d40c9a305523707c024f4 scipy-1.7.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 30206a19a96549f665bd608fe6bf2761 scipy-1.7.1-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl eb58d9f3797d47866bfe571d5df3b827 scipy-1.7.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl 808a907d994b98fd6dbe2050a48b8c69 scipy-1.7.1-cp38-cp38-win32.whl 688921def6681ee5abe8543aca8383c2 scipy-1.7.1-cp38-cp38-win_amd64.whl 2fe4e958cb14d0b071c494b9faee0c98 scipy-1.7.1-cp39-cp39-macosx_10_9_x86_64.whl dd5b4db9cf83a0594e0b651e198b16a4 scipy-1.7.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 67c5d75378d0ba2803c1b93fe670563b scipy-1.7.1-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl c1ea8eec1dd6dc9c1b3eae24e3b3a34a scipy-1.7.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl 93efd36f2c52dadbe7c9a377ad8c5be2 scipy-1.7.1-cp39-cp39-win32.whl f8d0f87aaa8929f059fcf840db345310 scipy-1.7.1-cp39-cp39-win_amd64.whl 8ac74369cdcabc097f602682c951197c scipy-1.7.1.tar.gz deb130f3959e5623fafb4a262c28183b scipy-1.7.1.tar.xz 5dd4ab895eaa141cb01b954ecae2ddfc scipy-1.7.1.zip SHA256 ~~~~~~ 2a0eeaab01258e0870c4022a6cd329aef3b7c6c2b606bd7cf7bb2ba9820ae561 scipy-1.7.1-cp37-cp37m-macosx_10_9_x86_64.whl 3f52470e0548cdb74fb8ddf06773ffdcca7c97550f903b1c51312ec19243a7f7 scipy-1.7.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 787749110a23502031fb1643c55a2236c99c6b989cca703ea2114d65e21728ef scipy-1.7.1-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.whl 3304bd5bc32e00954ac4b3f4cc382ca8824719bf348aacbec6347337d6b125fe scipy-1.7.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl d1388fbac9dd591ea630da75c455f4cc637a7ca5ecb31a6b6cef430914749cde scipy-1.7.1-cp37-cp37m-win32.whl d648aa85dd5074b1ed83008ae987c3fbb53d68af619fce1dee231f4d8bd40e2f scipy-1.7.1-cp37-cp37m-win_amd64.whl bc61e3e5ff92d2f32bb263621d54a9cff5e3f7c420af3d1fa122ce2529de2bd9 scipy-1.7.1-cp38-cp38-macosx_10_9_x86_64.whl a496b42dbcd04ea9924f5e92be63af3d8e0f43a274b769bfaca0a297327d54ee scipy-1.7.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl d13f31457f2216e5705304d9f28e2826edf75487410a57aa99263fa4ffd792c2 scipy-1.7.1-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl 90c07ba5f34f33299a428b0d4fa24c30d2ceba44d63f8385b2b05be460819fcb scipy-1.7.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl efdd3825d54c58df2cc394366ca4b9166cf940a0ebddeb87b6c10053deb625ea scipy-1.7.1-cp38-cp38-win32.whl 71cfc96297617eab911e22216e8a8597703202e95636d9406df9af5c2ac99a2b scipy-1.7.1-cp38-cp38-win_amd64.whl 4ee952f39a4a4c7ba775a32b664b1f4b74818548b65f765987adc14bb78f5802 scipy-1.7.1-cp39-cp39-macosx_10_9_x86_64.whl 611f9cb459d0707dd8e4de0c96f86e93f61aac7475fcb225e9ec71fecdc5cebf scipy-1.7.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl e101bceeb9e65a90dadbc5ca31283403a2d4667b9c178db29109750568e8d112 scipy-1.7.1-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl 4729b41a4cdaf4cd011aeac816b532f990bdf97710cef59149d3e293115cf467 scipy-1.7.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl c9951e3746b68974125e5e3445008a4163dd6d20ae0bbdae22b38cb8951dc11b scipy-1.7.1-cp39-cp39-win32.whl da9c6b336e540def0b7fd65603da8abeb306c5fc9a5f4238665cbbb5ff95cf58 scipy-1.7.1-cp39-cp39-win_amd64.whl 6b47d5fa7ea651054362561a28b1ccc8da9368a39514c1bbf6c0977a1c376764 scipy-1.7.1.tar.gz fdfe1d1eb1569846e331bd8d72106a8c446dafb2192c00adbb5376b02a0a1104 scipy-1.7.1.tar.xz 0dbea8556cb3770656770e5ed02e451dd3069c2f2f70ac885dea65f679a23afe scipy-1.7.1.zip -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Sun Aug 1 23:26:22 2021 From: andyfaff at gmail.com (Andrew Nelson) Date: Mon, 2 Aug 2021 13:26:22 +1000 Subject: [SciPy-Dev] scipy docs are down. Message-ID: https://docs.scipy.org/doc/scipy/reference/ is currently returning: Not Found The requested URL /doc/scipy/reference/ was not found on this server. ------------------------------ Apache/2.4.39 (Ubuntu) Server at docs.scipy.org Port 443 -- _____________________________________ Dr. Andrew Nelson _____________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyler.je.reddy at gmail.com Mon Aug 2 00:00:14 2021 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Sun, 1 Aug 2021 22:00:14 -0600 Subject: [SciPy-Dev] scipy docs are down. In-Reply-To: References: Message-ID: Probably because I just did a release. The new theme recently required an adjustment this should already be present on the maintenance branch: https://github.com/scipy/scipy/pull/14272 I can probably check the server a bit since I have ssh access I suppose On Sun, 1 Aug 2021 at 21:27, Andrew Nelson wrote: > https://docs.scipy.org/doc/scipy/reference/ > > is currently returning: > > Not Found > > The requested URL /doc/scipy/reference/ was not found on this server. > ------------------------------ > Apache/2.4.39 (Ubuntu) Server at docs.scipy.org Port 443 > > > > -- > _____________________________________ > Dr. Andrew Nelson > > > _____________________________________ > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyler.je.reddy at gmail.com Mon Aug 2 00:16:39 2021 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Sun, 1 Aug 2021 22:16:39 -0600 Subject: [SciPy-Dev] scipy docs are down. In-Reply-To: References: Message-ID: Ok, it seems to work now after some manual fiddling on the server. Please let me know if you see anything else that is strange--this is what I had to do to "fix" it: https://github.com/scipy/scipy/issues/14267#issuecomment-890698718 On Sun, 1 Aug 2021 at 22:00, Tyler Reddy wrote: > Probably because I just did a release. > > The new theme recently required an adjustment this should already be > present on the maintenance branch: > https://github.com/scipy/scipy/pull/14272 > > I can probably check the server a bit since I have ssh access I suppose > > On Sun, 1 Aug 2021 at 21:27, Andrew Nelson wrote: > >> https://docs.scipy.org/doc/scipy/reference/ >> >> is currently returning: >> >> Not Found >> >> The requested URL /doc/scipy/reference/ was not found on this server. >> ------------------------------ >> Apache/2.4.39 (Ubuntu) Server at docs.scipy.org Port 443 >> >> >> >> -- >> _____________________________________ >> Dr. Andrew Nelson >> >> >> _____________________________________ >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From evgeny.burovskiy at gmail.com Mon Aug 2 02:55:24 2021 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Mon, 2 Aug 2021 09:55:24 +0300 Subject: [SciPy-Dev] [Numpy-discussion] ANN: SciPy 1.7.1 In-Reply-To: References: Message-ID: Thank you Tyler for managing this one! ??, 2 ???. 2021 ?., 05:31 Tyler Reddy : > Hi all, > > On behalf of the SciPy development team I'm pleased to announce > the release of SciPy 1.7.1, which is a bug fix release. > > Sources and binary wheels can be found at: > https://pypi.org/project/scipy/ > and at: https://github.com/scipy/scipy/releases/tag/v1.7.1 > > > > One of a few ways to install this release with pip: > > pip install scipy==1.7.1 > > ===================== > SciPy 1.7.1 Release Notes > ===================== > > SciPy 1.7.1 is a bug-fix release with no new features > compared to 1.7.0. > > Authors > ======= > > * Peter Bell > * Evgeni Burovski > * Justin Charlong + > * Ralf Gommers > * Matti Picus > * Tyler Reddy > * Pamphile Roy > * Sebastian Wallk?tter > * Arthur Volant > > A total of 9 people contributed to this release. > People with a "+" by their names contributed a patch for the first time. > This list of names is automatically generated, and may not be fully > complete. > > Issues closed for 1.7.1 > ------------------------------ > > * `#14074 `__: Segmentation > fault when building cKDTree with Scipy 1.6.3. > * `#14271 `__: > scipy.io.loadmat failure in 1.7.0 > * `#14273 `__: > \`scipy.signal.{medfilt,medfilt2d}\` hit "Windows fatal exception:... > * `#14282 `__: DOC, CI: > stats skewtest refguide failure > * `#14363 `__: Huge stack > allocation in _sobol.pyx may cause stack overvflow > * `#14382 `__: Memory leak > in \`scipy.spatial.distance\` for \`cdist\` > * `#14396 `__: BUG: Sphinx > 4.1 breaks the banner's logo > * `#14444 `__: DOC/FEAT > Rotation.from_rotvec documents a degrees argument which... > > Pull requests for 1.7.1 > ------------------------------ > > * `#14178 `__: DEV: Update > Boschloo Exact test > * `#14264 `__: REL: prepare > for SciPy 1.7.1 > * `#14283 `__: BUG: fix > refguide-check namedtuple handling > * `#14303 `__: FIX: Check for > None before calling str methods > * `#14327 `__: BUG: medfilt > can access beyond the end of an array > * `#14355 `__: BUG: KDTree > balanced_tree is unbalanced for degenerate data > * `#14368 `__: BUG: avoid > large cython global variable in function > * `#14384 `__: BUG: Reference > count leak in distance_pybind > * `#14397 `__: DOC/CI: do not > allow sphinx 4.1. > * `#14417 `__: DOC/CI: pin > sphinx to !=4.1.0 > * `#14460 `__: DOC: add > required scipy version to kwarg > * `#14466 `__: MAINT: 1.7.1 > backports (round 1) > * `#14508 `__: MAINT: bump > scipy-mathjax > * `#14509 `__: MAINT: 1.7.1 > backports (round 2) > > > Checksums > ========= > > MD5 > ~~~ > > ef8b44a175818183de28f2c3dacf4b74 > scipy-1.7.1-cp37-cp37m-macosx_10_9_x86_64.whl > 4f717b62946a6306bba88696b4992c73 > scipy-1.7.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl > decf6837d0a28bdeb911e6e2d18b777c > scipy-1.7.1-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.whl > 6449932605e3284f731744eb207e5612 > scipy-1.7.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl > fb94deaf9b43bf18890b0bd12fc26fda scipy-1.7.1-cp37-cp37m-win32.whl > c5894e5811278243d7f4abeb1a5f230f scipy-1.7.1-cp37-cp37m-win_amd64.whl > 3aec592a699f835319cbb4649f30df71 > scipy-1.7.1-cp38-cp38-macosx_10_9_x86_64.whl > 774a74a6c81d40c9a305523707c024f4 > scipy-1.7.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl > 30206a19a96549f665bd608fe6bf2761 > scipy-1.7.1-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl > eb58d9f3797d47866bfe571d5df3b827 > scipy-1.7.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl > 808a907d994b98fd6dbe2050a48b8c69 scipy-1.7.1-cp38-cp38-win32.whl > 688921def6681ee5abe8543aca8383c2 scipy-1.7.1-cp38-cp38-win_amd64.whl > 2fe4e958cb14d0b071c494b9faee0c98 > scipy-1.7.1-cp39-cp39-macosx_10_9_x86_64.whl > dd5b4db9cf83a0594e0b651e198b16a4 > scipy-1.7.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl > 67c5d75378d0ba2803c1b93fe670563b > scipy-1.7.1-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl > c1ea8eec1dd6dc9c1b3eae24e3b3a34a > scipy-1.7.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl > 93efd36f2c52dadbe7c9a377ad8c5be2 scipy-1.7.1-cp39-cp39-win32.whl > f8d0f87aaa8929f059fcf840db345310 scipy-1.7.1-cp39-cp39-win_amd64.whl > 8ac74369cdcabc097f602682c951197c scipy-1.7.1.tar.gz > deb130f3959e5623fafb4a262c28183b scipy-1.7.1.tar.xz > 5dd4ab895eaa141cb01b954ecae2ddfc scipy-1.7.1.zip > > SHA256 > ~~~~~~ > > 2a0eeaab01258e0870c4022a6cd329aef3b7c6c2b606bd7cf7bb2ba9820ae561 > scipy-1.7.1-cp37-cp37m-macosx_10_9_x86_64.whl > 3f52470e0548cdb74fb8ddf06773ffdcca7c97550f903b1c51312ec19243a7f7 > scipy-1.7.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl > 787749110a23502031fb1643c55a2236c99c6b989cca703ea2114d65e21728ef > scipy-1.7.1-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.whl > 3304bd5bc32e00954ac4b3f4cc382ca8824719bf348aacbec6347337d6b125fe > scipy-1.7.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl > d1388fbac9dd591ea630da75c455f4cc637a7ca5ecb31a6b6cef430914749cde > scipy-1.7.1-cp37-cp37m-win32.whl > d648aa85dd5074b1ed83008ae987c3fbb53d68af619fce1dee231f4d8bd40e2f > scipy-1.7.1-cp37-cp37m-win_amd64.whl > bc61e3e5ff92d2f32bb263621d54a9cff5e3f7c420af3d1fa122ce2529de2bd9 > scipy-1.7.1-cp38-cp38-macosx_10_9_x86_64.whl > a496b42dbcd04ea9924f5e92be63af3d8e0f43a274b769bfaca0a297327d54ee > scipy-1.7.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl > d13f31457f2216e5705304d9f28e2826edf75487410a57aa99263fa4ffd792c2 > scipy-1.7.1-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl > 90c07ba5f34f33299a428b0d4fa24c30d2ceba44d63f8385b2b05be460819fcb > scipy-1.7.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl > efdd3825d54c58df2cc394366ca4b9166cf940a0ebddeb87b6c10053deb625ea > scipy-1.7.1-cp38-cp38-win32.whl > 71cfc96297617eab911e22216e8a8597703202e95636d9406df9af5c2ac99a2b > scipy-1.7.1-cp38-cp38-win_amd64.whl > 4ee952f39a4a4c7ba775a32b664b1f4b74818548b65f765987adc14bb78f5802 > scipy-1.7.1-cp39-cp39-macosx_10_9_x86_64.whl > 611f9cb459d0707dd8e4de0c96f86e93f61aac7475fcb225e9ec71fecdc5cebf > scipy-1.7.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl > e101bceeb9e65a90dadbc5ca31283403a2d4667b9c178db29109750568e8d112 > scipy-1.7.1-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl > 4729b41a4cdaf4cd011aeac816b532f990bdf97710cef59149d3e293115cf467 > scipy-1.7.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl > c9951e3746b68974125e5e3445008a4163dd6d20ae0bbdae22b38cb8951dc11b > scipy-1.7.1-cp39-cp39-win32.whl > da9c6b336e540def0b7fd65603da8abeb306c5fc9a5f4238665cbbb5ff95cf58 > scipy-1.7.1-cp39-cp39-win_amd64.whl > 6b47d5fa7ea651054362561a28b1ccc8da9368a39514c1bbf6c0977a1c376764 > scipy-1.7.1.tar.gz > fdfe1d1eb1569846e331bd8d72106a8c446dafb2192c00adbb5376b02a0a1104 > scipy-1.7.1.tar.xz > 0dbea8556cb3770656770e5ed02e451dd3069c2f2f70ac885dea65f679a23afe > scipy-1.7.1.zip > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Aug 2 10:53:15 2021 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 2 Aug 2021 08:53:15 -0600 Subject: [SciPy-Dev] [Numpy-discussion] ANN: SciPy 1.7.1 In-Reply-To: References: Message-ID: On Sun, Aug 1, 2021 at 8:31 PM Tyler Reddy wrote: > Hi all, > > On behalf of the SciPy development team I'm pleased to announce > the release of SciPy 1.7.1, which is a bug fix release. > > Sources and binary wheels can be found at: > https://pypi.org/project/scipy/ > and at: https://github.com/scipy/scipy/releases/tag/v1.7.1 > > > > One of a few ways to install this release with pip: > > pip install scipy==1.7.1 > > ===================== > SciPy 1.7.1 Release Notes > ===================== > > SciPy 1.7.1 is a bug-fix release with no new features > compared to 1.7.0. > > Authors > ======= > > * Peter Bell > * Evgeni Burovski > * Justin Charlong + > * Ralf Gommers > * Matti Picus > * Tyler Reddy > * Pamphile Roy > * Sebastian Wallk?tter > * Arthur Volant > > A total of 9 people contributed to this release. > People with a "+" by their names contributed a patch for the first time. > This list of names is automatically generated, and may not be fully > complete. > > Issues closed for 1.7.1 > ------------------------------ > > * `#14074 `__: Segmentation > fault when building cKDTree with Scipy 1.6.3. > * `#14271 `__: > scipy.io.loadmat failure in 1.7.0 > * `#14273 `__: > \`scipy.signal.{medfilt,medfilt2d}\` hit "Windows fatal exception:... > * `#14282 `__: DOC, CI: > stats skewtest refguide failure > * `#14363 `__: Huge stack > allocation in _sobol.pyx may cause stack overvflow > * `#14382 `__: Memory leak > in \`scipy.spatial.distance\` for \`cdist\` > * `#14396 `__: BUG: Sphinx > 4.1 breaks the banner's logo > * `#14444 `__: DOC/FEAT > Rotation.from_rotvec documents a degrees argument which... > > Pull requests for 1.7.1 > ------------------------------ > > * `#14178 `__: DEV: Update > Boschloo Exact test > * `#14264 `__: REL: prepare > for SciPy 1.7.1 > * `#14283 `__: BUG: fix > refguide-check namedtuple handling > * `#14303 `__: FIX: Check for > None before calling str methods > * `#14327 `__: BUG: medfilt > can access beyond the end of an array > * `#14355 `__: BUG: KDTree > balanced_tree is unbalanced for degenerate data > * `#14368 `__: BUG: avoid > large cython global variable in function > * `#14384 `__: BUG: Reference > count leak in distance_pybind > * `#14397 `__: DOC/CI: do not > allow sphinx 4.1. > * `#14417 `__: DOC/CI: pin > sphinx to !=4.1.0 > * `#14460 `__: DOC: add > required scipy version to kwarg > * `#14466 `__: MAINT: 1.7.1 > backports (round 1) > * `#14508 `__: MAINT: bump > scipy-mathjax > * `#14509 `__: MAINT: 1.7.1 > backports (round 2) > > > Checksums > ========= > > MD5 > ~~~ > > ef8b44a175818183de28f2c3dacf4b74 > scipy-1.7.1-cp37-cp37m-macosx_10_9_x86_64.whl > 4f717b62946a6306bba88696b4992c73 > scipy-1.7.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl > decf6837d0a28bdeb911e6e2d18b777c > scipy-1.7.1-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.whl > 6449932605e3284f731744eb207e5612 > scipy-1.7.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl > fb94deaf9b43bf18890b0bd12fc26fda scipy-1.7.1-cp37-cp37m-win32.whl > c5894e5811278243d7f4abeb1a5f230f scipy-1.7.1-cp37-cp37m-win_amd64.whl > 3aec592a699f835319cbb4649f30df71 > scipy-1.7.1-cp38-cp38-macosx_10_9_x86_64.whl > 774a74a6c81d40c9a305523707c024f4 > scipy-1.7.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl > 30206a19a96549f665bd608fe6bf2761 > scipy-1.7.1-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl > eb58d9f3797d47866bfe571d5df3b827 > scipy-1.7.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl > 808a907d994b98fd6dbe2050a48b8c69 scipy-1.7.1-cp38-cp38-win32.whl > 688921def6681ee5abe8543aca8383c2 scipy-1.7.1-cp38-cp38-win_amd64.whl > 2fe4e958cb14d0b071c494b9faee0c98 > scipy-1.7.1-cp39-cp39-macosx_10_9_x86_64.whl > dd5b4db9cf83a0594e0b651e198b16a4 > scipy-1.7.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl > 67c5d75378d0ba2803c1b93fe670563b > scipy-1.7.1-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl > c1ea8eec1dd6dc9c1b3eae24e3b3a34a > scipy-1.7.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl > 93efd36f2c52dadbe7c9a377ad8c5be2 scipy-1.7.1-cp39-cp39-win32.whl > f8d0f87aaa8929f059fcf840db345310 scipy-1.7.1-cp39-cp39-win_amd64.whl > 8ac74369cdcabc097f602682c951197c scipy-1.7.1.tar.gz > deb130f3959e5623fafb4a262c28183b scipy-1.7.1.tar.xz > 5dd4ab895eaa141cb01b954ecae2ddfc scipy-1.7.1.zip > > SHA256 > ~~~~~~ > > 2a0eeaab01258e0870c4022a6cd329aef3b7c6c2b606bd7cf7bb2ba9820ae561 > scipy-1.7.1-cp37-cp37m-macosx_10_9_x86_64.whl > 3f52470e0548cdb74fb8ddf06773ffdcca7c97550f903b1c51312ec19243a7f7 > scipy-1.7.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl > 787749110a23502031fb1643c55a2236c99c6b989cca703ea2114d65e21728ef > scipy-1.7.1-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.whl > 3304bd5bc32e00954ac4b3f4cc382ca8824719bf348aacbec6347337d6b125fe > scipy-1.7.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl > d1388fbac9dd591ea630da75c455f4cc637a7ca5ecb31a6b6cef430914749cde > scipy-1.7.1-cp37-cp37m-win32.whl > d648aa85dd5074b1ed83008ae987c3fbb53d68af619fce1dee231f4d8bd40e2f > scipy-1.7.1-cp37-cp37m-win_amd64.whl > bc61e3e5ff92d2f32bb263621d54a9cff5e3f7c420af3d1fa122ce2529de2bd9 > scipy-1.7.1-cp38-cp38-macosx_10_9_x86_64.whl > a496b42dbcd04ea9924f5e92be63af3d8e0f43a274b769bfaca0a297327d54ee > scipy-1.7.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl > d13f31457f2216e5705304d9f28e2826edf75487410a57aa99263fa4ffd792c2 > scipy-1.7.1-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl > 90c07ba5f34f33299a428b0d4fa24c30d2ceba44d63f8385b2b05be460819fcb > scipy-1.7.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl > efdd3825d54c58df2cc394366ca4b9166cf940a0ebddeb87b6c10053deb625ea > scipy-1.7.1-cp38-cp38-win32.whl > 71cfc96297617eab911e22216e8a8597703202e95636d9406df9af5c2ac99a2b > scipy-1.7.1-cp38-cp38-win_amd64.whl > 4ee952f39a4a4c7ba775a32b664b1f4b74818548b65f765987adc14bb78f5802 > scipy-1.7.1-cp39-cp39-macosx_10_9_x86_64.whl > 611f9cb459d0707dd8e4de0c96f86e93f61aac7475fcb225e9ec71fecdc5cebf > scipy-1.7.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl > e101bceeb9e65a90dadbc5ca31283403a2d4667b9c178db29109750568e8d112 > scipy-1.7.1-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl > 4729b41a4cdaf4cd011aeac816b532f990bdf97710cef59149d3e293115cf467 > scipy-1.7.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl > c9951e3746b68974125e5e3445008a4163dd6d20ae0bbdae22b38cb8951dc11b > scipy-1.7.1-cp39-cp39-win32.whl > da9c6b336e540def0b7fd65603da8abeb306c5fc9a5f4238665cbbb5ff95cf58 > scipy-1.7.1-cp39-cp39-win_amd64.whl > 6b47d5fa7ea651054362561a28b1ccc8da9368a39514c1bbf6c0977a1c376764 > scipy-1.7.1.tar.gz > fdfe1d1eb1569846e331bd8d72106a8c446dafb2192c00adbb5376b02a0a1104 > scipy-1.7.1.tar.xz > 0dbea8556cb3770656770e5ed02e451dd3069c2f2f70ac885dea65f679a23afe > scipy-1.7.1.zip > > Thanks Tyler... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From christoph.baumgarten at gmail.com Wed Aug 4 05:24:21 2021 From: christoph.baumgarten at gmail.com (Christoph Baumgarten) Date: Wed, 4 Aug 2021 11:24:21 +0200 Subject: [SciPy-Dev] Integrating UNU.RAN in scipy.stats In-Reply-To: References: Message-ID: Hi, there was already a lot of feedback on the PR and it has been approved by two developers. To meet the GSoC deadline, we plan to merge it early next week. I just wanted to send another short mail in case someone still would like to take a look. As Tirth mentioned, any additional feedback on the approach to embed the C library is welcome. Christoph schrieb am Sa., 17. Juli 2021, 18:01: > Send SciPy-Dev mailing list submissions to > scipy-dev at python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scipy-dev > or, via email, send a message with subject or body 'help' to > scipy-dev-request at python.org > > You can reach the person managing the list at > scipy-dev-owner at python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of SciPy-Dev digest..." > > > Today's Topics: > > 1. Integrating UNU.RAN in scipy.stats (Tirth Patel) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 17 Jul 2021 01:18:24 +0530 > From: Tirth Patel > To: scipy-dev > Subject: [SciPy-Dev] Integrating UNU.RAN in scipy.stats > Message-ID: > f1iToNnRgnzM3pQDEZZT5SdJww at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi all, > > I, Christoph, and Nicholas have been working on gh-14215 [1] to integrate > UNU.RAN library in SciPy. We'd appreciate your thoughts on it! > > We have designed an object-oriented API for sampling from any continuous or > discrete distributions using universal generators in UNU.RAN. For now, only > the `TransformedDensityRejection` (for continuous distributions) and > `DiscreteAliasUrn` (for discrete distributions) methods have been added. > These methods take a distribution object with required methods like PDF, > dPDF, CDF, etc as input and set-up a generator which can then be used to > sample from that distribution: > > >>> from scipy.stats import TransformedDensityRejection > >>> import numpy as np > >>> > >>> class StdNorm: > ... def pdf(self, x: float) -> float: > ... # notice that normalization constant is not required > ... # and the pdf accepts and returns scalars. > ... return np.exp(-0.5 * x*x) > ... def dpdf(self, x: float) -> float: > ... return -x * self.pdf(x) > ... > >>> dist = StdNorm() > >>> rng = TransformedDensityRejection(dist, seed=123) > >>> rng.rvs() > 0.474548717355228 > > One of the tricky parts about this PR is handling errors occurring in > Python callbacks and UNU.RAN C library. We could use some reviews and ideas > to build a maintainable infrastructure. Use of non-local returns causes > memory leaks making the code for error handling a lot less trivial and > possibly much more complex. I would really appreciate it if someone could > take a look at the Cython and C code in the PR and help verify the approach > or suggest an alternative approach, if any. There are some discussions > about this in [2] and [3] and you can also look at the review comments on > the main PR. Although the API design is not final yet, please feel free to > comment on it as well. Thanks! > > [1]: https://github.com/scipy/scipy/pull/14215 > [2]: https://github.com/tirthasheshpatel/scipy/pull/9 > [3]: https://github.com/tirthasheshpatel/unuran/pull/1 > > > -- > Kind Regards, > Tirth Patel > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > https://mail.python.org/pipermail/scipy-dev/attachments/20210717/65b4698c/attachment-0001.html > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > > ------------------------------ > > End of SciPy-Dev Digest, Vol 213, Issue 14 > ****************************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Fri Aug 6 13:19:29 2021 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Fri, 06 Aug 2021 10:19:29 -0700 Subject: [SciPy-Dev] In-place developer installation instructions Message-ID: <9aaa1e6f-426a-487b-8185-dea9d9f6c6ca@www.fastmail.com> Hi all, I noticed that the pip developer installation instructions are missing from the documentation. The instructions are in quickstart_pip.rst, but I don't see that included in contributor/index.rst. Besides that, the instructions do not show how to build in-place and then install that SciPy so it can be used locally. I would expect something like: pip install --no-build-isolation -e . Or: python setup.py build_ext -i Then add SciPy to the PYTHONPATH. If I can get a sense of what the instructions should look like, I'd be happy to file a PR. Best regards, St?fan From ralf.gommers at gmail.com Fri Aug 6 14:21:03 2021 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 6 Aug 2021 20:21:03 +0200 Subject: [SciPy-Dev] In-place developer installation instructions In-Reply-To: <9aaa1e6f-426a-487b-8185-dea9d9f6c6ca@www.fastmail.com> References: <9aaa1e6f-426a-487b-8185-dea9d9f6c6ca@www.fastmail.com> Message-ID: On Fri, Aug 6, 2021 at 7:20 PM Stefan van der Walt wrote: > Hi all, > > I noticed that the pip developer installation instructions are missing > from the documentation. The instructions are in quickstart_pip.rst, but I > don't see that included in contributor/index.rst. > > Besides that, the instructions do not show how to build in-place and then > install that SciPy so it can be used locally. I would expect something > like: > > pip install --no-build-isolation -e . > > Or: > > python setup.py build_ext -i > > Then add SciPy to the PYTHONPATH. > > If I can get a sense of what the instructions should look like, I'd be > happy to file a PR. > Hey St?fan, `build_ext` is used in the main "development workflow" guide: http://scipy.github.io/devdocs/dev/contributor/development_workflow.html#basic-workflow, and in several other guides. We already have too many guides; there's a tracking issue about reducing that complexity to make it easier to navigate. Subtracting >= adding here. I would not recommend using `pip -e`, it's worse than `setup.py build_ext -i` or `setup.py develop`. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Sat Aug 7 16:49:59 2021 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Sat, 07 Aug 2021 13:49:59 -0700 Subject: [SciPy-Dev] In-place developer installation instructions In-Reply-To: References: <9aaa1e6f-426a-487b-8185-dea9d9f6c6ca@www.fastmail.com> Message-ID: <50882ad9-a03f-4f0d-8dbf-fae00a1b496f@www.fastmail.com> On Fri, Aug 6, 2021, at 11:21, Ralf Gommers wrote: > Hey St?fan, `build_ext` is used in the main "development workflow" guide: http://scipy.github.io/devdocs/dev/contributor/development_workflow.html#basic-workflow, and in several other guides. We already have too many guides; there's a tracking issue about reducing that complexity to make it easier to navigate. Subtracting >= adding here. Great, thanks for the pointer. We can also remove the files not used in the documentation (or were they left there for a reason? I see that one has an explicit orphan tag.) > I would not recommend using `pip -e`, it's worse than `setup.py build_ext -i` or `setup.py develop`. What caught me by surprise is that the instructions stop after the in-place build step, but without showing how to add it to the Python path (instead simply pointing to the runtest script). St?fan -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Aug 10 14:32:30 2021 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 10 Aug 2021 12:32:30 -0600 Subject: [SciPy-Dev] NumPy Python 3.10.0rc1 wheels. Message-ID: Hi All, There are now NumPy Python 3.10.0rc1 wheels available for 64 bit Linux on Intel. You can install them from the nightly builds using "pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple numpy". Test away. Cheers, Charles Harris -------------- next part -------------- An HTML attachment was scrubbed... URL: From sarafridov at gmail.com Thu Aug 12 03:05:28 2021 From: sarafridov at gmail.com (Sara Fridovich-Keil) Date: Thu, 12 Aug 2021 00:05:28 -0700 Subject: [SciPy-Dev] directional derivatives in Wolfe line search In-Reply-To: <75EFAE77-F073-4061-9AFC-074428F1BFEC@gmail.com> References: <1DE64E20-BBDA-4094-B477-9BAEFD739228@gmail.com> <9A16DF8C-1823-4857-8A8B-002B0CB057D9@gmail.com> <75EFAE77-F073-4061-9AFC-074428F1BFEC@gmail.com> Message-ID: I just sent a PR for this, looking forward to feedback :) Best, Sara > On Jul 28, 2021, at 9:50 AM, Sara Fridovich-Keil wrote: > >> The first hurdle is to ensure that correctness isn't affects. The second hurdle is to show that changes improve things. I'm not sure that we have a comprehensive test suite for these kinds of comparisons for scipy. There are functions in benchmarks/test_functions.py, but we don't have something more comprehensive like CUTEST. It would be good to have that! > > I ended up translating the Mor? and Wild CUTEST benchmark into python (from matlab) for my own experiment anyway, so I?d be happy to include that as a benchmark for scipy. > >> If I understand correctly according to that graph your implementation in orange is better than the existing scipy implementation? > > It?s a bit nuanced; it looks like the current scipy implementation is faster than my orange implementation for some problems, but there are other problems where my orange implementation converges and the current scipy gets stuck. I suspect the former is probably because of the fancier finite differencing in scipy, and the latter might be because I added a little check in the BFGS update (since the benchmark problems are nonconvex): if BFGS ever tries to take an ascent direction, I replace it with steepest descent for that step, and reset the BFGS matrix to identity. > >> I'm guessing that the modification would be: >> >> 1. If user provides a callable(jac), then use that in derphi. >> 2. If jac is estimated via a FD method then estimate derphi by finite difference along the search direction, as you suggest. Estimate grad by FD of `fun`. Both grad and derphi are calculated in line_search_wolfe1, line_search_wolfe2. > > Yes, I think this would be good. > > Best, > Sara > >> On Jul 28, 2021, at 12:38 AM, Andrew Nelson > wrote: >> >> On Wed, 28 Jul 2021 at 15:35, Sara Fridovich-Keil > wrote: >> Thanks for the input. I can describe the experiment I?ve done so far that makes me believe this simpler finite differencing would be an improvement for the DFO case, but I?m not sure if there is a standard benchmark that is used for making these decisions in scipy. >> >> The first hurdle is to ensure that correctness isn't affects. The second hurdle is to show that changes improve things. I'm not sure that we have a comprehensive test suite for these kinds of comparisons for scipy. There are functions in benchmarks/test_functions.py, but we don't have something more comprehensive like CUTEST. It would be good to have that! >> >> My experiment is on the DFO benchmark of Mor? and Wild (https://www.mcs.anl.gov/~more/dfo/ ), comparing 3 algorithms. The green curve is just calling the current implementation of fmin_bfgs. The orange curve is my own implementation of fmin_bfgs based on scipy but using a simple forward finite differencing approximation to the full gradient. >> >> If I understand correctly according to that graph your implementation in orange is better than the existing scipy implementation? >> >> The blue curve is the same as the orange one, but replacing the forward finite differencing approximation of the full gradient with a forward finite differencing approximation of just the directional derivative (inside the Wolfe line search). The sampling distance for finite differencing is 1.4901161193847656e-08 (for both the orange and blue curves), which is the same default as scipy (https://github.com/scipy/scipy/blob/v1.7.0/scipy/optimize/optimize.py#L1448 , and the value recommended in Nocedal and Wright). The blue curve is the algorithm I?m proposing; I just included the orange one to compare the simple vs sophisticated full-gradient finite differencing methods (and any differences in the outer BFGS, which should be basically the same). >> >> Note that the finite differencing for most of the optimizers is done by approx_derivative (https://github.com/scipy/scipy/blob/master/scipy/optimize/_numdiff.py#L257 ), which chooses the step size automatically. If `jac=None` for `_minimize_bfgs` then the default is for absolute steps, but you can use relative steps by specifying `jac='2-point', etc. >> >> I'm guessing that the modification would be: >> >> 1. If user provides a callable(jac), then use that in derphi. >> 2. If jac is estimated via a FD method then estimate derphi by finite difference along the search direction, as you suggest. Estimate grad by FD of `fun`. Both grad and derphi are calculated in line_search_wolfe1, line_search_wolfe2. >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mhaberla at calpoly.edu Thu Aug 12 19:49:49 2021 From: mhaberla at calpoly.edu (Matt Haberland) Date: Thu, 12 Aug 2021 16:49:49 -0700 Subject: [SciPy-Dev] Welcome Tirth Patel to the SciPy core team Message-ID: Hi all, Please welcome Tirth Patel (@tirthasheshpatel) to the core team! Tirth has been an active contributor for a little over a year, authoring and reviewing several PRs involving scipy.stats and our CI suite ( https://github.com/scipy/scipy/pulls/tirthasheshpatel). He also has several projects in the pipeline, including the long-awaited addition of UNU.RAN ( https://github.com/scipy/scipy/pull/14215). Thank you and congratulations, Tirth! Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Thu Aug 12 20:28:06 2021 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Thu, 12 Aug 2021 17:28:06 -0700 Subject: [SciPy-Dev] Welcome Tirth Patel to the SciPy core team In-Reply-To: References: Message-ID: <93f832cb-f2e1-4557-8a0d-7ae1bafc6942@www.fastmail.com> On Thu, Aug 12, 2021, at 16:49, Matt Haberland wrote: > Please welcome Tirth Patel (@tirthasheshpatel) to the core team! Tirth has been an active contributor for a little over a year, authoring and reviewing several PRs involving scipy.stats and our CI suite (https://github.com/scipy/scipy/pulls/tirthasheshpatel). He also has several projects in the pipeline, including the long-awaited addition of UNU.RAN (https://github.com/scipy/scipy/pull/14215). Welcome to the team, Tirth! ? St?fan -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Fri Aug 13 02:36:57 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Fri, 13 Aug 2021 08:36:57 +0200 Subject: [SciPy-Dev] Welcome Tirth Patel to the SciPy core team In-Reply-To: References: Message-ID: <65AD61BC-A4D4-440C-AD3A-94DF8AF860DC@gmail.com> > On 13.08.2021, at 01:49, Matt Haberland wrote: > > Hi all, > > Please welcome Tirth Patel (@tirthasheshpatel) to the core team! Tirth has been an active contributor for a little over a year, authoring and reviewing several PRs involving scipy.stats and our CI suite (https://github.com/scipy/scipy/pulls/tirthasheshpatel ). He also has several projects in the pipeline, including the long-awaited addition of UNU.RAN (https://github.com/scipy/scipy/pull/14215 ). > > Thank you and congratulations, Tirth! > Matt Welcome Tirth! It?s a pleasure to work with you and I am looking forward to see what you will bring further. Cheers, Pamphile -------------- next part -------------- An HTML attachment was scrubbed... URL: From evgeny.burovskiy at gmail.com Fri Aug 13 04:54:51 2021 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Fri, 13 Aug 2021 11:54:51 +0300 Subject: [SciPy-Dev] Welcome Tirth Patel to the SciPy core team In-Reply-To: <65AD61BC-A4D4-440C-AD3A-94DF8AF860DC@gmail.com> References: <65AD61BC-A4D4-440C-AD3A-94DF8AF860DC@gmail.com> Message-ID: Welcome Tirth! ??, 13 ???. 2021 ?., 09:37 Pamphile Roy : > > On 13.08.2021, at 01:49, Matt Haberland wrote: > > Hi all, > > Please welcome Tirth Patel (@tirthasheshpatel) to the core team! Tirth has > been an active contributor for a little over a year, authoring and > reviewing several PRs involving scipy.stats and our CI suite ( > https://github.com/scipy/scipy/pulls/tirthasheshpatel). He also has > several projects in the pipeline, including the long-awaited addition of > UNU.RAN (https://github.com/scipy/scipy/pull/14215). > > Thank you and congratulations, Tirth! > Matt > > > Welcome Tirth! > > It?s a pleasure to work with you and I am looking forward to see what you > will bring further. > > Cheers, > Pamphile > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From christoph.baumgarten at gmail.com Fri Aug 13 16:52:21 2021 From: christoph.baumgarten at gmail.com (Christoph Baumgarten) Date: Fri, 13 Aug 2021 22:52:21 +0200 Subject: [SciPy-Dev] Welcome Tirth Patel to the SciPy core team In-Reply-To: References: Message-ID: Welcome and congratulations, Tirth! It is great to work with you on the UNURAN integration. Christoph schrieb am Fr., 13. Aug. 2021, 10:55: > Send SciPy-Dev mailing list submissions to > scipy-dev at python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scipy-dev > or, via email, send a message with subject or body 'help' to > scipy-dev-request at python.org > > You can reach the person managing the list at > scipy-dev-owner at python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of SciPy-Dev digest..." > > > Today's Topics: > > 1. Welcome Tirth Patel to the SciPy core team (Matt Haberland) > 2. Re: Welcome Tirth Patel to the SciPy core team > (Stefan van der Walt) > 3. Re: Welcome Tirth Patel to the SciPy core team (Pamphile Roy) > 4. Re: Welcome Tirth Patel to the SciPy core team (Evgeni Burovski) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 12 Aug 2021 16:49:49 -0700 > From: Matt Haberland > To: SciPy Developers List > Subject: [SciPy-Dev] Welcome Tirth Patel to the SciPy core team > Message-ID: > t9Q at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi all, > > Please welcome Tirth Patel (@tirthasheshpatel) to the core team! Tirth has > been an active contributor for a little over a year, authoring and > reviewing several PRs involving scipy.stats and our CI suite ( > https://github.com/scipy/scipy/pulls/tirthasheshpatel). He also has > several > projects in the pipeline, including the long-awaited addition of UNU.RAN ( > https://github.com/scipy/scipy/pull/14215). > > Thank you and congratulations, Tirth! > Matt > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > https://mail.python.org/pipermail/scipy-dev/attachments/20210812/d9f05c93/attachment-0001.html > > > > ------------------------------ > > Message: 2 > Date: Thu, 12 Aug 2021 17:28:06 -0700 > From: "Stefan van der Walt" > To: "SciPy Developers List" > Subject: Re: [SciPy-Dev] Welcome Tirth Patel to the SciPy core team > Message-ID: <93f832cb-f2e1-4557-8a0d-7ae1bafc6942 at www.fastmail.com> > Content-Type: text/plain; charset="utf-8" > > On Thu, Aug 12, 2021, at 16:49, Matt Haberland wrote: > > Please welcome Tirth Patel (@tirthasheshpatel) to the core team! Tirth > has been an active contributor for a little over a year, authoring and > reviewing several PRs involving scipy.stats and our CI suite ( > https://github.com/scipy/scipy/pulls/tirthasheshpatel). He also has > several projects in the pipeline, including the long-awaited addition of > UNU.RAN (https://github.com/scipy/scipy/pull/14215). > > Welcome to the team, Tirth! ? > > St?fan > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > https://mail.python.org/pipermail/scipy-dev/attachments/20210812/1003ed04/attachment-0001.html > > > > ------------------------------ > > Message: 3 > Date: Fri, 13 Aug 2021 08:36:57 +0200 > From: Pamphile Roy > To: SciPy Developers List > Subject: Re: [SciPy-Dev] Welcome Tirth Patel to the SciPy core team > Message-ID: <65AD61BC-A4D4-440C-AD3A-94DF8AF860DC at gmail.com> > Content-Type: text/plain; charset="utf-8" > > > > On 13.08.2021, at 01:49, Matt Haberland wrote: > > > > Hi all, > > > > Please welcome Tirth Patel (@tirthasheshpatel) to the core team! Tirth > has been an active contributor for a little over a year, authoring and > reviewing several PRs involving scipy.stats and our CI suite ( > https://github.com/scipy/scipy/pulls/tirthasheshpatel < > https://github.com/scipy/scipy/pulls/tirthasheshpatel>). He also has > several projects in the pipeline, including the long-awaited addition of > UNU.RAN (https://github.com/scipy/scipy/pull/14215 < > https://github.com/scipy/scipy/pull/14215>). > > > > Thank you and congratulations, Tirth! > > Matt > > Welcome Tirth! > > It?s a pleasure to work with you and I am looking forward to see what you > will bring further. > > Cheers, > Pamphile > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > https://mail.python.org/pipermail/scipy-dev/attachments/20210813/b439b6c0/attachment-0001.html > > > > ------------------------------ > > Message: 4 > Date: Fri, 13 Aug 2021 11:54:51 +0300 > From: Evgeni Burovski > To: SciPy Developers List > Subject: Re: [SciPy-Dev] Welcome Tirth Patel to the SciPy core team > Message-ID: > < > CAMRo0iuiC4AGLCLXVxyPNHELtrNiCtacVivAqY2R9vCtMkHosw at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Welcome Tirth! > > ??, 13 ???. 2021 ?., 09:37 Pamphile Roy : > > > > > On 13.08.2021, at 01:49, Matt Haberland wrote: > > > > Hi all, > > > > Please welcome Tirth Patel (@tirthasheshpatel) to the core team! Tirth > has > > been an active contributor for a little over a year, authoring and > > reviewing several PRs involving scipy.stats and our CI suite ( > > https://github.com/scipy/scipy/pulls/tirthasheshpatel). He also has > > several projects in the pipeline, including the long-awaited addition of > > UNU.RAN (https://github.com/scipy/scipy/pull/14215). > > > > Thank you and congratulations, Tirth! > > Matt > > > > > > Welcome Tirth! > > > > It?s a pleasure to work with you and I am looking forward to see what you > > will bring further. > > > > Cheers, > > Pamphile > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at python.org > > https://mail.python.org/mailman/listinfo/scipy-dev > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > https://mail.python.org/pipermail/scipy-dev/attachments/20210813/b0edf89b/attachment.html > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > > ------------------------------ > > End of SciPy-Dev Digest, Vol 214, Issue 10 > ****************************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Aug 15 16:41:02 2021 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 15 Aug 2021 14:41:02 -0600 Subject: [SciPy-Dev] NumPy 1.21.2 released Message-ID: Hi All, On behalf of the NumPy team I am pleased to announce the release of NumPy 1.21.2. NumPy 1.21.2 is a maintenance release that fixes bugs discovered after 1.21.1. It also provides 64 bit manylinux Python 3.10.0rc1 wheels for downstream testing. Note that Python 3.10 is not yet final. There is also preliminary support for Windows on ARM64 builds, but there is no OpenBLAS for that platform and no wheels are available. The Python versions supported for this release are 3.7-3.9. The 1.21.2 release is compatible with Python 3.10.0rc1 and Python 3.10 will be officially supported after it is released. The previous problems with gcc-11.1 have been fixed by gcc-11.2, check your version if you are using gcc-11. Wheels can be downloaded from PyPI ; source archives, release notes, and wheel hashes are available on Github . Linux users will need pip >= 0.19.3 in order to install manylinux2010 and manylinux2014 wheels. *Contributors* A total of 10 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Bas van Beek - Carl Johnsen + - Charles Harris - Gwyn Ciesla + - Matthieu Dartiailh - Matti Picus - Niyas Sait + - Ralf Gommers - Sayed Adel - Sebastian Berg *Pull requests merged* A total of 18 pull requests were merged for this release. - #19497: MAINT: set Python version for 1.21.x to ``<3.11`` - #19533: BUG: Fix an issue wherein importing ``numpy.typing`` could raise - #19646: MAINT: Update Cython version for Python 3.10. - #19648: TST: Bump the python 3.10 test version from beta4 to rc1 - #19651: TST: avoid distutils.sysconfig in runtests.py - #19652: MAINT: add missing dunder method to nditer type hints - #19656: BLD, SIMD: Fix testing extra checks when ``-Werror`` isn't applicable... - #19657: BUG: Remove logical object ufuncs with bool output - #19658: MAINT: Include .coveragerc in source distributions to support... - #19659: BUG: Fix bad write in masked iterator output copy paths - #19660: ENH: Add support for windows on arm targets - #19661: BUG: add base to templated arguments for platlib - #19662: BUG,DEP: Non-default UFunc signature/dtype usage should be deprecated - #19666: MAINT: Add Python 3.10 to supported versions. - #19668: TST,BUG: Sanitize path-separators when running ``runtest.py`` - #19671: BLD: load extra flags when checking for libflame - #19676: BLD: update circleCI docker image - #19677: REL: Prepare for 1.21.2 release. Cheers Charles Harris -------------- next part -------------- An HTML attachment was scrubbed... URL: From dparsons at debian.org Tue Aug 17 13:02:12 2021 From: dparsons at debian.org (Drew Parsons) Date: Tue, 17 Aug 2021 19:02:12 +0200 Subject: [SciPy-Dev] SciPy and pybind11 2.7 Message-ID: Hi SciPy developers, the release version of SciPy gives upper version caps on dependencies, e.g. pybind11>=2.4.3,<2.7.0 The latest pybind11 release is now 2.7.1. Is there a specific reason why SciPy adds the version cap to releases, i.e. a specific known point of failure? Or is the version cap simply used conservatively to help maintain release stability for users? Cheers, Drew Parsons From ralf.gommers at gmail.com Tue Aug 17 14:16:07 2021 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 17 Aug 2021 20:16:07 +0200 Subject: [SciPy-Dev] SciPy and pybind11 2.7 In-Reply-To: References: Message-ID: On Tue, Aug 17, 2021 at 7:02 PM Drew Parsons wrote: > Hi SciPy developers, > > the release version of SciPy gives upper version caps on dependencies, > e.g. pybind11>=2.4.3,<2.7.0 > > The latest pybind11 release is now 2.7.1. Is there a specific reason > why SciPy adds the version cap to releases, i.e. a specific known point > of failure? Or is the version cap simply used conservatively to help > maintain release stability for users? > There is no specific problem, it's just a matter of controlling the upgrade timeline so there is no risk of a new pybind11 release breaking an already released version of SciPy. Because it's a build-only dependency, that is usually the right strategy (runtime dependencies are a bit different, because the upper bound may bother users). If you want to bump the version in master, please let us know or open a PR. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From dparsons at debian.org Wed Aug 18 09:07:56 2021 From: dparsons at debian.org (Drew Parsons) Date: Wed, 18 Aug 2021 15:07:56 +0200 Subject: [SciPy-Dev] SciPy and pybind11 2.7 In-Reply-To: References: Message-ID: <319d1f1c552017520d03b74d122fb0d7@debian.org> On 2021-08-17 20:16, Ralf Gommers wrote: > On Tue, Aug 17, 2021 at 7:02 PM Drew Parsons > wrote: > >> Hi SciPy developers, >> >> the release version of SciPy gives upper version caps on >> dependencies, >> e.g. pybind11>=2.4.3,<2.7.0 >> >> The latest pybind11 release is now 2.7.1. Is there a specific >> reason >> why SciPy adds the version cap to releases, i.e. a specific known >> point >> of failure? Or is the version cap simply used conservatively to >> help >> maintain release stability for users? > > There is no specific problem, it's just a matter of controlling the > upgrade timeline so there is no risk of a new pybind11 release > breaking an already released version of SciPy. Because it's a > build-only dependency, that is usually the right strategy (runtime > dependencies are a bit different, because the upper bound may bother > users). > > If you want to bump the version in master, please let us know or open > a PR. > Thanks Ralf. I had a feeling that in principle scipy could build fine with pybind11 2.7, since the build wasn't reporting any errors. I just wanted assurance that there weren't any known but unseen problems. My build system (Debian) can happily ignore the recommended upper limit, so I've now uploaded scipy 1.7.1 to debian unstable, see https://buildd.debian.org/status/package.php?p=scipy Hence, no urgency on my account to bump the version in master, do it once you're comfortable that scipy is fine with pybind11 2.7. We can consider Debian unstable as a test environment for it :) Cheers, Drew From ralf.gommers at gmail.com Thu Aug 19 17:15:14 2021 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 19 Aug 2021 23:15:14 +0200 Subject: [SciPy-Dev] SciPy and pybind11 2.7 In-Reply-To: <319d1f1c552017520d03b74d122fb0d7@debian.org> References: <319d1f1c552017520d03b74d122fb0d7@debian.org> Message-ID: On Wed, Aug 18, 2021 at 3:07 PM Drew Parsons wrote: > On 2021-08-17 20:16, Ralf Gommers wrote: > > On Tue, Aug 17, 2021 at 7:02 PM Drew Parsons > > wrote: > > > >> Hi SciPy developers, > >> > >> the release version of SciPy gives upper version caps on > >> dependencies, > >> e.g. pybind11>=2.4.3,<2.7.0 > >> > >> The latest pybind11 release is now 2.7.1. Is there a specific > >> reason > >> why SciPy adds the version cap to releases, i.e. a specific known > >> point > >> of failure? Or is the version cap simply used conservatively to > >> help > >> maintain release stability for users? > > > > There is no specific problem, it's just a matter of controlling the > > upgrade timeline so there is no risk of a new pybind11 release > > breaking an already released version of SciPy. Because it's a > > build-only dependency, that is usually the right strategy (runtime > > dependencies are a bit different, because the upper bound may bother > > users). > > > > If you want to bump the version in master, please let us know or open > > a PR. > > > > > Thanks Ralf. I had a feeling that in principle scipy could build fine > with pybind11 2.7, since the build wasn't reporting any errors. I just > wanted assurance that there weren't any known but unseen problems. > > My build system (Debian) can happily ignore the recommended upper limit, > so I've now uploaded scipy 1.7.1 to debian unstable, see > https://buildd.debian.org/status/package.php?p=scipy > Great, thanks. Hence, no urgency on my account to bump the version in master, do it > once you're comfortable that scipy is fine with pybind11 2.7. We can > consider Debian unstable as a test environment for it :) > It should be fine. In master the dependencies are actually unpinned, so we have been testing it already. We just put an upper bound in for releases, so those cannot be broken by a future release of some dependency. I bumped to <2.8.0 for 1.7.2 in https://github.com/scipy/scipy/pull/14616. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From teddy at mit.edu Fri Aug 20 20:49:58 2021 From: teddy at mit.edu (Teddy Ort) Date: Fri, 20 Aug 2021 20:49:58 -0400 Subject: [SciPy-Dev] New PR for scipy.stats.binned_statistic Message-ID: Hi Scipy! I've made a small change to scipy.stats.binned_statistic that improves performance on the min, max, and median statistics. https://github.com/scipy/scipy/pull/14625 This is my first PR. I've tried to follow the checklist, but please let me know if I've missed something. Best, Teddy -------------- next part -------------- An HTML attachment was scrubbed... URL: From teddy at mit.edu Sun Aug 22 08:00:00 2021 From: teddy at mit.edu (Teddy Ort) Date: Sun, 22 Aug 2021 08:00:00 -0400 Subject: [SciPy-Dev] PR to update scipy.stats.binned_statistic Message-ID: Hi Scipy! https://github.com/scipy/scipy/pull/14629 is a PR that optimizes the *std* statistic for *scipy.stats.binned_statistic*. I recently submitted another PR that does the same for *min*, *max*, and *median*, and have since realized how to do something similar for *std.* Thanks for looking! Teddy -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyler.je.reddy at gmail.com Tue Aug 24 10:05:44 2021 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Tue, 24 Aug 2021 08:05:44 -0600 Subject: [SciPy-Dev] On the topic of separate merge commits Message-ID: Hi all, I recently received some criticism for merging a PR that had only 1 commit and producing a single merge commit on top of it (this happens to be the default option I have/what I've been doing for years). See here: https://github.com/scipy/scipy/pull/14630 I'd like to mention a few things here: - are we still "ok" with folks doing this? frankly, I find taking heat for precise merging philosophy a bit unwelcoming even as someone who has been contributing for a while; sometimes it is important to squash 100 messy commits, but otherwise do a good job reviewing PRs and relaxing on the purist "git history" attacks seem a bit more welcoming to me - I find those merge commits somewhat useful when navigating back to GitHub anyway, maybe because I'm so used to them - why does GitHub allow us to do this if it really isn't useful at all? Tyler -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.guyer at nist.gov Tue Aug 24 10:25:50 2021 From: jonathan.guyer at nist.gov (Guyer, Jonathan E. Dr. (Fed)) Date: Tue, 24 Aug 2021 14:25:50 +0000 Subject: [SciPy-Dev] On the topic of separate merge commits In-Reply-To: References: Message-ID: Git and GitHub allow a plethora of modalities. Many of them quite ill-advised. Others simply suitable for different development strategies. It?s up to the project to decide how it wants to work. I express no opinion about merge commits, but I found your response in that thread over the top. On Aug 24, 2021, at 10:05 AM, Tyler Reddy > wrote: Hi all, I recently received some criticism for merging a PR that had only 1 commit and producing a single merge commit on top of it (this happens to be the default option I have/what I've been doing for years). See here: https://github.com/scipy/scipy/pull/14630 I'd like to mention a few things here: - are we still "ok" with folks doing this? frankly, I find taking heat for precise merging philosophy a bit unwelcoming even as someone who has been contributing for a while; sometimes it is important to squash 100 messy commits, but otherwise do a good job reviewing PRs and relaxing on the purist "git history" attacks seem a bit more welcoming to me - I find those merge commits somewhat useful when navigating back to GitHub anyway, maybe because I'm so used to them - why does GitHub allow us to do this if it really isn't useful at all? Tyler _______________________________________________ SciPy-Dev mailing list SciPy-Dev at python.org https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Tue Aug 24 10:31:39 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Tue, 24 Aug 2021 16:31:39 +0200 Subject: [SciPy-Dev] On the topic of separate merge commits In-Reply-To: References: Message-ID: > On 24.08.2021, at 16:05, Tyler Reddy wrote: > > Hi all, > > I recently received some criticism for merging a PR that had only 1 commit and producing a single merge commit on top of it (this happens to be the default option I have/what I've been doing for years). See here: https://github.com/scipy/scipy/pull/14630 > > I'd like to mention a few things here: > - are we still "ok" with folks doing this? frankly, I find taking heat for precise merging philosophy a bit unwelcoming even as someone who has been contributing for a while; sometimes it is important to squash 100 messy commits, but otherwise do a good job reviewing PRs and relaxing on the purist "git history" attacks seem a bit more welcoming to me > - I find those merge commits somewhat useful when navigating back to GitHub anyway, maybe because I'm so used to them > - why does GitHub allow us to do this if it really isn't useful at all? Hi Tyler, I am sorry that you saw an attack in my comment. That was not my intention and I strongly think that it is not what my message and replies convey. The essence of my comment was: why merge instead of squash for a single commit. I asked you directly for 2 reasons: one, it?s on one of my PR; two, I saw you do this on multiple PRs. Hence I thought you would be the best placed to answer my interrogations. As I said, if it?s for the authorship and presence in the history (which I totally agree, when reviewing you should be in the history. I even opened a feature request on GitHub: https://github.com/github/feedback/discussions/4525 ), then I suggest to use the ?Co-authored-by? feature. As for the default, GitHub is considering the last action you did. If you merged a PR, next time it will show you ?Merge Pull Request?. In my case for instance, I last ?Squash and merge? and GitHub is showing me this. You can just click on the arrow to change this. Pamphile -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot 2021-08-24 at 16.25.22.png Type: image/png Size: 12103 bytes Desc: not available URL: From tyler.je.reddy at gmail.com Tue Aug 24 11:50:35 2021 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Tue, 24 Aug 2021 09:50:35 -0600 Subject: [SciPy-Dev] On the topic of separate merge commits In-Reply-To: References: Message-ID: > Hence I thought you would be the best placed to answer my interrogations. I am certainly not the appropriate person to target for "interrogations" on this matter. It is a long-standing practice in the community, and a feature GitHub has in their GUI to create separate merge commits even when there is only a single commit in the PR. If SciPy or the community strongly prefers not to do it, then I'll switch to squashing, but until then I strongly disagree that I should have to defend PRs where I do that. Tyler On Tue, 24 Aug 2021 at 08:32, Pamphile Roy wrote: > > > On 24.08.2021, at 16:05, Tyler Reddy wrote: > > Hi all, > > I recently received some criticism for merging a PR that had only 1 commit > and producing a single merge commit on top of it (this happens to be the > default option I have/what I've been doing for years). See here: > https://github.com/scipy/scipy/pull/14630 > > I'd like to mention a few things here: > - are we still "ok" with folks doing this? frankly, I find taking heat for > precise merging philosophy a bit unwelcoming even as someone who has been > contributing for a while; sometimes it is important to squash 100 messy > commits, but otherwise do a good job reviewing PRs and relaxing on the > purist "git history" attacks seem a bit more welcoming to me > - I find those merge commits somewhat useful when navigating back to > GitHub anyway, maybe because I'm so used to them > - why does GitHub allow us to do this if it really isn't useful at all? > > > Hi Tyler, > > I am sorry that you saw an attack in my comment. > > That was not my intention and I strongly think that it is not what my > message and replies convey. > > The essence of my comment was: why merge instead of squash for a single > commit. I asked you directly for 2 reasons: one, it?s on one of my PR; two, > I saw you do this on multiple PRs. > Hence I thought you would be the best placed to answer my interrogations. > > As I said, if it?s for the authorship and presence in the history (which I > totally agree, when reviewing you should be in the history. I even opened a > feature request on GitHub: > https://github.com/github/feedback/discussions/4525), > then I suggest to use the ?Co-authored-by? feature. > > As for the default, GitHub is considering the last action you did. If you > merged a PR, next time it will show you ?Merge Pull Request?. > In my case for instance, I last ?Squash and merge? and GitHub is showing > me this. You can just click on the arrow to change this. > > > Pamphile > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot 2021-08-24 at 16.25.22.png Type: image/png Size: 12103 bytes Desc: not available URL: From matthew.brett at gmail.com Tue Aug 24 13:45:16 2021 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 24 Aug 2021 18:45:16 +0100 Subject: [SciPy-Dev] On the topic of separate merge commits In-Reply-To: References: Message-ID: Hi, On Tue, Aug 24, 2021 at 4:51 PM Tyler Reddy wrote: > > > Hence I thought you would be the best placed to answer my interrogations. > > I am certainly not the appropriate person to target for "interrogations" on this matter. It is a long-standing practice in the community, and a feature GitHub has in their GUI to create separate merge commits even when there is only a single commit in the PR. > > If SciPy or the community strongly prefers not to do it, then I'll switch to squashing, but until then I strongly disagree that I should have to defend PRs where I do that. I must say - that's what I tend to do (merge, not squash). I can see arguments for squashing, if the history is very messy, but if the history is good, and the commits are well organized, squashing seems like the wrong thing to do. Why not leave it to the judgement of the person doing the merge? Cheers, Matthew From evgeny.burovskiy at gmail.com Wed Aug 25 07:14:29 2021 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Wed, 25 Aug 2021 14:14:29 +0300 Subject: [SciPy-Dev] On the topic of separate merge commits In-Reply-To: References: Message-ID: > On Tue, Aug 24, 2021 at 4:51 PM Tyler Reddy wrote: > > > > > Hence I thought you would be the best placed to answer my interrogations. > > > > I am certainly not the appropriate person to target for "interrogations" on this matter. It is a long-standing practice in the community, and a feature GitHub has in their GUI to create separate merge commits even when there is only a single commit in the PR. > > > > If SciPy or the community strongly prefers not to do it, then I'll switch to squashing, but until then I strongly disagree that I should have to defend PRs where I do that. > > I must say - that's what I tend to do (merge, not squash). I can see > arguments for squashing, if the history is very messy, but if the > history is good, and the commits are well organized, squashing seems > like the wrong thing to do. Why not leave it to the judgement of the > person doing the merge? I'd second what Mattew said. (I would also create a merge commit without much thinking, btw). There are clear limiting cases (messy history; a single 5kLOC commit etc), but otherwise it's really a judgement call of a person who does the merge. While Github does offer a bunch of options, deciding to just not think about the merge mode is a perfectly valid strategy. I personally don't think we need to standardize it, or even think about it too much. In the project of our size, there's going to be some variation between maintainers, and it's likely OK. One person prefers to squash, the other prefers a merge commit --- and it's fine IMO. Cheers, Evgeni From ralf.gommers at gmail.com Wed Aug 25 09:48:39 2021 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 25 Aug 2021 15:48:39 +0200 Subject: [SciPy-Dev] In-place developer installation instructions In-Reply-To: <50882ad9-a03f-4f0d-8dbf-fae00a1b496f@www.fastmail.com> References: <9aaa1e6f-426a-487b-8185-dea9d9f6c6ca@www.fastmail.com> <50882ad9-a03f-4f0d-8dbf-fae00a1b496f@www.fastmail.com> Message-ID: On Sat, Aug 7, 2021 at 10:51 PM Stefan van der Walt wrote: > On Fri, Aug 6, 2021, at 11:21, Ralf Gommers wrote: > > Hey St?fan, `build_ext` is used in the main "development workflow" guide: > http://scipy.github.io/devdocs/dev/contributor/development_workflow.html#basic-workflow, > and in several other guides. We already have too many guides; there's a > tracking issue about reducing that complexity to make it easier to > navigate. Subtracting >= adding here. > > > Great, thanks for the pointer. We can also remove the files not used in > the documentation (or were they left there for a reason? I see that one has > an explicit orphan tag.) > I'm not sure which files those are. Some orphan tags are on purpose (like for class methods/attributes IIRC), but not for narrative docs AFAIK. > > I would not recommend using `pip -e`, it's worse than `setup.py build_ext > -i` or `setup.py develop`. > > > What caught me by surprise is that the instructions stop after the > in-place build step, but without showing how to add it to the Python path > (instead simply pointing to the runtest script). > Yes, that is worth fixing. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From larson.eric.d at gmail.com Wed Aug 25 10:04:24 2021 From: larson.eric.d at gmail.com (Eric Larson) Date: Wed, 25 Aug 2021 10:04:24 -0400 Subject: [SciPy-Dev] On the topic of separate merge commits In-Reply-To: References: Message-ID: I'm okay with each maintainer following their preference, as there are arguments either way. That said, if people don't care much either way or want to think about it, I'd advocate for "squash and merge" as the default policy. I started using it (whenever they first introduced it years ago) across multiple repos with no observed practical drawbacks so far. On the other hand, it has improved maintainability in at least the following ways: 1) It makes `git bisect` easier as we are (almost) guaranteed that tests worked before the commit and after the commit. It's not 100% guaranteed because maintainers might miss a red CI/failing test in a PR, but this is rare. When we merge without squashing, intermediate commits often have errors. Squash+merge means there is no need for skipping steps in the bisect, all can be good/bad. 2) It keeps history smaller. 3) It makes a nicer/more direct mapping between commits to scipy:master and PRs. 4) Because of 1-3, browsing commits and running `git bisect` are faster. The only real disadvantage as I see it is that individual commits are lost (the flipside of (2)), so if these are really helpful being split for some reason, you'd have to go back to the original PR / branch to see them (assuming the branch is not deleted). For what it's worth, though, I have not run into a case where split commits are actually helpful across many `git bisect`s and many PRs across repos, so to me it's rare enough not to outweigh the benefits. The few remaining times I do a plain `merge` is when I see that the commits in the PR all have green CIs (or at least were likely to if they had run) and probably could have been individual PRs on their own (e.g., "move function", "fix bug", "add enhancement" all as part of making one thing more usable in a single PR). Some contributors are very good about structuring/restructuring their PRs this way, but it seems to be the (rare) exception and not the rule. If nobody objects to this line of reasoning, it might be worth putting a "if you don't care, make your default squash + merge and use it; if you do care, use your best judgment" in the maintainers guide somewhere. If people don't buy my arguments, then we should just put "use your best judgment". My 2c, Eric On Wed, Aug 25, 2021 at 7:15 AM Evgeni Burovski wrote: > > On Tue, Aug 24, 2021 at 4:51 PM Tyler Reddy > wrote: > > > > > > > Hence I thought you would be the best placed to answer my > interrogations. > > > > > > I am certainly not the appropriate person to target for > "interrogations" on this matter. It is a long-standing practice in the > community, and a feature GitHub has in their GUI to create separate merge > commits even when there is only a single commit in the PR. > > > > > > If SciPy or the community strongly prefers not to do it, then I'll > switch to squashing, but until then I strongly disagree that I should have > to defend PRs where I do that. > > > > I must say - that's what I tend to do (merge, not squash). I can see > > arguments for squashing, if the history is very messy, but if the > > history is good, and the commits are well organized, squashing seems > > like the wrong thing to do. Why not leave it to the judgement of the > > person doing the merge? > > I'd second what Mattew said. > (I would also create a merge commit without much thinking, btw). > There are clear limiting cases (messy history; a single 5kLOC commit > etc), but otherwise it's really a judgement call of a person who does > the merge. > While Github does offer a bunch of options, deciding to just not think > about the merge mode is a perfectly valid strategy. I personally don't > think we need to standardize it, or even think about it too much. > In the project of our size, there's going to be some variation between > maintainers, and it's likely OK. One person prefers to squash, the > other prefers a merge commit --- and it's fine IMO. > > Cheers, > > Evgeni > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Aug 25 10:29:57 2021 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 25 Aug 2021 10:29:57 -0400 Subject: [SciPy-Dev] On the topic of separate merge commits In-Reply-To: References: Message-ID: On Wed, Aug 25, 2021 at 10:04 AM Eric Larson wrote: > I'm okay with each maintainer following their preference, as there are > arguments either way. > > That said, if people don't care much either way or want to think about it, > I'd advocate for "squash and merge" as the default policy. I started using > it (whenever they first introduced it years ago) across multiple repos with > no observed practical drawbacks so far. On the other hand, it has improved > maintainability in at least the following ways: > > 1) It makes `git bisect` easier as we are (almost) guaranteed that tests > worked before the commit and after the commit. It's not 100% guaranteed > because maintainers might miss a red CI/failing test in a PR, but this is > rare. When we merge without squashing, intermediate commits often have > errors. Squash+merge means there is no need for skipping steps in the > bisect, all can be good/bad. > 2) It keeps history smaller. > 3) It makes a nicer/more direct mapping between commits to scipy:master > and PRs. > 4) Because of 1-3, browsing commits and running `git bisect` are faster. > For similar reasons we do the opposite in statsmodels. only merging pull requests is allowed. The commit history clearly marks merges by "Merge pull request ...." and is easy to search. intermediate commits can be ignored for maintenance, but are important to understand and bughunt the code. Jowd > > The only real disadvantage as I see it is that individual commits are lost > (the flipside of (2)), so if these are really helpful being split for some > reason, you'd have to go back to the original PR / branch to see them > (assuming the branch is not deleted). For what it's worth, though, I have > not run into a case where split commits are actually helpful across many > `git bisect`s and many PRs across repos, so to me it's rare enough not to > outweigh the benefits. > > The few remaining times I do a plain `merge` is when I see that the > commits in the PR all have green CIs (or at least were likely to if they > had run) and probably could have been individual PRs on their own (e.g., > "move function", "fix bug", "add enhancement" all as part of making one > thing more usable in a single PR). Some contributors are very good about > structuring/restructuring their PRs this way, but it seems to be the (rare) > exception and not the rule. > > If nobody objects to this line of reasoning, it might be worth putting a > "if you don't care, make your default squash + merge and use it; if you do > care, use your best judgment" in the maintainers guide somewhere. If people > don't buy my arguments, then we should just put "use your best judgment". > > My 2c, > Eric > > > On Wed, Aug 25, 2021 at 7:15 AM Evgeni Burovski < > evgeny.burovskiy at gmail.com> wrote: > >> > On Tue, Aug 24, 2021 at 4:51 PM Tyler Reddy >> wrote: >> > > >> > > > Hence I thought you would be the best placed to answer my >> interrogations. >> > > >> > > I am certainly not the appropriate person to target for >> "interrogations" on this matter. It is a long-standing practice in the >> community, and a feature GitHub has in their GUI to create separate merge >> commits even when there is only a single commit in the PR. >> > > >> > > If SciPy or the community strongly prefers not to do it, then I'll >> switch to squashing, but until then I strongly disagree that I should have >> to defend PRs where I do that. >> > >> > I must say - that's what I tend to do (merge, not squash). I can see >> > arguments for squashing, if the history is very messy, but if the >> > history is good, and the commits are well organized, squashing seems >> > like the wrong thing to do. Why not leave it to the judgement of the >> > person doing the merge? >> >> I'd second what Mattew said. >> (I would also create a merge commit without much thinking, btw). >> There are clear limiting cases (messy history; a single 5kLOC commit >> etc), but otherwise it's really a judgement call of a person who does >> the merge. >> While Github does offer a bunch of options, deciding to just not think >> about the merge mode is a perfectly valid strategy. I personally don't >> think we need to standardize it, or even think about it too much. >> In the project of our size, there's going to be some variation between >> maintainers, and it's likely OK. One person prefers to squash, the >> other prefers a merge commit --- and it's fine IMO. >> >> Cheers, >> >> Evgeni >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Wed Aug 25 12:05:42 2021 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Wed, 25 Aug 2021 09:05:42 -0700 Subject: [SciPy-Dev] On the topic of separate merge commits In-Reply-To: References: Message-ID: On Wed, Aug 25, 2021, at 07:04, Eric Larson wrote: > Some contributors are very good about structuring/restructuring their PRs this way, but it seems to be the (rare) exception and not the rule. This is in line with my experience. If you have developers who are willing to craft their patches very carefully, a merge strategy makes sense. Otherwise, since PRs are meant to be focused in scope already, squashing works well in practice. Github also implicitly encourages an append-to-PR workflow. They have "view only what has changed" when you only append patches, and if I'm not mistaken cannot always track comments across rebase. Perhaps most importantly, I think it is unreasonable to expect newcomers to rebase their work when merge is so much easier. So, for those PRs we should squash anyway. Eric, I agree with what you wrote. For `git bisect` the `--first-parent` flag could be useful in only evaluating commits on the root of the commit tree. St?fan -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Wed Aug 25 14:34:30 2021 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 25 Aug 2021 19:34:30 +0100 Subject: [SciPy-Dev] On the topic of separate merge commits In-Reply-To: References: Message-ID: Hi, On Wed, Aug 25, 2021 at 5:06 PM Stefan van der Walt wrote: > > On Wed, Aug 25, 2021, at 07:04, Eric Larson wrote: > > Some contributors are very good about structuring/restructuring their PRs this way, but it seems to be the (rare) exception and not the rule. > > > This is in line with my experience. If you have developers who are willing to craft their patches very carefully, a merge strategy makes sense. Otherwise, since PRs are meant to be focused in scope already, squashing works well in practice. > > Github also implicitly encourages an append-to-PR workflow. They have "view only what has changed" when you only append patches, and if I'm not mistaken cannot always track comments across rebase. > > Perhaps most importantly, I think it is unreasonable to expect newcomers to rebase their work when merge is so much easier. So, for those PRs we should squash anyway. > > Eric, I agree with what you wrote. For `git bisect` the `--first-parent` flag could be useful in only evaluating commits on the root of the commit tree. Surely the `--first-parent` option to git bisect switches the argument back in favor of merge rather than squash, in that you can get the advantages of squash from that flag, without the disadvantage of having to go back to the pull request history on the Github interface, or by recovering the PR history somehow. It's surely true that many less experienced developers don't give a clean history, but for those that do, it's a clear advantage to have it available with a standard checkout. So it still seems to me that a simple 'merge or squash, it's your call' policy is the best one. Cheers, Matthew From ralf.gommers at gmail.com Wed Aug 25 16:30:08 2021 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 25 Aug 2021 22:30:08 +0200 Subject: [SciPy-Dev] On the topic of separate merge commits In-Reply-To: References: Message-ID: On Wed, Aug 25, 2021 at 8:35 PM Matthew Brett wrote: > Hi, > > On Wed, Aug 25, 2021 at 5:06 PM Stefan van der Walt > wrote: > > > > On Wed, Aug 25, 2021, at 07:04, Eric Larson wrote: > > > > Some contributors are very good about structuring/restructuring their > PRs this way, but it seems to be the (rare) exception and not the rule. > > > > > > This is in line with my experience. If you have developers who are > willing to craft their patches very carefully, a merge strategy makes > sense. Otherwise, since PRs are meant to be focused in scope already, > squashing works well in practice. > > > > Github also implicitly encourages an append-to-PR workflow. They have > "view only what has changed" when you only append patches, and if I'm not > mistaken cannot always track comments across rebase. > > > > Perhaps most importantly, I think it is unreasonable to expect newcomers > to rebase their work when merge is so much easier. So, for those PRs we > should squash anyway. > > > > Eric, I agree with what you wrote. For `git bisect` the > `--first-parent` flag could be useful in only evaluating commits on the > root of the commit tree. > > Surely the `--first-parent` option to git bisect switches the argument > back in favor of merge rather than squash, in that you can get the > advantages of squash from that flag, without the disadvantage of > having to go back to the pull request history on the Github interface, > or by recovering the PR history somehow. > > It's surely true that many less experienced developers don't give a > clean history, but for those that do, it's a clear advantage to have > it available with a standard checkout. > > So it still seems to me that a simple 'merge or squash, it's your > call' policy is the best one. > I agree with this, the maintainer should apply some judgement. There are cases where it's a toss-up (like single-commit PRs), there are messy PRs that clearly need squashing, and there are PRs with good granular commits that should be preserved. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Thu Aug 26 15:24:32 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Thu, 26 Aug 2021 21:24:32 +0200 Subject: [SciPy-Dev] ENH: add Lloyd's algorithm Message-ID: Hi everyone, I opened a PR which proposes to add Lloyd?s algorithm. https://github.com/scipy/scipy/pull/14642 The TL;DR is: Lloyd's algorithm, or Central Voronoi Tesselation (CVT), post-process a sample to uniformize the inter-sample distances. It's an iterative process. From a sample, the Voronoi cells are calculated, then each sample is moved toward the centroid of its cell. Initially I though about adding this to scipy.stats.qmc as I intended to use this to improve a sample. But Matt rightly pointed out that this could just be put in scipy.spatial. What do you think? Thanks in advance for your inputs! Cheers, Pamphile -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyler.je.reddy at gmail.com Thu Aug 26 16:24:59 2021 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Thu, 26 Aug 2021 14:24:59 -0600 Subject: [SciPy-Dev] ENH: add Lloyd's algorithm In-Reply-To: References: Message-ID: It may be good to clarify why it goes in SciPy vs. say scikit-learn next to k-means clustering. Their docs for k-means ( https://scikit-learn.org/stable/modules/clustering.html#k-means ) even mention several things that sound very similar to what I saw in the PR when I took a quick look: "K-means is often referred to as Lloyd?s algorithm." and the description of the algo steps sounds very similar: "The algorithm can also be understood through the concept of Voronoi diagrams . First the Voronoi diagram of the points is calculated using the current centroids. Each segment in the Voronoi diagram becomes a separate cluster. Secondly, the centroids are updated to the mean of each segment. The algorithm then repeats this until a stopping criterion is fulfilled." Anyway, not saying SciPy shouldn't take it, but if it isn't tightly coupled to the stats qmc stuff, it might be well suited to be placed next to its closest siblings in the ecosystem? If it is mostly for qmc *internals* in SciPy and we can't depend on scikit-learn, I suppose that might be a reason, though if qmc only could it be private and used as a flag/option where needed in qmc? If it is used for i.e., *input* to qmc, or on the *output* from qmc stuff, then having an end-user use scikit-learn seems less of an ask. Tyler On Thu, 26 Aug 2021 at 13:25, Pamphile Roy wrote: > Hi everyone, > > I opened a PR which proposes to add Lloyd?s algorithm. > > https://github.com/scipy/scipy/pull/14642 > > The TL;DR is: Lloyd's algorithm, or Central Voronoi Tesselation (CVT), > post-process a sample to uniformize the inter-sample distances. > It's an iterative process. From a sample, the Voronoi cells are > calculated, then each sample is moved toward the centroid of its cell. > > Initially I though about adding this to scipy.stats.qmc as I intended to > use this to improve a sample. > But Matt rightly pointed out that this could just be put in scipy.spatial. > > What do you think? > > Thanks in advance for your inputs! > > Cheers, > Pamphile > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From deak.andris at gmail.com Thu Aug 26 16:41:54 2021 From: deak.andris at gmail.com (Andras Deak) Date: Thu, 26 Aug 2021 22:41:54 +0200 Subject: [SciPy-Dev] ENH: add Lloyd's algorithm In-Reply-To: References: Message-ID: On Thu, Aug 26, 2021 at 10:25 PM Tyler Reddy wrote: > > It may be good to clarify why it goes in SciPy vs. say scikit-learn next to k-means clustering. Their docs for k-means ( https://scikit-learn.org/stable/modules/clustering.html#k-means ) even mention several things that sound > very similar to what I saw in the PR when I took a quick look: > > "K-means is often referred to as Lloyd?s algorithm." > > and the description of the algo steps sounds very similar: > > "The algorithm can also be understood through the concept of Voronoi diagrams. First the Voronoi diagram of the points is calculated using the current centroids. Each segment in the Voronoi diagram becomes a separate cluster. Secondly, the centroids are updated to the mean of each segment. The algorithm then repeats this until a stopping criterion is fulfilled." That sounds like (ab)using K-means clustering by requesting n clusters for n points: you want each point to be its own "cluster" so that they each get updated. Except I suspect that this could mess with stopping criteria (assuming there's one for the makeup of each cluster in each iteration). I wonder if there would also be overhead from targetting what is probably a pathological case for clustering. I guess it sounds more like that a k-means implementation could make use of this new functionality (wherever it goes) to handle the centroid updates. Andr?s > > Anyway, not saying SciPy shouldn't take it, but if it isn't tightly coupled to the stats qmc stuff, it might be well suited to be placed next to its closest siblings in the ecosystem? > > If it is mostly for qmc *internals* in SciPy and we can't depend on scikit-learn, I suppose that might be a reason, though if qmc only could it be private and used as a flag/option where needed in qmc? > > If it is used for i.e., *input* to qmc, or on the *output* from qmc stuff, then having an end-user use scikit-learn seems less of an ask. > > Tyler > > On Thu, 26 Aug 2021 at 13:25, Pamphile Roy wrote: >> >> Hi everyone, >> >> I opened a PR which proposes to add Lloyd?s algorithm. >> >> https://github.com/scipy/scipy/pull/14642 >> >> The TL;DR is: Lloyd's algorithm, or Central Voronoi Tesselation (CVT), post-process a sample to uniformize the inter-sample distances. >> It's an iterative process. From a sample, the Voronoi cells are calculated, then each sample is moved toward the centroid of its cell. >> >> Initially I though about adding this to scipy.stats.qmc as I intended to use this to improve a sample. >> But Matt rightly pointed out that this could just be put in scipy.spatial. >> >> What do you think? >> >> Thanks in advance for your inputs! >> >> Cheers, >> Pamphile >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev From roy.pamphile at gmail.com Fri Aug 27 01:58:22 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Fri, 27 Aug 2021 07:58:22 +0200 Subject: [SciPy-Dev] ENH: add Lloyd's algorithm In-Reply-To: References: Message-ID: Thank you both for your inputs. K-means and Lloyd?s algorithm are very different and do not serve the same purpose. First of, let?s just focus on K-means and Voronoi diagrams. K-means will find the centroids of clusters of points. Hence all clusters would have a centroid and a boundary, or cell (based on L2 usually). Now, Voronoi diagrams separate each single point into a cell. And then you can compute the center of mass of the cell to get a centroid. This is why it can be viewed as K-means, but it?s very exaggerated. Also Voronoi diagram is deterministic and unique. Lloyd?s algorithm is a way to post process a sample of points in terms of L2. It?s used in a lot of fields to make blue noise, better meshes, integration, etc. It?s an iterative process where we build a Voronoi map and move the current points towards their centroid. This process tends to lower the variance in the volume of each cells, hence lowering the variance on the L2 between points. I would argue that it fits SciPy because it?s a slight extension of Voronoi diagrams (which we have). We don?t have much in spatial right now and that might be a good candidate for a new feature. Pamphile > On 26.08.2021, at 22:41, Andras Deak wrote: > > On Thu, Aug 26, 2021 at 10:25 PM Tyler Reddy > wrote: >> >> It may be good to clarify why it goes in SciPy vs. say scikit-learn next to k-means clustering. Their docs for k-means ( https://scikit-learn.org/stable/modules/clustering.html#k-means ) even mention several things that sound >> very similar to what I saw in the PR when I took a quick look: >> >> "K-means is often referred to as Lloyd?s algorithm." >> >> and the description of the algo steps sounds very similar: >> >> "The algorithm can also be understood through the concept of Voronoi diagrams. First the Voronoi diagram of the points is calculated using the current centroids. Each segment in the Voronoi diagram becomes a separate cluster. Secondly, the centroids are updated to the mean of each segment. The algorithm then repeats this until a stopping criterion is fulfilled." > > That sounds like (ab)using K-means clustering by requesting n clusters > for n points: you want each point to be its own "cluster" so that they > each get updated. Except I suspect that this could mess with stopping > criteria (assuming there's one for the makeup of each cluster in each > iteration). I wonder if there would also be overhead from targetting > what is probably a pathological case for clustering. > I guess it sounds more like that a k-means implementation could make > use of this new functionality (wherever it goes) to handle the > centroid updates. > > Andr?s > > >> >> Anyway, not saying SciPy shouldn't take it, but if it isn't tightly coupled to the stats qmc stuff, it might be well suited to be placed next to its closest siblings in the ecosystem? >> >> If it is mostly for qmc *internals* in SciPy and we can't depend on scikit-learn, I suppose that might be a reason, though if qmc only could it be private and used as a flag/option where needed in qmc? >> >> If it is used for i.e., *input* to qmc, or on the *output* from qmc stuff, then having an end-user use scikit-learn seems less of an ask. >> >> Tyler >> >> On Thu, 26 Aug 2021 at 13:25, Pamphile Roy wrote: >>> >>> Hi everyone, >>> >>> I opened a PR which proposes to add Lloyd?s algorithm. >>> >>> https://github.com/scipy/scipy/pull/14642 >>> >>> The TL;DR is: Lloyd's algorithm, or Central Voronoi Tesselation (CVT), post-process a sample to uniformize the inter-sample distances. >>> It's an iterative process. From a sample, the Voronoi cells are calculated, then each sample is moved toward the centroid of its cell. >>> >>> Initially I though about adding this to scipy.stats.qmc as I intended to use this to improve a sample. >>> But Matt rightly pointed out that this could just be put in scipy.spatial. >>> >>> What do you think? >>> >>> Thanks in advance for your inputs! >>> >>> Cheers, >>> Pamphile >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From keith at sloan-home.co.uk Sat Aug 28 09:45:05 2021 From: keith at sloan-home.co.uk (Keith Sloan) Date: Sat, 28 Aug 2021 14:45:05 +0100 Subject: [SciPy-Dev] Understanding scipy.stats.boxcox Message-ID: <714395c8-dc17-0bf1-f2a5-c121d58b8425@sloan-home.co.uk> I have the following questions about scipy.stats.boxcox and looking for pointers guidance. I would like to implement something similar to boxcox but with the following differences. 1) I would like to use a different transformation function from w=(y^x - 1) / x 2) I would separate from 1) like to have the target be other than a Normal/Gaussian distribution, I am thinking of a Plank function for a Blackbody I had a quick look at https://github.com/scipy/scipy/blob/master/scipy/stats/_continuous_distns.py but was still lacking enlightenment, so would welcome some pointers Thanks From xingyuliu at g.harvard.edu Mon Aug 30 13:19:04 2021 From: xingyuliu at g.harvard.edu (Xingyu Liu) Date: Mon, 30 Aug 2021 17:19:04 +0000 Subject: [SciPy-Dev] GSoC'21 Blog: Improve performance through use of Pythran Message-ID: Hi everyone: I?m Xingyu Liu, this year?s SciPy GSoC student :) This summer, I?m very fortunate to work on improving SciPy algorithms? performance via Pythran, under the supervision of my awesome mentors Ralf Gommers and Serge Guelton! I?m so happy to share the blog of our work: http://serge-sans-paille.github.io/pythran-stories/gsoc21-improve-performance-through-the-use-of-pythran.html It has been a great experience working on this project in GSoC'21, and much thanks to the SciPy community for this opportunity and help on my project! Special thanks to my mentors, Ralf and Serge, who provided immense support for me to get through the difficulties! I have learnt a lot this summer and I?ll continue to contribute to SciPy and Pythran in the future!!! Cheers, Xingyu -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Aug 30 14:37:51 2021 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 30 Aug 2021 20:37:51 +0200 Subject: [SciPy-Dev] GSoC'21 Blog: Improve performance through use of Pythran In-Reply-To: References: Message-ID: On Mon, Aug 30, 2021 at 7:19 PM Xingyu Liu wrote: > Hi everyone: > > > > I?m Xingyu Liu, this year?s SciPy GSoC student :) This summer, I?m very fortunate > to work on improving SciPy algorithms? performance via Pythran, under the > supervision of my awesome mentors Ralf Gommers and Serge Guelton! I?m so > happy to share the blog of our work: > http://serge-sans-paille.github.io/pythran-stories/gsoc21-improve-performance-through-the-use-of-pythran.html > Thanks for sharing Xingyu! This has been a really fun and productive GSoC project from my perspective. It definitely helped mature Pythran and its integration into SciPy. There are a number of open PRs left that are close to completion and that we'll get merged over the coming weeks, however overall a lot was already merged into both SciPy and Pythran as Xingyu's blog post shows. One of the main take-aways we learned recently was that we should use Pythran to accelerate the private part of a function, but leave the public signature and input validation in Python. This is something that wasn't obvious at the start of Xingyu's project, and we'll need to make sure we do that also for already-merged PRs (for consistently, and possibly to fix some undetected corner case bugs in master) before branching off 1.8.x. Other than that I think we are still pretty happy with Pythran; it gives us fast performance that's easier to obtain than with Cython. It has been a great experience working on this project in GSoC'21, and much > thanks to the SciPy community for this opportunity and help on my > project! Special thanks to my mentors, Ralf and Serge, who provided immense > support for me to get through the difficulties! I have learnt a lot this > summer and I?ll continue to contribute to SciPy and Pythran in the > future!!! > It's been a pleasure working with you too, and I'm happy you're going to continue contributing Xingyu! Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Tue Aug 31 05:09:56 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Tue, 31 Aug 2021 11:09:56 +0200 Subject: [SciPy-Dev] GSoC'21 Blog: Improve performance through use of Pythran In-Reply-To: References: Message-ID: Great job Xingyu! And happy to read that you are planning to contribute more. > On 30.08.2021, at 20:37, Ralf Gommers wrote: > > One of the main take-aways we learned recently was that we should use Pythran to accelerate the private part of a function, but leave the public signature and input validation in Python. This is something that wasn't obvious at the start of Xingyu's project, and we'll need to make sure we do that also for already-merged PRs (for consistently, and possibly to fix some undetected corner case bugs in master) before branching off 1.8.x. Other than that I think we are still pretty happy with Pythran; it gives us fast performance that's easier to obtain than with Cython. Ok, then I understand from this that we should do Pythran over Cython for new code. Just a thought. If we are going this road, it looks like we could have a Python API and a Pythran/(Cython) API. A user could then decide to skip the validation like that. Having the same signature would be less of a work in terms of doc as we could just add a note in the Python function about the existence of a Pythran version. Or we allow differences and clearly state that this API is not as strict as the Python API. Cheers, Pamphile From ralf.gommers at gmail.com Tue Aug 31 05:51:29 2021 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 31 Aug 2021 11:51:29 +0200 Subject: [SciPy-Dev] GSoC'21 Blog: Improve performance through use of Pythran In-Reply-To: References: Message-ID: On Tue, Aug 31, 2021 at 11:10 AM Pamphile Roy wrote: > Great job Xingyu! And happy to read that you are planning to contribute > more. > > > On 30.08.2021, at 20:37, Ralf Gommers wrote: > > > > One of the main take-aways we learned recently was that we should use > Pythran to accelerate the private part of a function, but leave the public > signature and input validation in Python. This is something that wasn't > obvious at the start of Xingyu's project, and we'll need to make sure we do > that also for already-merged PRs (for consistently, and possibly to fix > some undetected corner case bugs in master) before branching off 1.8.x. > Other than that I think we are still pretty happy with Pythran; it gives us > fast performance that's easier to obtain than with Cython. > > Ok, then I understand from this that we should do Pythran over Cython for > new code. > It's not as simple as that, it depends on what you're trying to do. If Pythran fits, like for individual kernels, then probably yes. Cython can do more complex things though, like interact with the Python and NumPy C APIs, threading, etc. Serge and I gave a talk at SciPy'21 that addresses this: https://www.youtube.com/watch?v=6a9D9WL6ZjQ&t=2s. > Just a thought. If we are going this road, it looks like we could have a > Python API and a Pythran/(Cython) API. A user could then decide to skip the > validation like that. > Having the same signature would be less of a work in terms of doc as we > could just add a note in the Python function about the existence of a > Pythran version. > Or we allow differences and clearly state that this API is not as strict > as the Python API. > There's no such thing as a Pythran API. Pythran generates a Python API, and there's no separate C-like API. So whether a Python function uses Pythran under the hood should always be an implementation detail. It is unrelated to potentially skipping validation. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy.pamphile at gmail.com Tue Aug 31 06:06:59 2021 From: roy.pamphile at gmail.com (Pamphile Roy) Date: Tue, 31 Aug 2021 12:06:59 +0200 Subject: [SciPy-Dev] GSoC'21 Blog: Improve performance through use of Pythran In-Reply-To: References: Message-ID: <1A27B96A-A44B-462C-A1E1-9737EFCA557E@gmail.com> Thanks for the precisions Ralf! Cheers, Pamphile > On 31.08.2021, at 11:51, Ralf Gommers wrote: > > > > On Tue, Aug 31, 2021 at 11:10 AM Pamphile Roy > wrote: > Great job Xingyu! And happy to read that you are planning to contribute more. > > > On 30.08.2021, at 20:37, Ralf Gommers > wrote: > > > > One of the main take-aways we learned recently was that we should use Pythran to accelerate the private part of a function, but leave the public signature and input validation in Python. This is something that wasn't obvious at the start of Xingyu's project, and we'll need to make sure we do that also for already-merged PRs (for consistently, and possibly to fix some undetected corner case bugs in master) before branching off 1.8.x. Other than that I think we are still pretty happy with Pythran; it gives us fast performance that's easier to obtain than with Cython. > > Ok, then I understand from this that we should do Pythran over Cython for new code. > > It's not as simple as that, it depends on what you're trying to do. If Pythran fits, like for individual kernels, then probably yes. Cython can do more complex things though, like interact with the Python and NumPy C APIs, threading, etc. Serge and I gave a talk at SciPy'21 that addresses this: https://www.youtube.com/watch?v=6a9D9WL6ZjQ&t=2s . > > > Just a thought. If we are going this road, it looks like we could have a Python API and a Pythran/(Cython) API. A user could then decide to skip the validation like that. > Having the same signature would be less of a work in terms of doc as we could just add a note in the Python function about the existence of a Pythran version. > Or we allow differences and clearly state that this API is not as strict as the Python API. > > There's no such thing as a Pythran API. Pythran generates a Python API, and there's no separate C-like API. So whether a Python function uses Pythran under the hood should always be an implementation detail. It is unrelated to potentially skipping validation. > > Cheers, > Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: