From spanda3 at jhu.edu Thu Jun 6 12:09:11 2019 From: spanda3 at jhu.edu (Sambit Panda) Date: Thu, 6 Jun 2019 16:09:11 +0000 Subject: [SciPy-Dev] Request to add functionality to scipy.stats Message-ID: <3690FF8B-1117-4890-9425-F94303D2A9B8@jh.edu> Request for project components' inclusion in scipy.stats - Project name: "mgcpy" - Authors: Satish Palaniappan (https://github.com/tpsatish95), Sambit Panda (https://github.com/sampan501), Junhao Xiong (https://github.com/junhaobearxiong), Ananya Swaminathan (https://github.com/ananyas713), Sandhya Ramachandran (https://github.com/sundaysundya), Richard Guo (https://github.com/rguo123) - Current repository: https://github.com/neurodata/mgcpy "mgcpy" is a Python package containing tools for independence testing and k-sample testing. Looking through the "scipy.stats" module, The module contains a host of independence and other hypothesis tests, but are limited by assumptions of normality, linearity, unidimensionality, etc. While this may be appropriate in a host of circumstances, it is increasingly important to analyze nonlinear and high dimensional trends, which is where the implementations in "mgcpy" could be very useful. Independence tests included can operate on multidimensional and nonlinear data. In addition, functionality has been extended to k-sample testing (with capabilities of operating on the same kinds of data). The tests included can not only be used for classification, but also for regression. Below is a list of some of the integrated tests contained within "mgcpy" and citations for relevant papers about it. - RV: P. Robert and Y. Escoufier, "A unifying tool for linear multivariate statistical methods: the rv-coefficient," Journal of the Royal Statistical Society: Series C (Applied Statistics), vol. 25, no. 3, pp. 257?265, 1976. 3 - CCA: D. R. Hardoon, S. Szedmak, and J. Shawe-Taylor, "Canonical correlation analysis: An overview with application to learning methods," Neural computation, vol. 16, no. 12, pp. 2639?2664, 2004. - HHG: R. Heller, Y. Heller, and M. Gorfine, "A consistent multivariate test of association based on ranks of distances," Biometrika, vol. 100, no. 2, pp. 503?510, 2012. - MDMR: N. J. Schork and M. A. Zapala, "Statistical properties of multivariate distance matrix regression for high-dimensional data analysis," Frontiers in Genetics, vol. 3, p. 190, 2012. - Biased Dcorr, Unbiased Dcorr**: G. J. Sz?kely, M. L. Rizzo, N. K. Bakirov et al., "Measuring and testing dependence by correlation of distances," The Annals of Statistics, vol. 35, no. 6, pp. 2769?2794, 2007. - Mantel: N. Mantel, "The detection of disease clustering and a generalized regression approach," Cancer research, vol. 27, no. 2 Part 1, pp. 209?220, 1967. - MANOVA: Warne, R. T. (2014). "A primer on multivariate analysis of variance (MANOVA) for behavioral scientists". Practical Assessment, Research & Evaluation. 19 (17): 1?10. - k-sample tests: Mart?nez-Camblor, P., & de U?a-?lvarez, J. (2009). Non-parametric k-sample tests: Density functions vs distribution functions. Computational Statistics & Data Analysis, 53(9), 3344-3357. Not included tests, but related useful readings: - Equivalency of Dcorr, HSIC, Energy, and MMD: C. Shen and J. T. Vogelstein, "The exact equivalence of distance and kernel methods for hypothesis testing," arXiv preprint arXiv:1806.05514, 2018. - Formulating k-sample tests as independence tests: C. Shen and J. T. Vogelstein, "The exact equivalence of distance and kernel methods for hypothesis testing," arXiv preprint arXiv:1806.05514, 2018. From tyler.je.reddy at gmail.com Thu Jun 6 22:04:38 2019 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Thu, 6 Jun 2019 19:04:38 -0700 Subject: [SciPy-Dev] ANN: SciPy 1.2.2 (LTS) Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi all, On behalf of the SciPy development team I'm pleased to announce the release of SciPy 1.2.2, which is a bug fix release. This is part of the long-term support (LTS) branch that includes Python 2.7. Sources and binary wheels can be found at: https://pypi.org/project/scipy/ and at: https://github.com/scipy/scipy/releases/tag/v1.2.2 One of a few ways to install this release with pip: pip install scipy==1.2.2 ===================== SciPy 1.2.2 Release Notes ===================== SciPy 1.2.2 is a bug-fix release with no new features compared to 1.2.1. Importantly, the SciPy 1.2.2 wheels are built with OpenBLAS 0.3.7.dev to alleviate issues with SkylakeX AVX512 kernels. Authors ====== * CJ Carey * Tyler Dawson + * Ralf Gommers * Kai Striega * Andrew Nelson * Tyler Reddy * Kevin Sheppard + A total of 7 people contributed to this release. People with a "+" by their names contributed a patch for the first time. This list of names is automatically generated, and may not be fully complete. Issues closed for 1.2.2 ------------------------------ * `#9611 `__: Overflow error with new way of p-value calculation in kendall tau correlation for perfectly monotonic vectors * `#9964 `__: optimize.newton : overwrites x0 argument when it is a numpy array * `#9784 `__: TST: Minimum NumPy version is not being CI tested * `#10132 `__: Docs: Description of nnz attribute of sparse.csc_matrix misleading Pull requests for 1.2.2 ----------------------------- * `#10056 `__: BUG: Ensure factorial is not too large in kendaltau * `#9991 `__: BUG: Avoid inplace modification of input array in newton * `#9788 `__: TST, BUG: f2py-related issues with NumPy < 1.14.0 * `#9749 `__: BUG: MapWrapper.__exit__ should terminate * `#10141 `__: Update description for nnz on csc.py Checksums ========= MD5 ~~~ f5d23361e78f230f70fd117be20930e1 scipy-1.2.2-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 44387030d96a2495e5576800b2a567d6 scipy-1.2.2-cp27-cp27m-manylinux1_i686.whl bc56bf862deadc96f6be1f67dc8eaf89 scipy-1.2.2-cp27-cp27m-manylinux1_x86_64.whl a45382978ff7d032041847f66e2f7351 scipy-1.2.2-cp27-cp27m-win32.whl 1140063ad53c44414f9feaae3c4fbf8c scipy-1.2.2-cp27-cp27m-win_amd64.whl 3407230bae0c36210c5d3fee717a3579 scipy-1.2.2-cp27-cp27mu-manylinux1_i686.whl fbb9867ea3ba38cc0c979c38b8c77871 scipy-1.2.2-cp27-cp27mu-manylinux1_x86_64.whl 8b4497e964c17135b6b2e8f691bed49e scipy-1.2.2-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 9139c344bc6ef05f7f22191af0810ef6 scipy-1.2.2-cp34-cp34m-manylinux1_i686.whl a62c1f316c33af02007da3374ebf02c3 scipy-1.2.2-cp34-cp34m-manylinux1_x86_64.whl 780ce592f99ade01a9b0883ac767f798 scipy-1.2.2-cp34-cp34m-win32.whl 498e740b099182df30c16144a109acdf scipy-1.2.2-cp34-cp34m-win_amd64.whl 8b157f5433846d8798ff6941d0f9671f scipy-1.2.2-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl e1692a9e3e9a9b2764bccd0c9575bfef scipy-1.2.2-cp35-cp35m-manylinux1_i686.whl 70863fc59dc034c07b73de765eb693f9 scipy-1.2.2-cp35-cp35m-manylinux1_x86_64.whl ce676f1adc72f8180b2eacec7e44c802 scipy-1.2.2-cp35-cp35m-win32.whl 21a9fac5e289682abe35ce6d54f5805f scipy-1.2.2-cp35-cp35m-win_amd64.whl 470fa57418223df8fc27e9ec45bc7a94 scipy-1.2.2-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 4001f322a2967de0aa0b8148e0116def scipy-1.2.2-cp36-cp36m-manylinux1_i686.whl 4e0d727cbbfe8410bd1229d197fb11d8 scipy-1.2.2-cp36-cp36m-manylinux1_x86_64.whl 352608fa1f48877fc76a55217e689240 scipy-1.2.2-cp36-cp36m-win32.whl 559ca5cda1935a9992436bb1398dbcd0 scipy-1.2.2-cp36-cp36m-win_amd64.whl 92b9356944c239520f5b2897ba531c16 scipy-1.2.2-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl d9b427be8fc3bfd5b2a8330e1215b0ee scipy-1.2.2-cp37-cp37m-manylinux1_i686.whl 4f2d513b1950ab7c147ddf3e4acb2542 scipy-1.2.2-cp37-cp37m-manylinux1_x86_64.whl 1598ffe78061854f7bed87290250c33f scipy-1.2.2-cp37-cp37m-win32.whl 9dad5d71152b714694e073d1c0c54288 scipy-1.2.2-cp37-cp37m-win_amd64.whl d94de858fba4f24de7d6dd16f1caeb5d scipy-1.2.2.tar.gz 136c5ee1bc4b259a12a7efe331b15d64 scipy-1.2.2.tar.xz b9a5b4cbdf54cf681eda3b4d94a73c18 scipy-1.2.2.zip SHA256 ~~~~~~ 271c6e56c8f9a3d6c3f0bc857d7a6e7cf7a8415c879a3915701cd011e82a83a3 scipy-1.2.2-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 2eb255b30dac7516c6f3c5237f2e0ad1f1213b5364de409d932249c9a8c5bffb scipy-1.2.2-cp27-cp27m-manylinux1_i686.whl 7f58faa422aa493d7b70dd56d6e8783223e84dd6e7f4b4161bd776b39ecbac92 scipy-1.2.2-cp27-cp27m-manylinux1_x86_64.whl d0d41a9ee3264f95820138170b447f5d3e453e5ebd10b411bca37c99237aac69 scipy-1.2.2-cp27-cp27m-win32.whl b074a83299a82eae617dc46a830cfa7aaa588d07523990507848ee1ded3c52ce scipy-1.2.2-cp27-cp27m-win_amd64.whl 49dcebc6f57bce0bd23cb55dbc6144f4990e5cbce9aab3128af03d6b1b4eab6a scipy-1.2.2-cp27-cp27mu-manylinux1_i686.whl 67d2210c7f6f585e1055bee3dc9f15610b5ebb04e80bfaa757868937ee744fec scipy-1.2.2-cp27-cp27mu-manylinux1_x86_64.whl 0bcababa06ff83138a7f30a68f334dee034ce1cc7604f9278b96f62265fe7fd7 scipy-1.2.2-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl a9fc1fcaa560edf771d4545d7e6dd865a213fc5b485bb127de5dfd32f40094e1 scipy-1.2.2-cp34-cp34m-manylinux1_i686.whl 7fb4efff9895116428ad65564d2232fb1cac4b9d84398512a858b09dd4a7fd59 scipy-1.2.2-cp34-cp34m-manylinux1_x86_64.whl fbdff021643c2dfa35efd29218e0318c4b4987f48ea432be7e8c02bdb1b0c314 scipy-1.2.2-cp34-cp34m-win32.whl f4e355afa8fdda11010de308c2376edda29e064cec699974097364115f71e16f scipy-1.2.2-cp34-cp34m-win_amd64.whl e99cd49daffe7384fd35046c3b14bee98ce87d97c95865469227001905534e13 scipy-1.2.2-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 447c40d33ec5e0020750fadbb8599220b9eb9fd8798030efe9b308247800f364 scipy-1.2.2-cp35-cp35m-manylinux1_i686.whl 9a21d64d002cb3a9239a55c0aa100b48d58b5e38382c0fdfcdfc68cf417d8142 scipy-1.2.2-cp35-cp35m-manylinux1_x86_64.whl 5fa84b467b5f77c243c5701628ed7a4238e53bc4120db87be7dafa416e842fb9 scipy-1.2.2-cp35-cp35m-win32.whl 682b210ff7a65f6f5245fdf73d26a348b57e42d2059bc5fcf7ed25d063f35c45 scipy-1.2.2-cp35-cp35m-win_amd64.whl bcd0d4b2de5cb3fab69007214a39737e917267f56f887ce9c7732ba3278fc33d scipy-1.2.2-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 4686d699f76068757a81269f1a111c0db689bf048a56b131a339803121534fa8 scipy-1.2.2-cp36-cp36m-manylinux1_i686.whl 97f26b4b5d4456f44849fd35cad8801f7cae4e64b75fc4e522d26a54aef17391 scipy-1.2.2-cp36-cp36m-manylinux1_x86_64.whl 922e2370674c82dd1367fc13a08c8765f4e5281a584d871e7cb454828d84600f scipy-1.2.2-cp36-cp36m-win32.whl c390f1721757ec983616149f00e1bd0432aa32d2c1d9398930d7e7cc9542c922 scipy-1.2.2-cp36-cp36m-win_amd64.whl 47d4623efa71948dc4a92f978fbf6b9fb69dac5b0f0fae4c1a1f3d955ac8aea9 scipy-1.2.2-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 162b803984ebb76927990d7233cab825d146be8e2a3f6a0efb1b3a61ebacae73 scipy-1.2.2-cp37-cp37m-manylinux1_i686.whl d18d1575d4a54f128c0f34422bd73ce0f177e462d6124f074388e211d8dc2616 scipy-1.2.2-cp37-cp37m-manylinux1_x86_64.whl c5b9db9e3f6537bf7b308de12c185b27f22fb9a66fd12efc7aefbcfa0adb4d82 scipy-1.2.2-cp37-cp37m-win32.whl f64e29a8b32d672fb6078f456bfff3cae8f36b6c8b64c337ad0942f29404b03f scipy-1.2.2-cp37-cp37m-win_amd64.whl a4331e0b8dab1ff75d2c67b5158a8bb9a83c799d7140094dda936d876c7cfbb1 scipy-1.2.2.tar.gz 8006216f7e99dbf639b9293c73c197f36c34389ea4a223547e31f1772d2626f7 scipy-1.2.2.tar.xz 578030ddb33eb093cdd2ebfb71ce052bb8b2d5cd33d14bcc452069b05ffa9dff scipy-1.2.2.zip -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJc+bQAAAoJELD/41ZX0J714DIP/1lo7lRql213ElrtmrXSetvc wagnMTnwKW6/zAWXrDaX5aQ3XoCbUyDyva427vCoAIacw7HDsdgCkw6hb8WCI2fM SxG7iBDuDnJM3iVBtM23qUH4aI4CvRoVZmG2oF4fwwMpjvx80bMzHmm1xkw5OVBz 9JaYYplT1PCcTUD4CnwX2jG66NlzLYomQgdg67I6NIubelKhVUEMRx1j9s5Ed76q bwPZbV6i52kzuG441VXhUq1Rhn/+j55/hgnlRpbQFAkwbz664OkqBZ1FPIH7/Wpq vGFxYPROECxjrpiPYiWtXZpJfRJySiQ5oltBHec3MdX1b/S7cAm7BCI0hW4NsPmU i36Ho0f6WhCTMGowl+V4uylE3hvWEW0zHp9MJwe2mUNWc9YKPu0pCZ3hif/YH+rh oQD8sI3IUZuyJ0ntPWN/SCXdE5kmE5zRLIBFoap15uRComuypuZWmMrWuX4oC1Qb 0pKuCa3UcDXmTVVc+ZypnbRfePUbocxeP9lrsQlT43nfhp9jXkPw1fWxXII9ChNL ORoyAvHzgfUxc4x5vka8Evs1OhSzMg6SYkH0SN1qNiOHq9RLBwdLqk4dgyNmLCPP fdr6HvLEK6rYNNDEq6IxY7h8zSFtOQRjtW348W6T611sAHAa8u+51nVMI5KljPlF x9AB2Fj0Q4rFQpVQtlz/ =jT1x -----END PGP SIGNATURE----- -------------- next part -------------- An HTML attachment was scrubbed... URL: From PeterBell10 at live.co.uk Mon Jun 10 10:22:37 2019 From: PeterBell10 at live.co.uk (Peter Bell) Date: Mon, 10 Jun 2019 14:22:37 +0000 Subject: [SciPy-Dev] scipy.fft submodule Message-ID: Following on from some of the discussion surrounding my backend proposal and the discussion on gh-10175; I've been working on adding a new scipy.fft submodule. This is now working, tested, documented and ready for review in gh-10238. The new submodule is almost a drop-in replacement for numpy.fft and scipy.fftpack, with a few exceptions: * Does not include NumPy's Hermitian transforms (hfft, ihfft) (but would be simple to add) * Uses numpy's conventions for rfft (complex array) instead of fftpack's (complex values packed into a real array) * Convolutions and pseudo-differential operators from fftpack are not included The new submodule adds pocketfft to implement the normal FFTs (not yet DCT and DST) which adds several new features beyond scipy.fftpack: * long double transforms * multi-dimensional real FFTs (rfftn) * Orthonormal transforms (norm='ortho' argument) * Bluestein's algorithm to avoid the worst case O(n^2) complexity of FFTPACK A few implementation details worth noting: * Unlike NumPy's version of pocketfft, this is C++ and uses templates to supporting the different floating point types * pocketfft also uses pybind11 which both adds a new build-time dependency and requires C++11. I believe this is the first use of C++11 in SciPy. * pocketfft has optional support for multithreading using OpenMP but this is currently disabled (and not compiled in at all). And finally, there is the rather awkward issue that scipy.fft already exists and is an alias for the numpy.fft.fft function. Currently I've had to add a workaround to allow the module scipy.fft to be callable, as discussed in gh-10253. This is only relevant to code that imports scipy.fft so I don't expect it to be used often, if at all. Hopefully this is a temporary solution and the NumPy functions can be removed from the scipy namespace in some future release, or at the very least the FFT functions can be removed. Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: From PeterBell10 at live.co.uk Mon Jun 10 10:49:29 2019 From: PeterBell10 at live.co.uk (Peter Bell) Date: Mon, 10 Jun 2019 14:49:29 +0000 Subject: [SciPy-Dev] Backend support for fftpack - GSoC 2019 In-Reply-To: References: Message-ID: I?ve had another look at the uarray docs and was quite surprised to see that the API has changed significantly since I last looked at them. It seems that uarray is now based on a __ua_function__ protocol which is similar to NEP 18 except that it?s held in the backend rather than the array type. I think I?ve tracked the change to PR #152 which is dated after the API was claimed to be stable. Hameer, would you be able to clarify what state uarray development is in: exactly how stable is the API? Peter From: SciPy-Dev On Behalf Of Hameer Abbasi Sent: 09 May 2019 13:39 To: SciPy Developers List Subject: Re: [SciPy-Dev] Backend support for fftpack - GSoC 2019 Hi, Since uarray came up: I?m the author. Yes, performance is slow (since it?s pure python) but the API is stable at this point (the docs are a bit out of date, I should update them). We?re welcoming any kind of contributions as I, myself, am not familiar enough with the CPython API to do it. However, I?d love to learn! It does have a focus on array computing but it is meant to be fairly general, and you shouldn?t have any problems with the interface. Best Regards, Hameer Abbasi On Thursday, May 09, 2019 at 8:25 AM, Peter Bell > wrote: >> Should backends be allowed to add functionality or should they be completely >> interchangeable. E.g. unlike fftpack, FFTW has support for long doubles; >> should this be exposed to SciPy users? > > If a user writes backend-specific code, then they may just as well bypass the > backend right, and call the other library directly? I agree, in fact I quote you in the proposal saying something similar. > Although for long double things will work out of the box perhaps, because the > dtype of an input array doesn't come back in the API. So the array can be > passed straight to the backend. This was why I picked it as an example. It doesn't add any new parameters or change the function signature at all, but it does change the result of a particular call to the API. So it's a very minimal change yet it still breaks the interchangeability of backends. > This is probably also a good time to point out http://uarray.readthedocs.io/. > It can provide a complete backend system without too much work After reading through the docs it does seem to cover this and more. However, it also seems to be unstable at this point so I?m not sure it?s wise to rely on it just yet. > it was designed exactly for the type of problem you're trying to solve here. Unless I'm misunderstanding it seems to be more focused on supporting different types of array, rather than different implementations for NumPy arrays. I can't see any utility in having a complicated dispatch mechanism when we really only want to have a single backend at a time for any given function. I would assume that because of this, uarray would come with a greater performance overhead. Whereas I think the overhead of an fftpack backend should just be some argument validation and a single function call. I could always implement it both ways and benchmark it if there's interest. Peter From: SciPy-Dev > On Behalf Of Ralf Gommers Sent: 08 May 2019 23:59 To: SciPy Developers List > Subject: Re: [SciPy-Dev] Backend support for fftpack - GSoC 2019 On Wed, May 8, 2019 at 5:50 PM Peter Bell > wrote: Hello all, I?m happy to say my GSoC proposal was accepted, Hey Peter, congratulations! We're happy to have you on board! so I?ll be working over the summer to add support for multiple fftpack backends. For those interested, I?ve extracted the main discussion from my proposal which you can read here: https://docs.google.com/document/d/1hGMc6ZrsmSi_LSixI5oFbo5cZkPUo92sfmnVthg1u6Y/edit Thanks, interesting. Will take a little while to process. I just give some very initial thoughts below (so take with a grain of salt). There are a several design decisions I raise in the proposal and any comments would be appreciated. Of particular note: * Should backends be allowed to add functionality or should they be completely interchangeable. E.g. unlike fftpack, FFTW has support for long doubles; should this be exposed to SciPy users? If a user writes backend-specific code, then they may just as well bypass the backend right, and call the other library directly? Although for long double things will work out of the box perhaps, because the dtype of an input array doesn't come back in the API. So the array can be passed straight to the backend. * Is there any interest in adding a SciPy config file? For just one option it?s probably not worthwhile but if there is interest in more config options then it starts to make more sense. My sense is no. Something like a context manager (`with backend('pyfftw'):`) is lightweight enough probably. This is probably also a good time to point out http://uarray.readthedocs.io/. It can provide a complete backend system without too much work, it was designed exactly for the type of problem you're trying to solve here. Would be good to have your thoughts on that. Ideally a clear set of design goals can be agreed upon now, before it gets too far into the coding period which begins at the end of the month. Yes great idea. Cheers, Ralf _______________________________________________ SciPy-Dev mailing list SciPy-Dev at python.org https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From einstein.edison at gmail.com Mon Jun 10 14:40:23 2019 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Mon, 10 Jun 2019 20:40:23 +0200 Subject: [SciPy-Dev] Backend support for fftpack - GSoC 2019 In-Reply-To: References: Message-ID: <80afec54-ef46-485f-b0b2-e5d3ec682f0b@Canary> Hi Peter, This was based on the feedback of the NumPy team, which preferred protocols over an OOP based model and wanted to avoid deep nesting. It also bought down the overhead factor tenfold. I also removed an arg/kwarg normalisation feature, which bought down the overhead another 15-fold. Ralf and I had some offline discussions about the overhead, which seemed to be more of a blocker than all of the other features combined. Rest assured, this was out of a desire to respond to feedback from the NumPy and SciPy teams and once there is code actually relying on uarray, we will not change the API any further. If you have code that is dependent on the last version, I?d be happy to revert my changes. Best Regards, Hameer Abbasi > On Monday, Jun 10, 2019 at 4:49 PM, Peter Bell wrote: > > I?ve had another look at the uarray docs and was quite surprised to see that the API has changed significantly since I last looked at them. It seems that uarray is now based on a __ua_function__ protocol which is similar to NEP 18 except that it?s held in the backend rather than the array type. I think I?ve tracked the change to PR #152 (https://github.com/Quansight-Labs/uarray/pull/152/) which is dated after the API was claimed to be stable. > > > > > > Hameer, would you be able to clarify what state uarray development is in: exactly how stable is the API? > > > > > > Peter > > > > > > > > > From: SciPy-Dev On Behalf Of Hameer Abbasi > Sent: 09 May 2019 13:39 > To: SciPy Developers List > Subject: Re: [SciPy-Dev] Backend support for fftpack - GSoC 2019 > > > > > > > > Hi, > > > > > > > > Since uarray came up: I?m the author. Yes, performance is slow (since it?s pure python) but the API is stable at this point (the docs are a bit out of date, I should update them). > > > > > > > > We?re welcoming any kind of contributions as I, myself, am not familiar enough with the CPython API to do it. However, I?d love to learn! > > > > > > > > It does have a focus on array computing but it is meant to be fairly general, and you shouldn?t have any problems with the interface. > > > > > > > > > > > Best Regards, > > > Hameer Abbasi > > > > > > > > > > > > > On Thursday, May 09, 2019 at 8:25 AM, Peter Bell wrote: > > > > > > > > >> Should backends be allowed to add functionality or should they be completely > > > > > > >> interchangeable. E.g. unlike fftpack, FFTW has support for long doubles; > > > > > > >> should this be exposed to SciPy users? > > > > > > > > > > > > > > If a user writes backend-specific code, then they may just as well bypass the > > > > > > > backend right, and call the other library directly? > > > > > > > > > > > > I agree, in fact I quote you in the proposal saying something similar. > > > > > > > > > > > > > Although for long double things will work out of the box perhaps, because the > > > > > > > dtype of an input array doesn't come back in the API. So the array can be > > > > > > > passed straight to the backend. > > > > > > > > > > > > This was why I picked it as an example. It doesn't add any new parameters or > > > > > > change the function signature at all, but it does change the result of a > > > > > > particular call to the API. So it's a very minimal change yet it still breaks > > > > > > the interchangeability of backends. > > > > > > > This is probably also a good time to point out http://uarray.readthedocs.io/. > > > > > > > It can provide a complete backend system without too much work > > > > > > > > > > > > After reading through the docs it does seem to cover this and more. However, it > > > > > > also seems to be unstable at this point so I?m not sure it?s wise to rely on it just > > > > > > yet. > > > > > > > > > > > > > it was designed exactly for the type of problem you're trying to solve here. > > > > > > > > > > > > Unless I'm misunderstanding it seems to be more focused on supporting different > > > > > > types of array, rather than different implementations for NumPy arrays. I can't > > > > > > see any utility in having a complicated dispatch mechanism when we really only > > > > > > want to have a single backend at a time for any given function. > > > > > > > > > > > > I would assume that because of this, uarray would come with a greater > > > > > > performance overhead. Whereas I think the overhead of an fftpack backend should > > > > > > just be some argument validation and a single function call. I could always > > > > > > implement it both ways and benchmark it if there's interest. > > > > > > > > > > > > Peter > > > > > > > > > > > > From: SciPy-Dev On Behalf Of Ralf Gommers > > Sent: 08 May 2019 23:59 > > To: SciPy Developers List > > Subject: Re: [SciPy-Dev] Backend support for fftpack - GSoC 2019 > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, May 8, 2019 at 5:50 PM Peter Bell wrote: > > > > > > > > > > Hello all, > > > > > > > > > > > > > > > > > > I?m happy to say my GSoC proposal was accepted, > > > > > > > > > > > > > > > > > > > > > > > Hey Peter, congratulations! We're happy to have you on board! > > > > > > > > > > > > > > > > > > so I?ll be working over the summer to add support for multiple fftpack backends. For those interested, I?ve extracted the main discussion from my proposal which you can read here: > > > > > > > > > https://docs.google.com/document/d/1hGMc6ZrsmSi_LSixI5oFbo5cZkPUo92sfmnVthg1u6Y/edit > > > > > > > > > > > > > > > > > > > > > > > Thanks, interesting. Will take a little while to process. I just give some very initial thoughts below (so take with a grain of salt). > > > > > > > > > > > > > > > > > > > > > > > > > > > There are a several design decisions I raise in the proposal and any comments would be appreciated. Of particular note: > > > > > > > > > * Should backends be allowed to add functionality or should they be completely interchangeable. E.g. unlike fftpack, FFTW has support for long doubles; should this be exposed to SciPy users? > > > > > > > > > > > > > > > > > > > > > > > If a user writes backend-specific code, then they may just as well bypass the backend right, and call the other library directly? Although for long double things will work out of the box perhaps, because the dtype of an input array doesn't come back in the API. So the array can be passed straight to the backend. > > > > > > > > > > > > > > > > > > * Is there any interest in adding a SciPy config file? For just one option it?s probably not worthwhile but if there is interest in more config options then it starts to make more sense. > > > > > > > > > > > > > > > > > > > > > > > My sense is no. Something like a context manager (`with backend('pyfftw'):`) is lightweight enough probably. > > > > > > > > > > > > > > > > This is probably also a good time to point out http://uarray.readthedocs.io/. It can provide a complete backend system without too much work, it was designed exactly for the type of problem you're trying to solve here. Would be good to have your thoughts on that. > > > > > > > > > > > > > > > > > > > > > > > > > > > Ideally a clear set of design goals can be agreed upon now, before it gets too far into the coding period which begins at the end of the month. > > > > > > > > > > > > > > > > > > > > > > > Yes great idea. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Ralf > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at python.org (mailto:SciPy-Dev at python.org) > > https://mail.python.org/mailman/listinfo/scipy-dev > > > > > > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From PeterBell10 at live.co.uk Mon Jun 10 15:51:53 2019 From: PeterBell10 at live.co.uk (Peter Bell) Date: Mon, 10 Jun 2019 19:51:53 +0000 Subject: [SciPy-Dev] Backend support for fftpack - GSoC 2019 In-Reply-To: <80afec54-ef46-485f-b0b2-e5d3ec682f0b@Canary> References: <80afec54-ef46-485f-b0b2-e5d3ec682f0b@Canary> Message-ID: >This was based on the feedback of the NumPy team Is this feedback available anywhere? I?ve had a look through the NumPy-Discussions list but couldn?t see anything. > If you have code that is dependent on the last version, I?d be happy to revert my changes. Don?t worry, I just want a sense of where the project is currently. Peter From: SciPy-Dev On Behalf Of Hameer Abbasi Sent: 10 June 2019 19:40 To: SciPy Developers List Subject: Re: [SciPy-Dev] Backend support for fftpack - GSoC 2019 Hi Peter, This was based on the feedback of the NumPy team, which preferred protocols over an OOP based model and wanted to avoid deep nesting. It also bought down the overhead factor tenfold. I also removed an arg/kwarg normalisation feature, which bought down the overhead another 15-fold. Ralf and I had some offline discussions about the overhead, which seemed to be more of a blocker than all of the other features combined. Rest assured, this was out of a desire to respond to feedback from the NumPy and SciPy teams and once there is code actually relying on uarray, we will not change the API any further. If you have code that is dependent on the last version, I?d be happy to revert my changes. Best Regards, Hameer Abbasi On Monday, Jun 10, 2019 at 4:49 PM, Peter Bell > wrote: I?ve had another look at the uarray docs and was quite surprised to see that the API has changed significantly since I last looked at them. It seems that uarray is now based on a __ua_function__ protocol which is similar to NEP 18 except that it?s held in the backend rather than the array type. I think I?ve tracked the change to PR #152 which is dated after the API was claimed to be stable. Hameer, would you be able to clarify what state uarray development is in: exactly how stable is the API? Peter From: SciPy-Dev > On Behalf Of Hameer Abbasi Sent: 09 May 2019 13:39 To: SciPy Developers List > Subject: Re: [SciPy-Dev] Backend support for fftpack - GSoC 2019 Hi, Since uarray came up: I?m the author. Yes, performance is slow (since it?s pure python) but the API is stable at this point (the docs are a bit out of date, I should update them). We?re welcoming any kind of contributions as I, myself, am not familiar enough with the CPython API to do it. However, I?d love to learn! It does have a focus on array computing but it is meant to be fairly general, and you shouldn?t have any problems with the interface. Best Regards, Hameer Abbasi On Thursday, May 09, 2019 at 8:25 AM, Peter Bell > wrote: >> Should backends be allowed to add functionality or should they be completely >> interchangeable. E.g. unlike fftpack, FFTW has support for long doubles; >> should this be exposed to SciPy users? > > If a user writes backend-specific code, then they may just as well bypass the > backend right, and call the other library directly? I agree, in fact I quote you in the proposal saying something similar. > Although for long double things will work out of the box perhaps, because the > dtype of an input array doesn't come back in the API. So the array can be > passed straight to the backend. This was why I picked it as an example. It doesn't add any new parameters or change the function signature at all, but it does change the result of a particular call to the API. So it's a very minimal change yet it still breaks the interchangeability of backends. > This is probably also a good time to point out http://uarray.readthedocs.io/. > It can provide a complete backend system without too much work After reading through the docs it does seem to cover this and more. However, it also seems to be unstable at this point so I?m not sure it?s wise to rely on it just yet. > it was designed exactly for the type of problem you're trying to solve here. Unless I'm misunderstanding it seems to be more focused on supporting different types of array, rather than different implementations for NumPy arrays. I can't see any utility in having a complicated dispatch mechanism when we really only want to have a single backend at a time for any given function. I would assume that because of this, uarray would come with a greater performance overhead. Whereas I think the overhead of an fftpack backend should just be some argument validation and a single function call. I could always implement it both ways and benchmark it if there's interest. Peter From: SciPy-Dev > On Behalf Of Ralf Gommers Sent: 08 May 2019 23:59 To: SciPy Developers List > Subject: Re: [SciPy-Dev] Backend support for fftpack - GSoC 2019 On Wed, May 8, 2019 at 5:50 PM Peter Bell > wrote: Hello all, I?m happy to say my GSoC proposal was accepted, Hey Peter, congratulations! We're happy to have you on board! so I?ll be working over the summer to add support for multiple fftpack backends. For those interested, I?ve extracted the main discussion from my proposal which you can read here: https://docs.google.com/document/d/1hGMc6ZrsmSi_LSixI5oFbo5cZkPUo92sfmnVthg1u6Y/edit Thanks, interesting. Will take a little while to process. I just give some very initial thoughts below (so take with a grain of salt). There are a several design decisions I raise in the proposal and any comments would be appreciated. Of particular note: * Should backends be allowed to add functionality or should they be completely interchangeable. E.g. unlike fftpack, FFTW has support for long doubles; should this be exposed to SciPy users? If a user writes backend-specific code, then they may just as well bypass the backend right, and call the other library directly? Although for long double things will work out of the box perhaps, because the dtype of an input array doesn't come back in the API. So the array can be passed straight to the backend. * Is there any interest in adding a SciPy config file? For just one option it?s probably not worthwhile but if there is interest in more config options then it starts to make more sense. My sense is no. Something like a context manager (`with backend('pyfftw'):`) is lightweight enough probably. This is probably also a good time to point out http://uarray.readthedocs.io/. It can provide a complete backend system without too much work, it was designed exactly for the type of problem you're trying to solve here. Would be good to have your thoughts on that. Ideally a clear set of design goals can be agreed upon now, before it gets too far into the coding period which begins at the end of the month. Yes great idea. Cheers, Ralf _______________________________________________ SciPy-Dev mailing list SciPy-Dev at python.org https://mail.python.org/mailman/listinfo/scipy-dev _______________________________________________ SciPy-Dev mailing list SciPy-Dev at python.org https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From einstein.edison at gmail.com Mon Jun 10 16:22:48 2019 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Mon, 10 Jun 2019 22:22:48 +0200 Subject: [SciPy-Dev] Backend support for fftpack - GSoC 2019 In-Reply-To: References: <80afec54-ef46-485f-b0b2-e5d3ec682f0b@Canary> Message-ID: <310a16c5135ed4f9bb6ca4dbc694b61a4e9e4e84.camel@gmail.com> On Mon, 2019-06-10 at 19:51 +0000, Peter Bell wrote: > >This was based on the feedback of the NumPy team > > > > Is this feedback available anywhere? I?ve had a look through the > NumPy-Discussions list but couldn?t see anything. Unfortunately not, this was at the NumPy dev summit at Berkeley and done in person. > > > > If you have code that is dependent on the last version, I?d be > happy to revert my changes. > > > > Don?t worry, I just want a sense of where the project is currently. That's great. Maybe I should be clearer in my wording: NumPy and SciPy are major potential users, and I'm willing to take uarray in the direction that's needed for those projects. > > Peter > > > > From: SciPy-Dev > On Behalf Of Hameer Abbasi > > Sent: 10 June 2019 19:40 > > To: SciPy Developers List > > Subject: Re: [SciPy-Dev] Backend support for fftpack - GSoC 2019 > > > > > > Hi Peter, > > > > > > This was based on the feedback of the NumPy team, which preferred > protocols over an OOP based model and wanted to avoid deep nesting. > It also bought down the overhead factor tenfold. > > > > > > I also removed an arg/kwarg normalisation feature, which bought down > the overhead another 15-fold. > > > > > > Ralf and I had some offline discussions about the overhead, which > seemed to be more of a blocker than all of the other features > combined. > > > > > > Rest assured, this was out of a desire to respond to feedback from > the NumPy and SciPy teams and once there is code actually relying on > uarray, we will not change the API any further. > > > > > > If you have code that is dependent on the last version, I?d be happy > to revert my changes. > > > > > > > > > Best Regards, > > Hameer Abbasi > > > > > > > > > > > On Monday, Jun 10, 2019 at 4:49 PM, Peter Bell < > > PeterBell10 at live.co.uk> wrote: > > > > > > > > I?ve had another look at the uarray docs and was quite surprised to > > see that the API has changed significantly since I last looked at > > them. > > It seems that uarray is now based on a __ua_function__ > > protocol which is similar to NEP 18 except that it?s held in the > > backend rather than the array type. I think I?ve tracked the > > change to > > PR #152 which is dated after the API was claimed to be stable. > > > > > > Hameer, would you be able to clarify what state uarray development > > is in: exactly how stable is the API? > > > > Peter > > > > > > > > > > From: SciPy-Dev < > > scipy-dev-bounces+peterbell10=live.co.uk at python.org> > > On Behalf Of Hameer Abbasi > > > > Sent: 09 May 2019 13:39 > > > > To: SciPy Developers List > > > > Subject: Re: [SciPy-Dev] Backend support for fftpack - GSoC 2019 > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > Since uarray came up: I?m the author. Yes, performance is slow > > (since it?s pure python) but the API is stable at this point (the > > docs are a bit out of > > date, I should update them). > > > > > > > > > > > > We?re welcoming any kind of contributions as I, myself, am not > > familiar enough with the CPython API to do it. However, I?d love to > > learn! > > > > > > > > > > > > It does have a focus on array computing but it is meant to be > > fairly general, and you shouldn?t have any problems with the > > interface. > > > > > > > > > > > > > > > > > > > > > > Best Regards, > > > > Hameer Abbasi > > > > > > > > > > > > > > > > > > > > On Thursday, May 09, 2019 at 8:25 AM, Peter Bell < > > > PeterBell10 at live.co.uk> wrote: > > > > > > > > > > > > >> Should backends be allowed to add functionality or should they > > > be completely > > > >> interchangeable. E.g. unlike fftpack, FFTW has support for > > > long doubles; > > > >> should this be exposed to SciPy users? > > > > > > > > If a user writes backend-specific code, then they may just as > > > well bypass the > > > > backend right, and call the other library directly? > > > > > > I agree, in fact I quote you in the proposal saying something > > > similar. > > > > > > > Although for long double things will work out of the box > > > perhaps, because the > > > > dtype of an input array doesn't come back in the API. So the > > > array can be > > > > passed straight to the backend. > > > > > > This was why I picked it as an example. It doesn't add any new > > > parameters or > > > change the function signature at all, but it does change the > > > result of a > > > particular call to the API. So it's a very minimal change yet it > > > still breaks > > > the interchangeability of backends. > > > > This is probably also a good time to point out > > > http://uarray.readthedocs.io/. > > > > It can provide a complete backend system without too much work > > > > > > After reading through the docs it does seem to cover this and > > > more. However, it > > > also seems to be unstable at this point so I?m not sure it?s wise > > > to rely on it just > > > yet. > > > > > > > it was designed exactly for the type of problem you're trying > > > to solve here. > > > > > > Unless I'm misunderstanding it seems to be more focused on > > > supporting different > > > types of array, rather than different implementations for NumPy > > > arrays. I can't > > > see any utility in having a complicated dispatch mechanism when > > > we really only > > > want to have a single backend at a time for any given function. > > > > > > I would assume that because of this, uarray would come with a > > > greater > > > performance overhead. Whereas I think the overhead of an fftpack > > > backend should > > > just be some argument validation and a single function call. I > > > could always > > > implement it both ways and benchmark it if there's interest. > > > > > > Peter > > > > > > From: SciPy-Dev < > > > scipy-dev-bounces+peterbell10=live.co.uk at python.org> > > > On Behalf Of Ralf Gommers > > > > > > Sent: 08 May 2019 23:59 > > > > > > To: SciPy Developers List > > > > > > Subject: Re: [SciPy-Dev] Backend support for fftpack - GSoC 2019 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, May 8, 2019 at 5:50 PM Peter Bell > > > wrote: > > > > > > > > > > > Hello all, > > > > > > > > I?m happy to say my GSoC proposal was accepted, > > > > > > > > > > > > > > > > > > > > > > > > > > > Hey Peter, congratulations! We're happy to have you on board! > > > > > > > > > > > > > > > > > > > > so I?ll be working over the summer to add support for multiple > > > > fftpack backends. For those interested, I?ve extracted the main > > > > discussion from my proposal which you can read here: > > > > > > > > https://docs.google.com/document/d/1hGMc6ZrsmSi_LSixI5oFbo5cZkPUo92sfmnVthg1u6Y/edit > > > > > > > > > > > > > > > > > > > > > > > Thanks, interesting. Will take a little while to process. I just > > > give some very initial thoughts below (so take with a grain of > > > salt). > > > > > > > > > > > > > > > > > > > > > > > > There are a several design decisions I raise in the proposal > > > > and any comments would be appreciated. Of particular note: > > > > > > > > * Should backends be allowed to add functionality or should > > > > they be completely interchangeable. E.g. unlike fftpack, FFTW > > > > has support for long doubles; should this be exposed to SciPy > > > > users? > > > > > > > > > > > > > > > > > > > > > > > If a user writes backend-specific code, then they may just as > > > well bypass the backend right, and call the other library > > > directly? Although for long double things will work out of > > > the box perhaps, because the dtype of an input array doesn't > > > come back in the API. So the array can be passed straight to the > > > backend. > > > > > > > > > > > > > > > > > > > > * Is there any interest in adding a SciPy config file? For just > > > > one option it?s probably not worthwhile but if there is > > > > interest in more config options then it starts to make more > > > > sense. > > > > > > > > > > > > > > > > > > > > > > > My sense is no. Something like a context manager (`with > > > backend('pyfftw'):`) is lightweight enough probably. > > > > > > > > > > > > > > > > > > This is probably also a good time to point out > > > http://uarray.readthedocs.io/. It can provide a complete backend > > > system without too much work, it was designed exactly for the > > > type of problem you're trying to solve here. Would be good to > > > have your thoughts on that. > > > > > > > > > > > > > > > > > > > > > > > > Ideally a clear set of design goals can be agreed upon now, > > > > before it gets too far into the coding period which begins at > > > > the end of the month. > > > > > > > > > > > > > > > > > > > > > > > > > > > Yes great idea. > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Ralf > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > SciPy-Dev mailing list > > > > > > SciPy-Dev at python.org > > > > > > https://mail.python.org/mailman/listinfo/scipy-dev > > > > > > > > > > > > > _______________________________________________ > > > > SciPy-Dev mailing list > > > > SciPy-Dev at python.org > > > > https://mail.python.org/mailman/listinfo/scipy-dev > > > > > > > > > > > _______________________________________________SciPy-Dev mailing > listSciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From larson.eric.d at gmail.com Mon Jun 10 16:34:52 2019 From: larson.eric.d at gmail.com (Eric Larson) Date: Mon, 10 Jun 2019 16:34:52 -0400 Subject: [SciPy-Dev] Deprecation of NumPy functions in SciPy root namespace Message-ID: We currently import a few NumPy functions into the root SciPy namespace, meaning you can currently do things like: import scipy scipy.diag(...) However, this seems like something we shouldn't support, as people should do the more standard `import numpy as np; np.diag(...)` instead. The functions we currently import in scipy/__init__.py are: from numpy import * from numpy.random import rand, randn from numpy.fft import fft, ifft from numpy.lib.scimath import * I opened a PR to deprecate these in the 1.4.0 release, marking them for removal in 1.6.0: https://github.com/scipy/scipy/pull/10290 Feel free to chime in if there are any objections to this plan. Ralf for example on the PR mentioned that he was deliberating between 1.6.0 or 2.0.0 being a better target. My vote is for 1.6.0 based on how infrequently (actually never) I have seen this pattern used in the wild, and that it appears to have also been undocumented behavior. -------------- next part -------------- An HTML attachment was scrubbed... URL: From PeterBell10 at live.co.uk Mon Jun 10 16:47:51 2019 From: PeterBell10 at live.co.uk (Peter Bell) Date: Mon, 10 Jun 2019 20:47:51 +0000 Subject: [SciPy-Dev] Deprecation of NumPy functions in SciPy root namespace In-Reply-To: References: Message-ID: > My vote is for 1.6.0 based on how infrequently (actually never) I have seen this pattern used in the wild, and that it appears to have also been undocumented behavior. Looking into this, I was able to find a few sentences that (partially) document this behaviour: Scipy Organization: In addition, many basic array functions from numpy are also available at the top-level of the scipy package. Interaction with NumPy: The top level of scipy also contains functions from numpy and numpy.lib.scimath. However, it is better to use them directly from the numpy module instead. Peter From: SciPy-Dev On Behalf Of Eric Larson Sent: 10 June 2019 21:35 To: SciPy Developers List Subject: [SciPy-Dev] Deprecation of NumPy functions in SciPy root namespace We currently import a few NumPy functions into the root SciPy namespace, meaning you can currently do things like: import scipy scipy.diag(...) However, this seems like something we shouldn't support, as people should do the more standard `import numpy as np; np.diag(...)` instead. The functions we currently import in scipy/__init__.py are: from numpy import * from numpy.random import rand, randn from numpy.fft import fft, ifft from numpy.lib.scimath import * I opened a PR to deprecate these in the 1.4.0 release, marking them for removal in 1.6.0: https://github.com/scipy/scipy/pull/10290 Feel free to chime in if there are any objections to this plan. Ralf for example on the PR mentioned that he was deliberating between 1.6.0 or 2.0.0 being a better target. My vote is for 1.6.0 based on how infrequently (actually never) I have seen this pattern used in the wild, and that it appears to have also been undocumented behavior. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Jun 11 02:49:15 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 11 Jun 2019 08:49:15 +0200 Subject: [SciPy-Dev] Deprecation of NumPy functions in SciPy root namespace In-Reply-To: References: Message-ID: On Mon, Jun 10, 2019 at 10:35 PM Eric Larson wrote: > We currently import a few NumPy functions into the root SciPy namespace, > meaning you can currently do things like: > > import scipy > scipy.diag(...) > > However, this seems like something we shouldn't support, as people should > do the more standard `import numpy as np; np.diag(...)` instead. The > functions we currently import in scipy/__init__.py are: > > from numpy import * > from numpy.random import rand, randn > from numpy.fft import fft, ifft > from numpy.lib.scimath import * > > I opened a PR to deprecate these in the 1.4.0 release, marking them for > removal in 1.6.0: > > https://github.com/scipy/scipy/pull/10290 > > Feel free to chime in if there are any objections to this plan. Ralf for > example on the PR mentioned that he was deliberating between 1.6.0 or 2.0.0 > being a better target. My vote is for 1.6.0 based on how infrequently > (actually never) I have seen this pattern used in the wild, and that it > appears to have also been undocumented behavior. > Based on Peter finding documentation for this, Warren digging up a number of actual usages (comment on the PR), this not really being an issue except for the scipy.fft workaround that needs to be applied, and people not reading deprecation warnings anyway, I'm in favor of deprecating now and then waiting with removal till a 2.0 release. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Jun 11 07:21:40 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 11 Jun 2019 13:21:40 +0200 Subject: [SciPy-Dev] scipy.fft submodule In-Reply-To: References: Message-ID: On Mon, Jun 10, 2019 at 4:22 PM Peter Bell wrote: > Following on from some of the discussion surrounding my backend proposal > and the discussion on gh-10175 > ; I?ve been working on > adding a new scipy.fft submodule. This is now working, tested, documented > and ready for review in gh-10238 > . > > > > The new submodule is almost a drop-in replacement for numpy.fft and > scipy.fftpack, with a few exceptions: > > - Does not include NumPy?s Hermitian transforms (hfft, ihfft) (but > would be simple to add) > - Uses numpy?s conventions for rfft (complex array) instead of fftpack?s > (complex values packed into a real array) > - Convolutions and pseudo-differential operators from fftpack are not > included > > > > The new submodule adds pocketfft to implement the normal FFTs (not yet DCT > and DST) which adds several new features beyond scipy.fftpack: > > - long double transforms > - multi-dimensional real FFTs (rfftn) > - Orthonormal transforms (norm=?ortho? argument) > - Bluestein?s algorithm to avoid the worst case O(n^2) complexity of > FFTPACK > > > > A few implementation details worth noting: > > - Unlike NumPy?s version of pocketfft, this is C++ and uses templates > to supporting the different floating point types > - pocketfft also uses pybind11 which both adds a new build-time > dependency and requires C++11. I believe this is the first use of C++11 in > SciPy. > > I've looked into this a little more. Compiler support for pybind11 is given in https://github.com/pybind/pybind11#supported-compilers. MSVC 2015 Update 3 seems compatible with the lowest version we need as far as I can tell (Python 3.5/3.6 require MSVC 14.0 == 2015; the "Update 3" part should be fine). GCC 4.8 is already the minimum we need today I believe. Minimum Clang versions needed are from 2013, so that's fine too. For Intel compilers a version from 2016 is needed. I think we're fine requiring a more recent version of the Intel compilers. For more exotic platforms, recent enough GCC versions will also do the job. So good to go I think. Cheers, Ralf > - pocketfft has optional support for multithreading using OpenMP but > this is currently disabled (and not compiled in at all). > > > > And finally, there is the rather awkward issue that scipy.fft already > exists and is an alias for the numpy.fft.fft function. Currently I?ve had > to add a workaround to allow the module scipy.fft to be callable, as > discussed in gh-10253 . This > is only relevant to code that imports scipy.fft so I don?t expect it to > be used often, if at all. Hopefully this is a temporary solution and the > NumPy functions can be removed from the scipy namespace in some future > release, or at the very least the FFT functions can be removed. > > > > Peter > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Fri Jun 14 16:50:49 2019 From: toddrjen at gmail.com (Todd) Date: Fri, 14 Jun 2019 16:50:49 -0400 Subject: [SciPy-Dev] Overlap-add convolution implementation Message-ID: Hi all, I am currently working an an implementation of overlap-add convolution [0]. I have a 1D implementation working and I am going to start expanding it to ND. Overlap-add convolution can provide significant performance benefits over FFT-based convolution. Does this make sense for scipy? Assuming it is, I have a couple questions. First is the name. We have "convolve" and "fftconvolve" already. A few options: overall_add_conv oaconvolve oadconvolve Second is how to calculate the optimal chunk size for the convolution. The equation for the number of complex convolutions, according to wikipedia, is: (Nx/(N-M+1))*N*(log2(N)+1) Where Nx is the signal length, M is the filter length, and N is the chunk size. I need to find the minimum of that equation. Nx is a constant scaling factor so I get rid of that. But I still can't find a closed-form solution to the equation. Assuming there is no closed-form solution, I need another approach. What I am doing now is calculating the minimum across all possible power of 2 indices (I calculated the log of the ends, do a linear range, then convert that back to linear steps to avoid doing logarithms on every step). However, this means I miss the fast powers of 3 and 5. I could do the minimum for all powers, but that would take longer. What would be a good trade-off? [0] https://en.wikipedia.org/wiki/Overlap%E2%80%93add_method -------------- next part -------------- An HTML attachment was scrubbed... URL: From deak.andris at gmail.com Fri Jun 14 19:28:32 2019 From: deak.andris at gmail.com (Andras Deak) Date: Sat, 15 Jun 2019 01:28:32 +0200 Subject: [SciPy-Dev] Overlap-add convolution implementation In-Reply-To: References: Message-ID: > Second is how to calculate the optimal chunk size for the convolution. The equation for the number of complex convolutions, according to wikipedia, is: > > (Nx/(N-M+1))*N*(log2(N)+1) > > Where Nx is the signal length, M is the filter length, and N is the chunk size. I need to find the minimum of that equation. Nx is a constant scaling factor so I get rid of that. But I still can't find a closed-form solution to the equation. If I understand correctly you're looking for an N(M) mapping of optimal chunk size, where the given formula for the number of convolutions is minimal. How about generating a (large) range of reasonable M values, numerically evaluating each optimal N_{opt} for the given M and plotting this N_{opt}(M) function, then trying to come up with a reasonable heuristic based on that? If the functions (number of convolutions vs N on one hand and N_{opt} vs M on the other) behave reasonably enough (and hopefully they will) you might be able to come up with a few intervals of M where you can approximate the N_{opt}(M) function with something nice. Basically hand-rolling a piecewise interpolating function for N_{opt}(M) to hard-wire in the implementation. I've never implemented anything like this so I might be completely wrong, but my hunch is that having a "good enough" heuristic is better than risking too much work (in terms of memory and CPU) to find the strictly optimal chunk size on each call. Regards, Andr?s > Assuming there is no closed-form solution, I need another approach. What I am doing now is calculating the minimum across all possible power of 2 indices (I calculated the log of the ends, do a linear range, then convert that back to linear steps to avoid doing logarithms on every step). However, this means I miss the fast powers of 3 and 5. I could do the minimum for all powers, but that would take longer. What would be a good trade-off? > > [0] https://en.wikipedia.org/wiki/Overlap%E2%80%93add_method > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev From ndbecker2 at gmail.com Fri Jun 14 20:12:41 2019 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 14 Jun 2019 20:12:41 -0400 Subject: [SciPy-Dev] Overlap-add convolution implementation In-Reply-To: References: Message-ID: I have implemented overlap-save before, which is similar. Neal On Fri, Jun 14, 2019 at 7:29 PM Andras Deak wrote: > > > Second is how to calculate the optimal chunk size for the convolution. The equation for the number of complex convolutions, according to wikipedia, is: > > > > (Nx/(N-M+1))*N*(log2(N)+1) > > > > Where Nx is the signal length, M is the filter length, and N is the chunk size. I need to find the minimum of that equation. Nx is a constant scaling factor so I get rid of that. But I still can't find a closed-form solution to the equation. > > If I understand correctly you're looking for an N(M) mapping of > optimal chunk size, where the given formula for the number of > convolutions is minimal. How about generating a (large) range of > reasonable M values, numerically evaluating each optimal N_{opt} for > the given M and plotting this N_{opt}(M) function, then trying to come > up with a reasonable heuristic based on that? If the functions (number > of convolutions vs N on one hand and N_{opt} vs M on the other) behave > reasonably enough (and hopefully they will) you might be able to come > up with a few intervals of M where you can approximate the N_{opt}(M) > function with something nice. Basically hand-rolling a piecewise > interpolating function for N_{opt}(M) to hard-wire in the > implementation. > I've never implemented anything like this so I might be completely > wrong, but my hunch is that having a "good enough" heuristic is better > than risking too much work (in terms of memory and CPU) to find the > strictly optimal chunk size on each call. > Regards, > > Andr?s > > > Assuming there is no closed-form solution, I need another approach. What I am doing now is calculating the minimum across all possible power of 2 indices (I calculated the log of the ends, do a linear range, then convert that back to linear steps to avoid doing logarithms on every step). However, this means I miss the fast powers of 3 and 5. I could do the minimum for all powers, but that would take longer. What would be a good trade-off? > > > > [0] https://en.wikipedia.org/wiki/Overlap%E2%80%93add_method > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at python.org > > https://mail.python.org/mailman/listinfo/scipy-dev > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -- Those who don't understand recursion are doomed to repeat it From 3ukip0s02 at sneakemail.com Sat Jun 15 13:01:41 2019 From: 3ukip0s02 at sneakemail.com (3ukip0s02 at sneakemail.com) Date: Sat, 15 Jun 2019 13:01:41 -0400 Subject: [SciPy-Dev] Overlap-add convolution implementation In-Reply-To: References: Message-ID: <21025-1560618124-553550@sneakemail.com> On Sat, Jun 15, 2019 at 12:01 PM Todd wrote: > I am currently working an an implementation of overlap-add convolution > [0]. I have a 1D implementation working and I am going to start expanding > it to ND. Overlap-add convolution can provide significant performance > benefits over FFT-based convolution. Does this make sense for scipy? > Yeah! > First is the name. We have "convolve" and "fftconvolve" already. A few > options: > > overall_add_conv > oaconvolve > oadconvolve > fftconvolve has been folded into convolve and can either be selected manually, or will automatically choose direct convolution, whichever is likely faster. Does it make sense for the OLA implementation to also be added as a convolve() option, using the "method" parameter? -------------- next part -------------- An HTML attachment was scrubbed... URL: From freddyrietdijk at fridh.nl Sat Jun 15 13:47:38 2019 From: freddyrietdijk at fridh.nl (Freddy Rietdijk) Date: Sat, 15 Jun 2019 19:47:38 +0200 Subject: [SciPy-Dev] Overlap-add convolution implementation In-Reply-To: <21025-1560618124-553550@sneakemail.com> References: <21025-1560618124-553550@sneakemail.com> Message-ID: I suppose whether it can be part of `convolve()` depends on whether any additional options are needed. Unless a working heuristic is found the block size will have to be a parameter. In case convolution with a variant signal should be supported the hop size should be considered as well. In my experience you will want to use an iterative solution though when doing convolution with a variant signal. On Sat, Jun 15, 2019 at 7:22 PM <3ukip0s02 at sneakemail.com> wrote: > > > On Sat, Jun 15, 2019 at 12:01 PM Todd wrote: > >> I am currently working an an implementation of overlap-add convolution >> [0]. I have a 1D implementation working and I am going to start expanding >> it to ND. Overlap-add convolution can provide significant performance >> benefits over FFT-based convolution. Does this make sense for scipy? >> > > Yeah! > > >> First is the name. We have "convolve" and "fftconvolve" already. A few >> options: >> >> overall_add_conv >> oaconvolve >> oadconvolve >> > > fftconvolve has been folded into convolve and can either be selected > manually, or will automatically choose direct convolution, whichever is > likely faster. > > Does it make sense for the OLA implementation to also be added as a > convolve() option, using the "method" parameter? > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sun Jun 16 06:01:28 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sun, 16 Jun 2019 12:01:28 +0200 Subject: [SciPy-Dev] Request to add functionality to scipy.stats In-Reply-To: <3690FF8B-1117-4890-9425-F94303D2A9B8@jh.edu> References: <3690FF8B-1117-4890-9425-F94303D2A9B8@jh.edu> Message-ID: Hi Sambit, This sounds interesting, thanks for bringing it up. I hadn't heard of mgcpy before - it looks good. On Thu, Jun 6, 2019 at 6:24 PM Sambit Panda wrote: > Request for project components' inclusion in scipy.stats > > - Project name: "mgcpy" > - Authors: Satish Palaniappan (https://github.com/tpsatish95), Sambit > Panda (https://github.com/sampan501), Junhao Xiong ( > https://github.com/junhaobearxiong), Ananya Swaminathan ( > https://github.com/ananyas713), Sandhya Ramachandran ( > https://github.com/sundaysundya), Richard Guo (https://github.com/rguo123) > - Current repository: https://github.com/neurodata/mgcpy > > "mgcpy" is a Python package containing tools for independence testing and > k-sample testing. Looking through the "scipy.stats" module, The module > contains a host of independence and other hypothesis tests, but are limited > by assumptions of normality, linearity, unidimensionality, etc. While this > may be appropriate in a host of circumstances, it is increasingly important > to analyze nonlinear and high dimensional trends, which is where the > implementations in "mgcpy" could be very useful. Independence tests > included can operate on multidimensional and nonlinear data. In addition, > functionality has been extended to k-sample testing (with capabilities of > operating on the same kinds of data). The tests included can not only be > used for classification, but also for regression. > The papers on which your implementations are based do meet our criteria for inclusion in SciPy. At this point my main questions are about maintenance and dependencies: 1. Why do you want to move your whole package into SciPy? Do you plan to keep maintaining mcgpy? Would you maintain the code included in SciPy? 2. Your code currently depends on pandas, scikit-learn and seaborn. Those are not dependencies of SciPy, and we'd prefer not to add new dependencies. I haven't looked in more detail at this - would it be easy to remove those dependencies? Cheers, Ralf > Below is a list of some of the integrated tests contained within "mgcpy" > and citations for relevant papers about it. > - RV: P. Robert and Y. Escoufier, "A unifying tool for linear multivariate > statistical methods: the rv-coefficient," Journal of the Royal Statistical > Society: Series C (Applied Statistics), vol. 25, no. 3, pp. 257?265, 1976. 3 > - CCA: D. R. Hardoon, S. Szedmak, and J. Shawe-Taylor, "Canonical > correlation analysis: An overview with application to learning methods," > Neural computation, vol. 16, no. 12, pp. 2639?2664, 2004. > - HHG: R. Heller, Y. Heller, and M. Gorfine, "A consistent multivariate > test of association based on ranks of distances," Biometrika, vol. 100, no. > 2, pp. 503?510, 2012. > - MDMR: N. J. Schork and M. A. Zapala, "Statistical properties of > multivariate distance matrix regression for high-dimensional data > analysis," Frontiers in Genetics, vol. 3, p. 190, 2012. > - Biased Dcorr, Unbiased Dcorr**: G. J. Sz?kely, M. L. Rizzo, N. K. > Bakirov et al., "Measuring and testing dependence by correlation of > distances," The Annals of Statistics, vol. 35, no. 6, pp. 2769?2794, 2007. > - Mantel: N. Mantel, "The detection of disease clustering and a > generalized regression approach," Cancer research, vol. 27, no. 2 Part 1, > pp. 209?220, 1967. > - MANOVA: Warne, R. T. (2014). "A primer on multivariate analysis of > variance (MANOVA) for behavioral scientists". Practical Assessment, > Research & Evaluation. 19 (17): 1?10. > - k-sample tests: Mart?nez-Camblor, P., & de U?a-?lvarez, J. (2009). > Non-parametric k-sample tests: Density functions vs distribution functions. > Computational Statistics & Data Analysis, 53(9), 3344-3357. > > Not included tests, but related useful readings: > - Equivalency of Dcorr, HSIC, Energy, and MMD: C. Shen and J. T. > Vogelstein, "The exact equivalence of distance and kernel methods for > hypothesis testing," arXiv preprint arXiv:1806.05514, 2018. > - Formulating k-sample tests as independence tests: C. Shen and J. T. > Vogelstein, "The exact equivalence of distance and kernel methods for > hypothesis testing," arXiv preprint arXiv:1806.05514, 2018. > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Jun 17 04:33:26 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 17 Jun 2019 10:33:26 +0200 Subject: [SciPy-Dev] scipy.org server down Message-ID: Hi, Before it starts raining issues/emails: yes the server is down, we're looking into it. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Jun 17 05:33:38 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 17 Jun 2019 11:33:38 +0200 Subject: [SciPy-Dev] scipy.org server down In-Reply-To: References: Message-ID: It's back up. Not sure yet what the cause was On Mon, Jun 17, 2019 at 10:33 AM Ralf Gommers wrote: > Hi, > > Before it starts raining issues/emails: yes the server is down, we're > looking into it. > > Cheers, > Ralf > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Mon Jun 17 10:01:10 2019 From: toddrjen at gmail.com (Todd) Date: Mon, 17 Jun 2019 10:01:10 -0400 Subject: [SciPy-Dev] Overlap-add convolution implementation In-Reply-To: <21025-1560618124-553550@sneakemail.com> References: <21025-1560618124-553550@sneakemail.com> Message-ID: On Sat, Jun 15, 2019 at 1:22 PM <3ukip0s02 at sneakemail.com> wrote: > > > On Sat, Jun 15, 2019 at 12:01 PM Todd wrote: > >> I am currently working an an implementation of overlap-add convolution >> [0]. I have a 1D implementation working and I am going to start expanding >> it to ND. Overlap-add convolution can provide significant performance >> benefits over FFT-based convolution. Does this make sense for scipy? >> > > Yeah! > > >> First is the name. We have "convolve" and "fftconvolve" already. A few >> options: >> >> overall_add_conv >> oaconvolve >> oadconvolve >> > > fftconvolve has been folded into convolve and can either be selected > manually, or will automatically choose direct convolution, whichever is > likely faster. > > Does it make sense for the OLA implementation to also be added as a > convolve() option, using the "method" parameter? > > The problem is that "convolve" lacks an "axes" argument. fftconvolve supports convolving along an arbitrary subset of dimensions, and this function will as well. But the conventional "convolve" doesn't (yet), so the "convolve" function doesn't, either. So this really needs a stand-alone function, just like fftconvolve (it would still be exposed through "convolve", just not exclusively). It could, in principle, be made a part of fftconvolve, but then we get into what I think are too many nested levels of "method" arguments that I think would get confusing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Mon Jun 17 10:04:45 2019 From: toddrjen at gmail.com (Todd) Date: Mon, 17 Jun 2019 10:04:45 -0400 Subject: [SciPy-Dev] Overlap-add convolution implementation In-Reply-To: References: <21025-1560618124-553550@sneakemail.com> Message-ID: The heuristic I have right now works fine. It is a negligible fraction of overall function run time for the large arrays where this approach makes sense to use in the first place. I am not sure whether it is optimal, but it is certainly usable. Still, I plan to allow people to specify a block size, which would skip that step completely. On Sat, Jun 15, 2019 at 1:48 PM Freddy Rietdijk wrote: > I suppose whether it can be part of `convolve()` depends on whether any > additional options are needed. Unless a working heuristic is found the > block size will have to be a parameter. In case convolution with a variant > signal should be supported the hop size should be considered as well. In my > experience you will want to use an iterative solution though when doing > convolution with a variant signal. > > On Sat, Jun 15, 2019 at 7:22 PM <3ukip0s02 at sneakemail.com> wrote: > >> >> >> On Sat, Jun 15, 2019 at 12:01 PM Todd wrote: >> >>> I am currently working an an implementation of overlap-add convolution >>> [0]. I have a 1D implementation working and I am going to start >>> expanding >>> it to ND. Overlap-add convolution can provide significant performance >>> benefits over FFT-based convolution. Does this make sense for scipy? >>> >> >> Yeah! >> >> >>> First is the name. We have "convolve" and "fftconvolve" already. A few >>> options: >>> >>> overall_add_conv >>> oaconvolve >>> oadconvolve >>> >> >> fftconvolve has been folded into convolve and can either be selected >> manually, or will automatically choose direct convolution, whichever is >> likely faster. >> >> Does it make sense for the OLA implementation to also be added as a >> convolve() option, using the "method" parameter? >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Mon Jun 17 10:16:45 2019 From: toddrjen at gmail.com (Todd) Date: Mon, 17 Jun 2019 10:16:45 -0400 Subject: [SciPy-Dev] Overlap-add convolution implementation In-Reply-To: References: Message-ID: On Fri, Jun 14, 2019 at 7:29 PM Andras Deak wrote: > > Second is how to calculate the optimal chunk size for the convolution. > The equation for the number of complex convolutions, according to > wikipedia, is: > > > > (Nx/(N-M+1))*N*(log2(N)+1) > > > > Where Nx is the signal length, M is the filter length, and N is the > chunk size. I need to find the minimum of that equation. Nx is a constant > scaling factor so I get rid of that. But I still can't find a closed-form > solution to the equation. > > If I understand correctly you're looking for an N(M) mapping of > optimal chunk size, where the given formula for the number of > convolutions is minimal. How about generating a (large) range of > reasonable M values, numerically evaluating each optimal N_{opt} for > the given M and plotting this N_{opt}(M) function, then trying to come > up with a reasonable heuristic based on that? If the functions (number > of convolutions vs N on one hand and N_{opt} vs M on the other) behave > reasonably enough (and hopefully they will) you might be able to come > up with a few intervals of M where you can approximate the N_{opt}(M) > function with something nice. Basically hand-rolling a piecewise > interpolating function for N_{opt}(M) to hard-wire in the > implementation. > I've never implemented anything like this so I might be completely > wrong, but my hunch is that having a "good enough" heuristic is better > than risking too much work (in terms of memory and CPU) to find the > strictly optimal chunk size on each call. > Regards, > I tried that but couldn't figure out anything. I will look again. Strictly speaking it is keeping the number of multiplications minimal, rather than the number of convolutions (no convolutions are actually being done). Keep in mind, though, that if I stick to only powers of two, there are only 32 values between, say, 128 and 1 trillion. The problem with that approach is that we lose powers of 3 and 5, which would more than double the number of values needed. But I am skeptical whether a heuristic would have much advantage in either performance or accuracy compared to simply calculating powers of 2. -------------- next part -------------- An HTML attachment was scrubbed... URL: From spanda3 at jhu.edu Mon Jun 17 14:58:05 2019 From: spanda3 at jhu.edu (Sambit Panda) Date: Mon, 17 Jun 2019 18:58:05 +0000 Subject: [SciPy-Dev] SciPy-Dev Digest, Vol 188, Issue 9 In-Reply-To: References: Message-ID: <38FE0871-2C0D-4E33-AEFB-B723EFDA5572@jh.edu> Hello, Thank you for your response. > 1. Why do you want to move your whole package into SciPy? Do you plan to > keep maintaining mcgpy? Would you maintain the code included in SciPy? Our main motivation for moving the package into SciPy is that we feel that more people would to have access to these useful independence tests. This is important because it appears that currently, most of those in SciPy operate on one-dimensional data vectors or have the assumption that the data is sampled from a normal distribution. We plan on deprecating mgcpy and helping to maintain the code included within SciPy. This includes porting mgcpy into SciPy and conforming mgcpy to the SciPy API where necessary. > 2. Your code currently depends on pandas, scikit-learn and seaborn. Those > are not dependencies of SciPy, and we'd prefer not to add new dependencies. > I haven't looked in more detail at this - would it be easy to remove those > dependencies? These are used when we were generating our demos and power curves to compare the independence tests to help make plots more visually appealing. None of the core independence tests nor their unit tests import anything from those packages. As such, it is not necessary to add them dependencies. Thank you, Sambit Panda > On Jun 16, 2019, at 5:59 AM, scipy-dev-request at python.org wrote: > > > Send SciPy-Dev mailing list submissions to > scipy-dev at python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scipy-dev > or, via email, send a message with subject or body 'help' to > scipy-dev-request at python.org > > You can reach the person managing the list at > scipy-dev-owner at python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of SciPy-Dev digest..." > > > Today's Topics: > > 1. Re: Overlap-add convolution implementation > (3ukip0s02 at sneakemail.com) > 2. Re: Overlap-add convolution implementation (Freddy Rietdijk) > 3. Re: Request to add functionality to scipy.stats (Ralf Gommers) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 15 Jun 2019 13:01:41 -0400 > From: 3ukip0s02 at sneakemail.com > To: scipy-dev at python.org > Subject: Re: [SciPy-Dev] Overlap-add convolution implementation > Message-ID: <21025-1560618124-553550 at sneakemail.com> > Content-Type: text/plain; charset="utf-8" > > On Sat, Jun 15, 2019 at 12:01 PM Todd wrote: > >> I am currently working an an implementation of overlap-add convolution >> [0]. I have a 1D implementation working and I am going to start expanding >> it to ND. Overlap-add convolution can provide significant performance >> benefits over FFT-based convolution. Does this make sense for scipy? >> > > Yeah! > > >> First is the name. We have "convolve" and "fftconvolve" already. A few >> options: >> >> overall_add_conv >> oaconvolve >> oadconvolve >> > > fftconvolve has been folded into convolve and can either be selected > manually, or will automatically choose direct convolution, whichever is > likely faster. > > Does it make sense for the OLA implementation to also be added as a > convolve() option, using the "method" parameter? > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > Message: 2 > Date: Sat, 15 Jun 2019 19:47:38 +0200 > From: Freddy Rietdijk > To: SciPy Developers List > Subject: Re: [SciPy-Dev] Overlap-add convolution implementation > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > I suppose whether it can be part of `convolve()` depends on whether any > additional options are needed. Unless a working heuristic is found the > block size will have to be a parameter. In case convolution with a variant > signal should be supported the hop size should be considered as well. In my > experience you will want to use an iterative solution though when doing > convolution with a variant signal. > > On Sat, Jun 15, 2019 at 7:22 PM <3ukip0s02 at sneakemail.com> wrote: > >> >> >> On Sat, Jun 15, 2019 at 12:01 PM Todd wrote: >> >>> I am currently working an an implementation of overlap-add convolution >>> [0]. I have a 1D implementation working and I am going to start expanding >>> it to ND. Overlap-add convolution can provide significant performance >>> benefits over FFT-based convolution. Does this make sense for scipy? >>> >> >> Yeah! >> >> >>> First is the name. We have "convolve" and "fftconvolve" already. A few >>> options: >>> >>> overall_add_conv >>> oaconvolve >>> oadconvolve >>> >> >> fftconvolve has been folded into convolve and can either be selected >> manually, or will automatically choose direct convolution, whichever is >> likely faster. >> >> Does it make sense for the OLA implementation to also be added as a >> convolve() option, using the "method" parameter? >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > Message: 3 > Date: Sun, 16 Jun 2019 12:01:28 +0200 > From: Ralf Gommers > To: SciPy Developers List > Subject: Re: [SciPy-Dev] Request to add functionality to scipy.stats > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > Hi Sambit, > > This sounds interesting, thanks for bringing it up. I hadn't heard of mgcpy > before - it looks good. > > > On Thu, Jun 6, 2019 at 6:24 PM Sambit Panda wrote: > >> Request for project components' inclusion in scipy.stats >> >> - Project name: "mgcpy" >> - Authors: Satish Palaniappan (https://github.com/tpsatish95), Sambit >> Panda (https://github.com/sampan501), Junhao Xiong ( >> https://github.com/junhaobearxiong), Ananya Swaminathan ( >> https://github.com/ananyas713), Sandhya Ramachandran ( >> https://github.com/sundaysundya), Richard Guo (https://github.com/rguo123) >> - Current repository: https://github.com/neurodata/mgcpy >> >> "mgcpy" is a Python package containing tools for independence testing and >> k-sample testing. Looking through the "scipy.stats" module, The module >> contains a host of independence and other hypothesis tests, but are limited >> by assumptions of normality, linearity, unidimensionality, etc. While this >> may be appropriate in a host of circumstances, it is increasingly important >> to analyze nonlinear and high dimensional trends, which is where the >> implementations in "mgcpy" could be very useful. Independence tests >> included can operate on multidimensional and nonlinear data. In addition, >> functionality has been extended to k-sample testing (with capabilities of >> operating on the same kinds of data). The tests included can not only be >> used for classification, but also for regression. >> > > The papers on which your implementations are based do meet our criteria for > inclusion in SciPy. At this point my main questions are about maintenance > and dependencies: > 1. Why do you want to move your whole package into SciPy? Do you plan to > keep maintaining mcgpy? Would you maintain the code included in SciPy? > 2. Your code currently depends on pandas, scikit-learn and seaborn. Those > are not dependencies of SciPy, and we'd prefer not to add new dependencies. > I haven't looked in more detail at this - would it be easy to remove those > dependencies? > > Cheers, > Ralf > > > >> Below is a list of some of the integrated tests contained within "mgcpy" >> and citations for relevant papers about it. >> - RV: P. Robert and Y. Escoufier, "A unifying tool for linear multivariate >> statistical methods: the rv-coefficient," Journal of the Royal Statistical >> Society: Series C (Applied Statistics), vol. 25, no. 3, pp. 257?265, 1976. 3 >> - CCA: D. R. Hardoon, S. Szedmak, and J. Shawe-Taylor, "Canonical >> correlation analysis: An overview with application to learning methods," >> Neural computation, vol. 16, no. 12, pp. 2639?2664, 2004. >> - HHG: R. Heller, Y. Heller, and M. Gorfine, "A consistent multivariate >> test of association based on ranks of distances," Biometrika, vol. 100, no. >> 2, pp. 503?510, 2012. >> - MDMR: N. J. Schork and M. A. Zapala, "Statistical properties of >> multivariate distance matrix regression for high-dimensional data >> analysis," Frontiers in Genetics, vol. 3, p. 190, 2012. >> - Biased Dcorr, Unbiased Dcorr**: G. J. Sz?kely, M. L. Rizzo, N. K. >> Bakirov et al., "Measuring and testing dependence by correlation of >> distances," The Annals of Statistics, vol. 35, no. 6, pp. 2769?2794, 2007. >> - Mantel: N. Mantel, "The detection of disease clustering and a >> generalized regression approach," Cancer research, vol. 27, no. 2 Part 1, >> pp. 209?220, 1967. >> - MANOVA: Warne, R. T. (2014). "A primer on multivariate analysis of >> variance (MANOVA) for behavioral scientists". Practical Assessment, >> Research & Evaluation. 19 (17): 1?10. >> - k-sample tests: Mart?nez-Camblor, P., & de U?a-?lvarez, J. (2009). >> Non-parametric k-sample tests: Density functions vs distribution functions. >> Computational Statistics & Data Analysis, 53(9), 3344-3357. >> >> Not included tests, but related useful readings: >> - Equivalency of Dcorr, HSIC, Energy, and MMD: C. Shen and J. T. >> Vogelstein, "The exact equivalence of distance and kernel methods for >> hypothesis testing," arXiv preprint arXiv:1806.05514, 2018. >> - Formulating k-sample tests as independence tests: C. Shen and J. T. >> Vogelstein, "The exact equivalence of distance and kernel methods for >> hypothesis testing," arXiv preprint arXiv:1806.05514, 2018. >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > > ------------------------------ > > End of SciPy-Dev Digest, Vol 188, Issue 9 > ***************************************** > From larson.eric.d at gmail.com Tue Jun 18 10:00:42 2019 From: larson.eric.d at gmail.com (Eric Larson) Date: Tue, 18 Jun 2019 10:00:42 -0400 Subject: [SciPy-Dev] Overlap-add convolution implementation In-Reply-To: References: <21025-1560618124-553550@sneakemail.com> Message-ID: I'd be fine with adding a new function `oaconvolve` or `olaconvolve` for this. It's sufficiently different implementation-wise that a new name seems reasonable, and it also matches how people typically probably think about doing a convolution (time-domain/convolve, frequency-domain/fftconvolve, or OLA/olaconvolve). It's something that has been implemented elsewhere because SciPy doesn't have it, so it would be great to have a proper implementation in SciPy. To keep speed up and release GIL would it be worth implementing in Cython? Eric On Mon, Jun 17, 2019 at 10:05 AM Todd wrote: > The heuristic I have right now works fine. It is a negligible fraction of > overall function run time for the large arrays where this approach makes > sense to use in the first place. I am not sure whether it is optimal, but > it is certainly usable. Still, I plan to allow people to specify a block > size, which would skip that step completely. > > On Sat, Jun 15, 2019 at 1:48 PM Freddy Rietdijk > wrote: > >> I suppose whether it can be part of `convolve()` depends on whether any >> additional options are needed. Unless a working heuristic is found the >> block size will have to be a parameter. In case convolution with a variant >> signal should be supported the hop size should be considered as well. In my >> experience you will want to use an iterative solution though when doing >> convolution with a variant signal. >> >> On Sat, Jun 15, 2019 at 7:22 PM <3ukip0s02 at sneakemail.com> wrote: >> >>> >>> >>> On Sat, Jun 15, 2019 at 12:01 PM Todd wrote: >>> >>>> I am currently working an an implementation of overlap-add convolution >>>> [0]. I have a 1D implementation working and I am going to start >>>> expanding >>>> it to ND. Overlap-add convolution can provide significant performance >>>> benefits over FFT-based convolution. Does this make sense for scipy? >>>> >>> >>> Yeah! >>> >>> >>>> First is the name. We have "convolve" and "fftconvolve" already. A few >>>> options: >>>> >>>> overall_add_conv >>>> oaconvolve >>>> oadconvolve >>>> >>> >>> fftconvolve has been folded into convolve and can either be selected >>> manually, or will automatically choose direct convolution, whichever is >>> likely faster. >>> >>> Does it make sense for the OLA implementation to also be added as a >>> convolve() option, using the "method" parameter? >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Wed Jun 19 10:10:22 2019 From: toddrjen at gmail.com (Todd) Date: Wed, 19 Jun 2019 10:10:22 -0400 Subject: [SciPy-Dev] Overlap-add convolution implementation In-Reply-To: References: <21025-1560618124-553550@sneakemail.com> Message-ID: On Tue, Jun 18, 2019 at 10:01 AM Eric Larson wrote: > I'd be fine with adding a new function `oaconvolve` or `olaconvolve` for > this. It's sufficiently different implementation-wise that a new name seems > reasonable, and it also matches how people typically probably think about > doing a convolution (time-domain/convolve, frequency-domain/fftconvolve, or > OLA/olaconvolve). > That is what I thought as well. > It's something that has been implemented elsewhere > > because SciPy doesn't have it, so it would be great to have a proper > implementation in SciPy. > I have seen some implementations, but that seem to be 1D. The approach I am working on should work with an arbitrary subset of an arbitrary number of dimensions. > To keep speed up and release GIL would it be worth implementing in Cython? > I am not using a loop-based approach, I am reshaping the array then broadcasting in the frequency domain. I already added the pieces to fftconvolve to get the broadcasting working, so what is needed in the new function is the reshaping at the beginning and overlap handling at the end. A loop-based approach would be slower but would use less memory. -------------- next part -------------- An HTML attachment was scrubbed... URL: From larson.eric.d at gmail.com Wed Jun 19 10:18:37 2019 From: larson.eric.d at gmail.com (Eric Larson) Date: Wed, 19 Jun 2019 10:18:37 -0400 Subject: [SciPy-Dev] Overlap-add convolution implementation In-Reply-To: References: <21025-1560618124-553550@sneakemail.com> Message-ID: > > A loop-based approach would be slower but would use less memory. > I would expect that a Python loop might be slower, but that a good Cython/C implementation of the loop would end up being at least as fast. In the memory-based reshaing approach, an implicit loop will be done under the hood in C anyway (probably by NumPy in array ops / the FFT routines with an axis arg). But this is really a performance concern that we can deal with later. We could start with the simpler reshape Python approach (with some basic benchmarks in place) and then some follow-up PR could use Cython to reduce the memory performance, showing it does not hurt (or maybe helps) performance. So I'd say it's worth getting an implementation PR going! Eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From josh.craig.wilson at gmail.com Thu Jun 20 00:01:16 2019 From: josh.craig.wilson at gmail.com (Joshua Wilson) Date: Wed, 19 Jun 2019 21:01:16 -0700 Subject: [SciPy-Dev] Enabling use of scipy.special from Numba Message-ID: Hey everyone, For a while I've been thinking about how we can improve the compatibility of `scipy.special` with Numba. There is a way to do that currently by using `cython_special`: https://github.com/numba/numba/issues/3086 but that has a few drawbacks: (1) it's not very user friendly (2) it relies on the (I think?) private implementation details of Cython's `__pyx_capi__` (including the name mangling scheme used for fused functions) (3) it's clumsy for complex functions I think (3) requires changes on the Numba side, e.g. https://github.com/numba/numba/issues/3860 For (1) you could imagine us implementing another Numba specific API, but that runs into some immediate issues: - We'd need Numba as a dependency (which has been discussed: https://mail.python.org/pipermail/scipy-dev/2018-March/022576.html) - We don't want to just keep implementing new APIs For those reasons I'd prefer to find another solution. In an ideal world I'd like to solve (2) in such a way that it's easy for a third-party library to wrap `cython_special` in Numba jitted functions. There is a solution to that already that doesn't involve `__pyx_capi__`: a third-party library writes some Cython glue that exports PyCapsules of function pointers to the specializations of all the `cython_special` functions and then you grab those using ctypes to construct the jitted functions. For the Cython side of that see the docs on specialization: http://docs.cython.org/en/latest/src/userguide/fusedtypes.html#selecting-specializations But that feels rather convoluted. In an ideal world it would be nice to have an official way to grab the PyCapsules out of `__pyx_capi__` instead of writing a new set of capsules yourself. Note that we run into a similar problem with our own `LowLevelCallable`: https://docs.scipy.org/doc/scipy/reference/generated/scipy.LowLevelCallable.from_cython.html#scipy.LowLevelCallable.from_cython Getting the required name of the exported function for a fused function requires searching through the mangled names for the one you want. Out of all that, my main questions are: - Maybe I'm missing something obvious? Do people have better ideas for improving compatibility? - If not, would it be crazy to talk to the Cython folks about an official method for getting the PyCapsules out of `__pyx_capi__`? - Josh From andyfaff at gmail.com Thu Jun 20 04:35:20 2019 From: andyfaff at gmail.com (Andrew Nelson) Date: Thu, 20 Jun 2019 18:35:20 +1000 Subject: [SciPy-Dev] SciPy paper Message-ID: A paper about SciPy 1.0 is in preparation. We appreciate all contributions made to SciPy, and we want to credit in the paper everyone who has made a substantial contribution. The definition of 'substantial' is "one new feature, significant improvement (e.g. performance or accuracy) to an existing feature, or multiple smaller contributions". We have tried to contact those with GitHub accounts (via the associated email address) who have made at least one commit, issue, pull request, review, or comment at the SciPy main repository prior to the release of SciPy 1.0. However, we realise that this may not have reached contributors whose associated GitHub email address is no longer reachable. If you believe that you have made substantial contributions to the project we ask that you fill out the following (short) form *by Friday, June 28,* if you wish to be considered for inclusion on the author list. SciPy 1.0 Paper Author Form ( https://docs.google.com/forms/d/e/1FAIpQLScApdlW3PvJBhdXvSzzpJ54oAxUnZTNqD8tYpxZVv4G0jkQHQ/viewform ) By submitting the form, you will certify that you have reviewed and agree to the submission of the manuscript ( https://github.com/scipy/scipy-articles/blob/master/scipy-1.0/paper.pdf), and that if selected as an author you will read the final version and take appropriate action if you disagree with any content. The coordination committee has determined that the author order will be 'The SciPy Developers' as first author; maintainers, paper writers, and other key contributors in order of contribution level; and all other authors in alphabetical order. The Author Contributions Statement at the end of the paper will document how the author order was determined. Final decisions about author order will be made by the paper coordination committee with a deciding vote, if needed, provided by the Chair of the SciPy Steering Council, Ralf Gommers. Once authorship decisions are made, those who submitted the form linked above will receive an email with their status and a link to a final version of the paper, which we intend to submit to Scientific Reports . Please post an issue on the scipy-articles GitHub repository ( https://github.com/scipy/scipy-articles) if you have any questions or comments so that a member of the coordination committee can respond. Thanks to everyone for your contributions to SciPy! -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.vanmulbregt at comcast.net Thu Jun 20 07:49:28 2019 From: p.vanmulbregt at comcast.net (Paul van Mulbregt) Date: Thu, 20 Jun 2019 07:49:28 -0400 Subject: [SciPy-Dev] Enabling use of scipy.special from Numba In-Reply-To: References: Message-ID: <87481890-99A3-42DD-B6DF-12D9C2E5780F@comcast.net> Hi Josh, For those of us who aren't intimately familiar with the incompatibilities, could you describe the issue(s) that you run into when mixing the two? Paul > On Jun 20, 2019, at 12:01 AM, Joshua Wilson wrote: > > Hey everyone, > > For a while I've been thinking about how we can improve the > compatibility of `scipy.special` with Numba. There is a way to do that > currently by using `cython_special`: > > https://github.com/numba/numba/issues/3086 > > but that has a few drawbacks: > > (1) it's not very user friendly > (2) it relies on the (I think?) private implementation details of > Cython's `__pyx_capi__` (including the name mangling scheme used for > fused functions) > (3) it's clumsy for complex functions > > I think (3) requires changes on the Numba side, e.g. > > https://github.com/numba/numba/issues/3860 > > For (1) you could imagine us implementing another Numba specific API, > but that runs into some immediate issues: > > - We'd need Numba as a dependency (which has been discussed: > https://mail.python.org/pipermail/scipy-dev/2018-March/022576.html) > - We don't want to just keep implementing new APIs > > For those reasons I'd prefer to find another solution. In an ideal > world I'd like to solve (2) in such a way that it's easy for a > third-party library to wrap `cython_special` in Numba jitted > functions. There is a solution to that already that doesn't involve > `__pyx_capi__`: a third-party library writes some Cython glue that > exports PyCapsules of function pointers to the specializations of all > the `cython_special` functions and then you grab those using ctypes to > construct the jitted functions. For the Cython side of that see the > docs on specialization: > > http://docs.cython.org/en/latest/src/userguide/fusedtypes.html#selecting-specializations > > But that feels rather convoluted. In an ideal world it would be nice > to have an official way to grab the PyCapsules out of `__pyx_capi__` > instead of writing a new set of capsules yourself. Note that we run > into a similar problem with our own `LowLevelCallable`: > > https://docs.scipy.org/doc/scipy/reference/generated/scipy.LowLevelCallable.from_cython.html#scipy.LowLevelCallable.from_cython > > Getting the required name of the exported function for a fused > function requires searching through the mangled names for the one you > want. > > Out of all that, my main questions are: > > - Maybe I'm missing something obvious? Do people have better ideas for > improving compatibility? > - If not, would it be crazy to talk to the Cython folks about an > official method for getting the PyCapsules out of `__pyx_capi__`? > > - Josh > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev From juanlu001 at gmail.com Thu Jun 20 08:30:18 2019 From: juanlu001 at gmail.com (Juan Luis Cano) Date: Thu, 20 Jun 2019 14:30:18 +0200 Subject: [SciPy-Dev] Enabling use of scipy.special from Numba In-Reply-To: <87481890-99A3-42DD-B6DF-12D9C2E5780F@comcast.net> References: <87481890-99A3-42DD-B6DF-12D9C2E5780F@comcast.net> Message-ID: (Copying here a comment I left in the original numba issue) FWIW, this is an experiment I did to wrap directly the underlying C code with CFFI and call it from numba: https://github.com/poliastro/pycephes/tree/v0.1.0 ([Explanatory article translated to English]( https://translate.google.com/translate?sl=auto&tl=en&u=https%3A%2F%2Fwww.pybonacci.org%2F2016%2F02%2F07%2Fcomo-crear-extensiones-en-c-para-python-usando-cffi-y-numba%2F )) @horta then followed a similar approach in https://github.com/limix/hcephes. On Thu, Jun 20, 2019 at 1:54 PM Paul van Mulbregt wrote: > Hi Josh, > For those of us who aren't intimately familiar with the incompatibilities, > could you describe the issue(s) that you run into when mixing the two? > > Paul > > > On Jun 20, 2019, at 12:01 AM, Joshua Wilson > wrote: > > > > Hey everyone, > > > > For a while I've been thinking about how we can improve the > > compatibility of `scipy.special` with Numba. There is a way to do that > > currently by using `cython_special`: > > > > https://github.com/numba/numba/issues/3086 > > > > but that has a few drawbacks: > > > > (1) it's not very user friendly > > (2) it relies on the (I think?) private implementation details of > > Cython's `__pyx_capi__` (including the name mangling scheme used for > > fused functions) > > (3) it's clumsy for complex functions > > > > I think (3) requires changes on the Numba side, e.g. > > > > https://github.com/numba/numba/issues/3860 > > > > For (1) you could imagine us implementing another Numba specific API, > > but that runs into some immediate issues: > > > > - We'd need Numba as a dependency (which has been discussed: > > https://mail.python.org/pipermail/scipy-dev/2018-March/022576.html) > > - We don't want to just keep implementing new APIs > > > > For those reasons I'd prefer to find another solution. In an ideal > > world I'd like to solve (2) in such a way that it's easy for a > > third-party library to wrap `cython_special` in Numba jitted > > functions. There is a solution to that already that doesn't involve > > `__pyx_capi__`: a third-party library writes some Cython glue that > > exports PyCapsules of function pointers to the specializations of all > > the `cython_special` functions and then you grab those using ctypes to > > construct the jitted functions. For the Cython side of that see the > > docs on specialization: > > > > > http://docs.cython.org/en/latest/src/userguide/fusedtypes.html#selecting-specializations > > > > But that feels rather convoluted. In an ideal world it would be nice > > to have an official way to grab the PyCapsules out of `__pyx_capi__` > > instead of writing a new set of capsules yourself. Note that we run > > into a similar problem with our own `LowLevelCallable`: > > > > > https://docs.scipy.org/doc/scipy/reference/generated/scipy.LowLevelCallable.from_cython.html#scipy.LowLevelCallable.from_cython > > > > Getting the required name of the exported function for a fused > > function requires searching through the mangled names for the one you > > want. > > > > Out of all that, my main questions are: > > > > - Maybe I'm missing something obvious? Do people have better ideas for > > improving compatibility? > > - If not, would it be crazy to talk to the Cython folks about an > > official method for getting the PyCapsules out of `__pyx_capi__`? > > > > - Josh > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at python.org > > https://mail.python.org/mailman/listinfo/scipy-dev > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -- Juan Luis Cano -------------- next part -------------- An HTML attachment was scrubbed... URL: From sseibert at anaconda.com Thu Jun 20 11:37:40 2019 From: sseibert at anaconda.com (Stanley Seibert) Date: Thu, 20 Jun 2019 10:37:40 -0500 Subject: [SciPy-Dev] Enabling use of scipy.special from Numba In-Reply-To: References: Message-ID: (Numba dev here...) I don't think SciPy would need to take Numba as a dependency to solve this, any more than cffi or ctypes does. All we need on the Numba side is a standard way, given a scipy.special function object to obtain the underlying C function pointer and the C type signature (basic types, no Python objects) of that function. If there is a standard set of attributes for us to pull that information from, the adapter can live inside the Numba code base and SciPy needs minimal changes. On Wed, Jun 19, 2019 at 11:02 PM Joshua Wilson wrote: > Hey everyone, > > For a while I've been thinking about how we can improve the > compatibility of `scipy.special` with Numba. There is a way to do that > currently by using `cython_special`: > > https://github.com/numba/numba/issues/3086 > > but that has a few drawbacks: > > (1) it's not very user friendly > (2) it relies on the (I think?) private implementation details of > Cython's `__pyx_capi__` (including the name mangling scheme used for > fused functions) > (3) it's clumsy for complex functions > > I think (3) requires changes on the Numba side, e.g. > > https://github.com/numba/numba/issues/3860 > > For (1) you could imagine us implementing another Numba specific API, > but that runs into some immediate issues: > > - We'd need Numba as a dependency (which has been discussed: > https://mail.python.org/pipermail/scipy-dev/2018-March/022576.html) > - We don't want to just keep implementing new APIs > > For those reasons I'd prefer to find another solution. In an ideal > world I'd like to solve (2) in such a way that it's easy for a > third-party library to wrap `cython_special` in Numba jitted > functions. There is a solution to that already that doesn't involve > `__pyx_capi__`: a third-party library writes some Cython glue that > exports PyCapsules of function pointers to the specializations of all > the `cython_special` functions and then you grab those using ctypes to > construct the jitted functions. For the Cython side of that see the > docs on specialization: > > > http://docs.cython.org/en/latest/src/userguide/fusedtypes.html#selecting-specializations > > But that feels rather convoluted. In an ideal world it would be nice > to have an official way to grab the PyCapsules out of `__pyx_capi__` > instead of writing a new set of capsules yourself. Note that we run > into a similar problem with our own `LowLevelCallable`: > > > https://docs.scipy.org/doc/scipy/reference/generated/scipy.LowLevelCallable.from_cython.html#scipy.LowLevelCallable.from_cython > > Getting the required name of the exported function for a fused > function requires searching through the mangled names for the one you > want. > > Out of all that, my main questions are: > > - Maybe I'm missing something obvious? Do people have better ideas for > improving compatibility? > - If not, would it be crazy to talk to the Cython folks about an > official method for getting the PyCapsules out of `__pyx_capi__`? > > - Josh > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josh.craig.wilson at gmail.com Thu Jun 20 12:18:27 2019 From: josh.craig.wilson at gmail.com (Joshua Wilson) Date: Thu, 20 Jun 2019 09:18:27 -0700 Subject: [SciPy-Dev] Enabling use of scipy.special from Numba In-Reply-To: References: Message-ID: <738C88BF-0A5A-4522-AE01-00E3C34F497D@gmail.com> > On Jun 20, 2019, at 08:37, Stanley Seibert wrote: > > All we need on the Numba side is a standard way, given a scipy.special function object to obtain the underlying C function pointer and the C type signature (basic types, no Python objects) of that function. Good, this is what I hoped the answer was. So as I see it, SciPy could either: (1) extend the cython_special API to make the pointers more available or (2) Talk to the Cython devs about making the function pointers Cython already creates more available. The SciPy changes are then probably just some sort of index of the Cython pointers so that they are easy to find. From josh.craig.wilson at gmail.com Thu Jun 20 12:21:26 2019 From: josh.craig.wilson at gmail.com (Joshua Wilson) Date: Thu, 20 Jun 2019 09:21:26 -0700 Subject: [SciPy-Dev] Enabling use of scipy.special from Numba In-Reply-To: <87481890-99A3-42DD-B6DF-12D9C2E5780F@comcast.net> References: <87481890-99A3-42DD-B6DF-12D9C2E5780F@comcast.net> Message-ID: <0E9FE0C7-F84F-45EC-B4E2-5F57C3B6869E@gmail.com> > On Jun 20, 2019, at 04:49, Paul van Mulbregt wrote: > > could you describe the issue(s) that you run into when mixing the two? As far as I know, the two issues are: - potential ABI incompatibility for complex types - getting function pointers from Cython modules is less user friendly than we would like and I think relies on Cython implementation details From ralf.gommers at gmail.com Thu Jun 20 14:22:18 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 20 Jun 2019 20:22:18 +0200 Subject: [SciPy-Dev] Enabling use of scipy.special from Numba In-Reply-To: References: Message-ID: On Thu, Jun 20, 2019 at 6:01 AM Joshua Wilson wrote: > Hey everyone, > > For a while I've been thinking about how we can improve the > compatibility of `scipy.special` with Numba. There is a way to do that > currently by using `cython_special`: > > https://github.com/numba/numba/issues/3086 > > but that has a few drawbacks: > > (1) it's not very user friendly > (2) it relies on the (I think?) private implementation details of > Cython's `__pyx_capi__` (including the name mangling scheme used for > fused functions) > Is there a difference on our side between cython_special and cython_blas/lapack? Because Numba already interfaces with those to accelerate numpy.linalg. We'd like Numba to understand scipy.linalg too (don't think it does, but didn't check recently), and ideally all those interfaces would work the same way. Cheers, Ralf (3) it's clumsy for complex functions > > I think (3) requires changes on the Numba side, e.g. > > https://github.com/numba/numba/issues/3860 > > For (1) you could imagine us implementing another Numba specific API, > but that runs into some immediate issues: > > - We'd need Numba as a dependency (which has been discussed: > https://mail.python.org/pipermail/scipy-dev/2018-March/022576.html) > - We don't want to just keep implementing new APIs > > For those reasons I'd prefer to find another solution. In an ideal > world I'd like to solve (2) in such a way that it's easy for a > third-party library to wrap `cython_special` in Numba jitted > functions. There is a solution to that already that doesn't involve > `__pyx_capi__`: a third-party library writes some Cython glue that > exports PyCapsules of function pointers to the specializations of all > the `cython_special` functions and then you grab those using ctypes to > construct the jitted functions. For the Cython side of that see the > docs on specialization: > > > http://docs.cython.org/en/latest/src/userguide/fusedtypes.html#selecting-specializations > > But that feels rather convoluted. In an ideal world it would be nice > to have an official way to grab the PyCapsules out of `__pyx_capi__` > instead of writing a new set of capsules yourself. Note that we run > into a similar problem with our own `LowLevelCallable`: > > > https://docs.scipy.org/doc/scipy/reference/generated/scipy.LowLevelCallable.from_cython.html#scipy.LowLevelCallable.from_cython > > Getting the required name of the exported function for a fused > function requires searching through the mangled names for the one you > want. > > Out of all that, my main questions are: > > - Maybe I'm missing something obvious? Do people have better ideas for > improving compatibility? > - If not, would it be crazy to talk to the Cython folks about an > official method for getting the PyCapsules out of `__pyx_capi__`? > > - Josh > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Jun 20 14:39:44 2019 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 20 Jun 2019 11:39:44 -0700 Subject: [SciPy-Dev] Enabling use of scipy.special from Numba In-Reply-To: References: Message-ID: On Thu, Jun 20, 2019, 08:38 Stanley Seibert wrote: > (Numba dev here...) > > I don't think SciPy would need to take Numba as a dependency to solve > this, any more than cffi or ctypes does. All we need on the Numba side is > a standard way, given a scipy.special function object to obtain the > underlying C function pointer and the C type signature (basic types, no > Python objects) of that function. If there is a standard set of attributes > for us to pull that information from, the adapter can live inside the Numba > code base and SciPy needs minimal changes. > I think scipy.special functions are all ufuncs, so I guess there is already a way to pull out the underlying loop pointers if you're willing to mess around with the numpy C API. And numba hasn't shown any hesitation about that in the past :-). Is there a reason numba can't pull out the loops already, or why this is a bad approach to approach to aim for? -n > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sseibert at anaconda.com Thu Jun 20 16:03:28 2019 From: sseibert at anaconda.com (Stanley Seibert) Date: Thu, 20 Jun 2019 15:03:28 -0500 Subject: [SciPy-Dev] Enabling use of scipy.special from Numba In-Reply-To: References: Message-ID: We're not looking for the ufunc, but the underlying scalar call, along with a type signature. Is that exposed? On Thu, Jun 20, 2019 at 1:41 PM Nathaniel Smith wrote: > On Thu, Jun 20, 2019, 08:38 Stanley Seibert wrote: > >> (Numba dev here...) >> >> I don't think SciPy would need to take Numba as a dependency to solve >> this, any more than cffi or ctypes does. All we need on the Numba side is >> a standard way, given a scipy.special function object to obtain the >> underlying C function pointer and the C type signature (basic types, no >> Python objects) of that function. If there is a standard set of attributes >> for us to pull that information from, the adapter can live inside the Numba >> code base and SciPy needs minimal changes. >> > > I think scipy.special functions are all ufuncs, so I guess there is > already a way to pull out the underlying loop pointers if you're willing to > mess around with the numpy C API. And numba hasn't shown any hesitation > about that in the past :-). > > Is there a reason numba can't pull out the loops already, or why this is a > bad approach to approach to aim for? > > -n > >> _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Jun 20 16:29:46 2019 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 20 Jun 2019 13:29:46 -0700 Subject: [SciPy-Dev] Enabling use of scipy.special from Numba In-Reply-To: References: Message-ID: On Thu, Jun 20, 2019, 13:03 Stanley Seibert wrote: > We're not looking for the ufunc, but the underlying scalar call, along > with a type signature. > I know that's what you're looking for. I'm hoping to understand why :-). The loop function pointer is very close to the raw scalar call, and it already has a type signature. Does the extra C level for loop add that much overhead? -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From sseibert at anaconda.com Thu Jun 20 16:41:03 2019 From: sseibert at anaconda.com (Stanley Seibert) Date: Thu, 20 Jun 2019 15:41:03 -0500 Subject: [SciPy-Dev] Enabling use of scipy.special from Numba In-Reply-To: References: Message-ID: Well, we'd prefer not to have an unnecessary indirection (and the overhead of making a nested call to the scalar inside the length 1 loop) since the user is doing their own iteration in Numba, but if that's the only option, we can certainly work around it. What is cython_special doing? I was under the impression that the scalar call is available there... On Thu, Jun 20, 2019 at 3:31 PM Nathaniel Smith wrote: > On Thu, Jun 20, 2019, 13:03 Stanley Seibert wrote: > >> We're not looking for the ufunc, but the underlying scalar call, along >> with a type signature. >> > > I know that's what you're looking for. I'm hoping to understand why :-). > The loop function pointer is very close to the raw scalar call, and it > already has a type signature. Does the extra C level for loop add that much > overhead? > > -n > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josh.craig.wilson at gmail.com Thu Jun 20 20:20:40 2019 From: josh.craig.wilson at gmail.com (Joshua Wilson) Date: Thu, 20 Jun 2019 17:20:40 -0700 Subject: [SciPy-Dev] Enabling use of scipy.special from Numba In-Reply-To: References: Message-ID: > On Jun 20, 2019, at 13:41, Stanley Seibert wrote: > > What is cython_special doing? I was under the impression that the scalar call is available there... It?s just a bunch of scalar functions overloaded using Cython fused types. So the scalar call for a particular type is available, but you have to sort through the Cython fused-types indirection to get it. From njs at pobox.com Thu Jun 20 20:30:26 2019 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 20 Jun 2019 17:30:26 -0700 Subject: [SciPy-Dev] Enabling use of scipy.special from Numba In-Reply-To: References: Message-ID: On Thu, Jun 20, 2019 at 1:41 PM Stanley Seibert wrote: > > Well, we'd prefer not to have an unnecessary indirection (and the overhead of making a nested call to the scalar inside the length 1 loop) since the user is doing their own iteration in Numba, but if that's the only option, we can certainly work around it. Well, it's computers, so there's never just one option :-). But if you can consume existing ufuncs and get 95% of the performance, then it might not be worth building custom infrastructure to expose scipy.special's raw scalar functions + an ad hoc metadata format + ad hoc tools to consume the metadata. Intuitively it seems like a few extra integer operations at entry and exit might not be noticeable at all compared to the cost of computing a bessel function or whatever, but I haven't checked. -n -- Nathaniel J. Smith -- https://vorpus.org From njs at pobox.com Thu Jun 20 23:31:48 2019 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 20 Jun 2019 20:31:48 -0700 Subject: [SciPy-Dev] Recently discovered bug in lots of code mixing C and Fortran Message-ID: Apparently there is a common situation where C code calling into Fortran violates the Fortran ABI, and has been accidentally getting away with it for years, but in the near future it will start causing mysterious crashes: https://lwn.net/SubscriberLink/791393/d04954027a732415/ https://developer.r-project.org/Blog/public/2019/05/15/gfortran-issues-with-lapack/ Both CBLAS and LAPACKE are specifically called out as including broken code, so I guess this is pretty widespread. I don't know if it affects scipy directly, but I'm guessing at least some people reading this list have some code that's affected, so... heads-up. -n -- Nathaniel J. Smith -- https://vorpus.org From warren.weckesser at gmail.com Fri Jun 21 00:02:23 2019 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Fri, 21 Jun 2019 00:02:23 -0400 Subject: [SciPy-Dev] Recently discovered bug in lots of code mixing C and Fortran In-Reply-To: References: Message-ID: On 6/20/19, Nathaniel Smith wrote: > Apparently there is a common situation where C code calling into > Fortran violates the Fortran ABI, and has been accidentally getting > away with it for years, but in the near future it will start causing > mysterious crashes: > > https://lwn.net/SubscriberLink/791393/d04954027a732415/ > https://developer.r-project.org/Blog/public/2019/05/15/gfortran-issues-with-lapack/ > > Both CBLAS and LAPACKE are specifically called out as including broken > code, so I guess this is pretty widespread. I don't know if it affects > scipy directly, but I'm guessing at least some people reading this > list have some code that's affected, so... heads-up. > Thanks Nathaniel. Pauli created an issue for this on the scipy github site earlier today: https://github.com/scipy/scipy/issues/10335 Warren > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > From ralf.gommers at gmail.com Fri Jun 21 22:01:19 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 22 Jun 2019 04:01:19 +0200 Subject: [SciPy-Dev] SciPy-Dev Digest, Vol 188, Issue 9 In-Reply-To: <38FE0871-2C0D-4E33-AEFB-B723EFDA5572@jh.edu> References: <38FE0871-2C0D-4E33-AEFB-B723EFDA5572@jh.edu> Message-ID: Hi Sambit, On Mon, Jun 17, 2019 at 9:16 PM Sambit Panda wrote: > Hello, > > Thank you for your response. > > > 1. Why do you want to move your whole package into SciPy? Do you plan to > > keep maintaining mcgpy? Would you maintain the code included in SciPy? > > Our main motivation for moving the package into SciPy is that we feel that > more people would to have access to these useful independence tests. This > is important because it appears that currently, most of those in SciPy > operate on one-dimensional data vectors or have the assumption that the > data is sampled from a normal distribution. We plan on deprecating mgcpy > and helping to maintain the code included within SciPy. This includes > porting mgcpy into SciPy and conforming mgcpy to the SciPy API where > necessary. > That's good to hear. This would make sense in principle I think. As you can tell from the slow replies though, we're pretty swamped. I looked at the mcgpy repo in some more detail now. There's a lot of code, and it looks very different to the code currently in scipy.stats. Therefore I would suggest not to convert all code in one go and send it as a PR - I'm afraid that we won't have the bandwidth to review it in a timely fashion. What I would suggest instead is to take one algorithm (perhaps MGC?), convert that to the same style (code/docs/test) as in the existing scipy.stats independence tests, and send that as a PR. Cheers, Ralf > > 2. Your code currently depends on pandas, scikit-learn and seaborn. Those > > are not dependencies of SciPy, and we'd prefer not to add new > dependencies. > > I haven't looked in more detail at this - would it be easy to remove > those > > dependencies? > > These are used when we were generating our demos and power curves to > compare the independence tests to help make plots more visually appealing. > None of the core independence tests nor their unit tests import anything > from those packages. As such, it is not necessary to add them dependencies. > > Thank you, > > Sambit Panda > > > On Jun 16, 2019, at 5:59 AM, scipy-dev-request at python.org wrote: > > > > > > Send SciPy-Dev mailing list submissions to > > scipy-dev at python.org > > > > To subscribe or unsubscribe via the World Wide Web, visit > > https://mail.python.org/mailman/listinfo/scipy-dev > > or, via email, send a message with subject or body 'help' to > > scipy-dev-request at python.org > > > > You can reach the person managing the list at > > scipy-dev-owner at python.org > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of SciPy-Dev digest..." > > > > > > Today's Topics: > > > > 1. Re: Overlap-add convolution implementation > > (3ukip0s02 at sneakemail.com) > > 2. Re: Overlap-add convolution implementation (Freddy Rietdijk) > > 3. Re: Request to add functionality to scipy.stats (Ralf Gommers) > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Sat, 15 Jun 2019 13:01:41 -0400 > > From: 3ukip0s02 at sneakemail.com > > To: scipy-dev at python.org > > Subject: Re: [SciPy-Dev] Overlap-add convolution implementation > > Message-ID: <21025-1560618124-553550 at sneakemail.com> > > Content-Type: text/plain; charset="utf-8" > > > > On Sat, Jun 15, 2019 at 12:01 PM Todd wrote: > > > >> I am currently working an an implementation of overlap-add convolution > >> [0]. I have a 1D implementation working and I am going to start > expanding > >> it to ND. Overlap-add convolution can provide significant performance > >> benefits over FFT-based convolution. Does this make sense for scipy? > >> > > > > Yeah! > > > > > >> First is the name. We have "convolve" and "fftconvolve" already. A few > >> options: > >> > >> overall_add_conv > >> oaconvolve > >> oadconvolve > >> > > > > fftconvolve has been folded into convolve and can either be selected > > manually, or will automatically choose direct convolution, whichever is > > likely faster. > > > > Does it make sense for the OLA implementation to also be added as a > > convolve() option, using the "method" parameter? > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: < > http://mail.python.org/pipermail/scipy-dev/attachments/20190615/4b2a1c16/attachment-0001.html > > > > > > ------------------------------ > > > > Message: 2 > > Date: Sat, 15 Jun 2019 19:47:38 +0200 > > From: Freddy Rietdijk > > To: SciPy Developers List > > Subject: Re: [SciPy-Dev] Overlap-add convolution implementation > > Message-ID: > > 76Z7teHwr2_TsqjSX33eoq6S_sg at mail.gmail.com> > > Content-Type: text/plain; charset="utf-8" > > > > I suppose whether it can be part of `convolve()` depends on whether any > > additional options are needed. Unless a working heuristic is found the > > block size will have to be a parameter. In case convolution with a > variant > > signal should be supported the hop size should be considered as well. In > my > > experience you will want to use an iterative solution though when doing > > convolution with a variant signal. > > > > On Sat, Jun 15, 2019 at 7:22 PM <3ukip0s02 at sneakemail.com> wrote: > > > >> > >> > >> On Sat, Jun 15, 2019 at 12:01 PM Todd wrote: > >> > >>> I am currently working an an implementation of overlap-add convolution > >>> [0]. I have a 1D implementation working and I am going to start > expanding > >>> it to ND. Overlap-add convolution can provide significant performance > >>> benefits over FFT-based convolution. Does this make sense for scipy? > >>> > >> > >> Yeah! > >> > >> > >>> First is the name. We have "convolve" and "fftconvolve" already. A > few > >>> options: > >>> > >>> overall_add_conv > >>> oaconvolve > >>> oadconvolve > >>> > >> > >> fftconvolve has been folded into convolve and can either be selected > >> manually, or will automatically choose direct convolution, whichever is > >> likely faster. > >> > >> Does it make sense for the OLA implementation to also be added as a > >> convolve() option, using the "method" parameter? > >> _______________________________________________ > >> SciPy-Dev mailing list > >> SciPy-Dev at python.org > >> https://mail.python.org/mailman/listinfo/scipy-dev > >> > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: < > http://mail.python.org/pipermail/scipy-dev/attachments/20190615/558742ad/attachment-0001.html > > > > > > ------------------------------ > > > > Message: 3 > > Date: Sun, 16 Jun 2019 12:01:28 +0200 > > From: Ralf Gommers > > To: SciPy Developers List > > Subject: Re: [SciPy-Dev] Request to add functionality to scipy.stats > > Message-ID: > > < > CABL7CQhSrexZjkiwvD3VKWkO_R2ow383jyE1V9fdjqhZ6t9LeQ at mail.gmail.com> > > Content-Type: text/plain; charset="utf-8" > > > > Hi Sambit, > > > > This sounds interesting, thanks for bringing it up. I hadn't heard of > mgcpy > > before - it looks good. > > > > > > On Thu, Jun 6, 2019 at 6:24 PM Sambit Panda wrote: > > > >> Request for project components' inclusion in scipy.stats > >> > >> - Project name: "mgcpy" > >> - Authors: Satish Palaniappan (https://github.com/tpsatish95), Sambit > >> Panda (https://github.com/sampan501), Junhao Xiong ( > >> https://github.com/junhaobearxiong), Ananya Swaminathan ( > >> https://github.com/ananyas713), Sandhya Ramachandran ( > >> https://github.com/sundaysundya), Richard Guo ( > https://github.com/rguo123) > >> - Current repository: https://github.com/neurodata/mgcpy > >> > >> "mgcpy" is a Python package containing tools for independence testing > and > >> k-sample testing. Looking through the "scipy.stats" module, The module > >> contains a host of independence and other hypothesis tests, but are > limited > >> by assumptions of normality, linearity, unidimensionality, etc. While > this > >> may be appropriate in a host of circumstances, it is increasingly > important > >> to analyze nonlinear and high dimensional trends, which is where the > >> implementations in "mgcpy" could be very useful. Independence tests > >> included can operate on multidimensional and nonlinear data. In > addition, > >> functionality has been extended to k-sample testing (with capabilities > of > >> operating on the same kinds of data). The tests included can not only be > >> used for classification, but also for regression. > >> > > > > The papers on which your implementations are based do meet our criteria > for > > inclusion in SciPy. At this point my main questions are about maintenance > > and dependencies: > > 1. Why do you want to move your whole package into SciPy? Do you plan to > > keep maintaining mcgpy? Would you maintain the code included in SciPy? > > 2. Your code currently depends on pandas, scikit-learn and seaborn. Those > > are not dependencies of SciPy, and we'd prefer not to add new > dependencies. > > I haven't looked in more detail at this - would it be easy to remove > those > > dependencies? > > > > Cheers, > > Ralf > > > > > > > >> Below is a list of some of the integrated tests contained within "mgcpy" > >> and citations for relevant papers about it. > >> - RV: P. Robert and Y. Escoufier, "A unifying tool for linear > multivariate > >> statistical methods: the rv-coefficient," Journal of the Royal > Statistical > >> Society: Series C (Applied Statistics), vol. 25, no. 3, pp. 257?265, > 1976. 3 > >> - CCA: D. R. Hardoon, S. Szedmak, and J. Shawe-Taylor, "Canonical > >> correlation analysis: An overview with application to learning methods," > >> Neural computation, vol. 16, no. 12, pp. 2639?2664, 2004. > >> - HHG: R. Heller, Y. Heller, and M. Gorfine, "A consistent multivariate > >> test of association based on ranks of distances," Biometrika, vol. 100, > no. > >> 2, pp. 503?510, 2012. > >> - MDMR: N. J. Schork and M. A. Zapala, "Statistical properties of > >> multivariate distance matrix regression for high-dimensional data > >> analysis," Frontiers in Genetics, vol. 3, p. 190, 2012. > >> - Biased Dcorr, Unbiased Dcorr**: G. J. Sz?kely, M. L. Rizzo, N. K. > >> Bakirov et al., "Measuring and testing dependence by correlation of > >> distances," The Annals of Statistics, vol. 35, no. 6, pp. 2769?2794, > 2007. > >> - Mantel: N. Mantel, "The detection of disease clustering and a > >> generalized regression approach," Cancer research, vol. 27, no. 2 Part > 1, > >> pp. 209?220, 1967. > >> - MANOVA: Warne, R. T. (2014). "A primer on multivariate analysis of > >> variance (MANOVA) for behavioral scientists". Practical Assessment, > >> Research & Evaluation. 19 (17): 1?10. > >> - k-sample tests: Mart?nez-Camblor, P., & de U?a-?lvarez, J. (2009). > >> Non-parametric k-sample tests: Density functions vs distribution > functions. > >> Computational Statistics & Data Analysis, 53(9), 3344-3357. > >> > >> Not included tests, but related useful readings: > >> - Equivalency of Dcorr, HSIC, Energy, and MMD: C. Shen and J. T. > >> Vogelstein, "The exact equivalence of distance and kernel methods for > >> hypothesis testing," arXiv preprint arXiv:1806.05514, 2018. > >> - Formulating k-sample tests as independence tests: C. Shen and J. T. > >> Vogelstein, "The exact equivalence of distance and kernel methods for > >> hypothesis testing," arXiv preprint arXiv:1806.05514, 2018. > >> _______________________________________________ > >> SciPy-Dev mailing list > >> SciPy-Dev at python.org > >> https://mail.python.org/mailman/listinfo/scipy-dev > >> > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: < > http://mail.python.org/pipermail/scipy-dev/attachments/20190616/de48bf44/attachment.html > > > > > > ------------------------------ > > > > Subject: Digest Footer > > > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at python.org > > https://mail.python.org/mailman/listinfo/scipy-dev > > > > > > ------------------------------ > > > > End of SciPy-Dev Digest, Vol 188, Issue 9 > > ***************************************** > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From haberland at ucla.edu Sun Jun 23 01:40:47 2019 From: haberland at ucla.edu (Matt Haberland) Date: Sat, 22 Jun 2019 22:40:47 -0700 Subject: [SciPy-Dev] Sprint at SciPy Conference? Message-ID: The SciPy Conference approaches! Who would be interested in a sprint ? Can you participate in-person or virtually? We don't necessarily need a theme, but would you like to recruit help for something in particular? If you already registered a sprint on behalf of the SciPy library, please let me know. Otherwise, I can register (and I'll ask to be located near NumPy so people don't have to choose one or the other). Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From harivallabha.rangarajan at gmail.com Sun Jun 23 07:19:08 2019 From: harivallabha.rangarajan at gmail.com (Harivallabha Rangarajan) Date: Sun, 23 Jun 2019 16:49:08 +0530 Subject: [SciPy-Dev] Contributing to SciPy's Documentation [Google Season Of Docs '19] Message-ID: Hi, I?m Harivallabha Rangarajan, a 3rd year undergraduate student pursuing *B.E. Computer Science* and *M.Sc. Mathematics*, with a *minor* in* Data Science*, at the *Birla Institute of Technology and Science (BITS) Pilani, Hyderabad, *one of India's premier research institutes. I am *deeply interested* in contributing to the *GSoD, 2019* project on Improving the documentation of the scipy.stats module*.* Right now, I'm participating in the *Google Summer of Code, '19 *with *PyMC3* under the NumFOCUS umbrella wherein my project is on implementing the Dirichlet Process module. [Proposal: Dirichlet Processes - GSoC, 2019 ] Therefore, I'm quite familiar with the open-source workflow. I'd be writing tests, documentation, blog posts and a final report for the module implemented - so by the end of the GSoC period, I'd be comfortable with technical writing as well. I have extensively used the scipy.stats module (and am using it for some writing some checks for distributions in PyMC3), so I hope I'd be able to contribute effectively to improving its documentation :) The API references and the documentation of the scipy.stats module, as is, is quite useful and straightforward. However, the module's documentation could be augmented with more extensive tutorials, and links to existing implementations using scipy.stats. By extensive tutorials, I mean notebooks such as these: https://docs.pymc.io/nb_examples/index.html As for adding links to existing implementations: The Dirichlet distribution documentation could direct users to something like https://dp.tdhopper.com/dirichlet-distribution/ In addition (as an ambitious goal, or as a nice-to-have) we could add some programming crestomathies and comparisons: Implementation of, say, a hierarchical regression model in scipy and pymc3 or stan, and compare them. I trust that I have the necessary statistics background to both "improve the tutorial site of the stats module" and "improve the documentation of the reference guide" - the two points mentioned in the SciPy - GSoD project ideas page. The GSoC coding period ends on Aug, 26. After that, I would be able to devote 5-6 hours a day for GSoD. I see this as an opportunity to both contribute to the SciPy docs as well as familiarize myself quite well with the scipy.stats codebase, which would help me contribute to SciPy beyond the GSoD period (in the same manner GSoC enabled me to acquaint myself with the PyMC3 codebase. I will be contributing to PyMC3 well beyond the GSoC period). Besides that, I'm currently working on a thesis project titled *Investigation of Learning Techniques for Regression Problems* with *Dr. Emre Ozkaya*, *Chair of Scientific Computing, TU - Kaiserslautern, *and* Dr. Anil Nemili, Department of Mathematics, BITS Pilani - Hyderabad. * Previously, I had worked on: *Performance Analysis of Algorithmic Differentiation based Adjoint Solvers* under *Dr. Anil Nemili.* I had designed and developed a *Scalable and Robust Journal Recommender System based on Stochastic Gradient Descent (SGD) and Singular Value Decomposition (SVD)*, implemented in Python, during my summer internship (2018) at* Indira Gandhi Centre for Atomic Research (IGCAR)*, Kalpakkam, India under the guidance of *Dr. N. Madurai Meenachi.* My coursework includes Partial Differential Equations, Functional Analysis, Numerical Methods, Optimization, Linear Algebra, Differential Geometry, Ordinary Differential Equations, Multivariable Calculus, Probability and Statistics, Complex Numbers, Real Analysis, Introduction to Topology, Graphs and Networks, Mathematical Methods, Discrete Mathematics, Operations Research, Data Structures and Algorithms, Database Management Systems, Object-Oriented Programming, Microprocessors and Interfacing, Digital Design, and Logic in Computer Science. I have a strong grasp of Data Structures and Algorithms, having implemented them in C++, for Competitive Coding. I?m interested in developing efficient and scalable computational methods for space and time optimizations in Machine Learning. *I would be extremely grateful if you could guide me towards getting started with the contributions, and continuing onward to participate in GSoD '19 with SciPy.* Looking forward to a positive response! Thank you for your time :) Yours sincerely, Harivallabha Rangarajan, BITS Pilani | Hyderabad Campus -------------- next part -------------- An HTML attachment was scrubbed... URL: From cwhanse at sandia.gov Mon Jun 24 12:11:29 2019 From: cwhanse at sandia.gov (Hansen, Clifford W) Date: Mon, 24 Jun 2019 16:11:29 +0000 Subject: [SciPy-Dev] [EXTERNAL] Overlap-add convolution implementation In-Reply-To: References: Message-ID: Optimal chunk size: there is a closed form solution, perhaps the following will help. Write f(N) = N / (N - K) * (log2(N) + 1) = N / (N - K) * log2(2N) Consider N as real, N>=1. df/dN = 1/(N - K) - K/(N - K)^2 * log2(2N) Set df/dN = 0, after some algebra one obtains 4N = (2^(1/K))^N = p^N writing p = 2^(1/K) The solutions to 4N = p^N are in terms of Lambert's W function, see Example 1, Solutions of equations section of wikipedia page for Lambert's W: https://en.wikipedia.org/wiki/Lambert_W_function Two real solutions will be found: one 01 from the W-1 branch. The solution wanted here is N = -1 / log(p) * W-1(-log(p) / 4) which is defined for p < 4/e, which certainly includes K>=1. Code to calculate: import numpy as np from scipy.special import lambertw as W K = 2 p = 2**(1 / K) z = -np.log(p)/4 xopt = -1 / np.log(p) * W(z, k=-1) N = xopt.real ________________________________ From: SciPy-Dev > on behalf of Todd > Sent: Friday, June 14, 2019 2:50 PM To: SciPy Developers List Subject: [EXTERNAL] [SciPy-Dev] Overlap-add convolution implementation Hi all, I am currently working an an implementation of overlap-add convolution [0]. I have a 1D implementation working and I am going to start expanding it to ND. Overlap-add convolution can provide significant performance benefits over FFT-based convolution. Does this make sense for scipy? Assuming it is, I have a couple questions. First is the name. We have "convolve" and "fftconvolve" already. A few options: overall_add_conv oaconvolve oadconvolve Second is how to calculate the optimal chunk size for the convolution. The equation for the number of complex convolutions, according to wikipedia, is: (Nx/(N-M+1))*N*(log2(N)+1) Where Nx is the signal length, M is the filter length, and N is the chunk size. I need to find the minimum of that equation. Nx is a constant scaling factor so I get rid of that. But I still can't find a closed-form solution to the equation. Assuming there is no closed-form solution, I need another approach. What I am doing now is calculating the minimum across all possible power of 2 indices (I calculated the log of the ends, do a linear range, then convert that back to linear steps to avoid doing logarithms on every step). However, this means I miss the fast powers of 3 and 5. I could do the minimum for all powers, but that would take longer. What would be a good trade-off? [0] https://en.wikipedia.org/wiki/Overlap%E2%80%93add_method -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Jun 25 13:50:38 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 25 Jun 2019 19:50:38 +0200 Subject: [SciPy-Dev] Sprint at SciPy Conference? In-Reply-To: References: Message-ID: On Sun, Jun 23, 2019 at 7:42 AM Matt Haberland wrote: > The SciPy Conference approaches! Who > would be interested in a sprint > ? > Can you participate in-person or virtually? > > We don't necessarily need a theme, but would you like to recruit help for > something in particular? > > If you already registered a sprint on behalf of the SciPy library, please > let me know. Otherwise, I can register (and I'll ask to be located near > NumPy so people don't have to choose one or the other). > Thanks Matt, that sounds great. I'll be there the whole weekend. And did not yet register the sprint. Nor do I think anyone else did, so please go for it. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From christoph.baumgarten at gmail.com Tue Jun 25 15:44:16 2019 From: christoph.baumgarten at gmail.com (Christoph Baumgarten) Date: Tue, 25 Jun 2019 21:44:16 +0200 Subject: [SciPy-Dev] Contributing to SciPy's Documentation [Google Season Of Docs '19] In-Reply-To: References: Message-ID: Hi Harivallabha, thanks for your interest to contribute to the documentation of scipy.stats. I think that writing a comprehensive tutorial is a good idea. You probably need to describe the proposal in a more concrete way. Do you have a tutorial in mind what you would be interested to work on? The application deadline is already on June 28th: https://developers.google.com/season-of-docs/docs/timeline Further information on the application process: https://developers.google.com/season-of-docs/docs/tech-writer-application-hints Christoph On Sun, Jun 23, 2019 at 6:03 PM wrote: > Send SciPy-Dev mailing list submissions to > scipy-dev at python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scipy-dev > or, via email, send a message with subject or body 'help' to > scipy-dev-request at python.org > > You can reach the person managing the list at > scipy-dev-owner at python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of SciPy-Dev digest..." > > > Today's Topics: > > 1. Sprint at SciPy Conference? (Matt Haberland) > 2. Contributing to SciPy's Documentation [Google Season Of Docs > '19] (Harivallabha Rangarajan) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 22 Jun 2019 22:40:47 -0700 > From: Matt Haberland > To: SciPy Developers List > Subject: [SciPy-Dev] Sprint at SciPy Conference? > Message-ID: > yET5AEkUyHQ at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > The SciPy Conference approaches! Who > would be interested in a sprint ? > Can you participate in-person or virtually? > > We don't necessarily need a theme, but would you like to recruit help for > something in particular? > > If you already registered a sprint on behalf of the SciPy library, please > let me know. Otherwise, I can register (and I'll ask to be located near > NumPy so people don't have to choose one or the other). > > Matt > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://mail.python.org/pipermail/scipy-dev/attachments/20190622/61ee8dd9/attachment-0001.html > > > > ------------------------------ > > Message: 2 > Date: Sun, 23 Jun 2019 16:49:08 +0530 > From: Harivallabha Rangarajan > To: scipy-dev at python.org > Cc: christoph.baumgarten at gmail.com > Subject: [SciPy-Dev] Contributing to SciPy's Documentation [Google > Season Of Docs '19] > Message-ID: > uXwCw at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi, > > I?m Harivallabha Rangarajan, a 3rd year undergraduate student pursuing > *B.E. > Computer Science* and *M.Sc. Mathematics*, with a *minor* in* Data > Science*, > at the *Birla Institute of Technology and Science (BITS) Pilani, > Hyderabad, *one of India's premier research institutes. I am *deeply > interested* in contributing to the *GSoD, 2019* project on Improving the > documentation of the scipy.stats module*.* > > Right now, I'm participating in the *Google Summer of Code, '19 *with > *PyMC3* under the NumFOCUS umbrella wherein my project is on implementing > the Dirichlet Process module. [Proposal: Dirichlet Processes - GSoC, 2019 > ] > Therefore, I'm quite familiar with the open-source workflow. I'd be writing > tests, documentation, blog posts and a final report for the module > implemented - so by the end of the GSoC period, I'd be comfortable with > technical writing as well. I have extensively used the scipy.stats module > (and am using it for some writing some checks for distributions in PyMC3), > so I hope I'd be able to contribute effectively to improving its > documentation :) > > The API references and the documentation of the scipy.stats module, as is, > is quite useful and straightforward. However, the module's documentation > could be augmented with more extensive tutorials, and links to existing > implementations using scipy.stats. > By extensive tutorials, I mean notebooks such as these: > https://docs.pymc.io/nb_examples/index.html > As for adding links to existing implementations: The Dirichlet distribution > documentation could direct users to something like > https://dp.tdhopper.com/dirichlet-distribution/ > In addition (as an ambitious goal, or as a nice-to-have) we could add some > programming crestomathies and comparisons: Implementation of, say, a > hierarchical regression model in scipy and pymc3 or stan, and compare them. > > I trust that I have the necessary statistics background to both "improve > the tutorial site of the stats module" and "improve the documentation of > the reference guide" - the two points mentioned in the SciPy - GSoD project > ideas page. > > The GSoC coding period ends on Aug, 26. After that, I would be able to > devote 5-6 hours a day for GSoD. I see this as an opportunity to both > contribute to the SciPy docs as well as familiarize myself quite well with > the scipy.stats codebase, which would help me contribute to SciPy beyond > the GSoD period (in the same manner GSoC enabled me to acquaint myself with > the PyMC3 codebase. I will be contributing to PyMC3 well beyond the GSoC > period). > > Besides that, > I'm currently working on a thesis project titled *Investigation of Learning > Techniques for Regression Problems* with *Dr. Emre Ozkaya*, *Chair of > Scientific Computing, TU - Kaiserslautern, *and* Dr. Anil Nemili, > Department of Mathematics, BITS Pilani - Hyderabad. * > Previously, > I had worked on: *Performance Analysis of Algorithmic Differentiation based > Adjoint Solvers* under *Dr. Anil Nemili.* > I had designed and developed a *Scalable and Robust Journal Recommender > System based on Stochastic Gradient Descent (SGD) and Singular Value > Decomposition (SVD)*, implemented in Python, during my summer internship > (2018) at* Indira Gandhi Centre for Atomic Research (IGCAR)*, Kalpakkam, > India under the guidance of *Dr. N. Madurai Meenachi.* > > My coursework includes Partial Differential Equations, Functional Analysis, > Numerical Methods, Optimization, Linear Algebra, Differential Geometry, > Ordinary Differential Equations, Multivariable Calculus, Probability and > Statistics, Complex Numbers, Real Analysis, Introduction to Topology, > Graphs and Networks, Mathematical Methods, Discrete Mathematics, Operations > Research, Data Structures and Algorithms, Database Management Systems, > Object-Oriented Programming, Microprocessors and Interfacing, Digital > Design, and Logic in Computer Science. I have a strong grasp of Data > Structures and Algorithms, having implemented them in C++, for Competitive > Coding. I?m interested in developing efficient and scalable computational > methods for space and time optimizations in Machine Learning. > > *I would be extremely grateful if you could guide me towards getting > started with the contributions, and continuing onward to participate in > GSoD '19 with SciPy.* > > Looking forward to a positive response! Thank you for your time :) > > Yours sincerely, > Harivallabha Rangarajan, > BITS Pilani | Hyderabad Campus > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://mail.python.org/pipermail/scipy-dev/attachments/20190623/7201b57d/attachment-0001.html > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > > ------------------------------ > > End of SciPy-Dev Digest, Vol 188, Issue 19 > ****************************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From scipy-dev at fuglede.dk Thu Jun 27 07:07:51 2019 From: scipy-dev at fuglede.dk (=?iso-8859-1?Q?S=F8ren_Fuglede_J=F8rgensen?=) Date: Thu, 27 Jun 2019 13:07:51 +0200 Subject: [SciPy-Dev] Combinatorial optimization in scipy.optimize In-Reply-To: <20190627080623.6m6pvbnwlvretphx@hackenturet.dk> References: <20190627075201.hna2k5hb6svrzzuj@hackenturet.dk> <20190627080623.6m6pvbnwlvretphx@hackenturet.dk> Message-ID: <20190627110751.zpcin5yzpsjfwbr4@hackenturet.dk> Hi SciPy-Dev This question is motivated by the recent improvements to scipy.optimize.linear_sum_assignment. [0] Here, we get a much improved algorithm for solving minimum weight perfect matching in bipartite graphs (which is excellent). Now, this problem is one in a large family of combinatorial optimization problems on packing and covering, and what I'm wondering is if SciPy should be the home for a general class of efficient algorithms for solving those? Most of these can be phrased in terms of graph theory, and indeed networkx already contains implementations of many classical algorithms (and I was surprised to find that it didn't have anything matching linear_sum_assignment directly; it does have an implementation of the Blossom algorithm). So there's some overlap of scope here, but I guess that's not a problem in and by itself, but it had me wondering where the split should be? Just to phrase this in terms of (a very incomplete list of) examples (some of which have polynomial solutions, some of which don't), would there be appetite for including into scipy.optimize algorithms for solving e.g. 1. other covering/packing problems (e.g. min set/vertex/edge cover, maximum matching, ...), 2. maximum flow, 3. shortest path, 4. travelling salesman, 5. circuit minimization? S?ren [0]: https://github.com/scipy/scipy/pull/10296 From harivallabha.rangarajan at gmail.com Thu Jun 27 16:14:24 2019 From: harivallabha.rangarajan at gmail.com (Harivallabha Rangarajan) Date: Fri, 28 Jun 2019 01:44:24 +0530 Subject: [SciPy-Dev] Contributing to SciPy's Documentation [Google Season Of Docs '19] Message-ID: Hi Christoph, Thanks for the reply! I've attached the proposal for the Google Season of Docs below. I'd be glad to incorporate your suggestions, and I'll submit it if this looks good to you :) Thanks! Yours sincerely, Hari -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: GSoD '19_Proposal_Harivallabha_Rangarajan.pdf Type: application/pdf Size: 246717 bytes Desc: not available URL: From charlesr.harris at gmail.com Thu Jun 27 18:25:58 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 27 Jun 2019 16:25:58 -0600 Subject: [SciPy-Dev] Fwd: 2-factor auth on PyPI & NumPy projects In-Reply-To: <69a68b0c-d0ea-397e-f8f5-c2e41a871b1b@changeset.nyc> References: <69a68b0c-d0ea-397e-f8f5-c2e41a871b1b@changeset.nyc> Message-ID: Forwarding a message from PyPI. I looked at the two factor login a while ago and decided it wasn't for me, but others might be interested. ---------- Forwarded message --------- From: Sumana Harihareswara Date: Thu, Jun 27, 2019 at 4:11 PM Subject: 2-factor auth on PyPI & NumPy projects To: Charles R Harris Dear Mr. Harris: Hi! I found you via the NumPy discussion mailing list, where I saw you are part of the release team. I'm the project manager for PyPI. Right now we're beta testing a new security feature on PyPI: https://pyfound.blogspot.com/2019/06/pypi-now-supports-two-factor-login-via.html 2-factor auth for website login with WebAuthn (hardware devices like Yubikeys). And during this beta, we'd love more testing from package maintainers who use a variety of operating systems, browsers, and browser plugins, including on mobile. There's more info at https://wiki.python.org/psf/WarehousePackageMaintainerTesting . And within the next few months we'll be adding support for API keys for uploading releases to PyPI https://github.com/pypa/warehouse/issues/994 , which should make release automation easier. Could you possibly pass this information on to the NumPy developers' list or another relevant group? Thanks, Sumana -- Sumana Harihareswara PyPI project manager Changeset Consulting https://changeset.nyc -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyler.je.reddy at gmail.com Thu Jun 27 22:35:26 2019 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Thu, 27 Jun 2019 20:35:26 -0600 Subject: [SciPy-Dev] Fwd: 2-factor auth on PyPI & NumPy projects In-Reply-To: References: <69a68b0c-d0ea-397e-f8f5-c2e41a871b1b@changeset.nyc> Message-ID: I've been using the 2FA via third party app for PyPI login. I just have 1password generate the code for me automatically based on the barcode & don't use a hardware key or phone. Maybe just reduces the chance of compromise a little bit. On Thu, 27 Jun 2019 at 16:26, Charles R Harris wrote: > > Forwarding a message from PyPI. I looked at the two factor login a while > ago and decided it wasn't for me, but others might be interested. > > ---------- Forwarded message --------- > From: Sumana Harihareswara > Date: Thu, Jun 27, 2019 at 4:11 PM > Subject: 2-factor auth on PyPI & NumPy projects > To: Charles R Harris > > > Dear Mr. Harris: > > Hi! I found you via the NumPy discussion mailing list, where I saw you > are part of the release team. > > I'm the project manager for PyPI. Right now we're beta testing a new > security feature on PyPI: > > https://pyfound.blogspot.com/2019/06/pypi-now-supports-two-factor-login-via.html > 2-factor auth for website login with WebAuthn (hardware devices like > Yubikeys). And during this beta, we'd love more testing from package > maintainers who use a variety of operating systems, browsers, and > browser plugins, including on mobile. > > There's more info at > https://wiki.python.org/psf/WarehousePackageMaintainerTesting . > > And within the next few months we'll be adding support for API keys for > uploading releases to PyPI https://github.com/pypa/warehouse/issues/994 > , which should make release automation easier. > > Could you possibly pass this information on to the NumPy developers' > list or another relevant group? > > Thanks, > Sumana > -- > Sumana Harihareswara > PyPI project manager > Changeset Consulting > https://changeset.nyc > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dashohoxha at gmail.com Sat Jun 29 14:19:30 2019 From: dashohoxha at gmail.com (Dashamir Hoxha) Date: Sat, 29 Jun 2019 20:19:30 +0200 Subject: [SciPy-Dev] Sprint at SciPy Conference? In-Reply-To: References: Message-ID: On Sun, Jun 23, 2019 at 7:42 AM Matt Haberland wrote: > The SciPy Conference approaches! Who > would be interested in a sprint > ? > Can you participate in-person or virtually? > I would be interested to participate virtually, at least on the tutorial. What do you usually do for virtual participation? Recently I have used successfully Guacamole for collaborating with some online students: https://www.researchgate.net/publication/332469806_Linux_Desktop_In_a_Container I can also offer my infrastructure for this, if needed: https://mint.fs.al:333/guac/ It works well, combined with some videoconferencing application, for example: https://meet.jit.si/ Dashamir -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sun Jun 30 14:10:16 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sun, 30 Jun 2019 20:10:16 +0200 Subject: [SciPy-Dev] Sprint at SciPy Conference? In-Reply-To: References: Message-ID: On Sat, Jun 29, 2019 at 8:19 PM Dashamir Hoxha wrote: > On Sun, Jun 23, 2019 at 7:42 AM Matt Haberland wrote: > >> The SciPy Conference approaches! Who >> would be interested in a sprint >> ? >> Can you participate in-person or virtually? >> > > I would be interested to participate virtually, at least on the tutorial. > There's no tutorial in the sprints normally. The "how to contribute" tutorial (walk through a first pull request) is centrally done by the conference organizers, and people at the SciPy sprint typically dive straight in after some informal discussion with one of the core developers. > What do you usually do for virtual participation? > Usually only via GitHub, and perhaps IRC/Gitter. > Recently I have used successfully Guacamole for collaborating with some > online students: > > https://www.researchgate.net/publication/332469806_Linux_Desktop_In_a_Container > > I can also offer my infrastructure for this, if needed: > https://mint.fs.al:333/guac/ > It works well, combined with some videoconferencing application, for > example: https://meet.jit.si/ > I have to admit I've never tried any kind of videoconferencing during a sprint other than short video calls planned beforehand. I'm not sure it would work well to be honest. Partcipating via our regular channels would be better probably. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Jun 30 18:47:25 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 30 Jun 2019 16:47:25 -0600 Subject: [SciPy-Dev] NumPy 1.17.0rc1 released Message-ID: Hi All, On behalf of the NumPy team I am pleased to announce the release of NumPy 1.17.0rc1. The 1.17 release contains a number of new features that should substantially improve its performance and usefulness. The Python versions supported are 3.5-3.7, note that Python 2.7 has been dropped. Python 3.8b1 should work with the released source packages, but there are no guarantees about future releases. Highlights of this release are: - A new extensible random module along with four selectable random numbe5 generators and improved seeding designed for use in parallel processes has been added. The currently available bit generators are MT19937, PCG64, Philox, and SFC64. - NumPy's FFT implementation was changed from fftpack to pocketfft, resulting in faster, more accurate transforms and better handling of datasets of prime length. - New radix sort and timsort sorting methods. It is currently not possible to choose which will be used, but they are hardwired to the datatype and used when either ``stable`` or ``mergesort`` is passed as the method. - Overriding numpy functions is now possible by default Downstream developers should use Cython >= 0.29.10 for Python 3.8 support and OpenBLAS >= 3.7 (not currently out) to avoid problems on the Skylake architecture. The NumPy wheels on PyPI are built from the OpenBLAS development branch in order to avoid those problems. Wheels for this release can be downloaded from PyPI , source archives and release notes are available from Github . *Contributors* A total of 142 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Aaron Voelker + - Abdur Rehman + - Abdur-Rahmaan Janhangeer + - Abhinav Sagar + - Adam J. Stewart + - Adam Orr + - Albert Thomas + - Alex Watt + - Alexander Blinne + - Alexander Shadchin - Allan Haldane - Ander Ustarroz + - Andras Deak - Andreas Schwab - Andrew Naguib + - Andy Scholand + - Ankit Shukla + - Anthony Sottile - Antoine Pitrou - Antony Lee - Arcesio Castaneda Medina + - Assem + - Bernardt Duvenhage + - Bharat Raghunathan + - Bharat123rox + - Bran + - Bruce Merry + - Charles Harris - Chirag Nighut + - Christoph Gohlke - Christopher Whelan + - Chuanzhu Xu + - Daniel Hrisca - Daniel Lawrence + - Debsankha Manik + - Dennis Zollo + - Dieter Werthm?ller + - Dominic Jack + - EelcoPeacs + - Eric Larson - Eric Wieser - Fabrice Fontaine + - Gary Gurlaskie + - Gregory Lee + - Gregory R. Lee - Hameer Abbasi - Haoyu Sun + - He Jia + - Hunter Damron + - Ian Sanders + - Ilja + - Isaac Virshup + - Isaiah Norton + - Jaime Fernandez - Jakub Wilk - Jan S. (Milania1) + - Jarrod Millman - Javier Dehesa + - Jeremy Lay + - Jim Turner + - Jingbei Li + - Joachim Hereth + - John Belmonte + - John Kirkham - John Law + - Jonas Jensen - Joseph Fox-Rabinovitz - Joseph Martinot-Lagarde - Josh Wilson - Juan Luis Cano Rodr?guez - Julian Taylor - J?r?mie du Boisberranger + - Kai Striega + - Katharine Hyatt + - Kevin Sheppard - Kexuan Sun - Kiko Correoso + - Kriti Singh + - Lars Grueter + - Maksim Shabunin + - Manvi07 + - Mark Harfouche - Marten van Kerkwijk - Martin Reinecke + - Matthew Brett - Matthias Bussonnier - Matti Picus - Michel Fruchart + - Mike Lui + - Mike Taves + - Min ho Kim + - Mircea Akos Bruma - Nick Minkyu Lee - Nick Papior - Nick R. Papior + - Nicola Soranzo + - Nimish Telang + - OBATA Akio + - Oleksandr Pavlyk - Ori Broda + - Paul Ivanov - Pauli Virtanen - Peter Andreas Entschev + - Peter Bell + - Pierre de Buyl - Piyush Jaipuriayar + - Prithvi MK + - Raghuveer Devulapalli + - Ralf Gommers - Richard Harris + - Rishabh Chakrabarti + - Riya Sharma + - Robert Kern - Roman Yurchak - Ryan Levy + - Sebastian Berg - Sergei Lebedev + - Shekhar Prasad Rajak + - Stefan van der Walt - Stephan Hoyer - SuryaChand P + - S?ren Rasmussen + - Thibault Hallouin + - Thomas A Caswell - Tobias Uelwer + - Tony LaTorre + - Toshiki Kataoka - Tyler Moncur + - Tyler Reddy - Valentin Haenel - Vrinda Narayan + - Warren Weckesser - Weitang Li - Wojtek Ruszczewski - Yu Feng - Yu Kobayashi + - Yury Kirienko + - @aashuli + - @euronion + - @luzpaz - @parul + - @spacescientist + Cheers, Charles Harris -------------- next part -------------- An HTML attachment was scrubbed... URL: