From rth.yurchak at gmail.com Wed Apr 1 16:20:58 2020 From: rth.yurchak at gmail.com (Roman Yurchak) Date: Wed, 1 Apr 2020 22:20:58 +0200 Subject: [SciPy-Dev] pyproject.toml for scipy dependent project Message-ID: <66f7eb45-22e5-8aea-7b5f-741bc0d7c900@gmail.com> Hello, a quick question: for a project that requires scipy at build time (e.g. because it using its the Cython BLAS API), how should the associated pyproject.toml look like? i.e. a) is specifying scipy>=min_version enough b) or does one need to pin minimal supported scipy for each python version as it's currently done e.g. for numpy https://github.com/scipy/scipy/blob/7ac26a178f385e4fa88d5c22951797a32ec8f38f/pyproject.toml#L6 ? I think I have seen suggestions on conda-forge in the past that b) makes sense, in a similar situation. Can someone confirm that this is also something that should be done in `pyproject.toml`? Or would any supported version work? Thanks, -- Roman From benjamin.liu at berkeley.edu Wed Apr 1 17:49:34 2020 From: benjamin.liu at berkeley.edu (Benjamin Liu) Date: Wed, 1 Apr 2020 17:49:34 -0400 Subject: [SciPy-Dev] Specifying SciPy requirement in .toml file Message-ID: Hello all, Wanna to know if it is necessary to specify SciPy version for each python version in .toml file. Like what pandas project do with numpy: https://github.com/pandas-dev/pandas/blob/master/pyproject.toml Many Thanks, Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu Apr 2 05:18:11 2020 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 2 Apr 2020 11:18:11 +0200 Subject: [SciPy-Dev] pyproject.toml for scipy dependent project In-Reply-To: <66f7eb45-22e5-8aea-7b5f-741bc0d7c900@gmail.com> References: <66f7eb45-22e5-8aea-7b5f-741bc0d7c900@gmail.com> Message-ID: On Wed, Apr 1, 2020 at 10:21 PM Roman Yurchak wrote: > Hello, > > a quick question: for a project that requires scipy at build time (e.g. > because it using its the Cython BLAS API), how should the associated > pyproject.toml look like? i.e. > a) is specifying scipy>=min_version enough > b) or does one need to pin minimal supported scipy for each python > version as it's currently done e.g. for numpy > > https://github.com/scipy/scipy/blob/7ac26a178f385e4fa88d5c22951797a32ec8f38f/pyproject.toml#L6 > ? > > I think I have seen suggestions on conda-forge in the past that b) makes > sense, in a similar situation. Can someone confirm that this is also > something that should be done in `pyproject.toml`? Or would any > supported version work? > I can't remember a discussion about this before. In SciPy we haven't made changes that would prevent (a) I believe, however the question is whether Cython preserves forwards ABI compatibility. My guess is it may have been stable for quite a while, but it's not guaranteed. So (a) could work, but (b) is healthier. Would be good to know what happens if you built SciPy master with latest Cython, built your project against it, then ran tests against an older SciPy binary - have you tried? Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu Apr 2 05:18:50 2020 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 2 Apr 2020 11:18:50 +0200 Subject: [SciPy-Dev] Specifying SciPy requirement in .toml file In-Reply-To: References: Message-ID: On Wed, Apr 1, 2020 at 11:49 PM Benjamin Liu wrote: > Hello all, > > Wanna to know if it is necessary to specify SciPy version for each python > version in .toml file. Like what pandas project do with numpy: > https://github.com/pandas-dev/pandas/blob/master/pyproject.toml > This is the same question as Roman asked a few hours earlier. Please see that thread. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From rth.yurchak at gmail.com Thu Apr 2 16:59:44 2020 From: rth.yurchak at gmail.com (Roman Yurchak) Date: Thu, 2 Apr 2020 22:59:44 +0200 Subject: [SciPy-Dev] pyproject.toml for scipy dependent project In-Reply-To: References: <66f7eb45-22e5-8aea-7b5f-741bc0d7c900@gmail.com> Message-ID: <21fa61b5-c0a5-2e21-7370-cd8fc83eda08@gmail.com> On 02/04/2020 11:18, Ralf Gommers wrote: > In SciPy we haven't made changes that would prevent (a) I believe, > however the question is whether Cython preserves forwards ABI > compatibility. My guess is it may have been stable for quite a while, > but it's not guaranteed. So (a) could work, but (b) is healthier. Thanks for the confirmation! Actually there was an earlier related discussion in https://github.com/scipy/scipy/issues/6440 with a somewhat similar conclusion. > Would be good to know what happens if you built SciPy master with latest > Cython, built your project against it, then ran tests against an older > SciPy binary - have you tried? You are right, it seems to work fine regardless of the scipy version for the build. Built scikit-learn in develop mode (to bypass pyproject.toml) with: Python 3.6, numpy 1.14.5, cython 0.29.16 and scipy master. Run scikit-learn test suite with scipy 0.19.1 -> no errors. Only a (small) part of the scipy Cython API is used there though. -- Roman From ralf.gommers at gmail.com Fri Apr 3 14:23:10 2020 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 3 Apr 2020 20:23:10 +0200 Subject: [SciPy-Dev] pyproject.toml for scipy dependent project In-Reply-To: <21fa61b5-c0a5-2e21-7370-cd8fc83eda08@gmail.com> References: <66f7eb45-22e5-8aea-7b5f-741bc0d7c900@gmail.com> <21fa61b5-c0a5-2e21-7370-cd8fc83eda08@gmail.com> Message-ID: On Thu, Apr 2, 2020 at 10:59 PM Roman Yurchak wrote: > On 02/04/2020 11:18, Ralf Gommers wrote: > > In SciPy we haven't made changes that would prevent (a) I believe, > > however the question is whether Cython preserves forwards ABI > > compatibility. My guess is it may have been stable for quite a while, > > but it's not guaranteed. So (a) could work, but (b) is healthier. > > Thanks for the confirmation! Actually there was an earlier related > discussion in https://github.com/scipy/scipy/issues/6440 with a somewhat > similar conclusion. > Great find, thanks! That issue says that the ABI is backwards but not forwards compatible I believe, given Pauli's comment about being free to add extra signatures. So go with (b) I'd say. You suggested in that issue to document this better, that seems like a good idea. That probably belongs in the reference guide (e.g. on the cython_blas page, and then link to that from cython_lapack). Plus we should mention this in the tutorial, and reference Ian's paper on this topic: http://conference.scipy.org/proceedings/scipy2015/pdfs/ian_henriksen.pdf Any chance I can interest you in this? Cheers, Ralf > > Would be good to know what happens if you built SciPy master with latest > > Cython, built your project against it, then ran tests against an older > > SciPy binary - have you tried? > > You are right, it seems to work fine regardless of the scipy version for > the build. > > Built scikit-learn in develop mode (to bypass pyproject.toml) with: > Python 3.6, numpy 1.14.5, cython 0.29.16 and scipy master. Run > scikit-learn test suite with scipy 0.19.1 -> no errors. Only a (small) > part of the scipy Cython API is used there though. > > -- > Roman > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Sat Apr 4 07:33:00 2020 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 04 Apr 2020 14:33:00 +0300 Subject: [SciPy-Dev] pyproject.toml for scipy dependent project In-Reply-To: References: <66f7eb45-22e5-8aea-7b5f-741bc0d7c900@gmail.com> <21fa61b5-c0a5-2e21-7370-cd8fc83eda08@gmail.com> Message-ID: <824b13ae6cc910583f9b3b194c8a1659dd40976e.camel@iki.fi> Hi, pe, 2020-04-03 kello 20:23 +0200, Ralf Gommers kirjoitti: > Great find, thanks! That issue says that the ABI is backwards but not > forwards compatible I believe, given Pauli's comment about being free > to add extra signatures. So go with (b) I'd say. The situation with the Cython APIs is to my understanding that currently the ABI is both forward and backward compatible, so projects built with newer Scipy versions work also with older versions. However, whether this remains true depends on what we do in the future. In particular, the situation is not necessarily the same as with Numpy: - For plain functions (e.g. cython_blas/lapack): Cython checks function signatures at import time, so if we change signatures later on (or you use functions not available in older Scipy), you'll get an import error at import time. This works on a per-function basis, so you don't need to care about functions you don't use. Any signature change will break both API and ABI compatibility, so there's no big separate ABI issue (binaries will work with any Scipy versions they can be compiled with). - With structs and types defined in our .pxd files: if we make changes to them, the ABI can become **both** forward and backward incompatible, depending on who allocates them and what changes were made and how they are used. AFAIK, Cython does not check type/struct signatures (only names), so errors made result to crashes at runtime. So the situation here is basically the same as in plain C. E.g. in cython_optimize the structs are supposed to be allocated on the caller side --- if we make changes to the structs by adding things at the end, code compiled with *new* Scipy versions is compatible with old versions, but not vice versa. So exactly opposite of the Numpy situation. If stuff is allocated on the callee side, it's then different. To avoid mess, it probably is simplest if we will **not allow any changes to be made in any public structs and types in the Cython API**. If changes have to be made, the names of the types/structs must be changed (and we maybe could decide on a naming scheme). Then the situation can be dealt similarly as with adding new APIs and deprecating/removing old ones. Even simpler is if we additionally will **not allow any non-opaque structs in the public Cython API**. I.e. no struct members may be listed in public .pxd files. All data structures must be allocated by a separate public allocation function, and all attributes accessed by accessor functions. (Types, defines, etc. would still have to be considered final, as above.) Maintaining a Numpy-style forward-compatible ABI situation requires a bit more care in managing who allocates what data structures and how they are used. However, because unlike Numpy we don't have big legacy API here, I think we would be able to maintain the simpler "binaries will work with any Scipy versions they can be compiled with" situation. Pauli > Would be good to know what happens if you built SciPy master with > > latest > > > Cython, built your project against it, then ran tests against an > > older > > > SciPy binary - have you tried? > > > > You are right, it seems to work fine regardless of the scipy > > version for > > the build. > > > > Built scikit-learn in develop mode (to bypass pyproject.toml) > > with: > > Python 3.6, numpy 1.14.5, cython 0.29.16 and scipy master. Run > > scikit-learn test suite with scipy 0.19.1 -> no errors. Only a > > (small) > > part of the scipy Cython API is used there though. > > > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at python.org > > https://mail.python.org/mailman/listinfo/scipy-dev From rth.yurchak at gmail.com Tue Apr 7 08:24:33 2020 From: rth.yurchak at gmail.com (Roman Yurchak) Date: Tue, 7 Apr 2020 14:24:33 +0200 Subject: [SciPy-Dev] pyproject.toml for scipy dependent project In-Reply-To: <824b13ae6cc910583f9b3b194c8a1659dd40976e.camel@iki.fi> References: <66f7eb45-22e5-8aea-7b5f-741bc0d7c900@gmail.com> <21fa61b5-c0a5-2e21-7370-cd8fc83eda08@gmail.com> <824b13ae6cc910583f9b3b194c8a1659dd40976e.camel@iki.fi> Message-ID: Thank you for the detailed explanations! I see you have also opened a PR to document it https://github.com/scipy/scipy/pull/11792 Thanks, that's very useful! -- Roman On 04/04/2020 13:33, Pauli Virtanen wrote: > Hi, > > pe, 2020-04-03 kello 20:23 +0200, Ralf Gommers kirjoitti: >> Great find, thanks! That issue says that the ABI is backwards but not >> forwards compatible I believe, given Pauli's comment about being free >> to add extra signatures. So go with (b) I'd say. > > The situation with the Cython APIs is to my understanding that > currently the ABI is both forward and backward compatible, so projects > built with newer Scipy versions work also with older versions. > > However, whether this remains true depends on what we do in the future. > In particular, the situation is not necessarily the same as with Numpy: > > - For plain functions (e.g. cython_blas/lapack): Cython checks function > signatures at import time, so if we change signatures later on (or you > use functions not available in older Scipy), you'll get an import error > at import time. This works on a per-function basis, so you don't need > to care about functions you don't use. Any signature change will break > both API and ABI compatibility, so there's no big separate ABI issue > (binaries will work with any Scipy versions they can be compiled with). > > - With structs and types defined in our .pxd files: if we make changes > to them, the ABI can become **both** forward and backward incompatible, > depending on who allocates them and what changes were made and how they > are used. AFAIK, Cython does not check type/struct signatures (only > names), so errors made result to crashes at runtime. So the situation > here is basically the same as in plain C. > > E.g. in cython_optimize the structs are supposed to be allocated on the > caller side --- if we make changes to the structs by adding things at > the end, code compiled with *new* Scipy versions is compatible with old > versions, but not vice versa. So exactly opposite of the Numpy > situation. If stuff is allocated on the callee side, it's then > different. > > To avoid mess, it probably is simplest if we will **not allow any > changes to be made in any public structs and types in the Cython API**. > If changes have to be made, the names of the types/structs must be > changed (and we maybe could decide on a naming scheme). Then the > situation can be dealt similarly as with adding new APIs and > deprecating/removing old ones. > > Even simpler is if we additionally will **not allow any non-opaque > structs in the public Cython API**. I.e. no struct members may be > listed in public .pxd files. All data structures must be allocated by a > separate public allocation function, and all attributes accessed by > accessor functions. (Types, defines, etc. would still have to be > considered final, as above.) > > Maintaining a Numpy-style forward-compatible ABI situation requires a > bit more care in managing who allocates what data structures and how > they are used. However, because unlike Numpy we don't have big legacy > API here, I think we would be able to maintain the simpler "binaries > will work with any Scipy versions they can be compiled with" situation. > > Pauli > > >> Would be good to know what happens if you built SciPy master with >>> latest >>>> Cython, built your project against it, then ran tests against an >>> older >>>> SciPy binary - have you tried? >>> >>> You are right, it seems to work fine regardless of the scipy >>> version for >>> the build. >>> >>> Built scikit-learn in develop mode (to bypass pyproject.toml) >>> with: >>> Python 3.6, numpy 1.14.5, cython 0.29.16 and scipy master. Run >>> scikit-learn test suite with scipy 0.19.1 -> no errors. Only a >>> (small) >>> part of the scipy Cython API is used there though. >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > From warren.weckesser at gmail.com Sun Apr 19 11:06:57 2020 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Sun, 19 Apr 2020 11:06:57 -0400 Subject: [SciPy-Dev] Run just the test functions that match a given pattern? Message-ID: With runtests.py, we can run the tests for a specific module (e.g. `-s stats`), a specific file (e.g. `-t scipy.stats.tests.test_stats`), or a specific class or function within a file (e.g. `-s scipy.stats.tests.test_stats::TestCorrPearsonr`, `-s scipy.stats.tests.test_stats::test_kendalltau`). Is there a way to run the tests whose names match a given pattern? For example, I'd like to use the pattern `*ttest_rel*` to run any test that has `ttest_rel` in its name. I can hack this by adding the line python_functions = *ttest_rel* to pytest.ini, and then comment out that line when I want to run a larger set of tests, but it would be nice if a pattern could be given in the command line. Warren From matti.picus at gmail.com Sun Apr 19 11:23:03 2020 From: matti.picus at gmail.com (Matti Picus) Date: Sun, 19 Apr 2020 18:23:03 +0300 Subject: [SciPy-Dev] Run just the test functions that match a given pattern? In-Reply-To: References: Message-ID: <2d12a446-f085-1fa1-f4a2-b7588b287a2f@gmail.com> On 19/4/20 6:06 pm, Warren Weckesser wrote: > With runtests.py, we can run the tests for a specific module (e.g. `-s > stats`), a specific file (e.g. `-t scipy.stats.tests.test_stats`), or > a specific class or function within a file (e.g. `-s > scipy.stats.tests.test_stats::TestCorrPearsonr`, `-s > scipy.stats.tests.test_stats::test_kendalltau`). Is there a way to > run the tests whose names match a given pattern? For example, I'd > like to use the pattern `*ttest_rel*` to run any test that has > `ttest_rel` in its name. > > I can hack this by adding the line > > python_functions = *ttest_rel* > > to pytest.ini, and then comment out that line when I want to run a > larger set of tests, but it would be nice if a pattern could be given > in the command line. > > Warren I think this is a documentation issue. Since scipy uses pytest under the hood to actually run the tests, any extra arguments after `--` are passed to pytest, so you should be able to use `python runtest.py -- -k "*ttest_rel*' which will use pytest's regex test chooser. At least that is the way it works on numpy ... Matti From warren.weckesser at gmail.com Sun Apr 19 11:26:28 2020 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Sun, 19 Apr 2020 11:26:28 -0400 Subject: [SciPy-Dev] Run just the test functions that match a given pattern? In-Reply-To: References: Message-ID: On 4/19/20, Warren Weckesser wrote: > With runtests.py, we can run the tests for a specific module (e.g. `-s > stats`), a specific file (e.g. `-t scipy.stats.tests.test_stats`), or > a specific class or function within a file (e.g. `-s > scipy.stats.tests.test_stats::TestCorrPearsonr`, `-s > scipy.stats.tests.test_stats::test_kendalltau`). Typo: those examples should use -t, not -s. > Is there a way to > run the tests whose names match a given pattern? For example, I'd > like to use the pattern `*ttest_rel*` to run any test that has > `ttest_rel` in its name. > Answering my own question: additional arguments given to runtests.py after `--` are passed to the pytest command, and pytest has the option -k to select a substring expression, so something like this works to run just the tests with `ttest_rel` in the test name within the stats module python runtests.py -n -s stats -- -k ttest_rel (The -n is there because I'm running the tests on a previous installation.) Warren > I can hack this by adding the line > > python_functions = *ttest_rel* > > to pytest.ini, and then comment out that line when I want to run a > larger set of tests, but it would be nice if a pattern could be given > in the command line. > > Warren > From warren.weckesser at gmail.com Sun Apr 19 11:28:39 2020 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Sun, 19 Apr 2020 11:28:39 -0400 Subject: [SciPy-Dev] Run just the test functions that match a given pattern? In-Reply-To: <2d12a446-f085-1fa1-f4a2-b7588b287a2f@gmail.com> References: <2d12a446-f085-1fa1-f4a2-b7588b287a2f@gmail.com> Message-ID: On 4/19/20, Matti Picus wrote: > > On 19/4/20 6:06 pm, Warren Weckesser wrote: >> With runtests.py, we can run the tests for a specific module (e.g. `-s >> stats`), a specific file (e.g. `-t scipy.stats.tests.test_stats`), or >> a specific class or function within a file (e.g. `-s >> scipy.stats.tests.test_stats::TestCorrPearsonr`, `-s >> scipy.stats.tests.test_stats::test_kendalltau`). Is there a way to >> run the tests whose names match a given pattern? For example, I'd >> like to use the pattern `*ttest_rel*` to run any test that has >> `ttest_rel` in its name. >> >> I can hack this by adding the line >> >> python_functions = *ttest_rel* >> >> to pytest.ini, and then comment out that line when I want to run a >> larger set of tests, but it would be nice if a pattern could be given >> in the command line. >> >> Warren > > > I think this is a documentation issue. Since scipy uses pytest under the > hood to actually run the tests, any extra arguments after `--` are > passed to pytest, so you should be able to use `python runtest.py -- -k > "*ttest_rel*' which will use pytest's regex test chooser. At least that > is the way it works on numpy ... Thanks Matti! I sent my previous email before I saw yours. -k works for me. Warren > > > Matti > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > From charlesr.harris at gmail.com Sun Apr 19 16:44:05 2020 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 19 Apr 2020 14:44:05 -0600 Subject: [SciPy-Dev] NumPy 1.18.3 released. Message-ID: Hi All, On behalf of the NumPy team I am pleased to announce that NumPy 1.18.3 has been released. This release contains various bug/regression fixes for the 1.18 series The Python versions supported in this release are 3.5-3.8. Downstream developers should use Cython >= 0.29.15 for Python 3.8 support and OpenBLAS >= 3.7 to avoid errors on the Skylake architecture. Wheels for this release can be downloaded from PyPI , source archives and release notes are available from Github . *Highlights* Fix for the method='eigh' and method='cholesky' options in numpy.random.multivariate_normal. Those were producing samples from the wrong distribution. *Contributors* A total of 6 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Charles Harris - Max Balandat + - @Mibu287 + - Pan Jan + - Sebastian Berg - @panpiort8 + *Pull requests merged* A total of 5 pull requests were merged for this release. - #15916: BUG: Fix eigh and cholesky methods of numpy.random.multivariate_normal - #15929: BUG,MAINT: Remove incorrect special case in string to number... - #15930: BUG: Guarantee array is in valid state after memory error occurs... - #15954: BUG: Check that pvals is 1D in _generator.multinomial. - #16017: BUG: Alpha parameter must be 1D in _generator.dirichlet Cheers, Charles Harris -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Sun Apr 19 17:46:02 2020 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Sun, 19 Apr 2020 17:46:02 -0400 Subject: [SciPy-Dev] NumPy 1.18.3 released. In-Reply-To: References: Message-ID: On 4/19/20, Charles R Harris wrote: > Hi All, > > On behalf of the NumPy team I am pleased to announce that NumPy 1.18.3 has > been released. This release contains various bug/regression fixes for the > 1.18 series Thanks Chuck! Warren > > The Python versions supported in this release are 3.5-3.8. Downstream > developers should use Cython >= 0.29.15 for Python 3.8 support and OpenBLAS >>= 3.7 to avoid errors on the Skylake architecture. Wheels for this > release can be downloaded from PyPI > , > source archives and release notes are available from Github > . > > *Highlights* > > Fix for the method='eigh' and method='cholesky' options in > numpy.random.multivariate_normal. Those were producing samples from the > wrong distribution. > > *Contributors* > > A total of 6 people contributed to this release. People with a "+" by > their > names contributed a patch for the first time. > > - Charles Harris > - Max Balandat + > - @Mibu287 + > - Pan Jan + > - Sebastian Berg > - @panpiort8 + > > > > *Pull requests merged* > A total of 5 pull requests were merged for this release. > > - #15916: BUG: Fix eigh and cholesky methods of > numpy.random.multivariate_normal > - #15929: BUG,MAINT: Remove incorrect special case in string to > number... > - #15930: BUG: Guarantee array is in valid state after memory error > occurs... > - #15954: BUG: Check that pvals is 1D in _generator.multinomial. > - #16017: BUG: Alpha parameter must be 1D in _generator.dirichlet > > > Cheers, > > Charles Harris > From wsw.raczek at gmail.com Fri Apr 24 04:10:18 2020 From: wsw.raczek at gmail.com (=?UTF-8?B?V8WCYWR5c8WCYXcgUmFjemVr?=) Date: Fri, 24 Apr 2020 10:10:18 +0200 Subject: [SciPy-Dev] gh-11787 Message-ID: Hi all, There is this issue on gh: https://github.com/scipy/scipy/issues/11787 Because I don't know who is the maintainer of the module, I'm writing hear to ask for (subjective) opinion (it would be great if the maintainer will answer): 1) Would it really be that useful? 2) Is there a reason we don't do that yet? I haven't deeply looked into issue, but I rather find that interesting and am ready to work on it. Kind regards, Vladyslav Rachek -------------- next part -------------- An HTML attachment was scrubbed... URL: From sujoykumargoswami at gmail.com Fri Apr 24 06:18:49 2020 From: sujoykumargoswami at gmail.com (Sujoy Kumar Goswami) Date: Fri, 24 Apr 2020 15:48:49 +0530 Subject: [SciPy-Dev] Regarding Contribution: a new image edge detection technique Message-ID: Hello Sir/ Madam, I want to contribute to SciPy (Python) with a new image edge detection technique which is more generic & better than the similar popular techniques for the same. The paper that I have published is attached here with this email. Please go through it. If you like it, please tell me the process to move next. Thanks & Regards, Sujoy K Goswami Principal Data Scientist TVS Motor Company India -------------- next part -------------- A non-text attachment was scrubbed... Name: a-better-first-derivative-approach-for-edge-detection-IJERTV2IS110616.pdf Type: application/pdf Size: 262790 bytes Desc: not available URL: From robert.kern at gmail.com Fri Apr 24 21:40:33 2020 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 24 Apr 2020 21:40:33 -0400 Subject: [SciPy-Dev] Regarding Contribution: a new image edge detection technique In-Reply-To: References: Message-ID: On Fri, Apr 24, 2020 at 6:19 AM Sujoy Kumar Goswami < sujoykumargoswami at gmail.com> wrote: > Hello Sir/ Madam, > > I want to contribute to SciPy (Python) with a new image edge detection > technique which is more generic & better than the similar popular > techniques for the same. > The paper that I have published is attached here with this email. > Please go through it. > If you like it, please tell me the process to move next. > Thank you for the paper. I don't think I've seen uses of these gradient filters in literature or software, so I don't feel a strong need to have this implemented explicitly in scipy.ndimage. If people do want to use these filters, they are straightforward to implement using the tools from scipy.ndimage. You may be interested in the paper "Consistent Gradient Operators" https://www.researchgate.net/publication/3193087_Consistent_gradient_operators It provides some interesting criteria for evaluating gradient operators that you might want to test yours against. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sat Apr 25 09:27:22 2020 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 25 Apr 2020 15:27:22 +0200 Subject: [SciPy-Dev] gh-11787 In-Reply-To: References: Message-ID: On Fri, Apr 24, 2020 at 10:10 AM W?adys?aw Raczek wrote: > Hi all, > > There is this issue on gh: https://github.com/scipy/scipy/issues/11787 > Because I don't know who is the maintainer of the module, I'm writing > hear to ask for (subjective) opinion (it would be great if the maintainer > will answer): > 1) Would it really be that useful? > 2) Is there a reason we don't do that yet? > I haven't deeply looked into issue, but I rather find that interesting and > am ready to work on it. > Thanks for asking Vladyslav! I've commented on the issue and pinged a couple of more knowledgeable people. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Tue Apr 28 19:52:36 2020 From: andyfaff at gmail.com (Andrew Nelson) Date: Wed, 29 Apr 2020 09:52:36 +1000 Subject: [SciPy-Dev] Using global minimizer methods from `optimize.minimize` function Message-ID: `optimize.minimize` offers a choice of many different methods for multivariate scalar minimisation. These methods are chosen using the `method` keyword. There are also different global minimisation routines that one can use (differential_evolution, basinhopping, dual_annealing, shgo). These minimisers have the same overall objective as `minimize`, just with a different approach to finding a minimum. The global minimiser routines are called individually, and are not accessible through the `minimize` function as different methods. A PR is open at https://github.com/scipy/scipy/pull/10778 which proposes to add a `differential-evolution` method to `minimize` that would permit this. This is a fairly straightforward change as the call interfaces are almost identical, and the problems are posed in similar ways. There are obviously pros and cons to this: Pros ------ - One could call any of the multivariate scalar minimizers through one function. - In user code this could simplify code significantly (code that offers all the different minimizers has to use if/elif constructs to call different functions depending on the method to be used). Cons ------- - A user may not appreciate the differences of how local and global minimisers work. e.g. a lot of the global minimisers are stochastic and some use local minimisers to polish the end solution. Could we have a discussion as to whether people think this is a good/bad idea? Would it confuse users, would it make `minimize` too convoluted, etc? A. -------------- next part -------------- An HTML attachment was scrubbed... URL: From newville at cars.uchicago.edu Wed Apr 29 11:25:20 2020 From: newville at cars.uchicago.edu (Matt Newville) Date: Wed, 29 Apr 2020 10:25:20 -0500 Subject: [SciPy-Dev] Using global minimizer methods from `optimize.minimize` function In-Reply-To: References: Message-ID: On Tue, Apr 28, 2020 at 6:53 PM Andrew Nelson wrote: > `optimize.minimize` offers a choice of many different methods for > multivariate scalar minimisation. These methods are chosen using the > `method` keyword. > > There are also different global minimisation routines that one can use > (differential_evolution, basinhopping, dual_annealing, shgo). These > minimisers have the same overall objective as `minimize`, just with a > different approach to finding a minimum. The global minimiser routines are > called individually, and are not accessible through the `minimize` function > as different methods. A PR is open at > https://github.com/scipy/scipy/pull/10778 which proposes to add a > `differential-evolution` method to `minimize` that would permit this. This > is a fairly straightforward change as the call interfaces are almost > identical, and the problems are posed in similar ways. > > There are obviously pros and cons to this: > > Pros > ------ > - One could call any of the multivariate scalar minimizers through one > function. > - In user code this could simplify code significantly (code that offers > all the different minimizers has to use if/elif constructs to call > different functions depending on the method to be used). > I sort of think these pros are overstated. The dispatching of what function to call does not seem that difficult to do (either in `minimize` or in user code). The benefit of having that dispatch of function name happen within `minimize` is small. Normalizing the APIs so that the options sent to the underlying methods is harder and also more valuable. That is, in order for the dispatching to really be valuable, it has to unite and offer a translation layer to the calls to the underlying functions. The global solvers have many different optional arguments with little overlap in name or meaning. Like, 'popsize' is only used by 'differential_evolution'. The plan would have to be to silently ignore keyword arguments for concepts not used by the currently used method. I'm not sure that helps achieve clarity and simplicity. To use these methods, the user has to read the docs for the actual solver to get the many optional arguments set anyway. At that point, they can just as easily change the name of the function. > Cons > ------- > - A user may not appreciate the differences of how local and global > minimisers work. e.g. a lot of the global minimisers are stochastic and > some use local minimisers to polish the end solution. > > Could we have a discussion as to whether people think this is a good/bad > idea? Would it confuse users, would it make `minimize` too convoluted, etc? > I don't think the distinction between "local" and "global" is actually that important. Well, actually, I think the label "global" is kind of misleading, as most of these methods require bounds. What they do is try to avoid getting stuck in the first minima they find. But, I think there is another concern that may not have been expressed yet. `x0` is a required, positional argument for `minimize()`, as an array of initial parameter values. Most of the global optimizers in scipy.optimize do not use `x0`. Instead, they require bounds and explore the range of values between those bounds. Would `x0` be required AND ignored for these global optimizers? Cheers, --Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Wed Apr 29 22:42:42 2020 From: andyfaff at gmail.com (Andrew Nelson) Date: Thu, 30 Apr 2020 12:42:42 +1000 Subject: [SciPy-Dev] Using global minimizer methods from `optimize.minimize` function In-Reply-To: References: Message-ID: > > I sort of think these pros are overstated. The dispatching of what > function to call does not seem that difficult to do (either in `minimize` > or in user code). The benefit of having that dispatch of function name > happen within `minimize` is small. Normalizing the APIs so that the > options sent to the underlying methods is harder and also more valuable. > That is, in order for the dispatching to really be valuable, it has to > unite and offer a translation layer to the calls to the underlying > functions. > A recent issue (https://github.com/scipy/scipy/issues/11956) highlighted this issue. Here the author wanted to use a constraints dict with differential_evolution, similar to that can be provided to `minimize` (de uses the new style `NonLinearConstraint`). If differential_evolution was a `minimize` method that translation would be done automatically. The same argument applies for translation of new style and old style bounds. Is that what you mean by normalising the API? > The global solvers have many different optional arguments with little > overlap in name or meaning. Like, 'popsize' is only used by > 'differential_evolution'. The plan would have to be to silently ignore > keyword arguments for concepts not used by the currently used method. I'm > not sure that helps achieve clarity and simplicity. To use these methods, > the user has to read the docs for the actual solver to get the many > optional arguments set anyway. At that point, they can just as easily > change the name of the function. > There are many optional arguments for each of the methods as-is. The most common are jac and hess which are used across some, but not all methods. L-BFGS-B has `iprint`, `gtol`, `maxls` which aren't used by most other methods. Over supply/non-supply (i.e. concepts not used by specified method) of these keywords is already handled by minimize, and by the minimizers themselves (via `_check_unknown_options`). Your line of reasoning runs counter to the design of the `minimize` function, and would suggest a return to the old-style minimize functions: fmin, fmin_l_bfgs_b, etc. (The documentation states "The functions below are not recommended for use in new scripts; all of these methods are accessible via a newer, more consistent interfaces, provided by the interfaces above.") But, I think there is another concern that may not have been expressed > yet. `x0` is a required, positional argument for `minimize()`, as an array > of initial parameter values. Most of the global optimizers in > scipy.optimize do not use `x0`. Instead, they require bounds and explore > the range of values between those bounds. Would `x0` be required AND > ignored for these global optimizers? > The call signature for required positional arguments for `minimize` is different to the global optimizers. Being able to call the global minimizers via `minimize` would alleviate that. As you say the global minimizers do explore within bounds, and don't use an `x0`. The PR (as it currently exists) would still require `x0`, and it would be ignored. It would be possible to change that behaviour for `differential_evolution`, but would require the modification of the code to accept an `x0` keyword. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mhaberla at calpoly.edu Wed Apr 29 23:12:07 2020 From: mhaberla at calpoly.edu (Matt Haberland) Date: Wed, 29 Apr 2020 20:12:07 -0700 Subject: [SciPy-Dev] Using global minimizer methods from `optimize.minimize` function In-Reply-To: References: Message-ID: Just wanted to mention that I'm in favor of adding the option of accessing `differential_evolution` via minimize, and my comments are at the PR . On Wed, Apr 29, 2020 at 7:43 PM Andrew Nelson wrote: > I sort of think these pros are overstated. The dispatching of what >> function to call does not seem that difficult to do (either in `minimize` >> or in user code). The benefit of having that dispatch of function name >> happen within `minimize` is small. Normalizing the APIs so that the >> options sent to the underlying methods is harder and also more valuable. >> That is, in order for the dispatching to really be valuable, it has to >> unite and offer a translation layer to the calls to the underlying >> functions. >> > > A recent issue (https://github.com/scipy/scipy/issues/11956) highlighted > this issue. Here the author wanted to use a constraints dict with > differential_evolution, similar to that can be provided to `minimize` (de > uses the new style `NonLinearConstraint`). If differential_evolution was a > `minimize` method that translation would be done automatically. The same > argument applies for translation of new style and old style bounds. Is that > what you mean by normalising the API? > > >> The global solvers have many different optional arguments with little >> overlap in name or meaning. Like, 'popsize' is only used by >> 'differential_evolution'. The plan would have to be to silently ignore >> keyword arguments for concepts not used by the currently used method. I'm >> not sure that helps achieve clarity and simplicity. To use these methods, >> the user has to read the docs for the actual solver to get the many >> optional arguments set anyway. At that point, they can just as easily >> change the name of the function. >> > > There are many optional arguments for each of the methods as-is. The most > common are jac and hess which are used across some, but not all methods. > L-BFGS-B has `iprint`, `gtol`, `maxls` which aren't used by most other > methods. Over supply/non-supply (i.e. concepts not used by specified > method) of these keywords is already handled by minimize, and by the > minimizers themselves (via `_check_unknown_options`). Your line of > reasoning runs counter to the design of the `minimize` function, and would > suggest a return to the old-style minimize functions: fmin, fmin_l_bfgs_b, > etc. (The documentation states "The functions below are not recommended > for use in new scripts; all of these methods are accessible via a newer, > more consistent interfaces, provided by the interfaces above.") > > But, I think there is another concern that may not have been expressed >> yet. `x0` is a required, positional argument for `minimize()`, as an array >> of initial parameter values. Most of the global optimizers in >> scipy.optimize do not use `x0`. Instead, they require bounds and explore >> the range of values between those bounds. Would `x0` be required AND >> ignored for these global optimizers? >> > > The call signature for required positional arguments for `minimize` is > different to the global optimizers. Being able to call the global > minimizers via `minimize` would alleviate that. As you say the global > minimizers do explore within bounds, and don't use an `x0`. The PR (as it > currently exists) would still require `x0`, and it would be ignored. > It would be possible to change that behaviour for > `differential_evolution`, but would require the modification of the code to > accept an `x0` keyword. > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -- Matt Haberland Assistant Professor BioResource and Agricultural Engineering 08A-3K, Cal Poly -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu Apr 30 00:24:23 2020 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 30 Apr 2020 06:24:23 +0200 Subject: [SciPy-Dev] Using global minimizer methods from `optimize.minimize` function In-Reply-To: References: Message-ID: On Thu, Apr 30, 2020 at 4:43 AM Andrew Nelson wrote: > > But, I think there is another concern that may not have been expressed >> yet. `x0` is a required, positional argument for `minimize()`, as an array >> of initial parameter values. Most of the global optimizers in >> scipy.optimize do not use `x0`. Instead, they require bounds and explore >> the range of values between those bounds. Would `x0` be required AND >> ignored for these global optimizers? >> > > The call signature for required positional arguments for `minimize` is > different to the global optimizers. Being able to call the global > minimizers via `minimize` would alleviate that. As you say the global > minimizers do explore within bounds, and don't use an `x0`. The PR (as it > currently exists) would still require `x0`, and it would be ignored. > It would be possible to change that behaviour for > `differential_evolution`, but would require the modification of the code to > accept an `x0` keyword. > Hmm, the signature issue seems quite problematic. The opposite is also true: `bounds` is optional for minimize(), and cannot be made non-optional, however for the global optimizers it is required. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Thu Apr 30 00:43:32 2020 From: andyfaff at gmail.com (Andrew Nelson) Date: Thu, 30 Apr 2020 14:43:32 +1000 Subject: [SciPy-Dev] Using global minimizer methods from `optimize.minimize` function In-Reply-To: References: Message-ID: On Thu, 30 Apr 2020 at 14:25, Ralf Gommers wrote: > > > On Thu, Apr 30, 2020 at 4:43 AM Andrew Nelson wrote: > >> >> But, I think there is another concern that may not have been expressed >>> yet. `x0` is a required, positional argument for `minimize()`, as an array >>> of initial parameter values. Most of the global optimizers in >>> scipy.optimize do not use `x0`. Instead, they require bounds and explore >>> the range of values between those bounds. Would `x0` be required AND >>> ignored for these global optimizers? >>> >> >> The call signature for required positional arguments for `minimize` is >> different to the global optimizers. Being able to call the global >> minimizers via `minimize` would alleviate that. As you say the global >> minimizers do explore within bounds, and don't use an `x0`. The PR (as it >> currently exists) would still require `x0`, and it would be ignored. >> It would be possible to change that behaviour for >> `differential_evolution`, but would require the modification of the code to >> accept an `x0` keyword. >> > > Hmm, the signature issue seems quite problematic. The opposite is also > true: `bounds` is optional for minimize(), and cannot be made non-optional, > however for the global optimizers it is required. > The call would be: ``` minimize(func, x0, bounds=bounds, method='differential-evolution') ``` Inside `minimize` it would look something like: ``` res = differential_evolution(func, bounds) ``` If bounds is None then an error would be raised, either from the `minimize` function or the underlying `differential_evolution` function. At the moment the x0 supplied to minimize would be ignored by the underlying de function, but it would be possible to amend de to use x0 as well. With this proposal the minimize signature wouldn't change, neither would that of `differential_evolution` (with the possible exception of adding an `x0` keyword to use an initial position). -------------- next part -------------- An HTML attachment was scrubbed... URL: From newville at cars.uchicago.edu Thu Apr 30 00:47:38 2020 From: newville at cars.uchicago.edu (Matt Newville) Date: Wed, 29 Apr 2020 23:47:38 -0500 Subject: [SciPy-Dev] Using global minimizer methods from `optimize.minimize` function In-Reply-To: References: Message-ID: On Wed, Apr 29, 2020 at 9:44 PM Andrew Nelson wrote: > I sort of think these pros are overstated. The dispatching of what >> function to call does not seem that difficult to do (either in `minimize` >> or in user code). The benefit of having that dispatch of function name >> happen within `minimize` is small. Normalizing the APIs so that the >> options sent to the underlying methods is harder and also more valuable. >> That is, in order for the dispatching to really be valuable, it has to >> unite and offer a translation layer to the calls to the underlying >> functions. >> > > A recent issue (https://github.com/scipy/scipy/issues/11956) highlighted > this issue. Here the author wanted to use a constraints dict with > differential_evolution, similar to that can be provided to `minimize` (de > uses the new style `NonLinearConstraint`). If differential_evolution was a > `minimize` method that translation would be done automatically. The same > argument applies for translation of new style and old style bounds. Is that > what you mean by normalising the API? > It would be? Or: how would it be without having it also work with the function `differential_evolution`. > > >> The global solvers have many different optional arguments with little >> overlap in name or meaning. Like, 'popsize' is only used by >> 'differential_evolution'. The plan would have to be to silently ignore >> keyword arguments for concepts not used by the currently used method. I'm >> not sure that helps achieve clarity and simplicity. To use these methods, >> the user has to read the docs for the actual solver to get the many >> optional arguments set anyway. At that point, they can just as easily >> change the name of the function. >> > > There are many optional arguments for each of the methods as-is. The most > common are jac and hess which are used across some, but not all methods. > L-BFGS-B has `iprint`, `gtol`, `maxls` which aren't used by most other > methods. Over supply/non-supply (i.e. concepts not used by specified > method) of these keywords is already handled by minimize, and by the > minimizers themselves (via `_check_unknown_options`). Your line of > reasoning runs counter to the design of the `minimize` function, and would > suggest a return to the old-style minimize functions: fmin, fmin_l_bfgs_b, > etc. (The documentation states "The functions below are not recommended > for use in new scripts; all of these methods are accessible via a newer, > more consistent interfaces, provided by the interfaces above.") > Yes, you are correct that my line of reasoning runs counter to the `minimize` function. Well, I would say not so much "counter" as seeing some value with `minimize`, but also seeing some value with the old style too. I am definitely not in favor of the documentation that claims working functions are deprecated in favor of a multi-dispatch function. Somehow, `minimize` and the deprecation of `fmin`, etc grates on me much less than the alleged deprecation for `leastsq` in favor of `least_squares`, which is a nest of interdependent options. `Minimize` does not seem as bad (but it appears that you're working on it ;)). My point is that the dispatching of the function names is not, by itself, really that big of a win. Functions are first-class objects. At least with most of the current solvers covered by `minimize`, most of the non-uniformly-named options are also truly optional (except, for some methods, `jac` -- but at least that is more or less a uniform name) and "advanced options for fine tuning". My sense is that the uniquely named arguments to the global solvers are more important, and are less like "advanced options". > But, I think there is another concern that may not have been expressed >> yet. `x0` is a required, positional argument for `minimize()`, as an array >> of initial parameter values. Most of the global optimizers in >> scipy.optimize do not use `x0`. Instead, they require bounds and explore >> the range of values between those bounds. Would `x0` be required AND >> ignored for these global optimizers? >> > > The call signature for required positional arguments for `minimize` is > different to the global optimizers. Being able to call the global > minimizers via `minimize` would alleviate that. As you say the global > minimizers do explore within bounds, and don't use an `x0`. The PR (as it > currently exists) would still require `x0`, and it would be ignored. > If I understand correctly, with the proposed changes, I hope you would have to continue supporting minimize(objective, x0, method='Nelder-Mead') to work as it currently does: x0 required. To switch to solve with `differential_evolution`, the user would have to do: minimize(objective, x0, bounds=bounds, method='differential_evolution') Now, although `bounds` is a keyword argument, it is actually required for the method to work. And `x0` is a required positional argument, but the value is ignored. That seems profoundly weird to me. Are there other examples in scipy (outside of scipy.optimize) for which a) a required, positional argument has a value that is ignored when an optional keyword argument has some value(s)? b) a keyword argument is changed from optional to required due to the value of a different keyword argument? Each of these seems like a problem to me. And, yes, b) is currently the case for `minimize`: some values of `method` require a `jac` option, while other values for `method` do not use `jac`. So, yes, I think the idea of putting all of `scipy.optimize` into a single function is an understandable desire but also sort of a design mistake. If important keyword arguments are different and not translatable, I don't see why minimize(objective, x0, method='solver', **kws) is actually better than solver(objective, x0, **kws) For sure, having uniform keyword arguments and using common infrastructure to, say, having a common way of setting bounds and constraints is valuable. Or (ahem), make Parameters objects that have names and bounds, etc.... Or if there was a Minimizer class with attributes and methods, maybe having a `method` attribute would make sense. Either way, if you're looking to improve the uniformity or the ability of downstream code to use the functions as if they were an API, then OptimizerResult really ought to include the name of the method used. Cheers, --Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Thu Apr 30 01:26:50 2020 From: andyfaff at gmail.com (Andrew Nelson) Date: Thu, 30 Apr 2020 15:26:50 +1000 Subject: [SciPy-Dev] Using global minimizer methods from `optimize.minimize` function In-Reply-To: References: Message-ID: > > My sense is that the uniquely named arguments to the global solvers are > more important, and are less like "advanced options". > When I use the global minimisers I hardly change the tuning of the default options. But, I think there is another concern that may not have been expressed >>> yet. `x0` is a required, positional argument for `minimize()`, as an array >>> of initial parameter values. Most of the global optimizers in >>> scipy.optimize do not use `x0`. Instead, they require bounds and explore >>> the range of values between those bounds. Would `x0` be required AND >>> ignored for these global optimizers? >>> >> >> The call signature for required positional arguments for `minimize` is >> different to the global optimizers. Being able to call the global >> minimizers via `minimize` would alleviate that. As you say the global >> minimizers do explore within bounds, and don't use an `x0`. The PR (as it >> currently exists) would still require `x0`, and it would be ignored. >> > > If I understand correctly, with the proposed changes, I hope you would > have to continue supporting > > minimize(objective, x0, method='Nelder-Mead') > You are correct. There would be no behavioural change of all the existing methods. minimize(objective, x0, bounds=bounds, > method='differential_evolution') > > Now, although `bounds` is a keyword argument, it is actually required for > the method to work. And `x0` is a required positional argument, but > the value is ignored. That seems profoundly weird to me. > `bounds` would required for the method to work. As you mention this is no different to 'newton-cg' that requires 'jac' for that method to work. `x0` would still be a required positional argument for `minimize`. If would be ignored for the `differential-evolution` method. However, it is also possible to change the underlying `differential_evolution` function to use an initial `x0` guess. Those guesses are less important for the global minimisers. > Are there other examples in scipy (outside of scipy.optimize) for which > > a) a required, positional argument has a value that is ignored when an > optional keyword argument has some value(s)? > b) a keyword argument is changed from optional to required due to the > value of a different keyword argument? > I am less familiar with other areas of scipy. Either way, if you're looking to improve the uniformity or the ability of > downstream code to use the functions as if they were an API, then > OptimizerResult really ought to include the name of the method used. > That's a good suggestion. -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Thu Apr 30 21:44:22 2020 From: andyfaff at gmail.com (Andrew Nelson) Date: Fri, 1 May 2020 11:44:22 +1000 Subject: [SciPy-Dev] special.factorial accepting floats Message-ID: The docstring for special.factorial says that the function accepts ints, and int arrays. However, it also seems to accept float arrays (including those that contain np.nan). Is there a rationale for it accepting floats/float arrays? One can't calculate factorials of non-integers. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Apr 30 22:40:08 2020 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 30 Apr 2020 22:40:08 -0400 Subject: [SciPy-Dev] special.factorial accepting floats In-Reply-To: References: Message-ID: On Thu, Apr 30, 2020 at 9:45 PM Andrew Nelson wrote: > The docstring for special.factorial says that the function accepts ints, > and int arrays. > However, it also seems to accept float arrays (including those that > contain np.nan). > > Is there a rationale for it accepting floats/float arrays? One can't > calculate factorials of non-integers. > When `exact=False`, the default, the result is computed with a gamma function, which generalizes the factorial function to non-integers. The answer that you get is a perfectly reasonable, meaningful, non-arbitrary result. When `exact=True`, you will see a `ValueError`, because we defer computation over to `math.factorial()` which does check for integralness because it's doing an exact computation that requires integralness. I guess that we could add in a check for integralness in the `exact=False` case, but to what end? Would we catch anyone's real errors? There's actually a possible use case here for representing integral data that have missing data represented as NaNs (i.e. a float array where all the non-NaN entries are integral-valued despite being floating point numbers). -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Thu Apr 30 23:35:30 2020 From: andyfaff at gmail.com (Andrew Nelson) Date: Fri, 1 May 2020 13:35:30 +1000 Subject: [SciPy-Dev] special.factorial accepting floats In-Reply-To: References: Message-ID: I came across this because I was setting up a CI run against Python3.9. On Py3.9 there is a DeprecationWarning if `math.factorial` is supplied a float: On Py3.8: math.factorial(2.0) --> 2 On Py3.9: math.factorial(2.0) --> 2 and emits a "DeprecationWarning: Using factorial() with floats is deprecated" Presumably with Py >> 3.9: math.factorial(2.0) will raise an Exception As you say, `math.factorial` is used by `special.factorial` for `exact=True`. When Python3.9 is released various tests will start to fail because of that DeprecationWarning. I know it's early days, I'm just trying to be proactive. - how to modify tests to account for that DeprecationWarning when `special.factorial` is supplied a float with `exact=True` - how test code and special.factorial code has to be modified for a post DeprecationWarning period. -------------- next part -------------- An HTML attachment was scrubbed... URL: