From ralf.gommers at googlemail.com Sat Jun 2 10:50:08 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 2 Jun 2012 16:50:08 +0200 Subject: [SciPy-Dev] contributing and maintainers docs Message-ID: Hi all, I've finally merged both the maintainers and contributing howto docs. If module maintainers could add some details on current status/issues and future plans and directions for development, that would be great. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjorn.forsman at gmail.com Sat Jun 2 14:16:52 2012 From: bjorn.forsman at gmail.com (=?UTF-8?Q?Bj=C3=B8rn_Forsman?=) Date: Sat, 2 Jun 2012 20:16:52 +0200 Subject: [SciPy-Dev] New contribution: bode() function for LTI systems In-Reply-To: References: Message-ID: On 20 May 2012 22:34, Pauli Virtanen wrote: > 20.05.2012 22:20, Bj?rn Forsman kirjoitti: >> I'm new on this list. I have been wanting to have a bode() function >> for LTI systems in Python for some time now. Today I got tired of >> waiting and wrote one myself :-) I have sent pull request on github: >> >> https://github.com/scipy/scipy/pull/224 - ENH: ltisys: new bode() function >> >> While at it I also fixed a small bug in the lti class init function: >> >> https://github.com/scipy/scipy/pull/223 - ENH: ltisys: make lti zpk >> initialization work with plain lists >> >> Is this OK? > > Yep, thanks for sending in fixes. > >> Should I post the patches here too? > > No, the pull requests are enough. Would it be possible to merge my changes into the upcoming release? It would make me happy :-) The pull requests: https://github.com/scipy/scipy/pull/224 - ENH: ltisys: new bode() function https://github.com/scipy/scipy/pull/225 - ENH: ltisys: make lti zpk initialization work with plain lists Best regards, Bj?rn Forsman From pav at iki.fi Sat Jun 2 19:12:18 2012 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 03 Jun 2012 01:12:18 +0200 Subject: [SciPy-Dev] New contribution: bode() function for LTI systems In-Reply-To: References: Message-ID: 02.06.2012 20:16, Bj?rn Forsman kirjoitti: [clip] > https://github.com/scipy/scipy/pull/224 - ENH: ltisys: new bode() function > https://github.com/scipy/scipy/pull/225 - ENH: ltisys: make lti zpk > initialization work with plain lists #225 is obvious, and should go in. #224 -- looks mostly good. However, you don't include a test case, and as I'm not familiar with the problem domain, I cannot easily double-check that the code works (and therefore, cannot merge it). Could you include a (simple but nontrivial) calculation where you know the correct result, and that uses this code? You most likely have already done a couple of checks like this already, so coding up one or a few of them as test cases hopefully is not too much work. Having such tests in place is useful for Scipy for several reasons --- for instance, it makes easier to check that new contributions work, and having them in place makes it more certain that any future changes in code (for instance in other parts of Scipy) do not break the functionality. Normally, I or someone else of the regulars can of course write these up themselves, but as Scipy spans a wide field of topics, it can take a while until someone with the correct expertise has time for it... So, it's really helpful if pull requests come with tests. Thanks, -- Pauli Virtanen From junkshops at gmail.com Sun Jun 3 02:13:33 2012 From: junkshops at gmail.com (Junkshops) Date: Sat, 02 Jun 2012 23:13:33 -0700 Subject: [SciPy-Dev] Adding t-test with unequal variances to stats.py In-Reply-To: <4FBC93FA.8010901@gmail.com> References: <4FBC93FA.8010901@gmail.com> Message-ID: <4FCB008D.5040907@gmail.com> After some of the recent list mails advising checking results against R, I checked this code against the R t.test() calls. I hadn't bothered before since I had checked the t statistic and df values, and they were right, and as I was just calling distribution.t.sf() like every other t-test function assumed I should be OK. However, it turns out that the statistics book I used as a source had the df function wrong, believe it or not. Hence I closed the old PR and opened a new one with a fix and more tests here: https://github.com/scipy/scipy/pull/235 Apparently the thing to do is to beg for your PRs to go into the upcoming version. I'm certainly willing to do so, but I was wondering if the scipy devs respond better to other inveiglements - say, threats of world destruction, under the table 'gifts', favors of a lascivious nature, or cake. All proposals considered. Cheers, Gavin On 5/23/2012 12:38 AM, Junkshops wrote: > Hi all, > > I've issued a pull request (http://github.com/scipy/scipy/pull/227) > for a version of scipy/stats/stats.py with the following changes: > > 1) Adds a method for running a t-test with unequal or unknown > population variances. ttest_ind assumes that population variances are > equal. > 2) Refactored common code in the 4 t-test methods into shared methods. > 3) This section of code, which has variations in multiple methods, > looks buggy to me: > > d = np.mean(a,axis) - np.mean(b,axis) > svar = ((n1-1)*v1+(n2-1)*v2) / float(df) > > t = d/np.sqrt(svar*(1.0/n1 + 1.0/n2)) > t = np.where((d==0)*(svar==0), 1.0, t) #define t=0/0 = 0, identical means > > Surely if d=0, regardless of svar, t should be set to 0, not 1. > Similarly, if svar = 0 then both variances are zero (assuming that > each data set has at least 2 points - perhaps there should be a check > for this?). In that case, if d==0 t should be zero. Otherwise, t > should be +/-inf. Hence, (svar==0) is redundant. > > Accordingly, I've changed the lines in all functions to be the > equivalent of > > t = np.where((d==0), 0.0, t) > > This handles the case where both d and svar are 0. The respective > tests have also been changed. > > If I'm missing something here, please let me know. > > Thanks, Gavin > From pav at iki.fi Sun Jun 3 06:02:27 2012 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 03 Jun 2012 12:02:27 +0200 Subject: [SciPy-Dev] Adding t-test with unequal variances to stats.py In-Reply-To: <4FCB008D.5040907@gmail.com> References: <4FBC93FA.8010901@gmail.com> <4FCB008D.5040907@gmail.com> Message-ID: 03.06.2012 08:13, Junkshops kirjoitti: [clip] > Apparently the thing to do is to beg for your PRs to go into the > upcoming version. I'm certainly willing to do so, but I was wondering if > the scipy devs respond better to other inveiglements - say, threats of > world destruction, under the table 'gifts', favors of a lascivious > nature, or cake. All proposals considered. The only thing needed is someone (= *anyone*, this doesn't have to be a "scipy developer") with some familiarity in stats to look at it, and say: "I know this stuff, and looks correct to me." So far, it seems nobody has found the interest to chip in. However, having the comparison to R in this case may be enough by itself to make it clear even to non-experts that things work now as intended. We nevertheless do have a "blame list" of people who have promised to be available for scipy.stats on some time scale: https://github.com/scipy/scipy/blob/master/doc/MAINTAINERS.rst.txt -- Pauli Virtanen From bjorn.forsman at gmail.com Sun Jun 3 13:07:02 2012 From: bjorn.forsman at gmail.com (=?UTF-8?Q?Bj=C3=B8rn_Forsman?=) Date: Sun, 3 Jun 2012 19:07:02 +0200 Subject: [SciPy-Dev] New contribution: bode() function for LTI systems In-Reply-To: References: Message-ID: On 3 June 2012 01:12, Pauli Virtanen wrote: > 02.06.2012 20:16, Bj?rn Forsman kirjoitti: > [clip] >> https://github.com/scipy/scipy/pull/224 - ENH: ltisys: new bode() function >> https://github.com/scipy/scipy/pull/225 - ENH: ltisys: make lti zpk >> initialization work with plain lists > > #225 is obvious, and should go in. I see it has been merged. Thanks. > #224 -- looks mostly good. However, you don't include a test case, and > as I'm not familiar with the problem domain, I cannot easily > double-check that the code works (and therefore, cannot merge it). > > Could you include a (simple but nontrivial) calculation where you know > the correct result, and that uses this code? ?You most likely have > already done a couple of checks like this already, so coding up one or a > few of them as test cases hopefully is not too much work. I have checked this bode() against the bode function in Matlab. I have now added (and pushed) 4 tests; test 1 and 2 sanity-checks the bode() magnitude and phase data against manually entered data. Test 3 and 4 calculate the magnitude and phase using the same algorithm as the bode() function, but without actually *using* bode(). This way you/we will be able to catch regression bugs. Let me know if there is anything more I need to do to get this merged. BTW, I think #224 will cause a small merge conflict in test_ltisys.py. How do you prefer to handle this? I normally rebase but some people don't like rebasing (so I thought I'd ask before doing it) and I don't know what github does to a pull-request if I rebase the branch. Best regards, Bj?rn Forsman From ralf.gommers at googlemail.com Sun Jun 3 13:57:55 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 3 Jun 2012 19:57:55 +0200 Subject: [SciPy-Dev] New contribution: bode() function for LTI systems In-Reply-To: References: Message-ID: On Sun, Jun 3, 2012 at 7:07 PM, Bj?rn Forsman wrote: > On 3 June 2012 01:12, Pauli Virtanen wrote: > > 02.06.2012 20:16, Bj?rn Forsman kirjoitti: > > [clip] > >> https://github.com/scipy/scipy/pull/224 - ENH: ltisys: new bode() > function > >> https://github.com/scipy/scipy/pull/225 - ENH: ltisys: make lti zpk > >> initialization work with plain lists > > > > #225 is obvious, and should go in. > > I see it has been merged. Thanks. > > > #224 -- looks mostly good. However, you don't include a test case, and > > as I'm not familiar with the problem domain, I cannot easily > > double-check that the code works (and therefore, cannot merge it). > > > > Could you include a (simple but nontrivial) calculation where you know > > the correct result, and that uses this code? You most likely have > > already done a couple of checks like this already, so coding up one or a > > few of them as test cases hopefully is not too much work. > > I have checked this bode() against the bode function in Matlab. > > I have now added (and pushed) 4 tests; test 1 and 2 sanity-checks the > bode() magnitude and phase data against manually entered data. Test 3 > and 4 calculate the magnitude and phase using the same algorithm as > the bode() function, but without actually *using* bode(). This way > you/we will be able to catch regression bugs. > > Let me know if there is anything more I need to do to get this merged. > Those tests look good. > > BTW, I think #224 will cause a small merge conflict in test_ltisys.py. > How do you prefer to handle this? I normally rebase but some people > don't like rebasing (so I thought I'd ask before doing it) and I don't > know what github does to a pull-request if I rebase the branch. > You can rebase and push again to the same branch (you need to force push). The only reason not to do that would be if there are a lot of comments directly on commits, because they'll be lost after rebasing. In this case there aren't, so rebasing is preferred to opening a new PR. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjorn.forsman at gmail.com Sun Jun 3 14:33:39 2012 From: bjorn.forsman at gmail.com (=?UTF-8?Q?Bj=C3=B8rn_Forsman?=) Date: Sun, 3 Jun 2012 20:33:39 +0200 Subject: [SciPy-Dev] New contribution: bode() function for LTI systems In-Reply-To: References: Message-ID: On 3 June 2012 19:57, Ralf Gommers wrote: [...] > You can rebase and push again to the same branch (you need to force push). > The only reason not to do that would be if there are a lot of comments > directly on commits, because they'll be lost after rebasing. In this case > there aren't, so rebasing is preferred to opening a new PR. Ok, now I have rebased, cleaned up the history a bit (2 commits instead of 4) and pushed: https://github.com/scipy/scipy/pull/224 Best regards, Bj?rn Forsman From ralf.gommers at googlemail.com Sun Jun 3 16:30:51 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 3 Jun 2012 22:30:51 +0200 Subject: [SciPy-Dev] Trac timeline broken Message-ID: Just noticed that http://projects.scipy.org/scipy/timeline is broken: Genshi UnicodeDecodeError error while rendering template '/srv/scipy/python-packages/lib/python2.4/site-packages/Trac-0.11.3-py2.4.egg/trac/timeline/templates/timeline.html', line 42, char -1 This is likely due to fc509992a, because Bjorn has a non-ascii name. That means it'll stay broken for about a month, unless someone has a look at it. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjorn.forsman at gmail.com Sun Jun 3 16:46:25 2012 From: bjorn.forsman at gmail.com (=?UTF-8?Q?Bj=C3=B8rn_Forsman?=) Date: Sun, 3 Jun 2012 22:46:25 +0200 Subject: [SciPy-Dev] Trac timeline broken In-Reply-To: References: Message-ID: On 3 June 2012 22:30, Ralf Gommers wrote: > Just noticed that http://projects.scipy.org/scipy/timeline is broken: > > Genshi UnicodeDecodeError error while rendering template > '/srv/scipy/python-packages/lib/python2.4/site-packages/Trac-0.11.3-py2.4.egg/trac/timeline/templates/timeline.html', > line 42, char -1 > > This is likely due to fc509992a, because Bjorn has a non-ascii name. That > means it'll stay broken for about a month, unless someone has a look at it. Sorry about that! :-) - Bj?rn From ralf.gommers at googlemail.com Sun Jun 3 16:49:32 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 3 Jun 2012 22:49:32 +0200 Subject: [SciPy-Dev] Trac timeline broken In-Reply-To: References: Message-ID: On Sun, Jun 3, 2012 at 10:46 PM, Bj?rn Forsman wrote: > On 3 June 2012 22:30, Ralf Gommers wrote: > > Just noticed that http://projects.scipy.org/scipy/timeline is broken: > > > > Genshi UnicodeDecodeError error while rendering template > > > '/srv/scipy/python-packages/lib/python2.4/site-packages/Trac-0.11.3-py2.4.egg/trac/timeline/templates/timeline.html', > > line 42, char -1 > > > > This is likely due to fc509992a, because Bjorn has a non-ascii name. That > > means it'll stay broken for about a month, unless someone has a look at > it. > > Sorry about that! :-) > I'd be perfectly fine with it being broken all the time, if that's because you contribute a nice improvement like signal.bode() every month:) Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sun Jun 3 16:55:16 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 3 Jun 2012 22:55:16 +0200 Subject: [SciPy-Dev] test failures with python 2.4 In-Reply-To: <1338479402.4062.24.camel@schloss.campus.mcgill.ca> References: <1338479402.4062.24.camel@schloss.campus.mcgill.ca> Message-ID: On Thu, May 31, 2012 at 5:50 PM, Denis Laxalde wrote: > Hi, > > Here is a list of test failures with python 2.4 (there are also failures > with 2.7, see below). > > Thanks. I see those too. Should all be fixed before the first beta ideally. Only the weave test we don't have to care about anymore. Ralf > Interpolate: > > test_interpnd.TestCloughTocher2DInterpolator.test_dense ... > Exception exceptions.TypeError: 0x7fd4d67fecf8> in 'scipy.spatial.qhull._get_delaunay_info' ignored > Segmentation fault > > > Spatial: > > test_qhull.TestUtilities.test_convex_hull ... ERROR > test_qhull.TestUtilities.test_degenerate_barycentric_transforms ... > ERROR > test_qhull.TestUtilities.test_find_simplex ... Exception > exceptions.TypeError: in > 'scipy.spatial.qhull._get_delaunay_info' ignored > FAIL > test_qhull.TestUtilities.test_more_barycentric_transforms ... ERROR > test_qhull.TestUtilities.test_plane_distance ... ok > > > ====================================================================== > ERROR: test_qhull.TestUtilities.test_convex_hull > > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/denis/.local/lib/python2.4/site-packages/nose/case.py", line 197, in > runTest > self.test(*self.arg) > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/spatial/tests/test_qhull.py", > line 75, in test_convex_hull > assert_equal(tri.convex_hull, [[1, 2], [3, 2], [1, 0], [3, 0]]) > File "qhull.pyx", line 1109, in > scipy.spatial.qhull.Delaunay.convex_hull (scipy/spatial/qhull.c:8406) > ValueError: cannot resize an array references or is referenced > by another array in this way. Use the resize function > > > ====================================================================== > ERROR: > test_qhull.TestUtilities.test_degenerate_barycentric_transforms > > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/denis/.local/lib/python2.4/site-packages/nose/case.py", line 197, in > runTest > self.test(*self.arg) > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/spatial/tests/test_qhull.py", > line 133, in test_degenerate_barycentric_transforms > bad_count = np.isnan(tri.transform[:,0,0]).sum() > File "qhull.pyx", line 1028, in > scipy.spatial.qhull.Delaunay.transform (scipy/spatial/qhull.c:7449) > File "qhull.pyx", line 398, in > scipy.spatial.qhull._get_barycentric_transforms (scipy/spatial/qhull.c:4300) > File > "/home/denis/.local/lib/python2.4/site-packages/numpy/core/numeric.py", > line 235, in asarray > return array(a, dtype, copy=False, order=order) > File "stringsource", line 366, in > View.MemoryView.memoryview.__getitem__ (scipy/spatial/qhull.c:14704) > File "stringsource", line 650, in View.MemoryView._unellipsify > (scipy/spatial/qhull.c:17965) > TypeError: Cannot index with type '' > > > ====================================================================== > ERROR: test_qhull.TestUtilities.test_more_barycentric_transforms > > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/denis/.local/lib/python2.4/site-packages/nose/case.py", line 197, in > runTest > self.test(*self.arg) > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/spatial/tests/test_qhull.py", > line 157, in test_more_barycentric_transforms > unit_cube=True) > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/spatial/tests/test_qhull.py", > line 96, in _check_barycentric_transforms > c = barycentric_transform(tri.transform, centroids) > File "qhull.pyx", line 1028, in > scipy.spatial.qhull.Delaunay.transform (scipy/spatial/qhull.c:7449) > File "qhull.pyx", line 398, in > scipy.spatial.qhull._get_barycentric_transforms (scipy/spatial/qhull.c:4300) > File > "/home/denis/.local/lib/python2.4/site-packages/numpy/core/numeric.py", > line 235, in asarray > return array(a, dtype, copy=False, order=order) > File "stringsource", line 366, in > View.MemoryView.memoryview.__getitem__ (scipy/spatial/qhull.c:14704) > File "stringsource", line 650, in View.MemoryView._unellipsify > (scipy/spatial/qhull.c:17965) > TypeError: Cannot index with type '' > > > ====================================================================== > FAIL: test_qhull.TestUtilities.test_find_simplex > > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/denis/.local/lib/python2.4/site-packages/nose/case.py", line 197, in > runTest > self.test(*self.arg) > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/spatial/tests/test_qhull.py", > line 33, in test_find_simplex > assert_equal(i, p[2], err_msg='%r' % (p,)) > File > "/home/denis/.local/lib/python2.4/site-packages/numpy/testing/utils.py", > line 256, in assert_equal > return assert_array_equal(actual, desired, err_msg, verbose) > File > "/home/denis/.local/lib/python2.4/site-packages/numpy/testing/utils.py", > line 707, in assert_array_equal > verbose=verbose, header='Arrays are not equal') > File > "/home/denis/.local/lib/python2.4/site-packages/numpy/testing/utils.py", > line 636, in assert_array_compare > raise AssertionError(msg) > AssertionError: > Arrays are not equal > (0.25, 0.25, 1) > (mismatch 100.0%) > x: array(-1, dtype=int32) > y: array(1) > > > ---------------------------------------------------------------------- > Ran 291 tests in 13.335s > > FAILED (errors=3, failures=1) > > > > Weave: > > FAIL: test_set_complex (test_scxx_object.TestObjectSetItemOpKey) > > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/weave/tests/test_scxx_object.py", > line 943, in test_set_complex > assert_equal(sys.getrefcount(key),4) # should be 3 > File > "/home/denis/.local/lib/python2.4/site-packages/numpy/testing/utils.py", > line 313, in assert_equal > raise AssertionError(msg) > AssertionError: > Items are not equal: > ACTUAL: 3 > DESIRED: 4 > > > Failures not specific to python 2.4. > > Sparse: > > > ====================================================================== > ERROR: adding a dense matrix to a sparse matrix > > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/tests/test_base.py", > line 527, in test_add_dense > sum1 = self.dat + self.datsp > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/dok.py", line > 133, in __getitem__ > raise TypeError('index must be a pair of integers or slices') > TypeError: index must be a pair of integers or slices > > > ====================================================================== > ERROR: test_matmat_sparse (test_base.TestDOK) > > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/tests/test_base.py", > line 425, in test_matmat_sparse > assert_array_almost_equal( a2*bsp, a*b) > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/dok.py", line > 133, in __getitem__ > raise TypeError('index must be a pair of integers or slices') > TypeError: index must be a pair of integers or slices > > > ====================================================================== > ERROR: test_radd (test_base.TestDOK) > > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/tests/test_base.py", > line 279, in test_radd > c = a + b > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/dok.py", line > 133, in __getitem__ > raise TypeError('index must be a pair of integers or slices') > TypeError: index must be a pair of integers or slices > > > ====================================================================== > ERROR: test_rsub (test_base.TestDOK) > > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/tests/test_base.py", > line 290, in test_rsub > assert_array_equal((self.dat - > self.datsp),[[0,0,0,0],[0,0,0,0],[0,0,0,0]]) > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/dok.py", line > 133, in __getitem__ > raise TypeError('index must be a pair of integers or slices') > TypeError: index must be a pair of integers or slices > > > ====================================================================== > ERROR: subtracting a dense matrix to/from a sparse matrix > > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/tests/test_base.py", > line 535, in test_sub_dense > sum1 = 3*self.dat - self.datsp > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/dok.py", line > 133, in __getitem__ > raise TypeError('index must be a pair of integers or slices') > TypeError: index must be a pair of integers or slices > > > ---------------------------------------------------------------------- > Ran 1210 tests in 12.543s > > FAILED (KNOWNFAIL=4, errors=5) > > Special: > > > ................../home/denis/.local/lib/python2.7/site-packages/scipy/special/tests/test_basic.py:1645: > RuntimeWarning: divide by zero encountered in iv > c1 = special.iv(v, x) > > /home/denis/.local/lib/python2.7/site-packages/scipy/special/tests/test_basic.py:1645: > RuntimeWarning: overflow encountered in iv > c1 = special.iv(v, x) > > /home/denis/.local/lib/python2.7/site-packages/scipy/special/tests/test_basic.py:1646: > RuntimeWarning: invalid value encountered in iv > c2 = special.iv(v, x+0j) > > /home/denis/.local/lib/python2.7/site-packages/scipy/special/tests/test_basic.py:1652: > RuntimeWarning: divide by zero encountered in divide > dc = abs(c1/c2 - 1) > > /home/denis/.local/lib/python2.7/site-packages/scipy/special/tests/test_basic.py:1652: > RuntimeWarning: invalid value encountered in divide > dc = abs(c1/c2 - 1) > > /home/denis/.local/lib/python2.7/site-packages/scipy/special/tests/test_basic.py:1659: > RuntimeWarning: overflow encountered in iv > assert_(dc[k] < 1e-9, (v[k], x[k], special.iv(v[k], x[k]), > special.iv(v[k], x[k]+0j))) > > F...................................................K.K.............................................................................................................................................................................................................................................................................................................................................................................................K........K.................................................. > > ====================================================================== > FAIL: test_iv_cephes_vs_amos_mass_test (test_basic.TestBessel) > > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/denis/.local/lib/python2.7/site-packages/scipy/special/tests/test_basic.py", > line 1659, in test_iv_cephes_vs_amos_mass_test > assert_(dc[k] < 1e-9, (v[k], x[k], special.iv(v[k], x[k]), > special.iv(v[k], x[k]+0j))) > File "/usr/lib/pymodules/python2.7/numpy/testing/utils.py", line > 34, in assert_ > raise AssertionError(msg) > AssertionError: (189.2947429454936, 3.0238805556481037, > 4.089165443940765e-317, 0j) > > > ---------------------------------------------------------------------- > Ran 514 tests in 6.908s > > FAILED (KNOWNFAIL=4, failures=1) > > > Building/testing environment for python 2.4 (Debian wheezy/sid x86_64): > > NumPy version 1.6.2 > NumPy is installed in > /home/denis/.local/lib/python2.4/site-packages/numpy > SciPy version 0.11.0.dev-127acfd > SciPy is installed in > /home/denis/.local/lib/python2.4/site-packages/scipy > Python version 2.4.6 (#2, Sep 25 2009, 22:22:06) [GCC 4.3.4] > nose version 1.1.2 > > > -- > Denis > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sun Jun 3 17:53:59 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 3 Jun 2012 17:53:59 -0400 Subject: [SciPy-Dev] Trac timeline broken In-Reply-To: References: Message-ID: On Sun, Jun 3, 2012 at 4:49 PM, Ralf Gommers wrote: > > > On Sun, Jun 3, 2012 at 10:46 PM, Bj?rn Forsman > wrote: >> >> On 3 June 2012 22:30, Ralf Gommers wrote: >> > Just noticed that http://projects.scipy.org/scipy/timeline is broken: >> > >> > Genshi UnicodeDecodeError error while rendering template >> > >> > '/srv/scipy/python-packages/lib/python2.4/site-packages/Trac-0.11.3-py2.4.egg/trac/timeline/templates/timeline.html', >> > line 42, char -1 >> > >> > This is likely due to fc509992a, because Bjorn has a non-ascii name. >> > That >> > means it'll stay broken for about a month, unless someone has a look at >> > it. I don't see any problems (maybe already solved) timeline looks fine to me with commits like Commit [97160c73] by Bj?rn Forsman Josef >> >> Sorry about that! :-) > > > I'd be perfectly fine with it being broken all the time, if that's because > you contribute a nice improvement like signal.bode() every month:) > > Ralf > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > From denis at laxalde.org Mon Jun 4 10:33:37 2012 From: denis at laxalde.org (Denis Laxalde) Date: Mon, 04 Jun 2012 10:33:37 -0400 Subject: [SciPy-Dev] contributing and maintainers docs In-Reply-To: References: Message-ID: <1338820417.2977.17.camel@schloss.campus.mcgill.ca> Ralf Gommers a ?crit : > If > module maintainers could add some details on current status/issues and > future plans and directions for development, that would be great. What level of details is expected here? Wouldn't it make more sense to put this kind of information in a wiki? (BTW, some times ago, it was suggested that the development wiki could be in github, is there a consensus on this?) -- Denis Laxalde From ralf.gommers at googlemail.com Mon Jun 4 11:27:10 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 4 Jun 2012 17:27:10 +0200 Subject: [SciPy-Dev] contributing and maintainers docs In-Reply-To: <1338820417.2977.17.camel@schloss.campus.mcgill.ca> References: <1338820417.2977.17.camel@schloss.campus.mcgill.ca> Message-ID: On Mon, Jun 4, 2012 at 4:33 PM, Denis Laxalde wrote: > Ralf Gommers a ?crit : > > If > > module maintainers could add some details on current status/issues and > > future plans and directions for development, that would be great. > > What level of details is expected here? > Something quite high-level I think, say between one sentence and two paragraphs. The constants and weave parts are one sentence, and that's enough. For large modules like stats or sparse I'd say one paragraph on status (what chucks of functionality are/aren't tested or documented, where are improvements needed) and one paragraph on future plans that developers have or that have been discussed on the list before. > Wouldn't it make more sense to put this kind of information in a wiki? > That tends to be maintained less well, can't be included in the refguide and can be easily spammed. Those are also reasons why we'd like to move the website from wiki-based to Sphinx-generated. > (BTW, some times ago, it was suggested that the development wiki could > be in github, is there a consensus on this?) > I thought so. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Mon Jun 4 14:56:39 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 4 Jun 2012 20:56:39 +0200 Subject: [SciPy-Dev] test failures with python 2.4 In-Reply-To: <1338479402.4062.24.camel@schloss.campus.mcgill.ca> References: <1338479402.4062.24.camel@schloss.campus.mcgill.ca> Message-ID: On Thu, May 31, 2012 at 5:50 PM, Denis Laxalde wrote: > > Sparse: > > > ====================================================================== > ERROR: adding a dense matrix to a sparse matrix > > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/tests/test_base.py", > line 527, in test_add_dense > sum1 = self.dat + self.datsp > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/dok.py", line > 133, in __getitem__ > raise TypeError('index must be a pair of integers or slices') > TypeError: index must be a pair of integers or slices > > > ====================================================================== > ERROR: test_matmat_sparse (test_base.TestDOK) > > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/tests/test_base.py", > line 425, in test_matmat_sparse > assert_array_almost_equal( a2*bsp, a*b) > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/dok.py", line > 133, in __getitem__ > raise TypeError('index must be a pair of integers or slices') > TypeError: index must be a pair of integers or slices > > > ====================================================================== > ERROR: test_radd (test_base.TestDOK) > > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/tests/test_base.py", > line 279, in test_radd > c = a + b > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/dok.py", line > 133, in __getitem__ > raise TypeError('index must be a pair of integers or slices') > TypeError: index must be a pair of integers or slices > > > ====================================================================== > ERROR: test_rsub (test_base.TestDOK) > > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/tests/test_base.py", > line 290, in test_rsub > assert_array_equal((self.dat - > self.datsp),[[0,0,0,0],[0,0,0,0],[0,0,0,0]]) > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/dok.py", line > 133, in __getitem__ > raise TypeError('index must be a pair of integers or slices') > TypeError: index must be a pair of integers or slices > > > ====================================================================== > ERROR: subtracting a dense matrix to/from a sparse matrix > > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/tests/test_base.py", > line 535, in test_sub_dense > sum1 = 3*self.dat - self.datsp > File > "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/dok.py", line > 133, in __getitem__ > raise TypeError('index must be a pair of integers or slices') > TypeError: index must be a pair of integers or slices > > > ---------------------------------------------------------------------- > Ran 1210 tests in 12.543s > > FAILED (KNOWNFAIL=4, errors=5) > > These are silences by https://github.com/scipy/scipy/pull/237. See http://projects.scipy.org/scipy/ticket/1559 for details. > Special: > > > ................../home/denis/.local/lib/python2.7/site-packages/scipy/special/tests/test_basic.py:1645: > RuntimeWarning: divide by zero encountered in iv > c1 = special.iv(v, x) > > /home/denis/.local/lib/python2.7/site-packages/scipy/special/tests/test_basic.py:1645: > RuntimeWarning: overflow encountered in iv > c1 = special.iv(v, x) > > /home/denis/.local/lib/python2.7/site-packages/scipy/special/tests/test_basic.py:1646: > RuntimeWarning: invalid value encountered in iv > c2 = special.iv(v, x+0j) > > /home/denis/.local/lib/python2.7/site-packages/scipy/special/tests/test_basic.py:1652: > RuntimeWarning: divide by zero encountered in divide > dc = abs(c1/c2 - 1) > > /home/denis/.local/lib/python2.7/site-packages/scipy/special/tests/test_basic.py:1652: > RuntimeWarning: invalid value encountered in divide > dc = abs(c1/c2 - 1) > > /home/denis/.local/lib/python2.7/site-packages/scipy/special/tests/test_basic.py:1659: > RuntimeWarning: overflow encountered in iv > assert_(dc[k] < 1e-9, (v[k], x[k], special.iv(v[k], x[k]), > special.iv(v[k], x[k]+0j))) > > F...................................................K.K.............................................................................................................................................................................................................................................................................................................................................................................................K........K.................................................. > > ====================================================================== > FAIL: test_iv_cephes_vs_amos_mass_test (test_basic.TestBessel) > > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/denis/.local/lib/python2.7/site-packages/scipy/special/tests/test_basic.py", > line 1659, in test_iv_cephes_vs_amos_mass_test > assert_(dc[k] < 1e-9, (v[k], x[k], special.iv(v[k], x[k]), > special.iv(v[k], x[k]+0j))) > File "/usr/lib/pymodules/python2.7/numpy/testing/utils.py", line > 34, in assert_ > raise AssertionError(msg) > AssertionError: (189.2947429454936, 3.0238805556481037, > 4.089165443940765e-317, 0j) > > This one I can't reproduce. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From denis at laxalde.org Mon Jun 4 15:05:06 2012 From: denis at laxalde.org (Denis Laxalde) Date: Mon, 04 Jun 2012 15:05:06 -0400 Subject: [SciPy-Dev] test failures with python 2.4 In-Reply-To: References: <1338479402.4062.24.camel@schloss.campus.mcgill.ca> Message-ID: <1338836706.2977.27.camel@schloss.campus.mcgill.ca> Le lundi 04 juin 2012 ? 20:56 +0200, Ralf Gommers a ?crit : > > Sparse: > These are silences by https://github.com/scipy/scipy/pull/237. See > http://projects.scipy.org/scipy/ticket/1559 for details. Looks reasonable. > > Special: [...] > > ====================================================================== > > FAIL: test_iv_cephes_vs_amos_mass_test (test_basic.TestBessel) > > > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File > > "/home/denis/.local/lib/python2.7/site-packages/scipy/special/tests/test_basic.py", > > line 1659, in test_iv_cephes_vs_amos_mass_test > > assert_(dc[k] < 1e-9, (v[k], x[k], special.iv(v[k], x[k]), > > special.iv(v[k], x[k]+0j))) > > File "/usr/lib/pymodules/python2.7/numpy/testing/utils.py", line > > 34, in assert_ > > raise AssertionError(msg) > > AssertionError: (189.2947429454936, 3.0238805556481037, > > 4.089165443940765e-317, 0j) > > > This one I can't reproduce. Did you run the test with 'full'? This particular test is marked as slow. From ralf.gommers at googlemail.com Mon Jun 4 15:10:07 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 4 Jun 2012 21:10:07 +0200 Subject: [SciPy-Dev] test failures with python 2.4 In-Reply-To: <1338836706.2977.27.camel@schloss.campus.mcgill.ca> References: <1338479402.4062.24.camel@schloss.campus.mcgill.ca> <1338836706.2977.27.camel@schloss.campus.mcgill.ca> Message-ID: On Mon, Jun 4, 2012 at 9:05 PM, Denis Laxalde wrote: > Le lundi 04 juin 2012 ? 20:56 +0200, Ralf Gommers a ?crit : > > > Sparse: > > > These are silences by https://github.com/scipy/scipy/pull/237. See > > http://projects.scipy.org/scipy/ticket/1559 for details. > > Looks reasonable. > > > > Special: > [...] > > > ====================================================================== > > > FAIL: test_iv_cephes_vs_amos_mass_test (test_basic.TestBessel) > > > > > > ---------------------------------------------------------------------- > > > Traceback (most recent call last): > > > File > > > > "/home/denis/.local/lib/python2.7/site-packages/scipy/special/tests/test_basic.py", > > > line 1659, in test_iv_cephes_vs_amos_mass_test > > > assert_(dc[k] < 1e-9, (v[k], x[k], special.iv(v[k], x[k]), > > > special.iv(v[k], x[k]+0j))) > > > File "/usr/lib/pymodules/python2.7/numpy/testing/utils.py", > line > > > 34, in assert_ > > > raise AssertionError(msg) > > > AssertionError: (189.2947429454936, 3.0238805556481037, > > > 4.089165443940765e-317, 0j) > > > > > This one I can't reproduce. > > Did you run the test with 'full'? This particular test is marked as > slow. > Yes, I know. Doesn't reproduce. It did sometime in the past though: http://projects.scipy.org/scipy/ticket/1453 Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Tue Jun 5 02:07:00 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 5 Jun 2012 08:07:00 +0200 Subject: [SciPy-Dev] scipy 0.11.0 release schedule In-Reply-To: References: Message-ID: On Wed, May 30, 2012 at 8:00 PM, Pauli Virtanen wrote: > Hi, > > 28.05.2012 17:26, Ralf Gommers kirjoitti: > > It's time for a new release I think; 0.10.0 was released just over 6 > > months ago. There are a lot of PRs to merge, but no release blockers > > anymore as far as I know. Does anyone still have important fixes or > > other things that should go in? > > > > For the release schedule I propose: > > June 7: beta 1 > > June 17: rc 1 > > June 30: rc 2 > > July 7: final release > > The schedule seems possible. > Everything will have to shift by half a week; there are still too much PRs to merge and some failures on 2.4 to be fixed. There aren't any serious issues though, so branching 0.11.x on Saturday should be possible. Help merging PRs would be much appreciated. If maintainers could look at what's still open for their module (not more than a couple of PRs per module) and comment on those, we can get as many contributions as possible in. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From junkshops at gmail.com Wed Jun 6 16:18:11 2012 From: junkshops at gmail.com (Junkshops) Date: Wed, 06 Jun 2012 13:18:11 -0700 Subject: [SciPy-Dev] Resolving PR 235: t-statistic = 0/0 case Message-ID: <4FCFBB03.4000102@gmail.com> Hi all, This mail references PR 235: https://github.com/scipy/scipy/pull/235 The PR adds a method for performing a t-test on 2 samples with unequal or unknown variances and changes the return value when the t-statistic = 0 / 0 to 0 from the previous code's return value, 1, for all t-test functions. The latter change is a point of contention for the pull. I take the position that t = 0/0 should return 0, while Josef, the primary scipy.stats maintainer, believes that t = 0/0 should return one. Normally I would try to resolve this directly with Josef, but unfortunately I haven't heard from him in over two days, and the beta release is scheduled for Saturday. As such, I'm writing the dev list to ask what to do next. Josef's position, to the best of my ability to understand it, is that: J1) Adding a small amount of noise to data sets that have otherwise equal means (e.g. [0,0,0,0], [0,0,0,1e-100]) results in a t-statistic of 1. Thus, as the mean difference approaches zero, t -> 1. J2) A data set with no mean difference and no variance is a zero probability event. As such, returning t = 1 is reasonable, as therefore p = 0.317-0.5 for a two tailed test depending on the degrees of freedom, and hence for standard values of alpha the null hypothesis will not be rejected, but the user gets some feedback that his data is suspect. I admit I don't completely understand the second argument. Hopefully when Josef resurfaces he can correct my representation of his argument if needed. My responses to these arguments are: J1) If you take the n-length vectors (x,...,x) and (x,...x,x+y) and solve for t, t = 1 for any value of x and y. This is simply due to the fact that the the mean difference is y/n and the pooled variance (e.g. the denominator of the t-statistic) is also y/n. Thus this is merely a special case and does not represent a limit as the mean difference approaches zero, since even for [0,0,0,0], [0,0,0,1e100] t = 1. J2) Strictly speaking, if we're pulling independent random samples from a normal distribution over the reals, say, any given set of samples has zero probability since there are an infinite number of possible samples. The sample [0,0,0] is just as probable as the sample [2.3, 7.4, 2.1]. In a discrete distribution, in fact, the sample [0,0,0] is *more* likely than any other sample. Also, we're now analyzing the legitimacy of the user's data, which is not the job of the t-test. The t-test is expected to take a data set(s) and provide a measure of the strength of the evidence that the means are different. Is it not expected to comment on the validity of the user's data. My arguments that 0 is the correct answer to return are: 1) if MD = mean difference and PV = pooled variance, then t = MD/PV. As MD -> 0 for any PV > 0, t -> 0. However, if we set t = 0/0 = 1 and MD = 0, as PV -> 0 we introduce a discontinuity at zero since t = 0 for any value of PV except 0, where t = 1. This implies that for MD = PV = 0, t = 1, but if the variance of the dataset is infinitesimally small but !=0, then t = 0. To me, this makes no sense. 2) The t-test's purpose is to measure the strength of the evidence against the null hypothesis that MD = 0. If MD = 0, as in the case we're discussing, by definition there is no evidence that MD != 0. Therefore p must = 1, and as a consequence t must = 0. I also consulted with a statistics professor - who agreed with my position - to make sure I wasn't talking out of turn. In summary, I think that if t = 0/0 we should return 0. However, both R and Matlab return NaN. To me this seems incorrect due to argument #2. Josef also mentioned there were users of scipy.stats.stats that didn't want NaN return values for some reason that I don't know offhand. How would the Scipy devs like to proceed? Cheers, Gavin From njs at pobox.com Wed Jun 6 16:48:26 2012 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 6 Jun 2012 21:48:26 +0100 Subject: [SciPy-Dev] Resolving PR 235: t-statistic = 0/0 case In-Reply-To: <4FCFBB03.4000102@gmail.com> References: <4FCFBB03.4000102@gmail.com> Message-ID: On Wed, Jun 6, 2012 at 9:18 PM, Junkshops wrote: > Hi all, > > This mail references PR 235: https://github.com/scipy/scipy/pull/235 > > The PR adds a method for performing a t-test on 2 samples with unequal > or unknown variances and changes the return value when the t-statistic = > 0 / 0 to 0 from the previous code's return value, 1, for all t-test > functions. > > The latter change is a point of contention for the pull. I take the > position that t = 0/0 should return 0, while Josef, the primary > scipy.stats maintainer, believes that t = 0/0 should return one. > Normally I would try to resolve this directly with Josef, but > unfortunately I haven't heard from him in over two days, and the beta > release is scheduled for Saturday. As such, I'm writing the dev list to > ask what to do next. > > Josef's position, to the best of my ability to understand it, is that: > > J1) Adding a small amount of noise to data sets that have otherwise > equal means (e.g. [0,0,0,0], [0,0,0,1e-100]) results in a t-statistic of > 1. Thus, as the mean difference approaches zero, t -> 1. > J2) A data set with no mean difference and no variance is a zero > probability event. As such, returning t = 1 is reasonable, as therefore > p = 0.317-0.5 for a two tailed test depending on the degrees of freedom, > and hence for standard values of alpha the null hypothesis will not be > rejected, but the user gets some feedback that his data is suspect. > > I admit I don't completely understand the second argument. Hopefully > when Josef resurfaces he can correct my representation of his argument > if needed. > > My responses to these arguments are: > > J1) If you take the n-length vectors (x,...,x) and (x,...x,x+y) and > solve for t, t = 1 for any value of x and y. This is simply due to the > fact that the the mean difference is y/n and the pooled variance (e.g. > the denominator of the t-statistic) is also y/n. Thus this is merely a > special case and does not represent a limit as the mean difference > approaches zero, since even for [0,0,0,0], [0,0,0,1e100] t = 1. > > J2) Strictly speaking, if we're pulling independent random samples from > a normal distribution over the reals, say, any given set of samples has > zero probability since there are an infinite number of possible samples. > The sample [0,0,0] is just as probable as the sample [2.3, 7.4, 2.1]. In > a discrete distribution, in fact, the sample [0,0,0] is *more* likely > than any other sample. Also, we're now analyzing the legitimacy of the > user's data, which is not the job of the t-test. The t-test is expected > to take a data set(s) and provide a measure of the strength of the > evidence that the means are different. Is it not expected to comment on > the validity of the user's data. > > My arguments that 0 is the correct answer to return are: > > 1) if MD = mean difference and PV = pooled variance, then t = MD/PV. As > MD -> 0 for any PV > 0, t -> 0. However, if we set t = 0/0 = 1 and MD = > 0, as PV -> 0 we introduce a discontinuity at zero since t = 0 for any > value of PV except 0, where t = 1. This implies that for MD = PV = 0, t > = 1, but if the variance of the dataset is infinitesimally small but > !=0, then t = 0. To me, this makes no sense. > > 2) The t-test's purpose is to measure the strength of the evidence > against the null hypothesis that MD = 0. If MD = 0, as in the case we're > discussing, by definition there is no evidence that MD != 0. Therefore p > must = 1, and as a consequence t must = 0. > > I also consulted with a statistics professor - who agreed with my > position - to make sure I wasn't talking out of turn. > > In summary, I think that if t = 0/0 we should return 0. However, both R > and Matlab return NaN. To me this seems incorrect due to argument #2. > Josef also mentioned there were users of scipy.stats.stats that didn't > want NaN return values for some reason that I don't know offhand. I won't comment on the question of the urgency of this, but have some thoughts on the behavior. My R seems to throw an exception whenever the variance is zero (regardless of the mean difference), not return NaN: > t.test(c(1, 1, 1), c(2, 2, 2)) Error in t.test.default(c(1, 1, 1), c(2, 2, 2)) : data are essentially constant I find the arguments about "convergence" really uncompelling. If you do a t-test between two equal, constant vectors with a tiny amount of noise added, then the t statistic does not converge on anything -- it has a t distribution! > almost.ones <- function() { c(1, 1, 1) + 1e-10 * rnorm(3) } > replicate(5, t.test(almost.ones(), almost.ones())$statistic) t t t t t -3.1954354 0.7893332 0.1240638 -0.6432737 -0.6349522 Like any parametric test, the t-test only makes sense under some kind of (at least approximate) assumptions about the data generating process. When the sample variance is 0, then those assumptions are clearly violated, and it doesn't seem appropriate to me to start making up numbers according to some other rule that we hope might give some sort-of appropriate result ("In the face of ambiguity, refuse the temptation to guess."). So I actually like the R/Matlab option of throwing an exception or returning NaN. -N From jsseabold at gmail.com Wed Jun 6 16:50:30 2012 From: jsseabold at gmail.com (Skipper Seabold) Date: Wed, 6 Jun 2012 16:50:30 -0400 Subject: [SciPy-Dev] Resolving PR 235: t-statistic = 0/0 case In-Reply-To: <4FCFBB03.4000102@gmail.com> References: <4FCFBB03.4000102@gmail.com> Message-ID: On Wed, Jun 6, 2012 at 4:18 PM, Junkshops wrote: > Hi all, > > This mail references PR 235: https://github.com/scipy/scipy/pull/235 > > The PR adds a method for performing a t-test on 2 samples with unequal > or unknown variances and changes the return value when the t-statistic = > 0 / 0 to 0 from the previous code's return value, 1, for all t-test > functions. > > The latter change is a point of contention for the pull. I take the > position that t = 0/0 should return 0, while Josef, the primary > scipy.stats maintainer, believes that t = 0/0 should return one. > Normally I would try to resolve this directly with Josef, but > unfortunately I haven't heard from him in over two days, and the beta > release is scheduled for Saturday. As such, I'm writing the dev list to > ask what to do next. > > Josef's position, to the best of my ability to understand it, is that: > > J1) Adding a small amount of noise to data sets that have otherwise > equal means (e.g. [0,0,0,0], [0,0,0,1e-100]) results in a t-statistic of > 1. Thus, as the mean difference approaches zero, t -> 1. > J2) A data set with no mean difference and no variance is a zero > probability event. As such, returning t = 1 is reasonable, as therefore > p = 0.317-0.5 for a two tailed test depending on the degrees of freedom, > and hence for standard values of alpha the null hypothesis will not be > rejected, but the user gets some feedback that his data is suspect. > > I admit I don't completely understand the second argument. Hopefully > when Josef resurfaces he can correct my representation of his argument > if needed. > > My responses to these arguments are: > > J1) If you take the n-length vectors (x,...,x) and (x,...x,x+y) and > solve for t, t = 1 for any value of x and y. This is simply due to the > fact that the the mean difference is y/n and the pooled variance (e.g. > the denominator of the t-statistic) is also y/n. Thus this is merely a > special case and does not represent a limit as the mean difference > approaches zero, since even for [0,0,0,0], [0,0,0,1e100] t = 1. > > J2) Strictly speaking, if we're pulling independent random samples from > a normal distribution over the reals, say, any given set of samples has > zero probability since there are an infinite number of possible samples. > The sample [0,0,0] is just as probable as the sample [2.3, 7.4, 2.1]. In > a discrete distribution, in fact, the sample [0,0,0] is *more* likely > than any other sample. Also, we're now analyzing the legitimacy of the > user's data, which is not the job of the t-test. The t-test is expected > to take a data set(s) and provide a measure of the strength of the > evidence that the means are different. Is it not expected to comment on > the validity of the user's data. Practically speaking, it's a bit of a stretch to assume that the data generating process for [0,0,0] is (even approximately) normal, so I think it is appropriate for the test to do some sanity checking. The t-test itself is only valid given that the underlying data satisfies the assumptions, and I don't think a constant random variable meets the requirements. > > My arguments that 0 is the correct answer to return are: > > 1) if MD = mean difference and PV = pooled variance, then t = MD/PV. As > MD -> 0 for any PV > 0, t -> 0. However, if we set t = 0/0 = 1 and MD = > 0, as PV -> 0 we introduce a discontinuity at zero since t = 0 for any > value of PV except 0, where t = 1. This implies that for MD = PV = 0, t > = 1, but if the variance of the dataset is infinitesimally small but > !=0, then t = 0. To me, this makes no sense. > > 2) The t-test's purpose is to measure the strength of the evidence > against the null hypothesis that MD = 0. If MD = 0, as in the case we're > discussing, by definition there is no evidence that MD != 0. Therefore p > must = 1, and as a consequence t must = 0. > > I also consulted with a statistics professor - who agreed with my > position - to make sure I wasn't talking out of turn. > > In summary, I think that if t = 0/0 we should return 0. However, both R > and Matlab return NaN. To me this seems incorrect due to argument #2. > Josef also mentioned there were users of scipy.stats.stats that didn't > want NaN return values for some reason that I don't know offhand. > > How would the Scipy devs like to proceed? > Until I see any math or a reference, I think returning NaN is the path of least resistance. My $.02, Skipper From junkshops at gmail.com Wed Jun 6 17:18:01 2012 From: junkshops at gmail.com (Junkshops) Date: Wed, 06 Jun 2012 14:18:01 -0700 Subject: [SciPy-Dev] Resolving PR 235: t-statistic = 0/0 case In-Reply-To: References: <4FCFBB03.4000102@gmail.com> Message-ID: <4FCFC909.2000805@gmail.com> Hi Nathaniel, At the outset, I'll just say that if the consensus is that we should return NaN, I'll accept that. I'll still try and argue my case though. > My R seems to throw an exception whenever the variance is zero > (regardless of the mean difference), not return NaN: Sorry, yes, that's correct. > Like any parametric test, the t-test only makes sense under some kind > of (at least approximate) assumptions about the data generating > process. When the sample variance is 0, then those assumptions are > clearly violated, So this seems similar to argument J2, and I still don't understand it. Let's say we assume our population data is normally distributed and we take three samples from the population and get [1,1,1]. How does that prove our assumption is incorrect? It's certainly possible to pull the same number three times from a normal distribution. > and it doesn't seem appropriate to me to start > making up numbers according to some other rule that we hope might give > some sort-of appropriate result ("In the face of ambiguity, refuse the > temptation to guess."). So I actually like the R/Matlab option of > throwing an exception or returning NaN. Well, we're not making up numbers here - we absolutely know the means are the same. Hence p = 1 and t = 0. -g From jsseabold at gmail.com Wed Jun 6 17:35:58 2012 From: jsseabold at gmail.com (Skipper Seabold) Date: Wed, 6 Jun 2012 17:35:58 -0400 Subject: [SciPy-Dev] Resolving PR 235: t-statistic = 0/0 case In-Reply-To: <4FCFC909.2000805@gmail.com> References: <4FCFBB03.4000102@gmail.com> <4FCFC909.2000805@gmail.com> Message-ID: On Wed, Jun 6, 2012 at 5:18 PM, Junkshops wrote: > Hi Nathaniel, > > At the outset, I'll just say that if the consensus is that we should > return NaN, I'll accept that. I'll still try and argue my case though. > >> My R seems to throw an exception whenever the variance is zero >> (regardless of the mean difference), not return NaN: > Sorry, yes, that's correct. > >> Like any parametric test, the t-test only makes sense under some kind >> of (at least approximate) assumptions about the data generating >> process. When the sample variance is 0, then those assumptions are >> clearly violated, > So this seems similar to argument J2, and I still don't understand it. > Let's say we assume our population data is normally distributed and we > take three samples from the population and get [1,1,1]. How does that > prove our assumption is incorrect? It's certainly possible to pull the > same number three times from a normal distribution. > How do you justify that 3 empirical observations [1,1,1] come from a normal distribution? If you have enough data for the central limit theorem to come into play, and your variance is still 0, this is so unlikely that I think the consequences of *possibly* incorrectly returning NaN here would be small. If you're simulating data from a known distribution, take another draw... >> and it doesn't seem appropriate to me to start >> making up numbers according to some other rule that we hope might give >> some sort-of appropriate result ("In the face of ambiguity, refuse the >> temptation to guess."). So I actually like the R/Matlab option of >> throwing an exception or returning NaN. > > Well, we're not making up numbers here - we absolutely know the means > are the same. Hence p ?= 1 and t = 0. But what we don't know is if the test is even appropriate, so why not be cautious and return NaN. It's very easy for a user to make the decision that NaN implies p = 1, if that's what you want to have. This doesn't seem to be of all that much practical importance. In what situation do you expect this to really matter? Skipper From angel.yanguas at gmail.com Wed Jun 6 17:37:36 2012 From: angel.yanguas at gmail.com (Angel Yanguas-Gil) Date: Wed, 6 Jun 2012 16:37:36 -0500 Subject: [SciPy-Dev] Resolving PR 235: t-statistic = 0/0 case In-Reply-To: <4FCFC909.2000805@gmail.com> References: <4FCFBB03.4000102@gmail.com> <4FCFC909.2000805@gmail.com> Message-ID: Hi, Regarding the 0/0 case: if your variance is nominally zero (and you are using your sample to estimate the variance), then essentially you are saying that any infinitesimal deviation from that value is statistically significative. From a mathematical perspective, in the case of a normal distribution a variance that tends to zero while keeping the same area tends to a dirac delta, which is no longer a traditional mathematical function, but it is a generalized function or a distribution: http://en.wikipedia.org/wiki/Distribution_(mathematics). So I guess the question is more: would you like to raise a flag when you apply your function to two zero variance samples? On Wed, Jun 6, 2012 at 4:18 PM, Junkshops wrote: > Hi Nathaniel, > > At the outset, I'll just say that if the consensus is that we should > return NaN, I'll accept that. I'll still try and argue my case though. > >> My R seems to throw an exception whenever the variance is zero >> (regardless of the mean difference), not return NaN: > Sorry, yes, that's correct. > >> Like any parametric test, the t-test only makes sense under some kind >> of (at least approximate) assumptions about the data generating >> process. When the sample variance is 0, then those assumptions are >> clearly violated, > So this seems similar to argument J2, and I still don't understand it. > Let's say we assume our population data is normally distributed and we > take three samples from the population and get [1,1,1]. How does that > prove our assumption is incorrect? It's certainly possible to pull the > same number three times from a normal distribution. > >> and it doesn't seem appropriate to me to start >> making up numbers according to some other rule that we hope might give >> some sort-of appropriate result ("In the face of ambiguity, refuse the >> temptation to guess."). So I actually like the R/Matlab option of >> throwing an exception or returning NaN. > > Well, we're not making up numbers here - we absolutely know the means > are the same. Hence p ?= 1 and t = 0. > > -g > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From njs at pobox.com Wed Jun 6 17:37:45 2012 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 6 Jun 2012 22:37:45 +0100 Subject: [SciPy-Dev] Resolving PR 235: t-statistic = 0/0 case In-Reply-To: <4FCFC909.2000805@gmail.com> References: <4FCFBB03.4000102@gmail.com> <4FCFC909.2000805@gmail.com> Message-ID: On Wed, Jun 6, 2012 at 10:18 PM, Junkshops wrote: > Hi Nathaniel, > > At the outset, I'll just say that if the consensus is that we should > return NaN, I'll accept that. I'll still try and argue my case though. > >> My R seems to throw an exception whenever the variance is zero >> (regardless of the mean difference), not return NaN: > Sorry, yes, that's correct. > >> Like any parametric test, the t-test only makes sense under some kind >> of (at least approximate) assumptions about the data generating >> process. When the sample variance is 0, then those assumptions are >> clearly violated, > So this seems similar to argument J2, and I still don't understand it. > Let's say we assume our population data is normally distributed and we > take three samples from the population and get [1,1,1]. How does that > prove our assumption is incorrect? It's certainly possible to pull the > same number three times from a normal distribution. Well, no, it isn't possible really -- taking n IID samples from a normal distribution and getting exactly the same number twice is an event that has probability zero. (Note that this is a stronger statement than the one you were comparing it to, that getting any specific vector of samples has probability zero. There are infinitely many vectors which contain at least one duplicate; the set of all of them collectively has probability zero.) OTOH there are many other common processes which do produce such samples, and in practice those are where such samples come from. >> and it doesn't seem appropriate to me to start >> making up numbers according to some other rule that we hope might give >> some sort-of appropriate result ("In the face of ambiguity, refuse the >> temptation to guess."). So I actually like the R/Matlab option of >> throwing an exception or returning NaN. > > Well, we're not making up numbers here - we absolutely know the means > are the same. Hence p ?= 1 and t = 0. That is not what p=1 and t=0 mean in any kind of frequentist test. If the means are the same (the null hypothesis is true), then p should be a sample from a uniform distribution and t a sample from a t distribution. And even if we pretend that such data actually did come from a normal distribution (which is never going to be what actually happened in practice), then we actually still don't have any evidence that the means are the same. What we know is that the difference in the means is too small for us to measure, and also that the variance is too small for us to measure. But we don't know their relative sizes. It sounds like you're mixing up substantative and significant differences -- certainly no-one cares if their means differ by 10^-20, but if you somehow had an instrument that could produce measurements accurate to 10^-30, than a 10^-20 difference could easily be statistically significant. Whatever R does is usually based on the consensus of a bunch of really picky, opinionated, professional statisticians; I think they have a good reason for making the choice they did here. -N From junkshops at gmail.com Wed Jun 6 17:43:49 2012 From: junkshops at gmail.com (Junkshops) Date: Wed, 06 Jun 2012 14:43:49 -0700 Subject: [SciPy-Dev] Resolving PR 235: t-statistic = 0/0 case In-Reply-To: References: <4FCFBB03.4000102@gmail.com> Message-ID: <4FCFCF15.8090707@gmail.com> Hi Skipper, > Practically speaking, it's a bit of a stretch to assume that the data > generating process for [0,0,0] is (even approximately) normal, so I > think it is appropriate for the test to do some sanity checking. > > The t-test itself is only valid given that the underlying data > satisfies the assumptions, and I don't think a constant random > variable meets the requirements. This is pretty much identical to Nathaniel's objection, so perhaps you wouldn't mind responding to my argument there so we don't end up with separate threads on the same topic? I'll try to respond to multiple emails that cover similar ground at once from now on. > Until I see any math or a reference, I think returning NaN is the path > of least resistance. I have a nasty feeling this is going to look obnoxious, but I think the math is: Pr(MD = 0|MD = 0) = 1 where MD is the mean difference. That does look extremely obnoxious. Sorry. But that's the case here, to the best of my ability to tell. Basically, if the means are equal, it doesn't matter what the distribution assumption is - because the means are equal with 100% probability. But again, if everyone wants NaN I'll capitulate. -g From junkshops at gmail.com Wed Jun 6 20:50:49 2012 From: junkshops at gmail.com (Junkshops) Date: Wed, 06 Jun 2012 17:50:49 -0700 Subject: [SciPy-Dev] Resolving PR 235: t-statistic = 0/0 case In-Reply-To: References: <4FCFBB03.4000102@gmail.com> <4FCFC909.2000805@gmail.com> Message-ID: <4FCFFAE9.5070007@gmail.com> OK, I give! NaN it is. That being said: Skipper said: > This doesn't seem to be of all that much practical importance. In what > situation do you expect this to really matter? Eh, you're probably right. I tend to enjoy arguing back and forth (as long as it doesn't get heated) and sometimes pick pointless battles. Plus sometimes you learn a lot, and I'm not much of a statistician, so there's lots of opportunities for such. If you're pulling data from a discrete distribution it could happen though (unless I'm mistaken). Nathan said: > Well, no, it isn't possible really -- taking n IID samples from a > normal distribution and getting exactly the same number twice is an > event that has probability zero. Would you mind humoring me and explaining why this is true? It seems counter intuitive that getting the same sample twice from independent random draws is impossible. OK, so what next? Shall I make the changes and push again? Or should we wait a bit and see if anyone else weighs in? If a push is warranted the other issue is the style of the 4 t-tests (1 sample, paired, 2 sample equal variances, 2 sample unequal variances): A. 4 separate functions (as in the PR) B. 1 combined function, select test via keyword arg, keep old function stubs for backward compatibility C. Functions for 1 sample, paired, 2 sample with keyword selection of equal vs unequal variances. I don't have strong feelings either way, but I think C is a little weird - should be all or none IMO. We could also go with A for now and change to B after the release; I think it's more important that the functionality gets in than consolidation of functions. Cheers, Gavin From josef.pktd at gmail.com Wed Jun 6 20:51:21 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 6 Jun 2012 20:51:21 -0400 Subject: [SciPy-Dev] Resolving PR 235: t-statistic = 0/0 case In-Reply-To: <4FCFCF15.8090707@gmail.com> References: <4FCFBB03.4000102@gmail.com> <4FCFCF15.8090707@gmail.com> Message-ID: On Wed, Jun 6, 2012 at 5:43 PM, Junkshops wrote: > Hi Skipper, > >> Practically speaking, it's a bit of a stretch to assume that the data >> generating process for [0,0,0] is (even approximately) normal, so I >> think it is appropriate for the test to do some sanity checking. >> >> The t-test itself is only valid given that the underlying data >> satisfies the assumptions, and I don't think a constant random >> variable meets the requirements. > This is pretty much identical to Nathaniel's objection, so perhaps you > wouldn't mind responding to my argument there so we don't end up with > separate threads on the same topic? I'll try to respond to multiple > emails that cover similar ground at once from now on. > >> Until I see any math or a reference, I think returning NaN is the path >> of least resistance. > I have a nasty feeling this is going to look obnoxious, but I think the > math is: > > Pr(MD = 0|MD = 0) = 1 where MD is the mean difference. not at all, if this is a statement on the posterior distribution, the first MD is our hypothesis on the true parameter, the second MD is the MD of a single pair of samples. If you just have one sample (of 2 series) or draw from two populations, you cannot have a p-value of 1 (with nonzero probability) unless you sample the entire populations (or you have a dogmatic prior, or your model assumptions are wrong). (I hope this has enough qualifiers and brackets to be correct.:) > > That does look extremely obnoxious. Sorry. But that's the case here, to > the best of my ability to tell. Basically, if the means are equal, it > doesn't matter what the distribution assumption is - because the means > are equal with 100% probability. > > But again, if everyone wants NaN I'll capitulate. I'm pretty much coming to the NaN position too, at least it will make future discussions about it less likely. My intention before was to get a "reasonable" non-NaN value, and a t-statistic of 1 is obtained if only one value is slightly different. What I didn't realize 3 years ago is that finding a limit is futile as Nathaniel mentioned above, since for every variance greater than zero the pvalues have to be uniformly distributed if the test distribution is correct. So we cannot find a degenerate limit. (And I got my grey hairs for nothing in return.) (I don't belief that users that might have a 0/0 problem will have data with zero variance, my bet would be on discretized or correlated data without variation, and the ttest is shaky in this case. I played with some discretized examples but didn't get anywhere clean either.) Stop guessing and returning nans sounds very good, or alternatively, for practical purposes, let the user choose zoz=np.nan (default), and zoz=0 or zoz=1 is allowed. Josef > > -g > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From josef.pktd at gmail.com Wed Jun 6 21:10:00 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 6 Jun 2012 21:10:00 -0400 Subject: [SciPy-Dev] Resolving PR 235: t-statistic = 0/0 case In-Reply-To: <4FCFFAE9.5070007@gmail.com> References: <4FCFBB03.4000102@gmail.com> <4FCFC909.2000805@gmail.com> <4FCFFAE9.5070007@gmail.com> Message-ID: On Wed, Jun 6, 2012 at 8:50 PM, Junkshops wrote: > OK, I give! NaN it is. > > That being said: > > Skipper said: >> This doesn't seem to be of all that much practical importance. In what >> situation do you expect this to really matter? > Eh, you're probably right. I tend to enjoy arguing back and forth (as > long as it doesn't get heated) and sometimes pick pointless battles. > Plus sometimes you learn a lot, and I'm not much of a statistician, so > there's lots of opportunities for such. > > If you're pulling data from a discrete distribution it could happen > though (unless I'm mistaken). It can happen in the discrete distribution case, but in this case this doesn't have zero probability and the calculations can follow the standard theory (no 0/0) > > Nathan said: >> Well, no, it isn't possible really -- taking n IID samples from a >> normal distribution and getting exactly the same number twice is an >> event that has probability zero. > Would you mind humoring me and explaining why this is true? It seems > counter intuitive that getting the same sample twice from independent > random draws is impossible. You have a continuum of numbers (an uncountable infinite number of possibilities). Each point has zero probability of being selected. (But we have a density that points in a neighborhood dx are selected.) You can select a first point, but then the second point has to be the same up to an infinite number of decimals. The probability that all decimals are the same is zero. With floating point doubles, someone might be able to calculate what the tiny discrete probability is. > > OK, so what next? Shall I make the changes and push again? Or should we > wait a bit and see if anyone else weighs in? > > If a push is warranted the other issue is the style of the 4 t-tests (1 > sample, paired, 2 sample equal variances, 2 sample unequal variances): > > A. 4 separate functions (as in the PR) > B. 1 combined function, select test via keyword arg, keep old function > stubs for backward compatibility > C. Functions for 1 sample, paired, 2 sample with keyword selection of > equal vs unequal variances. > > I don't have strong feelings either way, but I think C is a little weird > - should be all or none IMO. We could also go with A for now and change > to B after the release; I think it's more important that the > functionality gets in than consolidation of functions. I thought C is the most obvious solution, paired versus unpaired are two different sampling schemes. Assuming common versus heterogenous variances is just a detail compared to that. Users might want to test for the variance heteroscedasticity assumption, but they will usually know whether the sampling scheme is paired (repeated) or independent(unpaired). Josef > > Cheers, Gavin > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From junkshops at gmail.com Thu Jun 7 00:29:09 2012 From: junkshops at gmail.com (Junkshops) Date: Wed, 06 Jun 2012 21:29:09 -0700 Subject: [SciPy-Dev] Resolving PR 235: t-statistic = 0/0 case In-Reply-To: References: <4FCFBB03.4000102@gmail.com> <4FCFC909.2000805@gmail.com> <4FCFFAE9.5070007@gmail.com> Message-ID: <4FD02E15.109@gmail.com> Merging the last two messages from Josef here... > If you just have one sample (of 2 series) or draw from two > populations, you cannot have a p-value of 1 (with nonzero probability) > unless you sample the entire populations (or you have a dogmatic > prior, or your model assumptions are wrong). (I hope this has enough > qualifiers and brackets to be correct.:) Then why >>> stats.ttest_ind([1e-100, 0, -1e100], [1e-100, 0, -1e100]) (0.0, 1.0) > You have a continuum of numbers (an uncountable infinite number of > possibilities). Each point has zero probability of being selected. > (But we have a density that points in a neighborhood dx are selected.) > You can select a first point, but then the second point has to be the > same up to an infinite number of decimals. The probability that all > decimals are the same is zero. Heh, that's where my mind went but that seemed too simple for some reason. I thought I was missing something. > >If you're pulling data from a discrete distribution it could happen > >though (unless I'm mistaken). > It can happen in the discrete distribution case, but in this case this > doesn't have zero probability and the calculations can follow the > standard theory (no 0/0) Sorry, I'm not following this. If we have samples from a discrete distribution (and in practice, any computerized data set is discrete since you have to map the reals to a finite representation), then it's possible, however unlikely, to have [0,0,0], [0,0,0] in the data set. Right? In which case we could wind up with a 0/0. > I thought C is the most obvious solution, paired versus unpaired are > two different sampling schemes. *shrug* you're the maintainer, and I don't care enough to argue. So: - I'll merge the two 2 sample t-test functions - add an uneq_var=False kw arg, setting to true will use the new code - add an zoz=np.nan kw arg and a check that it's np.nan, 0 or 1. Otherwise raise ValueError - update docs & tests. - push. Anything else? Sorry for any grey hairs I may have caused you, Gavin From lists at hilboll.de Thu Jun 7 05:24:42 2012 From: lists at hilboll.de (Andreas Hilboll) Date: Thu, 7 Jun 2012 11:24:42 +0200 Subject: [SciPy-Dev] Ubuntu PPA for NumPy / SciPy / ... Message-ID: Hi, I just noticed that there's a PPA for NumPy/SciPy on Launchpad: https://launchpad.net/~scipy/+archive/ppa However, it's painfully outdated. Does anyone know of its status? Is it 'official'? Are there any plans in revitalizing it, possibly with adding other projects from the "scipy universe"? Is there help needed? Many questions, but possibly quite easy to answer ... Cheers, Andreas. From njs at pobox.com Thu Jun 7 16:13:31 2012 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 7 Jun 2012 21:13:31 +0100 Subject: [SciPy-Dev] Resolving PR 235: t-statistic = 0/0 case In-Reply-To: <4FD02E15.109@gmail.com> References: <4FCFBB03.4000102@gmail.com> <4FCFC909.2000805@gmail.com> <4FCFFAE9.5070007@gmail.com> <4FD02E15.109@gmail.com> Message-ID: On Thu, Jun 7, 2012 at 5:29 AM, Junkshops wrote: > - I'll merge the two 2 sample t-test functions > - add an uneq_var=False kw arg, setting to true will use the new code equal_var would be a better name, to avoid the double-negative. Would it be possible/desireable to make equal_var=False the default? Obviously this would require a deprecation period, but as semantic changes go it's relatively low risk -- anyone who misses the warnings etc. would just find one day that their t tests were producing more conservative/realistic values. (R defaults to doing the unequal variances test, and I have actually seen this fact used in their advocacy, as evidence for their branding as the tool for people who care about statistical rigor and soundness.) > - add an zoz=np.nan kw arg and a check that it's np.nan, 0 or 1. > Otherwise raise ValueError Let's please not add this "zoz=" feature. Adding features has a real cost (in terms of testing, writing docs, maintenance, and most importantly, the total time spent by all users reading about this pointless thing in the docs and being distracted by it). It's only benefit would be to smooth over this debate on the mailing list; I can't believe that any real user will actually care about this, ever. (And if such a user ever does show up, then adding a feature is an easy backwards-compatible change; removing it isn't.) > - update docs & tests. > - push. -n From cournape at gmail.com Fri Jun 8 02:07:37 2012 From: cournape at gmail.com (David Cournapeau) Date: Fri, 8 Jun 2012 14:07:37 +0800 Subject: [SciPy-Dev] Ubuntu PPA for NumPy / SciPy / ... In-Reply-To: References: Message-ID: On Thu, Jun 7, 2012 at 5:24 PM, Andreas Hilboll wrote: > Hi, > > I just noticed that there's a PPA for NumPy/SciPy on Launchpad: > > https://launchpad.net/~scipy/+archive/ppa > > However, it's painfully outdated. Does anyone know of its status? Is it > 'official'? Are there any plans in revitalizing it, possibly with adding > other projects from the "scipy universe"? Is there help needed? > > Many questions, but possibly quite easy to answer ... > > I set up this PPA a long time ago. I just don't have time to maintain it at that point, but would be happy to give someone the keys to make it up to date. David -------------- next part -------------- An HTML attachment was scrubbed... URL: From stuart at mumford.me.uk Fri Jun 8 09:17:25 2012 From: stuart at mumford.me.uk (Stuart Mumford) Date: Fri, 8 Jun 2012 14:17:25 +0100 Subject: [SciPy-Dev] Procedure for new code submission (pyHHT) Message-ID: Hello, I am working on a EMD / HHT code for some analysis I am doing as part of my PhD. (It is currently here: https://github.com/Cadair/pyhht (kicked off from jaidevd's inital work)) I was wondering what the general procedure for getting something like this included in SciPy is. I know it requires a lot more work to even be considered but I would like to focus what I am doing with that end in mind. The Web here: http://scipy.github.com/dev-zone.html says that initially you should write a SciKit, is this still the way to go or is it fine just to create a repo with good quality documentation etc? Also is a HHT toolbox something that should be integrated into scipy.signal or (at risk of restarting an amusing debate) should it go into scikit.signal?? Stuart -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Jun 8 09:50:44 2012 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 8 Jun 2012 15:50:44 +0200 Subject: [SciPy-Dev] Procedure for new code submission (pyHHT) In-Reply-To: References: Message-ID: On Fri, Jun 8, 2012 at 3:17 PM, Stuart Mumford wrote: > Hello, > > I am working on a EMD / HHT code for some analysis I am doing as part of my > PhD. (It is currently here: https://github.com/Cadair/pyhht (kicked off from > jaidevd's inital work)) > > I was wondering what the general procedure for getting something like this > included in SciPy is. I know it requires a lot more work to even be > considered but I would like to focus what I am doing with that end in mind. > > The Web here: http://scipy.github.com/dev-zone.html says that initially you > should write a SciKit, is this still the way to go or is it fine just to > create a repo with good quality documentation etc? It is fine to create your project with whatever structure you like. > Also is a HHT toolbox something that should be integrated into scipy.signal > or (at risk of restarting an amusing debate) should it go into > scikit.signal?? The Hilbert-Huang Transform is encumbered by a patent in the US. I recommend keeping it contained to a single package and not integrate it into another open source library. -- Robert Kern From stuart at mumford.me.uk Fri Jun 8 10:09:59 2012 From: stuart at mumford.me.uk (Stuart Mumford) Date: Fri, 8 Jun 2012 15:09:59 +0100 Subject: [SciPy-Dev] Procedure for new code submission (pyHHT) In-Reply-To: References: Message-ID: Hello, > It is fine to create your project with whatever structure you like. Fair enough!! > The Hilbert-Huang Transform is encumbered by a patent in the US. Oh great! > I recommend keeping it contained to a single package and not integrate it into another open source library. Seems sensible, then people can take it or leave it as required! Thanks for your help. Moving away from HHT, what are the documentation requirements and testing requirements for submissions in general? The test classes as written like https://github.com/numpy/numpy/blob/master/doc/TESTS.rst.txttest the execution of the code and the results? So in writing a test class do you need to develop a test set of data and check that the result of the test is what is expected? Then more over how do you justifiy what is expected is correct. For a certian function (like HHT or Wavelet) do you have to check the transform against a simple generated signal that makes it clear the output is correct or a test 'real life' dataset? Also for dicumentation am I right in thinking that API reference is created from docstrings and a tutorial style documentation should be made seperatly? Thanks Stuart -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.isaac at gmail.com Fri Jun 8 10:10:54 2012 From: alan.isaac at gmail.com (Alan G Isaac) Date: Fri, 08 Jun 2012 10:10:54 -0400 Subject: [SciPy-Dev] Procedure for new code submission (pyHHT) In-Reply-To: References: Message-ID: <4FD207EE.6000603@gmail.com> On 6/8/2012 9:50 AM, Robert Kern wrote: > The Hilbert-Huang Transform is encumbered by a patent in the US. Despite being developed with taxpayer dollars. And then Nasa bragged about "making the technology available" through its patent auction. What a bad joke on the public. Alan Isaac From stuart at mumford.me.uk Fri Jun 8 10:20:37 2012 From: stuart at mumford.me.uk (Stuart Mumford) Date: Fri, 8 Jun 2012 15:20:37 +0100 Subject: [SciPy-Dev] Procedure for new code submission (pyHHT) In-Reply-To: <4FD207EE.6000603@gmail.com> References: <4FD207EE.6000603@gmail.com> Message-ID: > Despite being developed with taxpayer dollars. > And then Nasa bragged about "making the technology available" > through its patent auction. What a bad joke on the public. > Indeed! What are the restiructions on use? Preseumably you can use it for acedemic purposes etc.? Stuart -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.isaac at gmail.com Fri Jun 8 10:40:55 2012 From: alan.isaac at gmail.com (Alan G Isaac) Date: Fri, 08 Jun 2012 10:40:55 -0400 Subject: [SciPy-Dev] Procedure for new code submission (pyHHT) In-Reply-To: References: <4FD207EE.6000603@gmail.com> Message-ID: <4FD20EF7.4000901@gmail.com> On 6/8/2012 10:20 AM, Stuart Mumford wrote: > What are the restiructions on use? Preseumably you can use it for > acedemic purposes etc.? Welcome to the ongoing US effort to stifle knowledge creation: http://www.invention-protection.com/ip/publications/docs/Can_Academic_Experimental_Use_Survive.html Note that in the US we have a single constitutional justification for patents and copyright: that they **promote progress**. Our patent system has been in serious violation of this since Warren Burger wrote Diamond vs. Chakrabarty: http://en.wikipedia.org/wiki/Diamond_v._Chakrabarty One reason that open source software is so absolutely critical in this environment is that it provides a well documented history of "prior art". Alan Isaac From pav at iki.fi Fri Jun 8 11:05:35 2012 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 8 Jun 2012 15:05:35 +0000 (UTC) Subject: [SciPy-Dev] Procedure for new code submission (pyHHT) References: Message-ID: Stuart Mumford mumford.me.uk> writes: > Hello,I am working on a EMD / HHT code for some analysis > I am doing as part of my PhD. (It is currently here: > https://github.com/Cadair/pyhht (kicked off from jaidevd's inital work)) > I was wondering what the general procedure for getting something > like this included in SciPy is. I know it requires a lot more work > to even be considered but I would like to focus what I am doing > with that end in mind. If the code is something that is useful across scientific fields, then IMO it can be bundled with Scipy. In this case, we have the misfortune of knowing beforehand that the algorithm is patented in US, and although as ridiculous it is, this makes the code poisonous for many US-based users. So, I'd say that this cannot be included in Scipy which does have US-based users and redistributors... > The Web here: > http://scipy.github.com/dev-zone.html says that initially you should > write a SciKit, is this still the way to go or is it fine just to > create a repo with good quality documentation etc? The information on that page is a bit outdated. This one is more up to date: https://github.com/scipy/scipy/blob/master/HACKING.rst.txt Note that being a "SciKit" is mostly just a matter of branding --- you make a separate Python package from your code as usual (which you seem to be working towards), call it "PyHHT Scikit", and add it to the "Topical software" list at scipy.org We earlier recommended naming the package as "scikits.XXX" but this turned out to bring in some technical problems in the long run, so that part is optional. > Also is a HHT toolbox something that should be integrated > into scipy.signal or (at risk of restarting an amusing debate) > should it go into scikit.signal?? Without the patent issues, scipy.signal would be the correct location. I'm not sure if scikit.signal is dead or just sleeping. -- Pauli Virtanen From deshpande.jaidev at gmail.com Fri Jun 8 15:39:34 2012 From: deshpande.jaidev at gmail.com (Jaidev Deshpande) Date: Sat, 9 Jun 2012 01:09:34 +0530 Subject: [SciPy-Dev] Procedure for new code submission (pyHHT) In-Reply-To: References: Message-ID: Hi, I agree that HHT should remain isolated for the moment. When it matures, the community can decide where it should reside. As for the IPR status of the HHT, I want to share the reply Dr Norden Huang gave when I asked him about it. "The NASA patent covers only the US. You are free to do whatever outside US territory as long as you do not make money in the US. Go ahead with your project, and good luck." Now, I would assume that this exempts research based on the HHT from the IPR regulations, even in the US. A lot of people have been doing it for almost a decade now, and many were Americans. And I don't think anyone pays or gets paid for using the HHT algorithms for academic purposes. And anyhow, there are many implementations of HHT on matlabcentral.com, there are freely available R packages too. Also, scipy.signal itself has a long way to go before we can include something like HHT. Regards, On Fri, Jun 8, 2012 at 6:47 PM, Stuart Mumford wrote: > Hello, > > I am working on a EMD / HHT code for some analysis I am doing as part of my > PhD. (It is currently here: https://github.com/Cadair/pyhht (kicked off from > jaidevd's inital work)) > > I was wondering what the general procedure for getting something like this > included in SciPy is. I know it requires a lot more work to even be > considered but I would like to focus what I am doing with that end in mind. > > The Web here: http://scipy.github.com/dev-zone.html says that initially you > should write a SciKit, is this still the way to go or is it fine just to > create a repo with good quality documentation etc? > > Also is a HHT toolbox something that should be integrated into scipy.signal > or (at risk of restarting an amusing debate) should it go into > scikit.signal?? > > Stuart > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > From robert.kern at gmail.com Fri Jun 8 16:11:31 2012 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 8 Jun 2012 22:11:31 +0200 Subject: [SciPy-Dev] Procedure for new code submission (pyHHT) In-Reply-To: References: Message-ID: On Fri, Jun 8, 2012 at 9:39 PM, Jaidev Deshpande wrote: > Hi, > > I agree that HHT should remain isolated for the moment. When it > matures, the community can decide where it should reside. > > As for the IPR status of the HHT, I want to share the reply Dr Norden > Huang gave when I asked him about it. > > "The NASA patent covers only the US. ?You are free to do whatever outside US > territory as long as you do not make money in the US. ?Go ahead with your > project, and good luck." Dr Huang does not own the patent anymore. Ocean Tomo Federal Services LLC does, unless if they've sold it off. I do not believe he can give you a license to the patent. His opinion about what can be done without acquiring a license to the patent is in conflict with US law. > Now, I would assume that this exempts research based on the HHT from > the IPR regulations, even in the US. A lot of people have been doing > it for almost a decade now, and many were Americans. And I don't think > anyone pays or gets paid for using the HHT algorithms for academic > purposes. Sorry, I'm afraid not. I believe the current law of the land was laid out in Madey v. Duke which interprets "commercial use" *extremely* broadly. Universities charge tuition and take in research grant money, and using patented inventions without paying royalties would make a university's program more attractive, both from students paying tuition and granting agencies, than otherwise. That was enough of a commercial nexus for that judge. It also draws on previous cases that limit the experimental use defense to cases where the use is "solely for amusement, to satisfy idle curiosity, or for strictly philosophical inquiry". University research programs are anything but idle curiosity, I'm afraid. The Supreme Court declined to review the case, leaving it the controlling precedent nationwide, and it has not been overruled to my knowledge. Here is the decision, edited down to the juiciest bits: http://cyber.law.harvard.edu/people/tfisher/2002Madeyedit.html > And anyhow, there are many implementations of HHT on > matlabcentral.com, there are freely available R packages too. Sure. You can write open source software that includes patented inventions. Users of that software have the responsibility to independently acquire a license for the patent. The fact that academic users have long ignored that responsibility just shows their ignorance of the law (and I don't entirely blame them since it's complicated); it is not a reflection of the law itself. -- Robert Kern From deshpande.jaidev at gmail.com Fri Jun 8 17:09:00 2012 From: deshpande.jaidev at gmail.com (Jaidev Deshpande) Date: Sat, 9 Jun 2012 02:39:00 +0530 Subject: [SciPy-Dev] Procedure for new code submission (pyHHT) In-Reply-To: References: Message-ID: > Universities charge tuition and take in research grant money, > and using patented inventions without paying royalties would make a > university's program more attractive, both from students paying > tuition and granting agencies, than otherwise. That was enough of a > commercial nexus for that judge. It also draws on previous cases that > limit the experimental use defense to cases where the use is "solely > for amusement, to satisfy idle curiosity, or for strictly > philosophical inquiry". University research programs are anything but > idle curiosity, I'm afraid. The Supreme Court declined to review the > case, leaving it the controlling precedent nationwide, and it has not > been overruled to my knowledge. > > Here is the decision, edited down to the juiciest bits: > > ?http://cyber.law.harvard.edu/people/tfisher/2002Madeyedit.html > Wow.. > Sure. You can write open source software that includes patented > inventions. Users of that software have the responsibility to > independently acquire a license for the patent. The fact that academic > users have long ignored that responsibility just shows their ignorance > of the law (and I don't entirely blame them since it's complicated); > it is not a reflection of the law itself. So then, what do you think we should do? From robert.kern at gmail.com Fri Jun 8 17:44:59 2012 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 8 Jun 2012 23:44:59 +0200 Subject: [SciPy-Dev] Procedure for new code submission (pyHHT) In-Reply-To: References: Message-ID: On Fri, Jun 8, 2012 at 11:09 PM, Jaidev Deshpande wrote: >> Universities charge tuition and take in research grant money, >> and using patented inventions without paying royalties would make a >> university's program more attractive, both from students paying >> tuition and granting agencies, than otherwise. That was enough of a >> commercial nexus for that judge. It also draws on previous cases that >> limit the experimental use defense to cases where the use is "solely >> for amusement, to satisfy idle curiosity, or for strictly >> philosophical inquiry". University research programs are anything but >> idle curiosity, I'm afraid. The Supreme Court declined to review the >> case, leaving it the controlling precedent nationwide, and it has not >> been overruled to my knowledge. >> >> Here is the decision, edited down to the juiciest bits: >> >> ?http://cyber.law.harvard.edu/people/tfisher/2002Madeyedit.html >> > > Wow.. > >> Sure. You can write open source software that includes patented >> inventions. Users of that software have the responsibility to >> independently acquire a license for the patent. The fact that academic >> users have long ignored that responsibility just shows their ignorance >> of the law (and I don't entirely blame them since it's complicated); >> it is not a reflection of the law itself. > > So then, what do you think we should do? I am not a lawyer, and this is not legal advice. If you're in the US, you will have to decide for yourself whether or not to continue developing the software. It's up to you, and possibly your institution's legal department. If you're not in the US, you're probably (*probably*) fine to continue to develop and publish your software. But please do clearly document that the algorithms are patented in the US. Ideally, you would include contact information for the patent owner; Dr Huang might have the correct contact information. It might help to include a brief warning like mine above (but in your own words, please) that most "research uses" people think of are not actually permitted without a patent license under US law. I don't know if there is a good web page describing the problem, but the EFF may have one. Maybe there's even a petition that you can link to. But that's purely up to you. In any case, it's best to keep the HHT code in its own package. -- Robert Kern From stuart at mumford.me.uk Fri Jun 8 17:46:31 2012 From: stuart at mumford.me.uk (Stuart Mumford) Date: Fri, 8 Jun 2012 22:46:31 +0100 Subject: [SciPy-Dev] Procedure for new code submission (pyHHT) In-Reply-To: References: Message-ID: Wow you really do have to love US (and international) patent law. As for what I think we should do? Write a decent HHT implementation, put it online and be careful what we use it for. Stuart -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri Jun 8 18:11:07 2012 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 09 Jun 2012 00:11:07 +0200 Subject: [SciPy-Dev] Procedure for new code submission (pyHHT) In-Reply-To: References: Message-ID: Hi, 08.06.2012 21:39, Jaidev Deshpande kirjoitti: [clip] > Also, scipy.signal itself has a long way to go before we can include > something like HHT. Out of curiosity, what's missing that would prevent inclusion of HHT? I'm not into this field myself, so I can't tell (and I admit that I don't even know what is in scipy.signal currently :) Cheers, From npkuin at gmail.com Sat Jun 9 02:47:38 2012 From: npkuin at gmail.com (Paul Kuin) Date: Sat, 9 Jun 2012 07:47:38 +0100 Subject: [SciPy-Dev] Procedure for new code submission (pyHHT) In-Reply-To: References: Message-ID: It was my understanding that patents developed under research contracts for NASA were encumbered with restrictions that allow the free use for teaching and (federally funded) research, but that may have changed in recent years. Perhaps it would be good to check if such a restriction is on the patent. A question to ask NASA?, or was there something passed into law that applies to all software developed with federal funding? Been away from the US for a while now, so it might have changed. But clearly, for any other use, the user will have to get a licence (in the US). Are there export restrictions ? -- Paul Kuin On Fri, Jun 8, 2012 at 9:11 PM, Robert Kern wrote: > On Fri, Jun 8, 2012 at 9:39 PM, Jaidev Deshpande > wrote: >> Hi, >> >> I agree that HHT should remain isolated for the moment. When it >> matures, the community can decide where it should reside. >> >> As for the IPR status of the HHT, I want to share the reply Dr Norden >> Huang gave when I asked him about it. >> >> "The NASA patent covers only the US. ?You are free to do whatever outside US >> territory as long as you do not make money in the US. ?Go ahead with your >> project, and good luck." > > Dr Huang does not own the patent anymore. Ocean Tomo Federal Services > LLC does, unless if they've sold it off. I do not believe he can give > you a license to the patent. His opinion about what can be done > without acquiring a license to the patent is in conflict with US law. > >> Now, I would assume that this exempts research based on the HHT from >> the IPR regulations, even in the US. A lot of people have been doing >> it for almost a decade now, and many were Americans. And I don't think >> anyone pays or gets paid for using the HHT algorithms for academic >> purposes. > > Sorry, I'm afraid not. I believe the current law of the land was laid > out in Madey v. Duke which interprets "commercial use" *extremely* > broadly. Universities charge tuition and take in research grant money, > and using patented inventions without paying royalties would make a > university's program more attractive, both from students paying > tuition and granting agencies, than otherwise. That was enough of a > commercial nexus for that judge. It also draws on previous cases that > limit the experimental use defense to cases where the use is "solely > for amusement, to satisfy idle curiosity, or for strictly > philosophical inquiry". University research programs are anything but > idle curiosity, I'm afraid. The Supreme Court declined to review the > case, leaving it the controlling precedent nationwide, and it has not > been overruled to my knowledge. > > Here is the decision, edited down to the juiciest bits: > > ?http://cyber.law.harvard.edu/people/tfisher/2002Madeyedit.html > >> And anyhow, there are many implementations of HHT on >> matlabcentral.com, there are freely available R packages too. > > Sure. You can write open source software that includes patented > inventions. Users of that software have the responsibility to > independently acquire a license for the patent. The fact that academic > users have long ignored that responsibility just shows their ignorance > of the law (and I don't entirely blame them since it's complicated); > it is not a reflection of the law itself. > > -- > Robert Kern > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From robert.kern at gmail.com Sat Jun 9 03:10:39 2012 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 9 Jun 2012 09:10:39 +0200 Subject: [SciPy-Dev] Procedure for new code submission (pyHHT) In-Reply-To: References: Message-ID: On Sat, Jun 9, 2012 at 8:47 AM, Paul Kuin wrote: > It was my understanding that patents developed under research > contracts for NASA were encumbered with restrictions that allow the > free use for teaching and (federally funded) research, but that may > have changed in recent years. Perhaps it would be good to check if > such a restriction is on the patent. A question to ask NASA?, or was > there something passed into law that applies to all software developed > with federal funding? ?Been away from the US for a while now, so it > might have changed. ?But clearly, for any other use, the user will > have to get a licence (in the US). ?Are there export restrictions ? The people to ask would be the current owners of the patent, Ocean Tomo Federal Services, LLC. -- Robert Kern From ralf.gommers at googlemail.com Sat Jun 9 07:04:29 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 9 Jun 2012 13:04:29 +0200 Subject: [SciPy-Dev] Resolving PR 235: t-statistic = 0/0 case In-Reply-To: References: <4FCFBB03.4000102@gmail.com> <4FCFC909.2000805@gmail.com> <4FCFFAE9.5070007@gmail.com> <4FD02E15.109@gmail.com> Message-ID: On Thu, Jun 7, 2012 at 10:13 PM, Nathaniel Smith wrote: > On Thu, Jun 7, 2012 at 5:29 AM, Junkshops wrote: > > - I'll merge the two 2 sample t-test functions > > - add an uneq_var=False kw arg, setting to true will use the new code > > equal_var would be a better name, to avoid the double-negative. > > Would it be possible/desireable to make equal_var=False the default? > Obviously this would require a deprecation period, but as semantic > changes go it's relatively low risk -- anyone who misses the warnings > etc. would just find one day that their t tests were producing more > conservative/realistic values. > I'm not in favor of adding a deprecation warning for this. It's a minor thing, and warnings are annoying - it does require the user to go and figure out what changed. My preference would be to merge the current PR as is, and add a new function that combines all four t-tests with an interface similar to R. There the new default can be equal_var=False without annoying anyone. > > (R defaults to doing the unequal variances test, and I have actually > seen this fact used in their advocacy, as evidence for their branding > as the tool for people who care about statistical rigor and > soundness.) > > > - add an zoz=np.nan kw arg and a check that it's np.nan, 0 or 1. > > Otherwise raise ValueError > > Let's please not add this "zoz=" feature. Adding features has a real > cost (in terms of testing, writing docs, maintenance, and most > importantly, the total time spent by all users reading about this > pointless thing in the docs and being distracted by it). It's only > benefit would be to smooth over this debate on the mailing list; I > can't believe that any real user will actually care about this, ever. > Agreed. And +1 for 0/0 --> NaN. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.kremer.dk at gmail.com Sat Jun 9 08:14:16 2012 From: david.kremer.dk at gmail.com (David Kremer) Date: Sat, 09 Jun 2012 14:14:16 +0200 Subject: [SciPy-Dev] [Probe] How amongst you are pro devs working for scipy ? Message-ID: <4FD33E18.9010909@gmail.com> Hello everyone, First I would like to tell you that I am a somehow satisfied scipy user, even if I feel it lacks official recognition. First, my question is concerning the scipy developers. I would like to know, amongst you, who is working on scipy as paid-job, or only on its spare time. Basically it makes the developer community divides in two parts : the first is the community of hobbyists, the second the community of profesional developers. Knowing grossly the proportion of pro and hobbyist, I would have a second question, related to the first : who uses scipy in business related project (I mean, no academic, that is to say, any work that is not included into a public research project). I feel that scipy is a very big project, and must have a lot of maintainers, and I am wondering if this project is actually used by real people out there, apart from PhD student (including myself) and researchers. I know that scipy, in its field, may encounter two great concurrents (for the least), namely matlab and octave, even if the goal of scipy may be slightly different. Well, any developer or member of the community that could answer this question or give some hints on the related topic would be very nice. Thank you. David Kremer PhD student Angers, France From jason-sage at creativetrax.com Sat Jun 9 08:56:18 2012 From: jason-sage at creativetrax.com (Jason Grout) Date: Sat, 09 Jun 2012 07:56:18 -0500 Subject: [SciPy-Dev] [Probe] How amongst you are pro devs working for scipy ? In-Reply-To: <4FD33E18.9010909@gmail.com> References: <4FD33E18.9010909@gmail.com> Message-ID: <4FD347F2.2090105@creativetrax.com> On 6/9/12 7:14 AM, David Kremer wrote: > Basically it makes the developer community divides in two > parts : the first is the community of hobbyists, the second the > community of profesional developers. Does it? For example, a college professor might work on scipy as part of his job in order to add needed functionality. Do they count as a professional developer, even if they don't code for a living? A person that writes code for a living might work on scipy in his spare time. Does that make them not a professional developer of scipy? A person might be unemployed, and so work a lot on scipy, as much as a regular job. Does that make them a "professional" or a "hobbyist"? Thanks, Jason From ralf.gommers at googlemail.com Sat Jun 9 09:46:46 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 9 Jun 2012 15:46:46 +0200 Subject: [SciPy-Dev] branching 0.11.x Message-ID: Hi all, We're almost ready to branch 0.11.x and tag the first beta. Here's some things that are still TODO. Necessary to finish before the first beta: - Fix http://projects.scipy.org/scipy/ticket/1599 (work around segfaults due to single-precision shift-invert mode in Arpack) - https://github.com/scipy/scipy/pull/217 (Finishing optimize.minimize() interface changes) Can be fixed after the first beta: - http://projects.scipy.org/scipy/ticket/1669 (sparse.csgraph test failures) - https://github.com/scipy/scipy/pull/198 (fitting discrete distributions issue) Things that would be good to include, but will be postponed unless they're merged in time: - https://github.com/scipy/scipy/pull/236 (use memoization in Minpack routines) - https://github.com/scipy/scipy/pull/235 (t-test with unequal variances) - https://github.com/scipy/scipy/pull/173 (binned statistics) - any other PRs / patches that are ready If we can get the necessary items resolved this weekend, I plan to create the branch tomorrow evening (at or after 9pm GMT). Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.kremer.dk at gmail.com Sat Jun 9 10:05:41 2012 From: david.kremer.dk at gmail.com (David Kremer) Date: Sat, 09 Jun 2012 16:05:41 +0200 Subject: [SciPy-Dev] [Probe] How amongst you are pro devs working for scipy ? In-Reply-To: <4FD347F2.2090105@creativetrax.com> References: <4FD33E18.9010909@gmail.com> <4FD347F2.2090105@creativetrax.com> Message-ID: <4FD35835.70801@gmail.com> On 09/06/2012 14:56, Jason Grout wrote: > Does it? For example, a college professor might work on scipy as part > of his job in order to add needed functionality. Do they count as a > professional developer, even if they don't code for a living? > > A person that writes code for a living might work on scipy in his spare > time. Does that make them not a professional developer of scipy? > > A person might be unemployed, and so work a lot on scipy, as much as a > regular job. Does that make them a "professional" or a "hobbyist"? I don't mean to oppose professional developer and hobbyist, actually my question would rather related to a curiosity about where come from the scipy base code ? I know that scientists are working on it, but honestly, it seems to me such a great piece of work that I would be astonished if absolutely no company was involved directly in the scipy code. From cournape at gmail.com Sat Jun 9 10:38:38 2012 From: cournape at gmail.com (David Cournapeau) Date: Sat, 9 Jun 2012 22:38:38 +0800 Subject: [SciPy-Dev] [Probe] How amongst you are pro devs working for scipy ? In-Reply-To: <4FD35835.70801@gmail.com> References: <4FD33E18.9010909@gmail.com> <4FD347F2.2090105@creativetrax.com> <4FD35835.70801@gmail.com> Message-ID: On Sat, Jun 9, 2012 at 10:05 PM, David Kremer wrote: > On 09/06/2012 14:56, Jason Grout wrote: > > Does it? For example, a college professor might work on scipy as part > > of his job in order to add needed functionality. Do they count as a > > professional developer, even if they don't code for a living? > > > > A person that writes code for a living might work on scipy in his spare > > time. Does that make them not a professional developer of scipy? > > > > A person might be unemployed, and so work a lot on scipy, as much as a > > regular job. Does that make them a "professional" or a "hobbyist"? > > I don't mean to oppose professional developer and hobbyist, actually my > question would rather related to a curiosity about where come from the > scipy base code ? I know that scientists are working on it, but > honestly, it seems to me such a great piece of work that I would be > astonished if absolutely no company was involved directly in the scipy > code. > Most of scipy code has been contributed by volunteers. Some paid programmers often contribute bug fixes or new features. Scipy is used by private companies as well (geophysics, finance, etc?) David -------------- next part -------------- An HTML attachment was scrubbed... URL: From npkuin at gmail.com Sat Jun 9 11:26:27 2012 From: npkuin at gmail.com (Paul Kuin) Date: Sat, 9 Jun 2012 16:26:27 +0100 Subject: [SciPy-Dev] [Probe] How amongst you are pro devs working for scipy ? In-Reply-To: References: <4FD33E18.9010909@gmail.com> <4FD347F2.2090105@creativetrax.com> <4FD35835.70801@gmail.com> Message-ID: On Sat, Jun 9, 2012 at 3:38 PM, David Cournapeau wrote: > > > On Sat, Jun 9, 2012 at 10:05 PM, David Kremer > wrote: >> >> On 09/06/2012 14:56, Jason Grout wrote: >> > Does it? ? For example, a college professor might work on scipy as part >> > of his job in order to add needed functionality. ?Do they count as a >> > professional developer, even if they don't code for a living? >> > >> > A person that writes code for a living might work on scipy in his spare >> > time. ?Does that make them not a professional developer of scipy? >> > >> > A person might be unemployed, and so work a lot on scipy, as much as a >> > regular job. ?Does that make them a "professional" or a "hobbyist"? >> >> I don't mean to oppose professional developer and hobbyist, actually my >> question would rather related to a curiosity about where come from the >> scipy base code ? I know that scientists are working on it, but >> honestly, it seems to me such a great piece of work that I would be >> astonished if absolutely no company was involved directly in the scipy >> code. > > > Most of scipy code has been contributed by volunteers. Some paid programmers > often contribute bug fixes or new features. > > Scipy is used by private companies as well (geophysics, finance, etc?) > There is a rich heritage of software in different languages that was developed over the years, some algorithms date back from before computers were machines (yes there once were people having a job as a computer). I think of scipy mainly as a handy collection of the most frequently used tools, and do follow this list mainly to learn about new approaches and changes since I use Scipy. It does not make sense to divide up who did what or if this was done for free or paid for by someone. That is not how free/open software development works. It is more 'organic' than that, the best stuff eventually gets in, and gets improved upon, or gets discarded. It is like asking about how London came about. Yes there is some organization, but it is like laying the roads down, not the whole of it. Paul From ralf.gommers at googlemail.com Sun Jun 10 06:33:27 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 10 Jun 2012 12:33:27 +0200 Subject: [SciPy-Dev] Resolving PR 235: t-statistic = 0/0 case In-Reply-To: References: <4FCFBB03.4000102@gmail.com> <4FCFC909.2000805@gmail.com> <4FCFFAE9.5070007@gmail.com> <4FD02E15.109@gmail.com> Message-ID: On Sat, Jun 9, 2012 at 1:04 PM, Ralf Gommers wrote: > > > On Thu, Jun 7, 2012 at 10:13 PM, Nathaniel Smith wrote: > >> On Thu, Jun 7, 2012 at 5:29 AM, Junkshops wrote: >> > - I'll merge the two 2 sample t-test functions >> > - add an uneq_var=False kw arg, setting to true will use the new code >> >> equal_var would be a better name, to avoid the double-negative. >> >> Would it be possible/desireable to make equal_var=False the default? >> Obviously this would require a deprecation period, but as semantic >> changes go it's relatively low risk -- anyone who misses the warnings >> etc. would just find one day that their t tests were producing more >> conservative/realistic values. >> > > I'm not in favor of adding a deprecation warning for this. It's a minor > thing, and warnings are annoying - it does require the user to go and > figure out what changed. My preference would be to merge the current PR as > is, and add a new function that combines all four t-tests with an interface > similar to R. There the new default can be equal_var=False without annoying > anyone. > > >> >> (R defaults to doing the unequal variances test, and I have actually >> seen this fact used in their advocacy, as evidence for their branding >> as the tool for people who care about statistical rigor and >> soundness.) >> >> > - add an zoz=np.nan kw arg and a check that it's np.nan, 0 or 1. >> > Otherwise raise ValueError >> >> Let's please not add this "zoz=" feature. Adding features has a real >> cost (in terms of testing, writing docs, maintenance, and most >> importantly, the total time spent by all users reading about this >> pointless thing in the docs and being distracted by it). It's only >> benefit would be to smooth over this debate on the mailing list; I >> can't believe that any real user will actually care about this, ever. >> > > Agreed. > > And +1 for 0/0 --> NaN. > The PR is now merged, with 0/0 --> NaN, and equal_var=True. Two things left to decide: 1) Do we want to transition to equal_var is False? 2) Do we want to unify the current 3 t-test function into one, like R/SAS? My answer to 2) would be yes, which also allows to do 1) without generating a deprecation warning. IMO this would simplify the API quite a bit, making things more understandable also for non-statisticians. Comparing APIs, I find ours quite poor: R: ttest SAS: TTEST Matlab: ttest, ttest2 SciPy: ttest_ind, ttest_1samp, ttest_rel The signature of a combined function ttest() would still be simple: def ttest(a, b=None, axis=0, popmean=0, equal_var=False) Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun Jun 10 07:29:44 2012 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 10 Jun 2012 12:29:44 +0100 Subject: [SciPy-Dev] Resolving PR 235: t-statistic = 0/0 case In-Reply-To: References: <4FCFBB03.4000102@gmail.com> <4FCFC909.2000805@gmail.com> <4FCFFAE9.5070007@gmail.com> <4FD02E15.109@gmail.com> Message-ID: On Sun, Jun 10, 2012 at 11:33 AM, Ralf Gommers wrote: > > > On Sat, Jun 9, 2012 at 1:04 PM, Ralf Gommers > wrote: >> >> >> >> On Thu, Jun 7, 2012 at 10:13 PM, Nathaniel Smith wrote: >>> >>> On Thu, Jun 7, 2012 at 5:29 AM, Junkshops wrote: >>> > - I'll merge the two 2 sample t-test functions >>> > - add an uneq_var=False kw arg, setting to true will use the new code >>> >>> equal_var would be a better name, to avoid the double-negative. >>> >>> Would it be possible/desireable to make equal_var=False the default? >>> Obviously this would require a deprecation period, but as semantic >>> changes go it's relatively low risk -- anyone who misses the warnings >>> etc. would just find one day that their t tests were producing more >>> conservative/realistic values. >> >> >> I'm not in favor of adding a deprecation warning for this. It's a minor >> thing, and warnings are annoying - it does require the user to go and figure >> out what changed. My preference would be to merge the current PR as is, and >> add a new function that combines all four t-tests with an interface similar >> to R. There the new default can be equal_var=False without annoying anyone. >> >>> >>> >>> (R defaults to doing the unequal variances test, and I have actually >>> seen this fact used in their advocacy, as evidence for their branding >>> as the tool for people who care about statistical rigor and >>> soundness.) >>> >>> > - add an zoz=np.nan kw arg and a check that it's np.nan, 0 or 1. >>> > Otherwise raise ValueError >>> >>> Let's please not add this "zoz=" feature. Adding features has a real >>> cost (in terms of testing, writing docs, maintenance, and most >>> importantly, the total time spent by all users reading about this >>> pointless thing in the docs and being distracted by it). It's only >>> benefit would be to smooth over this debate on the mailing list; I >>> can't believe that any real user will actually care about this, ever. >> >> >> Agreed. >> >> And +1 for 0/0 --> NaN. > > > The PR is now merged, with 0/0 --> NaN, and equal_var=True. > > Two things left to decide: > 1) Do we want to transition to equal_var is False? > 2) Do we want to unify the current 3 t-test function into one, like R/SAS? > > My answer to 2) would be yes, which also allows to do 1) without generating > a deprecation warning. IMO this would simplify the API quite a bit, making > things more understandable also for non-statisticians. Comparing APIs, I > find ours quite poor: > > R: ttest > SAS: TTEST > Matlab: ttest, ttest2 > SciPy: ttest_ind, ttest_1samp, ttest_rel > > The signature of a combined function ttest() would still be simple: > > def ttest(a, b=None, axis=0, popmean=0, equal_var=False) You need at least an argument for paired versus non-paired as well. R also has an argument to specify whether you want a two-tailed or one-tailed test (alternative="two.sided"/"less"/"greater"), which I guess is handy. I do think the combined signature is a little confusing, since many of the arguments only make sense for specific values of the other arguments. popmean is only meaningful for 1 sample tests (and paired tests, I guess, if we choose to interpret as the expected difference in that case?), equal_var and paired are only meaningful for two-sample tests, equal_var is only meaningful if paired is False. OTOH, I don't know if anyone cares -- obviously the rest of the world is getting by just fine with only 1 entry-point, and it's probably easier to find in the docs that way. -N From junkshops at gmail.com Sun Jun 10 11:49:42 2012 From: junkshops at gmail.com (Junkshops) Date: Sun, 10 Jun 2012 08:49:42 -0700 Subject: [SciPy-Dev] Resolving PR 235: t-statistic = 0/0 case In-Reply-To: References: <4FCFBB03.4000102@gmail.com> <4FCFC909.2000805@gmail.com> <4FCFFAE9.5070007@gmail.com> <4FD02E15.109@gmail.com> Message-ID: <4FD4C216.6000100@gmail.com> On 6/10/2012 4:29 AM, Nathaniel Smith wrote: > On Sun, Jun 10, 2012 at 11:33 AM, Ralf Gommers > wrote: >> *snip* >> >> The PR is now merged, with 0/0 --> NaN, and equal_var=True. >> >> Two things left to decide: >> 1) Do we want to transition to equal_var is False? >> 2) Do we want to unify the current 3 t-test function into one, like R/SAS? >> >> My answer to 2) would be yes, which also allows to do 1) without generating >> a deprecation warning. IMO this would simplify the API quite a bit, making >> things more understandable also for non-statisticians. Comparing APIs, I >> find ours quite poor: >> >> R: ttest >> SAS: TTEST >> Matlab: ttest, ttest2 >> SciPy: ttest_ind, ttest_1samp, ttest_rel >> >> The signature of a combined function ttest() would still be simple: >> >> def ttest(a, b=None, axis=0, popmean=0, equal_var=False) > You need at least an argument for paired versus non-paired as well. R > also has an argument to specify whether you want a two-tailed or > one-tailed test (alternative="two.sided"/"less"/"greater"), which I > guess is handy. > > I do think the combined signature is a little confusing, since many of > the arguments only make sense for specific values of the other > arguments. popmean is only meaningful for 1 sample tests (and paired > tests, I guess, if we choose to interpret as the expected difference > in that case?), equal_var and paired are only meaningful for > two-sample tests, equal_var is only meaningful if paired is False. > OTOH, I don't know if anyone cares -- obviously the rest of the world > is getting by just fine with only 1 entry-point, and it's probably > easier to find in the docs that way. > > -N If we go with def ttest(a, b=None, axis=0, popmean=0, equal_var=False, paired=False) Should the function have a hierarchy of tests that are performed when the function input is ambiguous regarding the desired test or raise an exception? Also, I'd suggest the default popmean=None to avoid ambiguity between the 1 sample and Welch's tests. An alternative: def ttest(a, b=None, axis=0, popmean=None, test='2sample_uneq_var') This doesn't have quite as many arguments, but breaks from the R-style parameters and the 'test' kw input strings might be unwieldy. However, the test the user wants to perform is unambiguous. Alternatively we could define constants rather than using strings, e.g. test = stats.2SAMPLE_UNEQ_VAR. The popmean and test keyword args could even be combined, but that is almost certainly too confusing. -g From bjorn.forsman at gmail.com Sun Jun 10 17:07:16 2012 From: bjorn.forsman at gmail.com (=?UTF-8?Q?Bj=C3=B8rn_Forsman?=) Date: Sun, 10 Jun 2012 23:07:16 +0200 Subject: [SciPy-Dev] Bug fix pull-request for signal.bode() Message-ID: Hi, I discovered a bug in signal.bode() that makes it unable to handle a systems with a zero-pole. signal.bode(([], [0], 1)) will provoke it. Please pull https://github.com/scipy/scipy/pull/253 to fix this bug. Best regards, Bj?rn Forsman From josef.pktd at gmail.com Sun Jun 10 23:09:49 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 10 Jun 2012 23:09:49 -0400 Subject: [SciPy-Dev] Resolving PR 235: t-statistic = 0/0 case In-Reply-To: <4FD4C216.6000100@gmail.com> References: <4FCFBB03.4000102@gmail.com> <4FCFC909.2000805@gmail.com> <4FCFFAE9.5070007@gmail.com> <4FD02E15.109@gmail.com> <4FD4C216.6000100@gmail.com> Message-ID: On Sun, Jun 10, 2012 at 11:49 AM, Junkshops wrote: > > > On 6/10/2012 4:29 AM, Nathaniel Smith wrote: >> On Sun, Jun 10, 2012 at 11:33 AM, Ralf Gommers >> ?wrote: >>> *snip* >>> >>> The PR is now merged, with 0/0 --> ?NaN, and equal_var=True. >>> >>> Two things left to decide: >>> 1) Do we want to transition to equal_var is False? >>> 2) Do we want to unify the current 3 t-test function into one, like R/SAS? >>> >>> My answer to 2) would be yes, which also allows to do 1) without generating >>> a deprecation warning. IMO this would simplify the API quite a bit, making >>> things more understandable also for non-statisticians. Comparing APIs, I >>> find ours quite poor: >>> >>> R: ttest >>> SAS: TTEST >>> Matlab: ttest, ttest2 >>> SciPy: ttest_ind, ttest_1samp, ttest_rel >>> >>> The signature of a combined function ttest() would still be simple: >>> >>> def ttest(a, b=None, axis=0, popmean=0, equal_var=False) >> You need at least an argument for paired versus non-paired as well. R >> also has an argument to specify whether you want a two-tailed or >> one-tailed test (alternative="two.sided"/"less"/"greater"), which I >> guess is handy. >> >> I do think the combined signature is a little confusing, since many of >> the arguments only make sense for specific values of the other >> arguments. popmean is only meaningful for 1 sample tests (and paired >> tests, I guess, if we choose to interpret as the expected difference >> in that case?), equal_var and paired are only meaningful for >> two-sample tests, equal_var is only meaningful if paired is False. >> OTOH, I don't know if anyone cares -- obviously the rest of the world >> is getting by just fine with only 1 entry-point, and it's probably >> easier to find in the docs that way. >> >> -N > > If we go with > > def ttest(a, b=None, axis=0, popmean=0, equal_var=False, paired=False) def ttest(a, b=None, value=0, paired=False, equal_var=False, axis=0) I wanted to suggest merging b and popmean, but I think the better solution is to allow for a more general version of the ttest, where the comparison is not necessarily 0. null hypothesis is mean(a) = value #ttest_1samp mean(a) - mean(b) = value #ttest_ind mean(a - b) = value #ttest_rel The only option that would then be ignored in two cases is equal_var, and paired would be irrelevant if b is None. (I didn't see an equal_var=False option for paired ttest in the SAS manual. ?) > > Should the function have a hierarchy of tests that are performed when > the function input is ambiguous regarding the desired test or raise an > exception? Also, I'd suggest the default popmean=None to avoid ambiguity > between the 1 sample and Welch's tests. > > An alternative: > > def ttest(a, b=None, axis=0, popmean=None, test='2sample_uneq_var') > > This doesn't have quite as many arguments, but breaks from the R-style > parameters and the 'test' kw input strings might be unwieldy. However, > the test the user wants to perform is unambiguous. Alternatively we > could define constants rather than using strings, e.g. test = > stats.2SAMPLE_UNEQ_VAR. > > The popmean and test keyword args could even be combined, but that is > almost certainly too confusing. There is no gain in usability in this case. I'd rather choose among three functions (that are next to each other with tab completion), than specifying the same three functions as a keyword argument. I find the current split into 3 functions for 3 different sampling cases quite natural. Josef > > -g > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From ralf.gommers at googlemail.com Mon Jun 11 02:28:44 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 11 Jun 2012 08:28:44 +0200 Subject: [SciPy-Dev] branching 0.11.x In-Reply-To: References: Message-ID: On Sat, Jun 9, 2012 at 3:46 PM, Ralf Gommers wrote: > Hi all, > > We're almost ready to branch 0.11.x and tag the first beta. Here's some > things that are still TODO. > > Necessary to finish before the first beta: > - Fix http://projects.scipy.org/scipy/ticket/1599 (work around segfaults > due to single-precision shift-invert mode in Arpack) > - https://github.com/scipy/scipy/pull/217 (Finishing optimize.minimize() > interface changes) > > Can be fixed after the first beta: > - http://projects.scipy.org/scipy/ticket/1669 (sparse.csgraph test > failures) > - https://github.com/scipy/scipy/pull/198 (fitting discrete distributions > issue) > > Things that would be good to include, but will be postponed unless they're > merged in time: > - https://github.com/scipy/scipy/pull/236 (use memoization in Minpack > routines) > - https://github.com/scipy/scipy/pull/235 (t-test with unequal variances) > - https://github.com/scipy/scipy/pull/173 (binned statistics) > - any other PRs / patches that are ready > > If we can get the necessary items resolved this weekend, I plan to create > the branch tomorrow evening (at or after 9pm GMT). > An update: we got a lot of the above done and are almost ready to branch and release the first beta. In total we got 17 PRs merged and 9 tickets closed this weekend. Thanks to everyone who pitched in. Unfortunately there's currently one (new) release blocker that makes all the Windows builds fail, http://projects.scipy.org/scipy/ticket/1670. Once this is resolved we can move ahead. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Mon Jun 11 02:29:36 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 11 Jun 2012 08:29:36 +0200 Subject: [SciPy-Dev] branching 0.11.x In-Reply-To: References: Message-ID: On Mon, Jun 11, 2012 at 8:28 AM, Ralf Gommers wrote: > > > On Sat, Jun 9, 2012 at 3:46 PM, Ralf Gommers wrote: > >> Hi all, >> >> We're almost ready to branch 0.11.x and tag the first beta. Here's some >> things that are still TODO. >> >> Necessary to finish before the first beta: >> - Fix http://projects.scipy.org/scipy/ticket/1599 (work around segfaults >> due to single-precision shift-invert mode in Arpack) >> - https://github.com/scipy/scipy/pull/217 (Finishing optimize.minimize() >> interface changes) >> >> Can be fixed after the first beta: >> - http://projects.scipy.org/scipy/ticket/1669 (sparse.csgraph test >> failures) >> - https://github.com/scipy/scipy/pull/198 (fitting discrete >> distributions issue) >> >> Things that would be good to include, but will be postponed unless >> they're merged in time: >> - https://github.com/scipy/scipy/pull/236 (use memoization in Minpack >> routines) >> - https://github.com/scipy/scipy/pull/235 (t-test with unequal variances) >> - https://github.com/scipy/scipy/pull/173 (binned statistics) >> - any other PRs / patches that are ready >> >> If we can get the necessary items resolved this weekend, I plan to create >> the branch tomorrow evening (at or after 9pm GMT). >> > > An update: we got a lot of the above done and are almost ready to branch > and release the first beta. In total we got 17 PRs merged and 9 tickets > closed this weekend. Thanks to everyone who pitched in. > > Unfortunately there's currently one (new) release blocker that makes all > the Windows builds fail, http://projects.scipy.org/scipy/ticket/1670. > Once this is resolved we can move ahead. > Sorry, should be http://projects.scipy.org/scipy/ticket/1673. > > Ralf > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nabil.freij at gmail.com Mon Jun 11 11:20:32 2012 From: nabil.freij at gmail.com (Nabil Freij) Date: Mon, 11 Jun 2012 16:20:32 +0100 Subject: [SciPy-Dev] Missing functions in scipy.signals reference manual Message-ID: Hey, It appears that several functions in scipy.signals.bspline (cspline1d_eval for example) are missing from the reference manual. Is there any reason for this? KInd regards, Nabil -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Mon Jun 11 16:18:27 2012 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Mon, 11 Jun 2012 15:18:27 -0500 Subject: [SciPy-Dev] scipy-central.org down? Message-ID: Is scipy-central.org working for anyone? I get a 403 error. Warren -------------- next part -------------- An HTML attachment was scrubbed... URL: From scopatz at gmail.com Mon Jun 11 16:28:49 2012 From: scopatz at gmail.com (Anthony Scopatz) Date: Mon, 11 Jun 2012 15:28:49 -0500 Subject: [SciPy-Dev] scipy-central.org down? In-Reply-To: References: Message-ID: Yup it is for me.... On Mon, Jun 11, 2012 at 3:18 PM, Warren Weckesser < warren.weckesser at enuthought.com > wrote: > Is scipy-central.org working for anyone? I get a 403 error. > > Warren > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eraldo.pomponi at gmail.com Mon Jun 11 16:33:27 2012 From: eraldo.pomponi at gmail.com (Eraldo Pomponi) Date: Mon, 11 Jun 2012 22:33:27 +0200 Subject: [SciPy-Dev] scipy-central.org down? In-Reply-To: References: Message-ID: Not for me .... Cheers, Eraldo On Mon, Jun 11, 2012 at 10:28 PM, Anthony Scopatz wrote: > Yup it is for me.... > > On Mon, Jun 11, 2012 at 3:18 PM, Warren Weckesser < > warren.weckesser at enuthought.com > wrote: > >> Is scipy-central.org working for anyone? I get a 403 error. >> >> Warren >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev >> >> > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From scopatz at gmail.com Mon Jun 11 16:40:35 2012 From: scopatz at gmail.com (Anthony Scopatz) Date: Mon, 11 Jun 2012 15:40:35 -0500 Subject: [SciPy-Dev] scipy-central.org down? In-Reply-To: References: Message-ID: Errr I should say that it is *down* for me, and therefore *not* working. On Mon, Jun 11, 2012 at 3:28 PM, Anthony Scopatz wrote: > Yup it is for me.... > > On Mon, Jun 11, 2012 at 3:18 PM, Warren Weckesser < > warren.weckesser at enuthought.com > wrote: > >> Is scipy-central.org working for anyone? I get a 403 error. >> >> Warren >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kgdunn at gmail.com Mon Jun 11 17:08:23 2012 From: kgdunn at gmail.com (Kevin Dunn) Date: Mon, 11 Jun 2012 17:08:23 -0400 Subject: [SciPy-Dev] scipy-central.org down? In-Reply-To: References: Message-ID: Thanks for the report. It's back up now. Not sure what went wrong. Will check the logs when I get back home tonight. Kevin On Mon, Jun 11, 2012 at 4:40 PM, Anthony Scopatz wrote: > Errr I should say that it is *down* for me, and therefore *not* working. > > > On Mon, Jun 11, 2012 at 3:28 PM, Anthony Scopatz wrote: >> >> Yup it is for me.... >> >> On Mon, Jun 11, 2012 at 3:18 PM, Warren Weckesser >> wrote: >>> >>> Is scipy-central.org working for anyone?? I get a 403 error. >>> >>> Warren >>> >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>> >> > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > From warren.weckesser at enthought.com Mon Jun 11 17:10:58 2012 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Mon, 11 Jun 2012 16:10:58 -0500 Subject: [SciPy-Dev] scipy-central.org down? In-Reply-To: References: Message-ID: On Mon, Jun 11, 2012 at 4:08 PM, Kevin Dunn wrote: > Thanks for the report. > > It's back up now. Not sure what went wrong. Will check the logs when I > get back home tonight. > Thanks, Kevin. Warren > > Kevin > > On Mon, Jun 11, 2012 at 4:40 PM, Anthony Scopatz > wrote: > > Errr I should say that it is *down* for me, and therefore *not* working. > > > > > > On Mon, Jun 11, 2012 at 3:28 PM, Anthony Scopatz > wrote: > >> > >> Yup it is for me.... > >> > >> On Mon, Jun 11, 2012 at 3:18 PM, Warren Weckesser > >> wrote: > >>> > >>> Is scipy-central.org working for anyone? I get a 403 error. > >>> > >>> Warren > >>> > >>> > >>> _______________________________________________ > >>> SciPy-Dev mailing list > >>> SciPy-Dev at scipy.org > >>> http://mail.scipy.org/mailman/listinfo/scipy-dev > >>> > >> > > > > > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-dev > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Tue Jun 12 17:29:52 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 12 Jun 2012 23:29:52 +0200 Subject: [SciPy-Dev] branching 0.11.x In-Reply-To: References: Message-ID: On Mon, Jun 11, 2012 at 8:29 AM, Ralf Gommers wrote: > > > On Mon, Jun 11, 2012 at 8:28 AM, Ralf Gommers > wrote: > >> >> >> On Sat, Jun 9, 2012 at 3:46 PM, Ralf Gommers > > wrote: >> >>> Hi all, >>> >>> We're almost ready to branch 0.11.x and tag the first beta. Here's some >>> things that are still TODO. >>> >>> Necessary to finish before the first beta: >>> - Fix http://projects.scipy.org/scipy/ticket/1599 (work around >>> segfaults due to single-precision shift-invert mode in Arpack) >>> - https://github.com/scipy/scipy/pull/217 (Finishing >>> optimize.minimize() interface changes) >>> >>> Can be fixed after the first beta: >>> - http://projects.scipy.org/scipy/ticket/1669 (sparse.csgraph test >>> failures) >>> - https://github.com/scipy/scipy/pull/198 (fitting discrete >>> distributions issue) >>> >>> Things that would be good to include, but will be postponed unless >>> they're merged in time: >>> - https://github.com/scipy/scipy/pull/236 (use memoization in Minpack >>> routines) >>> - https://github.com/scipy/scipy/pull/235 (t-test with unequal >>> variances) >>> - https://github.com/scipy/scipy/pull/173 (binned statistics) >>> - any other PRs / patches that are ready >>> >>> If we can get the necessary items resolved this weekend, I plan to >>> create the branch tomorrow evening (at or after 9pm GMT). >>> >> >> An update: we got a lot of the above done and are almost ready to branch >> and release the first beta. In total we got 17 PRs merged and 9 tickets >> closed this weekend. Thanks to everyone who pitched in. >> >> Unfortunately there's currently one (new) release blocker that makes all >> the Windows builds fail, http://projects.scipy.org/scipy/ticket/1670. >> Once this is resolved we can move ahead. >> > > Sorry, should be http://projects.scipy.org/scipy/ticket/1673. > Hi all, the last blocker has been resolved and the 0.11.x branch created. The first beta will follow soon. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Tue Jun 12 17:42:48 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 12 Jun 2012 23:42:48 +0200 Subject: [SciPy-Dev] Missing functions in scipy.signals reference manual In-Reply-To: References: Message-ID: On Mon, Jun 11, 2012 at 5:20 PM, Nabil Freij wrote: > Hey, > > It appears that several functions in scipy.signals.bspline (cspline1d_eval > for example) are missing from the reference manual. Is there any reason for > this? > It doesn't look like that was intentional. Missing are: cspline1d_eval qspline1d_eval cubic quadratic Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From npkuin at gmail.com Tue Jun 12 18:01:59 2012 From: npkuin at gmail.com (Paul Kuin) Date: Tue, 12 Jun 2012 23:01:59 +0100 Subject: [SciPy-Dev] Fwd: Software paper invitation (Swift UVOT grism Python software) In-Reply-To: References: Message-ID: Hi All, I was thinking that this email invitation I got for publishing about (my) software might be of interest to some of you also (for other stuff of course & Depending on what you develop & whether it is of interest to the 'research community'). Paul ---------- Forwarded message ---------- From: Brian Hole Date: Tue, Jun 12, 2012 at 11:36 AM Subject: Software paper invitation (Swift UVOT grism Python software) Dear Dr Kuin, I?m writing to you in connection with the Swift UVOT grism Python software you have ... We believe that this is important software of great use to the research community, and we would like to offer you a method of increasing its reuse and impact. Ubiquity Press (http://www.ubiquitypress.com/) is a researcher-led open access publisher based on the UCL campus, and we are in the process of launching a journal with the Software Sustainability Institute (http://www.software.ac.uk), called the Journal of Open Research Software (http://openresearchsoftware.metajnl.com). We would like to offer you the opportunity of contributing a software paper about the above software to the journal. Software papers are short descriptions of the software, which detail the methodology with which it was produced, link to its location in an archive, and very importantly, detail its reuse potential. The aim of the paper is to make other researchers aware of the software and encourage its reuse, and provide the creators of the software with a way to gain recognition for its production. The papers are of course citable, and we will track these citations as with any other article type so that you can see who is using the software and what kind of impact it is having. All software papers are peer reviewed, and require that the software itself be available in a repository such as UCL Discovery that can guarantee its long-term preservation and availability. Please let us know if you are interested in contributing a paper, or if you have any questions. All the best, Brian -- Brian Hole Ubiquity Press Ltd. ... www.ubiquitypress.com www.twitter.com/ubiquitypress From ralf.gommers at googlemail.com Wed Jun 13 13:56:30 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 13 Jun 2012 19:56:30 +0200 Subject: [SciPy-Dev] ANN: SciPy 0.11.0 beta 1 released Message-ID: Hi, I am pleased to announce the availability of the first beta release of SciPy0.11.0. For this release over 120 tickets and pull requests have been closed, and many new features have been added. Also noteworthy is that the number of contributors for this release has risen to over 50. Some of the highlights are: - A new module, sparse.csgraph, has been added which provides a number of common sparse graph algorithms. - New unified interfaces to the existing optimization and root finding functions have been added. Sources and binaries can be found at http://sourceforge.net/projects/scipy/files/scipy/0.11.0b1/, release notes are copied below. Please try this release and report any problems on the mailing list. Cheers, Ralf ========================== SciPy 0.11.0 Release Notes ========================== .. note:: Scipy 0.11.0 is not released yet! .. contents:: SciPy 0.11.0 is the culmination of 8 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation. Highlights of this release are: - A new module has been added which provides a number of common sparse graph algorithms. - New unified interfaces to the existing optimization and root finding functions have been added. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Our development attention will now shift to bug-fix releases on the 0.11.x branch, and on adding new features on the master branch. This release requires Python 2.4-2.7 or 3.1-3.2 and NumPy 1.5.1 or greater. New features ============ Sparse Graph Submodule ---------------------- The new submodule :mod:`scipy.sparse.csgraph` implements a number of efficient graph algorithms for graphs stored as sparse adjacency matrices. Available routines are: - :func:`connected_components` - determine connected components of a graph - :func:`laplacian` - compute the laplacian of a graph - :func:`shortest_path` - compute the shortest path between points on a positive graph - :func:`dijkstra` - use Dijkstra's algorithm for shortest path - :func:`floyd_warshall` - use the Floyd-Warshall algorithm for shortest path - :func:`breadth_first_order` - compute a breadth-first order of nodes - :func:`depth_first_order` - compute a depth-first order of nodes - :func:`breadth_first_tree` - construct the breadth-first tree from a given node - :func:`depth_first_tree` - construct a depth-first tree from a given node - :func:`minimum_spanning_tree` - construct the minimum spanning tree of a graph ``scipy.optimize`` improvements ------------------------------- The optimize module has received a lot of attention this release. In addition to added tests, documentation improvements, bug fixes and code clean-up, the following improvements were made: - A unified interface to minimizers of univariate and multivariate functions has been added. - A unified interface to root finding algorithms for multivariate functions has been added. - The L-BFGS-B algorithm has been updated to version 3.0. Unified interfaces to minimizers ```````````````````````````````` Two new functions ``scipy.optimize.minimize`` and ``scipy.optimize.minimize_scalar`` were added to provide a common interface to minimizers of multivariate and univariate functions respectively. For multivariate functions, ``scipy.optimize.minimize`` provides an interface to methods for unconstrained optimization (`fmin`, `fmin_powell`, `fmin_cg`, `fmin_ncg`, `fmin_bfgs` and `anneal`) or constrained optimization (`fmin_l_bfgs_b`, `fmin_tnc`, `fmin_cobyla` and `fmin_slsqp`). For univariate functions, ``scipy.optimize.minimize_scalar`` provides an interface to methods for unconstrained and bounded optimization (`brent`, `golden`, `fminbound`). This allows for easier comparing and switching between solvers. Unified interface to root finding algorithms ```````````````````````````````````````````` The new function ``scipy.optimize.root`` provides a common interface to root finding algorithms for multivariate functions, embeding `fsolve`, `leastsq` and `nonlin` solvers. ``scipy.linalg`` improvements ----------------------------- New matrix equation solvers ``````````````````````````` Solvers for the Sylvester equation (``scipy.linalg.solve_sylvester``, discrete and continuous Lyapunov equations (``scipy.linalg.solve_lyapunov``, ``scipy.linalg.solve_discrete_lyapunov``) and discrete and continuous algebraic Riccati equations (``scipy.linalg.solve_continuous_are``, ``scipy.linalg.solve_discrete_are``) have been added to ``scipy.linalg``. These solvers are often used in the field of linear control theory. QZ and QR Decomposition ```````````````````````` It is now possible to calculate the QZ, or Generalized Schur, decomposition using ``scipy.linalg.qz``. This function wraps the LAPACK routines sgges, dgges, cgges, and zgges. The function ``scipy.linalg.qr_multiply``, which allows efficient computation of the matrix product of Q (from a QR decompostion) and a vector, has been added. Pascal matrices ``````````````` A function for creating Pascal matrices, ``scipy.linalg.pascal``, was added. Sparse matrix construction and operations ----------------------------------------- Two new functions, ``scipy.sparse.diags`` and ``scipy.sparse.block_diag``, were added to easily construct diagonal and block-diagonal sparse matrices respectively. ``scipy.sparse.csc_matrix`` and ``csr_matrix`` now support the operations ``sin``, ``tan``, ``arcsin``, ``arctan``, ``sinh``, ``tanh``, ``arcsinh``, ``arctanh``, ``rint``, ``sign``, ``expm1``, ``log1p``, ``deg2rad``, ``rad2deg``, ``floor``, ``ceil`` and ``trunc``. Previously, these operations had to be performed by operating on the matrices' ``data`` attribute. LSMR iterative solver --------------------- LSMR, an iterative method for solving (sparse) linear and linear least-squares systems, was added as ``scipy.sparse.linalg.lsmr``. Discrete Sine Transform ----------------------- Bindings for the discrete sine transform functions have been added to ``scipy.fftpack``. ``scipy.interpolate`` improvements ---------------------------------- For interpolation in spherical coordinates, the three classes ``scipy.interpolate.SmoothSphereBivariateSpline``, ``scipy.interpolate.LSQSphereBivariateSpline``, and ``scipy.interpolate.RectSphereBivariateSpline`` have been added. Binned statistics (``scipy.stats``) ----------------------------------- The stats module has gained functions to do binned statistics, which are a generalization of histograms, in 1-D, 2-D and multiple dimensions: ``scipy.stats.binned_statistic``, ``scipy.stats.binned_statistic_2d`` and ``scipy.stats.binned_statistic_dd``. Deprecated features =================== ``scipy.sparse.cs_graph_components`` has been made a part of the sparse graph submodule, and renamed to ``scipy.sparse.csgraph.connected_components``. Calling the former routine will result in a deprecation warning. ``scipy.misc.radon`` has been deprecated. A more full-featured radon transform can be found in scikits-image. ``scipy.io.save_as_module`` has been deprecated. A better way to save multiple Numpy arrays is the ``numpy.savez`` function. The `xa` and `xb` parameters for all distributions in ``scipy.stats.distributions`` already weren't used; they have now been deprecated. Backwards incompatible changes ============================== Removal of ``scipy.maxentropy`` ------------------------------- The ``scipy.maxentropy`` module, which was deprecated in the 0.10.0 release, has been removed. Logistic regression in scikits.learn is a good and modern alternative for this functionality. alternative for this functionality. Minor change in behavior of ``splev`` ------------------------------------- The spline evaluation function now behaves similarly to ``interp1d`` for size-1 arrays. Previous behavior:: >>> from scipy.interpolate import splev, splrep, interp1d >>> x = [1,2,3,4,5] >>> y = [4,5,6,7,8] >>> tck = splrep(x, y) >>> splev([1], tck) 4. >>> splev(1, tck) 4. Corrected behavior:: >>> splev([1], tck) array([ 4.]) >>> splev(1, tck) array(4.) This affects also the ``UnivariateSpline`` classes. Behavior of ``scipy.integrate.complex_ode`` ------------------------------------------- The behavior of the ``y`` attribute of ``complex_ode`` is changed. Previously, it expressed the complex-valued solution in the form:: z = ode.y[::2] + 1j * ode.y[1::2] Now, it is directly the complex-valued solution:: z = ode.y Minor change in behavior of T-tests ----------------------------------- The T-tests ``scipy.stats.ttest_ind``, ``scipy.stats.ttest_rel`` and ``scipy.stats.ttest_1samp`` have been changed so that 0 / 0 now returns NaN instead of 1. Other changes ============= The SuperLU sources in ``scipy.sparse.linalg`` have been updated to version 4.3 from upstream. The function ``scipy.signal.bode``, which calculates magnitude and phase data for a continuous-time system, has been added. The two-sample T-test ``scipy.stats.ttest_ind`` gained an option to compare samples with unequal variances, i.e. Welch's T-test. ``scipy.misc.logsumexp`` now takes an optional ``axis`` keyword argument. Authors ======= This release contains work by the following people (contributed at least one patch to this release, names in alphabetical order): * Jeff Armstrong * Chad Baker * Brandon Beacher + * behrisch + * borishim + * Matthew Brett * Lars Buitinck * Luis Pedro Coelho + * Johann Cohen-Tanugi * David Cournapeau * dougal + * Ali Ebrahim + * endolith + * Bj?rn Forsman + * Robert Gantner + * Sebastian Gassner + * Christoph Gohlke * Ralf Gommers * Yaroslav Halchenko * Charles Harris * Jonathan Helmus + * Andreas Hilboll + * Marc Honnorat + * Jonathan Hunt + * Maxim Ivanov + * Thouis (Ray) Jones * Christopher Kuster + * Denis Laxalde + * Travis Oliphant * Joonas Paalasmaa + * Fabian Pedregosa * Josef Perktold * Gavin Price + * Jim Radford + * Andrew Schein + * Skipper Seabold * Jacob Silterra + * Scott Sinclair * Alexis Tabary + * Martin Teichmann * Matt Terry + * Nicky van Foreest + * Jacob Vanderplas * Patrick Varilly + * Pauli Virtanen * Nils Wagner + * Darryl Wally + * Stefan van der Walt * Liming Wang + * David Warde-Farley + * Warren Weckesser * Sebastian Werk + * Mike Wimmer + * Tony S Yu + A total of 54 people contributed to this release. People with a "+" by their names contributed a patch for the first time. -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott.sinclair.za at gmail.com Thu Jun 14 03:11:40 2012 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Thu, 14 Jun 2012 09:11:40 +0200 Subject: [SciPy-Dev] ANN: SciPy 0.11.0 beta 1 released In-Reply-To: References: Message-ID: On 13 June 2012 19:56, Ralf Gommers wrote: > I am pleased to announce the availability of the first beta release of SciPy > 0.11.0. > > Please try this release and report any problems on the mailing list. I see some Py3.2 test failures on a 64bit Ubuntu 12.04 machine. Py2.7 has no problems. -------------------------------------------------- Running unit tests for scipy NumPy version 1.6.2 NumPy is installed in /home/scott/.local/lib/python2.7/site-packages/numpy SciPy version 0.11.0b1 SciPy is installed in /home/scott/.virtualenvs/scipy-release-candidates/local/lib/python2.7/site-packages/scipy Python version 2.7.3 (default, Apr 20 2012, 22:39:59) [GCC 4.6.3] nose version 1.1.2 ... Ran 5480 tests in 48.055s OK (KNOWNFAIL=13, SKIP=35) -------------------------------------------------- -------------------------------------------------- Running unit tests for scipy NumPy version 1.6.2 NumPy is installed in /home/scott/.virtualenvs/scipy-py3/lib/python3.2/site-packages/numpy SciPy version 0.11.0b1 SciPy is installed in /home/scott/.virtualenvs/scipy-py3/lib/python3.2/site-packages/scipy Python version 3.2.3 (default, May 3 2012, 15:51:42) [GCC 4.6.3] nose version 1.1.2 ... ====================================================================== ERROR: test_smoke_bisplrep_bisplev (test_fitpack.TestSmokeTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/scott/.virtualenvs/scipy-py3/lib/python3.2/site-packages/scipy/interpolate/tests/test_fitpack.py", line 219, in test_smoke_bisplrep_bisplev self.check_5() File "/home/scott/.virtualenvs/scipy-py3/lib/python3.2/site-packages/scipy/interpolate/tests/test_fitpack.py", line 180, in check_5 xy=makepairs(x,y) File "/home/scott/.virtualenvs/scipy-py3/lib/python3.2/site-packages/scipy/interpolate/tests/test_fitpack.py", line 32, in makepairs xy.shape=sh[0]*sh[1],sh[2] IndexError: tuple index out of range ====================================================================== ERROR: Failure: AttributeError ('module' object has no attribute 'FileType') ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/scott/.virtualenvs/scipy-py3/lib/python3.2/site-packages/nose/failure.py", line 37, in runTest raise self.exc_class(self.exc_val).with_traceback(self.tb) File "/home/scott/.virtualenvs/scipy-py3/lib/python3.2/site-packages/nose/loader.py", line 390, in loadTestsFromName addr.filename, addr.module) File "/home/scott/.virtualenvs/scipy-py3/lib/python3.2/site-packages/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/home/scott/.virtualenvs/scipy-py3/lib/python3.2/site-packages/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/home/scott/.virtualenvs/scipy-py3/lib/python3.2/site-packages/scipy/weave/__init__.py", line 22, in from .blitz_tools import blitz File "/home/scott/.virtualenvs/scipy-py3/lib/python3.2/site-packages/scipy/weave/blitz_tools.py", line 6, in from . import converters File "/home/scott/.virtualenvs/scipy-py3/lib/python3.2/site-packages/scipy/weave/converters.py", line 19, in c_spec.file_converter(), File "/home/scott/.virtualenvs/scipy-py3/lib/python3.2/site-packages/scipy/weave/c_spec.py", line 74, in __init__ self.init_info() File "/home/scott/.virtualenvs/scipy-py3/lib/python3.2/site-packages/scipy/weave/c_spec.py", line 264, in init_info self.matching_types = [types.FileType] AttributeError: 'module' object has no attribute 'FileType' ====================================================================== FAIL: test_shortest_path.test_undirected(array([[ 0., 3., 3., 1., 3.], ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/scott/.virtualenvs/scipy-py3/lib/python3.2/site-packages/nose/case.py", line 198, in runTest self.test(*self.arg) File "/home/scott/.virtualenvs/scipy-py3/lib/python3.2/site-packages/numpy/testing/utils.py", line 800, in assert_array_almost_equal header=('Arrays are not almost equal to %d decimals' % decimal)) File "/home/scott/.virtualenvs/scipy-py3/lib/python3.2/site-packages/numpy/testing/utils.py", line 636, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not almost equal to 6 decimals (mismatch 16.0%) x: array([[ 0., 3., 3., 1., 3.], [ 3., 0., 6., 2., 4.], [ 3., 6., 0., 4., 6.],... y: array([[ 0., 3., 3., 1., 2.], [ 3., 0., 6., 2., 4.], [ 3., 6., 0., 4., 5.],... ---------------------------------------------------------------------- Ran 5343 tests in 53.577s FAILED (KNOWNFAIL=13, SKIP=41, errors=2, failures=1) -------------------------------------------------- Cheers, Scott From goxberry at gmail.com Tue Jun 19 12:32:02 2012 From: goxberry at gmail.com (Geoff Oxberry) Date: Tue, 19 Jun 2012 12:32:02 -0400 Subject: [SciPy-Dev] Adding new wrappers (dassl, daspk version 2, cvode version 2.7.0) to scipy.integrate? Message-ID: I posted a feature request ticket (#1678, http://projects.scipy.org/scipy/ticket/1678) about this on the SciPy Dev Wiki, and Ralf Gommers suggested that I follow up on the scipy-dev mailing list. To recap the gist of my ticket, there are problems for which DVODE (currently wrapped in scipy.integrate.ode) are inadequate, such as in computational combustion. It would be helpful to researchers in combustion to wrap updated numerical integration libraries (such as DASSL, DASPK, and CVODE); DAE solver wrappers would also be very helpful (technically, DASSL and DASPK are DAE solvers, but one thing at a time here). Since some of these wrappers already exist in scikits.odes, would it be possible to merge them into scipy.integrate.ode? If so, in the process, could the Sundials wrappers also be updated to operate with Sundials version 2.5.0 instead of Sundials version 2.4.0? The newest version of Sundials contains only minor changes to the API (a few integer arguments are changed to long integers to accommodate larger problem sizes), and several bugs are fixed. There have been many e-mails on the Sundials-users mailing list requesting a good, well-maintained and supported Python interface to Sundials with a reputable institution behind it. It looks like scikits.odes is the best-maintained Python interface out there, since pysundials and python-sundials are no longer maintained, and the Assimulo documentation uses some nomenclature that is nonstandard in numerical analysis (which makes me doubt the software). Furthermore, I know that users in the combustion community, frustrated with this situation, have implemented their own wrappers to such solvers (see for instance, https://github.com/jwallen/PyDAS, written by a colleague of mine). I am willing to help with this effort after I graduate (I am in the process of finishing my PhD) in a few months. Geoff From pav at iki.fi Tue Jun 19 15:58:30 2012 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 19 Jun 2012 21:58:30 +0200 Subject: [SciPy-Dev] Adding new wrappers (dassl, daspk version 2, cvode version 2.7.0) to scipy.integrate? In-Reply-To: References: Message-ID: Hi, 19.06.2012 18:32, Geoff Oxberry kirjoitti: > I posted a feature request ticket (#1678, > http://projects.scipy.org/scipy/ticket/1678) about this on the SciPy > Dev Wiki, and Ralf Gommers suggested that I follow up on the scipy-dev mailing list. > > To recap the gist of my ticket, there are problems for which DVODE > (currently wrapped in scipy.integrate.ode) are inadequate, such as > in computational combustion. It would be helpful to researchers in > combustion to wrap updated numerical integration libraries (such as > DASSL, DASPK, and CVODE); DAE solver wrappers would also be very helpful > (technically, DASSL and DASPK are DAE solvers, but one thing at a time here). Adding more ODE solver engines to scipy.integrate would certainly be useful, and doing it should be OK within the scipy.integrate.ode API. DAE solvers, I think, also fall within the scope for Scipy, and could be added. These I'd like however to see added with a cleaner API than what we currently have --- i.e. not bolted onto odeint/ode. At the same time, it could be useful to rewrite the ODE stuff to use a similar API design, as the present situation with duplicated but different `ode/odeint` and some issues with `ode` is not so desirable. *** The design in scikits.odes looks good. If the work there is mature enough, and there are no license issues, it might indeed be useful to concentrate these things under `scipy.integrate`. Would the people involved with scikits.odes like to move part of their development under scipy.integrate? The main issue is maybe the slower release cycle of Scipy and the more frozen API? -- Pauli Virtanen From claus at maths.lth.se Tue Jun 19 16:54:26 2012 From: claus at maths.lth.se (Claus =?utf-8?b?RsO8aHJlcg==?=) Date: Tue, 19 Jun 2012 20:54:26 +0000 (UTC) Subject: [SciPy-Dev] =?utf-8?q?Adding_new_wrappers_=28dassl=2C_daspk_versi?= =?utf-8?q?on_2=2C=09cvode_version_2=2E7=2E0=29_to_scipy=2Eintegrat?= =?utf-8?q?e=3F?= References: Message-ID: Geoff Oxberry gmail.com> writes: > > I posted a feature request ticket (#1678, http://projects.scipy.org/scipy/ticket/1678) about this on > the SciPy Dev Wiki, and Ralf Gommers suggested that I follow up on the scipy-dev mailing list. > > To recap the gist of my ticket, there are problems for which DVODE (currently wrapped in > scipy.integrate.ode) are inadequate, such as in computational combustion. It would be helpful to > researchers in combustion to wrap updated numerical integration libraries (such as DASSL, DASPK, and > CVODE); DAE solver wrappers would also be very helpful (technically, DASSL and DASPK are DAE solvers, but > one thing at a time here). > > Since some of these wrappers already exist in scikits.odes, would it be possible to merge them into > scipy.integrate.ode? If so, in the process, could the Sundials wrappers also be updated to operate with > Sundials version 2.5.0 instead of Sundials version 2.4.0? The newest version of Sundials contains only > minor changes to the API (a few integer arguments are changed to long integers to accommodate larger > problem sizes), and several bugs are fixed. > > There have been many e-mails on the Sundials-users mailing list requesting a good, well-maintained and > supported Python interface to Sundials with a reputable institution behind it. It looks like > scikits.odes is the best-maintained Python interface out there, since pysundials and python-sundials > are no longer maintained, and the Assimulo documentation uses some nomenclature that is nonstandard in > numerical analysis (which makes me doubt the software). Furthermore, I know that users in the combustion > community, frustrated with this situation, have implemented their own wrappers to such solvers (see for > instance, https://github.com/jwallen/PyDAS, written by a colleague of mine). > > I am willing to help with this effort after I graduate (I am in the process of finishing my PhD) in a few months. > > Geoff > Hi Geoff, I am not sure if Assimulo is the tool you are looking for but your comment about it makes me curious. As a co-author of this package and specialist in DAE numerics since 25 years I would be grateful to learn in which aspects the nomenclature is "non-standard" in numerical analysis. Can you give me some hints? Best regards Claus F?hrer Professor in Numerical Analysis Lund, Sweden From goxberry at gmail.com Tue Jun 19 21:38:31 2012 From: goxberry at gmail.com (Geoff Oxberry) Date: Tue, 19 Jun 2012 21:38:31 -0400 Subject: [SciPy-Dev] Adding new wrappers (dassl, daspk version 2, cvode version 2.7.0) to scipy.integrate? Message-ID: Claus, What made me skittish about the API documentation of Assimulo is when you describe the CVODE in the "Documentation contents" page of your documentation (http://www.jmodelica.org/assimulo) as an "explicit solver". That, to me, seemed "non-standard". After an initial moment of confusion, I realized that you were instead talking about explicit ordinary differential equations, not explicit methods for integrating ODEs. (I believe my comments on the Trac Ticket reflect this.) As someone who's seen enough software thrown out there by people who either don't maintain it, or don't document it, I get very skeptical when I see something that looks out of place in documentation. Anyone who jumps to the Assimulo home page will immediately realize the distinction you're making; I just happened to type in "assimulo documentation" because I had already been recommended the package by a user on Computational Science Stack Exchange (of which I am a moderator). When I downloaded your project to compile the Trac ticket report, I did notice that your project is among the more actively maintained wrappers of Sundials out there, and receiving an e-mail from you is a pleasant surprise. I also appreciate that your project has documentation. Of the six wrappers to Sundials or DASSL that I know of out there, Assimulo looks like it's the best documented. Recently, I attempted to integrate a stiff system of ODEs derived from the GRI-Mech 3.0 combustion chemistry model with DVODE, varying the absolute tolerances from 1e-9 to 1e-20 and the relative tolerances from 1e-6 to 1e-15. For whatever reason, I don't obtain the expected solution behavior, but I do obtain the expected solution behavior with the MATLAB wrapper to CVODE, and I also obtain this behavior with a modified version of DASSL (that simulation was programmed in Fortran 90). Perhaps I should give Assimulo's wrapper a try, especially since I've found that IDA tends to work well on my problems also. That said, I still believe that DVODE in scipy.integrate should be supplemented (or even replaced) by CVODE. DVODE has had a long history of successful use in combustion chemistry, but CVODE, DASSL, DASPK, and IDA seem to have superior stepping heuristics for these sorts of problems, based on my 5 or so years of experience working on them. Thank you for your response, Geoff From benny.malengier at gmail.com Wed Jun 20 03:43:10 2012 From: benny.malengier at gmail.com (Benny Malengier) Date: Wed, 20 Jun 2012 09:43:10 +0200 Subject: [SciPy-Dev] Adding new wrappers (dassl, daspk version 2, cvode version 2.7.0) to scipy.integrate? In-Reply-To: References: Message-ID: As co-author of scikits.odes, let me mention some things. Sundials has been wrapped a lot in 2011. Not only by my group, but also others. The main ones are: 1. https://openmodelica.org/ The author of python-sundials (http://code.google.com/p/python-sundials/) uses that now, so this last probably will not be further developed 2. http://www.jmodelica.org/assimulo Very extensive and good bindings. It is however more high-level. These two are targeted for use with modelica compiler, but work outside too using their api. 3. http://pysundials.sourceforge.net Not developed at the moment. I contributed some patches there, but they are happy with the current code, and concentrate on what they model. No time/funds to keep working on it. 4. our own scikit.odes. Very low level 1-1 mapping of sundials, combined with a higher level API to quickly start solving a problem. It was nice to work on that, now we use it is our models. Also for us, funding and time are uncertain in the near future. Our use case is solving PDE using method of lines (ODE), which is why we are not 100% happy with the approaches that come from other fields. 5. CasADi, http://sourceforge.net/apps/trac/casadi/ 6. DAE Tools, http://daetools.sourceforge.net/w/index.php/Main_Page by using pyDAE My main problem is that using some of these gives a full simulation environment, which is for many people much more than what you want. As a consequence, they are daunting for a master thesis student or somebody else less into modelling to use for his problem. Integration of sundials in scipy in some way with a basic interface would solve that. Benny 2012/6/20 Geoff Oxberry > Claus, > > What made me skittish about the API documentation of Assimulo is when you > describe the CVODE in the "Documentation contents" page of your > documentation (http://www.jmodelica.org/assimulo) as an "explicit > solver". That, to me, seemed "non-standard". > > After an initial moment of confusion, I realized that you were instead > talking about explicit ordinary differential equations, not explicit > methods for integrating ODEs. (I believe my comments on the Trac Ticket > reflect this.) As someone who's seen enough software thrown out there by > people who either don't maintain it, or don't document it, I get very > skeptical when I see something that looks out of place in documentation. > Anyone who jumps to the Assimulo home page will immediately realize the > distinction you're making; I just happened to type in "assimulo > documentation" because I had already been recommended the package by a user > on Computational Science Stack Exchange (of which I am a moderator). > > When I downloaded your project to compile the Trac ticket report, I did > notice that your project is among the more actively maintained wrappers of > Sundials out there, and receiving an e-mail from you is a pleasant > surprise. I also appreciate that your project has documentation. Of the six > wrappers to Sundials or DASSL that I know of out there, Assimulo looks like > it's the best documented. > > Recently, I attempted to integrate a stiff system of ODEs derived from the > GRI-Mech 3.0 combustion chemistry model with DVODE, varying the absolute > tolerances from 1e-9 to 1e-20 and the relative tolerances from 1e-6 to > 1e-15. For whatever reason, I don't obtain the expected solution behavior, > but I do obtain the expected solution behavior with the MATLAB wrapper to > CVODE, and I also obtain this behavior with a modified version of DASSL > (that simulation was programmed in Fortran 90). Perhaps I should give > Assimulo's wrapper a try, especially since I've found that IDA tends to > work well on my problems also. > > That said, I still believe that DVODE in scipy.integrate should be > supplemented (or even replaced) by CVODE. DVODE has had a long history of > successful use in combustion chemistry, but CVODE, DASSL, DASPK, and IDA > seem to have superior stepping heuristics for these sorts of problems, > based on my 5 or so years of experience working on them. > > Thank you for your response, > > Geoff > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From goxberry at gmail.com Wed Jun 20 08:46:58 2012 From: goxberry at gmail.com (Geoff Oxberry) Date: Wed, 20 Jun 2012 08:46:58 -0400 Subject: [SciPy-Dev] Adding new wrappers (dassl, daspk version 2, cvode version 2.7.0) to scipy.integrate? In-Reply-To: References: Message-ID: <5559489F-06C4-48EC-AFDD-F34C280F5D8F@gmail.com> Benny, To your list of wrappers, I'd add petsc4py. I like PETSc a lot, but right now, the petsc4py documentation only consists of some examples (and these don't cover all of the PETSc API, which is quite large). > My main problem is that using some of these gives a full simulation environment, which is for many people much more than what you want. As a consequence, they are daunting for a master thesis student or somebody else less into modelling to use for his problem. > > Integration of sundials in scipy in some way with a basic interface would solve that. That's my main motivation. I agree with you that the some of the interfaces you've mentioned aren't terribly simple. In order to be a good, drop-in replacement for MATLAB, there should be a good, full-featured suite of numerical integrators that are easy to use (in the sense that a newcomer who knows how to program in Python can pick up the API in a half-hour or less). Brevity of a good example is as good a proxy as any for how long it will take for someone to learn the API; the longer the example, the more commands there are to learn: - MATLAB's first ode45 example is 7 lines (http://www.mathworks.com/help/techdoc/ref/ode45.html) - scipy.integrate.ode's example is 13 lines - CasADi's SUNDIALS example is ~32 lines (http://casadi.sourceforge.net/tutorials/integration/sundials.pdf) - The scikits.odes example is effectively 9 lines, if you take some liberties (https://github.com/bmcage/odes/blob/master/docs/src/examples/ode/simpleoscillator.py) - Assimulo's most basic example is effectively ~10 lines (http://www.jmodelica.org/assimulo_home/_modules/assimulo/examples/cvode_basic.html#run_example) Given the number of open source ODE solver codes with permissive licenses out there, I think SciPy has the potential to be a lot better than MATLAB when it comes to numerical integrators because it could give users a wide variety of numerical methods (and implementations) to try out through a minimal interface. Geoff On Jun 20, 2012, at 3:43 AM, Benny Malengier wrote: > As co-author of scikits.odes, let me mention some things. Sundials has been wrapped a lot in 2011. Not only by my group, but also others. The main ones are: > > 1. https://openmodelica.org/ The author of python-sundials (http://code.google.com/p/python-sundials/) uses that now, so this last probably will not be further developed > > 2. http://www.jmodelica.org/assimulo Very extensive and good bindings. It is however more high-level. > > These two are targeted for use with modelica compiler, but work outside too using their api. > > 3. http://pysundials.sourceforge.net Not developed at the moment. I contributed some patches there, but they are happy with the current code, and concentrate on what they model. No time/funds to keep working on it. > > 4. our own scikit.odes. Very low level 1-1 mapping of sundials, combined with a higher level API to quickly start solving a problem. It was nice to work on that, now we use it is our models. Also for us, funding and time are uncertain in the near future. Our use case is solving PDE using method of lines (ODE), which is why we are not 100% happy with the approaches that come from other fields. > > 5. CasADi, http://sourceforge.net/apps/trac/casadi/ > > 6. DAE Tools, http://daetools.sourceforge.net/w/index.php/Main_Page by using pyDAE > > My main problem is that using some of these gives a full simulation environment, which is for many people much more than what you want. As a consequence, they are daunting for a master thesis student or somebody else less into modelling to use for his problem. > > Integration of sundials in scipy in some way with a basic interface would solve that. > > Benny > > 2012/6/20 Geoff Oxberry > Claus, > > What made me skittish about the API documentation of Assimulo is when you describe the CVODE in the "Documentation contents" page of your documentation (http://www.jmodelica.org/assimulo) as an "explicit solver". That, to me, seemed "non-standard". > > After an initial moment of confusion, I realized that you were instead talking about explicit ordinary differential equations, not explicit methods for integrating ODEs. (I believe my comments on the Trac Ticket reflect this.) As someone who's seen enough software thrown out there by people who either don't maintain it, or don't document it, I get very skeptical when I see something that looks out of place in documentation. Anyone who jumps to the Assimulo home page will immediately realize the distinction you're making; I just happened to type in "assimulo documentation" because I had already been recommended the package by a user on Computational Science Stack Exchange (of which I am a moderator). > > When I downloaded your project to compile the Trac ticket report, I did notice that your project is among the more actively maintained wrappers of Sundials out there, and receiving an e-mail from you is a pleasant surprise. I also appreciate that your project has documentation. Of the six wrappers to Sundials or DASSL that I know of out there, Assimulo looks like it's the best documented. > > Recently, I attempted to integrate a stiff system of ODEs derived from the GRI-Mech 3.0 combustion chemistry model with DVODE, varying the absolute tolerances from 1e-9 to 1e-20 and the relative tolerances from 1e-6 to 1e-15. For whatever reason, I don't obtain the expected solution behavior, but I do obtain the expected solution behavior with the MATLAB wrapper to CVODE, and I also obtain this behavior with a modified version of DASSL (that simulation was programmed in Fortran 90). Perhaps I should give Assimulo's wrapper a try, especially since I've found that IDA tends to work well on my problems also. > > That said, I still believe that DVODE in scipy.integrate should be supplemented (or even replaced) by CVODE. DVODE has had a long history of successful use in combustion chemistry, but CVODE, DASSL, DASPK, and IDA seem to have superior stepping heuristics for these sorts of problems, based on my 5 or so years of experience working on them. > > Thank you for your response, > > Geoff > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From johan.akesson at modelon.com Thu Jun 21 18:09:16 2012 From: johan.akesson at modelon.com (Johan =?utf-8?b?w4VrZXNzb24=?=) Date: Thu, 21 Jun 2012 22:09:16 +0000 (UTC) Subject: [SciPy-Dev] =?utf-8?q?Adding_new_wrappers_=28dassl=2C_daspk_versi?= =?utf-8?q?on_2=2C=09cvode_version_2=2E7=2E0=29_to_scipy=2Eintegrat?= =?utf-8?q?e=3F?= References: Message-ID: Geoff Oxberry gmail.com> writes: > > I posted a feature request ticket (#1678, http://projects.scipy.org/scipy/ticket/1678) about this on > the SciPy Dev Wiki, and Ralf Gommers suggested that I follow up on the scipy-dev mailing list. > > To recap the gist of my ticket, there are problems for which DVODE (currently wrapped in > scipy.integrate.ode) are inadequate, such as in computational combustion. It would be helpful to > researchers in combustion to wrap updated numerical integration libraries (such as DASSL, DASPK, and > CVODE); DAE solver wrappers would also be very helpful (technically, DASSL and DASPK are DAE solvers, but > one thing at a time here). > > Since some of these wrappers already exist in scikits.odes, would it be possible to merge them into > scipy.integrate.ode? If so, in the process, could the Sundials wrappers also be updated to operate with > Sundials version 2.5.0 instead of Sundials version 2.4.0? The newest version of Sundials contains only > minor changes to the API (a few integer arguments are changed to long integers to accommodate larger > problem sizes), and several bugs are fixed. > > There have been many e-mails on the Sundials-users mailing list requesting a good, well-maintained and > supported Python interface to Sundials with a reputable institution behind it. It looks like > scikits.odes is the best-maintained Python interface out there, since pysundials and python-sundials > are no longer maintained, and the Assimulo documentation uses some nomenclature that is nonstandard in > numerical analysis (which makes me doubt the software). Furthermore, I know that users in the combustion > community, frustrated with this situation, have implemented their own wrappers to such solvers (see for > instance, https://github.com/jwallen/PyDAS, written by a colleague of mine). > > I am willing to help with this effort after I graduate (I am in the process of finishing my PhD) in a few months. > > Geoff > Hi Geoff, I'm involved in the Assimulo development, and since Assimulo is mentioned in the post, I'd like to contribute some comments to the discussion. I think that Assimulo matches well the requirements mentioned in Geoff's post. The project is backed by an environment where departments at Lund University, Sweden, (notably the Center for Mathematical Sciences (www.maths.lth.se) and the Lund Center for Control of Complex Systems (www.lccc.lth.se)) collaborates with Modelon, a Lund-based company in the simulation and optimization industry. The project benefits from a mix contributions from academic research, ensuring sound numerics, and industrial applications, ensuring scalability to real world problems. The institutions involved in the development are committed to long term development and maintenance, which is one of the key differentiators to ensure quality in OSS projects. >From early on, the goal of the Assimulo project has been to provide an easy to use and well documented front-end to a variety of integration algorithms ? in addition to the Sundials solvers, we recently added interfaces to Hairer's Dopri5 and Radau codes, ODASSL and the Glimda solver. Additional solvers are planned for inclusion in the future. There was a publication recently that talks about Assimulo at the 2nd Joint International Conference on Multibody System Dynamics: http://www.control.lth.se/Publication/and_etal2012imsd.html. Through the Python-package PyFMI (www.pyfmi.org), Assimulo is conveniently connected to a wide range of state of the art tools for physical modeling, including AMESim, Dymola, JModelica.org, OpenModelica and SimulationX. The PyFMI package is based on the Functional Mock-up Interface Standard (www.fmi-standard.org), which is supported by a wide range of simulation tools. All input on how to improve Assimulo is very welcome! Best regards /Johan From goxberry at gmail.com Thu Jun 21 20:57:01 2012 From: goxberry at gmail.com (Geoff Oxberry) Date: Thu, 21 Jun 2012 20:57:01 -0400 Subject: [SciPy-Dev] Adding new wrappers (dassl, daspk version 2, cvode version 2.7.0) to scipy.integrate? In-Reply-To: References: Message-ID: <390E4DF1-9984-477E-BA28-F222EE879FD8@gmail.com> Johan, Assimulo indeed does many of the things I've mentioned in the post. I have had trouble installing Assimulo on my machine, so I haven't had a chance to test the software myself, but it is one candidate project that I believe does a lot of things well (there is a lot of documentation, there are organizations that back the software, and there are interfaces to many ODE and DAE solvers). It does not have a permissive (that is, BSD or MIT/X11 or similar) license, which means that other Python package developers may have trouble using it in their work. For instance, licensing issues are one reason my colleagues decided to roll-their-own DASSL wrapper. For my work right now, that's not a problem at all, because I'm not building a library; I just want to release the source code of my case studies so that others can reproduce my results. But I think when it would be beneficial to the computational science community to have these sorts of wrappers in SciPy, because they are so useful in the types of simulations Benny mentions in his scikits.odes README (solving ODEs arising from method of lines discretizations of PDEs) and in other types of problems (combustion chemistry, control problems with fast and slow time scales). In turn, developers and researchers working in these application areas would be able to use SciPy to build applications and other software libraries, increasing SciPy's user base. Geoff On Jun 21, 2012, at 6:09 PM, Johan ?kesson wrote: > > Geoff Oxberry gmail.com> writes: > >> >> I posted a feature request ticket (#1678, > http://projects.scipy.org/scipy/ticket/1678) about this on >> the SciPy Dev Wiki, and Ralf Gommers suggested that I follow up on the > scipy-dev mailing list. >> >> To recap the gist of my ticket, there are problems for which DVODE (currently > wrapped in >> scipy.integrate.ode) are inadequate, such as in computational combustion. It > would be helpful to >> researchers in combustion to wrap updated numerical integration libraries > (such as DASSL, DASPK, and >> CVODE); DAE solver wrappers would also be very helpful (technically, DASSL and > DASPK are DAE solvers, but >> one thing at a time here). >> >> Since some of these wrappers already exist in scikits.odes, would it be > possible to merge them into >> scipy.integrate.ode? If so, in the process, could the Sundials wrappers also > be updated to operate with >> Sundials version 2.5.0 instead of Sundials version 2.4.0? The newest version > of Sundials contains only >> minor changes to the API (a few integer arguments are changed to long integers > to accommodate larger >> problem sizes), and several bugs are fixed. >> >> There have been many e-mails on the Sundials-users mailing list requesting a > good, well-maintained and >> supported Python interface to Sundials with a reputable institution behind it. > It looks like >> scikits.odes is the best-maintained Python interface out there, since > pysundials and python-sundials >> are no longer maintained, and the Assimulo documentation uses some > nomenclature that is nonstandard in >> numerical analysis (which makes me doubt the software). Furthermore, I know > that users in the combustion >> community, frustrated with this situation, have implemented their own wrappers > to such solvers (see for >> instance, https://github.com/jwallen/PyDAS, written by a colleague of mine). >> >> I am willing to help with this effort after I graduate (I am in the process of > finishing my PhD) in a few months. >> >> Geoff >> > > Hi Geoff, > > I'm involved in the Assimulo development, and since Assimulo is mentioned in the > post, I'd like to contribute some comments to the discussion. I think that > Assimulo matches well the requirements mentioned in Geoff's post. The project is > backed by an environment where departments at Lund University, Sweden, (notably > the Center for Mathematical Sciences (www.maths.lth.se) and the Lund Center for > Control of Complex Systems (www.lccc.lth.se)) collaborates with Modelon, a > Lund-based company in the simulation and optimization industry. The project > benefits from a mix contributions from academic research, ensuring sound > numerics, and industrial applications, ensuring scalability to real world > problems. The institutions involved in the development are committed to long > term development and maintenance, which is one of the key differentiators to > ensure quality in OSS projects. > > From early on, the goal of the Assimulo project has been to provide an easy to > use and well documented front-end to a variety of integration algorithms ? in > addition to the Sundials solvers, we recently added interfaces to Hairer's > Dopri5 and Radau codes, ODASSL and the Glimda solver. Additional solvers are > planned for inclusion in the future. There was a publication recently that talks > about Assimulo at the 2nd Joint International Conference on Multibody System > Dynamics: http://www.control.lth.se/Publication/and_etal2012imsd.html. > > Through the Python-package PyFMI (www.pyfmi.org), Assimulo is conveniently > connected to a wide range of state of the art tools for physical modeling, > including AMESim, Dymola, JModelica.org, OpenModelica and SimulationX. The PyFMI > package is based on the Functional Mock-up Interface Standard > (www.fmi-standard.org), which is supported by a wide range of simulation tools. > > All input on how to improve Assimulo is very welcome! > > Best regards > /Johan > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From pierre.haessig at crans.org Mon Jun 25 11:14:42 2012 From: pierre.haessig at crans.org (Pierre Haessig) Date: Mon, 25 Jun 2012 17:14:42 +0200 Subject: [SciPy-Dev] interp2d bounds checking Message-ID: <4FE88062.1000103@crans.org> Hi, I just have a small question about scipy.interpolate.interp2d I noticed that bounds checking when calling the interpolation instance seems to be ineffective. I found a ticket about this issue, but not much activity around it. http://projects.scipy.org/scipy/ticket/1072 I looked at the code and at the documentation ( http://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.interp2d.html#scipy.interpolate.interp2d https://raw.github.com/scipy/scipy/master/scipy/interpolate/interpolate.py ) and it seems that the `bounds_error` argument is never used in the initialization of the interp2d object (unlike inter1d). I'm not at all a specialist of 2D interpolation but I guess that bounds checking in 2D is not as easy as in 1D... Maybe, if there is no easy way to check bounds in interp2d, we should just add a "big warning" in the docstring stating that `bounds_error` is not currently used (and therefore `fill_value` as well). What do you think ? Best, Pierre about docstring editing, I didn't find the "Edit page" link on interp2d page. Did I miss something ? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature URL: From pierre.haessig at crans.org Tue Jun 26 11:00:45 2012 From: pierre.haessig at crans.org (Pierre Haessig) Date: Tue, 26 Jun 2012 17:00:45 +0200 Subject: [SciPy-Dev] UnivariateSpline evaluated on a ND array Message-ID: <4FE9CE9D.8050107@crans.org> Hi, While playing a bit with UnivariateSpline... >>> from scipy.interpolate import UnivariateSpline >>> s = UnivariateSpline([0,1], [0,1],k=1) ...I noticed that the evaluation fails if the input is not flat: Fine with 1D input : >>> s(np.linspace(0,1,100)) array([ 0. , 0.11111111, 0.22222222, 0.33333333, 0.44444444, 0.55555556, 0.66666667, 0.77777778, 0.88888889, 1. ]) but fails with 2D input : >>> s(np.linspace(0,1,10).reshape(5,2)) [...] /usr/lib/python2.7/dist-packages/scipy/interpolate/fitpack.pyc in splev(x, tck, der, ext) [...] --> 548 y, ier =_fitpack._spl_(x, der, t, c, k, ext) [...] ValueError: object too deep for desired array Do you think it would be a good move to add *inside* the UnivariateSpline.__call__ method a small machinery which performs the "flatten -> spline eval -> unflatten" operation ? On the other hand, the Docstring of scipy.interpolate.splev which called within UnivariateSpline states explicately that input should be 1D. If this limitation should be propagated to UnivariateSpline, then it should be clarified in the Docstring of UnivariateSpline.__call__ (https://github.com/scipy/scipy/blob/master/scipy/interpolate/fitpack2.py#L215 if I'm right) What do you think is the best solution ? Best, Pierre -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature URL: From lists at hilboll.de Tue Jun 26 14:52:10 2012 From: lists at hilboll.de (Andreas Hilboll) Date: Tue, 26 Jun 2012 20:52:10 +0200 Subject: [SciPy-Dev] UnivariateSpline evaluated on a ND array In-Reply-To: <4FE9CE9D.8050107@crans.org> References: <4FE9CE9D.8050107@crans.org> Message-ID: <4FEA04DA.4010203@hilboll.de> > Hi, > > While playing a bit with UnivariateSpline... > >>> from scipy.interpolate import UnivariateSpline > > >>> s = UnivariateSpline([0,1], [0,1],k=1) > > > ...I noticed that the evaluation fails if the input is not flat: > > > Fine with 1D input : > > >>> s(np.linspace(0,1,100)) > array([ 0. , 0.11111111, 0.22222222, 0.33333333, 0.44444444, > 0.55555556, 0.66666667, 0.77777778, 0.88888889, 1. ]) > > > but fails with 2D input : > >>> s(np.linspace(0,1,10).reshape(5,2)) > [...] > /usr/lib/python2.7/dist-packages/scipy/interpolate/fitpack.pyc in > splev(x, tck, der, ext) > [...] > --> 548 y, ier =_fitpack._spl_(x, der, t, c, k, ext) > [...] > ValueError: object too deep for desired array > > > Do you think it would be a good move to add *inside* the > UnivariateSpline.__call__ method a small machinery which performs the > "flatten -> spline eval -> unflatten" operation ? +1 From arvind.sbia at gmail.com Thu Jun 28 08:47:45 2012 From: arvind.sbia at gmail.com (Arvind Rao) Date: Thu, 28 Jun 2012 08:47:45 -0400 Subject: [SciPy-Dev] Equidistant Sphere Points Message-ID: Hi All, For model fitting and numerical integration, I have developed a routine that generates N equidistant points on the sphere, by way of an intrinsic gradient flow. Now, I would like to port it over to python. Would there be interest in adding such a function to SciPy? Sincerely, -- *Arvind S. Rao, PhD* Section of Biomedical Image Analysis Department of Radiology University of Pennsylvania 3600 Market Street, Suite 380 Philadelphia, PA 19104 -------------- next part -------------- An HTML attachment was scrubbed... URL: From josh.k.lawrence at gmail.com Thu Jun 28 13:01:17 2012 From: josh.k.lawrence at gmail.com (Josh Lawrence) Date: Thu, 28 Jun 2012 13:01:17 -0400 Subject: [SciPy-Dev] scipy.signal.normalize problems? In-Reply-To: References: <3D094679-4BC5-4647-9739-776DA7539CBB@gmail.com> <95DB673F-E1B6-4C9D-9050-39D8A412D52F@gmail.com> <51356F2E-A648-400E-855B-D919B44330EA@gmail.com> Message-ID: <72BCC085-58FC-48E0-80C1-90C45EAB81D6@gmail.com> I updated the PR to link to this discussion and I included a unit-test. I applied the changes to the current scipy-master and running scipy.signal.test() (where the new test is located) produces no errors or failures. Will this be backported to 0.11? --Josh On May 30, 2012, at 1:38 PM, Ralf Gommers wrote: > > > On Wed, May 30, 2012 at 7:20 PM, Josh Lawrence wrote: > Okay, > > I made a pull request: > > https://github.com/scipy/scipy/pull/233 > > Do I need to file a ticket on Trac or is the PR good enough? > > The PR should be good enough. Two suggestions though for the PR: > - in a comment, link to this discussion (gmane archive) > - please add a unit test. You already have an example that didn't work before, and now gives correct results (or at least matches Matlab). You can convert that into a test, with the output of normalize() checked against the hardcoded correct numerical values. > > Thanks, > Ralf > > > --Josh > > On May 29, 2012, at 4:08 PM, Ralf Gommers wrote: > >> >> >> On Mon, May 21, 2012 at 9:11 PM, Josh Lawrence wrote: >> If I change the allclose lines to have >> >> allclose(0, outb[:,0], atol=1e-14) >> >> it works. I think that is what the original goal was, anyways. From the documentation of allclose, what I have written above result in ensuring >> >> np.abs(0 - outb[:,0]) > atol + rtol * np.abs(outb[:,0]) >> >> with rtol defaulting to 1e-5. I'm still not sure about how things have been written for the 'b' argument of normalize being rank-2, so I can't guarantee that my fix makes things work for that. Should I file a bug report/submit a patch/send a git pull request, etc? >> >> Sounds like you understood the issue and have a fix that does what you expect (matches Matlab), so a pull request would be nice. >> >> For the rank-2 thing, you can just note it on the PR if you can't figure it out. >> >> Ralf >> >> >> Cheers, >> >> Josh Lawrence >> >> On May 21, 2012, at 2:45 PM, Josh Lawrence wrote: >> >> > Hey all, >> > >> > I've been having some problems designing a Chebyshev filter and I think I have narrowed down the hang-up to scipy.signal.normalize. I think what's going on in my case is that line 286 of filter_design.py (the first allclose call in the normalize function) is producing a false positive. Here's the function definition: >> > >> > def normalize(b, a): >> > """Normalize polynomial representation of a transfer function. >> > >> > If values of b are too close to 0, they are removed. In that case, a >> > BadCoefficients warning is emitted. >> > """ >> > b, a = map(atleast_1d, (b, a)) >> > if len(a.shape) != 1: >> > raise ValueError("Denominator polynomial must be rank-1 array.") >> > if len(b.shape) > 2: >> > raise ValueError("Numerator polynomial must be rank-1 or" >> > " rank-2 array.") >> > if len(b.shape) == 1: >> > b = asarray([b], b.dtype.char) >> > while a[0] == 0.0 and len(a) > 1: >> > a = a[1:] >> > outb = b * (1.0) / a[0] >> > outa = a * (1.0) / a[0] >> > if allclose(outb[:, 0], 0, rtol=1e-14): <------------------ Line 286 >> > warnings.warn("Badly conditioned filter coefficients (numerator): the " >> > "results may be meaningless", BadCoefficients) >> > while allclose(outb[:, 0], 0, rtol=1e-14) and (outb.shape[-1] > 1): >> > outb = outb[:, 1:] >> > if outb.shape[0] == 1: >> > outb = outb[0] >> > return outb, outa >> > >> > I marked line 286. If I reproduce all the steps carried out by scipy.signal.iirdesign, I end up with a (b, a) pair which results of scipy.signal.lp2lp and looks like this: >> > >> > In [106]: b_lp2 >> > Out[106]: array([ 1.55431359e-06+0.j]) >> > >> > In [107]: a_lp2 >> > Out[107]: >> > array([ 1.00000000e+00 +0.00000000e+00j, >> > 3.46306104e-01 -2.01282794e-16j, >> > 2.42572185e-01 -6.08207573e-17j, >> > 5.92946943e-02 +0.00000000e+00j, >> > 1.82069156e-02 +5.55318531e-18j, >> > 2.89328123e-03 +0.00000000e+00j, >> > 4.36566281e-04 -2.95766719e-19j, >> > 3.50842810e-05 -3.19180568e-20j, 1.64641246e-06 -1.00966301e-21j]) >> > >> > scipy.signal.iirdesign takes b_lp2, a_lp2 (my local variable names to keep track of what's going on) and runs them through scipy.signal.bilinear (in filter_design.py bilinear is called on line 624 within iirfilter. iirdesign calls iirfilter which calls bilinear). Inside bilinear, normalize is called on line 445. I've made my own class with bilinear copied and pasted from filter_design.py to test things. In bilinear, the input to normalize is given by >> > >> > b = [ 1.55431359e-06 1.24345087e-05 4.35207804e-05 8.70415608e-05 >> > 1.08801951e-04 8.70415608e-05 4.35207804e-05 1.24345087e-05 >> > 1.55431359e-06] >> > a = [ 72269.02590913 -562426.61430468 1918276.19173089 -3745112.83646825 >> > 4577612.13937628 -3586970.61385926 1759651.18184723 -494097.93515708 >> > 60799.46134722] >> > >> > In normalize, right before the allclose() call, outb is defined by >> > >> > outb = [[ 2.15073272e-11 1.72058618e-10 6.02205162e-10 1.20441032e-09 >> > 1.50551290e-09 1.20441032e-09 6.02205162e-10 1.72058618e-10 >> > 2.15073272e-11]] >> > >> > From what I read of the function normalize, it should only evaluate true if all of the coefficients in outb are smaller than 1e-14. However, that's not what is going on. I have access to MATLAB and if I go through the same design criteria to design a chebyshev filter, I get the following: >> > >> > b = >> > >> > 1.0e-08 * >> > >> > Columns 1 through 5 >> > >> > 0.002150733144728 0.017205865157826 0.060220528052392 0.120441056104784 0.150551320130980 >> > >> > Columns 6 through 9 >> > >> > 0.120441056104784 0.060220528052392 0.017205865157826 0.002150733144728 >> > >> > which matches up rather well for several significant figures. >> > >> > I apologize if this is not clearly explained, but I'm not sure what to do. I tried messing around with the arguments to allclose (switching it to be allclose(0, outb[:,0], ...), or changing the keyword from rtol to atol). I am also not sure why normalize is setup to run on rank-2 arrays. I looked through filter_design.py and none of the functions contained in it send a rank-2 array to normalize from what I can tell. Any thoughts? >> > >> > Cheers, >> > >> > Josh Lawrence >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Thu Jun 28 13:06:54 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 28 Jun 2012 19:06:54 +0200 Subject: [SciPy-Dev] scipy.signal.normalize problems? In-Reply-To: <72BCC085-58FC-48E0-80C1-90C45EAB81D6@gmail.com> References: <3D094679-4BC5-4647-9739-776DA7539CBB@gmail.com> <95DB673F-E1B6-4C9D-9050-39D8A412D52F@gmail.com> <51356F2E-A648-400E-855B-D919B44330EA@gmail.com> <72BCC085-58FC-48E0-80C1-90C45EAB81D6@gmail.com> Message-ID: On Thu, Jun 28, 2012 at 7:01 PM, Josh Lawrence wrote: > I updated the PR to link to this discussion and I included a unit-test. I > applied the changes to the current scipy-master and running > scipy.signal.test() (where the new test is located) produces no errors or > failures. > > Will this be backported to 0.11? > Could one of the scipy.signal maintainers please have a look at this fix? https://github.com/scipy/scipy/pull/233 Backporting to 0.11.x is no problem. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From josh.k.lawrence at gmail.com Thu Jun 28 15:15:17 2012 From: josh.k.lawrence at gmail.com (Josh Lawrence) Date: Thu, 28 Jun 2012 15:15:17 -0400 Subject: [SciPy-Dev] scipy.signal.normalize problems? In-Reply-To: References: <3D094679-4BC5-4647-9739-776DA7539CBB@gmail.com> <95DB673F-E1B6-4C9D-9050-39D8A412D52F@gmail.com> <51356F2E-A648-400E-855B-D919B44330EA@gmail.com> <72BCC085-58FC-48E0-80C1-90C45EAB81D6@gmail.com> Message-ID: <9DE6F325-77C5-4626-884F-E615601E29E2@gmail.com> Sorry, I had my git repository all messed up. Here is the new pull request. https://github.com/scipy/scipy/pull/259 --Josh On Jun 28, 2012, at 1:06 PM, Ralf Gommers wrote: > > > On Thu, Jun 28, 2012 at 7:01 PM, Josh Lawrence wrote: > I updated the PR to link to this discussion and I included a unit-test. I applied the changes to the current scipy-master and running scipy.signal.test() (where the new test is located) produces no errors or failures. > > Will this be backported to 0.11? > > Could one of the scipy.signal maintainers please have a look at this fix? > https://github.com/scipy/scipy/pull/233 > > Backporting to 0.11.x is no problem. > > Ralf > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Thu Jun 28 16:10:14 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 28 Jun 2012 13:10:14 -0700 Subject: [SciPy-Dev] Equidistant Sphere Points In-Reply-To: References: Message-ID: Hi, On Thu, Jun 28, 2012 at 5:47 AM, Arvind Rao wrote: > Hi All, > > For model fitting and numerical integration, I have developed a routine that > generates N equidistant > points on the sphere, by way of an intrinsic gradient flow. Now, I would > like to port it > over to python. > > Would there be interest in adding such a function to SciPy? I suppose you might be working on diffusion imaging? The dipy [1] team has been talking quite a bit about different algorithms for sphere points ; maybe also ask on the nipy mailing list? [2] Cheers, Matthew [1] https://github.com/nipy/dipy [2] http://mail.scipy.org/mailman/listinfo/nipy-devel