From ralf.gommers at googlemail.com Sun Jul 1 05:21:32 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 1 Jul 2012 11:21:32 +0200 Subject: [SciPy-Dev] interp2d bounds checking In-Reply-To: <4FE88062.1000103@crans.org> References: <4FE88062.1000103@crans.org> Message-ID: On Mon, Jun 25, 2012 at 5:14 PM, Pierre Haessig wrote: > Hi, > > I just have a small question about scipy.interpolate.interp2d > > I noticed that bounds checking when calling the interpolation instance > seems to be ineffective. > I found a ticket about this issue, but not much activity around it. > http://projects.scipy.org/scipy/ticket/1072 > > I looked at the code and at the documentation > ( > > http://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.interp2d.html#scipy.interpolate.interp2d > > https://raw.github.com/scipy/scipy/master/scipy/interpolate/interpolate.py) > > and it seems that the `bounds_error` argument is never used in the > initialization of the interp2d object (unlike inter1d). > I'm not at all a specialist of 2D interpolation but I guess that bounds > checking in 2D is not as easy as in 1D... > > Maybe, if there is no easy way to check bounds in interp2d, we should > just add a "big warning" in the docstring stating that `bounds_error` is > not currently used (and therefore `fill_value` as well). > What do you think ? > Bounds checking on a 2-D grid isn't really more difficult than on a 1-D grid. It looks like someone just copied the interp1d interface and then ran out of time or energy to fully implement it. Would be good if someone feels like implementing the missing bits, and otherwise we should at least document that right now out-of-bounds input simply raises an error (I didn't check which one). > Best, > Pierre > > about docstring editing, I didn't find the "Edit page" link on interp2d > page. Did I miss something ? > > I see an edit button at http://docs.scipy.org/scipy/docs/scipy.interpolate.interpolate.interp2d/. Were you trying to find it there, or on the link you gave above (=built docs). Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Sun Jul 1 14:40:16 2012 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 01 Jul 2012 20:40:16 +0200 Subject: [SciPy-Dev] interp2d bounds checking In-Reply-To: References: <4FE88062.1000103@crans.org> Message-ID: 01.07.2012 11:21, Ralf Gommers kirjoitti: [clip] > Bounds checking on a 2-D grid isn't really more difficult than on a 1-D > grid. It looks like someone just copied the interp1d interface and then > ran out of time or energy to fully implement it. > > Would be good if someone feels like implementing the missing bits, and > otherwise we should at least document that right now out-of-bounds input > simply raises an error (I didn't check which one). IMO, interp2d should be deprecated. It is equivalent to SmoothBivariateSpline with s=0. It's sort of a almost-name-alias for a Matlab function, but it works differently --- scattered data interpolation vs. grid interpolation. However, as FITPACK's 2-D scattered data splines not so rarely explode with real data, I don't think this "replacement function" does its work properly, so it should be deprecated. If the users want these 2-D splines, it's better to make them ask for them explicitly... We can maybe think about adding a deprecation category "deprecated and removed from documentation, but we'll keep it around indefinitely for backwards compatibility". Move to e.g. to a new file '_deprecated.py' to keep the moldy code separate. *** What's needed is a proper grid-base interpolation. It would be rather easy to write a simple wrapper around ndimage.map_coordinates to get general spline interpolation. Such interpolants are also easy to construct up from the polynomial interpolants in Scipy. If someone is looking for something to do: *here* would be a nice, a relatively easy, and very likely a very useful project to work on. *** Also, where interp2d works, griddata should work better in most use cases. I've also been thinking about enlarging the griddata syntax from the possibly somewhat unfriendly griddata((x, y), z, (xi, yi), **kw) griddata((x, y, z), u, (xi, yi, zi), **kw) to the variable argument one griddata(x, y, z, xi, yi, **kw) griddata(x, y, z, u, xi, yi, zi, **kw) which may be more familiar to users. Thoughts? Pauli From thouis at gmail.com Thu Jul 5 09:10:43 2012 From: thouis at gmail.com (Thouis (Ray) Jones) Date: Thu, 5 Jul 2012 15:10:43 +0200 Subject: [SciPy-Dev] PR: ENH: rewrite ndimage.label() in Cython Message-ID: I just submitted this PR: https://github.com/scipy/scipy/pull/261 (summary below) I appreciate any feedback. Thanks, Ray Jones (note: I wrote "type templating" below when I meant to write "fused types", but didn't notice until just now). Summary: This reimplements ndimage.label() in Cython, with the following results: + easier to read code + faster (roughly 3x on my MacBook Pro) + output array can be nearly any numpy type (not just int32) + if passed in, output array is used for scratch space (minimal overhead) Downsides: - Uses type templating, so requires a dev version of Cython as of 2012-07 (Cython 0.17+ should work) - The type templating results in a very large C file (there's a small function that gets instantiated N^2 times for the N types the function takes as input/output). New tests have been added to ensure that label() returns the same values as before, and to test new functionality (in-place operation). From ralf.gommers at googlemail.com Thu Jul 5 14:25:48 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 5 Jul 2012 20:25:48 +0200 Subject: [SciPy-Dev] Cython as build dependency, file/dll size and current issues Message-ID: Hi all, On https://github.com/scipy/scipy/pull/261 the problem with large size of generated C files from Cython came up again, and Matthew suggested to add Cython as a build time dependency. He also pointed out that this was discussed before, with most people being in favor: http://mail.scipy.org/pipermail/scipy-dev/2009-November/013272.html http://mail.scipy.org/pipermail/scipy-dev/2009-November/013308.html We discussed the same issue on https://github.com/scipy/scipy/pull/211recently, and also the size of the binary. This is probably also the right moment to point out other recent Cython issues we've had: 1. A memoryview issue with Python 2.4, either a Cython or Numpy bug: https://github.com/numpy/numpy/pull/307 2. We had to manually patch the generated C files when using Cython 0.16, to make them work with MinGW: http://projects.scipy.org/scipy/ticket/1673 3. According to Ray, there's also an indexing bug in Cython 0.16 which requires to use 0.17-dev for https://github.com/scipy/scipy/pull/261 I think it's clear that PR's like #261 above (Ray's ndimage.label rewrite) are in principle a good thing: faster and more general code which is easier to maintain. Now the question is what to do though. Here's some options that I see: a) Keep things as is for now. Accept large file/binary sizes. Manually patch the generated C if necessary. b) Keep things as is for now. Either go back to Cython 0.15, or bump required numpy version to latest dev version to not have to manually patch the generated C files. c) Keep things as they are now, without accepting too large file/binary sizes. To be defined what too large. Means we can't get the full benefits of fused types for example. d) Move to Cython as a build dependency. Write down the required versions and incompatibilities in the docs. e) Include a Cython version in the scipy git repo, patch it to solve the above issues 2 and 3 (and any other ones that come along). f) Some combination of the above. g) Any other options? Anyone have an opinion on which way to go? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Thu Jul 5 14:31:54 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 5 Jul 2012 11:31:54 -0700 Subject: [SciPy-Dev] Cython as build dependency, file/dll size and current issues In-Reply-To: References: Message-ID: Hi, On Thu, Jul 5, 2012 at 11:25 AM, Ralf Gommers wrote: > Hi all, > > On https://github.com/scipy/scipy/pull/261 the problem with large size of > generated C files from Cython came up again, and Matthew suggested to add > Cython as a build time dependency. He also pointed out that this was > discussed before, with most people being in favor: > http://mail.scipy.org/pipermail/scipy-dev/2009-November/013272.html > http://mail.scipy.org/pipermail/scipy-dev/2009-November/013308.html > We discussed the same issue on https://github.com/scipy/scipy/pull/211 > recently, and also the size of the binary. > > This is probably also the right moment to point out other recent Cython > issues we've had: > 1. A memoryview issue with Python 2.4, either a Cython or Numpy bug: > https://github.com/numpy/numpy/pull/307 > 2. We had to manually patch the generated C files when using Cython 0.16, to > make them work with MinGW: http://projects.scipy.org/scipy/ticket/1673 > 3. According to Ray, there's also an indexing bug in Cython 0.16 which > requires to use 0.17-dev for https://github.com/scipy/scipy/pull/261 > > I think it's clear that PR's like #261 above (Ray's ndimage.label rewrite) > are in principle a good thing: faster and more general code which is easier > to maintain. Now the question is what to do though. Here's some options that > I see: > > a) Keep things as is for now. Accept large file/binary sizes. Manually patch > the generated C if necessary. > b) Keep things as is for now. Either go back to Cython 0.15, or bump > required numpy version to latest dev version to not have to manually patch > the generated C files. > c) Keep things as they are now, without accepting too large file/binary > sizes. To be defined what too large. Means we can't get the full benefits of > fused types for example. > d) Move to Cython as a build dependency. Write down the required versions > and incompatibilities in the docs. > e) Include a Cython version in the scipy git repo, patch it to solve the > above issues 2 and 3 (and any other ones that come along). > f) Some combination of the above. > g) Any other options? Am I right in thinking that Cython 0.17dev will generate usable C files without patching? Cheers, Matthew From ralf.gommers at googlemail.com Thu Jul 5 14:35:27 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 5 Jul 2012 20:35:27 +0200 Subject: [SciPy-Dev] Cython as build dependency, file/dll size and current issues In-Reply-To: References: Message-ID: On Thu, Jul 5, 2012 at 8:31 PM, Matthew Brett wrote: > Hi, > > On Thu, Jul 5, 2012 at 11:25 AM, Ralf Gommers > wrote: > > Hi all, > > > > On https://github.com/scipy/scipy/pull/261 the problem with large size > of > > generated C files from Cython came up again, and Matthew suggested to add > > Cython as a build time dependency. He also pointed out that this was > > discussed before, with most people being in favor: > > http://mail.scipy.org/pipermail/scipy-dev/2009-November/013272.html > > http://mail.scipy.org/pipermail/scipy-dev/2009-November/013308.html > > We discussed the same issue on https://github.com/scipy/scipy/pull/211 > > recently, and also the size of the binary. > > > > This is probably also the right moment to point out other recent Cython > > issues we've had: > > 1. A memoryview issue with Python 2.4, either a Cython or Numpy bug: > > https://github.com/numpy/numpy/pull/307 > > 2. We had to manually patch the generated C files when using Cython > 0.16, to > > make them work with MinGW: http://projects.scipy.org/scipy/ticket/1673 > > 3. According to Ray, there's also an indexing bug in Cython 0.16 which > > requires to use 0.17-dev for https://github.com/scipy/scipy/pull/261 > > > > I think it's clear that PR's like #261 above (Ray's ndimage.label > rewrite) > > are in principle a good thing: faster and more general code which is > easier > > to maintain. Now the question is what to do though. Here's some options > that > > I see: > > > > a) Keep things as is for now. Accept large file/binary sizes. Manually > patch > > the generated C if necessary. > > b) Keep things as is for now. Either go back to Cython 0.15, or bump > > required numpy version to latest dev version to not have to manually > patch > > the generated C files. > > c) Keep things as they are now, without accepting too large file/binary > > sizes. To be defined what too large. Means we can't get the full > benefits of > > fused types for example. > > d) Move to Cython as a build dependency. Write down the required versions > > and incompatibilities in the docs. > > e) Include a Cython version in the scipy git repo, patch it to solve the > > above issues 2 and 3 (and any other ones that come along). > > f) Some combination of the above. > > g) Any other options? > > Am I right in thinking that Cython 0.17dev will generate usable C > files without patching? > Yes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Thu Jul 5 14:57:26 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 5 Jul 2012 11:57:26 -0700 Subject: [SciPy-Dev] Cython as build dependency, file/dll size and current issues In-Reply-To: References: Message-ID: Hi, On Thu, Jul 5, 2012 at 11:35 AM, Ralf Gommers wrote: > > > On Thu, Jul 5, 2012 at 8:31 PM, Matthew Brett > wrote: >> >> Hi, >> >> On Thu, Jul 5, 2012 at 11:25 AM, Ralf Gommers >> wrote: >> > Hi all, >> > >> > On https://github.com/scipy/scipy/pull/261 the problem with large size >> > of >> > generated C files from Cython came up again, and Matthew suggested to >> > add >> > Cython as a build time dependency. He also pointed out that this was >> > discussed before, with most people being in favor: >> > http://mail.scipy.org/pipermail/scipy-dev/2009-November/013272.html >> > http://mail.scipy.org/pipermail/scipy-dev/2009-November/013308.html >> > We discussed the same issue on https://github.com/scipy/scipy/pull/211 >> > recently, and also the size of the binary. >> > >> > This is probably also the right moment to point out other recent Cython >> > issues we've had: >> > 1. A memoryview issue with Python 2.4, either a Cython or Numpy bug: >> > https://github.com/numpy/numpy/pull/307 >> > 2. We had to manually patch the generated C files when using Cython >> > 0.16, to >> > make them work with MinGW: http://projects.scipy.org/scipy/ticket/1673 >> > 3. According to Ray, there's also an indexing bug in Cython 0.16 which >> > requires to use 0.17-dev for https://github.com/scipy/scipy/pull/261 >> > >> > I think it's clear that PR's like #261 above (Ray's ndimage.label >> > rewrite) >> > are in principle a good thing: faster and more general code which is >> > easier >> > to maintain. Now the question is what to do though. Here's some options >> > that >> > I see: >> > >> > a) Keep things as is for now. Accept large file/binary sizes. Manually >> > patch >> > the generated C if necessary. >> > b) Keep things as is for now. Either go back to Cython 0.15, or bump >> > required numpy version to latest dev version to not have to manually >> > patch >> > the generated C files. >> > c) Keep things as they are now, without accepting too large file/binary >> > sizes. To be defined what too large. Means we can't get the full >> > benefits of >> > fused types for example. >> > d) Move to Cython as a build dependency. Write down the required >> > versions >> > and incompatibilities in the docs. >> > e) Include a Cython version in the scipy git repo, patch it to solve the >> > above issues 2 and 3 (and any other ones that come along). >> > f) Some combination of the above. >> > g) Any other options? >> >> Am I right in thinking that Cython 0.17dev will generate usable C >> files without patching? > > > Yes. How about making Cython 0.17 a developer build-time dependency? Meanwhile, making 'python setup.py sdist' dump the c files into the source distribution? Maybe with a nightly development snapshot pushed up to sourceforge or similar? That's what we are doing for dipy (the c files into the sdist): https://github.com/nipy/dipy/blob/master/setup.py#L95 https://github.com/nipy/dipy/blob/master/cythexts.py although a Bento way of doing that would probably be easier to maintain, Cheers, Matthew From ralf.gommers at googlemail.com Thu Jul 5 15:18:42 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 5 Jul 2012 21:18:42 +0200 Subject: [SciPy-Dev] Cython as build dependency, file/dll size and current issues In-Reply-To: References: Message-ID: On Thu, Jul 5, 2012 at 8:57 PM, Matthew Brett wrote: > Hi, > > On Thu, Jul 5, 2012 at 11:35 AM, Ralf Gommers > wrote: > > > > > > On Thu, Jul 5, 2012 at 8:31 PM, Matthew Brett > > wrote: > >> > >> Hi, > >> > >> On Thu, Jul 5, 2012 at 11:25 AM, Ralf Gommers > >> wrote: > >> > Hi all, > >> > > >> > On https://github.com/scipy/scipy/pull/261 the problem with large > size > >> > of > >> > generated C files from Cython came up again, and Matthew suggested to > >> > add > >> > Cython as a build time dependency. He also pointed out that this was > >> > discussed before, with most people being in favor: > >> > http://mail.scipy.org/pipermail/scipy-dev/2009-November/013272.html > >> > http://mail.scipy.org/pipermail/scipy-dev/2009-November/013308.html > >> > We discussed the same issue on > https://github.com/scipy/scipy/pull/211 > >> > recently, and also the size of the binary. > >> > > >> > This is probably also the right moment to point out other recent > Cython > >> > issues we've had: > >> > 1. A memoryview issue with Python 2.4, either a Cython or Numpy bug: > >> > https://github.com/numpy/numpy/pull/307 > >> > 2. We had to manually patch the generated C files when using Cython > >> > 0.16, to > >> > make them work with MinGW: > http://projects.scipy.org/scipy/ticket/1673 > >> > 3. According to Ray, there's also an indexing bug in Cython 0.16 which > >> > requires to use 0.17-dev for https://github.com/scipy/scipy/pull/261 > >> > > >> > I think it's clear that PR's like #261 above (Ray's ndimage.label > >> > rewrite) > >> > are in principle a good thing: faster and more general code which is > >> > easier > >> > to maintain. Now the question is what to do though. Here's some > options > >> > that > >> > I see: > >> > > >> > a) Keep things as is for now. Accept large file/binary sizes. Manually > >> > patch > >> > the generated C if necessary. > >> > b) Keep things as is for now. Either go back to Cython 0.15, or bump > >> > required numpy version to latest dev version to not have to manually > >> > patch > >> > the generated C files. > >> > c) Keep things as they are now, without accepting too large > file/binary > >> > sizes. To be defined what too large. Means we can't get the full > >> > benefits of > >> > fused types for example. > >> > d) Move to Cython as a build dependency. Write down the required > >> > versions > >> > and incompatibilities in the docs. > >> > e) Include a Cython version in the scipy git repo, patch it to solve > the > >> > above issues 2 and 3 (and any other ones that come along). > >> > f) Some combination of the above. > >> > g) Any other options? > >> > >> Am I right in thinking that Cython 0.17dev will generate usable C > >> files without patching? > > > > > > Yes. > > How about making Cython 0.17 a developer build-time dependency? > That's an option. Requiring a dev version will mean broken builds for some of the users that don't read the docs well but simply do "easyinstall cython". I'm not sure how acceptable that is. That still leaves the (mostly orthogonal) question about binary size. I just built Ray's PR, _nd_label.so is 1.4 Mb. For one function. Meanwhile, making 'python setup.py sdist' dump the c files into the > source distribution? Maybe with a nightly development snapshot pushed > up to sourceforge or similar? > > That's what we are doing for dipy (the c files into the sdist): > > https://github.com/nipy/dipy/blob/master/setup.py#L95 > https://github.com/nipy/dipy/blob/master/cythexts.py > > That sounds like a good idea, if we go for a Cython build dependency. Even if we don't, nightly builds would be great. Ralf although a Bento way of doing that would probably be easier to maintain, > > Cheers, > > Matthew > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanderplas at astro.washington.edu Thu Jul 5 15:19:20 2012 From: vanderplas at astro.washington.edu (Jacob VanderPlas) Date: Thu, 05 Jul 2012 12:19:20 -0700 Subject: [SciPy-Dev] Cython as build dependency, file/dll size and current issues In-Reply-To: References: Message-ID: <4FF5E8B8.70602@astro.washington.edu> On 07/05/12 11:57, Matthew Brett wrote: > Hi, [snip] > How about making Cython 0.17 a developer build-time dependency? > > Meanwhile, making 'python setup.py sdist' dump the c files into the > source distribution? Maybe with a nightly development snapshot pushed > up to sourceforge or similar? > > That's what we are doing for dipy (the c files into the sdist): > > https://github.com/nipy/dipy/blob/master/setup.py#L95 > https://github.com/nipy/dipy/blob/master/cythexts.py > > although a Bento way of doing that would probably be easier to maintain, I think this is the best compromise, if we can make it work. The flexibility of fused types is a major plus. Adding a general user-end dependency on cython would be a mistake, IMHO (scipy can be difficult enough to build as it is). This would also allow us to depend on bleeding-edge cython versions for development builds, and hopefully fix some of the by-hand patching issues we've seen in the past. Jake From matthew.brett at gmail.com Thu Jul 5 15:26:20 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 5 Jul 2012 12:26:20 -0700 Subject: [SciPy-Dev] Cython as build dependency, file/dll size and current issues In-Reply-To: References: Message-ID: Hi, On Thu, Jul 5, 2012 at 12:18 PM, Ralf Gommers wrote: > > > On Thu, Jul 5, 2012 at 8:57 PM, Matthew Brett > wrote: >> >> Hi, >> >> On Thu, Jul 5, 2012 at 11:35 AM, Ralf Gommers >> wrote: >> > >> > >> > On Thu, Jul 5, 2012 at 8:31 PM, Matthew Brett >> > wrote: >> >> >> >> Hi, >> >> >> >> On Thu, Jul 5, 2012 at 11:25 AM, Ralf Gommers >> >> wrote: >> >> > Hi all, >> >> > >> >> > On https://github.com/scipy/scipy/pull/261 the problem with large >> >> > size >> >> > of >> >> > generated C files from Cython came up again, and Matthew suggested to >> >> > add >> >> > Cython as a build time dependency. He also pointed out that this was >> >> > discussed before, with most people being in favor: >> >> > http://mail.scipy.org/pipermail/scipy-dev/2009-November/013272.html >> >> > http://mail.scipy.org/pipermail/scipy-dev/2009-November/013308.html >> >> > We discussed the same issue on >> >> > https://github.com/scipy/scipy/pull/211 >> >> > recently, and also the size of the binary. >> >> > >> >> > This is probably also the right moment to point out other recent >> >> > Cython >> >> > issues we've had: >> >> > 1. A memoryview issue with Python 2.4, either a Cython or Numpy bug: >> >> > https://github.com/numpy/numpy/pull/307 >> >> > 2. We had to manually patch the generated C files when using Cython >> >> > 0.16, to >> >> > make them work with MinGW: >> >> > http://projects.scipy.org/scipy/ticket/1673 >> >> > 3. According to Ray, there's also an indexing bug in Cython 0.16 >> >> > which >> >> > requires to use 0.17-dev for https://github.com/scipy/scipy/pull/261 >> >> > >> >> > I think it's clear that PR's like #261 above (Ray's ndimage.label >> >> > rewrite) >> >> > are in principle a good thing: faster and more general code which is >> >> > easier >> >> > to maintain. Now the question is what to do though. Here's some >> >> > options >> >> > that >> >> > I see: >> >> > >> >> > a) Keep things as is for now. Accept large file/binary sizes. >> >> > Manually >> >> > patch >> >> > the generated C if necessary. >> >> > b) Keep things as is for now. Either go back to Cython 0.15, or bump >> >> > required numpy version to latest dev version to not have to manually >> >> > patch >> >> > the generated C files. >> >> > c) Keep things as they are now, without accepting too large >> >> > file/binary >> >> > sizes. To be defined what too large. Means we can't get the full >> >> > benefits of >> >> > fused types for example. >> >> > d) Move to Cython as a build dependency. Write down the required >> >> > versions >> >> > and incompatibilities in the docs. >> >> > e) Include a Cython version in the scipy git repo, patch it to solve >> >> > the >> >> > above issues 2 and 3 (and any other ones that come along). >> >> > f) Some combination of the above. >> >> > g) Any other options? >> >> >> >> Am I right in thinking that Cython 0.17dev will generate usable C >> >> files without patching? >> > >> > >> > Yes. >> >> How about making Cython 0.17 a developer build-time dependency? > > > That's an option. Requiring a dev version will mean broken builds for some > of the users that don't read the docs well but simply do "easyinstall > cython". I'm not sure how acceptable that is. We could surely raise an informative error for that case? I hope that there won't be long before the 0.17 release - but we should check with the Cython folks. On the plus side, lowering the barrier to rewriting in Cython seems like a really big win, especially with memoryviews and fused types available. > That still leaves the (mostly orthogonal) question about binary size. I just > built Ray's PR, _nd_label.so is 1.4 Mb. For one function. Hmm. 1.4 Mb seems OK to me for the binary - but I can see that we'd have to watch that. Maybe it would be worth asking on the Cython list whether there is any way of reducing this, maybe by sharing across extensions. How is the load time for that extension? >> Meanwhile, making 'python setup.py sdist' dump the c files into the >> source distribution? Maybe with a nightly development snapshot pushed >> up to sourceforge or similar? >> >> That's what we are doing for dipy (the c files into the sdist): >> >> https://github.com/nipy/dipy/blob/master/setup.py#L95 >> https://github.com/nipy/dipy/blob/master/cythexts.py >> > That sounds like a good idea, if we go for a Cython build dependency. Even > if we don't, nightly builds would be great. Would a built sdist be widely used do you think? Compared to someone following .git? Just wondering out loud... Cheers, Matthew From pav at iki.fi Thu Jul 5 15:36:18 2012 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 05 Jul 2012 21:36:18 +0200 Subject: [SciPy-Dev] Cython as build dependency, file/dll size and current issues In-Reply-To: <4FF5E8B8.70602@astro.washington.edu> References: <4FF5E8B8.70602@astro.washington.edu> Message-ID: 05.07.2012 21:19, Jacob VanderPlas kirjoitti: > On 07/05/12 11:57, Matthew Brett wrote: [clip] >> That's what we are doing for dipy (the c files into the sdist): >> >> https://github.com/nipy/dipy/blob/master/setup.py#L95 >> https://github.com/nipy/dipy/blob/master/cythexts.py >> >> although a Bento way of doing that would probably be easier to maintain, > > I think this is the best compromise, if we can make it work. The > flexibility of fused types is a major plus. Adding a general user-end > dependency on cython would be a mistake, IMHO (scipy can be difficult > enough to build as it is). This would also allow us to depend on > bleeding-edge cython versions for development builds, and hopefully fix > some of the by-hand patching issues we've seen in the past. +1 for build-time dependency for Git builds, and dumping to .c files on sdist. For the binary size --- stripped, it comes down to 500k. That's still a pretty lot for a 500-line input file, but it's of the same order of magnitude as the sparse/sparsetools .so files generated by g++. Pauli From ralf.gommers at googlemail.com Thu Jul 5 15:37:30 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 5 Jul 2012 21:37:30 +0200 Subject: [SciPy-Dev] Cython as build dependency, file/dll size and current issues In-Reply-To: References: Message-ID: On Thu, Jul 5, 2012 at 9:26 PM, Matthew Brett wrote: > Hi, > > On Thu, Jul 5, 2012 at 12:18 PM, Ralf Gommers > wrote: > > > > > > On Thu, Jul 5, 2012 at 8:57 PM, Matthew Brett > > wrote: > >> > >> Hi, > >> > >> On Thu, Jul 5, 2012 at 11:35 AM, Ralf Gommers > >> wrote: > >> > > >> > > >> > On Thu, Jul 5, 2012 at 8:31 PM, Matthew Brett < > matthew.brett at gmail.com> > >> > wrote: > >> >> > >> >> Hi, > >> >> > >> >> On Thu, Jul 5, 2012 at 11:25 AM, Ralf Gommers > >> >> wrote: > >> >> > Hi all, > >> >> > > >> >> > On https://github.com/scipy/scipy/pull/261 the problem with large > >> >> > size > >> >> > of > >> >> > generated C files from Cython came up again, and Matthew suggested > to > >> >> > add > >> >> > Cython as a build time dependency. He also pointed out that this > was > >> >> > discussed before, with most people being in favor: > >> >> > > http://mail.scipy.org/pipermail/scipy-dev/2009-November/013272.html > >> >> > > http://mail.scipy.org/pipermail/scipy-dev/2009-November/013308.html > >> >> > We discussed the same issue on > >> >> > https://github.com/scipy/scipy/pull/211 > >> >> > recently, and also the size of the binary. > >> >> > > >> >> > This is probably also the right moment to point out other recent > >> >> > Cython > >> >> > issues we've had: > >> >> > 1. A memoryview issue with Python 2.4, either a Cython or Numpy > bug: > >> >> > https://github.com/numpy/numpy/pull/307 > >> >> > 2. We had to manually patch the generated C files when using Cython > >> >> > 0.16, to > >> >> > make them work with MinGW: > >> >> > http://projects.scipy.org/scipy/ticket/1673 > >> >> > 3. According to Ray, there's also an indexing bug in Cython 0.16 > >> >> > which > >> >> > requires to use 0.17-dev for > https://github.com/scipy/scipy/pull/261 > >> >> > > >> >> > I think it's clear that PR's like #261 above (Ray's ndimage.label > >> >> > rewrite) > >> >> > are in principle a good thing: faster and more general code which > is > >> >> > easier > >> >> > to maintain. Now the question is what to do though. Here's some > >> >> > options > >> >> > that > >> >> > I see: > >> >> > > >> >> > a) Keep things as is for now. Accept large file/binary sizes. > >> >> > Manually > >> >> > patch > >> >> > the generated C if necessary. > >> >> > b) Keep things as is for now. Either go back to Cython 0.15, or > bump > >> >> > required numpy version to latest dev version to not have to > manually > >> >> > patch > >> >> > the generated C files. > >> >> > c) Keep things as they are now, without accepting too large > >> >> > file/binary > >> >> > sizes. To be defined what too large. Means we can't get the full > >> >> > benefits of > >> >> > fused types for example. > >> >> > d) Move to Cython as a build dependency. Write down the required > >> >> > versions > >> >> > and incompatibilities in the docs. > >> >> > e) Include a Cython version in the scipy git repo, patch it to > solve > >> >> > the > >> >> > above issues 2 and 3 (and any other ones that come along). > >> >> > f) Some combination of the above. > >> >> > g) Any other options? > >> >> > >> >> Am I right in thinking that Cython 0.17dev will generate usable C > >> >> files without patching? > >> > > >> > > >> > Yes. > >> > >> How about making Cython 0.17 a developer build-time dependency? > > > > > > That's an option. Requiring a dev version will mean broken builds for > some > > of the users that don't read the docs well but simply do "easyinstall > > cython". I'm not sure how acceptable that is. > > We could surely raise an informative error for that case? I hope that > there won't be long before the 0.17 release - but we should check with > the Cython folks. > True. Perhaps it's not a big issue. > > On the plus side, lowering the barrier to rewriting in Cython seems > like a really big win, especially with memoryviews and fused types > available. > Agreed about lower barrier and fused types. Memoryviews are still not OK, because of https://github.com/numpy/numpy/pull/307. > > That still leaves the (mostly orthogonal) question about binary size. I > just > > built Ray's PR, _nd_label.so is 1.4 Mb. For one function. > > Hmm. 1.4 Mb seems OK to me for the binary Really? For one function? If we do that for each function, we end up with 4 Gb. > - but I can see that we'd > have to watch that. Maybe it would be worth asking on the Cython list > whether there is any way of reducing this, maybe by sharing across > extensions. How is the load time for that extension? > Very poor. (all hot cache): $ time python -c "" real 0m0.039s user 0m0.017s sys 0m0.017s $ time python -c "import numpy" real 0m0.187s user 0m0.080s sys 0m0.100s $ time python -c "import _ni_label" real 0m0.206s user 0m0.081s sys 0m0.109s > >> Meanwhile, making 'python setup.py sdist' dump the c files into the > >> source distribution? Maybe with a nightly development snapshot pushed > >> up to sourceforge or similar? > >> > >> That's what we are doing for dipy (the c files into the sdist): > >> > >> https://github.com/nipy/dipy/blob/master/setup.py#L95 > >> https://github.com/nipy/dipy/blob/master/cythexts.py > >> > > That sounds like a good idea, if we go for a Cython build dependency. > Even > > if we don't, nightly builds would be great. > > Would a built sdist be widely used do you think? Compared to someone > following .git? Just wondering out loud... > No idea to be honest. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Thu Jul 5 15:45:26 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 5 Jul 2012 12:45:26 -0700 Subject: [SciPy-Dev] Cython as build dependency, file/dll size and current issues In-Reply-To: References: Message-ID: Hi, On Thu, Jul 5, 2012 at 12:37 PM, Ralf Gommers wrote: > > > On Thu, Jul 5, 2012 at 9:26 PM, Matthew Brett > wrote: >> >> Hi, >> >> On Thu, Jul 5, 2012 at 12:18 PM, Ralf Gommers >> wrote: >> > >> > >> > On Thu, Jul 5, 2012 at 8:57 PM, Matthew Brett >> > wrote: >> >> >> >> Hi, >> >> >> >> On Thu, Jul 5, 2012 at 11:35 AM, Ralf Gommers >> >> wrote: >> >> > >> >> > >> >> > On Thu, Jul 5, 2012 at 8:31 PM, Matthew Brett >> >> > >> >> > wrote: >> >> >> >> >> >> Hi, >> >> >> >> >> >> On Thu, Jul 5, 2012 at 11:25 AM, Ralf Gommers >> >> >> wrote: >> >> >> > Hi all, >> >> >> > >> >> >> > On https://github.com/scipy/scipy/pull/261 the problem with large >> >> >> > size >> >> >> > of >> >> >> > generated C files from Cython came up again, and Matthew suggested >> >> >> > to >> >> >> > add >> >> >> > Cython as a build time dependency. He also pointed out that this >> >> >> > was >> >> >> > discussed before, with most people being in favor: >> >> >> > >> >> >> > http://mail.scipy.org/pipermail/scipy-dev/2009-November/013272.html >> >> >> > >> >> >> > http://mail.scipy.org/pipermail/scipy-dev/2009-November/013308.html >> >> >> > We discussed the same issue on >> >> >> > https://github.com/scipy/scipy/pull/211 >> >> >> > recently, and also the size of the binary. >> >> >> > >> >> >> > This is probably also the right moment to point out other recent >> >> >> > Cython >> >> >> > issues we've had: >> >> >> > 1. A memoryview issue with Python 2.4, either a Cython or Numpy >> >> >> > bug: >> >> >> > https://github.com/numpy/numpy/pull/307 >> >> >> > 2. We had to manually patch the generated C files when using >> >> >> > Cython >> >> >> > 0.16, to >> >> >> > make them work with MinGW: >> >> >> > http://projects.scipy.org/scipy/ticket/1673 >> >> >> > 3. According to Ray, there's also an indexing bug in Cython 0.16 >> >> >> > which >> >> >> > requires to use 0.17-dev for >> >> >> > https://github.com/scipy/scipy/pull/261 >> >> >> > >> >> >> > I think it's clear that PR's like #261 above (Ray's ndimage.label >> >> >> > rewrite) >> >> >> > are in principle a good thing: faster and more general code which >> >> >> > is >> >> >> > easier >> >> >> > to maintain. Now the question is what to do though. Here's some >> >> >> > options >> >> >> > that >> >> >> > I see: >> >> >> > >> >> >> > a) Keep things as is for now. Accept large file/binary sizes. >> >> >> > Manually >> >> >> > patch >> >> >> > the generated C if necessary. >> >> >> > b) Keep things as is for now. Either go back to Cython 0.15, or >> >> >> > bump >> >> >> > required numpy version to latest dev version to not have to >> >> >> > manually >> >> >> > patch >> >> >> > the generated C files. >> >> >> > c) Keep things as they are now, without accepting too large >> >> >> > file/binary >> >> >> > sizes. To be defined what too large. Means we can't get the full >> >> >> > benefits of >> >> >> > fused types for example. >> >> >> > d) Move to Cython as a build dependency. Write down the required >> >> >> > versions >> >> >> > and incompatibilities in the docs. >> >> >> > e) Include a Cython version in the scipy git repo, patch it to >> >> >> > solve >> >> >> > the >> >> >> > above issues 2 and 3 (and any other ones that come along). >> >> >> > f) Some combination of the above. >> >> >> > g) Any other options? >> >> >> >> >> >> Am I right in thinking that Cython 0.17dev will generate usable C >> >> >> files without patching? >> >> > >> >> > >> >> > Yes. >> >> >> >> How about making Cython 0.17 a developer build-time dependency? >> > >> > >> > That's an option. Requiring a dev version will mean broken builds for >> > some >> > of the users that don't read the docs well but simply do "easyinstall >> > cython". I'm not sure how acceptable that is. >> >> We could surely raise an informative error for that case? I hope that >> there won't be long before the 0.17 release - but we should check with >> the Cython folks. > > > True. Perhaps it's not a big issue. >> >> >> On the plus side, lowering the barrier to rewriting in Cython seems >> like a really big win, especially with memoryviews and fused types >> available. > > > Agreed about lower barrier and fused types. > > Memoryviews are still not OK, because of > https://github.com/numpy/numpy/pull/307. I'm afraid I didn't understand that discussion very well. Does that only apply to python 2.4? I had the impression we were dropping 2.4 compatibility, but I may be remembering wrong. >> >> > That still leaves the (mostly orthogonal) question about binary size. I >> > just >> > built Ray's PR, _nd_label.so is 1.4 Mb. For one function. >> >> Hmm. 1.4 Mb seems OK to me for the binary > > > Really? For one function? If we do that for each function, we end up with 4 > Gb. > >> >> - but I can see that we'd >> have to watch that. Maybe it would be worth asking on the Cython list >> whether there is any way of reducing this, maybe by sharing across >> extensions. How is the load time for that extension? > > > Very poor. (all hot cache): > > $ time python -c "" > real 0m0.039s > user 0m0.017s > sys 0m0.017s > > $ time python -c "import numpy" > real 0m0.187s > user 0m0.080s > sys 0m0.100s > > $ time python -c "import _ni_label" > real 0m0.206s > user 0m0.081s > sys 0m0.109s I guess that's much slower than the original C extension? I'd tend to prefer a slow loading but fast running and maintainable ndimage, but it's unfortunate we have to keep these tradeoffs in mind... Cheers, Matthew From pav at iki.fi Thu Jul 5 15:48:38 2012 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 05 Jul 2012 21:48:38 +0200 Subject: [SciPy-Dev] Cython as build dependency, file/dll size and current issues In-Reply-To: References: Message-ID: 05.07.2012 21:37, Ralf Gommers kirjoitti: [clip] > > That still leaves the (mostly orthogonal) question about binary > size. I just > > built Ray's PR, _nd_label.so is 1.4 Mb. For one function. > > Hmm. 1.4 Mb seems OK to me for the binary > > Really? For one function? If we do that for each function, we end up > with 4 Gb. The stripped code size is about 300k for the functions, and there's 200k of overhead code. That's still not so little, think 15 years back... Original file: $ strip ndimage/_ni_label.so ; ls -l ndimage/_ni_label.so -rwxr-x--- 1 pauli pauli 505752 hein? 5 21:40 ndimage/_ni_label.so All functions in the file duplicated and renamed: $ strip ndimage/_ni_label.so ; ls -l ndimage/_ni_label.so -rwxr-x--- 1 pauli pauli 819608 hein? 5 21:44 ndimage/_ni_label.so To get more insight what could be cut would need looking into the code Cython generates. -- Pauli Virtanen From ralf.gommers at googlemail.com Thu Jul 5 15:53:50 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 5 Jul 2012 21:53:50 +0200 Subject: [SciPy-Dev] Cython as build dependency, file/dll size and current issues In-Reply-To: References: Message-ID: On Thu, Jul 5, 2012 at 9:45 PM, Matthew Brett wrote: > Hi, > > On Thu, Jul 5, 2012 at 12:37 PM, Ralf Gommers > wrote: > > > > > > On Thu, Jul 5, 2012 at 9:26 PM, Matthew Brett > > wrote: > >> > >> Hi, > >> > >> On Thu, Jul 5, 2012 at 12:18 PM, Ralf Gommers > >> wrote: > >> > > >> > > >> > On Thu, Jul 5, 2012 at 8:57 PM, Matthew Brett < > matthew.brett at gmail.com> > >> > wrote: > >> >> > >> >> Hi, > >> >> > >> >> On Thu, Jul 5, 2012 at 11:35 AM, Ralf Gommers > >> >> wrote: > >> >> > > >> >> > > >> >> > On Thu, Jul 5, 2012 at 8:31 PM, Matthew Brett > >> >> > > >> >> > wrote: > >> >> >> > >> >> >> Hi, > >> >> >> > >> >> >> On Thu, Jul 5, 2012 at 11:25 AM, Ralf Gommers > >> >> >> wrote: > >> >> >> > Hi all, > >> >> >> > > >> >> >> > On https://github.com/scipy/scipy/pull/261 the problem with > large > >> >> >> > size > >> >> >> > of > >> >> >> > generated C files from Cython came up again, and Matthew > suggested > >> >> >> > to > >> >> >> > add > >> >> >> > Cython as a build time dependency. He also pointed out that this > >> >> >> > was > >> >> >> > discussed before, with most people being in favor: > >> >> >> > > >> >> >> > > http://mail.scipy.org/pipermail/scipy-dev/2009-November/013272.html > >> >> >> > > >> >> >> > > http://mail.scipy.org/pipermail/scipy-dev/2009-November/013308.html > >> >> >> > We discussed the same issue on > >> >> >> > https://github.com/scipy/scipy/pull/211 > >> >> >> > recently, and also the size of the binary. > >> >> >> > > >> >> >> > This is probably also the right moment to point out other recent > >> >> >> > Cython > >> >> >> > issues we've had: > >> >> >> > 1. A memoryview issue with Python 2.4, either a Cython or Numpy > >> >> >> > bug: > >> >> >> > https://github.com/numpy/numpy/pull/307 > >> >> >> > 2. We had to manually patch the generated C files when using > >> >> >> > Cython > >> >> >> > 0.16, to > >> >> >> > make them work with MinGW: > >> >> >> > http://projects.scipy.org/scipy/ticket/1673 > >> >> >> > 3. According to Ray, there's also an indexing bug in Cython 0.16 > >> >> >> > which > >> >> >> > requires to use 0.17-dev for > >> >> >> > https://github.com/scipy/scipy/pull/261 > >> >> >> > > >> >> >> > I think it's clear that PR's like #261 above (Ray's > ndimage.label > >> >> >> > rewrite) > >> >> >> > are in principle a good thing: faster and more general code > which > >> >> >> > is > >> >> >> > easier > >> >> >> > to maintain. Now the question is what to do though. Here's some > >> >> >> > options > >> >> >> > that > >> >> >> > I see: > >> >> >> > > >> >> >> > a) Keep things as is for now. Accept large file/binary sizes. > >> >> >> > Manually > >> >> >> > patch > >> >> >> > the generated C if necessary. > >> >> >> > b) Keep things as is for now. Either go back to Cython 0.15, or > >> >> >> > bump > >> >> >> > required numpy version to latest dev version to not have to > >> >> >> > manually > >> >> >> > patch > >> >> >> > the generated C files. > >> >> >> > c) Keep things as they are now, without accepting too large > >> >> >> > file/binary > >> >> >> > sizes. To be defined what too large. Means we can't get the full > >> >> >> > benefits of > >> >> >> > fused types for example. > >> >> >> > d) Move to Cython as a build dependency. Write down the required > >> >> >> > versions > >> >> >> > and incompatibilities in the docs. > >> >> >> > e) Include a Cython version in the scipy git repo, patch it to > >> >> >> > solve > >> >> >> > the > >> >> >> > above issues 2 and 3 (and any other ones that come along). > >> >> >> > f) Some combination of the above. > >> >> >> > g) Any other options? > >> >> >> > >> >> >> Am I right in thinking that Cython 0.17dev will generate usable C > >> >> >> files without patching? > >> >> > > >> >> > > >> >> > Yes. > >> >> > >> >> How about making Cython 0.17 a developer build-time dependency? > >> > > >> > > >> > That's an option. Requiring a dev version will mean broken builds for > >> > some > >> > of the users that don't read the docs well but simply do "easyinstall > >> > cython". I'm not sure how acceptable that is. > >> > >> We could surely raise an informative error for that case? I hope that > >> there won't be long before the 0.17 release - but we should check with > >> the Cython folks. > > > > > > True. Perhaps it's not a big issue. > >> > >> > >> On the plus side, lowering the barrier to rewriting in Cython seems > >> like a really big win, especially with memoryviews and fused types > >> available. > > > > > > Agreed about lower barrier and fused types. > > > > Memoryviews are still not OK, because of > > https://github.com/numpy/numpy/pull/307. > > I'm afraid I didn't understand that discussion very well. Does that > only apply to python 2.4? I had the impression we were dropping 2.4 > compatibility, but I may be remembering wrong. > Yes, only for python 2.4. But no, we're not dropping it. I'd like to, but each time it's brought up the result is the same. I'd rather not mix that discussion with this one. > > >> > >> > That still leaves the (mostly orthogonal) question about binary size. > I > >> > just > >> > built Ray's PR, _nd_label.so is 1.4 Mb. For one function. > >> > >> Hmm. 1.4 Mb seems OK to me for the binary > > > > > > Really? For one function? If we do that for each function, we end up > with 4 > > Gb. > > > >> > >> - but I can see that we'd > >> have to watch that. Maybe it would be worth asking on the Cython list > >> whether there is any way of reducing this, maybe by sharing across > >> extensions. How is the load time for that extension? > > > > > > Very poor. (all hot cache): > > > > $ time python -c "" > > real 0m0.039s > > user 0m0.017s > > sys 0m0.017s > > > > $ time python -c "import numpy" > > real 0m0.187s > > user 0m0.080s > > sys 0m0.100s > > > > $ time python -c "import _ni_label" > > real 0m0.206s > > user 0m0.081s > > sys 0m0.109s > > I guess that's much slower than the original C extension? I'd tend to > prefer a slow loading but fast running and maintainable ndimage, but > it's unfortunate we have to keep these tradeoffs in mind... > The whole ndimage module now imports in 0.280s. Not sure what the old version with label() included was, but probably similar. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Thu Jul 5 15:59:09 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 5 Jul 2012 21:59:09 +0200 Subject: [SciPy-Dev] Cython as build dependency, file/dll size and current issues In-Reply-To: References: Message-ID: On Thu, Jul 5, 2012 at 9:48 PM, Pauli Virtanen wrote: > 05.07.2012 21:37, Ralf Gommers kirjoitti: > [clip] > > > That still leaves the (mostly orthogonal) question about binary > > size. I just > > > built Ray's PR, _nd_label.so is 1.4 Mb. For one function. > > > > Hmm. 1.4 Mb seems OK to me for the binary > > > > Really? For one function? If we do that for each function, we end up > > with 4 Gb. > > The stripped code size is about 300k for the functions, and there's 200k > of overhead code. > > That's still not so little, think 15 years back... > > Original file: > > $ strip ndimage/_ni_label.so ; ls -l ndimage/_ni_label.so > -rwxr-x--- 1 pauli pauli 505752 hein? 5 21:40 ndimage/_ni_label.so > > Larger for me. Strip takes off 200 kb. On OS X everything is double (universal binary), not sure where the other factor of 2 comes from: $ strip -u -r _ni_label.so $ ls -l rwxr-xr-x 1 rgommers staff 1205404 Jul 5 21:56 _ni_label.so Ralf All functions in the file duplicated and renamed: > > $ strip ndimage/_ni_label.so ; ls -l ndimage/_ni_label.so > -rwxr-x--- 1 pauli pauli 819608 hein? 5 21:44 ndimage/_ni_label.so > > To get more insight what could be cut would need looking into the code > Cython generates. > > -- > Pauli Virtanen > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Thu Jul 5 16:06:03 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 5 Jul 2012 13:06:03 -0700 Subject: [SciPy-Dev] Cython as build dependency, file/dll size and current issues In-Reply-To: References: Message-ID: Hi, On Thu, Jul 5, 2012 at 12:53 PM, Ralf Gommers wrote: > > > On Thu, Jul 5, 2012 at 9:45 PM, Matthew Brett > wrote: >> >> Hi, >> >> On Thu, Jul 5, 2012 at 12:37 PM, Ralf Gommers >> wrote: >> > >> > >> > On Thu, Jul 5, 2012 at 9:26 PM, Matthew Brett >> > wrote: >> >> >> >> Hi, >> >> >> >> On Thu, Jul 5, 2012 at 12:18 PM, Ralf Gommers >> >> wrote: >> >> > >> >> > >> >> > On Thu, Jul 5, 2012 at 8:57 PM, Matthew Brett >> >> > >> >> > wrote: >> >> >> >> >> >> Hi, >> >> >> >> >> >> On Thu, Jul 5, 2012 at 11:35 AM, Ralf Gommers >> >> >> wrote: >> >> >> > >> >> >> > >> >> >> > On Thu, Jul 5, 2012 at 8:31 PM, Matthew Brett >> >> >> > >> >> >> > wrote: >> >> >> >> >> >> >> >> Hi, >> >> >> >> >> >> >> >> On Thu, Jul 5, 2012 at 11:25 AM, Ralf Gommers >> >> >> >> wrote: >> >> >> >> > Hi all, >> >> >> >> > >> >> >> >> > On https://github.com/scipy/scipy/pull/261 the problem with >> >> >> >> > large >> >> >> >> > size >> >> >> >> > of >> >> >> >> > generated C files from Cython came up again, and Matthew >> >> >> >> > suggested >> >> >> >> > to >> >> >> >> > add >> >> >> >> > Cython as a build time dependency. He also pointed out that >> >> >> >> > this >> >> >> >> > was >> >> >> >> > discussed before, with most people being in favor: >> >> >> >> > >> >> >> >> > >> >> >> >> > http://mail.scipy.org/pipermail/scipy-dev/2009-November/013272.html >> >> >> >> > >> >> >> >> > >> >> >> >> > http://mail.scipy.org/pipermail/scipy-dev/2009-November/013308.html >> >> >> >> > We discussed the same issue on >> >> >> >> > https://github.com/scipy/scipy/pull/211 >> >> >> >> > recently, and also the size of the binary. >> >> >> >> > >> >> >> >> > This is probably also the right moment to point out other >> >> >> >> > recent >> >> >> >> > Cython >> >> >> >> > issues we've had: >> >> >> >> > 1. A memoryview issue with Python 2.4, either a Cython or Numpy >> >> >> >> > bug: >> >> >> >> > https://github.com/numpy/numpy/pull/307 >> >> >> >> > 2. We had to manually patch the generated C files when using >> >> >> >> > Cython >> >> >> >> > 0.16, to >> >> >> >> > make them work with MinGW: >> >> >> >> > http://projects.scipy.org/scipy/ticket/1673 >> >> >> >> > 3. According to Ray, there's also an indexing bug in Cython >> >> >> >> > 0.16 >> >> >> >> > which >> >> >> >> > requires to use 0.17-dev for >> >> >> >> > https://github.com/scipy/scipy/pull/261 >> >> >> >> > >> >> >> >> > I think it's clear that PR's like #261 above (Ray's >> >> >> >> > ndimage.label >> >> >> >> > rewrite) >> >> >> >> > are in principle a good thing: faster and more general code >> >> >> >> > which >> >> >> >> > is >> >> >> >> > easier >> >> >> >> > to maintain. Now the question is what to do though. Here's some >> >> >> >> > options >> >> >> >> > that >> >> >> >> > I see: >> >> >> >> > >> >> >> >> > a) Keep things as is for now. Accept large file/binary sizes. >> >> >> >> > Manually >> >> >> >> > patch >> >> >> >> > the generated C if necessary. >> >> >> >> > b) Keep things as is for now. Either go back to Cython 0.15, or >> >> >> >> > bump >> >> >> >> > required numpy version to latest dev version to not have to >> >> >> >> > manually >> >> >> >> > patch >> >> >> >> > the generated C files. >> >> >> >> > c) Keep things as they are now, without accepting too large >> >> >> >> > file/binary >> >> >> >> > sizes. To be defined what too large. Means we can't get the >> >> >> >> > full >> >> >> >> > benefits of >> >> >> >> > fused types for example. >> >> >> >> > d) Move to Cython as a build dependency. Write down the >> >> >> >> > required >> >> >> >> > versions >> >> >> >> > and incompatibilities in the docs. >> >> >> >> > e) Include a Cython version in the scipy git repo, patch it to >> >> >> >> > solve >> >> >> >> > the >> >> >> >> > above issues 2 and 3 (and any other ones that come along). >> >> >> >> > f) Some combination of the above. >> >> >> >> > g) Any other options? >> >> >> >> >> >> >> >> Am I right in thinking that Cython 0.17dev will generate usable C >> >> >> >> files without patching? >> >> >> > >> >> >> > >> >> >> > Yes. >> >> >> >> >> >> How about making Cython 0.17 a developer build-time dependency? >> >> > >> >> > >> >> > That's an option. Requiring a dev version will mean broken builds for >> >> > some >> >> > of the users that don't read the docs well but simply do "easyinstall >> >> > cython". I'm not sure how acceptable that is. >> >> >> >> We could surely raise an informative error for that case? I hope that >> >> there won't be long before the 0.17 release - but we should check with >> >> the Cython folks. >> > >> > >> > True. Perhaps it's not a big issue. >> >> >> >> >> >> On the plus side, lowering the barrier to rewriting in Cython seems >> >> like a really big win, especially with memoryviews and fused types >> >> available. >> > >> > >> > Agreed about lower barrier and fused types. >> > >> > Memoryviews are still not OK, because of >> > https://github.com/numpy/numpy/pull/307. >> >> I'm afraid I didn't understand that discussion very well. Does that >> only apply to python 2.4? I had the impression we were dropping 2.4 >> compatibility, but I may be remembering wrong. > > > Yes, only for python 2.4. But no, we're not dropping it. I'd like to, but > each time it's brought up the result is the same. I'd rather not mix that > discussion with this one. Yes, good call. Maybe we could have that discussion in due course. Cheers, Matthew From thouis at broadinstitute.org Thu Jul 5 16:44:30 2012 From: thouis at broadinstitute.org (Thouis (Ray) Jones) Date: Thu, 5 Jul 2012 22:44:30 +0200 Subject: [SciPy-Dev] Cython as build dependency, file/dll size and current issues In-Reply-To: References: Message-ID: On Thu, Jul 5, 2012 at 9:48 PM, Pauli Virtanen wrote: > To get more insight what could be cut would need looking into the code > Cython generates. There's a (short) function that's specialized on two fused types, each of which can be 10 different individual types. I have an idea to rewrite it to specialize fewer and much shorter functions and only on single types, which I'll hopefully finish tomorrow. Ray From thouis at gmail.com Fri Jul 6 04:11:03 2012 From: thouis at gmail.com (Thouis (Ray) Jones) Date: Fri, 6 Jul 2012 10:11:03 +0200 Subject: [SciPy-Dev] Cython as build dependency, file/dll size and current issues In-Reply-To: References: Message-ID: On Thu, Jul 5, 2012 at 9:37 PM, Ralf Gommers wrote: > > > On Thu, Jul 5, 2012 at 9:26 PM, Matthew Brett > wrote: >> >> Hi, >> >> On Thu, Jul 5, 2012 at 12:18 PM, Ralf Gommers >> wrote: >> > >> > >> > On Thu, Jul 5, 2012 at 8:57 PM, Matthew Brett >> > wrote: >> >> >> >> Hi, >> >> >> >> On Thu, Jul 5, 2012 at 11:35 AM, Ralf Gommers >> >> wrote: >> >> > >> >> > >> >> > On Thu, Jul 5, 2012 at 8:31 PM, Matthew Brett >> >> > >> >> > wrote: >> >> >> >> >> >> Hi, >> >> >> >> >> >> On Thu, Jul 5, 2012 at 11:25 AM, Ralf Gommers >> >> >> wrote: >> >> >> > Hi all, >> >> >> > >> >> >> > On https://github.com/scipy/scipy/pull/261 the problem with large >> >> >> > size >> >> >> > of >> >> >> > generated C files from Cython came up again, and Matthew suggested >> >> >> > to >> >> >> > add >> >> >> > Cython as a build time dependency. He also pointed out that this >> >> >> > was >> >> >> > discussed before, with most people being in favor: >> >> >> > >> >> >> > http://mail.scipy.org/pipermail/scipy-dev/2009-November/013272.html >> >> >> > >> >> >> > http://mail.scipy.org/pipermail/scipy-dev/2009-November/013308.html >> >> >> > We discussed the same issue on >> >> >> > https://github.com/scipy/scipy/pull/211 >> >> >> > recently, and also the size of the binary. >> >> >> > >> >> >> > This is probably also the right moment to point out other recent >> >> >> > Cython >> >> >> > issues we've had: >> >> >> > 1. A memoryview issue with Python 2.4, either a Cython or Numpy >> >> >> > bug: >> >> >> > https://github.com/numpy/numpy/pull/307 >> >> >> > 2. We had to manually patch the generated C files when using >> >> >> > Cython >> >> >> > 0.16, to >> >> >> > make them work with MinGW: >> >> >> > http://projects.scipy.org/scipy/ticket/1673 >> >> >> > 3. According to Ray, there's also an indexing bug in Cython 0.16 >> >> >> > which >> >> >> > requires to use 0.17-dev for >> >> >> > https://github.com/scipy/scipy/pull/261 >> >> >> > >> >> >> > I think it's clear that PR's like #261 above (Ray's ndimage.label >> >> >> > rewrite) >> >> >> > are in principle a good thing: faster and more general code which >> >> >> > is >> >> >> > easier >> >> >> > to maintain. Now the question is what to do though. Here's some >> >> >> > options >> >> >> > that >> >> >> > I see: >> >> >> > >> >> >> > a) Keep things as is for now. Accept large file/binary sizes. >> >> >> > Manually >> >> >> > patch >> >> >> > the generated C if necessary. >> >> >> > b) Keep things as is for now. Either go back to Cython 0.15, or >> >> >> > bump >> >> >> > required numpy version to latest dev version to not have to >> >> >> > manually >> >> >> > patch >> >> >> > the generated C files. >> >> >> > c) Keep things as they are now, without accepting too large >> >> >> > file/binary >> >> >> > sizes. To be defined what too large. Means we can't get the full >> >> >> > benefits of >> >> >> > fused types for example. >> >> >> > d) Move to Cython as a build dependency. Write down the required >> >> >> > versions >> >> >> > and incompatibilities in the docs. >> >> >> > e) Include a Cython version in the scipy git repo, patch it to >> >> >> > solve >> >> >> > the >> >> >> > above issues 2 and 3 (and any other ones that come along). >> >> >> > f) Some combination of the above. >> >> >> > g) Any other options? >> >> >> >> >> >> Am I right in thinking that Cython 0.17dev will generate usable C >> >> >> files without patching? >> >> > >> >> > >> >> > Yes. >> >> >> >> How about making Cython 0.17 a developer build-time dependency? >> > >> > >> > That's an option. Requiring a dev version will mean broken builds for >> > some >> > of the users that don't read the docs well but simply do "easyinstall >> > cython". I'm not sure how acceptable that is. >> >> We could surely raise an informative error for that case? ?I hope that >> there won't be long before the 0.17 release - but we should check with >> the Cython folks. > > > True. Perhaps it's not a big issue. >> >> >> On the plus side, lowering the barrier to rewriting in Cython seems >> like a really big win, especially with memoryviews and fused types >> available. > > > Agreed about lower barrier and fused types. > > Memoryviews are still not OK, because of > https://github.com/numpy/numpy/pull/307. > >> >> > That still leaves the (mostly orthogonal) question about binary size. I >> > just >> > built Ray's PR, _nd_label.so is 1.4 Mb. For one function. >> >> Hmm. ? 1.4 Mb seems OK to me for the binary > > > Really? For one function? If we do that for each function, we end up with 4 > Gb. > >> >> - but I can see that we'd >> have to watch that. ?Maybe it would be worth asking on the Cython list >> whether there is any way of reducing this, maybe by sharing across >> extensions. ? How is the load time for that extension? > > > Very poor. (all hot cache): > > ?$ time python -c "" > real??? 0m0.039s > user??? 0m0.017s > sys??? 0m0.017s > > $ time python -c "import numpy" > real??? 0m0.187s > user??? 0m0.080s > sys??? 0m0.100s > > $ time python -c "import _ni_label" > real??? 0m0.206s > user??? 0m0.081s > sys??? 0m0.109s To be fair, the _ni_label module also imports numpy. So the delta is around 0.019 s, still not great, but not as bad as the test seems to show. (Unless I was missing something, and 0.019 is actually that bad.) Ray From ralf.gommers at googlemail.com Fri Jul 6 12:11:10 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 6 Jul 2012 18:11:10 +0200 Subject: [SciPy-Dev] Cython as build dependency, file/dll size and current issues In-Reply-To: References: Message-ID: On Fri, Jul 6, 2012 at 10:11 AM, Thouis (Ray) Jones wrote: > On Thu, Jul 5, 2012 at 9:37 PM, Ralf Gommers > wrote: > > > > > > On Thu, Jul 5, 2012 at 9:26 PM, Matthew Brett > > wrote: > >> > >> Hi, > >> > >> On Thu, Jul 5, 2012 at 12:18 PM, Ralf Gommers > >> wrote: > >> > > >> > > >> > On Thu, Jul 5, 2012 at 8:57 PM, Matthew Brett < > matthew.brett at gmail.com> > >> > wrote: > >> >> > >> >> Hi, > >> >> > >> >> On Thu, Jul 5, 2012 at 11:35 AM, Ralf Gommers > >> >> wrote: > >> >> > > >> >> > > >> >> > On Thu, Jul 5, 2012 at 8:31 PM, Matthew Brett > >> >> > > >> >> > wrote: > >> >> >> > >> >> >> Hi, > >> >> >> > >> >> >> On Thu, Jul 5, 2012 at 11:25 AM, Ralf Gommers > >> >> >> wrote: > >> >> >> > Hi all, > >> >> >> > > >> >> >> > On https://github.com/scipy/scipy/pull/261 the problem with > large > >> >> >> > size > >> >> >> > of > >> >> >> > generated C files from Cython came up again, and Matthew > suggested > >> >> >> > to > >> >> >> > add > >> >> >> > Cython as a build time dependency. He also pointed out that this > >> >> >> > was > >> >> >> > discussed before, with most people being in favor: > >> >> >> > > >> >> >> > > http://mail.scipy.org/pipermail/scipy-dev/2009-November/013272.html > >> >> >> > > >> >> >> > > http://mail.scipy.org/pipermail/scipy-dev/2009-November/013308.html > >> >> >> > We discussed the same issue on > >> >> >> > https://github.com/scipy/scipy/pull/211 > >> >> >> > recently, and also the size of the binary. > >> >> >> > > >> >> >> > This is probably also the right moment to point out other recent > >> >> >> > Cython > >> >> >> > issues we've had: > >> >> >> > 1. A memoryview issue with Python 2.4, either a Cython or Numpy > >> >> >> > bug: > >> >> >> > https://github.com/numpy/numpy/pull/307 > >> >> >> > 2. We had to manually patch the generated C files when using > >> >> >> > Cython > >> >> >> > 0.16, to > >> >> >> > make them work with MinGW: > >> >> >> > http://projects.scipy.org/scipy/ticket/1673 > >> >> >> > 3. According to Ray, there's also an indexing bug in Cython 0.16 > >> >> >> > which > >> >> >> > requires to use 0.17-dev for > >> >> >> > https://github.com/scipy/scipy/pull/261 > >> >> >> > > >> >> >> > I think it's clear that PR's like #261 above (Ray's > ndimage.label > >> >> >> > rewrite) > >> >> >> > are in principle a good thing: faster and more general code > which > >> >> >> > is > >> >> >> > easier > >> >> >> > to maintain. Now the question is what to do though. Here's some > >> >> >> > options > >> >> >> > that > >> >> >> > I see: > >> >> >> > > >> >> >> > a) Keep things as is for now. Accept large file/binary sizes. > >> >> >> > Manually > >> >> >> > patch > >> >> >> > the generated C if necessary. > >> >> >> > b) Keep things as is for now. Either go back to Cython 0.15, or > >> >> >> > bump > >> >> >> > required numpy version to latest dev version to not have to > >> >> >> > manually > >> >> >> > patch > >> >> >> > the generated C files. > >> >> >> > c) Keep things as they are now, without accepting too large > >> >> >> > file/binary > >> >> >> > sizes. To be defined what too large. Means we can't get the full > >> >> >> > benefits of > >> >> >> > fused types for example. > >> >> >> > d) Move to Cython as a build dependency. Write down the required > >> >> >> > versions > >> >> >> > and incompatibilities in the docs. > >> >> >> > e) Include a Cython version in the scipy git repo, patch it to > >> >> >> > solve > >> >> >> > the > >> >> >> > above issues 2 and 3 (and any other ones that come along). > >> >> >> > f) Some combination of the above. > >> >> >> > g) Any other options? > >> >> >> > >> >> >> Am I right in thinking that Cython 0.17dev will generate usable C > >> >> >> files without patching? > >> >> > > >> >> > > >> >> > Yes. > >> >> > >> >> How about making Cython 0.17 a developer build-time dependency? > >> > > >> > > >> > That's an option. Requiring a dev version will mean broken builds for > >> > some > >> > of the users that don't read the docs well but simply do "easyinstall > >> > cython". I'm not sure how acceptable that is. > >> > >> We could surely raise an informative error for that case? I hope that > >> there won't be long before the 0.17 release - but we should check with > >> the Cython folks. > > > > > > True. Perhaps it's not a big issue. > >> > >> > >> On the plus side, lowering the barrier to rewriting in Cython seems > >> like a really big win, especially with memoryviews and fused types > >> available. > > > > > > Agreed about lower barrier and fused types. > > > > Memoryviews are still not OK, because of > > https://github.com/numpy/numpy/pull/307. > > > >> > >> > That still leaves the (mostly orthogonal) question about binary size. > I > >> > just > >> > built Ray's PR, _nd_label.so is 1.4 Mb. For one function. > >> > >> Hmm. 1.4 Mb seems OK to me for the binary > > > > > > Really? For one function? If we do that for each function, we end up > with 4 > > Gb. > > > >> > >> - but I can see that we'd > >> have to watch that. Maybe it would be worth asking on the Cython list > >> whether there is any way of reducing this, maybe by sharing across > >> extensions. How is the load time for that extension? > > > > > > Very poor. (all hot cache): > > > > $ time python -c "" > > real 0m0.039s > > user 0m0.017s > > sys 0m0.017s > > > > $ time python -c "import numpy" > > real 0m0.187s > > user 0m0.080s > > sys 0m0.100s > > > > $ time python -c "import _ni_label" > > real 0m0.206s > > user 0m0.081s > > sys 0m0.109s > > To be fair, the _ni_label module also imports numpy. Sorry, I should have mentioned that. > So the delta is around 0.019 s, still not great, but not as bad as the > test seems to > show. (Unless I was missing something, and 0.019 is actually that > bad.) > Depends how you look at it. At the current rate of cythonizing, the damage is probably fairly limited. Although 10% of the import time of numpy may still be considered a problem by some. If we'd convert a significant fraction of code to Cython though, this would give a huge penalty on load time and memory usage. Scipy has a total of 1073 functions and objects at the moment - determined by the sum of len(module.__all__) for all modules. Therefore 20 ms load time and O(100 kb) binary size for one function is a bit much. Note that the above is not a criticism of your PR. _ni_label now has a similar footprint to other Cython code in scipy, so this discussion shouldn't hold up merging it in. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Fri Jul 6 12:19:50 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 6 Jul 2012 18:19:50 +0200 Subject: [SciPy-Dev] interp2d bounds checking In-Reply-To: References: <4FE88062.1000103@crans.org> Message-ID: On Sun, Jul 1, 2012 at 8:40 PM, Pauli Virtanen wrote: > 01.07.2012 11:21, Ralf Gommers kirjoitti: > [clip] > > Bounds checking on a 2-D grid isn't really more difficult than on a 1-D > > grid. It looks like someone just copied the interp1d interface and then > > ran out of time or energy to fully implement it. > > > > Would be good if someone feels like implementing the missing bits, and > > otherwise we should at least document that right now out-of-bounds input > > simply raises an error (I didn't check which one). > > IMO, interp2d should be deprecated. > > It is equivalent to SmoothBivariateSpline with s=0. > > It's sort of a almost-name-alias for a Matlab function, but it works > differently --- scattered data interpolation vs. grid interpolation. > However, as FITPACK's 2-D scattered data splines not so rarely explode > with real data, I don't think this "replacement function" does its work > properly, so it should be deprecated. If the users want these 2-D > splines, it's better to make them ask for them explicitly... > > We can maybe think about adding a deprecation category "deprecated and > removed from documentation, but we'll keep it around indefinitely for > backwards compatibility". Move to e.g. to a new file '_deprecated.py' to > keep the moldy code separate. > This sounds like a good idea. interp2d is widely used I think, and we don't usually deprecate functions like that. > *** > > What's needed is a proper grid-base interpolation. It would be rather > easy to write a simple wrapper around ndimage.map_coordinates to get > general spline interpolation. Such interpolants are also easy to > construct up from the polynomial interpolants in Scipy. > > If someone is looking for something to do: *here* would be a nice, a > relatively easy, and very likely a very useful project to work on. > > *** > > Also, where interp2d works, griddata should work better in most use cases. > > I've also been thinking about enlarging the griddata syntax from the > possibly somewhat unfriendly > > griddata((x, y), z, (xi, yi), **kw) > griddata((x, y, z), u, (xi, yi, zi), **kw) > > to the variable argument one > > griddata(x, y, z, xi, yi, **kw) > griddata(x, y, z, u, xi, yi, zi, **kw) > > which may be more familiar to users. Thoughts? That does look easier to use, but having two interfaces will be confusing. And deprecating the old one will be painful. What exactly do you have in mind to do here? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri Jul 6 14:51:53 2012 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 06 Jul 2012 20:51:53 +0200 Subject: [SciPy-Dev] interp2d bounds checking In-Reply-To: References: <4FE88062.1000103@crans.org> Message-ID: 06.07.2012 18:19, Ralf Gommers kirjoitti: > On Sun, Jul 1, 2012 at 8:40 PM, Pauli Virtanen wrote: [clip] > I've also been thinking about enlarging the griddata syntax from the > possibly somewhat unfriendly > > griddata((x, y), z, (xi, yi), **kw) > griddata((x, y, z), u, (xi, yi, zi), **kw) > > to the variable argument one > > griddata(x, y, z, xi, yi, **kw) > griddata(x, y, z, u, xi, yi, zi, **kw) > > which may be more familiar to users. Thoughts? > > That does look easier to use, but having two interfaces will be > confusing. And deprecating the old one will be painful. What exactly do > you have in mind to do here? I was thinking about making the latter syntax an alias for the former, keeping both around. But perhaps it's confusing. Another issue is that variable arguments don't play too well with function signatures. This is BTW one thing that is better in Python 3, which has keyword-only arguments. Pauli From njs at pobox.com Fri Jul 6 15:28:21 2012 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 6 Jul 2012 20:28:21 +0100 Subject: [SciPy-Dev] interp2d bounds checking In-Reply-To: References: <4FE88062.1000103@crans.org> Message-ID: On Fri, Jul 6, 2012 at 7:51 PM, Pauli Virtanen wrote: > 06.07.2012 18:19, Ralf Gommers kirjoitti: >> On Sun, Jul 1, 2012 at 8:40 PM, Pauli Virtanen wrote: > [clip] >> I've also been thinking about enlarging the griddata syntax from the >> possibly somewhat unfriendly >> >> griddata((x, y), z, (xi, yi), **kw) >> griddata((x, y, z), u, (xi, yi, zi), **kw) >> >> to the variable argument one >> >> griddata(x, y, z, xi, yi, **kw) >> griddata(x, y, z, u, xi, yi, zi, **kw) >> >> which may be more familiar to users. Thoughts? >> >> That does look easier to use, but having two interfaces will be >> confusing. And deprecating the old one will be painful. What exactly do >> you have in mind to do here? > > I was thinking about making the latter syntax an alias for the former, > keeping both around. But perhaps it's confusing. If I had the option I'd use the former ("old") syntax, but then, I'm weird. -N From patvarilly at gmail.com Fri Jul 6 19:36:50 2012 From: patvarilly at gmail.com (Patrick Varilly) Date: Sat, 7 Jul 2012 00:36:50 +0100 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree Message-ID: Dear all, I've recently been putting together a wrapper around scipy.spatial.KDTree to transparently handle periodic boundary conditions that are common in analyzing the results of molecular simulations ( https://github.com/patvarilly/periodic_kdtree). In the process, I've started enhancing some of scipy.spatial.cKDTree by adding a Cythonized query_ball_point method. A pull request is here https://github.com/scipy/scipy/pull/262 Two things I have noticed are that the interfaces to the KDTree and cKDTree query() methods aren't quite in line, and that it wouldn't take a whole lot of extra effort to Cythonize the rest of KDTree. The innards of the code could also be reorganized a bit to make it easier for other people to extend these classes. For the periodic kd-trees, for example, I had to jump through a few hoops, and ended up with two different codebases for wrapping KDTree's and cKDTree's, which is not ideal. I was writing to gauge if there's actually any interest in cleaning up KDTrees and Cythonizing the rest of the KDTree interface. If so, I'd be willing to put a bit of effort into doing so. A delicate question that comes up is whether anyone relies on the subtle differences between KDTree.query() and cKDTree.query() (e.g., passing k = None), so that bringing the two interfaces exactly in line with each other would break existing code. How is this issue usually dealt with? All the best, Patrick -------------- next part -------------- An HTML attachment was scrubbed... URL: From aia8v at virginia.edu Sat Jul 7 21:50:03 2012 From: aia8v at virginia.edu (Alexander Arsenovic) Date: Sat, 7 Jul 2012 21:50:03 -0400 Subject: [SciPy-Dev] SciPy-Dev Digest, Vol 105, Issue 9 In-Reply-To: References: Message-ID: thanks man. by the way, the docs look really good. the `ggplot` style is great. it would be cool to get journals to participate by making formal style files. i emailed ieee once about this and never heard back. maybe you could get them to endorse mpltools as a way to make an acceptably formatted plot, which would boost your user base. just a thought. alex On Sat, Jul 7, 2012 at 1:00 PM, wrote: > Send SciPy-Dev mailing list submissions to > scipy-dev at scipy.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.scipy.org/mailman/listinfo/scipy-dev > or, via email, send a message with subject or body 'help' to > scipy-dev-request at scipy.org > > You can reach the person managing the list at > scipy-dev-owner at scipy.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of SciPy-Dev digest..." > > > Today's Topics: > > 1. Re: interp2d bounds checking (Pauli Virtanen) > 2. Re: interp2d bounds checking (Nathaniel Smith) > 3. Enhancements to scipy.spatial.cKDTree (Patrick Varilly) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 06 Jul 2012 20:51:53 +0200 > From: Pauli Virtanen > Subject: Re: [SciPy-Dev] interp2d bounds checking > To: scipy-dev at scipy.org > Message-ID: > Content-Type: text/plain; charset=ISO-8859-1 > > 06.07.2012 18:19, Ralf Gommers kirjoitti: > > On Sun, Jul 1, 2012 at 8:40 PM, Pauli Virtanen wrote: > [clip] > > I've also been thinking about enlarging the griddata syntax from the > > possibly somewhat unfriendly > > > > griddata((x, y), z, (xi, yi), **kw) > > griddata((x, y, z), u, (xi, yi, zi), **kw) > > > > to the variable argument one > > > > griddata(x, y, z, xi, yi, **kw) > > griddata(x, y, z, u, xi, yi, zi, **kw) > > > > which may be more familiar to users. Thoughts? > > > > That does look easier to use, but having two interfaces will be > > confusing. And deprecating the old one will be painful. What exactly do > > you have in mind to do here? > > I was thinking about making the latter syntax an alias for the former, > keeping both around. But perhaps it's confusing. > > Another issue is that variable arguments don't play too well with > function signatures. This is BTW one thing that is better in Python 3, > which has keyword-only arguments. > > Pauli > > > > ------------------------------ > > Message: 2 > Date: Fri, 6 Jul 2012 20:28:21 +0100 > From: Nathaniel Smith > Subject: Re: [SciPy-Dev] interp2d bounds checking > To: SciPy Developers List > Message-ID: > y2MiMdqTX+bOAfBtgDi9SMFkpejtxatPYVb8gXnY3sA at mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > On Fri, Jul 6, 2012 at 7:51 PM, Pauli Virtanen wrote: > > 06.07.2012 18:19, Ralf Gommers kirjoitti: > >> On Sun, Jul 1, 2012 at 8:40 PM, Pauli Virtanen wrote: > > [clip] > >> I've also been thinking about enlarging the griddata syntax from the > >> possibly somewhat unfriendly > >> > >> griddata((x, y), z, (xi, yi), **kw) > >> griddata((x, y, z), u, (xi, yi, zi), **kw) > >> > >> to the variable argument one > >> > >> griddata(x, y, z, xi, yi, **kw) > >> griddata(x, y, z, u, xi, yi, zi, **kw) > >> > >> which may be more familiar to users. Thoughts? > >> > >> That does look easier to use, but having two interfaces will be > >> confusing. And deprecating the old one will be painful. What exactly do > >> you have in mind to do here? > > > > I was thinking about making the latter syntax an alias for the former, > > keeping both around. But perhaps it's confusing. > > If I had the option I'd use the former ("old") syntax, but then, I'm weird. > > -N > > > ------------------------------ > > Message: 3 > Date: Sat, 7 Jul 2012 00:36:50 +0100 > From: Patrick Varilly > Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree > To: scipy-dev at scipy.org > Message-ID: > < > CAHEe8R-y38kBx0an6RmWeVtTegRd2UKPROtyZcCzXLSCSMiviw at mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > Dear all, > > I've recently been putting together a wrapper around scipy.spatial.KDTree > to transparently handle periodic boundary conditions that are common in > analyzing the results of molecular simulations ( > https://github.com/patvarilly/periodic_kdtree). In the process, I've > started enhancing some of scipy.spatial.cKDTree by adding a Cythonized > query_ball_point method. A pull request is here > > https://github.com/scipy/scipy/pull/262 > > Two things I have noticed are that the interfaces to the KDTree and cKDTree > query() methods aren't quite in line, and that it wouldn't take a whole lot > of extra effort to Cythonize the rest of KDTree. The innards of the code > could also be reorganized a bit to make it easier for other people to > extend these classes. For the periodic kd-trees, for example, I had to > jump through a few hoops, and ended up with two different codebases for > wrapping KDTree's and cKDTree's, which is not ideal. > > I was writing to gauge if there's actually any interest in cleaning up > KDTrees and Cythonizing the rest of the KDTree interface. If so, I'd be > willing to put a bit of effort into doing so. A delicate question that > comes up is whether anyone relies on the subtle differences between > KDTree.query() and cKDTree.query() (e.g., passing k = None), so that > bringing the two interfaces exactly in line with each other would break > existing code. How is this issue usually dealt with? > > All the best, > > Patrick > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://mail.scipy.org/pipermail/scipy-dev/attachments/20120707/3e8f9a73/attachment-0001.html > > ------------------------------ > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > > End of SciPy-Dev Digest, Vol 105, Issue 9 > ***************************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanderplas at astro.washington.edu Sun Jul 8 11:10:25 2012 From: vanderplas at astro.washington.edu (Jake Vanderplas) Date: Sun, 08 Jul 2012 08:10:25 -0700 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: References: Message-ID: <4FF9A2E1.9040100@astro.washington.edu> Hi, Thanks for the PR, Patrick. Regarding the interface change: I personally have never used KDTree: in my experience it's simply too slow to do anything meaningful. cKDTree is very fast in some applications, however, and I have used it in research and in several packages which depend on scipy. If we're going to change an interface, we should change KDTree rather than cKDTree. It will likely affect fewer people. I wonder if, in addition, we should think about deprecating KDTree. It would be less confusing in the long run to have only the one object available, especially if the last few pieces of functionality can be ported to cKDTree. Jake On 07/06/2012 04:36 PM, Patrick Varilly wrote: > Dear all, > > I've recently been putting together a wrapper around > scipy.spatial.KDTree to transparently handle periodic boundary > conditions that are common in analyzing the results of molecular > simulations (https://github.com/patvarilly/periodic_kdtree). In the > process, I've started enhancing some of scipy.spatial.cKDTree by > adding a Cythonized query_ball_point method. A pull request is here > > https://github.com/scipy/scipy/pull/262 > > Two things I have noticed are that the interfaces to the KDTree and > cKDTree query() methods aren't quite in line, and that it wouldn't > take a whole lot of extra effort to Cythonize the rest of KDTree. The > innards of the code could also be reorganized a bit to make it easier > for other people to extend these classes. For the periodic kd-trees, > for example, I had to jump through a few hoops, and ended up with two > different codebases for wrapping KDTree's and cKDTree's, which is not > ideal. > > I was writing to gauge if there's actually any interest in cleaning up > KDTrees and Cythonizing the rest of the KDTree interface. If so, I'd > be willing to put a bit of effort into doing so. A delicate question > that comes up is whether anyone relies on the subtle differences > between KDTree.query() and cKDTree.query() (e.g., passing k = None), > so that bringing the two interfaces exactly in line with each other > would break existing code. How is this issue usually dealt with? > > All the best, > > Patrick > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From patvarilly at gmail.com Sun Jul 8 19:30:09 2012 From: patvarilly at gmail.com (Patrick Varilly) Date: Mon, 9 Jul 2012 00:30:09 +0100 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: <4FF9A2E1.9040100@astro.washington.edu> References: <4FF9A2E1.9040100@astro.washington.edu> Message-ID: Hi, Jake, I agree that the bare KDTree is painfully slow, and deprecating it once cKDTree has all its functionality sounds like the right thing to do. I've now finished porting over query_ball_tree and added it to the pull request, and the rest (query_pairs, count_neighbors and sparse_distance_matrix) shouldn't take too much longer, since they're conceptually very similar to query_ball_tree. That said, one good reason to keep the KDTree code alive, even if deprecated, is that it's very readable. This is useful if you want to know how each of the algorithms work. All the best, Patrick On Sun, Jul 8, 2012 at 4:10 PM, Jake Vanderplas < vanderplas at astro.washington.edu> wrote: > Hi, > Thanks for the PR, Patrick. > > Regarding the interface change: I personally have never used KDTree: in my > experience it's simply too slow to do anything meaningful. cKDTree is very > fast in some applications, however, and I have used it in research and in > several packages which depend on scipy. If we're going to change an > interface, we should change KDTree rather than cKDTree. It will likely > affect fewer people. > > I wonder if, in addition, we should think about deprecating KDTree. It > would be less confusing in the long run to have only the one object > available, especially if the last few pieces of functionality can be ported > to cKDTree. > Jake > > > On 07/06/2012 04:36 PM, Patrick Varilly wrote: > > Dear all, > > I've recently been putting together a wrapper around scipy.spatial.KDTree > to transparently handle periodic boundary conditions that are common in > analyzing the results of molecular simulations ( > https://github.com/patvarilly/periodic_kdtree). In the process, I've > started enhancing some of scipy.spatial.cKDTree by adding a Cythonized > query_ball_point method. A pull request is here > > https://github.com/scipy/scipy/pull/262 > > Two things I have noticed are that the interfaces to the KDTree and > cKDTree query() methods aren't quite in line, and that it wouldn't take a > whole lot of extra effort to Cythonize the rest of KDTree. The innards of > the code could also be reorganized a bit to make it easier for other people > to extend these classes. For the periodic kd-trees, for example, I had to > jump through a few hoops, and ended up with two different codebases for > wrapping KDTree's and cKDTree's, which is not ideal. > > I was writing to gauge if there's actually any interest in cleaning up > KDTrees and Cythonizing the rest of the KDTree interface. If so, I'd be > willing to put a bit of effort into doing so. A delicate question that > comes up is whether anyone relies on the subtle differences between > KDTree.query() and cKDTree.query() (e.g., passing k = None), so that > bringing the two interfaces exactly in line with each other would break > existing code. How is this issue usually dealt with? > > All the best, > > Patrick > > > _______________________________________________ > SciPy-Dev mailing listSciPy-Dev at scipy.orghttp://mail.scipy.org/mailman/listinfo/scipy-dev > > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericfode at gmail.com Mon Jul 9 14:51:57 2012 From: ericfode at gmail.com (Fode) Date: Mon, 9 Jul 2012 14:51:57 -0400 Subject: [SciPy-Dev] SSE optimization Message-ID: I would like to add sse optimization to numpy if that is still something the project is interested in. Do you have any suggestions as to where to start optimizing. Fode -------------- next part -------------- An HTML attachment was scrubbed... URL: From pepijndevos at yahoo.com Mon Jul 9 18:41:54 2012 From: pepijndevos at yahoo.com (pepijn de vos) Date: Mon, 9 Jul 2012 15:41:54 -0700 (PDT) Subject: [SciPy-Dev] f2py type patch submitted Message-ID: <1341873714.68106.YahooMailNeo@web120104.mail.ne1.yahoo.com> Hi, This is my first patch to NymPy. Hopefully more is to come this week. http://projects.scipy.org/numpy/ticket/2183 This patch is brought to you by the Hacker School open source week ;-) Pepijn de Vos -------------- next part -------------- An HTML attachment was scrubbed... URL: From emmanuelle.gouillart at normalesup.org Tue Jul 10 04:09:56 2012 From: emmanuelle.gouillart at normalesup.org (Emmanuelle Gouillart) Date: Tue, 10 Jul 2012 10:09:56 +0200 Subject: [SciPy-Dev] Euroscipy 2012: early bird registration ending soon Message-ID: <20120710080956.GA23990@phare.normalesup.org> Hello, early bird registration for Euroscipy 2012 is soon coming to an end, with the deadline on July 22nd. Don't forget to register soon! Reduced fees are available for academics, students and speakers. Registration takes place online on http://www.euroscipy.org/conference/euroscipy2012. Euroscipy 2012 is the annual European conference for scientists using the Python language. It will be held August 23-27 2012 in Brussels, Belgium. Our program has been online for a few weeks now: we're very excited to have a great selection of tutorials, as well as talks and poster presentations. During the two days of tutorials (August 23 and 24), it will possible to attend either the introduction track, or the advanced track, or to combine both tracks (see http://www.euroscipy.org/track/6538?tab=tracktalkslist and http://www.euroscipy.org/track/6539?tab=tracktalkslist). As for the highlights of the two days of conference (August 25 and 26), we are very happy to have David Beazley (http://www.dabeaz.com) and Eric Jones (http://www.enthought.com/company/support-team.php) as our keynote speakers. The list of talks is available on http://www.euroscipy.org/track/6540?tab=tracktalkslist, with subjects ranging from extension programming to machine learning, or cellular biology. We're looking forward to exciting discussions during the talk sessions or around the posters, as happened during the previous editions of Euroscipy! Sprints may be organized at the conference venue during the days following the conference, from Monday 27th on. Since there is a limited number of rooms booked for the sprints, please contact the organizers by July 22 if you intend to organize one. Two sprints are already planned by the scikit-learn and the scikits-image teams. The EuroSciPy 2012 conference will feature a best talk, a best poster and a jury award. All conference participants will be given the opportunity to cast a vote for the best talk and best poster awards while the jury award is selected by the members of the program committee. Each prize consists of a Commercial Use license for Wing IDE Professional, an integrated development environment designed specifically for Python programmers. The licenses are generously donated by Wingware. Financial support may be granted by Numfocus to a small number of eligible students. See http://www.euroscipy.org/card/euroscipy2012_support_numfocus for more details on how to apply. For information that cannot be found on the conference website, please contact the organizing team at org-team at lists.euroscipy.org Cheers, Emmanuelle, for the organizing team From ralf.gommers at googlemail.com Tue Jul 10 06:17:08 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 10 Jul 2012 12:17:08 +0200 Subject: [SciPy-Dev] SSE optimization In-Reply-To: References: Message-ID: Hi Eric, I'm not sure what the answer is, but the numpy-discussion at scipy.org list would be the right place to ask this question. Cheers, Ralf On Mon, Jul 9, 2012 at 8:51 PM, Fode wrote: > I would like to add sse optimization to numpy if that is still something > the project is interested in. Do you have any suggestions as to where to > start optimizing. > > Fode > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Tue Jul 10 07:01:00 2012 From: sturla at molden.no (Sturla Molden) Date: Tue, 10 Jul 2012 13:01:00 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: References: Message-ID: <4FFC0B6C.5030407@molden.no> On 07.07.2012 01:36, Patrick Varilly wrote: > I was writing to gauge if there's actually any interest in cleaning up > KDTrees and Cythonizing the rest of the KDTree interface. If so, I'd be > willing to put a bit of effort into doing so. A delicate question that > comes up is whether anyone relies on the subtle differences between > KDTree.query() and cKDTree.query() (e.g., passing k = None), so that > bringing the two interfaces exactly in line with each other would break > existing code. How is this issue usually dealt with? At least cKDTree have to be fixed, it will break as soon as the move to PyArray_DATA is mandatory. Preferably we should use Cython memoryviews and multidimensional arrays in the code, instead of just C pointer artithmetics (which is harder to understand). That will make the Cython code more readable to NumPy users. The GIL issue should also be fixed, as searching might take a while. I don't have time to do this now, but I might in the summer vacation. Preferably I'd like to see KDTree inherit from cKDTree. (I have numerous versions of cKDTree on my computer, including one that works with Python lists and a multithreaded versions.) Sturla From ralf.gommers at googlemail.com Tue Jul 10 10:13:16 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 10 Jul 2012 16:13:16 +0200 Subject: [SciPy-Dev] update on 0.11.0 status Message-ID: Hi all, Here's a short update on the status of the 0.11.0 release. Some of the reported issues for the beta release have been fixed or PRs for them are waiting to be merged. There is one failure not yet fixed (sparse.csgrapg.shortest_path failure) and one blocker also in the sparse.csgraph module: http://projects.scipy.org/scipy/ticket/1681. Once the csgraph issues are fixed we're ready to release the first RC. To be merged and backported: Umfpack failure: https://github.com/scipy/scipy/pull/257 Interpolate failure under Python 3.x: https://github.com/scipy/scipy/pull/258 To not fix: weave error when running the tests under Python 3.x. It's not easy to avoid the printed error, because importing the whole module fails. And in the end the result is the same, weave doesn't work at all for 3.x. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From josh.k.lawrence at gmail.com Tue Jul 10 11:30:21 2012 From: josh.k.lawrence at gmail.com (Josh Lawrence) Date: Tue, 10 Jul 2012 11:30:21 -0400 Subject: [SciPy-Dev] update on 0.11.0 status In-Reply-To: References: Message-ID: <5B03815A-88E4-4C22-A744-031462DAAB57@gmail.com> Also this pull request just needs to be reviewed by one of the maintainers of scipy.signal, right? https://github.com/scipy/scipy/pull/259 --Josh On Jul 10, 2012, at 10:13 AM, Ralf Gommers wrote: > Hi all, > > Here's a short update on the status of the 0.11.0 release. Some of the reported issues for the beta release have been fixed or PRs for them are waiting to be merged. There is one failure not yet fixed (sparse.csgrapg.shortest_path failure) and one blocker also in the sparse.csgraph module: http://projects.scipy.org/scipy/ticket/1681. Once the csgraph issues are fixed we're ready to release the first RC. > > To be merged and backported: > Umfpack failure: https://github.com/scipy/scipy/pull/257 > Interpolate failure under Python 3.x: https://github.com/scipy/scipy/pull/258 > > To not fix: weave error when running the tests under Python 3.x. It's not easy to avoid the printed error, because importing the whole module fails. And in the end the result is the same, weave doesn't work at all for 3.x. > > Cheers, > Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Tue Jul 10 11:32:59 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 10 Jul 2012 17:32:59 +0200 Subject: [SciPy-Dev] update on 0.11.0 status In-Reply-To: <5B03815A-88E4-4C22-A744-031462DAAB57@gmail.com> References: <5B03815A-88E4-4C22-A744-031462DAAB57@gmail.com> Message-ID: On Tue, Jul 10, 2012 at 5:30 PM, Josh Lawrence wrote: > Also this pull request just needs to be reviewed by one of the maintainers > of scipy.signal, right? > > https://github.com/scipy/scipy/pull/259 > Yes, would be good to get that in as well. If no one reviews it in time though, it may have to wait till 0.12.0. Ralf > > --Josh > > On Jul 10, 2012, at 10:13 AM, Ralf Gommers wrote: > > Hi all, > > Here's a short update on the status of the 0.11.0 release. Some of the > reported issues for the beta release have been fixed or PRs for them are > waiting to be merged. There is one failure not yet fixed > (sparse.csgrapg.shortest_path failure) and one blocker also in the > sparse.csgraph module: http://projects.scipy.org/scipy/ticket/1681. Once > the csgraph issues are fixed we're ready to release the first RC. > > To be merged and backported: > Umfpack failure: https://github.com/scipy/scipy/pull/257 > Interpolate failure under Python 3.x: > https://github.com/scipy/scipy/pull/258 > > To not fix: weave error when running the tests under Python 3.x. It's not > easy to avoid the printed error, because importing the whole module fails. > And in the end the result is the same, weave doesn't work at all for 3.x. > > Cheers, > Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanderplas at astro.washington.edu Tue Jul 10 12:25:30 2012 From: vanderplas at astro.washington.edu (Jake Vanderplas) Date: Tue, 10 Jul 2012 09:25:30 -0700 Subject: [SciPy-Dev] update on 0.11.0 status In-Reply-To: References: Message-ID: <4FFC577A.6070007@astro.washington.edu> Hi, The graph_laplacian issue is fixed in PR 266, which I just opened [1]. There is still the related but larger issue of errant fancy indexing in LIL matrix, discussed in ticket 1681 [2]. The LIL issue is more complicated, as it has potential for breaking backward compatibility. I have still been unable to reproduce the csgraph.shortest_path failure. Jake [1] https://github.com/scipy/scipy/pull/266 [2] http://projects.scipy.org/scipy/ticket/1681 On 07/10/2012 07:13 AM, Ralf Gommers wrote: > Hi all, > > Here's a short update on the status of the 0.11.0 release. Some of the > reported issues for the beta release have been fixed or PRs for them > are waiting to be merged. There is one failure not yet fixed > (sparse.csgrapg.shortest_path failure) and one blocker also in the > sparse.csgraph module: http://projects.scipy.org/scipy/ticket/1681. > Once the csgraph issues are fixed we're ready to release the first RC. > > To be merged and backported: > Umfpack failure: https://github.com/scipy/scipy/pull/257 > Interpolate failure under Python 3.x: > https://github.com/scipy/scipy/pull/258 > > To not fix: weave error when running the tests under Python 3.x. It's > not easy to avoid the printed error, because importing the whole > module fails. And in the end the result is the same, weave doesn't > work at all for 3.x. > > Cheers, > Ralf > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Tue Jul 10 16:34:40 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 10 Jul 2012 22:34:40 +0200 Subject: [SciPy-Dev] update on 0.11.0 status In-Reply-To: <4FFC577A.6070007@astro.washington.edu> References: <4FFC577A.6070007@astro.washington.edu> Message-ID: On Tue, Jul 10, 2012 at 6:25 PM, Jake Vanderplas < vanderplas at astro.washington.edu> wrote: > Hi, > The graph_laplacian issue is fixed in PR 266, which I just opened [1]. > There is still the related but larger issue of errant fancy indexing in LIL > matrix, discussed in ticket 1681 [2]. The LIL issue is more complicated, > as it has potential for breaking backward compatibility. > > Great. If that is fixed, I'm not sure if the LIL issue is still a blocker. I have still been unable to reproduce the csgraph.shortest_path failure. > I can reproduce this for about 1 in 10 runs, which is really weird because I can't find any randomness in the test or code. Plus, it once again shows how annoying generator tests are; can't see which method is failing. Failures are also not always the same: x and y nan location mismatch: x: array([[ 0., nan, 3., 2., 4.], [ nan, nan, nan, nan, nan], [ 1., nan, 0., 3., 5.], [ 1., nan, 4., 0., 3.]]) y: array([[ 0., 3., 3., 1., 2.], [ 3., 0., 6., 2., 4.], [ 3., 6., 0., 4., 5.], [ 1., 2., 4., 0., 2.]]) (mismatch 30.0%) x: array([[ 0., 3., 3., 2., 4.], [ 3., 0., 6., 2., 4.], [ 3., 6., 0., 5., 7.], [ 1., 1., 4., 0., 3.]]) y: array([[ 0., 3., 3., 1., 2.], [ 3., 0., 6., 2., 4.], [ 3., 6., 0., 4., 5.], [ 1., 2., 4., 0., 2.]]) Any idea where to look? Ralf > Jake > > [1] https://github.com/scipy/scipy/pull/266 > [2] http://projects.scipy.org/scipy/ticket/1681 > > > On 07/10/2012 07:13 AM, Ralf Gommers wrote: > > Hi all, > > Here's a short update on the status of the 0.11.0 release. Some of the > reported issues for the beta release have been fixed or PRs for them are > waiting to be merged. There is one failure not yet fixed > (sparse.csgrapg.shortest_path failure) and one blocker also in the > sparse.csgraph module: http://projects.scipy.org/scipy/ticket/1681. Once > the csgraph issues are fixed we're ready to release the first RC. > > To be merged and backported: > Umfpack failure: https://github.com/scipy/scipy/pull/257 > Interpolate failure under Python 3.x: > https://github.com/scipy/scipy/pull/258 > > To not fix: weave error when running the tests under Python 3.x. It's not > easy to avoid the printed error, because importing the whole module fails. > And in the end the result is the same, weave doesn't work at all for 3.x. > > Cheers, > Ralf > > > > _______________________________________________ > SciPy-Dev mailing listSciPy-Dev at scipy.orghttp://mail.scipy.org/mailman/listinfo/scipy-dev > > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanderplas at astro.washington.edu Tue Jul 10 16:52:44 2012 From: vanderplas at astro.washington.edu (Jake Vanderplas) Date: Tue, 10 Jul 2012 13:52:44 -0700 Subject: [SciPy-Dev] update on 0.11.0 status In-Reply-To: References: <4FFC577A.6070007@astro.washington.edu> Message-ID: <4FFC961C.8060605@astro.washington.edu> On 07/10/2012 01:34 PM, Ralf Gommers wrote: > > > On Tue, Jul 10, 2012 at 6:25 PM, Jake Vanderplas > > wrote: > > Hi, > The graph_laplacian issue is fixed in PR 266, which I just opened > [1]. There is still the related but larger issue of errant fancy > indexing in LIL matrix, discussed in ticket 1681 [2]. The LIL > issue is more complicated, as it has potential for breaking > backward compatibility. > > Great. If that is fixed, I'm not sure if the LIL issue is still a blocker. > > I have still been unable to reproduce the csgraph.shortest_path > failure. > > > I can reproduce this for about 1 in 10 runs, which is really weird > because I can't find any randomness in the test or code. Plus, it once > again shows how annoying generator tests are; can't see which method > is failing. Failures are also not always the same: > > > x and y nan location mismatch: > x: array([[ 0., nan, 3., 2., 4.], > [ nan, nan, nan, nan, nan], > [ 1., nan, 0., 3., 5.], > [ 1., nan, 4., 0., 3.]]) > y: array([[ 0., 3., 3., 1., 2.], > [ 3., 0., 6., 2., 4.], > [ 3., 6., 0., 4., 5.], > [ 1., 2., 4., 0., 2.]]) > > > > (mismatch 30.0%) > x: array([[ 0., 3., 3., 2., 4.], > [ 3., 0., 6., 2., 4.], > [ 3., 6., 0., 5., 7.], > [ 1., 1., 4., 0., 3.]]) > y: array([[ 0., 3., 3., 1., 2.], > [ 3., 0., 6., 2., 4.], > [ 3., 6., 0., 4., 5.], > [ 1., 2., 4., 0., 2.]]) > > > Any idea where to look? I just ran the test_shortest_path.py code a few dozen times, and didn't have any failures. Both these cases above seem to be from the `test_shortest_path_indices()` function, for indshape=(4,). Is that the only test you get errors from? Is the y array the same in each failure? The fact that it's not happening every time for you makes it sound like a memory issue. If that's the case, I suspect some variable is uninitialized in the Fibonacci heap code, as that's the only part of the code that uses raw pointers. I'll try running valgrind to see if I can find the culprit. Jake > > Ralf > > > Jake > > [1] https://github.com/scipy/scipy/pull/266 > [2] http://projects.scipy.org/scipy/ticket/1681 > > > On 07/10/2012 07:13 AM, Ralf Gommers wrote: >> Hi all, >> >> Here's a short update on the status of the 0.11.0 release. Some >> of the reported issues for the beta release have been fixed or >> PRs for them are waiting to be merged. There is one failure not >> yet fixed (sparse.csgrapg.shortest_path failure) and one blocker >> also in the sparse.csgraph module: >> http://projects.scipy.org/scipy/ticket/1681. Once the csgraph >> issues are fixed we're ready to release the first RC. >> >> To be merged and backported: >> Umfpack failure: https://github.com/scipy/scipy/pull/257 >> Interpolate failure under Python 3.x: >> https://github.com/scipy/scipy/pull/258 >> >> To not fix: weave error when running the tests under Python 3.x. >> It's not easy to avoid the printed error, because importing the >> whole module fails. And in the end the result is the same, weave >> doesn't work at all for 3.x. >> >> Cheers, >> Ralf >> >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Tue Jul 10 17:00:12 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 10 Jul 2012 23:00:12 +0200 Subject: [SciPy-Dev] update on 0.11.0 status In-Reply-To: <4FFC961C.8060605@astro.washington.edu> References: <4FFC577A.6070007@astro.washington.edu> <4FFC961C.8060605@astro.washington.edu> Message-ID: On Tue, Jul 10, 2012 at 10:52 PM, Jake Vanderplas < vanderplas at astro.washington.edu> wrote: > On 07/10/2012 01:34 PM, Ralf Gommers wrote: > > > > On Tue, Jul 10, 2012 at 6:25 PM, Jake Vanderplas < > vanderplas at astro.washington.edu> wrote: > >> Hi, >> The graph_laplacian issue is fixed in PR 266, which I just opened [1]. >> There is still the related but larger issue of errant fancy indexing in LIL >> matrix, discussed in ticket 1681 [2]. The LIL issue is more complicated, >> as it has potential for breaking backward compatibility. >> >> Great. If that is fixed, I'm not sure if the LIL issue is still a > blocker. > > I have still been unable to reproduce the csgraph.shortest_path failure. >> > > I can reproduce this for about 1 in 10 runs, which is really weird because > I can't find any randomness in the test or code. Plus, it once again shows > how annoying generator tests are; can't see which method is failing. > Failures are also not always the same: > > > x and y nan location mismatch: > x: array([[ 0., nan, 3., 2., 4.], > [ nan, nan, nan, nan, nan], > [ 1., nan, 0., 3., 5.], > [ 1., nan, 4., 0., 3.]]) > y: array([[ 0., 3., 3., 1., 2.], > [ 3., 0., 6., 2., 4.], > [ 3., 6., 0., 4., 5.], > [ 1., 2., 4., 0., 2.]]) > > > > (mismatch 30.0%) > x: array([[ 0., 3., 3., 2., 4.], > [ 3., 0., 6., 2., 4.], > [ 3., 6., 0., 5., 7.], > [ 1., 1., 4., 0., 3.]]) > y: array([[ 0., 3., 3., 1., 2.], > [ 3., 0., 6., 2., 4.], > [ 3., 6., 0., 4., 5.], > [ 1., 2., 4., 0., 2.]]) > > > Any idea where to look? > > > I just ran the test_shortest_path.py code a few dozen times, and didn't > have any failures. Both these cases above seem to be from the > `test_shortest_path_indices()` function, for indshape=(4,). Is that the > only test you get errors from? Is the y array the same in each failure? > Seems to be the same each time. Ralf > > The fact that it's not happening every time for you makes it sound like a > memory issue. If that's the case, I suspect some variable is uninitialized > in the Fibonacci heap code, as that's the only part of the code that uses > raw pointers. > > I'll try running valgrind to see if I can find the culprit. > Jake > > > > Ralf > > > >> Jake >> >> [1] https://github.com/scipy/scipy/pull/266 >> [2] http://projects.scipy.org/scipy/ticket/1681 >> >> >> On 07/10/2012 07:13 AM, Ralf Gommers wrote: >> >> Hi all, >> >> Here's a short update on the status of the 0.11.0 release. Some of the >> reported issues for the beta release have been fixed or PRs for them are >> waiting to be merged. There is one failure not yet fixed >> (sparse.csgrapg.shortest_path failure) and one blocker also in the >> sparse.csgraph module: http://projects.scipy.org/scipy/ticket/1681. Once >> the csgraph issues are fixed we're ready to release the first RC. >> >> To be merged and backported: >> Umfpack failure: https://github.com/scipy/scipy/pull/257 >> Interpolate failure under Python 3.x: >> https://github.com/scipy/scipy/pull/258 >> >> To not fix: weave error when running the tests under Python 3.x. It's not >> easy to avoid the printed error, because importing the whole module fails. >> And in the end the result is the same, weave doesn't work at all for 3.x. >> >> Cheers, >> Ralf >> >> >> >> _______________________________________________ >> SciPy-Dev mailing listSciPy-Dev at scipy.orghttp://mail.scipy.org/mailman/listinfo/scipy-dev >> >> >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev >> >> > > > _______________________________________________ > SciPy-Dev mailing listSciPy-Dev at scipy.orghttp://mail.scipy.org/mailman/listinfo/scipy-dev > > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanderplas at astro.washington.edu Tue Jul 10 17:33:41 2012 From: vanderplas at astro.washington.edu (Jake Vanderplas) Date: Tue, 10 Jul 2012 14:33:41 -0700 Subject: [SciPy-Dev] update on 0.11.0 status In-Reply-To: References: <4FFC577A.6070007@astro.washington.edu> <4FFC961C.8060605@astro.washington.edu> Message-ID: <4FFC9FB5.6090006@astro.washington.edu> Ralf, I tried running valgrind on the offending code. I don't see anything out of the ordinary come up on my system (for reference, the scikit-learn developer documentation has some tips on using valgrind for cython code [1]). This makes me suspect that the error you're seeing may be a system-specific issue (i.e. 32 bit/64 bit mismatch), or perhaps something related to the numpy version (I'm using the current numpy 1.8.0-dev). Could you run valgrind on your system with test_shortest_path.py and see if anything obvious comes up? Thanks Jake [1] http://scikit-learn.org/stable/developers/debugging.html On 07/10/2012 02:00 PM, Ralf Gommers wrote: > > > On Tue, Jul 10, 2012 at 10:52 PM, Jake Vanderplas > > wrote: > > On 07/10/2012 01:34 PM, Ralf Gommers wrote: >> >> >> On Tue, Jul 10, 2012 at 6:25 PM, Jake Vanderplas >> > > wrote: >> >> Hi, >> The graph_laplacian issue is fixed in PR 266, which I just >> opened [1]. There is still the related but larger issue of >> errant fancy indexing in LIL matrix, discussed in ticket 1681 >> [2]. The LIL issue is more complicated, as it has potential >> for breaking backward compatibility. >> >> Great. If that is fixed, I'm not sure if the LIL issue is still a >> blocker. >> >> I have still been unable to reproduce the >> csgraph.shortest_path failure. >> >> >> I can reproduce this for about 1 in 10 runs, which is really >> weird because I can't find any randomness in the test or code. >> Plus, it once again shows how annoying generator tests are; can't >> see which method is failing. Failures are also not always the same: >> >> >> x and y nan location mismatch: >> x: array([[ 0., nan, 3., 2., 4.], >> [ nan, nan, nan, nan, nan], >> [ 1., nan, 0., 3., 5.], >> [ 1., nan, 4., 0., 3.]]) >> y: array([[ 0., 3., 3., 1., 2.], >> [ 3., 0., 6., 2., 4.], >> [ 3., 6., 0., 4., 5.], >> [ 1., 2., 4., 0., 2.]]) >> >> >> >> (mismatch 30.0%) >> x: array([[ 0., 3., 3., 2., 4.], >> [ 3., 0., 6., 2., 4.], >> [ 3., 6., 0., 5., 7.], >> [ 1., 1., 4., 0., 3.]]) >> y: array([[ 0., 3., 3., 1., 2.], >> [ 3., 0., 6., 2., 4.], >> [ 3., 6., 0., 4., 5.], >> [ 1., 2., 4., 0., 2.]]) >> >> >> Any idea where to look? > > I just ran the test_shortest_path.py code a few dozen times, and > didn't have any failures. Both these cases above seem to be from > the `test_shortest_path_indices()` function, for indshape=(4,). > Is that the only test you get errors from? Is the y array the same > in each failure? > > > Seems to be the same each time. > > Ralf > > > The fact that it's not happening every time for you makes it sound > like a memory issue. If that's the case, I suspect some variable > is uninitialized in the Fibonacci heap code, as that's the only > part of the code that uses raw pointers. > > I'll try running valgrind to see if I can find the culprit. > Jake > > >> >> Ralf >> >> >> Jake >> >> [1] https://github.com/scipy/scipy/pull/266 >> [2] http://projects.scipy.org/scipy/ticket/1681 >> >> >> On 07/10/2012 07:13 AM, Ralf Gommers wrote: >>> Hi all, >>> >>> Here's a short update on the status of the 0.11.0 release. >>> Some of the reported issues for the beta release have been >>> fixed or PRs for them are waiting to be merged. There is one >>> failure not yet fixed (sparse.csgrapg.shortest_path failure) >>> and one blocker also in the sparse.csgraph module: >>> http://projects.scipy.org/scipy/ticket/1681. Once the >>> csgraph issues are fixed we're ready to release the first RC. >>> >>> To be merged and backported: >>> Umfpack failure: https://github.com/scipy/scipy/pull/257 >>> Interpolate failure under Python 3.x: >>> https://github.com/scipy/scipy/pull/258 >>> >>> To not fix: weave error when running the tests under Python >>> 3.x. It's not easy to avoid the printed error, because >>> importing the whole module fails. And in the end the result >>> is the same, weave doesn't work at all for 3.x. >>> >>> Cheers, >>> Ralf >>> >>> >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-dev >> >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev >> >> >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Wed Jul 11 17:45:45 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 11 Jul 2012 23:45:45 +0200 Subject: [SciPy-Dev] update on 0.11.0 status In-Reply-To: <4FFC9FB5.6090006@astro.washington.edu> References: <4FFC577A.6070007@astro.washington.edu> <4FFC961C.8060605@astro.washington.edu> <4FFC9FB5.6090006@astro.washington.edu> Message-ID: On Tue, Jul 10, 2012 at 11:33 PM, Jake Vanderplas < vanderplas at astro.washington.edu> wrote: > Ralf, > I tried running valgrind on the offending code. I don't see anything out > of the ordinary come up on my system (for reference, the scikit-learn > developer documentation has some tips on using valgrind for cython code > [1]). This makes me suspect that the error you're seeing may be a > system-specific issue (i.e. 32 bit/64 bit mismatch), or perhaps something > related to the numpy version (I'm using the current numpy 1.8.0-dev). > > Could you run valgrind on your system with test_shortest_path.py and see > if anything obvious comes up? > It's clean, some noise related to Python itself but nothing related to csgraph. This was 32-bit python 2.6, numpy 1.8-dev by the way. I haven't yet checked if I get the same when running in 64-bit mode on the same machine yet. Ralf Jake > > [1] http://scikit-learn.org/stable/developers/debugging.html > > > On 07/10/2012 02:00 PM, Ralf Gommers wrote: > > > > On Tue, Jul 10, 2012 at 10:52 PM, Jake Vanderplas < > vanderplas at astro.washington.edu> wrote: > >> On 07/10/2012 01:34 PM, Ralf Gommers wrote: >> >> >> >> On Tue, Jul 10, 2012 at 6:25 PM, Jake Vanderplas < >> vanderplas at astro.washington.edu> wrote: >> >>> Hi, >>> The graph_laplacian issue is fixed in PR 266, which I just opened [1]. >>> There is still the related but larger issue of errant fancy indexing in LIL >>> matrix, discussed in ticket 1681 [2]. The LIL issue is more complicated, >>> as it has potential for breaking backward compatibility. >>> >>> Great. If that is fixed, I'm not sure if the LIL issue is still a >> blocker. >> >> I have still been unable to reproduce the csgraph.shortest_path >>> failure. >>> >> >> I can reproduce this for about 1 in 10 runs, which is really weird >> because I can't find any randomness in the test or code. Plus, it once >> again shows how annoying generator tests are; can't see which method is >> failing. Failures are also not always the same: >> >> >> x and y nan location mismatch: >> x: array([[ 0., nan, 3., 2., 4.], >> [ nan, nan, nan, nan, nan], >> [ 1., nan, 0., 3., 5.], >> [ 1., nan, 4., 0., 3.]]) >> y: array([[ 0., 3., 3., 1., 2.], >> [ 3., 0., 6., 2., 4.], >> [ 3., 6., 0., 4., 5.], >> [ 1., 2., 4., 0., 2.]]) >> >> >> >> (mismatch 30.0%) >> x: array([[ 0., 3., 3., 2., 4.], >> [ 3., 0., 6., 2., 4.], >> [ 3., 6., 0., 5., 7.], >> [ 1., 1., 4., 0., 3.]]) >> y: array([[ 0., 3., 3., 1., 2.], >> [ 3., 0., 6., 2., 4.], >> [ 3., 6., 0., 4., 5.], >> [ 1., 2., 4., 0., 2.]]) >> >> >> Any idea where to look? >> >> >> I just ran the test_shortest_path.py code a few dozen times, and didn't >> have any failures. Both these cases above seem to be from the >> `test_shortest_path_indices()` function, for indshape=(4,). Is that the >> only test you get errors from? Is the y array the same in each failure? >> > > Seems to be the same each time. > > Ralf > > >> >> The fact that it's not happening every time for you makes it sound like a >> memory issue. If that's the case, I suspect some variable is uninitialized >> in the Fibonacci heap code, as that's the only part of the code that uses >> raw pointers. >> >> I'll try running valgrind to see if I can find the culprit. >> Jake >> >> >> >> Ralf >> >> >> >>> Jake >>> >>> [1] https://github.com/scipy/scipy/pull/266 >>> [2] http://projects.scipy.org/scipy/ticket/1681 >>> >>> >>> On 07/10/2012 07:13 AM, Ralf Gommers wrote: >>> >>> Hi all, >>> >>> Here's a short update on the status of the 0.11.0 release. Some of the >>> reported issues for the beta release have been fixed or PRs for them are >>> waiting to be merged. There is one failure not yet fixed >>> (sparse.csgrapg.shortest_path failure) and one blocker also in the >>> sparse.csgraph module: http://projects.scipy.org/scipy/ticket/1681. >>> Once the csgraph issues are fixed we're ready to release the first RC. >>> >>> To be merged and backported: >>> Umfpack failure: https://github.com/scipy/scipy/pull/257 >>> Interpolate failure under Python 3.x: >>> https://github.com/scipy/scipy/pull/258 >>> >>> To not fix: weave error when running the tests under Python 3.x. It's >>> not easy to avoid the printed error, because importing the whole module >>> fails. And in the end the result is the same, weave doesn't work at all for >>> 3.x. >>> >>> Cheers, >>> Ralf >>> >>> >>> >>> _______________________________________________ >>> SciPy-Dev mailing listSciPy-Dev at scipy.orghttp://mail.scipy.org/mailman/listinfo/scipy-dev >>> >>> >>> >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>> >>> >> >> >> _______________________________________________ >> SciPy-Dev mailing listSciPy-Dev at scipy.orghttp://mail.scipy.org/mailman/listinfo/scipy-dev >> >> >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev >> >> > > > _______________________________________________ > SciPy-Dev mailing listSciPy-Dev at scipy.orghttp://mail.scipy.org/mailman/listinfo/scipy-dev > > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From patvarilly at gmail.com Wed Jul 11 18:26:55 2012 From: patvarilly at gmail.com (Patrick Varilly) Date: Wed, 11 Jul 2012 23:26:55 +0100 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: <4FFC0B6C.5030407@molden.no> References: <4FFC0B6C.5030407@molden.no> Message-ID: On Tue, Jul 10, 2012 at 12:01 PM, Sturla Molden wrote: At least cKDTree have to be fixed, it will break as soon as the move to > PyArray_DATA is mandatory. > > Preferably we should use Cython memoryviews and multidimensional arrays > in the code, instead of just C pointer artithmetics (which is harder to > understand). That will make the Cython code more readable to NumPy users. > > The GIL issue should also be fixed, as searching might take a while. > I'm relatively new to Cython. Could you tell me where I could read up on these issues? I don't have time to do this now, but I might in the summer vacation. > I'm nearly done porting most of KDTree to cKDTree (only count_neighbors to go), and could contribute to this clean-up when you have the time. > Preferably I'd like to see KDTree inherit from cKDTree. > > > (I have numerous versions of cKDTree on my computer, including one that > works with Python lists and a multithreaded versions.) > This sounds very interesting, and I'm sure many people would benefit from incorporating some of these things (especially multithreading) and thus having easy access to a high-performance cKDTree. Patrick -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Thu Jul 12 12:42:42 2012 From: sturla at molden.no (Sturla Molden) Date: Thu, 12 Jul 2012 18:42:42 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: References: <4FFC0B6C.5030407@molden.no> Message-ID: <4FFEFE82.5090303@molden.no> On 12.07.2012 00:26, Patrick Varilly wrote: > On Tue, Jul 10, 2012 at 12:01 PM, Sturla Molden > wrote: > > At least cKDTree have to be fixed, it will break as soon as the move to > PyArray_DATA is mandatory. > > Preferably we should use Cython memoryviews and multidimensional arrays > in the code, instead of just C pointer artithmetics (which is harder to > understand). That will make the Cython code more readable to NumPy > users. > > The GIL issue should also be fixed, as searching might take a while. > > I'm relatively new to Cython. Could you tell me where I could read up > on these issues? The main issue is the use of the .data attribute. See here: http://wiki.cython.org/tutorials/NumpyPointerToC Another is that Cython's ndarray interface is (more or less) deprecated in favour of typed memoryviews: http://docs.cython.org/src/userguide/memoryviews.html So preferably the cKDTree code should use these, but I my experience they can generate compile-time warnings. There is also a 64-bit issue with cKDTree if I remember correctly. And the only dtype it supports is float64. We should replace the current pointer artimetics with multidimensional arrays. It had (or still has) non-portable code like dependency on unions and binary layout (tree and heap nodes). And there the issue of making it release the GIL whenever it should. So several things needs be fixed. Sturla From patvarilly at gmail.com Thu Jul 12 19:34:44 2012 From: patvarilly at gmail.com (Patrick Varilly) Date: Fri, 13 Jul 2012 00:34:44 +0100 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: <4FFEFE82.5090303@molden.no> References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> Message-ID: Alright, I've uploaded the last bit of cKDTree that was missing for it to be functionally equivalent to KDTree. As it stands, I think it's a useful addition in its own right, so it would be nice if someone else could look the code over and see if it can be merged in. Over the coming weeks, I will look into the issues that Sturla has brought up and see if I can make some progress on these. All the best, Patrick On Thu, Jul 12, 2012 at 5:42 PM, Sturla Molden wrote: > On 12.07.2012 00:26, Patrick Varilly wrote: > > On Tue, Jul 10, 2012 at 12:01 PM, Sturla Molden > > wrote: > > > > At least cKDTree have to be fixed, it will break as soon as the move > to > > PyArray_DATA is mandatory. > > > > Preferably we should use Cython memoryviews and multidimensional > arrays > > in the code, instead of just C pointer artithmetics (which is harder > to > > understand). That will make the Cython code more readable to NumPy > > users. > > > > The GIL issue should also be fixed, as searching might take a while. > > > > I'm relatively new to Cython. Could you tell me where I could read up > > on these issues? > > The main issue is the use of the .data attribute. See here: > > http://wiki.cython.org/tutorials/NumpyPointerToC > > Another is that Cython's ndarray interface is (more or less) deprecated > in favour of typed memoryviews: > > http://docs.cython.org/src/userguide/memoryviews.html > > So preferably the cKDTree code should use these, but I my experience > they can generate compile-time warnings. > > There is also a 64-bit issue with cKDTree if I remember correctly. And > the only dtype it supports is float64. We should replace the current > pointer artimetics with multidimensional arrays. It had (or still has) > non-portable code like dependency on unions and binary layout (tree and > heap nodes). And there the issue of making it release the GIL whenever > it should. So several things needs be fixed. > > Sturla > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Fri Jul 13 03:09:18 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 13 Jul 2012 09:09:18 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> Message-ID: On Fri, Jul 13, 2012 at 1:34 AM, Patrick Varilly wrote: > Alright, I've uploaded the last bit of cKDTree that was missing for it to > be functionally equivalent to KDTree. As it stands, I think it's a useful > addition in its own right, so it would be nice if someone else could look > the code over and see if it can be merged in. > > Over the coming weeks, I will look into the issues that Sturla has brought > up and see if I can make some progress on these. > Please note that memoryviews can't be used yet in scipy due them not working with python 2.4 (see https://github.com/numpy/numpy/pull/307). All the other things Sturla mentioned do sound like improvements that can be made. Cheers, Ralf > All the best, > > Patrick > > > On Thu, Jul 12, 2012 at 5:42 PM, Sturla Molden wrote: > >> On 12.07.2012 00:26, Patrick Varilly wrote: >> > On Tue, Jul 10, 2012 at 12:01 PM, Sturla Molden > > > wrote: >> > >> > At least cKDTree have to be fixed, it will break as soon as the >> move to >> > PyArray_DATA is mandatory. >> > >> > Preferably we should use Cython memoryviews and multidimensional >> arrays >> > in the code, instead of just C pointer artithmetics (which is >> harder to >> > understand). That will make the Cython code more readable to NumPy >> > users. >> > >> > The GIL issue should also be fixed, as searching might take a while. >> > >> > I'm relatively new to Cython. Could you tell me where I could read up >> > on these issues? >> >> The main issue is the use of the .data attribute. See here: >> >> http://wiki.cython.org/tutorials/NumpyPointerToC >> >> Another is that Cython's ndarray interface is (more or less) deprecated >> in favour of typed memoryviews: >> >> http://docs.cython.org/src/userguide/memoryviews.html >> >> So preferably the cKDTree code should use these, but I my experience >> they can generate compile-time warnings. >> >> There is also a 64-bit issue with cKDTree if I remember correctly. And >> the only dtype it supports is float64. We should replace the current >> pointer artimetics with multidimensional arrays. It had (or still has) >> non-portable code like dependency on unions and binary layout (tree and >> heap nodes). And there the issue of making it release the GIL whenever >> it should. So several things needs be fixed. >> >> Sturla >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev >> > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri Jul 13 05:57:01 2012 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 13 Jul 2012 09:57:01 +0000 (UTC) Subject: [SciPy-Dev] Generator tests References: <4FFC577A.6070007@astro.washington.edu> Message-ID: Ralf Gommers googlemail.com> writes: [clip] > in the test or code. Plus, it once again shows how > annoying generator tests are; can't see which method is > failing. Here's a recommendation for generator test: - No computation related to the features tested should be done in the test_* generator function, but delegated to a corresponding _check_* function (can be inside the generator, to share namespace). - The generators should be used *solely* for loops over parameters. - These parameters should *not* be arrays. If a parameter is "complicated", as in the case of ARPACK tests, wrap it into a class with a descriptive __repr__ method. Like so: def test_foo(): x = [1, 2, 3] z = 9 def check(a, b): x = a + b + z y = a - b + z assert x == y + 2*b for a in [1, 2, 3]: for b in [4, 5, 6]: yield check, a, b -- Pauli Virtanen From sturla at molden.no Fri Jul 13 07:02:09 2012 From: sturla at molden.no (Sturla Molden) Date: Fri, 13 Jul 2012 13:02:09 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> Message-ID: <50000031.7040709@molden.no> I've quickly looked at the code, most of it looks good. Care if I help? I'd like to focus on these changes first: ** Make sure the code behaves correctly on 64-bit. Currently there are numerous possibilities for integer overflow, as C int is used consistently instead of npy_intp or Py_ssize_t. ** Make sure malloc and free does not leak memory in case of a Python exception. This means either using try/finally religiously or wrap it in a Python object that use "RAII" with __cinit__, __init__ and __dealloc__. Currently it is littered with possible memory leaks. ** Raise MemoryError on malloc failure. ** Use PyMem_Malloc/PyMem_Free instead of libc, so we are sure to use the same heap as Python. ** Correct the "ndarray.data" issue, but I will refrain from using typed memoryviews to maintain compatibility with Python 2.4. That will probably mean to use the Python API directly (PyArrayObject and PyArray_DATA) instead of Cython's numpy.pxd. ** Use Cython's fused types (a bloatware generator) instead of C double to autogenerate cKDTrees for any NumPy dtype. ** Avoid using struct hacks to accomodate tree node structs of different sizes (or else we are only compatible with C99). ** Start to release the GIL wherever possible, as searching a kd-tree is expensive. (Not just for the sake of multi-threading, but also to avoid locking up the interpreter forever.) Sturla On 13.07.2012 09:09, Ralf Gommers wrote: > > > On Fri, Jul 13, 2012 at 1:34 AM, Patrick Varilly > wrote: > > Alright, I've uploaded the last bit of cKDTree that was missing for > it to be functionally equivalent to KDTree. As it stands, I think > it's a useful addition in its own right, so it would be nice if > someone else could look the code over and see if it can be merged in. > > Over the coming weeks, I will look into the issues that Sturla has > brought up and see if I can make some progress on these. > > > Please note that memoryviews can't be used yet in scipy due them not > working with python 2.4 (see https://github.com/numpy/numpy/pull/307). > All the other things Sturla mentioned do sound like improvements that > can be made. > > Cheers, > Ralf > > > > All the best, > > Patrick > > > On Thu, Jul 12, 2012 at 5:42 PM, Sturla Molden > wrote: > > On 12.07.2012 00:26, Patrick Varilly wrote: > > On Tue, Jul 10, 2012 at 12:01 PM, Sturla Molden > > > >> wrote: > > > > At least cKDTree have to be fixed, it will break as soon > as the move to > > PyArray_DATA is mandatory. > > > > Preferably we should use Cython memoryviews and > multidimensional arrays > > in the code, instead of just C pointer artithmetics > (which is harder to > > understand). That will make the Cython code more readable > to NumPy > > users. > > > > The GIL issue should also be fixed, as searching might > take a while. > > > > I'm relatively new to Cython. Could you tell me where I > could read up > > on these issues? > > The main issue is the use of the .data attribute. See here: > > http://wiki.cython.org/tutorials/NumpyPointerToC > > Another is that Cython's ndarray interface is (more or less) > deprecated > in favour of typed memoryviews: > > http://docs.cython.org/src/userguide/memoryviews.html > > So preferably the cKDTree code should use these, but I my experience > they can generate compile-time warnings. > > There is also a 64-bit issue with cKDTree if I remember > correctly. And > the only dtype it supports is float64. We should replace the current > pointer artimetics with multidimensional arrays. It had (or > still has) > non-portable code like dependency on unions and binary layout > (tree and > heap nodes). And there the issue of making it release the GIL > whenever > it should. So several things needs be fixed. > > Sturla > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From sturla at molden.no Fri Jul 13 08:19:39 2012 From: sturla at molden.no (Sturla Molden) Date: Fri, 13 Jul 2012 14:19:39 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: <50000031.7040709@molden.no> References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> <50000031.7040709@molden.no> Message-ID: <5000125B.5010607@molden.no> On 13.07.2012 13:02, Sturla Molden wrote: > ** Make sure malloc and free does not leak memory in case of a Python > exception. This means either using try/finally religiously or wrap it in > a Python object that use "RAII" with __cinit__, __init__ and > __dealloc__. Currently it is littered with possible memory leaks. > > ** Raise MemoryError on malloc failure. That is, it is NOT sufficient to always pair malloc and free as we can do in C, since Python (like C++) can raise exceptions. A Python exception in Cython can separate a malloc/free pair. That is the same issue that mandates the use of RAII in robust C++ code. This idiom is currently used all over cKDTree: ptr = stdlib.malloc(n) stdlib.free(ptr) It probably looks ok to C programmers and inexperienced C++ programmers, but it is not. If raises any Python exception, we can have a memory leak. The other idioms that should be used instead. Either we can use try/finally religiously in the code, which makes it hard to read and maintain: cdef void *ptr = NULL # intialize to NULL outside try/finally try: ptr = cpython.PyMem_Malloc(n) if ptr == NULL: raise MemoryError finally: if ptr != NULL: cpython.PyMem_Free(ptr) RAII-like coding with a Python extension object is better (IMO): cdef class buffer: cdef void *ptr def __cinit__(buffer self, Py_ssize_t n): self.ptr = NULL def __init__(buffer self, Py_ssize_t n): self.ptr = cpython.PyMem_Malloc(n) if (self.ptr == NULL): raise MemoryError def __dealloc__(buffer self): if self.ptr != NULL: cpython.PyMem_Free(self.ptr) def __enter__(buffer self): return cpython.PyLong_FromVoidPtr(self.ptr) def __exit__(buffer self, type, value, traceback): # Tear-down is strictly speaking not required in # Cython, inlike in Python, since __dealloc__ # behaves differently from __del__. if self.ptr != NULL: cpython.PyMem_Free(self.ptr) self.ptr = NULL # pass on all exceptions return (type is None) Now we can safely do this: cdef buffer buf = buffer(n) cdef void *ptr = buf.ptr # ptr is invalid when reference to # buf is lost ... about here ... or preferably: cdef void *ptr cdef object addr with buffer(n) as addr: ptr = cpython.PyLong_AsVoidPtr(addr) # ptr is invalid here If multiple buffers are needed, then we can e.g. use contextlib.nested to avoid nested with blocks: cdef void *ptr1 cdef void *ptr2 cdef object addr1 cdef object addr2 with nested(buffer(n1),buffer(n2)) as (addr1,addr2): ptr1 = cpython.PyLong_AsVoidPtr(addr1) ptr2 = cpython.PyLong_AsVoidPtr(addr2) # ptr1,ptr2 are invalid here I believe the issue of exceptions and malloc/free is not sufficiently know among scientific users of Cython, so we might put this on the Cookbook wiki as well. Sturla From sturla at molden.no Fri Jul 13 11:08:46 2012 From: sturla at molden.no (Sturla Molden) Date: Fri, 13 Jul 2012 17:08:46 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> Message-ID: <500039FE.6020002@molden.no> I have changed the code to use PyArray_DATA and squashed countless 32-bit vs. 64-bit integer bugs. I think it should work correctly on 64-bit now. All C integers are changed to npy_intp, and corresponding dtype (np.int32 or np.int64) is inferred on module import. Patrick: When you code Cython, observe that a Python int and np.int is a C long on Python 2.x, which is 32-bit on Windows 64. It is very important to make sure all integeres have the correct size. Run-time inference is actually required. This is because in Cython we are often using NumPy from the Python interface, and not using the NumPy C API directly. dtype=int and dtype=np.int are almost never correct in Cython. All cases for memory leaks due to exception handling are still there, getting oversight is terrible... (See Attachment, I don't know how to use git. Perhaps Patrick could put this on Github?) Sturla On 13.07.2012 01:34, Patrick Varilly wrote: > Alright, I've uploaded the last bit of cKDTree that was missing for it > to be functionally equivalent to KDTree. As it stands, I think it's a > useful addition in its own right, so it would be nice if someone else > could look the code over and see if it can be merged in. > > Over the coming weeks, I will look into the issues that Sturla has > brought up and see if I can make some progress on these. > > All the best, > > Patrick > > On Thu, Jul 12, 2012 at 5:42 PM, Sturla Molden > wrote: > > On 12.07.2012 00:26, Patrick Varilly wrote: > > On Tue, Jul 10, 2012 at 12:01 PM, Sturla Molden > > >> wrote: > > > > At least cKDTree have to be fixed, it will break as soon as > the move to > > PyArray_DATA is mandatory. > > > > Preferably we should use Cython memoryviews and > multidimensional arrays > > in the code, instead of just C pointer artithmetics (which is > harder to > > understand). That will make the Cython code more readable to > NumPy > > users. > > > > The GIL issue should also be fixed, as searching might take a > while. > > > > I'm relatively new to Cython. Could you tell me where I could > read up > > on these issues? > > The main issue is the use of the .data attribute. See here: > > http://wiki.cython.org/tutorials/NumpyPointerToC > > Another is that Cython's ndarray interface is (more or less) deprecated > in favour of typed memoryviews: > > http://docs.cython.org/src/userguide/memoryviews.html > > So preferably the cKDTree code should use these, but I my experience > they can generate compile-time warnings. > > There is also a 64-bit issue with cKDTree if I remember correctly. And > the only dtype it supports is float64. We should replace the current > pointer artimetics with multidimensional arrays. It had (or still has) > non-portable code like dependency on unions and binary layout (tree and > heap nodes). And there the issue of making it release the GIL whenever > it should. So several things needs be fixed. > > Sturla > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: cKDTree.pyx Type: / Size: 95945 bytes Desc: not available URL: From sturla at molden.no Fri Jul 13 12:23:20 2012 From: sturla at molden.no (Sturla Molden) Date: Fri, 13 Jul 2012 18:23:20 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> Message-ID: <50004B78.7000809@molden.no> On 13.07.2012 09:09, Ralf Gommers wrote: > Please note that memoryviews can't be used yet in scipy due them not > working with python 2.4 (see https://github.com/numpy/numpy/pull/307). > All the other things Sturla mentioned do sound like improvements that > can be made. Then I suggest using PyArray_DATA instead of array.data. (See the code I posted.) Sturla From sturla at molden.no Fri Jul 13 12:39:23 2012 From: sturla at molden.no (Sturla Molden) Date: Fri, 13 Jul 2012 18:39:23 +0200 Subject: [SciPy-Dev] SSE optimization In-Reply-To: References: Message-ID: <50004F3B.6060301@molden.no> On 09.07.2012 20:51, Fode wrote: > I would like to add sse optimization to numpy if that is still something > the project is interested in. Do you have any suggestions as to where to > start optimizing. Before you waste your time, observe that NumPy is memory bound (due to creation of temporary arrays). SSE matters for CPU bound code. Sturla From sturla at molden.no Fri Jul 13 13:15:32 2012 From: sturla at molden.no (Sturla Molden) Date: Fri, 13 Jul 2012 19:15:32 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> Message-ID: <500057B4.4020505@molden.no> On 13.07.2012 01:34, Patrick Varilly wrote: > Alright, I've uploaded the last bit of cKDTree that was missing for it > to be functionally equivalent to KDTree. As it stands, I think it's a > useful addition in its own right, so it would be nice if someone else > could look the code over and see if it can be merged in. > > Over the coming weeks, I will look into the issues that Sturla has > brought up and see if I can make some progress on these. Your changes + mine: https://github.com/sturlamolden/scipy/blob/master/scipy/spatial/ckdtree.pyx Sturla From sturla at molden.no Sat Jul 14 01:54:30 2012 From: sturla at molden.no (Sturla Molden) Date: Sat, 14 Jul 2012 07:54:30 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: <500057B4.4020505@molden.no> References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> <500057B4.4020505@molden.no> Message-ID: <50010996.3030802@molden.no> I am more or less finished with cleaning up Patrick's cKDTree code. Here are the main changes I have made so far to Patrick's cKDTree rewrite/additions: - Integer sizes are correct for 32 and 64 bit (it was totally confused) and npy_intp are used consistently. - np.float64_t instead of double - All variables are properly declared (many were left undeclared). - All for loops with a range are Cython loops (none of Patrick's loops were correct...). - Python exceptions propagate correctly through cdef functions. - Memory leaks are prevented with try/finally. - MemoryError is raised on malloc or realloc failure (Cython handles "raise MemoryError" correctly). - PyArray_DATA is used for future compatibility with NumPy. - Made sure the common case of Euclidian distance is processed faster. I believe it should be more or less OK, but it needs testing before it's worthy of a pull request to SciPy master. The code is here: https://github.com/sturlamolden/scipy/blob/master/scipy/spatial/ckdtree.pyx Sturla Den 13.07.2012 19:15, skrev Sturla Molden: > On 13.07.2012 01:34, Patrick Varilly wrote: >> Alright, I've uploaded the last bit of cKDTree that was missing for it >> to be functionally equivalent to KDTree. As it stands, I think it's a >> useful addition in its own right, so it would be nice if someone else >> could look the code over and see if it can be merged in. >> >> Over the coming weeks, I will look into the issues that Sturla has >> brought up and see if I can make some progress on these. > Your changes + mine: > > https://github.com/sturlamolden/scipy/blob/master/scipy/spatial/ckdtree.pyx > > > Sturla > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From ralf.gommers at googlemail.com Sat Jul 14 07:17:52 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 14 Jul 2012 13:17:52 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: <50010996.3030802@molden.no> References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> <500057B4.4020505@molden.no> <50010996.3030802@molden.no> Message-ID: On Sat, Jul 14, 2012 at 7:54 AM, Sturla Molden wrote: > > I am more or less finished with cleaning up Patrick's cKDTree code. > > Here are the main changes I have made so far to Patrick's cKDTree > rewrite/additions: > > - Integer sizes are correct for 32 and 64 bit (it was totally confused) > and npy_intp are used consistently. > - np.float64_t instead of double > - All variables are properly declared (many were left undeclared). > - All for loops with a range are Cython loops (none of Patrick's loops > were correct...). > - Python exceptions propagate correctly through cdef functions. > - Memory leaks are prevented with try/finally. > - MemoryError is raised on malloc or realloc failure (Cython handles > "raise MemoryError" correctly). > - PyArray_DATA is used for future compatibility with NumPy. > - Made sure the common case of Euclidian distance is processed faster. > > I believe it should be more or less OK, but it needs testing before it's > worthy of a pull request to SciPy master. > > The code is here: > > https://github.com/sturlamolden/scipy/blob/master/scipy/spatial/ckdtree.pyx > I took your edits and committed them as a diff to Patricks last commit. Then I added a few style fixes: https://github.com/rgommers/scipy/tree/fixes-sturla. The best way to get this in would be for Patrick to add these commits to his branch so they show up in his PR (after finding a bug, see below). I did some testing (python 2.6, 32-bit, OS X 10.6), I get one test failure with Sturla's edits. Patrick's branch given me no failures. ====================================================================== FAIL: test_kdtree.test_count_neighbors_compiled.test_multiple_radius ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/nose-1.1.2-py2.6.egg/nose/case.py", line 197, in runTest self.test(*self.arg) File "/Users/rgommers/Code/scipy/scipy/spatial/tests/test_kdtree.py", line 496, in test_multiple_radius assert_equal(self.T1.count_neighbors(self.T2, r), result) File "/Users/rgommers/Code/numpy/numpy/testing/utils.py", line 313, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: 6 DESIRED: 54 The actual and desired numbers change due to random numbers without a seed being used, but it fails consistently. The missing seed is no real problem here, because the code simply does the same twice; it only tests if the "r" input to count_neighbors is correctly vectorized. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sat Jul 14 10:17:58 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 14 Jul 2012 16:17:58 +0200 Subject: [SciPy-Dev] Generator tests In-Reply-To: References: <4FFC577A.6070007@astro.washington.edu> Message-ID: On Fri, Jul 13, 2012 at 11:57 AM, Pauli Virtanen wrote: > Ralf Gommers googlemail.com> writes: > [clip] > > in the test or code. Plus, it once again shows how > > annoying generator tests are; can't see which method is > > failing. > > Here's a recommendation for generator test: > > - No computation related to the features tested should be > done in the test_* generator function, but delegated to > a corresponding _check_* function (can be inside the > generator, to share namespace). > > - The generators should be used *solely* for loops over parameters. > > - These parameters should *not* be arrays. > > If a parameter is "complicated", as in the case of ARPACK > tests, wrap it into a class with a descriptive __repr__ method. > > Like so: > > def test_foo(): > x = [1, 2, 3] > z = 9 > > def check(a, b): > x = a + b + z > y = a - b + z > assert x == y + 2*b > > for a in [1, 2, 3]: > for b in [4, 5, 6]: > yield check, a, b That's good advice. The ARPACK tests are a lot easier to deal with due to doing it like this. We should also add the above to https://github.com/numpy/numpy/blob/master/doc/TESTS.rst.txt Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Sat Jul 14 13:36:45 2012 From: sturla at molden.no (Sturla Molden) Date: Sat, 14 Jul 2012 19:36:45 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> <500057B4.4020505@molden.no> <50010996.3030802@molden.no> Message-ID: <5001AE2D.9060806@molden.no> I have traced down the bug, it was related to an array being altered inplace instead of copied. I have also taken out two other bugs related to C long (and Python int) being 32 bit on Windows 64. I still get four fails on Win 64. They are there in Patrick's code as well, so they are related to his coding. An update is commited here: https://github.com/sturlamolden/scipy/blob/master/scipy/spatial/ckdtree.pyx Sturla D:\kd-tree\20120714-cKDTree>python test.py ..............................................................................................................................FFFF... ====================================================================== FAIL: test.test_onetree_query_compiled(, 0.1) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "D:\kd-tree\20120714-cKDTree\test.py", line 570, in check_onetree_query assert_(s == T.query_pairs(d)) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 34, in assert_ raise AssertionError(msg) AssertionError ====================================================================== FAIL: test.test_onetree_query_compiled(, 0.001) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "D:\kd-tree\20120714-cKDTree\test.py", line 570, in check_onetree_query assert_(s == T.query_pairs(d)) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 34, in assert_ raise AssertionError(msg) AssertionError ====================================================================== FAIL: test.test_onetree_query_compiled(, 1e-05) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "D:\kd-tree\20120714-cKDTree\test.py", line 570, in check_onetree_query assert_(s == T.query_pairs(d)) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 34, in assert_ raise AssertionError(msg) AssertionError ====================================================================== FAIL: test.test_onetree_query_compiled(, 1e-06) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "D:\kd-tree\20120714-cKDTree\test.py", line 570, in check_onetree_query assert_(s == T.query_pairs(d)) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 34, in assert_ raise AssertionError(msg) AssertionError ---------------------------------------------------------------------- Ran 133 tests in 4.351s FAILED (failures=4) Den 14.07.2012 13:17, skrev Ralf Gommers: > > I took your edits and committed them as a diff to Patricks last > commit. Then I added a few style fixes: > https://github.com/rgommers/scipy/tree/fixes-sturla. The best way to > get this in would be for Patrick to add these commits to his branch so > they show up in his PR (after finding a bug, see below). > > I did some testing (python 2.6, 32-bit, OS X 10.6), I get one test > failure with Sturla's edits. Patrick's branch given me no failures. > > ====================================================================== > FAIL: test_kdtree.test_count_neighbors_compiled.test_multiple_radius > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/nose-1.1.2-py2.6.egg/nose/case.py", > line 197, in runTest > self.test(*self.arg) > File > "/Users/rgommers/Code/scipy/scipy/spatial/tests/test_kdtree.py", line > 496, in test_multiple_radius > assert_equal(self.T1.count_neighbors(self.T2, r), result) > File "/Users/rgommers/Code/numpy/numpy/testing/utils.py", line 313, > in assert_equal > raise AssertionError(msg) > AssertionError: > Items are not equal: > ACTUAL: 6 > DESIRED: 54 > > The actual and desired numbers change due to random numbers without a > seed being used, but it fails consistently. The missing seed is no > real problem here, because the code simply does the same twice; it > only tests if the "r" input to count_neighbors is correctly vectorized. > > Ralf > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Sat Jul 14 14:03:01 2012 From: sturla at molden.no (Sturla Molden) Date: Sat, 14 Jul 2012 20:03:01 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: <5001AE2D.9060806@molden.no> References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> <500057B4.4020505@molden.no> <50010996.3030802@molden.no> <5001AE2D.9060806@molden.no> Message-ID: <5001B455.7080406@molden.no> Raph, do you mind testing again on your Mac? (I am suspecting another integer problem related to Windows 64...) Sturla Den 14.07.2012 19:36, skrev Sturla Molden: > I have traced down the bug, it was related to an array being altered > inplace instead of copied. > > I have also taken out two other bugs related to C long (and Python > int) being 32 bit on Windows 64. > > I still get four fails on Win 64. They are there in Patrick's code as > well, so they are related to his coding. > > An update is commited here: > > https://github.com/sturlamolden/scipy/blob/master/scipy/spatial/ckdtree.pyx > > > Sturla > > > > D:\kd-tree\20120714-cKDTree>python test.py > ..............................................................................................................................FFFF... > ====================================================================== > FAIL: test.test_onetree_query_compiled( 0x000000000583AE18>, 0.1) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest > self.test(*self.arg) > File "D:\kd-tree\20120714-cKDTree\test.py", line 570, in > check_onetree_query > assert_(s == T.query_pairs(d)) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line > 34, in assert_ > raise AssertionError(msg) > AssertionError > > ====================================================================== > FAIL: test.test_onetree_query_compiled( 0x000000000583AE18>, 0.001) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest > self.test(*self.arg) > File "D:\kd-tree\20120714-cKDTree\test.py", line 570, in > check_onetree_query > assert_(s == T.query_pairs(d)) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line > 34, in assert_ > raise AssertionError(msg) > AssertionError > > ====================================================================== > FAIL: test.test_onetree_query_compiled( 0x000000000583AE18>, 1e-05) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest > self.test(*self.arg) > File "D:\kd-tree\20120714-cKDTree\test.py", line 570, in > check_onetree_query > assert_(s == T.query_pairs(d)) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line > 34, in assert_ > raise AssertionError(msg) > AssertionError > > ====================================================================== > FAIL: test.test_onetree_query_compiled( 0x000000000583AE18>, 1e-06) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest > self.test(*self.arg) > File "D:\kd-tree\20120714-cKDTree\test.py", line 570, in > check_onetree_query > assert_(s == T.query_pairs(d)) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line > 34, in assert_ > raise AssertionError(msg) > AssertionError > > ---------------------------------------------------------------------- > Ran 133 tests in 4.351s > > FAILED (failures=4) > > > > Den 14.07.2012 13:17, skrev Ralf Gommers: >> >> I took your edits and committed them as a diff to Patricks last >> commit. Then I added a few style fixes: >> https://github.com/rgommers/scipy/tree/fixes-sturla. The best way to >> get this in would be for Patrick to add these commits to his branch >> so they show up in his PR (after finding a bug, see below). >> >> I did some testing (python 2.6, 32-bit, OS X 10.6), I get one test >> failure with Sturla's edits. Patrick's branch given me no failures. >> >> ====================================================================== >> FAIL: test_kdtree.test_count_neighbors_compiled.test_multiple_radius >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File >> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/nose-1.1.2-py2.6.egg/nose/case.py", >> line 197, in runTest >> self.test(*self.arg) >> File >> "/Users/rgommers/Code/scipy/scipy/spatial/tests/test_kdtree.py", line >> 496, in test_multiple_radius >> assert_equal(self.T1.count_neighbors(self.T2, r), result) >> File "/Users/rgommers/Code/numpy/numpy/testing/utils.py", line 313, >> in assert_equal >> raise AssertionError(msg) >> AssertionError: >> Items are not equal: >> ACTUAL: 6 >> DESIRED: 54 >> >> The actual and desired numbers change due to random numbers without a >> seed being used, but it fails consistently. The missing seed is no >> real problem here, because the code simply does the same twice; it >> only tests if the "r" input to count_neighbors is correctly vectorized. >> >> Ralf >> >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sat Jul 14 17:03:54 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 14 Jul 2012 23:03:54 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: <5001B455.7080406@molden.no> References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> <500057B4.4020505@molden.no> <50010996.3030802@molden.no> <5001AE2D.9060806@molden.no> <5001B455.7080406@molden.no> Message-ID: On Sat, Jul 14, 2012 at 8:03 PM, Sturla Molden wrote: > Raph, do you mind testing again on your Mac? > On my system all tests pass now. > (I am suspecting another integer problem related to Windows 64...) > > Sturla > > > > > Den 14.07.2012 19:36, skrev Sturla Molden: > > I have traced down the bug, it was related to an array being altered > inplace instead of copied. > > I have also taken out two other bugs related to C long (and Python int) > being 32 bit on Windows 64. > > I still get four fails on Win 64. They are there in Patrick's code as > well, so they are related to his coding. > > An update is commited here: > > https://github.com/sturlamolden/scipy/blob/master/scipy/spatial/ckdtree.pyx > > Taken over in https://github.com/rgommers/scipy/tree/fixes-sturla I know you may not have the time or interest to learn about git right now, but it may make both our lives easier if you try the below steps. It will allow you to put your commits on top of mine without any manual copying. $ git remote add rgommers https://github.com/rgommers/scipy $ git fetch rgommers $ git checkout -b fixes-sturla rgommers/fixes-sturla Now your local copy of the code is up to date and in sync with mine; you can now make commits in this branch, preferably with a commit message that is somewhat more descriptive than "update master". Then to push them to github: $ git push origin fixes-sturla Cheers, Ralf > Sturla > > > > D:\kd-tree\20120714-cKDTree>python test.py > > ..............................................................................................................................FFFF... > ====================================================================== > FAIL: test.test_onetree_query_compiled( 0x000000000583AE18>, 0.1) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest > self.test(*self.arg) > File "D:\kd-tree\20120714-cKDTree\test.py", line 570, in > check_onetree_query > assert_(s == T.query_pairs(d)) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 34, in > assert_ > raise AssertionError(msg) > AssertionError > > ====================================================================== > FAIL: test.test_onetree_query_compiled( 0x000000000583AE18>, 0.001) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest > self.test(*self.arg) > File "D:\kd-tree\20120714-cKDTree\test.py", line 570, in > check_onetree_query > assert_(s == T.query_pairs(d)) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 34, in > assert_ > raise AssertionError(msg) > AssertionError > > ====================================================================== > FAIL: test.test_onetree_query_compiled( 0x000000000583AE18>, 1e-05) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest > self.test(*self.arg) > File "D:\kd-tree\20120714-cKDTree\test.py", line 570, in > check_onetree_query > assert_(s == T.query_pairs(d)) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 34, in > assert_ > raise AssertionError(msg) > AssertionError > > ====================================================================== > FAIL: test.test_onetree_query_compiled( 0x000000000583AE18>, 1e-06) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest > self.test(*self.arg) > File "D:\kd-tree\20120714-cKDTree\test.py", line 570, in > check_onetree_query > assert_(s == T.query_pairs(d)) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 34, in > assert_ > raise AssertionError(msg) > AssertionError > > ---------------------------------------------------------------------- > Ran 133 tests in 4.351s > > FAILED (failures=4) > > > > Den 14.07.2012 13:17, skrev Ralf Gommers: > > > I took your edits and committed them as a diff to Patricks last commit. > Then I added a few style fixes: > https://github.com/rgommers/scipy/tree/fixes-sturla. The best way to get > this in would be for Patrick to add these commits to his branch so they > show up in his PR (after finding a bug, see below). > > I did some testing (python 2.6, 32-bit, OS X 10.6), I get one test failure > with Sturla's edits. Patrick's branch given me no failures. > > ====================================================================== > FAIL: test_kdtree.test_count_neighbors_compiled.test_multiple_radius > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/nose-1.1.2-py2.6.egg/nose/case.py", > line 197, in runTest > self.test(*self.arg) > File "/Users/rgommers/Code/scipy/scipy/spatial/tests/test_kdtree.py", > line 496, in test_multiple_radius > assert_equal(self.T1.count_neighbors(self.T2, r), result) > File "/Users/rgommers/Code/numpy/numpy/testing/utils.py", line 313, in > assert_equal > raise AssertionError(msg) > AssertionError: > Items are not equal: > ACTUAL: 6 > DESIRED: 54 > > The actual and desired numbers change due to random numbers without a seed > being used, but it fails consistently. The missing seed is no real problem > here, because the code simply does the same twice; it only tests if the "r" > input to count_neighbors is correctly vectorized. > > Ralf > > > > _______________________________________________ > SciPy-Dev mailing listSciPy-Dev at scipy.orghttp://mail.scipy.org/mailman/listinfo/scipy-dev > > > > > _______________________________________________ > SciPy-Dev mailing listSciPy-Dev at scipy.orghttp://mail.scipy.org/mailman/listinfo/scipy-dev > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Sat Jul 14 18:36:33 2012 From: sturla at molden.no (Sturla Molden) Date: Sun, 15 Jul 2012 00:36:33 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> <500057B4.4020505@molden.no> <50010996.3030802@molden.no> <5001AE2D.9060806@molden.no> <5001B455.7080406@molden.no> Message-ID: <5001F471.1010404@molden.no> Den 14.07.2012 23:03, skrev Ralf Gommers: > > On Sat, Jul 14, 2012 at 8:03 PM, Sturla Molden > wrote: > > Raph, do you mind testing again on your Mac? > > > On my system all tests pass now. Thanks. On Windows 64, quary_pair returns bad results depending on the size of the tree. I don't know why but Patrick's code hade the same problem. > Taken over in https://github.com/rgommers/scipy/tree/fixes-sturla > > I know you may not have the time or interest to learn about git right > now, but it may make both our lives easier if you try the below steps. > It will allow you to put your commits on top of mine without any > manual copying. > > $ git remote add rgommers https://github.com/rgommers/scipy > $ git fetch rgommers > $ git checkout -b fixes-sturla rgommers/fixes-sturla > > Now your local copy of the code is up to date and in sync with mine; > you can now make commits in this branch, preferably with a commit > message that is somewhat more descriptive than "update master". > > Then to push them to github: > $ git push origin fixes-sturla I'm an idiot with tools like git, svn and hg, which unfortunately prevents me from contributing as much as I could. I should perhaps try to learn it :) Sturla -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Sat Jul 14 19:29:14 2012 From: sturla at molden.no (Sturla Molden) Date: Sun, 15 Jul 2012 01:29:14 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: <5001F471.1010404@molden.no> References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> <500057B4.4020505@molden.no> <50010996.3030802@molden.no> <5001AE2D.9060806@molden.no> <5001B455.7080406@molden.no> <5001F471.1010404@molden.no> Message-ID: <500200CA.1030109@molden.no> Den 15.07.2012 00:36, skrev Sturla Molden: > > Thanks. On Windows 64, query_pair returns bad results depending on the > size of the tree. I don't know why but Patrick's code hade the same > problem. > Which brings me to the other issue I'd like to discuss... In its current state, Patrick's enhancement to cKDTree (IMHO) is close to unmaintainable. It might be 'fast', but it is not estetically nice looking code (no pun intended), about 2500 lines of low-level C masquerading as Cython, very difficult to read and follow with respect to logic, it was full of bugs (and might still be), and still don't pass all the tests on my computer (even though I have been debugging it quite extensively). Who is going to maintain it? Patrick? Ralph? Anne? (I am not volunteering...) On the other hand, Anne's KDTree.py is an example of very beautiful Python code, and very easy to read and understand. KDTree is also robust and rather bug-free, after having been tested for quite a long time, and more or less feature-complete for a KDTree. But it is a bit slow, courtesy to Python... How fast do we need it to be? I think it might be better to just gently 'Cythonize' out the worst bottlenecks in KDTree.py, but leave the rest as it is. It might not be equally fast as Patrick's rewrite of cKDTree enhancement, but will be 100 times easier to maintain, and probably "fast enough" for most usecases. cKDTree was intended as a very fast kd-tree for nearest-neighbor queries. It was never intended as general-purpose kd-tree like, well, KDTree. Currently in SciPy master it is very "lean and mean" for its particular purpose. (It could still use some minor updating, such as 64-bit support.) I think at least this should be discussed before this major rewrite is pulled into SciPy master. I am not saying Patrick's rewrite is bad anbs should not be pulled into SciPy. But I don't think we should drop code into SciPy if it will never be properly maintained. It must be maintainable for years ahead. Sturla -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sun Jul 15 08:26:20 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 15 Jul 2012 14:26:20 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: <500200CA.1030109@molden.no> References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> <500057B4.4020505@molden.no> <50010996.3030802@molden.no> <5001AE2D.9060806@molden.no> <5001B455.7080406@molden.no> <5001F471.1010404@molden.no> <500200CA.1030109@molden.no> Message-ID: On Sun, Jul 15, 2012 at 1:29 AM, Sturla Molden wrote: > Den 15.07.2012 00:36, skrev Sturla Molden: > > > Thanks. On Windows 64, query_pair returns bad results depending on the > size of the tree. I don't know why but Patrick's code hade the same problem. > > > Which brings me to the other issue I'd like to discuss... > > In its current state, Patrick's enhancement to cKDTree (IMHO) is close to > unmaintainable. It might be 'fast', but it is not estetically nice looking > code (no pun intended), about 2500 lines of low-level C masquerading as > Cython, very difficult to read and follow with respect to logic, it was > full of bugs (and might still be), and still don't pass all the tests on my > computer (even though I have been debugging it quite extensively). > > Who is going to maintain it? Patrick? Ralph? Anne? (I am not > volunteering...) > Good question. For me, ``cython_skills * available_time`` is too small to volunteer for that. > > On the other hand, Anne's KDTree.py is an example of very beautiful Python > code, and very easy to read and understand. KDTree is also robust and > rather bug-free, after having been tested for quite a long time, and more > or less feature-complete for a KDTree. But it is a bit slow, courtesy to > Python... > > How fast do we need it to be? > > I think it might be better to just gently 'Cythonize' out the worst > bottlenecks in KDTree.py, but leave the rest as it is. It might not be > equally fast as Patrick's rewrite of cKDTree enhancement, but will be 100 > times easier to maintain, and probably "fast enough" for most usecases. > > cKDTree was intended as a very fast kd-tree for nearest-neighbor queries. > It was never intended as general-purpose kd-tree like, well, KDTree. > Currently in SciPy master it is very "lean and mean" for its particular > purpose. (It could still use some minor updating, such as 64-bit support.) > No matter what we do with cKDTree, KDTree should not be removed. Readability alone is enough reason to keep it. Ralf > > I think at least this should be discussed before this major rewrite is > pulled into SciPy master. I am not saying Patrick's rewrite is bad anbs > should not be pulled into SciPy. But I don't think we should drop code into > SciPy if it will never be properly maintained. It must be maintainable for > years ahead. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sun Jul 15 10:57:02 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 15 Jul 2012 16:57:02 +0200 Subject: [SciPy-Dev] update on 0.11.0 status In-Reply-To: References: <4FFC577A.6070007@astro.washington.edu> <4FFC961C.8060605@astro.washington.edu> <4FFC9FB5.6090006@astro.washington.edu> Message-ID: On Wed, Jul 11, 2012 at 11:45 PM, Ralf Gommers wrote: > > > On Tue, Jul 10, 2012 at 11:33 PM, Jake Vanderplas < > vanderplas at astro.washington.edu> wrote: > >> Ralf, >> I tried running valgrind on the offending code. I don't see anything out >> of the ordinary come up on my system (for reference, the scikit-learn >> developer documentation has some tips on using valgrind for cython code >> [1]). This makes me suspect that the error you're seeing may be a >> system-specific issue (i.e. 32 bit/64 bit mismatch), or perhaps something >> related to the numpy version (I'm using the current numpy 1.8.0-dev). >> >> Could you run valgrind on your system with test_shortest_path.py and see >> if anything obvious comes up? >> > > It's clean, some noise related to Python itself but nothing related to > csgraph. > > This was 32-bit python 2.6, numpy 1.8-dev by the way. I haven't yet > checked if I get the same when running in 64-bit mode on the same machine > yet. > Found the issue (use of an array created with np.empty), see http://projects.scipy.org/scipy/ticket/1701. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanderplas at astro.washington.edu Sun Jul 15 19:49:01 2012 From: vanderplas at astro.washington.edu (Jake Vanderplas) Date: Sun, 15 Jul 2012 16:49:01 -0700 Subject: [SciPy-Dev] update on 0.11.0 status In-Reply-To: References: <4FFC577A.6070007@astro.washington.edu> <4FFC961C.8060605@astro.washington.edu> <4FFC9FB5.6090006@astro.washington.edu> Message-ID: <500356ED.7010401@astro.washington.edu> Good catch, Ralf! Initializing all values of dist_array to zero is the correct thing to do here. I don't know how I missed that... Jake On 07/15/2012 07:57 AM, Ralf Gommers wrote: > > > On Wed, Jul 11, 2012 at 11:45 PM, Ralf Gommers > > wrote: > > > > On Tue, Jul 10, 2012 at 11:33 PM, Jake Vanderplas > > wrote: > > Ralf, > I tried running valgrind on the offending code. I don't see > anything out of the ordinary come up on my system (for > reference, the scikit-learn developer documentation has some > tips on using valgrind for cython code [1]). This makes me > suspect that the error you're seeing may be a system-specific > issue (i.e. 32 bit/64 bit mismatch), or perhaps something > related to the numpy version (I'm using the current numpy > 1.8.0-dev). > > Could you run valgrind on your system with > test_shortest_path.py and see if anything obvious comes up? > > > It's clean, some noise related to Python itself but nothing > related to csgraph. > > This was 32-bit python 2.6, numpy 1.8-dev by the way. I haven't > yet checked if I get the same when running in 64-bit mode on the > same machine yet. > > > Found the issue (use of an array created with np.empty), see > http://projects.scipy.org/scipy/ticket/1701. > > Ralf > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From patvarilly at gmail.com Sun Jul 15 20:00:35 2012 From: patvarilly at gmail.com (Patrick Varilly) Date: Mon, 16 Jul 2012 01:00:35 +0100 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: <500200CA.1030109@molden.no> References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> <500057B4.4020505@molden.no> <50010996.3030802@molden.no> <5001AE2D.9060806@molden.no> <5001B455.7080406@molden.no> <5001F471.1010404@molden.no> <500200CA.1030109@molden.no> Message-ID: Wow, I must confess I wasn't quite expecting the Spanish Inquisition :-D The original reason I got into this project was to bring a fast query_ball_point to cKDTree (which I really needed, the KDTree implementation is so slow as to be unusable for analysing a long molecular dynamics trajectory). When I started, I thought I was only porting over one method of KDTree, so I strove to keep the code changes to a minimum (hence not abstracting out the incremental updates to the minimum/maximum distances between the query point and a node's bounding hyperrectangle) and to keep a very similar style to the existing code (hence the int's, double's, malloc/free, etc.). I only went about porting over the rest of the methods essentially to submit a complete enhancement to SciPy instead of just the juicy bit I cared about (after parsing Anne's code, I figured 90% of the effort was done already). I also agree that Anne's code is really transparent, and should stay no matter what, but speed does matter a fair bit to users of kd-trees. And in my defense with respect to the code being "full of bugs", I tried hard to make all the unit tests pass on my machine before submitting and was completely unaware of the nightmares of 32 vs 64-bit ints in Windows 64. That said, now that the initial effort is complete, it's obvious that a clear pattern recurs in the kd-tree walking algorithms: keeping incremental track of the minimum/maximum distances between two hyperrectangles as they're successively split (alternatively, a hyperrectangle and a point). If that primitive is abstracted, the code becomes a lot more readable again. I've had a first go at what such an abstraction might look like, and updated the code to query_ball_tree. Does this look like a way forward for enhancing the code's maintainability? Here are another couple of issues / questions: * Is there any difference between np.PyArray_DATA(my_array) and &my_array[0] ? If not, the second choice is clearly preferable. * I get the issues with int vs np.int vs np.npy_intp, but is there really any SciPy-supported platform in which double doesn't mean np.float64_t? * As for memory management, it's probably easier to let NumPy & Python handle most things, i.e. make Rectangle store two NumPy arrays. This needs a bit of surgery to do right, but I've tried to show what I mean in the query_ball_tree version I'm submitting now. * I fixed a small glitch with calls to realloc(): Sturla's changes for handling memory errors ended up setting the first argument to realloc to NULL, which effectively wipes out the contents of the data that were there before the resize. * There seems to be something funny about raw_mins and raw_maxes: they are member variables of cKDTree, but the data they point to seems to me like it should go out of scope at the end of cKDTree.__init__. Am I misunderstanding something here? All the best, Patrick On Sun, Jul 15, 2012 at 12:29 AM, Sturla Molden wrote: > Den 15.07.2012 00:36, skrev Sturla Molden: > > > Thanks. On Windows 64, query_pair returns bad results depending on the > size of the tree. I don't know why but Patrick's code hade the same problem. > > > Which brings me to the other issue I'd like to discuss... > > In its current state, Patrick's enhancement to cKDTree (IMHO) is close to > unmaintainable. It might be 'fast', but it is not estetically nice looking > code (no pun intended), about 2500 lines of low-level C masquerading as > Cython, very difficult to read and follow with respect to logic, it was > full of bugs (and might still be), and still don't pass all the tests on my > computer (even though I have been debugging it quite extensively). > > Who is going to maintain it? Patrick? Ralph? Anne? (I am not > volunteering...) > > On the other hand, Anne's KDTree.py is an example of very beautiful Python > code, and very easy to read and understand. KDTree is also robust and > rather bug-free, after having been tested for quite a long time, and more > or less feature-complete for a KDTree. But it is a bit slow, courtesy to > Python... > > How fast do we need it to be? > > I think it might be better to just gently 'Cythonize' out the worst > bottlenecks in KDTree.py, but leave the rest as it is. It might not be > equally fast as Patrick's rewrite of cKDTree enhancement, but will be 100 > times easier to maintain, and probably "fast enough" for most usecases. > > cKDTree was intended as a very fast kd-tree for nearest-neighbor queries. > It was never intended as general-purpose kd-tree like, well, KDTree. > Currently in SciPy master it is very "lean and mean" for its particular > purpose. (It could still use some minor updating, such as 64-bit support.) > > > I think at least this should be discussed before this major rewrite is > pulled into SciPy master. I am not saying Patrick's rewrite is bad anbs > should not be pulled into SciPy. But I don't think we should drop code into > SciPy if it will never be properly maintained. It must be maintainable for > years ahead. > > Sturla > > > > > > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Mon Jul 16 08:34:34 2012 From: sturla at molden.no (Sturla Molden) Date: Mon, 16 Jul 2012 14:34:34 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> <500057B4.4020505@molden.no> <50010996.3030802@molden.no> <5001AE2D.9060806@molden.no> <5001B455.7080406@molden.no> <5001F471.1010404@molden.no> <500200CA.1030109@molden.no> Message-ID: <50040A5A.4060905@molden.no> Den 16.07.2012 02:00, skrev Patrick Varilly: > Wow, I must confess I wasn't quite expecting the Spanish Inquisition :-D Sorry, don't take it personally. It is important that code that goes into SciPy can be maintained for several years ahead, and preferably not just by you. > > The original reason I got into this project was to bring a fast > query_ball_point to cKDTree (which I really needed, the KDTree > implementation is so slow as to be unusable for analysing a long > molecular dynamics trajectory). When I started, I thought I was only > porting over one method of KDTree, so I strove to keep the code > changes to a minimum (hence not abstracting out the incremental > updates to the minimum/maximum distances between the query point and a > node's bounding hyperrectangle) and to keep a very similar style to > the existing code (hence the int's, double's, malloc/free, etc.). I > only went about porting over the rest of the methods essentially to > submit a complete enhancement to SciPy instead of just the juicy bit I > cared about (after parsing Anne's code, I figured 90% of the effort > was done already). I also agree that Anne's code is really > transparent, and should stay no matter what, but speed does matter a > fair bit to users of kd-trees. And in my defense with respect to the > code being "full of bugs", I tried hard to make all the unit tests > pass on my machine before submitting and was completely unaware of the > nightmares of 32 vs 64-bit ints in Windows 64. Yes, Linux and Apple cheated and made a C long 64 bits. Whereas AMD designed AMD64 (aka x64) to use a 64-bit pointer with a 32-bit offset. Windows followed that pattern and uses a 32-bit long. Whereas on IA64, Intel has specified a 64-bit offset, so Windows uses a 64-bit long there. It is actually Windows that does this correctly (pedantially speaking), but the 64-bit long on Apple and Linux obviously makes life easier. On the good side, this makes Windows 64 an excellent pantform for tracing down integer size bugs :) > > That said, now that the initial effort is complete, it's obvious that > a clear pattern recurs in the kd-tree walking algorithms: keeping > incremental track of the minimum/maximum distances between two > hyperrectangles as they're successively split (alternatively, a > hyperrectangle and a point). If that primitive is abstracted, the > code becomes a lot more readable again. I've had a first go at what > such an abstraction might look like, and updated the code to > query_ball_tree. Does this look like a way forward for enhancing the > code's maintainability? I'll look at it, but this is a very important issue. I think this rewrite cannot be used in SciPy unless it is so readable/understandable that several SciPy developers can understand and change the code. At least this is my main objection with it right now. Also observe that Cython has closures now, and you can nest Python functions in Cython (and declaring them cpdef makes the call almost as efficient as a cdef). > > Here are another couple of issues / questions: > > * Is there any difference between > np.PyArray_DATA(my_array) and &my_array[0] ? If not, > the second choice is clearly preferable. > I checked Dag Sverre's Cython code again. np.PyArray_DATA(my_array) and &my_array[0] are equivalent. my_Array.data is wrong. Go ahead and change my code. &my_array[0] will also make it easier to update to memoryviews when Python 2.4 support is dropped. > * I get the issues with int vs np.int vs np.npy_intp, > but is there really any SciPy-supported platform in which double > doesn't mean np.float64_t? Do not use C int, ever. (Except where we know it does not overflow, such as the void functions I changed to propagate errors.) np.int should be a C long (or C int?), for which you don't know the size, and it might not correspond to the indexer we want (npy_intp or Py_ssize_t). That is why you should either use np.int32 or np.int64, and check at runtime which is applicable. Also do not use dtype='i' or anything of that sort. > > * As for memory management, it's probably easier to let NumPy & Python > handle most things, i.e. make Rectangle store two NumPy arrays. This > needs a bit of surgery to do right, but I've tried to show what I mean > in the query_ball_tree version I'm submitting now. > I thought the same thing. You can often use NumPy instead of malloc in your code. Then you could also remove the try/finally blocks I added to your code (but leave those I added to Anne's cKDTree code). > * I fixed a small glitch with calls to realloc(): Sturla's changes for > handling memory errors ended up setting the first argument to realloc > to NULL, which effectively wipes out the contents of the data that > were there before the resize. Thanks. > * There seems to be something funny about raw_mins and raw_maxes: they > are member variables of cKDTree, but the data they point to seems to > me like it should go out of scope at the end of cKDTree.__init__. Am > I misunderstanding something here? Yes you do :) See below: cdef class cKDTree: cdef readonly object maxes def __init__(cKDTree self): cdef np.ndarray[np.float64_t, ndim=1] inner_maxes self.maxes = np.ascontiguousarray(np.amax(self.data,axis=0), dtype=np.float64) inner_maxes = self.maxes self.raw_maxes = np.PyArray_DATA(inner_maxes) "maxes" stores the contiguous array as an attribute of the cKDTree extension object. It will be alive until the object of type cKDTree is reclaimed. "inner_maxes" unboxes the array struct to a collection of local variables for fast indexing access inside cKDTree. It also tells Cython that the type is np.ndarray (which extends NumPy's PyArrayObject). Then finally "raw_maxes" gets the C pointer to the contiguous buffer stored in "maxes", by using the unboxed "inner_maxes". We could also have done this (at least for Python 2.x): self.raw_maxes = cpython.PyLong_AsVoidPtr(self.maxes.__array_interface__['data'][0]) or this self.raw_maxes = &inner_maxes[0] But the current "inner_maxes.data" should not be used. Another issue is this: Where ever you use list.append and set.add to save a tuple of indexes (i,j), do this: if sizeof(long) < sizeof(np.npy_intp): # Win 64 if self.raw_indices[i] < self.raw_indices[j]: results.add((int(self.raw_indices[i]), int(self.raw_indices[j]))) else: results.add((int(self.raw_indices[j]), int(self.raw_indices[i]))) else: # Other platforms if self.raw_indices[i] < self.raw_indices[j]: results.add((self.raw_indices[i], self.raw_indices[j])) else: results.add((self.raw_indices[j], self.raw_indices[i])) sizeof(long) < sizeof(np.npy_intp) is a compile time constant so it will be optimized by the compiler. Then you can take out the stuff I added to fix the set for Win64 before the return of query_pairs. You might want to put this into two inline functions (for list.append and set.add) to avoid repeating yourself. cdef inline int set_add_pair(set results, np.npy_intp i, np.npy_intp j) except -1: if sizeof(long) < sizeof(np.npy_intp): # Win 64 if i < j: results.add((int(j), int(j))) else: results.add((int(j), int(i))) else: # Other platforms if i < j: results.add((i, j)) else: results.add((j, i)) return 0 The difference is that on Win 64 we should use Python long istead of Python int if a C long (i.e. Pyhton int) overflows, which the function int() will ensure. Cython automatically converts np.npy_intp to Python long on Win 64, which we want to convert to a Python int if it is possible. On other platforms we don't want this extra overhead. I think Cython will optimize set.add, otherwise you should put it in a local variable istead of getting a dict lookup on results.add (I know list.append is optimized). The int return value for set_add_pair is used to propagate Python exceptions, which can result from results.add() or int(). Sturla -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Mon Jul 16 08:54:53 2012 From: sturla at molden.no (Sturla Molden) Date: Mon, 16 Jul 2012 14:54:53 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> <500057B4.4020505@molden.no> <50010996.3030802@molden.no> <5001AE2D.9060806@molden.no> <5001B455.7080406@molden.no> <5001F471.1010404@molden.no> <500200CA.1030109@molden.no> Message-ID: <50040F1D.70109@molden.no> Den 16.07.2012 02:00, skrev Patrick Varilly: > > * I get the issues with int vs np.int vs np.npy_intp, > but is there really any SciPy-supported platform in which double > doesn't mean np.float64_t? Sorry, I misread you question. (I don't speak English natively.) np.float64_t is the C typedef corresponding to np.float64, i.e. npy_float64. np.double_t is the C typedef corresponding to np.double, i.e. npy_double. Both are C double by definition, AFAIK. So on the C side we could use them interchangeably on all platforms I know about. So this is more pedantery, unless there are very strange platforms I don't know about. Well, internally on x86 a C double is 80 bits, I don't know if npy_float64 sets the mantissa of the FPU to 53 bits or not. (You might ask on the NumPy list.) But np.float64_t will ensure consistency with NumPy and KDTree. Sturla -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Mon Jul 16 14:08:31 2012 From: sturla at molden.no (Sturla Molden) Date: Mon, 16 Jul 2012 20:08:31 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> <500057B4.4020505@molden.no> <50010996.3030802@molden.no> <5001AE2D.9060806@molden.no> <5001B455.7080406@molden.no> <5001F471.1010404@molden.no> <500200CA.1030109@molden.no> Message-ID: <5004589F.7040908@molden.no> Den 16.07.2012 02:00, skrev Patrick Varilly: > > That said, now that the initial effort is complete, it's obvious that > a clear pattern recurs in the kd-tree walking algorithms: keeping > incremental track of the minimum/maximum distances between two > hyperrectangles as they're successively split (alternatively, a > hyperrectangle and a point). If that primitive is abstracted, the > code becomes a lot more readable again. Yes, that was much better :-) +1 for doping it like this (even if there is a small speed penalty). We might even do this to the original cKDTree.query function as well. Sturla From ralf.gommers at googlemail.com Mon Jul 16 14:34:53 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 16 Jul 2012 20:34:53 +0200 Subject: [SciPy-Dev] update on 0.11.0 status In-Reply-To: <500356ED.7010401@astro.washington.edu> References: <4FFC577A.6070007@astro.washington.edu> <4FFC961C.8060605@astro.washington.edu> <4FFC9FB5.6090006@astro.washington.edu> <500356ED.7010401@astro.washington.edu> Message-ID: On Mon, Jul 16, 2012 at 1:49 AM, Jake Vanderplas < vanderplas at astro.washington.edu> wrote: > Good catch, Ralf! > Initializing all values of dist_array to zero is the correct thing to do > here. I don't know how I missed that... > Great, thanks for confirming. Now that all issues have fixes available, I plan to merge all open PRs by tomorrow and prepare a release candidate asap. Some PRs haven't been reviewed yet but they're very straightforward - please have a look at them or let me know you intend to do so if you want me to wait. https://github.com/scipy/scipy/pull/267 https://github.com/scipy/scipy/pull/268 https://github.com/scipy/scipy/pull/258 https://github.com/scipy/scipy/pull/257 https://github.com/scipy/scipy/pull/266 Ralf > On 07/15/2012 07:57 AM, Ralf Gommers wrote: > > > > On Wed, Jul 11, 2012 at 11:45 PM, Ralf Gommers < > ralf.gommers at googlemail.com> wrote: > >> >> >> On Tue, Jul 10, 2012 at 11:33 PM, Jake Vanderplas < >> vanderplas at astro.washington.edu> wrote: >> >>> Ralf, >>> I tried running valgrind on the offending code. I don't see anything >>> out of the ordinary come up on my system (for reference, the scikit-learn >>> developer documentation has some tips on using valgrind for cython code >>> [1]). This makes me suspect that the error you're seeing may be a >>> system-specific issue (i.e. 32 bit/64 bit mismatch), or perhaps something >>> related to the numpy version (I'm using the current numpy 1.8.0-dev). >>> >>> Could you run valgrind on your system with test_shortest_path.py and see >>> if anything obvious comes up? >>> >> >> It's clean, some noise related to Python itself but nothing related to >> csgraph. >> >> This was 32-bit python 2.6, numpy 1.8-dev by the way. I haven't yet >> checked if I get the same when running in 64-bit mode on the same machine >> yet. >> > > Found the issue (use of an array created with np.empty), see > http://projects.scipy.org/scipy/ticket/1701. > > Ralf > > > > _______________________________________________ > SciPy-Dev mailing listSciPy-Dev at scipy.orghttp://mail.scipy.org/mailman/listinfo/scipy-dev > > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From patvarilly at gmail.com Mon Jul 16 14:50:39 2012 From: patvarilly at gmail.com (Patrick Varilly) Date: Mon, 16 Jul 2012 19:50:39 +0100 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: <50040A5A.4060905@molden.no> References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> <500057B4.4020505@molden.no> <50010996.3030802@molden.no> <5001AE2D.9060806@molden.no> <5001B455.7080406@molden.no> <5001F471.1010404@molden.no> <500200CA.1030109@molden.no> <50040A5A.4060905@molden.no> Message-ID: > Also observe that Cython has closures now, and you can nest Python > functions in Cython (and declaring them cpdef makes the call almost as > efficient as a cdef). > By this, do you mean putting back the workhorse "traverse" methods inside their respective function (e.g., query_ball_tree), like KDTree does? I would certainly like to do that (it would save passing in tons of arguments in the recursive calls), but I tried the following test and got the Cython error "C function definition not allowed here" (here being "print_i()"): cdef class Test(object): cpdef test(): cdef int i = 2 cpdef print_i(): print i print_i() * There seems to be something funny about raw_mins and raw_maxes: they are member variables of cKDTree, but the data they point to seems to me like it should go out of scope at the end of cKDTree.__init__. Am I misunderstanding something here? > Yes you do :) See below: > > > cdef class cKDTree: > > cdef readonly object maxes > > def __init__(cKDTree self): > > cdef np.ndarray[np.float64_t, ndim=1] inner_maxes > > self.maxes = np.ascontiguousarray(np.amax(self.data,axis=0), > dtype=np.float64) > > inner_maxes = self.maxes > > self.raw_maxes = np.PyArray_DATA(inner_maxes) > > > "maxes" stores the contiguous array as an attribute of the cKDTree > extension object. It will be alive until the object of type cKDTree is > reclaimed. > > "inner_maxes" unboxes the array struct to a collection of local variables > for fast indexing access inside cKDTree. It also tells Cython that the > type is np.ndarray (which extends NumPy's PyArrayObject). > > Then finally "raw_maxes" gets the C pointer to the contiguous buffer > stored in "maxes", by using the unboxed "inner_maxes". > > > We could also have done this (at least for Python 2.x): > > self.raw_maxes = cpython.PyLong_AsVoidPtr( > self.maxes.__array_interface__['data'][0]) > > or this > > self.raw_maxes = &inner_maxes[0] > > But the current "inner_maxes.data" should not be used. > Thanks, this makes more sense. I thought there might be a circumstance when "inner_maxes = maxes" might create a copy of maxes, but I now see that it will only check that ndim is 1 and dtype is np.float64 before making the data available. > > > Another issue is this: > > Where ever you use list.append and set.add to save a tuple of indexes > (i,j), do this: > > > if sizeof(long) < sizeof(np.npy_intp): > # Win 64 > if self.raw_indices[i] < self.raw_indices[j]: > results.add((int(self.raw_indices[i]), int(self.raw_indices[j]))) > else: > results.add((int(self.raw_indices[j]), int(self.raw_indices[i]))) > > else: > # Other platforms > if self.raw_indices[i] < self.raw_indices[j]: > results.add((self.raw_indices[i], self.raw_indices[j])) > else: > results.add((self.raw_indices[j], self.raw_indices[i])) > > > sizeof(long) < sizeof(np.npy_intp) is a compile time constant so it will > be optimized by the compiler. Then you can take out the stuff I added to > fix the set for Win64 before the return of query_pairs. > > You might want to put this into two inline functions (for list.append and > set.add) to avoid repeating yourself. > > > cdef inline int set_add_pair(set results, np.npy_intp i, np.npy_intp j) > except -1: > if sizeof(long) < sizeof(np.npy_intp): > # Win 64 > if i < j: > results.add((int(j), int(j))) > else: > results.add((int(j), int(i))) > else: > # Other platforms > if i < j: > results.add((i, j)) > else: > results.add((j, i)) > return 0 > > > The difference is that on Win 64 we should use Python long istead of > Python int if a C long (i.e. Pyhton int) overflows, which the function > int() will ensure. Cython automatically converts np.npy_intp to Python > long on Win 64, which we want to convert to a Python int if it is possible.On other platforms we don't want this extra overhead. > Okay, sounds good, I'll do this. Best, Patrick -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Mon Jul 16 15:20:09 2012 From: sturla at molden.no (Sturla Molden) Date: Mon, 16 Jul 2012 21:20:09 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> <500057B4.4020505@molden.no> <50010996.3030802@molden.no> <5001AE2D.9060806@molden.no> <5001B455.7080406@molden.no> <5001F471.1010404@molden.no> <500200CA.1030109@molden.no> <50040A5A.4060905@molden.no> Message-ID: <50046969.9000008@molden.no> Den 16.07.2012 20:50, skrev Patrick Varilly: > > Also observe that Cython has closures now, and you can nest Python > functions in Cython (and declaring them cpdef makes the call > almost as efficient as a cdef). > > > By this, do you mean putting back the workhorse "traverse" methods > inside their respective function (e.g., query_ball_tree), like KDTree > does? I would certainly like to do that (it would save passing in > tons of arguments in the recursive calls), but I tried the following > test and got the Cython error "C function definition not allowed here" > (here being "print_i()"): > > cdef class Test(object): > cpdef test(): > cdef int i = 2 > cpdef print_i(): > print i > print_i() > Unfortunately it seems it is only allowed with Python functions (Cython 0.16): cdef class Test(object): def test(self): cdef int i = 2 def print_i(): print i print_i() :-( > > Thanks, this makes more sense. I thought there might be a > circumstance when "inner_maxes = maxes" might create a copy of maxes, > but I now see that it will only check that ndim is 1 and dtype is > np.float64 before making the data available. "inner_maxes" extends the PyArrayObject "maxes" with a PEP 3118 buffer, checks the assigned NumPy array "maxes", uses PyArray_DATA et al. to grab the data pointer, dimensions and strides, and unboxes them into the local scope. I does not make a copy. This is also why the current "inner_maxes.data" is wrong, becuase this will access the "data" member in the PyArrayObject struct "maxes" (from which np.ndarray in Cython inherits) - and we should not touch that member directly. Sturla -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Mon Jul 16 16:31:42 2012 From: sturla at molden.no (Sturla Molden) Date: Mon, 16 Jul 2012 22:31:42 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> <500057B4.4020505@molden.no> <50010996.3030802@molden.no> <5001AE2D.9060806@molden.no> <5001B455.7080406@molden.no> Message-ID: <50047A2E.4070402@molden.no> Den 14.07.2012 23:03, skrev Ralf Gommers: > > $ git remote add rgommers https://github.com/rgommers/scipy > $ git fetch rgommers > $ git checkout -b fixes-sturla rgommers/fixes-sturla At least I am trying... D:\test>git remote add rgommers https://github.com/rgommers/scipy fatal: Not a git repository (or any of the parent directories): .git ? Sturla From ralf.gommers at googlemail.com Mon Jul 16 16:39:17 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 16 Jul 2012 22:39:17 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: <50047A2E.4070402@molden.no> References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> <500057B4.4020505@molden.no> <50010996.3030802@molden.no> <5001AE2D.9060806@molden.no> <5001B455.7080406@molden.no> <50047A2E.4070402@molden.no> Message-ID: On Mon, Jul 16, 2012 at 10:31 PM, Sturla Molden wrote: > Den 14.07.2012 23:03, skrev Ralf Gommers: > > > > $ git remote add rgommers https://github.com/rgommers/scipy > > $ git fetch rgommers > > $ git checkout -b fixes-sturla rgommers/fixes-sturla > > At least I am trying... > > D:\test>git remote add rgommers https://github.com/rgommers/scipy > fatal: Not a git repository (or any of the parent directories): .git > Sorry, should have been "git remote add rgommers https://github.com/rgommers/scipy.git" Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From samtygier at yahoo.co.uk Mon Jul 16 17:43:24 2012 From: samtygier at yahoo.co.uk (sam tygier) Date: Mon, 16 Jul 2012 22:43:24 +0100 Subject: [SciPy-Dev] SSE optimization In-Reply-To: References: Message-ID: On 09/07/12 19:51, Fode wrote: > I would like to add sse optimization to numpy if that is still something the project is interested in. Do you have any suggestions as to where to start optimizing. > > Fode If you want to speed up numpy operations you might want to look at numexpr. sam From sturla at molden.no Mon Jul 16 18:21:21 2012 From: sturla at molden.no (Sturla Molden) Date: Tue, 17 Jul 2012 00:21:21 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> <500057B4.4020505@molden.no> <50010996.3030802@molden.no> <5001AE2D.9060806@molden.no> <5001B455.7080406@molden.no> Message-ID: <500493E1.50306@molden.no> Den 14.07.2012 23:03, skrev Ralf Gommers: > > Then to push them to github: > $ git push origin fixes-sturla > D:\GitHub\scipy [fixes-sturla]> git push origin fixes-sturla error: The requested URL returned error: 403 while accessing https://github.com/ rgommers/scipy.git/info/refs fatal: HTTP request failed Nope, doesn't work. Enough learning git for today :) Sturla From patvarilly at gmail.com Mon Jul 16 19:21:04 2012 From: patvarilly at gmail.com (Patrick Varilly) Date: Tue, 17 Jul 2012 00:21:04 +0100 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: <5004589F.7040908@molden.no> References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> <500057B4.4020505@molden.no> <50010996.3030802@molden.no> <5001AE2D.9060806@molden.no> <5001B455.7080406@molden.no> <5001F471.1010404@molden.no> <500200CA.1030109@molden.no> <5004589F.7040908@molden.no> Message-ID: Ok, I finished cleaning up the other functions that keep track of rectangle-rectangle min/max distances. The other enhancements with respect to int's and PyArray_DATA are also in (one case where it's not so easy to get rid of PyArray_DATA(arr) is when arr is only defined as an np.ndarray, as it must if it's a class member variable). I'll attack query_ball_point and perhaps query tomorrow night and do some more general clean-up then as well (e.g., in how Rectangle objects get set up). I haven't yet looked into the ref-count issue Sturla brought up. Thanks for all the useful feedback today! Patrick On Mon, Jul 16, 2012 at 7:08 PM, Sturla Molden wrote: > Den 16.07.2012 02:00, skrev Patrick Varilly: > >> >> That said, now that the initial effort is complete, it's obvious that a >> clear pattern recurs in the kd-tree walking algorithms: keeping incremental >> track of the minimum/maximum distances between two hyperrectangles as >> they're successively split (alternatively, a hyperrectangle and a point). >> If that primitive is abstracted, the code becomes a lot more readable >> again. >> > > Yes, that was much better :-) > +1 for doping it like this (even if there is a small speed penalty). > > We might even do this to the original cKDTree.query function as well. > > Sturla > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josh.k.lawrence at gmail.com Mon Jul 16 19:30:35 2012 From: josh.k.lawrence at gmail.com (Josh Lawrence) Date: Mon, 16 Jul 2012 19:30:35 -0400 Subject: [SciPy-Dev] update on 0.11.0 status In-Reply-To: References: <4FFC577A.6070007@astro.washington.edu> <4FFC961C.8060605@astro.washington.edu> <4FFC9FB5.6090006@astro.washington.edu> <500356ED.7010401@astro.washington.edu> Message-ID: This one needs to be reviewed as well: https://github.com/scipy/scipy/pull/259 --Josh On Jul 16, 2012, at 2:34 PM, Ralf Gommers wrote: > > > On Mon, Jul 16, 2012 at 1:49 AM, Jake Vanderplas wrote: > Good catch, Ralf! > Initializing all values of dist_array to zero is the correct thing to do here. I don't know how I missed that... > > Great, thanks for confirming. > > Now that all issues have fixes available, I plan to merge all open PRs by tomorrow and prepare a release candidate asap. Some PRs haven't been reviewed yet but they're very straightforward - please have a look at them or let me know you intend to do so if you want me to wait. > > https://github.com/scipy/scipy/pull/267 > https://github.com/scipy/scipy/pull/268 > https://github.com/scipy/scipy/pull/258 > https://github.com/scipy/scipy/pull/257 > https://github.com/scipy/scipy/pull/266 > > Ralf > > > > On 07/15/2012 07:57 AM, Ralf Gommers wrote: > >> >> >> On Wed, Jul 11, 2012 at 11:45 PM, Ralf Gommers wrote: >> >> >> On Tue, Jul 10, 2012 at 11:33 PM, Jake Vanderplas wrote: >> Ralf, >> I tried running valgrind on the offending code. I don't see anything out of the ordinary come up on my system (for reference, the scikit-learn developer documentation has some tips on using valgrind for cython code [1]). This makes me suspect that the error you're seeing may be a system-specific issue (i.e. 32 bit/64 bit mismatch), or perhaps something related to the numpy version (I'm using the current numpy 1.8.0-dev). >> >> Could you run valgrind on your system with test_shortest_path.py and see if anything obvious comes up? >> >> It's clean, some noise related to Python itself but nothing related to csgraph. >> >> This was 32-bit python 2.6, numpy 1.8-dev by the way. I haven't yet checked if I get the same when running in 64-bit mode on the same machine yet. >> >> Found the issue (use of an array created with np.empty), see http://projects.scipy.org/scipy/ticket/1701. >> >> Ralf >> >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Mon Jul 16 19:44:21 2012 From: sturla at molden.no (Sturla Molden) Date: Tue, 17 Jul 2012 01:44:21 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> <500057B4.4020505@molden.no> <50010996.3030802@molden.no> <5001AE2D.9060806@molden.no> <5001B455.7080406@molden.no> <5001F471.1010404@molden.no> <500200CA.1030109@molden.no> <5004589F.7040908@molden.no> Message-ID: <5004A755.1050106@molden.no> Excellent :) Minor issue: On line 1750 in ckdtree.pyx, edit dtype from np.int to np.intp. With this edit all tests pass on Windows 64. Sturla Den 17.07.2012 01:21, skrev Patrick Varilly: > Ok, I finished cleaning up the other functions that keep track of > rectangle-rectangle min/max distances. The other enhancements with > respect to int's and PyArray_DATA are also in (one case where it's not > so easy to get rid of PyArray_DATA(arr) is when arr is only defined as > an np.ndarray, as it must if it's a class member variable). I'll > attack query_ball_point and perhaps query tomorrow night and do some > more general clean-up then as well (e.g., in how Rectangle objects get > set up). I haven't yet looked into the ref-count issue Sturla brought up. > > Thanks for all the useful feedback today! > > Patrick > > On Mon, Jul 16, 2012 at 7:08 PM, Sturla Molden > wrote: > > Den 16.07.2012 02:00, skrev Patrick Varilly: > > > That said, now that the initial effort is complete, it's > obvious that a clear pattern recurs in the kd-tree walking > algorithms: keeping incremental track of the minimum/maximum > distances between two hyperrectangles as they're successively > split (alternatively, a hyperrectangle and a point). If that > primitive is abstracted, the code becomes a lot more readable > again. > > > Yes, that was much better :-) > +1 for doping it like this (even if there is a small speed penalty). > > We might even do this to the original cKDTree.query function as well. > > Sturla > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Mon Jul 16 19:54:33 2012 From: sturla at molden.no (Sturla Molden) Date: Tue, 17 Jul 2012 01:54:33 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> <500057B4.4020505@molden.no> <50010996.3030802@molden.no> <5001AE2D.9060806@molden.no> <5001B455.7080406@molden.no> <5001F471.1010404@molden.no> <500200CA.1030109@molden.no> <5004589F.7040908@molden.no> Message-ID: <5004A9B9.1060001@molden.no> Den 17.07.2012 01:21, skrev Patrick Varilly: > Ok, I finished cleaning up the other functions that keep track of > rectangle-rectangle min/max distances. The other enhancements with > respect to int's and PyArray_DATA are also in (one case where it's not > so easy to get rid of PyArray_DATA(arr) is when arr is only defined as > an np.ndarray, as it must if it's a class member variable). Then you need an ndarray in the local scope for unboxing, cf. maxes vs. inner_maxes (it's the same issue). Or just use PyArray_DATA, it's ok too. Sturla From sturla at molden.no Mon Jul 16 20:34:55 2012 From: sturla at molden.no (Sturla Molden) Date: Tue, 17 Jul 2012 02:34:55 +0200 Subject: [SciPy-Dev] update on 0.11.0 status In-Reply-To: References: <4FFC577A.6070007@astro.washington.edu> <4FFC961C.8060605@astro.washington.edu> <4FFC9FB5.6090006@astro.washington.edu> <500356ED.7010401@astro.washington.edu> Message-ID: <5004B32F.4080105@molden.no> I believe changing the order of arguments is correct. Currently the rtol test in the call to allclose fails since it is relative to 0. I think rtol should be kept and/or atol should probably be much smaller than 1E-14. Sturla Den 17.07.2012 01:30, skrev Josh Lawrence: > This one needs to be reviewed as well: > > https://github.com/scipy/scipy/pull/259 > > > --Josh > > On Jul 16, 2012, at 2:34 PM, Ralf Gommers wrote: > >> >> >> On Mon, Jul 16, 2012 at 1:49 AM, Jake Vanderplas >> > > wrote: >> >> Good catch, Ralf! >> Initializing all values of dist_array to zero is the correct >> thing to do here. I don't know how I missed that... >> >> >> Great, thanks for confirming. >> >> Now that all issues have fixes available, I plan to merge all open >> PRs by tomorrow and prepare a release candidate asap. Some PRs >> haven't been reviewed yet but they're very straightforward - please >> have a look at them or let me know you intend to do so if you want me >> to wait. >> >> https://github.com/scipy/scipy/pull/267 >> https://github.com/scipy/scipy/pull/268 >> https://github.com/scipy/scipy/pull/258 >> https://github.com/scipy/scipy/pull/257 >> https://github.com/scipy/scipy/pull/266 >> >> Ralf >> >> >> >> On 07/15/2012 07:57 AM, Ralf Gommers wrote: >> >>> >>> >>> On Wed, Jul 11, 2012 at 11:45 PM, Ralf Gommers >>> >> > wrote: >>> >>> >>> >>> On Tue, Jul 10, 2012 at 11:33 PM, Jake Vanderplas >>> >> > wrote: >>> >>> Ralf, >>> I tried running valgrind on the offending code. I don't >>> see anything out of the ordinary come up on my system >>> (for reference, the scikit-learn developer documentation >>> has some tips on using valgrind for cython code [1]). >>> This makes me suspect that the error you're seeing may >>> be a system-specific issue (i.e. 32 bit/64 bit >>> mismatch), or perhaps something related to the numpy >>> version (I'm using the current numpy 1.8.0-dev). >>> >>> Could you run valgrind on your system with >>> test_shortest_path.py and see if anything obvious comes up? >>> >>> >>> It's clean, some noise related to Python itself but nothing >>> related to csgraph. >>> >>> This was 32-bit python 2.6, numpy 1.8-dev by the way. I >>> haven't yet checked if I get the same when running in 64-bit >>> mode on the same machine yet. >>> >>> >>> Found the issue (use of an array created with np.empty), see >>> http://projects.scipy.org/scipy/ticket/1701. >>> >>> Ralf >>> >>> >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-dev >> >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From josh.k.lawrence at gmail.com Mon Jul 16 20:43:13 2012 From: josh.k.lawrence at gmail.com (Josh Lawrence) Date: Mon, 16 Jul 2012 20:43:13 -0400 Subject: [SciPy-Dev] update on 0.11.0 status In-Reply-To: <5004B32F.4080105@molden.no> References: <4FFC577A.6070007@astro.washington.edu> <4FFC961C.8060605@astro.washington.edu> <4FFC9FB5.6090006@astro.washington.edu> <500356ED.7010401@astro.washington.edu> <5004B32F.4080105@molden.no> Message-ID: <1960FFB6-A39A-48CB-91EE-DCFA4DCCF686@gmail.com> I'll follow on your suggestion of keeping rtol instead of atol. Let me rerun the tests to make sure 1e-14 is not too small for rtol. As far as git goes, should I rebase or should I just leave the rtol commit as is? --Josh On Jul 16, 2012, at 8:34 PM, Sturla Molden wrote: > I believe changing the order of arguments is correct. Currently the rtol test in the call to allclose fails since it is relative to 0. > > I think rtol should be kept and/or atol should probably be much smaller than 1E-14. > > Sturla > > > > Den 17.07.2012 01:30, skrev Josh Lawrence: >> >> This one needs to be reviewed as well: >> >> https://github.com/scipy/scipy/pull/259 >> >> >> --Josh >> >> On Jul 16, 2012, at 2:34 PM, Ralf Gommers wrote: >> >>> >>> >>> On Mon, Jul 16, 2012 at 1:49 AM, Jake Vanderplas wrote: >>> Good catch, Ralf! >>> Initializing all values of dist_array to zero is the correct thing to do here. I don't know how I missed that... >>> >>> Great, thanks for confirming. >>> >>> Now that all issues have fixes available, I plan to merge all open PRs by tomorrow and prepare a release candidate asap. Some PRs haven't been reviewed yet but they're very straightforward - please have a look at them or let me know you intend to do so if you want me to wait. >>> >>> https://github.com/scipy/scipy/pull/267 >>> https://github.com/scipy/scipy/pull/268 >>> https://github.com/scipy/scipy/pull/258 >>> https://github.com/scipy/scipy/pull/257 >>> https://github.com/scipy/scipy/pull/266 >>> >>> Ralf >>> >>> >>> >>> On 07/15/2012 07:57 AM, Ralf Gommers wrote: >>> >>>> >>>> >>>> On Wed, Jul 11, 2012 at 11:45 PM, Ralf Gommers wrote: >>>> >>>> >>>> On Tue, Jul 10, 2012 at 11:33 PM, Jake Vanderplas wrote: >>>> Ralf, >>>> I tried running valgrind on the offending code. I don't see anything out of the ordinary come up on my system (for reference, the scikit-learn developer documentation has some tips on using valgrind for cython code [1]). This makes me suspect that the error you're seeing may be a system-specific issue (i.e. 32 bit/64 bit mismatch), or perhaps something related to the numpy version (I'm using the current numpy 1.8.0-dev). >>>> >>>> Could you run valgrind on your system with test_shortest_path.py and see if anything obvious comes up? >>>> >>>> It's clean, some noise related to Python itself but nothing related to csgraph. >>>> >>>> This was 32-bit python 2.6, numpy 1.8-dev by the way. I haven't yet checked if I get the same when running in 64-bit mode on the same machine yet. >>>> >>>> Found the issue (use of an array created with np.empty), see http://projects.scipy.org/scipy/ticket/1701. >>>> >>>> Ralf >>>> >>>> >>>> >>>> _______________________________________________ >>>> SciPy-Dev mailing list >>>> SciPy-Dev at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>> >>> >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>> >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-dev >> >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Jul 16 21:28:18 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 16 Jul 2012 19:28:18 -0600 Subject: [SciPy-Dev] update on 0.11.0 status In-Reply-To: <5004B32F.4080105@molden.no> References: <4FFC577A.6070007@astro.washington.edu> <4FFC961C.8060605@astro.washington.edu> <4FFC9FB5.6090006@astro.washington.edu> <500356ED.7010401@astro.washington.edu> <5004B32F.4080105@molden.no> Message-ID: On Mon, Jul 16, 2012 at 6:34 PM, Sturla Molden wrote: > I believe changing the order of arguments is correct. Currently the rtol > test in the call to allclose fails since it is relative to 0. > > I think rtol should be kept and/or atol should probably be much smaller > than 1E-14. > Usually both should be specified. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Tue Jul 17 02:18:52 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 17 Jul 2012 08:18:52 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: <500493E1.50306@molden.no> References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> <500057B4.4020505@molden.no> <50010996.3030802@molden.no> <5001AE2D.9060806@molden.no> <5001B455.7080406@molden.no> <500493E1.50306@molden.no> Message-ID: On Tue, Jul 17, 2012 at 12:21 AM, Sturla Molden wrote: > Den 14.07.2012 23:03, skrev Ralf Gommers: > > > > Then to push them to github: > > $ git push origin fixes-sturla > > > > D:\GitHub\scipy [fixes-sturla]> git push origin fixes-sturla > error: The requested URL returned error: 403 while accessing > https://github.com/ > rgommers/scipy.git/info/refs > fatal: HTTP request failed > > Nope, doesn't work. > > Enough learning git for today :) > "origin" is the default name for your own Github repo. If it's named differently for you, change it to the correct name. You managed to push to your own master branch, so it can't be ssh problems or something like that. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Tue Jul 17 02:53:55 2012 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 17 Jul 2012 08:53:55 +0200 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: <500493E1.50306@molden.no> References: <4FFEFE82.5090303@molden.no> <500057B4.4020505@molden.no> <50010996.3030802@molden.no> <5001AE2D.9060806@molden.no> <5001B455.7080406@molden.no> <500493E1.50306@molden.no> Message-ID: <20120717065355.GB12154@phare.normalesup.org> On Tue, Jul 17, 2012 at 12:21:21AM +0200, Sturla Molden wrote: > Enough learning git for today :) Maybe you could try http://windows.github.com/ >From what I understand it's a GUI for git under windows. The github guys tend to be fairly good at usability. G From patvarilly at gmail.com Tue Jul 17 04:22:34 2012 From: patvarilly at gmail.com (Patrick Varilly) Date: Tue, 17 Jul 2012 09:22:34 +0100 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: <5004A755.1050106@molden.no> References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> <500057B4.4020505@molden.no> <50010996.3030802@molden.no> <5001AE2D.9060806@molden.no> <5001B455.7080406@molden.no> <5001F471.1010404@molden.no> <500200CA.1030109@molden.no> <5004589F.7040908@molden.no> <5004A755.1050106@molden.no> Message-ID: Brilliant! Done, Patrick On Tue, Jul 17, 2012 at 12:44 AM, Sturla Molden wrote: > Excellent :) > > Minor issue: > > On line 1750 in ckdtree.pyx, edit dtype from np.int to np.intp. With this > edit all tests pass on Windows 64. > > Sturla > > > > Den 17.07.2012 01:21, skrev Patrick Varilly: > > Ok, I finished cleaning up the other functions that keep track of > rectangle-rectangle min/max distances. The other enhancements with respect > to int's and PyArray_DATA are also in (one case where it's not so easy to > get rid of PyArray_DATA(arr) is when arr is only defined as an np.ndarray, > as it must if it's a class member variable). I'll attack query_ball_point > and perhaps query tomorrow night and do some more general clean-up then as > well (e.g., in how Rectangle objects get set up). I haven't yet looked > into the ref-count issue Sturla brought up. > > Thanks for all the useful feedback today! > > Patrick > > On Mon, Jul 16, 2012 at 7:08 PM, Sturla Molden wrote: > >> Den 16.07.2012 02:00, skrev Patrick Varilly: >> >>> >>> That said, now that the initial effort is complete, it's obvious that a >>> clear pattern recurs in the kd-tree walking algorithms: keeping incremental >>> track of the minimum/maximum distances between two hyperrectangles as >>> they're successively split (alternatively, a hyperrectangle and a point). >>> If that primitive is abstracted, the code becomes a lot more readable >>> again. >>> >> >> Yes, that was much better :-) >> +1 for doping it like this (even if there is a small speed penalty). >> >> We might even do this to the original cKDTree.query function as well. >> >> Sturla >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Wed Jul 18 14:33:09 2012 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 18 Jul 2012 14:33:09 -0400 Subject: [SciPy-Dev] update on 0.11.0 status In-Reply-To: References: Message-ID: On Tue, Jul 10, 2012 at 10:13 AM, Ralf Gommers wrote: > Here's a short update on the status of the 0.11.0 release. Some of the Could we also have a look at: https://github.com/scipy/scipy/pull/269 This is holding up some other packages in Debian. Thanks St?fan From ralf.gommers at googlemail.com Wed Jul 18 15:40:27 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 18 Jul 2012 21:40:27 +0200 Subject: [SciPy-Dev] update on 0.11.0 status In-Reply-To: References: Message-ID: On Wed, Jul 18, 2012 at 8:33 PM, St?fan van der Walt wrote: > On Tue, Jul 10, 2012 at 10:13 AM, Ralf Gommers > wrote: > > Here's a short update on the status of the 0.11.0 release. Some of the > > Could we also have a look at: > > https://github.com/scipy/scipy/pull/269 > > This is holding up some other packages in Debian. > That's a good reason to still include it. I have already compiled an RC though, and I think that patch needs some more review/testing before putting it into 0.11.x. How about I tag RC1 now, and if this patch is OK consider it for a second RC? If there's no other reason for a second RC, it would also be possible to apply the patch only for Debian, right? It seems to only affect one exotic platform. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Wed Jul 18 15:49:40 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 18 Jul 2012 12:49:40 -0700 Subject: [SciPy-Dev] Segfaults building against numpy / scipy on OSX Lion Message-ID: Hi, Please forgive the half-baked question, but, I'm trying to build our package nipy (nipy.org) against stock downloaded numpy-1.6.2-py2.7-python.org-macosx10.6.dmg scipy-0.10.1-py2.7-python.org-macosx10.6.dmg I'm using latest XCode 4.3.3 $ clang --version Apple clang version 3.1 (tags/Apple/clang-318.0.61) (based on LLVM 3.1svn) Target: x86_64-apple-darwin11.4.0 Thread model: posix on Lion: $ uname -a Darwin pis-macbook-air 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr 9 19:33:05 PDT 2012; root:xnu-1699.26.8~1/RELEASE_I386 i386 I believe the correct incantation for the build is: http://scipy.org/Installing_SciPy/Mac_OS_X CC=clang CXX=clang FFLAGS=-ff2c python setup.py build_ext -i This does build, but during the tests, and only on this platform, segfaults (report attached). Is this build / link expected to work? Did I get the right flags? Thanks a lot, Matthew -------------- next part -------------- A non-text attachment was scrubbed... Name: lion-segmentation-crash.debug.gz Type: application/x-gzip Size: 11418 bytes Desc: not available URL: From ralf.gommers at googlemail.com Thu Jul 19 11:29:21 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 19 Jul 2012 17:29:21 +0200 Subject: [SciPy-Dev] Segfaults building against numpy / scipy on OSX Lion In-Reply-To: References: Message-ID: On Wed, Jul 18, 2012 at 9:49 PM, Matthew Brett wrote: > Hi, > > Please forgive the half-baked question, but, I'm trying to build our > package nipy (nipy.org) against stock downloaded > > numpy-1.6.2-py2.7-python.org-macosx10.6.dmg > scipy-0.10.1-py2.7-python.org-macosx10.6.dmg > You're only building against numpy, and scipy is a runtime dependency, right? > I'm using latest XCode 4.3.3 > > $ clang --version > Apple clang version 3.1 (tags/Apple/clang-318.0.61) (based on LLVM 3.1svn) > Target: x86_64-apple-darwin11.4.0 > Thread model: posix > > on Lion: > > $ uname -a > Darwin pis-macbook-air 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr 9 > 19:33:05 PDT 2012; root:xnu-1699.26.8~1/RELEASE_I386 i386 > > I believe the correct incantation for the build is: > > http://scipy.org/Installing_SciPy/Mac_OS_X > > CC=clang CXX=clang FFLAGS=-ff2c python setup.py build_ext -i > It has been reported to work for scipy, that's as much as I know. You can try if building scipy in the same way works for you. If it doesn't, the problem may not be in nipy. > > This does build, but during the tests, and only on this platform, > segfaults (report attached). > > Is this build / link expected to work? Did I get the right flags? > Not sure. Some things to check: that you have the 64-bit python from python.org, that MACOSX_DEPLOYMENT_TARGET is set to 10.6 during the build. The by far most reliable way to make everything work together is to use the exact same compiler. Which is not available anymore with the most recent XCode. You could look into ways to get gcc-4.2 to install on 10.7. And make sure to build your binaries on 10.6 for 64-bit 2.7 and on 10.5 for the other Python 2.x flavors if you can. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From slasley at space.umd.edu Thu Jul 19 11:40:03 2012 From: slasley at space.umd.edu (Scott Lasley) Date: Thu, 19 Jul 2012 11:40:03 -0400 Subject: [SciPy-Dev] Segfaults building against numpy / scipy on OSX Lion In-Reply-To: References: Message-ID: On Jul 19, 2012, at 11:29 AM, Ralf Gommers wrote: > > > On Wed, Jul 18, 2012 at 9:49 PM, Matthew Brett wrote: > Hi, > > Please forgive the half-baked question, but, I'm trying to build our > package nipy (nipy.org) against stock downloaded > > numpy-1.6.2-py2.7-python.org-macosx10.6.dmg > scipy-0.10.1-py2.7-python.org-macosx10.6.dmg > > You're only building against numpy, and scipy is a runtime dependency, right? > > I'm using latest XCode 4.3.3 > > $ clang --version > Apple clang version 3.1 (tags/Apple/clang-318.0.61) (based on LLVM 3.1svn) > Target: x86_64-apple-darwin11.4.0 > Thread model: posix > > on Lion: > > $ uname -a > Darwin pis-macbook-air 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr 9 > 19:33:05 PDT 2012; root:xnu-1699.26.8~1/RELEASE_I386 i386 > > I believe the correct incantation for the build is: > > http://scipy.org/Installing_SciPy/Mac_OS_X > > CC=clang CXX=clang FFLAGS=-ff2c python setup.py build_ext -i > > It has been reported to work for scipy, that's as much as I know. You can try if building scipy in the same way works for you. If it doesn't, the problem may not be in nipy. > > > This does build, but during the tests, and only on this platform, > segfaults (report attached). > > Is this build / link expected to work? Did I get the right flags? > > Not sure. Some things to check: that you have the 64-bit python from python.org, that MACOSX_DEPLOYMENT_TARGET is set to 10.6 during the build. > > The by far most reliable way to make everything work together is to use the exact same compiler. Which is not available anymore with the most recent XCode. You could look into ways to get gcc-4.2 to install on 10.7. And make sure to build your binaries on 10.6 for 64-bit 2.7 and on 10.5 for the other Python 2.x flavors if you can. > > Ralf You can get installer packages for gcc-4.2 and gfortran-4.2 for Lion through Tiger from the R project http://r.research.att.com/tools/ Scott From ralf.gommers at googlemail.com Thu Jul 19 14:10:28 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 19 Jul 2012 20:10:28 +0200 Subject: [SciPy-Dev] ANN: SciPy 0.11.0 release candidate 1 Message-ID: Hi, I am pleased to announce the availability of the first release candidate of SciPy 0.11.0. For this release many new features have been added, and over 120 tickets and pull requests have been closed. Also noteworthy is that the number of contributors for this release has risen to over 50. Some of the highlights are: - A new module, sparse.csgraph, has been added which provides a number of common sparse graph algorithms. - New unified interfaces to the existing optimization and root finding functions have been added. Sources and binaries can be found at https://sourceforge.net/projects/scipy/files/scipy/0.11.0rc1/, release notes are copied below. Please try this release candidate and report any problems on the scipy mailing lists. Cheers, Ralf ========================== SciPy 0.11.0 Release Notes ========================== .. note:: Scipy 0.11.0 is not released yet! .. contents:: SciPy 0.11.0 is the culmination of 8 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation. Highlights of this release are: - A new module has been added which provides a number of common sparse graph algorithms. - New unified interfaces to the existing optimization and root finding functions have been added. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Our development attention will now shift to bug-fix releases on the 0.11.x branch, and on adding new features on the master branch. This release requires Python 2.4-2.7 or 3.1-3.2 and NumPy 1.5.1 or greater. New features ============ Sparse Graph Submodule ---------------------- The new submodule :mod:`scipy.sparse.csgraph` implements a number of efficient graph algorithms for graphs stored as sparse adjacency matrices. Available routines are: - :func:`connected_components` - determine connected components of a graph - :func:`laplacian` - compute the laplacian of a graph - :func:`shortest_path` - compute the shortest path between points on a positive graph - :func:`dijkstra` - use Dijkstra's algorithm for shortest path - :func:`floyd_warshall` - use the Floyd-Warshall algorithm for shortest path - :func:`breadth_first_order` - compute a breadth-first order of nodes - :func:`depth_first_order` - compute a depth-first order of nodes - :func:`breadth_first_tree` - construct the breadth-first tree from a given node - :func:`depth_first_tree` - construct a depth-first tree from a given node - :func:`minimum_spanning_tree` - construct the minimum spanning tree of a graph ``scipy.optimize`` improvements ------------------------------- The optimize module has received a lot of attention this release. In addition to added tests, documentation improvements, bug fixes and code clean-up, the following improvements were made: - A unified interface to minimizers of univariate and multivariate functions has been added. - A unified interface to root finding algorithms for multivariate functions has been added. - The L-BFGS-B algorithm has been updated to version 3.0. Unified interfaces to minimizers ```````````````````````````````` Two new functions ``scipy.optimize.minimize`` and ``scipy.optimize.minimize_scalar`` were added to provide a common interface to minimizers of multivariate and univariate functions respectively. For multivariate functions, ``scipy.optimize.minimize`` provides an interface to methods for unconstrained optimization (`fmin`, `fmin_powell`, `fmin_cg`, `fmin_ncg`, `fmin_bfgs` and `anneal`) or constrained optimization (`fmin_l_bfgs_b`, `fmin_tnc`, `fmin_cobyla` and `fmin_slsqp`). For univariate functions, ``scipy.optimize.minimize_scalar`` provides an interface to methods for unconstrained and bounded optimization (`brent`, `golden`, `fminbound`). This allows for easier comparing and switching between solvers. Unified interface to root finding algorithms ```````````````````````````````````````````` The new function ``scipy.optimize.root`` provides a common interface to root finding algorithms for multivariate functions, embeding `fsolve`, `leastsq` and `nonlin` solvers. ``scipy.linalg`` improvements ----------------------------- New matrix equation solvers ``````````````````````````` Solvers for the Sylvester equation (``scipy.linalg.solve_sylvester``, discrete and continuous Lyapunov equations (``scipy.linalg.solve_lyapunov``, ``scipy.linalg.solve_discrete_lyapunov``) and discrete and continuous algebraic Riccati equations (``scipy.linalg.solve_continuous_are``, ``scipy.linalg.solve_discrete_are``) have been added to ``scipy.linalg``. These solvers are often used in the field of linear control theory. QZ and QR Decomposition ```````````````````````` It is now possible to calculate the QZ, or Generalized Schur, decomposition using ``scipy.linalg.qz``. This function wraps the LAPACK routines sgges, dgges, cgges, and zgges. The function ``scipy.linalg.qr_multiply``, which allows efficient computation of the matrix product of Q (from a QR decompostion) and a vector, has been added. Pascal matrices ``````````````` A function for creating Pascal matrices, ``scipy.linalg.pascal``, was added. Sparse matrix construction and operations ----------------------------------------- Two new functions, ``scipy.sparse.diags`` and ``scipy.sparse.block_diag``, were added to easily construct diagonal and block-diagonal sparse matrices respectively. ``scipy.sparse.csc_matrix`` and ``csr_matrix`` now support the operations ``sin``, ``tan``, ``arcsin``, ``arctan``, ``sinh``, ``tanh``, ``arcsinh``, ``arctanh``, ``rint``, ``sign``, ``expm1``, ``log1p``, ``deg2rad``, ``rad2deg``, ``floor``, ``ceil`` and ``trunc``. Previously, these operations had to be performed by operating on the matrices' ``data`` attribute. LSMR iterative solver --------------------- LSMR, an iterative method for solving (sparse) linear and linear least-squares systems, was added as ``scipy.sparse.linalg.lsmr``. Discrete Sine Transform ----------------------- Bindings for the discrete sine transform functions have been added to ``scipy.fftpack``. ``scipy.interpolate`` improvements ---------------------------------- For interpolation in spherical coordinates, the three classes ``scipy.interpolate.SmoothSphereBivariateSpline``, ``scipy.interpolate.LSQSphereBivariateSpline``, and ``scipy.interpolate.RectSphereBivariateSpline`` have been added. Binned statistics (``scipy.stats``) ----------------------------------- The stats module has gained functions to do binned statistics, which are a generalization of histograms, in 1-D, 2-D and multiple dimensions: ``scipy.stats.binned_statistic``, ``scipy.stats.binned_statistic_2d`` and ``scipy.stats.binned_statistic_dd``. Deprecated features =================== ``scipy.sparse.cs_graph_components`` has been made a part of the sparse graph submodule, and renamed to ``scipy.sparse.csgraph.connected_components``. Calling the former routine will result in a deprecation warning. ``scipy.misc.radon`` has been deprecated. A more full-featured radon transform can be found in scikits-image. ``scipy.io.save_as_module`` has been deprecated. A better way to save multiple Numpy arrays is the ``numpy.savez`` function. The `xa` and `xb` parameters for all distributions in ``scipy.stats.distributions`` already weren't used; they have now been deprecated. Backwards incompatible changes ============================== Removal of ``scipy.maxentropy`` ------------------------------- The ``scipy.maxentropy`` module, which was deprecated in the 0.10.0 release, has been removed. Logistic regression in scikits.learn is a good and modern alternative for this functionality. Minor change in behavior of ``splev`` ------------------------------------- The spline evaluation function now behaves similarly to ``interp1d`` for size-1 arrays. Previous behavior:: >>> from scipy.interpolate import splev, splrep, interp1d >>> x = [1,2,3,4,5] >>> y = [4,5,6,7,8] >>> tck = splrep(x, y) >>> splev([1], tck) 4. >>> splev(1, tck) 4. Corrected behavior:: >>> splev([1], tck) array([ 4.]) >>> splev(1, tck) array(4.) This affects also the ``UnivariateSpline`` classes. Behavior of ``scipy.integrate.complex_ode`` ------------------------------------------- The behavior of the ``y`` attribute of ``complex_ode`` is changed. Previously, it expressed the complex-valued solution in the form:: z = ode.y[::2] + 1j * ode.y[1::2] Now, it is directly the complex-valued solution:: z = ode.y Minor change in behavior of T-tests ----------------------------------- The T-tests ``scipy.stats.ttest_ind``, ``scipy.stats.ttest_rel`` and ``scipy.stats.ttest_1samp`` have been changed so that 0 / 0 now returns NaN instead of 1. Other changes ============= The SuperLU sources in ``scipy.sparse.linalg`` have been updated to version 4.3 from upstream. The function ``scipy.signal.bode``, which calculates magnitude and phase data for a continuous-time system, has been added. The two-sample T-test ``scipy.stats.ttest_ind`` gained an option to compare samples with unequal variances, i.e. Welch's T-test. ``scipy.misc.logsumexp`` now takes an optional ``axis`` keyword argument. Authors ======= This release contains work by the following people (contributed at least one patch to this release, names in alphabetical order): * Jeff Armstrong * Chad Baker * Brandon Beacher + * behrisch + * borishim + * Matthew Brett * Lars Buitinck * Luis Pedro Coelho + * Johann Cohen-Tanugi * David Cournapeau * dougal + * Ali Ebrahim + * endolith + * Bj?rn Forsman + * Robert Gantner + * Sebastian Gassner + * Christoph Gohlke * Ralf Gommers * Yaroslav Halchenko * Charles Harris * Jonathan Helmus + * Andreas Hilboll + * Marc Honnorat + * Jonathan Hunt + * Maxim Ivanov + * Thouis (Ray) Jones * Christopher Kuster + * Josh Lawrence + * Denis Laxalde + * Travis Oliphant * Joonas Paalasmaa + * Fabian Pedregosa * Josef Perktold * Gavin Price + * Jim Radford + * Andrew Schein + * Skipper Seabold * Jacob Silterra + * Scott Sinclair * Alexis Tabary + * Martin Teichmann * Matt Terry + * Nicky van Foreest + * Jacob Vanderplas * Patrick Varilly + * Pauli Virtanen * Nils Wagner + * Darryl Wally + * Stefan van der Walt * Liming Wang + * David Warde-Farley + * Warren Weckesser * Sebastian Werk + * Mike Wimmer + * Tony S Yu + A total of 55 people contributed to this release. People with a "+" by their names contributed a patch for the first time. -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Thu Jul 19 15:58:30 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 19 Jul 2012 12:58:30 -0700 Subject: [SciPy-Dev] Segfaults building against numpy / scipy on OSX Lion In-Reply-To: References: Message-ID: Hi, On Thu, Jul 19, 2012 at 8:29 AM, Ralf Gommers wrote: > > > On Wed, Jul 18, 2012 at 9:49 PM, Matthew Brett > wrote: >> >> Hi, >> >> Please forgive the half-baked question, but, I'm trying to build our >> package nipy (nipy.org) against stock downloaded >> >> numpy-1.6.2-py2.7-python.org-macosx10.6.dmg >> scipy-0.10.1-py2.7-python.org-macosx10.6.dmg > > > You're only building against numpy, and scipy is a runtime dependency, > right? Right - thanks for the clarification. >> >> I'm using latest XCode 4.3.3 >> >> $ clang --version >> Apple clang version 3.1 (tags/Apple/clang-318.0.61) (based on LLVM 3.1svn) >> Target: x86_64-apple-darwin11.4.0 >> Thread model: posix >> >> on Lion: >> >> $ uname -a >> Darwin pis-macbook-air 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr 9 >> 19:33:05 PDT 2012; root:xnu-1699.26.8~1/RELEASE_I386 i386 >> >> I believe the correct incantation for the build is: >> >> http://scipy.org/Installing_SciPy/Mac_OS_X >> >> CC=clang CXX=clang FFLAGS=-ff2c python setup.py build_ext -i > > > It has been reported to work for scipy, that's as much as I know. You can > try if building scipy in the same way works for you. If it doesn't, the > problem may not be in nipy. Having linked the http://r.research.att.com/tools/ gfortran-4.2 to gfortran, this scipy build: CC=clang FFLAGS=-ff2c python setup.py build_ext -i builds and passes all tests. Building and linking nipy with the same flags, and with this scipy on the path, passes all tests. Take this scipy off the path (reverting to the system scipy as downloaded) gives another segfault in the same routine as before, but a little later: Thread 0 Crashed:: Dispatch queue: com.apple.main-thread 0 _segmentation.so 0x00000001035eef14 _get_message_mf + 20 (mrf.c:59) 1 _segmentation.so 0x00000001035eebee ve_step + 798 (mrf.c:172) 2 _segmentation.so 0x00000001035ed4dc __pyx_pf_4nipy_10algorithms_12segmentation_13_segmentation__ve_step + 4604 (_segmentation.c:1103) 3 org.python.python 0x00000001000c1e7d PyEval_EvalFrameEx + 25213 >> This does build, but during the tests, and only on this platform, >> segfaults (report attached). >> >> Is this build / link expected to work? Did I get the right flags? > > > Not sure. Some things to check: that you have the 64-bit python from > python.org, that MACOSX_DEPLOYMENT_TARGET is set to 10.6 during the build. The MACOSX_DEPLOYMENT_TARGET did not seem to make a difference. > The by far most reliable way to make everything work together is to use the > exact same compiler. Which is not available anymore with the most recent > XCode. You could look into ways to get gcc-4.2 to install on 10.7. And make > sure to build your binaries on 10.6 for 64-bit 2.7 and on 10.5 for the other > Python 2.x flavors if you can. Hum. I had previously tried the R gcc-4.2 compiler (thanks Scott for the link) and run into: gcc-4.2: /Users/mb312/dev_trees/nipy/libcstat/wrapper/fffpy.c In file included from /Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/unicodeobject.h:4, from /Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/Python.h:85, from /Users/mb312/dev_trees/nipy/libcstat/wrapper/fffpy.h:1, from /Users/mb312/dev_trees/nipy/libcstat/wrapper/fffpy.c:1: /Developer/SDKs/MacOSX10.6.sdk/usr/include/stdarg.h:4:25: error: stdarg.h: No such file or directory Anything obvious that I missed? Cheers, Matthew From cgohlke at uci.edu Thu Jul 19 16:15:44 2012 From: cgohlke at uci.edu (Christoph Gohlke) Date: Thu, 19 Jul 2012 13:15:44 -0700 Subject: [SciPy-Dev] ANN: SciPy 0.11.0 release candidate 1 In-Reply-To: References: Message-ID: <50086AF0.8030501@uci.edu> On 7/19/2012 11:10 AM, Ralf Gommers wrote: > Hi, > > I am pleased to announce the availability of the first release candidate > of SciPy 0.11.0. For this release many new features have been added, and > over 120 tickets and pull requests have been closed. Also noteworthy is > that the number of contributors for this release has risen to over 50. > Some of the highlights are: > > - A new module, sparse.csgraph, has been added which provides a > number of common sparse graph algorithms. > - New unified interfaces to the existing optimization and root > finding functions have been added. > > Sources and binaries can be found at > https://sourceforge.net/projects/scipy/files/scipy/0.11.0rc1/, release > notes are copied below. > > Please try this release candidate and report any problems on the scipy > mailing lists. > > Cheers, > Ralf > Hi Ralf, looks good: scipy 0.11.0rc1 builds and tests OK with numpy-MKL 1.6.2 on win32 and win-amd64, Python 2.6 to 3.2. Only one known Weave test error on Python 3.x. Could be applied to the final? Christoph From ralf.gommers at googlemail.com Thu Jul 19 16:17:54 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 19 Jul 2012 22:17:54 +0200 Subject: [SciPy-Dev] ANN: SciPy 0.11.0 release candidate 1 In-Reply-To: <50086AF0.8030501@uci.edu> References: <50086AF0.8030501@uci.edu> Message-ID: On Thu, Jul 19, 2012 at 10:15 PM, Christoph Gohlke wrote: > On 7/19/2012 11:10 AM, Ralf Gommers wrote: > > Hi, > > > > I am pleased to announce the availability of the first release candidate > > of SciPy 0.11.0. For this release many new features have been added, and > > over 120 tickets and pull requests have been closed. Also noteworthy is > > that the number of contributors for this release has risen to over 50. > > Some of the highlights are: > > > > - A new module, sparse.csgraph, has been added which provides a > > number of common sparse graph algorithms. > > - New unified interfaces to the existing optimization and root > > finding functions have been added. > > > > Sources and binaries can be found at > > https://sourceforge.net/projects/scipy/files/scipy/0.11.0rc1/, release > > notes are copied below. > > > > Please try this release candidate and report any problems on the scipy > > mailing lists. > > > > Cheers, > > Ralf > > > > Hi Ralf, > > looks good: scipy 0.11.0rc1 builds and tests OK with numpy-MKL 1.6.2 on > win32 and win-amd64, Python 2.6 to 3.2. > Great, thanks for the quick feedback! > > Only one known Weave test error on Python 3.x. Could > < > https://github.com/scipy/scipy/commit/c2e8029054e7fbb8083b11996398db8634a8b564 > > > be applied to the final? > Yes, I meant to do that but somehow overlooked it. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Thu Jul 19 17:14:37 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 19 Jul 2012 23:14:37 +0200 Subject: [SciPy-Dev] Segfaults building against numpy / scipy on OSX Lion In-Reply-To: References: Message-ID: On Thu, Jul 19, 2012 at 9:58 PM, Matthew Brett wrote: > Hi, > > On Thu, Jul 19, 2012 at 8:29 AM, Ralf Gommers > wrote: > > > > > > On Wed, Jul 18, 2012 at 9:49 PM, Matthew Brett > > wrote: > >> > >> Hi, > >> > >> Please forgive the half-baked question, but, I'm trying to build our > >> package nipy (nipy.org) against stock downloaded > >> > >> numpy-1.6.2-py2.7-python.org-macosx10.6.dmg > >> scipy-0.10.1-py2.7-python.org-macosx10.6.dmg > > > > > > You're only building against numpy, and scipy is a runtime dependency, > > right? > > Right - thanks for the clarification. > > >> > >> I'm using latest XCode 4.3.3 > >> > >> $ clang --version > >> Apple clang version 3.1 (tags/Apple/clang-318.0.61) (based on LLVM > 3.1svn) > >> Target: x86_64-apple-darwin11.4.0 > >> Thread model: posix > >> > >> on Lion: > >> > >> $ uname -a > >> Darwin pis-macbook-air 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr 9 > >> 19:33:05 PDT 2012; root:xnu-1699.26.8~1/RELEASE_I386 i386 > >> > >> I believe the correct incantation for the build is: > >> > >> http://scipy.org/Installing_SciPy/Mac_OS_X > >> > >> CC=clang CXX=clang FFLAGS=-ff2c python setup.py build_ext -i > > > > > > It has been reported to work for scipy, that's as much as I know. You can > > try if building scipy in the same way works for you. If it doesn't, the > > problem may not be in nipy. > > Having linked the http://r.research.att.com/tools/ gfortran-4.2 to > gfortran, this scipy build: > > CC=clang FFLAGS=-ff2c python setup.py build_ext -i > > builds and passes all tests. > > Building and linking nipy with the same flags, and with this scipy on > the path, passes all tests. > > Take this scipy off the path (reverting to the system scipy as > downloaded) gives another segfault in the same routine as before, but > a little later: > > Thread 0 Crashed:: Dispatch queue: com.apple.main-thread > 0 _segmentation.so 0x00000001035eef14 _get_message_mf > + 20 (mrf.c:59) > 1 _segmentation.so 0x00000001035eebee ve_step + 798 > (mrf.c:172) > 2 _segmentation.so 0x00000001035ed4dc > __pyx_pf_4nipy_10algorithms_12segmentation_13_segmentation__ve_step + > 4604 (_segmentation.c:1103) > 3 org.python.python 0x00000001000c1e7d > PyEval_EvalFrameEx + 25213 > > >> This does build, but during the tests, and only on this platform, > >> segfaults (report attached). > >> > >> Is this build / link expected to work? Did I get the right flags? > > > > > > Not sure. Some things to check: that you have the 64-bit python from > > python.org, that MACOSX_DEPLOYMENT_TARGET is set to 10.6 during the > build. > > The MACOSX_DEPLOYMENT_TARGET did not seem to make a difference. > > > The by far most reliable way to make everything work together is to use > the > > exact same compiler. Which is not available anymore with the most recent > > XCode. You could look into ways to get gcc-4.2 to install on 10.7. And > make > > sure to build your binaries on 10.6 for 64-bit 2.7 and on 10.5 for the > other > > Python 2.x flavors if you can. > > Hum. I had previously tried the R gcc-4.2 compiler (thanks Scott for > the link) and run into: > > gcc-4.2: /Users/mb312/dev_trees/nipy/libcstat/wrapper/fffpy.c > In file included from > > /Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/unicodeobject.h:4, > from > > /Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/Python.h:85, > from > /Users/mb312/dev_trees/nipy/libcstat/wrapper/fffpy.h:1, > from > /Users/mb312/dev_trees/nipy/libcstat/wrapper/fffpy.c:1: > /Developer/SDKs/MacOSX10.6.sdk/usr/include/stdarg.h:4:25: error: > stdarg.h: No such file or directory > > Anything obvious that I missed? > This is typically caused by not having the required SDKs installed, they're under "Optional Installs" on your XCode dvd. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Thu Jul 19 17:29:34 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 19 Jul 2012 14:29:34 -0700 Subject: [SciPy-Dev] Segfaults building against numpy / scipy on OSX Lion In-Reply-To: References: Message-ID: Hi, On Thu, Jul 19, 2012 at 2:14 PM, Ralf Gommers wrote: > > > On Thu, Jul 19, 2012 at 9:58 PM, Matthew Brett > wrote: >> >> Hi, >> >> On Thu, Jul 19, 2012 at 8:29 AM, Ralf Gommers >> wrote: >> > >> > >> > On Wed, Jul 18, 2012 at 9:49 PM, Matthew Brett >> > wrote: >> >> >> >> Hi, >> >> >> >> Please forgive the half-baked question, but, I'm trying to build our >> >> package nipy (nipy.org) against stock downloaded >> >> >> >> numpy-1.6.2-py2.7-python.org-macosx10.6.dmg >> >> scipy-0.10.1-py2.7-python.org-macosx10.6.dmg >> > >> > >> > You're only building against numpy, and scipy is a runtime dependency, >> > right? >> >> Right - thanks for the clarification. >> >> >> >> >> I'm using latest XCode 4.3.3 >> >> >> >> $ clang --version >> >> Apple clang version 3.1 (tags/Apple/clang-318.0.61) (based on LLVM >> >> 3.1svn) >> >> Target: x86_64-apple-darwin11.4.0 >> >> Thread model: posix >> >> >> >> on Lion: >> >> >> >> $ uname -a >> >> Darwin pis-macbook-air 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr 9 >> >> 19:33:05 PDT 2012; root:xnu-1699.26.8~1/RELEASE_I386 i386 >> >> >> >> I believe the correct incantation for the build is: >> >> >> >> http://scipy.org/Installing_SciPy/Mac_OS_X >> >> >> >> CC=clang CXX=clang FFLAGS=-ff2c python setup.py build_ext -i >> > >> > >> > It has been reported to work for scipy, that's as much as I know. You >> > can >> > try if building scipy in the same way works for you. If it doesn't, the >> > problem may not be in nipy. >> >> Having linked the http://r.research.att.com/tools/ gfortran-4.2 to >> gfortran, this scipy build: >> >> CC=clang FFLAGS=-ff2c python setup.py build_ext -i >> >> builds and passes all tests. >> >> Building and linking nipy with the same flags, and with this scipy on >> the path, passes all tests. >> >> Take this scipy off the path (reverting to the system scipy as >> downloaded) gives another segfault in the same routine as before, but >> a little later: >> >> Thread 0 Crashed:: Dispatch queue: com.apple.main-thread >> 0 _segmentation.so 0x00000001035eef14 _get_message_mf >> + 20 (mrf.c:59) >> 1 _segmentation.so 0x00000001035eebee ve_step + 798 >> (mrf.c:172) >> 2 _segmentation.so 0x00000001035ed4dc >> __pyx_pf_4nipy_10algorithms_12segmentation_13_segmentation__ve_step + >> 4604 (_segmentation.c:1103) >> 3 org.python.python 0x00000001000c1e7d >> PyEval_EvalFrameEx + 25213 >> >> >> >> This does build, but during the tests, and only on this platform, >> >> segfaults (report attached). >> >> >> >> Is this build / link expected to work? Did I get the right flags? >> > >> > >> > Not sure. Some things to check: that you have the 64-bit python from >> > python.org, that MACOSX_DEPLOYMENT_TARGET is set to 10.6 during the >> > build. >> >> The MACOSX_DEPLOYMENT_TARGET did not seem to make a difference. >> >> > The by far most reliable way to make everything work together is to use >> > the >> > exact same compiler. Which is not available anymore with the most recent >> > XCode. You could look into ways to get gcc-4.2 to install on 10.7. And >> > make >> > sure to build your binaries on 10.6 for 64-bit 2.7 and on 10.5 for the >> > other >> > Python 2.x flavors if you can. >> >> Hum. I had previously tried the R gcc-4.2 compiler (thanks Scott for >> the link) and run into: >> >> gcc-4.2: /Users/mb312/dev_trees/nipy/libcstat/wrapper/fffpy.c >> In file included from >> >> /Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/unicodeobject.h:4, >> from >> >> /Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/Python.h:85, >> from >> /Users/mb312/dev_trees/nipy/libcstat/wrapper/fffpy.h:1, >> from >> /Users/mb312/dev_trees/nipy/libcstat/wrapper/fffpy.c:1: >> /Developer/SDKs/MacOSX10.6.sdk/usr/include/stdarg.h:4:25: error: >> stdarg.h: No such file or directory >> >> Anything obvious that I missed? > > > This is typically caused by not having the required SDKs installed, they're > under "Optional Installs" on your XCode dvd. I believe the only way to install Xcode for Lion 10.7.4 at least - is via the App Store. I can't see any options for other SDKs in the App Store or on the Xcode components menu. Do you know, has anyone got a gcc-4.2 build working with Lion 10.7.4? See you, Matthew From slasley at space.umd.edu Thu Jul 19 18:22:37 2012 From: slasley at space.umd.edu (Scott Lasley) Date: Thu, 19 Jul 2012 18:22:37 -0400 Subject: [SciPy-Dev] Segfaults building against numpy / scipy on OSX Lion In-Reply-To: References: Message-ID: On Jul 19, 2012, at 5:29 PM, Matthew Brett wrote: > Hi, > > On Thu, Jul 19, 2012 at 2:14 PM, Ralf Gommers > wrote: >> >> >> On Thu, Jul 19, 2012 at 9:58 PM, Matthew Brett >> wrote: >>> >>> Hi, >>> >>> On Thu, Jul 19, 2012 at 8:29 AM, Ralf Gommers >>> wrote: >>>> >>>> >>>> On Wed, Jul 18, 2012 at 9:49 PM, Matthew Brett >>>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> Please forgive the half-baked question, but, I'm trying to build our >>>>> package nipy (nipy.org) against stock downloaded >>>>> >>>>> numpy-1.6.2-py2.7-python.org-macosx10.6.dmg >>>>> scipy-0.10.1-py2.7-python.org-macosx10.6.dmg >>>> >>>> >>>> You're only building against numpy, and scipy is a runtime dependency, >>>> right? >>> >>> Right - thanks for the clarification. >>> >>>>> >>>>> I'm using latest XCode 4.3.3 >>>>> >>>>> $ clang --version >>>>> Apple clang version 3.1 (tags/Apple/clang-318.0.61) (based on LLVM >>>>> 3.1svn) >>>>> Target: x86_64-apple-darwin11.4.0 >>>>> Thread model: posix >>>>> >>>>> on Lion: >>>>> >>>>> $ uname -a >>>>> Darwin pis-macbook-air 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr 9 >>>>> 19:33:05 PDT 2012; root:xnu-1699.26.8~1/RELEASE_I386 i386 >>>>> >>>>> I believe the correct incantation for the build is: >>>>> >>>>> http://scipy.org/Installing_SciPy/Mac_OS_X >>>>> >>>>> CC=clang CXX=clang FFLAGS=-ff2c python setup.py build_ext -i >>>> >>>> >>>> It has been reported to work for scipy, that's as much as I know. You >>>> can >>>> try if building scipy in the same way works for you. If it doesn't, the >>>> problem may not be in nipy. >>> >>> Having linked the http://r.research.att.com/tools/ gfortran-4.2 to >>> gfortran, this scipy build: >>> >>> CC=clang FFLAGS=-ff2c python setup.py build_ext -i >>> >>> builds and passes all tests. >>> >>> Building and linking nipy with the same flags, and with this scipy on >>> the path, passes all tests. >>> >>> Take this scipy off the path (reverting to the system scipy as >>> downloaded) gives another segfault in the same routine as before, but >>> a little later: >>> >>> Thread 0 Crashed:: Dispatch queue: com.apple.main-thread >>> 0 _segmentation.so 0x00000001035eef14 _get_message_mf >>> + 20 (mrf.c:59) >>> 1 _segmentation.so 0x00000001035eebee ve_step + 798 >>> (mrf.c:172) >>> 2 _segmentation.so 0x00000001035ed4dc >>> __pyx_pf_4nipy_10algorithms_12segmentation_13_segmentation__ve_step + >>> 4604 (_segmentation.c:1103) >>> 3 org.python.python 0x00000001000c1e7d >>> PyEval_EvalFrameEx + 25213 >>> >>> >>>>> This does build, but during the tests, and only on this platform, >>>>> segfaults (report attached). >>>>> >>>>> Is this build / link expected to work? Did I get the right flags? >>>> >>>> >>>> Not sure. Some things to check: that you have the 64-bit python from >>>> python.org, that MACOSX_DEPLOYMENT_TARGET is set to 10.6 during the >>>> build. >>> >>> The MACOSX_DEPLOYMENT_TARGET did not seem to make a difference. >>> >>>> The by far most reliable way to make everything work together is to use >>>> the >>>> exact same compiler. Which is not available anymore with the most recent >>>> XCode. You could look into ways to get gcc-4.2 to install on 10.7. And >>>> make >>>> sure to build your binaries on 10.6 for 64-bit 2.7 and on 10.5 for the >>>> other >>>> Python 2.x flavors if you can. >>> >>> Hum. I had previously tried the R gcc-4.2 compiler (thanks Scott for >>> the link) and run into: >>> >>> gcc-4.2: /Users/mb312/dev_trees/nipy/libcstat/wrapper/fffpy.c >>> In file included from >>> >>> /Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/unicodeobject.h:4, >>> from >>> >>> /Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/Python.h:85, >>> from >>> /Users/mb312/dev_trees/nipy/libcstat/wrapper/fffpy.h:1, >>> from >>> /Users/mb312/dev_trees/nipy/libcstat/wrapper/fffpy.c:1: >>> /Developer/SDKs/MacOSX10.6.sdk/usr/include/stdarg.h:4:25: error: >>> stdarg.h: No such file or directory >>> >>> Anything obvious that I missed? >> >> >> This is typically caused by not having the required SDKs installed, they're >> under "Optional Installs" on your XCode dvd. > > I believe the only way to install Xcode for Lion 10.7.4 at least - is > via the App Store. I can't see any options for other SDKs in the App > Store or on the Xcode components menu. Do you know, has anyone got a > gcc-4.2 build working with Lion 10.7.4? > > See you, > > Matthew I'm not sure if it will fix the issue you're having, but you could try downloading the command line tools from https://developer.apple.com/downloads/ You will need an Apple ID to download the disk image. Enter the word "command" in the search field on the Downloads page to narrow down the list, and pick the command line tools for the version of XCode you have. The installer will install the standard headers and libs for making programs from the command line. Scott From patvarilly at gmail.com Thu Jul 19 20:48:04 2012 From: patvarilly at gmail.com (Patrick Varilly) Date: Fri, 20 Jul 2012 01:48:04 +0100 Subject: [SciPy-Dev] Enhancements to scipy.spatial.cKDTree In-Reply-To: References: <4FFC0B6C.5030407@molden.no> <4FFEFE82.5090303@molden.no> <500057B4.4020505@molden.no> <50010996.3030802@molden.no> <5001AE2D.9060806@molden.no> <5001B455.7080406@molden.no> <5001F471.1010404@molden.no> <500200CA.1030109@molden.no> <5004589F.7040908@molden.no> <5004A755.1050106@molden.no> Message-ID: Dear all, I've finished cleaning up the cKDTree code, which should now be much more readable and maintainable than before. It's ready for a final review from top to bottom. I'm going on holidays for two weeks starting tomorrow, but look forward to any comments when I come back. All the best, Patrick On Tue, Jul 17, 2012 at 9:22 AM, Patrick Varilly wrote: > Brilliant! Done, > > Patrick > > > On Tue, Jul 17, 2012 at 12:44 AM, Sturla Molden wrote: > >> Excellent :) >> >> Minor issue: >> >> On line 1750 in ckdtree.pyx, edit dtype from np.int to np.intp. With >> this edit all tests pass on Windows 64. >> >> Sturla >> >> >> >> Den 17.07.2012 01:21, skrev Patrick Varilly: >> >> Ok, I finished cleaning up the other functions that keep track of >> rectangle-rectangle min/max distances. The other enhancements with respect >> to int's and PyArray_DATA are also in (one case where it's not so easy to >> get rid of PyArray_DATA(arr) is when arr is only defined as an np.ndarray, >> as it must if it's a class member variable). I'll attack query_ball_point >> and perhaps query tomorrow night and do some more general clean-up then as >> well (e.g., in how Rectangle objects get set up). I haven't yet looked >> into the ref-count issue Sturla brought up. >> >> Thanks for all the useful feedback today! >> >> Patrick >> >> On Mon, Jul 16, 2012 at 7:08 PM, Sturla Molden wrote: >> >>> Den 16.07.2012 02:00, skrev Patrick Varilly: >>> >>>> >>>> That said, now that the initial effort is complete, it's obvious that >>>> a clear pattern recurs in the kd-tree walking algorithms: keeping >>>> incremental track of the minimum/maximum distances between two >>>> hyperrectangles as they're successively split (alternatively, a >>>> hyperrectangle and a point). If that primitive is abstracted, the code >>>> becomes a lot more readable again. >>>> >>> >>> Yes, that was much better :-) >>> +1 for doping it like this (even if there is a small speed penalty). >>> >>> We might even do this to the original cKDTree.query function as well. >>> >>> Sturla >>> >>> >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott.sinclair.za at gmail.com Fri Jul 20 04:16:24 2012 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Fri, 20 Jul 2012 10:16:24 +0200 Subject: [SciPy-Dev] ANN: SciPy 0.11.0 release candidate 1 In-Reply-To: References: <50086AF0.8030501@uci.edu> Message-ID: On 19 July 2012 22:17, Ralf Gommers wrote: > > > On Thu, Jul 19, 2012 at 10:15 PM, Christoph Gohlke wrote: >> >> looks good: scipy 0.11.0rc1 builds and tests OK with numpy-MKL 1.6.2 on >> win32 and win-amd64, Python 2.6 to 3.2. > > > Great, thanks for the quick feedback! >> >> >> Only one known Weave test error on Python 3.x. Could Python 2.5 - 3.2 look fine on Ubuntu Linux 64-bit (except for the weave error on Py3). On Python 2.4 I see some failures: Running unit tests for scipy NumPy version 1.6.2 NumPy is installed in /home/scott/Downloads/scipy-0.11.0rc1/.tox/py24/lib/python2.4/site-packages/numpy SciPy version 0.11.0rc1 SciPy is installed in /home/scott/Downloads/scipy-0.11.0rc1/.tox/py24/lib/python2.4/site-packages/scipy Python version 2.4.6 (#2, May 21 2012, 16:12:34) [GCC 4.6.3] nose version 1.1.2 ... ====================================================================== ERROR: adding a dense matrix to a sparse matrix ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/scott/Downloads/scipy-0.11.0rc1/.tox/py24/lib/python2.4/site-packages/scipy/sparse/tests/test_base.py", line 573, in test_add_dense sum1 = self.dat + self.datsp File "/home/scott/Downloads/scipy-0.11.0rc1/.tox/py24/lib/python2.4/site-packages/scipy/sparse/dok.py", line 133, in __getitem__ raise TypeError('index must be a pair of integers or slices') TypeError: index must be a pair of integers or slices ====================================================================== ERROR: test_matmat_sparse (test_base.TestDOK) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/scott/Downloads/scipy-0.11.0rc1/.tox/py24/lib/python2.4/site-packages/scipy/sparse/tests/test_base.py", line 471, in test_matmat_sparse assert_array_almost_equal( a2*bsp, a*b) File "/home/scott/Downloads/scipy-0.11.0rc1/.tox/py24/lib/python2.4/site-packages/scipy/sparse/dok.py", line 133, in __getitem__ raise TypeError('index must be a pair of integers or slices') TypeError: index must be a pair of integers or slices ====================================================================== ERROR: test_radd (test_base.TestDOK) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/scott/Downloads/scipy-0.11.0rc1/.tox/py24/lib/python2.4/site-packages/scipy/sparse/tests/test_base.py", line 325, in test_radd c = a + b File "/home/scott/Downloads/scipy-0.11.0rc1/.tox/py24/lib/python2.4/site-packages/scipy/sparse/dok.py", line 133, in __getitem__ raise TypeError('index must be a pair of integers or slices') TypeError: index must be a pair of integers or slices ====================================================================== ERROR: test_rsub (test_base.TestDOK) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/scott/Downloads/scipy-0.11.0rc1/.tox/py24/lib/python2.4/site-packages/scipy/sparse/tests/test_base.py", line 336, in test_rsub assert_array_equal((self.dat - self.datsp),[[0,0,0,0],[0,0,0,0],[0,0,0,0]]) File "/home/scott/Downloads/scipy-0.11.0rc1/.tox/py24/lib/python2.4/site-packages/scipy/sparse/dok.py", line 133, in __getitem__ raise TypeError('index must be a pair of integers or slices') TypeError: index must be a pair of integers or slices ====================================================================== ERROR: subtracting a dense matrix to/from a sparse matrix ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/scott/Downloads/scipy-0.11.0rc1/.tox/py24/lib/python2.4/site-packages/scipy/sparse/tests/test_base.py", line 581, in test_sub_dense sum1 = 3*self.dat - self.datsp File "/home/scott/Downloads/scipy-0.11.0rc1/.tox/py24/lib/python2.4/site-packages/scipy/sparse/dok.py", line 133, in __getitem__ raise TypeError('index must be a pair of integers or slices') TypeError: index must be a pair of integers or slices Cheers, Scott From lists at hilboll.de Fri Jul 20 05:34:01 2012 From: lists at hilboll.de (Andreas Hilboll) Date: Fri, 20 Jul 2012 11:34:01 +0200 Subject: [SciPy-Dev] ANN: SciPy 0.11.0 release candidate 1 In-Reply-To: References: Message-ID: <3fb4522421d3b87335b5d040a0e6cd4b.squirrel@srv2.s4y.tournesol-consulting.eu> > Hi, > > I am pleased to announce the availability of the first release candidate > of > SciPy 0.11.0. For this release many new features have been added, and over > 120 tickets and pull requests have been closed. Also noteworthy is that > the > number of contributors for this release has risen to over 50. Some of the > highlights are: > > - A new module, sparse.csgraph, has been added which provides a number > of > common sparse graph algorithms. > - New unified interfaces to the existing optimization and root finding > functions have been added. > > Sources and binaries can be found at > https://sourceforge.net/projects/scipy/files/scipy/0.11.0rc1/, release > notes are copied below. > > Please try this release candidate and report any problems on the scipy > mailing lists. Failure on Archlinux 64bit, Python 2.7.3, Numpy 1.6.1: This is what I did: mkvirtualenv --system-site-packages --distribute scipy_test_rc1 cd ~/.virtualenvs/scipy_test_rc1 mkdir src wget -O src/scipy-0.11.0rc1.tar.gz "http://downloads.sourceforge.net/project/scipy/scipy/0.11.0rc1/scipy-0.11.0rc1.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fscipy%2Ffiles%2Fscipy%2F0.11.0rc1%2F&ts=1342772081&use_mirror=netcologne" cd src tar xzf scipy-0.11.0rc1.tar.gz cd scipy-0.11.0rc1 python setup.py build python setup.py install cd python -c "import scipy; scipy.test('full')" Running unit tests for scipy NumPy version 1.6.1 NumPy is installed in /usr/lib/python2.7/site-packages/numpy SciPy version 0.11.0rc1 SciPy is installed in /home2/hilboll/.virtualenvs/scipy_test_rc1/lib/python2.7/site-packages/scipy Python version 2.7.3 (default, Apr 24 2012, 00:00:54) [GCC 4.7.0 20120414 (prerelease)] nose version 1.1.2 [...] ====================================================================== FAIL: test_basic.TestNorm.test_stable ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/home2/hilboll/.virtualenvs/scipy_test_rc1/lib/python2.7/site-packages/scipy/linalg/tests/test_basic.py", line 585, in test_stable assert_almost_equal(norm(a) - 1e4, 0.5) File "/usr/lib/python2.7/site-packages/numpy/testing/utils.py", line 468, in assert_almost_equal raise AssertionError(msg) AssertionError: Arrays are not almost equal to 7 decimals ACTUAL: 0.0 DESIRED: 0.5 ---------------------------------------------------------------------- FAILED (KNOWNFAIL=16, SKIP=42, failures=1) From ondrej at certik.cz Fri Jul 20 07:41:40 2012 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 20 Jul 2012 13:41:40 +0200 Subject: [SciPy-Dev] [Numpy-discussion] ANN: SciPy 0.11.0 release candidate 1 In-Reply-To: References: Message-ID: Hi Ralf, On Thu, Jul 19, 2012 at 8:10 PM, Ralf Gommers wrote: > Hi, > > I am pleased to announce the availability of the first release candidate of > SciPy 0.11.0. For this release many new features have been added, and over > 120 tickets and pull requests have been closed. Also noteworthy is that the > number of contributors for this release has risen to over 50. Some of the > highlights are: > > - A new module, sparse.csgraph, has been added which provides a number of > common sparse graph algorithms. > - New unified interfaces to the existing optimization and root finding > functions have been added. > > Sources and binaries can be found at > https://sourceforge.net/projects/scipy/files/scipy/0.11.0rc1/, release notes > are copied below. > > Please try this release candidate and report any problems on the scipy > mailing lists. I've installed the 2.7 superpack on win64, with Python from python.org 32bit and NumPy 1.6.2 superpack (I first tested numpy, all tests pass) and when I run the scipy tests, I get a segfault when executing: test_qz_double_sort (test_decomp.TestQZ) I am using Windows XP 64 bit in a VM. Ondrej From ralf.gommers at googlemail.com Fri Jul 20 12:30:05 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 20 Jul 2012 18:30:05 +0200 Subject: [SciPy-Dev] ANN: SciPy 0.11.0 release candidate 1 In-Reply-To: References: <50086AF0.8030501@uci.edu> Message-ID: On Fri, Jul 20, 2012 at 10:16 AM, Scott Sinclair < scott.sinclair.za at gmail.com> wrote: > On 19 July 2012 22:17, Ralf Gommers wrote: > > > > > > On Thu, Jul 19, 2012 at 10:15 PM, Christoph Gohlke > wrote: > >> > >> looks good: scipy 0.11.0rc1 builds and tests OK with numpy-MKL 1.6.2 on > >> win32 and win-amd64, Python 2.6 to 3.2. > > > > > > Great, thanks for the quick feedback! > >> > >> > >> Only one known Weave test error on Python 3.x. Could > > Python 2.5 - 3.2 look fine on Ubuntu Linux 64-bit (except for the > weave error on Py3). > > On Python 2.4 I see some failures: > > Running unit tests for scipy > NumPy version 1.6.2 > NumPy is installed in > > /home/scott/Downloads/scipy-0.11.0rc1/.tox/py24/lib/python2.4/site-packages/numpy > SciPy version 0.11.0rc1 > SciPy is installed in > > /home/scott/Downloads/scipy-0.11.0rc1/.tox/py24/lib/python2.4/site-packages/scipy > Python version 2.4.6 (#2, May 21 2012, 16:12:34) [GCC 4.6.3] > nose version 1.1.2 > ... > ====================================================================== > ERROR: adding a dense matrix to a sparse matrix > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/scott/Downloads/scipy-0.11.0rc1/.tox/py24/lib/python2.4/site-packages/scipy/sparse/tests/test_base.py", > line 573, in test_add_dense > sum1 = self.dat + self.datsp > File > "/home/scott/Downloads/scipy-0.11.0rc1/.tox/py24/lib/python2.4/site-packages/scipy/sparse/dok.py", > line 133, in __getitem__ > raise TypeError('index must be a pair of integers or slices') > TypeError: index must be a pair of integers or slices > > ====================================================================== > ERROR: test_matmat_sparse (test_base.TestDOK) > ---------------------------------------------------------------------- > There was a ticket for this, http://projects.scipy.org/scipy/ticket/1559, and a PR to mark as knownfail (as it was on 0.10.0), https://github.com/scipy/scipy/pull/237. For some reason this failure went away on some platforms with 2.4, but apparently not for you. Since we had already decided that this wasn't fixable for 2.4 and didn't understand why PR-237 helped at all, I think we should just mark as knownfail now and be done with it. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri Jul 20 15:08:51 2012 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 20 Jul 2012 21:08:51 +0200 Subject: [SciPy-Dev] ANN: SciPy 0.11.0 release candidate 1 In-Reply-To: <3fb4522421d3b87335b5d040a0e6cd4b.squirrel@srv2.s4y.tournesol-consulting.eu> References: <3fb4522421d3b87335b5d040a0e6cd4b.squirrel@srv2.s4y.tournesol-consulting.eu> Message-ID: <5009ACC3.5090701@iki.fi> 20.07.2012 11:34, Andreas Hilboll kirjoitti: [clip] > ====================================================================== > FAIL: test_basic.TestNorm.test_stable > ---------------------------------------------------------------------- [clip] Broken BLAS or something? Try this fortran program: !!! test.f90 program main implicit none real a(10001) real b, snrm2 external snrm2 a(1) = 1e4 a(2:10001) = 1.0 b = snrm2(10001, a, 1) write(*,*) b ! should give 10000.5 end program main -- Pauli Virtanen From scopatz at gmail.com Fri Jul 20 16:40:43 2012 From: scopatz at gmail.com (Anthony Scopatz) Date: Fri, 20 Jul 2012 15:40:43 -0500 Subject: [SciPy-Dev] ANN: PyTables 2.4.0 released Message-ID: ========================== Announcing PyTables 2.4.0 ========================== We are happy to announce PyTables 2.4.0. This is an incremental release which includes many changes to prepare for future Python 3 support. What's new ========== This release includes support for the float16 data type and read-only support for variable length string attributes. The handling of HDF5 errors has been improved. The user will no longer see HDF5 error stacks dumped to the console. All HDF5 error messages are trapped and attached to a proper Python exception. Now PyTables only supports HDF5 v1.8.4+. All the code has been updated to the new HDF5 API. Supporting only HDF5 1.8 series is beneficial for future development. Documentation has been improved. As always, a large amount of bugs have been addressed and squashed as well. In case you want to know more in detail what has changed in this version, please refer to: http://pytables.github.com/release_notes.html You can download a source package with generated PDF and HTML docs, as well as binaries for Windows, from: http://sourceforge.net/projects/pytables/files/pytables/2.4.0 For an online version of the manual, visit: http://pytables.github.com/usersguide/index.html What it is? =========== PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. PyTables includes OPSI, a new indexing technology, allowing to perform data lookups in tables exceeding 10 gigarows (10**10 rows) in less than a tenth of a second. Resources ========= About PyTables: http://www.pytables.org About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments =============== Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Most specially, a lot of kudos go to the HDF5 and NumPy (and numarray!) makers. Without them, PyTables simply would not exist. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Team -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Sat Jul 21 13:10:13 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 21 Jul 2012 10:10:13 -0700 Subject: [SciPy-Dev] Segfaults building against numpy / scipy on OSX Lion In-Reply-To: References: Message-ID: Hi, On Thu, Jul 19, 2012 at 3:22 PM, Scott Lasley wrote: > > On Jul 19, 2012, at 5:29 PM, Matthew Brett wrote: > >> Hi, >> >> On Thu, Jul 19, 2012 at 2:14 PM, Ralf Gommers >> wrote: >>> >>> >>> On Thu, Jul 19, 2012 at 9:58 PM, Matthew Brett >>> wrote: >>>> >>>> Hi, >>>> >>>> On Thu, Jul 19, 2012 at 8:29 AM, Ralf Gommers >>>> wrote: >>>>> >>>>> >>>>> On Wed, Jul 18, 2012 at 9:49 PM, Matthew Brett >>>>> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> Please forgive the half-baked question, but, I'm trying to build our >>>>>> package nipy (nipy.org) against stock downloaded >>>>>> >>>>>> numpy-1.6.2-py2.7-python.org-macosx10.6.dmg >>>>>> scipy-0.10.1-py2.7-python.org-macosx10.6.dmg >>>>> >>>>> >>>>> You're only building against numpy, and scipy is a runtime dependency, >>>>> right? >>>> >>>> Right - thanks for the clarification. >>>> >>>>>> >>>>>> I'm using latest XCode 4.3.3 >>>>>> >>>>>> $ clang --version >>>>>> Apple clang version 3.1 (tags/Apple/clang-318.0.61) (based on LLVM >>>>>> 3.1svn) >>>>>> Target: x86_64-apple-darwin11.4.0 >>>>>> Thread model: posix >>>>>> >>>>>> on Lion: >>>>>> >>>>>> $ uname -a >>>>>> Darwin pis-macbook-air 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr 9 >>>>>> 19:33:05 PDT 2012; root:xnu-1699.26.8~1/RELEASE_I386 i386 >>>>>> >>>>>> I believe the correct incantation for the build is: >>>>>> >>>>>> http://scipy.org/Installing_SciPy/Mac_OS_X >>>>>> >>>>>> CC=clang CXX=clang FFLAGS=-ff2c python setup.py build_ext -i >>>>> >>>>> >>>>> It has been reported to work for scipy, that's as much as I know. You >>>>> can >>>>> try if building scipy in the same way works for you. If it doesn't, the >>>>> problem may not be in nipy. >>>> >>>> Having linked the http://r.research.att.com/tools/ gfortran-4.2 to >>>> gfortran, this scipy build: >>>> >>>> CC=clang FFLAGS=-ff2c python setup.py build_ext -i >>>> >>>> builds and passes all tests. >>>> >>>> Building and linking nipy with the same flags, and with this scipy on >>>> the path, passes all tests. >>>> >>>> Take this scipy off the path (reverting to the system scipy as >>>> downloaded) gives another segfault in the same routine as before, but >>>> a little later: >>>> >>>> Thread 0 Crashed:: Dispatch queue: com.apple.main-thread >>>> 0 _segmentation.so 0x00000001035eef14 _get_message_mf >>>> + 20 (mrf.c:59) >>>> 1 _segmentation.so 0x00000001035eebee ve_step + 798 >>>> (mrf.c:172) >>>> 2 _segmentation.so 0x00000001035ed4dc >>>> __pyx_pf_4nipy_10algorithms_12segmentation_13_segmentation__ve_step + >>>> 4604 (_segmentation.c:1103) >>>> 3 org.python.python 0x00000001000c1e7d >>>> PyEval_EvalFrameEx + 25213 >>>> >>>> >>>>>> This does build, but during the tests, and only on this platform, >>>>>> segfaults (report attached). >>>>>> >>>>>> Is this build / link expected to work? Did I get the right flags? >>>>> >>>>> >>>>> Not sure. Some things to check: that you have the 64-bit python from >>>>> python.org, that MACOSX_DEPLOYMENT_TARGET is set to 10.6 during the >>>>> build. >>>> >>>> The MACOSX_DEPLOYMENT_TARGET did not seem to make a difference. >>>> >>>>> The by far most reliable way to make everything work together is to use >>>>> the >>>>> exact same compiler. Which is not available anymore with the most recent >>>>> XCode. You could look into ways to get gcc-4.2 to install on 10.7. And >>>>> make >>>>> sure to build your binaries on 10.6 for 64-bit 2.7 and on 10.5 for the >>>>> other >>>>> Python 2.x flavors if you can. >>>> >>>> Hum. I had previously tried the R gcc-4.2 compiler (thanks Scott for >>>> the link) and run into: >>>> >>>> gcc-4.2: /Users/mb312/dev_trees/nipy/libcstat/wrapper/fffpy.c >>>> In file included from >>>> >>>> /Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/unicodeobject.h:4, >>>> from >>>> >>>> /Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/Python.h:85, >>>> from >>>> /Users/mb312/dev_trees/nipy/libcstat/wrapper/fffpy.h:1, >>>> from >>>> /Users/mb312/dev_trees/nipy/libcstat/wrapper/fffpy.c:1: >>>> /Developer/SDKs/MacOSX10.6.sdk/usr/include/stdarg.h:4:25: error: >>>> stdarg.h: No such file or directory >>>> >>>> Anything obvious that I missed? >>> >>> >>> This is typically caused by not having the required SDKs installed, they're >>> under "Optional Installs" on your XCode dvd. >> >> I believe the only way to install Xcode for Lion 10.7.4 at least - is >> via the App Store. I can't see any options for other SDKs in the App >> Store or on the Xcode components menu. Do you know, has anyone got a >> gcc-4.2 build working with Lion 10.7.4? >> >> See you, >> >> Matthew > > I'm not sure if it will fix the issue you're having, but you could try downloading the command line tools from > https://developer.apple.com/downloads/ > You will need an Apple ID to download the disk image. Enter the word "command" in the search field on the Downloads page to narrow down the list, and pick the command line tools for the version of XCode you have. The installer will install the standard headers and libs for making programs from the command line. Just to update: I did try downloading the command line tools to fix the stdarg.h error with gcc-4.2, but it did not help. I'm guessing that the command line tools download is a subset of the full XCode install, but I haven't investigated further. A new version of the nipy code with edits in the relevant C routines, does not segfault; it is possible that we were picking up a nipy error that for some reason was only showing up on OSX Lion with clang... Cheers, Matthew From ralf.gommers at googlemail.com Sat Jul 21 15:57:34 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 21 Jul 2012 21:57:34 +0200 Subject: [SciPy-Dev] DST I precision (was ANN: SciPy 0.11.0 release candidate 1) Message-ID: On Sat, Jul 21, 2012 at 6:06 PM, John Hassler wrote: > > On 7/21/2012 10:43 AM, Ralf Gommers wrote: > > > On Fri, Jul 20, 2012 at 9:57 PM, John Hassler wrote: > > ====================================================================== >> FAIL: test_definition (test_real_transforms.TestDSTIDouble) >> >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File >> "C:\Python32\lib\site-packages\scipy\fftpack\tests\test_real_transforms.py", >> line 213, in test_definition >> err_msg="Size %d failed" % i) >> File "C:\Python32\lib\site-packages\numpy\testing\utils.py", line 800, >> in assert_array_almost_equal >> header=('Arrays are not almost equal to %d decimals' % decimal)) >> File "C:\Python32\lib\site-packages\numpy\testing\utils.py", line 636, >> in assert_array_compare >> raise AssertionError(msg) >> AssertionError: >> Arrays are not almost equal to 15 decimals >> Size 256 failed >> (mismatch 3.515625%) >> x: array([ 1.00000000e+00, -5.03902743e-01, 3.33300126e-01, >> -2.51913719e-01, 1.99940224e-01, -1.67900640e-01, >> 1.42771743e-01, -1.25881543e-01, 1.11000399e-01,... >> y: array([ 1.00000000e+00, -5.03902743e-01, 3.33300126e-01, >> -2.51913719e-01, 1.99940224e-01, -1.67900640e-01, >> 1.42771743e-01, -1.25881543e-01, 1.11000399e-01,... >> > > This looks like the test precision being a little too high. Could you > adjust it to decimal=14 or lower, and tell us when the test passes on your > machine? > > Thanks, > Ralf > > > decimal = 14 passes. > john > This test and the failure look a bit odd, so I just want to make sure: is it OK to lower the test precision? DST II passes with decimal=15 and DST III with decimal=16, so not sure why DST I is worse. Looks like this test could use some improvement: # XXX: we divide by np.max(y) because the tests fail otherwise. We # should really use something like assert_array_approx_equal. The # difference is due to fftw using a better algorithm w.r.t error # propagation compared to the ones from fftpack. assert_array_almost_equal(y / np.max(y), yr / np.max(y), decimal=self.dec, err_msg="Size %d failed" % i) Any takers? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From cristeab at gmail.com Sat Jul 21 17:33:21 2012 From: cristeab at gmail.com (Bogdan Cristea) Date: Sat, 21 Jul 2012 23:33:21 +0200 Subject: [SciPy-Dev] Algorithm for butterworth filter implementation Message-ID: <6521901.HqHg5kiBu9@desktop.cristea.fr> Hi Could someone provide me with some hints about the algorithm used to compute the coefficients for the butterworth filter found in scipy ? I am interested in the passband version of this filter since concatenating a low pass and a high pass filter does not produce the same frequency response as the passband filter provided by scipy. thanks -- Bogdan From sturla at molden.no Sun Jul 22 08:34:27 2012 From: sturla at molden.no (Sturla Molden) Date: Sun, 22 Jul 2012 14:34:27 +0200 Subject: [SciPy-Dev] Algorithm for butterworth filter implementation In-Reply-To: <6521901.HqHg5kiBu9@desktop.cristea.fr> References: <6521901.HqHg5kiBu9@desktop.cristea.fr> Message-ID: <879529A4-5953-4F2D-A26E-19BDAA3F11BC@molden.no> You probably did the common mistake of assuming an N-order LP followed by an N-order HP produces an N-order BP. That is not an algorithmic issue. Sturla Den 21. juli 2012 kl. 23:33 skrev Bogdan Cristea : > Hi > > Could someone provide me with some hints about the algorithm used to compute > the coefficients for the butterworth filter found in scipy ? I am interested > in the passband version of this filter since concatenating a low pass and a > high pass filter does not produce the same frequency response as the passband > filter provided by scipy. > > thanks > -- > Bogdan > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From ralf.gommers at googlemail.com Mon Jul 23 04:37:43 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 23 Jul 2012 10:37:43 +0200 Subject: [SciPy-Dev] DST I precision (was ANN: SciPy 0.11.0 release candidate 1) In-Reply-To: References: Message-ID: On Sat, Jul 21, 2012 at 9:57 PM, Ralf Gommers wrote: > > > On Sat, Jul 21, 2012 at 6:06 PM, John Hassler wrote: > >> >> On 7/21/2012 10:43 AM, Ralf Gommers wrote: >> >> >> On Fri, Jul 20, 2012 at 9:57 PM, John Hassler wrote: >> >> ====================================================================== >>> FAIL: test_definition (test_real_transforms.TestDSTIDouble) >>> >>> ---------------------------------------------------------------------- >>> Traceback (most recent call last): >>> File >>> "C:\Python32\lib\site-packages\scipy\fftpack\tests\test_real_transforms.py", >>> line 213, in test_definition >>> err_msg="Size %d failed" % i) >>> File "C:\Python32\lib\site-packages\numpy\testing\utils.py", line 800, >>> in assert_array_almost_equal >>> header=('Arrays are not almost equal to %d decimals' % decimal)) >>> File "C:\Python32\lib\site-packages\numpy\testing\utils.py", line 636, >>> in assert_array_compare >>> raise AssertionError(msg) >>> AssertionError: >>> Arrays are not almost equal to 15 decimals >>> Size 256 failed >>> (mismatch 3.515625%) >>> x: array([ 1.00000000e+00, -5.03902743e-01, 3.33300126e-01, >>> -2.51913719e-01, 1.99940224e-01, -1.67900640e-01, >>> 1.42771743e-01, -1.25881543e-01, 1.11000399e-01,... >>> y: array([ 1.00000000e+00, -5.03902743e-01, 3.33300126e-01, >>> -2.51913719e-01, 1.99940224e-01, -1.67900640e-01, >>> 1.42771743e-01, -1.25881543e-01, 1.11000399e-01,... >>> >> >> This looks like the test precision being a little too high. Could you >> adjust it to decimal=14 or lower, and tell us when the test passes on your >> machine? >> >> Thanks, >> Ralf >> >> >> decimal = 14 passes. >> john >> > > This test and the failure look a bit odd, so I just want to make sure: is > it OK to lower the test precision? DST II passes with decimal=15 and DST > III with decimal=16, so not sure why DST I is worse. Looks like this test > could use some improvement: > > # XXX: we divide by np.max(y) because the tests fail > otherwise. We > # should really use something like assert_array_approx_equal. > The > # difference is due to fftw using a better algorithm w.r.t > error > # propagation compared to the ones from fftpack. > assert_array_almost_equal(y / np.max(y), yr / np.max(y), > decimal=self.dec, > err_msg="Size %d failed" % i) > > Actually I get a failure myself for DSTIIIDouble, only with Python 2.4. It seems that the DST test tolerances are set much higher than the DCT ones. Lowering them all to decimal=14 would make sense to avoid further failures. Any objections? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at hilboll.de Mon Jul 23 08:41:14 2012 From: lists at hilboll.de (Andreas Hilboll) Date: Mon, 23 Jul 2012 14:41:14 +0200 Subject: [SciPy-Dev] ANN: SciPy 0.11.0 release candidate 1 In-Reply-To: <5009ACC3.5090701@iki.fi> References: <3fb4522421d3b87335b5d040a0e6cd4b.squirrel@srv2.s4y.tournesol-consulting.eu> <5009ACC3.5090701@iki.fi> Message-ID: <477862d6d96a2d4b41ad524711d02fe9.squirrel@srv2.s4y.tournesol-consulting.eu> > 20.07.2012 11:34, Andreas Hilboll kirjoitti: > [clip] >> ====================================================================== >> FAIL: test_basic.TestNorm.test_stable >> ---------------------------------------------------------------------- > [clip] > > Broken BLAS or something? Try this fortran program: > > > !!! test.f90 > program main > implicit none > real a(10001) > real b, snrm2 > external snrm2 > > a(1) = 1e4 > a(2:10001) = 1.0 > > b = snrm2(10001, a, 1) > write(*,*) b ! should give 10000.5 > end program main > > % gfortran -lblas test_blas.f90 % ./a.out 10000.0000 So apparently, something's wrong. But what? Cheers, A. From mello at elipse.com.br Mon Jul 23 12:49:28 2012 From: mello at elipse.com.br (=?iso-8859-1?Q?Mello=2C_C=E9sar_Menin_de_Mello_=28Elipse_Software=29?=) Date: Mon, 23 Jul 2012 13:49:28 -0300 Subject: [SciPy-Dev] MSI Message-ID: <45A24C2DBFC7D04A98990169453563D54FDE6E4815@madrid.elipse.com.br> Hi! Please I would like to know if there are any plans to release an "official" MSI redistributable for numpy for Windows (like the official Python installer for Windows). This would make it easier to embed Python+numpy in other installers. Thank you a lot for the attention! Best regards Mello -------------- next part -------------- An HTML attachment was scrubbed... URL: From cristeab at gmail.com Mon Jul 23 12:54:50 2012 From: cristeab at gmail.com (Bogdan Cristea) Date: Mon, 23 Jul 2012 18:54:50 +0200 Subject: [SciPy-Dev] MSI In-Reply-To: <45A24C2DBFC7D04A98990169453563D54FDE6E4815@madrid.elipse.com.br> References: <45A24C2DBFC7D04A98990169453563D54FDE6E4815@madrid.elipse.com.br> Message-ID: <5767254.KS8iN3BRUl@desktop.cristea.fr> On Monday 23 July 2012 13:49:28 Mello, C?sar Menin de Mello wrote: > Please I would like to know if there are any plans to release an "official" > MSI redistributable for numpy for Windows (like the official Python > installer for Windows). This would make it easier to embed Python+numpy in > other installers. Have a look at Enthough Python Distribution -- Bogdan From ruediger.kessel at gmail.com Tue Jul 24 09:43:56 2012 From: ruediger.kessel at gmail.com (=?utf-8?b?UsO8ZGlnZXI=?= Kessel) Date: Tue, 24 Jul 2012 13:43:56 +0000 (UTC) Subject: [SciPy-Dev] ANN: SciPy 0.11.0 release candidate 1 References: <3fb4522421d3b87335b5d040a0e6cd4b.squirrel@srv2.s4y.tournesol-consulting.eu> <5009ACC3.5090701@iki.fi> <477862d6d96a2d4b41ad524711d02fe9.squirrel@srv2.s4y.tournesol-consulting.eu> Message-ID: Andreas Hilboll hilboll.de> writes: > > > 20.07.2012 11:34, Andreas Hilboll kirjoitti: > > [clip] > >> ====================================================================== > >> FAIL: test_basic.TestNorm.test_stable > >> ---------------------------------------------------------------------- > > [clip] > > > > Broken BLAS or something? Try this fortran program: > > > > > > !!! test.f90 > > program main > > implicit none > > real a(10001) > > real b, snrm2 > > external snrm2 > > > > a(1) = 1e4 > > a(2:10001) = 1.0 > > > > b = snrm2(10001, a, 1) > > write(*,*) b ! should give 10000.5 > > end program main > > > > > > % gfortran -lblas test_blas.f90 > % ./a.out > 10000.0000 > > So apparently, something's wrong. But what? > > Cheers, A. > Hi, I think BLAS is right. I implemented the core of the original BLAS routine (http://www.netlib.org/blas/snrm2.f) in python: def snrm2(X,INCX=1): import math from numpy import float32 N=len(X) SCALE = float32(0.0) SSQ = float32(0.0) for ix in range(0,N,INCX): if X[ix]!=0.0: ABSXI = float32(abs(X[ix])) if SCALE < ABSXI: SSQ = float32(1.0) + SSQ * float32(float32(SCALE/ABSXI)**2) SCALE = ABSXI else: SSQ = SSQ + float32(float32(ABSXI/SCALE)**2) NORM = SCALE*float32(math.sqrt(SSQ)) return NORM It shows the same behavior. The reason is that the ratio between 10000**2 / 1**2 is 1e8. So the loop is adding a small value to a large value in single precision. float32(1e8+1) = 1e8 This can only be improved if snrm2 would internally use double precision. But this was obviously not the intention. Greetings R?diger From ruediger.kessel at gmail.com Tue Jul 24 12:07:04 2012 From: ruediger.kessel at gmail.com (=?utf-8?b?UsO8ZGlnZXI=?= Kessel) Date: Tue, 24 Jul 2012 16:07:04 +0000 (UTC) Subject: [SciPy-Dev] ANN: SciPy 0.11.0 release candidate 1 References: <3fb4522421d3b87335b5d040a0e6cd4b.squirrel@srv2.s4y.tournesol-consulting.eu> <5009ACC3.5090701@iki.fi> <477862d6d96a2d4b41ad524711d02fe9.squirrel@srv2.s4y.tournesol-consulting.eu> Message-ID: R?diger Kessel gmail.com> writes: PS: With test_stable() one can tell implementations apart. If snrm2() is implemented completely in single precision then the result of norm(a) will be 10000.0. If internally a double precision implementation is used but the result is rounded to single precision then the result of norm(a) is 10000.5. Both implementations are possible and probably present in different libs. Therefore the test should allow for both results. It might produce some printed comment about the implementation. From ralf.gommers at googlemail.com Tue Jul 24 15:57:18 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 24 Jul 2012 21:57:18 +0200 Subject: [SciPy-Dev] MSI In-Reply-To: <45A24C2DBFC7D04A98990169453563D54FDE6E4815@madrid.elipse.com.br> References: <45A24C2DBFC7D04A98990169453563D54FDE6E4815@madrid.elipse.com.br> Message-ID: On Mon, Jul 23, 2012 at 6:49 PM, Mello, C?sar Menin de Mello (Elipse Software) wrote: > Hi!**** > > ** ** > > Please I would like to know if there are any plans to release an > "official" MSI redistributable for numpy for Windows (like the official > Python installer for Windows). This would make it easier to embed > Python+numpy in other installers. > Not that I'm aware of. Would be a nice-to-have though. Ralf > **** > > ** ** > > Thank you a lot for the attention!**** > > ** ** > > Best regards**** > > Mello**** > > ** ** > > ** ** > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt.terry at gmail.com Tue Jul 24 16:09:31 2012 From: matt.terry at gmail.com (Matt Terry) Date: Tue, 24 Jul 2012 13:09:31 -0700 Subject: [SciPy-Dev] DST I precision (was ANN: SciPy 0.11.0 release candidate 1) In-Reply-To: References: Message-ID: Nope. The tolerances are based on what worked on my macbook, but I didn't have machines to do further testing. The correct values are from fftw. Compared to fftw the algorithms in fftpack are not as accurate. That and i would be very careful about believing the 15th decimal of anything. -matt On Mon, Jul 23, 2012 at 1:37 AM, Ralf Gommers wrote: > > > On Sat, Jul 21, 2012 at 9:57 PM, Ralf Gommers > wrote: >> >> >> >> On Sat, Jul 21, 2012 at 6:06 PM, John Hassler >> wrote: >>> >>> >>> On 7/21/2012 10:43 AM, Ralf Gommers wrote: >>> >>> >>> On Fri, Jul 20, 2012 at 9:57 PM, John Hassler >>> wrote: >>> >>>> ====================================================================== >>>> FAIL: test_definition (test_real_transforms.TestDSTIDouble) >>>> >>>> ---------------------------------------------------------------------- >>>> Traceback (most recent call last): >>>> File >>>> "C:\Python32\lib\site-packages\scipy\fftpack\tests\test_real_transforms.py", >>>> line 213, in test_definition >>>> err_msg="Size %d failed" % i) >>>> File "C:\Python32\lib\site-packages\numpy\testing\utils.py", line 800, >>>> in assert_array_almost_equal >>>> header=('Arrays are not almost equal to %d decimals' % decimal)) >>>> File "C:\Python32\lib\site-packages\numpy\testing\utils.py", line 636, >>>> in assert_array_compare >>>> raise AssertionError(msg) >>>> AssertionError: >>>> Arrays are not almost equal to 15 decimals >>>> Size 256 failed >>>> (mismatch 3.515625%) >>>> x: array([ 1.00000000e+00, -5.03902743e-01, 3.33300126e-01, >>>> -2.51913719e-01, 1.99940224e-01, -1.67900640e-01, >>>> 1.42771743e-01, -1.25881543e-01, 1.11000399e-01,... >>>> y: array([ 1.00000000e+00, -5.03902743e-01, 3.33300126e-01, >>>> -2.51913719e-01, 1.99940224e-01, -1.67900640e-01, >>>> 1.42771743e-01, -1.25881543e-01, 1.11000399e-01,... >>> >>> >>> This looks like the test precision being a little too high. Could you >>> adjust it to decimal=14 or lower, and tell us when the test passes on your >>> machine? >>> >>> Thanks, >>> Ralf >>> >>> >>> decimal = 14 passes. >>> john >> >> >> This test and the failure look a bit odd, so I just want to make sure: is >> it OK to lower the test precision? DST II passes with decimal=15 and DST III >> with decimal=16, so not sure why DST I is worse. Looks like this test could >> use some improvement: >> >> # XXX: we divide by np.max(y) because the tests fail >> otherwise. We >> # should really use something like assert_array_approx_equal. >> The >> # difference is due to fftw using a better algorithm w.r.t >> error >> # propagation compared to the ones from fftpack. >> assert_array_almost_equal(y / np.max(y), yr / np.max(y), >> decimal=self.dec, >> err_msg="Size %d failed" % i) >> > > Actually I get a failure myself for DSTIIIDouble, only with Python 2.4. It > seems that the DST test tolerances are set much higher than the DCT ones. > Lowering them all to decimal=14 would make sense to avoid further failures. > Any objections? > > Ralf > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > From kyle.mandli at gmail.com Wed Jul 25 12:57:22 2012 From: kyle.mandli at gmail.com (Kyle Mandli) Date: Wed, 25 Jul 2012 11:57:22 -0500 Subject: [SciPy-Dev] SciPy.org Website Move and Redesign Message-ID: At the scipy conference this year there were a number of people who expressed interest in moving the scipy.org website off of the current server and migrate away from using MoinMoinWiki. It turns out there have been a number of attempts at this over the past couple of years but nothing has ever stuck (as can be seen in the github org page). After discussing a number of approaches it sounds like the best strategy is one similar to what IPython and many other packages are using, namely using github pages and sphinx. The plan would be: 1) Mailing lists - Leave these as there's a lot of content there and we are worried about losing history and users if we even attempted to move this (plus we don't have a good alternative). 2) Scipy.org (and numpy.org) - Move the content to a repository under the Scipy organization on github that would contain a sphinx representation of the content that people can make pull requests to. There would be another repository that the built content would then be pushed to and that github would show as the website. David Warde-Farley has made an attempt at converting many of the scipy pages over to sphinx but it was never used (https://github.com/dwf/scipy-website). I noticed there were a few attempts at this a while ago (> 2 years). 3) Trac Projects - Numpy is in the process (or so we thought) of moving all the trac issues to github issues and as far as we are aware represents really the last reason these trac projects are still being used. Once this is done we can dispense with these as well. The scipy trac pages have not been moved at all and would also have to be done. These represent some effort but the website could make the transition before this is complete anyway. 4) Anything else? I am hoping that we can move the content quickly and set up the system first to get it off the ground and update the pages as we can. Hopefully we can get something going quickly and avoid the fates of previous attempts at this. Kyle From ralf.gommers at googlemail.com Wed Jul 25 13:56:40 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 25 Jul 2012 19:56:40 +0200 Subject: [SciPy-Dev] DST I precision (was ANN: SciPy 0.11.0 release candidate 1) In-Reply-To: References: Message-ID: On Tue, Jul 24, 2012 at 10:09 PM, Matt Terry wrote: > Nope. The tolerances are based on what worked on my macbook, but I > didn't have machines to do further testing. > Thanks Matt. Changed in master now. Ralf > The correct values are from fftw. Compared to fftw the algorithms in > fftpack are not as accurate. That and i would be very careful about > believing the 15th decimal of anything. > > -matt > > On Mon, Jul 23, 2012 at 1:37 AM, Ralf Gommers > wrote: > > > > > > On Sat, Jul 21, 2012 at 9:57 PM, Ralf Gommers < > ralf.gommers at googlemail.com> > > wrote: > >> > >> > >> > >> On Sat, Jul 21, 2012 at 6:06 PM, John Hassler > >> wrote: > >>> > >>> > >>> On 7/21/2012 10:43 AM, Ralf Gommers wrote: > >>> > >>> > >>> On Fri, Jul 20, 2012 at 9:57 PM, John Hassler > >>> wrote: > >>> > >>>> ====================================================================== > >>>> FAIL: test_definition (test_real_transforms.TestDSTIDouble) > >>>> > >>>> ---------------------------------------------------------------------- > >>>> Traceback (most recent call last): > >>>> File > >>>> > "C:\Python32\lib\site-packages\scipy\fftpack\tests\test_real_transforms.py", > >>>> line 213, in test_definition > >>>> err_msg="Size %d failed" % i) > >>>> File "C:\Python32\lib\site-packages\numpy\testing\utils.py", line > 800, > >>>> in assert_array_almost_equal > >>>> header=('Arrays are not almost equal to %d decimals' % decimal)) > >>>> File "C:\Python32\lib\site-packages\numpy\testing\utils.py", line > 636, > >>>> in assert_array_compare > >>>> raise AssertionError(msg) > >>>> AssertionError: > >>>> Arrays are not almost equal to 15 decimals > >>>> Size 256 failed > >>>> (mismatch 3.515625%) > >>>> x: array([ 1.00000000e+00, -5.03902743e-01, 3.33300126e-01, > >>>> -2.51913719e-01, 1.99940224e-01, -1.67900640e-01, > >>>> 1.42771743e-01, -1.25881543e-01, 1.11000399e-01,... > >>>> y: array([ 1.00000000e+00, -5.03902743e-01, 3.33300126e-01, > >>>> -2.51913719e-01, 1.99940224e-01, -1.67900640e-01, > >>>> 1.42771743e-01, -1.25881543e-01, 1.11000399e-01,... > >>> > >>> > >>> This looks like the test precision being a little too high. Could you > >>> adjust it to decimal=14 or lower, and tell us when the test passes on > your > >>> machine? > >>> > >>> Thanks, > >>> Ralf > >>> > >>> > >>> decimal = 14 passes. > >>> john > >> > >> > >> This test and the failure look a bit odd, so I just want to make sure: > is > >> it OK to lower the test precision? DST II passes with decimal=15 and > DST III > >> with decimal=16, so not sure why DST I is worse. Looks like this test > could > >> use some improvement: > >> > >> # XXX: we divide by np.max(y) because the tests fail > >> otherwise. We > >> # should really use something like > assert_array_approx_equal. > >> The > >> # difference is due to fftw using a better algorithm w.r.t > >> error > >> # propagation compared to the ones from fftpack. > >> assert_array_almost_equal(y / np.max(y), yr / np.max(y), > >> decimal=self.dec, > >> err_msg="Size %d failed" % i) > >> > > > > Actually I get a failure myself for DSTIIIDouble, only with Python 2.4. > It > > seems that the DST test tolerances are set much higher than the DCT ones. > > Lowering them all to decimal=14 would make sense to avoid further > failures. > > Any objections? > > > > Ralf > > > > > > > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-dev > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Wed Jul 25 17:59:29 2012 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 25 Jul 2012 21:59:29 +0000 (UTC) Subject: [SciPy-Dev] SciPy.org Website Move and Redesign References: Message-ID: Kyle Mandli gmail.com> writes: [clip] > 1) Mailing lists - Leave these as there's a lot of content there > and we are worried about losing history and users if we even > attempted to move this (plus we don't have a good alternative). Mailing lists are going to be needed. They don't necessarily need to be hosted on the current scipy.org machine, but it's probably best not to touch what has been working reliably. > 2) Scipy.org (and numpy.org) - Move the content to a repository under the > Scipy organization on github that would contain a sphinx representation > of the content that people can make pull requests to. There would be > another repository that the built content would then be pushed to and > that github would show as the website. David Warde-Farley has made > an attempt at converting many of the scipy pages over to sphinx but it > was never used (https://github.com/dwf/scipy-website). I noticed there > were a few attempts at this a while ago (> 2 years). Note that the scipy.github.com site is up with a Git repo set up at https://github.com/scipy/scipy.org-new/ Similarly, numpy.scipy.org site has already moved to Github. I would see these repos as the central point for any further work. David's work is not merged there, but it could be merged (where applicable --- some of the content in scipy.org is outdated). > 3) Trac Projects - Numpy is in the process (or so we thought) [clip] I think the trac projects are a separate issue --- web site redesign can either precede or come after what is done with Trac. IMO, one of the first steps to do is to design a website layout skeleton that can be used appropriately on scipy.org, and daughter sites, docs.scipy.org, scipy-central.org, etc. This does not have to be in Sphinx format at first, just plain HTML+whatever, it's easy to hack in later. It also doesn't have to be great, but it does have to exist, and one has to think a bit about how and where to put the navigation tools. The amount of relevant "official" content on scipy.org is actually not very big, I think. The main issue is more on what to do with the Cookbook/Topical software sections, which in my opinion are not very well-suited to a Sphinx-generated static site. Why this hasn't moved forwards is probably largely due to not reaching a decision on what actually to do with this material. The current best bet as a replacement (apart from keeping these in Moin) would probably be scipy-central.org --- which I'm sure would also benefit greatly from additional love. Enthusiastic Django-capable people might be interested to take a look at improving it. -- Pauli Virtanen From charlesr.harris at gmail.com Wed Jul 25 20:36:48 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 25 Jul 2012 18:36:48 -0600 Subject: [SciPy-Dev] SciPy.org Website Move and Redesign In-Reply-To: References: Message-ID: On Wed, Jul 25, 2012 at 3:59 PM, Pauli Virtanen wrote: > Kyle Mandli gmail.com> writes: > [clip] > > 1) Mailing lists - Leave these as there's a lot of content there > > and we are worried about losing history and users if we even > > attempted to move this (plus we don't have a good alternative). > > Mailing lists are going to be needed. They don't necessarily need to > be hosted on the current scipy.org machine, but it's probably best not to > touch what has been working reliably. > > > 2) Scipy.org (and numpy.org) - Move the content to a repository under > the > > Scipy organization on github that would contain a sphinx representation > > of the content that people can make pull requests to. There would be > > another repository that the built content would then be pushed to and > > that github would show as the website. David Warde-Farley has made > > an attempt at converting many of the scipy pages over to sphinx but it > > was never used (https://github.com/dwf/scipy-website). I noticed there > > were a few attempts at this a while ago (> 2 years). > > Note that the scipy.github.com site is up with a Git repo set up at > https://github.com/scipy/scipy.org-new/ > Similarly, numpy.scipy.org site has already moved to Github. > > I think the www.scipy.org site looks a lot nicer than the replacement, which still looks unfinished. > I would see these repos as the central point for any further work. > David's work is not merged there, but it could be merged (where > applicable --- some of the content in scipy.org is outdated). > > > 3) Trac Projects - Numpy is in the process (or so we thought) > [clip] > > I think the trac projects are a separate issue --- web site redesign > can either precede or come after what is done with Trac. > > IMO, one of the first steps to do is to design a website layout > skeleton that can be used appropriately on scipy.org, and daughter > sites, docs.scipy.org, scipy-central.org, etc. This does not have > to be in Sphinx format at first, just plain HTML+whatever, it's easy > to hack in later. It also doesn't have to be great, but it does have > to exist, and one has to think a bit about how and where to put the > navigation tools. > > The amount of relevant "official" content on scipy.org is actually > not very big, I think. > > The main issue is more on what to do with the Cookbook/Topical software > sections, which in my opinion are not very well-suited to a > Sphinx-generated static site. Why this hasn't moved forwards is > probably largely due to not reaching a decision on what actually > to do with this material. > > The current best bet as a replacement (apart from keeping these in Moin) > would probably be scipy-central.org --- which I'm sure would also benefit > greatly from additional love. Enthusiastic Django-capable people might be > interested to take a look at improving it. > > Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From scopatz at gmail.com Wed Jul 25 21:57:24 2012 From: scopatz at gmail.com (Anthony Scopatz) Date: Wed, 25 Jul 2012 20:57:24 -0500 Subject: [SciPy-Dev] SciPy.org Website Move and Redesign In-Reply-To: References: Message-ID: On Wed, Jul 25, 2012 at 7:36 PM, Charles R Harris wrote: > > > On Wed, Jul 25, 2012 at 3:59 PM, Pauli Virtanen wrote: > >> Kyle Mandli gmail.com> writes: >> [clip] >> > 1) Mailing lists - Leave these as there's a lot of content there >> > and we are worried about losing history and users if we even >> > attempted to move this (plus we don't have a good alternative). >> >> Mailing lists are going to be needed. They don't necessarily need to >> be hosted on the current scipy.org machine, but it's probably best not to >> touch what has been working reliably. >> >> > 2) Scipy.org (and numpy.org) - Move the content to a repository under >> the >> > Scipy organization on github that would contain a sphinx representation >> > of the content that people can make pull requests to. There would be >> > another repository that the built content would then be pushed to and >> > that github would show as the website. David Warde-Farley has made >> > an attempt at converting many of the scipy pages over to sphinx but it >> > was never used (https://github.com/dwf/scipy-website). I noticed there >> > were a few attempts at this a while ago (> 2 years). >> >> Note that the scipy.github.com site is up with a Git repo set up at >> https://github.com/scipy/scipy.org-new/ >> Similarly, numpy.scipy.org site has already moved to Github. >> >> > I think the www.scipy.org site looks a lot nicer than the replacement, > which still looks unfinished. > I disagree and am in favor of the replacement though I probably don't have the spare cycles to contribute in any meaningful way. ~A. > > >> I would see these repos as the central point for any further work. >> David's work is not merged there, but it could be merged (where >> applicable --- some of the content in scipy.org is outdated). >> >> > 3) Trac Projects - Numpy is in the process (or so we thought) >> [clip] >> >> I think the trac projects are a separate issue --- web site redesign >> can either precede or come after what is done with Trac. >> >> IMO, one of the first steps to do is to design a website layout >> skeleton that can be used appropriately on scipy.org, and daughter >> sites, docs.scipy.org, scipy-central.org, etc. This does not have >> to be in Sphinx format at first, just plain HTML+whatever, it's easy >> to hack in later. It also doesn't have to be great, but it does have >> to exist, and one has to think a bit about how and where to put the >> navigation tools. >> >> The amount of relevant "official" content on scipy.org is actually >> not very big, I think. >> >> The main issue is more on what to do with the Cookbook/Topical software >> sections, which in my opinion are not very well-suited to a >> Sphinx-generated static site. Why this hasn't moved forwards is >> probably largely due to not reaching a decision on what actually >> to do with this material. >> >> The current best bet as a replacement (apart from keeping these in Moin) >> would probably be scipy-central.org --- which I'm sure would also benefit >> greatly from additional love. Enthusiastic Django-capable people might be >> interested to take a look at improving it. >> >> > Chuck > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Wed Jul 25 22:27:00 2012 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 25 Jul 2012 19:27:00 -0700 Subject: [SciPy-Dev] SciPy.org Website Move and Redesign In-Reply-To: References: Message-ID: On Wed, Jul 25, 2012 at 6:57 PM, Anthony Scopatz wrote: >> >> I think the www.scipy.org site looks a lot nicer than the replacement, >> which still looks unfinished. > > > I disagree and am in favor of the replacement though I probably don't have > the spare cycles to contribute in any meaningful way. I think the point that's more important here is that the new approach is *maintainable* moving forward and makes it trivial to delegate website work to a separate team of contributors. The moin-based setup is more or less impossible to update in any meaningful way, so for all intents and purposes it's a complete dead-end. Cheers, f From pav at iki.fi Thu Jul 26 17:59:42 2012 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 26 Jul 2012 21:59:42 +0000 (UTC) Subject: [SciPy-Dev] SciPy.org Website Move and Redesign References: Message-ID: Charles R Harris gmail.com> writes: [clip] > I think the www.scipy.org site looks a lot nicer than > the replacement, which still looks unfinished.? Well, it *is* unfinished, and the point in this thread was to make it finished. The current state is not very relevant for this discussion. Pauli From charlesr.harris at gmail.com Thu Jul 26 18:21:58 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 26 Jul 2012 16:21:58 -0600 Subject: [SciPy-Dev] SciPy.org Website Move and Redesign In-Reply-To: References: Message-ID: On Thu, Jul 26, 2012 at 3:59 PM, Pauli Virtanen wrote: > Charles R Harris gmail.com> writes: > [clip] > > I think the www.scipy.org site looks a lot nicer than > > the replacement, which still looks unfinished. > > Well, it *is* unfinished, and the point in this thread was to make > it finished. The current state is not very relevant for this discussion. > May I suggest making it look more like the www.scipy.org site then? I think that site looks pretty good. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at enthought.com Thu Jul 26 21:15:06 2012 From: eric at enthought.com (Eric Jones) Date: Thu, 26 Jul 2012 20:15:06 -0500 Subject: [SciPy-Dev] SciPy.org Website Move and Redesign In-Reply-To: References: Message-ID: Hey Kyle and Pauli, Ognen is out this week, but he will be helpful for how to best work with on server options. We'll be glad to help getting the scipy server set up as needed to host static/Django/whatever. I am, unfortunately, heading out on vacation early tomorrow morning and am off the grid through next week. I'll catch up with the conversation at that point. eric On Jul 25, 2012, at 4:59 PM, Pauli Virtanen wrote: > Kyle Mandli gmail.com> writes: > [clip] >> 1) Mailing lists - Leave these as there's a lot of content there >> and we are worried about losing history and users if we even >> attempted to move this (plus we don't have a good alternative). > > Mailing lists are going to be needed. They don't necessarily need to > be hosted on the current scipy.org machine, but it's probably best not to > touch what has been working reliably. > >> 2) Scipy.org (and numpy.org) - Move the content to a repository under the >> Scipy organization on github that would contain a sphinx representation >> of the content that people can make pull requests to. There would be >> another repository that the built content would then be pushed to and >> that github would show as the website. David Warde-Farley has made >> an attempt at converting many of the scipy pages over to sphinx but it >> was never used (https://github.com/dwf/scipy-website). I noticed there >> were a few attempts at this a while ago (> 2 years). > > Note that the scipy.github.com site is up with a Git repo set up at > https://github.com/scipy/scipy.org-new/ > Similarly, numpy.scipy.org site has already moved to Github. > > I would see these repos as the central point for any further work. > David's work is not merged there, but it could be merged (where > applicable --- some of the content in scipy.org is outdated). > >> 3) Trac Projects - Numpy is in the process (or so we thought) > [clip] > > I think the trac projects are a separate issue --- web site redesign > can either precede or come after what is done with Trac. > > IMO, one of the first steps to do is to design a website layout > skeleton that can be used appropriately on scipy.org, and daughter > sites, docs.scipy.org, scipy-central.org, etc. This does not have > to be in Sphinx format at first, just plain HTML+whatever, it's easy > to hack in later. It also doesn't have to be great, but it does have > to exist, and one has to think a bit about how and where to put the > navigation tools. > > The amount of relevant "official" content on scipy.org is actually > not very big, I think. > > The main issue is more on what to do with the Cookbook/Topical software > sections, which in my opinion are not very well-suited to a > Sphinx-generated static site. Why this hasn't moved forwards is > probably largely due to not reaching a decision on what actually > to do with this material. > > The current best bet as a replacement (apart from keeping these in Moin) > would probably be scipy-central.org --- which I'm sure would also benefit > greatly from additional love. Enthusiastic Django-capable people might be > interested to take a look at improving it. > > -- > Pauli Virtanen > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From ralf.gommers at googlemail.com Fri Jul 27 02:17:13 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 27 Jul 2012 08:17:13 +0200 Subject: [SciPy-Dev] SciPy.org Website Move and Redesign In-Reply-To: References: Message-ID: On Fri, Jul 27, 2012 at 12:21 AM, Charles R Harris < charlesr.harris at gmail.com> wrote: > > > On Thu, Jul 26, 2012 at 3:59 PM, Pauli Virtanen wrote: > >> Charles R Harris gmail.com> writes: >> [clip] >> > I think the www.scipy.org site looks a lot nicer than >> > the replacement, which still looks unfinished. >> >> Well, it *is* unfinished, and the point in this thread was to make >> it finished. The current state is not very relevant for this discussion. >> > > May I suggest making it look more like the www.scipy.org site then? I > think that site looks pretty good. > Well, I'd be interested to see what someone with some design talent could do. scipy.org doesn't look that good imho. And can we please lose the baby blue background? The days of hand-written html with plain colored backgrounds are over. A more interesting discussion is the content. scipy.org talks about the *project* scipy, while http://scipy.github.com/ is about the *ecosystem*. I'd prefer the front page to be about the ecosystem, and have a new page about the project. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From margaret_doll at brown.edu Fri Jul 27 13:48:55 2012 From: margaret_doll at brown.edu (Doll, Margaret Ann) Date: Fri, 27 Jul 2012 13:48:55 -0400 Subject: [SciPy-Dev] Problems building SciPy-0.11.0b1 on CentOS Message-ID: I have CentOS release 5.6 (Final) I have defined the variables ATLAS=/usr/local/lib/libsatlas.so:/usr/local/lib/libtatlas.so BLAS=/usr/lib64/libblas.so LAPACK= FLAPACK=/usr/lib/liblapack.so.3 Executed the line to build and install scipy-0.11.0b1]# python setup.py install This ends with File "/usr/lib64/python2.4/site-packages/numpy/distutils/system_info.py", line 321, in get_info return cl().get_info(notfound_action) File "/usr/lib64/python2.4/site-packages/numpy/distutils/system_info.py", line 472, in get_info raise self.notfounderror(self.notfounderror.__doc__) numpy.distutils.system_info.LapackNotFoundError: Lapack (http://www.netlib.org/lapack/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [lapack]) or by setting the LAPACK environment variable. I have also tried LAPACK=/usr/lib/liblapack.so.3 How do I fix this problem? Atlas was built with ./configure --shared --prefix=/share/apps --with-netlib-lapack-tarfile=/share/apps/lapack-3.4.1.tgz -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Fri Jul 27 14:35:48 2012 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 27 Jul 2012 11:35:48 -0700 Subject: [SciPy-Dev] SciPy.org Website Move and Redesign In-Reply-To: References: Message-ID: On Thu, Jul 26, 2012 at 11:17 PM, Ralf Gommers wrote: > Well, I'd be interested to see what someone with some design talent could > do. scipy.org doesn't look that good imho. And can we please lose the baby > blue background? The days of hand-written html with plain colored > backgrounds are over. > > A more interesting discussion is the content. scipy.org talks about the > *project* scipy, while http://scipy.github.com/ is about the *ecosystem*. > I'd prefer the front page to be about the ecosystem, and have a new page > about the project. And precisely having the site moved over to a system where: - it's trivial to delegate website work to a separate team who can collaborate with the regular github flow (branches, PRs, etc) to manage the web content. - form and content are separated, with templates and CSS for all formal display makes progress on both of those fronts possible. The ipython site isn't about to win any web design awards, and we deliberately kept it very minimalistic, but it shows that it's easy to make a sphinx site that looks very different from your typical sphinx doc build. With a bit more work, a talented web team could make it look even more different, while still using a toolchain (sphinx, reST and github) that we use for other things and therefore leveraging existing skills. Cheers, f From sturla at molden.no Mon Jul 30 10:10:47 2012 From: sturla at molden.no (Sturla Molden) Date: Mon, 30 Jul 2012 16:10:47 +0200 Subject: [SciPy-Dev] scipy.org is very slow Message-ID: <501695E7.6080108@molden.no> scipy.org takes forever to do anything. I think the server might need a restart. Sturla From sturla at molden.no Mon Jul 30 15:15:14 2012 From: sturla at molden.no (Sturla Molden) Date: Mon, 30 Jul 2012 21:15:14 +0200 Subject: [SciPy-Dev] SciPy bugtracker on GitHub (a la NumPy)? Message-ID: <5016DD42.5090404@molden.no> (This might have been discussed before, forgive me if I've missed it.) NumPy is using "Issues" on GitHub as bugtracker in addition to trac it seems. It has several advantages. Everything is kept in one place on GitHub. The interface is more tidy. And pull requests can be attached to "issues" (cf. tickets with attached .diff files on trac). In my experience the patches contributed on trac often get "forgotten". Someone has to take time to get them into git/svn. (Which quickly discouraged me from contributing anything.) I think this is the same reason Ralph Gommers complained when I mailed cKDTree code here instead of using git myself: "I know you may not have the time or interest to learn about git right now, but it may make both our lives easier if you try the below steps. It will allow you to put your commits on top of mine without any manual copying." I guess that applies to anything attached to a ticket on trac as well? When SciPy is on GitHub I think we should consider using GitHub's issue tracker. (And NumPy is already using it.) Sturla From ralf.gommers at googlemail.com Mon Jul 30 15:31:26 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 30 Jul 2012 21:31:26 +0200 Subject: [SciPy-Dev] SciPy bugtracker on GitHub (a la NumPy)? In-Reply-To: <5016DD42.5090404@molden.no> References: <5016DD42.5090404@molden.no> Message-ID: On Mon, Jul 30, 2012 at 9:15 PM, Sturla Molden wrote: > > (This might have been discussed before, forgive me if I've missed it.) > > NumPy is using "Issues" on GitHub as bugtracker in addition to trac it > seems. It has several advantages. Everything is kept in one place on > GitHub. The interface is more tidy. And pull requests can be attached to > "issues" (cf. tickets with attached .diff files on trac). > > In my experience the patches contributed on trac often get "forgotten". > Someone has to take time to get them into git/svn. (Which quickly > discouraged me from contributing anything.) I think this is the same > reason Ralph Gommers complained when I mailed cKDTree code here instead > of using git myself: "I know you may not have the time or interest to > learn about git right now, but it may make both our lives easier if you > try the below steps. It will allow you to put your commits on top of > mine without any manual copying." I guess that applies to anything > attached to a ticket on trac as well? > > Indeed. Patches as Trac attachments have some overhead for review and getting it merged. That's why unfortunately some patches have languished there for too long. > When SciPy is on GitHub I think we should consider using GitHub's issue > tracker. (And NumPy is already using it.) > The Trac --> Github conversion is still in progress for numpy. Once that's completed and it all looks good, the plan is to do the same for scipy. I'm glad you like git/Github. Looking forward to more patches:) Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Mon Jul 30 15:39:06 2012 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 30 Jul 2012 21:39:06 +0200 Subject: [SciPy-Dev] SciPy bugtracker on GitHub (a la NumPy)? In-Reply-To: <5016DD42.5090404@molden.no> References: <5016DD42.5090404@molden.no> Message-ID: 30.07.2012 21:15, Sturla Molden kirjoitti: [clip] > When SciPy is on GitHub I think we should consider using GitHub's issue > tracker. (And NumPy is already using it.) I would wait until Thouis finishes the trac ticket -> Github migration path. Github issues IMHO do not feature-wise offer much that Trac does not have, and the lack of attachments will mean in the context of Scipy that people who want to attach data files will have to work a bit harder to find separate hosting (and those links will become dead eventually). It also does not help with the "have-to-learn-git-first" issue regarding patches. Switching is more of an issue of shifting the server maintenance for somebody else to do. The integration between tickets and PRs becomes a bit better, but this in my experience can be worked tolerably around by pasting the corresponding cross-links in the comments. The better response for pull requests as compared to patches on Trac is a real effect. I think this is mainly due to: (i) the pull requests are much easier to locate (whereas in Trac, everything is mixed up with bug reports etc.) and look at, and (ii) what is submitted as pull request on average seems to be in a more "finished" form than code snippets attached on Trac. Pauli From sturla at molden.no Mon Jul 30 15:43:17 2012 From: sturla at molden.no (Sturla Molden) Date: Mon, 30 Jul 2012 21:43:17 +0200 Subject: [SciPy-Dev] SciPy bugtracker on GitHub (a la NumPy)? In-Reply-To: References: <5016DD42.5090404@molden.no> Message-ID: <5016E3D5.7020009@molden.no> Den 30.07.2012 21:39, skrev Pauli Virtanen: > I would wait until Thouis finishes the trac ticket -> Github migration path. > Yes, I've seen that now, sorry for bothering you with this :) Sturla From sturla at molden.no Mon Jul 30 15:58:34 2012 From: sturla at molden.no (Sturla Molden) Date: Mon, 30 Jul 2012 21:58:34 +0200 Subject: [SciPy-Dev] SciPy bugtracker on GitHub (a la NumPy)? In-Reply-To: References: <5016DD42.5090404@molden.no> Message-ID: <5016E76A.2070509@molden.no> Den 30.07.2012 21:31, skrev Ralf Gommers: > > I'm glad you like git/Github. Looking forward to more patches:)/scipy-dev I finally figured out how git/Github works. (That is, after reading two books on git, I'm still not comfortable using it, but I'm trying to learn.) Hopefully I can contribute something without just having it lost on trac. I discovered that Patrick Varilly's last PR did not have a corresponding ticket on trac. I don't think a PR without a ticket is a good idea, although his PR explains what it does. I guess this is easier to not to mess up if the issues/tickets are on Github as well. (That is the reason I started this thread.) Pauli: Waiting for Thouis' track -> Github migration seems like a good idea. I wasn't saying that trac should be closed either. :) Sturla From vanderplas at astro.washington.edu Mon Jul 30 16:08:16 2012 From: vanderplas at astro.washington.edu (Jacob VanderPlas) Date: Mon, 30 Jul 2012 13:08:16 -0700 Subject: [SciPy-Dev] SciPy bugtracker on GitHub (a la NumPy)? In-Reply-To: References: <5016DD42.5090404@molden.no> Message-ID: <5016E9B0.1040500@astro.washington.edu> On 07/30/12 12:39, Pauli Virtanen wrote: [clip] > Github issues IMHO do not feature-wise offer much that Trac does not > have, and the lack of attachments will mean in the context of Scipy that > people who want to attach data files will have to work a bit harder to > find separate hosting (and those links will become dead eventually). It > also does not help with the "have-to-learn-git-first" issue regarding > patches. Switching is more of an issue of shifting the server > maintenance for somebody else to do. Regarding attaching files in github issues and pull-requests: using http://gist.github.com streamlines this. > > The integration between tickets and PRs becomes a bit better, but this > in my experience can be worked tolerably around by pasting the > corresponding cross-links in the comments. > > The better response for pull requests as compared to patches on Trac is > a real effect. I think this is mainly due to: (i) the pull requests are > much easier to locate (whereas in Trac, everything is mixed up with bug > reports etc.) and look at, and (ii) what is submitted as pull request on > average seems to be in a more "finished" form than code snippets > attached on Trac. > > Pauli > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From pav at iki.fi Mon Jul 30 16:27:30 2012 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 30 Jul 2012 22:27:30 +0200 Subject: [SciPy-Dev] SciPy bugtracker on GitHub (a la NumPy)? In-Reply-To: <5016E9B0.1040500@astro.washington.edu> References: <5016DD42.5090404@molden.no> <5016E9B0.1040500@astro.washington.edu> Message-ID: 30.07.2012 22:08, Jacob VanderPlas kirjoitti: [clip] > Regarding attaching files in github issues and pull-requests: using > http://gist.github.com streamlines this. Can you attach binary files? I think it's possible by pushing directly to the Gist via Git, but not via the web interface. Also, when using the web interface, you need to copy-paste everything, right? Pauli From ralf.gommers at googlemail.com Mon Jul 30 16:36:42 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 30 Jul 2012 22:36:42 +0200 Subject: [SciPy-Dev] [SciPy-User] ANN: SciPy 0.11.0 release candidate 1 In-Reply-To: <500AE520.6030507@comcast.net> References: <50096161.8060207@comcast.net> <5009BE8A.3080902@comcast.net> <5009C6DF.2030806@iki.fi> <500AE520.6030507@comcast.net> Message-ID: On Sat, Jul 21, 2012 at 7:21 PM, John Hassler wrote: > > On 7/20/2012 5:00 PM, Pauli Virtanen wrote: > > 20.07.2012 22:24, John Hassler kirjoitti: > >> The offending line is: > >> AA,BB,Q,Z,sdim = qz(A,B,sort=sort) > >> > >> I couldn't find 'sort' defined anywhere, so I arbitrarily changed it to > >> sort='lhp'. Then it runs, although the test fails. > >> > >> Is there something else I can try? > > Seems to be a problem with the callable sort function then. > > > > That it works with sort='lhp' is strange, and probably means that there > > is not a problem with all callback functions, but something goes wrong > > in the algorithm. (Which would be expected, if *gees callbacks work > > without problems.) > > > > If you can/know how to recompile, try recompiling with > > > > set OPT=-g -DDEBUGCFUNCS > > python setup.py ......... > > > > This should make the f2py wrappers spit out extra information on what is > > going on. > > > I'm not set up to recompile. Is there anyone who can reproduce the issue and investigate this further? Alternatively I can build some binaries with the above debug flags. I can't reproduce the issue though, so that may be a painful way to debug. Ralf > I set up a little test program to > reproduce the problem (below). It leads to the Fortran call to gges in > _decomp_qz.py. This crashes if sort_t = 1 (which calls the lambda), but > not for sort_t = 0. I didn't play with it any further. > john > > # Test qz > import numpy as np > from scipy.linalg import qz > > A = np.array([[3.9, 12.5, -34.5, -0.5], > [ 4.3, 21.5, -47.5, 7.5], > [ 4.3, 21.5, -43.5, 3.5], > [ 4.4, 26.0, -46.0, 6.0 ]]) > > B = np.array([[ 1.0, 2.0, -3.0, 1.0], > [1.0, 3.0, -5.0, 4.0], > [1.0, 3.0, -4.0, 3.0], > [1.0, 3.0, -4.0, 4.0]]) > > > #sort = lambda ar,ai,beta : ai == 0 ## crashes > sort = None ## runs > > print(qz(A,B,sort=sort)) > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanderplas at astro.washington.edu Mon Jul 30 16:45:05 2012 From: vanderplas at astro.washington.edu (Jacob VanderPlas) Date: Mon, 30 Jul 2012 13:45:05 -0700 Subject: [SciPy-Dev] SciPy bugtracker on GitHub (a la NumPy)? In-Reply-To: References: <5016DD42.5090404@molden.no> <5016E9B0.1040500@astro.washington.edu> Message-ID: <5016F251.30201@astro.washington.edu> On 07/30/12 13:27, Pauli Virtanen wrote: > 30.07.2012 22:08, Jacob VanderPlas kirjoitti: > [clip] >> Regarding attaching files in github issues and pull-requests: using >> http://gist.github.com streamlines this. > Can you attach binary files? I think it's possible by pushing directly > to the Gist via Git, but not via the web interface. Also, when using the > web interface, you need to copy-paste everything, right? You can attach any sort of file from the git interface (i.e. by cloning the gist, adding files locally, then pushing to origin). There's not a web interface for this that I'm aware of. The web interface is only copy/paste of text. Jake From sturla at molden.no Mon Jul 30 17:01:06 2012 From: sturla at molden.no (Sturla Molden) Date: Mon, 30 Jul 2012 23:01:06 +0200 Subject: [SciPy-Dev] SciPy bugtracker on GitHub (a la NumPy)? In-Reply-To: <5016F251.30201@astro.washington.edu> References: <5016DD42.5090404@molden.no> <5016E9B0.1040500@astro.washington.edu> <5016F251.30201@astro.washington.edu> Message-ID: <5016F612.5040603@molden.no> Den 30.07.2012 22:45, skrev Jacob VanderPlas: > > You can attach any sort of file from the git interface (i.e. by cloning > the gist, adding files locally, then pushing to origin). There's not a > web interface for this that I'm aware of. The web interface is only > copy/paste of text. It seems that gist only accepts certain text formats, not binary. But it can accept base64 as ascii text I think. Sturla From thomas.haslwanter at alumni.ethz.ch Sun Jul 29 08:23:28 2012 From: thomas.haslwanter at alumni.ethz.ch (Thomas Haslwanter) Date: Sun, 29 Jul 2012 12:23:28 +0000 (UTC) Subject: [SciPy-Dev] Contributing to SciPy? Message-ID: Hi, I found an error on the SciPy Cookbook (described below), and wanted to fix it. So I wrote a corrected version and a unittest procedure. But now I don't know how I can submit/contribute it to SciPy. (BTW, the wiki did not let me to change it, even after I registered.) Is the only way to pull the whole sourcecode, install Fortran and C++ compilers, etc? Or can I just submit Python code? Please let me know. The problem: The Cookbook entry http://www.scipy.org/Cookbook/SavitzkyGolay has a number of problems: 1) As it is, it does in general not run. The line "import numpy as np" should be inserted before the first "try" in line 49 2) I don't completely understand the workings of the code. However, when you put in a sine-wave and calculate the first derivative, you get out a negative(!) cosine - not a cosine. The correct result appears if you replace "m," in line 68 with "m[::-1]," Since the Savitzky-Golay filter is a very commonly used filter, I regard this mistake as significant. thomas