From c.j.lee at tnw.utwente.nl Wed Apr 1 08:41:54 2009 From: c.j.lee at tnw.utwente.nl (Chris Lee) Date: Wed, 1 Apr 2009 14:41:54 +0200 Subject: [SciPy-user] cross correlation functions Message-ID: <8F06DE86-FA4A-4E1A-BF61-17ABB543FCFB@tnw.utwente.nl> Hi All, I have two arrays of data and I want to perform a cross correlation on them. Will correlate(input, weights, output=None, mode='reflect', cval=0.0, origin=0) do this for me? Would the structure be something like this (pseudo code): arr1 = [bunch of data] arr2 = [bunch of data] input_arr = append(arr1, arr2) weigths = zeros(input_arr.shape, float) weigth[max(arr1.shape)::] = 1 out_mat = correlate(input_arr, weights) Otherwise, I may have to loop... the prospect is not appealing as these arrays are reasonably large. Cheers Chris *************************************************** Chris Lee Laser Physics and Nonlinear Optics Group MESA+ Research Institute for Nanotechnology University of Twente Phone: ++31 (0)53 489 3968 fax: ++31 (0)53 489 1102 *************************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.pincus at yale.edu Wed Apr 1 11:30:34 2009 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Wed, 1 Apr 2009 11:30:34 -0400 Subject: [SciPy-user] How to fit a surface from a list of measured 3D points ? In-Reply-To: References: Message-ID: <3EBDDD15-1C15-4F9A-BCE1-5B8123BC5C0D@yale.edu> Hello, > I would like to estimate some geoemetric characteristic of this > surface and view the variation of radius of curvature. > >> What's your eventual goal for the fit surface? There are a lot of >> possible approaches possible with the tools in scipy (e.g. fit a >> spline, as you have done, or fit a simpler parameteric surface in a >> least-squares fashion, or go for a nonparametric approach and >> interpolate between the points with radial basis functions). The >> best >> approach will likely depend on what you want to do with the >> surface... >> but what you have seems like a reasonable start. > I think a 2D surface spline could do the trick, but I think I didn't > fit it well : > - the estimated z can go far below the range of my inputs ( [48-62] ) > - As you can see on small_data.jpg, the surface is really 3D, there is > a kind of crease in my surface, and there are two z value for (x,y) ~ > (2,2). So I 'm not sure if this is the good approach Hmm, good point. Can you rotate the data points in the 3D space so that the new z values do become a proper function in two dimensions? If not, then you'll have to: a) fit a surface to all of the data in 3D (something done a lot by computer graphics and robotics people, who get point clouds as return data from LIDAR scanners and similar, and then try to fit the points to 3D surfaces for visualization / navigation) b) Find locally-smooth patches and fit surfaces to these individually (the manifold-learning folks do this, e.g. "Hessian LLE"). Say you're interested in curvature around a given data point (x, y, z)... you could take the points within some neighborhood and then either fit them to a simple 3d surface (like some kind of paraboloid), or figure out (with e.g. PCA) the best projection of those data points to a plane, and then fit a surface to f(x, y) -> z for the transformed data. or perhaps even c) just calculate what you need from the data points directly. If you just need very local curvature data, you could probably calculate that from a point and its nearest neighbors. (This is really just a degenerate case of b...) Lots of tools for these tasks are in scipy, but nothing off-the-shelf that I know if. Zach From guyer at nist.gov Wed Apr 1 13:36:47 2009 From: guyer at nist.gov (Jonathan Guyer) Date: Wed, 1 Apr 2009 13:36:47 -0400 Subject: [SciPy-user] "smooth" plots Message-ID: <26B3CE4E-5D3E-4904-A0BC-36F319103DDA@nist.gov> Lets say I want to plot y = f(x). f() is non-analytical and expensive. Are there any facilities in SciPy (or elsewhere in Python) for automating the selection of values of x that will produce a "smooth" plot of y vs. x? The only mentions I've seen for adaptive step size selection are for the ode integrators, which I'm not doing. Likewise, the splines and other smoothing functions don't seem appropriate because I don't have noisy data; I'm trying to figure out what data I even need to calculate (which will probably not be noisy at all). Years ago I did things like this in Mathematica, which was really good at it (even though the actual plots were hideous (necessitating export of the plot points (which Mma was also really bad at))). From berthe.loic at gmail.com Wed Apr 1 13:40:57 2009 From: berthe.loic at gmail.com (LB) Date: Wed, 1 Apr 2009 10:40:57 -0700 (PDT) Subject: [SciPy-user] How to fit a surface from a list of measured 3D points ? In-Reply-To: <3EBDDD15-1C15-4F9A-BCE1-5B8123BC5C0D@yale.edu> References: <3EBDDD15-1C15-4F9A-BCE1-5B8123BC5C0D@yale.edu> Message-ID: <8ba863ef-eee5-4a0b-b298-918eca024b1c@v15g2000yqn.googlegroups.com> > Hmm, good point. Can you rotate the data points in the 3D space so ? > that the new z values do become a proper function in two dimensions? ? It may be possible, with some manual transformation of the data points, but I would prefer a more generic approach if possible. > If not, then you'll have to: > a) fit a surface to all of the data in 3D (something done a lot by ? > computer graphics and robotics people, who get point clouds as return ? > data from LIDAR scanners and similar, and then try to fit the points ? > to 3D surfaces for visualization / navigation) > > b) Find locally-smooth patches and fit surfaces to these individually ? > (the manifold-learning folks do this, e.g. "Hessian LLE"). Say you're ? > interested in curvature around a given data point (x, y, z)... you ? > could take the points within some neighborhood and then either fit ? > them to a simple 3d surface (like some kind of paraboloid), or figure ? > out (with e.g. PCA) the best projection of those data points to a ? > plane, and then fit a surface to f(x, y) -> z for the transformed data. > > or perhaps even c) just calculate what you need from the data points ? > directly. If you just need very local curvature data, you could ? > probably calculate that from a point and its nearest neighbors. (This ? > is really just a degenerate case of b...) > > Lots of tools for these tasks are in scipy, but nothing off-the-shelf ? > that I know if. The method c) seems the simplest at first sight but I see two issues for this local approach : - the measured data are noisy. Using the nearest neighbor could give a noisy result two, especially when looking at a radius of curvature - I don't see how to use this approach to plot the variation of radius of curvature along the surface, It can give me an array of radius of curvature, but as my data are not regularly spaced, it won't be easy to handle. Th method b) seems very fuzzy to me : I don't have any knowledge in manifold-learning and I would have the second issue of the method c) too. The method a) is what I had initially in mind, but I didn't see how to do this in scipy :-( I believed at first that I could make a sort of parametric bispline fit with the functions available in scipy.interpolate, but I didn't succeed in. Do you have any example or hint for doing this kind of treatment in scipy ? LB From josef.pktd at gmail.com Wed Apr 1 13:51:30 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 1 Apr 2009 13:51:30 -0400 Subject: [SciPy-user] How to fit a surface from a list of measured 3D points ? In-Reply-To: <8ba863ef-eee5-4a0b-b298-918eca024b1c@v15g2000yqn.googlegroups.com> References: <3EBDDD15-1C15-4F9A-BCE1-5B8123BC5C0D@yale.edu> <8ba863ef-eee5-4a0b-b298-918eca024b1c@v15g2000yqn.googlegroups.com> Message-ID: <1cd32cbb0904011051h8ac630fua35ecaa28b000720@mail.gmail.com> On Wed, Apr 1, 2009 at 1:40 PM, LB wrote: >> Hmm, good point. Can you rotate the data points in the 3D space so >> that the new z values do become a proper function in two dimensions? > It may be possible, with some manual transformation of the data > points, but I would prefer a more generic approach if possible. > >> If not, then you'll have to: >> a) fit a surface to all of the data in 3D (something done a lot by >> computer graphics and robotics people, who get point clouds as return >> data from LIDAR scanners and similar, and then try to fit the points >> to 3D surfaces for visualization / navigation) >> >> b) Find locally-smooth patches and fit surfaces to these individually >> (the manifold-learning folks do this, e.g. "Hessian LLE"). Say you're >> interested in curvature around a given data point (x, y, z)... you >> could take the points within some neighborhood and then either fit >> them to a simple 3d surface (like some kind of paraboloid), or figure >> out (with e.g. PCA) the best projection of those data points to a >> plane, and then fit a surface to f(x, y) -> z for the transformed data. >> >> or perhaps even c) just calculate what you need from the data points >> directly. If you just need very local curvature data, you could >> probably calculate that from a point and its nearest neighbors. (This >> is really just a degenerate case of b...) >> >> Lots of tools for these tasks are in scipy, but nothing off-the-shelf >> that I know if. > > The method c) seems the simplest at first sight but I see two issues > for this local approach : > ?- the measured data are noisy. Using the nearest neighbor could give > a noisy result two, especially when looking at a radius of curvature > ?- I don't see how to use this approach to plot the variation of > radius of curvature along the surface, It can give me an array of > radius of curvature, but as my data are not regularly spaced, it won't > be easy to handle. > > Th method b) seems very fuzzy to me : I don't have any knowledge in > manifold-learning and I would have the second issue of the method c) > too. > > The method a) is what I had initially in mind, but I didn't see how to > do this in scipy :-( > I believed at first that I could make a sort of parametric bispline > fit with the functions available in scipy.interpolate, but I didn't > succeed in. > Do you have any example or hint for doing this kind of treatment in > scipy ? > > LB If you have noisy data, then a kernel ridge regression or fitting a gaussian process might be more appropriate than just interpolation. I posted a simple example code for it a while ago, and pymvpa has a more complete implementation. Around 900 points should still be ok, since it builds the square distance matrix (900,900) I don't know anything about your curvature measure, but it should be worth a try. Josef From robert.kern at gmail.com Wed Apr 1 15:22:47 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 1 Apr 2009 14:22:47 -0500 Subject: [SciPy-user] "smooth" plots In-Reply-To: <26B3CE4E-5D3E-4904-A0BC-36F319103DDA@nist.gov> References: <26B3CE4E-5D3E-4904-A0BC-36F319103DDA@nist.gov> Message-ID: <3d375d730904011222y3d8b4864x465e101079ebc546@mail.gmail.com> On Wed, Apr 1, 2009 at 12:36, Jonathan Guyer wrote: > Lets say I want to plot y = f(x). > > f() is non-analytical and expensive. > > Are there any facilities in SciPy (or elsewhere in Python) for > automating the selection of values of x that will produce a "smooth" > plot of y vs. x? The only mentions I've seen for adaptive step size > selection are for the ode integrators, which I'm not doing. Likewise, > the splines and other smoothing functions don't seem appropriate > because I don't have noisy data; I'm trying to figure out what data I > even need to calculate (which will probably not be noisy at all). Most of the spline fitting in scipy.interpolate is for exact interpolation and not noisy data. If your function is smooth (C0 and C1 continuous), and there are no singularities, then the most straightforward thing to do would be to do a coarse sampling of your function, spline fit it, and then evaluate the spline finely. This is, of course, approximate, but probably not too bad. There is no adaptive sampling in scipy. That would require much more information about the screen geometry and line thicknesses. We leave that to the plotting frameworks. You might want to check SAGE for this functionality; they would often have need for it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From guyer at nist.gov Wed Apr 1 15:29:04 2009 From: guyer at nist.gov (Jonathan Guyer) Date: Wed, 1 Apr 2009 15:29:04 -0400 Subject: [SciPy-user] "smooth" plots In-Reply-To: <3d375d730904011222y3d8b4864x465e101079ebc546@mail.gmail.com> References: <26B3CE4E-5D3E-4904-A0BC-36F319103DDA@nist.gov> <3d375d730904011222y3d8b4864x465e101079ebc546@mail.gmail.com> Message-ID: <734DE825-2DF6-440D-98CC-34ED550F9BEF@nist.gov> On Apr 1, 2009, at 3:22 PM, Robert Kern wrote: > straightforward thing to do would be to do a coarse sampling of your > function, spline fit it, and then evaluate the spline finely. This is, > of course, approximate, but probably not too bad. That's probably a reasonable approach for my immediate need. > There is no adaptive sampling in scipy. That would require much more > information about the screen geometry and line thicknesses. I fully expected to have to tell it what I thought was "good" or "smooth", but I'd hoped to find something that would automate the guesses. I've got some papers on it around here someplace; I just was hoping to exploit somebody else's labor. > You might want to check SAGE for this > functionality; they would often have need for it. Good idea, thanks. From josef.pktd at gmail.com Wed Apr 1 15:41:44 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 1 Apr 2009 15:41:44 -0400 Subject: [SciPy-user] cross correlation functions In-Reply-To: <8F06DE86-FA4A-4E1A-BF61-17ABB543FCFB@tnw.utwente.nl> References: <8F06DE86-FA4A-4E1A-BF61-17ABB543FCFB@tnw.utwente.nl> Message-ID: <1cd32cbb0904011241t3d3d48c3hb230e08be0aad3f7@mail.gmail.com> On Wed, Apr 1, 2009 at 8:41 AM, Chris Lee wrote: > Hi All, > I have two arrays of data and I want to perform a cross correlation on them. > Will?correlate(input, weights, output=None, mode='reflect', cval=0.0, > origin=0) do this for me? > Would the structure be something like this (pseudo code): > arr1 = [bunch of data] > arr2 = [bunch of data] > input_arr =?append(arr1, arr2) > weigths = zeros(input_arr.shape, float) > weigth[max(arr1.shape)::] = 1 > out_mat = correlate(input_arr, weights) > Otherwise, I may have to loop... the prospect is not appealing as these > arrays are reasonably large. > Cheers > Chris What's the dimension of your arr1, arr2? there are 3 different correlate in numpy/scipy, for 1d numpy.correlate and for 2d scipy.signal.correlate2d might do more directly what you want. Josef From simpson at math.toronto.edu Wed Apr 1 19:32:07 2009 From: simpson at math.toronto.edu (Gideon Simpson) Date: Wed, 1 Apr 2009 19:32:07 -0400 Subject: [SciPy-user] blitz error Message-ID: <8164ACA8-6B03-418B-AD13-DCC6B0AF2BBF@math.toronto.edu> I tried this little bit of code: def linear_p(qp_hat, qm_hat, vg, kappa, n1, il, ik): linp = np.zeros(qp_hat.shape) + 0.j expr = "linp = -vg * qp_hat * il + kappa * ik * n1 * qm_hat" weave.blitz(expr, check_size=1) return linp and got the following error: Traceback (most recent call last): File "./soliton1_blitz.py", line 185, in k1p = filter(k1p) + linop_p(Ep_hat, Em_hat) File "./soliton1_blitz.py", line 79, in vg, kappa, n1, il, ik) File "/Users/gideon/Code/CME/CMEops_blitz.py", line 10, in linear_p weave.blitz(expr, check_size=1) File "/opt/lib/python2.5/site-packages/scipy/weave/blitz_tools.py", line 62, in blitz **kw) File "/opt/lib/python2.5/site-packages/scipy/weave/ inline_tools.py", line 462, in compile_function verbose=verbose, **kw) File "/opt/lib/python2.5/site-packages/scipy/weave/ext_tools.py", line 367, in compile verbose = verbose, **kw) File "/opt/lib/python2.5/site-packages/scipy/weave/build_tools.py", line 272, in build_extension setup(name = module_name, ext_modules = [ext],verbose=verb) File "/opt/lib/python2.5/site-packages/numpy/distutils/core.py", line 184, in setup return old_setup(**new_attr) File "/sw/lib/python2.5/distutils/core.py", line 168, in setup raise SystemExit, "error: " + str(msg) scipy.weave.build_tools.CompileError: error: Command "g++ -fno-strict- aliasing -mno-fused-madd -DNDEBUG -g -fwrapv -O3 -Wall -I/opt/lib/ python2.5/site-packages/scipy/weave -I/opt/lib/python2.5/site-packages/ scipy/weave/scxx -I/opt/lib/python2.5/site-packages/scipy/weave/blitz - I/opt/lib/python2.5/site-packages/numpy/core/include -I/sw/include/ python2.5 -c /Users/gideon/.python25_compiled/ sc_24bab5a447df6354e489345507f46f0d2.cpp -o /var/folders/yL/ yLD5tRJiGWa7oyM6mWJUN++++TI/-Tmp-/gideon/python25_intermediate/ compiler_eded75b75a17d5bc9afe97cc30cf08c0/Users/ gideon/.python25_compiled/sc_24bab5a447df6354e489345507f46f0d2.o" failed with exit status 1 Could it be because I'm using complex numbers? -gideon From c.j.lee at tnw.utwente.nl Thu Apr 2 02:52:38 2009 From: c.j.lee at tnw.utwente.nl (Chris Lee) Date: Thu, 2 Apr 2009 08:52:38 +0200 Subject: [SciPy-user] cross correlation functions In-Reply-To: <1cd32cbb0904011241t3d3d48c3hb230e08be0aad3f7@mail.gmail.com> References: <8F06DE86-FA4A-4E1A-BF61-17ABB543FCFB@tnw.utwente.nl> <1cd32cbb0904011241t3d3d48c3hb230e08be0aad3f7@mail.gmail.com> Message-ID: Thanks, I think the 2d function is what I want. I have two 1-d arrays and I would like a 2-d cross correlation of the two. Cheers Chris On Apr 1, 2009, at 9:41 PM, josef.pktd at gmail.com wrote: > On Wed, Apr 1, 2009 at 8:41 AM, Chris Lee > wrote: >> Hi All, >> I have two arrays of data and I want to perform a cross correlation >> on them. >> Will correlate(input, weights, output=None, mode='reflect', cval=0.0, >> origin=0) do this for me? >> Would the structure be something like this (pseudo code): >> arr1 = [bunch of data] >> arr2 = [bunch of data] >> input_arr = append(arr1, arr2) >> weigths = zeros(input_arr.shape, float) >> weigth[max(arr1.shape)::] = 1 >> out_mat = correlate(input_arr, weights) >> Otherwise, I may have to loop... the prospect is not appealing as >> these >> arrays are reasonably large. >> Cheers >> Chris > > What's the dimension of your arr1, arr2? there are 3 different > correlate in numpy/scipy, for 1d numpy.correlate and for 2d > scipy.signal.correlate2d might do more directly what you want. > > Josef > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user *************************************************** Chris Lee Laser Physics and Nonlinear Optics Group MESA+ Research Institute for Nanotechnology University of Twente Phone: ++31 (0)53 489 3968 fax: ++31 (0)53 489 1102 *************************************************** From rchrdlyon1 at gmail.com Thu Apr 2 04:02:04 2009 From: rchrdlyon1 at gmail.com (Richard Lyon) Date: Thu, 02 Apr 2009 19:02:04 +1100 Subject: [SciPy-user] Memory Leak in Scipy.Signal.lfilter? Message-ID: <49D470FC.3030107@gmail.com> Hi, Problem: ======= Been successfully using scipy to simulate a signal decoding algorithm. Recently upgraded from release 0.6.0 to release 0.7.0 and discovered the simulation leaks memory, which brings the computer to a grinding halt after a few minutes. Details: ======= Microsoft Vista Python 2.5.2 Numpy 1.2.1 Scipy 0.7.0 The following code leaks memory: --------------------------------------------------------- # Check if scipy.signal.lfilter causes memory leak import numpy from scipy.signal import lfilter def main(): # low pass filter coefficients (maybe this should be an array) hlp = [-2.145760E-003, -4.058501E-003, -4.629715E-003, -1.294677E-003, 6.771358E-003, 1.692096E-002, 2.300385E-002, 1.825164E-002, 1.908418E-004, -2.526333E-002, -4.382636E-002, -3.841010E-002, 1.735187E-003, 7.313454E-002, 1.565082E-001, 2.235226E-001, 2.491718E-001, 2.235226E-001, 1.565082E-001, 7.313454E-002, 1.735187E-003, -3.841010E-002, -4.382636E-002, -2.526333E-002, 1.908418E-004, 1.825164E-002, 2.300385E-002, 1.692096E-002, 6.771358E-003, -1.294677E-003, -4.629715E-003, -4.058501E-003, -2.145760E-003] # initialise low pass filter registers (maybe this should be an array) lp_filter_registers = [0.0]*(len(hlp)-1) # read individual samples and filter s = 0 while s < 1000000: # for test just use zero value incoming_sample = numpy.array([0.0]); # low pass filter samples (lp_filtered_sample, lp_filter_registers) = lfilter(hlp, 1.0, incoming_sample, -1, lp_filter_registers) s = s + 1 # main entry point if __name__ == '__main__': print 'Running memory leak test' main() print 'Finished.' --------------------------------------------------------- When I view the Physical Memory Usage History in the Windows task manager it is ramp which only halts growing when memory is exhausted. For the time being I have reverted back to the previous version of scipy to keep working. Anyone else observed the same problem? Regards ... From david at ar.media.kyoto-u.ac.jp Thu Apr 2 04:01:11 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 02 Apr 2009 17:01:11 +0900 Subject: [SciPy-user] Memory Leak in Scipy.Signal.lfilter? In-Reply-To: <49D470FC.3030107@gmail.com> References: <49D470FC.3030107@gmail.com> Message-ID: <49D470C7.7020303@ar.media.kyoto-u.ac.jp> Hi Rchard, Richard Lyon wrote: > Anyone else observed the same problem? > On my machine, it segfaults right away. And I am more than likely the one responsible for it. The problem looks relatively simple, though - I will try to look at it later today, Could you fill a bug on trac ? cheers, David From pav at iki.fi Thu Apr 2 04:24:16 2009 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 2 Apr 2009 08:24:16 +0000 (UTC) Subject: [SciPy-user] Memory Leak in Scipy.Signal.lfilter? References: <49D470FC.3030107@gmail.com> <49D470C7.7020303@ar.media.kyoto-u.ac.jp> Message-ID: Thu, 02 Apr 2009 17:01:11 +0900, David Cournapeau kirjoitti: > Richard Lyon wrote: >> Anyone else observed the same problem? > > On my machine, it segfaults right away. And I am more than likely the > one responsible for it. The problem looks relatively simple, though - I > will try to look at it later today, > > Could you fill a bug on trac ? I was faster ;) http://projects.scipy.org/scipy/ticket/911 And leaks memory here, too. -- Pauli Virtanen From zvalim at gmail.com Thu Apr 2 09:35:22 2009 From: zvalim at gmail.com (zvalim at gmail.com) Date: Thu, 2 Apr 2009 15:35:22 +0200 Subject: [SciPy-user] problem installing latest scikit timeseries (0.91) Message-ID: <656a77fc0904020635w65d5ded5pef811ad87512cba7@mail.gmail.com> Dear all, I am trying to install scikits.timeseries 0.91.0 and am getting the following error during installation (I have installed setuptools, numpy 1.3.0rc1 and scipy0.7. All installed without a glitch.) -------- [snip] Processing dependencies for scikits.timeseries==0.91.0 Searching for numpy>=1.3.0b1 Reading http://pypi.python.org/simple/numpy/ Reading http://numpy.scipy.org Reading http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 Reading http://numeric.scipy.org No local packages or download links found for numpy>=1.3.0b1 error: Could not find suitable distribution for Requirement.parse('numpy>=1.3.0b1') ---- AFAICT, I am running the newest version of numpy. Has anyone run into this? Cheers, Uri From josef.pktd at gmail.com Thu Apr 2 09:36:26 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 2 Apr 2009 09:36:26 -0400 Subject: [SciPy-user] cross correlation functions In-Reply-To: References: <8F06DE86-FA4A-4E1A-BF61-17ABB543FCFB@tnw.utwente.nl> <1cd32cbb0904011241t3d3d48c3hb230e08be0aad3f7@mail.gmail.com> Message-ID: <1cd32cbb0904020636wd03f7f4g97e8f529fcedee96@mail.gmail.com> On Thu, Apr 2, 2009 at 2:52 AM, Chris Lee wrote: > Thanks, > > I think the 2d function is what I want. I have two 1-d arrays and I > would like a 2-d cross correlation of the two. > > Cheers > Chris > > On Apr 1, 2009, at 9:41 PM, josef.pktd at gmail.com wrote: > >> On Wed, Apr 1, 2009 at 8:41 AM, Chris Lee >> wrote: >>> Hi All, >>> I have two arrays of data and I want to perform a cross correlation >>> on them. >>> Will correlate(input, weights, output=None, mode='reflect', cval=0.0, >>> origin=0) do this for me? >>> Would the structure be something like this (pseudo code): >>> arr1 = [bunch of data] >>> arr2 = [bunch of data] >>> input_arr = append(arr1, arr2) >>> weigths = zeros(input_arr.shape, float) >>> weigth[max(arr1.shape)::] = 1 >>> out_mat = correlate(input_arr, weights) >>> Otherwise, I may have to loop... the prospect is not appealing as >>> these >>> arrays are reasonably large. >>> Cheers >>> Chris >> >> What's the dimension of your arr1, arr2? there are 3 different >> correlate in numpy/scipy, for 1d numpy.correlate and for 2d >> scipy.signal.correlate2d might do more directly what you want. >> >> Josef for two 1-d arrays np.correlate should be enough. I had forgotten to add some references to current implementation to my previous email, they also show how to normalize the cross covariance. an example implementation for masked arrays is in http://scipy.org/scipy/scikits/browser/trunk/timeseries/scikits/timeseries/lib/avcf.py#L67 http://scipy.org/scipy/scikits/browser/trunk/timeseries/scikits/timeseries/lib/avcf.py#L135 Pierre also had an example for different implementations in http://www.mail-archive.com/matplotlib-users at lists.sourceforge.net/msg01487.html matplotlib also has an implementation based on np.correlate matplotlib.pyplot.acorr matplotlib.pyplot.xcorr Josef From josef.pktd at gmail.com Thu Apr 2 09:42:26 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 2 Apr 2009 09:42:26 -0400 Subject: [SciPy-user] problem installing latest scikit timeseries (0.91) In-Reply-To: <656a77fc0904020635w65d5ded5pef811ad87512cba7@mail.gmail.com> References: <656a77fc0904020635w65d5ded5pef811ad87512cba7@mail.gmail.com> Message-ID: <1cd32cbb0904020642k76871c36p2f2aee38642f1455@mail.gmail.com> On Thu, Apr 2, 2009 at 9:35 AM, wrote: > Dear all, > > I am trying to install scikits.timeseries 0.91.0 and am getting the > following error during installation (I have installed setuptools, > numpy 1.3.0rc1 and scipy0.7. All installed without a glitch.) > > -------- > [snip] > Processing dependencies for scikits.timeseries==0.91.0 > Searching for numpy>=1.3.0b1 > Reading http://pypi.python.org/simple/numpy/ > Reading http://numpy.scipy.org > Reading http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 > Reading http://numeric.scipy.org > No local packages or download links found for numpy>=1.3.0b1 > error: Could not find suitable distribution for > Requirement.parse('numpy>=1.3.0b1') > ---- > > AFAICT, I am running the newest version of numpy. Has anyone run into this? > > Cheers, > Uri setuptools and easyinstall had in the past often problems to recognize the already installed numpy and I ususally just comment out any numpy or scipy requirements in a setup.py file. But that's a quick fix not a cure for whatever the underlying problem is. Josef From pgmdevlist at gmail.com Thu Apr 2 11:59:39 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 2 Apr 2009 11:59:39 -0400 Subject: [SciPy-user] problem installing latest scikit timeseries (0.91) In-Reply-To: <656a77fc0904020635w65d5ded5pef811ad87512cba7@mail.gmail.com> References: <656a77fc0904020635w65d5ded5pef811ad87512cba7@mail.gmail.com> Message-ID: Uri, Sorry about that. Hopefully it should be sorted when 1.3.0 is officially released. In the mean time, you can follow Josef's advice and comment the requirement line in setup.py (the one at the same level as the scikits directory in the distribution). On Apr 2, 2009, at 9:35 AM, zvalim at gmail.com wrote: > Dear all, > > I am trying to install scikits.timeseries 0.91.0 and am getting the > following error during installation (I have installed setuptools, > numpy 1.3.0rc1 and scipy0.7. All installed without a glitch.) > > -------- > [snip] > Processing dependencies for scikits.timeseries==0.91.0 > Searching for numpy>=1.3.0b1 > Reading http://pypi.python.org/simple/numpy/ > Reading http://numpy.scipy.org > Reading http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 > Reading http://numeric.scipy.org > No local packages or download links found for numpy>=1.3.0b1 > error: Could not find suitable distribution for > Requirement.parse('numpy>=1.3.0b1') > ---- > > AFAICT, I am running the newest version of numpy. Has anyone run > into this? > > Cheers, > Uri > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From zachary.pincus at yale.edu Thu Apr 2 12:17:41 2009 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Thu, 2 Apr 2009 12:17:41 -0400 Subject: [SciPy-user] How to fit a surface from a list of measured 3D points ? In-Reply-To: <8ba863ef-eee5-4a0b-b298-918eca024b1c@v15g2000yqn.googlegroups.com> References: <3EBDDD15-1C15-4F9A-BCE1-5B8123BC5C0D@yale.edu> <8ba863ef-eee5-4a0b-b298-918eca024b1c@v15g2000yqn.googlegroups.com> Message-ID: <7D2F401F-7183-4621-BA59-8027699580C5@yale.edu> >> If not, then you'll have to: >> a) fit a surface to all of the data in 3D (something done a lot by >> computer graphics and robotics people, who get point clouds as return >> data from LIDAR scanners and similar, and then try to fit the points >> to 3D surfaces for visualization / navigation) >> >> b) Find locally-smooth patches and fit surfaces to these individually >> (the manifold-learning folks do this, e.g. "Hessian LLE"). Say you're >> interested in curvature around a given data point (x, y, z)... you >> could take the points within some neighborhood and then either fit >> them to a simple 3d surface (like some kind of paraboloid), or figure >> out (with e.g. PCA) the best projection of those data points to a >> plane, and then fit a surface to f(x, y) -> z for the transformed >> data. >> >> or perhaps even c) just calculate what you need from the data points >> directly. If you just need very local curvature data, you could >> probably calculate that from a point and its nearest neighbors. (This >> is really just a degenerate case of b...) >> >> Lots of tools for these tasks are in scipy, but nothing off-the-shelf >> that I know if. > > The method c) seems the simplest at first sight but I see two issues > for this local approach : > - the measured data are noisy. Using the nearest neighbor could give > a noisy result two, especially when looking at a radius of curvature > - I don't see how to use this approach to plot the variation of > radius of curvature along the surface, It can give me an array of > radius of curvature, but as my data are not regularly spaced, it won't > be easy to handle. > > Th method b) seems very fuzzy to me : I don't have any knowledge in > manifold-learning and I would have the second issue of the method c) > too. > > The method a) is what I had initially in mind, but I didn't see how to > do this in scipy :-( > I believed at first that I could make a sort of parametric bispline > fit with the functions available in scipy.interpolate, but I didn't > succeed in. Right. A bivariate spline is to fit data of the form f(x, y) -> z, whereas your data are just scattered (x, y, z) points. That's not going to work unless you can reliably transform them to a function in 2D. (Perhaps a PCA step to find two axes of maximum variance in the data, and then a projection down to these axes will be a general method to transform the data to something a bi-spline can fit. On the other hand, if the data are, e.g., helical, there would be no transform you could do to render them as something that could be fit by a function f(x, y) -> z...) So you'll have to choose some parameterized family of surfaces that you feel will fit the data sufficiently, and then find the best-fit parameters to give a curve that approximates the data. This is a very general problem (nonlinear regression, or just function optimization, depending on how you cast it) for which there are many tools in scipy. But for figuring out the right family of surfaces to use, you'll have to dig into the graphics or robotics literature. Quite likely there's some generalized spline surface you could use, but there's nothing of the shelf in scipy to do that, other than the basic numerical tools. Zach From josef.pktd at gmail.com Thu Apr 2 15:01:57 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 2 Apr 2009 15:01:57 -0400 Subject: [SciPy-user] blitz error In-Reply-To: <8164ACA8-6B03-418B-AD13-DCC6B0AF2BBF@math.toronto.edu> References: <8164ACA8-6B03-418B-AD13-DCC6B0AF2BBF@math.toronto.edu> Message-ID: <1cd32cbb0904021201na71f74dt3981fadaefa7ebca@mail.gmail.com> On Wed, Apr 1, 2009 at 7:32 PM, Gideon Simpson wrote: > I tried this little bit of code: > > def linear_p(qp_hat, qm_hat, vg, kappa, n1, il, ik): > > ? ? ? ?linp = np.zeros(qp_hat.shape) + 0.j > ? ? ? ?expr = "linp = -vg * qp_hat * il + kappa * ik * n1 * qm_hat" > ? ? ? ?weave.blitz(expr, check_size=1) > ? ? ? ?return linp > > and got the following error: > > Traceback (most recent call last): > ? File "./soliton1_blitz.py", line 185, in > ? ? k1p = filter(k1p) + linop_p(Ep_hat, Em_hat) > ? File "./soliton1_blitz.py", line 79, in > ? ? vg, kappa, n1, il, ik) > ? File "/Users/gideon/Code/CME/CMEops_blitz.py", line 10, in linear_p > ? ? weave.blitz(expr, check_size=1) > ? File "/opt/lib/python2.5/site-packages/scipy/weave/blitz_tools.py", > line 62, in blitz > ? ? **kw) > ? File "/opt/lib/python2.5/site-packages/scipy/weave/ > inline_tools.py", line 462, in compile_function > ? ? verbose=verbose, **kw) > ? File "/opt/lib/python2.5/site-packages/scipy/weave/ext_tools.py", > line 367, in compile > ? ? verbose = verbose, **kw) > ? File "/opt/lib/python2.5/site-packages/scipy/weave/build_tools.py", > line 272, in build_extension > ? ? setup(name = module_name, ext_modules = [ext],verbose=verb) > ? File "/opt/lib/python2.5/site-packages/numpy/distutils/core.py", > line 184, in setup > ? ? return old_setup(**new_attr) > ? File "/sw/lib/python2.5/distutils/core.py", line 168, in setup > ? ? raise SystemExit, "error: " + str(msg) > scipy.weave.build_tools.CompileError: error: Command "g++ -fno-strict- > aliasing -mno-fused-madd -DNDEBUG -g -fwrapv -O3 -Wall -I/opt/lib/ > python2.5/site-packages/scipy/weave -I/opt/lib/python2.5/site-packages/ > scipy/weave/scxx -I/opt/lib/python2.5/site-packages/scipy/weave/blitz - > I/opt/lib/python2.5/site-packages/numpy/core/include -I/sw/include/ > python2.5 -c /Users/gideon/.python25_compiled/ > sc_24bab5a447df6354e489345507f46f0d2.cpp -o /var/folders/yL/ > yLD5tRJiGWa7oyM6mWJUN++++TI/-Tmp-/gideon/python25_intermediate/ > compiler_eded75b75a17d5bc9afe97cc30cf08c0/Users/ > gideon/.python25_compiled/sc_24bab5a447df6354e489345507f46f0d2.o" > failed with exit status 1 > > Could it be because I'm using complex numbers? > > -gideon > from the test files for weave.blitz (test_blitz_tools.py) def _check_5point_avg_2d_complex_float(self): """ Note: THIS TEST is KNOWN TO FAIL ON GCC 3.x. It will not adversely affect 99.99 percent of weave result[1:-1,1:-1] = (b[1:-1,1:-1] + b[2:,1:-1] + b[:-2,1:-1] + b[1:-1,2:] + b[1:-1,:-2]) / 5. Note: THIS TEST is KNOWN TO FAIL ON GCC 3.x. The reason is that 5. is a double and b is a complex32. blitz doesn't know how to handle complex32/double. See: http://www.oonumerics.org/MailArchives/blitz-support/msg00541.php Unfortunately, the fix isn't trivial. Instead of fixing it, I prefer to wait until we replace blitz++ with Pat Miller's code that doesn't rely on blitz.. """ expr = "result[1:-1,1:-1] = (b[1:-1,1:-1] + b[2:,1:-1] + b[:-2,1:-1]" \ "+ b[1:-1,2:] + b[1:-1,:-2]) / 5." self.generic_2d(expr,complex64) your script runs if I don't mix complex and float, or switch to complex128, see below. I don't know if the results are what you intended or if this is really the source of your problem. Josef --------------------------------- import numpy as np from scipy import weave def linear_p(qp_hat, qm_hat, vg, kappa, n1, il, ik): linp = np.zeros(qp_hat.shape) + 0.j # in one case I got: TypeError: __neg__() takes exactly 2 arguments (1 given) expr = "linp = 0.0 - vg * qp_hat * il + kappa * ik * n1 * qm_hat" weave.blitz(expr, check_size=1) return linp qp_hat, qm_hat, vg, kappa, n1, il, ik = np.array([1+ 0.1j,1,1]),1,1,1,1,1,1 #print linear_p(qp_hat, qm_hat, vg, kappa, n1, il, ik) # results in CompileError: error: c = 1+ 0.1j qp_hat, qm_hat, vg, kappa, n1, il, ik = np.array([1+ 0.1j,1,1]),c,c,c,c,c,c print linear_p(qp_hat, qm_hat, vg, kappa, n1, il, ik) #[-0.0299+0.097j -0.0499+0.196j -0.0499+0.196j] c = 1.0 qp_hat, qm_hat, vg, kappa, n1, il, ik = np.array([1+ 0.1j,1,1],'complex128'),c,c,c,c,c,c print linear_p(qp_hat, qm_hat, vg, kappa, n1, il, ik) #[ 0.-0.1j 0.+0.j 0.+0.j ] From wcoulter at berkeley.edu Thu Apr 2 16:35:42 2009 From: wcoulter at berkeley.edu (William K. Coulter) Date: Thu, 02 Apr 2009 13:35:42 -0700 Subject: [SciPy-user] Getting the right numerical libraries for scipy Message-ID: <49D5219E.8060408@berkeley.edu> I wanted to optimize my python code to use the scipy.sparse library; however, benchmarking reveals that multiplying a sparse and dense matrix takes over 100 times as long as multiplying the equivalent two dense matrices. This suggests that I have at least one of the numerical libraries linked with python, but that the sparse ones aren't. I understand that numpy/distutils/system_info.py should contain some helpful information, but I can't read the output and don't know how to ensure that the libraries are there in a clean install. How could I check for the libraries and what would I do to install them if they are not there? I'm on a redhat system. Thanks much, -- Will PS: Here's an example of what I'm comparing (b is the dense representation of a sparse matrix; c is dense). counter2 > 100*counter1 for many iterations: bsp = sparse.csc_matrix(b) t1 = time.time() for i in range(n): np.dot(b,c) t2 = time.time() counter1 += t2-t1 t1 = time.time() for i in range(n): bsp.matmat(c) t2 = time.time() counter2 += t2-t1 -- William K. Coulter Graduate Student, Helen Wills Neuroscience Institute, UC Berkeley From ceputter at googlemail.com Thu Apr 2 17:52:41 2009 From: ceputter at googlemail.com (Christiaan Putter) Date: Thu, 2 Apr 2009 23:52:41 +0200 Subject: [SciPy-user] [Timeseries] Linux installation error Message-ID: <3a59ce2f0904021452s27622d1dt251e8b65f58bc44a@mail.gmail.com> Hi guys and girls, I just installed the timeseries module on linux after updating to the latest numpy 1.3 on python2.5. I followed the normal "python setup.py install" routine for installing both timeseries and numpy. There weren't any errors during building and installation. Though when I import the timeseries module I get an error: Traceback (most recent call last): File "/data/workspace/tests/timeseries.py", line 4, in import scikits.timeseries as ts File "/usr/local/lib/python2.5/site-packages/scikits.timeseries-0.91.0-py2.5-linux-x86_64.egg/scikits/timeseries/__init__.py", line 13, in import const File "/usr/local/lib/python2.5/site-packages/scikits.timeseries-0.91.0-py2.5-linux-x86_64.egg/scikits/timeseries/const.py", line 79, in from cseries import freq_constants ImportError: /usr/local/lib/python2.5/site-packages/scikits.timeseries-0.91.0-py2.5-linux-x86_64.egg/scikits/timeseries/cseries.so: undefined symbol: _time64 I'm obviously running a 64bit version of linux. Might this be the cause of the above error? Another question: Is numpy1.3 really necessary seeing as it's still only in testing? Most of my users use the enthought python distribution which still has an older numpy version bundled with it. Hope someone can help me out. Have a great day, Christian From pgmdevlist at gmail.com Thu Apr 2 18:10:36 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 2 Apr 2009 18:10:36 -0400 Subject: [SciPy-user] [Timeseries] Linux installation error In-Reply-To: <3a59ce2f0904021452s27622d1dt251e8b65f58bc44a@mail.gmail.com> References: <3a59ce2f0904021452s27622d1dt251e8b65f58bc44a@mail.gmail.com> Message-ID: <0E2BD09D-1EBA-48DD-89D8-AFFE993388C0@gmail.com> On Apr 2, 2009, at 5:52 PM, Christiaan Putter wrote: > Hi guys and girls, > > I just installed the timeseries module on linux after updating to the > latest numpy 1.3 on python2.5. Ah, an early adopter who failed into our trap... We detected a pb in the latest sources of scikits.timeseries. I'll send you a proper dist off-list, the files should be uploaded on sourceforge in the next 24h. > Another question: Is numpy1.3 really necessary seeing as it's still > only in testing? Most of my users use the enthought python > distribution which still has an older numpy version bundled with it. Oh yes. numpy 1.3 improves supports for structured arrays in numpy.ma and introduces numpy.lib.io.genfromtxt, two new features scikits.timeseries make extensive use of. That's basically why we didn't make any official release announcement yet: we're waiting for numpy 1.3 to be released first. We're just getting ready, with relative success as you have unfortunately experienced. In any case, please accept all our sincere apologies for any inconvenience. P. From timmichelsen at gmx-topmail.de Thu Apr 2 18:19:05 2009 From: timmichelsen at gmx-topmail.de (Tim Michelsen) Date: Fri, 03 Apr 2009 00:19:05 +0200 Subject: [SciPy-user] [Timeseries] Linux installation error In-Reply-To: <0E2BD09D-1EBA-48DD-89D8-AFFE993388C0@gmail.com> References: <3a59ce2f0904021452s27622d1dt251e8b65f58bc44a@mail.gmail.com> <0E2BD09D-1EBA-48DD-89D8-AFFE993388C0@gmail.com> Message-ID: > In any case, please accept all our sincere apologies for any > inconvenience. No need for excuse. Indeed, the newly organised documentation with its logo is just impressive. You may still include the recipes I sent... I am looking forward into using easy_install timseries after numpy is out. Thanks for all your efforts! Timmie. From pgmdevlist at gmail.com Thu Apr 2 18:28:09 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 2 Apr 2009 18:28:09 -0400 Subject: [SciPy-user] [Timeseries] Linux installation error In-Reply-To: References: <3a59ce2f0904021452s27622d1dt251e8b65f58bc44a@mail.gmail.com> <0E2BD09D-1EBA-48DD-89D8-AFFE993388C0@gmail.com> Message-ID: <7A4EF6BC-F253-41D4-8CBD-263F95A2D017@gmail.com> On Apr 2, 2009, at 6:19 PM, Tim Michelsen wrote: >> In any case, please accept all our sincere apologies for any >> inconvenience. > No need for excuse. > > Indeed, the newly organised documentation with its logo is just > impressive. Thank Matt, he did a really great job. > > You may still include the recipes I sent... That's on our todo list, worry not. We intend to add an Examples section to the doc. > I am looking forward into using easy_install timseries after numpy > is out. Hopefully that'll work. We still have some ironing to do here and there... > Thanks for all your efforts! and thanks a lot for your feeback! From Chris.Barker at noaa.gov Thu Apr 2 18:43:49 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 02 Apr 2009 15:43:49 -0700 Subject: [SciPy-user] [Timeseries] Linux installation error In-Reply-To: <7A4EF6BC-F253-41D4-8CBD-263F95A2D017@gmail.com> References: <3a59ce2f0904021452s27622d1dt251e8b65f58bc44a@mail.gmail.com> <0E2BD09D-1EBA-48DD-89D8-AFFE993388C0@gmail.com> <7A4EF6BC-F253-41D4-8CBD-263F95A2D017@gmail.com> Message-ID: <49D53FA5.9080308@noaa.gov> Pierre GM wrote: > On Apr 2, 2009, at 6:19 PM, Tim Michelsen wrote: >> Indeed, the newly organised documentation with its logo is just >> impressive. where to I find these impressive docs? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From pgmdevlist at gmail.com Thu Apr 2 18:54:54 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 2 Apr 2009 18:54:54 -0400 Subject: [SciPy-user] [Timeseries] Linux installation error In-Reply-To: <49D53FA5.9080308@noaa.gov> References: <3a59ce2f0904021452s27622d1dt251e8b65f58bc44a@mail.gmail.com> <0E2BD09D-1EBA-48DD-89D8-AFFE993388C0@gmail.com> <7A4EF6BC-F253-41D4-8CBD-263F95A2D017@gmail.com> <49D53FA5.9080308@noaa.gov> Message-ID: On Apr 2, 2009, at 6:43 PM, Christopher Barker wrote: > Pierre GM wrote: >> On Apr 2, 2009, at 6:19 PM, Tim Michelsen wrote: >>> Indeed, the newly organised documentation with its logo is just >>> impressive. > > where to I find these impressive docs? http://pytseries.sourceforge.net/ (or google "scikits timeseries" and feel lucky) From Chris.Barker at noaa.gov Thu Apr 2 19:36:49 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 02 Apr 2009 16:36:49 -0700 Subject: [SciPy-user] [Timeseries] Linux installation error In-Reply-To: References: <3a59ce2f0904021452s27622d1dt251e8b65f58bc44a@mail.gmail.com> <0E2BD09D-1EBA-48DD-89D8-AFFE993388C0@gmail.com> <7A4EF6BC-F253-41D4-8CBD-263F95A2D017@gmail.com> <49D53FA5.9080308@noaa.gov> Message-ID: <49D54C11.10105@noaa.gov> Pierre GM wrote: > On Apr 2, 2009, at 6:43 PM, Christopher Barker wrote: > >> Pierre GM wrote: > http://pytseries.sourceforge.net/ > > (or google "scikits timeseries" and feel lucky) for some odd reason, I google "time_series", and didn't find it. Yes the docs look great, and so does the package -- nice work! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From ceputter at googlemail.com Thu Apr 2 19:51:16 2009 From: ceputter at googlemail.com (Christiaan Putter) Date: Fri, 3 Apr 2009 01:51:16 +0200 Subject: [SciPy-user] [Timeseries] Linux installation error In-Reply-To: References: <3a59ce2f0904021452s27622d1dt251e8b65f58bc44a@mail.gmail.com> <0E2BD09D-1EBA-48DD-89D8-AFFE993388C0@gmail.com> <7A4EF6BC-F253-41D4-8CBD-263F95A2D017@gmail.com> <49D53FA5.9080308@noaa.gov> Message-ID: <3a59ce2f0904021651r664cd27chdb4a4f587a5b6de9@mail.gmail.com> Hi Pierre, Thanks for your swift reply, I'll test the code tomorrow. I'm looking forward to using your timeseries module. I'm writing a finance app using Enthought's tool suite and it seems timeseries will come in quite handy. Up until now I've been using normal numpy arrays with pytables for storing actual historical data and postgres (with SQLAlchemy) for storing some 'higher' level information about stocks and the results of analysis done on said historical data. I've found it's a pretty good combination since hdf5 compression keeps the data size down to only a few hundred megs and SQLAlchemy is simply awesome for running queries. Pytables doesn't play well with threading though even though I'm using locks in any block of code that so much as sniffs at the hdf5 file (strangely though it's rather stable on linux but crashes horribly on windows without even the courtesy of a trace back). I'll test h5py some time next week and if it performs better (which they claim on their site :-) I'll see if I can cook something up similar to what you did for integrating timeseries and pytables. I'll send it along to you once it's usable. In case that doesn't work I'll probably resort to storing the historical data in sql as well. Can someone give me some pointers on how I would go about that perhaps? It's about 20 000 - 30 000 stocks with on average about a decades worth of daily data. Would I dump all of that into a single table? 20 000 tables? hdf5 certainly is much better suited for something like that... Hope everyone is having a great day. Regards, Christian 2009/4/3 Pierre GM : > > On Apr 2, 2009, at 6:43 PM, Christopher Barker wrote: > >> Pierre GM wrote: >>> On Apr 2, 2009, at 6:19 PM, Tim Michelsen wrote: >>>> Indeed, the newly organised documentation with its logo is just >>>> impressive. >> >> where to I find these impressive docs? > > http://pytseries.sourceforge.net/ > > (or google "scikits timeseries" and feel lucky) > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From ceputter at googlemail.com Thu Apr 2 20:21:32 2009 From: ceputter at googlemail.com (Christiaan Putter) Date: Fri, 3 Apr 2009 02:21:32 +0200 Subject: [SciPy-user] [Timeseries] Linux installation error In-Reply-To: <3a59ce2f0904021651r664cd27chdb4a4f587a5b6de9@mail.gmail.com> References: <3a59ce2f0904021452s27622d1dt251e8b65f58bc44a@mail.gmail.com> <0E2BD09D-1EBA-48DD-89D8-AFFE993388C0@gmail.com> <7A4EF6BC-F253-41D4-8CBD-263F95A2D017@gmail.com> <49D53FA5.9080308@noaa.gov> <3a59ce2f0904021651r664cd27chdb4a4f587a5b6de9@mail.gmail.com> Message-ID: <3a59ce2f0904021721i112b896v9b61f53b298d1e2a@mail.gmail.com> Ok, just tested the code you sent me and it works just fine (had to kill numpy 1.2 though) Some simple copy / pastable examples would be great to get new users (that's me) going :-) Quick question: Is there any possibility to use python's decimal data type? I saw you used it in your sql example on the database side, but I'm guessing numpy doesn't allow for this? I sometimes have problems with equality testing after several divisions / multiplications, so at the moment I revert to the "is kind of" instead of the "is equal to" approach of comparison... 2009/4/3 Christiaan Putter : > Hi Pierre, > > Thanks for your swift reply, I'll test the code tomorrow. > > I'm looking forward to using your timeseries module. ?I'm writing a > finance app using Enthought's tool suite and it seems timeseries will > come in quite handy. ?Up until now I've been using normal numpy arrays > with pytables for storing actual historical data and postgres (with > SQLAlchemy) for storing some 'higher' level information about stocks > and the results of analysis done on said historical data. ?I've found > it's a pretty good combination since hdf5 compression keeps the data > size down to only a few hundred megs and SQLAlchemy is simply awesome > for running queries. > > Pytables doesn't play well with threading though even though I'm using > locks in any block of code that so much as sniffs at the hdf5 file > (strangely though it's rather stable on linux but crashes horribly on > windows without even the courtesy of a trace back). ?I'll test h5py > some time next week and if it performs better (which they claim on > their site :-) I'll see if I can cook something up similar to what you > did for integrating timeseries and pytables. ?I'll send it along to > you once it's usable. > > In case that doesn't work I'll probably resort to storing the > historical data in sql as well. ?Can someone give me some pointers on > how I would go about that perhaps? ?It's about 20 000 - 30 000 stocks > with on average about a decades worth of daily data. ?Would I dump all > of that into a single table? ?20 000 tables? ?hdf5 certainly is much > better suited for something like that... > > Hope everyone is having a great day. > > Regards, > Christian > > > > 2009/4/3 Pierre GM : >> >> On Apr 2, 2009, at 6:43 PM, Christopher Barker wrote: >> >>> Pierre GM wrote: >>>> On Apr 2, 2009, at 6:19 PM, Tim Michelsen wrote: >>>>> Indeed, the newly organised documentation with its logo is just >>>>> impressive. >>> >>> where to I find these impressive docs? >> >> http://pytseries.sourceforge.net/ >> >> (or google "scikits timeseries" and feel lucky) >> _______________________________________________ >> SciPy-user mailing list >> SciPy-user at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > From xiangyuding at gmail.com Thu Apr 2 20:51:36 2009 From: xiangyuding at gmail.com (Xiangyu Ding) Date: Thu, 2 Apr 2009 19:51:36 -0500 Subject: [SciPy-user] quesiton about integrate.odeint Message-ID: <6ed40ad40904021751x70154ba7w386b4e82149b5b7b@mail.gmail.com> Hi Sorry to bother, I just started using Scipy for research and so far I'm pretty impress with how fast scipy is compare to matlab. I just had a quick question about odeint. When I try to integrate a large ode system, like dy/dt = A*y, where A is a very large matrix say like 70000 by 70000 sparse matrix, the odeint function seem to breakdown, so I was wondering if that was a bug or is the function simply not made to handle function of that size? Thanks a lot, Xiangyu Ding From pgmdevlist at gmail.com Thu Apr 2 21:04:35 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 2 Apr 2009 21:04:35 -0400 Subject: [SciPy-user] [Timeseries] Linux installation error In-Reply-To: <3a59ce2f0904021721i112b896v9b61f53b298d1e2a@mail.gmail.com> References: <3a59ce2f0904021452s27622d1dt251e8b65f58bc44a@mail.gmail.com> <0E2BD09D-1EBA-48DD-89D8-AFFE993388C0@gmail.com> <7A4EF6BC-F253-41D4-8CBD-263F95A2D017@gmail.com> <49D53FA5.9080308@noaa.gov> <3a59ce2f0904021651r664cd27chdb4a4f587a5b6de9@mail.gmail.com> <3a59ce2f0904021721i112b896v9b61f53b298d1e2a@mail.gmail.com> Message-ID: <7206B522-6596-4D76-9CB8-50F2A26C9A15@gmail.com> On Apr 2, 2009, at 8:21 PM, Christiaan Putter wrote: > Ok, just tested the code you sent me and it works just fine (had to > kill numpy 1.2 though) Great !! > Some simple copy / pastable examples would be great to get new users > (that's me) going :-) FYI, you can find some examples of application at http://hydroclimpy.sourceforge.net/ . That's a package of extensions for timeseries focused on environmental series. That should get you started. Once again, we'll soon put an Example section online. For financial applications, check with Matt Knox, the co-author of the package. I'm the hydrologist of the duo. > Quick question: > > Is there any possibility to use python's decimal data type? I saw you > used it in your sql example on the database side, but I'm guessing > numpy doesn't allow for this? I don't think that numpy can interact w/ Decimal, but I never actually tried myself. > I sometimes have problems with equality > testing after several divisions / multiplications, so at the moment I > revert to the "is kind of" instead of the "is equal to" approach of > comparison... You can use assert_almost_equal from the testing modules. > I'll test h5py > some time next week and if it performs better (which they claim on > their site :-) I'll see if I can cook something up similar to what you > did for integrating timeseries and pytables. I'll send it along to > you once it's usable. Please do, it's always useful indeed. > > > > > 2009/4/3 Christiaan Putter : >> Hi Pierre, >> >> Thanks for your swift reply, I'll test the code tomorrow. >> >> I'm looking forward to using your timeseries module. I'm writing a >> finance app using Enthought's tool suite and it seems timeseries will >> come in quite handy. Up until now I've been using normal numpy >> arrays >> with pytables for storing actual historical data and postgres (with >> SQLAlchemy) for storing some 'higher' level information about stocks >> and the results of analysis done on said historical data. I've found >> it's a pretty good combination since hdf5 compression keeps the data >> size down to only a few hundred megs and SQLAlchemy is simply awesome >> for running queries. >> >> Pytables doesn't play well with threading though even though I'm >> using >> locks in any block of code that so much as sniffs at the hdf5 file >> (strangely though it's rather stable on linux but crashes horribly on >> windows without even the courtesy of a trace back). I'll test h5py >> some time next week and if it performs better (which they claim on >> their site :-) I'll see if I can cook something up similar to what >> you >> did for integrating timeseries and pytables. I'll send it along to >> you once it's usable. >> >> In case that doesn't work I'll probably resort to storing the >> historical data in sql as well. Can someone give me some pointers on >> how I would go about that perhaps? It's about 20 000 - 30 000 stocks >> with on average about a decades worth of daily data. Would I dump >> all >> of that into a single table? 20 000 tables? hdf5 certainly is much >> better suited for something like that... >> >> Hope everyone is having a great day. >> >> Regards, >> Christian >> >> >> >> 2009/4/3 Pierre GM : >>> >>> On Apr 2, 2009, at 6:43 PM, Christopher Barker wrote: >>> >>>> Pierre GM wrote: >>>>> On Apr 2, 2009, at 6:19 PM, Tim Michelsen wrote: >>>>>> Indeed, the newly organised documentation with its logo is just >>>>>> impressive. >>>> >>>> where to I find these impressive docs? >>> >>> http://pytseries.sourceforge.net/ >>> >>> (or google "scikits timeseries" and feel lucky) >>> _______________________________________________ >>> SciPy-user mailing list >>> SciPy-user at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >> > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From josef.pktd at gmail.com Thu Apr 2 21:14:34 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 2 Apr 2009 21:14:34 -0400 Subject: [SciPy-user] [Timeseries] Linux installation error In-Reply-To: <3a59ce2f0904021721i112b896v9b61f53b298d1e2a@mail.gmail.com> References: <3a59ce2f0904021452s27622d1dt251e8b65f58bc44a@mail.gmail.com> <0E2BD09D-1EBA-48DD-89D8-AFFE993388C0@gmail.com> <7A4EF6BC-F253-41D4-8CBD-263F95A2D017@gmail.com> <49D53FA5.9080308@noaa.gov> <3a59ce2f0904021651r664cd27chdb4a4f587a5b6de9@mail.gmail.com> <3a59ce2f0904021721i112b896v9b61f53b298d1e2a@mail.gmail.com> Message-ID: <1cd32cbb0904021814l4da58700k36799f29b04da05f@mail.gmail.com> On Thu, Apr 2, 2009 at 8:21 PM, Christiaan Putter wrote: > Ok, just tested the code you sent me and it works just fine ?(had to > kill numpy 1.2 though) > > Some simple copy / pastable examples would be great to get new users > (that's me) going :-) > > Quick question: > > Is there any possibility to use python's decimal data type? ?I saw you > used it in your sql example on the database side, but I'm guessing > numpy doesn't allow for this? ?I sometimes have problems with equality > testing after several divisions / multiplications, so at the moment I > revert to the "is kind of" instead of the "is equal to" approach of > comparison... > for floating point comparison it is better to use something like numpy.allclose(a, b, rtol=1.0000000000000001e-05, atol=1e-08) numpy.ma.allclose(a, b, masked_equal=True, rtol=1.0000000000000001e-05, atol=1e-08, fill_value=None) Returns True if two arrays are element-wise equal within a tolerance. absolute(a - b) <= (atol + rtol * absolute(b)) instead of equal, Josef From mattknox.ca at gmail.com Thu Apr 2 21:56:19 2009 From: mattknox.ca at gmail.com (Matt Knox) Date: Fri, 3 Apr 2009 01:56:19 +0000 (UTC) Subject: [SciPy-user] [Timeseries] Linux installation error References: <3a59ce2f0904021452s27622d1dt251e8b65f58bc44a@mail.gmail.com> <0E2BD09D-1EBA-48DD-89D8-AFFE993388C0@gmail.com> <7A4EF6BC-F253-41D4-8CBD-263F95A2D017@gmail.com> <49D53FA5.9080308@noaa.gov> <3a59ce2f0904021651r664cd27chdb4a4f587a5b6de9@mail.gmail.com> <3a59ce2f0904021721i112b896v9b61f53b298d1e2a@mail.gmail.com> Message-ID: >> In case that doesn't work I'll probably resort to storing the >> historical data in sql as well. Can someone give me some pointers on >> how I would go about that perhaps? It's about 20 000 - 30 000 stocks >> with on average about a decades worth of daily data. Would I dump all >> of that into a single table? 20 000 tables? hdf5 certainly is much >> better suited for something like that... in terms of raw throughput on reading/writing simple stock price history, yes HDF5 will be better, but if you need any kind of concurrent access or plan to throw some stuff up on a web-site, you will quickly find that the robustness of a modern relational database system will outweigh the performance benefits of a pure HDF5 solution. You could look at a hybrid solution, but the complexity often would not be worth it. I think HDF5 and similar types of storage are great for research projects and ad-hoc analysis, but if you are talking about a large scale production system it is going to be hard to beat a modern relational database. And you may be surprised by the performance you get from a modern relational db running on modern hardware. These systems are designed to handle LOTS of data. Anyway, your question really comes down to database design, and I would highly recommend you do some introductory reading on the basics of table design and such. One thing with relational dbs is that it is generally very hard to change the schema (ie. table structures) once your application gets to a certain size so you really need to plan an overall architecture ahead of time. Things like HDF5 allow for a little bit more of a care free approach. Also, having managed a large equity database before, I can say that it is not something I can adequately describe how to do well in a brief email on a mailing list. But I will say this... a typical setup would have a "master security" table (which may actually take several tables to properly describe the securities) which would have an integer id for every security and map it to various identifiers like cusip's, isin's, ticker's, etc. You will also need to account for changing tickers. To store your actual data (lets just say it is price data for now), you could have a table with 3 fields: [security_id] [int], [date] [datetime], [price] [decimal](12,5) The natural primary key here would be ([security_id], [date]) I won't get into the topic of "surrogate keys", but you may want to google that too. You would probably have a foreign key for the security_id field referencing your master security table -- although maybe not depending on performance considerations. Then there are things like indexes to consider to optimize the performance for your usage patterns. And you definitely DO NOT want 20,000 tables to store 20,000 stocks. You'll also need to think about how you handle corporate actions in your db like stock splits, mergers, etc. If you mess up on the design of how to do this, you will be in for a world of pain as far as maintenance of the database. Now as for reading your data into Python, you will find numpy arrays using compound data types work quite nicely. I often query out a big chunk of data, store it in a numpy array with a compound data type for caching purposes, then filter that array to get the specific chunks of data I need (eg. read 100 stocks worth of data at a time into a single array, then filter that array after rather than hitting the db one time for each stock). This is something I'll probably add to the database examples in the documentation at some point. >> Is there any possibility to use python's decimal data type? I saw you >> used it in your sql example on the database side, but I'm guessing >> numpy doesn't allow for this? I sometimes have problems with equality >> testing after several divisions / multiplications, so at the moment I >> revert to the "is kind of" instead of the "is equal to" approach of >> comparison... Numpy does not support a decimal type. I agree it would be a really nice addition, but I can generally live without it and probably won't have the motivation to contribute that to the numpy community any time soon. You can use an object array to store decimal values, but that is not recommended. You should always just cast the decimal values to floats prior to pulling them into Python unless you are talking about some accounting system or something where that level of accuracy matters. And I would agree with Josef's comments with regards to the approach for equality checks. From wkcochran at gmail.com Thu Apr 2 23:02:25 2009 From: wkcochran at gmail.com (Bill Cochran) Date: Fri, 3 Apr 2009 03:02:25 +0000 (UTC) Subject: [SciPy-user] Obtaining factors of sparse matrices Message-ID: I am new to scipy. I would like to compare fill-in between a standard GEPP and one that I have developed to schedule pivoting in parallel codes. My matrices are large and sparse. In MATLAB code, I need this: [L,U] = lu ( A ) nnz(U) I have looked at the scipy documentation and I believe the function I am looking for is sparse.linalg.splu (...). I do not know what permc_spec means. Neither I nor Google know what a "factored_lu" object is. And I don't understand the instruction: "See scipy.sparse.linalg.dsolve._superlu.dgstrf for more info." Neither does Google. Any help would be appreciated. I am kind of crunched for time and thought I'd see if scipy lived up to the hype. Please help scipy live up to the hype. From josef.pktd at gmail.com Fri Apr 3 00:15:55 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 3 Apr 2009 00:15:55 -0400 Subject: [SciPy-user] Obtaining factors of sparse matrices In-Reply-To: References: Message-ID: <1cd32cbb0904022115s3b016905o3428fd9ccc8769d2@mail.gmail.com> On Thu, Apr 2, 2009 at 11:02 PM, Bill Cochran wrote: > I am new to scipy. ?I would like to compare fill-in between a standard GEPP and > one that I have developed to schedule pivoting in parallel codes. ?My matrices > are large and sparse. ?In MATLAB code, I need this: > > [L,U] = lu ( A ) > nnz(U) > > I have looked at the scipy documentation and I believe the function I am looking > for is sparse.linalg.splu (...). ?I do not know what permc_spec means. ?Neither > I nor Google know what a "factored_lu" object is. ?And I don't understand the > instruction: ?"See scipy.sparse.linalg.dsolve._superlu.dgstrf for more info." > Neither does Google. I'm just looking up the source, I don't know this part of scipy: the above means that the only documentation of the details is in the c source e.g. http://projects.scipy.org/scipy/browser/trunk/scipy/sparse/linalg/dsolve/_dsuperlumodule.c#L124 144 permc_spec specifies the matrix ordering used for the factorization\n\ 145 0: natural ordering\n\ 146 1: MMD applied to the structure of A^T * A\n\ 147 2: MMD applied to the structure of A^T + A\n\ 148 3: COLAMD, approximate minimum degree column ordering\n\ 149 (default: 2)\n\ and http://projects.scipy.org/scipy/browser/trunk/scipy/sparse/linalg/dsolve/SuperLU/SRC/dgstrf.c#L25 splu is just a thin wrapper around dgrstrf (for double), see http://projects.scipy.org/scipy/browser/trunk/scipy/sparse/linalg/dsolve/linsolve.py#L103 if umfpack (scikits) is installed, than LU is calculated using umfpack That's browsing the source, for more info you need someone who knows this part. The return type of functions, I usually just inspect for an example dir(result), ... Josef > > Any help would be appreciated. ?I am kind of crunched for time and thought I'd > see if scipy lived up to the hype. ?Please help scipy live up to the hype. > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From josef.pktd at gmail.com Fri Apr 3 00:24:45 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 3 Apr 2009 00:24:45 -0400 Subject: [SciPy-user] Obtaining factors of sparse matrices In-Reply-To: <1cd32cbb0904022115s3b016905o3428fd9ccc8769d2@mail.gmail.com> References: <1cd32cbb0904022115s3b016905o3428fd9ccc8769d2@mail.gmail.com> Message-ID: <1cd32cbb0904022124x51be73a8m8c7b52f0c8aab4ea@mail.gmail.com> On Fri, Apr 3, 2009 at 12:15 AM, wrote: > On Thu, Apr 2, 2009 at 11:02 PM, Bill Cochran wrote: >> I am new to scipy. ?I would like to compare fill-in between a standard GEPP and >> one that I have developed to schedule pivoting in parallel codes. ?My matrices >> are large and sparse. ?In MATLAB code, I need this: >> >> [L,U] = lu ( A ) >> nnz(U) >> >> I have looked at the scipy documentation and I believe the function I am looking >> for is sparse.linalg.splu (...). ?I do not know what permc_spec means. ?Neither >> I nor Google know what a "factored_lu" object is. ?And I don't understand the >> instruction: ?"See scipy.sparse.linalg.dsolve._superlu.dgstrf for more info." >> Neither does Google. > > I'm just looking up the source, I don't know this part of scipy: > > the above means that the only documentation of the details is in the c source > > e.g. > http://projects.scipy.org/scipy/browser/trunk/scipy/sparse/linalg/dsolve/_dsuperlumodule.c#L124 > 144 ? ? permc_spec ? ? ? ? ?specifies the matrix ordering used for the > factorization\n\ > 145 ? ? ? ? ? ? ? ? ? ? ? ? 0: natural ordering\n\ > 146 ? ? ? ? ? ? ? ? ? ? ? ? 1: MMD applied to the structure of A^T * A\n\ > 147 ? ? ? ? ? ? ? ? ? ? ? ? 2: MMD applied to the structure of A^T + A\n\ > 148 ? ? ? ? ? ? ? ? ? ? ? ? 3: COLAMD, approximate minimum degree column ordering\n\ > 149 ? ? ? ? ? ? ? ? ? ? ? ? (default: 2)\n\ > > and > http://projects.scipy.org/scipy/browser/trunk/scipy/sparse/linalg/dsolve/SuperLU/SRC/dgstrf.c#L25 > > splu is just a thin wrapper around dgrstrf (for double), see > http://projects.scipy.org/scipy/browser/trunk/scipy/sparse/linalg/dsolve/linsolve.py#L103 > > if umfpack (scikits) is installed, than LU is calculated using umfpack > > That's browsing the source, for more info you need someone who knows > this part. The return type of functions, I usually just inspect for an > example dir(result), ... > > Josef > >> >> Any help would be appreciated. ?I am kind of crunched for time and thought I'd >> see if scipy lived up to the hype. ?Please help scipy live up to the hype. >> >> _______________________________________________ just one more reference, I think this part of sparse linalg came from pysparse: http://pysparse.sourceforge.net/fact.html#module-pysparseSuperLU Josef From faltet at pytables.org Fri Apr 3 02:44:58 2009 From: faltet at pytables.org (Francesc Alted) Date: Fri, 3 Apr 2009 08:44:58 +0200 Subject: [SciPy-user] [Timeseries] Linux installation error In-Reply-To: <3a59ce2f0904021651r664cd27chdb4a4f587a5b6de9@mail.gmail.com> References: <3a59ce2f0904021452s27622d1dt251e8b65f58bc44a@mail.gmail.com> <3a59ce2f0904021651r664cd27chdb4a4f587a5b6de9@mail.gmail.com> Message-ID: <200904030844.58484.faltet@pytables.org> A Friday 03 April 2009, Christiaan Putter escrigu?: > > Pytables doesn't play well with threading though even though I'm > using locks in any block of code that so much as sniffs at the hdf5 > file (strangely though it's rather stable on linux but crashes > horribly on windows without even the courtesy of a trace back). Please could you be more explicit on the sort of problems that you are experiencing? I'd be glad to discuss them (in the PyTables list, preferably) and see if they can be addressed in some way or another. Cheers, -- Francesc Alted "One would expect people to feel threatened by the 'giant brains or machines that think'. In fact, the fightening computer becomes less frightening if it is used only to simulate a familiar noncomputer." -- Edsger W. Dykstra "On the cruelty of really teaching computer science" From pav at iki.fi Fri Apr 3 04:23:37 2009 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 3 Apr 2009 08:23:37 +0000 (UTC) Subject: [SciPy-user] quesiton about integrate.odeint References: <6ed40ad40904021751x70154ba7w386b4e82149b5b7b@mail.gmail.com> Message-ID: Thu, 02 Apr 2009 19:51:36 -0500, Xiangyu Ding kirjoitti: [clip] > I just had a quick question about odeint. When I try to integrate a > large ode system, like dy/dt = A*y, where A is a very large matrix say > like 70000 by 70000 sparse matrix, the odeint function seem to > breakdown, so I was wondering if that was a bug or is the function > simply not made to handle function of that size? The method used by odeint tries to automatically detect if the problem is stiff, so it always allocates a dense Jacobian. Since the Jacobian here is 70000x70000, it does not fit into memory. You can try using `ode` instead of `odeint`, it doesn't seem to have this particular restriction. -- Pauli Virtanen From stefan at sun.ac.za Fri Apr 3 07:13:17 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 3 Apr 2009 13:13:17 +0200 Subject: [SciPy-user] Getting the right numerical libraries for scipy In-Reply-To: <49D5219E.8060408@berkeley.edu> References: <49D5219E.8060408@berkeley.edu> Message-ID: <9457e7c80904030413l2dd46fcagc2c1bdb7bb201c46@mail.gmail.com> 2009/4/2 William K. Coulter : > > I wanted to optimize my python code to use the scipy.sparse library; > however, benchmarking reveals that multiplying a sparse and dense matrix > takes over 100 times as long as multiplying the equivalent two dense > matrices. I did some benchmarks now (see attached), and I see the same behaviour: http://mentat.za.net/refer/bench_sparse.png This seems odd. Regards St?fan -------------- next part -------------- A non-text attachment was scrubbed... Name: bench_sparse.py Type: text/x-python-script Size: 1131 bytes Desc: not available URL: From david at ar.media.kyoto-u.ac.jp Fri Apr 3 07:05:26 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 03 Apr 2009 20:05:26 +0900 Subject: [SciPy-user] Getting the right numerical libraries for scipy In-Reply-To: <9457e7c80904030413l2dd46fcagc2c1bdb7bb201c46@mail.gmail.com> References: <49D5219E.8060408@berkeley.edu> <9457e7c80904030413l2dd46fcagc2c1bdb7bb201c46@mail.gmail.com> Message-ID: <49D5ED76.9010206@ar.media.kyoto-u.ac.jp> St?fan van der Walt wrote: > 2009/4/2 William K. Coulter : > >> I wanted to optimize my python code to use the scipy.sparse library; >> however, benchmarking reveals that multiplying a sparse and dense matrix >> takes over 100 times as long as multiplying the equivalent two dense >> matrices. >> > > I did some benchmarks now (see attached), and I see the same behaviour: > > http://mentat.za.net/refer/bench_sparse.png > Isn't this expected ? I thought that for sparse matrix to be useful, the density had to be much lower than the figures you used ? Maybe a more useful benchmark would be the dense/sparse ratio as a function of density for a given size, cheers, David From sebastian.walter at gmail.com Fri Apr 3 07:24:08 2009 From: sebastian.walter at gmail.com (Sebastian Walter) Date: Fri, 3 Apr 2009 13:24:08 +0200 Subject: [SciPy-user] Getting the right numerical libraries for scipy In-Reply-To: <49D5ED76.9010206@ar.media.kyoto-u.ac.jp> References: <49D5219E.8060408@berkeley.edu> <9457e7c80904030413l2dd46fcagc2c1bdb7bb201c46@mail.gmail.com> <49D5ED76.9010206@ar.media.kyoto-u.ac.jp> Message-ID: i agree with david. Sparse matrix packages are for a density of O(N) for a (N,N) matrix. On Fri, Apr 3, 2009 at 1:05 PM, David Cournapeau wrote: > St?fan van der Walt wrote: >> 2009/4/2 William K. Coulter : >> >>> I wanted to optimize my python code to use the scipy.sparse library; >>> however, benchmarking reveals that multiplying a sparse and dense matrix >>> takes over 100 times as long as multiplying the equivalent two dense >>> matrices. >>> >> >> I did some benchmarks now (see attached), and I see the same behaviour: >> >> http://mentat.za.net/refer/bench_sparse.png >> > > Isn't this expected ? I thought that for sparse matrix to be useful, the > density had to be much lower than the figures you used ? > > Maybe a more useful benchmark would be the dense/sparse ratio as a > function of density for a given size, > > cheers, > > David > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From josef.pktd at gmail.com Fri Apr 3 09:53:39 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 3 Apr 2009 09:53:39 -0400 Subject: [SciPy-user] Getting the right numerical libraries for scipy In-Reply-To: References: <49D5219E.8060408@berkeley.edu> <9457e7c80904030413l2dd46fcagc2c1bdb7bb201c46@mail.gmail.com> <49D5ED76.9010206@ar.media.kyoto-u.ac.jp> Message-ID: <1cd32cbb0904030653u12e5869ia804f82fda799abc@mail.gmail.com> On Fri, Apr 3, 2009 at 7:24 AM, Sebastian Walter wrote: > i agree with david. > Sparse matrix packages are for a density of O(N) for a (N,N) matrix. > > > > On Fri, Apr 3, 2009 at 1:05 PM, David Cournapeau > wrote: >> St?fan van der Walt wrote: >>> 2009/4/2 William K. Coulter : >>> >>>> I wanted to optimize my python code to use the scipy.sparse library; >>>> however, benchmarking reveals that multiplying a sparse and dense matrix >>>> takes over 100 times as long as multiplying the equivalent two dense >>>> matrices. >>>> >>> >>> I did some benchmarks now (see attached), and I see the same behaviour: >>> >>> http://mentat.za.net/refer/bench_sparse.png >>> >> >> Isn't this expected ? I thought that for sparse matrix to be useful, the >> density had to be much lower than the figures you used ? >> >> Maybe a more useful benchmark would be the dense/sparse ratio as a >> function of density for a given size, >> >> cheers, >> >> David in my examples the break-point is around 55% of non-zero elements at random positions. But even with 100% density sparse dot only takes about twice the time of dense dot >>> b.shape (1000, 1000) >>> c.shape (1000, 3) 550763 55.0763 percent non-zero elements np.dot(b,c) bsp.matmat(c) bsp*c print counter1/float(n)*1000, counter2/float(n)*1000, counter3/float(n)*1000 9.21999931335 9.22000169754 9.6799993515 np.max(np.abs(bcd - bcsp)) is 7.27595761418e-012 1000000 100.0 percent non-zero elements 9.21999931335 16.400001049 16.7199993134 maxabsdiff is 2.43289832724e-011 when b is also (m,m) then the breakpoint seems to be araund 60%. But I thought the advantage of sparse is also memory usage, if I increase the matrix size much higher my computer starts to swap or I get out of memory errors. Josef From stefan at sun.ac.za Fri Apr 3 09:55:19 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 3 Apr 2009 15:55:19 +0200 Subject: [SciPy-user] Getting the right numerical libraries for scipy In-Reply-To: <49D5ED76.9010206@ar.media.kyoto-u.ac.jp> References: <49D5219E.8060408@berkeley.edu> <9457e7c80904030413l2dd46fcagc2c1bdb7bb201c46@mail.gmail.com> <49D5ED76.9010206@ar.media.kyoto-u.ac.jp> Message-ID: <9457e7c80904030655g12748c91h4c6041fcf71f28fb@mail.gmail.com> 2009/4/3 David Cournapeau : > Maybe a more useful benchmark would be the dense/sparse ratio as a > function of density for a given size, http://mentat.za.net/refer/bench_ratio.png Cheers St?fan -------------- next part -------------- A non-text attachment was scrubbed... Name: bench_sparse.py Type: text/x-python-script Size: 1154 bytes Desc: not available URL: From david at ar.media.kyoto-u.ac.jp Fri Apr 3 10:09:24 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 03 Apr 2009 23:09:24 +0900 Subject: [SciPy-user] Getting the right numerical libraries for scipy In-Reply-To: <1cd32cbb0904030653u12e5869ia804f82fda799abc@mail.gmail.com> References: <49D5219E.8060408@berkeley.edu> <9457e7c80904030413l2dd46fcagc2c1bdb7bb201c46@mail.gmail.com> <49D5ED76.9010206@ar.media.kyoto-u.ac.jp> <1cd32cbb0904030653u12e5869ia804f82fda799abc@mail.gmail.com> Message-ID: <49D61894.10509@ar.media.kyoto-u.ac.jp> josef.pktd at gmail.com wrote: > > in my examples the break-point is around 55% of non-zero elements at > random positions. But even with 100% density sparse dot only takes > about twice the time of dense dot > I think it will depend on your dot implementation (does it uses atlas or other heavily optimized implementation). This is to be taken with a (big) grain of salt since I don't know much about sparse matrices, but if the distribution is purely random, then I can see how sparse matrices would be much slower than contiguous arrays. Memory access is often the bottleneck for simple FPU operations on big data, and random memory access just kills access performances (can be order of magnitude slower - a cache miss on modern CPU costs ~ 250 cycles). > But I thought the advantage of sparse is also memory usage, if I > increase the matrix size much higher my computer starts to swap or I > get out of memory errors. > I believe that's the biggest advantage in some areas. If you need to compute the SVD of a matrix with millions of rows and/or columns, you won't do it on 'normal' computers with plain matrices :) (there are some methods for recursive computation which do not need the whole matrix in memory, but that's not always available/desirable/possible), cheers, David From david at ar.media.kyoto-u.ac.jp Fri Apr 3 10:12:24 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 03 Apr 2009 23:12:24 +0900 Subject: [SciPy-user] Getting the right numerical libraries for scipy In-Reply-To: <49D61894.10509@ar.media.kyoto-u.ac.jp> References: <49D5219E.8060408@berkeley.edu> <9457e7c80904030413l2dd46fcagc2c1bdb7bb201c46@mail.gmail.com> <49D5ED76.9010206@ar.media.kyoto-u.ac.jp> <1cd32cbb0904030653u12e5869ia804f82fda799abc@mail.gmail.com> <49D61894.10509@ar.media.kyoto-u.ac.jp> Message-ID: <49D61948.6080601@ar.media.kyoto-u.ac.jp> David Cournapeau wrote: > > I think it will depend on your dot implementation (does it uses atlas or > other heavily optimized implementation). This is to be taken with a > (big) grain of salt since I don't know much about sparse matrices, but > if the distribution is purely random, then I can see how sparse matrices > would be much slower than contiguous arrays. Memory access is often the > bottleneck for simple FPU operations on big data, and random memory > access just kills access performances (can be order of magnitude slower > - a cache miss on modern CPU costs ~ 250 cycles). > Sorry, there lacks one sentence here: memory access patterns for sparse matrices are more difficult to optimize than plain arrays. cheers, David From josef.pktd at gmail.com Fri Apr 3 10:49:57 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 3 Apr 2009 10:49:57 -0400 Subject: [SciPy-user] Getting the right numerical libraries for scipy In-Reply-To: <49D61948.6080601@ar.media.kyoto-u.ac.jp> References: <49D5219E.8060408@berkeley.edu> <9457e7c80904030413l2dd46fcagc2c1bdb7bb201c46@mail.gmail.com> <49D5ED76.9010206@ar.media.kyoto-u.ac.jp> <1cd32cbb0904030653u12e5869ia804f82fda799abc@mail.gmail.com> <49D61894.10509@ar.media.kyoto-u.ac.jp> <49D61948.6080601@ar.media.kyoto-u.ac.jp> Message-ID: <1cd32cbb0904030749l18e8d8eetc6d1d7392d82b2e2@mail.gmail.com> On Fri, Apr 3, 2009 at 10:12 AM, David Cournapeau wrote: > David Cournapeau wrote: >> >> I think it will depend on your dot implementation (does it uses atlas or >> other heavily optimized implementation). This is to be taken with a >> (big) grain of salt since I don't know much about sparse matrices, but >> if the distribution is purely random, then I can see how sparse matrices >> would be much slower than contiguous arrays. Memory access is often the >> bottleneck for simple FPU operations on big data, and random memory >> access just kills access performances ?(can be order of magnitude slower >> - a cache miss on modern CPU costs ~ 250 cycles). I think, csr and csc matrices are row and column contiguous in the non-zero elements, so the position of the elements might not matter so much for random memory access. I'm surprised sparse dot only takes twice the time on my computer than np.dot (with atlas from 1.3.0b1 installer) for matrices larger than (500,500) with 100% density. William mentioned 100 times slower, and Stefan has more than 5 times slower. The only time, I tried to use sparse matrices for kernel ridge regression. When we use the "kernel trick" from machine learning, the kernel matrix is number of observation squared, which can get very large and might take a long time to invert or solve a linear matrix equation. On my "small" computer, I worried more about memory than speed. Josef From stefan at sun.ac.za Fri Apr 3 11:14:25 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 3 Apr 2009 17:14:25 +0200 Subject: [SciPy-user] Getting the right numerical libraries for scipy In-Reply-To: <1cd32cbb0904030749l18e8d8eetc6d1d7392d82b2e2@mail.gmail.com> References: <49D5219E.8060408@berkeley.edu> <9457e7c80904030413l2dd46fcagc2c1bdb7bb201c46@mail.gmail.com> <49D5ED76.9010206@ar.media.kyoto-u.ac.jp> <1cd32cbb0904030653u12e5869ia804f82fda799abc@mail.gmail.com> <49D61894.10509@ar.media.kyoto-u.ac.jp> <49D61948.6080601@ar.media.kyoto-u.ac.jp> <1cd32cbb0904030749l18e8d8eetc6d1d7392d82b2e2@mail.gmail.com> Message-ID: <9457e7c80904030814i71d1dd53sad48f8123eaa8d69@mail.gmail.com> 2009/4/3 : > I'm surprised sparse dot only takes twice the time on my computer than > np.dot (with atlas from 1.3.0b1 installer) for matrices larger than > (500,500) with 100% density. > William mentioned 100 times slower, and Stefan has more than 5 times slower. Looks more like 500x! Cheers St?fan From david at ar.media.kyoto-u.ac.jp Fri Apr 3 11:03:46 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 04 Apr 2009 00:03:46 +0900 Subject: [SciPy-user] Getting the right numerical libraries for scipy In-Reply-To: <1cd32cbb0904030749l18e8d8eetc6d1d7392d82b2e2@mail.gmail.com> References: <49D5219E.8060408@berkeley.edu> <9457e7c80904030413l2dd46fcagc2c1bdb7bb201c46@mail.gmail.com> <49D5ED76.9010206@ar.media.kyoto-u.ac.jp> <1cd32cbb0904030653u12e5869ia804f82fda799abc@mail.gmail.com> <49D61894.10509@ar.media.kyoto-u.ac.jp> <49D61948.6080601@ar.media.kyoto-u.ac.jp> <1cd32cbb0904030749l18e8d8eetc6d1d7392d82b2e2@mail.gmail.com> Message-ID: <49D62552.8090400@ar.media.kyoto-u.ac.jp> josef.pktd at gmail.com wrote: > On Fri, Apr 3, 2009 at 10:12 AM, David Cournapeau > wrote: > >> David Cournapeau wrote: >> >>> I think it will depend on your dot implementation (does it uses atlas or >>> other heavily optimized implementation). This is to be taken with a >>> (big) grain of salt since I don't know much about sparse matrices, but >>> if the distribution is purely random, then I can see how sparse matrices >>> would be much slower than contiguous arrays. Memory access is often the >>> bottleneck for simple FPU operations on big data, and random memory >>> access just kills access performances (can be order of magnitude slower >>> - a cache miss on modern CPU costs ~ 250 cycles). >>> > > I think, csr and csc matrices are row and column contiguous in the > non-zero elements, so the position of the elements might not matter so > much for random memory access. > Yes, they are contiguous - but the parts which are contiguous are not "homogeneous". For example, if you want to compute the matrix product of two matrices of size (n, m) and (m, k), knowing the shape is enough to split your computation to the best size for the fast kernels on a given machine. For sparse matrices, not so much: you need to read the indexes as well, and it becomes much more complex to optimize this. cheers, David From wcoulter at berkeley.edu Fri Apr 3 11:32:43 2009 From: wcoulter at berkeley.edu (William K. Coulter) Date: Fri, 03 Apr 2009 08:32:43 -0700 Subject: [SciPy-user] Getting the right numerical libraries for scipy In-Reply-To: <9457e7c80904030655g12748c91h4c6041fcf71f28fb@mail.gmail.com> References: <49D5219E.8060408@berkeley.edu> <9457e7c80904030413l2dd46fcagc2c1bdb7bb201c46@mail.gmail.com> <49D5ED76.9010206@ar.media.kyoto-u.ac.jp> <9457e7c80904030655g12748c91h4c6041fcf71f28fb@mail.gmail.com> Message-ID: <49D62C1B.4040907@berkeley.edu> Thanks all for your replies. While this benchmark is interesting, it is not the one that I was trying. I'm interested in computations where just *one* of the matrices is actually sparse. I'm looking at something more like the following in your script's language -- I'm guessing on the syntax for Timer: c = np.random.random((s,s)) # c is dense b = np.random.random((s, s)) b[b > d] = 0 # b sparse, but dense representation b_s = ss.csc_matrix(b) # b_s sparse, with sparse rep. timer_dense = Timer("np.dot(b,c)", "import __main__ as m") # test1 timer_sparse = Timer("b_s.matmat(c)", "import __main__ as m") # test2 Could you look at those curves? Would you not expect the multiplication using the sparse version of b (b_s) to be faster? On matlab, the equivalent to test2 runs far faster (20x) than test1, whereas in python, I'm seeing completely the opposite. Since there are questions about the dense/sparse ratio, I get speedup in matlab on using sparse representation for one matrix for ratios (d) of 0.5 and lower. Any ideas for the performance difference? -- Will From St?fan van der Walt on 4/3/2009 6.55: > 2009/4/3 David Cournapeau : >> Maybe a more useful benchmark would be the dense/sparse ratio as a >> function of density for a given size, > > http://mentat.za.net/refer/bench_ratio.png > > Cheers > St?fan > > > ------------------------------------------------------------------------ > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -- William K. Coulter wcoulter at berkeley.edu Graduate Student, Helen Wills Neuroscience Institute, UC Berkeley From david at ar.media.kyoto-u.ac.jp Fri Apr 3 11:18:18 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 04 Apr 2009 00:18:18 +0900 Subject: [SciPy-user] [ANN] Numpy 1.3.0rc2 Message-ID: <49D628BA.2000404@ar.media.kyoto-u.ac.jp> Hi, I am pleased to announce the release of the rc2 for numpy 1.3.0. I have decided to go for a rc2 instead of the release directly because of the serious mac os X issue. You can find source tarballs and installers for both Mac OS X and Windows on the sourceforge page: https://sourceforge.net/projects/numpy/ > The release note for the 1.3.0 release are below, the numpy developers. ========================= NumPy 1.3.0 Release Notes ========================= This minor includes numerous bug fixes, official python 2.6 support, and several new features such as generalized ufuncs. Highlights ========== Python 2.6 support ~~~~~~~~~~~~~~~~~~ Python 2.6 is now supported on all previously supported platforms, including windows. http://www.python.org/dev/peps/pep-0361/ Generalized ufuncs ~~~~~~~~~~~~~~~~~~ There is a general need for looping over not only functions on scalars but also over functions on vectors (or arrays), as explained on http://scipy.org/scipy/numpy/wiki/GeneralLoopingFunctions. We propose to realize this concept by generalizing the universal functions (ufuncs), and provide a C implementation that adds ~500 lines to the numpy code base. In current (specialized) ufuncs, the elementary function is limited to element-by-element operations, whereas the generalized version supports "sub-array" by "sub-array" operations. The Perl vector library PDL provides a similar functionality and its terms are re-used in the following. Each generalized ufunc has information associated with it that states what the "core" dimensionality of the inputs is, as well as the corresponding dimensionality of the outputs (the element-wise ufuncs have zero core dimensions). The list of the core dimensions for all arguments is called the "signature" of a ufunc. For example, the ufunc numpy.add has signature "(),()->()" defining two scalar inputs and one scalar output. Another example is (see the GeneralLoopingFunctions page) the function inner1d(a,b) with a signature of "(i),(i)->()". This applies the inner product along the last axis of each input, but keeps the remaining indices intact. For example, where a is of shape (3,5,N) and b is of shape (5,N), this will return an output of shape (3,5). The underlying elementary function is called 3*5 times. In the signature, we specify one core dimension "(i)" for each input and zero core dimensions "()" for the output, since it takes two 1-d arrays and returns a scalar. By using the same name "i", we specify that the two corresponding dimensions should be of the same size (or one of them is of size 1 and will be broadcasted). The dimensions beyond the core dimensions are called "loop" dimensions. In the above example, this corresponds to (3,5). The usual numpy "broadcasting" rules apply, where the signature determines how the dimensions of each input/output object are split into core and loop dimensions: While an input array has a smaller dimensionality than the corresponding number of core dimensions, 1's are pre-pended to its shape. The core dimensions are removed from all inputs and the remaining dimensions are broadcasted; defining the loop dimensions. The output is given by the loop dimensions plus the output core dimensions. Experimental Windows 64 bits support ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Numpy can now be built on windows 64 bits (amd64 only, not IA64), with both MS compilers and mingw-w64 compilers: This is *highly experimental*: DO NOT USE FOR PRODUCTION USE. See INSTALL.txt, Windows 64 bits section for more information on limitations and how to build it by yourself. New features ============ Formatting issues ~~~~~~~~~~~~~~~~~ Float formatting is now handled by numpy instead of the C runtime: this enables locale independent formatting, more robust fromstring and related methods. Special values (inf and nan) are also more consistent across platforms (nan vs IND/NaN, etc...), and more consistent with recent python formatting work (in 2.6 and later). Nan handling in max/min ~~~~~~~~~~~~~~~~~~~~~~~ The maximum/minimum ufuncs now reliably propagate nans. If one of the arguments is a nan, then nan is retured. This affects np.min/np.max, amin/amax and the array methods max/min. New ufuncs fmax and fmin have been added to deal with non-propagating nans. Nan handling in sign ~~~~~~~~~~~~~~~~~~~~ The ufunc sign now returns nan for the sign of anan. New ufuncs ~~~~~~~~~~ #. fmax - same as maximum for integer types and non-nan floats. Returns the non-nan argument if one argument is nan and returns nan if both arguments are nan. #. fmin - same as minimum for integer types and non-nan floats. Returns the non-nan argument if one argument is nan and returns nan if both arguments are nan. #. deg2rad - converts degrees to radians, same as the radians ufunc. #. rad2deg - converts radians to degrees, same as the degrees ufunc. #. log2 - base 2 logarithm. #. exp2 - base 2 exponential. #. trunc - truncate floats to nearest integer towards zero. #. logaddexp - add numbers stored as logarithms and return the logarithm of the result. #. logaddexp2 - add numbers stored as base 2 logarithms and return the base 2 logarithm of the result result. Masked arrays ~~~~~~~~~~~~~ Several new features and bug fixes, including: * structured arrays should now be fully supported by MaskedArray (r6463, r6324, r6305, r6300, r6294...) * Minor bug fixes (r6356, r6352, r6335, r6299, r6298) * Improved support for __iter__ (r6326) * made baseclass, sharedmask and hardmask accesible to the user (but read-only) * doc update gfortran support on windows ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Gfortran can now be used as a fortran compiler for numpy on windows, even when the C compiler is Visual Studio (VS 2005 and above; VS 2003 will NOT work). Gfortran + Visual studio does not work on windows 64 bits (but gcc + gfortran does). It is unclear whether it will be possible to use gfortran and visual studio at all on x64. Arch option for windows binary ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Automatic arch detection can now be bypassed from the command line for the superpack installed: numpy-1.3.0-superpack-win32.exe /arch=nosse will install a numpy which works on any x86, even if the running computer supports SSE set. Deprecated features =================== Histogram ~~~~~~~~~ The semantics of histogram has been modified to fix long-standing issues with outliers handling. The main changes concern #. the definition of the bin edges, now including the rightmost edge, and #. the handling of upper outliers, now ignored rather than tallied in the rightmost bin. The previous behavior is still accessible using `new=False`, but this is deprecated, and will be removed entirely in 1.4.0. Documentation changes ===================== A lot of documentation has been added. Both user guide and references can be built from sphinx. New C API ========= Multiarray API ~~~~~~~~~~~~~~ The following functions have been added to the multiarray C API: * PyArray_GetEndianness: to get runtime endianness Ufunc API ~~~~~~~~~~~~~~ The following functions have been added to the ufunc API: * PyUFunc_FromFuncAndDataAndSignature: to declare a more general ufunc (generalized ufunc). New defines ~~~~~~~~~~~ New public C defines are available for ARCH specific code through numpy/npy_cpu.h: * NPY_CPU_X86: x86 arch (32 bits) * NPY_CPU_AMD64: amd64 arch (x86_64, NOT Itanium) * NPY_CPU_PPC: 32 bits ppc * NPY_CPU_PPC64: 64 bits ppc * NPY_CPU_SPARC: 32 bits sparc * NPY_CPU_SPARC64: 64 bits sparc * NPY_CPU_S390: S390 * NPY_CPU_IA64: ia64 * NPY_CPU_PARISC: PARISC New macros for CPU endianness has been added as well (see internal changes below for details): * NPY_BYTE_ORDER: integer * NPY_LITTLE_ENDIAN/NPY_BIG_ENDIAN defines Those provide portable alternatives to glibc endian.h macros for platforms without it. Portable NAN, INFINITY, etc... ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ npy_math.h now makes available several portable macro to get NAN, INFINITY: * NPY_NAN: equivalent to NAN, which is a GNU extension * NPY_INFINITY: equivalent to C99 INFINITY * NPY_PZERO, NPY_NZERO: positive and negative zero respectively Corresponding single and extended precision macros are available as well. All references to NAN, or home-grown computation of NAN on the fly have been removed for consistency. Internal changes ================ numpy.core math configuration revamp ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This should make the porting to new platforms easier, and more robust. In particular, the configuration stage does not need to execute any code on the target platform, which is a first step toward cross-compilation. http://projects.scipy.org/numpy/browser/trunk/doc/neps/math_config_clean.txt umath refactor ~~~~~~~~~~~~~~ A lot of code cleanup for umath/ufunc code (charris). Improvements to build warnings ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Numpy can now build with -W -Wall without warnings http://projects.scipy.org/numpy/browser/trunk/doc/neps/warnfix.txt Separate core math library ~~~~~~~~~~~~~~~~~~~~~~~~~~ The core math functions (sin, cos, etc... for basic C types) have been put into a separate library; it acts as a compatibility layer, to support most C99 maths functions (real only for now). The library includes platform-specific fixes for various maths functions, such as using those versions should be more robust than using your platform functions directly. The API for existing functions is exactly the same as the C99 math functions API; the only difference is the npy prefix (npy_cos vs cos). The core library will be made available to any extension in 1.4.0. CPU arch detection ~~~~~~~~~~~~~~~~~~ npy_cpu.h defines numpy specific CPU defines, such as NPY_CPU_X86, etc... Those are portable across OS and toolchains, and set up when the header is parsed, so that they can be safely used even in the case of cross-compilation (the values is not set when numpy is built), or for multi-arch binaries (e.g. fat binaries on Max OS X). npy_endian.h defines numpy specific endianness defines, modeled on the glibc endian.h. NPY_BYTE_ORDER is equivalent to BYTE_ORDER, and one of NPY_LITTLE_ENDIAN or NPY_BIG_ENDIAN is defined. As for CPU archs, those are set when the header is parsed by the compiler, and as such can be used for cross-compilation and multi-arch binaries. Checksums ========= 55a2d2c9ba056133f61ad70e6445d2f9 release/installers/numpy-1.3.0rc2.tar.gz ea88cd161ac8b267bd85419dddde40a5 release/installers/numpy-1.3.0rc2.zip 77b5a1ed12f86353f7ab998e1ce7d4b9 release/installers/numpy-1.3.0rc2-win32-superpack-python2.5.exe 803425416d6e86ce5cc2704aab16eedf release/installers/numpy-1.3.0rc2-win32-superpack-python2.6.exe b00b22484080b58141961af9a9707b01 release/installers/numpy-1.3.0rc2-py2.5-macosx10.5.dmg From Jim.Vickroy at noaa.gov Fri Apr 3 11:40:49 2009 From: Jim.Vickroy at noaa.gov (Jim Vickroy) Date: Fri, 03 Apr 2009 09:40:49 -0600 Subject: [SciPy-user] scipy.interpolate.Rbf() MemoryError for more than 4000 (x, y) pairs ? Message-ID: <49D62E01.1030702@noaa.gov> Hello Group, I'm a radial basis functions novice so I may be making a mistake; however, it seems that scipy.interpolate.Rbf() can handle a maximum of ~4000 (x,y) points. Is that correct? Here is a sample output from the attached script (rbf-demo.py) for 5000 (x,y) points: DOS>rbf-demo.py Python version: 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)] numpy version: 1.2.1 scipy version: 0.7.0 multiquadric in-painting required 200 seconds for 5000 points Traceback (most recent call last): File "rbf-demo.py", line 28, in rbf = scipy.interpolate.Rbf(x, y, z, function=function) File "C:\Python25\lib\site-packages\scipy\interpolate\rbf.py", line 132, in __init__ r = self._call_norm(self.xi, self.xi) File "C:\Python25\lib\site-packages\scipy\interpolate\rbf.py", line 147, in _call_norm return self.norm(x1, x2) File "C:\Python25\lib\site-packages\scipy\interpolate\rbf.py", line 100, in _euclidean_norm return sqrt( ((x1 - x2)**2).sum(axis=0) ) MemoryError DOS> Thanks, -- jv P.S. I'm considering using Rbf to replace a horizontal band of bad-pixel values in 512x512 images obtained from a damaged detector. -------------- next part -------------- A non-text attachment was scrubbed... Name: rbf-demo.py Type: application/x-python Size: 1436 bytes Desc: not available URL: From josef.pktd at gmail.com Fri Apr 3 12:22:26 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 3 Apr 2009 12:22:26 -0400 Subject: [SciPy-user] scipy.interpolate.Rbf() MemoryError for more than 4000 (x, y) pairs ? In-Reply-To: <49D62E01.1030702@noaa.gov> References: <49D62E01.1030702@noaa.gov> Message-ID: <1cd32cbb0904030922s39f6cb4eu90cc38f52cd8aad3@mail.gmail.com> On Fri, Apr 3, 2009 at 11:40 AM, Jim Vickroy wrote: > Hello Group, > > I'm a radial basis functions novice so I may be making a mistake; however, > it seems that scipy.interpolate.Rbf() can handle a maximum of ~4000 (x,y) > points. ?Is that correct? > > Here is a sample output from the attached script (rbf-demo.py) for 5000 > (x,y) points: > > > DOS>rbf-demo.py > Python version: 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit > (Intel)] > numpy version: ?1.2.1 > scipy version: ?0.7.0 > multiquadric ? ? ? ? in-painting required 200 seconds for 5000 points > Traceback (most recent call last): > File "rbf-demo.py", line 28, in > ?rbf ? = scipy.interpolate.Rbf(x, y, z, function=function) > File "C:\Python25\lib\site-packages\scipy\interpolate\rbf.py", line 132, in > __init__ > ?r = self._call_norm(self.xi, self.xi) > File "C:\Python25\lib\site-packages\scipy\interpolate\rbf.py", line 147, in > _call_norm > ?return self.norm(x1, x2) > File "C:\Python25\lib\site-packages\scipy\interpolate\rbf.py", line 100, in > _euclidean_norm > ?return sqrt( ((x1 - x2)**2).sum(axis=0) ) > MemoryError > > DOS> > > > Thanks, > -- jv > > P.S. > I'm considering using Rbf to replace a horizontal band of bad-pixel values > in 512x512 images obtained from a damaged detector. > Besides the memory error, I don't think rbf can do this, rbf uses all points to do the interpolation, the kernel is number of observation squared. In some examples, that I tried, if the number of points is to large, then you don't get a smooth solution, the interpolation has too much variation (like fitting a very high order polynomial). >From my experimentation with rbf, it only works well with a small number of points, so the best is to run rbf on a local neighborhood of the missing points. For 1d, I have the script somewhere but I never tried the 2d version. Josef From pav at iki.fi Fri Apr 3 12:39:48 2009 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 3 Apr 2009 16:39:48 +0000 (UTC) Subject: [SciPy-user] scipy.interpolate.Rbf() MemoryError for more than 4000 (x, y) pairs ? References: <49D62E01.1030702@noaa.gov> <1cd32cbb0904030922s39f6cb4eu90cc38f52cd8aad3@mail.gmail.com> Message-ID: Fri, 03 Apr 2009 12:22:26 -0400, josef.pktd kirjoitti: [clip] > Besides the memory error, I don't think rbf can do this, rbf uses all > points to do the interpolation, the kernel is number of observation > squared. In some examples, that I tried, if the number of points is to > large, then you don't get a smooth solution, the interpolation has too > much variation (like fitting a very high order polynomial). Btw, the different Rbf functions (thin-plate, linear, etc.) seem to behave slightly differently in how stably they can fit many points. I'd imagine why this happens is answered somewhere in the literature... -- Pauli Virtanen From josef.pktd at gmail.com Fri Apr 3 12:50:07 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 3 Apr 2009 12:50:07 -0400 Subject: [SciPy-user] Fwd: scipy.interpolate.Rbf() MemoryError for more than 4000 (x, y) pairs ? In-Reply-To: <1cd32cbb0904030936r4bd1a47el56c0f78361c74460@mail.gmail.com> References: <49D62E01.1030702@noaa.gov> <1cd32cbb0904030922s39f6cb4eu90cc38f52cd8aad3@mail.gmail.com> <1cd32cbb0904030936r4bd1a47el56c0f78361c74460@mail.gmail.com> Message-ID: <1cd32cbb0904030950p4ca5e3c9kc0762d2620cfb74e@mail.gmail.com> send again, previous message bounced ---------- Forwarded message ---------- From: Date: Fri, Apr 3, 2009 at 12:36 PM Subject: Re: [SciPy-user] scipy.interpolate.Rbf() MemoryError for more than 4000 (x, y) pairs ? To: SciPy Users List On Fri, Apr 3, 2009 at 12:22 PM, ? wrote: > On Fri, Apr 3, 2009 at 11:40 AM, Jim Vickroy wrote: >> Hello Group, >> >> I'm a radial basis functions novice so I may be making a mistake; however, >> it seems that scipy.interpolate.Rbf() can handle a maximum of ~4000 (x,y) >> points. ?Is that correct? >> >> Here is a sample output from the attached script (rbf-demo.py) for 5000 >> (x,y) points: >> >> >> DOS>rbf-demo.py >> Python version: 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit >> (Intel)] >> numpy version: ?1.2.1 >> scipy version: ?0.7.0 >> multiquadric ? ? ? ? in-painting required 200 seconds for 5000 points >> Traceback (most recent call last): >> File "rbf-demo.py", line 28, in >> ?rbf ? = scipy.interpolate.Rbf(x, y, z, function=function) >> File "C:\Python25\lib\site-packages\scipy\interpolate\rbf.py", line 132, in >> __init__ >> ?r = self._call_norm(self.xi, self.xi) >> File "C:\Python25\lib\site-packages\scipy\interpolate\rbf.py", line 147, in >> _call_norm >> ?return self.norm(x1, x2) >> File "C:\Python25\lib\site-packages\scipy\interpolate\rbf.py", line 100, in >> _euclidean_norm >> ?return sqrt( ((x1 - x2)**2).sum(axis=0) ) >> MemoryError >> >> DOS> >> >> >> Thanks, >> -- jv >> >> P.S. >> I'm considering using Rbf to replace a horizontal band of bad-pixel values >> in 512x512 images obtained from a damaged detector. >> > > > Besides the memory error, I don't think rbf can do this, rbf uses all > points to do the interpolation, the kernel is number of observation > squared. In some examples, that I tried, if the number of points is to > large, then you don't get a smooth solution, the interpolation has too > much variation (like fitting a very high order polynomial). > > From my experimentation with rbf, it only works well with a small > number of points, so the best is to run rbf on a local neighborhood of > the missing points. > > For 1d, I have the script somewhere but I never tried the 2d version. > > Josef > here is the script, it produces 12 graphs to illustrate this Josef -------------- next part -------------- '''script to try out scipy.interpolate.rbf rbf solves a system of equations with the number of equations and variables equal to the number of node points. This produces reasonable results for up to 10 or 20 points, but for 100 points the results are very jagged. Scott Sinclair recommended using moving windows, which I implemented for the one dimensional case. For the higher dimensional case, an efficient neighbor search could be done with the scipy.distance functions. RBF kernel regression ? Josef ''' import numpy as np import matplotlib.pyplot as plt from scipy import interpolate def show_rbf_example(basefn,n): '''example for global rbf Parameters ---------- basefn: function sample points are created with basefn plus noise n: integer number of sample points to create returns ------- None output: one plots Notes ----- If the number of sample points is larger than 20 (in the examples), then the interpolation is very jagged. ''' xr = np.linspace(0,3,n) yr = basefn(xr)+0.1*np.random.randn(xr.size) plt.figure() plt.plot(xr,yr,'o') xi = np.linspace(0,3,n*5) rbfi = interpolate.Rbf(xr, yr) #use default 'multiquadric':: #yi = rbfi(xi) plt.plot(xi,rbfi(xi),'r.-') rbfig = interpolate.Rbf(xr, yr, function='gaussian') plt.plot(xi,rbfig(xi),'g.-') plt.title('interpolation with global rbf, n = %d' % n + \ '\nred: multiquadric, green: gaussian') def windowedrbf(xi,x,d, hw=2): '''interpolate 1D points using local rbf Parameters ---------- xi: 1D array points for which interpolated values are calculated x: 1D array coordinates of node points d: 1D array values, hight at coordinates x hw: integer (default = 2) half window size, number of points to include in the rbf in each direction. The number of included points in the rbf is 2*hw for points that are not nodes and 2*hw+1 for node points ''' ind = np.argsort(x) x = x[ind] y = d[ind] ny = len(y) # find neighborhoods indx = np.argsort(xi) revindx = np.argsort(indx) # inverse sort index xs = xi[indx] intervals = np.searchsorted(x,xs) neigh = {} for ii,iv in enumerate(intervals): neigh.setdefault(iv,[]).append(ii) # interpolate for points in each neighborhood di = [] for k in neigh: sl = slice(max(0,k-hw),min(ny,k+hw),1) #print sl #print x[sl], y[sl] # get rbf for neighborhood rbfi = interpolate.Rbf(x[sl], y[sl]) xiindlocal = neigh[k] #print xiindlocal xilocal = xs[xiindlocal] di.extend(zip(xilocal,rbfi(xilocal))) xdi = np.array(di) return xdi[revindx,:], neigh, intervals def show_rbfw_example(basefn, n, gap=True, extrapol=False): '''example to plot comparison between global and local rbf Parameters ---------- basefn: function sample points are created with basefn plus noise n: integer number of sample points to create gap: boolean (default: True) if true, create a 10% gap starting in middle of sample extrapol: boolean if true, extrapolate 5 points (1 node point distance) returns ------- None output: two plots ''' xr = np.linspace(0,3,n) yr = basefn(xr)+0.1*np.random.randn(xr.size) if gap: rm = slice(n/2,np.int(n*(1/2.0+0.1))) xr = np.delete(xr,rm) yr = np.delete(yr,rm) if extrapol: extrap = 3/float(n) # extrapolate 5 points (1 nodepoint distance) else: extrap = 0 xi = 3 + extrap - np.linspace(0,3+extrap,5*n) res, resn, iv = windowedrbf(xi,xr,yr) assert np.any(res[:,0]==xi), 'original order of xi array not preserved' ## plt.figure() ## plt.plot(res[:,0],res[:,1],'r.',xr,yr,'o') ## plt.title('interprolation with local rbf, n = %d' % n) plt.figure() plt.plot(xr,yr,'o',res[:,0],res[:,1],'r.-') plt.title('interprolation with local rbf, n = %d' % n) plt.figure() rbfi = interpolate.Rbf(xr, yr) #use default 'multiquadric':: plt.plot(xi,rbfi(xi),'r.-',xr,yr,'o',) plt.title('interpolation with global rbf, n = %d' % n) #basefn = lambda (x): np.sin(x**2) def basefn1(x): return np.sin(x**2) def basefn2(x): return x**2 # global rbf works ok up to 20 points ##for basefn in [basefn1, basefn2]: ## for n in [20]:#[3,10,20,100]: ## show_rbf_example(basefn,n) for basefn in [basefn1, basefn2]: for n in [10,20,100]: show_rbfw_example(basefn,n) #plt.show() -------------- next part -------------- A non-text attachment was scrubbed... Name: rbf100.png Type: image/png Size: 60550 bytes Desc: not available URL: From josef.pktd at gmail.com Fri Apr 3 12:51:03 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 3 Apr 2009 12:51:03 -0400 Subject: [SciPy-user] scipy.interpolate.Rbf() MemoryError for more than 4000 (x, y) pairs ? In-Reply-To: <1cd32cbb0904030950p4ca5e3c9kc0762d2620cfb74e@mail.gmail.com> References: <49D62E01.1030702@noaa.gov> <1cd32cbb0904030922s39f6cb4eu90cc38f52cd8aad3@mail.gmail.com> <1cd32cbb0904030936r4bd1a47el56c0f78361c74460@mail.gmail.com> <1cd32cbb0904030950p4ca5e3c9kc0762d2620cfb74e@mail.gmail.com> Message-ID: <1cd32cbb0904030951lb4f404bob1dc815df88f6d78@mail.gmail.com> > > here is the script, it produces 12 graphs to illustrate this > > Josef > graph with local rbf -------------- next part -------------- A non-text attachment was scrubbed... Name: rbf100local.png Type: image/png Size: 47671 bytes Desc: not available URL: From millman at berkeley.edu Fri Apr 3 13:03:42 2009 From: millman at berkeley.edu (Jarrod Millman) Date: Fri, 3 Apr 2009 10:03:42 -0700 Subject: [SciPy-user] [Numpy-discussion] [ANN] Numpy 1.3.0rc2 In-Reply-To: <49D628BA.2000404@ar.media.kyoto-u.ac.jp> References: <49D628BA.2000404@ar.media.kyoto-u.ac.jp> Message-ID: On Fri, Apr 3, 2009 at 8:18 AM, David Cournapeau wrote: > I am pleased to announce the release of the rc2 for numpy 1.3.0. I have > decided to go for a rc2 instead of the release directly because of the > serious mac os X issue. You can find source tarballs and installers for > both Mac OS X and Windows on the sourceforge page: Thanks to David and everyone else who worked on this release! I am looking forward to 1.3.0 final. Jarrod From josef.pktd at gmail.com Fri Apr 3 13:05:35 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 3 Apr 2009 13:05:35 -0400 Subject: [SciPy-user] scipy.interpolate.Rbf() MemoryError for more than 4000 (x, y) pairs ? In-Reply-To: References: <49D62E01.1030702@noaa.gov> <1cd32cbb0904030922s39f6cb4eu90cc38f52cd8aad3@mail.gmail.com> Message-ID: <1cd32cbb0904031005s2e803939l5ef30c395e3f1a5b@mail.gmail.com> On Fri, Apr 3, 2009 at 12:39 PM, Pauli Virtanen wrote: > Fri, 03 Apr 2009 12:22:26 -0400, josef.pktd kirjoitti: > [clip] >> Besides the memory error, I don't think rbf can do this, rbf uses all >> points to do the interpolation, the kernel is number of observation >> squared. In some examples, that I tried, if the number of points is to >> large, then you don't get a smooth solution, the interpolation has too >> much variation (like fitting a very high order polynomial). > > Btw, the different Rbf functions (thin-plate, linear, etc.) seem to > behave slightly differently in how stably they can fit many points. I'd > imagine why this happens is answered somewhere in the literature... > I had only tried and looked at the code for gaussian and multiquadratic. linear and thin-plate work in my example, but according to the comments 'thin-plate' is the first that has a memory error. Josef -------------- next part -------------- A non-text attachment was scrubbed... Name: rbf100_thinplate.png Type: image/png Size: 43707 bytes Desc: not available URL: From Jim.Vickroy at noaa.gov Fri Apr 3 13:08:44 2009 From: Jim.Vickroy at noaa.gov (Jim Vickroy) Date: Fri, 03 Apr 2009 11:08:44 -0600 Subject: [SciPy-user] Fwd: scipy.interpolate.Rbf() MemoryError for more than 4000 (x, y) pairs ? In-Reply-To: <1cd32cbb0904030950p4ca5e3c9kc0762d2620cfb74e@mail.gmail.com> References: <49D62E01.1030702@noaa.gov> <1cd32cbb0904030922s39f6cb4eu90cc38f52cd8aad3@mail.gmail.com> <1cd32cbb0904030936r4bd1a47el56c0f78361c74460@mail.gmail.com> <1cd32cbb0904030950p4ca5e3c9kc0762d2620cfb74e@mail.gmail.com> Message-ID: <49D6429C.3070506@noaa.gov> Thanks for this valuable information, Josef. -- jv josef.pktd at gmail.com wrote: > send again, previous message bounced > > > ---------- Forwarded message ---------- > From: > Date: Fri, Apr 3, 2009 at 12:36 PM > Subject: Re: [SciPy-user] scipy.interpolate.Rbf() MemoryError for more > than 4000 (x, y) pairs ? > To: SciPy Users List > > > On Fri, Apr 3, 2009 at 12:22 PM, wrote: > >> On Fri, Apr 3, 2009 at 11:40 AM, Jim Vickroy wrote: >> >>> Hello Group, >>> >>> I'm a radial basis functions novice so I may be making a mistake; however, >>> it seems that scipy.interpolate.Rbf() can handle a maximum of ~4000 (x,y) >>> points. Is that correct? >>> >>> Here is a sample output from the attached script (rbf-demo.py) for 5000 >>> (x,y) points: >>> >>> >>> DOS>rbf-demo.py >>> Python version: 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit >>> (Intel)] >>> numpy version: 1.2.1 >>> scipy version: 0.7.0 >>> multiquadric in-painting required 200 seconds for 5000 points >>> Traceback (most recent call last): >>> File "rbf-demo.py", line 28, in >>> rbf = scipy.interpolate.Rbf(x, y, z, function=function) >>> File "C:\Python25\lib\site-packages\scipy\interpolate\rbf.py", line 132, in >>> __init__ >>> r = self._call_norm(self.xi, self.xi) >>> File "C:\Python25\lib\site-packages\scipy\interpolate\rbf.py", line 147, in >>> _call_norm >>> return self.norm(x1, x2) >>> File "C:\Python25\lib\site-packages\scipy\interpolate\rbf.py", line 100, in >>> _euclidean_norm >>> return sqrt( ((x1 - x2)**2).sum(axis=0) ) >>> MemoryError >>> >>> DOS> >>> >>> >>> Thanks, >>> -- jv >>> >>> P.S. >>> I'm considering using Rbf to replace a horizontal band of bad-pixel values >>> in 512x512 images obtained from a damaged detector. >>> >>> >> Besides the memory error, I don't think rbf can do this, rbf uses all >> points to do the interpolation, the kernel is number of observation >> squared. In some examples, that I tried, if the number of points is to >> large, then you don't get a smooth solution, the interpolation has too >> much variation (like fitting a very high order polynomial). >> >> From my experimentation with rbf, it only works well with a small >> number of points, so the best is to run rbf on a local neighborhood of >> the missing points. >> >> For 1d, I have the script somewhere but I never tried the 2d version. >> >> Josef >> >> > > here is the script, it produces 12 graphs to illustrate this > > Josef > > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------ > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 60550 bytes Desc: not available URL: From guyer at nist.gov Fri Apr 3 15:04:53 2009 From: guyer at nist.gov (Jonathan Guyer) Date: Fri, 3 Apr 2009 15:04:53 -0400 Subject: [SciPy-user] "smooth" plots In-Reply-To: <26B3CE4E-5D3E-4904-A0BC-36F319103DDA@nist.gov> References: <26B3CE4E-5D3E-4904-A0BC-36F319103DDA@nist.gov> Message-ID: <54118B02-1375-4D46-907E-F22C8CAF8D1C@nist.gov> On Apr 1, 2009, at 1:36 PM, I wrote: > Lets say I want to plot y = f(x). > > f() is non-analytical and expensive. > > Are there any facilities in SciPy (or elsewhere in Python) for > automating the selection of values of x that will produce a "smooth" > plot of y vs. x? David Huard sent me a perfectly serviceable bit of code off-line (he described it as "not pretty", but it does the trick for me and sure beats manually hunting and pecking). When I get back from vacation, I may pester him to post a cookbook example, or to let me do it. From ceputter at googlemail.com Fri Apr 3 16:34:46 2009 From: ceputter at googlemail.com (Christiaan Putter) Date: Fri, 3 Apr 2009 22:34:46 +0200 Subject: [SciPy-user] "smooth" plots In-Reply-To: <54118B02-1375-4D46-907E-F22C8CAF8D1C@nist.gov> References: <26B3CE4E-5D3E-4904-A0BC-36F319103DDA@nist.gov> <54118B02-1375-4D46-907E-F22C8CAF8D1C@nist.gov> Message-ID: <3a59ce2f0904031334n5ace8f5er7d93c3cdfcdd60fb@mail.gmail.com> There's a function plotter in chaco that one can do that with. It will compute the required values depending on the view (zoom, index bounds). There's an usage example that shows how to use it in case anyone's interested. 2009/4/3 Jonathan Guyer : > > On Apr 1, 2009, at 1:36 PM, I wrote: > >> Lets say I want to plot y = f(x). >> >> f() is non-analytical and expensive. >> >> Are there any facilities in SciPy (or elsewhere in Python) for >> automating the selection of values of x that will produce a "smooth" >> plot of y vs. x? > > David Huard sent me a perfectly serviceable bit of code off-line (he > described it as "not pretty", but it does the trick for me and sure > beats manually hunting and pecking). When I get back from vacation, I > may pester him to post a cookbook example, or to let me do it. > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From peter.skomoroch at gmail.com Fri Apr 3 16:39:36 2009 From: peter.skomoroch at gmail.com (Peter Skomoroch) Date: Fri, 3 Apr 2009 16:39:36 -0400 Subject: [SciPy-user] SciPy installed on Amazon Elastic MapReduce Message-ID: I thought the mailing list would be interested to know that NumPy and SciPy are among the few default packages installed on Amazon Elastic MapReduce, so you can use them in Python streaming Hadoop jobs on Amazon. http://aws.amazon.com/elasticmapreduce/ * Amazon Elastic MapReduce is a web service that enables businesses, researchers, data analysts, and developers to easily and cost-effectively process vast amounts of data. It utilizes a hosted Hadoop framework running on the web-scale infrastructure of Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3). Using Amazon Elastic MapReduce, you can instantly provision as much or as little capacity as you like to perform data-intensive tasks for applications such as web indexing, data mining, log file analysis, machine learning, financial analysis, scientific simulation, and bioinformatics research. Amazon Elastic MapReduce lets you focus on crunching or analyzing your data without having to worry about time-consuming set-up, management or tuning of Hadoop clusters or the compute capacity upon which they sit. * List of installed packages is here: http://developer.amazonwebservices.com/connect/thread.jspa?messageID=123216𞅐 -- Peter N. Skomoroch 617.285.8348 http://www.datawrangling.com http://delicious.com/pskomoroch http://twitter.com/peteskomoroch -------------- next part -------------- An HTML attachment was scrubbed... URL: From xiangyuding at gmail.com Fri Apr 3 18:35:35 2009 From: xiangyuding at gmail.com (Xiangyu Ding) Date: Fri, 3 Apr 2009 17:35:35 -0500 Subject: [SciPy-user] Simulink API to Python? Message-ID: <6ed40ad40904031535y6c48235fj39bbf9007ae8cd5a@mail.gmail.com> I just started using python and plan to port a matlab simulation software to python. The main problem I had so far is that there doesn't seem to be a good alternative to simulink in python. so my plan was to compile the simulink model into C code and use some kind of API to transfer the results into python. Currently I'm saving the result to hard drive and load it in scipy, but this is very slow. I was wondering, is there any existing module or something that will allow the transfer to be done in memory? Another problem is that I need to extract the state space matrix from a simulink model, does anyone know a way to do this without using matlab? Thanks, Xiangyu Ding From ceputter at googlemail.com Fri Apr 3 18:46:08 2009 From: ceputter at googlemail.com (Christiaan Putter) Date: Sat, 4 Apr 2009 00:46:08 +0200 Subject: [SciPy-user] [timeseries] Missing dates Message-ID: <3a59ce2f0904031546m1731580fxb2af4ba79057295a@mail.gmail.com> Hi guys, I've been playing with timeseries for the last hour or so and it's pretty cool. Still lots of things I have to go through. In the one plotting example (using yahoo finance) I saw that one can fill missing dates before plotting so that the missing ones get masked. Though when applying some moving windows functions that caused all periods that were effected by the missing values to also become masked, which isn't the behaviour I was expecting. It does make sense to do it that way though. Obviously it's simple enough to use the original timeseries to calculate the moving window functions, or interpolate or something. The question I'm trying to get at though is if I'm going to store my timeseries in hdf5 will I fill in the missing dates before I do so, or only do that whenever I plot the timeseries? I'm working with stock prices, so the "missing" dates over the weekends will increase file size by more then 30%. Is there any other reason to fill in missing dates besides for plotting? Have a nice weekend, Christian From pgmdevlist at gmail.com Fri Apr 3 19:23:21 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 3 Apr 2009 19:23:21 -0400 Subject: [SciPy-user] [timeseries] Missing dates In-Reply-To: <3a59ce2f0904031546m1731580fxb2af4ba79057295a@mail.gmail.com> References: <3a59ce2f0904031546m1731580fxb2af4ba79057295a@mail.gmail.com> Message-ID: <975EF0FC-DB66-4097-B580-758474A625CA@gmail.com> On Apr 3, 2009, at 6:46 PM, Christiaan Putter wrote: > Hi guys, > > I've been playing with timeseries for the last hour or so and it's > pretty cool. Still lots of things I have to go through. Write your experience down and keep us posted, that'd be great material for a FAQ. > In the one plotting example (using yahoo finance) I saw that one can > fill missing dates before plotting so that the missing ones get > masked. Though when applying some moving windows functions that > caused all periods that were effected by the missing values to also > become masked, which isn't the behaviour I was expecting. It does > make sense to do it that way though. > > I'm working with stock > prices, so the "missing" dates over the weekends will increase file > size by more then 30%. Is there any other reason to fill in missing > dates besides for plotting? Most functions (like convert or align_series) do require consecutive dates, hence the need for fill_missing_dates. This function ensures that your dates are all consecutive, and the values corresponding to the initially missing dates are replaced by the constant masked. For the case of movering average (for example), if you don't fill the dates, you may end up grouping values separated by more than your window size, which may not give the results you'd expect. For the case of plotting: when plotting a series w/ missing dates using some line between the points, you'll connect existing points (eg, Friday w/ Monday). If you want a separation between Fridays and Mondays, fill the dates first. OK, one picture is worth a lot of words, so compare the two plots: >>> import numpy as np, numpy.ma as ma, scikits.timeseries as ts, scikits.timeseries.lib.plotlib as tpl >>> s=ts.time_series(np.range(10), dates=ts.date_array(['2001-%02i' % i for i in (1,2,3,4,5,6,7,10,11,12)],freq='M')) >>> tpl.tsplot(s, 'o-b') >>> tpl.tsplot(s.fill_missing_dates(), 's-r') > The question I'm trying to get at though is if I'm going to store my > timeseries in hdf5 will I fill in the missing dates before I do so, or > only do that whenever I plot the timeseries? If you need to save space, don't save the series w/ missing data. You can use the `compressed` method to get rid of those before saving, and use fill_missing_dates after loading. From mattknox.ca at gmail.com Fri Apr 3 21:03:18 2009 From: mattknox.ca at gmail.com (Matt Knox) Date: Sat, 4 Apr 2009 01:03:18 +0000 (UTC) Subject: [SciPy-user] [timeseries] Missing dates References: <3a59ce2f0904031546m1731580fxb2af4ba79057295a@mail.gmail.com> Message-ID: > In the one plotting example (using yahoo finance) I saw that one can > fill missing dates before plotting so that the missing ones get > masked. Though when applying some moving windows functions that > caused all periods that were effected by the missing values to also > become masked, which isn't the behaviour I was expecting. It does > make sense to do it that way though. > > Obviously it's simple enough to use the original timeseries to > calculate the moving window functions, or interpolate or something. You hit the nail on the head here. There is no way for the timeseries module to know what the user thinks is the proper way to handle the masked values here, so the sensible thing to do is mask the whole region. You can calculate the moving average on the original series (ie. before you call fill_missing_dates), or interpolate the data somehow first (eg. using forward_fill), etc. > The question I'm trying to get at though is if I'm going to store my > timeseries in hdf5 will I fill in the missing dates before I do so, or > only do that whenever I plot the timeseries? I'm working with stock > prices, so the "missing" dates over the weekends will increase file > size by more then 30%. Is there any other reason to fill in missing > dates besides for plotting? Note that in the example you are talking about, the series is a "BUSINESS" frequency series dates = ts.date_array([q[0] for q in quotes], freq='DAILY').asfreq('BUSINESS') so calling fill_missing_dates on this has the effect of adding masked values for the HOLIDAYS, but not Saturday and Sunday. Now as to whether or not one should fill in the holidays for storage purposes is a judgement call, but I generally find it simpler to just forward fill all holidays (see the forward_fill function in the interpolation section of the docs) in a batch job overnight and that way any reports or models don't have to think about adding special logic to handle holidays which can be somewhat complicated, especially if you are talking about global data with different calendars and so forth. Yes, this can introduce inaccuracies to some degree, but for most use cases I have found the gains in simplicity more than outweigh those costs. - Matt From dwf at cs.toronto.edu Sat Apr 4 02:46:26 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Sat, 4 Apr 2009 02:46:26 -0400 Subject: [SciPy-user] Simulink API to Python? In-Reply-To: <6ed40ad40904031535y6c48235fj39bbf9007ae8cd5a@mail.gmail.com> References: <6ed40ad40904031535y6c48235fj39bbf9007ae8cd5a@mail.gmail.com> Message-ID: <7DDCE24D-3E80-485A-B2D7-A7218C08E9DB@cs.toronto.edu> Hi Xiangyu, On 3-Apr-09, at 6:35 PM, Xiangyu Ding wrote: > > I was wondering, is there any existing module or something that will > allow the transfer to be done in memory? This sounds roughly like a job for Cython's numpy buffer support. It would allow you call whatever C functions you need to and expose contiguous blocks of memory (to which you have a pointer) to Python code. See the tutorial at: http://wiki.cython.org/tutorials/numpy David From matthieu.brucher at gmail.com Sat Apr 4 05:15:21 2009 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Sat, 4 Apr 2009 11:15:21 +0200 Subject: [SciPy-user] [ANN][optimization] OpenOpt 0.23 In-Reply-To: <9457e7c80903170609g5214f970sec5b9f17b60d5a04@mail.gmail.com> References: <49BD506F.5000900@ukr.net> <49BEB66E.2040501@lpta.in2p3.fr> <9457e7c80903170144w5b7d9f87p568104f9dad81e15@mail.gmail.com> <49BF6351.3080203@ukr.net> <9457e7c80903170609g5214f970sec5b9f17b60d5a04@mail.gmail.com> Message-ID: 2009/3/17 St?fan van der Walt : > 2009/3/17 Dmitrey : >>> I agree. ?Dmitrey, would that be ok with you? >>> >> I hadn't removed that one yet because there is GenericOpt code there. > > Why did the GenericOpt code not travel along? > > Matthieu, can we integrate GenericOpt into scipy.optimize? > > Cheers > St?fan Hi, I think I forgot to reply to this... GenericOpt is not finished, I think. It lacks several tools, but the basis is there. For the moment, the design is ready for any unconstrained optimization, provided you implement some of the missing tools (for instance rank updates, ...). The problem is, I don't have much time to complete this now I'm outside the university and I don't work that much with Python ATM :( Matthieu -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From s.mientki at ru.nl Sat Apr 4 12:41:10 2009 From: s.mientki at ru.nl (Stef Mientki) Date: Sat, 04 Apr 2009 18:41:10 +0200 Subject: [SciPy-user] Simulink API to Python? In-Reply-To: <6ed40ad40904031535y6c48235fj39bbf9007ae8cd5a@mail.gmail.com> References: <6ed40ad40904031535y6c48235fj39bbf9007ae8cd5a@mail.gmail.com> Message-ID: <49D78DA6.6090303@ru.nl> Xiangyu Ding wrote: > I just started using python and plan to port a matlab simulation > software to python. > > The main problem I had so far is that there doesn't seem to be a good > alternative to simulink in python. I'm working on PyLab_Works, which should be able to be a replacement for both Simulink and PowerSim. although not yet released, you might get an impression here: http://code.google.com/p/pylab-works/ cheers, Stef From ceputter at googlemail.com Sat Apr 4 13:41:42 2009 From: ceputter at googlemail.com (Christiaan Putter) Date: Sat, 4 Apr 2009 19:41:42 +0200 Subject: [SciPy-user] [timeseries] Missing dates In-Reply-To: References: <3a59ce2f0904031546m1731580fxb2af4ba79057295a@mail.gmail.com> Message-ID: <3a59ce2f0904041041h48a934eev523acb63689c154a@mail.gmail.com> 2009/4/4 Matt Knox : >> In the one plotting example (using yahoo finance) I saw that one can >> fill missing dates before plotting so that the missing ones get >> masked. ?Though when applying some moving windows functions that >> caused all periods that were effected by the missing values to also >> become masked, which isn't the behaviour I was expecting. ?It does >> make sense to do it that way though. >> >> Obviously it's simple enough to use the original timeseries to >> calculate the moving window functions, or interpolate or something. > > You hit the nail on the head here. There is no way for the timeseries module to > know what the user thinks is the proper way to handle the masked values here, so > the sensible thing to do is mask the whole region. You can calculate the moving > average on the original series (ie. before you call fill_missing_dates), or > interpolate the data somehow first (eg. using forward_fill), etc. > >> The question I'm trying to get at though is if I'm going to store my >> timeseries in hdf5 will I fill in the missing dates before I do so, or >> only do that whenever I plot the timeseries? ?I'm working with stock >> prices, so the "missing" dates over the weekends will increase file >> size by more then 30%. ?Is there any other reason ?to fill in missing >> dates besides for plotting? > > Note that in the example you are talking about, the series is a "BUSINESS" > frequency series > > dates = ts.date_array([q[0] for q in quotes], freq='DAILY').asfreq('BUSINESS') > > so calling fill_missing_dates on this has the effect of adding masked values for > the HOLIDAYS, but not Saturday and Sunday. I'm trying to wrap my head around how the different frequencies behave... Correct me if I'm wrong. Using the yahoo example as a reference: A timeseries with daily frequency for one year does not need to have a date value for every day in that year in its date array. But when it gets plotted the index (x-axis) runs over the entire year, and a line plot will simply connect all the dots basically as if it were linearly interpolating the values for the missing dates. Thus even though the frequency is expecting values for every day, the process of masking the missing dates needs to be done explicitly. Which is done by fill_missing_dates(), which then adds a date to our array for every date that the frequency expects, and sets the mask to true for those dates that weren't in the array initially. So when I'm using a business frequency the index on plots doesn't behave like a calender, but mondays immediately follow fridays. And the only dates 'missing' are in fact the holidays as you pointed out, which we need to add explicitly so that we can mask them. > Now as to whether or not one should fill in the holidays for storage purposes is > a judgement call, but I generally find it simpler to just forward fill all > holidays (see the forward_fill function in the interpolation section of the > docs) in a batch job overnight and that way any reports or models don't have to > think about adding special logic to handle holidays which can be somewhat > complicated, especially if you are talking about global data with different > calendars and so forth. Yes, this can introduce inaccuracies to some degree, but > for most use cases I have found the gains in simplicity more than outweigh those > costs. The way I've been working with my data up until now was purely looking at trading days, ignoring weekends, holidays etc. So any analysis that takes time into account basically has 'trading day' as its unit of time. This makes some things simpler. close[-11] would always be the closing price 10 trading days ago. Rate of change or other indicators aren't effected by long holidays, though that's just a minor issue. The disadvantage is obviously that indexing the array with an actual date becomes a bit harder and you always need to do a search. What would really be the advantage if I were to use 'business day' as my frequency? And another question: Do you use the closing price from yahoo or the adjusted closing price? It seems they use the adjusted prices themselves, though I've come across one or two graphs where they didn't. Thanks for all your advice Matt and Pierre. Regards, Christian > - Matt > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From ndbecker2 at gmail.com Sat Apr 4 13:42:58 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Sat, 04 Apr 2009 13:42:58 -0400 Subject: [SciPy-user] Simulink API to Python? References: <6ed40ad40904031535y6c48235fj39bbf9007ae8cd5a@mail.gmail.com> Message-ID: Xiangyu Ding wrote: > I just started using python and plan to port a matlab simulation > software to python. > > The main problem I had so far is that there doesn't seem to be a good > alternative to simulink in python. so my plan was to compile the > simulink model into C code and use some kind of API to transfer the > results into python. Currently I'm saving the result to hard drive and > load it in scipy, but this is very slow. What does simulink do that you miss? Is it the gui? For what it's worth, electrical engineers used to design using schematics. Now design is described in text. There's a reason for this. Schematics are only useful for simple designs. Once they become more complex, it's much more efficient to work with text. Not to mention the ability to manipulate text, and the ability to transfer text-based designs between systems. From pgmdevlist at gmail.com Sat Apr 4 13:54:46 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Sat, 4 Apr 2009 13:54:46 -0400 Subject: [SciPy-user] [timeseries] Missing dates In-Reply-To: <3a59ce2f0904041041h48a934eev523acb63689c154a@mail.gmail.com> References: <3a59ce2f0904031546m1731580fxb2af4ba79057295a@mail.gmail.com> <3a59ce2f0904041041h48a934eev523acb63689c154a@mail.gmail.com> Message-ID: <139F0D93-18B4-4B49-A300-2E5BEC5D3001@gmail.com> On Apr 4, 2009, at 1:41 PM, Christiaan Putter wrote: > > I'm trying to wrap my head around how the different frequencies > behave... Correct me if I'm wrong. Using the yahoo example as a > reference: A timeseries with daily frequency for one year does not > need to have a date value for every day in that year in its date > array. Correct. For a TimeSeries, the dates don't have to be consecutive nor even in the right order... > But when it gets plotted the index (x-axis) runs over the > entire year, and a line plot will simply connect all the dots > basically as if it were linearly interpolating the values for the > missing dates. Correct again. You just plot a set of couples (value,date) > Thus even though the frequency is expecting values for > every day, the process of masking the missing dates needs to be done > explicitly. Correct. > Which is done by fill_missing_dates(), which then adds a > date to our array for every date that the frequency expects, and sets > the mask to true for those dates that weren't in the array initially. Correct > So when I'm using a business frequency the index on plots doesn't > behave like a calender, but mondays immediately follow fridays. Yes. > And > the only dates 'missing' are in fact the holidays as you pointed out, > which we need to add explicitly so that we can mask them. Yes, and this is somethng *you* have to do, depending on your own definition of holidays. > > The way I've been working with my data up until now was purely looking > at trading days, ignoring weekends, holidays etc. So any analysis > that takes time into account basically has 'trading day' as its unit > of time. This makes some things simpler. close[-11] would always be > the closing price 10 trading days ago. Rate of change or other > indicators aren't effected by long holidays, though that's just a > minor issue. > > The disadvantage is obviously that indexing the array with an actual > date becomes a bit harder and you always need to do a search. > > What would really be the advantage if I were to use 'business day' as > my frequency? * should you want to compute statistics over months or years * should you want to plot some graphs where the axis ticks are automatically adjusted depending on the zoom level. * should you need to find data falling between two given dates... > > > And another question: Do you use the closing price from yahoo or the > adjusted closing price? It seems they use the adjusted prices > themselves, though I've come across one or two graphs where they > didn't. I'll let Matt answering that... > Thanks for all your advice Matt and Pierre. Quite welcome. From xiangyuding at gmail.com Sat Apr 4 14:16:16 2009 From: xiangyuding at gmail.com (Xiangyu Ding) Date: Sat, 4 Apr 2009 13:16:16 -0500 Subject: [SciPy-user] Simulink API to Python? In-Reply-To: References: <6ed40ad40904031535y6c48235fj39bbf9007ae8cd5a@mail.gmail.com> Message-ID: <6ed40ad40904041116ldfb0e31s2f8c38841790cf1c@mail.gmail.com> Thanks to everyone for the suggestions. Neal wrote: > For what it's worth, electrical engineers used to design using schematics. > Now design is described in text. ?There's a reason for this. ?Schematics are > only useful for simple designs. ?Once they become more complex, it's much > more efficient to work with text. ?Not to mention the ability to manipulate > text, and the ability to transfer text-based designs between systems. I guess I need to clarify that the software I'm writing is for Reliability analysis purpose and lots of the system that it will be used to test have schematics in Simulink, that's the main reason I have to work with Simulink. I was considering porting everything to python if there were parser that will convert Simulink model into some equivalent model in python, the gui doesn't really matter. but I don't know of any tools that will do this. From mattknox.ca at gmail.com Sat Apr 4 15:04:10 2009 From: mattknox.ca at gmail.com (Matt Knox) Date: Sat, 4 Apr 2009 19:04:10 +0000 (UTC) Subject: [SciPy-user] [timeseries] Missing dates References: <3a59ce2f0904031546m1731580fxb2af4ba79057295a@mail.gmail.com> <3a59ce2f0904041041h48a934eev523acb63689c154a@mail.gmail.com> Message-ID: > And another question: Do you use the closing price from yahoo or the > adjusted closing price? It seems they use the adjusted prices > themselves, though I've come across one or two graphs where they > didn't. I don't know, sorry. I don't actually use data from yahoo. That example on the web page is just for illustration purposes. For any serious equity number crunching, yahoo probably isn't going to cut it as a data provider. I actually don't do any equity stuff anymore really. I've crossed over into the fixed income world which appeals much more to my mathematical sensibilities :) - Matt From david at ar.media.kyoto-u.ac.jp Sun Apr 5 06:04:39 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sun, 05 Apr 2009 19:04:39 +0900 Subject: [SciPy-user] [ANN] Numpy 1.3.0 released Message-ID: <49D88237.7050106@ar.media.kyoto-u.ac.jp> Hi, I am pleased to announce the release of numpy 1.3.0. You can find source tarballs and binary installers for both Mac OS X and Windows on the sourceforge page: https://sourceforge.net/projects/numpy/ The release note for the 1.3.0 release are below. There are no changes compared to the release candidate 2. Thank you to everyone involved in this release, developers, users who reported bugs, enjoy, the numpy developers. ---- ========================= NumPy 1.3.0 Release Notes ========================= This minor includes numerous bug fixes, official python 2.6 support, and several new features such as generalized ufuncs. Highlights ========== Python 2.6 support ~~~~~~~~~~~~~~~~~~ Python 2.6 is now supported on all previously supported platforms, including windows. http://www.python.org/dev/peps/pep-0361/ Generalized ufuncs ~~~~~~~~~~~~~~~~~~ There is a general need for looping over not only functions on scalars but also over functions on vectors (or arrays), as explained on http://scipy.org/scipy/numpy/wiki/GeneralLoopingFunctions. We propose to realize this concept by generalizing the universal functions (ufuncs), and provide a C implementation that adds ~500 lines to the numpy code base. In current (specialized) ufuncs, the elementary function is limited to element-by-element operations, whereas the generalized version supports "sub-array" by "sub-array" operations. The Perl vector library PDL provides a similar functionality and its terms are re-used in the following. Each generalized ufunc has information associated with it that states what the "core" dimensionality of the inputs is, as well as the corresponding dimensionality of the outputs (the element-wise ufuncs have zero core dimensions). The list of the core dimensions for all arguments is called the "signature" of a ufunc. For example, the ufunc numpy.add has signature "(),()->()" defining two scalar inputs and one scalar output. Another example is (see the GeneralLoopingFunctions page) the function inner1d(a,b) with a signature of "(i),(i)->()". This applies the inner product along the last axis of each input, but keeps the remaining indices intact. For example, where a is of shape (3,5,N) and b is of shape (5,N), this will return an output of shape (3,5). The underlying elementary function is called 3*5 times. In the signature, we specify one core dimension "(i)" for each input and zero core dimensions "()" for the output, since it takes two 1-d arrays and returns a scalar. By using the same name "i", we specify that the two corresponding dimensions should be of the same size (or one of them is of size 1 and will be broadcasted). The dimensions beyond the core dimensions are called "loop" dimensions. In the above example, this corresponds to (3,5). The usual numpy "broadcasting" rules apply, where the signature determines how the dimensions of each input/output object are split into core and loop dimensions: While an input array has a smaller dimensionality than the corresponding number of core dimensions, 1's are pre-pended to its shape. The core dimensions are removed from all inputs and the remaining dimensions are broadcasted; defining the loop dimensions. The output is given by the loop dimensions plus the output core dimensions. Experimental Windows 64 bits support ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Numpy can now be built on windows 64 bits (amd64 only, not IA64), with both MS compilers and mingw-w64 compilers: This is *highly experimental*: DO NOT USE FOR PRODUCTION USE. See INSTALL.txt, Windows 64 bits section for more information on limitations and how to build it by yourself. New features ============ Formatting issues ~~~~~~~~~~~~~~~~~ Float formatting is now handled by numpy instead of the C runtime: this enables locale independent formatting, more robust fromstring and related methods. Special values (inf and nan) are also more consistent across platforms (nan vs IND/NaN, etc...), and more consistent with recent python formatting work (in 2.6 and later). Nan handling in max/min ~~~~~~~~~~~~~~~~~~~~~~~ The maximum/minimum ufuncs now reliably propagate nans. If one of the arguments is a nan, then nan is retured. This affects np.min/np.max, amin/amax and the array methods max/min. New ufuncs fmax and fmin have been added to deal with non-propagating nans. Nan handling in sign ~~~~~~~~~~~~~~~~~~~~ The ufunc sign now returns nan for the sign of anan. New ufuncs ~~~~~~~~~~ #. fmax - same as maximum for integer types and non-nan floats. Returns the non-nan argument if one argument is nan and returns nan if both arguments are nan. #. fmin - same as minimum for integer types and non-nan floats. Returns the non-nan argument if one argument is nan and returns nan if both arguments are nan. #. deg2rad - converts degrees to radians, same as the radians ufunc. #. rad2deg - converts radians to degrees, same as the degrees ufunc. #. log2 - base 2 logarithm. #. exp2 - base 2 exponential. #. trunc - truncate floats to nearest integer towards zero. #. logaddexp - add numbers stored as logarithms and return the logarithm of the result. #. logaddexp2 - add numbers stored as base 2 logarithms and return the base 2 logarithm of the result result. Masked arrays ~~~~~~~~~~~~~ Several new features and bug fixes, including: * structured arrays should now be fully supported by MaskedArray (r6463, r6324, r6305, r6300, r6294...) * Minor bug fixes (r6356, r6352, r6335, r6299, r6298) * Improved support for __iter__ (r6326) * made baseclass, sharedmask and hardmask accesible to the user (but read-only) * doc update gfortran support on windows ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Gfortran can now be used as a fortran compiler for numpy on windows, even when the C compiler is Visual Studio (VS 2005 and above; VS 2003 will NOT work). Gfortran + Visual studio does not work on windows 64 bits (but gcc + gfortran does). It is unclear whether it will be possible to use gfortran and visual studio at all on x64. Arch option for windows binary ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Automatic arch detection can now be bypassed from the command line for the superpack installed: numpy-1.3.0-superpack-win32.exe /arch=nosse will install a numpy which works on any x86, even if the running computer supports SSE set. Deprecated features =================== Histogram ~~~~~~~~~ The semantics of histogram has been modified to fix long-standing issues with outliers handling. The main changes concern #. the definition of the bin edges, now including the rightmost edge, and #. the handling of upper outliers, now ignored rather than tallied in the rightmost bin. The previous behavior is still accessible using `new=False`, but this is deprecated, and will be removed entirely in 1.4.0. Documentation changes ===================== A lot of documentation has been added. Both user guide and references can be built from sphinx. New C API ========= Multiarray API ~~~~~~~~~~~~~~ The following functions have been added to the multiarray C API: * PyArray_GetEndianness: to get runtime endianness Ufunc API ~~~~~~~~~~~~~~ The following functions have been added to the ufunc API: * PyUFunc_FromFuncAndDataAndSignature: to declare a more general ufunc (generalized ufunc). New defines ~~~~~~~~~~~ New public C defines are available for ARCH specific code through numpy/npy_cpu.h: * NPY_CPU_X86: x86 arch (32 bits) * NPY_CPU_AMD64: amd64 arch (x86_64, NOT Itanium) * NPY_CPU_PPC: 32 bits ppc * NPY_CPU_PPC64: 64 bits ppc * NPY_CPU_SPARC: 32 bits sparc * NPY_CPU_SPARC64: 64 bits sparc * NPY_CPU_S390: S390 * NPY_CPU_IA64: ia64 * NPY_CPU_PARISC: PARISC New macros for CPU endianness has been added as well (see internal changes below for details): * NPY_BYTE_ORDER: integer * NPY_LITTLE_ENDIAN/NPY_BIG_ENDIAN defines Those provide portable alternatives to glibc endian.h macros for platforms without it. Portable NAN, INFINITY, etc... ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ npy_math.h now makes available several portable macro to get NAN, INFINITY: * NPY_NAN: equivalent to NAN, which is a GNU extension * NPY_INFINITY: equivalent to C99 INFINITY * NPY_PZERO, NPY_NZERO: positive and negative zero respectively Corresponding single and extended precision macros are available as well. All references to NAN, or home-grown computation of NAN on the fly have been removed for consistency. Internal changes ================ numpy.core math configuration revamp ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This should make the porting to new platforms easier, and more robust. In particular, the configuration stage does not need to execute any code on the target platform, which is a first step toward cross-compilation. http://projects.scipy.org/numpy/browser/trunk/doc/neps/math_config_clean.txt umath refactor ~~~~~~~~~~~~~~ A lot of code cleanup for umath/ufunc code (charris). Improvements to build warnings ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Numpy can now build with -W -Wall without warnings http://projects.scipy.org/numpy/browser/trunk/doc/neps/warnfix.txt Separate core math library ~~~~~~~~~~~~~~~~~~~~~~~~~~ The core math functions (sin, cos, etc... for basic C types) have been put into a separate library; it acts as a compatibility layer, to support most C99 maths functions (real only for now). The library includes platform-specific fixes for various maths functions, such as using those versions should be more robust than using your platform functions directly. The API for existing functions is exactly the same as the C99 math functions API; the only difference is the npy prefix (npy_cos vs cos). The core library will be made available to any extension in 1.4.0. CPU arch detection ~~~~~~~~~~~~~~~~~~ npy_cpu.h defines numpy specific CPU defines, such as NPY_CPU_X86, etc... Those are portable across OS and toolchains, and set up when the header is parsed, so that they can be safely used even in the case of cross-compilation (the values is not set when numpy is built), or for multi-arch binaries (e.g. fat binaries on Max OS X). npy_endian.h defines numpy specific endianness defines, modeled on the glibc endian.h. NPY_BYTE_ORDER is equivalent to BYTE_ORDER, and one of NPY_LITTLE_ENDIAN or NPY_BIG_ENDIAN is defined. As for CPU archs, those are set when the header is parsed by the compiler, and as such can be used for cross-compilation and multi-arch binaries. Checksums ========= e8d2b1f0d30416ee72bc29e3b6762fef release/installers/numpy-1.3.0-win32-superpack-python2.5.exe 3f7773ff0971a5ebb8591536d8ec7bd6 release/installers/numpy-1.3.0.tar.gz 18a7352f6ed8e6507f7cba1e56b4adc8 release/installers/numpy-1.3.0.zip 82e1cc25d945652c8408a48e00ba6726 release/installers/numpy-1.3.0-win32-superpack-python2.6.exe 0504197b83c7ed442aa684556bd0c519 release/installers/numpy-1.3.0-py2.5-macosx10.5.dmg 1a8089e34d7435d2b43bdda4f0a85443 release/installers/numpy-1.3.0.win-amd64-py2.6.msi From ceputter at googlemail.com Sun Apr 5 13:05:54 2009 From: ceputter at googlemail.com (Christiaan Putter) Date: Sun, 5 Apr 2009 19:05:54 +0200 Subject: [SciPy-user] [timeseries] Missing dates In-Reply-To: References: <3a59ce2f0904031546m1731580fxb2af4ba79057295a@mail.gmail.com> <3a59ce2f0904041041h48a934eev523acb63689c154a@mail.gmail.com> Message-ID: <3a59ce2f0904051005i632aaa60hcbf503e0f4baa73f@mail.gmail.com> Hi there, Again some missing dates questions... I'm trying to use a timeseries with records, the setup looks like this: historics = ydp.quotes_historical_yahoo('INTC') dates = ts.date_array(historics['date'], freq='DAILY').asfreq('BUSINESS') dtype=[('open', 'f4'), ('close', 'f4'), ('high', 'f4'), ('low', 'f4'), ('volume','u4'), ] data = zip(historics['open'], historics['close'], historics['high'], historics['low'], historics['volume'], ) raw_series = ts.time_series(data, dates, dtype=dtype) series = ts.fill_missing_dates(raw_series) Doing a report on this results in all records being masked... basicReport = rl.Report(raw_series, series) <-- I get a traceback when passing in only one argument (at the bottom of this email somewhere) basicReport() prints: *********** 05-Feb-2004 | (--, --, --, --, --) | (--, --, --, --, --) 06-Feb-2004 | (--, --, --, --, --) | (--, --, --, --, --) 09-Feb-2004 | (--, --, --, --, --) | (--, --, --, --, --) 10-Feb-2004 | (--, --, --, --, --) | (--, --, --, --, --) 11-Feb-2004 | (--, --, --, --, --) | (--, --, --, --, --) 12-Feb-2004 | (--, --, --, --, --) | (--, --, --, --, --) 13-Feb-2004 | (--, --, --, --, --) | (--, --, --, --, --) 16-Feb-2004 | (--, --, --, --, --) | (--, --, --, --, --) 17-Feb-2004 | (--, --, --, --, --) | (--, --, --, --, --) 18-Feb-2004 | (--, --, --, --, --) | (--, --, --, --, --) 19-Feb-2004 | (--, --, --, --, --) | (--, --, --, --, --) 20-Feb-2004 | (--, --, --, --, --) | (--, --, --, --, --) 23-Feb-2004 | (--, --, --, --, --) | (--, --, --, --, --) 24-Feb-2004 | (--, --, --, --, --) | (--, --, --, --, --) ************ I'm guessing the report is using ts.fill_missing_dates() which doesn't play well with records. That's why the series and raw_series look the same in the report... Reporting only a specific column: basicReport = rl.Report(raw_series['close'], series['close']) ********** 01-Jul-2008 | 21.03 | -- 02-Jul-2008 | 20.41 | -- 03-Jul-2008 | 20.15 | -- 04-Jul-2008 | -- | -- 07-Jul-2008 | 20.2 | -- 08-Jul-2008 | 20.4 | -- 09-Jul-2008 | 19.32 | -- 10-Jul-2008 | 20.11 | -- 11-Jul-2008 | 20.13 | -- ************ So how would I go about doing fill missing dates on a timeseries with records? Besides that I managed to use timeseries as a data source in chaco plots. Basic functionality is there, though I'm sure some guys at enthought will be able to improve on it. For their tick axis to work with dates properly I had to convert series.dates into posix timestamps: index = [time.mktime(t.timetuple()) for t in series.dates.tolist()] time_ds = ArrayDataSource(index) Not very efficient though I guess. In chaco a plot's index and values are in seperate data sources, which means timeseries with different frequencies require different indexes. I'll talk with one of their devs to see what's the best approach to deal with that. I'm assuming you merge the index axis somehow in matplotlib when plotting series with different frequencies? So if I were to plot something with a business frequency for the entire year 2009, and on the same figure plot a series with daily frequency just in January I'm guessing weekends will show up as gaps in the first plot? Just for january or for the entire year? In any case, I'll send along an example once it's actually usable. Figure attached... Have a nice weekend. ***************** Traceback from calling report with only one argument. I'm guessing it's something to do with the records. Traceback (most recent call last): File "/data/workspace/lynxgraphs/qtest_chaco.py", line 91, in basicReport() File "/usr/local/lib/python2.5/site-packages/scikits.timeseries-0.91.0-py2.5-linux-x86_64.egg/scikits/timeseries/lib/reportlib.py", line 338, in __call__ rows.append([datefmt_func(d)]+[fmt_func[i](ser.series[d - _sd]) for i, ser in enumerate(tseries)]) File "/usr/local/lib/python2.5/site-packages/numpy/ma/core.py", line 2578, in __getitem__ dout = ndarray.__getitem__(ndarray.view(self, ndarray), indx) IndexError: index out of bounds -------------- next part -------------- A non-text attachment was scrubbed... Name: chaco_timeseries.jpeg Type: image/jpeg Size: 58706 bytes Desc: not available URL: From pgmdevlist at gmail.com Sun Apr 5 13:46:02 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Sun, 5 Apr 2009 13:46:02 -0400 Subject: [SciPy-user] [timeseries] Missing dates In-Reply-To: <3a59ce2f0904051005i632aaa60hcbf503e0f4baa73f@mail.gmail.com> References: <3a59ce2f0904031546m1731580fxb2af4ba79057295a@mail.gmail.com> <3a59ce2f0904041041h48a934eev523acb63689c154a@mail.gmail.com> <3a59ce2f0904051005i632aaa60hcbf503e0f4baa73f@mail.gmail.com> Message-ID: <1FD641CD-EE15-46C3-A6C5-88F2AC066BA1@gmail.com> On Apr 5, 2009, at 1:05 PM, Christiaan Putter wrote: > Hi there, > <...> > > So how would I go about doing fill missing dates on a timeseries > with records? Well, you found a bug. It won't be corrected before the first release (in the next 48h), but will in the few days. Meanwhile, here's a workaround: >>> newlength = series.dates[-1] - series.dates[0] + 1 >>> newseries = ts.time_series(np.empty(newlength, dtype=series.dtype), start_date=series.dates[0]) >>> newlength = series.dates[-1] - series.dates[0] + 1 >>> newseries = ts.time_series(np.empty(newlength, dtype=series.dtype), start_date=series.dates[0]) >>> for n in series.dtype.names: newseries[n] = series[n].fill_missing_dates() > > > > Besides that I managed to use timeseries as a data source in chaco > plots. Basic functionality is there, though I'm sure some guys at > enthought will be able to improve on it. For their tick axis to work > with dates properly I had to convert series.dates into posix > timestamps: > > index = [time.mktime(t.timetuple()) for t in series.dates.tolist()] > time_ds = ArrayDataSource(index) > > Not very efficient though I guess. > > In chaco a plot's index and values are in seperate data sources, which > means timeseries with different frequencies require different indexes. > I'll talk with one of their devs to see what's the best approach to > deal with that. > > I'm assuming you merge the index axis somehow in matplotlib when > plotting series with different frequencies? When you plot a timeseries, the frequency of the plot is the frequency of the series. If you use multiple series w/ different frequencies, they're all converted to the plot frequency. If this latter was still undefined, it is set to the frequency of the first series, and the other ones are converted to this frequency. In other terms: * Either you convert all the series to a same frequency beforehand (using .asfreq, not .convert) * Either you plot the series with the highest frequency first, then the others > So if I were to plot > something with a business frequency for the entire year 2009, and on > the same figure plot a series with daily frequency just in January I'm > guessing weekends will show up as gaps in the first plot? Nope: because you plotted a business frequency first, there won't be any gap on week-ends. Instead, your second series will be internally expressed in business days, so you'll have several points falling on the same date (week-ends on fridays). From timmichelsen at gmx-topmail.de Sun Apr 5 14:31:03 2009 From: timmichelsen at gmx-topmail.de (Tim Michelsen) Date: Sun, 05 Apr 2009 20:31:03 +0200 Subject: [SciPy-user] [ANN] Numpy 1.3.0 released In-Reply-To: <49D88237.7050106@ar.media.kyoto-u.ac.jp> References: <49D88237.7050106@ar.media.kyoto-u.ac.jp> Message-ID: Hello, thanks to all developers and the releasemanager who made this possible! No I can keep up with the development version other packages that were already using the 1.3.* devel version. > I am pleased to announce the release of numpy 1.3.0. You can find source tarballs and binary installers for > both Mac OS X and Windows on the sourceforge page: > > https://sourceforge.net/projects/numpy/ I tried to upgrade with easy_install but it failed: C:\>easy_install --upgrade numpy Searching for numpy Reading http://pypi.python.org/simple/numpy/ Reading http://numpy.scipy.org Reading http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 Reading http://numeric.scipy.org Best match: numpy 1.2.1 Adding numpy 1.2.1 to easy-install.pth file Using c:\programme\pythonxy\python\lib\site-packages Processing dependencies for numpy Finished processing dependencies for numpy C:\>easy_install --upgrade "numpy==1.3.0" Searching for numpy==1.3.0 Reading http://pypi.python.org/simple/numpy/ Reading http://numpy.scipy.org Reading http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 Reading http://numeric.scipy.org No local packages or download links found for numpy==1.3.0 error: Could not find suitable distribution for Requirement.parse('numpy==1.3.0') Seems that I will have to wait until PythonXY gets updated. Kind regards, Timmie From timmichelsen at gmx-topmail.de Sun Apr 5 14:53:59 2009 From: timmichelsen at gmx-topmail.de (Tim Michelsen) Date: Sun, 05 Apr 2009 20:53:59 +0200 Subject: [SciPy-user] package hint: pyspread Message-ID: Hello, Interactive and visual analysis and editing can be desired in some situations. I just found a new contendor in this areas: PySpread http://pyspread.sourceforge.net/ A similar approach to ResolverOne but natively working with Python (not IronPython), free and more trageted towards science. Really worth a look! Kind ragards, Timmie From mattknox.ca at gmail.com Sun Apr 5 15:04:29 2009 From: mattknox.ca at gmail.com (Matt Knox) Date: Sun, 5 Apr 2009 19:04:29 +0000 (UTC) Subject: [SciPy-user] [timeseries] Missing dates References: <3a59ce2f0904031546m1731580fxb2af4ba79057295a@mail.gmail.com> <3a59ce2f0904041041h48a934eev523acb63689c154a@mail.gmail.com> <3a59ce2f0904051005i632aaa60hcbf503e0f4baa73f@mail.gmail.com> Message-ID: > In chaco a plot's index and values are in seperate data sources, which > means timeseries with different frequencies require different indexes. > I'll talk with one of their devs to see what's the best approach to > deal with that. > > I'm assuming you merge the index axis somehow in matplotlib when > plotting series with different frequencies? So if I were to plot > something with a business frequency for the entire year 2009, and on > the same figure plot a series with daily frequency just in January I'm > guessing weekends will show up as gaps in the first plot? Just for > january or for the entire year? I would generally recommend explicitly converting all the time series that you want to plot to the same frequency first rather than relying on the implicit conversion to avoid any unexpected behavior. Also, I don't use chaco personally, but if you come up with some good strategies for plotting TimeSeries objects with it that you can share with us, it may make a nice addition to the plotlib submodule in the timeseries package. And while we are on the topic of time series plotting... you may have noticed that matplotlib, chaco and the timeseries module all implement their own separate logic for date/time axis label auto scaling. I haven't used chaco, but in my (biased) opinion, the timeseries module currently does a better job at this than the plot_date function in matplotlib (although it is not perfect and there is still room for improvement). The plot_date auto_scaling does not handle all levels of zoom well (you can get overlapping labels at many zoom levels) and so far as I can tell will not work well with "business" frequency data (forces it to a "daily" plot with missing values on weekends). I would be interested to hear your thoughts on chaco's scaling compared to the timeseries module, particularly for business frequency and other non-daily types of data. - Matt From josef.pktd at gmail.com Sun Apr 5 15:42:07 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 5 Apr 2009 15:42:07 -0400 Subject: [SciPy-user] package hint: pyspread In-Reply-To: References: Message-ID: <1cd32cbb0904051242n7837987dxa711b9aef08aa742@mail.gmail.com> On Sun, Apr 5, 2009 at 2:53 PM, Tim Michelsen wrote: > Hello, > Interactive and visual analysis and editing can be desired in some > situations. > > I just found a new contendor in this areas: > > PySpread > http://pyspread.sourceforge.net/ > > A similar approach to ResolverOne but natively working with Python (not > IronPython), free and more trageted towards science. > > Really worth a look! > > Kind ragards, > Timmie > In a similar vein, I just looked at the new version of reinteract http://www.reinteract.org/trac/ which now also runs under windows. It looks useful for writing documentation and tutorials, it allows interactive editing and has a "copy as doctest" feature that allows examples to be directly copied to the docs, instead of piecewise cut and paste from the command line. Josef From josef.pktd at gmail.com Sun Apr 5 16:36:39 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 5 Apr 2009 16:36:39 -0400 Subject: [SciPy-user] removal of stats.mean et al breaks untested code Message-ID: <1cd32cbb0904051336p61d06ff0o279b8460d9d0e993@mail.gmail.com> I just found another victim of the removal of stats.mean and stats.var stats.gaussian_kde doesn't have any tests, and when I tried to rewrite the option to set the smoothing factor, I found that several methods are broken that use the removed stats.mean and stats.var. If the users of scipy have code similar to the one in scipy.stats, without sufficient test coverage, then fixing all the hidden (untested) usages of removed functions can be pretty painful and will take quite a bit of time. In case that the release cycle of scipy gets shorter, we might want to consider a longer depreciation period. Josef From stefan at sun.ac.za Sun Apr 5 17:48:29 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 5 Apr 2009 23:48:29 +0200 Subject: [SciPy-user] [ANN][optimization] OpenOpt 0.23 In-Reply-To: References: <49BD506F.5000900@ukr.net> <49BEB66E.2040501@lpta.in2p3.fr> <9457e7c80903170144w5b7d9f87p568104f9dad81e15@mail.gmail.com> <49BF6351.3080203@ukr.net> <9457e7c80903170609g5214f970sec5b9f17b60d5a04@mail.gmail.com> Message-ID: <9457e7c80904051448y185276a7m84c5bb5e770b681d@mail.gmail.com> 2009/4/4 Matthieu Brucher : > GenericOpt is not finished, I think. It lacks several tools, but the > basis is there. For the moment, the design is ready for any > unconstrained optimization, provided you implement some of the missing > tools (for instance rank updates, ...). > The problem is, I don't have much time to complete this now I'm > outside the university and I don't work that much with Python ATM :( I would hate to see GenericOpt disappear, though. I think it can be a very handy educational and research tool. If we document its deficiencies, it can be packaged as a SciKit? Regards St?fan From josef.pktd at gmail.com Sun Apr 5 21:46:40 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 5 Apr 2009 21:46:40 -0400 Subject: [SciPy-user] another gaussian_kde question Message-ID: <1cd32cbb0904051846n1d70af0as501394c8f15cc735@mail.gmail.com> Does anybody know what integrate_kde and integrate_gaussian in stats.kde are good for? I was trying for a while to find out what they can be used for, but so far without success. def integrate_kde(self, other): """Computes the integral of the product of this kernel density estimate with another. def integrate_gaussian(self, mean, cov): """Multiply estimated density by a multivariate Gaussian and integrate over the wholespace. Josef From robert.kern at gmail.com Sun Apr 5 22:06:35 2009 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 5 Apr 2009 21:06:35 -0500 Subject: [SciPy-user] another gaussian_kde question In-Reply-To: <1cd32cbb0904051846n1d70af0as501394c8f15cc735@mail.gmail.com> References: <1cd32cbb0904051846n1d70af0as501394c8f15cc735@mail.gmail.com> Message-ID: <3d375d730904051906r4a4b4e2dv2a29d26b19b26f89@mail.gmail.com> On Sun, Apr 5, 2009 at 20:46, wrote: > Does anybody know what integrate_kde and integrate_gaussian in > stats.kde are good for? I was trying for a while to find out what they > can be used for, but so far without success. > > ? ?def integrate_kde(self, other): > ? ? ? ?"""Computes the integral of the product of this kernel density estimate > ? ? ? ?with another. > ? ?def integrate_gaussian(self, mean, cov): > ? ? ? ?"""Multiply estimated density by a multivariate Gaussian and integrate > ? ? ? ?over the wholespace. My particular use case at the time I wrote gaussian_kde involved a Bayesian model comparison. The computations required integrations of a KDE against a rectangular uniform distribution (hence mvndst), a Gaussian distribution, or another KDE. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From josef.pktd at gmail.com Sun Apr 5 22:33:31 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 5 Apr 2009 22:33:31 -0400 Subject: [SciPy-user] another gaussian_kde question In-Reply-To: <3d375d730904051906r4a4b4e2dv2a29d26b19b26f89@mail.gmail.com> References: <1cd32cbb0904051846n1d70af0as501394c8f15cc735@mail.gmail.com> <3d375d730904051906r4a4b4e2dv2a29d26b19b26f89@mail.gmail.com> Message-ID: <1cd32cbb0904051933n25eea150q4ded7d99dcc5ec95@mail.gmail.com> On Sun, Apr 5, 2009 at 10:06 PM, Robert Kern wrote: > On Sun, Apr 5, 2009 at 20:46, ? wrote: >> Does anybody know what integrate_kde and integrate_gaussian in >> stats.kde are good for? I was trying for a while to find out what they >> can be used for, but so far without success. >> >> ? ?def integrate_kde(self, other): >> ? ? ? ?"""Computes the integral of the product of this kernel density estimate >> ? ? ? ?with another. >> ? ?def integrate_gaussian(self, mean, cov): >> ? ? ? ?"""Multiply estimated density by a multivariate Gaussian and integrate >> ? ? ? ?over the wholespace. > > My particular use case at the time I wrote gaussian_kde involved a > Bayesian model comparison. The computations required integrations of a > KDE against a rectangular uniform distribution (hence mvndst), a > Gaussian distribution, or another KDE. > Thanks, Do you still have the reference for this? A hint for what the use cases for the functions are would be useful in the docs. I was also wondering whether it can be used for cross validation of the bandwith, but I don't have the time right now to figure this out. I'm just writing some tests in preparation for introducing a keyword parameter as option for bandwith selection as we discussed a while ago. covariance factor, covfact = 'scotts', 'silverman' or a float (or maybe float array (1,n) if I figure out how to have different covariance factors by array dimension) Josef From matthieu.brucher at gmail.com Mon Apr 6 01:47:35 2009 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 6 Apr 2009 07:47:35 +0200 Subject: [SciPy-user] [ANN][optimization] OpenOpt 0.23 In-Reply-To: <9457e7c80904051448y185276a7m84c5bb5e770b681d@mail.gmail.com> References: <49BD506F.5000900@ukr.net> <49BEB66E.2040501@lpta.in2p3.fr> <9457e7c80903170144w5b7d9f87p568104f9dad81e15@mail.gmail.com> <49BF6351.3080203@ukr.net> <9457e7c80903170609g5214f970sec5b9f17b60d5a04@mail.gmail.com> <9457e7c80904051448y185276a7m84c5bb5e770b681d@mail.gmail.com> Message-ID: 2009/4/5 St?fan van der Walt : > 2009/4/4 Matthieu Brucher : >> GenericOpt is not finished, I think. It lacks several tools, but the >> basis is there. For the moment, the design is ready for any >> unconstrained optimization, provided you implement some of the missing >> tools (for instance rank updates, ...). >> The problem is, I don't have much time to complete this now I'm >> outside the university and I don't work that much with Python ATM :( > > I would hate to see GenericOpt disappear, though. ?I think it can be a > very handy educational and research tool. ?If we document its > deficiencies, it can be packaged as a SciKit? Yes, it can. Besides, it is needed by learn. I don't have the time right now to do the switch (and update learn), but I'll try. Matthieu -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From dave.hirschfeld at gmail.com Mon Apr 6 06:16:41 2009 From: dave.hirschfeld at gmail.com (Dave Hirschfeld) Date: Mon, 6 Apr 2009 10:16:41 +0000 (UTC) Subject: [SciPy-user] =?utf-8?q?cdf_and_integration_for_multivariate_norma?= =?utf-8?q?l=09distribution_in_stats=2Ekde?= References: <1cd32cbb0901101829w3890212cwa18694d958b6700b@mail.gmail.com> <3d375d730901101835u4e644898n50424f436b545775@mail.gmail.com> <1cd32cbb0901101853q4696aeb4l7e21c10aee1c92ee@mail.gmail.com> <1cd32cbb0903200755o3653ed82v2b9040fcd40288f1@mail.gmail.com> <1cd32cbb0903310804j128ae041w657970e77dc9f49@mail.gmail.com> Message-ID: gmail.com> writes: > > A question on copulas got me initially started to look for the > multivariate normal cdf. I would be nice to have also the cdf of the > multivariate t distribution by Alan Genz, the author of mvndst. > > Do you have a simple example for the normal copula that you are > willing to share for the cookbook? > > Thanks for the comments, > > Josef > The multivariate t-distribution would also be very useful! I did have plans to post the working copula function to the list in the hope that others may find it useful and to get any feedback but unfortunately my hard drive crashed and I hadn't committed the code to my repository :( I'll still look to post it to the list or as a cookbook recipe once I find the time to re-create it. -Dave From esemenko at gmail.com Mon Apr 6 06:32:43 2009 From: esemenko at gmail.com (Eugene Semenko) Date: Mon, 6 Apr 2009 14:32:43 +0400 Subject: [SciPy-user] selection from array Message-ID: <23435cd40904060332q227a5a0as20608587ec02937f@mail.gmail.com> Hello! I'm new in Python and just started to learn it. My question is about method to make selection from array. Suppose there is an array: arr = np.arange(20,300,0.5) How can I select from this array elements where 30.5 < arr < 125? Sincerely, Eugene. -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott.sinclair.za at gmail.com Mon Apr 6 06:51:20 2009 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Mon, 6 Apr 2009 12:51:20 +0200 Subject: [SciPy-user] selection from array In-Reply-To: <23435cd40904060332q227a5a0as20608587ec02937f@mail.gmail.com> References: <23435cd40904060332q227a5a0as20608587ec02937f@mail.gmail.com> Message-ID: <6a17e9ee0904060351o389ec154ieecc763d633e648e@mail.gmail.com> > 2009/4/6 Eugene Semenko : > ??? Hello! I'm new in Python and just started to learn it. My question is > about method to make selection from array. Suppose there is an array: > arr = np.arange(20,300,0.5) > How can I select from this array elements where 30.5 < arr < 125? Hi Eugene, >>> arr = np.arange(20, 300, 0.5) >>> cond = (arr > 30.5) & (arr < 125) >>> ind = cond.nonzero() >>> arr[ind] This works because the False values in the boolean array cond evaluate to zero, therefore the nonzero method returns the indices where your condition evaluates to True. You could also use the special case of np.where() to get the indices of the entries where your condition is True. >>> ind = np.where(cond) >>> arr[ind] Cheers, Scott From faltet at pytables.org Mon Apr 6 06:58:39 2009 From: faltet at pytables.org (Francesc Alted) Date: Mon, 6 Apr 2009 12:58:39 +0200 Subject: [SciPy-user] selection from array In-Reply-To: <23435cd40904060332q227a5a0as20608587ec02937f@mail.gmail.com> References: <23435cd40904060332q227a5a0as20608587ec02937f@mail.gmail.com> Message-ID: <200904061258.40424.faltet@pytables.org> A Monday 06 April 2009, Eugene Semenko escrigu?: > Hello! I'm new in Python and just started to learn it. My > question is about method to make selection from array. Suppose there > is an array: arr = np.arange(20,300,0.5) > How can I select from this array elements where 30.5 < arr < 125? arr[(arr > 30.5) & (arr < 125)] This is a FAQ, so perhaps it would be worth if someone can add this example (a boolean composition of two conditions) to: http://docs.scipy.org/doc/numpy/user/basics.indexing.html#boolean-or-mask-index-arrays (or whatever place this is appropriate). -- Francesc Alted "One would expect people to feel threatened by the 'giant brains or machines that think'. In fact, the frightening computer becomes less frightening if it is used only to simulate a familiar noncomputer." -- Edsger W. Dykstra "On the cruelty of really ting computer science" From stefan at sun.ac.za Mon Apr 6 09:03:11 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 6 Apr 2009 15:03:11 +0200 Subject: [SciPy-user] [ANN][optimization] OpenOpt 0.23 In-Reply-To: References: <49BD506F.5000900@ukr.net> <49BEB66E.2040501@lpta.in2p3.fr> <9457e7c80903170144w5b7d9f87p568104f9dad81e15@mail.gmail.com> <49BF6351.3080203@ukr.net> <9457e7c80903170609g5214f970sec5b9f17b60d5a04@mail.gmail.com> <9457e7c80904051448y185276a7m84c5bb5e770b681d@mail.gmail.com> Message-ID: <9457e7c80904060603o627e752dsc127ca29994b99ad@mail.gmail.com> 2009/4/6 Matthieu Brucher : >> I would hate to see GenericOpt disappear, though. ?I think it can be a >> very handy educational and research tool. ?If we document its >> deficiencies, it can be packaged as a SciKit? > > Yes, it can. Besides, it is needed by learn. I don't have the time > right now to do the switch (and update learn), but I'll try. Thanks, Matthieu. If David doesn't mind, we can even include the code in "learn" for now, with an eye on factoring it out later. That way, we don't lose the code, and you don't have to setup all the infrastructure from scratch. Regards St?fan From matthieu.brucher at gmail.com Mon Apr 6 09:06:30 2009 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 6 Apr 2009 15:06:30 +0200 Subject: [SciPy-user] [ANN][optimization] OpenOpt 0.23 In-Reply-To: <9457e7c80904060603o627e752dsc127ca29994b99ad@mail.gmail.com> References: <49BD506F.5000900@ukr.net> <49BEB66E.2040501@lpta.in2p3.fr> <9457e7c80903170144w5b7d9f87p568104f9dad81e15@mail.gmail.com> <49BF6351.3080203@ukr.net> <9457e7c80903170609g5214f970sec5b9f17b60d5a04@mail.gmail.com> <9457e7c80904051448y185276a7m84c5bb5e770b681d@mail.gmail.com> <9457e7c80904060603o627e752dsc127ca29994b99ad@mail.gmail.com> Message-ID: 2009/4/6 St?fan van der Walt : > 2009/4/6 Matthieu Brucher : >>> I would hate to see GenericOpt disappear, though. ?I think it can be a >>> very handy educational and research tool. ?If we document its >>> deficiencies, it can be packaged as a SciKit? >> >> Yes, it can. Besides, it is needed by learn. I don't have the time >> right now to do the switch (and update learn), but I'll try. > > Thanks, Matthieu. ?If David doesn't mind, we can even include the code > in "learn" for now, with an eye on factoring it out later. ?That way, > we don't lose the code, and you don't have to setup all the > infrastructure from scratch. There will be some internal changes to do (as well as in learn), so David may have other stuff to worry about :| The infrastructure for GenericOpt is very simple: a simple setup.py is enough as it is a pure Python package. It's more a question of checking every location where it is needed: - GenericOpt itself, - learn in the code, - learn in the setup step, which means uploading the package to pypi. Matthieu -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From david at ar.media.kyoto-u.ac.jp Mon Apr 6 08:57:47 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 06 Apr 2009 21:57:47 +0900 Subject: [SciPy-user] [ANN][optimization] OpenOpt 0.23 In-Reply-To: <9457e7c80904060603o627e752dsc127ca29994b99ad@mail.gmail.com> References: <49BD506F.5000900@ukr.net> <49BEB66E.2040501@lpta.in2p3.fr> <9457e7c80903170144w5b7d9f87p568104f9dad81e15@mail.gmail.com> <49BF6351.3080203@ukr.net> <9457e7c80903170609g5214f970sec5b9f17b60d5a04@mail.gmail.com> <9457e7c80904051448y185276a7m84c5bb5e770b681d@mail.gmail.com> <9457e7c80904060603o627e752dsc127ca29994b99ad@mail.gmail.com> Message-ID: <49D9FC4B.6040708@ar.media.kyoto-u.ac.jp> St?fan van der Walt wrote: > > Thanks, Matthieu. If David doesn't mind, we can even include the code > in "learn" for now, with an eye on factoring it out later. That way, > we don't lose the code, and you don't have to setup all the > infrastructure from scratch. > To be honest, the learn scikits is in such a limbo stage that at this point, it can't be worse. So Matthieu can go ahead, I won't mind, cheers, David From bsouthey at gmail.com Mon Apr 6 09:46:40 2009 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 06 Apr 2009 08:46:40 -0500 Subject: [SciPy-user] removal of stats.mean et al breaks untested code In-Reply-To: <1cd32cbb0904051336p61d06ff0o279b8460d9d0e993@mail.gmail.com> References: <1cd32cbb0904051336p61d06ff0o279b8460d9d0e993@mail.gmail.com> Message-ID: <49DA07C0.2080701@gmail.com> josef.pktd at gmail.com wrote: > I just found another victim of the removal of stats.mean and stats.var > > stats.gaussian_kde doesn't have any tests, and when I tried to rewrite > the option to set the smoothing factor, I found that several methods > are broken that use the removed stats.mean and stats.var. > > If the users of scipy have code similar to the one in scipy.stats, > without sufficient test coverage, then fixing all the hidden > (untested) usages of removed functions can be pretty painful and will > take quite a bit of time. > > In case that the release cycle of scipy gets shorter, we might want to > consider a longer depreciation period. > > Josef > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > The only other one that I could easily find by looking for the files that import stats: stats/_support.py in the collapse function line 179: if cfcn is None: cfcn = stats.mean a = asarray(a) Bruce From matthieu.brucher at gmail.com Mon Apr 6 09:50:03 2009 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 6 Apr 2009 15:50:03 +0200 Subject: [SciPy-user] [ANN][optimization] OpenOpt 0.23 In-Reply-To: <49D9FC4B.6040708@ar.media.kyoto-u.ac.jp> References: <49BD506F.5000900@ukr.net> <9457e7c80903170144w5b7d9f87p568104f9dad81e15@mail.gmail.com> <49BF6351.3080203@ukr.net> <9457e7c80903170609g5214f970sec5b9f17b60d5a04@mail.gmail.com> <9457e7c80904051448y185276a7m84c5bb5e770b681d@mail.gmail.com> <9457e7c80904060603o627e752dsc127ca29994b99ad@mail.gmail.com> <49D9FC4B.6040708@ar.media.kyoto-u.ac.jp> Message-ID: 2009/4/6 David Cournapeau : > St?fan van der Walt wrote: >> >> Thanks, Matthieu. ?If David doesn't mind, we can even include the code >> in "learn" for now, with an eye on factoring it out later. ?That way, >> we don't lose the code, and you don't have to setup all the >> infrastructure from scratch. >> > > To be honest, the learn scikits is in such a limbo stage that at this > point, it can't be worse. So Matthieu can go ahead, I won't mind, > > cheers, Let's be honest, if I have some time to put GenericOpt in learn, I would also have the time to put it in a specific scikit (won't be much longer) ;) Matthieu -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From josef.pktd at gmail.com Mon Apr 6 10:49:13 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 6 Apr 2009 10:49:13 -0400 Subject: [SciPy-user] removal of stats.mean et al breaks untested code In-Reply-To: <49DA07C0.2080701@gmail.com> References: <1cd32cbb0904051336p61d06ff0o279b8460d9d0e993@mail.gmail.com> <49DA07C0.2080701@gmail.com> Message-ID: <1cd32cbb0904060749t2db8111cv9fcc6f4a98fccb2c@mail.gmail.com> On Mon, Apr 6, 2009 at 9:46 AM, Bruce Southey wrote: > josef.pktd at gmail.com wrote: >> I just found another victim of the removal of stats.mean and stats.var >> >> stats.gaussian_kde doesn't have any tests, and when I tried to rewrite >> the option to set the smoothing factor, I found that several methods >> are broken that use the removed stats.mean and stats.var. >> >> If the users of scipy have code similar to the one in scipy.stats, >> without sufficient test coverage, then fixing all the hidden >> (untested) usages of removed functions can be pretty painful and will >> take quite a bit of time. >> >> In case that the release cycle of scipy gets shorter, we might want to >> consider a longer depreciation period. >> >> Josef >> _______________________________________________ >> SciPy-user mailing list >> SciPy-user at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > The only other one that I could easily find by looking for the files > that import stats: > stats/_support.py in the collapse function line 179: > ? ?if cfcn is None: > ? ? ? ?cfcn = stats.mean > ? ?a = asarray(a) > Thank you for finding this. I tried to fix this function, but it is more thoroughly broken. I never looked at function in stats._support that are not used anywhere else, like collapse. From the description, it actually looks like a useful function, there where some questions on the mailing list asking for something similar. But the current collapse mixes inconsistently array and list operations, and after a short time, I only get the simplest case to work. I have a rewrite in a similar function that, I guess, will be faster. So for now I will just open a ticket to fix this. Josef From mattknox.ca at gmail.com Mon Apr 6 17:40:02 2009 From: mattknox.ca at gmail.com (matt knox) Date: Mon, 6 Apr 2009 17:40:02 -0400 Subject: [SciPy-user] [ANN] scikits.timeseries 0.91.0 Message-ID: We are pleased to announce the release of scikits.timeseries 0.91.0 This is the first official public release of the scikits.timeseries module. Due to the long birthing period, the api is considered stable and the version number was chosen to reflect the relative maturity of the initial release compared to a typical first release. Home page: http://pytseries.sourceforge.net/ Please see the website for installation requirements and download details. Note that the recently released numpy 1.3.0 is a strict requirement. Windows binaries are provided for Python 2.5 and 2.6. About the package ================= The scikits.timeseries module provides classes and functions for manipulating, reporting, and plotting time series of various frequencies. The focus is on convenient data access and manipulation while leveraging the existing mathematical functionality in numpy and scipy. If the following scenarios sound familiar to you, then you will likely find the scikits.timeseries module useful: * Compare many time series with different ranges of data (eg. stock prices); * Create time series plots with intelligently spaced axis labels; * Convert a daily time series to monthly by taking the average value during each month; * Work with data that has missing values; * Determine the last business day of the previous month/quarter/year for reporting purposes; * Compute a moving standard deviation efficiently; These are just some of the scenarios that are made very simple with the scikits.timeseries module. Thanks, Matt Knox & Pierre Gerard-Marchant -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Apr 6 23:07:10 2009 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 6 Apr 2009 22:07:10 -0500 Subject: [SciPy-user] another gaussian_kde question In-Reply-To: <1cd32cbb0904051933n25eea150q4ded7d99dcc5ec95@mail.gmail.com> References: <1cd32cbb0904051846n1d70af0as501394c8f15cc735@mail.gmail.com> <3d375d730904051906r4a4b4e2dv2a29d26b19b26f89@mail.gmail.com> <1cd32cbb0904051933n25eea150q4ded7d99dcc5ec95@mail.gmail.com> Message-ID: <3d375d730904062007v307577bvd5bd61ad02e8d26e@mail.gmail.com> On Sun, Apr 5, 2009 at 21:33, wrote: > On Sun, Apr 5, 2009 at 10:06 PM, Robert Kern wrote: >> On Sun, Apr 5, 2009 at 20:46, ? wrote: >>> Does anybody know what integrate_kde and integrate_gaussian in >>> stats.kde are good for? I was trying for a while to find out what they >>> can be used for, but so far without success. >>> >>> ? ?def integrate_kde(self, other): >>> ? ? ? ?"""Computes the integral of the product of this kernel density estimate >>> ? ? ? ?with another. >>> ? ?def integrate_gaussian(self, mean, cov): >>> ? ? ? ?"""Multiply estimated density by a multivariate Gaussian and integrate >>> ? ? ? ?over the wholespace. >> >> My particular use case at the time I wrote gaussian_kde involved a >> Bayesian model comparison. The computations required integrations of a >> KDE against a rectangular uniform distribution (hence mvndst), a >> Gaussian distribution, or another KDE. >> > > Thanks, > Do you still have the reference for this? A hint for what the use > cases for the functions are would be useful in the docs. Not really, sorry. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david.huard at gmail.com Tue Apr 7 09:54:28 2009 From: david.huard at gmail.com (David Huard) Date: Tue, 7 Apr 2009 09:54:28 -0400 Subject: [SciPy-user] [ANN] scikits.timeseries 0.91.0 In-Reply-To: References: Message-ID: <91cf711d0904070654n5c46578akb2ee42693cb4e460@mail.gmail.com> Congratulations to Matt and Pierre, I use the timeseries scikit on a regular basis and it's a really useful piece of work. Regards, David On Mon, Apr 6, 2009 at 5:40 PM, matt knox wrote: > We are pleased to announce the release of scikits.timeseries 0.91.0 > > This is the first official public release of the scikits.timeseries module. > Due > to the long birthing period, the api is considered stable and the version > number was chosen to reflect the relative maturity of the initial release > compared to a typical first release. > > Home page: http://pytseries.sourceforge.net/ > Please see the website for installation requirements and download details. > Note that the recently released numpy 1.3.0 is a strict requirement. > > Windows binaries are provided for Python 2.5 and 2.6. > > About the package > ================= > > The scikits.timeseries module provides classes and functions for > manipulating, > reporting, and plotting time series of various frequencies. The focus is on > convenient data access and manipulation while leveraging the existing > mathematical functionality in numpy and scipy. > > If the following scenarios sound familiar to you, then you will likely > find the scikits.timeseries module useful: > > * Compare many time series with different ranges of data (eg. stock > prices); > * Create time series plots with intelligently spaced axis labels; > * Convert a daily time series to monthly by taking the average value > during > each month; > * Work with data that has missing values; > * Determine the last business day of the previous month/quarter/year for > reporting purposes; > * Compute a moving standard deviation efficiently; > > These are just some of the scenarios that are made very simple with the > scikits.timeseries module. > > Thanks, > Matt Knox & Pierre Gerard-Marchant > > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abiel.x.reinhart at jpmchase.com Tue Apr 7 12:17:29 2009 From: abiel.x.reinhart at jpmchase.com (Abiel X Reinhart) Date: Tue, 7 Apr 2009 12:17:29 -0400 Subject: [SciPy-user] Performance problem/suggestion for scikits.timeseries.convert Message-ID: I have recently begun working with the useful scikits.timeseries package, and noticed some performance issues in the ts.convert() function. For example, when converting 1000 monthly values to a quarterly frequency using the ma.mean() function, it took me about 0.6 seconds. This isn't that bad, but it definitely can become an issue when working with many series or longer timespans. After looking at the scikits.timeseries source code, I found essentially all the delay was coming from the ma.apply_along_axis() call inside _convert1d() function. I am not that familiar with the numpy functions, but it seems that ma.apply_along_axis can be be rather slow. For instance, consider the following code: a = np.arange(300000).reshape(30000,10) b = ma.mean(a,-1) c = ma.apply_along_axis(ma.mean, -1, a) In this example, b = c, but b is generated much quicker. My system was always able to generate b in less than 0.02 seconds. but took about 4.3 seconds to generate c. Perhaps an improvement could be made to the convert() function by recognizing a standard set of built-in numpy functions like ma.mean and applying the method used to generate "b" above, and only using ma.apply_along_axis() for custom functions. Since I imagine most people use standard aggregation functions like ma.mean and ma.sum, this could lead to a big speed improvement. I am building a GUI application, and this would make the difference between an application that reacts essentially instantly and one that hangs slightly in many situations. One other possible solution seems to be leave scikits.timeseries unchanged, and do something like the following: Let t be a monthly time series. t = t.convert(freq="Q") t = ts.time_series(ma.mean(t,-1), freq="Q", start_date=t.start_date) The downside of this is its just more verbose, and many users may not even think of it. Thanks very much. Abiel This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is privileged, confidential, legally privileged, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Tue Apr 7 12:38:54 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 7 Apr 2009 12:38:54 -0400 Subject: [SciPy-user] Performance problem/suggestion for scikits.timeseries.convert In-Reply-To: References: Message-ID: <95869807-8047-42CB-9C53-BE4F981DAB4A@gmail.com> Abiel, Thanks a lot for the report. There could be some improvement to do on ma.apply_along_axis, but I'm afraid I won;'t be able to work on it any time soon. I'll try to find a substitute to apply_along_axis in the mean time, we'll keep you posted. Personally, I rarely use the func parameter of convert, preferring to deal directly w/ the 2D series a basic .convert(.., func=None) outputs. It's quite useful when you have several operations to perform on the same series (eg, getting the mean & std deviation...), as you convert the series only once. Note that in your example, you shouldn't have to recreate a series: qt = t.convert(freq="Q") mqt = qt.mean(-1) Mmh. We really need to get these "Examples/FAQ" sections in the doc... Cheers and thx again P. On Apr 7, 2009, at 12:17 PM, Abiel X Reinhart wrote: > I have recently begun working with the useful scikits.timeseries > package, and noticed some performance issues in the ts.convert() > function. For example, when converting 1000 monthly values to a > quarterly frequency using the ma.mean() function, it took me about > 0.6 seconds. This isn't that bad, but it definitely can become an > issue when working with many series or longer timespans. > > After looking at the scikits.timeseries source code, I found > essentially all the delay was coming from the ma.apply_along_axis() > call inside _convert1d() function. I am not that familiar with the > numpy functions, but it seems that ma.apply_along_axis can be be > rather slow. For instance, consider the following code: > > a = np.arange(300000).reshape(30000,10) > b = ma.mean(a,-1) > c = ma.apply_along_axis(ma.mean, -1, a) > > In this example, b = c, but b is generated much quicker. My system > was always able to generate b in less than 0.02 seconds. but took > about 4.3 seconds to generate c. > > Perhaps an improvement could be made to the convert() function by > recognizing a standard set of built-in numpy functions like ma.mean > and applying the method used to generate "b" above, and only using > ma.apply_along_axis() for custom functions. Since I imagine most > people use standard aggregation functions like ma.mean and ma.sum, > this could lead to a big speed improvement. I am building a GUI > application, and this would make the difference between an > application that reacts essentially instantly and one that hangs > slightly in many situations. > > One other possible solution seems to be leave scikits.timeseries > unchanged, and do something like the following: > > Let t be a monthly time series. > > t = t.convert(freq="Q") > t = ts.time_series(ma.mean(t,-1), freq="Q", start_date=t.start_date) > > The downside of this is its just more verbose, and many users may > not even think of it. > > Thanks very much. > > Abiel > > > > This communication is for informational purposes only. It is not > intended as an offer or solicitation for the purchase or sale of any > financial instrument or as an official confirmation of any > transaction. All market prices, data and other information are not > warranted as to completeness or accuracy and are subject to change > without notice. Any comments or statements made herein do not > necessarily reflect those of JPMorgan Chase & Co., its subsidiaries > and affiliates. This transmission may contain information that is > privileged, confidential, legally privileged, and/or exempt from > disclosure under applicable law. If you are not the intended > recipient, you are hereby notified that any disclosure, copying, > distribution, or use of the information contained herein (including > any reliance thereon) is STRICTLY PROHIBITED. Although this > transmission and any attachments are believed to be free of any > virus or other defect that might affect any computer system into > which it is received and opened, it is the responsibility of the > recipient to ensure that it is virus free and no responsibility is > accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, > as applicable, for any loss or damage arising in any way from its > use. If you received this transmission in error, please immediately > contact the sender and destroy the material in its entirety, whether > in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures > for disclosures relating to European legal entities. > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From mattknox.ca at gmail.com Tue Apr 7 12:40:56 2009 From: mattknox.ca at gmail.com (Matt Knox) Date: Tue, 7 Apr 2009 16:40:56 +0000 (UTC) Subject: [SciPy-user] =?utf-8?q?Performance_problem/suggestion_for=09sciki?= =?utf-8?q?ts=2Etimeseries=2Econvert?= References: Message-ID: > Perhaps an improvement could be made to the convert() > function by recognizing a standard set of built-in numpy functions like ma.mean > and applying the method used to generate "b" above, and only using > ma.apply_along_axis() for custom functions. Since I imagine most people use > standard aggregation functions like ma.mean and ma.sum, this could lead to a > big speed improvement. I am building a GUI application, and this would make the > difference between an application that reacts essentially instantly and one > that hangs slightly in many situations. Hi Alex, thanks for your feedback. That is a good suggestion. Would you mind adding a note to the scikits bug/feature tracker for this? http://scipy.org/scipy/scikits/query?component=timeseries&order=status Also, patches are welcome and would increase the chances of it getting into a future release quickly. Thanks, - Matt From pgmdevlist at gmail.com Tue Apr 7 17:52:00 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 7 Apr 2009 17:52:00 -0400 Subject: [SciPy-user] Compile scipy from SVN on OS X: no clapack ? Message-ID: <082B0946-B5C8-4F8C-AAA4-681CCA52C54F@gmail.com> All, I'm working on a piece of code that requires numerous calls to scipy.linalg.eigvals. I was surprised by the slowness of the code, so I figured I'd check whether my installation of scipy was correct. Turns out it's probably not: running the tests, I end up w/ some messages indicating that clapack is empty. That never really mattered until now... Could anybody try and hold my hand on this one (I'm still a newbie on Mac) ? * I do have an Accelerate framework in my /System/Library * The python I use is the 2.5.4 from macports * I have a macports atlas (v3.8.2) * I do want to compile from source: I usually install scipy/numpy w/ python -c "import setuptools; execfile('setup.py')" develop in virtual environments (that way, I can get the latest changes I make straight away). * extra config info below Obviously I'm missing some configuration step somewhere, or I'm mixing things that shouldn't be. Note that following the info in INSTALL.txt, I tried to run python setup_atlas_version.py build_ext --inplace --force but failed w/ a Traceback (most recent call last): File "setup_atlas_version.py", line 7, in from numpy.distutils.misc_util import get_path, default_config_dict ImportError: cannot import name get_path Any help welcome. Thanks an awful lot in advance P. __________ >>> python setup.py build 2>&1 | tee build.log File downloadable at http://rapidshare.com/files/218658464/build.log.html __________ >>> python -c "from numpy.f2py.diagnose import run; run()" ------ os.name='posix' ------ sys.platform='darwin' ------ sys.version: 2.5.4 (r254:67916, Apr 1 2009, 17:15:22) [GCC 4.0.1 (Apple Inc. build 5490)] ------ sys.prefix: /Users/pierregm/Computing/.pythonenvs/default/bin/.. ------ sys.path=':/Users/pierregm/Computing/.pythonenvs/default/lib/python2.5/ site-packages/setuptools-0.6c9-py2.5.egg:/Users/pierregm/ Computing/.pythonenvs/default/lib/python2.5/site-packages/ ipython-0.9.1-py2.5.egg:/Users/pierregm/Computing/workspace/svn_numpy:/ Users/pierregm/Computing/.pythonenvs/default/lib/python2.5/site- packages/nose-0.10.4-py2.5.egg:/Users/pierregm/Computing/workspace/ svn_scipy:/Users/pierregm/Computing/.pythonenvs/default/lib/python2.5/ site-packages/BeautifulSoup-3.1.0.1-py2.5.egg:/Users/pierregm/ Computing/.pythonenvs/default/lib/python2.5/site-packages/xlrd-0.6.1- py2.5.egg:/Users/pierregm/Computing/.pythonenvs/default/lib/python2.5/ site-packages/Sphinx-0.6.1-py2.5.egg:/Users/pierregm/ Computing/.pythonenvs/default/lib/python2.5/site-packages/Jinja2-2.1.1- py2.5-macosx-10.5-i386.egg:/Users/pierregm/Computing/.pythonenvs/ default/lib/python2.5/site-packages/SquareMap-1.0.0b23-py2.5.egg:/ Users/pierregm/Computing/.pythonenvs/default/lib/python2.5/site- packages/RunSnakeRun-2.0.0b4-py2.5.egg:/opt/local/Library/Frameworks/ Python.framework/Versions/2.5/lib/python2.5/site-packages/Jinja-1.2- py2.5-macosx-10.3-i386.egg:/opt/local/Library/Frameworks/ Python.framework/Versions/2.5/lib/python2.5/site-packages/ Pygments-0.11.1-py2.5.egg:/opt/local/Library/Frameworks/ Python.framework/Versions/2.5/lib/python2.5/site-packages/yolk-0.4.1- py2.5.egg:/opt/local/Library/Frameworks/Python.framework/Versions/2.5/ lib/python2.5/site-packages/Sphinx-0.5-py2.5.egg:/opt/local/Library/ Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/ cocos2d-0.3.0-py2.5.egg:/opt/local/Library/Frameworks/Python.framework/ Versions/2.5/lib/python2.5/site-packages/virtualenv-1.3.2-py2.5.egg:/ Users/pierregm/Computing/workspace/hydroclimpy:/Users/pierregm/ Computing/workspace/svn_scikits_timeseries:/Library/Frameworks/ GDAL.framework/Versions/1.6/lib:/Users/pierregm/Computing/.pythonenvs/ default/lib/python25.zip:/Users/pierregm/Computing/.pythonenvs/default/ lib/python2.5:/Users/pierregm/Computing/.pythonenvs/default/lib/ python2.5/plat-darwin:/Users/pierregm/Computing/.pythonenvs/default/ lib/python2.5/plat-mac:/Users/pierregm/Computing/.pythonenvs/default/ lib/python2.5/plat-mac/lib-scriptpackages:/Users/pierregm/ Computing/.pythonenvs/default/lib/python2.5/lib-tk:/Users/pierregm/ Computing/.pythonenvs/default/lib/python2.5/lib-dynload:/opt/local/ Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5:/opt/ local/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/ lib-tk:/opt/local/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/plat-darwin:/opt/local/Library/Frameworks/Python.framework/ Versions/2.5/lib/python2.5/plat-mac:/opt/local/Library/Frameworks/ Python.framework/Versions/2.5/lib/python2.5/plat-mac/lib- scriptpackages:/Users/pierregm/Computing/.pythonenvs/default/lib/ python2.5/site-packages:/opt/local/Library/Frameworks/Python.framework/ Versions/2.5/lib/python2.5/site-packages:/opt/local/Library/Frameworks/ Python.framework/Versions/2.5/lib/python2.5/site-packages/Numeric:/opt/ local/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/ site-packages/PyObjC:/opt/local/Library/Frameworks/Python.framework/ Versions/2.5/lib/python2.5/site-packages/gtk-2.0:/opt/local/lib/ python2.5/site-packages:/opt/local/lib/svn-python2.5:/opt/local/ Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- packages/wx-2.8-mac-unicode' ------ Failed to import numarray: No module named numarray Found Numeric version '24.2' in /opt/local/Library/Frameworks/ Python.framework/Versions/2.5/lib/python2.5/site-packages/Numeric/ Numeric.pyc Found new numpy version '1.4.0.dev' in /Users/pierregm/Computing/ workspace/svn_numpy/numpy/__init__.pyc Found f2py2e version '2' in /Users/pierregm/Computing/workspace/ svn_numpy/numpy/f2py/f2py2e.pyc Found numpy.distutils version '0.4.0' in '/Users/pierregm/Computing/ workspace/svn_numpy/numpy/distutils/__init__.pyc' ------ Importing numpy.distutils.fcompiler ... ok ------ Checking availability of supported Fortran compilers: Using built-in specs. Target: powerpc-apple-darwin8 Configured with: /Builds/unix/gcc/gcc-4.2/configure --prefix=/usr/ local --mandir=/share/man --program-transform-name=/^[cg][^.-]*$/s/ $/-4.2/ --build=i686-apple-darwin8 --host=i686-apple-darwin8 -- target=powerpc-apple-darwin8 --enable-languages=fortran Thread model: posix gcc version 4.2.3 Using built-in specs. Target: i686-apple-darwin8 Configured with: /Builds/unix/gcc/gcc-4.2/configure --prefix=/usr/ local --mandir=/share/man --program-transform-name=/^[cg][^.-]*$/s/ $/-4.2/ --build=i686-apple-darwin8 --host=i686-apple-darwin8 -- target=i686-apple-darwin8 --enable-languages=fortran Thread model: posix gcc version 4.2.3 Using built-in specs. Target: powerpc-apple-darwin8 Configured with: /Builds/unix/gcc/gcc-4.2/configure --prefix=/usr/ local --mandir=/share/man --program-transform-name=/^[cg][^.-]*$/s/ $/-4.2/ --build=i686-apple-darwin8 --host=i686-apple-darwin8 -- target=powerpc-apple-darwin8 --enable-languages=fortran Thread model: posix gcc version 4.2.3 Using built-in specs. Target: i686-apple-darwin8 Configured with: /Builds/unix/gcc/gcc-4.2/configure --prefix=/usr/ local --mandir=/share/man --program-transform-name=/^[cg][^.-]*$/s/ $/-4.2/ --build=i686-apple-darwin8 --host=i686-apple-darwin8 -- target=i686-apple-darwin8 --enable-languages=fortran Thread model: posix gcc version 4.2.3 Gnu95FCompiler instance properties: archiver = ['/usr/local/bin/gfortran', '-cr'] compile_switch = '-c' compiler_f77 = ['/usr/local/bin/gfortran', '-Wall', '-ffixed- form', '- fno-second-underscore', '-arch', 'ppc', '-arch', 'i686', ' -fPIC', '-O3', '-funroll-loops'] compiler_f90 = ['/usr/local/bin/gfortran', '-Wall', '-fno-second- underscore', '-arch', 'ppc', '-arch', 'i686', '- fPIC', '- O3', '-funroll-loops'] compiler_fix = ['/usr/local/bin/gfortran', '-Wall', '-ffixed- form', '- fno-second-underscore', '-Wall', '-fno-second- underscore', '-arch', 'ppc', '-arch', 'i686', '-fPIC', '-O3', '- funroll -loops'] libraries = ['gfortran'] library_dirs = [] linker_exe = ['/usr/local/bin/gfortran', '-Wall', '-Wall'] linker_so = ['/usr/local/bin/gfortran', '-Wall', '-arch', 'ppc', '- arch', 'i686', '-Wall', '-undefined', 'dynamic_lookup', '- bundle'] object_switch = '-o ' ranlib = ['/usr/local/bin/gfortran'] version = LooseVersion ('4.2.3') version_cmd = ['/usr/local/bin/gfortran', '--version'] Fortran compilers found: --fcompiler=gnu95 GNU Fortran 95 compiler (4.2.3) Compilers available for this platform, but not found: --fcompiler=absoft Absoft Corp Fortran Compiler --fcompiler=g95 G95 Fortran Compiler --fcompiler=gnu GNU Fortran 77 compiler --fcompiler=ibm IBM XL Fortran Compiler --fcompiler=intel Intel Fortran Compiler for 32-bit apps --fcompiler=nag NAGWare Fortran 95 Compiler Compilers not available on this platform: --fcompiler=compaq Compaq Fortran Compiler --fcompiler=hpux HP Fortran 90 Compiler --fcompiler=intele Intel Fortran Compiler for Itanium apps --fcompiler=intelem Intel Fortran Compiler for EM64T-based apps --fcompiler=intelev Intel Visual Fortran Compiler for Itanium apps --fcompiler=intelv Intel Visual Fortran Compiler for 32-bit apps --fcompiler=lahey Lahey/Fujitsu Fortran 95 Compiler --fcompiler=mips MIPSpro Fortran Compiler --fcompiler=none Fake Fortran compiler --fcompiler=pg Portland Group Fortran Compiler --fcompiler=sun Sun or Forte Fortran 95 Compiler --fcompiler=vast Pacific-Sierra Research Fortran 90 Compiler For compiler details, run 'config_fc --verbose' setup command. ------ Importing numpy.distutils.cpuinfo ... ok ------ CPU information: CPUInfoBase__get_nbits getNCPUs is_32bit is_i386 ------ From david at ar.media.kyoto-u.ac.jp Tue Apr 7 22:41:11 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 08 Apr 2009 11:41:11 +0900 Subject: [SciPy-user] Compile scipy from SVN on OS X: no clapack ? In-Reply-To: <082B0946-B5C8-4F8C-AAA4-681CCA52C54F@gmail.com> References: <082B0946-B5C8-4F8C-AAA4-681CCA52C54F@gmail.com> Message-ID: <49DC0EC7.60302@ar.media.kyoto-u.ac.jp> Pierre GM wrote: > All, > I'm working on a piece of code that requires numerous calls to > scipy.linalg.eigvals. I was surprised by the slowness of the code, so > I figured I'd check whether my installation of scipy was correct. > When you said surprised by the slowness, is it comparatively to scipy on another OS on the same/comparable machine ? > Turns out it's probably not: running the tests, I end up w/ some > messages indicating that clapack is empty. That never really mattered > until now... > Accelerate does not have a clapack interface, IIRC. It should not matter much if you have an accelerated lapack - maybe the different you could see, assuming it is faster on linux, is due to atlas having an optimized version on Linux but not on mac os x (Accelerate mostly use atlas, I believe - and atlas recently included more optimized lapack functions). cheers, David From pgmdevlist at gmail.com Wed Apr 8 03:01:41 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 8 Apr 2009 03:01:41 -0400 Subject: [SciPy-user] Compile scipy from SVN on OS X: no clapack ? In-Reply-To: <49DC0EC7.60302@ar.media.kyoto-u.ac.jp> References: <082B0946-B5C8-4F8C-AAA4-681CCA52C54F@gmail.com> <49DC0EC7.60302@ar.media.kyoto-u.ac.jp> Message-ID: <6ED4714D-079A-4ACD-B037-167B1BB340CA@gmail.com> On Apr 7, 2009, at 10:41 PM, David Cournapeau wrote: > Pierre GM wrote: >> All, >> I'm working on a piece of code that requires numerous calls to >> scipy.linalg.eigvals. I was surprised by the slowness of the code, so >> I figured I'd check whether my installation of scipy was correct. >> > > When you said surprised by the slowness, is it comparatively to > scipy on > another OS on the same/comparable machine ? Oh no, I don't have any particular reference, it's just that ~3s to compute the eigenvalues of a 365x365 matrix looked a bit long (especially as I need to compute a lot of those)... I was just afraid I had missed something somewhere. >> Turns out it's probably not: running the tests, I end up w/ some >> messages indicating that clapack is empty. That never really mattered >> until now... >> > > Accelerate does not have a clapack interface, IIRC. It should not > matter > much if you have an accelerated lapack But I don;t know if I do! Do I need to compile lapack from source, when I have an atlas from macports and the Accelerate framework ? From david at ar.media.kyoto-u.ac.jp Wed Apr 8 03:29:58 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 08 Apr 2009 16:29:58 +0900 Subject: [SciPy-user] Compile scipy from SVN on OS X: no clapack ? In-Reply-To: <6ED4714D-079A-4ACD-B037-167B1BB340CA@gmail.com> References: <082B0946-B5C8-4F8C-AAA4-681CCA52C54F@gmail.com> <49DC0EC7.60302@ar.media.kyoto-u.ac.jp> <6ED4714D-079A-4ACD-B037-167B1BB340CA@gmail.com> Message-ID: <49DC5276.7040805@ar.media.kyoto-u.ac.jp> Pierre GM wrote: > On Apr 7, 2009, at 10:41 PM, David Cournapeau wrote: > > >> Pierre GM wrote: >> >>> All, >>> I'm working on a piece of code that requires numerous calls to >>> scipy.linalg.eigvals. I was surprised by the slowness of the code, so >>> I figured I'd check whether my installation of scipy was correct. >>> >>> >> When you said surprised by the slowness, is it comparatively to >> scipy on >> another OS on the same/comparable machine ? >> > > Oh no, I don't have any particular reference, it's just that ~3s to > compute the eigenvalues of a 365x365 matrix looked a bit long > (especially as I need to compute a lot of those)... I was just afraid > I had missed something somewhere. > Sounds a bit long, but not that much. Finding N eigenvalues for a NxN matrix is N**3. I have just tested on my machine, it takes ~ 1.5 on a random 400x400 matrix, and the machine is a bit old (P4 @ 3ghz), with the debian atlas (that is not very fast). Recent atlas have included quite a few more optimized LAPACK functions, maybe the one used by eigvals is one of them. > But I don;t know if I do! Do I need to compile lapack from source, > when I have an atlas from macports and the Accelerate framework ? > Oh, sorry: I meant that the accelerate framework includes optimized blas (and hopefully some lapack). You did against the accelerate framework, not with your own ATLAS, right ? cheers, David From cournape at gmail.com Wed Apr 8 03:56:48 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 8 Apr 2009 16:56:48 +0900 Subject: [SciPy-user] Memory Leak in Scipy.Signal.lfilter? In-Reply-To: <49D470FC.3030107@gmail.com> References: <49D470FC.3030107@gmail.com> Message-ID: <5b8d13220904080056i168a30ffj2f6bcefbb6d1e443@mail.gmail.com> On Thu, Apr 2, 2009 at 5:02 PM, Richard Lyon wrote: > Hi, > > Anyone else observed the same problem? Ok, I think I fixed the problem in the trunk. Could you test it ? I will backport it for upcoming 0.7.1 as well, cheers, David From rocksportrocker at googlemail.com Wed Apr 8 05:24:37 2009 From: rocksportrocker at googlemail.com (Uwe Schmitt) Date: Wed, 8 Apr 2009 02:24:37 -0700 (PDT) Subject: [SciPy-user] sparse matrix multiplication elementwise ? Message-ID: Hi, is there a way to multiply a sparse matrix with a dense numpy.array elementwise ? Greetings, Uwe From sebastien.maret at gmail.com Wed Apr 8 06:16:30 2009 From: sebastien.maret at gmail.com (Sebastien Maret) Date: Wed, 08 Apr 2009 12:16:30 +0200 Subject: [SciPy-user] Scipy-0.7.0 build error on Mac OSX 10.5.6 Message-ID: Hello, I am trying to compile Scipy-0.7.0 on Mac OS 10.5.6 with gcc-4.3.3. I have numpy-1.3.0 installed. The compilation fails with the following error message: building 'dfftpack' library compiling Fortran sources Fortran f77 compiler: /sw/bin/gfortran -Wall -ffixed-form -fno-second-underscore -arch i686 -fPIC -O3 -funroll-loops Fortran f90 compiler: /sw/bin/gfortran -Wall -fno-second-underscore -arch i686 -fPIC -O3 -funroll-loops Fortran fix compiler: /sw/bin/gfortran -Wall -ffixed-form -fno-second-underscore -Wall -fno-second-underscore -arch i686 -fPIC -O3 -funroll-loops compile options: '-I/sw/lib/python2.6/site-packages/numpy/core/include -c' gfortran:f77: scipy/fftpack/src/dfftpack/dcosqb.f f951: error: unrecognized command line option "-arch" f951: error: unrecognized command line option "-arch" error: Command "/sw/bin/gfortran -Wall -ffixed-form -fno-second-underscore -arch i686 -fPIC -O3 -funroll-loops -I/sw/lib/python2.6/site-packages/numpy/core/include -c -c scipy/fftpack/src/dfftpack/dcosqb.f -o build/temp.macosx-10.5-i386-2.6/scipy/fftpack/src/dfftpack/dcosqb.o" failed with exit status 1 Thanks in advance for you help. Sebastien From Ross.Williamson at usap.gov Wed Apr 8 07:12:03 2009 From: Ross.Williamson at usap.gov (Ross Williamson) Date: Wed, 08 Apr 2009 23:12:03 +1200 Subject: [SciPy-user] Python loops too slow Message-ID: <49DC8683.6070201@usap.gov> Hi All I'm trying to convert some IDL code into Python and am coming across the incredibly slow for loop issue in python. Now I detest IDL but is there any fancy way to make the following code not use for loops? Sorry if this is obvious. def make_ellgrid(ngrid, reso): result = zeros([ngrid, ngrid]) for i in arange((ngrid / 2)+1): for j in arange((ngrid / 2)+1): result[j,i] = 2. * pi * sqrt(i ** 2. + j ** 2) / reso / (ngrid*1.0) for i in xrange(ngrid): result[i,ngrid / 2+1:] = result[i,1:(ngrid / 2)][::-1] for i in xrange(ngrid): result[ngrid / 2+1:,i] = result[1:(ngrid / 2),i][::-1] return result Cheers Ross From gnurser at googlemail.com Wed Apr 8 08:09:48 2009 From: gnurser at googlemail.com (George Nurser) Date: Wed, 8 Apr 2009 13:09:48 +0100 Subject: [SciPy-user] Python loops too slow In-Reply-To: <49DC8683.6070201@usap.gov> References: <49DC8683.6070201@usap.gov> Message-ID: <1d1e6ea70904080509l148f4d4ta44e99e8df3cf9b2@mail.gmail.com> I can see how to do the inner loop of the double loop -- this should speed it up a lot. Also where you can, it's probably best to have the second index as the inner loop, since C and python have the last index as fastest varying i.e. result[j,i],result[j,i+1] are stored adjacently in memory. Try replacing for i in arange((ngrid / 2)+1): for j in arange((ngrid / 2)+1): result[j,i] = 2. * pi * sqrt(i ** 2. + j ** 2) / reso / (ngrid*1.0) by factor = 2. * pi/(reso *ngrid) nhalf = ngrid /2+1 ii = arange(nhalf) for j in arange(nhalf): j2 = j**2 result[j,:nhalf] = factor*sqrt(j2+ii**2) check this gives the right answer, though! HTH. George Nurser. 2009/4/8 Ross Williamson : > Hi All > > I'm trying to convert some IDL code into Python and am coming across the > incredibly slow for loop issue in python. Now I detest IDL but is there > any fancy way to make the following code not use for loops? Sorry if > this is obvious. > > def make_ellgrid(ngrid, reso): > > ? result = zeros([ngrid, ngrid]) > > ? for i in arange((ngrid / 2)+1): > ? ? ?for j in arange((ngrid / 2)+1): > ? ? ? ? result[j,i] = 2. * pi * sqrt(i ** 2. + j ** 2) / reso / (ngrid*1.0) > > ? for i in xrange(ngrid): > ? ? ?result[i,ngrid / 2+1:] = result[i,1:(ngrid / 2)][::-1] > ? for i in xrange(ngrid): > ? ? ?result[ngrid / 2+1:,i] = result[1:(ngrid / 2),i][::-1] > > ? return result > > Cheers > > Ross > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From rocksportrocker at googlemail.com Wed Apr 8 08:37:27 2009 From: rocksportrocker at googlemail.com (Uwe Schmitt) Date: Wed, 8 Apr 2009 05:37:27 -0700 (PDT) Subject: [SciPy-user] sparse matrix multiplication elementwise ? In-Reply-To: References: Message-ID: <031a28ef-ad3b-405c-96d6-25c9bdab1a77@y7g2000yqa.googlegroups.com> I solved my question as follows: import numpy as np import scipy.sparse as sp def scale_sparse_matrix(A, by): assert isinstance(A, sp.csr_matrix), "wrong format" now = 0 for row in range(A.shape[0]): upto = A.indptr[row+1] while now < upto: col = A.indices[now] A.data[now] *= by[row, col] now += 1 if __name__ == "__main__": A = np.zeros((3,4)) A[1,1] = 1 A[2] = 2 aa = sp.csr_matrix(A) print "before:\n", aa b = 2*np.ones_like(A) scale_sparse_matrix(aa, by=b) print "after:\n", aa On 8 Apr., 11:24, Uwe Schmitt wrote: > Hi, > > is there a way to multiply a sparse matrix with a dense numpy.array > elementwise ? > > Greetings, Uwe > _______________________________________________ > SciPy-user mailing list > SciPy-u... at scipy.orghttp://mail.scipy.org/mailman/listinfo/scipy-user From cournape at gmail.com Wed Apr 8 08:51:23 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 8 Apr 2009 21:51:23 +0900 Subject: [SciPy-user] Scipy-0.7.0 build error on Mac OSX 10.5.6 In-Reply-To: References: Message-ID: <5b8d13220904080551t551b800cuebb836bf85b78d7e@mail.gmail.com> Hi Sebastien, On Wed, Apr 8, 2009 at 7:16 PM, Sebastien Maret wrote: > Hello, > > I am trying to compile Scipy-0.7.0 on Mac OS 10.5.6 with gcc-4.3.3. I > have numpy-1.3.0 installed. The compilation fails with the following > error message: > > building 'dfftpack' library > compiling Fortran sources > Fortran f77 compiler: /sw/bin/gfortran -Wall -ffixed-form > -fno-second-underscore -arch i686 -fPIC -O3 -funroll-loops > Fortran f90 compiler: /sw/bin/gfortran -Wall -fno-second-underscore > -arch i686 -fPIC -O3 -funroll-loops > Fortran fix compiler: /sw/bin/gfortran -Wall -ffixed-form > -fno-second-underscore -Wall -fno-second-underscore -arch i686 -fPIC -O3 > -funroll-loops > compile options: '-I/sw/lib/python2.6/site-packages/numpy/core/include > -c' > gfortran:f77: scipy/fftpack/src/dfftpack/dcosqb.f > f951: error: unrecognized command line option "-arch" > f951: error: unrecognized command line option "-arch" > error: Command "/sw/bin/gfortran -Wall -ffixed-form > -fno-second-underscore -arch i686 -fPIC -O3 -funroll-loops > -I/sw/lib/python2.6/site-packages/numpy/core/include -c -c > scipy/fftpack/src/dfftpack/dcosqb.f -o > build/temp.macosx-10.5-i386-2.6/scipy/fftpack/src/dfftpack/dcosqb.o" > failed with exit status 1 You should use a universal capable fortran compiler, like this one: http://r.research.att.com/tools/ cheers, David From cournape at gmail.com Wed Apr 8 08:53:20 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 8 Apr 2009 21:53:20 +0900 Subject: [SciPy-user] Scipy-0.7.0 build error on Mac OSX 10.5.6 In-Reply-To: References: Message-ID: <5b8d13220904080553h2533116asa105c0a3fbc19ace@mail.gmail.com> Hi Sebastien, On Wed, Apr 8, 2009 at 7:16 PM, Sebastien Maret wrote: > Hello, > > I am trying to compile Scipy-0.7.0 on Mac OS 10.5.6 with gcc-4.3.3. I > have numpy-1.3.0 installed. The compilation fails with the following > error message: > > building 'dfftpack' library > compiling Fortran sources > Fortran f77 compiler: /sw/bin/gfortran -Wall -ffixed-form > -fno-second-underscore -arch i686 -fPIC -O3 -funroll-loops > Fortran f90 compiler: /sw/bin/gfortran -Wall -fno-second-underscore > -arch i686 -fPIC -O3 -funroll-loops > Fortran fix compiler: /sw/bin/gfortran -Wall -ffixed-form > -fno-second-underscore -Wall -fno-second-underscore -arch i686 -fPIC -O3 > -funroll-loops > compile options: '-I/sw/lib/python2.6/site-packages/numpy/core/include > -c' > gfortran:f77: scipy/fftpack/src/dfftpack/dcosqb.f > f951: error: unrecognized command line option "-arch" > f951: error: unrecognized command line option "-arch" > error: Command "/sw/bin/gfortran -Wall -ffixed-form > -fno-second-underscore -arch i686 -fPIC -O3 -funroll-loops > -I/sw/lib/python2.6/site-packages/numpy/core/include -c -c > scipy/fftpack/src/dfftpack/dcosqb.f -o > build/temp.macosx-10.5-i386-2.6/scipy/fftpack/src/dfftpack/dcosqb.o" > failed with exit status 1 You should use a universal capable fortran compiler, like this one: http://r.research.att.com/tools/ cheers, David From argriffi at ncsu.edu Wed Apr 8 09:06:51 2009 From: argriffi at ncsu.edu (alex) Date: Wed, 08 Apr 2009 09:06:51 -0400 Subject: [SciPy-user] Python loops too slow In-Reply-To: <49DC8683.6070201@usap.gov> References: <49DC8683.6070201@usap.gov> Message-ID: <49DCA16B.8080205@ncsu.edu> Ross Williamson wrote: > Hi All > > I'm trying to convert some IDL code into Python and am coming across the > incredibly slow for loop issue in python. Now I detest IDL but is there > any fancy way to make the following code not use for loops? Sorry if > this is obvious. > > def make_ellgrid(ngrid, reso): > > result = zeros([ngrid, ngrid]) > > for i in arange((ngrid / 2)+1): > for j in arange((ngrid / 2)+1): > result[j,i] = 2. * pi * sqrt(i ** 2. + j ** 2) / reso / (ngrid*1.0) > > for i in xrange(ngrid): > result[i,ngrid / 2+1:] = result[i,1:(ngrid / 2)][::-1] > for i in xrange(ngrid): > result[ngrid / 2+1:,i] = result[1:(ngrid / 2),i][::-1] > > return result > > Cheers > > Ross By factoring out a constant and using hypot you might be able to speed up the inner loop without doing anything fancy. Maybe something like: c = ... for i in ...: for j in ...: result[j,i] = hypot(j, i) result = c * result From here it is easy to vectorize hypot. To get started, you can see that: >>> a = np.array([[1, 1], [2, 2]]) >>> np.hypot(a, a.T) array([[ 1.41421356, 2.23606798], [ 2.23606798, 2.82842712]]) This should be like: for i in (1,2): for j in (1,2): result[j,i] = hypot(j,i) Now all you need to do is make the 'a' matrix where a[i,j] is equal to i, and I'll bet you know how to do that fast (I don't because I don't really know numpy). Alex From pav at iki.fi Wed Apr 8 09:14:07 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 8 Apr 2009 13:14:07 +0000 (UTC) Subject: [SciPy-user] Python loops too slow References: <49DC8683.6070201@usap.gov> Message-ID: Wed, 08 Apr 2009 23:12:03 +1200, Ross Williamson kirjoitti: [clip] > result = zeros([ngrid, ngrid]) > > for i in arange((ngrid / 2)+1): > for j in arange((ngrid / 2)+1): > result[j,i] = 2. * pi * sqrt(i ** 2. + j ** 2) / reso / > (ngrid*1.0) i = arange(ngrid/2 + 1) j = arange(ngrid/2 + 1) result = 2*pi*hypot(i[None,:], j[:,None])/reso/ngrid -- Pauli Virtanen From josef.pktd at gmail.com Wed Apr 8 09:50:45 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 8 Apr 2009 09:50:45 -0400 Subject: [SciPy-user] sparse matrix multiplication elementwise ? In-Reply-To: <031a28ef-ad3b-405c-96d6-25c9bdab1a77@y7g2000yqa.googlegroups.com> References: <031a28ef-ad3b-405c-96d6-25c9bdab1a77@y7g2000yqa.googlegroups.com> Message-ID: <1cd32cbb0904080650q6394b7c7te6fe3ff00ab6c3eb@mail.gmail.com> On Wed, Apr 8, 2009 at 8:37 AM, Uwe Schmitt wrote: > I solved my question as follows: > > > import numpy as np > import scipy.sparse as sp > > > def scale_sparse_matrix(A, by): > > ? ?assert isinstance(A, sp.csr_matrix), "wrong format" > > ? ?now = 0 > ? ?for row in range(A.shape[0]): > ? ? ? ?upto = A.indptr[row+1] > ? ? ? ?while now < upto: > ? ? ? ? ? ?col = A.indices[now] > ? ? ? ? ? ?A.data[now] *= by[row, col] > ? ? ? ? ? ?now += 1 > > > if __name__ == "__main__": > > ? ?A = np.zeros((3,4)) > ? ?A[1,1] = 1 > ? ?A[2] = 2 > > ? ?aa = sp.csr_matrix(A) > ? ?print "before:\n", aa > > ? ?b = 2*np.ones_like(A) > ? ?scale_sparse_matrix(aa, by=b) > > ? ?print "after:\n", aa > > > On 8 Apr., 11:24, Uwe Schmitt wrote: >> Hi, >> >> is there a way to multiply a sparse matrix with a dense numpy.array >> elementwise ? >> I don't know if this is intended use, but it works. try this addition to your example if __name__ == "__main__": A = np.zeros((3,4)) A[1,1] = 1 A[2] = 2 aa = sp.csr_matrix(A) print "before:\n", aa b = np.arange(12).reshape(3,4) #2*np.ones_like(A) scale_sparse_matrix(aa, by=b) print "after:\n", aa # new version #multiply non-zero elements by corresponding elements of array b aa2 = sp.csr_matrix(A) aao=aa.tocoo() aa2.data *= b[aao.row,aao.col] print "new version:\n", aa2 From josef.pktd at gmail.com Wed Apr 8 09:53:46 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 8 Apr 2009 09:53:46 -0400 Subject: [SciPy-user] Python loops too slow In-Reply-To: References: <49DC8683.6070201@usap.gov> Message-ID: <1cd32cbb0904080653w772fd9bdga0e1ea834346ba8@mail.gmail.com> On Wed, Apr 8, 2009 at 9:14 AM, Pauli Virtanen wrote: > Wed, 08 Apr 2009 23:12:03 +1200, Ross Williamson kirjoitti: > [clip] >> ? ?result = zeros([ngrid, ngrid]) >> >> ? ?for i in arange((ngrid / 2)+1): >> ? ? ? for j in arange((ngrid / 2)+1): >> ? ? ? ? ?result[j,i] = 2. * pi * sqrt(i ** 2. + j ** 2) / reso / >> ? ? ? ? ?(ngrid*1.0) > > i = arange(ngrid/2 + 1) > j = arange(ngrid/2 + 1) > result = 2*pi*hypot(i[None,:], j[:,None])/reso/ngrid > second part can also be vectorized half = (ngrid / 2) result[:, half:] = result[:, half:0:-1] result[half:, :] = result[half:0:-1, :] Josef From dave.hirschfeld at gmail.com Wed Apr 8 10:00:38 2009 From: dave.hirschfeld at gmail.com (Dave Hirschfeld) Date: Wed, 8 Apr 2009 14:00:38 +0000 (UTC) Subject: [SciPy-user] Python loops too slow References: <49DC8683.6070201@usap.gov> Message-ID: Pauli Virtanen iki.fi> writes: > > i = arange(ngrid/2 + 1) > j = arange(ngrid/2 + 1) > result = 2*pi*hypot(i[None,:], j[:,None])/reso/ngrid > It seems Pauli's solution may need to be repeated in the final output as below. Is that what you really want though because the submatrix isn't reflected uniformly - i.e. the first row and column are missing. Anyway, timings are below - it seems using meshgrid is more efficient than hypot. -Dave Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)] Type "copyright", "credits" or "license" for more information. IPython 0.9.1 -- An enhanced Interactive Python. Welcome to pylab, a matplotlib-based Python environment. For more information, type 'help(pylab)'. In [2]: def f1(ngrid, reso): ...: result = zeros([ngrid, ngrid]) ...: for i in arange((ngrid / 2)+1): ...: for j in arange((ngrid / 2)+1): ...: result[j,i] = 2. * pi * sqrt(i ** 2. + j ** 2) / reso / (ngrid*1.0) ...: for i in xrange(ngrid): ...: result[i,ngrid / 2+1:] = result[i,1:(ngrid / 2)][::-1] ...: for i in xrange(ngrid): ...: result[ngrid / 2+1:,i] = result[1:(ngrid / 2),i][::-1] ...: return result ...: # ...: In [3]: def f2(ngrid, reso): ...: nsub = ngrid/2+1 ...: i = np.arange(nsub) ...: j = np.arange(nsub) ...: submatrix = 2*pi*np.hypot(i[None,:], j[:,None])/reso/np.float(ngrid) ...: result = zeros([ngrid, ngrid]) ...: result[0:nsub,0:nsub] = submatrix ...: result[0:nsub,nsub:] = np.fliplr(submatrix[:,1:-1]) ...: result[nsub:,:] = np.flipud(result[1:nsub-1,:]) ...: return result ...: # ...: In [4]: def f3(ngrid, reso): ...: nsub = ngrid/2+1 ...: i, j = meshgrid(np.arange(nsub),np.arange(nsub)) ...: submatrix = 2*pi*np.sqrt(i**2 + j**2)/reso/np.float(ngrid) ...: result = zeros([ngrid, ngrid]) ...: result[0:nsub,0:nsub] = submatrix ...: result[0:nsub,nsub:] = np.fliplr(submatrix[:,1:-1]) ...: result[nsub:,:] = np.flipud(result[1:nsub-1,:]) ...: return result ...: # ...: In [5]: np.allclose(f1(100,1),f2(100,1)) Out[5]: True In [6]: np.allclose(f1(100,1),f3(100,1)) Out[6]: True In [7]: In [8]: timeit f1(100,1) 10 loops, best of 3: 60.6 ms per loop In [9]: In [10]: timeit f2(100,1) 1000 loops, best of 3: 840 ?s per loop In [11]: In [12]: timeit f3(100,1) 1000 loops, best of 3: 258 ?s per loop In [13]: From pav at iki.fi Wed Apr 8 14:33:06 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 8 Apr 2009 18:33:06 +0000 (UTC) Subject: [SciPy-user] Memory Leak in Scipy.Signal.lfilter? References: <49D470FC.3030107@gmail.com> <5b8d13220904080056i168a30ffj2f6bcefbb6d1e443@mail.gmail.com> Message-ID: Wed, 08 Apr 2009 16:56:48 +0900, David Cournapeau wrote: > On Thu, Apr 2, 2009 at 5:02 PM, Richard Lyon > wrote: >> Anyone else observed the same problem? > > Ok, I think I fixed the problem in the trunk. Could you test it ? I will > backport it for upcoming 0.7.1 as well, Could you add the test case from the ticket in signal/tests as a regression test? -- Pauli Virtanen From pav at iki.fi Wed Apr 8 15:09:07 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 8 Apr 2009 19:09:07 +0000 (UTC) Subject: [SciPy-user] Memory Leak in Scipy.Signal.lfilter? References: <49D470FC.3030107@gmail.com> <5b8d13220904080056i168a30ffj2f6bcefbb6d1e443@mail.gmail.com> Message-ID: Wed, 08 Apr 2009 18:33:06 +0000, Pauli Virtanen wrote: > Wed, 08 Apr 2009 16:56:48 +0900, David Cournapeau wrote: >> On Thu, Apr 2, 2009 at 5:02 PM, Richard Lyon >> wrote: >>> Anyone else observed the same problem? >> >> Ok, I think I fixed the problem in the trunk. Could you test it ? I >> will backport it for upcoming 0.7.1 as well, > > Could you add the test case from the ticket in signal/tests as a > regression test? Sorry, I'm typing faster than thinking; testing memory leaks that aren't refcount errors isn't straightforward. -- Pauli Virtanen From rob.patro at gmail.com Wed Apr 8 18:04:21 2009 From: rob.patro at gmail.com (Rob Patro) Date: Wed, 08 Apr 2009 18:04:21 -0400 Subject: [SciPy-user] sparse SVD Message-ID: <49DD1F65.40404@gmail.com> Hello, Is there any implementation of sparse SVD available in scipy? If not, does anyone know of an implementation available in python at all? I'd like to port a project on which I'm working from Matlab to Python, but it is crucial that I am able to perform the SVD of large and *very* sparse matrices. Thanks, Rob From stefan at sun.ac.za Wed Apr 8 18:35:32 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 9 Apr 2009 00:35:32 +0200 Subject: [SciPy-user] sparse SVD In-Reply-To: <49DD1F65.40404@gmail.com> References: <49DD1F65.40404@gmail.com> Message-ID: <9457e7c80904081535o203026e7k7ed4e51bdc0c8b10@mail.gmail.com> Hi Rob 2009/4/9 Rob Patro : > ?Is there any implementation of sparse SVD available in scipy? ?If not, > does anyone know of an implementation available in python at all? ?I'd > like to port a project on which I'm working from Matlab to Python, but > it is crucial that I am able to perform the SVD of large and *very* > sparse matrices. Currently, there is no sparse SVD in SciPy, but I'd love to see something like http://tedlab.mit.edu/~dr/svdlibc/ wrapped. I may even have an application for this, so I wouldn't mind helping. I've also come across http://soi.stanford.edu/~rmunk/PROPACK/ but I have no idea which library is best. I'll send an email to the authors of SVDPACK as a start. Regards St?fan From Ross.Williamson at usap.gov Wed Apr 8 19:15:59 2009 From: Ross.Williamson at usap.gov (Ross Williamson) Date: Thu, 09 Apr 2009 11:15:59 +1200 Subject: [SciPy-user] Python loops too slow In-Reply-To: References: <49DC8683.6070201@usap.gov> Message-ID: <49DD302F.2040904@usap.gov> Hi Guys Thanks for all your help - My code is now super quick and I'll implement meshgrid in other areas where I've got double loops. Cheers Ross > Pauli Virtanen iki.fi> writes: > >> i = arange(ngrid/2 + 1) >> j = arange(ngrid/2 + 1) >> result = 2*pi*hypot(i[None,:], j[:,None])/reso/ngrid >> >> > > It seems Pauli's solution may need to be repeated in the final output as below. > Is that what you really want though because the submatrix isn't reflected > uniformly - i.e. the first row and column are missing. > > Anyway, timings are below - it seems using meshgrid is more efficient than > hypot. > > -Dave > > Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)] > Type "copyright", "credits" or "license" for more information. > > IPython 0.9.1 -- An enhanced Interactive Python. > > Welcome to pylab, a matplotlib-based Python environment. > For more information, type 'help(pylab)'. > > In [2]: def f1(ngrid, reso): > ...: result = zeros([ngrid, ngrid]) > ...: for i in arange((ngrid / 2)+1): > ...: for j in arange((ngrid / 2)+1): > ...: result[j,i] = 2. * pi * sqrt(i ** 2. + j ** 2) / reso / > (ngrid*1.0) > ...: for i in xrange(ngrid): > ...: result[i,ngrid / 2+1:] = result[i,1:(ngrid / 2)][::-1] > ...: for i in xrange(ngrid): > ...: result[ngrid / 2+1:,i] = result[1:(ngrid / 2),i][::-1] > ...: return result > ...: # > ...: > > In [3]: def f2(ngrid, reso): > ...: nsub = ngrid/2+1 > ...: i = np.arange(nsub) > ...: j = np.arange(nsub) > ...: submatrix = 2*pi*np.hypot(i[None,:], j[:,None])/reso/np.float(ngrid) > ...: result = zeros([ngrid, ngrid]) > ...: result[0:nsub,0:nsub] = submatrix > ...: result[0:nsub,nsub:] = np.fliplr(submatrix[:,1:-1]) > ...: result[nsub:,:] = np.flipud(result[1:nsub-1,:]) > ...: return result > ...: # > ...: > > In [4]: def f3(ngrid, reso): > ...: nsub = ngrid/2+1 > ...: i, j = meshgrid(np.arange(nsub),np.arange(nsub)) > ...: submatrix = 2*pi*np.sqrt(i**2 + j**2)/reso/np.float(ngrid) > ...: result = zeros([ngrid, ngrid]) > ...: result[0:nsub,0:nsub] = submatrix > ...: result[0:nsub,nsub:] = np.fliplr(submatrix[:,1:-1]) > ...: result[nsub:,:] = np.flipud(result[1:nsub-1,:]) > ...: return result > ...: # > ...: > > In [5]: np.allclose(f1(100,1),f2(100,1)) > Out[5]: True > > In [6]: np.allclose(f1(100,1),f3(100,1)) > Out[6]: True > > In [7]: > > In [8]: timeit f1(100,1) > 10 loops, best of 3: 60.6 ms per loop > > In [9]: > > In [10]: timeit f2(100,1) > 1000 loops, best of 3: 840 ?s per loop > > In [11]: > > In [12]: timeit f3(100,1) > 1000 loops, best of 3: 258 ?s per loop > > In [13]: > > > > > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From Ross.Williamson at usap.gov Wed Apr 8 19:36:21 2009 From: Ross.Williamson at usap.gov (Ross Williamson) Date: Thu, 09 Apr 2009 11:36:21 +1200 Subject: [SciPy-user] Python loops too slow In-Reply-To: <49DD302F.2040904@usap.gov> References: <49DC8683.6070201@usap.gov> <49DD302F.2040904@usap.gov> Message-ID: <49DD34F5.8060705@usap.gov> Actually I've got another puzzler: I currently do the following to get an integrated vector from a 2d array: for i in xrange(ngrid): for j in xrange(ngrid): itemp = binned_ell[j,i] nsamp[itemp] = nsamp[itemp] + mask[j,i] where binned_ell is a set of indices and mask is just some numbers - Is there a more efficient way of doing this? I'm trying to work it out at the moment. Cheers Ross > Hi Guys > > Thanks for all your help - My code is now super quick and I'll implement > meshgrid in other areas where I've got double loops. > > Cheers > > Ross > >> Pauli Virtanen iki.fi> writes: >> >> >>> i = arange(ngrid/2 + 1) >>> j = arange(ngrid/2 + 1) >>> result = 2*pi*hypot(i[None,:], j[:,None])/reso/ngrid >>> >>> >>> >> It seems Pauli's solution may need to be repeated in the final output as below. >> Is that what you really want though because the submatrix isn't reflected >> uniformly - i.e. the first row and column are missing. >> >> Anyway, timings are below - it seems using meshgrid is more efficient than >> hypot. >> >> -Dave >> >> Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)] >> Type "copyright", "credits" or "license" for more information. >> >> IPython 0.9.1 -- An enhanced Interactive Python. >> >> Welcome to pylab, a matplotlib-based Python environment. >> For more information, type 'help(pylab)'. >> >> In [2]: def f1(ngrid, reso): >> ...: result = zeros([ngrid, ngrid]) >> ...: for i in arange((ngrid / 2)+1): >> ...: for j in arange((ngrid / 2)+1): >> ...: result[j,i] = 2. * pi * sqrt(i ** 2. + j ** 2) / reso / >> (ngrid*1.0) >> ...: for i in xrange(ngrid): >> ...: result[i,ngrid / 2+1:] = result[i,1:(ngrid / 2)][::-1] >> ...: for i in xrange(ngrid): >> ...: result[ngrid / 2+1:,i] = result[1:(ngrid / 2),i][::-1] >> ...: return result >> ...: # >> ...: >> >> In [3]: def f2(ngrid, reso): >> ...: nsub = ngrid/2+1 >> ...: i = np.arange(nsub) >> ...: j = np.arange(nsub) >> ...: submatrix = 2*pi*np.hypot(i[None,:], j[:,None])/reso/np.float(ngrid) >> ...: result = zeros([ngrid, ngrid]) >> ...: result[0:nsub,0:nsub] = submatrix >> ...: result[0:nsub,nsub:] = np.fliplr(submatrix[:,1:-1]) >> ...: result[nsub:,:] = np.flipud(result[1:nsub-1,:]) >> ...: return result >> ...: # >> ...: >> >> In [4]: def f3(ngrid, reso): >> ...: nsub = ngrid/2+1 >> ...: i, j = meshgrid(np.arange(nsub),np.arange(nsub)) >> ...: submatrix = 2*pi*np.sqrt(i**2 + j**2)/reso/np.float(ngrid) >> ...: result = zeros([ngrid, ngrid]) >> ...: result[0:nsub,0:nsub] = submatrix >> ...: result[0:nsub,nsub:] = np.fliplr(submatrix[:,1:-1]) >> ...: result[nsub:,:] = np.flipud(result[1:nsub-1,:]) >> ...: return result >> ...: # >> ...: >> >> In [5]: np.allclose(f1(100,1),f2(100,1)) >> Out[5]: True >> >> In [6]: np.allclose(f1(100,1),f3(100,1)) >> Out[6]: True >> >> In [7]: >> >> In [8]: timeit f1(100,1) >> 10 loops, best of 3: 60.6 ms per loop >> >> In [9]: >> >> In [10]: timeit f2(100,1) >> 1000 loops, best of 3: 840 ?s per loop >> >> In [11]: >> >> In [12]: timeit f3(100,1) >> 1000 loops, best of 3: 258 ?s per loop >> >> In [13]: >> >> >> >> >> >> _______________________________________________ >> SciPy-user mailing list >> SciPy-user at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From Ross.Williamson at usap.gov Wed Apr 8 23:02:10 2009 From: Ross.Williamson at usap.gov (Ross Williamson) Date: Thu, 09 Apr 2009 15:02:10 +1200 Subject: [SciPy-user] Python loops too slow In-Reply-To: <49DD34F5.8060705@usap.gov> References: <49DC8683.6070201@usap.gov> <49DD302F.2040904@usap.gov> <49DD34F5.8060705@usap.gov> Message-ID: <49DD6532.6030204@usap.gov> Ok so this is really annoying and slowing my code down. I've tried: nsamp = zeros(711) mask = ones(1000*1000) binned_all is (1000*1000 set of indices including numbers between 0 and 710) rbl = binned_ell.ravel() rmask = mask.ravel() nsamp[rbl] = nsamp[rbl] + mask And all it does is just add 1 to each element of nsamp - i.e. nsamp is now all ones rather than zeros and not the cumulative sum of the number of elements in binned_all (Note I'm using one here to make life easy - in reality it will be a float of different values) Any help much appriciated Ross > Actually I've got another puzzler: > > I currently do the following to get an integrated vector from a 2d array: > > for i in xrange(ngrid): > for j in xrange(ngrid): > itemp = binned_ell[j,i] > nsamp[itemp] = nsamp[itemp] + mask[j,i] > > where binned_ell is a set of indices and mask is just some numbers - Is > there a more efficient way of doing this? I'm trying to work it out at > the moment. > > Cheers > > Ross > >> Hi Guys >> >> Thanks for all your help - My code is now super quick and I'll implement >> meshgrid in other areas where I've got double loops. >> >> Cheers >> >> Ross >> >> >>> Pauli Virtanen iki.fi> writes: >>> >>> >>> >>>> i = arange(ngrid/2 + 1) >>>> j = arange(ngrid/2 + 1) >>>> result = 2*pi*hypot(i[None,:], j[:,None])/reso/ngrid >>>> >>>> >>>> >>>> >>> It seems Pauli's solution may need to be repeated in the final output as below. >>> Is that what you really want though because the submatrix isn't reflected >>> uniformly - i.e. the first row and column are missing. >>> >>> Anyway, timings are below - it seems using meshgrid is more efficient than >>> hypot. >>> >>> -Dave >>> >>> Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)] >>> Type "copyright", "credits" or "license" for more information. >>> >>> IPython 0.9.1 -- An enhanced Interactive Python. >>> >>> Welcome to pylab, a matplotlib-based Python environment. >>> For more information, type 'help(pylab)'. >>> >>> In [2]: def f1(ngrid, reso): >>> ...: result = zeros([ngrid, ngrid]) >>> ...: for i in arange((ngrid / 2)+1): >>> ...: for j in arange((ngrid / 2)+1): >>> ...: result[j,i] = 2. * pi * sqrt(i ** 2. + j ** 2) / reso / >>> (ngrid*1.0) >>> ...: for i in xrange(ngrid): >>> ...: result[i,ngrid / 2+1:] = result[i,1:(ngrid / 2)][::-1] >>> ...: for i in xrange(ngrid): >>> ...: result[ngrid / 2+1:,i] = result[1:(ngrid / 2),i][::-1] >>> ...: return result >>> ...: # >>> ...: >>> >>> In [3]: def f2(ngrid, reso): >>> ...: nsub = ngrid/2+1 >>> ...: i = np.arange(nsub) >>> ...: j = np.arange(nsub) >>> ...: submatrix = 2*pi*np.hypot(i[None,:], j[:,None])/reso/np.float(ngrid) >>> ...: result = zeros([ngrid, ngrid]) >>> ...: result[0:nsub,0:nsub] = submatrix >>> ...: result[0:nsub,nsub:] = np.fliplr(submatrix[:,1:-1]) >>> ...: result[nsub:,:] = np.flipud(result[1:nsub-1,:]) >>> ...: return result >>> ...: # >>> ...: >>> >>> In [4]: def f3(ngrid, reso): >>> ...: nsub = ngrid/2+1 >>> ...: i, j = meshgrid(np.arange(nsub),np.arange(nsub)) >>> ...: submatrix = 2*pi*np.sqrt(i**2 + j**2)/reso/np.float(ngrid) >>> ...: result = zeros([ngrid, ngrid]) >>> ...: result[0:nsub,0:nsub] = submatrix >>> ...: result[0:nsub,nsub:] = np.fliplr(submatrix[:,1:-1]) >>> ...: result[nsub:,:] = np.flipud(result[1:nsub-1,:]) >>> ...: return result >>> ...: # >>> ...: >>> >>> In [5]: np.allclose(f1(100,1),f2(100,1)) >>> Out[5]: True >>> >>> In [6]: np.allclose(f1(100,1),f3(100,1)) >>> Out[6]: True >>> >>> In [7]: >>> >>> In [8]: timeit f1(100,1) >>> 10 loops, best of 3: 60.6 ms per loop >>> >>> In [9]: >>> >>> In [10]: timeit f2(100,1) >>> 1000 loops, best of 3: 840 ?s per loop >>> >>> In [11]: >>> >>> In [12]: timeit f3(100,1) >>> 1000 loops, best of 3: 258 ?s per loop >>> >>> In [13]: >>> >>> >>> >>> >>> >>> _______________________________________________ >>> SciPy-user mailing list >>> SciPy-user at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >>> >> _______________________________________________ >> SciPy-user mailing list >> SciPy-user at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From robert.kern at gmail.com Wed Apr 8 23:06:26 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 8 Apr 2009 22:06:26 -0500 Subject: [SciPy-user] Python loops too slow In-Reply-To: <49DD6532.6030204@usap.gov> References: <49DC8683.6070201@usap.gov> <49DD302F.2040904@usap.gov> <49DD34F5.8060705@usap.gov> <49DD6532.6030204@usap.gov> Message-ID: <3d375d730904082006j4b65239v45c1eda5355b555d@mail.gmail.com> On Wed, Apr 8, 2009 at 22:02, Ross Williamson wrote: > Ok so this is really annoying and slowing my code down. > > I've tried: > > nsamp = zeros(711) > mask = ones(1000*1000) > binned_all is (1000*1000 set of indices including numbers between 0 and 710) > > rbl = binned_ell.ravel() > rmask = mask.ravel() > > nsamp[rbl] = nsamp[rbl] + mask > > And all it does is just add 1 to each element of nsamp - i.e. nsamp is > now all ones rather than zeros and not the cumulative sum of the number > of elements in binned_all (Note I'm using one here to make life easy - > in reality it will be ?a float of different values) The reason why this can't work has been explained many times on this list. The right way to do this is to use bincount(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Ross.Williamson at usap.gov Wed Apr 8 23:26:56 2009 From: Ross.Williamson at usap.gov (Ross Williamson) Date: Thu, 09 Apr 2009 15:26:56 +1200 Subject: [SciPy-user] Python loops too slow In-Reply-To: <3d375d730904082006j4b65239v45c1eda5355b555d@mail.gmail.com> References: <49DC8683.6070201@usap.gov> <49DD302F.2040904@usap.gov> <49DD34F5.8060705@usap.gov> <49DD6532.6030204@usap.gov> <3d375d730904082006j4b65239v45c1eda5355b555d@mail.gmail.com> Message-ID: <49DD6B00.80505@usap.gov> Thanks Robert Apologies for not searching the mailing list but I'm working over Iridium and only have email - no WWW :( bincount worked a treat Ross > On Wed, Apr 8, 2009 at 22:02, Ross Williamson wrote: > >> Ok so this is really annoying and slowing my code down. >> >> I've tried: >> >> nsamp = zeros(711) >> mask = ones(1000*1000) >> binned_all is (1000*1000 set of indices including numbers between 0 and 710) >> >> rbl = binned_ell.ravel() >> rmask = mask.ravel() >> >> nsamp[rbl] = nsamp[rbl] + mask >> >> And all it does is just add 1 to each element of nsamp - i.e. nsamp is >> now all ones rather than zeros and not the cumulative sum of the number >> of elements in binned_all (Note I'm using one here to make life easy - >> in reality it will be a float of different values) >> > > The reason why this can't work has been explained many times on this > list. The right way to do this is to use bincount(). > > From robert.kern at gmail.com Wed Apr 8 23:28:20 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 8 Apr 2009 22:28:20 -0500 Subject: [SciPy-user] Python loops too slow In-Reply-To: <49DD6B00.80505@usap.gov> References: <49DC8683.6070201@usap.gov> <49DD302F.2040904@usap.gov> <49DD34F5.8060705@usap.gov> <49DD6532.6030204@usap.gov> <3d375d730904082006j4b65239v45c1eda5355b555d@mail.gmail.com> <49DD6B00.80505@usap.gov> Message-ID: <3d375d730904082028m8492d3ake105fe4199daac04@mail.gmail.com> On Wed, Apr 8, 2009 at 22:26, Ross Williamson wrote: > Thanks Robert > > Apologies for not searching the mailing list but I'm working over > Iridium and only have email - no WWW :( No worries. I wasn't chastising, just giving the short version so I don't have to give the lengthy explanation. :-) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From imrehg at gmail.com Wed Apr 8 23:51:40 2009 From: imrehg at gmail.com (Gergely Imreh) Date: Thu, 9 Apr 2009 11:51:40 +0800 Subject: [SciPy-user] sparse matrices - list assignment to rows and columns Message-ID: <371b60070904082051p8da59d1mea2102d6b48928ab@mail.gmail.com> Hi, I was trying figure out the scipy sparse matrix handling, but run into some difficulties assigning a list of values to rows and columns. The scipy tutorial has the following example [1]: from scipy import sparse Asp = sparse.lil_matrix((50000,50000)) Asp.setdiag(ones(50000)) Asp[20,100:250] = 10*random.rand(150) Asp[200:250,30] = 10*random.rand(50) That looks straightforward enough, make a large, diagonal sparse matrix, and set some additional elements to non-zero. What I get, however, is different: Asp[20,100:250] = 10*random.rand(150) sets the matrix elements at row 20, column 100-249 to random values. Asp[200:250,30] = 10*random.rand(50) sets the matrix element at row 200, column 30 to a 50 element row vector with random values.... (elements at row 201-249, column 30 are still 0) If I reshape the results of random.rand(50) to be in a column instead of row, the assignment will results in the elements of row 200-249, column 30 to be set to a single element array values (So, for exaple Asp[200,30] will be an array, which will have a single random value at [0,0]) I'm using Python 2.6 (that comes with my distro, or 2.4 for which I'd have to recompile scipy) and scipy 0.7.0. Is this kind of behaviour due to the changes (and incompatibilites) of 2.6 (since I know scipy is writtend to be compatible up to 2.5) or something else? The other sparse matrix types would handle this differently? A workaround is to do single element assignments but I'd think that's probably slower in general. Cheers! Greg [1] http://www.scipy.org/SciPy_Tutorial From josef.pktd at gmail.com Thu Apr 9 04:51:01 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 9 Apr 2009 04:51:01 -0400 Subject: [SciPy-user] sparse matrices - list assignment to rows and columns In-Reply-To: <371b60070904082051p8da59d1mea2102d6b48928ab@mail.gmail.com> References: <371b60070904082051p8da59d1mea2102d6b48928ab@mail.gmail.com> Message-ID: <1cd32cbb0904090151x4590f7cak8d6cbc666131f908@mail.gmail.com> On Wed, Apr 8, 2009 at 11:51 PM, Gergely Imreh wrote: > Hi, > > ?I was trying figure out the scipy sparse matrix handling, but run > into some difficulties assigning a list of values to rows and columns. > ?The scipy tutorial has the following example [1]: > > from scipy import sparse > Asp = sparse.lil_matrix((50000,50000)) > Asp.setdiag(ones(50000)) > Asp[20,100:250] = 10*random.rand(150) > Asp[200:250,30] = 10*random.rand(50) > > ?That looks straightforward enough, make a large, diagonal sparse > matrix, and set some additional elements to non-zero. What I get, > however, is different: > Asp[20,100:250] = 10*random.rand(150) ?sets the matrix elements at row > 20, column 100-249 to random values. > Asp[200:250,30] = 10*random.rand(50) sets the matrix element at row > 200, column 30 to a 50 element row vector with random values.... > (elements at row 201-249, column 30 are still 0) > ?If I reshape the results of random.rand(50) to be in a column > instead of row, the assignment will results in the elements of row > 200-249, column 30 to be set to a single element array values (So, for > exaple Asp[200,30] will be an array, which will have a single random > value at ?[0,0]) > > ?I'm using Python 2.6 (that comes with my distro, or 2.4 for which > I'd have to recompile scipy) and scipy 0.7.0. Is this kind of > behaviour due to the changes (and incompatibilites) of 2.6 (since I > know scipy is writtend to be compatible up to 2.5) or something else? > The other sparse matrix types would handle this differently? > ?A workaround is to do single element assignments but I'd think > that's probably slower in general. > > ?Cheers! > ? ? ?Greg > [1] http://www.scipy.org/SciPy_Tutorial There is an assignment error: Asp[200:250,30] seems to assign all 50 elements to to the position Asp[200,30] >>> Asp[200,30] <1x50 sparse matrix of type '' with 50 stored elements in LInked List format> >>> Aspc = Asp.tocrc() Traceback (most recent call last): File "", line 1, in Aspc = Asp.tocrc() File "c:\josef\_progs_scipy\scipy\sparse\base.py", line 429, in __getattr__ raise AttributeError, attr + " not found" AttributeError: tocrc not found this is with >>> scipy.version.version '0.8.0.dev5551' there is a related assignment error that got fixed in trunk, http://thread.gmane.org/gmane.comp.python.scientific.user/19996 I don't know if it also handles this case, a bug report might be useful to make sure this case is handled correctly I think, for this example dok format would be better to build the matrix, since column slices need to access many lists Asp = sparse.dok_matrix((50000,50000)) Aspr = Asp.tocsr() works without problems I checked the history of the scipy tutorial that you linked to, the main editing has been done in 2006, and maybe it isn't up to date. The current docs are being written and are available at http://docs.scipy.org/doc/ Josef From josef.pktd at gmail.com Thu Apr 9 05:05:29 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 9 Apr 2009 05:05:29 -0400 Subject: [SciPy-user] sparse matrices - list assignment to rows and columns In-Reply-To: <1cd32cbb0904090151x4590f7cak8d6cbc666131f908@mail.gmail.com> References: <371b60070904082051p8da59d1mea2102d6b48928ab@mail.gmail.com> <1cd32cbb0904090151x4590f7cak8d6cbc666131f908@mail.gmail.com> Message-ID: <1cd32cbb0904090205n7db86bf8j9f8d2e2e51c46a9b@mail.gmail.com> On Thu, Apr 9, 2009 at 4:51 AM, wrote: > On Wed, Apr 8, 2009 at 11:51 PM, Gergely Imreh wrote: >> Hi, >> >> ?I was trying figure out the scipy sparse matrix handling, but run >> into some difficulties assigning a list of values to rows and columns. >> ?The scipy tutorial has the following example [1]: >> >> from scipy import sparse >> Asp = sparse.lil_matrix((50000,50000)) >> Asp.setdiag(ones(50000)) >> Asp[20,100:250] = 10*random.rand(150) >> Asp[200:250,30] = 10*random.rand(50) >> >> ?That looks straightforward enough, make a large, diagonal sparse >> matrix, and set some additional elements to non-zero. What I get, >> however, is different: >> Asp[20,100:250] = 10*random.rand(150) ?sets the matrix elements at row >> 20, column 100-249 to random values. >> Asp[200:250,30] = 10*random.rand(50) sets the matrix element at row >> 200, column 30 to a 50 element row vector with random values.... >> (elements at row 201-249, column 30 are still 0) >> ?If I reshape the results of random.rand(50) to be in a column >> instead of row, the assignment will results in the elements of row >> 200-249, column 30 to be set to a single element array values (So, for >> exaple Asp[200,30] will be an array, which will have a single random >> value at ?[0,0]) >> >> ?I'm using Python 2.6 (that comes with my distro, or 2.4 for which >> I'd have to recompile scipy) and scipy 0.7.0. Is this kind of >> behaviour due to the changes (and incompatibilites) of 2.6 (since I >> know scipy is writtend to be compatible up to 2.5) or something else? >> The other sparse matrix types would handle this differently? >> ?A workaround is to do single element assignments but I'd think >> that's probably slower in general. >> >> ?Cheers! >> ? ? ?Greg >> [1] http://www.scipy.org/SciPy_Tutorial > > > There is an assignment error: > Asp[200:250,30] ?seems to assign all 50 elements to to the position Asp[200,30] > >>>> Asp[200,30] > <1x50 sparse matrix of type '' > ? ? ? ?with 50 stored elements in LInked List format> >>>> Aspc = Asp.tocrc() > Traceback (most recent call last): > ?File "", line 1, in > ? ?Aspc = Asp.tocrc() > ?File "c:\josef\_progs_scipy\scipy\sparse\base.py", line 429, in __getattr__ > ? ?raise AttributeError, attr + " not found" > AttributeError: tocrc not found sorry, I copied the wrong traceback, it should be: >>> Aspc = Asp.tocsr() Traceback (most recent call last): File "", line 1, in Aspc = Asp.tocsr() File "c:\josef\_progs_scipy\scipy\sparse\lil.py", line 427, in tocsr data = np.asarray(data, dtype=self.dtype) File "C:\Programs\Python25\Lib\site-packages\numpy\core\numeric.py", line 230, in asarray return array(a, dtype, copy=False, order=order) ValueError: setting an array element with a sequence. > > this is with >>>> scipy.version.version > '0.8.0.dev5551' > > there is a related assignment error that got fixed in trunk, > http://thread.gmane.org/gmane.comp.python.scientific.user/19996 > I don't know if it also handles this case, a bug report might be > useful to make sure this case is handled correctly > > I think, for this example dok format would be better to build the > matrix, since column slices need to access many lists > > Asp = sparse.dok_matrix((50000,50000)) > Aspr = Asp.tocsr() > > works without problems > > I checked the history of the scipy tutorial that you linked to, the > main editing has been done in 2006, and maybe it isn't up to date. > > The current docs are being written and are available at > http://docs.scipy.org/doc/ > > Josef > From imrehg at gmail.com Thu Apr 9 05:17:35 2009 From: imrehg at gmail.com (Gergely Imreh) Date: Thu, 9 Apr 2009 17:17:35 +0800 Subject: [SciPy-user] sparse matrices - list assignment to rows and columns In-Reply-To: <1cd32cbb0904090205n7db86bf8j9f8d2e2e51c46a9b@mail.gmail.com> References: <371b60070904082051p8da59d1mea2102d6b48928ab@mail.gmail.com> <1cd32cbb0904090151x4590f7cak8d6cbc666131f908@mail.gmail.com> <1cd32cbb0904090205n7db86bf8j9f8d2e2e51c46a9b@mail.gmail.com> Message-ID: <371b60070904090217g440eac9fna7500aac41c183a4@mail.gmail.com> 2009/4/9 : > On Thu, Apr 9, 2009 at 4:51 AM, ? wrote: >> On Wed, Apr 8, 2009 at 11:51 PM, Gergely Imreh wrote: >>> Hi, >>> >>> ?I was trying figure out the scipy sparse matrix handling, but run >>> into some difficulties assigning a list of values to rows and columns. >>> ?The scipy tutorial has the following example [1]: >>> >>> from scipy import sparse >>> Asp = sparse.lil_matrix((50000,50000)) >>> Asp.setdiag(ones(50000)) >>> Asp[20,100:250] = 10*random.rand(150) >>> Asp[200:250,30] = 10*random.rand(50) >>> >>> ?That looks straightforward enough, make a large, diagonal sparse >>> matrix, and set some additional elements to non-zero. What I get, >>> however, is different: >>> Asp[20,100:250] = 10*random.rand(150) ?sets the matrix elements at row >>> 20, column 100-249 to random values. >>> Asp[200:250,30] = 10*random.rand(50) sets the matrix element at row >>> 200, column 30 to a 50 element row vector with random values.... >>> (elements at row 201-249, column 30 are still 0) >>> ?If I reshape the results of random.rand(50) to be in a column >>> instead of row, the assignment will results in the elements of row >>> 200-249, column 30 to be set to a single element array values (So, for >>> exaple Asp[200,30] will be an array, which will have a single random >>> value at ?[0,0]) >>> >>> ?I'm using Python 2.6 (that comes with my distro, or 2.4 for which >>> I'd have to recompile scipy) and scipy 0.7.0. Is this kind of >>> behaviour due to the changes (and incompatibilites) of 2.6 (since I >>> know scipy is writtend to be compatible up to 2.5) or something else? >>> The other sparse matrix types would handle this differently? >>> ?A workaround is to do single element assignments but I'd think >>> that's probably slower in general. >>> >>> ?Cheers! >>> ? ? ?Greg >>> [1] http://www.scipy.org/SciPy_Tutorial >> >> >> There is an assignment error: >> Asp[200:250,30] ?seems to assign all 50 elements to to the position Asp[200,30] >> >>>>> Asp[200,30] >> <1x50 sparse matrix of type '' >> ? ? ? ?with 50 stored elements in LInked List format> >>>>> Aspc = Asp.tocrc() >> Traceback (most recent call last): >> ?File "", line 1, in >> ? ?Aspc = Asp.tocrc() >> ?File "c:\josef\_progs_scipy\scipy\sparse\base.py", line 429, in __getattr__ >> ? ?raise AttributeError, attr + " not found" >> AttributeError: tocrc not found > > sorry, I copied the wrong traceback, it should be: > >>>> Aspc = Asp.tocsr() > Traceback (most recent call last): > ?File "", line 1, in > ? ?Aspc = Asp.tocsr() > ?File "c:\josef\_progs_scipy\scipy\sparse\lil.py", line 427, in tocsr > ? ?data = np.asarray(data, dtype=self.dtype) > ?File "C:\Programs\Python25\Lib\site-packages\numpy\core\numeric.py", > line 230, in asarray > ? ?return array(a, dtype, copy=False, order=order) > ValueError: setting an array element with a sequence. > >> >> this is with >>>>> scipy.version.version >> '0.8.0.dev5551' >> >> there is a related assignment error that got fixed in trunk, >> http://thread.gmane.org/gmane.comp.python.scientific.user/19996 >> I don't know if it also handles this case, a bug report might be >> useful to make sure this case is handled correctly >> >> I think, for this example dok format would be better to build the >> matrix, since column slices need to access many lists >> >> Asp = sparse.dok_matrix((50000,50000)) >> Aspr = Asp.tocsr() >> >> works without problems >> >> I checked the history of the scipy tutorial that you linked to, the >> main editing has been done in 2006, and maybe it isn't up to date. >> >> The current docs are being written and are available at >> http://docs.scipy.org/doc/ >> >> Josef >> Yes, I think is the same, I got a ValueError as well, having upgraded to the latest (r5655) version. Traceback (most recent call last): File "sp2.py", line 6, in Asp[200:250,30] = 10*random.rand(50) File "/usr/lib/python2.6/site-packages/scipy/sparse/lil.py", line 329, in __setitem__ self._insertat3(row, data, j, xx) File "/usr/lib/python2.6/site-packages/scipy/sparse/lil.py", line 285, in _insertat3 self._insertat2(row, data, j, x) File "/usr/lib/python2.6/site-packages/scipy/sparse/lil.py", line 246, in _insertat2 raise ValueError('setting an array element with a sequence') ValueError: setting an array element with a sequence Checked out the new documentation you referenced[1] and there is only same-row assignment (e.g. A[0, :100] = rand(100) ) but no same-column assignment... So still, my question is that is there something inherently different between array -> row and array -> column assigment in this case? Cheers, Greg [1] http://docs.scipy.org/doc/scipy/reference/sparse.html From josef.pktd at gmail.com Thu Apr 9 09:36:30 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 9 Apr 2009 09:36:30 -0400 Subject: [SciPy-user] sparse matrices - list assignment to rows and columns In-Reply-To: <371b60070904090217g440eac9fna7500aac41c183a4@mail.gmail.com> References: <371b60070904082051p8da59d1mea2102d6b48928ab@mail.gmail.com> <1cd32cbb0904090151x4590f7cak8d6cbc666131f908@mail.gmail.com> <1cd32cbb0904090205n7db86bf8j9f8d2e2e51c46a9b@mail.gmail.com> <371b60070904090217g440eac9fna7500aac41c183a4@mail.gmail.com> Message-ID: <1cd32cbb0904090636u4f931d05jec70075bdc85e962@mail.gmail.com> On Thu, Apr 9, 2009 at 5:17 AM, Gergely Imreh wrote: > 2009/4/9 ?: >> On Thu, Apr 9, 2009 at 4:51 AM, ? wrote: >>> On Wed, Apr 8, 2009 at 11:51 PM, Gergely Imreh wrote: >>>> Hi, >>>> >>>> ?I was trying figure out the scipy sparse matrix handling, but run >>>> into some difficulties assigning a list of values to rows and columns. >>>> ?The scipy tutorial has the following example [1]: >>>> >>>> from scipy import sparse >>>> Asp = sparse.lil_matrix((50000,50000)) >>>> Asp.setdiag(ones(50000)) >>>> Asp[20,100:250] = 10*random.rand(150) >>>> Asp[200:250,30] = 10*random.rand(50) >>>> >>>> ?That looks straightforward enough, make a large, diagonal sparse >>>> matrix, and set some additional elements to non-zero. What I get, >>>> however, is different: >>>> Asp[20,100:250] = 10*random.rand(150) ?sets the matrix elements at row >>>> 20, column 100-249 to random values. >>>> Asp[200:250,30] = 10*random.rand(50) sets the matrix element at row >>>> 200, column 30 to a 50 element row vector with random values.... >>>> (elements at row 201-249, column 30 are still 0) >>>> ?If I reshape the results of random.rand(50) to be in a column >>>> instead of row, the assignment will results in the elements of row >>>> 200-249, column 30 to be set to a single element array values (So, for >>>> exaple Asp[200,30] will be an array, which will have a single random >>>> value at ?[0,0]) >>>> >>>> ?I'm using Python 2.6 (that comes with my distro, or 2.4 for which >>>> I'd have to recompile scipy) and scipy 0.7.0. Is this kind of >>>> behaviour due to the changes (and incompatibilites) of 2.6 (since I >>>> know scipy is writtend to be compatible up to 2.5) or something else? >>>> The other sparse matrix types would handle this differently? >>>> ?A workaround is to do single element assignments but I'd think >>>> that's probably slower in general. >>>> >>>> ?Cheers! >>>> ? ? ?Greg >>>> [1] http://www.scipy.org/SciPy_Tutorial >>> >>> >>> There is an assignment error: >>> Asp[200:250,30] ?seems to assign all 50 elements to to the position Asp[200,30] >>> >>>>>> Asp[200,30] >>> <1x50 sparse matrix of type '' >>> ? ? ? ?with 50 stored elements in LInked List format> >>>>>> Aspc = Asp.tocrc() >>> Traceback (most recent call last): >>> ?File "", line 1, in >>> ? ?Aspc = Asp.tocrc() >>> ?File "c:\josef\_progs_scipy\scipy\sparse\base.py", line 429, in __getattr__ >>> ? ?raise AttributeError, attr + " not found" >>> AttributeError: tocrc not found >> >> sorry, I copied the wrong traceback, it should be: >> >>>>> Aspc = Asp.tocsr() >> Traceback (most recent call last): >> ?File "", line 1, in >> ? ?Aspc = Asp.tocsr() >> ?File "c:\josef\_progs_scipy\scipy\sparse\lil.py", line 427, in tocsr >> ? ?data = np.asarray(data, dtype=self.dtype) >> ?File "C:\Programs\Python25\Lib\site-packages\numpy\core\numeric.py", >> line 230, in asarray >> ? ?return array(a, dtype, copy=False, order=order) >> ValueError: setting an array element with a sequence. >> >>> >>> this is with >>>>>> scipy.version.version >>> '0.8.0.dev5551' >>> >>> there is a related assignment error that got fixed in trunk, >>> http://thread.gmane.org/gmane.comp.python.scientific.user/19996 >>> I don't know if it also handles this case, a bug report might be >>> useful to make sure this case is handled correctly >>> >>> I think, for this example dok format would be better to build the >>> matrix, since column slices need to access many lists >>> >>> Asp = sparse.dok_matrix((50000,50000)) >>> Aspr = Asp.tocsr() >>> >>> works without problems >>> >>> I checked the history of the scipy tutorial that you linked to, the >>> main editing has been done in 2006, and maybe it isn't up to date. >>> >>> The current docs are being written and are available at >>> http://docs.scipy.org/doc/ >>> >>> Josef >>> > > > Yes, I think is the same, I got a ValueError as well, having upgraded > to the latest (r5655) version. > > Traceback (most recent call last): > ?File "sp2.py", line 6, in > ? ?Asp[200:250,30] = 10*random.rand(50) > ?File "/usr/lib/python2.6/site-packages/scipy/sparse/lil.py", line > 329, in __setitem__ > ? ?self._insertat3(row, data, j, xx) > ?File "/usr/lib/python2.6/site-packages/scipy/sparse/lil.py", line > 285, in _insertat3 > ? ?self._insertat2(row, data, j, x) > ?File "/usr/lib/python2.6/site-packages/scipy/sparse/lil.py", line > 246, in _insertat2 > ? ?raise ValueError('setting an array element with a sequence') > ValueError: setting an array element with a sequence > > Checked out the new documentation you referenced[1] and there is only > same-row assignment (e.g. A[0, :100] = rand(100) ?) but no same-column > assignment... > > So still, my question is that is there something inherently different > between array -> row and array -> column assigment in this case? I'm not completely sure about the internal structure, but essentially the non-zero values are stored in row-wise lists, to access a column slice it needs to access all the rowlists and insert to each, and this is much slower. see the explanation in the documentation describing the different formats. Josef From josef.pktd at gmail.com Thu Apr 9 09:53:25 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 9 Apr 2009 09:53:25 -0400 Subject: [SciPy-user] sparse matrices - list assignment to rows and columns In-Reply-To: <1cd32cbb0904090636u4f931d05jec70075bdc85e962@mail.gmail.com> References: <371b60070904082051p8da59d1mea2102d6b48928ab@mail.gmail.com> <1cd32cbb0904090151x4590f7cak8d6cbc666131f908@mail.gmail.com> <1cd32cbb0904090205n7db86bf8j9f8d2e2e51c46a9b@mail.gmail.com> <371b60070904090217g440eac9fna7500aac41c183a4@mail.gmail.com> <1cd32cbb0904090636u4f931d05jec70075bdc85e962@mail.gmail.com> Message-ID: <1cd32cbb0904090653h60031d12o9440d44d0fe74fc2@mail.gmail.com> On Thu, Apr 9, 2009 at 9:36 AM, wrote: > On Thu, Apr 9, 2009 at 5:17 AM, Gergely Imreh wrote: >> 2009/4/9 ?: >>> On Thu, Apr 9, 2009 at 4:51 AM, ? wrote: >>>> On Wed, Apr 8, 2009 at 11:51 PM, Gergely Imreh wrote: >>>>> Hi, >>>>> >>>>> ?I was trying figure out the scipy sparse matrix handling, but run >>>>> into some difficulties assigning a list of values to rows and columns. >>>>> ?The scipy tutorial has the following example [1]: >>>>> >>>>> from scipy import sparse >>>>> Asp = sparse.lil_matrix((50000,50000)) >>>>> Asp.setdiag(ones(50000)) >>>>> Asp[20,100:250] = 10*random.rand(150) >>>>> Asp[200:250,30] = 10*random.rand(50) >>>>> >>>>> ?That looks straightforward enough, make a large, diagonal sparse >>>>> matrix, and set some additional elements to non-zero. What I get, >>>>> however, is different: >>>>> Asp[20,100:250] = 10*random.rand(150) ?sets the matrix elements at row >>>>> 20, column 100-249 to random values. >>>>> Asp[200:250,30] = 10*random.rand(50) sets the matrix element at row >>>>> 200, column 30 to a 50 element row vector with random values.... >>>>> (elements at row 201-249, column 30 are still 0) >>>>> ?If I reshape the results of random.rand(50) to be in a column >>>>> instead of row, the assignment will results in the elements of row >>>>> 200-249, column 30 to be set to a single element array values (So, for >>>>> exaple Asp[200,30] will be an array, which will have a single random >>>>> value at ?[0,0]) >>>>> >>>>> ?I'm using Python 2.6 (that comes with my distro, or 2.4 for which >>>>> I'd have to recompile scipy) and scipy 0.7.0. Is this kind of >>>>> behaviour due to the changes (and incompatibilites) of 2.6 (since I >>>>> know scipy is writtend to be compatible up to 2.5) or something else? >>>>> The other sparse matrix types would handle this differently? >>>>> ?A workaround is to do single element assignments but I'd think >>>>> that's probably slower in general. >>>>> >>>>> ?Cheers! >>>>> ? ? ?Greg >>>>> [1] http://www.scipy.org/SciPy_Tutorial >>>> >>>> >>>> There is an assignment error: >>>> Asp[200:250,30] ?seems to assign all 50 elements to to the position Asp[200,30] >>>> >>>>>>> Asp[200,30] >>>> <1x50 sparse matrix of type '' >>>> ? ? ? ?with 50 stored elements in LInked List format> >>>>>>> Aspc = Asp.tocrc() >>>> Traceback (most recent call last): >>>> ?File "", line 1, in >>>> ? ?Aspc = Asp.tocrc() >>>> ?File "c:\josef\_progs_scipy\scipy\sparse\base.py", line 429, in __getattr__ >>>> ? ?raise AttributeError, attr + " not found" >>>> AttributeError: tocrc not found >>> >>> sorry, I copied the wrong traceback, it should be: >>> >>>>>> Aspc = Asp.tocsr() >>> Traceback (most recent call last): >>> ?File "", line 1, in >>> ? ?Aspc = Asp.tocsr() >>> ?File "c:\josef\_progs_scipy\scipy\sparse\lil.py", line 427, in tocsr >>> ? ?data = np.asarray(data, dtype=self.dtype) >>> ?File "C:\Programs\Python25\Lib\site-packages\numpy\core\numeric.py", >>> line 230, in asarray >>> ? ?return array(a, dtype, copy=False, order=order) >>> ValueError: setting an array element with a sequence. >>> >>>> >>>> this is with >>>>>>> scipy.version.version >>>> '0.8.0.dev5551' >>>> >>>> there is a related assignment error that got fixed in trunk, >>>> http://thread.gmane.org/gmane.comp.python.scientific.user/19996 >>>> I don't know if it also handles this case, a bug report might be >>>> useful to make sure this case is handled correctly >>>> >>>> I think, for this example dok format would be better to build the >>>> matrix, since column slices need to access many lists >>>> >>>> Asp = sparse.dok_matrix((50000,50000)) >>>> Aspr = Asp.tocsr() >>>> >>>> works without problems >>>> >>>> I checked the history of the scipy tutorial that you linked to, the >>>> main editing has been done in 2006, and maybe it isn't up to date. >>>> >>>> The current docs are being written and are available at >>>> http://docs.scipy.org/doc/ >>>> >>>> Josef >>>> >> >> >> Yes, I think is the same, I got a ValueError as well, having upgraded >> to the latest (r5655) version. >> >> Traceback (most recent call last): >> ?File "sp2.py", line 6, in >> ? ?Asp[200:250,30] = 10*random.rand(50) >> ?File "/usr/lib/python2.6/site-packages/scipy/sparse/lil.py", line >> 329, in __setitem__ >> ? ?self._insertat3(row, data, j, xx) >> ?File "/usr/lib/python2.6/site-packages/scipy/sparse/lil.py", line >> 285, in _insertat3 >> ? ?self._insertat2(row, data, j, x) >> ?File "/usr/lib/python2.6/site-packages/scipy/sparse/lil.py", line >> 246, in _insertat2 >> ? ?raise ValueError('setting an array element with a sequence') >> ValueError: setting an array element with a sequence >> >> Checked out the new documentation you referenced[1] and there is only >> same-row assignment (e.g. A[0, :100] = rand(100) ?) but no same-column >> assignment... >> >> So still, my question is that is there something inherently different >> between array -> row and array -> column assigment in this case? > > I'm not completely sure about the internal structure, but essentially > the non-zero values are stored in row-wise lists, to access a column > slice it needs to access all the rowlists and insert to each, and this > is much slower. > > see the explanation in the documentation describing the different formats. > > Josef > I opened a ticket http://projects.scipy.org/scipy/ticket/917 Josef From jsalvati at u.washington.edu Thu Apr 9 13:43:35 2009 From: jsalvati at u.washington.edu (John Salvatier) Date: Thu, 9 Apr 2009 10:43:35 -0700 Subject: [SciPy-user] Question including a boundary value problem solver in SciPy Message-ID: <113e17f20904091043m5762fbd0m696bf032812934ba@mail.gmail.com> Hello, I am not sure this is the right place to ask this question, but I have wrapped a boundary value problem solver based on the Fortran code BVP_SOLVER. The PyPi page is here , and I have a tutorial for it here . I am interested in integrating this into SciPy, so I have two questions: 1) Does this type of thing belong in SciPy? 2) if it is, what are the steps to getting this included in into SciPy? I am in the process of getting BVP_SOLVER available under a BSD license. One of the authors has made his license compatible with the BSD license, but the Fortran algorithm currently relies on COLROW, which is subject to the ACM license. The same author has told me he plans on replacing COLROW with RESCALE which is free for both commercial and non-commercial use. Thank you for your time, John Salvatier -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtravs at gmail.com Fri Apr 10 05:23:23 2009 From: jtravs at gmail.com (John Travers) Date: Fri, 10 Apr 2009 10:23:23 +0100 Subject: [SciPy-user] Question including a boundary value problem solver in SciPy In-Reply-To: <113e17f20904091043m5762fbd0m696bf032812934ba@mail.gmail.com> References: <113e17f20904091043m5762fbd0m696bf032812934ba@mail.gmail.com> Message-ID: <3a1077e70904100223n580472dev4ca683f30feef718@mail.gmail.com> On Thu, Apr 9, 2009 at 6:43 PM, John Salvatier wrote: > Hello, > > I am not sure this is the right place to ask this question, but I have > wrapped a boundary value problem solver based on the Fortran code > BVP_SOLVER. The PyPi page is here, and I have a tutorial for it here. scipy-dev. I have cross-posted this reply. Please respond to the scipy-dev thread. > > I am interested in integrating this into SciPy, so I have two questions: 1) > Does this type of thing belong in SciPy? 2) if it is, what are the steps to > getting this included in into SciPy? > Hi John, This would be a very welcome addition! I also made a wrapper for this code a few years back. Apart from the license issues, which it sounds like you are solving, the main problem with direct inclusion is that it is written in fortran 90, which scipy currently does not depend on. At the time I was told that making it a scikit would be the best option for inclusion, and I think that is still true. This allows scipy to still be independent of fortran 90 code. On the other hand, as gfortran is now getting more stable and reasonably fast maybe it is time to re-visit this issue. Given that over the past two years I haven't made any moves to actually make a scikit, then I guess you have more momentum to go ahead with this. Note that there is a different boundary bvalue problem solver available (http://www.elisanet.fi/ptvirtan/software/bvp/index.html), it would be nice if both were combined into a single scikit. Cheers, John From shchelokovskyy at gmail.com Fri Apr 10 11:58:29 2009 From: shchelokovskyy at gmail.com (Pavlo Shchelokovskyy) Date: Fri, 10 Apr 2009 17:58:29 +0200 Subject: [SciPy-user] 16-bit images and ndimage filters Message-ID: Hi all, I'm experiencing strange behaviour of SciPy concerning the subject. I need to process a 16-bit png image. The PIL is installed and scipy.misc.imread loads the image just fine as int32 numpy array. But then filters from scipy.ndimage give me the error "RuntimeError: array type 5 not supported". Only after converting the image back and forth (i.e. to float and back to int32) the error is gone. So I do not know whether this problem is of PIL or SciPy but it seems that we have two different int32-types. I'm using Python 2.5.4 with NumPy 1.3, SciPy 0.7 and PIL 1.1.6 on Vista Here is an example: Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from scipy.ndimage import gaussian_filter >>> from scipy.misc import imread >>> from scipy import asarray, asfarray, int32 >>> img = imread('16bit.png') >>> img.dtype dtype('int32') >>> gaussian_filter(img, 1) Traceback (most recent call last): File "", line 1, in File "C:\Python25\lib\site-packages\scipy\ndimage\filters.py", line 268, in gaussian_filter mode, cval) File "C:\Python25\lib\site-packages\scipy\ndimage\filters.py", line 220, in gaussian_filter1d return correlate1d(input, weights, axis, output, mode, cval, 0) File "C:\Python25\lib\site-packages\scipy\ndimage\filters.py", line 129, in correlate1d origin) RuntimeError: array type 5 not supported >>> imgi = asarray(asfarray(img), int32) >>> imgi.dtype dtype('int32') >>> gaussian_filter(imgi,1) array([[11181, 11194, 11240, ..., 11412, 11451, 11438], [11332, 11328, 11350, ..., 11429, 11467, 11445], [11419, 11433, 11442, ..., 11437, 11451, 11411], ..., [11373, 11373, 11354, ..., 11688, 11712, 11728], [11486, 11472, 11461, ..., 11725, 11737, 11777], [11556, 11545, 11535, ..., 11834, 11807, 11803]]) From stefan at sun.ac.za Fri Apr 10 15:09:33 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 10 Apr 2009 21:09:33 +0200 Subject: [SciPy-user] 16-bit images and ndimage filters In-Reply-To: References: Message-ID: <9457e7c80904101209s2a205246v2c2ef0266e9c997b@mail.gmail.com> Hi Pavlo 2009/4/10 Pavlo Shchelokovskyy : > scipy.misc.imread loads the image just fine as int32 numpy array. But > then filters from scipy.ndimage give me the error "RuntimeError: array > type 5 not supported". Only after converting the image back and forth > (i.e. to float and back to int32) the error is gone. It sounds like ndimage should be comparing dtypes but is, in fact, comparing array types. I would file it as a bug. Cheers St?fan From stefan at sun.ac.za Fri Apr 10 15:47:22 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 10 Apr 2009 21:47:22 +0200 Subject: [SciPy-user] sparse SVD In-Reply-To: <49DD1F65.40404@gmail.com> References: <49DD1F65.40404@gmail.com> Message-ID: <9457e7c80904101247t1c12d0f7y4810e4454f071a42@mail.gmail.com> 2009/4/9 Rob Patro : > ?Is there any implementation of sparse SVD available in scipy? ?If not, > does anyone know of an implementation available in python at all? ?I'd > like to port a project on which I'm working from Matlab to Python, but > it is crucial that I am able to perform the SVD of large and *very* > sparse matrices. Michael Berry informed me that ARPACK can also do Sparse SVD. That may be a good place to start, since we already have ARPACK wrappers in place. Regards St?fan From gael.varoquaux at normalesup.org Fri Apr 10 16:04:50 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 10 Apr 2009 22:04:50 +0200 Subject: [SciPy-user] LARS algorithm Message-ID: <20090410200450.GA3641@phare.normalesup.org> Has anybody implemented the LARS[1] (least angle regression) algorithm for regularized regression in Python or in C binded in Python? I am about to start in such endeaviour, but I wanted to check if someone was willing to share code under a BSD-compatible license. Ga?l [1] http://en.wikipedia.org/wiki/Least-angle_regression PS: Sorry for the cross-post, I just wanted to be really sure that if anybody was willing to share such code, I would reach him :) From emanuele at relativita.com Fri Apr 10 16:42:01 2009 From: emanuele at relativita.com (emanuele at relativita.com) Date: Fri, 10 Apr 2009 22:42:01 +0200 (CEST) Subject: [SciPy-user] LARS algorithm In-Reply-To: <20090410200450.GA3641@phare.normalesup.org> References: <20090410200450.GA3641@phare.normalesup.org> Message-ID: Per Sedeberg wrapped LARS (through RPy) in PyMVPA. Later he mentioned about a C implementation he found and it seems he's going to work on it: http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2009q1/000404.html I guess you should contact him. Emanuele On Fri, April 10, 2009 10:04 pm, Gael Varoquaux wrote: > Has anybody implemented the LARS[1] (least angle regression) algorithm for > regularized regression in Python or in C binded in Python? > > I am about to start in such endeaviour, but I wanted to check if someone > was willing to share code under a BSD-compatible license. > > Ga?l > > [1] http://en.wikipedia.org/wiki/Least-angle_regression > > PS: Sorry for the cross-post, I just wanted to be really sure that if > anybody was willing to share such code, I would reach him :) > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From josef.pktd at gmail.com Sat Apr 11 14:17:45 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 11 Apr 2009 14:17:45 -0400 Subject: [SciPy-user] failurs with running scipy inplace Message-ID: <1cd32cbb0904111117vcffc7e2w8c845c81cdb37ca1@mail.gmail.com> I was trying to see if working inplace with bzr branches, can improve the workflow, but I got some errors that I'm not sure are because of my setup or "real" I did python setup.py build_ext -i and added it to the python path scipy.stsci doesn't have a proper package structure in the source tree, no __init__.py in convolve and image. there are no problems with a regular, non-inplace build I don't know anything about the other test failures, and I would like to know if someone else also gets these or if they are setup specific. working from a bzr branch doesn't add the revision number Josef >>> scipy.test() Running unit tests for scipy NumPy version 1.3.0b1 NumPy is installed in c:\programs\python25\lib\site-packages\numpy SciPy version 0.8.0.dev SciPy is installed in C:\Josef\eclipsegworkspace\scipy-branch\scipy ====================================================================== ERROR: Failure: ImportError (No module named convolve) ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.10.4-py2.5.egg\nose\loader .py", line 364, in loadTestsFromName addr.filename, addr.module) File "c:\programs\python25\lib\site-packages\nose-0.10.4-py2.5.egg\nose\import er.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "c:\programs\python25\lib\site-packages\nose-0.10.4-py2.5.egg\nose\import er.py", line 84, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "C:\Josef\eclipsegworkspace\scipy-branch\scipy\stsci\__init__.py", line 1 , in import convolve ImportError: No module named convolve ====================================================================== FAIL: test whether all methods converge ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Josef\eclipsegworkspace\scipy-branch\scipy\sparse\linalg\isolve\tests \test_iterative.py", line 101, in test_convergence assert_equal(info,0) File "C:\Programs\Python25\Lib\site-packages\numpy\testing\utils.py", line 190 , in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: -10 DESIRED: 0 ====================================================================== FAIL: test whether maxiter is respected ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Josef\eclipsegworkspace\scipy-branch\scipy\sparse\linalg\isolve\tests \test_iterative.py", line 82, in test_maxiter assert_equal(len(residuals), 3) File "C:\Programs\Python25\Lib\site-packages\numpy\testing\utils.py", line 190 , in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: 1 DESIRED: 3 ====================================================================== FAIL: test whether all methods accept a trivial preconditioner ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Josef\eclipsegworkspace\scipy-branch\scipy\sparse\linalg\isolve\tests \test_iterative.py", line 132, in test_precond assert_equal(info,0) File "C:\Programs\Python25\Lib\site-packages\numpy\testing\utils.py", line 190 , in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: -10 DESIRED: 0 ====================================================================== FAIL: test_yv_cephes_vs_amos (test_basic.TestBessel) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Josef\eclipsegworkspace\scipy-branch\scipy\special\tests\test_basic.p y", line 1664, in test_yv_cephes_vs_amos self.check_cephes_vs_amos(yv, yn, rtol=1e-11, atol=1e-305) File "C:\Josef\eclipsegworkspace\scipy-branch\scipy\special\tests\test_basic.p y", line 1653, in check_cephes_vs_amos assert c2.imag != 0, (v, z) AssertionError: (301, 1.0) ---------------------------------------------------------------------- Ran 3514 tests in 69.891s FAILED (KNOWNFAIL=2, SKIP=29, errors=1, failures=4) From josef.pktd at gmail.com Sat Apr 11 16:10:01 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 11 Apr 2009 16:10:01 -0400 Subject: [SciPy-user] failurs with running scipy inplace In-Reply-To: <1cd32cbb0904111117vcffc7e2w8c845c81cdb37ca1@mail.gmail.com> References: <1cd32cbb0904111117vcffc7e2w8c845c81cdb37ca1@mail.gmail.com> Message-ID: <1cd32cbb0904111310p3f17e8cs372189ec4b24e597@mail.gmail.com> > ---------------------------------------------------------------------- > Ran 3514 tests in 69.891s > > FAILED (KNOWNFAIL=2, SKIP=29, errors=1, failures=4) > > I think the isolve errors are a consequence of a failed attempt to install umfpack, and is independent of inplace. I currently get compile errors if I follow my usual build routine, if I remove umfpack from numpy/distutils/site.cfg. So I need to debug my compile setup. Josef From general.mooney at googlemail.com Sun Apr 12 09:59:28 2009 From: general.mooney at googlemail.com (=?ISO-8859-1?Q?Ciar=E1n_Mooney?=) Date: Sun, 12 Apr 2009 14:59:28 +0100 Subject: [SciPy-user] [Announcement] Europython 2009 Message-ID: <3e4e51a80904120659r6ff6270la2dcbbd7de16b640@mail.gmail.com> Hi, There has been a inkling of interest in a Scipy/Numpy tutorial on the Europython 2009 lists. But we need someone to lead it! So even though the deadline for papers has passed, they may still be open to an offer. If not, please come along anyway to mingle, meet and network with other python-ista's. The following is from the Europython Commitee. ====== On behalf of the EuroPython 2009 organisation it is my privilege and honour to announce the 'Call for Participation' for EuroPython 2009! EuroPython is the conference for the communities around Python, including the Django, Zope and Plone communities. This years conference will be held in Birmingham, UK from Monday 29th June to Saturday 4th July 2009. Talk & Themes Do you have something you wish to present at EuroPython? Go to http://www.europython.eu/talks/cfp/ for this years themes and submissions criteria, the deadline is on 5th April 2009. Other Talks, Activities and Events Have you got something which does not fit the above? Visit http://www.europython.eu/talks/ Help Us Out We could use a hand - any contribution is welcome. Please take a look at http://www.europython.eu/contact/ . Sponsors An unique opportunity to affiliate with the prestigious EuroPython conference! http://www.europython.eu/sponsors/ Spread the Word Improve our publicity by distributing this announcement in your corner of the community, please coordinate this with the organizers: http://www.europython.eu/contact/ General Information For more information about the conference, please visit http://www.europython.eu/ Looking forward to seeing you! The EuroPython Team =========== Regards, Ciar?n From josef.pktd at gmail.com Sun Apr 12 23:52:11 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 12 Apr 2009 23:52:11 -0400 Subject: [SciPy-user] failurs with running scipy inplace In-Reply-To: <1cd32cbb0904111310p3f17e8cs372189ec4b24e597@mail.gmail.com> References: <1cd32cbb0904111117vcffc7e2w8c845c81cdb37ca1@mail.gmail.com> <1cd32cbb0904111310p3f17e8cs372189ec4b24e597@mail.gmail.com> Message-ID: <1cd32cbb0904122052i5ccc72e9ha732108995109f9f@mail.gmail.com> On Sat, Apr 11, 2009 at 4:10 PM, wrote: >> ---------------------------------------------------------------------- >> Ran 3514 tests in 69.891s >> >> FAILED (KNOWNFAIL=2, SKIP=29, errors=1, failures=4) >> >> > > I think the isolve errors are a consequence of a failed attempt to > install umfpack, and is independent of inplace. > > I currently get compile errors if I follow my usual build routine, if > I remove umfpack from numpy/distutils/site.cfg. > So I need to debug my compile setup. > > Josef > rebuilding several times and the failures persist, My mistake, I should check my own emails and the trac tickets, all failures have been reported before. I just got too used to seeing zero failures. Sorry for the noise. The only difference between inplace and bdist build seems then to be in stsci. Josef From Andrew.G.York+scipy at gmail.com Mon Apr 13 02:39:05 2009 From: Andrew.G.York+scipy at gmail.com (Andrew York) Date: Mon, 13 Apr 2009 02:39:05 -0400 Subject: [SciPy-user] Orthogonal polynomial evaluation Message-ID: <744bb3c80904122339sc7eed7dva6a1716fcf6ba0ba@mail.gmail.com> I am running a simulation with Scipy that needs to numerically evaluate Jacobi polynomials. I have been using the function scipy.special.orthogonal.jacobi() heavily. Evaluations of this function are >90% of my simulation running time. Much more troubling, I begin to see unphysical results when I need to evaluate high-order Jacobi polynomials, and I suspect I am running into the problem warned against here: http://docs.scipy.org/doc/scipy/reference/special.html "Warning Evaluating large-order polynomials using these functions can be numerically unstable." Another Scipy user seems to have run into a similar problem: http://thread.gmane.org/gmane.comp.python.scientific.user/11206 So, it seems the correct course of action is clear: I should identify a stable algorithm to numerically evaluate Jacobi polynomials, I should implement this algorithm in something fast like C, and then call it from within python. I found some promising information on a possible algorithm: http://portal.acm.org/citation.cfm?id=363393 http://portal.acm.org/citation.cfm?id=362686.362700 https://people.sc.fsu.edu/~burkardt/f77_src/toms332/toms332.html But I do not speak FORTRAN. I am also new to python and not yet familiar with how to integrate a fast low-level language with python, nor am I especially proficient in any low-level language. I suspect if numerical evaluation of these functions was faster or more robust, other scipy users would also benefit. Should I attempt this myself (in C with weave?)? If I do, would others be interested in the results? Is this an easy task for anyone else? I would greatly appreciate any help or guidance. Thank you for your attention. From peridot.faceted at gmail.com Mon Apr 13 04:59:39 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Mon, 13 Apr 2009 04:59:39 -0400 Subject: [SciPy-user] Orthogonal polynomial evaluation In-Reply-To: <744bb3c80904122339sc7eed7dva6a1716fcf6ba0ba@mail.gmail.com> References: <744bb3c80904122339sc7eed7dva6a1716fcf6ba0ba@mail.gmail.com> Message-ID: 2009/4/13 Andrew York : > ?I am running a simulation with Scipy that needs to numerically > evaluate Jacobi polynomials. I have been using the function > scipy.special.orthogonal.jacobi() heavily. Evaluations of this > function are >90% of my simulation running time. Much more troubling, > I begin to see unphysical results when I need to evaluate high-order > Jacobi polynomials, and I suspect I am running into the problem warned > against here: > > http://docs.scipy.org/doc/scipy/reference/special.html > > "Warning Evaluating large-order polynomials using these functions can > be numerically unstable." Yes, unfortunately all scipy's orthogonal polynomials are computed using generic recurrence relations. This allows a quick flexible implementation, but as you and I both discovered it suffers from numerical instability. > Another Scipy user seems to have run into a similar problem: > > http://thread.gmane.org/gmane.comp.python.scientific.user/11206 > > ?So, it seems the correct course of action is clear: I should identify > a stable algorithm to numerically evaluate Jacobi polynomials, I > should implement this algorithm in something fast like C, and then > call it from within python. I found some promising information on a > possible algorithm: > > http://portal.acm.org/citation.cfm?id=363393 > http://portal.acm.org/citation.cfm?id=362686.362700 > https://people.sc.fsu.edu/~burkardt/f77_src/toms332/toms332.html > > But I do not speak FORTRAN. I am also new to python and not yet > familiar with how to integrate a fast low-level language with python, > nor am I especially proficient in any low-level language. I suspect if > numerical evaluation of these functions was faster or more robust, > other scipy users would also benefit. Should I attempt this myself (in > C with weave?)? If I do, would others be interested in the results? > > Is this an easy task for anyone else? I would greatly appreciate any > help or guidance. If you are comfortable with python but not FORTRAN or python-to-other-language linking, then I recommend using cython to implement compiled extensions. It can be very fast, and the path to a working implementation is all modest steps: * write a python point-at-a-time implementation based on the articles * write a python unit testing module (makes the rest much easier!) * compile the point-at-a-time implementation with cython * add enough type annotations that you can see that the generated C code contains no python calls * write a wrapper function in cython that calls the point-at-a-time function on each element of an array Ideally, I'd like to see the special functions module modified so that the orthogonal polynomials could use better implementations where available, falling back to the recurrence relation. But it's kind of a maze. Good luck, Anne From lou_boog2000 at yahoo.com Mon Apr 13 09:52:16 2009 From: lou_boog2000 at yahoo.com (Lou Pecora) Date: Mon, 13 Apr 2009 06:52:16 -0700 (PDT) Subject: [SciPy-user] Orthogonal polynomial evaluation Message-ID: <730577.76173.qm@web34403.mail.mud.yahoo.com> I recall that stable routines to calculate Bessel functions use descending recurrence relations (descending in indices), rather than ascending recurrence which is unstable. ?Only trick is that the values have to be renormalized at the end (or as you go to keep the numbers in floating point range, I think) since descending recurrence does not set the scale initially. ?I wonder if this is the case for other recurrence relations. ?That is, if one recurrence is unstable (e.g. ascending), then the opposite (descending) will be stable. ?Perhaps the scipy routines can be easily reworked, if so. ?Just a thought. -- Lou Pecora, my views are my own. --- On Mon, 4/13/09, Anne Archibald wrote: From: Anne Archibald Yes, unfortunately all scipy's orthogonal polynomials are computed using generic recurrence relations. This allows a quick flexible implementation, but as you and I both discovered it suffers from numerical instability. > Another Scipy user seems to have run into a similar problem: > > http://thread.gmane.org/gmane.comp.python.scientific.user/11206 > > ?So, it seems the correct course of action is clear: I should identify > a stable algorithm to numerically evaluate Jacobi polynomials, I > should implement this algorithm in something fast like C, and then > call it from within python. I found some promising information on a > possible algorithm: > > http://portal.acm.org/citation.cfm?id=363393 > http://portal.acm.org/citation.cfm?id=362686.362700 > https://people.sc.fsu.edu/~burkardt/f77_src/toms332/toms332.html > > But I do not speak FORTRAN. I am also new to python and not yet > familiar with how to integrate a fast low-level language with python, > nor am I especially proficient in any low-level language. I suspect if > numerical evaluation of these functions was faster or more robust, > other scipy users would also benefit. Should I attempt this myself (in > C with weave?)? If I do, would others be interested in the results? > > Is this an easy task for anyone else? I would greatly appreciate any > help or guidance. If you are comfortable with python but not FORTRAN or python-to-other-language linking, then I recommend using cython to implement compiled extensions. It can be very fast, and the path to a working implementation is all modest steps: * write a python point-at-a-time implementation based on the articles * write a python unit testing module (makes the rest much easier!) * compile the point-at-a-time implementation with cython * add enough type annotations that you can see that the generated C code contains no python calls * write a wrapper function in cython that calls the point-at-a-time function on each element of an array Ideally, I'd like to see the special functions module modified so that the orthogonal polynomials could use better implementations where available, falling back to the recurrence relation. But it's kind of a maze. Good luck, Anne _______________________________________________ SciPy-user mailing list SciPy-user at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Mon Apr 13 13:35:19 2009 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 13 Apr 2009 17:35:19 +0000 (UTC) Subject: [SciPy-user] Orthogonal polynomial evaluation References: <730577.76173.qm@web34403.mail.mud.yahoo.com> Message-ID: Mon, 13 Apr 2009 06:52:16 -0700, Lou Pecora wrote: > > I recall that stable routines to calculate Bessel functions use > descending recurrence relations (descending in indices), rather than > ascending recurrence which is unstable. ?Only trick is that the values > have to be renormalized at the end (or as you go to keep the numbers in > floating point range, I think) since descending recurrence does not set > the scale initially. ?I wonder if this is the case for other recurrence > relations. ?That is, if one recurrence is unstable (e.g. ascending), > then the opposite (descending) will be stable. ?Perhaps the scipy > routines can be easily reworked, if so. ?Just a thought. Yes, the errors in polynomial evaluation come from a source somewhat similar to the reason why Bessel recurrence relations have to be evaluated in a certaion direction: loss of precision. The problem here is that how the routines in Scipy currently work is that they compute the polynomial coefficients and produce numpy.poly1d objects. All poly1d objects evaluate the polynomials using the *same* Horner recurrence scheme, but it is not numerically stable for high-order polynomials. The problem is that each of the orthogonal polynomial would need to be evaluated using a specialized recurrence relation. We don't currently have implementations of stable evaluation algorithms. So, a complete rewrite of the orthogonal polynomial routines is required. Ticket: http://projects.scipy.org/scipy/ticket/921 -- Pauli Virtanen From peridot.faceted at gmail.com Mon Apr 13 14:26:27 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Mon, 13 Apr 2009 14:26:27 -0400 Subject: [SciPy-user] Orthogonal polynomial evaluation In-Reply-To: References: <730577.76173.qm@web34403.mail.mud.yahoo.com> Message-ID: 2009/4/13 Pauli Virtanen : > Mon, 13 Apr 2009 06:52:16 -0700, Lou Pecora wrote: >> >> I recall that stable routines to calculate Bessel functions use >> descending recurrence relations (descending in indices), rather than >> ascending recurrence which is unstable. ?Only trick is that the values >> have to be renormalized at the end (or as you go to keep the numbers in >> floating point range, I think) since descending recurrence does not set >> the scale initially. ?I wonder if this is the case for other recurrence >> relations. ?That is, if one recurrence is unstable (e.g. ascending), >> then the opposite (descending) will be stable. ?Perhaps the scipy >> routines can be easily reworked, if so. ?Just a thought. > > Yes, the errors in polynomial evaluation come from a source somewhat > similar to the reason why Bessel recurrence relations have to be > evaluated in a certaion direction: loss of precision. > > The problem here is that how the routines in Scipy currently work is that > they compute the polynomial coefficients and produce numpy.poly1d > objects. All poly1d objects evaluate the polynomials using the *same* > Horner recurrence scheme, but it is not numerically stable for high-order > polynomials. The problem is that each of the orthogonal polynomial would > need to be evaluated using a specialized recurrence relation. > > We don't currently have implementations of stable evaluation algorithms. > So, a complete rewrite of the orthogonal polynomial routines is required. It's not just a question of polynomial evaluation schemes; the polynomial coefficients themselves present severe problems. I'd say that 1, x, ..., x**n is often a very poor basis for expressing polynomials. It would actually be nice to have a polynomials class that could use other bases - in fact families of orthogonal polynomials are natural bases for polynomials. But let's leave aside poly1d entirely. There are algorithms, and I thought there was code in scipy, to evaluate orthogonal polynomials, their zeros, and their derivatives and integrals, in terms of recurrence relations. These are quite general, fairly efficient, and reasonably accurate - for small orders. But many orthogonal polynomials have special-purpose evaluation algorithms: for example the cos(n*arccos(x)) expression for Chebyshev polynomials. Such algorithms are usually fast and accurate, but require separate implementation for each family that has one. I think making it possible to simply override the evaluation of a particular family of orthogonal polynomials would be a better solution to the OP's problem. Of course, once one had efficient accurate evaluation, then one could think about implementing 1D polynomials represented in other bases... Anne > Ticket: http://projects.scipy.org/scipy/ticket/921 From pav at iki.fi Mon Apr 13 16:00:36 2009 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 13 Apr 2009 20:00:36 +0000 (UTC) Subject: [SciPy-user] Orthogonal polynomial evaluation References: <730577.76173.qm@web34403.mail.mud.yahoo.com> Message-ID: Mon, 13 Apr 2009 14:26:27 -0400, Anne Archibald wrote: > 2009/4/13 Pauli Virtanen : [clip] >> We don't currently have implementations of stable evaluation >> algorithms. So, a complete rewrite of the orthogonal polynomial >> routines is required. > > It's not just a question of polynomial evaluation schemes; the > polynomial coefficients themselves present severe problems. I'd say that > 1, x, ..., x**n is often a very poor basis for expressing polynomials. > It would actually be nice to have a polynomials class that could use > other bases - in fact families of orthogonal polynomials are natural > bases for polynomials. Yes, this was exactly my point: evaluating polynomials using the basic coefficients is not suitable for high-order polynomials. > But let's leave aside poly1d entirely. There are algorithms, and I > thought there was code in scipy, to evaluate orthogonal polynomials, > their zeros, and their derivatives and integrals, in terms of recurrence > relations. These are quite general, fairly efficient, and reasonably > accurate - for small orders. But many orthogonal polynomials have > special-purpose evaluation algorithms: for example the cos(n*arccos(x)) > expression for Chebyshev polynomials. Such algorithms are usually fast > and accurate, but require separate implementation for each family that > has one. I think making it possible to simply override the evaluation of > a particular family of orthogonal polynomials would be a better solution > to the OP's problem. Perhaps I said it earlier in some strange way, but this is exactly the solution I proposed. The problem is that the current code in orthogonal.py only computes the coefficients in the 1, x, x^2, ... basis, so we don't have specialized algorithms for evaluating the polynomials. -- Pauli Virtanen From pav at iki.fi Mon Apr 13 16:09:58 2009 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 13 Apr 2009 20:09:58 +0000 (UTC) Subject: [SciPy-user] Orthogonal polynomial evaluation References: <730577.76173.qm@web34403.mail.mud.yahoo.com> Message-ID: Mon, 13 Apr 2009 20:00:36 +0000, Pauli Virtanen wrote: [clip] > The problem is that the current code in orthogonal.py only computes the > coefficients in the 1, x, x^2, ... basis, so we don't have specialized > algorithms for evaluating the polynomials. Sorry, this was inaccurate. The algorithms actually compute the polynomial roots and an overall coefficient. I wonder if simply making orthopoly1d evaluate the polynomials using the Newton scheme would fix stability issues... Needs some thinking. -- Pauli Virtanen From pav at iki.fi Mon Apr 13 20:21:15 2009 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 14 Apr 2009 00:21:15 +0000 (UTC) Subject: [SciPy-user] Orthogonal polynomial evaluation References: <730577.76173.qm@web34403.mail.mud.yahoo.com> Message-ID: Mon, 13 Apr 2009 20:09:58 +0000, Pauli Virtanen wrote: [clip] > I wonder if simply making orthopoly1d evaluate the polynomials using the > Newton scheme would fix stability issues... Needs some thinking. This seems to actually help quite a lot: see the branch at http://github.com/pv/scipy-work/tree/ticket/921-orthogonal For example, Before (Horner): >>> import scipy.special as sc >>> sc.chebyt(300)(0.999) - np.cos(300*np.arccos(0.999)) -2.9807351165313966e+117 After (using roots): >>> import scipy.special as sc >>> sc.chebyt(300)(0.999) - np.cos(300*np.arccos(0.999)) 1.9129142714291447e-13 Specialized evaluation routines could of course be faster; finding the zeros of the polynomial only for evaluating it is overkill... -- Pauli Virtanen From jh at physics.ucf.edu Wed Apr 15 00:56:31 2009 From: jh at physics.ucf.edu (Joe Harrington) Date: Wed, 15 Apr 2009 00:56:31 -0400 Subject: [SciPy-user] JOB: write numpy docs In-Reply-To: (message from Joe Harrington on Mon, 30 Mar 2009 12:26:39 -0400) References: Message-ID: So far I have no takers for the numpy doc job (below), and the lack of activity on docs.scipy.org indicates nobody new is preparing an application. This job is open (as a subcontract) to international applicants. It could be part-time, if a suitable full-time person is not available. It could be a volunteer job, if you're gainfully employed and would prefer it that way. If you have substantial free time this summer and you want to have full docs for numpy, please contact me. Thanks, --jh-- Date: Mon, 30 Mar 2009 12:26:39 -0400 To: Scipy Developers , Scipy Users , Numpy Discussion Subject: JOB: write numpy docs From: Joe Harrington CC: jh at physics.ucf.edu Reply-to: jh at physics.ucf.edu Last year's Doc Marathon got us off to a great start on documenting NumPy! But, there's still much work to be done, and SciPy after that. It's time to gear up for doing it again. Critical to last year's success was Stefan van der Walt's committed time, but he will be unable to play that role this year. So, I am looking to hire someone to write NumPy docs and help coordinate the doc project and its volunteers. The job includes working with me, the doc team, doc volunteers, and developers to: write and review a lot of docs, mainly those that others don't want to write help define milestones organize campaigns and volunteer teams to meet them research the NumPy and SciPy source codes to help plan: the eventual SciPy documentation the writing of a good User Manual work with the packaging team to meet their release deadlines perform other duties as assigned I am seeking someone to work full time if possible, and at least half time, from mid-April (or soon thereafter) through at least the (northern) summer. Candidates must be experienced NumPy and SciPy programmers; familiarity under the hood is a strong plus. They must also demonstrate their ability to produce excellent docs on the docs.SciPy.org wiki. Having contributed at a high level to an open-source community, especially to SciPy, is a big plus. Ability to take direction, work with and lead a team, and to work for extended periods without direct supervision on a list of assigned tasks are all critical. The applicant must be able to function well in a Linux environment; familiarity with multiple platforms is a plus. Please reply directly to me by email only. Include the following (PDF or ASCII formats strongly preferred): CV Statement of interest, qualifications per requirements above, availability, and wage expectations. Contact info for at least 3 professional references. Links to doc wiki pages for which you wrote the initial draft Links to doc wiki pages started by others to which you contributed significantly (edited, reviewed, proofed) The position is open until filled; candidates with complete applications by April 15 will receive full consideration. This is an open posting. Candidates who have not written any pages on the doc wiki yet have several weeks in which to do so. Pay will be commensurate with experience (up to a point). Relocation is not necessary. Candidates will need to provide their own computer and internet access. The University of Central Florida is an equal opportunity, equal access, affirmative action employer. --jh-- Prof. Joseph Harrington Department of Physics MAP 414 4000 Central Florida Blvd. University of Central Florida Orlando, FL 32816-2385 (407) 823-3416 voice (407) 823-5112 fax (407) 823-2325 physics office jh at physics.ucf.edu From ramercer at gmail.com Wed Apr 15 01:03:45 2009 From: ramercer at gmail.com (Adam Mercer) Date: Wed, 15 Apr 2009 00:03:45 -0500 Subject: [SciPy-user] Build error on PPC Mac OS X 10.4 Message-ID: <799406d60904142203n560627dcp1525d978c817818c@mail.gmail.com> Hi I maintain scipy for MacPorts and a user has reported a build problem with PowerPC Mac OS X 10.4 and scipy-0.7.0 (against numpy-1.3.0). The build is failing with the following error: building 'dfftpack' library compiling Fortran sources Fortran f77 compiler: /opt/local/bin/gfortran-mp-4.3 -Wall -ffixed-form -fno-second-underscore -arch ppc -fPIC -O3 -funroll-loops Fortran f90 compiler: /opt/local/bin/gfortran-mp-4.3 -Wall -fno-second-underscore -arch ppc -fPIC -O3 -funroll-loops Fortran fix compiler: /opt/local/bin/gfortran-mp-4.3 -Wall -ffixed-form-fno-second-underscore -Wall -fno-second-underscore -arch ppc -fPIC -O3 -funroll-loops creating build/temp.macosx-10.4-ppc-2.5 creating build/temp.macosx-10.4-ppc-2.5/scipy creating build/temp.macosx-10.4-ppc-2.5/scipy/fftpack creating build/temp.macosx-10.4-ppc-2.5/scipy/fftpack/src creating build/temp.macosx-10.4-ppc-2.5/scipy/fftpack/src/dfftpack compile options: '-I/opt/local/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/core/include -c' gfortran-mp-4.3:f77: scipy/fftpack/src/dfftpack/dcosqb.f f951: error: unrecognized command line option "-arch" f951: error: unrecognized command line option "-arch" error: Command "/opt/local/bin/gfortran-mp-4.3 -Wall -ffixed-form -fno-second-underscore -arch ppc -fPIC -O3 -funroll-loops -I/opt/local/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/core/include -c -c scipy/fftpack/src/dfftpack/dcosqb.f -o build/temp.macosx-10.4-ppc-2.5/scipy/fftpack/src/dfftpack/dcosqb.o" failed with exit status 1 >From what I can gather this is due to the flags "-arch ppc" being specified on the command line to the compiler. On my Intel 10.5 system, no architecture specific flags are passed to the compiler. Does anyone why these options are being passed? Or how to try and solve this? This seems to be a similar problem to Cheers Adam From david at ar.media.kyoto-u.ac.jp Wed Apr 15 02:13:39 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 15 Apr 2009 15:13:39 +0900 Subject: [SciPy-user] Build error on PPC Mac OS X 10.4 In-Reply-To: <799406d60904142203n560627dcp1525d978c817818c@mail.gmail.com> References: <799406d60904142203n560627dcp1525d978c817818c@mail.gmail.com> Message-ID: <49E57B13.10709@ar.media.kyoto-u.ac.jp> Adam Mercer wrote: > Hi > > I maintain scipy for MacPorts and a user has reported a build problem > with PowerPC Mac OS X 10.4 and scipy-0.7.0 (against numpy-1.3.0). The > build is failing with the following error: > Which compiler macport relies on ? The arch specific flags are necessary for universal builds. cheers, David From stefan at sun.ac.za Wed Apr 15 03:25:54 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 15 Apr 2009 09:25:54 +0200 Subject: [SciPy-user] [SciPy-dev] JOB: write numpy docs In-Reply-To: References: Message-ID: <9457e7c80904150025r34e967a4w7506b2fb7ae6be12@mail.gmail.com> 2009/4/15 Joe Harrington : > So far I have no takers for the numpy doc job (below), and the lack of > activity on docs.scipy.org indicates nobody new is preparing an > application. ?This job is open (as a subcontract) to international > applicants. ?It could be part-time, if a suitable full-time person is > not available. ?It could be a volunteer job, if you're gainfully > employed and would prefer it that way. ?If you have substantial free > time this summer and you want to have full docs for numpy, please > contact me. I'm surprised no-one wants to take this deal! * Learn the inner workings of NumPy and SciPy, and * Get paid to do it * Have your work published for all to see * Make a highly valued contribution to a central open source project I thoroughly enjoyed working on this project last year -- wish I had the opportunity to do it again! Cheers St?fan From Andrew.G.York+scipy at gmail.com Wed Apr 15 03:52:21 2009 From: Andrew.G.York+scipy at gmail.com (Andrew York) Date: Wed, 15 Apr 2009 03:52:21 -0400 Subject: [SciPy-user] Orthogonal polynomial evaluation In-Reply-To: References: <730577.76173.qm@web34403.mail.mud.yahoo.com> Message-ID: <744bb3c80904150052qc85c5d4wc3fa65d0f9295ca6@mail.gmail.com> I found an inelegant but satisfactory solution to my problem, perhaps. I'm more comfortable with MATLAB than FORTRAN, and the MATLAB algorithm here: https://people.sc.fsu.edu/~burkardt/m_src/polpak/jacobi_poly.m was very straightforward to translate into python. Perhaps this algorithm is similar or equivalent to what Paul Virtanen suggested? This algorithm seems to run faster and resist explosion longer than scipy.special.orthogonal.jacobi(). I suppose that's not surprising if the scipy algorithm is meant to find zeros and only evaluates the polynomial unstably as an afterthought. If it's helpful, there are many similar algorithms in MATLAB (and other languages) for special polynomials here: https://people.sc.fsu.edu/~burkardt/m_src/polpak/polpak.html On Mon, Apr 13, 2009 at 8:21 PM, Pauli Virtanen wrote: > Mon, 13 Apr 2009 20:09:58 +0000, Pauli Virtanen wrote: > [clip] >> I wonder if simply making orthopoly1d evaluate the polynomials using the >> Newton scheme would fix stability issues... Needs some thinking. > > This seems to actually help quite a lot: see the branch at > > ? ? ? ?http://github.com/pv/scipy-work/tree/ticket/921-orthogonal > > For example, > > Before (Horner): > >>>> import scipy.special as sc >>>> sc.chebyt(300)(0.999) - np.cos(300*np.arccos(0.999)) > -2.9807351165313966e+117 > > After (using roots): > >>>> import scipy.special as sc >>>> sc.chebyt(300)(0.999) - np.cos(300*np.arccos(0.999)) > 1.9129142714291447e-13 > > Specialized evaluation routines could of course be faster; finding the > zeros of the polynomial only for evaluating it is overkill... > > -- > Pauli Virtanen > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From pav at iki.fi Wed Apr 15 04:10:50 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 15 Apr 2009 08:10:50 +0000 (UTC) Subject: [SciPy-user] Orthogonal polynomial evaluation References: <730577.76173.qm@web34403.mail.mud.yahoo.com> <744bb3c80904150052qc85c5d4wc3fa65d0f9295ca6@mail.gmail.com> Message-ID: Wed, 15 Apr 2009 03:52:21 -0400, Andrew York kirjoitti: [clip] > I found an inelegant but satisfactory solution to my problem, > perhaps. I'm more comfortable with MATLAB than FORTRAN, and the MATLAB > algorithm here: > > https://people.sc.fsu.edu/~burkardt/m_src/polpak/jacobi_poly.m > > was very straightforward to translate into python. Perhaps this > algorithm is similar or equivalent to what Paul Virtanen suggested? Yes, this was the idea. Thanks for the links. Unfortunately, we cannot really translate these codes, due to license issues. (It's LGPL, which is incompatible with BSD.) However, based on a quick look, it appears that they are straightforward implementations of the algorithm from Abramowitz & Stegun chapter 22.18. These should be straightforward to implement (using Cython indeed seems like a good idea here). -- Pauli Virtanen From gaedol at gmail.com Wed Apr 15 05:46:12 2009 From: gaedol at gmail.com (Marco) Date: Wed, 15 Apr 2009 11:46:12 +0200 Subject: [SciPy-user] efficiently write multiple arrays in the same file Message-ID: Hi all, any suggestion on how to (as efficiently as possible) write 2 or more n-dimensional arrays *in the same file*? Thanks in advance, marco From stefan at sun.ac.za Wed Apr 15 05:58:48 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 15 Apr 2009 11:58:48 +0200 Subject: [SciPy-user] efficiently write multiple arrays in the same file In-Reply-To: References: Message-ID: <9457e7c80904150258g3bf7d98crf3fc9a51a5cdf5a8@mail.gmail.com> Hi Marco 2009/4/15 Marco : > any suggestion on how to (as efficiently as possible) write 2 or more > n-dimensional arrays *in the same file*? Use numpy.savez. Regards St?fan From gaedol at gmail.com Wed Apr 15 06:04:03 2009 From: gaedol at gmail.com (Marco) Date: Wed, 15 Apr 2009 12:04:03 +0200 Subject: [SciPy-user] efficiently write multiple arrays in the same file In-Reply-To: <9457e7c80904150258g3bf7d98crf3fc9a51a5cdf5a8@mail.gmail.com> References: <9457e7c80904150258g3bf7d98crf3fc9a51a5cdf5a8@mail.gmail.com> Message-ID: 2009/4/15 St?fan van der Walt : > Use numpy.savez. Hi Stefan, thanks. Will try ASAP. marco From ramercer at gmail.com Wed Apr 15 08:44:42 2009 From: ramercer at gmail.com (Adam Mercer) Date: Wed, 15 Apr 2009 07:44:42 -0500 Subject: [SciPy-user] Build error on PPC Mac OS X 10.4 In-Reply-To: <49E57B13.10709@ar.media.kyoto-u.ac.jp> References: <799406d60904142203n560627dcp1525d978c817818c@mail.gmail.com> <49E57B13.10709@ar.media.kyoto-u.ac.jp> Message-ID: <799406d60904150544u41988da9re03e672729b91f0d@mail.gmail.com> On Wed, Apr 15, 2009 at 01:13, David Cournapeau wrote: > Which compiler macport relies on ? The arch specific flags are necessary > for universal builds. For fortran the user is given a choice between gfortran from gcc-4.2.x or gfortran from gcc-4.3.x, the default being from gcc-4.3.x. Its installed as gfortran-mp-4.{2,3} and it is explicitly not compiled universally. SciPy is also explictly set not to compile universally. Cheers Adam From rjchacko at gmail.com Wed Apr 15 14:19:43 2009 From: rjchacko at gmail.com (Ranjit Chacko) Date: Wed, 15 Apr 2009 14:19:43 -0400 Subject: [SciPy-user] enthought distribution problem In-Reply-To: References: Message-ID: <1E31E1B5-C22C-4CD7-9BA8-24C120B3B1DD@gmail.com> I just installed the Enthought distribution of scipy on my Mac running Leopard and I got the following error message when I tried to run Ipython. Any ideas on how to fix this?: ------------------------------------------------------------ Traceback (most recent call last): File "", line 1, in ImportError: No module named sage.misc.preparser_ipython WARNING: Failure executing code: 'import sage.misc.preparser_ipython; sage.misc.preparser_ipython.magma_colon_equals=True' ERROR: history is not a magic function ------------------------------------------------------------ Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/4.1.30101/lib/ python2.5/site-packages/ipython-0.9.1.0001_s-py2.5.egg/IPython/ ipmaker.py", line 66, in force_import __import__(modname) ImportError: No module named ipy_user_conf WARNING: /Users/rjchacko/.ipython/ipy_user_conf.py does not exist, please run %upgrade! WARNING: Loading of ipy_user_conf failed. From michael.abshoff at googlemail.com Wed Apr 15 15:38:38 2009 From: michael.abshoff at googlemail.com (Michael Abshoff) Date: Wed, 15 Apr 2009 12:38:38 -0700 Subject: [SciPy-user] enthought distribution problem In-Reply-To: <1E31E1B5-C22C-4CD7-9BA8-24C120B3B1DD@gmail.com> References: <1E31E1B5-C22C-4CD7-9BA8-24C120B3B1DD@gmail.com> Message-ID: <49E637BE.4040104@gmail.com> Ranjit Chacko wrote: Hi, > I just installed the Enthought distribution of scipy on my Mac running > Leopard and I got the following error message when I tried to run > Ipython. Any ideas on how to fix this?: > > ------------------------------------------------------------ > Traceback (most recent call last): > File "", line 1, in > ImportError: No module named sage.misc.preparser_ipython > > WARNING: Failure executing code: 'import sage.misc.preparser_ipython; > sage.misc.preparser_ipython.magma_colon_equals=True' > ERROR: history is not a magic function > ------------------------------------------------------------ > Traceback (most recent call last): > File "/Library/Frameworks/Python.framework/Versions/4.1.30101/lib/ > python2.5/site-packages/ipython-0.9.1.0001_s-py2.5.egg/IPython/ > ipmaker.py", line 66, in force_import > __import__(modname) > ImportError: No module named ipy_user_conf > > WARNING: /Users/rjchacko/.ipython/ipy_user_conf.py does not exist, > please run %upgrade! > WARNING: Loading of ipy_user_conf failed. > > Hmm, did you ever use Sage on that computer via sage -ipython for example. I would move /Users/rjchacko/.ipython/ out of the way and try again. Cheers, Michael > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From robert.kern at gmail.com Wed Apr 15 15:50:33 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 15 Apr 2009 14:50:33 -0500 Subject: [SciPy-user] Build error on PPC Mac OS X 10.4 In-Reply-To: <799406d60904150544u41988da9re03e672729b91f0d@mail.gmail.com> References: <799406d60904142203n560627dcp1525d978c817818c@mail.gmail.com> <49E57B13.10709@ar.media.kyoto-u.ac.jp> <799406d60904150544u41988da9re03e672729b91f0d@mail.gmail.com> Message-ID: <3d375d730904151250u3b96c775h18f2982f8964b817@mail.gmail.com> On Wed, Apr 15, 2009 at 07:44, Adam Mercer wrote: > On Wed, Apr 15, 2009 at 01:13, David Cournapeau > wrote: > >> Which compiler macport relies on ? The arch specific flags are necessary >> for universal builds. > > For fortran the user is given a choice between gfortran from gcc-4.2.x > or gfortran from gcc-4.3.x, the default being from gcc-4.3.x. Its > installed as gfortran-mp-4.{2,3} and it is explicitly not compiled > universally. SciPy is also explictly set not to compile universally. How? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ramercer at gmail.com Wed Apr 15 21:31:42 2009 From: ramercer at gmail.com (Adam Mercer) Date: Wed, 15 Apr 2009 20:31:42 -0500 Subject: [SciPy-user] Build error on PPC Mac OS X 10.4 In-Reply-To: <3d375d730904151250u3b96c775h18f2982f8964b817@mail.gmail.com> References: <799406d60904142203n560627dcp1525d978c817818c@mail.gmail.com> <49E57B13.10709@ar.media.kyoto-u.ac.jp> <799406d60904150544u41988da9re03e672729b91f0d@mail.gmail.com> <3d375d730904151250u3b96c775h18f2982f8964b817@mail.gmail.com> Message-ID: <799406d60904151831h5c98743cnf9f12ae909a962a2@mail.gmail.com> On Wed, Apr 15, 2009 at 14:50, Robert Kern wrote: >> For fortran the user is given a choice between gfortran from gcc-4.2.x >> or gfortran from gcc-4.3.x, the default being from gcc-4.3.x. Its >> installed as gfortran-mp-4.{2,3} and it is explicitly not compiled >> universally. SciPy is also explictly set not to compile universally. > > How? MacPorts has infrastructure to tell a port not to build to universally although come to think of it it doesn't work properly with python ports. But never the less the gfortran is _not_ universal. Cheers Adam From robert.kern at gmail.com Wed Apr 15 21:34:30 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 15 Apr 2009 20:34:30 -0500 Subject: [SciPy-user] Build error on PPC Mac OS X 10.4 In-Reply-To: <799406d60904151831h5c98743cnf9f12ae909a962a2@mail.gmail.com> References: <799406d60904142203n560627dcp1525d978c817818c@mail.gmail.com> <49E57B13.10709@ar.media.kyoto-u.ac.jp> <799406d60904150544u41988da9re03e672729b91f0d@mail.gmail.com> <3d375d730904151250u3b96c775h18f2982f8964b817@mail.gmail.com> <799406d60904151831h5c98743cnf9f12ae909a962a2@mail.gmail.com> Message-ID: <3d375d730904151834qa5d7445u5faa0a9b617c74b8@mail.gmail.com> On Wed, Apr 15, 2009 at 20:31, Adam Mercer wrote: > On Wed, Apr 15, 2009 at 14:50, Robert Kern wrote: > >>> For fortran the user is given a choice between gfortran from gcc-4.2.x >>> or gfortran from gcc-4.3.x, the default being from gcc-4.3.x. Its >>> installed as gfortran-mp-4.{2,3} and it is explicitly not compiled >>> universally. SciPy is also explictly set not to compile universally. >> >> How? > > MacPorts has infrastructure to tell a port not to build to universally > although come to think of it it doesn't work properly with python > ports. But never the less the gfortran is _not_ universal. Sorry, I'm aware of the "universal" flag in MacPorts. My question was intended to ask for the specific details about how you try to tell scipy not to build Universal. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ramercer at gmail.com Wed Apr 15 22:34:23 2009 From: ramercer at gmail.com (Adam Mercer) Date: Wed, 15 Apr 2009 21:34:23 -0500 Subject: [SciPy-user] Build error on PPC Mac OS X 10.4 In-Reply-To: <3d375d730904151834qa5d7445u5faa0a9b617c74b8@mail.gmail.com> References: <799406d60904142203n560627dcp1525d978c817818c@mail.gmail.com> <49E57B13.10709@ar.media.kyoto-u.ac.jp> <799406d60904150544u41988da9re03e672729b91f0d@mail.gmail.com> <3d375d730904151250u3b96c775h18f2982f8964b817@mail.gmail.com> <799406d60904151831h5c98743cnf9f12ae909a962a2@mail.gmail.com> <3d375d730904151834qa5d7445u5faa0a9b617c74b8@mail.gmail.com> Message-ID: <799406d60904151934k7d35579mbbef3384f8520086@mail.gmail.com> On Wed, Apr 15, 2009 at 20:34, Robert Kern wrote: > Sorry, I'm aware of the "universal" flag in MacPorts. My question was > intended to ask for the specific details about how you try to tell > scipy not to build Universal. I was under the impression that if all the dependencies its building against are non-universal, that scipy would build non-universal? Is this not the case? Cheers Adam From robert.kern at gmail.com Wed Apr 15 22:43:31 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 15 Apr 2009 21:43:31 -0500 Subject: [SciPy-user] Build error on PPC Mac OS X 10.4 In-Reply-To: <799406d60904151934k7d35579mbbef3384f8520086@mail.gmail.com> References: <799406d60904142203n560627dcp1525d978c817818c@mail.gmail.com> <49E57B13.10709@ar.media.kyoto-u.ac.jp> <799406d60904150544u41988da9re03e672729b91f0d@mail.gmail.com> <3d375d730904151250u3b96c775h18f2982f8964b817@mail.gmail.com> <799406d60904151831h5c98743cnf9f12ae909a962a2@mail.gmail.com> <3d375d730904151834qa5d7445u5faa0a9b617c74b8@mail.gmail.com> <799406d60904151934k7d35579mbbef3384f8520086@mail.gmail.com> Message-ID: <3d375d730904151943i1cb564a3n27dc3d71ae06db24@mail.gmail.com> On Wed, Apr 15, 2009 at 21:34, Adam Mercer wrote: > On Wed, Apr 15, 2009 at 20:34, Robert Kern wrote: > >> Sorry, I'm aware of the "universal" flag in MacPorts. My question was >> intended to ask for the specific details about how you try to tell >> scipy not to build Universal. > > I was under the impression that if all the dependencies its building > against are non-universal, that scipy would build non-universal? Is > this not the case? We do try to determine if the gfortran is capable of accepting the -arch flags before using them, but it appears this mechanism is not the one adding -arch ppc. It must be coming from somewhere else. Are you using any $LDFLAGS? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From arokem at berkeley.edu Thu Apr 16 00:29:10 2009 From: arokem at berkeley.edu (Ariel Rokem) Date: Wed, 15 Apr 2009 21:29:10 -0700 Subject: [SciPy-user] fourier series In-Reply-To: <22776639.post@talk.nabble.com> References: <22776639.post@talk.nabble.com> Message-ID: <43958ee60904152129x5a7dfc23tf3f409675300e3a5@mail.gmail.com> Hi - in pylab, which is part of matplotlib: >>>from matplotlib import pylab there is a function called psd: >>>pylab.psd which can help you graphically show your frequency decomposition Cheers, Ariel On Sun, Mar 29, 2009 at 10:29 PM, vvinuv wrote: > > Hi > I would like to demonstrate the fourier series graphically. I have a > function in an array and want to decompose it. I used rfft to do that. But > I > am not able to interpret the coefficients I am getting. How can I use the > array returns by rfft in the fourier series, ie how can I extract the > coefficients (a_0, a_1, b_1, a_2, b_2 etc.) out of that? > Thanks > Vinu > -- > View this message in context: > http://www.nabble.com/fourier-series-tp22776639p22776639.html > Sent from the Scipy-User mailing list archive at Nabble.com. > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Ariel Rokem Helen Wills Neuroscience Institute University of California, Berkeley http://argentum.ucbso.berkeley.edu/ariel -------------- next part -------------- An HTML attachment was scrubbed... URL: From mattknox.ca at gmail.com Thu Apr 16 22:11:42 2009 From: mattknox.ca at gmail.com (Matt Knox) Date: Fri, 17 Apr 2009 02:11:42 +0000 (UTC) Subject: [SciPy-user] [ANN] scikits.timeseries 0.91.1 Message-ID: We are pleased to announce the release of scikits.timeseries 0.91.1 This release contains some minor new features and bug fixes and is recommended for all users. Home page: http://pytseries.sourceforge.net/ Please see the website for installation requirements and download details. New Features ------------ * Calling the convert method (or the corresponding function) with an explicit func parameter should now be faster when the function accepts an axis parameter. * The functions first_unmasked_val and last_unmasked_val now accept an optional axis parameter (with a default of None). * The display of dates for a series with undefined frequency has been simplified: dates are presented as integers. * date_array function now accepts strings for start_date and end_date parameters. Bug Fixes --------- * max / min methods now work for multi-dimension TimeSeries objects * The method fill_missing_dates (or the corresponding function) now works with structured arrays. * Methods and functions accepting an axis parameter should now return a TimeSeries when possible. Thanks, Matt Knox & Pierre Gerard-Marchant From ckkart at hoc.net Fri Apr 17 05:14:26 2009 From: ckkart at hoc.net (Christian K.) Date: Fri, 17 Apr 2009 09:14:26 +0000 (UTC) Subject: [SciPy-user] problems with splrep,splev Message-ID: Hi, I am having problems with the spline functions from scipy.interpolate. Reading the doc strings I expected the following to work: from scipy.interpolate import splev,splrep import numpy as N x = N.linspace(-4,10,5) y = x**2 t,c,l = splrep(x,y,xb=0,xe=4) Traceback (most recent call last): File "C:\pythonxy\python\lib\site-packages\scipy\interpolate\fitpack.py", line 406, in splrep n,c,fp,ier = dfitpack.curfit(task, x, y, w, t, wrk, iwrk, xb, xe, k, s) error: (xb<=x[0]) failed for 1st keyword xb Second, I want to fit several curves using the same knots, thus I tried the following: y2 = x**2-3*x+4 t2,c2,l2 = splrep(x,y2,t=t) Traceback (most recent call last): File "C:\pythonxy\python\lib\site-packages\scipy\interpolate\fitpack.py", line 418, in splrep raise _iermess[ier][1],_iermess[ier][0] ValueError: Error on input data Are those bugs or am I misunderstanding something? Eventually I would like to morph one spline into another, therefor I want to stick with the knots. If anyone has experiences with splines and morphing I would be very interested to hear about it. Regards, Christian From warren.weckesser at gmail.com Fri Apr 17 11:39:54 2009 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Fri, 17 Apr 2009 10:39:54 -0500 Subject: [SciPy-user] problems with splrep,splev In-Reply-To: References: Message-ID: <114880320904170839q26855a1doa3fce5423901f8c1@mail.gmail.com> Hi Christian, About the first error... On Fri, Apr 17, 2009 at 4:14 AM, Christian K. wrote: > > Hi, > > I am having problems with the spline functions from scipy.interpolate. > Reading > the doc strings I expected the following to work: > > from scipy.interpolate import splev,splrep > import numpy as N > x = N.linspace(-4,10,5) > y = x**2 > t,c,l = splrep(x,y,xb=0,xe=4) > > Traceback (most recent call last): > File "C:\pythonxy\python\lib\site-packages\scipy\interpolate\fitpack.py", > line 406, in splrep > n,c,fp,ier = dfitpack.curfit(task, x, y, w, t, wrk, iwrk, xb, xe, k, s) > error: (xb<=x[0]) failed for 1st keyword xb > The error message is telling you what the problem is. From the docstring: Given the set of data points (x[i], y[i]) determine a smooth spline approximation of degree k on the interval xb <= x <= xe. You have values in your x array outside the interval [xb,xe]. Warren -------------- next part -------------- An HTML attachment was scrubbed... URL: From janislaw at o2.pl Fri Apr 17 11:29:29 2009 From: janislaw at o2.pl (Jan Wicijowski) Date: Fri, 17 Apr 2009 17:29:29 +0200 Subject: [SciPy-user] fsolve with sparse matrices Message-ID: Hi all! I would like to ask you, if anyone was ever confronted with solving nonlinear system with scipy.optimize.fsolve, with huge no. of equations, say 10000. The usual approach fails, because the size of jacobian matrix is proportional to the square of equations number. The paging kills my OS. In my problem, which is solving a nonlinear electrical circuit, the matrix is mostly zeros, which has immediately led me to trying sparse matrices. The first obstacle was numpy.atleast_1d contained in scipy.optimize.minpack.check_func - instead of preserving sparse matrix as is, it has a suprising behaviour of boxing the matrix in a list. This may be a bug, but I may post it elsewhere. Injecting custom function instead of scipy.optimize.minpack.check_func didn't work as well with fsolve. This wasn't surprising, as I guessed, that FORTRAN hybrj won't be able to deal with interfacing with scipy matrices. So, am I left with rewriting the original FORTRAN hybrj source to python, or is there somebody, who dealt with such problem? Regards, Jan Wicijowski From imrehg at gmail.com Fri Apr 17 11:53:58 2009 From: imrehg at gmail.com (Gergely Imreh) Date: Fri, 17 Apr 2009 23:53:58 +0800 Subject: [SciPy-user] fsolve with sparse matrices In-Reply-To: References: Message-ID: <371b60070904170853o17a2a4fcm69af0e8155d82909@mail.gmail.com> 2009/4/17 Jan Wicijowski : > Hi all! > > I would like to ask you, if anyone was ever confronted with solving > nonlinear system with scipy.optimize.fsolve, with huge no. of equations, > say 10000. > > The usual approach fails, because the size of jacobian matrix is > proportional to the square of equations number. The paging kills my OS. > > In my problem, which is solving a nonlinear electrical circuit, the matrix > is mostly zeros, which has immediately led me to trying sparse matrices. > > The first obstacle was numpy.atleast_1d contained in > scipy.optimize.minpack.check_func - instead of preserving sparse matrix as > is, it has a suprising behaviour of boxing the matrix in a list. This may > be a bug, but I may post it elsewhere. > > Injecting custom function instead of scipy.optimize.minpack.check_func > didn't work as well with fsolve. This wasn't surprising, as I guessed, > that FORTRAN hybrj won't be able to deal with interfacing with scipy > matrices. > > So, am I left with rewriting the original FORTRAN hybrj source to python, > or is there somebody, who dealt with such problem? > > Regards, > Jan Wicijowski Hi, I was working on a similar problem (large sparse matrix, solving optical Bloch-equations), and been planning to check out interfacing with Fortran. The idea was, that I know the NAG library [1] works very well for these kinds of calculations, just have to get the interface between the sparse matrices and Fortran working. I checked it now, and they have some documents showing how to use the NAG with F2PY [2]. Though they only use numpy, so no mention of sparse matrices, but NAG does sparse. This supposed to be just a heads up, need to check it out. Though in my experience even the NAG people can be contacted, and they helped to resolve issues (or extend functionality) before. Cheers, Greg [1] http://www.nag.co.uk/ [2] http://www.nag.co.uk/doc/techrep/index.asp#np3665 From janislaw at o2.pl Fri Apr 17 09:24:21 2009 From: janislaw at o2.pl (Jan Wicijowski) Date: Fri, 17 Apr 2009 15:24:21 +0200 Subject: [SciPy-user] fsolve with sparse matrices Message-ID: Hi all! I would like to ask you, if anyone was ever confronted with solving nonlinear system with scipy.optimize.fsolve, with huge no. of equations, say 10000. The usual approach fails, because the size of jacobian matrix is proportional to the square of equations number. The paging kills my OS. In my problem, which is solving a nonlinear electrical circuit, the matrix is mostly zeros, which has immediately led me to trying sparse matrices. The first obstacle was numpy.atleast_1d contained in scipy.optimize.minpack.check_func - instead of preserving sparse matrix as is, it has a suprising behaviour of boxing the matrix in a list. This may be a bug, but I may post it elsewhere. Injecting custom function instead of scipy.optimize.minpack.check_func didn't work as well with fsolve. This wasn't surprising, as I guessed, that FORTRAN hybrj won't be able to deal with interfacing with scipy matrices. So, am I left with rewriting the original FORTRAN hybrj source to python, or is there somebody, who dealt with such problem? Regards, Jan Wicijowski From pav at iki.fi Fri Apr 17 12:28:14 2009 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 17 Apr 2009 16:28:14 +0000 (UTC) Subject: [SciPy-user] fsolve with sparse matrices References: Message-ID: Fri, 17 Apr 2009 17:29:29 +0200, Jan Wicijowski kirjoitti: [clip] > I would like to ask you, if anyone was ever confronted with solving > nonlinear system with scipy.optimize.fsolve, with huge no. of equations, > say 10000. Sure, but as you noticed, fsolve won't cut it, as it assumes dense matrices. [clip: fsolve & sparse jacobian] > Injecting custom function instead of scipy.optimize.minpack.check_func > didn't work as well with fsolve. This wasn't surprising, as I guessed, > that FORTRAN hybrj won't be able to deal with interfacing with scipy > matrices. > > So, am I left with rewriting the original FORTRAN hybrj source to > python, or is there somebody, who dealt with such problem? Translating hybrj directly to sparse matrices is a bit of work; it's a trust-region Newton method, so it doesn't simply invert the Jacobian, but solves a restricted linear programming problem involving it. (I think some of the tricks it does work well only with sparse matrices.) In principle, writing something like this with sparse matrices should nevertheless be possible with the tools in Scipy (though Scipy does not have a sparse QR decomposition). If you write a sparse trust-region Newton algorithm in Python, I'm sure there's a place in Scipy for it :) *** The easier way would be just to implement a Newton method combined with line search: x = x0 maxiter = 100 abs_tolerance = 1e-8 for k in xrange(maxiter): J = jacobian(x) r = residual(x) d = -sparse.linalg.spsolve(J, r) # or, iterative r_norm = norm(r) # check convergence if r_norm < abs_tolerance: break # naive line search, could consider using # scipy.optimize.line_search instead for alpha in [1, 0.5, 0.1, 0.01]: x2 = x + alpha*d r2 = residual(x2) if norm(r2) < r_norm: break x = x2 else: raise RuntimeError("Didn't converge") It's very naive, but if your problem is easy enough, it works. -- Pauli Virtanen From keflavich at gmail.com Fri Apr 17 12:38:46 2009 From: keflavich at gmail.com (Keflavich) Date: Fri, 17 Apr 2009 09:38:46 -0700 (PDT) Subject: [SciPy-user] 64 bit on Mac? Message-ID: <60cc3bb5-ab28-42e6-874c-ef49dd2bf015@d2g2000pra.googlegroups.com> Hi folks, I'd really like to work with some of the large images I have; I work on a galactic plane survey that has lots of images with ~10^8 floating point pixels. I'm also using a mac. Unfortunately I can't get a 64-bit version of python compiled and google has been unhelpful in resolving the problem. Is there a workaround to get 64 bit numpy working without 64 bit python installed? I doubt that, but if anyone has tips I'd appreciate them. On a related note, I also use a linux machine with 64 bit python installed, but it crashes with large arrays giving a segfault where my mac spits out the "ValueError: dimensions too large." error. Any ideas there? Thanks, Adam From ramercer at gmail.com Fri Apr 17 12:51:39 2009 From: ramercer at gmail.com (Adam Mercer) Date: Fri, 17 Apr 2009 11:51:39 -0500 Subject: [SciPy-user] Build error on PPC Mac OS X 10.4 In-Reply-To: <3d375d730904151943i1cb564a3n27dc3d71ae06db24@mail.gmail.com> References: <799406d60904142203n560627dcp1525d978c817818c@mail.gmail.com> <49E57B13.10709@ar.media.kyoto-u.ac.jp> <799406d60904150544u41988da9re03e672729b91f0d@mail.gmail.com> <3d375d730904151250u3b96c775h18f2982f8964b817@mail.gmail.com> <799406d60904151831h5c98743cnf9f12ae909a962a2@mail.gmail.com> <3d375d730904151834qa5d7445u5faa0a9b617c74b8@mail.gmail.com> <799406d60904151934k7d35579mbbef3384f8520086@mail.gmail.com> <3d375d730904151943i1cb564a3n27dc3d71ae06db24@mail.gmail.com> Message-ID: <799406d60904170951i1e4e6a4fua0e60c25535ae6d5@mail.gmail.com> On Wed, Apr 15, 2009 at 21:43, Robert Kern wrote: > We do try to determine if the gfortran is capable of accepting the > -arch flags before using them, but it appears this mechanism is not > the one adding -arch ppc. It must be coming from somewhere else. Are > you using any $LDFLAGS? No, the only extra environment variables being set are: DEBUG: Environment: MACOSX_DEPLOYMENT_TARGET='10.4' CCFLAGS='-I/opt/local/include -L/opt/local/lib' Cheers Adam From stefan at sun.ac.za Fri Apr 17 12:54:57 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 17 Apr 2009 18:54:57 +0200 Subject: [SciPy-user] 64 bit on Mac? In-Reply-To: <60cc3bb5-ab28-42e6-874c-ef49dd2bf015@d2g2000pra.googlegroups.com> References: <60cc3bb5-ab28-42e6-874c-ef49dd2bf015@d2g2000pra.googlegroups.com> Message-ID: <9457e7c80904170954k63633e58w39179a5d089fead1@mail.gmail.com> Hi Adam 2009/4/17 Keflavich : > Hi folks, I'd really like to work with some of the large images I > have; I work on a galactic plane survey that has lots of images with > ~10^8 floating point pixels. ?I'm also using a mac. ?Unfortunately I > can't get a 64-bit version of python compiled and google has been > unhelpful in resolving the problem. ?Is there a workaround to get 64 > bit numpy working without 64 bit python installed? ?I doubt that, but > if anyone has tips I'd appreciate them. I think SAGE has a 64-bit version available. Try it with sage -python or sage-python Cheers St?fan From michael.abshoff at googlemail.com Fri Apr 17 15:27:17 2009 From: michael.abshoff at googlemail.com (Michael Abshoff) Date: Fri, 17 Apr 2009 12:27:17 -0700 Subject: [SciPy-user] 64 bit on Mac? In-Reply-To: <9457e7c80904170954k63633e58w39179a5d089fead1@mail.gmail.com> References: <60cc3bb5-ab28-42e6-874c-ef49dd2bf015@d2g2000pra.googlegroups.com> <9457e7c80904170954k63633e58w39179a5d089fead1@mail.gmail.com> Message-ID: <49E8D815.5020604@gmail.com> St?fan van der Walt wrote: > Hi Adam > > 2009/4/17 Keflavich : >> Hi folks, I'd really like to work with some of the large images I >> have; I work on a galactic plane survey that has lots of images with >> ~10^8 floating point pixels. I'm also using a mac. Unfortunately I >> can't get a 64-bit version of python compiled and google has been >> unhelpful in resolving the problem. Is there a workaround to get 64 >> bit numpy working without 64 bit python installed? I doubt that, but >> if anyone has tips I'd appreciate them. > > I think SAGE has a 64-bit version available. Try it with You can build such a release, but it isn't available from the default tarball yet. The goal for Sage 4.0 out in a month is to add official support for it. But we haven't updated to numpy 1.3 and scipy 0.7 yet, but you can expect that to happen in the next couple weeks. Maybe scipy 0.7.1 will be ouy by then. > sage -python > > or > > sage-python > > Cheers > St?fan Cheers, Michael > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From robert.kern at gmail.com Fri Apr 17 17:43:02 2009 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 17 Apr 2009 16:43:02 -0500 Subject: [SciPy-user] Build error on PPC Mac OS X 10.4 In-Reply-To: <799406d60904170951i1e4e6a4fua0e60c25535ae6d5@mail.gmail.com> References: <799406d60904142203n560627dcp1525d978c817818c@mail.gmail.com> <49E57B13.10709@ar.media.kyoto-u.ac.jp> <799406d60904150544u41988da9re03e672729b91f0d@mail.gmail.com> <3d375d730904151250u3b96c775h18f2982f8964b817@mail.gmail.com> <799406d60904151831h5c98743cnf9f12ae909a962a2@mail.gmail.com> <3d375d730904151834qa5d7445u5faa0a9b617c74b8@mail.gmail.com> <799406d60904151934k7d35579mbbef3384f8520086@mail.gmail.com> <3d375d730904151943i1cb564a3n27dc3d71ae06db24@mail.gmail.com> <799406d60904170951i1e4e6a4fua0e60c25535ae6d5@mail.gmail.com> Message-ID: <3d375d730904171443m4d2b3e3dm73fbc54128bfd202@mail.gmail.com> On Fri, Apr 17, 2009 at 11:51, Adam Mercer wrote: > On Wed, Apr 15, 2009 at 21:43, Robert Kern wrote: > >> We do try to determine if the gfortran is capable of accepting the >> -arch flags before using them, but it appears this mechanism is not >> the one adding -arch ppc. It must be coming from somewhere else. Are >> you using any $LDFLAGS? > > No, the only extra environment variables being set are: > > DEBUG: Environment: MACOSX_DEPLOYMENT_TARGET='10.4' > CCFLAGS='-I/opt/local/include -L/opt/local/lib' I'm afraid that I don't know the source of "-arch ppc", then. You may want to grep through your installed numpy for "-arch"; it is possible that it has been modified. You don't happen to have any site.cfg or ~/.numpy-site.cfg files laying about, do you? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david.huard at gmail.com Fri Apr 17 22:11:22 2009 From: david.huard at gmail.com (David Huard) Date: Fri, 17 Apr 2009 22:11:22 -0400 Subject: [SciPy-user] fsolve with sparse matrices In-Reply-To: References: Message-ID: <91cf711d0904171911q5df95b65m566c07c0b757420@mail.gmail.com> Have you looked at pyamg ? It has a number of sparse matrix solvers based on multigrid. David On Fri, Apr 17, 2009 at 12:28 PM, Pauli Virtanen wrote: > Fri, 17 Apr 2009 17:29:29 +0200, Jan Wicijowski kirjoitti: > [clip] > > I would like to ask you, if anyone was ever confronted with solving > > nonlinear system with scipy.optimize.fsolve, with huge no. of equations, > > say 10000. > > Sure, but as you noticed, fsolve won't cut it, as it assumes dense > matrices. > > [clip: fsolve & sparse jacobian] > > Injecting custom function instead of scipy.optimize.minpack.check_func > > didn't work as well with fsolve. This wasn't surprising, as I guessed, > > that FORTRAN hybrj won't be able to deal with interfacing with scipy > > matrices. > > > > So, am I left with rewriting the original FORTRAN hybrj source to > > python, or is there somebody, who dealt with such problem? > > Translating hybrj directly to sparse matrices is a bit of work; it's a > trust-region Newton method, so it doesn't simply invert the Jacobian, but > solves a restricted linear programming problem involving it. (I think > some of the tricks it does work well only with sparse matrices.) In > principle, writing something like this with sparse matrices should > nevertheless be possible with the tools in Scipy (though Scipy does not > have a sparse QR decomposition). > > If you write a sparse trust-region Newton algorithm in Python, I'm sure > there's a place in Scipy for it :) > > *** > > The easier way would be just to implement a Newton method combined with > line search: > > x = x0 > maxiter = 100 > abs_tolerance = 1e-8 > for k in xrange(maxiter): > J = jacobian(x) > r = residual(x) > d = -sparse.linalg.spsolve(J, r) # or, iterative > > r_norm = norm(r) > > # check convergence > if r_norm < abs_tolerance: > break > > # naive line search, could consider using > # scipy.optimize.line_search instead > for alpha in [1, 0.5, 0.1, 0.01]: > x2 = x + alpha*d > r2 = residual(x2) > if norm(r2) < r_norm: > break > x = x2 > else: > raise RuntimeError("Didn't converge") > > It's very naive, but if your problem is easy enough, it works. > > -- > Pauli Virtanen > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Sat Apr 18 03:20:29 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 18 Apr 2009 16:20:29 +0900 Subject: [SciPy-user] Build error on PPC Mac OS X 10.4 In-Reply-To: <3d375d730904171443m4d2b3e3dm73fbc54128bfd202@mail.gmail.com> References: <799406d60904142203n560627dcp1525d978c817818c@mail.gmail.com> <49E57B13.10709@ar.media.kyoto-u.ac.jp> <799406d60904150544u41988da9re03e672729b91f0d@mail.gmail.com> <3d375d730904151250u3b96c775h18f2982f8964b817@mail.gmail.com> <799406d60904151831h5c98743cnf9f12ae909a962a2@mail.gmail.com> <3d375d730904151834qa5d7445u5faa0a9b617c74b8@mail.gmail.com> <799406d60904151934k7d35579mbbef3384f8520086@mail.gmail.com> <3d375d730904151943i1cb564a3n27dc3d71ae06db24@mail.gmail.com> <799406d60904170951i1e4e6a4fua0e60c25535ae6d5@mail.gmail.com> <3d375d730904171443m4d2b3e3dm73fbc54128bfd202@mail.gmail.com> Message-ID: <49E97F3D.7010009@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > On Fri, Apr 17, 2009 at 11:51, Adam Mercer wrote: > >> On Wed, Apr 15, 2009 at 21:43, Robert Kern wrote: >> >> >>> We do try to determine if the gfortran is capable of accepting the >>> -arch flags before using them, but it appears this mechanism is not >>> the one adding -arch ppc. It must be coming from somewhere else. Are >>> you using any $LDFLAGS? >>> >> No, the only extra environment variables being set are: >> >> DEBUG: Environment: MACOSX_DEPLOYMENT_TARGET='10.4' >> CCFLAGS='-I/opt/local/include -L/opt/local/lib' >> > > I'm afraid that I don't know the source of "-arch ppc", then. You may > want to grep through your installed numpy for "-arch"; it is possible > that it has been modified. This may be caused by my recent changes in numpy.distutils (r6793 and r6801). The logic to detect universal gfortran was broken for some fortran compilers for quite some time - that's why our 0.7.0 binary is broken for mac os x. http://projects.scipy.org/numpy/ticket/1067 In theory, it should not add universal flag if gfortran does not handle it, but I did not test on fink or macport, so it may be broken. cheers, David From janislaw at o2.pl Sat Apr 18 08:59:47 2009 From: janislaw at o2.pl (Jan Wicijowski) Date: Sat, 18 Apr 2009 14:59:47 +0200 Subject: [SciPy-user] fsolve with sparse matrices In-Reply-To: <91cf711d0904171911q5df95b65m566c07c0b757420@mail.gmail.com> References: <91cf711d0904171911q5df95b65m566c07c0b757420@mail.gmail.com> Message-ID: Dnia 18-04-2009 o 04:11:22 David Huard napisa?(a): > Have you looked at pyamg ? It has a number of sparse matrix solvers > based on > multigrid. > > David Just have taken a quick peek. Scipy also has got them, so when I find the proper method in which I _could_ use sparse matrix solver I may consider it. Thanks Jan Wicijowski From ckkart at hoc.net Sat Apr 18 09:31:23 2009 From: ckkart at hoc.net (Christian K.) Date: Sat, 18 Apr 2009 15:31:23 +0200 Subject: [SciPy-user] problems with splrep,splev In-Reply-To: <114880320904170839q26855a1doa3fce5423901f8c1@mail.gmail.com> References: <114880320904170839q26855a1doa3fce5423901f8c1@mail.gmail.com> Message-ID: Hi Warren, Warren Weckesser schrieb: > About the first error... > > On Fri, Apr 17, 2009 at 4:14 AM, Christian K. > wrote: > > > Hi, > > I am having problems with the spline functions from > scipy.interpolate. Reading > the doc strings I expected the following to work: > > from scipy.interpolate import splev,splrep > import numpy as N > x = N.linspace(-4,10,5) > y = x**2 > t,c,l = splrep(x,y,xb=0,xe=4) > > Traceback (most recent call last): > File > "C:\pythonxy\python\lib\site-packages\scipy\interpolate\fitpack.py", > line 406, in splrep > n,c,fp,ier = dfitpack.curfit(task, x, y, w, t, wrk, iwrk, xb, xe, > k, s) > error: (xb<=x[0]) failed for 1st keyword xb > > > The error message is telling you what the problem is. From the docstring: > > Given the set of data points (x[i], y[i]) determine a smooth spline > approximation of degree k on the interval xb <= x <= xe. > > You have values in your x array outside the interval [xb,xe]. Right, but then I don't get the meaning of xb and xe at all. What sense does it make to choose a fit interval larger than the input data? IMHO x[0] < xb < xe < x[-1] should hold but obviously the docs tell the opposite. Christian From janislaw at o2.pl Sat Apr 18 09:45:13 2009 From: janislaw at o2.pl (Jan Wicijowski) Date: Sat, 18 Apr 2009 15:45:13 +0200 Subject: [SciPy-user] fsolve with sparse matrices References: Message-ID: Dnia 17-04-2009 o 18:28:14 Pauli Virtanen napisa?(a): > If you write a sparse trust-region Newton algorithm in Python, I'm sure > there's a place in Scipy for it :) In my work I immediately picked ready solution of scipy.optimize.fsolve, and didn't look at simpler methods. Fortran hybrj algorithm works blazingly fast for my problem, and I expect simpler methods to converge much slower. Although it may be a reasonable trade-off with sparse matrices. > The easier way would be just to implement a Newton method combined with > line search: > > x = x0 > maxiter = 100 > abs_tolerance = 1e-8 > for k in xrange(maxiter): > J = jacobian(x) > r = residual(x) > d = -sparse.linalg.spsolve(J, r) # or, iterative > > r_norm = norm(r) > > # check convergence > if r_norm < abs_tolerance: > break > > # naive line search, could consider using > # scipy.optimize.line_search instead > for alpha in [1, 0.5, 0.1, 0.01]: > x2 = x + alpha*d > r2 = residual(x2) > if norm(r2) < r_norm: > break > x = x2 > else: > raise RuntimeError("Didn't converge") > > It's very naive, but if your problem is easy enough, it works. Cool, it looks nice and self-explanatory. I'll try it ASAP. If the results for my problem won't satisfy me, expect to see hybrj algorithm ported to python in two weeks or so ;) Thanks for your help - you've pointed out many important facts for my work. Regards, Jan Wicijowski -- Jan Wicijowski a29jaGFtIEp1bGnqIEcu From ramercer at gmail.com Sat Apr 18 11:07:24 2009 From: ramercer at gmail.com (Adam Mercer) Date: Sat, 18 Apr 2009 10:07:24 -0500 Subject: [SciPy-user] Build error on PPC Mac OS X 10.4 In-Reply-To: <3d375d730904171443m4d2b3e3dm73fbc54128bfd202@mail.gmail.com> References: <799406d60904142203n560627dcp1525d978c817818c@mail.gmail.com> <49E57B13.10709@ar.media.kyoto-u.ac.jp> <799406d60904150544u41988da9re03e672729b91f0d@mail.gmail.com> <3d375d730904151250u3b96c775h18f2982f8964b817@mail.gmail.com> <799406d60904151831h5c98743cnf9f12ae909a962a2@mail.gmail.com> <3d375d730904151834qa5d7445u5faa0a9b617c74b8@mail.gmail.com> <799406d60904151934k7d35579mbbef3384f8520086@mail.gmail.com> <3d375d730904151943i1cb564a3n27dc3d71ae06db24@mail.gmail.com> <799406d60904170951i1e4e6a4fua0e60c25535ae6d5@mail.gmail.com> <3d375d730904171443m4d2b3e3dm73fbc54128bfd202@mail.gmail.com> Message-ID: <799406d60904180807g679b9528r6c2cbe69ee8488b8@mail.gmail.com> On Fri, Apr 17, 2009 at 16:43, Robert Kern wrote: > I'm afraid that I don't know the source of "-arch ppc", then. You may > want to grep through your installed numpy for "-arch"; it is possible > that it has been modified. You don't happen to have any site.cfg or > ~/.numpy-site.cfg files laying about, do you? No site.cfg or .numpy-site.cfg are used. The only patch that is applied to numpy is attached, it ensures that MacPorts python is used to run f2py. Cheers Adam -------------- next part -------------- A non-text attachment was scrubbed... Name: patch-f2py_setup.py.diff Type: application/octet-stream Size: 597 bytes Desc: not available URL: From dyamins at gmail.com Sat Apr 18 19:44:59 2009 From: dyamins at gmail.com (Dan Yamins) Date: Sat, 18 Apr 2009 19:44:59 -0400 Subject: [SciPy-user] building 64bit scipy Message-ID: <15e4667e0904181644q255d0038t69012c10722c17a7@mail.gmail.com> Hi, I'm trying to build scipy 64bit binaries. I'm on OS10.5.6, and using python2.6 built as 64bit universal from macports. When I run: python2.6 setup.py install the build doesn't fail along the way. However, when I then go into python and try something, e.g. "from scipy import linalg", I get the error: Python 2.6.2 (r262:71600, Apr 18 2009, 15:53:12) [GCC 4.0.1 (Apple Inc. build 5490)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from scipy import linalg Traceback (most recent call last): File "", line 1, in File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/__init__.py", line 8, in from basic import * File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/basic.py", line 17, in from lapack import get_lapack_funcs File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/lapack.py", line 17, in from scipy.linalg import flapack ImportError: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/flapack.so, 2): no suitable image found. Did find: /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/flapack.so: no matching architecture in universal wrapper Evidently flapack.so has not been build as a 64bit. Why is this, and how do I solve it? Thanks, Dan -----Below are some more details from my own investigation of the problem, for those interested in helping: If I run file /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/flapack.so this yields the output: /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/flapack.so: Mach-O universal binary with 2 architectures /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/flapack.so (for architecture ppc7400): Mach-O bundle ppc /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/flapack.so (for architecture i386): Mach-O bundle i386 So obviously it really isn't built right. Now, I think I see when this is happening. If I look back at the output during the building of scipy, I see commands like: usr/local/bin/gfortran -Wall -arch ppc -arch i686 -Wall -undefined dynamic_lookup -bundle build/temp.macosx-10.5-universal-2.6/build/src.macosx-10.5-universal- 2.6/scipy/linalg/_flinalgmodule.o build/temp.macosx-10.5-universal-2.6/build/src.macosx-10.5-universal-2.6/fortranobject.o build/temp.macosx-10.5-universal-2.6/scipy/linalg/src/det.o build/temp.macosx-10.5-universal-2.6/scipy/linalg/src/lu.o -Lbuild/temp.macosx-10.5-universal-2.6 -lgfortran -o build/lib.macosx-10.5-universal-2.6/scipy/linalg/_flinalg.so -Wl,-framework -Wl,Accelerate e.g. the fortran compiler is not being given the flag -arch x86_64 (though reading the output shows that the C compiler IS.) Morevoer, if I do: python2.6 setup.py config_fc --help-fcompiler then the output is: archiver = ['/usr/local/bin/gfortran', '-cr'] compile_switch = '-c' compiler_f77 = ['/usr/local/bin/gfortran', '-Wall', '-ffixed-form', '- fno-second-underscore', '-arch', 'ppc', '-arch', 'i686', ' -fPIC', '-O3', '-funroll-loops'] compiler_f90 = ['/usr/local/bin/gfortran', '-Wall', '-fno-second- underscore', '-arch', 'ppc', '-arch', 'i686', '-fPIC', '- O3', '-funroll-loops'] compiler_fix = ['/usr/local/bin/gfortran', '-Wall', '-ffixed-form', '- fno-second-underscore', '-Wall', '-fno-second-underscore', '-arch', 'ppc', '-arch', 'i686', '-fPIC', '-O3', '-funroll -loops'] So I have three questions: A) is this lack of proper flag to the fortran compiler the reason my flapack is not being build properly? B) If so, how do I change the compiler options to the fortran compiler when I run setup.py? C) If not, what is the reason, and what do I do about it? Thanks for reading this long thing , Dan -------------- next part -------------- An HTML attachment was scrubbed... URL: From joebarfett at yahoo.ca Sun Apr 19 10:56:57 2009 From: joebarfett at yahoo.ca (Joe Barfett) Date: Sun, 19 Apr 2009 07:56:57 -0700 (PDT) Subject: [SciPy-user] error with interpolation import? Message-ID: <9445.51160.qm@web59409.mail.ac4.yahoo.com> Hello,I'm a MATLAB user that's recently switched to Python and Scipy. I'm trying to use scipy with macPython to fit a 1D spline to some data.?I'm getting an error message...The command:?from scipy.signal import cspline1dReturns the error... Traceback (most recent call last):??File "/Users/joe/", line 1, in ??File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/signal/__init__.py", line 9, in ?? ?from bsplines import *??File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/signal/bsplines.py", line 3, in ?? ?import scipy.special??File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/special/__init__.py", line 8, in ?? ?from basic import *??File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/special/basic.py", line 8, in ?? ?from _cephes import *ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/special/_cephes.so, 2): Library not loaded: /usr/local/lib/libgfortran.2.dylib??Referenced from: /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/special/_cephes.so??Reason: image not found I've reinstalled scipy a couple times. If anyone knows what's going on I'd appreciate some help. I know scipy can do splines.?Thanks!joe __________________________________________________________________ Make your browsing faster, safer, and easier with the new Internet Explorer? 8. Optimized for Yahoo! Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ckkart at hoc.net Sun Apr 19 14:56:11 2009 From: ckkart at hoc.net (Christian K.) Date: Sun, 19 Apr 2009 20:56:11 +0200 Subject: [SciPy-user] error with interpolation import? In-Reply-To: <9445.51160.qm@web59409.mail.ac4.yahoo.com> References: <9445.51160.qm@web59409.mail.ac4.yahoo.com> Message-ID: Joe Barfett schrieb: > Hello, > I'm a MATLAB user that's recently switched to Python and Scipy. I'm > trying to use scipy with macPython to fit a 1D spline to some data. > I'm getting an error message... > The command: from scipy.signal import cspline1d > Returns the error... > > Traceback (most recent call last): > File "/Users/joe/", line 1, in > File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/signal/__init__.py", > line 9, in > from bsplines import * > File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/signal/bsplines.py", > line 3, in > import scipy.special > File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/special/__init__.py", > line 8, in > from basic import * > File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/special/basic.py", > line 8, in > from _cephes import * > ImportError: > dlopen(/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/special/_cephes.so, > 2): Library not loaded: /usr/local/lib/libgfortran.2.dylib > Referenced from: > /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/special/_cephes.so > Reason: image not found > > I've reinstalled scipy a couple times. If anyone knows what's going on I can't help you solving that error (I don't see it on linux), but ... > I'd appreciate some help. I know scipy can do splines. ... in the meanwhile you might try splrep and splev from scipy.interpolate Christian From robert.kern at gmail.com Sun Apr 19 15:50:20 2009 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 19 Apr 2009 14:50:20 -0500 Subject: [SciPy-user] error with interpolation import? In-Reply-To: <9445.51160.qm@web59409.mail.ac4.yahoo.com> References: <9445.51160.qm@web59409.mail.ac4.yahoo.com> Message-ID: <3d375d730904191250yd55372ob472f44adfd15fec@mail.gmail.com> On Sun, Apr 19, 2009 at 09:56, Joe Barfett wrote: > Hello, > I'm a MATLAB user that's recently switched to Python and Scipy. I'm trying > to use scipy with macPython to fit a 1D spline to some data. > I'm getting an error message... > The command:?from scipy.signal import cspline1d > Returns the error... > Traceback (most recent call last): > ??File "/Users/joe/", line 1, in > ??File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/signal/__init__.py", > line 9, in > ?? ?from bsplines import * > ??File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/signal/bsplines.py", > line 3, in > ?? ?import scipy.special > ??File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/special/__init__.py", > line 8, in > ?? ?from basic import * > ??File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/special/basic.py", > line 8, in > ?? ?from _cephes import * > ImportError: > dlopen(/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/special/_cephes.so, > 2): Library not loaded: /usr/local/lib/libgfortran.2.dylib > ??Referenced from: > /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/special/_cephes.so > ??Reason: image not found > I've reinstalled scipy a couple times. If anyone knows what's going on I'd > appreciate some help. I know scipy can do splines. How are you installing scipy? It appears that you are installing from a binary installer that was built using gfortran, which you do not have the runtime libraries for. You can probably work around this by installing gfortran from here: http://r.research.att.com/tools -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dyamins at gmail.com Sun Apr 19 16:17:08 2009 From: dyamins at gmail.com (Dan Yamins) Date: Sun, 19 Apr 2009 16:17:08 -0400 Subject: [SciPy-user] Building 64-bit binaries, OS 10.5.6 Message-ID: <15e4667e0904191317w75340a87p8f0e15e68a69431d@mail.gmail.com> Hi, (Sorry if this was posted multiple times...) I'm trying to build scipy 64bit binaries. I'm on OS10.5.6, and using python2.6 built as 64bit universal from macports. When I run: python2.6 setup.py install the build doesn't fail along the way. However, when I then go into python and try something, e.g. "from scipy import linalg", I get the error: Python 2.6.2 (r262:71600, Apr 18 2009, 15:53:12) [GCC 4.0.1 (Apple Inc. build 5490)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from scipy import linalg Traceback (most recent call last): File "", line 1, in File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/__init__.py", line 8, in from basic import * File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/basic.py", line 17, in from lapack import get_lapack_funcs File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/lapack.py", line 17, in from scipy.linalg import flapack ImportError: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/flapack.so, 2): no suitable image found. Did find: /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/flapack.so: no matching architecture in universal wrapper Evidently flapack.so has not been build as a 64bit. Why is this, and how do I solve it? Thanks, Dan -----Below are some more details from my own investigation of the problem, for those interested in helping: If I run file /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/flapack.so this yields the output: /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/flapack.so: Mach-O universal binary with 2 architectures /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/flapack.so (for architecture ppc7400): Mach-O bundle ppc /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/flapack.so (for architecture i386): Mach-O bundle i386 So obviously it really isn't built right. Now, I think I see when this is happening. If I look back at the output during the building of scipy, I see commands like: usr/local/bin/gfortran -Wall -arch ppc -arch i686 -Wall -undefined dynamic_lookup -bundle build/temp.macosx-10.5-universal-2.6/build/src.macosx-10.5-universal- 2.6/scipy/linalg/_flinalgmodule.o build/temp.macosx-10.5-universal-2.6/build/src.macosx-10.5-universal-2.6/fortranobject.o build/temp.macosx-10.5-universal-2.6/scipy/linalg/src/det.o build/temp.macosx-10.5-universal-2.6/scipy/linalg/src/lu.o -Lbuild/temp.macosx-10.5-universal-2.6 -lgfortran -o build/lib.macosx-10.5-universal-2.6/scipy/linalg/_flinalg.so -Wl,-framework -Wl,Accelerate e.g. the fortran compiler is not being given the flag -arch x86_64 (though reading the output shows that the C compiler IS.) Morevoer, if I do: python2.6 setup.py config_fc --help-fcompiler then the output is: archiver = ['/usr/local/bin/gfortran', '-cr'] compile_switch = '-c' compiler_f77 = ['/usr/local/bin/gfortran', '-Wall', '-ffixed-form', '- fno-second-underscore', '-arch', 'ppc', '-arch', 'i686', ' -fPIC', '-O3', '-funroll-loops'] compiler_f90 = ['/usr/local/bin/gfortran', '-Wall', '-fno-second- underscore', '-arch', 'ppc', '-arch', 'i686', '-fPIC', '- O3', '-funroll-loops'] compiler_fix = ['/usr/local/bin/gfortran', '-Wall', '-ffixed-form', '- fno-second-underscore', '-Wall', '-fno-second-underscore', '-arch', 'ppc', '-arch', 'i686', '-fPIC', '-O3', '-funroll -loops'] So I have three questions: A) is this lack of proper flag to the fortran compiler the reason my flapack is not being build properly? B) If so, how do I change the compiler options to the fortran compiler when I run setup.py? C) If not, what is the reason, and what do I do about it? Thanks for reading this long thing , Dan -------------- next part -------------- An HTML attachment was scrubbed... URL: From yosefmel at post.tau.ac.il Mon Apr 20 04:38:14 2009 From: yosefmel at post.tau.ac.il (Yosef Meller) Date: Mon, 20 Apr 2009 11:38:14 +0300 Subject: [SciPy-user] Bilingual implementation of the epsilon-MOEA algorithm Message-ID: <200904201138.14139.yosefmel@post.tau.ac.il> Hi all, I've recently had to do a Matlab implementation of the epsilon-MOEA algorithm for multi-objective optimization. I decided to make this a benchmark case and implemented it using NumPy as well. I thought others may have use for it. The code: http://github.com/yosefm/eps-moea/tree A wiki page on my development impressions is here: http://wiki.github.com/yosefm/eps-moea/account-of-development I hope anyone finds it useful. Yours, Yosef. From aaulagnier at gmail.com Mon Apr 20 08:04:58 2009 From: aaulagnier at gmail.com (Adrien Aulagnier) Date: Mon, 20 Apr 2009 14:04:58 +0200 Subject: [SciPy-user] Optimization fmin_tnc with equality constraint Message-ID: <20f2e9f60904200504x1b2ef230w7192b9092a3cb8af@mail.gmail.com> Hello Sorry to disturb you. My question might be simple: The function to minimize is simply f (w) = w' * Sigma * w with w a vector and Sigma a square matrix The constraints are : 0 <= w <= 1 sum(w) = 1 This last constraint is my problem. Could you give me some clue ? Thanks for your help -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.hirschfeld at gmail.com Mon Apr 20 08:26:14 2009 From: dave.hirschfeld at gmail.com (Dave Hirschfeld) Date: Mon, 20 Apr 2009 12:26:14 +0000 (UTC) Subject: [SciPy-user] =?utf-8?q?Optimization_fmin=5Ftnc_with_equality_cons?= =?utf-8?q?traint?= References: <20f2e9f60904200504x1b2ef230w7192b9092a3cb8af@mail.gmail.com> Message-ID: Adrien Aulagnier gmail.com> writes: > > HelloSorry to disturb you. My question might be simple:The function to minimize is simply f (w) = w' *? Sigma * wwith w a vector and Sigma a square matrixThe constraints are :?0 <= w <= 1 > ?sum(w) = 1This last constraint is my problem.Could you give me some clue ?Thanks for your help > > _______________________________________________ > SciPy-user mailing list > SciPy-user scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > This discussion may help: http://thread.gmane.org/gmane.comp.python.scientific.user/16180/focus=16187 From josef.pktd at gmail.com Mon Apr 20 08:40:44 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 20 Apr 2009 08:40:44 -0400 Subject: [SciPy-user] Optimization fmin_tnc with equality constraint In-Reply-To: References: <20f2e9f60904200504x1b2ef230w7192b9092a3cb8af@mail.gmail.com> Message-ID: <1cd32cbb0904200540n6378c64fm3ee9912c40d129a7@mail.gmail.com> On Mon, Apr 20, 2009 at 8:26 AM, Dave Hirschfeld wrote: > Adrien Aulagnier gmail.com> writes: > > > > > HelloSorry to disturb you. My question might be simple:The function to > minimize is simply f (w) = w' * Sigma * wwith w a vector and Sigma a > square > matrixThe constraints are : 0 <= w <= 1 > > sum(w) = 1This last constraint is my problem.Could you give me some clue > ?Thanks for your help > > > > _______________________________________________ > > SciPy-user mailing list > > SciPy-user scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > This discussion may help: > > http://thread.gmane.org/gmane.comp.python.scientific.user/16180/focus=16187 > > For a constraint on a simplex, I have seen a multinomial logit, or dirichlet (?) reparameterization most often: something like: w_i = exp(wn_i)/(1+sum_i(exp(wn_i)) for i>0 w_i = 1/(1+sum_i(exp(wn_i)) for i=0 wn has length = len(w)-1 Josef -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsalvador.wk at gmail.com Mon Apr 20 09:58:01 2009 From: rsalvador.wk at gmail.com (Ruben Salvador) Date: Mon, 20 Apr 2009 15:58:01 +0200 Subject: [SciPy-user] Numpy/Scipy float precision vs Python list float issue Message-ID: <4fe028e30904200658w44126eah305e06bb0865b1b9@mail.gmail.com> Hi everybody! First of all I should say I am a newbie with Python/Scipy. Have been searching a little bit (google and lists) and haven't found a helpful answer...so I'm posting. (sorry for double posting, since I also just posted this in numpy-discussion...just in case some of you are in both lists....but I didn't figure out which only list should I post this to) I'm using Scipy/Numpy to do image wavelet transforms via the lifting scheme. I grabbed some code implementing the transforms with Python lists (float type). This code works perfectly, but slow for my needs (I'll be doing some genetic algorithms to evolve coefficients of the filters and the forward and inverse transform will be done many times). It's just implemented by looping in the lists and making computations this way. Reconstructed image after doing a forward and inverse transform is perfect, this is, original and reconstructed images difference is 0. With Scipy/Numpy float arrays slicing this code is much faster as you know. But the reconstructed image is not perfect. The image difference maximum and minimum values returns: maximum difference => 3.5527136788e-15 minimum difference => -3.5527136788e-15 Is this behavior expected? Because it seems sooo weird to me. If expected, anyway to override it? I include some test code for you to reproduce. It's part of a transform over a 8x8 2D signal (for simplicity). It's not complete (again for simplicity), but enough to reproduce. It does part of a forward and inverse transform both with lists and arrays, printing the differences (there is commented code showing some plots with the results as I used when transforming real images, but for the purpose, is enough with the return results I think). Code is simple (maybe long but it's always the same). Instead of using the faster array slicing as commented above, I am using here array looping, so that the math code is exactly the same as in the case list. This happens in the next three system/platforms. * System 1 (laptop): --------------------------- 64 bit processor running Kubuntu 8.04 32 bits Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52 Numpy version: 1:1.0.4-6ubuntu3 Scipy version: 0.6.0-8ubuntu1 * System 2 (PC): -------------------------- Windows Xp on 64 bit processor Enthought Python distribution (EPD Py25 v4.1.30101). This is a Python 2.5.2 with Numpy 1.1.1 and Scipy 0.6.0 * System 3 (same PC as 2): -------------------------------------- Debian Lenny 64 bit on 64 bit processor Not sure about versions here, but doesn't mind because behavior is prety much the same in the 3 systems Thanks everybody in advance for the help! Ruben Salvador PhD Student Industrial Electronics Center Universidad Politecnica de Madrid -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: floattest.py Type: text/x-python Size: 7161 bytes Desc: not available URL: From david at ar.media.kyoto-u.ac.jp Mon Apr 20 09:53:37 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 20 Apr 2009 22:53:37 +0900 Subject: [SciPy-user] building 64bit scipy In-Reply-To: <15e4667e0904181644q255d0038t69012c10722c17a7@mail.gmail.com> References: <15e4667e0904181644q255d0038t69012c10722c17a7@mail.gmail.com> Message-ID: <49EC7E61.30408@ar.media.kyoto-u.ac.jp> Hi Dan, Dan Yamins wrote: > Hi, > > I'm trying to build scipy 64bit binaries. I'm on OS10.5.6, and using > python2.6 built as 64bit universal from macports. > > When I run: > python2.6 setup.py install > > the build doesn't fail along the way. > > However, when I then go into python and try something, e.g. "from > scipy import linalg", I get the error That's because the fortran compiler flags do not handle the x86_64 arch. As there is no official 64 bits build for python yet, it is a bit difficult to test. In the meantime, I managed to build a 64 bits version of python 2.6.2, and built numpy and scipy on it. Nothing is needed for numpy, and for scipy, you will want something like LDFLAGS="-arch x86_64 -undefined dynamic_lookup -bundle" FFLAGS="-arch x86_64 -O2 -Wall -fPIC" python setup.py install This will build a 64 bits only binary. If you want to build universal binaries, you will need to add the corresponding -arch flags. You should remove the build directory entirely before starting (you may want to remove the -O2 flag at first for testing purpose, as the build time is significantly shorter without it). There are a couple of failures, and I think you should consider the 64 bits on mac os X as experimental for the time being, cheers, David From aaulagnier at gmail.com Mon Apr 20 10:53:13 2009 From: aaulagnier at gmail.com (Adrien Aulagnier) Date: Mon, 20 Apr 2009 16:53:13 +0200 Subject: [SciPy-user] Optimization fmin_tnc with equality constraint In-Reply-To: References: <20f2e9f60904200504x1b2ef230w7192b9092a3cb8af@mail.gmail.com> Message-ID: <20f2e9f60904200753v33ef50ccq2fef0b694136519f@mail.gmail.com> 2009/4/20 Dave Hirschfeld > Adrien Aulagnier gmail.com> writes: > > > > > HelloSorry to disturb you. My question might be simple:The function to > minimize is simply f (w) = w' * Sigma * wwith w a vector and Sigma a > square > matrixThe constraints are : 0 <= w <= 1 > > sum(w) = 1This last constraint is my problem.Could you give me some clue > ?Thanks for your help > > > > _______________________________________________ > > SciPy-user mailing list > > SciPy-user scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > This discussion may help: > > http://thread.gmane.org/gmane.comp.python.scientific.user/16180/focus=16187 > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > Thank you for the help but my description was not complet: The function to minimize is simply f (w) = w' * Sigma * w with w a vector and Sigma a square matrix The constraints are : 0 <= w <= 1 sum(w) = 1 R' * w = a constant and R a vector of the same size as w Sorry for the missing information. I think that the Aitchison transformation won't work in this case... Do you have an idea to solve this system ? Thanks for your help -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Mon Apr 20 11:10:47 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 21 Apr 2009 00:10:47 +0900 Subject: [SciPy-user] building 64bit scipy In-Reply-To: <49EC7E61.30408@ar.media.kyoto-u.ac.jp> References: <15e4667e0904181644q255d0038t69012c10722c17a7@mail.gmail.com> <49EC7E61.30408@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220904200810u2a09ede3u5b765938ea389e99@mail.gmail.com> On Mon, Apr 20, 2009 at 10:53 PM, David Cournapeau wrote: > Hi Dan, > > Dan Yamins wrote: >> Hi, >> >> I'm trying to build scipy 64bit binaries. ?I'm on OS10.5.6, and using >> python2.6 built as 64bit universal from macports. >> >> When I run: >> ? ? python2.6 setup.py install >> >> the build doesn't fail along the way. >> >> However, when I then go into python and try something, e.g. "from >> scipy import linalg", ?I get the error > > That's because the fortran compiler flags do not handle the x86_64 arch. Actually, if you don't mind build numpy by yourself from the trunk, it should now handle the x86_64 arch. I did not test this thoroughly, but it is working when scipy is built from a 64 bits python, and it looks like it does not break 32 bits python. cheers, David From dyamins at gmail.com Mon Apr 20 11:20:53 2009 From: dyamins at gmail.com (Dan Yamins) Date: Mon, 20 Apr 2009 11:20:53 -0400 Subject: [SciPy-user] building 64bit scipy In-Reply-To: <49EC7E61.30408@ar.media.kyoto-u.ac.jp> References: <15e4667e0904181644q255d0038t69012c10722c17a7@mail.gmail.com> <49EC7E61.30408@ar.media.kyoto-u.ac.jp> Message-ID: <15e4667e0904200820i3dde34cal49818f5936c54c2b@mail.gmail.com> On Mon, Apr 20, 2009 at 9:53 AM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Hi Dan, > > Dan Yamins wrote: > > Hi, > > > > I'm trying to build scipy 64bit binaries. I'm on OS10.5.6, and using > > python2.6 built as 64bit universal from macports. > > > > When I run: > > python2.6 setup.py install > > > > the build doesn't fail along the way. > > > > However, when I then go into python and try something, e.g. "from > > scipy import linalg", I get the error > > That's because the fortran compiler flags do not handle the x86_64 arch. > As there is no official 64 bits build for python yet, it is a bit > difficult to test. In the meantime, I managed to build a 64 bits version > of python 2.6.2, and built numpy and scipy on it. Yes -- this is exactly what I figured, since I could see that during the scipy build, the fortran compiler wasn't getting the -arch x86_64 flag. > Nothing is needed for > numpy, Precisely. I built numpy 1.3.0 first before trying scipy, the build worked perfectly, and passed all the tests. I guess this is either be because only the C compiler is being used (and it is, even in the scipy build, getting the right arch flag), or because for some reason any use of the fortran compiler is being called with the right flags? > and for scipy, you will want something like > > LDFLAGS="-arch x86_64 -undefined dynamic_lookup -bundle" FFLAGS="-arch > x86_64 -O2 -Wall -fPIC" python setup.py install So are you saying that I should 1) find and edit the LDFLAGS and FFLAGS lines of Make file with the options shown above, and 2) then do "python26 setup.py install" at the shell or: just execute the command LDFLAGS="-arch x86_64 -undefined dynamic_lookup -bundle" FFLAGS="-arch x86_64 -O2 -Wall -fPIC" python26 setup.py install at the shell? Sorry for this question and my ignorance. Thanks very much for your help (I really had been struggling with this)! Dan -------------- next part -------------- An HTML attachment was scrubbed... URL: From dyamins at gmail.com Mon Apr 20 11:22:37 2009 From: dyamins at gmail.com (Dan Yamins) Date: Mon, 20 Apr 2009 11:22:37 -0400 Subject: [SciPy-user] building 64bit scipy In-Reply-To: <5b8d13220904200810u2a09ede3u5b765938ea389e99@mail.gmail.com> References: <15e4667e0904181644q255d0038t69012c10722c17a7@mail.gmail.com> <49EC7E61.30408@ar.media.kyoto-u.ac.jp> <5b8d13220904200810u2a09ede3u5b765938ea389e99@mail.gmail.com> Message-ID: <15e4667e0904200822x23d118c1m3deee49d89f4de82@mail.gmail.com> > Actually, if you don't mind build numpy by yourself from the trunk, it > should now handle the x86_64 arch. I did not test this thoroughly, but > it is working when scipy is built from a 64 bits python, and it looks > like it does not break 32 bits python. > > In fact, the numpy 1.3 release seems to handle it fine too .... although I built numpy separately from scipy, so I'm not sure how this relates to what your saying ... Dan -------------- next part -------------- An HTML attachment was scrubbed... URL: From dyamins at gmail.com Mon Apr 20 11:33:39 2009 From: dyamins at gmail.com (Dan Yamins) Date: Mon, 20 Apr 2009 11:33:39 -0400 Subject: [SciPy-user] building 64bit scipy In-Reply-To: <15e4667e0904200820i3dde34cal49818f5936c54c2b@mail.gmail.com> References: <15e4667e0904181644q255d0038t69012c10722c17a7@mail.gmail.com> <49EC7E61.30408@ar.media.kyoto-u.ac.jp> <15e4667e0904200820i3dde34cal49818f5936c54c2b@mail.gmail.com> Message-ID: <15e4667e0904200833y214e4b03p17574a961719b37d@mail.gmail.com> > > So are you saying that I should > 1) find and edit the LDFLAGS and FFLAGS lines of Make file with the > options shown above, and > 2) then do "python26 setup.py install" at the shell > > or: > > just execute the command > > LDFLAGS="-arch x86_64 -undefined dynamic_lookup -bundle" FFLAGS="-arch > x86_64 -O2 -Wall -fPIC" python26 setup.py install > > at the shell? > > Sorry for this question and my ignorance. > I guess you obviously don't mean the second thing (e.g. the one shell command)?? Maybe a better way to ask this is: could you give me a step-by-step description of exactly what I should do? E.g. which files to edit, where, and then what shell commans to run. Mostly until now for installing python packages that have setup.py files for setuptools installations, I've just been using the setuptools, and not running any compiler configuration steps or makes. So I'm not sure where exactly I'm supposed to make the flags edits that you suggested. (E.g. I don't know where to look for the Make file, &c) And then I'm not sure what shell commands to run. Sorry for my ignorance, and thanks for all the help! best, Dan -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Mon Apr 20 11:47:21 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 21 Apr 2009 00:47:21 +0900 Subject: [SciPy-user] building 64bit scipy In-Reply-To: <15e4667e0904200822x23d118c1m3deee49d89f4de82@mail.gmail.com> References: <15e4667e0904181644q255d0038t69012c10722c17a7@mail.gmail.com> <49EC7E61.30408@ar.media.kyoto-u.ac.jp> <5b8d13220904200810u2a09ede3u5b765938ea389e99@mail.gmail.com> <15e4667e0904200822x23d118c1m3deee49d89f4de82@mail.gmail.com> Message-ID: <5b8d13220904200847n775ffa8dj4a7aada478f5185c@mail.gmail.com> On Tue, Apr 21, 2009 at 12:22 AM, Dan Yamins wrote: > > >> >> Actually, if you don't mind build numpy by yourself from the trunk, it >> should now handle the x86_64 arch. I did not test this thoroughly, but >> it is working when scipy is built from a 64 bits python, and it looks >> like it does not break 32 bits python. >> > > In fact,? the numpy 1.3 release seems to handle it fine too .... Sorry, this only makes sense if you know that scipy uses code in numpy to build itself. In particular, fortran compiler flags are taken from numpy code. cheers, David From cournape at gmail.com Mon Apr 20 11:51:12 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 21 Apr 2009 00:51:12 +0900 Subject: [SciPy-user] building 64bit scipy In-Reply-To: <15e4667e0904200833y214e4b03p17574a961719b37d@mail.gmail.com> References: <15e4667e0904181644q255d0038t69012c10722c17a7@mail.gmail.com> <49EC7E61.30408@ar.media.kyoto-u.ac.jp> <15e4667e0904200820i3dde34cal49818f5936c54c2b@mail.gmail.com> <15e4667e0904200833y214e4b03p17574a961719b37d@mail.gmail.com> Message-ID: <5b8d13220904200851y3fa9b1f7v9c3376a0bd09a8a7@mail.gmail.com> On Tue, Apr 21, 2009 at 12:33 AM, Dan Yamins wrote: > I guess you obviously don't mean the second thing (e.g. the one shell > command)?? Actually, I mean the second one :) > > Maybe a better way to ask this is: ? could you give me a step-by-step > description of exactly what I should do? ? E.g. which files to edit, where, > and then what shell commans to run. Just copy and paste the given command in the terminal - or better, install numpy first from svn, so that you don't need any other command. > > Mostly until now for installing python packages that have setup.py files for > setuptools installations, I've just been using the setuptools, and not > running any compiler configuration steps or makes. Most python packages do not use fortran. Numpy/scipy are not conventional packages in that respect, because they use much more compiled code than usually. David From dyamins at gmail.com Mon Apr 20 12:25:28 2009 From: dyamins at gmail.com (Dan Yamins) Date: Mon, 20 Apr 2009 12:25:28 -0400 Subject: [SciPy-user] building 64bit scipy In-Reply-To: <5b8d13220904200851y3fa9b1f7v9c3376a0bd09a8a7@mail.gmail.com> References: <15e4667e0904181644q255d0038t69012c10722c17a7@mail.gmail.com> <49EC7E61.30408@ar.media.kyoto-u.ac.jp> <15e4667e0904200820i3dde34cal49818f5936c54c2b@mail.gmail.com> <15e4667e0904200833y214e4b03p17574a961719b37d@mail.gmail.com> <5b8d13220904200851y3fa9b1f7v9c3376a0bd09a8a7@mail.gmail.com> Message-ID: <15e4667e0904200925y497d5c60x55f8fd14826dfca@mail.gmail.com> > Just copy and paste the given command in the terminal - or better, > install numpy first from svn, so that you don't need any other > command. So I did try to do this. But the command fails if you first put "sudo" in front of it. I needed to run it sudo (otherwise the final stage of copying over the built things to the target directory would fail). Hence, I thought perhaps that this wasn't actually a command and I had misinterpreted it. But what works is that to just run the command you said first, and then run "sudo python26 setup.py install", e.g. >>> LDFLAGS="-arch x86_64 -undefined dynamic_lookup -bundle" FFLAGS="-arch x86_64 -O2 -Wall -fPIC" python26 setup.py install and then >>> sudo python26 setup.py install And now it works fine. Yay! But I have a few questions: a) the syntax of that command that you gave me is new to me. I'm not an expert in make files, and I don't really instand the way that the scipy setup.py module commands interact with the setting of compiler flags. I wouldn't have known how to put together a command like this -- where would I look to learn about this? Is it a make file thing? A scipy thing? I feel dumb asking this, but ... b) Why did it fail with sudo? What would I need to do to make it work with sudo? Sorry this is dumb too, it's prbably a unix thing I should know but again, I don't. c) Why would having built numpy from the svn have avoided the need for this? Is there something different about the way that that numpy retains or exposes information about the compiler flags that were used to build it? Sorry for all these ignorant questions. Dan > > > > > > Mostly until now for installing python packages that have setup.py files > for > > setuptools installations, I've just been using the setuptools, and not > > running any compiler configuration steps or makes. > > Most python packages do not use fortran. Numpy/scipy are not > conventional packages in that respect, because they use much more > compiled code than usually. > > David > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Mon Apr 20 13:08:57 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 21 Apr 2009 02:08:57 +0900 Subject: [SciPy-user] building 64bit scipy In-Reply-To: <15e4667e0904200925y497d5c60x55f8fd14826dfca@mail.gmail.com> References: <15e4667e0904181644q255d0038t69012c10722c17a7@mail.gmail.com> <49EC7E61.30408@ar.media.kyoto-u.ac.jp> <15e4667e0904200820i3dde34cal49818f5936c54c2b@mail.gmail.com> <15e4667e0904200833y214e4b03p17574a961719b37d@mail.gmail.com> <5b8d13220904200851y3fa9b1f7v9c3376a0bd09a8a7@mail.gmail.com> <15e4667e0904200925y497d5c60x55f8fd14826dfca@mail.gmail.com> Message-ID: <5b8d13220904201008g2e9f6a5cwd76cb5aaa19a78d@mail.gmail.com> On Tue, Apr 21, 2009 at 1:25 AM, Dan Yamins wrote: > > >> >> Just copy and paste the given command in the terminal - or better, >> install numpy first from svn, so that you don't need any other >> command. > > So I did try to do this.? But the command fails if you first put "sudo" in > front of it. Yes, sudo affects environment variables. > But I have a few questions: > ?? a) the syntax of that command that you gave me is new to me. It is simple: you just set environment variables which are imported and interpreted during the build process. >?? I'm not an > expert in make files, and I don't really instand the way that the scipy > setup.py? module commands interact with the setting of compiler flags. They interact in a non standard manner. setup.py drives the build through a package called distutils, which is used by the majority of python packages out there (setuptools is a set of extensions on top of distutils). Unfortunately, it does not work the same as the standard configure/make/make install, and on top of it, we (numpy/scipy) use our own extensions which also add some complexity. > I > wouldn't have known how to put together a command like this -- where would I > look to learn about this?? Is it a make file thing? There is no make file in scipy or numpy. > ?? c) Why would having built numpy from the svn have avoided the need for > this?? Is there something different about the way that that numpy retains or > exposes information about the compiler flags that were used to build it? Because distutils is not enough to build numpy and scipy (very few packages in python community needs fortran support, for example, so we needed to add this). So we have our own extensions in numpy - IOW, scipy requires numpy.distutils to build correctly. FOllowing your email, I added the necessary code to support intel 64 bits in numpy.distutils, which would be in turn used by scipy. Unfortunately, there is no easy way to just modify one flag in the build process as is the case with the usual (and much better) autotools. David From mickael.paris at gmail.com Mon Apr 20 13:27:06 2009 From: mickael.paris at gmail.com (Mickael) Date: Mon, 20 Apr 2009 19:27:06 +0200 Subject: [SciPy-user] Convolve an input with a seri of impulses Message-ID: <5df0b3120904201027g25f66692la426bf838be78b0c@mail.gmail.com> Hi, I'm trying to convolve a reference signal, like a step signal, with a sequence of impulses. Impulses have different amplitudes and timing : ---------------------------------------------------------------------------- zeta = 0.0193 wn = 35037.0 wd = wn*math.sqrt(1.0-zeta**2) K = math.exp(((-zeta*pi)/math.sqrt(1.0-zeta**2))) num = array([9.871]) den = array([1,1354,1.228*10**9]) H = signal.lti(num, den) # Impulses : amplitudes A1 = (1.0/(1.0+K)) A2 = (K/(1.0+K)) # Impulses : timing t1 = 0 t2 = (pi/wd) ----------------------------------------------------------------------------- I would like to apply to "H" the result of a step signal convolve with these two pulses but I'don't know to define the vector of the impulses. --->T,yout,xout = signal.lsim(H, entree, t) <=== entree = scipy.signal. convolve(*A_step*, My_Impulses?, mode='full') Thanks, Mickael.**** -------------- next part -------------- An HTML attachment was scrubbed... URL: From dyamins at gmail.com Mon Apr 20 13:28:10 2009 From: dyamins at gmail.com (Dan Yamins) Date: Mon, 20 Apr 2009 13:28:10 -0400 Subject: [SciPy-user] building 64bit scipy In-Reply-To: <5b8d13220904201008g2e9f6a5cwd76cb5aaa19a78d@mail.gmail.com> References: <15e4667e0904181644q255d0038t69012c10722c17a7@mail.gmail.com> <49EC7E61.30408@ar.media.kyoto-u.ac.jp> <15e4667e0904200820i3dde34cal49818f5936c54c2b@mail.gmail.com> <15e4667e0904200833y214e4b03p17574a961719b37d@mail.gmail.com> <5b8d13220904200851y3fa9b1f7v9c3376a0bd09a8a7@mail.gmail.com> <15e4667e0904200925y497d5c60x55f8fd14826dfca@mail.gmail.com> <5b8d13220904201008g2e9f6a5cwd76cb5aaa19a78d@mail.gmail.com> Message-ID: <15e4667e0904201028q726c7b84hc830c185862f32cc@mail.gmail.com> On Mon, Apr 20, 2009 at 1:08 PM, David Cournapeau wrote: > > > > Unfortunately, there is no easy way to just modify one flag in the > build process as is the case with the usual (and much better) > autotools. > > David > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > Great, that explains everything, thanks very much for all your help. I guess I'd have to learn distutils, and then numpy.distutils before I understood how to write such commands myself. BTW, scipy.test('full') showed an error when run on 64bit python. I've included below the results from the test, first for 64 bit and then for comparison for the 32 bit, where no unknown failure showed. (don't feel compelled to look into this if you don't have time/interest, the error is unlikely to affect any of the work I do, so I don't really much care to fix it right now) thanks again for all your help! Dan ------python 2.6.2, 64 bit version ====================================================================== ERROR: test_implicit (test_odr.TestODR) ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/odr/tests/test_odr.py", line 88, in test_implicit out = implicit_odr.run() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/odr/odrpack.py", line 1055, in run self.output = Output(apply(odr, args, kwds)) TypeError: y must be a sequence or integer (if model is implicit) ====================================================================== ERROR: test_complex_nonsymmetric_modes (test_arpack.TestEigenComplexNonSymmetric) ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py", line 264, in test_complex_nonsymmetric_modes self.eval_evec(m,typ,k,which) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py", line 245, in eval_evec eval,evec=eigen(a,k,which=which) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/sparse/linalg/eigen/arpack/arpack.py", line 220, in eigen raise RuntimeError("Error info=%d in arpack"%info) RuntimeError: Error info=-8 in arpack ====================================================================== ERROR: test_nonsymmetric_modes (test_arpack.TestEigenNonSymmetric) ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py", line 201, in test_nonsymmetric_modes self.eval_evec(m,typ,k,which) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py", line 183, in eval_evec eval,evec=eigen(a,k,which=which,**kwds) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/sparse/linalg/eigen/arpack/arpack.py", line 220, in eigen raise RuntimeError("Error info=%d in arpack"%info) RuntimeError: Error info=-8 in arpack ====================================================================== ERROR: test_starting_vector (test_arpack.TestEigenNonSymmetric) ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py", line 211, in test_starting_vector self.eval_evec(self.symmetric[0],typ,k,which='LM',v0=v0) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py", line 183, in eval_evec eval,evec=eigen(a,k,which=which,**kwds) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/sparse/linalg/eigen/arpack/arpack.py", line 220, in eigen raise RuntimeError("Error info=%d in arpack"%info) RuntimeError: Error info=-8 in arpack ====================================================================== FAIL: test_complex_symmetric_modes (test_arpack.TestEigenComplexSymmetric) ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py", line 153, in test_complex_symmetric_modes self.eval_evec(self.symmetric[0],typ,k,which) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py", line 142, in eval_evec assert_array_almost_equal(eval,exact_eval,decimal=_ndigits[typ]) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/testing/utils.py", line 537, in assert_array_almost_equal header='Arrays are not almost equal') File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/testing/utils.py", line 395, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not almost equal (mismatch 100.0%) x: array([ 1.07188725 +6.23436023e-08j, 4.91291142 -3.25412906e-08j], dtype=complex64) y: array([ 5.+0.j, 6.+0.j], dtype=complex64) ---------------------------------------------------------------------- Ran 4020 tests in 350.895s FAILED (KNOWNFAIL=9, SKIP=29, errors=4, failures=1) ----32 bit python 2.5.4: ====================================================================== ERROR: Failure: ImportError (cannot import name set_package_path) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/loader.py", line 364, in loadTestsFromName addr.filename, addr.module) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/importer.py", line 82, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/io/tests/test_mio.py", line 7, in from numpy.testing import set_package_path, restore_path, NumpyTestCase, NumpyTest ImportError: cannot import name set_package_path ====================================================================== ERROR: Failure: NameError (name 'set_package_path' is not defined) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/loader.py", line 364, in loadTestsFromName addr.filename, addr.module) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/importer.py", line 84, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/linalg/tests/test_iterative.py", line 22, in set_package_path() NameError: name 'set_package_path' is not defined ====================================================================== ERROR: Failure: ImportError (cannot import name NumpyTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/loader.py", line 364, in loadTestsFromName addr.filename, addr.module) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/importer.py", line 84, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/linsolve/umfpack/__init__.py", line 6, in from numpy.testing import NumpyTest ImportError: cannot import name NumpyTest ====================================================================== ERROR: Failure: NameError (name 'set_package_path' is not defined) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/loader.py", line 364, in loadTestsFromName addr.filename, addr.module) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/importer.py", line 84, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/sparse/tests/test_sparse.py", line 23, in set_package_path() NameError: name 'set_package_path' is not defined ====================================================================== ERROR: Failure: ImportError (No module named Numeric) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/loader.py", line 364, in loadTestsFromName addr.filename, addr.module) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/importer.py", line 84, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/special/tests/Test.py", line 4, in import Numeric, cephes, RandomArray ImportError: No module named Numeric ====================================================================== ERROR: Failure: NameError (name 'set_local_path' is not defined) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/loader.py", line 364, in loadTestsFromName addr.filename, addr.module) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/importer.py", line 84, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/weave/tests/test_scxx.py", line 8, in set_local_path() NameError: name 'set_local_path' is not defined ====================================================================== ERROR: Failure: NameError (name 'set_package_path' is not defined) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/loader.py", line 364, in loadTestsFromName addr.filename, addr.module) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/importer.py", line 84, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/weave/tests/test_wx_spec.py", line 12, in set_package_path() NameError: name 'set_package_path' is not defined ---------------------------------------------------------------------- Ran 4027 tests in 366.557s FAILED (KNOWNFAIL=8, SKIP=29, errors=7) Out[3]: -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Apr 20 13:31:09 2009 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 20 Apr 2009 12:31:09 -0500 Subject: [SciPy-user] building 64bit scipy In-Reply-To: <15e4667e0904201028q726c7b84hc830c185862f32cc@mail.gmail.com> References: <15e4667e0904181644q255d0038t69012c10722c17a7@mail.gmail.com> <49EC7E61.30408@ar.media.kyoto-u.ac.jp> <15e4667e0904200820i3dde34cal49818f5936c54c2b@mail.gmail.com> <15e4667e0904200833y214e4b03p17574a961719b37d@mail.gmail.com> <5b8d13220904200851y3fa9b1f7v9c3376a0bd09a8a7@mail.gmail.com> <15e4667e0904200925y497d5c60x55f8fd14826dfca@mail.gmail.com> <5b8d13220904201008g2e9f6a5cwd76cb5aaa19a78d@mail.gmail.com> <15e4667e0904201028q726c7b84hc830c185862f32cc@mail.gmail.com> Message-ID: <3d375d730904201031x1217d590i6b615679a8e7e6d@mail.gmail.com> On Mon, Apr 20, 2009 at 12:28, Dan Yamins wrote: > > Great, that explains everything, thanks very much for all your help. ? I > guess I'd have to learn distutils, and then numpy.distutils before I > understood how to write such commands myself. > > BTW, scipy.test('full') showed an error when run on 64bit python.?? I've > included below the results from the test, first for 64 bit and then for > comparison for the 32 bit, where no unknown failure showed.? (don't feel > compelled to look into this if you don't have time/interest, the error is > unlikely to affect any of the work I do, so I don't really much care to fix > it right now) It looks like you have incompatible versions of numpy and scipy. What versions are you trying to install together? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ivo.maljevic at gmail.com Mon Apr 20 13:53:49 2009 From: ivo.maljevic at gmail.com (Ivo Maljevic) Date: Mon, 20 Apr 2009 13:53:49 -0400 Subject: [SciPy-user] Convolve an input with a seri of impulses In-Reply-To: <5df0b3120904201027g25f66692la426bf838be78b0c@mail.gmail.com> References: <5df0b3120904201027g25f66692la426bf838be78b0c@mail.gmail.com> Message-ID: <826c64da0904201053o11158f3chac74e81c390be95d@mail.gmail.com> In general, you would do it the same way you would in Matlab: 1) create a sequence of pulses 2) add zeros in between (e.g., using kron function) Mybe you will find this simple example useful: Ivo #!/usr/bin/python # import from scipy import * from pylab import * L = 50 A_1 = 1 A_2 = -2 # one pulse pulse_vec = zeros(L) pulse_vec[0] = 1 #sequence of pulses my_seq = array([A_1, A_2]) my_pulses = kron(my_seq, pulse_vec) t = linspace(0,10,2000); h = 0.1*exp(-t); # just copy. No need for this in general x = my_pulses[:] print x y = convolve(h,x) subplot(211) plot(x) grid(True) title('two pulses') subplot(212) plot(y) title('output') grid(True) show() 2009/4/20 Mickael > Hi, > > I'm trying to convolve a reference signal, like a step signal, with a > sequence of impulses. Impulses have different amplitudes and timing : > > > ---------------------------------------------------------------------------- > zeta = 0.0193 > wn = 35037.0 > wd = wn*math.sqrt(1.0-zeta**2) > K = math.exp(((-zeta*pi)/math.sqrt(1.0-zeta**2))) > > num = array([9.871]) > den = array([1,1354,1.228*10**9]) > H = signal.lti(num, den) > > # Impulses : amplitudes > A1 = (1.0/(1.0+K)) > A2 = (K/(1.0+K)) > # Impulses : timing > t1 = 0 > t2 = (pi/wd) > > ----------------------------------------------------------------------------- > > I would like to apply to "H" the result of a step signal convolve with > these two pulses but I'don't know to define the vector of the impulses. > --->T,yout,xout = signal.lsim(H, entree, t) <=== entree = scipy.signal. > convolve(*A_step*, My_Impulses?, mode='full') > > Thanks, > > Mickael.**** > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dyamins at gmail.com Mon Apr 20 14:45:04 2009 From: dyamins at gmail.com (Dan Yamins) Date: Mon, 20 Apr 2009 14:45:04 -0400 Subject: [SciPy-user] building 64bit scipy In-Reply-To: <3d375d730904201031x1217d590i6b615679a8e7e6d@mail.gmail.com> References: <15e4667e0904181644q255d0038t69012c10722c17a7@mail.gmail.com> <49EC7E61.30408@ar.media.kyoto-u.ac.jp> <15e4667e0904200820i3dde34cal49818f5936c54c2b@mail.gmail.com> <15e4667e0904200833y214e4b03p17574a961719b37d@mail.gmail.com> <5b8d13220904200851y3fa9b1f7v9c3376a0bd09a8a7@mail.gmail.com> <15e4667e0904200925y497d5c60x55f8fd14826dfca@mail.gmail.com> <5b8d13220904201008g2e9f6a5cwd76cb5aaa19a78d@mail.gmail.com> <15e4667e0904201028q726c7b84hc830c185862f32cc@mail.gmail.com> <3d375d730904201031x1217d590i6b615679a8e7e6d@mail.gmail.com> Message-ID: <15e4667e0904201145g42dbb0a8r83ff870b51475737@mail.gmail.com> On Mon, Apr 20, 2009 at 1:31 PM, Robert Kern wrote: > On Mon, Apr 20, 2009 at 12:28, Dan Yamins wrote: > > > > Great, that explains everything, thanks very much for all your help. I > > guess I'd have to learn distutils, and then numpy.distutils before I > > understood how to write such commands myself. > > > > BTW, scipy.test('full') showed an error when run on 64bit python. I've > > included below the results from the test, first for 64 bit and then for > > comparison for the 32 bit, where no unknown failure showed. (don't feel > > compelled to look into this if you don't have time/interest, the error is > > unlikely to affect any of the work I do, so I don't really much care to > fix > > it right now) > > It looks like you have incompatible versions of numpy and scipy. What > versions are you trying to install together? > Well, I've installed numpy 1.3.0 (from downloading the most recent stable source release from numpy's sourceforge page). Then I installed scipy 0.7.0 (downloading again the most recent stable source release from scipy's sourceforge page) I installed both first using python 2.5.4 (using the most recent OSX binary from www.python.org -- this is a 32bit binary) and then also on python 2.6.2 that I build as a 64-bit binary using macports. Are you saying that the installation both for both versions of python are bad? Or just the 64bit one? thanks! Dan -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Apr 20 14:53:53 2009 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 20 Apr 2009 13:53:53 -0500 Subject: [SciPy-user] building 64bit scipy In-Reply-To: <15e4667e0904201145g42dbb0a8r83ff870b51475737@mail.gmail.com> References: <15e4667e0904181644q255d0038t69012c10722c17a7@mail.gmail.com> <49EC7E61.30408@ar.media.kyoto-u.ac.jp> <15e4667e0904200820i3dde34cal49818f5936c54c2b@mail.gmail.com> <15e4667e0904200833y214e4b03p17574a961719b37d@mail.gmail.com> <5b8d13220904200851y3fa9b1f7v9c3376a0bd09a8a7@mail.gmail.com> <15e4667e0904200925y497d5c60x55f8fd14826dfca@mail.gmail.com> <5b8d13220904201008g2e9f6a5cwd76cb5aaa19a78d@mail.gmail.com> <15e4667e0904201028q726c7b84hc830c185862f32cc@mail.gmail.com> <3d375d730904201031x1217d590i6b615679a8e7e6d@mail.gmail.com> <15e4667e0904201145g42dbb0a8r83ff870b51475737@mail.gmail.com> Message-ID: <3d375d730904201153g688d7ea6ob2ad048c7ee501dd@mail.gmail.com> On Mon, Apr 20, 2009 at 13:45, Dan Yamins wrote: > > > On Mon, Apr 20, 2009 at 1:31 PM, Robert Kern wrote: >> >> On Mon, Apr 20, 2009 at 12:28, Dan Yamins wrote: >> > >> > Great, that explains everything, thanks very much for all your help. ? I >> > guess I'd have to learn distutils, and then numpy.distutils before I >> > understood how to write such commands myself. >> > >> > BTW, scipy.test('full') showed an error when run on 64bit python.?? I've >> > included below the results from the test, first for 64 bit and then for >> > comparison for the 32 bit, where no unknown failure showed.? (don't feel >> > compelled to look into this if you don't have time/interest, the error >> > is >> > unlikely to affect any of the work I do, so I don't really much care to >> > fix >> > it right now) >> >> It looks like you have incompatible versions of numpy and scipy. What >> versions are you trying to install together? > > Well, I've installed numpy 1.3.0 (from downloading the most recent stable > source release from numpy's sourceforge page). > > Then I installed scipy 0.7.0 (downloading again the most recent stable > source release from scipy's sourceforge page) > > I installed both first using python 2.5.4? (using the most recent OSX binary > from www.python.org -- this is a 32bit binary) and then also on python > 2.6.2? that I build as a 64-bit binary using macports. > > Are you saying that the installation both for both versions of python are > bad? Or just the 64bit one? Sorry, I was just referring to the 32 bit one. I really doubt that you have a clean install of scipy 0.7.0; the "set_package_path" stuff was removed before the 0.7.0 release. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dyamins at gmail.com Mon Apr 20 15:06:13 2009 From: dyamins at gmail.com (Dan Yamins) Date: Mon, 20 Apr 2009 15:06:13 -0400 Subject: [SciPy-user] building 64bit scipy In-Reply-To: <3d375d730904201153g688d7ea6ob2ad048c7ee501dd@mail.gmail.com> References: <15e4667e0904181644q255d0038t69012c10722c17a7@mail.gmail.com> <15e4667e0904200820i3dde34cal49818f5936c54c2b@mail.gmail.com> <15e4667e0904200833y214e4b03p17574a961719b37d@mail.gmail.com> <5b8d13220904200851y3fa9b1f7v9c3376a0bd09a8a7@mail.gmail.com> <15e4667e0904200925y497d5c60x55f8fd14826dfca@mail.gmail.com> <5b8d13220904201008g2e9f6a5cwd76cb5aaa19a78d@mail.gmail.com> <15e4667e0904201028q726c7b84hc830c185862f32cc@mail.gmail.com> <3d375d730904201031x1217d590i6b615679a8e7e6d@mail.gmail.com> <15e4667e0904201145g42dbb0a8r83ff870b51475737@mail.gmail.com> <3d375d730904201153g688d7ea6ob2ad048c7ee501dd@mail.gmail.com> Message-ID: <15e4667e0904201206p579f87c5j9168a12ca6b8ddf@mail.gmail.com> On Mon, Apr 20, 2009 at 2:53 PM, Robert Kern wrote: > On Mon, Apr 20, 2009 at 13:45, Dan Yamins wrote: > > > > > > Sorry, I was just referring to the 32 bit one. I really doubt that you > have a clean install of scipy 0.7.0; the "set_package_path" stuff was > removed before the 0.7.0 release. > Hm ... well my procedure for installing the 32bit version was: a) download http://sourceforge.net/project/downloading.php?group_id=27747&filename=scipy-0.7.0.tar.gz&a=56623687 b) unzipping c) build running: python setup.py install in the source directory. Have I done something obviously wrong? I'm on OSX, so I don't need to build blas or lapack, .... Thanks, dan -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Apr 20 15:08:39 2009 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 20 Apr 2009 14:08:39 -0500 Subject: [SciPy-user] building 64bit scipy In-Reply-To: <15e4667e0904201206p579f87c5j9168a12ca6b8ddf@mail.gmail.com> References: <15e4667e0904181644q255d0038t69012c10722c17a7@mail.gmail.com> <15e4667e0904200833y214e4b03p17574a961719b37d@mail.gmail.com> <5b8d13220904200851y3fa9b1f7v9c3376a0bd09a8a7@mail.gmail.com> <15e4667e0904200925y497d5c60x55f8fd14826dfca@mail.gmail.com> <5b8d13220904201008g2e9f6a5cwd76cb5aaa19a78d@mail.gmail.com> <15e4667e0904201028q726c7b84hc830c185862f32cc@mail.gmail.com> <3d375d730904201031x1217d590i6b615679a8e7e6d@mail.gmail.com> <15e4667e0904201145g42dbb0a8r83ff870b51475737@mail.gmail.com> <3d375d730904201153g688d7ea6ob2ad048c7ee501dd@mail.gmail.com> <15e4667e0904201206p579f87c5j9168a12ca6b8ddf@mail.gmail.com> Message-ID: <3d375d730904201208t31d7e3fev9a6e564e82c862dd@mail.gmail.com> On Mon, Apr 20, 2009 at 14:06, Dan Yamins wrote: > > On Mon, Apr 20, 2009 at 2:53 PM, Robert Kern wrote: >> >> On Mon, Apr 20, 2009 at 13:45, Dan Yamins wrote: >> > >> > >> >> Sorry, I was just referring to the 32 bit one. I really doubt that you >> have a clean install of scipy 0.7.0; the "set_package_path" stuff was >> removed before the 0.7.0 release. > > > Hm ...? well my procedure for installing the 32bit version was: > ? ? a) download > http://sourceforge.net/project/downloading.php?group_id=27747&filename=scipy-0.7.0.tar.gz&a=56623687 > ??? b) unzipping > ? ? c) build running: > ? ? ? ?? python setup.py install > ? ? in the source directory. > > Have I done something obviously wrong??? I'm on OSX, so I don't need to > build blas or lapack, .... I suspect you have an old install already. You can double-check that the files raising the exceptions are not the same as those in your source tarball. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dyamins at gmail.com Mon Apr 20 15:37:03 2009 From: dyamins at gmail.com (Dan Yamins) Date: Mon, 20 Apr 2009 15:37:03 -0400 Subject: [SciPy-user] building 64bit scipy In-Reply-To: <3d375d730904201208t31d7e3fev9a6e564e82c862dd@mail.gmail.com> References: <15e4667e0904181644q255d0038t69012c10722c17a7@mail.gmail.com> <5b8d13220904200851y3fa9b1f7v9c3376a0bd09a8a7@mail.gmail.com> <15e4667e0904200925y497d5c60x55f8fd14826dfca@mail.gmail.com> <5b8d13220904201008g2e9f6a5cwd76cb5aaa19a78d@mail.gmail.com> <15e4667e0904201028q726c7b84hc830c185862f32cc@mail.gmail.com> <3d375d730904201031x1217d590i6b615679a8e7e6d@mail.gmail.com> <15e4667e0904201145g42dbb0a8r83ff870b51475737@mail.gmail.com> <3d375d730904201153g688d7ea6ob2ad048c7ee501dd@mail.gmail.com> <15e4667e0904201206p579f87c5j9168a12ca6b8ddf@mail.gmail.com> <3d375d730904201208t31d7e3fev9a6e564e82c862dd@mail.gmail.com> Message-ID: <15e4667e0904201237t7e864f17icdd88c0cc19d5746@mail.gmail.com> On Mon, Apr 20, 2009 at 3:08 PM, Robert Kern wrote: > On Mon, Apr 20, 2009 at 14:06, Dan Yamins wrote: > > I suspect you have an old install already. You can double-check that > the files raising the exceptions are not the same as those in your > source tarball. You're quite right, my python 2.5 installation had a (failed) scipy 0.6 installation from some time ago. I cleared out the scipy directory and egg in the site-packages directory and re-installed. Now I have no (unknown) errors reports in the unit tests. In terms of the errors with the 2.6 64-installation (that was certainly a clean install), should I merely ignore those? Thanks! Dan -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivo.maljevic at gmail.com Mon Apr 20 16:05:50 2009 From: ivo.maljevic at gmail.com (Ivo Maljevic) Date: Mon, 20 Apr 2009 16:05:50 -0400 Subject: [SciPy-user] Convolve an input with a seri of impulses In-Reply-To: <5df0b3120904201027g25f66692la426bf838be78b0c@mail.gmail.com> References: <5df0b3120904201027g25f66692la426bf838be78b0c@mail.gmail.com> Message-ID: <826c64da0904201305j7032e54dt76556f92e09c918e@mail.gmail.com> I never use LTI systems defined in this way, but out of curiosity I tried to make some changes into my original code in a hope I will learn something and help you in a way. Try the attached script, it uses your code. Ivo 2009/4/20 Mickael > Hi, > > I'm trying to convolve a reference signal, like a step signal, with a > sequence of impulses. Impulses have different amplitudes and timing : > > > ---------------------------------------------------------------------------- > zeta = 0.0193 > wn = 35037.0 > wd = wn*math.sqrt(1.0-zeta**2) > K = math.exp(((-zeta*pi)/math.sqrt(1.0-zeta**2))) > > num = array([9.871]) > den = array([1,1354,1.228*10**9]) > H = signal.lti(num, den) > > # Impulses : amplitudes > A1 = (1.0/(1.0+K)) > A2 = (K/(1.0+K)) > # Impulses : timing > t1 = 0 > t2 = (pi/wd) > > ----------------------------------------------------------------------------- > > I would like to apply to "H" the result of a step signal convolve with > these two pulses but I'don't know to define the vector of the impulses. > --->T,yout,xout = signal.lsim(H, entree, t) <=== entree = scipy.signal. > convolve(*A_step*, My_Impulses?, mode='full') > > Thanks, > > Mickael.**** > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: conv2.py Type: text/x-python Size: 858 bytes Desc: not available URL: From Chris.Barker at noaa.gov Mon Apr 20 17:53:21 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 20 Apr 2009 14:53:21 -0700 Subject: [SciPy-user] Build error on PPC Mac OS X 10.4 In-Reply-To: <799406d60904151831h5c98743cnf9f12ae909a962a2@mail.gmail.com> References: <799406d60904142203n560627dcp1525d978c817818c@mail.gmail.com> <49E57B13.10709@ar.media.kyoto-u.ac.jp> <799406d60904150544u41988da9re03e672729b91f0d@mail.gmail.com> <3d375d730904151250u3b96c775h18f2982f8964b817@mail.gmail.com> <799406d60904151831h5c98743cnf9f12ae909a962a2@mail.gmail.com> Message-ID: <49ECEED1.70404@noaa.gov> Adam Mercer wrote: > MacPorts has infrastructure to tell a port not to build to universally > although come to think of it it doesn't work properly with python > ports. But never the less the gfortran is _not_ universal. The gfortran binaries themselves? I can't see that that would be an issue, but it would be nice if gfortran could build universal binaries. Which would mean a universal libgfortran, I guess. I'd love it if macports could build scipy universal... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From david at ar.media.kyoto-u.ac.jp Mon Apr 20 22:52:40 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 21 Apr 2009 11:52:40 +0900 Subject: [SciPy-user] Build error on PPC Mac OS X 10.4 In-Reply-To: <49ECEED1.70404@noaa.gov> References: <799406d60904142203n560627dcp1525d978c817818c@mail.gmail.com> <49E57B13.10709@ar.media.kyoto-u.ac.jp> <799406d60904150544u41988da9re03e672729b91f0d@mail.gmail.com> <3d375d730904151250u3b96c775h18f2982f8964b817@mail.gmail.com> <799406d60904151831h5c98743cnf9f12ae909a962a2@mail.gmail.com> <49ECEED1.70404@noaa.gov> Message-ID: <49ED34F8.8090607@ar.media.kyoto-u.ac.jp> Christopher Barker wrote: > Adam Mercer wrote: > >> MacPorts has infrastructure to tell a port not to build to universally >> although come to think of it it doesn't work properly with python >> ports. But never the less the gfortran is _not_ universal. >> > > The gfortran binaries themselves? I can't see that that would be an > issue, but it would be nice if gfortran could build universal binaries. > Which would mean a universal libgfortran, I guess. > > I'd love it if macports could build scipy universal... > The problem is that building a universal gcc is damn hard: http://r.research.att.com/tools/ As you have to build the driver by yourself from the cross compilers. It also means building times are significantly longer, which may be a problem for something like macport, David From mickael.paris at gmail.com Tue Apr 21 02:36:28 2009 From: mickael.paris at gmail.com (Mickael) Date: Tue, 21 Apr 2009 08:36:28 +0200 Subject: [SciPy-user] Convolve an input with a seri of impulses In-Reply-To: <826c64da0904201305j7032e54dt76556f92e09c918e@mail.gmail.com> References: <5df0b3120904201027g25f66692la426bf838be78b0c@mail.gmail.com> <826c64da0904201305j7032e54dt76556f92e09c918e@mail.gmail.com> Message-ID: <5df0b3120904202336v69f81af9naa7b49c32a0edb26@mail.gmail.com> 2009/4/20 Ivo Maljevic > I never use LTI systems defined in this way, but out of curiosity I tried > to make some changes into my original code in a hope I will learn something > and help you in a way. > > Try the attached script, it uses your code. > > Ivo > > 2009/4/20 Mickael > >> Hi, >> >> >> I'm trying to convolve a reference signal, like a step signal, with a >> sequence of impulses. Impulses have different amplitudes and timing : >> >> >> ---------------------------------------------------------------------------- >> zeta = 0.0193 >> wn = 35037.0 >> wd = wn*math.sqrt(1.0-zeta**2) >> K = math.exp(((-zeta*pi)/math.sqrt(1.0-zeta**2))) >> >> num = array([9.871]) >> den = array([1,1354,1.228*10**9]) >> H = signal.lti(num, den) >> >> # Impulses : amplitudes >> A1 = (1.0/(1.0+K)) >> A2 = (K/(1.0+K)) >> # Impulses : timing >> t1 = 0 >> t2 = (pi/wd) >> >> ----------------------------------------------------------------------------- >> >> I would like to apply to "H" the result of a step signal convolve with >> these two pulses but I'don't know to define the vector of the impulses. >> --->T,yout,xout = signal.lsim(H, entree, t) <=== entree = scipy.signal. >> convolve(*A_step*, My_Impulses?, mode='full') >> >> Thanks, >> >> Mickael.**** >> _______________________________________________ >> SciPy-user mailing list >> SciPy-user at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > Hi Ivo, thank you very much for your help. i will try your code today and post the result thanks, Mickael. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivo.maljevic at gmail.com Tue Apr 21 07:10:36 2009 From: ivo.maljevic at gmail.com (Ivo Maljevic) Date: Tue, 21 Apr 2009 07:10:36 -0400 Subject: [SciPy-user] Convolve an input with a seri of impulses In-Reply-To: <5df0b3120904202336v69f81af9naa7b49c32a0edb26@mail.gmail.com> References: <5df0b3120904201027g25f66692la426bf838be78b0c@mail.gmail.com> <826c64da0904201305j7032e54dt76556f92e09c918e@mail.gmail.com> <5df0b3120904202336v69f81af9naa7b49c32a0edb26@mail.gmail.com> Message-ID: <826c64da0904210410j937acaeyd9621ff866ee81f6@mail.gmail.com> Mickael, I do not know why you have to characterise your LTI system in a such way. I would probably find an impulse response of the system characterised with num and den using filter() and than I would have full control. I am only guessing that you are trying to describe an analog system, and this is probably a convenient way, but one I never use. See if the time output has a different sampling rate (tout variable) than the time axis I've added. In this case you would have to use tout and set your pulse sequence to zeros at the appropriate points. All I'm saying is that you have more work to do there. Ivo 2009/4/21 Mickael > > > 2009/4/20 Ivo Maljevic > > I never use LTI systems defined in this way, but out of curiosity I tried >> to make some changes into my original code in a hope I will learn something >> and help you in a way. >> >> Try the attached script, it uses your code. >> >> Ivo >> >> 2009/4/20 Mickael >> >>> Hi, >>> >>> >>> I'm trying to convolve a reference signal, like a step signal, with a >>> sequence of impulses. Impulses have different amplitudes and timing : >>> >>> >>> ---------------------------------------------------------------------------- >>> zeta = 0.0193 >>> wn = 35037.0 >>> wd = wn*math.sqrt(1.0-zeta**2) >>> K = math.exp(((-zeta*pi)/math.sqrt(1.0-zeta**2))) >>> >>> num = array([9.871]) >>> den = array([1,1354,1.228*10**9]) >>> H = signal.lti(num, den) >>> >>> # Impulses : amplitudes >>> A1 = (1.0/(1.0+K)) >>> A2 = (K/(1.0+K)) >>> # Impulses : timing >>> t1 = 0 >>> t2 = (pi/wd) >>> >>> ----------------------------------------------------------------------------- >>> >>> I would like to apply to "H" the result of a step signal convolve with >>> these two pulses but I'don't know to define the vector of the impulses. >>> --->T,yout,xout = signal.lsim(H, entree, t) <=== entree = scipy.signal. >>> convolve(*A_step*, My_Impulses?, mode='full') >>> >>> Thanks, >>> >>> Mickael.**** >>> _______________________________________________ >>> SciPy-user mailing list >>> SciPy-user at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> >> _______________________________________________ >> SciPy-user mailing list >> SciPy-user at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> Hi Ivo, > > thank you very much for your help. > > i will try your code today and post the result > > thanks, > > Mickael. > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mickael.paris at gmail.com Tue Apr 21 08:01:40 2009 From: mickael.paris at gmail.com (Mickael) Date: Tue, 21 Apr 2009 14:01:40 +0200 Subject: [SciPy-user] Convolve an input with a seri of impulses In-Reply-To: <826c64da0904210410j937acaeyd9621ff866ee81f6@mail.gmail.com> References: <5df0b3120904201027g25f66692la426bf838be78b0c@mail.gmail.com> <826c64da0904201305j7032e54dt76556f92e09c918e@mail.gmail.com> <5df0b3120904202336v69f81af9naa7b49c32a0edb26@mail.gmail.com> <826c64da0904210410j937acaeyd9621ff866ee81f6@mail.gmail.com> Message-ID: <5df0b3120904210501i2e6c8e44k36b3ddb4902cb3c5@mail.gmail.com> Hi Ivo, My LTI system is just characterised by the "num" and "den" arrays. What I'm trying to do is to shape my input signal which will apply to H(nem,dem) in order to control the vibration. Convolving a step entry (for exemple) with some impulses allows to decrease the natural vibration of H, this method is known as "Input-shaping". Since this morning I'm working on your code "conv2.py" and I've seen that you convolve Hout with x (the sequence of two impulses). In fact, this the step "u" that must be convolved with "x": x = kron(my_seq, pulse_vec) #y = convolve(Hout,x) y = signal.fftconvolve(u,x) --->fftconvolve allows to obtain the good shaped command (the step command is transformed to a staircase command) The result is a staircase where the number of stair is equal to the number of impulse. Just before to post the results, I' m looking if the ouput times are correct (as you explain) Again, thanks for your help, it seems to work ;) Mickael. 2009/4/21 Ivo Maljevic > Mickael, > I do not know why you have to characterise your LTI system in a such way. I > would probably > find an impulse response of the system characterised with num and den using > filter() and than > I would have full control. I am only guessing that you are trying to > describe an analog system, > and this is probably a convenient way, but one I never use. See if the time > output has a different > sampling rate (tout variable) than the time axis I've added. In this case > you would have to use tout > and set your pulse sequence to zeros at the appropriate points. > > All I'm saying is that you have more work to do there. > > Ivo > > 2009/4/21 Mickael > > >> >> 2009/4/20 Ivo Maljevic >> >> I never use LTI systems defined in this way, but out of curiosity I tried >>> to make some changes into my original code in a hope I will learn something >>> and help you in a way. >>> >>> Try the attached script, it uses your code. >>> >>> Ivo >>> >>> 2009/4/20 Mickael >>> >>>> Hi, >>>> >>>> >>>> I'm trying to convolve a reference signal, like a step signal, with a >>>> sequence of impulses. Impulses have different amplitudes and timing : >>>> >>>> >>>> ---------------------------------------------------------------------------- >>>> zeta = 0.0193 >>>> wn = 35037.0 >>>> wd = wn*math.sqrt(1.0-zeta**2) >>>> K = math.exp(((-zeta*pi)/math.sqrt(1.0-zeta**2))) >>>> >>>> num = array([9.871]) >>>> den = array([1,1354,1.228*10**9]) >>>> H = signal.lti(num, den) >>>> >>>> # Impulses : amplitudes >>>> A1 = (1.0/(1.0+K)) >>>> A2 = (K/(1.0+K)) >>>> # Impulses : timing >>>> t1 = 0 >>>> t2 = (pi/wd) >>>> >>>> ----------------------------------------------------------------------------- >>>> >>>> I would like to apply to "H" the result of a step signal convolve with >>>> these two pulses but I'don't know to define the vector of the impulses. >>>> --->T,yout,xout = signal.lsim(H, entree, t) <=== entree = >>>> scipy.signal.convolve(*A_step*, My_Impulses?, mode='full') >>>> >>>> Thanks, >>>> >>>> Mickael.**** >>>> _______________________________________________ >>>> SciPy-user mailing list >>>> SciPy-user at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/scipy-user >>>> >>>> >>> >>> _______________________________________________ >>> SciPy-user mailing list >>> SciPy-user at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> Hi Ivo, >> >> thank you very much for your help. >> >> i will try your code today and post the result >> >> thanks, >> >> Mickael. >> >> _______________________________________________ >> SciPy-user mailing list >> SciPy-user at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at vrbka.net Tue Apr 21 17:42:01 2009 From: lists at vrbka.net (lists at vrbka.net) Date: Tue, 21 Apr 2009 23:42:01 +0200 Subject: [SciPy-user] conjugate gradients solver - operator adjoint Message-ID: hi guys, in my program for solving the integral equations in the liquid state theory (based on scipy/numpy; check pyoz.vrbka.net if you are interested in details) i need to implement a solver of a linear system AX=B - generally, this shouldn't be a big problem. there are some issues, however... 1) X and B are sets of functions - X_ij and B_ij, X and B are NxN matrices (most frequently N=2-3) 2) the functions are discretized (generally 2^10-2^12 discretization points) therefore, i have (in the easiest case) 2^10 of 2x2 problems 3) A is slightly complicated linear operator, involving fourier transforms in the literature, i found that this system of equations could be solved using the nonsymmetric (iterative) conjugate gradients procedure. in the paper (i can provide the reference on request, don't have it here at the moment), actually, the author presented the equations for the CG algorithm, involving the adjoint of the operator A - the resulting equations were relatively simple. the author speaks also of matrices of the size of 10^3 x 10^3 without giving further details - i presume functions at all the discretization points would be put into one big block diagonal matrix... having a look at the sparse.linalg.solve.cg, there's no way of specifying the adjoint (which should be, if i understand the method correctly, necessary for getting the result, since it's impossible to invert something that big efficiently). i'm searching for help here, since my knowledge of the mathematics is quite limited in this area... would be really very glad for any hints or comments - how would you proceed in this case? thanks in advance for your help! best, lubos From pav at iki.fi Wed Apr 22 03:46:49 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 22 Apr 2009 07:46:49 +0000 (UTC) Subject: [SciPy-user] conjugate gradients solver - operator adjoint References: Message-ID: Tue, 21 Apr 2009 23:42:01 +0200, lists kirjoitti: > in my program for solving the integral equations in the liquid state > theory (based on scipy/numpy; check pyoz.vrbka.net if you are interested > in details) i need to implement a solver of a linear system AX=B - > generally, this shouldn't be a big problem. there are some issues, > however... [clip] > having a look at the sparse.linalg.solve.cg, there's no way of > specifying the adjoint The conjugate gradient method assumes that the matrix is hermitian, hence no need for specifying adjoint separately. The algorithm you refer to is probably something different. A reference to the paper would come handy. Anyway, as far as I see, none of the solvers in sparse.linalg.* make use of the adjoint, so the particular algorithm is probably not implemented in scipy. > (which should be, if i understand the method > correctly, necessary for getting the result, since it's impossible to > invert something that big efficiently). This is probably debatable. There are many iterative algorithms that can invert non-symmetric matrices not needing to know the adjoint. You could try using one of scipy.sparse.linalg.cgs scipy.sparse.linalg.bicgstab scipy.sparse.linalg.gmres to see if one of them works well with your problem. Note that in general, large problems may require preconditioning for the solvers to work efficiently. If you have the sparse matrix at hand and don't want to think, possibilities include incomplete (LU) decompositions (scipy.sparse.linalg.splu), algebraic multigrid (see http://code.google.com/p/pyamg/), etc... Especially PyAMG could be worth trying out, as it's fairly easy to use. -- Pauli Virtanen From lists at vrbka.net Wed Apr 22 04:00:07 2009 From: lists at vrbka.net (Lubos Vrbka) Date: Wed, 22 Apr 2009 10:00:07 +0200 Subject: [SciPy-user] conjugate gradients solver - operator adjoint In-Reply-To: References: Message-ID: <49EECE87.2020801@vrbka.net> hi, > The conjugate gradient method assumes that the matrix is hermitian, hence > no need for specifying adjoint separately. > > The algorithm you refer to is probably something different. A reference > to the paper would come handy. the paper is journal of computational physics 61 (1985) 280-285. in case of need/interest i could forward it to you. for obvious reasons i don't attach it to this message... would be glad to know your opinion on this problem... will check also the other options. i think the problem is, that in order for the algorithm to work (it's newton-raphson + conjugate gradients), it's necessary to have the exact adjoint in the solution... best, -- Lubos _ at _" http://www.lubos.vrbka.net From pav at iki.fi Wed Apr 22 04:00:48 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 22 Apr 2009 08:00:48 +0000 (UTC) Subject: [SciPy-user] conjugate gradients solver - operator adjoint References: Message-ID: Wed, 22 Apr 2009 07:46:49 +0000, Pauli Virtanen kirjoitti: [clip] >> (which should be, if i understand the method correctly, necessary for >> getting the result, since it's impossible to invert something that big >> efficiently). > > This is probably debatable. There are many iterative algorithms that can > invert non-symmetric matrices not needing to know the adjoint. Oops, I'm mistaken here, scipy.sparse.linalg.bicg and .qmr require the adjoint. (However, it is automatically computed if you give in a sparse matrix.) So the algorithm you described probably is scipy.sparse.linalg.bicg or some related method. -- Pauli Virtanen From pav at iki.fi Wed Apr 22 04:33:09 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 22 Apr 2009 08:33:09 +0000 (UTC) Subject: [SciPy-user] conjugate gradients solver - operator adjoint References: <49EECE87.2020801@vrbka.net> Message-ID: Wed, 22 Apr 2009 10:00:07 +0200, Lubos Vrbka kirjoitti: [clip] > would be glad to know your opinion on this problem... will check also > the other options. i think the problem is, that in order for the > algorithm to work (it's newton-raphson + conjugate gradients), it's > necessary to have the exact adjoint in the solution... Not knowing more about the particular problem (our univ. doesn't have a subscription to that old JCP articles...), but I'd say Netwon-Raphson should in general work with any iterative method. In fact, I'm presently solving equations with > 1e5 unknowns with Newton + preconditioned GMRES, and this at least worksforme. However, it could be that the fluid dynamics equations have some property that makes one iterative method favorable over others. -- Pauli Virtanen From lorenzo.isella at gmail.com Wed Apr 22 05:39:56 2009 From: lorenzo.isella at gmail.com (Lorenzo Isella) Date: Wed, 22 Apr 2009 11:39:56 +0200 Subject: [SciPy-user] Relatively simple PDE's and Scipy Message-ID: Dear All, I hope this is not too off-topic. Some time ago I posted about the general availability of PDE solvers under SciPy. Now I can be more specific; say you are given the equation \partial_t f(x,t)+\partial_x(f(x,t)/x)=0, where t is time and x is position in 1D. I would say that I am dealing with a 1D PDE (so, nothing like solving a multi-dimensional PDE on a complex domain). Do you think there is any ready-to-use method to tackle this kind of equation under (or tightly integrated with) SciPy? I stress that x is a scalar variable and not a vector. Overall, it does not look (naively) a terrible equation, but I am having a hard time with it. Many thanks Lorenzo From sebastian.walter at gmail.com Wed Apr 22 05:58:25 2009 From: sebastian.walter at gmail.com (Sebastian Walter) Date: Wed, 22 Apr 2009 11:58:25 +0200 Subject: [SciPy-user] Relatively simple PDE's and Scipy In-Reply-To: References: Message-ID: I'm not sure I understand correctly: but if you want a quick way to solve such problems you can use the "method of lines": I.e. discretize either the time t or the space x and then apply an ODE solver available in scipy: e.g. the above equation leads to a system of ODEs \partial_t f(x_n,t) + [g(x_{n+1}) - g(x_n})]/h = 0 n=0,...,N_gridsize where h = x_{n+1} - x_n and g = f(t,x)/x On Wed, Apr 22, 2009 at 11:39 AM, Lorenzo Isella wrote: > Dear All, > I hope this is not too off-topic. > Some time ago I posted about the general availability of PDE solvers > under SciPy. > Now I can be more specific; say you are given the equation > > \partial_t f(x,t)+\partial_x(f(x,t)/x)=0, > where t is time and x is position in 1D. I would say that I am dealing > with a 1D PDE (so, nothing like solving a multi-dimensional PDE on a > complex domain). > > Do you think there is any ready-to-use method to tackle this kind of > equation under (or tightly integrated with) SciPy? > I stress that x is a scalar variable and not a vector. Overall, it > does not look (naively) a terrible equation, but I am having a hard > time with it. > Many thanks > > Lorenzo > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From aaulagnier at gmail.com Wed Apr 22 06:08:31 2009 From: aaulagnier at gmail.com (Adrien Aulagnier) Date: Wed, 22 Apr 2009 12:08:31 +0200 Subject: [SciPy-user] Optimization fmin_tnc with equality constraint In-Reply-To: <20f2e9f60904200753v33ef50ccq2fef0b694136519f@mail.gmail.com> References: <20f2e9f60904200504x1b2ef230w7192b9092a3cb8af@mail.gmail.com> <20f2e9f60904200753v33ef50ccq2fef0b694136519f@mail.gmail.com> Message-ID: <20f2e9f60904220308s709fa481ice7d75058e440087@mail.gmail.com> 2009/4/20 Adrien Aulagnier > > > 2009/4/20 Dave Hirschfeld > > Adrien Aulagnier gmail.com> writes: >> >> > >> > HelloSorry to disturb you. My question might be simple:The function to >> minimize is simply f (w) = w' * Sigma * wwith w a vector and Sigma a >> square >> matrixThe constraints are : 0 <= w <= 1 >> > sum(w) = 1This last constraint is my problem.Could you give me some >> clue >> ?Thanks for your help >> > >> > _______________________________________________ >> > SciPy-user mailing list >> > SciPy-user scipy.org >> > http://mail.scipy.org/mailman/listinfo/scipy-user >> > >> >> This discussion may help: >> >> >> http://thread.gmane.org/gmane.comp.python.scientific.user/16180/focus=16187 >> >> _______________________________________________ >> SciPy-user mailing list >> SciPy-user at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > > Thank you for the help but my description was not complet: > > The function to minimize is simply f (w) = w' * Sigma * w > > with w a vector and Sigma a square matrix > > The constraints are : > > 0 <= w <= 1 > sum(w) = 1 > R' * w = a constant > > and R a vector of the same size as w > > Sorry for the missing information. > > I think that the Aitchison transformation won't work in this case... > > Do you have an idea to solve this system ? > > Thanks for your help > > > The problem is simply a quadratic programming problem scipy.optimize is not provided with a direct tool for this problem This type of problem is used a lot in Markowitz portfolio theory or efficient frontier construction One should use an addon module call cvxopt http://abel.ee.ucla.edu/cvxopt/ This is a software for convex optimization which is really useful with scipy A simple usage is provided with openopt I used openopt f = scipy.zeros(numberOfAssets) Aeq = scipy.ones((2, numberOfAssets)) Aeq[0,:] = expReturns.T beq = scipy.ones((2, 1)) # adapt this lb = scipy.zeros((numberOfAssets, 1)) lh = scipy.ones((numberOfAssets, 1)) p = scikits.openopt.QP(cov, f, Aeq = Aeq, beq = beq, lb = lb, lh = lh, iprint = -1) r = p.solve('cvxopt_qp') That's all folks Hope this might help someone Thanks for this mailing list -------------- next part -------------- An HTML attachment was scrubbed... URL: From gnurser at googlemail.com Wed Apr 22 06:13:02 2009 From: gnurser at googlemail.com (George Nurser) Date: Wed, 22 Apr 2009 11:13:02 +0100 Subject: [SciPy-user] Relatively simple PDE's and Scipy In-Reply-To: References: Message-ID: <1d1e6ea70904220313j61ef149do5e7df79935d0a415@mail.gmail.com> I believe you can solve this particular equation analytically. For if F(x,t)=f(x,t)/x x\partial_t F(x,t)+\partial_x F(x,t)=0 is satisfied by F(x,t)=phi(t-.5x^2), where phi is set by e.g. the boundary condition at x_0 or the initial condition at t_0 Then f(x,t)=phi(t-.5x^2)/x + HTH. George Nurser. From gnurser at googlemail.com Wed Apr 22 06:20:18 2009 From: gnurser at googlemail.com (George Nurser) Date: Wed, 22 Apr 2009 11:20:18 +0100 Subject: [SciPy-user] Relatively simple PDE's and Scipy In-Reply-To: <1d1e6ea70904220313j61ef149do5e7df79935d0a415@mail.gmail.com> References: <1d1e6ea70904220313j61ef149do5e7df79935d0a415@mail.gmail.com> Message-ID: <1d1e6ea70904220320p3bc535d5h91fbcf4ec3db8138@mail.gmail.com> correction.. 2009/4/22 George Nurser : > I believe you can solve this particular equation analytically. > For if F(x,t)=f(x,t)/x > x\partial_t F(x,t)+\partial_x F(x,t)=0 > > is satisfied by F(x,t)=phi(t-.5x^2), where phi is set by e.g. the > boundary condition at x_0 or the initial condition at t_0 > > Then f(x,t)=phi(t-.5x^2)/x No! f(x,t)=x.phi(t-.5x^2) ;) > HTH. George Nurser. > From dmitrey15 at ukr.net Wed Apr 22 06:40:25 2009 From: dmitrey15 at ukr.net (dmitrey) Date: Wed, 22 Apr 2009 13:40:25 +0300 Subject: [SciPy-user] Optimization fmin_tnc with equality constraint In-Reply-To: <20f2e9f60904220308s709fa481ice7d75058e440087@mail.gmail.com> References: <20f2e9f60904200504x1b2ef230w7192b9092a3cb8af@mail.gmail.com> <20f2e9f60904200753v33ef50ccq2fef0b694136519f@mail.gmail.com> <20f2e9f60904220308s709fa481ice7d75058e440087@mail.gmail.com> Message-ID: <49EEF419.6040405@ukr.net> Adrien Aulagnier ?????: > > > 2009/4/20 Adrien Aulagnier > > > > > 2009/4/20 Dave Hirschfeld > > > Adrien Aulagnier gmail.com > > writes: > > > > > HelloSorry to disturb you. My question might be simple:The > function to > minimize is simply f (w) = w' * Sigma * wwith w a vector and > Sigma a square > matrixThe constraints are : 0 <= w <= 1 > > sum(w) = 1This last constraint is my problem.Could you give > me some clue > ?Thanks for your help > > > > _______________________________________________ > > SciPy-user mailing list > > SciPy-user scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > This discussion may help: > > http://thread.gmane.org/gmane.comp.python.scientific.user/16180/focus=16187 > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > Thank you for the help but my description was not complet: > > > The function to minimize is simply f (w) = w' * Sigma * w > > with w a vector and Sigma a square matrix > > The constraints are : > > 0 <= w <= 1 > sum(w) = 1 > R' * w = a constant > > and R a vector of the same size as w > > Sorry for the missing information. > > I think that the Aitchison transformation won't work in this case... > > Do you have an idea to solve this system ? > > Thanks for your help > > > The problem is simply a quadratic programming problem > > scipy.optimize is not provided with a direct tool for this problem > > This type of problem is used a lot in Markowitz portfolio theory or > efficient frontier construction > > One should use an addon module call cvxopt http://abel.ee.ucla.edu/cvxopt/ > This is a software for convex optimization which is really useful with > scipy > > A simple usage is provided with openopt > > I used openopt > > f = scipy.zeros(numberOfAssets) > > Aeq = scipy.ones((2, numberOfAssets)) > Aeq[0,:] = expReturns.T > beq = scipy.ones((2, 1)) # adapt this > > lb = scipy.zeros((numberOfAssets, 1)) > lh = scipy.ones((numberOfAssets, 1)) > > p = scikits.openopt.QP(cov, f, Aeq = Aeq, beq = beq, lb = lb, lh = lh, > iprint = -1) > r = p.solve('cvxopt_qp') > > > That's all folks > > Hope this might help someone > > Thanks for this mailing list Hi there, I don't know what lh stands for, it is unused, maybe you meant ub here? p = scikits.openopt.QP(cov, f, Aeq = Aeq, beq = beq, lb = lb, ub = lh, iprint = -1) Also, using scikits.openopt means you use obsolete openopt ver <= 0.19, v 0.21 and 0.23 are outside of scikits. Regards, D. From haase at msg.ucsf.edu Wed Apr 22 08:23:13 2009 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 22 Apr 2009 14:23:13 +0200 Subject: [SciPy-user] building svn failed Message-ID: Hi, short note: I'm on debian and I get for a svn checkout: scipy/signal/sigtoolsmodule.c:1635: warning: initialization from incompatible pointer type scipy/signal/sigtoolsmodule.c:161: warning: parameter names (without types) in function declaration scipy/signal/sigtoolsmodule.c:161: error: redefinition of parameter 'NPY_UNUSED' scipy/signal/sigtoolsmodule.c:161: error: previous definition of 'NPY_UNUSED' was here The official version 0.7 builds ok (I have two systems: 32 and 64 bit) - Sebastian Haase From david at ar.media.kyoto-u.ac.jp Wed Apr 22 08:08:44 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 22 Apr 2009 21:08:44 +0900 Subject: [SciPy-user] building svn failed In-Reply-To: References: Message-ID: <49EF08CC.7090102@ar.media.kyoto-u.ac.jp> Sebastian Haase wrote: > Hi, > short note: I'm on debian and I get for a svn checkout: > > scipy/signal/sigtoolsmodule.c:1635: warning: initialization from > incompatible pointer type > scipy/signal/sigtoolsmodule.c:161: warning: parameter names (without > types) in function declaration > scipy/signal/sigtoolsmodule.c:161: error: redefinition of parameter 'NPY_UNUSED' > scipy/signal/sigtoolsmodule.c:161: error: previous definition of > 'NPY_UNUSED' was here > > The official version 0.7 builds ok (I have two systems: 32 and 64 bit) > You need numpy 1.3.0 or above to build scipy svn David From mathieu.dubois at limsi.fr Wed Apr 22 09:09:20 2009 From: mathieu.dubois at limsi.fr (Mathieu Dubois) Date: Wed, 22 Apr 2009 15:09:20 +0200 Subject: [SciPy-user] Save a list of arrays Message-ID: <49EF1700.2040907@limsi.fr> [I have posted this message this morning but apparently it is stuck somewhere - sorry for multi-posting] Hello, I would like to save a list of arrays (each one has a different shape) to a file. For instance: >>> array1 = numpy.ones(2); >>> array2 = numpy.ones(5); >>> array3 = numpy.ones(1000); >>> list = [array1, array2, array3] As my arrays are huge (each one contains several thousands values) I would like a compressed file. numpy.savez would be perfect (because it produces an archive of binary files) but unfortunately numpy.savez(list) doesn't work because savez needs each array individually. So what's the best way to do that? Maybe I could create an archive file in Python (using ZipFile) save each array in a npy file and then write each file in the archive. But then, will numpy.load be able to open the file? Any help appreciated. Kind regards, Mathieu From bnuttall at uky.edu Wed Apr 22 09:35:58 2009 From: bnuttall at uky.edu (Nuttall, Brandon C) Date: Wed, 22 Apr 2009 09:35:58 -0400 Subject: [SciPy-user] Save a list of arrays In-Reply-To: <49EF1700.2040907@limsi.fr> References: <49EF1700.2040907@limsi.fr> Message-ID: I thought that is what pickling does, see http://docs.python.org/library/pickle.html Brandon Nuttall bnuttall at uky.edu (KGS) ________________________________________ From: scipy-user-bounces at scipy.org [scipy-user-bounces at scipy.org] On Behalf Of Mathieu Dubois [mathieu.dubois at limsi.fr] Sent: Wednesday, April 22, 2009 9:09 AM To: SciPy Users List Subject: [SciPy-user] Save a list of arrays [I have posted this message this morning but apparently it is stuck somewhere - sorry for multi-posting] Hello, I would like to save a list of arrays (each one has a different shape) to a file. For instance: >>> array1 = numpy.ones(2); >>> array2 = numpy.ones(5); >>> array3 = numpy.ones(1000); >>> list = [array1, array2, array3] As my arrays are huge (each one contains several thousands values) I would like a compressed file. numpy.savez would be perfect (because it produces an archive of binary files) but unfortunately numpy.savez(list) doesn't work because savez needs each array individually. So what's the best way to do that? Maybe I could create an archive file in Python (using ZipFile) save each array in a npy file and then write each file in the archive. But then, will numpy.load be able to open the file? Any help appreciated. Kind regards, Mathieu _______________________________________________ SciPy-user mailing list SciPy-user at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user From pav at iki.fi Wed Apr 22 09:43:48 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 22 Apr 2009 13:43:48 +0000 (UTC) Subject: [SciPy-user] Save a list of arrays References: <49EF1700.2040907@limsi.fr> Message-ID: Wed, 22 Apr 2009 15:09:20 +0200, Mathieu Dubois kirjoitti: > [I have posted this message this morning but apparently it is stuck > somewhere - sorry for multi-posting] > > Hello, > > I would like to save a list of arrays (each one has a different shape) > to a file. > For instance: > >>> array1 = numpy.ones(2); > >>> array2 = numpy.ones(5); > >>> array3 = numpy.ones(1000); > >>> list = [array1, array2, array3] > > As my arrays are huge (each one contains several thousands values) I > would like a compressed file. > numpy.savez would be perfect (because it produces an archive of binary > files) but unfortunately numpy.savez(list) doesn't work because savez > needs each array individually. > > So what's the best way to do that? savez(filename, *list) The star is Python syntax for unpacking a sequence to arguments. -- Pauli Virtanen From lev at columbia.edu Wed Apr 22 09:43:28 2009 From: lev at columbia.edu (Lev Givon) Date: Wed, 22 Apr 2009 09:43:28 -0400 Subject: [SciPy-user] Save a list of arrays In-Reply-To: <49EF1700.2040907@limsi.fr> References: <49EF1700.2040907@limsi.fr> Message-ID: <20090422134327.GA22591@avicenna.ee.columbia.edu> Received from Mathieu Dubois on Wed, Apr 22, 2009 at 09:09:20AM EDT: > [I have posted this message this morning but apparently it is stuck > somewhere - sorry for multi-posting] > > Hello, > > I would like to save a list of arrays (each one has a different shape) > to a file. > For instance: > >>> array1 = numpy.ones(2); > >>> array2 = numpy.ones(5); > >>> array3 = numpy.ones(1000); > >>> list = [array1, array2, array3] > > As my arrays are huge (each one contains several thousands values) I > would like a compressed file. > numpy.savez would be perfect (because it produces an archive of binary > files) but unfortunately numpy.savez(list) doesn't work because savez > needs each array individually. > > So what's the best way to do that? > > Maybe I could create an archive file in Python (using ZipFile) save each > array in a npy file and then write each file in the archive. But then, > will numpy.load be able to open the file? > > Any help appreciated. > Kind regards, > Mathieu You might find hdf5pickle useful: http://www.elisanet.fi/ptvirtan/software/hdf5pickle (If you would like a compressed file, you can specify a compression filter when creating the HDF5 file in which to store your data.) L.G. From beckers at orn.mpg.de Wed Apr 22 09:53:06 2009 From: beckers at orn.mpg.de (Gabriel Beckers) Date: Wed, 22 Apr 2009 15:53:06 +0200 Subject: [SciPy-user] Save a list of arrays In-Reply-To: <49EF1700.2040907@limsi.fr> References: <49EF1700.2040907@limsi.fr> Message-ID: <1240408386.14242.19.camel@gabriel-desktop> If you want your data saved in a generic way, you could use PyTables ( http://www.pytables.org/ ) to save your arrays in hdf5 format. A script like the one below already does the trick, although it could be greatly improved in terms of compression (look into pytables' CArray). Also if the number of arrays gets really large you have to adapt the approach, and avoid saving everything in root. It would be trivial to write a function to read. You can also look at the contents of the file visually by using ViTables ( http://vitables.berlios.de/ ) Gabriel ------------------------------------------ import numpy import tables def write_arraylist(arraylist, filename): f = tables.openFile(filename, 'w') for i, array in enumerate(arraylist): f.createArray(f.root, 'a%d'%i, array) f.close() array1 = numpy.ones(2); array2 = numpy.ones(5); array3 = numpy.ones(1000); arraylist = [array1, array2, array3] write_arraylist(arraylist, 'testfile.h5') On Wed, 2009-04-22 at 15:09 +0200, Mathieu Dubois wrote: > [I have posted this message this morning but apparently it is stuck > somewhere - sorry for multi-posting] > > Hello, > > I would like to save a list of arrays (each one has a different shape) > to a file. > For instance: > >>> array1 = numpy.ones(2); > >>> array2 = numpy.ones(5); > >>> array3 = numpy.ones(1000); > >>> list = [array1, array2, array3] > > As my arrays are huge (each one contains several thousands values) I > would like a compressed file. > numpy.savez would be perfect (because it produces an archive of binary > files) but unfortunately numpy.savez(list) doesn't work because savez > needs each array individually. > > So what's the best way to do that? > > Maybe I could create an archive file in Python (using ZipFile) save each > array in a npy file and then write each file in the archive. But then, > will numpy.load be able to open the file? > > Any help appreciated. > Kind regards, > Mathieu > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -- Dr. Gabri?l J.L. Beckers Max Planck Institute for Ornithology Department of Behavioural Neurobiology Web: http://www.gbeckers.nl From mathieu.dubois at limsi.fr Wed Apr 22 10:19:07 2009 From: mathieu.dubois at limsi.fr (Mathieu Dubois) Date: Wed, 22 Apr 2009 16:19:07 +0200 Subject: [SciPy-user] Save a list of arrays In-Reply-To: References: <49EF1700.2040907@limsi.fr> Message-ID: <49EF275B.1050900@limsi.fr> Pauli Virtanen wrote: > Wed, 22 Apr 2009 15:09:20 +0200, Mathieu Dubois kirjoitti: > > >> [I have posted this message this morning but apparently it is stuck >> somewhere - sorry for multi-posting] >> >> Hello, >> >> I would like to save a list of arrays (each one has a different shape) >> to a file. >> For instance: >> >>> array1 = numpy.ones(2); >> >>> array2 = numpy.ones(5); >> >>> array3 = numpy.ones(1000); >> >>> list = [array1, array2, array3] >> >> As my arrays are huge (each one contains several thousands values) I >> would like a compressed file. >> numpy.savez would be perfect (because it produces an archive of binary >> files) but unfortunately numpy.savez(list) doesn't work because savez >> needs each array individually. >> >> So what's the best way to do that? >> > > Hello Pauli, > savez(filename, *list) > > The star is Python syntax for unpacking a sequence to arguments Thank you for the tip this opens interesting possibilities. Do you know something that works with named arguments (keyword arguments)? This would allow to set the name of the array in the archive file (by default it's 'arr_0.npy'). Mathieu From rmay31 at gmail.com Wed Apr 22 10:29:40 2009 From: rmay31 at gmail.com (Ryan May) Date: Wed, 22 Apr 2009 09:29:40 -0500 Subject: [SciPy-user] Save a list of arrays In-Reply-To: <49EF275B.1050900@limsi.fr> References: <49EF1700.2040907@limsi.fr> <49EF275B.1050900@limsi.fr> Message-ID: On Wed, Apr 22, 2009 at 9:19 AM, Mathieu Dubois wrote: > Pauli Virtanen wrote: > > Wed, 22 Apr 2009 15:09:20 +0200, Mathieu Dubois kirjoitti: > > > > > >> [I have posted this message this morning but apparently it is stuck > >> somewhere - sorry for multi-posting] > >> > >> Hello, > >> > >> I would like to save a list of arrays (each one has a different shape) > >> to a file. > >> For instance: > >> >>> array1 = numpy.ones(2); > >> >>> array2 = numpy.ones(5); > >> >>> array3 = numpy.ones(1000); > >> >>> list = [array1, array2, array3] > >> > >> As my arrays are huge (each one contains several thousands values) I > >> would like a compressed file. > >> numpy.savez would be perfect (because it produces an archive of binary > >> files) but unfortunately numpy.savez(list) doesn't work because savez > >> needs each array individually. > >> > >> So what's the best way to do that? > >> > > > > > Hello Pauli, > > savez(filename, *list) > > > > The star is Python syntax for unpacking a sequence to arguments > Thank you for the tip this opens interesting possibilities. > > Do you know something that works with named arguments (keyword arguments)? > This would allow to set the name of the array in the archive file (by > default it's 'arr_0.npy'). > A dictionary and two stars: arrays = {'a':a, 'b':b, 'c':c} savez(filename, **arrays) Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma -------------- next part -------------- An HTML attachment was scrubbed... URL: From beckers at orn.mpg.de Wed Apr 22 10:31:06 2009 From: beckers at orn.mpg.de (Gabriel Beckers) Date: Wed, 22 Apr 2009 16:31:06 +0200 Subject: [SciPy-user] Save a list of arrays In-Reply-To: <20090422134327.GA22591@avicenna.ee.columbia.edu> References: <49EF1700.2040907@limsi.fr> <20090422134327.GA22591@avicenna.ee.columbia.edu> Message-ID: <1240410666.14242.21.camel@gabriel-desktop> On Wed, 2009-04-22 at 09:43 -0400, Lev Givon wrote: > You might find hdf5pickle useful: > > http://www.elisanet.fi/ptvirtan/software/hdf5pickle Extremely useful link, thanks! Gabriel From camejia at raytheon.com Wed Apr 22 12:50:17 2009 From: camejia at raytheon.com (Christopher A Mejia) Date: Wed, 22 Apr 2009 09:50:17 -0700 Subject: [SciPy-user] Using savemat with (nested) NumPy record arrays? Message-ID: Hi, I'm trying to write a NumPy record array using the savemat function, using the format='5' default, but I am not having much success. Here's an example using a NumPy record array defined in the NumPy User Guide: ----------------------------------------- >>> import numpy as np >>> x = np.zeros(3, dtype=[(?x?,?f4?),(?y?,np.float32),(?value?,?f4?,(2,2))]) SyntaxError: invalid syntax >>> x = np.zeros(3, dtype=[('x','f4'),('y',np.float32),('value','f4',(2,2))]) >>> x array([(0.0, 0.0, [[0.0, 0.0], [0.0, 0.0]]), (0.0, 0.0, [[0.0, 0.0], [0.0, 0.0]]), (0.0, 0.0, [[0.0, 0.0], [0.0, 0.0]])], dtype=[('x', '>> from scipy.io.matlab.mio import savemat >>> savemat('record_array_test.mat', {'x': x}) Traceback (most recent call last): File "", line 1, in savemat('record_array_test.mat', {'x': x}) File "C:\Python25\lib\site-packages\scipy\io\matlab\mio.py", line 159, in savemat MW.put_variables(mdict) File "C:\Python25\lib\site-packages\scipy\io\matlab\mio5.py", line 974, in put_variables mat_writer.write() File "C:\Python25\lib\site-packages\scipy\io\matlab\mio5.py", line 736, in write self.arr = self.arr.astype('f8') ValueError: setting an array element with a sequence. >>> ----------------------------------------- Actually, what I'd like to do is to be able to handle an arbitrarily nested record array, as in: ----------------------------------------- >>> spam = np.zeros(2, dtype=[('a','f4'), ('b', [('x', 'f4'), ('y', 'f4', (2,2))])]) >>> spam array([(0.0, (0.0, [[0.0, 0.0], [0.0, 0.0]])), (0.0, (0.0, [[0.0, 0.0], [0.0, 0.0]]))], dtype=[('a', '>> savemat('record_array_test2.mat', {'spam': spam}) Traceback (most recent call last): File "", line 1, in savemat('record_array_test2.mat', {'spam': spam}) File "C:\Python25\lib\site-packages\scipy\io\matlab\mio.py", line 159, in savemat MW.put_variables(mdict) File "C:\Python25\lib\site-packages\scipy\io\matlab\mio5.py", line 974, in put_variables mat_writer.write() File "C:\Python25\lib\site-packages\scipy\io\matlab\mio5.py", line 736, in write self.arr = self.arr.astype('f8') ValueError: setting an array element with a sequence. ----------------------------------------- As you can see, I get the same error for the nested case. I know what I am trying to do is possible, because I can generate my desired nested structure array in MATLAB, then do a "round-trip" loadmat(,struct_as_record=True) and savemat() to get back the same thing in MATLAB. However, I cannot seem to reverse engineer what loadmat(,struct_as_record=True) does to create the NumPy record array. Two differences appear to be that the dtype definition created by loadmat(,struct_as_record=True) does not print out as being nested, it just shows a '|04' type (set by the keyword "object"); also scalars and one-dimensional vectors appear to be upconverted to 2-d matrices. Perhaps someone has a routine that I can use to pre-process my nested record array so it works with savemat? FYI, I'm using Python 2.5.4, NumPy 1.2.1 and SciPy 0.7.0. Thanks in advance for any help, --Chris ( P.S. I apologize in advance if this post shows up twice...my first attempt seems to have gotten lost.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From weixie2007 at gmail.com Wed Apr 22 13:12:18 2009 From: weixie2007 at gmail.com (Wei Xie) Date: Wed, 22 Apr 2009 12:12:18 -0500 Subject: [SciPy-user] Internal parameter changes while appending scipy.stats function to a list Message-ID: <35d8a30f0904221012r19f4686cjba3c65f60decc814@mail.gmail.com> Dear Scipy users: While playing SCIPY (numpy version 1.3.0, scipy version 0.7.0, windows), it seems sometimes the internal parameters change quite unexpectedly. For example, >>> from scipy.stats import norm >>> from numpy.random import normal >>> e = [] >>> e.append((1,lambda q:norm.ppf(q,loc=1,scale=0.1),lambda sz=1:normal(loc=1,scale=0.1,size=sz))) >>> e.append((10,lambda q:norm.ppf(q,loc=10,scale=0.1),lambda sz=1:normal(loc=10,scale=0.1,size=sz))) >>> e.append((100,lambda q:norm.ppf(q,loc=100,scale=0.1),lambda sz=1:normal(loc=100,scale=0.1,size=sz))) This one works as >>> e[0][1](0.5) 1.0 >>> e[1][1](0.5) 10.0 >>> e[2][1](0.5) 100.0 suggests the internal location parameters still holds the value as assigned. On the other hand: >>> d = [] >>> for i in (1,10,100): d.append((i,lambda q:norm.ppf(q,loc=i,scale=0.1),lambda sz=1:normal(loc=i,scale=0.1,size=sz))) does not work, as >>> d[0][1](0.5) 100.0 >>> d[1][1](0.5) 100.0 >>> d[2][1](0.5) 100.0 suggess the internal loc parameters to 100.0, i.e., the last one. I am a bit puzzled by this finding as I am expecting lambda expression will give me a closure and therefore the internal parameter loc and scale should not change. I am looking forward to your input. Thanks. Wei -------------- next part -------------- An HTML attachment was scrubbed... URL: From mathieu.dubois at limsi.fr Wed Apr 22 13:14:29 2009 From: mathieu.dubois at limsi.fr (Mathieu Dubois) Date: Wed, 22 Apr 2009 19:14:29 +0200 Subject: [SciPy-user] Save a list of arrays In-Reply-To: References: <49EF1700.2040907@limsi.fr> <49EF275B.1050900@limsi.fr> Message-ID: <49EF5075.1050209@limsi.fr> Ryan May wrote: > On Wed, Apr 22, 2009 at 9:19 AM, Mathieu Dubois > > wrote: > > Pauli Virtanen wrote: > > Wed, 22 Apr 2009 15:09:20 +0200, Mathieu Dubois kirjoitti: > > > > > >> [I have posted this message this morning but apparently it is stuck > >> somewhere - sorry for multi-posting] > >> > >> Hello, > >> > >> I would like to save a list of arrays (each one has a different > shape) > >> to a file. > >> For instance: > >> >>> array1 = numpy.ones(2); > >> >>> array2 = numpy.ones(5); > >> >>> array3 = numpy.ones(1000); > >> >>> list = [array1, array2, array3] > >> > >> As my arrays are huge (each one contains several thousands > values) I > >> would like a compressed file. > >> numpy.savez would be perfect (because it produces an archive of > binary > >> files) but unfortunately numpy.savez(list) doesn't work because > savez > >> needs each array individually. > >> > >> So what's the best way to do that? > >> > > > > > Hello Pauli, > > savez(filename, *list) > > > > The star is Python syntax for unpacking a sequence to arguments > Thank you for the tip this opens interesting possibilities. > > Do you know something that works with named arguments (keyword > arguments)? > This would allow to set the name of the array in the archive file (by > default it's 'arr_0.npy'). > > > A dictionary and two stars: > > arrays = {'a':a, 'b':b, 'c':c} > savez(filename, **arrays) Hello Ryan, Thank you very much this is exactly what I wanted to do. Python is very powerful. Thanks also to Lev and Gabriel but using PyTable and HDF5 would have forced me to changed all my scripts... Kind regards, Mathieu From josef.pktd at gmail.com Wed Apr 22 15:23:16 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 22 Apr 2009 15:23:16 -0400 Subject: [SciPy-user] Internal parameter changes while appending scipy.stats function to a list In-Reply-To: <35d8a30f0904221012r19f4686cjba3c65f60decc814@mail.gmail.com> References: <35d8a30f0904221012r19f4686cjba3c65f60decc814@mail.gmail.com> Message-ID: <1cd32cbb0904221223l230505fbm7b6007d2ab715920@mail.gmail.com> On Wed, Apr 22, 2009 at 1:12 PM, Wei Xie wrote: > Dear Scipy users: > > While playing SCIPY (numpy version 1.3.0, scipy version 0.7.0, windows), it > seems sometimes the internal parameters change quite unexpectedly.? For > example, > >>>> from scipy.stats import norm >>>> from numpy.random import normal > >>>> e = [] >>>> e.append((1,lambda q:norm.ppf(q,loc=1,scale=0.1),lambda >>>> sz=1:normal(loc=1,scale=0.1,size=sz))) >>>> e.append((10,lambda q:norm.ppf(q,loc=10,scale=0.1),lambda >>>> sz=1:normal(loc=10,scale=0.1,size=sz))) >>>> e.append((100,lambda q:norm.ppf(q,loc=100,scale=0.1),lambda >>>> sz=1:normal(loc=100,scale=0.1,size=sz))) > > This one works as >>>> e[0][1](0.5) > 1.0 >>>> e[1][1](0.5) > 10.0 >>>> e[2][1](0.5) > 100.0 > > suggests the internal location parameters still holds the value as assigned. > > On the other hand: >>>> d = [] >>>> for i in (1,10,100): d.append((i,lambda >>>> q:norm.ppf(q,loc=i,scale=0.1),lambda sz=1:normal(loc=i,scale=0.1,size=sz))) > > does not work, as > >>>> d[0][1](0.5) > 100.0 >>>> d[1][1](0.5) > 100.0 >>>> d[2][1](0.5) > 100.0 > > suggess the internal loc parameters to 100.0, i.e., the last one. > > I am a bit puzzled by this finding as I am expecting lambda expression will > give me a closure and therefore the internal parameter loc and scale should > not change. > > I am looking forward to your input.? Thanks. > > Wei > That's a python question not related to stats, see minimal example below. I don't remember the exact arguments on binding of the variables, but this is a recurrent discussion on the python mailing list. You can search the python list. Josef li = [] for i in range(5): li.append(lambda (x): x+i) print li[0](1) print li[3](1) for ff in li: print ff(1) def nest(ii): return lambda (x): x+ii li2 = [] for i in range(5): li2.append(nest(i)) print li2[0](1) print li2[3](1) for ff in li2: print ff(1) this prints: 5 5 5 5 5 5 5 1 4 1 2 3 4 5 From oanjao at yahoo.com Wed Apr 22 18:31:08 2009 From: oanjao at yahoo.com (Craig Finch) Date: Wed, 22 Apr 2009 15:31:08 -0700 (PDT) Subject: [SciPy-user] Building Python, Numpy and Scipy with Intel compilers and MKL Message-ID: <858048.93360.qm@web30404.mail.mud.yahoo.com> I'm trying to build Python, Numpy, and SciPy on a Red Hat system using the Intel Fortran, C, and C++ compilers, along with the Intel Math Kernel Library. This is proving to be difficult--I've tried everything I could find on the web, and it's still not working (everything works fine if I use GCC). I would really appreciate any suggestions. I can get Python and Numpy working correctly after compiling with Intel compilers, but I can't import anything from SciPy without getting an "undefined symbol" error. According to what I've read, this can happen when gfortran is used to compile some library (such as BLAS) while Intel fortran is used to compile SciPy. I don't think this is my problem, because I'm only using the Intel Math Kernel Libraries. Is this correct? Here's my latest attempt at installation: Python install: CC='icc' CXX='icc' LDFLAGS='-L/opt/intel/cce/10.1.008/lib' CFLAGS='-I/opt/intel/cce/10.1.008/include' CXXFLAGS='-I/opt/intel/cce/10.1.008/include' ./configure -prefix=/home/me Numpy install: python setup.py config --compiler=intel --fcompiler=intel install --home=/home/me Scipy install: python setup.py config --compiler=intel --fcompiler=intel build_ext --fcompiler=intel -L/opt/intel/cce/10.1.008/lib install --home=/home/me Here's something that bothers me: when I start my Python interpreter which was compiled with icc, this is what I see: Python 2.5.4 (r254:67916, Apr 22 2009, 15:52:10) [GCC 4.1.1 20070105 (Red Hat 4.1.1-52)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> Why does the interpreter think it's built with GCC? Can that be causing problems with SciPy? How can I ensure that only Intel libraries are used? Craig From david at ar.media.kyoto-u.ac.jp Wed Apr 22 22:08:39 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 23 Apr 2009 11:08:39 +0900 Subject: [SciPy-user] Building Python, Numpy and Scipy with Intel compilers and MKL In-Reply-To: <858048.93360.qm@web30404.mail.mud.yahoo.com> References: <858048.93360.qm@web30404.mail.mud.yahoo.com> Message-ID: <49EFCDA7.8000001@ar.media.kyoto-u.ac.jp> Hi Craig, Craig Finch wrote: > I'm trying to build Python, Numpy, and SciPy on a Red Hat system using the Intel Fortran, C, and C++ compilers, along with the Intel Math Kernel Library. This is proving to be difficult--I've tried everything I could find on the web, and it's still not working (everything works fine if I use GCC). I would really appreciate any suggestions. > > I can get Python and Numpy working correctly after compiling with Intel compilers, but I can't import anything from SciPy without getting an "undefined symbol" error. According to what I've read, this can happen when gfortran is used to compile some library (such as BLAS) while Intel fortran is used to compile SciPy. I don't think this is my problem, because I'm only using the Intel Math Kernel Libraries. Is this correct? > You need to be much more precise than that. You need to give us the exact missing symbols, and ideally the build log of both numpy and scipy. > Here's something that bothers me: when I start my Python interpreter which was compiled with icc, this is what I see: > Python 2.5.4 (r254:67916, Apr 22 2009, 15:52:10) > [GCC 4.1.1 20070105 (Red Hat 4.1.1-52)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > > > Why > does the interpreter think it's built with GCC? Because it almost certainly is. You are not running the python you think you are running for some reason. You don't need to build python with intel compiler, BTW. cheers, David From camejia at Raytheon.com Wed Apr 22 23:34:53 2009 From: camejia at Raytheon.com (Christopher A Mejia) Date: Wed, 22 Apr 2009 20:34:53 -0700 Subject: [SciPy-user] Using savemat with (nested) NumPy record arrays? In-Reply-To: Message-ID: Hi, Well, it turns out I found a solution myself, so I'll share that, but I still need some further help... What I found was: 1. By declaring the dtype as "object" (no quotes) instead of a nested dtype, I got past the part where the Python call to savemat was breaking. 2. However, I didn't get any data showing up in MATLAB after loading my file, unless I made sure that all of the arrays had a shape with at least one dimension. Here is an example of code that works: -------------------------------- >>> import numpy as np >>> x = np.zeros((1,), dtype=[('a', object)]) >>> x[0]['a'] = np.zeros((1,)) >>> savemat('record_array_test3.mat', {'x': x}) -------------------------------- The problem I'm running into now is that the savemat function is too slow. The top level of data I'm trying to save is an array of structures. It seems that the time for savemat increases exponentially as the number of records in this structure array increases. Is there a better way to organize the storage of data into savemat, or is there a simple way to modify savemat to speed it up? Other approaches? I'm trying to keep all of the "metadata" (i.e. field names) in Python available to MATLAB. I got the field names into Python using SWIG and C++ code. I've looked into PyTables but didn't like the way the tables loaded into MATLAB. Thanks in advance, --Chris Christopher A Mejia Sent by: scipy-user-bounces at scipy.org 04/22/2009 09:51 AM Please respond to SciPy Users List To scipy-user at scipy.org cc Subject [SciPy-user] Using savemat with (nested) NumPy record arrays? Hi, I'm trying to write a NumPy record array using the savemat function, using the format='5' default, but I am not having much success. Here's an example using a NumPy record array defined in the NumPy User Guide: ----------------------------------------- >>> import numpy as np >>> x = np.zeros(3, dtype=[(?x?,?f4?),(?y?,np.float32),(?value?,?f4?,(2,2))]) SyntaxError: invalid syntax >>> x = np.zeros(3, dtype=[('x','f4'),('y',np.float32),('value','f4',(2,2))]) >>> x array([(0.0, 0.0, [[0.0, 0.0], [0.0, 0.0]]), (0.0, 0.0, [[0.0, 0.0], [0.0, 0.0]]), (0.0, 0.0, [[0.0, 0.0], [0.0, 0.0]])], dtype=[('x', '>> from scipy.io.matlab.mio import savemat >>> savemat('record_array_test.mat', {'x': x}) Traceback (most recent call last): File "", line 1, in savemat('record_array_test.mat', {'x': x}) File "C:\Python25\lib\site-packages\scipy\io\matlab\mio.py", line 159, in savemat MW.put_variables(mdict) File "C:\Python25\lib\site-packages\scipy\io\matlab\mio5.py", line 974, in put_variables mat_writer.write() File "C:\Python25\lib\site-packages\scipy\io\matlab\mio5.py", line 736, in write self.arr = self.arr.astype('f8') ValueError: setting an array element with a sequence. >>> ----------------------------------------- Actually, what I'd like to do is to be able to handle an arbitrarily nested record array, as in: ----------------------------------------- >>> spam = np.zeros(2, dtype=[('a','f4'), ('b', [('x', 'f4'), ('y', 'f4', (2,2))])]) >>> spam array([(0.0, (0.0, [[0.0, 0.0], [0.0, 0.0]])), (0.0, (0.0, [[0.0, 0.0], [0.0, 0.0]]))], dtype=[('a', '>> savemat('record_array_test2.mat', {'spam': spam}) Traceback (most recent call last): File "", line 1, in savemat('record_array_test2.mat', {'spam': spam}) File "C:\Python25\lib\site-packages\scipy\io\matlab\mio.py", line 159, in savemat MW.put_variables(mdict) File "C:\Python25\lib\site-packages\scipy\io\matlab\mio5.py", line 974, in put_variables mat_writer.write() File "C:\Python25\lib\site-packages\scipy\io\matlab\mio5.py", line 736, in write self.arr = self.arr.astype('f8') ValueError: setting an array element with a sequence. ----------------------------------------- As you can see, I get the same error for the nested case. I know what I am trying to do is possible, because I can generate my desired nested structure array in MATLAB, then do a "round-trip" loadmat(,struct_as_record=True) and savemat() to get back the same thing in MATLAB. However, I cannot seem to reverse engineer what loadmat(,struct_as_record=True) does to create the NumPy record array. Two differences appear to be that the dtype definition created by loadmat(,struct_as_record=True) does not print out as being nested, it just shows a '|04' type (set by the keyword "object"); also scalars and one-dimensional vectors appear to be upconverted to 2-d matrices. Perhaps someone has a routine that I can use to pre-process my nested record array so it works with savemat? FYI, I'm using Python 2.5.4, NumPy 1.2.1 and SciPy 0.7.0. Thanks in advance for any help, --Chris ( P.S. I apologize in advance if this post shows up twice...my first attempt seems to have gotten lost.) _______________________________________________ SciPy-user mailing list SciPy-user at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Thu Apr 23 00:09:22 2009 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 22 Apr 2009 21:09:22 -0700 Subject: [SciPy-user] Using savemat with (nested) NumPy record arrays? In-Reply-To: References: Message-ID: <1e2af89e0904222109x66060a8ew7817701ee224e19c@mail.gmail.com> Hi, > 2. ?However, I didn't get any data showing up in MATLAB after loading my > file, unless I made sure that all of the arrays had a shape with at least > one dimension. There was a bug in scipy 0.7 release - do you have the latest SVN? Does that work? > The problem I'm running into now is that the savemat function is too slow. > ?The top level of data I'm trying to save is an array of structures. ?It > seems that the time for savemat increases exponentially as the number of > records in this structure array increases. ?Is there a better way to > organize the storage of data into savemat, or is there a simple way to > modify savemat to speed it up? Yes, that's a known problem as well. The hope is that the current SVN implementation is a lot faster. I'd be very glad of your feedback about that... Thanks, Matthew From bevan.jenkins at ew.govt.nz Thu Apr 23 00:24:02 2009 From: bevan.jenkins at ew.govt.nz (Bevan Jenkins) Date: Thu, 23 Apr 2009 04:24:02 +0000 (UTC) Subject: [SciPy-user] plot curve of a series of equations Message-ID: Hello, I have a series of equations that describe a curve. Each equation is only valid for a range of x values. For example: Equation Upperlimit Lowerlimit 24 + 18.75 * x + 4.79616e-013 * x ** 2 + 31.25 * x ** 3 7.800 7.600 28 + 22.5 * x + 5.68434e-013 * x ** 2 + 0 * x ** 3 8.000 7.800 There is a lower limit for eqn1, after that the upper limit of the previous eqn is lower of the next. >From (X) scale linear, To (Y) scale linear y = A0 + A1(x) + A2(x)2 + A3(x)3 where: x= value ? LL LL = the lower limit of the applicable equation I would like to generate a sequence of values (linspace(min(lowerlimit),max (upperlimit)), work out x (x=value-approriate lower limit) and then using the appropriate equation generate the y values. Below is the answer I am looking for Value x y 7.600 0 24.000 7.650 0.05 24.941 7.700 0.1 25.906 7.750 0.15 26.918 7.800 0.2 28.000 7.850 0.050 29.125 7.900 0.100 30.25 7.950 0.150 31.375 8.000 0.200 32.5 Thanks for any help, Bevan Jenkins From matthieu.brucher at gmail.com Thu Apr 23 03:38:53 2009 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 23 Apr 2009 09:38:53 +0200 Subject: [SciPy-user] Building Python, Numpy and Scipy with Intel compilers and MKL In-Reply-To: <858048.93360.qm@web30404.mail.mud.yahoo.com> References: <858048.93360.qm@web30404.mail.mud.yahoo.com> Message-ID: Hi, I've done the same on my configuration at work, and I didn't have a glitch (once I got the appropriate MKL version, don't forget that any MKL > 10.0.2 will not work). Don't forget to install your new Python in a local folder and then to export this local/bin in your PATH and local/lib in LD_LIBRARY_PATH. Matthieu 2009/4/23 Craig Finch : > > ?I'm trying to build Python, Numpy, and SciPy on a Red Hat system using the Intel Fortran, C, and C++ compilers, along with the Intel Math Kernel Library. ?This is proving to be difficult--I've tried everything I could find on the web, and it's still not working (everything works fine if I use GCC). ?I would really appreciate any suggestions. > > I can get Python and Numpy working correctly after compiling with Intel compilers, but I can't import anything from SciPy without getting an "undefined symbol" error. ?According to what I've read, this can happen when gfortran is used to compile some library (such as BLAS) while Intel fortran is used to compile SciPy. ?I don't think this is my problem, because I'm only using the Intel Math Kernel Libraries. ?Is this correct? > > Here's my latest attempt at installation: > Python install: > CC='icc' > CXX='icc' LDFLAGS='-L/opt/intel/cce/10.1.008/lib' CFLAGS='-I/opt/intel/cce/10.1.008/include' CXXFLAGS='-I/opt/intel/cce/10.1.008/include' ./configure -prefix=/home/me > > Numpy install: > python setup.py config --compiler=intel --fcompiler=intel install --home=/home/me > > Scipy install: > python > setup.py config --compiler=intel --fcompiler=intel build_ext > --fcompiler=intel -L/opt/intel/cce/10.1.008/lib ?install --home=/home/me > > Here's something that bothers me: when I start my Python interpreter which was compiled with icc, this is what I see: > Python 2.5.4 (r254:67916, Apr 22 2009, 15:52:10) > [GCC 4.1.1 20070105 (Red Hat 4.1.1-52)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> > > Why > does the interpreter think it's built with GCC? ?Can that be causing > problems with SciPy? ?How can I ensure that only Intel libraries are > used? > > ? ?Craig > > > > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From rcsqtc at iqac.csic.es Thu Apr 23 04:26:00 2009 From: rcsqtc at iqac.csic.es (Ramon Crehuet) Date: Thu, 23 Apr 2009 10:26:00 +0200 Subject: [SciPy-user] Numpy in parallel Message-ID: <49F02618.5080707@iqac.csic.es> Dear all, I would like to use the 4 processors in my AMD Phenom machine to do some calcualtions with Numpy. According to: http://www.scipy.org/ParallelProgramming This can be achieved if numpy is compiled using a BLAS that is able to use multi-cores. I have installed the rpm numpy that comes with my Opensuse 11.0 distribution. Apparently only one processor is used when I calculate things like: c=numpy.dot(a,b) d=numpy.linalg.inv(c) My first question is: is this a good choice? Is this Numpy optimized for my arquitecture or I could do much better compiling ATLAS and Numpy from scratch? Or is Intel MKL better? My second question is: I have to create, many times, a large array of normally distributed vectors (the stochastic term in a Langevin dynamics run). Do you know if numpy.random.randn will be able to use multi-cores? Thanks in advance, Ramon From gruben at bigpond.net.au Thu Apr 23 05:39:59 2009 From: gruben at bigpond.net.au (Gary Ruben) Date: Thu, 23 Apr 2009 19:39:59 +1000 Subject: [SciPy-user] plot curve of a series of equations In-Reply-To: References: Message-ID: <49F0376F.9070003@bigpond.net.au> Like this? This basically just what you said, so I'm not sure what your exact question is. Is it just the use of eval that you weren't clear on? import numpy as np import matplotlib.pyplot as plt eqs = [ ['24 + 18.75*x + 4.79616e-013*x**2 + 31.25*x**3', 7.800, 7.600], ['28 + 22.5*x + 5.68434e-013*x**2 + 0*x**3', 8.000, 7.800], ] for i in eqs: xr = np.linspace(i[2], i[1], 10) x = xr - i[2] y = eval(i[0]) plt.plot(xr,y) plt.show() Gary R. Bevan Jenkins wrote: > Hello, > > I have a series of equations that describe a curve. Each equation is only > valid for a range of x values. For example: > > Equation Upperlimit Lowerlimit > 24 + 18.75 * x + 4.79616e-013 * x ** 2 + 31.25 * x ** 3 7.800 7.600 > 28 + 22.5 * x + 5.68434e-013 * x ** 2 + 0 * x ** 3 8.000 7.800 > > There is a lower limit for eqn1, after that the upper limit of the previous > eqn is lower of the next. > > From (X) scale linear, To (Y) scale linear > y = A0 + A1(x) + A2(x)2 + A3(x)3 > where: x= value ? LL > > LL = the lower limit of the applicable equation > > I would like to generate a sequence of values (linspace(min(lowerlimit),max > (upperlimit)), work out x (x=value-approriate lower limit) and then using the > appropriate equation generate the y values. > Below is the answer I am looking for > > Value x y > 7.600 0 24.000 > 7.650 0.05 24.941 > 7.700 0.1 25.906 > 7.750 0.15 26.918 > 7.800 0.2 28.000 > 7.850 0.050 29.125 > 7.900 0.100 30.25 > 7.950 0.150 31.375 > 8.000 0.200 32.5 > > Thanks for any help, > > Bevan Jenkins From gruben at bigpond.net.au Thu Apr 23 05:40:13 2009 From: gruben at bigpond.net.au (Gary Ruben) Date: Thu, 23 Apr 2009 19:40:13 +1000 Subject: [SciPy-user] plot curve of a series of equations In-Reply-To: References: Message-ID: <49F0377D.8080201@bigpond.net.au> Like this? This basically just what you said, so I'm not sure what your exact question is. Is it just the use of eval that you weren't clear on? import numpy as np import matplotlib.pyplot as plt eqs = [ ['24 + 18.75*x + 4.79616e-013*x**2 + 31.25*x**3', 7.800, 7.600], ['28 + 22.5*x + 5.68434e-013*x**2 + 0*x**3', 8.000, 7.800], ] for i in eqs: xr = np.linspace(i[2], i[1], 10) x = xr - i[2] y = eval(i[0]) plt.plot(xr,y) plt.show() Gary R. Bevan Jenkins wrote: > Hello, > > I have a series of equations that describe a curve. Each equation is only > valid for a range of x values. For example: > > Equation Upperlimit Lowerlimit > 24 + 18.75 * x + 4.79616e-013 * x ** 2 + 31.25 * x ** 3 7.800 7.600 > 28 + 22.5 * x + 5.68434e-013 * x ** 2 + 0 * x ** 3 8.000 7.800 > > There is a lower limit for eqn1, after that the upper limit of the previous > eqn is lower of the next. > > From (X) scale linear, To (Y) scale linear > y = A0 + A1(x) + A2(x)2 + A3(x)3 > where: x= value ? LL > > LL = the lower limit of the applicable equation > > I would like to generate a sequence of values (linspace(min(lowerlimit),max > (upperlimit)), work out x (x=value-approriate lower limit) and then using the > appropriate equation generate the y values. > Below is the answer I am looking for > > Value x y > 7.600 0 24.000 > 7.650 0.05 24.941 > 7.700 0.1 25.906 > 7.750 0.15 26.918 > 7.800 0.2 28.000 > 7.850 0.050 29.125 > 7.900 0.100 30.25 > 7.950 0.150 31.375 > 8.000 0.200 32.5 > > Thanks for any help, > > Bevan Jenkins From fccoelho at gmail.com Thu Apr 23 09:31:34 2009 From: fccoelho at gmail.com (Flavio Coelho) Date: Thu, 23 Apr 2009 14:31:34 +0100 Subject: [SciPy-user] bug in stats.randint Message-ID: Hi, I stumbled upon something I think is a bug in scipy: In [4]: stats.randint(1.,15.).ppf([.1,.2,.3,.4,.5]) Out[4]: array([ 2., 3., 5., 6., 7.]) When you pass float arguments to stats.randint and then call the ppf method, you get an array of floats, which clearly wrong. The rvs method doesn't display this bug so I think is a matter of poor type checking in the ppf implementation... -- --- Fl?vio Code?o Coelho -------------- next part -------------- An HTML attachment was scrubbed... URL: From oanjao at yahoo.com Thu Apr 23 12:40:08 2009 From: oanjao at yahoo.com (Craig Finch) Date: Thu, 23 Apr 2009 09:40:08 -0700 (PDT) Subject: [SciPy-user] ***[Possible UCE]*** Re: Building Python, Numpy and Scipy with Intel compilers and MKL In-Reply-To: References: <858048.93360.qm@web30404.mail.mud.yahoo.com> Message-ID: <875112.67646.qm@web30402.mail.mud.yahoo.com> Matthieu and David--thank you for your replies. Today I went back and did two installs in different locations, one with icc and one with gcc. Looking at the two side-by-side, I noticed something I didn't catch yesterday: Here's the startup message from Python built with gcc: Python 2.5.4 (r254:67916, Apr 22 2009, 15:52:10) [GCC 4.1.1 20070105 (Red Hat 4.1.1-52)] on linux2 Here's the startup message from Python built with icc: Python 2.5.4 (r254:67916, Apr 23 2009, 10:48:58) [GCC Intel(R) C++ gcc 4.1 mode] on linux2 I didn't notice the difference until I had them side-by-side (in fact, I pasted the wrong one into my first email). The Python I built with icc is, in fact, built with icc. So, that solves the mystery of the Python interpreter. Numpy (1.3.0) works fine, whether I compile it with gcc or icc. Here is a snippet of output to show that it is actually being built with icc and linked with MKL 10.0.1: icc -O2 -fPIC -shared build/temp.linux-x86_64-2.5/numpy/linalg/lapack_litemodule.o build/temp.linux-x86_64-2.5/numpy/linalg/python_xerbla.o -L/opt/intel/mkl/10.0.1.014/lib/em64t/ -Lbuild/temp.linux-x86_64-2.5 -lmkl_lapack -lmkl -lguide -lpthread -o build/lib.linux-x86_64-2.5/numpy/linalg/lapack_lite.so Scipy (0.7.0) is giving me a compilation error today when I attempt to build with icc, which is different from yesterday. I had to modify numpy/distutiles/fcompiler/intel.py to manually set the version to 10.1, and I commented out all the options in method get_flags_arch which are deprecated or obsolete for modern intel compilers. The Scipy build proceeds until it tries to build scipy.interpolate.dfitpack. Here is the error I get: icc -O2 -fPIC -shared build/temp.linux-x86_64-2.5/scipy/interpolate/src/_fitpackmodule.o -Lbuild/temp.linux-x86_64-2.5 -lfitpack -o build/lib.linux-x86_64-2.5/scipy/interpolate/_fitpack.so building 'scipy.interpolate.dfitpack' extension error: extension 'scipy.interpolate.dfitpack' has Fortran sources but no Fortran compiler found Further, there are several warnings like this, and I don't know how important they are: building 'scipy.fftpack._fftpack' extension warning: build_ext: extension 'scipy.fftpack._fftpack' has Fortran libraries but no Fortran linker found, using default linker I have saved the entire output of all these builds, but the Scipy build log output is about 300k and I'm open to suggestions as to the best way to share it. For starters, here is a snippet showing the compiler configuration: building 'statlib' library compiling Fortran sources Fortran f77 compiler: /apps/intel/fce/10.1.008/bin/ifort -72 -w90 -w95 -fPIC -cm -O3 -unroll Fortran f90 compiler: /apps/intel/fce/10.1.008/bin/ifort -FR -fPIC -cm -O3 -unroll Fortran fix compiler: /apps/intel/fce/10.1.008/bin/ifort -FI -fPIC -cm -O3 -unroll creating build/temp.linux-x86_64-2.5/scipy/stats creating build/temp.linux-x86_64-2.5/scipy/stats/statlib compile options: '-I/home/me/Python_Test/lib/python/numpy/core/include -c' ifort:f77: scipy/stats/statlib/swilk.f scipy/stats/statlib/swilk.f(85): (col. 15) remark: LOOP WAS VECTORIZED. scipy/stats/statlib/swilk.f(120): (col. 10) remark: PARTIAL LOOP WAS VECTORIZED. ifort:f77: scipy/stats/statlib/spearman.f scipy/stats/statlib/spearman.f(52): (col. 12) remark: LOOP WAS VECTORIZED. ifort:f77: scipy/stats/statlib/ansari.f Further suggestions would be appreciated. Craig ----- Original Message ---- From: Matthieu Brucher To: Craig Finch ; SciPy Users List Sent: Thursday, April 23, 2009 3:38:53 AM Subject: ***[Possible UCE]*** Re: [SciPy-user] Building Python, Numpy and Scipy with Intel compilers and MKL Hi, I've done the same on my configuration at work, and I didn't have a glitch (once I got the appropriate MKL version, don't forget that any MKL > 10.0.2 will not work). Don't forget to install your new Python in a local folder and then to export this local/bin in your PATH and local/lib in LD_LIBRARY_PATH. Matthieu 2009/4/23 Craig Finch : > > I'm trying to build Python, Numpy, and SciPy on a Red Hat system using the Intel Fortran, C, and C++ compilers, along with the Intel Math Kernel Library. This is proving to be difficult--I've tried everything I could find on the web, and it's still not working (everything works fine if I use GCC). I would really appreciate any suggestions. > > I can get Python and Numpy working correctly after compiling with Intel compilers, but I can't import anything from SciPy without getting an "undefined symbol" error. According to what I've read, this can happen when gfortran is used to compile some library (such as BLAS) while Intel fortran is used to compile SciPy. I don't think this is my problem, because I'm only using the Intel Math Kernel Libraries. Is this correct? > > Here's my latest attempt at installation: > Python install: > CC='icc' > CXX='icc' LDFLAGS='-L/opt/intel/cce/10.1.008/lib' CFLAGS='-I/opt/intel/cce/10.1.008/include' CXXFLAGS='-I/opt/intel/cce/10.1.008/include' ./configure -prefix=/home/me > > Numpy install: > python setup.py config --compiler=intel --fcompiler=intel install --home=/home/me > > Scipy install: > python > setup.py config --compiler=intel --fcompiler=intel build_ext > --fcompiler=intel -L/opt/intel/cce/10.1.008/lib install --home=/home/me > > Here's something that bothers me: when I start my Python interpreter which was compiled with icc, this is what I see: > Python 2.5.4 (r254:67916, Apr 22 2009, 15:52:10) > [GCC 4.1.1 20070105 (Red Hat 4.1.1-52)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> > > Why > does the interpreter think it's built with GCC? Can that be causing > problems with SciPy? How can I ensure that only Intel libraries are > used? > > Craig > > > > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From josef.pktd at gmail.com Thu Apr 23 14:37:01 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 23 Apr 2009 14:37:01 -0400 Subject: [SciPy-user] plot curve of a series of equations In-Reply-To: References: Message-ID: <1cd32cbb0904231137g5bf81a17i23d3c9b84b507843@mail.gmail.com> On Thu, Apr 23, 2009 at 12:24 AM, Bevan Jenkins wrote: > Hello, > > I have a series of equations that describe a curve. ?Each equation is only > valid for a range of x values. ?For example: > > Equation ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Upperlimit Lowerlimit > 24 + 18.75 * x + 4.79616e-013 * x ** 2 + 31.25 * x ** 3 ? ?7.800 ? ? ?7.600 > 28 + 22.5 * x + 5.68434e-013 * x ** 2 + 0 * x ** 3 ? ? ? ? 8.000 ? ? ?7.800 > > There is a lower limit for eqn1, after that the upper limit of the previous > eqn is lower of the next. > > From (X) scale linear, To (Y) scale linear > y = A0 + A1(x) + A2(x)2 + A3(x)3 > where: ? ?x= value ? LL > > LL = the lower limit of the applicable equation > > I would like to generate a sequence of values (linspace(min(lowerlimit),max > (upperlimit)), work out x (x=value-approriate lower limit) and then using the > appropriate equation generate the y values. > Below is the answer I am looking for > > Value ? x ? ? ? y > 7.600 ? 0 ? ? ? 24.000 > 7.650 ? 0.05 ? ?24.941 > 7.700 ? 0.1 ? ? 25.906 > 7.750 ? 0.15 ? ?26.918 > 7.800 ? 0.2 ? ? 28.000 > 7.850 ? 0.050 ? 29.125 > 7.900 ? 0.100 ? 30.25 > 7.950 ? 0.150 ? 31.375 > 8.000 ? 0.200 ? 32.5 > > Thanks for any help, > > Bevan Jenkins > your explanation is not completely clear, but you seem to be looking for a piece-wise function function. If you have a partitioning of the domain with non-overlapping but connected intervals, and x is scalar, then I would use np.searchsorted to find the interval index and then call the function by index in a list of functions. Kind of a python version of a case statement with searchsorted instead of dictionary. If the calculation in each equation is cheap/fast and many x, then you can vectorize it, calculate all equations for all x and then get the one corresponding to the searchsorted index, this will be much faster in some cases than looping over x. Josef From bevan.jenkins at ew.govt.nz Thu Apr 23 16:26:56 2009 From: bevan.jenkins at ew.govt.nz (Bevan Jenkins) Date: Thu, 23 Apr 2009 20:26:56 +0000 (UTC) Subject: [SciPy-user] plot curve of a series of equations References: <49F0376F.9070003@bigpond.net.au> Message-ID: Thanks Gary, Gary Ruben bigpond.net.au> writes: > > Like this? This basically just what you said, so I'm not sure what your > exact question is. Is it just the use of eval that you weren't clear on? > > import numpy as np > import matplotlib.pyplot as plt > > eqs = [ > ['24 + 18.75*x + 4.79616e-013*x**2 + 31.25*x**3', 7.800, 7.600], > ['28 + 22.5*x + 5.68434e-013*x**2 + 0*x**3', 8.000, 7.800], > ] > > for i in eqs: > xr = np.linspace(i[2], i[1], 10) > x = xr - i[2] > y = eval(i[0]) > plt.plot(xr,y) > plt.show() > > Gary R. Yes that is what I am looking for. Sorry if I wasn't clear but maybe i didn't have it clear in my head. Eval is new to me but also combing the upper limits, lower limits and equations into the one variable is mcuh tidier than what I had in mind. I was thinking of for i in eqs: but then struggling to get the approriate limits. Thanks, Bevan From jsseabold at gmail.com Thu Apr 23 16:56:35 2009 From: jsseabold at gmail.com (Skipper Seabold) Date: Thu, 23 Apr 2009 16:56:35 -0400 Subject: [SciPy-user] Numpy in parallel In-Reply-To: <49F02618.5080707@iqac.csic.es> References: <49F02618.5080707@iqac.csic.es> Message-ID: On Thu, Apr 23, 2009 at 4:26 AM, Ramon Crehuet wrote: > Dear all, > I would like to use the 4 processors in my AMD Phenom machine to do some > calcualtions with Numpy. > According to: > http://www.scipy.org/ParallelProgramming > This can be achieved if numpy is compiled using a BLAS that is able to > use multi-cores. I have installed the rpm numpy that comes with my > Opensuse 11.0 distribution. Apparently only one processor is used when I > calculate things like: > c=numpy.dot(a,b) > d=numpy.linalg.inv(c) > My first question is: is this a good choice? Is this Numpy optimized for > my arquitecture or I could do much better compiling ATLAS and Numpy from > scratch? Or is Intel MKL better? I don't know much about parallel processing or the actual dependence on BLAS, but I have recently gone through trying to build Numpy and Scipy with these dependencies from source on openSUSE and long story short, I gave up and switched to a debian-based distribution (Kubuntu Jaunty). I never actually identified what the compiler issue was, but Scipy didn't like how something was done in the build process. I can't recall the actual step that I think was hanging me up off the top of my head. You can see the discussion here: Does your build of scipy pass all of the tests? Also what does show_config tell you? -Skipper From ashish.rastogi at gmail.com Thu Apr 23 17:35:18 2009 From: ashish.rastogi at gmail.com (Ashish Rastogi) Date: Thu, 23 Apr 2009 17:35:18 -0400 Subject: [SciPy-user] problems installing scipy on a mac. Message-ID: <8A301854-AEFA-47B4-A377-1299834311DC@gmail.com> Hi all, I am trying to install scipy and getting a strange error on my Mac. I have installed all the prerequisites, including FFTW and gfortran, Xcode tools and numpy. I am including the output that I receive, Any help is much appreciated. Thanks, Ashish dhcp-172-26-76-239:scipy$ python setup.py build Warning: No configuration returned, assuming unavailable. blas_opt_info: FOUND: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3)] extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/ vecLib.framework/Headers'] lapack_opt_info: FOUND: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3)] extra_compile_args = ['-msse3'] umfpack_info: libraries umfpack not found in /System/Library/Frameworks/ Python.framework/Versions/2.5/lib libraries umfpack not found in /usr/local/lib libraries umfpack not found in /usr/lib libraries umfpack not found in /sw/lib /System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/ python/numpy/distutils/system_info.py:401: UserWarning: UMFPACK sparse solver (http://www.cise.ufl.edu/research/sparse/umfpack/ ) not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [umfpack]) or by setting the UMFPACK environment variable. warnings.warn(self.notfounderror.__doc__) NOT AVAILABLE running build running config_fc running build_src building py_modules sources building library "dfftpack" sources building library "fftpack" sources building library "linpack_lite" sources building library "mach" sources building library "quadpack" sources building library "odepack" sources building library "dop" sources building library "fitpack" sources building library "odrpack" sources building library "minpack" sources building library "rootfind" sources building library "superlu_src" sources building library "arpack" sources building library "sc_c_misc" sources building library "sc_cephes" sources building library "sc_mach" sources building library "sc_toms" sources building library "sc_amos" sources building library "sc_cdf" sources building library "sc_specfun" sources building library "statlib" sources building extension "scipy.cluster._vq" sources building extension "scipy.cluster._hierarchy_wrap" sources building extension "scipy.fftpack._fftpack" sources f2py options: [] adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. building extension "scipy.fftpack.convolve" sources f2py options: [] adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. building extension "scipy.integrate._quadpack" sources building extension "scipy.integrate._odepack" sources building extension "scipy.integrate.vode" sources f2py options: [] adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. building extension "scipy.integrate.dop" sources f2py options: [] adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. building extension "scipy.interpolate._fitpack" sources building extension "scipy.interpolate.dfitpack" sources f2py options: [] adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. adding 'build/src.macosx-10.5-i386-2.5/scipy/interpolate/src/ dfitpack-f2pywrappers.f' to sources. building extension "scipy.interpolate._interpolate" sources building extension "scipy.io.numpyio" sources building extension "scipy.lib.blas.fblas" sources f2py options: ['skip:', ':'] adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. adding 'build/src.macosx-10.5-i386-2.5/build/src.macosx-10.5- i386-2.5/scipy/lib/blas/fblas-f2pywrappers.f' to sources. building extension "scipy.lib.blas.cblas" sources adding 'build/src.macosx-10.5-i386-2.5/scipy/lib/blas/cblas.pyf' to sources. f2py options: ['skip:', ':'] adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. building extension "scipy.lib.lapack.flapack" sources f2py options: ['skip:', ':'] adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. building extension "scipy.lib.lapack.clapack" sources adding 'build/src.macosx-10.5-i386-2.5/scipy/lib/lapack/ clapack.pyf' to sources. f2py options: ['skip:', ':'] adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. building extension "scipy.lib.lapack.calc_lwork" sources f2py options: [] adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. building extension "scipy.lib.lapack.atlas_version" sources building extension "scipy.linalg.fblas" sources adding 'build/src.macosx-10.5-i386-2.5/scipy/linalg/fblas.pyf' to sources. f2py options: [] adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. adding 'build/src.macosx-10.5-i386-2.5/build/src.macosx-10.5- i386-2.5/scipy/linalg/fblas-f2pywrappers.f' to sources. building extension "scipy.linalg.cblas" sources adding 'build/src.macosx-10.5-i386-2.5/scipy/linalg/cblas.pyf' to sources. f2py options: [] adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. building extension "scipy.linalg.flapack" sources adding 'build/src.macosx-10.5-i386-2.5/scipy/linalg/flapack.pyf' to sources. f2py options: [] adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. adding 'build/src.macosx-10.5-i386-2.5/build/src.macosx-10.5- i386-2.5/scipy/linalg/flapack-f2pywrappers.f' to sources. building extension "scipy.linalg.clapack" sources adding 'build/src.macosx-10.5-i386-2.5/scipy/linalg/clapack.pyf' to sources. f2py options: [] adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. building extension "scipy.linalg._flinalg" sources f2py options: [] adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. building extension "scipy.linalg.calc_lwork" sources f2py options: [] adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. building extension "scipy.linalg.atlas_version" sources building extension "scipy.odr.__odrpack" sources building extension "scipy.optimize._minpack" sources building extension "scipy.optimize._zeros" sources building extension "scipy.optimize._lbfgsb" sources f2py options: [] adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. building extension "scipy.optimize.moduleTNC" sources building extension "scipy.optimize._cobyla" sources f2py options: [] adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. building extension "scipy.optimize.minpack2" sources f2py options: [] adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. building extension "scipy.optimize._slsqp" sources f2py options: [] adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. building extension "scipy.optimize._nnls" sources f2py options: [] adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. building extension "scipy.signal.sigtools" sources conv_template:> build/src.macosx-10.5-i386-2.5/scipy/signal/lfilter.inc Traceback (most recent call last): File "setup.py", line 92, in setup_package() File "setup.py", line 84, in setup_package configuration=configuration ) File "/System/Library/Frameworks/Python.framework/Versions/2.5/ Extras/lib/python/numpy/distutils/core.py", line 174, in setup return old_setup(**new_attr) File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/distutils/core.py", line 151, in setup dist.run_commands() File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/distutils/dist.py", line 974, in run_commands self.run_command(cmd) File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/distutils/command/build.py", line 112, in run self.run_command(cmd_name) File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/System/Library/Frameworks/Python.framework/Versions/2.5/ Extras/lib/python/numpy/distutils/command/build_src.py", line 87, in run self.build_sources() File "/System/Library/Frameworks/Python.framework/Versions/2.5/ Extras/lib/python/numpy/distutils/command/build_src.py", line 106, in build_sources self.build_extension_sources(ext) File "/System/Library/Frameworks/Python.framework/Versions/2.5/ Extras/lib/python/numpy/distutils/command/build_src.py", line 214, in build_extension_sources sources = self.template_sources(sources, ext) File "/System/Library/Frameworks/Python.framework/Versions/2.5/ Extras/lib/python/numpy/distutils/command/build_src.py", line 322, in template_sources outstr = process_c_file(source) File "/System/Library/Frameworks/Python.framework/Versions/2.5/ Extras/lib/python/numpy/distutils/conv_template.py", line 191, in process_file % (sourcefile, process_str(''.join(lines)))) File "/System/Library/Frameworks/Python.framework/Versions/2.5/ Extras/lib/python/numpy/distutils/conv_template.py", line 156, in process_str newstr[sub[0]:sub[1]], sub[4]) File "/System/Library/Frameworks/Python.framework/Versions/2.5/ Extras/lib/python/numpy/distutils/conv_template.py", line 120, in expand_sub for k in range(numsubs): TypeError: range() integer end argument expected, got NoneType. dhcp-172-26-76-239:scipy$ python -c 'from numpy.f2py.diagnose import run; run()' ------ os.name='posix' ------ sys.platform='darwin' ------ sys.version: 2.5.1 (r251:54863, Jan 13 2009, 10:26:13) [GCC 4.0.1 (Apple Inc. build 5465)] ------ sys.prefix: /System/Library/Frameworks/Python.framework/Versions/2.5 ------ sys.path=':/System/Library/Frameworks/Python.framework/Versions/2.5/ lib/python25.zip:/System/Library/Frameworks/Python.framework/Versions/ 2.5/lib/python2.5:/System/Library/Frameworks/Python.framework/Versions/ 2.5/lib/python2.5/plat-darwin:/System/Library/Frameworks/ Python.framework/Versions/2.5/lib/python2.5/plat-mac:/System/Library/ Frameworks/Python.framework/Versions/2.5/lib/python2.5/plat-mac/lib- scriptpackages:/System/Library/Frameworks/Python.framework/Versions/ 2.5/Extras/lib/python:/System/Library/Frameworks/Python.framework/ Versions/2.5/lib/python2.5/lib-tk:/System/Library/Frameworks/ Python.framework/Versions/2.5/lib/python2.5/lib-dynload:/Library/ Python/2.5/site-packages:/System/Library/Frameworks/Python.framework/ Versions/2.5/Extras/lib/python/PyObjC' ------ Failed to import Numeric: No module named Numeric Failed to import numarray: No module named numarray Found new numpy version '1.0.1' in /System/Library/Frameworks/ Python.framework/Versions/2.5/Extras/lib/python/numpy/__init__.pyc Found f2py2e version '2_3473' in /System/Library/Frameworks/ Python.framework/Versions/2.5/Extras/lib/python/numpy/f2py/f2py2e.pyc Found numpy.distutils version '0.4.0' in '/System/Library/Frameworks/ Python.framework/Versions/2.5/Extras/lib/python/numpy/distutils/ __init__.pyc' ------ Importing numpy.distutils.fcompiler ... ok ------ Checking availability of supported Fortran compilers: customize CompaqFCompiler customize NoneFCompiler customize AbsoftFCompiler Could not locate executable ifort Could not locate executable ifc Could not locate executable ifort Could not locate executable efort Could not locate executable efc Could not locate executable ifort Could not locate executable efort Could not locate executable efc customize IntelFCompiler Could not locate executable f77 Could not locate executable f95 customize GnuFCompiler customize SunFCompiler customize NAGFCompiler customize VastFCompiler customize GnuFCompiler customize IbmFCompiler customize Gnu95FCompiler Couldn't match compiler version for 'GNU Fortran (GCC) 4.2.3\nCopyright (C) 2007 Free Software Foundation, Inc.\n\nGNU Fortran comes with NO WARRANTY, to the extent permitted by law.\nYou may redistribute copies of GNU Fortran\nunder the terms of the GNU General Public License.\nFor more information about these matters, see the file named COPYING\n' customize IntelVisualFCompiler customize G95FCompiler customize IntelItaniumFCompiler customize PGroupFCompiler customize LaheyFCompiler customize CompaqVisualFCompiler customize MipsFCompiler customize HPUXFCompiler customize IntelItaniumVisualFCompiler customize IntelEM64TFCompiler List of available Fortran compilers: --fcompiler=gnu GNU Fortran Compiler (3.4.0) List of unavailable Fortran compilers: --fcompiler=absoft Absoft Corp Fortran Compiler --fcompiler=compaq Compaq Fortran Compiler --fcompiler=compaqv DIGITAL|Compaq Visual Fortran Compiler --fcompiler=g95 G95 Fortran Compiler --fcompiler=gnu95 GNU 95 Fortran Compiler --fcompiler=hpux HP Fortran 90 Compiler --fcompiler=ibm IBM XL Fortran Compiler --fcompiler=intel Intel Fortran Compiler for 32-bit apps --fcompiler=intele Intel Fortran Compiler for Itanium apps --fcompiler=intelem Intel Fortran Compiler for EM64T-based apps --fcompiler=intelev Intel Visual Fortran Compiler for Itanium apps --fcompiler=intelv Intel Visual Fortran Compiler for 32-bit apps --fcompiler=lahey Lahey/Fujitsu Fortran 95 Compiler --fcompiler=mips MIPSpro Fortran Compiler --fcompiler=nag NAGWare Fortran 95 Compiler --fcompiler=none Fake Fortran compiler --fcompiler=pg Portland Group Fortran Compiler --fcompiler=sun Sun|Forte Fortran 95 Compiler --fcompiler=vast Pacific-Sierra Research Fortran 90 Compiler List of unimplemented Fortran compilers: --fcompiler=f Fortran Company/NAG F Compiler For compiler details, run 'config_fc --verbose' setup command. ------ Importing numpy.distutils.cpuinfo ... ok ------ CPU information: getNCPUs is_32bit is_i386 ------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwf at cs.toronto.edu Thu Apr 23 18:53:58 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 23 Apr 2009 18:53:58 -0400 Subject: [SciPy-user] problems installing scipy on a mac. In-Reply-To: <8A301854-AEFA-47B4-A377-1299834311DC@gmail.com> References: <8A301854-AEFA-47B4-A377-1299834311DC@gmail.com> Message-ID: <4854B3DC-579D-4BE1-B658-F8D989BC4238@cs.toronto.edu> Hi, What version of SciPy are you trying to compile? 0.7.0 or an SVN revision? David On 23-Apr-09, at 5:35 PM, Ashish Rastogi wrote: > Hi all, > > I am trying to install scipy and getting a strange error on my Mac. > I have installed all the prerequisites, including FFTW and gfortran, > Xcode tools and numpy. I am including the output that I receive, > > Any help is much appreciated. > > Thanks, > Ashish > > > dhcp-172-26-76-239:scipy$ python setup.py build > Warning: No configuration returned, assuming unavailable. > blas_opt_info: > FOUND: > extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] > define_macros = [('NO_ATLAS_INFO', 3)] > extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/ > vecLib.framework/Headers'] > > lapack_opt_info: > FOUND: > extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] > define_macros = [('NO_ATLAS_INFO', 3)] > extra_compile_args = ['-msse3'] > > umfpack_info: > libraries umfpack not found in /System/Library/Frameworks/ > Python.framework/Versions/2.5/lib > libraries umfpack not found in /usr/local/lib > libraries umfpack not found in /usr/lib > libraries umfpack not found in /sw/lib > /System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/ > python/numpy/distutils/system_info.py:401: UserWarning: > UMFPACK sparse solver (http://www.cise.ufl.edu/research/sparse/umfpack/ > ) > not found. Directories to search for the libraries can be > specified in the > numpy/distutils/site.cfg file (section [umfpack]) or by setting > the UMFPACK environment variable. > warnings.warn(self.notfounderror.__doc__) > NOT AVAILABLE > > running build > running config_fc > running build_src > building py_modules sources > building library "dfftpack" sources > building library "fftpack" sources > building library "linpack_lite" sources > building library "mach" sources > building library "quadpack" sources > building library "odepack" sources > building library "dop" sources > building library "fitpack" sources > building library "odrpack" sources > building library "minpack" sources > building library "rootfind" sources > building library "superlu_src" sources > building library "arpack" sources > building library "sc_c_misc" sources > building library "sc_cephes" sources > building library "sc_mach" sources > building library "sc_toms" sources > building library "sc_amos" sources > building library "sc_cdf" sources > building library "sc_specfun" sources > building library "statlib" sources > building extension "scipy.cluster._vq" sources > building extension "scipy.cluster._hierarchy_wrap" sources > building extension "scipy.fftpack._fftpack" sources > f2py options: [] > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > building extension "scipy.fftpack.convolve" sources > f2py options: [] > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > building extension "scipy.integrate._quadpack" sources > building extension "scipy.integrate._odepack" sources > building extension "scipy.integrate.vode" sources > f2py options: [] > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > building extension "scipy.integrate.dop" sources > f2py options: [] > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > building extension "scipy.interpolate._fitpack" sources > building extension "scipy.interpolate.dfitpack" sources > f2py options: [] > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > adding 'build/src.macosx-10.5-i386-2.5/scipy/interpolate/src/ > dfitpack-f2pywrappers.f' to sources. > building extension "scipy.interpolate._interpolate" sources > building extension "scipy.io.numpyio" sources > building extension "scipy.lib.blas.fblas" sources > f2py options: ['skip:', ':'] > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > adding 'build/src.macosx-10.5-i386-2.5/build/src.macosx-10.5- > i386-2.5/scipy/lib/blas/fblas-f2pywrappers.f' to sources. > building extension "scipy.lib.blas.cblas" sources > adding 'build/src.macosx-10.5-i386-2.5/scipy/lib/blas/cblas.pyf' to > sources. > f2py options: ['skip:', ':'] > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > building extension "scipy.lib.lapack.flapack" sources > f2py options: ['skip:', ':'] > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > building extension "scipy.lib.lapack.clapack" sources > adding 'build/src.macosx-10.5-i386-2.5/scipy/lib/lapack/ > clapack.pyf' to sources. > f2py options: ['skip:', ':'] > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > building extension "scipy.lib.lapack.calc_lwork" sources > f2py options: [] > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > building extension "scipy.lib.lapack.atlas_version" sources > building extension "scipy.linalg.fblas" sources > adding 'build/src.macosx-10.5-i386-2.5/scipy/linalg/fblas.pyf' to > sources. > f2py options: [] > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > adding 'build/src.macosx-10.5-i386-2.5/build/src.macosx-10.5- > i386-2.5/scipy/linalg/fblas-f2pywrappers.f' to sources. > building extension "scipy.linalg.cblas" sources > adding 'build/src.macosx-10.5-i386-2.5/scipy/linalg/cblas.pyf' to > sources. > f2py options: [] > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > building extension "scipy.linalg.flapack" sources > adding 'build/src.macosx-10.5-i386-2.5/scipy/linalg/flapack.pyf' to > sources. > f2py options: [] > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > adding 'build/src.macosx-10.5-i386-2.5/build/src.macosx-10.5- > i386-2.5/scipy/linalg/flapack-f2pywrappers.f' to sources. > building extension "scipy.linalg.clapack" sources > adding 'build/src.macosx-10.5-i386-2.5/scipy/linalg/clapack.pyf' to > sources. > f2py options: [] > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > building extension "scipy.linalg._flinalg" sources > f2py options: [] > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > building extension "scipy.linalg.calc_lwork" sources > f2py options: [] > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > building extension "scipy.linalg.atlas_version" sources > building extension "scipy.odr.__odrpack" sources > building extension "scipy.optimize._minpack" sources > building extension "scipy.optimize._zeros" sources > building extension "scipy.optimize._lbfgsb" sources > f2py options: [] > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > building extension "scipy.optimize.moduleTNC" sources > building extension "scipy.optimize._cobyla" sources > f2py options: [] > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > building extension "scipy.optimize.minpack2" sources > f2py options: [] > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > building extension "scipy.optimize._slsqp" sources > f2py options: [] > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > building extension "scipy.optimize._nnls" sources > f2py options: [] > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > building extension "scipy.signal.sigtools" sources > conv_template:> build/src.macosx-10.5-i386-2.5/scipy/signal/ > lfilter.inc > Traceback (most recent call last): > File "setup.py", line 92, in > setup_package() > File "setup.py", line 84, in setup_package > configuration=configuration ) > File "/System/Library/Frameworks/Python.framework/Versions/2.5/ > Extras/lib/python/numpy/distutils/core.py", line 174, in setup > return old_setup(**new_attr) > File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ > python2.5/distutils/core.py", line 151, in setup > dist.run_commands() > File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ > python2.5/distutils/dist.py", line 974, in run_commands > self.run_command(cmd) > File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ > python2.5/distutils/dist.py", line 994, in run_command > cmd_obj.run() > File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ > python2.5/distutils/command/build.py", line 112, in run > self.run_command(cmd_name) > File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ > python2.5/distutils/cmd.py", line 333, in run_command > self.distribution.run_command(command) > File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ > python2.5/distutils/dist.py", line 994, in run_command > cmd_obj.run() > File "/System/Library/Frameworks/Python.framework/Versions/2.5/ > Extras/lib/python/numpy/distutils/command/build_src.py", line 87, in > run > self.build_sources() > File "/System/Library/Frameworks/Python.framework/Versions/2.5/ > Extras/lib/python/numpy/distutils/command/build_src.py", line 106, > in build_sources > self.build_extension_sources(ext) > File "/System/Library/Frameworks/Python.framework/Versions/2.5/ > Extras/lib/python/numpy/distutils/command/build_src.py", line 214, > in build_extension_sources > sources = self.template_sources(sources, ext) > File "/System/Library/Frameworks/Python.framework/Versions/2.5/ > Extras/lib/python/numpy/distutils/command/build_src.py", line 322, > in template_sources > outstr = process_c_file(source) > File "/System/Library/Frameworks/Python.framework/Versions/2.5/ > Extras/lib/python/numpy/distutils/conv_template.py", line 191, in > process_file > % (sourcefile, process_str(''.join(lines)))) > File "/System/Library/Frameworks/Python.framework/Versions/2.5/ > Extras/lib/python/numpy/distutils/conv_template.py", line 156, in > process_str > newstr[sub[0]:sub[1]], sub[4]) > File "/System/Library/Frameworks/Python.framework/Versions/2.5/ > Extras/lib/python/numpy/distutils/conv_template.py", line 120, in > expand_sub > for k in range(numsubs): > TypeError: range() integer end argument expected, got NoneType. > > > dhcp-172-26-76-239:scipy$ python -c 'from numpy.f2py.diagnose import > run; run()' > ------ > os.name='posix' > ------ > sys.platform='darwin' > ------ > sys.version: > 2.5.1 (r251:54863, Jan 13 2009, 10:26:13) > [GCC 4.0.1 (Apple Inc. build 5465)] > ------ > sys.prefix: > /System/Library/Frameworks/Python.framework/Versions/2.5 > ------ > sys.path=':/System/Library/Frameworks/Python.framework/Versions/2.5/ > lib/python25.zip:/System/Library/Frameworks/Python.framework/ > Versions/2.5/lib/python2.5:/System/Library/Frameworks/ > Python.framework/Versions/2.5/lib/python2.5/plat-darwin:/System/ > Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/plat- > mac:/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ > python2.5/plat-mac/lib-scriptpackages:/System/Library/Frameworks/ > Python.framework/Versions/2.5/Extras/lib/python:/System/Library/ > Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-tk:/ > System/Library/Frameworks/Python.framework/Versions/2.5/lib/ > python2.5/lib-dynload:/Library/Python/2.5/site-packages:/System/ > Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python/ > PyObjC' > ------ > Failed to import Numeric: No module named Numeric > Failed to import numarray: No module named numarray > Found new numpy version '1.0.1' in /System/Library/Frameworks/ > Python.framework/Versions/2.5/Extras/lib/python/numpy/__init__.pyc > Found f2py2e version '2_3473' in /System/Library/Frameworks/ > Python.framework/Versions/2.5/Extras/lib/python/numpy/f2py/f2py2e.pyc > Found numpy.distutils version '0.4.0' in '/System/Library/Frameworks/ > Python.framework/Versions/2.5/Extras/lib/python/numpy/distutils/ > __init__.pyc' > ------ > Importing numpy.distutils.fcompiler ... ok > ------ > Checking availability of supported Fortran compilers: > customize CompaqFCompiler > customize NoneFCompiler > customize AbsoftFCompiler > Could not locate executable ifort > Could not locate executable ifc > Could not locate executable ifort > Could not locate executable efort > Could not locate executable efc > Could not locate executable ifort > Could not locate executable efort > Could not locate executable efc > customize IntelFCompiler > Could not locate executable f77 > Could not locate executable f95 > customize GnuFCompiler > customize SunFCompiler > customize NAGFCompiler > customize VastFCompiler > customize GnuFCompiler > customize IbmFCompiler > customize Gnu95FCompiler > Couldn't match compiler version for 'GNU Fortran (GCC) > 4.2.3\nCopyright (C) 2007 Free Software Foundation, Inc.\n\nGNU > Fortran comes with NO WARRANTY, to the extent permitted by law.\nYou > may redistribute copies of GNU Fortran\nunder the terms of the GNU > General Public License.\nFor more information about these matters, > see the file named COPYING\n' > customize IntelVisualFCompiler > customize G95FCompiler > customize IntelItaniumFCompiler > customize PGroupFCompiler > customize LaheyFCompiler > customize CompaqVisualFCompiler > customize MipsFCompiler > customize HPUXFCompiler > customize IntelItaniumVisualFCompiler > customize IntelEM64TFCompiler > List of available Fortran compilers: > --fcompiler=gnu GNU Fortran Compiler (3.4.0) > List of unavailable Fortran compilers: > --fcompiler=absoft Absoft Corp Fortran Compiler > --fcompiler=compaq Compaq Fortran Compiler > --fcompiler=compaqv DIGITAL|Compaq Visual Fortran Compiler > --fcompiler=g95 G95 Fortran Compiler > --fcompiler=gnu95 GNU 95 Fortran Compiler > --fcompiler=hpux HP Fortran 90 Compiler > --fcompiler=ibm IBM XL Fortran Compiler > --fcompiler=intel Intel Fortran Compiler for 32-bit apps > --fcompiler=intele Intel Fortran Compiler for Itanium apps > --fcompiler=intelem Intel Fortran Compiler for EM64T-based apps > --fcompiler=intelev Intel Visual Fortran Compiler for Itanium apps > --fcompiler=intelv Intel Visual Fortran Compiler for 32-bit apps > --fcompiler=lahey Lahey/Fujitsu Fortran 95 Compiler > --fcompiler=mips MIPSpro Fortran Compiler > --fcompiler=nag NAGWare Fortran 95 Compiler > --fcompiler=none Fake Fortran compiler > --fcompiler=pg Portland Group Fortran Compiler > --fcompiler=sun Sun|Forte Fortran 95 Compiler > --fcompiler=vast Pacific-Sierra Research Fortran 90 Compiler > List of unimplemented Fortran compilers: > --fcompiler=f Fortran Company/NAG F Compiler > For compiler details, run 'config_fc --verbose' setup command. > ------ > Importing numpy.distutils.cpuinfo ... ok > ------ > CPU information: getNCPUs is_32bit is_i386 ------ > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From dwf at cs.toronto.edu Thu Apr 23 19:57:07 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 23 Apr 2009 19:57:07 -0400 Subject: [SciPy-user] Numpy in parallel In-Reply-To: <49F02618.5080707@iqac.csic.es> References: <49F02618.5080707@iqac.csic.es> Message-ID: On 23-Apr-09, at 4:26 AM, Ramon Crehuet wrote: > My first question is: is this a good choice? Is this Numpy optimized > for > my arquitecture or I could do much better compiling ATLAS and Numpy > from > scratch? Or is Intel MKL better? Almost certainly you could do better than you are currently by compiling your own ATLAS, I don't know about the OpenSUSE packages but distribution binaries usually can't be optimized very much since they can't count on everyone having certain CPU features. The Intel MKL, while it can be used on AMD chips (apparently), is mainly designed to optimize the heck out of *Intel* chips, and from what I understand outperforms ATLAS on that hardware. I would not expect it to do very much better than ATLAS (if at all) on non-Intel hardware. > My second question is: I have to create, many times, a large array of > normally distributed vectors (the stochastic term in a Langevin > dynamics > run). Do you know if numpy.random.randn will be able to use multi- > cores? Not on its own, I don't think, but since it's C code it should release the GIL. This means that you can use the Python threading module and the different threads (which are full POSIX threads on Linux) can run simultaneously on your multiple cores: http://docs.python.org/library/threading.html The multiprocessing module in Python 2.6 (or available in Python 2.5 if you install it from PyPI) let's you do the same thing but with full processes: http://docs.python.org/library/multiprocessing.html David From ashish.rastogi at gmail.com Thu Apr 23 20:28:43 2009 From: ashish.rastogi at gmail.com (Ashish Rastogi) Date: Thu, 23 Apr 2009 20:28:43 -0400 Subject: [SciPy-user] problems installing scipy on a mac. In-Reply-To: <4854B3DC-579D-4BE1-B658-F8D989BC4238@cs.toronto.edu> References: <8A301854-AEFA-47B4-A377-1299834311DC@gmail.com> <4854B3DC-579D-4BE1-B658-F8D989BC4238@cs.toronto.edu> Message-ID: An SVN revision. The version.py file under the scipy directory says 0.8.0 Thanks, Ashish On Thu, Apr 23, 2009 at 6:53 PM, David Warde-Farley wrote: > Hi, > > What version of SciPy are you trying to compile? 0.7.0 or an SVN > revision? > > David > > On 23-Apr-09, at 5:35 PM, Ashish Rastogi wrote: > > > Hi all, > > > > I am trying to install scipy and getting a strange error on my Mac. > > I have installed all the prerequisites, including FFTW and gfortran, > > Xcode tools and numpy. I am including the output that I receive, > > > > Any help is much appreciated. > > > > Thanks, > > Ashish > > > > > > dhcp-172-26-76-239:scipy$ python setup.py build > > Warning: No configuration returned, assuming unavailable. > > blas_opt_info: > > FOUND: > > extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] > > define_macros = [('NO_ATLAS_INFO', 3)] > > extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/ > > vecLib.framework/Headers'] > > > > lapack_opt_info: > > FOUND: > > extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] > > define_macros = [('NO_ATLAS_INFO', 3)] > > extra_compile_args = ['-msse3'] > > > > umfpack_info: > > libraries umfpack not found in /System/Library/Frameworks/ > > Python.framework/Versions/2.5/lib > > libraries umfpack not found in /usr/local/lib > > libraries umfpack not found in /usr/lib > > libraries umfpack not found in /sw/lib > > /System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/ > > python/numpy/distutils/system_info.py:401: UserWarning: > > UMFPACK sparse solver ( > http://www.cise.ufl.edu/research/sparse/umfpack/ > > ) > > not found. Directories to search for the libraries can be > > specified in the > > numpy/distutils/site.cfg file (section [umfpack]) or by setting > > the UMFPACK environment variable. > > warnings.warn(self.notfounderror.__doc__) > > NOT AVAILABLE > > > > running build > > running config_fc > > running build_src > > building py_modules sources > > building library "dfftpack" sources > > building library "fftpack" sources > > building library "linpack_lite" sources > > building library "mach" sources > > building library "quadpack" sources > > building library "odepack" sources > > building library "dop" sources > > building library "fitpack" sources > > building library "odrpack" sources > > building library "minpack" sources > > building library "rootfind" sources > > building library "superlu_src" sources > > building library "arpack" sources > > building library "sc_c_misc" sources > > building library "sc_cephes" sources > > building library "sc_mach" sources > > building library "sc_toms" sources > > building library "sc_amos" sources > > building library "sc_cdf" sources > > building library "sc_specfun" sources > > building library "statlib" sources > > building extension "scipy.cluster._vq" sources > > building extension "scipy.cluster._hierarchy_wrap" sources > > building extension "scipy.fftpack._fftpack" sources > > f2py options: [] > > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > > building extension "scipy.fftpack.convolve" sources > > f2py options: [] > > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > > building extension "scipy.integrate._quadpack" sources > > building extension "scipy.integrate._odepack" sources > > building extension "scipy.integrate.vode" sources > > f2py options: [] > > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > > building extension "scipy.integrate.dop" sources > > f2py options: [] > > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > > building extension "scipy.interpolate._fitpack" sources > > building extension "scipy.interpolate.dfitpack" sources > > f2py options: [] > > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > > adding 'build/src.macosx-10.5-i386-2.5/scipy/interpolate/src/ > > dfitpack-f2pywrappers.f' to sources. > > building extension "scipy.interpolate._interpolate" sources > > building extension "scipy.io.numpyio" sources > > building extension "scipy.lib.blas.fblas" sources > > f2py options: ['skip:', ':'] > > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > > adding 'build/src.macosx-10.5-i386-2.5/build/src.macosx-10.5- > > i386-2.5/scipy/lib/blas/fblas-f2pywrappers.f' to sources. > > building extension "scipy.lib.blas.cblas" sources > > adding 'build/src.macosx-10.5-i386-2.5/scipy/lib/blas/cblas.pyf' to > > sources. > > f2py options: ['skip:', ':'] > > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > > building extension "scipy.lib.lapack.flapack" sources > > f2py options: ['skip:', ':'] > > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > > building extension "scipy.lib.lapack.clapack" sources > > adding 'build/src.macosx-10.5-i386-2.5/scipy/lib/lapack/ > > clapack.pyf' to sources. > > f2py options: ['skip:', ':'] > > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > > building extension "scipy.lib.lapack.calc_lwork" sources > > f2py options: [] > > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > > building extension "scipy.lib.lapack.atlas_version" sources > > building extension "scipy.linalg.fblas" sources > > adding 'build/src.macosx-10.5-i386-2.5/scipy/linalg/fblas.pyf' to > > sources. > > f2py options: [] > > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > > adding 'build/src.macosx-10.5-i386-2.5/build/src.macosx-10.5- > > i386-2.5/scipy/linalg/fblas-f2pywrappers.f' to sources. > > building extension "scipy.linalg.cblas" sources > > adding 'build/src.macosx-10.5-i386-2.5/scipy/linalg/cblas.pyf' to > > sources. > > f2py options: [] > > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > > building extension "scipy.linalg.flapack" sources > > adding 'build/src.macosx-10.5-i386-2.5/scipy/linalg/flapack.pyf' to > > sources. > > f2py options: [] > > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > > adding 'build/src.macosx-10.5-i386-2.5/build/src.macosx-10.5- > > i386-2.5/scipy/linalg/flapack-f2pywrappers.f' to sources. > > building extension "scipy.linalg.clapack" sources > > adding 'build/src.macosx-10.5-i386-2.5/scipy/linalg/clapack.pyf' to > > sources. > > f2py options: [] > > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > > building extension "scipy.linalg._flinalg" sources > > f2py options: [] > > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > > building extension "scipy.linalg.calc_lwork" sources > > f2py options: [] > > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > > building extension "scipy.linalg.atlas_version" sources > > building extension "scipy.odr.__odrpack" sources > > building extension "scipy.optimize._minpack" sources > > building extension "scipy.optimize._zeros" sources > > building extension "scipy.optimize._lbfgsb" sources > > f2py options: [] > > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > > building extension "scipy.optimize.moduleTNC" sources > > building extension "scipy.optimize._cobyla" sources > > f2py options: [] > > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > > building extension "scipy.optimize.minpack2" sources > > f2py options: [] > > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > > building extension "scipy.optimize._slsqp" sources > > f2py options: [] > > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > > building extension "scipy.optimize._nnls" sources > > f2py options: [] > > adding 'build/src.macosx-10.5-i386-2.5/fortranobject.c' to sources. > > adding 'build/src.macosx-10.5-i386-2.5' to include_dirs. > > building extension "scipy.signal.sigtools" sources > > conv_template:> build/src.macosx-10.5-i386-2.5/scipy/signal/ > > lfilter.inc > > Traceback (most recent call last): > > File "setup.py", line 92, in > > setup_package() > > File "setup.py", line 84, in setup_package > > configuration=configuration ) > > File "/System/Library/Frameworks/Python.framework/Versions/2.5/ > > Extras/lib/python/numpy/distutils/core.py", line 174, in setup > > return old_setup(**new_attr) > > File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ > > python2.5/distutils/core.py", line 151, in setup > > dist.run_commands() > > File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ > > python2.5/distutils/dist.py", line 974, in run_commands > > self.run_command(cmd) > > File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ > > python2.5/distutils/dist.py", line 994, in run_command > > cmd_obj.run() > > File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ > > python2.5/distutils/command/build.py", line 112, in run > > self.run_command(cmd_name) > > File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ > > python2.5/distutils/cmd.py", line 333, in run_command > > self.distribution.run_command(command) > > File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ > > python2.5/distutils/dist.py", line 994, in run_command > > cmd_obj.run() > > File "/System/Library/Frameworks/Python.framework/Versions/2.5/ > > Extras/lib/python/numpy/distutils/command/build_src.py", line 87, in > > run > > self.build_sources() > > File "/System/Library/Frameworks/Python.framework/Versions/2.5/ > > Extras/lib/python/numpy/distutils/command/build_src.py", line 106, > > in build_sources > > self.build_extension_sources(ext) > > File "/System/Library/Frameworks/Python.framework/Versions/2.5/ > > Extras/lib/python/numpy/distutils/command/build_src.py", line 214, > > in build_extension_sources > > sources = self.template_sources(sources, ext) > > File "/System/Library/Frameworks/Python.framework/Versions/2.5/ > > Extras/lib/python/numpy/distutils/command/build_src.py", line 322, > > in template_sources > > outstr = process_c_file(source) > > File "/System/Library/Frameworks/Python.framework/Versions/2.5/ > > Extras/lib/python/numpy/distutils/conv_template.py", line 191, in > > process_file > > % (sourcefile, process_str(''.join(lines)))) > > File "/System/Library/Frameworks/Python.framework/Versions/2.5/ > > Extras/lib/python/numpy/distutils/conv_template.py", line 156, in > > process_str > > newstr[sub[0]:sub[1]], sub[4]) > > File "/System/Library/Frameworks/Python.framework/Versions/2.5/ > > Extras/lib/python/numpy/distutils/conv_template.py", line 120, in > > expand_sub > > for k in range(numsubs): > > TypeError: range() integer end argument expected, got NoneType. > > > > > > dhcp-172-26-76-239:scipy$ python -c 'from numpy.f2py.diagnose import > > run; run()' > > ------ > > os.name='posix' > > ------ > > sys.platform='darwin' > > ------ > > sys.version: > > 2.5.1 (r251:54863, Jan 13 2009, 10:26:13) > > [GCC 4.0.1 (Apple Inc. build 5465)] > > ------ > > sys.prefix: > > /System/Library/Frameworks/Python.framework/Versions/2.5 > > ------ > > sys.path=':/System/Library/Frameworks/Python.framework/Versions/2.5/ > > lib/python25.zip:/System/Library/Frameworks/Python.framework/ > > Versions/2.5/lib/python2.5:/System/Library/Frameworks/ > > Python.framework/Versions/2.5/lib/python2.5/plat-darwin:/System/ > > Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/plat- > > mac:/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ > > python2.5/plat-mac/lib-scriptpackages:/System/Library/Frameworks/ > > Python.framework/Versions/2.5/Extras/lib/python:/System/Library/ > > Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-tk:/ > > System/Library/Frameworks/Python.framework/Versions/2.5/lib/ > > python2.5/lib-dynload:/Library/Python/2.5/site-packages:/System/ > > Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python/ > > PyObjC' > > ------ > > Failed to import Numeric: No module named Numeric > > Failed to import numarray: No module named numarray > > Found new numpy version '1.0.1' in /System/Library/Frameworks/ > > Python.framework/Versions/2.5/Extras/lib/python/numpy/__init__.pyc > > Found f2py2e version '2_3473' in /System/Library/Frameworks/ > > Python.framework/Versions/2.5/Extras/lib/python/numpy/f2py/f2py2e.pyc > > Found numpy.distutils version '0.4.0' in '/System/Library/Frameworks/ > > Python.framework/Versions/2.5/Extras/lib/python/numpy/distutils/ > > __init__.pyc' > > ------ > > Importing numpy.distutils.fcompiler ... ok > > ------ > > Checking availability of supported Fortran compilers: > > customize CompaqFCompiler > > customize NoneFCompiler > > customize AbsoftFCompiler > > Could not locate executable ifort > > Could not locate executable ifc > > Could not locate executable ifort > > Could not locate executable efort > > Could not locate executable efc > > Could not locate executable ifort > > Could not locate executable efort > > Could not locate executable efc > > customize IntelFCompiler > > Could not locate executable f77 > > Could not locate executable f95 > > customize GnuFCompiler > > customize SunFCompiler > > customize NAGFCompiler > > customize VastFCompiler > > customize GnuFCompiler > > customize IbmFCompiler > > customize Gnu95FCompiler > > Couldn't match compiler version for 'GNU Fortran (GCC) > > 4.2.3\nCopyright (C) 2007 Free Software Foundation, Inc.\n\nGNU > > Fortran comes with NO WARRANTY, to the extent permitted by law.\nYou > > may redistribute copies of GNU Fortran\nunder the terms of the GNU > > General Public License.\nFor more information about these matters, > > see the file named COPYING\n' > > customize IntelVisualFCompiler > > customize G95FCompiler > > customize IntelItaniumFCompiler > > customize PGroupFCompiler > > customize LaheyFCompiler > > customize CompaqVisualFCompiler > > customize MipsFCompiler > > customize HPUXFCompiler > > customize IntelItaniumVisualFCompiler > > customize IntelEM64TFCompiler > > List of available Fortran compilers: > > --fcompiler=gnu GNU Fortran Compiler (3.4.0) > > List of unavailable Fortran compilers: > > --fcompiler=absoft Absoft Corp Fortran Compiler > > --fcompiler=compaq Compaq Fortran Compiler > > --fcompiler=compaqv DIGITAL|Compaq Visual Fortran Compiler > > --fcompiler=g95 G95 Fortran Compiler > > --fcompiler=gnu95 GNU 95 Fortran Compiler > > --fcompiler=hpux HP Fortran 90 Compiler > > --fcompiler=ibm IBM XL Fortran Compiler > > --fcompiler=intel Intel Fortran Compiler for 32-bit apps > > --fcompiler=intele Intel Fortran Compiler for Itanium apps > > --fcompiler=intelem Intel Fortran Compiler for EM64T-based apps > > --fcompiler=intelev Intel Visual Fortran Compiler for Itanium apps > > --fcompiler=intelv Intel Visual Fortran Compiler for 32-bit apps > > --fcompiler=lahey Lahey/Fujitsu Fortran 95 Compiler > > --fcompiler=mips MIPSpro Fortran Compiler > > --fcompiler=nag NAGWare Fortran 95 Compiler > > --fcompiler=none Fake Fortran compiler > > --fcompiler=pg Portland Group Fortran Compiler > > --fcompiler=sun Sun|Forte Fortran 95 Compiler > > --fcompiler=vast Pacific-Sierra Research Fortran 90 Compiler > > List of unimplemented Fortran compilers: > > --fcompiler=f Fortran Company/NAG F Compiler > > For compiler details, run 'config_fc --verbose' setup command. > > ------ > > Importing numpy.distutils.cpuinfo ... ok > > ------ > > CPU information: getNCPUs is_32bit is_i386 ------ > > > > _______________________________________________ > > SciPy-user mailing list > > SciPy-user at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwf at cs.toronto.edu Thu Apr 23 20:39:41 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 23 Apr 2009 20:39:41 -0400 Subject: [SciPy-user] problems with splrep,splev In-Reply-To: References: <114880320904170839q26855a1doa3fce5423901f8c1@mail.gmail.com> Message-ID: <2F3F6D5A-1F16-40DD-94F5-0AA1EE7358BB@cs.toronto.edu> On 18-Apr-09, at 9:31 AM, Christian K. wrote: > > Right, but then I don't get the meaning of xb and xe at all. What > sense > does it make to choose a fit interval larger than the input data? IMHO > x[0] < xb < xe < x[-1] should hold but obviously the docs tell the > opposite. What sense does it make to fit using a smaller interval than x[0] ... x[-1]? You'd then be throwing away some of your observations. An xb and xe value < or > might do something I don't see why you're specifying xb and xe to begin with. If you're fitting it to all the data, simply omitting those arguments makes most sense. You can slice your input arrays if you'd prefer not to use all of your data. David From david at ar.media.kyoto-u.ac.jp Fri Apr 24 03:58:49 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 24 Apr 2009 16:58:49 +0900 Subject: [SciPy-user] problems installing scipy on a mac. In-Reply-To: References: <8A301854-AEFA-47B4-A377-1299834311DC@gmail.com> <4854B3DC-579D-4BE1-B658-F8D989BC4238@cs.toronto.edu> Message-ID: <49F17139.1040407@ar.media.kyoto-u.ac.jp> Ashish Rastogi wrote: > An SVN revision. The version.py file under the scipy directory says 0.8.0 Which version of numpy are you using ? It looks like you are using the system numpy, which is quite old. To build the trunk, you need numpy 1.3.0 (or above). David From faltet at pytables.org Fri Apr 24 06:15:20 2009 From: faltet at pytables.org (Francesc Alted) Date: Fri, 24 Apr 2009 12:15:20 +0200 Subject: [SciPy-user] Numpy in parallel In-Reply-To: References: <49F02618.5080707@iqac.csic.es> Message-ID: <200904241215.21439.faltet@pytables.org> A Friday 24 April 2009, David Warde-Farley escrigu?: > The Intel MKL, while it can be used on AMD chips (apparently), is > mainly designed to optimize the heck out of *Intel* chips, and from > what I understand outperforms ATLAS on that hardware. I would not > expect it to do very much better than ATLAS (if at all) on non-Intel > hardware. Well, following the excellent "Optimizing software in C++" [1], MKL explicitely disables its optimizations when an Intel CPU is not detected. However, in this manual it is explained how to override MKL's CPU detection for taking advantage of other CPUs (i.e. AMD) too. In fact, looking at table 11.4 one can see that there is not much difference between using Intel or AMD CPUs when the CPU detection has been overridden (and sometimes AMDs can be faster, as show there). [1] http://www.agner.org/optimize/optimizing_cpp.pdf -- Francesc Alted "One would expect people to feel threatened by the 'giant brains or machines that think'. In fact, the frightening computer becomes less frightening if it is used only to simulate a familiar noncomputer." -- Edsger W. Dykstra "On the cruelty of really teaching computer science" From ashish.rastogi at gmail.com Fri Apr 24 08:01:46 2009 From: ashish.rastogi at gmail.com (Ashish Rastogi) Date: Fri, 24 Apr 2009 08:01:46 -0400 Subject: [SciPy-user] problems installing scipy on a mac. In-Reply-To: <49F17139.1040407@ar.media.kyoto-u.ac.jp> References: <8A301854-AEFA-47B4-A377-1299834311DC@gmail.com> <4854B3DC-579D-4BE1-B658-F8D989BC4238@cs.toronto.edu> <49F17139.1040407@ar.media.kyoto-u.ac.jp> Message-ID: I'm using numpy version 1.4.0. -- Ashish On Apr 24, 2009, at 3:58 AM, David Cournapeau wrote: > Ashish Rastogi wrote: >> An SVN revision. The version.py file under the scipy directory says >> 0.8.0 > > Which version of numpy are you using ? It looks like you are using the > system numpy, which is quite old. To build the trunk, you need numpy > 1.3.0 (or above). > > David > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From cournape at gmail.com Fri Apr 24 09:41:43 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 24 Apr 2009 22:41:43 +0900 Subject: [SciPy-user] problems installing scipy on a mac. In-Reply-To: References: <8A301854-AEFA-47B4-A377-1299834311DC@gmail.com> <4854B3DC-579D-4BE1-B658-F8D989BC4238@cs.toronto.edu> <49F17139.1040407@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220904240641m2db5f264pf348627ac363ff24@mail.gmail.com> On Fri, Apr 24, 2009 at 9:01 PM, Ashish Rastogi wrote: > I'm using numpy version 1.4.0. Are you sure that's the version you are using when installing scipy ? What does python -c "import numpy; print numpy.version.version" returns in the shell you install scipy from. David From ashish.rastogi at gmail.com Fri Apr 24 09:43:39 2009 From: ashish.rastogi at gmail.com (Ashish Rastogi) Date: Fri, 24 Apr 2009 09:43:39 -0400 Subject: [SciPy-user] problems installing scipy on a mac. In-Reply-To: <5b8d13220904240641m2db5f264pf348627ac363ff24@mail.gmail.com> References: <8A301854-AEFA-47B4-A377-1299834311DC@gmail.com> <4854B3DC-579D-4BE1-B658-F8D989BC4238@cs.toronto.edu> <49F17139.1040407@ar.media.kyoto-u.ac.jp> <5b8d13220904240641m2db5f264pf348627ac363ff24@mail.gmail.com> Message-ID: Thanks. It worked. On Fri, Apr 24, 2009 at 9:41 AM, David Cournapeau wrote: > On Fri, Apr 24, 2009 at 9:01 PM, Ashish Rastogi > wrote: > > I'm using numpy version 1.4.0. > > Are you sure that's the version you are using when installing scipy ? What > does > > python -c "import numpy; print numpy.version.version" > > returns in the shell you install scipy from. > > David > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From camejia at raytheon.com Fri Apr 24 09:46:56 2009 From: camejia at raytheon.com (Christopher A Mejia) Date: Fri, 24 Apr 2009 06:46:56 -0700 Subject: [SciPy-user] Using savemat with (nested) NumPy record arrays? In-Reply-To: <1e2af89e0904222109x66060a8ew7817701ee224e19c@mail.gmail.com> Message-ID: Matthew, Yes and yes! I downloaded the latest SVN, and savemat of NumPy scalars (shape==()) worked correctly, and savemat ran much faster. A data set which did not complete in writing overnight (~16 hours) succesfully wrote in 4.5 minutes with the update. Thank you! I did notice that the files created by savemat are rather large; if I read it in MATLAB and write it back out, it is much smaller. Maybe MATLAB is intelligent deciding when to compress the data? Anyway, this is not a big deal and savemat certainly provides the functionality I need. Another "nice to have" would be to upgrade savemat to work with nested dtypes, in addition to dtype=object. I don't know enough about Python to know if this is simple or complicated, and as long as one knows that you need to set dtype=object instead of defining a nested dtype, again all of the functionality is there. Thanks again, --Chris Matthew Brett Sent by: scipy-user-bounces at scipy.org 04/22/2009 09:09 PM Please respond to SciPy Users List To SciPy Users List cc Subject Re: [SciPy-user] Using savemat with (nested) NumPy record arrays? Hi, > 2. However, I didn't get any data showing up in MATLAB after loading my > file, unless I made sure that all of the arrays had a shape with at least > one dimension. There was a bug in scipy 0.7 release - do you have the latest SVN? Does that work? > The problem I'm running into now is that the savemat function is too slow. > The top level of data I'm trying to save is an array of structures. It > seems that the time for savemat increases exponentially as the number of > records in this structure array increases. Is there a better way to > organize the storage of data into savemat, or is there a simple way to > modify savemat to speed it up? Yes, that's a known problem as well. The hope is that the current SVN implementation is a lot faster. I'd be very glad of your feedback about that... Thanks, Matthew _______________________________________________ SciPy-user mailing list SciPy-user at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From ashish.rastogi at gmail.com Fri Apr 24 09:48:12 2009 From: ashish.rastogi at gmail.com (Ashish Rastogi) Date: Fri, 24 Apr 2009 09:48:12 -0400 Subject: [SciPy-user] problems installing scipy on a mac. In-Reply-To: <5b8d13220904240641m2db5f264pf348627ac363ff24@mail.gmail.com> References: <8A301854-AEFA-47B4-A377-1299834311DC@gmail.com> <4854B3DC-579D-4BE1-B658-F8D989BC4238@cs.toronto.edu> <49F17139.1040407@ar.media.kyoto-u.ac.jp> <5b8d13220904240641m2db5f264pf348627ac363ff24@mail.gmail.com> Message-ID: The version was indeed incorrect. -- Ashish On Fri, Apr 24, 2009 at 9:41 AM, David Cournapeau wrote: > On Fri, Apr 24, 2009 at 9:01 PM, Ashish Rastogi > wrote: > > I'm using numpy version 1.4.0. > > Are you sure that's the version you are using when installing scipy ? What > does > > python -c "import numpy; print numpy.version.version" > > returns in the shell you install scipy from. > > David > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.robitaille at gmail.com Fri Apr 24 10:58:22 2009 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Fri, 24 Apr 2009 10:58:22 -0400 Subject: [SciPy-user] Ignoring pixels with gaussian_filter Message-ID: Hello, I am currently using gaussian_filter to smooth an image stored in a numpy array. My problem is that some pixels have no defined value, and are set to NaN. If gaussian_filter stumbles upon a NaN value, it will set all pixels within a certain radius of that value to NaN. Consider the following example: --- import numpy as np from scipy.ndimage import gaussian_filter a = np.ones((9,9)) a[4,4] = np.nan b = gaussian_filter(a,sigma=2) print "Original:" print a print "Smoothed:" print b --- Of course, I can simply use a[np.where(np.isnan(a))] = 0. to reset all NaN pixels to zero, but then the result depends on the value I replace NaNs by. Is there a way to get gaussian_filter to simply ignore such pixels when smoothing? (apart from writing a gaussian filter algorithm from scratch!) Thanks for any help, Thomas From mudit_19a at yahoo.com Fri Apr 24 13:41:30 2009 From: mudit_19a at yahoo.com (mudit sharma) Date: Fri, 24 Apr 2009 23:11:30 +0530 (IST) Subject: [SciPy-user] realtime axes issues Message-ID: <543906.80630.qm@web94916.mail.in2.yahoo.com> I am trying to debug this issue may be someone can help. I have drawn multiple axes in one figure under "ipython -pylab " which contains line that are updated in realtime. Then I remove these axses using fig.delaxes from figures and draw another sets of axeses but the previous sets of axeses still exists and there line data keep getting updated in realtime. It causes current sets of axeses and x and y axis in figure to flicker. Any idea what might be causing this? Thanks! From zachary.pincus at yale.edu Fri Apr 24 15:12:36 2009 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Fri, 24 Apr 2009 15:12:36 -0400 Subject: [SciPy-user] Ignoring pixels with gaussian_filter In-Reply-To: References: Message-ID: Hi Thomas, > Hello, > > I am currently using gaussian_filter to smooth an image stored in a > numpy array. My problem is that some pixels have no defined value, and > are set to NaN. If gaussian_filter stumbles upon a NaN value, it will > set all pixels within a certain radius of that value to NaN. ... > Of course, I can simply use > > a[np.where(np.isnan(a))] = 0. > > to reset all NaN pixels to zero, but then the result depends on the > value I replace NaNs by. > > Is there a way to get gaussian_filter to simply ignore such pixels > when smoothing? (apart from writing a gaussian filter algorithm from > scratch!) No simple way to get gaussian_filter to ignore nan pixels when doing the convolution. You could write your own convolution function in cython (not as bad as it sounds, or you could fill in the missing data with something reasonable (like the median of the neighbors) before gaussian filtering. These should give pretty similar results. You could check out an earlier thread on numpy-discussion titled "Help with interpolating missing values from a 3D scanner" for some discussion about filling in missing values. Zach From davide.cittaro at ifom-ieo-campus.it Fri Apr 24 15:44:15 2009 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Fri, 24 Apr 2009 21:44:15 +0200 Subject: [SciPy-user] canny1d filter? Message-ID: <5C957654-B8F7-49B2-BAE6-4B1BA221096A@ifom-ieo-campus.it> Hi all, I'm new to image (and signal processing). I have to detect peaks in a 1d signal. I'm collecting documentation about this and tested some procedures (actually LoG and DoG in the past two days...). I would like now to test the Canny filter which is not included in scipy.ndimage filters... I've found a bunch of implementations in 2d cases, in many languages, but not for the 1d case... Does anybody want to share some hints on this? :-) Any suggestion is appreciated Thanks d From Chris.Barker at noaa.gov Fri Apr 24 17:55:54 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 24 Apr 2009 14:55:54 -0700 Subject: [SciPy-user] plot curve of a series of equations In-Reply-To: References: <49F0376F.9070003@bigpond.net.au> Message-ID: <49F2356A.6070700@noaa.gov> >> eqs = [ >> ['24 + 18.75*x + 4.79616e-013*x**2 + 31.25*x**3', 7.800, 7.600], >> ['28 + 22.5*x + 5.68434e-013*x**2 + 0*x**3', 8.000, 7.800], >> ] >> >> for i in eqs: >> xr = np.linspace(i[2], i[1], 10) >> x = xr - i[2] >> y = eval(i[0]) eval makes me nervous. One of the nifty things about python is that functions are objects too, so you can do something like: (untested) eqs = [ (lambda x: 24 + 18.75*x + 4.79616e-013*x**2 + 31.25*x**3, 7.800, 7.600), (lambda x: 28 + 22.5*x + 5.68434e-013*x**2 + 0*x**3, 8.000, 7.800), ] for f, x1, x2 in eqs: xr = np.linspace(x1, x2, 10) x = xr - x2 y = f(x) If the function are more complicated, you can use def, rather than lambda, and still put them in list or whatever. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From zachary.pincus at yale.edu Fri Apr 24 19:30:16 2009 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Fri, 24 Apr 2009 19:30:16 -0400 Subject: [SciPy-user] canny1d filter? In-Reply-To: <5C957654-B8F7-49B2-BAE6-4B1BA221096A@ifom-ieo-campus.it> References: <5C957654-B8F7-49B2-BAE6-4B1BA221096A@ifom-ieo-campus.it> Message-ID: <2821C09D-5738-4BD8-9BBE-EF371D713B5A@yale.edu> Hi Davide, I can send you code for a 2D Canny filter that uses numpy and scipy.ndimage to good effect. It's a decent demonstration of the filter, and of how to make good use out of the tools in ndimage. However, I don't think that Canny edge-detection, as I understand it, is applicable to 1D signaling. Basically, Canny filtering finds the gradient magnitude and direction in an image, and then uses the direction information to thin out edges except along their "ridges". It then uses a hysteresis thresholding approach to link the edges (by finding seeds with a stringent threshold, and then extending the seeds out along the edges using a less-stringent threshold). None of this stuff makes sense in the 1D case -- there's no "edge direction", etc. Zach On Apr 24, 2009, at 3:44 PM, Davide Cittaro wrote: > Hi all, I'm new to image (and signal processing). > I have to detect peaks in a 1d signal. I'm collecting documentation > about this and tested some procedures (actually LoG and DoG in the > past two days...). I would like now to test the Canny filter which is > not included in scipy.ndimage filters... > I've found a bunch of implementations in 2d cases, in many languages, > but not for the 1d case... Does anybody want to share some hints on > this? :-) > Any suggestion is appreciated > > Thanks > > d > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From thomas.robitaille at gmail.com Fri Apr 24 23:20:44 2009 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Fri, 24 Apr 2009 23:20:44 -0400 Subject: [SciPy-user] Ignoring pixels with gaussian_filter In-Reply-To: References: Message-ID: <274C4D39-07A2-4236-B14C-6C104EE11352@gmail.com> Hi Zach, >> Is there a way to get gaussian_filter to simply ignore such pixels >> when smoothing? (apart from writing a gaussian filter algorithm from >> scratch!) > > No simple way to get gaussian_filter to ignore nan pixels when doing > the convolution. > You could write your own convolution function in cython [snip] I looked into this, and figured out how to write my own filter. It does the job, but is very slow. The code is below, and takes 11s on a 300x300 array with a filter size of 31x31 on my computer. How could I use cython to speed things up? (I know that I should really pass the filter size and the filter sigma as arguments using e.g. extra_arguments, but I want to try and get it to run as fast as possible by computing the gaussian filter only once.) Thanks! Thomas --- import numpy as np from scipy.ndimage import generic_filter # size of filter (in both directions) s = 31 # center of filter (in both directions) c = 16 # sigma of filter (in both directions) sigma = 5. # define gaussian function def gaussian(cx, cy, w): return lambda x,y: np.exp(-(((cx-x)/w)**2+((cy-y)/w)**2)/2) # define gaussian filter x,y = np.mgrid[0:s,0:s] filt = gaussian(c,c,sigma)(x,y).ravel() # define custom filter def custom_filter(values): mask = np.where(np.isnan(values) == False) return np.sum(values[mask]*filt[mask])/np.sum(filt[mask]) # the function to test the custom filter def do(): a = np.ones((300,300)) b = generic_filter(a,custom_filter,size=s,mode='constant',cval=np.nan) --- From argriffi at ncsu.edu Fri Apr 24 23:51:03 2009 From: argriffi at ncsu.edu (alex) Date: Fri, 24 Apr 2009 23:51:03 -0400 Subject: [SciPy-user] Ignoring pixels with gaussian_filter In-Reply-To: <274C4D39-07A2-4236-B14C-6C104EE11352@gmail.com> References: <274C4D39-07A2-4236-B14C-6C104EE11352@gmail.com> Message-ID: <49F288A7.7040609@ncsu.edu> Thomas Robitaille wrote: > ... > # define gaussian function > def gaussian(cx, cy, w): > return lambda x,y: np.exp(-(((cx-x)/w)**2+((cy-y)/w)**2)/2) > ... A neat mathematical property of gaussian blur that is not true of 2d kernels in general is that it can be applied to the x and y axes separately. That is, it can be implemented as two 1d passes instead of one 2d pass. This could speed up your code a lot if you aren't already doing it. Alex From davide.cittaro at ifom-ieo-campus.it Sat Apr 25 02:59:02 2009 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Sat, 25 Apr 2009 08:59:02 +0200 Subject: [SciPy-user] canny1d filter? In-Reply-To: <2821C09D-5738-4BD8-9BBE-EF371D713B5A@yale.edu> References: <5C957654-B8F7-49B2-BAE6-4B1BA221096A@ifom-ieo-campus.it> <2821C09D-5738-4BD8-9BBE-EF371D713B5A@yale.edu> Message-ID: <0621D77C-2281-4EE6-977B-4063EAD2D14D@ifom-ieo-campus.it> Hi Zach, On Apr 25, 2009, at 1:30 AM, Zachary Pincus wrote: > Hi Davide, > > I can send you code for a 2D Canny filter that uses numpy and > scipy.ndimage to good effect. It's a decent demonstration of the > filter, and of how to make good use out of the tools in ndimage. > Can you send it, please? I would really appreciate... > However, I don't think that Canny edge-detection, as I understand it, > is applicable to 1D signaling. Basically, Canny filtering finds the > gradient magnitude and direction in an image, and then uses the > direction information to thin out edges except along their "ridges". > It then uses a hysteresis thresholding approach to link the edges (by > finding seeds with a stringent threshold, and then extending the seeds > out along the edges using a less-stringent threshold). None of this > stuff makes sense in the 1D case -- there's no "edge direction", etc. I see... I still have to read the original Canny paper, but AFAIK he used a 1d example in it (but I may be wrong...). Thanks, d From zachary.pincus at yale.edu Sat Apr 25 18:24:38 2009 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Sat, 25 Apr 2009 18:24:38 -0400 Subject: [SciPy-user] canny1d filter? In-Reply-To: <0621D77C-2281-4EE6-977B-4063EAD2D14D@ifom-ieo-campus.it> References: <5C957654-B8F7-49B2-BAE6-4B1BA221096A@ifom-ieo-campus.it> <2821C09D-5738-4BD8-9BBE-EF371D713B5A@yale.edu> <0621D77C-2281-4EE6-977B-4063EAD2D14D@ifom-ieo-campus.it> Message-ID: >> I can send you code for a 2D Canny filter that uses numpy and >> scipy.ndimage to good effect. It's a decent demonstration of the >> filter, and of how to make good use out of the tools in ndimage. >> > > Can you send it, please? I would really appreciate... Attached. The implementation of the hysteresis thresholding bit is worth noting, because the technique for doing this with ndimage was not obvious to me at first, but has proven quite useful in general: I use the lower threshold to make a mask, and then dilate the edge map thresholded at the higher threshold for -1 iterations (which makes ndimage keep dilating until no pixels change), using the lower threshold mask to constrain the dilation. This has the effect of "growing out" the regions that pass the higher threshold out into regions above the lower threshold. (I'm pretty sure this is a standard operation, given that it's available in ndimage, but like I said, it was not obvious to me at first.) Other than that, the implementation is basically straight from the wikipedia entry. >> However, I don't think that Canny edge-detection, as I understand it, >> is applicable to 1D signaling. Basically, Canny filtering finds the >> gradient magnitude and direction in an image, and then uses the >> direction information to thin out edges except along their "ridges". >> It then uses a hysteresis thresholding approach to link the edges (by >> finding seeds with a stringent threshold, and then extending the >> seeds >> out along the edges using a less-stringent threshold). None of this >> stuff makes sense in the 1D case -- there's no "edge direction", etc. > > I see... I still have to read the original Canny paper, but AFAIK he > used a 1d example in it (but I may be wrong...) I just looked over the paper -- there are plenty of 1D examples and calculations, but I think the results from that part aren't that useful: basically it says that there's a theoretical reason to use a DoG as the edge filter. It's probably interesting to read through the logic, because if your problem setting is different, then you could see if a different filter shape is optimal for what you're looking for. But as far as the "canny filter" as anyone would implement it, it's a 2D-only thing, because his real contribution was the 2D post- processing after doing a DoG. Zach -------------- next part -------------- A non-text attachment was scrubbed... Name: canny.py Type: text/x-python-script Size: 2174 bytes Desc: not available URL: -------------- next part -------------- From gruben at bigpond.net.au Sat Apr 25 23:18:59 2009 From: gruben at bigpond.net.au (Gary Ruben) Date: Sun, 26 Apr 2009 13:18:59 +1000 Subject: [SciPy-user] Looking for a way to cluster data Message-ID: <49F3D2A3.3060002@bigpond.net.au> Hi all, I'm looking for some advice on how to order data points so that I can visualise them. I've been looking at scipy.cluster for this purpose but I'm not sure whether it is suitable so I thought I'd see whether anyone had suggestions for a simpler suggestion of how to order the coordinates. I have a binary 3D array containing 1's that form a shape in a 3D volume against a background of 0's - they form a skeleton of a connected, branched structure. Furthermore, the points are all 26-connected to each other, i.e. there are no gaps in the skeleton. The longest chains may be 1000's of points long. It would be nice to visualise these using the mayavi mlab plot3d function, which draws tubes and which requires ordered coordinates as input, so I need to get ordered coordinate lists that traverse the points along the branches of the skeleton. It would also be nice to preferentially cluster long chains since then I can cull very short chains from the visualisation. scipy.cluster seems to be able to cluster the points but I'm not sure how to get the x,y,z coordinates of the original points out of its linkage data. This may not be possible. Maybe the scipy.spatial module is a better match to my problem. Any suggestions? Gary From tpk at kraussfamily.org Sun Apr 26 16:51:48 2009 From: tpk at kraussfamily.org (Tom K.) Date: Sun, 26 Apr 2009 13:51:48 -0700 (PDT) Subject: [SciPy-user] firwin upgrades Message-ID: <23246480.post@talk.nabble.com> Hi folks, About a month ago, a feature for high / pass / stop filters was requested in firwin (or somewhere in scipy.signal) (see http://projects.scipy.org/scipy/ticket/902). I'm looking at how that might be added and had a few alternatives I would like to run by the group: SUPPORT FOR OTHER FILTER TYPES - PROPOSALS 1) add "btype" kwarg after cutoff that may be any key of the band_dict mapping. Allow cutoff to be a list or 1D array, and make sure that the cutoff and btype arguments are consistent. h = firwin(N, .1) # lowpass h = firwin(N, .1, btype='high') # highpass h = firwin(N, [.1, .2]) # bandpass h = firwin(N, [.1, .2], btype='stop') # bandstop For this, I propose the btype defaults to None, and its value would be 'low' for length 1 cutoff, and 'pass' for length 2 cutoff, since I think bandpass filters are more common than bandstop (maybe I am wrong though). 2) Allow cutoff to be an arbitrary length list. Only need a boolean to indicate whether to start with a value of 0 at DC (the default), or start with a value of 1 at DC (for highpass and bandpass filters). I propose a new 'lowpass' boolean with default True for that in this option. For this, the calls would be e.g.: h = firwin(N, .1) # lowpass h = firwin(N, .1, lowpass=False) # highpass h = firwin(N, [.1, .2], lowpass=False) # bandpass h = firwin(N, [.1, .2]) # bandstop 3) Same as option 2), but instead of a new boolean argument, allow "0" as the first cutoff frequency: h = firwin(N, .1) # lowpass h = firwin(N, [0, .1]) # highpass h = firwin(N, [0, .1, .2]) # bandpass h = firwin(N, [.1, .2]) # bandstop In each of these cases, instead of scaling by 1./sum(h, axis=0), scale by the DTFT of the filter at the center of the first band. What are your preferences from the above (or, suggest an alternative)? NULL AT NYQUIST ISSUE Also, what do we want to do about having an even length filter with a passband at Nyquist (that is pi radians frequency)? All even length filters have a null at pi (because h[0]-h[1]+h[2]-h[3]... = 0 for linear phase, even length). If arguments indicate a passband at Nyquist (that is pi radians frequency), here are some options: a) issue a warning, increase length by 1, and return the longer filter [this is the behavior of another popular signal processing package] b) design the filter anyway, and issue either an error if noScale is False (since the scaling would cause a divide by 0 - see proposal below) or a warning if noScale is True. SUPPORT FOR NO SCALING Currently, the filter is scaled so that the DC value of the filter is unity. This filter no longer minimizes the integral of the square of the error because the scaling is not natural. I propose we provide a boolean "noScale" argument that will allow the filter to float according to the actual least-squares filter. How does that sound? -- View this message in context: http://www.nabble.com/firwin-upgrades-tp23246480p23246480.html Sent from the Scipy-User mailing list archive at Nabble.com. From zachary.pincus at yale.edu Sun Apr 26 19:51:55 2009 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Sun, 26 Apr 2009 19:51:55 -0400 Subject: [SciPy-user] Looking for a way to cluster data In-Reply-To: <49F3D2A3.3060002@bigpond.net.au> References: <49F3D2A3.3060002@bigpond.net.au> Message-ID: <98993A99-513E-4EAF-A33F-7F5B7FFD53F3@yale.edu> Hi Gary, > I'm looking for some advice on how to order data points so that I can > visualise them. I've been looking at scipy.cluster for this purpose > but > I'm not sure whether it is suitable so I thought I'd see whether > anyone > had suggestions for a simpler suggestion of how to order the > coordinates. > > I have a binary 3D array containing 1's that form a shape in a 3D > volume > against a background of 0's - they form a skeleton of a connected, > branched structure. Furthermore, the points are all 26-connected to > each > other, i.e. there are no gaps in the skeleton. The longest chains > may be > 1000's of points long. > It would be nice to visualise these using the mayavi mlab plot3d > function, which draws tubes and which requires ordered coordinates as > input, so I need to get ordered coordinate lists that traverse the > points along the branches of the skeleton. It would also be nice to > preferentially cluster long chains since then I can cull very short > chains from the visualisation. It sounds like what you want from the clustering is to get groups of pixels that form straight-ish lines (and could thus be visualized with 3D rods). Is this correct? Most clustering algorithms are likely to just give you groups of pixels that are nearby spatially -- which is probably not exactly what you want, since if you (say) have a long rod- structure, you'd want all the pixels in that rod grouped together, and not grouped with separate rods that cross nearby in space but aren't physically connected. So if you do want to cluster the pixels, you'll need to use the geodesic distance between two pixels, not the euclidian distance. But that still wouldn't give you sets of rods... more like rods and blobs at the junctions. Another option would be to try to detect junction points on the skeleton (I think this is a common operation -- sometimes in 2D people brute-force it by looking for all possible patterns of pixels indicative of branching, but there's probably a nicer way, especially for 3d). Once the junctions are known, a simple flood-fill along the skeleton gives you sets of pixels that lie between each junction. Having such a connectivity graph is a useful thing -- lots of neat stuff one can calculate from that. In fact, I think you'd need to do this to calculate geodesic distances anyway... Anyhow, you could then try fitting the points from each branch to a poly-line (using some information criterion for determining how many lines to use), to simplify the representation down to rods for plotting. But perhaps it's worth figuring out if there's another visualization method you could use, before going to all this effort... Would volume rendering work? Perhaps you could go as far as junction-point-finding and branch-labeling. Just doing that would let you exclude short branches (or short terminal branches) and then you could simply volume- render the rest and not fiddle with line-fitting. Zach On Apr 25, 2009, at 11:18 PM, Gary Ruben wrote: > Hi all, > > I'm looking for some advice on how to order data points so that I can > visualise them. I've been looking at scipy.cluster for this purpose > but > I'm not sure whether it is suitable so I thought I'd see whether > anyone > had suggestions for a simpler suggestion of how to order the > coordinates. > > I have a binary 3D array containing 1's that form a shape in a 3D > volume > against a background of 0's - they form a skeleton of a connected, > branched structure. Furthermore, the points are all 26-connected to > each > other, i.e. there are no gaps in the skeleton. The longest chains > may be > 1000's of points long. > It would be nice to visualise these using the mayavi mlab plot3d > function, which draws tubes and which requires ordered coordinates as > input, so I need to get ordered coordinate lists that traverse the > points along the branches of the skeleton. It would also be nice to > preferentially cluster long chains since then I can cull very short > chains from the visualisation. > > scipy.cluster seems to be able to cluster the points but I'm not sure > how to get the x,y,z coordinates of the original points out of its > linkage data. This may not be possible. Maybe the scipy.spatial module > is a better match to my problem. > > Any suggestions? > > Gary > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From tmbdev at gmail.com Mon Apr 27 01:01:25 2009 From: tmbdev at gmail.com (Thomas Breuel) Date: Mon, 27 Apr 2009 07:01:25 +0200 Subject: [SciPy-user] weave with C99 converters? Message-ID: <7e51d15d0904262201o5232afcmc2e5ad2b77a4a642@mail.gmail.com> I'm wondering whether there already are, or are planned to be, C99 converters for weave. These converters would rely on the dynamic array facilities of C99 instead of Blitz for representing multidimensional arrays: int n = 2*r, m = 3*r; float a[n][m]; ... Of course, "a" would have to be contiguous. But the advantage would be no dependence on a C++ compiler, faster compilation, and a simpler language to learn for end users. I suppose the same question could also be asked for weaving Fortran code (77, 90, or later; and, yes, I know about f2py). Tom -------------- next part -------------- An HTML attachment was scrubbed... URL: From gruben at bigpond.net.au Mon Apr 27 05:56:55 2009 From: gruben at bigpond.net.au (Gary Ruben) Date: Mon, 27 Apr 2009 19:56:55 +1000 Subject: [SciPy-user] Looking for a way to cluster data In-Reply-To: <98993A99-513E-4EAF-A33F-7F5B7FFD53F3@yale.edu> References: <49F3D2A3.3060002@bigpond.net.au> <98993A99-513E-4EAF-A33F-7F5B7FFD53F3@yale.edu> Message-ID: <49F58167.30403@bigpond.net.au> Hi Zach, Many thanks for your thoughts. I've had a good go at this myself now and am getting some useful results, although the processing is a bit slow at the moment. The approach I took is as follows: Using scipy.ndimage I label the 26-connected regions so that I can process each connected branch-like structure in turn. For each of these, I form a graph with nodes corresponding to pixels and edges corresponding to the Minimum Spanning Tree weighted by euclidean distance. I then use NetworkX to traverse this graph, extracting the node visitation order as a single list. Whenever this list makes jumps of distance >sqrt(2), it is jumping to a different branch, so I identify these jumps and subdivide the list into a list of lists, each containing nodes containing only one branch. I then plot these using plot3d. Problems with this approach are that generating the Minimum Spanning Tree in pure Python is slow - It turns out for the example I'm trying that I have one large region containing 25000 pixels. For the moment I've ignored this region and am only dealing with the smaller region which are only up to a few hundred pixels. I may need to look at Cython for this if I pursue this approach. I've made a few in-line responses to your comments. Zachary Pincus wrote: > Hi Gary, > It sounds like what you want from the clustering is to get groups of > pixels that form straight-ish lines (and could thus be visualized > with 3D rods). Is this correct? Actually, I probably want curved lines so your later comments and suggestions about fitting poly-lines are what I may try next. > Most clustering algorithms are likely > to just give you groups of pixels that are nearby spatially -- which > is probably not exactly what you want, since if you (say) have a long > rod- structure, you'd want all the pixels in that rod grouped > together, and not grouped with separate rods that cross nearby in > space but aren't physically connected. So if you do want to cluster > the pixels, you'll need to use the geodesic distance between two > pixels, not the euclidian distance. But that still wouldn't give you > sets of rods... more like rods and blobs at the junctions. I'm not sure what you mean by geodesic distance here. It's true that I have unwanted connections due to the proximity of regions to one another, but without some very fancy code to detect these cases, I think I'll be satisfied with my rough approximate method. I know one way of doing this is to look at the behaviour of tangent vectors to the curves but this is impractical in pure Python and probably not worth the effort for my purposes. > Another option would be to try to detect junction points on the > skeleton (I think this is a common operation -- sometimes in 2D > people brute-force it by looking for all possible patterns of pixels > indicative of branching, but there's probably a nicer way, > especially for 3d). Once the junctions are known, a simple flood-fill > along the skeleton gives you sets of pixels that lie between each > junction. > > Having such a connectivity graph is a useful thing -- lots of neat > stuff one can calculate from that. In fact, I think you'd need to do > this to calculate geodesic distances anyway... Anyhow, you could > then try fitting the points from each branch to a poly-line (using > some information criterion for determining how many lines to use), to > simplify the representation down to rods for plotting. I'm planning to do this next. I'm hoping I can think of a very simple criterion. > But perhaps it's worth figuring out if there's another visualization > method you could use, before going to all this effort... Would > volume rendering work? It may. I haven't tried volume rendering yet, but rendering tubes is the ideal. I have tried just isosurfacing the detected pixels and this isn't so bad, but tubes give the eye extra depth cues that I'm hoping to exploit to make things look more informative. > Perhaps you could go as far as > junction-point-finding and branch-labeling. Just doing that would let > you exclude short branches (or short terminal branches) and then you > could simply volume- render the rest and not fiddle with > line-fitting. I'll think about this too, but I may try straight volume rendering first and see how it looks. > Zach Thanks for taking the time to respond. regards, Gary From etrade.griffiths at dsl.pipex.com Mon Apr 27 05:49:21 2009 From: etrade.griffiths at dsl.pipex.com (etrade.griffiths at dsl.pipex.com) Date: Mon, 27 Apr 2009 10:49:21 +0100 Subject: [SciPy-user] problem with fmin_powell Message-ID: <1240825761.49f57fa163015@netmail.pipex.net> Hi I am running an optimisation program using fmin_powell with the following statement: vals = scipy.optimize.fmin_powell(my_obj.ix_func,ivalues,maxiter=my_obj.maxiter,full_output=True) Scipy makes around 10 calls to the function my_obj.ix_func then crashes with these error messages: Traceback (most recent call last): File "D:\AlunG\2008\convergence\test_script.py", line 383, in vals = scipy.optimize.fmin_powell(my_obj.ix_func,ivalues,maxiter=my_obj.maxiter,full_output=True) File "C:\Python25\lib\site-packages\scipy-0.6.0.0006-py2.5-win32.egg\scipy\optimize\optimize.py", line 1852, in fmin_powell fval, x, direc1 = _linesearch_powell(func, x, direc1, tol=xtol*100) File "C:\Python25\lib\site-packages\scipy-0.6.0.0006-py2.5-win32.egg\scipy\optimize\optimize.py", line 1737, in _linesearch_powell alpha_min, fret, iter, num = brent(myfunc, full_output=1, tol=tol) File "C:\Python25\lib\site-packages\scipy-0.6.0.0006-py2.5-win32.egg\scipy\optimize\optimize.py", line 1549, in brent brent.optimize() File "C:\Python25\lib\site-packages\scipy-0.6.0.0006-py2.5-win32.egg\scipy\optimize\optimize.py", line 1432, in optimize tmp1 = (x-w)*(fx-fv) TypeError: unsupported operand type(s) for -: 'str' and 'str' I have no problems if I use fmin instead of fmin_powell. I also tried passing the argument direc as a unit vector ([1.,1.,1.,1.] in this case) but have the same problem. I am using Python 2.5, SciPy 0.6, NumPy 1.5.2. All suggestions gratefully received Alun Griffiths ------------------------------------------------- Visit Pipex Business: The homepage for UK Small Businesses Go to http://www.pipex.co.uk/business-services From sebastian.walter at gmail.com Mon Apr 27 06:52:37 2009 From: sebastian.walter at gmail.com (Sebastian Walter) Date: Mon, 27 Apr 2009 12:52:37 +0200 Subject: [SciPy-user] problem with fmin_powell In-Reply-To: <1240825761.49f57fa163015@netmail.pipex.net> References: <1240825761.49f57fa163015@netmail.pipex.net> Message-ID: The source of the full example would be a lot more helpful... My bet is that something is wrong with your objective function. TypeError: unsupported operand type(s) for -: 'str' and 'str' might possibly be the result of an objective function that uses global variables that are set differently in the context of fmin_powell. But that's just a wild guess. On Mon, Apr 27, 2009 at 11:49 AM, wrote: > Hi > > I am running an optimisation program using fmin_powell with the following statement: > > vals = scipy.optimize.fmin_powell(my_obj.ix_func,ivalues,maxiter=my_obj.maxiter,full_output=True) > > Scipy makes around 10 calls to the function my_obj.ix_func then crashes with these error messages: > > Traceback (most recent call last): > ?File "D:\AlunG\2008\convergence\test_script.py", line 383, in > ? ?vals = scipy.optimize.fmin_powell(my_obj.ix_func,ivalues,maxiter=my_obj.maxiter,full_output=True) > ?File "C:\Python25\lib\site-packages\scipy-0.6.0.0006-py2.5-win32.egg\scipy\optimize\optimize.py", line 1852, in > fmin_powell > ? ?fval, x, direc1 = _linesearch_powell(func, x, direc1, tol=xtol*100) > ?File "C:\Python25\lib\site-packages\scipy-0.6.0.0006-py2.5-win32.egg\scipy\optimize\optimize.py", line 1737, in > _linesearch_powell > ? ?alpha_min, fret, iter, num = brent(myfunc, full_output=1, tol=tol) > ?File "C:\Python25\lib\site-packages\scipy-0.6.0.0006-py2.5-win32.egg\scipy\optimize\optimize.py", line 1549, in > brent > ? ?brent.optimize() > ?File "C:\Python25\lib\site-packages\scipy-0.6.0.0006-py2.5-win32.egg\scipy\optimize\optimize.py", line 1432, in > optimize > ? ?tmp1 = (x-w)*(fx-fv) > TypeError: unsupported operand type(s) for -: 'str' and 'str' > > I have no problems if I use fmin instead of fmin_powell. ?I also tried passing the argument direc as a unit vector > ([1.,1.,1.,1.] in this case) but have the same problem. ?I am using Python 2.5, SciPy 0.6, NumPy 1.5.2. ?All > suggestions gratefully received > > Alun Griffiths > > > > > ------------------------------------------------- > Visit Pipex Business: The homepage for UK Small Businesses > > Go to http://www.pipex.co.uk/business-services > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From davide.cittaro at ifom-ieo-campus.it Mon Apr 27 08:28:53 2009 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Mon, 27 Apr 2009 14:28:53 +0200 Subject: [SciPy-user] canny1d filter? In-Reply-To: References: <5C957654-B8F7-49B2-BAE6-4B1BA221096A@ifom-ieo-campus.it> <2821C09D-5738-4BD8-9BBE-EF371D713B5A@yale.edu> <0621D77C-2281-4EE6-977B-4063EAD2D14D@ifom-ieo-campus.it> Message-ID: Hi Zach, > > This has the effect of "growing out" the regions that pass the > higher threshold out into regions above the lower threshold. (I'm > pretty sure this is a standard operation, given that it's available > in ndimage, but like I said, it was not obvious to me at first.) > Other than that, the implementation is basically straight from the > wikipedia entry. > Got it! Indeed this is much more elegant than other implementations I've seen (where each magnitude is tested for being higher, lower or within...) > I just looked over the paper -- there are plenty of 1D examples and > calculations, but I think the results from that part aren't that > useful: basically it says that there's a theoretical reason to use a > DoG as the edge filter. It's probably interesting to read through > the logic, because if your problem setting is different, then you > could see if a different filter shape is optimal for what you're > looking for. But as far as the "canny filter" as anyone would > implement it, it's a 2D-only thing, because his real contribution > was the 2D post-processing after doing a DoG. I've asked to my institution to get the paper (it is not in our library... you know, here's all about biology ^__^) so I can figure how it works. BTW, I'm testing a 1d implementation (basically a 2d with only W footprint and no angles), I still have some doubt on choosing the thresholds... bah... I'll take some time to think about it. Thanks d /* Davide Cittaro Cogentech - Consortium for Genomic Technologies via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro at ifom-ieo-campus.it */ -------------- next part -------------- An HTML attachment was scrubbed... URL: From etrade.griffiths at dsl.pipex.com Mon Apr 27 08:49:00 2009 From: etrade.griffiths at dsl.pipex.com (etrade.griffiths at dsl.pipex.com) Date: Mon, 27 Apr 2009 13:49:00 +0100 Subject: [SciPy-user] problem with fmin_powell In-Reply-To: References: Message-ID: <1240836540.49f5a9bccf74e@netmail.pipex.net> Sebastian your wild guess was not so wild! The objective function returns the CPU time used by an external program with the current settings of the variables. I got this value from a text file and forgot to return it as a float, hence the problems SCIPY had trying to operate on strings. Thanks for the hint! Best regards Alun Griffiths ------------------------------------------------- Visit Pipex Business: The homepage for UK Small Businesses Go to http://www.pipex.co.uk/business-services From josef.pktd at gmail.com Mon Apr 27 11:48:09 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 27 Apr 2009 11:48:09 -0400 Subject: [SciPy-user] goodness-of-fit test - power discrepancy Message-ID: <1cd32cbb0904270848u91d9271oc8d8f4cd9bb3c78c@mail.gmail.com> A while ago, I wrote a function for power discrepancy which is a generalization of the chi-square test for goodness-of-fit for discrete samples or probabilities. If someone wants to try it, I would welcome comments, before adding it to scipy.stats. Josef -------------- next part -------------- ''' Author: Josef Perktold and pymc authors ''' import numpy as np from numpy.linalg.linalg import LinAlgError import numpy.linalg as nplinalg from numpy.testing import assert_almost_equal, assert_equal, assert_array_almost_equal from scipy import linalg, stats #JP: note: Freeman-Tukey statistics is a goodness-of-fit test for # discrete data, ## other gof tests ## The test statistics: ## Pearson?s Chi-Square, ## the Kolmogorov-Smirnov test statistic for discrete data, ## the Log-Likelihood Ratio, ## the Freeman-Tukey and ## the Power Divergence statistic with ?=?. def discrepancy(observed, simulated, expected): """Calculates Freeman-Tukey statistics (Freeman and Tukey 1950) as a measure of discrepancy between observed and simulated data. This is a convenient method for assessing goodness-of-fit (see Brooks et al. 2000). D(x|\theta) = \sum_j (\sqrt{x_j} - \sqrt{e_j})^2 :Parameters: observed : Iterable of observed values simulated : Iterable of simulated values expected : Iterable of expeted values :Returns: D_obs : Discrepancy of observed values D_sim : Discrepancy of simulated values Notes: ------ this is the pymc version """ D_obs = np.sum([(np.sqrt(x)-np.sqrt(e))**2 for x,e in zip(observed, expected)]) D_sim = np.sum([(np.sqrt(s)-np.sqrt(e))**2 for s,e in zip(simulated, expected)]) return D_obs, D_sim def powerdiscrepancy(o, e, lambd=0.0, axis=0): """Calculates power discrepancy, a class of goodness-of-fit tests as a measure of discrepancy between observed and expected data. This contains several goodness-of-fit tests as special cases, see the describtion of lambd, the exponent of the power discrepancy. The pvalue is based on the asymptotic chi-square distribution of the test statistic. freeman_tukey: D(x|\theta) = \sum_j (\sqrt{x_j} - \sqrt{e_j})^2 Parameters ---------- o : Iterable of observed values e : Iterable of expeted values lambd : float or string * float : exponent `a` for power discrepancy * 'loglikeratio': a = 0 * 'freeman_tukey': a = -0.5 * 'pearson': a = 1 * 'modified_loglikeratio': a = -1 * 'cressie_read': a = 2/3 Returns ------- D_obs : Discrepancy of observed values pvalue : pvalue References ---------- Cressie, Noel and Timothy R. C. Read, Multinomial Goodness-of-Fit Tests, Journal of the Royal Statistical Society. Series B (Methodological), Vol. 46, No. 3 (1984), pp. 440-464 Campbell B. Read: Freeman-Tukey chi-squared goodness-of-fit statistics, Statistics & Probability Letters 18 (1993) 271-278 Nobuhiro Taneichi, Yuri Sekiya, Akio Suzukawa, Asymptotic Approximations for the Distributions of the Multinomial Goodness-of-Fit Statistics under Local Alternatives, Journal of Multivariate Analysis 81, 335?359 (2002) Steele, M. 1,2, C. Hurst 3 and J. Chaseling, Simulated Power of Discrete Goodness-of-Fit Tests for Likert Type Data Examples -------- >>> observed = np.array([ 2., 4., 2., 1., 1.]) >>> expected = np.array([ 0.2, 0.2, 0.2, 0.2, 0.2]) for checking correct dimension with multiple series >>> powerdiscrepancy(np.column_stack((observed,observed)).T, 10*expected, lambd='freeman_tukey',axis=1) (array([[ 2.745166, 2.745166]]), array([[ 0.6013346, 0.6013346]])) >>> powerdiscrepancy(np.column_stack((observed,observed)).T, 10*expected,axis=1) (array([[ 2.77258872, 2.77258872]]), array([[ 0.59657359, 0.59657359]])) >>> powerdiscrepancy(np.column_stack((observed,observed)).T, 10*expected, lambd=0,axis=1) (array([[ 2.77258872, 2.77258872]]), array([[ 0.59657359, 0.59657359]])) >>> powerdiscrepancy(np.column_stack((observed,observed)).T, 10*expected, lambd=1,axis=1) (array([[ 3., 3.]]), array([[ 0.5578254, 0.5578254]])) >>> powerdiscrepancy(np.column_stack((observed,observed)).T, 10*expected, lambd=2/3.0,axis=1) (array([[ 2.89714546, 2.89714546]]), array([[ 0.57518277, 0.57518277]])) >>> powerdiscrepancy(np.column_stack((observed,observed)).T, expected, lambd=2/3.0,axis=1) (array([[ 2.89714546, 2.89714546]]), array([[ 0.57518277, 0.57518277]])) >>> powerdiscrepancy(np.column_stack((observed,observed)), expected, lambd=2/3.0, axis=0) (array([[ 2.89714546, 2.89714546]]), array([[ 0.57518277, 0.57518277]])) each random variable can have different total count/sum >>> powerdiscrepancy(np.column_stack((observed,2*observed)), expected, lambd=2/3.0, axis=0) (array([[ 2.89714546, 5.79429093]]), array([[ 0.57518277, 0.21504648]])) >>> powerdiscrepancy(np.column_stack((observed,2*observed)), 10*expected, lambd=2/3.0, axis=0) Traceback (most recent call last): ... ValueError: observed and expected need to have the samenumber of observations, or e needs to add to 1 >>> powerdiscrepancy(np.column_stack((2*observed,2*observed)), expected, lambd=2/3.0, axis=0) (array([[ 5.79429093, 5.79429093]]), array([[ 0.21504648, 0.21504648]])) >>> powerdiscrepancy(np.column_stack((2*observed,2*observed)), 20*expected, lambd=2/3.0, axis=0) (array([[ 5.79429093, 5.79429093]]), array([[ 0.21504648, 0.21504648]])) >>> powerdiscrepancy(np.column_stack((observed,2*observed)), np.column_stack((10*expected,20*expected)), lambd=2/3.0, axis=0) (array([[ 2.89714546, 5.79429093]]), array([[ 0.57518277, 0.21504648]])) >>> powerdiscrepancy(np.column_stack((observed,2*observed)), np.column_stack((10*expected,20*expected)), lambd=-1, axis=0) (array([[ 2.77258872, 5.54517744]]), array([[ 0.59657359, 0.2357868 ]])) """ o = np.array(o) e = np.array(e) if np.isfinite(lambd) == True: # check whether lambd is a number a = lambd else: if lambd == 'loglikeratio': a = 0 elif lambd == 'freeman_tukey': a = -0.5 elif lambd == 'pearson': a = 1 elif lambd == 'modified_loglikeratio': a = -1 elif lambd == 'cressie_read': a = 2/3.0 else: raise ValueError, 'lambd has to be a number or one of ' + \ 'loglikeratio, freeman_tukey, pearson, ' +\ 'modified_loglikeratio or cressie_read' n = np.sum(o, axis=axis) nt = n if n.size>1: n = np.atleast_2d(n) if axis == 1: nt = n.T # need both for 2d, n and nt for broadcasting if e.ndim == 1: e = np.atleast_2d(e) if axis == 0: e = e.T if np.all(np.sum(e, axis=axis) == n): p = e/(1.0*nt) elif np.all(np.sum(e, axis=axis) == 1): p = e e = nt * e else: raise ValueError, 'observed and expected need to have the same' \ 'number of observations, or e needs to add to 1' k = o.shape[axis] if e.shape[axis] != k: raise ValueError, 'observed and expected need to have the same' \ 'number of bins' # Note: taken from formulas, to simplify cancel n if a == 0: # log likelihood ratio D_obs = 2*n * np.sum(o/(1.0*nt) * np.log(o/e), axis=axis) elif a == -1: # modified log likelihood ratio D_obs = 2*n * np.sum(e/(1.0*nt) * np.log(e/o), axis=axis) else: D_obs = 2*n/a/(a+1) * np.sum(o/(1.0*nt) * ((o/e)**a - 1), axis=axis) return D_obs, stats.chi2.sf(D_obs,k-1) def itemfreq2d(a, scores=None, axis=0): # JP: from scipy.stats.stats.py with changes """Returns a 2D array of item frequencies. Column 1 contains item values, column 2 to n contain their respective counts. Parameters ---------- a : array 1D or 2D data, either columns or rows represent variables, see describtion for axis scores : array_like (optional) Contains list or array of items for which frequencies are found. If scores is None, then the frequency is found for those items which occur at least once. If scores are given, then also items that do not occur in the data are listed in the frequency table axis : 0, 1 or None If axis = 0, then the frequency count is calculated for each column. If axis = 1, then the frequency count is calculated for each row. If axis = None, then the frequency count is calculated for the entire data, (array is raveled first). Returns ------- A 2D frequency table (col [0]=scores, col [1:n]=frequencies) >>> rvs = np.array([[4, 6, 3], [3, 6, 4], [4, 7, 6], [6, 1, 6]]) >>> itemfreq2d(rvs,range(1,8),axis=1) array([[ 1., 0., 0., 0., 1.], [ 2., 0., 0., 0., 0.], [ 3., 1., 1., 0., 0.], [ 4., 1., 1., 1., 0.], [ 5., 0., 0., 0., 0.], [ 6., 1., 1., 1., 2.], [ 7., 0., 0., 1., 0.]]) >>> itemfreq2d(rvs,range(1,8),axis=0) array([[ 1., 0., 1., 0.], [ 2., 0., 0., 0.], [ 3., 1., 0., 1.], [ 4., 2., 0., 1.], [ 5., 0., 0., 0.], [ 6., 1., 2., 2.], [ 7., 0., 1., 0.]]) >>> itemfreq2d(rvs,axis=1) array([[ 1., 0., 0., 0., 1.], [ 3., 1., 1., 0., 0.], [ 4., 1., 1., 1., 0.], [ 6., 1., 1., 1., 2.], [ 7., 0., 0., 1., 0.]]) >>> itemfreq2d(rvs,axis=0) array([[ 1., 0., 1., 0.], [ 3., 1., 0., 1.], [ 4., 2., 0., 1.], [ 6., 1., 2., 2.], [ 7., 0., 1., 0.]]) >>> itemfreq2d(rvs[:,0],range(1,8)) array([[ 1., 0.], [ 2., 0.], [ 3., 1.], [ 4., 2.], [ 5., 0.], [ 6., 1.], [ 7., 0.]]) >>> itemfreq2d(rvs[:,0]) array([[ 3., 1.], [ 4., 2.], [ 6., 1.]]) >>> itemfreq2d(rvs,axis=None) array([[ 1., 1.], [ 3., 2.], [ 4., 3.], [ 6., 5.], [ 7., 1.]]) >>> itemfreq2d(rvs,range(1,8),axis=None) array([[ 1., 1.], [ 2., 0.], [ 3., 2.], [ 4., 3.], [ 5., 0.], [ 6., 5.], [ 7., 1.]]) """ if axis is None: a = a.ravel() axis = 0 if a.ndim == 1: k = 1 elif a.ndim == 2: k = a.shape[1-axis] if scores is None: #scores = stats._support.unique(a.ravel()) scores = np.unique(a.ravel()) scores = np.sort(scores) freq = np.zeros((len(scores),k)) for i in range(len(scores)): freq[i] = np.add.reduce(np.equal(a,scores[i]),axis=axis) #return np.array(stats._support.abut(scores, freq)) return np.array(np.column_stack((scores, freq))) if __name__ == '__main__': import doctest doctest.testmod(verbose=True) #rvsunif = stats.randint.rvs(1,5,size=10) rvs0 = np.array([3, 2, 5, 2, 1, 2, 4, 3, 2, 1]) # get requencies with old version freq = {}; for s in range(1,6): freq[s]=0 freq.update(stats.itemfreq(rvs0)) # get requencies with new version observed = itemfreq2d(rvs0,range(1,6),axis=0)[:,1] expected = np.ones(5)/5.0 print powerdiscrepancy(observed, expected, lambd=0) print 'the next two are identical Pearson chisquare' print powerdiscrepancy(observed, expected, lambd=1) print stats.chisquare(observed, 10*expected) assert_array_almost_equal(powerdiscrepancy(observed, expected, lambd=1), stats.chisquare(observed, 10*expected)) print 'the next two are identical freeman_tukey' print powerdiscrepancy(observed, expected, lambd='freeman_tukey') print discrepancy(observed, observed, 10*expected)[0]*4 assert_array_almost_equal(powerdiscrepancy(observed, expected, lambd='freeman_tukey')[0], discrepancy(observed, observed, 10*expected)[0]*4) powerdiscrepancy(np.column_stack((observed,observed)), 10*expected[:,np.newaxis], lambd='freeman_tukey') powerdiscrepancy(np.column_stack((observed,2*observed)), np.column_stack((10*expected,20*expected)), lambd=-1, axis=0) #print powerdiscrepancy(np.array(sorted(freq.items())), [1/4.0]*4, lambd=0) ## print powerdiscrepancy(np.array(sorted(freq.items()))[:,1], np.asarray([1/4.0]*4), lambd=0) ## print 'the next two are identical Pearson chisquare' ## print powerdiscrepancy(np.array(sorted(freq.items()))[:,1], np.asarray([1/4.0]*4), lambd=1) ## print stats.chisquare(np.array(sorted(freq.items()))[:,1], 10*np.asarray([1/4.0]*4)) ## print 'the next two are identical freeman_tukey' ## print powerdiscrepancy(np.array(sorted(freq.items()))[:,1], np.asarray([1/4.0]*4), lambd='freeman_tukey') ## print discrepancy(np.array(sorted(freq.items()))[:,1],np.array(sorted(freq.items()))[:,1], 10*np.asarray([1/4.0]*4) )[0]*4 From josef.pktd at gmail.com Mon Apr 27 12:28:51 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 27 Apr 2009 12:28:51 -0400 Subject: [SciPy-user] bootstrap statistics, higher moments Message-ID: <1cd32cbb0904270928y390f5069m6bfccfc4c0d8b116@mail.gmail.com> (continuing the bug-hunting adventure) Until now I was missing a good test for higher order moments in stats.distributions. I finally figured out that I can use bootstrap estimation to get a confidence interval for the tests of skew and kurtosis, or 3rd and 4th moments. However, the bootstrap variance of the estimator is not very good with my "primitive" bootstrap. Does anyone have a good bootstrap function, that can be used for skew and kurtosis tests for samples from arbitrary distributions? --- The first incorrect results in stats.distribution, that I found and checked, are the raw moments for the non-central F distribution, stats.ncf.moment. The only explicit reference I found is http://mathworld.wolfram.com/NoncentralF-Distribution.html However, their general formula for the raw moments is identical to the current implementation, and I don't know whether the formula is wrong or whether there are some problems with the numerical implementation. The explicit formulas for the first 4 raw moments produce the same result as the generic numerical integration.The variance in stats.ncf.stats() is also a little bit off (not enough to be caught by the existing tests). I didn't find any moments for the ncf-distribution in R for comparison. Does mathworld have typos? Or are there some differences in the definitions of the special functions? I'm just curious, I will use the explicit formulas in stats.ncf. Josef >>> args (10, 20, 0.41578399999999999) >>> stats.ncf.moment(4, *args) 0.97927472463095944 moment 1 stats 1.15730933333 integration 1.15730292082 explicit 1.15730933333 moment 2 stats 1.78949632988 integration 1.80763055597 explicit 1.80766239354 moment 3 stats 1.41137486702 integration 3.761650183 explicit 3.76366631204 moment 4 stats 0.979274724631 integration 10.4651526972 explicit 10.4455970627 From zachary.pincus at yale.edu Mon Apr 27 12:32:43 2009 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Mon, 27 Apr 2009 12:32:43 -0400 Subject: [SciPy-user] Looking for a way to cluster data In-Reply-To: <49F58167.30403@bigpond.net.au> References: <49F3D2A3.3060002@bigpond.net.au> <98993A99-513E-4EAF-A33F-7F5B7FFD53F3@yale.edu> <49F58167.30403@bigpond.net.au> Message-ID: <95BE990C-83E7-44E3-86BD-638F9BA043D7@yale.edu> Hi Gary, > Using scipy.ndimage I label the 26-connected regions so that I can > process each connected branch-like structure in turn. For each of > these, > I form a graph with nodes corresponding to pixels and edges > corresponding to the Minimum Spanning Tree weighted by euclidean > distance. I then use NetworkX to traverse this graph, extracting the > node visitation order as a single list. Whenever this list makes jumps > of distance >sqrt(2), it is jumping to a different branch, so I > identify > these jumps and subdivide the list into a list of lists, each > containing > nodes containing only one branch. I then plot these using plot3d. This sounds like a pretty good approach. Unfortunately, as you've recognized, any operations that loop over pixels in pure python are pretty much too slow to use for large images. I definitely think cython might be a good thing to look into. Another possibility, if you were coding this in c or cython, would be to just do a flood-fill directly. (You could do the recursive algorithm, or just use a stack. Either way it's simple.) Every time you encounter a branch-point, increment a counter and use that new value for the "flood-fill" color. Then you have a labeled image where each branch has a given value. Conveniently, ndimage has lots of useful tools for dealing with such label images. You could also build up the connectivity graph during the flood-fill (nodes = branch points, edge weights = the number of pixels along each branch), for calculating the MST. >> It sounds like what you want from the clustering is to get groups of >> pixels that form straight-ish lines (and could thus be visualized >> with 3D rods). Is this correct? > > Actually, I probably want curved lines so your later comments and > suggestions about fitting poly-lines are what I may try next. I have another idea that might be good: take the x,y,z coords of each pixel along each branch, and fit a parametric smoothing spline through them with scipy.interpolate.splprep (use a non-zero / non-None smoothing parameter 's'). Then to render the curves, just evaluate the spline (scipy.interpolate.splev) at however many internal points you want, to give a poly-line. Use more points to get better resolution... I do this a lot when I need to "post-filter" binarized image data into smooth continuous arcs. >> Most clustering algorithms are likely >> to just give you groups of pixels that are nearby spatially -- which >> is probably not exactly what you want, since if you (say) have a long >> rod- structure, you'd want all the pixels in that rod grouped >> together, and not grouped with separate rods that cross nearby in >> space but aren't physically connected. So if you do want to cluster >> the pixels, you'll need to use the geodesic distance between two >> pixels, not the euclidian distance. But that still wouldn't give you >> sets of rods... more like rods and blobs at the junctions. > > I'm not sure what you mean by geodesic distance here. The geodesic distance between two pixels would be the shortest path *along the binarzied skeleton* between those pixels. As opposed to the shortest straight-line between the points. This would be relevant, say, if you have two branches that are nearby in space, but rather far from one another in terms of connectivity -- then using a euclidian distance to cluster pixels might put neaby pixels from the two different branches in the same cluster. >> Having such a connectivity graph is a useful thing -- lots of neat >> stuff one can calculate from that. In fact, I think you'd need to do >> this to calculate geodesic distances anyway... Anyhow, you could >> then try fitting the points from each branch to a poly-line (using >> some information criterion for determining how many lines to use), to >> simplify the representation down to rods for plotting. > > I'm planning to do this next. I'm hoping I can think of a very simple > criterion. Check out the standard, if maybe hokey, information criteria for model selection, like AIC and BIC. They offer semi-principled ways to trade off goodness-of-fit vs. model parsimony. http://en.wikipedia.org/wiki/Akaike_information_criterion and related pages. Zach From magnusp at astro.su.se Mon Apr 27 13:59:28 2009 From: magnusp at astro.su.se (n.l.o) Date: Mon, 27 Apr 2009 10:59:28 -0700 (PDT) Subject: [SciPy-user] scipy.interpolate.bisplrep MemoryError Message-ID: <23261686.post@talk.nabble.com> Hi, I am trying to interpolate a 512x512 floating-point array with scipy.interpolate.bisplrep and bisplev. First some info about the variables: SIZE_X=SIZE_Y=512, a90.shape=(512,512) dx90 and dy90 are sub-pixel,i.e floating-point shifts Then the code: x,y = mgrid[0:SIZE_X, 0:SIZE_Y] tck = interpolate.bisplrep(x,y,a90) xnew = x - dx90 ynew = x - dy90 a90new = interpolate.bisplev(xnew,ynew,tck) BUT at tck = interpolate.bisplrep(x,y,a90) i get: --------------------------------------------------------------------------- MemoryError Traceback (most recent call last) /home/magnusp/msc/reduction/reduce.py in () 657 658 x,y = mgrid[0:SIZE_X, 0:SIZE_Y] --> 659 tck = interpolate.bisplrep(x,y,a90) 660 xnew = x - dx90 661 ynew = x - dy90 /usr/lib/python2.6/dist-packages/scipy/interpolate/fitpack.py in bisplrep(x, y, z, w, xb, xe, yb, ye, kx, ky, task, s, eps, tx, ty, full_output, nxest, nyest, quiet) 760 except OverflowError: 761 raise OverflowError("Too many data points to interpolate") --> 762 tx,ty,c,o = _fitpack._surfit(x,y,z,w,xb,xe,yb,ye,kx,ky,task,s,eps, 763 tx,ty,nxest,nyest,wrk,lwrk1,lwrk2) 764 _curfit_cache['tx']=tx MemoryError: WARNING: Failure executing file: What can I do to make it work? Cheers, Magnus -- View this message in context: http://www.nabble.com/scipy.interpolate.bisplrep-MemoryError-tp23261686p23261686.html Sent from the Scipy-User mailing list archive at Nabble.com. From peridot.faceted at gmail.com Mon Apr 27 14:51:26 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Mon, 27 Apr 2009 14:51:26 -0400 Subject: [SciPy-user] Looking for a way to cluster data In-Reply-To: <49F3D2A3.3060002@bigpond.net.au> References: <49F3D2A3.3060002@bigpond.net.au> Message-ID: 2009/4/25 Gary Ruben : > Hi all, > > I'm looking for some advice on how to order data points so that I can > visualise them. I've been looking at scipy.cluster for this purpose but > I'm not sure whether it is suitable so I thought I'd see whether anyone > had suggestions for a simpler suggestion of how to order the coordinates. > > I have a binary 3D array containing 1's that form a shape in a 3D volume > against a background of 0's - they form a skeleton of a connected, > branched structure. Furthermore, the points are all 26-connected to each > other, i.e. there are no gaps in the skeleton. The longest chains may be > 1000's of points long. > It would be nice to visualise these using the mayavi mlab plot3d > function, which draws tubes and which requires ordered coordinates as > input, so I need to get ordered coordinate lists that traverse the > points along the branches of the skeleton. It would also be nice to > preferentially cluster long chains since then I can cull very short > chains from the visualisation. > > scipy.cluster seems to be able to cluster the points but I'm not sure > how to get the x,y,z coordinates of the original points out of its > linkage data. This may not be possible. Maybe the scipy.spatial module > is a better match to my problem. > > Any suggestions? If I understand you correctly, what you have is not a list of coordinates of freely-located points, it's a binary mask indicating which voxels are part of your object. So first of all, have you considered using volumetric visualization tools? These seem like they might be a better fit to your problem. If what you want to know about is the connectivity of your object, though, I can see why you might want to build chains of rods. The most direct approach is, for each cell that is a 1, to draw rods from it to each of its neighbors that is on. This may not give you what you want: if you have regions where all the cells are on, they'll be a dense grid of rods. It will also not allow you to provide long strings of rods to your 3D toolkit, or to eliminate short chains. As I see it, then, your problem is graph-theoretic: you have this fairly dense adjacency graph of "on" cells, and you want to pare it down. One good choice would be to produce a (minimum diameter?) spanning tree, which should be pretty easy to convert to a collection of strings of rods. But I think what you want is a graph library of some sort. On the other hand, if what you have is "fat" chains of on cells, and you want to build up a "skeleton" of them (like converting the pixels of an image of the letter o back to a circle), you might look at machine vision for help, they do this sort of thing often. Anne From pav at iki.fi Mon Apr 27 16:46:56 2009 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 27 Apr 2009 20:46:56 +0000 (UTC) Subject: [SciPy-user] scipy.interpolate.bisplrep MemoryError References: <23261686.post@talk.nabble.com> Message-ID: Mon, 27 Apr 2009 10:59:28 -0700, n.l.o wrote: > I am trying to interpolate a 512x512 floating-point array with > scipy.interpolate.bisplrep and bisplev. [clip] > x,y = mgrid[0:SIZE_X, 0:SIZE_Y] > tck = interpolate.bisplrep(x,y,a90) > xnew = x - dx90 > ynew = x - dy90 > a90new = interpolate.bisplev(xnew,ynew,tck) > > BUT at tck = interpolate.bisplrep(x,y,a90) i get: [clip] > MemoryError: > WARNING: Failure executing file: > > What can I do to make it work? Try using the interpolation tools from ndimage, or RectBivariateSpline, instead; your data is on a regular grid so specialized algorithms should be used. -- Pauli Virtanen From wesmckinn at gmail.com Mon Apr 27 17:05:21 2009 From: wesmckinn at gmail.com (Wes McKinney) Date: Mon, 27 Apr 2009 17:05:21 -0400 Subject: [SciPy-user] Managing Python with NumPy and many external libraries on multiple Windows machines Message-ID: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> Hello, I am wondering if anyone can offer some suggestions on this problem. Over the last year or so I have been building a number of libraries on top of NumPy + SciPy + matplotlib and other libraries which are being used for investigative research for my company's problem domain in place of, say, Matlab and R (which are more "ready out of the box" systems). I have approximately 20 users, all of whom are running Windows on a very Microsoft-centric network with databases, etc. Has anyone had any luck managing a standardized Python environment on lots of Windows machines with a large number of 3rd-party Python libraries? Upgrading packages right now involves getting 20 people to click through an .exe installer, which is hardly a good solution. For example, I was recently forced to upgrade everyone's NumPy to 1.3 after I discovered that a DLL I had built against 1.3 was incompatible with 1.2.x. It seems like on a Unix / OS X environment with root access this problem would be fairly easy to script away, but on Windows it's rather painful. Any advice would be much appreciated. Thanks, Wes -------------- next part -------------- An HTML attachment was scrubbed... URL: From dickinsm at gmail.com Mon Apr 27 17:30:56 2009 From: dickinsm at gmail.com (Mark Dickinson) Date: Mon, 27 Apr 2009 22:30:56 +0100 Subject: [SciPy-user] Proposed change to Python's complex number formatting Message-ID: <5c6f2a5d0904271430m3233bfd0r3199d20193af54b2@mail.gmail.com> There's a proposal on python-dev[1] to change the way that the repr and str functions behave for complex numbers in Python 3.1. In brief, the suggestion is that we should get: >>> repr(4+2j) '(4.0+2.0j)' instead of >>> repr(4+2j) '(4+2j)' That is, repr and str for complex numbers should behave more like repr and str for floats: both real and imaginary parts always include *either* an exponent *or* a decimal point and at least one digit following the point. Raymond Hettinger pointed out that Numpy/Scipy users are amongst those most likely to be affected, so I'm soliciting feedback from you all. Any strong feelings about this proposal, one way or the other? Comments either here, or (even better) on python-dev or Python's issue tracker[2] would be very welcome. The first beta of Python 3.1 is due this Saturday, so I'm hoping to resolve this one way or the other by then. Links: the thread is at: [1] http://mail.python.org/pipermail/python-dev/2009-April/089030.html and there's a corresponding issue in the Python bug tracker: [2] http://bugs.python.org/issue5858 Mark Dickinson From david at ar.media.kyoto-u.ac.jp Mon Apr 27 23:20:06 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 28 Apr 2009 12:20:06 +0900 Subject: [SciPy-user] [Numpy-discussion] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> Message-ID: <49F675E6.3030108@ar.media.kyoto-u.ac.jp> Wes McKinney wrote: > Hello, > > I am wondering if anyone can offer some suggestions on this problem. > Over the last year or so I have been building a number of libraries on > top of NumPy + SciPy + matplotlib and other libraries which are being > used for investigative research for my company's problem domain in > place of, say, Matlab and R (which are more "ready out of the box" > systems). I have approximately 20 users, all of whom are running > Windows on a very Microsoft-centric network with databases, etc. Has > anyone had any luck managing a standardized Python environment on lots > of Windows machines with a large number of 3rd-party Python libraries? > Upgrading packages right now involves getting 20 people to click > through an .exe installer, which is hardly a good solution. There is no good solution that I know of for python for this problem. Upgrades on windows and mac os x are usually handled on a per-application basis, but that requires that you control everything. Existing solutions (e.g. based on eggs) will require you to do something in any case. > For example, I was recently forced to upgrade everyone's NumPy to 1.3 > after I discovered that a DLL I had built against 1.3 was incompatible > with 1.2.x. The solution is to build against numpy 1.2.x. You can't expect to link against a library v2 and runs the application with v1 is loaded when v1 < v2. Very few, if any library handle forward compatibility. cheers, David From robince at gmail.com Tue Apr 28 05:46:04 2009 From: robince at gmail.com (Robin) Date: Tue, 28 Apr 2009 10:46:04 +0100 Subject: [SciPy-user] loadmat speed - structs and cells Message-ID: Hi, I recently had some colleagues starting to use NumPy/SciPy and they ran into a problem with performance loading some of their matlab files. It is a small 1MB file, but contains lots of nested structs and cell arrays... it takes a second or so to open in matlab, but python was taking 4 minutes +. I thought it might be related to the problem previously discussed, and after upgrading to SVN there was a big improvement - it now takes 40s on my machine, but this is still an order of magnitude slower than matlab. Since the previous problem was observed with a file that was too large for testing I thought this 1MB file might be more useful to use as a test against regressions and maybe even an example to help optimising. It can be found here: http://www.robince.net/robince/structs_cells.mat Please let me know if there is anything else for them to try to get a speed up (I suggested trying to save in different formats - but I don't really know if v5 vs v6 etc. is likely to make a difference to loading). Cheers Robin From bgunem at gmail.com Tue Apr 28 08:21:19 2009 From: bgunem at gmail.com (Gunten) Date: Tue, 28 Apr 2009 12:21:19 +0000 (UTC) Subject: [SciPy-user] Conversion problem References: <4984F58C.5070605@gmail.com> <49A02E1A.5050703@gmail.com> <49A67E0C.4010002@ru.nl> Message-ID: I have my matlabcodes but i could not convert them into matlabcodes can you help me with this conversion? From argriffi at ncsu.edu Tue Apr 28 08:32:29 2009 From: argriffi at ncsu.edu (alex) Date: Tue, 28 Apr 2009 08:32:29 -0400 Subject: [SciPy-user] Conversion problem In-Reply-To: References: <4984F58C.5070605@gmail.com> <49A02E1A.5050703@gmail.com> <49A67E0C.4010002@ru.nl> Message-ID: <49F6F75D.6020408@ncsu.edu> Gunten wrote: > I have my matlabcodes but i could not convert them into matlabcodes can you > help me with this conversion? Sounds easy. But maybe you mean something like this: http://www.scipy.org/NumPy_for_Matlab_Users Alex From gruben at bigpond.net.au Tue Apr 28 08:36:25 2009 From: gruben at bigpond.net.au (Gary Ruben) Date: Tue, 28 Apr 2009 22:36:25 +1000 Subject: [SciPy-user] Looking for a way to cluster data In-Reply-To: References: <49F3D2A3.3060002@bigpond.net.au> Message-ID: <49F6F849.7060607@bigpond.net.au> Hi Anne, > If I understand you correctly, what you have is not a list of > coordinates of freely-located points, it's a binary mask indicating > which voxels are part of your object. So first of all, have you > considered using volumetric visualization tools? These seem like they > might be a better fit to your problem. I haven't tried this yet, but I will - Zach suggested it too elsewhere in this thread. > If what you want to know about is the connectivity of your object, > though, I can see why you might want to build chains of rods. The most > direct approach is, for each cell that is a 1, to draw rods from it to > each of its neighbors that is on. This may not give you what you want: > if you have regions where all the cells are on, they'll be a dense > grid of rods. It will also not allow you to provide long strings of > rods to your 3D toolkit, or to eliminate short chains. > > As I see it, then, your problem is graph-theoretic: you have this > fairly dense adjacency graph of "on" cells, and you want to pare it > down. One good choice would be to produce a (minimum diameter?) > spanning tree, which should be pretty easy to convert to a collection > of strings of rods. But I think what you want is a graph library of > some sort. I may have (pre-emptively) read your mind because that's basically the approach I took, but I'm not really interested directly in connectivity information, just in ordering the voxel coordinates so I can draw the lines as tubes. I used NetworkX but I've built the Minimum Spanning Tree myself because NetworkX needs the edge information to create its graphs. > On the other hand, if what you have is "fat" chains of on cells, and > you want to build up a "skeleton" of them (like converting the pixels > of an image of the letter o back to a circle), you might look at > machine vision for help, they do this sort of thing often. > > Anne Thanks - that's not my problem - my chains are all lines of single-voxel thickness to start. Thanks for the suggestions. I've got a workable approach now that needs some tweaking but basically works. Volumetric visualisation may do the trick too and will be simple for me to try thanks to the wonderful mayavi. regards, Gary From argriffi at ncsu.edu Tue Apr 28 09:03:39 2009 From: argriffi at ncsu.edu (alex) Date: Tue, 28 Apr 2009 09:03:39 -0400 Subject: [SciPy-user] Conversion problem In-Reply-To: References: <4984F58C.5070605@gmail.com> <49A02E1A.5050703@gmail.com> <49A67E0C.4010002@ru.nl> Message-ID: <49F6FEAB.5090402@ncsu.edu> Gunten wrote: > I have my matlabcodes but i could not convert them into matlabcodes can you > help me with this conversion? Sup dawg, I heard you like idempotence so we converted your matlabcodes into matlabcodes so that they still run in matlab. From scott.sinclair.za at gmail.com Tue Apr 28 10:10:05 2009 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Tue, 28 Apr 2009 16:10:05 +0200 Subject: [SciPy-user] Conversion problem In-Reply-To: <49F6F75D.6020408@ncsu.edu> References: <4984F58C.5070605@gmail.com> <49A02E1A.5050703@gmail.com> <49A67E0C.4010002@ru.nl> <49F6F75D.6020408@ncsu.edu> Message-ID: <6a17e9ee0904280710u3c84e00fu4eab88c769f1fbdc@mail.gmail.com> > 2009/4/28 alex : > Gunten wrote: >> ?I have my matlabcodes but i could not convert them into matlabcodes can you >> help me with this conversion? > > http://www.scipy.org/NumPy_for_Matlab_Users Hi Gunten, Glad to hear that you're keen to start using NumPy and SciPy. I suggest reading through the web page suggested by Alex and starting to convert your code. Once you hit issues that you are unable to solve, you should post detailed questions here. It will be easier for someone to give you a helpful response to a detailed query instead of a general request to convert all of your code. Cheers, Scott From oliphant at enthought.com Tue Apr 28 10:58:29 2009 From: oliphant at enthought.com (Travis Oliphant) Date: Tue, 28 Apr 2009 09:58:29 -0500 Subject: [SciPy-user] Managing Python with NumPy and many external libraries on multiple Windows machines In-Reply-To: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> References: <6c476c8a0904271405q553a9a37h73d38b4be2c46a6b@mail.gmail.com> Message-ID: On Apr 27, 2009, at 4:05 PM, Wes McKinney wrote: > Hello, > > I am wondering if anyone can offer some suggestions on this problem. > Over the last year or so I have been building a number of libraries > on top of NumPy + SciPy + matplotlib and other libraries which are > being used for investigative research for my company's problem > domain in place of, say, Matlab and R (which are more "ready out of > the box" systems). I have approximately 20 users, all of whom are > running Windows on a very Microsoft-centric network with databases, > etc. Has anyone had any luck managing a standardized Python > environment on lots of Windows machines with a large number of 3rd- > party Python libraries? Upgrading packages right now involves > getting 20 people to click through an .exe installer, which is > hardly a good solution. For example, I was recently forced to > upgrade everyone's NumPy to 1.3 after I discovered that a DLL I had > built against 1.3 was incompatible with 1.2.x. There has been a good discussion of this with others. I'm pretty sure we had fixed this in NumPy so at least you would get a sane error instead of a segfault. The solution may not be ideal and it can be hard to remember to edit the right place in the headers when a change is made so it may not have been done for 1.3 > > > It seems like on a Unix / OS X environment with root access this > problem would be fairly easy to script away, but on Windows it's > rather painful. Any advice would be much appreciated. This is a real problem we have encountered at Enthought as well. The Enthought Python Distribution solves this problem using a repository of eggs that the users have access to and a system that can update their machines on a regular schedule (i.e. nightly). I think others have done similar things (copy eggs into a directory and update the search path). This kind of large scale use and maintenance problem is something we are trying to solve with the Enthought Python Distribution. If anybody is interested in it, feel free to email or call me off line. EPD is more than a one-click installer. It's also a repo that can be locally mirrored and added to for your own distribution needs. Best regards, -Travis From oanjao at yahoo.com Tue Apr 28 14:56:22 2009 From: oanjao at yahoo.com (Craig Finch) Date: Tue, 28 Apr 2009 11:56:22 -0700 (PDT) Subject: [SciPy-user] Building Python, Numpy and Scipy with Intel compilers and MKL Message-ID: <850372.12898.qm@web30405.mail.mud.yahoo.com> I figured it out--I am now able to build Python, Numpy, and Scipy with the Intel C, C++, and Fortran compilers. Here is the command line that I used: /home/me/Python_Test/bin/python setup.py config --compiler=intel --fcompiler=intelem config_fc --fcompiler=intelem install --home=/home/me/Python_Test > scipy_build_log.txt What I did differently to make it work: pass the --fcompiler=intelem option to the config_fc module. The option "--fcompiler=intel" did not work for me (perhaps because I am compiling on a 64-bit machine?) Unfortunately, Numpy distutils are not well documented, so I had to read the source to figure this out. Thanks to those who offered advice. Now that I have two separate installs, one built with GCC and one with Intel, I will try to run some benchmarks to compare the two. Craig ----- Original Message ---- From: Craig Finch To: scipy-user at scipy.org Sent: Wednesday, April 22, 2009 6:31:08 PM Subject: Building Python, Numpy and Scipy with Intel compilers and MKL I'm trying to build Python, Numpy, and SciPy on a Red Hat system using the Intel Fortran, C, and C++ compilers, along with the Intel Math Kernel Library. This is proving to be difficult--I've tried everything I could find on the web, and it's still not working (everything works fine if I use GCC). I would really appreciate any suggestions. I can get Python and Numpy working correctly after compiling with Intel compilers, but I can't import anything from SciPy without getting an "undefined symbol" error. According to what I've read, this can happen when gfortran is used to compile some library (such as BLAS) while Intel fortran is used to compile SciPy. I don't think this is my problem, because I'm only using the Intel Math Kernel Libraries. Is this correct? Here's my latest attempt at installation: Python install: CC='icc' CXX='icc' LDFLAGS='-L/opt/intel/cce/10.1.008/lib' CFLAGS='-I/opt/intel/cce/10.1.008/include' CXXFLAGS='-I/opt/intel/cce/10.1.008/include' ./configure -prefix=/home/me Numpy install: python setup.py config --compiler=intel --fcompiler=intel install --home=/home/me Scipy install: python setup.py config --compiler=intel --fcompiler=intel build_ext --fcompiler=intel -L/opt/intel/cce/10.1.008/lib install --home=/home/me Here's something that bothers me: when I start my Python interpreter which was compiled with icc, this is what I see: Python 2.5.4 (r254:67916, Apr 22 2009, 15:52:10) [GCC 4.1.1 20070105 (Red Hat 4.1.1-52)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> Why does the interpreter think it's built with GCC? Can that be causing problems with SciPy? How can I ensure that only Intel libraries are used? Craig From matthieu.brucher at gmail.com Tue Apr 28 14:58:42 2009 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 28 Apr 2009 20:58:42 +0200 Subject: [SciPy-user] Building Python, Numpy and Scipy with Intel compilers and MKL In-Reply-To: <850372.12898.qm@web30405.mail.mud.yahoo.com> References: <850372.12898.qm@web30405.mail.mud.yahoo.com> Message-ID: I'm still surprised you had so many troubles. I compile Python with several extensions without having to specify the C compiler. Matthieu 2009/4/28 Craig Finch : > > I figured it out--I am now able to build Python, Numpy, and Scipy with the Intel C, C++, and Fortran compilers. ?Here is the command line that I used: > > /home/me/Python_Test/bin/python setup.py config --compiler=intel --fcompiler=intelem config_fc --fcompiler=intelem install --home=/home/me/Python_Test > scipy_build_log.txt > > What I did differently to make it work: pass the --fcompiler=intelem option to the config_fc module. The option "--fcompiler=intel" did not work for me (perhaps because I am compiling on a 64-bit machine?) ?Unfortunately, Numpy distutils are not well documented, so I had to read the source to figure this out. > > Thanks to those who offered advice. ?Now that I have two separate installs, one built with GCC and one with Intel, I will try to run some benchmarks to compare the two. > > ? Craig > > > > ----- Original Message ---- > From: Craig Finch > To: scipy-user at scipy.org > Sent: Wednesday, April 22, 2009 6:31:08 PM > Subject: Building Python, Numpy and Scipy with Intel compilers and MKL > > I'm trying to build Python, Numpy, and SciPy on a Red Hat system using the Intel Fortran, C, and C++ compilers, along with the Intel Math Kernel Library. ?This is proving to be difficult--I've tried everything I could find on the web, and it's still not working (everything works fine if I use GCC). ?I would really appreciate any suggestions. > > I can get Python and Numpy working correctly after compiling with Intel compilers, but I can't import anything from SciPy without getting an "undefined symbol" error. ?According to what I've read, this can happen when gfortran is used to compile some library (such as BLAS) while Intel fortran is used to compile SciPy. ?I don't think this is my problem, because I'm only using the Intel Math Kernel Libraries. ?Is this correct? > > Here's my latest attempt at installation: > Python install: > CC='icc' > CXX='icc' LDFLAGS='-L/opt/intel/cce/10.1.008/lib' CFLAGS='-I/opt/intel/cce/10.1.008/include' CXXFLAGS='-I/opt/intel/cce/10.1.008/include' ./configure -prefix=/home/me > > Numpy install: > python setup.py config --compiler=intel --fcompiler=intel install --home=/home/me > > Scipy install: > python > setup.py config --compiler=intel --fcompiler=intel build_ext > --fcompiler=intel -L/opt/intel/cce/10.1.008/lib ?install --home=/home/me > > Here's something that bothers me: when I start my Python interpreter which was compiled with icc, this is what I see: > Python 2.5.4 (r254:67916, Apr 22 2009, 15:52:10) > [GCC 4.1.1 20070105 (Red Hat 4.1.1-52)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> > > Why > does the interpreter think it's built with GCC? ?Can that be causing > problems with SciPy? ?How can I ensure that only Intel libraries are > used? > > ? ?Craig > > > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From oanjao at yahoo.com Tue Apr 28 22:29:23 2009 From: oanjao at yahoo.com (Craig Finch) Date: Tue, 28 Apr 2009 19:29:23 -0700 (PDT) Subject: [SciPy-user] Building Python, Numpy and Scipy with Intel compilers and MKL Message-ID: <247988.31946.qm@web30406.mail.mud.yahoo.com> I spoke too soon! Upon further testing of Numpy, I find that I cannot import the "sparsetools" package, which is built using Swig (details of the error below). The system version of Swig is 1.3.29, built with g++. Is it possible that Swig is causing a problem here? Any other suggestions? >>> from scipy.interpolate import interp1d Traceback (most recent call last): File "", line 1, in File "/home/me/Python_Test/lib/python/scipy/interpolate/__init__.py", line 13, in from rbf import Rbf File "/home/me/Python_Test/lib/python/scipy/interpolate/rbf.py", line 47, in from scipy import linalg File "/home/me/Python_Test/lib/python/scipy/linalg/__init__.py", line 13, in from iterative import * File "/home/me/Python_Test/lib/python/scipy/linalg/iterative.py", line 5, in from scipy.sparse.linalg import isolve File "/home/me/Python_Test/lib/python/scipy/sparse/__init__.py", line 6, in from csr import * File "/home/me/Python_Test/lib/python/scipy/sparse/csr.py", line 12, in from sparsetools import csr_tocsc, csr_tobsr, csr_count_blocks, \ File "/home/me/Python_Test/lib/python/scipy/sparse/sparsetools/__init__.py", line 4, in from csr import * File "/home/me/Python_Test/lib/python/scipy/sparse/sparsetools/csr.py", line 7, in import _csr ImportError: /home/me/Python_Test/lib/python/scipy/sparse/sparsetools/_csr.so: undefined symbol: _ZNSt8ios_base4InitD1Ev I looked at the symbols defined in the library with "nm" and sure enough, that one is undefined: ... 00000000000d2fdc W _ZNSt6vectorIySaIyEE6resizeEmy U _ZNSt8ios_base4InitC1Ev U _ZNSt8ios_base4InitD1Ev ... For what it's worth, here is the build output for the building of _csr.so: ------------------------------------------------------------------------------------------ building 'scipy.sparse.sparsetools._csr' extension compiling C++ sources C compiler: icc -O2 -fPIC creating build/temp.linux-x86_64-2.5/scipy/sparse/sparsetools compile options: '-I/home/cfinch/lib/python/numpy/core/include -I/home/cfinch/Python_Test/include/python2.5 -c' icc: scipy/sparse/sparsetools/csr_wrap.cxx scipy/sparse/sparsetools/csr_wrap.cxx(11526): (col. 28) remark: LOOP WAS VECTORIZED. scipy/sparse/sparsetools/csr_wrap.cxx(11572): (col. 28) remark: LOOP WAS VECTORIZED. scipy/sparse/sparsetools/csr_wrap.cxx(10056): (col. 3) remark: LOOP WAS VECTORIZED. scipy/sparse/sparsetools/csr_wrap.cxx(10182): (col. 3) remark: LOOP WAS VECTORIZED. scipy/sparse/sparsetools/csr_wrap.cxx(10308): (col. 3) remark: LOOP WAS VECTORIZED. scipy/sparse/sparsetools/csr_wrap.cxx(10434): (col. 3) remark: LOOP WAS VECTORIZED. scipy/sparse/sparsetools/csr_wrap.cxx(10560): (col. 3) remark: LOOP WAS VECTORIZED. scipy/sparse/sparsetools/csr_wrap.cxx(10686): (col. 3) remark: LOOP WAS VECTORIZED. scipy/sparse/sparsetools/csr_wrap.cxx(10812): (col. 3) remark: LOOP WAS VECTORIZED. scipy/sparse/sparsetools/csr_wrap.cxx(10938): (col. 3) remark: LOOP WAS VECTORIZED. scipy/sparse/sparsetools/csr_wrap.cxx(11064): (col. 3) remark: LOOP WAS VECTORIZED. scipy/sparse/sparsetools/csr_wrap.cxx(11190): (col. 3) remark: LOOP WAS VECTORIZED. scipy/sparse/sparsetools/csr_wrap.cxx(11316): (col. 3) remark: LOOP WAS VECTORIZED. scipy/sparse/sparsetools/csr_wrap.cxx(11442): (col. 3) remark: LOOP WAS VECTORIZED. icc -O2 -fPIC -shared build/temp.linux-x86_64-2.5/scipy/sparse/sparsetools/csr_wrap.o -Lbuild/temp.linux-x86_64-2.5 -o build/lib.linux-x86_64-2.5/scipy/sparse/sparsetools/_csr.so ... copying build/lib.linux-x86_64-2.5/scipy/sparse/sparsetools/_csr.so -> /home/cfinch/Python_Test/lib/python/scipy/sparse/sparsetools Any ideas? Craig ----- Original Message ---- From: Craig Finch To: scipy-user at scipy.org Sent: Tuesday, April 28, 2009 2:56:22 PM Subject: Re: Building Python, Numpy and Scipy with Intel compilers and MKL I figured it out--I am now able to build Python, Numpy, and Scipy with the Intel C, C++, and Fortran compilers. Here is the command line that I used: /home/me/Python_Test/bin/python setup.py config --compiler=intel --fcompiler=intelem config_fc --fcompiler=intelem install --home=/home/me/Python_Test > scipy_build_log.txt What I did differently to make it work: pass the --fcompiler=intelem option to the config_fc module. The option "--fcompiler=intel" did not work for me (perhaps because I am compiling on a 64-bit machine?) Unfortunately, Numpy distutils are not well documented, so I had to read the source to figure this out. Thanks to those who offered advice. Now that I have two separate installs, one built with GCC and one with Intel, I will try to run some benchmarks to compare the two. Craig ----- Original Message ---- From: Craig Finch To: scipy-user at scipy.org Sent: Wednesday, April 22, 2009 6:31:08 PM Subject: Building Python, Numpy and Scipy with Intel compilers and MKL I'm trying to build Python, Numpy, and SciPy on a Red Hat system using the Intel Fortran, C, and C++ compilers, along with the Intel Math Kernel Library. This is proving to be difficult--I've tried everything I could find on the web, and it's still not working (everything works fine if I use GCC). I would really appreciate any suggestions. I can get Python and Numpy working correctly after compiling with Intel compilers, but I can't import anything from SciPy without getting an "undefined symbol" error. According to what I've read, this can happen when gfortran is used to compile some library (such as BLAS) while Intel fortran is used to compile SciPy. I don't think this is my problem, because I'm only using the Intel Math Kernel Libraries. Is this correct? Here's my latest attempt at installation: Python install: CC='icc' CXX='icc' LDFLAGS='-L/opt/intel/cce/10.1.008/lib' CFLAGS='-I/opt/intel/cce/10.1.008/include' CXXFLAGS='-I/opt/intel/cce/10.1.008/include' ./configure -prefix=/home/me Numpy install: python setup.py config --compiler=intel --fcompiler=intel install --home=/home/me Scipy install: python setup.py config --compiler=intel --fcompiler=intel build_ext --fcompiler=intel -L/opt/intel/cce/10.1.008/lib install --home=/home/me Here's something that bothers me: when I start my Python interpreter which was compiled with icc, this is what I see: Python 2.5.4 (r254:67916, Apr 22 2009, 15:52:10) [GCC 4.1.1 20070105 (Red Hat 4.1.1-52)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> Why does the interpreter think it's built with GCC? Can that be causing problems with SciPy? How can I ensure that only Intel libraries are used? Craig From david at ar.media.kyoto-u.ac.jp Wed Apr 29 00:43:06 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 29 Apr 2009 13:43:06 +0900 Subject: [SciPy-user] Building Python, Numpy and Scipy with Intel compilers and MKL In-Reply-To: <247988.31946.qm@web30406.mail.mud.yahoo.com> References: <247988.31946.qm@web30406.mail.mud.yahoo.com> Message-ID: <49F7DADA.20401@ar.media.kyoto-u.ac.jp> Craig Finch wrote: > I spoke too soon! Upon further testing of Numpy, I find that I cannot import the "sparsetools" package, which is built using Swig (details of the error below). The system version of Swig is 1.3.29, built with g++. Is it possible that Swig is causing a problem here? Any other suggestions? > I think C++ ABI incompatibility is the problem, not swig. Icc has some options to set up the g++ ABI, and some icc are incompatible with some g++ versions as well (the error seems to be related to the io C++ library). Maybe you could try compiling the c++ part with g++, cheers, David From dave.hirschfeld at gmail.com Wed Apr 29 10:10:28 2009 From: dave.hirschfeld at gmail.com (Dave Hirschfeld) Date: Wed, 29 Apr 2009 14:10:28 +0000 (UTC) Subject: [SciPy-user] timeseries.convert and structured arrays Message-ID: Hi, I ran into a problem using the convert function with a structured array (see below) so I just wanted to check if the timeseries function convert is intended to support structured arrays? If not are there any plans to do so? And if it does, what am I doing wrong below?? Thanks, Dave dates = ts.date_array(start_date=ts.Date('D','2008-01-01'), end_date=ts.Date('D','2008-12-31')) # data = np.array(zip(np.random.rand(366),np.random.rand(366)), dtype=[('field1',np.float),('field2',np.object)]) # data = ts.time_series(data,dates) data.convert('M') C:\dev\bin\Python25\lib\site-packages\scikits\timeseries\tseries.pyc in convert(series, freq, func, position, *args, **kwargs) 1999 2000 if series.ndim == 1: -> 2001 obj = _convert1d(series, freq, func, position, *args, **kwargs) 2002 elif series.ndim == 2: 2003 base = _convert1d(series[:,0], freq, func, position, *args, **kwargs) C:\dev\bin\Python25\lib\site-packages\scikits\timeseries\tseries.pyc in _convert1d(series, freq, func, position, *args, **kwargs) 1909 1910 cdictresult = cseries.TS_convert(data_, from_freq, to_freq, position, -> 1911 int(start_date), mask_) 1912 start_date = Date(freq=to_freq, value=cdictresult['startindex']) 1913 data_ = masked_array(cdictresult['values'], mask=cdictresult['mask']) ValueError: data type must provide an itemsize From mudit_19a at yahoo.com Wed Apr 29 10:27:39 2009 From: mudit_19a at yahoo.com (mudit sharma) Date: Wed, 29 Apr 2009 19:57:39 +0530 (IST) Subject: [SciPy-user] Fw: realtime axes flickers Message-ID: <263726.36163.qm@web94914.mail.in2.yahoo.com> I tried reloading the figure and whole widget before plotting but the axes still flickers when I plot after reloading. Here's my code: def __init__(self): QtGui.QMainWindow.__init__(self) self.plot_widget = QtGui.QWidget(self) self.plot_box = QtGui.QVBoxLayout(self.plot_widget) self.cfig = LiveFigure(figsize=(8, 4), dpi=100) self.canvas = LiveCanvas(self.cfig, self.plot_widget) self.plot_box.addWidget(self.canvas) self.plot_widget.setFocus() self.setCentralWidget(self.plot_widget) def reload_figure(self): self.plot_box.removeWidget(self.canvas) self.plot_widget.destroy() self.plot_widget = QtGui.QWidget(self) self.plot_box = QtGui.QVBoxLayout(self.plot_widget) self.cfig = LiveFigure(figsize=(8, 4), dpi=100) self..canvas = LiveCanvas(self.cfig, self.plot_widget) self.plot_box..addWidget(self.canvas) self.plot_widget.setFocus() self.setCentralWidget(self.plot_widget) ----- Forwarded Message ---- From: mudit sharma To: scipy-user at scipy.org Sent: Friday, 24 April, 2009 18:41:30 Subject: realtime axes issues I am trying to debug this issue may be someone can help. I have drawn multiple axes in one figure under "ipython -pylab " which contains line that are updated in realtime. Then I remove these axses using fig.delaxes from figures and draw another sets of axeses but the previous sets of axeses still exists and there line data keep getting updated in realtime. It causes current sets of axeses and x and y axis in figure to flicker. Any idea what might be causing this? Thanks! From pgmdevlist at gmail.com Wed Apr 29 13:21:29 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 29 Apr 2009 13:21:29 -0400 Subject: [SciPy-user] timeseries.convert and structured arrays In-Reply-To: References: Message-ID: On Apr 29, 2009, at 10:10 AM, Dave Hirschfeld wrote: > Hi, > I ran into a problem using the convert function with a structured > array (see > below) so I just wanted to check if the timeseries function convert > is intended > to support structured arrays? Doesn't work yet, sorry... > If not are there any plans to do so? When I'll have some time working on it, yes. Sometime this summer if I'm lucky... Meanwhile, you can emulate it by converting each field independently and repatching the result >>> data = ts.time_series(data,dates) >>> mfield1 = data['field1'].convert("M") >>> mdata = ts.time_series(np.empty(mfield1.shape, dtype=data.dtype), ... dates=mfield1.dates) >>> mdata['field1'] = mfield1 >>> mdata['field2'] = data['field2'].convert("M") > And if it > does, what am I doing wrong below?? > > Thanks, > Dave > > dates = ts.date_array(start_date=ts.Date('D','2008-01-01'), > end_date=ts.Date('D','2008-12-31')) > # > > data = np.array(zip(np.random.rand(366),np.random.rand(366)), > dtype=[('field1',np.float),('field2',np.object)]) > # > > data = ts.time_series(data,dates) > data.convert('M') > > > > C:\dev\bin\Python25\lib\site-packages\scikits\timeseries\tseries.pyc > in > convert(series, freq, func, position, *args, **kwargs) > 1999 > 2000 if series.ndim == 1: > -> 2001 obj = _convert1d(series, freq, func, position, > *args, **kwargs) > 2002 elif series.ndim == 2: > 2003 base = _convert1d(series[:,0], freq, func, position, > *args, > **kwargs) > > C:\dev\bin\Python25\lib\site-packages\scikits\timeseries\tseries.pyc > in > _convert1d(series, freq, func, position, *args, **kwargs) > 1909 > 1910 cdictresult = cseries.TS_convert(data_, from_freq, > to_freq, position, > -> 1911 int(start_date), mask_) > 1912 start_date = Date(freq=to_freq, > value=cdictresult['startindex']) > 1913 data_ = masked_array(cdictresult['values'], > mask=cdictresult['mask']) > > ValueError: data type must provide an itemsize > > > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From R.Springuel at umit.maine.edu Wed Apr 29 17:14:16 2009 From: R.Springuel at umit.maine.edu (R. Padraic Springuel) Date: Wed, 29 Apr 2009 17:14:16 -0400 Subject: [SciPy-user] SciPy for Python 2.6 Message-ID: <49F8C328.3070109@umit.maine.edu> I found a Windows installer file for Numpy for Python 2.6 but couldn't find one for SciPy. Does one exist yet? -- R. Padraic Springuel Research Assistant Department of Physics and Astronomy University of Maine Bennett 309 Office Hours: By appointment only -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 5632 bytes Desc: S/MIME Cryptographic Signature URL: From david at ar.media.kyoto-u.ac.jp Wed Apr 29 21:51:23 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 30 Apr 2009 10:51:23 +0900 Subject: [SciPy-user] SciPy for Python 2.6 In-Reply-To: <49F8C328.3070109@umit.maine.edu> References: <49F8C328.3070109@umit.maine.edu> Message-ID: <49F9041B.7040700@ar.media.kyoto-u.ac.jp> Hi Padraic, R. Padraic Springuel wrote: > I found a Windows installer file for Numpy for Python 2.6 but couldn't > find one for SciPy. Does one exist yet? Not at the moment, no. David From gael.varoquaux at normalesup.org Thu Apr 30 09:56:19 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 30 Apr 2009 15:56:19 +0200 Subject: [SciPy-user] Sparse factorisation Message-ID: <20090430135619.GB9195@phare.normalesup.org> Hi there, This is a rather general question about sparse factorisation and Scipy. I no nothing about sparse linear algebra, but I happen to need to do a sparse Cholesky. By sparse Cholesky, I don't mean a sparse implementation of Cholesky, my matrices are small, I don't actually care about sparse linear algebra, but I mean finding P, permutation matrix, and L, lower-only matrix, such that the sparse positive definite matrix A can be written: A = P L L' P', where P is a reordering of the rows and column of A used to maximise the sparsity of L. What is really of interest to me is P. P can be computed using the colamd routine, present in UMFPACK. Scipy sparse does make use of the colamd routine (by setting the permc_spec argument of scipy.sparse.splu to 3). However, I cannot figure out a way of extracting P. I have been looking around in the sparse code source, as well as the scikits.umfpack code source, and I must admit I am a bit at loss to what is the best way to achieve my goals. The umfpack.py module, both in the scipy code base and in the scikit, is broken. Scipy seems to hint that it is depreciated, but nothing replaces it to provide the functionality I want. In addition, the tests of the scikit don't run with the svn versions of numpy and scipy. So I am wondering, in what direction is all this going. Where should I invest my efforts to get things working, and get my 'P' in a reasonnably short amount of time? Any help much appreciated, as I am no pro of these problems. By the way, progress on this issue could close ticket 261: http://projects.scipy.org/scipy/ticket/261 Cheers, Ga?l From gael.varoquaux at normalesup.org Thu Apr 30 11:54:13 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 30 Apr 2009 17:54:13 +0200 Subject: [SciPy-user] Sparse factorisation In-Reply-To: <20090430135619.GB9195@phare.normalesup.org> References: <20090430135619.GB9195@phare.normalesup.org> Message-ID: <20090430155413.GD9195@phare.normalesup.org> On Thu, Apr 30, 2009 at 03:56:19PM +0200, Gael Varoquaux wrote: > I have been looking around in the sparse code source, as well as the > scikits.umfpack code source, and I must admit I am a bit at loss to what > is the best way to achieve my goals. I have dug a bit further. It seems that all that needs to be done is expose the L, U, perm_c and perm_r attributes of the SciPyLUObject (AKA factored_lu object) returned by splu. Of course this easier said than done, as exposing these objects requires creating scipy sparse matrices from the inner SuperLU representation of these objects. I'd really appreciate if someone (most probably Nathan) could give me a hand with this. I realise that I am asking people to do my work, and I know exactly what my reaction is when someone comes around to me with this request (ie, not happy), but I am not sure I have to time required to learn the library and the tools to do this myself, and would hate to have to find an ugly workaround (this does sound like a bad excuse). :) Ga?l From wnbell at gmail.com Thu Apr 30 23:47:38 2009 From: wnbell at gmail.com (Nathan Bell) Date: Thu, 30 Apr 2009 23:47:38 -0400 Subject: [SciPy-user] Sparse factorisation In-Reply-To: <20090430155413.GD9195@phare.normalesup.org> References: <20090430135619.GB9195@phare.normalesup.org> <20090430155413.GD9195@phare.normalesup.org> Message-ID: On Thu, Apr 30, 2009 at 11:54 AM, Gael Varoquaux wrote: > > I have dug a bit further. It seems that all that needs to be done is > expose the L, U, perm_c and perm_r attributes of the SciPyLUObject (AKA > factored_lu object) returned by splu. Of course this easier said than > done, as exposing these objects requires creating scipy sparse matrices > from the inner SuperLU representation of these objects. > > I'd really appreciate if someone (most probably Nathan) could give me a > hand with this. I realise that I am asking people to do my work, and I > know exactly what my reaction is when someone comes around to me with > this request (ie, not happy), but I am not sure I have to time required > to learn the library and the tools to do this myself, and would hate to > have to find an ugly workaround (this does sound like a bad excuse). > Hi Gael, I hate to disappoint, but I don't have enough free time right now. If you or someone else wants to give it a try I can probably provide some support. I don't know what's "broken" in the umfpack wrappers, but I suspect they may be the path of least resistance. -- Nathan Bell wnbell at gmail.com http://graphics.cs.uiuc.edu/~wnbell/ From ondrej at certik.cz Thu Apr 30 23:52:28 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 30 Apr 2009 20:52:28 -0700 Subject: [SciPy-user] Sparse factorisation In-Reply-To: <20090430135619.GB9195@phare.normalesup.org> References: <20090430135619.GB9195@phare.normalesup.org> Message-ID: <85b5c3130904302052k7fe30d7ekeb3de8fbf9cd8f72@mail.gmail.com> On Thu, Apr 30, 2009 at 6:56 AM, Gael Varoquaux wrote: > Hi there, > > This is a rather general question about sparse factorisation and Scipy. I > no nothing about sparse linear algebra, but I happen to need to do a > sparse Cholesky. > > By sparse Cholesky, I don't mean a sparse implementation of Cholesky, my > matrices are small, I don't actually care about sparse linear algebra, > but I mean finding P, permutation matrix, and L, lower-only matrix, such > that the sparse positive definite matrix A can be written: > ? ? ? ?A = P L L' P', > where P is a reordering of the rows and column of A used to maximise the > sparsity of L. > > What is really of interest to me is P. P can be computed using the colamd > routine, present in UMFPACK. Scipy sparse does make use of the colamd > routine (by setting the permc_spec argument of scipy.sparse.splu to 3). > However, I cannot figure out a way of extracting P. > > I have been looking around in the sparse code source, as well as the > scikits.umfpack code source, and I must admit I am a bit at loss to what > is the best way to achieve my goals. The umfpack.py module, both in the > scipy code base and in the scikit, is broken. Scipy seems to hint that it Is umfpack in scipy really broken? That is very disappointing. I was planning to use umfpack through scipy. I hope it will not be difficult for me to fix it. Ondrej