From jake.biesinger at gmail.com Sun Aug 1 01:16:20 2010 From: jake.biesinger at gmail.com (Jacob Biesinger) Date: Sat, 31 Jul 2010 22:16:20 -0700 Subject: [SciPy-User] Off by one bug in Scipy.stats.hypergeom In-Reply-To: References: Message-ID: I have seen the error of my ways and apologize for the noise in the list-- I should have been using the probability mass function instead of the survival function. Doing the summation manually shows that all is well: def hypergeom(x, fgTotal, bgMatching, bgTotal): return scipy.comb(fgTotal,x) * scipy.comb(bgTotal-fgTotal,bgMatching-x) / scipy.comb(bgTotal,bgMatching) >>> scipy.stats.hypergeom.sf(4,50,5,10, loc=1) 0.0040835205497095073 >>> sum(scipy.stats.hypergeom.pmf(x,50,5,10) for x in range(4,min(10+1,50+1))) 0.0040835205497557125 >>> sum([hypergeom(x, 10, 5, 50) for x in scipy.arange(4, min(10+1, 50+1))]) 0.0040835205497557186 >>> scipy.stats.hypergeom.sf(5,50,5,10, loc=1) 0.00011893749169422652 >>> sum(scipy.stats.hypergeom.pmf(x,50,5,10) for x in range(5,min(10+1,50+1))) 0.00011893749174045688 >>> sum([hypergeom(x, 10, 5, 50) for x in scipy.arange(5, min(10+1, 50+1))]) 0.00011893749174045679 Thanks anyway! -- Jake Biesinger Graduate Student Xie Lab, UC Irvine (949) 231-7587 On Sat, Jul 31, 2010 at 5:48 PM, Jacob Biesinger wrote: > Hi! > > Perhaps I'm using the module incorrectly, but it looks like the x parameter > in scipy.stats.hypergeom is off by one. Specifically, I think it's > one-too-high. > > From the wikipedia article > http://en.wikipedia.org/wiki/Hypergeometric_distribution#Application_and_example (I > know they could be wrong-- just hear me out on this), > > scipy.stats.hypergeom? > Hypergeometric distribution > > Models drawing objects from a bin. > M is total number of objects, n is total number of Type I objects. > RV counts number of Type I objects in N drawn without replacement > from > population. > So translating wikipedia's example... > Pr(x=4; M=50, n=5, N=10) = (choose(5,4) * choose(50-5, 10-4)) / > choose(50,10) = .003964583 > Pr(x=5; M=50, n=5, N=10) = (choose(5,5) * choose(50-5, 10-5)) / > choose(50,10) = .0001189375 > > Which you can check with the python code: > from scipy import comb as chse # "combination" => choose > > float((chse(5,4, exact=1) * chse(50-5,10-4, exact=1))) / > chse(50,10,exact=1) # example one > 0.0039645830580150654155 # okay! > > float((chse(5,5, exact=1) * chse(50-5,10-5, exact=1))) / > chse(50,10,exact=1) # example two > 0.00011893749174045196247 # okay! > > Try example one with scipy.stats.hypergeom: > # scipy.stats.hypergeom.sf(x, M, n, N) > scipy.stats.hypergeom.sf(4,50,5,10) > 0.00011893749169422652 # correct value for x=5, not x=4 > scipy.stats.hypergeom.sf(5,50,5,10) > -4.6185277824406512e-14 # wrong > > It seems that changing the loc value from =0 (default) to =1 fixes the > issue... > scipy.stats.hypergeom.sf(4,50,5,10, loc=1) > 0.0040835205497095073 # close enough > > scipy.stats.hypergeom.sf(5,50,5,10, loc=1) > 0.00011893749169422652 # okay! > > Am I using the package wrong? > -- > Jake Biesinger > Graduate Student > Xie Lab, UC Irvine > (949) 231-7587 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sun Aug 1 05:33:27 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 1 Aug 2010 05:33:27 -0400 Subject: [SciPy-User] Off by one bug in Scipy.stats.hypergeom In-Reply-To: References: Message-ID: On Sun, Aug 1, 2010 at 1:16 AM, Jacob Biesinger wrote: > I have seen the error of my ways and apologize for the noise in the list-- I > should have been using the probability mass function instead of the survival > function. ?Doing the summation manually shows that all is well: > def hypergeom(x, fgTotal, bgMatching, bgTotal): > ?? ? ? ?return scipy.comb(fgTotal,x) * > scipy.comb(bgTotal-fgTotal,bgMatching-x) / scipy.comb(bgTotal,bgMatching) >>>> scipy.stats.hypergeom.sf(4,50,5,10, loc=1) > 0.0040835205497095073 >>>>?sum(scipy.stats.hypergeom.pmf(x,50,5,10) for x in >>>> range(4,min(10+1,50+1))) > 0.0040835205497557125 >>>>?sum([hypergeom(x, 10, 5, 50) for x in scipy.arange(4, min(10+1, 50+1))]) > 0.0040835205497557186 > >>>>?scipy.stats.hypergeom.sf(5,50,5,10, loc=1) > 0.00011893749169422652 >>>>?sum(scipy.stats.hypergeom.pmf(x,50,5,10) for x in >>>> range(5,min(10+1,50+1))) > 0.00011893749174045688 >>>>?sum([hypergeom(x, 10, 5, 50) for x in scipy.arange(5, min(10+1, 50+1))]) > 0.00011893749174045679 Note that the distribution methods are vectorized. You can use an array_like for x instead of the list comprehension. Josef > > Thanks anyway! > -- > Jake Biesinger > Graduate Student > Xie Lab, UC Irvine > (949) 231-7587 > > > On Sat, Jul 31, 2010 at 5:48 PM, Jacob Biesinger > wrote: >> >> Hi! >> Perhaps I'm using the module incorrectly, but it looks like the x >> parameter in scipy.stats.hypergeom is off by one. ?Specifically, I think >> it's one-too-high. >> From the wikipedia article >> http://en.wikipedia.org/wiki/Hypergeometric_distribution#Application_and_example?(I >> know they could be wrong-- just hear me out on this), >> scipy.stats.hypergeom? >> Hypergeometric distribution >> >> ?? ? ? Models drawing objects from a bin. >> ?? ? ? M is total number of objects, n is total number of Type I objects. >> ?? ? ? RV counts number of Type I objects in N drawn without replacement >> from >> ?? ? ? population. >> So translating wikipedia's example... >> Pr(x=4; M=50, n=5, N=10) ?= (choose(5,4) * choose(50-5, 10-4)) / >> choose(50,10) = .003964583 >> Pr(x=5; M=50, n=5, N=10) ?= (choose(5,5) * choose(50-5, 10-5)) / >> choose(50,10) = .0001189375 >> Which you can check with the python code: >> from scipy import comb as chse ? # "combination" => choose >> >> float((chse(5,4, exact=1) * chse(50-5,10-4, exact=1))) / >> chse(50,10,exact=1) ?# example one >> 0.0039645830580150654155 ?# okay! >> float((chse(5,5, exact=1) * chse(50-5,10-5, exact=1))) / >> chse(50,10,exact=1) # example two >> 0.00011893749174045196247 ?# okay! >> Try example one with scipy.stats.hypergeom: >> # scipy.stats.hypergeom.sf(x, M, n, N) >> scipy.stats.hypergeom.sf(4,50,5,10) >> 0.00011893749169422652 ? ? # correct value for x=5, not x=4 >> scipy.stats.hypergeom.sf(5,50,5,10) >> -4.6185277824406512e-14 ? ?# wrong >> It seems that changing the loc value from =0 (default) to =1 fixes the >> issue... >> scipy.stats.hypergeom.sf(4,50,5,10, loc=1) >> 0.0040835205497095073 ? ?# close enough >> scipy.stats.hypergeom.sf(5,50,5,10, loc=1) >> 0.00011893749169422652 ? # okay! >> Am I using the package wrong? >> -- >> Jake Biesinger >> Graduate Student >> Xie Lab, UC Irvine >> (949) 231-7587 > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From faltet at pytables.org Sun Aug 1 07:33:52 2010 From: faltet at pytables.org (Francesc Alted) Date: Sun, 1 Aug 2010 13:33:52 +0200 Subject: [SciPy-User] ANN: Numexpr 1.4 released Message-ID: <201008011333.52074.faltet@pytables.org> Hi, After the Theano talk in last EuroSciPy I suddenly realized that it would not be too difficult to implement a multi-threaded version of Numexpr. Well, as usual I was terribly wrong and it took me a *long* week to do the job :-/ Anyway the thing is done now, so... enjoy! Note for PyTables users: Numexpr does not include changes in the API/ABI, so the upgrade from 1.3.1 is recommended. ======================== Announcing Numexpr 1.4 ======================== Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. What's new ========== The main improvement in this version is the support for multi-threading in pure C. Threading in C provides the best performance in nowadays multi-core machines. In addition, this avoids the GIL that hampers performance in many Python apps. Just to wet your appetite, look into this page where the implementation is briefly described and where some benchmarks are shown: http://code.google.com/p/numexpr/wiki/MultiThreadVM In case you want to know more in detail what has changed in this version, see: http://code.google.com/p/numexpr/wiki/ReleaseNotes or have a look at RELEASE_NOTES.txt in the tarball. Where I can find Numexpr? ========================= The project is hosted at Google code in: http://code.google.com/p/numexpr/ And you can get the packages from PyPI as well: http://pypi.python.org/pypi Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy! -- Francesc Alted From sebastian.walter at gmail.com Sun Aug 1 08:05:58 2010 From: sebastian.walter at gmail.com (Sebastian Walter) Date: Sun, 1 Aug 2010 14:05:58 +0200 Subject: [SciPy-User] [ANN] ALGOPY 0.21, algorithmic differentiation in Python Message-ID: I'm happy to announce the first official release of ALGOPY in version 0.2.1. Rationale: ~~~~~~~~ The purpose of ALGOPY is the evaluation of higher-order derivatives in the forward and reverse mode of Algorithmic Differentiation (AD) using univariate Taylor polynomial arithmetic. Particular focus are functions that contain numerical linear algebra functions (e.g. inv, dot, eigh, qr, cholesky,...) as they often appear in statistically motivated functions. Download: ~~~~~~~~~ http://pypi.python.org/pypi/algopy/0.2.1 or bleeding edge versions from from http://github.com/b45ch1/algopy Documentation: ~~~~~~~~~~~~ available at http://packages.python.org/algopy/ OS Support: ~~~~~~~~~~ Linux, Windows (tested with pythonxy), should also work on Mac Software Dependencies: ~~~~~~~~~~~~~~~~~~~~ for the core: numpy, scipy for testing: nose Exampe Session: ~~~~~~~~~~~~~ Consider the contrived example where it is the goal to compute the directional derivative df/dx_1 : >>> import numpy; from numpy import log, exp, sin, cos >>> import algopy; from algopy import UTPM, dot, inv, zeros >>> >>> def f(x): ... A = zeros((2,2),dtype=x) ... A[0,0] = numpy.log(x[0]*x[1]) ... A[0,1] = numpy.log(x[1]) + exp(x[0]) ... A[1,0] = sin(x[1])**2 + cos(x[0])**3.1 ... A[1,1] = x[0]**cos(x[1]) ... return log( dot(x.T, dot( inv(A), x))) ... >>> >>> x = UTPM(zeros((2,1,2),dtype=float)) >>> x.data[0,0] = [1,2] >>> x.data[1,0] = [1,0] >>> y = f(x) >>> >>> print 'normal function evaluation f(x) = ',y.data[0,0] normal function evaluation f(x) = 0.641250189986 >>> print 'directional derivative df/dx1 = ',y.data[1,0] directional derivative df/dx1 = 1.62982340133 From sjsrey at gmail.com Sun Aug 1 11:36:48 2010 From: sjsrey at gmail.com (Serge Rey) Date: Sun, 1 Aug 2010 08:36:48 -0700 Subject: [SciPy-User] ANN: PySAL 1.0.0 Message-ID: We are pleased to announce the release of PySAL 1.0.0. PySAL is a library of tools for spatial data analysis written in Python. This release comes after an extensive integration and refactoring effort to combine common analytical functionality in the projects STARS and PySpace into a shared open source library for geospatial analysis and geocomputation. For more information, please see the release notes that follow below. You can download PySAL from here: http://code.google.com/p/pysal/downloads/list Python 2.6 binaries for Windows and OS X are available, as well as source tarballs for other platforms and documentation in html form. Enjoy, The PySAL developers ================================================= Python Spatial Analysis Library ================================================= .. Contents:: What is PySAL -------------- PySAL modules +++++++++++++ * pysal.core ? Core Data Structures and IO * pysal.cg ? Computational Geometry * pysal.esda ? Exploratory Spatial Data Analysis * pysal.inequality ? Spatial Inequality Analysis * pysal.spatial_dynamics ? Spatial Dynamics * pysal.region ? Spatially constrained clustering * pysal.weights ? Spatial Weights * pysal.FileIO ? PySAL FileIO: Module for reading and writing various file types in a Pythonic way Documentation ------------- The documentation site is here http://pysal.org/contents.html Web sites --------- PySAL's home is here http://pysal.org/ The developer's site is here http://code.google.com/p/pysal/ Mailing Lists ------------- Please see the developer's list here http://groups.google.com/group/pysal-dev Help for users is here http://groups.google.com/group/openspace-list Bug reports ----------- To search for or report bugs, please see http://code.google.com/p/pysal/issues/list License information ------------------- See the file "LICENSE.txt" for information on the history of this software, terms & conditions for usage, and a DISCLAIMER OF ALL WARRANTIES. -- Sergio (Serge) Rey Professor, School of Geographical Sciences and Urban Planning Arizona State University http://geoplan.asu.edu/rey Editor, International Regional Science Review http://irx.sagepub.com From ernesto.adorio at gmail.com Mon Aug 2 07:13:13 2010 From: ernesto.adorio at gmail.com (Ernesto Adorio) Date: Mon, 2 Aug 2010 19:13:13 +0800 Subject: [SciPy-User] Installing scipy 0.8.0 from sources. (where is recaster.py?) Message-ID: Here is the last message from trying to install scipy from sources.(Numpy installed fine) creating /usr/lib/python2.5/site-packages/scipy/io/arff copying build/lib.linux-i686-2.5/scipy/io/arff/myfunctools.py -> /usr/lib/python2.5/site-packages/scipy/io/arff copying build/lib.linux-i686-2.5/scipy/io/arff/utils.py -> /usr/lib/python2.5/site-packages/scipy/io/arff copying build/lib.linux-i686-2.5/scipy/io/arff/__init__.py -> /usr/lib/python2.5/site-packages/scipy/io/arff copying build/lib.linux-i686-2.5/scipy/io/arff/setup.py -> /usr/lib/python2.5/site-packages/scipy/io/arff copying build/lib.linux-i686-2.5/scipy/io/arff/arffread.py -> /usr/lib/python2.5/site-packages/scipy/io/arff copying build/lib.linux-i686-2.5/scipy/io/recaster.py -> /usr/lib/python2.5/site-packages/scipy/io error: /usr/lib/python2.5/site-packages/scipy/io/recaster.py: No such file or directory I am wondering what causes the last line error? error: /usr/lib/python2.5/site-packages/scipy/io/recaster.py: No such file or directory. Regards, ernie From ralf.gommers at googlemail.com Mon Aug 2 10:17:21 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 2 Aug 2010 22:17:21 +0800 Subject: [SciPy-User] Installing scipy 0.8.0 from sources. (where is recaster.py?) In-Reply-To: References: Message-ID: On Mon, Aug 2, 2010 at 7:13 PM, Ernesto Adorio wrote: > Here is the last message from trying to install scipy from > sources.(Numpy installed fine) > > creating /usr/lib/python2.5/site-packages/scipy/io/arff > copying build/lib.linux-i686-2.5/scipy/io/arff/myfunctools.py -> > /usr/lib/python2.5/site-packages/scipy/io/arff > copying build/lib.linux-i686-2.5/scipy/io/arff/utils.py -> > /usr/lib/python2.5/site-packages/scipy/io/arff > copying build/lib.linux-i686-2.5/scipy/io/arff/__init__.py -> > /usr/lib/python2.5/site-packages/scipy/io/arff > copying build/lib.linux-i686-2.5/scipy/io/arff/setup.py -> > /usr/lib/python2.5/site-packages/scipy/io/arff > copying build/lib.linux-i686-2.5/scipy/io/arff/arffread.py -> > /usr/lib/python2.5/site-packages/scipy/io/arff > copying build/lib.linux-i686-2.5/scipy/io/recaster.py -> > /usr/lib/python2.5/site-packages/scipy/io > error: /usr/lib/python2.5/site-packages/scipy/io/recaster.py: No such > file or directory > > I am wondering what causes the last line error? > > error: /usr/lib/python2.5/site-packages/scipy/io/recaster.py: No such > file or directory. > > The recaster.py file is present, but somehow you can't copy it over - strange error What OS, compilers and install command are you using? And can you please provide your whole build log? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From thoeger at fys.ku.dk Mon Aug 2 10:19:15 2010 From: thoeger at fys.ku.dk (=?ISO-8859-1?Q?Th=F8ger?= Emil Juul Thorsen) Date: Mon, 02 Aug 2010 16:19:15 +0200 Subject: [SciPy-User] Points with given distance to a polygon In-Reply-To: References: <1280162079.6595.5.camel@falconeer> Message-ID: <1280758755.2110.2.camel@zetkin> Shapely indeed looks like exactly what I was searching for! Thanks! I love Python; I'm able to do stuff after a couple of months which I couldn't figure out how to do in IDL after years of usage. :-D /Emil On Mon, 2010-07-26 at 14:28 -0500, Joe Kington wrote: > On Mon, Jul 26, 2010 at 1:26 PM, Zachary Pincus > wrote: > > I am working on a project where I am defining some regions > of > > interest. > > I have a 2200x2200 px 2D Array in which my ROI is defined by > a > > polygon. > > However, my data are smoothed by a gaussian kernel of width > 300px, > > and I > > would like to draw some lines indicating this inner 150px > distance to > > the borders of the ROI. I cannot come up with any way to do > this, does > > anyone have an idea? > > > Two broad options spring to mind: > (1) Geometric -- shrink the polygon along the normals to the > vertices. > [Oh, I see that eat has given pseudocode for same... good] > (2) Gridded -- rasterize the polygon to a binary mask (no > tools for > this in scipy, I fear... but if you're handy with opengl or > something, > that's not too hard), and then use scipy.ndimage to erode or > dilate > the mask as necessary. > > Zach > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > > If you go with a purely geometric method, look into a module like > shapely to buffer the polygon. (similar to the code snipped eat > posted, but more flexible) > > See this example for a general idea. > > -Joe > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From vanforeest at gmail.com Tue Aug 3 15:35:42 2010 From: vanforeest at gmail.com (nicky van foreest) Date: Tue, 3 Aug 2010 21:35:42 +0200 Subject: [SciPy-User] scipy.stats.uniform gives strange behavior (error?) Message-ID: Hi, It seems that the sum over the density of the uniform distribution does not always produces 1 (after proper division). Here is the code: ==== #!/usr/bin/env python from numpy import * from scipy.stats import uniform start = 0.; end = 2.; delta = 0.005; grid = arange(start,end,delta) rv = uniform(loc=1., scale = 0.1) f = rv.pdf(arange(delta, len(grid)*delta,delta))*delta # 1 print sum(f) print f ====== The result: 1.05 [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.05 0.05 0.05 0.05 0.05 0.05 0.05 0.05 0.05 0.05 0.05 0.05 0.05 0.05 0.05 0.05 0.05 0.05 0.05 0.05 0.05 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. ] This is weird, since the sum must be 1. Moreover, if I change line #1 into: f = rv.pdf(arange(0., len(grid)*delta,delta))*delta # 1 I get a sum of 1, as it should. If I change delta to 0.05 I get 1.0 in both cases. Does anybody have an idea about how this strange behavior may occur? Thanks Nicky From jrennie at gmail.com Tue Aug 3 15:53:57 2010 From: jrennie at gmail.com (Jason Rennie) Date: Tue, 3 Aug 2010 15:53:57 -0400 Subject: [SciPy-User] scipy.stats.uniform gives strange behavior (error?) In-Reply-To: References: Message-ID: Sounds like you're accidentally counting both fence-posts when you should only be counting one. Try summing densities from the middle of the buckets: f = rv.pdf(arange(delta+delta/2., len(grid)*delta+delta/2.,delta))*delta Jason On Tue, Aug 3, 2010 at 3:35 PM, nicky van foreest wrote: > Hi, > > It seems that the sum over the density of the uniform distribution > does not always produces 1 (after proper division). Here is the code: > > ==== > #!/usr/bin/env python > > from numpy import * > from scipy.stats import uniform > > start = 0.; end = 2.; delta = 0.005; grid = arange(start,end,delta) > > rv = uniform(loc=1., scale = 0.1) > > f = rv.pdf(arange(delta, len(grid)*delta,delta))*delta # 1 > print sum(f) > print f > ====== > > The result: > > 1.05 > [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0.05 0.05 0.05 0.05 0.05 0.05 0.05 0.05 > 0.05 0.05 0.05 0.05 0.05 0.05 0.05 0.05 0.05 0.05 0.05 0.05 > 0.05 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. > 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. ] > > This is weird, since the sum must be 1. Moreover, if I change line #1 into: > > f = rv.pdf(arange(0., len(grid)*delta,delta))*delta # 1 > > I get a sum of 1, as it should. > > If I change delta to 0.05 I get 1.0 in both cases. > > Does anybody have an idea about how this strange behavior may occur? > > Thanks > > Nicky > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanforeest at gmail.com Tue Aug 3 16:07:00 2010 From: vanforeest at gmail.com (nicky van foreest) Date: Tue, 3 Aug 2010 22:07:00 +0200 Subject: [SciPy-User] scipy.stats.uniform gives strange behavior (error?) In-Reply-To: References: Message-ID: Hi Jason, Thanks, this works. However, this solution leaves me with an uneasy feeling. From the shape of f (see my former mail) it appears that the fence posts of f are zero, since f starts at 0, and ends at 0. (Or do I misunderstand the meaning of fence posts?) It makes me actually wonder whether sum(rv.pdf(...) ) for any pdf in scipy might be allowed to exceed 1 ever. Shouldn't a (naive) user be protected from this? bye Nicky On 3 August 2010 21:53, Jason Rennie wrote: > Sounds like you're accidentally counting both fence-posts when you should > only be counting one. ?Try summing densities from the middle of the buckets: > f = ?rv.pdf(arange(delta+delta/2., len(grid)*delta+delta/2.,delta))*delta > Jason > > On Tue, Aug 3, 2010 at 3:35 PM, nicky van foreest > wrote: >> >> Hi, >> >> It seems that the sum over the density of the uniform distribution >> does not always produces 1 (after proper division). Here is the code: >> >> ==== >> #!/usr/bin/env python >> >> from numpy import * >> from scipy.stats import ?uniform >> >> start = 0.; end = 2.; delta = 0.005; grid = arange(start,end,delta) >> >> rv = uniform(loc=1., scale = 0.1) >> >> f = ?rv.pdf(arange(delta, len(grid)*delta,delta))*delta ? # 1 >> print ?sum(f) >> print f >> ====== >> >> The result: >> >> 1.05 >> [ 0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0.05 ?0.05 ?0.05 ?0.05 ?0.05 ?0.05 ?0.05 ?0.05 >> ?0.05 ?0.05 ?0.05 ?0.05 ?0.05 ?0.05 ?0.05 ?0.05 ?0.05 ?0.05 ?0.05 ?0.05 >> ?0.05 ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. >> ?0. >> ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ? ?0. ?] >> >> This is weird, since the sum must be 1. Moreover, if I change line #1 >> into: >> >> f = ?rv.pdf(arange(0., len(grid)*delta,delta))*delta ? # 1 >> >> I get a sum of 1, as it should. >> >> If I change delta to 0.05 I get 1.0 in both cases. >> >> Does anybody have an idea about how this strange behavior may occur? >> >> Thanks >> >> Nicky >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user > > > > -- > Jason Rennie > Research Scientist, ITA Software > 617-714-2645 > http://www.itasoftware.com/ > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From pav at iki.fi Tue Aug 3 16:14:54 2010 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 3 Aug 2010 20:14:54 +0000 (UTC) Subject: [SciPy-User] scipy.stats.uniform gives strange behavior (error?) References: Message-ID: Tue, 03 Aug 2010 21:35:42 +0200, nicky van foreest wrote: > It seems that the sum over the density of the uniform distribution does > not always produces 1 (after proper division). Here is the code: [clip] > rv = uniform(loc=1., scale = 0.1) > f = rv.pdf(arange(delta, len(grid)*delta,delta))*delta # 1 print It's a continuous distribution, and the *integral* over the PDF should give 1, and it indeed does: >>> from scipy.integrate import quad >>> quad(rv.pdf, 0, 2) (0.99999999999999989, 1.1102230246251563e-15) If you approximate the integral with a sum, as you did above, be prepared to get finite integration error. -- Pauli Virtanen From nbwood at lamar.colostate.edu Tue Aug 3 19:48:42 2010 From: nbwood at lamar.colostate.edu (Norm Wood) Date: Tue, 3 Aug 2010 17:48:42 -0600 Subject: [SciPy-User] Undefined symbols in _flinalg.so Message-ID: <20100803234842.GA7380@wombat.atmos.colostate.edu> Hi, I'd run into a problem with determinants last October while trying to use the kernel density estimation package in (at that time) SciPy 0.7.0 and 0.7.1. In scipy/linalg/basic.py, the determinant code has the lines: fdet, = get_flinalg_funcs(('det',),(a1,)) a_det,info = fdet(a1,overwrite_a=overwrite_a) and the second line was producing a TypeError: TypeError: 'NoneType' object is not callable I never resolved this with SciPy 0.7, but I've managed to dig further while trying to install 0.8.0. get_flinalg_funcs() is in sciyp/linalg/flinalg.py. The problem is occurring when flinalg.py tries to import _flinalg.so. An ImportError is raised, causing _flinalg to be set to None, and this in turn causes get_flinalg_funcs() to return None for fdet. If I go into my site-packages/scipy/linalg directory, start a python shell and try to import _flinalg.so, I get: >>> import _flinalg Traceback (most recent call last): File "", line 1, in ? ImportError: ./_flinalg.so: undefined symbol: ddet_c_ The routine ddet_c() is in scipy/linalg/src/det.f and appears mainly to be a wrapper which calls LAPACK's dgetrf to compute the LU factorization, then computes the determinant. I'd appreciate any suggestions about where to look further to resolve this. Running nm on _flinalg.so shows for ddet_c: U ddet_c_ 0000bed0 T ddet_c__ indicating ddet_c_ is undefined. dgetrf_ appears in _flinalg.so, but also has the symbol type "U" (undefined). Running ldd on _flinalg.so shows that it is linked to the liblapack.so and libatlas.so that I just built (Atlas 3.8.0). dgetrf_ shows up as a defined symbol (type "T") in liblapack.so. Note that the code for doing determinants in numpy (in numpy/linalg/linalg.py) also uses dgetrf, but imports it from a different shared object file, lapack_lite.so, and seems to work fine. Thanks for any assistance. I've attached below a summary of my system and build info. It's slightly different from that in my earlier message (newer numpy and scipy). I've also given a link to that October thread. Regards, Norm The previous thread: http://old.nabble.com/scipy.linalg.det-TypeError-td26100345.html and my current system and build info: These results are using LAPACK 3.1.1, Atlas 3.8.0, numpy 1.5.0b1, and SciPy 0.8.0, all built with gcc 3.3.6 and g77. Python is 2.4..1 built with gcc 3.3.5. I'm running Slackware 10.2 with kernel 2.4.31. Here's partial output from system_info.py: ATLAS version 3.8.0 built by norm on Tue Aug 3 13:09:42 MDT 2010: UNAME : Linux wombat 2.4.31-1 #1 SMP Wed Dec 7 16:52:41 MST 2005 i686 unknown unknown GNU/Linux INSTFLG : -1 0 -a 1 ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_P4E -DATL_CPUMHZ=3014 -DATL_SSE3 -DATL_SSE2 -DATL_SSE1 -DATL_GAS_x8632 F2CDEFS : -DAdd__ -DF77_INTEGER=int -DStringSunStyle CACHEEDGE: 1048576 F77 : g77, version GNU Fortran (GCC) 3.3.6 F77FLAGS : -O -fPIC -m32 SMC : gcc, version gcc (GCC) 3.3.6 SMCFLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 -fPIC -m32 SKC : gcc, version gcc (GCC) 3.3.6 SKCFLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 -fPIC -m32 success! removing: _configtest.c _configtest.o _configtest FOUND: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/home/norm/lusr/atlas/lib'] language = f77 define_macros = [('ATLAS_INFO', '"\\"3.8.0\\""')] include_dirs = ['/home/norm/lusr/atlas/include'] atlas_blas_info: libraries f77blas,cblas,atlas not found in /usr/local/lib FOUND: libraries = ['f77blas', 'cblas', 'atlas'] library_dirs = ['/home/norm/lusr/atlas/lib'] language = c include_dirs = ['/home/norm/lusr/atlas/include'] From thoeger at fys.ku.dk Tue Aug 3 20:05:13 2010 From: thoeger at fys.ku.dk (=?ISO-8859-1?Q?Th=F8ger?= Emil Juul Thorsen) Date: Wed, 04 Aug 2010 02:05:13 +0200 Subject: [SciPy-User] ROI, the sequel Message-ID: <1280880313.1937.115.camel@zetkin> Hello list; I need a tool for polygon clipping of 2d NumPy arrays, but I cannot seem to find anything. Shapely will only export the vertices of the polygon. NXutils and patches in matplotlib seem to be purely visual, in any case I cannot see how they would be saved as arrays in data coordinates. There is a very neat IDL routine, POYFILLV which returns a 1d array of the (1d) indices of the pixels that are inside the polygon, sort of like a "where" function. Is there anything like it out there for numpy arrays? Best; Emil From PHobson at Geosyntec.com Tue Aug 3 21:50:25 2010 From: PHobson at Geosyntec.com (PHobson at Geosyntec.com) Date: Tue, 3 Aug 2010 21:50:25 -0400 Subject: [SciPy-User] stats vs stats.mstats Message-ID: I was curious about the state of scipy.stats and scipy.stats.mstats. It seems to me that mstats is more advanced for some functions (like computing percentiles/quantiles). Is that the case? Any reason not to use mstats over stats entirely? Thanks, -paul From zachary.pincus at yale.edu Tue Aug 3 22:06:48 2010 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Tue, 3 Aug 2010 22:06:48 -0400 Subject: [SciPy-User] ROI, the sequel In-Reply-To: <1280880313.1937.115.camel@zetkin> References: <1280880313.1937.115.camel@zetkin> Message-ID: <0F1FEA94-5D89-4A40-A0B1-DD41B6BFA04E@yale.edu> Hello, The task you want to accomplish is called "polygon rasterization", and I have long looked for a simple python (or better, portable C that I could use from anywhere) module that does this well. (Ideally with optional antialiasing.) I really haven't found much... any graphics toolkit (e.g. for throwing up a GUI) can be convinced to rasterize to a memory buffer, which can be converted to a numpy array, but it's not simple or particularly portable. Lately I use OpenGL, but again not exactly simple. Perhaps the python bindings for the AGG library would be the best bet, but this is a pretty simple problem to solve with such a heavyweight dependency. Does anyone else know of any simple rasterization libs out there? Zach On Aug 3, 2010, at 8:05 PM, Th?ger Emil Juul Thorsen wrote: > Hello list; > > I need a tool for polygon clipping of 2d NumPy arrays, but I cannot > seem > to find anything. Shapely will only export the vertices of the > polygon. > NXutils and patches in matplotlib seem to be purely visual, in any > case > I cannot see how they would be saved as arrays in data coordinates. > > There is a very neat IDL routine, POYFILLV which returns a 1d array of > the (1d) indices of the pixels that are inside the polygon, sort of > like > a "where" function. Is there anything like it out there for numpy > arrays? > > Best; > > Emil > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From Chris.Barker at noaa.gov Tue Aug 3 22:26:03 2010 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 03 Aug 2010 19:26:03 -0700 Subject: [SciPy-User] ROI, the sequel In-Reply-To: <0F1FEA94-5D89-4A40-A0B1-DD41B6BFA04E@yale.edu> References: <1280880313.1937.115.camel@zetkin> <0F1FEA94-5D89-4A40-A0B1-DD41B6BFA04E@yale.edu> Message-ID: <4C58CFBB.7040207@noaa.gov> Zachary Pincus wrote: > The task you want to accomplish is called "polygon rasterization", and > I have long looked for a simple python (or better, portable C that I > could use from anywhere) module that does this well. (Ideally with > optional antialiasing.) I really haven't found much... any graphics > toolkit (e.g. for throwing up a GUI) can be convinced to rasterize to > a memory buffer, which can be converted to a numpy array, but it's not > simple or particularly portable. I've used PIL for that -- much less overhead than a GUI toolkit, if you don't need a GUI. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From ernesto.adorio at gmail.com Tue Aug 3 22:41:30 2010 From: ernesto.adorio at gmail.com (Ernesto Adorio) Date: Wed, 4 Aug 2010 10:41:30 +0800 Subject: [SciPy-User] SciPy-User Digest, Vol 84, Issue 3 In-Reply-To: References: Message-ID: Hi all, > Message: 2 > Date: Mon, 2 Aug 2010 22:17:21 +0800 > From: Ralf Gommers > Subject: Re: [SciPy-User] Installing scipy 0.8.0 from sources. (where > ? ? ? ?is ? ? ?recaster.py?) > To: SciPy Users List > Message-ID: > ? ? ? ? > Content-Type: text/plain; charset="iso-8859-1" > > On Mon, Aug 2, 2010 at 7:13 PM, Ernesto Adorio wrote: > >> Here is the last message from trying to install scipy from >> sources.(Numpy installed fine) >> >> creating /usr/lib/python2.5/site-packages/scipy/io/arff >> copying build/lib.linux-i686-2.5/scipy/io/arff/myfunctools.py -> >> /usr/lib/python2.5/site-packages/scipy/io/arff >> copying build/lib.linux-i686-2.5/scipy/io/arff/utils.py -> >> /usr/lib/python2.5/site-packages/scipy/io/arff >> copying build/lib.linux-i686-2.5/scipy/io/arff/__init__.py -> >> /usr/lib/python2.5/site-packages/scipy/io/arff >> copying build/lib.linux-i686-2.5/scipy/io/arff/setup.py -> >> /usr/lib/python2.5/site-packages/scipy/io/arff >> copying build/lib.linux-i686-2.5/scipy/io/arff/arffread.py -> >> /usr/lib/python2.5/site-packages/scipy/io/arff >> copying build/lib.linux-i686-2.5/scipy/io/recaster.py -> >> /usr/lib/python2.5/site-packages/scipy/io >> error: /usr/lib/python2.5/site-packages/scipy/io/recaster.py: No such >> file or directory >> >> I am wondering what causes the last line error? >> >> error: /usr/lib/python2.5/site-packages/scipy/io/recaster.py: No such >> file or directory. >> >> The recaster.py file is present, but somehow you can't copy it over - > strange error > > What OS, compilers and install command are you using? And can you please > provide your whole build log? > > Ralf > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: http://mail.scipy.org/pipermail/scipy-user/attachments/20100802/b3ea9d23/attachment-0001.html > Sorry, if I had not save the build log.The OS was 8.04 Hardy But in my impatience, I removed the alternate python2.4 directory and redid the installation from sources. [not the distribution package] using the standard configure; make; sudo make install The problem went away. Regards, -ernie http://adorio-research.org/wordpress From zachary.pincus at yale.edu Tue Aug 3 22:48:33 2010 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Tue, 3 Aug 2010 22:48:33 -0400 Subject: [SciPy-User] ROI, the sequel In-Reply-To: <4C58CFBB.7040207@noaa.gov> References: <1280880313.1937.115.camel@zetkin> <0F1FEA94-5D89-4A40-A0B1-DD41B6BFA04E@yale.edu> <4C58CFBB.7040207@noaa.gov> Message-ID: Hmm, I didn't realize PIL has a rasterizer. That's useful... (PIL and I parted ways a while ago, but I should dig back into the code to see if I can pull out the rasterizer and use it standalone.) Emil: PIL ought to be an easy way to get what you need, it seems. On Aug 3, 2010, at 10:26 PM, Christopher Barker wrote: > Zachary Pincus wrote: >> The task you want to accomplish is called "polygon rasterization", >> and >> I have long looked for a simple python (or better, portable C that I >> could use from anywhere) module that does this well. (Ideally with >> optional antialiasing.) I really haven't found much... any graphics >> toolkit (e.g. for throwing up a GUI) can be convinced to rasterize to >> a memory buffer, which can be converted to a numpy array, but it's >> not >> simple or particularly portable. > > I've used PIL for that -- much less overhead than a GUI toolkit, if > you > don't need a GUI. > > -Chris > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From Luke.Pinner at environment.gov.au Tue Aug 3 23:07:11 2010 From: Luke.Pinner at environment.gov.au (Pinner, Luke) Date: Wed, 4 Aug 2010 13:07:11 +1000 Subject: [SciPy-User] ROI, the sequel In-Reply-To: <1280880313.1937.115.camel@zetkin> References: <1280880313.1937.115.camel@zetkin> Message-ID: <64102B2FB850C345B04D09C82018F45F0273ED9B@ACT01VEXCCL04.internal.govt> GDAL/OGR perhaps? www.gdal.org Some semi-pseudo code below. from osgeo import gdal,gdal_array,ogr import numpy # Open a polygon layer to rasterize from. poly_ds = ogr.Open( some file dataset ) #Can also create an in-memory poly layer from scratch poly_lyr=poly_ds.GetLayer(0) # Create an in-memory raster to rasterize into target_ds = gdal.GetDriverByName('MEM').Create( approriate args...) #Rasterize... gdal.RasterizeLayer( appropriate args... ) numpy_array = gdal_aray.DatasetReadAsArray(target_ds) Then use the numpy_array as you lie. Luke -----Original Message----- From: scipy-user-bounces at scipy.org [mailto:scipy-user-bounces at scipy.org] On Behalf Of Th?ger Emil Juul Thorsen Sent: Wednesday, 4 August 2010 10:05 AM To: scipy-user at scipy.org Subject: [SciPy-User] ROI, the sequel Hello list; I need a tool for polygon clipping of 2d NumPy arrays, but I cannot seem to find anything. Shapely will only export the vertices of the polygon. NXutils and patches in matplotlib seem to be purely visual, in any case I cannot see how they would be saved as arrays in data coordinates. There is a very neat IDL routine, POYFILLV which returns a 1d array of the (1d) indices of the pixels that are inside the polygon, sort of like a "where" function. Is there anything like it out there for numpy arrays? Best; Emil _______________________________________________ SciPy-User mailing list SciPy-User at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user ------ If you have received this transmission in error please notify us immediately by return e-mail and delete all copies. If this e-mail or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the e-mail or attachments. Please consider the environment before printing this email. ------ From rwgk at yahoo.com Wed Aug 4 03:35:48 2010 From: rwgk at yahoo.com (Ralf W. Grosse-Kunstleve) Date: Wed, 4 Aug 2010 00:35:48 -0700 (PDT) Subject: [SciPy-User] ANN: Automatic Fortran to C++ conversion Message-ID: <582853.46216.qm@web111409.mail.gq1.yahoo.com> I thought this may be of interest to the readers of this list: http://cci.lbl.gov/fable/ fable - Automatic Fortran to C++ conversion * fable converts fixed-format Fortran sources to C++. The generated C++ code is designed to be human-readable and suitable for further development. In many cases it can be compiled and run without manual intervention. * fable comes with the C++ fem Fortran EMulation library. The entire fem library is inlined and therefore very easy to use. * The fem library has no dependencies other than a standard C++ compiler. * fable is written in Python (version 2.x) * The fable Fortran reader could be re-used to generate code for other target languages. * fable/fem is open source. From thoeger at fys.ku.dk Wed Aug 4 07:40:03 2010 From: thoeger at fys.ku.dk (=?ISO-8859-1?Q?Th=F8ger?= Emil Juul Thorsen) Date: Wed, 04 Aug 2010 13:40:03 +0200 Subject: [SciPy-User] ROI, the sequel In-Reply-To: <64102B2FB850C345B04D09C82018F45F0273ED9B@ACT01VEXCCL04.internal.govt> References: <1280880313.1937.115.camel@zetkin> <64102B2FB850C345B04D09C82018F45F0273ED9B@ACT01VEXCCL04.internal.govt> Message-ID: <1280922003.1937.121.camel@zetkin> Thanks to all of your for your suggestions - I think I'm going to try out the gdal solution first. But where can I request a similar function for includion in NumPy? I cannot be the only one needing to work with non-trivial Regions Of Interest on large arrays? Best; Emil On Wed, 2010-08-04 at 13:07 +1000, Pinner, Luke wrote: > GDAL/OGR perhaps? www.gdal.org > > Some semi-pseudo code below. > > from osgeo import gdal,gdal_array,ogr > import numpy > > # Open a polygon layer to rasterize from. > poly_ds = ogr.Open( some file dataset ) #Can also create an in-memory poly layer from scratch > poly_lyr=poly_ds.GetLayer(0) > > # Create an in-memory raster to rasterize into > target_ds = gdal.GetDriverByName('MEM').Create( approriate args...) > > #Rasterize... > gdal.RasterizeLayer( appropriate args... ) > numpy_array = gdal_aray.DatasetReadAsArray(target_ds) > > Then use the numpy_array as you lie. > > Luke > > -----Original Message----- > From: scipy-user-bounces at scipy.org [mailto:scipy-user-bounces at scipy.org] On Behalf Of Th?ger Emil Juul Thorsen > Sent: Wednesday, 4 August 2010 10:05 AM > To: scipy-user at scipy.org > Subject: [SciPy-User] ROI, the sequel > > Hello list; > > I need a tool for polygon clipping of 2d NumPy arrays, but I cannot seem > to find anything. Shapely will only export the vertices of the polygon. > NXutils and patches in matplotlib seem to be purely visual, in any case > I cannot see how they would be saved as arrays in data coordinates. > > There is a very neat IDL routine, POYFILLV which returns a 1d array of > the (1d) indices of the pixels that are inside the polygon, sort of like > a "where" function. Is there anything like it out there for numpy > arrays? > > Best; > > Emil > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > > ------ > If you have received this transmission in error please notify us immediately by return e-mail and delete all copies. If this e-mail or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the e-mail or attachments. > > > > Please consider the environment before printing this email. > > ------ > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From pav at iki.fi Wed Aug 4 07:50:08 2010 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 4 Aug 2010 11:50:08 +0000 (UTC) Subject: [SciPy-User] ROI, the sequel References: <1280880313.1937.115.camel@zetkin> <64102B2FB850C345B04D09C82018F45F0273ED9B@ACT01VEXCCL04.internal.govt> <1280922003.1937.121.camel@zetkin> Message-ID: Wed, 04 Aug 2010 13:40:03 +0200, Th?ger Emil Juul Thorsen wrote: > Thanks to all of your for your suggestions - I think I'm going to try > out the gdal solution first. > > But where can I request a similar function for includion in NumPy? I > cannot be the only one needing to work with non-trivial Regions Of > Interest on large arrays? I think this function is out-of-scope for Numpy, it's too specialized. In Scipy, there probably would be room for it, though, if someone writes it (patches are accepted ;). -- Pauli Virtanen From gammelmark at phys.au.dk Wed Aug 4 07:55:56 2010 From: gammelmark at phys.au.dk (=?ISO-8859-1?Q?S=F8ren_Gammelmark?=) Date: Wed, 04 Aug 2010 13:55:56 +0200 Subject: [SciPy-User] Compiling SciPy almost works Message-ID: <4C59554C.3060704@phys.au.dk> Hi everyone I just finished compiling NumPy 1.4.1 and SciPy 0.8.0 using Intel C++ and Fortran compilers (Version 9.1) and linking to Intel MKL 10.2. SciPy appears to compile fine, but a couple of the modules seems to be broken. When I import scipy.sparse i get the following error import scipy.sparse File "/home/sgamark/usr/lib/python2.6/site-packages/scipy/sparse/__init__.py", line 6, in from csr import * File "/home/sgamark/usr/lib/python2.6/site-packages/scipy/sparse/csr.py", line 12, in from sparsetools import csr_tocsc, csr_tobsr, csr_count_blocks, \ File "/home/sgamark/usr/lib/python2.6/site-packages/scipy/sparse/sparsetools/__init__.py", line 4, in from csr import * File "/home/sgamark/usr/lib/python2.6/site-packages/scipy/sparse/sparsetools/csr.py", line 7, in import _csr ImportError: /home/sgamark/usr/lib/python2.6/site-packages/scipy/sparse/sparsetools/_csr.so: undefined symbol: __gxx_personality_v0 It is the same problem with scipy.interpolate. From what I can find out by googling gxx_personality, it has something to do with gcc and g++ linking. This is odd however, since SciPy was compiled using python setup.py --compiler=intel --fcompiler=intelem Have anyone come across this problem before? (I couldn't find anything in the archives). Does anyone have a suggestion how to fix this? S?ren Gammelmark From bsouthey at gmail.com Wed Aug 4 09:44:29 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 04 Aug 2010 08:44:29 -0500 Subject: [SciPy-User] stats vs stats.mstats In-Reply-To: References: Message-ID: <4C596EBD.909@gmail.com> On 08/03/2010 08:50 PM, PHobson at Geosyntec.com wrote: > I was curious about the state of scipy.stats and scipy.stats.mstats. It seems to me that mstats is more advanced for some functions (like computing percentiles/quantiles). Is that the case? > > Any reason not to use mstats over stats entirely? > > Thanks, > -paul > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user scipy.stats uses 'standard' arrays. scipy.mstats uses masked arrays. Most of scipy.stats was directly implemented for masked arrays in scipy.mstats with some additional functionality. So while you can use scipy.mstats for non-masked arrays, it is can be 'overkill' and you should expect certain consequences such as masked arrays being returned. Bruce From ben.root at ou.edu Wed Aug 4 11:08:58 2010 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 4 Aug 2010 10:08:58 -0500 Subject: [SciPy-User] Compiling SciPy almost works In-Reply-To: <4C59554C.3060704@phys.au.dk> References: <4C59554C.3060704@phys.au.dk> Message-ID: On Wed, Aug 4, 2010 at 6:55 AM, S?ren Gammelmark wrote: > Hi everyone > > I just finished compiling NumPy 1.4.1 and SciPy 0.8.0 using Intel C++ > and Fortran compilers (Version 9.1) and linking to Intel MKL 10.2. SciPy > appears to compile fine, but a couple of the modules seems to be broken. > When I import scipy.sparse i get the following error > > import scipy.sparse > File > "/home/sgamark/usr/lib/python2.6/site-packages/scipy/sparse/__init__.py", > line > 6, in > from csr import * > File > "/home/sgamark/usr/lib/python2.6/site-packages/scipy/sparse/csr.py", > line 12, in > from sparsetools import csr_tocsc, csr_tobsr, csr_count_blocks, \ > File > > "/home/sgamark/usr/lib/python2.6/site-packages/scipy/sparse/sparsetools/__init__.py", > line 4, in > from csr import * > File > > "/home/sgamark/usr/lib/python2.6/site-packages/scipy/sparse/sparsetools/csr.py", > line 7, in > import _csr > ImportError: > > /home/sgamark/usr/lib/python2.6/site-packages/scipy/sparse/sparsetools/_csr.so: > undefined symbol: __gxx_personality_v0 > > It is the same problem with scipy.interpolate. From what I can find out > by googling gxx_personality, it has something to do with gcc and g++ > linking. This is odd however, since SciPy was compiled using > > python setup.py --compiler=intel --fcompiler=intelem > > Have anyone come across this problem before? (I couldn't find anything > in the archives). Does anyone have a suggestion how to fix this? > > S?ren Gammelmark > S?ren, Possibly you might have a dirty build where some stuff from your previous attempts compiled and some stuff that didn't is now getting compiled differently. Try completely removing the build directory and the numpy/scipy installs and start fresh. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From PHobson at Geosyntec.com Wed Aug 4 12:13:31 2010 From: PHobson at Geosyntec.com (PHobson at Geosyntec.com) Date: Wed, 4 Aug 2010 12:13:31 -0400 Subject: [SciPy-User] stats vs stats.mstats In-Reply-To: <4C596EBD.909@gmail.com> References: <4C596EBD.909@gmail.com> Message-ID: > -----Original Message----- > From: scipy-user-bounces at scipy.org [mailto:scipy-user- > bounces at scipy.org] On Behalf Of Bruce Southey > Sent: Wednesday, August 04, 2010 6:44 AM > To: scipy-user at scipy.org > Subject: Re: [SciPy-User] stats vs stats.mstats > > On 08/03/2010 08:50 PM, PHobson at Geosyntec.com wrote: > > I was curious about the state of scipy.stats and scipy.stats.mstats. > It seems to me that mstats is more advanced for some functions (like > computing percentiles/quantiles). Is that the case? > > > > Any reason not to use mstats over stats entirely? > > > > Thanks, > > -paul > Most of scipy.stats was directly implemented for masked arrays in > scipy.mstats with some additional functionality. So while you can use > scipy.mstats for non-masked arrays, it is can be 'overkill' and you > should expect certain consequences such as masked arrays being > returned. > > Bruce Thanks, Bruce. I can live with possibly getting a masked array back. -paul From cournape at gmail.com Wed Aug 4 12:50:03 2010 From: cournape at gmail.com (David Cournapeau) Date: Thu, 5 Aug 2010 01:50:03 +0900 Subject: [SciPy-User] Compiling SciPy almost works In-Reply-To: <4C59554C.3060704@phys.au.dk> References: <4C59554C.3060704@phys.au.dk> Message-ID: On Wed, Aug 4, 2010 at 8:55 PM, S?ren Gammelmark wrote: > Hi everyone > > I just finished compiling NumPy 1.4.1 and SciPy 0.8.0 using Intel C++ > and Fortran compilers (Version 9.1) and linking to Intel MKL 10.2. SciPy > appears to compile fine, but a couple of the modules seems to be broken. > When I import scipy.sparse i get the following error > > ? ? import scipy.sparse > ? File > "/home/sgamark/usr/lib/python2.6/site-packages/scipy/sparse/__init__.py", line > 6, in > ? ? from csr import * > ? File > "/home/sgamark/usr/lib/python2.6/site-packages/scipy/sparse/csr.py", > line 12, in > ? ? from sparsetools import csr_tocsc, csr_tobsr, csr_count_blocks, \ > ? File > "/home/sgamark/usr/lib/python2.6/site-packages/scipy/sparse/sparsetools/__init__.py", > line 4, in > ? ? from csr import * > ? File > "/home/sgamark/usr/lib/python2.6/site-packages/scipy/sparse/sparsetools/csr.py", > line 7, in > ? ? import _csr > ImportError: > /home/sgamark/usr/lib/python2.6/site-packages/scipy/sparse/sparsetools/_csr.so: > undefined symbol: __gxx_personality_v0 That's often caused by incompatibilities between C++ runtimes and compilers. Can you post the build.log with the relevant parts for sparsetools ? David From Dharhas.Pothina at twdb.state.tx.us Wed Aug 4 12:52:12 2010 From: Dharhas.Pothina at twdb.state.tx.us (Dharhas Pothina) Date: Wed, 04 Aug 2010 11:52:12 -0500 Subject: [SciPy-User] More efficient way of sorting and filtering structured array. Message-ID: <4C59546B.63BA.009B.0@twdb.state.tx.us> Hi, I have a structured array that contains intersection points between two sets of lines (specified by LINENAME and FILENAME). For each unique combination of LINENAME and FILENAME, there are a number of matches in the array intersection_points and I need only the closest match (i.e. the pdist field in the array is the smallest). The following snippet works but is extremely slow. I was wondering if there is a more efficient way to do this. # Creates closest_points array with zeros closest_points = np.zeros(0,dtype=pnt_dtype) # loops through unique linenames of intersection_points for linename in np.unique(intersection_points['LINENAME']): # loops through unique filenames of intersection_points for filename in np.unique(intersection_points['FILENAME']): # create seperate temporary arrays from intersection_points matching linename THEN filename idx_line = intersection_points['LINENAME'] == linename idx_file = intersection_points['FILENAME'] == filename # create temporary array from only points of correct linename AND filename tmp_points = intersection_points[idx_line * idx_file] # Eliminates empty array errors by making sure something is present if tmp_points.size > 0: # sort tmp_points array by pdist idx_sort = np.argsort(tmp_points, order='pdist') # add closest tmp_point to bottom of closest_points file closest_points = np.hstack((closest_points,tmp_points[idx_sort][0])) for reference pnt_dtype is : pnt_dtype = np.dtype([('lon','f8'),('lat','f8'),('x','f8'),('y','f8'), ('FILENAME','S50'), ('FILE_NUM','i4'), ('SSID','i8'), ('Lidx', 'i4'), ('pdist', 'f8'), ('LINENAME','S50'), ('HE_Code','i4'), ('PairNum','i4'), ('dist', 'S50') ]) thanks, - dharhas From Dharhas.Pothina at twdb.state.tx.us Wed Aug 4 13:26:35 2010 From: Dharhas.Pothina at twdb.state.tx.us (Dharhas Pothina) Date: Wed, 04 Aug 2010 12:26:35 -0500 Subject: [SciPy-User] More efficient way of sorting and filtering structuredarray. In-Reply-To: <4C59546B.63BA.009B.0@twdb.state.tx.us> References: <4C59546B.63BA.009B.0@twdb.state.tx.us> Message-ID: <4C595C7B.63BA.009B.0@twdb.state.tx.us> Just to add, one idea I had was to use argsort to sort the array by FILENAME, LINENAME and pdist. This would result in an array that had blocks with common FILENAME and LINENAME. I would then have to select the first line of each block which I'm not sure there is a easy way to do. - dharhas >>> "Dharhas Pothina" 8/4/2010 11:52 AM >>> Hi, I have a structured array that contains intersection points between two sets of lines (specified by LINENAME and FILENAME). For each unique combination of LINENAME and FILENAME, there are a number of matches in the array intersection_points and I need only the closest match (i.e. the pdist field in the array is the smallest). The following snippet works but is extremely slow. I was wondering if there is a more efficient way to do this. # Creates closest_points array with zeros closest_points = np.zeros(0,dtype=pnt_dtype) # loops through unique linenames of intersection_points for linename in np.unique(intersection_points['LINENAME']): # loops through unique filenames of intersection_points for filename in np.unique(intersection_points['FILENAME']): # create seperate temporary arrays from intersection_points matching linename THEN filename idx_line = intersection_points['LINENAME'] == linename idx_file = intersection_points['FILENAME'] == filename # create temporary array from only points of correct linename AND filename tmp_points = intersection_points[idx_line * idx_file] # Eliminates empty array errors by making sure something is present if tmp_points.size > 0: # sort tmp_points array by pdist idx_sort = np.argsort(tmp_points, order='pdist') # add closest tmp_point to bottom of closest_points file closest_points = np.hstack((closest_points,tmp_points[idx_sort][0])) for reference pnt_dtype is : pnt_dtype = np.dtype([('lon','f8'),('lat','f8'),('x','f8'),('y','f8'), ('FILENAME','S50'), ('FILE_NUM','i4'), ('SSID','i8'), ('Lidx', 'i4'), ('pdist', 'f8'), ('LINENAME','S50'), ('HE_Code','i4'), ('PairNum','i4'), ('dist', 'S50') ]) thanks, - dharhas _______________________________________________ SciPy-User mailing list SciPy-User at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user From jkington at wisc.edu Wed Aug 4 13:56:34 2010 From: jkington at wisc.edu (Joe Kington) Date: Wed, 04 Aug 2010 12:56:34 -0500 Subject: [SciPy-User] ROI, the sequel In-Reply-To: <1280922003.1937.121.camel@zetkin> References: <1280880313.1937.115.camel@zetkin> <64102B2FB850C345B04D09C82018F45F0273ED9B@ACT01VEXCCL04.internal.govt> <1280922003.1937.121.camel@zetkin> Message-ID: For whatever it's worth, matplotlib has a fairly fast function matplotlib.nxutils.points_inside_poly that will do this for polygons with no "holes". To use it to rasterize a polygon region of interest, just build a 2xN array of the x,y locations of the verticies of your grid, pass it to to points_inside_poly along with the polygon verticies, and then reshape the output boolean mask to the size of your original grid. Hope that helps a bit, -Joe 2010/8/4 Th?ger Emil Juul Thorsen > Thanks to all of your for your suggestions - I think I'm going to try > out the gdal solution first. > > But where can I request a similar function for includion in NumPy? I > cannot be the only one needing to work with non-trivial Regions Of > Interest on large arrays? > > Best; > > Emil > > > > On Wed, 2010-08-04 at 13:07 +1000, Pinner, Luke wrote: > > GDAL/OGR perhaps? www.gdal.org > > > > Some semi-pseudo code below. > > > > from osgeo import gdal,gdal_array,ogr > > import numpy > > > > # Open a polygon layer to rasterize from. > > poly_ds = ogr.Open( some file dataset ) #Can also create an in-memory > poly layer from scratch > > poly_lyr=poly_ds.GetLayer(0) > > > > # Create an in-memory raster to rasterize into > > target_ds = gdal.GetDriverByName('MEM').Create( approriate args...) > > > > #Rasterize... > > gdal.RasterizeLayer( appropriate args... ) > > numpy_array = gdal_aray.DatasetReadAsArray(target_ds) > > > > Then use the numpy_array as you lie. > > > > Luke > > > > -----Original Message----- > > From: scipy-user-bounces at scipy.org [mailto:scipy-user-bounces at scipy.org] > On Behalf Of Th?ger Emil Juul Thorsen > > Sent: Wednesday, 4 August 2010 10:05 AM > > To: scipy-user at scipy.org > > Subject: [SciPy-User] ROI, the sequel > > > > Hello list; > > > > I need a tool for polygon clipping of 2d NumPy arrays, but I cannot seem > > to find anything. Shapely will only export the vertices of the polygon. > > NXutils and patches in matplotlib seem to be purely visual, in any case > > I cannot see how they would be saved as arrays in data coordinates. > > > > There is a very neat IDL routine, POYFILLV which returns a 1d array of > > the (1d) indices of the pixels that are inside the polygon, sort of like > > a "where" function. Is there anything like it out there for numpy > > arrays? > > > > Best; > > > > Emil > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > > > ------ > > If you have received this transmission in error please notify us > immediately by return e-mail and delete all copies. If this e-mail or any > attachments have been sent to you in error, that error does not constitute > waiver of any confidentiality, privilege or copyright in respect of > information in the e-mail or attachments. > > > > > > > > Please consider the environment before printing this email. > > > > ------ > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From schrabacke at web.de Sun Aug 1 15:25:37 2010 From: schrabacke at web.de (Jana Schulz) Date: Sun, 1 Aug 2010 21:25:37 +0200 (CEST) Subject: [SciPy-User] Interpolation in 3D with interp2d In-Reply-To: References: Message-ID: <1456192421.951467.1280690737282.JavaMail.fmail@mwmweb001> Hi, I'm trying to interpolate a 3D data (from the pic attached) with the interp2d command. What I have, are three vectors f, z, A (x, y, z respectively, A is the percentage data given on the isolines). I first put the f and z in a meshgrid and afterwards pruduced a mesh with the A vector then started to interpolate. I plotted the the data after gridding, and I observed that almost all nodes are ignored. Do you have any idea how to prepare data to the interp2d command? my code so far is: import numpy as np from matplotlib import rc from mpl_toolkits.mplot3d import axes3d from scipy.interpolate import interp2d from scipy import interpolate, mgrid import matplotlib.pyplot as plt from matplotlib import mlab import griddata plt.close() plt.clf() fig = plt.figure(1) ax = axes3d.Axes3D(fig) #read data (ff,ZZ,A,a) = np.loadtxt("accuracy-map_pointsonly.txt", unpack=True) f=np.log10(ff) z=np.log10(ZZ) ##grid everything fgrid, zgrid=np.meshgrid(f,z) #define grid n=1e2 #number of nodes (linear) ef=np.linspace(min(f), max(f), n) ez=np.linspace(min(z), max(z), n) efmesh, ezmesh=np.meshgrid(ef,ez) efg, ezg=np.meshgrid(f, z) Ag,Ag=np.meshgrid(A,A) int2d=interp2d(fgrid, zgrid, Ag, kind='linear') intnew=int2d(ef, ez) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? #return the new function ax.plot(f,z,A,'ok', markerfacecolor='w') ax.plot_surface(efmesh,ezmesh, intnew) ax.view_init(elev=220, azim=270) ax.set_xlim3d((min(f), max(f))) ax.set_ylim3d(min(z), max(z)) ax.set_zlim3d(0,100) plt.show() ___________________________________________________________ WEB.DE DSL ab 19,99 Euro/Monat. Bis zu 150,- Euro Startguthaben und 50,- Euro Geldpr?mie inklusive! https://freundschaftswerbung.web.de -------------- next part -------------- A non-text attachment was scrubbed... Name: novo-error.pdf Type: application/octet-stream Size: 151539 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: accuracy-map_pointsonly.txt Type: application/octet-stream Size: 2817 bytes Desc: not available URL: From robert.streubel at googlemail.com Mon Aug 2 16:42:12 2010 From: robert.streubel at googlemail.com (Robert Streubel) Date: Mon, 2 Aug 2010 22:42:12 +0200 Subject: [SciPy-User] Optimization with boundary conditions Message-ID: Hello there, I'd like to minimize a scalar function F(a(x,y),b(x,y)) - with other words a functional - under certain boundary conditions, such as a+b<1. Since Scipy supports minimization, I wonder if there could be a rather convenient way to imply boundary conditions. Because this condition causes a different minimum, it can not be considered recursively by a second fmin routine. Hence, could anyway assist me, please? Cheers Robert From jsseabold at gmail.com Wed Aug 4 14:28:59 2010 From: jsseabold at gmail.com (Skipper Seabold) Date: Wed, 4 Aug 2010 14:28:59 -0400 Subject: [SciPy-User] scipy.spatial.distance.mahalanobis & inverse covariance matrix In-Reply-To: References: Message-ID: On Wed, Jul 21, 2010 at 8:22 AM, bearce meanu wrote: > Dear experts, > > i just switched from matlab to scipy/numpy and i am sorry for this > very basic question. > > my goal is to calculate the mahalanobis distance btw to vectors x & y. > Here is my code: > > from scipy.spatial.distance import mahalanobis > import numpy as np > x=np.random.normal(size=25) > y=np.random.normal(size=25) > V = np.linalg.inv(np.cov(np.concatenate((x, y)).T)) # inverse covariance matrix > Have a look at what concatenate actually does here. The problem is that inv is getting a scalar array (which I think would be nice to have linalg.inv and linalg.det handle...), though I think it could use a more useful error message. In [59]: np.linalg.inv(np.array(1.5)) --------------------------------------------------------------------------- IndexError Traceback (most recent call last) /home/skipper/ in () /usr/local/lib/python2.6/dist-packages/numpy/linalg/linalg.pyc in inv(a) 443 """ 444 a, wrap = _makearray(a) --> 445 return wrap(solve(a, identity(a.shape[0], dtype=a.dtype))) 446 447 IndexError: tuple index out of range You probably want something like this. Note that you don't have to stack x and y, but you could do np.linalg.cov(np.column_stack((x,y)), rowvar=0) or np.linalg.cov(x,y, rowvar=0) Note that the rowvar=0 tells the cov function that you have your variables in columns, so you don't have to take the transpose. hth, Skipper From dwf at cs.toronto.edu Wed Aug 4 15:00:22 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Wed, 4 Aug 2010 15:00:22 -0400 Subject: [SciPy-User] Scipy.io .mat files are bigger than those made with Matlab In-Reply-To: <4C5023E4.5040900@pasteur.fr> References: <4C5023E4.5040900@pasteur.fr> Message-ID: <8EE73C6F-5F65-4AC3-AAD0-3F192692991A@cs.toronto.edu> On 2010-07-28, at 8:34 AM, Hua Wong wrote: > Is there any option I should set to make these .mat smaller? > > One 2726*2726 matrix gives 10Mo with matlab and goes up to 57Mo when > exported with scipy.io help(scipy.io.matlab.savemat), look for the do_compression option. It's there since at least 0.7.2. David From pav at iki.fi Wed Aug 4 15:21:32 2010 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 4 Aug 2010 19:21:32 +0000 (UTC) Subject: [SciPy-User] Interpolation in 3D with interp2d References: <1456192421.951467.1280690737282.JavaMail.fmail@mwmweb001> Message-ID: Sun, 01 Aug 2010 21:25:37 +0200, Jana Schulz wrote: > I'm trying to interpolate a 3D data (from the pic attached) with the > interp2d command. What I have, are three vectors f, z, A (x, y, z > respectively, A is the percentage data given on the isolines). I first > put the f and z in a meshgrid and afterwards pruduced a mesh with the A > vector then started to interpolate. This sounds like you are trying to use `interp2d` for something it does not do -- it requires that your data is already gridded. So let's clarify: you have points (f[i], z[i]), and function values A[i]. The points (f[i], z[i]) do not form a regular grid. If yes, then you almost certainly are looking for the `griddata` function: >>> from matplotlib.mlab import griddata Or possibly, you could also try to use splines: >>> from scipy.intepolate import SmoothBivariateSpline >>> int2d = SmoothBivariateSpline(f, z, A, s=0) >>> intnew = int2d(ef, ez) Mind the `s=0` -- otherwise it will try to smooth your data. However, if your data point distribution is irregular, the spline method will likely produce bad results -- they're OK for smoothing, but not so nice for data regridding. -- Pauli Virtanen From pav at iki.fi Wed Aug 4 15:25:13 2010 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 4 Aug 2010 19:25:13 +0000 (UTC) Subject: [SciPy-User] Interpolation in 3D with interp2d References: <1456192421.951467.1280690737282.JavaMail.fmail@mwmweb001> Message-ID: Sun, 01 Aug 2010 21:25:37 +0200, Jana Schulz wrote: > I'm trying to interpolate a 3D data (from the pic attached) with the > interp2d command. What I have, are three vectors f, z, A (x, y, z > respectively, A is the percentage data given on the isolines). [clip] Out of curiosity, this mail is almost identical with this one sent here last Thursday: From: "Hardock, Andreas" Newsgroups: gmane.comp.python.scientific.user Subject: Linear interpolation in 3D Date: Thu, 29 Jul 2010 08:09:03 +0200 Same person? -- Pauli Virtanen From zachary.pincus at yale.edu Wed Aug 4 15:40:54 2010 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Wed, 4 Aug 2010 15:40:54 -0400 Subject: [SciPy-User] Interpolation in 3D with interp2d In-Reply-To: References: <1456192421.951467.1280690737282.JavaMail.fmail@mwmweb001> Message-ID: > Sun, 01 Aug 2010 21:25:37 +0200, Jana Schulz wrote: >> I'm trying to interpolate a 3D data (from the pic attached) with the >> interp2d command. What I have, are three vectors f, z, A (x, y, z >> respectively, A is the percentage data given on the isolines). I >> first >> put the f and z in a meshgrid and afterwards pruduced a mesh with >> the A >> vector then started to interpolate. > > This sounds like you are trying to use `interp2d` for something it > does > not do -- it requires that your data is already gridded. > > So let's clarify: you have points (f[i], z[i]), and function values > A[i]. > The points (f[i], z[i]) do not form a regular grid. If yes, then you > almost certainly are looking for the `griddata` function: > >>>> from matplotlib.mlab import griddata > > Or possibly, you could also try to use splines: > >>>> from scipy.intepolate import SmoothBivariateSpline >>>> int2d = SmoothBivariateSpline(f, z, A, s=0) >>>> intnew = int2d(ef, ez) > > Mind the `s=0` -- otherwise it will try to smooth your data. > However, if > your data point distribution is irregular, the spline method will > likely > produce bad results -- they're OK for smoothing, but not so nice for > data > regridding. See also: http://www.scipy.org/Cookbook/RadialBasisFunctions and the delaunay scikit, both of which deal directly with interpolating scattered data, if you don't necessarily need to interpolate on a complete grid. (The griddata function uses these methods under the hood, I think.) Zach From ben.root at ou.edu Wed Aug 4 19:38:51 2010 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 4 Aug 2010 18:38:51 -0500 Subject: [SciPy-User] random points within an ellipse Message-ID: Hi, For a project, I need to create sets of random coordinates within a 2d domain. To start, I have been creating random x and y coordinates, which has worked very nicely. However, I would like to start doing some fancier domains like ellipses and crescents. Does anybody know of any useful tricks for doing this? Thanks, Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From argriffi at ncsu.edu Wed Aug 4 20:18:25 2010 From: argriffi at ncsu.edu (alex) Date: Wed, 4 Aug 2010 20:18:25 -0400 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: Message-ID: On Wed, Aug 4, 2010 at 7:38 PM, Benjamin Root wrote: > Hi, > > For a project, I need to create sets of random coordinates within a 2d > domain. To start, I have been creating random x and y coordinates, which > has worked very nicely. However, I would like to start doing some fancier > domains like ellipses and crescents. Does anybody know of any useful tricks > for doing this? > > Thanks, > Ben Root > For an ellipse you could start with random points in a disk like this http://mathworld.wolfram.com/DiskPointPicking.html and then I think you can just stretch along an axis to give the points inside the ellipse disk. For fancier shapes you can do rejection sampling as long as you can get a bounding box and you can tell what is inside vs outside the domain. Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Aug 4 21:11:21 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 4 Aug 2010 21:11:21 -0400 Subject: [SciPy-User] Porting matlab's ksdensity to scipy In-Reply-To: <31ca5275-e3b6-4f65-a896-740f9f1eb0ba@l14g2000yql.googlegroups.com> References: <31ca5275-e3b6-4f65-a896-740f9f1eb0ba@l14g2000yql.googlegroups.com> Message-ID: On Mon, Jul 19, 2010 at 12:20 PM, Tadas Vilkeliskis wrote: > Hi guys! > > I am trying to port the code written in matlab to python and I got > stuck with matlab's ksdensity function. The matlab code I have looks > something like this: > > [f, x] = ksdensity(values) > > I used scipy's gaussian_kde to achieve the same result; however, I > modified the matlab's code to f = ksdensity(values, x) and where x are > fixed values. When x are fixed gaussian_kde works fine, but this is > not what I need. I want to select x based on the input values as in > [f, x] = ksdensity(values) which returns x based on the input. Is > there a way to do this in scipy? For instance, if I have input values > [0, 1, 40, 2, 3, 2] how do I get the range of x values? isn't this just x = np.linspace(values.min(), values.max(), 100) ? from only a quick look at the matlab help: "The density is evaluated at 100 equally spaced points that cover the range of the data in x" Josef > > Thank you very much. > > Tadas > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From josef.pktd at gmail.com Wed Aug 4 21:12:48 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 4 Aug 2010 21:12:48 -0400 Subject: [SciPy-User] Porting matlab's ksdensity to scipy In-Reply-To: References: <31ca5275-e3b6-4f65-a896-740f9f1eb0ba@l14g2000yql.googlegroups.com> Message-ID: On Wed, Aug 4, 2010 at 9:11 PM, wrote: > On Mon, Jul 19, 2010 at 12:20 PM, Tadas Vilkeliskis > wrote: >> Hi guys! >> >> I am trying to port the code written in matlab to python and I got >> stuck with matlab's ksdensity function. The matlab code I have looks >> something like this: >> >> [f, x] = ksdensity(values) >> >> I used scipy's gaussian_kde to achieve the same result; however, I >> modified the matlab's code to f = ksdensity(values, x) and where x are >> fixed values. When x are fixed gaussian_kde works fine, but this is >> not what I need. I want to select x based on the input values as in >> [f, x] = ksdensity(values) which returns x based on the input. Is >> there a way to do this in scipy? For instance, if I have input values >> [0, 1, 40, 2, 3, 2] how do I get the range of x values? > > isn't this just x = np.linspace(values.min(), values.max(), 100) ?? > > from only a quick look at the matlab help: > "The density is evaluated at 100 equally spaced points that cover the > range of the data in x" > > Josef Ignore if this is too late, I just realized that I got some old emails today. Josef > >> >> Thank you very much. >> >> Tadas >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > From ben.root at ou.edu Wed Aug 4 23:13:03 2010 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 4 Aug 2010 22:13:03 -0500 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: Message-ID: On Wed, Aug 4, 2010 at 7:18 PM, alex wrote: > On Wed, Aug 4, 2010 at 7:38 PM, Benjamin Root wrote: > >> Hi, >> >> For a project, I need to create sets of random coordinates within a 2d >> domain. To start, I have been creating random x and y coordinates, which >> has worked very nicely. However, I would like to start doing some fancier >> domains like ellipses and crescents. Does anybody know of any useful tricks >> for doing this? >> >> Thanks, >> Ben Root >> > > For an ellipse you could start with random points in a disk like this > http://mathworld.wolfram.com/DiskPointPicking.html > and then I think you can just stretch along an axis to give the points > inside the ellipse disk. > For fancier shapes you can do rejection sampling as long as you can get a > bounding box and > you can tell what is inside vs outside the domain. > > Alex > > Thanks Alex, that certainly does put me in the correct direction. It appears that there isn't an analogous expression for an ellipse as there was for a disc. I probably will have to just take your suggestion and do some scaling/rotation upon the set of points to get what I want. It should be more than sufficient, and it does look better than my original attempts. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From gammelmark at phys.au.dk Thu Aug 5 02:27:19 2010 From: gammelmark at phys.au.dk (=?UTF-8?B?U8O4cmVuIEdhbW1lbG1hcms=?=) Date: Thu, 05 Aug 2010 08:27:19 +0200 Subject: [SciPy-User] Compiling SciPy almost works In-Reply-To: References: <4C59554C.3060704@phys.au.dk> Message-ID: <4C5A59C7.5070204@phys.au.dk> I have attached the build-log. As far as I can see it chooses the Intel compiler with the correct options as it should. Also recompiling after removing build did not change anything (I do not believe it should have, since I was quite careful of doing this with every attempt). Thanks again S?ren On 2010-08-04 18:50, David Cournapeau wrote: > On Wed, Aug 4, 2010 at 8:55 PM, S?ren Gammelmark wrote: > >> Hi everyone >> >> I just finished compiling NumPy 1.4.1 and SciPy 0.8.0 using Intel C++ >> and Fortran compilers (Version 9.1) and linking to Intel MKL 10.2. SciPy >> appears to compile fine, but a couple of the modules seems to be broken. >> When I import scipy.sparse i get the following error >> >> import scipy.sparse >> File >> "/home/sgamark/usr/lib/python2.6/site-packages/scipy/sparse/__init__.py", line >> 6, in >> from csr import * >> File >> "/home/sgamark/usr/lib/python2.6/site-packages/scipy/sparse/csr.py", >> line 12, in >> from sparsetools import csr_tocsc, csr_tobsr, csr_count_blocks, \ >> File >> "/home/sgamark/usr/lib/python2.6/site-packages/scipy/sparse/sparsetools/__init__.py", >> line 4, in >> from csr import * >> File >> "/home/sgamark/usr/lib/python2.6/site-packages/scipy/sparse/sparsetools/csr.py", >> line 7, in >> import _csr >> ImportError: >> /home/sgamark/usr/lib/python2.6/site-packages/scipy/sparse/sparsetools/_csr.so: >> undefined symbol: __gxx_personality_v0 >> > That's often caused by incompatibilities between C++ runtimes and > compilers. Can you post the build.log with the relevant parts for > sparsetools ? > > David > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: build.log URL: From d.l.goldsmith at gmail.com Thu Aug 5 02:54:23 2010 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Wed, 4 Aug 2010 23:54:23 -0700 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: Message-ID: Whoops, sorry, in the example, the call to rand_in_ellipse should have 4./3 as a single argument, i.e., the line should look like sample = np.array([rand_in_ellipse(4./3) for i in np.arange(10000)]) DG On Wed, Aug 4, 2010 at 11:50 PM, David Goldsmith wrote: > For the ellipse: > > import numpy as np > from numpy.random import random_sample as random > > pi = np.pi > def rand_in_ellipse(a, b=1, offset=0): > angle = 2*pi*random() - offset > x = a * random() * np.cos(angle) > y = b * random() * np.sin(angle) > return np.array((x,y)) > > import matplotlib.pyplot as plt > > fig = plt.figure() > ax = fig.add_subplot(111) > > sample = np.array([rand_in_ellipse() for i in np.arange(10000)]) > ax.scatter(sample[:,0], sample[:,1]) > plt.show() # Figure attached > > DG > > > On Wed, Aug 4, 2010 at 8:13 PM, Benjamin Root wrote: > >> On Wed, Aug 4, 2010 at 7:18 PM, alex wrote: >> >>> On Wed, Aug 4, 2010 at 7:38 PM, Benjamin Root wrote: >>> >>>> Hi, >>>> >>>> For a project, I need to create sets of random coordinates within a 2d >>>> domain. To start, I have been creating random x and y coordinates, which >>>> has worked very nicely. However, I would like to start doing some fancier >>>> domains like ellipses and crescents. Does anybody know of any useful tricks >>>> for doing this? >>>> >>>> Thanks, >>>> Ben Root >>>> >>> >>> For an ellipse you could start with random points in a disk like this >>> http://mathworld.wolfram.com/DiskPointPicking.html >>> and then I think you can just stretch along an axis to give the points >>> inside the ellipse disk. >>> For fancier shapes you can do rejection sampling as long as you can get a >>> bounding box and >>> you can tell what is inside vs outside the domain. >>> >>> Alex >>> >>> >> Thanks Alex, that certainly does put me in the correct direction. It >> appears that there isn't an analogous expression for an ellipse as there was >> for a disc. I probably will have to just take your suggestion and do some >> scaling/rotation upon the set of points to get what I want. It should be >> more than sufficient, and it does look better than my original attempts. >> >> Ben Root >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > > -- > Mathematician: noun, someone who disavows certainty when their uncertainty > set is non-empty, even if that set has measure zero. > > Hope: noun, that delusive spirit which escaped Pandora's jar and, with her > lies, prevents mankind from committing a general suicide. (As interpreted > by Robert Graves) > -- Mathematician: noun, someone who disavows certainty when their uncertainty set is non-empty, even if that set has measure zero. Hope: noun, that delusive spirit which escaped Pandora's jar and, with her lies, prevents mankind from committing a general suicide. (As interpreted by Robert Graves) -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at silveregg.co.jp Thu Aug 5 03:11:33 2010 From: david at silveregg.co.jp (David) Date: Thu, 05 Aug 2010 16:11:33 +0900 Subject: [SciPy-User] Compiling SciPy almost works In-Reply-To: <4C5A59C7.5070204@phys.au.dk> References: <4C59554C.3060704@phys.au.dk> <4C5A59C7.5070204@phys.au.dk> Message-ID: <4C5A6425.5040604@silveregg.co.jp> On 08/05/2010 03:27 PM, S?ren Gammelmark wrote: > I have attached the build-log. As far as I can see it chooses the Intel > compiler with the correct options as it should. > > Also recompiling after removing build did not change anything (I do not > believe it should have, since I was quite careful of doing this with > every attempt). Could you try linking the extension "manually" and add -lstdc++, i.e.. icc -fPIC -fomit-frame-pointer -O2 -g -parallel -shared build/temp.linux-x86_64-2.6/scipy/sparse/sparsetools/csr_wrap.o -Lbuild/temp.linux-x86_64-2.6 -o build/lib.linux-x86_64-2.6/scipy/sparse/sparsetools/_csr.so -lstdc++ and see it you can do python -c "import _csr" in the directory where _csr.so is located. If this does not work, can you get the verbose output of the link (I think something like icc -## or icc -v should do it - the goal is to see every line used by the driver icc to get the exact options used for linking) cheers, David From gammelmark at phys.au.dk Thu Aug 5 03:28:39 2010 From: gammelmark at phys.au.dk (=?UTF-8?B?U8O4cmVuIEdhbW1lbG1hcms=?=) Date: Thu, 05 Aug 2010 09:28:39 +0200 Subject: [SciPy-User] Compiling SciPy almost works In-Reply-To: <4C5A6425.5040604@silveregg.co.jp> References: <4C59554C.3060704@phys.au.dk> <4C5A59C7.5070204@phys.au.dk> <4C5A6425.5040604@silveregg.co.jp> Message-ID: <4C5A6827.5090803@phys.au.dk> When trying the manual link everything works out fine. "import _csr" executes without any problem. I also tried using the Intel C++ compiler (icpc) and then everything works out fine. I guess the default link-libraries are different for the programs. Should I then just append -lstdc++ to the icc-command or what would be the best way of solving this? S?ren On 2010-08-05 09:11, David wrote: > On 08/05/2010 03:27 PM, S?ren Gammelmark wrote: > >> I have attached the build-log. As far as I can see it chooses the Intel >> compiler with the correct options as it should. >> >> Also recompiling after removing build did not change anything (I do not >> believe it should have, since I was quite careful of doing this with >> every attempt). >> > Could you try linking the extension "manually" and add -lstdc++, i.e.. > > icc -fPIC -fomit-frame-pointer -O2 -g -parallel -shared > build/temp.linux-x86_64-2.6/scipy/sparse/sparsetools/csr_wrap.o > -Lbuild/temp.linux-x86_64-2.6 -o > build/lib.linux-x86_64-2.6/scipy/sparse/sparsetools/_csr.so -lstdc++ > > and see it you can do python -c "import _csr" in the directory where > _csr.so is located. > > If this does not work, can you get the verbose output of the link (I > think something like icc -## or icc -v should do it - the goal is to see > every line used by the driver icc to get the exact options used for linking) > > cheers, > > David > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From david at silveregg.co.jp Thu Aug 5 03:33:51 2010 From: david at silveregg.co.jp (David) Date: Thu, 05 Aug 2010 16:33:51 +0900 Subject: [SciPy-User] Compiling SciPy almost works In-Reply-To: <4C5A6827.5090803@phys.au.dk> References: <4C59554C.3060704@phys.au.dk> <4C5A59C7.5070204@phys.au.dk> <4C5A6425.5040604@silveregg.co.jp> <4C5A6827.5090803@phys.au.dk> Message-ID: <4C5A695F.7090104@silveregg.co.jp> On 08/05/2010 04:28 PM, S?ren Gammelmark wrote: > When trying the manual link everything works out fine. "import _csr" > executes without any problem. I also tried using the Intel C++ compiler > (icpc) and then everything works out fine. I guess the default > link-libraries are different for the programs. Hm, so they change the C++ driver - that's almost certainly the root of the issue. Does setting CXX to icpc works ? David From sebastian.walter at gmail.com Thu Aug 5 03:46:26 2010 From: sebastian.walter at gmail.com (Sebastian Walter) Date: Thu, 5 Aug 2010 09:46:26 +0200 Subject: [SciPy-User] Optimization with boundary conditions In-Reply-To: References: Message-ID: It really depends on what kind of optimization problem you have. On http://openopt.org/NLP there is a nice chart that summarizes the different classes of optimization problems. Once you have identified the problem class you can look for an appropriate algorithm. Hope that helps a little. Sebastian On Mon, Aug 2, 2010 at 10:42 PM, Robert Streubel wrote: > Hello there, > > I'd like to minimize a scalar function F(a(x,y),b(x,y)) - with other words a functional - under certain boundary conditions, such as a+b<1. > Since Scipy supports minimization, I wonder if there could be a rather convenient way to imply boundary conditions. Because this condition causes a different minimum, it can not be considered recursively by a second fmin routine. > Hence, could anyway assist me, please? > > Cheers > Robert > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From sebastian.walter at gmail.com Thu Aug 5 03:47:34 2010 From: sebastian.walter at gmail.com (Sebastian Walter) Date: Thu, 5 Aug 2010 09:47:34 +0200 Subject: [SciPy-User] Optimization with boundary conditions In-Reply-To: References: Message-ID: sorry, posted the wrong link: it was meant to be http://openopt.org/Problems and not http://openopt.org/NLP On Thu, Aug 5, 2010 at 9:46 AM, Sebastian Walter wrote: > It really depends on what kind of optimization problem you have. > On http://openopt.org/NLP there is a nice chart that summarizes the > different classes of optimization problems. > Once you have identified the problem class you can look for an > appropriate algorithm. > > Hope that helps a little. > > Sebastian > > > > > > On Mon, Aug 2, 2010 at 10:42 PM, Robert Streubel > wrote: >> Hello there, >> >> I'd like to minimize a scalar function F(a(x,y),b(x,y)) - with other words a functional - under certain boundary conditions, such as a+b<1. >> Since Scipy supports minimization, I wonder if there could be a rather convenient way to imply boundary conditions. Because this condition causes a different minimum, it can not be considered recursively by a second fmin routine. >> Hence, could anyway assist me, please? >> >> Cheers >> Robert >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > From gammelmark at phys.au.dk Thu Aug 5 04:21:28 2010 From: gammelmark at phys.au.dk (=?UTF-8?B?U8O4cmVuIEdhbW1lbG1hcms=?=) Date: Thu, 05 Aug 2010 10:21:28 +0200 Subject: [SciPy-User] Compiling SciPy almost works In-Reply-To: <4C5A695F.7090104@silveregg.co.jp> References: <4C59554C.3060704@phys.au.dk> <4C5A59C7.5070204@phys.au.dk> <4C5A6425.5040604@silveregg.co.jp> <4C5A6827.5090803@phys.au.dk> <4C5A695F.7090104@silveregg.co.jp> Message-ID: <4C5A7488.6050104@phys.au.dk> You mean setting the environment-variable CXX to icpc? That does not work (I tried it), but I would not expect it to either -- I did not think that the numpy distutils listen to those environment-variables? When I compiled numpy/scipy I had to do a bit of editing in numpy/distutils/intelcompilers.py, where I changed the line cc_exe = 'icc' to include more command-line arguments, i.e. cc_exe = 'icc -fPIC -O2 -fomit-frame-pointer -parallel'. One possibility would be to put in icpc here instead, but then everything will be compiled with icpc. I tried this, but then the numpy-compilation sprouts errors about missing libraries (I did not try thi SciPy-compilation). I then tried adding a cxx_exe = 'icpc .....' line and modifying to cxx_compiler option i intelcompilers.py. This appears to work (and make excellent sense). I can then compiler NumPy without problems and SciPy compiles fine and sparsetools work as they should. Now, however, the scipy.test() crashes with "*** glibc detected *** free(): invalid next size (fast): 0x0000000001f6f2b0 ***" when running test_arpack.TestEigenNonSymmetric. This is not a big program for me and seems to be related to the issue stated here http://projects.scipy.org/scipy/ticket/848, but it still seems a bit strange. If I remove the ARPACK tests things work for the most part. There is still some issues regarding the interpretation of NaN and Inf, but I believe that is quite normal. It would be nice if we could get better support for the Intel-compilers since I believe they are used by many scientific institutions. For future reference: A peculiar thing I noticed when compiling SciPy was that the Intel Fortran compiler setup in numpy.distutils appended the option "-arch SSE2" on the AMD Opteron machines. This had the peculiar effect of an error complaining about the command-line " 'ifort .... -arch SSE2 ??[]^_?=0??@ .....'. I guess it is a bug in the Intel Fortran 9.1 compiler (just try ifort -arch SSE2 on an AMD Opteron) and in addition the compiler claims that neither -arch SSE2 nor -msse2 is supported on that platform. So if people are trying to compile SciPy with this compiler they need to change this. Thanks for all the help S?ren On 2010-08-05 09:33, David wrote: > On 08/05/2010 04:28 PM, S?ren Gammelmark wrote: > >> When trying the manual link everything works out fine. "import _csr" >> executes without any problem. I also tried using the Intel C++ compiler >> (icpc) and then everything works out fine. I guess the default >> link-libraries are different for the programs. >> > Hm, so they change the C++ driver - that's almost certainly the root of > the issue. Does setting CXX to icpc works ? > > David > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From david at silveregg.co.jp Thu Aug 5 05:16:53 2010 From: david at silveregg.co.jp (David) Date: Thu, 05 Aug 2010 18:16:53 +0900 Subject: [SciPy-User] Compiling SciPy almost works In-Reply-To: <4C5A7488.6050104@phys.au.dk> References: <4C59554C.3060704@phys.au.dk> <4C5A59C7.5070204@phys.au.dk> <4C5A6425.5040604@silveregg.co.jp> <4C5A6827.5090803@phys.au.dk> <4C5A695F.7090104@silveregg.co.jp> <4C5A7488.6050104@phys.au.dk> Message-ID: <4C5A8185.4030708@silveregg.co.jp> On 08/05/2010 05:21 PM, S?ren Gammelmark wrote: > You mean setting the environment-variable CXX to icpc? That does not > work (I tried it), but I would not expect it to either -- I did not > think that the numpy distutils listen to those environment-variables? > > When I compiled numpy/scipy I had to do a bit of editing in > numpy/distutils/intelcompilers.py, where I changed the line cc_exe = > 'icc' to include more command-line arguments, i.e. cc_exe = 'icc -fPIC > -O2 -fomit-frame-pointer -parallel'. One possibility would be to put in > icpc here instead, but then everything will be compiled with icpc. I > tried this, but then the numpy-compilation sprouts errors about missing > libraries (I did not try thi SciPy-compilation). > > I then tried adding a cxx_exe = 'icpc .....' line and modifying to > cxx_compiler option i intelcompilers.py. This appears to work (and make > excellent sense). I can then compiler NumPy without problems and SciPy > compiles fine and sparsetools work as they should. Now, however, the > scipy.test() crashes with "*** glibc detected *** free(): invalid next > size (fast): 0x0000000001f6f2b0 ***" when running > test_arpack.TestEigenNonSymmetric. This may indeed be unrelated. ARPACK (or at least our wrapper) has several obscure bugs which are unfortunately hard to reproduce. > It would be nice if we could get better support for the Intel-compilers > since I believe they are used by many scientific institutions. The problem is those are not free, and that our build infrastructure sucks. So we mostly rely on volunteers to make this work, but I don't think people can expect to get the same support for Intel compilers and Gnu compilers on linux. cheers, David From matthieu.brucher at gmail.com Thu Aug 5 05:28:18 2010 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 5 Aug 2010 11:28:18 +0200 Subject: [SciPy-User] Compiling SciPy almost works In-Reply-To: <4C5A8185.4030708@silveregg.co.jp> References: <4C59554C.3060704@phys.au.dk> <4C5A59C7.5070204@phys.au.dk> <4C5A6425.5040604@silveregg.co.jp> <4C5A6827.5090803@phys.au.dk> <4C5A695F.7090104@silveregg.co.jp> <4C5A7488.6050104@phys.au.dk> <4C5A8185.4030708@silveregg.co.jp> Message-ID: > The problem is those are not free, and that our build infrastructure > sucks. So we mostly rely on volunteers to make this work, but I don't > think people can expect to get the same support for Intel compilers and > Gnu compilers on linux. I tested with numpy 1.4.1rc2 and scipy 0.7.2rc3 (I know, they are pretty old now) with Intel compilers and the associated MKL (11.1) for Python 2.6, I never had this kind of issue. Of course, my own Python version is compiled with Intel compilers. Matthieu -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher From alan.isaac at gmail.com Thu Aug 5 07:38:37 2010 From: alan.isaac at gmail.com (Alan G Isaac) Date: Thu, 05 Aug 2010 07:38:37 -0400 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: Message-ID: <4C5AA2BD.2020306@gmail.com> For finding random points in an arbitrary simple closed figure, you could draw it in say PIL, fill it with a known color, convert to array, random sample the array, and keep the locations of the colored points. fwiw, Alan Isaac From denis-bz-gg at t-online.de Thu Aug 5 09:11:40 2010 From: denis-bz-gg at t-online.de (denis) Date: Thu, 5 Aug 2010 06:11:40 -0700 (PDT) Subject: [SciPy-User] Interpolation in 3D with interp2d In-Reply-To: References: <1456192421.951467.1280690737282.JavaMail.fmail@mwmweb001> Message-ID: <59fe2dfc-f236-4c43-a44f-331abf7f2453@i28g2000yqa.googlegroups.com> Jana, Andreas, Take a look at http://scipy.org/Cookbook/Matplotlib/Gridding_irregularly_spaced_data and http://en.wikipedia.org/wiki/Natural_neighbor for a picture. To use radial basis functions on N given data points, one first solves an N x N linear system, so N can't be more than 1000 or so; then each interpolated point takes time ~ N. Also there are arbitrary choices of basis family and r0 -- your mileage will vary. As an alternative to RBFs I'd suggest the combination of scipy KDTree and inverse-distance weighting given under http://stackoverflow.com/questions/3104781/inverse-distance-weighted-idw-interpolation-with-python . KDTree finds the nearest say 8 neighbours to a point to interpolate, then you take a weighted average of the 8 values A(f,z). This works for data in 2d, 3d, 10d too. But for your 2d data, Gridding_irregularly_spaced_data is faster and simpler. cheers -- denis From ben.root at ou.edu Thu Aug 5 11:06:00 2010 From: ben.root at ou.edu (Benjamin Root) Date: Thu, 5 Aug 2010 10:06:00 -0500 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: Message-ID: On Thu, Aug 5, 2010 at 1:54 AM, David Goldsmith wrote: > Whoops, sorry, in the example, the call to rand_in_ellipse should have 4./3 > as a single argument, i.e., the line should look like > > sample = np.array([rand_in_ellipse(4./3) for i in np.arange(10000)]) > > > DG > > On Wed, Aug 4, 2010 at 11:50 PM, David Goldsmith wrote: > >> For the ellipse: >> >> import numpy as np >> from numpy.random import random_sample as random >> >> pi = np.pi >> def rand_in_ellipse(a, b=1, offset=0): >> angle = 2*pi*random() - offset >> x = a * random() * np.cos(angle) >> y = b * random() * np.sin(angle) >> return np.array((x,y)) >> >> import matplotlib.pyplot as plt >> >> fig = plt.figure() >> >> >> ax = fig.add_subplot(111) >> >> sample = np.array([rand_in_ellipse() for i in np.arange(10000)]) >> ax.scatter(sample[:,0], sample[:,1]) >> >> >> plt.show() # Figure attached >> >> DG >> >> Unfortunately, that does not produce a uniform sample. It is heavily biased along the major and minor axes. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From denis-bz-gg at t-online.de Thu Aug 5 11:28:29 2010 From: denis-bz-gg at t-online.de (denis) Date: Thu, 5 Aug 2010 08:28:29 -0700 (PDT) Subject: [SciPy-User] Interpolation in 3D with interp2d In-Reply-To: <1456192421.951467.1280690737282.JavaMail.fmail@mwmweb001> References: <1456192421.951467.1280690737282.JavaMail.fmail@mwmweb001> Message-ID: <7401bcc8-05e4-43fa-8f0b-0d094a5d51ee@t20g2000yqa.googlegroups.com> Jana, Andreas, looking a little less hastily, Jana's mail of 1 August imports griddata but doesn't call it ? Do I understand that you want to 1) first griddata to get a A(f,z) on a uniform grid 2) then do bilinear interpolation from that ? If so, why not just griddata() to your fine final grid in one shot ? A gotcha, griddata() can return masked arrays which may mess up downstream interpolation: print ma.count_masked( griddataresult ) (experts correct me / suggest what do do -- separate thread.) (To do bilinear interpolation from one regular grid to another, from scipy.interpolate import RectBivariateSpline x, y = linspace(...), linspace(...) interpolator = RectBivariateSpline( x,y,z, kx=1, ky=1, s=0 ) # s=0 interpolates xi, yi = linspace(...), linspace(...) zi = interpolator( xi, yi ) cheers -- denis From erin.sheldon at gmail.com Thu Aug 5 12:32:20 2010 From: erin.sheldon at gmail.com (Erin Sheldon) Date: Thu, 05 Aug 2010 12:32:20 -0400 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: Message-ID: <1281024260-sup-9515@theshire> Excerpts from Benjamin Root's message of Wed Aug 04 19:38:51 -0400 2010: > Hi, > > For a project, I need to create sets of random coordinates within a 2d > domain. To start, I have been creating random x and y coordinates, which > has worked very nicely. However, I would like to start doing some fancier > domains like ellipses and crescents. Does anybody know of any useful tricks > for doing this? > > Thanks, > Ben Root Hi Ben - A simple way is to generate random x and y coordinates in a bounding box and then trim back to the shape you want using your constraint. E.g. for an ellipse: # equation of ellipse: # (x/x0)^2 + (y/y0)^2 = 1 # or x=2*x0*(random.random(n-nkeep) - 0.5) y=2*y0*(random.random(n-nkeep) - 0.5) # get points that fall within constraint w,=where( ( (x/x0)**2 + (y/y0)**2 ) < 1 ) You can put this in a while loop to get the desired number: import numpy from numpy import random, where, cos, sin, arctan2, abs def get_random_ellipse(n, x0, y0): xout = numpy.zeros(n) yout = numpy.zeros(n) nkeep=0 while nkeep < n: x=2*x0*(random.random(n-nkeep) - 0.5) y=2*y0*(random.random(n-nkeep) - 0.5) w,=where( ( (x/x0)**2 + (y/y0)**2 ) < 1 ) if w.size > 0: xout[nkeep:nkeep+w.size] = x[w] yout[nkeep:nkeep+w.size] = y[w] nkeep += w.size return xout,yout From kumk.pdf at gmail.com Thu Aug 5 13:35:10 2010 From: kumk.pdf at gmail.com (Mahendra Kumar) Date: Thu, 5 Aug 2010 23:05:10 +0530 Subject: [SciPy-User] import error scipy.io.array_import Message-ID: Dear Scipy Users i am facing problem with importing io module in scipy and python shell shows an error as following. import scipy.io.array_import ImportError: No module named array_import Please let me know how to solve this. I am using WinXp -OS and python version -2.5 and rest are also for the same version. Mahendra -------------- next part -------------- An HTML attachment was scrubbed... URL: From iannimmosmith at gmail.com Thu Aug 5 13:53:24 2010 From: iannimmosmith at gmail.com (Ian Nimmo-Smith) Date: Thu, 5 Aug 2010 18:53:24 +0100 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: Message-ID: Ben You could try the following to generate a uniform distribution in the interior of an ellipse with semi-axes of length (a,b). It scales the x-coordinate of a random point (x,y) in the circular disc of radius 1 by a. This will have the correct marginal density of the x-coordinate for the desired ellipse. Conditional on x, the desired y distribution is uniform on the chord of length 2*b*sqrt(1-(x/a)**2). (It's a bit wasteful in that it uses 3 calls to random() but discards the y-coordinate of the random point in the disc. I don't think you can utilise that to generate a second point in the ellipse because y and x are not independent.) Ian --- import sympy import numpy as np import scipy as sc from numpy.random import random_sample as random def random_uniform_in_disc(): # returns a tuple which is uniform in the disc theta = 2*np.pi*random() r2 = random() r = np.sqrt(r2) return np.array((r*np.sin(theta),r*np.cos(theta))) def random_uniform_in_ellipse(a=1,b=1): x = a*random_uniform_in_disc()[0] y = b*np.sqrt(1-(x/a)**2)*(1-2*random()) return np.array((x,y)) import matplotlib.pyplot as plt fig = plt.figure() ax = fig.add_subplot(111) sample = np.array([random_uniform_in_ellipse(a=2,b=1) for i in np.arange(10000)]) ax.scatter(*sample.T) plt.show() On 5 August 2010 16:06, Benjamin Root wrote: > On Thu, Aug 5, 2010 at 1:54 AM, David Goldsmith wrote: > >> Whoops, sorry, in the example, the call to rand_in_ellipse should have >> 4./3 as a single argument, i.e., the line should look like >> >> sample = np.array([rand_in_ellipse(4./3) for i in np.arange(10000)]) >> >> >> >> DG >> >> On Wed, Aug 4, 2010 at 11:50 PM, David Goldsmith > > wrote: >> >>> For the ellipse: >>> >>> import numpy as np >>> from numpy.random import random_sample as random >>> >>> pi = np.pi >>> def rand_in_ellipse(a, b=1, offset=0): >>> angle = 2*pi*random() - offset >>> x = a * random() * np.cos(angle) >>> y = b * random() * np.sin(angle) >>> return np.array((x,y)) >>> >>> import matplotlib.pyplot as plt >>> >>> fig = plt.figure() >>> >>> >>> >>> ax = fig.add_subplot(111) >>> >>> sample = np.array([rand_in_ellipse() for i in np.arange(10000)]) >>> ax.scatter(sample[:,0], sample[:,1]) >>> >>> >>> >>> plt.show() # Figure attached >>> >>> DG >>> >>> > Unfortunately, that does not produce a uniform sample. It is heavily > biased along the major and minor axes. > > Ben Root > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Thu Aug 5 13:59:21 2010 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 5 Aug 2010 17:59:21 +0000 (UTC) Subject: [SciPy-User] import error scipy.io.array_import References: Message-ID: Thu, 05 Aug 2010 23:05:10 +0530, Mahendra Kumar wrote: > I am facing problem with importing io module in scipy > and python shell shows an error as following. > > import scipy.io.array_import > > ImportError: No module named array_import array_import was removed in Scipy 0.8. If you are looking for read_array and write_array, you should use 'numpy.savetxt' and 'numpy.loadtxt' instead. If this is some other error, please include in your mail (i) the code that you typed in, (ii) the full error message (starting with Traceback: ...) -- Pauli Virtanen From vanforeest at gmail.com Thu Aug 5 17:16:28 2010 From: vanforeest at gmail.com (nicky van foreest) Date: Thu, 5 Aug 2010 23:16:28 +0200 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: Message-ID: Hi, I actually wonder whether this is the type of distribution you want to see. Lets apply the method described below to a circle (that is just a special type of ellipse) with unit radius and center at (0,0). With p =1/2 you will hit the segments of the circle at the left of the line such that x=-1/2 or at the right of the line with x =1/2. However the segments above the line with y=1/2 or below the line with y = -1/2 will be hit with a smaller probability. Thus, the distribution of points on the circle produced by the method below is certainly not rotationally invariant. I recall having read three ways to produce a random point in a circle, but I forgot where (perhaps it can be found in Feller, or the book by Jaynes), and each of the three methods gives different distributions with different properties. Nicky On 5 August 2010 19:53, Ian Nimmo-Smith wrote: > Ben > > You could try the following to generate a uniform distribution in the > interior of an ellipse with semi-axes of length (a,b). It scales the > x-coordinate of a random point (x,y) in the circular disc of radius 1 by a. > This will have the correct marginal density of the x-coordinate for the > desired ellipse. Conditional on x, the desired y distribution is uniform on > the chord of length 2*b*sqrt(1-(x/a)**2).? (It's a bit wasteful in that it > uses 3 calls to random() but discards the y-coordinate of the random point > in the disc. I don't think you can utilise that to generate a second point > in the ellipse because y and x are not independent.) > > Ian > --- > > import sympy > import numpy as np > import scipy as sc > from numpy.random import random_sample as random > > def random_uniform_in_disc(): > ??? # returns a tuple which is uniform in the disc > ??? theta = 2*np.pi*random() > ??? r2 = random() > ??? r = np.sqrt(r2) > ??? return np.array((r*np.sin(theta),r*np.cos(theta))) > > def random_uniform_in_ellipse(a=1,b=1): > ??? x = a*random_uniform_in_disc()[0] > ??? y = b*np.sqrt(1-(x/a)**2)*(1-2*random()) > ??? return np.array((x,y)) > > import matplotlib.pyplot as plt > fig = plt.figure() > ax = fig.add_subplot(111) > sample = np.array([random_uniform_in_ellipse(a=2,b=1) for i in > np.arange(10000)]) > ax.scatter(*sample.T) > plt.show() > > > On 5 August 2010 16:06, Benjamin Root wrote: >> >> On Thu, Aug 5, 2010 at 1:54 AM, David Goldsmith >> wrote: >>> >>> Whoops, sorry, in the example, the call to rand_in_ellipse should have >>> 4./3 as a single argument, i.e., the line should look like >>> >>> sample = np.array([rand_in_ellipse(4./3) for i in np.arange(10000)]) >>> >>> >>> >>> >>> >>> DG >>> >>> On Wed, Aug 4, 2010 at 11:50 PM, David Goldsmith >>> wrote: >>>> >>>> For the ellipse: >>>> >>>> import numpy as np >>>> from numpy.random import random_sample as random >>>> >>>> pi = np.pi >>>> def rand_in_ellipse(a, b=1, offset=0): >>>> ??? angle = 2*pi*random() - offset >>>> ??? x = a * random() * np.cos(angle) >>>> ??? y = b * random() * np.sin(angle) >>>> ??? return np.array((x,y)) >>>> >>>> import matplotlib.pyplot as plt >>>> >>>> fig = plt.figure() >>>> >>>> >>>> >>>> >>>> >>>> ax = fig.add_subplot(111) >>>> >>>> >>>> sample = np.array([rand_in_ellipse() for i in np.arange(10000)]) >>>> ax.scatter(sample[:,0], sample[:,1]) >>>> >>>> >>>> >>>> >>>> >>>> plt.show() # Figure attached >>>> >>>> DG >> >> Unfortunately, that does not produce a uniform sample. ? It is heavily >> biased along the major and minor axes. >> >> Ben Root >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From d.l.goldsmith at gmail.com Thu Aug 5 21:23:25 2010 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Thu, 5 Aug 2010 18:23:25 -0700 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: Message-ID: Ben: Nicky brings up a good point: you need to be clear on precisely how you want the points to be distributed; I was thinking more about my final "working" example in which the area density appears to be inversely related to r (which I attributed to the size of the points being more significant with decreasing r) and I think (I'm too lazy/have too many other priorities to prove) that the true explanation is that my distribution is uniform with respect to r (and axially symmetric, which you almost certainly want), which makes the expected number of points per unit area decrease as 1/r (again, this is a conjecture, I'm not claiming to have proven it). If you want a distribution in which the expected number of points per unit area is independent of r (and theta), i.e., a "unit-area-uniform" distribution, I think either Alan's or Erin's methods are what you want. DG On Thu, Aug 5, 2010 at 2:16 PM, nicky van foreest wrote: > Hi, > > I actually wonder whether this is the type of distribution you want to > see. Lets apply the method described below to a circle (that is just a > special type of ellipse) with unit radius and center at (0,0). With p > =1/2 you will hit the segments of the circle at the left of the line > such that x=-1/2 or at the right of the line with x =1/2. However the > segments above the line with y=1/2 or below the line with y = -1/2 > will be hit with a smaller probability. Thus, the distribution of > points on the circle produced by the method below is certainly not > rotationally invariant. > > I recall having read three ways to produce a random point in a circle, > but I forgot where (perhaps it can be found in Feller, or the book by > Jaynes), and each of the three methods gives different distributions > with different properties. > > Nicky > > On 5 August 2010 19:53, Ian Nimmo-Smith wrote: > > Ben > > > > You could try the following to generate a uniform distribution in the > > interior of an ellipse with semi-axes of length (a,b). It scales the > > x-coordinate of a random point (x,y) in the circular disc of radius 1 by > a. > > This will have the correct marginal density of the x-coordinate for the > > desired ellipse. Conditional on x, the desired y distribution is uniform > on > > the chord of length 2*b*sqrt(1-(x/a)**2). (It's a bit wasteful in that > it > > uses 3 calls to random() but discards the y-coordinate of the random > point > > in the disc. I don't think you can utilise that to generate a second > point > > in the ellipse because y and x are not independent.) > > > > Ian > > --- > > > > import sympy > > import numpy as np > > import scipy as sc > > from numpy.random import random_sample as random > > > > def random_uniform_in_disc(): > > # returns a tuple which is uniform in the disc > > theta = 2*np.pi*random() > > r2 = random() > > r = np.sqrt(r2) > > return np.array((r*np.sin(theta),r*np.cos(theta))) > > > > def random_uniform_in_ellipse(a=1,b=1): > > x = a*random_uniform_in_disc()[0] > > y = b*np.sqrt(1-(x/a)**2)*(1-2*random()) > > return np.array((x,y)) > > > > import matplotlib.pyplot as plt > > fig = plt.figure() > > ax = fig.add_subplot(111) > > sample = np.array([random_uniform_in_ellipse(a=2,b=1) for i in > > np.arange(10000)]) > > ax.scatter(*sample.T) > > plt.show() > > > > > > On 5 August 2010 16:06, Benjamin Root wrote: > >> > >> On Thu, Aug 5, 2010 at 1:54 AM, David Goldsmith < > d.l.goldsmith at gmail.com> > >> wrote: > >>> > >>> Whoops, sorry, in the example, the call to rand_in_ellipse should have > >>> 4./3 as a single argument, i.e., the line should look like > >>> > >>> sample = np.array([rand_in_ellipse(4./3) for i in np.arange(10000)]) > >>> > >>> > >>> > >>> > >>> > >>> DG > >>> > >>> On Wed, Aug 4, 2010 at 11:50 PM, David Goldsmith > >>> wrote: > >>>> > >>>> For the ellipse: > >>>> > >>>> import numpy as np > >>>> from numpy.random import random_sample as random > >>>> > >>>> pi = np.pi > >>>> def rand_in_ellipse(a, b=1, offset=0): > >>>> angle = 2*pi*random() - offset > >>>> x = a * random() * np.cos(angle) > >>>> y = b * random() * np.sin(angle) > >>>> return np.array((x,y)) > >>>> > >>>> import matplotlib.pyplot as plt > >>>> > >>>> fig = plt.figure() > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> ax = fig.add_subplot(111) > >>>> > >>>> > >>>> sample = np.array([rand_in_ellipse() for i in np.arange(10000)]) > >>>> ax.scatter(sample[:,0], sample[:,1]) > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> plt.show() # Figure attached > >>>> > >>>> DG > >> > >> Unfortunately, that does not produce a uniform sample. It is heavily > >> biased along the major and minor axes. > >> > >> Ben Root > >> > >> _______________________________________________ > >> SciPy-User mailing list > >> SciPy-User at scipy.org > >> http://mail.scipy.org/mailman/listinfo/scipy-user > >> > > > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Mathematician: noun, someone who disavows certainty when their uncertainty set is non-empty, even if that set has measure zero. Hope: noun, that delusive spirit which escaped Pandora's jar and, with her lies, prevents mankind from committing a general suicide. (As interpreted by Robert Graves) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Thu Aug 5 21:45:06 2010 From: ben.root at ou.edu (Benjamin Root) Date: Thu, 5 Aug 2010 20:45:06 -0500 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: Message-ID: On Thu, Aug 5, 2010 at 12:53 PM, Ian Nimmo-Smith wrote: > Ben > > You could try the following to generate a uniform distribution in the > interior of an ellipse with semi-axes of length (a,b). It scales the > x-coordinate of a random point (x,y) in the circular disc of radius 1 by a. > This will have the correct marginal density of the x-coordinate for the > desired ellipse. Conditional on x, the desired y distribution is uniform on > the chord of length 2*b*sqrt(1-(x/a)**2). (It's a bit wasteful in that it > uses 3 calls to random() but discards the y-coordinate of the random point > in the disc. I don't think you can utilise that to generate a second point > in the ellipse because y and x are not independent.) > > Ian > --- > > import sympy > import numpy as np > import scipy as sc > > from numpy.random import random_sample as random > > def random_uniform_in_disc(): > # returns a tuple which is uniform in the disc > theta = 2*np.pi*random() > r2 = random() > r = np.sqrt(r2) > return np.array((r*np.sin(theta),r*np.cos(theta))) > > def random_uniform_in_ellipse(a=1,b=1): > x = a*random_uniform_in_disc()[0] > y = b*np.sqrt(1-(x/a)**2)*(1-2*random()) > > return np.array((x,y)) > > import matplotlib.pyplot as plt > fig = plt.figure() > ax = fig.add_subplot(111) > sample = np.array([random_uniform_in_ellipse(a=2,b=1) for i in > np.arange(10000)]) > ax.scatter(*sample.T) > plt.show() > > Ian, This seems to work very nicely. And I am not too worried about the extra call to random. I am only looking to create about 50 or some points at any given run (I am simulating storm tracks and want to be able to define a region of storm initiation). Thank you, Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Thu Aug 5 21:58:29 2010 From: ben.root at ou.edu (Benjamin Root) Date: Thu, 5 Aug 2010 20:58:29 -0500 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: Message-ID: On Thu, Aug 5, 2010 at 4:16 PM, nicky van foreest wrote: > Hi, > > I actually wonder whether this is the type of distribution you want to > see. Lets apply the method described below to a circle (that is just a > special type of ellipse) with unit radius and center at (0,0). With p > =1/2 you will hit the segments of the circle at the left of the line > such that x=-1/2 or at the right of the line with x =1/2. However the > segments above the line with y=1/2 or below the line with y = -1/2 > will be hit with a smaller probability. Thus, the distribution of > points on the circle produced by the method below is certainly not > rotationally invariant. > > I recall having read three ways to produce a random point in a circle, > but I forgot where (perhaps it can be found in Feller, or the book by > Jaynes), and each of the three methods gives different distributions > with different properties. > > Nicky > > Running the code for just a regular circle (and shrinking the size of the markers), I can not see any visual evidence of biases within the ellipse. Maybe it, mathematically, has a slight bias one way or another, but I think Ian's approach to be sufficient for my purposes. Thanks, Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Thu Aug 5 22:11:00 2010 From: ben.root at ou.edu (Benjamin Root) Date: Thu, 5 Aug 2010 21:11:00 -0500 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: Message-ID: On Thu, Aug 5, 2010 at 8:23 PM, David Goldsmith wrote: > Ben: > > Nicky brings up a good point: you need to be clear on precisely how you > want the points to be distributed; I was thinking more about my final > "working" example in which the area density appears to be inversely related > to r (which I attributed to the size of the points being more significant > with decreasing r) and I think (I'm too lazy/have too many other priorities > to prove) that the true explanation is that my distribution is uniform with > respect to r (and axially symmetric, which you almost certainly want), which > makes the expected number of points per unit area decrease as 1/r (again, > this is a conjecture, I'm not claiming to have proven it). If you want a > distribution in which the expected number of points per unit area is > independent of r (and theta), i.e., a "unit-area-uniform" distribution, I > think either Alan's or Erin's methods are what you want. > > DG > > Yeah, I am looking more for a unit-area-uniform distribution, and various references have pointed out that simply using random r and theta and plugging them in will produce unexpected biases. I have been hesitant about the rejection methods because I thought there might be a better, more direct method. However, it does look like I might have to go towards Alan's or Erin's methods as I edge towards a more general case of allowing arbitrary regions for random point generation. Thanks for all your help! Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcmcclur at unca.edu Fri Aug 6 00:48:22 2010 From: mcmcclur at unca.edu (Mark McClure) Date: Fri, 6 Aug 2010 00:48:22 -0400 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: Message-ID: On Wed, Aug 4, 2010 at 8:18 PM, alex wrote: > On Wed, Aug 4, 2010 at 7:38 PM, Benjamin Root wrote: > >> For a project, I need to create sets of random coordinates within a 2d >> domain.? To start, I have been creating random x and y coordinates, which >> has worked very nicely.? However, I would like to start doing some fancier >> domains like ellipses and crescents. > > For an ellipse you could start with random points in a disk like this > http://mathworld.wolfram.com/DiskPointPicking.html > and then I think you can just stretch along an axis to give the points > inside the ellipse disk. This is definitely the way to go for an ellipse. Essentially, this works because the determinant of the Jacobian of the underlying transformation is constant. Given a parametrized domain with non-constant Jacobian determinant, it may be possible to re-scale the parametrization to one with constant Jacobian determinant. Finding such a rescaling essentially boils down to solving differential equations. If you are interested, I set up a Sage notebook that illustrates a few such examples: http://sagenb.org/home/pub/2336/ Mark McClure From ben.root at ou.edu Fri Aug 6 10:39:43 2010 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 6 Aug 2010 09:39:43 -0500 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: Message-ID: On Thu, Aug 5, 2010 at 11:48 PM, Mark McClure wrote: > On Wed, Aug 4, 2010 at 8:18 PM, alex wrote: > > On Wed, Aug 4, 2010 at 7:38 PM, Benjamin Root wrote: > > > >> For a project, I need to create sets of random coordinates within a 2d > >> domain. To start, I have been creating random x and y coordinates, > which > >> has worked very nicely. However, I would like to start doing some > fancier > >> domains like ellipses and crescents. > > > > For an ellipse you could start with random points in a disk like this > > http://mathworld.wolfram.com/DiskPointPicking.html > > and then I think you can just stretch along an axis to give the points > > inside the ellipse disk. > > This is definitely the way to go for an ellipse. Essentially, this > works because the determinant of the Jacobian of the underlying > transformation is constant. Given a parametrized domain with > non-constant Jacobian determinant, it may be possible to re-scale the > parametrization to one with constant Jacobian determinant. Finding > such a rescaling essentially boils down to solving differential > equations. If you are interested, I set up a Sage notebook that > illustrates a few such examples: > http://sagenb.org/home/pub/2336/ > > Mark McClure > Oh, wow. I think this is the best explanation yet about how to determine if the point picking is uniform or not. I have been doing a rescaling of the unit circle to get my ellipse, and the results looked good, but it was against my intuition that the points would still be uniform. Thanks! Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From denis-bz-gg at t-online.de Fri Aug 6 12:07:52 2010 From: denis-bz-gg at t-online.de (denis) Date: Fri, 6 Aug 2010 09:07:52 -0700 (PDT) Subject: [SciPy-User] masked griddata() from Cookbook example Message-ID: While the doc for matplotlib.mlab.griddata clearly says A masked array is returned if any grid points are outside convex hull defined by input data (no extrapolation is done). it may not be obvious to someone running the example in http://www.scipy.org/Cookbook/Matplotlib/Gridding_irregularly_spaced_data how easily "outside convex hull" can happen. I'd suggest growing the example a bit along the lines 
nrand = 10
ngrid = 21
hull = 0
exec "\n".join( sys.argv[1:] )  # run this.py nrand= ...
...
x = np.random.uniform(-2, 2, nrand)
y = np.random.uniform(-2, 2, nrand)
if hull:
    x = np.r_[ -2, 2, 2, -2, x]  # add 4 corners to griddata all, not
just the convex hull
    y = np.r_[ -2, -2, 2, 2, y]
z = f(x,y)
xi = np.linspace(-2,2,ngrid)
yi = np.linspace(-2,2,ngrid)

#...............................................................................
zi = griddata(x,y,z,xi,yi)  # 1d x y z -> 2d zi on meshgrid(xi,yi)

nmask = np.ma.count_masked(zi)
if nmask > 0:
    print >>sys.stderr, "warning: griddata: %d masked points in
output" % nmask
    np.savetxt( sys.stderr, zi.mask.astype(int), "%d", "" )
...
Would anyone care to talk this over, before editing the Cookbook example ? (Actually I think griddata() should add the 4 corners by itself, but.) cheers -- denis From dwf at cs.toronto.edu Fri Aug 6 12:42:59 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Fri, 6 Aug 2010 12:42:59 -0400 Subject: [SciPy-User] Compiling SciPy almost works In-Reply-To: References: <4C59554C.3060704@phys.au.dk> <4C5A59C7.5070204@phys.au.dk> <4C5A6425.5040604@silveregg.co.jp> <4C5A6827.5090803@phys.au.dk> <4C5A695F.7090104@silveregg.co.jp> <4C5A7488.6050104@phys.au.dk> <4C5A8185.4030708@silveregg.co.jp> Message-ID: <5A074B07-4E1E-43CF-9A9C-509F1E5842DA@cs.toronto.edu> On 2010-08-05, at 5:28 AM, Matthieu Brucher wrote: >> The problem is those are not free, and that our build infrastructure >> sucks. So we mostly rely on volunteers to make this work, but I don't >> think people can expect to get the same support for Intel compilers and >> Gnu compilers on linux. > > I tested with numpy 1.4.1rc2 and scipy 0.7.2rc3 (I know, they are > pretty old now) with Intel compilers and the associated MKL (11.1) for > Python 2.6, I never had this kind of issue. > Of course, my own Python version is compiled with Intel compilers. I just tried compiling 0.8.0 and I get the same issue, even with Intel-compiled Python. I was using NumPy 1.5.0b1. Also, this is off topic but Matthieu -- when building Python with the Intel compilers did you get ctypes to build properly with icc? It fails for me. David From dwf at cs.toronto.edu Fri Aug 6 12:51:45 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Fri, 6 Aug 2010 12:51:45 -0400 Subject: [SciPy-User] Compiling SciPy almost works In-Reply-To: <4C5A6425.5040604@silveregg.co.jp> References: <4C59554C.3060704@phys.au.dk> <4C5A59C7.5070204@phys.au.dk> <4C5A6425.5040604@silveregg.co.jp> Message-ID: <64A22683-5E7D-4DC0-AFDA-1E6C919B3CBB@cs.toronto.edu> On 2010-08-05, at 3:11 AM, David wrote: > On 08/05/2010 03:27 PM, S?ren Gammelmark wrote: >> I have attached the build-log. As far as I can see it chooses the Intel >> compiler with the correct options as it should. >> >> Also recompiling after removing build did not change anything (I do not >> believe it should have, since I was quite careful of doing this with >> every attempt). > > Could you try linking the extension "manually" and add -lstdc++, i.e.. > > icc -fPIC -fomit-frame-pointer -O2 -g -parallel -shared > build/temp.linux-x86_64-2.6/scipy/sparse/sparsetools/csr_wrap.o > -Lbuild/temp.linux-x86_64-2.6 -o > build/lib.linux-x86_64-2.6/scipy/sparse/sparsetools/_csr.so -lstdc++ Hey David, The command I'm using is similar but the build actually fails for me with 11.1... It seems it does not like these sprintf statements: gpc-f101n084-$ icc -xHost -O3 -fPIC -openmp -I/home/dwf/pkg/lib/python2.7/site-packages/numpy/core/include -I/home/dwf/pkg/include/python2.7 -c scipy/sparse/sparsetools/csr_wrap.cxx -o build/temp.linux-x86_64-2.7/scipy/sparse/sparsetools/csr_wrap.o scipy/sparse/sparsetools/csr_wrap.cxx(2910): error: expected a ")" sprintf(s,"%" NPY_INTP_FMT ",", size[i]); ^ scipy/sparse/sparsetools/csr_wrap.cxx(2917): error: expected a ")" sprintf(s,"%" NPY_INTP_FMT ",", array_size(ary,i)); ^ compilation aborted for scipy/sparse/sparsetools/csr_wrap.cxx (code 2) gpc-f101n084-$ icc --version icc (ICC) 11.1 20100414 Copyright (C) 1985-2010 Intel Corporation. All rights reserved. Any thoughts? David From cimrman3 at ntc.zcu.cz Fri Aug 6 12:52:31 2010 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Fri, 06 Aug 2010 18:52:31 +0200 Subject: [SciPy-User] ANN: SfePy 2010.3 Message-ID: <4C5C3DCF.5010100@ntc.zcu.cz> I am pleased to announce release 2010.3 of SfePy. Description ----------- SfePy (simple finite elements in Python) is a software for solving systems of coupled partial differential equations by the finite element method. The code is based on NumPy and SciPy packages. It is distributed under the new BSD license. Development, mailing lists, issue tracking: http://sfepy.org Documentation: http://docs.sfepy.org/doc Git repository: http://github.com/sfepy Project page: http://sfepy.kme.zcu.cz Highlights of this release -------------------------- - significantly rewritten code for better interactive use - cleaner and simpler high level interface - new tutorial section: - Interactive Example: Linear Elasticity [1] [1] http://docs.sfepy.org/doc/tutorial.html#interactive-example-linear-elasticity Major improvements ------------------ Apart from many bug-fixes, let us mention: - new examples: - demonstration of the high level interface - new tests: - tests of the new high level interface - simplified but more powerful homogenization engine For more information on this release, see http://sfepy.googlecode.com/svn/web/releases/2010.3_RELEASE_NOTES.txt (full release notes, rather long and technical). Best regards, Robert Cimrman and Contributors (*) (*) Contributors to this release (alphabetical order): Vladim?r Luke?, Osman, Andre Smit, Logan Sorenson From bsouthey at gmail.com Fri Aug 6 14:11:29 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Fri, 06 Aug 2010 13:11:29 -0500 Subject: [SciPy-User] Compiling SciPy almost works In-Reply-To: <64A22683-5E7D-4DC0-AFDA-1E6C919B3CBB@cs.toronto.edu> References: <4C59554C.3060704@phys.au.dk> <4C5A59C7.5070204@phys.au.dk> <4C5A6425.5040604@silveregg.co.jp> <64A22683-5E7D-4DC0-AFDA-1E6C919B3CBB@cs.toronto.edu> Message-ID: <4C5C5051.3090402@gmail.com> On 08/06/2010 11:51 AM, David Warde-Farley wrote: > On 2010-08-05, at 3:11 AM, David wrote: > >> On 08/05/2010 03:27 PM, S?ren Gammelmark wrote: >>> I have attached the build-log. As far as I can see it chooses the Intel >>> compiler with the correct options as it should. >>> >>> Also recompiling after removing build did not change anything (I do not >>> believe it should have, since I was quite careful of doing this with >>> every attempt). >> Could you try linking the extension "manually" and add -lstdc++, i.e.. >> >> icc -fPIC -fomit-frame-pointer -O2 -g -parallel -shared >> build/temp.linux-x86_64-2.6/scipy/sparse/sparsetools/csr_wrap.o >> -Lbuild/temp.linux-x86_64-2.6 -o >> build/lib.linux-x86_64-2.6/scipy/sparse/sparsetools/_csr.so -lstdc++ > Hey David, > > The command I'm using is similar but the build actually fails for me with 11.1... It seems it does not like these sprintf statements: > > gpc-f101n084-$ icc -xHost -O3 -fPIC -openmp -I/home/dwf/pkg/lib/python2.7/site-packages/numpy/core/include -I/home/dwf/pkg/include/python2.7 -c scipy/sparse/sparsetools/csr_wrap.cxx -o build/temp.linux-x86_64-2.7/scipy/sparse/sparsetools/csr_wrap.o > scipy/sparse/sparsetools/csr_wrap.cxx(2910): error: expected a ")" > sprintf(s,"%" NPY_INTP_FMT ",", size[i]); > ^ > > scipy/sparse/sparsetools/csr_wrap.cxx(2917): error: expected a ")" > sprintf(s,"%" NPY_INTP_FMT ",", array_size(ary,i)); > ^ > > compilation aborted for scipy/sparse/sparsetools/csr_wrap.cxx (code 2) > gpc-f101n084-$ icc --version > icc (ICC) 11.1 20100414 > Copyright (C) 1985-2010 Intel Corporation. All rights reserved. > > > Any thoughts? > > David > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user This is the closed ticket 1180: http://projects.scipy.org/scipy/ticket/1180 See the thread from about two weeks ago: http://thread.gmane.org/gmane.comp.python.scientific.devel/14323/focus=14324 Hopefully that will give you sufficient information to address it for icc. Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.l.goldsmith at gmail.com Thu Aug 5 02:50:46 2010 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Wed, 4 Aug 2010 23:50:46 -0700 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: Message-ID: For the ellipse: import numpy as np from numpy.random import random_sample as random pi = np.pi def rand_in_ellipse(a, b=1, offset=0): angle = 2*pi*random() - offset x = a * random() * np.cos(angle) y = b * random() * np.sin(angle) return np.array((x,y)) import matplotlib.pyplot as plt fig = plt.figure() ax = fig.add_subplot(111) sample = np.array([rand_in_ellipse() for i in np.arange(10000)]) ax.scatter(sample[:,0], sample[:,1]) plt.show() # Figure attached DG On Wed, Aug 4, 2010 at 8:13 PM, Benjamin Root wrote: > On Wed, Aug 4, 2010 at 7:18 PM, alex wrote: > >> On Wed, Aug 4, 2010 at 7:38 PM, Benjamin Root wrote: >> >>> Hi, >>> >>> For a project, I need to create sets of random coordinates within a 2d >>> domain. To start, I have been creating random x and y coordinates, which >>> has worked very nicely. However, I would like to start doing some fancier >>> domains like ellipses and crescents. Does anybody know of any useful tricks >>> for doing this? >>> >>> Thanks, >>> Ben Root >>> >> >> For an ellipse you could start with random points in a disk like this >> http://mathworld.wolfram.com/DiskPointPicking.html >> and then I think you can just stretch along an axis to give the points >> inside the ellipse disk. >> For fancier shapes you can do rejection sampling as long as you can get a >> bounding box and >> you can tell what is inside vs outside the domain. >> >> Alex >> >> > Thanks Alex, that certainly does put me in the correct direction. It > appears that there isn't an analogous expression for an ellipse as there was > for a disc. I probably will have to just take your suggestion and do some > scaling/rotation upon the set of points to get what I want. It should be > more than sufficient, and it does look better than my original attempts. > > Ben Root > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- Mathematician: noun, someone who disavows certainty when their uncertainty set is non-empty, even if that set has measure zero. Hope: noun, that delusive spirit which escaped Pandora's jar and, with her lies, prevents mankind from committing a general suicide. (As interpreted by Robert Graves) -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 4thirds_ellipse.png Type: image/png Size: 213518 bytes Desc: not available URL: From d.l.goldsmith at gmail.com Thu Aug 5 03:49:26 2010 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Thu, 5 Aug 2010 00:49:26 -0700 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: Message-ID: This is embarrassing: clearly, that was not a uniform distribution over the ellipse; here's the fix: def rand_in_ellipse(a, b=1, offset=0): angle = 2*pi*random() - offset r = np.sqrt( (a*np.cos(angle))**2 + (b*np.sin(angle))**2 ) r *= random() x = r * np.cos(angle) y = r * np.sin(angle) return np.array((x,y)) Which yields the attached. (It looks denser in the center because the uniformity in the size of the points increases in significance as that size becomes of the same order as the distance away from other points.) DG On Wed, Aug 4, 2010 at 11:54 PM, David Goldsmith wrote: > Whoops, sorry, in the example, the call to rand_in_ellipse should have 4./3 > as a single argument, i.e., the line should look like > > sample = np.array([rand_in_ellipse(4./3) for i in np.arange(10000)]) > > > DG > > > On Wed, Aug 4, 2010 at 11:50 PM, David Goldsmith wrote: > >> For the ellipse: >> >> import numpy as np >> from numpy.random import random_sample as random >> >> pi = np.pi >> def rand_in_ellipse(a, b=1, offset=0): >> angle = 2*pi*random() - offset >> x = a * random() * np.cos(angle) >> y = b * random() * np.sin(angle) >> return np.array((x,y)) >> >> import matplotlib.pyplot as plt >> >> fig = plt.figure() >> >> ax = fig.add_subplot(111) >> >> sample = np.array([rand_in_ellipse() for i in np.arange(10000)]) >> ax.scatter(sample[:,0], sample[:,1]) >> >> plt.show() # Figure attached >> >> DG >> >> >> On Wed, Aug 4, 2010 at 8:13 PM, Benjamin Root wrote: >> >>> On Wed, Aug 4, 2010 at 7:18 PM, alex wrote: >>> >>>> On Wed, Aug 4, 2010 at 7:38 PM, Benjamin Root wrote: >>>> >>>>> Hi, >>>>> >>>>> For a project, I need to create sets of random coordinates within a 2d >>>>> domain. To start, I have been creating random x and y coordinates, which >>>>> has worked very nicely. However, I would like to start doing some fancier >>>>> domains like ellipses and crescents. Does anybody know of any useful tricks >>>>> for doing this? >>>>> >>>>> Thanks, >>>>> Ben Root >>>>> >>>> >>>> For an ellipse you could start with random points in a disk like this >>>> http://mathworld.wolfram.com/DiskPointPicking.html >>>> and then I think you can just stretch along an axis to give the points >>>> inside the ellipse disk. >>>> For fancier shapes you can do rejection sampling as long as you can get >>>> a bounding box and >>>> you can tell what is inside vs outside the domain. >>>> >>>> Alex >>>> >>>> >>> Thanks Alex, that certainly does put me in the correct direction. It >>> appears that there isn't an analogous expression for an ellipse as there was >>> for a disc. I probably will have to just take your suggestion and do some >>> scaling/rotation upon the set of points to get what I want. It should be >>> more than sufficient, and it does look better than my original attempts. >>> >>> Ben Root >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> >> >> -- >> Mathematician: noun, someone who disavows certainty when their uncertainty >> set is non-empty, even if that set has measure zero. >> >> Hope: noun, that delusive spirit which escaped Pandora's jar and, with her >> lies, prevents mankind from committing a general suicide. (As interpreted >> by Robert Graves) >> > > > > -- > Mathematician: noun, someone who disavows certainty when their uncertainty > set is non-empty, even if that set has measure zero. > > Hope: noun, that delusive spirit which escaped Pandora's jar and, with her > lies, prevents mankind from committing a general suicide. (As interpreted > by Robert Graves) > -- Mathematician: noun, someone who disavows certainty when their uncertainty set is non-empty, even if that set has measure zero. Hope: noun, that delusive spirit which escaped Pandora's jar and, with her lies, prevents mankind from committing a general suicide. (As interpreted by Robert Graves) -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 4thirds_ellipse.png Type: image/png Size: 213362 bytes Desc: not available URL: From d.l.goldsmith at gmail.com Thu Aug 5 04:12:46 2010 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Thu, 5 Aug 2010 01:12:46 -0700 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: Message-ID: One more fix (I've learned my lesson: thoroughly test before posting code) :-( def rand_in_ellipse(a, b=1, t=0): angle = 2*pi*random() r = np.sqrt( (a*np.cos(angle-t))**2 + (b*np.sin(angle-t))**2 ) r *= random() x = r * np.cos(angle) y = r * np.sin(angle) return np.array((x,y)) fig = plt.figure() ax = fig.add_subplot(111) sample = np.array([rand_in_ellipse(4., 3., np.pi/4) for i in np.arange(10000)]) ax.scatter(sample[:,0], sample[:,1], s=1) plt.show() # Figure attached DG On Thu, Aug 5, 2010 at 12:49 AM, David Goldsmith wrote: > This is embarrassing: clearly, that was not a uniform distribution over the > ellipse; here's the fix: > > > def rand_in_ellipse(a, b=1, offset=0): > angle = 2*pi*random() - offset > r = np.sqrt( (a*np.cos(angle))**2 + (b*np.sin(angle))**2 ) > r *= random() > x = r * np.cos(angle) > y = r * np.sin(angle) > > return np.array((x,y)) > > Which yields the attached. (It looks denser in the center because the > uniformity in the size of the points increases in significance as that size > becomes of the same order as the distance away from other points.) > > DG > > On Wed, Aug 4, 2010 at 11:54 PM, David Goldsmith wrote: > >> Whoops, sorry, in the example, the call to rand_in_ellipse should have >> 4./3 as a single argument, i.e., the line should look like >> >> sample = np.array([rand_in_ellipse(4./3) for i in np.arange(10000)]) >> >> >> >> DG >> >> >> On Wed, Aug 4, 2010 at 11:50 PM, David Goldsmith > > wrote: >> >>> For the ellipse: >>> >>> import numpy as np >>> from numpy.random import random_sample as random >>> >>> pi = np.pi >>> def rand_in_ellipse(a, b=1, offset=0): >>> angle = 2*pi*random() - offset >>> x = a * random() * np.cos(angle) >>> y = b * random() * np.sin(angle) >>> return np.array((x,y)) >>> >>> import matplotlib.pyplot as plt >>> >>> fig = plt.figure() >>> >>> >>> >>> ax = fig.add_subplot(111) >>> >>> sample = np.array([rand_in_ellipse() for i in np.arange(10000)]) >>> ax.scatter(sample[:,0], sample[:,1]) >>> >>> >>> >>> plt.show() # Figure attached >>> >>> DG >>> >>> >>> On Wed, Aug 4, 2010 at 8:13 PM, Benjamin Root wrote: >>> >>>> On Wed, Aug 4, 2010 at 7:18 PM, alex wrote: >>>> >>>>> On Wed, Aug 4, 2010 at 7:38 PM, Benjamin Root wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> For a project, I need to create sets of random coordinates within a 2d >>>>>> domain. To start, I have been creating random x and y coordinates, which >>>>>> has worked very nicely. However, I would like to start doing some fancier >>>>>> domains like ellipses and crescents. Does anybody know of any useful tricks >>>>>> for doing this? >>>>>> >>>>>> Thanks, >>>>>> Ben Root >>>>>> >>>>> >>>>> For an ellipse you could start with random points in a disk like this >>>>> http://mathworld.wolfram.com/DiskPointPicking.html >>>>> and then I think you can just stretch along an axis to give the points >>>>> inside the ellipse disk. >>>>> For fancier shapes you can do rejection sampling as long as you can get >>>>> a bounding box and >>>>> you can tell what is inside vs outside the domain. >>>>> >>>>> Alex >>>>> >>>>> >>>> Thanks Alex, that certainly does put me in the correct direction. It >>>> appears that there isn't an analogous expression for an ellipse as there was >>>> for a disc. I probably will have to just take your suggestion and do some >>>> scaling/rotation upon the set of points to get what I want. It should be >>>> more than sufficient, and it does look better than my original attempts. >>>> >>>> Ben Root >>>> >>>> _______________________________________________ >>>> SciPy-User mailing list >>>> SciPy-User at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/scipy-user >>>> >>>> >>> >>> >>> -- >>> Mathematician: noun, someone who disavows certainty when their >>> uncertainty set is non-empty, even if that set has measure zero. >>> >>> Hope: noun, that delusive spirit which escaped Pandora's jar and, with >>> her lies, prevents mankind from committing a general suicide. (As >>> interpreted by Robert Graves) >>> >> >> >> >> -- >> Mathematician: noun, someone who disavows certainty when their uncertainty >> set is non-empty, even if that set has measure zero. >> >> Hope: noun, that delusive spirit which escaped Pandora's jar and, with her >> lies, prevents mankind from committing a general suicide. (As interpreted >> by Robert Graves) >> > > > > -- > Mathematician: noun, someone who disavows certainty when their uncertainty > set is non-empty, even if that set has measure zero. > > Hope: noun, that delusive spirit which escaped Pandora's jar and, with her > lies, prevents mankind from committing a general suicide. (As interpreted > by Robert Graves) > -- Mathematician: noun, someone who disavows certainty when their uncertainty set is non-empty, even if that set has measure zero. Hope: noun, that delusive spirit which escaped Pandora's jar and, with her lies, prevents mankind from committing a general suicide. (As interpreted by Robert Graves) -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 43piover4ellipse.png Type: image/png Size: 230918 bytes Desc: not available URL: From sturla at molden.no Sat Aug 7 15:37:17 2010 From: sturla at molden.no (Sturla Molden) Date: Sat, 7 Aug 2010 21:37:17 +0200 Subject: [SciPy-User] absoft and f2py Message-ID: <7b42c743f36f8a4b71e6da41fd0209b0.squirrel@webmail.uio.no> I asked before, but did not get an answer. What is the problem with absoft and f2py? Sturla C:\Absoft11.0>f2py -c --help-fcompiler Forcing DISTUTILS_USE_SDK=1 Unexpected ValueError in C:\Python26\lib\site-packages\numpy\distutils\fcompiler \compaq.py Traceback (most recent call last): File "C:\Python26\Scripts\f2py-script.py", line 24, in main() File "C:\Python26\lib\site-packages\numpy\f2py\f2py2e.py", line 557, in main run_compile() File "C:\Python26\lib\site-packages\numpy\f2py\f2py2e.py", line 543, in run_co mpile setup(ext_modules = [ext]) File "C:\Python26\lib\site-packages\numpy\distutils\core.py", line 186, in set up return old_setup(**new_attr) File "C:\Python26\lib\distutils\core.py", line 138, in setup ok = dist.parse_command_line() File "C:\Python26\lib\distutils\dist.py", line 460, in parse_command_line args = self._parse_command_opts(parser, args) File "C:\Python26\lib\distutils\dist.py", line 574, in _parse_command_opts func() File "C:\Python26\lib\site-packages\numpy\distutils\command\config_compiler.py ", line 13, in show_fortran_compilers show_fcompilers(dist) File "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\__init__.py", li ne 848, in show_fcompilers load_all_fcompiler_classes() File "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\__init__.py", li ne 715, in load_all_fcompiler_classes __import__ (module_name) File "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\compaq.py", line 55, in class CompaqVisualFCompiler(FCompiler): File "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\compaq.py", line 95, in CompaqVisualFCompiler raise e ValueError: [u'path', u'lib'] From kwmsmith at gmail.com Sat Aug 7 20:33:39 2010 From: kwmsmith at gmail.com (Kurt Smith) Date: Sat, 7 Aug 2010 19:33:39 -0500 Subject: [SciPy-User] ANN fwrap v0.1.0 Message-ID: Fwrap v0.1.0 ============ I am pleased to announce the first release of Fwrap v0.1.0, a utility for wrapping Fortran code in C, Cython and Python. Fwrap focuses on supporting the features of Fortran 90/95, but will work with most well-behaved Fortran 77 code, too. Fwrap is BSD licensed. Fwrap is beta-level software; all public APIs and commandline options are subject to change. Features in the v0.1.0 release: * Fortran source parsing and automatic wrapper generation; * Top-level (non-module) Fortran subroutines and functions; * Supports all intrinsic types (integer, real, complex, logical & character); * Default and non-default intrinsic types properly wrapped; * Scalar and array arguments supported; * Supports assumed-shape, assumed-size, and explicit-shape array arguments; * Intent 'in', 'inout', 'out' and no intent arguments supported; * Automatic docstring generation for extension module and functions; * Wrappers are portable w.r.t. both Fortran and C compilers. Upcoming features: * "Use" statement support; * Python & Cython callback support; * Kind-parameters in type specification; * Module-level data and parameters; * User-derived types. See the README and documentation for requirements, compiler support, etc. Download -------- You can get fwrap from pypi or from its sourceforge download page: https://sourceforge.net/projects/fwrap/files/ More Info --------- Project homepage, including links to wiki & bug tracker: http://fwrap.sourceforge.net/ Mailing list: http://groups.google.com/group/fwrap-users Development blog: http://fortrancython.wordpress.com/ From sierra_mtnview at sbcglobal.net Sun Aug 8 19:33:21 2010 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Sun, 08 Aug 2010 16:33:21 -0700 Subject: [SciPy-User] Win7. Why Don't Matplotlib, ... Show up in Control Panel Add-Remove? Message-ID: <4C5F3EC1.50309@sbcglobal.net> See Subject. I use matplotlib, scipy, numpy and possibly one other module. If I go to the control panel, I only see numpy listed. Why? I use a search and find only numpy and Python itself. How can matplotlib and scipy be uninstalled? -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet "An experiment is a question which science poses to Nature, and a measurement is the recording of Nature?s answer." -- Max Planck Web Page: From josef.pktd at gmail.com Sun Aug 8 19:42:51 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 8 Aug 2010 19:42:51 -0400 Subject: [SciPy-User] Win7. Why Don't Matplotlib, ... Show up in Control Panel Add-Remove? In-Reply-To: <4C5F3EC1.50309@sbcglobal.net> References: <4C5F3EC1.50309@sbcglobal.net> Message-ID: On Sun, Aug 8, 2010 at 7:33 PM, Wayne Watson wrote: > See Subject. I use matplotlib, scipy, numpy and possibly one other > module. If I go to the control panel, I only see numpy listed. Why? I > use a search and find only numpy and Python itself. How can matplotlib > and scipy be uninstalled? numpy-discussion Feb 5 and Feb 6 Josef > > -- > ? ? ? ? ? ?Wayne Watson (Watson Adventures, Prop., Nevada City, CA) > > ? ? ? ? ? ? ?(121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) > ? ? ? ? ? ? ? Obz Site: ?39? 15' 7" N, 121? 2' 32" W, 2700 feet > > ? ? ? ? ? ? "An experiment is a question which science poses to > ? ? ? ? ? ? ?Nature, and a measurement is the recording of > ? ? ? ? ? ? ?Nature?s answer." -- Max Planck > > > ? ? ? ? ? ? ? ? ? ? Web Page: > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From sierra_mtnview at sbcglobal.net Sun Aug 8 20:24:59 2010 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Sun, 08 Aug 2010 17:24:59 -0700 Subject: [SciPy-User] Win7. Why Don't Matplotlib, ... Show up in Control Panel Add-Remove? In-Reply-To: References: <4C5F3EC1.50309@sbcglobal.net> Message-ID: <4C5F4ADB.6050505@sbcglobal.net> Sorry I don't understand this. If you mean I should look at postings around that date, I was not a member of the list then. On 8/8/2010 4:42 PM, josef.pktd at gmail.com wrote: > On Sun, Aug 8, 2010 at 7:33 PM, Wayne Watson > wrote: > >> See Subject. I use matplotlib, scipy, numpy and possibly one other >> module. If I go to the control panel, I only see numpy listed. Why? I >> use a search and find only numpy and Python itself. How can matplotlib >> and scipy be uninstalled? >> > numpy-discussion Feb 5 and Feb 6 > > Josef > > >> -- >> Wayne Watson (Watson Adventures, Prop., Nevada City, CA) >> >> (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) >> Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet >> >> "An experiment is a question which science poses to >> Nature, and a measurement is the recording of >> Nature?s answer." -- Max Planck >> >> >> Web Page: >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet "An experiment is a question which science poses to Nature, and a measurement is the recording of Nature?s answer." -- Max Planck Web Page: From felipeacsi at gmail.com Sun Aug 8 20:34:26 2010 From: felipeacsi at gmail.com (Felipe Raimann) Date: Sun, 8 Aug 2010 20:34:26 -0400 Subject: [SciPy-User] Win7. Why Don't Matplotlib, ... Show up in Control Panel Add-Remove? In-Reply-To: <4C5F4ADB.6050505@sbcglobal.net> References: <4C5F3EC1.50309@sbcglobal.net> <4C5F4ADB.6050505@sbcglobal.net> Message-ID: See the list archives (http://mail.scipy.org/pipermail/scipy-user/) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tgrav at mac.com Sun Aug 8 20:35:43 2010 From: tgrav at mac.com (Tommy Grav) Date: Sun, 08 Aug 2010 20:35:43 -0400 Subject: [SciPy-User] Win7. Why Don't Matplotlib, ... Show up in Control Panel Add-Remove? In-Reply-To: References: <4C5F3EC1.50309@sbcglobal.net> <4C5F4ADB.6050505@sbcglobal.net> Message-ID: <1F40F91C-55B8-4151-930B-33D150DB4B5B@mac.com> On Aug 8, 2010, at 8:34 PM, Felipe Raimann wrote: > See the list archives (http://mail.scipy.org/pipermail/scipy-user/) > ______________________________________________ or http://mail.scipy.org/mailman/listinfo for archive for all the appropriate lists. Tommy -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sun Aug 8 20:58:39 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 8 Aug 2010 20:58:39 -0400 Subject: [SciPy-User] Win7. Why Don't Matplotlib, ... Show up in Control Panel Add-Remove? In-Reply-To: <1F40F91C-55B8-4151-930B-33D150DB4B5B@mac.com> References: <4C5F3EC1.50309@sbcglobal.net> <4C5F4ADB.6050505@sbcglobal.net> <1F40F91C-55B8-4151-930B-33D150DB4B5B@mac.com> Message-ID: On Sun, Aug 8, 2010 at 8:35 PM, Tommy Grav wrote: > > On Aug 8, 2010, at 8:34 PM, Felipe Raimann wrote: > > See the list archives (http://mail.scipy.org/pipermail/scipy-user/) > ______________________________________________ > > or?http://mail.scipy.org/mailman/listinfo > for archive for all the appropriate lists. > Tommy or more specifically http://mail.scipy.org/pipermail/numpy-discussion/2010-February/048315.html Josef > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From sierra_mtnview at sbcglobal.net Sun Aug 8 21:42:01 2010 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Sun, 08 Aug 2010 18:42:01 -0700 Subject: [SciPy-User] Win7. Why Don't Matplotlib, ... Show up in Control Panel Add-Remove? In-Reply-To: References: <4C5F3EC1.50309@sbcglobal.net> <4C5F4ADB.6050505@sbcglobal.net> <1F40F91C-55B8-4151-930B-33D150DB4B5B@mac.com> Message-ID: <4C5F5CE9.4050902@sbcglobal.net> Funny, I wrote the post; however, my intention here is to find out why that seems to be true. Why is MPL so special that it doesn't list in Control Panel Add-Remove? On 8/8/2010 5:58 PM, josef.pktd at gmail.com wrote: > On Sun, Aug 8, 2010 at 8:35 PM, Tommy Grav wrote: > >> On Aug 8, 2010, at 8:34 PM, Felipe Raimann wrote: >> >> See the list archives (http://mail.scipy.org/pipermail/scipy-user/) >> ______________________________________________ >> >> or http://mail.scipy.org/mailman/listinfo >> for archive for all the appropriate lists. >> Tommy >> > or more specifically > http://mail.scipy.org/pipermail/numpy-discussion/2010-February/048315.html > > Josef > > >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> >> > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet "An experiment is a question which science poses to Nature, and a measurement is the recording of Nature?s answer." -- Max Planck Web Page: From robert.kern at gmail.com Sun Aug 8 21:50:57 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 8 Aug 2010 20:50:57 -0500 Subject: [SciPy-User] Win7. Why Don't Matplotlib, ... Show up in Control Panel Add-Remove? In-Reply-To: <4C5F5CE9.4050902@sbcglobal.net> References: <4C5F3EC1.50309@sbcglobal.net> <4C5F4ADB.6050505@sbcglobal.net> <1F40F91C-55B8-4151-930B-33D150DB4B5B@mac.com> <4C5F5CE9.4050902@sbcglobal.net> Message-ID: On Sun, Aug 8, 2010 at 20:42, Wayne Watson wrote: > Funny, I wrote the post; however, my intention here is to find out why > that seems to be true. Why is MPL so special that it doesn't list in > Control Panel Add-Remove? Most likely you used an MSI installer for numpy and the .exe installers for scipy and matplotlib. The things in the Add-Remove panel are only MSIs. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From sierra_mtnview at sbcglobal.net Sun Aug 8 22:24:55 2010 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Sun, 08 Aug 2010 19:24:55 -0700 Subject: [SciPy-User] Win7. Why Don't Matplotlib, ... Show up in Control Panel Add-Remove? In-Reply-To: References: <4C5F3EC1.50309@sbcglobal.net> <4C5F4ADB.6050505@sbcglobal.net> <1F40F91C-55B8-4151-930B-33D150DB4B5B@mac.com> <4C5F5CE9.4050902@sbcglobal.net> Message-ID: <4C5F66F7.9060206@sbcglobal.net> That's interesting. I just searched for numpy, and found all the installers were exe, and not msi. Same for matplotlib, and scipy. I just looked at the users manual, and it shows only exe. OK, at one time the product distributed msi files. Well, this is still a mystery. On 8/8/2010 6:50 PM, Robert Kern wrote: > On Sun, Aug 8, 2010 at 20:42, Wayne Watson wrote: > >> Funny, I wrote the post; however, my intention here is to find out why >> that seems to be true. Why is MPL so special that it doesn't list in >> Control Panel Add-Remove? >> > Most likely you used an MSI installer for numpy and the .exe > installers for scipy and matplotlib. The things in the Add-Remove > panel are only MSIs. > > -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet "An experiment is a question which science poses to Nature, and a measurement is the recording of Nature?s answer." -- Max Planck Web Page: From robert.kern at gmail.com Sun Aug 8 22:33:14 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 8 Aug 2010 21:33:14 -0500 Subject: [SciPy-User] Win7. Why Don't Matplotlib, ... Show up in Control Panel Add-Remove? In-Reply-To: <4C5F66F7.9060206@sbcglobal.net> References: <4C5F3EC1.50309@sbcglobal.net> <4C5F4ADB.6050505@sbcglobal.net> <1F40F91C-55B8-4151-930B-33D150DB4B5B@mac.com> <4C5F5CE9.4050902@sbcglobal.net> <4C5F66F7.9060206@sbcglobal.net> Message-ID: On Sun, Aug 8, 2010 at 21:24, Wayne Watson wrote: > That's interesting. I just searched for numpy, and found all the > installers were exe, and not msi. Same for matplotlib, and scipy. > > I just looked at the users manual, and it shows only exe. OK, at one > time the product distributed msi files. Well, this is still a mystery. numpy 1.3.0 had an .msi installer. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From cgohlke at uci.edu Sun Aug 8 23:54:10 2010 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sun, 08 Aug 2010 20:54:10 -0700 Subject: [SciPy-User] Win7. Why Don't Matplotlib, ... Show up in Control Panel Add-Remove? In-Reply-To: <4C5F5CE9.4050902@sbcglobal.net> References: <4C5F3EC1.50309@sbcglobal.net> <4C5F4ADB.6050505@sbcglobal.net> <1F40F91C-55B8-4151-930B-33D150DB4B5B@mac.com> <4C5F5CE9.4050902@sbcglobal.net> Message-ID: <4C5F7BE2.50101@uci.edu> On 8/8/2010 6:42 PM, Wayne Watson wrote: > Funny, I wrote the post; however, my intention here is to find out why > that seems to be true. Why is MPL so special that it doesn't list in > Control Panel Add-Remove? First, it would really help if you specify the versions of Python, matplotlib, etc. you are using. Second, I believe your question has previously been answered in the numpy-discussion thread referenced by Josef. Chances are you are using Python 2.5 and the matplotlib and scipy installers created with Python 2.5's distutils. These installers don't know about Windows 7 UAC and unless they are run specifically with administrator privileges will not be able to write the uninstall entries to the to the Windows registry. The numpy installers are using NSIS, which is UAC aware. > > On 8/8/2010 5:58 PM, josef.pktd at gmail.com wrote: >> On Sun, Aug 8, 2010 at 8:35 PM, Tommy Grav wrote: >> >>> On Aug 8, 2010, at 8:34 PM, Felipe Raimann wrote: >>> >>> See the list archives (http://mail.scipy.org/pipermail/scipy-user/) >>> ______________________________________________ >>> >>> or http://mail.scipy.org/mailman/listinfo >>> for archive for all the appropriate lists. >>> Tommy >>> >> or more specifically >> http://mail.scipy.org/pipermail/numpy-discussion/2010-February/048315.html >> >> Josef -- Christoph From josef.pktd at gmail.com Mon Aug 9 00:02:45 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 9 Aug 2010 00:02:45 -0400 Subject: [SciPy-User] stats-mstats plotting_positions, quantiles Message-ID: I stumbled over plotting positions in mstats while looking at the Pareto family of distributions. plotting_positions, quantiles in stats.mstats has more options that the functions in stats (similar to a ecdf proposal by David Huard) Instead of writing plain ndarray versions, I was trying to have a common interface for plain ndarrays, ndarrays with nans and masked arrays. The implementations are different enough that merging the ndarray and masked array version didn't look useful (e.g. masked arrays or nans require apply_along_axis). Instead I just delegated the messy cases (ma, nans, limit) to stats.mstats and only the nice cases go through the plain ndarray version. Main question: Would it be useful to have this delegation in scipy.stats so that there is a single entry point for users, or is it better to keep the plain ndarray, nan and ma versions separate? The pattern could apply to quite a few functions in stats-mstats that are too difficult to merge. As a bonus, I added a plotting_positions_w1d that handles weights (since I recently saw the question somewhere). I am not completely sure about the definition for the plotting position correction, but if desired it will be easy enough to include it in the other functions or write also versions of quantiles and scoreatpercentile that take weights. Josef -------------- next part -------------- '''get versions of mstats percentile functions that also work with non-masked arrays uses dispatch to mstats version for difficult cases: - data is masked array - data requires nan handling (masknan=True) - data should be trimmed (limit is non-empty) handle simple cases directly, which doesn't require apply_along_axis changes compared to mstats: plotting_positions for n-dim with axis argument addition: plotting_positions_w1d: with weights, 1d ndarray only TODO: consistency with scipy.stats versions not checked docstrings from mstats not updated yet code duplication, better solutions (?) convert examples to tests rename alphap, betap for consistency timing question: one additional argsort versus apply_along_axis weighted plotting_positions - I haven't figured out nd version of weighted plotting_positions - add weighted quantiles ''' import numpy as np from numpy import ma from scipy import stats #from numpy.ma import nomask #####-------------------------------------------------------------------------- #---- --- Percentiles --- #####-------------------------------------------------------------------------- def quantiles(a, prob=list([.25,.5,.75]), alphap=.4, betap=.4, axis=None, limit=(), masknan=False): """ Computes empirical quantiles for a data array. Samples quantile are defined by :math:`Q(p) = (1-g).x[i] +g.x[i+1]`, where :math:`x[j]` is the *j*th order statistic, and `i = (floor(n*p+m))`, `m=alpha+p*(1-alpha-beta)` and `g = n*p + m - i`. Typical values of (alpha,beta) are: - (0,1) : *p(k) = k/n* : linear interpolation of cdf (R, type 4) - (.5,.5) : *p(k) = (k+1/2.)/n* : piecewise linear function (R, type 5) - (0,0) : *p(k) = k/(n+1)* : (R type 6) - (1,1) : *p(k) = (k-1)/(n-1)*. In this case, p(k) = mode[F(x[k])]. That's R default (R type 7) - (1/3,1/3): *p(k) = (k-1/3)/(n+1/3)*. Then p(k) ~ median[F(x[k])]. The resulting quantile estimates are approximately median-unbiased regardless of the distribution of x. (R type 8) - (3/8,3/8): *p(k) = (k-3/8)/(n+1/4)*. Blom. The resulting quantile estimates are approximately unbiased if x is normally distributed (R type 9) - (.4,.4) : approximately quantile unbiased (Cunnane) - (.35,.35): APL, used with PWM Parameters ---------- a : array-like Input data, as a sequence or array of dimension at most 2. prob : array-like, optional List of quantiles to compute. alpha : float, optional Plotting positions parameter, default is 0.4. beta : float, optional Plotting positions parameter, default is 0.4. axis : int, optional Axis along which to perform the trimming. If None (default), the input array is first flattened. limit : tuple Tuple of (lower, upper) values. Values of `a` outside this closed interval are ignored. Returns ------- quants : MaskedArray An array containing the calculated quantiles. Examples -------- >>> from scipy.stats.mstats import mquantiles >>> a = np.array([6., 47., 49., 15., 42., 41., 7., 39., 43., 40., 36.]) >>> mquantiles(a) array([ 19.2, 40. , 42.8]) Using a 2D array, specifying axis and limit. >>> data = np.array([[ 6., 7., 1.], [ 47., 15., 2.], [ 49., 36., 3.], [ 15., 39., 4.], [ 42., 40., -999.], [ 41., 41., -999.], [ 7., -999., -999.], [ 39., -999., -999.], [ 43., -999., -999.], [ 40., -999., -999.], [ 36., -999., -999.]]) >>> mquantiles(data, axis=0, limit=(0, 50)) array([[ 19.2 , 14.6 , 1.45], [ 40. , 37.5 , 2.5 ], [ 42.8 , 40.05, 3.55]]) >>> data[:, 2] = -999. >>> mquantiles(data, axis=0, limit=(0, 50)) masked_array(data = [[19.2 14.6 --] [40.0 37.5 --] [42.8 40.05 --]], mask = [[False False True] [False False True] [False False True]], fill_value = 1e+20) """ if isinstance(a, np.ma.MaskedArray): return stats.mstats.mquantiles(a, prob=prob, alphap=alphap, betap=alphap, axis=axis, limit=limit) if limit: marr = stats.mstats.mquantiles(a, prob=prob, alphap=alphap, betap=alphap, axis=axis, limit=limit) return ma.filled(marr, fill_value=np.nan) if masknan: nanmask = np.isnan(a) if nanmask.any(): marr = ma.array(a, mask=nanmask) marr = stats.mstats.mquantiles(marr, prob=prob, alphap=alphap, betap=alphap, axis=axis, limit=limit) return ma.filled(marr, fill_value=np.nan) # Initialization & checks --------- data = np.asarray(a) p = np.array(prob, copy=False, ndmin=1) m = alphap + p*(1.-alphap-betap) isrolled = False #from _quantiles1d if (axis is None): data = data.ravel() #reshape(-1,1) axis = 0 else: axis = np.arange(data.ndim)[axis] data = np.rollaxis(data, axis) isrolled = True # keep track, maybe can be removed x = np.sort(data, axis=0) n = x.shape[0] returnshape = list(data.shape) returnshape[axis] = p #TODO: check these if n == 0: return np.empty(len(p), dtype=float) elif n == 1: return np.resize(x, p.shape) aleph = (n*p + m) k = np.floor(aleph.clip(1, n-1)).astype(int) ind = [None]*x.ndim ind[0] = slice(None) gamma = (aleph-k).clip(0,1)[ind] q = (1.-gamma)*x[k-1] + gamma*x[k] if isrolled: return np.rollaxis(q, 0, axis+1) else: return q def scoreatpercentile(data, per, limit=(), alphap=.4, betap=.4, axis=0, masknan=None): """Calculate the score at the given 'per' percentile of the sequence a. For example, the score at per=50 is the median. This function is a shortcut to mquantile """ per = np.asarray(per, float) if (per < 0).any() or (per > 100.).any(): raise ValueError("The percentile should be between 0. and 100. !"\ " (got %s)" % per) return quantiles(data, prob=[per/100.], alphap=alphap, betap=betap, limit=limit, axis=axis, masknan=masknan).squeeze() def plotting_positions(data, alpha=0.4, beta=0.4, axis=0, masknan=False): """Returns the plotting positions (or empirical percentile points) for the data. Plotting positions are defined as (i-alpha)/(n-alpha-beta), where: - i is the rank order statistics - n is the number of unmasked values along the given axis - alpha and beta are two parameters. Typical values for alpha and beta are: - (0,1) : *p(k) = k/n* : linear interpolation of cdf (R, type 4) - (.5,.5) : *p(k) = (k-1/2.)/n* : piecewise linear function (R, type 5) - (0,0) : *p(k) = k/(n+1)* : Weibull (R type 6) - (1,1) : *p(k) = (k-1)/(n-1)*. In this case, p(k) = mode[F(x[k])]. That's R default (R type 7) - (1/3,1/3): *p(k) = (k-1/3)/(n+1/3)*. Then p(k) ~ median[F(x[k])]. The resulting quantile estimates are approximately median-unbiased regardless of the distribution of x. (R type 8) - (3/8,3/8): *p(k) = (k-3/8)/(n+1/4)*. Blom. The resulting quantile estimates are approximately unbiased if x is normally distributed (R type 9) - (.4,.4) : approximately quantile unbiased (Cunnane) - (.35,.35): APL, used with PWM Parameters ---------- x : sequence Input data, as a sequence or array of dimension at most 2. prob : sequence List of quantiles to compute. alpha : {0.4, float} optional Plotting positions parameter. beta : {0.4, float} optional Plotting positions parameter. """ if isinstance(data, np.ma.MaskedArray): if axis is None or data.ndim == 1: return stats.mstats.plotting_positions(data, alpha=alpha, beta=beta) else: return ma.apply_along_axis(stats.mstats.plotting_positions, axis, data, alpha=alpha, beta=beta) if masknan: nanmask = np.isnan(data) if nanmask.any(): marr = ma.array(data, mask=nanmask) #code duplication: if axis is None or data.ndim == 1: marr = stats.mstats.plotting_positions(marr, alpha=alpha, beta=beta) else: marr = ma.apply_along_axis(stats.mstats.plotting_positions, axis, marr, alpha=alpha, beta=beta) return ma.filled(marr, fill_value=np.nan) data = np.asarray(data) if data.size == 1: # use helper function instead data = np.atleast_1d(data) axis = 0 if axis is None: data = data.ravel() axis = 0 n = data.shape[axis] if data.ndim == 1: plpos = np.empty(data.shape, dtype=float) plpos[data.argsort()] = (np.arange(1,n+1) - alpha)/(n+1.-alpha-beta) else: #nd assignment instead of second argsort doesn't look easy plpos = (data.argsort(axis).argsort(axis) + 1. - alpha)/(n+1.-alpha-beta) return plpos meppf = plotting_positions def plotting_positions_w1d(data, weights=None, alpha=0.4, beta=0.4, method='notnormed'): '''Weighted plotting positions (or empirical percentile points) for the data. observations are weighted and the plotting positions are defined as (ws-alpha)/(n-alpha-beta), where: - ws is the weighted rank order statistics or cumulative weighted sum, normalized to n if method is "normed" - n is the number of values along the given axis if method is "normed" and total weight otherwise - alpha and beta are two parameters. wtd.quantile in R package Hmisc seems to use the "notnormed" version. notnormed coincides with unweighted segmetn in example, drop "normed" version ? See Also -------- plotting_positions : unweighted version that works also with more than one dimension and has other options ''' x = np.atleast_1d(data) if x.ndim > 1: raise ValueError('currently implemented only for 1d') if weights is None: weights = np.ones(x.shape) else: weights = np.array(weights, float, copy=False, ndmin=1) #atleast_1d(weights) if weights.shape != x.shape: raise ValueError('if weights is given, it needs to be the same' 'shape as data') n = len(x) xargsort = x.argsort() ws = weights[xargsort].cumsum() res = np.empty(x.shape) if method == 'normed': res[xargsort] = (1.*ws/ws[-1]*n-alpha)/(n+1.-alpha-beta) else: res[xargsort] = (1.*ws-alpha)/(ws[-1]+1.-alpha-beta) return res if __name__ == '__main__': x = np.arange(5) print plotting_positions(x) x = np.arange(10).reshape(-1,2) print plotting_positions(x) print quantiles(x, axis=0) print quantiles(x, axis=None) print quantiles(x, axis=1) xm = ma.array(x) x2 = x.astype(float) x2[1,0] = np.nan print plotting_positions(xm, axis=0) # test 0d, 1d for sl1 in [slice(None), 0]: print (plotting_positions(xm[sl1,0]) == plotting_positions(x[sl1,0])).all(), print (quantiles(xm[sl1,0]) == quantiles(x[sl1,0])).all(), print (stats.mstats.mquantiles(ma.fix_invalid(x2[sl1,0])) == quantiles(x2[sl1,0], masknan=1)).all(), #test 2d for ax in [0, 1, None, -1]: print (plotting_positions(xm, axis=ax) == plotting_positions(x, axis=ax)).all(), print (quantiles(xm, axis=ax) == quantiles(x, axis=ax)).all(), print (stats.mstats.mquantiles(ma.fix_invalid(x2), axis=ax) == quantiles(x2, axis=ax, masknan=1)).all(), #stats version doesn't have axis print (stats.mstats.plotting_positions(ma.fix_invalid(x2)) == plotting_positions(x2, axis=None, masknan=1)).all(), #test 3d x3 = np.dstack((x,x)).T for ax in [1,2]: print (plotting_positions(x3, axis=ax)[0] == plotting_positions(x.T, axis=ax-1)).all(), print print scoreatpercentile(x, [10,90]) print plotting_positions_w1d(x[:,0]) print (plotting_positions_w1d(x[:,0]) == plotting_positions(x[:,0])).all() #weights versus replicating multiple occurencies of same x value w1 = [1, 1, 2, 1, 1] plotexample = 1 if plotexample: import matplotlib.pyplot as plt plt.figure() plt.title('ppf, cdf values on horizontal axis') plt.step(plotting_positions_w1d(x[:,0], weights=w1, method='0'), x[:,0], where='post') plt.step(stats.mstats.plotting_positions(np.repeat(x[:,0],w1,axis=0)),np.repeat(x[:,0],w1,axis=0),where='post') plt.plot(plotting_positions_w1d(x[:,0], weights=w1, method='0'), x[:,0], '-o') plt.plot(stats.mstats.plotting_positions(np.repeat(x[:,0],w1,axis=0)),np.repeat(x[:,0],w1,axis=0), '-o') plt.figure() plt.title('cdf, cdf values on vertical axis') plt.step(x[:,0], plotting_positions_w1d(x[:,0], weights=w1, method='0'),where='post') plt.step(np.repeat(x[:,0],w1,axis=0), stats.mstats.plotting_positions(np.repeat(x[:,0],w1,axis=0)),where='post') plt.plot(x[:,0], plotting_positions_w1d(x[:,0], weights=w1, method='0'), '-o') plt.plot(np.repeat(x[:,0],w1,axis=0), stats.mstats.plotting_positions(np.repeat(x[:,0],w1,axis=0)), '-o') plt.show() From pgmdevlist at gmail.com Mon Aug 9 01:14:10 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 9 Aug 2010 01:14:10 -0400 Subject: [SciPy-User] stats-mstats plotting_positions, quantiles In-Reply-To: References: Message-ID: On Aug 9, 2010, at 12:02 AM, josef.pktd at gmail.com wrote: > I stumbled over plotting positions in mstats while looking at the > Pareto family of distributions. > > plotting_positions, quantiles in stats.mstats has more options that > the functions in stats (similar to a ecdf proposal by David Huard) I followed the example of R on that one, and took the definitions from the Hydrology Handbook. I kinda suspect that David did the same... > Instead of writing plain ndarray versions, I was trying to have a > common interface for plain ndarrays, ndarrays with nans and masked > arrays. The implementations are different enough that merging the > ndarray and masked array version didn't look useful (e.g. masked > arrays or nans require apply_along_axis). Because of a variable nb of nans/missing in each column... Mmh, looks like the ndarray case is the special one here. > > Instead I just delegated the messy cases (ma, nans, limit) to > stats.mstats and only the nice cases go through the plain ndarray > version. > > Main question: Would it be useful to have this delegation in > scipy.stats so that there is a single entry point for users, or is it > better to keep the plain ndarray, nan and ma versions separate? > The pattern could apply to quite a few functions in stats-mstats that > are too difficult to merge. As long as you import numpy.ma inside the function, and not at the module level... I expect some people won't like numpy.ma overhead by default. > As a bonus, I added a plotting_positions_w1d that handles weights > (since I recently saw the question somewhere). I am not completely > sure about the definition for the plotting position correction, but if > desired it will be easy enough to include it in the other functions or > write also versions of quantiles and scoreatpercentile that take > weights. Well, me neither. Float weights kinda defeat the purpose of plotting positions, don't you think ? From super.inframan at gmail.com Mon Aug 9 11:19:28 2010 From: super.inframan at gmail.com (Mr Nilsson) Date: Mon, 9 Aug 2010 08:19:28 -0700 (PDT) Subject: [SciPy-User] soft clipping values above... oh nevermind Message-ID: Hi Ive been trying to figure out this mathematical expression for a while, but got stuck so thought maybe someone here could help me. but as I tried to write down the problem in a structured and easy to understand way, then I realised what I need to do to make it work lol thanks guys! From zachary.pincus at yale.edu Mon Aug 9 11:47:56 2010 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Mon, 9 Aug 2010 11:47:56 -0400 Subject: [SciPy-User] soft clipping values above... oh nevermind In-Reply-To: References: Message-ID: I was told that at CMU once upon a time, the computer-science TAs would have a teddy bear outside their office, and before you could ask the TAs a question about a bug, you had to ask the bear. And darn if the teddy bear didn't answer about half of the questions... Can anyone confirm this, which has got to be among my favorite probably-apocryphal stories? Zach On Aug 9, 2010, at 11:19 AM, Mr Nilsson wrote: > Hi > Ive been trying to figure out this mathematical expression for a > while, but got stuck so thought maybe someone here could help me. but > as I tried to write down the problem in a structured and easy to > understand way, then I realised what I need to do to make it work lol > > thanks guys! > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From zachary.pincus at yale.edu Mon Aug 9 15:19:32 2010 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Mon, 9 Aug 2010 15:19:32 -0400 Subject: [SciPy-User] stats.pearsonr divide by zero warning Message-ID: Hello, I just svn-up'd scipy, and now find that stats.pearsonr is causing divide-by-zero warnings foolishly. the function contains the following stanzas: rs = np.corrcoef(ar,br,rowvar=axisout) t = rs * np.sqrt((n-2) / ((rs+1.0)*(1.0-rs))) prob = distributions.t.sf(np.abs(t),n-2)*2 if rs.shape == (2,2): return rs[1,0], prob[1,0] else: return rs, prob Given that the diagonal of the correlation matrix returned by corrcoef will *always* be 1s, the t matrix will have divide-by-zero issues on the diagonal, and give inf values -- which get zero values for the t- distribution's survival function, so everything's fine, output-wise. Presumably, though, the t-calculating line should be flanked by err = np.seterr(divide='ignore') / np.seterr(**err), right? Should I add a bug in the tracker? Someone want to just commit this fix? Zach From josef.pktd at gmail.com Mon Aug 9 15:46:27 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 9 Aug 2010 15:46:27 -0400 Subject: [SciPy-User] stats.pearsonr divide by zero warning In-Reply-To: References: Message-ID: On Mon, Aug 9, 2010 at 3:19 PM, Zachary Pincus wrote: > Hello, > > I just svn-up'd scipy, and now find that stats.pearsonr is causing > divide-by-zero warnings foolishly. > > the function contains the following stanzas: > > ? ? rs = np.corrcoef(ar,br,rowvar=axisout) > > ? ? t = rs * np.sqrt((n-2) / ((rs+1.0)*(1.0-rs))) > ? ? prob = distributions.t.sf(np.abs(t),n-2)*2 > > ? ? if rs.shape == (2,2): > ? ? ? ? return rs[1,0], prob[1,0] > ? ? else: > ? ? ? ? return rs, prob > > Given that the diagonal of the correlation matrix returned by corrcoef > will *always* be 1s, the t matrix will have divide-by-zero issues on > the diagonal, and give inf values -- which get zero values for the t- > distribution's survival function, so everything's fine, output-wise. > Presumably, though, the t-calculating line should be flanked by err = > np.seterr(divide='ignore') / np.seterr(**err), right? > > Should I add a bug in the tracker? Someone want to just commit this fix? I guess you mean spearmanr, pearsonr hasn't been rewritten as far as I can see. The old trick (still used in pearsonr) was to add TINY in the calculation of the test statistic. Maybe we should add TINY to the diagonal, which would keep a zero division warning if any of the series are perfectly correlated. seterr is also fine with me. a ticket is always good, at least for the record so we know what to watch out for. I have warnings turned off globally, so no zero division problems for me. np.corrcoef might throw a warning if there is zero variance, but I'm not sure this applies in this case Josef > > Zach > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From josef.pktd at gmail.com Mon Aug 9 16:18:14 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 9 Aug 2010 16:18:14 -0400 Subject: [SciPy-User] stats.pearsonr divide by zero warning In-Reply-To: References: Message-ID: On Mon, Aug 9, 2010 at 3:46 PM, wrote: > On Mon, Aug 9, 2010 at 3:19 PM, Zachary Pincus wrote: >> Hello, >> >> I just svn-up'd scipy, and now find that stats.pearsonr is causing >> divide-by-zero warnings foolishly. >> >> the function contains the following stanzas: >> >> ? ? rs = np.corrcoef(ar,br,rowvar=axisout) >> >> ? ? t = rs * np.sqrt((n-2) / ((rs+1.0)*(1.0-rs))) >> ? ? prob = distributions.t.sf(np.abs(t),n-2)*2 >> >> ? ? if rs.shape == (2,2): >> ? ? ? ? return rs[1,0], prob[1,0] >> ? ? else: >> ? ? ? ? return rs, prob >> >> Given that the diagonal of the correlation matrix returned by corrcoef >> will *always* be 1s, the t matrix will have divide-by-zero issues on >> the diagonal, and give inf values -- which get zero values for the t- >> distribution's survival function, so everything's fine, output-wise. >> Presumably, though, the t-calculating line should be flanked by err = >> np.seterr(divide='ignore') / np.seterr(**err), right? >> >> Should I add a bug in the tracker? Someone want to just commit this fix? > > I guess you mean spearmanr, ?pearsonr hasn't been rewritten as far as I can see. > > The old trick (still used in pearsonr) was to add TINY in the > calculation of the test statistic. > Maybe we should add TINY to the diagonal, which would keep a zero > division warning if any of the series are perfectly correlated. > > seterr is also fine with me. > > a ticket is always good, at least for the record so we know what to > watch out for. I have warnings turned off globally, so no zero > division problems for me. > > np.corrcoef might throw a warning if there is zero variance, but I'm > not sure this applies in this case > > Josef just a follow-up because I think there is a similar case in the contingency table code mut_inf = np.nansum(self.probability * np.log(self.observed / self.expected)) Do we really need to protect everywhere for zero division warnings? I think in statsmodels we worked around the warning in 0*np.log(0) or something like this. Josef > > > >> >> Zach >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > From jsseabold at gmail.com Mon Aug 9 16:29:23 2010 From: jsseabold at gmail.com (Skipper Seabold) Date: Mon, 9 Aug 2010 16:29:23 -0400 Subject: [SciPy-User] stats.pearsonr divide by zero warning In-Reply-To: References: Message-ID: On Mon, Aug 9, 2010 at 4:18 PM, wrote: > On Mon, Aug 9, 2010 at 3:46 PM, ? wrote: >> On Mon, Aug 9, 2010 at 3:19 PM, Zachary Pincus wrote: >>> Hello, >>> >>> I just svn-up'd scipy, and now find that stats.pearsonr is causing >>> divide-by-zero warnings foolishly. >>> >>> the function contains the following stanzas: >>> >>> ? ? rs = np.corrcoef(ar,br,rowvar=axisout) >>> >>> ? ? t = rs * np.sqrt((n-2) / ((rs+1.0)*(1.0-rs))) >>> ? ? prob = distributions.t.sf(np.abs(t),n-2)*2 >>> >>> ? ? if rs.shape == (2,2): >>> ? ? ? ? return rs[1,0], prob[1,0] >>> ? ? else: >>> ? ? ? ? return rs, prob >>> >>> Given that the diagonal of the correlation matrix returned by corrcoef >>> will *always* be 1s, the t matrix will have divide-by-zero issues on >>> the diagonal, and give inf values -- which get zero values for the t- >>> distribution's survival function, so everything's fine, output-wise. >>> Presumably, though, the t-calculating line should be flanked by err = >>> np.seterr(divide='ignore') / np.seterr(**err), right? >>> >>> Should I add a bug in the tracker? Someone want to just commit this fix? >> >> I guess you mean spearmanr, ?pearsonr hasn't been rewritten as far as I can see. >> >> The old trick (still used in pearsonr) was to add TINY in the >> calculation of the test statistic. >> Maybe we should add TINY to the diagonal, which would keep a zero >> division warning if any of the series are perfectly correlated. >> >> seterr is also fine with me. >> >> a ticket is always good, at least for the record so we know what to >> watch out for. I have warnings turned off globally, so no zero >> division problems for me. >> >> np.corrcoef might throw a warning if there is zero variance, but I'm >> not sure this applies in this case >> >> Josef > > just a follow-up because I think there is a similar case in the > contingency table code > > mut_inf = np.nansum(self.probability * np.log(self.observed / self.expected)) > > Do we really need to protect everywhere for zero division warnings? > Is this protecting against a zero division warning? > I think in statsmodels we worked around the warning in 0*np.log(0) or > something like this. > IIRC, we use a mask for these kind of cases in Shannon entropy to avoid the ugly warning. I don't know about speed. The current implementation of stats.entropy uses where, but this doesn't avoid the warning. Skipper From josef.pktd at gmail.com Mon Aug 9 16:35:49 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 9 Aug 2010 16:35:49 -0400 Subject: [SciPy-User] stats.pearsonr divide by zero warning In-Reply-To: References: Message-ID: On Mon, Aug 9, 2010 at 4:29 PM, Skipper Seabold wrote: > On Mon, Aug 9, 2010 at 4:18 PM, ? wrote: >> On Mon, Aug 9, 2010 at 3:46 PM, ? wrote: >>> On Mon, Aug 9, 2010 at 3:19 PM, Zachary Pincus wrote: >>>> Hello, >>>> >>>> I just svn-up'd scipy, and now find that stats.pearsonr is causing >>>> divide-by-zero warnings foolishly. >>>> >>>> the function contains the following stanzas: >>>> >>>> ? ? rs = np.corrcoef(ar,br,rowvar=axisout) >>>> >>>> ? ? t = rs * np.sqrt((n-2) / ((rs+1.0)*(1.0-rs))) >>>> ? ? prob = distributions.t.sf(np.abs(t),n-2)*2 >>>> >>>> ? ? if rs.shape == (2,2): >>>> ? ? ? ? return rs[1,0], prob[1,0] >>>> ? ? else: >>>> ? ? ? ? return rs, prob >>>> >>>> Given that the diagonal of the correlation matrix returned by corrcoef >>>> will *always* be 1s, the t matrix will have divide-by-zero issues on >>>> the diagonal, and give inf values -- which get zero values for the t- >>>> distribution's survival function, so everything's fine, output-wise. >>>> Presumably, though, the t-calculating line should be flanked by err = >>>> np.seterr(divide='ignore') / np.seterr(**err), right? >>>> >>>> Should I add a bug in the tracker? Someone want to just commit this fix? >>> >>> I guess you mean spearmanr, ?pearsonr hasn't been rewritten as far as I can see. >>> >>> The old trick (still used in pearsonr) was to add TINY in the >>> calculation of the test statistic. >>> Maybe we should add TINY to the diagonal, which would keep a zero >>> division warning if any of the series are perfectly correlated. >>> >>> seterr is also fine with me. >>> >>> a ticket is always good, at least for the record so we know what to >>> watch out for. I have warnings turned off globally, so no zero >>> division problems for me. >>> >>> np.corrcoef might throw a warning if there is zero variance, but I'm >>> not sure this applies in this case >>> >>> Josef >> >> just a follow-up because I think there is a similar case in the >> contingency table code >> >> mut_inf = np.nansum(self.probability * np.log(self.observed / self.expected)) >> >> Do we really need to protect everywhere for zero division warnings? >> > > Is this protecting against a zero division warning? No, it will cause a warning by design whenever there is a zero probability. The question is whether we need a work-around in the code as we did in statsmodels. I never tried to see how many zero division and other warnings are raised in the regular usage in scipy.stats, but I guess a lot. Josef > >> I think in statsmodels we worked around the warning in 0*np.log(0) or >> something like this. >> > > IIRC, we use a mask for these kind of cases in Shannon entropy to > avoid the ugly warning. ?I don't know about speed. ?The current > implementation of stats.entropy uses where, but this doesn't avoid the > warning. > > Skipper > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From ben.root at ou.edu Mon Aug 9 16:51:08 2010 From: ben.root at ou.edu (Benjamin Root) Date: Mon, 9 Aug 2010 15:51:08 -0500 Subject: [SciPy-User] soft clipping values above... oh nevermind In-Reply-To: References: Message-ID: I use my cat... best debugger on four paws I have ever seen! :-P Ben Root On Mon, Aug 9, 2010 at 10:47 AM, Zachary Pincus wrote: > I was told that at CMU once upon a time, the computer-science TAs > would have a teddy bear outside their office, and before you could ask > the TAs a question about a bug, you had to ask the bear. And darn if > the teddy bear didn't answer about half of the questions... > > Can anyone confirm this, which has got to be among my favorite > probably-apocryphal stories? > > Zach > > > > On Aug 9, 2010, at 11:19 AM, Mr Nilsson wrote: > > > Hi > > Ive been trying to figure out this mathematical expression for a > > while, but got stuck so thought maybe someone here could help me. but > > as I tried to write down the problem in a structured and easy to > > understand way, then I realised what I need to do to make it work lol > > > > thanks guys! > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From R.Springuel at umit.maine.edu Mon Aug 9 17:36:56 2010 From: R.Springuel at umit.maine.edu (R. Padraic Springuel) Date: Mon, 09 Aug 2010 17:36:56 -0400 Subject: [SciPy-User] Elementwise equivalent to allclose Message-ID: <4C6074F8.7090501@umit.maine.edu> Is there an element wise equivalent to allclose? I.e. a function which doesn't apply the final all() step? -- R. Padraic Springuel Research Assistant Department of Physics and Astronomy University of Maine Bennett 309 Office Hours: By Appointment Only From zachary.pincus at yale.edu Mon Aug 9 17:48:07 2010 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Mon, 9 Aug 2010 17:48:07 -0400 Subject: [SciPy-User] stats.pearsonr divide by zero warning In-Reply-To: References: Message-ID: <722CA1A9-C752-41C6-AD05-A504A7504B81@yale.edu> On Aug 9, 2010, at 3:46 PM, josef.pktd at gmail.com wrote: > On Mon, Aug 9, 2010 at 3:19 PM, Zachary Pincus > wrote: >> Hello, >> >> I just svn-up'd scipy, and now find that stats.pearsonr is causing >> divide-by-zero warnings foolishly. >> >> the function contains the following stanzas: >> >> rs = np.corrcoef(ar,br,rowvar=axisout) >> >> t = rs * np.sqrt((n-2) / ((rs+1.0)*(1.0-rs))) >> prob = distributions.t.sf(np.abs(t),n-2)*2 >> >> if rs.shape == (2,2): >> return rs[1,0], prob[1,0] >> else: >> return rs, prob >> >> Given that the diagonal of the correlation matrix returned by >> corrcoef >> will *always* be 1s, the t matrix will have divide-by-zero issues on >> the diagonal, and give inf values -- which get zero values for the t- >> distribution's survival function, so everything's fine, output-wise. >> Presumably, though, the t-calculating line should be flanked by err = >> np.seterr(divide='ignore') / np.seterr(**err), right? >> >> Should I add a bug in the tracker? Someone want to just commit this >> fix? > > I guess you mean spearmanr, pearsonr hasn't been rewritten as far > as I can see. Ugh, yes. Fingers typed a different thing than I was thinking! > The old trick (still used in pearsonr) was to add TINY in the > calculation of the test statistic. > Maybe we should add TINY to the diagonal, which would keep a zero > division warning if any of the series are perfectly correlated. > > seterr is also fine with me. > > a ticket is always good, at least for the record so we know what to > watch out for. I have warnings turned off globally, so no zero > division problems for me. http://projects.scipy.org/scipy/ticket/1259 > np.corrcoef might throw a warning if there is zero variance, but I'm > not sure this applies in this case > > Josef > > > >> >> Zach >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From fperez.net at gmail.com Tue Aug 10 03:51:22 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 10 Aug 2010 00:51:22 -0700 Subject: [SciPy-User] soft clipping values above... oh nevermind In-Reply-To: References: Message-ID: On Mon, Aug 9, 2010 at 8:47 AM, Zachary Pincus wrote: > Can anyone confirm this, which has got to be among my favorite > probably-apocryphal stories? > The highly technical term you are looking for is 'rubber-duck debugging': http://en.wikipedia.org/wiki/Rubber_duck_debugging See, without knowing the precise terminology, it's nigh impossible to advance in these complex and jargon-laden fields... ;) Cheers, f From rigal at rapideye.de Tue Aug 10 10:11:27 2010 From: rigal at rapideye.de (Matthieu Rigal) Date: Tue, 10 Aug 2010 16:11:27 +0200 Subject: [SciPy-User] Best method to pick-up every N-th sample of an array Message-ID: <201008101611.28165.rigal@rapideye.de> Hi folks, I could not find an appropriate function to pick-up every N-th value of an array into another one... for example : [0,1,2,3,4,5] could return [0,2,4], for N = 2 Neither I could find help browsing the dev-lists The functions numpy.choose and numpy.take may give correct results, but after having created the appropriate mask.. which may not be the best when you handle arrays containing hundred million of values, and I wouldn't be sure how to build it. Therefore I wrote the following subfunction, but it is not very powerful: def ReduceArray(x,y): x0=[] j=0 for i in x: j=j+1 if j==y: x0.append(i) j=0 return numpy.array(x0) Yes, using a simple Python list in between ... :-((( So if you had some hints on which way to follow to have this computed quickly, on a flat or a ndarray, you will make someone happy :-) Best regards, Matthieu PS : I apologize it this should have go to the numpy list, I was unsure... RapidEye AG Molkenmarkt 30 14776 Brandenburg an der Havel Germany Follow us on Twitter! www.twitter.com/rapideye_ag Head Office/Sitz der Gesellschaft: Brandenburg an der Havel Management Board/Vorstand: Wolfgang G. Biedermann Chairman of Supervisory Board/Vorsitzender des Aufsichtsrates: Juergen Breitkopf Commercial Register/Handelsregister Potsdam HRB 17 796 Tax Number/Steuernummer: 048/100/00053 VAT-Ident-Number/Ust.-ID: DE 199331235 DIN EN ISO 9001 certified ************************************************************************* Diese E-Mail enthaelt vertrauliche und/oder rechtlich geschuetzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtuemlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet. The information in this e-mail is intended for the named recipients only. It may contain privileged and confidential information. If you have received this communication in error, any use, copying or dissemination of its contents is strictly prohibited. Please erase all copies of the message along with any included attachments and notify RapidEye AG or the sender immediately by telephone at the number indicated on this page. From rmay31 at gmail.com Tue Aug 10 10:15:33 2010 From: rmay31 at gmail.com (Ryan May) Date: Tue, 10 Aug 2010 09:15:33 -0500 Subject: [SciPy-User] Best method to pick-up every N-th sample of an array In-Reply-To: <201008101611.28165.rigal@rapideye.de> References: <201008101611.28165.rigal@rapideye.de> Message-ID: On Tue, Aug 10, 2010 at 9:11 AM, Matthieu Rigal wrote: > Hi folks, > > I could not find an appropriate function to pick-up every N-th value of an > array into another one... for example : > [0,1,2,3,4,5] could return [0,2,4], for N = 2 > Neither I could find help browsing the dev-lists > > The functions numpy.choose and numpy.take may give correct results, but after > having created the appropriate mask.. which may not be the best when you > handle arrays containing hundred million of values, and I wouldn't be sure how > to build it. Python's slice notation supports it, with slices being noted as: start:stop:step If start is not given, it defaults to the start of the array (0). If stop is not given, it defaults to the end. The following examples should explain what I mean: In [1]: a = np.arange(10) In [2]: a Out[2]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) In [3]: a[::2] Out[3]: array([0, 2, 4, 6, 8]) In [4]: a[::3] Out[4]: array([0, 3, 6, 9]) In [5]: a[1::2] Out[5]: array([1, 3, 5, 7, 9]) Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma From ben.root at ou.edu Tue Aug 10 10:18:11 2010 From: ben.root at ou.edu (Benjamin Root) Date: Tue, 10 Aug 2010 09:18:11 -0500 Subject: [SciPy-User] Best method to pick-up every N-th sample of an array In-Reply-To: <201008101611.28165.rigal@rapideye.de> References: <201008101611.28165.rigal@rapideye.de> Message-ID: On Tue, Aug 10, 2010 at 9:11 AM, Matthieu Rigal wrote: > Hi folks, > > I could not find an appropriate function to pick-up every N-th value of an > array into another one... for example : > [0,1,2,3,4,5] could return [0,2,4], for N = 2 > Neither I could find help browsing the dev-lists > > The functions numpy.choose and numpy.take may give correct results, but > after > having created the appropriate mask.. which may not be the best when you > handle arrays containing hundred million of values, and I wouldn't be sure > how > to build it. > > Therefore I wrote the following subfunction, but it is not very powerful: > > def ReduceArray(x,y): > x0=[] > j=0 > for i in x: > j=j+1 > if j==y: > x0.append(i) > j=0 > return numpy.array(x0) > > Yes, using a simple Python list in between ... :-((( > > So if you had some hints on which way to follow to have this computed > quickly, > on a flat or a ndarray, you will make someone happy :-) > > Best regards, > Matthieu > > PS : I apologize it this should have go to the numpy list, I was unsure... > > RapidEye AG > Molkenmarkt 30 > 14776 Brandenburg an der Havel > Germany > > Would slicing the array help? >>> a = numpy.array([0, 1, 2, 3, 4, 5]) >>> a[::2] # every second element array([0, 2, 4]) As a side benefit, you can save on memory because this 'slice' is actually a view into the original array. I hope that helps, Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertwb at math.washington.edu Sun Aug 8 00:12:20 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 7 Aug 2010 21:12:20 -0700 Subject: [SciPy-User] [cython-users] ANN fwrap v0.1.0 In-Reply-To: References: Message-ID: Excellent news! May you get lots of users and (easy to address) bug reports. Sounds like it does a lot already. - Robert On Sat, Aug 7, 2010 at 5:33 PM, Kurt Smith wrote: > Fwrap v0.1.0 > ============ > > I am pleased to announce the first release of Fwrap v0.1.0, a utility for > wrapping Fortran code in C, Cython and Python. ?Fwrap focuses on supporting the > features of Fortran 90/95, but will work with most well-behaved Fortran 77 > code, too. > > Fwrap is BSD licensed. > > Fwrap is beta-level software; all public APIs and commandline options are > subject to change. > > Features in the v0.1.0 release: > > ? ?* Fortran source parsing and automatic wrapper generation; > > ? ?* Top-level (non-module) Fortran subroutines and functions; > > ? ?* Supports all intrinsic types (integer, real, complex, logical & > ? ? ?character); > > ? ?* Default and non-default intrinsic types properly wrapped; > > ? ?* Scalar and array arguments supported; > > ? ?* Supports assumed-shape, assumed-size, and explicit-shape array arguments; > > ? ?* Intent 'in', 'inout', 'out' and no intent arguments supported; > > ? ?* Automatic docstring generation for extension module and functions; > > ? ?* Wrappers are portable w.r.t. both Fortran and C compilers. > > Upcoming features: > > ? ?* "Use" statement support; > > ? ?* Python & Cython callback support; > > ? ?* Kind-parameters in type specification; > > ? ?* Module-level data and parameters; > > ? ?* User-derived types. > > See the README and documentation for requirements, compiler support, etc. > > Download > -------- > > You can get fwrap from pypi or from its sourceforge download page: > > ? ?https://sourceforge.net/projects/fwrap/files/ > > More Info > --------- > > Project homepage, including links to wiki & bug tracker: > > ? ?http://fwrap.sourceforge.net/ > > Mailing list: > > ? ?http://groups.google.com/group/fwrap-users > > Development blog: > > ? ?http://fortrancython.wordpress.com/ > From vanforeest at gmail.com Sun Aug 8 17:03:07 2010 From: vanforeest at gmail.com (nicky van foreest) Date: Sun, 8 Aug 2010 23:03:07 +0200 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: Message-ID: Hi, Good idea to include the graphical result. I must admit that the results of the latest mail by David do not satisfy my "random intuition". There is a thick cloud around the origin, which seems unnatural to me. Just for fun I implemented a rejection method that seems to give good results. From an algorithmic perspective my implementation is far from optimal (BTW, I welcome feedback on how to speed up code like this.) However, the rejection method does not seem that bad, roughly 20% of the numbers is rejected, and there is no need to compute cosines and the like. The resulting graph seems ok to me. I modified the example of DG somewhat for the example below. Nicky #!/usr/bin/env python import numpy as np from numpy.random import random_sample as random a = 1.; b = 0.5 def inEllipse(x,y): return (x/a)**2 + (y/b)**2 < 1. # taking sqrt is unnecessary num = 1e4 x = -a + 2*a*random(num) y = -b + 2*b*random(num) sample = [] reject = 0 for xx, yy in zip(x,y): if inEllipse(xx,yy): sample.append([xx,yy]) else: reject += 1 print reject sample = np.array(sample) import matplotlib.pyplot as plt fig = plt.figure() ax = fig.add_subplot(111) ax.scatter(sample[:,0], sample[:,1]) plt.show() # Figure attached -------------- next part -------------- A non-text attachment was scrubbed... Name: ellips.png Type: image/png Size: 387665 bytes Desc: not available URL: From tgrav at me.com Sun Aug 8 20:29:59 2010 From: tgrav at me.com (Tommy Grav) Date: Sun, 08 Aug 2010 20:29:59 -0400 Subject: [SciPy-User] Win7. Why Don't Matplotlib, ... Show up in Control Panel Add-Remove? In-Reply-To: <4C5F4ADB.6050505@sbcglobal.net> References: <4C5F3EC1.50309@sbcglobal.net> <4C5F4ADB.6050505@sbcglobal.net> Message-ID: <5B01C396-1426-4509-8070-8C7AD7A8AB5A@me.com> On Aug 8, 2010, at 8:24 PM, Wayne Watson wrote: > Sorry I don't understand this. If you mean I should look at postings > around that date, I was not a member of the list then. http://mail.scipy.org/pipermail/numpy-discussion/ An archive of the numpy-discussion list. Took me 10 seconds to google. Look at the february 5-6 postings. Tommy From seb.haase at gmail.com Tue Aug 10 11:29:34 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Tue, 10 Aug 2010 17:29:34 +0200 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: Message-ID: Hi, in case you need speed... if you generate (and test and reject) always 1000 xy points at a time, you will be MUCH (100x ?) faster. Try using vector operations where possible. Regards, Sebastian Haase On Sun, Aug 8, 2010 at 11:03 PM, nicky van foreest wrote: > Hi, > > Good idea to include the graphical result. > > I must admit that the results of the latest mail by David do not > satisfy my "random intuition". There is a thick cloud around the > origin, which seems unnatural to me. > > Just for fun I implemented a rejection method that seems to give good > results. From an algorithmic perspective my implementation is far from > optimal (BTW, I welcome feedback on how to speed up code like this.) > However, the rejection method does not seem that bad, roughly 20% of > the numbers is rejected, and there is no need to compute cosines and > the like. The resulting graph seems ok to me. > > I modified the example of DG somewhat for the example below. > > Nicky > > #!/usr/bin/env python > > import numpy as np > from numpy.random import random_sample as random > > a = 1.; b = 0.5 > > def inEllipse(x,y): > ? ?return (x/a)**2 + (y/b)**2 < 1. ?# taking sqrt is unnecessary > > num = 1e4 > x = -a + 2*a*random(num) > y = -b + 2*b*random(num) > > sample = [] > reject = 0 > for xx, yy in zip(x,y): > ? ?if inEllipse(xx,yy): > ? ? ? ?sample.append([xx,yy]) > ? ?else: > ? ? ? ?reject += 1 > > print reject > sample = np.array(sample) > > import matplotlib.pyplot as plt > > fig = plt.figure() > > ax = fig.add_subplot(111) > > ax.scatter(sample[:,0], sample[:,1]) > > plt.show() # Figure attached > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From timmichelsen at gmx-topmail.de Tue Aug 10 11:41:32 2010 From: timmichelsen at gmx-topmail.de (Timmie) Date: Tue, 10 Aug 2010 15:41:32 +0000 (UTC) Subject: [SciPy-User] help in starting with netCDF Message-ID: Hello, I am finally diving into netCDF. I need some assistance in defining coordinate variables such as "humidity:coordinates" May someone please help me creating a python file with this CDL: dimensions: station = 10 ; // measurement locations pressure = 11 ; // pressure levels time = UNLIMITED ; variables: float humidity(time,pressure,station) ; humidity:long_name = "specific humidity" ; humidity:coordinates = "lat lon" ; double time(time) ; time:long_name = "time of measurement" ; time:units = "days since 1970-01-01 00:00:00" ; float lon(station) ; lon:long_name = "station longitude"; lon:units = "degrees_east"; float lat(station) ; lat:long_name = "station latitude" ; lat:units = "degrees_north" ; float pressure(pressure) ; pressure:long_name = "pressure" ; pressure:units = "hPa" ; Source: http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.4/ch05s04.html#id3141969 Here is my start: from netCDF4 import Dataset ncfile = Dataset('station_data.nc','w') ncfile.createDimension('station', 10) ncfile.createDimension('pressure', 11) ncfile.createDimension('time',None) hum = ncfile.createVariable('hum',dtype('float32'), ('lat', 'lon', 'time', 'pressure', 'station')) hum.long_name = 'specific humidity' timevar = ncfile.createVariable('time',dtype('float32'), ('time')) timevar.long_name = 'time of measurement' timevar.units = 'days since 1970-01-01 00:00:00' lon = ncfile.createVariable('lon',dtype('float32'), ('station')) lon.long_name = 'station longitude' lon.units = 'degrees_east' lat = ncfile.createVariable('lat',dtype('float32'), ('station')) lat.long_name = 'station latitude' lat.units = 'degrees_west' pressure = ncfile.createVariable('pressure',dtype('float32'), ('pressure')) pressure.long_name = 'pressure' pressure.units = 'hPa' ncfile.close() I also have the following question: Is it possible with some python library to convert cdl or ncml into a nc file? I would like to use this NC file in order to stort 4D data: network of measurements stations with a corresponding time series for each station. Thanks a lot in advance, Timmie From gnurser at gmail.com Tue Aug 10 12:03:15 2010 From: gnurser at gmail.com (George Nurser) Date: Tue, 10 Aug 2010 17:03:15 +0100 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: Message-ID: Hi, Perhaps #!/usr/bin/env python import numpy as np from numpy.random import random_sample as random a = 1.; b = 0.5 num = 1e4 x = -a + 2*a*random(num) y = -b + 2*b*random(num) ra2 = 1./(a*a) rb2 = 1./(b*b) # create boolean mask inside = x*x*ra2 + y*y*rb2 <1. xv = x[inside] yv = y[inside] import matplotlib.pyplot as plt fig = plt.figure() ax = fig.add_subplot(111) ax.scatter(xv,yv) ax.axis('equal') plt.show() # Figure attached HTH. George. On 10 August 2010 16:29, Sebastian Haase wrote: > Hi, > in case you need speed... > if you generate (and test and reject) always 1000 xy points at a time, > you will be MUCH (100x ?) faster. > Try using vector operations where possible. > > Regards, > Sebastian Haase > > > On Sun, Aug 8, 2010 at 11:03 PM, nicky van foreest wrote: >> Hi, >> >> Good idea to include the graphical result. >> >> I must admit that the results of the latest mail by David do not >> satisfy my "random intuition". There is a thick cloud around the >> origin, which seems unnatural to me. >> >> Just for fun I implemented a rejection method that seems to give good >> results. From an algorithmic perspective my implementation is far from >> optimal (BTW, I welcome feedback on how to speed up code like this.) >> However, the rejection method does not seem that bad, roughly 20% of >> the numbers is rejected, and there is no need to compute cosines and >> the like. The resulting graph seems ok to me. >> >> I modified the example of DG somewhat for the example below. >> >> Nicky >> >> #!/usr/bin/env python >> >> import numpy as np >> from numpy.random import random_sample as random >> >> a = 1.; b = 0.5 >> >> def inEllipse(x,y): >> ? ?return (x/a)**2 + (y/b)**2 < 1. ?# taking sqrt is unnecessary >> >> num = 1e4 >> x = -a + 2*a*random(num) >> y = -b + 2*b*random(num) >> >> sample = [] >> reject = 0 >> for xx, yy in zip(x,y): >> ? ?if inEllipse(xx,yy): >> ? ? ? ?sample.append([xx,yy]) >> ? ?else: >> ? ? ? ?reject += 1 >> >> print reject >> sample = np.array(sample) >> >> import matplotlib.pyplot as plt >> >> fig = plt.figure() >> >> ax = fig.add_subplot(111) >> >> ax.scatter(sample[:,0], sample[:,1]) >> >> plt.show() # Figure attached >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From erin.sheldon at gmail.com Tue Aug 10 12:11:07 2010 From: erin.sheldon at gmail.com (Erin Sheldon) Date: Tue, 10 Aug 2010 12:11:07 -0400 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: Message-ID: <1281456532-sup-8954@theshire> Excerpts from nicky van foreest's message of Sun Aug 08 17:03:07 -0400 2010: > Hi, > > Good idea to include the graphical result. > > I must admit that the results of the latest mail by David do not > satisfy my "random intuition". There is a thick cloud around the > origin, which seems unnatural to me. > > Just for fun I implemented a rejection method that seems to give good > results. From an algorithmic perspective my implementation is far from > optimal (BTW, I welcome feedback on how to speed up code like this.) > However, the rejection method does not seem that bad, roughly 20% of > the numbers is rejected, and there is no need to compute cosines and > the like. The resulting graph seems ok to me. > > I modified the example of DG somewhat for the example below. Nicky - I think this version (from my original post) is fairly optimal: def get_random_ellipse(n, x0, y0): xout = numpy.zeros(n) yout = numpy.zeros(n) nkeep=0 while nkeep < n: x=2*x0*(random.random(n-nkeep) - 0.5) y=2*y0*(random.random(n-nkeep) - 0.5) w,=where( ( (x/x0)**2 + (y/y0)**2 ) < 1 ) if w.size > 0: xout[nkeep:nkeep+w.size] = x[w] yout[nkeep:nkeep+w.size] = y[w] nkeep += w.size return xout,yout From josef.pktd at gmail.com Tue Aug 10 12:26:10 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 10 Aug 2010 12:26:10 -0400 Subject: [SciPy-User] random points within an ellipse In-Reply-To: <1281456532-sup-8954@theshire> References: <1281456532-sup-8954@theshire> Message-ID: On Tue, Aug 10, 2010 at 12:11 PM, Erin Sheldon wrote: > Excerpts from nicky van foreest's message of Sun Aug 08 17:03:07 -0400 2010: >> Hi, >> >> Good idea to include the graphical result. >> >> I must admit that the results of the latest mail by David do not >> satisfy my "random intuition". There is a thick cloud around the >> origin, which seems unnatural to me. >> >> Just for fun I implemented a rejection method that seems to give good >> results. From an algorithmic perspective my implementation is far from >> optimal (BTW, I welcome feedback on how to speed up code like this.) >> However, the rejection method does not seem that bad, roughly 20% of >> the numbers is rejected, and there is no need to compute cosines and >> the like. The resulting graph seems ok to me. >> >> I modified the example of DG somewhat for the example below. > > Nicky - > > I think this version (from my original post) is fairly optimal: > > ?def get_random_ellipse(n, x0, y0): > > ? ? ?xout = numpy.zeros(n) > ? ? ?yout = numpy.zeros(n) > > ? ? ?nkeep=0 > > ? ? ?while nkeep < n: > ? ? ? ? ?x=2*x0*(random.random(n-nkeep) - 0.5) > ? ? ? ? ?y=2*y0*(random.random(n-nkeep) - 0.5) > > ? ? ? ? ?w,=where( ( (x/x0)**2 + (y/y0)**2 ) < 1 ) > ? ? ? ? ?if w.size > 0: > ? ? ? ? ? ? ?xout[nkeep:nkeep+w.size] = x[w] > ? ? ? ? ? ? ?yout[nkeep:nkeep+w.size] = y[w] > ? ? ? ? ? ? ?nkeep += w.size > > ? ? ?return xout,yout I would oversample if the rejection rate is around 20% Josef > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From ben.root at ou.edu Tue Aug 10 12:33:45 2010 From: ben.root at ou.edu (Benjamin Root) Date: Tue, 10 Aug 2010 11:33:45 -0500 Subject: [SciPy-User] help in starting with netCDF In-Reply-To: References: Message-ID: On Tue, Aug 10, 2010 at 10:41 AM, Timmie wrote: > Hello, > I am finally diving into netCDF. > > I need some assistance in defining coordinate variables such as > "humidity:coordinates" > > May someone please help me creating a python file with this CDL: > > dimensions: > station = 10 ; // measurement locations > pressure = 11 ; // pressure levels > time = UNLIMITED ; > variables: > float humidity(time,pressure,station) ; > humidity:long_name = "specific humidity" ; > humidity:coordinates = "lat lon" ; > > double time(time) ; > time:long_name = "time of measurement" ; > time:units = "days since 1970-01-01 00:00:00" ; > > float lon(station) ; > lon:long_name = "station longitude"; > lon:units = "degrees_east"; > float lat(station) ; > lat:long_name = "station latitude" ; > lat:units = "degrees_north" ; > float pressure(pressure) ; > pressure:long_name = "pressure" ; > pressure:units = "hPa" ; > > Source: > http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.4/ch05s04.html#id3141969 > > Here is my start: > > from netCDF4 import Dataset > > ncfile = Dataset('station_data.nc','w') > > ncfile.createDimension('station', 10) > ncfile.createDimension('pressure', 11) > ncfile.createDimension('time',None) > > hum = ncfile.createVariable('hum',dtype('float32'), > ('lat', 'lon', 'time', 'pressure', 'station')) > hum.long_name = 'specific humidity' > > timevar = ncfile.createVariable('time',dtype('float32'), ('time')) > timevar.long_name = 'time of measurement' > timevar.units = 'days since 1970-01-01 00:00:00' > > lon = ncfile.createVariable('lon',dtype('float32'), ('station')) > lon.long_name = 'station longitude' > lon.units = 'degrees_east' > > lat = ncfile.createVariable('lat',dtype('float32'), ('station')) > lat.long_name = 'station latitude' > lat.units = 'degrees_west' > > pressure = ncfile.createVariable('pressure',dtype('float32'), ('pressure')) > pressure.long_name = 'pressure' > pressure.units = 'hPa' > > ncfile.close() > > I also have the following question: > Is it possible with some python library to convert cdl or ncml into a nc > file? > > I would like to use this NC file in order to stort 4D data: network of > measurements stations with a corresponding time series for each station. > > Thanks a lot in advance, > Timmie > > Timmie, If you are already using scipy, then you can use scipy.io.netcdf to help you (this presumes that the arrays 'humidityData', 'lonData', 'latData', 'timeData', and 'pressureData' exists). from scipy.io import netcdf ncfile = netcdf.netcdf_file('station_data.nc','w') ncfile.createDimension('station', 10) ncfile.createDimension('pressure', 11) ncfile.createDimension('time',None) hum = ncfile.createVariable('humidity', 'f', ('time', 'pressure', 'station')) hum.long_name = 'specific humidity' hum.coordinates = 'lat lon' hum[:] = humidityData timevar = ncfile.createVariable('time', 'f', ('time',)) timevar.long_name = 'time of measurement' timevar.units = 'days since 1970-01-01 00:00:00' timevar[:] = timeData lon = ncfile.createVariable('lon', 'f', ('station',)) lon.long_name = 'station longitude' lon.units = 'degrees_east' lon[:] = lonData lat = ncfile.createVariable('lat', 'f', ('station',)) lat.long_name = 'station latitude' lat.units = 'degrees_west' lat[:] = latData pressure = ncfile.createVariable('pressure', 'f', ('pressure',)) pressure.long_name = 'pressure' pressure.units = 'hPa' pressure[:] = pressureData ncfile.close() Now, if you want hum.coordinates to be an array, you can do: hum.coordinates = ['lat', 'lon'] Also, this won't exactly produce the right CDL, as the type for timevar is set as a float, not a double. I haven't checked, but I think a 'd' is a valid descriptor. Anyway, to find out the lat and lon for a particular humidity observation (lets say it is the 4th station): stationHum = ncfile.variables['humidity'][:, :, 3] stationLat = ncfile.variables['lat'][3] stationLon = ncfile.variables['lon'][3] I hope this is helpful! Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanforeest at gmail.com Tue Aug 10 14:08:35 2010 From: vanforeest at gmail.com (nicky van foreest) Date: Tue, 10 Aug 2010 20:08:35 +0200 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: <1281456532-sup-8954@theshire> Message-ID: Hi, Thanks for the nice feedback. This is a bit too short for me: > I would oversample if the rejection rate is around 20% > > Josef BTW: The rejection fraction of an ellipse is 1 - \pi/4 \approx 0.21. What is wrong with using the rejection method? bye Nicky From josef.pktd at gmail.com Tue Aug 10 14:17:29 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 10 Aug 2010 14:17:29 -0400 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: <1281456532-sup-8954@theshire> Message-ID: On Tue, Aug 10, 2010 at 2:08 PM, nicky van foreest wrote: > Hi, > > Thanks for the nice feedback. > > This is a bit too short for me: > >> I would oversample if the rejection rate is around 20% >> >> Josef > > BTW: The rejection fraction of an ellipse is 1 - \pi/4 \approx 0.21. > What is wrong with using the rejection method? Everything is fine. I meant that in Erin's code the exact number of still missing random variables are drawn in each iteration. So it will take many iterations, with smaller and smaller sample sizes to get to the desired number. If instead 20% more random variables are drawn in each iteration, then in expected terms only one iteration would be needed. Of course actually there will still be too few or already too many. Getting an extra random draw from numpy is cheap, I guess. Josef > > bye > > Nicky > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From josef.pktd at gmail.com Tue Aug 10 14:25:43 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 10 Aug 2010 14:25:43 -0400 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: <1281456532-sup-8954@theshire> Message-ID: On Tue, Aug 10, 2010 at 2:17 PM, wrote: > On Tue, Aug 10, 2010 at 2:08 PM, nicky van foreest wrote: >> Hi, >> >> Thanks for the nice feedback. >> >> This is a bit too short for me: >> >>> I would oversample if the rejection rate is around 20% >>> >>> Josef >> >> BTW: The rejection fraction of an ellipse is 1 - \pi/4 \approx 0.21. >> What is wrong with using the rejection method? > > Everything is fine. > I meant that in Erin's code the exact number of still missing random > variables are drawn in each iteration. > So it will take many iterations, with smaller and smaller sample sizes > to get to the desired number. > > If instead 20% more random variables are drawn in each iteration, then > in expected terms only one iteration would be needed. Of course > actually there will still be too few or already too many. Getting an > extra random draw from numpy is cheap, I guess. That's a nice dynamic optimization, dynamic programming problem: What's the optimal (expected cost minimizing) oversampling rate if there are still n random variables to draw and the success probability for each draw is 79%. Josef > > Josef > > >> >> bye >> >> Nicky >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > From ndbecker2 at gmail.com Tue Aug 10 14:49:02 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 10 Aug 2010 14:49:02 -0400 Subject: [SciPy-User] 2d inverse parabolic interpolation Message-ID: In 1-d, inverse parabolic interpolation is a useful operation. It is described in e.g., NRC sec 10.2 (Numerical Recipes in C). Is there an equivalent for 2d? Is there any scipy code? From charlesr.harris at gmail.com Tue Aug 10 15:19:23 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 10 Aug 2010 13:19:23 -0600 Subject: [SciPy-User] 2d inverse parabolic interpolation In-Reply-To: References: Message-ID: On Tue, Aug 10, 2010 at 12:49 PM, Neal Becker wrote: > In 1-d, inverse parabolic interpolation is a useful operation. It is > described in e.g., NRC sec 10.2 (Numerical Recipes in C). > > Is there an equivalent for 2d? Is there any scipy code? > > What do you want to do? What do you mean by equivalent? I suspect the answer is no since a map from 2d to 1d doesn't have an inverse and the inverse image is likely to be a manifold. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanforeest at gmail.com Tue Aug 10 15:29:04 2010 From: vanforeest at gmail.com (nicky van foreest) Date: Tue, 10 Aug 2010 21:29:04 +0200 Subject: [SciPy-User] random points within an ellipse In-Reply-To: References: <1281456532-sup-8954@theshire> Message-ID: >> If instead 20% more random variables are drawn in each iteration, then >> in expected terms only one iteration would be needed. Of course >> actually there will still be too few or already too many. Getting an >> extra random draw from numpy is cheap, I guess. I get it now. I assume that making random deviates is very cheap, as many very bright people spent lots of time on this issue. > > That's a nice dynamic optimization, dynamic programming problem: > What's the optimal (expected cost minimizing) oversampling rate if > there are still n random variables to draw and the success probability > for each draw is 79%. >From a practical point of view: the expected number of needed random deviates is N = n/p, where p is the success probability. I am inclined then to take somewhat more than N deviates, for instance three times the sigma which in this case is \sqrt{Np(1-p)}, and just throw away the surplus. The actual minimization problem is indeed interesting. Nicky From alexlz at lmn.pub.ro Tue Aug 10 19:30:12 2010 From: alexlz at lmn.pub.ro (Ioan-Alexandru Lazar) Date: Wed, 11 Aug 2010 02:30:12 +0300 Subject: [SciPy-User] Correctly compiling SciPy with UMFPACK support? Message-ID: <73e6e58816fa7aac519ec03b9017298f.squirrel@wm.lmn.pub.ro> Hello everyone, I know that SciPy vanilla support for UMFPACK is going to be deprecated due to license issues, so I hope I am not incorrectly directing my message. I have spent the last two days trying to build SciPy with UMFPACK. Normally, I would simply use the packages available from my distribution; however, I want SciPy to use a version of UMFPACK I am compiling myself using AMD's ACML (because I need multithreaded BLAS support). Unfortunately, there's something that I don't seem to be getting straight because regardless of how I do it, I end up with an error message. If I am using scikits.umfpack or scipy version 0.7.0, the error message I eventually receive is: Traceback (most recent call last): File "/home/alexlz/Projects/2010/llssolve/scipytest.py", line 16, in umf = scikits.umfpack.UmfpackContext("zi") File "/home/alexlz/.local/lib/python2.6/site-packages/scikits.umfpack-5.1.0-py2.6-linux-x86_64.egg/scikits/umfpack/umfpack.py", line 278, in __init__ self.control = nm.zeros( (UMFPACK_CONTROL, ), dtype = nm.double ) NameError: global name 'UMFPACK_CONTROL' is not defined If I am using scipy.sparse.linalg.dsolve.umfpack in scipy version 0.8.0, it complains that I have not compiled it with UMFPACK support. However, I am quite certain that I have done so -- __umfpack.so appears in the module directory and in fact, if I replace it with the __umfpack.so file that my distribution installed in its systemwide installation directory, it works; however, that one is not linked against ACML's BLAS. I am further baffled by this because building the UMFPACK module actually works -- I have made sure it doesn't fail, as I've read around the mailing list and noted that building it fails silently if it cannot be correctly compiled, being an optional extension of SciPy. I am compiling UMFPACK using the following options (snippets from UFconfig.mk): CFLAGS = -fPIC -m64 -O3 -fexceptions -L/home/alexlz/libraries/acml/gfortran64_mp/lib -I//home/alexlz/libraries/acml/gfortran64_mp/include -L/home/alexlz/src/CBLAS/lib/LINUX -I/home/alexlz/src/CBLAS/src # Fortran compiler (not normally required) F77 = gfortran F77FLAGS = -O F77LIB = # C and Fortran libraries LIB = -lm -lgomp -lgfortran BLAS = -lcblas -lacml_mp -lgfortran (Note: cblas is required because ACML doesn't follow the Netlib calling convention. Also, note that I'm not trying to use it to build the entire SciPy package -- I'm only interested in having multithreaded BLAS with UMFPACK) And the relevant entries in my site.cfg are: [amd] library_dirs = /home/alexlz/libraries include_dirs = /home/alexlz/includes amd_libs = amd # [umfpack] library_dirs = /home/alexlz/libraries include_dirs = /home/alexlz/includes umfpack_libs = umfpack Judging from the behavior (replacing the wrapper-built __umfpack.so with the one supplied by my systemwide distribution) I am assuming this is an issue with the way I am building it, especially considering that the __umfpack.so I'm building and the one in the systemwide installation have radically different sizes (mine is about 1M). However, I haven't found anything in the installation instruction -- I appear to be building it correctly for all I know. It doesn't work regardless of whether I am using the scikits.umfpack version or the one to be deprecated from scipy. Do you have any hint as to what I am doing wrong? I am currently out of ideas. Thank you, Alexandru Lazar, LMN, Politehnica University of Bucharest PS: If this is not an option, I would be grateful for any other suggestion about how to use a version of UMFPACK compiled with a multithreaded BLAS. Not necessarily the one from ACML (which is actually a bit complicated because they're not following the netlib calling convention so I'm forced to use CBLAS as well). If a readily-built binary is available, or there's any other approach to this, I'd be happy to hear about it. The background: I am trying to build my HPC application using Python, and this is the only obstacle I am currently encountering. Unfortunately, the very peculiar structure of the matrices I'm dealing with means that using SuperLU is not an option, *but* a multithreaded BLAS library and UMFPACK gives us a significant speedup on the 8-core nodes in our cluster. From lutz.maibaum at gmail.com Tue Aug 10 20:16:04 2010 From: lutz.maibaum at gmail.com (Lutz Maibaum) Date: Tue, 10 Aug 2010 17:16:04 -0700 Subject: [SciPy-User] Arpack changes from 0.7.2 to 0.8.0 Message-ID: <3901099D-9353-44FD-A382-A49AED99E5E2@gmail.com> Hello, It seems that the location of the arpack routines has changed from SciPy 0.7.2 to 0.8.0. In the older version, scipy.sparse.linalg.eigen was a module, with another submodule arpack that provided various functions. For example, one could do import scipy.sparse.linalg.eigen.arpack as arpack arpack.eigen (?) This no longer works in 0.8.0, because scipy.sparse.linalg.eigen is now a function. Were all the functions scipy.sparse.linalg.eigen.arpack.* simply moved to scipy.sparse.linalg? I couldn't find any information abut this in the 0.8.0 release notes. Of course I might also simply have a borked installation. Any insight wold be much appreciated. Thanks, Lutz From josef.pktd at gmail.com Tue Aug 10 20:26:42 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 10 Aug 2010 20:26:42 -0400 Subject: [SciPy-User] Arpack changes from 0.7.2 to 0.8.0 In-Reply-To: <3901099D-9353-44FD-A382-A49AED99E5E2@gmail.com> References: <3901099D-9353-44FD-A382-A49AED99E5E2@gmail.com> Message-ID: On Tue, Aug 10, 2010 at 8:16 PM, Lutz Maibaum wrote: > Hello, > > It seems that the location of the arpack routines has changed from SciPy 0.7.2 to 0.8.0. In the older version, scipy.sparse.linalg.eigen was a module, with another submodule arpack that provided various functions. For example, one could do > > ? ? ? ?import scipy.sparse.linalg.eigen.arpack as arpack > ? ? ? ?arpack.eigen (?) > > This no longer works in 0.8.0, because scipy.sparse.linalg.eigen is now a function. Were all the functions scipy.sparse.linalg.eigen.arpack.* simply moved to scipy.sparse.linalg? I couldn't find any information abut this in the 0.8.0 release notes. > > Of course I might also simply have a borked installation. Any insight wold be much appreciated. I think there is a problem that the function is shadowing the module. I remember I had problems with this before. I don't remember whether there was a work around. scipy.sparse.linalg.eigen.arpack.* are imported into the scipy.sparse.linalg namespace, as far as I can see. Josef > > Thanks, > > ?Lutz > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From lutz.maibaum at gmail.com Wed Aug 11 00:05:36 2010 From: lutz.maibaum at gmail.com (Lutz Maibaum) Date: Tue, 10 Aug 2010 21:05:36 -0700 Subject: [SciPy-User] Arpack changes from 0.7.2 to 0.8.0 In-Reply-To: References: <3901099D-9353-44FD-A382-A49AED99E5E2@gmail.com> Message-ID: On Tue, Aug 10, 2010 at 5:26 PM, wrote: > I think there is a problem that the function is shadowing the module. > I remember I had problems with this before. I don't remember whether > there was a work around. Thanks, that's what I thought. Was this done on purpose? It seems like this is an API change that breaks backwards compatibility (that's how I stumbled upon it). It would be nice if there was a warning in the release notes, and/or in the scipy.sparse.linalg documentation. Best, Lutz From josef.pktd at gmail.com Wed Aug 11 00:56:02 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 11 Aug 2010 00:56:02 -0400 Subject: [SciPy-User] Arpack changes from 0.7.2 to 0.8.0 In-Reply-To: References: <3901099D-9353-44FD-A382-A49AED99E5E2@gmail.com> Message-ID: On Wed, Aug 11, 2010 at 12:05 AM, Lutz Maibaum wrote: > On Tue, Aug 10, 2010 at 5:26 PM, ? wrote: >> I think there is a problem that the function is shadowing the module. >> I remember I had problems with this before. I don't remember whether >> there was a work around. > > Thanks, that's what I thought. Was this done on purpose? It seems like > this is an API change that breaks backwards compatibility (that's how > I stumbled upon it). It would be nice if there was a warning in the > release notes, and/or in the scipy.sparse.linalg documentation. reading the thread again from jan 11 [Numpy-discussion] TypeError: 'module' object is not callable I'm still not clear what was going on. The import into the sparse.linalg namespace was missing only for 2 months, after ARPACK had been temporarily removed, between r:5154 and r:5527, which it looks like was just when 0.7.0 was branched. I think I added the import in 5527 just to restore the previous state. I don't think this was an intended change in behavior. Josef > > Best, > > ?Lutz > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From schrabacke at web.de Wed Aug 11 04:01:45 2010 From: schrabacke at web.de (Jana Schulz) Date: Wed, 11 Aug 2010 10:01:45 +0200 (CEST) Subject: [SciPy-User] Interpolation in 3D with interp2d In-Reply-To: References: <1456192421.951467.1280690737282.JavaMail.fmail@mwmweb001>, Message-ID: <1379466308.2338901.1281513705129.JavaMail.fmail@mwmweb003> Hi everyone, thanks for all your answers. I still work on the problem. I tried the splines and its working but not linear, which ist the most important thing. It is working with the RBF too, but again not linear even if i set the RBF interpolation to 'linear'. I'm not sure if the griddata and interp2d works well on my example (or in general) because iPython gets stuck by running the interp2d command. Is there a bug in interp2d or griddata?? I was looking for a function which is similiar to the matlab's 'TriScatteredInterp' but there is nothing like this. I would be glad to have some other suggestions! -----Urspr?ngliche Nachricht----- Von: denis Gesendet: 06.08.2010 14:39:24 An: Jana Schulz Betreff: Re: Interpolation in 3D with interp2d >Hi Jana, hi Andreas, > > were the answers to your question of any use or have you given up : >( > >On Aug 1, 9:25?pm, Jana Schulz wrote: >> Hi, >> >> I'm trying to interpolate a 3D data (from the pic attached) with the > >Gruss > -- denis ___________________________________________________________ Neu: WEB.DE De-Mail - Einfach wie E-Mail, sicher wie ein Brief! Jetzt De-Mail-Adresse reservieren: https://produkte.web.de/go/demail02 From dagss at student.matnat.uio.no Wed Aug 11 04:10:24 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Aug 2010 10:10:24 +0200 Subject: [SciPy-User] Correctly compiling SciPy with UMFPACK support? In-Reply-To: <73e6e58816fa7aac519ec03b9017298f.squirrel@wm.lmn.pub.ro> References: <73e6e58816fa7aac519ec03b9017298f.squirrel@wm.lmn.pub.ro> Message-ID: <4C625AF0.6090908@student.matnat.uio.no> Ioan-Alexandru Lazar wrote: > Hello everyone, > > I know that SciPy vanilla support for UMFPACK is going to be deprecated > due to license issues, so I hope I am not incorrectly directing my > message. > > I have spent the last two days trying to build SciPy with UMFPACK. > Normally, I would simply use the packages available from my distribution; > however, I want SciPy to use a version of UMFPACK I am compiling myself > using AMD's ACML (because I need multithreaded BLAS support). > > Unfortunately, there's something that I don't seem to be getting straight > because regardless of how I do it, I end up with an error message. > > If I am using scikits.umfpack or scipy version 0.7.0, the error message I > eventually receive is: > > Traceback (most recent call last): > File "/home/alexlz/Projects/2010/llssolve/scipytest.py", line 16, in > > umf = scikits.umfpack.UmfpackContext("zi") > File > "/home/alexlz/.local/lib/python2.6/site-packages/scikits.umfpack-5.1.0-py2.6-linux-x86_64.egg/scikits/umfpack/umfpack.py", > line 278, in __init__ > self.control = nm.zeros( (UMFPACK_CONTROL, ), dtype = nm.double ) > NameError: global name 'UMFPACK_CONTROL' is not defined > > > If I am using scipy.sparse.linalg.dsolve.umfpack in scipy version 0.8.0, > it complains that I have not compiled it with UMFPACK support. > I just wanted to make you aware that the UMFPACK wrapper has a new home over at scikits (scikits.appspot.com). Perhaps you'll have better luch with that one... Haven't tried myself though. Dag Sverre From rigal at rapideye.de Wed Aug 11 04:29:47 2010 From: rigal at rapideye.de (Matthieu Rigal) Date: Wed, 11 Aug 2010 10:29:47 +0200 Subject: [SciPy-User] Best method to pick-up every N-th sample of an array In-Reply-To: References: Message-ID: <201008111029.47954.rigal@rapideye.de> Thanks Ryan and Ben, Indeed, this was a much better idea I had not explored... It is much faster and directly usable on numpy arrays ! I just did not thought about it and I was looking hopeless for a numpy subfunction ... Have a nice day, Best regards, Matthieu On Tuesday 10 August 2010 17:12:01 scipy-user-request at scipy.org wrote: > Would slicing the array help? > > >>> a = numpy.array([0, 1, 2, 3, 4, 5]) > >>> a[::2] # every second element > > array([0, 2, 4]) > > As a side benefit, you can save on memory because this 'slice' is actually > a view into the original array. > > I hope that helps, > > Ben Root RapidEye AG Molkenmarkt 30 14776 Brandenburg an der Havel Germany Follow us on Twitter! www.twitter.com/rapideye_ag Head Office/Sitz der Gesellschaft: Brandenburg an der Havel Management Board/Vorstand: Wolfgang G. Biedermann Chairman of Supervisory Board/Vorsitzender des Aufsichtsrates: Juergen Breitkopf Commercial Register/Handelsregister Potsdam HRB 17 796 Tax Number/Steuernummer: 048/100/00053 VAT-Ident-Number/Ust.-ID: DE 199331235 DIN EN ISO 9001 certified ************************************************************************* Diese E-Mail enthaelt vertrauliche und/oder rechtlich geschuetzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtuemlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet. The information in this e-mail is intended for the named recipients only. It may contain privileged and confidential information. If you have received this communication in error, any use, copying or dissemination of its contents is strictly prohibited. Please erase all copies of the message along with any included attachments and notify RapidEye AG or the sender immediately by telephone at the number indicated on this page. From ndbecker2 at gmail.com Wed Aug 11 06:55:02 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 11 Aug 2010 06:55:02 -0400 Subject: [SciPy-User] 2d inverse parabolic interpolation References: Message-ID: Charles R Harris wrote: > On Tue, Aug 10, 2010 at 12:49 PM, Neal Becker wrote: > >> In 1-d, inverse parabolic interpolation is a useful operation. It is >> described in e.g., NRC sec 10.2 (Numerical Recipes in C). >> >> Is there an equivalent for 2d? Is there any scipy code? >> >> > What do you want to do? What do you mean by equivalent? I suspect the > answer is no since a map from 2d to 1d doesn't have an inverse and the > inverse image is likely to be a manifold. > > Chuck In 1d, the operation is: giving 3 grid points of which the center point is the minimum of the 3, find the location of the minimum (a position not necessarily on the grid) (of a parabola passing through these 3 points). I was wondering if there is a similar operation for 2d, which would take perhaps the 8 grid points surrounding 1 point and find position of a minimum. The application is a time-frequency search. A set of points is given in 2d, quantized in time and frequency. Find the position (time and frequency) of the true minimum (approximately, and inexpensively). From ralf.gommers at googlemail.com Wed Aug 11 06:58:57 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 11 Aug 2010 18:58:57 +0800 Subject: [SciPy-User] Arpack changes from 0.7.2 to 0.8.0 In-Reply-To: References: <3901099D-9353-44FD-A382-A49AED99E5E2@gmail.com> Message-ID: On Wed, Aug 11, 2010 at 12:56 PM, wrote: > On Wed, Aug 11, 2010 at 12:05 AM, Lutz Maibaum > wrote: > > On Tue, Aug 10, 2010 at 5:26 PM, wrote: > >> I think there is a problem that the function is shadowing the module. > >> I remember I had problems with this before. I don't remember whether > >> there was a work around. > > > > Thanks, that's what I thought. Was this done on purpose? It seems like > > this is an API change that breaks backwards compatibility (that's how > > I stumbled upon it). It would be nice if there was a warning in the > > release notes, and/or in the scipy.sparse.linalg documentation. > > reading the thread again from jan 11 > [Numpy-discussion] TypeError: 'module' object is not callable > > I'm still not clear what was going on. > > The import into the sparse.linalg namespace was missing only for 2 > months, after ARPACK had been temporarily removed, between r:5154 and > r:5527, which it looks like was just when 0.7.0 was branched. > > I think I added the import in 5527 just to restore the previous state. > I don't think this was an intended change in behavior. > > This removal in r5088 made import scipy.sparse.linalg.eigen.arpack as arpack work, by accident. This was just before the 0.7.x branch was made, so there it worked. The issue is that arpack.py contains a function called eigen, and in eigen/__init__.py there is a line from arpack import * and then the function gets added to eigen.__all__, which then in turn is imported from one level up. This looks like a bad idea, so either the module of the function should be renamed. Or the functions in eigen.__all__ should not be made available in the sparse.linalg namespace. But I'd prefer renaming. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From denis-bz-gg at t-online.de Wed Aug 11 08:48:23 2010 From: denis-bz-gg at t-online.de (denis) Date: Wed, 11 Aug 2010 05:48:23 -0700 (PDT) Subject: [SciPy-User] Interpolation in 3D with interp2d In-Reply-To: <1379466308.2338901.1281513705129.JavaMail.fmail@mwmweb003> References: <1456192421.951467.1280690737282.JavaMail.fmail@mwmweb001>, <1379466308.2338901.1281513705129.JavaMail.fmail@mwmweb003> Message-ID: <627f01fa-f1c6-410e-9a40-47f49691b77e@i31g2000yqm.googlegroups.com> On Aug 11, 10:01?am, Jana Schulz wrote: > Hi everyone, > > thanks for all your answers. I still work on the problem. > I tried the splines and its working but not linear, which ist the most important thing. Jana, griddata( ... interp="linear" ) is much like TriScatteredInterp for ndim=2, interpolates a plane over each triangle in a Delaunay triangulation. (The default interp="nn" is smoother but not linear.) Is "linear" what you want ? Repeat, are you doing BOTH griddata() to a grid of rectangles then RectBivariateSpline() inside those rectangles ? When you interpolate the corner values 0 0 0 1 inside a square, 0 -- 0 | | 0 -- 1 RectBivariateSpline( x,y,image, kx=1, ky=1 ) gives the function x*y, which is of course BI linear -- linear along horizontal and vertical lines, quadratic along diagonals like x == y. If you could post half-a-dozen of your points (f, z, A) with an expected interpolated value or two, that would help. cheers -- denis From renato.fabbri at gmail.com Wed Aug 11 11:00:22 2010 From: renato.fabbri at gmail.com (Renato Fabbri) Date: Wed, 11 Aug 2010 12:00:22 -0300 Subject: [SciPy-User] how to fit a given pdf Message-ID: Dear All, help appreciated, thanks in advance. how do you fit a pdf you have with a given pdf (say gamma). with the file attached, you can go like: a=open("AC-010_ED-1m37F100P0.txt","rb") aa=a.read() aaa=aa[1:-1].split(",") data=[int(i) for i in aaa] if you do pylab.plot(data); pylab.show() The data is something like: ___|\___ It is my pdf (probability density function). how can i find the right parameters to make that fit with a gamma? if i was looking for a normal pdf, for example, i would just find mean and std and ask for the pdf. i've been playing with scipy.stats.distributions.gamma but i have not reached anything. we can extend the discussion further, but this is a good starting point. any idea? thanks! best of luck, rf -- GNU/Linux User #479299 skype: fabbri.renato -------------- next part -------------- [3, 1, 3, 2, 2, 2, 1, 1, 1, 1, 5, 4, 9, 16, 26, 51, 55, 83, 134, 228, 222, 202, 203, 180, 142, 114, 75, 55, 48, 41, 28, 26, 17, 13, 11, 14, 15, 14, 16, 9, 3, 3, 4, 3, 7, 4, 3, 3, 5, 7, 2, 2, 1, 8, 10, 14, 22, 11, 10, 12, 6, 3, 2, 1] From denis-bz-gg at t-online.de Wed Aug 11 11:20:08 2010 From: denis-bz-gg at t-online.de (denis) Date: Wed, 11 Aug 2010 08:20:08 -0700 (PDT) Subject: [SciPy-User] 2d inverse parabolic interpolation In-Reply-To: References: Message-ID: <216712ee-2353-44f2-b920-5f6e536b931b@p7g2000yqa.googlegroups.com> On Aug 10, 8:49?pm, Neal Becker wrote: > In 1-d, inverse parabolic interpolation is a useful operation. ?It is > described in e.g., NRC sec 10.2 (Numerical Recipes in C). > Is there an equivalent for 2d? ?Is there any scipy code? Neal, in 2d you need of course 6 xyz points to fit z = a + bx + cy + dx^2 + exy + fy^2 and even then may get a hyperbola, not a parabola or ellipse. Nonetheless the idea is interesting for convex optimization i.e. minimize 2 vars at once in a plane instead of 1 along a line (but, correct me, convex opt in general is a mighty hammer looking for nails ?) cheers -- denis From alexlz at lmn.pub.ro Wed Aug 11 11:44:29 2010 From: alexlz at lmn.pub.ro (Ioan-Alexandru Lazar) Date: Wed, 11 Aug 2010 18:44:29 +0300 Subject: [SciPy-User] Correctly compiling SciPy with UMFPACK support? In-Reply-To: References: Message-ID: Hello Dag, As I mentioned in my initial message, I get the same error with the UMFPACK wrapper in scikits. With my message being fairly long, this was probably hidden somewhere non-obvious so I'll reiterate this here -- I get exactly the same error message with the scikits UMFPACK wrapper. I suspect it's something about how I build UMFPACK because if I replace the __umfpack.so that gets built from source with the one supplied by Fedora's scipy package, it works (although that's not linked with the BLAS libraries I need). Perhaps someone who has managed to compile the scikits wrapper from source with a custom-built UMFPACK can shed some light on this? In the meantime, I think I'll try to look for Fedora's Scipy and suitesparse or UMFPACK SRPMs to see if I can find something interesting in there. Thanks for your time! Sincerely yours, Alexandru Lazar > I just wanted to make you aware that the UMFPACK wrapper has a new home > over at scikits (scikits.appspot.com). Perhaps you'll have better luch > with that one... > > Haven't tried myself though. > > Dag Sverre > > From jjstickel at vcn.com Wed Aug 11 12:14:05 2010 From: jjstickel at vcn.com (Jonathan Stickel) Date: Wed, 11 Aug 2010 10:14:05 -0600 Subject: [SciPy-User] how to fit a given pdf In-Reply-To: References: Message-ID: <4C62CC4D.7020604@vcn.com> On 8/11/10 09:00 , scipy-user-request at scipy.org wrote: > Date: Wed, 11 Aug 2010 12:00:22 -0300 > From: Renato Fabbri > Subject: [SciPy-User] how to fit a given pdf > To: Discussion of Numerical Python, > scipy-user at scipy.org > Message-ID: > > Content-Type: text/plain; charset="iso-8859-1" > > Dear All, > > help appreciated, thanks in advance. > > how do you fit a pdf you have with a given pdf (say gamma). > > with the file attached, you can go like: > > a=open("AC-010_ED-1m37F100P0.txt","rb") > aa=a.read() > aaa=aa[1:-1].split(",") > data=[int(i) for i in aaa] > > if you do pylab.plot(data); pylab.show() > > The data is something like: > ___|\___ > > It is my pdf (probability density function). > > how can i find the right parameters to make that fit with a gamma? > > if i was looking for a normal pdf, for example, i would just find mean > and std and ask for the pdf. > > i've been playing with scipy.stats.distributions.gamma but i have not > reached anything. > > we can extend the discussion further, but this is a good starting point. > > any idea? > I am not familiar with the scipy.stats module, and so I do not know what it can do for you. However, I would just generate a model gamma distribution from the mean and variance, just as for a normal distribution. The gamma distribution equation can be written as p(x) = p0/(b^a*Gamma(a))*x^(a-1)*exp(-x/b) assuming x starts at zero (x>=0) (http://en.wikipedia.org/wiki/Gamma_distribution). Then the the parameters a and b are related to the mean and variance by a = mean^2/var b = var/mean (I did the algebra quickly just now, and so you might want to double-check). p0 is the area under the distribution and may be simply 1 if your distribution is normalized. Hope this helps. Jonathan From josef.pktd at gmail.com Wed Aug 11 13:40:18 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 11 Aug 2010 13:40:18 -0400 Subject: [SciPy-User] how to fit a given pdf In-Reply-To: <4C62CC4D.7020604@vcn.com> References: <4C62CC4D.7020604@vcn.com> Message-ID: On Wed, Aug 11, 2010 at 12:14 PM, Jonathan Stickel wrote: > On 8/11/10 09:00 , scipy-user-request at scipy.org wrote: >> Date: Wed, 11 Aug 2010 12:00:22 -0300 >> From: Renato Fabbri >> Subject: [SciPy-User] how to fit a given pdf >> To: Discussion of Numerical Python, >> ? ? ? scipy-user at scipy.org >> Message-ID: >> ? ? ? >> Content-Type: text/plain; charset="iso-8859-1" >> >> Dear All, >> >> help appreciated, thanks in advance. >> >> how do you fit a pdf you have with a given pdf (say gamma). >> >> with the file attached, you can go like: >> >> a=open("AC-010_ED-1m37F100P0.txt","rb") >> aa=a.read() >> aaa=aa[1:-1].split(",") >> data=[int(i) for i in aaa] >> >> if you do pylab.plot(data); pylab.show() The first step is to try the fit method. If generic Maximum Likelihood estimation works with your data, then it's good. An example is at http://projects.scipy.org/scipy/ticket/832 If you want to fix the support, the lower bound then you could use the link on the ticket, current scipy trunk or statsmodels, to estimate. In some uncommitted changes to statsmodels, I'm preparing the standard errors for the MLE estimates using the Hessian of the log-likelihood, which I think will work in this case. Since your data are all integers, is there a discrete analog to the gamma distribution? I never checked this. I hope that helps, keep asking if there are additional issues. I just spend some time fighting with estimation for pareto and genpareto and I would like to get this to work "out-of-the-box". Josef >> >> The data is something like: >> ___|\___ >> >> It is my pdf (probability density function). >> >> how can i find the right parameters to make that fit with a gamma? >> >> if i was looking for a normal pdf, for example, i would just find mean >> and std and ask for the pdf. >> >> i've been playing with scipy.stats.distributions.gamma but i have not >> reached anything. >> >> we can extend the discussion further, but this is a good starting point. >> >> any idea? >> > > I am not familiar with the scipy.stats module, and so I do not know what > it can do for you. ?However, I would just generate a model gamma > distribution from the mean and variance, just as for a normal > distribution. ?The gamma distribution equation can be written as > > p(x) = p0/(b^a*Gamma(a))*x^(a-1)*exp(-x/b) > > assuming x starts at zero (x>=0) > (http://en.wikipedia.org/wiki/Gamma_distribution). ?Then the the > parameters a and b are related to the mean and variance by > > a = mean^2/var > b = var/mean > > (I did the algebra quickly just now, and so you might want to > double-check). ?p0 is the area under the distribution and may be simply > 1 if your distribution is normalized. > > Hope this helps. > > Jonathan > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From josef.pktd at gmail.com Wed Aug 11 13:48:10 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 11 Aug 2010 13:48:10 -0400 Subject: [SciPy-User] Arpack changes from 0.7.2 to 0.8.0 In-Reply-To: References: <3901099D-9353-44FD-A382-A49AED99E5E2@gmail.com> Message-ID: On Wed, Aug 11, 2010 at 6:58 AM, Ralf Gommers wrote: > > > On Wed, Aug 11, 2010 at 12:56 PM, wrote: >> >> On Wed, Aug 11, 2010 at 12:05 AM, Lutz Maibaum >> wrote: >> > On Tue, Aug 10, 2010 at 5:26 PM, ? wrote: >> >> I think there is a problem that the function is shadowing the module. >> >> I remember I had problems with this before. I don't remember whether >> >> there was a work around. >> > >> > Thanks, that's what I thought. Was this done on purpose? It seems like >> > this is an API change that breaks backwards compatibility (that's how >> > I stumbled upon it). It would be nice if there was a warning in the >> > release notes, and/or in the scipy.sparse.linalg documentation. >> >> reading the thread again from jan 11 >> [Numpy-discussion] TypeError: 'module' object is not callable >> >> I'm still not clear what was going on. >> >> The import into the sparse.linalg namespace was missing only for 2 >> months, after ARPACK had been temporarily removed, between r:5154 and >> r:5527, ? which it looks like was just when 0.7.0 was branched. >> >> I think I added the import in 5527 just to restore the previous state. >> I don't think this was an intended change in behavior. >> > This removal in r5088 made > ??? import scipy.sparse.linalg.eigen.arpack as arpack > work, by accident. This was just before the 0.7.x branch was made, so there > it worked. The issue is that arpack.py contains a function called eigen, and > in eigen/__init__.py there is a line > ??? from arpack import * > and then the function gets added to eigen.__all__, which then in turn is > imported from one level up. > > This looks like a bad idea, so either the module of the function should be > renamed. Or the functions in eigen.__all__ should not be made available in > the sparse.linalg namespace. But I'd prefer renaming. I also think renaming would be the best solution, renaming the module wouldn't break the current code (the damage with 0.7 is already done) ode is (or was) another case where the class/function shadows the module name. The previous reply was that ode should be rewritten. But since it doesn't look like that is happening soon, a renaming of the shadowed module might a better short tem solution. (this is from memory, I haven't checked recent changes to ode) Josef > > Cheers, > Ralf > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From ben.root at ou.edu Wed Aug 11 13:48:32 2010 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 11 Aug 2010 12:48:32 -0500 Subject: [SciPy-User] how to fit a given pdf In-Reply-To: <4C62CC4D.7020604@vcn.com> References: <4C62CC4D.7020604@vcn.com> Message-ID: On Wed, Aug 11, 2010 at 11:14 AM, Jonathan Stickel wrote: > On 8/11/10 09:00 , scipy-user-request at scipy.org wrote: > > Date: Wed, 11 Aug 2010 12:00:22 -0300 > > From: Renato Fabbri > > Subject: [SciPy-User] how to fit a given pdf > > To: Discussion of Numerical Python, > > scipy-user at scipy.org > > Message-ID: > > > > > > Content-Type: text/plain; charset="iso-8859-1" > > > > Dear All, > > > > help appreciated, thanks in advance. > > > > how do you fit a pdf you have with a given pdf (say gamma). > > > > with the file attached, you can go like: > > > > a=open("AC-010_ED-1m37F100P0.txt","rb") > > aa=a.read() > > aaa=aa[1:-1].split(",") > > data=[int(i) for i in aaa] > > > > if you do pylab.plot(data); pylab.show() > > > > The data is something like: > > ___|\___ > > > > It is my pdf (probability density function). > > > > how can i find the right parameters to make that fit with a gamma? > > > > if i was looking for a normal pdf, for example, i would just find mean > > and std and ask for the pdf. > > > > i've been playing with scipy.stats.distributions.gamma but i have not > > reached anything. > > > > we can extend the discussion further, but this is a good starting point. > > > > any idea? > > > > I am not familiar with the scipy.stats module, and so I do not know what > it can do for you. However, I would just generate a model gamma > distribution from the mean and variance, just as for a normal > distribution. The gamma distribution equation can be written as > > p(x) = p0/(b^a*Gamma(a))*x^(a-1)*exp(-x/b) > > assuming x starts at zero (x>=0) > (http://en.wikipedia.org/wiki/Gamma_distribution). Then the the > parameters a and b are related to the mean and variance by > > a = mean^2/var > b = var/mean > > (I did the algebra quickly just now, and so you might want to > double-check). p0 is the area under the distribution and may be simply > 1 if your distribution is normalized. > > Hope this helps. > > Jonathan > Note, those are the point estimators for gamma distribution, but it does not use maximum likeihood estimation (MLE). Using MLE, finding the estimator for alpha is tricky, requiring convergence. Probably best to use the fit module as Josef mentioned. However, there is a decent estimator for the estimator (I heard you liked estimation...) a = 0.5 / (ln(mean(X)) - mean(ln(X))) Note that if you have a zero value in your dataset this won't work. Also, the reference is not 100% clear if it is a base 10 log or not. I am pretty certain it is a natural log. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Aug 11 14:15:56 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 11 Aug 2010 14:15:56 -0400 Subject: [SciPy-User] how to fit a given pdf In-Reply-To: References: <4C62CC4D.7020604@vcn.com> Message-ID: On Wed, Aug 11, 2010 at 1:48 PM, Benjamin Root wrote: > On Wed, Aug 11, 2010 at 11:14 AM, Jonathan Stickel > wrote: >> >> On 8/11/10 09:00 , scipy-user-request at scipy.org wrote: >> > Date: Wed, 11 Aug 2010 12:00:22 -0300 >> > From: Renato Fabbri >> > Subject: [SciPy-User] how to fit a given pdf >> > To: Discussion of Numerical Python, >> > ? ? ? scipy-user at scipy.org >> > Message-ID: >> > ? ? ? >> > Content-Type: text/plain; charset="iso-8859-1" >> > >> > Dear All, >> > >> > help appreciated, thanks in advance. >> > >> > how do you fit a pdf you have with a given pdf (say gamma). >> > >> > with the file attached, you can go like: >> > >> > a=open("AC-010_ED-1m37F100P0.txt","rb") >> > aa=a.read() >> > aaa=aa[1:-1].split(",") >> > data=[int(i) for i in aaa] >> > >> > if you do pylab.plot(data); pylab.show() >> > >> > The data is something like: >> > ___|\___ >> > >> > It is my pdf (probability density function). >> > >> > how can i find the right parameters to make that fit with a gamma? >> > >> > if i was looking for a normal pdf, for example, i would just find mean >> > and std and ask for the pdf. >> > >> > i've been playing with scipy.stats.distributions.gamma but i have not >> > reached anything. >> > >> > we can extend the discussion further, but this is a good starting point. >> > >> > any idea? >> > >> >> I am not familiar with the scipy.stats module, and so I do not know what >> it can do for you. ?However, I would just generate a model gamma >> distribution from the mean and variance, just as for a normal >> distribution. ?The gamma distribution equation can be written as >> >> p(x) = p0/(b^a*Gamma(a))*x^(a-1)*exp(-x/b) >> >> assuming x starts at zero (x>=0) >> (http://en.wikipedia.org/wiki/Gamma_distribution). ?Then the the >> parameters a and b are related to the mean and variance by >> >> a = mean^2/var >> b = var/mean >> >> (I did the algebra quickly just now, and so you might want to >> double-check). ?p0 is the area under the distribution and may be simply >> 1 if your distribution is normalized. >> >> Hope this helps. >> >> Jonathan > > Note, those are the point estimators for gamma distribution, but it does not > use maximum likeihood estimation (MLE).? Using MLE, finding the estimator > for alpha is tricky, requiring convergence.? Probably best to use the fit > module as Josef mentioned.? However, there is a decent estimator for the > estimator (I heard you liked estimation...) > > a = 0.5 / (ln(mean(X)) - mean(ln(X))) > > Note that if you have a zero value in your dataset this won't work.? Also, > the reference is not 100% clear if it is a base 10 log or not.? I am pretty > certain it is a natural log. Travis also added a different estimator in http://projects.scipy.org/scipy/browser/trunk/scipy/stats/distributions.py#L2867 I never looked specifically at gamma, only as example for generic MLE, and have no idea what this estimator is. Ben, do you have the reference or know what type of estimators your's is? (I like references, the R package POT has 17 estimators for genpareto, and I have 20 pdf files for pareto/genpareto/genextreme open) Josef > > Ben Root > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From ben.root at ou.edu Wed Aug 11 14:29:36 2010 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 11 Aug 2010 13:29:36 -0500 Subject: [SciPy-User] how to fit a given pdf In-Reply-To: References: <4C62CC4D.7020604@vcn.com> Message-ID: On Wed, Aug 11, 2010 at 1:15 PM, wrote: > On Wed, Aug 11, 2010 at 1:48 PM, Benjamin Root wrote: > > On Wed, Aug 11, 2010 at 11:14 AM, Jonathan Stickel > > wrote: > >> > >> On 8/11/10 09:00 , scipy-user-request at scipy.org wrote: > >> > Date: Wed, 11 Aug 2010 12:00:22 -0300 > >> > From: Renato Fabbri > >> > Subject: [SciPy-User] how to fit a given pdf > >> > To: Discussion of Numerical Python, > >> > scipy-user at scipy.org > >> > Message-ID: > >> > > > > >> > Content-Type: text/plain; charset="iso-8859-1" > >> > > >> > Dear All, > >> > > >> > help appreciated, thanks in advance. > >> > > >> > how do you fit a pdf you have with a given pdf (say gamma). > >> > > >> > with the file attached, you can go like: > >> > > >> > a=open("AC-010_ED-1m37F100P0.txt","rb") > >> > aa=a.read() > >> > aaa=aa[1:-1].split(",") > >> > data=[int(i) for i in aaa] > >> > > >> > if you do pylab.plot(data); pylab.show() > >> > > >> > The data is something like: > >> > ___|\___ > >> > > >> > It is my pdf (probability density function). > >> > > >> > how can i find the right parameters to make that fit with a gamma? > >> > > >> > if i was looking for a normal pdf, for example, i would just find mean > >> > and std and ask for the pdf. > >> > > >> > i've been playing with scipy.stats.distributions.gamma but i have not > >> > reached anything. > >> > > >> > we can extend the discussion further, but this is a good starting > point. > >> > > >> > any idea? > >> > > >> > >> I am not familiar with the scipy.stats module, and so I do not know what > >> it can do for you. However, I would just generate a model gamma > >> distribution from the mean and variance, just as for a normal > >> distribution. The gamma distribution equation can be written as > >> > >> p(x) = p0/(b^a*Gamma(a))*x^(a-1)*exp(-x/b) > >> > >> assuming x starts at zero (x>=0) > >> (http://en.wikipedia.org/wiki/Gamma_distribution). Then the the > >> parameters a and b are related to the mean and variance by > >> > >> a = mean^2/var > >> b = var/mean > >> > >> (I did the algebra quickly just now, and so you might want to > >> double-check). p0 is the area under the distribution and may be simply > >> 1 if your distribution is normalized. > >> > >> Hope this helps. > >> > >> Jonathan > > > > Note, those are the point estimators for gamma distribution, but it does > not > > use maximum likeihood estimation (MLE). Using MLE, finding the estimator > > for alpha is tricky, requiring convergence. Probably best to use the fit > > module as Josef mentioned. However, there is a decent estimator for the > > estimator (I heard you liked estimation...) > > > > a = 0.5 / (ln(mean(X)) - mean(ln(X))) > > > > Note that if you have a zero value in your dataset this won't work. > Also, > > the reference is not 100% clear if it is a base 10 log or not. I am > pretty > > certain it is a natural log. > > Travis also added a different estimator in > > http://projects.scipy.org/scipy/browser/trunk/scipy/stats/distributions.py#L2867 > I never looked specifically at gamma, only as example for generic MLE, > and have no idea what this estimator is. > > Ben, do you have the reference or know what type of estimators your's is? > > (I like references, the R package POT has 17 estimators for genpareto, > and I have 20 pdf files for pareto/genpareto/genextreme open) > > Josef > > Josef, The reference for that particular estimator is http://research.microsoft.com/en-us/um/people/minka/papers/minka-gamma.pdf There is another estimator that I have used that came from a very good 'statistics for meteorologists' book, but I don't have that on me. I can get it to you tomorrow, though. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Aug 11 14:49:25 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 11 Aug 2010 14:49:25 -0400 Subject: [SciPy-User] how to fit a given pdf In-Reply-To: References: <4C62CC4D.7020604@vcn.com> Message-ID: On Wed, Aug 11, 2010 at 2:29 PM, Benjamin Root wrote: > On Wed, Aug 11, 2010 at 1:15 PM, wrote: >> >> On Wed, Aug 11, 2010 at 1:48 PM, Benjamin Root wrote: >> > On Wed, Aug 11, 2010 at 11:14 AM, Jonathan Stickel >> > wrote: >> >> >> >> On 8/11/10 09:00 , scipy-user-request at scipy.org wrote: >> >> > Date: Wed, 11 Aug 2010 12:00:22 -0300 >> >> > From: Renato Fabbri >> >> > Subject: [SciPy-User] how to fit a given pdf >> >> > To: Discussion of Numerical Python, >> >> > ? ? ? scipy-user at scipy.org >> >> > Message-ID: >> >> > ? ? ? >> >> > Content-Type: text/plain; charset="iso-8859-1" >> >> > >> >> > Dear All, >> >> > >> >> > help appreciated, thanks in advance. >> >> > >> >> > how do you fit a pdf you have with a given pdf (say gamma). >> >> > >> >> > with the file attached, you can go like: >> >> > >> >> > a=open("AC-010_ED-1m37F100P0.txt","rb") >> >> > aa=a.read() >> >> > aaa=aa[1:-1].split(",") >> >> > data=[int(i) for i in aaa] >> >> > >> >> > if you do pylab.plot(data); pylab.show() >> >> > >> >> > The data is something like: >> >> > ___|\___ >> >> > >> >> > It is my pdf (probability density function). >> >> > >> >> > how can i find the right parameters to make that fit with a gamma? >> >> > >> >> > if i was looking for a normal pdf, for example, i would just find >> >> > mean >> >> > and std and ask for the pdf. >> >> > >> >> > i've been playing with scipy.stats.distributions.gamma but i have not >> >> > reached anything. >> >> > >> >> > we can extend the discussion further, but this is a good starting >> >> > point. >> >> > >> >> > any idea? >> >> > >> >> >> >> I am not familiar with the scipy.stats module, and so I do not know >> >> what >> >> it can do for you. ?However, I would just generate a model gamma >> >> distribution from the mean and variance, just as for a normal >> >> distribution. ?The gamma distribution equation can be written as >> >> >> >> p(x) = p0/(b^a*Gamma(a))*x^(a-1)*exp(-x/b) >> >> >> >> assuming x starts at zero (x>=0) >> >> (http://en.wikipedia.org/wiki/Gamma_distribution). ?Then the the >> >> parameters a and b are related to the mean and variance by >> >> >> >> a = mean^2/var >> >> b = var/mean >> >> >> >> (I did the algebra quickly just now, and so you might want to >> >> double-check). ?p0 is the area under the distribution and may be simply >> >> 1 if your distribution is normalized. >> >> >> >> Hope this helps. >> >> >> >> Jonathan >> > >> > Note, those are the point estimators for gamma distribution, but it does >> > not >> > use maximum likeihood estimation (MLE).? Using MLE, finding the >> > estimator >> > for alpha is tricky, requiring convergence.? Probably best to use the >> > fit >> > module as Josef mentioned.? However, there is a decent estimator for the >> > estimator (I heard you liked estimation...) >> > >> > a = 0.5 / (ln(mean(X)) - mean(ln(X))) >> > >> > Note that if you have a zero value in your dataset this won't work. >> > Also, >> > the reference is not 100% clear if it is a base 10 log or not.? I am >> > pretty >> > certain it is a natural log. >> >> Travis also added a different estimator in >> >> http://projects.scipy.org/scipy/browser/trunk/scipy/stats/distributions.py#L2867 >> I never looked specifically at gamma, only as example for generic MLE, >> and have no idea what this estimator is. >> >> Ben, do you have the reference or know what type of estimators your's is? >> >> (I like references, the R package POT has 17 estimators for genpareto, >> and I have 20 pdf files for pareto/genpareto/genextreme open) >> >> Josef >> > > Josef, > > The reference for that particular estimator is > http://research.microsoft.com/en-us/um/people/minka/papers/minka-gamma.pdf thanks, that also describes the current scipy trunk estimator. Josef > > There is another estimator that I have used that came from a very good > 'statistics for meteorologists' book, but I don't have that on me.? I can > get it to you tomorrow, though. > > Ben Root > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From brennan.williams at visualreservoir.com Wed Aug 11 18:16:40 2010 From: brennan.williams at visualreservoir.com (Brennan Williams) Date: Thu, 12 Aug 2010 10:16:40 +1200 Subject: [SciPy-User] scipy.interpolate.Rbf Message-ID: <4C632148.3070504@visualreservoir.com> I'm coding up using Rbf where the number of coordinates x,y,z.... will vary from one dataset to another. So I may have... rbfi=Rbf(x,y,z,d) for one dataset and then.... rbfi=Rbf(u,v,w,x,y,z,d) for another This is all inside a Traits GUI app rather than in a user-editable script. So how should I do this? Thanks Brennan From robert.kern at gmail.com Wed Aug 11 18:24:52 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 11 Aug 2010 17:24:52 -0500 Subject: [SciPy-User] scipy.interpolate.Rbf In-Reply-To: <4C632148.3070504@visualreservoir.com> References: <4C632148.3070504@visualreservoir.com> Message-ID: On Wed, Aug 11, 2010 at 17:16, Brennan Williams wrote: > ?I'm coding up using Rbf where the number of coordinates x,y,z.... will > vary from one dataset to another. > So I may have... > > rbfi=Rbf(x,y,z,d) for one dataset > > and then.... > > rbfi=Rbf(u,v,w,x,y,z,d) for another > > This is all inside a Traits GUI app rather than in a user-editable script. > So how should I do this? Collect the arguments in a list and use Rbf(*args). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From chico at cnpmf.embrapa.br Thu Aug 12 10:07:08 2010 From: chico at cnpmf.embrapa.br (chico at cnpmf.embrapa.br) Date: Thu, 12 Aug 2010 11:07:08 -0300 Subject: [SciPy-User] scipy.interpolate.Rbf In-Reply-To: References: <4C632148.3070504@visualreservoir.com> Message-ID: <28a6ba60a6d02a77e836321bb8e636ac.squirrel@www.cnpmf.embrapa.br> Dear All, Is there a function for interpolation using non-regular spaced data or data with missing points? Also, what would recommend for geostatistics? Thank you, Chico L. > On Wed, Aug 11, 2010 at 17:16, Brennan Williams > wrote: >> ??I'm coding up using Rbf where the number of coordinates x,y,z.... will >> vary from one dataset to another. >> So I may have... >> >> rbfi=Rbf(x,y,z,d) for one dataset >> >> and then.... >> >> rbfi=Rbf(u,v,w,x,y,z,d) for another >> >> This is all inside a Traits GUI app rather than in a user-editable >> script. >> So how should I do this? > > Collect the arguments in a list and use Rbf(*args). > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ?? -- Umberto Eco > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From gdmcbain at freeshell.org Wed Aug 11 20:00:54 2010 From: gdmcbain at freeshell.org (Geordie McBain) Date: Thu, 12 Aug 2010 10:00:54 +1000 Subject: [SciPy-User] [Numpy-discussion] how to fit a given pdf In-Reply-To: References: Message-ID: 2010/8/12 Renato Fabbri : > Dear All, > > help appreciated, thanks in advance. > > how do you fit a pdf you have with a given pdf (say gamma). > > with the file attached, you can go like: > > a=open("AC-010_ED-1m37F100P0.txt","rb") > aa=a.read() > aaa=aa[1:-1].split(",") > data=[int(i) for i in aaa] > > if you do pylab.plot(data); pylab.show() > > The data is something like: > ___|\___ > > It is my pdf (probability density function). > > how can i find the right parameters to make that fit with a gamma? > > if i was looking for a normal pdf, for example, i would just find mean > and std and ask for the pdf. > > i've been playing with scipy.stats.distributions.gamma but i have not > reached anything. > > we can extend the discussion further, but this is a good starting point. > > any idea? A general point on fitting empirical probability density functions is that it is often much easier to fit the cumulative distribution function instead. For one thing, this means you don't have to decide on the intervals of the bins in the histogram. For another, it's actually often the cdf that is more related to the final answer (though I don't know your application, of course). Here's a quote. `So far the discussion of plots of distributions has emphasized frequency (or probability) vs. size plots, whereas for many applications cumulative plots are more important. Cumulative curves are produced by plotting the percentage of particles (or weight, volume, or surface) having particle diameters greater than (or less than) a given particle size against the particle size. ? Such curves have the advantage over histograms for plotting data that the class interval is eliminated, and they can be used to represent data which are obtained in classified form having unequal class intervals' (Cadle, R. D. 1965. Particle Size. New York: Reinhold Publishing Corporation, pp. 38-39) Once you've got your empirical cdf, the problem reduces to one of nonlinear curve fitting, for whichever theoretical distribution you like. For a tutorial on nonlinear curve fitting, see scipy.optimize.leastsq at http://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html. You could of course use this approach for the pdf too, but I fancy the cdf result will be more robust. On the other hand, if you want something like your `mean and variance' approach to fitting normal distributions, you could still compare your mean and variance with the known values for the Gamma distribution (available e.g. on its Wikipedia page) and back-out the two parameters of the distribution from them. I'm not too sure how well this will work, but it's pretty easy. Another idea occurs to me and is about as easy as this is to compute the two parameters of the Gamma distribution by collocation with the empirical cdf; i.e. pick two quantiles, e.g. 0.25 and 0.75, or whatever, and get two equations for the two unknown parameters by insisting on the Gamma cdf agreeing with the empirical for these quantiles. This might be more robust than the mean & variance approach, but I haven't tried either. Good luck! From schrabacke at web.de Thu Aug 12 16:35:09 2010 From: schrabacke at web.de (Jana Schulz) Date: Thu, 12 Aug 2010 22:35:09 +0200 (CEST) Subject: [SciPy-User] Interpolation in 3D with interp2d In-Reply-To: <627f01fa-f1c6-410e-9a40-47f49691b77e@i31g2000yqm.googlegroups.com> References: <1456192421.951467.1280690737282.JavaMail.fmail@mwmweb001>, <1379466308.2338901.1281513705129.JavaMail.fmail@mwmweb003>, <627f01fa-f1c6-410e-9a40-47f49691b77e@i31g2000yqm.googlegroups.com> Message-ID: <1506300512.361757.1281645309362.JavaMail.fmail@mwmweb001> Hi, RectBivariateSpline() can not handle randomly spaced data (that's what the error message says). The option 'linear' in griddata works only for constant spacing data. I attached the surface plot of my data and the txt-file including my data. As you see the surface plot works pretty well. All I want is to get any z-value for a given x,y-position. Sounds easy, but a didn't find any function besides interp2d. Here is the error message: C:\Python26\lib\site-packages\scipy\interpolate\interpolate.py in __init__(self, x, y, z, kind, copy, bounds_error, fill_value) 111 raise ValueError("Unsupported interpolation type.") 112 --> 113 self.tck = fitpack.bisplrep(self.x, self.y, self.z, kx=kx, ky=ky , s=0.) 114 115 def __call__(self,x,y,dx=0,dy=0): C:\Python26\lib\site-packages\scipy\interpolate\fitpack.py in bisplrep(x, y, z, w, xb, xe, yb, ye, kx, ky, task, s, eps, tx, ty, full_output, nxest, nyest, quie t) 780 raise OverflowError("Too many data points to interpolate") 781 tx,ty,c,o = _fitpack._surfit(x,y,z,w,xb,xe,yb,ye,kx,ky,task,s,eps, --> 782 tx,ty,nxest,nyest,wrk,lwrk1,lwrk2) 783 _curfit_cache['tx']=tx 784 _curfit_cache['ty']=ty MemoryError: WARNING: Failure executing file: any idea what it means? -----Urspr?ngliche Nachricht----- Von: denis Gesendet: 11.08.2010 14:48:23 An: scipy-user at scipy.org Betreff: Re: [SciPy-User] Interpolation in 3D with interp2d >On Aug 11, 10:01?am, Jana Schulz wrote: >> Hi everyone, >> >> thanks for all your answers. I still work on the problem. >> I tried the splines and its working but not linear, which ist the most important thing. > >Jana, > >griddata( ... interp="linear" ) is much like TriScatteredInterp for >ndim=2, >interpolates a plane over each triangle in a Delaunay triangulation. >(The default interp="nn" is smoother but not linear.) >Is "linear" what you want ? > >Repeat, are you doing BOTH griddata() to a grid of rectangles >then RectBivariateSpline() inside those rectangles ? >When you interpolate the corner values 0 0 0 1 inside a square, > 0 -- 0 > | | > 0 -- 1 >RectBivariateSpline( x,y,image, kx=1, ky=1 ) gives the function x*y, >which is of course BI linear -- linear along horizontal and vertical >lines, >quadratic along diagonals like x == y. > >If you could post half-a-dozen of your points (f, z, A) >with an expected interpolated value or two, that would help. > >cheers > -- denis >_______________________________________________ >SciPy-User mailing list >SciPy-User at scipy.org >http://mail.scipy.org/mailman/listinfo/scipy-user ___________________________________________________________ Neu: WEB.DE De-Mail - Einfach wie E-Mail, sicher wie ein Brief! Jetzt De-Mail-Adresse reservieren: https://produkte.web.de/go/demail02 -------------- next part -------------- A non-text attachment was scrubbed... Name: accuracy-map_v2.txt Type: application/octet-stream Size: 11171 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: surface.png Type: application/octet-stream Size: 163391 bytes Desc: not available URL: From emcquinn at cs.ucsd.edu Thu Aug 12 20:24:00 2010 From: emcquinn at cs.ucsd.edu (Emmett McQuinn) Date: Thu, 12 Aug 2010 17:24:00 -0700 (PDT) Subject: [SciPy-User] Cross platform animated plot with widgets Message-ID: <499214819.414661281659040499.JavaMail.root@csemailbox.ucsd.edu> Hello, I've been trying to create an animated plot with matplotlib that supports parameter updating with a slider. I've been able to have animation, and also create a slider (using matplotlib.widgets), but not both at the same time. I've tried various permutations and it seems exclusively either the graph animates or the widgets work and not both. Is there a good method to provide a simple lightweight GUI that provides both animation and sliders for Linux, Windows, and Mac? Thanks, Emmett McQuinn From fbouffard at gmail.com Fri Aug 13 15:51:38 2010 From: fbouffard at gmail.com (=?ISO-8859-1?Q?Fran=E7ois_Bouffard?=) Date: Fri, 13 Aug 2010 15:51:38 -0400 Subject: [SciPy-User] Parallel operations on the columns of a numpy array Message-ID: I chose Python and numpy for a project of mine which had to be programmed really quickly, and this solution really shines in terms of development time. However I encountered a speed bottleneck when trying to perform "embarassingly parallel" operations on the columns of a medium-sized numpy array. More specifically, I need to perform fftshift() and fft() on every column of this array, as quickly as possible. I cannot zero-pad the array to the next power-of-two, so of course I use the fft(x, None, 0) syntax. Comparing my results to a Matlab implementation, I quickly realized that I suffered a factor-C penality, where C is the number of CPUs available on the machine I'm working on. Whereas in Matlab, all CPUs are used at 100%, only one (in average) is used in Python. This can be a major drawback since the production machine is a dual-quadcore computer. I tested numpy.fft.fft and scipy.fftpack.fft and quickly found a important speed difference in favor of fftpack; however multiprocessor support is still not there. I understand that FFTW is not distributed with numpy anymore, and using a wrapper for that library may be an option. I also understand that scipy and numpy must be built for ATLAS, LAPACK or Intel MKL to maximize the performances of matrix operations. I have not come around to building numpy with ATLAS support yet, but I have installed Christoph Gholke's MKL-enabled packages. With these, multi-processor support seems to be active for matrix operations such as dot(); however it doesn't help, it seems, at splitting a a common operation across columns of an array. Another attempt at a speed-up consisted in using the parallel_map function found in the handythread module graciously provided in the Multithreading Cookbook on scipy.org. I simply split the array in as many blocks as there are processors, and feed that to parallel_map. While it seems that more than one processor is used in average, the speed gain is just above negligible and still quite far from Matlab's implementation. Maybe the overhead of this method is killing me for arrays that are not that large (typical size is 100-by-10,000) So I'm just looking for any hints... Would FFTW help? Would ATLAS offer better performances in that regard than Intel's MKL? Is there a better way to parallelize column operations? Thanks for any idea (and sorry for this rather long post). Fran?ois From sturla at molden.no Fri Aug 13 16:33:09 2010 From: sturla at molden.no (Sturla Molden) Date: Fri, 13 Aug 2010 22:33:09 +0200 Subject: [SciPy-User] Parallel operations on the columns of a numpy array In-Reply-To: References: Message-ID: <6283a674b0dc3adff5eafa736a75ec9c.squirrel@webmail.uio.no> > So I'm just looking for any hints... Would FFTW help? Would ATLAS > offer better performances in that regard than Intel's MKL? Is there a > better way to parallelize column operations? The FFTs in FFTW, MKL and ACML uses multithreading out of the box. NumPy's fftpack_lite does not. NumPy's FFTs can be multithreaded e.g. with threading.Thread if the GIL was released. I once opened a ticket for this, and supplied the required code, but it's not in NumPy trunk, nor in the "unified diff". Basically we must put Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS pragmas inside fftpack_litemodule.c. The library is thread safe. It is easy to recompile fftpack_lite to use multithreading even automatically (OpenMP). Also note that FFTW, MKL or ACML will give you faster FFTs than FFTPACK, multithreaded or not. So you will in any case be better off with one of these. FFTW is GPL. MKL and ACML are not free (as in speech), but ACML can be used for free (as in beer). ATLAS will not help as there are no FFT in ATLAS. Nor will compiling NumPy against MKL or ACML help, as the FFTs are not used by NumPy. http://projects.scipy.org/numpy/ticket/1055 http://svn.scipy.org/svn/numpy/trunk/numpy/fft/fftpack_litemodule.c Note that when you use FFT libraries like FFTW, arrays must be promoted to 16 byte boundaries (that's what fftw_malloc does). We can do that from NumPy, e.g. see here: http://blog.gmane.org/gmane.comp.python.scientific.user/month=20090301/page=15 P.S. SciPy's fftpack is not thread safe due to the way info arrays are cached, and cannot be freely threaded in its current state. But free threading is OK for NumPy's fftpack_lite. Regards, Sturla From lutz.maibaum at gmail.com Fri Aug 13 16:47:33 2010 From: lutz.maibaum at gmail.com (Lutz Maibaum) Date: Fri, 13 Aug 2010 13:47:33 -0700 Subject: [SciPy-User] Arpack changes from 0.7.2 to 0.8.0 In-Reply-To: References: <3901099D-9353-44FD-A382-A49AED99E5E2@gmail.com> Message-ID: On Aug 11, 2010, at 10:48 AM, josef.pktd at gmail.com wrote: > On Wed, Aug 11, 2010 at 6:58 AM, ralf.gommers at googlemail.com wrote: >> >> This removal in r5088 made >> import scipy.sparse.linalg.eigen.arpack as arpack >> work, by accident. This was just before the 0.7.x branch was made, so there >> it worked. The issue is that arpack.py contains a function called eigen, and >> in eigen/__init__.py there is a line >> from arpack import * >> and then the function gets added to eigen.__all__, which then in turn is >> imported from one level up. >> >> This looks like a bad idea, so either the module of the function should be >> renamed. Or the functions in eigen.__all__ should not be made available in >> the sparse.linalg namespace. But I'd prefer renaming. > > I also think renaming would be the best solution, renaming the module > wouldn't break the current code (the damage with 0.7 is already done) > > ode is (or was) another case where the class/function shadows the > module name. The previous reply was that ode should be rewritten. But > since it doesn't look like that is happening soon, a renaming of the > shadowed module might a better short tem solution. (this is from > memory, I haven't checked recent changes to ode) Thanks for looking into this. I created a ticket for this issue: http://projects.scipy.org/scipy/ticket/1263 Best, Lutz From jake.biesinger at gmail.com Fri Aug 13 21:17:45 2010 From: jake.biesinger at gmail.com (Jacob Biesinger) Date: Fri, 13 Aug 2010 18:17:45 -0700 Subject: [SciPy-User] Efficient iteration over a scipy.sparse.csc_matrix? Message-ID: Hi! I have some large sparse matrices (csc_matrix since they are 1e10 rows x 4 columns). I expect that some (many) of the rows are completely empty so is there an efficient way to iterate over *only* the rows that have entries? Perhaps something along the lines of useRows=set(ma.indices) ? Thanks! -- Jake Biesinger Graduate Student Xie Lab, UC Irvine (949) 231-7587 -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Aug 13 22:28:17 2010 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 13 Aug 2010 19:28:17 -0700 Subject: [SciPy-User] Efficient iteration over a scipy.sparse.csc_matrix? In-Reply-To: References: Message-ID: On Fri, Aug 13, 2010 at 6:17 PM, Jacob Biesinger wrote: > Hi! > I have some large sparse matrices (csc_matrix since they are 1e10 rows x 4 > columns). ?I expect that some (many) of the rows are completely empty so is > there an efficient way to iterate over *only* the rows that have entries? > ?Perhaps something along the lines of > useRows=set(ma.indices) ? Not sure what kind of API you're thinking of there, but the CSC format is not *too* hard to work with directly, once you've wrapped your head around it. See any description on the web and the matrix attributes ma.data, ma.indices, ma.indptr. (In particular, note that ma.data is the non-zero values in your matrix, ma.indices is the same length as ma.data and gives the row index where each corresponding data point resides, and ma.indptr encodes column information in a slightly more complicated way. So you can probably slice ma.data/ma.indices and then iterate over them.) -- Nathaniel From timmichelsen at gmx-topmail.de Sat Aug 14 06:38:29 2010 From: timmichelsen at gmx-topmail.de (Tim Michelsen) Date: Sat, 14 Aug 2010 12:38:29 +0200 Subject: [SciPy-User] Datasets in Scipy Code Message-ID: Hello, I am wondering what happened to the Dataset for scipy: design proposal (at old scikits page: http://www.scipy.org/scipy/scikits/wiki/DataSets). Where can I find the current status of this initiative? I have seen that some is ued in statsmodels at: http://bazaar.launchpad.net/~scipystats/statsmodels/trunk/files/head%3A/scikits/statsmodels/datasets/ I would appreciate any pointer. Thanks for your help and kind regards. Timmie From jsseabold at gmail.com Sat Aug 14 13:28:42 2010 From: jsseabold at gmail.com (Skipper Seabold) Date: Sat, 14 Aug 2010 13:28:42 -0400 Subject: [SciPy-User] Datasets in Scipy Code In-Reply-To: References: Message-ID: On Sat, Aug 14, 2010 at 6:38 AM, Tim Michelsen wrote: > Hello, > I am wondering what happened to the Dataset for scipy: design proposal > (at old scikits page: http://www.scipy.org/scipy/scikits/wiki/DataSets). > > Where can I find the current status of this initiative? > > I have seen that some is ued in statsmodels at: > http://bazaar.launchpad.net/~scipystats/statsmodels/trunk/files/head%3A/scikits/statsmodels/datasets/ > > I would appreciate any pointer. > > Thanks for your help and kind regards. > Yeah, I think what we have in statsmodels is about as far as it's gotten. I rewrote a lot of the code and David's NEP at the beginning of the summer based on our needs to keep it maintanable and flexible. There is also an incarnation in scikits-learn with a few differences, but we tried to keep them similar. It might make sense to combine the two at some point and distribute as a standalone scikit. Skipper From timmichelsen at gmx-topmail.de Sat Aug 14 16:31:27 2010 From: timmichelsen at gmx-topmail.de (Tim Michelsen) Date: Sat, 14 Aug 2010 22:31:27 +0200 Subject: [SciPy-User] Datasets in Scipy Code In-Reply-To: References: Message-ID: > Yeah, I think what we have in statsmodels is about as far as it's > gotten. I rewrote a lot of the code and David's NEP at the beginning > of the summer based on our needs to keep it maintanable and flexible. > There is also an incarnation in scikits-learn with a few differences, > but we tried to keep them similar. Comparing with Learn at: http://scikit-learn.git.sourceforge.net/git/gitweb.cgi?p=scikit-learn/scikit-learn;a=tree;f=scikits/learn/datasets You have increase the variety. > It might make sense to combine the two at some point and distribute as > a standalone scikit. For time series analysis I'd appreciate to have a data set with time stamps of frequency >= 1min. I currently do not have one free of copyright. I will use your code a starter and submit a good data set as soon as I get roalty-free data. Thanks. From jsseabold at gmail.com Sat Aug 14 16:57:37 2010 From: jsseabold at gmail.com (Skipper Seabold) Date: Sat, 14 Aug 2010 16:57:37 -0400 Subject: [SciPy-User] Datasets in Scipy Code In-Reply-To: References: Message-ID: On Sat, Aug 14, 2010 at 4:31 PM, Tim Michelsen wrote: >> Yeah, I think what we have in statsmodels is about as far as it's >> gotten. ?I rewrote a lot of the code and David's NEP at the beginning >> of the summer based on our needs to keep it maintanable and flexible. >> There is also an incarnation in scikits-learn with a few differences, >> but we tried to keep them similar. > Comparing with Learn at: > http://scikit-learn.git.sourceforge.net/git/gitweb.cgi?p=scikit-learn/scikit-learn;a=tree;f=scikits/learn/datasets > You have increase the variety. > >> It might make sense to combine the two at some point and distribute as >> a standalone scikit. > For time series analysis I'd appreciate to have a data set with time > stamps of frequency >= 1min. > I currently do not have one free of copyright. > What do you have in mind? I have some US macro data in there at the quarterly frequency. I would like to get some higher frequency finance stuff. > I will use your code a starter and submit a good data set as soon as I > get roalty-free data. That'd be great. There are some utility functions and templates so adding datasets is easy, so let me know when you have some data and I can walk you through it if you need it. It should also be documented in the updated datasets proposal. The license is the rub. Mostly I've contacted the original authors and have had no problems getting expressed written permission for reuse. Authors have told me that I am the only one who has ever asked, including datasets that are included in the R datasets library and other packages, and I've never gotten a straight answer on the licensing of datasets in R. Other stuff is often public domain. Skipper From gael.varoquaux at normalesup.org Sat Aug 14 18:05:53 2010 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 15 Aug 2010 00:05:53 +0200 Subject: [SciPy-User] Datasets in Scipy Code In-Reply-To: References: Message-ID: <20100814220553.GA609@phare.normalesup.org> On Sat, Aug 14, 2010 at 10:31:27PM +0200, Tim Michelsen wrote: > > Yeah, I think what we have in statsmodels is about as far as it's > > gotten. I rewrote a lot of the code and David's NEP at the beginning > > of the summer based on our needs to keep it maintanable and flexible. > > There is also an incarnation in scikits-learn with a few differences, > > but we tried to keep them similar. > Comparing with Learn at: > http://scikit-learn.git.sourceforge.net/git/gitweb.cgi?p=scikit-learn/scikit-learn;a=tree;f=scikits/learn/datasets > You have increase the variety. Yeah, maybe we need to loop back again. I am not happy with our current implementation in scikit learn. It is a bit messy. We had a common discussion a while ago to make sure that we were on the same track, and I think that we both found some snags in the implementation, that we both solved in different ways :). Ga?l From stef.mientki at gmail.com Mon Aug 16 10:27:49 2010 From: stef.mientki at gmail.com (Stef Mientki) Date: Mon, 16 Aug 2010 16:27:49 +0200 Subject: [SciPy-User] does someone has a binary 32-bit windows installer for arac ? Message-ID: <4C694AE5.2020907@gmail.com> thanks, Stef Mientki From cgohlke at uci.edu Mon Aug 16 15:23:48 2010 From: cgohlke at uci.edu (Christoph Gohlke) Date: Mon, 16 Aug 2010 12:23:48 -0700 Subject: [SciPy-User] does someone has a binary 32-bit windows installer for arac ? In-Reply-To: <4C694AE5.2020907@gmail.com> References: <4C694AE5.2020907@gmail.com> Message-ID: <4C699044.10204@uci.edu> On 8/16/2010 7:27 AM, Stef Mientki wrote: > thanks, > Stef Mientki Seems that building arac on Windows is not currently supported. From the SConstruct file: if sys.platform == 'darwin': ... elif sys.platform == 'linux2': ... else: raise SystemExit("Cannot build on %s." % sys.platform) -- Christoph From stef.mientki at gmail.com Mon Aug 16 16:11:51 2010 From: stef.mientki at gmail.com (Stef Mientki) Date: Mon, 16 Aug 2010 22:11:51 +0200 Subject: [SciPy-User] does someone has a binary 32-bit windows installer for arac ? In-Reply-To: <4C699044.10204@uci.edu> References: <4C694AE5.2020907@gmail.com> <4C699044.10204@uci.edu> Message-ID: <4C699B87.8080409@gmail.com> On 16-08-2010 21:23, Christoph Gohlke wrote: > > On 8/16/2010 7:27 AM, Stef Mientki wrote: >> thanks, >> Stef Mientki > > Seems that building arac on Windows is not currently supported. From the > SConstruct file: > > if sys.platform == 'darwin': > ... > elif sys.platform == 'linux2': > ... > else: > raise SystemExit("Cannot build on %s." % sys.platform) > > > -- > Christoph > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user thanks Cristhoph, at least I now can stop searching. cheers, Stef From mhubig at gmail.com Mon Aug 16 16:45:42 2010 From: mhubig at gmail.com (Markus Hubig) Date: Mon, 16 Aug 2010 22:45:42 +0200 Subject: [SciPy-User] Installing issue with python 2.7, numpy 1.5.0b1 in my Mac Message-ID: Hi @all, maybe this is a known issue but I didn't find any useful information out there. When I try to install scipy-0.8.0 with pip I get this strange error about missing files ... File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/distutils/npy_pkg_config.py", line 276, in parse_config raise PkgNotFound("Could not find file(s) %s" % str(filenames)) numpy.distutils.npy_pkg_config.PkgNotFound: Could not find file(s) ['/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/lib/npy-pkg-config/npymath.ini'] The complete error can be found here: http://gist.github.com/527697 Is there a known solution for this? Any help would be nice ... - Markus -- Can't read my mail? Just don't hold it that way! -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Mon Aug 16 22:48:34 2010 From: cournape at gmail.com (David Cournapeau) Date: Tue, 17 Aug 2010 11:48:34 +0900 Subject: [SciPy-User] Installing issue with python 2.7, numpy 1.5.0b1 in my Mac In-Reply-To: References: Message-ID: On Tue, Aug 17, 2010 at 5:45 AM, Markus Hubig wrote: > Hi @all, > maybe this is a known issue but I didn't find any?useful information out > there. > When I try to install?scipy-0.8.0 with pip Do you get the error without pip ? Also, does the file '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/lib/npy-pkg-config/npymath.ini' exists or not, and if it does, what's its content ? cheers, David From denis-bz-gg at t-online.de Tue Aug 17 07:17:39 2010 From: denis-bz-gg at t-online.de (denis) Date: Tue, 17 Aug 2010 04:17:39 -0700 (PDT) Subject: [SciPy-User] Interpolation in 3D with interp2d In-Reply-To: <1506300512.361757.1281645309362.JavaMail.fmail@mwmweb001> References: <1456192421.951467.1280690737282.JavaMail.fmail@mwmweb001>, <1379466308.2338901.1281513705129.JavaMail.fmail@mwmweb003>, <627f01fa-f1c6-410e-9a40-47f49691b77e@i31g2000yqm.googlegroups.com> <1506300512.361757.1281645309362.JavaMail.fmail@mwmweb001> Message-ID: <3f482ae8-94ab-4805-85d9-5ec05337a5a8@c10g2000yqi.googlegroups.com> On Aug 12, 10:35?pm, Jana Schulz wrote: > Hi, > RectBivariateSpline() can not handle randomly spaced data (that's what the error message says). The option 'linear' in griddata works only for ?constant spacing data. I attached the surface plot of my data and the txt-file including my data. > > As you see the surface plot works pretty well. All I want is to get any z-value ?for a given x,y-position. Jana, griddata( your data ) with the default interp="nn" looks good *once you take log10 of all your data*; any interpolator can misbehave on data from 1e-6 .. 1e6. I sent sample code and a plot to you at web.de on 16Aug (since my mails via gmane aren't getting through); did it make sense ? If you want to interpolate a single point at a time with Delaunay triangulation with a 2-stage interpolator like interp2d, see Triinterpolate below. (Yes the many interpolators in scipy are, hmm, scattered, nonuniform.) Is this closer to what you want ? cheers -- denis # tri = Triinterpolate(x,y,z), tri( single points ) from __future__ import division import sys import numpy as np import matplotlib.delaunay as delaunay # $matplotlib/delaunay/ interpolate.py __date__ = "2010-08-17 Aug" class Triinterpolate: """ interpolate scattered data a point at a time: tri = Triinterpolate( x,y,z ) -- 1d arrays first build a Delaunay triangulation; then zi = tri(xi,yi) -- interpolates one point NaN if xi,yi is outside the convex hull of the x,y points See also http://www.scipy.org/Cookbook/Matplotlib/Gridding_irregularly_spaced_data http://en.wikipedia.org/wiki/Natural_neighbor .../matplotlib/ mlab.py griddata(), delaunay/interpolate.py """ def __init__( self, x,y,z ): if not (len(x) == len(y) == len(z) and x.ndim == y.ndim == z.ndim == 1): raise TypeError("inputs x,y,z must all be 1D arrays of the same length") tri = delaunay.Triangulation(x,y) self.interp = tri.nn_interpolator(z) # linear_ has no __call__ def __call__( self, xi,yi ): if not (np.isscalar(xi) and np.isscalar(yi)): raise TypeError("inputs xi,yi must be scalars") return self.interp( xi, yi ) # NaN outside convex hull, cf corners() #............................................................................... N = 20 # N random in side x side side = 10 exec "\n".join( sys.argv[1:] ) # run this.py N= ... np.random.seed(1) np.set_printoptions( 1, threshold=100, suppress=True ) # .1f def terrainf( x, y ): return x**2 + y**2 x,y = np.random.uniform( 0, side, (2,N) ) z = terrainf( x, y ) print "x y z:\n", x, "\n", y, "\n", z, "\n" tri = Triinterpolate( x,y,z ) print "Triinterpolate a point at a time:" for yi in range(side): for xi in range(side): print "%5.0f" % tri( xi,yi ) , print "" From denis-bz-gg at t-online.de Tue Aug 17 08:42:58 2010 From: denis-bz-gg at t-online.de (denis) Date: Tue, 17 Aug 2010 05:42:58 -0700 (PDT) Subject: [SciPy-User] brunch-and-bound search ? Message-ID: The curse of dimensionality appears to cause typos too, "brunch-and-bound search" in similarity search wiki http://www.sswiki.tierra-aoi.net From dan.lussier at sjc.ox.ac.uk Tue Aug 17 13:41:26 2010 From: dan.lussier at sjc.ox.ac.uk (Dan Lussier) Date: Tue, 17 Aug 2010 12:41:26 -0500 Subject: [SciPy-User] IO of large ASCII table data Message-ID: I am looking to read in large (many million rows) ASCII space separated tables into numpy arrays. In the past I have heard of people using Miller's TableIO to do this but was wondering if a similarly fast method has been more recently integrated into scipy/numpy? In consulting the documentation the most likely candidate is numpy.genfromtext(...). Is this function pure python or does it rely on a C extension as was the case with Miller's TableIO? Any advice here would be great as my application could get seriously bogged down (both time and memory) in reading these files into arrays if I get onto the wrong track. Thanks. From Chris.Barker at noaa.gov Tue Aug 17 13:47:15 2010 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 17 Aug 2010 10:47:15 -0700 Subject: [SciPy-User] IO of large ASCII table data In-Reply-To: References: Message-ID: <4C6ACB23.708@noaa.gov> Dan Lussier wrote: > I am looking to read in large (many million rows) ASCII space > separated tables into numpy arrays. If it's all space separated, all one data type, and no comment lines or anything like that, then np.fromfile(file, sep=' ', dtype=np.float) is about as fast as you can get. fromfile() is very limited for reading text files, and doesn't handle error cases well, but if it works, it's fast. -Chris > In consulting the documentation the most likely candidate is > numpy.genfromtext(...). Is this function pure python or does it rely > on a C extension as was the case with Miller's TableIO? it's python, and not all that fast. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From edepagne at lcogt.net Tue Aug 17 13:53:07 2010 From: edepagne at lcogt.net (=?iso-8859-1?q?=C9ric_Depagne?=) Date: Tue, 17 Aug 2010 10:53:07 -0700 Subject: [SciPy-User] IO of large ASCII table data In-Reply-To: References: Message-ID: <201008171053.07876.edepagne@lcogt.net> Le mardi 17 ao?t 2010 10:41:26, Dan Lussier a ?crit : > I am looking to read in large (many million rows) ASCII space > separated tables into numpy arrays. > > In the past I have heard of people using Miller's TableIO to do this > but was wondering if a similarly fast method has been more recently > integrated into scipy/numpy? > > In consulting the documentation the most likely candidate is > numpy.genfromtext(...). Is this function pure python or does it rely > on a C extension as was the case with Miller's TableIO? > > Any advice here would be great as my application could get seriously > bogged down (both time and memory) in reading these files into arrays > if I get onto the wrong track. > > Thanks. There is the numpy.loadtxt() method that can also read data from file. I use it to read large datasets. Considering its speed, here are numbers I typically get. To extract 2.5 million lines and 10 columns it needs ~3mn. ?ric. -- Un clavier azerty en vaut deux ---------------------------------------------------------- ?ric Depagne edepagne at lcogt.net Las Cumbres Observatory 6740 Cortona Dr Goleta CA, 93117 ---------------------------------------------------------- From kwgoodman at gmail.com Tue Aug 17 14:03:13 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 17 Aug 2010 11:03:13 -0700 Subject: [SciPy-User] IO of large ASCII table data In-Reply-To: <201008171053.07876.edepagne@lcogt.net> References: <201008171053.07876.edepagne@lcogt.net> Message-ID: On Tue, Aug 17, 2010 at 10:53 AM, ?ric Depagne wrote: > Le mardi 17 ao?t 2010 10:41:26, Dan Lussier a ?crit : >> I am looking to read in large (many million rows) ASCII space >> separated tables into numpy arrays. >> >> In the past I have heard of people using Miller's TableIO to do this >> but was wondering if a similarly fast method has been more recently >> integrated into scipy/numpy? >> >> In consulting the documentation the most likely candidate is >> numpy.genfromtext(...). ?Is this function pure python or does it rely >> on a C extension as was the case with Miller's TableIO? >> >> Any advice here would be great as my application could get seriously >> bogged down (both time and memory) in reading these files into arrays >> if I get onto the wrong track. >> >> Thanks. > There is the numpy.loadtxt() method that can also read data from file. > I use it to read large datasets. Considering its speed, here are numbers I > typically get. To extract 2.5 million lines and 10 columns it needs ~3mn. For comparison, h5py (and pytables) are over 1500 times faster: Save data: >> arr = np.random.rand(2500000, 10) >> import h5py >> f = h5py.File('/tmp/speed.hdf5') >> f['arr'] = arr Time the loading of data: $ ipython >> import time >> import h5py >> f = h5py.File('/tmp/speed.hdf5') >> t1=time.time(); a = f['arr'][:]; print time.time() - t1 0.0953390598297 Speed up: >> 3*60/0.0953390598297 1887.9984795479013 From erin.sheldon at gmail.com Tue Aug 17 14:04:24 2010 From: erin.sheldon at gmail.com (Erin Sheldon) Date: Tue, 17 Aug 2010 14:04:24 -0400 Subject: [SciPy-User] IO of large ASCII table data In-Reply-To: References: Message-ID: <1282067797-sup-919@theshire> Excerpts from Dan Lussier's message of Tue Aug 17 13:41:26 -0400 2010: > I am looking to read in large (many million rows) ASCII space > separated tables into numpy arrays. > > In the past I have heard of people using Miller's TableIO to do this > but was wondering if a similarly fast method has been more recently > integrated into scipy/numpy? > > In consulting the documentation the most likely candidate is > numpy.genfromtext(...). Is this function pure python or does it rely > on a C extension as was the case with Miller's TableIO? > > Any advice here would be great as my application could get seriously > bogged down (both time and memory) in reading these files into arrays > if I get onto the wrong track. > > Thanks. The recfile package is designed specifically for this purpose: http://code.google.com/p/recfile/ It can read ascii or binary into record arrays. It is a C++ extension, so it is efficient. import recfile # Read from an ascii file. Number of rows will be determined from # the file data and dtype if not entered. fname='test.csv' dtype=[('field1','f8'),('field2','2i4'),('field3','i8')] robj = recfile.Open(fname, dtype=dtype, delim=',') # read all rows and columns data = robj[:] # read subset of columns data = robj[ ['x','y'] ][:] # read subset of rows data = robj[25:50] data = robj[rowlist] data = robj[ ['x','y'] ][rowlist] Erin Scott Sheldon From ben.root at ou.edu Tue Aug 17 14:07:59 2010 From: ben.root at ou.edu (Benjamin Root) Date: Tue, 17 Aug 2010 13:07:59 -0500 Subject: [SciPy-User] IO of large ASCII table data In-Reply-To: References: <201008171053.07876.edepagne@lcogt.net> Message-ID: On Tue, Aug 17, 2010 at 1:03 PM, Keith Goodman wrote: > On Tue, Aug 17, 2010 at 10:53 AM, ?ric Depagne wrote: > > Le mardi 17 ao?t 2010 10:41:26, Dan Lussier a ?crit : > >> I am looking to read in large (many million rows) ASCII space > >> separated tables into numpy arrays. > >> > >> In the past I have heard of people using Miller's TableIO to do this > >> but was wondering if a similarly fast method has been more recently > >> integrated into scipy/numpy? > >> > >> In consulting the documentation the most likely candidate is > >> numpy.genfromtext(...). Is this function pure python or does it rely > >> on a C extension as was the case with Miller's TableIO? > >> > >> Any advice here would be great as my application could get seriously > >> bogged down (both time and memory) in reading these files into arrays > >> if I get onto the wrong track. > >> > >> Thanks. > > There is the numpy.loadtxt() method that can also read data from file. > > I use it to read large datasets. Considering its speed, here are numbers > I > > typically get. To extract 2.5 million lines and 10 columns it needs ~3mn. > > For comparison, h5py (and pytables) are over 1500 times faster: > > Save data: > > >> arr = np.random.rand(2500000, 10) > >> import h5py > >> f = h5py.File('/tmp/speed.hdf5') > >> f['arr'] = arr > > Time the loading of data: > > $ ipython > >> import time > >> import h5py > >> f = h5py.File('/tmp/speed.hdf5') > >> t1=time.time(); a = f['arr'][:]; print time.time() - t1 > 0.0953390598297 > > Speed up: > > >> 3*60/0.0953390598297 > 1887.9984795479013 > Keith, Note that files saved to the /tmp directory are likely using tmpfs, which is heavily RAM oriented. Your speed-up might not be reflecting the impact of disk I/O. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Tue Aug 17 14:13:14 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 17 Aug 2010 11:13:14 -0700 Subject: [SciPy-User] IO of large ASCII table data In-Reply-To: References: <201008171053.07876.edepagne@lcogt.net> Message-ID: On Tue, Aug 17, 2010 at 11:07 AM, Benjamin Root wrote: > > > On Tue, Aug 17, 2010 at 1:03 PM, Keith Goodman wrote: >> >> On Tue, Aug 17, 2010 at 10:53 AM, ?ric Depagne wrote: >> > Le mardi 17 ao?t 2010 10:41:26, Dan Lussier a ?crit : >> >> I am looking to read in large (many million rows) ASCII space >> >> separated tables into numpy arrays. >> >> >> >> In the past I have heard of people using Miller's TableIO to do this >> >> but was wondering if a similarly fast method has been more recently >> >> integrated into scipy/numpy? >> >> >> >> In consulting the documentation the most likely candidate is >> >> numpy.genfromtext(...). ?Is this function pure python or does it rely >> >> on a C extension as was the case with Miller's TableIO? >> >> >> >> Any advice here would be great as my application could get seriously >> >> bogged down (both time and memory) in reading these files into arrays >> >> if I get onto the wrong track. >> >> >> >> Thanks. >> > There is the numpy.loadtxt() method that can also read data from file. >> > I use it to read large datasets. Considering its speed, here are numbers >> > I >> > typically get. To extract 2.5 million lines and 10 columns it needs >> > ~3mn. >> >> For comparison, h5py (and pytables) are over 1500 times faster: >> >> Save data: >> >> >> arr = np.random.rand(2500000, 10) >> >> import h5py >> >> f = h5py.File('/tmp/speed.hdf5') >> >> f['arr'] = arr >> >> Time the loading of data: >> >> $ ipython >> >> import time >> >> import h5py >> >> f = h5py.File('/tmp/speed.hdf5') >> >> t1=time.time(); a = f['arr'][:]; print time.time() - t1 >> 0.0953390598297 >> >> Speed up: >> >> >> 3*60/0.0953390598297 >> ? 1887.9984795479013 > > Keith, > > Note that files saved to the /tmp directory are likely using tmpfs, which is > heavily RAM oriented.? Your speed-up might not be reflecting the impact of > disk I/O. Here's the time it takes to read from my home directory (regular HD): >> f = h5py.File('/home/kg/speed.hdf5') >> t1=time.time(); a = f['arr'][:]; print time.time() - t1 0.0879709720612 And here's the time it takes to read from my ram disk: >> f = h5py.File('/dev/shm/speed.hdf5') >> t1=time.time(); a = f['arr'][:]; print time.time() - t1 0.086874961853 From almar.klein at gmail.com Tue Aug 17 14:38:40 2010 From: almar.klein at gmail.com (Almar Klein) Date: Tue, 17 Aug 2010 20:38:40 +0200 Subject: [SciPy-User] Cross platform animated plot with widgets In-Reply-To: <499214819.414661281659040499.JavaMail.root@csemailbox.ucsd.edu> References: <499214819.414661281659040499.JavaMail.root@csemailbox.ucsd.edu> Message-ID: On 13 August 2010 02:24, Emmett McQuinn wrote: > Hello, > > I've been trying to create an animated plot with matplotlib that supports > parameter updating with a slider. I've been able to have animation, and also > create a slider (using matplotlib.widgets), but not both at the same time. > I've tried various permutations and it seems exclusively either the graph > animates or the widgets work and not both. Is there a good method to provide > a simple lightweight GUI that provides both animation and sliders for Linux, > Windows, and Mac? > > Thanks, > Emmett McQuinn > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > Visvis can be used for that. It currently does not have a slider widget, but you can embed a visvis figure in a simple GUI application that has a slider. (There's an example how to do that in the examples dir.) Supported GUI toolkits are Qt4, WX and FLTK (if you really want to go lightweight :)). http://code.google.com/p/visvis/ Almar -------------- next part -------------- An HTML attachment was scrubbed... URL: From dtlussier at gmail.com Tue Aug 17 14:41:21 2010 From: dtlussier at gmail.com (Dan Lussier) Date: Tue, 17 Aug 2010 13:41:21 -0500 Subject: [SciPy-User] IO of large ASCII table data In-Reply-To: References: <201008171053.07876.edepagne@lcogt.net> Message-ID: That's great. Thanks. I am going to give np.fromfile(...) a try as I have data that is pretty uniform and will fixup the output as necessary. On my system reading a 1.2M row by 11 column data file np.genfromtxt(...) took 50 seconds while np.fromfile(...) plus a np.reshape(...) to change the shape of the output took under 2 seconds. Dan On Tue, Aug 17, 2010 at 1:13 PM, Keith Goodman wrote: > On Tue, Aug 17, 2010 at 11:07 AM, Benjamin Root wrote: >> >> >> On Tue, Aug 17, 2010 at 1:03 PM, Keith Goodman wrote: >>> >>> On Tue, Aug 17, 2010 at 10:53 AM, ?ric Depagne wrote: >>> > Le mardi 17 ao?t 2010 10:41:26, Dan Lussier a ?crit : >>> >> I am looking to read in large (many million rows) ASCII space >>> >> separated tables into numpy arrays. >>> >> >>> >> In the past I have heard of people using Miller's TableIO to do this >>> >> but was wondering if a similarly fast method has been more recently >>> >> integrated into scipy/numpy? >>> >> >>> >> In consulting the documentation the most likely candidate is >>> >> numpy.genfromtext(...). ?Is this function pure python or does it rely >>> >> on a C extension as was the case with Miller's TableIO? >>> >> >>> >> Any advice here would be great as my application could get seriously >>> >> bogged down (both time and memory) in reading these files into arrays >>> >> if I get onto the wrong track. >>> >> >>> >> Thanks. >>> > There is the numpy.loadtxt() method that can also read data from file. >>> > I use it to read large datasets. Considering its speed, here are numbers >>> > I >>> > typically get. To extract 2.5 million lines and 10 columns it needs >>> > ~3mn. >>> >>> For comparison, h5py (and pytables) are over 1500 times faster: >>> >>> Save data: >>> >>> >> arr = np.random.rand(2500000, 10) >>> >> import h5py >>> >> f = h5py.File('/tmp/speed.hdf5') >>> >> f['arr'] = arr >>> >>> Time the loading of data: >>> >>> $ ipython >>> >> import time >>> >> import h5py >>> >> f = h5py.File('/tmp/speed.hdf5') >>> >> t1=time.time(); a = f['arr'][:]; print time.time() - t1 >>> 0.0953390598297 >>> >>> Speed up: >>> >>> >> 3*60/0.0953390598297 >>> ? 1887.9984795479013 >> >> Keith, >> >> Note that files saved to the /tmp directory are likely using tmpfs, which is >> heavily RAM oriented.? Your speed-up might not be reflecting the impact of >> disk I/O. > > Here's the time it takes to read from my home directory (regular HD): > >>> f = h5py.File('/home/kg/speed.hdf5') >>> t1=time.time(); a = f['arr'][:]; print time.time() - t1 > 0.0879709720612 > > And here's the time it takes to read from my ram disk: > >>> f = h5py.File('/dev/shm/speed.hdf5') >>> t1=time.time(); a = f['arr'][:]; print time.time() - t1 > 0.086874961853 > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From edepagne at lcogt.net Tue Aug 17 14:43:20 2010 From: edepagne at lcogt.net (=?iso-8859-1?q?=C9ric_Depagne?=) Date: Tue, 17 Aug 2010 11:43:20 -0700 Subject: [SciPy-User] IO of large ASCII table data In-Reply-To: References: Message-ID: <201008171143.20994.edepagne@lcogt.net> > > Here's the time it takes to read from my home directory (regular HD): > >> f = h5py.File('/home/kg/speed.hdf5') > >> t1=time.time(); a = f['arr'][:]; print time.time() - t1 > > 0.0879709720612 > > And here's the time it takes to read from my ram disk: > >> f = h5py.File('/dev/shm/speed.hdf5') > >> t1=time.time(); a = f['arr'][:]; print time.time() - t1 > > 0.086874961853 > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > That's impressive. Any ascii file can be read with h5py? And the can we "convert" the data read into numpy arrays ? -- Un clavier azerty en vaut deux ---------------------------------------------------------- ?ric Depagne edepagne at lcogt.net Las Cumbres Observatory 6740 Cortona Dr Goleta CA, 93117 ---------------------------------------------------------- From alan.isaac at gmail.com Tue Aug 17 14:44:55 2010 From: alan.isaac at gmail.com (Alan G Isaac) Date: Tue, 17 Aug 2010 14:44:55 -0400 Subject: [SciPy-User] IO of large ASCII table data In-Reply-To: <4C6ACB23.708@noaa.gov> References: <4C6ACB23.708@noaa.gov> Message-ID: <4C6AD8A7.7050509@gmail.com> > Dan Lussier wrote: >> I am looking to read in large (many million rows) ASCII space >> separated tables into numpy arrays. On 8/17/2010 1:47 PM, Christopher Barker wrote: > If it's all space separated, all one data type, and no comment lines or > anything like that, then np.fromfile(file, sep=' ', dtype=np.float) is > about as fast as you can get. > > fromfile() is very limited for reading text files, and doesn't handle > error cases well, but if it works, it's fast. Apparently TableIO has gotten harder to find, but: http://kochanski.org/gpk/misc/TableIO.html Didn't check the posted license, but Mike Miller did say he was open to a SciPy compatible license. Alan Isaac PS Sorry for the delayed reply... From pgmdevlist at gmail.com Tue Aug 17 14:51:23 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 17 Aug 2010 14:51:23 -0400 Subject: [SciPy-User] IO of large ASCII table data In-Reply-To: References: <201008171053.07876.edepagne@lcogt.net> Message-ID: <59DB4D58-621E-4F7F-B94E-FC6996B92B57@gmail.com> On Aug 17, 2010, at 2:41 PM, Dan Lussier wrote: > That's great. Thanks. > > I am going to give np.fromfile(...) a try as I have data that is > pretty uniform and will fixup the output as necessary. > > On my system reading a 1.2M row by 11 column data file > np.genfromtxt(...) took 50 seconds while np.fromfile(...) plus a > np.reshape(...) to change the shape of the output took under 2 > seconds. There are multiple loops involved in genfromtxt that you don't have in fromtxt. That's unfortunately the price to pay for flexibility (ie, you don't know what your array should look like beforehand, and don't know want to deal w/ missing values)... From kwgoodman at gmail.com Tue Aug 17 14:54:28 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 17 Aug 2010 11:54:28 -0700 Subject: [SciPy-User] IO of large ASCII table data In-Reply-To: <201008171143.20994.edepagne@lcogt.net> References: <201008171143.20994.edepagne@lcogt.net> Message-ID: On Tue, Aug 17, 2010 at 11:43 AM, ?ric Depagne wrote: >> >> Here's the time it takes to read from my home directory (regular HD): >> >> f = h5py.File('/home/kg/speed.hdf5') >> >> t1=time.time(); a = f['arr'][:]; print time.time() - t1 >> >> 0.0879709720612 >> >> And here's the time it takes to read from my ram disk: >> >> f = h5py.File('/dev/shm/speed.hdf5') >> >> t1=time.time(); a = f['arr'][:]; print time.time() - t1 >> >> 0.086874961853 >> > That's impressive. > Any ascii file can be read with h5py? > And the can we "convert" the data read into numpy arrays ? h5py can only read hdf5 files, so you'd have to convert the ascii file to hdf5. Might be worth it if you read the same ascii file many times. Even better would be to change the program that is writing the ascii files to write hdf5 files. To convert from ascii to hdf5: arr = np.fromfile(...) f = h5py.File('converted.hdf5') f['arr'] = arr From schlesin at cshl.edu Tue Aug 17 15:55:17 2010 From: schlesin at cshl.edu (Felix Schlesinger) Date: Tue, 17 Aug 2010 15:55:17 -0400 Subject: [SciPy-User] IO of large ASCII table data Message-ID: > The recfile package is designed specifically for this purpose: > http://code.google.com/p/recfile/ recfile looks very useful. Any reason its not on pypi? That would make maintaining it as a dependency much easier. Felix From erin.sheldon at gmail.com Tue Aug 17 16:21:55 2010 From: erin.sheldon at gmail.com (erin.sheldon) Date: Tue, 17 Aug 2010 16:21:55 -0400 Subject: [SciPy-User] IO of large ASCII table data In-Reply-To: References: Message-ID: <1282076276-sup-9537@theshire> Excerpts from Felix Schlesinger's message of Tue Aug 17 15:55:17 -0400 2010: > > The recfile package is designed specifically for this purpose: > > http://code.google.com/p/recfile/ > > recfile looks very useful. Any reason its not on pypi? That would make > maintaining it as a dependency much easier. > > Felix Felix - Interesting, I had never paid much attention to PyPI before as I have my own methods for dealing with dependencies. But that looks like a great resource and a nice way to distribute packages. I wouldn't know where to start with it though. I would appreciate any tips you could give (offline if you wish). Erin Scott Sheldon From robert.kern at gmail.com Tue Aug 17 16:32:18 2010 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 17 Aug 2010 15:32:18 -0500 Subject: [SciPy-User] IO of large ASCII table data In-Reply-To: <1282076276-sup-9537@theshire> References: <1282076276-sup-9537@theshire> Message-ID: On Tue, Aug 17, 2010 at 15:21, erin.sheldon wrote: > Excerpts from Felix Schlesinger's message of Tue Aug 17 15:55:17 -0400 2010: >> > The recfile package is designed specifically for this purpose: >> > http://code.google.com/p/recfile/ >> >> recfile looks very useful. Any reason its not on pypi? That would make >> maintaining it as a dependency much easier. >> >> Felix > > Felix - > > Interesting, I had never paid much attention to PyPI before as I have my > own methods for dealing with dependencies. But that looks like a great > resource and a nice way to distribute packages. > > I wouldn't know where to start with it though. ?I would appreciate any > tips you could give (offline if you wish). http://wiki.python.org/moin/CheeseShopTutorial Go down to the section "Submitting Packages to the Package Index". I recommend removing the "ups" parts of the setup.py before making a general release. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From mhubig at gmail.com Wed Aug 18 03:13:39 2010 From: mhubig at gmail.com (Markus Hubig) Date: Wed, 18 Aug 2010 09:13:39 +0200 Subject: [SciPy-User] Installing issue with python 2.7, numpy 1.5.0b1 in my Mac In-Reply-To: References: Message-ID: I get the same error if I try to install SciPy with setup.py. And I dont have the missing file npymath.ini, I not even have the directory "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/lib" - Markus On Tue, Aug 17, 2010 at 4:48 AM, David Cournapeau wrote: > On Tue, Aug 17, 2010 at 5:45 AM, Markus Hubig wrote: > > Hi @all, > > maybe this is a known issue but I didn't find any useful information out > > there. > > When I try to install scipy-0.8.0 with pip > > Do you get the error without pip ? Also, does the file > > '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/lib/npy-pkg-config/npymath.ini' > exists or not, and if it does, what's its content ? > > cheers, > > David > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Can't read my mail? Just don't hold it that way! -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at silveregg.co.jp Wed Aug 18 03:40:16 2010 From: david at silveregg.co.jp (David) Date: Wed, 18 Aug 2010 16:40:16 +0900 Subject: [SciPy-User] Installing issue with python 2.7, numpy 1.5.0b1 in my Mac In-Reply-To: References: Message-ID: <4C6B8E60.80903@silveregg.co.jp> On 08/18/2010 04:13 PM, Markus Hubig wrote: > I get the same error if I try to install SciPy with setup.py. And I dont > have the missing file npymath.ini, > I not even have the directory > "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/lib" How did you install numpy (the exact command you used) ? David From mhubig at gmail.com Wed Aug 18 07:38:42 2010 From: mhubig at gmail.com (Markus Hubig) Date: Wed, 18 Aug 2010 13:38:42 +0200 Subject: [SciPy-User] Installing issue with python 2.7, numpy 1.5.0b1 in my Mac In-Reply-To: <4C6B8E60.80903@silveregg.co.jp> References: <4C6B8E60.80903@silveregg.co.jp> Message-ID: pip install numpy-1.5.0b1.tar.gz Runs without problems. - Markus On Wed, Aug 18, 2010 at 9:40 AM, David wrote: > On 08/18/2010 04:13 PM, Markus Hubig wrote: > > I get the same error if I try to install SciPy with setup.py. And I dont > > have the missing file npymath.ini, > > I not even have the directory > > > "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/lib" > > How did you install numpy (the exact command you used) ? > > David > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Can't read my mail? Just don't hold it that way! -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at silveregg.co.jp Wed Aug 18 07:48:32 2010 From: david at silveregg.co.jp (David) Date: Wed, 18 Aug 2010 20:48:32 +0900 Subject: [SciPy-User] Installing issue with python 2.7, numpy 1.5.0b1 in my Mac In-Reply-To: References: <4C6B8E60.80903@silveregg.co.jp> Message-ID: <4C6BC890.8020009@silveregg.co.jp> On 08/18/2010 08:38 PM, Markus Hubig wrote: > pip install numpy-1.5.0b1.tar.gz > > Runs without problems. Where is numpy installed with pip (what does numpy.__file__ say) ? The fact that the ini file is not there means something is wrong in the install, cheers, David From jeanluc.menut at free.fr Wed Aug 18 10:30:09 2010 From: jeanluc.menut at free.fr (Jean-Luc Menut) Date: Wed, 18 Aug 2010 16:30:09 +0200 Subject: [SciPy-User] Glitch in the map_coordinates doc Message-ID: <4C6BEE71.80309@free.fr> Hello all, In the map_coordinate reference document page (http://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.interpolation.map_coordinates.html), there is this example : >>> import scipy.ndimage >>> a = np.arange(12.).reshape((4, 3)) >>> a array([[ 0., 1., 2.], [ 3., 4., 5.], [ 6., 7., 8.], [ 9., 10., 11.]]) >>> sp.ndimage.map_coordinates(a, [[0.5, 2], [0.5, 1]], order=1) [ 2. 7.] According to the last line, import scipy.ndimage should be import scipy as sp. Best regards, J.L. From mhubig at gmail.com Wed Aug 18 18:56:43 2010 From: mhubig at gmail.com (Markus Hubig) Date: Thu, 19 Aug 2010 00:56:43 +0200 Subject: [SciPy-User] Installing issue with python 2.7, numpy 1.5.0b1 in my Mac In-Reply-To: <4C6BC890.8020009@silveregg.co.jp> References: <4C6B8E60.80903@silveregg.co.jp> <4C6BC890.8020009@silveregg.co.jp> Message-ID: >>> import numpy >>> numpy.__file__ '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/__init__.pyc' >>> And as you can see here http://gist.github.com/536443 I didn't get an error installing the new numpy-1.5.0b2 but SciPy also didn't install with this one ... On Wed, Aug 18, 2010 at 1:48 PM, David wrote: > On 08/18/2010 08:38 PM, Markus Hubig wrote: > > pip install numpy-1.5.0b1.tar.gz > > > > Runs without problems. > > Where is numpy installed with pip (what does numpy.__file__ say) ? The > fact that the ini file is not there means something is wrong in the > install, > > cheers, > > David > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Can't read my mail? Just don't hold it that way! -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Wed Aug 18 21:07:39 2010 From: cournape at gmail.com (David Cournapeau) Date: Thu, 19 Aug 2010 10:07:39 +0900 Subject: [SciPy-User] Installing issue with python 2.7, numpy 1.5.0b1 in my Mac In-Reply-To: References: <4C6B8E60.80903@silveregg.co.jp> <4C6BC890.8020009@silveregg.co.jp> Message-ID: On Thu, Aug 19, 2010 at 7:56 AM, Markus Hubig wrote: >>>> import numpy >>>> numpy.__file__ > '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/__init__.pyc' >>>> > And as you can see here?http://gist.github.com/536443 I didn't get an error > installing the > new?numpy-1.5.0b2?but SciPy also didn't install with this one ... Could you install both numpy and scipy the standard way, without pip being involved at all ? I tried the last beta with python2.7 on Linux without pip, and it installs the numpy/core/lib/npy-pkg-config directory as expected. So I suspect either pip or some mac os x-specific bug, cheers, David From ralf.gommers at googlemail.com Thu Aug 19 10:55:11 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 19 Aug 2010 22:55:11 +0800 Subject: [SciPy-User] Glitch in the map_coordinates doc In-Reply-To: <4C6BEE71.80309@free.fr> References: <4C6BEE71.80309@free.fr> Message-ID: On Wed, Aug 18, 2010 at 10:30 PM, Jean-Luc Menut wrote: > Hello all, > > In the map_coordinate reference document page > ( > http://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.interpolation.map_coordinates.html > ), > there is this example : > > >>> import scipy.ndimage > >>> a = np.arange(12.).reshape((4, 3)) > >>> a > array([[ 0., 1., 2.], > [ 3., 4., 5.], > [ 6., 7., 8.], > [ 9., 10., 11.]]) > >>> sp.ndimage.map_coordinates(a, [[0.5, 2], [0.5, 1]], order=1) > [ 2. 7.] > > > According to the last line, import scipy.ndimage should be import scipy > as sp. > > In all examples the lines "import scipy as sp" and "import numpy as np" are assumed to have been executed, no need to repeat that in every docstring. Subpackages like ndimage do need to be imported seperately. So the example is fine. See http://projects.scipy.org/numpy/wiki/CodingStyleGuidelines. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Thu Aug 19 12:21:11 2010 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Thu, 19 Aug 2010 11:21:11 -0500 Subject: [SciPy-User] Glitch in the map_coordinates doc In-Reply-To: References: <4C6BEE71.80309@free.fr> Message-ID: <4C6D59F7.5070304@enthought.com> Ralf Gommers wrote: > > > On Wed, Aug 18, 2010 at 10:30 PM, Jean-Luc Menut > > wrote: > > Hello all, > > In the map_coordinate reference document page > (http://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.interpolation.map_coordinates.html), > there is this example : > > >>> import scipy.ndimage > >>> a = np.arange(12.).reshape((4, 3)) > >>> a > array([[ 0., 1., 2.], > [ 3., 4., 5.], > [ 6., 7., 8.], > [ 9., 10., 11.]]) > >>> sp.ndimage.map_coordinates(a, [[0.5, 2], [0.5, 1]], order=1) > [ 2. 7.] > > > According to the last line, import scipy.ndimage should be import > scipy > as sp. > > In all examples the lines "import scipy as sp" and "import numpy as > np" are assumed to have been executed, no need to repeat that in every > docstring. Subpackages like ndimage do need to be imported seperately. > So the example is fine. See > http://projects.scipy.org/numpy/wiki/CodingStyleGuidelines. > The first line of the example, "import scipy.ndimage", can be removed. Warren > Cheers, > Ralf > > > ------------------------------------------------------------------------ > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From warren.weckesser at enthought.com Thu Aug 19 12:33:19 2010 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Thu, 19 Aug 2010 11:33:19 -0500 Subject: [SciPy-User] Glitch in the map_coordinates doc In-Reply-To: <4C6D59F7.5070304@enthought.com> References: <4C6BEE71.80309@free.fr> <4C6D59F7.5070304@enthought.com> Message-ID: <4C6D5CCF.5020609@enthought.com> Warren Weckesser wrote: > Ralf Gommers wrote: > >> On Wed, Aug 18, 2010 at 10:30 PM, Jean-Luc Menut >> > wrote: >> >> Hello all, >> >> In the map_coordinate reference document page >> (http://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.interpolation.map_coordinates.html), >> there is this example : >> >> >>> import scipy.ndimage >> >>> a = np.arange(12.).reshape((4, 3)) >> >>> a >> array([[ 0., 1., 2.], >> [ 3., 4., 5.], >> [ 6., 7., 8.], >> [ 9., 10., 11.]]) >> >>> sp.ndimage.map_coordinates(a, [[0.5, 2], [0.5, 1]], order=1) >> [ 2. 7.] >> >> >> According to the last line, import scipy.ndimage should be import >> scipy >> as sp. >> >> In all examples the lines "import scipy as sp" and "import numpy as >> np" are assumed to have been executed, no need to repeat that in every >> docstring. Subpackages like ndimage do need to be imported seperately. >> So the example is fine. See >> http://projects.scipy.org/numpy/wiki/CodingStyleGuidelines. >> >> > > The first line of the example, "import scipy.ndimage", can be removed. > > No, that is not correct. It needs to be there, so Ralf was correct. Warren From mhubig at gmail.com Thu Aug 19 13:25:34 2010 From: mhubig at gmail.com (Markus Hubig) Date: Thu, 19 Aug 2010 19:25:34 +0200 Subject: [SciPy-User] Installing issue with python 2.7, numpy 1.5.0b1 in my Mac In-Reply-To: References: <4C6B8E60.80903@silveregg.co.jp> <4C6BC890.8020009@silveregg.co.jp> Message-ID: Hmm it seems SciPy don't like me at all ... ;-) Now the installation of numpy (the usual way without using pip) was OK (I have the numpy/core/lib/npy-pkg-config dir) but SciPy still fails to compile. But now I get gfortran and ln errors ... see: http://gist.github.com/538418 Damn! On Thu, Aug 19, 2010 at 3:07 AM, David Cournapeau wrote: > On Thu, Aug 19, 2010 at 7:56 AM, Markus Hubig wrote: > >>>> import numpy > >>>> numpy.__file__ > > > '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/__init__.pyc' > >>>> > > And as you can see here http://gist.github.com/536443 I didn't get an > error > > installing the > > new numpy-1.5.0b2 but SciPy also didn't install with this one ... > > Could you install both numpy and scipy the standard way, without pip > being involved at all ? I tried the last beta with python2.7 on Linux > without pip, and it installs the numpy/core/lib/npy-pkg-config > directory as expected. So I suspect either pip or some mac os > x-specific bug, > > cheers, > > David > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Can't read my mail? Just don't hold it that way! -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcragun at gmail.com Thu Aug 19 17:08:49 2010 From: mcragun at gmail.com (Matthew Cragun) Date: Thu, 19 Aug 2010 17:08:49 -0400 Subject: [SciPy-User] Interpolating discrete values with splprep and splev Message-ID: I'm trying to fit a curve to some data points using splprep and splev. Then I would like to take a few discrete values and get the inerpolated values back using the spline. I am having a problem getting the correct y values out based on the spline fit. In my code below I would like a plot of the saex values against their interpolated values, but can't get them out properly. saex2 gives: [ 37.27079882 55.12775627 106.64397968 142.10925103 158.67651813 176.27838508] saey gives: [ 1.76685635e-01 -3.89842080e+00 -5.48621169e+01 -1.42948834e+02 -2.03504645e+02 -2.83162159e+02] Any thoughts? Here is my code: from numpy import arange, cos, linspace, pi, sin, random, r_, array from scipy.interpolate import splprep, splev # Original points x = array([0,3,6,9,12,15]) y = array([0.530,0.546,0.586,0.658,0.727,0.801]) #Interpolated X locations saex = array([2.484,3.671,7.063,9.354,10.408,11.515]) # spline parameters s=3.0 # smoothness parameter k=3 # spline order nest=-1 # estimate of number of knots needed (-1 = maximal) # find the knot points tckp,u = splprep([x,y],s=s,k=k,nest=nest) # evaluate spline splinex,spliney= splev(linspace(0,1,400),tckp) #evaluate at discrete points saex2,saey= splev(saex,tckp,der=0) print saex2 #plot import pylab #plot the data data,=pylab.plot(x,y,'o',label='data') #plot the fitted curve fit,=pylab.plot(splinex,spliney,'r-',label='fit') #plot the discrete locations #sae,=pylab.plot(saex2,saey,'x',label='sae points') pylab.legend() pylab.xlabel('x') pylab.ylabel('y') pylab.savefig('splprep_demo.png') -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at silveregg.co.jp Thu Aug 19 20:44:12 2010 From: david at silveregg.co.jp (David) Date: Fri, 20 Aug 2010 09:44:12 +0900 Subject: [SciPy-User] Installing issue with python 2.7, numpy 1.5.0b1 in my Mac In-Reply-To: References: <4C6B8E60.80903@silveregg.co.jp> <4C6BC890.8020009@silveregg.co.jp> Message-ID: <4C6DCFDC.4050203@silveregg.co.jp> On 08/20/2010 02:25 AM, Markus Hubig wrote: > Hmm it seems SciPy don't like me at all ... ;-) Now the installation of > numpy (the usual way > without using pip) was OK (I have the numpy/core/lib/npy-pkg-config dir) > but SciPy still fails > to compile. But now I get gfortran and ln errors ... see: Hm, this one looks annoying... One temporary hack would be to disable ppc arch altogether in fortran, something like: export FFLAGS="-Wall -ffixed-form -fno-second-underscore -arch i686 -arch x86_64 -fPIC -O3 -funroll-loops" export LDFLAGS=" -Wall -arch i686 -arch x86_64 -Wall -undefined dynamic_lookup -bundle" I don't have a mac to test this ATM, so you may have to tweak things a bit. The idea is to remove the -arch ppc64 flags which seem unsupported on recent mac os x version (do you use snow leopard by any chance ?) cheers, David From josef.pktd at gmail.com Thu Aug 19 20:57:01 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 19 Aug 2010 20:57:01 -0400 Subject: [SciPy-User] Interpolating discrete values with splprep and splev In-Reply-To: References: Message-ID: On Thu, Aug 19, 2010 at 5:08 PM, Matthew Cragun wrote: > I'm trying to fit a curve to some data points using splprep and splev. ?Then > I would like to take a few discrete values and get the inerpolated values > back using the spline. > I am having a problem getting the correct y values out based on the spline > fit. ?In my code below I would like a plot of the saex values against their > interpolated values, but can't get them out properly. > saex2 gives: [ ?37.27079882 ? 55.12775627 ?106.64397968 ?142.10925103 > ?158.67651813?176.27838508] > saey ?gives: ?[ ?1.76685635e-01 ?-3.89842080e+00 ?-5.48621169e+01 > ?-1.42948834e+02??-2.03504645e+02 ?-2.83162159e+02] > Any thoughts? ?Here is my code: > from numpy import arange, cos, linspace, pi, sin, random, r_, array > from scipy.interpolate import splprep, splev > # Original points > x = array([0,3,6,9,12,15]) > y = array([0.530,0.546,0.586,0.658,0.727,0.801]) > #Interpolated X locations > saex = array([2.484,3.671,7.063,9.354,10.408,11.515]) > # spline parameters > s=3.0 # smoothness parameter > k=3 # spline order > nest=-1 # estimate of number of knots needed (-1 = maximal) > # find the knot points > tckp,u = splprep([x,y],s=s,k=k,nest=nest) > # evaluate spline > splinex,spliney= splev(linspace(0,1,400),tckp) > #evaluate at discrete points > saex2,saey= splev(saex,tckp,der=0) > print saex2 > #plot > import pylab > #plot the data > data,=pylab.plot(x,y,'o',label='data') > #plot the fitted curve > fit,=pylab.plot(splinex,spliney,'r-',label='fit') > #plot the discrete locations > #sae,=pylab.plot(saex2,saey,'x',label='sae points') > pylab.legend() > pylab.xlabel('x') > pylab.ylabel('y') > > pylab.savefig('splprep_demo.png') I'm not quite sure what you are asking for, nor why you are parametric splines, but if you want the interpolated values for the original x, then this seems to work up to 4, 5 decimals >>> res = splev(u,tckp) >>> res [array([ -1.40838043e-05, 3.00005731e+00, 5.99991159e+00, 9.00006221e+00, 1.19999820e+01, 1.50000010e+01]), array([ 0.53034055, 0.54429613, 0.58940979, 0.65458787, 0.7287074 , 0.80065826])] >>> x-res[0] array([ 1.40838043e-05, -5.73105777e-05, 8.84070598e-05, -6.22059350e-05, 1.80088155e-05, -9.83166856e-07]) >>> y-res[1] array([-0.00034055, 0.00170387, -0.00340979, 0.00341213, -0.0017074 , 0.00034174]) >>> u array([ 0. , 0.19996446, 0.39994386, 0.59996306, 0.79997756, 1. ]) >>> x array([ 0, 3, 6, 9, 12, 15]) >>> 1.*x/x.max() array([ 0. , 0.2, 0.4, 0.6, 0.8, 1. ]) >>> res = splev((x-x.min())*1.0/x.max(),tckp) >>> res [array([ -1.40838043e-05, 3.00059043e+00, 6.00075376e+00, 9.00061633e+00, 1.20003186e+01, 1.50000010e+01]), array([ 0.53034055, 0.54430171, 0.5894258 , 0.65460107, 0.72871581, 0.80065826])] >>> x-res[0] array([ 1.40838043e-05, -5.90429918e-04, -7.53755099e-04, -6.16330116e-04, -3.18593347e-04, -9.83166856e-07]) >>> y-res[1] array([-0.00034055, 0.00169829, -0.0034258 , 0.00339893, -0.00171581, 0.00034174]) non-parametric might work easier in this case, but I don't know about the details. Are the parametric splines using a symmetric least-squares distance and the non-parametric a one-sided ? >>> tckp2 = interpolate.splrep(x,y,s=s,k=k) >>> res2 = splev(x,tckp2) >>> y array([ 0.53 , 0.546, 0.586, 0.658, 0.727, 0.801]) >>> res - y array([ 0.00034127, -0.00170635, 0.0034127 , -0.0034127 , 0.00170635, -0.00034127]) >>> res = splev(u,tckp) >>> res[1] - res2 array([ -7.21620543e-07, 2.47969098e-06, -2.90360301e-06, 5.65085112e-07, 1.05423256e-06, -4.73785097e-07]) Josef > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From mhubig at gmail.com Fri Aug 20 08:30:07 2010 From: mhubig at gmail.com (Markus Hubig) Date: Fri, 20 Aug 2010 14:30:07 +0200 Subject: [SciPy-User] Installing issue with python 2.7, numpy 1.5.0b1 in my Mac In-Reply-To: <4C6DCFDC.4050203@silveregg.co.jp> References: <4C6B8E60.80903@silveregg.co.jp> <4C6BC890.8020009@silveregg.co.jp> <4C6DCFDC.4050203@silveregg.co.jp> Message-ID: Yes I'm using Snow Leopard ... I'll try the FFLAGS this evening and giving some feedback ... On Fri, Aug 20, 2010 at 2:44 AM, David wrote: > On 08/20/2010 02:25 AM, Markus Hubig wrote: > > Hmm it seems SciPy don't like me at all ... ;-) Now the installation of > > numpy (the usual way > > without using pip) was OK (I have the numpy/core/lib/npy-pkg-config dir) > > but SciPy still fails > > to compile. But now I get gfortran and ln errors ... see: > > Hm, this one looks annoying... > > One temporary hack would be to disable ppc arch altogether in fortran, > something like: > > export FFLAGS="-Wall -ffixed-form -fno-second-underscore -arch i686 > -arch x86_64 -fPIC -O3 -funroll-loops" > export LDFLAGS=" -Wall -arch i686 -arch x86_64 -Wall -undefined > dynamic_lookup -bundle" > > I don't have a mac to test this ATM, so you may have to tweak things a > bit. The idea is to remove the -arch ppc64 flags which seem unsupported > on recent mac os x version (do you use snow leopard by any chance ?) > > cheers, > > David > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Can't read my mail? Just don't hold it that way! -------------- next part -------------- An HTML attachment was scrubbed... URL: From dpinte at enthought.com Fri Aug 20 11:27:00 2010 From: dpinte at enthought.com (Didrik Pinte) Date: Fri, 20 Aug 2010 17:27:00 +0200 Subject: [SciPy-User] Cross platform animated plot with widgets In-Reply-To: <499214819.414661281659040499.JavaMail.root@csemailbox.ucsd.edu> References: <499214819.414661281659040499.JavaMail.root@csemailbox.ucsd.edu> Message-ID: Le 13/08/2010 2:24, Emmett McQuinn a ?crit : Hello, I've been trying to create an animated plot with matplotlib that supports parameter updating with a slider. I've been able to have animation, and also create a slider (using matplotlib.widgets), but not both at the same time. I've tried various permutations and it seems exclusively either the graph animates or the widgets work and not both. Is there a good method to provide a simple lightweight GUI that provides both animation and sliders for Linux, Windows, and Mac? Hi Emmet, You should take a look at Chaco. They are a some examples using a slider and updating the content of the plot (can be much more complex than a slider with for example a zoomable legend colorbar). -- Didrik -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at pytables.org Fri Aug 20 11:29:11 2010 From: faltet at pytables.org (Francesc Alted) Date: Fri, 20 Aug 2010 17:29:11 +0200 Subject: [SciPy-User] [ANN] carray: an in-memory compressed data container Message-ID: ====================== Announcing carray 0.1 ====================== What it is ========== carray is a container for numerical data that can be compressed in-memory. The compresion process is carried out internally by Blosc, a high-performance compressor that is optimized for binary data. Having data compressed in-memory can reduce the stress of the memory subsystem. The net result is that carray operations can be faster than using a traditional ndarray object from NumPy. What's new ========== Everything ;-) This is the first public release of carray, and it is more a proof of concept to see which possibilities opens efficient compression for in-memory data containers. Currently carray 0.1 implements the basic container creation and enlargeability. Multidimensional arrays are not supported yet, and elements cannot be modified either (i.e. no `__setitem__()` support). Resources ========= Visit the main carray site repository at: http://github.com/FrancescAlted/carray You can download a source package from: http://github.com/FrancescAlted/carray/downloads Short tutorial: http://github.com/FrancescAlted/carray/blob/master/USAGE.txt Home of Blosc compressor: http://blosc.pytables.org Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- Enjoy! -- Francesc Alted -------------- next part -------------- An HTML attachment was scrubbed... URL: From kparfrey at gmail.com Fri Aug 20 11:52:49 2010 From: kparfrey at gmail.com (Kyle Parfrey) Date: Fri, 20 Aug 2010 11:52:49 -0400 Subject: [SciPy-User] Problems with 2D interpolation of data on polar grid Message-ID: Hi all, I've been having some trouble doing 2D interpolation with both interp2d and bisplrep/bisplev. My data is on a spherical polar (r, theta) grid, and I'm trying to interpolate functions similar to the vector components of a dipole field. The output looks very wrong, and I keep getting warnings like (from interp2d): Warning: No more knots can be added because the number of B-spline coefficients already exceeds the number of data points m. Probably causes: either s or m too small. (fp>s) kx,ky=1,1 nx,ny=16,9 m=90 fp=0.000000 s=0.000000 or, from bisplrep: /usr/lib/python2.6/dist-packages/scipy/interpolate/fitpack.py:763: DeprecationWarning: integer argument expected, got float tx,ty,nxest,nyest,wrk,lwrk1,lwrk2) One thing: I'm not trying to regrid---I'll need to be able to interpolate to any arbitrary point, since I'll be using this to calculate field lines. I've pasted some code below that should give some idea of the problem. I hope I haven't done something obviously stupid! Thanks in advance, Kyle ---------------------------------------------------- import numpy as np from scipy import interpolate from math import * import matplotlib.pyplot as plt ### Make polar grid ### rvec = np.arange(1.0, 11.0, 1.0) tvec = np.arange(pi/10.0, pi, pi/10.0) Nr = len(rvec) Nt = len(tvec) X = np.empty([Nr,Nt]) Y = np.empty([Nr,Nt]) Z = np.empty([Nr,Nt]) for i in range(Nr): for j in range(Nt): r = rvec[i] t = tvec[j] X[i,j] = r*sin(t) Y[i,j] = r*cos(t) Z[i,j] = cos(t)/pow(r,3) # cos(theta)/r^3: Br of dipole ### Do the interpolation ### #interp_poly = interpolate.interp2d(X,Y,Z, kind='linear') tck = interpolate.bisplrep(X,Y,Z, kx=3, ky=3) ### interpolate onto new grid ### rnew = np.arange(1.0, 10.1, 0.1) tnew = np.arange(pi/100.0, pi, pi/100.0) Nr2 = len(rnew) Nt2 = len(tnew) X2 = np.empty([Nr2,Nt2]) Y2 = np.empty([Nr2,Nt2]) Z2 = np.empty([Nr2,Nt2]) for i in range(Nr2): for j in range(Nt2): r = rnew[i] t = tnew[j] X2[i,j] = r*sin(t) Y2[i,j] = r*cos(t) #Z2[i,j] = interp_poly(X2[i,j], Y2[i,j]) Z2[i,j] = interpolate.bisplev(X2[i,j], Y2[i,j], tck) ### Pseudocolour plot ### fig = plt.figure() fig.add_subplot(111, aspect='equal') plt.pcolor(X2,Y2,Z2) plt.colorbar() plt.show() From william.t.bridgman at nasa.gov Fri Aug 20 12:06:04 2010 From: william.t.bridgman at nasa.gov (Bridgman, William T.) Date: Fri, 20 Aug 2010 12:06:04 -0400 Subject: [SciPy-User] Problem building scikits.vectorplot v1.1 Message-ID: <1EDFA30B-C806-4F74-8780-BC2410AC8C49@nasa.gov> Trying to build this on MacOS 10.5+ under python 2.6, numpy 1.3, scipy 0.7 lic_internal.c would not compile until I edited scikits/vectorplot/setup.py adding/modifying lines to find numpy headers: import numpy as np include_dirs = [',',np.get_include()], After that, lic_internal.c compiles with % python setup.py build using the setup.py within vectorplot, but % sudo python setup.py install at the top level does not seem to actually install (though it gives no obvious errors). I can't find it in site-packages. I'm tempted to just copy the scipy tree into site-packages but it looks like parts of vectorplot are scattered between build/lib/scikits/ and scikits/vectorplot/build/lib.macosx-10.5-fat-2.6 and it is unclear what items are complete. BTW, cleaning out the build/ directories and trying to build from the top-level setup.py fails to compile lic_internal.c. Suggestions? Thanks, Tom -- Dr. William T."Tom" Bridgman Scientific Visualization Studio Global Science & Technology, Inc. NASA/Goddard Space Flight Center Email: William.T.Bridgman at nasa.gov Code 610.3 Phone: 301-286-1346 Greenbelt, MD 20771 FAX: 301-286-1634 http://svs.gsfc.nasa.gov/ From josef.pktd at gmail.com Fri Aug 20 12:58:05 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 20 Aug 2010 12:58:05 -0400 Subject: [SciPy-User] Problems with 2D interpolation of data on polar grid In-Reply-To: References: Message-ID: On Fri, Aug 20, 2010 at 11:52 AM, Kyle Parfrey wrote: > Hi all, > > I've been having some trouble doing 2D interpolation with both > interp2d and bisplrep/bisplev. My data is on a spherical polar (r, > theta) grid, and I'm trying to interpolate functions similar to the > vector components of a dipole field. The output looks very wrong, and > I keep getting warnings like (from interp2d): > > Warning: ? ? No more knots can be added because the number of B-spline > coefficients > ? ?already exceeds the number of data points m. Probably causes: either > ? ?s or m too small. (fp>s) > ? ? ? ?kx,ky=1,1 nx,ny=16,9 m=90 fp=0.000000 s=0.000000 > > or, from bisplrep: > > /usr/lib/python2.6/dist-packages/scipy/interpolate/fitpack.py:763: > DeprecationWarning: integer argument expected, got float > ?tx,ty,nxest,nyest,wrk,lwrk1,lwrk2) > > One thing: I'm not trying to regrid---I'll need to be able to > interpolate to any arbitrary point, since I'll be using this to > calculate field lines. I've pasted some code below that should give > some idea of the problem. I hope I haven't done something obviously > stupid! > > Thanks in advance, > Kyle > > ---------------------------------------------------- > > import numpy as np > from scipy import interpolate > from math import * > import matplotlib.pyplot as plt > > ### Make polar grid ### > rvec = np.arange(1.0, 11.0, 1.0) > tvec = np.arange(pi/10.0, pi, pi/10.0) > Nr = len(rvec) > Nt = len(tvec) > X = np.empty([Nr,Nt]) > Y = np.empty([Nr,Nt]) > Z = np.empty([Nr,Nt]) > > for i in range(Nr): > ?for j in range(Nt): > ? ?r = rvec[i] > ? ?t = tvec[j] > ? ?X[i,j] = r*sin(t) > ? ?Y[i,j] = r*cos(t) > ? ?Z[i,j] = cos(t)/pow(r,3) # cos(theta)/r^3: Br of dipole > > > ### Do the interpolation ### > #interp_poly = interpolate.interp2d(X,Y,Z, kind='linear') > tck = interpolate.bisplrep(X,Y,Z, kx=3, ky=3) > > > ### interpolate onto new grid ### > rnew = np.arange(1.0, 10.1, 0.1) > tnew = np.arange(pi/100.0, pi, pi/100.0) > Nr2 = len(rnew) > Nt2 = len(tnew) > X2 = np.empty([Nr2,Nt2]) > Y2 = np.empty([Nr2,Nt2]) > Z2 = np.empty([Nr2,Nt2]) > for i in range(Nr2): > ?for j in range(Nt2): > ? ?r = rnew[i] > ? ?t = tnew[j] > ? ?X2[i,j] = r*sin(t) > ? ?Y2[i,j] = r*cos(t) > ? ?#Z2[i,j] = interp_poly(X2[i,j], Y2[i,j]) > ? ?Z2[i,j] = interpolate.bisplev(X2[i,j], Y2[i,j], tck) > > > ### Pseudocolour plot ### > fig = plt.figure() > fig.add_subplot(111, aspect='equal') > plt.pcolor(X2,Y2,Z2) > plt.colorbar() > plt.show() I'm not a graphical person, but if I reduce the radius ### interpolate onto new grid ### rnew = np.arange(1.0, 10.1, 0.1)[:20] the plot seems to look reasonable. plotting the original X,Y,Z also has color variation only close to the origin. contour plot also shows "action" only around the origin. looks like most of the area is flat close to zero. I don't see a problem with the spline interpolation itself. Josef > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From erin.sheldon at gmail.com Fri Aug 20 14:08:45 2010 From: erin.sheldon at gmail.com (Erin Sheldon) Date: Fri, 20 Aug 2010 14:08:45 -0400 Subject: [SciPy-User] IO of large ASCII table data In-Reply-To: References: <1282076276-sup-9537@theshire> Message-ID: <1282327403-sup-4250@theshire> Excerpts from Robert Kern's message of Tue Aug 17 16:32:18 -0400 2010: > On Tue, Aug 17, 2010 at 15:21, erin.sheldon wrote: > > Excerpts from Felix Schlesinger's message of Tue Aug 17 15:55:17 -0400 2010: > >> > The recfile package is designed specifically for this purpose: > >> > http://code.google.com/p/recfile/ > >> > >> recfile looks very useful. Any reason its not on pypi? That would make > >> maintaining it as a dependency much easier. > >> > >> Felix > > > > Felix - > > > > Interesting, I had never paid much attention to PyPI before as I have my > > own methods for dealing with dependencies. But that looks like a great > > resource and a nice way to distribute packages. > > > > I wouldn't know where to start with it though. ?I would appreciate any > > tips you could give (offline if you wish). > > http://wiki.python.org/moin/CheeseShopTutorial > > Go down to the section "Submitting Packages to the Package Index". > > I recommend removing the "ups" parts of the setup.py before making a > general release. > For those who are interested, recfile is now on pypi. http://pypi.python.org/pypi/recfile/0.40 Thanks for Robert and Felix for pointers. Erin Scott Sheldon Brookhaven National Laboratory From kyle at astro.columbia.edu Fri Aug 20 14:17:29 2010 From: kyle at astro.columbia.edu (Kyle Parfrey) Date: Fri, 20 Aug 2010 14:17:29 -0400 Subject: [SciPy-User] Problems with 2D interpolation of data on polar grid In-Reply-To: References: Message-ID: The interpolated field doesn't look as crazy in the centre of the grid, but I think it's still wrong. Have a go with the code below, in which I've fixed a few little things, and now you see the original field on the left, interpolated on the right. (I'm having trouble forcing the colour ranges to be the same in both plots.) Is the "hole" in the middle of the grid causing problems? Thanks, Kyle import numpy as np from scipy import interpolate from math import * import matplotlib.pyplot as plt ### Make polar grid ### rvec = np.linspace(1.0, 5.0, num=50) tvec = np.linspace(0, pi, num=50) Nr = len(rvec) Nt = len(tvec) X = np.empty([Nr,Nt]) Y = np.empty([Nr,Nt]) Z = np.empty([Nr,Nt]) for i in range(Nr): for j in range(Nt): r = rvec[i] t = tvec[j] X[i,j] = r*sin(t) Y[i,j] = r*cos(t) Z[i,j] = cos(t)/pow(r,3) # cos(theta)/r^3: Br of dipole ### Do the interpolation ### #interp_poly = interpolate.interp2d(X,Y,Z, kind='linear') tck = interpolate.bisplrep(X,Y,Z, kx=3, ky=3) ### interpolate onto new grid ### rnew = np.linspace(1.0, 5.0, num=100) tnew = np.linspace(0, pi, num=100) Nr2 = len(rnew) Nt2 = len(tnew) X2 = np.empty([Nr2,Nt2]) Y2 = np.empty([Nr2,Nt2]) Z2 = np.empty([Nr2,Nt2]) for i in range(Nr2): for j in range(Nt2): r = rnew[i] t = tnew[j] X2[i,j] = r*sin(t) Y2[i,j] = r*cos(t) #Z2[i,j] = interp_poly(X2[i,j], Y2[i,j]) Z2[i,j] = interpolate.bisplev(X2[i,j], Y2[i,j], tck) ### Plot pseudocolour plot ### fig = plt.figure() fig.add_subplot(121, aspect='equal') plt.pcolor(X,Y,Z) plt.colorbar() fig.add_subplot(122, aspect='equal') plt.pcolor(X2,Y2,Z2) plt.colorbar() plt.show() On 20 August 2010 12:58, wrote: > On Fri, Aug 20, 2010 at 11:52 AM, Kyle Parfrey wrote: >> Hi all, >> >> I've been having some trouble doing 2D interpolation with both >> interp2d and bisplrep/bisplev. My data is on a spherical polar (r, >> theta) grid, and I'm trying to interpolate functions similar to the >> vector components of a dipole field. The output looks very wrong, and >> I keep getting warnings like (from interp2d): >> >> Warning: ? ? No more knots can be added because the number of B-spline >> coefficients >> ? ?already exceeds the number of data points m. Probably causes: either >> ? ?s or m too small. (fp>s) >> ? ? ? ?kx,ky=1,1 nx,ny=16,9 m=90 fp=0.000000 s=0.000000 >> >> or, from bisplrep: >> >> /usr/lib/python2.6/dist-packages/scipy/interpolate/fitpack.py:763: >> DeprecationWarning: integer argument expected, got float >> ?tx,ty,nxest,nyest,wrk,lwrk1,lwrk2) >> >> One thing: I'm not trying to regrid---I'll need to be able to >> interpolate to any arbitrary point, since I'll be using this to >> calculate field lines. I've pasted some code below that should give >> some idea of the problem. I hope I haven't done something obviously >> stupid! >> >> Thanks in advance, >> Kyle >> >> ---------------------------------------------------- >> >> import numpy as np >> from scipy import interpolate >> from math import * >> import matplotlib.pyplot as plt >> >> ### Make polar grid ### >> rvec = np.arange(1.0, 11.0, 1.0) >> tvec = np.arange(pi/10.0, pi, pi/10.0) >> Nr = len(rvec) >> Nt = len(tvec) >> X = np.empty([Nr,Nt]) >> Y = np.empty([Nr,Nt]) >> Z = np.empty([Nr,Nt]) >> >> for i in range(Nr): >> ?for j in range(Nt): >> ? ?r = rvec[i] >> ? ?t = tvec[j] >> ? ?X[i,j] = r*sin(t) >> ? ?Y[i,j] = r*cos(t) >> ? ?Z[i,j] = cos(t)/pow(r,3) # cos(theta)/r^3: Br of dipole >> >> >> ### Do the interpolation ### >> #interp_poly = interpolate.interp2d(X,Y,Z, kind='linear') >> tck = interpolate.bisplrep(X,Y,Z, kx=3, ky=3) >> >> >> ### interpolate onto new grid ### >> rnew = np.arange(1.0, 10.1, 0.1) >> tnew = np.arange(pi/100.0, pi, pi/100.0) >> Nr2 = len(rnew) >> Nt2 = len(tnew) >> X2 = np.empty([Nr2,Nt2]) >> Y2 = np.empty([Nr2,Nt2]) >> Z2 = np.empty([Nr2,Nt2]) >> for i in range(Nr2): >> ?for j in range(Nt2): >> ? ?r = rnew[i] >> ? ?t = tnew[j] >> ? ?X2[i,j] = r*sin(t) >> ? ?Y2[i,j] = r*cos(t) >> ? ?#Z2[i,j] = interp_poly(X2[i,j], Y2[i,j]) >> ? ?Z2[i,j] = interpolate.bisplev(X2[i,j], Y2[i,j], tck) >> >> >> ### Pseudocolour plot ### >> fig = plt.figure() >> fig.add_subplot(111, aspect='equal') >> plt.pcolor(X2,Y2,Z2) >> plt.colorbar() >> plt.show() > > > I'm not a graphical person, but if I reduce the radius > > ### interpolate onto new grid ### > rnew = np.arange(1.0, 10.1, 0.1)[:20] > > the plot seems to look reasonable. > > plotting the original X,Y,Z also has color variation only close to the > origin. contour plot also shows "action" only around the origin. > looks like most of the area is flat close to zero. > > I don't see a problem with the spline interpolation itself. > > Josef > >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From josef.pktd at gmail.com Fri Aug 20 14:35:51 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 20 Aug 2010 14:35:51 -0400 Subject: [SciPy-User] Problems with 2D interpolation of data on polar grid In-Reply-To: References: Message-ID: On Fri, Aug 20, 2010 at 2:17 PM, Kyle Parfrey wrote: > The interpolated field doesn't look as crazy in the centre of the > grid, but I think it's still wrong. Have a go with the code below, in > which I've fixed a few little things, and > now you see the original field on the left, interpolated on the right. > (I'm having trouble forcing the colour ranges to be the same in both > plots.) > > Is the "hole" in the middle of the grid causing problems? maybe the radial placement of the points ? I don't know. adding a small positive s seems to help in the graph (I tried s= 0.001 to s=0.1) ### Do the interpolation ### #interp_poly = interpolate.interp2d(X,Y,Z, kind='linear') tck = interpolate.bisplrep(X.ravel(),Y.ravel(),Z.ravel(), kx=3, ky=3, s=0.001) looks much closer to the original but I didn't check what error you get at the original points. Josef > > Thanks, > Kyle > > > import numpy as np > from scipy import interpolate > from math import * > import matplotlib.pyplot as plt > > ### Make polar grid ### > rvec = np.linspace(1.0, 5.0, num=50) > tvec = np.linspace(0, pi, num=50) > Nr = len(rvec) > Nt = len(tvec) > X = np.empty([Nr,Nt]) > Y = np.empty([Nr,Nt]) > Z = np.empty([Nr,Nt]) > > for i in range(Nr): > ?for j in range(Nt): > ? ?r = rvec[i] > ? ?t = tvec[j] > ? ?X[i,j] = r*sin(t) > ? ?Y[i,j] = r*cos(t) > ? ?Z[i,j] = cos(t)/pow(r,3) # cos(theta)/r^3: Br of dipole > > > ### Do the interpolation ### > #interp_poly = interpolate.interp2d(X,Y,Z, kind='linear') > tck = interpolate.bisplrep(X,Y,Z, kx=3, ky=3) > > > ### interpolate onto new grid ### > rnew = np.linspace(1.0, 5.0, num=100) > tnew = np.linspace(0, pi, num=100) > Nr2 = len(rnew) > Nt2 = len(tnew) > X2 = np.empty([Nr2,Nt2]) > Y2 = np.empty([Nr2,Nt2]) > Z2 = np.empty([Nr2,Nt2]) > for i in range(Nr2): > ?for j in range(Nt2): > ? ?r = rnew[i] > ? ?t = tnew[j] > ? ?X2[i,j] = r*sin(t) > ? ?Y2[i,j] = r*cos(t) > ? ?#Z2[i,j] = interp_poly(X2[i,j], Y2[i,j]) > ? ?Z2[i,j] = interpolate.bisplev(X2[i,j], Y2[i,j], tck) > > > ### Plot pseudocolour plot ### > fig = plt.figure() > fig.add_subplot(121, aspect='equal') > plt.pcolor(X,Y,Z) > plt.colorbar() > > fig.add_subplot(122, aspect='equal') > plt.pcolor(X2,Y2,Z2) > plt.colorbar() > plt.show() > > > > > > > On 20 August 2010 12:58, ? wrote: >> On Fri, Aug 20, 2010 at 11:52 AM, Kyle Parfrey wrote: >>> Hi all, >>> >>> I've been having some trouble doing 2D interpolation with both >>> interp2d and bisplrep/bisplev. My data is on a spherical polar (r, >>> theta) grid, and I'm trying to interpolate functions similar to the >>> vector components of a dipole field. The output looks very wrong, and >>> I keep getting warnings like (from interp2d): >>> >>> Warning: ? ? No more knots can be added because the number of B-spline >>> coefficients >>> ? ?already exceeds the number of data points m. Probably causes: either >>> ? ?s or m too small. (fp>s) >>> ? ? ? ?kx,ky=1,1 nx,ny=16,9 m=90 fp=0.000000 s=0.000000 >>> >>> or, from bisplrep: >>> >>> /usr/lib/python2.6/dist-packages/scipy/interpolate/fitpack.py:763: >>> DeprecationWarning: integer argument expected, got float >>> ?tx,ty,nxest,nyest,wrk,lwrk1,lwrk2) >>> >>> One thing: I'm not trying to regrid---I'll need to be able to >>> interpolate to any arbitrary point, since I'll be using this to >>> calculate field lines. I've pasted some code below that should give >>> some idea of the problem. I hope I haven't done something obviously >>> stupid! >>> >>> Thanks in advance, >>> Kyle >>> >>> ---------------------------------------------------- >>> >>> import numpy as np >>> from scipy import interpolate >>> from math import * >>> import matplotlib.pyplot as plt >>> >>> ### Make polar grid ### >>> rvec = np.arange(1.0, 11.0, 1.0) >>> tvec = np.arange(pi/10.0, pi, pi/10.0) >>> Nr = len(rvec) >>> Nt = len(tvec) >>> X = np.empty([Nr,Nt]) >>> Y = np.empty([Nr,Nt]) >>> Z = np.empty([Nr,Nt]) >>> >>> for i in range(Nr): >>> ?for j in range(Nt): >>> ? ?r = rvec[i] >>> ? ?t = tvec[j] >>> ? ?X[i,j] = r*sin(t) >>> ? ?Y[i,j] = r*cos(t) >>> ? ?Z[i,j] = cos(t)/pow(r,3) # cos(theta)/r^3: Br of dipole >>> >>> >>> ### Do the interpolation ### >>> #interp_poly = interpolate.interp2d(X,Y,Z, kind='linear') >>> tck = interpolate.bisplrep(X,Y,Z, kx=3, ky=3) >>> >>> >>> ### interpolate onto new grid ### >>> rnew = np.arange(1.0, 10.1, 0.1) >>> tnew = np.arange(pi/100.0, pi, pi/100.0) >>> Nr2 = len(rnew) >>> Nt2 = len(tnew) >>> X2 = np.empty([Nr2,Nt2]) >>> Y2 = np.empty([Nr2,Nt2]) >>> Z2 = np.empty([Nr2,Nt2]) >>> for i in range(Nr2): >>> ?for j in range(Nt2): >>> ? ?r = rnew[i] >>> ? ?t = tnew[j] >>> ? ?X2[i,j] = r*sin(t) >>> ? ?Y2[i,j] = r*cos(t) >>> ? ?#Z2[i,j] = interp_poly(X2[i,j], Y2[i,j]) >>> ? ?Z2[i,j] = interpolate.bisplev(X2[i,j], Y2[i,j], tck) >>> >>> >>> ### Pseudocolour plot ### >>> fig = plt.figure() >>> fig.add_subplot(111, aspect='equal') >>> plt.pcolor(X2,Y2,Z2) >>> plt.colorbar() >>> plt.show() >> >> >> I'm not a graphical person, but if I reduce the radius >> >> ### interpolate onto new grid ### >> rnew = np.arange(1.0, 10.1, 0.1)[:20] >> >> the plot seems to look reasonable. >> >> plotting the original X,Y,Z also has color variation only close to the >> origin. contour plot also shows "action" only around the origin. >> looks like most of the area is flat close to zero. >> >> I don't see a problem with the spline interpolation itself. >> >> Josef >> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From pav at iki.fi Sat Aug 21 14:44:44 2010 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 21 Aug 2010 18:44:44 +0000 (UTC) Subject: [SciPy-User] Problems with 2D interpolation of data on polar grid References: Message-ID: Fri, 20 Aug 2010 11:52:49 -0400, Kyle Parfrey wrote: > I've been having some trouble doing 2D interpolation with both interp2d > and bisplrep/bisplev. My data is on a spherical polar (r, theta) grid, > and I'm trying to interpolate functions similar to the vector components > of a dipole field. The output looks very wrong, and I keep getting > warnings like (from interp2d): Consider using RectBivariateSpline, if your data indeed is on a regular grid. This way FITPACK might choose better knots... If your data, furthermore, is given on a homogeneous grid, another alternative is scipy.ndimage.map_coordinates. Using that requires some thinking, though. -- Pauli Virtanen From spons at utm.csic.es Sat Aug 21 21:18:04 2010 From: spons at utm.csic.es (Sergi Pons Freixes) Date: Sat, 21 Aug 2010 18:18:04 -0700 Subject: [SciPy-User] Populating a recarray from 0 size Message-ID: Hi everybody, I'm interested in populating a recarray a row at a time. I thought in using: ------- # Creation of empty recarray names = ["cruise", "SEQ", "param", "r2", "stderr", "slope", "intercept" formats = ["i4", "i4", "S10", "f4", "f4", "f4", "f4"] statsc = scipy.empty(0, dtype={"names":names, "formats":formats}) # Adding row and setting values statsc = scipy.resize(statsc, statsc.size + 1) statsc["cruise"][-1] = cr statsc["SEQ"][-1] = ca statsc["param"][-1] = yvar ... ------- But scipy.resize complains with: statsc = scipy.resize(statsc, statsc.size + 1) File "/usr/lib/python2.6/site-packages/numpy/core/fromnumeric.py", line 833, in resize if not Na: return mu.zeros(new_shape, a.dtype.char) ValueError: Empty data-type So, is scipy not happy because statsc.size is 0 on the first resize? In this case, how could I overcome this limitation? Regards, Sergi -- Sergi Pons Freixes Ph.D. Student Marine Technology Unit Centre Mediterrani d'Investigacions Marines i Ambientals (CMIMA-CSIC) Pg. Mar?tim Barceloneta, 37-49 E-08003 Barcelona (Spain) Ph.? +34 93 230 95 00 (ext. 1510) Fax. +34 93 230 95 55 spons at utm.csic.es http://www.utm.csic.es From josef.pktd at gmail.com Sat Aug 21 21:29:23 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 21 Aug 2010 21:29:23 -0400 Subject: [SciPy-User] Populating a recarray from 0 size In-Reply-To: References: Message-ID: On Sat, Aug 21, 2010 at 9:18 PM, Sergi Pons Freixes wrote: > Hi everybody, > > I'm interested in populating a recarray a row at a time. I thought in using: > > ------- > # Creation of empty recarray > names = ["cruise", "SEQ", "param", "r2", "stderr", "slope", "intercept" > formats = ["i4", "i4", "S10", "f4", "f4", "f4", "f4"] > statsc = scipy.empty(0, dtype={"names":names, "formats":formats}) > > # Adding row and setting values > statsc = scipy.resize(statsc, statsc.size + 1) > statsc["cruise"][-1] = cr > statsc["SEQ"][-1] = ca > statsc["param"][-1] = yvar > ... > ------- > > But scipy.resize complains with: > statsc = scipy.resize(statsc, statsc.size + 1) > ?File "/usr/lib/python2.6/site-packages/numpy/core/fromnumeric.py", > line 833, in resize > ? ?if not Na: return mu.zeros(new_shape, a.dtype.char) > ValueError: Empty data-type > > So, is scipy not happy because statsc.size is 0 on the first resize? > In this case, how could I overcome this limitation? BTW: These are numpy functions. I don't know the answer, but it would be more efficient to correctly size the empty array, if you know the size, or to build up the array first as a list of tuples before converting to an array. Josef > > Regards, > Sergi > > -- > Sergi Pons Freixes > Ph.D. Student > Marine Technology Unit > Centre Mediterrani d'Investigacions Marines i Ambientals (CMIMA-CSIC) > Pg. Mar?tim Barceloneta, 37-49 > E-08003 Barcelona (Spain) > Ph.? +34 93 230 95 00 (ext. 1510) > Fax. +34 93 230 95 55 > spons at utm.csic.es > http://www.utm.csic.es > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From asmund.hjulstad at gmail.com Sun Aug 22 11:19:04 2010 From: asmund.hjulstad at gmail.com (=?ISO-8859-1?Q?=C5smund_Hjulstad?=) Date: Sun, 22 Aug 2010 18:19:04 +0300 Subject: [SciPy-User] Populating a recarray from 0 size In-Reply-To: References: Message-ID: 2010/8/22 > On Sat, Aug 21, 2010 at 9:18 PM, Sergi Pons Freixes > wrote: > > Hi everybody, > > > > I'm interested in populating a recarray a row at a time. I thought in > using: > > > I don't know the answer, but it would be more efficient to correctly > size the empty array, if you know the size, or to build up the array > first as a list of tuples before converting to an array. > > My approach in this kind of situation has been using pytables, creating an empty table with a specific dtype, and then populating the table with the append method. It seems very efficient, and quite convenient in situations where I eventually will need to persist data to disk somehow. Reading the table into an in memory recarray is then easily done using data = h5file.root.datatable[:].view(numpy.recarray) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Sun Aug 22 13:44:54 2010 From: ben.root at ou.edu (Benjamin Root) Date: Sun, 22 Aug 2010 12:44:54 -0500 Subject: [SciPy-User] Populating a recarray from 0 size In-Reply-To: References: Message-ID: On Sat, Aug 21, 2010 at 8:18 PM, Sergi Pons Freixes wrote: > Hi everybody, > > I'm interested in populating a recarray a row at a time. I thought in > using: > > ------- > # Creation of empty recarray > names = ["cruise", "SEQ", "param", "r2", "stderr", "slope", "intercept" > formats = ["i4", "i4", "S10", "f4", "f4", "f4", "f4"] > statsc = scipy.empty(0, dtype={"names":names, "formats":formats}) > > # Adding row and setting values > statsc = scipy.resize(statsc, statsc.size + 1) > statsc["cruise"][-1] = cr > statsc["SEQ"][-1] = ca > statsc["param"][-1] = yvar > ... > ------- > > But scipy.resize complains with: > statsc = scipy.resize(statsc, statsc.size + 1) > File "/usr/lib/python2.6/site-packages/numpy/core/fromnumeric.py", > line 833, in resize > if not Na: return mu.zeros(new_shape, a.dtype.char) > ValueError: Empty data-type > > So, is scipy not happy because statsc.size is 0 on the first resize? > In this case, how could I overcome this limitation? > > Regards, > Sergi > > Sergi, I had a similar need recently. I was populating recarrays where I didn't know the final size beforehand. The way I solved the problem was to create an iterator function and use .fromiter() to build the array. A simplified example: import numpy class foobar(object) : def __init__(self) : self._myval = 0 def __iter__(self) : return self def next(self) : if self._myval >= 7 : raise StopIteration self._myval += 1 return self._myval gen = foobar() a = numpy.fromiter(gen, dtype=[('Column', 'i4')]) print a However... for some reason, I can't seem to get this working. Maybe someone else can spot my error? Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From spons at utm.csic.es Sun Aug 22 14:23:43 2010 From: spons at utm.csic.es (Sergi Pons Freixes) Date: Sun, 22 Aug 2010 11:23:43 -0700 Subject: [SciPy-User] Populating a recarray from 0 size In-Reply-To: References: Message-ID: On Sat, Aug 21, 2010 at 6:29 PM, wrote: > BTW: These are numpy functions. Oh, sorry, I am so used to call numpy functions via scipy that I forgot about it. > > I don't know the answer, but it would be more efficient to correctly > size the empty array, if you know the size, or to build up the array > first as a list of tuples before converting to an array. The problem is that the size is not known. I'll think about the list of tuples. On Sun, Aug 22, 2010 at 8:19 AM, ?smund Hjulstad wrote: > > > My approach in this kind of situation has been using pytables, creating an > empty table with a specific dtype, and then populating the table with the > append method. It seems very efficient, and quite convenient in situations > where I eventually will need to persist data to disk somehow. > Reading the table into an in memory recarray is then easily done using > data = h5file.root.datatable[:].view(numpy.recarray) > Mmm, it sounds a bit overkill to use pytables just for having a recarray with dynamic size. Anyway, I'll have a look at it, thank you. I would also appreciate is a reasoning of why is scipy.resize (e.g., numpy.resize) raising this exception. The idea increasing the size of a 0-size array doesn't sound very weird to me, so I was shocked that i wasn't working. Actually, I've been successfully using the method array.resize(newsize) previously, but now I'm "forced" to use scipy.resize because of the "ValueError: cannot resize an array that has been referenced or is referencing another array in this way." issue (see [1]). If this conversation is more appropriate to be developed in the numpy mailing list, please let me now and I will migrate it. Regards, Sergi [1] http://www.scipy.org/Numpy_Example_List#head-d5585278b02f43f78eb75a4fdc337fdc12417426 From jkington at wisc.edu Sun Aug 22 14:46:05 2010 From: jkington at wisc.edu (Joe Kington) Date: Sun, 22 Aug 2010 13:46:05 -0500 Subject: [SciPy-User] Populating a recarray from 0 size In-Reply-To: References: Message-ID: On Sun, Aug 22, 2010 at 12:44 PM, Benjamin Root wrote: > On Sat, Aug 21, 2010 at 8:18 PM, Sergi Pons Freixes wrote: > >> Hi everybody, >> >> I'm interested in populating a recarray a row at a time. I thought in >> using: >> >> ------- >> # Creation of empty recarray >> names = ["cruise", "SEQ", "param", "r2", "stderr", "slope", "intercept" >> formats = ["i4", "i4", "S10", "f4", "f4", "f4", "f4"] >> statsc = scipy.empty(0, dtype={"names":names, "formats":formats}) >> >> # Adding row and setting values >> statsc = scipy.resize(statsc, statsc.size + 1) >> statsc["cruise"][-1] = cr >> statsc["SEQ"][-1] = ca >> statsc["param"][-1] = yvar >> ... >> ------- >> >> But scipy.resize complains with: >> statsc = scipy.resize(statsc, statsc.size + 1) >> File "/usr/lib/python2.6/site-packages/numpy/core/fromnumeric.py", >> line 833, in resize >> if not Na: return mu.zeros(new_shape, a.dtype.char) >> ValueError: Empty data-type >> >> So, is scipy not happy because statsc.size is 0 on the first resize? >> In this case, how could I overcome this limitation? >> >> Regards, >> Sergi >> >> > Sergi, > > I had a similar need recently. I was populating recarrays where I didn't > know the final size beforehand. The way I solved the problem was to create > an iterator function and use .fromiter() to build the array. > > A simplified example: > > import numpy > > class foobar(object) : > def __init__(self) : > self._myval = 0 > > def __iter__(self) : > return self > > def next(self) : > if self._myval >= 7 : > raise StopIteration > > self._myval += 1 > return self._myval > > gen = foobar() > a = numpy.fromiter(gen, dtype=[('Column', 'i4')]) > print a > > > However... for some reason, I can't seem to get this working. Maybe > someone else can spot my error? > I think the iterator needs to return a tuple, rather than a "bare" int. At any rate, if you change "return self._myval" to "return (self._myval,)", things work perfectly. > > Ben Root > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sun Aug 22 14:52:09 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 22 Aug 2010 14:52:09 -0400 Subject: [SciPy-User] Populating a recarray from 0 size In-Reply-To: References: Message-ID: On Sun, Aug 22, 2010 at 2:23 PM, Sergi Pons Freixes wrote: > On Sat, Aug 21, 2010 at 6:29 PM, ? wrote: >> BTW: These are numpy functions. > > Oh, sorry, I am so used to call numpy functions via scipy that I > forgot about it. > >> >> I don't know the answer, but it would be more efficient to correctly >> size the empty array, if you know the size, or to build up the array >> first as a list of tuples before converting to an array. > > The problem is that the size is not known. I'll think about the list of tuples. > > > On Sun, Aug 22, 2010 at 8:19 AM, ?smund Hjulstad > wrote: >> >> >> My approach in this kind of situation has been using pytables, creating an >> empty table with a specific dtype, and then populating the table with the >> append method. It seems very efficient, and quite convenient in situations >> where I eventually will need to persist data to disk somehow. >> Reading the table into an in memory recarray is then easily done using >> data = h5file.root.datatable[:].view(numpy.recarray) >> > > Mmm, it sounds a bit overkill to use pytables just for having a > recarray with dynamic size. Anyway, I'll have a look at it, thank you. > > > I would also appreciate is a reasoning of why is scipy.resize (e.g., > numpy.resize) raising this exception. The idea increasing the size of > a 0-size array doesn't sound very weird to me, so I was shocked that i > wasn't working. Actually, I've been successfully using the method > array.resize(newsize) previously, but now I'm "forced" to use > scipy.resize because of the "ValueError: cannot resize an array that > has been referenced or is referencing another array in this way." > issue (see [1]). I think it should be a bug report on the numpy trac, given that the method works but not the function. To use the method, you might also be able to make a copy yourself to break the second reference. (Not tried out). I still think copying the entire array whenever a new item is added is not a good idea (very costly in a loop). Josef > > If this conversation is more appropriate to be developed in the numpy > mailing list, please let me now and I will migrate it. > > Regards, > Sergi > > [1] http://www.scipy.org/Numpy_Example_List#head-d5585278b02f43f78eb75a4fdc337fdc12417426 > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From ben.root at ou.edu Sun Aug 22 15:13:36 2010 From: ben.root at ou.edu (Benjamin Root) Date: Sun, 22 Aug 2010 14:13:36 -0500 Subject: [SciPy-User] Populating a recarray from 0 size In-Reply-To: References: Message-ID: On Sun, Aug 22, 2010 at 1:46 PM, Joe Kington wrote: > > > On Sun, Aug 22, 2010 at 12:44 PM, Benjamin Root wrote: > >> On Sat, Aug 21, 2010 at 8:18 PM, Sergi Pons Freixes wrote: >> >>> Hi everybody, >>> >>> I'm interested in populating a recarray a row at a time. I thought in >>> using: >>> >>> ------- >>> # Creation of empty recarray >>> names = ["cruise", "SEQ", "param", "r2", "stderr", "slope", "intercept" >>> formats = ["i4", "i4", "S10", "f4", "f4", "f4", "f4"] >>> statsc = scipy.empty(0, dtype={"names":names, "formats":formats}) >>> >>> # Adding row and setting values >>> statsc = scipy.resize(statsc, statsc.size + 1) >>> statsc["cruise"][-1] = cr >>> statsc["SEQ"][-1] = ca >>> statsc["param"][-1] = yvar >>> ... >>> ------- >>> >>> But scipy.resize complains with: >>> statsc = scipy.resize(statsc, statsc.size + 1) >>> File "/usr/lib/python2.6/site-packages/numpy/core/fromnumeric.py", >>> line 833, in resize >>> if not Na: return mu.zeros(new_shape, a.dtype.char) >>> ValueError: Empty data-type >>> >>> So, is scipy not happy because statsc.size is 0 on the first resize? >>> In this case, how could I overcome this limitation? >>> >>> Regards, >>> Sergi >>> >>> >> Sergi, >> >> I had a similar need recently. I was populating recarrays where I didn't >> know the final size beforehand. The way I solved the problem was to create >> an iterator function and use .fromiter() to build the array. >> >> A simplified example: >> >> import numpy >> >> class foobar(object) : >> def __init__(self) : >> self._myval = 0 >> >> def __iter__(self) : >> return self >> >> def next(self) : >> if self._myval >= 7 : >> raise StopIteration >> >> self._myval += 1 >> return self._myval >> >> gen = foobar() >> a = numpy.fromiter(gen, dtype=[('Column', 'i4')]) >> print a >> >> >> However... for some reason, I can't seem to get this working. Maybe >> someone else can spot my error? >> > > I think the iterator needs to return a tuple, rather than a "bare" int. At > any rate, if you change "return self._myval" to "return (self._myval,)", > things work perfectly. > Ah, right! An even easier example would be: def foobar(start, end) : curVal = start while curVal < end : yield (curVal,) curVal += 1 gen = foo(1, 10) a = numpy.fromiter(gen, [('Column', 'i4')]) print a['Column'] Which woulld output: [1 2 3 4 5 6 7 8 9] I hope that helps! Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Sun Aug 22 15:16:04 2010 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Sun, 22 Aug 2010 12:16:04 -0700 Subject: [SciPy-User] Populating a recarray from 0 size In-Reply-To: References: Message-ID: <4C717774.2040704@noaa.gov> Sergi Pons Freixes wrote: > The problem is that the size is not known. I'll think about the list of tuples. It is the generally accepted method for building up an array -- build a list first, then convert to an array. However, I think it would be great to have a built-in way to do this with numpy efficiently, particularly for things like recarrays and data types that pure python does not have. > I would also appreciate is a reasoning of why is scipy.resize (e.g., > numpy.resize) raising this exception. true -- probably a bug/missing feature. However, calling resize on every addition is likely to be slow and inefficient anyway -- it has to re-allocate memory every single time. The way this is usually handled (and is handled by python lists...) is to pre-allocate more than required at first, and then when that amount gets full, re-allocate some more -- this way you only re-allocate every once in a while, rather than at every append -- at the expense of some wasted memory. With that in mind, I wrote a pure python numpy array "accumulator", which used .resize() under the hood, but pre-allocates. The performance isn't' great, but should be better than calling a re-size() every time. I'd like to see something like this written in C some day -- but who knows if I'll ever get around to it... Enclosed, if you want to try it out. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- A non-text attachment was scrubbed... Name: Accumulator.zip Type: application/zip Size: 4701 bytes Desc: not available URL: From david_baddeley at yahoo.com.au Sun Aug 22 16:44:02 2010 From: david_baddeley at yahoo.com.au (David Baddeley) Date: Sun, 22 Aug 2010 13:44:02 -0700 (PDT) Subject: [SciPy-User] Populating a recarray from 0 size In-Reply-To: References: Message-ID: <896848.14025.qm@web33003.mail.mud.yahoo.com> Is there any way of knowing the final size in advance so you can pre-allocate the array? Or at least guessing it with reasonable accuracy so that you can pre-allocate a slightly larger array and then cut it down to size? Either of these is likely to be significantly faster than resizing an array at each iteration. Something else to look at would be the hstack/vstack functions, although you pay a similar performance penalty as with resizing. David ----- Original Message ---- From: Sergi Pons Freixes To: SciPy Users List Sent: Sun, 22 August, 2010 1:18:04 PM Subject: [SciPy-User] Populating a recarray from 0 size Hi everybody, I'm interested in populating a recarray a row at a time. I thought in using: ------- # Creation of empty recarray names = ["cruise", "SEQ", "param", "r2", "stderr", "slope", "intercept" formats = ["i4", "i4", "S10", "f4", "f4", "f4", "f4"] statsc = scipy.empty(0, dtype={"names":names, "formats":formats}) # Adding row and setting values statsc = scipy.resize(statsc, statsc.size + 1) statsc["cruise"][-1] = cr statsc["SEQ"][-1] = ca statsc["param"][-1] = yvar ... ------- But scipy.resize complains with: statsc = scipy.resize(statsc, statsc.size + 1) File "/usr/lib/python2.6/site-packages/numpy/core/fromnumeric.py", line 833, in resize if not Na: return mu.zeros(new_shape, a.dtype.char) ValueError: Empty data-type So, is scipy not happy because statsc.size is 0 on the first resize? In this case, how could I overcome this limitation? Regards, Sergi -- Sergi Pons Freixes Ph.D. Student Marine Technology Unit Centre Mediterrani d'Investigacions Marines i Ambientals (CMIMA-CSIC) Pg. Mar?tim Barceloneta, 37-49 E-08003 Barcelona (Spain) Ph. +34 93 230 95 00 (ext. 1510) Fax. +34 93 230 95 55 spons at utm.csic.es http://www.utm.csic.es _______________________________________________ SciPy-User mailing list SciPy-User at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user From ben.root at ou.edu Sun Aug 22 17:41:42 2010 From: ben.root at ou.edu (Benjamin Root) Date: Sun, 22 Aug 2010 16:41:42 -0500 Subject: [SciPy-User] Populating a recarray from 0 size In-Reply-To: References: Message-ID: On Sun, Aug 22, 2010 at 2:13 PM, Benjamin Root wrote: > On Sun, Aug 22, 2010 at 1:46 PM, Joe Kington wrote: > >> >> >> On Sun, Aug 22, 2010 at 12:44 PM, Benjamin Root wrote: >> >>> On Sat, Aug 21, 2010 at 8:18 PM, Sergi Pons Freixes wrote: >>> >>>> Hi everybody, >>>> >>>> I'm interested in populating a recarray a row at a time. I thought in >>>> using: >>>> >>>> ------- >>>> # Creation of empty recarray >>>> names = ["cruise", "SEQ", "param", "r2", "stderr", "slope", "intercept" >>>> formats = ["i4", "i4", "S10", "f4", "f4", "f4", "f4"] >>>> statsc = scipy.empty(0, dtype={"names":names, "formats":formats}) >>>> >>>> # Adding row and setting values >>>> statsc = scipy.resize(statsc, statsc.size + 1) >>>> statsc["cruise"][-1] = cr >>>> statsc["SEQ"][-1] = ca >>>> statsc["param"][-1] = yvar >>>> ... >>>> ------- >>>> >>>> But scipy.resize complains with: >>>> statsc = scipy.resize(statsc, statsc.size + 1) >>>> File "/usr/lib/python2.6/site-packages/numpy/core/fromnumeric.py", >>>> line 833, in resize >>>> if not Na: return mu.zeros(new_shape, a.dtype.char) >>>> ValueError: Empty data-type >>>> >>>> So, is scipy not happy because statsc.size is 0 on the first resize? >>>> In this case, how could I overcome this limitation? >>>> >>>> Regards, >>>> Sergi >>>> >>>> >>> Sergi, >>> >>> I had a similar need recently. I was populating recarrays where I didn't >>> know the final size beforehand. The way I solved the problem was to create >>> an iterator function and use .fromiter() to build the array. >>> >>> A simplified example: >>> >>> import numpy >>> >>> class foobar(object) : >>> def __init__(self) : >>> self._myval = 0 >>> >>> def __iter__(self) : >>> return self >>> >>> def next(self) : >>> if self._myval >= 7 : >>> raise StopIteration >>> >>> self._myval += 1 >>> return self._myval >>> >>> gen = foobar() >>> a = numpy.fromiter(gen, dtype=[('Column', 'i4')]) >>> print a >>> >>> >>> However... for some reason, I can't seem to get this working. Maybe >>> someone else can spot my error? >>> >> >> I think the iterator needs to return a tuple, rather than a "bare" int. >> At any rate, if you change "return self._myval" to "return (self._myval,)", >> things work perfectly. >> > > Ah, right! > > An even easier example would be: > > def foobar(start, end) : > curVal = start > while curVal < end : > yield (curVal,) > curVal += 1 > > gen = foo(1, 10) > a = numpy.fromiter(gen, [('Column', 'i4')]) > print a['Column'] > > Which woulld output: > > [1 2 3 4 5 6 7 8 9] > > I hope that helps! > Ben Root > > I should also add that the .fromiter() method is extremely flexible. It is perfect for those who don't know the number of elements ahead of time, but it also works for those who do know how many elements they will be loading. One can pass a "count" keyword to the .fromiter() to have the function pre-allocate the array for you. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From brennan.williams at visualreservoir.com Sun Aug 22 18:56:30 2010 From: brennan.williams at visualreservoir.com (Brennan Williams) Date: Mon, 23 Aug 2010 10:56:30 +1200 Subject: [SciPy-User] scipy.interpolate.Rbf In-Reply-To: References: <4C632148.3070504@visualreservoir.com> Message-ID: <4C71AB1E.9040907@visualreservoir.com> On 12/08/2010 10:24 a.m., Robert Kern wrote: > On Wed, Aug 11, 2010 at 17:16, Brennan Williams > wrote: >> I'm coding up using Rbf where the number of coordinates x,y,z.... will >> vary from one dataset to another. >> So I may have... >> >> rbfi=Rbf(x,y,z,d) for one dataset >> >> and then.... >> >> rbfi=Rbf(u,v,w,x,y,z,d) for another >> >> This is all inside a Traits GUI app rather than in a user-editable script. >> So how should I do this? > Collect the arguments in a list and use Rbf(*args). > See attached code example. I've put the arrays into a list and then called Rbf but I'm getting... Traceback (most recent call last): File "F:\dev\rezen\code\testrbf.py", line 36, in rbfi=Rbf(alist) File "C:\Python26\lib\site-packages\scipy\interpolate\rbf.py", line 176, in __init__ 'All arrays must be equal length' AssertionError: All arrays must be equal length and as far as I can tell my arrays are float32 arrays of equal length. So is my syntax in calling Rbf incorrect? Brennan -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: testrbf.py URL: From robert.kern at gmail.com Sun Aug 22 20:54:55 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 22 Aug 2010 19:54:55 -0500 Subject: [SciPy-User] scipy.interpolate.Rbf In-Reply-To: <4C71AB1E.9040907@visualreservoir.com> References: <4C632148.3070504@visualreservoir.com> <4C71AB1E.9040907@visualreservoir.com> Message-ID: On Sun, Aug 22, 2010 at 17:56, Brennan Williams wrote: > > On 12/08/2010 10:24 a.m., Robert Kern wrote: >> >> On Wed, Aug 11, 2010 at 17:16, Brennan Williams >> ?wrote: >>> >>> ?I'm coding up using Rbf where the number of coordinates x,y,z.... will >>> vary from one dataset to another. >>> So I may have... >>> >>> rbfi=Rbf(x,y,z,d) for one dataset >>> >>> and then.... >>> >>> rbfi=Rbf(u,v,w,x,y,z,d) for another >>> >>> This is all inside a Traits GUI app rather than in a user-editable >>> script. >>> So how should I do this? >> >> Collect the arguments in a list and use Rbf(*args). >> > See attached code example. I've put the arrays into a list and then called > Rbf but I'm getting... > > Traceback (most recent call last): > ?File "F:\dev\rezen\code\testrbf.py", line 36, in > ? ?rbfi=Rbf(alist) > ?File "C:\Python26\lib\site-packages\scipy\interpolate\rbf.py", line 176, in > __init__ > ? ?'All arrays must be equal length' > AssertionError: All arrays must be equal length > > and as far as I can tell my arrays are float32 arrays of equal length. So is > my syntax in calling Rbf incorrect? You didn't do what I said. --- testrbf.py 2010-08-22 19:47:02.000000000 -0500 +++ correctrbf.py 2010-08-22 19:53:31.000000000 -0500 @@ -33,5 +33,5 @@ alist.append(a6) print 'alist' print alist -rbfi=Rbf(alist) +rbfi=Rbf(*alist) print 'rbfi=',rbfi -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From brennan.williams at visualreservoir.com Sun Aug 22 21:29:29 2010 From: brennan.williams at visualreservoir.com (Brennan Williams) Date: Mon, 23 Aug 2010 13:29:29 +1200 Subject: [SciPy-User] scipy.interpolate.Rbf In-Reply-To: References: <4C632148.3070504@visualreservoir.com> <4C71AB1E.9040907@visualreservoir.com> Message-ID: <4C71CEF9.6060101@visualreservoir.com> On 23/08/2010 12:54 p.m., Robert Kern wrote: > On Sun, Aug 22, 2010 at 17:56, Brennan Williams > wrote: >> On 12/08/2010 10:24 a.m., Robert Kern wrote: >>> On Wed, Aug 11, 2010 at 17:16, Brennan Williams >>> wrote: >>>> I'm coding up using Rbf where the number of coordinates x,y,z.... will >>>> vary from one dataset to another. >>>> So I may have... >>>> >>>> rbfi=Rbf(x,y,z,d) for one dataset >>>> >>>> and then.... >>>> >>>> rbfi=Rbf(u,v,w,x,y,z,d) for another >>>> >>>> This is all inside a Traits GUI app rather than in a user-editable >>>> script. >>>> So how should I do this? >>> Collect the arguments in a list and use Rbf(*args). >>> >> See attached code example. I've put the arrays into a list and then called >> Rbf but I'm getting... >> >> Traceback (most recent call last): >> File "F:\dev\rezen\code\testrbf.py", line 36, in >> rbfi=Rbf(alist) >> File "C:\Python26\lib\site-packages\scipy\interpolate\rbf.py", line 176, in >> __init__ >> 'All arrays must be equal length' >> AssertionError: All arrays must be equal length >> >> and as far as I can tell my arrays are float32 arrays of equal length. So is >> my syntax in calling Rbf incorrect? > You didn't do what I said. Fair enough. > --- testrbf.py 2010-08-22 19:47:02.000000000 -0500 > +++ correctrbf.py 2010-08-22 19:53:31.000000000 -0500 > @@ -33,5 +33,5 @@ > alist.append(a6) > print 'alist' > print alist > -rbfi=Rbf(alist) > +rbfi=Rbf(*alist) > print 'rbfi=',rbfi > Corrected, but now I get... Traceback (most recent call last): File "F:\dev\rezen\code\testrbf.py", line 40, in rbfi2=Rbf(*alist) File "C:\Python26\lib\site-packages\scipy\interpolate\rbf.py", line 191, in __init__ self.A = self._init_function(r) - eye(self.N)*self.smooth File "C:\Python26\lib\site-packages\scipy\interpolate\rbf.py", line 164, in _init_function a0 = self._function(r) TypeError: _h_multiquadric() takes exactly 2 arguments (3 given) BTW I also tried out the 2D example from http://www.scipy.org/Cookbook/RadialBasisFunctions and I'm getting the same message. I'm using EPD 6.2-2 (Python 2.6.5) under Win 7 32-bit which has scipy 0.8.0b1 and numpy 1.4.0 Brennan From robert.kern at gmail.com Sun Aug 22 21:40:56 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 22 Aug 2010 20:40:56 -0500 Subject: [SciPy-User] scipy.interpolate.Rbf In-Reply-To: <4C71CEF9.6060101@visualreservoir.com> References: <4C632148.3070504@visualreservoir.com> <4C71AB1E.9040907@visualreservoir.com> <4C71CEF9.6060101@visualreservoir.com> Message-ID: It's a bug in that version of scipy. See the SVN for the fix or explicitly use function='multiquadric' after the *args. Robert Kern On Aug 22, 2010 8:29 PM, "Brennan Williams" < brennan.williams at visualreservoir.com> wrote: On 23/08/2010 12:54 p.m., Robert Kern wrote: > On Sun, Aug 22, 2010 at 17:56, Brennan Williams > <... Fair enough. > --- testrbf.py 2010-08-22 19:47:02.000000000 -0500 > +++ correctrbf.py 2010-08-22 19:53:31.... Corrected, but now I get... Traceback (most recent call last): File "F:\dev\rezen\code\testrbf.py", line 40, in rbfi2=Rbf(*alist) File "C:\Python26\lib\site-packages\scipy\interpolate\rbf.py", line 191, in __init__ self.A = self._init_function(r) - eye(self.N)*self.smooth File "C:\Python26\lib\site-packages\scipy\interpolate\rbf.py", line 164, in _init_function a0 = self._function(r) TypeError: _h_multiquadric() takes exactly 2 arguments (3 given) BTW I also tried out the 2D example from http://www.scipy.org/Cookbook/RadialBasisFunctions and I'm getting the same message. I'm using EPD 6.2-2 (Python 2.6.5) under Win 7 32-bit which has scipy 0.8.0b1 and numpy 1.4.0 Brennan _______________________________________________ SciPy-User mailing list SciPy-User at scip... -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Mon Aug 23 04:35:00 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 23 Aug 2010 10:35:00 +0200 Subject: [SciPy-User] Problems with 2D interpolation of data on polar grid In-Reply-To: References: Message-ID: On 21 August 2010 20:44, Pauli Virtanen wrote: > If your data, furthermore, is given on a homogeneous grid, another > alternative is scipy.ndimage.map_coordinates. Using that requires some > thinking, though. We use this routine in scikits.image.transform.homography, which serves as a good example on how to setup the coordinates: http://github.com/stefanv/scikits.image/blob/master/scikits/image/transform/project.py Regards St?fan From david.d.marsh at huskymail.uconn.edu Mon Aug 23 10:27:34 2010 From: david.d.marsh at huskymail.uconn.edu (David Marsh) Date: Mon, 23 Aug 2010 10:27:34 -0400 (EDT) Subject: [SciPy-User] Buffet Style Scipy? Message-ID: Hello, I wasn't sure where to start looking for this answer so I figured I'd send my question to the mailing list. I'm writing a small program that requires the use of SciPy but is currently limited to the interpolation library. The rest of the package (which is quite large) remains unused. Is there anyway for me to just distribute the pieces of SciPy I need (keeping all copyright and other such notices in tact) or will my users have to install the entire library? Thank you From jdh2358 at gmail.com Mon Aug 23 12:19:11 2010 From: jdh2358 at gmail.com (John Hunter) Date: Mon, 23 Aug 2010 11:19:11 -0500 Subject: [SciPy-User] Buffet Style Scipy? In-Reply-To: References: Message-ID: On Mon, Aug 23, 2010 at 9:27 AM, David Marsh wrote: > Hello, > > ? I wasn't sure where to start looking for this answer so I figured I'd > send my question to the mailing list. > > ? I'm writing a small program that requires the use of SciPy but is > currently limited to the interpolation library. ?The rest of the > package (which is quite large) remains unused. ?Is there anyway for me > to just distribute the pieces of SciPy I need (keeping all copyright > and other such notices in tact) or will my users have to install the > entire library? scipy is BSD licensed which means you are free to re-distribute it as you like (in part or whole), provided you abide by clauses a-c in the license (retention of copyright, no endorsement w/o permission by contributors). From hodgson at cse.lehigh.edu Tue Aug 24 02:30:40 2010 From: hodgson at cse.lehigh.edu (Bryan Hodgson) Date: Tue, 24 Aug 2010 02:30:40 -0400 Subject: [SciPy-User] Notes on building scipy on Solaris 10 with SunStudio. Message-ID: <20100824063040.GA23129@ganymede.eecs.lehigh.edu> Folks, This is not a problem report, it's a (mostly) success report. I'm a sysadmin, and not a python-ist or an scipy end-user. I found the process of getting a working scipy build on Solaris 10 to be rather too opaque (even with assistance from Google), and decided to post my results in the hope that they might help others. The list seems about as good a spot as any. Building scipy 0.8.0 on Solaris 10 x86 and sparc, update 7 and 5 respectively. Same results on x86 update 8 with current recommended patches. All machines built with SUNWCprog metacluster (e.g. the Solaris developer environment), and Sun Studio 12 update 1 (unpatched original release) at /opt/sunstudio12.1 (the default location for this release). The default system /usr/bin/python (from Sun) is 2.4.4. I'm willing to avoid building BLAS / LAPACK if I can avoid doing that, and the sunmath / sunperf libs bundled with Sun Studio work. Details: The build environment is: site.cfg (for both numpy and scipy) says: [DEFAULT] library_dirs = /opt/sunstudio12.1/lib [blas_opt] libraries = sunperf [lapack_opt] libraries = sunmath The relevant fraction of the shell environment for both is: PATH=/opt/sunstudio12.1/bin:/usr/ccs/bin:/usr/bin:/usr/sfw/bin:/usr/sbin; export PATH ATLAS=None; export ATLAS BLAS=/opt/sunstudio12.1/lib/libsunperf.so; export BLAS LAPACK=/opt/sunstudio12.1/lib/libsunmath.so; export LAPACK pyCC finds the compilers by their inclusion in PATH. SunStudio was installed with '--create-symlinks', creating links in /usr/bin; that may have helped. CC,CXX,CFLAGS and friends are not set. For scipy, LDFLAGS='-lCrun' was added to the environment to get resolve run-time linker errors produced scipy.test() (discussed below). crle is set to installation default, no modifications present. LD_LIBRARY_PATH is unset for the build. numpy 1.4.1 builds with the usual warnings about mis-matched pointer declarations from system headers and code not reached. 'python -c "import numpy;numpy.test()"' fails with a complaint that the default nose is too old. nose 0.11.x does not work, nose 0.10.(3,4) worked, no errors from numpy.test(). (All modules were packaged to permit clean re-installation, and as a final step all was built / installed on a machine that had not yet seen scipy.) Building scipy 0.8.0 out of the box ends with: scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/zneupd.f: zneupd: f90: Internal Error, code=fw-interface-cexp-277, last src=scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/zneupd.f:673 f90: Warning: The option -xcode=pic32 has no effect on x86 scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/zneupd.f: zneupd: f90: Internal Error, code=fw-interface-cexp-277, last src=scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/zneupd.f:673 error: Command "/opt/sunstudio12.1/bin/f90 -ftrap=%none -f77 -xcode=pic32 -Iscipy/sparse/linalg/eigen/arpack/ARPACK/SRC -I/usr/lib/python2.4/site-packages/numpy/core/include -c -c scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/zneupd.f -o build/temp.solaris-2.10-i86pc-2.4/scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/zneupd.o" failed with exit status 1 This is a SunStudio problem fixed by installing patches: sparc: 128231-05 128232-03 141854-01 141860-03 i386: 141852-06 141853-03 141855-01 141858-06 There is a gcc compiler directive (-Wall) in scipy/ndimage/setup.py that must be removed for the compiler to succeed. Without LDFLAGS=lCrun, scipy.test() records 6 errors against scipy/sparse/sparsetools/_csr.so: ImportError: ld.so.1: python2.4: fatal: relocation error: file /usr/lib/python2.4/site-packages/scipy/sparse/sparsetools/_csr.so: symbol __1c2k6Fpv0_v_: referenced symbol not found The missing mangled C++ symbol is in /usr/lib/libCrun.so.1. It's not clear to me why the linker didn't pick it up, but moving on ... There are 3 python assertion failures (not detailed here). Building scipy with LDFLAGS=-lCrun results in an abend of scipy.test() evidently originating in scipy/io/tests/test_netcdf.py (line 104) at test_read_write_sio. After running our scipy code against the resulting scipy, we concluded that this was either an isolated error or a problem on the test suite in our environment. All done. Hope this helps someone ... Bryan From balazovic.peter at gmail.com Tue Aug 24 05:32:59 2010 From: balazovic.peter at gmail.com (pepe_emspin) Date: Tue, 24 Aug 2010 02:32:59 -0700 (PDT) Subject: [SciPy-User] Simulation of dynamic systems and its control with SciPy Message-ID: <15316ce4-e50c-4d26-86f6-3c9c6248f0a4@f42g2000yqn.googlegroups.com> Hello, I am a new to SciPy, I want to create continuous time simulation of dynamical system (xdot=A*x+B*u) than incrementally add controllers in cascade. Is it somehow possible to create simulation in "simulink way" (I don't want GUI - just simple coding) it means step by step add new controllers without a need to rewrite already defined equation, e.g. 1/ build motor system with load 2/ add servo control to motor 3/ add observer systems to eliminate position sensing 3/ add manipulation control (master control of servo) etc... It means gradually build whole system under control and keep it existing - already build equation system? Thanks! From eneide.odissea at gmail.com Tue Aug 24 07:43:40 2010 From: eneide.odissea at gmail.com (eneide.odissea) Date: Tue, 24 Aug 2010 13:43:40 +0200 Subject: [SciPy-User] Simulation of dynamic systems and its control with SciPy In-Reply-To: <15316ce4-e50c-4d26-86f6-3c9c6248f0a4@f42g2000yqn.googlegroups.com> References: <15316ce4-e50c-4d26-86f6-3c9c6248f0a4@f42g2000yqn.googlegroups.com> Message-ID: HI pepe have you look at this package? http://www.siue.edu/~rkrauss/python_intro.html On Tue, Aug 24, 2010 at 11:32 AM, pepe_emspin wrote: > Hello, > > I am a new to SciPy, I want to create continuous time simulation of > dynamical system (xdot=A*x+B*u) than incrementally add controllers in > cascade. > Is it somehow possible to create simulation in "simulink way" (I don't > want GUI - just simple coding) it means step by step add new > controllers without a need to rewrite already defined equation, e.g. > 1/ build motor system with load > 2/ add servo control to motor > 3/ add observer systems to eliminate position sensing > 3/ add manipulation control (master control of servo) etc... > > It means gradually build whole system under control and keep it > existing - already build equation system? > > Thanks! > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From almar.klein at gmail.com Tue Aug 24 08:23:29 2010 From: almar.klein at gmail.com (Almar Klein) Date: Tue, 24 Aug 2010 14:23:29 +0200 Subject: [SciPy-User] ANN: IEP, the interactive Editor for Python Message-ID: I am pleased to announce IEP, the interactive Editor for Python. website: http://code.google.com/p/iep/ downloads: http://code.google.com/p/iep/downloads/list (binaries are available for Windows and Linux) group: http://groups.google.com/group/iep_ I'm interested in what you think of it and whether everything works as it should on all OS's. So if you try it, please drop me a line! Description: IEP is a cross-platform Python IDE focused on interactivity and introspection, which makes it very suitable for scientific computing. Its practical design is aimed at simplicity and efficiency. IEP consists of two main components, the editor and the shell, and uses a set of pluggable tools to help the programmer in various ways (such as source structure and interactive help). Cheers, Almar -------------- next part -------------- An HTML attachment was scrubbed... URL: From jorgesmbox-ml at yahoo.es Tue Aug 24 09:17:39 2010 From: jorgesmbox-ml at yahoo.es (Jorge Scandaliaris) Date: Tue, 24 Aug 2010 13:17:39 +0000 (UTC) Subject: [SciPy-User] Least median of squares for regression in scipy? Message-ID: Hi, I have to perform a linear regression on noisy data. On the last paper I read least median of squares was suggested for dealing with outliers. I have searched the scipy docs but it seems nothing is readily available. Searching the web for "(python OR scipy OR numpy) least median square" doesn't yield meaningfull results. The best I found were fortran and matlab code, which I would need to wrap (I have zero knowledge about fortran or wrapping it into python, except that there's a tool called f2py, but I would have to learn that as well) or rewrite (I used matlab in the past, so this should be feasible). I am asking here in the hope there's something I overlooked before I jump into one of the (probably more time demanding) possibilities I mentioned above. Thanks in advance, Jorge From jrennie at gmail.com Tue Aug 24 09:27:59 2010 From: jrennie at gmail.com (Jason Rennie) Date: Tue, 24 Aug 2010 09:27:59 -0400 Subject: [SciPy-User] Least median of squares for regression in scipy? In-Reply-To: References: Message-ID: "least median of squares" doesn't mean anything to me. But, I know that minimizing sum of absolute differences will provide a good estimate of the median and is a good technique for dealing with outliers: http://en.wikipedia.org/wiki/Least_absolute_deviation http://en.wikipedia.org/wiki/L1_norm Note that you'll need an LP solver. Another option is a hybrid between the squared and absolute value loss functions, such as the one that Peter Huber devised: http://en.wikipedia.org/wiki/Huber_Loss_Function This loss provides the outlier-insensitivity of L1 while being easy to solve using gradient descent & line search. Cheers, Jason On Tue, Aug 24, 2010 at 9:17 AM, Jorge Scandaliaris wrote: > Hi, > I have to perform a linear regression on noisy data. On the last paper I > read > least median of squares was suggested for dealing with outliers. I have > searched > the scipy docs but it seems nothing is readily available. Searching the web > for > "(python OR scipy OR numpy) least median square" doesn't yield meaningfull > results. The best I found were fortran and matlab code, which I would need > to > wrap (I have zero knowledge about fortran or wrapping it into python, > except > that there's a tool called f2py, but I would have to learn that as well) or > rewrite (I used matlab in the past, so this should be feasible). > I am asking here in the hope there's something I overlooked before I jump > into > one of the (probably more time demanding) possibilities I mentioned above. > > Thanks in advance, > > Jorge > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Aug 24 09:29:50 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 24 Aug 2010 09:29:50 -0400 Subject: [SciPy-User] Least median of squares for regression in scipy? In-Reply-To: References: Message-ID: On Tue, Aug 24, 2010 at 9:17 AM, Jorge Scandaliaris wrote: > Hi, > I have to perform a linear regression on noisy data. On the last paper I read > least median of squares was suggested for dealing with outliers. I have searched > the scipy docs but it seems nothing is readily available. Searching the web for > "(python OR scipy OR numpy) least median square" doesn't yield meaningfull > results. The best I found were fortran and matlab code, which I would need to > wrap (I have zero knowledge about fortran or wrapping it into python, except > that there's a tool called f2py, but I would have to learn that as well) or > rewrite (I used matlab in the past, so this should be feasible). > I am asking here in the hope there's something I overlooked before I jump into > one of the (probably more time demanding) possibilities I mentioned above. scikits.statsmodels has robust linear model estimators, rlm Josef > > Thanks in advance, > > Jorge > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From jorgesmbox-ml at yahoo.es Tue Aug 24 10:05:01 2010 From: jorgesmbox-ml at yahoo.es (Jorge Scandaliaris) Date: Tue, 24 Aug 2010 14:05:01 +0000 (UTC) Subject: [SciPy-User] Least median of squares for regression in scipy? References: Message-ID: Jason Rennie gmail.com> writes: > > > "least median of squares" doesn't mean anything to me. ?But, I know that > minimizing sum of absolute differences will provide a good estimate of the > median and is a good technique for dealing with outliers: > > > > http://en.wikipedia.org/wiki/Least_absolute_deviation > http://en.wikipedia.org/wiki/L1_norm > > Note that you'll need an LP solver. ?Another option is a hybrid between the > squared and absolute value loss functions, such as the one that Peter Huber > devised: > > http://en.wikipedia.org/wiki/Huber_Loss_Function > > This loss provides the outlier-insensitivity of L1 while being easy to solve > using gradient descent & line search. > Thanks, I am not very proficient with the subject, I am just trying to use it as a tool in a very particular task. I'll check the links you provide. BTW, this reference is what I was taking about with "least median of squares": http://www.stats.uwo.ca/faculty/aim/2004/04-259/ExtraReading/LeastMedianOfSquaresRegression.pdf Regards, Jorge From jorgesmbox-ml at yahoo.es Tue Aug 24 10:06:08 2010 From: jorgesmbox-ml at yahoo.es (Jorge Scandaliaris) Date: Tue, 24 Aug 2010 14:06:08 +0000 (UTC) Subject: [SciPy-User] Least median of squares for regression in scipy? References: Message-ID: gmail.com> writes: > > scikits.statsmodels has robust linear model estimators, rlm > > Josef Thanks, I'll check it out Jorge From josef.pktd at gmail.com Tue Aug 24 10:16:22 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 24 Aug 2010 10:16:22 -0400 Subject: [SciPy-User] Least median of squares for regression in scipy? In-Reply-To: References: Message-ID: On Tue, Aug 24, 2010 at 10:06 AM, Jorge Scandaliaris wrote: > ? gmail.com> writes: > >> >> scikits.statsmodels has robust linear model estimators, ?rlm >> >> Josef > > Thanks, I'll check it out The wikipedia link for Huber loss function, and http://statsmodels.sourceforge.net/rlm.html might get you started. It should be possible to try out different options to see which version can handle your outliers. (I'm a bit busy and don't know rlm so well, but if you have any questions, I could look at them tonight, or maybe Skipper is available. Skipper worked on RLM.) Josef > > > Jorge > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From ralf.gommers at googlemail.com Tue Aug 24 10:29:26 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 24 Aug 2010 22:29:26 +0800 Subject: [SciPy-User] Notes on building scipy on Solaris 10 with SunStudio. In-Reply-To: <20100824063040.GA23129@ganymede.eecs.lehigh.edu> References: <20100824063040.GA23129@ganymede.eecs.lehigh.edu> Message-ID: Hi Bryan, On Tue, Aug 24, 2010 at 2:30 PM, Bryan Hodgson wrote: > > Folks, > > This is not a problem report, it's a (mostly) success report. I'm > a sysadmin, and not a python-ist or an scipy end-user. > > I found the process of getting a working scipy build on Solaris 10 > to be rather too opaque (even with assistance from Google), and > decided to post my results in the hope that they might help others. > The list seems about as good a spot as any. > Thanks for the write-up, that's definitely useful. If you don't mind, I'll add this to the wiki as Solaris-specific details under http://www.scipy.org/Installing_SciPy. > > Building scipy 0.8.0 on Solaris 10 x86 and sparc, update 7 and 5 > respectively. Same results on x86 update 8 with current > recommended patches. All machines built with SUNWCprog metacluster > (e.g. the Solaris developer environment), and Sun Studio 12 update > 1 (unpatched original release) at /opt/sunstudio12.1 (the default > location for this release). The default system /usr/bin/python > (from Sun) is 2.4.4. I'm willing to avoid building BLAS / LAPACK > if I can avoid doing that, and the sunmath / sunperf libs bundled > with Sun Studio work. Details: > > The build environment is: > > site.cfg (for both numpy and scipy) says: > > [DEFAULT] > library_dirs = /opt/sunstudio12.1/lib > [blas_opt] > libraries = sunperf > [lapack_opt] > libraries = sunmath > > The relevant fraction of the shell environment for both is: > > PATH=/opt/sunstudio12.1/bin:/usr/ccs/bin:/usr/bin:/usr/sfw/bin:/usr/sbin; > export PATH > ATLAS=None; export ATLAS > BLAS=/opt/sunstudio12.1/lib/libsunperf.so; export BLAS > LAPACK=/opt/sunstudio12.1/lib/libsunmath.so; export LAPACK > > pyCC finds the compilers by their inclusion in PATH. SunStudio was > installed with '--create-symlinks', creating links in /usr/bin; > that may have helped. > > CC,CXX,CFLAGS and friends are not set. > > For scipy, LDFLAGS='-lCrun' was added to the environment to get > resolve run-time linker errors produced scipy.test() (discussed > below). > > crle is set to installation default, no modifications present. > LD_LIBRARY_PATH is unset for the build. > > numpy 1.4.1 builds with the usual warnings about mis-matched > pointer declarations from system headers and code not reached. > > 'python -c "import numpy;numpy.test()"' fails with a complaint that > the default nose is too old. nose 0.11.x does not work, nose > 0.10.(3,4) worked, no errors from numpy.test(). (All modules were > packaged to permit clean re-installation, and as a final step all > was built / installed on a machine that had not yet seen scipy.) > > Building scipy 0.8.0 out of the box ends with: > > scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/zneupd.f: > zneupd: > f90: Internal Error, code=fw-interface-cexp-277, last > src=scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/zneupd.f:673 > f90: Warning: The option -xcode=pic32 has no effect on x86 > scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/zneupd.f: > zneupd: > f90: Internal Error, code=fw-interface-cexp-277, last > src=scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/zneupd.f:673 > error: Command "/opt/sunstudio12.1/bin/f90 -ftrap=%none -f77 -xcode=pic32 > -Iscipy/sparse/linalg/eigen/arpack/ARPACK/SRC > -I/usr/lib/python2.4/site-packages/numpy/core/include -c -c > scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/zneupd.f -o > build/temp.solaris-2.10-i86pc-2.4/scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/zneupd.o" > failed with exit status 1 > > This is a SunStudio problem fixed by installing patches: > > sparc: 128231-05 128232-03 141854-01 141860-03 > > i386: 141852-06 141853-03 141855-01 141858-06 > > There is a gcc compiler directive (-Wall) in scipy/ndimage/setup.py > that must be removed for the compiler to succeed. > That looks like debug code that got merged recently by accident, r6339, and should be removed. The same goes for "import time" in that commit. Stefan, is that correct? Cheers, Ralf > Without LDFLAGS=lCrun, scipy.test() records 6 errors against > scipy/sparse/sparsetools/_csr.so: > > ImportError: ld.so.1: python2.4: fatal: relocation error: file > /usr/lib/python2.4/site-packages/scipy/sparse/sparsetools/_csr.so: symbol > __1c2k6Fpv0_v_: referenced symbol not found > > The missing mangled C++ symbol is in /usr/lib/libCrun.so.1. It's > not clear to me why the linker didn't pick it up, but moving on > ... > > There are 3 python assertion failures (not detailed here). > > Building scipy with LDFLAGS=-lCrun results in an abend of > scipy.test() evidently originating in scipy/io/tests/test_netcdf.py > (line 104) at test_read_write_sio. After running our scipy code > against the resulting scipy, we concluded that this was either an > isolated error or a problem on the test suite in our environment. > > All done. > > Hope this helps someone ... > > Bryan > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jrennie at gmail.com Tue Aug 24 10:38:17 2010 From: jrennie at gmail.com (Jason Rennie) Date: Tue, 24 Aug 2010 10:38:17 -0400 Subject: [SciPy-User] Least median of squares for regression in scipy? In-Reply-To: References: Message-ID: On Tue, Aug 24, 2010 at 10:05 AM, Jorge Scandaliaris wrote: > Thanks, I am not very proficient with the subject, I am just trying to use > it as > a tool in a very particular task. I'll check the links you provide. BTW, > this > reference is what I was taking about with "least median of squares": > > http://www.stats.uwo.ca/faculty/aim/2004/04-259/ExtraReading/LeastMedianOfSquaresRegression.pdf Ah, I see. Least median of squares is different, using the median (squared/absolute) deviation as the objective to minimize. Huber's estimators seem to be more popular FWIW... Jason -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsseabold at gmail.com Tue Aug 24 10:51:13 2010 From: jsseabold at gmail.com (Skipper Seabold) Date: Tue, 24 Aug 2010 10:51:13 -0400 Subject: [SciPy-User] Least median of squares for regression in scipy? In-Reply-To: References: Message-ID: On Tue, Aug 24, 2010 at 10:16 AM, wrote: > On Tue, Aug 24, 2010 at 10:06 AM, Jorge Scandaliaris > wrote: >> ? gmail.com> writes: >> >>> >>> scikits.statsmodels has robust linear model estimators, ?rlm >>> >>> Josef >> >> Thanks, I'll check it out > > > The wikipedia link for Huber loss function, and > http://statsmodels.sourceforge.net/rlm.html might get you started. > This, with your other link should get you moving. We have as of now only M-estimators with several weighting norms (http://statsmodels.sourceforge.net/rlm_techn1.html) and scale estimators that are proposed in your reference. The framework is there for extension using iteratively reweighted least squares, if you need to go down this route. Let me know if any of the documentation or code is not clear. > It should be possible to try out different options to see which > version can handle your outliers. > > > (I'm a bit busy and don't know rlm so well, but if you have any > questions, I could look at them tonight, or maybe Skipper is > available. Skipper worked on RLM.) > > Josef > >> >> >> Jorge >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From vincentdegroof at gmail.com Tue Aug 24 11:03:09 2010 From: vincentdegroof at gmail.com (Vincent De Groof) Date: Tue, 24 Aug 2010 17:03:09 +0200 Subject: [SciPy-User] Solving an eigenvalue problem Message-ID: Hi, I have to solve an eigenvalue problem for which I currently use the scipy.linalg.eig function. My matrix A is squared and symmetric. So far the problem size was quite limited (10000*10000) so it could find all the eigenvalues and eigenvectors within a reasonable time frame. However, I'm now scaling up the problem drastically and my fear is that it will just take too much time. I'm running a test right now (62000*62000). However I don't need all the eigenvalues and eigenvectors. Is there an existing scipy function where I can define the number of eigenvectors I want? So the output would become e.g. the 250 largest eigenvalues and its corresponding eigenvectors? thanks, Vincent -------------- next part -------------- An HTML attachment was scrubbed... URL: From balazovic.peter at gmail.com Tue Aug 24 11:08:16 2010 From: balazovic.peter at gmail.com (pepe_emspin) Date: Tue, 24 Aug 2010 08:08:16 -0700 (PDT) Subject: [SciPy-User] Simulation of dynamic systems and its control with SciPy In-Reply-To: References: <15316ce4-e50c-4d26-86f6-3c9c6248f0a4@f42g2000yqn.googlegroups.com> Message-ID: <0ec55c5a-0986-4aae-9c9e-df301b2b930b@v8g2000yqe.googlegroups.com> it seems to be interesting for certain purposes but I am looking for an approach where I have a system (MIMO) under control then several controllers (continues time domain). There are "ode", "lsim" and so on and it does not help me. Therefore a question is how can SciPy can help me to perform whole simulation of dynamic system (controllers, observers, MIMO system)? best regards, Peter On 24 srp, 13:43, "eneide.odissea" wrote: > HI pepe > have you look at this package?http://www.siue.edu/~rkrauss/python_intro.html > > > On Tue, Aug 24, 2010 at 11:32 AM, pepe_emspin wrote: > > > > > Hello, > > > I am a new to SciPy, I want to create continuous time simulation of > > dynamical system (xdot=A*x+B*u) than incrementally add controllers in > > cascade. > > Is it somehow possible to create simulation in "simulink way" (I don't > > want GUI - just simple coding) it means step by step add new > > controllers without a need to rewrite already defined equation, e.g. > > 1/ build motor system with load > > 2/ add servo control to motor > > 3/ add observer systems to eliminate position sensing > > 3/ add manipulation control (master control of servo) etc... > > > It means gradually build whole system under control and keep it > > existing - already build equation system? > > > Thanks! > > _______________________________________________ > > SciPy-User mailing list > > SciPy-U... at scipy.org > >http://mail.scipy.org/mailman/listinfo/scipy-user > > > > _______________________________________________ > SciPy-User mailing list > SciPy-U... at scipy.orghttp://mail.scipy.org/mailman/listinfo/scipy-user From stefan at sun.ac.za Tue Aug 24 11:09:57 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 24 Aug 2010 17:09:57 +0200 Subject: [SciPy-User] Solving an eigenvalue problem In-Reply-To: References: Message-ID: Hi Vincent On 24 August 2010 17:03, Vincent De Groof wrote: > However I don't need all the eigenvalues and eigenvectors. Is there an > existing scipy function where I can define the number of eigenvectors I > want? So the output would become e.g. the 250 largest eigenvalues and its > corresponding eigenvectors? Also have a look at the ARPACK wrappers in scipy.sparse.linalg. Regards St?fan From lutz.maibaum at gmail.com Tue Aug 24 11:12:25 2010 From: lutz.maibaum at gmail.com (Lutz Maibaum) Date: Tue, 24 Aug 2010 08:12:25 -0700 Subject: [SciPy-User] Solving an eigenvalue problem In-Reply-To: References: Message-ID: On Tue, Aug 24, 2010 at 8:03 AM, Vincent De Groof wrote: > However I don't need all the eigenvalues and eigenvectors. Is there an > existing scipy function where I can define the number of eigenvectors I > want? So the output would become e.g. the 250 largest eigenvalues and its > corresponding eigenvectors? You could try scipy.sparse.linalg.eigen_symmetric (or the non-symmetric counterpart). Hope this helps, Lutz From stefan at sun.ac.za Tue Aug 24 11:27:13 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 24 Aug 2010 17:27:13 +0200 Subject: [SciPy-User] ANN: IEP, the interactive Editor for Python In-Reply-To: References: Message-ID: Hi Almar On 24 August 2010 14:23, Almar Klein wrote: > I am pleased to announce IEP, the interactive Editor for Python. > > website: http://code.google.com/p/iep/ > downloads: http://code.google.com/p/iep/downloads/list (binaries are > available for Windows and Linux) > group: http://groups.google.com/group/iep_ Nifty! An interesting Windows-like distribution model (shipping all the .so's) :-) The only thing that bugs me is that the numpy docstrings are mangled by default. Maybe "no newlines" should be ticked to start with? (Or, parsed using the provided code from the docs project) Also, a variable inspector would come in handy. Regards St?fan From almar.klein at gmail.com Tue Aug 24 11:42:14 2010 From: almar.klein at gmail.com (Almar Klein) Date: Tue, 24 Aug 2010 17:42:14 +0200 Subject: [SciPy-User] ANN: IEP, the interactive Editor for Python In-Reply-To: References: Message-ID: 2010/8/24 St?fan van der Walt > Hi Almar > > On 24 August 2010 14:23, Almar Klein wrote: > > I am pleased to announce IEP, the interactive Editor for Python. > > > > website: http://code.google.com/p/iep/ > > downloads: http://code.google.com/p/iep/downloads/list (binaries are > > available for Windows and Linux) > > group: http://groups.google.com/group/iep_ > > Nifty! An interesting Windows-like distribution model (shipping all > the .so's) :-) > The main reason for doing this is that it can be quite a bit of work to install Python3 + PyQT4 + Qscintilla on Linux. Prebuilding saves a lot of people that work :) I hope to provide something similar for Mac users in the near future (but I need some help with that since I do not own a Mac). > The only thing that bugs me is that the numpy docstrings are mangled > by default. Maybe "no newlines" should be ticked to start with? (Or, > parsed using the provided code from the docs project) Also, a > variable inspector would come in handy. > I will take that under consideration. A workspace like tool is on my wishlist (see issues :) ) . Cheers, Almar -------------- next part -------------- An HTML attachment was scrubbed... URL: From almar.klein at gmail.com Tue Aug 24 11:52:02 2010 From: almar.klein at gmail.com (Almar Klein) Date: Tue, 24 Aug 2010 17:52:02 +0200 Subject: [SciPy-User] ANN: IEP, the interactive Editor for Python In-Reply-To: References: Message-ID: > ... (Or, > parsed using the provided code from the docs project) > Euhm, to what "docs project" are you referring? All I found was a small algorithm in PEP257 to trim indentation in docstrings. Almar -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Aug 24 11:57:01 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 24 Aug 2010 11:57:01 -0400 Subject: [SciPy-User] Solving an eigenvalue problem In-Reply-To: References: Message-ID: On Tue, Aug 24, 2010 at 11:12 AM, Lutz Maibaum wrote: > On Tue, Aug 24, 2010 at 8:03 AM, Vincent De Groof > wrote: >> However I don't need all the eigenvalues and eigenvectors. Is there an >> existing scipy function where I can define the number of eigenvectors I >> want? So the output would become e.g. the 250 largest eigenvalues and its >> corresponding eigenvectors? scipy.linalg.eigh has the option to only return some eigenvectors eigvals : tuple (lo, hi) Indexes of the smallest and largest (in ascending order) eigenvalues and corresponding eigenvectors to be returned: 0 <= lo < hi <= M-1. If omitted, all eigenvalues and eigenvectors are returned. David C was discussing about rewriting and enhancing this feature, but I don't know what the status is. Josef > > You could try scipy.sparse.linalg.eigen_symmetric (or the > non-symmetric counterpart). > > Hope this helps, > > ?Lutz > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From ijstokes at hkl.hms.harvard.edu Tue Aug 24 11:57:32 2010 From: ijstokes at hkl.hms.harvard.edu (Ian Stokes-Rees) Date: Tue, 24 Aug 2010 11:57:32 -0400 Subject: [SciPy-User] ANN: IEP, the interactive Editor for Python In-Reply-To: References: Message-ID: <4C73EBEC.40703@hkl.hms.harvard.edu> > The main reason for doing this is that it can be quite a bit of work > to install Python3 + PyQT4 + Qscintilla on Linux. Prebuilding saves a > lot of people that work :) > > I hope to provide something similar for Mac users in the near future > (but I need some help with that since I do not own a Mac). We run a developer network with a number of different platforms available (including OS X 10.5 and 10.6). I could probably get you an account there. Desktop/graphical access is via VNC, AFAIK. Ian -------------- next part -------------- A non-text attachment was scrubbed... Name: ijstokes.vcf Type: text/x-vcard Size: 394 bytes Desc: not available URL: From josef.pktd at gmail.com Tue Aug 24 12:01:00 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 24 Aug 2010 12:01:00 -0400 Subject: [SciPy-User] Simulation of dynamic systems and its control with SciPy In-Reply-To: <0ec55c5a-0986-4aae-9c9e-df301b2b930b@v8g2000yqe.googlegroups.com> References: <15316ce4-e50c-4d26-86f6-3c9c6248f0a4@f42g2000yqn.googlegroups.com> <0ec55c5a-0986-4aae-9c9e-df301b2b930b@v8g2000yqe.googlegroups.com> Message-ID: On Tue, Aug 24, 2010 at 11:08 AM, pepe_emspin wrote: > it seems to be interesting for certain purposes but I am looking for > an approach where I have a system (MIMO) under control then several > controllers (continues time domain). There are "ode", "lsim" and so on > and it does not help me. > Therefore a question is how can SciPy can help me to perform whole > simulation of dynamic system (controllers, observers, MIMO system)? > > best regards, > Peter > > On 24 srp, 13:43, "eneide.odissea" wrote: >> HI pepe >> have you look at this package?http://www.siue.edu/~rkrauss/python_intro.html >> >> >> On Tue, Aug 24, 2010 at 11:32 AM, pepe_emspin wrote: >> >> >> >> > Hello, >> >> > I am a new to SciPy, I want to create continuous time simulation of >> > dynamical system (xdot=A*x+B*u) than incrementally add controllers in >> > cascade. >> > Is it somehow possible to create simulation in "simulink way" (I don't >> > want GUI - just simple coding) it means step by step add new >> > controllers without a need to rewrite already defined equation, e.g. >> > 1/ build motor system with load >> > 2/ add servo control to motor >> > 3/ add observer systems to eliminate position sensing >> > 3/ add manipulation control (master control of servo) etc... >> >> > It means gradually build whole system under control and keep it >> > existing - already build equation system? >> >> > Thanks! Maybe the enhancements and referenced code by Roberto Bucher can help you. http://mail.scipy.org/pipermail/scipy-dev/2010-August/015465.html Josef From christian at prinoth.name Tue Aug 24 12:03:05 2010 From: christian at prinoth.name (Christian Prinoth) Date: Tue, 24 Aug 2010 18:03:05 +0200 Subject: [SciPy-User] Least median of squares for regression in scipy? In-Reply-To: References: Message-ID: if I am not wrong you are looking for quantile regression, which has least absolute deviation as a special case equivalent to median regression. There is some matlab code on Roger Koenker's site to do this, and it should be very easy to translate to python. On Tue, Aug 24, 2010 at 16:51, Skipper Seabold wrote: > On Tue, Aug 24, 2010 at 10:16 AM, wrote: > > On Tue, Aug 24, 2010 at 10:06 AM, Jorge Scandaliaris > > wrote: > >> gmail.com> writes: > >> > >>> > >>> scikits.statsmodels has robust linear model estimators, rlm > >>> > >>> Josef > >> > >> Thanks, I'll check it out > > > > > > The wikipedia link for Huber loss function, and > > http://statsmodels.sourceforge.net/rlm.html might get you started. > > > > This, with your other link should get you moving. We have as of now > only M-estimators with several weighting norms > (http://statsmodels.sourceforge.net/rlm_techn1.html) and scale > estimators that are proposed in your reference. The framework is > there for extension using iteratively reweighted least squares, if you > need to go down this route. Let me know if any of the documentation > or code is not clear. > > > It should be possible to try out different options to see which > > version can handle your outliers. > > > > > > (I'm a bit busy and don't know rlm so well, but if you have any > > questions, I could look at them tonight, or maybe Skipper is > > available. Skipper worked on RLM.) > > > > Josef > > > >> > >> > >> Jorge > >> > >> _______________________________________________ > >> SciPy-User mailing list > >> SciPy-User at scipy.org > >> http://mail.scipy.org/mailman/listinfo/scipy-user > >> > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From almar.klein at gmail.com Tue Aug 24 12:09:06 2010 From: almar.klein at gmail.com (Almar Klein) Date: Tue, 24 Aug 2010 18:09:06 +0200 Subject: [SciPy-User] ANN: IEP, the interactive Editor for Python In-Reply-To: <4C73EBEC.40703@hkl.hms.harvard.edu> References: <4C73EBEC.40703@hkl.hms.harvard.edu> Message-ID: On 24 August 2010 17:57, Ian Stokes-Rees wrote: > > > The main reason for doing this is that it can be quite a bit of work > > to install Python3 + PyQT4 + Qscintilla on Linux. Prebuilding saves a > > lot of people that work :) > > > > I hope to provide something similar for Mac users in the near future > > (but I need some help with that since I do not own a Mac). > > We run a developer network with a number of different platforms > available (including OS X 10.5 and 10.6). I could probably get you an > account there. Desktop/graphical access is via VNC, AFAIK. > Wow, thanks for the offer. I'll first ask a friend who owns a Mac if he can help me out though. I get back to you if I still need your suggestion :) Thanks, Almar -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Tue Aug 24 12:10:39 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 24 Aug 2010 18:10:39 +0200 Subject: [SciPy-User] ANN: IEP, the interactive Editor for Python In-Reply-To: References: Message-ID: On 24 August 2010 17:52, Almar Klein wrote: > >> ... (Or, >> parsed using the provided code from the docs project) > > Euhm, to what "docs project" are you referring? All I found was a small > algorithm in PEP257 to trim indentation in docstrings. http://projects.scipy.org/numpy/browser/trunk/doc/sphinxext/docscrape.py St?fan From ijstokes at hkl.hms.harvard.edu Tue Aug 24 12:13:19 2010 From: ijstokes at hkl.hms.harvard.edu (Ian Stokes-Rees) Date: Tue, 24 Aug 2010 12:13:19 -0400 Subject: [SciPy-User] ANN: IEP, the interactive Editor for Python In-Reply-To: References: <4C73EBEC.40703@hkl.hms.harvard.edu> Message-ID: <4C73EF9F.5020700@hkl.hms.harvard.edu> > > We run a developer network with a number of different platforms > available (including OS X 10.5 and 10.6). I could probably get you an > account there. Desktop/graphical access is via VNC, AFAIK. > > > Wow, thanks for the offer. I'll first ask a friend who owns a Mac if > he can help me out though. I get back to you if I still need your > suggestion :) Sure thing. It is free for approved users, and it is for scientific software development, so I suspect you'd be approved no questions asked. Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ijstokes.vcf Type: text/x-vcard Size: 394 bytes Desc: not available URL: From philmorefield at yahoo.com Tue Aug 24 16:37:49 2010 From: philmorefield at yahoo.com (Phil Morefield) Date: Tue, 24 Aug 2010 13:37:49 -0700 (PDT) Subject: [SciPy-User] scipy.io.netcdf question Message-ID: <189258.29565.qm@web53406.mail.re2.yahoo.com> A question for scipy experts: I'm trying to read in a very large three dimensional NetCDF file and failing.?The code is simple: import numpy from scipy.io import netcdf ncFile = netcdf.netcdf_file(my_input_file,"r") Traceback (most recent call last): ? File "", line 1, in ? File "C:\Python25\lib\site-packages\scipy\io\netcdf.py", line 43, in __init__ ??? self._parse() ? File "C:\Python25\lib\site-packages\scipy\io\netcdf.py", line 79, in _parse ??? self._var_array() ? File "C:\Python25\lib\site-packages\scipy\io\netcdf.py", line 135, in _var_array ??? self.variables[name] = self._read_var() ? File "C:\Python25\lib\site-packages\scipy\io\netcdf.py", line 180, in _read_var ??? attributes = self._att_array() ? File "C:\Python25\lib\site-packages\scipy\io\netcdf.py", line 119, in _att_array ??? attributes[name] = self._read_values(n, nc_type) ? File "C:\Python25\lib\site-packages\scipy\io\netcdf.py", line 193, in _read_values ??? count = n * bytes[nc_type-1] IndexError: list index out of range ? I went into the scipy code and determined that the variable 'nc_type'?is being passed to _read_values with a?value of 1333097057, while the list object 'bytes' only holds six items. That explains the "list index out of range". My suspicion is that some identifying information?is wrong with my NetCDF file, but I can't figure out where the 'nc_type' variable comes from exactly, so I'm just guessing. I'd really appreciate some help with this. Any ideas?? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Tue Aug 24 16:49:35 2010 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 24 Aug 2010 13:49:35 -0700 Subject: [SciPy-User] scipy.io.netcdf question In-Reply-To: <189258.29565.qm@web53406.mail.re2.yahoo.com> References: <189258.29565.qm@web53406.mail.re2.yahoo.com> Message-ID: <4C74305F.2040105@noaa.gov> Phil Morefield wrote: > I'd really appreciate some help with this. Any ideas? No ideas, but I've had good luck with the NetCDF4 package: http://code.google.com/p/netcdf4-python/ It may be worth a try (and it does fully support netcdf3 files) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From ben.root at ou.edu Tue Aug 24 17:36:34 2010 From: ben.root at ou.edu (Benjamin Root) Date: Tue, 24 Aug 2010 16:36:34 -0500 Subject: [SciPy-User] scipy.io.netcdf question In-Reply-To: <189258.29565.qm@web53406.mail.re2.yahoo.com> References: <189258.29565.qm@web53406.mail.re2.yahoo.com> Message-ID: On Tue, Aug 24, 2010 at 3:37 PM, Phil Morefield wrote: > A question for scipy experts: > > I'm trying to read in a very large three dimensional NetCDF file and > failing. The code is simple: > > import numpy > from scipy.io import netcdf > > ncFile = netcdf.netcdf_file(my_input_file,"r") > > Traceback (most recent call last): > File "", line 1, in > File "C:\Python25\lib\site-packages\scipy\io\netcdf.py", line 43, in > __init__ > self._parse() > File "C:\Python25\lib\site-packages\scipy\io\netcdf.py", line 79, in > _parse > self._var_array() > File "C:\Python25\lib\site-packages\scipy\io\netcdf.py", line 135, in > _var_array > self.variables[name] = self._read_var() > File "C:\Python25\lib\site-packages\scipy\io\netcdf.py", line 180, in > _read_var > attributes = self._att_array() > File "C:\Python25\lib\site-packages\scipy\io\netcdf.py", line 119, in > _att_array > attributes[name] = self._read_values(n, nc_type) > File "C:\Python25\lib\site-packages\scipy\io\netcdf.py", line 193, in > _read_values > count = n * bytes[nc_type-1] > IndexError: list index out of range > > I went into the scipy code and determined that the variable 'nc_type' is > being passed to _read_values with a value of 1333097057, while the list > object 'bytes' only holds six items. That explains the "list index out of > range". My suspicion is that some identifying information is wrong with my > NetCDF file, but I can't figure out where the 'nc_type' variable comes from > exactly, so I'm just guessing. > > I'd really appreciate some help with this. Any ideas? > > Thanks! > > Phil, Can you please do a ncdump -h on the file and post the output here. That might help explain things. Also, how many megabytes is this file. It might be possible that the code is not properly detecting the newer mode (which contains extra data at the beginning of the header) and therefore is reading bad values in the wrong spots. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From dagss at student.matnat.uio.no Wed Aug 25 04:04:50 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 25 Aug 2010 10:04:50 +0200 Subject: [SciPy-User] ANN: Cython 0.13 released! Message-ID: <4C74CEA2.1070703@student.matnat.uio.no> Forwarding; Crag Citro wrote: It is with *great* pleasure that I email to announce the release of Cython version 0.13! This release sets another milestone on the path towards Python compatibility and brings major new features and improvements for the usability of the Cython language. Download it here: http://cython.org/release/Cython-0.13.tar.gz == New Features == * Closures are fully supported for Python functions. Cython supports inner functions and lambda expressions. Generators and generator expressions are __not__ supported in this release. * Proper C++ support. Cython knows about C++ classes, templates and overloaded function signatures, so that Cython code can interact with them in a straight forward way. * Type inference is enabled by default for safe C types (e.g. double, bint, C++ classes) and known extension types. This reduces the need for explicit type declarations and can improve the performance of untyped code in some cases. There is also a verbose compile mode for testing the impact on user code. * Cython's for-in-loop can iterate over C arrays and sliced pointers. The type of the loop variable will be inferred automatically in this case. * The Py_UNICODE integer type for Unicode code points is fully supported, including for-loops and 'in' tests on unicode strings. It coerces from and to single character unicode strings. Note that untyped for-loop variables will automatically be inferred as Py_UNICODE when iterating over a unicode string. In most cases, this will be much more efficient than yielding sliced string objects, but can also have a negative performance impact when the variable is used in a Python context multiple times, so that it needs to coerce to a unicode string object more than once. If this happens, typing the loop variable as unicode or object will help. * The built-in functions any(), all(), sum(), list(), set() and dict() are inlined as plain `for` loops when called on generator expressions. Note that generator expressions are not generally supported apart from this feature. Also, tuple(genexpr) is not currently supported - use tuple([listcomp]) instead. * More shipped standard library declarations. The python_* and stdlib/stdio .pxd files have been deprecated in favor of clib.* and cpython[.*] and may get removed in a future release. == Python compatibility == * Pure Python mode no longer disallows non-Python keywords like 'cdef', 'include' or 'cimport'. It also no longer recognises syntax extensions like the for-from loop. * Parsing has improved for Python 3 syntax in Python code, although not all features are correctly supported. The missing Python 3 features are being worked on for the next release. * from __future__ import print_function is supported in Python 2.6 and later. Note that there is currently no emulation for earlier Python versions, so code that uses print() with this future import will require at least Python 2.6. * New compiler directive language_level (valid values: 2 or 3) with corresponding command line options -2 and -3 requests source code compatibility with Python 2.x or Python 3.x respectively. Language level 3 currently enforces unicode literals for unprefixed string literals, enables the print function (requires Python 2.6 or later) and keeps loop variables in list comprehensions from leaking. * Loop variables in set/dict comprehensions no longer leak into the surrounding scope (following Python 2.7). List comprehensions are unchanged in language level 2. == Incompatible changes == * The availability of type inference by default means that Cython will also infer the type of pointers on assignments. Previously, code like this cdef char* s = ... untyped_variable = s would convert the char* to a Python bytes string and assign that. This is no longer the case and no coercion will happen in the example above. The correct way of doing this is through an explicit cast or by typing the target variable, i.e. cdef char* s = ... untyped_variable1 = s untyped_variable2 = s cdef object py_object = s cdef bytes bytes_string = s * bool is no longer a valid type name by default. The problem is that it's not clear whether bool should refer to the Python type or the C++ type, and expecting one and finding the other has already led to several hard-to-find bugs. Both types are available for importing: you can use from cpython cimport bool for the Python bool type, and from libcpp cimport bool for the C++ type. == Contributors == Many people contributed to this release, including: * David Barnett * Stefan Behnel * Chuck Blake * Robert Bradshaw * Craig Citro * Bryan Cole * Lisandro Dalcin * Eric Firing * Danilo Freitas * Christoph Gohlke * Dag Sverre Seljebotn * Kurt Smith * Erik Tollerud * Carl Witty -cc _______________________________________________ Cython-dev mailing list Cython-dev at codespeak.net http://codespeak.net/mailman/listinfo/cython-dev From balazovic.peter at gmail.com Wed Aug 25 05:28:13 2010 From: balazovic.peter at gmail.com (pepe_emspin) Date: Wed, 25 Aug 2010 02:28:13 -0700 (PDT) Subject: [SciPy-User] Simulation of dynamic systems and its control with SciPy In-Reply-To: References: <15316ce4-e50c-4d26-86f6-3c9c6248f0a4@f42g2000yqn.googlegroups.com> <0ec55c5a-0986-4aae-9c9e-df301b2b930b@v8g2000yqe.googlegroups.com> Message-ID: <7d9ac9df-7f8e-40a3-a160-6731a71dfb55@z10g2000yqb.googlegroups.com> Thanks for posting! I find some of the functions very useful. My question is: Does anybody considered to extend SciPy or create a kind of "control toolbox" as an extension of scipy/numpy/python? Peter On 24 srp, 18:01, josef.p... at gmail.com wrote: > On Tue, Aug 24, 2010 at 11:08 AM, pepe_emspin wrote: > > it seems to be interesting for certain purposes but I am looking for > > an approach where I have a system (MIMO) under control then several > > controllers (continues time domain). There are "ode", "lsim" and so on > > and it does not help me. > > Therefore a question is how can SciPy can help me to perform whole > > simulation of dynamic system (controllers, observers, MIMO system)? > > > best regards, > > Peter > > > On 24 srp, 13:43, "eneide.odissea" wrote: > >> HI pepe > >> have you look at this package?http://www.siue.edu/~rkrauss/python_intro.html > >> > > >> On Tue, Aug 24, 2010 at 11:32 AM, pepe_emspin wrote: > > >> > Hello, > > >> > I am a new to SciPy, I want to create continuous time simulation of > >> > dynamical system (xdot=A*x+B*u) than incrementally add controllers in > >> > cascade. > >> > Is it somehow possible to create simulation in "simulink way" (I don't > >> > want GUI - just simple coding) it means step by step add new > >> > controllers without a need to rewrite already defined equation, e.g. > >> > 1/ build motor system with load > >> > 2/ add servo control to motor > >> > 3/ add observer systems to eliminate position sensing > >> > 3/ add manipulation control (master control of servo) etc... > > >> > It means gradually build whole system under control and keep it > >> > existing - already build equation system? > > >> > Thanks! > > Maybe the enhancements and referenced code by Roberto Bucher can help you. > > http://mail.scipy.org/pipermail/scipy-dev/2010-August/015465.html > > Josef > _______________________________________________ > SciPy-User mailing list > SciPy-U... at scipy.orghttp://mail.scipy.org/mailman/listinfo/scipy-user From christian at prinoth.name Wed Aug 25 08:52:16 2010 From: christian at prinoth.name (Christian Prinoth) Date: Wed, 25 Aug 2010 14:52:16 +0200 Subject: [SciPy-User] ANN: IEP, the interactive Editor for Python In-Reply-To: References: <4C73EBEC.40703@hkl.hms.harvard.edu> Message-ID: FYI, in my experience QScintilla has problems on OSX, graphical glitches and various artifacts. I am looking forward to your experience. Christian On Tue, Aug 24, 2010 at 18:09, Almar Klein wrote: > > On 24 August 2010 17:57, Ian Stokes-Rees wrote: > >> >> > The main reason for doing this is that it can be quite a bit of work >> > to install Python3 + PyQT4 + Qscintilla on Linux. Prebuilding saves a >> > lot of people that work :) >> > >> > I hope to provide something similar for Mac users in the near future >> > (but I need some help with that since I do not own a Mac). >> >> We run a developer network with a number of different platforms >> available (including OS X 10.5 and 10.6). I could probably get you an >> account there. Desktop/graphical access is via VNC, AFAIK. >> > > Wow, thanks for the offer. I'll first ask a friend who owns a Mac if he can > help me out though. I get back to you if I still need your suggestion :) > > Thanks, > Almar > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Aug 25 09:46:49 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 25 Aug 2010 09:46:49 -0400 Subject: [SciPy-User] ANN: IEP, the interactive Editor for Python In-Reply-To: References: <4C73EBEC.40703@hkl.hms.harvard.edu> Message-ID: On Wed, Aug 25, 2010 at 8:52 AM, Christian Prinoth wrote: > FYI, in my experience QScintilla has problems on OSX, graphical glitches and > various artifacts. I am looking forward to your experience. > Christian > > On Tue, Aug 24, 2010 at 18:09, Almar Klein wrote: >> >> On 24 August 2010 17:57, Ian Stokes-Rees >> wrote: >>> >>> > The main reason for doing this is that it can be quite a bit of work >>> > to install Python3 + PyQT4 + Qscintilla on Linux. Prebuilding saves a >>> > lot of people that work :) >>> > >>> > I hope to provide something similar for Mac users in the near future >>> > (but I need some help with that since I do not own a Mac). >>> >>> We run a developer network with a number of different platforms >>> available (including OS X 10.5 and 10.6). ?I could probably get you an >>> account there. ?Desktop/graphical access is via VNC, AFAIK. >> >> Wow, thanks for the offer. I'll first ask a friend who owns a Mac if he >> can help me out though. I get back to you if I still need your suggestion :) >> >> Thanks, >> ? Almar >> >> unfortunately it doesn't seem to like my matplotlib setup with Tkinter backend on WindowsXP. The file runs without problems in IDLE. Are there special options in the matplotlib rc file required? Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) on win32. This is the IEP interpreterType "help" for help, type "?" for a list of *magic* commands. >>> [executing "firms.py"] c:\programs\python25\lib\site-packages\matplotlib-0.99.1-py2.5-win32.egg\matplot lib\rcsetup.py:117: UserWarning: rcParams key "numerix" is obsolete and has no e ffect; please delete it from your matplotlibrc file warnings.warn('rcParams key "numerix" is obsolete and has no effect;\n' Traceback (most recent call last): File "C:/Josef/eclipsegworkspace/openecon/pyecon/pyecon/micro/firms.py", line 551, in f1.plot() File "C:/Josef/eclipsegworkspace/openecon/pyecon/pyecon/micro/firms.py", line 300, in plot pl.plot() File "C:/Josef/eclipsegworkspace/openecon/pyecon/pyecon/micro/firms.py", line 437, in plot plt.figure() File "C:\Programs\Python25\lib\site-packages\matplotlib-0.99.1-py2.5-win32.egg \matplotlib\pyplot.py", line 254, in figure # do not solve using profit_function as difficulties with numerical File "C:\Programs\Python25\lib\site-packages\matplotlib-0.99.1-py2.5-win32.egg \matplotlib\backends\backend_tkagg.py", line 90, in new_figure_manager __call__ returns function values, i.e. total cost File "C:\Programs\Python25\lib\lib-tk\Tkinter.py", line 1631, in __init__ baseName = os.path.basename(sys.argv[0]) IndexError: list index out of range Josef From josef.pktd at gmail.com Wed Aug 25 10:11:45 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 25 Aug 2010 10:11:45 -0400 Subject: [SciPy-User] ANN: IEP, the interactive Editor for Python In-Reply-To: References: <4C73EBEC.40703@hkl.hms.harvard.edu> Message-ID: On Wed, Aug 25, 2010 at 9:46 AM, wrote: > On Wed, Aug 25, 2010 at 8:52 AM, Christian Prinoth > wrote: >> FYI, in my experience QScintilla has problems on OSX, graphical glitches and >> various artifacts. I am looking forward to your experience. >> Christian >> >> On Tue, Aug 24, 2010 at 18:09, Almar Klein wrote: >>> >>> On 24 August 2010 17:57, Ian Stokes-Rees >>> wrote: >>>> >>>> > The main reason for doing this is that it can be quite a bit of work >>>> > to install Python3 + PyQT4 + Qscintilla on Linux. Prebuilding saves a >>>> > lot of people that work :) >>>> > >>>> > I hope to provide something similar for Mac users in the near future >>>> > (but I need some help with that since I do not own a Mac). >>>> >>>> We run a developer network with a number of different platforms >>>> available (including OS X 10.5 and 10.6). ?I could probably get you an >>>> account there. ?Desktop/graphical access is via VNC, AFAIK. >>> >>> Wow, thanks for the offer. I'll first ask a friend who owns a Mac if he >>> can help me out though. I get back to you if I still need your suggestion :) >>> >>> Thanks, >>> ? Almar >>> >>> > > unfortunately it doesn't seem to like my matplotlib setup with Tkinter > backend on WindowsXP. The file runs without problems in IDLE. Are > there special options in the matplotlib rc file required? > > > Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) on win32. > This is the IEP interpreterType "help" for help, type "?" for a list of *magic* > commands. >>>> [executing "firms.py"] > c:\programs\python25\lib\site-packages\matplotlib-0.99.1-py2.5-win32.egg\matplot > lib\rcsetup.py:117: UserWarning: rcParams key "numerix" is obsolete and has no e > ffect; > ?please delete it from your matplotlibrc file > ?warnings.warn('rcParams key "numerix" is obsolete and has no effect;\n' > Traceback (most recent call last): > ?File "C:/Josef/eclipsegworkspace/openecon/pyecon/pyecon/micro/firms.py", line > 551, in > ? ?f1.plot() > ?File "C:/Josef/eclipsegworkspace/openecon/pyecon/pyecon/micro/firms.py", line > 300, in plot > ? ?pl.plot() > ?File "C:/Josef/eclipsegworkspace/openecon/pyecon/pyecon/micro/firms.py", line > 437, in plot > ? ?plt.figure() > ?File "C:\Programs\Python25\lib\site-packages\matplotlib-0.99.1-py2.5-win32.egg > \matplotlib\pyplot.py", line 254, in figure > ? ?# do not solve using profit_function as difficulties with numerical > ?File "C:\Programs\Python25\lib\site-packages\matplotlib-0.99.1-py2.5-win32.egg > \matplotlib\backends\backend_tkagg.py", line 90, in new_figure_manager > ? ?__call__ returns function values, i.e. total cost > ?File "C:\Programs\Python25\lib\lib-tk\Tkinter.py", line 1631, in __init__ > ? ?baseName = os.path.basename(sys.argv[0]) > IndexError: list index out of range Changing shell setting to using TK and using "run file as script" instead of "run file" is working so far. Josef > > Josef > From almar.klein at gmail.com Wed Aug 25 10:17:41 2010 From: almar.klein at gmail.com (Almar Klein) Date: Wed, 25 Aug 2010 16:17:41 +0200 Subject: [SciPy-User] ANN: IEP, the interactive Editor for Python In-Reply-To: References: <4C73EBEC.40703@hkl.hms.harvard.edu> Message-ID: > unfortunately it doesn't seem to like my matplotlib setup with Tkinter > backend on WindowsXP. The file runs without problems in IDLE. Are > there special options in the matplotlib rc file required? > > > Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) on win32. > This is the IEP interpreterType "help" for help, type "?" for a list of > *magic* > commands. > >>> [executing "firms.py"] > > c:\programs\python25\lib\site-packages\matplotlib-0.99.1-py2.5-win32.egg\matplot > lib\rcsetup.py:117: UserWarning: rcParams key "numerix" is obsolete and has > no e > ffect; > please delete it from your matplotlibrc file > warnings.warn('rcParams key "numerix" is obsolete and has no effect;\n' > Traceback (most recent call last): > File "C:/Josef/eclipsegworkspace/openecon/pyecon/pyecon/micro/firms.py", > line > 551, in > f1.plot() > File "C:/Josef/eclipsegworkspace/openecon/pyecon/pyecon/micro/firms.py", > line > 300, in plot > pl.plot() > File "C:/Josef/eclipsegworkspace/openecon/pyecon/pyecon/micro/firms.py", > line > 437, in plot > plt.figure() > File > "C:\Programs\Python25\lib\site-packages\matplotlib-0.99.1-py2.5-win32.egg > \matplotlib\pyplot.py", line 254, in figure > # do not solve using profit_function as difficulties with numerical > File > "C:\Programs\Python25\lib\site-packages\matplotlib-0.99.1-py2.5-win32.egg > \matplotlib\backends\backend_tkagg.py", line 90, in new_figure_manager > __call__ returns function values, i.e. total cost > File "C:\Programs\Python25\lib\lib-tk\Tkinter.py", line 1631, in __init__ > baseName = os.path.basename(sys.argv[0]) > IndexError: list index out of range > Right, the IEP interpreter sets sys.argv to [], while it should have set it to ['']. (I used the standard Python shell as an example and must have misread it). I will fix this bug right now. In the mean time, you can put "sys.argv = ['']" somewhere before importing matplotlib, or better yet, put that code in your PYTHONSTARTUP file. Another option is to run your code as a script, which sets the first element in sys.argv to the scripts path. Thanks for reporting this! Almar -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Aug 25 10:34:56 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 25 Aug 2010 10:34:56 -0400 Subject: [SciPy-User] ANN: IEP, the interactive Editor for Python In-Reply-To: References: <4C73EBEC.40703@hkl.hms.harvard.edu> Message-ID: On Wed, Aug 25, 2010 at 10:17 AM, Almar Klein wrote: > >> unfortunately it doesn't seem to like my matplotlib setup with Tkinter >> backend on WindowsXP. The file runs without problems in IDLE. Are >> there special options in the matplotlib rc file required? >> >> >> Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) on win32. >> This is the IEP interpreterType "help" for help, type "?" for a list of >> *magic* >> commands. >> >>> [executing "firms.py"] >> >> c:\programs\python25\lib\site-packages\matplotlib-0.99.1-py2.5-win32.egg\matplot >> lib\rcsetup.py:117: UserWarning: rcParams key "numerix" is obsolete and >> has no e >> ffect; >> ?please delete it from your matplotlibrc file >> ?warnings.warn('rcParams key "numerix" is obsolete and has no effect;\n' >> Traceback (most recent call last): >> ?File "C:/Josef/eclipsegworkspace/openecon/pyecon/pyecon/micro/firms.py", >> line >> 551, in >> ? ?f1.plot() >> ?File "C:/Josef/eclipsegworkspace/openecon/pyecon/pyecon/micro/firms.py", >> line >> 300, in plot >> ? ?pl.plot() >> ?File "C:/Josef/eclipsegworkspace/openecon/pyecon/pyecon/micro/firms.py", >> line >> 437, in plot >> ? ?plt.figure() >> ?File >> "C:\Programs\Python25\lib\site-packages\matplotlib-0.99.1-py2.5-win32.egg >> \matplotlib\pyplot.py", line 254, in figure >> ? ?# do not solve using profit_function as difficulties with numerical >> ?File >> "C:\Programs\Python25\lib\site-packages\matplotlib-0.99.1-py2.5-win32.egg >> \matplotlib\backends\backend_tkagg.py", line 90, in new_figure_manager >> ? ?__call__ returns function values, i.e. total cost >> ?File "C:\Programs\Python25\lib\lib-tk\Tkinter.py", line 1631, in __init__ >> ? ?baseName = os.path.basename(sys.argv[0]) >> IndexError: list index out of range > > > Right, the IEP interpreter sets sys.argv to [], while it should have set it > to ['']. (I used the standard Python shell as an example and must have > misread it). I will fix this bug right now. > > In the mean time, you can put "sys.argv = ['']" somewhere before importing > matplotlib, Done this and it works, I can successfully "run file" several times in the same shell without restarting. or better yet, put that code in your PYTHONSTARTUP file. Another > option is to run your code as a script, which sets the first element in > sys.argv to the scripts path. this restarts the interpreter which I don't always want. A cosmetic problem is that I have many cells because IDLE uses ## for commenting out a block. I haven't found the Command history yet. Thanks, Josef > > Thanks for reporting this! > ? Almar > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From almar.klein at gmail.com Wed Aug 25 10:57:21 2010 From: almar.klein at gmail.com (Almar Klein) Date: Wed, 25 Aug 2010 16:57:21 +0200 Subject: [SciPy-User] ANN: IEP, the interactive Editor for Python In-Reply-To: References: <4C73EBEC.40703@hkl.hms.harvard.edu> Message-ID: > > In the mean time, you can put "sys.argv = ['']" somewhere before > importing > > matplotlib, > > Done this and it works, I can successfully "run file" several times in > the same shell without restarting. > Great! > A cosmetic problem is that I have many cells because IDLE uses ## for > commenting out a block. > Yes, I know some editors do that. If you're not interested in this feature, you can kind of "deactivate" it: in the source structure tool you can simply deselect showing the cells. If you also want to change the appearance of the cells, you can edit its style (Settings > Edit syntax styles). > I haven't found the Command history yet. > Using the up/down keys in the shell should do the trick. Cheers, Almar -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Aug 25 11:02:22 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 25 Aug 2010 11:02:22 -0400 Subject: [SciPy-User] ANN: IEP, the interactive Editor for Python In-Reply-To: References: <4C73EBEC.40703@hkl.hms.harvard.edu> Message-ID: On Wed, Aug 25, 2010 at 10:57 AM, Almar Klein wrote: > >> > In the mean time, you can put "sys.argv = ['']" somewhere before >> > importing >> > matplotlib, >> >> Done this and it works, I can successfully "run file" several times in >> the same shell without restarting. > > Great! > > >> >> A cosmetic problem is that I have many cells because IDLE uses ## for >> commenting out a block. > > Yes, I know some editors do that. If you're not interested in this feature, > you can kind of "deactivate" it: in the source structure tool you can simply > deselect showing the cells. If you also want to change the appearance of the > cells, you can edit its style (Settings > Edit syntax styles). > > >> >> I haven't found the Command history yet. > > Using the up/down keys in the shell should do the trick. Ok, I found that, but I misinterpreted this, I thought command history is the log file of all commands in a session, similar to matlab or spyder. I usually like to save my "experiments" for possible future reference. Josef > > Cheers, > ? Almar > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From almar.klein at gmail.com Wed Aug 25 11:10:02 2010 From: almar.klein at gmail.com (Almar Klein) Date: Wed, 25 Aug 2010 17:10:02 +0200 Subject: [SciPy-User] ANN: IEP, the interactive Editor for Python In-Reply-To: References: <4C73EBEC.40703@hkl.hms.harvard.edu> Message-ID: > >> I haven't found the Command history yet. > > > > Using the up/down keys in the shell should do the trick. > > Ok, I found that, but I misinterpreted this, I thought command history > is the log file of all commands in a session, similar to matlab or > spyder. I usually like to save my "experiments" for possible future > reference. > Ah ok. IEP does not yet have such a feature. But you could post a feature request for it in the issues list at the website. A tool that does this should not be hard to make. Almar -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Aug 25 11:16:28 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 25 Aug 2010 11:16:28 -0400 Subject: [SciPy-User] ANN: IEP, the interactive Editor for Python In-Reply-To: References: <4C73EBEC.40703@hkl.hms.harvard.edu> Message-ID: On Wed, Aug 25, 2010 at 11:10 AM, Almar Klein wrote: > >> >> I haven't found the Command history yet. >> > >> > Using the up/down keys in the shell should do the trick. >> >> Ok, I found that, but I misinterpreted this, I thought command history >> is the log file of all commands in a session, similar to matlab or >> spyder. I usually like to save my "experiments" for possible future >> reference. > > Ah ok. IEP does not yet have such a feature. But you could post a feature > request for it in the issues list at the website. A tool that does this > should not be hard to make. will do. One more question. Is there a way to close shells? I have too many of them now (after playing for a while), and don't find a way to close some of them. Thanks, Josef > > ? Almar > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From almar.klein at gmail.com Wed Aug 25 11:48:50 2010 From: almar.klein at gmail.com (Almar Klein) Date: Wed, 25 Aug 2010 17:48:50 +0200 Subject: [SciPy-User] ANN: IEP, the interactive Editor for Python In-Reply-To: References: <4C73EBEC.40703@hkl.hms.harvard.edu> Message-ID: On 25 August 2010 17:16, wrote: > On Wed, Aug 25, 2010 at 11:10 AM, Almar Klein > wrote: > > > >> >> I haven't found the Command history yet. > >> > > >> > Using the up/down keys in the shell should do the trick. > >> > >> Ok, I found that, but I misinterpreted this, I thought command history > >> is the log file of all commands in a session, similar to matlab or > >> spyder. I usually like to save my "experiments" for possible future > >> reference. > > > > Ah ok. IEP does not yet have such a feature. But you could post a feature > > request for it in the issues list at the website. A tool that does this > > should not be hard to make. > > will do. > > One more question. Is there a way to close shells? I have too many of > them now (after playing for a while), and don't find a way to close > some of them. > Shell > Terminate current shell (or ctrl+K) -------------- next part -------------- An HTML attachment was scrubbed... URL: From philmorefield at yahoo.com Wed Aug 25 14:58:06 2010 From: philmorefield at yahoo.com (Phil Morefield) Date: Wed, 25 Aug 2010 11:58:06 -0700 (PDT) Subject: [SciPy-User] scipy.io.netcdf question In-Reply-To: References: <189258.29565.qm@web53406.mail.re2.yahoo.com> Message-ID: <581101.494.qm@web53404.mail.re2.yahoo.com> ? Ben- The .nc files I'm using range from 550 MB to 915 MB. Large, I know. Here is the output from ncdump: ?dimensions: xc = 140 ; yc = 115 ; time = UNLIMITED ; // (14600 currently) variables: double level ; level:units = "m" ; level:long_name = "height" ; level:axis = "Z" ; level:positive = "up" ; level:coordinate_defines = "point" ; level:actual_range = 2.f, 2.f ; double yc(yc) ; yc:units = "m" ; yc:long_name = "y-coordinate of polar_stereographic projection" ; yc:standard_name = "projection_y_coordinate" ; yc:axis = "Y" ; yc:coordinate_defines = "point" ; double xc(xc) ; xc:units = "m" ; xc:long_name = "x-coordinate of polar_stereographic projection" ; xc:standard_name = "projection_x_coordinate" ; xc:axis = "X" ; xc:coordinate_defines = "point" ; double time(time) ; time:long_name = "time" ; time:calendar = "365_day" ; time:standard_name = "time" ; time:axis = "T" ; time:units = "days since 1968-01-01 00:00:0.0" ; time:delta_t = "0000-00-00 03:00:00" ; time:coordinate_defines = "point" ; time:actual_range = 1095.125f, 2920.f ; double lon(yc, xc) ; lon:units = "degrees_east" ; lon:long_name = "longitude" ; lon:standard_name = "longitude" ; lon:axis = "X" ; double lat(yc, xc) ; lat:units = "degrees_north" ; lat:long_name = "latitude" ; lat:standard_name = "latitude" ; lat:axis = "Y" ; char polar_stereographic ; polar_stereographic:straight_vertical_longitude_from_pole = 263. ; polar_stereographic:standard_parallel = 60. ; polar_stereographic:false_easting = 3475000. ; polar_stereographic:false_northing = 7475000. ; polar_stereographic:latitude_of_projection_origin = 90. ; polar_stereographic:grid_mapping_name = "polar_stereographic" ; polar_stereographic:resolution_at_standard_parallel = 50000. ; polar_stereographic:earth_radius = 6371000. ; float tas(time, yc, xc) ; tas:units = "K" ; tas:type = "instantaneaous" ; tas:long_name = "Surface Air Temperature" ; tas:standard_name = "air_temperature" ; tas:missing_value = 1.e+020f ; tas:_FillValue = 1.e+020f ; tas:actual_range = 204.64909f, 327.28021f ; tas:add_offset = 0.f ; tas:scale_factor = 1.f ; tas:coordinates = "lon lat level" ; tas:grid_mapping = "polar_stereographic" ; tas:level_desc = "2 m" ; tas:grid_desc = "polar_stereographic" ; // global attributes: :Conventions = "CF-1.0" ; :institution = "Ouranos (Ouranos, Montreal, PQ, Canada)" ; :source = "CRCM(2006):atmosphere:CRCMv4.2.0(50km,29levels):sea ice:AMIPII:land:CLASSv2.7" ; :project_id = "NARCCAP" ; :realization = "1" ; :experiment_id = "1968-2000 climate simulation using CGCM3 member #4 (mc_abv experiment)" ; :experiment_id2 = "Ouranos operational simulation afv" ;:history = "Tue Feb 3 14:36:39 2009: ncrename -a conventions,Conventions tas_CRCM_1971010103.nc\n", "created 24/04/2004 by Ton Nom (netCDF3.6.1 udunits1.12.4)" ; :table_id = "Table 2" ; } ? And lastly, I was incorrectly about 'nc_type' last time. The value being passed to?_read_values is only 10. Not sure where I got that big number.?Konrad Hinsen's ScientificPython module works fine for what I'm doing, but I'd really like to find a way to make scipy do the work. ? Thanks in advance, ? Phil ________________________________ From: Benjamin Root To: SciPy Users List Sent: Tue, August 24, 2010 5:36:34 PM Subject: Re: [SciPy-User] scipy.io.netcdf question On Tue, Aug 24, 2010 at 3:37 PM, Phil Morefield wrote: A question for scipy experts: > >I'm trying to read in a very large three dimensional NetCDF file and >failing.?The code is simple: > >import numpy >from scipy.io import netcdf > >ncFile = netcdf.netcdf_file(my_input_file,"r") > >Traceback (most recent call last): >? File "", line 1, in >? File "C:\Python25\lib\site-packages\scipy\io\netcdf.py", line 43, in __init__ >??? self._parse() >? File "C:\Python25\lib\site-packages\scipy\io\netcdf.py", line 79, in _parse >??? self._var_array() >? File "C:\Python25\lib\site-packages\scipy\io\netcdf.py", line 135, in >_var_array >??? self.variables[name] = self._read_var() >? File "C:\Python25\lib\site-packages\scipy\io\netcdf.py", line 180, in >_read_var >??? attributes = self._att_array() >? File "C:\Python25\lib\site-packages\scipy\io\netcdf.py", line 119, in >_att_array >??? attributes[name] = self._read_values(n, nc_type) >? File "C:\Python25\lib\site-packages\scipy\io\netcdf.py", line 193, in >_read_values >??? count = n * bytes[nc_type-1] >IndexError: list index out of range >? >I went into the scipy code and determined that the variable 'nc_type'?is being >passed to _read_values with a?value of 1333097057, while the list object 'bytes' >only holds six items. That explains the "list index out of range". My suspicion >is that some identifying information?is wrong with my NetCDF file, but I can't >figure out where the 'nc_type' variable comes from exactly, so I'm just >guessing. > >I'd really appreciate some help with this. Any ideas?? > >Thanks! > Phil, Can you please do a ncdump -h on the file and post the output here.? That might help explain things.? Also, how many megabytes is this file.? It might be possible that the code is not properly detecting the newer mode (which contains extra data at the beginning of the header) and therefore is reading bad values in the wrong spots. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From Michael.Graham at cdicorp.com Wed Aug 25 15:51:50 2010 From: Michael.Graham at cdicorp.com (Graham, Michael) Date: Wed, 25 Aug 2010 15:51:50 -0400 Subject: [SciPy-User] [JOB] Developers/engineers wanted for mechanics simulation software Message-ID: <61A807FD6A92CC45A6064B98D79D343B25F7994CD8@CDIMSXMBCL01.CDI.CDICorp.net> CDI Marine Band Lavis Division is seeking developers with skills in engineering and software development in Baltimore, MD (relocation available) for full-time work on exciting, research-oriented marine engineering projects. CDI is a large, stable company of which Band Lavis Division is a small, profitable division with a laid back atmosphere and small-company flexibility. Candidates should be quick learners, enthusiastic workers, and have strong technical expertise. The following skills, knowledge, and passions are sought: - Python, C, C++, and Fortran knowledge and experience: Our current work is mostly Python but there will certainly be C development for performance and library bindings. - Parallel programming experience and knowledge. Knowedge of MPI. - Network programming experience would be extremely valuable, especially experience with Twisted and AMQP. Experience designing and implementing network protocols. - Knowledge of solid mechanics, structural mechanics, and fluid mechanics. - Knowledge of numerical methods in general and of FEA or CFD in particular is highly desirable. - Skills with the following scientific and engineering software packages could be useful: Abaqus, OpenFOAM, Ansys, CFX, Gridgen, LAPACK, Metis, InterComm and--of course--numpy and scipy. - An interest in performing some engineering analysis in addition to primary software development responsibilities. - Knowledge of parsing. We deal with a lot of input files, most of which are not our design; skills working with XML and experience parsing a variety of formats would be an asset. - Linux cluster design and administration. - Experience with unit testing, smoke testing, coverage testing, and performance testing. - Test and build automation experience. - Database design and development skills. - Desire to contribute to open source software. - GUI development in Python. PyGTK. - 3D graphics and CAD visualization. If this describes you, please reply with information about your background and skills. This is a great opportunity to develop high-quality, cross-platform engineering software in Python and contribute to the open source community. This position requires US citizenship and the ability to obtain a security clearance. Please confirm that you are a US citizen in your reply. Thank you, Mike Graham Software Engineer CDI Marine Company, Band Lavis Division ______________________________________________________________________ CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain information which is confidential to, and/or privileged in favor of, CDI Corporation or its affiliated companies (CDI) or CDI's customers. Any review, use, reproduction, disclosure or distribution by the recipient is prohibited without prior written approval from an authorized CDI representative. This notice must appear in any such authorized reproduction, disclosure or distribution. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and any attachments. Thank you. From ralf.gommers at googlemail.com Thu Aug 26 07:30:16 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 26 Aug 2010 19:30:16 +0800 Subject: [SciPy-User] ANN: NumPy 1.5.0 release candidate 1 Message-ID: I am pleased to announce the availability of the first release candidate of NumPy 1.5.0. This will be the first NumPy release to include support for Python 3, as well as for Python 2.7. Please try this RC and report any problems on the NumPy mailing list. Especially with Python 3 testing will be very useful. On Linux and OS X building from source should be straightforward, for Windows a binary installer is provided. Binaries, sources and release notes can be found at https://sourceforge.net/projects/numpy/files/ Enjoy, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From wkerzendorf at googlemail.com Fri Aug 27 05:10:20 2010 From: wkerzendorf at googlemail.com (Wolfgang Kerzendorf) Date: Fri, 27 Aug 2010 11:10:20 +0200 Subject: [SciPy-User] ndimage find_objects and label Message-ID: <4C7780FC.8020507@gmail.com> Dear all, ndimage has find_object and label functions to find and cluster objects above a certain threshold. I wanted to know if that happens in one pass through the data? There is an algorithm called the lutz algorithm (lutz 1979: http://comjnl.oxfordjournals.org/cgi/reprint/23/3/262) which only needs one pass through the data. I If this is already implemented in scipy I wouldn't bother my summer student to implement it, if it isn't I would implement it and also make it available. Cheers Wolfgang From stefan at sun.ac.za Fri Aug 27 10:43:07 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 27 Aug 2010 16:43:07 +0200 Subject: [SciPy-User] ndimage find_objects and label In-Reply-To: <4C7780FC.8020507@gmail.com> References: <4C7780FC.8020507@gmail.com> Message-ID: Dear Wolfgang On 27 August 2010 11:10, Wolfgang Kerzendorf wrote: > ndimage has find_object and label functions to find and cluster objects > above a certain threshold. > I wanted to know if that happens in one pass through the data? There is > an algorithm called the lutz algorithm (lutz 1979: > http://comjnl.oxfordjournals.org/cgi/reprint/23/3/262) which only needs > one pass through the data. I Have a look in the file scipy/ndimage/src/ni_measure.c -- search for "FindObject". The source should reveal which algorithm was chosen (unfortunately, the documentation doesn't tell). Regards St?fan From denis-bz-gg at t-online.de Fri Aug 27 12:03:01 2010 From: denis-bz-gg at t-online.de (denis) Date: Fri, 27 Aug 2010 09:03:01 -0700 (PDT) Subject: [SciPy-User] Problems with 2D interpolation of data on polar grid In-Reply-To: References: Message-ID: <3e143837-6a30-455c-8079-e65e6907994d@q22g2000yqm.googlegroups.com> Kyle, folks, for nonuniform 2d interpolation I'd recommend matplotlib.mlab.griddata, which does a Delaunay triangulation then Natural-neighbor; see http://scipy.org/Cookbook/Matplotlib/Gridding_irregularly_spaced_data and http://en.wikipedia.org/wiki/Natural_neighbor . To interpolate to arbitrary points with Delaunay, use the little class Triinterpolate in the scipy-user thread "Interpolation in 3D with interp2d " Aug 17. griddata returns a masked array, masked outside the convex hull of the input x y -- if that's a problem, you can extrapolate to the corners. I've found fitpack BivariateSpline to be *very* sensitive to s= (which is a sum of squares -- print sqrt( fit.get_residual() / N ), also print once fit.get_knots() ). What you really want to specify for BivariateSpline is the number of knots -- but you can't, afaik. cheers -- denis On Aug 20, 5:52?pm, Kyle Parfrey wrote: > Hi all, > > I've been having some trouble doing 2D interpolation with both > interp2d and bisplrep/bisplev. My data is on a spherical polar (r, > theta) grid, and I'm trying to interpolate functions similar to the > vector components of a dipole field. The output looks very wrong, and > I keep getting warnings like (from interp2d): From opossumnano at gmail.com Sat Aug 28 10:58:03 2010 From: opossumnano at gmail.com (Tiziano Zito) Date: Sat, 28 Aug 2010 16:58:03 +0200 (CEST) Subject: [SciPy-User] =?utf-8?q?=5BANN=5D_Reminder=3A_Autumn_School_=22Adv?= =?utf-8?q?anced_Scientific_Programming_in_Python=22_in_Trento=2C_Italy?= Message-ID: <20100828145803.7A992249721@mail.bccn-berlin> Reminder: Application deadline is August 31st, 2010! =================================================== Advanced Scientific Programming in Python ========================================= an Autumn School by the G-Node, the Center for Mind/Brain Sciences and the Fondazione Bruno Kessler Scientists spend more and more time writing, maintaining, and debugging software. While techniques for doing this efficiently have evolved, only few scientists actually use them. As a result, instead of doing their research, they spend far too much time writing deficient code and reinventing the wheel. In this course we will present a selection of advanced programming techniques with theoretical lectures and practical exercises tailored to the needs of a programming scientist. New skills will be tested in a real programming project: we will team up to develop an entertaining scientific computer game. We'll use the Python programming language for the entire course. Python works as a simple programming language for beginners, but more importantly, it also works great in scientific simulations and data analysis. Clean language design and easy extensibility are driving Python to become a standard tool for scientific computing. Some of the most useful open source libraries for scientific computing and visualization will be presented. This school is targeted at Post-docs and PhD students from all areas of science. Competence in Python or in another language such as Java, C/C++, MATLAB, or Mathematica is absolutely required. A basic knowledge of the Python language is assumed. Participants without prior experience with Python should work through the proposed introductory materials. Date and Location ================= October 4th?8th, 2010. Trento, Italy. Preliminary Program =================== Day 0 (Mon Oct 4) ? Software Carpentry & Advanced Python ? Documenting code and using version control ? Object-oriented programming, design patterns, and agile programming ? Exception handling, lambdas, decorators, context managers, metaclasses Day 1 (Tue Oct 5) ? Software Carpentry ? Test-driven development, unit testing & Quality Assurance ? Debugging, profiling and benchmarking techniques ? Data serialization: from pickle to databases Day 2 (Wed Oct 6) ? Scientific Tools for Python ? Advanced NumPy ? The Quest for Speed (intro): Interfacing to C ? Programming project Day 3 (Thu Oct 7) ? The Quest for Speed ? Writing parallel applications in Python ? When parallelization does not help: the starving CPUs problem ? Programming project Day 4 (Fri Oct 8) ? Practical Software Development ? Efficient programming in teams ? Programming project ? The Pac-Man Tournament Every evening we will have the tutors' consultation hour: Tutors will answer your questions and give suggestions for your own projects Applications ============ You can apply on-line at http://www.g-node.org/python-autumnschool Applications must be submitted before August 31st, 2010. Notifications of acceptance will be sent by September 4th, 2010. No fee is charged but participants should take care of travel, living, and accommodation expenses. Candidates will be selected on the basis of their profile. Places are limited: acceptance rate in past editions was around 30%. Prerequisites ============= You are supposed to know the basics of Python to participate in the lectures! Look on the website for a list of introductory material. Faculty ======= ? Francesc Alted, author of PyTables, Castell? de la Plana, Spain ? Pietro Berkes, Volen Center for Complex Systems, Brandeis University, USA ? Valentin Haenel, Berlin Institute of Technology and Bernstein Center for Computational Neuroscience Berlin, Germany ? Zbigniew J?drzejewski-Szmek, Faculty of Physics, University of Warsaw, Poland ? Eilif Muller, The Blue Brain Project, Ecole Polytechnique F?d?rale de Lausanne, Switzerland ? Emanuele Olivetti, NeuroInformatics Laboratory, Fondazione Bruno Kessler and University of Trento, Italy ? Rike-Benjamin Schuppner, Bernstein Center for Computational Neuroscience Berlin, Germany ? Bartosz Tele?czuk, Institute for Theoretical Biology, Humboldt-Universit?t zu Berlin, Germany ? Bastian Venthur, Berlin Institute of Technology and Bernstein Focus: Neurotechnology, Germany ? St?fan van der Walt, Applied Mathematics, University of Stellenbosch, South Africa ? Tiziano Zito, Berlin Institute of Technology and Bernstein Center for Computational Neuroscience Berlin, Germany Organized by Paolo Avesani for the Center for Mind/Brain Sciences and the Fondazione Bruno Kessler, and by Zbigniew J?drzejewski-Szmek and Tiziano Zito for the German Neuroinformatics Node of the INCF. Website: http://www.g-node.org/python-autumnschool Contact: python-info at g-node.org From josef.pktd at gmail.com Sat Aug 28 23:23:08 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 28 Aug 2010 23:23:08 -0400 Subject: [SciPy-User] OT: matplotlib.testing Message-ID: How do I get access to the testing facilities of matplotlib? (I don't seem to be able to sign up to matlab-user yet, so I'm trying here.) I finally managed my first (kind of baby) GUI with matplotlib, the diagrams/plots look good but I don't have any tests to keep it this way. for the graph testing (GUI testing is indefinitely postponed), I would like matplotlib.testing.compare as in http://matplotlib.svn.sourceforge.net/viewvc/matplotlib/trunk/matplotlib/test/mplTest/MplTestCase.py?revision=7649&view=markup&pathrev=7652 How do I get access to it, and are there any docs for doing testing of matplotlib plots ? my outcome so far: >>> mpl.__version__ '0.99.1' >>> import matplotlib.testing Traceback (most recent call last): File "", line 1, in ImportError: No module named testing >>> import matplotlib.testing.compare as compare Traceback (most recent call last): File "", line 1, in ImportError: No module named testing.compare Thanks, and sorry for being off-topic. Josef (TDD by staring at plots and graphs and diagrams; T&E instead of RTFM, where F=Full) From alexlz at lmn.pub.ro Sun Aug 29 01:42:16 2010 From: alexlz at lmn.pub.ro (Ioan-Alexandru Lazar) Date: Sun, 29 Aug 2010 08:42:16 +0300 Subject: [SciPy-User] Correctly compiling SciPy with UMFPACK support? In-Reply-To: References: Message-ID: <3d98c06a76ca4d84bb31995cb32b40a8.squirrel@wm.lmn.pub.ro> Hello everyone, A while ago I posted a problem I had -- namely both the scipy and the scikits UMFPACK interface failing to work when I was trying to compile them by hand with UMFPACK also being compiled by hand. A couple of friendly folks over here gave me a hand but even with their advice, I wasn't able to do much. I postponed it for a while (it wasn't really urgent -- now it is, but it's also solved), so I though I'd post here just in case someone bumps against the same issue. There are two problems, one that is mentioned in most instructions and one that seems not to be, namely: 1. When building, SciPy (and the Scikits umfpack kit) both assume that UFconfig.h is also in the include directory -- so umfpack and amd's won't suffice. 2. The umfpack interface module needs the shared umfpack and amd libraries. However, SuiteSparse does *not* compile them as such; as many C programs actually expect to end up being statically linked with SuiteSparse's library, it's not such a good idea to tweak UFconfig.mk so as to build them all as shared libraries. It's best to manually compile AMD and UMFPACK as shared libraries (with gcc -shared -Wl and remembering to link the math library and whatever blas you're using with umfpack) and place them in the libraries folder defined in site.cfg, and add that path to LD_LIBRARY_PATH. I hope this helps. I hope I haven't made an ass of myself by not reading the docs correctly -- in any case, I haven't seen it mentioned at http://http://www.scipy.org/Installing_SciPy/Linux --- Slightly less interesting section following --- By the way -- I think I could have figured this out slightly faster; when I looked at it again, I only needed a morning of quick hacking. The culprit was actually here: try: # Silence import error. import _umfpack as _um except: _um = None ( http://svn.scipy.org/svn/scikits/trunk/umfpack/scikits/umfpack/umfpack.py , but this is how it currently looks in SciPy as well) Unfortunately, the error message that appears when _um = None is simply that SciPy was compiled with umfpack support in scipy, or that the symbol UMFPACK_CONTROL is undefined. However, this can be misleading -- in my particular case, the import error initially pointed to a symbol in cholmod (!) not being defined, and after sume further digging, to umfpack.so simply not being found. The deadline for the paper I'm working on is in about three weeks and I'm kind of entagled right now (you know how academics is...) but as soon as I'm done with it, I'll try to provide a fix for this (a more particular error message) and update the docs if anyone else thinks this is useful. Since direct umfpack support is going to be deprecated in SciPy, I assume I should do this in the scikits version, right? Thank you all for your help and your time. Alexandru Lazar Numerical Modeling Laboratory, Politehnica University of Bucharest From strawman at astraw.com Mon Aug 30 03:52:23 2010 From: strawman at astraw.com (Andrew Straw) Date: Mon, 30 Aug 2010 00:52:23 -0700 Subject: [SciPy-User] OT: matplotlib.testing In-Reply-To: References: Message-ID: <4C7B6337.3090302@astraw.com> On 8/28/10 8:23 PM, josef.pktd at gmail.com wrote: > How do I get access to the testing facilities of matplotlib? > > How do I get access to it, and are there any docs for doing testing of > matplotlib plots ? > There's not much in the way of documentation for the testing infrastructure. You can see example tests in the lib/matplotlib/tests/*.py files in the MPL source. The MPL testing facilities only made their debut in 1.0, if I recall correctly. From mischa at astro.uni-bonn.de Mon Aug 30 07:10:48 2010 From: mischa at astro.uni-bonn.de (Mischa Schirmer) Date: Mon, 30 Aug 2010 13:10:48 +0200 Subject: [SciPy-User] scipy.interpolate.rbf: how is "smooth" defined? Message-ID: <20100830111045.GB22961@aibn201.astro.uni-bonn.de> Hello, I'm doing a 2D fit using scipy.interpolate.rbf. I have no problem with the fit itself, it works fine. Data points are randomly scattered in a x-y plane, and have a z-value each. The data is fairly well behaved, in the sense that variations across-x-y plane are slow. The data is somewhat noisy, and thus I want to smooth it. rbf has a 'smooth' parameter which is working well, but it is exteremly poorly documented. Basically, for smooth=0 no smoothing takes place, and for smooth>0 some smoothing takes place. Does anybody know how large the smoothing length is? Is it internally modified depending on the number of nodes entering the fit? I'm asking because the number of nodes in my data can vary from a few 10s to more than 10000. Clearly, in the latter case I can do with a much smaller smoothing scale, but I'd like to determine it in some automatic fashion. It appears that the smoothing length is somehow normalised. At the moment I set 'smooth' to 10% of the average distance between nodes (in data units), resulting in a (numerically) much larger effective smoothing scale. Any insight is much appreciated! Mischa Schirmer From josef.pktd at gmail.com Mon Aug 30 08:25:09 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 30 Aug 2010 08:25:09 -0400 Subject: [SciPy-User] scipy.interpolate.rbf: how is "smooth" defined? In-Reply-To: <20100830111045.GB22961@aibn201.astro.uni-bonn.de> References: <20100830111045.GB22961@aibn201.astro.uni-bonn.de> Message-ID: On Mon, Aug 30, 2010 at 7:10 AM, Mischa Schirmer wrote: > Hello, > > I'm doing a 2D fit using scipy.interpolate.rbf. > I have no problem with the fit itself, it works fine. > > Data points are randomly scattered in a x-y plane, > and have a z-value each. The data is fairly well > behaved, in the sense that variations across-x-y > plane are slow. The data is somewhat noisy, and > thus I want to smooth it. > > rbf has a 'smooth' parameter which is working well, > but it is exteremly poorly documented. Basically, > for smooth=0 no smoothing takes place, and for > smooth>0 some smoothing takes place. smooth<0 also works, and, I think, is the correct sign for gaussian > > Does anybody know how large the smoothing length > is? Is it internally modified depending on the > number of nodes entering the fit? No, smoothing factor is directly used. My intuition mainly for gaussian case: the estimate depends on the data and a prior that biases towards zero. The matrix in the normal equations are a weighted average (except negative sign) between data (kernel matrix) with weight 1, and an identity matrix with weight smooth. It's a bit similar to Ridge Regression, where relative weights can be selected depending on the data (but often just given as parameter that depends on the "prior beliefs"). I think, it's the usual bandwidth selection problem and scipy offers no support for this (until someone adds it). Also, I don't know the rbf literature, so I have no idea if there are any easy or standard solutions for choosing the smoothing parameter, but there should be bandwidth selection criteria analogous to other smoothers. > > I'm asking because the number of nodes in my data > can vary from a few 10s to more than 10000. > Clearly, in the latter case I can do with a much > smaller smoothing scale, but I'd like to determine > it in some automatic fashion. Is this true for all radial basis functions ? If I remember correctly, I had to smooth more with many points, maybe it depends on the density of points and the amount of noise. Josef > > It appears that the smoothing length is > somehow normalised. At the moment I set 'smooth' > to 10% of the average distance between nodes > (in data units), resulting in a (numerically) > much larger effective smoothing scale. > > Any insight is much appreciated! > > Mischa Schirmer > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From josef.pktd at gmail.com Mon Aug 30 08:40:40 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 30 Aug 2010 08:40:40 -0400 Subject: [SciPy-User] OT: matplotlib.testing In-Reply-To: <4C7B6337.3090302@astraw.com> References: <4C7B6337.3090302@astraw.com> Message-ID: On Mon, Aug 30, 2010 at 3:52 AM, Andrew Straw wrote: > > On 8/28/10 8:23 PM, josef.pktd at gmail.com wrote: >> How do I get access to the testing facilities of matplotlib? >> >> How do I get access to it, and are there any docs for doing testing of >> matplotlib plots ? >> > There's not much in the way of documentation for the testing > infrastructure. You can see example tests in the > lib/matplotlib/tests/*.py files in the MPL source. > > The MPL testing facilities only made their debut in 1.0, if I recall > correctly. Thanks for the information, I will look at whether I can copy some of the testing facilities, so I can follow the examples in the tests. So far, I only looked briefly at the mpl testing source, because I thought I'm missing something. (Unfortunately, I don't want to risk upgrading to mpl 1.0 until I find the time to get away from numpy 1.4.0) Josef > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From denis-bz-gg at t-online.de Mon Aug 30 10:48:11 2010 From: denis-bz-gg at t-online.de (denis) Date: Mon, 30 Aug 2010 07:48:11 -0700 (PDT) Subject: [SciPy-User] Problems with 2D interpolation of data on polar grid In-Reply-To: References: Message-ID: Kyle, it looks as though there's a mixup here between Cartesion X,Y and polar Xnew,Ynew. griddata() will catch this to some extent with masked array Znew but the the Fitpack routines will extrapolate *without warning*. See the too-long notes under http://stackoverflow.com/questions/3526514/problem-with-2d-interpolation-in-scipy-non-rectangular-grid cheers -- denis From denis-bz-gg at t-online.de Mon Aug 30 11:18:03 2010 From: denis-bz-gg at t-online.de (denis) Date: Mon, 30 Aug 2010 08:18:03 -0700 (PDT) Subject: [SciPy-User] scipy.interpolate.rbf: how is "smooth" defined? In-Reply-To: <20100830111045.GB22961@aibn201.astro.uni-bonn.de> References: <20100830111045.GB22961@aibn201.astro.uni-bonn.de> Message-ID: On Aug 30, 1:10?pm, Mischa Schirmer wrote: > Hello, ... > rbf has a 'smooth' parameter which is working well, > but it is exteremly poorly documented. Basically, > for smooth=0 no smoothing takes place, and for > smooth>0 some smoothing takes place nope, it's just an offset for the matrix A that RBF solves: interpolate/rbf.py line 191 (scipy 0.8.0rc3) self.A = self._init_function(r) - eye(self.N)*self.smooth self.nodes = linalg.solve(self.A, self.di) Josef, we went back and forth on this back in February, and I thought agreed -- the -= should be a += (but no ticket) -- for gaussian, use smooth = -1e-6 to nudge the eigenvalues of A > 0; others have eigenvalues < 0 and > 0, so leave smooth=0 . Mischa, are you using Gaussian ? You could then try varying epsilon aka r0 in exp( - (r / r0)^2 ) which has a big affect on the fit. Its "default to approximate average distance between [all] nodes" is not so hot, better av distance to nearby nodes, i.e. smaller Gaussian peaks. cheers -- denis From josef.pktd at gmail.com Mon Aug 30 11:57:47 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 30 Aug 2010 11:57:47 -0400 Subject: [SciPy-User] scipy.interpolate.rbf: how is "smooth" defined? In-Reply-To: References: <20100830111045.GB22961@aibn201.astro.uni-bonn.de> Message-ID: On Mon, Aug 30, 2010 at 11:18 AM, denis wrote: > On Aug 30, 1:10?pm, Mischa Schirmer wrote: >> Hello, > ... >> rbf has a 'smooth' parameter which is working well, >> but it is exteremly poorly documented. Basically, >> for smooth=0 no smoothing takes place, and for >> smooth>0 some smoothing takes place > > nope, it's just an offset for the matrix A that RBF solves: > interpolate/rbf.py line 191 (scipy 0.8.0rc3) > ? ?self.A = self._init_function(r) - eye(self.N)*self.smooth > ? ?self.nodes = linalg.solve(self.A, self.di) > > Josef, we went back and forth on this back in February, and I thought > agreed > -- the -= should be a += ?(but no ticket) Yes, do you want to open a ticket? > -- for gaussian, use smooth = -1e-6 ?to nudge the eigenvalues of A > > 0; 1e-6 might be more like a minimum, in many cases it might need to be much larger. But it would be useful to have better heuristics than just trial and error. Josef > others have eigenvalues < 0 and > 0, so leave smooth=0 . > > Mischa, are you using Gaussian ? > You could then try varying epsilon aka r0 in exp( - (r / r0)^2 ) > which has a big affect on the fit. > Its "default to approximate average distance between [all] nodes" is > not so hot, > better av distance to nearby nodes, i.e. smaller Gaussian peaks. > > cheers > ?-- denis > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From gael.varoquaux at normalesup.org Mon Aug 30 12:01:41 2010 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 30 Aug 2010 18:01:41 +0200 Subject: [SciPy-User] scipy.interpolate.rbf: how is "smooth" defined? In-Reply-To: References: <20100830111045.GB22961@aibn201.astro.uni-bonn.de> Message-ID: <20100830160141.GB21959@phare.normalesup.org> On Mon, Aug 30, 2010 at 11:57:47AM -0400, josef.pktd at gmail.com wrote: > 1e-6 might be more like a minimum, in many cases it might need to be > much larger. But it would be useful to have better heuristics than > just trial and error. I have not really been following the conversation (sorry), but are you not talking about Tikhonov regularization of a covariance matrix? In which case, I have had good experience using the Ledoit-Wolf approach to setting the regularization factor. I can provide numpy code for that, if needed. Ga?l From kyle at astro.columbia.edu Mon Aug 30 12:11:31 2010 From: kyle at astro.columbia.edu (Kyle Parfrey) Date: Mon, 30 Aug 2010 12:11:31 -0400 Subject: [SciPy-User] Problems with 2D interpolation of data on polar grid In-Reply-To: References: Message-ID: Thanks everyone for all your replies, especially to Denis for those detailed notes on stackoverflow. I've managed to get it to work well with interpolate.RectBivariateSpline (as suggested above) on a rectangular (r, theta) grid, with quintic splines in theta and cubic in r, and no smoothing (s=0). I don't know why that particular combination works so well (quintic in both directions requires some smoothing, and the amount needed depends on the dataset, not just the grid, which is really bad for what I'm trying to do), I think I was being careful enough to always convert to Cartesian coordinates (both X, Y and Xnew, Ynew) etc, but the routines just seem to do a much better job on rectangular grids. If I run into problems with this later I'll try griddata() as described. Thanks again, Kyle On 30 August 2010 10:48, denis wrote: > Kyle, > ?it looks as though there's a mixup here between Cartesion X,Y and > polar Xnew,Ynew. > griddata() will catch this to some extent with masked array Znew > but the the Fitpack routines will extrapolate *without warning*. > See the too-long notes under > http://stackoverflow.com/questions/3526514/problem-with-2d-interpolation-in-scipy-non-rectangular-grid > > cheers > ?-- denis > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From josef.pktd at gmail.com Mon Aug 30 12:18:58 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 30 Aug 2010 12:18:58 -0400 Subject: [SciPy-User] scipy.interpolate.rbf: how is "smooth" defined? In-Reply-To: <20100830160141.GB21959@phare.normalesup.org> References: <20100830111045.GB22961@aibn201.astro.uni-bonn.de> <20100830160141.GB21959@phare.normalesup.org> Message-ID: On Mon, Aug 30, 2010 at 12:01 PM, Gael Varoquaux wrote: > On Mon, Aug 30, 2010 at 11:57:47AM -0400, josef.pktd at gmail.com wrote: >> 1e-6 might be more like a minimum, in many cases it might need to be >> much larger. But it would be useful to have better heuristics than >> just trial and error. > > I have not really been following the conversation (sorry), but are you > not talking about Tikhonov regularization of a covariance matrix? In > which case, I have had good experience using the Ledoit-Wolf approach to > setting the regularization factor. It's the same idea, but instead of the covariance matrix, RBF uses the kernel matrix. It will be worth a try, but I don't know whether the choice of the regularization factor will apply in the same way here. > > I can provide numpy code for that, if needed. Please do, I can also use it for other things. (I never read the small print in Ledoit-Wolf.) Josef > > Ga?l > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From gael.varoquaux at normalesup.org Mon Aug 30 12:22:29 2010 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 30 Aug 2010 18:22:29 +0200 Subject: [SciPy-User] scipy.interpolate.rbf: how is "smooth" defined? In-Reply-To: References: <20100830111045.GB22961@aibn201.astro.uni-bonn.de> <20100830160141.GB21959@phare.normalesup.org> Message-ID: <20100830162229.GC21959@phare.normalesup.org> On Mon, Aug 30, 2010 at 12:18:58PM -0400, josef.pktd at gmail.com wrote: > Please do, I can also use it for other things. (I never read the small > print in Ledoit-Wolf.) There you go (BSD licensed). This will land in scikit learn at some point because we are going to need it in different places, but I haven't found the time to polish it. Gael def ledoit_wolf(x, return_factor=False): """ Estimates the shrunk Ledoit-Wolf covariance matrix. Parameters ---------- x: 2D ndarray, shape (n, p) The data matrix, with p features and n samples. return_factor: boolean, optional If return_factor is True, the regularisation_factor is returned. Returns ------- regularised_cov: 2D ndarray Regularized covariance regularisation_factor: float Regularisation factor Notes ----- The regularised covariance is:: (1 - regularisation_factor)*cov + regularisation_factor*np.identity(n_features) """ n_samples, n_features = x.shape if n_features == 1: if return_factor: return np.atleast_2d(x.std()), 0 return np.atleast_2d(x.std()) cov = np.dot(x.T, x)/n_samples i = np.identity(n_features) mu = np.trace(cov)/n_features delta = ((cov - mu*i)**2).sum()/n_features x2 = x**2 beta_ = 1./(n_features*n_samples) * np.sum( np.dot(x2.T, x2)/n_samples - cov**2 ) beta = min(beta_, delta) alpha = delta - beta if not return_factor: return beta/delta * mu * i + alpha/delta * cov else: return beta/delta * mu * i + alpha/delta * cov, beta/delta From josef.pktd at gmail.com Mon Aug 30 12:45:27 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 30 Aug 2010 12:45:27 -0400 Subject: [SciPy-User] scipy.interpolate.rbf: how is "smooth" defined? In-Reply-To: <20100830162229.GC21959@phare.normalesup.org> References: <20100830111045.GB22961@aibn201.astro.uni-bonn.de> <20100830160141.GB21959@phare.normalesup.org> <20100830162229.GC21959@phare.normalesup.org> Message-ID: On Mon, Aug 30, 2010 at 12:22 PM, Gael Varoquaux wrote: > On Mon, Aug 30, 2010 at 12:18:58PM -0400, josef.pktd at gmail.com wrote: >> Please do, I can also use it for other things. (I never read the small >> print in Ledoit-Wolf.) > > There you go (BSD licensed). This will land in scikit learn at some point > because we are going to need it in different places, but I haven't found > the time to polish it. > > Gael > > def ledoit_wolf(x, return_factor=False): > ? ?""" Estimates the shrunk Ledoit-Wolf covariance matrix. > > ? ? ? ?Parameters > ? ? ? ?---------- > ? ? ? ?x: 2D ndarray, shape (n, p) > ? ? ? ? ? ?The data matrix, with p features and n samples. > ? ? ? ?return_factor: boolean, optional > ? ? ? ? ? ?If return_factor is True, the regularisation_factor is > ? ? ? ? ? ?returned. > > ? ? ? ?Returns > ? ? ? ?------- > ? ? ? ?regularised_cov: 2D ndarray > ? ? ? ? ? ?Regularized covariance > ? ? ? ?regularisation_factor: float > ? ? ? ? ? ?Regularisation factor > > ? ? ? ?Notes > ? ? ? ?----- > ? ? ? ?The regularised covariance is:: > > ? ? ? ? ? ?(1 - regularisation_factor)*cov > ? ? ? ? ? ? ? ? ? ?+ regularisation_factor*np.identity(n_features) > ? ?""" > ? ?n_samples, n_features = x.shape > ? ?if n_features == 1: > ? ? ? ?if return_factor: > ? ? ? ? ? ?return np.atleast_2d(x.std()), 0 > ? ? ? ?return np.atleast_2d(x.std()) > ? ?cov = np.dot(x.T, x)/n_samples > ? ?i = np.identity(n_features) > ? ?mu = np.trace(cov)/n_features > ? ?delta = ((cov - mu*i)**2).sum()/n_features > ? ?x2 = x**2 > ? ?beta_ = 1./(n_features*n_samples) * np.sum( > ? ? ? ? ? ? ? ? ? ? ? ? ? ?np.dot(x2.T, x2)/n_samples - cov**2 > ? ? ? ? ? ? ? ?) > > ? ?beta = min(beta_, delta) > ? ?alpha = delta - beta > ? ?if not return_factor: > ? ? ? ?return beta/delta * mu * i + alpha/delta * cov > ? ?else: > ? ? ? ?return beta/delta * mu * i + alpha/delta * cov, beta/delta > Thanks, If you use this with moment/normal equations for penalized least-squares, do you have to multiply also `dot(x,y)/n_samples` by `alpha/delta` ? In the normalization of RBF, smooth would be `beta/delta * mu / (alpha/delta)` or maybe additionally also divided by n_samples or something like this. Josef > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From josh.k.lawrence at gmail.com Mon Aug 30 13:08:22 2010 From: josh.k.lawrence at gmail.com (Josh Lawrence) Date: Mon, 30 Aug 2010 13:08:22 -0400 Subject: [SciPy-User] fftpack documentation error? Message-ID: <4C7BE586.8040708@gmail.com> Hey all, I think there is an error in the doc for the fftpack module. Under the section "Helper functions" there is an entry for dftfreq but scipy.fftpack doesn't have an dftfreq function. It has fftfreq. I believe the doc for scipy.fftpack is contained in scipy/fftpack/info.py in the source tree. Do I also need to file a ticket somewhere? If so, where? Cheers, Josh From robert.kern at gmail.com Mon Aug 30 13:14:07 2010 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 30 Aug 2010 12:14:07 -0500 Subject: [SciPy-User] fftpack documentation error? In-Reply-To: <4C7BE586.8040708@gmail.com> References: <4C7BE586.8040708@gmail.com> Message-ID: On Mon, Aug 30, 2010 at 12:08, Josh Lawrence wrote: > ?Hey all, > > I think there is an error in the doc for the fftpack module. Under the > section "Helper functions" there is an entry for dftfreq but > scipy.fftpack doesn't have an dftfreq function. It has fftfreq. I > believe the doc for scipy.fftpack is contained in scipy/fftpack/info.py > in the source tree. Do I also need to file a ticket somewhere? If so, where? I have fixed it through the docstring editor: http://docs.scipy.org/scipy/docs/scipy-docs/fftpack.rst/ See the information here about how to get access to the editor to make fixes on your own: http://docs.scipy.org/numpy/Front%20Page/ All of the reference manual pages on docs.scipy.org have an "Edit this page" link on the left, which will take you to the appropriate wiki page to edit: http://docs.scipy.org/doc/ -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From ndbecker2 at gmail.com Mon Aug 30 13:33:45 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 30 Aug 2010 13:33:45 -0400 Subject: [SciPy-User] signal.bilinear question Message-ID: Doc is pretty sketchy. A, B = bilinear (a, b, fs) a, b are numerator, denominator, respectively? each are polynomials in _descending_ negative powers of s? e.g.: a = a0 + a1 * s**-1 + a2 * s**-2 ... A, B are numerator, denominator, respectively? each are polynomials in descending negative powers of z? e.g.: A = A0 + A1 * z**-1 + A2 * z**-2 ... From gael.varoquaux at normalesup.org Mon Aug 30 14:15:28 2010 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 30 Aug 2010 20:15:28 +0200 Subject: [SciPy-User] scipy.interpolate.rbf: how is "smooth" defined? In-Reply-To: References: <20100830111045.GB22961@aibn201.astro.uni-bonn.de> <20100830160141.GB21959@phare.normalesup.org> <20100830162229.GC21959@phare.normalesup.org> Message-ID: <20100830181528.GA559@phare.normalesup.org> On Mon, Aug 30, 2010 at 12:45:27PM -0400, josef.pktd at gmail.com wrote: > If you use this with moment/normal equations for penalized > least-squares, do you have to multiply also `dot(x,y)/n_samples` by > `alpha/delta` ? I do not know, and I do not want to reply with something stupid. If you find out the answer, I would be interest. > In the normalization of RBF, smooth would be `beta/delta * mu / > (alpha/delta)` or maybe additionally also divided by n_samples or > something like this. I believe it would be 'beta/delta * mu / (alpha/delta)', but I am not sure whether or not the multiplication with n_samples should be applied, as I haven't really looked at the exact formulation of the problem in the context of RBFs (sorry, no time). All I know is that the trace of the covariance matrix estimate should be kept constant as much as possible. Ga?l From warren.weckesser at enthought.com Mon Aug 30 14:36:20 2010 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Mon, 30 Aug 2010 13:36:20 -0500 Subject: [SciPy-User] signal.bilinear question In-Reply-To: References: Message-ID: <4C7BFA24.3080009@enthought.com> Neal Becker wrote: > Doc is pretty sketchy. > Yup. > A, B = bilinear (a, b, fs) > > a, b are numerator, denominator, respectively? > No. a is the denominator, b is the numerator. > each are polynomials in _descending_ negative powers of s? > Yes. > e.g.: a = a0 + a1 * s**-1 + a2 * s**-2 ... > > A, B are numerator, denominator, respectively? > No--the opposite. > each are polynomials in descending negative powers of z? > Yes. For example, the wikipedia page http://en.wikipedia.org/wiki/Bilinear_transform shows that 1/(1 + (RC)s) becomes (1 + z^-1) / (1+2RC/T) + (1-2RC/T)*z^-1 With RC=3 and T=1, this means 1/(3*s + 1) becomes (1+z^-1) / (7 - 5*z^-1) Here's that calculation with bilinear: ----- In [32]: b = np.array([1.0]) In [33]: a = np.array([3.0, 1.0]) In [34]: B, A = bilinear(b, a) In [35]: B, A Out[35]: (array([ 0.14285714, 0.14285714]), array([ 1. , -0.71428571])) In [36]: 7*B, 7*A Out[36]: (array([ 1., 1.]), array([ 7., -5.])) ----- Another example, from the bottom of the 5th page of these notes: www.cs.man.ac.uk/~barry/mydocs/courses/EE4192/LFIL3.pdf ----- In [55]: b = np.array([1.0]) In [56]: a = np.array([1.0/0.828**2, np.sqrt(2.)/0.828, 1.0]) In [57]: B, A = bilinear(b, a) In [58]: B, A Out[58]: (array([ 0.09755701, 0.19511402, 0.09755701]), array([ 1. , -0.94326739, 0.33349543])) In [59]: B/B[0] Out[59]: array([ 1., 2., 1.]) ----- Warren From josef.pktd at gmail.com Mon Aug 30 14:38:42 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 30 Aug 2010 14:38:42 -0400 Subject: [SciPy-User] scipy.interpolate.rbf: how is "smooth" defined? In-Reply-To: <20100830181528.GA559@phare.normalesup.org> References: <20100830111045.GB22961@aibn201.astro.uni-bonn.de> <20100830160141.GB21959@phare.normalesup.org> <20100830162229.GC21959@phare.normalesup.org> <20100830181528.GA559@phare.normalesup.org> Message-ID: On Mon, Aug 30, 2010 at 2:15 PM, Gael Varoquaux wrote: > On Mon, Aug 30, 2010 at 12:45:27PM -0400, josef.pktd at gmail.com wrote: >> If you use this with moment/normal equations for penalized >> least-squares, do you have to multiply also `dot(x,y)/n_samples` by >> `alpha/delta` ? > > I do not know, and I do not want to reply with something stupid. If you > find out the answer, I would be interest. Maybe you find out before me, with all the other things I don't find time for, Ridge, informative Bayesian priors and Tychonov will have to wait. Josef > >> In the normalization of RBF, smooth would be `beta/delta * mu / >> (alpha/delta)` ?or maybe additionally also divided by n_samples or >> something like this. > > I believe it would be 'beta/delta * mu / (alpha/delta)', but I am not > sure whether or not the multiplication with n_samples should be applied, > as I haven't really looked at the exact formulation of the problem in the > context of RBFs (sorry, no time). All I know is that the trace of the > covariance matrix estimate should be kept constant as much as possible. > > Ga?l > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From warren.weckesser at enthought.com Mon Aug 30 14:47:17 2010 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Mon, 30 Aug 2010 13:47:17 -0500 Subject: [SciPy-User] signal.bilinear question In-Reply-To: <4C7BFA24.3080009@enthought.com> References: <4C7BFA24.3080009@enthought.com> Message-ID: <4C7BFCB5.1080807@enthought.com> Correction to my previous email--I ignored the change of convention for the names of the variables in the function call and return values: Warren Weckesser wrote: > Neal Becker wrote: > >> Doc is pretty sketchy. >> >> > > Yup. > >> A, B = bilinear (a, b, fs) >> >> a, b are numerator, denominator, respectively? >> >> > > No. a is the denominator, b is the numerator. > > Actually, you were right. I didn't pay attention to your change of convention for the names of the arguments and return values. Same for A and B below. (I don't recommend going against the conventional use of the names 'a' and 'b'.) Warren >> each are polynomials in _descending_ negative powers of s? >> >> > > Yes. > > >> e.g.: a = a0 + a1 * s**-1 + a2 * s**-2 ... >> >> A, B are numerator, denominator, respectively? >> >> > > No--the opposite. > > >> each are polynomials in descending negative powers of z? >> >> > > Yes. > > > For example, the wikipedia page > http://en.wikipedia.org/wiki/Bilinear_transform > shows that > > 1/(1 + (RC)s) > > becomes > > (1 + z^-1) / (1+2RC/T) + (1-2RC/T)*z^-1 > > With RC=3 and T=1, this means > > 1/(3*s + 1) > > becomes > > (1+z^-1) / (7 - 5*z^-1) > > Here's that calculation with bilinear: > > ----- > In [32]: b = np.array([1.0]) > > In [33]: a = np.array([3.0, 1.0]) > > In [34]: B, A = bilinear(b, a) > > In [35]: B, A > Out[35]: (array([ 0.14285714, 0.14285714]), array([ 1. , > -0.71428571])) > > In [36]: 7*B, 7*A > Out[36]: (array([ 1., 1.]), array([ 7., -5.])) > ----- > > Another example, from the bottom of the 5th page of these notes: > www.cs.man.ac.uk/~barry/mydocs/courses/EE4192/LFIL3.pdf > > ----- > In [55]: b = np.array([1.0]) > > In [56]: a = np.array([1.0/0.828**2, np.sqrt(2.)/0.828, 1.0]) > > In [57]: B, A = bilinear(b, a) > > In [58]: B, A > Out[58]: > (array([ 0.09755701, 0.19511402, 0.09755701]), > array([ 1. , -0.94326739, 0.33349543])) > > In [59]: B/B[0] > Out[59]: array([ 1., 2., 1.]) > ----- > > > Warren > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From denis-bz-gg at t-online.de Tue Aug 31 11:20:50 2010 From: denis-bz-gg at t-online.de (denis) Date: Tue, 31 Aug 2010 08:20:50 -0700 (PDT) Subject: [SciPy-User] scipy.interpolate.rbf: how is "smooth" defined? In-Reply-To: <20100830111045.GB22961@aibn201.astro.uni-bonn.de> References: <20100830111045.GB22961@aibn201.astro.uni-bonn.de> Message-ID: On Aug 30, 1:10?pm, Mischa Schirmer wrote: > Hello, > > I'm doing a 2D fit using scipy.interpolate.rbf. > I have no problem with the fit itself, it works fine. > > Data points are randomly scattered in a x-y plane, > and have a z-value each. The data is fairly well > behaved, in the sense that variations across-x-y > plane are slow. The data is somewhat noisy, and > thus I want to smooth it. > > rbf has a 'smooth' parameter which is working well, > but it is exteremly poorly documented. Basically, > for smooth=0 no smoothing takes place, and for > smooth>0 some smoothing takes place. Mischa, are you seeing smoothing with smooth > 0 ? measured how ? For varying epsilon aka r0 in exp( - (r / r0)^2 ), r0 = side * sqrt( 2 / N ) looks to be of the right scale: on a checkerboard of N points in a side^2 square, a circle of that radius will enclose its 8 neighbours. Well, the points are scattered, but. Can you try epsilon = this r0, *2, *3 keeping smooth=0 ? Of course any interpolation depends heavily on what you're fitting. (Opinion: You really want smaller Gaussian hills / smaller r0 where points are clustered, bigger where sparse. RBF is not at all adaptive -- all hills are of size r0. griddata (Delaunay triangulation + Natural Neighbour) is adaptive, Invdisttree (kd tree + inverse-distance weighting) is adaptive. ) cheers -- denis r0 = side * np.sqrt( 2 / N ) for smooth in ( 0, -1e-6 ): print "smooth %.2g" % smooth for c in range( 1, 5+1 ): rbf = Rbf( x,y,z, function=func, epsilon=c*r0, smooth=smooth ) zi = rbf( xm.flatten(), ym.flatten() ) .reshape((ngrid,ngrid)) From ralf.gommers at googlemail.com Tue Aug 31 12:18:23 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 1 Sep 2010 00:18:23 +0800 Subject: [SciPy-User] ANN: NumPy 1.5.0 release Message-ID: I am pleased to announce the availability of NumPy 1.5.0. This is the first NumPy release to include support for Python 3, as well as for Python 2.7. Binaries, sources, documentation and release notes can be found at https://sourceforge.net/projects/numpy/files/. Thank you to everyone who contributed to this release. Enjoy, the Numpy developers ========================= NumPy 1.5.0 Release Notes ========================= Highlights ========== Python 3 compatibility ---------------------- This is the first NumPy release which is compatible with Python 3. Support for Python 3 and Python 2 is done from a single code base. Extensive notes on changes can be found at ``_. Note that the Numpy testing framework relies on nose, which does not have a Python 3 compatible release yet. A working Python 3 branch of nose can be found at ``_ however. Porting of SciPy to Python 3 is expected to be completed soon. :pep:`3118` compatibility ------------------------- The new buffer protocol described by PEP 3118 is fully supported in this version of Numpy. On Python versions >= 2.6 Numpy arrays expose the buffer interface, and array(), asarray() and other functions accept new-style buffers as input. New features ============ Warning on casting complex to real ---------------------------------- Numpy now emits a `numpy.ComplexWarning` when a complex number is cast into a real number. For example: >>> x = np.array([1,2,3]) >>> x[:2] = np.array([1+2j, 1-2j]) ComplexWarning: Casting complex values to real discards the imaginary part The cast indeed discards the imaginary part, and this may not be the intended behavior in all cases, hence the warning. This warning can be turned off in the standard way: >>> import warnings >>> warnings.simplefilter("ignore", np.ComplexWarning) Dot method for ndarrays ----------------------- Ndarrays now have the dot product also as a method, which allows writing chains of matrix products as >>> a.dot(b).dot(c) instead of the longer alternative >>> np.dot(a, np.dot(b, c)) linalg.slogdet function ----------------------- The slogdet function returns the sign and logarithm of the determinant of a matrix. Because the determinant may involve the product of many small/large values, the result is often more accurate than that obtained by simple multiplication. new header ---------- The new header file ndarraytypes.h contains the symbols from ndarrayobject.h that do not depend on the PY_ARRAY_UNIQUE_SYMBOL and NO_IMPORT/_ARRAY macros. Broadly, these symbols are types, typedefs, and enumerations; the array function calls are left in ndarrayobject.h. This allows users to include array-related types and enumerations without needing to concern themselves with the macro expansions and their side- effects. Changes ======= polynomial.polynomial --------------------- * The polyint and polyder functions now check that the specified number integrations or derivations is a non-negative integer. The number 0 is a valid value for both functions. * A degree method has been added to the Polynomial class. * A trimdeg method has been added to the Polynomial class. It operates like truncate except that the argument is the desired degree of the result, not the number of coefficients. * Polynomial.fit now uses None as the default domain for the fit. The default Polynomial domain can be specified by using [] as the domain value. * Weights can be used in both polyfit and Polynomial.fit * A linspace method has been added to the Polynomial class to ease plotting. * The polymulx function was added. polynomial.chebyshev -------------------- * The chebint and chebder functions now check that the specified number integrations or derivations is a non-negative integer. The number 0 is a valid value for both functions. * A degree method has been added to the Chebyshev class. * A trimdeg method has been added to the Chebyshev class. It operates like truncate except that the argument is the desired degree of the result, not the number of coefficients. * Chebyshev.fit now uses None as the default domain for the fit. The default Chebyshev domain can be specified by using [] as the domain value. * Weights can be used in both chebfit and Chebyshev.fit * A linspace method has been added to the Chebyshev class to ease plotting. * The chebmulx function was added. * Added functions for the Chebyshev points of the first and second kind. histogram --------- After a two years transition period, the old behavior of the histogram function has been phased out, and the "new" keyword has been removed. correlate --------- The old behavior of correlate was deprecated in 1.4.0, the new behavior (the usual definition for cross-correlation) is now the default. Checksums ========= 738572dd3e5d4cd89e98c76cc3f162a9 release/installers/numpy-1.5.0-py2.5-python.org.dmg f58ebc1840974cf2ef8b4f41516dc288 release/installers/numpy-1.5.0-py2.6-python.org.dmg d7232048392ede8d8d8fb57839cb4b91 release/installers/numpy-1.5.0-py2.7-python.org.dmg c5130a11f920706cdc665ef1e07772e2 release/installers/numpy-1.5.0-win32-superpack-python2.5.exe b46a52f82126ace1eb7cb627623c64dc release/installers/numpy-1.5.0-win32-superpack-python2.6.exe 8a93c004a104f6231de4c398e2b3c48f release/installers/numpy-1.5.0-win32-superpack-python2.7.exe 1d255b58764d465e64b7b3eee832aa9e release/installers/numpy-1.5.0-win32-superpack-python3.1.exe 3a8bfdc434df782d647161c48943ee09 release/installers/numpy-1.5.0.tar.gz 11354c0ca15ca6f37df9589bd4f25943 release/installers/numpy-1.5.0.zip -------------- next part -------------- An HTML attachment was scrubbed... URL: From phil.cummins at anu.edu.au Mon Aug 23 19:22:29 2010 From: phil.cummins at anu.edu.au (Phil Cummins) Date: Mon, 23 Aug 2010 23:22:29 -0000 Subject: [SciPy-User] help with scipy cholesky_banded References: <29817_1282541862_4C720925_29817_44347_1_16355C68-89E7-4BFA-8EF5-3A286B575451@anu.edu.au> Message-ID: <3734769F-D030-4D72-B2A7-DAAE95314683@anu.edu.au> Hi, I am using Enthought epd64 version 6.2-2 on Mac OS 10.6.4. The version of scipy used is 0.8.0.dev6485. I am trying to create a finite element solution to a simple, 1D differential equation. I have constructed a stiffness matrix and called scipy.linalg.cholesky_banded to decompose it. Cholesky_banded complains that the matrix is not positive definite, which should not happen if it is a properly constructed FEM stiffness matrix. I checked my construction many times, and finally started looking into why cholesky_banded thinks it's not positive definite My understanding is that, since my matrix is real, cholesky_banded reduces to a call to the lapack routine dpbtrf, and since I don't think I'm using the 'blocked' version, I think it should be using dpbtrf2. So I went and found dpbtrf2, and discovered that its check for positive definiteness appears to consist of testing whether any of the diagonals are <= 0. I downloaded dpbtrf2 and commented out the lines that called external routines (i.e., leaving pretty much just the check for positive definiteness). I used f2py to import it as a module, so that I could investigate why the positive definitiveness check fails. Attached is a python script which sets the elements of the stiffness matrix. It then does the following (note that the matrix Stiff is banded symmetric, with only 1 super diagonal, stored as upper triangular): import scipy.linalg import math from numpy import * import dpbtf2 # My own dpbtrf2 module created with "f2py -c dpbtf2.pyf dpbtf2.f" . . . for i in range(0,len(rg)): if Stiff[1][i] <= 0.: print 'non positive diagonal %d <= 0.: %g' % (i,Stiff[1][i]) info = dpbtf2.dpbtf2('U',len(rg),1,Stiff,2) print 'dpbtf2:info=',info C, info = scipy.linalg.flapack.dpbtrf(Stiff) print 'dpbtrf:info=',info So, in python I check whether any diagonal elements are <= 0., then I check this with my modified dpbtrf2 fortran subroutine, and then I call the linalg one. Here's what happens: $ ./scipy_bug.py dpbtf2:info= 0 dpbtrf:info= 1001 Python doesn't find a diagonal element <=0., nor does the fortran routine dpbtrf2. But the scipy.linalg.dpbtrf does - so the latter fails. Can anyone please help me understand why scipy.linalg.dpbtr doesn't seem to work for me? Thanks, - Phil > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: scipy_bug.py Type: text/x-python-script Size: 1676 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: dpbtf2.f Type: application/octet-stream Size: 5795 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: dpbtf2.pyf Type: application/octet-stream Size: 627 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: