From evilper at gmail.com Wed Aug 1 08:15:51 2012 From: evilper at gmail.com (Per Nielsen) Date: Wed, 1 Aug 2012 14:15:51 +0200 Subject: [SciPy-User] Matrix-free version of connected_components Message-ID: Hi all, I work with large sparse matrices which consists of several disconnected components and I only need one of these. So far I have succesfully been using scipy.sparse.csgraph.connected_components to dig out the component I needed. This algorithm does however require the entire sparse matrix as input, which sometimes is too large to fit in memory. For this reason I wondered whether there existed a matrix-free version of connected_components, or another algorithm achieving the same, where I would only need to provide a function calculating the sparse matrix-vector product. Any help, hints, or information would be greatly appreciated :) Cheers, Per -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at hilboll.de Wed Aug 1 09:13:04 2012 From: lists at hilboll.de (Andreas Hilboll) Date: Wed, 1 Aug 2012 15:13:04 +0200 Subject: [SciPy-User] calculating the mean for each factor (like tapply in R) In-Reply-To: References: Message-ID: > Hi there, > > I've just moved from R to IPython and wondered if there was a good way of > finding the means and/or variance of values in a dataframe given a factor > > e.g.: > if df = > x experiment > 10 1 > 13 1 > 12 1 > 3 2 > 4 2 > 6 2 > 33 3 > 44 3 > 55 3 > > in tapply you would do: > > tapply(df$x, list(df$experiment), mean) > tapply(df$x, list(df$experiment), var) > > I guess I can always loop through the array for each experiment type, but > thought that this is the kind of functionality that would be included in a > core library. Pandas (http://pandas.pydata.org/) seems to be what you're looking for. It has a DataFrame class which allows grouping of data. Cheers, Andreas. From gael.varoquaux at normalesup.org Wed Aug 1 09:18:54 2012 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 1 Aug 2012 15:18:54 +0200 Subject: [SciPy-User] Matrix-free version of connected_components In-Reply-To: References: Message-ID: <20120801131854.GA9137@phare.normalesup.org> You can try spectral graph partioning, computing the Fiedler with arpack, that only needs matrix-vector product: http://mathworld.wolfram.com/FiedlerVector.html http://en.wikipedia.org/wiki/Algebraic_connectivity HTH, Gael On Wed, Aug 01, 2012 at 02:15:51PM +0200, Per Nielsen wrote: > Hi all, > I work with large sparse matrices which consists of several disconnected > components and I only need one of these. So far I have succesfully been > using scipy.sparse.csgraph.connected_components to dig out the component I > needed. This algorithm does however require the entire sparse matrix as > input, which sometimes is too large to fit in memory. > For this reason I wondered whether there existed a matrix-free version > of connected_components, or another algorithm achieving the same, where I > would only need to provide a function calculating the sparse matrix-vector > product. > Any help, hints, or information would be greatly appreciated :) > Cheers, > Per -- Gael Varoquaux Researcher, INRIA Parietal Laboratoire de Neuro-Imagerie Assistee par Ordinateur NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France Phone: ++ 33-1-69-08-79-68 http://gael-varoquaux.info http://twitter.com/GaelVaroquaux From guziy.sasha at gmail.com Wed Aug 1 09:35:53 2012 From: guziy.sasha at gmail.com (Oleksandr Huziy) Date: Wed, 1 Aug 2012 09:35:53 -0400 Subject: [SciPy-User] calculating the mean for each factor (like tapply in R) In-Reply-To: References: Message-ID: Hi, It is pretty much the same as looping, but you could do the following In [1]: import numpy as np In [2]: exps = np.array([10,13,12,3,4,6,33,44,55]) In [3]: x = np.array([10,13,12,3,4,6,33,44,55]) In [4]: exps = np.array([1,1,1,2,2,2,3,3,3]) z = [np.mean(x[exps == i]) for i in np.unique( exps )] -- Oleksandr (Sasha) Huziy 2012/8/1 Andreas Hilboll > > Hi there, > > > > I've just moved from R to IPython and wondered if there was a good way of > > finding the means and/or variance of values in a dataframe given a factor > > > > e.g.: > > if df = > > x experiment > > 10 1 > > 13 1 > > 12 1 > > 3 2 > > 4 2 > > 6 2 > > 33 3 > > 44 3 > > 55 3 > > > > in tapply you would do: > > > > tapply(df$x, list(df$experiment), mean) > > tapply(df$x, list(df$experiment), var) > > > > I guess I can always loop through the array for each experiment type, but > > thought that this is the kind of functionality that would be included in > a > > core library. > > Pandas (http://pandas.pydata.org/) seems to be what you're looking for. It > has a DataFrame class which allows grouping of data. > > Cheers, Andreas. > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergio_r at mail.com Wed Aug 1 09:58:17 2012 From: sergio_r at mail.com (Sergio Rojas) Date: Wed, 01 Aug 2012 09:58:17 -0400 Subject: [SciPy-User] (no subject) Message-ID: <20120801135818.4290@gmx.com> Dear all, The sentences bellow will always raise an Error or Exception just as follows, which is a little anomaly. Is this a bug? >>> u1=numpy.random.rand(100000) >>> u2=numpy.random.rand(100000) >>> scipy.stats.kendalltau(u1,u2) --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) /home/zfyuan/phd/paper1/pyvine_lap/ in () ----> 1 sp.stats.kendalltau(u1,u2) /usr/lib64/python2.7/site-packages/scipy/stats/stats.pyc in kendalltau(x, y, initial_lexsort) 2673 2674 tau = ((tot - (v + u - t)) - 2.0 * exchanges) / \ -> 2675 np.sqrt((tot - u) * (tot - v)) 2676 2677 # what follows reproduces the ending of Gary Strangman's original AttributeError: sqrt -- Jeffrey ------------------------------ >>> import numpy as np >>> import scipy.stats >>> u1=np.random.rand(100000) >>> u2=np.random.rand(100000) >>> scipy.stats.kendalltau(u1,u2) (0.00094913269132691487, 0.65256243280384563) >>> np.version.version '1.6.1' >>> scipy.version.version '0.10.1' >>> import os >>> os.sys.version '2.7.2 (default, Apr 21 2012, 14:16:53) \n[GCC 4.6.1]' Sergio -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergio_r at mail.com Wed Aug 1 10:06:47 2012 From: sergio_r at mail.com (Sergio Rojas) Date: Wed, 01 Aug 2012 10:06:47 -0400 Subject: [SciPy-User] scipy.stats.kendalltau bug? (Jeffrey) Message-ID: <20120801140647.4310@gmx.com> Dear all, The sentences bellow will always raise an Error or Exception just as follows, which is a little anomaly. Is this a bug? >>> u1=numpy.random.rand(100000) >>> u2=numpy.random.rand(100000) >>> scipy.stats.kendalltau(u1,u2) --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) /home/zfyuan/phd/paper1/pyvine_lap/ in () ----> 1 sp.stats.kendalltau(u1,u2) /usr/lib64/python2.7/site-packages/scipy/stats/stats.pyc in kendalltau(x, y, initial_lexsort) 2673 2674 tau = ((tot - (v + u - t)) - 2.0 * exchanges) / \ -> 2675 np.sqrt((tot - u) * (tot - v)) 2676 2677 # what follows reproduces the ending of Gary Strangman's original AttributeError: sqrt -- Jeffrey ------------------------------ >>> import numpy as np >>> import scipy.stats >>> u1=np.random.rand(100000) >>> u2=np.random.rand(100000) >>> scipy.stats.kendalltau(u1,u2) (0.00094913269132691487, 0.65256243280384563) >>> np.version.version '1.6.1' >>> scipy.version.version '0.10.1' >>> import os >>> os.sys.version '2.7.2 (default, Apr 21 2012, 14:16:53) \n[GCC 4.6.1]' Sergio -------------- next part -------------- An HTML attachment was scrubbed... URL: From srey at asu.edu Wed Aug 1 10:55:10 2012 From: srey at asu.edu (Serge Rey) Date: Wed, 1 Aug 2012 07:55:10 -0700 Subject: [SciPy-User] ANN: PySAL 1.4 Message-ID: On behalf of the PySAL development team, I'm happy to announce the official release of PySAL 1.4. PySAL is a library of tools for spatial data analysis and geocomputation written in Python. PySAL 1.4, the fifth official release of PySAL brings the following key enhancements: ? ?- ?spatial regression (spreg): - Integration of scipy.sparse structures in all the regression classes - Probit - Probit with diagnostics for spatial effects ? - ?exploratory spatial data analysis (esda): - Generalized Gamma statistics for spatial autocorrelation ? - ?spatial dynamics (spatial_dynamics): - O(n log n) algorithm for spatial tau (spatial rank correlation) - LISA Markov transitions and tests for spatial dynamics ? - computational geometry (cg) and spatial networks - efficient point to nearest LineSegment search - generate network topology from shapefiles - utilities for interfacing with networkx - integration of network cluster code (beta) among the 155 commits and bug fixes since the last release, 6 months ago. PySAL modules ------------- ? ? - pysal.core ? Core Data Structures and IO ? ? - pysal.cg ? Computational Geometry ? ? - pysal.esda ? Exploratory Spatial Data Analysis ? ? - pysal.inequality ? Spatial Inequality Analysis ? ? - pysal.spatial_dynamics ? Spatial Dynamics ? ? - pysal.spreg - Regression and Diagnostics ? ? - pysal.region ? Spatially Constrained Clustering ? ? - pysal.weights ? Spatial Weights ? ? - pysal.FileIO ? PySAL FileIO: Module for reading and writing various file types in a Pythonic way Downloads -------------- Binary installers and source distributions are available for download at http://code.google.com/p/pysal/downloads/list Documentation ------------- The documentation site is here http://pysal.geodacenter.org/1.4 Web sites --------- PySAL's home is here http://pysal.org/ The developer's site is here http://code.google.com/p/pysal/ Mailing Lists ------------- Please see the developer's list here http://groups.google.com/group/pysal-dev Help for users is here http://groups.google.com/group/openspace-list Bug reports and feature requests -------------------------------- To search for or report bugs, as well as request enhancements, please see http://code.google.com/p/pysal/issues/list License information ------------------- See the file "LICENSE.txt" for information on the history of this software, terms & conditions for usage, and a DISCLAIMER OF ALL WARRANTIES. Many thanks to all who contributed! Serge, on behalf of the PySAL development team. -- Sergio (Serge) Rey Professor, School of Geographical Sciences and Urban Planning GeoDa Center for Geospatial Analysis and Computation Arizona State University http://geoplan.asu.edu/rey Editor, International Regional Science Review http://irx.sagepub.com From njs at pobox.com Wed Aug 1 12:53:14 2012 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 1 Aug 2012 17:53:14 +0100 Subject: [SciPy-User] B-spline basis functions? In-Reply-To: References: Message-ID: On Tue, Jul 31, 2012 at 4:46 PM, Charles R Harris wrote: > On Tue, Jul 31, 2012 at 9:37 AM, Nathaniel Smith wrote: >> >> Hi all, >> >> I'd like to be able to do spline regression in patsy[1], which means >> that I need to be able to compute b-spline basis functions. I am not >> an initiate into the mysteries of practical spline computations, but I >> *think* the stuff in scipy.signal is not quite usable as is, because >> it's focused on doing interpolation directly rather than exposing the >> basis functions themselves? >> >> Specifically, to achieve feature parity with R [2], I need to be able to >> take >> - an arbitrary order >> - an arbitrary collection of knot positions (which may be irregularly >> spaced) >> - a vector x of points at which to evaluate the basis functions >> and spit out the value of each spline basis function evaluated at each >> point in the x vector. >> >> It looks like scipy.signal.bspline *might* be useful, but I can't >> quite tell? Or alternatively someone might have some code lying around >> to do this already? >> >> Basically I have a copy of Schumaker here and I'm hoping someone will >> save me from having to read it :-). >> > > I have this floating around [...] > v[i] = spl.splev(x, (knots, d[i], deg)) This looks fabulous, thank you! From a quick look it seems to be producing numerically identical results to R's spline.des(). The bit with the coefficient vectors like [0, 0, 0, 1, 0] worries me a bit -- do you know if it's producing the entire spline basis internally on every call, and then throwing away most of them when forming the inner product? If so then in practice it's probably not a show-stopper, but it would be a pointless factor-of-len(knots) increase in running time for no reason. -n From charlesr.harris at gmail.com Wed Aug 1 13:21:59 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 1 Aug 2012 11:21:59 -0600 Subject: [SciPy-User] B-spline basis functions? In-Reply-To: References: Message-ID: On Wed, Aug 1, 2012 at 10:53 AM, Nathaniel Smith wrote: > On Tue, Jul 31, 2012 at 4:46 PM, Charles R Harris > wrote: > > On Tue, Jul 31, 2012 at 9:37 AM, Nathaniel Smith wrote: > >> > >> Hi all, > >> > >> I'd like to be able to do spline regression in patsy[1], which means > >> that I need to be able to compute b-spline basis functions. I am not > >> an initiate into the mysteries of practical spline computations, but I > >> *think* the stuff in scipy.signal is not quite usable as is, because > >> it's focused on doing interpolation directly rather than exposing the > >> basis functions themselves? > >> > >> Specifically, to achieve feature parity with R [2], I need to be able to > >> take > >> - an arbitrary order > >> - an arbitrary collection of knot positions (which may be irregularly > >> spaced) > >> - a vector x of points at which to evaluate the basis functions > >> and spit out the value of each spline basis function evaluated at each > >> point in the x vector. > >> > >> It looks like scipy.signal.bspline *might* be useful, but I can't > >> quite tell? Or alternatively someone might have some code lying around > >> to do this already? > >> > >> Basically I have a copy of Schumaker here and I'm hoping someone will > >> save me from having to read it :-). > >> > > > > I have this floating around > [...] > > v[i] = spl.splev(x, (knots, d[i], deg)) > > This looks fabulous, thank you! From a quick look it seems to be > producing numerically identical results to R's spline.des(). > > The bit with the coefficient vectors like [0, 0, 0, 1, 0] worries me a > bit -- do you know if it's producing the entire spline basis > internally on every call, and then throwing away most of them when > forming the inner product? If so then in practice it's probably not a > show-stopper, but it would be a pointless factor-of-len(knots) > increase in running time for no reason. > Yes it is. It was a quick and dirty routine and producing the array wasn't the bottleneck. Also, since the b-splines themselves have small support, a lot of extra zero values are produced. It wouldn't be difficult to improve things quite a bit using a hacked version of the splev.f routine (De Boor's algorithm ). IIRC, the b-splines in signal are periodic b-splines with reflected boundary conditions, so aren't as general. I could be wrong about that, however. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Wed Aug 1 15:03:46 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 1 Aug 2012 21:03:46 +0200 Subject: [SciPy-User] scipy.stats.kendalltau bug? In-Reply-To: References: <501411E4.6060308@mail.ustc.edu.cn> <501417D1.8050402@mail.ustc.edu.cn> <5014E5E2.1010002@mail.ustc.edu.cn> <5015059C.60406@mail.ustc.edu.cn> Message-ID: On Sun, Jul 29, 2012 at 12:30 PM, Nathaniel Smith wrote: > On Sun, Jul 29, 2012 at 10:42 AM, Jeffrey wrote: > > On 07/29/2012 03:47 PM, Nathaniel Smith wrote: > >> On Sun, Jul 29, 2012 at 8:27 AM, Jeffrey > wrote: > >>> Thanks eat. I found the reason is that numpy.sqrt cannot deal with too > large > >>> number. When calculating kendalltau, assume n=len(x),then the total > pair > >>> number is 'tot' below: > >>> > >>> tot=(n-1)*n//2 > >>> > >>> when calculating tau, the de-numerator is as below: > >>> > >>> np.sqrt((tot-u)*(tot-v)) > >>> > >>> u and v stands for ties in x[] and y[perm[]], which is zero if the two > array > >>> sample from continuous dist. Hence (tot-u)*(tot-v) may be out of range > for > >>> the C written ufunc 'np.sqrt', and an Error is then raised. > >>> > >>> What about using math.sqrt here, or multiply two np.sqrt in the > >>> de-numerator? Since big data sets are often seen these days. > >> It seems like the bug is that np.sqrt is raising an AttributeError on > >> valid input... can you give an example of a value that np.sqrt fails > >> on? Like > > > > Assume the input array x and y has n=100000 length, which is common > > seen, and assume there is no tie in both x and y, hence u=0, v=0 and t=0 > > in the scipy.stats.kendalltau subroutine. Hence the de-numerator of > > expression for calculating tau would be as follows: > > > > np.sqrt( (tot-u) * (tot-v) ) > > > > Here above, tot= n * (n-1) //2=499950000, and (tot-u) * (tot-v)= tot*tot > > = 24999500002500000000L, this long int will raise Error when np.sqrt is > > applied. I think type convert, like 'float()' should be done before > > np.sqrt, or write like np.sqrt(tot-u) * np.sqrt(tot-v) to avoid long > > integer. > > > > Thanks a lot : ) > > Thanks, that clarifies things: https://github.com/numpy/numpy/issues/368 > > For now, yeah, some sort of workaround makes sense, though... in > addition to the ones you mention, I noticed that this also seems to > work: > > np.sqrt(bignum, dtype=float) > > You should submit a pull request :-). > This was already fixed for 0.10.x: https://github.com/scipy/scipy/commit/ce14ddb Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Wed Aug 1 17:29:41 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 1 Aug 2012 23:29:41 +0200 Subject: [SciPy-User] scipy.pkgload() error In-Reply-To: <1627248.DDPmKElzJ1@kruimel> References: <5011A0F2.9090708@sun.ac.za> <1627248.DDPmKElzJ1@kruimel> Message-ID: On Tue, Jul 31, 2012 at 8:18 AM, Johann Rohwer wrote: > On Monday 30 July 2012 22:29:40 Pauli Virtanen wrote: > > 30.07.2012 21:52, Ralf Gommers kirjoitti: > > [clip] > > > > > My proposal would be to delete pkgload from the scipy namespace. > > > > Or maybe have it do __import__('scipy.' + pkgname) > > I had some old code that assumed pkgload would add the submodule to > > the main namespace, so maybe trying to restore this could be one > > option. > Since that currently doesn't work (also for released versions) and isn't even any shorter to type than "from scipy import module", I don't really see the point of that. Alternatively, just remove it... > > I agree that > scipy.pkgload('io') and import scipy.io > are probably not much different. However, I did find the convenience > function > scipy.pkgload() > useful to load all the subpackages and it would be nice to have that > option, although it's of course not a show stopper. > Reimplementing pkgload() which imports all modules seems fine to me, as long as we document that it's meant for interactive usage only. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From wesmckinn at gmail.com Wed Aug 1 21:32:39 2012 From: wesmckinn at gmail.com (Wes McKinney) Date: Wed, 1 Aug 2012 21:32:39 -0400 Subject: [SciPy-User] calculating the mean for each factor (like tapply in R) In-Reply-To: References: Message-ID: On Wed, Aug 1, 2012 at 9:35 AM, Oleksandr Huziy wrote: > Hi, > > It is pretty much the same as looping, but you could do the following > > In [1]: import numpy as np > > In [2]: exps = np.array([10,13,12,3,4,6,33,44,55]) > > In [3]: x = np.array([10,13,12,3,4,6,33,44,55]) > > In [4]: exps = np.array([1,1,1,2,2,2,3,3,3]) > > z = [np.mean(x[exps == i]) for i in np.unique( exps )] > > -- > Oleksandr (Sasha) Huziy > > > 2012/8/1 Andreas Hilboll >> >> > Hi there, >> > >> > I've just moved from R to IPython and wondered if there was a good way >> > of >> > finding the means and/or variance of values in a dataframe given a >> > factor >> > >> > e.g.: >> > if df = >> > x experiment >> > 10 1 >> > 13 1 >> > 12 1 >> > 3 2 >> > 4 2 >> > 6 2 >> > 33 3 >> > 44 3 >> > 55 3 >> > >> > in tapply you would do: >> > >> > tapply(df$x, list(df$experiment), mean) >> > tapply(df$x, list(df$experiment), var) >> > >> > I guess I can always loop through the array for each experiment type, >> > but >> > thought that this is the kind of functionality that would be included in >> > a >> > core library. >> >> Pandas (http://pandas.pydata.org/) seems to be what you're looking for. It >> has a DataFrame class which allows grouping of data. >> >> Cheers, Andreas. >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > For the #lazyweb, here is what this looks like in pandas: In [24]: df Out[24]: x experiment 0 10 1 1 13 1 2 12 1 3 3 2 4 4 2 5 6 2 6 33 3 7 44 3 8 55 3 In [25]: df.groupby('experiment').x.mean() Out[25]: experiment 1 11.666667 2 4.333333 3 44.000000 Name: x In [26]: df.groupby('experiment').x.var() Out[26]: experiment 1 2.333333 2 2.333333 3 121.000000 Name: x or if you want to be fancy: In [27]: df.groupby('experiment').x.agg(['mean', 'var']) Out[27]: mean var experiment 1 11.666667 2.333333 2 4.333333 2.333333 3 44.000000 121.000000 There are good reasons to use pandas over a DIY approach with NumPy array operations; notably I use smart algorithms so that the runtime scales linearly with the side of the data instead of quadratically. - Wes From jrocher at enthought.com Wed Aug 1 22:43:41 2012 From: jrocher at enthought.com (Jonathan Rocher) Date: Wed, 1 Aug 2012 21:43:41 -0500 Subject: [SciPy-User] [CONF] Call for papers AMS Jan 6-10 2013 Message-ID: Dear all, Sorry for the cross post. Abstract submission is open for the Third Symposium on Advances in Modeling and Analysis Using Python at the AMS Annual Meeting in Austin, TX, January 6-10, 2013. The call for papers and link to submit an abstract is located here: http://annual.ametsoc.org/2013/index.cfm/programs-and-events/conferences-and-symposia/third-symposium-on-advances-in-modeling-and-analysis-using-python/ The abstract submission deadline is extended to August 28, 2012. We are soliciting papers related to all areas of the use of Python in the atmospheric and oceanic sciences. Please pass along this announcement to your friends and colleagues! Thanks! Regards, Jonathan for the organizing committee -- Jonathan Rocher, PhD Scientific software developer Enthought, Inc. jrocher at enthought.com 1-512-536-1057 http://www.enthought.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From evilper at gmail.com Thu Aug 2 03:07:27 2012 From: evilper at gmail.com (Per Nielsen) Date: Thu, 2 Aug 2012 09:07:27 +0200 Subject: [SciPy-User] Matrix-free version of connected_components In-Reply-To: <20120801131854.GA9137@phare.normalesup.org> References: <20120801131854.GA9137@phare.normalesup.org> Message-ID: Thanks for the reply. This approach looks like it could do the job, although it appaers to be significantly more computationally demanding than connected_components, due to the repeated need to find the Fiedler vector of very large systems. But I will give it a go! :) Per On Wed, Aug 1, 2012 at 3:18 PM, Gael Varoquaux < gael.varoquaux at normalesup.org> wrote: > You can try spectral graph partioning, computing the Fiedler with arpack, > that only needs matrix-vector product: > http://mathworld.wolfram.com/FiedlerVector.html > http://en.wikipedia.org/wiki/Algebraic_connectivity > > HTH, > > Gael > > On Wed, Aug 01, 2012 at 02:15:51PM +0200, Per Nielsen wrote: > > Hi all, > > > I work with large sparse matrices which consists of several disconnected > > components and I only need one of these. So far I have succesfully been > > using scipy.sparse.csgraph.connected_components to dig out the component > I > > needed. This algorithm does however require the entire sparse matrix as > > input, which sometimes is too large to fit in memory. > > > For this reason I wondered whether there existed a matrix-free version > > of connected_components, or another algorithm achieving the same, where I > > would only need to provide a function calculating the sparse > matrix-vector > > product. > > > Any help, hints, or information would be greatly appreciated :) > > > Cheers, > > Per > > > -- > Gael Varoquaux > Researcher, INRIA Parietal > Laboratoire de Neuro-Imagerie Assistee par Ordinateur > NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France > Phone: ++ 33-1-69-08-79-68 > http://gael-varoquaux.info http://twitter.com/GaelVaroquaux > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidmenhur at gmail.com Thu Aug 2 06:50:15 2012 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Thu, 2 Aug 2012 11:50:15 +0100 Subject: [SciPy-User] Matrix-free version of connected_components In-Reply-To: References: <20120801131854.GA9137@phare.normalesup.org> Message-ID: I am not sure if this is what you are looking for, but an alternative approach would be to transform the matrix into a graph, and then get the connected components using, for eample, networkx. You can find a complete description about the topic in: "Complex networks: Structure and dynamics" S. Boccaletti et al. (2006) Phys. Reports. Sorry that I cannot provide anything more concise. David. On Thu, Aug 2, 2012 at 8:07 AM, Per Nielsen wrote: > Thanks for the reply. > > This approach looks like it could do the job, although it appaers to be > significantly more computationally demanding than connected_components, due > to the repeated need to find the Fiedler vector of very large systems. But I > will give it a go! :) > > Per > > > On Wed, Aug 1, 2012 at 3:18 PM, Gael Varoquaux > wrote: >> >> You can try spectral graph partioning, computing the Fiedler with arpack, >> that only needs matrix-vector product: >> http://mathworld.wolfram.com/FiedlerVector.html >> http://en.wikipedia.org/wiki/Algebraic_connectivity >> >> HTH, >> >> Gael >> >> On Wed, Aug 01, 2012 at 02:15:51PM +0200, Per Nielsen wrote: >> > Hi all, >> >> > I work with large sparse matrices which consists of several disconnected >> > components and I only need one of these. So far I have succesfully been >> > using scipy.sparse.csgraph.connected_components to dig out the component >> > I >> > needed. This algorithm does however require the entire sparse matrix as >> > input, which sometimes is too large to fit in memory. >> >> > For this reason I wondered whether there existed a matrix-free version >> > of connected_components, or another algorithm achieving the same, where >> > I >> > would only need to provide a function calculating the sparse >> > matrix-vector >> > product. >> >> > Any help, hints, or information would be greatly appreciated :) >> >> > Cheers, >> > Per >> >> >> -- >> Gael Varoquaux >> Researcher, INRIA Parietal >> Laboratoire de Neuro-Imagerie Assistee par Ordinateur >> NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France >> Phone: ++ 33-1-69-08-79-68 >> http://gael-varoquaux.info http://twitter.com/GaelVaroquaux >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From evilper at gmail.com Thu Aug 2 08:15:07 2012 From: evilper at gmail.com (Per Nielsen) Date: Thu, 2 Aug 2012 14:15:07 +0200 Subject: [SciPy-User] Matrix-free version of connected_components In-Reply-To: References: <20120801131854.GA9137@phare.normalesup.org> Message-ID: I have looked into the functions provided by the networkX package, however, as far as I have been able to understand both networkX and scipy employ similar methods for finding the connected components. Thus I still need the full sparse matrix (or graph which I think is similar in memory usage) in memory, which I would like to avoid. Thank you for your reply and the reference. Per On Thu, Aug 2, 2012 at 12:50 PM, Da?id wrote: > I am not sure if this is what you are looking for, but an alternative > approach would be to transform the matrix into a graph, and then get > the connected components using, for eample, networkx. You can find a > complete description about the topic in: "Complex networks: Structure > and dynamics" S. Boccaletti et al. (2006) Phys. Reports. Sorry that I > cannot provide anything more concise. > > > David. > > On Thu, Aug 2, 2012 at 8:07 AM, Per Nielsen wrote: > > Thanks for the reply. > > > > This approach looks like it could do the job, although it appaers to be > > significantly more computationally demanding than connected_components, > due > > to the repeated need to find the Fiedler vector of very large systems. > But I > > will give it a go! :) > > > > Per > > > > > > On Wed, Aug 1, 2012 at 3:18 PM, Gael Varoquaux > > wrote: > >> > >> You can try spectral graph partioning, computing the Fiedler with > arpack, > >> that only needs matrix-vector product: > >> http://mathworld.wolfram.com/FiedlerVector.html > >> http://en.wikipedia.org/wiki/Algebraic_connectivity > >> > >> HTH, > >> > >> Gael > >> > >> On Wed, Aug 01, 2012 at 02:15:51PM +0200, Per Nielsen wrote: > >> > Hi all, > >> > >> > I work with large sparse matrices which consists of several > disconnected > >> > components and I only need one of these. So far I have succesfully > been > >> > using scipy.sparse.csgraph.connected_components to dig out the > component > >> > I > >> > needed. This algorithm does however require the entire sparse matrix > as > >> > input, which sometimes is too large to fit in memory. > >> > >> > For this reason I wondered whether there existed a matrix-free version > >> > of connected_components, or another algorithm achieving the same, > where > >> > I > >> > would only need to provide a function calculating the sparse > >> > matrix-vector > >> > product. > >> > >> > Any help, hints, or information would be greatly appreciated :) > >> > >> > Cheers, > >> > Per > >> > >> > >> -- > >> Gael Varoquaux > >> Researcher, INRIA Parietal > >> Laboratoire de Neuro-Imagerie Assistee par Ordinateur > >> NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France > >> Phone: ++ 33-1-69-08-79-68 > >> http://gael-varoquaux.info > http://twitter.com/GaelVaroquaux > >> _______________________________________________ > >> SciPy-User mailing list > >> SciPy-User at scipy.org > >> http://mail.scipy.org/mailman/listinfo/scipy-user > > > > > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Thu Aug 2 08:36:01 2012 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 2 Aug 2012 14:36:01 +0200 Subject: [SciPy-User] Matrix-free version of connected_components In-Reply-To: References: <20120801131854.GA9137@phare.normalesup.org> Message-ID: <20120802123601.GF3396@phare.normalesup.org> On Thu, Aug 02, 2012 at 02:15:07PM +0200, Per Nielsen wrote: > I have looked into the functions provided by the networkX package, > however, as far as I have been able to understand both networkX and scipy > employ similar methods for finding the connected components. Thus I still > need the full sparse matrix (or graph which I think is similar in memory > usage) in memory, which I would like to avoid. But you can easily recode them to replace the indexing by a function call that would return the list of neighbors for a given node. That way you don't have to store the graph, you can generate it on the fly. G From charlesr.harris at gmail.com Thu Aug 2 08:46:41 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 2 Aug 2012 06:46:41 -0600 Subject: [SciPy-User] Matrix-free version of connected_components In-Reply-To: References: <20120801131854.GA9137@phare.normalesup.org> Message-ID: On Thu, Aug 2, 2012 at 6:15 AM, Per Nielsen wrote: > I have looked into the functions provided by the networkX package, > however, as far as I have been able to understand both networkX and scipy > employ similar methods for finding the connected components. Thus I still > need the full sparse matrix (or graph which I think is similar in memory > usage) in memory, which I would like to avoid. > > Thank you for your reply and the reference. > I'm not sure what your application is, but if you just need connected components and have an easy way to find neighbors, then unionfind will partition the set for you. Although the common version doesn't make it easy to extract them, I have an implementation that keeps the connected nodes in a circular list for just that application. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Thu Aug 2 09:36:33 2012 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 2 Aug 2012 13:36:33 +0000 (UTC) Subject: [SciPy-User] ANN: PySAL 1.4 References: Message-ID: Hi, Serge Rey asu.edu> writes: > On behalf of the PySAL development team, I'm happy to announce the > official release of PySAL 1.4. Please consider listing your software here: http://scipy.org/Topical_Software http://scipy-central.org/ -- Pauli Virtanen From srey at asu.edu Thu Aug 2 13:16:32 2012 From: srey at asu.edu (Serge Rey) Date: Thu, 2 Aug 2012 10:16:32 -0700 Subject: [SciPy-User] ANN: PySAL 1.4 In-Reply-To: References: Message-ID: On Thu, Aug 2, 2012 at 6:36 AM, Pauli Virtanen wrote: > Hi, > > Serge Rey asu.edu> writes: >> On behalf of the PySAL development team, I'm happy to announce the >> official release of PySAL 1.4. > > Please consider listing your software here: > > http://scipy.org/Topical_Software i tried to but get the following error: [Errno 31] Too many links: '/home/scipy/wiki/data/pages/SergeRey(2f)MoinEditorBackup' anyone know of a fix? -- Sergio (Serge) Rey Professor, School of Geographical Sciences and Urban Planning GeoDa Center for Geospatial Analysis and Computation Arizona State University http://geoplan.asu.edu/rey Editor, International Regional Science Review http://irx.sagepub.com From fperez.net at gmail.com Thu Aug 2 19:10:40 2012 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 2 Aug 2012 16:10:40 -0700 Subject: [SciPy-User] [ANN] SIAM Conference on Computational Science & Engineering Submission Deadlines Approaching! In-Reply-To: References: Message-ID: Dear Colleagues, the SIAM CSE13 conference will be held next year in Boston, and this is a conference that is well suited for much of the type of work that goes on in the open source scientific Python development community (and Julia). The conference is co-chaired by Hans-Petter Langtangen, well known around these parts for his several books on scientific computing with Python and for having led a campus-wide adoption of Python as the core computational foundation across the University of Oslo. I am also on the program committee, as well as Randy LeVeque and other Python-friendly folks. An excellent way to participate is to organize a one- or two-part minisymposium on a specific topic with a group of related speakers (instructions at http://www.siam.org/meetings/cse13/submissions.php). Please note that the MS deadline is fast approaching: August 13, 2012. If you have any further questions, don't hesitate to contact me or one of the other organizers if you feel they can address your concerns more directly: "Fernando Perez" "Randy LeVeque" (Reproducible research track) "Hans Petter Langtangen" (Conference co-chair) "Karen Willcox" (conference chair) ---------- Forwarded message ---------- From: Karen Willcox Date: Tue, Jul 24, 2012 at 6:08 AM Subject: [SIAM-CSE] SIAM Conference on Computational Science & Engineering Submission Deadlines Approaching! To: SIAM-CSE at siam.org *SIAM Conference on Computational Science & Engineering (CSE13)* February 25-March 1, 2013 The Westin Boston Waterfront, Boston, Massachusetts, USA**** ** ** SUBMISSION DEADLINES ARE APPROACHING!**** August 13, 2012: Minisymposium proposals September 10, 2012: Abstracts for contributed and minisymposium speakers**** Visit http://www.siam.org/meetings/cse13/submissions.php to submit.**** ** ** Twitter hashtag: #SIAMcse13**** ** ** For more information about the conference, visit * http://www.siam.org/meetings/cse13/* or contact SIAM Conference Department at meetings at siam.org.**** -- Karen Willcox Professor and Associate Department Head Department of Aeronautics and Astronautics, MIT http://acdl.mit.edu/willcox.html _______________________________________________ SIAM-CSE mailing list To post messages to the list please send them to: SIAM-CSE at siam.org http://lists.siam.org/mailman/listinfo/siam-cse -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryanlists at gmail.com Fri Aug 3 13:02:42 2012 From: ryanlists at gmail.com (Ryan Krauss) Date: Fri, 3 Aug 2012 12:02:42 -0500 Subject: [SciPy-User] help speeding up a Runge-Kuta algorithm (cython, f2py, ...) Message-ID: I need help speeding up some code I wrote to perform a Runge-Kuta integration. I need to do the integration as part of a real-time control algorithm, so it needs to be fairly fast. scipy.integrate.odeint does too much error checking to be fast enough. My pure Python version was just a little too slow, so I tried coding it up in Cython. I have only used Cython once before, so I don't know if I did it correctly (the .pyx file is attached). The code runs just fine, but there is almost no speed up. I think the core issue is that my dxdt_runge_kuta function gets called about 4000 times per second, so most of my overhead is in the function calls (I think). I am running my real-time control algorithm at 500 Hz and I need at least 2 Runge-Kuta integration steps per real-time steps for numeric stability. And the Runge-Kuta algorithm needs to evaluate the derivative 4 times per times step. So, 500 Hz * 2 * 4 = 4000 calls per second. I also tried coding this up in fortran and using f2py, but I am getting a type mismatch error I don't understand. I have a function that declares its return values as double precision: double precision function dzdt(x,voltage) and I declare the variable I want to store the returned value in to also be double precision: double precision F,z,vel,accel,zdot1,zdot2,zdot3,zdot4 zdot1 = dzdt(x_prev,volts) but some how it is not happy. My C skills are pretty weak (the longer I use Python, the more C I forget, and I didn't know that much to start with). I started looking into Boost as well as using f2py on C code, but I got stuck. Can anyone either make my Cython or Fortran approaches work or point me in a different direction? Thanks, Ryan -------------- next part -------------- A non-text attachment was scrubbed... Name: runge_kuta.pyx Type: application/octet-stream Size: 2940 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: runge_kuta_f.f Type: application/octet-stream Size: 2058 bytes Desc: not available URL: From jim.vickroy at noaa.gov Fri Aug 3 13:15:13 2012 From: jim.vickroy at noaa.gov (Jim Vickroy) Date: Fri, 03 Aug 2012 11:15:13 -0600 Subject: [SciPy-User] help speeding up a Runge-Kuta algorithm (cython, f2py, ...) In-Reply-To: References: Message-ID: <501C0721.1090305@noaa.gov> On 8/3/2012 11:02 AM, Ryan Krauss wrote: > I need help speeding up some code I wrote to perform a Runge-Kuta > integration. I need to do the integration as part of a real-time > control algorithm, so it needs to be fairly fast. > scipy.integrate.odeint does too much error checking to be fast enough. > My pure Python version was just a little too slow, so I tried coding > it up in Cython. I have only used Cython once before, so I don't know > if I did it correctly (the .pyx file is attached). > > The code runs just fine, but there is almost no speed up. I think the > core issue is that my dxdt_runge_kuta function gets called about 4000 > times per second, so most of my overhead is in the function calls (I > think). I am running my real-time control algorithm at 500 Hz and I > need at least 2 Runge-Kuta integration steps per real-time steps for > numeric stability. And the Runge-Kuta algorithm needs to evaluate the > derivative 4 times per times step. So, 500 Hz * 2 * 4 = 4000 calls > per second. > > I also tried coding this up in fortran and using f2py, but I am > getting a type mismatch error I don't understand. I have a function > that declares its return values as double precision: > > double precision function dzdt(x,voltage) > > and I declare the variable I want to store the returned value in to > also be double precision: > > double precision F,z,vel,accel,zdot1,zdot2,zdot3,zdot4 > > zdot1 = dzdt(x_prev,volts) > > but some how it is not happy. I'm not much of a Fortran programmer and I may misunderstand the above, but have you tried adding dzdt to your double precision declaration? > > > My C skills are pretty weak (the longer I use Python, the more C I > forget, and I didn't know that much to start with). I started looking > into Boost as well as using f2py on C code, but I got stuck. > > > Can anyone either make my Cython or Fortran approaches work or point > me in a different direction? > > Thanks, > > Ryan > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From carmeldudley at gmail.com Fri Aug 3 13:24:44 2012 From: carmeldudley at gmail.com (carmel) Date: Fri, 3 Aug 2012 13:24:44 -0400 Subject: [SciPy-User] trouble installing scipy on os x 10.7.4 Message-ID: Hi, I'm on OS X 10.7.4 and am having trouble installing scipy. $ gcc --version: i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.9.00) $ gfortran --version: GNU Fortran (GCC) 4.2.3 I got gfortran from here: http://r.research.att.com/tools/ I already have numpy 1.6.1, am trying to install scipy-0.11.0rc1. The full output of $ python setup.py build is in this paste: http://pastebin.com/xs1fQ0fW All help welcome, I don't know anything about this stuff... I also downloaded and supposedly successfully installed the 10.7 specific add-on, from here: http://r.research.att.com/tools/gcc-42-5666.3-darwin11.pkg So I was surprised to see that the gcc version is the llvm-gcc-4.2 one that the installation guide warns about. Not sure how to change that or what I did wrong... Thanks, Carmel -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidmenhur at gmail.com Fri Aug 3 13:37:58 2012 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Fri, 3 Aug 2012 18:37:58 +0100 Subject: [SciPy-User] trouble installing scipy on os x 10.7.4 In-Reply-To: References: Message-ID: In my experience, the easiest fail-proof for installing everything is Macports. It will download and compile everything, and all dependences will be linked. On Fri, Aug 3, 2012 at 6:24 PM, carmel wrote: > Hi, I'm on OS X 10.7.4 and am having trouble installing scipy. > > $ gcc --version: > i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build > 5658) (LLVM build 2336.9.00) > > $ gfortran --version: > GNU Fortran (GCC) 4.2.3 > > I got gfortran from here: > http://r.research.att.com/tools/ > I already have numpy 1.6.1, am trying to install scipy-0.11.0rc1. > > The full output of $ python setup.py build is in this paste: > http://pastebin.com/xs1fQ0fW > > All help welcome, I don't know anything about this stuff... I also > downloaded and supposedly successfully installed the 10.7 specific add-on, > from here: > http://r.research.att.com/tools/gcc-42-5666.3-darwin11.pkg > So I was surprised to see that the gcc version is the llvm-gcc-4.2 one that > the installation guide warns about. Not sure how to change that or what I > did wrong... > > Thanks, > Carmel > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From pav at iki.fi Fri Aug 3 14:56:45 2012 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 03 Aug 2012 20:56:45 +0200 Subject: [SciPy-User] help speeding up a Runge-Kuta algorithm (cython, f2py, ...) In-Reply-To: References: Message-ID: 03.08.2012 19:02, Ryan Krauss kirjoitti: [clip] > Can anyone either make my Cython or Fortran approaches work or point > me in a different direction? Regarding Cython: run cython -a runge_kuta.pyx and check the created HTML file. Slow points are highlighted with yellow. Regarding this case: - `cdef`, not `def` for the dxdt_* function - from libc.math import exp - Do not use small numpy arrays inside loops. Use C constructs instead. - Use @cython.cdivision(True), @cython.boundscheck(False) PS. Runge-Kutta -- Pauli Virtanen From pav at iki.fi Fri Aug 3 15:05:38 2012 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 03 Aug 2012 21:05:38 +0200 Subject: [SciPy-User] help speeding up a Runge-Kuta algorithm (cython, f2py, ...) In-Reply-To: References: Message-ID: 03.08.2012 19:02, Ryan Krauss kirjoitti: [clip] > zdot1 = dzdt(x_prev,volts) > > but some how it is not happy. It's Fortran 77. You need to declare double precision dzdt I'd suggest writing Fortran 90 --- no need to bring more F77 code into existence ;) -- Pauli Virtanen From ryanlists at gmail.com Fri Aug 3 15:16:48 2012 From: ryanlists at gmail.com (Ryan Krauss) Date: Fri, 3 Aug 2012 14:16:48 -0500 Subject: [SciPy-User] help speeding up a Runge-Kuta algorithm (cython, f2py, ...) In-Reply-To: References: Message-ID: Thanks for the suggestions. > - Do not use small numpy arrays inside loops. > Use C constructs instead. This is where I ran into trouble with my knowledge of C. I have several 3x1 arrays that I need to pass into the dxdt function, multiply by scalars, and add together. I don't know how to do that cleanly in C. For example: x_out = x_prev + 1.0/6*(g1 + 2*g2 + 2*g3 + g4) where x_prev, g1, g2, g3, and g4 are all 3x1. A little googling lead me to valarray's, but I don't know if that is the best approach or how to use them within Cython. How would you do basic math on small arrays in pure C? On Fri, Aug 3, 2012 at 1:56 PM, Pauli Virtanen wrote: > 03.08.2012 19:02, Ryan Krauss kirjoitti: > [clip] >> Can anyone either make my Cython or Fortran approaches work or point >> me in a different direction? > > Regarding Cython: run > > cython -a runge_kuta.pyx > > and check the created HTML file. Slow points are highlighted with yellow. > > Regarding this case: > > - `cdef`, not `def` for the dxdt_* function > > - from libc.math import exp > > - Do not use small numpy arrays inside loops. > Use C constructs instead. > > - Use @cython.cdivision(True), @cython.boundscheck(False) > > > > PS. Runge-Kutta > > -- > Pauli Virtanen > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From ryanlists at gmail.com Fri Aug 3 15:41:00 2012 From: ryanlists at gmail.com (Ryan Krauss) Date: Fri, 3 Aug 2012 14:41:00 -0500 Subject: [SciPy-User] help speeding up a Runge-Kuta algorithm (cython, f2py, ...) In-Reply-To: References: Message-ID: Fortran, so fast, yet so painful. Once I got it working, it was 94 times faster than my pure Python version. Thanks to Jim and Pauli for helping me find my error. Ironically, I was thinking like a C programmer. Just because a Fortran function declares its return value data type doesn't mean all calling functions or subroutines will know the data of the function when they call it. I am still open to Cython suggestions. I don't want to bring more F77 code into the world..... On Fri, Aug 3, 2012 at 2:16 PM, Ryan Krauss wrote: > Thanks for the suggestions. > >> - Do not use small numpy arrays inside loops. >> Use C constructs instead. > > This is where I ran into trouble with my knowledge of C. I have > several 3x1 arrays that I need to pass into the dxdt function, > multiply by scalars, and add together. I don't know how to do that > cleanly in C. For example: > x_out = x_prev + 1.0/6*(g1 + 2*g2 + 2*g3 + g4) > where x_prev, g1, g2, g3, and g4 are all 3x1. > > A little googling lead me to valarray's, but I don't know if that is > the best approach or how to use them within Cython. > > How would you do basic math on small arrays in pure C? > > > > On Fri, Aug 3, 2012 at 1:56 PM, Pauli Virtanen wrote: >> 03.08.2012 19:02, Ryan Krauss kirjoitti: >> [clip] >>> Can anyone either make my Cython or Fortran approaches work or point >>> me in a different direction? >> >> Regarding Cython: run >> >> cython -a runge_kuta.pyx >> >> and check the created HTML file. Slow points are highlighted with yellow. >> >> Regarding this case: >> >> - `cdef`, not `def` for the dxdt_* function >> >> - from libc.math import exp >> >> - Do not use small numpy arrays inside loops. >> Use C constructs instead. >> >> - Use @cython.cdivision(True), @cython.boundscheck(False) >> >> >> >> PS. Runge-Kutta >> >> -- >> Pauli Virtanen >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user From sturla at molden.no Fri Aug 3 21:03:38 2012 From: sturla at molden.no (Sturla Molden) Date: Sat, 04 Aug 2012 03:03:38 +0200 Subject: [SciPy-User] help speeding up a Runge-Kuta algorithm (cython, f2py, ...) In-Reply-To: References: Message-ID: <501C74EA.7080103@molden.no> Den 03.08.2012 21:05, skrev Pauli Virtanen: > It's Fortran 77. You need to declare > > double precision dzdt > > I'd suggest writing Fortran 90 --- no need to bring more F77 code into > existence ;) > With the new typed memoryviews in Cython, there is no need to bring more Fortran of any sort into existance. ;-) Sturla From ralf.gommers at googlemail.com Sat Aug 4 04:50:08 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 4 Aug 2012 10:50:08 +0200 Subject: [SciPy-User] trouble installing scipy on os x 10.7.4 In-Reply-To: References: Message-ID: On Fri, Aug 3, 2012 at 7:24 PM, carmel wrote: > Hi, I'm on OS X 10.7.4 and am having trouble installing scipy. > > $ gcc --version: > i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build > 5658) (LLVM build 2336.9.00) > > $ gfortran --version: > GNU Fortran (GCC) 4.2.3 > > I got gfortran from here: > http://r.research.att.com/tools/ > I already have numpy 1.6.1, am trying to install scipy-0.11.0rc1. > > The full output of $ python setup.py build is in this paste: > http://pastebin.com/xs1fQ0fW > You're trying to use a Python version that doesn't support OS X 10.7 properly. You can see on the python.org download site that Python 2.7 has separate versions for 10.3-10.6 and 10.6+: - Python 2.7.3 Mac OS X 64-bit/32-bit x86-64/i386 Installer (for Mac OS X 10.6 and 10.7 [2]) - Python 2.7.3 Mac OS X 32-bit i386/PPC Installer (for Mac OS X 10.3 through 10.6 [2]) The python 2.6 you have tries to use gcc-4.0, which isn't available on OS X 10.7 anymore. > > All help welcome, I don't know anything about this stuff... I also > downloaded and supposedly successfully installed the 10.7 specific add-on, > from here: > http://r.research.att.com/tools/gcc-42-5666.3-darwin11.pkg > So I was surprised to see that the gcc version is the llvm-gcc-4.2 one > that the installation guide warns about. Not sure how to change that or > what I did wrong... > You have to look at how the binary is called. From http://r.research.att.com/tools/ it looks like it might be "gcc-4.2". If that's the case, make your build use it by: $ export gcc=/usr/bin/gcc-4.2 $ export g++=/usr/bin/g++-4.2 $ python setup.py install Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Sat Aug 4 05:07:40 2012 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Sat, 04 Aug 2012 11:07:40 +0200 Subject: [SciPy-User] help speeding up a Runge-Kuta algorithm (cython, f2py, ...) In-Reply-To: References: Message-ID: <1344071260.26109.14.camel@sebastian-laptop> Hey, Just to add what was said previously, isn't float in Cython single precision? I doubt this was intended here and should be replaced with DTYPE_t everywhere. Other then that it was already said, np.zeros/np.exp is bad there... Regards, Sebastian On Fr, 2012-08-03 at 14:41 -0500, Ryan Krauss wrote: > Fortran, so fast, yet so painful. Once I got it working, it was 94 > times faster than my pure Python version. > > Thanks to Jim and Pauli for helping me find my error. Ironically, I > was thinking like a C programmer. Just because a Fortran function > declares its return value data type doesn't mean all calling functions > or subroutines will know the data of the function when they call it. > > I am still open to Cython suggestions. I don't want to bring more F77 > code into the world..... > > On Fri, Aug 3, 2012 at 2:16 PM, Ryan Krauss wrote: > > Thanks for the suggestions. > > > >> - Do not use small numpy arrays inside loops. > >> Use C constructs instead. > > > > This is where I ran into trouble with my knowledge of C. I have > > several 3x1 arrays that I need to pass into the dxdt function, > > multiply by scalars, and add together. I don't know how to do that > > cleanly in C. For example: > > x_out = x_prev + 1.0/6*(g1 + 2*g2 + 2*g3 + g4) > > where x_prev, g1, g2, g3, and g4 are all 3x1. > > > > A little googling lead me to valarray's, but I don't know if that is > > the best approach or how to use them within Cython. > > > > How would you do basic math on small arrays in pure C? > > > > > > > > On Fri, Aug 3, 2012 at 1:56 PM, Pauli Virtanen wrote: > >> 03.08.2012 19:02, Ryan Krauss kirjoitti: > >> [clip] > >>> Can anyone either make my Cython or Fortran approaches work or point > >>> me in a different direction? > >> > >> Regarding Cython: run > >> > >> cython -a runge_kuta.pyx > >> > >> and check the created HTML file. Slow points are highlighted with yellow. > >> > >> Regarding this case: > >> > >> - `cdef`, not `def` for the dxdt_* function > >> > >> - from libc.math import exp > >> > >> - Do not use small numpy arrays inside loops. > >> Use C constructs instead. > >> > >> - Use @cython.cdivision(True), @cython.boundscheck(False) > >> > >> > >> > >> PS. Runge-Kutta > >> > >> -- > >> Pauli Virtanen > >> > >> _______________________________________________ > >> SciPy-User mailing list > >> SciPy-User at scipy.org > >> http://mail.scipy.org/mailman/listinfo/scipy-user > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From x.piter at gmail.com Sat Aug 4 11:48:23 2012 From: x.piter at gmail.com (x.piter at gmail.com) Date: Sat, 04 Aug 2012 17:48:23 +0200 Subject: [SciPy-User] loadmat and figure Message-ID: <87lihulku0.fsf@cica.cica> Hi all. Is it possible to load a malab figure using io.loadmat. It works well with a matrix saved as mat file, but if I try to load a fig file I get the following output: scipy.io.loadmat('test2.mat') {'a': 'Read error: Cannot read matlab functions', '__version__': '1.0', '__header__': 'MATLAB 5.0 MAT-file, Platform: GLNX86, Created on: Sat Aug 4 16:56:09 2012', '__globals__': []} >>> I use scipy shipped with the Debian stable repositories. Thanks. Petro. From silva at lma.cnrs-mrs.fr Sat Aug 4 12:13:30 2012 From: silva at lma.cnrs-mrs.fr (Fabrice Silva) Date: Sat, 04 Aug 2012 18:13:30 +0200 Subject: [SciPy-User] loadmat and figure In-Reply-To: <87lihulku0.fsf@cica.cica> References: <87lihulku0.fsf@cica.cica> Message-ID: <1344096810.8004.13.camel@amilo.coursju> Le samedi 04 ao?t 2012 ? 17:48 +0200, x.piter at gmail.com a ?crit : > Hi all. > Is it possible to load a malab figure using io.loadmat. > It works well with a matrix saved as mat file, but if I try to load a > fig file I get the following output: > scipy.io.loadmat('test2.mat') > {'a': 'Read error: Cannot read matlab functions', '__version__': '1.0', '__header__': 'MATLAB 5.0 MAT-file, Platform: GLNX86, Created on: Sat Aug 4 16:56:09 2012', '__globals__': []} > >>> > I use scipy shipped with the Debian stable repositories. > Thanks. Are not matlab figure supposed to be .fig files ? The 'a' items of the resulting dictionary seems weird. I can load (with loadmat, Debian unstable) matlab figure files, but the trouble is in converting the whole tree stored in the file into some matplotlib (or similar) instances... -- Fabrice Silva From jjstickel at gmail.com Sat Aug 4 15:35:17 2012 From: jjstickel at gmail.com (Jonathan Stickel) Date: Sat, 04 Aug 2012 13:35:17 -0600 Subject: [SciPy-User] help speeding up a Runge-Kuta algorithm (cython, f2py, ...) In-Reply-To: References: Message-ID: <501D7975.80306@gmail.com> I am sure properly coded Cython is great, but I really struggled when I tried to use it. I found that it allows you to write really slow code without errors or warnings. I found the profiling tools to be only marginally helpful. So many different ways to do the same thing... which is the best? All the documentation is nice, but very long and dense. I am having much more success with f2py (using F90 syntax). Either your code runs fast, or it simply will not compile (excepting segfault bugs that can sometimes be difficult to track down). Improved and updated documentation would be helpful, but otherwise f2py is now what I turn to when speed is crucial. My 2 cents. YMMV. Jonathan On 08/04/2012 02:45 AM, scipy-user-request at scipy.org wrote: > Date: Sat, 04 Aug 2012 03:03:38 +0200 > From: Sturla Molden > Subject: Re: [SciPy-User] help speeding up a Runge-Kuta algorithm > (cython, f2py, ...) > > Den 03.08.2012 21:05, skrev Pauli Virtanen: >> >It's Fortran 77. You need to declare >> > >> > double precision dzdt >> > >> >I'd suggest writing Fortran 90 --- no need to bring more F77 code into >> >existence;) >> > > With the new typed memoryviews in Cython, there is no need to bring more > Fortran of any sort into existance.;-) From sturla at molden.no Sat Aug 4 17:49:29 2012 From: sturla at molden.no (Sturla Molden) Date: Sat, 04 Aug 2012 23:49:29 +0200 Subject: [SciPy-User] help speeding up a Runge-Kuta algorithm (cython, f2py, ...) In-Reply-To: <501D7975.80306@gmail.com> References: <501D7975.80306@gmail.com> Message-ID: <501D98E9.3060400@molden.no> Den 04.08.2012 21:35, skrev Jonathan Stickel: > I am sure properly coded Cython is great, but I really struggled when I > tried to use it. I uses np.ndarray declarations and array expressions. Those will be slow. To write fast and numpyonic Cython, use typed memoryviews instead, and write out all loops. I.e. there is no support for array expressions with these yet. And unfortunately there is a huge lack of documentation on how to use typed memoryviews efficiently. Here is an example code of how to use Cython as a Fortran killer: https://github.com/sturlamolden/memview_benchmarks/blob/master/memview.pyx In this case, the performance with -O2 was just 2.2% slower than "plain C" with pointer arithmetics. It is possible to write very fast array code with Cython, but you must do it right. For comparison, this is very slow: https://github.com/sturlamolden/memview_benchmarks/blob/master/cythonized_numpy_2b.pyx What this mean is this: For anything but trivial code, the NumPy syntax is just too slow and should be avoided! I breiefly looked at the Cython code posted in this thread, and it suffers form all these issues. Sturla From sturla at molden.no Sat Aug 4 19:28:48 2012 From: sturla at molden.no (Sturla Molden) Date: Sun, 05 Aug 2012 01:28:48 +0200 Subject: [SciPy-User] help speeding up a Runge-Kuta algorithm (cython, f2py, ...) In-Reply-To: References: Message-ID: <501DB030.6080108@molden.no> Not tested and debugged, but to me it looks like something like this might be what you want. Sturla Den 03.08.2012 19:02, skrev Ryan Krauss: > I need help speeding up some code I wrote to perform a Runge-Kuta > integration. I need to do the integration as part of a real-time > control algorithm, so it needs to be fairly fast. > scipy.integrate.odeint does too much error checking to be fast enough. > My pure Python version was just a little too slow, so I tried coding > it up in Cython. I have only used Cython once before, so I don't know > if I did it correctly (the .pyx file is attached). > > The code runs just fine, but there is almost no speed up. I think the > core issue is that my dxdt_runge_kuta function gets called about 4000 > times per second, so most of my overhead is in the function calls (I > think). I am running my real-time control algorithm at 500 Hz and I > need at least 2 Runge-Kuta integration steps per real-time steps for > numeric stability. And the Runge-Kuta algorithm needs to evaluate the > derivative 4 times per times step. So, 500 Hz * 2 * 4 = 4000 calls > per second. > > I also tried coding this up in fortran and using f2py, but I am > getting a type mismatch error I don't understand. I have a function > that declares its return values as double precision: > > double precision function dzdt(x,voltage) > > and I declare the variable I want to store the returned value in to > also be double precision: > > double precision F,z,vel,accel,zdot1,zdot2,zdot3,zdot4 > > zdot1 = dzdt(x_prev,volts) > > but some how it is not happy. > > > My C skills are pretty weak (the longer I use Python, the more C I > forget, and I didn't know that much to start with). I started looking > into Boost as well as using f2py on C code, but I got stuck. > > > Can anyone either make my Cython or Fortran approaches work or point > me in a different direction? > > Thanks, > > Ryan > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- #cython: boundscheck=False #cython: wraparound=False import numpy as np from libc.math cimport exp, fabs cdef inline void dxdt_runge_kuta(double *x "const double *", double voltage "const double", double *dxdt): cdef double J = 0.0011767297528720126 "const double" # these are all constants, using the "const trick" cdef double alpha0 = 4.1396263800000002 "const double" cdef double alpha1 = 8.7203167700000002 "const double" cdef double alpha2 = 0.010979707700000001 "const double" cdef double sigma0 = 2.391621670693391 "const double" cdef double sigma1 = 0.005368467489711875 "const double" cdef double v_gain = 1.3319218899042409 "const double" cdef double v_stribeck = 655.33634300000006 "const double" cdef double vel = x[1] cdef double z = x[2] cdef double g = alpha0 + alpha1*exp(-1*(vel/v_stribeck)**2) cdef double dzdt = vel - sigma0*z*fabs(vel)/g cdef double F = sigma0*z + sigma1*dzdt + alpha2*vel cdef double accel = 1.0/J*(voltage*v_gain - F) dxdt[0] = vel dxdt[1] = accel dxdt[2] = dzdt def runge_kuta_one_step(double _x[::1], Py_ssize_t factor, double volts, double dt_rk): cdef double g1[4], g2[4], g3[4], g4[4], dxdt[4], tmpx[4] # padding with one element for better alignment cdef double x[4] cdef double out[::1] = np.zeros(3) cdef Py_ssize_t q cdef int i cdef double *x_prev, *x_out, *swap x_out = &out[0] x_prev = &x[0] x[0] = _x[0] x[1] = _x[1] x[2] = _x[2] for q in range(factor): dxdt_runge_kuta(x_prev, volts, dxdt) for i in range(3): g1[i] = dt_rk * dxdt[i] tmpx[i] = x_prev[i] + 0.5*g1[i] dxdt_runge_kuta(tmpx, volts, dxdt) for i in range(3): g2[i] = dt_rk * dxdt[i] tmpx[i] = x_prev[i] + 0.5*g2[i] dxdt_runge_kuta(tmpx, volts, dxdt) for i in range(3): g3[i] = dt_rk * dxdt[i] tmpx[i] = x_prev[i] + g3[i] dxdt_runge_kuta(tmpx, volts, dxdt) for i in range(3): g4[i] = dt_rk * dxdt[i] for i in range(3): x_out[i] = x_prev[i] + 1.0/6*(g1[i] + 2*g2[i] + 2*g3[i] + g4[i]) swap = x_prev x_prev = x_out x_out = swap for i in range(3): out[i] = x_prev[i] return out From gael.varoquaux at normalesup.org Sun Aug 5 04:44:17 2012 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 5 Aug 2012 10:44:17 +0200 Subject: [SciPy-User] help speeding up a Runge-Kuta algorithm (cython, f2py, ...) In-Reply-To: <501D7975.80306@gmail.com> References: <501D7975.80306@gmail.com> Message-ID: <20120805084417.GA19254@phare.normalesup.org> On Sat, Aug 04, 2012 at 01:35:17PM -0600, Jonathan Stickel wrote: > I am sure properly coded Cython is great, but I really struggled when I > tried to use it. I found that it allows you to write really slow code > without errors or warnings. I found the profiling tools to be only > marginally helpful. To write fast cython code, compile it with 'cython -a', open the resulting html file in a web browser. The yellow lines are where the problems are: click on them and you'll find that they correspond to lines of Cython code that lead to long and complex C code. Improve your code (by making sure that it relies on typed variables and fast array access) until it has not yellow lines. HTH, Gael From x.piter at gmail.com Sun Aug 5 05:50:36 2012 From: x.piter at gmail.com (x.piter at gmail.com) Date: Sun, 05 Aug 2012 11:50:36 +0200 Subject: [SciPy-User] loadmat and figure References: <87lihulku0.fsf@cica.cica> <1344096810.8004.13.camel@amilo.coursju> Message-ID: <87wr1dfz0z.fsf@cica.cica> Hi Fabrice Silva writes: > Are not matlab figure supposed to be .fig files ? > The 'a' items of the resulting dictionary seems weird. I was not sure if fig files are compartiple with loadmat, so I had loaded figure in matlab with -mat key and saved the structure a.mat > I can load (with loadmat, Debian unstable) matlab figure files, but the > trouble is in converting the whole tree stored in the file into some > matplotlib (or similar) instances... This is another trouble but I will think about it when I reach this stage (I need only numbers). May be I need the latest scipy. Regards Petro From silva at lma.cnrs-mrs.fr Sun Aug 5 06:29:31 2012 From: silva at lma.cnrs-mrs.fr (Fabrice Silva) Date: Sun, 05 Aug 2012 12:29:31 +0200 Subject: [SciPy-User] loadmat and figure In-Reply-To: <87wr1dfz0z.fsf@cica.cica> References: <87lihulku0.fsf@cica.cica> <1344096810.8004.13.camel@amilo.coursju> <87wr1dfz0z.fsf@cica.cica> Message-ID: <1344162571.16423.9.camel@amilo.coursju> Le dimanche 05 ao?t 2012 ? 11:50 +0200, x.piter at gmail.com a ?crit : > Hi > Fabrice Silva writes: > > Are not matlab figure supposed to be .fig files ? > > The 'a' items of the resulting dictionary seems weird. > I was not sure if fig files are compartiple with loadmat, so I had > loaded figure in matlab with -mat key and saved the structure a.mat Debian unstable ships scipy 0.10 which succeed in loading .fig files. Example : In [1]: import scipy; scipy.__version__ Out[1]: '0.10.1' In [2]: import scipy.io as io In [3]: d = io.loadmat('AR.fig', squeeze_me=True, struct_as_record=False); d.keys() Out[3]: ['hgS_070000', '__version__', '__header__', '__globals__'] Note the hgS_* magic key indicating matlab v7 figure type In [4]: print d['__header__'] MATLAB 5.0 MAT-file, Platform: GLNXA64, Created on: Fri Jul 6 13:43:19 2012 In [5]: print d['hgS_070000'].type figure You then need to introspect through the structured hierarchy to extract your data In [6]: print d['hgS_070000'].children.type axes In [7]: print d['hgS_070000'].children.children.shape (43,) In [8]: print d['hgS_070000'].children.children[0].type graph2d.lineseries In [9]: dir(d['hgS_070000'].children.children[0].properties) Out[9]: ['ApplicationData', 'CodeGenColorMode', 'CodeGenLineStyleMode', 'CodeGenMarkerMode', 'Color', 'LineWidth', 'ObeyXDataMode', 'OldSwitchProps', 'OldSwitchVals', 'SwitchProps', 'XData', 'XDataJitter', 'XDataMode', 'XDataSource', 'YData', 'YDataSource', 'ZDataSource', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__slotnames__', '__str__', '__subclasshook__', '__weakref__', '_fieldnames'] Getting X and Y arrays for the first line of the first axes In [10]: X = d['hgS_070000'].children.children[0].properties.XData In [11]: Y = d['hgS_070000'].children.children[0].properties.YData In [12]: import matplotlib.pyplot as plt In [13]: plt.plot(X,Y) Out[13]: [] > This is another trouble but I will think about it when I reach this > stage (I need only numbers). May be I need the latest scipy. Have you seen the note at the end of this page, concerning v7.3 format not handled by scipy 0.7 (also neither by recent versions) ? http://docs.scipy.org/doc/scipy-0.7.x/reference/generated/scipy.io.loadmat.html#scipy.io.loadmat -- Fabrice Silva LMA UPR CNRS 7051 From x.piter at gmail.com Sun Aug 5 07:14:20 2012 From: x.piter at gmail.com (x.piter at gmail.com) Date: Sun, 05 Aug 2012 13:14:20 +0200 Subject: [SciPy-User] scipy installation error Message-ID: <87obmpfv5f.fsf@cica.cica> Hi I try to install scpy 10.1 (or 11) on debian testing, but get the following error: error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -I/usr/lib/pymodules/python2.6/numpy/core/include -I/usr/lib/pymodules/python2.6/numpy/core/include -c scipy/spatial/qhull/src/poly.c -o build/temp.linux-i686-2.6/scipy/spatial/qhull/src/poly.o" failed with exit status 1 Does anybody knows a solution for it? Thanks. Petro From cournape at gmail.com Sun Aug 5 07:40:12 2012 From: cournape at gmail.com (David Cournapeau) Date: Sun, 5 Aug 2012 12:40:12 +0100 Subject: [SciPy-User] scipy installation error In-Reply-To: <87obmpfv5f.fsf@cica.cica> References: <87obmpfv5f.fsf@cica.cica> Message-ID: Hi Petro, On Sun, Aug 5, 2012 at 12:14 PM, wrote: > Hi > I try to install scpy 10.1 (or 11) on debian testing, but get the > following error: > error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -I/usr/lib/pymodules/python2.6/numpy/core/include -I/usr/lib/pymodules/python2.6/numpy/core/include -c scipy/spatial/qhull/src/poly.c -o build/temp.linux-i686-2.6/scipy/spatial/qhull/src/poly.o" failed with exit status 1 > Does anybody knows a solution for it? We would need a bit more context to help you there. Can you give the rest of the log ? David From pav at iki.fi Sun Aug 5 07:47:34 2012 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 05 Aug 2012 13:47:34 +0200 Subject: [SciPy-User] scipy installation error In-Reply-To: <87obmpfv5f.fsf@cica.cica> References: <87obmpfv5f.fsf@cica.cica> Message-ID: 05.08.2012 13:14, x.piter at gmail.com kirjoitti: > Hi > I try to install scpy 10.1 (or 11) on debian testing, but get the > following error: > error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -I/usr/lib/pymodules/python2.6/numpy/core/include -I/usr/lib/pymodules/python2.6/numpy/core/include -c scipy/spatial/qhull/src/poly.c -o build/temp.linux-i686-2.6/scipy/spatial/qhull/src/poly.o" failed with exit status 1 > Does anybody knows a solution for it? [clip] Paste the whole build log on gist.github.com or pastebin. It's impossible to say much based on this snippet. -- Pauli Virtanen From x.piter at gmail.com Sun Aug 5 12:15:13 2012 From: x.piter at gmail.com (x.piter at gmail.com) Date: Sun, 05 Aug 2012 18:15:13 +0200 Subject: [SciPy-User] scipy installation error References: <87obmpfv5f.fsf@cica.cica> Message-ID: <878vdtpb72.fsf@cica.cica> David Cournapeau writes: > We would need a bit more context to help you there. Can you give the > rest of the log ? Hi, thanks for these followups. Here is the end part were troubles start I am an illiterate in C :( ########################################## compiling C sources C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC compile options: '-I/usr/include/python2.6 -I/usr/lib/pymodules/python2.6/numpy/core/include -I/usr/lib/pymodules/python2.6/numpy/core/include -c' gcc: scipy/spatial/qhull/src/poly.c In file included from /usr/include/python2.6/Python.h:8, from scipy/spatial/qhull/src/mem.h:22, from scipy/spatial/qhull/src/qhull_a.h:31, from scipy/spatial/qhull/src/poly.c:17: /usr/include/python2.6/pyconfig.h:1031:1: warning: "_POSIX_C_SOURCE" redefined In file included from /usr/include/time.h:28, from scipy/spatial/qhull/src/user.h:27, from scipy/spatial/qhull/src/libqhull.h:34, from scipy/spatial/qhull/src/qhull_a.h:27, from scipy/spatial/qhull/src/poly.c:17: /usr/include/features.h:210:1: warning: this is the location of the previous definition In file included from scipy/spatial/qhull/src/qhull_a.h:31, from scipy/spatial/qhull/src/poly.c:17: scipy/spatial/qhull/src/mem.h:23:32: error: numpy/ndarraytypes.h: No such file or directory In file included from scipy/spatial/qhull/src/qhull_a.h:31, from scipy/spatial/qhull/src/poly.c:17: scipy/spatial/qhull/src/mem.h:85: error: expected ?=?, ?,?, ?;?, ?asm? or ?__attribute__? before ?ptr_intT? scipy/spatial/qhull/src/poly.c: In function ?qh_gethash?: scipy/spatial/qhull/src/poly.c:407: error: ?ptr_intT? undeclared (first use in this function) scipy/spatial/qhull/src/poly.c:407: error: (Each undeclared identifier is reported only once scipy/spatial/qhull/src/poly.c:407: error: for each function it appears in.) scipy/spatial/qhull/src/poly.c:407: error: expected ?;? before ?hash? scipy/spatial/qhull/src/poly.c:417: error: ?hash? undeclared (first use in this function) scipy/spatial/qhull/src/poly.c:417: error: expected ?;? before ?skipelem? scipy/spatial/qhull/src/poly.c:420: error: expected ?;? before ?elemp? scipy/spatial/qhull/src/poly.c:423: error: expected ?;? before ?elemp? scipy/spatial/qhull/src/poly.c:427: error: expected ?;? before ?elemp? scipy/spatial/qhull/src/poly.c:431: error: expected ?;? before ?elemp? scipy/spatial/qhull/src/poly.c:435: error: expected ?;? before ?elemp? scipy/spatial/qhull/src/poly.c:443: error: ?elem? undeclared (first use in this function) scipy/spatial/qhull/src/poly.c:443: error: expected ?)? before ?skipelem? scipy/spatial/qhull/src/poly.c: In function ?qh_pointid?: scipy/spatial/qhull/src/poly.c:1047: error: ?ptr_intT? undeclared (first use in this function) scipy/spatial/qhull/src/poly.c:1047: error: expected ?;? before ?offset? scipy/spatial/qhull/src/poly.c:1055: error: ?offset? undeclared (first use in this function) scipy/spatial/qhull/src/poly.c:1056: error: ?id? undeclared (first use in this function) In file included from /usr/include/python2.6/Python.h:8, from scipy/spatial/qhull/src/mem.h:22, from scipy/spatial/qhull/src/qhull_a.h:31, from scipy/spatial/qhull/src/poly.c:17: /usr/include/python2.6/pyconfig.h:1031:1: warning: "_POSIX_C_SOURCE" redefined In file included from /usr/include/time.h:28, from scipy/spatial/qhull/src/user.h:27, from scipy/spatial/qhull/src/libqhull.h:34, from scipy/spatial/qhull/src/qhull_a.h:27, from scipy/spatial/qhull/src/poly.c:17: /usr/include/features.h:210:1: warning: this is the location of the previous definition In file included from scipy/spatial/qhull/src/qhull_a.h:31, from scipy/spatial/qhull/src/poly.c:17: scipy/spatial/qhull/src/mem.h:23:32: error: numpy/ndarraytypes.h: No such file or directory In file included from scipy/spatial/qhull/src/qhull_a.h:31, from scipy/spatial/qhull/src/poly.c:17: scipy/spatial/qhull/src/mem.h:85: error: expected ?=?, ?,?, ?;?, ?asm? or ?__attribute__? before ?ptr_intT? scipy/spatial/qhull/src/poly.c: In function ?qh_gethash?: scipy/spatial/qhull/src/poly.c:407: error: ?ptr_intT? undeclared (first use in this function) scipy/spatial/qhull/src/poly.c:407: error: (Each undeclared identifier is reported only once scipy/spatial/qhull/src/poly.c:407: error: for each function it appears in.) scipy/spatial/qhull/src/poly.c:407: error: expected ?;? before ?hash? scipy/spatial/qhull/src/poly.c:417: error: ?hash? undeclared (first use in this function) scipy/spatial/qhull/src/poly.c:417: error: expected ?;? before ?skipelem? scipy/spatial/qhull/src/poly.c:420: error: expected ?;? before ?elemp? scipy/spatial/qhull/src/poly.c:423: error: expected ?;? before ?elemp? scipy/spatial/qhull/src/poly.c:427: error: expected ?;? before ?elemp? scipy/spatial/qhull/src/poly.c:431: error: expected ?;? before ?elemp? scipy/spatial/qhull/src/poly.c:435: error: expected ?;? before ?elemp? scipy/spatial/qhull/src/poly.c:443: error: ?elem? undeclared (first use in this function) scipy/spatial/qhull/src/poly.c:443: error: expected ?)? before ?skipelem? scipy/spatial/qhull/src/poly.c: In function ?qh_pointid?: scipy/spatial/qhull/src/poly.c:1047: error: ?ptr_intT? undeclared (first use in this function) scipy/spatial/qhull/src/poly.c:1047: error: expected ?;? before ?offset? scipy/spatial/qhull/src/poly.c:1055: error: ?offset? undeclared (first use in this function) scipy/spatial/qhull/src/poly.c:1056: error: ?id? undeclared (first use in this function) error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -I/usr/lib/pymodules/python2.6/numpy/core/include -I/usr/lib/pymodules/python2.6/numpy/core/include -c scipy/spatial/qhull/src/poly.c -o build/temp.linux-i686-2.6/scipy/spatial/qhull/src/poly.o" failed with exit status 1 From x.piter at gmail.com Sun Aug 5 12:52:43 2012 From: x.piter at gmail.com (Petro) Date: Sun, 05 Aug 2012 18:52:43 +0200 Subject: [SciPy-User] loadmat and figure References: <87lihulku0.fsf@cica.cica> <1344096810.8004.13.camel@amilo.coursju> <87wr1dfz0z.fsf@cica.cica> <1344162571.16423.9.camel@amilo.coursju> Message-ID: <871ujlp9gk.fsf@cica.cica> Fabrice Silva writes: > Le dimanche 05 ao?t 2012 ? 11:50 +0200, x.piter at gmail.com a ?crit : > > Have you seen the note at the end of this page, concerning v7.3 format > not handled by scipy 0.7 (also neither by recent versions) ? > http://docs.scipy.org/doc/scipy-0.7.x/reference/generated/scipy.io.loadmat.html#scipy.io.loadmat Yes, I have seen it and tried to load different version of figures, none of them worked. Now I try to install scipy 10.1 instead of default 7th version, so far without success. From silva at lma.cnrs-mrs.fr Sun Aug 5 14:31:29 2012 From: silva at lma.cnrs-mrs.fr (Fabrice Silva) Date: Sun, 05 Aug 2012 20:31:29 +0200 Subject: [SciPy-User] loadmat and figure In-Reply-To: <871ujlp9gk.fsf@cica.cica> References: <87lihulku0.fsf@cica.cica> <1344096810.8004.13.camel@amilo.coursju> <87wr1dfz0z.fsf@cica.cica> <1344162571.16423.9.camel@amilo.coursju> <871ujlp9gk.fsf@cica.cica> Message-ID: <1344191489.16423.20.camel@amilo.coursju> Le dimanche 05 ao?t 2012 ? 18:52 +0200, Petro a ?crit : > Fabrice Silva writes: > > > Le dimanche 05 ao?t 2012 ? 11:50 +0200, x.piter at gmail.com a ?crit : > > > > Have you seen the note at the end of this page, concerning v7.3 format > > not handled by scipy 0.7 (also neither by recent versions) ? > > http://docs.scipy.org/doc/scipy-0.7.x/reference/generated/scipy.io.loadmat.html#scipy.io.loadmat > Yes, I have seen it and tried to load different version of figures, none > of them worked. Now I try to install scipy 10.1 instead of default 7th > version, so far without success. Could you post a complete minimal example (with the file, and traceback) ? -- Fabrice Silva From sergio_r at mail.com Sun Aug 5 14:33:43 2012 From: sergio_r at mail.com (Sergio Rojas) Date: Sun, 05 Aug 2012 14:33:43 -0400 Subject: [SciPy-User] help speeding up a Runge-Kuta algorithm (cython, f2py, ...) (Sturla Molden) Message-ID: <20120805183344.4280@gmx.com> ************************************************************ Here is an example code of how to use Cython as a Fortran killer: https://github.com/sturlamolden/memview_benchmarks/blob/master/memview.pyx In this case, the performance with -O2 was just 2.2% slower than "plain C" with pointer arithmetics. It is possible to write very fast array code with Cython, but you must do it right. For comparison, this is very slow: https://github.com/sturlamolden/memview_benchmarks/blob/master/cythonized_numpy_2b.pyx What this mean is this: For anything but trivial code, the NumPy syntax is just too slow and should be avoided! I breiefly looked at the Cython code posted in this thread, and it suffers form all these issues. Sturla ********************************************** Hello Sturla, I am wondering whether there is a problem with my phyton installation or something is missing in your "memview.pyx" file as the following compiling error arises: cythoning memview.pyx to memview.c Error compiling Cython file: ------------------------------------------------------------ ... cimport numpy as np cimport cython @cython.boundscheck(False) @cython.wraparound(False) cdef np.float64_t slice_func(np.float64_t[::1] Xi, ^ ------------------------------------------------------------ memview.pyx:7:42: Expected an identifier or literal Sergio PD. Compilation works fine in the other files (cythonized_numpy_2b.pyx cythonized_numpy_4.pyx pointer_arith.pyx cythonized_numpy_2.pyx cythonized_numpy.pyx cythonized_numpy_3.pyx). Taking out that module in the "runme.py" file the output is: >$ python runme.py numpy_only: 3.13 sec cythonized_numpy: 2.38 sec cythonized_numpy_2: 3.8 sec cythonized_numpy_2b: 3.27 sec cythonized_numpy_3: 1.52 sec cythonized_numpy_4: 1.19 sec pointer_arith: 1.19 sec -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Sun Aug 5 16:05:44 2012 From: sturla at molden.no (Sturla Molden) Date: Sun, 05 Aug 2012 22:05:44 +0200 Subject: [SciPy-User] help speeding up a Runge-Kuta algorithm (cython, f2py, ...) (Sturla Molden) In-Reply-To: <20120805183344.4280@gmx.com> References: <20120805183344.4280@gmx.com> Message-ID: <501ED218.5050403@molden.no> Which version of Cython do you have (use at least 0.16). Sturla Den 05.08.2012 20:33, skrev Sergio Rojas: > Sturla ********************************************** Hello Sturla, I > am wondering whether there is a problem with my phyton installation or > something is missing in your "memview.pyx" file as the following > compiling error arises: cythoning memview.pyx to memview.c Error > compiling Cython file: > ------------------------------------------------------------ ... > cimport numpy as np cimport cython @cython.boundscheck(False) > @cython.wraparound(False) cdef np.float64_t > slice_func(np.float64_t[::1] Xi, ^ > ------------------------------------------------------------ > memview.pyx:7:42: Expected an identifier or literal Sergio PD. > Compilation works fine in the other files (cythonized_numpy_2b.pyx > cythonized_numpy_4.pyx pointer_arith.pyx cythonized_numpy_2.pyx > cythonized_numpy.pyx cythonized_numpy_3.pyx). Taking out that module > in the "runme.py" file the output is: >$ python runme.py numpy_only: > 3.13 sec cythonized_numpy: 2.38 sec cythonized_numpy_2: 3.8 sec > cythonized_numpy_2b: 3.27 sec cythonized_numpy_3: 1.52 sec > cythonized_numpy_4: 1.19 sec pointer_arith: 1.19 sec > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From j at abel.co Sun Aug 5 19:09:06 2012 From: j at abel.co (James Abel) Date: Sun, 5 Aug 2012 16:09:06 -0700 Subject: [SciPy-User] kmeans2 question/issue Message-ID: <000c01cd735f$5082d010$f1887030$@co> Hi, I'm trying to use scipy.cluster.vq.kmeans2() but I'm getting inconsistent output. With a simple test input that should have 3 clusters, I'm getting good results most of the time but every so often the output creates the wrong clustering. If anyone could point to what I'm doing wrong I'd appreciate it! Code and sample output below. Thanks! James Code: import sys import scipy from scipy.cluster.vq import * print sys.version vals = scipy.array((0.0,0.1,0.5,0.6,1.0,1.1)) print vals white_vals = whiten(vals) print white_vals.shape, white_vals # try it several times to see if we get similar answers count = 0 while count < 5: res, idx = kmeans2(white_vals, 3) # changing iter doesn't seem to matter print res, idx count += 1 Output: 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] [ 0. 0.1 0.5 0.6 1. 1.1] (6,) [ 0. 0.24313227 1.21566135 1.45879362 2.4313227 2.67445496] [ 0.12156613 2.55288883 1.33722748] [0 0 2 2 1 1] [ 0.12156613 2.55288883 1.33722748] [0 0 2 2 1 1] [ 1.33722748 2.55288883 0.12156613] [2 2 0 0 1 1] [ 2.18819043 0.48626454 -0.97292963] [1 1 1 0 0 0] <-- unexpected result [ 0.12156613 2.55288883 1.33722748] [0 0 2 2 1 1] C:\PYTHON27\lib\site-packages\scipy\cluster\vq.py:588: UserWarning: One of the clusters is empty. Re-run kmean with a different initialization. warnings.warn("One of the clusters is empty. " From helmrp at yahoo.com Sun Aug 5 23:07:12 2012 From: helmrp at yahoo.com (The Helmbolds) Date: Sun, 5 Aug 2012 20:07:12 -0700 (PDT) Subject: [SciPy-User] Brentq return values Message-ID: <1344222432.75833.YahooMailNeo@web31808.mail.mud.yahoo.com> When the brentq routine is called with `full_output` set to `True` it returns a tuple. The first element of that tuple is the root, i.e., a simple real number. So far, so good. ? However, the second element of that tuple is: ??? I don't know how to read or "decode" such a thing. I'd welcome any instruction on how to do that. ? I also observe that perhaps the return value could be more transparent, i.e., easy to interpret. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Mon Aug 6 01:30:48 2012 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 6 Aug 2012 07:30:48 +0200 Subject: [SciPy-User] kmeans2 question/issue In-Reply-To: <000c01cd735f$5082d010$f1887030$@co> References: <000c01cd735f$5082d010$f1887030$@co> Message-ID: <20120806053048.GB32434@phare.normalesup.org> On Sun, Aug 05, 2012 at 04:09:06PM -0700, James Abel wrote: > I'm trying to use scipy.cluster.vq.kmeans2() but I'm getting inconsistent > output. With a simple test input that should have 3 clusters, I'm getting > good results most of the time but every so often the output creates the > wrong clustering. K Means is a non-convex problem: it is dependent on the (random) initialization. In addition, it is not garantied to find the 'true' clusters, because quite often it is not possible from the data. You are not doing anything wrong, you are just asking for something that is not possible. HTH, Gael From evilper at gmail.com Mon Aug 6 05:32:01 2012 From: evilper at gmail.com (Per Nielsen) Date: Mon, 6 Aug 2012 11:32:01 +0200 Subject: [SciPy-User] Matrix-free version of connected_components In-Reply-To: <20120802123601.GF3396@phare.normalesup.org> References: <20120801131854.GA9137@phare.normalesup.org> <20120802123601.GF3396@phare.normalesup.org> Message-ID: You are completely right and the networkX code seems simple enough so that I can actually manage it. Now I just need to optimize my graph/matrix function as it calculates the entire graph/matrix MUCH faster than going all the rows/columns individually. Thanks. Per On Thu, Aug 2, 2012 at 2:36 PM, Gael Varoquaux < gael.varoquaux at normalesup.org> wrote: > On Thu, Aug 02, 2012 at 02:15:07PM +0200, Per Nielsen wrote: > > I have looked into the functions provided by the networkX package, > > however, as far as I have been able to understand both networkX and > scipy > > employ similar methods for finding the connected components. Thus I > still > > need the full sparse matrix (or graph which I think is similar in > memory > > usage) in memory, which I would like to avoid. > > But you can easily recode them to replace the indexing by a function call > that would return the list of neighbors for a given node. That way you > don't have to store the graph, you can generate it on the fly. > > G > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From evilper at gmail.com Mon Aug 6 05:33:59 2012 From: evilper at gmail.com (Per Nielsen) Date: Mon, 6 Aug 2012 11:33:59 +0200 Subject: [SciPy-User] Matrix-free version of connected_components In-Reply-To: References: <20120801131854.GA9137@phare.normalesup.org> Message-ID: > > > I'm not sure what your application is, but if you just need connected > components and have an easy way to find neighbors, then unionfind will > partition the set for you. Although the common version doesn't make it easy > to extract them, I have an implementation that keeps the connected nodes in > a circular list for just that application. > > I would very like to have copy of your algorithm, it might be easier to modify than the networkX code as Gael suggested. Per -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfurmston at googlemail.com Mon Aug 6 09:29:08 2012 From: tfurmston at googlemail.com (Tom Furmston) Date: Mon, 6 Aug 2012 14:29:08 +0100 Subject: [SciPy-User] gamma.pdf numerically unstable? Message-ID: Hi, I'm trying to use the gamma distribution in the SciPy library, but it appears to be numerically unstable. First, I've imported the library with from scipy.stats import gamma I want to evaluate the gamma distribution are x = 1.6, where the gamma distribution has the parameters shape = 142, scale = 0.0098. When I type gamma.pdf(1.6, 142, loc=0, scale=0.0098) I get a value of inf. According to matlab the value should be around 0.7030. Am I using this distribution correctly? Is this distribution known to suffer from numerical issues? TAI From ryanlists at gmail.com Mon Aug 6 09:51:46 2012 From: ryanlists at gmail.com (Ryan Krauss) Date: Mon, 6 Aug 2012 08:51:46 -0500 Subject: [SciPy-User] help speeding up a Runge-Kuta algorithm (cython, f2py, ...) In-Reply-To: <501DB030.6080108@molden.no> References: <501DB030.6080108@molden.no> Message-ID: Thanks to Sturla for helping me get this working in Cython. I am trying to compile the code to compare it against fortran for speed. I have run into two bugs so far (I mentioned that my C skills are weak). The first has to do with the "const trick": Error compiling Cython file: ------------------------------------------------------------ ... cdef inline void dxdt_runge_kuta(double *x "const double *", double voltage "const double", double *dxdt): #cdef double J = 0.0011767297528720126 "const double" cdef double J = 0.0011767297528720126 cdef double alpha0 = 4.1396263800000002 "const double" ^ ------------------------------------------------------------ runge_kuta_v2.pyx:12:44: Syntax error in C variable declaration I don't know what the problem is here, so for now I just got rid of all the "const double" statements. (In case the formatting doesn't come through, the little error carrot ^ points to the space between the last number and the quote. After getting rid of all the "const double" expressions (just to see if everything else would compile), I got this: Error compiling Cython file: ------------------------------------------------------------ ... dxdt[0] = vel dxdt[1] = accel dxdt[2] = dzdt def runge_kuta_one_step(double _x[::1], Py_ssize_t factor, double volts, ^ ------------------------------------------------------------ runge_kuta_v2.pyx:31:34: Expected an identifier or literal The carrot points to the first square bracket. Thanks, Ryan On Sat, Aug 4, 2012 at 6:28 PM, Sturla Molden wrote: > Not tested and debugged, but to me it looks like something like this might > be what you want. > > Sturla > > > Den 03.08.2012 19:02, skrev Ryan Krauss: > > I need help speeding up some code I wrote to perform a Runge-Kuta > integration. I need to do the integration as part of a real-time > control algorithm, so it needs to be fairly fast. > scipy.integrate.odeint does too much error checking to be fast enough. > My pure Python version was just a little too slow, so I tried coding > it up in Cython. I have only used Cython once before, so I don't know > if I did it correctly (the .pyx file is attached). > > The code runs just fine, but there is almost no speed up. I think the > core issue is that my dxdt_runge_kuta function gets called about 4000 > times per second, so most of my overhead is in the function calls (I > think). I am running my real-time control algorithm at 500 Hz and I > need at least 2 Runge-Kuta integration steps per real-time steps for > numeric stability. And the Runge-Kuta algorithm needs to evaluate the > derivative 4 times per times step. So, 500 Hz * 2 * 4 = 4000 calls > per second. > > I also tried coding this up in fortran and using f2py, but I am > getting a type mismatch error I don't understand. I have a function > that declares its return values as double precision: > > double precision function dzdt(x,voltage) > > and I declare the variable I want to store the returned value in to > also be double precision: > > double precision F,z,vel,accel,zdot1,zdot2,zdot3,zdot4 > > zdot1 = dzdt(x_prev,volts) > > but some how it is not happy. > > > My C skills are pretty weak (the longer I use Python, the more C I > forget, and I didn't know that much to start with). I started looking > into Boost as well as using f2py on C code, but I got stuck. > > > Can anyone either make my Cython or Fortran approaches work or point > me in a different direction? > > Thanks, > > Ryan > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From jsseabold at gmail.com Mon Aug 6 09:48:44 2012 From: jsseabold at gmail.com (Skipper Seabold) Date: Mon, 6 Aug 2012 09:48:44 -0400 Subject: [SciPy-User] gamma.pdf numerically unstable? In-Reply-To: References: Message-ID: On Mon, Aug 6, 2012 at 9:29 AM, Tom Furmston wrote: > Hi, > > I'm trying to use the gamma distribution in the SciPy library, but it > appears to be numerically unstable. > > First, I've imported the library with > > from scipy.stats import gamma > > I want to evaluate the gamma distribution are x = 1.6, where the gamma > distribution has the parameters shape = 142, scale = 0.0098. When I > type > > gamma.pdf(1.6, 142, loc=0, scale=0.0098) > > I get a value of inf. > > According to matlab the value should be around 0.7030. > > Am I using this distribution correctly? Is this distribution known to > suffer from numerical issues? > What SciPy version? [~/] [1]: from scipy.stats import gamma [~/] [2]: gamma.pdf(1.6, 142, loc=0, scale=.0098) [2]: 0.69705038205764014 [~/] [3]: np.exp(gamma.logpdf(1.6, 142, loc=0, scale=.0098)) [3]: 0.69705038205763981 [~/] [4]: import scipy [~/] [5]: scipy.__version__ [5]: '0.12.0.dev-8647010' In R > dgamma(1.6, 142, scale=.0098) [1] 0.6970504 Skipper From ryanlists at gmail.com Mon Aug 6 10:02:44 2012 From: ryanlists at gmail.com (Ryan Krauss) Date: Mon, 6 Aug 2012 09:02:44 -0500 Subject: [SciPy-User] help speeding up a Runge-Kuta algorithm (cython, f2py, ...) In-Reply-To: References: <501DB030.6080108@molden.no> Message-ID: So, I get the same error when I try to compile Stula's memview.pyx example. I think I have too old of a version of cython: Cython version 0.15.1 Let me look into that... On Mon, Aug 6, 2012 at 8:51 AM, Ryan Krauss wrote: > Thanks to Sturla for helping me get this working in Cython. > > I am trying to compile the code to compare it against fortran for > speed. I have run into two bugs so far (I mentioned that my C skills > are weak). > > The first has to do with the "const trick": > Error compiling Cython file: > ------------------------------------------------------------ > ... > cdef inline void dxdt_runge_kuta(double *x "const double *", > double voltage "const double", > double *dxdt): > #cdef double J = 0.0011767297528720126 "const double" > cdef double J = 0.0011767297528720126 > cdef double alpha0 = 4.1396263800000002 "const double" > ^ > ------------------------------------------------------------ > > runge_kuta_v2.pyx:12:44: Syntax error in C variable declaration > > I don't know what the problem is here, so for now I just got rid of > all the "const double" statements. (In case the formatting doesn't > come through, the little error carrot ^ points to the space between > the last number and the quote. > > After getting rid of all the "const double" expressions (just to see > if everything else would compile), I got this: > Error compiling Cython file: > ------------------------------------------------------------ > ... > dxdt[0] = vel > dxdt[1] = accel > dxdt[2] = dzdt > > > def runge_kuta_one_step(double _x[::1], Py_ssize_t factor, double volts, > ^ > ------------------------------------------------------------ > > runge_kuta_v2.pyx:31:34: Expected an identifier or literal > > The carrot points to the first square bracket. > > Thanks, > > Ryan > > > On Sat, Aug 4, 2012 at 6:28 PM, Sturla Molden wrote: >> Not tested and debugged, but to me it looks like something like this might >> be what you want. >> >> Sturla >> >> >> Den 03.08.2012 19:02, skrev Ryan Krauss: >> >> I need help speeding up some code I wrote to perform a Runge-Kuta >> integration. I need to do the integration as part of a real-time >> control algorithm, so it needs to be fairly fast. >> scipy.integrate.odeint does too much error checking to be fast enough. >> My pure Python version was just a little too slow, so I tried coding >> it up in Cython. I have only used Cython once before, so I don't know >> if I did it correctly (the .pyx file is attached). >> >> The code runs just fine, but there is almost no speed up. I think the >> core issue is that my dxdt_runge_kuta function gets called about 4000 >> times per second, so most of my overhead is in the function calls (I >> think). I am running my real-time control algorithm at 500 Hz and I >> need at least 2 Runge-Kuta integration steps per real-time steps for >> numeric stability. And the Runge-Kuta algorithm needs to evaluate the >> derivative 4 times per times step. So, 500 Hz * 2 * 4 = 4000 calls >> per second. >> >> I also tried coding this up in fortran and using f2py, but I am >> getting a type mismatch error I don't understand. I have a function >> that declares its return values as double precision: >> >> double precision function dzdt(x,voltage) >> >> and I declare the variable I want to store the returned value in to >> also be double precision: >> >> double precision F,z,vel,accel,zdot1,zdot2,zdot3,zdot4 >> >> zdot1 = dzdt(x_prev,volts) >> >> but some how it is not happy. >> >> >> My C skills are pretty weak (the longer I use Python, the more C I >> forget, and I didn't know that much to start with). I started looking >> into Boost as well as using f2py on C code, but I got stuck. >> >> >> Can anyone either make my Cython or Fortran approaches work or point >> me in a different direction? >> >> Thanks, >> >> Ryan >> >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> From josef.pktd at gmail.com Mon Aug 6 10:04:12 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 6 Aug 2012 10:04:12 -0400 Subject: [SciPy-User] gamma.pdf numerically unstable? In-Reply-To: References: Message-ID: On Mon, Aug 6, 2012 at 9:29 AM, Tom Furmston wrote: > Hi, > > I'm trying to use the gamma distribution in the SciPy library, but it > appears to be numerically unstable. > > First, I've imported the library with > > from scipy.stats import gamma > > I want to evaluate the gamma distribution are x = 1.6, where the gamma > distribution has the parameters shape = 142, scale = 0.0098. When I > type > > gamma.pdf(1.6, 142, loc=0, scale=0.0098) > > I get a value of inf. > > According to matlab the value should be around 0.7030. > > Am I using this distribution correctly? Is this distribution known to > suffer from numerical issues? > looks like numerical instability in the pdf, I guess it should use exp(logpdf) instead >>> stats.gamma.pdf(1.6, 142, loc=0, scale=0.0098) inf >>> np.exp(stats.gamma.logpdf(1.6, 142, loc=0, scale=0.0098)) 0.69705038205771896 would be worth a check with a multiprecision library. >>> x, a = 1.6, 142; loc=0; scale=0.0098 >>> mpmath.power(x/scale, (a-1))*mpmath.exp(-x/scale)/mpmath.gamma(a) / scale mpf('0.6970503820577183') a bit lower than matlab >>> np.exp(stats.gamma.logpdf(1.6, 142, loc=0, scale=0.098)) 4.5494987629369171e-79 >>> stats.gamma.pdf(1.6, 142, loc=0, scale=0.098) 4.5494987629366552e-79 It's not a known numerical problem, some numerical problems only surface when someone tries unusual parameters. Josef > > TAI > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Mon Aug 6 10:05:50 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 6 Aug 2012 10:05:50 -0400 Subject: [SciPy-User] gamma.pdf numerically unstable? In-Reply-To: References: Message-ID: On Mon, Aug 6, 2012 at 9:48 AM, Skipper Seabold wrote: > On Mon, Aug 6, 2012 at 9:29 AM, Tom Furmston > wrote: > > Hi, > > > > I'm trying to use the gamma distribution in the SciPy library, but it > > appears to be numerically unstable. > > > > First, I've imported the library with > > > > from scipy.stats import gamma > > > > I want to evaluate the gamma distribution are x = 1.6, where the gamma > > distribution has the parameters shape = 142, scale = 0.0098. When I > > type > > > > gamma.pdf(1.6, 142, loc=0, scale=0.0098) > > > > I get a value of inf. > > > > According to matlab the value should be around 0.7030. > > > > Am I using this distribution correctly? Is this distribution known to > > suffer from numerical issues? > > > > What SciPy version? > >>> import scipy >>> scipy.__version__ '0.9.0' already fixed then > > [~/] > [1]: from scipy.stats import gamma > > [~/] > [2]: gamma.pdf(1.6, 142, loc=0, scale=.0098) > [2]: 0.69705038205764014 > > [~/] > [3]: np.exp(gamma.logpdf(1.6, 142, loc=0, scale=.0098)) > [3]: 0.69705038205763981 > > [~/] > [4]: import scipy > > [~/] > [5]: scipy.__version__ > [5]: '0.12.0.dev-8647010' > > In R > > > dgamma(1.6, 142, scale=.0098) > [1] 0.6970504 > > > Skipper > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryanlists at gmail.com Mon Aug 6 10:21:51 2012 From: ryanlists at gmail.com (Ryan Krauss) Date: Mon, 6 Aug 2012 09:21:51 -0500 Subject: [SciPy-User] help speeding up a Runge-Kuta algorithm (cython, f2py, ...) In-Reply-To: References: <501DB030.6080108@molden.no> Message-ID: I upgraded to cython 0.16 and made a bit more progress. I don't know if this is headed in the right direction or not, but based on the memview.pyx example I changed double _x[::1] to np.float64_t[::1] _x and did the same thing with cdef double out[::1] = np.zeros(3) I seem to be closer to compiling successfully, but now have this error: Error compiling Cython file: ------------------------------------------------------------ ... import numpy as np cimport numpy as np from libc.math cimport exp, fabs cdef inline void dxdt_runge_kuta(double *x "const double *", ^ ------------------------------------------------------------ runge_kuta_v2.pyx:8:32: Function argument cannot have C name specification (carrot points to the last a in runge_kuta Thanks again, Ryan On Mon, Aug 6, 2012 at 9:02 AM, Ryan Krauss wrote: > So, I get the same error when I try to compile Stula's memview.pyx > example. I think I have too old of a version of cython: > > Cython version 0.15.1 > > Let me look into that... > > On Mon, Aug 6, 2012 at 8:51 AM, Ryan Krauss wrote: >> Thanks to Sturla for helping me get this working in Cython. >> >> I am trying to compile the code to compare it against fortran for >> speed. I have run into two bugs so far (I mentioned that my C skills >> are weak). >> >> The first has to do with the "const trick": >> Error compiling Cython file: >> ------------------------------------------------------------ >> ... >> cdef inline void dxdt_runge_kuta(double *x "const double *", >> double voltage "const double", >> double *dxdt): >> #cdef double J = 0.0011767297528720126 "const double" >> cdef double J = 0.0011767297528720126 >> cdef double alpha0 = 4.1396263800000002 "const double" >> ^ >> ------------------------------------------------------------ >> >> runge_kuta_v2.pyx:12:44: Syntax error in C variable declaration >> >> I don't know what the problem is here, so for now I just got rid of >> all the "const double" statements. (In case the formatting doesn't >> come through, the little error carrot ^ points to the space between >> the last number and the quote. >> >> After getting rid of all the "const double" expressions (just to see >> if everything else would compile), I got this: >> Error compiling Cython file: >> ------------------------------------------------------------ >> ... >> dxdt[0] = vel >> dxdt[1] = accel >> dxdt[2] = dzdt >> >> >> def runge_kuta_one_step(double _x[::1], Py_ssize_t factor, double volts, >> ^ >> ------------------------------------------------------------ >> >> runge_kuta_v2.pyx:31:34: Expected an identifier or literal >> >> The carrot points to the first square bracket. >> >> Thanks, >> >> Ryan >> >> >> On Sat, Aug 4, 2012 at 6:28 PM, Sturla Molden wrote: >>> Not tested and debugged, but to me it looks like something like this might >>> be what you want. >>> >>> Sturla >>> >>> >>> Den 03.08.2012 19:02, skrev Ryan Krauss: >>> >>> I need help speeding up some code I wrote to perform a Runge-Kuta >>> integration. I need to do the integration as part of a real-time >>> control algorithm, so it needs to be fairly fast. >>> scipy.integrate.odeint does too much error checking to be fast enough. >>> My pure Python version was just a little too slow, so I tried coding >>> it up in Cython. I have only used Cython once before, so I don't know >>> if I did it correctly (the .pyx file is attached). >>> >>> The code runs just fine, but there is almost no speed up. I think the >>> core issue is that my dxdt_runge_kuta function gets called about 4000 >>> times per second, so most of my overhead is in the function calls (I >>> think). I am running my real-time control algorithm at 500 Hz and I >>> need at least 2 Runge-Kuta integration steps per real-time steps for >>> numeric stability. And the Runge-Kuta algorithm needs to evaluate the >>> derivative 4 times per times step. So, 500 Hz * 2 * 4 = 4000 calls >>> per second. >>> >>> I also tried coding this up in fortran and using f2py, but I am >>> getting a type mismatch error I don't understand. I have a function >>> that declares its return values as double precision: >>> >>> double precision function dzdt(x,voltage) >>> >>> and I declare the variable I want to store the returned value in to >>> also be double precision: >>> >>> double precision F,z,vel,accel,zdot1,zdot2,zdot3,zdot4 >>> >>> zdot1 = dzdt(x_prev,volts) >>> >>> but some how it is not happy. >>> >>> >>> My C skills are pretty weak (the longer I use Python, the more C I >>> forget, and I didn't know that much to start with). I started looking >>> into Boost as well as using f2py on C code, but I got stuck. >>> >>> >>> Can anyone either make my Cython or Fortran approaches work or point >>> me in a different direction? >>> >>> Thanks, >>> >>> Ryan >>> >>> >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> From tfurmston at googlemail.com Mon Aug 6 10:28:26 2012 From: tfurmston at googlemail.com (Tom Furmston) Date: Mon, 6 Aug 2012 15:28:26 +0100 Subject: [SciPy-User] gamma.pdf numerically unstable? In-Reply-To: References: Message-ID: Thanks for all the responses. My SciPy version is 0.7.2, so it looks like it might be a version issue. Will update and so what happens. On 8/6/12, josef.pktd at gmail.com wrote: > On Mon, Aug 6, 2012 at 9:48 AM, Skipper Seabold > wrote: > >> On Mon, Aug 6, 2012 at 9:29 AM, Tom Furmston >> wrote: >> > Hi, >> > >> > I'm trying to use the gamma distribution in the SciPy library, but it >> > appears to be numerically unstable. >> > >> > First, I've imported the library with >> > >> > from scipy.stats import gamma >> > >> > I want to evaluate the gamma distribution are x = 1.6, where the gamma >> > distribution has the parameters shape = 142, scale = 0.0098. When I >> > type >> > >> > gamma.pdf(1.6, 142, loc=0, scale=0.0098) >> > >> > I get a value of inf. >> > >> > According to matlab the value should be around 0.7030. >> > >> > Am I using this distribution correctly? Is this distribution known to >> > suffer from numerical issues? >> > >> >> What SciPy version? >> > >>>> import scipy >>>> scipy.__version__ > '0.9.0' > > already fixed then > > >> >> [~/] >> [1]: from scipy.stats import gamma >> >> [~/] >> [2]: gamma.pdf(1.6, 142, loc=0, scale=.0098) >> [2]: 0.69705038205764014 >> >> [~/] >> [3]: np.exp(gamma.logpdf(1.6, 142, loc=0, scale=.0098)) >> [3]: 0.69705038205763981 >> >> [~/] >> [4]: import scipy >> >> [~/] >> [5]: scipy.__version__ >> [5]: '0.12.0.dev-8647010' >> >> In R >> >> > dgamma(1.6, 142, scale=.0098) >> [1] 0.6970504 >> >> >> Skipper >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > From sergio_r at mail.com Mon Aug 6 09:51:23 2012 From: sergio_r at mail.com (Sergio Rojas) Date: Mon, 06 Aug 2012 09:51:23 -0400 Subject: [SciPy-User] help speeding up a Runge-Kuta algorithm (cython, f2py, ...) (Sturla Molden) (Sturla Molden) Message-ID: <20120806135123.4280@gmx.com> > Which version of Cython do you have (use at least 0.16). > Sturla I missed that. Now the full stuff compiles and runs: >$ ?python -c "from Cython.Compiler.Version import version; print version" 0.16 >$ python runme.py numpy_only: 3.5 sec cythonized_numpy: 2.62 sec cythonized_numpy_2: 4.15 sec cythonized_numpy_2b: 3.35 sec cythonized_numpy_3: 1.48 sec cythonized_numpy_4: 1.18 sec pointer_arith: 1.19 sec memview: 1.22 sec Sergio ************************************************************ Here is an example code of how to use Cython as a Fortran killer: https://github.com/sturlamolden/memview_benchmarks/blob/master/memview.pyx In this case, the performance with -O2 was just 2.2% slower than "plain C" with pointer arithmetics. It is possible to write very fast array code with Cython, but you must do it right. For comparison, this is very slow: https://github.com/sturlamolden/memview_benchmarks/blob/master/cythonized_numpy_2b.pyx What this mean is this: For anything but trivial code, the NumPy syntax is just too slow and should be avoided! I breiefly looked at the Cython code posted in this thread, and it suffers form all these issues. Sturla ********************************************** From davesmivers at gmail.com Mon Aug 6 14:11:49 2012 From: davesmivers at gmail.com (Dave Smith) Date: Mon, 6 Aug 2012 19:11:49 +0100 Subject: [SciPy-User] Saving result of sparse.linalg.splu and spilu to file\disk Message-ID: Hi guys/girls, I'm trying to save the result of a (sparse.linalg.spilu) LU decomposition of A such that Ax = b to disk. I only need the result once to apply to many sets of b (solutions). The spilu takes ~ 4hours to compute. Unfortunately, the object returned is a 'factored_lu' type and I don't know how to save this to disk. It won't pickel or mashal: Below is some test code that builds a small ILU 'factored_lu' object called EquSpaInvish ____________________________________________________________________________ from scipy.sparse import lil_matrix from scipy.sparse.linalg import spsolve,spilu from numpy.linalg import solve, norm, inv from numpy.random import rand size = 250 A = lil_matrix((size, size)) A[0, :size] = rand(size) A.setdiag(rand(size)) A = A.tocsc() b = rand(size) EquSpaInvish = spilu(A) iluelap = tm.time() - ilustart x = EquSpaInvish.solve(b) _____________________________________________________________________________ I want to save\load EquSpaInvish to\from disk so i don't have to calculate it each time. Any ideas......? Kind Regards Dave :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Mon Aug 6 14:31:48 2012 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 06 Aug 2012 20:31:48 +0200 Subject: [SciPy-User] Saving result of sparse.linalg.splu and spilu to file\disk In-Reply-To: References: Message-ID: 06.08.2012 20:11, Dave Smith kirjoitti: > I'm trying to save the result of a (sparse.linalg.spilu) LU > decomposition of A such that Ax = b to disk. I only need the result > once to apply to many sets of b (solutions). The spilu takes ~ 4hours > to compute. > > Unfortunately, the object returned is a 'factored_lu' type and I don't > know how to save this to disk. It won't pickel or mashal: [clip] > Any ideas......? The result of the LU decomposition is in SuperLU internal format that is not exposed via the Python interface. You'll need to dig into SuperLU source code and take a look at the Scipy wrapper. The relevant source files are under scipy/sparse/linalg/dsolve/ in Scipy's source code. -- Pauli Virtanen From ralf.gommers at gmail.com Sun Aug 5 11:41:21 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sun, 5 Aug 2012 17:41:21 +0200 Subject: [SciPy-User] qHull zero division error, scipy 0.10.1, linux In-Reply-To: <1343695343.11264.YahooMailNeo@web113409.mail.gq1.yahoo.com> References: <1343695343.11264.YahooMailNeo@web113409.mail.gq1.yahoo.com> Message-ID: On Tue, Jul 31, 2012 at 2:42 AM, David Baddeley wrote: > I get a zero division error following error when trying to use the qHull > bindings in scipy.spatial, e.g. > > In [8]: from scipy.spatial import qhull > In [9]: qhull.Delaunay(randn(100, 2)) > --------------------------------------------------------------------------- > ZeroDivisionError Traceback (most recent call last) > /home/david/ in () > /usr/local/lib/python2.6/dist-packages/scipy-0.10.1-py2.6-linux-x86_64.egg/scipy/spatial/qhull.so > in scipy.spatial.qhull.Delaunay.__init__ (scipy/spatial/qhull.c:4109)() > /usr/local/lib/python2.6/dist-packages/scipy-0.10.1-py2.6-linux-x86_64.egg/scipy/spatial/qhull.so > in scipy.spatial.qhull._construct_delaunay (scipy/spatial/qhull.c:1344)() > ZeroDivisionError: float division > > All the tests involving the qhull module also fail with the same error if > I run scipy.test(). > > full system details below: > > NumPy version 1.6.2 > NumPy is installed in > /usr/local/lib/python2.6/dist-packages/numpy-1.6.2-py2.6-linux-x86_64.egg/numpy > SciPy version 0.10.1 > SciPy is installed in > /usr/local/lib/python2.6/dist-packages/scipy-0.10.1-py2.6-linux-x86_64.egg/scipy > Python version 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) [GCC 4.4.3] > nose version 1.1.2 > Ubuntu 10.04 LTS x64 > > both numpy and scipy were installed using easy_install. > > before I download the source and start messing with qhull.pyx / the qhull > sources themselves. I thought I'd ask if there were any known bugs / clues. > I can't reproduce this with current master and it doesn't look familiar. There were some fixes to qhull though for 0.11, so I'd check first if those fixed your issue. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Mon Aug 6 16:38:49 2012 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 06 Aug 2012 22:38:49 +0200 Subject: [SciPy-User] qHull zero division error, scipy 0.10.1, linux In-Reply-To: References: <1343695343.11264.YahooMailNeo@web113409.mail.gq1.yahoo.com> Message-ID: 05.08.2012 17:41, Ralf Gommers kirjoitti: [clip] > On Tue, Jul 31, 2012 at 2:42 AM, David Baddeley [clip] > before I download the source and start messing with qhull.pyx / the > qhull sources themselves. I thought I'd ask if there were any known > bugs / clues. > > I can't reproduce this with current master and it doesn't look familiar. > There were some fixes to qhull though for 0.11, so I'd check first if > those fixed your issue. This seems like a failure occurring inside Qhull --- the scale parameters seem to not have been computed for a reason or another. I haven't seen any reports about anyone else seeing such a failure before, so compiler problems may also come into question. -- Pauli Virtanen From ralf.gommers at gmail.com Mon Aug 6 17:28:23 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 6 Aug 2012 23:28:23 +0200 Subject: [SciPy-User] Brentq return values In-Reply-To: <1344222432.75833.YahooMailNeo@web31808.mail.mud.yahoo.com> References: <1344222432.75833.YahooMailNeo@web31808.mail.mud.yahoo.com> Message-ID: On Mon, Aug 6, 2012 at 5:07 AM, The Helmbolds wrote: > When the brentq routine is called with `full_output` set to `True` it > returns a tuple. > The first element of that tuple is the root, i.e., a simple real number. > So far, so good. > > However, the second element of that tuple is: > > I don't know how to read or "decode" such a thing. > I'd welcome any instruction on how to do that. > > I also observe that perhaps the return value could be more transparent, > i.e., easy to interpret. > The docstring says that (if we call the RootResults instance r) there's an attribute r.converged. Would be good to show in an example the rest: r.root r.iterations r.function_calls r.converged What those things are seems self-explanatory. You can find out things like the above by tab completion in IPython for example. Type r. and you'll see what attributes and methods are present (alternatively, look at the source code). Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Mon Aug 6 19:18:42 2012 From: sturla at molden.no (Sturla Molden) Date: Tue, 07 Aug 2012 01:18:42 +0200 Subject: [SciPy-User] help speeding up a Runge-Kuta algorithm (cython, f2py, ...) In-Reply-To: References: <501DB030.6080108@molden.no> Message-ID: <502050D2.2020308@molden.no> Den 06.08.2012 15:51, skrev Ryan Krauss: > Thanks to Sturla for helping me get this working in Cython. > > I am trying to compile the code to compare it against fortran for > speed. I have run into two bugs so far (I mentioned that my C skills > are weak). > Sorry, I should have debugged :( This one compiles with $ python setup.py build_ext Is this what you wanted? Sturla -------------- next part -------------- A non-text attachment was scrubbed... Name: runge_kutta.zip Type: application/x-zip-compressed Size: 1231 bytes Desc: not available URL: From charlesr.harris at gmail.com Mon Aug 6 21:41:09 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 6 Aug 2012 19:41:09 -0600 Subject: [SciPy-User] Matrix-free version of connected_components In-Reply-To: References: <20120801131854.GA9137@phare.normalesup.org> Message-ID: On Mon, Aug 6, 2012 at 3:33 AM, Per Nielsen wrote: > >> I'm not sure what your application is, but if you just need connected >> components and have an easy way to find neighbors, then unionfind will >> partition the set for you. Although the common version doesn't make it easy >> to extract them, I have an implementation that keeps the connected nodes in >> a circular list for just that application. >> >> > I would very like to have copy of your algorithm, it might be easier to > modify than the networkX code as Gael suggested. > Well, it's in Cython ;) It is set up to use integer indices as node labels and all components are extracted as lists of integers. It's also pretty old and needs documentation, so I'll do that and clean it up a bit. I'll be on travel for the next few days, so unless matters are pressing, I won't get to it until the weekend. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From mohanr at fss.co.in Tue Aug 7 01:11:07 2012 From: mohanr at fss.co.in (Mohan Radhakrishnan) Date: Tue, 7 Aug 2012 10:41:07 +0530 Subject: [SciPy-User] Sample queueing analysis Message-ID: <3BAA9108A76CF149A553864B23D47E13013A261B@fssbemail.fss.india> Hi, I am trying to learn to draw graphs for performance models. As part of this process I have a sample problem from a technical paper "The Operational Analysis of Queueing Network Models" There are 3 componens and their visit rations and service times. V1S1 = (20)(.05) = 1.00 seconds (Total CPU time is the bottleneck ) V2S2 = (11)(.08) = .88 seconds (Total Disk time) V3S3 = (8)(.04)= .32 seconds (Total Drum time) These products sum to the minimal response time of 2.2 seconds M1 = Z/VISI = 20/1 = 20 terminals Now I want to draw the graph by sending these values as inputs to a queueing model. So if I plot terminals vs seconds the asymptote of the response time curve should intersect The y-axis at 20 which is M1 give above. Is this type of work done using Sci-Py to solve a queueing model ? Can I read some references ? Thanks, Mohan DISCLAIMER: ==========================================================================================================================================================The information contained in this e-mail message may be privileged and/or confidential and protected from disclosure under applicable law. It is intended only for the individual to whom or entity to which it is addressed as shown at the beginning of the message. If the reader of this message is not the intended recipient, or if the employee or agent responsible for delivering the message is not an employee or agent of the intended recipient, you are hereby notified that any review, dissemination,distribution, use, or copying of this message is strictly prohibited. If you have received this message in error, please notify us immediately by return e-mail and permanently delete this message and your reply to the extent it includes this message. Any views or opinions presented in this message or attachments are those of the author and do not necessarily represent those of the Company. All e-mails and attachments sent and received are subject to monitoring, reading, and archival by the Company.========================================================================================================================================================== -------------- next part -------------- An HTML attachment was scrubbed... URL: From evilper at gmail.com Tue Aug 7 03:23:22 2012 From: evilper at gmail.com (Per Nielsen) Date: Tue, 7 Aug 2012 09:23:22 +0200 Subject: [SciPy-User] Matrix-free version of connected_components In-Reply-To: References: <20120801131854.GA9137@phare.normalesup.org> Message-ID: Ah hehe. I have had a hard time optimizing my graph neighbor extraction code, so right now it is actually to slow to be of any use. I will try to find a different approach and I dont want you spending any of your time cleaning code, but thank you very much for the offer :) Per On Tue, Aug 7, 2012 at 3:41 AM, Charles R Harris wrote: > > > On Mon, Aug 6, 2012 at 3:33 AM, Per Nielsen wrote: > >> >>> I'm not sure what your application is, but if you just need connected >>> components and have an easy way to find neighbors, then unionfind will >>> partition the set for you. Although the common version doesn't make it easy >>> to extract them, I have an implementation that keeps the connected nodes in >>> a circular list for just that application. >>> >>> >> I would very like to have copy of your algorithm, it might be easier to >> modify than the networkX code as Gael suggested. >> > > Well, it's in Cython ;) It is set up to use integer indices as node labels > and all components are extracted as lists of integers. It's also pretty old > and needs documentation, so I'll do that and clean it up a bit. I'll be on > travel for the next few days, so unless matters are pressing, I won't get > to it until the weekend. > > Chuck > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pawel.kw at gmail.com Tue Aug 7 09:16:46 2012 From: pawel.kw at gmail.com (=?ISO-8859-2?Q?Pawe=B3_Kwa=B6niewski?=) Date: Tue, 7 Aug 2012 15:16:46 +0200 Subject: [SciPy-User] Question about errors (uncertainties) in non-linear least squares fitting Message-ID: Hi, I'm fitting some data using a wrapper around the scipy.optimize.leastsq method which can be found under http://code.google.com/p/nmrglue/source/browse/trunk/nmrglue/analysis/leastsqbound.pyBasically it allows for putting bounds on the fitted parameters, which is very important for me. I'm using the covariance matrix, returned by leastsq() function to estimate the errors of the fitted parameters. The fitting is done using real measurement uncertainties (which are ridiculously small, by the way), so I would expect the resulting parameter error to be reasonable. What don't understand, is that I'm getting extremely small errors on the fitted parameters (I calculate the errors as perr = sqrt(diag(fitres[1])), where fitres[1] is the covariance matrix returned by leastsq() function). For example, a parameter which has a fitted value of ~100 gets an error of ~1e-6. At the same time, when I calculate the reduced chi squared of the fit I'm getting an extremely large number (of the order of 1e8). I can understand the large chi^2 value - the data variance is extremely small and the model curve is not perfect, so even slight deviations of the fitted model from the data will blow up chi^2 into space. But how can the fitted parameter variance be so small, while at the same time the fit is garbage according to chi^2? I guess this requires a better understanding of how the covariance matrix is calculated. Some suggestions anyone? Cheers, Pawe? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jjhelmus at gmail.com Tue Aug 7 10:25:51 2012 From: jjhelmus at gmail.com (Jonathan Helmus) Date: Tue, 07 Aug 2012 10:25:51 -0400 Subject: [SciPy-User] Question about errors (uncertainties) in non-linear least squares fitting In-Reply-To: References: Message-ID: <5021256F.6040606@gmail.com> Pawel, First off you may want to use a more up to date version of leastsqbound which can be found at https://github.com/jjhelmus/leastsqbound-scipy Second, when you perform a constrained optimization using internal parameters like leastsqbound does, if one of more of the parameters is close to a bound, the values in the covariance matrix can take on meaningless values. Section 1.3 of the The Minuit User's Guide [1] gives a good overview of this, especially look at the discussion on page 5. For best results an unconstrained optimization should be performed, often times you can rewrite your model in such a way that the constraints are automatically imposed (this is what is done internally in leastsqbound, but transforming back to the original model can introduce large errors if a parameter is close to the bounds). Third, since you have measurement uncertainties make sure you include them in the chi^2 calculation. I find the discussion by P.H. Richter [2] to be quite good. Cheers, - Jonathan Helmus [1] http://seal.cern.ch/documents/minuit/mnusersguide.pdf [2] Estimating Errors in Least-Squares Fitting, P.H. Richter TDA Progress Report 42-122 http://tmo.jpl.nasa.gov/progress_report/42-122/122E.pdf On 08/07/2012 09:16 AM, Pawe? Kwas'niewski wrote: > Hi, > > I'm fitting some data using a wrapper around the > scipy.optimize.leastsq method which can be found under > http://code.google.com/p/nmrglue/source/browse/trunk/nmrglue/analysis/leastsqbound.py > Basically it allows for putting bounds on the fitted parameters, which > is very important for me. > > I'm using the covariance matrix, returned by leastsq() function to > estimate the errors of the fitted parameters. The fitting is done > using real measurement uncertainties (which are ridiculously small, by > the way), so I would expect the resulting parameter error to be > reasonable. What don't understand, is that I'm getting extremely small > errors on the fitted parameters (I calculate the errors as perr = > sqrt(diag(fitres[1])), where fitres[1] is the covariance matrix > returned by leastsq() function). For example, a parameter which has a > fitted value of ~100 gets an error of ~1e-6. At the same time, when I > calculate the reduced chi squared of the fit I'm getting an extremely > large number (of the order of 1e8). I can understand the large chi^2 > value - the data variance is extremely small and the model curve is > not perfect, so even slight deviations of the fitted model from the > data will blow up chi^2 into space. But how can the fitted parameter > variance be so small, while at the same time the fit is garbage > according to chi^2? > > I guess this requires a better understanding of how the covariance > matrix is calculated. Some suggestions anyone? > > Cheers, > > Pawe? > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryanlists at gmail.com Tue Aug 7 12:37:31 2012 From: ryanlists at gmail.com (Ryan Krauss) Date: Tue, 7 Aug 2012 11:37:31 -0500 Subject: [SciPy-User] help speeding up a Runge-Kuta algorithm (cython, f2py, ...) In-Reply-To: <502050D2.2020308@molden.no> References: <501DB030.6080108@molden.no> <502050D2.2020308@molden.no> Message-ID: Thanks Stula. That code compiles just fine and will go a long way toward helping me understand how to use Cython to write fast code for these kinds of applications. For many Runge-Kutta steps, your Cython code is 200 times faster than my pure Python version. Fortran is still 1.6 times faster than the Cython version, but the Fortran version is much more work to code up. Thanks again, Ryan On Mon, Aug 6, 2012 at 6:18 PM, Sturla Molden wrote: > Den 06.08.2012 15:51, skrev Ryan Krauss: > >> Thanks to Sturla for helping me get this working in Cython. >> >> I am trying to compile the code to compare it against fortran for >> speed. I have run into two bugs so far (I mentioned that my C skills >> are weak). >> > > Sorry, I should have debugged :( > > This one compiles with > > $ python setup.py build_ext > > Is this what you wanted? > > > > Sturla > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From sturla at molden.no Tue Aug 7 14:10:55 2012 From: sturla at molden.no (Sturla Molden) Date: Tue, 07 Aug 2012 20:10:55 +0200 Subject: [SciPy-User] help speeding up a Runge-Kuta algorithm (cython, f2py, ...) In-Reply-To: References: <501DB030.6080108@molden.no> <502050D2.2020308@molden.no> Message-ID: <50215A2F.1070303@molden.no> On 07.08.2012 18:37, Ryan Krauss wrote: > For many Runge-Kutta steps, your Cython code is 200 times faster than > my pure Python version. Fortran is still 1.6 times faster than the > Cython version, but the Fortran version is much more work to code up. Don't expect anything to be "faster than Fortran" for certain kind of numerical work. Cython has a certain overhead (larger than C and Fortran), and since it compiles to ANSI C (not ISO C) we cannot restrict pointers. But still, ~75% of Fortran performance is often acceptable! Another thing is you need to look at "scalability". How much of that extra runtime is constant due to differences between Cython and f2py? How much is variable due to the numerical kernel being faster in Fortran? Will differently sized problems give you the same overhead from using Cython? It often helps to plot a graph of the performance (mean and error bars) for various problem sizes, rather than benchmarking at one single point. Correctness is always more important than speed. That is one thing to consider too. With Cython we can begin with a tested Python prototype and optimize along the way, using the Python profiler to pinpoint where it matters the most. Python, NumPy and Cython will not win the world championship of being "fastest on the CPU" for simple numerical kernels, but that is not the idea either. Implementing complex algorithms in Fortran can be a PITA compared to Python. But Cython helps us in a stright forward way to speed up Python code and/or interface with C or C++. Fortran is only nice for helping us scientists to avoid the pointer arithmetics of C, but Cython's memoryviews do that too. Sturla From ryanlists at gmail.com Tue Aug 7 15:24:39 2012 From: ryanlists at gmail.com (Ryan Krauss) Date: Tue, 7 Aug 2012 14:24:39 -0500 Subject: [SciPy-User] help speeding up a Runge-Kuta algorithm (cython, f2py, ...) In-Reply-To: <50215A2F.1070303@molden.no> References: <501DB030.6080108@molden.no> <502050D2.2020308@molden.no> <50215A2F.1070303@molden.no> Message-ID: I agree. Thanks again. On Tue, Aug 7, 2012 at 1:10 PM, Sturla Molden wrote: > On 07.08.2012 18:37, Ryan Krauss wrote: > >> For many Runge-Kutta steps, your Cython code is 200 times faster than >> my pure Python version. Fortran is still 1.6 times faster than the >> Cython version, but the Fortran version is much more work to code up. > > Don't expect anything to be "faster than Fortran" for certain kind of > numerical work. Cython has a certain overhead (larger than C and > Fortran), and since it compiles to ANSI C (not ISO C) we cannot restrict > pointers. But still, ~75% of Fortran performance is often acceptable! > Another thing is you need to look at "scalability". How much of that > extra runtime is constant due to differences between Cython and f2py? > How much is variable due to the numerical kernel being faster in > Fortran? Will differently sized problems give you the same overhead from > using Cython? It often helps to plot a graph of the performance (mean > and error bars) for various problem sizes, rather than benchmarking at > one single point. > > Correctness is always more important than speed. That is one thing to > consider too. With Cython we can begin with a tested Python prototype > and optimize along the way, using the Python profiler to pinpoint where > it matters the most. Python, NumPy and Cython will not win the world > championship of being "fastest on the CPU" for simple numerical kernels, > but that is not the idea either. Implementing complex algorithms in > Fortran can be a PITA compared to Python. But Cython helps us in a > stright forward way to speed up Python code and/or interface with C or > C++. Fortran is only nice for helping us scientists to avoid the pointer > arithmetics of C, but Cython's memoryviews do that too. > > > Sturla > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From pawel.kw at gmail.com Tue Aug 7 17:31:37 2012 From: pawel.kw at gmail.com (=?ISO-8859-2?Q?Pawe=B3_Kwa=B6niewski?=) Date: Tue, 7 Aug 2012 23:31:37 +0200 Subject: [SciPy-User] Question about errors (uncertainties) in non-linear least squares fitting In-Reply-To: <5021256F.6040606@gmail.com> References: <5021256F.6040606@gmail.com> Message-ID: Dear Jonathan, Thank you for the quick answer and pointing me to the updated code and the literature. I'll certainly have a look into it. Just for the record, I wanted to use constrained fitting because the unconstrained fit sometimes gave me unphysical parameters. It's interesting what you wrote about how the constraint influences the parameter error estimate. Just to give it a shot I tried an unconstrained fit on one data set. It did converge, but still - I got enormous chi^2 (reduced, and yes - I'm using the experimental errors to calculate it) with ridiculously small parameter errors... I guess I need to understand better what's going on. By the way, the constrained fitting function is great, but I'd like to add another feature - the possibility to fix a parameter. This would largely facilitate testing of the fitting function, or can even be useful in real applications, where one of the parameters is known from a previous fit of a slightly different model or a different measurement or whatever... Anyway - maybe you have any ideas how to implement that? I already tried setting the upper and lower bound to the same value - it doesn't work. The method you use to apply the bounds throws division by zero error. ** Cheers, Pawe? 2012/8/7 Jonathan Helmus > Pawel, > > First off you may want to use a more up to date version of > leastsqbound which can be found at > https://github.com/jjhelmus/leastsqbound-scipy > > Second, when you perform a constrained optimization using internal > parameters like leastsqbound does, > if one of more of the parameters is close to a bound, the values in the > covariance matrix can take on meaningless values. Section 1.3 of the The > Minuit User's Guide [1] gives a good overview of this, especially look at > the discussion on page 5. For best results an unconstrained optimization > should be performed, often times you can rewrite your model in such a way > that the constraints are automatically imposed (this is what is done > internally in leastsqbound, but transforming back to the original model can > introduce large errors if a parameter is close to the bounds). > > Third, since you have measurement uncertainties make sure you include > them in the chi^2 calculation. I find the discussion by P.H. Richter [2] > to be quite good. > > Cheers, > > - Jonathan Helmus > > > > > [1] http://seal.cern.ch/documents/minuit/mnusersguide.pdf > [2] Estimating Errors in Least-Squares Fitting, P.H. Richter TDA Progress > Report 42-122 > http://tmo.jpl.nasa.gov/progress_report/42-122/122E.pdf > > > On 08/07/2012 09:16 AM, Pawe? Kwa?niewski wrote: > > Hi, > > I'm fitting some data using a wrapper around the scipy.optimize.leastsq > method which can be found under > http://code.google.com/p/nmrglue/source/browse/trunk/nmrglue/analysis/leastsqbound.pyBasically it allows for putting bounds on the fitted parameters, which is > very important for me. > > I'm using the covariance matrix, returned by leastsq() function to > estimate the errors of the fitted parameters. The fitting is done using > real measurement uncertainties (which are ridiculously small, by the way), > so I would expect the resulting parameter error to be reasonable. What > don't understand, is that I'm getting extremely small errors on the fitted > parameters (I calculate the errors as perr = sqrt(diag(fitres[1])), where > fitres[1] is the covariance matrix returned by leastsq() function). For > example, a parameter which has a fitted value of ~100 gets an error of > ~1e-6. At the same time, when I calculate the reduced chi squared of the > fit I'm getting an extremely large number (of the order of 1e8). I can > understand the large chi^2 value - the data variance is extremely small and > the model curve is not perfect, so even slight deviations of the fitted > model from the data will blow up chi^2 into space. But how can the fitted > parameter variance be so small, while at the same time the fit is garbage > according to chi^2? > > I guess this requires a better understanding of how the covariance matrix > is calculated. Some suggestions anyone? > > Cheers, > > Pawe? > > > _______________________________________________ > SciPy-User mailing listSciPy-User at scipy.orghttp://mail.scipy.org/mailman/listinfo/scipy-user > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david_baddeley at yahoo.com.au Tue Aug 7 18:24:52 2012 From: david_baddeley at yahoo.com.au (David Baddeley) Date: Tue, 7 Aug 2012 15:24:52 -0700 (PDT) Subject: [SciPy-User] qHull zero division error, scipy 0.10.1, linux In-Reply-To: References: <1343695343.11264.YahooMailNeo@web113409.mail.gq1.yahoo.com> Message-ID: <1344378292.5898.YahooMailNeo@web113417.mail.gq1.yahoo.com> Many thanks for taking the time to look at this. Was hoping someone had already come across it, but seems I'm out of luck.?Having delved into the internals of qHull before in a?separate?project I don't relish the prospect.?Am beginning to suspect there is some funny compiler problem/interactions with old numpy/scipy verisons as an extension module of mine (using the numpy c api) which previously ran fine is now also failing. Will try downloading the tarballs and compiling from them after deleting any previous versions rather than using easy_install. cheers, David ________________________________ From: Pauli Virtanen To: scipy-user at scipy.org Sent: Tuesday, 7 August 2012 8:38 AM Subject: Re: [SciPy-User] qHull zero division error, scipy 0.10.1, linux 05.08.2012 17:41, Ralf Gommers kirjoitti: [clip] > On Tue, Jul 31, 2012 at 2:42 AM, David Baddeley [clip] >? ? before I download the source and start messing with qhull.pyx / the >? ? qhull sources themselves. I thought I'd ask if there were any known >? ? bugs / clues. > > I can't reproduce this with current master and it doesn't look familiar. > There were some fixes to qhull though for 0.11, so I'd check first if > those fixed your issue. This seems like a failure occurring inside Qhull --- the scale parameters seem to not have been computed for a reason or another. I haven't seen any reports about anyone else seeing such a failure before, so compiler problems may also come into question. -- Pauli Virtanen _______________________________________________ SciPy-User mailing list SciPy-User at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevin.gullikson at gmail.com Tue Aug 7 17:37:09 2012 From: kevin.gullikson at gmail.com (Kevin Gullikson) Date: Tue, 7 Aug 2012 16:37:09 -0500 Subject: [SciPy-User] Question about errors (uncertainties) in non-linear least squares fitting In-Reply-To: References: <5021256F.6040606@gmail.com> Message-ID: Pawell, I do some fitting with leastq and fix parameters by just giving them as part of the 'args' keyword. I would assume you could do the same. On Aug 7, 2012 4:32 PM, "Pawe? Kwa?niewski" wrote: > Dear Jonathan, > > Thank you for the quick answer and pointing me to the updated code and the > literature. I'll certainly have a look into it. Just for the record, I > wanted to use constrained fitting because the unconstrained fit sometimes > gave me unphysical parameters. > > It's interesting what you wrote about how the constraint influences the > parameter error estimate. Just to give it a shot I tried an unconstrained > fit on one data set. It did converge, but still - I got enormous chi^2 > (reduced, and yes - I'm using the experimental errors to calculate it) with > ridiculously small parameter errors... I guess I need to understand better > what's going on. > > By the way, the constrained fitting function is great, but I'd like to add > another feature - the possibility to fix a parameter. This would largely > facilitate testing of the fitting function, or can even be useful in real > applications, where one of the parameters is known from a previous fit of a > slightly different model or a different measurement or whatever... Anyway - > maybe you have any ideas how to implement that? I already tried setting the > upper and lower bound to the same value - it doesn't work. The method you > use to apply the bounds throws division by zero error. ** > > Cheers, > > Pawe? > > > 2012/8/7 Jonathan Helmus > >> Pawel, >> >> First off you may want to use a more up to date version of >> leastsqbound which can be found at >> https://github.com/jjhelmus/leastsqbound-scipy >> >> Second, when you perform a constrained optimization using internal >> parameters like leastsqbound does, >> if one of more of the parameters is close to a bound, the values in the >> covariance matrix can take on meaningless values. Section 1.3 of the The >> Minuit User's Guide [1] gives a good overview of this, especially look at >> the discussion on page 5. For best results an unconstrained optimization >> should be performed, often times you can rewrite your model in such a way >> that the constraints are automatically imposed (this is what is done >> internally in leastsqbound, but transforming back to the original model can >> introduce large errors if a parameter is close to the bounds). >> >> Third, since you have measurement uncertainties make sure you include >> them in the chi^2 calculation. I find the discussion by P.H. Richter [2] >> to be quite good. >> >> Cheers, >> >> - Jonathan Helmus >> >> >> >> >> [1] http://seal.cern.ch/documents/minuit/mnusersguide.pdf >> [2] Estimating Errors in Least-Squares Fitting, P.H. Richter TDA Progress >> Report 42-122 >> http://tmo.jpl.nasa.gov/progress_report/42-122/122E.pdf >> >> >> On 08/07/2012 09:16 AM, Pawe? Kwa?niewski wrote: >> >> Hi, >> >> I'm fitting some data using a wrapper around the scipy.optimize.leastsq >> method which can be found under >> http://code.google.com/p/nmrglue/source/browse/trunk/nmrglue/analysis/leastsqbound.pyBasically it allows for putting bounds on the fitted parameters, which is >> very important for me. >> >> I'm using the covariance matrix, returned by leastsq() function to >> estimate the errors of the fitted parameters. The fitting is done using >> real measurement uncertainties (which are ridiculously small, by the way), >> so I would expect the resulting parameter error to be reasonable. What >> don't understand, is that I'm getting extremely small errors on the fitted >> parameters (I calculate the errors as perr = sqrt(diag(fitres[1])), where >> fitres[1] is the covariance matrix returned by leastsq() function). For >> example, a parameter which has a fitted value of ~100 gets an error of >> ~1e-6. At the same time, when I calculate the reduced chi squared of the >> fit I'm getting an extremely large number (of the order of 1e8). I can >> understand the large chi^2 value - the data variance is extremely small and >> the model curve is not perfect, so even slight deviations of the fitted >> model from the data will blow up chi^2 into space. But how can the fitted >> parameter variance be so small, while at the same time the fit is garbage >> according to chi^2? >> >> I guess this requires a better understanding of how the covariance matrix >> is calculated. Some suggestions anyone? >> >> Cheers, >> >> Pawe? >> >> >> _______________________________________________ >> SciPy-User mailing listSciPy-User at scipy.orghttp://mail.scipy.org/mailman/listinfo/scipy-user >> >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dlibyp at gmail.com Wed Aug 8 15:11:28 2012 From: dlibyp at gmail.com (dlibyp at gmail.com) Date: Wed, 08 Aug 2012 21:11:28 +0200 Subject: [SciPy-User] watershed Message-ID: <5022B9E0.2000708@gmail.com> Hi all, My question is a digest code question, so that I post on scipy-user, please tell me if I should no more do that. I would like to see source code for watershed algorithm. Code in scipy.ndimage.measurements for watershed is a wrapper for method watershed_ift in _nd_image. Module is imported in measurements.py, but I cannot find it in parent repository. Sorry for naive question: does _nd_image refers to C implementation of watershed_ift() method? Thanks. Rig From vanforeest at gmail.com Wed Aug 8 17:47:05 2012 From: vanforeest at gmail.com (nicky van foreest) Date: Wed, 8 Aug 2012 23:47:05 +0200 Subject: [SciPy-User] Sample queueing analysis In-Reply-To: <3BAA9108A76CF149A553864B23D47E13013A261B@fssbemail.fss.india> References: <3BAA9108A76CF149A553864B23D47E13013A261B@fssbemail.fss.india> Message-ID: Hi Mohammed You might be interested in some of python code for queueing analysis at my homepage Http://Nicky.vanforeest.com Nicky On Aug 7, 2012 7:13 AM, "Mohan Radhakrishnan" wrote: > Hi,**** > > ** ** > > I am trying to learn to draw graphs for performance models. As part of > this process I have a sample problem from a technical paper ?The > Operational Analysis of Queueing Network Models?**** > > ** ** > > There are 3 componens and their visit rations and service times.**** > > ** ** > > V1S1 = (20)(.05) = 1.00 seconds (Total CPU time *is the bottleneck* ) **** > > ** ** > > V2S2 = (11)(.08) = .88 seconds (Total Disk time)**** > > ** ** > > V3S3 = (8)(.04)= .32 seconds (Total Drum time)**** > > ** ** > > These products sum to the minimal response**** > > time of *2.2* seconds**** > > ** ** > > M1 = Z/VISI = 20/1 = 20 terminals**** > > ** ** > > ** ** > > Now I want to draw the graph by sending these values as inputs to a > queueing model. So if I plot terminals vs seconds the asymptote of the > response time curve should intersect**** > > The y-axis at 20 which is M1 give above.**** > > ** ** > > Is this type of work done using Sci-Py to solve a queueing model ? Can I > read some references ?**** > > ** ** > > Thanks,**** > > Mohan**** > > DISCLAIMER: > ========================================================================================================================================================== > > The information contained in this e-mail message may be privileged and/or > confidential and protected from disclosure under applicable law. It is > intended only for the individual to whom or entity to which it is addressed > as shown at the beginning of the message. If the reader of this message is > not the intended recipient, or if the employee or agent responsible for > delivering the message is not an employee or agent of the intended > recipient, you are hereby notified that any review, > dissemination,distribution, use, or copying of this message is strictly > prohibited. If you have received this message in error, please notify us > immediately by return e-mail and permanently delete this message and your > reply to the extent it includes this message. Any views or opinions > presented in this message or attachments are those of the author and do not > necessarily represent those of the Company. All e-mails and attachments > sent and received are subject to monitoring, reading, and archival by the > Company. > > ========================================================================================================================================================== > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alia_khouri at yahoo.com Wed Aug 8 20:32:15 2012 From: alia_khouri at yahoo.com (Alia) Date: Wed, 8 Aug 2012 17:32:15 -0700 (PDT) Subject: [SciPy-User] (no subject) Message-ID: <1344472335.75305.YahooMailNeo@web162104.mail.bf1.yahoo.com> http://izou33.com/01/site.php?perfect233.jpg -------------- next part -------------- An HTML attachment was scrubbed... URL: From j at abel.co Wed Aug 8 23:53:19 2012 From: j at abel.co (James Abel) Date: Wed, 8 Aug 2012 20:53:19 -0700 Subject: [SciPy-User] kmeans2 question/issue Message-ID: <001701cd75e2$844bef00$8ce3cd00$@co> Thanks Gael. BTW, I modified my code to loop until it gets the same clustering twice in a row. This yields more consistent results. I don't know if this is a general solution but it worked for my simple test case. Code below. James import sys import scipy import warnings from scipy.cluster.vq import * print sys.version vals = scipy.array((0.0,0.1,0.5,0.6,1.0,1.1)) print vals white_vals = whiten(vals) print white_vals.shape, white_vals # Check for same clustering def clustering_test(a,b): # have to create copies, then sort so we don't modify the original ea = a.copy() eb = b.copy() ea.sort() eb.sort() r = (ea == eb).all() print a,b,ea,eb,r return r # try it until we get the same clustering twice in a row found = False prior_idx = None while not found: with warnings.catch_warnings(): warnings.simplefilter("ignore") # suppress the warning message (happens if it doesn't find the right number of clusters) res, idx = kmeans2(white_vals, 3) # changing iter doesn't seem to matter #print res, idx if prior_idx is not None: eq = clustering_test(idx, prior_idx) #print eq.all() if eq: found = True prior_idx = idx print "result", res, idx -------------- next part -------------- An HTML attachment was scrubbed... URL: From nelle.varoquaux at gmail.com Thu Aug 9 00:06:53 2012 From: nelle.varoquaux at gmail.com (Nelle Varoquaux) Date: Wed, 8 Aug 2012 21:06:53 -0700 Subject: [SciPy-User] kmeans2 question/issue In-Reply-To: <001701cd75e2$844bef00$8ce3cd00$@co> References: <001701cd75e2$844bef00$8ce3cd00$@co> Message-ID: Hi James, Usually, we run the optimisation several times and take the solution with the smallest inertia. The technic you use don't ensure you to keep the best solution. There's a full implementation in scikit-learn with several runs. You can have a look at the code to see how it works. Cheers, N On 8 Aug 2012 20:53, "James Abel" wrote: > Thanks Gael.**** > > ** ** > > BTW, I modified my code to loop until it gets the same clustering twice in > a row. This yields more consistent results. I don?t know if this is a > general solution but it worked for my simple test case. Code below.**** > > ** ** > > James**** > > ** ** > > import sys**** > > import scipy**** > > import warnings**** > > from scipy.cluster.vq import ***** > > ** ** > > print sys.version**** > > vals = scipy.array((0.0,0.1,0.5,0.6,1.0,1.1))**** > > print vals**** > > white_vals = whiten(vals)**** > > print white_vals.shape, white_vals**** > > ** ** > > # Check for same clustering**** > > def clustering_test(a,b):**** > > # have to create copies, then sort so we don't modify the original**** > > ea = a.copy()**** > > eb = b.copy()**** > > ea.sort()**** > > eb.sort()**** > > r = (ea == eb).all()**** > > print a,b,ea,eb,r**** > > return r**** > > ** ** > > # try it until we get the same clustering twice in a row**** > > found = False**** > > prior_idx = None**** > > while not found:**** > > with warnings.catch_warnings():**** > > warnings.simplefilter("ignore") # suppress the warning message > (happens if it doesn't find the right number of clusters)**** > > res, idx = kmeans2(white_vals, 3) # changing iter doesn't seem to > matter**** > > #print res, idx**** > > if prior_idx is not None:**** > > eq = clustering_test(idx, prior_idx)**** > > #print eq.all()**** > > if eq:**** > > found = True**** > > prior_idx = idx**** > > print "result", res, idx**** > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mohanr at fss.co.in Thu Aug 9 02:32:36 2012 From: mohanr at fss.co.in (Mohan Radhakrishnan) Date: Thu, 9 Aug 2012 12:02:36 +0530 Subject: [SciPy-User] Sample queueing analysis In-Reply-To: Message-ID: <3BAA9108A76CF149A553864B23D47E130143142D@fssbemail.fss.india> Hi, I tried to read your page earlier but it is several steps ahead of me ! There may be simpler material. Thanks, Mohan ________________________________ From: scipy-user-bounces at scipy.org [mailto:scipy-user-bounces at scipy.org] On Behalf Of nicky van foreest Sent: Thursday, August 09, 2012 3:17 AM To: SciPy Users List Subject: Re: [SciPy-User] Sample queueing analysis Hi Mohammed You might be interested in some of python code for queueing analysis at my homepage Http://Nicky.vanforeest.com Nicky On Aug 7, 2012 7:13 AM, "Mohan Radhakrishnan" wrote: Hi, I am trying to learn to draw graphs for performance models. As part of this process I have a sample problem from a technical paper "The Operational Analysis of Queueing Network Models" There are 3 componens and their visit rations and service times. V1S1 = (20)(.05) = 1.00 seconds (Total CPU time is the bottleneck ) V2S2 = (11)(.08) = .88 seconds (Total Disk time) V3S3 = (8)(.04)= .32 seconds (Total Drum time) These products sum to the minimal response time of 2.2 seconds M1 = Z/VISI = 20/1 = 20 terminals Now I want to draw the graph by sending these values as inputs to a queueing model. So if I plot terminals vs seconds the asymptote of the response time curve should intersect The y-axis at 20 which is M1 give above. Is this type of work done using Sci-Py to solve a queueing model ? Can I read some references ? Thanks, Mohan DISCLAIMER: ======================================================================== ======================================================================== ========== The information contained in this e-mail message may be privileged and/or confidential and protected from disclosure under applicable law. It is intended only for the individual to whom or entity to which it is addressed as shown at the beginning of the message. If the reader of this message is not the intended recipient, or if the employee or agent responsible for delivering the message is not an employee or agent of the intended recipient, you are hereby notified that any review, dissemination,distribution, use, or copying of this message is strictly prohibited. If you have received this message in error, please notify us immediately by return e-mail and permanently delete this message and your reply to the extent it includes this message. Any views or opinions presented in this message or attachments are those of the author and do not necessarily represent those of the Company. All e-mails and attachments sent and received are subject to monitoring, reading, and archival by the Company. ======================================================================== ======================================================================== ========== _______________________________________________ SciPy-User mailing list SciPy-User at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user DISCLAIMER: ==========================================================================================================================================================The information contained in this e-mail message may be privileged and/or confidential and protected from disclosure under applicable law. It is intended only for the individual to whom or entity to which it is addressed as shown at the beginning of the message. If the reader of this message is not the intended recipient, or if the employee or agent responsible for delivering the message is not an employee or agent of the intended recipient, you are hereby notified that any review, dissemination,distribution, use, or copying of this message is strictly prohibited. If you have received this message in error, please notify us immediately by return e-mail and permanently delete this message and your reply to the extent it includes this message. Any views or opinions presented in this message or attachments are those of the author and do not necessarily represent those of the Company. All e-mails and attachments sent and received are subject to monitoring, reading, and archival by the Company.========================================================================================================================================================== -------------- next part -------------- An HTML attachment was scrubbed... URL: From pawel.kw at gmail.com Thu Aug 9 05:02:03 2012 From: pawel.kw at gmail.com (=?ISO-8859-2?Q?Pawe=B3_Kwa=B6niewski?=) Date: Thu, 9 Aug 2012 11:02:03 +0200 Subject: [SciPy-User] watershed In-Reply-To: <5022B9E0.2000708@gmail.com> References: <5022B9E0.2000708@gmail.com> Message-ID: Hi, I looked into the functions you're interested, although I'm not an expert but as far as I understand it - the watershed_ift() function is implemented in C, since the _nd_image module is a binary package. If you download the SciPy source code (http://sourceforge.net/projects/scipy/files/scipy/) you can find the C source of the function you're interested in the scipy/ndimage/src/ni_measure.c file. The function is called NI_WatershedIFT. I hope this helps, Pawe? 2012/8/8 dlibyp at gmail.com > Hi all, > My question is a digest code question, so that I post on scipy-user, > please tell me if I should no more do that. > I would like to see source code for watershed algorithm. Code in > scipy.ndimage.measurements for watershed is a wrapper for method > watershed_ift in _nd_image. Module is imported in measurements.py, but I > cannot find it in parent repository. > Sorry for naive question: does _nd_image refers to C implementation of > watershed_ift() method? > Thanks. > Rig > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cweisiger at msg.ucsf.edu Thu Aug 9 13:54:17 2012 From: cweisiger at msg.ucsf.edu (Chris Weisiger) Date: Thu, 9 Aug 2012 10:54:17 -0700 Subject: [SciPy-User] argsort TypeError Message-ID: I'm trying to find the indices in an array that have the lowest values, so that I can plot the corresponding values in another array. numpy.argsort would seem to be the proper approach here, but when I try it I get this error: Traceback (most recent call last): File "../scripts/findWorstFitPixel.py", line 48, in indices = np.argsort(result) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/fromnumeric.py", line 680, in argsort return argsort(axis, kind, order) TypeError: an integer is required The weird thing is that when I try to replicate the circumstances in the REPL (same argument shape/dtype passed to argsort), it succeeds. So there must be something subtle going on here. My input is a length-270336 1D array of float32s, each of which was generated by scipy.stats.pearsonr and almost all of which have values between .95 and 1. Any ideas what I can investigate to get this working? Unfortunately my script relies on reading a bunch of files, so I can't very well upload a working sample, but the script itself is here: http://pastebin.com/Rxcbyb8p Thanks for any assistance! -Chris From guziy.sasha at gmail.com Thu Aug 9 16:25:10 2012 From: guziy.sasha at gmail.com (Oleksandr Huziy) Date: Thu, 9 Aug 2012 16:25:10 -0400 Subject: [SciPy-User] argsort TypeError In-Reply-To: References: Message-ID: Hi, your problem is interesting, could you pickle the array result, like this we could play with it? But I can tell you one thing, I am not comfortable with the way you are reshaping arrays... Though it may be correct, try using reshape and copy the result, rather than inplace changing the arrays. Cheers -- Oleksandr (Sasha) Huziy 2012/8/9 Chris Weisiger > I'm trying to find the indices in an array that have the lowest > values, so that I can plot the corresponding values in another array. > numpy.argsort would seem to be the proper approach here, but when I > try it I get this error: > > Traceback (most recent call last): > File "../scripts/findWorstFitPixel.py", line 48, in > indices = np.argsort(result) > File > "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/fromnumeric.py", > line 680, in argsort > return argsort(axis, kind, order) > TypeError: an integer is required > > The weird thing is that when I try to replicate the circumstances in > the REPL (same argument shape/dtype passed to argsort), it succeeds. > So there must be something subtle going on here. My input is a > length-270336 1D array of float32s, each of which was generated by > scipy.stats.pearsonr and almost all of which have values between .95 > and 1. Any ideas what I can investigate to get this working? > Unfortunately my script relies on reading a bunch of files, so I can't > very well upload a working sample, but the script itself is here: > http://pastebin.com/Rxcbyb8p > > Thanks for any assistance! > > -Chris > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanforeest at gmail.com Thu Aug 9 17:15:45 2012 From: vanforeest at gmail.com (nicky van foreest) Date: Thu, 9 Aug 2012 23:15:45 +0200 Subject: [SciPy-User] odeint Message-ID: Hi, I am using odeint to integrate the following delay differential equation df/dx = c + f(x) - f(x-1) To solve it (numerically) I use the following iterative approach. The values of f(x) are given on the interval [0,1]. Then with odeint I can obtain f on [1,2]. Now that f in the interval [1,2] is known, I can use this to compute f on [2,3], etc. All in all this is workable, but somewhat cumbersome. Is it possible in odeint to somehow get intermediate values of the function values? Thanks Nicky From alia_khouri at yahoo.com Thu Aug 9 19:49:52 2012 From: alia_khouri at yahoo.com (Alia) Date: Thu, 9 Aug 2012 16:49:52 -0700 (PDT) Subject: [SciPy-User] (no subject) Message-ID: <1344556192.282.YahooMailNeo@web162103.mail.bf1.yahoo.com> http://interracialmobilesex.com/index10_files/goog.php?neighbor233.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.clewley at gmail.com Fri Aug 10 04:09:55 2012 From: rob.clewley at gmail.com (Rob Clewley) Date: Fri, 10 Aug 2012 01:09:55 -0700 Subject: [SciPy-User] odeint In-Reply-To: References: Message-ID: Hi, On Thu, Aug 9, 2012 at 2:15 PM, nicky van foreest wrote: > I am using odeint to integrate the following delay differential equation > > df/dx = c + f(x) - f(x-1) > > somewhat cumbersome. Is it possible in odeint to somehow get > intermediate values of the function values? Nicky, I think your easiest way ahead here is to use a spline to fit your previous intervals' solution function, building up your solution using one that meets values and first derivatives at the end points to retain smoothness as you go. Scipy offers several such spline tools. Then you can sample that to your heart's content. -- Robert Clewley, Ph.D. Assistant Professor Neuroscience Institute and Department of Mathematics and Statistics Georgia State University PO Box 5030 Atlanta, GA 30302, USA tel: 404-413-6420 fax: 404-413-5446 http://www2.gsu.edu/~matrhc http://neuroscience.gsu.edu/rclewley.html From matthew.brett at gmail.com Fri Aug 10 15:57:36 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 10 Aug 2012 12:57:36 -0700 Subject: [SciPy-User] loadmat and figure In-Reply-To: <1344191489.16423.20.camel@amilo.coursju> References: <87lihulku0.fsf@cica.cica> <1344096810.8004.13.camel@amilo.coursju> <87wr1dfz0z.fsf@cica.cica> <1344162571.16423.9.camel@amilo.coursju> <871ujlp9gk.fsf@cica.cica> <1344191489.16423.20.camel@amilo.coursju> Message-ID: Hi, On Sun, Aug 5, 2012 at 11:31 AM, Fabrice Silva wrote: > Le dimanche 05 ao?t 2012 ? 18:52 +0200, Petro a ?crit : >> Fabrice Silva writes: >> >> > Le dimanche 05 ao?t 2012 ? 11:50 +0200, x.piter at gmail.com a ?crit : >> > >> > Have you seen the note at the end of this page, concerning v7.3 format >> > not handled by scipy 0.7 (also neither by recent versions) ? >> > http://docs.scipy.org/doc/scipy-0.7.x/reference/generated/scipy.io.loadmat.html#scipy.io.loadmat >> Yes, I have seen it and tried to load different version of figures, none >> of them worked. Now I try to install scipy 10.1 instead of default 7th >> version, so far without success. > > Could you post a complete minimal example (with the file, and > traceback) ? An example would be very good. It looks like figure file have matlab functions inside, which we can't do much with, because the format is rather opaque. They should load OK though, in recent scipy versions. Did you succeed in installing scipy 10.1? Best, Matthew From dlibyp at gmail.com Fri Aug 10 22:54:09 2012 From: dlibyp at gmail.com (dlibyp at gmail.com) Date: Sat, 11 Aug 2012 04:54:09 +0200 Subject: [SciPy-User] watershed Message-ID: <5025C951.5090204@gmail.com> Thanks Pawel, This is what I was looking for. Rig Message: 2 Date: Thu, 9 Aug 2012 11:02:03 +0200 From: Pawe? Kwa?niewski Subject: Re: [SciPy-User] watershed To: SciPy Users List Message-ID: Content-Type: text/plain; charset="utf-8" Hi, I looked into the functions you're interested, although I'm not an expert but as far as I understand it - the watershed_ift() function is implemented in C, since the _nd_image module is a binary package. If you download the SciPy source code (http://sourceforge.net/projects/scipy/files/scipy/) you can find the C source of the function you're interested in the scipy/ndimage/src/ni_measure.c file. The function is called NI_WatershedIFT. I hope this helps, Pawe? 2012/8/8dlibyp at gmail.com > Hi all, > My question is a digest code question, so that I post on scipy-user, > please tell me if I should no more do that. > I would like to see source code for watershed algorithm. Code in > scipy.ndimage.measurements for watershed is a wrapper for method > watershed_ift in _nd_image. Module is imported in measurements.py, but I > cannot find it in parent repository. > Sorry for naive question: does _nd_image refers to C implementation of > watershed_ift() method? > Thanks. > Rig > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From j at abel.co Sat Aug 11 14:19:04 2012 From: j at abel.co (James Abel) Date: Sat, 11 Aug 2012 11:19:04 -0700 Subject: [SciPy-User] kmeans2 question/issue Message-ID: <002701cd77ed$ca9d66c0$5fd83440$@co> Thanks - sklearn works great! I got exactly what I expected each time I ran it! James import sys import numpy from sklearn.cluster import * print sys.version vals = numpy.array([[0.0],[0.1],[0.5],[0.6],[1.0],[1.1]]) print vals k_means_ex = KMeans(k=3) x = k_means_ex.fit_predict(vals) print x print k_means_ex.cluster_centers_ print k_means_ex.score(vals) 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] [[ 0. ] [ 0.1] [ 0.5] [ 0.6] [ 1. ] [ 1.1]] [1 1 0 0 2 2] [[ 0.55] [ 0.05] [ 1.05]] -0.015 -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlossbaptista at gmail.com Sun Aug 12 09:49:43 2012 From: carlossbaptista at gmail.com (Carlos Baptista) Date: Sun, 12 Aug 2012 15:49:43 +0200 Subject: [SciPy-User] Disable singular matrix warning Message-ID: Hello there, I am fairly new to Python. As a self-test I re-programmed an old MATLAB assignment in Python using NumPy, SciPy and Matplotlib. At a certain point I am getting the following warning: warning: (almost) singular matrix! (estimated cond. number: 1.87e+14) This result is however expected and therefore I would like to disable this warning. I am getting this warning from: scipy.sparse.linalg.spsolve(A,b) where A has a large condition number. I have already tried the following to no avail: ======================== import warnings with warnings.catch_warnings(): warnings.filterwarnings("ignore",category=Warning) from poisson import solvePoisson ======================== where solvePoisson is the function (located in poisson.py) in which the above solve statement is executed. How can I disable this singular matrix warning? With kind regards, Carlos -------------- next part -------------- An HTML attachment was scrubbed... URL: From helmrp at yahoo.com Sun Aug 12 11:47:41 2012 From: helmrp at yahoo.com (The Helmbolds) Date: Sun, 12 Aug 2012 08:47:41 -0700 (PDT) Subject: [SciPy-User] brentq and brenth Message-ID: <1344786461.52280.YahooMailNeo@web31811.mail.mud.yahoo.com> It would seem more logical to call brentq "brentp" ("p" for parabola) to follow the same pattern as "brenth" ("h" for hyperbola). ? ? Unless there is an inside joke to use "q" because it is "p" reflected about a vertical axis, and hence represents?Brent's "inverse parabolic" estimate of the root? -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanforeest at gmail.com Sun Aug 12 15:49:25 2012 From: vanforeest at gmail.com (nicky van foreest) Date: Sun, 12 Aug 2012 21:49:25 +0200 Subject: [SciPy-User] odeint In-Reply-To: References: Message-ID: Hi Rob, Thanks for the suggestion. I didn't think of using splines. Nicky On 10 August 2012 10:09, Rob Clewley wrote: > Hi, > > On Thu, Aug 9, 2012 at 2:15 PM, nicky van foreest wrote: >> I am using odeint to integrate the following delay differential equation >> >> df/dx = c + f(x) - f(x-1) >> >> somewhat cumbersome. Is it possible in odeint to somehow get >> intermediate values of the function values? > > Nicky, I think your easiest way ahead here is to use a spline to fit > your previous intervals' solution function, building up your solution > using one that meets values and first derivatives at the end points to > retain smoothness as you go. Scipy offers several such spline tools. > Then you can sample that to your heart's content. > > -- > Robert Clewley, Ph.D. > Assistant Professor > Neuroscience Institute and > Department of Mathematics and Statistics > Georgia State University > PO Box 5030 > Atlanta, GA 30302, USA > > tel: 404-413-6420 fax: 404-413-5446 > http://www2.gsu.edu/~matrhc > http://neuroscience.gsu.edu/rclewley.html > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From pawel.kw at gmail.com Sun Aug 12 17:47:58 2012 From: pawel.kw at gmail.com (=?ISO-8859-2?Q?Pawe=B3_Kwa=B6niewski?=) Date: Sun, 12 Aug 2012 23:47:58 +0200 Subject: [SciPy-User] Disable singular matrix warning In-Reply-To: References: Message-ID: Hi, Have you tried the simplest example from http://docs.python.org/library/warnings.html import warnings with warnings.catch_warnings(): warnings.simplefilter("ignore") And then your function call? Cheers, Pawe? 2012/8/12 Carlos Baptista > Hello there, > > I am fairly new to Python. As a self-test I re-programmed an old MATLAB > assignment in Python using NumPy, SciPy and Matplotlib. At a certain point > I am getting the following warning: > > warning: (almost) singular matrix! (estimated cond. number: 1.87e+14) > > This result is however expected and therefore I would like to disable this > warning. I am getting this warning from: > > scipy.sparse.linalg.spsolve(A,b) > > where A has a large condition number. > > I have already tried the following to no avail: > > ======================== > > import warnings > > > with warnings.catch_warnings(): > > warnings.filterwarnings("ignore",category=Warning) > > from poisson import solvePoisson > > > ======================== > > > where solvePoisson is the function (located in poisson.py) in which the > above solve statement is executed. > > > How can I disable this singular matrix warning? > > > > With kind regards, > > Carlos > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlossbaptista at gmail.com Sun Aug 12 22:06:13 2012 From: carlossbaptista at gmail.com (Carlos Baptista) Date: Mon, 13 Aug 2012 04:06:13 +0200 Subject: [SciPy-User] Disable singular matrix warning In-Reply-To: References: Message-ID: I had not tried it before. But I did it just now and it did not work unfortunately. On 12 August 2012 23:47, Pawe? Kwa?niewski wrote: > Hi, > > Have you tried the simplest example from > http://docs.python.org/library/warnings.html > > import warnings > > > with warnings.catch_warnings(): > warnings.simplefilter("ignore") > > And then your function call? > > Cheers, > > Pawe? > > > 2012/8/12 Carlos Baptista > >> Hello there, >> >> I am fairly new to Python. As a self-test I re-programmed an old MATLAB >> assignment in Python using NumPy, SciPy and Matplotlib. At a certain point >> I am getting the following warning: >> >> warning: (almost) singular matrix! (estimated cond. number: 1.87e+14) >> >> This result is however expected and therefore I would like to disable >> this warning. I am getting this warning from: >> >> scipy.sparse.linalg.spsolve(A,b) >> >> where A has a large condition number. >> >> I have already tried the following to no avail: >> >> ======================== >> >> import warnings >> >> >> with warnings.catch_warnings(): >> >> warnings.filterwarnings("ignore",category=Warning) >> >> from poisson import solvePoisson >> >> >> ======================== >> >> >> where solvePoisson is the function (located in poisson.py) in which the >> above solve statement is executed. >> >> >> How can I disable this singular matrix warning? >> >> >> >> With kind regards, >> >> Carlos >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- met de vriendelijke groeten van / with kind regards, Carlos Fernando Baptista -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Aug 13 02:15:28 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 13 Aug 2012 08:15:28 +0200 Subject: [SciPy-User] Disable singular matrix warning In-Reply-To: References: Message-ID: On Mon, Aug 13, 2012 at 4:06 AM, Carlos Baptista wrote: > I had not tried it before. But I did it just now and it did not work > unfortunately. > > That's because sparse/linalg/dsolve/umfpack/umfpack.py contains: if econd > self.maxCond: print 'warning: (almost) singular matrix! '\ + '(estimated cond. number: %.2e)' % econd We have to change that to use the warnings module. For now all you can do is comment out those 3 lines in your installed version of scipy. Ralf > On 12 August 2012 23:47, Pawe? Kwa?niewski wrote: > >> Hi, >> >> Have you tried the simplest example from >> http://docs.python.org/library/warnings.html >> >> import warnings >> >> >> with warnings.catch_warnings(): >> warnings.simplefilter("ignore") >> >> And then your function call? >> >> Cheers, >> >> Pawe? >> >> >> 2012/8/12 Carlos Baptista >> >>> Hello there, >>> >>> I am fairly new to Python. As a self-test I re-programmed an old MATLAB >>> assignment in Python using NumPy, SciPy and Matplotlib. At a certain point >>> I am getting the following warning: >>> >>> warning: (almost) singular matrix! (estimated cond. number: 1.87e+14) >>> >>> This result is however expected and therefore I would like to disable >>> this warning. I am getting this warning from: >>> >>> scipy.sparse.linalg.spsolve(A,b) >>> >>> where A has a large condition number. >>> >>> I have already tried the following to no avail: >>> >>> ======================== >>> >>> import warnings >>> >>> >>> with warnings.catch_warnings(): >>> >>> warnings.filterwarnings("ignore",category=Warning) >>> >>> from poisson import solvePoisson >>> >>> >>> ======================== >>> >>> >>> where solvePoisson is the function (located in poisson.py) in which the >>> above solve statement is executed. >>> >>> >>> How can I disable this singular matrix warning? >>> >>> >>> >>> With kind regards, >>> >>> Carlos >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > > -- > met de vriendelijke groeten van / with kind regards, > > Carlos Fernando Baptista > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlossbaptista at gmail.com Mon Aug 13 08:33:16 2012 From: carlossbaptista at gmail.com (Carlos Baptista) Date: Mon, 13 Aug 2012 14:33:16 +0200 Subject: [SciPy-User] Disable singular matrix warning In-Reply-To: References: Message-ID: Thank you very much Ralf. That did the trick! On 13 August 2012 08:15, Ralf Gommers wrote: > > > On Mon, Aug 13, 2012 at 4:06 AM, Carlos Baptista < > carlossbaptista at gmail.com> wrote: > >> I had not tried it before. But I did it just now and it did not work >> unfortunately. >> >> That's because sparse/linalg/dsolve/umfpack/umfpack.py contains: > > if econd > self.maxCond: > print 'warning: (almost) singular matrix! '\ > + '(estimated cond. number: %.2e)' % econd > > We have to change that to use the warnings module. For now all you can do > is comment out those 3 lines in your installed version of scipy. > > Ralf > > >> On 12 August 2012 23:47, Pawe? Kwa?niewski wrote: >> >>> Hi, >>> >>> Have you tried the simplest example from >>> http://docs.python.org/library/warnings.html >>> >>> import warnings >>> >>> >>> with warnings.catch_warnings(): >>> warnings.simplefilter("ignore") >>> >>> And then your function call? >>> >>> Cheers, >>> >>> Pawe? >>> >>> >>> 2012/8/12 Carlos Baptista >>> >>>> Hello there, >>>> >>>> I am fairly new to Python. As a self-test I re-programmed an old MATLAB >>>> assignment in Python using NumPy, SciPy and Matplotlib. At a certain point >>>> I am getting the following warning: >>>> >>>> warning: (almost) singular matrix! (estimated cond. number: 1.87e+14) >>>> >>>> This result is however expected and therefore I would like to disable >>>> this warning. I am getting this warning from: >>>> >>>> scipy.sparse.linalg.spsolve(A,b) >>>> >>>> where A has a large condition number. >>>> >>>> I have already tried the following to no avail: >>>> >>>> ======================== >>>> >>>> import warnings >>>> >>>> >>>> with warnings.catch_warnings(): >>>> >>>> warnings.filterwarnings("ignore",category=Warning) >>>> >>>> from poisson import solvePoisson >>>> >>>> >>>> ======================== >>>> >>>> >>>> where solvePoisson is the function (located in poisson.py) in which the >>>> above solve statement is executed. >>>> >>>> >>>> How can I disable this singular matrix warning? >>>> >>>> >>>> >>>> With kind regards, >>>> >>>> Carlos >>>> >>>> _______________________________________________ >>>> SciPy-User mailing list >>>> SciPy-User at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/scipy-user >>>> >>>> >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> >> >> -- >> met de vriendelijke groeten van / with kind regards, >> >> Carlos Fernando Baptista >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- met de vriendelijke groeten van / with kind regards, Carlos Fernando Baptista -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Aug 13 14:30:44 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 13 Aug 2012 20:30:44 +0200 Subject: [SciPy-User] ANN: SciPy 0.11.0 release candidate 2 Message-ID: Hi, I am pleased to announce the availability of the second release candidate of SciPy 0.11.0. For this release many new features have been added, and over 120 tickets and pull requests have been closed. Also noteworthy is that the number of contributors for this release has risen to over 50. Some of the highlights are: - A new module, sparse.csgraph, has been added which provides a number of common sparse graph algorithms. - New unified interfaces to the existing optimization and root finding functions have been added. Sources and binaries can be found at http://sourceforge.net/projects/scipy/files/scipy/0.11.0rc2/, release notes are copied below. For this release candidate all known issues (with the exception of one Qhull issue on Debian, s390x platform) have been solved. In the meantime also OS X 10.8 was released, this RC contains a few build fixes for that platform. If no more serious issues are reported, the final release will be in one week. Cheers, Ralf ========================== SciPy 0.11.0 Release Notes ========================== .. note:: Scipy 0.11.0 is not released yet! .. contents:: SciPy 0.11.0 is the culmination of 8 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation. Highlights of this release are: - A new module has been added which provides a number of common sparse graph algorithms. - New unified interfaces to the existing optimization and root finding functions have been added. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Our development attention will now shift to bug-fix releases on the 0.11.x branch, and on adding new features on the master branch. This release requires Python 2.4-2.7 or 3.1-3.2 and NumPy 1.5.1 or greater. New features ============ Sparse Graph Submodule ---------------------- The new submodule :mod:`scipy.sparse.csgraph` implements a number of efficient graph algorithms for graphs stored as sparse adjacency matrices. Available routines are: - :func:`connected_components` - determine connected components of a graph - :func:`laplacian` - compute the laplacian of a graph - :func:`shortest_path` - compute the shortest path between points on a positive graph - :func:`dijkstra` - use Dijkstra's algorithm for shortest path - :func:`floyd_warshall` - use the Floyd-Warshall algorithm for shortest path - :func:`breadth_first_order` - compute a breadth-first order of nodes - :func:`depth_first_order` - compute a depth-first order of nodes - :func:`breadth_first_tree` - construct the breadth-first tree from a given node - :func:`depth_first_tree` - construct a depth-first tree from a given node - :func:`minimum_spanning_tree` - construct the minimum spanning tree of a graph ``scipy.optimize`` improvements ------------------------------- The optimize module has received a lot of attention this release. In addition to added tests, documentation improvements, bug fixes and code clean-up, the following improvements were made: - A unified interface to minimizers of univariate and multivariate functions has been added. - A unified interface to root finding algorithms for multivariate functions has been added. - The L-BFGS-B algorithm has been updated to version 3.0. Unified interfaces to minimizers ```````````````````````````````` Two new functions ``scipy.optimize.minimize`` and ``scipy.optimize.minimize_scalar`` were added to provide a common interface to minimizers of multivariate and univariate functions respectively. For multivariate functions, ``scipy.optimize.minimize`` provides an interface to methods for unconstrained optimization (`fmin`, `fmin_powell`, `fmin_cg`, `fmin_ncg`, `fmin_bfgs` and `anneal`) or constrained optimization (`fmin_l_bfgs_b`, `fmin_tnc`, `fmin_cobyla` and `fmin_slsqp`). For univariate functions, ``scipy.optimize.minimize_scalar`` provides an interface to methods for unconstrained and bounded optimization (`brent`, `golden`, `fminbound`). This allows for easier comparing and switching between solvers. Unified interface to root finding algorithms ```````````````````````````````````````````` The new function ``scipy.optimize.root`` provides a common interface to root finding algorithms for multivariate functions, embeding `fsolve`, `leastsq` and `nonlin` solvers. ``scipy.linalg`` improvements ----------------------------- New matrix equation solvers ``````````````````````````` Solvers for the Sylvester equation (``scipy.linalg.solve_sylvester``, discrete and continuous Lyapunov equations (``scipy.linalg.solve_lyapunov``, ``scipy.linalg.solve_discrete_lyapunov``) and discrete and continuous algebraic Riccati equations (``scipy.linalg.solve_continuous_are``, ``scipy.linalg.solve_discrete_are``) have been added to ``scipy.linalg``. These solvers are often used in the field of linear control theory. QZ and QR Decomposition ```````````````````````` It is now possible to calculate the QZ, or Generalized Schur, decomposition using ``scipy.linalg.qz``. This function wraps the LAPACK routines sgges, dgges, cgges, and zgges. The function ``scipy.linalg.qr_multiply``, which allows efficient computation of the matrix product of Q (from a QR decompostion) and a vector, has been added. Pascal matrices ``````````````` A function for creating Pascal matrices, ``scipy.linalg.pascal``, was added. Sparse matrix construction and operations ----------------------------------------- Two new functions, ``scipy.sparse.diags`` and ``scipy.sparse.block_diag``, were added to easily construct diagonal and block-diagonal sparse matrices respectively. ``scipy.sparse.csc_matrix`` and ``csr_matrix`` now support the operations ``sin``, ``tan``, ``arcsin``, ``arctan``, ``sinh``, ``tanh``, ``arcsinh``, ``arctanh``, ``rint``, ``sign``, ``expm1``, ``log1p``, ``deg2rad``, ``rad2deg``, ``floor``, ``ceil`` and ``trunc``. Previously, these operations had to be performed by operating on the matrices' ``data`` attribute. LSMR iterative solver --------------------- LSMR, an iterative method for solving (sparse) linear and linear least-squares systems, was added as ``scipy.sparse.linalg.lsmr``. Discrete Sine Transform ----------------------- Bindings for the discrete sine transform functions have been added to ``scipy.fftpack``. ``scipy.interpolate`` improvements ---------------------------------- For interpolation in spherical coordinates, the three classes ``scipy.interpolate.SmoothSphereBivariateSpline``, ``scipy.interpolate.LSQSphereBivariateSpline``, and ``scipy.interpolate.RectSphereBivariateSpline`` have been added. Binned statistics (``scipy.stats``) ----------------------------------- The stats module has gained functions to do binned statistics, which are a generalization of histograms, in 1-D, 2-D and multiple dimensions: ``scipy.stats.binned_statistic``, ``scipy.stats.binned_statistic_2d`` and ``scipy.stats.binned_statistic_dd``. Deprecated features =================== ``scipy.sparse.cs_graph_components`` has been made a part of the sparse graph submodule, and renamed to ``scipy.sparse.csgraph.connected_components``. Calling the former routine will result in a deprecation warning. ``scipy.misc.radon`` has been deprecated. A more full-featured radon transform can be found in scikits-image. ``scipy.io.save_as_module`` has been deprecated. A better way to save multiple Numpy arrays is the ``numpy.savez`` function. The `xa` and `xb` parameters for all distributions in ``scipy.stats.distributions`` already weren't used; they have now been deprecated. Backwards incompatible changes ============================== Removal of ``scipy.maxentropy`` ------------------------------- The ``scipy.maxentropy`` module, which was deprecated in the 0.10.0 release, has been removed. Logistic regression in scikits.learn is a good and modern alternative for this functionality. Minor change in behavior of ``splev`` ------------------------------------- The spline evaluation function now behaves similarly to ``interp1d`` for size-1 arrays. Previous behavior:: >>> from scipy.interpolate import splev, splrep, interp1d >>> x = [1,2,3,4,5] >>> y = [4,5,6,7,8] >>> tck = splrep(x, y) >>> splev([1], tck) 4. >>> splev(1, tck) 4. Corrected behavior:: >>> splev([1], tck) array([ 4.]) >>> splev(1, tck) array(4.) This affects also the ``UnivariateSpline`` classes. Behavior of ``scipy.integrate.complex_ode`` ------------------------------------------- The behavior of the ``y`` attribute of ``complex_ode`` is changed. Previously, it expressed the complex-valued solution in the form:: z = ode.y[::2] + 1j * ode.y[1::2] Now, it is directly the complex-valued solution:: z = ode.y Minor change in behavior of T-tests ----------------------------------- The T-tests ``scipy.stats.ttest_ind``, ``scipy.stats.ttest_rel`` and ``scipy.stats.ttest_1samp`` have been changed so that 0 / 0 now returns NaN instead of 1. Other changes ============= The SuperLU sources in ``scipy.sparse.linalg`` have been updated to version 4.3 from upstream. The function ``scipy.signal.bode``, which calculates magnitude and phase data for a continuous-time system, has been added. The two-sample T-test ``scipy.stats.ttest_ind`` gained an option to compare samples with unequal variances, i.e. Welch's T-test. ``scipy.misc.logsumexp`` now takes an optional ``axis`` keyword argument. Authors ======= This release contains work by the following people (contributed at least one patch to this release, names in alphabetical order): * Jeff Armstrong * Chad Baker * Brandon Beacher + * behrisch + * borishim + * Matthew Brett * Lars Buitinck * Luis Pedro Coelho + * Johann Cohen-Tanugi * David Cournapeau * dougal + * Ali Ebrahim + * endolith + * Bj?rn Forsman + * Robert Gantner + * Sebastian Gassner + * Christoph Gohlke * Ralf Gommers * Yaroslav Halchenko * Charles Harris * Jonathan Helmus + * Andreas Hilboll + * Marc Honnorat + * Jonathan Hunt + * Maxim Ivanov + * Thouis (Ray) Jones * Christopher Kuster + * Josh Lawrence + * Denis Laxalde + * Travis Oliphant * Joonas Paalasmaa + * Fabian Pedregosa * Josef Perktold * Gavin Price + * Jim Radford + * Andrew Schein + * Skipper Seabold * Jacob Silterra + * Scott Sinclair * Alexis Tabary + * Martin Teichmann * Matt Terry + * Nicky van Foreest + * Jacob Vanderplas * Patrick Varilly + * Pauli Virtanen * Nils Wagner + * Darryl Wally + * Stefan van der Walt * Liming Wang + * David Warde-Farley + * Warren Weckesser * Sebastian Werk + * Mike Wimmer + * Tony S Yu + A total of 55 people contributed to this release. People with a "+" by their names contributed a patch for the first time. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hasslerjc at comcast.net Mon Aug 13 14:52:27 2012 From: hasslerjc at comcast.net (John Hassler) Date: Mon, 13 Aug 2012 14:52:27 -0400 Subject: [SciPy-User] ANN: SciPy 0.11.0 release candidate 2 In-Reply-To: References: Message-ID: <50294CEB.4070103@comcast.net> An HTML attachment was scrubbed... URL: From jjhelmus at gmail.com Mon Aug 13 15:01:05 2012 From: jjhelmus at gmail.com (Jonathan Helmus) Date: Mon, 13 Aug 2012 15:01:05 -0400 Subject: [SciPy-User] Question about errors (uncertainties) in non-linear least squares fitting In-Reply-To: References: <5021256F.6040606@gmail.com> Message-ID: <50294EF1.50309@gmail.com> Pawel, leastsqbound cannot fix a parameter. If you want a fixed parameter in the fit you can either rewrite your fitting/error function to take the fixed parameters as an extra argument, or use a more "full featured" constrained least squares fitting package like lmfit (http://newville.github.com/lmfit-py/) whose class based Parameters allow for additional control. For any aspiring developers out there, fixed parameter could be added to leastsqbound (although it wouldn't be easy) but I'm not planning on add it in the near term. I would accept a patch which added this. - Jonathan Helmus On 08/07/2012 05:31 PM, Pawe? Kwas'niewski wrote: > Dear Jonathan, > > Thank you for the quick answer and pointing me to the updated code and > the literature. I'll certainly have a look into it. Just for the > record, I wanted to use constrained fitting because the unconstrained > fit sometimes gave me unphysical parameters. > > It's interesting what you wrote about how the constraint influences > the parameter error estimate. Just to give it a shot I tried an > unconstrained fit on one data set. It did converge, but still - I got > enormous chi^2 (reduced, and yes - I'm using the experimental errors > to calculate it) with ridiculously small parameter errors... I guess I > need to understand better what's going on. > > By the way, the constrained fitting function is great, but I'd like to > add another feature - the possibility to fix a parameter. This would > largely facilitate testing of the fitting function, or can even be > useful in real applications, where one of the parameters is known from > a previous fit of a slightly different model or a different > measurement or whatever... Anyway - maybe you have any ideas how to > implement that? I already tried setting the upper and lower bound to > the same value - it doesn't work. The method you use to apply the > bounds throws division by zero error. > > Cheers, > > Pawe? > > > 2012/8/7 Jonathan Helmus > > > Pawel, > > First off you may want to use a more up to date version of > leastsqbound which can be found at > https://github.com/jjhelmus/leastsqbound-scipy > > Second, when you perform a constrained optimization using > internal parameters like leastsqbound does, > if one of more of the parameters is close to a bound, the values > in the covariance matrix can take on meaningless values. Section > 1.3 of the The Minuit User's Guide [1] gives a good overview of > this, especially look at the discussion on page 5. For best > results an unconstrained optimization should be performed, often > times you can rewrite your model in such a way that the > constraints are automatically imposed (this is what is done > internally in leastsqbound, but transforming back to the original > model can introduce large errors if a parameter is close to the > bounds). > > Third, since you have measurement uncertainties make sure you > include them in the chi^2 calculation. I find the discussion by > P.H. Richter [2] to be quite good. > > Cheers, > > - Jonathan Helmus > > > > > [1] http://seal.cern.ch/documents/minuit/mnusersguide.pdf > [2] Estimating Errors in Least-Squares Fitting, P.H. Richter TDA > Progress Report 42-122 > http://tmo.jpl.nasa.gov/progress_report/42-122/122E.pdf > > > On 08/07/2012 09:16 AM, Pawe? Kwas'niewski wrote: >> Hi, >> >> I'm fitting some data using a wrapper around the >> scipy.optimize.leastsq method which can be found under >> http://code.google.com/p/nmrglue/source/browse/trunk/nmrglue/analysis/leastsqbound.py >> Basically it allows for putting bounds on the fitted parameters, >> which is very important for me. >> >> I'm using the covariance matrix, returned by leastsq() function >> to estimate the errors of the fitted parameters. The fitting is >> done using real measurement uncertainties (which are ridiculously >> small, by the way), so I would expect the resulting parameter >> error to be reasonable. What don't understand, is that I'm >> getting extremely small errors on the fitted parameters (I >> calculate the errors as perr = sqrt(diag(fitres[1])), where >> fitres[1] is the covariance matrix returned by leastsq() >> function). For example, a parameter which has a fitted value of >> ~100 gets an error of ~1e-6. At the same time, when I calculate >> the reduced chi squared of the fit I'm getting an extremely large >> number (of the order of 1e8). I can understand the large chi^2 >> value - the data variance is extremely small and the model curve >> is not perfect, so even slight deviations of the fitted model >> from the data will blow up chi^2 into space. But how can the >> fitted parameter variance be so small, while at the same time the >> fit is garbage according to chi^2? >> >> I guess this requires a better understanding of how the >> covariance matrix is calculated. Some suggestions anyone? >> >> Cheers, >> >> Pawe? >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnl at cs.wisc.edu Mon Aug 13 16:49:35 2012 From: johnl at cs.wisc.edu (J. David Lee) Date: Mon, 13 Aug 2012 15:49:35 -0500 Subject: [SciPy-User] optimization with multiple processors. Message-ID: <5029685F.70001@cs.wisc.edu> Hello everyone. For my work I will need to perform an optimization using an objective function that is slow to evaluate (a few to tens of minutes). I have access to a virtually unlimited number of processors for my work to speed this up, but the optimization routines in scipy are all single threaded. I see two simple ways to utilize multiple processors for newton-like or gradient-descent optimization methods: 1) numerical gradient or Jacobian calculation 2) line search To some extent, parallel processing could be applied to a simplex-like method as well (the simplest example would be doing parallel computation of reflection, extension and contraction points). I think this functionality would fit perfectly in scipy, and I will be working on it in any event, so if anyone would be interested in collaborating or providing guidance, please let me know. Thanks, David From ralf.gommers at gmail.com Mon Aug 13 17:17:04 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 13 Aug 2012 23:17:04 +0200 Subject: [SciPy-User] ANN: SciPy 0.11.0 release candidate 2 In-Reply-To: <50294CEB.4070103@comcast.net> References: <50294CEB.4070103@comcast.net> Message-ID: On Mon, Aug 13, 2012 at 8:52 PM, John Hassler wrote: > > On 8/13/2012 2:30 PM, Ralf Gommers wrote: > > Hi, > > I am pleased to announce the availability of the second release candidate > of SciPy 0.11.0. For this release many new features have been added, and > over 120 tickets and pull requests have been closed. Also noteworthy is > that the number of contributors for this release has risen to over 50. Some > of the highlights are: > > - A new module, sparse.csgraph, has been added which provides a number > of common sparse graph algorithms. > - New unified interfaces to the existing optimization and root finding > functions have been added. > > Sources and binaries can be found at > http://sourceforge.net/projects/scipy/files/scipy/0.11.0rc2/, release > notes are copied below. > > For this release candidate all known issues (with the exception of one > Qhull issue on Debian, s390x platform) have been solved. In the meantime > also OS X 10.8 was released, this RC contains a few build fixes for that > platform. > > If no more serious issues are reported, the final release will be in one > week. > > Cheers, > Ralf > > The diagnostics all run: one warning, one error, one failure. > > Thanks John. Is that test_singular failure repeatable? You didn't report that as failing for RC1. Ralf > > > Windows XP SP 3 > Intel Core 2 Quad Q6600 > 3 GB RAM > > Python 3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit (Intel)] > on win32 > Type "copyright", "credits" or "license()" for more information. > >>> import scipy > >>> scipy.test() > Running unit tests for scipy > NumPy version 1.6.2 > NumPy is installed in C:\Python32\lib\site-packages\numpy > SciPy version 0.11.0rc2 > SciPy is installed in C:\Python32\lib\site-packages\scipy > Python version 3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit > (Intel)] > nose version 1.0.0 > - - - - - - - - > ....................................................................... > Warning (from warnings module): > File "C:\Python32\lib\site-packages\scipy\special\tests\test_basic.py", > line 1606 > assert_(np.abs(c2) >= 1e300, (v, z)) > RuntimeWarning: invalid value encountered in absolute > .........................K.K.............................................................................................................................................................................................................................................................................................................................................................................................K........K..............SSSSSSS............................................................................................................................................................................S......................................................................................................................................................................................................................................................................................................................................................................................... > ....... > ..........................................................................................................................E > ====================================================================== > ERROR: Failure: ImportError (scipy.weave only supports Python 2.x) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "C:\Python32\lib\site-packages\nose-1.0.0-py3.2.egg\nose\failure.py", line > 37, in runTest > raise self.exc_class(self.exc_val).with_traceback(self.tb) > File > "C:\Python32\lib\site-packages\nose-1.0.0-py3.2.egg\nose\loader.py", line > 390, in loadTestsFromName > addr.filename, addr.module) > File > "C:\Python32\lib\site-packages\nose-1.0.0-py3.2.egg\nose\importer.py", line > 39, in importFromPath > return self.importFromDir(dir_path, fqname) > File > "C:\Python32\lib\site-packages\nose-1.0.0-py3.2.egg\nose\importer.py", line > 86, in importFromDir > mod = load_module(part_fqname, fh, filename, desc) > File "C:\Python32\lib\site-packages\scipy\weave\__init__.py", line 22, > in > raise ImportError("scipy.weave only supports Python 2.x") > ImportError: scipy.weave only supports Python 2.x > > ====================================================================== > FAIL: test_decomp.TestEig.test_singular > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python32\lib\site-packages\nose-1.0.0-py3.2.egg\nose\case.py", > line 188, in runTest > self.test(*self.arg) > File "C:\Python32\lib\site-packages\scipy\linalg\tests\test_decomp.py", > line 201, in test_singular > self._check_gen_eig(A, B) > File "C:\Python32\lib\site-packages\scipy\linalg\tests\test_decomp.py", > line 188, in _check_gen_eig > err_msg=msg) > File "C:\Python32\lib\site-packages\numpy\testing\utils.py", line 800, > in assert_array_almost_equal > header=('Arrays are not almost equal to %d decimals' % decimal)) > File "C:\Python32\lib\site-packages\numpy\testing\utils.py", line 636, > in assert_array_compare > raise AssertionError(msg) > AssertionError: > Arrays are not almost equal to 6 decimals > > array([[22, 34, 31, 31, 17], > [45, 45, 42, 19, 29], > [39, 47, 49, 26, 34], > [27, 31, 26, 21, 15], > [38, 44, 44, 24, 30]]) > array([[13, 26, 25, 17, 24], > [31, 46, 40, 26, 37], > [26, 40, 19, 25, 25], > [16, 25, 27, 14, 23], > [24, 35, 18, 21, 22]]) > (mismatch 25.0%) > x: array([ -2.45037885e-01 +0.00000000e+00j, > 5.17637463e-16 -4.01120590e-08j, > 5.17637463e-16 +4.01120590e-08j, 2.00000000e+00 > +0.00000000e+00j]) > y: array([ -3.74550285e-01 +0.00000000e+00j, > -5.17716907e-17 -1.15230800e-08j, > -5.17716907e-17 +1.15230800e-08j, 2.00000000e+00 > +0.00000000e+00j]) > > ---------------------------------------------------------------------- > Ran 5351 tests in 107.672s > > FAILED (KNOWNFAIL=14, SKIP=42, errors=1, failures=1) > > >>> > > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Mon Aug 13 18:09:02 2012 From: cgohlke at uci.edu (Christoph Gohlke) Date: Mon, 13 Aug 2012 15:09:02 -0700 Subject: [SciPy-User] ANN: SciPy 0.11.0 release candidate 2 In-Reply-To: References: Message-ID: <50297AFE.3020806@uci.edu> On 8/13/2012 11:30 AM, Ralf Gommers wrote: > Hi, > > I am pleased to announce the availability of the second release > candidate of SciPy 0.11.0. For this release many new features have been > added, and over 120 tickets and pull requests have been closed. Also > noteworthy is that the number of contributors for this release has risen > to over 50. Some of the highlights are: > > - A new module, sparse.csgraph, has been added which provides a > number of common sparse graph algorithms. > - New unified interfaces to the existing optimization and root > finding functions have been added. > > Sources and binaries can be found at > http://sourceforge.net/projects/scipy/files/scipy/0.11.0rc2/, release > notes are copied below. > > For this release candidate all known issues (with the exception of one > Qhull issue on Debian, s390x platform) have been solved. In the meantime > also OS X 10.8 was released, this RC contains a few build fixes for that > platform. > > If no more serious issues are reported, the final release will be in one > week. > > Cheers, > Ralf Hi Ralf, test_qz_double_sort is now failing in all msvc9/MKL builds (Python 2.6 to 3.2, 32 and 64 bit): FAIL: test_qz_double_sort (test_decomp.TestQZ) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python32\lib\site-packages\scipy\linalg\tests\test_decomp.py", line 1728, in test_qz_double_sort [ 0. , 0. , 0. , -12.8217]]), 4) File "X:\Python32\lib\site-packages\numpy\testing\utils.py", line 800, in assert_array_almost_equal header=('Arrays are not almost equal to %d decimals' % decimal)) File "X:\Python32\lib\site-packages\numpy\testing\utils.py", line 636, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not almost equal to 4 decimals (mismatch 62.5%) x: array([[-14.66007968, -27.25220511, -31.55732717, -29.0823765 ], [ 0. , 30.23027809, 42.47668118, 52.55438253], [ 0. , 0. , 0.71600413, -2.77147791], [ 0. , 0. , 0. , 2.50096525]]) y: array([[ 3.57864000e+01, -8.09061000e+01, -1.20629000e+01, -9.49800000e+00], [ 0.00000000e+00, 2.76380000e+00, -2.35050000e+00,... Christoph From hasslerjc at comcast.net Mon Aug 13 18:09:19 2012 From: hasslerjc at comcast.net (John Hassler) Date: Mon, 13 Aug 2012 18:09:19 -0400 Subject: [SciPy-User] ANN: SciPy 0.11.0 release candidate 2 In-Reply-To: References: <50294CEB.4070103@comcast.net> Message-ID: <50297B0F.2090400@comcast.net> An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Aug 14 02:36:05 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 14 Aug 2012 08:36:05 +0200 Subject: [SciPy-User] ANN: SciPy 0.11.0 release candidate 2 In-Reply-To: <50297B0F.2090400@comcast.net> References: <50294CEB.4070103@comcast.net> <50297B0F.2090400@comcast.net> Message-ID: On Tue, Aug 14, 2012 at 12:09 AM, John Hassler wrote: > > On 8/13/2012 5:17 PM, Ralf Gommers wrote: > > > > On Mon, Aug 13, 2012 at 8:52 PM, John Hassler wrote: > >> >> On 8/13/2012 2:30 PM, Ralf Gommers wrote: >> >> Hi, >> >> I am pleased to announce the availability of the second release candidate >> of SciPy 0.11.0. For this release many new features have been added, and >> over 120 tickets and pull requests have been closed. Also noteworthy is >> that the number of contributors for this release has risen to over 50. Some >> of the highlights are: >> >> - A new module, sparse.csgraph, has been added which provides a number >> of common sparse graph algorithms. >> - New unified interfaces to the existing optimization and root finding >> functions have been added. >> >> Sources and binaries can be found at >> http://sourceforge.net/projects/scipy/files/scipy/0.11.0rc2/, release >> notes are copied below. >> >> For this release candidate all known issues (with the exception of one >> Qhull issue on Debian, s390x platform) have been solved. In the meantime >> also OS X 10.8 was released, this RC contains a few build fixes for that >> platform. >> >> If no more serious issues are reported, the final release will be in one >> week. >> >> Cheers, >> Ralf >> >> The diagnostics all run: one warning, one error, one failure. >> >> Thanks John. Is that test_singular failure repeatable? You didn't > report that as failing for RC1. > > Ralf > > > If I restart the shell, I always get the same result that I posted. If I > re-run scipy.test() without a restart, I get two new errors every time, but > I get the test_singular failure only about half the time. The new ones are: > Those two new ones are normal; Python only raises a warning from the same place once, so in the second test the check for raising it fails. Does anyone have an idea about that test_singular failure? Ralf > ====================================================================== > FAIL: test_mio.test_mat4_3d > > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python32\lib\site-packages\nose-1.0.0-py3.2.egg\nose\case.py", > line 188, in runTest > self.test(*self.arg) > File "C:\Python32\lib\site-packages\scipy\io\matlab\tests\test_mio.py", > line 771, in test_mat4_3d > stream, {'a': arr}, True, '4') > File "C:\Python32\lib\site-packages\numpy\testing\utils.py", line 1008, > in assert_raises > return nose.tools.assert_raises(*args,**kwargs) > AssertionError: DeprecationWarning not raised by > functools.partial(, oned_as='row') > > ====================================================================== > FAIL: Regression test for #651: better handling of badly conditioned > > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "C:\Python32\lib\site-packages\scipy\signal\tests\test_filter_design.py", > line 34, in test_bad_filter > assert_raises(BadCoefficients, tf2zpk, [1e-15], [1.0, 1.0]) > File "C:\Python32\lib\site-packages\numpy\testing\utils.py", line 1008, > in assert_raises > return nose.tools.assert_raises(*args,**kwargs) > AssertionError: BadCoefficients not raised by tf2zpk > > ---------------------------------------------------------------------- > Ran 5351 tests in 103.766s > > FAILED (KNOWNFAIL=14, SKIP=42, errors=1, failures=2) > > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Tue Aug 14 07:21:52 2012 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 14 Aug 2012 11:21:52 +0000 (UTC) Subject: [SciPy-User] ANN: SciPy 0.11.0 release candidate 2 References: <50294CEB.4070103@comcast.net> <50297B0F.2090400@comcast.net> Message-ID: Ralf Gommers gmail.com> writes: [clip] > Does anyone have an idea about that test_singular failure? That's very likely some problem with the underlying LAPACK library. I think the problem solved is close to a numerical instability. The failing comparison compares eigenvalues computed by eig(A, B) eig(A, B, left=False, right=False) which differ solely in passing 'N' vs. 'V' to DGGEV. The eigenvalue property of the former is also checked and seems to pass. Interestingly, the result obtained from the two seems to differ (therefore, the latter is probably wrong), which appears to point to a LAPACK issue. Here, it would be interesting to know if the problem occurs with the official Scipy binaries, or something else. -- Pauli Virtanen From hasslerjc at comcast.net Tue Aug 14 09:35:32 2012 From: hasslerjc at comcast.net (John Hassler) Date: Tue, 14 Aug 2012 09:35:32 -0400 Subject: [SciPy-User] ANN: SciPy 0.11.0 release candidate 2 In-Reply-To: References: <50294CEB.4070103@comcast.net> <50297B0F.2090400@comcast.net> Message-ID: <502A5424.2070408@comcast.net> On 8/14/2012 7:21 AM, Pauli Virtanen wrote: > Ralf Gommers gmail.com> writes: > [clip] >> Does anyone have an idea about that test_singular failure? > That's very likely some problem with the underlying LAPACK library. > I think the problem solved is close to a numerical instability. > > The failing comparison compares eigenvalues computed by > > eig(A, B) > eig(A, B, left=False, right=False) > > which differ solely in passing 'N' vs. 'V' to DGGEV. The eigenvalue > property of the former is also checked and seems to pass. Interestingly, > the result obtained from the two seems to differ (therefore, the latter > is probably wrong), which appears to point to a LAPACK issue. > > Here, it would be interesting to know if the problem occurs with > the official Scipy binaries, or something else. > I installed rc2 on Python 2.7.3. Same problem. I get the test_singular error on some, but not all, of the runs. Both are win32-superpack from http://sourceforge.net/projects/scipy/files/scipy/0.11.0rc2/. The error occurs on less than half but more than 1/3 (based on a very small sample) of the runs on both 2.7 and 3.2. I've been working on computers for more than 50 years. Somehow, I had developed the delusion that they were deterministic ..... john ------------- Python 2.7 -------------------- >>> import scipy >>> scipy.test() Running unit tests for scipy NumPy version 1.6.2 NumPy is installed in C:\Python27\lib\site-packages\numpy SciPy version 0.11.0rc2 SciPy is installed in C:\Python27\lib\site-packages\scipy Python version 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] nose version 0.11.2 ====================================================================== FAIL: test_decomp.TestEig.test_singular Test singular pair ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python27\lib\site-packages\nose-0.11.2-py2.7.egg\nose\case.py", line 186, in runTest self.test(*self.arg) File "C:\Python27\lib\site-packages\scipy\linalg\tests\test_decomp.py", line 201, in test_singular self._check_gen_eig(A, B) File "C:\Python27\lib\site-packages\scipy\linalg\tests\test_decomp.py", line 188, in _check_gen_eig err_msg=msg) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 800, in assert_array_almost_equal header=('Arrays are not almost equal to %d decimals' % decimal)) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 636, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not almost equal to 6 decimals array([[22, 34, 31, 31, 17], [45, 45, 42, 19, 29], [39, 47, 49, 26, 34], [27, 31, 26, 21, 15], [38, 44, 44, 24, 30]]) array([[13, 26, 25, 17, 24], [31, 46, 40, 26, 37], [26, 40, 19, 25, 25], [16, 25, 27, 14, 23], [24, 35, 18, 21, 22]]) (mismatch 25.0%) x: array([ -2.45037885e-01 +0.00000000e+00j, 5.17637463e-16 -4.01120590e-08j, 5.17637463e-16 +4.01120590e-08j, 2.00000000e+00 +0.00000000e+00j]) y: array([ -3.74550285e-01 +0.00000000e+00j, -5.17716907e-17 -1.15230800e-08j, -5.17716907e-17 +1.15230800e-08j, 2.00000000e+00 +0.00000000e+00j]) ---------------------------------------------------------------------- Ran 5490 tests in 103.250s FAILED (KNOWNFAIL=14, SKIP=36, failures=1) >>> -------------- Python 3.2 -------------- >>> scipy.test() Running unit tests for scipy NumPy version 1.6.2 NumPy is installed in C:\Python32\lib\site-packages\numpy SciPy version 0.11.0rc2 SciPy is installed in C:\Python32\lib\site-packages\scipy Python version 3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit (Intel)] nose version 1.0.0 ====================================================================== FAIL: test_decomp.TestEig.test_singular ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python32\lib\site-packages\nose-1.0.0-py3.2.egg\nose\case.py", line 188, in runTest self.test(*self.arg) File "C:\Python32\lib\site-packages\scipy\linalg\tests\test_decomp.py", line 201, in test_singular self._check_gen_eig(A, B) File "C:\Python32\lib\site-packages\scipy\linalg\tests\test_decomp.py", line 188, in _check_gen_eig err_msg=msg) File "C:\Python32\lib\site-packages\numpy\testing\utils.py", line 800, in assert_array_almost_equal header=('Arrays are not almost equal to %d decimals' % decimal)) File "C:\Python32\lib\site-packages\numpy\testing\utils.py", line 636, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not almost equal to 6 decimals array([[22, 34, 31, 31, 17], [45, 45, 42, 19, 29], [39, 47, 49, 26, 34], [27, 31, 26, 21, 15], [38, 44, 44, 24, 30]]) array([[13, 26, 25, 17, 24], [31, 46, 40, 26, 37], [26, 40, 19, 25, 25], [16, 25, 27, 14, 23], [24, 35, 18, 21, 22]]) (mismatch 25.0%) x: array([ -2.450e-01 +0.000e+00j, 5.176e-16 -4.011e-08j, 5.176e-16 +4.011e-08j, 2.000e+00 +0.000e+00j]) y: array([ -3.746e-01 +0.000e+00j, -5.177e-17 -1.152e-08j, -5.177e-17 +1.152e-08j, 2.000e+00 +0.000e+00j]) From newville at cars.uchicago.edu Tue Aug 14 11:04:53 2012 From: newville at cars.uchicago.edu (Matt Newville) Date: Tue, 14 Aug 2012 10:04:53 -0500 Subject: [SciPy-User] Question about errors (uncertainties) in non-linear least squares fitting (Jonathan Helmus) Message-ID: Hi, On Mon, 13 Aug 2012 15:01:05 -0400 Jonathan Helmus wrote: > Pawel, > > leastsqbound cannot fix a parameter. If you want a fixed parameter > in the fit you can either rewrite your fitting/error function to take > the fixed parameters as an extra argument, or use a more "full featured" > constrained least squares fitting package like lmfit > (http://newville.github.com/lmfit-py/) whose class based Parameters > allow for additional control. > > For any aspiring developers out there, fixed parameter could be > added to leastsqbound (although it wouldn't be easy) but I'm not > planning on add it in the near term. I would accept a patch which added > this. > > - Jonathan Helmus Reading this conversation with Pawel over the past week or so, I was reminded and re-inspired to fix the setting of bounds in lmfit-py which had been pretty fragile to use the much more robust MINUT-style transformations for min/max bounds as Jonathan did in leastsqbound. Lmfit-py version 0.6.0 borrows heavily from leastsqbound (thanks Jonathan!) for it's implementation of setting bounds, and also includes the ability for users to change whether parameters in the fitting model are fixed or varied, and to set up simple mathematical constraints between parameters. It also provides support tools for more thoroughly investigating confidence intervals beyond the simple use of the covariance matrix (thanks to Till Stensitzki!). This latest version is now available from http://pypi.python.org/pypi/lmfit/0.6 with the development version at http://newville.github.com/lmfit-py/ Feedback, suggestions, and bug reports are most welcome. --Matt Newville From ralf.gommers at gmail.com Tue Aug 14 13:12:01 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 14 Aug 2012 19:12:01 +0200 Subject: [SciPy-User] ANN: SciPy 0.11.0 release candidate 2 In-Reply-To: <50297AFE.3020806@uci.edu> References: <50297AFE.3020806@uci.edu> Message-ID: On Tue, Aug 14, 2012 at 12:09 AM, Christoph Gohlke wrote: > On 8/13/2012 11:30 AM, Ralf Gommers wrote: > > Hi, > > > > I am pleased to announce the availability of the second release > > candidate of SciPy 0.11.0. For this release many new features have been > > added, and over 120 tickets and pull requests have been closed. Also > > noteworthy is that the number of contributors for this release has risen > > to over 50. Some of the highlights are: > > > > - A new module, sparse.csgraph, has been added which provides a > > number of common sparse graph algorithms. > > - New unified interfaces to the existing optimization and root > > finding functions have been added. > > > > Sources and binaries can be found at > > http://sourceforge.net/projects/scipy/files/scipy/0.11.0rc2/, release > > notes are copied below. > > > > For this release candidate all known issues (with the exception of one > > Qhull issue on Debian, s390x platform) have been solved. In the meantime > > also OS X 10.8 was released, this RC contains a few build fixes for that > > platform. > > > > If no more serious issues are reported, the final release will be in one > > week. > > > > Cheers, > > Ralf > > > Hi Ralf, > > test_qz_double_sort is now failing in all msvc9/MKL builds (Python 2.6 > to 3.2, 32 and 64 bit): > Hmm, I think that that excludes anything compiler or ATLAS specific. This is the only test which really checks the results are correct for float input, almost all the other test only check that the output of qz is self-consistent (no hard-coded expected output). It seems that a couple more tests would be helpful. This issue has held up the release for too long already, so unless someone has time to get it resolved this week, I propose the following: 1. Figure out if the problem is the sort function or something else. 2. If it's sort, disable it. Otherwise remove the qz function from the 0.11.x branch. Ralf > > FAIL: test_qz_double_sort (test_decomp.TestQZ) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "X:\Python32\lib\site-packages\scipy\linalg\tests\test_decomp.py", line > 1728, in test_qz_double_sort > [ 0. , 0. , 0. , -12.8217]]), 4) > File "X:\Python32\lib\site-packages\numpy\testing\utils.py", line > 800, in assert_array_almost_equal > header=('Arrays are not almost equal to %d decimals' % decimal)) > File "X:\Python32\lib\site-packages\numpy\testing\utils.py", line > 636, in assert_array_compare > raise AssertionError(msg) > AssertionError: > Arrays are not almost equal to 4 decimals > > (mismatch 62.5%) > x: array([[-14.66007968, -27.25220511, -31.55732717, -29.0823765 ], > [ 0. , 30.23027809, 42.47668118, 52.55438253], > [ 0. , 0. , 0.71600413, -2.77147791], > [ 0. , 0. , 0. , 2.50096525]]) > y: array([[ 3.57864000e+01, -8.09061000e+01, -1.20629000e+01, > -9.49800000e+00], > [ 0.00000000e+00, 2.76380000e+00, -2.35050000e+00,... > > > Christoph > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanforeest at gmail.com Wed Aug 15 16:15:56 2012 From: vanforeest at gmail.com (nicky van foreest) Date: Wed, 15 Aug 2012 22:15:56 +0200 Subject: [SciPy-User] integrals and quad Message-ID: Hi, Given some function f it is easy with scipy.integrate.quad to compute the integral of f for some given endpoint. However, I need the integral at many endpoints, that is, I want to plot \int_0^t f(x) dx. How can this be done in an efficient and elegant way? To illustrate I used the following code. from numpy import cumsum, linspace, vectorize from scipy.integrate import quad from pylab import plot, show def f(x): return x F = vectorize(lambda t: quad(f, 0, t)[0]) # must be wasteful t = linspace(0,3, 50) FF = cumsum(f(t))*(t[1]-t[0]) # simple, but inaccurate, note that t[1]-t[0] is the grid size, a bit like dx in the integral plot(t,f(t)) plot(t, F(t)) plot(t, F) show() I suspect that calling F at many values is wasteful, since the integral is evaluated at the same points many times. The trick with using cumsum must save some work (an O(n) algo), but is less accurate as is shown by the graphs. So, I don't like to use cumsum, and I also don't like to use a vectorized quad. Is there something better? thanks Nicky From josef.pktd at gmail.com Wed Aug 15 16:20:37 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 15 Aug 2012 16:20:37 -0400 Subject: [SciPy-User] integrals and quad In-Reply-To: References: Message-ID: On Wed, Aug 15, 2012 at 4:15 PM, nicky van foreest wrote: > Hi, > > Given some function f it is easy with scipy.integrate.quad to compute > the integral of f for some given endpoint. However, I need the > integral at many endpoints, that is, I want to plot \int_0^t f(x) dx. > How can this be done in an efficient and elegant way? > > To illustrate I used the following code. > > from numpy import cumsum, linspace, vectorize > from scipy.integrate import quad > from pylab import plot, show > > def f(x): > return x > > F = vectorize(lambda t: quad(f, 0, t)[0]) # must be wasteful > > > t = linspace(0,3, 50) > > FF = cumsum(f(t))*(t[1]-t[0]) # simple, but inaccurate, note that > t[1]-t[0] is the grid size, a bit like dx in the integral > > plot(t,f(t)) > plot(t, F(t)) > plot(t, F) > show() > > I suspect that calling F at many values is wasteful, since the > integral is evaluated at the same points many times. The trick with > using cumsum must save some work (an O(n) algo), but is less accurate > as is shown by the graphs. So, I don't like to use cumsum, and I also > don't like to use a vectorized quad. Is there something better? cumtrapz is the only one that works (when I looked at this) I also tried odeint for this once before, but didn't really use it. Josef > > thanks > > Nicky > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From guziy.sasha at gmail.com Wed Aug 15 16:36:47 2012 From: guziy.sasha at gmail.com (Oleksandr Huziy) Date: Wed, 15 Aug 2012 16:36:47 -0400 Subject: [SciPy-User] integrals and quad In-Reply-To: References: Message-ID: Hi, I am not sure if it falls under the elegant, but what if you take the integrals for parts and then apply cumsum: >>> import numpy as np >>> t1 = np.linspace(0,3,50) >>> t2 = t1[1:] >>> t1 = t1[:-1] >>> parts = map(lambda x1,x2: quad(f,x1,x2)[0], t1,t2) >>> F = np.cumsum(parts) Probably you'd want to add 0 at the start of the result... Cheers -- Oleksandr (Sasha) Huziy 2012/8/15 > On Wed, Aug 15, 2012 at 4:15 PM, nicky van foreest > wrote: > > Hi, > > > > Given some function f it is easy with scipy.integrate.quad to compute > > the integral of f for some given endpoint. However, I need the > > integral at many endpoints, that is, I want to plot \int_0^t f(x) dx. > > How can this be done in an efficient and elegant way? > > > > To illustrate I used the following code. > > > > from numpy import cumsum, linspace, vectorize > > from scipy.integrate import quad > > from pylab import plot, show > > > > def f(x): > > return x > > > > F = vectorize(lambda t: quad(f, 0, t)[0]) # must be wasteful > > > > > > t = linspace(0,3, 50) > > > > FF = cumsum(f(t))*(t[1]-t[0]) # simple, but inaccurate, note that > > t[1]-t[0] is the grid size, a bit like dx in the integral > > > > plot(t,f(t)) > > plot(t, F(t)) > > plot(t, F) > > show() > > > > I suspect that calling F at many values is wasteful, since the > > integral is evaluated at the same points many times. The trick with > > using cumsum must save some work (an O(n) algo), but is less accurate > > as is shown by the graphs. So, I don't like to use cumsum, and I also > > don't like to use a vectorized quad. Is there something better? > > cumtrapz is the only one that works (when I looked at this) > I also tried odeint for this once before, but didn't really use it. > > Josef > > > > > thanks > > > > Nicky > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Aug 15 21:58:48 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 15 Aug 2012 19:58:48 -0600 Subject: [SciPy-User] odeint In-Reply-To: References: Message-ID: On Sun, Aug 12, 2012 at 1:49 PM, nicky van foreest wrote: > Hi Rob, > > Thanks for the suggestion. I didn't think of using splines. > > Nicky > > On 10 August 2012 10:09, Rob Clewley wrote: > > Hi, > > > > On Thu, Aug 9, 2012 at 2:15 PM, nicky van foreest > wrote: > >> I am using odeint to integrate the following delay differential equation > >> > >> df/dx = c + f(x) - f(x-1) > >> > >> somewhat cumbersome. Is it possible in odeint to somehow get > >> intermediate values of the function values? > > > > Nicky, I think your easiest way ahead here is to use a spline to fit > > your previous intervals' solution function, building up your solution > > using one that meets values and first derivatives at the end points to > > retain smoothness as you go. Scipy offers several such spline tools. > > Then you can sample that to your heart's content. > > > > -- > > Robert Clewley, Ph.D. > > Assistant Professor > > Neuroscience Institute and > > Department of Mathematics and Statistics > > Georgia State University > > PO Box 5030 > > Atlanta, GA 30302, USA > Depending on the time interval, it might work to treat the equation as a Chebyshev series by using its values at the Chebyshev points. Both the derivative and the delay are then linear operators that can be represented by matrices, see Trefethen's bookfor the differentiation matrices. The boundary conditions might be a bit tricky, but you don't mention how you handle the delay there. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanforeest at gmail.com Thu Aug 16 04:08:05 2012 From: vanforeest at gmail.com (nicky van foreest) Date: Thu, 16 Aug 2012 10:08:05 +0200 Subject: [SciPy-User] odeint In-Reply-To: References: Message-ID: Hi Chuck, Thanks for your suggestion. I already have something working now, based on the Rob's idea with splines. BTW, I first used interp1d, but that gave some problems at the boundaries due to tiny rounding errors. I now use univariatespline as this gives a function that also extends beyond the boundaries, hence can deal with the rounding problems. Nicky On 16 August 2012 03:58, Charles R Harris wrote: > > > On Sun, Aug 12, 2012 at 1:49 PM, nicky van foreest > wrote: >> >> Hi Rob, >> >> Thanks for the suggestion. I didn't think of using splines. >> >> Nicky >> >> On 10 August 2012 10:09, Rob Clewley wrote: >> > Hi, >> > >> > On Thu, Aug 9, 2012 at 2:15 PM, nicky van foreest >> > wrote: >> >> I am using odeint to integrate the following delay differential >> >> equation >> >> >> >> df/dx = c + f(x) - f(x-1) >> >> >> >> somewhat cumbersome. Is it possible in odeint to somehow get >> >> intermediate values of the function values? >> > >> > Nicky, I think your easiest way ahead here is to use a spline to fit >> > your previous intervals' solution function, building up your solution >> > using one that meets values and first derivatives at the end points to >> > retain smoothness as you go. Scipy offers several such spline tools. >> > Then you can sample that to your heart's content. >> > >> > -- >> > Robert Clewley, Ph.D. >> > Assistant Professor >> > Neuroscience Institute and >> > Department of Mathematics and Statistics >> > Georgia State University >> > PO Box 5030 >> > Atlanta, GA 30302, USA > > > Depending on the time interval, it might work to treat the equation as a > Chebyshev series by using its values at the Chebyshev points. Both the > derivative and the delay are then linear operators that can be represented > by matrices, see Trefethen's book for the differentiation matrices. The > boundary conditions might be a bit tricky, but you don't mention how you > handle the delay there. > > Chuck > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From vanforeest at gmail.com Thu Aug 16 04:13:42 2012 From: vanforeest at gmail.com (nicky van foreest) Date: Thu, 16 Aug 2012 10:13:42 +0200 Subject: [SciPy-User] integrals and quad In-Reply-To: References: Message-ID: Hi Josef and Sasha, Thanks for the hints. I tried cumtrapz, and it gives nice results, at least for the trivial integration of f(x) = x. Now that I know cumtrapz, I realize I could have found this myself, but just reading the scipy.integrate docs... Nicky On 15 August 2012 22:36, Oleksandr Huziy wrote: > Hi, > > I am not sure if it falls under the elegant, but what if you take the > integrals for parts and then apply cumsum: >>>> import numpy as np >>>> t1 = np.linspace(0,3,50) >>>> t2 = t1[1:] >>>> t1 = t1[:-1] >>>> parts = map(lambda x1,x2: quad(f,x1,x2)[0], t1,t2) >>>> F = np.cumsum(parts) > > Probably you'd want to add 0 at the start of the result... > > Cheers > -- > Oleksandr (Sasha) Huziy > > 2012/8/15 > >> On Wed, Aug 15, 2012 at 4:15 PM, nicky van foreest >> wrote: >> > Hi, >> > >> > Given some function f it is easy with scipy.integrate.quad to compute >> > the integral of f for some given endpoint. However, I need the >> > integral at many endpoints, that is, I want to plot \int_0^t f(x) dx. >> > How can this be done in an efficient and elegant way? >> > >> > To illustrate I used the following code. >> > >> > from numpy import cumsum, linspace, vectorize >> > from scipy.integrate import quad >> > from pylab import plot, show >> > >> > def f(x): >> > return x >> > >> > F = vectorize(lambda t: quad(f, 0, t)[0]) # must be wasteful >> > >> > >> > t = linspace(0,3, 50) >> > >> > FF = cumsum(f(t))*(t[1]-t[0]) # simple, but inaccurate, note that >> > t[1]-t[0] is the grid size, a bit like dx in the integral >> > >> > plot(t,f(t)) >> > plot(t, F(t)) >> > plot(t, F) >> > show() >> > >> > I suspect that calling F at many values is wasteful, since the >> > integral is evaluated at the same points many times. The trick with >> > using cumsum must save some work (an O(n) algo), but is less accurate >> > as is shown by the graphs. So, I don't like to use cumsum, and I also >> > don't like to use a vectorized quad. Is there something better? >> >> cumtrapz is the only one that works (when I looked at this) >> I also tried odeint for this once before, but didn't really use it. >> >> Josef >> >> > >> > thanks >> > >> > Nicky >> > _______________________________________________ >> > SciPy-User mailing list >> > SciPy-User at scipy.org >> > http://mail.scipy.org/mailman/listinfo/scipy-user >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From guyer at nist.gov Thu Aug 16 13:42:41 2012 From: guyer at nist.gov (Jonathan Guyer) Date: Thu, 16 Aug 2012 13:42:41 -0400 Subject: [SciPy-User] ANN: FiPy 3.0 Message-ID: <18B2EC1A-D7BA-4598-B50E-152A0F405B55@nist.gov> We are pleased to announce the release of FiPy 3.0. http://www.ctcms.nist.gov/fipy The bump in major version number reflects more on the substantial increase in capabilities and ease of use than it does on a break in compatibility with FiPy 2.x. Few, if any, changes to your existing scripts should be necessary. The significant changes since version 2.1 are: ? Coupled and vector equations are now supported. ? A more robust mechanism for specifying boundary conditions is now used. ? Most Meshes can be partitioned by meshing with Gmsh. ? PyAMG and SciPy have been added to the solvers. ? FiPy is capable of running under Python 3. ? ?getter? and ?setter? methods have been pervasively changed to Python properties. ? The test suite now runs much faster. ? Tests can now be run on a full install using fipy.test(). This release addresses 66 tickets. ======================================================================== FiPy is an object oriented, partial differential equation (PDE) solver, written in Python, based on a standard finite volume (FV) approach. The framework has been developed in the Metallurgy Division and Center for Theoretical and Computational Materials Science (CTCMS), in the Material Measurement Laboratory (MML) at the National Institute of Standards and Technology (NIST). The solution of coupled sets of PDEs is ubiquitous to the numerical simulation of science problems. Numerous PDE solvers exist, using a variety of languages and numerical approaches. Many are proprietary, expensive and difficult to customize. As a result, scientists spend considerable resources repeatedly developing limited tools for specific problems. Our approach, combining the FV method and Python, provides a tool that is extensible, powerful and freely available. A significant advantage to Python is the existing suite of tools for array calculations, sparse matrices and data rendering. The FiPy framework includes terms for transient diffusion, convection and standard sources, enabling the solution of arbitrary combinations of coupled elliptic, hyperbolic and parabolic PDEs. Currently implemented models include phase field treatments of polycrystalline, dendritic, and electrochemical phase transformations as well as a level set treatment of the electrodeposition process. From pmhobson at gmail.com Thu Aug 16 15:29:52 2012 From: pmhobson at gmail.com (Paul Hobson) Date: Thu, 16 Aug 2012 12:29:52 -0700 Subject: [SciPy-User] brentq and brenth In-Reply-To: <1344786461.52280.YahooMailNeo@web31811.mail.mud.yahoo.com> References: <1344786461.52280.YahooMailNeo@web31811.mail.mud.yahoo.com> Message-ID: Perhaps it's intended to be interpreted as "Brent-Quadratic" and "Brend-Hyperbolic" -p On Sun, Aug 12, 2012 at 8:47 AM, The Helmbolds wrote: > It would seem more logical to call brentq "brentp" ("p" for parabola) to > follow the same pattern as "brenth" ("h" for hyperbola). > > Unless there is an inside joke to use "q" because it is "p" reflected about > a vertical axis, and hence represents Brent's "inverse parabolic" estimate > of the root? > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From mdekauwe at gmail.com Sun Aug 19 05:07:59 2012 From: mdekauwe at gmail.com (Martin De Kauwe) Date: Sun, 19 Aug 2012 02:07:59 -0700 (PDT) Subject: [SciPy-User] Avoiding inner for loops?? Message-ID: <82d5dfed-e68c-44b6-94f9-7f2557ddece4@googlegroups.com> Hi, I need to avoid (at least) two inner for loops in what I am trying to do otherwise my processing takes forever. What is the best way to transfer what I am doing into a more "numpy way"? Essentially I am trying to call a model again for various different parameter combinations. The example is fictional, by the grid_size would ideally grow > 500 and by doing so the processing speed becomes very slow the way I have set things up.. thanks. example. import numpy as np def fake_model(data1, data2, p1, p2, p3): """ complete nonsense """ return data1 + data2 * p1 * p2 * p3 data1 = np.random.rand(10) # the size of this arrays varies might be 10 might be 15 etc data2 = np.random.rand(10) # the size of this arrays varies might be 10 might be 15 etc obs = np.random.rand(10) # the size of this arrays varies might be 10 might be 15 etc grid_size = 10 # Ideally this would be a large number param1 = np.linspace(5.0, 350, grid_size) param2 = np.linspace(5.0, 550, grid_size) param3 = np.linspace(1E-8, 10.5, grid_size) ss = np.zeros(0) for p1 in param1: for p2 in param2: for p3 in param3: ans = fake_model(data1, data2, p1, p2, p3) ss = np.append(ss, np.sum(obs - ans)**2) print np.sum(obs - ans)**2 -------------- next part -------------- An HTML attachment was scrubbed... URL: From aeronaelius at gmail.com Sat Aug 18 19:07:29 2012 From: aeronaelius at gmail.com (Carlos Baptista) Date: Sun, 19 Aug 2012 01:07:29 +0200 Subject: [SciPy-User] Strange import behaviour Message-ID: Dear all, I have come across a very strange behaviour of Scipy. It has to do with importing and using Scipy functions. For example, this works: >>> import scipy.sparse as ss >>> ss However this does not work: >>> import scipy as sp >>> sp.sparse Traceback (most recent call last): File "", line 1, in sp.sparse AttributeError: 'module' object has no attribute 'sparse' The above example is merely a condensation of my real problem (using spsolve from scipy.sparse.linalg). What might cause this problem? Kind regards, Carlos -------------- next part -------------- An HTML attachment was scrubbed... URL: From silva at lma.cnrs-mrs.fr Sun Aug 19 07:02:49 2012 From: silva at lma.cnrs-mrs.fr (Fabrice Silva) Date: Sun, 19 Aug 2012 13:02:49 +0200 Subject: [SciPy-User] Strange import behaviour In-Reply-To: References: Message-ID: <1345374169.3880.0.camel@amilo.coursju> Have a look at http://docs.scipy.org/doc/scipy/reference/api.html#guidelines-for-importing-functions-from-scipy From santosh.singh at ansys.com Thu Aug 16 09:38:38 2012 From: santosh.singh at ansys.com (Santosh Pratap Singh) Date: Thu, 16 Aug 2012 19:08:38 +0530 Subject: [SciPy-User] scipy-0.10.1 compilation on 64 bit windows in debug mode Message-ID: Hi All, I am trying to compile a debug version of the scipy-0.10.1 extension on 64 bit windows. I have also compiled the python-2.7.3 and numpy-1.6.2 in debug mode. When I am building scipy, I ran into the problem C:\cpython\Debug\scipy\0_10_1\scipy-0.10.1>python_d setup.py config --compiler=msvc --fcompiler=intelvem build --debug compile options: '-g -IC:\cpython\Debug\CPython\2_7_3\winx64\Debug\python\lib\site-packages\numpy\core\include -c' ifort.exe:f77: scipy\fftpack\src\dfftpack\dcosqb.f ifort: command line warning #10131: extension 'M' not supported ignored in optio n '/Qax' *ifort: command line error: option '/g' is ambiguous* error: Command "C:\Program Files (x86)\Intel\Compiler\11.1\060\Bin\intel64\ifort.exe -FI -w90 -w95 /nologo /MD /nbs /Qlowercase /us /O1 /QaxM -g -IC:\cpython\Debug\CPython\2_7_3\winx64\Debug\python\lib\site-packages\numpy\core\include -c /c scipy\fftpack\src\dfftpack\dcosqb.f /Fobuild\temp.win-amd64-2.7\scipy\fftpack\src\dfftpack\dcosqb.o" failed with exit status 1 Can some one help me in this? Am I missing something? Thanks. Santosh -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevin.gullikson at gmail.com Fri Aug 17 17:34:51 2012 From: kevin.gullikson at gmail.com (Kevin Gullikson) Date: Fri, 17 Aug 2012 16:34:51 -0500 Subject: [SciPy-User] UnivariateSpline returning NaN Message-ID: Hi all, I am having a weird issue with UnivariateSpline return nan when trying to interpolate some data. I have had this happen before a couple times, but it was always because a nan was in the input arrays. Here, they are all real numbers. Here is some minimal code that recreates my problem: import numpy x,y = numpy.loadtxt("test.out", usecols=(0,1), unpack=True) from scipy.interpolate import UnivariateSpline fcn = UnivariateSpline(x,y,s=0) print fcn(numpy.median(x)) (the result of the print is nan) Here is the data that I used: 960.38174 2218580.3 2205540 960.3904 2209433.1 2205465.9 960.39906 2202205.2 2205391.9 960.40773 2209385.9 2205317.8 960.41639 2218550.3 2205243.7 960.42505 2220656.8 2205169.7 960.43372 2220779.4 2205095.6 960.44238 2220771.8 2205021.5 960.45104 2220764.3 2204947.5 960.67545 2220409.5 2203030.8 960.68725 2220387.5 2202930.1 960.69905 2220365.5 2202829.5 960.71085 2220343.4 2202728.8 960.72265 2220321.4 2202628.2 960.73445 2220299.3 2202527.6 960.74625 2220277.3 2202427 960.75805 2220222.8 2202326.4 960.76985 2220200.7 2202225.7 960.78165 2220178.7 2202125.2 960.79345 2220156.6 2202024.6 960.80525 2220134.6 2201924 960.81705 2220080.1 2201823.4 961.16055 2219013.1 2198897.2 961.16699 2218983.4 2198842.3 961.17344 2218953.6 2198787.4 961.17988 2218923.8 2198732.5 961.18788 2218886.9 2198664.3 961.19587 2218882.5 2198596.2 961.20386 2218910.5 2198528.1 961.21186 2218906.1 2198459.9 961.21963 2218837.8 2198393.7 961.2274 2218769.4 2198327.4 961.23517 2218701.1 2198261.2 961.24294 2218697.7 2198194.9 961.25071 2218661.8 2198128.7 961.25695 2218633 2198075.6 961.26317 2218604.3 2198022.5 961.27485 2218550.4 2197922.9 961.28653 2218496.5 2197823.4 961.2982 2218475 2197723.8 961.30988 2218421.1 2197624.3 961.32156 2218367.3 2197524.8 961.75259 2216379.3 2193857.4 961.75259 2225525.5 2193857.4 962.04986 2222983.5 2191338.5 962.05175 2222942.3 2191322.5 962.69908 2217396.7 2185841.9 962.71837 2217210.7 2185678.5 962.73766 2217024.8 2185515.2 962.75695 2214605.6 2185352 962.77624 2180663.5 2185188.7 962.79552 2080767.6 2185025.6 962.81481 2027222.1 2184862.4 962.83411 2080471.4 2184699.3 962.8534 2180022.7 2184536.2 962.87269 2213620.2 2184373.1 Thanks, Kevin Gullikson -------------- next part -------------- An HTML attachment was scrubbed... URL: From vagabondaero at gmail.com Sun Aug 19 09:44:42 2012 From: vagabondaero at gmail.com (Vagabond_Aero) Date: Sun, 19 Aug 2012 06:44:42 -0700 Subject: [SciPy-User] UnivariateSpline returning NaN In-Reply-To: References: Message-ID: Looking at your data, about 3/4 of the way down you have 2 x values exactly the same. Leaving those in, I could recreate your problem, but removing one of them made the problem go away. On Fri, Aug 17, 2012 at 2:34 PM, Kevin Gullikson wrote: > Hi all, > > I am having a weird issue with UnivariateSpline return nan when trying to > interpolate some data. I have had this happen before a couple times, but it > was always because a nan was in the input arrays. Here, they are all real > numbers. Here is some minimal code that recreates my problem: > > import numpy > x,y = numpy.loadtxt("test.out", usecols=(0,1), unpack=True) > from scipy.interpolate import UnivariateSpline > fcn = UnivariateSpline(x,y,s=0) > print fcn(numpy.median(x)) > > (the result of the print is nan) > > > Here is the data that I used: > > 960.38174 2218580.3 2205540 > 960.3904 2209433.1 2205465.9 > 960.39906 2202205.2 2205391.9 > 960.40773 2209385.9 2205317.8 > 960.41639 2218550.3 2205243.7 > 960.42505 2220656.8 2205169.7 > 960.43372 2220779.4 2205095.6 > 960.44238 2220771.8 2205021.5 > 960.45104 2220764.3 2204947.5 > 960.67545 2220409.5 2203030.8 > 960.68725 2220387.5 2202930.1 > 960.69905 2220365.5 2202829.5 > 960.71085 2220343.4 2202728.8 > 960.72265 2220321.4 2202628.2 > 960.73445 2220299.3 2202527.6 > 960.74625 2220277.3 2202427 > 960.75805 2220222.8 2202326.4 > 960.76985 2220200.7 2202225.7 > 960.78165 2220178.7 2202125.2 > 960.79345 2220156.6 2202024.6 > 960.80525 2220134.6 2201924 > 960.81705 2220080.1 2201823.4 > 961.16055 2219013.1 2198897.2 > 961.16699 2218983.4 2198842.3 > 961.17344 2218953.6 2198787.4 > 961.17988 2218923.8 2198732.5 > 961.18788 2218886.9 2198664.3 > 961.19587 2218882.5 2198596.2 > 961.20386 2218910.5 2198528.1 > 961.21186 2218906.1 2198459.9 > 961.21963 2218837.8 2198393.7 > 961.2274 2218769.4 2198327.4 > 961.23517 2218701.1 2198261.2 > 961.24294 2218697.7 2198194.9 > 961.25071 2218661.8 2198128.7 > 961.25695 2218633 2198075.6 > 961.26317 2218604.3 2198022.5 > 961.27485 2218550.4 2197922.9 > 961.28653 2218496.5 2197823.4 > 961.2982 2218475 2197723.8 > 961.30988 2218421.1 2197624.3 > 961.32156 2218367.3 2197524.8 > 961.75259 2216379.3 2193857.4 > 961.75259 2225525.5 2193857.4 > 962.04986 2222983.5 2191338.5 > 962.05175 2222942.3 2191322.5 > 962.69908 2217396.7 2185841.9 > 962.71837 2217210.7 2185678.5 > 962.73766 2217024.8 2185515.2 > 962.75695 2214605.6 2185352 > 962.77624 2180663.5 2185188.7 > 962.79552 2080767.6 2185025.6 > 962.81481 2027222.1 2184862.4 > 962.83411 2080471.4 2184699.3 > 962.8534 2180022.7 2184536.2 > 962.87269 2213620.2 2184373.1 > > > > Thanks, > > Kevin Gullikson > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- Co-discoverer of KBO: IH-X-694190 -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnl at cs.wisc.edu Sun Aug 19 10:22:19 2012 From: johnl at cs.wisc.edu (J. David Lee) Date: Sun, 19 Aug 2012 09:22:19 -0500 Subject: [SciPy-User] Avoiding inner for loops?? In-Reply-To: <82d5dfed-e68c-44b6-94f9-7f2557ddece4@googlegroups.com> References: <82d5dfed-e68c-44b6-94f9-7f2557ddece4@googlegroups.com> Message-ID: <5030F69B.7040000@cs.wisc.edu> Hi Martin, It looks like you are trying to do a brute force minimization of your model with three parameters, is that right? For that you could look at scipy.optimize.brute, which will probably be faster. scipy.optimize also has several more sophisticated minimizers that you might consider: http://docs.scipy.org/doc/scipy/reference/optimize.html If your function works with numpy arrays, you could try generating all of your parameters at once with mgrid and calling your function on the returned arrays, p1, p2, p3 = np.mgrid[:10,:10,:10] David On 08/19/2012 04:07 AM, Martin De Kauwe wrote: > Hi, > > I need to avoid (at least) two inner for loops in what I am trying to > do otherwise my processing takes forever. What is the best way to > transfer what I am doing into a more "numpy way"? Essentially I am > trying to call a model again for various different parameter > combinations. The example is fictional, by the grid_size would ideally > grow > 500 and by doing so the processing speed becomes very slow the > way I have set things up.. > > thanks. > > example. > > > import numpy as np > > def fake_model(data1, data2, p1, p2, p3): > """ complete nonsense """ > return data1 + data2 * p1 * p2 * p3 > > data1 = np.random.rand(10) # the size of this arrays varies might be > 10 might be 15 etc > data2 = np.random.rand(10) # the size of this arrays varies might be > 10 might be 15 etc > obs = np.random.rand(10) # the size of this arrays varies might be 10 > might be 15 etc > > grid_size = 10 # Ideally this would be a large number > param1 = np.linspace(5.0, 350, grid_size) > param2 = np.linspace(5.0, 550, grid_size) > param3 = np.linspace(1E-8, 10.5, grid_size) > ss = np.zeros(0) > > for p1 in param1: > for p2 in param2: > for p3 in param3: > ans = fake_model(data1, data2, p1, p2, p3) > > ss = np.append(ss, np.sum(obs - ans)**2) > print np.sum(obs - ans)**2 -------------- next part -------------- An HTML attachment was scrubbed... URL: From e.antero.tammi at gmail.com Sun Aug 19 10:40:39 2012 From: e.antero.tammi at gmail.com (eat) Date: Sun, 19 Aug 2012 17:40:39 +0300 Subject: [SciPy-User] Avoiding inner for loops?? In-Reply-To: <82d5dfed-e68c-44b6-94f9-7f2557ddece4@googlegroups.com> References: <82d5dfed-e68c-44b6-94f9-7f2557ddece4@googlegroups.com> Message-ID: Hi, On Sun, Aug 19, 2012 at 12:07 PM, Martin De Kauwe wrote: > Hi, > > I need to avoid (at least) two inner for loops in what I am trying to do > otherwise my processing takes forever. What is the best way to transfer > what I am doing into a more "numpy way"? Essentially I am trying to call a > model again for various different parameter combinations. The example is > fictional, by the grid_size would ideally grow > 500 and by doing so the > processing speed becomes very slow the way I have set things up.. > > thanks. > > example. > > > import numpy as np > > def fake_model(data1, data2, p1, p2, p3): > """ complete nonsense """ > return data1 + data2 * p1 * p2 * p3 > > data1 = np.random.rand(10) # the size of this arrays varies might be 10 > might be 15 etc > data2 = np.random.rand(10) # the size of this arrays varies might be 10 > might be 15 etc > obs = np.random.rand(10) # the size of this arrays varies might be 10 > might be 15 etc > > grid_size = 10 # Ideally this would be a large number > param1 = np.linspace(5.0, 350, grid_size) > param2 = np.linspace(5.0, 550, grid_size) > param3 = np.linspace(1E-8, 10.5, grid_size) > ss = np.zeros(0) > > for p1 in param1: > for p2 in param2: > for p3 in param3: > ans = fake_model(data1, data2, p1, p2, p3) > > ss = np.append(ss, np.sum(obs - ans)**2) > print np.sum(obs - ans)**2 > > You may like to utilize functionality of ix_ and newaxis, like: In []: p1, p2, p3= ix_(param1, param2, param3) In []: n_= newaxis In []: ans= fake_model(data1[:, n_, n_, n_], data2[:, n_, n_, n_], p1, p2, p3) In []: ss2= ((obs[:, n_, n_, n_]- ans).sum(0)** 2).reshape(grid_size** 3) In []: allclose(ss, ss2) Out[]: True My 2 cents, -eat > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdekauwe at gmail.com Sun Aug 19 03:56:19 2012 From: mdekauwe at gmail.com (mdekauwe) Date: Sun, 19 Aug 2012 00:56:19 -0700 (PDT) Subject: [SciPy-User] [SciPy-user] Avoiding inner for loops Message-ID: <34319609.post@talk.nabble.com> Hi, I need to avoid (at least) two inner for loops in what I am trying to do otherwise my processing takes forever. What is the best way to transfer what I am doing into a more "numpy way"? Essentially I am trying to call a model again for various different parameter combinations. The example is fictional, by the grid_size would ideally grow > 500 and by doing so the processing speed becomes very slow the way I have set things up.. thanks. example. import numpy as np def fake_model(data1, data2, p1, p2, p3): """ complete nonsense """ return data1 + data2 * p1 * p2 * p3 data1 = np.random.rand(10) # the size of this arrays varies might be 10 might be 15 etc data2 = np.random.rand(10) # the size of this arrays varies might be 10 might be 15 etc obs = np.random.rand(10) # the size of this arrays varies might be 10 might be 15 etc grid_size = 10 # Ideally this would be a large number param1 = np.linspace(5.0, 350, grid_size) param2 = np.linspace(5.0, 550, grid_size) param3 = np.linspace(1E-8, 10.5, grid_size) ss = np.zeros(0) for p1 in param1: for p2 in param2: for p3 in param3: ans = fake_model(data1, data2, p1, p2, p3) ss = np.append(ss, np.sum(obs - ans)**2) print np.sum(obs - ans)**2 -- View this message in context: http://old.nabble.com/Avoiding-inner-for-loops-tp34319609p34319609.html Sent from the Scipy-User mailing list archive at Nabble.com. From issa at aims.ac.za Sun Aug 19 15:36:05 2012 From: issa at aims.ac.za (Issa Karambal) Date: Sun, 19 Aug 2012 20:36:05 +0100 Subject: [SciPy-User] [SciPy-user] Avoiding inner for loops In-Reply-To: <34319609.post@talk.nabble.com> References: <34319609.post@talk.nabble.com> Message-ID: Hi, This will help. You do not need to have 'fake_model' function ss = [] params = param1*param2*param3 ans = data1+data2*params[:, None] for i in range(0,len(ans)): ss.append( np.sum( obs-ans[i] )**2 ) On 19 August 2012 08:56, mdekauwe wrote: > > Hi, > > I need to avoid (at least) two inner for loops in what I am trying to do > otherwise my processing takes forever. What is the best way to transfer > what > I am doing into a more "numpy way"? Essentially I am trying to call a model > again for various different parameter combinations. The example is > fictional, by the grid_size would ideally grow > 500 and by doing so the > processing speed becomes very slow the way I have set things up.. > > thanks. > > example. > > > import numpy as np > > def fake_model(data1, data2, p1, p2, p3): > """ complete nonsense """ > return data1 + data2 * p1 * p2 * p3 > > data1 = np.random.rand(10) # the size of this arrays varies might be 10 > might be 15 etc > data2 = np.random.rand(10) # the size of this arrays varies might be 10 > might be 15 etc > obs = np.random.rand(10) # the size of this arrays varies might be 10 might > be 15 etc > > grid_size = 10 # Ideally this would be a large number > param1 = np.linspace(5.0, 350, grid_size) > param2 = np.linspace(5.0, 550, grid_size) > param3 = np.linspace(1E-8, 10.5, grid_size) > ss = np.zeros(0) > > for p1 in param1: > for p2 in param2: > for p3 in param3: > ans = fake_model(data1, data2, p1, p2, p3) > > ss = np.append(ss, np.sum(obs - ans)**2) > print np.sum(obs - ans)**2 > -- > View this message in context: > http://old.nabble.com/Avoiding-inner-for-loops-tp34319609p34319609.html > Sent from the Scipy-User mailing list archive at Nabble.com. > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From issa at aims.ac.za Sun Aug 19 15:47:29 2012 From: issa at aims.ac.za (Issa Karambal) Date: Sun, 19 Aug 2012 20:47:29 +0100 Subject: [SciPy-User] [SciPy-user] Avoiding inner for loops In-Reply-To: References: <34319609.post@talk.nabble.com> Message-ID: If you do not want to use any loop, you might try this params = param1*param2*param3 ans = data1+data2*params[:, None] ss = np.sum(obs - ans, axis=1)**2 On 19 August 2012 20:36, Issa Karambal wrote: > Hi, > This will help. You do not need to have 'fake_model' function > > ss = [] > params = param1*param2*param3 > ans = data1+data2*params[:, None] > for i in range(0,len(ans)): > ss.append( np.sum( obs-ans[i] )**2 ) > > > On 19 August 2012 08:56, mdekauwe wrote: > >> >> Hi, >> >> I need to avoid (at least) two inner for loops in what I am trying to do >> otherwise my processing takes forever. What is the best way to transfer >> what >> I am doing into a more "numpy way"? Essentially I am trying to call a >> model >> again for various different parameter combinations. The example is >> fictional, by the grid_size would ideally grow > 500 and by doing so the >> processing speed becomes very slow the way I have set things up.. >> >> thanks. >> >> example. >> >> >> import numpy as np >> >> def fake_model(data1, data2, p1, p2, p3): >> """ complete nonsense """ >> return data1 + data2 * p1 * p2 * p3 >> >> data1 = np.random.rand(10) # the size of this arrays varies might be 10 >> might be 15 etc >> data2 = np.random.rand(10) # the size of this arrays varies might be 10 >> might be 15 etc >> obs = np.random.rand(10) # the size of this arrays varies might be 10 >> might >> be 15 etc >> >> grid_size = 10 # Ideally this would be a large number >> param1 = np.linspace(5.0, 350, grid_size) >> param2 = np.linspace(5.0, 550, grid_size) >> param3 = np.linspace(1E-8, 10.5, grid_size) >> ss = np.zeros(0) >> >> for p1 in param1: >> for p2 in param2: >> for p3 in param3: >> ans = fake_model(data1, data2, p1, p2, p3) >> >> ss = np.append(ss, np.sum(obs - ans)**2) >> print np.sum(obs - ans)**2 >> -- >> View this message in context: >> http://old.nabble.com/Avoiding-inner-for-loops-tp34319609p34319609.html >> Sent from the Scipy-User mailing list archive at Nabble.com. >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevin.gullikson.signup at gmail.com Sun Aug 19 15:49:21 2012 From: kevin.gullikson.signup at gmail.com (Kevin Gullikson) Date: Sun, 19 Aug 2012 14:49:21 -0500 Subject: [SciPy-User] UnivariateSpline returning NaN In-Reply-To: References: Message-ID: Thank you, that was it! I was banging my head against the wall trying to figure out what was wrong! On Sun, Aug 19, 2012 at 8:44 AM, Vagabond_Aero wrote: > Looking at your data, about 3/4 of the way down you have 2 x values > exactly the same. Leaving those in, I could recreate your problem, but > removing one of them made the problem go away. > > On Fri, Aug 17, 2012 at 2:34 PM, Kevin Gullikson < > kevin.gullikson at gmail.com> wrote: > >> Hi all, >> >> I am having a weird issue with UnivariateSpline return nan when trying to >> interpolate some data. I have had this happen before a couple times, but it >> was always because a nan was in the input arrays. Here, they are all real >> numbers. Here is some minimal code that recreates my problem: >> >> import numpy >> x,y = numpy.loadtxt("test.out", usecols=(0,1), unpack=True) >> from scipy.interpolate import UnivariateSpline >> fcn = UnivariateSpline(x,y,s=0) >> print fcn(numpy.median(x)) >> >> (the result of the print is nan) >> >> >> Here is the data that I used: >> >> 960.38174 2218580.3 2205540 >> 960.3904 2209433.1 2205465.9 >> 960.39906 2202205.2 2205391.9 >> 960.40773 2209385.9 2205317.8 >> 960.41639 2218550.3 2205243.7 >> 960.42505 2220656.8 2205169.7 >> 960.43372 2220779.4 2205095.6 >> 960.44238 2220771.8 2205021.5 >> 960.45104 2220764.3 2204947.5 >> 960.67545 2220409.5 2203030.8 >> 960.68725 2220387.5 2202930.1 >> 960.69905 2220365.5 2202829.5 >> 960.71085 2220343.4 2202728.8 >> 960.72265 2220321.4 2202628.2 >> 960.73445 2220299.3 2202527.6 >> 960.74625 2220277.3 2202427 >> 960.75805 2220222.8 2202326.4 >> 960.76985 2220200.7 2202225.7 >> 960.78165 2220178.7 2202125.2 >> 960.79345 2220156.6 2202024.6 >> 960.80525 2220134.6 2201924 >> 960.81705 2220080.1 2201823.4 >> 961.16055 2219013.1 2198897.2 >> 961.16699 2218983.4 2198842.3 >> 961.17344 2218953.6 2198787.4 >> 961.17988 2218923.8 2198732.5 >> 961.18788 2218886.9 2198664.3 >> 961.19587 2218882.5 2198596.2 >> 961.20386 2218910.5 2198528.1 >> 961.21186 2218906.1 2198459.9 >> 961.21963 2218837.8 2198393.7 >> 961.2274 2218769.4 2198327.4 >> 961.23517 2218701.1 2198261.2 >> 961.24294 2218697.7 2198194.9 >> 961.25071 2218661.8 2198128.7 >> 961.25695 2218633 2198075.6 >> 961.26317 2218604.3 2198022.5 >> 961.27485 2218550.4 2197922.9 >> 961.28653 2218496.5 2197823.4 >> 961.2982 2218475 2197723.8 >> 961.30988 2218421.1 2197624.3 >> 961.32156 2218367.3 2197524.8 >> 961.75259 2216379.3 2193857.4 >> 961.75259 2225525.5 2193857.4 >> 962.04986 2222983.5 2191338.5 >> 962.05175 2222942.3 2191322.5 >> 962.69908 2217396.7 2185841.9 >> 962.71837 2217210.7 2185678.5 >> 962.73766 2217024.8 2185515.2 >> 962.75695 2214605.6 2185352 >> 962.77624 2180663.5 2185188.7 >> 962.79552 2080767.6 2185025.6 >> 962.81481 2027222.1 2184862.4 >> 962.83411 2080471.4 2184699.3 >> 962.8534 2180022.7 2184536.2 >> 962.87269 2213620.2 2184373.1 >> >> >> >> Thanks, >> >> Kevin Gullikson >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > > -- > Co-discoverer of KBO: IH-X-694190 > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanforeest at gmail.com Sun Aug 19 15:51:02 2012 From: vanforeest at gmail.com (nicky van foreest) Date: Sun, 19 Aug 2012 21:51:02 +0200 Subject: [SciPy-User] UnivariateSpline returning NaN In-Reply-To: References: Message-ID: Hi, I also had this problem, but this was due to setting s = 0 in UnivariateSpline. In my case, setting s = 0.0001 helped remove the nans. I am not a specialist on splines, but settting s = 0 requires the splines to go through each and every point. That may be, perhaps, a too strong requirement. Nicky On 19 August 2012 15:44, Vagabond_Aero wrote: > Looking at your data, about 3/4 of the way down you have 2 x values exactly > the same. Leaving those in, I could recreate your problem, but removing one > of them made the problem go away. > > On Fri, Aug 17, 2012 at 2:34 PM, Kevin Gullikson > wrote: >> >> Hi all, >> >> I am having a weird issue with UnivariateSpline return nan when trying to >> interpolate some data. I have had this happen before a couple times, but it >> was always because a nan was in the input arrays. Here, they are all real >> numbers. Here is some minimal code that recreates my problem: >> >> import numpy >> x,y = numpy.loadtxt("test.out", usecols=(0,1), unpack=True) >> from scipy.interpolate import UnivariateSpline >> fcn = UnivariateSpline(x,y,s=0) >> print fcn(numpy.median(x)) >> >> (the result of the print is nan) >> >> >> Here is the data that I used: >> >> 960.38174 2218580.3 2205540 >> 960.3904 2209433.1 2205465.9 >> 960.39906 2202205.2 2205391.9 >> 960.40773 2209385.9 2205317.8 >> 960.41639 2218550.3 2205243.7 >> 960.42505 2220656.8 2205169.7 >> 960.43372 2220779.4 2205095.6 >> 960.44238 2220771.8 2205021.5 >> 960.45104 2220764.3 2204947.5 >> 960.67545 2220409.5 2203030.8 >> 960.68725 2220387.5 2202930.1 >> 960.69905 2220365.5 2202829.5 >> 960.71085 2220343.4 2202728.8 >> 960.72265 2220321.4 2202628.2 >> 960.73445 2220299.3 2202527.6 >> 960.74625 2220277.3 2202427 >> 960.75805 2220222.8 2202326.4 >> 960.76985 2220200.7 2202225.7 >> 960.78165 2220178.7 2202125.2 >> 960.79345 2220156.6 2202024.6 >> 960.80525 2220134.6 2201924 >> 960.81705 2220080.1 2201823.4 >> 961.16055 2219013.1 2198897.2 >> 961.16699 2218983.4 2198842.3 >> 961.17344 2218953.6 2198787.4 >> 961.17988 2218923.8 2198732.5 >> 961.18788 2218886.9 2198664.3 >> 961.19587 2218882.5 2198596.2 >> 961.20386 2218910.5 2198528.1 >> 961.21186 2218906.1 2198459.9 >> 961.21963 2218837.8 2198393.7 >> 961.2274 2218769.4 2198327.4 >> 961.23517 2218701.1 2198261.2 >> 961.24294 2218697.7 2198194.9 >> 961.25071 2218661.8 2198128.7 >> 961.25695 2218633 2198075.6 >> 961.26317 2218604.3 2198022.5 >> 961.27485 2218550.4 2197922.9 >> 961.28653 2218496.5 2197823.4 >> 961.2982 2218475 2197723.8 >> 961.30988 2218421.1 2197624.3 >> 961.32156 2218367.3 2197524.8 >> 961.75259 2216379.3 2193857.4 >> 961.75259 2225525.5 2193857.4 >> 962.04986 2222983.5 2191338.5 >> 962.05175 2222942.3 2191322.5 >> 962.69908 2217396.7 2185841.9 >> 962.71837 2217210.7 2185678.5 >> 962.73766 2217024.8 2185515.2 >> 962.75695 2214605.6 2185352 >> 962.77624 2180663.5 2185188.7 >> 962.79552 2080767.6 2185025.6 >> 962.81481 2027222.1 2184862.4 >> 962.83411 2080471.4 2184699.3 >> 962.8534 2180022.7 2184536.2 >> 962.87269 2213620.2 2184373.1 >> >> >> >> Thanks, >> >> Kevin Gullikson >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > > > -- > Co-discoverer of KBO: > > IH-X-694190 > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From mdekauwe at gmail.com Sun Aug 19 19:47:45 2012 From: mdekauwe at gmail.com (Martin De Kauwe) Date: Sun, 19 Aug 2012 16:47:45 -0700 (PDT) Subject: [SciPy-User] Avoiding inner for loops?? In-Reply-To: <82d5dfed-e68c-44b6-94f9-7f2557ddece4@googlegroups.com> References: <82d5dfed-e68c-44b6-94f9-7f2557ddece4@googlegroups.com> Message-ID: Perhaps simplifying, in 2D this is what I want if using loops def fake_model(data1, data2, p1, p2): """ complete nonsense """ return data1 + data2 * p1 * p2 grid_size = 5 nobs = 5 obs = np.zeros(nobs) data1 = np.arange(nobs) data2 = np.arange(nobs) a = np.arange(grid_size) b = np.arange(grid_size) c = np.arange(grid_size) ss = np.zeros(0) for p1 in a: for p2 in b: ans = fake_model(data1, data2, p1, p2) #ss = np.append(ss, np.sum(obs - ans)**2) print ans which would produce [0 1 2 3 4] [0 1 2 3 4] [0 1 2 3 4] [0 1 2 3 4] [0 1 2 3 4] [0 1 2 3 4] [0 2 4 6 8] . snip . [0 1 2 3 4] [ 0 5 10 15 20] [ 0 9 18 27 36] [ 0 13 26 39 52] [ 0 17 34 51 68] And so I figured something like... a = np.ones((grid_size,grid_size)) * np.arange(grid_size)[None,:] b = np.ones((grid_size,grid_size)) * np.arange(grid_size)[:,None] ans = fake_model(data1, data2, a, b) Although this doesn't seem to work, but I think this might be along the right lines? This produces [[ 0. 1. 2. 3. 4.] [ 0. 2. 6. 12. 20.] [ 0. 3. 10. 21. 36.] [ 0. 4. 14. 30. 52.] [ 0. 5. 18. 39. 68.]] On Sunday, August 19, 2012 7:07:59 PM UTC+10, Martin De Kauwe wrote: > > Hi, > > I need to avoid (at least) two inner for loops in what I am trying to do > otherwise my processing takes forever. What is the best way to transfer > what I am doing into a more "numpy way"? Essentially I am trying to call a > model again for various different parameter combinations. The example is > fictional, by the grid_size would ideally grow > 500 and by doing so the > processing speed becomes very slow the way I have set things up.. > > thanks. > > example. > > > import numpy as np > > def fake_model(data1, data2, p1, p2, p3): > """ complete nonsense """ > return data1 + data2 * p1 * p2 * p3 > > data1 = np.random.rand(10) # the size of this arrays varies might be 10 > might be 15 etc > data2 = np.random.rand(10) # the size of this arrays varies might be 10 > might be 15 etc > obs = np.random.rand(10) # the size of this arrays varies might be 10 > might be 15 etc > > grid_size = 10 # Ideally this would be a large number > param1 = np.linspace(5.0, 350, grid_size) > param2 = np.linspace(5.0, 550, grid_size) > param3 = np.linspace(1E-8, 10.5, grid_size) > ss = np.zeros(0) > > for p1 in param1: > for p2 in param2: > for p3 in param3: > ans = fake_model(data1, data2, p1, p2, p3) > > ss = np.append(ss, np.sum(obs - ans)**2) > print np.sum(obs - ans)**2 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jumper at uchicago.edu Mon Aug 20 03:00:04 2012 From: jumper at uchicago.edu (John Jumper) Date: Mon, 20 Aug 2012 02:00:04 -0500 Subject: [SciPy-User] Notification when array is modified Message-ID: I am looking for a method so that I can be notified when the contents of a numpy array change. A simplified class that illustrates the desired functionality is below. class ComputeDerived(object): def __init__(self, n): self.A = np.zeros(n) self.B = np.zeros(n) self.change_mask = np.ones(n, dtype=np.bool) self._Derived = expensive(self.A,self.B) def set_A(self, i, x): self.A[i] = x self.change_mask[i] = True def set_B(self, i, x): self.B[i] = x self.change_mask[i] = True def Derived(self): if np.any(self.change_mask): for i in np.nonzero(self.change_mask)[0]: self._Derived[i] = expensive(A[i],B[i]) self.change_mask[:] = False return self._Derived This class works, but I lack the power of numpy's indexing syntax to change A or B. I would like to be able to write compute_derived.A[(3,5,6)] = 1 and then be notified that A has changed. Otherwise, if I want to use a numpy interface for A and B, I would have to store old copies of A and B and traverse these copies every time Derived is called. Is there any interface in numpy so that I could expose A and B directly as writeable numpy arrays but be notified when they are modified? Alternatively, if I subclass ndarray, is there some exhaustive list of methods that can change the underlying data (I would be worried about view objects here)? In case it makes a difference, my actual use case is a C++ class wrapped using Boost::Python. The numpy arrays A and B would point at aligned float* members of my C++ class. Thank you, John Jumper From david_baddeley at yahoo.com.au Mon Aug 20 05:12:12 2012 From: david_baddeley at yahoo.com.au (David Baddeley) Date: Mon, 20 Aug 2012 02:12:12 -0700 (PDT) Subject: [SciPy-User] Notification when array is modified In-Reply-To: References: Message-ID: <1345453932.10936.YahooMailNeo@web113410.mail.gq1.yahoo.com> I'd make a proxy class to hold your array and implement the __getitem__ and __setitem__ methods. Something like (your actual implementation might need to be a bit more complex - numpy arrays seem to override __setslice__ as well as ?__setitem__): class ChangeTrackingArray(object): def __init__(self, mask, callback): self.mask = mask self.data = np.zeros(mask.shape) self.callback = callback def __getitem__(self, keys): return self.data.__getitem__(keys) def __setitem__(self, keys, values): self.data.__setitem__(keys, values) self.mask.__setitem__(keys, 1) self.callback() ________________________________ From: John Jumper To: scipy-user at scipy.org Sent: Monday, 20 August 2012 7:00 PM Subject: [SciPy-User] Notification when array is modified I am looking for a method so that I can be notified when the contents of a numpy array change. A simplified class that illustrates the desired functionality is below. class ComputeDerived(object): ? ? def __init__(self, n): ? ? ? ? self.A = np.zeros(n) ? ? ? ? self.B = np.zeros(n) ? ? ? ? self.change_mask = np.ones(n, dtype=np.bool) ? ? ? ? self._Derived = expensive(self.A,self.B) ? ? def set_A(self, i, x): ? ? ? ? self.A[i] = x ? ? ? ? self.change_mask[i] = True ? ? def set_B(self, i, x): ? ? ? ? self.B[i] = x ? ? ? ? self.change_mask[i] = True ? ? def Derived(self): ? ? ? ? if np.any(self.change_mask): ? ? ? ? ? ? for i in np.nonzero(self.change_mask)[0]: ? ? ? ? ? ? ? ? self._Derived[i] = expensive(A[i],B[i]) ? ? ? ? ? ? self.change_mask[:] = False ? ? ? ? return self._Derived This class works, but I lack the power of numpy's indexing syntax to change A or B.? I would like to be able to write compute_derived.A[(3,5,6)] = 1 and then be notified that A has changed.? Otherwise, if I want to use a numpy interface for A and B, I would have to store old copies of A and B and traverse these copies every time Derived is called. Is there any interface in numpy so that I could expose A and B directly as writeable numpy arrays but be notified when they are modified?? Alternatively, if I subclass ndarray, is there some exhaustive list of methods that can change the underlying data (I would be worried about view objects here)? In case it makes a difference, my actual use case is a C++ class wrapped using Boost::Python.? The numpy arrays A and B would point at aligned float* members of my C++ class. Thank you, John Jumper _______________________________________________ SciPy-User mailing list SciPy-User at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From e.antero.tammi at gmail.com Mon Aug 20 08:01:14 2012 From: e.antero.tammi at gmail.com (eat) Date: Mon, 20 Aug 2012 15:01:14 +0300 Subject: [SciPy-User] Avoiding inner for loops?? In-Reply-To: References: <82d5dfed-e68c-44b6-94f9-7f2557ddece4@googlegroups.com> Message-ID: Hi, On Mon, Aug 20, 2012 at 2:47 AM, Martin De Kauwe wrote: > Perhaps simplifying, in 2D this is what I want if using loops > > def fake_model(data1, data2, p1, p2): > """ complete nonsense """ > return data1 + data2 * p1 * p2 > > grid_size = 5 > nobs = 5 > obs = np.zeros(nobs) > data1 = np.arange(nobs) > data2 = np.arange(nobs) > a = np.arange(grid_size) > b = np.arange(grid_size) > c = np.arange(grid_size) > ss = np.zeros(0) > for p1 in a: > for p2 in b: > ans = fake_model(data1, data2, p1, p2) > #ss = np.append(ss, np.sum(obs - ans)**2) > print ans > > Your script with a slightly modified data and parameters, will produce [10 13 16 19 22] [15 19 23 27 31] [20 25 30 35 40] snip [100 121 142 163 184] [125 151 177 203 229] [150 181 212 243 274] which is equivalent to: In []: p1, p2= ix_(a, b) In []: n_= newaxis In []: ans= fake_model(data1[:, n_, n_], data2[:, n_, n_], p1, p2) In []: ans.reshape(-1, grid_size** 2).T Out[]: array([[ 10, 13, 16, 19, 22], [ 15, 19, 23, 27, 31], [ 20, 25, 30, 35, 40], snip [100, 121, 142, 163, 184], [125, 151, 177, 203, 229], [150, 181, 212, 243, 274]]) Regards, -eat > > which would produce > > [0 1 2 3 4] > [0 1 2 3 4] > [0 1 2 3 4] > [0 1 2 3 4] > [0 1 2 3 4] > [0 1 2 3 4] > [0 2 4 6 8] > . > snip > . > [0 1 2 3 4] > [ 0 5 10 15 20] > [ 0 9 18 27 36] > [ 0 13 26 39 52] > [ 0 17 34 51 68] > > And so I figured something like... > > a = np.ones((grid_size,grid_size)) * np.arange(grid_size)[None,:] > b = np.ones((grid_size,grid_size)) * np.arange(grid_size)[:,None] > ans = fake_model(data1, data2, a, b) > > Although this doesn't seem to work, but I think this might be along the > right lines? This produces > > [[ 0. 1. 2. 3. 4.] > [ 0. 2. 6. 12. 20.] > [ 0. 3. 10. 21. 36.] > [ 0. 4. 14. 30. 52.] > [ 0. 5. 18. 39. 68.]] > > > > > > > > > > On Sunday, August 19, 2012 7:07:59 PM UTC+10, Martin De Kauwe wrote: >> >> Hi, >> >> I need to avoid (at least) two inner for loops in what I am trying to do >> otherwise my processing takes forever. What is the best way to transfer >> what I am doing into a more "numpy way"? Essentially I am trying to call a >> model again for various different parameter combinations. The example is >> fictional, by the grid_size would ideally grow > 500 and by doing so the >> processing speed becomes very slow the way I have set things up.. >> >> thanks. >> >> example. >> >> >> import numpy as np >> >> def fake_model(data1, data2, p1, p2, p3): >> """ complete nonsense """ >> return data1 + data2 * p1 * p2 * p3 >> >> data1 = np.random.rand(10) # the size of this arrays varies might be 10 >> might be 15 etc >> data2 = np.random.rand(10) # the size of this arrays varies might be 10 >> might be 15 etc >> obs = np.random.rand(10) # the size of this arrays varies might be 10 >> might be 15 etc >> >> grid_size = 10 # Ideally this would be a large number >> param1 = np.linspace(5.0, 350, grid_size) >> param2 = np.linspace(5.0, 550, grid_size) >> param3 = np.linspace(1E-8, 10.5, grid_size) >> ss = np.zeros(0) >> >> for p1 in param1: >> for p2 in param2: >> for p3 in param3: >> ans = fake_model(data1, data2, p1, p2, p3) >> >> ss = np.append(ss, np.sum(obs - ans)**2) >> print np.sum(obs - ans)**2 >> >> > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guyer at nist.gov Mon Aug 20 09:03:43 2012 From: guyer at nist.gov (Jonathan Guyer) Date: Mon, 20 Aug 2012 09:03:43 -0400 Subject: [SciPy-User] Notification when array is modified In-Reply-To: References: Message-ID: On Aug 20, 2012, at 3:00 AM, John Jumper wrote: > I am looking for a method so that I can be notified when the contents > of a numpy array change. : : > Is there any interface in numpy so that I could expose A and B > directly as writeable numpy arrays but be notified when they are > modified? Alternatively, if I subclass ndarray, is there some > exhaustive list of methods that can change the underlying data (I > would be worried about view objects here)? Our FiPy Variable class does this in the course of setting up a mechanism for lazy expression evaluation. http://matforge.org/fipy/browser/trunk/fipy/variables/variable.py There's a lot in there that you don't need, but see _requires(), _markFresh(), and _markStale(), and the value property. See also http://matforge.org/fipy/browser/trunk/fipy/variables/operatorVariable.py http://matforge.org/fipy/browser/trunk/fipy/variables/binaryOperatorVariable.py http://matforge.org/fipy/browser/trunk/fipy/variables/unaryOperatorVariable.py There's unfortunately a lot of work to do to make it all act like an ndarray without subclassing (which was impossible when we originally wrote this and discouraged later. I don't know what the current thinking is on ndarray subclassing). From mdekauwe at gmail.com Mon Aug 20 10:28:37 2012 From: mdekauwe at gmail.com (Martin De Kauwe) Date: Mon, 20 Aug 2012 07:28:37 -0700 (PDT) Subject: [SciPy-User] How to minimize a simple multi variable function In-Reply-To: References: Message-ID: <7b492a96-88c2-40cf-9740-5c55a7b0e587@googlegroups.com> I am not sure what pulp is. But the lmfit package has options for constraints http://cars9.uchicago.edu/software/python/lmfit/ On Monday, August 20, 2012 8:41:37 PM UTC+10, Roderick Bartolo wrote: > > Hi all, > I would like to minimize the following simple function f(x,y,z) = x*y*z > with some constraints x<=20 + y and y < z+2 using Pulp. Does any body know > how this can be done please? > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From animator333 at gmail.com Mon Aug 20 05:02:44 2012 From: animator333 at gmail.com (Prashant) Date: Mon, 20 Aug 2012 02:02:44 -0700 (PDT) Subject: [SciPy-User] Vector Field Generation Message-ID: Hi, I was googling for this problem where I myself don't know that exact term to use for. It could be vector field, follow behaviour, flow field, streamlines or anything else. Please have a look at attached image "vector_field.jpg". 1. Three black rectangles represent polygons in 2d space. 2. Red color points are Master Points/Vectors in 2d space 3. Green color points are child points (no vector, only position) Based on Master points I would like to generate vectors (vector field) on child points. I have also attached an image (image1.jpg) that shows desired result. In this image resulting child vectors has smooth transition which is what I am looking for. The four white arrows can be considered as Master vectors and rest of the points are childrens. I am not sure if nearest neighbour search algorithm would be used but if it's required then we have to consider a situation. I have pointed out a child point using darker black arrow in vector_field.jpg. If we calculate nearest master point for this point, it would give point "C", where correct point would be "A" as point "c" is far away based on polygon distribution. I would really appreciate any pointers or suggestions for the problem. Best Regards Prashant -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: vector_field.jpg Type: image/jpeg Size: 15135 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image1.jpg Type: image/jpeg Size: 43218 bytes Desc: not available URL: From rodbartolo at gmail.com Mon Aug 20 06:41:37 2012 From: rodbartolo at gmail.com (Roderick Bartolo) Date: Mon, 20 Aug 2012 03:41:37 -0700 (PDT) Subject: [SciPy-User] How to minimize a simple multi variable function Message-ID: Hi all, I would like to minimize the following simple function f(x,y,z) = x*y*z with some constraints x<=20 + y and y < z+2 using Pulp. Does any body know how this can be done please? -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanforeest at gmail.com Mon Aug 20 11:56:41 2012 From: vanforeest at gmail.com (nicky van foreest) Date: Mon, 20 Aug 2012 17:56:41 +0200 Subject: [SciPy-User] How to minimize a simple multi variable function In-Reply-To: References: Message-ID: Hi, Pulp can be used to solve *linear* programming problems. Your objective function is not linear, hence I don't think you can use pulp. HTH Nicky On 20 August 2012 12:41, Roderick Bartolo wrote: > Hi all, > I would like to minimize the following simple function f(x,y,z) = x*y*z with > some constraints x<=20 + y and y < z+2 using Pulp. Does any body know how > this can be done please? > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From helmrp at yahoo.com Mon Aug 20 13:52:00 2012 From: helmrp at yahoo.com (The Helmbolds) Date: Mon, 20 Aug 2012 10:52:00 -0700 (PDT) Subject: [SciPy-User] minimization of constrained mltivariate problem Message-ID: <1345485120.4201.YahooMailNeo@web31813.mail.mud.yahoo.com> Take alook at SciPy's "optimize" stuff, especially fmin_bfgs and fmin_cobyla. The writeups on these may be hard to follow, but we are working an better documentation of these and other "optimize" routines. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ckkart at hoc.net Mon Aug 20 20:44:43 2012 From: ckkart at hoc.net (Christian K.) Date: Mon, 20 Aug 2012 21:44:43 -0300 Subject: [SciPy-User] Notification when array is modified In-Reply-To: References: Message-ID: For that purpose I use a subclass of ndarray. See the attached sample. Hope it helps. Christian Am 20.08.12 04:00, schrieb John Jumper: > I am looking for a method so that I can be notified when the contents > of a numpy array change. > > A simplified class that illustrates the desired functionality is below. > > class ComputeDerived(object): > def __init__(self, n): > self.A = np.zeros(n) > self.B = np.zeros(n) > self.change_mask = np.ones(n, dtype=np.bool) > self._Derived = expensive(self.A,self.B) > > def set_A(self, i, x): > self.A[i] = x > self.change_mask[i] = True > > def set_B(self, i, x): > self.B[i] = x > self.change_mask[i] = True > > def Derived(self): > if np.any(self.change_mask): > for i in np.nonzero(self.change_mask)[0]: > self._Derived[i] = expensive(A[i],B[i]) > self.change_mask[:] = False > return self._Derived > > This class works, but I lack the power of numpy's indexing syntax to > change A or B. I would like to be able to write > compute_derived.A[(3,5,6)] = 1 and then be notified that A has > changed. Otherwise, if I want to use a numpy interface for A and B, I > would have to store old copies of A and B and traverse these copies > every time Derived is called. > > Is there any interface in numpy so that I could expose A and B > directly as writeable numpy arrays but be notified when they are > modified? Alternatively, if I subclass ndarray, is there some > exhaustive list of methods that can change the underlying data (I > would be worried about view objects here)? > > In case it makes a difference, my actual use case is a C++ class > wrapped using Boost::Python. The numpy arrays A and B would point at > aligned float* members of my C++ class. > > Thank you, > John Jumper > -------------- next part -------------- A non-text attachment was scrubbed... Name: callback_array.py Type: text/x-python-script Size: 1357 bytes Desc: not available URL: From issa at aims.ac.za Mon Aug 20 21:00:14 2012 From: issa at aims.ac.za (Issa Karambal) Date: Tue, 21 Aug 2012 02:00:14 +0100 Subject: [SciPy-User] Avoiding inner for loops?? In-Reply-To: References: <82d5dfed-e68c-44b6-94f9-7f2557ddece4@googlegroups.com> Message-ID: This produces the same result like using loops """ params = param1*param2[:, None] fake_model = data1+data2*params1.reshape(-1)[:, None] ans = np.reshape( fake_model, (5,5,5) ) print ans[0] array([[0, 1, 2, 3, 4], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4]]) print ans[-1] array([[ 0, 1, 2, 3, 4], [ 0, 5, 10, 15, 20], [ 0, 9, 18, 27, 36], [ 0, 13, 26, 39, 52], [ 0, 17, 34, 51, 68]]) Issa On 20 August 2012 00:47, Martin De Kauwe wrote: > Perhaps simplifying, in 2D this is what I want if using loops > > def fake_model(data1, data2, p1, p2): > """ complete nonsense """ > return data1 + data2 * p1 * p2 > > grid_size = 5 > nobs = 5 > obs = np.zeros(nobs) > data1 = np.arange(nobs) > data2 = np.arange(nobs) > a = np.arange(grid_size) > b = np.arange(grid_size) > c = np.arange(grid_size) > ss = np.zeros(0) > for p1 in a: > for p2 in b: > ans = fake_model(data1, data2, p1, p2) > #ss = np.append(ss, np.sum(obs - ans)**2) > print ans > > > which would produce > > [0 1 2 3 4] > [0 1 2 3 4] > [0 1 2 3 4] > [0 1 2 3 4] > [0 1 2 3 4] > [0 1 2 3 4] > [0 2 4 6 8] > . > snip > . > [0 1 2 3 4] > [ 0 5 10 15 20] > [ 0 9 18 27 36] > [ 0 13 26 39 52] > [ 0 17 34 51 68] > > And so I figured something like... > > a = np.ones((grid_size,grid_size)) * np.arange(grid_size)[None,:] > b = np.ones((grid_size,grid_size)) * np.arange(grid_size)[:,None] > ans = fake_model(data1, data2, a, b) > > Although this doesn't seem to work, but I think this might be along the > right lines? This produces > > [[ 0. 1. 2. 3. 4.] > [ 0. 2. 6. 12. 20.] > [ 0. 3. 10. 21. 36.] > [ 0. 4. 14. 30. 52.] > [ 0. 5. 18. 39. 68.]] > > > > > > > > > > On Sunday, August 19, 2012 7:07:59 PM UTC+10, Martin De Kauwe wrote: >> >> Hi, >> >> I need to avoid (at least) two inner for loops in what I am trying to do >> otherwise my processing takes forever. What is the best way to transfer >> what I am doing into a more "numpy way"? Essentially I am trying to call a >> model again for various different parameter combinations. The example is >> fictional, by the grid_size would ideally grow > 500 and by doing so the >> processing speed becomes very slow the way I have set things up.. >> >> thanks. >> >> example. >> >> >> import numpy as np >> >> def fake_model(data1, data2, p1, p2, p3): >> """ complete nonsense """ >> return data1 + data2 * p1 * p2 * p3 >> >> data1 = np.random.rand(10) # the size of this arrays varies might be 10 >> might be 15 etc >> data2 = np.random.rand(10) # the size of this arrays varies might be 10 >> might be 15 etc >> obs = np.random.rand(10) # the size of this arrays varies might be 10 >> might be 15 etc >> >> grid_size = 10 # Ideally this would be a large number >> param1 = np.linspace(5.0, 350, grid_size) >> param2 = np.linspace(5.0, 550, grid_size) >> param3 = np.linspace(1E-8, 10.5, grid_size) >> ss = np.zeros(0) >> >> for p1 in param1: >> for p2 in param2: >> for p3 in param3: >> ans = fake_model(data1, data2, p1, p2, p3) >> >> ss = np.append(ss, np.sum(obs - ans)**2) >> print np.sum(obs - ans)**2 >> >> > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdekauwe at gmail.com Mon Aug 20 21:11:22 2012 From: mdekauwe at gmail.com (Martin De Kauwe) Date: Mon, 20 Aug 2012 18:11:22 -0700 (PDT) Subject: [SciPy-User] Avoiding inner for loops?? In-Reply-To: <82d5dfed-e68c-44b6-94f9-7f2557ddece4@googlegroups.com> References: <82d5dfed-e68c-44b6-94f9-7f2557ddece4@googlegroups.com> Message-ID: Or perhaps an easier solution would be to build the sampling grid first and then use a single for loop to run the model? import itertools import numpy as np grid_size = 500 a = np.linspace(5.0, 350, grid_size) b = np.linspace(5.0, 550, grid_size) c = np.linspace(1E-8, 10.5, grid_size) x = [] for (i,j,k) in itertools.product(a, b, c): x.extend((i,j,k)) This would achieve what I want but is again very slow, so is there a way to jump over the need for the two inner loops? thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From issa at aims.ac.za Mon Aug 20 21:27:00 2012 From: issa at aims.ac.za (Issa Karambal) Date: Tue, 21 Aug 2012 02:27:00 +0100 Subject: [SciPy-User] Avoiding inner for loops?? In-Reply-To: References: <82d5dfed-e68c-44b6-94f9-7f2557ddece4@googlegroups.com> Message-ID: perhaps you can consider 'np.ogrid' for the product. Ps: Is this code not helpful? param1 = np.arange(5) param2 = param1 params1 = param1*param2[:, None] fake_model = data1+data2*params1.reshape(- 1)[:, None] ans = np.reshape( fake_model, (5,5,5) ) print ans[0] array([[0, 1, 2, 3, 4], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4]]) print ans[1] array([[ 0, 1, 2, 3, 4], [ 0, 2, 4, 6, 8], [ 0, 3, 6, 9, 12], [ 0, 4, 8, 12, 16], [ 0, 5, 10, 15, 20]]) print ans[-1] array([[ 0, 1, 2, 3, 4], [ 0, 5, 10, 15, 20], [ 0, 9, 18, 27, 36], [ 0, 13, 26, 39, 52], [ 0, 17, 34, 51, 68]]) On 21 August 2012 02:11, Martin De Kauwe wrote: > Or perhaps an easier solution would be to build the sampling grid first > and then use a single for loop to run the model? > > import itertools > import numpy as np > > grid_size = 500 > a = np.linspace(5.0, 350, grid_size) > b = np.linspace(5.0, 550, grid_size) > c = np.linspace(1E-8, 10.5, grid_size) > > x = [] > for (i,j,k) in itertools.product(a, b, c): > x.extend((i,j,k)) > > This would achieve what I want but is again very slow, so is there a way > to jump over the need for the two inner loops? > > thanks > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jumper at uchicago.edu Mon Aug 20 23:52:51 2012 From: jumper at uchicago.edu (John Jumper) Date: Mon, 20 Aug 2012 22:52:51 -0500 Subject: [SciPy-User] Notification when array is modified In-Reply-To: References: Message-ID: Thank you all. I will probably use a proxy object, but all the responses are helpful. John On Mon, Aug 20, 2012 at 7:44 PM, Christian K. wrote: > For that purpose I use a subclass of ndarray. See the attached sample. > Hope it helps. > > Christian > > Am 20.08.12 04:00, schrieb John Jumper: >> I am looking for a method so that I can be notified when the contents >> of a numpy array change. >> >> A simplified class that illustrates the desired functionality is below. >> >> class ComputeDerived(object): >> def __init__(self, n): >> self.A = np.zeros(n) >> self.B = np.zeros(n) >> self.change_mask = np.ones(n, dtype=np.bool) >> self._Derived = expensive(self.A,self.B) >> >> def set_A(self, i, x): >> self.A[i] = x >> self.change_mask[i] = True >> >> def set_B(self, i, x): >> self.B[i] = x >> self.change_mask[i] = True >> >> def Derived(self): >> if np.any(self.change_mask): >> for i in np.nonzero(self.change_mask)[0]: >> self._Derived[i] = expensive(A[i],B[i]) >> self.change_mask[:] = False >> return self._Derived >> >> This class works, but I lack the power of numpy's indexing syntax to >> change A or B. I would like to be able to write >> compute_derived.A[(3,5,6)] = 1 and then be notified that A has >> changed. Otherwise, if I want to use a numpy interface for A and B, I >> would have to store old copies of A and B and traverse these copies >> every time Derived is called. >> >> Is there any interface in numpy so that I could expose A and B >> directly as writeable numpy arrays but be notified when they are >> modified? Alternatively, if I subclass ndarray, is there some >> exhaustive list of methods that can change the underlying data (I >> would be worried about view objects here)? >> >> In case it makes a difference, my actual use case is a C++ class >> wrapped using Boost::Python. The numpy arrays A and B would point at >> aligned float* members of my C++ class. >> >> Thank you, >> John Jumper >> > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From dan.stowell at eecs.qmul.ac.uk Tue Aug 21 05:14:30 2012 From: dan.stowell at eecs.qmul.ac.uk (Dan Stowell) Date: Tue, 21 Aug 2012 10:14:30 +0100 Subject: [SciPy-User] Vector Field Generation In-Reply-To: References: Message-ID: <50335176.8040909@eecs.qmul.ac.uk> Hi - It's not clear to me, if you are "generating" child points or "classifying" them. Your question about nearest-neighbour makes it sound like you're classifying the child points according to their parent. Nearest-neighbour search would work fine in your example BUT not in the space of location [x,y]. it looks like nearest-neighbours in a joint space of location and velocity [x,y, a * dx, a * dy] would work OK (where "a" is a scaling factor to weight position-vs-velocity); or even, just velocity alone. HTH Dan On 20/08/12 10:02, Prashant wrote: > Hi, > > I was googling for this problem where I myself don't know that exact > term to use for. It could be vector field, follow behaviour, flow field, > streamlines or anything else. > > Please have a look at attached image "vector_field.jpg". > > 1. Three black rectangles represent polygons in 2d space. > 2. Red color points are Master Points/Vectors in 2d space > 3. Green color points are child points (no vector, only position) > > Based on Master points I would like to generate vectors (vector field) > on child points. > I have also attached an image (image1.jpg) that shows desired result. In > this image resulting > child vectors has smooth transition which is what I am looking for. The > four white arrows can be considered > as Master vectors and rest of the points are childrens. > > I am not sure if nearest neighbour search algorithm would be used but if > it's required then we have to consider > a situation. > I have pointed out a child point using darker black arrow in > vector_field.jpg. If we calculate > nearest master point for this point, it would give point "C", where > correct point would be "A" as point > "c" is far away based on polygon distribution. > > I would really appreciate any pointers or suggestions for the problem. > > Best Regards > > Prashant > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Dan Stowell Postdoctoral Research Assistant Centre for Digital Music Queen Mary, University of London Mile End Road, London E1 4NS http://www.elec.qmul.ac.uk/digitalmusic/people/dans.htm http://www.mcld.co.uk/ From alind_sap at yahoo.com Tue Aug 21 08:11:56 2012 From: alind_sap at yahoo.com (alind sharma) Date: Tue, 21 Aug 2012 20:11:56 +0800 (SGT) Subject: [SciPy-User] (no subject) Message-ID: <1345551116.45477.YahooMailNeo@web190606.mail.sg3.yahoo.com> http://www.transporthirereviews.co.uk/wp-content/themes/twentyten/makoler.html?ldb=johqjba -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdekauwe at gmail.com Tue Aug 21 12:20:11 2012 From: mdekauwe at gmail.com (mdekauwe) Date: Tue, 21 Aug 2012 09:20:11 -0700 (PDT) Subject: [SciPy-User] [SciPy-user] Avoiding inner for loops?? In-Reply-To: <5030F69B.7040000@cs.wisc.edu> References: <82d5dfed-e68c-44b6-94f9-7f2557ddece4@googlegroups.com> <5030F69B.7040000@cs.wisc.edu> Message-ID: <34330080.post@talk.nabble.com> Hi Apologies I didn't see this response as I was viewing on google groups and on there it looks like no one has responded!! Odd. Sort of. I already have a minimisation script set up using the lmfit package. I was trying to sample the parameter space to find a decent starting point for my initial parameter guess. J. David Lee wrote: > > Hi Martin, > > It looks like you are trying to do a brute force minimization of your > model with three parameters, is that right? For that you could look at > scipy.optimize.brute, which will probably be faster. scipy.optimize also > has several more sophisticated minimizers that you might consider: > > http://docs.scipy.org/doc/scipy/reference/optimize.html > > If your function works with numpy arrays, you could try generating all > of your parameters at once with mgrid and calling your function on the > returned arrays, > > p1, p2, p3 = np.mgrid[:10,:10,:10] > > David > > On 08/19/2012 04:07 AM, Martin De Kauwe wrote: >> Hi, >> >> I need to avoid (at least) two inner for loops in what I am trying to >> do otherwise my processing takes forever. What is the best way to >> transfer what I am doing into a more "numpy way"? Essentially I am >> trying to call a model again for various different parameter >> combinations. The example is fictional, by the grid_size would ideally >> grow > 500 and by doing so the processing speed becomes very slow the >> way I have set things up.. >> >> thanks. >> >> example. >> >> >> import numpy as np >> >> def fake_model(data1, data2, p1, p2, p3): >> """ complete nonsense """ >> return data1 + data2 * p1 * p2 * p3 >> >> data1 = np.random.rand(10) # the size of this arrays varies might be >> 10 might be 15 etc >> data2 = np.random.rand(10) # the size of this arrays varies might be >> 10 might be 15 etc >> obs = np.random.rand(10) # the size of this arrays varies might be 10 >> might be 15 etc >> >> grid_size = 10 # Ideally this would be a large number >> param1 = np.linspace(5.0, 350, grid_size) >> param2 = np.linspace(5.0, 550, grid_size) >> param3 = np.linspace(1E-8, 10.5, grid_size) >> ss = np.zeros(0) >> >> for p1 in param1: >> for p2 in param2: >> for p3 in param3: >> ans = fake_model(data1, data2, p1, p2, p3) >> >> ss = np.append(ss, np.sum(obs - ans)**2) >> print np.sum(obs - ans)**2 > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- View this message in context: http://old.nabble.com/Avoiding-inner-for-loops---tp34319763p34330080.html Sent from the Scipy-User mailing list archive at Nabble.com. From josef.pktd at gmail.com Tue Aug 21 19:16:29 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 21 Aug 2012 19:16:29 -0400 Subject: [SciPy-User] cholesky of sparse symmetric banded matrix Message-ID: I would like to get a cholesky decomposition of a symmetric banded matrix and multiply it with a dense array I found http://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.cholesky_banded.html and this http://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.dia_matrix.html either 1) is there a way to do linear algebra (dot multiplication) directly in the "upper diagonal ordered form"? or 2) is there an efficient way to go from the "upper diagonal ordered form" to a sparse diagonal matrix (or both ways)? Is there code that uses this and that I can look at for the pattern? ------- my problem is standard linear least squares, where I have an explicit banded form for the (nobs, nobs) weighting matrix X'WX and X'Wy and I need a transformation X2 = W^(0.5) X and y2 = W^(0.5) y so I get X2'X2 and X2'y2 (nobs: number of observations, prime is transpose) My first example only has one upper and one lower off-diagonal, so working with dense is wasteful. Josef -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdekauwe at gmail.com Tue Aug 21 20:14:59 2012 From: mdekauwe at gmail.com (mdekauwe) Date: Tue, 21 Aug 2012 17:14:59 -0700 (PDT) Subject: [SciPy-User] [SciPy-user] Avoiding inner for loops?? In-Reply-To: References: <82d5dfed-e68c-44b6-94f9-7f2557ddece4@googlegroups.com> Message-ID: <34332203.post@talk.nabble.com> Hi, > You may like to utilize functionality of ix_ and newaxis, like: In []: p1, p2, p3= ix_(param1, param2, param3) In []: n_= newaxis In []: ans= fake_model(data1[:, n_, n_, n_], data2[:, n_, n_, n_], p1, p2, p3) In []: ss2= ((obs[:, n_, n_, n_]- ans).sum(0)** 2).reshape(grid_size** 3) In []: allclose(ss, ss2) Out[]: True My 2 cents, -eat This solution is exactly what I was looking for thank you. I hadn't come across np.ix_ before. -- View this message in context: http://old.nabble.com/Avoiding-inner-for-loops---tp34319763p34332203.html Sent from the Scipy-User mailing list archive at Nabble.com. From mdekauwe at gmail.com Tue Aug 21 20:52:28 2012 From: mdekauwe at gmail.com (mdekauwe) Date: Tue, 21 Aug 2012 17:52:28 -0700 (PDT) Subject: [SciPy-User] [SciPy-user] Avoiding inner for loops?? In-Reply-To: <34332203.post@talk.nabble.com> References: <82d5dfed-e68c-44b6-94f9-7f2557ddece4@googlegroups.com> <34332203.post@talk.nabble.com> Message-ID: <34332296.post@talk.nabble.com> Actually one more thing I can't figure out, is I want to use the smallest difference, ie. from ss2 to get the relevant paramater values. e.g. ss2= ((obs[:, n_, n_, n_]- ans).sum(0)** 2).reshape(grid_size** 3) index = np.argmin(ss2, 0) but this index will be from the full grid (shape=8000) and not give me something I can pull the values from the param1, param2, param3 arrays (shape=20)? thanks again. -- View this message in context: http://old.nabble.com/Avoiding-inner-for-loops---tp34319763p34332296.html Sent from the Scipy-User mailing list archive at Nabble.com. From e.antero.tammi at gmail.com Wed Aug 22 05:04:09 2012 From: e.antero.tammi at gmail.com (eat) Date: Wed, 22 Aug 2012 12:04:09 +0300 Subject: [SciPy-User] [SciPy-user] Avoiding inner for loops?? In-Reply-To: <34332296.post@talk.nabble.com> References: <82d5dfed-e68c-44b6-94f9-7f2557ddece4@googlegroups.com> <34332203.post@talk.nabble.com> <34332296.post@talk.nabble.com> Message-ID: Hi, On Wed, Aug 22, 2012 at 3:52 AM, mdekauwe wrote: > > Actually one more thing I can't figure out, is I want to use the smallest > difference, ie. from ss2 to get the relevant paramater values. > > e.g. > > ss2= ((obs[:, n_, n_, n_]- ans).sum(0)** 2).reshape(grid_size** 3) > index = np.argmin(ss2, 0) > > but this index will be from the full grid (shape=8000) and not give me > something I can pull the values from the param1, param2, param3 arrays > (shape=20)? > Perhaps In []: ss= ((obs[:, n_, n_, n_]- ans).sum(0)** 2) In []: ndx= where(ss.min()== ss) In []: ndx Out[]: (array([0]), array([0]), array([0])) In []: ss[ndx] Out[]: array([ 2500.]) Regards, -eat > > thanks again. > -- > View this message in context: > http://old.nabble.com/Avoiding-inner-for-loops---tp34319763p34332296.html > Sent from the Scipy-User mailing list archive at Nabble.com. > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanforeest at gmail.com Wed Aug 22 05:40:00 2012 From: vanforeest at gmail.com (nicky van foreest) Date: Wed, 22 Aug 2012 11:40:00 +0200 Subject: [SciPy-User] rv_discrete Message-ID: Hi, How can I find out whether a frozen rv is a subclass of rv_discrete or rv_continuous. This doesn't work: X = stats.norm(10, np.sqrt(10)) print issubclass(X.__class__, stats.rv_continuous) thanks Nicky From ralf.gommers at gmail.com Wed Aug 22 05:58:21 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 22 Aug 2012 11:58:21 +0200 Subject: [SciPy-User] rv_discrete In-Reply-To: References: Message-ID: On Wed, Aug 22, 2012 at 11:40 AM, nicky van foreest wrote: > Hi, > > How can I find out whether a frozen rv is a subclass of rv_discrete or > rv_continuous. This doesn't work: > > X = stats.norm(10, np.sqrt(10)) > print issubclass(X.__class__, stats.rv_continuous) > It's not a subclass of either one. But I guess this is what you want: In [11]: isinstance(X.dist, stats.rv_continuous) Out[11]: True In [12]: isinstance(X.dist, stats.rv_discrete) Out[12]: False Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanforeest at gmail.com Wed Aug 22 06:02:16 2012 From: vanforeest at gmail.com (nicky van foreest) Date: Wed, 22 Aug 2012 12:02:16 +0200 Subject: [SciPy-User] rv_discrete In-Reply-To: References: Message-ID: Hi Ralf, Thanks. Nicky On 22 August 2012 11:58, Ralf Gommers wrote: > > > On Wed, Aug 22, 2012 at 11:40 AM, nicky van foreest > wrote: >> >> Hi, >> >> How can I find out whether a frozen rv is a subclass of rv_discrete or >> rv_continuous. This doesn't work: >> >> X = stats.norm(10, np.sqrt(10)) >> print issubclass(X.__class__, stats.rv_continuous) > > > It's not a subclass of either one. But I guess this is what you want: > > In [11]: isinstance(X.dist, stats.rv_continuous) > Out[11]: True > > In [12]: isinstance(X.dist, stats.rv_discrete) > Out[12]: False > > Ralf > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From njs at pobox.com Wed Aug 22 10:19:25 2012 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 22 Aug 2012 15:19:25 +0100 Subject: [SciPy-User] cholesky of sparse symmetric banded matrix In-Reply-To: References: Message-ID: On Wed, Aug 22, 2012 at 12:16 AM, wrote: > I would like to get a cholesky decomposition of a symmetric banded matrix > and multiply it with a dense array > > I found > http://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.cholesky_banded.html > and this > http://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.dia_matrix.html > > either > 1) is there a way to do linear algebra (dot multiplication) directly in the > "upper diagonal ordered form"? > or > 2) is there an efficient way to go from the "upper diagonal ordered form" to > a sparse diagonal matrix (or both ways)? > > Is there code that uses this and that I can look at for the pattern? > > > ------- > my problem is standard linear least squares, where I have an explicit banded > form for the (nobs, nobs) weighting matrix > > X'WX and X'Wy > > and I need a transformation X2 = W^(0.5) X and y2 = W^(0.5) y > > so I get X2'X2 and X2'y2 > > (nobs: number of observations, prime is transpose) > My first example only has one upper and one lower off-diagonal, so working > with dense is wasteful. If you only have one off-diagonal, then you may be best off using CHOLMOD on the CSC representation: http://packages.python.org/scikits.sparse/cholmod.html http://lists.vorpus.org/pipermail/scikits-sparse-discuss/2012-August/000038.html Of course this uses GPLed code. -n From sunghwanchoi91 at gmail.com Wed Aug 22 10:56:31 2012 From: sunghwanchoi91 at gmail.com (SungHwan Choi) Date: Wed, 22 Aug 2012 23:56:31 +0900 Subject: [SciPy-User] Hi I have a quastion to weave Message-ID: Hi I am Sunghwan Choi Now, I am interested on weave. First of all, I have c code to be wrapped by python. tmp.c file #include #include int main(){ int np =3; int return_val; xc_func_type func; xc_func_init(&func, XC_GGA_X_B88, XC_UNPOLARIZED); xc_func_end(&func); return_val=np+131; printf("%d", return_val); } gcc -o exe tmp.c -L/home/shchoi/git/LagChem/LagChem/Basic/XCs/libxc-1.2.0/src/.libs/ -lxc -lm ./exe above code is my target code and now, from scipy.weave import inline np=3 code=\ """ xc_func_type func; xc_func_init(&func, XC_GGA_X_B88, XC_UNPOLARIZED); xc_func_end(&func); return_val=np+131; """ print inline(code,['np'],include_dirs = ["/home/shchoi/git/LagChem/LagChem/Basic/XCs/libxc-1.2.0/src/"],library_dirs=["/home/shchoi/git/LagChem/LagChem/Basic/XCs/libxc-1.2.0/src/.libs"],libraries=["m","xc"],headers=["",""],compiler='gcc') print np This is my solution but it give me a error In file included from /usr/lib64/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1975:0, from /usr/lib64/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17, from /usr/lib64/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:15, from /home/shchoi/.python27_compiled/sc_a383cf366f9347f9b6ff017e9aab78882.cpp:22: /usr/lib64/python2.7/site-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" /usr/bin/ld: /home/shchoi/git/LagChem/LagChem/Basic/XCs/libxc-1.2.0/src/.libs/libxc.a(libxc_la-functionals.o): relocation R_X86_64_32 against `xc_functional_keys' can not be used when making a shared object; recompile with -fPIC /home/shchoi/git/LagChem/LagChem/Basic/XCs/libxc-1.2.0/src/.libs/libxc.a: could not read symbols: Bad value collect2: ld returned 1 exit status /usr/bin/ld: /home/shchoi/git/LagChem/LagChem/Basic/XCs/libxc-1.2.0/src/.libs/libxc.a(libxc_la-functionals.o): relocation R_X86_64_32 against `xc_functional_keys' can not be used when making a shared object; recompile with -fPIC /home/shchoi/git/LagChem/LagChem/Basic/XCs/libxc-1.2.0/src/.libs/libxc.a: could not read symbols: Bad value collect2: ld returned 1 exit status Traceback (most recent call last): File "tmp.py", line 12, in print inline(code,['np'],include_dirs = ["/home/shchoi/git/LagChem/LagChem/Basic/XCs/libxc-1.2.0/src/"],library_dirs=["/home/shchoi/git/LagChem/LagChem/Basic/XCs/libxc-1.2.0/src/.libs"],libraries=["m","xc"],headers=["",""],compiler='gcc') File "/usr/lib64/python2.7/site-packages/scipy/weave/inline_tools.py", line 355, in inline **kw) File "/usr/lib64/python2.7/site-packages/scipy/weave/inline_tools.py", line 482, in compile_function verbose=verbose, **kw) File "/usr/lib64/python2.7/site-packages/scipy/weave/ext_tools.py", line 367, in compile verbose = verbose, **kw) File "/usr/lib64/python2.7/site-packages/scipy/weave/build_tools.py", line 272, in build_extension setup(name = module_name, ext_modules = [ext],verbose=verb) File "/usr/lib64/python2.7/site-packages/numpy/distutils/core.py", line 186, in setup return old_setup(**new_attr) File "/usr/lib64/python2.7/distutils/core.py", line 169, in setup raise SystemExit, "error: " + str(msg) scipy.weave.build_tools.CompileError: error: Command "g++ -pthread -shared /tmp/shchoi/python27_intermediate/compiler_81874d258832299909898bc232bd652b/home/shchoi/.python27_compiled/sc_a383cf366f9347f9b6ff017e9aab78882.o /tmp/shchoi/python27_intermediate/compiler_81874d258832299909898bc232bd652b/usr/lib64/python2.7/site-packages/scipy/weave/scxx/weave_imp.o -L/home/shchoi/git/LagChem/LagChem/Basic/XCs/libxc-1.2.0/src/.libs -L/usr/lib64 -lm -lxc -lpython2.7 -o /home/shchoi/.python27_compiled/sc_a383cf366f9347f9b6ff017e9aab78882.so" failed with exit status 1 I have tried to solve for a while but now I give up. please help me anyone~ -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanforeest at gmail.com Wed Aug 22 16:49:47 2012 From: vanforeest at gmail.com (nicky van foreest) Date: Wed, 22 Aug 2012 22:49:47 +0200 Subject: [SciPy-User] generic expectation operator Message-ID: Hi, For numerous purposes I need to compute the expectation of some function with respect to some probability distribution. i.e. E f(X) = \int f(x) dG(x), where f is the function and G the distribution. When G is continuous, this is simple, just use scipy.integrate.quad. However, if G is discrete, things become more complicated since quad often misses the spikes in the density. To deal with this problem, in part, I came up with the following code, but it is not completely ok yet, see below. #!/usr/bin/env python import scipy.stats as stats import scipy.integrate from math import sqrt def E(f, X): if hasattr(X,'dist'): if isinstance(X.dist, stats.rv_discrete): # hint from Ralph g = X.pmf else: g = X.pdf return scipy.integrate.quad(lambda x: x*g(x), X.dist.a, X.dist.b) else: return sum(f(x)*p for x,p in zip(X.xk, X.pk)) # now the following works: X = stats.norm(3,sqrt(3)) print E(sqrt, X) # this is ok too grid = [50, 100, 150, 200, 250, 300] probs=[2,3,2,1.,0.5,0.5] tot = sum(probs) probs = [p/tot for p in probs] X = stats.rv_discrete(values=(grid,probs), name='sizeDist') print E(sqrt, X) # but this is not ok X = stats.poisson(3) print E(sqrt, X) ------------------------------------------ This is the output: 11.0383463182 (7.771634331185016e-19, 1.916327339868129e-17) (2.9999999999999996, 1.1770368678494054e-08) So indeed, quad misses the spikes in the poisson distribution. Is there any nice way to resolve this? Moreover, is there a better way to build the expectation operator? Perhaps, if all works, it would be a useful add-on to distributions.py, or else I can include it in the distribution tutorial, if other people also think this is useful. thanks Nicky From rainexpected at theo.to Wed Aug 22 16:53:14 2012 From: rainexpected at theo.to (Ted To) Date: Wed, 22 Aug 2012 16:53:14 -0400 Subject: [SciPy-User] Constrained optimization Message-ID: <503546BA.8060603@theo.to> Hi, The documentation for the constrained optimization routines seems pretty sparse and the googling I did revealed nothing in the way of examples of usage. I defined a function W(q) where q is a vector of length 2. Using fmin obviously fails (too many iterations) but works in the sense that it doesn't outright fail with an error. Using one of the constrained optimizers, fmin_l_bfgs_b for example, completely chokes with the Traceback output at http://pastebin.com/t8tU8YBm. The commands leading to this are: x0=[.1,.1] fmin_l_bfgs_b(W,x0,bounds=[(0,1),(0,1)]) so that the intention is to bound q[0] and q[1] between 0 and 1. Any help with this would be much appreciated. Cheers, Ted From gael.varoquaux at normalesup.org Wed Aug 22 17:01:41 2012 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 22 Aug 2012 23:01:41 +0200 Subject: [SciPy-User] Constrained optimization In-Reply-To: <503546BA.8060603@theo.to> References: <503546BA.8060603@theo.to> Message-ID: <20120822210141.GA22432@phare.normalesup.org> On Wed, Aug 22, 2012 at 04:53:14PM -0400, Ted To wrote: > The documentation for the constrained optimization routines seems pretty > sparse and the googling I did revealed nothing in the way of examples of > usage. I defined a function W(q) where q is a vector of length 2. > Using fmin obviously fails (too many iterations) but works in the sense > that it doesn't outright fail with an error. Using one of the > constrained optimizers, fmin_l_bfgs_b for example, completely chokes > with the Traceback output at http://pastebin.com/t8tU8YBm. The commands > leading to this are: > x0=[.1,.1] > fmin_l_bfgs_b(W,x0,bounds=[(0,1),(0,1)]) I think that your problem is simply one of reading more carefully the documentation for fmin_l_bfgs_b: if your function 'W' does not return both the value of the function and of the gradient, and if you do not specify the gradient, you must specify approx_grad=1. Hope this helps, Gael From josef.pktd at gmail.com Wed Aug 22 17:03:02 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 22 Aug 2012 17:03:02 -0400 Subject: [SciPy-User] generic expectation operator In-Reply-To: References: Message-ID: On Wed, Aug 22, 2012 at 4:49 PM, nicky van foreest wrote: > Hi, > > For numerous purposes I need to compute the expectation of some > function with respect to some probability distribution. i.e. E f(X) = > \int f(x) dG(x), where f is the function and G the distribution. When > G is continuous, this is simple, just use scipy.integrate.quad. > However, if G is discrete, things become more complicated since quad > often misses the spikes in the density. To deal with this problem, in > part, I came up with the following code, but it is not completely ok > yet, see below. > > #!/usr/bin/env python > > import scipy.stats as stats > import scipy.integrate > from math import sqrt > > def E(f, X): > if hasattr(X,'dist'): > if isinstance(X.dist, stats.rv_discrete): # hint from Ralph > g = X.pmf > else: > g = X.pdf > return scipy.integrate.quad(lambda x: x*g(x), X.dist.a, X.dist.b) > else: > return sum(f(x)*p for x,p in zip(X.xk, X.pk)) > > # now the following works: > X = stats.norm(3,sqrt(3)) > print E(sqrt, X) > > # this is ok too > grid = [50, 100, 150, 200, 250, 300] > probs=[2,3,2,1.,0.5,0.5] > tot = sum(probs) > probs = [p/tot for p in probs] > X = stats.rv_discrete(values=(grid,probs), name='sizeDist') > print E(sqrt, X) > > # but this is not ok > > X = stats.poisson(3) > print E(sqrt, X) > > ------------------------------------------ > > This is the output: > > 11.0383463182 > (7.771634331185016e-19, 1.916327339868129e-17) > (2.9999999999999996, 1.1770368678494054e-08) > > So indeed, quad misses the spikes in the poisson distribution. > > Is there any nice way to resolve this? Moreover, is there a better way > to build the expectation operator? Perhaps, if all works, it would be > a useful add-on to distributions.py, or else I can include it in the > distribution tutorial, if other people also think this is useful. quad works only for continuous functions did you look at >>> stats.poisson.expect > >>> stats.norm.expect > improvements welcome, especially making it more robust Josef > > thanks > > Nicky > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From a.klein at science-applied.nl Wed Aug 22 18:37:16 2012 From: a.klein at science-applied.nl (Almar Klein) Date: Thu, 23 Aug 2012 00:37:16 +0200 Subject: [SciPy-User] ANN: visvis 1.7 - The object oriented approach to visualization - now also in Py3k Message-ID: Dear all, On behalf of the Visvis development team, I'm pleased to announce version 1.7 of Visvis - the object oriented approach to visualization. Visvis is a pure Python library for visualization of 1D to 4D data in an object oriented way. Essentially, visvis is an object oriented layer of Python on top of OpenGl, thereby combining the power of OpenGl with the usability of Python. A Matlab-like interface in the form of a set of functions allows easy creation of objects (e.g. plot(), imshow(), volshow(), surf()). The most notable changes are: - support for Py3k - a PySide backend - full Unicode support and prettier rendering of text (by using FreeType) Website: https://code.google.com/p/visvis/ Discussion group: http://groups.google.com/group/visvis Release notes: https://code.google.com/p/visvis/wiki/releaseNotes Regards, Almar -- Almar Klein, PhD Science Applied phone: +31 6 19268652 e-mail: a.klein at science-applied.nl -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Aug 22 19:48:56 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 22 Aug 2012 19:48:56 -0400 Subject: [SciPy-User] generic expectation operator In-Reply-To: References: Message-ID: On Wed, Aug 22, 2012 at 5:03 PM, wrote: > On Wed, Aug 22, 2012 at 4:49 PM, nicky van foreest wrote: >> Hi, >> >> For numerous purposes I need to compute the expectation of some >> function with respect to some probability distribution. i.e. E f(X) = >> \int f(x) dG(x), where f is the function and G the distribution. When >> G is continuous, this is simple, just use scipy.integrate.quad. >> However, if G is discrete, things become more complicated since quad >> often misses the spikes in the density. To deal with this problem, in >> part, I came up with the following code, but it is not completely ok >> yet, see below. >> >> #!/usr/bin/env python >> >> import scipy.stats as stats >> import scipy.integrate >> from math import sqrt >> >> def E(f, X): >> if hasattr(X,'dist'): >> if isinstance(X.dist, stats.rv_discrete): # hint from Ralph >> g = X.pmf >> else: >> g = X.pdf >> return scipy.integrate.quad(lambda x: x*g(x), X.dist.a, X.dist.b) >> else: >> return sum(f(x)*p for x,p in zip(X.xk, X.pk)) >> >> # now the following works: >> X = stats.norm(3,sqrt(3)) >> print E(sqrt, X) I don't know why this works, sqrt of a normal distributed random variable has lots of nans (or complex numbers) >>> f = lambda x: np.sqrt(np.maximum(x,0)) >>> stats.norm.expect(f, loc=3, scale=np.sqrt(3)) 1.6394037171420484 >>> f = lambda x: np.sqrt(np.abs(x)) >>> stats.norm.expect(f, loc=3, scale=np.sqrt(3)) 1.6708150951924068 >> >> # this is ok too >> grid = [50, 100, 150, 200, 250, 300] >> probs=[2,3,2,1.,0.5,0.5] >> tot = sum(probs) >> probs = [p/tot for p in probs] >> X = stats.rv_discrete(values=(grid,probs), name='sizeDist') >> print E(sqrt, X) >> >> # but this is not ok >> >> X = stats.poisson(3) >> print E(sqrt, X) >>> stats.poisson.expect(np.sqrt, args=(3,)) 1.6309535375094115 Josef >> >> ------------------------------------------ >> >> This is the output: >> >> 11.0383463182 >> (7.771634331185016e-19, 1.916327339868129e-17) >> (2.9999999999999996, 1.1770368678494054e-08) >> >> So indeed, quad misses the spikes in the poisson distribution. >> >> Is there any nice way to resolve this? Moreover, is there a better way >> to build the expectation operator? Perhaps, if all works, it would be >> a useful add-on to distributions.py, or else I can include it in the >> distribution tutorial, if other people also think this is useful. > > quad works only for continuous functions > > did you look at > >>>> stats.poisson.expect > > >>>> stats.norm.expect > object at 0x031EB890>> > > improvements welcome, especially making it more robust > > Josef > >> >> thanks >> >> Nicky >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user From josef.pktd at gmail.com Thu Aug 23 08:19:20 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 23 Aug 2012 08:19:20 -0400 Subject: [SciPy-User] cholesky of sparse symmetric banded matrix In-Reply-To: References: Message-ID: On Wed, Aug 22, 2012 at 10:19 AM, Nathaniel Smith wrote: > On Wed, Aug 22, 2012 at 12:16 AM, wrote: >> I would like to get a cholesky decomposition of a symmetric banded matrix >> and multiply it with a dense array >> >> I found >> http://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.cholesky_banded.html >> and this >> http://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.dia_matrix.html >> >> either >> 1) is there a way to do linear algebra (dot multiplication) directly in the >> "upper diagonal ordered form"? >> or >> 2) is there an efficient way to go from the "upper diagonal ordered form" to >> a sparse diagonal matrix (or both ways)? >> >> Is there code that uses this and that I can look at for the pattern? >> >> >> ------- >> my problem is standard linear least squares, where I have an explicit banded >> form for the (nobs, nobs) weighting matrix >> >> X'WX and X'Wy >> >> and I need a transformation X2 = W^(0.5) X and y2 = W^(0.5) y >> >> so I get X2'X2 and X2'y2 >> >> (nobs: number of observations, prime is transpose) >> My first example only has one upper and one lower off-diagonal, so working >> with dense is wasteful. > > If you only have one off-diagonal, then you may be best off using > CHOLMOD on the CSC representation: > http://packages.python.org/scikits.sparse/cholmod.html > http://lists.vorpus.org/pipermail/scikits-sparse-discuss/2012-August/000038.html > > Of course this uses GPLed code. Thanks, which makes it however difficult or impossible to use with statsmodels I ended up doing the transformation just with numpy: with banded matrix, I only need to multiply with diagonals and add something like this for my special case def chol_vinv_diags(self, diags): '''diags is list of 2 diagonals ''' #what's minimum scipy version ? from scipy.linalg import cholesky_banded #use that we only have one off-diagonal band_low = np.concatenate((diags[0], diags[1], [0])).reshape(2,-1) result = cholesky_banded(band_low, lower=True) return result[0], result[1][:-1] def whiten(self, x): ''' special case diags is banded symmetric with 1 off-diagonal ''' diags = self.vinv_diags() chdiags = self.chol_vinv_diags(diags) if x.ndim == 2: chdiags = [chdiags[0][:,None], chdiags[1][:,None]] #cholesky has only diagonal and one lower off-diagonal res = x * chdiags[0] res[:-1] += x[1:] * chdiags[1] return res Josef > > -n > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From vanforeest at gmail.com Thu Aug 23 12:36:00 2012 From: vanforeest at gmail.com (nicky van foreest) Date: Thu, 23 Aug 2012 18:36:00 +0200 Subject: [SciPy-User] generic expectation operator In-Reply-To: References: Message-ID: Hi Josef, Thanks for your answers. >>> return scipy.integrate.quad(lambda x: x*g(x), X.dist.a, X.dist.b) >>> X = stats.norm(3,sqrt(3)) >>> print E(sqrt, X) > I don't know why this works, sqrt of a normal distributed random > variable has lots of nans (or complex numbers) You are right, it shouldn't work. The point is that my code above is this: lambda x: x*g(x), while it should have been lambda x: f(x)*g(x). I'll give expect a try. Nicky From rainexpected at theo.to Thu Aug 23 12:37:40 2012 From: rainexpected at theo.to (Ted To) Date: Thu, 23 Aug 2012 12:37:40 -0400 Subject: [SciPy-User] Constrained optimization In-Reply-To: <20120822210141.GA22432@phare.normalesup.org> References: <503546BA.8060603@theo.to> <20120822210141.GA22432@phare.normalesup.org> Message-ID: <50365C54.8070201@theo.to> On 08/22/2012 05:01 PM, Gael Varoquaux wrote: > On Wed, Aug 22, 2012 at 04:53:14PM -0400, Ted To wrote: >> The documentation for the constrained optimization routines seems pretty >> sparse and the googling I did revealed nothing in the way of examples of >> usage. I defined a function W(q) where q is a vector of length 2. >> Using fmin obviously fails (too many iterations) but works in the sense >> that it doesn't outright fail with an error. Using one of the >> constrained optimizers, fmin_l_bfgs_b for example, completely chokes >> with the Traceback output at http://pastebin.com/t8tU8YBm. The commands >> leading to this are: > >> x0=[.1,.1] >> fmin_l_bfgs_b(W,x0,bounds=[(0,1),(0,1)]) > > I think that your problem is simply one of reading more carefully the > documentation for fmin_l_bfgs_b: if your function 'W' does not return > both the value of the function and of the gradient, and if you do not > specify the gradient, you must specify approx_grad=1. > > Hope this helps, Worked perfectly! Many thanks! From vanforeest at gmail.com Thu Aug 23 13:11:58 2012 From: vanforeest at gmail.com (nicky van foreest) Date: Thu, 23 Aug 2012 19:11:58 +0200 Subject: [SciPy-User] generic expectation operator In-Reply-To: References: Message-ID: Hi, I noticed that rv_frozen does not have an expect attribute. Is there a design reason for this? It feels somewhat unnatural to me that frozen distributions cannot be used directly to compute expectations. bye Nicky On 23 August 2012 18:36, nicky van foreest wrote: > Hi Josef, > > Thanks for your answers. > >>>> return scipy.integrate.quad(lambda x: x*g(x), X.dist.a, X.dist.b) > >>>> X = stats.norm(3,sqrt(3)) >>>> print E(sqrt, X) > >> I don't know why this works, sqrt of a normal distributed random >> variable has lots of nans (or complex numbers) > > You are right, it shouldn't work. The point is that my code above is > this: lambda x: x*g(x), while it should have been lambda x: f(x)*g(x). > > I'll give expect a try. > > Nicky From josef.pktd at gmail.com Thu Aug 23 13:24:26 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 23 Aug 2012 13:24:26 -0400 Subject: [SciPy-User] generic expectation operator In-Reply-To: References: Message-ID: On Thu, Aug 23, 2012 at 1:11 PM, nicky van foreest wrote: > Hi, > > I noticed that rv_frozen does not have an expect attribute. Is there a > design reason for this? It feels somewhat unnatural to me that frozen > distributions cannot be used directly to compute expectations. oversight to add it, I guess. can be added and it can delegate like in the other methods. PR welcome. frozen_dist.dist.expect(..) (I almost never work with frozen distribution and didn't see this.) Josef > > bye > > Nicky > > > > On 23 August 2012 18:36, nicky van foreest wrote: >> Hi Josef, >> >> Thanks for your answers. >> >>>>> return scipy.integrate.quad(lambda x: x*g(x), X.dist.a, X.dist.b) >> >>>>> X = stats.norm(3,sqrt(3)) >>>>> print E(sqrt, X) >> >>> I don't know why this works, sqrt of a normal distributed random >>> variable has lots of nans (or complex numbers) >> >> You are right, it shouldn't work. The point is that my code above is >> this: lambda x: x*g(x), while it should have been lambda x: f(x)*g(x). >> >> I'll give expect a try. >> >> Nicky > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From josef.pktd at gmail.com Thu Aug 23 13:30:00 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 23 Aug 2012 13:30:00 -0400 Subject: [SciPy-User] generic expectation operator In-Reply-To: References: Message-ID: On Thu, Aug 23, 2012 at 1:24 PM, wrote: > On Thu, Aug 23, 2012 at 1:11 PM, nicky van foreest wrote: >> Hi, >> >> I noticed that rv_frozen does not have an expect attribute. Is there a >> design reason for this? It feels somewhat unnatural to me that frozen >> distributions cannot be used directly to compute expectations. > > oversight to add it, I guess. > > can be added and it can delegate like in the other methods. PR welcome. > frozen_dist.dist.expect(..) > > (I almost never work with frozen distribution and didn't see this.) >>> miss = set(dir(stats.norm)) - set(dir(stats.norm(loc=5))) >>> [i for i in miss if not i[0]=='_'] ['moment_type', 'xb', 'xa', 'freeze', 'badvalue', 'expect', 'xtol', 'vecentropy', 'fit_loc_scale', 'expandarr', 'fit', 'vecfunc', 'est_loc_scale', 'logsf', 'logcdf', 'logpdf', 'generic_moment', 'numargs', 'name', 'a', 'nnlf', 'b', 'm', 'shapes', 'extradoc', 'veccdf'] >>> import scipy >>> scipy.__version__ '0.9.0' 'logsf', 'logcdf', 'logpdf', are also missing (in my version of scipy) Josef > > Josef > > >> >> bye >> >> Nicky >> >> >> >> On 23 August 2012 18:36, nicky van foreest wrote: >>> Hi Josef, >>> >>> Thanks for your answers. >>> >>>>>> return scipy.integrate.quad(lambda x: x*g(x), X.dist.a, X.dist.b) >>> >>>>>> X = stats.norm(3,sqrt(3)) >>>>>> print E(sqrt, X) >>> >>>> I don't know why this works, sqrt of a normal distributed random >>>> variable has lots of nans (or complex numbers) >>> >>> You are right, it shouldn't work. The point is that my code above is >>> this: lambda x: x*g(x), while it should have been lambda x: f(x)*g(x). >>> >>> I'll give expect a try. >>> >>> Nicky >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user From vanforeest at gmail.com Thu Aug 23 14:09:24 2012 From: vanforeest at gmail.com (nicky van foreest) Date: Thu, 23 Aug 2012 20:09:24 +0200 Subject: [SciPy-User] generic expectation operator In-Reply-To: References: Message-ID: Hi Josef, > oversight to add it, I guess. Ok. > > can be added and it can delegate like in the other methods. PR welcome. > frozen_dist.dist.expect(..) oops. I saw this request coming. While it is trivial to implement (nearly a copy paste from the other examples in rv_frozen), I have to admit that working with git takes me a considerable amount of time, as I hardly use it. > > (I almost never work with frozen distribution and didn't see this.) Is there a specific reason for this? I find it very convenient. For the moment I solve the problem with the expectation with this function: def expect(X, f, lb, ub): if hasattr(X, 'dist'): return X.dist.expect(f, lb = lb, ub = ub) else: return X.expect(f, lb = lb, ub = ub) Like this I can pass frozen distributions and the ones that I make with rv_discrete. As your suggestion above is not yet part of standard scipy, I 'll use this function to show my students how convenient python is. Nicky From fperez.net at gmail.com Thu Aug 23 14:26:49 2012 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 23 Aug 2012 11:26:49 -0700 Subject: [SciPy-User] [ANN] Call for abstracts: BigData minisymposium at CSE'13, February 2013, Boston Message-ID: Dear colleagues, next year's SIAM conference on Computational Science and Engineering, CSE'13, will take place in Boston, February 25-March 1 (http://www.siam.org/meetings/cse13), and for this version there will be a track focused on the topic of Big Data. This term has rapidly risen in recent discussions of science and even of mainstream business computing, and for good reasons. Today virtually all disciplines are facing a flood of quantitative information whose volumes have often grown faster than the quality of our tools for extracting insight from these data. SIAM hopes that CSE'13 will provide an excellent venue for discussing these problems, from the vantage point offered by a community whose expertise combines analytical insights, algorithmic development, software engineering and domain-specific applications. As part of this event, Titus Brown (http://ged.msu.edu) and I are organizing a minisymposium where we would like to have a group of presentations that address both novel algorithmic ideas and computational approaches as well as domain-specific problems. Data doesn't appear in a vacuum, and data from different domains presents a mix of common problems along with questions that may be specific to each; we hope that by engaging a dialog between those working on algorithmic and implementation questions and those with specific problems from the field, valuable insights can be obtained. If you would like to contribute to this minisymposium, please contact us directly at: "C. Titus Brown" , "Fernando Perez" with your name and affiliation, the title of your proposed talk and a brief description (actual abstracts are due later so an informal description will suffice for now), by Wednesday August 29. For more details on the submission process, see: http://www.siam.org/meetings/cse13/submissions.php Please forward this to any interested colleagues. Regards, Titus and Fernando. From mount.sarah at gmail.com Thu Aug 23 14:29:09 2012 From: mount.sarah at gmail.com (Sarah Mount) Date: Thu, 23 Aug 2012 19:29:09 +0100 Subject: [SciPy-User] scipy.ndimage.measurements.label Message-ID: Hi all, Can someone please tell me what algorithm scipy.ndimage.measurements.label implements? Many thanks, Sarah -- Sarah Mount, Senior Lecturer, University of Wolverhampton website: http://www.snim2.org/ twitter: @snim2 From josef.pktd at gmail.com Thu Aug 23 15:07:21 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 23 Aug 2012 15:07:21 -0400 Subject: [SciPy-User] generic expectation operator In-Reply-To: References: Message-ID: On Thu, Aug 23, 2012 at 2:09 PM, nicky van foreest wrote: > Hi Josef, > >> oversight to add it, I guess. > > Ok. > >> >> can be added and it can delegate like in the other methods. PR welcome. >> frozen_dist.dist.expect(..) > > oops. I saw this request coming. While it is trivial to implement > (nearly a copy paste from the other examples in rv_frozen), I have to > admit that working with git takes me a considerable amount of time, as > I hardly use it. I'm using mostly git gui so I can keep the number of git commands, sub-commands and options that I have to remember or to look up to a small number. Then it's ok. > >> >> (I almost never work with frozen distribution and didn't see this.) > > Is there a specific reason for this? I find it very convenient. habit, unnecessary detour and useless for fitting. > > For the moment I solve the problem with the expectation with this function: > > def expect(X, f, lb, ub): > if hasattr(X, 'dist'): > return X.dist.expect(f, lb = lb, ub = ub) > else: > return X.expect(f, lb = lb, ub = ub) > > Like this I can pass frozen distributions and the ones that I make > with rv_discrete. As your suggestion above is not yet part of > standard scipy, I 'll use this function to show my students how > convenient python is. If you run into problems with the discrete expect, then please report them or file a ticket (if not a PR). I have no idea how much usage it has seen, and it's based on a truncated sum, that has problems in some cases (edge cases or cases where tails are important). Josef > > Nicky > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From josef.pktd at gmail.com Thu Aug 23 15:34:41 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 23 Aug 2012 15:34:41 -0400 Subject: [SciPy-User] cholesky of sparse symmetric banded matrix In-Reply-To: References: Message-ID: On Thu, Aug 23, 2012 at 8:19 AM, wrote: > On Wed, Aug 22, 2012 at 10:19 AM, Nathaniel Smith wrote: >> On Wed, Aug 22, 2012 at 12:16 AM, wrote: >>> I would like to get a cholesky decomposition of a symmetric banded matrix >>> and multiply it with a dense array >>> >>> I found >>> http://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.cholesky_banded.html >>> and this >>> http://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.dia_matrix.html >>> >>> either >>> 1) is there a way to do linear algebra (dot multiplication) directly in the >>> "upper diagonal ordered form"? >>> or >>> 2) is there an efficient way to go from the "upper diagonal ordered form" to >>> a sparse diagonal matrix (or both ways)? >>> >>> Is there code that uses this and that I can look at for the pattern? >>> >>> >>> ------- >>> my problem is standard linear least squares, where I have an explicit banded >>> form for the (nobs, nobs) weighting matrix >>> >>> X'WX and X'Wy >>> >>> and I need a transformation X2 = W^(0.5) X and y2 = W^(0.5) y >>> >>> so I get X2'X2 and X2'y2 >>> >>> (nobs: number of observations, prime is transpose) >>> My first example only has one upper and one lower off-diagonal, so working >>> with dense is wasteful. >> >> If you only have one off-diagonal, then you may be best off using >> CHOLMOD on the CSC representation: >> http://packages.python.org/scikits.sparse/cholmod.html >> http://lists.vorpus.org/pipermail/scikits-sparse-discuss/2012-August/000038.html >> >> Of course this uses GPLed code. > > Thanks, which makes it however difficult or impossible to use with statsmodels > > I ended up doing the transformation just with numpy: with banded > matrix, I only need to multiply with diagonals and add > > something like this for my special case > > def chol_vinv_diags(self, diags): > '''diags is list of 2 diagonals > ''' > #what's minimum scipy version ? > from scipy.linalg import cholesky_banded > > #use that we only have one off-diagonal > band_low = np.concatenate((diags[0], diags[1], [0])).reshape(2,-1) > result = cholesky_banded(band_low, lower=True) > return result[0], result[1][:-1] > > def whiten(self, x): > ''' > > special case diags is banded symmetric with 1 off-diagonal > > ''' > diags = self.vinv_diags() > chdiags = self.chol_vinv_diags(diags) > if x.ndim == 2: > chdiags = [chdiags[0][:,None], chdiags[1][:,None]] > > #cholesky has only diagonal and one lower off-diagonal > res = x * chdiags[0] > res[:-1] += x[1:] * chdiags[1] > return res usage, in case anyone is interested: estimating loc and scale of a distribution with no or fixed shape parameters using empirical quantiles. works also in the case when variance is infinite cauchy, t(2) and can also use robust estimation if there are outliers (reference paper is on Weibull) https://github.com/josef-pkt/statsmodels/commit/f59ecd2f1903b91a58d47f94496c44adff4333a1#L0R191 uses statsmodels Josef > > Josef > >> >> -n >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user From chrisfs59 at hotmail.com Fri Aug 24 16:52:39 2012 From: chrisfs59 at hotmail.com (Chris Stehlik) Date: Fri, 24 Aug 2012 13:52:39 -0700 Subject: [SciPy-User] Scipy.test() crashes Idle Message-ID: Hi, I am new to scipy and python itself. I was told to run scipy.test() to check it. I have a pc running Windows XP with the Enthought Free Python distribution 2.7.3 with Numpy 1.6.1 and Scipy 0.10.1. When I run scipy.test() in Idle, it crashes part way through. Numpy.test() goes through with no problems, but scipy.test() crashes each time. How does one go about reporting a bug one this ? Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott.sinclair.za at gmail.com Sat Aug 25 02:58:45 2012 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Sat, 25 Aug 2012 08:58:45 +0200 Subject: [SciPy-User] Scipy.test() crashes Idle In-Reply-To: References: Message-ID: On Aug 24, 2012 10:52 PM, "Chris Stehlik" wrote: > > Hi, > > I am new to scipy and python itself. I was told to run scipy.test() to check it. I have a pc running Windows XP with the Enthought Free Python distribution 2.7.3 with Numpy 1.6.1 and Scipy 0.10.1. When I run scipy.test() in Idle, it crashes part way through. Numpy.test() goes through with no problems, but scipy.test() crashes each time. > How does one go about reporting a bug one this ? What happens if you open a command prompt and run: python -c "import scipy; scipy.test()" Cheers, Scott -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sat Aug 25 10:10:03 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 25 Aug 2012 10:10:03 -0400 Subject: [SciPy-User] Scipy.test() crashes Idle In-Reply-To: References: Message-ID: On Sat, Aug 25, 2012 at 2:58 AM, Scott Sinclair wrote: > On Aug 24, 2012 10:52 PM, "Chris Stehlik" wrote: >> >> Hi, >> >> I am new to scipy and python itself. I was told to run scipy.test() to >> check it. I have a pc running Windows XP with the Enthought Free Python >> distribution 2.7.3 with Numpy 1.6.1 and Scipy 0.10.1. When I run >> scipy.test() in Idle, it crashes part way through. Numpy.test() goes through >> with no problems, but scipy.test() crashes each time. >> How does one go about reporting a bug one this ? > > What happens if you open a command prompt and run: > > python -c "import scipy; scipy.test()" a question given this and 2 tickets, and since I haven't updated scipy yet scipy 0.10.1 does scipy.test() now run also weave? or the slow tests. weave tests (compiled with mingw) have been crashing on windows for ages in my experience, so I never run them. Josef > > Cheers, > Scott > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From ralf.gommers at gmail.com Sat Aug 25 10:13:29 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 25 Aug 2012 16:13:29 +0200 Subject: [SciPy-User] Scipy.test() crashes Idle In-Reply-To: References: Message-ID: On Sat, Aug 25, 2012 at 4:10 PM, wrote: > On Sat, Aug 25, 2012 at 2:58 AM, Scott Sinclair > wrote: > > On Aug 24, 2012 10:52 PM, "Chris Stehlik" wrote: > >> > >> Hi, > >> > >> I am new to scipy and python itself. I was told to run scipy.test() to > >> check it. I have a pc running Windows XP with the Enthought Free Python > >> distribution 2.7.3 with Numpy 1.6.1 and Scipy 0.10.1. When I run > >> scipy.test() in Idle, it crashes part way through. Numpy.test() goes > through > >> with no problems, but scipy.test() crashes each time. > >> How does one go about reporting a bug one this ? > > > > What happens if you open a command prompt and run: > > > > python -c "import scipy; scipy.test()" > > a question given this and 2 tickets, and since I haven't updated scipy yet > > scipy 0.10.1 > does scipy.test() now run also weave? or the slow tests. > That should run also the weave tests, but not tests marked as slow. > > weave tests (compiled with mingw) have been crashing on windows for > ages in my experience, so I never run them. > I haven't seen any crashes or failures. If you do, please open a ticket. Ralf > Josef > > > > > > Cheers, > > Scott > > > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chrisfs59 at hotmail.com Sat Aug 25 13:51:23 2012 From: chrisfs59 at hotmail.com (Chris Stehlik) Date: Sat, 25 Aug 2012 10:51:23 -0700 Subject: [SciPy-User] SciPy crashes Idle In-Reply-To: References: Message-ID: I ran as scipy.test(verbose=2) and it crashed at test_random_tall (test_decomp.TestQR) ... I opened a ticket. It's #1723. The whole file is long, so I won't post it here unless people specifically ask. > From: scipy-user-request at scipy.org > Subject: SciPy-User Digest, Vol 108, Issue 42 > To: scipy-user at scipy.org > Date: Sat, 25 Aug 2012 12:00:14 -0500 > > Send SciPy-User mailing list submissions to > scipy-user at scipy.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.scipy.org/mailman/listinfo/scipy-user > or, via email, send a message with subject or body 'help' to > scipy-user-request at scipy.org > > You can reach the person managing the list at > scipy-user-owner at scipy.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of SciPy-User digest..." > > > Today's Topics: > > 1. Scipy.test() crashes Idle (Chris Stehlik) > 2. Re: Scipy.test() crashes Idle (Scott Sinclair) > 3. Re: Scipy.test() crashes Idle (josef.pktd at gmail.com) > 4. Re: Scipy.test() crashes Idle (Ralf Gommers) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 24 Aug 2012 13:52:39 -0700 > From: Chris Stehlik > Subject: [SciPy-User] Scipy.test() crashes Idle > To: > Message-ID: > Content-Type: text/plain; charset="iso-8859-1" > > > Hi, > > I am new to scipy and python itself. I was told to run scipy.test() to check it. I have a pc running Windows XP with the Enthought Free Python distribution 2.7.3 with Numpy 1.6.1 and Scipy 0.10.1. When I run scipy.test() in Idle, it crashes part way through. Numpy.test() goes through with no problems, but scipy.test() crashes each time. > How does one go about reporting a bug one this ? > > Chris > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: http://mail.scipy.org/pipermail/scipy-user/attachments/20120824/6b4238b8/attachment-0001.html > > ------------------------------ > > Message: 2 > Date: Sat, 25 Aug 2012 08:58:45 +0200 > From: Scott Sinclair > Subject: Re: [SciPy-User] Scipy.test() crashes Idle > To: SciPy Users List > Message-ID: > > Content-Type: text/plain; charset="iso-8859-1" > > On Aug 24, 2012 10:52 PM, "Chris Stehlik" wrote: > > > > Hi, > > > > I am new to scipy and python itself. I was told to run scipy.test() to > check it. I have a pc running Windows XP with the Enthought Free Python > distribution 2.7.3 with Numpy 1.6.1 and Scipy 0.10.1. When I run > scipy.test() in Idle, it crashes part way through. Numpy.test() goes > through with no problems, but scipy.test() crashes each time. > > How does one go about reporting a bug one this ? > > What happens if you open a command prompt and run: > > python -c "import scipy; scipy.test()" > > Cheers, > Scott > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: http://mail.scipy.org/pipermail/scipy-user/attachments/20120825/181e1165/attachment-0001.html > > ------------------------------ > > Message: 3 > Date: Sat, 25 Aug 2012 10:10:03 -0400 > From: josef.pktd at gmail.com > Subject: Re: [SciPy-User] Scipy.test() crashes Idle > To: SciPy Users List > Message-ID: > > Content-Type: text/plain; charset=ISO-8859-1 > > On Sat, Aug 25, 2012 at 2:58 AM, Scott Sinclair > wrote: > > On Aug 24, 2012 10:52 PM, "Chris Stehlik" wrote: > >> > >> Hi, > >> > >> I am new to scipy and python itself. I was told to run scipy.test() to > >> check it. I have a pc running Windows XP with the Enthought Free Python > >> distribution 2.7.3 with Numpy 1.6.1 and Scipy 0.10.1. When I run > >> scipy.test() in Idle, it crashes part way through. Numpy.test() goes through > >> with no problems, but scipy.test() crashes each time. > >> How does one go about reporting a bug one this ? > > > > What happens if you open a command prompt and run: > > > > python -c "import scipy; scipy.test()" > > a question given this and 2 tickets, and since I haven't updated scipy yet > > scipy 0.10.1 > does scipy.test() now run also weave? or the slow tests. > > weave tests (compiled with mingw) have been crashing on windows for > ages in my experience, so I never run them. > > Josef > > > > > > Cheers, > > Scott > > > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > > ------------------------------ > > Message: 4 > Date: Sat, 25 Aug 2012 16:13:29 +0200 > From: Ralf Gommers > Subject: Re: [SciPy-User] Scipy.test() crashes Idle > To: SciPy Users List > Message-ID: > > Content-Type: text/plain; charset="iso-8859-1" > > On Sat, Aug 25, 2012 at 4:10 PM, wrote: > > > On Sat, Aug 25, 2012 at 2:58 AM, Scott Sinclair > > wrote: > > > On Aug 24, 2012 10:52 PM, "Chris Stehlik" wrote: > > >> > > >> Hi, > > >> > > >> I am new to scipy and python itself. I was told to run scipy.test() to > > >> check it. I have a pc running Windows XP with the Enthought Free Python > > >> distribution 2.7.3 with Numpy 1.6.1 and Scipy 0.10.1. When I run > > >> scipy.test() in Idle, it crashes part way through. Numpy.test() goes > > through > > >> with no problems, but scipy.test() crashes each time. > > >> How does one go about reporting a bug one this ? > > > > > > What happens if you open a command prompt and run: > > > > > > python -c "import scipy; scipy.test()" > > > > a question given this and 2 tickets, and since I haven't updated scipy yet > > > > scipy 0.10.1 > > does scipy.test() now run also weave? or the slow tests. > > > > That should run also the weave tests, but not tests marked as slow. > > > > > > weave tests (compiled with mingw) have been crashing on windows for > > ages in my experience, so I never run them. > > > > I haven't seen any crashes or failures. If you do, please open a ticket. > > Ralf > > > > > Josef > > > > > > > > > > Cheers, > > > Scott > > > > > > > > > _______________________________________________ > > > SciPy-User mailing list > > > SciPy-User at scipy.org > > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: http://mail.scipy.org/pipermail/scipy-user/attachments/20120825/0d624ccd/attachment-0001.html > > ------------------------------ > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > > End of SciPy-User Digest, Vol 108, Issue 42 > ******************************************* -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.pincus at yale.edu Sun Aug 26 09:55:49 2012 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Sun, 26 Aug 2012 09:55:49 -0400 Subject: [SciPy-User] scipy.ndimage.measurements.label In-Reply-To: References: Message-ID: <2D2B2DBF-5B10-4B67-86E7-354BAFDF51E4@yale.edu> Is your question what the function does, or how it does it? The docstring is I think pretty clear on the former, but you can ask back if you have any specific questions. As to how that result is generated, the most authoritative answer is to peruse the source code to NI_Label() in https://github.com/scipy/scipy/blob/master/scipy/ndimage/src/ni_measure.c Based on a quick look, this appears to be the standard two-pass binary connected-component labeling algorithm ( http://en.wikipedia.org/wiki/Connected-component_labeling ), generalized to ND, and with the nice feature that the connectivity can be defined by an arbitrary structuring element. Zach On Aug 23, 2012, at 2:29 PM, Sarah Mount wrote: > Hi all, > > Can someone please tell me what algorithm > scipy.ndimage.measurements.label implements? > > Many thanks, > > Sarah > > -- > Sarah Mount, Senior Lecturer, University of Wolverhampton > website: http://www.snim2.org/ > twitter: @snim2 > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From mount.sarah at gmail.com Sun Aug 26 09:58:06 2012 From: mount.sarah at gmail.com (Sarah Mount) Date: Sun, 26 Aug 2012 14:58:06 +0100 Subject: [SciPy-User] scipy.ndimage.measurements.label In-Reply-To: <2D2B2DBF-5B10-4B67-86E7-354BAFDF51E4@yale.edu> References: <2D2B2DBF-5B10-4B67-86E7-354BAFDF51E4@yale.edu> Message-ID: On Sun, Aug 26, 2012 at 2:55 PM, Zachary Pincus wrote: > Is your question what the function does, or how it does it? How it does it... > The docstring is I think pretty clear on the former, but you can ask back if you have any specific questions. > > As to how that result is generated, the most authoritative answer is to peruse the source code to NI_Label() in > https://github.com/scipy/scipy/blob/master/scipy/ndimage/src/ni_measure.c > > Based on a quick look, this appears to be the standard two-pass binary connected-component labeling algorithm ( http://en.wikipedia.org/wiki/Connected-component_labeling ), generalized to ND, and with the nice feature that the connectivity can be defined by an arbitrary structuring element. > Brilliant. I was ideally looking for a paper which describes the algorithm and I think there's a couple of good links from the Wikipedia page you mention above. Thanks for your help, that was massively useful! Sarah > On Aug 23, 2012, at 2:29 PM, Sarah Mount wrote: > >> Hi all, >> >> Can someone please tell me what algorithm >> scipy.ndimage.measurements.label implements? >> >> Many thanks, >> >> Sarah >> >> -- >> Sarah Mount, Senior Lecturer, University of Wolverhampton >> website: http://www.snim2.org/ >> twitter: @snim2 >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -- Sarah Mount, Senior Lecturer, University of Wolverhampton website: http://www.snim2.org/ twitter: @snim2 From arsbbr at gmx.net Tue Aug 28 06:33:11 2012 From: arsbbr at gmx.net (arsbbr) Date: Tue, 28 Aug 2012 03:33:11 -0700 (PDT) Subject: [SciPy-User] Re[SciPy-user] binning to polar coordinates Message-ID: <34348219.post@talk.nabble.com> Hi, I'm trying to rebin some uniform gridded data to polar coordinates. The original data (Z) is noisy but on a fine grid. ### snip # define grid in cartesian coordinates x = arange(-500, 500) y = arange(-500, 500) X, Y = meshgrid(x, y) Z = X**2 + Y**2 Z += uniform(-5000, 5000, size=Z.shape)) # transform to polar coordinates def cart2pol(x, y): theta = arctan2(y, x) rho = sqrt(x**2 + y**2) return (theta, rho) THETA_xy, RHO_xy = cart2pol(X, Y) # define new polar grid theta = arange(-pi, pi, 0.1) rho = arange(0.1 ,500, 2.0) THETA, RHO = meshgrid(theta, rho) # Tried to use griddata, but it gives of course very jumpy results, because it # does not take the average over the pixels in a patch (dTHETA, dRHO) but an interpolation at the # exact new point. interpolate.griddata((THETA_xy.ravel(), RHO_xy.ravel()), Z.ravel(), (THETA, RHO), method='linear') ### snip Is there a method that rebins the data to a new grid, possibly taking the average of the Z-values inside a patch (dTHETA, dRHO)? Thanks for any hints! -- View this message in context: http://old.nabble.com/Rebinning-to-polar-coordinates-tp34348219p34348219.html Sent from the Scipy-User mailing list archive at Nabble.com. From Jerome.Kieffer at esrf.fr Tue Aug 28 09:01:07 2012 From: Jerome.Kieffer at esrf.fr (Jerome Kieffer) Date: Tue, 28 Aug 2012 15:01:07 +0200 Subject: [SciPy-User] Re[SciPy-user] binning to polar coordinates In-Reply-To: <34348219.post@talk.nabble.com> References: <34348219.post@talk.nabble.com> Message-ID: <20120828150107.be7a0dfb.Jerome.Kieffer@esrf.fr> On Tue, 28 Aug 2012 03:33:11 -0700 (PDT) arsbbr wrote: > > Hi, > > I'm trying to rebin some uniform gridded data to polar coordinates. The > original data (Z) is noisy but on a fine grid. > > ### snip > > # define grid in cartesian coordinates > x = arange(-500, 500) > y = arange(-500, 500) > X, Y = meshgrid(x, y) > Z = X**2 + Y**2 > Z += uniform(-5000, 5000, size=Z.shape)) > > # transform to polar coordinates > > def cart2pol(x, y): > theta = arctan2(y, x) > rho = sqrt(x**2 + y**2) > return (theta, rho) > > THETA_xy, RHO_xy = cart2pol(X, Y) > > # define new polar grid > theta = arange(-pi, pi, 0.1) > rho = arange(0.1 ,500, 2.0) > > THETA, RHO = meshgrid(theta, rho) > > # Tried to use griddata, but it gives of course very jumpy results, because > it > # does not take the average over the pixels in a patch (dTHETA, dRHO) but an > interpolation at the > # exact new point. > > interpolate.griddata((THETA_xy.ravel(), RHO_xy.ravel()), > Z.ravel(), (THETA, RHO), method='linear') > > ### snip > > Is there a method that rebins the data to a new grid, possibly taking the > average > of the Z-values inside a patch (dTHETA, dRHO)? Try to use 2D-weighted histogram (divided by unweighted histograms). This unfortunately does dot split pixels over various bins so it works badly when the number of output bins is large. w=histogram2d(THETA_xy.ravel(),RHO_xy.ravel(),(int(2*pi/0.1),500//2),weights=Z.ravel()) u=histogram2d(THETA_xy.ravel(),RHO_xy.ravel(),(int(2*pi/0.1),500//2)) imshow (w[0]/u[0]) If you want to split pixels, I am working on more sophisticated algorithm to do that in https://github.com/kif/pyFAI Tell me if you are interested. Cheers, -- J?r?me Kieffer On-Line Data analysis / Software Group ISDD / ESRF tel +33 476 882 445 From jeffspencerd at gmail.com Wed Aug 29 01:20:50 2012 From: jeffspencerd at gmail.com (Jeffrey Spencer) Date: Wed, 29 Aug 2012 15:20:50 +1000 Subject: [SciPy-User] Bug in freqz returning angular frequency and frequency response in backward order Message-ID: The freqz function appears to return the frequency values and the frequency response backwards. Like it is supposed to return in the order (h,w) but it actually returns in the order (w,h). This can even be seen in the documentation here: http://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.freqz.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Wed Aug 29 06:35:55 2012 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Wed, 29 Aug 2012 05:35:55 -0500 Subject: [SciPy-User] Bug in freqz returning angular frequency and frequency response in backward order In-Reply-To: References: Message-ID: On Wed, Aug 29, 2012 at 12:20 AM, Jeffrey Spencer wrote: > The freqz function appears to return the frequency values and the > frequency response backwards. Like it is supposed to return in the order > (h,w) but it actually returns in the order (w,h). > > This can even be seen in the documentation here: > http://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.freqz.html > > Jeff, freqz and freqs both return (w, h), which is correct. The mistake is in the example in the docstring for freqz. The line: >>> h, w = signal.freqz(b) should be >>> w, h = signal.freqz(b) Warren > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeffspencerd at gmail.com Wed Aug 29 07:48:57 2012 From: jeffspencerd at gmail.com (Jeffrey Spencer) Date: Wed, 29 Aug 2012 21:48:57 +1000 Subject: [SciPy-User] Bug in freqz returning angular frequency and frequency response in backward order In-Reply-To: References: Message-ID: K, that makes sense. Knew it had to be backwards somewhere. On Aug 29, 2012 8:36 PM, "Warren Weckesser" wrote: > > > On Wed, Aug 29, 2012 at 12:20 AM, Jeffrey Spencer wrote: > >> The freqz function appears to return the frequency values and the >> frequency response backwards. Like it is supposed to return in the order >> (h,w) but it actually returns in the order (w,h). >> >> This can even be seen in the documentation here: >> >> http://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.freqz.html >> >> > > Jeff, > > freqz and freqs both return (w, h), which is correct. > > The mistake is in the example in the docstring for freqz. The line: > > >>> h, w = signal.freqz(b) > > should be > > >>> w, h = signal.freqz(b) > > > Warren > > > >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Wed Aug 29 08:32:56 2012 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Wed, 29 Aug 2012 07:32:56 -0500 Subject: [SciPy-User] Bug in freqz returning angular frequency and frequency response in backward order In-Reply-To: References: Message-ID: On Wed, Aug 29, 2012 at 6:48 AM, Jeffrey Spencer wrote: > K, that makes sense. Knew it had to be backwards somewhere. > On Aug 29, 2012 8:36 PM, "Warren Weckesser" < > warren.weckesser at enthought.com> wrote: > >> >> >> On Wed, Aug 29, 2012 at 12:20 AM, Jeffrey Spencer > > wrote: >> >>> The freqz function appears to return the frequency values and the >>> frequency response backwards. Like it is supposed to return in the order >>> (h,w) but it actually returns in the order (w,h). >>> >>> This can even be seen in the documentation here: >>> >>> http://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.freqz.html >>> >>> >> >> Jeff, >> >> freqz and freqs both return (w, h), which is correct. >> >> The mistake is in the example in the docstring for freqz. The line: >> >> >>> h, w = signal.freqz(b) >> >> should be >> >> >>> w, h = signal.freqz(b) >> >> Pull request to fix this and add an example to signal.freqs is here: https://github.com/scipy/scipy/pull/299 Warren > >> Warren >> >> >> >>> >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Andrew.G.York+scipy at gmail.com Wed Aug 29 09:37:30 2012 From: Andrew.G.York+scipy at gmail.com (Andrew York) Date: Wed, 29 Aug 2012 09:37:30 -0400 Subject: [SciPy-User] 2D integration of irregularly gridded data Message-ID: I have 2D data sampled at irregular points. I want to estimate the integral of this data over a finite x-y region. Is there a standard way to do this in python/scipy, or should I roll my own integration? The algorithm I imagine is to use Delaunay triangulation to construct a surface f(x, y), and then sum the volume under this approximate surface. The form my data currently takes: x = [1.1, 1.3, 1.31, 1.33, 2, 2.05] #x-coordinates at which data is known y = [0.15, 0.7, 0.01, 0.01, 0.9, 0] #y-coordinates at which data is known f = [1.1, 1.3, 1.15, 1.2, 1.18, 1.3] # Data at these x-y coordinates (This isn't real data, just meant to concretely illustrate the type of data I have) I have to process roughly a million of these datasets to produce a single image, so a vectorized solution would be nice. A version of my question is described here: http://math.stackexchange.com/questions/187730/efficiently-estimate-a-2d-integral-from-irregularly-sampled-limited-data -------------- next part -------------- An HTML attachment was scrubbed... URL: From e.antero.tammi at gmail.com Wed Aug 29 15:29:18 2012 From: e.antero.tammi at gmail.com (eat) Date: Wed, 29 Aug 2012 22:29:18 +0300 Subject: [SciPy-User] Interpolation problem with interp1d(.) Message-ID: Hi, Apparently I'm somehow misusing the functionality of interp1d(.) or does following behavior imply a bug in scipy. A minimum snippet (with plots) to demonstrate the problem: In []: from scipy.interpolate import interp1d In []: n= 1000 In []: x, y= randn(n), linspace(0, 1, n) In []: x.sort() In []: plot(x, y, lw= 2) Out[]: [] In []: f= interp1d(x, y, 'cubic') In []: xi= linspace(x.min(), x.max(), n) In []: plot(xi, f(xi)) Out[]: [] Regards, -eat -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: p0.png Type: image/png Size: 34991 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: p1.png Type: image/png Size: 37514 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: p2.png Type: image/png Size: 54543 bytes Desc: not available URL: From josef.pktd at gmail.com Wed Aug 29 15:46:25 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 29 Aug 2012 15:46:25 -0400 Subject: [SciPy-User] Interpolation problem with interp1d(.) In-Reply-To: References: Message-ID: On Wed, Aug 29, 2012 at 3:29 PM, eat wrote: > Hi, > > Apparently I'm somehow misusing the functionality of interp1d(.) or does > following behavior imply a bug in scipy. A minimum snippet (with plots) to > demonstrate the problem: > In []: from scipy.interpolate import interp1d > In []: n= 1000 > In []: x, y= randn(n), linspace(0, 1, n) > In []: x.sort() > In []: plot(x, y, lw= 2) > Out[]: [] > > In []: f= interp1d(x, y, 'cubic') > In []: xi= linspace(x.min(), x.max(), n) > In []: plot(xi, f(xi)) > Out[]: [] I guess, some x's are too close to each other to fit a cubic interpolation without a lot of overshooting. Reversing x and y looks fine. Using the splines directly and add a small s>0 might also work. Josef > > Regards, > -eat > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From e.antero.tammi at gmail.com Wed Aug 29 15:49:14 2012 From: e.antero.tammi at gmail.com (eat) Date: Wed, 29 Aug 2012 22:49:14 +0300 Subject: [SciPy-User] Interpolation problem with interp1d(.) In-Reply-To: References: Message-ID: Oops, I forgot to mention that I'm on: In []: sys.version Out[]: '2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)]' In []: np.version.version Out[]: '1.6.0' In []: sp.version.version Out[]: '0.9.0' On Wed, Aug 29, 2012 at 10:29 PM, eat wrote: > Hi, > > Apparently I'm somehow misusing the functionality of interp1d(.) or does > following behavior imply a bug in scipy. A minimum snippet (with plots) to > demonstrate the problem: > In []: from scipy.interpolate import interp1d > In []: n= 1000 > In []: x, y= randn(n), linspace(0, 1, n) > In []: x.sort() > In []: plot(x, y, lw= 2) > Out[]: [] > > In []: f= interp1d(x, y, 'cubic') > In []: xi= linspace(x.min(), x.max(), n) > In []: plot(xi, f(xi)) > Out[]: [] > > Regards, > -eat > -------------- next part -------------- An HTML attachment was scrubbed... URL: From e.antero.tammi at gmail.com Wed Aug 29 15:56:29 2012 From: e.antero.tammi at gmail.com (eat) Date: Wed, 29 Aug 2012 22:56:29 +0300 Subject: [SciPy-User] Interpolation problem with interp1d(.) In-Reply-To: References: Message-ID: Hi, On Wed, Aug 29, 2012 at 10:46 PM, wrote: > On Wed, Aug 29, 2012 at 3:29 PM, eat wrote: > > Hi, > > > > Apparently I'm somehow misusing the functionality of interp1d(.) or does > > following behavior imply a bug in scipy. A minimum snippet (with plots) > to > > demonstrate the problem: > > In []: from scipy.interpolate import interp1d > > In []: n= 1000 > > In []: x, y= randn(n), linspace(0, 1, n) > > In []: x.sort() > > In []: plot(x, y, lw= 2) > > Out[]: [] > > > > In []: f= interp1d(x, y, 'cubic') > > In []: xi= linspace(x.min(), x.max(), n) > > In []: plot(xi, f(xi)) > > Out[]: [] > > > I guess, some x's are too close to each other to fit a cubic > interpolation without a lot of overshooting. > Makes sense, although what are too close seems to be quite conservative: In []: d= x[1:]- x[:-1] In []: d.sort() In []: d[:3] Out[]: array([ 2.09893021e-06, 2.36059137e-06, 7.52680662e-06]) Regards, -eat > > Reversing x and y looks fine. > > Using the splines directly and add a small s>0 might also work. > > Josef > > > > > Regards, > > -eat > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Wed Aug 29 17:20:48 2012 From: takowl at gmail.com (Thomas Kluyver) Date: Wed, 29 Aug 2012 22:20:48 +0100 Subject: [SciPy-User] A unified face for the scipy ecosystem Message-ID: Hi all, Almar Klein and I floated this idea at Euroscipy this year, to some positive feedback. I'm posting it here to get some more opinions. We think it's a problem for newcomers that there isn't a clear starting point with the scipy ecosystem. We all understand that tools like numpy and matplotlib are critical for our field, but a newcomer searching for 'Python' will have to install these things separately. The conferences are called 'scipy', and scipy.org has some ambition to be a general portal for scientific computing in Python, but the name is confounded by the scipy package, and the downloads page is for packages, not complete distributions. Pylab is even more confusing. EPD and Python(x,y) are more helpful, but code is often not specific to those distributions. This is in contrast to systems like R and Matlab, where searching for the most obvious name turns up a complete system that's ready for scientific work. So what we propose comes in three parts: - A standard: like Linux Standard Base, it would specify minimum versions of packages for any distribution that wanted to advertise its compliance. I envisage it would include Python itself, numpy, scipy and matplotlib, but the exact list is open to debate. - A name: something that we would all use to refer to the core packages, regardless of how we installed them. If we don't use the name, newcomers won't search for it. I'm terrible with names, but Almar has proposed: scipy-base, unipy, pyengine, pycraft. Personally, I'd lean towards an abstract name without 'py'. Your suggestions are welcome! - A website: The first thing newcomers will see when searching for the name. It will describe the set of packages, and how to get them installed (probably giving precedence to inclusive Python distributions, but with instructions for separate installation as well). It will also link in to relevant documentation and example code, and it could have curated lists of packages for specific topics. It's important that this doesn't obscure the identity of the individual packages; the aim is not to stop anyone getting the credit they deserve. But I know we do often work closely together - getting involved with IPython, I've seen that clearly - and I think we sell ourselves short by not presenting a single face. Another important question is whether the standard should include a coding interface - the Matlab built-in IDE is touted as an advantage of that platform, for instance. As an IPython developer, I'm keen to see IPython included, but then I know there are other intriguing solutions, such as Spyder and IEP. We'd love to hear what you think, whether you're an experienced developer, a beginner with scientific Python, someone trying to introduce others to Python, or indeed anyone else with an interest in this. Best wishes, Thomas From travis at continuum.io Wed Aug 29 18:40:53 2012 From: travis at continuum.io (Travis Oliphant) Date: Wed, 29 Aug 2012 17:40:53 -0500 Subject: [SciPy-User] A unified face for the scipy ecosystem In-Reply-To: References: Message-ID: <54AD300D-0DE1-47E9-9CDA-B7CDCF77DFF0@continuum.io> This is closely related to the idea of "sciome" that we have been discussing on the numfocus mailing list for a few months: https://groups.google.com/group/numfocus/browse_frm/month/2012-05?pli=1 I agree that such an entry point is needed to assist new-comers. The idea of sciome was to have a meta-package that was basically a "tagging" of the various individual projects and a single web-site and a place to have documentation links to the various sub-projects, etc. Fernando Perez also suggested common help searching tools, etc. that would make sciome a single concept. The name is not fixed in stone, of course, but it does have a nice ring to it IMHO. Best, -Travis On Aug 29, 2012, at 4:20 PM, Thomas Kluyver wrote: > Hi all, > > Almar Klein and I floated this idea at Euroscipy this year, to some > positive feedback. I'm posting it here to get some more opinions. > > We think it's a problem for newcomers that there isn't a clear > starting point with the scipy ecosystem. We all understand that tools > like numpy and matplotlib are critical for our field, but a newcomer > searching for 'Python' will have to install these things separately. > The conferences are called 'scipy', and scipy.org has some ambition to > be a general portal for scientific computing in Python, but the name > is confounded by the scipy package, and the downloads page is for > packages, not complete distributions. Pylab is even more confusing. > EPD and Python(x,y) are more helpful, but code is often not specific > to those distributions. > > This is in contrast to systems like R and Matlab, where searching for > the most obvious name turns up a complete system that's ready for > scientific work. > > So what we propose comes in three parts: > > - A standard: like Linux Standard Base, it would specify minimum > versions of packages for any distribution that wanted to advertise its > compliance. I envisage it would include Python itself, numpy, scipy > and matplotlib, but the exact list is open to debate. > - A name: something that we would all use to refer to the core > packages, regardless of how we installed them. If we don't use the > name, newcomers won't search for it. I'm terrible with names, but > Almar has proposed: scipy-base, unipy, pyengine, pycraft. Personally, > I'd lean towards an abstract name without 'py'. Your suggestions are > welcome! > - A website: The first thing newcomers will see when searching for the > name. It will describe the set of packages, and how to get them > installed (probably giving precedence to inclusive Python > distributions, but with instructions for separate installation as > well). It will also link in to relevant documentation and example > code, and it could have curated lists of packages for specific topics. > > It's important that this doesn't obscure the identity of the > individual packages; the aim is not to stop anyone getting the credit > they deserve. But I know we do often work closely together - getting > involved with IPython, I've seen that clearly - and I think we sell > ourselves short by not presenting a single face. > > Another important question is whether the standard should include a > coding interface - the Matlab built-in IDE is touted as an advantage > of that platform, for instance. As an IPython developer, I'm keen to > see IPython included, but then I know there are other intriguing > solutions, such as Spyder and IEP. > > We'd love to hear what you think, whether you're an experienced > developer, a beginner with scientific Python, someone trying to > introduce others to Python, or indeed anyone else with an interest in > this. > > Best wishes, > Thomas > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Wed Aug 29 22:59:47 2012 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 29 Aug 2012 19:59:47 -0700 Subject: [SciPy-User] A sad day for our community. John Hunter: 1968-2012. Message-ID: Dear friends and colleagues, [please excuse a possible double-post of this message, in-flight internet glitches] I am terribly saddened to report that yesterday, August 28 2012 at 10am, John D. Hunter died from complications arising from cancer treatment at the University of Chicago hospital, after a brief but intense battle with this terrible illness. John is survived by his wife Miriam, his three daughters Rahel, Ava and Clara, his sisters Layne and Mary, and his mother Sarah. Note: If you decide not to read any further (I know this is a long message), please go to this page for some important information about how you can thank John for everything he gave in a decade of generous contributions to the Python and scientific communities: http://numfocus.org/johnhunter. Just a few weeks ago, John delivered his keynote address at the SciPy 2012 conference in Austin centered around the evolution of matplotlib: http://www.youtube.com/watch?v=e3lTby5RI54 but tragically, shortly after his return home he was diagnosed with advanced colon cancer. This diagnosis was a terrible discovery to us all, but John took it with his usual combination of calm and resolve, and initiated treatment procedures. Unfortunately, the first round of chemotherapy treatments led to severe complications that sent him to the intensive care unit, and despite the best efforts of the University of Chicago medical center staff, he never fully recovered from these. Yesterday morning, he died peacefully at the hospital with his loved ones at his bedside. John fought with grace and courage, enduring every necessary procedure with a smile on his face and a kind word for all of his caretakers and becoming a loved patient of the many teams that ended up involved with his case. This was no surprise for those of us who knew him, but he clearly left a deep and lasting mark even amongst staff hardened by the rigors of oncology floors and intensive care units. I don't need to explain to this community the impact of John's work, but allow me to briefly recap, in case this is read by some who don't know the whole story. In 2002, John was a postdoc at the University of Chicago hospital working on the analysis of epilepsy seizure data in children. Frustrated with the state of the existing proprietary solutions for this class of problems, he started using Python for his work, back when the scientific Python ecosystem was much, much smaller than it is today and this could have been seen as a crazy risk. Furthermore, he found that there were many half-baked solutions for data visualization in Python at the time, but none that truly met his needs. Undeterred, he went on to create matplotlib (http://matplotlib.org) and thus overcome one of the key obstacles for Python to become the best solution for open source scientific and technical computing. Matplotlib is both an amazing technical achievement and a shining example of open source community building, as John not only created its backbone but also fostered the development of a very strong development team, ensuring that the talent of many others could also contribute to this project. The value and importance of this are now painfully clear: despite having lost John, matplotlib continues to thrive thanks to the leadership of Michael Droetboom, the support of Perry Greenfield at the Hubble Telescope Science Institute, and the daily work of the rest of the team. I want to thank Perry and Michael for putting their resources and talent once more behind matplotlib, securing the future of the project. It is difficult to overstate the value and importance of matplotlib, and therefore of John's contributions (which do not end in matplotlib, by the way; but a biography will have to wait for another day...). Python has become a major force in the technical and scientific computing world, leading the open source offers and challenging expensive proprietary platforms with large teams and millions of dollars of resources behind them. But this would be impossible without a solid data visualization tool that would allow both ad-hoc data exploration and the production of complex, fine-tuned figures for papers, reports or websites. John had the vision to make matplotlib easy to use, but powerful and flexible enough to work in graphical user interfaces and as a server-side library, enabling a myriad use cases beyond his personal needs. This means that now, matplotlib powers everything from plots in dissertations and journal articles to custom data analysis projects and websites. And despite having left his academic career a few years ago for a job in industry, he remained engaged enough that as of today, he is still the top committer to matplotlib; this is the git shortlog of those with more than 1000 commits to the project: 2145 John Hunter 2130 Michael Droettboom 1060 Eric Firing All of this was done by a man who had three children to raise and who still always found the time to help those on the mailing lists, solve difficult technical problems in matplotlib, teach courses and seminars about scientific Python, and more recently help create the NumFOCUS foundation project. Despite the challenges that raising three children in an expensive city like Chicago presented, he never once wavered from his commitment to open source. But unfortunately now he is not here anymore to continue providing for their well-being, and I hope that all those who have so far benefited from his generosity, will thank this wonderful man who always gave far more than he received. Thanks to the rapid action of Travis Oliphant, the NumFOCUS foundation is now acting as an escrow agent to accept donations that will go into a fund to support the education and care of his wonderful girls Rahel, Ava and Clara. If you have benefited from John's many contributions, please say thanks in the way that would matter most to him, by helping Miriam continue the task of caring for and educating Rahel, Ava and Clara. You will find all the information necessary to make a donation here: http://numfocus.org/johnhunter Remember that even a small donation helps! If all those who ever use matplotlib give just a little bit, in the long run I am sure that we can make a difference. If you are a company that benefits in a serious way from matplotlib, remember that John was a staunch advocate of keeping all scientific Python projects under the BSD license so that commercial users could benefit from them without worry. Please say thanks to John in a way commensurate with your resources (and check how much a yearly matlab license would cost you in case you have any doubts about the value you are getting...). John's family is planning a private burial in Tennessee, but (most likely in September) there will also be a memorial service in Chicago that friends and members of the community can attend. We don't have the final scheduling details at this point, but I will post them once we know. I would like to again express my gratitude to Travis Oliphant for moving quickly with the setup of the donation support, and to Eric Jones (the founder of Enthought and another one of the central figures in our community) who immediately upon learning of John's plight contributed resources to support the family with everyday logistics while John was facing treatment as well as my travel to Chicago to assist. This kind of immediate urge to come to the help of others that Eric and Travis displayed is a hallmark of our community. Before closing, I want to take a moment to publicly thank the incredible staff of the University of Chicago medical center. The last two weeks were an intense and brutal ordeal for John and his loved ones, but the hospital staff offered a sometimes hard to believe, unending supply of generosity, care and humanity in addition to their technical competence. The latter is something we expect from a first-rate hospital at a top university, where the attending physicians can be world-renowned specialists in their field. But the former is often forgotten in a world often ruled by a combination of science and concerns about regulations and liability. Instead, we found generous and tireless staff who did everything in their power to ease the pain, always putting our well being ahead of any mindless adherence to protocol, patiently tending to every need we had and working far beyond their stated responsibilities to support us. To name only one person (and many others are equally deserving), I want to thank Dr. Carla Moreira, chief surgical resident, who spent the last few hours of John's life with us despite having just completed a solid night shift of surgical work. Instead of resting she came to the ICU and worked to ensure that those last hours were as comfortable as possible for John; her generous actions helped us through a very difficult moment. It is now time to close this already too long message... John, thanks for everything you gave all of us, and for the privilege of knowing you. Fernando. ps - I have sent this with my 'mailing lists' email. If you need to contact me directly for anything regarding the above, please write to my regular address at Fernando.Perez at berkeley.edu, where I do my best to reply more promptly. From gary.ruben at gmail.com Thu Aug 30 03:22:21 2012 From: gary.ruben at gmail.com (gary ruben) Date: Thu, 30 Aug 2012 17:22:21 +1000 Subject: [SciPy-User] Tensioned Spline interpolation Message-ID: Hi, I'm rewriting some IDL code in Python. The original code uses IDL's spline function to interpolate a 1D ungridded data series onto a regular grid. Unfortunately, I can't find a close enough equivalent in scipy. I'm guessing the IDL spline function implements a "tensioned spline." The signature of the IDL spline function is Result = SPLINE(X, Y, T [, Sigma]) where Sigma is 'The amount of "tension" that is applied to the curve. If sigma is close to 0, (e.g., .01), then effectively there is a cubic spline fit. If sigma is large, (e.g., greater than 10), then the fit will be like a polynomial interpolation.' In the code I'm trying to reproduce, the author has set sigma=15 so the result is quite different to a simple cubic spline. Scipy's pchip algorithm gives a similar result, but is unfortunately too slow for my application. Can anyone confirm that IDL's spline is a tensioned spline routine? Does anyone know of a BSD-licensed Python module that implements a tensioned spline. Failing that any suggestions for a good routine I should look at wrapping (maybe Alan Cline's fitpack on netlib?) thanks, Gary From fperez.net at gmail.com Wed Aug 29 22:32:23 2012 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 29 Aug 2012 19:32:23 -0700 Subject: [SciPy-User] A sad day for our community. John Hunter: 1968-2012. Message-ID: Dear friends and colleagues, I am terribly saddened to report that yesterday, August 28 2012 at 10am, John D. Hunter died from complications arising from cancer treatment at the University of Chicago hospital, after a brief but intense battle with this terrible illness. John is survived by his wife Miriam, his three daughters Rahel, Ava and Clara, his sisters Layne and Mary, and his mother Sarah. Note: If you decide not to read any further (I know this is a long message), please go to this page for some important information about how you can thank John for everything he gave in a decade of generous contributions to the Python and scientific communities: http://numfocus.org/johnhunter. Just a few weeks ago, John delivered his keynote address at the SciPy 2012 conference in Austin centered around the evolution of matplotlib: http://www.youtube.com/watch?v=e3lTby5RI54 but tragically, shortly after his return home he was diagnosed with advanced colon cancer. This diagnosis was a terrible discovery to us all, but John took it with his usual combination of calm and resolve, and initiated treatment procedures. Unfortunately, the first round of chemotherapy treatments led to severe complications that sent him to the intensive care unit, and despite the best efforts of the University of Chicago medical center staff, he never fully recovered from these. Yesterday morning, he died peacefully at the hospital with his loved ones at his bedside. John fought with grace and courage, enduring every necessary procedure with a smile on his face and a kind word for all of his caretakers and becoming a loved patient of the many teams that ended up involved with his case. This was no surprise for those of us who knew him, but he clearly left a deep and lasting mark even amongst staff hardened by the rigors of oncology floors and intensive care units. I don't need to explain to this community the impact of John's work, but allow me to briefly recap, in case this is read by some who don't know the whole story. In 2002, John was a postdoc at the University of Chicago hospital working on the analysis of epilepsy seizure data in children. Frustrated with the state of the existing proprietary solutions for this class of problems, he started using Python for his work, back when the scientific Python ecosystem was much, much smaller than it is today and this could have been seen as a crazy risk. Furthermore, he found that there were many half-baked solutions for data visualization in Python at the time, but none that truly met his needs. Undeterred, he went on to create matplotlib (http://matplotlib.org) and thus overcome one of the key obstacles for Python to become the best solution for open source scientific and technical computing. Matplotlib is both an amazing technical achievement and a shining example of open source community building, as John not only created its backbone but also fostered the development of a very strong development team, ensuring that the talent of many others could also contribute to this project. The value and importance of this are now painfully clear: despite having lost John, matplotlib continues to thrive thanks to the leadership of Michael Droetboom, the support of Perry Greenfield at the Hubble Telescope Science Institute, and the daily work of the rest of the team. I want to thank Perry and Michael for putting their resources and talent once more behind matplotlib, securing the future of the project. It is difficult to overstate the value and importance of matplotlib, and therefore of John's contributions (which do not end in matplotlib, by the way; but a biography will have to wait for another day...). Python has become a major force in the technical and scientific computing world, leading the open source offers and challenging expensive proprietary platforms with large teams and millions of dollars of resources behind them. But this would be impossible without a solid data visualization tool that would allow both ad-hoc data exploration and the production of complex, fine-tuned figures for papers, reports or websites. John had the vision to make matplotlib easy to use, but powerful and flexible enough to work in graphical user interfaces and as a server-side library, enabling a myriad use cases beyond his personal needs. This means that now, matplotlib powers everything from plots in dissertations and journal articles to custom data analysis projects and websites. And despite having left his academic career a few years ago for a job in industry, he remained engaged enough that as of today, he is still the top committer to matplotlib; this is the git shortlog of those with more than 1000 commits to the project: 2145 John Hunter 2130 Michael Droettboom 1060 Eric Firing All of this was done by a man who had three children to raise and who still always found the time to help those on the mailing lists, solve difficult technical problems in matplotlib, teach courses and seminars about scientific Python, and more recently help create the NumFOCUS foundation project. Despite the challenges that raising three children in an expensive city like Chicago presented, he never once wavered from his commitment to open source. But unfortunately now he is not here anymore to continue providing for their well-being, and I hope that all those who have so far benefited from his generosity, will thank this wonderful man who always gave far more than he received. Thanks to the rapid action of Travis Oliphant, the NumFOCUS foundation is now acting as an escrow agent to accept donations that will go into a fund to support the education and care of his wonderful girls Rahel, Ava and Clara. If you have benefited from John's many contributions, please say thanks in the way that would matter most to him, by helping Miriam continue the task of caring for and educating Rahel, Ava and Clara. You will find all the information necessary to make a donation here: http://numfocus.org/johnhunter Remember that even a small donation helps! If all those who ever use matplotlib give just a little bit, in the long run I am sure that we can make a difference. If you are a company that benefits in a serious way from matplotlib, remember that John was a staunch advocate of keeping all scientific Python projects under the BSD license so that commercial users could benefit from them without worry. Please say thanks to John in a way commensurate with your resources (and check how much a yearly matlab license would cost you in case you have any doubts about the value you are getting...). John's family is planning a private burial in Tennessee, but (most likely in September) there will also be a memorial service in Chicago that friends and members of the community can attend. We don't have the final scheduling details at this point, but I will post them once we know. I would like to again express my gratitude to Travis Oliphant for moving quickly with the setup of the donation support, and to Eric Jones (the founder of Enthought and another one of the central figures in our community) who immediately upon learning of John's plight contributed resources to support the family with everyday logistics while John was facing treatment as well as my travel to Chicago to assist. This kind of immediate urge to come to the help of others that Eric and Travis displayed is a hallmark of our community. Before closing, I want to take a moment to publicly thank the incredible staff of the University of Chicago medical center. The last two weeks were an intense and brutal ordeal for John and his loved ones, but the hospital staff offered a sometimes hard to believe, unending supply of generosity, care and humanity in addition to their technical competence. The latter is something we expect from a first-rate hospital at a top university, where the attending physicians can be world-renowned specialists in their field. But the former is often forgotten in a world often ruled by a combination of science and concerns about regulations and liability. Instead, we found generous and tireless staff who did everything in their power to ease the pain, always putting our well being ahead of any mindless adherence to protocol, patiently tending to every need we had and working far beyond their stated responsibilities to support us. To name only one person (and many others are equally deserving), I want to thank Dr. Carla Moreira, chief surgical resident, who spent the last few hours of John's life with us despite having just completed a solid night shift of surgical work. Instead of resting she came to the ICU and worked to ensure that those last hours were as comfortable as possible for John; her generous actions helped us through a very difficult moment. It is now time to close this already too long message... John, thanks for everything you gave all of us, and for the privilege of knowing you. Fernando. ps - I have sent this with my 'mailing lists' email. If you need to contact me directly for anything regarding the above, please write to my regular address at Fernando.Perez at berkeley.edu, where I do my best to reply more promptly. From nils106 at googlemail.com Wed Aug 29 14:16:24 2012 From: nils106 at googlemail.com (Nils Wagner) Date: Wed, 29 Aug 2012 19:16:24 +0100 Subject: [SciPy-User] Matlab figure Message-ID: Hi all, I have received a Matlab figure stored in a matlab file. How can I extract the curves from the file ? Cheers, Nils -------------- next part -------------- An HTML attachment was scrubbed... URL: From talcite at gmail.com Wed Aug 29 17:20:04 2012 From: talcite at gmail.com (Matt Chan) Date: Wed, 29 Aug 2012 14:20:04 -0700 (PDT) Subject: [SciPy-User] possible bug in scipy.optimize.newton_krylov Message-ID: Hi, I'm having trouble setting the tolerance for the inner method of the newton-krylov method. Although the documentation says it is possible, setting inner_tol returns an error complaining about duplicate arguments. http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.newton_krylov.html I've pasted the stack trace below. Is there something I am doing incorrectly? Traceback (most recent call last): File "", line 758, in x_star= op.newton_krylov(my_fn, x0, method='lgmres', inner_tol=1e-8) File "", line 8, in newton_krylov File "/usr/lib64/python2.7/site-packages/scipy/optimize/nonlin.py", line 294, in nonlin_solve dx = -jacobian.solve(Fx, tol=tol) File "/usr/lib64/python2.7/site-packages/scipy/optimize/nonlin.py", line 1394, in solve sol, info = self.method(self.op, rhs, tol=tol, **self.method_kw) TypeError: lgmres() got multiple values for keyword argument 'tol' Thanks, Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From SergeRogov at minifab.com.au Thu Aug 30 00:41:47 2012 From: SergeRogov at minifab.com.au (Serge Rogov) Date: Thu, 30 Aug 2012 14:41:47 +1000 Subject: [SciPy-User] scipy.stats.gaussian_kde ppf? Message-ID: <1A8615610E06C54BA9257EB465F21E33D2D2F6@mfbsrv002.minifab.local> Hi all, I need to compute confidence intervals from gaussian_kde, but I found that the ppf function is missing. I have implemented my own na?ve version that precomputes the cdf for the kde and then performs a search for the desired probability, but it's extremely slow and shows a bit of error when compared to norm.ppf even at large sample sizes. Has anyone had to do something like this before? Kind regards, Serge -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Thu Aug 30 12:05:12 2012 From: sturla at molden.no (Sturla Molden) Date: Thu, 30 Aug 2012 18:05:12 +0200 Subject: [SciPy-User] Matlab figure In-Reply-To: References: Message-ID: <503F8F38.1040406@molden.no> If you have Matlab and Adobe CS, I'd say save the figure as EPS and let Illustrator extract the data. Sturla On 29.08.2012 20:16, Nils Wagner wrote: > Hi all, > > I have received a Matlab figure stored in a matlab file. > How can I extract the curves from the file ? > > Cheers, > Nils > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From hturesson at gmail.com Thu Aug 30 12:13:05 2012 From: hturesson at gmail.com (Hjalmar Turesson) Date: Thu, 30 Aug 2012 12:13:05 -0400 Subject: [SciPy-User] Matlab figure In-Reply-To: <503F8F38.1040406@molden.no> References: <503F8F38.1040406@molden.no> Message-ID: Inkscape will work as well (instead of illustrator). Hjalmar On Thu, Aug 30, 2012 at 12:05 PM, Sturla Molden wrote: > If you have Matlab and Adobe CS, I'd say save the figure as EPS and let > Illustrator extract the data. > > Sturla > > On 29.08.2012 20:16, Nils Wagner wrote: > > Hi all, > > > > I have received a Matlab figure stored in a matlab file. > > How can I extract the curves from the file ? > > > > Cheers, > > Nils > > > > > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Aug 30 12:15:06 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 30 Aug 2012 12:15:06 -0400 Subject: [SciPy-User] scipy.stats.gaussian_kde ppf? In-Reply-To: <1A8615610E06C54BA9257EB465F21E33D2D2F6@mfbsrv002.minifab.local> References: <1A8615610E06C54BA9257EB465F21E33D2D2F6@mfbsrv002.minifab.local> Message-ID: On Thu, Aug 30, 2012 at 12:41 AM, Serge Rogov wrote: > Hi all, > > > > I need to compute confidence intervals from gaussian_kde, but I found that > the ppf function is missing. I have implemented my own na?ve version that > precomputes the cdf for the kde and then performs a search for the desired > probability, but it?s extremely slow and shows a bit of error when compared > to norm.ppf even at large sample sizes. Has anyone had to do something like > this before? I've never seen a ppf for a kde. gaussian_kde is multivariate, and I don't know if there is a definition of ppf for multivariate distributions. My guess is that there is nothing better than 2 rootfinding calls to find the confidence interval for 1d (maybe with norm.ppf as starting values if the distribution is approximately normal) (aside: for distributions with multiple modes the minimum length confidence region in 1d might be non-overlapping intervals.) What would be the best to find a confidence region in the multivariate case ? recipes, example code and/or pull request would be useful for future users. Josef > > > > Kind regards, > > Serge > > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From aronne.merrelli at gmail.com Thu Aug 30 12:33:42 2012 From: aronne.merrelli at gmail.com (Aronne Merrelli) Date: Thu, 30 Aug 2012 11:33:42 -0500 Subject: [SciPy-User] Matlab figure In-Reply-To: References: Message-ID: On Wed, Aug 29, 2012 at 1:16 PM, Nils Wagner wrote: > Hi all, > > I have received a Matlab figure stored in a matlab file. > How can I extract the curves from the file ? > I guess it depends on what exactly you mean by "figure stored in a matlab file". I assume you mean save as fig-format; after I fiddling around for a bit, I find that this seems to work: >From MATLAB: >> x = sqrt(1:9); >> plot(x) >> saveas(gcf(), 'test.fig', 'fig') >> version ans = 7.9.0.529 (R2009b) SciPy: In [11]: from scipy.io.matlab import loadmat In [12]: m = loadmat('test.fig') In [13]: m.keys() Out[13]: ['hgS_070000', '__version__', '__header__', '__globals__'] In [15]: m['hgS_070000']['children'].item()['children'].item()['properties'].item()['XData'].item() Out[15]: array([[1, 2, 3, 4, 5, 6, 7, 8, 9]], dtype=uint8) In [16]: m['hgS_070000']['children'].item()['children'].item()['properties'].item()['YData'].item() Out[16]: array([[ 1. , 1.41421356, 1.73205081, 2. , 2.23606798, 2.44948974, 2.64575131, 2.82842712, 3. ]]) In [17]: np.version.version Out[17]: '1.6.1' In [18]: scipy.version.version Out[18]: '0.9.0' I have no idea where the key name 'hgS_070000' comes from, but it should be obvious to find if the figure is the only thing saved in the file, since it would be the only variable. The rest of the syntax crawling through the data structure matches how the figure is built in MATLAB. Meaning, if you load it back, you would have done the same thing in MATLAB like so: (Note I renamed it to test.mat to get MATLAB to reload it here) >> m = load('test.mat') >> m.hgS_070000.children.children.properties.XData ans = 1 2 3 4 5 6 7 8 9 >> m.hgS_070000.children.children.properties.YData ans = Columns 1 through 8 1.0000 1.4142 1.7321 2.0000 2.2361 2.4495 2.6458 2.8284 Column 9 3.0000 If you reloaded it via the GUI load figure, and it is in figure #2, then it would have been: >> get(get(get(2,'Children'),'Children'),'XData') etc... Cheers, Aronne From robert.kern at gmail.com Thu Aug 30 12:35:31 2012 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 30 Aug 2012 17:35:31 +0100 Subject: [SciPy-User] scipy.stats.gaussian_kde ppf? In-Reply-To: References: <1A8615610E06C54BA9257EB465F21E33D2D2F6@mfbsrv002.minifab.local> Message-ID: On Thu, Aug 30, 2012 at 5:15 PM, wrote: > On Thu, Aug 30, 2012 at 12:41 AM, Serge Rogov wrote: >> Hi all, >> >> I need to compute confidence intervals from gaussian_kde, but I found that >> the ppf function is missing. I have implemented my own na?ve version that >> precomputes the cdf for the kde and then performs a search for the desired >> probability, but it?s extremely slow and shows a bit of error when compared >> to norm.ppf even at large sample sizes. Has anyone had to do something like >> this before? > > I've never seen a ppf for a kde. > > gaussian_kde is multivariate, and I don't know if there is a > definition of ppf for multivariate distributions. > > My guess is that there is nothing better than 2 rootfinding calls to > find the confidence interval for 1d (maybe with norm.ppf as starting > values if the distribution is approximately normal) You can invert the empirical CDF for a nicer starting value, regardless of distribution. -- Robert Kern From josef.pktd at gmail.com Thu Aug 30 13:17:54 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 30 Aug 2012 13:17:54 -0400 Subject: [SciPy-User] scipy.stats.gaussian_kde ppf? In-Reply-To: References: <1A8615610E06C54BA9257EB465F21E33D2D2F6@mfbsrv002.minifab.local> Message-ID: On Thu, Aug 30, 2012 at 12:35 PM, Robert Kern wrote: > On Thu, Aug 30, 2012 at 5:15 PM, wrote: >> On Thu, Aug 30, 2012 at 12:41 AM, Serge Rogov wrote: >>> Hi all, >>> >>> I need to compute confidence intervals from gaussian_kde, but I found that >>> the ppf function is missing. I have implemented my own na?ve version that >>> precomputes the cdf for the kde and then performs a search for the desired >>> probability, but it?s extremely slow and shows a bit of error when compared >>> to norm.ppf even at large sample sizes. Has anyone had to do something like >>> this before? >> >> I've never seen a ppf for a kde. >> >> gaussian_kde is multivariate, and I don't know if there is a >> definition of ppf for multivariate distributions. >> >> My guess is that there is nothing better than 2 rootfinding calls to >> find the confidence interval for 1d (maybe with norm.ppf as starting >> values if the distribution is approximately normal) > > You can invert the empirical CDF for a nicer starting value, > regardless of distribution. good idea, or even with a linear interpolation of the inverse empirical cdf, which might already be in statsmodels. Josef > > -- > Robert Kern > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From jjstickel at gmail.com Thu Aug 30 13:30:44 2012 From: jjstickel at gmail.com (Jonathan Stickel) Date: Thu, 30 Aug 2012 11:30:44 -0600 Subject: [SciPy-User] Tensioned Spline interpolation In-Reply-To: References: Message-ID: <503FA344.4070006@gmail.com> Gary I am not familiar with the "tension" terminology, but it sounds like a smoothing parameter. scipy.interpolate.UnivariateSpline provides splines with smoothing. You might also try the scikit "datasmooth" http://pypi.python.org/pypi/scikits.datasmooth/0.61 While not exactly a spline implementation, it computes a trend-line through 1D data with user-specified or automatically determined smoothing. HTH, Jonathan On 8/30/12 10:08 , scipy-user-request at scipy.org wrote: > Date: Thu, 30 Aug 2012 17:22:21 +1000 > From: gary ruben > Subject: [SciPy-User] Tensioned Spline interpolation > > Hi, > > I'm rewriting some IDL code in Python. The original code uses IDL's > spline function to interpolate a 1D ungridded data series onto a > regular grid. Unfortunately, I can't find a close enough equivalent in > scipy. I'm guessing the IDL spline function implements a "tensioned > spline." > > The signature of the IDL spline function is > Result = SPLINE(X, Y, T [, Sigma]) > where Sigma is 'The amount of "tension" that is applied to the curve. > If sigma is close to 0, (e.g., .01), then effectively there is a cubic > spline fit. If sigma is large, (e.g., greater than 10), then the fit > will be like a polynomial interpolation.' > > In the code I'm trying to reproduce, the author has set sigma=15 so > the result is quite different to a simple cubic spline. > Scipy's pchip algorithm gives a similar result, but is unfortunately > too slow for my application. Can anyone confirm that IDL's spline is a > tensioned spline routine? Does anyone know of a BSD-licensed Python > module that implements a tensioned spline. Failing that any > suggestions for a good routine I should look at wrapping (maybe Alan > Cline's fitpack on netlib?) > > thanks, > Gary From takowl at gmail.com Thu Aug 30 15:30:13 2012 From: takowl at gmail.com (Thomas Kluyver) Date: Thu, 30 Aug 2012 20:30:13 +0100 Subject: [SciPy-User] A unified face for the scipy ecosystem In-Reply-To: <54AD300D-0DE1-47E9-9CDA-B7CDCF77DFF0@continuum.io> References: <54AD300D-0DE1-47E9-9CDA-B7CDCF77DFF0@continuum.io> Message-ID: On 29 August 2012 23:40, Travis Oliphant wrote: > This is closely related to the idea of "sciome" that we have been discussing > on the numfocus mailing list for a few months: > https://groups.google.com/group/numfocus/browse_frm/month/2012-05?pli=1 Thanks Travis. I see we're thinking along very similar lines. Sciome has a nice ring to it, although as mentioned on the numfocus list there's the potential for confusion (and trademark issues?) with sciome.com. I'm glad I'm not the only one to have thought about this, and I've love to help push it forward. Almar: > The way I see it, this standard should be something that distributions can say they are > compliant with, and not more than that. It should remain abstract; you cannot download > it. Therefore I think it should *not* include a coding interface. That would make perfect sense, except that the interface can affect the way you share code. We're already seeing people sharing both demonstration code and real working code as IPython notebook files, and that will probably increase as the format settles down. It's important that everyone using the standard can easily exchange code, although we've got ideas for a website that will aid that by making it easy to convert between file formats. Thanks, Thomas From gary.ruben at gmail.com Thu Aug 30 20:19:51 2012 From: gary.ruben at gmail.com (gary ruben) Date: Fri, 31 Aug 2012 10:19:51 +1000 Subject: [SciPy-User] Tensioned Spline interpolation In-Reply-To: <503FA344.4070006@gmail.com> References: <503FA344.4070006@gmail.com> Message-ID: Thanks Jonathan, I think the UnivariateSpline is from the *other* fitpack on netlib, which doesn't appear to support tensioned splines (aka tension splines). As I understand it the smoothing parameter in UnivariateSpline changes the spline from an interpolating spline that passes through all data points to an approximating curve that passes near the points. In contrast, a tensioned spline always passes through all data points and is typically a cubic spline with extra knots added to change the behaviour of the interpolating curve between the data points. Gary On 31 August 2012 03:30, Jonathan Stickel wrote: > Gary > > I am not familiar with the "tension" terminology, but it sounds like a > smoothing parameter. scipy.interpolate.UnivariateSpline provides splines > with smoothing. You might also try the scikit "datasmooth" > > http://pypi.python.org/pypi/scikits.datasmooth/0.61 > > While not exactly a spline implementation, it computes a trend-line through > 1D data with user-specified or automatically determined smoothing. > > HTH, > Jonathan > > > On 8/30/12 10:08 , scipy-user-request at scipy.org wrote: >> >> Date: Thu, 30 Aug 2012 17:22:21 +1000 >> From: gary ruben >> Subject: [SciPy-User] Tensioned Spline interpolation >> >> >> Hi, >> >> I'm rewriting some IDL code in Python. The original code uses IDL's >> spline function to interpolate a 1D ungridded data series onto a >> regular grid. Unfortunately, I can't find a close enough equivalent in >> scipy. I'm guessing the IDL spline function implements a "tensioned >> spline." >> >> The signature of the IDL spline function is >> Result = SPLINE(X, Y, T [, Sigma]) >> where Sigma is 'The amount of "tension" that is applied to the curve. >> If sigma is close to 0, (e.g., .01), then effectively there is a cubic >> spline fit. If sigma is large, (e.g., greater than 10), then the fit >> will be like a polynomial interpolation.' >> >> In the code I'm trying to reproduce, the author has set sigma=15 so >> the result is quite different to a simple cubic spline. >> Scipy's pchip algorithm gives a similar result, but is unfortunately >> too slow for my application. Can anyone confirm that IDL's spline is a >> tensioned spline routine? Does anyone know of a BSD-licensed Python >> module that implements a tensioned spline. Failing that any >> suggestions for a good routine I should look at wrapping (maybe Alan >> Cline's fitpack on netlib?) >> >> thanks, >> Gary From arsbbr at gmx.net Fri Aug 31 03:36:21 2012 From: arsbbr at gmx.net (arsbbr) Date: Fri, 31 Aug 2012 00:36:21 -0700 (PDT) Subject: [SciPy-User] Re[SciPy-user] binning to polar coordinates In-Reply-To: <20120828150107.be7a0dfb.Jerome.Kieffer@esrf.fr> References: <34348219.post@talk.nabble.com> <20120828150107.be7a0dfb.Jerome.Kieffer@esrf.fr> Message-ID: <34372124.post@talk.nabble.com> Dear Jerome, thank you for your tip! Unfortunately I'm getting big gaps for my real data in the small rho region. The optimal algorithm should interpolate in the small rho region and bin in the high rho region. I guess your split pixel solution would be best. But this a whole API with a lot of corrections and methods for real experimental data. Is it possible to use your sophisticated algorithm only for the transformation from Cartesian to a evenly stepped polar grid? Speed is not the issue with my data, so a numpy solution would be fine. Thanks a lot! Jerome Kieffer wrote: > > On Tue, 28 Aug 2012 03:33:11 -0700 (PDT) > arsbbr wrote: > >> >> Hi, >> >> I'm trying to rebin some uniform gridded data to polar coordinates. The >> original data (Z) is noisy but on a fine grid. >> >> ### snip >> >> # define grid in cartesian coordinates >> x = arange(-500, 500) >> y = arange(-500, 500) >> X, Y = meshgrid(x, y) >> Z = X**2 + Y**2 >> Z += uniform(-5000, 5000, size=Z.shape)) >> >> # transform to polar coordinates >> >> def cart2pol(x, y): >> theta = arctan2(y, x) >> rho = sqrt(x**2 + y**2) >> return (theta, rho) >> >> THETA_xy, RHO_xy = cart2pol(X, Y) >> >> # define new polar grid >> theta = arange(-pi, pi, 0.1) >> rho = arange(0.1 ,500, 2.0) >> >> THETA, RHO = meshgrid(theta, rho) >> >> # Tried to use griddata, but it gives of course very jumpy results, >> because >> it >> # does not take the average over the pixels in a patch (dTHETA, dRHO) but >> an >> interpolation at the >> # exact new point. >> >> interpolate.griddata((THETA_xy.ravel(), RHO_xy.ravel()), >> Z.ravel(), (THETA, RHO), method='linear') >> >> ### snip >> >> Is there a method that rebins the data to a new grid, possibly taking the >> average >> of the Z-values inside a patch (dTHETA, dRHO)? > > Try to use 2D-weighted histogram (divided by unweighted histograms). > This unfortunately does dot split pixels over various bins so it works > badly when the number of output bins is large. > > w=histogram2d(THETA_xy.ravel(),RHO_xy.ravel(),(int(2*pi/0.1),500//2),weights=Z.ravel()) > u=histogram2d(THETA_xy.ravel(),RHO_xy.ravel(),(int(2*pi/0.1),500//2)) > imshow (w[0]/u[0]) > > If you want to split pixels, I am working on more sophisticated algorithm > to do that in > https://github.com/kif/pyFAI Tell me if you are interested. > > Cheers, > -- > J?r?me Kieffer > On-Line Data analysis / Software Group > ISDD / ESRF > tel +33 476 882 445 > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- View this message in context: http://old.nabble.com/Rebinning-to-polar-coordinates-tp34348219p34372124.html Sent from the Scipy-User mailing list archive at Nabble.com. From Jerome.Kieffer at esrf.fr Fri Aug 31 04:31:15 2012 From: Jerome.Kieffer at esrf.fr (Jerome Kieffer) Date: Fri, 31 Aug 2012 10:31:15 +0200 Subject: [SciPy-User] Re[SciPy-user] binning to polar coordinates In-Reply-To: <34372124.post@talk.nabble.com> References: <34348219.post@talk.nabble.com> <20120828150107.be7a0dfb.Jerome.Kieffer@esrf.fr> <34372124.post@talk.nabble.com> Message-ID: <20120831103115.930e5b5d.Jerome.Kieffer@esrf.fr> On Fri, 31 Aug 2012 00:36:21 -0700 (PDT) arsbbr wrote: > thank you for your tip! Unfortunately I'm getting big gaps for my real data > in the small rho region. The optimal algorithm should interpolate in the > small rho region and bin in the high rho region. > I guess your split pixel solution would be best. But this a whole API with a > lot of corrections and methods for real experimental data. Is it possible to > use your sophisticated algorithm only for the transformation from Cartesian > to a evenly stepped polar grid? Speed is not the issue with my data, so a > numpy solution would be fine. I wish this would be possible using only numpy but I failed. I had to write it in a low-level (Cython) way to get decent speed because the algorithm is iterating over all input pixels. What you are looking for is in: https://github.com/kif/pyFAI/blob/master/src/splitBBox.pyx Use the function histoBBox2d(weights, pos0, delta_pos0, pos1, delta_pos1, bins=(100, 36)) If you refer to the manual of numpy.histogram2d: -> bins has the same meaning, i.e. the shape of the output image -> pos0 and pos1 are the same a x and y; they should contain the coordinated in (rho,theta) of the center of input pixels. -> delta_pos0 and delta_pos1 are the distance in (rho,theta) from pixel center to the pixel corner. This is of course new compared to numpy -> weights is as in numpy the intensity of input pixels. All array need to have the same size. The output is: -> your transformed image (maybe it's transposed), i.e. weighted_histogram/unweighted_histogram -> axis in rho (or theta) 1D -> axis in theta (or rho) 1D -> weighted histogram (2D) -> unweighted histogram (2D) splitBBox should be stand alone: just use this setup.py to compile the module (needs Cython) ######################################## try: from setuptools import setup except ImportError: from distutils.core import setup from distutils.core import Extension from Cython.Distutils import build_ext # for numpy from numpy.distutils.misc_util import get_numpy_include_dirs split_ext = Extension("splitBBox", include_dirs=get_numpy_include_dirs(), sources=['splitBBox.pyx'], setup(name='histogram', version="0.3.0", author="Jerome Kieffer", author_email="jerome.kieffer at esrf.eu", description='test for azim int', ext_modules=[split_ext], cmdclass={'build_ext': build_ext}, ) ##################################################### Nota: I am working on X-ray diffraction so Your "rho" could be named "2theta", tth or q. Your "theta" is usually named "chi" in my comments. Different sciences; different conventions :) I hope this helps, cheers, -- J?r?me Kieffer On-Line Data analysis / Software Group ISDD / ESRF tel +33 476 882 445 From Pierre.RAYBAUT at CEA.FR Fri Aug 31 11:39:40 2012 From: Pierre.RAYBAUT at CEA.FR (Pierre.RAYBAUT at CEA.FR) Date: Fri, 31 Aug 2012 17:39:40 +0200 Subject: [SciPy-User] [ANN] guiqwt v2.2.0 Message-ID: Hi all, I am pleased to announce that `guiqwt` v2.2.0 has been released (http://guiqwt.googlecode.com). Based on PyQwt (plotting widgets for PyQt4 graphical user interfaces) and on the scientific modules NumPy and SciPy, guiqwt is a Python library providing efficient 2D data-plotting features (curve/image visualization and related tools) for interactive computing and signal/image processing application development. The Mercurial repository is now publicly available here: http://code.google.com/p/guiqwt/source/checkout Complete change log is available here: http://code.google.com/p/guiqwt/wiki/ChangeLog Documentation with examples, API reference, etc. is available here: http://packages.python.org/guiqwt/ This version of `guiqwt` includes a demo software, Sift (for Signal and Image Filtering Tool), based on `guidata` and `guiqwt`: http://packages.python.org/guiqwt/sift.html Windows users may even download the portable version of Sift 0.2.6 to test it without having to install anything: http://code.google.com/p/guiqwt/downloads/detail?name=sift-0.2.6-guiqwt-2.2-win32.zip When compared to the excellent module `matplotlib`, the main advantages of `guiqwt` are: * Performance: see http://packages.python.org/guiqwt/overview.html#performances * Interactivity: see for example http://packages.python.org/guiqwt/_images/plot.png * Powerful signal processing tools: see for example http://packages.python.org/guiqwt/_images/fit.png * Powerful image processing tools: * Real-time contrast adjustment: http://packages.python.org/guiqwt/_images/contrast.png * Cross sections (line/column, averaged and oblique cross sections!): http://packages.python.org/guiqwt/_images/cross_section.png * Arbitrary affine transforms on images: http://packages.python.org/guiqwt/_images/transform.png * Interactive filters: http://packages.python.org/guiqwt/_images/imagefilter.png * Geometrical shapes/Measurement tools: http://packages.python.org/guiqwt/_images/image_plot_tools.png * Perfect integration of `guidata` features for image data editing: http://packages.python.org/guiqwt/_images/simple_window.png But `guiqwt` is more than a plotting library; it also provides: * Framework for signal/image processing application development: see http://packages.python.org/guiqwt/examples.html * And many other features like making executable Windows programs easily (py2exe helpers): see http://packages.python.org/guiqwt/disthelpers.html guiqwt has been successfully tested on GNU/Linux and Windows platforms. Python package index page: http://pypi.python.org/pypi/guiqwt/ Documentation, screenshots: http://packages.python.org/guiqwt/ Downloads (source + Python(x,y) plugin): http://guiqwt.googlecode.com -- Dr. Pierre Raybaut CEA - Commissariat ? l'Energie Atomique et aux Energies Alternatives From Pierre.RAYBAUT at CEA.FR Fri Aug 31 11:39:37 2012 From: Pierre.RAYBAUT at CEA.FR (Pierre.RAYBAUT at CEA.FR) Date: Fri, 31 Aug 2012 17:39:37 +0200 Subject: [SciPy-User] [ANN] guidata v1.5.0 Message-ID: Hi all, I am pleased to announce that `guidata` v1.5.0 has been released (http://guidata.googlecode.com). Based on the Qt Python binding module PyQt4 (and mostly compatible with PySide), guidata is a Python library generating graphical user interfaces for easy dataset editing and display. It also provides helpers and application development tools for PyQt4. guidata also provides the following features: * guidata.qthelpers: PyQt4 helpers * guidata.disthelpers: cx_Freeze/py2exe helpers (or how to build a self-consistent executable in three lines of code!) * guidata.userconfig: .ini configuration management helpers (based on Python standard module ConfigParser) * guidata.configtools: library/application data management * guidata.gettext_helpers: translation helpers (based on the GNU tool gettext) * guidata.guitest: automatic GUI-based test launcher * guidata.utils: miscelleneous utilities guidata has been successfully tested on GNU/Linux and Windows platforms. This is mostly a maintenance release with a couple of bugfixes and minor new features (see changelog here: http://code.google.com/p/guidata/wiki/ChangeLog). The Mercurial repository is now publicly available here: http://code.google.com/p/guidata/source/checkout The `guidata` documentation with examples, API reference, etc. is available here: http://packages.python.org/guidata/ Python package index page: http://pypi.python.org/pypi/guidata/ Documentation, screenshots: http://packages.python.org/guidata/ Downloads (source + Windows installers): http://guidata.googlecode.com -- Dr. Pierre Raybaut CEA - Commissariat ? l'Energie Atomique et aux Energies Alternatives From thoger.emil at gmail.com Fri Aug 31 11:58:03 2012 From: thoger.emil at gmail.com (=?ISO-8859-1?Q?Th=F8ger_Rivera-Thorsen?=) Date: Fri, 31 Aug 2012 17:58:03 +0200 Subject: [SciPy-User] [ANN] guidata v1.5.0 In-Reply-To: References: Message-ID: <5040DF0B.7000805@gmail.com> Looks very interesting! Should have known about it before I started my journey into the dark and largely undocumented woodlands of Chaco and Traits... ;-) On 08/31/2012 05:39 PM, Pierre.RAYBAUT at CEA.FR wrote: > Hi all, > > I am pleased to announce that `guidata` v1.5.0 has been released (http://guidata.googlecode.com). > > Based on the Qt Python binding module PyQt4 (and mostly compatible with PySide), guidata is a Python library generating graphical user interfaces for easy dataset editing and display. It also provides helpers and application development tools for PyQt4. > > guidata also provides the following features: > * guidata.qthelpers: PyQt4 helpers > * guidata.disthelpers: cx_Freeze/py2exe helpers (or how to build a self-consistent executable in three lines of code!) > * guidata.userconfig: .ini configuration management helpers (based on Python standard module ConfigParser) > * guidata.configtools: library/application data management > * guidata.gettext_helpers: translation helpers (based on the GNU tool gettext) > * guidata.guitest: automatic GUI-based test launcher > * guidata.utils: miscelleneous utilities > > guidata has been successfully tested on GNU/Linux and Windows platforms. > > This is mostly a maintenance release with a couple of bugfixes and minor new features (see changelog here: http://code.google.com/p/guidata/wiki/ChangeLog). > > The Mercurial repository is now publicly available here: > http://code.google.com/p/guidata/source/checkout > > The `guidata` documentation with examples, API reference, etc. is available here: > http://packages.python.org/guidata/ > > Python package index page: > http://pypi.python.org/pypi/guidata/ > > Documentation, screenshots: > http://packages.python.org/guidata/ > > Downloads (source + Windows installers): > http://guidata.googlecode.com > > -- > Dr. Pierre Raybaut > CEA - Commissariat ? l'Energie Atomique et aux Energies Alternatives > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user