From saullogiovani at gmail.com Fri Nov 1 11:01:09 2013 From: saullogiovani at gmail.com (Saullo Castro) Date: Fri, 1 Nov 2013 16:01:09 +0100 Subject: [SciPy-User] How to obtain the LU matrices when using `scipy.sparse.linalg.splu`? Message-ID: I've tried to obtain the `LU` matrices using this SuperLU wrapper without succes. In the `dgstrf` function documentation it says that the outputs are `L`, `U` and a `stat`. Is there any hidden parameter that I can pass to `scipy.sparse.linalg.splu` or `scipy.sparse.linalg.spilu` in order to obtain the `L` and `U` matrices instead of the `factored_lu` object? Greetings! Saullo -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Sun Nov 3 09:06:10 2013 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 03 Nov 2013 16:06:10 +0200 Subject: [SciPy-User] How to obtain the LU matrices when using `scipy.sparse.linalg.splu`? In-Reply-To: References: Message-ID: 01.11.2013 17:01, Saullo Castro kirjoitti: > I've tried to obtain the `LU` matrices using this SuperLU wrapper without > succes. http://article.gmane.org/gmane.comp.python.scientific.user/32315 From jeff.c.taylor at gmail.com Mon Nov 4 14:23:04 2013 From: jeff.c.taylor at gmail.com (Jeff Taylor) Date: Mon, 4 Nov 2013 14:23:04 -0500 Subject: [SciPy-User] Parameters in scipy.signal.find_peaks_cwt Message-ID: I am currently looking at find_peaks_cwt to see how well it works as part of a peak finding task. I need to know what each of the parameters do and whether their defaults are sensible in terms of using the algorithm for arc lines. For example, the default wavelet is the ricker wavelet, but are there any other wavelets that can be used? And are the widths the full-width half maximum or something else? Is there anywhere that more information can be found about each of the parameters, maybe along with examples of what happens when their values are changed? Thank you, Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Mon Nov 4 14:30:50 2013 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 04 Nov 2013 21:30:50 +0200 Subject: [SciPy-User] Parameters in scipy.signal.find_peaks_cwt In-Reply-To: References: Message-ID: 04.11.2013 21:23, Jeff Taylor kirjoitti: > I am currently looking at find_peaks_cwt to see how well it works > as part of a peak finding task. I need to know what each of the > parameters do and whether their defaults are sensible in terms of > using the algorithm for arc lines. For example, the default wavelet > is the ricker wavelet, but are there any other wavelets that can be > used? And are the widths the full-width half maximum or something > else? > > Is there anywhere that more information can be found about each of > the parameters, maybe along with examples of what happens when > their values are changed? Your best bet is probably to look at the publication referred to in the documentation of that function. I'm not sure if the original author of that function is still around. -- Pauli Virtanen From a.klein at science-applied.nl Tue Nov 5 11:05:09 2013 From: a.klein at science-applied.nl (Almar Klein) Date: Tue, 5 Nov 2013 17:05:09 +0100 Subject: [SciPy-User] [ANN] Vispy 0.2 for OpenGL-based data visualization Message-ID: This did not seem to come through. ---------- Forwarded message ---------- From: Cyrille Rossant Date: 2013/11/4 Subject: [ANN] Vispy 0.2 for OpenGL-based data visualization To: scipy-user at scipy.org Dear all, I'm pleased to announce that we've just released Vispy 0.2 (http://vispy.org) ! Vispy is a hardware-accelerated interactive visualization library in Python that brings OpenGL to the masses. In this release we improved the OpenGL object-oriented interface. The next step for us will be to design higher-level interfaces. Those will let users not necessarily familiar with OpenGL design beautiful and fast visualizations in Python (including data visualization and scientific plotting). Please report bugs and feature requests at our issue tracker: https://github.com/vispy/vispy/issues All the best, Cyrille PS: Vispy will be at the Budapest BI Forum this week! http://budapestbiforum.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Tue Nov 5 15:44:18 2013 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 05 Nov 2013 22:44:18 +0200 Subject: [SciPy-User] Scipy cookbook to Ipython notebooks Message-ID: Hi, It would be useful to have the set of code snippets currently in Scipy cookbook (http://wiki.scipy.org/Cookbook/) be converted to a set of Ipython notebooks. The notebook format is probably the easiest-to-work-with format for this available currently, and I think I prefer it over Pweave as the source format. Here's the conversion, which was essentially done by Matti Pastell half a year ago: https://github.com/pv/Scipy-CookBook The files still need some manual tuning up --- the conversion was not 100% flawless, and there's stuff that cannot be done automatically (replacing images with those outputted by the plots etc.). To work with them, you can just launch ipython notebook in the directory where the .ipynb files are. I don't have the time to work on this on the short term myself, so if someone wants to give a whirl at it, that would be useful. The plan is to either stick these up at scipy.org, or put them up on scipy-central.org when it gets ipython support. Some of the examples are probably quite outdated, but it may be of historical interest to preserve them. (A notice with a date could be added on top of outdated content.) Best regards, Pauli Virtanen From dpm at umn.edu Wed Nov 6 02:06:58 2013 From: dpm at umn.edu (David P Morrissey) Date: Wed, 6 Nov 2013 07:06:58 +0000 (UTC) Subject: [SciPy-User] Sparse Jacobian for optimize.root Message-ID: Hi, I have been writing some continuation programs using SciPy to solve differential equations. The main step is solving a large set of nonlinear equations using the optimize package. For a model case the Jacobian below is tridiagonal. Is there a way to use sparse solvers for multivariate root solvers in SciPY? Ideally I would be able to enter a sparse LinearOperator for the Jacobian and have linear computation times. Is something like this possible? Thanks, David import numpy as np from scipy import optimize from scipy import sparse import matplotlib.pyplot as plt N = 200 L = 2*np.pi x = np.linspace(0,L,N) mu = 10 h = float(L)/(N-1) BC = sparse.lil_matrix( (N,N) ) BC[0,0]=1 BC[N-1,N-1]=1 BC.tocsr() lin = (1/h/h)*( sparse.diags([np.ones(N-1),-2*np.ones(N),np.ones(N-1)], [-1,0,1]) + BC ) + mu*sparse.eye(N) def f(u): return lin*u - u**3 def J(u): J = lin - sparse.diags(3*u**2,0) #J = sparse.linalg.aslinearoperator(J) return J.todense() soln = optimize.root(f,np.sqrt(4*mu/3)*np.cos(x),jac=J) y = soln.x plt.plot(x,y) plt.show() From pav at iki.fi Wed Nov 6 11:23:19 2013 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 6 Nov 2013 16:23:19 +0000 (UTC) Subject: [SciPy-User] Sparse Jacobian for optimize.root References: Message-ID: David P Morrissey umn.edu> writes: > I have been writing some continuation programs using SciPy to solve > differential equations. The main step is solving a large set of > nonlinear equations using the optimize package. For a model case the > Jacobian below is tridiagonal. Is there a way to use sparse solvers for > multivariate root solvers in SciPY? This is not supported at the moment, but it's on the roadmap for Scipy 1.0. It's however relatively easy to whip up a line search Newton solver using the existing line searches. Also, the HYBRD solver however does support banded matrices IIRC. -- Pauli Virtanen From anthony.j.mannucci at jpl.nasa.gov Wed Nov 6 13:35:11 2013 From: anthony.j.mannucci at jpl.nasa.gov (Mannucci, Anthony J (335G)) Date: Wed, 6 Nov 2013 18:35:11 +0000 Subject: [SciPy-User] Netcdf exception behavior Message-ID: This snippet of code does not provide the behavior I expect: >>> import scipy.io.netcdf as CDF import sys fitfile = 'simpleXXX.nc' try: fnc = CDF.netcdf_file(fitfile, 'r') except: sys.stderr.write("WARNING: error.\n") else: print fnc.variables fnc.close() <<< If fitfile exists, the read occurs as expected. If the fitfile does not exist, the following message is generated: WARNING: error. Exception AttributeError: "'netcdf_file' object has no attribute 'fp'" in > ignored I expected the exception handling code to eliminate the python exception handler. Thanks for any help. -Tony -- Tony Mannucci Supervisor, Ionospheric and Atmospheric Remote Sensing Group Mail-Stop 138-308, Tel > (818) 354-1699 Jet Propulsion Laboratory, Fax > (818) 393-5115 California Institute of Technology, Email > Tony.Mannucci at jpl.nasa.gov 4800 Oak Grove Drive, http://scienceandtechnology.jpl.nasa.gov/people/a_mannucci/ Pasadena, CA 91109 -------------- next part -------------- An HTML attachment was scrubbed... URL: From doutriaux1 at llnl.gov Wed Nov 6 18:46:15 2013 From: doutriaux1 at llnl.gov (Doutriaux, Charles) Date: Wed, 6 Nov 2013 23:46:15 +0000 Subject: [SciPy-User] scipy 0.13 on Maverick Message-ID: Hello, I?m trying to build scipy on Maverick (Apple latest OS) When running python setup.py install I get: 8463 /usr/local/bin/gfortran -Wall -Wall -undefined dynamic_lookup -bundle build/temp.macosx-10.4-x86_64-2.7/build/src.macosx-10.4-x86_64-2.7/scipy/fftpack/_fftpac kmodule.o build/temp.macosx-10.4-x86_64-2.7/scipy/fftpack/src/zfft.o build/temp.macosx-10.4-x86_64-2.7/scipy/fftpack/src/drfft.o build/temp.macosx-10.4-x86_64 -2.7/scipy/fftpack/src/zrfft.o build/temp.macosx-10.4-x86_64-2.7/scipy/fftpack/src/zfftnd.o build/temp.macosx-10.4-x86_64-2.7/build/src.macosx-10.4-x86_64-2.7 /scipy/fftpack/src/dct.o build/temp.macosx-10.4-x86_64-2.7/build/src.macosx-10.4-x86_64-2.7/scipy/fftpack/src/dst.o build/temp.macosx-10.4-x86_64-2.7/build/sr c.macosx-10.4-x86_64-2.7/fortranobject.o -L/usr/local/lib/gcc/x86_64-apple-darwin13.0.0/4.9.0 -Lbuild/temp.macosx-10.4-x86_64-2.7 -ldfftpack -lfftpack -lgfort ran -o build/lib.macosx-10.4-x86_64-2.7/scipy/fftpack/_fftpack.so 8464 ld: library not found for -lbundle1.o 8465 collect2: error: ld returned 1 exit status 8466 ld: library not found for -lbundle1.o 8467 collect2: error: ld returned 1 exit status But if I copy paste this command and run it straight from the command line, it works. Any idea what?s going on? PS. I tried Python 2.7.4 and 2.7.5 C. -------------- next part -------------- An HTML attachment was scrubbed... URL: From h.chr at mail.ru Thu Nov 7 00:18:17 2013 From: h.chr at mail.ru (Horea Christian) Date: Thu, 07 Nov 2013 06:18:17 +0100 Subject: [SciPy-User] ttest_rel with unequal groups Message-ID: <527B2299.60706@mail.ru> Hey there! I would like to use the ttest_rel function to compare reaction times for two conditions tested over 10 participants. We have done 100 trials per participant, but some of them had errors and were excluded. For instance for prticipants 1 and 2 I have condition1: 95 trials, condition2: 100 trials AND condition1:100 trials and condition2: 99 trials. depending on whether or not I transpose my dataframe I get a complaint either at if a.shape[axis] != b.shape[axis]: raise ValueError('unequal length arrays') or at d = (a - b).astype(np.float64) . What can I do about this? I found it surprising that it doesn't "just work" since in most experiments it is expected for some of the measurements to fail. Many Thanks! Christian -- Horea Christian http://chymera.eu From lzh at mwlab.pmo.ac.cn Thu Nov 7 01:22:17 2013 From: lzh at mwlab.pmo.ac.cn (mr_lin) Date: Wed, 6 Nov 2013 22:22:17 -0800 (PST) Subject: [SciPy-User] scipy.signal.periodogram problem Message-ID: <1383805337629-18844.post@n7.nabble.com> when using periodogram() to calculate sine wave spectrum with hanning window, I find a problem, the following is data processing : case one: 1) sine-wave is modulated with hanning window first, 2) then periodogram function is used to estimate sine-wave amplitude spectrum(parameter window=None);3) a factor U=1/N*sum(win[i]**2) is corrected in spectrum. case two: 1) periodogram is directly used to calculate sine-wave(window is set hanning window); 2) then amplitude spectrum is calculated. the result in case two/the one in case on = 1.2247822.... WHY two results is not equal in these two case? post my code: fs = 10e3 N = 16384 f0 = 2345 time = np.arange(N)/fs sig = 5.* np.sin(2*np.pi*f0*time) h_win = han_win(N) U = 0. for i in range(N): U += h_win[i]**2 U /= N sig_win = (sig-np.mean(sig))* h_win freq1,Pxx_spec_w1 = sp_sig.periodogram(sig,fs,h_win,return_onesided=True,scaling='spectrum') freq2,Pxx_spec_w2 = sp_sig.periodogram(sig_win,fs,window=None,return_onesided=True,scaling='spectrum') sig_amp_w1 = np.sqrt(Pxx_spec_w1) sig_amp_w2 = np.sqrt(Pxx_spec_w2/U) print 'E{sig_amp_w1/sig_amp_w2}=',np.mean(sig_amp_w1[1:]/sig_amp_w2[1:]) -- View this message in context: http://scipy-user.10969.n7.nabble.com/scipy-signal-periodogram-problem-tp18844.html Sent from the Scipy-User mailing list archive at Nabble.com. From hturesson at gmail.com Thu Nov 7 04:52:57 2013 From: hturesson at gmail.com (Hjalmar Turesson) Date: Thu, 7 Nov 2013 07:52:57 -0200 Subject: [SciPy-User] ttest_rel with unequal groups In-Reply-To: <527B2299.60706@mail.ru> References: <527B2299.60706@mail.ru> Message-ID: Hi, If I'm not confused, ttest_rel is a paired samples ttest ( http://en.wikipedia.org/wiki/Paired_difference_test), and thus requires that all samples are paired (this does not depend on the particular scipy implementation). If occasional samples in a group are missing, and you still want perform the paired ttest, then you will probably have to exclude the corresponding sample in the other 2nd, or generate pseudo-values to replace the missing values in the 1st group. Alternatively, you can use ttest_ind ( http://en.wikipedia.org/wiki/Ttest#Independent_samples), which doesn't require exactly the same number of samples in the two groups. On Thu, Nov 7, 2013 at 2:18 AM, Horea Christian wrote: > Hey there! I would like to use the ttest_rel function to compare > reaction times for two conditions tested over 10 participants. We have > done 100 trials per participant, but some of them had errors and were > excluded. For instance for prticipants 1 and 2 I have condition1: 95 > trials, condition2: 100 trials AND condition1:100 trials and condition2: > 99 trials. > > depending on whether or not I transpose my dataframe I get a complaint > either at > > if a.shape[axis] != b.shape[axis]: > raise ValueError('unequal length arrays') > > or at > > d = (a - b).astype(np.float64) > > . > > > What can I do about this? I found it surprising that it doesn't "just > work" since in most experiments it is expected for some of the > measurements to fail. > > Many Thanks! > Christian > > -- > Horea Christian > http://chymera.eu > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From h.chr at mail.ru Thu Nov 7 22:51:06 2013 From: h.chr at mail.ru (Horea Christian) Date: Fri, 08 Nov 2013 04:51:06 +0100 Subject: [SciPy-User] ttest_rel with unequal groups In-Reply-To: References: <527B2299.60706@mail.ru> Message-ID: <527C5FAA.6040500@mail.ru> I managed to get tteste_rel to work by replacinf my missing values either with NaN or with False . I am yet to determine whether or not that distorts my data (could be that d = (a - b).astype(np.float64) is zero for entries where one value is false, or that false is read as zero and d = (a - b).astype(np.float64) will be -b[x] wherever a[x] is false...) In any case, I am a bit uncertain as to the usage of this method - am I supposed to pass it a 1d array? or a 2d array? I am thinking 2d shouled be mandatory because otherwise the method can't tell which groups measures are related. I tried doing that (my array being N(participants) x N(measurements) ) but that gave me a 2d output - that can't be right, I just want one t and one p value, not a multidim array. So, how do I use this? (The docs are not very informative on what happens to 2d vs 1d inputs). Cheers, christian On Do 07 Nov 2013 10:52:57 CET, Hjalmar Turesson wrote: > Hi, > > If I'm not confused, ttest_rel is a paired samples ttest > (http://en.wikipedia.org/wiki/Paired_difference_test), and thus > requires that all samples are paired (this does not depend on the > particular scipy implementation). > If occasional samples in a group are missing, and you still want > perform the paired ttest, then you will probably have to exclude the > corresponding sample in the other 2nd, or generate pseudo-values to > replace the missing values in the 1st group. Alternatively, you can > use ttest_ind > (http://en.wikipedia.org/wiki/Ttest#Independent_samples), which > doesn't require exactly the same number of samples in the two groups. > > > On Thu, Nov 7, 2013 at 2:18 AM, Horea Christian > wrote: > > Hey there! I would like to use the ttest_rel function to compare > reaction times for two conditions tested over 10 participants. We have > done 100 trials per participant, but some of them had errors and were > excluded. For instance for prticipants 1 and 2 I have condition1: 95 > trials, condition2: 100 trials AND condition1:100 trials and > condition2: > 99 trials. > > depending on whether or not I transpose my dataframe I get a complaint > either at > > if a.shape[axis] != b.shape[axis]: > raise ValueError('unequal length arrays') > > or at > > d = (a - b).astype(np.float64) > > . > > > What can I do about this? I found it surprising that it doesn't "just > work" since in most experiments it is expected for some of the > measurements to fail. > > Many Thanks! > Christian > > -- > Horea Christian > http://chymera.eu > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -- Horea Christian http://chymera.eu From josef.pktd at gmail.com Fri Nov 8 06:04:08 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 8 Nov 2013 06:04:08 -0500 Subject: [SciPy-User] ttest_rel with unequal groups In-Reply-To: <527C5FAA.6040500@mail.ru> References: <527B2299.60706@mail.ru> <527C5FAA.6040500@mail.ru> Message-ID: On Thu, Nov 7, 2013 at 10:51 PM, Horea Christian wrote: > I managed to get tteste_rel to work by replacinf my missing values > either with NaN or with False . I am yet to determine whether or not > that distorts my data (could be that d = (a - b).astype(np.float64) is > zero for entries where one value is false, or that false is read as > zero and d = (a - b).astype(np.float64) will be -b[x] wherever a[x] is > false...) > It will distort your results, since it is treated as non-missing observation which affects both the estimated difference and the number of observations, the degrees of freedom for the p-value. > > In any case, I am a bit uncertain as to the usage of this method - am I > supposed to pass it a 1d array? or a 2d array? I am thinking 2d shouled > be mandatory because otherwise the method can't tell which groups > measures are related. I tried doing that (my array being > N(participants) x N(measurements) ) but that gave me a 2d output - that > can't be right, I just want one t and one p value, not a multidim array. > > So, how do I use this? (The docs are not very informative on what > happens to 2d vs 1d inputs). > you need to give it two arrays, the difference between the arrays is calculated internally. If the arrays are 2d, then the test is calculated for each column (or along axis) of the broadcasted difference. These are separate tests for each column that give the same result as looping over the columns. If one array has only one column (for example a benchmark treatment), the other array has several columns, then we get ttest_rel for each comparison of a second column to the first array. The result will be as many tstatistics and pvalues as there are columns. There is no multiple testing correction for the pvalues >>> outcome = np.random.randn(20, 4) + [0, 0, 1, 2] >>> from scipy import stats >>> stats.ttest_rel(outcome[:, :1], outcome[:, 1:]) (array([-1.60220806, -3.13556782, -7.1567637 ]), array([ 1.25604679e-01, 5.44534856e-03, 8.41006537e-07])) >>> [stats.ttest_rel(outcome[:, 0], outcome[:, k]) for k in range(1, 4)] [(array(-1.6022080647700057), 0.12560467940402195), (array(-3.135567822455234), 0.005445348556616313), (array(-7.156763700790868), 8.4100653703218436e-07)] aside: I think the following is doing the right thing for testing the joint hypothesis >>> diff = outcome[:, 1:] - outcome[:, :1] >>> stats.f_oneway(*diff.T) (10.606594036595835, 0.00012132595252973279) Josef > > Cheers, > christian > > On Do 07 Nov 2013 10:52:57 CET, Hjalmar Turesson wrote: > > Hi, > > > > If I'm not confused, ttest_rel is a paired samples ttest > > (http://en.wikipedia.org/wiki/Paired_difference_test), and thus > > requires that all samples are paired (this does not depend on the > > particular scipy implementation). > > If occasional samples in a group are missing, and you still want > > perform the paired ttest, then you will probably have to exclude the > > corresponding sample in the other 2nd, or generate pseudo-values to > > replace the missing values in the 1st group. Alternatively, you can > > use ttest_ind > > (http://en.wikipedia.org/wiki/Ttest#Independent_samples), which > > doesn't require exactly the same number of samples in the two groups. > > > > > > On Thu, Nov 7, 2013 at 2:18 AM, Horea Christian > > wrote: > > > > Hey there! I would like to use the ttest_rel function to compare > > reaction times for two conditions tested over 10 participants. We > have > > done 100 trials per participant, but some of them had errors and were > > excluded. For instance for prticipants 1 and 2 I have condition1: 95 > > trials, condition2: 100 trials AND condition1:100 trials and > > condition2: > > 99 trials. > > > > depending on whether or not I transpose my dataframe I get a > complaint > > either at > > > > if a.shape[axis] != b.shape[axis]: > > raise ValueError('unequal length arrays') > > > > or at > > > > d = (a - b).astype(np.float64) > > > > . > > > > > > What can I do about this? I found it surprising that it doesn't "just > > work" since in most experiments it is expected for some of the > > measurements to fail. > > > > Many Thanks! > > Christian > > > > -- > > Horea Christian > > http://chymera.eu > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > > > > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- > Horea Christian > http://chymera.eu > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Fri Nov 8 06:26:13 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 8 Nov 2013 06:26:13 -0500 Subject: [SciPy-User] ttest_rel with unequal groups In-Reply-To: References: <527B2299.60706@mail.ru> <527C5FAA.6040500@mail.ru> Message-ID: On Fri, Nov 8, 2013 at 6:04 AM, wrote: > > > > On Thu, Nov 7, 2013 at 10:51 PM, Horea Christian wrote: > >> I managed to get tteste_rel to work by replacinf my missing values >> either with NaN or with False . I am yet to determine whether or not >> that distorts my data (could be that d = (a - b).astype(np.float64) is >> zero for entries where one value is false, or that false is read as >> zero and d = (a - b).astype(np.float64) will be -b[x] wherever a[x] is >> false...) >> > > It will distort your results, since it is treated as non-missing > observation which affects both the estimated difference and the number of > observations, the degrees of freedom for the p-value. > Instead of deleting the missing observations, you can also use stats.mstats, which does the deletion internally >>> stats.mstats.ttest_rel(outcome[:, :1], outcome[:, 1:], axis=0) (array([-1.60220806, -3.13556782, -7.1567637 ]), masked_array(data = [ 1.25604679e-01 5.44534856e-03 8.41006537e-07], mask = False, fill_value = 1e+20) >>> [stats.mstats.ttest_rel(np.ma.masked_array(outcome[:, 0]), outcome[:, k]) for k in range(1, 4)] [(array(-1.6022080647700057), masked_array(data = 0.125604679404, mask = False, fill_value = 1e+20) ), (array(-3.135567822455234), masked_array(data = 0.00544534855662, mask = False, fill_value = 1e+20) ), (array(-7.156763700790863), masked_array(data = 8.41006537032e-07, mask = False, fill_value = 1e+20) )] mstats.ttest_rel has the wrong axis default (None instead of 0) and raises an exception on non masked arrays, when axis=None Looks like a BUG. Josef > > > >> >> In any case, I am a bit uncertain as to the usage of this method - am I >> supposed to pass it a 1d array? or a 2d array? I am thinking 2d shouled >> be mandatory because otherwise the method can't tell which groups >> measures are related. I tried doing that (my array being >> N(participants) x N(measurements) ) but that gave me a 2d output - that >> can't be right, I just want one t and one p value, not a multidim array. >> >> So, how do I use this? (The docs are not very informative on what >> happens to 2d vs 1d inputs). >> > > you need to give it two arrays, the difference between the arrays is > calculated internally. > > If the arrays are 2d, then the test is calculated for each column (or > along axis) of the broadcasted difference. > These are separate tests for each column that give the same result as > looping over the columns. > > If one array has only one column (for example a benchmark treatment), the > other array has several columns, then we get ttest_rel for each comparison > of a second column to the first array. > > The result will be as many tstatistics and pvalues as there are columns. > There is no multiple testing correction for the pvalues > > >>> outcome = np.random.randn(20, 4) + [0, 0, 1, 2] > >>> from scipy import stats > >>> stats.ttest_rel(outcome[:, :1], outcome[:, 1:]) > (array([-1.60220806, -3.13556782, -7.1567637 ]), array([ 1.25604679e-01, > 5.44534856e-03, 8.41006537e-07])) > > >>> [stats.ttest_rel(outcome[:, 0], outcome[:, k]) for k in range(1, 4)] > [(array(-1.6022080647700057), 0.12560467940402195), > (array(-3.135567822455234), 0.005445348556616313), > (array(-7.156763700790868), 8.4100653703218436e-07)] > > > aside: I think the following is doing the right thing for testing the > joint hypothesis > > >>> diff = outcome[:, 1:] - outcome[:, :1] > >>> stats.f_oneway(*diff.T) > (10.606594036595835, 0.00012132595252973279) > > > Josef > > > >> >> Cheers, >> christian >> >> On Do 07 Nov 2013 10:52:57 CET, Hjalmar Turesson wrote: >> > Hi, >> > >> > If I'm not confused, ttest_rel is a paired samples ttest >> > (http://en.wikipedia.org/wiki/Paired_difference_test), and thus >> > requires that all samples are paired (this does not depend on the >> > particular scipy implementation). >> > If occasional samples in a group are missing, and you still want >> > perform the paired ttest, then you will probably have to exclude the >> > corresponding sample in the other 2nd, or generate pseudo-values to >> > replace the missing values in the 1st group. Alternatively, you can >> > use ttest_ind >> > (http://en.wikipedia.org/wiki/Ttest#Independent_samples), which >> > doesn't require exactly the same number of samples in the two groups. >> > >> > >> > On Thu, Nov 7, 2013 at 2:18 AM, Horea Christian > > > wrote: >> > >> > Hey there! I would like to use the ttest_rel function to compare >> > reaction times for two conditions tested over 10 participants. We >> have >> > done 100 trials per participant, but some of them had errors and >> were >> > excluded. For instance for prticipants 1 and 2 I have condition1: 95 >> > trials, condition2: 100 trials AND condition1:100 trials and >> > condition2: >> > 99 trials. >> > >> > depending on whether or not I transpose my dataframe I get a >> complaint >> > either at >> > >> > if a.shape[axis] != b.shape[axis]: >> > raise ValueError('unequal length arrays') >> > >> > or at >> > >> > d = (a - b).astype(np.float64) >> > >> > . >> > >> > >> > What can I do about this? I found it surprising that it doesn't >> "just >> > work" since in most experiments it is expected for some of the >> > measurements to fail. >> > >> > Many Thanks! >> > Christian >> > >> > -- >> > Horea Christian >> > http://chymera.eu >> > >> > _______________________________________________ >> > SciPy-User mailing list >> > SciPy-User at scipy.org >> > http://mail.scipy.org/mailman/listinfo/scipy-user >> > >> > >> > >> > >> > _______________________________________________ >> > SciPy-User mailing list >> > SciPy-User at scipy.org >> > http://mail.scipy.org/mailman/listinfo/scipy-user >> >> -- >> Horea Christian >> http://chymera.eu >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Fri Nov 8 06:38:40 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 8 Nov 2013 06:38:40 -0500 Subject: [SciPy-User] ttest_rel with unequal groups In-Reply-To: References: <527B2299.60706@mail.ru> <527C5FAA.6040500@mail.ru> Message-ID: On Fri, Nov 8, 2013 at 6:26 AM, wrote: > > > > On Fri, Nov 8, 2013 at 6:04 AM, wrote: > >> >> >> >> On Thu, Nov 7, 2013 at 10:51 PM, Horea Christian wrote: >> >>> I managed to get tteste_rel to work by replacinf my missing values >>> either with NaN or with False . I am yet to determine whether or not >>> that distorts my data (could be that d = (a - b).astype(np.float64) is >>> zero for entries where one value is false, or that false is read as >>> zero and d = (a - b).astype(np.float64) will be -b[x] wherever a[x] is >>> false...) >>> >> >> It will distort your results, since it is treated as non-missing >> observation which affects both the estimated difference and the number of >> observations, the degrees of freedom for the p-value. >> > > Instead of deleting the missing observations, you can also use > stats.mstats, which does the deletion internally > > > >>> stats.mstats.ttest_rel(outcome[:, :1], outcome[:, 1:], axis=0) > (array([-1.60220806, -3.13556782, -7.1567637 ]), masked_array(data = [ > 1.25604679e-01 5.44534856e-03 8.41006537e-07], > mask = False, > fill_value = 1e+20) > > >>> [stats.mstats.ttest_rel(np.ma.masked_array(outcome[:, 0]), outcome[:, > k]) for k in range(1, 4)] > [(array(-1.6022080647700057), masked_array(data = 0.125604679404, > mask = False, > fill_value = 1e+20) > ), (array(-3.135567822455234), masked_array(data = 0.00544534855662, > mask = False, > fill_value = 1e+20) > ), (array(-7.156763700790863), masked_array(data = 8.41006537032e-07, > mask = False, > fill_value = 1e+20) > )] > > > mstats.ttest_rel has the wrong axis default (None instead of 0) > and raises an exception on non masked arrays, when axis=None > Looks like a BUG. > https://github.com/scipy/scipy/issues/3047 > > Josef > > > >> >> >> >>> >>> In any case, I am a bit uncertain as to the usage of this method - am I >>> supposed to pass it a 1d array? or a 2d array? I am thinking 2d shouled >>> be mandatory because otherwise the method can't tell which groups >>> measures are related. I tried doing that (my array being >>> N(participants) x N(measurements) ) but that gave me a 2d output - that >>> can't be right, I just want one t and one p value, not a multidim array. >>> >>> So, how do I use this? (The docs are not very informative on what >>> happens to 2d vs 1d inputs). >>> >> >> you need to give it two arrays, the difference between the arrays is >> calculated internally. >> >> If the arrays are 2d, then the test is calculated for each column (or >> along axis) of the broadcasted difference. >> These are separate tests for each column that give the same result as >> looping over the columns. >> >> If one array has only one column (for example a benchmark treatment), the >> other array has several columns, then we get ttest_rel for each comparison >> of a second column to the first array. >> >> The result will be as many tstatistics and pvalues as there are columns. >> There is no multiple testing correction for the pvalues >> >> >>> outcome = np.random.randn(20, 4) + [0, 0, 1, 2] >> >>> from scipy import stats >> >>> stats.ttest_rel(outcome[:, :1], outcome[:, 1:]) >> (array([-1.60220806, -3.13556782, -7.1567637 ]), array([ 1.25604679e-01, >> 5.44534856e-03, 8.41006537e-07])) >> >> >>> [stats.ttest_rel(outcome[:, 0], outcome[:, k]) for k in range(1, 4)] >> [(array(-1.6022080647700057), 0.12560467940402195), >> (array(-3.135567822455234), 0.005445348556616313), >> (array(-7.156763700790868), 8.4100653703218436e-07)] >> >> >> aside: I think the following is doing the right thing for testing the >> joint hypothesis >> >> >>> diff = outcome[:, 1:] - outcome[:, :1] >> >>> stats.f_oneway(*diff.T) >> (10.606594036595835, 0.00012132595252973279) >> >> >> Josef >> >> >> >>> >>> Cheers, >>> christian >>> >>> On Do 07 Nov 2013 10:52:57 CET, Hjalmar Turesson wrote: >>> > Hi, >>> > >>> > If I'm not confused, ttest_rel is a paired samples ttest >>> > (http://en.wikipedia.org/wiki/Paired_difference_test), and thus >>> > requires that all samples are paired (this does not depend on the >>> > particular scipy implementation). >>> > If occasional samples in a group are missing, and you still want >>> > perform the paired ttest, then you will probably have to exclude the >>> > corresponding sample in the other 2nd, or generate pseudo-values to >>> > replace the missing values in the 1st group. Alternatively, you can >>> > use ttest_ind >>> > (http://en.wikipedia.org/wiki/Ttest#Independent_samples), which >>> > doesn't require exactly the same number of samples in the two groups. >>> > >>> > >>> > On Thu, Nov 7, 2013 at 2:18 AM, Horea Christian >> > > wrote: >>> > >>> > Hey there! I would like to use the ttest_rel function to compare >>> > reaction times for two conditions tested over 10 participants. We >>> have >>> > done 100 trials per participant, but some of them had errors and >>> were >>> > excluded. For instance for prticipants 1 and 2 I have condition1: >>> 95 >>> > trials, condition2: 100 trials AND condition1:100 trials and >>> > condition2: >>> > 99 trials. >>> > >>> > depending on whether or not I transpose my dataframe I get a >>> complaint >>> > either at >>> > >>> > if a.shape[axis] != b.shape[axis]: >>> > raise ValueError('unequal length arrays') >>> > >>> > or at >>> > >>> > d = (a - b).astype(np.float64) >>> > >>> > . >>> > >>> > >>> > What can I do about this? I found it surprising that it doesn't >>> "just >>> > work" since in most experiments it is expected for some of the >>> > measurements to fail. >>> > >>> > Many Thanks! >>> > Christian >>> > >>> > -- >>> > Horea Christian >>> > http://chymera.eu >>> > >>> > _______________________________________________ >>> > SciPy-User mailing list >>> > SciPy-User at scipy.org >>> > http://mail.scipy.org/mailman/listinfo/scipy-user >>> > >>> > >>> > >>> > >>> > _______________________________________________ >>> > SciPy-User mailing list >>> > SciPy-User at scipy.org >>> > http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> -- >>> Horea Christian >>> http://chymera.eu >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yw5aj at virginia.edu Fri Nov 8 08:18:25 2013 From: yw5aj at virginia.edu (Yuxiang Wang) Date: Fri, 8 Nov 2013 08:18:25 -0500 Subject: [SciPy-User] Extracting data from a figure in a paper Message-ID: Dear all, Does anyone know any python alternatives to GRABIT or DataThief (both attached below)? GRABIT: http://www.mathworks.com.au/matlabcentral/fileexchange/7173-grabit DataThief: http://www.datathief.org/ These two software are used to extract data points from a research paper figure, I guess usually happens when the author did not reply the email... And I truly believe I am not the only person who needs this, from this post: http://academia.stackexchange.com/questions/7671/software-for-extracting-data-from-a-graph I have written a simple python snippet myself using pyHook, yet I was wondering, probably there is already something already out there, which may be much more powerful. Thank you! -Shawn -------------- next part -------------- An HTML attachment was scrubbed... URL: From gary.ruben at gmail.com Fri Nov 8 08:30:37 2013 From: gary.ruben at gmail.com (gary ruben) Date: Sat, 9 Nov 2013 00:30:37 +1100 Subject: [SciPy-User] Extracting data from a figure in a paper In-Reply-To: References: Message-ID: I haven't tried it, but here: http://yoinkery.com/ On 9 November 2013 00:18, Yuxiang Wang wrote: > Dear all, > > Does anyone know any python alternatives to GRABIT or DataThief (both > attached below)? > GRABIT: http://www.mathworks.com.au/matlabcentral/fileexchange/7173-grabit > DataThief: http://www.datathief.org/ > > These two software are used to extract data points from a research paper > figure, I guess usually happens when the author did not reply the email... > And I truly believe I am not the only person who needs this, from this post: > > http://academia.stackexchange.com/questions/7671/software-for-extracting-data-from-a-graph > > I have written a simple python snippet myself using pyHook, yet I was > wondering, probably there is already something already out there, which may > be much more powerful. > > Thank you! > > -Shawn > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jr at sun.ac.za Fri Nov 8 08:55:26 2013 From: jr at sun.ac.za (Johann Rohwer) Date: Fri, 08 Nov 2013 15:55:26 +0200 Subject: [SciPy-User] Extracting data from a figure in a paper In-Reply-To: References: Message-ID: <4631948.sFSAzLLRcD@biochem433789> Does it have to be python? We've used Engauge Digitizer (http://digitizer.sourceforge.net/) quite effectively for this purpose. It's written in C++ but fully open-source and multi platform. Johann On Friday 08 November 2013 08:18:25 Yuxiang Wang wrote: > Dear all, > > Does anyone know any python alternatives to GRABIT or DataThief (both > attached below)? > GRABIT: http://www.mathworks.com.au/matlabcentral/fileexchange/7173-grabit > DataThief: http://www.datathief.org/ > > These two software are used to extract data points from a research paper > figure, I guess usually happens when the author did not reply the email... > And I truly believe I am not the only person who needs this, from this post: > http://academia.stackexchange.com/questions/7671/software-for-extracting-da > ta-from-a-graph > > I have written a simple python snippet myself using pyHook, yet I was > wondering, probably there is already something already out there, which may > be much more powerful. > > Thank you! > > -Shawn E-pos vrywaringsklousule Hierdie e-pos mag vertroulike inligting bevat en mag regtens geprivilegeerd wees en is slegs bedoel vir die persoon aan wie dit geadresseer is. Indien u nie die bedoelde ontvanger is nie, word u hiermee in kennis gestel dat u hierdie dokument geensins mag gebruik, versprei of kopieer nie. Stel ook asseblief die sender onmiddellik per telefoon in kennis en vee die e-pos uit. Die Universiteit aanvaar nie aanspreeklikheid vir enige skade, verlies of uitgawe wat voortspruit uit hierdie e-pos en/of die oopmaak van enige l??s aangeheg by hierdie e-pos nie. E-mail disclaimer This e-mail may contain confidential information and may be legally privileged and is intended only for the person to whom it is addressed. If you are not the intended recipient, you are notified that you may not use, distribute or copy this document in any manner whatsoever. Kindly also notify the sender immediately by telephone, and delete the e-mail. The University does not accept liability for any damage, loss or expense arising from this e-mail and/or accessing any files attached to this e-mail. From yw5aj at virginia.edu Fri Nov 8 10:01:14 2013 From: yw5aj at virginia.edu (Yuxiang Wang) Date: Fri, 8 Nov 2013 10:01:14 -0500 Subject: [SciPy-User] Extracting data from a figure in a paper In-Reply-To: <4631948.sFSAzLLRcD@biochem433789> References: <4631948.sFSAzLLRcD@biochem433789> Message-ID: Gary and Johann, Thank you very much for both suggestions! 1) I have tried Engauge Digitizer and it worked great; I highly recommend it. It solved my problem! 2) I also tried yoink, and since I have this inexplicable love for python =D I really love it too! Hopefully it will grow more mature and become more and more popular! -Shawn On Fri, Nov 8, 2013 at 8:55 AM, Johann Rohwer wrote: > Does it have to be python? We've used Engauge Digitizer > (http://digitizer.sourceforge.net/) quite effectively for this purpose. > It's > written in C++ but fully open-source and multi platform. > > Johann > > On Friday 08 November 2013 08:18:25 Yuxiang Wang wrote: > > Dear all, > > > > Does anyone know any python alternatives to GRABIT or DataThief (both > > attached below)? > > GRABIT: > http://www.mathworks.com.au/matlabcentral/fileexchange/7173-grabit > > DataThief: http://www.datathief.org/ > > > > These two software are used to extract data points from a research paper > > figure, I guess usually happens when the author did not reply the > email... > > And I truly believe I am not the only person who needs this, from this > post: > > > http://academia.stackexchange.com/questions/7671/software-for-extracting-da > > ta-from-a-graph > > > > I have written a simple python snippet myself using pyHook, yet I was > > wondering, probably there is already something already out there, which > may > > be much more powerful. > > > > Thank you! > > > > -Shawn > > > > > E-pos vrywaringsklousule > > Hierdie e-pos mag vertroulike inligting bevat en mag regtens > geprivilegeerd wees en is slegs bedoel vir die persoon aan wie dit > geadresseer is. Indien u nie die bedoelde ontvanger is nie, word u hiermee > in kennis gestel dat u hierdie dokument geensins mag gebruik, versprei of > kopieer nie. Stel ook asseblief die sender onmiddellik per telefoon in > kennis en vee die e-pos uit. Die Universiteit aanvaar nie aanspreeklikheid > vir enige skade, verlies of uitgawe wat voortspruit uit hierdie e-pos en/of > die oopmaak van enige l?s aangeheg by hierdie e-pos nie. > > E-mail disclaimer > > This e-mail may contain confidential information and may be legally > privileged and is intended only for the person to whom it is addressed. If > you are not the intended recipient, you are notified that you may not use, > distribute or copy this document in any manner whatsoever. Kindly also > notify the sender immediately by telephone, and delete the e-mail. The > University does not accept liability for any damage, loss or expense > arising from this e-mail and/or accessing any files attached to this e-mail. > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- Yuxiang "Shawn" Wang Gerling Research Lab University of Virginia yw5aj at virginia.edu +1 (434) 284-0836 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jkhilmer at chemistry.montana.edu Fri Nov 8 11:14:09 2013 From: jkhilmer at chemistry.montana.edu (jkhilmer at chemistry.montana.edu) Date: Fri, 8 Nov 2013 09:14:09 -0700 Subject: [SciPy-User] Extracting data from a figure in a paper In-Reply-To: References: <4631948.sFSAzLLRcD@biochem433789> Message-ID: It is not written in Python, but I believe the most feature-rich software for extracting information from raster images is ImageJ and its associated plugins: http://rsbweb.nih.gov/ij/. It is basically public-domain, but not all plugins are. If you are digitizing things like scatterplots or images which have shape-encoded data, then you may be able to make CellProfiler work: http://www.cellprofiler.org/ Jonathan On Fri, Nov 8, 2013 at 8:01 AM, Yuxiang Wang wrote: > Gary and Johann, > > Thank you very much for both suggestions! > > 1) I have tried Engauge Digitizer and it worked great; I highly recommend > it. It solved my problem! > 2) I also tried yoink, and since I have this inexplicable love for python > =D I really love it too! Hopefully it will grow more mature and become more > and more popular! > > -Shawn > > > On Fri, Nov 8, 2013 at 8:55 AM, Johann Rohwer wrote: > >> Does it have to be python? We've used Engauge Digitizer >> (http://digitizer.sourceforge.net/) quite effectively for this purpose. >> It's >> written in C++ but fully open-source and multi platform. >> >> Johann >> >> On Friday 08 November 2013 08:18:25 Yuxiang Wang wrote: >> > Dear all, >> > >> > Does anyone know any python alternatives to GRABIT or DataThief (both >> > attached below)? >> > GRABIT: >> http://www.mathworks.com.au/matlabcentral/fileexchange/7173-grabit >> > DataThief: http://www.datathief.org/ >> > >> > These two software are used to extract data points from a research paper >> > figure, I guess usually happens when the author did not reply the >> email... >> > And I truly believe I am not the only person who needs this, from this >> post: >> > >> http://academia.stackexchange.com/questions/7671/software-for-extracting-da >> > ta-from-a-graph >> > >> > I have written a simple python snippet myself using pyHook, yet I was >> > wondering, probably there is already something already out there, which >> may >> > be much more powerful. >> > >> > Thank you! >> > >> > -Shawn >> >> >> >> >> E-pos vrywaringsklousule >> >> Hierdie e-pos mag vertroulike inligting bevat en mag regtens >> geprivilegeerd wees en is slegs bedoel vir die persoon aan wie dit >> geadresseer is. Indien u nie die bedoelde ontvanger is nie, word u hiermee >> in kennis gestel dat u hierdie dokument geensins mag gebruik, versprei of >> kopieer nie. Stel ook asseblief die sender onmiddellik per telefoon in >> kennis en vee die e-pos uit. Die Universiteit aanvaar nie aanspreeklikheid >> vir enige skade, verlies of uitgawe wat voortspruit uit hierdie e-pos en/of >> die oopmaak van enige l?s aangeheg by hierdie e-pos nie. >> >> E-mail disclaimer >> >> This e-mail may contain confidential information and may be legally >> privileged and is intended only for the person to whom it is addressed. If >> you are not the intended recipient, you are notified that you may not use, >> distribute or copy this document in any manner whatsoever. Kindly also >> notify the sender immediately by telephone, and delete the e-mail. The >> University does not accept liability for any damage, loss or expense >> arising from this e-mail and/or accessing any files attached to this e-mail. >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > > -- > Yuxiang "Shawn" Wang > Gerling Research Lab > University of Virginia > yw5aj at virginia.edu > +1 (434) 284-0836 > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kyle.mandli at gmail.com Fri Nov 8 11:39:12 2013 From: kyle.mandli at gmail.com (Kyle Mandli) Date: Fri, 8 Nov 2013 10:39:12 -0600 Subject: [SciPy-User] scipy 0.13 on Maverick In-Reply-To: References: Message-ID: Have you tried a different gcc version? It looks like you might be using 4.9. On Wed, Nov 6, 2013 at 5:46 PM, Doutriaux, Charles wrote: > Hello, > > I?m trying to build scipy on Maverick (Apple latest OS) > > When running > python setup.py install > > I get: > > 8463 /usr/local/bin/gfortran -Wall -Wall -undefined dynamic_lookup -bundle > build/temp.macosx-10.4-x86_64-2.7/build/src.macosx-10.4-x86_64-2.7/scipy/fftpack/_fftpac > kmodule.o build/temp.macosx-10.4-x86_64-2.7/scipy/fftpack/src/zfft.o > build/temp.macosx-10.4-x86_64-2.7/scipy/fftpack/src/drfft.o > build/temp.macosx-10.4-x86_64 -2.7/scipy/fftpack/src/zrfft.o > build/temp.macosx-10.4-x86_64-2.7/scipy/fftpack/src/zfftnd.o > build/temp.macosx-10.4-x86_64-2.7/build/src.macosx-10.4-x86_64-2.7 > /scipy/fftpack/src/dct.o > build/temp.macosx-10.4-x86_64-2.7/build/src.macosx-10.4-x86_64-2.7/scipy/fftpack/src/dst.o > build/temp.macosx-10.4-x86_64-2.7/build/sr > c.macosx-10.4-x86_64-2.7/fortranobject.o > -L/usr/local/lib/gcc/x86_64-apple-darwin13.0.0/4.9.0 > -Lbuild/temp.macosx-10.4-x86_64-2.7 -ldfftpack -lfftpack -lgfort ran -o > build/lib.macosx-10.4-x86_64-2.7/scipy/fftpack/_fftpack.so > > 8464 ld: library not found for -lbundle1.o > > 8465 collect2: error: ld returned 1 exit status > > 8466 ld: library not found for -lbundle1.o > > 8467 collect2: error: ld returned 1 exit status > > > But if I copy paste this command and run it straight from the command line, > it works. > > Any idea what?s going on? > > PS. I tried Python 2.7.4 and 2.7.5 > > C. > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From h.chr at mail.ru Sat Nov 9 23:06:24 2013 From: h.chr at mail.ru (Horea Christian) Date: Sun, 10 Nov 2013 05:06:24 +0100 Subject: [SciPy-User] overlapping samples t test Message-ID: <527F0640.2030309@mail.ru> Hey there - relating to my previous question about t-tests. I have (only very little) missing data and I was told on CrossValidated (a stack exchange for data analysis) in this thread http://stats.stackexchange.com/questions/75046/related-sample-t-test-with-missing-values that I should use an overlapping samples t-test. is this in scipy - or is there any easy way to get similar functionality? (I could push it upstream if you can tell me how to implement it) Many Thanks, -- Horea Christian http://chymera.eu -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sun Nov 10 05:07:43 2013 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sun, 10 Nov 2013 11:07:43 +0100 Subject: [SciPy-User] scipy 0.13 on Maverick In-Reply-To: References: Message-ID: On Fri, Nov 8, 2013 at 5:39 PM, Kyle Mandli wrote: > Have you tried a different gcc version? It looks like you might be using > 4.9. > > On Wed, Nov 6, 2013 at 5:46 PM, Doutriaux, Charles > wrote: > > Hello, > > > > I?m trying to build scipy on Maverick (Apple latest OS) > > > > When running > > python setup.py install > > > > I get: > > > > 8463 /usr/local/bin/gfortran -Wall -Wall -undefined dynamic_lookup > -bundle > > > build/temp.macosx-10.4-x86_64-2.7/build/src.macosx-10.4-x86_64-2.7/scipy/fftpack/_fftpac > > kmodule.o build/temp.macosx-10.4-x86_64-2.7/scipy/fftpack/src/zfft.o > > build/temp.macosx-10.4-x86_64-2.7/scipy/fftpack/src/drfft.o > > build/temp.macosx-10.4-x86_64 -2.7/scipy/fftpack/src/zrfft.o > > build/temp.macosx-10.4-x86_64-2.7/scipy/fftpack/src/zfftnd.o > > build/temp.macosx-10.4-x86_64-2.7/build/src.macosx-10.4-x86_64-2.7 > > /scipy/fftpack/src/dct.o > > > build/temp.macosx-10.4-x86_64-2.7/build/src.macosx-10.4-x86_64-2.7/scipy/fftpack/src/dst.o > > build/temp.macosx-10.4-x86_64-2.7/build/sr > > c.macosx-10.4-x86_64-2.7/fortranobject.o > > -L/usr/local/lib/gcc/x86_64-apple-darwin13.0.0/4.9.0 > > -Lbuild/temp.macosx-10.4-x86_64-2.7 -ldfftpack -lfftpack -lgfort ran > -o > > build/lib.macosx-10.4-x86_64-2.7/scipy/fftpack/_fftpack.so > > > > 8464 ld: library not found for -lbundle1.o > > > > 8465 collect2: error: ld returned 1 exit status > > > > 8466 ld: library not found for -lbundle1.o > > > > 8467 collect2: error: ld returned 1 exit status > > > > > > But if I copy paste this command and run it straight from the command > line, > > it works. > > > > Any idea what?s going on? > > > > PS. I tried Python 2.7.4 and 2.7.5 > Works for me on 10.9. There are a lot of ways to install Python and compilers on OS X, so to be able to reproduce your issue we need to know: - how you installed Python - XCode version - compiler versions - where you got gfortran from - numpy version In the traceback the 10.4 in "macosx-10.4-x86_64" could be wrong, do you have MACOSX_DEPLOYMENT_TARGET set somewhere? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sun Nov 10 11:22:28 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 10 Nov 2013 11:22:28 -0500 Subject: [SciPy-User] overlapping samples t test In-Reply-To: <527F0640.2030309@mail.ru> References: <527F0640.2030309@mail.ru> Message-ID: On Sat, Nov 9, 2013 at 11:06 PM, Horea Christian wrote: > Hey there - relating to my previous question about t-tests. I have (only > very little) missing data and I was told on CrossValidated (a stack > exchange for data analysis) in this thread > http://stats.stackexchange.com/questions/75046/related-sample-t-test-with-missing-valuesthat I should use an overlapping samples t-test. is this in scipy - or is > there any easy way to get similar functionality? (I could push it upstream > if you can tell me how to implement it) > I've never heard of overlapping samples t-test. If it mixes independent and paired t-tests, then it might require additional assumptions that might not be appropriate for a repeated measures setting, when we have to worry about unobserved differences across individuals or correlation within individuals. Josef > > Many Thanks, > > -- > Horea Christianhttp://chymera.eu > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aronne.merrelli at gmail.com Tue Nov 12 10:59:23 2013 From: aronne.merrelli at gmail.com (Aronne Merrelli) Date: Tue, 12 Nov 2013 09:59:23 -0600 Subject: [SciPy-User] Netcdf exception behavior In-Reply-To: References: Message-ID: Tony, I can easily replicate this behavior, I do think it is a bug. Here is another example and a clue for what is happening (I trimmed the exceptions down for brevity): In [7]: import scipy.io.netcdf as CDF In [8]: test = CDF.netcdf_file('foo.cdf','r') --------------------------------------------------------------------------- IOError: [Errno 2] No such file or directory: 'foo.cdf' In [9]: print test Exception AttributeError: "'netcdf_file' object has no attribute 'fp'" in > ignored --------------------------------------------------------------------------- NameError Traceback (most recent call last) in () ----> 1 print test NameError: name 'test' is not defined I think what happens is that when you run netcdf_file() with a non-existent file, you wind up with a partly constructed object because the IOError happens during the __init__() method. The partly constructed object will eventually be garbage collected, which means its __del__() method will be called. The way the netcdf_file class is written, it assumes that the self.fp attribute exists, but it does not exist because of the IOError. So, you get a second AttributeError, but at some point later, depending on when the partial object is deleted. In my example above, attempting to print "test" causes the garbage collection on the next line (also causing a NameError because "test" is not defined, but that is not part of the netcdf_file problem). In your example it is garbage collected immediately, so you get the two exceptions right in a row. A possible fix is to just test for the fp attribute first, in the __del__ method, so it won't raise the additional AttributeError. I could file a Bug report / PR if that sounds reasonable to others - I am not an expert in python internals, so what I've described here might not be the "correct" fix - maybe someone on list who is a python expert could comment =) Cheers, Aronne On Wed, Nov 6, 2013 at 12:35 PM, Mannucci, Anthony J (335G) < anthony.j.mannucci at jpl.nasa.gov> wrote: > This snippet of code does not provide the behavior I expect: > > >>> > import scipy.io.netcdf as CDF > import sys > > fitfile = 'simpleXXX.nc' > try: > fnc = CDF.netcdf_file(fitfile, 'r') > except: > sys.stderr.write("WARNING: error.\n") > else: > print fnc.variables > fnc.close() > <<< > > If fitfile exists, the read occurs as expected. > > If the fitfile does not exist, the following message is generated: > > WARNING: error. > Exception AttributeError: "'netcdf_file' object has no attribute 'fp'" in > 0x44c9290>> ignored > > I expected the exception handling code to eliminate the python exception > handler. > > Thanks for any help. > > -Tony > > -- > Tony Mannucci > Supervisor, Ionospheric and Atmospheric Remote Sensing Group > Mail-Stop 138-308, Tel > (818) 354-1699 > Jet Propulsion Laboratory, Fax > (818) 393-5115 > California Institute of Technology, Email > > Tony.Mannucci at jpl.nasa.gov > 4800 Oak Grove Drive, > http://scienceandtechnology.jpl.nasa.gov/people/a_mannucci/ > Pasadena, CA 91109 > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrea.gavana at gmail.com Wed Nov 13 08:47:14 2013 From: andrea.gavana at gmail.com (Andrea Gavana) Date: Wed, 13 Nov 2013 14:47:14 +0100 Subject: [SciPy-User] Library for constrained (?) networks Message-ID: Hi, I'm currently investigating a problem, and my Google-fu is completely failing me - possibly because I am not sure how these problems are defined in the mathematical world. If you refer to the attached picture, I have a number of wells attached to a platform, which can be in turn attached to another platform, and all the platforms end into a node called "Field". It's basically a network graph. These wells produce some fluids, and the platforms (and the Field) may have a "maximum capacity" (indicated by numbers in the picture) which may not be exceeded. What I am trying to do is to produce as much as possible from all the wells while avoiding breaking the capacity constraints of the platforms and the Field. My googling around suggested that I should look for constraint satisfaction problems libraries, but I am unsure which ones or if this is the correct approach or if there are better strategies around. Or maybe I'm just completely missing something obvious. Thank you in advance for any suggestions. Andrea. "Imagination Is The Only Weapon In The War Against Reality." http://www.infinity77.net -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: network_sample.png Type: image/png Size: 61693 bytes Desc: not available URL: From julien.brule at obspm.fr Wed Nov 13 12:38:13 2013 From: julien.brule at obspm.fr (=?ISO-8859-1?Q?julien_brul=E9?=) Date: Wed, 13 Nov 2013 18:38:13 +0100 Subject: [SciPy-User] Library for constrained (?) networks In-Reply-To: References: Message-ID: <5283B905.1090004@obspm.fr> On 13/11/2013 14:47, Andrea Gavana wrote: > Hi, > Hi > I'm currently investigating a problem, and my Google-fu is > completely failing me - possibly because I am not sure how these > problems are defined in the mathematical world. > > If you refer to the attached picture, I have a number of wells > attached to a platform, which can be in turn attached to another > platform, and all the platforms end into a node called "Field". It's > basically a network graph. These wells produce some fluids, and the > platforms (and the Field) may have a "maximum capacity" (indicated by > numbers in the picture) which may not be exceeded. > > What I am trying to do is to produce as much as possible from all the > wells while avoiding breaking the capacity constraints of the > platforms and the Field. My googling around suggested that I should > look for constraint satisfaction problems libraries, but I am unsure > which ones or if this is the correct approach or if there are better > strategies around. Or maybe I'm just completely missing something obvious. > > Thank you in advance for any suggestions. http://stackoverflow.com/questions/2517026/graph-theory-in-python ? j > > Andrea. > > "Imagination Is The Only Weapon In The War Against Reality." > http://www.infinity77.net > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrea.gavana at gmail.com Wed Nov 13 14:18:35 2013 From: andrea.gavana at gmail.com (Andrea Gavana) Date: Wed, 13 Nov 2013 20:18:35 +0100 Subject: [SciPy-User] Library for constrained (?) networks In-Reply-To: <5283B905.1090004@obspm.fr> References: <5283B905.1090004@obspm.fr> Message-ID: Hi, On Nov 13, 2013 6:38 PM, "julien brul?" wrote: > > On 13/11/2013 14:47, Andrea Gavana wrote: >> >> Hi, >> > > Hi > >> I'm currently investigating a problem, and my Google-fu is completely failing me - possibly because I am not sure how these problems are defined in the mathematical world. >> >> If you refer to the attached picture, I have a number of wells attached to a platform, which can be in turn attached to another platform, and all the platforms end into a node called "Field". It's basically a network graph. These wells produce some fluids, and the platforms (and the Field) may have a "maximum capacity" (indicated by numbers in the picture) which may not be exceeded. >> >> What I am trying to do is to produce as much as possible from all the wells while avoiding breaking the capacity constraints of the platforms and the Field. My googling around suggested that I should look for constraint satisfaction problems libraries, but I am unsure which ones or if this is the correct approach or if there are better strategies around. Or maybe I'm just completely missing something obvious. >> >> Thank you in advance for any suggestions. > > > > > http://stackoverflow.com/questions/2517026/graph-theory-in-python ? > Thank you, but that's way too generic as explanation. It doesn't really seem to apply directly to my question, or at least I'm way too ignorant to apply that explanation to the problem at hand... > j >> >> >> Andrea. >> >> "Imagination Is The Only Weapon In The War Against Reality." >> http://www.infinity77.net >> >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidmenhur at gmail.com Thu Nov 14 07:05:42 2013 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Thu, 14 Nov 2013 13:05:42 +0100 Subject: [SciPy-User] Range keyword in binned_statistics does not work Message-ID: Hi, See the following example: import numpy as np from scipy import stats x = np.arange(30) data = np.random.random(30) mean, _, _ = stats.binned_statistic(x, data) mean_range, _, _ = stats.binned_statistic(x, data, range=(3, 7)) The first call works as expected, but the second throws a TypeError. The documentation says range should be a (float, float), as it is. The error is present in versions 0.11.0 (when it was first introduced) and 0.13. (If I try anything more recent, I get a RuntimeError: maximum recursion depth exceeded at build time). --------------------------------------------------------------------------- TypeError Traceback (most recent call last) in () ----> 1 mean, _, _ = stats.binned_statistic(x, data, range=(3, 7)) /afs/ pdc.kth.se/home/d/davidmh/all/.virtualenv/local/lib/python2.7/site-packages/scipy/stats/_binned_statistic.pycin binned_statistic(x, values, statistic, bins, range) 96 97 medians, edges, xy = binned_statistic_dd([x], values, statistic, ---> 98 bins, range) 99 100 return medians, edges[0], xy /afs/ pdc.kth.se/home/d/davidmh/all/.virtualenv/local/lib/python2.7/site-packages/scipy/stats/_binned_statistic.pycin binned_statistic_dd(sample, values, statistic, bins, range) 292 smax = np.zeros(D) 293 for i in np.arange(D): --> 294 smin[i], smax[i] = range[i] 295 296 # Make sure the bins have a finite width. TypeError: 'float' object is not iterable The offending piece of code, which I don't quite understand what is it there for, is here: https://github.com/scipy/scipy/blob/master/scipy/stats/_binned_statistic.py#L287 if range is None: smin = np.atleast_1d(np.array(sample.min(0), float)) smax = np.atleast_1d(np.array(sample.max(0), float)) else: smin = np.zeros(D) smax = np.zeros(D) for i in np.arange(D): smin[i], smax[i] = range[i] And here is a test case: import numpy as np from scipy import stats x = np.arange(30) data = np.random.random(30) mean, bins, counts = stats.binned_statistic(x[:15], data[:15]) mean_range, bins_range, counts_range= stats.binned_statistic(x, data, range=(0, 15)) np.testing.assert_allclose(mean, mean_range) np.testing.assert_allclose(bins, bins_range) np.testing.assert_allclose(counts, counts_range) /David. -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Nov 14 08:18:01 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 14 Nov 2013 08:18:01 -0500 Subject: [SciPy-User] Range keyword in binned_statistics does not work In-Reply-To: References: Message-ID: On Thu, Nov 14, 2013 at 7:05 AM, Da?id wrote: > Hi, > > See the following example: > > > import numpy as np > from scipy import stats > > x = np.arange(30) > data = np.random.random(30) > > mean, _, _ = stats.binned_statistic(x, data) > mean_range, _, _ = stats.binned_statistic(x, data, range=(3, 7)) > > > The first call works as expected, but the second throws a TypeError. The > documentation says range should be a (float, float), as it is. > > The error is present in versions 0.11.0 (when it was first introduced) and > 0.13. (If I try anything more recent, I get a RuntimeError: maximum > recursion depth exceeded at build time). > > --------------------------------------------------------------------------- > TypeError Traceback (most recent call last) > in () > ----> 1 mean, _, _ = stats.binned_statistic(x, data, range=(3, 7)) > > /afs/pdc.kth.se/home/d/davidmh/all/.virtualenv/local/lib/python2.7/site-packages/scipy/stats/_binned_statistic.pyc > in binned_statistic(x, values, statistic, bins, range) > 96 > 97 medians, edges, xy = binned_statistic_dd([x], values, statistic, > ---> 98 bins, range) > 99 > 100 return medians, edges[0], xy > > /afs/pdc.kth.se/home/d/davidmh/all/.virtualenv/local/lib/python2.7/site-packages/scipy/stats/_binned_statistic.pyc > in binned_statistic_dd(sample, values, statistic, bins, range) > 292 smax = np.zeros(D) > 293 for i in np.arange(D): > --> 294 smin[i], smax[i] = range[i] > 295 > 296 # Make sure the bins have a finite width. > > TypeError: 'float' object is not iterable > > > > The offending piece of code, which I don't quite understand what is it there > for, is here: > https://github.com/scipy/scipy/blob/master/scipy/stats/_binned_statistic.py#L287 > > > > if range is None: > > smin = np.atleast_1d(np.array(sample.min(0), float)) > > smax = np.atleast_1d(np.array(sample.max(0), float)) > > else: > smin = np.zeros(D) > > smax = np.zeros(D) > > for i in np.arange(D): > > smin[i], smax[i] = range[i] > > > > > And here is a test case: > > import numpy as np > from scipy import stats > > x = np.arange(30) > data = np.random.random(30) > > mean, bins, counts = stats.binned_statistic(x[:15], data[:15]) > mean_range, bins_range, counts_range= stats.binned_statistic(x, data, > range=(0, 15)) > > np.testing.assert_allclose(mean, mean_range) > np.testing.assert_allclose(bins, bins_range) > np.testing.assert_allclose(counts, counts_range) > from the code it looks like `range` need to be a list (or iterable) of tuples, with length equal to the number of dimensions range = [(3, 7)] Josef > > > /David. > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From davidmenhur at gmail.com Thu Nov 14 09:50:01 2013 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Thu, 14 Nov 2013 15:50:01 +0100 Subject: [SciPy-User] Range keyword in binned_statistics does not work In-Reply-To: References: Message-ID: On 14 November 2013 14:18, wrote: > from the code it looks like `range` need to be a list (or iterable) of > tuples, with length equal to the number of dimensions > > range = [(3, 7)] > Then binned_statistics (the 1D version) should do that transformation for you. This may break some code, but given that no one has reported the documentation mistaken in two versions, I wonder how many people are actually using range argument. Should we just change the docstring? range=[[xmin, xmax]] looks absurdly verbose to me. /David. -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Nov 14 10:16:35 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 14 Nov 2013 10:16:35 -0500 Subject: [SciPy-User] Range keyword in binned_statistics does not work In-Reply-To: References: Message-ID: On Thu, Nov 14, 2013 at 9:50 AM, Da?id wrote: > On 14 November 2013 14:18, wrote: >> >> from the code it looks like `range` need to be a list (or iterable) of >> tuples, with length equal to the number of dimensions >> >> range = [(3, 7)] > > > Then binned_statistics (the 1D version) should do that transformation for > you. This may break some code, but given that no one has reported the > documentation mistaken in two versions, I wonder how many people are > actually using range argument. I agree, binned_statistics is based on np.histogram However, np.histogram has a 1D specific implementation, where range doesn't need to be a list of tuples. issue and PR to make binned_statistics accept just the tuple would be welcome IMO. If Ralf wants backwards compatibility, then it would be possible to add a check, so both versions, tuple or list of tuples, work. Josef > > Should we just change the docstring? range=[[xmin, xmax]] looks absurdly > verbose to me. > > > /David. > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From josef.pktd at gmail.com Thu Nov 14 10:22:29 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 14 Nov 2013 10:22:29 -0500 Subject: [SciPy-User] Range keyword in binned_statistics does not work In-Reply-To: References: Message-ID: On Thu, Nov 14, 2013 at 10:16 AM, wrote: > On Thu, Nov 14, 2013 at 9:50 AM, Da?id wrote: >> On 14 November 2013 14:18, wrote: >>> >>> from the code it looks like `range` need to be a list (or iterable) of >>> tuples, with length equal to the number of dimensions >>> >>> range = [(3, 7)] >> >> >> Then binned_statistics (the 1D version) should do that transformation for >> you. This may break some code, but given that no one has reported the >> documentation mistaken in two versions, I wonder how many people are >> actually using range argument. > > I agree, binned_statistics is based on np.histogram > However, np.histogram has a 1D specific implementation, where range > doesn't need to be a list of tuples. > > issue and PR to make binned_statistics accept just the tuple would be > welcome IMO. > > If Ralf wants backwards compatibility, then it would be possible to > add a check, so both versions, tuple or list of tuples, work. if len(range) == 2: range = [range] aside: why is `range` a variable name? it has the wrong color. Josef > > Josef > > >> >> Should we just change the docstring? range=[[xmin, xmax]] looks absurdly >> verbose to me. >> >> >> /David. >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> From ecarlson at eng.ua.edu Thu Nov 14 10:43:02 2013 From: ecarlson at eng.ua.edu (Eric Carlson) Date: Thu, 14 Nov 2013 09:43:02 -0600 Subject: [SciPy-User] Library for constrained (?) networks In-Reply-To: References: Message-ID: Hello Andrea, Your problem certainly involves, but is much more than networking. It sounds like you need to couple some reservoir simulation with surface facilities, then assess various production strategies that lead to optimal profitability (based on your definitions for profitability). Once you have this field flow model set up, I think this turns into a constrained optimization problem, for which there are many libraries - some of them with python wrapping. The objective function will have something to do with net profitability, and you will have a number of parameters that can be varied. At its simplest, based on your figure you might have an objective function F = sum( Q_i*net_value_per_unit_i ), subject to Q_4+Q_5<=300, sum_{i=4}^9 Q_i<=530.5, and sum Q_i <=1150.3 (perhaps also Q_i_min<=Q_i<=Q_i_max ) My expertise is in the reservoir engineering and simulation part of the problem, so I can't give any recommendations on the best methods for solution (although I think the system given could be handled by linear programming). I hope this can help you narrow your searches... Cheers, Eric From andrea.gavana at gmail.com Fri Nov 15 03:00:56 2013 From: andrea.gavana at gmail.com (Andrea Gavana) Date: Fri, 15 Nov 2013 09:00:56 +0100 Subject: [SciPy-User] Library for constrained (?) networks In-Reply-To: References: Message-ID: Hi Eric, Il giorno gioved? 14 novembre 2013, Eric Carlson ha scritto: > Hello Andrea, > Your problem certainly involves, but is much more than networking. It > sounds like you need to couple some reservoir simulation with surface > facilities, then assess various production strategies that lead to > optimal profitability (based on your definitions for profitability). > > Once you have this field flow model set up, I think this turns into a > constrained optimization problem, for which there are many libraries - > some of them with python wrapping. The objective function will have > something to do with net profitability, and you will have a number of > parameters that can be varied. > > At its simplest, based on your figure you might have an objective function > > F = sum( Q_i*net_value_per_unit_i ), subject to Q_4+Q_5<=300, > sum_{i=4}^9 Q_i<=530.5, and sum Q_i <=1150.3 (perhaps also > Q_i_min<=Q_i<=Q_i_max ) > > My expertise is in the reservoir engineering and simulation part of the > problem, so I can't give any recommendations on the best methods for > solution (although I think the system given could be handled by linear > programming). I'm in the same area of expertise, and I know most reservoir simulators can do this (and also pretty much all the surface network tools can). But my problem is way less complicated than coupling a reservoir simulator with a network tool. I'm just trying to see if there is any Python library (or any intelligent approach) that will allow me to solve a problem similar to the one I sketched in the attached picture. I'm basically looking at ways to (1) transform that network into a single-source single-sink graph and then solve it with any of the available graphing libraries in Python or (2) to convert it into a constrained linear programming problem and solve it with other Python libraries. The problem is, I have no idea of how to do either (1) or (2)... Andrea. > > > I hope this can help you narrow your searches... > > Cheers, > Eric > > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Andrea. "Imagination Is The Only Weapon In The War Against Reality." http://www.infinity77.net # ------------------------------------------------------------- # def ask_mailing_list_support(email): if mention_platform_and_version() and include_sample_app(): send_message(email) else: install_malware() erase_hard_drives() # ------------------------------------------------------------- # -------------- next part -------------- An HTML attachment was scrubbed... URL: From ecarlson at eng.ua.edu Fri Nov 15 17:57:09 2013 From: ecarlson at eng.ua.edu (Eric Carlson) Date: Fri, 15 Nov 2013 16:57:09 -0600 Subject: [SciPy-User] Library for constrained (?) networks In-Reply-To: References: Message-ID: Hello Andrea, So, I must say that it is not exactly clear to me what you are trying to do. That said, if it involves linear programming, it looks like pulp-or (http://code.google.com/p/pulp-or/) might be pretty easy to use. Not so easy, but a little more general is cvxopt (http://cvxopt.org/). Also not so easy but very general is OpenOpt (http://openopt.org/Welcome). Cheers, Eric From ralf.gommers at gmail.com Sun Nov 17 04:53:13 2013 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sun, 17 Nov 2013 10:53:13 +0100 Subject: [SciPy-User] ANN: Scipy 0.13.1 release Message-ID: Hi, I'm happy to announce the availability of the scipy 0.13.1 release. This is a bugfix only release; it contains several fixes for issues in ndimage. Thanks to Pauli Virtanen and Ray Jones for fixing these issues quickly. Source tarballs, binaries and release notes can be found at http://sourceforge.net/projects/scipy/files/scipy/0.13.1/. Cheers, Ralf ========================== SciPy 0.13.1 Release Notes ========================== SciPy 0.13.1 is a bug-fix release with no new features compared to 0.13.0. The only changes are several fixes in ``ndimage``, one of which was a serious regression in ``ndimage.label`` (Github issue 3025), which gave incorrect results in 0.13.0. Issues fixed ------------ - 3025: ``ndimage.label`` returns incorrect results in scipy 0.13.0 - 1992: ``ndimage.label`` return type changed from int32 to uint32 - 1992: ``ndimage.find_objects`` doesn't work with int32 input in some cases -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy at jeremysanders.net Mon Nov 18 13:26:13 2013 From: jeremy at jeremysanders.net (Jeremy Sanders) Date: Mon, 18 Nov 2013 19:26:13 +0100 Subject: [SciPy-User] ANN: Veusz 1.19 Message-ID: <528A5BC5.4090504@jeremysanders.net> I'm pleased to announce Veusz 1.19, the Python plotting package/module which adds support for Python 3. Jeremy Veusz 1.19 ---------- http://home.gna.org/veusz/ Veusz is a scientific plotting package. It is designed to produce publication-ready Postscript/PDF/SVG output. Graphs are built-up by combining plotting widgets. The user interface aims to be simple, consistent and powerful. Veusz provides GUI, Python module, command line, scripting, DBUS and SAMP interfaces to its plotting facilities. It also allows for manipulation and editing of datasets. Data can be captured from external sources such as Internet sockets or other programs. Changes in 1.19: * Make code compatible with python3 * Separation of python module and resources in source package (packagers may need to consult INSTALL) * Move to v2 of PyQt API * Remove deprecated numpy API from helpers module * Allow FITS dataset names to be blank if prefix/suffix are not * New ast-based code security checker * Picker uses 1-based index for consistency * Allow non-text datasets to be used for labels * Add number->text conversion dataset plugin * Add binning dataset plugin * Allow more significant figures in float values * Intelligent choice of significant figures when interactively changing axis range * Decrease minimum range of axis * Add notes setting to document, pages and graphs Bug fixes: * Fix crash if other linked axis not initialized * Fix crash if trying to edit non-editable datasets * Make ignore exception work in error reporting dialog * Renaming datasets, widgets and editing data now allow original text to be edited, rather than being cleared * Fix line positions on broken axes * OS X Mavericks - fixes for opening files from finder and fonts Features of package: Plotting features: * X-Y plots (with errorbars) * Line and function plots * Contour plots * Images (with colour mappings and colorbars) * Stepped plots (for histograms) * Bar graphs * Vector field plots * Box plots * Polar plots * Ternary plots * Plotting dates * Fitting functions to data * Stacked plots and arrays of plots * Nested plots * Plot keys * Plot labels * Shapes and arrows on plots * LaTeX-like formatting for text * Multiple axes * Axes with steps in axis scale (broken axes) * Axis scales using functional forms * Plotting functions of datasets Input and output: * EPS/PDF/PNG/SVG/EMF export * Dataset creation/manipulation * Embed Veusz within other programs * Text, CSV, FITS, NPY/NPZ, QDP, binary and user-plugin importing * Data can be captured from external sources Extending: * Use as a Python module * User defined functions, constants and can import external Python functions * Plugin interface to allow user to write or load code to - import data using new formats - make new datasets, optionally linked to existing datasets - arbitrarily manipulate the document * Scripting interface * Control with DBUS and SAMP Other features: * Data picker * Interactive tutorial * Multithreaded rendering Requirements for source install: Python 2.x (2.6 or greater required) or 3.x (3.3 or greater required) http://www.python.org/ Qt >= 4.4 (free edition) http://www.trolltech.com/products/qt/ PyQt >= 4.5 (SIP is required to be installed first) http://www.riverbankcomputing.co.uk/software/pyqt/ http://www.riverbankcomputing.co.uk/software/sip/ numpy >= 1.0 http://numpy.scipy.org/ Optional: PyFITS >= 1.1 (optional for FITS import) http://www.stsci.edu/resources/software_hardware/pyfits pyemf >= 2.0.0 (optional for EMF export) http://pyemf.sourceforge.net/ PyMinuit >= 1.1.2 (optional improved fitting) http://code.google.com/p/pyminuit/ For EMF and better SVG export, PyQt >= 4.6 or better is required, to fix a bug in the C++ wrapping dbus-python, for dbus interface http://dbus.freedesktop.org/doc/dbus-python/ astropy (optional for VO table import) http://www.astropy.org/ SAMPy (optional for SAMP support) http://pypi.python.org/pypi/sampy/ Veusz is Copyright (C) 2003-2013 Jeremy Sanders and contributors. It is licenced under the GPL (version 2 or greater). For documentation on using Veusz, see the "Documents" directory. The manual is in PDF, HTML and text format (generated from docbook). The examples are also useful documentation. Please also see and contribute to the Veusz wiki: http://barmag.net/veusz-wiki/ Issues with the current version: * Due to a bug in the Qt XML processing, some MathML elements containing purely white space (e.g. thin space) will give an error. If you enjoy using Veusz, we would love to hear from you. Please join the mailing lists at https://gna.org/mail/?group=veusz to discuss new features or if you'd like to contribute code. The latest code can always be found in the Git repository at https://github.com/jeremysanders/veusz.git. From andrea.gavana at gmail.com Mon Nov 18 15:59:29 2013 From: andrea.gavana at gmail.com (Andrea Gavana) Date: Mon, 18 Nov 2013 21:59:29 +0100 Subject: [SciPy-User] (Possible) new optimization routine (2) - scipy.optimize Message-ID: Hi All, since I posted the last time about (possible) new optimization routines in scipy.optimize, I have been working here and there in making the code for AMPGO (Adaptive Memory Programming for Global Optimization) a bit more robust and in expanding the benchmark test suite. Since recently I saw a few messages about optimization methods flying around in the mailing list, I thought I may share some more findings and possibly (finally) start integrating AMPGO into scipy. First things first: I would love to see AMPGO into scipy, but there are still a couple of issues to be solved: 1. Some of the local optimization methods AMPGO can use (like L-BFGS-B, TNC and so on) can take advantage of gradient information, and sometimes people do actually have access to the gradient of the objective function. The problem is, given the definition of the Tunnelling function used by AMPGO: T(x) = [f(x) - aspiration)**2.0] / prod(dist(s, x)) for s in tabu_list Where "dist" is the euclidean distance between the current point "x" and one of the previous local optima "s" ("tabu_list" is a list containing 2 or more of these local optima). (or see page 4 at http://leeds-faculty.colorado.edu/glover/fred%20pubs/416%20-%20AMP%20(TS)%20for%20Constrained%20Global%20Opt%20w%20Lasdon%20et%20al%20.pdffor a clearer formula for the Tunnelling function). I have absolutely no idea of how to get the analytical expression of the gradient of the Tunnelling function, given the gradient of the objective function. I'm sure it's very much doable but my calculus skills are way too inadequate. 2. As the current code for AMPGO supports local solvers from scipy.optimize and OpenOpt, it turns out to be a pain to generalize its interface in order for AMPGO to be compatible with the minimize() general API. Not to mention the general PR process. In any case, I'll try to gather enough willpower to get AMPGO up to scipy standards and I'll contribute the benchmarks I created as well. So, mentioning the benchmarks and the results, I managed to expand the multi-dimensional test suite from 85 to 184 test functions. The test suite is now one of the most complete and comprehensive in the Open Source world, and it's not in Fortran 77 (luckily enough). The test suite currently contains: - 18 one-dimensional test functions with multiple local/global minima; - 184 multivariate problems (where the number of independent variables ranges from 2 to 17), again with multiple local/global minima. The main page describing the rules, algorithms and motivation is here: http://infinity77.net/global_optimization/index.html A fairly in-depth summary page on AMPGO and sensitivities on its input parameters (local solver, tunnelling strategy, etc...): http://infinity77.net/global_optimization/ampgo.html Algorithms comparisons: http://infinity77.net/global_optimization/multidimensional.html http://infinity77.net/global_optimization/univariate.html Test functions and how they rank in a "difficult-to-solve" context: http://infinity77.net/global_optimization/test_functions.html The overall conclusion is that AMPGO is superior to all the other algorithms I have tried, leaving the second-best (pyasa) behind by a full 20% of number of solved problems. It's also the fastest (function-evaluation-wise), as it is able to outperform all the other algorithms' best results within 200 function evaluations or less (even though the other algorithms limit is 2,000). However, to be fair, it is an algorithm designed for low-dimensional optimization problems (i.e., 1-20 variables). If anyone has any suggestion on how to implement my point (1) above, please feel free to share your thoughts. I have no clue whatsoever. Any comment or requests to add additional benchmarks, please give me a shout. Enjoy :-) . Andrea. "Imagination Is The Only Weapon In The War Against Reality." http://www.infinity77.net # ------------------------------------------------------------- # def ask_mailing_list_support(email): if mention_platform_and_version() and include_sample_app(): send_message(email) else: install_malware() erase_hard_drives() # ------------------------------------------------------------- # -------------- next part -------------- An HTML attachment was scrubbed... URL: From gblive at gmail.com Tue Nov 19 13:38:40 2013 From: gblive at gmail.com (Mike Timonin) Date: Tue, 19 Nov 2013 18:38:40 +0000 Subject: [SciPy-User] fmin_slsqp constraint violation with 0 exit code Message-ID: Hi! I am getting rather weird results from fmin_slsqp and would be grateful for any help in debugging it. Basically, I run the following piece of code sol = fmin_slsqp(objfunc,x0, fprime=fprime, f_eqcons=f_eqcons, f_ieqcons=f_ineqcons_w, fprime_eqcons = fprime_eqcons,fprime_ieqcons = fprime_ineqcons_w, iprint=0,full_output=1,acc=1e-7,iter=600) print sol if sol[3] == 0: print f_eqcons(sol[0]),f_ineqcons_w(sol[0]) and ocassionally get results such as (array([ 1. , 27.22012122, 31.52718928, 41.00537142, 40.6788792 , -142.11913408, 4.68757296]), -1.0000000020906914, 206, 0, 'Optimization terminated successfully.') [-0.] [ -0. -25.22012122 -29.52718928 -39.00537142 -38.6788792 144.11913408 -2.68757296 27.22012122 31.52718928 41.00537142 40.6788792 -142.11913408 4.68757296] According to the documentation, f_ineqcons_w(sol[0]) output should be all non-negative. However, as can be seen, it contains negative values. How can this be investigated further? Am I hitting some known bug here? Thanks! Mikhail Timonin -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Nov 19 14:14:02 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 19 Nov 2013 14:14:02 -0500 Subject: [SciPy-User] fmin_slsqp constraint violation with 0 exit code In-Reply-To: References: Message-ID: On Tue, Nov 19, 2013 at 1:38 PM, Mike Timonin wrote: > Hi! > > I am getting rather weird results from fmin_slsqp and would be grateful for > any help in debugging it. > > Basically, I run the following piece of code > > sol = fmin_slsqp(objfunc,x0, fprime=fprime, f_eqcons=f_eqcons, > f_ieqcons=f_ineqcons_w, fprime_eqcons = fprime_eqcons,fprime_ieqcons = > fprime_ineqcons_w, iprint=0,full_output=1,acc=1e-7,iter=600) > print sol > if sol[3] == 0: > print f_eqcons(sol[0]),f_ineqcons_w(sol[0]) > > > and ocassionally get results such as > > (array([ 1. , 27.22012122, 31.52718928, 41.00537142, > 40.6788792 , -142.11913408, 4.68757296]), -1.0000000020906914, 206, 0, > 'Optimization terminated successfully.') > [-0.] [ -0. -25.22012122 -29.52718928 -39.00537142 -38.6788792 > 144.11913408 -2.68757296 27.22012122 31.52718928 41.00537142 > 40.6788792 -142.11913408 4.68757296] > > > According to the documentation, f_ineqcons_w(sol[0]) output should be all > non-negative. However, as can be seen, it contains negative values. > > How can this be investigated further? Am I hitting some known bug here? I cannot see an open issue for this, but I think it's likely a bug. I ran into something similar a while ago, but didn't investigate (it was just a quick example). In my case the violation of one inequality constraint went away when I rearranged my constraints. IIRC I think it only happens with some combination of equality and inequality constraints. The first task would be to get an example that can be used to replicate the results and open an issue on github. Then someone needs to dig into the code, if it really turns out to be a bug. Josef > > Thanks! > > Mikhail Timonin > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From bryanv at continuum.io Tue Nov 19 23:52:17 2013 From: bryanv at continuum.io (Bryan Van de Ven) Date: Tue, 19 Nov 2013 22:52:17 -0600 Subject: [SciPy-User] ANN: Bokeh 0.3 released Message-ID: I am pleased to announce the release of Bokeh 0.3! Bokeh is a Python interactive visualization library for large datasets that natively uses the latest web technologies. Its goal is to provide elegant, concise construction of novel graphics in the style of Protovis/D3, while delivering high-performance interactivity over large data to thin clients. If you are using Anaconda, you can install through conda: conda install bokeh Alternatively you can install from PyPI using pip: pip install bokeh This release was largely an internal refactor to merge the BokehJS and Bokeh projects into one repository, and to greatly improve and simplify the BokehJS coffee script build process. Additionally, this release also includes a number of bug and stability fixes, and some enhancements. See the CHANGELOG for full details. Many new examples were added including a reproduction of Burtin's Antibiotics, and examples of animation using the Bokeh plot server inside IPython notebooks. ColorBrewer palettes were also added on the python side. Finally, the user guide has been flushed out and will continually be updated as features and API changes are made. Check out the full documentation and interactive gallery at http://bokeh.pydata.org The release of Bokeh 0.4 is planned for early January. Some notable features to be included are: * Integrate Abstract Rendering into bokeh server * Better grid-based layout system; use Cassowary.js for layout solver * Tool Improvements (pan always on, box zoom always on, passive resize with hot corners) * Basic MPL compatibility interface (enough to make ggplot.py work) * Expose image plot in Python interface: Add BSON for sending large data Issues or enhancement requests can be logged on the Bokeh Github page: https://github.com/continuumio/bokeh Questions can be directed to the Bokeh mailing list: bokeh at continuum.io Regards, Bryan Van de Ven From zhou_heng at yahoo.com Wed Nov 20 06:09:56 2013 From: zhou_heng at yahoo.com (Heng Zhou) Date: Wed, 20 Nov 2013 11:09:56 +0000 (UTC) Subject: [SciPy-User] How to install scipy in virtualenv+python3+unbuntu12.04? Message-ID: I want to install scipy in virtualenv and I'm using python3 and the OS is ubuntu 12.04. Because virtualenv use pip to install third-party packages, apt-get is not applicable. I read the install webpage but couldn't understand what it is talking about. So, could you please give me a list of workable command lines that I can follow to install scipy in virtualenv+python3+unbuntu12.04? In addition, if there is any depandencies needed before installation of scipy, please also included. Thank you very much. From lists at hilboll.de Wed Nov 20 06:26:31 2013 From: lists at hilboll.de (Andreas Hilboll) Date: Wed, 20 Nov 2013 12:26:31 +0100 Subject: [SciPy-User] How to install scipy in virtualenv+python3+unbuntu12.04? In-Reply-To: References: Message-ID: <528C9C67.8020509@hilboll.de> On 20.11.2013 12:09, Heng Zhou wrote: > I want to install scipy in virtualenv and I'm using python3 and the OS is > ubuntu 12.04. Because virtualenv use pip to install third-party packages, > apt-get is not applicable. I read the install webpage but couldn't > understand what it is talking about. So, could you please give me a list of > workable command lines that I can follow to install scipy in > virtualenv+python3+unbuntu12.04? In addition, if there is any depandencies > needed before installation of scipy, please also included. Thank you very much. You can actually create a virtualenv which uses the global site-packages, so the following should work: sudo apt-get install python3-scipy virtualenv --system-site-packages path/to/my/virtualenv Otherwise, if you don't want --system-site-packages, you can just do the standard virtualenv path/to/my/virtualenv source path/to/my/virtualenv/bin/activate pip install scipy Cheers, Andreas. From evgeny.burovskiy at gmail.com Wed Nov 20 06:39:04 2013 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Wed, 20 Nov 2013 11:39:04 +0000 Subject: [SciPy-User] How to install scipy in virtualenv+python3+unbuntu12.04? In-Reply-To: References: Message-ID: An easy trick is to first install it system-wide using apt-get to satisfy non-python dependencies, then install it manually in virtualenv, run the test suite to make sure it works all right. After that apt-get remove packages installed in step one. Evgeni On Nov 20, 2013 11:21 AM, "Heng Zhou" wrote: > I want to install scipy in virtualenv and I'm using python3 and the OS is > ubuntu 12.04. Because virtualenv use pip to install third-party packages, > apt-get is not applicable. I read the install webpage but couldn't > understand what it is talking about. So, could you please give me a list of > workable command lines that I can follow to install scipy in > virtualenv+python3+unbuntu12.04? In addition, if there is any depandencies > needed before installation of scipy, please also included. Thank you very > much. > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Wed Nov 20 12:11:58 2013 From: takowl at gmail.com (Thomas Kluyver) Date: Wed, 20 Nov 2013 09:11:58 -0800 Subject: [SciPy-User] How to install scipy in virtualenv+python3+unbuntu12.04? In-Reply-To: References: Message-ID: On 20 November 2013 03:39, Evgeni Burovski wrote: > An easy trick is to first install it system-wide using apt-get to satisfy > non-python dependencies, then install it manually in virtualenv, run the > test suite to make sure it works all right. After that apt-get remove > packages installed in step one. You probably want to 'apt-get build-dep python3-scipy', rather than simply installing it. That will install the packages required to compile it, whereas installing it will only get runtime dependencies. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhou_heng at yahoo.com Wed Nov 20 17:15:16 2013 From: zhou_heng at yahoo.com (zzzhhh) Date: Wed, 20 Nov 2013 22:15:16 +0000 (UTC) Subject: [SciPy-User] =?utf-8?q?How_to_install_scipy_in=09virtualenv+pytho?= =?utf-8?q?n3+unbuntu12=2E04=3F?= References: Message-ID: Thomas Kluyver gmail.com> writes: > > > > On 20 November 2013 03:39, Evgeni Burovski gmail.com> wrote:An easy trick is to first install it system-wide using apt- get to satisfy non-python dependencies, then install it manually in virtualenv, run the test suite to make sure it works all right. After that apt-get remove packages installed in step one. > > > > > You probably want to 'apt-get build-dep python3-scipy', rather than simply installing it. That will install the packages required to compile it, whereas installing it will only get runtime dependencies. > > Thomas > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > Thank you, but, after excuting the following commands (in order) in virtualenv: sudo apt-get install python3-scipy sudo apt-get build-dep python3-scipy sudo pip install scipy , I did successfully installed scipy because I was told "Successfully installed scipy". But in python3, when I input "import scipy", an error says "ImportError: No module named scipy". Could you please tell me if I missed something? Should I apt-get remove the global package? Thanks a lot! Heng From lists at hilboll.de Wed Nov 20 17:19:03 2013 From: lists at hilboll.de (Andreas Hilboll) Date: Wed, 20 Nov 2013 23:19:03 +0100 Subject: [SciPy-User] How to install scipy in virtualenv+python3+unbuntu12.04? In-Reply-To: References: Message-ID: <528D3557.40709@hilboll.de> On 20.11.2013 23:15, zzzhhh wrote: > Thomas Kluyver gmail.com> writes: > >> >> >> >> On 20 November 2013 03:39, Evgeni Burovski > gmail.com> wrote:An easy trick is to first install it system-wide using apt- > get to satisfy non-python dependencies, then install it manually in > virtualenv, run the test suite to make sure it works all right. After that > apt-get remove packages installed in step one. >> >> >> >> >> You probably want to 'apt-get build-dep python3-scipy', rather than > simply installing it. That will install the packages required to compile > it, whereas installing it will only get runtime dependencies. >> >> Thomas >> >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > Thank you, but, after excuting the following commands (in order) in > virtualenv: > > sudo apt-get install python3-scipy > sudo apt-get build-dep python3-scipy > sudo pip install scipy > > , I did successfully installed scipy because I was told "Successfully > installed scipy". But in python3, when I input "import scipy", an error > says "ImportError: No module named scipy". Could you please tell me if I > missed something? Should I apt-get remove the global package? Thanks a lot! > > Heng > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > Don't do "sudo pip install scipy". Do "pip install scipy" within the virtualenv (i.e. after executing "source /path/to/venv/bin/activate". If you call "sudo pip install scipy", the root user executes "pip install scipy", and the root user doesn't have the virtualenv enabled. Cheers, Andreas. From pav at iki.fi Wed Nov 20 17:57:16 2013 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 20 Nov 2013 22:57:16 +0000 (UTC) Subject: [SciPy-User] =?utf-8?q?fmin=5Fslsqp_constraint_violation_with_0_e?= =?utf-8?q?xit_code?= References: Message-ID: gmail.com> writes: [clip: SLSQP constraint violation] > The first task would be to get an example that can be used to > replicate the results and open an issue on github. > Then someone needs to dig into the code, if it really turns out > to be a bug. I've seen some people also complain about SLSQP returning constraint violations. However, so far nobody has provided a reproducible test case, and therefore it has not been possible to look into this. So if someone has a program that fails on SLSQP, please step up. Scipy's wrappers for this method are quite thin, so it seems likely the bug is in the original 1994 Fortran source code. -- Pauli Virtanen From zhou_heng at yahoo.com Wed Nov 20 20:48:22 2013 From: zhou_heng at yahoo.com (zzzhhh) Date: Thu, 21 Nov 2013 01:48:22 +0000 (UTC) Subject: [SciPy-User] =?utf-8?q?How_to_install_scipy_in=09virtualenv+pytho?= =?utf-8?q?n3+unbuntu12=2E04=3F?= References: <528D3557.40709@hilboll.de> Message-ID: Andreas Hilboll hilboll.de> writes: > > On 20.11.2013 23:15, zzzhhh wrote: > > Thomas Kluyver gmail.com> writes: > > > >> > >> > >> > >> On 20 November 2013 03:39, Evgeni Burovski > > gmail.com> wrote:An easy trick is to first install it system-wide using apt- > > get to satisfy non-python dependencies, then install it manually in > > virtualenv, run the test suite to make sure it works all right. After that > > apt-get remove packages installed in step one. > >> > >> > >> > >> > >> You probably want to 'apt-get build-dep python3-scipy', rather than > > simply installing it. That will install the packages required to compile > > it, whereas installing it will only get runtime dependencies. > >> > >> Thomas > >> > >> > >> > >> _______________________________________________ > >> SciPy-User mailing list > >> SciPy-User scipy.org > >> http://mail.scipy.org/mailman/listinfo/scipy-user > >> > > > > Thank you, but, after excuting the following commands (in order) in > > virtualenv: > > > > sudo apt-get install python3-scipy > > sudo apt-get build-dep python3-scipy > > sudo pip install scipy > > > > , I did successfully installed scipy because I was told "Successfully > > installed scipy". But in python3, when I input "import scipy", an error > > says "ImportError: No module named scipy". Could you please tell me if I > > missed something? Should I apt-get remove the global package? Thanks a lot! > > > > Heng > > > > > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > Don't do "sudo pip install scipy". Do "pip install scipy" within the > virtualenv (i.e. after executing "source /path/to/venv/bin/activate". If > you call "sudo pip install scipy", the root user executes "pip install > scipy", and the root user doesn't have the virtualenv enabled. > > Cheers, Andreas. > Thank you very much! I finally made it! Thank Andreas Hilboll, thank Thomas Kluyver, thank Evgeni Burovski, thank everyone who helped me, I loved you! Here I summarize all the steps as follows: In ubuntu 12.04 server that comes only with python 2.7, to install a python3 virtual environment that doesn't conflict with python2, and scipy package that can be used by python3, run the following commands: sudo apt-get install python-pip sudo pip install virtualenv sudo apt-get install python3 virtualenv -p /usr/bin/python3 py3env . py3env/bin/activate sudo apt-get install python3-scipy sudo apt-get build-dep python3-scipy pip install numpy pip install scipy Now you are all set. Finally a small hint: to find where the python3 is, type: whereis python3. Thank you all again! Heng Zhou From cimrman3 at ntc.zcu.cz Fri Nov 22 07:50:26 2013 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Fri, 22 Nov 2013 13:50:26 +0100 Subject: [SciPy-User] ANN: SfePy 2013.4 Message-ID: <528F5312.30700@ntc.zcu.cz> I am pleased to announce release 2013.4 of SfePy. Description ----------- SfePy (simple finite elements in Python) is a software for solving systems of coupled partial differential equations by the finite element method. The code is based on NumPy and SciPy packages. It is distributed under the new BSD license. Home page: http://sfepy.org Mailing list: http://groups.google.com/group/sfepy-devel Git (source) repository, issue tracker, wiki: http://github.com/sfepy Highlights of this release -------------------------- - simplified quadrature definition - equation sequence solver - initial support for 'plate' integration/connectivity type - script for visualization of quadrature points and weights For full release notes see http://docs.sfepy.org/doc/release_notes.html#id1 (rather long and technical). Best regards, Robert Cimrman and Contributors (*) (*) Contributors to this release (alphabetical order): Vladim?r Luke?, Jaroslav Vond?ejc From ndbecker2 at gmail.com Fri Nov 22 08:51:18 2013 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 22 Nov 2013 08:51:18 -0500 Subject: [SciPy-User] wiki.scipy.org down? Message-ID: I'm getting unreachable for: http://wiki.scipy.org/Cookbook/ From robert.kern at gmail.com Fri Nov 22 10:48:01 2013 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 22 Nov 2013 15:48:01 +0000 Subject: [SciPy-User] wiki.scipy.org down? In-Reply-To: References: Message-ID: On Fri, Nov 22, 2013 at 1:51 PM, Neal Becker wrote: > I'm getting unreachable for: > > http://wiki.scipy.org/Cookbook/ Back up now. Thanks for reporting it. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzuhone at gmail.com Sat Nov 23 15:53:13 2013 From: jzuhone at gmail.com (John ZuHone) Date: Sat, 23 Nov 2013 15:53:13 -0500 Subject: [SciPy-User] Announcing yt 2.6 Message-ID: I'm forwarding this release announcement on behalf of our release manager, Kacper Kowalik. Please forward to other interested parties. ---------- Forwarded message ---------- From: Kacper Kowalik Date: Sat, Nov 23, 2013 at 12:19 AM Subject: [yt-dev] Announcing: yt 2.6 To: Discussion of the yt analysis package , "yt-dev at lists.spacepope.org" Hi all, We are pleased to announce the release of yt 2.6. This is a major release that includes new features and major updates along with all the bugs identified and fixed since the release of 2.5.5 on August 28. We currently plan for this to be the final major release of the yt 2.X release series, although bug fix releases will continue for the foreseeable future. Please forward this email to other interested parties. Most notable changes include: * Relicensing whole project to BSD 3-clause * Automated absorption line fitting module developed by Hilary Egan * Significant improvement in documentation in a joint effort of the dev team led by Cameron Hummels * Documentation now includes inlined IPython notebooks, thanks to Nathan Goldbaum * RAMSES, ART, Tiger, Maestro and Castro frontends removal as their counterparts in 3.0 branch exceeded them both performance and capability wise. All users of the aforementioned codes are strongly advised to migrate to yt-3.0 * Python stack update, which now features: ipython-1.1.0, hg-2.8, python-2.7.6 numpy-1.7.1 * New ProfilePlot and PhasePlot classes thanks to Britton Smith, Nathan Goldbaum and Matt Turk. Previous mechanisms (i.e. profiles using PlotCollection) are deprecated and will be removed in future versions. * Projections of the Sunyaev-Zeldovich effect using SZpack implemented by John ZuHone * Generating Mock X-ray Photons now possible thanks to John ZuHone * Introduction of Transfer Function Helper by Sam Skillman * Initial support for installing yt with the package management system Conda. * The addition of a frontend for the Pluto code by Andrew Myers * The addition of an OBJ exporter to enable transparent and multi-surface exports of surfaces to Blender and Sketchfab by Jiill Naiman * yt can now export to RADMC3D thanks to Andrew Myers * Considerably (10x+) faster kD-tree building for volume rendering thanks to Sam Skillman * Full integration and demonstration of all available colormaps included in documentation thanks to Cameron Hummels There have also been a number of minor bugs fixed in this release. Some highlights: * Center of the plot in off_axis_projection plots is now calculated correctly, thanks to William Gray * Several enhancements to PlotWindow made by Nathan Goldbaum: - Save accepts paths to directories - Better handling of highly rectangular domains - Axes and colorbars are now toggleable - Nicer output in ipython notebook - eps_writer is now compatible with PlotWindow, thanks to John Wise * yt command line enhancements thanks to Devin Silvia and Matt Turk * Several enhancements to Athena frontend by John ZuHone * Improvements for OSX support (up to Mavericks) * Updates to halo finding routines by Britton Smith We?d also like to recognize the new contributors to the yt 2.x source code and documentation during the yt 2.5 development cycle: * Hilary Egan * John Forbes * William Gray * Stuart Mumford * Jill Naiman * Doug Rudd * Noel Scudder * Elizabeth Tasker If you are using the stable branch of yt from an installation script, you can upgrade using "yt update" or "yt update --all" to upgrade your full dependency stack. If you are using the development branch, you may already have these fixes. A tarball of this release has been uploaded to the Python Package Index (PyPI). Documentation for this release is available at: http://yt-project.org/docs/2.6/ Previous announcements from the 2.5 release cycle can be found at: 2.5.5: http://lists.spacepope.org/pipermail/yt-users-spacepope.org/2013-August/003929.html 2.5.4: http://lists.spacepope.org/pipermail/yt-users-spacepope.org/2013-July/003716.html 2.5.3: http://lists.spacepope.org/pipermail/yt-users-spacepope.org/2013-June/003669.html 2.5.2: http://lists.spacepope.org/pipermail/yt-users-spacepope.org/2013-May/003595.html 2.5.1: http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2013-March/002988.html Thank you very much, Kacper, on behalf of the yt development team -------------- next part -------------- An HTML attachment was scrubbed... URL: From mailinglists at xgm.de Sun Nov 24 06:50:01 2013 From: mailinglists at xgm.de (Florian Lindner) Date: Sun, 24 Nov 2013 12:50:01 +0100 Subject: [SciPy-User] Cleaning/feature extraction of e-mail messages Message-ID: <1658818.VX4gHe01CO@horus> Hello, I want to use scikit-lean for mail classification (no spam detection). I haven't really worked with machine learning software (besides end-user spamfilters). What I have done so far: vectorizer = TfidfVectorizer(input='filename', preprocessor=mail_preprocessor, decode_error="ignore") X = vectorizer.fit_transform(["testmail2"]) testmail2 is raw email message (taken from a servers maildir), The decode_error I've set due to utf8 decoding issues that I decided to ignore for the time being. This works perfectly for the scikit-learn part. But one challenge (for me) seems to be to prepare the mail for feature extraction. My idea would be to take the plain/text parts of the mails, maybe additionally the From header. def mail_preprocessor(str): msg = email.message_from_string(str) msg_body = "" for part in msg.walk(): if part.get_content_type() == "text/plain": msg_body += part.get_payload(decode=True) msg_body = msg_body.lower() msg_body = msg_body.replace("\n", " ") msg_body = msg_body.replace("\t", " ") return msg_body I know that this may be slightly offtopic and I apologize if it's too offtopic. Is there already some code in the wild that prepares mail messages for feature extraction? The topic seems to be much more fancy then I had suspected, regarding issues like HTML, MIME encodings, multipart stuff, ... Thanks! Florian From mailinglists at xgm.de Sun Nov 24 06:56:10 2013 From: mailinglists at xgm.de (Florian Lindner) Date: Sun, 24 Nov 2013 12:56:10 +0100 Subject: [SciPy-User] Cleaning/feature extraction of e-mail messages In-Reply-To: <1658818.VX4gHe01CO@horus> References: <1658818.VX4gHe01CO@horus> Message-ID: <6482996.HAFUGefqFb@horus> I double apologize, this message was intended for the scikit-learn list. ;-) Am Sonntag, 24. November 2013, 12:50:01 schrieb Florian Lindner: > Hello, From chris.waigl at gmail.com Mon Nov 25 12:16:45 2013 From: chris.waigl at gmail.com (Chris Waigl) Date: Mon, 25 Nov 2013 08:16:45 -0900 Subject: [SciPy-User] Install scipy in virtualenv under OS X Snow Leopard with Python 3 Message-ID: Good morning! I am setting up my first scientific Python environment with Python 3 (3.3.3, from python.org). My production stuff is running under 2.7.5, where I installed scipy globally via the .dmg file from scipy.org. Now, I have set up a virtual environment (with --no-site-packages) and am installing packages with pip. Scipy is the first to give me trouble, though. (Numpy and matplotlib are already installed.) pip install scipy errors out while compiling _umfpack_wrap.c. The final error message is error: Command "gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -isysroot /Developer/SDKs/MacOSX10.6.sdk -arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk -g -DSCIPY_UMFPACK_H -DSCIPY_AMD_H -DNO_ATLAS_INFO=3 -I/usr/local/include -I/Users/cwaigl/.virtualenvs/science3/lib/python3.3/site-packages/numpy/core/include -I/Library/Frameworks/Python.framework/Versions/3.3/include/python3.3m -c build/src.macosx-10.6-intel-3.3/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c -o build/temp.macosx-10.6-intel-3.3/build/src.macosx-10.6-intel-3.3/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.o -msse3 -I/System/Library/Frameworks/vecLib.framework/Headers" failed with exit status 1 This happens with gcc 4.2. I also installed gcc4.6, in case, but the makefile still picks up gcc4.2. Is this a known issue? Any advice how to resolve it? (I could force it to use 4.6 if that would help.) Thanks, Chris Waigl Error log excerpt: build/src.macosx-10.6-intel-3.3/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c: In function ???PySwigPacked_repr???: build/src.macosx-10.6-intel-3.3/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c:1715: warning: return makes pointer from integer without a cast build/src.macosx-10.6-intel-3.3/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c:1717: warning: return makes pointer from integer without a cast build/src.macosx-10.6-intel-3.3/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c: In function ???PySwigPacked_str???: build/src.macosx-10.6-intel-3.3/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c:1726: warning: return makes pointer from integer without a cast build/src.macosx-10.6-intel-3.3/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c:1728: warning: return makes pointer from integer without a cast build/src.macosx-10.6-intel-3.3/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c: In function ???_PySwigPacked_type???: build/src.macosx-10.6-intel-3.3/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c:1773: warning: missing braces around initializer build/src.macosx-10.6-intel-3.3/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c:1773: warning: (near initialization for ???tmp.ob_base.ob_base???) build/src.macosx-10.6-intel-3.3/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c:1775: warning: initialization makes integer from pointer without a cast build/src.macosx-10.6-intel-3.3/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c:1778: warning: initialization from incompatible pointer type build/src.macosx-10.6-intel-3.3/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c:1779: warning: initialization from incompatible pointer type build/src.macosx-10.6-intel-3.3/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c:1780: warning: initialization from incompatible pointer type build/src.macosx-10.6-intel-3.3/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c:1782: error: ???cmpfunc??? undeclared (first use in this function) build/src.macosx-10.6-intel-3.3/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c:1782: error: expected ???}??? before ???PySwigPacked_compare??? build/src.macosx-10.6-intel-3.3/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c:1829: error: ???PyTypeObject??? has no member named ???ob_type??? build/src.macosx-10.6-intel-3.3/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c:1767: warning: unused variable ???swigpacked_doc??? From ralf.gommers at gmail.com Mon Nov 25 15:27:31 2013 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 25 Nov 2013 21:27:31 +0100 Subject: [SciPy-User] Install scipy in virtualenv under OS X Snow Leopard with Python 3 In-Reply-To: References: Message-ID: On Mon, Nov 25, 2013 at 6:16 PM, Chris Waigl wrote: > Good morning! > > I am setting up my first scientific Python environment with Python 3 > (3.3.3, from python.org). My production stuff is running under 2.7.5, > where I installed scipy globally via the .dmg file from scipy.org. Now, I > have set up a virtual environment (with --no-site-packages) and am > installing packages with pip. Scipy is the first to give me trouble, > though. (Numpy and matplotlib are already installed.) > > pip install scipy > > errors out while compiling _umfpack_wrap.c. The final error message is > > error: Command "gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG > -g -fwrapv -O3 -Wall -Wstrict-prototypes -isysroot > /Developer/SDKs/MacOSX10.6.sdk -arch i386 -arch x86_64 -isysroot > /Developer/SDKs/MacOSX10.6.sdk -g -DSCIPY_UMFPACK_H -DSCIPY_AMD_H > -DNO_ATLAS_INFO=3 -I/usr/local/include > -I/Users/cwaigl/.virtualenvs/science3/lib/python3.3/site-packages/numpy/core/include > -I/Library/Frameworks/Python.framework/Versions/3.3/include/python3.3m -c > build/src.macosx-10.6-intel-3.3/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c > -o > build/temp.macosx-10.6-intel-3.3/build/src.macosx-10.6-intel-3.3/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.o > -msse3 -I/System/Library/Frameworks/vecLib.framework/Headers" failed with > exit status 1 > > This happens with gcc 4.2. I also installed gcc4.6, in case, but the > makefile still picks up gcc4.2. > > Is this a known issue? Any advice how to resolve it? (I could force it to > use 4.6 if that would help.) > Hi, I'm not sure what the exact issue on your machine is, but ``export UMFPACK=None`` should fix (or work around) it. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From lanceboyle at qwest.net Mon Nov 25 22:17:33 2013 From: lanceboyle at qwest.net (Jerry) Date: Mon, 25 Nov 2013 20:17:33 -0700 Subject: [SciPy-User] Announcing yt 2.6 In-Reply-To: References: Message-ID: My usual grumpy reply for this kind of announcement: I am really glad and thankful that people have written the fine software yt. But could you please tell us in your announcement what yt does? Your announcement fails in this regard. Jerry On Nov 23, 2013, at 1:53 PM, John ZuHone wrote: > I'm forwarding this release announcement on behalf of our release manager, Kacper Kowalik. Please forward to other interested parties. > > ---------- Forwarded message ---------- > From: Kacper Kowalik > Date: Sat, Nov 23, 2013 at 12:19 AM > Subject: [yt-dev] Announcing: yt 2.6 > To: Discussion of the yt analysis package , "yt-dev at lists.spacepope.org" > > > Hi all, > > We are pleased to announce the release of yt 2.6. This is a major > release that includes new features and major updates along with all the > bugs identified and fixed since the release of 2.5.5 on August 28. We > currently plan for this to be the final major release of the yt 2.X > release series, although bug fix releases will continue for the > foreseeable future. > > Please forward this email to other interested parties. > > Most notable changes include: > > * Relicensing whole project to BSD 3-clause > * Automated absorption line fitting module developed by Hilary Egan > * Significant improvement in documentation in a joint effort of the dev > team led by Cameron Hummels > * Documentation now includes inlined IPython notebooks, thanks to > Nathan Goldbaum > * RAMSES, ART, Tiger, Maestro and Castro frontends removal as their > counterparts in 3.0 branch exceeded them both performance and capability > wise. All users of the aforementioned codes are strongly advised to > migrate to yt-3.0 > * Python stack update, which now features: ipython-1.1.0, hg-2.8, > python-2.7.6 numpy-1.7.1 > * New ProfilePlot and PhasePlot classes thanks to Britton Smith, Nathan > Goldbaum and Matt Turk. Previous mechanisms (i.e. profiles using > PlotCollection) are deprecated and will be removed in future versions. > * Projections of the Sunyaev-Zeldovich effect using SZpack implemented > by John ZuHone > * Generating Mock X-ray Photons now possible thanks to John ZuHone > * Introduction of Transfer Function Helper by Sam Skillman > * Initial support for installing yt with the package management system > Conda. > * The addition of a frontend for the Pluto code by Andrew Myers > * The addition of an OBJ exporter to enable transparent and > multi-surface exports of surfaces to Blender and Sketchfab by Jiill Naiman > * yt can now export to RADMC3D thanks to Andrew Myers > * Considerably (10x+) faster kD-tree building for volume rendering > thanks to Sam Skillman > * Full integration and demonstration of all available colormaps > included in documentation thanks to Cameron Hummels > > There have also been a number of minor bugs fixed in this release. Some > highlights: > > * Center of the plot in off_axis_projection plots is now calculated > correctly, thanks to William Gray > * Several enhancements to PlotWindow made by Nathan Goldbaum: > - Save accepts paths to directories > - Better handling of highly rectangular domains > - Axes and colorbars are now toggleable > - Nicer output in ipython notebook > - eps_writer is now compatible with PlotWindow, thanks to John Wise > * yt command line enhancements thanks to Devin Silvia and Matt Turk > * Several enhancements to Athena frontend by John ZuHone > * Improvements for OSX support (up to Mavericks) > * Updates to halo finding routines by Britton Smith > > We?d also like to recognize the new contributors to the yt 2.x source > code and documentation during the yt 2.5 development cycle: > > * Hilary Egan > * John Forbes > * William Gray > * Stuart Mumford > * Jill Naiman > * Doug Rudd > * Noel Scudder > * Elizabeth Tasker > > If you are using the stable branch of yt from an installation script, > you can upgrade using "yt update" or "yt update --all" to upgrade your > full dependency stack. If you are using the development branch, you > may already have these fixes. A tarball of this release has been > uploaded to the Python Package Index (PyPI). > > Documentation for this release is available at: > > http://yt-project.org/docs/2.6/ > > Previous announcements from the 2.5 release cycle can be found at: > > 2.5.5: > http://lists.spacepope.org/pipermail/yt-users-spacepope.org/2013-August/003929.html > 2.5.4: > http://lists.spacepope.org/pipermail/yt-users-spacepope.org/2013-July/003716.html > 2.5.3: > http://lists.spacepope.org/pipermail/yt-users-spacepope.org/2013-June/003669.html > 2.5.2: > http://lists.spacepope.org/pipermail/yt-users-spacepope.org/2013-May/003595.html > 2.5.1: > http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2013-March/002988.html > > Thank you very much, > > Kacper, on behalf of the yt development team > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From dineshbvadhia at hotmail.com Tue Nov 26 03:37:48 2013 From: dineshbvadhia at hotmail.com (Dinesh Vadhia) Date: Tue, 26 Nov 2013 00:37:48 -0800 Subject: [SciPy-User] Upcasting of sparse matrix Message-ID: This question was asked a few years ago but still not convinced what the right approach is. Consider a simple binary matrix-vector calculation: |1 0 1 0| |2.0| |0 0 1 0| |1.1| = Ax |0 0 0 1| |4.5| If A is defined as a dtype=int matrix then during the Ax calculation is it upcast to a float to match x? The previous answer was that it is upcast and hence, to avoid a performance penalty define A as a dtype=float matrix. However, it has been pointed out that A's dtype doesn't change, see , where 'other.data' is not cast before being passed to the wrapped c++ routines. If so, maybe the blas module performs an automatic upcast or maybe not. 'A' has been tested on large matrices with both dtype=int and dtype=float and no significant timing differences were observed between the two. The reason for raising the question again is that A with dtype=int uses less memory than with dtype=float. Would like to understand what is really going on. Thx! -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.tollerud at gmail.com Tue Nov 26 15:18:03 2013 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Tue, 26 Nov 2013 15:18:03 -0500 Subject: [SciPy-User] ANN: Astropy v0.3 released Message-ID: Dear colleagues, We are very happy to announce the second major public release (v0.3) of the Astropy package, a core Python package for Astronomy: http://www.astropy.org Astropy is a community-driven package intended to contain much of the core functionality and common tools needed for performing astronomy and astrophysics with Python. New and improved functionality in this release includes: * A new modeling package which provides a framework for fitting models to data * Quantity has been re-implemented as a numpy array subclass, enhancing performance and usability * Unit conversion and Quantities are better integrated with other parts of Astropy, simplifying many APIs and user code. * New Table functionality for joining and aggregating tables * Support for arrays of celestial coordinates * A new virtual observatory cone search package * A dedicated convolution sub-package with many predefined kernels A full list of improvements, including examples, is provided at: http://docs.astropy.org/en/latest/whatsnew/0.3.html Instructions for installing Astropy are provided at the http://www.astropy.org website, and extensive documentation can be found at: http://docs.astropy.org Please report any issues, or request new features via our GitHub repository: https://github.com/astropy/astropy/issues Over 50 developers have contributed code to Astropy so far, and you can find out more about the team behind Astropy here: http://www.astropy.org/team.html If you use Astropy directly - or as a dependency to another package - for your work, please remember to include the following acknowledgment at the end of paperst: "This research made use of Astropy, a community-developed core Python package for Astronomy (Astropy Collaboration, 2013)." where ?(Astropy Collaboration, 2013)? is the Astropy paper which was published this year: http://adsabs.harvard.edu/abs/2013A%26A...558A..33A Please feel free to forward this announcement to anyone you think might be interested in this release. We hope that you enjoy using Astropy as much as we enjoyed developing it! Erik Tollerud, Thomas Robitaille, and Perry Greenfield on behalf of The Astropy Collaboration From chris.waigl at gmail.com Wed Nov 27 01:39:22 2013 From: chris.waigl at gmail.com (Chris Waigl) Date: Tue, 26 Nov 2013 21:39:22 -0900 Subject: [SciPy-User] Install scipy in virtualenv under OS X Snow Leopard with Python 3 In-Reply-To: References: Message-ID: <86459E76-2A00-48B6-A29E-432B5D6D2250@gmail.com> On 25 Nov 2013, at 11:27, Ralf Gommers wrote: > export UMFPACK=None Sorry for the late reply -- work intervened before I could get back to the computer in question. This of course worked like a charm, thanks! I guess I can live without unsymmetric multifrontal sparse LU factorization for a little while. Should this be reported somewhere? Chris From ralf.gommers at gmail.com Wed Nov 27 01:53:34 2013 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 27 Nov 2013 07:53:34 +0100 Subject: [SciPy-User] Install scipy in virtualenv under OS X Snow Leopard with Python 3 In-Reply-To: <86459E76-2A00-48B6-A29E-432B5D6D2250@gmail.com> References: <86459E76-2A00-48B6-A29E-432B5D6D2250@gmail.com> Message-ID: On Wed, Nov 27, 2013 at 7:39 AM, Chris Waigl wrote: > > On 25 Nov 2013, at 11:27, Ralf Gommers wrote: > > > export UMFPACK=None > > Sorry for the late reply -- work intervened before I could get back to the > computer in question. This of course worked like a charm, thanks! I guess I > can live without unsymmetric multifrontal sparse LU factorization for a > little while. > > Should this be reported somewhere? > I've added it to https://github.com/scipy/scipy/issues/3002. For the next release this umfpack binding is going to be removed (will be available as separate scikit though), so that will solve the install issue for good. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From hturesson at gmail.com Wed Nov 27 17:26:27 2013 From: hturesson at gmail.com (Hjalmar Turesson) Date: Wed, 27 Nov 2013 20:26:27 -0200 Subject: [SciPy-User] Higher order Slepian (DPSS) windows? Message-ID: Hi, Is there some way to get higher order slepian/dpss windows from scipy.signal.slepian(). As far as I understand, it only returns the 0th order window/sequence. For example Nitime's nitime.algorithms.spectral.dpss_windows(), takes as a parameter Kmax, which specifies the number of DPSS windows to return (orders 0 through Kmax-1). Thanks, Hjalmar -------------- next part -------------- An HTML attachment was scrubbed... URL: From Sean.Peng at Instinet.com Wed Nov 27 17:58:40 2013 From: Sean.Peng at Instinet.com (Sean.Peng at Instinet.com) Date: Wed, 27 Nov 2013 17:58:40 -0500 Subject: [SciPy-User] user defined function for mlab.rec_groupby? Message-ID: Hi all, Is there a way to specify a self-defined function to pass into mlab.rec_groupby? So far, all I have seen is simple functions like len(), np.mean(), np.var(). Thanks, Sean ========================================================================================================= <<<< Disclaimer >>>> This message is intended solely for use by the named addressee(s). If you receive this transmission in error, please immediately notify the sender and destroy this message in its entirety, whether in electronic or hard copy format. Any unauthorized use (and reliance thereon), copying, disclosure, retention, or distribution of this transmission or the material in this transmission is forbidden. We reserve the right to monitor and archive electronic communications. This material does not constitute an offer or solicitation with respect to the purchase or sale of any security. It should not be construed to contain any recommendation regarding any security or strategy. Any views expressed are those of the individual sender, except where the message states otherwise and the sender is authorized to state them to be the views of any such entity. This communication is provided on an ?as is? basis. It contains material that is owned by Instinet Incorporated, its subsidiaries or its or their licensors, and may not, in whole or in part, be (i) copied, photocopied or duplicated in any form, by any means, or (ii) redistributed, posted, published, excerpted, or quoted without Instinet Incorporated's prior written consent. Please access the following link for important information and instructions: http://instinet.com/includes/index.jsp?thePage=/html/le_index.txt Securities products and services are provided by locally registered brokerage subsidiaries of Instinet Incorporated: Instinet Australia Pty Limited (ACN: 131 253 686 AFSL No: 327834), regulated by the Australian Securities & Investments Commission; Instinet Canada Limited, member IIROC/CIPF; Instinet Pacific Limited, authorized and regulated by the Securities and Futures Commission of Hong Kong; Instinet Singapore Services Private Limited, regulated by the Monetary Authority of Singapore, trading member of The Singapore Exchange Securities Trading Private Limited and clearing member of The Central Depository (Pte) Limited; and Instinet, LLC, member SIPC. ========================================================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From questions.anon at gmail.com Wed Nov 27 22:48:25 2013 From: questions.anon at gmail.com (questions anon) Date: Thu, 28 Nov 2013 14:48:25 +1100 Subject: [SciPy-User] sum of array for masked area only Message-ID: Hi All, I am not completely sure if this is the correct place for this question, I have a separate text file for daily rainfall data that covers the whole country. I would like to calculate the monthly mean, min, max and the mean of the sum for one state. The mean, max and min are just the mean, max and min for all data in that month however the sum data needs to work out the total for the month across the array and then sum that value. I use gdal tools to mask out the rest of the country and I use numpy tools for the summary stats. I can get the max, min and mean for the state, but the mean of the sum keeps giving me a result for the whole country rather than just the state, even though I am performing the analysis on the state only data. I am not sure if this is a masking issue or a numpy calculation issue. The mask works fine for the other summary statistics. Any feedback will be greatly appreciated! import numpy as np import matplotlib.pyplot as plt from numpy import ma as MA from mpl_toolkits.basemap import Basemap from datetime import datetime as dt from datetime import timedelta import os from StringIO import StringIO from osgeo import gdal, gdalnumeric, ogr, osr import glob import matplotlib.dates as mdates import sys shapefile=r"/Users/state.shp" ## Create masked array from shapefile xmin,ymin,xmax,ymax=[111.975,-9.975, 156.275,-44.525] ncols,nrows=[886, 691] #Your rows/cols maskvalue = 1 xres=(xmax-xmin)/float(ncols) yres=(ymax-ymin)/float(nrows) geotransform=(xmin,xres,0,ymax,0, -yres) 0 src_ds = ogr.Open(shapefile) src_lyr=src_ds.GetLayer() dst_ds = gdal.GetDriverByName('MEM').Create('',ncols, nrows, 1 ,gdal.GDT_Byte) dst_rb = dst_ds.GetRasterBand(1) dst_rb.Fill(0) #initialise raster with zeros dst_rb.SetNoDataValue(0) dst_ds.SetGeoTransform(geotransform) err = gdal.RasterizeLayer(dst_ds, [maskvalue], src_lyr) dst_ds.FlushCache() mask_arr=dst_ds.GetRasterBand(1).ReadAsArray() np.set_printoptions(threshold='nan') mask_arr[mask_arr == 255] = 1 newmask=MA.masked_equal(mask_arr,0) ### calculate monthly summary stats for state Only rainmax=[] rainmin=[] rainmean=[] rainsum=[] yearmonthlist=[] yearmonth_int=[] errors=[] OutputFolder=r"/outputfolder" GLOBTEMPLATE = r"/daily-rainfall/combined/rainfall-{year}/r{year}{month:02}??.txt" def accumulate_month(year, month): files = glob.glob(GLOBTEMPLATE.format(year=year, month=month)) monthlyrain=[] for ifile in files: try: f=np.genfromtxt(ifile,skip_header=6) except: print "ERROR with file:", ifile errors.append(ifile) f=np.flipud(f) stateonly_f=np.ma.masked_array(f, mask=newmask.mask) # this masks data to state print "stateonly_f:", stateonly_f.max(), stateonly_f.mean(), stateonly_f.sum() monthlyrain.append(stateonly_f) yearmonth=dt(year,month,1) yearmonthlist.append(yearmonth) yearmonthint=str(year)+str(month) d=dt.strptime(yearmonthint, '%Y%m') print d date_string=dt.strftime(d,'%Y%m') yearmonthint=int(date_string) yearmonth_int.append(yearmonthint) r_sum=np.sum(monthlyrain, axis=0) r_mean_of_sum=MA.mean(r_sum) r_max, r_mean, r_min=MA.max(monthlyrain), MA.mean(monthlyrain), MA.min(monthlyrain) rainmax.append(r_max) rainmean.append(r_mean) rainmin.append(r_min) rainsum.append(r_mean_of_sum) print " state only:", yearmonthint,r_max, r_mean, r_min, r_mean_of_sum -------------- next part -------------- An HTML attachment was scrubbed... URL: From guziy.sasha at gmail.com Thu Nov 28 10:26:23 2013 From: guziy.sasha at gmail.com (Oleksandr Huziy) Date: Thu, 28 Nov 2013 10:26:23 -0500 Subject: [SciPy-User] sum of array for masked area only In-Reply-To: References: Message-ID: Hi what is your scipy version? Does changing np.sum to MA.sum fix the problem? This is the only reason I can why it might not work. You are calculating your group statistics in time and space simultaneously, is this really what you need?... Cheers 2013/11/27 questions anon > Hi All, > I am not completely sure if this is the correct place for this question, > > I have a separate text file for daily rainfall data that covers the whole > country. I would like to calculate the monthly mean, min, max and the mean > of the sum for one state. > The mean, max and min are just the mean, max and min for all data in that > month however the sum data needs to work out the total for the month across > the array and then sum that value. > > I use gdal tools to mask out the rest of the country and I use numpy tools > for the summary stats. > > I can get the max, min and mean for the state, but the mean of the sum > keeps giving me a result for the whole country rather than just the state, > even though I am performing the analysis on the state only data. I am not > sure if this is a masking issue or a numpy calculation issue. The mask > works fine for the other summary statistics. > > Any feedback will be greatly appreciated! > > > import numpy as np > import matplotlib.pyplot as plt > from numpy import ma as MA > from mpl_toolkits.basemap import Basemap > from datetime import datetime as dt > from datetime import timedelta > import os > from StringIO import StringIO > from osgeo import gdal, gdalnumeric, ogr, osr > import glob > import matplotlib.dates as mdates > import sys > > shapefile=r"/Users/state.shp" > > ## Create masked array from shapefile > xmin,ymin,xmax,ymax=[111.975,-9.975, 156.275,-44.525] > ncols,nrows=[886, 691] #Your rows/cols > maskvalue = 1 > > xres=(xmax-xmin)/float(ncols) > yres=(ymax-ymin)/float(nrows) > geotransform=(xmin,xres,0,ymax,0, -yres) > 0 > src_ds = ogr.Open(shapefile) > src_lyr=src_ds.GetLayer() > > dst_ds = gdal.GetDriverByName('MEM').Create('',ncols, nrows, 1 > ,gdal.GDT_Byte) > dst_rb = dst_ds.GetRasterBand(1) > dst_rb.Fill(0) #initialise raster with zeros > dst_rb.SetNoDataValue(0) > dst_ds.SetGeoTransform(geotransform) > > err = gdal.RasterizeLayer(dst_ds, [maskvalue], src_lyr) > dst_ds.FlushCache() > > mask_arr=dst_ds.GetRasterBand(1).ReadAsArray() > np.set_printoptions(threshold='nan') > mask_arr[mask_arr == 255] = 1 > > newmask=MA.masked_equal(mask_arr,0) > > > ### calculate monthly summary stats for state Only > > rainmax=[] > rainmin=[] > rainmean=[] > rainsum=[] > > yearmonthlist=[] > yearmonth_int=[] > errors=[] > > OutputFolder=r"/outputfolder" > GLOBTEMPLATE = > r"/daily-rainfall/combined/rainfall-{year}/r{year}{month:02}??.txt" > > def accumulate_month(year, month): > files = glob.glob(GLOBTEMPLATE.format(year=year, month=month)) > monthlyrain=[] > for ifile in files: > try: > f=np.genfromtxt(ifile,skip_header=6) > except: > print "ERROR with file:", ifile > errors.append(ifile) > f=np.flipud(f) > > stateonly_f=np.ma.masked_array(f, mask=newmask.mask) # this masks > data to state > > > print "stateonly_f:", stateonly_f.max(), stateonly_f.mean(), > stateonly_f.sum() > > monthlyrain.append(stateonly_f) > > yearmonth=dt(year,month,1) > yearmonthlist.append(yearmonth) > yearmonthint=str(year)+str(month) > d=dt.strptime(yearmonthint, '%Y%m') > print d > date_string=dt.strftime(d,'%Y%m') > yearmonthint=int(date_string) > yearmonth_int.append(yearmonthint) > > r_sum=np.sum(monthlyrain, axis=0) > r_mean_of_sum=MA.mean(r_sum) > > r_max, r_mean, r_min=MA.max(monthlyrain), MA.mean(monthlyrain), > MA.min(monthlyrain) > rainmax.append(r_max) > rainmean.append(r_mean) > rainmin.append(r_min) > rainsum.append(r_mean_of_sum) > > > > print " state only:", yearmonthint,r_max, r_mean, r_min, r_mean_of_sum > > > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- Sasha -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevinkunzmann at gmx.net Fri Nov 29 04:29:43 2013 From: kevinkunzmann at gmx.net (Kevin Kunzmann) Date: Fri, 29 Nov 2013 10:29:43 +0100 Subject: [SciPy-User] scipy.optimize.minimize constraint optimization Message-ID: <52985E87.9030902@gmx.net> Hi, I have a question concerning constraint optimization with scipy. After setting up my problem using scipy.optimize.minimize and the SLSQP method the procedure failed to converge and the final iteration was a bit awkward. So I inspected the iteration steps closely and the problem seems to be, that the bounds / constraints are not respected in every step. This blows up some function evaluations, as they are only well defined for the feasible set. Is there a way of doing general purpose, nonlinear, constraint optimization, that respects the feasible set at ALL times in scipy (or at least in python?). The Matlab implementations of fmincon using sqp and trust-region-reflective do implement that feature http://www.mathworks.de/de/help/optim/ug/writing-constraints.html#br9p_ry but i would rather stick with python if possible... If that feature is available already in scipy an comment on that in the help would be superb :) thank you, Kevin From davidmenhur at gmail.com Fri Nov 29 08:54:01 2013 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Fri, 29 Nov 2013 14:54:01 +0100 Subject: [SciPy-User] scipy.optimize.minimize constraint optimization In-Reply-To: <52985E87.9030902@gmx.net> References: <52985E87.9030902@gmx.net> Message-ID: On 29 November 2013 10:29, Kevin Kunzmann wrote: > So I inspected the iteration steps closely and the problem > seems to be, that the bounds / constraints are not respected in every > step. > That is a heisenbug that has been heard of, but it would be great if you could provide a working (failing) example to investigate it further. If you want to be sure, you could do what software like Minuit does, apply a transformation of your input variables, so that the problem becomes unbounded. If you variable y must lie between a and b, the unbounded variable ? is defined as: ?= arcsin(2((y -a)/(b- a))-1) All the details: http://root.cern.ch/root/html/TMinuit.html Regards /David. -------------- next part -------------- An HTML attachment was scrubbed... URL: From djpine at gmail.com Fri Nov 29 10:10:34 2013 From: djpine at gmail.com (David Pine) Date: Fri, 29 Nov 2013 10:10:34 -0500 Subject: [SciPy-User] adding linear fitting routine Message-ID: <582DB7CD-EB73-4B07-BA69-2AEDC927D081@gmail.com> I have written a function called linfit for linear least square fitting that I am proposing to have added to one of the numpy or scipy libraries. linfit performs a full least squares chi-squared fit. Thus, it can handle data with error estimates (aka error bars), weighting the data accordingly. linfit provides estimates of the uncertainties of the fitted parameters, the slope and y-intercept. linfit allows one to optionally: (1) use no weighting, or (2) to weight data according the residuals, often called relative weighting (the way it's often done in work in the social sciences), or (3) to use the absolute measure of uncertainties either for each data point or for all the data points at once (the way it's often done in the physical sciences). These options were included with the recent discussion on weighted least squares fitting in mind. See scipy/scipy#448. I am not sure where linfit best belongs in the numpy/scipy universe. The most reasonable places would seem to be either the polynomial package (a straight line is the simplest polynomial) or perhaps the scipy.optimize package along with curve_fit, which fits nonlinear functions to data. I wrote the function because there really is nothing like it in numpy or scipy, and it is so basic that in my opinion, something like it should be available. I tried to write it so that it is useful to a very wide range of users that cross all branches of social and physical sciences as well as engineering. I have added linfit to a cloned version of the numpy.polynomial module, which can be found at https://github.com/numpy/numpy/pull/4080. The standalone linfit function can be found at https://github.com/djpine/linfit. An ipython notebook demonstrating various ways of using linfit is available at the same site; its output can be viewed at http://nbviewer.ipython.org/github/djpine/linfit/blob/master/linfit.ipynb. I have in included a unit test test_linfit.py that, among other things, compares the speed of linfit to numpy.polyfit, scipy.linalg.lstsq, and scipy.stats.linregress. linfit is faster for all cases I tested, typically by several times. Finally, I am new to using Github and to contributing to numpy/scipy, so I am not sure if I have submitted everything properly, but I hope this gets the process going. David Pine From newville at cars.uchicago.edu Fri Nov 29 14:13:53 2013 From: newville at cars.uchicago.edu (Matt Newville) Date: Fri, 29 Nov 2013 13:13:53 -0600 Subject: [SciPy-User] scipy.optimize.minimize constraint optimization In-Reply-To: <52985E87.9030902@gmx.net> References: <52985E87.9030902@gmx.net> Message-ID: Kevin, On Fri, Nov 29, 2013 at 3:29 AM, Kevin Kunzmann wrote: > Hi, > > I have a question concerning constraint optimization with scipy. After > setting up my problem using scipy.optimize.minimize and the SLSQP method > the procedure failed to converge and the final iteration was a bit > awkward. So I inspected the iteration steps closely and the problem > seems to be, that the bounds / constraints are not respected in every > step. This blows up some function evaluations, as they are only well > defined for the feasible set. Is there a way of doing general purpose, > nonlinear, constraint optimization, that respects the feasible set at > ALL times in scipy (or at least in python?). The Matlab implementations > of fmincon using sqp and trust-region-reflective do implement that > feature > http://www.mathworks.de/de/help/optim/ug/writing-constraints.html#br9p_ry but > i would rather stick with python if possible... > If that feature is available already in scipy an comment on that in the > help would be superb :) > > thank you, > > Kevin > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > You might find the lmfit module (http://newville.github.io/lmfit-py/) useful. It takes a slightly different approach, wrapping many of the scipy optimization routines (with .leastsq() and .fmin() being the best tested) to be written as a function of Parameter objects instead of floating point numbers. By using a richer object, user-defined bounds can be set for Parameter (each Parameter has a .min and .max attribute, and the Minuit re-casting as David suggested, and as implemented in Python by JJ Helmus, is used). In addition, you can fix (ie, not vary) a Parameter used by the objective function without having to rewrite the objective function. You can also set constraints, including inequality constraints, by writing an expression to be evaluated for a Parameters value. While the .leastsq() function provides an automatically generated estimate of uncertainties and correlations between Parameters, the lmfit module also provides convenient functions to do a brute force exploration of confidence intervals. Finally, the module also makes it easy (no recoding of objective functions) to switch fitting algorithms. Some people use the Nelder-Mead algorithm (fmin) to find the best solution, the running leastsq starting with that solution to get estimates of the uncertainties. Hope that helps, --Matt Newville From questions.anon at gmail.com Fri Nov 29 17:10:15 2013 From: questions.anon at gmail.com (questions anon) Date: Sat, 30 Nov 2013 09:10:15 +1100 Subject: [SciPy-User] sum of array for masked area only In-Reply-To: References: Message-ID: Thanks for responding. I received very helpful instruction on the numpy forum (posted below) thanks At this point monthlyrain is a list of masked arrays > r_sum=np.sum(monthlyrain, axis=0) ^^^^^^^^^^^ Passing a list of masked arrays to np.sum returns an np.ndarray object (*not* a masked array) > r_mean_of_sum=MA.mean(r_sum) Therefore this call to MA.mean returns the mean of all values in the ndarray r_sum. To fix: convert your monthlyrain list to a 3D maksed array before calling np.sum(monthlyrain, axis=0). In this case np.sum will call the masked array's .sum() method which knows about the mask. monthlyrain = np.ma.asarray(monthlyrain) r_sum=np.sum(monthlyrain, axis=0) Consider the following simplified example: alist = [] for k in range(2): a = np.arange(4).reshape((2,2)) alist.append(np.ma.masked_ array(a, mask=[[0,1],[0,0]])) print(alist) print(type(alist)) alist = np.ma.asarray(alist) print(alist) print(type(alist)) asum = np.sum(alist, axis=0) print(asum) print(type(asum)) print(asum.mean()) Cheers, Scott On Fri, Nov 29, 2013 at 2:26 AM, Oleksandr Huziy wrote: > Hi > > what is your scipy version? Does changing np.sum to MA.sum fix the problem? > This is the only reason I can why it might not work. > > You are calculating your group statistics in time and space > simultaneously, is this really what you need?... > Cheers > > > 2013/11/27 questions anon > >> Hi All, >> I am not completely sure if this is the correct place for this question, >> >> I have a separate text file for daily rainfall data that covers the whole >> country. I would like to calculate the monthly mean, min, max and the mean >> of the sum for one state. >> The mean, max and min are just the mean, max and min for all data in that >> month however the sum data needs to work out the total for the month across >> the array and then sum that value. >> >> I use gdal tools to mask out the rest of the country and I use numpy >> tools for the summary stats. >> >> I can get the max, min and mean for the state, but the mean of the sum >> keeps giving me a result for the whole country rather than just the state, >> even though I am performing the analysis on the state only data. I am not >> sure if this is a masking issue or a numpy calculation issue. The mask >> works fine for the other summary statistics. >> >> Any feedback will be greatly appreciated! >> >> >> import numpy as np >> import matplotlib.pyplot as plt >> from numpy import ma as MA >> from mpl_toolkits.basemap import Basemap >> from datetime import datetime as dt >> from datetime import timedelta >> import os >> from StringIO import StringIO >> from osgeo import gdal, gdalnumeric, ogr, osr >> import glob >> import matplotlib.dates as mdates >> import sys >> >> shapefile=r"/Users/state.shp" >> >> ## Create masked array from shapefile >> xmin,ymin,xmax,ymax=[111.975,-9.975, 156.275,-44.525] >> ncols,nrows=[886, 691] #Your rows/cols >> maskvalue = 1 >> >> xres=(xmax-xmin)/float(ncols) >> yres=(ymax-ymin)/float(nrows) >> geotransform=(xmin,xres,0,ymax,0, -yres) >> 0 >> src_ds = ogr.Open(shapefile) >> src_lyr=src_ds.GetLayer() >> >> dst_ds = gdal.GetDriverByName('MEM').Create('',ncols, nrows, 1 >> ,gdal.GDT_Byte) >> dst_rb = dst_ds.GetRasterBand(1) >> dst_rb.Fill(0) #initialise raster with zeros >> dst_rb.SetNoDataValue(0) >> dst_ds.SetGeoTransform(geotransform) >> >> err = gdal.RasterizeLayer(dst_ds, [maskvalue], src_lyr) >> dst_ds.FlushCache() >> >> mask_arr=dst_ds.GetRasterBand(1).ReadAsArray() >> np.set_printoptions(threshold='nan') >> mask_arr[mask_arr == 255] = 1 >> >> newmask=MA.masked_equal(mask_arr,0) >> >> >> ### calculate monthly summary stats for state Only >> >> rainmax=[] >> rainmin=[] >> rainmean=[] >> rainsum=[] >> >> yearmonthlist=[] >> yearmonth_int=[] >> errors=[] >> >> OutputFolder=r"/outputfolder" >> GLOBTEMPLATE = >> r"/daily-rainfall/combined/rainfall-{year}/r{year}{month:02}??.txt" >> >> def accumulate_month(year, month): >> files = glob.glob(GLOBTEMPLATE.format(year=year, month=month)) >> monthlyrain=[] >> for ifile in files: >> try: >> f=np.genfromtxt(ifile,skip_header=6) >> except: >> print "ERROR with file:", ifile >> errors.append(ifile) >> f=np.flipud(f) >> >> stateonly_f=np.ma.masked_array(f, mask=newmask.mask) # this masks >> data to state >> >> >> print "stateonly_f:", stateonly_f.max(), stateonly_f.mean(), >> stateonly_f.sum() >> >> monthlyrain.append(stateonly_f) >> >> yearmonth=dt(year,month,1) >> yearmonthlist.append(yearmonth) >> yearmonthint=str(year)+str(month) >> d=dt.strptime(yearmonthint, '%Y%m') >> print d >> date_string=dt.strftime(d,'%Y%m') >> yearmonthint=int(date_string) >> yearmonth_int.append(yearmonthint) >> >> r_sum=np.sum(monthlyrain, axis=0) >> r_mean_of_sum=MA.mean(r_sum) >> >> r_max, r_mean, r_min=MA.max(monthlyrain), MA.mean(monthlyrain), >> MA.min(monthlyrain) >> rainmax.append(r_max) >> rainmean.append(r_mean) >> rainmin.append(r_min) >> rainsum.append(r_mean_of_sum) >> >> >> >> print " state only:", yearmonthint,r_max, r_mean, r_min, r_mean_of_sum >> >> >> >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > > -- > Sasha > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.l.goldsmith at gmail.com Sat Nov 30 20:35:56 2013 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Sat, 30 Nov 2013 17:35:56 -0800 Subject: [SciPy-User] "Genetic Algorithm" method support in Python/SciPy Message-ID: Hi, folks. Does SciPy have a sub-package for so-called Genetic Algorithmwork? If not in SciPy, does anyone know of a Python package for this? Thanks! DG -------------- next part -------------- An HTML attachment was scrubbed... URL: From joseluismietta at yahoo.com.ar Fri Nov 1 11:13:23 2013 From: joseluismietta at yahoo.com.ar (=?iso-8859-1?Q?Jos=E8_Luis_Mietta?=) Date: Fri, 01 Nov 2013 15:13:23 -0000 Subject: [SciPy-User] log normal distribution random number array generation Message-ID: <1383319029.73853.YahooMailNeo@web142305.mail.bf1.yahoo.com> Hi experts! I wanna generate a random number array of size=N using a log normal distribution. From http://en.wikipedia.org/wiki/Log-normal_distribution i wanna use the parameters mu and sigma. I know that I must do: form scipy.stats import lognorm new_array = lognorm.rvs(......, size=N) What must I set like parameters (loc, s, scale, etc.) for use mu and sigma distribution parameters. In the same way: what must I do in?? new_array = norm.rvs(......, size=N) for generate a array of random numbers using a gaussian distribution with parameters mu and sigma? Waitign for your answers. Thanks a lot! -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Sun Nov 3 11:37:08 2013 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Sun, 03 Nov 2013 16:37:08 -0000 Subject: [SciPy-User] ANN: NumPy 1.7.2rc1 release Message-ID: <52767CF1.60003@googlemail.com> Hi all, I'm happy to announce the release candidate of Numpy 1.7.2. This is a bugfix only release supporting Python 2.4 - 2.7 and 3.1 - 3.3. More than 37 issues were fixed, the most important issues are listed in the release notes: https://github.com/numpy/numpy/blob/v1.7.2rc1/doc/release/1.7.2-notes.rst It is supposed to not break any existing code, so please test the releases and report any issues you find. Source tarballs and release notes can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.7.2rc1/. Currently only Windows installers are available. OS X installer will follow soon. Concerning OS X currently only a single person can create the binary installers which is not a good situation. If you have a suitable machine [0] and want to help out please contact us. Cheers, Julian Taylor [0] we currently base the releases on macos 10.6 using python.org python versions -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From bravo.loic at gmail.com Mon Nov 4 03:15:13 2013 From: bravo.loic at gmail.com (LB) Date: Mon, 04 Nov 2013 08:15:13 -0000 Subject: [SciPy-User] How to efficiently modify a sparse matrix ? Message-ID: -------------- next part -------------- An HTML attachment was scrubbed... URL: From bravo.loic at gmail.com Mon Nov 4 03:44:24 2013 From: bravo.loic at gmail.com (LB) Date: Mon, 04 Nov 2013 08:44:24 -0000 Subject: [SciPy-User] How to efficiently modify a sparse matrix ? In-Reply-To: References: Message-ID: Hi, Sorry for the empty mail below. I'm using scipy to solve a 2D thermal analysis. This analysis is made in three steps: - I define a stiffness matrix using scipy.sparse.lil_matrix - Then I add some boundary conditions by updating some rows of this matrix - Finally, I translate this matrix to CSR format and solve the system with scipy.sparse.linalg.spsolve This works. You could find attached a working example. But it seems a bit slow on real applications. When I looked at the time spent in each step, I've been surprised to see that the second step takes most of the computation time in these cases. The code lines which seems to be the bottleneck is : i_CL = np.flatnonzero(CL) n_CL = i_CL.size K_CL = sparse.coo_matrix((np.ones(n_CL), (np.arange(n_CL), i_CL)), shape=(n_CL, n)).tolil() K[i_CL, :] = K_CL This code define an array i_CL defining the lines of the matrix K to be modified. These lines should match those of an identity matrix, in order to define fixed boundary conditions. I'm not very familiar with sparse matrix, so I'm not sure it is the correct way to go. Have you got any hint to improve this code ? Is there a better approach to solve this kind of sparse problem ? Regards, Loic -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: conduction.py Type: application/octet-stream Size: 2061 bytes Desc: not available URL: From cyrille.rossant at gmail.com Mon Nov 4 10:43:29 2013 From: cyrille.rossant at gmail.com (Cyrille Rossant) Date: Mon, 04 Nov 2013 15:43:29 -0000 Subject: [SciPy-User] [ANN] Vispy 0.2 for OpenGL-based data visualization Message-ID: Dear all, I'm pleased to announce that we've just released Vispy 0.2 (http://vispy.org) ! Vispy is a hardware-accelerated interactive visualization library in Python that brings OpenGL to the masses. In this release we improved the OpenGL object-oriented interface. The next step for us will be to design higher-level interfaces. Those will let users not necessarily familiar with OpenGL design beautiful and fast visualizations in Python (including data visualization and scientific plotting). Please report bugs and feature requests at our issue tracker: https://github.com/vispy/vispy/issues All the best, Cyrille PS: Vispy will be at the Budapest BI Forum this week! http://budapestbiforum.com From horea.christ at gmail.com Wed Nov 6 08:44:26 2013 From: horea.christ at gmail.com (Horea Christian) Date: Wed, 06 Nov 2013 13:44:26 -0000 Subject: [SciPy-User] Repeated Measure ANOVA Message-ID: Hi guys, I would like to compare reaction times for a series of experimental conditions. My data comes from ~100 trial repetitions over 10 participants (yielding ~1000 trials). I was told that just doing an ANOVA on this dataset would be improper, because the 1000 measurements are not truly independent - and that the proper way to do this is called a repeated measure ANOVA. I have tried to look for a scipy function for this and found nothing. In this relevant discussiona participant pointed the following out: "Repeated measures" ANOVA is just a misnomer for using the "randomized > block design" as a substitute for not knowing MANOVA or Hotelling's T- > square test, and as such leads to conclusions that are very hard to > interpret. The real value of repeated measures ANOVA in medical > litterature is often to inform the reader that the authors don't > understand the statistics they use ;-) I would like to know whether I'm looking for the right thing at all, and if yes how I could accomplish this with scipy -------------- next part -------------- An HTML attachment was scrubbed... URL: From ggventurini at gmail.com Thu Nov 7 12:26:20 2013 From: ggventurini at gmail.com (Giuseppe Venturini) Date: Thu, 07 Nov 2013 17:26:20 -0000 Subject: [SciPy-User] zpk2ss results are different in SciPy and Matlab Message-ID: Hi all, I am usually able to run all my code in Numpy/SciPy instead of MATLAB but I seem to have met an obstacle I can't get past. I am trying to convert a zpk representation to ss consistently the results from MATLAB. For example take the following code: from numpy import * import scipy.signal z = array([ 0.745-0.088j, 0.745+0.088j, 0.850-0.252j, 0.850+0.252j]) p = array([ 0.997-0.083j, 0.997+0.083j, 1.000+0.j , 1.000+0.j ]) z.sort() p.sort() A, B, C, D = scipy.signal.zpk2ss(z, p, [-1]) for i in A, B, C, D: print i Which results in the ABCD matrices: [[ 3.994 -5.988898 3.995796 -1.000898] [ 1. 0. 0. 0. ] [ 0. 1. 0. 0. ] [ 0. 0. 1. 0. ]] [[ 1.] [ 0.] [ 0.] [ 0.]] [[-0.804 2.107125 -1.86794274 0.55855931]] [-1.] while in MATLAB I get the results loaded by this excerpt: MA = array([[1.993119459099803, -1.000000000000000, 0, 0], [1.000000000000000, 0, 0, 0], [0.502449960669436, -0.436680054705503, 2.000000000000000, -1], [0., 0., 1.000000000000000, 0.]]) MB2 = np.array([[1],[0],[1],[0]]) MC = np.array([-0.502449960669436, 0.436680054705503, -0.299569824826267, 0.213738831336429]) MD2 = np.array([[-1]]) for i in MA, MB2, MC, MD2: print i Which prints: [[ 1.99311946 -1. 0. 0. ] [ 1. 0. 0. 0. ] [ 0.50244996 -0.43668005 2. -1. ] [ 0. 0. 1. 0. ]] [[1] [0] [1] [0]] [-0.50244996 0.43668005 -0.29956982 0.21373883] [[-1]] The two seem to be two valid representations of the same LTI system, in fact, if I convert back the MATLAB matrices to poles and zeros with: Mz, Mp, Mk = scipy.signal.ss2zpk(MA, MB2, MC, MD2) Mz.sort(), Mp.sort() print "Do zeros match? (rtol=1e-3):", np.allclose(Mz, z, rtol=1e-3) print "Do poles match? (rtol=1e-3):", np.allclose(Mp, p, rtol=1e-3) The roots match: Do zeros match? (rtol=1e-3): True Do poles match? (rtol=1e-3): True Can somebody tell whether this is a bug? Further, is it possible to have a consistent behavior in SciPy and MATLAB? Thank you! GV From almar.klein at gmail.com Fri Nov 8 06:07:10 2013 From: almar.klein at gmail.com (Almar Klein) Date: Fri, 08 Nov 2013 11:07:10 -0000 Subject: [SciPy-User] ANN: Pyzo 2013c - now based on conda! Message-ID: Hi all, We are pleased to announce release version 2013c of the Pyzo scientific Python distribution. A very significant change is that Pyzo is now build on conda. This significantly simplifies our build process, since we can profit from the binary packages build by Continuum. We created a channel on binstar ( binstar.org/pyzo) where we upload a few missing or improved packages. For example, we have packages for PySide on python3, as well as scikit-learn. Package management is no much easier for Pyzo users. You can install new packages from the IDE's shell using e.g. "conda install numba" or "pip install vispy". Further, we now have a 64bit version for Windows, and the OSX version is now also 64bit. Try it out at http://pyzo.og ! Happy coding, Almar -------------- next part -------------- An HTML attachment was scrubbed... URL: From almar.klein at gmail.com Fri Nov 8 06:08:43 2013 From: almar.klein at gmail.com (Almar Klein) Date: Fri, 08 Nov 2013 11:08:43 -0000 Subject: [SciPy-User] ANN: Pyzo 2013c - now based on conda! In-Reply-To: References: Message-ID: * that should have been http://pyzo.org On 8 November 2013 12:12, Almar Klein wrote: > Hi all, > > We are pleased to announce release version 2013c of the Pyzo scientific > Python distribution. > > A very significant change is that Pyzo is now build on conda. This > significantly simplifies our build process, since we can profit from the > binary packages build by Continuum. We created a channel on binstar ( > binstar.org/pyzo) where we upload a few missing or improved packages. For > example, we have packages for PySide on python3, as well as scikit-learn. > > Package management is no much easier for Pyzo users. You can install new > packages from the IDE's shell using e.g. "conda install numba" or "pip > install vispy". > > Further, we now have a 64bit version for Windows, and the OSX version is > now also 64bit. > > Try it out at http://pyzo.og ! > > Happy coding, > Almar > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From boris.burle at univ-amu.fr Fri Nov 8 07:56:44 2013 From: boris.burle at univ-amu.fr (=?ISO-8859-1?Q?Bor=EDs_BURLE?=) Date: Fri, 08 Nov 2013 12:56:44 -0000 Subject: [SciPy-User] ttest_rel with unequal groups In-Reply-To: <527B2299.60706@mail.ru> References: <527B2299.60706@mail.ru> Message-ID: <527CE0D8.5020001@univ-amu.fr> Dear Christian, On 07/11/2013 06:18, Horea Christian wrote: > Hey there! I would like to use the ttest_rel function to compare > reaction times for two conditions tested over 10 participants. We have > done 100 trials per participant, but some of them had errors and were > excluded. For instance for prticipants 1 and 2 I have condition1: 95 > trials, condition2: 100 trials AND condition1:100 trials and condition2: > 99 trials. If you're dealing with reaction time (RT) task, what 99% of people would do is to compute the mean RT per condition, and then perform a within subject t-test. inthis case, the number of mean per condition will necessarily be the same. Although I have not looked at the code in details, given the description of ttest_rel, I guess it will compute the difference scores between "corresponding" values, and test whether the distribution of the differences differ from 0. If you do that on raw data (single trials), you compute differences between unrelated trials, which does not make any sense: trial 1 of condition 1 has nothing to do with trial 1 of condition 2 (at least not more than with trial 2 or 3 etc... of condition 2). Hope it helps, B. > > depending on whether or not I transpose my dataframe I get a complaint > either at > > if a.shape[axis] != b.shape[axis]: > raise ValueError('unequal length arrays') > > or at > > d = (a - b).astype(np.float64) > > . > > > What can I do about this? I found it surprising that it doesn't "just > work" since in most experiments it is expected for some of the > measurements to fail. > > Many Thanks! > Christian > -- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% ATTENTION NOUVELLES COORDONNEES / WARNING NEW CONTACT DETAILS %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Boris BURLE Laboratoire de Neurosciences Cognitives P?le 3C, Aix-Marseille Universit?, CNRS tel: (+33) 4 13 55 09 40 fax: (+33) 4 13 55 09 58 web page: http://sites.univ-provence.fr/lnc/ From almar.klein at gmail.com Tue Nov 12 07:38:30 2013 From: almar.klein at gmail.com (Almar Klein) Date: Tue, 12 Nov 2013 12:38:30 -0000 Subject: [SciPy-User] ANN: IEP 3.3.2 - the Interactive Editor for Python Message-ID: We just released 3.3.2 of the Interactive Editor for Python. This is mostly a maintenance release. We fixed some problems with the binaries and IEP can now run using PyQt4 again. See https://bitbucket.org/iep-project/iep/wiki/Release_notes for a complete list. People using Pyzo can update IEP by running "conda update pyzolib iep". Regards, Almar -------------- next part -------------- An HTML attachment was scrubbed... URL: From geovis at xtra.co.nz Wed Nov 13 10:47:54 2013 From: geovis at xtra.co.nz (Brennan Williams) Date: Wed, 13 Nov 2013 15:47:54 -0000 Subject: [SciPy-User] Library for constrained (?) networks In-Reply-To: References: Message-ID: <1384358004.954.YahooMailNeo@web140206.mail.bf1.yahoo.com> constrained linear programming??http://openopt.org?perhaps? ________________________________ From: Andrea Gavana To: SciPy Users List Sent: Wednesday, 13 November 2013 11:47 AM Subject: [SciPy-User] Library for constrained (?) networks Hi, ? ? I'm currently investigating a problem, and my Google-fu is completely failing me - possibly because I am not sure how these problems are defined in the mathematical world. If you refer to the attached picture, I have a number of wells attached to a platform, which can be in turn attached to another platform, and all the platforms end into a node called "Field". It's basically a network graph. These wells produce some fluids, and the platforms (and the Field) may have a "maximum capacity" (indicated by numbers in the picture) which may not be exceeded. What I am trying to do is to produce as much as possible from all the wells while avoiding breaking the capacity constraints of the platforms and the Field. My googling around suggested that I should look for constraint satisfaction problems libraries, but I am unsure which ones or if this is the correct approach or if there are better strategies around. Or maybe I'm just completely missing something obvious. Thank you in advance for any suggestions. Andrea. "Imagination Is The Only Weapon In The War Against Reality." http://www.infinity77.net _______________________________________________ SciPy-User mailing list SciPy-User at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at dkjones.org Wed Nov 13 15:52:29 2013 From: dave at dkjones.org (David Jones) Date: Wed, 13 Nov 2013 20:52:29 -0000 Subject: [SciPy-User] Library for constrained (?) networks In-Reply-To: References: <5283B905.1090004@obspm.fr> Message-ID: Hi Andrea, It looks like you have a "max flow" problem: https://en.wikipedia.org/wiki/Maximum_flow_problem I only glanced at the Stack Overflow page, but I believe that NetworkX has algorithm(s) for this class of problems. Dave On Wed, Nov 13, 2013 at 2:18 PM, Andrea Gavana wrote: > Hi, > > On Nov 13, 2013 6:38 PM, "julien brul?" wrote: > > > > On 13/11/2013 14:47, Andrea Gavana wrote: > >> > >> Hi, > >> > > > > Hi > > > >> I'm currently investigating a problem, and my Google-fu is > completely failing me - possibly because I am not sure how these problems > are defined in the mathematical world. > >> > >> If you refer to the attached picture, I have a number of wells attached > to a platform, which can be in turn attached to another platform, and all > the platforms end into a node called "Field". It's basically a network > graph. These wells produce some fluids, and the platforms (and the Field) > may have a "maximum capacity" (indicated by numbers in the picture) which > may not be exceeded. > >> > >> What I am trying to do is to produce as much as possible from all the > wells while avoiding breaking the capacity constraints of the platforms and > the Field. My googling around suggested that I should look for constraint > satisfaction problems libraries, but I am unsure which ones or if this is > the correct approach or if there are better strategies around. Or maybe I'm > just completely missing something obvious. > >> > >> Thank you in advance for any suggestions. > > > > > > > > > > http://stackoverflow.com/questions/2517026/graph-theory-in-python ? > > > > Thank you, but that's way too generic as explanation. It doesn't really > seem to apply directly to my question, or at least I'm way too ignorant to > apply that explanation to the problem at hand... > > > j > >> > >> > >> Andrea. > >> > >> "Imagination Is The Only Weapon In The War Against Reality." > >> http://www.infinity77.net > >> > >> > >> > >> _______________________________________________ > >> SciPy-User mailing list > >> SciPy-User at scipy.org > >> http://mail.scipy.org/mailman/listinfo/scipy-user > > > > > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From almugabo at googlemail.com Thu Nov 14 01:17:36 2013 From: almugabo at googlemail.com (almugabo at googlemail.com) Date: Thu, 14 Nov 2013 06:17:36 -0000 Subject: [SciPy-User] scipy.cluster : linkagematrix to tree Message-ID: <3c47df58-2066-4fdd-ac79-637c5cdfa800@googlegroups.com> (I posted this to stackoverflow but a friend thought that this is best place to look for help for scipy specific issue) I am struggling to transform results of a hierarchical clusterring in a json format Here my code : import pandas as pd import scipy.spatialimport scipy.cluster d = {'employee' : ['A', 'B', 'C', 'D', 'E', 'F'], 'skillX': [2,8,3,6,8,10], 'skillY': [8,15,6,9,7,10]} d1 = pd.DataFrame(d) distMat = xPairWiseDist = scipy.spatial.distance.pdist(np.array(d1[['skillX', 'skillY']]), 'euclidean') clusters = scipy.cluster.hierarchy.linkage(distMat, method='single') dendo = scipy.cluster.hierarchy.dendrogram(clusters, labels = list(d1.employee), orientation = 'right') dendo my question : how to transform the linkage matrix in json format that d3.js understands {'name': 'Root1?, 'children':[{'name' : 'B'}, {'name': 'E-D-F-C-A', 'children' : [{'name': 'C-A', 'children' : {'name': 'A'}, {'name' : 'C'}] } } ]} I understand this is what the totree method is for but could not find any example which will help me to understand it (despite the excellent documentation). I get so far a = scipy.cluster.hierarchy.to_tree(clusters , rd=True)for x in a[1]: #print x.get_id() if x.is_leaf() != True : print x.get_left().get_id(), x.get_right().get_id(), x.get_count() -------------- next part -------------- An HTML attachment was scrubbed... URL: From klaus.rheinberger at gmail.com Sun Nov 17 06:07:20 2013 From: klaus.rheinberger at gmail.com (Klaus) Date: Sun, 17 Nov 2013 11:07:20 -0000 Subject: [SciPy-User] Matplotlib 1.3.1: plot(matrix("1, 2, 3")) -> RuntimeError: maximum recursion depth exceeded Message-ID: Hi, I am working with python 2.7.5 using - numpy.__version__: 1.7.1 - - matplotlib.__version__: 1.3.1 When I start "ipython2 --pylab" and execute the following code x = matrix("1,2,3") > plot(x) I get the error message [...] > /usr/lib/python2.7/site-packages/matplotlib/units.pyc in > get_converter(self, x) > 146 except AttributeError: > 147 # not a masked_array > --> 148 converter = self.get_converter(xravel[0]) > 149 return converter > 150 > /usr/lib/python2.7/site-packages/numpy/matrixlib/defmatrix.py in > __getitem__(self, index) > 303 > 304 try: > --> 305 out = N.ndarray.__getitem__(self, index) > 306 finally: > 307 self._getitem = False > RuntimeError: maximum recursion depth exceeded In the older matplotlib version 1.3.0 this error was not present. Any help is highly appreciated! -------------- next part -------------- An HTML attachment was scrubbed... URL: From howarth at bromo.med.uc.edu Sun Nov 17 10:39:16 2013 From: howarth at bromo.med.uc.edu (Jack Howarth) Date: Sun, 17 Nov 2013 15:39:16 -0000 Subject: [SciPy-User] [SciPy-Dev] ANN: Scipy 0.13.1 release In-Reply-To: References: Message-ID: <20131117154450.GA29454@bromo.med.uc.edu> On Sun, Nov 17, 2013 at 10:53:13AM +0100, Ralf Gommers wrote: > Hi, > > I'm happy to announce the availability of the scipy 0.13.1 release. This is > a bugfix only release; it contains several fixes for issues in ndimage. > Thanks to Pauli Virtanen and Ray Jones for fixing these issues quickly. > > Source tarballs, binaries and release notes can be found at > http://sourceforge.net/projects/scipy/files/scipy/0.13.1/. > > Cheers, > Ralf Ralf, Unfortunately, the scipy 0.13.1 release has introduced two new failures on the x86_64-apple-darwin11 target... FAIL: test_cases (test_solvers.TestSolveLyapunov) ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/src/fink.build/root-scipy-py26-0.13.1-1/sw/lib/python2.6/site-packages/scipy/linalg/tests/test_solvers.py", line 45, in test_cases self.check_continuous_case(case[0], case[1]) File "/sw/src/fink.build/root-scipy-py26-0.13.1-1/sw/lib/python2.6/site-packages/scipy/linalg/tests/test_solvers.py", line 37, in check_continuous_case assert_array_almost_equal(np.dot(a, x) + np.dot(x, a.conj().transpose()), q) File "/sw/lib/python2.6/site-packages/numpy/testing/utils.py", line 811, in assert_array_almost_equal header=('Arrays are not almost equal to %d decimals' % decimal)) File "/sw/lib/python2.6/site-packages/numpy/testing/utils.py", line 644, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not almost equal to 6 decimals (mismatch 100.0%) x: array([[ 0.66940063, 1.82351325, 0.09693803, 0.74958987, 5.37149648], [ 2.35737286, -1.68687652, -1.11483151, 2.92536995, 5.39661969], [ 0.08560546, -1.49569258, 2.3794113 , 0.51512191, 6.00411424],... y: array([[2, 4, 1, 0, 1], [4, 1, 0, 2, 0], [1, 0, 3, 0, 3],... ====================================================================== FAIL: test_cases (test_solvers.TestSolveSylvester) ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/src/fink.build/root-scipy-py26-0.13.1-1/sw/lib/python2.6/site-packages/scipy/linalg/tests/test_solvers.py", line 179, in test_cases self.check_case(case[0], case[1], case[2]) File "/sw/src/fink.build/root-scipy-py26-0.13.1-1/sw/lib/python2.6/site-packages/scipy/linalg/tests/test_solvers.py", line 175, in check_case assert_array_almost_equal(np.dot(a, x) + np.dot(x, b), c) File "/sw/lib/python2.6/site-packages/numpy/testing/utils.py", line 811, in assert_array_almost_equal header=('Arrays are not almost equal to %d decimals' % decimal)) File "/sw/lib/python2.6/site-packages/numpy/testing/utils.py", line 644, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not almost equal to 6 decimals (mismatch 25.0%) x: array([[ 1.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00], [ 0.00000000e+00, 1.00000000e+00, 0.00000000e+00,... y: array([[ 1., 0., 0., 0.], [ 0., 1., 0., 0.], [ 0., 0., 1., 0.], [ 0., 0., 0., 1.]]) ---------------------------------------------------------------------- Ran 8933 tests in 186.410s FAILED (KNOWNFAIL=114, SKIP=210, failures=2) whereas both x86_64-apple-darwin12 and x86_64-apple-darwin13, there are no failures. OK (KNOWNFAIL=114, SKIP=203) What darwin releases was this update tested on? Jack > > > > ========================== > SciPy 0.13.1 Release Notes > ========================== > > SciPy 0.13.1 is a bug-fix release with no new features compared to 0.13.0. > The only changes are several fixes in ``ndimage``, one of which was a > serious > regression in ``ndimage.label`` (Github issue 3025), which gave > incorrect results in 0.13.0. > > Issues fixed > ------------ > > - 3025: ``ndimage.label`` returns incorrect results in scipy 0.13.0 > - 1992: ``ndimage.label`` return type changed from int32 to uint32 > - 1992: ``ndimage.find_objects`` doesn't work with int32 input in some cases > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From jeremy at jeremysanders.net Sun Nov 17 14:10:18 2013 From: jeremy at jeremysanders.net (Jeremy Sanders) Date: Sun, 17 Nov 2013 19:10:18 -0000 Subject: [SciPy-User] ANN: Veusz 1.19 Message-ID: <528915E4.3020802@jeremysanders.net> I'm pleased to announce the release of Veusz 1.19. This version now supports joint Python 2 and 3 compatibility. Jeremy Veusz 1.19 ---------- http://home.gna.org/veusz/ Veusz is a scientific plotting package. It is designed to produce publication-ready Postscript/PDF/SVG output. Graphs are built-up by combining plotting widgets. The user interface aims to be simple, consistent and powerful. Veusz provides GUI, Python module, command line, scripting, DBUS and SAMP interfaces to its plotting facilities. It also allows for manipulation and editing of datasets. Data can be captured from external sources such as Internet sockets or other programs. Changes in 1.19: * Make code compatible with python3 * Separation of python module and resources in source package (packagers may need to consult INSTALL) * Move to v2 of PyQt API * Remove deprecated numpy API from helpers module * Allow FITS dataset names to be blank if prefix/suffix are not * New ast-based code security checker * Picker uses 1-based index for consistency * Allow non-text datasets to be used for labels * Add number->text conversion dataset plugin * Add binning dataset plugin * Add notes properties to document, pages and graphs * Allow more significant figures in float values * Intelligent choice of significant figures when interactively changing axis range * Decrease minimum range of axis * Add notes setting to document, pages and graphs Bug fixes: * Fix crash if other linked axis not initialized * Fix crash if trying to edit non-editable datasets * Make ignore exception work in error reporting dialog * Renaming datasets, widgets and editing data now allow original text to be edited, rather than being cleared * Fix line positions on broken axes * OS X Mavericks - fixes for opening files from finder and fonts Features of package: Plotting features: * X-Y plots (with errorbars) * Line and function plots * Contour plots * Images (with colour mappings and colorbars) * Stepped plots (for histograms) * Bar graphs * Vector field plots * Box plots * Polar plots * Ternary plots * Plotting dates * Fitting functions to data * Stacked plots and arrays of plots * Nested plots * Plot keys * Plot labels * Shapes and arrows on plots * LaTeX-like formatting for text * Multiple axes * Axes with steps in axis scale (broken axes) * Axis scales using functional forms * Plotting functions of datasets Input and output: * EPS/PDF/PNG/SVG/EMF export * Dataset creation/manipulation * Embed Veusz within other programs * Text, CSV, FITS, NPY/NPZ, QDP, binary and user-plugin importing * Data can be captured from external sources Extending: * Use as a Python module * User defined functions, constants and can import external Python functions * Plugin interface to allow user to write or load code to - import data using new formats - make new datasets, optionally linked to existing datasets - arbitrarily manipulate the document * Scripting interface * Control with DBUS and SAMP Other features: * Data picker * Interactive tutorial * Multithreaded rendering Requirements for source install: Python 2.x (2.6 or greater required) or 3.x (3.3 or greater required) http://www.python.org/ Qt >= 4.4 (free edition) http://www.trolltech.com/products/qt/ PyQt >= 4.5 (SIP is required to be installed first) http://www.riverbankcomputing.co.uk/software/pyqt/ http://www.riverbankcomputing.co.uk/software/sip/ numpy >= 1.0 http://numpy.scipy.org/ Optional: PyFITS >= 1.1 (optional for FITS import) http://www.stsci.edu/resources/software_hardware/pyfits pyemf >= 2.0.0 (optional for EMF export) http://pyemf.sourceforge.net/ PyMinuit >= 1.1.2 (optional improved fitting) http://code.google.com/p/pyminuit/ For EMF and better SVG export, PyQt >= 4.6 or better is required, to fix a bug in the C++ wrapping dbus-python, for dbus interface http://dbus.freedesktop.org/doc/dbus-python/ astropy (optional for VO table import) http://www.astropy.org/ SAMPy (optional for SAMP support) http://pypi.python.org/pypi/sampy/ Veusz is Copyright (C) 2003-2013 Jeremy Sanders and contributors. It is licenced under the GPL (version 2 or greater). For documentation on using Veusz, see the "Documents" directory. The manual is in PDF, HTML and text format (generated from docbook). The examples are also useful documentation. Please also see and contribute to the Veusz wiki: http://barmag.net/veusz-wiki/ Issues with the current version: * Due to a bug in the Qt XML processing, some MathML elements containing purely white space (e.g. thin space) will give an error. If you enjoy using Veusz, we would love to hear from you. Please join the mailing lists at https://gna.org/mail/?group=veusz to discuss new features or if you'd like to contribute code. The latest code can always be found in the Git repository at https://github.com/jeremysanders/veusz.git. From chris.barker at noaa.gov Mon Nov 18 13:31:58 2013 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 18 Nov 2013 10:31:58 -0800 Subject: [SciPy-User] [Numpy-discussion] ANN: Scipy 0.13.1 release In-Reply-To: References: Message-ID: On Sun, Nov 17, 2013 at 1:53 AM, Ralf Gommers wrote: > I'm happy to announce the availability of the scipy 0.13.1 release. This > is a bugfix only release; it contains several fixes for issues in ndimage. > Thanks to Pauli Virtanen and Ray Jones for fixing these issues quickly. > > Thanks Ralf et al. for more great work! Source tarballs, binaries and release notes can be found at > http://sourceforge.net/projects/scipy/files/scipy/0.13.1/. > > What's the plan for OS-X binaries? It would be great to have those up sooner than later...Can I help? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Nov 18 13:45:32 2013 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 18 Nov 2013 10:45:32 -0800 Subject: [SciPy-User] [Numpy-discussion] ANN: Scipy 0.13.1 release In-Reply-To: References: Message-ID: On Mon, Nov 18, 2013 at 10:31 AM, Chris Barker wrote: > What's the plan for OS-X binaries? It would be great to have those up > sooner than later...Can I help? > > oops sorry there they are! Were they just uploaded -- I really did look before posting that! Thanks all! -Chris > -Chris > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: