From kkappler at sjgeophysics.com Thu Nov 1 20:09:19 2012 From: kkappler at sjgeophysics.com (Karl Kappler) Date: Thu, 1 Nov 2012 17:09:19 -0700 Subject: [SciPy-User] permuted output arguments in online tutorial for scipy.signal.freqz Message-ID: Hi, Not sure I am sending this to the right place, just wanted to let you know that the output arguments w and h in the worked example on the webpage: http://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.freqz.html are flipped. i.e. the line: h, w = signal.freqz(b) should read w, h = signal.freqz(b) Otherwise the plot is not a function, i.e. its many-to-one. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pengyu.ut at gmail.com Thu Nov 1 20:28:47 2012 From: pengyu.ut at gmail.com (Peng Yu) Date: Thu, 1 Nov 2012 19:28:47 -0500 Subject: [SciPy-User] ks_2samp is not giving the same results as ks.test in R Message-ID: Hi, The ks_2samp function does not give the same answer as ks.test in R. Does anybody know why they are different? Is ks_2samp compute something different? helium:~/linux/test/python/man/library/scipy/stats/ks_2samp$ Rscript main.R > ks.test(1:5, 11:15) Two-sample Kolmogorov-Smirnov test data: 1:5 and 11:15 D = 1, p-value = 0.007937 alternative hypothesis: two-sided > ks.test(1:5, 11:15, alternative='less') Two-sample Kolmogorov-Smirnov test data: 1:5 and 11:15 D^- = 0, p-value = 1 alternative hypothesis: the CDF of x lies below that of y > ks.test(1:5, 11:15, alternative='greater') Two-sample Kolmogorov-Smirnov test data: 1:5 and 11:15 D^+ = 1, p-value = 0.006738 alternative hypothesis: the CDF of x lies above that of y > > helium:~/linux/test/python/man/library/scipy/stats/ks_2samp$ ./main.py (1.0, 0.0037813540593701006) helium:~/linux/test/python/man/library/scipy/stats/ks_2samp$ cat main.py #!/usr/bin/env python from scipy.stats import ks_2samp print ks_2samp([1,2,3,4,5], [11,12,13,14,15]) -- Regards, Peng From josef.pktd at gmail.com Thu Nov 1 21:14:25 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 1 Nov 2012 21:14:25 -0400 Subject: [SciPy-User] ks_2samp is not giving the same results as ks.test in R In-Reply-To: References: Message-ID: On Thu, Nov 1, 2012 at 8:28 PM, Peng Yu wrote: > Hi, > > The ks_2samp function does not give the same answer as ks.test in R. > Does anybody know why they are different? Is ks_2samp compute > something different? > > helium:~/linux/test/python/man/library/scipy/stats/ks_2samp$ Rscript main.R >> ks.test(1:5, 11:15) > > Two-sample Kolmogorov-Smirnov test > > data: 1:5 and 11:15 > D = 1, p-value = 0.007937 > alternative hypothesis: two-sided > >> ks.test(1:5, 11:15, alternative='less') > > Two-sample Kolmogorov-Smirnov test > > data: 1:5 and 11:15 > D^- = 0, p-value = 1 > alternative hypothesis: the CDF of x lies below that of y > >> ks.test(1:5, 11:15, alternative='greater') > > Two-sample Kolmogorov-Smirnov test > > data: 1:5 and 11:15 > D^+ = 1, p-value = 0.006738 > alternative hypothesis: the CDF of x lies above that of y > >> >> > helium:~/linux/test/python/man/library/scipy/stats/ks_2samp$ ./main.py > (1.0, 0.0037813540593701006) > helium:~/linux/test/python/man/library/scipy/stats/ks_2samp$ cat main.py > #!/usr/bin/env python > > from scipy.stats import ks_2samp > print ks_2samp([1,2,3,4,5], [11,12,13,14,15]) R uses by default an "exact" distribution for small samples if there are no ties. If there are ties or with a large sample, R uses the asymptotic distribution. If I read the function correctly, then scipy.stats is using a small sample approximation by Stephens. (But I would have to look up the formula to verify this.) In the example below with a bit larger sample and no ties, our approximation is closer to R's "exact" pvalue than the asymptotic distribution if exact=FALSE. > ks.test(1:25, (10:30)-0.5, exact=FALSE) Two-sample Kolmogorov-Smirnov test data: 1:25 and (10:30) - 0.5 D = 0.36, p-value = 0.1038 alternative hypothesis: two-sided > ks.test(1:25, (10:30)-0.5, exact=TRUE) Two-sample Kolmogorov-Smirnov test data: 1:25 and (10:30) - 0.5 D = 0.36, p-value = 0.07608 alternative hypothesis: two-sided >>> stats.ks_2samp(np.arange(1.,26), np.arange(10,31.)-0.5) (0.35999999999999999, 0.078993426961291274) For the 1 sample kstest I used (when I rewrote stats.kstest) an approximation that is closer to the exact distribution than the asymptotic distribution, but it's also not exact. It would be good to have better small sample approximations or exact distributions, but I worked on this in scipy.stats when I barely had any idea about goodness-of-fit tests. Also, ks_2samp never got the enhancement for one-sided alternatives. (In statsmodels I have been working so far only on one sample tests, but not on two-sample tests.) (I don't remember if there is a minimum size recommendation, but the examples I usually checked were larger.) since it's a community project: Pull Request are welcome Josef > > > -- > Regards, > Peng > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From josef.pktd at gmail.com Thu Nov 1 21:41:11 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 1 Nov 2012 21:41:11 -0400 Subject: [SciPy-User] ks_2samp is not giving the same results as ks.test in R In-Reply-To: References: Message-ID: On Thu, Nov 1, 2012 at 9:14 PM, wrote: > On Thu, Nov 1, 2012 at 8:28 PM, Peng Yu wrote: >> Hi, >> >> The ks_2samp function does not give the same answer as ks.test in R. >> Does anybody know why they are different? Is ks_2samp compute >> something different? >> >> helium:~/linux/test/python/man/library/scipy/stats/ks_2samp$ Rscript main.R >>> ks.test(1:5, 11:15) >> >> Two-sample Kolmogorov-Smirnov test >> >> data: 1:5 and 11:15 >> D = 1, p-value = 0.007937 >> alternative hypothesis: two-sided >> >>> ks.test(1:5, 11:15, alternative='less') >> >> Two-sample Kolmogorov-Smirnov test >> >> data: 1:5 and 11:15 >> D^- = 0, p-value = 1 >> alternative hypothesis: the CDF of x lies below that of y >> >>> ks.test(1:5, 11:15, alternative='greater') >> >> Two-sample Kolmogorov-Smirnov test >> >> data: 1:5 and 11:15 >> D^+ = 1, p-value = 0.006738 >> alternative hypothesis: the CDF of x lies above that of y >> >>> >>> >> helium:~/linux/test/python/man/library/scipy/stats/ks_2samp$ ./main.py >> (1.0, 0.0037813540593701006) >> helium:~/linux/test/python/man/library/scipy/stats/ks_2samp$ cat main.py >> #!/usr/bin/env python >> >> from scipy.stats import ks_2samp >> print ks_2samp([1,2,3,4,5], [11,12,13,14,15]) > > R uses by default an "exact" distribution for small samples if there > are no ties. > If there are ties or with a large sample, R uses the asymptotic distribution. > > If I read the function correctly, then scipy.stats is using a small > sample approximation by Stephens. (But I would have to look up the > formula to verify this.) http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test#Two-sample_Kolmogorov.E2.80.93Smirnov_test has the weighted sample size: en = np.sqrt(n1*n2/float(n1+n2)) the small sample weighting ((en+0.12+0.11/en)*d) is the same as in Stephens (1970, 1985?) for the one sample test. I don't have a reference for the two sample approximation right now. (another bit of random information) tables are often only available for 0.01 to 0.25 and approximations are targeted on that range and might not be as accurate outside of it Josef > > In the example below with a bit larger sample and no ties, our > approximation is closer to R's "exact" pvalue than the asymptotic > distribution if exact=FALSE. > >> ks.test(1:25, (10:30)-0.5, exact=FALSE) > > Two-sample Kolmogorov-Smirnov test > > data: 1:25 and (10:30) - 0.5 > D = 0.36, p-value = 0.1038 > alternative hypothesis: two-sided > >> ks.test(1:25, (10:30)-0.5, exact=TRUE) > > Two-sample Kolmogorov-Smirnov test > > data: 1:25 and (10:30) - 0.5 > D = 0.36, p-value = 0.07608 > alternative hypothesis: two-sided > > >>>> stats.ks_2samp(np.arange(1.,26), np.arange(10,31.)-0.5) > (0.35999999999999999, 0.078993426961291274) > > > For the 1 sample kstest I used (when I rewrote stats.kstest) an > approximation that is closer to the exact distribution than the > asymptotic distribution, but it's also not exact. > > It would be good to have better small sample approximations or exact > distributions, but I worked on this in scipy.stats when I barely had > any idea about goodness-of-fit tests. > Also, ks_2samp never got the enhancement for one-sided alternatives. > (In statsmodels I have been working so far only on one sample tests, > but not on two-sample tests.) > > (I don't remember if there is a minimum size recommendation, but the > examples I usually checked were larger.) matlab help: http://www.mathworks.com/help/stats/kstest2.html "The asymptotic p value becomes very accurate for large sample sizes, and is believed to be reasonably accurate for sample sizes n1 and n2 such that (n1*n2)/(n1 + n2) >= 4." > > > since it's a community project: Pull Request are welcome > > Josef > >> >> >> -- >> Regards, >> Peng >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user From josef.pktd at gmail.com Thu Nov 1 21:42:03 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 1 Nov 2012 21:42:03 -0400 Subject: [SciPy-User] ks_2samp is not giving the same results as ks.test in R In-Reply-To: References: Message-ID: On Thu, Nov 1, 2012 at 9:41 PM, wrote: > On Thu, Nov 1, 2012 at 9:14 PM, wrote: >> On Thu, Nov 1, 2012 at 8:28 PM, Peng Yu wrote: >>> Hi, >>> >>> The ks_2samp function does not give the same answer as ks.test in R. >>> Does anybody know why they are different? Is ks_2samp compute >>> something different? >>> >>> helium:~/linux/test/python/man/library/scipy/stats/ks_2samp$ Rscript main.R >>>> ks.test(1:5, 11:15) >>> >>> Two-sample Kolmogorov-Smirnov test >>> >>> data: 1:5 and 11:15 >>> D = 1, p-value = 0.007937 >>> alternative hypothesis: two-sided >>> >>>> ks.test(1:5, 11:15, alternative='less') >>> >>> Two-sample Kolmogorov-Smirnov test >>> >>> data: 1:5 and 11:15 >>> D^- = 0, p-value = 1 >>> alternative hypothesis: the CDF of x lies below that of y >>> >>>> ks.test(1:5, 11:15, alternative='greater') >>> >>> Two-sample Kolmogorov-Smirnov test >>> >>> data: 1:5 and 11:15 >>> D^+ = 1, p-value = 0.006738 >>> alternative hypothesis: the CDF of x lies above that of y >>> >>>> >>>> >>> helium:~/linux/test/python/man/library/scipy/stats/ks_2samp$ ./main.py >>> (1.0, 0.0037813540593701006) >>> helium:~/linux/test/python/man/library/scipy/stats/ks_2samp$ cat main.py >>> #!/usr/bin/env python >>> >>> from scipy.stats import ks_2samp >>> print ks_2samp([1,2,3,4,5], [11,12,13,14,15]) >> >> R uses by default an "exact" distribution for small samples if there >> are no ties. >> If there are ties or with a large sample, R uses the asymptotic distribution. >> >> If I read the function correctly, then scipy.stats is using a small >> sample approximation by Stephens. (But I would have to look up the >> formula to verify this.) > > http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test#Two-sample_Kolmogorov.E2.80.93Smirnov_test > has the weighted sample size: en = np.sqrt(n1*n2/float(n1+n2)) > the small sample weighting ((en+0.12+0.11/en)*d) is the same as in > Stephens (1970, 1985?) for the one sample test. > I don't have a reference for the two sample approximation right now. > > (another bit of random information) > tables are often only available for 0.01 to 0.25 and approximations (hit send too fast) 0.001 to 0.25 > are targeted on that range and might not be as accurate outside of it > > Josef > > >> >> In the example below with a bit larger sample and no ties, our >> approximation is closer to R's "exact" pvalue than the asymptotic >> distribution if exact=FALSE. >> >>> ks.test(1:25, (10:30)-0.5, exact=FALSE) >> >> Two-sample Kolmogorov-Smirnov test >> >> data: 1:25 and (10:30) - 0.5 >> D = 0.36, p-value = 0.1038 >> alternative hypothesis: two-sided >> >>> ks.test(1:25, (10:30)-0.5, exact=TRUE) >> >> Two-sample Kolmogorov-Smirnov test >> >> data: 1:25 and (10:30) - 0.5 >> D = 0.36, p-value = 0.07608 >> alternative hypothesis: two-sided >> >> >>>>> stats.ks_2samp(np.arange(1.,26), np.arange(10,31.)-0.5) >> (0.35999999999999999, 0.078993426961291274) >> >> >> For the 1 sample kstest I used (when I rewrote stats.kstest) an >> approximation that is closer to the exact distribution than the >> asymptotic distribution, but it's also not exact. >> >> It would be good to have better small sample approximations or exact >> distributions, but I worked on this in scipy.stats when I barely had >> any idea about goodness-of-fit tests. >> Also, ks_2samp never got the enhancement for one-sided alternatives. >> (In statsmodels I have been working so far only on one sample tests, >> but not on two-sample tests.) >> >> (I don't remember if there is a minimum size recommendation, but the >> examples I usually checked were larger.) > > matlab help: http://www.mathworks.com/help/stats/kstest2.html > "The asymptotic p value becomes very accurate for large sample sizes, > and is believed to be reasonably accurate for sample sizes n1 and n2 > such that (n1*n2)/(n1 + n2) >= 4." >> >> >> since it's a community project: Pull Request are welcome >> >> Josef >> >>> >>> >>> -- >>> Regards, >>> Peng >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user From josef.pktd at gmail.com Thu Nov 1 23:27:01 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 1 Nov 2012 23:27:01 -0400 Subject: [SciPy-User] ks_2samp is not giving the same results as ks.test in R In-Reply-To: References: Message-ID: On Thu, Nov 1, 2012 at 9:42 PM, wrote: > On Thu, Nov 1, 2012 at 9:41 PM, wrote: >> On Thu, Nov 1, 2012 at 9:14 PM, wrote: >>> On Thu, Nov 1, 2012 at 8:28 PM, Peng Yu wrote: >>>> Hi, >>>> >>>> The ks_2samp function does not give the same answer as ks.test in R. >>>> Does anybody know why they are different? Is ks_2samp compute >>>> something different? >>>> >>>> helium:~/linux/test/python/man/library/scipy/stats/ks_2samp$ Rscript main.R >>>>> ks.test(1:5, 11:15) >>>> >>>> Two-sample Kolmogorov-Smirnov test >>>> >>>> data: 1:5 and 11:15 >>>> D = 1, p-value = 0.007937 >>>> alternative hypothesis: two-sided >>>> >>>>> ks.test(1:5, 11:15, alternative='less') >>>> >>>> Two-sample Kolmogorov-Smirnov test >>>> >>>> data: 1:5 and 11:15 >>>> D^- = 0, p-value = 1 >>>> alternative hypothesis: the CDF of x lies below that of y >>>> >>>>> ks.test(1:5, 11:15, alternative='greater') >>>> >>>> Two-sample Kolmogorov-Smirnov test >>>> >>>> data: 1:5 and 11:15 >>>> D^+ = 1, p-value = 0.006738 >>>> alternative hypothesis: the CDF of x lies above that of y >>>> >>>>> >>>>> >>>> helium:~/linux/test/python/man/library/scipy/stats/ks_2samp$ ./main.py >>>> (1.0, 0.0037813540593701006) >>>> helium:~/linux/test/python/man/library/scipy/stats/ks_2samp$ cat main.py >>>> #!/usr/bin/env python >>>> >>>> from scipy.stats import ks_2samp >>>> print ks_2samp([1,2,3,4,5], [11,12,13,14,15]) >>> >>> R uses by default an "exact" distribution for small samples if there >>> are no ties. >>> If there are ties or with a large sample, R uses the asymptotic distribution. >>> >>> If I read the function correctly, then scipy.stats is using a small >>> sample approximation by Stephens. (But I would have to look up the >>> formula to verify this.) >> >> http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test#Two-sample_Kolmogorov.E2.80.93Smirnov_test >> has the weighted sample size: en = np.sqrt(n1*n2/float(n1+n2)) >> the small sample weighting ((en+0.12+0.11/en)*d) is the same as in >> Stephens (1970, 1985?) for the one sample test. >> I don't have a reference for the two sample approximation right now. >> >> (another bit of random information) >> tables are often only available for 0.01 to 0.25 and approximations > (hit send too fast) 0.001 to 0.25 >> are targeted on that range and might not be as accurate outside of it >> >> Josef >> >> >>> >>> In the example below with a bit larger sample and no ties, our >>> approximation is closer to R's "exact" pvalue than the asymptotic >>> distribution if exact=FALSE. >>> >>>> ks.test(1:25, (10:30)-0.5, exact=FALSE) >>> >>> Two-sample Kolmogorov-Smirnov test >>> >>> data: 1:25 and (10:30) - 0.5 >>> D = 0.36, p-value = 0.1038 >>> alternative hypothesis: two-sided >>> >>>> ks.test(1:25, (10:30)-0.5, exact=TRUE) >>> >>> Two-sample Kolmogorov-Smirnov test >>> >>> data: 1:25 and (10:30) - 0.5 >>> D = 0.36, p-value = 0.07608 >>> alternative hypothesis: two-sided >>> >>> >>>>>> stats.ks_2samp(np.arange(1.,26), np.arange(10,31.)-0.5) >>> (0.35999999999999999, 0.078993426961291274) >>> >>> >>> For the 1 sample kstest I used (when I rewrote stats.kstest) an >>> approximation that is closer to the exact distribution than the >>> asymptotic distribution, but it's also not exact. >>> >>> It would be good to have better small sample approximations or exact >>> distributions, but I worked on this in scipy.stats when I barely had >>> any idea about goodness-of-fit tests. >>> Also, ks_2samp never got the enhancement for one-sided alternatives. >>> (In statsmodels I have been working so far only on one sample tests, >>> but not on two-sample tests.) >>> >>> (I don't remember if there is a minimum size recommendation, but the >>> examples I usually checked were larger.) >> >> matlab help: http://www.mathworks.com/help/stats/kstest2.html >> "The asymptotic p value becomes very accurate for large sample sizes, >> and is believed to be reasonably accurate for sample sizes n1 and n2 >> such that (n1*n2)/(n1 + n2) >= 4." >>> >>> >>> since it's a community project: Pull Request are welcome (since I haven't spammed the mailing list in a while, and it's fast) I think, except in very small sample, ks_2samp looks ok. (However, independent of our p-values, the power of Kolmogorov-Smirnov in small samples is awful. >>> stats.ttest_ind(np.arange(1.,26), np.arange(10,31.)-0.5) (-3.2015129142545442, 0.0025398589272691129) reject Null >>> stats.ks_2samp(np.arange(1.,26), np.arange(10,31.)-0.5) (0.35999999999999999, 0.078993426961291274) don't reject Null at 0.05 ) --------------------------- # -*- coding: utf-8 -*- """Comparing pvalues for ks_2samp in small sample, permutation test Created on Thu Nov 01 22:42:30 2012 Author: Josef Perktold Results: sample: 1 permutation 0.07704 scipy 0.0789934269613 R exact 0.07608 R asymp 0.1038 subset results [ 0.0774 0.0783 0.0732 0.0772 0.0775 0.0755 0.0791 0.0743 0.0816 0.0763] >>> execfile('permutations_ks1samp.py') sample: 0 permutation 0.00791 scipy 0.00378135405937 R exact 0.007937 R asymp 0.01348 subset results [ 0.008 0.0094 0.0074 0.0071 0.0071 0.0084 0.0073 0.0084 0.0083 0.0077] """ import numpy as np from scipy import stats sample = 0 if sample == 0: x1, y1 = np.arange(1.,6), np.arange(11,16.) else: x1, y1 = np.arange(1.,26), np.arange(10,31.)-0.5 nx = len(x1) D, pval = stats.ks_2samp(x1, y1) #permutation p-values n_rep = 100000 #make it large results = np.empty((n_rep, 2)) results.fill(np.nan) x, y = x1.copy(), y1.copy() xy = np.concatenate((x,y)) np.random.seed(2345678) for ii in xrange(n_rep): np.random.shuffle(xy) results[ii] = stats.ks_2samp(xy[:nx], xy[nx:]) print '\nsample:', sample print 'permutation', (results[:,0] >= D).mean() print 'scipy ', pval if sample == 0: print 'R exact ', 0.007937 print 'R asymp ', 0.01348 else: print 'R exact ', 0.07608 print 'R asymp ', 0.1038 print '\nsubset results' print (results[:,0].reshape(10000, -1) >= D).mean(0) ------------------------- Josef >>> >>> Josef >>> >>>> >>>> >>>> -- >>>> Regards, >>>> Peng >>>> _______________________________________________ >>>> SciPy-User mailing list >>>> SciPy-User at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/scipy-user From pav at iki.fi Fri Nov 2 10:09:00 2012 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 2 Nov 2012 14:09:00 +0000 (UTC) Subject: [SciPy-User] =?utf-8?q?permuted_output_arguments_in_online_tutori?= =?utf-8?q?al_for=09scipy=2Esignal=2Efreqz?= References: Message-ID: Karl Kappler sjgeophysics.com> writes: > > > Hi, Not sure I am sending this to the right place, just wanted to let you know that the output arguments w and h in the worked example on the webpage:http://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.freqz.ht ml > > are flipped.? i.e. the line: > h, w = signal.freqz(b)should readw, h = signal.freqz(b) This has apparently been fixed some time ago: http://docs.scipy.org/doc/scipy-dev/reference/generated/scipy.signal.freqz.html From josef.pktd at gmail.com Fri Nov 2 11:41:46 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 2 Nov 2012 11:41:46 -0400 Subject: [SciPy-User] ks_2samp is not giving the same results as ks.test in R In-Reply-To: References: Message-ID: On Thu, Nov 1, 2012 at 11:27 PM, wrote: > On Thu, Nov 1, 2012 at 9:42 PM, wrote: >> On Thu, Nov 1, 2012 at 9:41 PM, wrote: >>> On Thu, Nov 1, 2012 at 9:14 PM, wrote: >>>> On Thu, Nov 1, 2012 at 8:28 PM, Peng Yu wrote: >>>>> Hi, >>>>> >>>>> The ks_2samp function does not give the same answer as ks.test in R. >>>>> Does anybody know why they are different? Is ks_2samp compute >>>>> something different? >>>>> >>>>> helium:~/linux/test/python/man/library/scipy/stats/ks_2samp$ Rscript main.R >>>>>> ks.test(1:5, 11:15) >>>>> >>>>> Two-sample Kolmogorov-Smirnov test >>>>> >>>>> data: 1:5 and 11:15 >>>>> D = 1, p-value = 0.007937 >>>>> alternative hypothesis: two-sided >>>>> >>>>>> ks.test(1:5, 11:15, alternative='less') >>>>> >>>>> Two-sample Kolmogorov-Smirnov test >>>>> >>>>> data: 1:5 and 11:15 >>>>> D^- = 0, p-value = 1 >>>>> alternative hypothesis: the CDF of x lies below that of y >>>>> >>>>>> ks.test(1:5, 11:15, alternative='greater') >>>>> >>>>> Two-sample Kolmogorov-Smirnov test >>>>> >>>>> data: 1:5 and 11:15 >>>>> D^+ = 1, p-value = 0.006738 >>>>> alternative hypothesis: the CDF of x lies above that of y >>>>> >>>>>> >>>>>> >>>>> helium:~/linux/test/python/man/library/scipy/stats/ks_2samp$ ./main.py >>>>> (1.0, 0.0037813540593701006) >>>>> helium:~/linux/test/python/man/library/scipy/stats/ks_2samp$ cat main.py >>>>> #!/usr/bin/env python >>>>> >>>>> from scipy.stats import ks_2samp >>>>> print ks_2samp([1,2,3,4,5], [11,12,13,14,15]) >>>> >>>> R uses by default an "exact" distribution for small samples if there >>>> are no ties. >>>> If there are ties or with a large sample, R uses the asymptotic distribution. >>>> >>>> If I read the function correctly, then scipy.stats is using a small >>>> sample approximation by Stephens. (But I would have to look up the >>>> formula to verify this.) >>> >>> http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test#Two-sample_Kolmogorov.E2.80.93Smirnov_test >>> has the weighted sample size: en = np.sqrt(n1*n2/float(n1+n2)) >>> the small sample weighting ((en+0.12+0.11/en)*d) is the same as in >>> Stephens (1970, 1985?) for the one sample test. >>> I don't have a reference for the two sample approximation right now. >>> >>> (another bit of random information) >>> tables are often only available for 0.01 to 0.25 and approximations >> (hit send too fast) 0.001 to 0.25 >>> are targeted on that range and might not be as accurate outside of it >>> >>> Josef >>> >>> >>>> >>>> In the example below with a bit larger sample and no ties, our >>>> approximation is closer to R's "exact" pvalue than the asymptotic >>>> distribution if exact=FALSE. >>>> >>>>> ks.test(1:25, (10:30)-0.5, exact=FALSE) >>>> >>>> Two-sample Kolmogorov-Smirnov test >>>> >>>> data: 1:25 and (10:30) - 0.5 >>>> D = 0.36, p-value = 0.1038 >>>> alternative hypothesis: two-sided >>>> >>>>> ks.test(1:25, (10:30)-0.5, exact=TRUE) >>>> >>>> Two-sample Kolmogorov-Smirnov test >>>> >>>> data: 1:25 and (10:30) - 0.5 >>>> D = 0.36, p-value = 0.07608 >>>> alternative hypothesis: two-sided >>>> >>>> >>>>>>> stats.ks_2samp(np.arange(1.,26), np.arange(10,31.)-0.5) >>>> (0.35999999999999999, 0.078993426961291274) >>>> >>>> >>>> For the 1 sample kstest I used (when I rewrote stats.kstest) an >>>> approximation that is closer to the exact distribution than the >>>> asymptotic distribution, but it's also not exact. >>>> >>>> It would be good to have better small sample approximations or exact >>>> distributions, but I worked on this in scipy.stats when I barely had >>>> any idea about goodness-of-fit tests. >>>> Also, ks_2samp never got the enhancement for one-sided alternatives. >>>> (In statsmodels I have been working so far only on one sample tests, >>>> but not on two-sample tests.) >>>> >>>> (I don't remember if there is a minimum size recommendation, but the >>>> examples I usually checked were larger.) >>> >>> matlab help: http://www.mathworks.com/help/stats/kstest2.html >>> "The asymptotic p value becomes very accurate for large sample sizes, >>> and is believed to be reasonably accurate for sample sizes n1 and n2 >>> such that (n1*n2)/(n1 + n2) >= 4." >>>> >>>> >>>> since it's a community project: Pull Request are welcome > > (since I haven't spammed the mailing list in a while, and it's fast) > > I think, except in very small sample, ks_2samp looks ok. > > (However, independent of our p-values, the power of Kolmogorov-Smirnov > in small samples is awful. >>>> stats.ttest_ind(np.arange(1.,26), np.arange(10,31.)-0.5) > (-3.2015129142545442, 0.0025398589272691129) reject Null >>>> stats.ks_2samp(np.arange(1.,26), np.arange(10,31.)-0.5) > (0.35999999999999999, 0.078993426961291274) don't reject Null at 0.05 > ) Anderson-Darling has in most cases higher power than Kolmogorov-Smirnov on my wishlist for a long time: "K-sample Anderson?Darling tests" http://scholar.google.com/scholar?cluster=1766356837451056669&hl=en&as_sdt=0,5&sciodt=0,5 volunteers? Josef > > --------------------------- > # -*- coding: utf-8 -*- > """Comparing pvalues for ks_2samp in small sample, permutation test > > Created on Thu Nov 01 22:42:30 2012 > Author: Josef Perktold > > Results: > > sample: 1 > permutation 0.07704 > scipy 0.0789934269613 > R exact 0.07608 > R asymp 0.1038 > > subset results > [ 0.0774 0.0783 0.0732 0.0772 0.0775 0.0755 0.0791 0.0743 0.0816 > 0.0763] >>>> execfile('permutations_ks1samp.py') > > sample: 0 > permutation 0.00791 > scipy 0.00378135405937 > R exact 0.007937 > R asymp 0.01348 > > subset results > [ 0.008 0.0094 0.0074 0.0071 0.0071 0.0084 0.0073 0.0084 0.0083 > 0.0077] > > """ > > import numpy as np > from scipy import stats > > sample = 0 > > if sample == 0: > x1, y1 = np.arange(1.,6), np.arange(11,16.) > else: > x1, y1 = np.arange(1.,26), np.arange(10,31.)-0.5 > nx = len(x1) > > D, pval = stats.ks_2samp(x1, y1) > > #permutation p-values > n_rep = 100000 #make it large > results = np.empty((n_rep, 2)) > results.fill(np.nan) > > x, y = x1.copy(), y1.copy() > xy = np.concatenate((x,y)) > np.random.seed(2345678) > for ii in xrange(n_rep): > np.random.shuffle(xy) > results[ii] = stats.ks_2samp(xy[:nx], xy[nx:]) > > print '\nsample:', sample > print 'permutation', (results[:,0] >= D).mean() > print 'scipy ', pval > if sample == 0: > print 'R exact ', 0.007937 > print 'R asymp ', 0.01348 > else: > print 'R exact ', 0.07608 > print 'R asymp ', 0.1038 > > print '\nsubset results' > print (results[:,0].reshape(10000, -1) >= D).mean(0) > ------------------------- > > Josef > > >>>> >>>> Josef >>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> Peng >>>>> _______________________________________________ >>>>> SciPy-User mailing list >>>>> SciPy-User at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/scipy-user From ralf.gommers at gmail.com Fri Nov 2 15:37:44 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 2 Nov 2012 20:37:44 +0100 Subject: [SciPy-User] numpy.distutils cross compile question In-Reply-To: <508F471F.8030703@mail.ustc.edu.cn> References: <508E8213.3070305@molden.no> <508F471F.8030703@mail.ustc.edu.cn> Message-ID: On Tue, Oct 30, 2012 at 4:18 AM, Zhenfei Yuan wrote: > Dear all. > > I've written a small package in my field containing some pure > python scripts and fortran extension libraries. At first I just use f2py > for the compiling work before importing these extensions into python and > they all worked well. This time, I write a "setup.py" file which import > "setup" function in "numpy.distutils.core" and will build a scipy sub > package on my ubuntu 64 machine, with gfortran specified by the command > "python setup.py build fgnu95", which works well. > > My questions comes with the problem when I'd like to build a win > 32/64 package. I think it concerns cross compiling, so I installed mingw > compilers like "i686-w64-mingw32-gfortran" and "i686-w64-mingw-32-gcc" > on my ubuntu 12.04. However I don't know how to write the setup.py file > for cross compiling using numpy.distutils. I tried typing "python > setup.py" and specify fortran compiler by typing > "fi686-w64-mingw32-gfortran" for building, however it doesn't work. So > I'm wondering whether I have to build this package on linux 32, 64 bit > and win 32, 64 bit? > I don't think that will work. Even if the compilers would work, distutils isn't flexible enough to pass the right flags to them. If you want to build Windows binaries on Linux, you should use install MinGW under Wine. Even that is a little painful, but Ondrej Certik recently wrote some scripts to install everything that's needed: https://github.com/certik/numpy-vendor Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From zfyuan at mail.ustc.edu.cn Fri Nov 2 22:44:28 2012 From: zfyuan at mail.ustc.edu.cn (Zhenfei Yuan) Date: Sat, 03 Nov 2012 10:44:28 +0800 Subject: [SciPy-User] numpy.distutils cross compile question In-Reply-To: References: <508E8213.3070305@molden.no> <508F471F.8030703@mail.ustc.edu.cn> Message-ID: <5094850C.8090408@mail.ustc.edu.cn> On 11/03/2012 03:37 AM, Ralf Gommers wrote: > > > > On Tue, Oct 30, 2012 at 4:18 AM, Zhenfei Yuan > wrote: > > Dear all. > > I've written a small package in my field containing some pure > python scripts and fortran extension libraries. At first I just > use f2py > for the compiling work before importing these extensions into > python and > they all worked well. This time, I write a "setup.py" file which > import > "setup" function in "numpy.distutils.core" and will build a scipy sub > package on my ubuntu 64 machine, with gfortran specified by the > command > "python setup.py build fgnu95", which works well. > > My questions comes with the problem when I'd like to build a win > 32/64 package. I think it concerns cross compiling, so I installed > mingw > compilers like "i686-w64-mingw32-gfortran" and "i686-w64-mingw-32-gcc" > on my ubuntu 12.04. However I don't know how to write the setup.py > file > for cross compiling using numpy.distutils. I tried typing "python > setup.py" and specify fortran compiler by typing > "fi686-w64-mingw32-gfortran" for building, however it doesn't work. So > I'm wondering whether I have to build this package on linux 32, 64 bit > and win 32, 64 bit? > > > I don't think that will work. Even if the compilers would work, > distutils isn't flexible enough to pass the right flags to them. If > you want to build Windows binaries on Linux, you should use install > MinGW under Wine. Even that is a little painful, but Ondrej Certik > recently wrote some scripts to install everything that's needed: > https://github.com/certik/numpy-vendor > > Ralf > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user It confused me for some time, and I found a win machine and compiled on it with mingw32. Thanks very much, Ralf. -- zfyuan -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sat Nov 3 13:49:08 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 3 Nov 2012 18:49:08 +0100 Subject: [SciPy-User] overflow in .sum() of dtype bool sparse matrix In-Reply-To: <20121030133830.GE5955@onerussian.com> References: <20121030133830.GE5955@onerussian.com> Message-ID: On Tue, Oct 30, 2012 at 2:38 PM, Yaroslav Halchenko wrote: > I wonder if that is somehow considered a feature and manual casting is > generally advised in such cases: calling .sum on a bool matrix can easily > lead > to overflows causing bogus results (works fine on ndarrays): > > % git describe --tags > v0.4.3-6232-g43c7982 > > % PYTHONPATH=$PWD ../demo-scipy-sparse-negativesoverflow.py > summing 128 booleans in leads to answer [128] > summing 128 booleans in leads to > answer [[-128]] > > % cat ../demo-scipy-sparse-negativesoverflow.py > #!/usr/bin/python > > import numpy as np > import scipy.sparse as sp > test = np.random.rand(128, 1) > test_m= sp.csc_matrix(test) > > for t in test, test_m: > test_bool=t.astype('bool') > sum = test_bool.sum(axis=0) > print "summing %d booleans in %s leads to answer %s" \ > % (t.shape[0], test_bool.__class__, sum) > Boolean sparse matrices are apparently not supported: In [37]: sparse.sputils.supported_dtypes Out[37]: [numpy.int8, numpy.uint8, numpy.int16, numpy.uint16, numpy.int32, numpy.uint32, numpy.int64, numpy.uint64, numpy.float32, numpy.float64, numpy.float96, numpy.complex64, numpy.complex128, numpy.complex192] In [38]: x = sp.csc_matrix(np.random.rand(3, 1)).astype(np.bool) In [39]: x.tocsr() Out[39]: <3x1 sparse matrix of type '' with 3 stored elements in Compressed Sparse Row format> Does anyone know why? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sat Nov 3 14:04:20 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 3 Nov 2012 14:04:20 -0400 Subject: [SciPy-User] special: the missing documentation Message-ID: I_{|n|} in http://en.wikipedia.org/wiki/Von_Mises_distribution#Moments >>> special.ivp(0, kappa, n=0) 11.301921952136331 >>> special.i0(kappa) 11.301921952136331 >>> special.ivp(0, kappa, n=1) 9.7594651537044488 >>> special.i1(kappa) 9.7594651537044488 >>> special.ivp(1, kappa, n=0) 9.7594651537044488 induction fails >>> special.ivp(2, kappa, n=0) 6.4221893752841046 >>> special.ivp(1, kappa, n=1) 8.8620556637102172 >>> special.ivp(0, kappa, n=2) 8.8620556637102172 >>> xx = np.random.vonmises(mu, kappa, size=10000) >>> circ_moment(xx, p=2, cent=False) ((-0.57140066760819208-0.012030738424083472j), 0.57152730609317026, -3.1205409436169562) >>> special.ivp(2, kappa, n=0) / special.i0(kappa) * np.exp(2 * 1j*mu) (-0.56823869448772457+6.9589169835606258e-17j) Josef proof by example works without understanding From pav at iki.fi Sat Nov 3 14:55:30 2012 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 03 Nov 2012 20:55:30 +0200 Subject: [SciPy-User] special: the missing documentation In-Reply-To: References: Message-ID: 03.11.2012 20:04, josef.pktd at gmail.com kirjoitti: > I_{|n|} in http://en.wikipedia.org/wiki/Von_Mises_distribution#Moments >>>> special.ivp(0, kappa, n=0) > 11.301921952136331 If I understand what you are getting at: http://docs.scipy.org/doc/scipy/reference/generated/scipy.special.ivp.html http://functions.wolfram.com/Bessel-TypeFunctions/BesselI/20/01/02/0003/ http://functions.wolfram.com/Bessel-TypeFunctions/BesselI/20/02/02/0005/ -- Pauli Virtanen From josef.pktd at gmail.com Sat Nov 3 15:24:03 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 3 Nov 2012 15:24:03 -0400 Subject: [SciPy-User] special: the missing documentation In-Reply-To: References: Message-ID: On Sat, Nov 3, 2012 at 2:55 PM, Pauli Virtanen wrote: > 03.11.2012 20:04, josef.pktd at gmail.com kirjoitti: >> I_{|n|} in http://en.wikipedia.org/wiki/Von_Mises_distribution#Moments > >>>>> special.ivp(0, kappa, n=0) >> 11.301921952136331 > > If I understand what you are getting at: > > http://docs.scipy.org/doc/scipy/reference/generated/scipy.special.ivp.html yes, that's what I used: find a likely sounding function in the list in the documentation then try to figure out what the "cryptic" docstring means I get confused because there are so many bessel functions, and I have not much of an idea about them. And I knew that i1 is the derivative of i0, but non of the other relationships. (wikipedia has a link to the handbook, http://people.math.sfu.ca/~cbm/aands/page_376.htm that I checked now. But it has too many different variable names and not enough english.) But it's very useful to find most of the special functions readily available in scipy. The moments of the Von Mises distribution are just some ``special`` calls. (aside: When I get the next scipy release, I will try to find out what a complex normal distribution looks like.) Thanks, Josef > > http://functions.wolfram.com/Bessel-TypeFunctions/BesselI/20/01/02/0003/ > > http://functions.wolfram.com/Bessel-TypeFunctions/BesselI/20/02/02/0005/ > > -- > Pauli Virtanen > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From helmrp at yahoo.com Mon Nov 5 10:56:54 2012 From: helmrp at yahoo.com (The Helmbolds) Date: Mon, 5 Nov 2012 07:56:54 -0800 (PST) Subject: [SciPy-User] Questions about the "ode" differential equation solver Message-ID: <1352131014.4663.YahooMailNeo@web31812.mail.mud.yahoo.com> My questions about the "ode" differential equation sover are stated after I lay out the specific setups used. ? Given the following inputs to "odeint" and to "ode" for the pair of differential equations:? "xdot = -Dy",? "ydot = -Ax". ??? (A, D) = (1.0e-2, 1.5e-2) ??# Parameters. ?params = (A, D) ??? (x0, y0) = (1000, 1500) ??# Initial values. ??? w0 = np.asarray([x0, y0], dtype = 'float') ??? def J(w, t, *params): ??????? A, D = params ??????? x, y = w ??????? JJ = [[0, -A], [-D, 0]] ??????? return JJ ??? def func(w, t, *params): ??????? A, D = params ??????? x, y = w ??????? xdot = -D*y ??????? ydot = -A*x ??????? temp = [xdot, ydot] ??????? return temp ??? tseq = range(11) #+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++? First, use "odeint" on this problem. ? ????from scipy import integrate ??? res = integrate.odeint(func, w0, tseq, args=params, Dfun=J, col_deriv=0,? ??????full_output=0, ml=None, mu=None, rtol=None, atol=None,? ??????tcrit=None, h0=0.0, hmax=0.0, hmin=0.0, ixpr=0, mxstep=0,? ??????mxhnil=0, mxordn=12, mxords=5, printmessg=0) ?? ????print res ?? This yields the values shown here (in a different format) ????????? t???????? x?????????? y????? ????? ======== =========== =========== ???????? 0.00???? 1000.00???? 1500.00 ???????? 1.00????? 977.57???? 1490.11 ???????? 2.00????? 955.30???? 1480.45 ???????? 3.00????? 933.16???? 1471.01 ???????? 4.00????? 911.16???? 1461.78 ???????? 5.00????? 889.31???? 1452.78 ???????? 6.00????? 867.58???? 1444.00 ???????? 7.00????? 845.98???? 1435.43 ???????? 8.00????? 824.52???? 1427.08 ???????? 9.00????? 803.17???? 1418.94 ??????? 10.00????? 781.95???? 1411.01 #++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++?? Now use "ode" on same problem.?? ????t0 = 0 ??? # Order of variables changed from (w, t) to (t, w). ??? def f(t, w, params):?????????????????? ?# Note -- no *args !!! ??????? A, D = params ??????? x, y = w ??????? return [-D * y, -A * x]???????????? # Returns a list. ??? def jac(t, w, params):????????????????? # Note -- no *args !!! ??????? A, D = params ??????? x, y = w ??????? return [ [0, -D], [-A, 0] ]???????? # Returns a list of lists. ???? ????from scipy.integrate import ode ??? r = ode(f, jac) ??? r.set_integrator('vode', method = 'adams', with_jacobian = True) ??? r.set_initial_value(w0, t0) ??? r.set_f_params(params)????????????????? # Note -- no *args !!! ??? r.set_jac_params(params)??????????????? # Note -- no *args !!! ??? tf = 10 ??? dt = 1 ??? while r.successful() and r.t < tf: ??????? r.integrate(r.t + dt) ??????? temp = [r.t, r.y] ??????? print 'temp = ', temp ? This produces essentially the same values as when using "odeint". ? My questions are as follows: Why the quirky change in the order of the variables? ? I'm puzzled by the absence of *args or *params in "ode". Moreover, if I change any of these to the "*" form, "ode" crashes with an error of the general form: ????"ValueError: need more than 0 values to unpack" What am I doing wrong? ? Bob H -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.fortunov at man.com Tue Nov 6 05:55:09 2012 From: daniel.fortunov at man.com (Daniel Fortunov) Date: Tue, 6 Nov 2012 10:55:09 +0000 (UTC) Subject: [SciPy-User] ImportError exits with code 0 after importing scipy.stats (WTF?!) Message-ID: Hello, 01234567890123456789012345678901234567890123456789012345678901234567890123456789 I've been struggling to track down a most curious behaviour that we're observing in production, which has the nasty side-effect of causing job to crash and burn, but silently, so that we don't notice until much later. I've got it down to the following repro steps, on Windows... An ImportError causes the process to exit with return code 1 (as expected): (scipytest) C:>python -c "import donkey" Traceback (most recent call last): File "", line 1, in ImportError: No module named donkey (scipytest) C:>echo %errorlevel% 1 However, if scipy.stats has been imported, that same ImportError causes the process to exit with return code 0 (most unexpected!): (scipytest) C:>python -c "import scipy.stats; import donkey" Traceback (most recent call last): File "", line 1, in ImportError: No module named donkey (scipytest) C:>echo %errorlevel% 0 I notice that numpy has a custom PackageLoader in _import_tools, which scipy makes use of, and I think this might be related, but I'm struggling to get to the bottom of the issue. Does anyone have any ideas on what is going on here? Can anyone reproduce this with the same (or similar) software versions below? Versions:- OS: Windows 7 Enterprise (SP1) Python: 2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit (Intel)] numpy: 1.6.2 (built from source) scipy: 0.9.0 (built from source) We are using a rather old version of scipy (0.9.0) and I wanted to get an "official build" of something newer, but the EPD repository only has scipy 0.8.0 built for Python 2.6 (https://www.enthought.com/repo/epd/eggs/python2.6.html) Any help is much appreciated. Thanks in advance, Daniel Fortunov From takowl at gmail.com Tue Nov 6 06:17:29 2012 From: takowl at gmail.com (Thomas Kluyver) Date: Tue, 6 Nov 2012 11:17:29 +0000 Subject: [SciPy-User] ImportError exits with code 0 after importing scipy.stats (WTF?!) In-Reply-To: References: Message-ID: On 6 November 2012 10:55, Daniel Fortunov wrote: > Can anyone reproduce this with the same (or similar) software versions > below? For reference, it appears to be working correctly on Linux (Ubuntu 12.04, Python 2.7, SciPy 0.9.0): ~$ python -c "import scipy.stats; import woiejr" Traceback (most recent call last): File "", line 1, in ImportError: No module named woiejr ~$ echo $? 1 Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From klonuo at gmail.com Tue Nov 6 07:19:46 2012 From: klonuo at gmail.com (klo uo) Date: Tue, 6 Nov 2012 13:19:46 +0100 Subject: [SciPy-User] ImportError exits with code 0 after importing scipy.stats (WTF?!) In-Reply-To: References: Message-ID: It's also fine on XP: ======================================== >python -c "import scipy.stats; import donkey" Traceback (most recent call last): File "", line 1, in ImportError: No module named donkey >echo %errorlevel% 1 ======================================== On Tue, Nov 6, 2012 at 12:17 PM, Thomas Kluyver wrote: > On 6 November 2012 10:55, Daniel Fortunov wrote: > >> Can anyone reproduce this with the same (or similar) software versions >> below? > > > For reference, it appears to be working correctly on Linux (Ubuntu 12.04, > Python 2.7, SciPy 0.9.0): > > ~$ python -c "import scipy.stats; import woiejr" > > Traceback (most recent call last): > File "", line 1, in > ImportError: No module named woiejr > ~$ echo $? > 1 > > Thomas > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Nov 6 08:02:01 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 6 Nov 2012 08:02:01 -0500 Subject: [SciPy-User] ImportError exits with code 0 after importing scipy.stats (WTF?!) In-Reply-To: References: Message-ID: I can reproduce the original changed error level, with older numpy >python -c "import scipy; print scipy.__version__; import numpy; print numpy.__version__" 0.9.0 1.5.1 python 2.6.5, Windows 7 32bit and also with python 2.7 and >"C:\Programs\Python27\python.exe" -c "import scipy; print(scipy.__version__); import numpy; print(numpy.__version__") 0.11.0b1 1.6.1 Josef On Tue, Nov 6, 2012 at 7:19 AM, klo uo wrote: > It's also fine on XP: > > ======================================== >>python -c "import scipy.stats; import donkey" > > Traceback (most recent call last): > File "", line 1, in > ImportError: No module named donkey > >>echo %errorlevel% > 1 > ======================================== > > > On Tue, Nov 6, 2012 at 12:17 PM, Thomas Kluyver wrote: >> >> On 6 November 2012 10:55, Daniel Fortunov wrote: >>> >>> Can anyone reproduce this with the same (or similar) software versions >>> below? >> >> >> For reference, it appears to be working correctly on Linux (Ubuntu 12.04, >> Python 2.7, SciPy 0.9.0): >> >> ~$ python -c "import scipy.stats; import woiejr" >> >> Traceback (most recent call last): >> File "", line 1, in >> ImportError: No module named woiejr >> ~$ echo $? >> 1 >> >> Thomas >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From mhoffman at gmail.com Mon Nov 5 15:50:07 2012 From: mhoffman at gmail.com (Matt Hoffman) Date: Mon, 5 Nov 2012 12:50:07 -0800 Subject: [SciPy-User] Calling scipy.special._cephes functions directly from C. Message-ID: Hi all - I need to compute the inverse incomplete beta function, which can be found in scipy.special.betaincinv, many times within a loop. As a result I've coded this particular loop in cython, and made calls instead directly to the cephes function incbi, which looks like it should be defined in scipy/special/_cephes.so as cephes_incbi. In my cython I've declared this extern as cdef extern double cephes_incbi(double, double, double) And made sure to include the relevant scipy.special._cephes. This works on a mac machine of mine, but not under linux. I'm sure I'm having some issues with name mangling, or something, but I can't quite put my finger on it and I'm wondering if anyone else has done something similar and can give me a few pointers. Or even can let me know if there's a better way to go about this. Thanks! -matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Tue Nov 6 12:40:48 2012 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 06 Nov 2012 19:40:48 +0200 Subject: [SciPy-User] ImportError exits with code 0 after importing scipy.stats (WTF?!) In-Reply-To: References: Message-ID: 06.11.2012 12:55, Daniel Fortunov kirjoitti: [clip] > However, if scipy.stats has been imported, that same ImportError causes the > process to exit with return code 0 (most unexpected!): Known Windows-only issue, exact cause still unknown: http://projects.scipy.org/scipy/ticket/1314 -- Pauli Virtanen From zfyuan at mail.ustc.edu.cn Wed Nov 7 03:26:14 2012 From: zfyuan at mail.ustc.edu.cn (Zhenfei Yuan) Date: Wed, 07 Nov 2012 16:26:14 +0800 Subject: [SciPy-User] numpy.distutils cross compile question In-Reply-To: References: <508E8213.3070305@molden.no> <508F471F.8030703@mail.ustc.edu.cn> Message-ID: <509A1B26.8060407@mail.ustc.edu.cn> On 11/03/2012 03:37 AM, Ralf Gommers wrote: > > > > On Tue, Oct 30, 2012 at 4:18 AM, Zhenfei Yuan > wrote: > > Dear all. > > I've written a small package in my field containing some pure > python scripts and fortran extension libraries. At first I just > use f2py > for the compiling work before importing these extensions into > python and > they all worked well. This time, I write a "setup.py" file which > import > "setup" function in "numpy.distutils.core" and will build a scipy sub > package on my ubuntu 64 machine, with gfortran specified by the > command > "python setup.py build fgnu95", which works well. > > My questions comes with the problem when I'd like to build a win > 32/64 package. I think it concerns cross compiling, so I installed > mingw > compilers like "i686-w64-mingw32-gfortran" and "i686-w64-mingw-32-gcc" > on my ubuntu 12.04. However I don't know how to write the setup.py > file > for cross compiling using numpy.distutils. I tried typing "python > setup.py" and specify fortran compiler by typing > "fi686-w64-mingw32-gfortran" for building, however it doesn't work. So > I'm wondering whether I have to build this package on linux 32, 64 bit > and win 32, 64 bit? > > > I don't think that will work. Even if the compilers would work, > distutils isn't flexible enough to pass the right flags to them. If > you want to build Windows binaries on Linux, you should use install > MinGW under Wine. Even that is a little painful, but Ondrej Certik > recently wrote some scripts to install everything that's needed: > https://github.com/certik/numpy-vendor > > Ralf > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user Now I have some problem for creating a win64 package using mingw-w64 targeting for either win32 or win64 machine, maybe as you said that distutils isn't flexible enough, since I found that it could only accept some popular compilers but mingw-w64 excluded. I wonder that can numpy-vendor do this work, or is there any other way? I have seen some unofficial win32 and win-amd64 binaries, and I wonder how they make it. With msvc flag on win 64 machine? Thanks a lot, Ralf. -- zfyuan -------------- next part -------------- An HTML attachment was scrubbed... URL: From beamesleach at gmail.com Wed Nov 7 09:47:51 2012 From: beamesleach at gmail.com (Alex Leach) Date: Wed, 07 Nov 2012 14:47:51 +0000 Subject: [SciPy-User] faster expm Message-ID: <1527543.Uu3C0B2Dp3@metabuntu> Sorry to pick up on this old thread, but I too was looking for a faster expm implementation, and had some joy. Thought I'd share... The expokit library seems to be a highly regarded matrix exponentiation library (that Matlab probably uses), which is written in Fortran. I downloaded it a while ago and compiled it with f2py; first time I've used f2py and it worked pretty much out of the box! The Expokit source code is available from:- http://www.maths.uq.edu.au/expokit/download.html Expokit just needs to be compiled and linked against LAPACK and BLAS libraries, so I've got two versions: one linked against NVIDIA's libcublas and another linked against Intel's libmkl (most people would link against blas and lapack libraries from ATLAS, I guess). The library can be specified manually, but it's probably easiest to just let numpy choose (like the command below does). I've had it kicking around for a while, but just checked, and a compile command that just worked:- f2py --fcompiler=intelem -c expokit.f -m expokit --link-blas_opt --link- lapack_opt Unless you use Intel's 64bit Fortran compiler, you'll want to remove or change the --fcompiler option. This creates the Python shared module: expokit.so This assumes a working numpy installation, and fortran compiler. I've only needed to use Pad? approximation myself, which is done with the subroutines [ZD]GPADM. These should give equivalent results to scipy.linalg.expm, but they take a lot more function arguments. time_expm.py, attached, has an example wrapper function and runs some basic timings against scipy.linalg.expm. The only change I made to the Fortran code was to add f2py style comments, so the output variables do return changed, and can be seen when inspecting with the numpy.info function. e.g. >>> import numpy as np >>> import expokit >>> np.info(expokit.dgpadm) dgpadm - Function signature: dgpadm(ideg,t,h,wsp,ipiv,iexph,ns,iflag,[m,ldh,lwsp]) Required arguments: ideg : input int t : input float h : input rank-2 array('d') with bounds (ldh,m) wsp : input rank-1 array('d') with bounds (lwsp) ipiv : input rank-1 array('i') with bounds (m) iexph : in/output rank-0 array(int,'i') ns : in/output rank-0 array(int,'i') iflag : in/output rank-0 array(int,'i') Optional arguments: m := shape(h,1) input int ldh := shape(h,0) input int lwsp := len(wsp) in/output rank-0 array(int,'i') For further information on the subroutine names and arguments, see the Expokit download link, above, which describes each subroutine, and provides separate links to each subroutine's source code. The Fortran source code is where to find the best documentation of each function and its arguments. In terms of merging this into scipy, some work would need to be done... 1) Wrapper functions would need to be created to conform to scipy's expm API; time_expm.py, attached, demonstrates python wrappers for dgpadm and zgpadm. Using PyFort to automatially write and compile C wrappers should give further, marginal performance improvements. 2) Unit-tests on the wrapper functions. I think scipy's expm unit-tests could be used here. 3) The scipy build systems would need to be informed about it. 4) scipy.linalg.__init__.py should probably try and import expokit wrapper functions over the expm, expm2 and expm3 functions. If that fails, fall back to the original scipy versions. Performance improvements? These are the results I got when just running it, averaging the times over 3 runs. Note, both scipy and expokit use the same BLAS dot product functions, which does a lot of the heavy lifting; over multiple processor cores, I might add. ----- Mean time +- S.D. ----- Array size Expokit scipy..expm 25 0.149 += 0.008 0.410 += 0.025 50 0.341 += 0.011 0.678 += 0.019 75 0.650 += 0.049 1.282 += 0.030 100 1.372 += 0.087 1.995 += 0.032 125 1.972 += 0.029 3.007 += 0.145 150 6.038 += 0.062 7.933 += 0.217 175 9.169 += 0.050 11.785 += 0.593 200 12.049 += 0.190 16.281 += 0.293 Hope this might be of help to someone. Kind regards, Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: expokit.f Type: text/x-fortran Size: 161471 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: time_expm.py Type: text/x-python Size: 2935 bytes Desc: not available URL: From jjhelmus at gmail.com Wed Nov 7 10:04:55 2012 From: jjhelmus at gmail.com (Jonathan Helmus) Date: Wed, 07 Nov 2012 10:04:55 -0500 Subject: [SciPy-User] faster expm In-Reply-To: <1527543.Uu3C0B2Dp3@metabuntu> References: <1527543.Uu3C0B2Dp3@metabuntu> Message-ID: <509A7897.5000408@gmail.com> Alex this is some very nice work. Unfortunately I don't think the Expokit licensed (http://www.maths.uq.edu.au/expokit/copyright) is compatible with the BSD license SciPy uses. Specifically approval must be obtain before commercial use. Including the package in the Topical Software list and/or the SciPy Cookbook/SciPy Central might be prudent. - Jonathan Helmus On 11/07/2012 09:47 AM, Alex Leach wrote: > > Sorry to pick up on this old thread, but I too was looking for a > faster expm > > implementation, and had some joy. Thought I'd share... > > The expokit library seems to be a highly regarded matrix exponentiation > > library (that Matlab probably uses), which is written in Fortran. I > downloaded > > it a while ago and compiled it with f2py; first time I've used f2py > and it > > worked pretty much out of the box! > > The Expokit source code is available from:- > > http://www.maths.uq.edu.au/expokit/download.html > > Expokit just needs to be compiled and linked against LAPACK and BLAS > libraries, so I've got two versions: one linked against NVIDIA's > libcublas and another linked against Intel's libmkl (most people would > link against blas and lapack libraries from ATLAS, I guess). The > library can be specified manually, but it's probably easiest to just > let numpy choose (like the command below does). > > I've had it kicking around for a while, but just checked, and a compile > > command that just worked:- > > f2py --fcompiler=intelem -c expokit.f -m expokit --link-blas_opt > --link-lapack_opt > > Unless you use Intel's 64bit Fortran compiler, you'll want to remove > or change the --fcompiler option. > > This creates the Python shared module: expokit.so > > This assumes a working numpy installation, and fortran compiler. > > I've only needed to use Pad? approximation myself, which is done with the > > subroutines [ZD]GPADM. These should give equivalent results to > scipy.linalg.expm, but they take a lot more function arguments. > > time_expm.py, attached, has an example wrapper function and runs some > basic timings against scipy.linalg.expm. > > The only change I made to the Fortran code was to add f2py style > comments, so > > the output variables do return changed, and can be seen when > inspecting with the numpy.info function. e.g. > > >>> import numpy as np > > >>> import expokit > > >>> np.info(expokit.dgpadm) > > dgpadm - Function signature: > > dgpadm(ideg,t,h,wsp,ipiv,iexph,ns,iflag,[m,ldh,lwsp]) > > Required arguments: > > ideg : input int > > t : input float > > h : input rank-2 array('d') with bounds (ldh,m) > > wsp : input rank-1 array('d') with bounds (lwsp) > > ipiv : input rank-1 array('i') with bounds (m) > > iexph : in/output rank-0 array(int,'i') > > ns : in/output rank-0 array(int,'i') > > iflag : in/output rank-0 array(int,'i') > > Optional arguments: > > m := shape(h,1) input int > > ldh := shape(h,0) input int > > lwsp := len(wsp) in/output rank-0 array(int,'i') > > For further information on the subroutine names and arguments, see the > Expokit download link, above, which describes each subroutine, and > provides separate links to each subroutine's source code. The Fortran > source code is where to find the best documentation of each function > and its arguments. > > In terms of merging this into scipy, some work would need to be done... > > 1) Wrapper functions would need to be created to conform to scipy's > expm API; time_expm.py, attached, demonstrates python wrappers for > dgpadm and zgpadm. Using PyFort to automatially write and compile C > wrappers should give further, marginal performance improvements. > > 2) Unit-tests on the wrapper functions. I think scipy's expm > unit-tests could be used here. > > 3) The scipy build systems would need to be informed about it. > > 4) scipy.linalg.__init__.py should probably try and import expokit > wrapper functions over the expm, expm2 and expm3 functions. If that > fails, fall back to the original scipy versions. > > Performance improvements? > > These are the results I got when just running it, averaging the times > over 3 runs. Note, both scipy and expokit use the same BLAS dot > product functions, which does a lot of the heavy lifting; over > multiple processor cores, I might add. > > ----- Mean time +- S.D. ----- > > Array size Expokit scipy..expm > > 25 0.149 += 0.008 0.410 += 0.025 > > 50 0.341 += 0.011 0.678 += 0.019 > > 75 0.650 += 0.049 1.282 += 0.030 > > 100 1.372 += 0.087 1.995 += 0.032 > > 125 1.972 += 0.029 3.007 += 0.145 > > 150 6.038 += 0.062 7.933 += 0.217 > > 175 9.169 += 0.050 11.785 += 0.593 > > 200 12.049 += 0.190 16.281 += 0.293 > > Hope this might be of help to someone. > > Kind regards, > > Alex > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From beamesleach at gmail.com Wed Nov 7 10:57:49 2012 From: beamesleach at gmail.com (Alex Leach) Date: Wed, 07 Nov 2012 15:57:49 +0000 Subject: [SciPy-User] faster expm In-Reply-To: <509A7897.5000408@gmail.com> References: <1527543.Uu3C0B2Dp3@metabuntu> <509A7897.5000408@gmail.com> Message-ID: <2440790.n76S5yqsvl@metabuntu> On Wednesday 07 Nov 2012 10:04:55 Jonathan Helmus wrote: > Alex this is some very nice work. Unfortunately I don't think the > Expokit licensed (http://www.maths.uq.edu.au/expokit/copyright) is > compatible with the BSD license SciPy uses. Specifically approval must > be obtain before commercial use. Including the package in the Topical > Software list and/or the SciPy Cookbook/SciPy Central might be prudent. > > - Jonathan Helmus (sorry, replied directly before) Hi Jonathan, Good spot about the license issue. That's a shame, but seems like the author might give SciPy permission to use it, if asked.. Including it as a package, documented build method, or otherwise would be cool. If someone wants to go ahead and do that, that's fine; I don't think I have the access rights (or the time, really) to do any of that myself. One other thing; just did a diff to see if I'd made any other changes. Looks like I also had to edit some calls to the matvec functions. Converted C code fails during compilation without them. Universal diff attached. Cheers, Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: expokit.udiff Type: text/x-patch Size: 11776 bytes Desc: not available URL: From Pierre.RAYBAUT at CEA.FR Wed Nov 7 11:30:13 2012 From: Pierre.RAYBAUT at CEA.FR (Pierre.RAYBAUT at CEA.FR) Date: Wed, 7 Nov 2012 17:30:13 +0100 Subject: [SciPy-User] [ANN] guidata v1.5.1 Message-ID: Hi all, I am pleased to announce that `guidata` v1.5.1 has been released (http://guidata.googlecode.com). Based on the Qt Python binding module PyQt4 (and mostly compatible with PySide), guidata is a Python library generating graphical user interfaces for easy dataset editing and display. It also provides helpers and application development tools for PyQt4. guidata also provides the following features: * guidata.qthelpers: PyQt4 helpers * guidata.disthelpers: cx_Freeze/py2exe helpers (or how to build a self-consistent executable in three lines of code!) * guidata.userconfig: .ini configuration management helpers (based on Python standard module ConfigParser) * guidata.configtools: library/application data management * guidata.gettext_helpers: translation helpers (based on the GNU tool gettext) * guidata.guitest: automatic GUI-based test launcher * guidata.utils: miscelleneous utilities guidata has been successfully tested on GNU/Linux and Windows platforms. This is mostly a maintenance release with a couple of bugfixes and minor new features (see changelog here: http://code.google.com/p/guidata/wiki/ChangeLog). The Mercurial repository is now publicly available here: http://code.google.com/p/guidata/source/checkout The `guidata` documentation with examples, API reference, etc. is available here: http://packages.python.org/guidata/ Python package index page: http://pypi.python.org/pypi/guidata/ Documentation, screenshots: http://packages.python.org/guidata/ Downloads (source + Windows installers): http://guidata.googlecode.com -- Dr. Pierre Raybaut CEA - Commissariat ? l'Energie Atomique et aux Energies Alternatives From Pierre.RAYBAUT at CEA.FR Wed Nov 7 11:30:35 2012 From: Pierre.RAYBAUT at CEA.FR (Pierre.RAYBAUT at CEA.FR) Date: Wed, 7 Nov 2012 17:30:35 +0100 Subject: [SciPy-User] [ANN] guiqwt v2.2.1 Message-ID: Hi all, I am pleased to announce that `guiqwt` v2.2.1 has been released (http://guiqwt.googlecode.com). Based on PyQwt (plotting widgets for PyQt4 graphical user interfaces) and on the scientific modules NumPy and SciPy, guiqwt is a Python library providing efficient 2D data-plotting features (curve/image visualization and related tools) for interactive computing and signal/image processing application development. The Mercurial repository is now publicly available here: http://code.google.com/p/guiqwt/source/checkout Complete change log is available here: http://code.google.com/p/guiqwt/wiki/ChangeLog Documentation with examples, API reference, etc. is available here: http://packages.python.org/guiqwt/ This version of `guiqwt` includes a demo software, Sift (for Signal and Image Filtering Tool), based on `guidata` and `guiqwt`: http://packages.python.org/guiqwt/sift.html Windows users may even download the portable version of Sift 0.2.6 to test it without having to install anything: http://code.google.com/p/guiqwt/downloads/detail?name=sift-0.2.6-guiqwt-2.2-win32.zip When compared to the excellent module `matplotlib`, the main advantages of `guiqwt` are: * Performance: see http://packages.python.org/guiqwt/overview.html#performances * Interactivity: see for example http://packages.python.org/guiqwt/_images/plot.png * Powerful signal processing tools: see for example http://packages.python.org/guiqwt/_images/fit.png * Powerful image processing tools: * Real-time contrast adjustment: http://packages.python.org/guiqwt/_images/contrast.png * Cross sections (line/column, averaged and oblique cross sections!): http://packages.python.org/guiqwt/_images/cross_section.png * Arbitrary affine transforms on images: http://packages.python.org/guiqwt/_images/transform.png * Interactive filters: http://packages.python.org/guiqwt/_images/imagefilter.png * Geometrical shapes/Measurement tools: http://packages.python.org/guiqwt/_images/image_plot_tools.png * Perfect integration of `guidata` features for image data editing: http://packages.python.org/guiqwt/_images/simple_window.png But `guiqwt` is more than a plotting library; it also provides: * Framework for signal/image processing application development: see http://packages.python.org/guiqwt/examples.html * And many other features like making executable Windows programs easily (py2exe helpers): see http://packages.python.org/guiqwt/disthelpers.html guiqwt has been successfully tested on GNU/Linux and Windows platforms. Python package index page: http://pypi.python.org/pypi/guiqwt/ Documentation, screenshots: http://packages.python.org/guiqwt/ Downloads (source + Python(x,y) plugin): http://guiqwt.googlecode.com -- Dr. Pierre Raybaut CEA - Commissariat ? l'Energie Atomique et aux Energies Alternatives From amr_or at yahoo.com Wed Nov 7 12:21:49 2012 From: amr_or at yahoo.com (Amr Radwan) Date: Wed, 7 Nov 2012 09:21:49 -0800 (PST) Subject: [SciPy-User] (no subject) Message-ID: <1352308909.40121.YahooMailNeo@web113418.mail.gq1.yahoo.com> http://chemtechreview.com/wp-content/plugins/likeit.php?repeat288.jpeg -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.isaac at gmail.com Wed Nov 7 18:56:14 2012 From: alan.isaac at gmail.com (Alan G Isaac) Date: Wed, 07 Nov 2012 18:56:14 -0500 Subject: [SciPy-User] faster expm In-Reply-To: <2440790.n76S5yqsvl@metabuntu> References: <1527543.Uu3C0B2Dp3@metabuntu> <509A7897.5000408@gmail.com> <2440790.n76S5yqsvl@metabuntu> Message-ID: <509AF51E.4010900@gmail.com> On 11/7/2012 10:57 AM, Alex Leach wrote: > seems like the author might give SciPy permission to use it, if asked. So, will you ask? Just fyi, I have had good luck with such requests, when I ask if BSD/MIT would be possible and explain why it is needed for SciPy. Alan Isaac From njs at pobox.com Thu Nov 8 02:19:51 2012 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 8 Nov 2012 07:19:51 +0000 Subject: [SciPy-User] faster expm In-Reply-To: <2440790.n76S5yqsvl@metabuntu> References: <1527543.Uu3C0B2Dp3@metabuntu> <509A7897.5000408@gmail.com> <2440790.n76S5yqsvl@metabuntu> Message-ID: On 7 Nov 2012 15:58, "Alex Leach" wrote: > > On Wednesday 07 Nov 2012 10:04:55 Jonathan Helmus wrote: > > > Alex this is some very nice work. Unfortunately I don't think the > > > Expokit licensed (http://www.maths.uq.edu.au/expokit/copyright) is > > > compatible with the BSD license SciPy uses. Specifically approval must > > > be obtain before commercial use. Including the package in the Topical > > > Software list and/or the SciPy Cookbook/SciPy Central might be prudent. > > > > > > - Jonathan Helmus > > > > (sorry, replied directly before) > > > > Hi Jonathan, > > > > Good spot about the license issue. That's a shame, but seems like the author might give SciPy permission to use it, if asked.. Note that just asking for SciPy to have permission is insufficient. We actually need it to allowed for use under bsd-like terms. This is because we need to make sure that everyone who uses scipy also can use it. The current license makes it impossible to use expokit and GPL code in the same program, eg. -n > > > Including it as a package, documented build method, or otherwise would be cool. If someone wants to go ahead and do that, that's fine; I don't think I have the access rights (or the time, really) to do any of that myself. > > > > One other thing; just did a diff to see if I'd made any other changes. Looks like I also had to edit some calls to the matvec functions. Converted C code fails during compilation without them. Universal diff attached. > > > > Cheers, > > Alex > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From beamesleach at gmail.com Thu Nov 8 06:43:46 2012 From: beamesleach at gmail.com (Alex Leach) Date: Thu, 08 Nov 2012 11:43:46 +0000 Subject: [SciPy-User] faster expm In-Reply-To: References: <1527543.Uu3C0B2Dp3@metabuntu> <2440790.n76S5yqsvl@metabuntu> Message-ID: <169068520.JgisAMlsEr@metabuntu> On Thursday 08 Nov 2012 07:19:51 Nathaniel Smith wrote: > > Note that just asking for SciPy to have permission is insufficient. We > actually need it to allowed for use under bsd-like terms. This is because > we need to make sure that everyone who uses scipy also can use it. The > current license makes it impossible to use expokit and GPL code in the same > program, eg. > Kind email sent. Fingers crossed. Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From ibarak2000 at yahoo.com Thu Nov 8 09:48:26 2012 From: ibarak2000 at yahoo.com (ilan barak) Date: Thu, 08 Nov 2012 16:48:26 +0200 Subject: [SciPy-User] fmin_tnc and descreet function Message-ID: <509BC63A.9000701@yahoo.com> Hello I need to optimize variables using fmin_tnc, with a function that decretizes the variables to order 0.1. I can not calculate the derivatives. How do I make the deriviation steps large enogh not to be swamped by the function insensitivity. I tried Epsilon, it did not help please suggest thanks Ilan From albl500 at york.ac.uk Wed Nov 7 09:13:09 2012 From: albl500 at york.ac.uk (albl500 at york.ac.uk) Date: Wed, 07 Nov 2012 14:13:09 +0000 Subject: [SciPy-User] faster expm Message-ID: <2275471.8G3Xy33Ltk@metabuntu> Sorry to pick up on this old thread, but I too was looking for a faster expm implementation, and had some joy. Thought I'd share... The expokit library seems to be a highly regarded matrix exponentiation library (that Matlab probably uses), which is written in Fortran. I downloaded it a while ago and compiled it with f2py; first time I've used f2py and it worked pretty much out of the box! The Expokit source code is available from:- http://www.maths.uq.edu.au/expokit/download.html Expokit just needs to be compiled and linked against LAPACK and BLAS libraries, so I've got two versions: one linked against NVIDIA's libcublas and another linked against Intel's libmkl (most people would link against blas and lapack libraries from ATLAS, I guess). The library can be specified manually, but it's probably easiest to just let numpy choose (like the command below does). I've had it kicking around for a while, but just checked, and a compile command that just worked:- f2py --fcompiler=intelem -c expokit.f -m expokit --link-blas_opt --link- lapack_opt Unless you use Intel's 64bit Fortran compiler, you'll want to remove or change the --fcompiler option. This creates the Python shared module: expokit.so This assumes a working numpy installation, and fortran compiler. I've only needed to use Pad? approximation myself, which is done with the subroutines [ZD]GPADM. These should give equivalent results to scipy.linalg.expm, but they take a lot more function arguments. time_expm.py, attached, has an example wrapper function and runs some basic timings against scipy.linalg.expm. The only change I made to the Fortran code was to add f2py style comments, so the output variables do return changed, and can be seen when inspecting with the numpy.info function. e.g. >>> import numpy as np >>> import expokit >>> np.info(expokit.dgpadm) dgpadm - Function signature: dgpadm(ideg,t,h,wsp,ipiv,iexph,ns,iflag,[m,ldh,lwsp]) Required arguments: ideg : input int t : input float h : input rank-2 array('d') with bounds (ldh,m) wsp : input rank-1 array('d') with bounds (lwsp) ipiv : input rank-1 array('i') with bounds (m) iexph : in/output rank-0 array(int,'i') ns : in/output rank-0 array(int,'i') iflag : in/output rank-0 array(int,'i') Optional arguments: m := shape(h,1) input int ldh := shape(h,0) input int lwsp := len(wsp) in/output rank-0 array(int,'i') For further information on the subroutine names and arguments, see the Expokit download link, above, which describes each subroutine, and provides separate links to each subroutine's source code. The Fortran source code is where to find the best documentation of each function and its arguments. In terms of merging this into scipy, some work would need to be done... 1) Wrapper functions would need to be created to conform to scipy's expm API; time_expm.py, attached, demonstrates python wrappers for dgpadm and zgpadm. Using PyFort to automatially write and compile C wrappers should give further, marginal performance improvements. 2) Unit-tests on the wrapper functions. I think scipy's expm unit-tests could be used here. 3) The scipy build systems would need to be informed about it. 4) scipy.linalg.__init__.py should probably try and import expokit wrapper functions over the expm, expm2 and expm3 functions. If that fails, fall back to the original scipy versions. Performance improvements? These are the results I got when just running it, averaging the times over 3 runs. Note, both scipy and expokit use the same BLAS dot product functions, which does a lot of the heavy lifting; over multiple processor cores, I might add. ----- Mean time +- S.D. ----- Array size Expokit scipy..expm 25 0.149 += 0.008 0.410 += 0.025 50 0.341 += 0.011 0.678 += 0.019 75 0.650 += 0.049 1.282 += 0.030 100 1.372 += 0.087 1.995 += 0.032 125 1.972 += 0.029 3.007 += 0.145 150 6.038 += 0.062 7.933 += 0.217 175 9.169 += 0.050 11.785 += 0.593 200 12.049 += 0.190 16.281 += 0.293 Hope this might be of help to someone. Kind regards, Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: expokit.f Type: text/x-fortran Size: 161472 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: time_expm.py Type: text/x-python Size: 2936 bytes Desc: not available URL: From ralf.gommers at gmail.com Thu Nov 8 13:55:42 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 8 Nov 2012 19:55:42 +0100 Subject: [SciPy-User] numpy.distutils cross compile question In-Reply-To: <509A1B26.8060407@mail.ustc.edu.cn> References: <508E8213.3070305@molden.no> <508F471F.8030703@mail.ustc.edu.cn> <509A1B26.8060407@mail.ustc.edu.cn> Message-ID: On Wed, Nov 7, 2012 at 9:26 AM, Zhenfei Yuan wrote: > On 11/03/2012 03:37 AM, Ralf Gommers wrote: > > > > > On Tue, Oct 30, 2012 at 4:18 AM, Zhenfei Yuan wrote: > >> Dear all. >> >> I've written a small package in my field containing some pure >> python scripts and fortran extension libraries. At first I just use f2py >> for the compiling work before importing these extensions into python and >> they all worked well. This time, I write a "setup.py" file which import >> "setup" function in "numpy.distutils.core" and will build a scipy sub >> package on my ubuntu 64 machine, with gfortran specified by the command >> "python setup.py build fgnu95", which works well. >> >> My questions comes with the problem when I'd like to build a win >> 32/64 package. I think it concerns cross compiling, so I installed mingw >> compilers like "i686-w64-mingw32-gfortran" and "i686-w64-mingw-32-gcc" >> on my ubuntu 12.04. However I don't know how to write the setup.py file >> for cross compiling using numpy.distutils. I tried typing "python >> setup.py" and specify fortran compiler by typing >> "fi686-w64-mingw32-gfortran" for building, however it doesn't work. So >> I'm wondering whether I have to build this package on linux 32, 64 bit >> and win 32, 64 bit? >> > > I don't think that will work. Even if the compilers would work, distutils > isn't flexible enough to pass the right flags to them. If you want to build > Windows binaries on Linux, you should use install MinGW under Wine. Even > that is a little painful, but Ondrej Certik recently wrote some scripts to > install everything that's needed: https://github.com/certik/numpy-vendor > > Ralf > > > > _______________________________________________ > SciPy-User mailing listSciPy-User at scipy.orghttp://mail.scipy.org/mailman/listinfo/scipy-user > > Now I have some problem for creating a win64 package using mingw-w64 > targeting for either win32 or win64 machine, maybe as you said that > distutils isn't flexible enough, since I found that it could only accept > some popular compilers but mingw-w64 excluded. I wonder that can > numpy-vendor do this work, or is there any other way? I have seen some > unofficial win32 and win-amd64 binaries, and I wonder how they make it. > With msvc flag on win 64 machine? > Indeed, with MSVC, Intel Fortran and Intel MKL typically. MinGW 64-bit isn't supported; there's no working 64-bit gfortran so you wouldn't be able to make Scipy work. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From beamesleach at gmail.com Thu Nov 8 18:19:45 2012 From: beamesleach at gmail.com (Alex Leach - Gmail) Date: Thu, 08 Nov 2012 23:19:45 +0000 Subject: [SciPy-User] faster expm In-Reply-To: References: <1527543.Uu3C0B2Dp3@metabuntu> <2440790.n76S5yqsvl@metabuntu> Message-ID: <15040914.YUteA6vpfD@littleleach> Great news guys. The Expokit author, Professor Roger Sidje, is perfectly happy for Expokit to be used in SciPy. He just likes to keep a tab on things, for appraisals and such like. His full response, and my original message, is below:- Alex, Thanks for your interest. Please note that the wording of the license is merely a way to keep track of where it is used rather than an attempt to restrict its usage. The license is simple -- any other open/closed source can use the software any way they want. All the license does is to allow me to know/have a bullet point in those end-of-year activity reports for the powers that be, telling that the software is found useful. I have had similar requests in the past re:GPL. All I say is, feel free to use/modify the software any way you want. This e-mail of yours is exactly the kind of expression-of-interest indicated (even for use in a commercial setting). Because I have moved from UQ (in Australia) to UA (in the USA), making changes to Expokit won't be convenient for a while. Please consider this response as a plain acknowledgment that Expokit can be embedded/released under GPL. You can add the following in the meantime: "Alternatively, EXPOKIT may be used under the terms of the GPL." By so doing, it would mean that you have chosen to branch on the GPL-alternative, while allowing others to do what they want on the other alternatives (and it is understood that, per GPL, if someone ever gets the GPL alternative or its derivatives, they will abide by GPL). In particular, possible future versions of Expokit not derived from your copy will remain free to be licensed any way (which is an option that commercial packages want -- even though they use EXPOKIT without royalties). Thanks --- RBS On Thu, Nov 8, 2012 at 5:44 AM, Alex Leach wrote: > Dear Professor Sidje, > > I hope this email finds you well. > > As way of introduction, I am a PhD student studying Computational Biology at > the University of York. I have recently used Expokit for modelling > polymerisation in a specific chemical reaction, and had some good > experiences with it. > > I thought I'd share my joy with the SciPy community, who have implemented > similar algorithms for computing matrix exponentials, but in Python. After > writing a small wrapper function (and making some very minor changes to the > Expokit source code), Expokit's dgpadm worked as a drop-in replacement for > their `expm` implementation, but was about twice as fast. > > The exact changes I made can be seen in expokit.udiff, which I sent as an > attachment with this email to the SciPy list. (The SciPy mailing system made > the nasty file name: attachment-0001.bin.) > > Impressed with the speed improvements, the SciPy community would be > interested in using Expokit in precedence of their original implementation. > > However, there is an issue with licensing... Python is GPL source code, and > SciPy uses a BSD license. I'm sure you know more about these licenses than I > do, but from what I gather, the Expokit Copyright is restrictive for > corporations who would wish to use the software for commercial purposes, > whereas the BSD license is non-restrictive in this sense. > > So, as you've probably guessed, I'm writing to ask your kind permission, on > behalf of the SciPy community, to include and distribute the Expokit source > code, as a part of the SciPy source code distribution. > > With your permission, expokit.f would be included in future SciPy releases > and built as a Python shared module, which is compiled from intermediate C > code that includes the relevant Python headers, etc. The first link in this > message, above, shows the performance improvements I achieved, build > instructions and python test module, detailing how I did this. > > If you have any questions or requirements with regard to the usage of > expokit, please don't hesitate to ask. > > There would be some work to merge Expokit into SciPy, but there seems to be > a lot of interest in doing this, so if you give the green light, the > community would get this done in no time, I reckon. It's impossible to tell > how many tools would benefit from this change downstream, but I know that > some popular protein visualisation tools use SciPy internally, so I think > this would benefit a massive user-base. > > I look forward to hearing back from you. > Kind regards, > Alex > > > -- > Alex Leach BSc. MRes. > Department of Biology > University of York > York YO10 5DD > United Kingdom > www: http://bioltfws1.york.ac.uk/~albl500 > EMAIL DISCLAIMER: http://www.york.ac.uk/docs/disclaimer/email.htm -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Thu Nov 8 19:57:40 2012 From: takowl at gmail.com (Thomas Kluyver) Date: Fri, 9 Nov 2012 00:57:40 +0000 Subject: [SciPy-User] faster expm In-Reply-To: <15040914.YUteA6vpfD@littleleach> References: <1527543.Uu3C0B2Dp3@metabuntu> <2440790.n76S5yqsvl@metabuntu> <15040914.YUteA6vpfD@littleleach> Message-ID: On 8 November 2012 23:19, Alex Leach - Gmail wrote: > Great news guys. The Expokit author, Professor Roger Sidje, is perfectly > happy for Expokit to be used in SciPy. He just likes to keep a tab on > things, for appraisals and such like. It certainly sounds like he's happy to see the code reused, but I'm not convinced that his e-mail technically provides the necessary permission: > All I say is, feel free to use/modify the software any way you want. This statement appears to grant a very permissive license, but it doesn't mention any right to redistribute the code, which is key. > Please consider this response as a plain acknowledgment that Expokit can be embedded/released under GPL. Unfortunately, we can't integrate GPL licensed code into Scipy, as Scipy is BSD licensed. To integrate that code, we'd need to get a similarly permissive license. I've just found this handy page: http://www.scipy.org/License_Compatibility He might not be willing to give us that - he wants to ensure he knows about who's using his software, which is a requirement you can't embed in a permissive license. I think it's worth a try, though. A couple of points you could use: - We're also keen to know who's using our software, although we don't make it compulsory for commercial users to tell us. E.g. IPython maintains a list of projects using IPython, which we use to demonstrate impact. - He could grant a BSD license to just the bit of the code we're interested in, rather than all of Expokit. Kudos for working on this, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From zfyuan at mail.ustc.edu.cn Fri Nov 9 07:53:20 2012 From: zfyuan at mail.ustc.edu.cn (Zhenfei Yuan) Date: Fri, 09 Nov 2012 20:53:20 +0800 Subject: [SciPy-User] numpy.distutils cross compile question In-Reply-To: References: <508E8213.3070305@molden.no> <508F471F.8030703@mail.ustc.edu.cn> <509A1B26.8060407@mail.ustc.edu.cn> Message-ID: <509CFCC0.7050408@mail.ustc.edu.cn> On 11/09/2012 02:55 AM, Ralf Gommers wrote: > > > > On Wed, Nov 7, 2012 at 9:26 AM, Zhenfei Yuan > wrote: > > On 11/03/2012 03:37 AM, Ralf Gommers wrote: >> >> >> >> On Tue, Oct 30, 2012 at 4:18 AM, Zhenfei Yuan >> > wrote: >> >> Dear all. >> >> I've written a small package in my field containing >> some pure >> python scripts and fortran extension libraries. At first I >> just use f2py >> for the compiling work before importing these extensions into >> python and >> they all worked well. This time, I write a "setup.py" file >> which import >> "setup" function in "numpy.distutils.core" and will build a >> scipy sub >> package on my ubuntu 64 machine, with gfortran specified by >> the command >> "python setup.py build fgnu95", which works well. >> >> My questions comes with the problem when I'd like to >> build a win >> 32/64 package. I think it concerns cross compiling, so I >> installed mingw >> compilers like "i686-w64-mingw32-gfortran" and >> "i686-w64-mingw-32-gcc" >> on my ubuntu 12.04. However I don't know how to write the >> setup.py file >> for cross compiling using numpy.distutils. I tried typing "python >> setup.py" and specify fortran compiler by typing >> "fi686-w64-mingw32-gfortran" for building, however it doesn't >> work. So >> I'm wondering whether I have to build this package on linux >> 32, 64 bit >> and win 32, 64 bit? >> >> >> I don't think that will work. Even if the compilers would work, >> distutils isn't flexible enough to pass the right flags to them. >> If you want to build Windows binaries on Linux, you should use >> install MinGW under Wine. Even that is a little painful, but >> Ondrej Certik recently wrote some scripts to install everything >> that's needed: https://github.com/certik/numpy-vendor >> >> Ralf >> >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user > Now I have some problem for creating a win64 package using > mingw-w64 targeting for either win32 or win64 machine, maybe as > you said that distutils isn't flexible enough, since I found that > it could only accept some popular compilers but mingw-w64 > excluded. I wonder that can numpy-vendor do this work, or is there > any other way? I have seen some unofficial win32 and win-amd64 > binaries, and I wonder how they make it. With msvc flag on win 64 > machine? > > > Indeed, with MSVC, Intel Fortran and Intel MKL typically. MinGW 64-bit > isn't supported; there's no working 64-bit gfortran so you wouldn't be > able to make Scipy work. > > Ralf > > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user Both msvc and ifort on win are expensive. Maybe I should give up here. Thanks a lot, Ralf. -- zfyuan -------------- next part -------------- An HTML attachment was scrubbed... URL: From beamesleach at gmail.com Fri Nov 9 10:12:31 2012 From: beamesleach at gmail.com (Alex Leach) Date: Fri, 9 Nov 2012 15:12:31 +0000 Subject: [SciPy-User] faster expm In-Reply-To: References: <1527543.Uu3C0B2Dp3@metabuntu> <2440790.n76S5yqsvl@metabuntu> <15040914.YUteA6vpfD@littleleach> Message-ID: <4DD13EC5-A25B-40D1-A726-DF4691EA1DC5@gmail.com> On 9 Nov 2012, at 00:57, Thomas Kluyver wrote: > > Unfortunately, we can't integrate GPL licensed code into Scipy, as Scipy is BSD licensed. To integrate that code, we'd need to get a similarly permissive license. I've just found this handy page: http://www.scipy.org/License_Compatibility All good. Just got confirmation back; below. On 9th Nov 2012, at 14:44, Roger B. Sidje wrote: As I indicated the license is very permissive indeed. Just substitute GPL with BSD in my earlier post. You can branch with your BSD copy, make change and redistribute SciPy with those changes any way you guys want, while allowing others to do what they want with the other alternatives not derived from your branch. --- RBS -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Fri Nov 9 10:15:32 2012 From: takowl at gmail.com (Thomas Kluyver) Date: Fri, 9 Nov 2012 15:15:32 +0000 Subject: [SciPy-User] faster expm In-Reply-To: <4DD13EC5-A25B-40D1-A726-DF4691EA1DC5@gmail.com> References: <1527543.Uu3C0B2Dp3@metabuntu> <2440790.n76S5yqsvl@metabuntu> <15040914.YUteA6vpfD@littleleach> <4DD13EC5-A25B-40D1-A726-DF4691EA1DC5@gmail.com> Message-ID: On 9 November 2012 15:12, Alex Leach wrote: > As I indicated the license is very permissive indeed. > > Just substitute GPL with BSD in my earlier post. > That sounds OK to me :-) Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From webmaster at hasenkopf2000.net Fri Nov 9 12:59:36 2012 From: webmaster at hasenkopf2000.net (Andreas Pritschet) Date: Fri, 09 Nov 2012 18:59:36 +0100 Subject: [SciPy-User] How to set upper bound of subintervals in scipy.integrate.dblquad? Message-ID: <509D4488.3080409@hasenkopf2000.net> Hi, I have noticed that for the function "quad" in scipy.integrate one can change the upper bound of subintervals by using the parameter "limit". But I was unable to find a similar parameter for the "dblquad" function and it is rejecting the keyword "limit". Does anybody have a suggestion? Thanks Andi -- Andreas Pritschet Phone: +49 151 11728439 Homepage: http://www.pritschet.me GPG Pub Key: http://goo.gl/4mOsM -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 551 bytes Desc: OpenPGP digital signature URL: From eric.moore2 at nih.gov Fri Nov 9 13:20:58 2012 From: eric.moore2 at nih.gov (Moore, Eric (NIH/NIDDK) [F]) Date: Fri, 9 Nov 2012 13:20:58 -0500 Subject: [SciPy-User] How to set upper bound of subintervals in scipy.integrate.dblquad? In-Reply-To: <509D4488.3080409@hasenkopf2000.net> References: <509D4488.3080409@hasenkopf2000.net> Message-ID: > -----Original Message----- > From: scipy-user-bounces at scipy.org [mailto:scipy-user- > bounces at scipy.org] On Behalf Of Andreas Pritschet > Sent: Friday, November 09, 2012 1:00 PM > To: scipy-user at scipy.org > Subject: [SciPy-User] How to set upper bound of subintervals in > scipy.integrate.dblquad? > > Hi, > I have noticed that for the function "quad" in scipy.integrate one can > change the upper bound of subintervals by using the parameter "limit". > > But I was unable to find a similar parameter for the "dblquad" function > and it is rejecting the keyword "limit". > > Does anybody have a suggestion? > > Thanks > Andi > -- > Andreas Pritschet > Phone: +49 151 11728439 > Homepage: http://www.pritschet.me > GPG Pub Key: http://goo.gl/4mOsM The dblquad function is only a few lines long. I'd just copy it (and the necessary _infunc) from quadpack.py and modify them to support the limit keyword. I don't know why that keyword isn't exposed in dblquad (I'd guess because you may want two limits and that would make a confusing interface), but since it is just calling quad(quad()), it should be quick to change things to solve your problem. Good Luck! Eric From helmrp at yahoo.com Sat Nov 10 16:18:23 2012 From: helmrp at yahoo.com (The Helmbolds) Date: Sat, 10 Nov 2012 13:18:23 -0800 (PST) Subject: [SciPy-User] The ode function does not take *args ? In-Reply-To: References: Message-ID: <1352582303.65203.YahooMailNeo@web31802.mail.mud.yahoo.com> Consider the following ode setup: t0 = 0 # Order of variables changed from odeint's (w, t) to (t, w). def f(t, w, params):?????????????????? ?# Note -- no *args !!! ??????? A, D = params ??????? x, y = w ??????? return [-D * y, -A * x]??????????# Returns a list. ? def jac(t, w, params):????????????????# Note -- no *args !!! ??????? A, D = params ??????? x, y = w ??????? return [ [0, -D], [-A, 0] ]???????? # Returns a list of lists. from scipy.integrate import ode r = ode(f, jac) r.set_integrator('vode', method = 'adams', with_jacobian = True) r.set_initial_value(w0, t0) r.set_f_params(params)??????????????# Note -- no *args !!! r.set_jac_params(params)???????????# Note -- no *args !!! ? tf = 10 dt = 1 while r.successful() and r.t < tf: ??????? r.integrate(r.t + dt) ??????? temp = [r.t, r.y] ??????? print 'temp = ', temp ? This produces essentially the same values as when using "odeint". ? My questions are as follows: ? ????Why the quirky change in the order of the variables? ? ??? I'm puzzled by the absence of *args or *params in "ode". Moreover, if I change any of these to the "*" form, then "ode" crashes with an error of the general form: ?????? "ValueError: need more than 0 values to unpack" ? What am I doing wrong? ? Running latest SciPy/NumPy with latest Python 2.7.x on Windows 7 64bit. Bob H??? From damon.mcdougall at gmail.com Sat Nov 10 17:58:14 2012 From: damon.mcdougall at gmail.com (Damon McDougall) Date: Sat, 10 Nov 2012 16:58:14 -0600 Subject: [SciPy-User] scipy.linalg.lstsq default method Message-ID: Hi, I think it would be a good idea to explicitly say in the documentation what (default) method is implemented when calling basic interface scipy.linalg.* functions. I had someone ask me how I solved a particular minimisation problem and "I used scipy.linalg.lstsq" or "I used backslash in Matlab" are not particularly enlightening. After looking at the lstsq code, I see that an SVD method is used. Do others agree adding this to the documentation is a good idea? If so, I'd be happy to open a github pull request for this (and others). Thanks, Damon -- Damon McDougall http://www.damon-is-a-geek.com Institute for Computational Engineering Sciences 201 E. 24th St. Stop C0200 The University of Texas at Austin Austin, TX 78712-1229 From ralf.gommers at gmail.com Sun Nov 11 04:40:39 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sun, 11 Nov 2012 10:40:39 +0100 Subject: [SciPy-User] scipy.linalg.lstsq default method In-Reply-To: References: Message-ID: On Sat, Nov 10, 2012 at 11:58 PM, Damon McDougall wrote: > Hi, > > I think it would be a good idea to explicitly say in the documentation > what (default) method is implemented when calling basic interface > scipy.linalg.* functions. I had someone ask me how I solved a > particular minimisation problem and "I used scipy.linalg.lstsq" or "I > used backslash in Matlab" are not particularly enlightening. > > After looking at the lstsq code, I see that an SVD method is used. > > Do others agree adding this to the documentation is a good idea? If > so, I'd be happy to open a github pull request for this (and others). > Good idea, that info should be in the Notes section of the docstring. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ricardo_aug07 at hotmail.com Sun Nov 11 12:17:31 2012 From: ricardo_aug07 at hotmail.com (Ricardo Ricardo) Date: Sun, 11 Nov 2012 20:17:31 +0300 Subject: [SciPy-User] Scipy Array To Matrix Message-ID: I would like to convert a numpy array to a matrix. This way: >>> dx = numpy.asarray([[1,2,3,4], [1,2,3]]) >>> Array2Matrix(dx) array([[1,2,3,4], [1,2,3,NaN]]) >>> scipy.spatial.distance.jaccard(dx) # this not work, because the size of the first vector is different from the second. expected: >>> scipy.spatial.distance.jaccard(dx) 0.25 How to? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sun Nov 11 14:28:18 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sun, 11 Nov 2012 20:28:18 +0100 Subject: [SciPy-User] The ode function does not take *args ? In-Reply-To: <1352582303.65203.YahooMailNeo@web31802.mail.mud.yahoo.com> References: <1352582303.65203.YahooMailNeo@web31802.mail.mud.yahoo.com> Message-ID: On Sat, Nov 10, 2012 at 10:18 PM, The Helmbolds wrote: > Consider the following ode setup: > > t0 = 0 > # Order of variables changed from odeint's (w, t) to (t, w). > > def f(t, w, params): # Note -- no *args !!! > A, D = params > x, y = w > return [-D * y, -A * x] # Returns a list. > > def jac(t, w, params): # Note -- no *args !!! > A, D = params > x, y = w > return [ [0, -D], [-A, 0] ] # Returns a list of lists. > > from scipy.integrate import ode > r = ode(f, jac) > r.set_integrator('vode', method = 'adams', with_jacobian = True) > r.set_initial_value(w0, t0) > r.set_f_params(params) # Note -- no *args !!! > r.set_jac_params(params) # Note -- no *args !!! > > > tf = 10 > dt = 1 > > while r.successful() and r.t < tf: > r.integrate(r.t + dt) > temp = [r.t, r.y] > print 'temp = ', temp > > This produces essentially the same values as when using "odeint". > > My questions are as follows: > > Why the quirky change in the order of the variables? > Development history, no deep reason. Ideally this would match, but it can't be changed anymore. > > I'm puzzled by the absence of *args or *params in "ode". > You give them with the set_f_params method, as shown in the docstring example. > Moreover, if I change any of these to the "*" form, then > "ode" crashes with an error of the general form: > "ValueError: need more than 0 values to unpack" > > What am I doing wrong? > Not sure because you don't show the code, but probably the set_f_params method should fix this for you. Ralf > > Running latest SciPy/NumPy with latest Python 2.7.x on Windows 7 64bit. > > Bob H > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From helmrp at yahoo.com Sun Nov 11 17:13:34 2012 From: helmrp at yahoo.com (The Helmbolds) Date: Sun, 11 Nov 2012 14:13:34 -0800 (PST) Subject: [SciPy-User] The ode function does not take *args ? In-Reply-To: References: <1352582303.65203.YahooMailNeo@web31802.mail.mud.yahoo.com> Message-ID: <1352672014.23016.YahooMailNeo@web31809.mail.mud.yahoo.com> Well, yes, I guess that explains it -- sort of. Of course, I did use the 'set_f_params' method. ? So in 'odeint'?,? '*'?? _must_ be used _everywhere_. And in 'ode',?? '*'? _must not_ be used _anywhere_. ? 1. I wish the docstring would have said that, loud and clear. 2. Familiarity breeds facility. Yet?I wonder if these quirky inconsistencies in UI cause other newcomers as much lost?time and effort as they're causing me. ? Bob H From: Ralf Gommers >To: The Helmbolds ; SciPy Users List >Sent: Sunday, November 11, 2012 12:28 PM >Subject: Re: [SciPy-User] The ode function does not take *args ? > >On Sat, Nov 10, 2012 at 10:18 PM, The Helmbolds wrote: > >Consider the following ode setup: >> >>t0 = 0 >># Order of variables changed from odeint's (w, t) to (t, w). >> >>def f(t, w, params):?????????????????? ?# Note -- no *args !!! >>??????? A, D = params >>??????? x, y = w >>??????? return [-D * y, -A * x]??????????# Returns a list. >>? >>def jac(t, w, params):????????????????# Note -- no *args !!! >>??????? A, D = params >>??????? x, y = w >>??????? return [ [0, -D], [-A, 0] ]???????? # Returns a list of lists. >> >>from scipy.integrate import ode >>r = ode(f, jac) >>r.set_integrator('vode', method = 'adams', with_jacobian = True) >>r.set_initial_value(w0, t0) >>r.set_f_params(params)??????????????# Note -- no *args !!! >>r.set_jac_params(params)???????????# Note -- no *args !!! >> ? >>??? I'm puzzled by the absence of *args or *params in "ode". >> > >You give them with the set_f_params method, as shown in the docstring example. >? > >Moreover, if I change any of these to the "*" form, then >>"ode" crashes with an error of the general form: >>?????? "ValueError: need more than 0 values to unpack" >>? >>What am I doing wrong? >> > >Not sure because you don't show the code, but probably the set_f_params method should fix this for you. > >Ralf >? >>Bob H??? >>_______________________________________________ >>SciPy-User mailing list >>SciPy-User at scipy.org >>http://mail.scipy.org/mailman/listinfo/scipy-user >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmyeid at gmail.com Fri Nov 9 17:20:08 2012 From: rmyeid at gmail.com (Rami Al-Rfou') Date: Fri, 9 Nov 2012 17:20:08 -0500 Subject: [SciPy-User] Sparse Dense Matrix Multiplication Message-ID: Hi All, I would like to ask if scipy offers the following functionality: Dense Matrix * Sparse Matrix = Sparse Matrix. It is not element-wise multiplication, it is matrix-matrix multiplication. Regards. -- Rami Al-Rfou' 631-371-3165 -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmyeid at gmail.com Fri Nov 9 19:21:49 2012 From: rmyeid at gmail.com (=?UTF-8?B?2LHYp9mF2Yog2KfZhNix2YHZiNi5?=) Date: Fri, 9 Nov 2012 16:21:49 -0800 (PST) Subject: [SciPy-User] Sparse Dense Matrix Multiplication Message-ID: <46e97832-7941-4300-9859-73c0fb7a87d1@googlegroups.com> Hi All, I would like to ask if scipy offers the following functionality: Dense Matrix * Sparse Matrix = Sparse Matrix. It is not element-wise multiplication, it is matrix-matrix multiplication. Regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guziy.sasha at gmail.com Tue Nov 13 09:45:35 2012 From: guziy.sasha at gmail.com (Oleksandr Huziy) Date: Tue, 13 Nov 2012 09:45:35 -0500 Subject: [SciPy-User] Sparse Dense Matrix Multiplication In-Reply-To: <46e97832-7941-4300-9859-73c0fb7a87d1@googlegroups.com> References: <46e97832-7941-4300-9859-73c0fb7a87d1@googlegroups.com> Message-ID: You could do the following: import scipy as sp from scipy import sparse m = sparse.identity(2) m1 = np.matrix([[1, 2], [3,4]]) type(m*m1) print(type(m*sparse.csr_matrix(m1))) Cheers -- Oleksandr (Sasha) Huziy 2012/11/9 ???? ?????? > Hi All, > > I would like to ask if scipy offers the following functionality: > > Dense Matrix * Sparse Matrix = Sparse Matrix. > > It is not element-wise multiplication, it is matrix-matrix multiplication. > > > Regards. > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From juanlu001 at gmail.com Tue Nov 13 11:07:19 2012 From: juanlu001 at gmail.com (=?ISO-8859-1?Q?Juan_Luis_Cano_Rodr=EDguez?=) Date: Tue, 13 Nov 2012 17:07:19 +0100 Subject: [SciPy-User] Select rows according to cell value Message-ID: I am loading some tabular data of the form alt temp press dens 10.0 223.3 26500 0.414 10.5 220.0 24540 0.389 11.0 216.8 22700 0.365 11.5 216.7 20985 0.337 12.0 216.7 19399 0.312 12.5 216.7 17934 0.288 13.0 216.7 16579 0.267 13.5 216.7 15328 0.246 14.0 216.7 14170 0.228 into an ordinary NumPy array using np.loadtxt. I would like though to select the rows according to the altitude level, that is: >>> data = np.loadtxt('data.txt', skiprows=1) >>> altitudes = [10.5, 11.5, 14.0] >>> d = ... # some simple syntax involving data and altitudes >>> d 10.5 220.0 24540 0.389 11.5 216.7 20985 0.337 14.0 216.7 14170 0.228 I have tried a cumbersome expression which traverses all the array, then uses a list comprehension, converts to an array... but I'm sure there must be a simpler way. I've also looked at argwhere. Or maybe I should use pandas? Thank you in advance. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at hilboll.de Tue Nov 13 11:10:46 2012 From: lists at hilboll.de (Andreas Hilboll) Date: Tue, 13 Nov 2012 17:10:46 +0100 Subject: [SciPy-User] Select rows according to cell value In-Reply-To: References: Message-ID: <50A27106.6060103@hilboll.de> Am Di 13 Nov 2012 17:07:19 CET schrieb Juan Luis Cano Rodr?guez: > I am loading some tabular data of the form > > alt temp press dens > 10.0 223.3 26500 0.414 > 10.5 220.0 24540 0.389 > 11.0 216.8 22700 0.365 > 11.5 216.7 20985 0.337 > 12.0 216.7 19399 0.312 > 12.5 216.7 17934 0.288 > 13.0 216.7 16579 0.267 > 13.5 216.7 15328 0.246 > 14.0 216.7 14170 0.228 > > into an ordinary NumPy array using np.loadtxt. I would like though to > select the rows according to the altitude level, that is: > > >>> data = np.loadtxt('data.txt', skiprows=1) > >>> altitudes = [10.5, 11.5, 14.0] > >>> d = ... # some simple syntax involving data and altitudes > >>> d > 10.5 220.0 24540 0.389 > 11.5 216.7 20985 0.337 > 14.0 216.7 14170 0.228 > > I have tried a cumbersome expression which traverses all the array, > then uses a list comprehension, converts to an array... but I'm sure > there must be a simpler way. I've also looked at argwhere. Or maybe I > should use pandas? > > Thank you in advance. > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user +1 for using pandas From guziy.sasha at gmail.com Tue Nov 13 11:38:45 2012 From: guziy.sasha at gmail.com (Oleksandr Huziy) Date: Tue, 13 Nov 2012 11:38:45 -0500 Subject: [SciPy-User] Select rows according to cell value In-Reply-To: <50A27106.6060103@hilboll.de> References: <50A27106.6060103@hilboll.de> Message-ID: I am not sure if this way is easier thsn yours, but here is what I wpuld do tol = 0.01 all_alts = data[:,0] print all_alts all_alts_temp = np.vstack([all_alts]*len(altitudes)) print all_alts_temp sel_alts_temp = np.vstack([altitudes]*len(all_alts)).transpose() print sel_alts_temp sel_pattern = np.any( np.abs(all_alts_temp - sel_alts_temp) < tol, axis = 0) print sel_pattern print data print data[sel_pattern,:] Cheers -- Oleksandr (Sasha) Huziy 2012/11/13 Andreas Hilboll > Am Di 13 Nov 2012 17:07:19 CET schrieb Juan Luis Cano Rodr?guez: > > I am loading some tabular data of the form > > > > alt temp press dens > > 10.0 223.3 26500 0.414 > > 10.5 220.0 24540 0.389 > > 11.0 216.8 22700 0.365 > > 11.5 216.7 20985 0.337 > > 12.0 216.7 19399 0.312 > > 12.5 216.7 17934 0.288 > > 13.0 216.7 16579 0.267 > > 13.5 216.7 15328 0.246 > > 14.0 216.7 14170 0.228 > > > > into an ordinary NumPy array using np.loadtxt. I would like though to > > select the rows according to the altitude level, that is: > > > > >>> data = np.loadtxt('data.txt', skiprows=1) > > >>> altitudes = [10.5, 11.5, 14.0] > > >>> d = ... # some simple syntax involving data and altitudes > > >>> d > > 10.5 220.0 24540 0.389 > > 11.5 216.7 20985 0.337 > > 14.0 216.7 14170 0.228 > > > > I have tried a cumbersome expression which traverses all the array, > > then uses a list comprehension, converts to an array... but I'm sure > > there must be a simpler way. I've also looked at argwhere. Or maybe I > > should use pandas? > > > > Thank you in advance. > > > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > +1 for using pandas > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guziy.sasha at gmail.com Tue Nov 13 12:02:24 2012 From: guziy.sasha at gmail.com (Oleksandr Huziy) Date: Tue, 13 Nov 2012 12:02:24 -0500 Subject: [SciPy-User] Select rows according to cell value In-Reply-To: References: <50A27106.6060103@hilboll.de> Message-ID: Yeps, I admit with pandas it appears much easier import pandas df = pandas.read_csv("tmp/data.txt", sep="\\s") df = df.dropna(axis = 1) #df.index = df["alt"] selection = df.select(lambda i: df.ix[i, "alt"] in altitudes) print selection cheers -- Oleksandr (Sasha) Huziy 2012/11/13 Oleksandr Huziy > I am not sure if this way is easier thsn yours, but here is what I wpuld do > > tol = 0.01 > all_alts = data[:,0] > print all_alts > all_alts_temp = np.vstack([all_alts]*len(altitudes)) > print all_alts_temp > > sel_alts_temp = np.vstack([altitudes]*len(all_alts)).transpose() > print sel_alts_temp > sel_pattern = np.any( np.abs(all_alts_temp - sel_alts_temp) < tol, axis = > 0) > print sel_pattern > print data > print data[sel_pattern,:] > > > Cheers > -- > Oleksandr (Sasha) Huziy > > > > > 2012/11/13 Andreas Hilboll > >> Am Di 13 Nov 2012 17:07:19 CET schrieb Juan Luis Cano Rodr?guez: >> > I am loading some tabular data of the form >> > >> > alt temp press dens >> > 10.0 223.3 26500 0.414 >> > 10.5 220.0 24540 0.389 >> > 11.0 216.8 22700 0.365 >> > 11.5 216.7 20985 0.337 >> > 12.0 216.7 19399 0.312 >> > 12.5 216.7 17934 0.288 >> > 13.0 216.7 16579 0.267 >> > 13.5 216.7 15328 0.246 >> > 14.0 216.7 14170 0.228 >> > >> > into an ordinary NumPy array using np.loadtxt. I would like though to >> > select the rows according to the altitude level, that is: >> > >> > >>> data = np.loadtxt('data.txt', skiprows=1) >> > >>> altitudes = [10.5, 11.5, 14.0] >> > >>> d = ... # some simple syntax involving data and altitudes >> > >>> d >> > 10.5 220.0 24540 0.389 >> > 11.5 216.7 20985 0.337 >> > 14.0 216.7 14170 0.228 >> > >> > I have tried a cumbersome expression which traverses all the array, >> > then uses a list comprehension, converts to an array... but I'm sure >> > there must be a simpler way. I've also looked at argwhere. Or maybe I >> > should use pandas? >> > >> > Thank you in advance. >> > >> > >> > _______________________________________________ >> > SciPy-User mailing list >> > SciPy-User at scipy.org >> > http://mail.scipy.org/mailman/listinfo/scipy-user >> >> +1 for using pandas >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From juanlu001 at gmail.com Tue Nov 13 12:41:46 2012 From: juanlu001 at gmail.com (=?ISO-8859-1?Q?Juan_Luis_Cano_Rodr=EDguez?=) Date: Tue, 13 Nov 2012 18:41:46 +0100 Subject: [SciPy-User] Select rows according to cell value In-Reply-To: References: <50A27106.6060103@hilboll.de> Message-ID: Actually I arrived to a couple of one-liners: d = np.take(data, [np.argwhere(data[:, 0] == a).flatten()[0] for a in altitudes], axis=0) or d = np.array([data[data[:, 0] == a][0] for a in altitudes]) I find them sort of ugly but maybe it's the only way. The same way I'd do data[[1, 3, 8]] to retrieve the first, third and eighth I'd like to do data[np.magic_indices(altitudes)] On Tue, Nov 13, 2012 at 6:02 PM, Oleksandr Huziy wrote: > Yeps, I admit with pandas it appears much easier > > import pandas > df = pandas.read_csv("tmp/data.txt", sep="\\s") > df = df.dropna(axis = 1) > > #df.index = df["alt"] > selection = df.select(lambda i: df.ix[i, "alt"] in altitudes) > print selection > > > cheers > -- > Oleksandr (Sasha) Huziy > > > > 2012/11/13 Oleksandr Huziy > >> I am not sure if this way is easier thsn yours, but here is what I wpuld >> do >> >> tol = 0.01 >> all_alts = data[:,0] >> print all_alts >> all_alts_temp = np.vstack([all_alts]*len(altitudes)) >> print all_alts_temp >> >> sel_alts_temp = np.vstack([altitudes]*len(all_alts)).transpose() >> print sel_alts_temp >> sel_pattern = np.any( np.abs(all_alts_temp - sel_alts_temp) < tol, axis = >> 0) >> print sel_pattern >> print data >> print data[sel_pattern,:] >> >> >> Cheers >> -- >> Oleksandr (Sasha) Huziy >> >> >> >> >> 2012/11/13 Andreas Hilboll >> >>> Am Di 13 Nov 2012 17:07:19 CET schrieb Juan Luis Cano Rodr?guez: >>> > I am loading some tabular data of the form >>> > >>> > alt temp press dens >>> > 10.0 223.3 26500 0.414 >>> > 10.5 220.0 24540 0.389 >>> > 11.0 216.8 22700 0.365 >>> > 11.5 216.7 20985 0.337 >>> > 12.0 216.7 19399 0.312 >>> > 12.5 216.7 17934 0.288 >>> > 13.0 216.7 16579 0.267 >>> > 13.5 216.7 15328 0.246 >>> > 14.0 216.7 14170 0.228 >>> > >>> > into an ordinary NumPy array using np.loadtxt. I would like though to >>> > select the rows according to the altitude level, that is: >>> > >>> > >>> data = np.loadtxt('data.txt', skiprows=1) >>> > >>> altitudes = [10.5, 11.5, 14.0] >>> > >>> d = ... # some simple syntax involving data and altitudes >>> > >>> d >>> > 10.5 220.0 24540 0.389 >>> > 11.5 216.7 20985 0.337 >>> > 14.0 216.7 14170 0.228 >>> > >>> > I have tried a cumbersome expression which traverses all the array, >>> > then uses a list comprehension, converts to an array... but I'm sure >>> > there must be a simpler way. I've also looked at argwhere. Or maybe I >>> > should use pandas? >>> > >>> > Thank you in advance. >>> > >>> > >>> > _______________________________________________ >>> > SciPy-User mailing list >>> > SciPy-User at scipy.org >>> > http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> +1 for using pandas >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >> >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From slasley at space.umd.edu Tue Nov 13 12:53:48 2012 From: slasley at space.umd.edu (Scott Lasley) Date: Tue, 13 Nov 2012 12:53:48 -0500 Subject: [SciPy-User] Select rows according to cell value In-Reply-To: References: <50A27106.6060103@hilboll.de> Message-ID: I don't know if this is any less ugly, but you could use magic_indices = lambda a: np.any([data[:,0] == x for x in a],axis=0) d = data[magic_indices(altitudes)] Or, as Oleksandr pointed out, if you're concerned about comparing floats you might use tol = 0.01 magic_indices = lambda a: np.any([np.abs(data[:,0] - x) < tol for x in a],axis=0) d = data[magic_indices(altitudes)] or just tol = 0.01 d = data[np.any([np.abs(data[:,0] - x) < tol for x in altitudes],axis=0)] Scott On Nov 13, 2012, at 12:41 PM, Juan Luis Cano Rodr?guez wrote: > Actually I arrived to a couple of one-liners: > > d = np.take(data, [np.argwhere(data[:, 0] == a).flatten()[0] for a in altitudes], axis=0) > > or > > d = np.array([data[data[:, 0] == a][0] for a in altitudes]) > > I find them sort of ugly but maybe it's the only way. The same way I'd do > > data[[1, 3, 8]] > > to retrieve the first, third and eighth I'd like to do > > data[np.magic_indices(altitudes)] > > > On Tue, Nov 13, 2012 at 6:02 PM, Oleksandr Huziy wrote: > Yeps, I admit with pandas it appears much easier > > import pandas > df = pandas.read_csv("tmp/data.txt", sep="\\s") > df = df.dropna(axis = 1) > > #df.index = df["alt"] > selection = df.select(lambda i: df.ix[i, "alt"] in altitudes) > print selection > > > cheers > -- > Oleksandr (Sasha) Huziy > > > > 2012/11/13 Oleksandr Huziy > I am not sure if this way is easier thsn yours, but here is what I wpuld do > > tol = 0.01 > all_alts = data[:,0] > print all_alts > all_alts_temp = np.vstack([all_alts]*len(altitudes)) > print all_alts_temp > > sel_alts_temp = np.vstack([altitudes]*len(all_alts)).transpose() > print sel_alts_temp > sel_pattern = np.any( np.abs(all_alts_temp - sel_alts_temp) < tol, axis = 0) > print sel_pattern > print data > print data[sel_pattern,:] > > > Cheers > -- > Oleksandr (Sasha) Huziy > > > > > 2012/11/13 Andreas Hilboll > Am Di 13 Nov 2012 17:07:19 CET schrieb Juan Luis Cano Rodr?guez: > > I am loading some tabular data of the form > > > > alt temp press dens > > 10.0 223.3 26500 0.414 > > 10.5 220.0 24540 0.389 > > 11.0 216.8 22700 0.365 > > 11.5 216.7 20985 0.337 > > 12.0 216.7 19399 0.312 > > 12.5 216.7 17934 0.288 > > 13.0 216.7 16579 0.267 > > 13.5 216.7 15328 0.246 > > 14.0 216.7 14170 0.228 > > > > into an ordinary NumPy array using np.loadtxt. I would like though to > > select the rows according to the altitude level, that is: > > > > >>> data = np.loadtxt('data.txt', skiprows=1) > > >>> altitudes = [10.5, 11.5, 14.0] > > >>> d = ... # some simple syntax involving data and altitudes > > >>> d > > 10.5 220.0 24540 0.389 > > 11.5 216.7 20985 0.337 > > 14.0 216.7 14170 0.228 > > > > I have tried a cumbersome expression which traverses all the array, > > then uses a list comprehension, converts to an array... but I'm sure > > there must be a simpler way. I've also looked at argwhere. Or maybe I > > should use pandas? > > > > Thank you in advance. > > > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > +1 for using pandas > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From wesmckinn at gmail.com Tue Nov 13 15:56:55 2012 From: wesmckinn at gmail.com (Wes McKinney) Date: Tue, 13 Nov 2012 15:56:55 -0500 Subject: [SciPy-User] Select rows according to cell value In-Reply-To: References: <50A27106.6060103@hilboll.de> Message-ID: On Tue, Nov 13, 2012 at 12:53 PM, Scott Lasley wrote: > I don't know if this is any less ugly, but you could use > > magic_indices = lambda a: np.any([data[:,0] == x for x in a],axis=0) > d = data[magic_indices(altitudes)] > > Or, as Oleksandr pointed out, if you're concerned about comparing floats you might use > > tol = 0.01 > magic_indices = lambda a: np.any([np.abs(data[:,0] - x) < tol for x in a],axis=0) > d = data[magic_indices(altitudes)] > > or just > > tol = 0.01 > d = data[np.any([np.abs(data[:,0] - x) < tol for x in altitudes],axis=0)] > > Scott > > On Nov 13, 2012, at 12:41 PM, Juan Luis Cano Rodr?guez wrote: > >> Actually I arrived to a couple of one-liners: >> >> d = np.take(data, [np.argwhere(data[:, 0] == a).flatten()[0] for a in altitudes], axis=0) >> >> or >> >> d = np.array([data[data[:, 0] == a][0] for a in altitudes]) >> >> I find them sort of ugly but maybe it's the only way. The same way I'd do >> >> data[[1, 3, 8]] >> >> to retrieve the first, third and eighth I'd like to do >> >> data[np.magic_indices(altitudes)] >> >> >> On Tue, Nov 13, 2012 at 6:02 PM, Oleksandr Huziy wrote: >> Yeps, I admit with pandas it appears much easier >> >> import pandas >> df = pandas.read_csv("tmp/data.txt", sep="\\s") >> df = df.dropna(axis = 1) >> >> #df.index = df["alt"] >> selection = df.select(lambda i: df.ix[i, "alt"] in altitudes) >> print selection >> >> >> cheers >> -- >> Oleksandr (Sasha) Huziy >> >> >> >> 2012/11/13 Oleksandr Huziy >> I am not sure if this way is easier thsn yours, but here is what I wpuld do >> >> tol = 0.01 >> all_alts = data[:,0] >> print all_alts >> all_alts_temp = np.vstack([all_alts]*len(altitudes)) >> print all_alts_temp >> >> sel_alts_temp = np.vstack([altitudes]*len(all_alts)).transpose() >> print sel_alts_temp >> sel_pattern = np.any( np.abs(all_alts_temp - sel_alts_temp) < tol, axis = 0) >> print sel_pattern >> print data >> print data[sel_pattern,:] >> >> >> Cheers >> -- >> Oleksandr (Sasha) Huziy >> >> >> >> >> 2012/11/13 Andreas Hilboll >> Am Di 13 Nov 2012 17:07:19 CET schrieb Juan Luis Cano Rodr?guez: >> > I am loading some tabular data of the form >> > >> > alt temp press dens >> > 10.0 223.3 26500 0.414 >> > 10.5 220.0 24540 0.389 >> > 11.0 216.8 22700 0.365 >> > 11.5 216.7 20985 0.337 >> > 12.0 216.7 19399 0.312 >> > 12.5 216.7 17934 0.288 >> > 13.0 216.7 16579 0.267 >> > 13.5 216.7 15328 0.246 >> > 14.0 216.7 14170 0.228 >> > >> > into an ordinary NumPy array using np.loadtxt. I would like though to >> > select the rows according to the altitude level, that is: >> > >> > >>> data = np.loadtxt('data.txt', skiprows=1) >> > >>> altitudes = [10.5, 11.5, 14.0] >> > >>> d = ... # some simple syntax involving data and altitudes >> > >>> d >> > 10.5 220.0 24540 0.389 >> > 11.5 216.7 20985 0.337 >> > 14.0 216.7 14170 0.228 >> > >> > I have tried a cumbersome expression which traverses all the array, >> > then uses a list comprehension, converts to an array... but I'm sure >> > there must be a simpler way. I've also looked at argwhere. Or maybe I >> > should use pandas? >> > >> > Thank you in advance. >> > >> > >> > _______________________________________________ >> > SciPy-User mailing list >> > SciPy-User at scipy.org >> > http://mail.scipy.org/mailman/listinfo/scipy-user >> >> +1 for using pandas >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user Use isin: In [5]: df[df.alt.isin(altitudes)] Out[5]: alt temp press dens 1 10.5 220.0 24540 0.389 3 11.5 216.7 20985 0.337 8 14.0 216.7 14170 0.228 - Wes From juanlu001 at gmail.com Tue Nov 13 16:19:33 2012 From: juanlu001 at gmail.com (=?ISO-8859-1?Q?Juan_Luis_Cano_Rodr=EDguez?=) Date: Tue, 13 Nov 2012 22:19:33 +0100 Subject: [SciPy-User] Select rows according to cell value In-Reply-To: References: <50A27106.6060103@hilboll.de> Message-ID: Oh Wes, you showed me the light :) On Tue, Nov 13, 2012 at 9:56 PM, Wes McKinney wrote: > On Tue, Nov 13, 2012 at 12:53 PM, Scott Lasley > wrote: > > I don't know if this is any less ugly, but you could use > > > > magic_indices = lambda a: np.any([data[:,0] == x for x in a],axis=0) > > d = data[magic_indices(altitudes)] > > > > Or, as Oleksandr pointed out, if you're concerned about comparing floats > you might use > > > > tol = 0.01 > > magic_indices = lambda a: np.any([np.abs(data[:,0] - x) < tol for x in > a],axis=0) > > d = data[magic_indices(altitudes)] > > > > or just > > > > tol = 0.01 > > d = data[np.any([np.abs(data[:,0] - x) < tol for x in altitudes],axis=0)] > > > > Scott > > > > On Nov 13, 2012, at 12:41 PM, Juan Luis Cano Rodr?guez < > juanlu001 at gmail.com> wrote: > > > >> Actually I arrived to a couple of one-liners: > >> > >> d = np.take(data, [np.argwhere(data[:, 0] == a).flatten()[0] for a > in altitudes], axis=0) > >> > >> or > >> > >> d = np.array([data[data[:, 0] == a][0] for a in altitudes]) > >> > >> I find them sort of ugly but maybe it's the only way. The same way I'd > do > >> > >> data[[1, 3, 8]] > >> > >> to retrieve the first, third and eighth I'd like to do > >> > >> data[np.magic_indices(altitudes)] > >> > >> > >> On Tue, Nov 13, 2012 at 6:02 PM, Oleksandr Huziy > wrote: > >> Yeps, I admit with pandas it appears much easier > >> > >> import pandas > >> df = pandas.read_csv("tmp/data.txt", sep="\\s") > >> df = df.dropna(axis = 1) > >> > >> #df.index = df["alt"] > >> selection = df.select(lambda i: df.ix[i, "alt"] in altitudes) > >> print selection > >> > >> > >> cheers > >> -- > >> Oleksandr (Sasha) Huziy > >> > >> > >> > >> 2012/11/13 Oleksandr Huziy > >> I am not sure if this way is easier thsn yours, but here is what I > wpuld do > >> > >> tol = 0.01 > >> all_alts = data[:,0] > >> print all_alts > >> all_alts_temp = np.vstack([all_alts]*len(altitudes)) > >> print all_alts_temp > >> > >> sel_alts_temp = np.vstack([altitudes]*len(all_alts)).transpose() > >> print sel_alts_temp > >> sel_pattern = np.any( np.abs(all_alts_temp - sel_alts_temp) < tol, axis > = 0) > >> print sel_pattern > >> print data > >> print data[sel_pattern,:] > >> > >> > >> Cheers > >> -- > >> Oleksandr (Sasha) Huziy > >> > >> > >> > >> > >> 2012/11/13 Andreas Hilboll > >> Am Di 13 Nov 2012 17:07:19 CET schrieb Juan Luis Cano Rodr?guez: > >> > I am loading some tabular data of the form > >> > > >> > alt temp press dens > >> > 10.0 223.3 26500 0.414 > >> > 10.5 220.0 24540 0.389 > >> > 11.0 216.8 22700 0.365 > >> > 11.5 216.7 20985 0.337 > >> > 12.0 216.7 19399 0.312 > >> > 12.5 216.7 17934 0.288 > >> > 13.0 216.7 16579 0.267 > >> > 13.5 216.7 15328 0.246 > >> > 14.0 216.7 14170 0.228 > >> > > >> > into an ordinary NumPy array using np.loadtxt. I would like though to > >> > select the rows according to the altitude level, that is: > >> > > >> > >>> data = np.loadtxt('data.txt', skiprows=1) > >> > >>> altitudes = [10.5, 11.5, 14.0] > >> > >>> d = ... # some simple syntax involving data and altitudes > >> > >>> d > >> > 10.5 220.0 24540 0.389 > >> > 11.5 216.7 20985 0.337 > >> > 14.0 216.7 14170 0.228 > >> > > >> > I have tried a cumbersome expression which traverses all the array, > >> > then uses a list comprehension, converts to an array... but I'm sure > >> > there must be a simpler way. I've also looked at argwhere. Or maybe I > >> > should use pandas? > >> > > >> > Thank you in advance. > >> > > >> > > >> > _______________________________________________ > >> > SciPy-User mailing list > >> > SciPy-User at scipy.org > >> > http://mail.scipy.org/mailman/listinfo/scipy-user > >> > >> +1 for using pandas > >> _______________________________________________ > >> SciPy-User mailing list > >> SciPy-User at scipy.org > >> http://mail.scipy.org/mailman/listinfo/scipy-user > >> > >> > >> > >> _______________________________________________ > >> SciPy-User mailing list > >> SciPy-User at scipy.org > >> http://mail.scipy.org/mailman/listinfo/scipy-user > >> > >> > >> _______________________________________________ > >> SciPy-User mailing list > >> SciPy-User at scipy.org > >> http://mail.scipy.org/mailman/listinfo/scipy-user > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > Use isin: > > In [5]: df[df.alt.isin(altitudes)] > Out[5]: > alt temp press dens > 1 10.5 220.0 24540 0.389 > 3 11.5 216.7 20985 0.337 > 8 14.0 216.7 14170 0.228 > > - Wes > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wesmckinn at gmail.com Wed Nov 14 22:33:25 2012 From: wesmckinn at gmail.com (Wes McKinney) Date: Wed, 14 Nov 2012 22:33:25 -0500 Subject: [SciPy-User] ANN: pandas 0.9.1 released Message-ID: hi all, I'm pleased to announce the 0.9.1 release of pandas. This is primarily a bugfix release but contains a number of new features and enhancements. There are also a few minor API changes having to do with time series using the Period data type. I strongly recommend that all users upgrade to this release. Note that I plan to make a 0.9.2 release shortly within the next few weeks (hopefully!) that includes the new file parser branch (http://wesmckinney.com/blog/?p=543). This includes vastly improved performance and memory usage, per-column user data type specification (in addition to the default type inference), column subsetting by name (for those "really wide" CSV files), improved support for European decimal formats, and many other goodies. I'm hopeful someone will help port this code to go in a near-future version of NumPy. As always source archives and Windows installers can be found on PyPI. Thanks to all who contributed to this release, especially Yoval P, Chang She, Jeff Reback, and everyone listed below. A special thanks also goes out to all those who answer questions on StackOverflow and the mailing lists, helping to make the pandas community vibrant and helpful. What's new: http://pandas.pydata.org/pandas-docs/stable/whatsnew.html $ git log v0.9.0..v0.9.1 --pretty=format:%aN | sort | uniq -c | sort -rn 90 Wes McKinney 66 y-p 57 Chang She 7 Jeff Reback 6 Tobias Brandt 4 Wouter Overmeire 4 Brenda Moon 2 timmie 1 Martin Blais 1 K.-Michael Aye 1 Justin C Johnson Happy data hacking! - Wes What is it ========== pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with relational, time series, or any other kind of labeled data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Links ===== Release Notes: http://github.com/pydata/pandas/blob/master/RELEASE.rst Documentation: http://pandas.pydata.org Installers: http://pypi.python.org/pypi/pandas Code Repository: http://github.com/pydata/pandas Mailing List: http://groups.google.com/group/pydata From vs at it.uu.se Thu Nov 15 08:48:01 2012 From: vs at it.uu.se (Virgil Stokes) Date: Thu, 15 Nov 2012 14:48:01 +0100 Subject: [SciPy-User] ANN: pandas 0.9.1 released In-Reply-To: References: Message-ID: <50A4F291.8040505@it.uu.se> On 15-Nov-2012 04:33, Wes McKinney wrote: > hi all, > > I'm pleased to announce the 0.9.1 release of pandas. This is > primarily a bugfix release but contains a number of new features > and enhancements. There are also a few minor API changes having > to do with time series using the Period data type. I strongly > recommend that all users upgrade to this release. > > Note that I plan to make a 0.9.2 release shortly within the next > few weeks (hopefully!) that includes the new file parser > branch (http://wesmckinney.com/blog/?p=543). This includes vastly > improved performance and memory usage, per-column user data type > specification (in addition to the default type inference), column > subsetting by name (for those "really wide" CSV files), improved > support for European decimal formats, and many other goodies. I'm > hopeful someone will help port this code to go in a near-future > version of NumPy. > > As always source archives and Windows installers can be found on > PyPI. Thanks to all who contributed to this release, especially > Yoval P, Chang She, Jeff Reback, and everyone listed below. A > special thanks also goes out to all those who answer questions on > StackOverflow and the mailing lists, helping to make the pandas > community vibrant and helpful. > > What's new: http://pandas.pydata.org/pandas-docs/stable/whatsnew.html > > $ git log v0.9.0..v0.9.1 --pretty=format:%aN | sort | uniq -c | sort -rn > 90 Wes McKinney > 66 y-p > 57 Chang She > 7 Jeff Reback > 6 Tobias Brandt > 4 Wouter Overmeire > 4 Brenda Moon > 2 timmie > 1 Martin Blais > 1 K.-Michael Aye > 1 Justin C Johnson > > Happy data hacking! > > - Wes > > What is it > ========== > pandas is a Python package providing fast, flexible, and > expressive data structures designed to make working with > relational, time series, or any other kind of labeled data both > easy and intuitive. It aims to be the fundamental high-level > building block for doing practical, real world data analysis in > Python. > > Links > ===== > Release Notes: http://github.com/pydata/pandas/blob/master/RELEASE.rst > Documentation: http://pandas.pydata.org > Installers: http://pypi.python.org/pypi/pandas > Code Repository: http://github.com/pydata/pandas > Mailing List: http://groups.google.com/group/pydata > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user Great job Wes --- keep up the good work. Best regards, --V :-) From deil.christoph at googlemail.com Thu Nov 15 09:53:20 2012 From: deil.christoph at googlemail.com (Christoph Deil) Date: Thu, 15 Nov 2012 15:53:20 +0100 Subject: [SciPy-User] Python packages for parallel numerical optimization? Message-ID: Hi, are there any good Python packages for parallel numerical optimisation? Just to be clear, I mean minimising a function that is time-consuming to compute and needs to be computed many times to find the minimum. I'm mostly interested in message passing solutions that work on a cluster of 10 to 1000 CPUs. The only thing I found with Google if FiPy, which uses mpi4py: http://www.ctcms.nist.gov/fipy/documentation/USAGE.html#parallel But FiPy is a finite volume PDE solver, which is not the application I want to solve. I've heard many good things about zeromq ( http://www.zeromq.org/bindings:python ) and it looks very simple and powerful, so I'd be especially interested in zeromq-based solutions. Any links welcome, also on parallel optimisation on CPU clusters in general! Thanks a lot! Christoph From gabe.g at me.com Thu Nov 15 12:19:40 2012 From: gabe.g at me.com (gabe.g at me.com) Date: Thu, 15 Nov 2012 11:19:40 -0600 Subject: [SciPy-User] Sparse Matrices, summing columns !=sum Message-ID: <49FBA830-EC6C-4C8A-8523-F1499BFC6E65@me.com> Hi all, I'm new to this list. Have a specific question about Sparse matrices. I posted this to StackOverflow and Anony-Mousse suggested it might be a bug and that I should post to this list: So-- is this a bug or am I doing something wrong? Thanks much. Gabe I have a sparse matrix that I arrived at through a complicated bunch of calculations which I cannot reproduce here. I will try to find a simpler example of this. For now, does anyone know how it might be (even remotely) possible that I could have a sparse matrix X with the property that: In [143]: X.sum(0).sum() Out[143]: 131138 In [144]: X.sum() Out[144]: 327746 In [145]: X.sum(1).sum() Out[145]: 327746 In [146]: type(X) Out[146]: scipy.sparse.csr.csr_matrix My only guess is that if I want to sum columns correctly, I need to first cast the matrix as csc -- which makes sense. Although one would think that the sparse package would handle column sums gracefully (or throw an error) instead of just giving a WRONG answer. After more thought, I tried the following: In [164]: X.tocsr().sum(0).sum() Out[164]: 131138 In [165]: X.tocsc().sum(0).sum() Out[165]: 131138 In [166]: X.tocoo().sum(0).sum() Out[166]: 131138 In [167]: X.tolil().sum(0).sum() Out[167]: 131138 In [168]: X.todok().sum(0).sum() Out[168]: 131138 In [169]: X.shape Out[169]: (196980, 43) In [170]: X Out[170]: <196980x43 sparse matrix of type '' with 70875 stored elements in Compressed Sparse Row format> In [172]: X.todense().sum(0) Out[172]: matrix([[170726, 1041, 117398, 3526, 13202, 3585, 2355, 1895, 1392, 2189, 2070, 2603, 1676, 496, 1194, 933, 129, 529, 544, 256, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], dtype=uint64) In [173]: X.sum(0) Out[173]: matrix([[39654, 1041, 51862, 3526, 13202, 3585, 2355, 1895, 1392, 2189, 2070, 2603, 1676, 496, 1194, 933, 129, 529, 544, 256, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], dtype=uint16) -------------- next part -------------- An HTML attachment was scrubbed... URL: From L.J.Buitinck at uva.nl Fri Nov 16 10:08:46 2012 From: L.J.Buitinck at uva.nl (Lars Buitinck) Date: Fri, 16 Nov 2012 16:08:46 +0100 Subject: [SciPy-User] Sparse Matrices, summing columns !=sum Message-ID: 2012/11/16 : > Subject: [SciPy-User] Sparse Matrices, summing columns !=sum > To: scipy-user at scipy.org > > I have a sparse matrix that I arrived at through a complicated bunch of calculations which I cannot reproduce here. I will try to find a simpler example of this. > > For now, does anyone know how it might be (even remotely) possible that I could have a sparse matrix X with the property that: > > In [143]: X.sum(0).sum() > Out[143]: 131138 I tried this in SciPy 0.7.2 by constructing X from the dense matrix you posted below, your result for X.todense().sum(0). So >>> X matrix([[39654, 1041, 51862, 3526, 13202, 3585, 2355, 1895, 1392, 2189, 2070, 2603, 1676, 496, 1194, 933, 129, 529, 544, 256, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], dtype=uint16) >>> Xs = csr_matrix(Xs) >>> Xs <1x43 sparse matrix of type '' with 21 stored elements in Compressed Sparse Row format> With this, I get >>> Xs.sum(axis=1) matrix([[66]], dtype=uint16) >>> X.sum(axis=1) matrix([[131138]], dtype=uint64) >>> Xs.toarray().sum(axis=1) array([131138], dtype=uint64) So it would seem that csr_matrix.sum tries to retain the dtype for the sum, overflowing np.uint16 halfway through the computation, while np.matrix.sum picks a larger integer type. One solution/workaround would be to use a larger dtype. I haven't tried this on more recent SciPy. -- Lars Buitinck Scientific programmer, ILPS University of Amsterdam From gabe at gabegaster.com Fri Nov 16 10:59:11 2012 From: gabe at gabegaster.com (Gabriel Gaster) Date: Fri, 16 Nov 2012 09:59:11 -0600 Subject: [SciPy-User] SciPy-User Digest, Vol 111, Issue 20 In-Reply-To: References: Message-ID: Sorry -- never mind -- this was figured out on StackOverflow. It was actually an overflow error. http://stackoverflow.com/questions/13402393/scipy-sparse-matrices-inconsistent-sums -- Gabriel Gaster Data Scientist gabe at gabegaster.com, twitter, blog 773 816 3349, skype: gabrielgaster On Nov 16, 2012, at 8:04 AM, scipy-user-request at scipy.org wrote: > Send SciPy-User mailing list submissions to > scipy-user at scipy.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.scipy.org/mailman/listinfo/scipy-user > or, via email, send a message with subject or body 'help' to > scipy-user-request at scipy.org > > You can reach the person managing the list at > scipy-user-owner at scipy.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of SciPy-User digest..." > > > Today's Topics: > > 1. Sparse Matrices, summing columns !=sum (gabe.g at me.com) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 15 Nov 2012 11:19:40 -0600 > From: gabe.g at me.com > Subject: [SciPy-User] Sparse Matrices, summing columns !=sum > To: scipy-user at scipy.org > Message-ID: <49FBA830-EC6C-4C8A-8523-F1499BFC6E65 at me.com> > Content-Type: text/plain; charset="us-ascii" > > Hi all, I'm new to this list. Have a specific question about Sparse matrices. > > I posted this to StackOverflow and Anony-Mousse suggested it might be a bug and that I should post to this list: > > So-- is this a bug or am I doing something wrong? > Thanks much. > Gabe > > > I have a sparse matrix that I arrived at through a complicated bunch of calculations which I cannot reproduce here. I will try to find a simpler example of this. > > For now, does anyone know how it might be (even remotely) possible that I could have a sparse matrix X with the property that: > > In [143]: X.sum(0).sum() > Out[143]: 131138 > > In [144]: X.sum() > Out[144]: 327746 > > In [145]: X.sum(1).sum() > Out[145]: 327746 > > In [146]: type(X) > Out[146]: scipy.sparse.csr.csr_matrix > My only guess is that if I want to sum columns correctly, I need to first cast the matrix as csc -- which makes sense. Although one would think that the sparse package would handle column sums gracefully (or throw an error) instead of just giving a WRONG answer. > > After more thought, I tried the following: > > In [164]: X.tocsr().sum(0).sum() > Out[164]: 131138 > > In [165]: X.tocsc().sum(0).sum() > Out[165]: 131138 > > In [166]: X.tocoo().sum(0).sum() > Out[166]: 131138 > > In [167]: X.tolil().sum(0).sum() > Out[167]: 131138 > > In [168]: X.todok().sum(0).sum() > Out[168]: 131138 > > In [169]: X.shape > Out[169]: (196980, 43) > > In [170]: X > Out[170]: > <196980x43 sparse matrix of type '' > with 70875 stored elements in Compressed Sparse Row format> > > In [172]: X.todense().sum(0) > Out[172]: > matrix([[170726, 1041, 117398, 3526, 13202, 3585, 2355, 1895, 1392, 2189, 2070, 2603, 1676, 496, 1194, 933, 129, > 529, 544, 256, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0]], dtype=uint64) > > In [173]: X.sum(0) > Out[173]: > matrix([[39654, 1041, 51862, 3526, 13202, 3585, 2355, 1895, 1392, 2189, 2070, 2603, 1676, 496, 1194, 933, 129, 529, 544, 256, > 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0]], dtype=uint16) > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: http://mail.scipy.org/pipermail/scipy-user/attachments/20121115/2a8bbe49/attachment.html > > ------------------------------ > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > > End of SciPy-User Digest, Vol 111, Issue 20 > ******************************************* -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri Nov 16 11:57:23 2012 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 16 Nov 2012 16:57:23 +0000 (UTC) Subject: [SciPy-User] Sparse Matrices, summing columns !=sum References: <49FBA830-EC6C-4C8A-8523-F1499BFC6E65@me.com> Message-ID: me.com> writes: [clip] > So-- is this a bug or am I doing something wrong? > > I have a sparse matrix that I arrived at through > a complicated bunch of calculations which I cannot > reproduce here. I will try to find a simpler example of this. [clip] > In [170]: X > Out[170]: > <196980x43 sparse matrix of type '' > with 70875 stored elements in Compressed Sparse Row format> It's an integer overflow due to using the same integer type as the accumulator. Dense Numpy arrays however appear to use the platform default integer as the accumulator: >>> np.array([[1,1],[1,1]], dtype=np.uint16).sum(axis=0) array([2, 2], dtype=uint64) IIRC, this was changed in Numpy at some point to work like this. The sparse matrices perhaps also should mirror this behavior, at least it would avoid overflows in the most common cases. -- Pauli Virtanen From aronne.merrelli at gmail.com Fri Nov 16 13:25:30 2012 From: aronne.merrelli at gmail.com (Aronne Merrelli) Date: Fri, 16 Nov 2012 12:25:30 -0600 Subject: [SciPy-User] Sparse Matrices, summing columns !=sum In-Reply-To: References: <49FBA830-EC6C-4C8A-8523-F1499BFC6E65@me.com> Message-ID: On Fri, Nov 16, 2012 at 10:57 AM, Pauli Virtanen wrote: > me.com> writes: > [clip] >> So-- is this a bug or am I doing something wrong? >> >> I have a sparse matrix that I arrived at through >> a complicated bunch of calculations which I cannot >> reproduce here. I will try to find a simpler example of this. > [clip] >> In [170]: X >> Out[170]: >> <196980x43 sparse matrix of type '' >> with 70875 stored elements in Compressed Sparse Row format> > > It's an integer overflow due to using the same integer type > as the accumulator. > > Dense Numpy arrays however appear to use the platform default integer > as the accumulator: > >>>> np.array([[1,1],[1,1]], dtype=np.uint16).sum(axis=0) > array([2, 2], dtype=uint64) > > IIRC, this was changed in Numpy at some point to work like this. > > The sparse matrices perhaps also should mirror this behavior, > at least it would avoid overflows in the most common cases. > Tracing the code, it looks like the sum method of the spmatrix class is used in this case (see base.py in the sparse module). It computes column or row sums by a matrix product: if axis == 0: # sum over columns return np.asmatrix(np.ones((1, m), dtype=self.dtype)) * self elif axis == 1: # sum over rows return self * np.asmatrix(np.ones((n, 1), dtype=self.dtype)) My guess is by turning this into a multiplication it uses a more efficient sparse matrix product. I'd think this could be modified to take an optional keyword dtype argument instead of defaulting to the same type as itself. The user could pick a larger dtype to prevent overflow. Something like the following (which also might be a workaround for the OP?): In [1]: import scipy.sparse In [2]: denseA = np.zeros((2,2), dtype=uint16) In [3]: denseA[:,0] = 60000 In [4]: denseA.sum(0) Out[4]: array([120000, 0], dtype=uint64) In [5]: sparseA = scipy.sparse.csr.csr_matrix(denseA) In [6]: sparseA Out[6]: <2x2 sparse matrix of type '' with 2 stored elements in Compressed Sparse Row format> In [7]: sparseA.sum(0) # overflows Out[7]: matrix([[54464, 0]], dtype=uint16) In [12]: np.asmatrix(np.ones((1,2),dtype=uint64)) * sparseA # does not overflow Out[12]: matrix([[120000, 0]], dtype=uint64) From juanlu001 at gmail.com Sat Nov 17 05:19:52 2012 From: juanlu001 at gmail.com (=?ISO-8859-1?Q?Juan_Luis_Cano_Rodr=EDguez?=) Date: Sat, 17 Nov 2012 11:19:52 +0100 Subject: [SciPy-User] Question about boolean indexing in NumPy Message-ID: Hello everybody, I was trying to understand the use of boolean indexing in NumPy, and looking both in the User Guide and the Reference has driven me a bit confused. On the one hand, in the User Guide http://docs.scipy.org/doc/numpy/user/basics.indexing.html#boolean-or-mask-index-arrays is stated that "Boolean arrays must be [...] broadcastable to the same shape", and then you have this example: >>> y = np.arange(35).reshape(5,7) >>> b = y > 20 >>> b[:,5] # use a 1-D boolean that broadcasts with y array([False, False, False, True, True], dtype=bool) >>> y[b[:,5]] array([[21, 22, 23, 24, 25, 26, 27], [28, 29, 30, 31, 32, 33, 34]]) but, indeed, these two arrays don't broadcast! Though they do if you transpose one of them: >>> np.broadcast_arrays(y, b[:, 5]) Traceback (most recent call last): File "", line 1, in File "/usr/lib/python3.3/site-packages/numpy/lib/stride_tricks.py", line 94, in broadcast_arrays "incompatible dimensions on axis %r." % (axis,)) ValueError: shape mismatch: two or more arrays have incompatible dimensions on axis 1. On the other hand, it is also stated that "Boolean arrays must be of the same shape as the array being indexed", but indeed this is not true, because according to the reference guide http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#boolean "This advanced indexing [...] is always equivalent to [...] x[obj.nonzero()]". So I can come up with an example: >>> y[0, :] array([0, 1, 2, 3, 4, 5, 6]) >>> y[0, :].shape (7,) >>> np.array([True, False, True, False]).shape # Not the same shape! (4,) >>> y[0, :][np.array([True, False, True, False])] # But actually it works -> [].nonzero() array([0, 2]) Is there anything that I am misunderstanding? Can everyone shine the light on me in this topic? Thanks in advance, regards Juan Luis Cano -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Sat Nov 17 06:08:21 2012 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Sat, 17 Nov 2012 12:08:21 +0100 Subject: [SciPy-User] Question about boolean indexing in NumPy In-Reply-To: References: Message-ID: <1353150501.9591.14.camel@sebastian-laptop> On Sat, 2012-11-17 at 11:19 +0100, Juan Luis Cano Rodr?guez wrote: > Hello everybody, I was trying to understand the use of boolean > indexing in NumPy, and looking both in the User Guide and the > Reference has driven me a bit confused. > > On the one hand, in the User Guide > > http://docs.scipy.org/doc/numpy/user/basics.indexing.html#boolean-or-mask-index-arrays > > is stated that "Boolean arrays must be [...] broadcastable to the same > shape", and then you have this example: > I guess this is probably not quite correct. In indexing you always give indices from the first to last dimension, while broadcasting aligns from the last to first dimension. Under the hood, what happens works basically like this: 1. indices = np.nonzero(boolean_index) 2. result = array[indices] Note that indices would be a tuple, so you are indexing multiple dimensions. But you start with the first dimension not with the last. This makes sense since it generalizes also if you would do `array[:,b]`. then you got `array[:, b.nonzero[0], b.nonzero[1]]` (assuming b is 2-d). When thinking about broadcasting you could also think that a 1-dimensional axis is repeated. This does not happen, if you would want that you would have to rather remove that axes and let normal indexing do its job. Ie. not array[b], but array[:,b[0]]. There is a small thing that due to use of np.nonzero like functionality, the array shapes do not have to match exactly (if out of bound values are all False), but I personally think that one shouldn't rely on that. I hope this clarifies things for you, Regards, Sebastian > >>> y = np.arange(35).reshape(5,7) > >>> b = y > 20 > >>> b[:,5] # use a 1-D boolean that broadcasts with y > array([False, False, False, True, True], dtype=bool) > >>> y[b[:,5]] > array([[21, 22, 23, 24, 25, 26, 27], > [28, 29, 30, 31, 32, 33, 34]]) > > but, indeed, these two arrays don't broadcast! Though they do if you > transpose one of them: > > >>> np.broadcast_arrays(y, b[:, 5]) > Traceback (most recent call last): > File "", line 1, in > File "/usr/lib/python3.3/site-packages/numpy/lib/stride_tricks.py", > line 94, in broadcast_arrays > "incompatible dimensions on axis %r." % (axis,)) > ValueError: shape mismatch: two or more arrays have incompatible > dimensions on axis 1. > > On the other hand, it is also stated that "Boolean arrays must be of > the same shape as the array being indexed", but indeed this is not > true, because according to the reference guide > > http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#boolean > > "This advanced indexing [...] is always equivalent to [...] > x[obj.nonzero()]". So I can come up with an example: > > >>> y[0, :] > array([0, 1, 2, 3, 4, 5, 6]) > >>> y[0, :].shape > (7,) > >>> np.array([True, False, True, False]).shape # Not the same shape! > (4,) > >>> y[0, :][np.array([True, False, True, False])] # But actually it > works -> [].nonzero() > array([0, 2]) > > Is there anything that I am misunderstanding? Can everyone shine the > light on me in this topic? > > Thanks in advance, regards > > Juan Luis Cano > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From dshean at gmail.com Mon Nov 19 03:28:06 2012 From: dshean at gmail.com (David Shean) Date: Mon, 19 Nov 2012 00:28:06 -0800 Subject: [SciPy-User] ndimage masked array handling Message-ID: <01D655C1-A8EC-44B6-953E-B6E4CA3F0E24@gmail.com> Hi all, I'm running into issues similar to those described in this ticket (opened 3 years ago): http://projects.scipy.org/scipy/ticket/1155 My workflows typically involve loading a geotif as a masked array: #Given input dataset, return a masked array for the input band def fn_getma(src_fn, bnum=1, ndv=0): src_ds = gdal.Open(src_fn, gdal.GA_ReadOnly) b = src_ds.GetRasterBand(bnum) b_ndv = b.GetNoDataValue() if (b_ndv is not None): ndv = b_ndv bma = numpy.ma.masked_equal(b.ReadAsArray(), ndv) return bma The scipy.ndimage package is great, and I use it often. Unfortunately, the filters and interpolation routines can't handle masked arrays. I've had some success using the following hack, passing a masked array filled with numpy.nan to the desired scipy.ndimage function f_a: def nanfill(a, f_a, *args, **kwargs): ndv = a.fill_value b = f_a(a.filled(numpy.nan), *args, **kwargs) out = numpy.ma.fix_invalid(b, copy=False) out.set_fill_value(ndv) return out For example: b = fn_getma('img.tif') nanfill(b, scipy.ndimage.gaussian_filter, 2) This works, but the nan regions are improperly expanded. The desired behavior would be to ignore any nans within the filter window and return an output value using the remaining valid input values. Instead, nan is returned whenever a single nan is encountered within the filter window. My current issue is with scipy.ndimage.interpolation.map_coordinates. I need to interpolate values for non-integer indices over a masked array containing "holes": #Arrays containing pixel coordinates pX = array([ 521.61974178, 520.18007917, 531.9847472 , ..., 382.70468842, 382.04165718, 381.38110078]) pY = array([ 1551.08334089, 1540.31571092, 1626.53056875, ..., 2556.5575993 , 2562.73110509, 2568.90483958]) #Rounding to nearest int b[numpy.around(pY).astype(int), numpy.around(pX).astype(int)] masked_array(data = [194.732543945 187.755401611 192.118453979 ..., 308.895629883 311.699554443 306.658691406], mask = [False False False ..., False False False], fill_value = 0.0) #Interpolating with map_coordinates, uses default b.fill_value, which will be considered valid during interpolation scipy.ndimage.interpolation.map_coordinates(b, [pY, pX]) array([ 194.61734009, 187.88977051, 192.112854 , ..., 309.16894531, 312.19412231, 306.87319946], dtype=float32) #Fill holes with nan. Returns all nans, even for regions with sufficient valid values scipy.ndimage.interpolation.map_coordinates(b.filled(numpy.nan), [pY, pX]) array([ nan, nan, nan, ..., nan, nan, nan], dtype=float32) An alternative approach would be to fill all holes up front with inpainting, but I would prefer to limit my analysis to valid values. Hopefully this is all clear. Any other ideas for workarounds? Perhaps the ticket should be updated, or priority bumped? Thanks. -David From vs at it.uu.se Mon Nov 19 18:01:38 2012 From: vs at it.uu.se (Virgil Stokes) Date: Tue, 20 Nov 2012 00:01:38 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? Message-ID: <50AABA52.7020808@it.uu.se> I am using the latest versions of numpy (from numpy-1.7.0b2-win32-superpack-python2.7.exe) and scipy (from scipy-0.11.0-win32-superpack-python2.7.exe ) on a windows 7 (32-bit) platform. I have used import numpy as np q,r = np.linalg.qr(A) and compared the results to what I get from MATLAB (R2010B) [q,r] = qr(A) The q,r returned from numpy are both the negative of the q,r returned from MATLAB for the same matrix A. I believe that theq,r returned from MATLAB are correct. Why am I getting their negative from numpy? Note, I have tried this on several different matrices --- numpy always gives the negative of MATLAB's. From davidmenhur at gmail.com Tue Nov 20 16:33:30 2012 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Tue, 20 Nov 2012 22:33:30 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: <50AABA52.7020808@it.uu.se> References: <50AABA52.7020808@it.uu.se> Message-ID: The QR descomposition is finding two matrices with certain properties such that: A = Q?R But, if both Q and R are multiplied by -1, (-Q)?(-R) = Q?R = A, still the same matrix. If Q is orthogonal, -Q is also. The sign is, therefore, arbitrary. On Tue, Nov 20, 2012 at 12:01 AM, Virgil Stokes wrote: > I am using the latest versions of numpy (from > numpy-1.7.0b2-win32-superpack-python2.7.exe) and scipy (from > scipy-0.11.0-win32-superpack-python2.7.exe ) on a windows 7 (32-bit) > platform. > > I have used > > import numpy as np > q,r = np.linalg.qr(A) > > and compared the results to what I get from MATLAB (R2010B) > > [q,r] = qr(A) > > The q,r returned from numpy are both the negative of the q,r returned > from MATLAB for the same matrix A. I believe that theq,r returned from > MATLAB are correct. Why am I getting their negative from numpy? > > Note, I have tried this on several different matrices --- numpy always > gives the negative of MATLAB's. > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From vs at it.uu.se Tue Nov 20 17:03:54 2012 From: vs at it.uu.se (Virgil Stokes) Date: Tue, 20 Nov 2012 23:03:54 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: References: <50AABA52.7020808@it.uu.se> Message-ID: <50ABFE4A.3040308@it.uu.se> On 2012-11-20 22:33, Da?id wrote: > The QR descomposition is finding two matrices with certain properties such that: > > A = Q?R > > But, if both Q and R are multiplied by -1, (-Q)?(-R) = Q?R = A, still > the same matrix. If Q is orthogonal, -Q is also. The sign is, > therefore, arbitrary. > > On Tue, Nov 20, 2012 at 12:01 AM, Virgil Stokes wrote: >> I am using the latest versions of numpy (from >> numpy-1.7.0b2-win32-superpack-python2.7.exe) and scipy (from >> scipy-0.11.0-win32-superpack-python2.7.exe ) on a windows 7 (32-bit) >> platform. >> >> I have used >> >> import numpy as np >> q,r = np.linalg.qr(A) >> >> and compared the results to what I get from MATLAB (R2010B) >> >> [q,r] = qr(A) >> >> The q,r returned from numpy are both the negative of the q,r returned >> from MATLAB for the same matrix A. I believe that theq,r returned from >> MATLAB are correct. Why am I getting their negative from numpy? >> >> Note, I have tried this on several different matrices --- numpy always >> gives the negative of MATLAB's. >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user Thanks David, I am well aware of this; but, I am using the QR decomposition for a convariance (PD matrix) and the negative R is not very useful in this case and the numpy result, IMHO should not be the default. Why is numpy/Python different from that returned by MATLAB and MATHEMATICA? This makes translations rather tricky and one begins to wonder if there are other differences. From jsseabold at gmail.com Tue Nov 20 17:13:19 2012 From: jsseabold at gmail.com (Skipper Seabold) Date: Tue, 20 Nov 2012 17:13:19 -0500 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: <50ABFE4A.3040308@it.uu.se> References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> Message-ID: On Tue, Nov 20, 2012 at 5:03 PM, Virgil Stokes wrote: > > On 2012-11-20 22:33, Da?id wrote: > > The QR descomposition is finding two matrices with certain properties such that: > > > > A = Q?R > > > > But, if both Q and R are multiplied by -1, (-Q)?(-R) = Q?R = A, still > > the same matrix. If Q is orthogonal, -Q is also. The sign is, > > therefore, arbitrary. > > > > On Tue, Nov 20, 2012 at 12:01 AM, Virgil Stokes wrote: > >> I am using the latest versions of numpy (from > >> numpy-1.7.0b2-win32-superpack-python2.7.exe) and scipy (from > >> scipy-0.11.0-win32-superpack-python2.7.exe ) on a windows 7 (32-bit) > >> platform. > >> > >> I have used > >> > >> import numpy as np > >> q,r = np.linalg.qr(A) > >> > >> and compared the results to what I get from MATLAB (R2010B) > >> > >> [q,r] = qr(A) > >> > >> The q,r returned from numpy are both the negative of the q,r returned > >> from MATLAB for the same matrix A. I believe that theq,r returned from > >> MATLAB are correct. Why am I getting their negative from numpy? > >> > >> Note, I have tried this on several different matrices --- numpy always > >> gives the negative of MATLAB's. > >> > >> _______________________________________________ > >> SciPy-User mailing list > >> SciPy-User at scipy.org > >> http://mail.scipy.org/mailman/listinfo/scipy-user > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > Thanks David, > I am well aware of this; but, I am using the QR decomposition for a > convariance (PD matrix) and the negative R is not very useful in this > case and the numpy result, IMHO should not be the default. > Can't you guard against this in your code? > Why is numpy/Python different from that returned by MATLAB and > MATHEMATICA? This makes translations rather tricky and one begins to > wonder if there are other differences. It can often depend on the version of the underlying LAPACK functions used (or maybe even where/how it was compiled). In my experience, I've seen linear algebra functions in MATLAB give different results up to an arbitrary sign when I know for a fact they were using the same underling LAPACK routine. I later upgraded the LAPACK that I used to build scipy and the signs agreed. I do not know if MATLAB does any kind of normalization after the fact, but you could file an issue or better yet provide a PR for the sign check in scipy if it's something you don't want to check for in your code in the future. The beauty of scipy is that you can look at the code to see why you're getting the results you're getting. You can find out the LAPACK version and then look at the helper functions that calls these routines to see what's going on. Good luck figuring that out with MATLAB, etc. Skipper Skipper From charlesr.harris at gmail.com Tue Nov 20 17:43:05 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 20 Nov 2012 15:43:05 -0700 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: <50ABFE4A.3040308@it.uu.se> References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> Message-ID: On Tue, Nov 20, 2012 at 3:03 PM, Virgil Stokes wrote: > On 2012-11-20 22:33, Da?id wrote: > > The QR descomposition is finding two matrices with certain properties > such that: > > > > A = Q?R > > > > But, if both Q and R are multiplied by -1, (-Q)?(-R) = Q?R = A, still > > the same matrix. If Q is orthogonal, -Q is also. The sign is, > > therefore, arbitrary. > > > > On Tue, Nov 20, 2012 at 12:01 AM, Virgil Stokes wrote: > >> I am using the latest versions of numpy (from > >> numpy-1.7.0b2-win32-superpack-python2.7.exe) and scipy (from > >> scipy-0.11.0-win32-superpack-python2.7.exe ) on a windows 7 (32-bit) > >> platform. > >> > >> I have used > >> > >> import numpy as np > >> q,r = np.linalg.qr(A) > >> > >> and compared the results to what I get from MATLAB (R2010B) > >> > >> [q,r] = qr(A) > >> > >> The q,r returned from numpy are both the negative of the q,r returned > >> from MATLAB for the same matrix A. I believe that theq,r returned from > >> MATLAB are correct. Why am I getting their negative from numpy? > >> > >> Note, I have tried this on several different matrices --- numpy always > >> gives the negative of MATLAB's. > >> > >> _______________________________________________ > >> SciPy-User mailing list > >> SciPy-User at scipy.org > >> http://mail.scipy.org/mailman/listinfo/scipy-user > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > Thanks David, > I am well aware of this; but, I am using the QR decomposition for a > convariance (PD matrix) and the negative R is not very useful in this > case and the numpy result, IMHO should not be the default. > What is your application? I don't see that it should matter for most things, you are just using a slightly different set of basis vectors in the q. Is orientation something you are concerned about? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From vs at it.uu.se Tue Nov 20 17:49:34 2012 From: vs at it.uu.se (Virgil Stokes) Date: Tue, 20 Nov 2012 23:49:34 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> Message-ID: <50AC08FE.7050703@it.uu.se> On 2012-11-20 23:13, Skipper Seabold wrote: > On Tue, Nov 20, 2012 at 5:03 PM, Virgil Stokes wrote: >> On 2012-11-20 22:33, Da?id wrote: >>> The QR descomposition is finding two matrices with certain properties such that: >>> >>> A = Q?R >>> >>> But, if both Q and R are multiplied by -1, (-Q)?(-R) = Q?R = A, still >>> the same matrix. If Q is orthogonal, -Q is also. The sign is, >>> therefore, arbitrary. >>> >>> On Tue, Nov 20, 2012 at 12:01 AM, Virgil Stokes wrote: >>>> I am using the latest versions of numpy (from >>>> numpy-1.7.0b2-win32-superpack-python2.7.exe) and scipy (from >>>> scipy-0.11.0-win32-superpack-python2.7.exe ) on a windows 7 (32-bit) >>>> platform. >>>> >>>> I have used >>>> >>>> import numpy as np >>>> q,r = np.linalg.qr(A) >>>> >>>> and compared the results to what I get from MATLAB (R2010B) >>>> >>>> [q,r] = qr(A) >>>> >>>> The q,r returned from numpy are both the negative of the q,r returned >>>> from MATLAB for the same matrix A. I believe that theq,r returned from >>>> MATLAB are correct. Why am I getting their negative from numpy? >>>> >>>> Note, I have tried this on several different matrices --- numpy always >>>> gives the negative of MATLAB's. >>>> >>>> _______________________________________________ >>>> SciPy-User mailing list >>>> SciPy-User at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >> Thanks David, >> I am well aware of this; but, I am using the QR decomposition for a >> convariance (PD matrix) and the negative R is not very useful in this >> case and the numpy result, IMHO should not be the default. >> > Can't you guard against this in your code? > >> Why is numpy/Python different from that returned by MATLAB and >> MATHEMATICA? This makes translations rather tricky and one begins to >> wonder if there are other differences. > It can often depend on the version of the underlying LAPACK functions > used (or maybe even where/how it was compiled). In my experience, I've > seen linear algebra functions in MATLAB give different results up to > an arbitrary sign when I know for a fact they were using the same > underling LAPACK routine. I later upgraded the LAPACK that I used to > build scipy and the signs agreed. I do not know if MATLAB does any > kind of normalization after the fact, but you could file an issue or > better yet provide a PR for the sign check in scipy if it's something > you don't want to check for in your code in the future. The beauty of > scipy is that you can look at the code to see why you're getting the > results you're getting. You can find out the LAPACK version and then > look at the helper functions that calls these routines to see what's > going on. Good luck figuring that out with MATLAB, etc. > > Skipper > > Skipper > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user Ok Skipper, Unfortunately, things are worse than I had hoped, numpy sometimes returns the negative of the q,r and other times the same as MATLAB! Thus, as someone has already mentioned in this discussion, the "sign" seems to depend on the matrix being decomposed. This could be a nightmare to track down. I hope that I can return to some older versions of numpy/scipy to work around this problem until this problem is fixed. Any suggestions on how to recover earlier versions would be appreciated. From charlesr.harris at gmail.com Tue Nov 20 17:57:40 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 20 Nov 2012 15:57:40 -0700 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: <50AC08FE.7050703@it.uu.se> References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC08FE.7050703@it.uu.se> Message-ID: On Tue, Nov 20, 2012 at 3:49 PM, Virgil Stokes wrote: > On 2012-11-20 23:13, Skipper Seabold wrote: > > On Tue, Nov 20, 2012 at 5:03 PM, Virgil Stokes wrote: > >> On 2012-11-20 22:33, Da?id wrote: > >>> The QR descomposition is finding two matrices with certain properties > such that: > >>> > >>> A = Q?R > >>> > >>> But, if both Q and R are multiplied by -1, (-Q)?(-R) = Q?R = A, still > >>> the same matrix. If Q is orthogonal, -Q is also. The sign is, > >>> therefore, arbitrary. > >>> > >>> On Tue, Nov 20, 2012 at 12:01 AM, Virgil Stokes wrote: > >>>> I am using the latest versions of numpy (from > >>>> numpy-1.7.0b2-win32-superpack-python2.7.exe) and scipy (from > >>>> scipy-0.11.0-win32-superpack-python2.7.exe ) on a windows 7 (32-bit) > >>>> platform. > >>>> > >>>> I have used > >>>> > >>>> import numpy as np > >>>> q,r = np.linalg.qr(A) > >>>> > >>>> and compared the results to what I get from MATLAB (R2010B) > >>>> > >>>> [q,r] = qr(A) > >>>> > >>>> The q,r returned from numpy are both the negative of the q,r returned > >>>> from MATLAB for the same matrix A. I believe that theq,r returned from > >>>> MATLAB are correct. Why am I getting their negative from numpy? > >>>> > >>>> Note, I have tried this on several different matrices --- numpy > always > >>>> gives the negative of MATLAB's. > >>>> > >>>> _______________________________________________ > >>>> SciPy-User mailing list > >>>> SciPy-User at scipy.org > >>>> http://mail.scipy.org/mailman/listinfo/scipy-user > >>> _______________________________________________ > >>> SciPy-User mailing list > >>> SciPy-User at scipy.org > >>> http://mail.scipy.org/mailman/listinfo/scipy-user > >> Thanks David, > >> I am well aware of this; but, I am using the QR decomposition for a > >> convariance (PD matrix) and the negative R is not very useful in this > >> case and the numpy result, IMHO should not be the default. > >> > > Can't you guard against this in your code? > > > >> Why is numpy/Python different from that returned by MATLAB and > >> MATHEMATICA? This makes translations rather tricky and one begins to > >> wonder if there are other differences. > > It can often depend on the version of the underlying LAPACK functions > > used (or maybe even where/how it was compiled). In my experience, I've > > seen linear algebra functions in MATLAB give different results up to > > an arbitrary sign when I know for a fact they were using the same > > underling LAPACK routine. I later upgraded the LAPACK that I used to > > build scipy and the signs agreed. I do not know if MATLAB does any > > kind of normalization after the fact, but you could file an issue or > > better yet provide a PR for the sign check in scipy if it's something > > you don't want to check for in your code in the future. The beauty of > > scipy is that you can look at the code to see why you're getting the > > results you're getting. You can find out the LAPACK version and then > > look at the helper functions that calls these routines to see what's > > going on. Good luck figuring that out with MATLAB, etc. > > > > Skipper > > > > Skipper > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > Ok Skipper, > Unfortunately, things are worse than I had hoped, numpy sometimes > returns the negative of the q,r and other times the same as MATLAB! > Thus, as someone has already mentioned in this discussion, the "sign" > seems to depend on the matrix being decomposed. This could be a > nightmare to track down. > > I hope that I can return to some older versions of numpy/scipy to work > around this problem until this problem is fixed. Any suggestions on how > to recover earlier versions would be appreciated. > But why is it a problem? Why is Matlab "right"? What is the property that you need to have in the decomposition? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From vs at it.uu.se Tue Nov 20 17:59:00 2012 From: vs at it.uu.se (Virgil Stokes) Date: Tue, 20 Nov 2012 23:59:00 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> Message-ID: <50AC0B34.6030200@it.uu.se> On 2012-11-20 23:43, Charles R Harris wrote: > > > On Tue, Nov 20, 2012 at 3:03 PM, Virgil Stokes > wrote: > > On 2012-11-20 22:33, Da?id wrote: > > The QR descomposition is finding two matrices with certain > properties such that: > > > > A = Q?R > > > > But, if both Q and R are multiplied by -1, (-Q)?(-R) = Q?R = A, > still > > the same matrix. If Q is orthogonal, -Q is also. The sign is, > > therefore, arbitrary. > > > > On Tue, Nov 20, 2012 at 12:01 AM, Virgil Stokes > wrote: > >> I am using the latest versions of numpy (from > >> numpy-1.7.0b2-win32-superpack-python2.7.exe) and scipy (from > >> scipy-0.11.0-win32-superpack-python2.7.exe ) on a windows 7 > (32-bit) > >> platform. > >> > >> I have used > >> > >> import numpy as np > >> q,r = np.linalg.qr(A) > >> > >> and compared the results to what I get from MATLAB (R2010B) > >> > >> [q,r] = qr(A) > >> > >> The q,r returned from numpy are both the negative of the q,r > returned > >> from MATLAB for the same matrix A. I believe that theq,r > returned from > >> MATLAB are correct. Why am I getting their negative from numpy? > >> > >> Note, I have tried this on several different matrices --- > numpy always > >> gives the negative of MATLAB's. > >> > >> _______________________________________________ > >> SciPy-User mailing list > >> SciPy-User at scipy.org > >> http://mail.scipy.org/mailman/listinfo/scipy-user > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > Thanks David, > I am well aware of this; but, I am using the QR decomposition for a > convariance (PD matrix) and the negative R is not very useful in this > case and the numpy result, IMHO should not be the default. > > > What is your application? My application is the propagation of the factorized R matrix in the Kalman filter, where the QR factorization is for the covariance matrix in the KF recursions. > I don't see that it should matter for most things, you > are just using a slightly different set of basis vectors in the q. Is > orientation something you > are concerned about? And it does make a lot of difference! I now have found that sometimes the "sign" of the factorization switches (wrt to the MATLAB version). The MATLAB QR factorization (however it may differ from that of numpy) is consistent in the sense there is no sign switching and the results obtained from the KF are correct (this I have verified). On the other hand, it is very unlikely that one can obtain the correct results with the current implementation of the QR factorization in Python. > > > > Chuck > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Nov 20 17:59:55 2012 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 20 Nov 2012 22:59:55 +0000 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: <50AC08FE.7050703@it.uu.se> References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC08FE.7050703@it.uu.se> Message-ID: On Tue, Nov 20, 2012 at 10:49 PM, Virgil Stokes wrote: > Ok Skipper, > Unfortunately, things are worse than I had hoped, numpy sometimes > returns the negative of the q,r and other times the same as MATLAB! > Thus, as someone has already mentioned in this discussion, the "sign" > seems to depend on the matrix being decomposed. This could be a > nightmare to track down. > > I hope that I can return to some older versions of numpy/scipy to work > around this problem until this problem is fixed. Any suggestions on how > to recover earlier versions would be appreciated. That's not going to help you. The only thing that we guarantee (or have *ever* guaranteed) is that the result is a valid QR decomposition. If you need to swap signs to normalize things to your desired convention, you will need to do that as a postprocessing step. -- Robert Kern From vs at it.uu.se Tue Nov 20 18:10:19 2012 From: vs at it.uu.se (Virgil Stokes) Date: Wed, 21 Nov 2012 00:10:19 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC08FE.7050703@it.uu.se> Message-ID: <50AC0DDB.708@it.uu.se> On 2012-11-20 23:57, Charles R Harris wrote: > > > On Tue, Nov 20, 2012 at 3:49 PM, Virgil Stokes > wrote: > > On 2012-11-20 23:13, Skipper Seabold wrote: > > On Tue, Nov 20, 2012 at 5:03 PM, Virgil Stokes > wrote: > >> On 2012-11-20 22:33, Da?id wrote: > >>> The QR descomposition is finding two matrices with certain > properties such that: > >>> > >>> A = Q?R > >>> > >>> But, if both Q and R are multiplied by -1, (-Q)?(-R) = Q?R = > A, still > >>> the same matrix. If Q is orthogonal, -Q is also. The sign is, > >>> therefore, arbitrary. > >>> > >>> On Tue, Nov 20, 2012 at 12:01 AM, Virgil Stokes > wrote: > >>>> I am using the latest versions of numpy (from > >>>> numpy-1.7.0b2-win32-superpack-python2.7.exe) and scipy (from > >>>> scipy-0.11.0-win32-superpack-python2.7.exe ) on a windows 7 > (32-bit) > >>>> platform. > >>>> > >>>> I have used > >>>> > >>>> import numpy as np > >>>> q,r = np.linalg.qr(A) > >>>> > >>>> and compared the results to what I get from MATLAB (R2010B) > >>>> > >>>> [q,r] = qr(A) > >>>> > >>>> The q,r returned from numpy are both the negative of the q,r > returned > >>>> from MATLAB for the same matrix A. I believe that theq,r > returned from > >>>> MATLAB are correct. Why am I getting their negative from numpy? > >>>> > >>>> Note, I have tried this on several different matrices --- > numpy always > >>>> gives the negative of MATLAB's. > >>>> > >>>> _______________________________________________ > >>>> SciPy-User mailing list > >>>> SciPy-User at scipy.org > >>>> http://mail.scipy.org/mailman/listinfo/scipy-user > >>> _______________________________________________ > >>> SciPy-User mailing list > >>> SciPy-User at scipy.org > >>> http://mail.scipy.org/mailman/listinfo/scipy-user > >> Thanks David, > >> I am well aware of this; but, I am using the QR decomposition for a > >> convariance (PD matrix) and the negative R is not very useful > in this > >> case and the numpy result, IMHO should not be the default. > >> > > Can't you guard against this in your code? > > > >> Why is numpy/Python different from that returned by MATLAB and > >> MATHEMATICA? This makes translations rather tricky and one > begins to > >> wonder if there are other differences. > > It can often depend on the version of the underlying LAPACK > functions > > used (or maybe even where/how it was compiled). In my > experience, I've > > seen linear algebra functions in MATLAB give different results up to > > an arbitrary sign when I know for a fact they were using the same > > underling LAPACK routine. I later upgraded the LAPACK that I used to > > build scipy and the signs agreed. I do not know if MATLAB does any > > kind of normalization after the fact, but you could file an issue or > > better yet provide a PR for the sign check in scipy if it's > something > > you don't want to check for in your code in the future. The > beauty of > > scipy is that you can look at the code to see why you're getting the > > results you're getting. You can find out the LAPACK version and then > > look at the helper functions that calls these routines to see what's > > going on. Good luck figuring that out with MATLAB, etc. > > > > Skipper > > > > Skipper > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > Ok Skipper, > Unfortunately, things are worse than I had hoped, numpy sometimes > returns the negative of the q,r and other times the same as MATLAB! > Thus, as someone has already mentioned in this discussion, the "sign" > seems to depend on the matrix being decomposed. This could be a > nightmare to track down. > > I hope that I can return to some older versions of numpy/scipy to work > around this problem until this problem is fixed. Any suggestions > on how > to recover earlier versions would be appreciated. > > > But why is it a problem? Why is Matlab "right"? What is the property > that you > need to have in the decomposition? I have already posted an answer to your first question, Chuck. MATLAB is correct because the results of my application (Kalman filter using QR factorization of the covariance matrix) are correct -- I have verified this. The QR factorization is used to propagate the R matrix and clearly if the "sign" of R changes in an unpredictable manner (at least I have been unable to predict the "sign" changes that occur in numpy), then the answer is unlikely to be correct. I believe that I have answered your questions. If not, then you might look at a paper/tutorial/book that discusses the "square-root Kalman filter". Then, if you have any additional questions on my application, I will try my best to answer them for you. > > Chuck > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Nov 20 18:11:12 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 20 Nov 2012 16:11:12 -0700 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: <50AC0B34.6030200@it.uu.se> References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC0B34.6030200@it.uu.se> Message-ID: On Tue, Nov 20, 2012 at 3:59 PM, Virgil Stokes wrote: > On 2012-11-20 23:43, Charles R Harris wrote: > > > > On Tue, Nov 20, 2012 at 3:03 PM, Virgil Stokes wrote: > >> On 2012-11-20 22:33, Da?id wrote: >> > The QR descomposition is finding two matrices with certain properties >> such that: >> > >> > A = Q?R >> > >> > But, if both Q and R are multiplied by -1, (-Q)?(-R) = Q?R = A, still >> > the same matrix. If Q is orthogonal, -Q is also. The sign is, >> > therefore, arbitrary. >> > >> > On Tue, Nov 20, 2012 at 12:01 AM, Virgil Stokes wrote: >> >> I am using the latest versions of numpy (from >> >> numpy-1.7.0b2-win32-superpack-python2.7.exe) and scipy (from >> >> scipy-0.11.0-win32-superpack-python2.7.exe ) on a windows 7 (32-bit) >> >> platform. >> >> >> >> I have used >> >> >> >> import numpy as np >> >> q,r = np.linalg.qr(A) >> >> >> >> and compared the results to what I get from MATLAB (R2010B) >> >> >> >> [q,r] = qr(A) >> >> >> >> The q,r returned from numpy are both the negative of the q,r returned >> >> from MATLAB for the same matrix A. I believe that theq,r returned from >> >> MATLAB are correct. Why am I getting their negative from numpy? >> >> >> >> Note, I have tried this on several different matrices --- numpy always >> >> gives the negative of MATLAB's. >> >> >> >> _______________________________________________ >> >> SciPy-User mailing list >> >> SciPy-User at scipy.org >> >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > _______________________________________________ >> > SciPy-User mailing list >> > SciPy-User at scipy.org >> > http://mail.scipy.org/mailman/listinfo/scipy-user >> Thanks David, >> I am well aware of this; but, I am using the QR decomposition for a >> convariance (PD matrix) and the negative R is not very useful in this >> case and the numpy result, IMHO should not be the default. >> > > What is your application? > > My application is the propagation of the factorized R matrix in the Kalman > filter, where the QR factorization is for the covariance matrix in the KF > recursions. > That is what I suspected. However, the factorized matrices are usually U^t*D*U or U^t * U, so I think you are doing something wrong. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From vs at it.uu.se Tue Nov 20 18:21:00 2012 From: vs at it.uu.se (Virgil Stokes) Date: Wed, 21 Nov 2012 00:21:00 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC08FE.7050703@it.uu.se> Message-ID: <50AC105C.5090401@it.uu.se> On 2012-11-20 23:59, Robert Kern wrote: > On Tue, Nov 20, 2012 at 10:49 PM, Virgil Stokes wrote: > >> Ok Skipper, >> Unfortunately, things are worse than I had hoped, numpy sometimes >> returns the negative of the q,r and other times the same as MATLAB! >> Thus, as someone has already mentioned in this discussion, the "sign" >> seems to depend on the matrix being decomposed. This could be a >> nightmare to track down. >> >> I hope that I can return to some older versions of numpy/scipy to work >> around this problem until this problem is fixed. Any suggestions on how >> to recover earlier versions would be appreciated. > That's not going to help you. The only thing that we guarantee (or > have *ever* guaranteed) is that the result is a valid QR > decomposition. If you need to swap signs to normalize things to your > desired convention, you will need to do that as a postprocessing step. But why do I need to normalize with numpy (at least with latest release); but not with MATLAB. A simple question for you. In my application MATLAB generates a sequence of QR factorizations for covariance matrices in which R is always PD --- which is corect! For the same application, numpy generate a sequence of QR factorizations for covariance matrices in which R is not always PD. How can I predict when I will get an R that is not PD? > > -- > Robert Kern > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From vs at it.uu.se Tue Nov 20 18:26:19 2012 From: vs at it.uu.se (Virgil Stokes) Date: Wed, 21 Nov 2012 00:26:19 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC0B34.6030200@it.uu.se> Message-ID: <50AC119B.1040603@it.uu.se> On 2012-11-21 00:11, Charles R Harris wrote: > > > On Tue, Nov 20, 2012 at 3:59 PM, Virgil Stokes > wrote: > > On 2012-11-20 23:43, Charles R Harris wrote: >> >> >> On Tue, Nov 20, 2012 at 3:03 PM, Virgil Stokes > > wrote: >> >> On 2012-11-20 22:33, Da?id wrote: >> > The QR descomposition is finding two matrices with certain >> properties such that: >> > >> > A = Q?R >> > >> > But, if both Q and R are multiplied by -1, (-Q)?(-R) = Q?R >> = A, still >> > the same matrix. If Q is orthogonal, -Q is also. The sign is, >> > therefore, arbitrary. >> > >> > On Tue, Nov 20, 2012 at 12:01 AM, Virgil Stokes >> > wrote: >> >> I am using the latest versions of numpy (from >> >> numpy-1.7.0b2-win32-superpack-python2.7.exe) and scipy (from >> >> scipy-0.11.0-win32-superpack-python2.7.exe ) on a windows >> 7 (32-bit) >> >> platform. >> >> >> >> I have used >> >> >> >> import numpy as np >> >> q,r = np.linalg.qr(A) >> >> >> >> and compared the results to what I get from MATLAB (R2010B) >> >> >> >> [q,r] = qr(A) >> >> >> >> The q,r returned from numpy are both the negative of the >> q,r returned >> >> from MATLAB for the same matrix A. I believe that theq,r >> returned from >> >> MATLAB are correct. Why am I getting their negative from >> numpy? >> >> >> >> Note, I have tried this on several different matrices --- >> numpy always >> >> gives the negative of MATLAB's. >> >> >> >> _______________________________________________ >> >> SciPy-User mailing list >> >> SciPy-User at scipy.org >> >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > _______________________________________________ >> > SciPy-User mailing list >> > SciPy-User at scipy.org >> > http://mail.scipy.org/mailman/listinfo/scipy-user >> Thanks David, >> I am well aware of this; but, I am using the QR decomposition >> for a >> convariance (PD matrix) and the negative R is not very useful >> in this >> case and the numpy result, IMHO should not be the default. >> >> >> What is your application? > My application is the propagation of the factorized R matrix in > the Kalman filter, where the QR factorization is for the > covariance matrix in the KF recursions. > > > That is what I suspected. However, the factorized matrices are usually > U^t*D*U or U^t * U, so I think you are doing something wrong. No Chuck, You are referring to Bierman's factorization which is just one of the factorizations possible. I am using a standard and well-documented form of the so-called "square-root" Kalman filters (just Google on this and be enlightened). Again, there many papers/books that discuss the QR factorization implementation for both the Kalman filter and Kalman smoother. > > Chuck > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Nov 20 18:29:31 2012 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 20 Nov 2012 23:29:31 +0000 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: <50AC105C.5090401@it.uu.se> References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC08FE.7050703@it.uu.se> <50AC105C.5090401@it.uu.se> Message-ID: On Tue, Nov 20, 2012 at 11:21 PM, Virgil Stokes wrote: > On 2012-11-20 23:59, Robert Kern wrote: >> On Tue, Nov 20, 2012 at 10:49 PM, Virgil Stokes wrote: >> >>> Ok Skipper, >>> Unfortunately, things are worse than I had hoped, numpy sometimes >>> returns the negative of the q,r and other times the same as MATLAB! >>> Thus, as someone has already mentioned in this discussion, the "sign" >>> seems to depend on the matrix being decomposed. This could be a >>> nightmare to track down. >>> >>> I hope that I can return to some older versions of numpy/scipy to work >>> around this problem until this problem is fixed. Any suggestions on how >>> to recover earlier versions would be appreciated. >> That's not going to help you. The only thing that we guarantee (or >> have *ever* guaranteed) is that the result is a valid QR >> decomposition. If you need to swap signs to normalize things to your >> desired convention, you will need to do that as a postprocessing step. > But why do I need to normalize with numpy (at least with latest > release); but not with MATLAB. Because MATLAB decided to do the normalization step for you. That's a valid decision. And so is ours. > A simple question for you. > > In my application MATLAB generates a sequence of QR factorizations for > covariance matrices in which R is always PD --- which is corect! That is not part of the definition of a QR decomposition. Failing to meet that property does not make the QR decomposition incorrect. The only thing that is incorrect is passing an arbitrary, but valid, QR decomposition to something that is expecting a strict *subset* of valid QR decompositions. -- Robert Kern From alejandro.weinstein at gmail.com Tue Nov 20 18:37:09 2012 From: alejandro.weinstein at gmail.com (Alejandro Weinstein) Date: Tue, 20 Nov 2012 16:37:09 -0700 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: <50AC119B.1040603@it.uu.se> References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC0B34.6030200@it.uu.se> <50AC119B.1040603@it.uu.se> Message-ID: On Tue, Nov 20, 2012 at 4:26 PM, Virgil Stokes wrote: > No Chuck, > You are referring to Bierman's factorization which is just one of the > factorizations possible. I am using a standard and well-documented form of > the so-called "square-root" Kalman filters (just Google on this and be > enlightened). Again, there many papers/books that discuss the QR > factorization implementation for both the Kalman filter and Kalman smoother. Can you show the particular implementation you're using? According to Wikipedia [1] there are a few alternatives that can be classified as "square root" KF. Alejandro. [1] http://en.wikipedia.org/wiki/Kalman_filter#Square_root_form From charlesr.harris at gmail.com Tue Nov 20 18:55:16 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 20 Nov 2012 16:55:16 -0700 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: <50AC119B.1040603@it.uu.se> References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC0B34.6030200@it.uu.se> <50AC119B.1040603@it.uu.se> Message-ID: On Tue, Nov 20, 2012 at 4:26 PM, Virgil Stokes wrote: > On 2012-11-21 00:11, Charles R Harris wrote: > > > > On Tue, Nov 20, 2012 at 3:59 PM, Virgil Stokes wrote: > >> On 2012-11-20 23:43, Charles R Harris wrote: >> >> >> >> On Tue, Nov 20, 2012 at 3:03 PM, Virgil Stokes wrote: >> >>> On 2012-11-20 22:33, Da?id wrote: >>> > The QR descomposition is finding two matrices with certain properties >>> such that: >>> > >>> > A = Q?R >>> > >>> > But, if both Q and R are multiplied by -1, (-Q)?(-R) = Q?R = A, still >>> > the same matrix. If Q is orthogonal, -Q is also. The sign is, >>> > therefore, arbitrary. >>> > >>> > On Tue, Nov 20, 2012 at 12:01 AM, Virgil Stokes wrote: >>> >> I am using the latest versions of numpy (from >>> >> numpy-1.7.0b2-win32-superpack-python2.7.exe) and scipy (from >>> >> scipy-0.11.0-win32-superpack-python2.7.exe ) on a windows 7 (32-bit) >>> >> platform. >>> >> >>> >> I have used >>> >> >>> >> import numpy as np >>> >> q,r = np.linalg.qr(A) >>> >> >>> >> and compared the results to what I get from MATLAB (R2010B) >>> >> >>> >> [q,r] = qr(A) >>> >> >>> >> The q,r returned from numpy are both the negative of the q,r returned >>> >> from MATLAB for the same matrix A. I believe that theq,r returned from >>> >> MATLAB are correct. Why am I getting their negative from numpy? >>> >> >>> >> Note, I have tried this on several different matrices --- numpy >>> always >>> >> gives the negative of MATLAB's. >>> >> >>> >> _______________________________________________ >>> >> SciPy-User mailing list >>> >> SciPy-User at scipy.org >>> >> http://mail.scipy.org/mailman/listinfo/scipy-user >>> > _______________________________________________ >>> > SciPy-User mailing list >>> > SciPy-User at scipy.org >>> > http://mail.scipy.org/mailman/listinfo/scipy-user >>> Thanks David, >>> I am well aware of this; but, I am using the QR decomposition for a >>> convariance (PD matrix) and the negative R is not very useful in this >>> case and the numpy result, IMHO should not be the default. >>> >> >> What is your application? >> >> My application is the propagation of the factorized R matrix in the >> Kalman filter, where the QR factorization is for the covariance matrix in >> the KF recursions. >> > > That is what I suspected. However, the factorized matrices are usually > U^t*D*U or U^t * U, so I think you are doing something wrong. > > No Chuck, > You are referring to Bierman's factorization which is just one of the > factorizations possible. I am using a standard and well-documented form of > the so-called "square-root" Kalman filters (just Google on this and be > enlightened). Again, there many papers/books that discuss the QR > factorization implementation for both the Kalman filter and Kalman smoother. > Yes I am familiar with square root Kalman filters, I've even written a few. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From vs at it.uu.se Tue Nov 20 19:36:08 2012 From: vs at it.uu.se (Virgil Stokes) Date: Wed, 21 Nov 2012 01:36:08 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC08FE.7050703@it.uu.se> <50AC105C.5090401@it.uu.se> Message-ID: <50AC21F8.9050106@it.uu.se> On 2012-11-21 00:29, Robert Kern wrote: > On Tue, Nov 20, 2012 at 11:21 PM, Virgil Stokes wrote: >> On 2012-11-20 23:59, Robert Kern wrote: >>> On Tue, Nov 20, 2012 at 10:49 PM, Virgil Stokes wrote: >>> >>>> Ok Skipper, >>>> Unfortunately, things are worse than I had hoped, numpy sometimes >>>> returns the negative of the q,r and other times the same as MATLAB! >>>> Thus, as someone has already mentioned in this discussion, the "sign" >>>> seems to depend on the matrix being decomposed. This could be a >>>> nightmare to track down. >>>> >>>> I hope that I can return to some older versions of numpy/scipy to work >>>> around this problem until this problem is fixed. Any suggestions on how >>>> to recover earlier versions would be appreciated. >>> That's not going to help you. The only thing that we guarantee (or >>> have *ever* guaranteed) is that the result is a valid QR >>> decomposition. If you need to swap signs to normalize things to your >>> desired convention, you will need to do that as a postprocessing step. >> But why do I need to normalize with numpy (at least with latest >> release); but not with MATLAB. > Because MATLAB decided to do the normalization step for you. That's a > valid decision. And so is ours. > >> A simple question for you. >> >> In my application MATLAB generates a sequence of QR factorizations for >> covariance matrices in which R is always PD --- which is corect! > That is not part of the definition of a QR decomposition. Failing to > meet that property does not make the QR decomposition incorrect. > > The only thing that is incorrect is passing an arbitrary, but valid, > QR decomposition to something that is expecting a strict *subset* of > valid QR decompositions. Sorry but I do not understand this... Let me give you an example that I believe illustrates the problem in numpy I have the following matrix, A: array([[ 7.07106781e+02, 5.49702852e-04, 1.66675481e-19], [ -3.53553391e+01, 7.07104659e+01, 1.66675481e-19], [ 0.00000000e+00, -3.97555166e+00, 7.07106781e-03], [ -7.07106781e+02, -6.48214647e-04, 1.66675481e-19], [ 3.53553391e+01, -7.07104226e+01, 1.66675481e-19], [ 0.00000000e+00, 3.97560687e+00, -7.07106781e-03], [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00], [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00], [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00]]) Note, this is clearly not a covariance matrix, but, it does contain a covariance matrix (3x3). I refer you to the paper for how this matrix was generated. Using np.linalg.qr(A) I get the following for R (3x3) which is "square-root" of the covariance matrix: array([[ -1.00124922e+03, 4.99289918e+00, 0.00000000e+00], [ 0.00000000e+00, -1.00033071e+02, 5.62045938e-04], [ 0.00000000e+00, 0.00000000e+00, -9.98419272e-03]]) which is clearly not PD, since the it's 3 eigenvalues (diagonal elements) are all negative. Now, if I use qr(A,0) in MATLAB: I get the following for R (3x3) 1001.24922, -4.99290, 0.00000 0.00000, 100.03307, -0.00056 -0.00000, 0.00000, 0.00998 This is obviously PD, as it should be, and gives the correct results. Note, it is the negative of the R obtained with numpy. I can provide other examples in which both R's obtained are the same and they both lead to correct results. That is, when the R's are different, the R obtained with MATLAB is always PD and always gives the correct end result, while the R with numpy is not PD and does not give the correct end result. I hope that this helps you to understand my problem better. If there are more details that you need then let me know what, please. From vs at it.uu.se Tue Nov 20 19:39:09 2012 From: vs at it.uu.se (Virgil Stokes) Date: Wed, 21 Nov 2012 01:39:09 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: <50AC21F8.9050106@it.uu.se> References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC08FE.7050703@it.uu.se> <50AC105C.5090401@it.uu.se> <50AC21F8.9050106@it.uu.se> Message-ID: <50AC22AD.4030507@it.uu.se> On 2012-11-21 01:36, Virgil Stokes wrote: > On 2012-11-21 00:29, Robert Kern wrote: >> On Tue, Nov 20, 2012 at 11:21 PM, Virgil Stokes wrote: >>> On 2012-11-20 23:59, Robert Kern wrote: >>>> On Tue, Nov 20, 2012 at 10:49 PM, Virgil Stokes wrote: >>>> >>>>> Ok Skipper, >>>>> Unfortunately, things are worse than I had hoped, numpy sometimes >>>>> returns the negative of the q,r and other times the same as MATLAB! >>>>> Thus, as someone has already mentioned in this discussion, the "sign" >>>>> seems to depend on the matrix being decomposed. This could be a >>>>> nightmare to track down. >>>>> >>>>> I hope that I can return to some older versions of numpy/scipy to >>>>> work >>>>> around this problem until this problem is fixed. Any suggestions >>>>> on how >>>>> to recover earlier versions would be appreciated. >>>> That's not going to help you. The only thing that we guarantee (or >>>> have *ever* guaranteed) is that the result is a valid QR >>>> decomposition. If you need to swap signs to normalize things to your >>>> desired convention, you will need to do that as a postprocessing step. >>> But why do I need to normalize with numpy (at least with latest >>> release); but not with MATLAB. >> Because MATLAB decided to do the normalization step for you. That's a >> valid decision. And so is ours. >> >>> A simple question for you. >>> >>> In my application MATLAB generates a sequence of QR factorizations for >>> covariance matrices in which R is always PD --- which is corect! >> That is not part of the definition of a QR decomposition. Failing to >> meet that property does not make the QR decomposition incorrect. >> >> The only thing that is incorrect is passing an arbitrary, but valid, >> QR decomposition to something that is expecting a strict *subset* of >> valid QR decompositions. > Sorry but I do not understand this... > Let me give you an example that I believe illustrates the problem in > numpy > > I have the following matrix, A: > > array([[ 7.07106781e+02, 5.49702852e-04, 1.66675481e-19], > [ -3.53553391e+01, 7.07104659e+01, 1.66675481e-19], > [ 0.00000000e+00, -3.97555166e+00, 7.07106781e-03], > [ -7.07106781e+02, -6.48214647e-04, 1.66675481e-19], > [ 3.53553391e+01, -7.07104226e+01, 1.66675481e-19], > [ 0.00000000e+00, 3.97560687e+00, -7.07106781e-03], > [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00], > [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00], > [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00]]) > > Note, this is clearly not a covariance matrix, but, it does contain a > covariance matrix (3x3). I refer you to the paper for how this matrix > was generated. > > Using np.linalg.qr(A) I get the following for R (3x3) which is > "square-root" of the covariance matrix: > > array([[ -1.00124922e+03, 4.99289918e+00, 0.00000000e+00], > [ 0.00000000e+00, -1.00033071e+02, 5.62045938e-04], > [ 0.00000000e+00, 0.00000000e+00, -9.98419272e-03]]) > > which is clearly not PD, since the it's 3 eigenvalues (diagonal > elements) are all negative. > > Now, if I use qr(A,0) in MATLAB: > > I get the following for R (3x3) > > 1001.24922, -4.99290, 0.00000 > 0.00000, 100.03307, -0.00056 > -0.00000, 0.00000, 0.00998 > > This is obviously PD, as it should be, and gives the correct results. > Note, it is the negative of the R obtained with numpy. > > I can provide other examples in which both R's obtained are the same > and they both lead to correct results. That is, when the R's are > different, the R obtained with MATLAB is always PD and always gives > the correct end result, while the R with numpy is not PD and does not > give the correct end result. > > I hope that this helps you to understand my problem better. If there > are more details that you need then let me know what, please. Sorry, I forgot the paper (it is attached). -------------- next part -------------- A non-text attachment was scrubbed... Name: Arasaratnam08_QKF-Sqrt_VG.pdf Type: application/unknown Size: 302193 bytes Desc: not available URL: From vs at it.uu.se Tue Nov 20 19:45:32 2012 From: vs at it.uu.se (Virgil Stokes) Date: Wed, 21 Nov 2012 01:45:32 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC0B34.6030200@it.uu.se> <50AC119B.1040603@it.uu.se> Message-ID: <50AC242C.50306@it.uu.se> On 2012-11-21 00:37, Alejandro Weinstein wrote: > On Tue, Nov 20, 2012 at 4:26 PM, Virgil Stokes wrote: >> No Chuck, >> You are referring to Bierman's factorization which is just one of the >> factorizations possible. I am using a standard and well-documented form of >> the so-called "square-root" Kalman filters (just Google on this and be >> enlightened). Again, there many papers/books that discuss the QR >> factorization implementation for both the Kalman filter and Kalman smoother. > Can you show the particular implementation you're using? According to > Wikipedia [1] there are a few alternatives that can be classified as > "square root" KF. > > Alejandro. > > [1] http://en.wikipedia.org/wiki/Kalman_filter#Square_root_form > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user Yes, I have just sent an email with a paper attached that shows the method that I have implemented. Note, again I have no trouble with the MATLAB code --- it works correctly (according to the author of the paper). And, I was able to isolate the problem with the numpy/Python implementation to the QR factorization obtained from numpy.linalg.qr. From alejandro.weinstein at gmail.com Tue Nov 20 19:48:35 2012 From: alejandro.weinstein at gmail.com (Alejandro Weinstein) Date: Tue, 20 Nov 2012 17:48:35 -0700 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: <50AC21F8.9050106@it.uu.se> References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC08FE.7050703@it.uu.se> <50AC105C.5090401@it.uu.se> <50AC21F8.9050106@it.uu.se> Message-ID: On Tue, Nov 20, 2012 at 5:36 PM, Virgil Stokes wrote: > Using np.linalg.qr(A) I get the following for R (3x3) which is > "square-root" of the covariance matrix: > > array([[ -1.00124922e+03, 4.99289918e+00, 0.00000000e+00], > [ 0.00000000e+00, -1.00033071e+02, 5.62045938e-04], > [ 0.00000000e+00, 0.00000000e+00, -9.98419272e-03]]) > > which is clearly not PD, since the it's 3 eigenvalues (diagonal > elements) are all negative. But why you expect R to be PD? The QR decomposition [1] is A = QR with Q^T Q = I and R upper diagonal. [1] http://en.wikipedia.org/wiki/QR_factorization From vs at it.uu.se Tue Nov 20 19:49:44 2012 From: vs at it.uu.se (Virgil Stokes) Date: Wed, 21 Nov 2012 01:49:44 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC0B34.6030200@it.uu.se> <50AC119B.1040603@it.uu.se> Message-ID: <50AC2528.8000404@it.uu.se> On 2012-11-21 00:55, Charles R Harris wrote: > > > On Tue, Nov 20, 2012 at 4:26 PM, Virgil Stokes > wrote: > > On 2012-11-21 00:11, Charles R Harris wrote: >> >> >> On Tue, Nov 20, 2012 at 3:59 PM, Virgil Stokes > > wrote: >> >> On 2012-11-20 23:43, Charles R Harris wrote: >>> >>> >>> On Tue, Nov 20, 2012 at 3:03 PM, Virgil Stokes >> > wrote: >>> >>> On 2012-11-20 22:33, Da?id wrote: >>> > The QR descomposition is finding two matrices with >>> certain properties such that: >>> > >>> > A = Q?R >>> > >>> > But, if both Q and R are multiplied by -1, (-Q)?(-R) = >>> Q?R = A, still >>> > the same matrix. If Q is orthogonal, -Q is also. The >>> sign is, >>> > therefore, arbitrary. >>> > >>> > On Tue, Nov 20, 2012 at 12:01 AM, Virgil Stokes >>> > wrote: >>> >> I am using the latest versions of numpy (from >>> >> numpy-1.7.0b2-win32-superpack-python2.7.exe) and >>> scipy (from >>> >> scipy-0.11.0-win32-superpack-python2.7.exe ) on a >>> windows 7 (32-bit) >>> >> platform. >>> >> >>> >> I have used >>> >> >>> >> import numpy as np >>> >> q,r = np.linalg.qr(A) >>> >> >>> >> and compared the results to what I get from MATLAB >>> (R2010B) >>> >> >>> >> [q,r] = qr(A) >>> >> >>> >> The q,r returned from numpy are both the negative of >>> the q,r returned >>> >> from MATLAB for the same matrix A. I believe that >>> theq,r returned from >>> >> MATLAB are correct. Why am I getting their negative >>> from numpy? >>> >> >>> >> Note, I have tried this on several different matrices >>> --- numpy always >>> >> gives the negative of MATLAB's. >>> >> >>> >> _______________________________________________ >>> >> SciPy-User mailing list >>> >> SciPy-User at scipy.org >>> >> http://mail.scipy.org/mailman/listinfo/scipy-user >>> > _______________________________________________ >>> > SciPy-User mailing list >>> > SciPy-User at scipy.org >>> > http://mail.scipy.org/mailman/listinfo/scipy-user >>> Thanks David, >>> I am well aware of this; but, I am using the QR >>> decomposition for a >>> convariance (PD matrix) and the negative R is not very >>> useful in this >>> case and the numpy result, IMHO should not be the default. >>> >>> >>> What is your application? >> My application is the propagation of the factorized R matrix >> in the Kalman filter, where the QR factorization is for the >> covariance matrix in the KF recursions. >> >> >> That is what I suspected. However, the factorized matrices are >> usually U^t*D*U or U^t * U, so I think you are doing something wrong. > No Chuck, > You are referring to Bierman's factorization which is just one of > the factorizations possible. I am using a standard and > well-documented form of the so-called "square-root" Kalman filters > (just Google on this and be enlightened). Again, there many > papers/books that discuss the QR factorization implementation for > both the Kalman filter and Kalman smoother. > > > Yes I am familiar with square root Kalman filters, I've even written a > few. > > Chuck > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user Sorry, It is now 01:49 in Stockholm and I need to get a few hours of sleep. I will try to respond to further comments/questions on this topic tomorrow morning. Thanks for looking at this problem. I believe that this is a serious issue that does deserve some attention. Best, V -------------- next part -------------- An HTML attachment was scrubbed... URL: From vs at it.uu.se Tue Nov 20 19:56:13 2012 From: vs at it.uu.se (Virgil Stokes) Date: Wed, 21 Nov 2012 01:56:13 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC08FE.7050703@it.uu.se> <50AC105C.5090401@it.uu.se> <50AC21F8.9050106@it.uu.se> Message-ID: <50AC26AD.2080904@it.uu.se> On 2012-11-21 01:48, Alejandro Weinstein wrote: > On Tue, Nov 20, 2012 at 5:36 PM, Virgil Stokes wrote: >> Using np.linalg.qr(A) I get the following for R (3x3) which is >> "square-root" of the covariance matrix: >> >> array([[ -1.00124922e+03, 4.99289918e+00, 0.00000000e+00], >> [ 0.00000000e+00, -1.00033071e+02, 5.62045938e-04], >> [ 0.00000000e+00, 0.00000000e+00, -9.98419272e-03]]) >> >> which is clearly not PD, since the it's 3 eigenvalues (diagonal >> elements) are all negative. > But why you expect R to be PD? Because R*R^T = P (a covariance matrix). One important reason for using the QR factorization in the KF is to ensure that R is always PD during the recursions. > The QR decomposition [1] is > > A = QR with Q^T Q = I and R upper diagonal. > > [1] http://en.wikipedia.org/wiki/QR_factorization > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From jsseabold at gmail.com Tue Nov 20 20:17:25 2012 From: jsseabold at gmail.com (Skipper Seabold) Date: Tue, 20 Nov 2012 20:17:25 -0500 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: <50AC21F8.9050106@it.uu.se> References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC08FE.7050703@it.uu.se> <50AC105C.5090401@it.uu.se> <50AC21F8.9050106@it.uu.se> Message-ID: On Tue, Nov 20, 2012 at 7:36 PM, Virgil Stokes wrote: > On 2012-11-21 00:29, Robert Kern wrote: >> On Tue, Nov 20, 2012 at 11:21 PM, Virgil Stokes wrote: >>> On 2012-11-20 23:59, Robert Kern wrote: >>>> On Tue, Nov 20, 2012 at 10:49 PM, Virgil Stokes wrote: >>>> >>>>> Ok Skipper, >>>>> Unfortunately, things are worse than I had hoped, numpy sometimes >>>>> returns the negative of the q,r and other times the same as MATLAB! >>>>> Thus, as someone has already mentioned in this discussion, the "sign" >>>>> seems to depend on the matrix being decomposed. This could be a >>>>> nightmare to track down. >>>>> >>>>> I hope that I can return to some older versions of numpy/scipy to work >>>>> around this problem until this problem is fixed. Any suggestions on how >>>>> to recover earlier versions would be appreciated. >>>> That's not going to help you. The only thing that we guarantee (or >>>> have *ever* guaranteed) is that the result is a valid QR >>>> decomposition. If you need to swap signs to normalize things to your >>>> desired convention, you will need to do that as a postprocessing step. >>> But why do I need to normalize with numpy (at least with latest >>> release); but not with MATLAB. >> Because MATLAB decided to do the normalization step for you. That's a >> valid decision. And so is ours. >> >>> A simple question for you. >>> >>> In my application MATLAB generates a sequence of QR factorizations for >>> covariance matrices in which R is always PD --- which is corect! >> That is not part of the definition of a QR decomposition. Failing to >> meet that property does not make the QR decomposition incorrect. >> >> The only thing that is incorrect is passing an arbitrary, but valid, >> QR decomposition to something that is expecting a strict *subset* of >> valid QR decompositions. > Sorry but I do not understand this... > Let me give you an example that I believe illustrates the problem in numpy > > I have the following matrix, A: > > array([[ 7.07106781e+02, 5.49702852e-04, 1.66675481e-19], > [ -3.53553391e+01, 7.07104659e+01, 1.66675481e-19], > [ 0.00000000e+00, -3.97555166e+00, 7.07106781e-03], > [ -7.07106781e+02, -6.48214647e-04, 1.66675481e-19], > [ 3.53553391e+01, -7.07104226e+01, 1.66675481e-19], > [ 0.00000000e+00, 3.97560687e+00, -7.07106781e-03], > [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00], > [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00], > [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00]]) > > Note, this is clearly not a covariance matrix, but, it does contain a > covariance matrix (3x3). I refer you to the paper for how this matrix > was generated. > > Using np.linalg.qr(A) I get the following for R (3x3) which is > "square-root" of the covariance matrix: > > array([[ -1.00124922e+03, 4.99289918e+00, 0.00000000e+00], > [ 0.00000000e+00, -1.00033071e+02, 5.62045938e-04], > [ 0.00000000e+00, 0.00000000e+00, -9.98419272e-03]]) > > which is clearly not PD, since the it's 3 eigenvalues (diagonal > elements) are all negative. > > Now, if I use qr(A,0) in MATLAB: > > I get the following for R (3x3) > > 1001.24922, -4.99290, 0.00000 > 0.00000, 100.03307, -0.00056 > -0.00000, 0.00000, 0.00998 > > This is obviously PD, as it should be, and gives the correct results. > Note, it is the negative of the R obtained with numpy. > > I can provide other examples in which both R's obtained are the same and > they both lead to correct results. That is, when the R's are different, > the R obtained with MATLAB is always PD and always gives the correct end > result, while the R with numpy is not PD and does not give the correct > end result. > > I hope that this helps you to understand my problem better. If there are > more details that you need then let me know what, please. To recap. No one is disputing that MATLAB does a normalization that may make *some* use cases of a QR decomposition easier. However, *both* are correct answers. Nothing in the definition of a QR decomposition says R has to be positive definite. If you want R positive definite, then, as mentioned, you'll need to check this yourself and/or make a patch to numpy's QR function to do this to make sure you get a PD R. I happen to get a PD R in your example using LAPACK 3.3.1. I believe you can ensure R positive definite like so Q,R = np.linalg.qr(A) trace_r = np.trace(R) Q *= trace_r > 0 or -1 R *= trace_r > 0 or -1 Skipper Skipper From jason-sage at creativetrax.com Tue Nov 20 20:29:27 2012 From: jason-sage at creativetrax.com (Jason Grout) Date: Tue, 20 Nov 2012 19:29:27 -0600 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: <50AC2528.8000404@it.uu.se> References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC0B34.6030200@it.uu.se> <50AC119B.1040603@it.uu.se> <50AC2528.8000404@it.uu.se> Message-ID: <50AC2E77.2070906@creativetrax.com> On 11/20/12 6:49 PM, Virgil Stokes wrote: > Thanks for looking at this problem. I believe that this is a serious > issue that does deserve some attention. I'm curious: are you *guaranteed* by matlab (e.g., it's documented) to get a PD R matrix, or does it just happen in the cases you checked? If you aren't guaranteed it, then it seems very shaky to depend on it even from matlab. Of course, if it doesn't mention it in the docs, then who knows if it is guaranteed---matlab is closed source, so you can't just go check the code yourself. The way I read the docs at http://www.mathworks.com/help/matlab/ref/qr.html: "[Q,R] = qr(A), where A is m-by-n, produces an m-by-n upper triangular matrix R and an m-by-m unitary matrix Q so that A = Q*R." it sounds like you shouldn't depend on R being PD even from matlab, even if it's worked out that way in a few of your cases. Thanks, Jason From beamesleach at gmail.com Tue Nov 20 20:30:55 2012 From: beamesleach at gmail.com (Alex Leach) Date: Wed, 21 Nov 2012 01:30:55 +0000 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: <50AC21F8.9050106@it.uu.se> References: <50AABA52.7020808@it.uu.se> <50AC21F8.9050106@it.uu.se> Message-ID: <1422781.E2OrQfupyW@metabuntu> On Wednesday 21 Nov 2012 01:36:08 Virgil Stokes wrote: > > I have the following matrix, A: > > array([[ 7.07106781e+02, 5.49702852e-04, 1.66675481e-19], > [ -3.53553391e+01, 7.07104659e+01, 1.66675481e-19], > [ 0.00000000e+00, -3.97555166e+00, 7.07106781e-03], > [ -7.07106781e+02, -6.48214647e-04, 1.66675481e-19], > [ 3.53553391e+01, -7.07104226e+01, 1.66675481e-19], > [ 0.00000000e+00, 3.97560687e+00, -7.07106781e-03], > [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00], > [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00], > [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00]]) > > > Using np.linalg.qr(A) I get the following for R (3x3) which is > "square-root" of the covariance matrix: > > array([[ -1.00124922e+03, 4.99289918e+00, 0.00000000e+00], > [ 0.00000000e+00, -1.00033071e+02, 5.62045938e-04], > [ 0.00000000e+00, 0.00000000e+00, -9.98419272e-03]]) > I thought this would be down to the Lapack library you have linked against, as numpy uses Lapack's [zd]gqrf internally. I know Matlab comes distributed with Intel's MKL, so I imagine Matlab would use Intel's MKL routines internally. Without knowing anything about the math, here are some test cases on Linux and Mac OS X. Looks like it's dependent on platform, from these results... >>> a = np.array( ... ) >>> q, r = np.linalg.qr( a ) >>> r **Linux** Intel MKL: array([[ -1.00124922e+03, 4.99289919e+00, 2.40741243e-35], [ 0.00000000e+00, -1.00033071e+02, 5.62045938e-04], [ 0.00000000e+00, 0.00000000e+00, -9.98419272e-03]]) ATLAS Lapack: array([[ -1.00124922e+03, 4.99289919e+00, -2.40741243e-35], [ 0.00000000e+00, -1.00033071e+02, 5.62045938e-04], [ 0.00000000e+00, 0.00000000e+00, -9.98419272e-03]]) **Mac OS X**: Accelerate Framework:- array([[ 1.00124922e+03, -4.99289919e+00, 2.40741243e-35], [ 0.00000000e+00, 1.00033071e+02, -5.62045938e-04], [ 0.00000000e+00, 0.00000000e+00, 9.98419272e-03]]) ATLAS Lapack:- array([[ 1.00124922e+03, -4.99289919e+00, 2.40741243e-35], [ 0.00000000e+00, 1.00033071e+02, -5.62045938e-04], [ 0.00000000e+00, 0.00000000e+00, 9.98419272e-03]]) Cheers, Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From alejandro.weinstein at gmail.com Tue Nov 20 21:47:46 2012 From: alejandro.weinstein at gmail.com (Alejandro Weinstein) Date: Tue, 20 Nov 2012 19:47:46 -0700 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: <50AC26AD.2080904@it.uu.se> References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC08FE.7050703@it.uu.se> <50AC105C.5090401@it.uu.se> <50AC21F8.9050106@it.uu.se> <50AC26AD.2080904@it.uu.se> Message-ID: On Tue, Nov 20, 2012 at 5:56 PM, Virgil Stokes wrote: > On 2012-11-21 01:48, Alejandro Weinstein wrote: >> On Tue, Nov 20, 2012 at 5:36 PM, Virgil Stokes wrote: >>> which is clearly not PD, since the it's 3 eigenvalues (diagonal >>> elements) are all negative. >> But why you expect R to be PD? > Because R*R^T = P (a covariance matrix). One important reason for > using the QR factorization in the KF is to ensure that R is always PD > during the recursions. As you said, P = R * R^T, which is PD, even if R is not. Please check the definition of QR decomposition: R is _not_ required to be PD. Looking at the paper, they in fact use P = R * R ^ T, as in eq. (1). They never use R alone. So the fact that R is not PD should not be an issue. Can you show your code? I'm curious to see how the fact that R is not PD makes a difference. Alejandro. From vs at it.uu.se Wed Nov 21 02:36:38 2012 From: vs at it.uu.se (Virgil Stokes) Date: Wed, 21 Nov 2012 08:36:38 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC08FE.7050703@it.uu.se> <50AC105C.5090401@it.uu.se> <50AC21F8.9050106@it.uu.se> Message-ID: <50AC8486.9050808@it.uu.se> On 21-Nov-2012 02:17, Skipper Seabold wrote: > On Tue, Nov 20, 2012 at 7:36 PM, Virgil Stokes wrote: >> On 2012-11-21 00:29, Robert Kern wrote: >>> On Tue, Nov 20, 2012 at 11:21 PM, Virgil Stokes wrote: >>>> On 2012-11-20 23:59, Robert Kern wrote: >>>>> On Tue, Nov 20, 2012 at 10:49 PM, Virgil Stokes wrote: >>>>> >>>>>> Ok Skipper, >>>>>> Unfortunately, things are worse than I had hoped, numpy sometimes >>>>>> returns the negative of the q,r and other times the same as MATLAB! >>>>>> Thus, as someone has already mentioned in this discussion, the "sign" >>>>>> seems to depend on the matrix being decomposed. This could be a >>>>>> nightmare to track down. >>>>>> >>>>>> I hope that I can return to some older versions of numpy/scipy to work >>>>>> around this problem until this problem is fixed. Any suggestions on how >>>>>> to recover earlier versions would be appreciated. >>>>> That's not going to help you. The only thing that we guarantee (or >>>>> have *ever* guaranteed) is that the result is a valid QR >>>>> decomposition. If you need to swap signs to normalize things to your >>>>> desired convention, you will need to do that as a postprocessing step. >>>> But why do I need to normalize with numpy (at least with latest >>>> release); but not with MATLAB. >>> Because MATLAB decided to do the normalization step for you. That's a >>> valid decision. And so is ours. >>> >>>> A simple question for you. >>>> >>>> In my application MATLAB generates a sequence of QR factorizations for >>>> covariance matrices in which R is always PD --- which is corect! >>> That is not part of the definition of a QR decomposition. Failing to >>> meet that property does not make the QR decomposition incorrect. >>> >>> The only thing that is incorrect is passing an arbitrary, but valid, >>> QR decomposition to something that is expecting a strict *subset* of >>> valid QR decompositions. >> Sorry but I do not understand this... >> Let me give you an example that I believe illustrates the problem in numpy >> >> I have the following matrix, A: >> >> array([[ 7.07106781e+02, 5.49702852e-04, 1.66675481e-19], >> [ -3.53553391e+01, 7.07104659e+01, 1.66675481e-19], >> [ 0.00000000e+00, -3.97555166e+00, 7.07106781e-03], >> [ -7.07106781e+02, -6.48214647e-04, 1.66675481e-19], >> [ 3.53553391e+01, -7.07104226e+01, 1.66675481e-19], >> [ 0.00000000e+00, 3.97560687e+00, -7.07106781e-03], >> [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00], >> [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00], >> [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00]]) >> >> Note, this is clearly not a covariance matrix, but, it does contain a >> covariance matrix (3x3). I refer you to the paper for how this matrix >> was generated. >> >> Using np.linalg.qr(A) I get the following for R (3x3) which is >> "square-root" of the covariance matrix: >> >> array([[ -1.00124922e+03, 4.99289918e+00, 0.00000000e+00], >> [ 0.00000000e+00, -1.00033071e+02, 5.62045938e-04], >> [ 0.00000000e+00, 0.00000000e+00, -9.98419272e-03]]) >> >> which is clearly not PD, since the it's 3 eigenvalues (diagonal >> elements) are all negative. >> >> Now, if I use qr(A,0) in MATLAB: >> >> I get the following for R (3x3) >> >> 1001.24922, -4.99290, 0.00000 >> 0.00000, 100.03307, -0.00056 >> -0.00000, 0.00000, 0.00998 >> >> This is obviously PD, as it should be, and gives the correct results. >> Note, it is the negative of the R obtained with numpy. >> >> I can provide other examples in which both R's obtained are the same and >> they both lead to correct results. That is, when the R's are different, >> the R obtained with MATLAB is always PD and always gives the correct end >> result, while the R with numpy is not PD and does not give the correct >> end result. >> >> I hope that this helps you to understand my problem better. If there are >> more details that you need then let me know what, please. > To recap. No one is disputing that MATLAB does a normalization that > may make *some* use cases of a QR decomposition easier. However, > *both* are correct answers. Nothing in the definition of a QR > decomposition says R has to be positive definite. If you want R > positive definite, then, as mentioned, you'll need to check this > yourself and/or make a patch to numpy's QR function to do this to make > sure you get a PD R. I happen to get a PD R in your example using > LAPACK 3.3.1. I believe you can ensure R positive definite like so > > Q,R = np.linalg.qr(A) > trace_r = np.trace(R) > Q *= trace_r > 0 or -1 > R *= trace_r > 0 or -1 Yes, I agree with you. And your method works fine. In my particular case (with my current installation), it seems just checking R[0,0] > 0 can also be used. How can I force the use of LAPACK 3.3.1? From vs at it.uu.se Wed Nov 21 02:42:31 2012 From: vs at it.uu.se (Virgil Stokes) Date: Wed, 21 Nov 2012 08:42:31 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: <50AC2E77.2070906@creativetrax.com> References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC0B34.6030200@it.uu.se> <50AC119B.1040603@it.uu.se> <50AC2528.8000404@it.uu.se> <50AC2E77.2070906@creativetrax.com> Message-ID: <50AC85E7.8000107@it.uu.se> On 21-Nov-2012 02:29, Jason Grout wrote: > On 11/20/12 6:49 PM, Virgil Stokes wrote: > >> Thanks for looking at this problem. I believe that this is a serious >> issue that does deserve some attention. > I'm curious: are you *guaranteed* by matlab (e.g., it's documented) to > get a PD R matrix, or does it just happen in the cases you checked? If > you aren't guaranteed it, then it seems very shaky to depend on it even > from matlab. Of course, if it doesn't mention it in the docs, then who > knows if it is guaranteed---matlab is closed source, so you can't just > go check the code yourself. > > The way I read the docs at http://www.mathworks.com/help/matlab/ref/qr.html: > > "[Q,R] = qr(A), where A is m-by-n, produces an m-by-n upper triangular > matrix R and an m-by-m unitary matrix Q so that A = Q*R." > > it sounds like you shouldn't depend on R being PD even from matlab, even > if it's worked out that way in a few of your cases. Of course you are correct Jason, and I have not checked every QR decomposition in MATLAB and numpy. However, for my particular application I do know that the R returned from MATLAB always has diagonal elements > 0, while this is not the case for my installation of numpy. > > Thanks, > > Jason > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From vs at it.uu.se Wed Nov 21 02:46:31 2012 From: vs at it.uu.se (Virgil Stokes) Date: Wed, 21 Nov 2012 08:46:31 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: <1422781.E2OrQfupyW@metabuntu> References: <50AABA52.7020808@it.uu.se> <50AC21F8.9050106@it.uu.se> <1422781.E2OrQfupyW@metabuntu> Message-ID: <50AC86D7.8020903@it.uu.se> On 21-Nov-2012 02:30, Alex Leach wrote: > > On Wednesday 21 Nov 2012 01:36:08 Virgil Stokes wrote: > > > > > > I have the following matrix, A: > > > > > > array([[ 7.07106781e+02, 5.49702852e-04, 1.66675481e-19], > > > [ -3.53553391e+01, 7.07104659e+01, 1.66675481e-19], > > > [ 0.00000000e+00, -3.97555166e+00, 7.07106781e-03], > > > [ -7.07106781e+02, -6.48214647e-04, 1.66675481e-19], > > > [ 3.53553391e+01, -7.07104226e+01, 1.66675481e-19], > > > [ 0.00000000e+00, 3.97560687e+00, -7.07106781e-03], > > > [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00], > > > [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00], > > > [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00]]) > > > > > > > > > Using np.linalg.qr(A) I get the following for R (3x3) which is > > > "square-root" of the covariance matrix: > > > > > > array([[ -1.00124922e+03, 4.99289918e+00, 0.00000000e+00], > > > [ 0.00000000e+00, -1.00033071e+02, 5.62045938e-04], > > > [ 0.00000000e+00, 0.00000000e+00, -9.98419272e-03]]) > > > > > I thought this would be down to the Lapack library you have linked against, as > numpy uses Lapack's [zd]gqrf internally. I know Matlab comes distributed with > Intel's MKL, so I imagine Matlab would use Intel's MKL routines internally. > > Without knowing anything about the math, here are some test cases on Linux and > Mac OS X. Looks like it's dependent on platform, from these results... > > >>> a = np.array( ... ) > > >>> q, r = np.linalg.qr( a ) > > >>> r > > **Linux** > > Intel MKL: > > array([[ -1.00124922e+03, 4.99289919e+00, 2.40741243e-35], > > [ 0.00000000e+00, -1.00033071e+02, 5.62045938e-04], > > [ 0.00000000e+00, 0.00000000e+00, -9.98419272e-03]]) > > ATLAS Lapack: > > array([[ -1.00124922e+03, 4.99289919e+00, -2.40741243e-35], > > [ 0.00000000e+00, -1.00033071e+02, 5.62045938e-04], > > [ 0.00000000e+00, 0.00000000e+00, -9.98419272e-03]]) > > **Mac OS X**: > > Accelerate Framework:- > > array([[ 1.00124922e+03, -4.99289919e+00, 2.40741243e-35], > > [ 0.00000000e+00, 1.00033071e+02, -5.62045938e-04], > > [ 0.00000000e+00, 0.00000000e+00, 9.98419272e-03]]) > > ATLAS Lapack:- > > array([[ 1.00124922e+03, -4.99289919e+00, 2.40741243e-35], > > [ 0.00000000e+00, 1.00033071e+02, -5.62045938e-04], > > [ 0.00000000e+00, 0.00000000e+00, 9.98419272e-03]]) > > Cheers, > > Alex > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user Very interesting Alex --- Guess I should get a Mac :-) How can I force the use of the Lapack (on my Win32 system) that is being used by the Mac OS X? -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Wed Nov 21 02:49:24 2012 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 21 Nov 2012 08:49:24 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: <50AC86D7.8020903@it.uu.se> References: <50AABA52.7020808@it.uu.se> <50AC21F8.9050106@it.uu.se> <1422781.E2OrQfupyW@metabuntu> <50AC86D7.8020903@it.uu.se> Message-ID: Shor answer: you shouldn't! This time, it will work, next time it won't. You must never depend on an implementation detail as it is the case here. Add a post processing step, and it will be allright on all platforms and also in all languages. Cheers, 2012/11/21 Virgil Stokes > On 21-Nov-2012 02:30, Alex Leach wrote: > > On Wednesday 21 Nov 2012 01:36:08 Virgil Stokes wrote: > > > > > > I have the following matrix, A: > > > > > > array([[ 7.07106781e+02, 5.49702852e-04, 1.66675481e-19], > > > [ -3.53553391e+01, 7.07104659e+01, 1.66675481e-19], > > > [ 0.00000000e+00, -3.97555166e+00, 7.07106781e-03], > > > [ -7.07106781e+02, -6.48214647e-04, 1.66675481e-19], > > > [ 3.53553391e+01, -7.07104226e+01, 1.66675481e-19], > > > [ 0.00000000e+00, 3.97560687e+00, -7.07106781e-03], > > > [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00], > > > [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00], > > > [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00]]) > > > > > > > > > > > Using np.linalg.qr(A) I get the following for R (3x3) which is > > > "square-root" of the covariance matrix: > > > > > > array([[ -1.00124922e+03, 4.99289918e+00, 0.00000000e+00], > > > [ 0.00000000e+00, -1.00033071e+02, 5.62045938e-04], > > > [ 0.00000000e+00, 0.00000000e+00, -9.98419272e-03]]) > > > > > > > I thought this would be down to the Lapack library you have linked > against, as numpy uses Lapack's [zd]gqrf internally. I know Matlab comes > distributed with Intel's MKL, so I imagine Matlab would use Intel's MKL > routines internally. > > Without knowing anything about the math, here are some test cases on Linux > and Mac OS X. Looks like it's dependent on platform, from these results... > > > > >>> a = np.array( ... ) > > >>> q, r = np.linalg.qr( a ) > > >>> r > > > > **Linux** > > Intel MKL: > > array([[ -1.00124922e+03, 4.99289919e+00, 2.40741243e-35], > > [ 0.00000000e+00, -1.00033071e+02, 5.62045938e-04], > > [ 0.00000000e+00, 0.00000000e+00, -9.98419272e-03]]) > > > > ATLAS Lapack: > > array([[ -1.00124922e+03, 4.99289919e+00, -2.40741243e-35], > > [ 0.00000000e+00, -1.00033071e+02, 5.62045938e-04], > > [ 0.00000000e+00, 0.00000000e+00, -9.98419272e-03]]) > > > > > > **Mac OS X**: > > Accelerate Framework:- > > array([[ 1.00124922e+03, -4.99289919e+00, 2.40741243e-35], > > [ 0.00000000e+00, 1.00033071e+02, -5.62045938e-04], > > [ 0.00000000e+00, 0.00000000e+00, 9.98419272e-03]]) > > > > ATLAS Lapack:- > > array([[ 1.00124922e+03, -4.99289919e+00, 2.40741243e-35], > > [ 0.00000000e+00, 1.00033071e+02, -5.62045938e-04], > > [ 0.00000000e+00, 0.00000000e+00, 9.98419272e-03]]) > > > > Cheers, > > Alex > > > _______________________________________________ > SciPy-User mailing listSciPy-User at scipy.orghttp://mail.scipy.org/mailman/listinfo/scipy-user > > Very interesting Alex --- Guess I should get a Mac :-) > > How can I force the use of the Lapack (on my Win32 system) that is being > used by the Mac OS X? > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher Music band: http://liliejay.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From vs at it.uu.se Wed Nov 21 03:34:29 2012 From: vs at it.uu.se (Virgil Stokes) Date: Wed, 21 Nov 2012 09:34:29 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC08FE.7050703@it.uu.se> <50AC105C.5090401@it.uu.se> <50AC21F8.9050106@it.uu.se> <50AC26AD.2080904@it.uu.se> Message-ID: <50AC9215.1010609@it.uu.se> On 21-Nov-2012 03:47, Alejandro Weinstein wrote: > On Tue, Nov 20, 2012 at 5:56 PM, Virgil Stokes wrote: >> On 2012-11-21 01:48, Alejandro Weinstein wrote: >>> On Tue, Nov 20, 2012 at 5:36 PM, Virgil Stokes wrote: >>>> which is clearly not PD, since the it's 3 eigenvalues (diagonal >>>> elements) are all negative. >>> But why you expect R to be PD? >> Because R*R^T = P (a covariance matrix). One important reason for >> using the QR factorization in the KF is to ensure that R is always PD >> during the recursions. > As you said, P = R * R^T, which is PD, even if R is not. Please check > the definition of QR decomposition: R is _not_ required to be PD. Yes, again Alejandro I know this --- my statement was really not correct (sloppy on my part). > > Looking at the paper, they in fact use P = R * R ^ T, as in eq. (1). > They never use R alone. So the fact that R is not PD should not be an > issue. But, again, it is an issue for the algorithm given in Table 3 (p. 2248 of paper). Look at step 8. and equation (30). As stated in this step "The square-root of the filtered state-error covariance" is returned as $T_{22}$ from step 5. where a QR decomposition is performed for the triangularization. The $S_{k+1|k+1}$ matrix must have diagonal elements > 0 (I leave this to you to think about). When I perform step 5. with MATLAB, I *always* get a $T_{22}$ that has diagonal elements > 0 for my application (which is satisfying). When I perform step 5. with numpy, for the same matrix on the RHS of (27) I *do not always* get diagonal elements > 0 for $T_{22}$ --- herein lies the problem that I have been trying to explain. > Can you show your code? I'm curious to see how the fact that R > is not PD makes a difference. You should be able to write a few lines of code to test this yourself. If this problem does not occur on your system then please tell me what version of Scipy/numpy you have installed and on what platform. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: tblatex-1.png Type: image/png Size: 721 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: tblatex-2.png Type: image/png Size: 907 bytes Desc: not available URL: From davidmenhur at gmail.com Wed Nov 21 05:31:17 2012 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Wed, 21 Nov 2012 11:31:17 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: <50AC9215.1010609@it.uu.se> References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC08FE.7050703@it.uu.se> <50AC105C.5090401@it.uu.se> <50AC21F8.9050106@it.uu.se> <50AC26AD.2080904@it.uu.se> <50AC9215.1010609@it.uu.se> Message-ID: On Wed, Nov 21, 2012 at 9:34 AM, Virgil Stokes wrote: > When I perform step 5. with MATLAB, I *always* get a [image: $T_{22}$]that has diagonal elements > 0 for my application (which is satisfying). > When I perform step 5. with numpy, for the same matrix on the RHS of (27) I > *do not always* get diagonal elements > 0 for [image: $T_{22}$] --- > herein lies the problem that I have been trying to explain. > http://www.mathworks.es/es/help/matlab/ref/qr.html There is no reference to the positiveness of r, nor it is required by the definition of QR decomposition, therefore it is wrong to assume it is true. It may be in your current examples, but you don't have guaranteed that you will find another A matrix where this will not be true. Also, the portability of your code is tricky, as another person, with a different version of MATLAB could have different results, even if it is not documented (because the change would not affect the API). The same holds true for Mathematica. http://reference.wolfram.com/mathematica/ref/QRDecomposition.html And even, in one of the examples, the matrix r is negative. If you want to make sure, just use a qr function that checks for the sign and reverts it, if necesary. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: tblatex-1.png Type: image/png Size: 721 bytes Desc: not available URL: From sturla at molden.no Wed Nov 21 06:20:15 2012 From: sturla at molden.no (Sturla Molden) Date: Wed, 21 Nov 2012 12:20:15 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: <50AC9215.1010609@it.uu.se> References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC08FE.7050703@it.uu.se> <50AC105C.5090401@it.uu.se> <50AC21F8.9050106@it.uu.se> <50AC26AD.2080904@it.uu.se> <50AC9215.1010609@it.uu.se> Message-ID: <12B0A7E3-7880-459D-B127-273C6EB74D3D@molden.no> Den 21. nov. 2012 kl. 09:34 skrev Virgil Stokes : > But, again, it is an issue for the algorithm given in Table 3 (p. 2248 of paper). Look at step 8. and equation (30). As stated in this step "The square-root of the filtered state-error covariance" is returned as from step 5. where a QR decomposition is performed for the triangularization. The matrix must have diagonal elements > 0 (I leave this to you to think about). When I perform step 5. with MATLAB, I always get a that has diagonal elements > 0 for my application (which is satisfying). When I perform step 5. with numpy, for the same matrix on the RHS of (27) I do not always get diagonal elements > 0 for --- herein lies the problem that I have been trying to explain. There is nothing in the definition of QR that says R must be Positive definite. NumPy will give you a valid QR fatorization computed with LAPACK, and so will Matlab. You might blame it on the LAPACK version, or Matlab might do an undocumented normalization as post-processing. Matlab is not a standard for the "right ansver" in linear algebra. Any numerical code requiring R to be PD is errorneous, even in Matlab. Then again, it is not rocket science to post-process Q and R so that R is PD. If you need to decompose a covariance matrix P = R*R' with R PD it sounds like you use QR as a Cholesky factorization. If you want a Cholesky factor of P, you know where to find it. (Not that I recommend using it.) Sturla -------------- next part -------------- An HTML attachment was scrubbed... URL: From alejandro.weinstein at gmail.com Wed Nov 21 08:22:22 2012 From: alejandro.weinstein at gmail.com (Alejandro Weinstein) Date: Wed, 21 Nov 2012 06:22:22 -0700 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: <50AC9215.1010609@it.uu.se> References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC08FE.7050703@it.uu.se> <50AC105C.5090401@it.uu.se> <50AC21F8.9050106@it.uu.se> <50AC26AD.2080904@it.uu.se> <50AC9215.1010609@it.uu.se> Message-ID: On Wed, Nov 21, 2012 at 1:34 AM, Virgil Stokes wrote: > But, again, it is an issue for the algorithm given in Table 3 (p. 2248 > of paper). Look at step 8. and equation (30). > I see no Table 3, nor page 2248 in the paper you attached. Which paper are you talking about? > As stated in this step "The square-root of the filtered state-error > covariance" is returned as [image: $T_{22}$] from step 5. where a QR > decomposition is performed for the triangularization. The [image: > $S_{k+1|k+1}$] matrix must have diagonal elements > 0 (I leave this to > you to think about). When I perform step 5. with MATLAB, I *always* get a [image: > $T_{22}$] that has diagonal elements > 0 for my application (which is > satisfying). When I perform step 5. with numpy, for the same matrix on the > RHS of (27) I *do not always* get diagonal elements > 0 for [image: > $T_{22}$] --- herein lies the problem that I have been trying to explain. > > Anyway, I would say that an algorithm that depend on the sign of the entries of R is not correct. Alejandro. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: tblatex-2.png Type: image/png Size: 907 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: tblatex-1.png Type: image/png Size: 721 bytes Desc: not available URL: From sturla at molden.no Wed Nov 21 09:15:15 2012 From: sturla at molden.no (Sturla Molden) Date: Wed, 21 Nov 2012 15:15:15 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC08FE.7050703@it.uu.se> <50AC105C.5090401@it.uu.se> <50AC21F8.9050106@it.uu.se> <50AC26AD.2080904@it.uu.se> <50AC9215.1010609@it.uu.se> Message-ID: <50ACE1F3.2060606@molden.no> On 21.11.2012 14:22, Alejandro Weinstein wrote: > On Wed, Nov 21, 2012 at 1:34 AM, Virgil Stokes > wrote: > > But, again, it is an issue for the algorithm given in Table 3 (p. > 2248 of paper). Look at step 8. and equation (30). > > > I see no Table 3, nor page 2248 in the paper you attached. Which paper > are you talking about? I think what he means is that in equation 22 (page 2591) S must be PD, and thus R = S' must be PD as well. Which (AFAIK) must be an error in the paper. Too bad referees didn't catch it. They should have used Cholesky of P as S. So instead of the second equation in section IV which states that S = R', equation (6) should Cholesky factorize P to obtain S. And thus S would be PD for equation 22. In conclusion, there is an error in the paper which was silenced by Matlab's QR. Sturla From vs at it.uu.se Wed Nov 21 10:01:17 2012 From: vs at it.uu.se (Virgil Stokes) Date: Wed, 21 Nov 2012 16:01:17 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC08FE.7050703@it.uu.se> <50AC105C.5090401@it.uu.se> <50AC21F8.9050106@it.uu.se> <50AC26AD.2080904@it.uu.se> <50AC9215.1010609@it.uu.se> Message-ID: <50ACECBD.6000307@it.uu.se> On 21-Nov-2012 14:22, Alejandro Weinstein wrote: > On Wed, Nov 21, 2012 at 1:34 AM, Virgil Stokes > wrote: > > But, again, it is an issue for the algorithm given in Table 3 (p. 2248 of > paper). Look at step 8. and equation (30). > > > I see no Table 3, nor page 2248 in the paper you attached. Which paper are > you talking about? > > As stated in this step "The square-root of the filtered state-error > covariance" is returned as $T_{22}$ from step 5. where a QR decomposition > is performed for the triangularization. The $S_{k+1|k+1}$ matrix must have > diagonal elements > 0 (I leave this to you to think about). When I > perform step 5. with MATLAB, I *always* get a $T_{22}$ that has diagonal > elements > 0 for my application (which is satisfying). When I perform step > 5. with numpy, for the same matrix on the RHS of (27) I *do not always* > get diagonal elements > 0 for $T_{22}$ --- herein lies the problem that I > have been trying to explain. > > > Anyway, I would say that an algorithm that depend on the sign of the entries > of R is not correct. > > Alejandro. > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user I am referring to the following paper: Arasaratnam, I. and Haykin, S. (2011) Cubature Kalman Smoothers. /Automatica/ *47*, pp. 2245-2250. Sorry about the confusion -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 721 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 907 bytes Desc: not available URL: From vs at it.uu.se Wed Nov 21 10:01:33 2012 From: vs at it.uu.se (Virgil Stokes) Date: Wed, 21 Nov 2012 16:01:33 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: <50ACE1F3.2060606@molden.no> References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC08FE.7050703@it.uu.se> <50AC105C.5090401@it.uu.se> <50AC21F8.9050106@it.uu.se> <50AC26AD.2080904@it.uu.se> <50AC9215.1010609@it.uu.se> <50ACE1F3.2060606@molden.no> Message-ID: <50ACECCD.50204@it.uu.se> On 21-Nov-2012 15:15, Sturla Molden wrote: > On 21.11.2012 14:22, Alejandro Weinstein wrote: >> On Wed, Nov 21, 2012 at 1:34 AM, Virgil Stokes > > wrote: >> >> But, again, it is an issue for the algorithm given in Table 3 (p. >> 2248 of paper). Look at step 8. and equation (30). >> >> >> I see no Table 3, nor page 2248 in the paper you attached. Which paper >> are you talking about? > I think what he means is that in equation 22 (page 2591) S must be PD, > and thus R = S' must be PD as well. Which (AFAIK) must be an error in > the paper. Too bad referees didn't catch it. > > They should have used Cholesky of P as S. So instead of the second > equation in section IV which states that S = R', equation (6) should > Cholesky factorize P to obtain S. And thus S would be PD for equation > 22. In conclusion, there is an error in the paper which was silenced by > Matlab's QR. > > Sturla > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user I am referring to the following paper: Arasaratnam, I. and Haykin, S. (2011) Cubature Kalman Smoothers. /Automatica/ *47*, pp. 2245-2250. Sorry about the confusion :-[ -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Wed Nov 21 10:14:09 2012 From: sturla at molden.no (Sturla Molden) Date: Wed, 21 Nov 2012 16:14:09 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: <50ACECCD.50204@it.uu.se> References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC08FE.7050703@it.uu.se> <50AC105C.5090401@it.uu.se> <50AC21F8.9050106@it.uu.se> <50AC26AD.2080904@it.uu.se> <50AC9215.1010609@it.uu.se> <50ACE1F3.2060606@molden.no> <50ACECCD.50204@it.uu.se> Message-ID: <50ACEFC1.6090100@molden.no> On 21.11.2012 16:01, Virgil Stokes wrote: > I am referring to the following paper: > Arasaratnam, I. and Haykin, S. (2011) Cubature Kalman Smoothers. > /Automatica/ *47*, pp. 2245-2250. Not the one you attached, and sorry I don't have it. Sturla From sturla at molden.no Wed Nov 21 10:25:26 2012 From: sturla at molden.no (Sturla Molden) Date: Wed, 21 Nov 2012 16:25:26 +0100 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: <50ACEFC1.6090100@molden.no> References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC08FE.7050703@it.uu.se> <50AC105C.5090401@it.uu.se> <50AC21F8.9050106@it.uu.se> <50AC26AD.2080904@it.uu.se> <50AC9215.1010609@it.uu.se> <50ACE1F3.2060606@molden.no> <50ACECCD.50204@it.uu.se> <50ACEFC1.6090100@molden.no> Message-ID: <50ACF266.1030207@molden.no> On 21.11.2012 16:14, Sturla Molden wrote: > On 21.11.2012 16:01, Virgil Stokes wrote: > >> I am referring to the following paper: >> Arasaratnam, I. and Haykin, S. (2011) Cubature Kalman Smoothers. >> /Automatica/ *47*, pp. 2245-2250. > > Not the one you attached, and sorry I don't have it. Ok, found it. It basically has the same error as I pointed out. It needs a PD lower triangular matrix, which is not returned by QR but Cholesky. Sturla From andy.belisson.ext at safran-engineering.com Wed Nov 21 09:17:25 2012 From: andy.belisson.ext at safran-engineering.com (BELISSON Andy [EXT]) Date: Wed, 21 Nov 2012 15:17:25 +0100 Subject: [SciPy-User] [SCIPY]Belisson Andy: please help build/install/testing Message-ID: Hi, I am a french developer, and I have some problem during Scipy Testing. ... ============= ERROR: Failure: ImportError (/home/cmn/HelicoStation/Linux_x86_64/HS_SDK_GTK-2.24/lib/python2.7/site -packages/scipy/special/_cephes.so: undefined symbol: s_stop) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/cmn/HelicoStation/Linux_x86_64/HS_SDK_GTK-2.24/lib/python2.7/site -packages/nose/loader.py", line 390, in loadTestsFromName addr.filename, addr.module) File "/home/cmn/HelicoStation/Linux_x86_64/HS_SDK_GTK-2.24/lib/python2.7/site -packages/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/home/cmn/HelicoStation/Linux_x86_64/HS_SDK_GTK-2.24/lib/python2.7/site -packages/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/home/cmn/HelicoStation/Linux_x86_64/HS_SDK_GTK-2.24/lib/python2.7/site -packages/scipy/stats/__init__.py", line 321, in from stats import * File "/home/cmn/HelicoStation/Linux_x86_64/HS_SDK_GTK-2.24/lib/python2.7/site -packages/scipy/stats/stats.py", line 193, in import scipy.special as special File "/home/cmn/HelicoStation/Linux_x86_64/HS_SDK_GTK-2.24/lib/python2.7/site -packages/scipy/special/__init__.py", line 525, in from _cephes import * ImportError: /home/cmn/HelicoStation/Linux_x86_64/HS_SDK_GTK-2.24/lib/python2.7/site- packages/scipy/special/_cephes.so: undefined symbol: s_stop ---------------------------------------------------------------------- Ran 2016 tests in 10.695s FAILED (KNOWNFAIL=3, SKIP=22, errors=14) I search many ways, but nothing work. Please I have encountered this problem, one week ago, I test many things, like try to link -lg2c. But I don't know how solve this problem. There is a full .log attached to this mail. Please help me ... Best Regards Belisson Andy Mail : andy.belisson at club-internet.fr or andy.belisson.ext at safran-engineering.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Troubleshooting_scipy.txt URL: From Christos.Siopis at ulb.ac.be Wed Nov 21 10:27:40 2012 From: Christos.Siopis at ulb.ac.be (Christos Siopis) Date: Wed, 21 Nov 2012 16:27:40 +0100 Subject: [SciPy-User] FOSS for scientists devroom at FOSDEM 2013 Message-ID: <20121121152740.GA6241@ulb.ac.be> Dear members of the Numerical Python ecosystem (with apologies for cross-postings), A day-long session ("devroom") on Free/Libre and Open Source Software (FLOSS) for scientists will be held during the next FOSDEM conference, Brussels, 2-3 February 2013 (http://fosdem.org/2013). We aim at having a dozen or two short talks introducing projects, advertising brand new features of established tools, discussing issues relevant to the development of software for scientific computing, and touching on the interdependence of FLOSS and open science. You can find more info on the call for talks at: http://slayoo.github.com/fosdem2013/ The deadline for sending talk proposals is December 16th 2012. Please send your submissions or comments to: foss4scientists-devroom at lists.fosdem.org Please do forward this message to anyone potentially interested. Please also let us know if you have any suggestions for what would you like to hear about in the devroom. Looking forward to meeting you in Brussels. Thanks in advance. The conveners, Sylwester Arabas, Juan Antonio A?el, Christos Siopis P.S. There are open calls for main-track talks, lightning talks, and stands at FOSDEM as well, see: http://fosdem.org/2013/ -------------- I would like to add to the above general announcement, that it would be great if a main track talk were to be given at FOSDEM about the importance of scientific open source software in science and engineering today. Main track talks last 50 minutes, and are addressed to all FOSDEM participants, something that would add to the visibility of scientific software. As an extra bonus, main track speakers have their travel and hotel expenses covered by FOSDEM. I think that the numerical python "ecosystem" could serve as an excellent "case study" of the data processing and visualisation workflow, while adding an interesting historical dimension, being one of the oldest projects of its sort. If you decide to respond to the call for main track speakers, you should start here: https://fosdem.org/2013/call_for_main_speakers/ Please note the December 1 deadline. I urge you to let us (the science software devroom conveners) know about your proposed talk, so that we may send a word of recommendation to the FOSDEM committee who will make the ultimate selection. We thank you in advance for your expressions of interest and participation! From cimrman3 at ntc.zcu.cz Wed Nov 21 13:08:51 2012 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Wed, 21 Nov 2012 19:08:51 +0100 Subject: [SciPy-User] ANN: SfePy 2012.4 Message-ID: <50AD18B3.1010705@ntc.zcu.cz> I am pleased to announce release 2012.4 of SfePy. Description ----------- SfePy (simple finite elements in Python) is a software for solving systems of coupled partial differential equations by the finite element method. The code is based on NumPy and SciPy packages. It is distributed under the new BSD license. Home page: http://sfepy.org Downloads, mailing list, wiki: http://code.google.com/p/sfepy/ Git (source) repository, issue tracker: http://github.com/sfepy Highlights of this release -------------------------- - initial support for hierarchical basis on quadrilateral and brick elements - unified C/Cython structures for reference mappings - new linear combination boundary condition: edge direction - new examples showing some advanced features For full release notes see http://docs.sfepy.org/doc/release_notes.html#id1 (rather long and technical). Best regards, Robert Cimrman and Contributors (*) (*) Contributors to this release (alphabetical order): Bjarke Dalslet, Vladim?r Luke?, Maty?? Nov?k From josef.pktd at gmail.com Thu Nov 22 00:10:05 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 22 Nov 2012 00:10:05 -0500 Subject: [SciPy-User] PeriodicUnivariateSpline Message-ID: I just found a well kept secret: "Periodic Splines in scipy", that I have to advertise. http://stackoverflow.com/questions/13490533/interpolating-data-in-python-with-fixed-points my quick plot based on their example: https://picasaweb.google.com/lh/photo/0YnsV6Ad6MlixKNfYW-g0dMTjNZETYmyPJy0liipFm0?feat=directlink Next time I need seasonal splines, I go for scipy. periodic at 3 decimals: >>> points[0], points[-1] (14.875105965386629, 14.874590980930265) Josef Look, a new toy for my tool box. From pav at iki.fi Thu Nov 22 08:51:11 2012 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 22 Nov 2012 13:51:11 +0000 (UTC) Subject: [SciPy-User] qr decompostion gives negative q, r ? References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC08FE.7050703@it.uu.se> <50AC105C.5090401@it.uu.se> <50AC21F8.9050106@it.uu.se> Message-ID: Virgil Stokes it.uu.se> writes: [clip] > Now, if I use qr(A,0) in MATLAB: > > I get the following for R (3x3) > > 1001.24922, -4.99290, 0.00000 > 0.00000, 100.03307, -0.00056 > -0.00000, 0.00000, 0.00998 > > This is obviously PD, as it should be, and gives the correct results. > Note, it is the negative of the R obtained with numpy. Case in point: MATLAB R2011b, Linux x86_64 >> A=[707.106781, 0.000549702852, 1.66675481e-19;-35.3553391, 70.7104659, 1.66675481e-19;0.0, -3.97555166, 0.00707106781;-707.106781, -0.000648214647, 1.66675481e-19;35.3553391, -70.7104226, 1.66675481e-19;0.0, 3.97560687, -0.00707106781;0.0, 0.0, 0.0;0.0, 0.0, 0.0;0.0, 0.0, 0.0] >> [Q,R] = qr(A, 0); >> R R = 1.0e+03 * -1.001249219464423 0.004992899186307 0.000000000000000 0 -0.100033070889029 0.000000562045938 0 0 -0.000009984192722 The statements "MATLAB gives always positive definite R" "MATLAB does post-processing in QR" are both false. Indeed, any "normalization" of the result is not mentioned in the documentation. You simply cannot assume that the returned R is PD, because what you actually get ultimately depends on the underlying LAPACK library. -- Pauli Virtanen From pav at iki.fi Thu Nov 22 10:10:13 2012 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 22 Nov 2012 15:10:13 +0000 (UTC) Subject: [SciPy-User] qr decompostion gives negative q, r ? References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC0B34.6030200@it.uu.se> <50AC119B.1040603@it.uu.se> <50AC2528.8000404@it.uu.se> <50AC2E77.2070906@creativetrax.com> <50AC85E7.8000107@it.uu.se> Message-ID: Virgil Stokes it.uu.se> writes: [clip] > Of course you are correct Jason, and I have not checked every QR decomposition > in MATLAB and numpy. However, for my particular application I do know that the > R returned from MATLAB always has diagonal elements > 0, while this is not the > case for my installation of numpy. It is incorrect to assume this. Also Matlab does not guarantee that R is PD. That you get it is just due to lucky choice of matrices and the MATLAB version/platform. Compare (these are on the same machine!): MATLAB(R) Version 7.10.0.499 (R2010a) 64-bit (glnxa64) February 5, 2010 >> A=[707.106781, 0.000549702852, 1.66675481e-19;-35.3553391, 70.7104659, 1.66675481e-19;0.0, -3.97555166, 0.00707106781;-707.106781, -0.000648214647, 1.66675481e-19;35.3553391, -70.7104226, 1.66675481e-19;0.0, 3.97560687, -0.00707106781;0.0, 0.0, 0.0;0.0, 0.0, 0.0;0.0, 0.0, 0.0]; [Q,R]=qr(A,0); R R = 1.0e+03 * 1.0012 -0.0050 0.0000 0 0.1000 -0.0000 0 0 0.0000 MATLAB(R) R2012a (7.14.0.739) 64-bit (glnxa64) February 9, 2012 >> A=[707.106781, 0.000549702852, 1.66675481e-19;-35.3553391, 70.7104659, 1.66675481e-19;0.0, -3.97555166, 0.00707106781;-707.106781, -0.000648214647, 1.66675481e-19;35.3553391, -70.7104226, 1.66675481e-19;0.0, 3.97560687, -0.00707106781;0.0, 0.0, 0.0;0.0, 0.0, 0.0;0.0, 0.0, 0.0]; [Q,R]=qr(A,0); R R = 1.0e+03 * -1.0012 0.0050 0.0000 0 -0.1000 0.0000 0 0 -0.0000 These, and the results returned by Numpy/Scipy are all correct QR decompositions. There are several signs in QR that can be freely chosen, and in general there is no particular reason to stick with a given choice. Cheers, Pauli Virtanen From charlesr.harris at gmail.com Thu Nov 22 15:12:14 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 22 Nov 2012 13:12:14 -0700 Subject: [SciPy-User] qr decompostion gives negative q, r ? In-Reply-To: References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC0B34.6030200@it.uu.se> <50AC119B.1040603@it.uu.se> <50AC2528.8000404@it.uu.se> <50AC2E77.2070906@creativetrax.com> <50AC85E7.8000107@it.uu.se> Message-ID: On Thu, Nov 22, 2012 at 8:10 AM, Pauli Virtanen wrote: > Virgil Stokes it.uu.se> writes: > [clip] > > Of course you are correct Jason, and I have not checked every QR > decomposition > > in MATLAB and numpy. However, for my particular application I do know > that the > > R returned from MATLAB always has diagonal elements > 0, while this is > not the > > case for my installation of numpy. > > It is incorrect to assume this. Also Matlab does not guarantee > that R is PD. That you get it is just due to lucky choice > of matrices and the MATLAB version/platform. > > Compare (these are on the same machine!): > > MATLAB(R) Version 7.10.0.499 (R2010a) 64-bit (glnxa64) February 5, 2010 > > >> A=[707.106781, 0.000549702852, 1.66675481e-19;-35.3553391, 70.7104659, > 1.66675481e-19;0.0, -3.97555166, 0.00707106781;-707.106781, > -0.000648214647, > 1.66675481e-19;35.3553391, -70.7104226, 1.66675481e-19;0.0, 3.97560687, > -0.00707106781;0.0, 0.0, 0.0;0.0, 0.0, 0.0;0.0, 0.0, 0.0]; [Q,R]=qr(A,0); R > > R = > > 1.0e+03 * > > 1.0012 -0.0050 0.0000 > 0 0.1000 -0.0000 > 0 0 0.0000 > > > MATLAB(R) R2012a (7.14.0.739) 64-bit (glnxa64) February 9, 2012 > > >> A=[707.106781, 0.000549702852, 1.66675481e-19;-35.3553391, 70.7104659, > 1.66675481e-19;0.0, -3.97555166, 0.00707106781;-707.106781, > -0.000648214647, > 1.66675481e-19;35.3553391, -70.7104226, 1.66675481e-19;0.0, 3.97560687, > -0.00707106781;0.0, 0.0, 0.0;0.0, 0.0, 0.0;0.0, 0.0, 0.0]; [Q,R]=qr(A,0); R > > R = > > 1.0e+03 * > > -1.0012 0.0050 0.0000 > 0 -0.1000 0.0000 > 0 0 -0.0000 > > > These, and the results returned by Numpy/Scipy are all correct QR > decompositions. There are several signs in QR that can be freely > chosen, and in general there is no particular reason to stick with > a given choice. > > The term 'positive definite' doesn't really apply either. Just making the diagonals of r come positive in the qr factorization of a covariance doesn't make the quadratic form associated with r PD: In [17]: m[0] Out[17]: array([[ 1. , -0.9], [-0.9, 1. ]]) In [18]: eigvalsh(m[0]) Out[18]: array([ 0.1, 1.9]) In [19]: rp[0] Out[19]: array([[ 1.3453624 , -1.33792946], [ 0. , 0.14122589]]) In [20]: eigvalsh(.5*(rp[0].T + rp[0])) Out[20]: array([-0.15670585, 1.64329415]) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From edwardsamery at ymail.com Wed Nov 21 23:09:38 2012 From: edwardsamery at ymail.com (edwardsamery) Date: Wed, 21 Nov 2012 20:09:38 -0800 (PST) Subject: [SciPy-User] [SciPy-user] Problem with Installation of Scipy on Macbook In-Reply-To: References: Message-ID: <34709315.post@talk.nabble.com> I think just by following all the installment procedure you could easily get installed it. You have to just follow all the steps correctly. You should also take care of it that you shouldn't skip any step. -------------------------------------------------------------- Get the best http://therepairstop.com/ HTC glass replacement | http://therepairstop.com/ iMac Glass replacement -- View this message in context: http://old.nabble.com/Problem-with-Installation-of-Scipy-on-Macbook-tp33537462p34709315.html Sent from the Scipy-User mailing list archive at Nabble.com. From flower_des_iles at hotmail.com Thu Nov 22 10:23:02 2012 From: flower_des_iles at hotmail.com (=?iso-8859-1?B?U3TpcGhhbmllIGhhYWFhYWFhYQ==?=) Date: Thu, 22 Nov 2012 15:23:02 +0000 Subject: [SciPy-User] cubic spline interpolation using scipy Message-ID: Hello everybody, I have dataset which look like this : position number_of_tag_at_this_position 3 4 8 6 13 25 23 12 I want to apply cubic spline interpolation to this dataset to interpolate tag density; to do so, i run : import numpy as np from scipy import interpolate` x = [3,8,13,23]` y = [4,6,25,12]` tck = interpolate.splrep(x,y) # cubic` And now, i would like to calculate the derivative of the function at each point of the interpolation, How can i do this ? Thanks for your help ! -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcp.stras at gmail.com Fri Nov 23 01:14:28 2012 From: mcp.stras at gmail.com (Martin Campos Pinto) Date: Fri, 23 Nov 2012 07:14:28 +0100 Subject: [SciPy-User] Gmres does not work with Sparse Complex Matrices Message-ID: <50AF1444.9030502@gmail.com> Hi all, I am having some troubles using iterative solvers with sparse complex matrices. Here is an example with gmres: the script from scipy import sparse from numpy.linalg import norm from numpy.random import rand import scipy.sparse.linalg as spla C = sparse.lil_matrix((10, 10), dtype=complex) C.setdiag(rand(10)) C[0,3] = 1j C = C.tocsr() c = rand(10)+rand(10)*1j x = spla.spsolve(C, c) print "spsolve: norm(C*x - c) = ", norm(C*x - c) (x,info) = spla.gmres(C, c) print "gmres: norm(C*x - c) = ", norm(C*x - c) gives as output: spsolve: norm(C*x - c) = 1.57621370006e-16 Segmentation fault Actually, sometimes I get this error message instead of a Segmentation fault: Traceback (most recent call last): File "test_gmres_cplx.py", line 45, in (x,info) = spla.gmres(C, c) File "/Library/Frameworks/EPD64.framework/Versions/7.2/lib/python2.7/site-packages/scipy/sparse/linalg/isolve/iterative.py", line 437, in gmres revcom(b, x, restrt, work, work2, iter_, resid, info, ndx1, ndx2, ijob) SystemError: NULL result without error in PyObject_Call Note that I have tested gmres with real sparse matrices, and it runs fine: Indeed C = sparse.lil_matrix((10, 10)) C.setdiag(rand(10)) C = C.tocsr() c = rand(10) x = spla.spsolve(C, c) print "spsolve: norm(C*x - c) = ", norm(C*x - c) (x,info) = spla.gmres(C, c) print "gmres: norm(C*x - c) = ", norm(C*x - c) gives spsolve: norm(C*x - c) = 6.93889390391e-18 gmres: norm(C*x - c) = 5.28860261481e-16 I have looked on the web for solutions but haven't found any. Some very old posts indicate similar errors but they don't come with an answer, and I imagine that if those were due to bugs, they should have been fixed by now. Am I doing something stupid here, or is that a real problem ? Is somebody aware of a solution ? (I am using scipy version 0.10.1) Thanks, Martin -- Martin Campos Pinto LJLL, UPMC & CNRS From spamaxx at gmail.com Fri Nov 23 04:19:41 2012 From: spamaxx at gmail.com (Maxim) Date: Fri, 23 Nov 2012 09:19:41 +0000 (UTC) Subject: [SciPy-User] cubic spline interpolation - derivative value in an end point Message-ID: I have a 1D function which is defined on [-1; 1] and it's symmetrical (with respect to 0). I want to construct a cubic spline interpolation of the function on [0; 1] (using, for example, interpolate.splrep and interpolate.splev functions) and as I know that the function is smooth, I must specify a derivative of the function in 0 to be 0). How can I manually define a derivative in one of the end points? Thanks, Maxim From pawel.kw at gmail.com Fri Nov 23 04:27:11 2012 From: pawel.kw at gmail.com (=?ISO-8859-2?Q?Pawe=B3_Kwa=B6niewski?=) Date: Fri, 23 Nov 2012 10:27:11 +0100 Subject: [SciPy-User] cubic spline interpolation using scipy In-Reply-To: References: Message-ID: Hi, Once you have your spline, you can use interpolate.splev() function to evaluate it on a given set of x values. If you look into the documentation of this function you will see that it can also give you the derivatives - in your case, if you want the first derivative at each point of the interpolation, you should do the following: dy = interpolate.splev(x,tck,der=1) That's it. Cheers, Pawe? 2012/11/22 St?phanie haaaaaaaa > [4,6,25,12] -------------- next part -------------- An HTML attachment was scrubbed... URL: From pawel.kw at gmail.com Fri Nov 23 05:52:45 2012 From: pawel.kw at gmail.com (=?ISO-8859-2?Q?Pawe=B3_Kwa=B6niewski?=) Date: Fri, 23 Nov 2012 11:52:45 +0100 Subject: [SciPy-User] cubic spline interpolation - derivative value in an end point In-Reply-To: References: Message-ID: Hi, I don't really understand why you want to specify the derivative at 0. If, as you say, your function has a 0 derivative at 0, if you interpolate the function on the complete range where it is defined: [-1:1] with interpolate.splrep(), you can later evaluate it with splev() on whatever range you wish, like [0:1], and it should have the proper derivative. Cheers, Pawe? 2012/11/23 Maxim > I have a 1D function which is defined on [-1; 1] and it's symmetrical (with > respect to 0). I want to construct a cubic spline interpolation of the > function > on [0; 1] (using, for example, interpolate.splrep and interpolate.splev > functions) and as I know that the function is smooth, I must specify a > derivative of the function in 0 to be 0). > > How can I manually define a derivative in one of the end points? > > Thanks, > Maxim > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From spamaxx at gmail.com Fri Nov 23 07:17:47 2012 From: spamaxx at gmail.com (Maxim) Date: Fri, 23 Nov 2012 12:17:47 +0000 (UTC) Subject: [SciPy-User] cubic spline interpolation - derivative value in an end point References: Message-ID: Hi Pawel, in fact I analyse a set of data from an experiment (the data represents temperature values). I have a cylindrical system of coordinates and experimental measurements were done along the radius (that means on [0;1]). I know that temperature has a maximum in the centre and that there is no sharp peak there (the derivative there must be zero). I just want to implement this feature in my model. Cheers, Maxim From spamaxx at gmail.com Fri Nov 23 07:23:16 2012 From: spamaxx at gmail.com (Maxim) Date: Fri, 23 Nov 2012 12:23:16 +0000 (UTC) Subject: [SciPy-User] =?utf-8?q?cubic_spline_interpolation_-_derivative_va?= =?utf-8?q?lue_in=09an_end_point?= References: Message-ID: sorry, just one more short comment. I can?t just symmetrically reflect data from [0;1] on [-1;0] to construct a spline interpolation on [-1;1] because of particularities of the experimental data (it's not just a set like [coordinate, value]). From pawel.kw at gmail.com Fri Nov 23 07:52:19 2012 From: pawel.kw at gmail.com (=?ISO-8859-2?Q?Pawe=B3_Kwa=B6niewski?=) Date: Fri, 23 Nov 2012 13:52:19 +0100 Subject: [SciPy-User] cubic spline interpolation - derivative value in an end point In-Reply-To: References: Message-ID: Can you give an example of the experimental data you have? I suppose you could find a way to transform your data into a [coordinate, value] matrix and then reflect it... Otherwise, I'm not aware of a way in which you could impose the derivative to be 0 in a given point (which does not mean there is no way to do it). Pawe? 2012/11/23 Maxim > sorry, just one more short comment. > I can?t just symmetrically reflect data from [0;1] on [-1;0] to construct a > spline interpolation on [-1;1] because of particularities of the > experimental > data (it's not just a set like [coordinate, value]). > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From spamaxx at gmail.com Fri Nov 23 08:26:12 2012 From: spamaxx at gmail.com (Maxim) Date: Fri, 23 Nov 2012 13:26:12 +0000 (UTC) Subject: [SciPy-User] cubic spline interpolation - derivative value in an end point References: Message-ID: Here is an example: I need to determine a temperature profile in a substance. The system has cylindrical symmetry so I need to determine 1D profile along a radius. I parametrise profile with cubic splines. That means that I define a grid like x = linspace(0,1,20) and y-values are parameters that I need to find. I have two types of measurements: the first one is usual measurements like radius-temperature, the second one is integrated measurements (integrals of temperature along arbitrary lines (which are not diameters). If I try to reflect measurements symmetrically from [0;1] to [-1;0] then it causes a problem with integrated measurements (how to reflect them correctly) And in fact it causes lots of calculations which are absolutely waste and could be avoided by just defining a value of derivative. From francois.beaubert at univ-valenciennes.fr Fri Nov 23 09:33:56 2012 From: francois.beaubert at univ-valenciennes.fr (=?ISO-8859-1?Q?Fran=E7ois_Beaubert?=) Date: Fri, 23 Nov 2012 15:33:56 +0100 Subject: [SciPy-User] Can loadtxt do this: complex file format Message-ID: <50AF8954.20108@univ-valenciennes.fr> Hi alll, I'm wondering if loadtxt can be used to read a file with a rather complicated file format (well at least for me :)) I would like to read data in a file (100 000 lines) which contains the Ux, Uy and Uz velocity components versus time for different points (x, y, z coordinates). The file's format looks like this: # x 0 0.005 # y 0 0.0025 # z 0.3 0.3 # Time 100 (7.665475124e-05 -0.0002514304858 0.07519945952) (0.0007792405436 -0.001695097619 0.1563337147) 200 (7.060958587e-05 -0.0002343546897 0.07631018852) (0.0007458315198 -0.001621335844 0.1705269028) 300 (6.511531713e-05 -0.0002186820111 0.07784758393) (0.0007208548956 -0.001563533763 0.1811116196) 400 (6.019627747e-05 -0.0002045291128 0.07981817652) (0.000700274915 -0.001515310657 0.1896792906) The first column is ... the time The 3 values in the first parentheses are the Ux, Uy and Uz velocity components on point 1, the values enclosed in the second parentheses are the Ux,Uy and Uz velocity components for point 2, etc 1. Is it possible to skip those parentheses to be able to read the data with loadtxt ? 2. Is it possible to store the points coordinates (referenced in the heading) in variables without a priori knowing the total numbers of positions ? Sorry if it looks like to easy for you, but I"m a newbie in python Any help will be welcome. Thanks a lot ! Cheers Francois -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: francois_beaubert.vcf Type: text/x-vcard Size: 454 bytes Desc: not available URL: From tmp50 at ukr.net Fri Nov 23 11:22:28 2012 From: tmp50 at ukr.net (Dmitrey) Date: Fri, 23 Nov 2012 18:22:28 +0200 Subject: [SciPy-User] [ANN] Stochastic programming and optimization addon for FuncDesigner v. 0.421 Message-ID: <31182.1353687748.10255534963624443904@ffe16.ukr.net> hi all, I'm glad to inform you that stochastic programming and optimization addon for FuncDesigner v. 0.421 has been released. Now you can use gradient-based solvers for numerical optimization, such as ALGENCAN, IPOPT, ralg, gsubg etc. Usually they work faster than derivative-free (such as scipy_cobyla, BOBYQA) or global (GLP) solvers, e.g. on this example ALGENCAN time elapsed is less than 1 second while scipy_cobyla spend ~20 sec. However pay attention that having function P() in your problem may bring nonconvexity or some other issues to the solver optimization trajectory, thus sometimes you'll have to use derivative-free or GLP solvers (e.g. de) instead. FuncDesigner is free (BSD license) cross-platform Python language written software, while its stochastic programming and optimization addon, written by same authors, is free for small-scaled problems with educational or research purposes only. For more details visit our website http://openopt.org ------------------------------------- Regards, D. http://openopt.org/Dmitrey -------------- next part -------------- An HTML attachment was scrubbed... URL: From runar.tenfjord at gmail.com Fri Nov 23 11:24:46 2012 From: runar.tenfjord at gmail.com (Runar Tenfjord) Date: Fri, 23 Nov 2012 16:24:46 +0000 (UTC) Subject: [SciPy-User] ANN : occmodel v0.1.0 Message-ID: I am pleased to announce the first official release of occmodel (v0.1.0) and the releated libraries geotools/gltools. Description ---------- occmodel is a small library which gives a high level access to the OpenCASCADE modelling kernel. For most users a direct use of the OpenCASCADE modelling kernel can be quite a hurdle as it is a huge library. The geometry can be visualized with the included viewer. This viewer is utilizing modern OpenGL methods like GLSL shaders and vertex buffers to ensure visual quality and maximum speed. To use the viewer OpenGL version 2.1 is needed. Home page : http://github.com/tenko/occmodel Documentation : http://tenko.github.com/occmodel/index.html In addition the following related libraries are released: geotools (required) : http://github.com/tenko/geotools Documentation : http://tenko.github.com/geotools/index.html gltools (optional) : http://github.com/tenko/gltools Documentation : http://tenko.github.com/gltools/index.html As this is the first official release some hurdles are expected Binary installers are available for the Windows platform. Best regards Runar Tenfjord From ecarlson at eng.ua.edu Fri Nov 23 12:03:20 2012 From: ecarlson at eng.ua.edu (Eric Carlson) Date: Fri, 23 Nov 2012 11:03:20 -0600 Subject: [SciPy-User] cubic spline interpolation - derivative value in an end point In-Reply-To: References: Message-ID: On 11/23/2012 4:52 AM, Pawe? Kwa?niewski wrote: > Hi, > > I don't really understand why you want to specify the derivative at 0. > If, as you say, your function has a 0 derivative at 0, if you > interpolate the function on the complete range where it is defined: > [-1:1] with interpolate.splrep(), you can later evaluate it with splev() > on whatever range you wish, like [0:1], and it should have the proper > derivative. Hello Pawe?, In fact, there are many cases where the derivative at the ends is available. For several years I used a (crudely numpy-wrapped) spline interpolation version from netlib, the same sources as those used for octave, in order to get clamped ends and (mostly) pchip. My wrapped versions performed much more slowly on evaluations than native scipy, though, so I quit using my wrapped versions once I found that pchip was available. My ugly hack for getting a clamped condition on normal interpolating cubic splines has been to insert a point, (x_near_end=x_end+small_dx, y_near_end=y_end+yprime_end*small_dx) into my sorted data array at position 1 or -2. The results of this have pretty decent compared with my wrapped spline routine, and even though the spacing is not uniform, evaluation is still very fast. Cheers, Eric From davidmenhur at gmail.com Fri Nov 23 20:24:51 2012 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Sat, 24 Nov 2012 02:24:51 +0100 Subject: [SciPy-User] Can loadtxt do this: complex file format In-Reply-To: <50AF8954.20108@univ-valenciennes.fr> References: <50AF8954.20108@univ-valenciennes.fr> Message-ID: Answering your first question: yes, you can use the arg "converters". I used it when I had to load data where the decimal separator was a comma for the first column and a dot for the rest of the file (a professional equipment, BTW). In your case, it would be something like (not tested): converters = {2: lambda s: float(s[1:)), 4: lambda s: float(s[:-1)), 5: lambda s: float(s[1:)), 7: lambda s: float(s[:-1))) I don't think I understand your second question. loadtxt will load the whole file, regardless of the number of columns or rows (and alocate memory dynamically). David. On Fri, Nov 23, 2012 at 3:33 PM, Fran?ois Beaubert wrote: > Hi alll, > > I'm wondering if loadtxt can be used to read a file with a rather > complicated file format (well at least for me :)) > > I would like to read data in a file (100 000 lines) which contains the Ux, > Uy and Uz velocity components versus time for different points (x, y, z > coordinates). > > The file's format looks like this: > > # x 0 0.005 > # y 0 0.0025 > # z 0.3 0.3 > # Time > 100 (7.665475124e-05 -0.0002514304858 0.07519945952) > (0.0007792405436 -0.001695097619 0.1563337147) > 200 (7.060958587e-05 -0.0002343546897 0.07631018852) > (0.0007458315198 -0.001621335844 0.1705269028) > 300 (6.511531713e-05 -0.0002186820111 0.07784758393) > (0.0007208548956 -0.001563533763 0.1811116196) > 400 (6.019627747e-05 -0.0002045291128 0.07981817652) > (0.000700274915 -0.001515310657 0.1896792906) > > The first column is ... the time > The 3 values in the first parentheses are the Ux, Uy and Uz velocity > components on point 1, the values enclosed in the second parentheses are the > Ux,Uy and Uz velocity components for point 2, etc > > 1. Is it possible to skip those parentheses to be able to read the data with > loadtxt ? > 2. Is it possible to store the points coordinates (referenced in the > heading) in variables without a priori knowing the total numbers of > positions ? > > Sorry if it looks like to easy for you, but I"m a newbie in python > Any help will be welcome. > > Thanks a lot ! > Cheers > Francois > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From mcp.stras at gmail.com Fri Nov 23 23:26:31 2012 From: mcp.stras at gmail.com (Martin Campos Pinto) Date: Sat, 24 Nov 2012 05:26:31 +0100 Subject: [SciPy-User] Iterative solver for Sparse Complex Matrices In-Reply-To: <50AF1444.9030502@gmail.com> References: <50AF1444.9030502@gmail.com> Message-ID: <76E91259-D493-4029-A908-9A6B0CCEF92F@gmail.com> Hi everybody, I haven't had much success with my first question so let me ask a new one : does anybody know an iterative solver (in scipy) that works for linear systems of equations with complex coefficients ? Thank you, Martin On 23 nov. 2012, at 07:14, Martin Campos Pinto wrote: > Hi all, I am having some troubles using iterative solvers with sparse complex matrices. > > Here is an example with gmres: the script > > from scipy import sparse > from numpy.linalg import norm > from numpy.random import rand > import scipy.sparse.linalg as spla > > C = sparse.lil_matrix((10, 10), dtype=complex) > C.setdiag(rand(10)) > C[0,3] = 1j > C = C.tocsr() > c = rand(10)+rand(10)*1j > > x = spla.spsolve(C, c) > print "spsolve: norm(C*x - c) = ", norm(C*x - c) > > (x,info) = spla.gmres(C, c) > print "gmres: norm(C*x - c) = ", norm(C*x - c) > > gives as output: > > spsolve: norm(C*x - c) = 1.57621370006e-16 > Segmentation fault > > > Actually, sometimes I get this error message instead of a Segmentation fault: > > Traceback (most recent call last): > File "test_gmres_cplx.py", line 45, in > (x,info) = spla.gmres(C, c) > File "/Library/Frameworks/EPD64.framework/Versions/7.2/lib/python2.7/site-packages/scipy/sparse/linalg/isolve/iterative.py", line 437, in gmres > revcom(b, x, restrt, work, work2, iter_, resid, info, ndx1, ndx2, ijob) > SystemError: NULL result without error in PyObject_Call > > > > Note that I have tested gmres with real sparse matrices, and it runs fine: Indeed > > C = sparse.lil_matrix((10, 10)) > C.setdiag(rand(10)) > C = C.tocsr() > c = rand(10) > > x = spla.spsolve(C, c) > print "spsolve: norm(C*x - c) = ", norm(C*x - c) > > (x,info) = spla.gmres(C, c) > print "gmres: norm(C*x - c) = ", norm(C*x - c) > > gives > > spsolve: norm(C*x - c) = 6.93889390391e-18 > gmres: norm(C*x - c) = 5.28860261481e-16 > > > I have looked on the web for solutions but haven't found any. Some very old posts indicate similar errors but they don't come with an answer, > and I imagine that if those were due to bugs, they should have been fixed by now. > > Am I doing something stupid here, or is that a real problem ? Is somebody aware of a solution ? > (I am using scipy version 0.10.1) > > Thanks, > Martin > > -- > Martin Campos Pinto > LJLL, UPMC & CNRS > From vs at it.uu.se Sat Nov 24 09:55:59 2012 From: vs at it.uu.se (Virgil Stokes) Date: Sat, 24 Nov 2012 15:55:59 +0100 Subject: [SciPy-User] Final (summary) on: qr decompostion gives negative q, r ? In-Reply-To: References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC0B34.6030200@it.uu.se> <50AC119B.1040603@it.uu.se> <50AC2528.8000404@it.uu.se> <50AC2E77.2070906@creativetrax.com> <50AC85E7.8000107@it.uu.se> Message-ID: <50B0DFFF.1070607@it.uu.se> After studying the qr decomposition using both numpy and MATLAB, I now realize that what I thought to be true about the results of the qr decomposition was incorrect. That is, as many of the those who replied to my emails on this pointed out: 1) The "sign" of the returned 'r' matrix really doesn't matter. 2) The same input to MATLAB and numpy (for a qr decomposition) can give different results (sign differences), and again this really doesn't matter (at least in my application). Thanks: Charles R. Harris, Pauli Virtanen, Sturla Molden, Alejandro Weinstein, Matthieu Brucher, Alex Leach, Jason Grout, Skipper Seabold, Robert Kern, and David Menhur for forcing me to think a little more about this. I am very grateful for your help. The final results (smoothed states and covariances) obtained from my Python/numpy implementation are the same as those returned from the original MATLAB implementation :-) Best regards, --V From davidmenhur at gmail.com Sat Nov 24 10:13:57 2012 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Sat, 24 Nov 2012 16:13:57 +0100 Subject: [SciPy-User] Final (summary) on: qr decompostion gives negative q, r ? In-Reply-To: <50B0DFFF.1070607@it.uu.se> References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC0B34.6030200@it.uu.se> <50AC119B.1040603@it.uu.se> <50AC2528.8000404@it.uu.se> <50AC2E77.2070906@creativetrax.com> <50AC85E7.8000107@it.uu.se> <50B0DFFF.1070607@it.uu.se> Message-ID: Glad we were useful. On Sat, Nov 24, 2012 at 3:55 PM, Virgil Stokes wrote: > I am very grateful for your help. The final results (smoothed states and > covariances) obtained from my Python/numpy implementation are the same as those > returned from the original MATLAB implementation :-) I am curious: what about timing? Is any of them significantly faster than the other? From charlesr.harris at gmail.com Sat Nov 24 11:17:34 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 24 Nov 2012 09:17:34 -0700 Subject: [SciPy-User] Final (summary) on: qr decompostion gives negative q, r ? In-Reply-To: <50B0DFFF.1070607@it.uu.se> References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC0B34.6030200@it.uu.se> <50AC119B.1040603@it.uu.se> <50AC2528.8000404@it.uu.se> <50AC2E77.2070906@creativetrax.com> <50AC85E7.8000107@it.uu.se> <50B0DFFF.1070607@it.uu.se> Message-ID: On Sat, Nov 24, 2012 at 7:55 AM, Virgil Stokes wrote: > After studying the qr decomposition using both numpy and MATLAB, I now > realize > that what I thought to be true about the results of the qr decomposition > was > incorrect. That is, as many of the those who replied to my emails on this > pointed out: > > 1) The "sign" of the returned 'r' matrix really doesn't matter. > 2) The same input to MATLAB and numpy (for a qr decomposition) can give > different results (sign differences), and again this really doesn't matter > (at > least in my application). > > Thanks: Charles R. Harris, Pauli Virtanen, Sturla Molden, Alejandro > Weinstein, > Matthieu Brucher, Alex Leach, Jason Grout, Skipper Seabold, Robert Kern, > and > David Menhur for forcing me to think a little more about this. > > I am very grateful for your help. The final results (smoothed states and > covariances) obtained from my Python/numpy implementation are the same as > those > returned from the original MATLAB implementation :-) > > That's a very graceful end to the discussion. Thanks for the update. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From vs at it.uu.se Sat Nov 24 13:02:11 2012 From: vs at it.uu.se (Virgil Stokes) Date: Sat, 24 Nov 2012 19:02:11 +0100 Subject: [SciPy-User] Final (summary) on: qr decompostion gives negative q, r ? In-Reply-To: References: <50AABA52.7020808@it.uu.se> <50ABFE4A.3040308@it.uu.se> <50AC0B34.6030200@it.uu.se> <50AC119B.1040603@it.uu.se> <50AC2528.8000404@it.uu.se> <50AC2E77.2070906@creativetrax.com> <50AC85E7.8000107@it.uu.se> <50B0DFFF.1070607@it.uu.se> Message-ID: <50B10BA3.6070702@it.uu.se> On 24-Nov-2012 16:13, Da?id wrote: > Glad we were useful. > > On Sat, Nov 24, 2012 at 3:55 PM, Virgil Stokes wrote: >> I am very grateful for your help. The final results (smoothed states and >> covariances) obtained from my Python/numpy implementation are the same as those >> returned from the original MATLAB implementation :-) > I am curious: what about timing? Is any of them significantly faster > than the other? Good question, MATLAB is faster (how significantly I have not quantified). --V > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From spamaxx at gmail.com Mon Nov 26 03:34:11 2012 From: spamaxx at gmail.com (Maxim) Date: Mon, 26 Nov 2012 08:34:11 +0000 (UTC) Subject: [SciPy-User] =?utf-8?q?cubic_spline_interpolation_-_derivative_va?= =?utf-8?q?lue_in=09an_end_point?= References: Message-ID: Thank you, Eric. But there is a problem with the method you proposed (maybe you have overcome it...): I use 'smooth' parameter in splrep function, and there is a jump between first and second node which is smoothed out by splrep. If I add another node between the two, it will be just smoothed out again and thus it makes no impact on the derivative. Cheers, Maxim From francois.beaubert at univ-valenciennes.fr Mon Nov 26 04:16:41 2012 From: francois.beaubert at univ-valenciennes.fr (=?UTF-8?B?RnJhbsOnb2lzIEJlYXViZXJ0?=) Date: Mon, 26 Nov 2012 10:16:41 +0100 Subject: [SciPy-User] Can loadtxt do this: complex file format In-Reply-To: References: <50AF8954.20108@univ-valenciennes.fr> Message-ID: <50B33379.8020804@univ-valenciennes.fr> Thank you very much David for your kind answer ! I will try your suggestion for question 1. Regarding question 2. I realized that my email client has messed up the formatting of my file. This file is written like this: # x 0 0.005 # y 0 0.0025 # z 0.3 0.3 # Time 100 (7.6e-05 -0.02 0.75) (0.06 -0.01 0.16) 200 (7.06e-05 -0.12 0.76) (0.07 -0.11 0.17) 300 (6.51e-05 -0.21 0.77) (0.72 -0.15 0.26) 400 (6.01e-05 -0.20 0.79) (0.05 -0.13 0.19) I just want to plot my data versus time: After the last line of the header (beginning with "#") the file is formatted like this: * The first column is the time * The 3 values in the first parentheses are the Ux, Uy and Uz velocity components on point 1 (coordinates x1, y1, z1) * The 3 values enclosed in the second parentheses are the Ux,Uy and Uz velocity components for point 2, etc So I have to read the first column (easy part) and the other ones but without the "(" and ")". I'm wondering if loadtxt() can skip/remove the "(" and ")" chars to be able to read my data with something like : time,Ux1,Uy1,Uz1,Ux2,Uy2,Uz2 = loadtxt(filename, comments='#', delimiter=None,converters=None, usecols=(0,1,2,3,4,5,6), unpack=True) Do I have to first remove all the "(" and ")" in the file or can I use some tricky loadtxt() features to do this for me ? Thank you very much for your help Have a nice day Fran?ois Le 24/11/2012 02:24, Da?id a ?crit : > Answering your first question: yes, you can use the arg "converters". > I used it when I had to load data where the decimal separator was a > comma for the first column and a dot for the rest of the file (a > professional equipment, BTW). > > In your case, it would be something like (not tested): > > converters = {2: lambda s: float(s[1:)), 4: lambda s: float(s[:-1)), > 5: lambda s: float(s[1:)), 7: lambda s: float(s[:-1))) > > > I don't think I understand your second question. loadtxt will load the > whole file, regardless of the number of columns or rows (and alocate > memory dynamically). > > > > David. -------------- next part -------------- A non-text attachment was scrubbed... Name: francois_beaubert.vcf Type: text/x-vcard Size: 465 bytes Desc: not available URL: From davidmenhur at gmail.com Mon Nov 26 05:59:44 2012 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Mon, 26 Nov 2012 11:59:44 +0100 Subject: [SciPy-User] Can loadtxt do this: complex file format In-Reply-To: <50B33379.8020804@univ-valenciennes.fr> References: <50AF8954.20108@univ-valenciennes.fr> <50B33379.8020804@univ-valenciennes.fr> Message-ID: On Mon, Nov 26, 2012 at 10:16 AM, Fran?ois Beaubert wrote: converters = {2: lambda s: float(s[1:)), 4: lambda s: float(s[:-1)), 5: lambda s: float(s[1:)), 7: lambda s: float(s[:-1))) time,Ux1,Uy1,Uz1,Ux2,Uy2,Uz2 = loadtxt(filename, comments='#', skiprows = 4, delimiter=None, usecols=(0,1,2,3,4,5,6), unpack=True, converters = converters) I have added skiprows to jump over the first few lines, and the converters = converters to get rid off the " ( ". I insist, I haven't tested my converters function, so it may be wrong, but it should be easy to fix. David. From francois.beaubert at univ-valenciennes.fr Mon Nov 26 08:18:00 2012 From: francois.beaubert at univ-valenciennes.fr (=?UTF-8?B?RnJhbsOnb2lzIEJlYXViZXJ0?=) Date: Mon, 26 Nov 2012 14:18:00 +0100 Subject: [SciPy-User] Can loadtxt do this: complex file format In-Reply-To: References: <50AF8954.20108@univ-valenciennes.fr> <50B33379.8020804@univ-valenciennes.fr> Message-ID: <50B36C08.40306@univ-valenciennes.fr> Great David, it works for me ! I just have to change your previous converters dictionary with this one: converters = {1: lambda s: float(s[1:]), 3: lambda s: float(s[:-1]),4: lambda s: float(s[1:]), 6: lambda s: float(s[:-1])} Just have to find a way to automate this with the number of points I have in the header. Many, many thanks David ! You've maid my day. Cheers Fran?ois Le lun. 26 nov. 2012 11:59:44 CET, Da?id a ?crit : > On Mon, Nov 26, 2012 at 10:16 AM, Fran?ois Beaubert > wrote: > > converters = {2: lambda s: float(s[1:)), 4: lambda s: float(s[:-1)), > 5: lambda s: float(s[1:)), 7: lambda s: float(s[:-1))) > time,Ux1,Uy1,Uz1,Ux2,Uy2,Uz2 = loadtxt(filename, comments='#', > skiprows = 4, delimiter=None, usecols=(0,1,2,3,4,5,6), unpack=True, > converters = converters) > > I have added skiprows to jump over the first few lines, and the > converters = converters to get rid off the " ( ". I insist, I haven't > tested my converters function, so it may be wrong, but it should be > easy to fix. > > > David. -------------- next part -------------- A non-text attachment was scrubbed... Name: francois_beaubert.vcf Type: text/x-vcard Size: 465 bytes Desc: not available URL: From takowl at gmail.com Mon Nov 26 09:14:39 2012 From: takowl at gmail.com (Thomas Kluyver) Date: Mon, 26 Nov 2012 14:14:39 +0000 Subject: [SciPy-User] New Scipy website Message-ID: I'd like to discuss how we can move the new SciPy website (currently at http://scipy.github.com/ ) to be the homepage of scipy.org. The new site is still something of a work in progress, but it's been there for years, and we can't continue to maintain two sites indefinitely. We've already been bitten by people referring to out of date FAQs, and I think the aesthetics of the new site are a marked improvement. So, first, what does the new site need before it can go live?* *Two things I'm already aware of: - Installation instructions for other package managers (so far we cover apt and macports). Please can users of other Linux distros send me the corresponding installation commands? http://scipy.github.com/install.html - The 'Getting started' page should be rewritten to describe the Scipy stack, and from a more newcomer-ish perspective. Of course, there's a lot of useful information on the existing site - such as the Cookbook - that it's impractical to move to the new site. So, I suggest that we move the current MoinMoin-based site to wiki.scipy.org, and provide redirects so we don't break existing links. Unfortunately, Github pages can't do proper HTTP-based redirects. So we can either use hackish HTML/JS redirects, or host the new site on a server where we can set up proper redirects. Thanks, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Mon Nov 26 09:27:03 2012 From: takowl at gmail.com (Thomas Kluyver) Date: Mon, 26 Nov 2012 14:27:03 +0000 Subject: [SciPy-User] New Scipy website In-Reply-To: <50B37AC6.1050603@hilboll.de> References: <50B37AC6.1050603@hilboll.de> Message-ID: On 26 November 2012 14:20, Andreas Hilboll wrote: > > Wouldn't it be possible to put "empty" HTML files into github pages with > a header content of > > > > While being inefficient and ugly, I think it should work in principle, > and there's no JS involved. Yep, that's the HTML side of the 'ugly HTML/JS redirects' ;-). Apparently, it can break the back button in some browsers, because it goes back to the page which triggers the redirect. If we go down that route, I'd probably also include a plain link saying "this page can now be found at http://..." Thanks, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From erichuns at gmail.com Sat Nov 24 10:36:57 2012 From: erichuns at gmail.com (Eric Hunsberger) Date: Sat, 24 Nov 2012 10:36:57 -0500 Subject: [SciPy-User] scipy.linalg cannot find OpenBLAS Message-ID: I recently compiled my numpy/scipy to use OpenBLAS, and it seemed to be working fine (i.e., normal numpy functions seem to use it). However, when I try to import scipy.linalg, I get an error that libopenblas.so.0 cannot be found: In [7]: import scipy.linalg as spl --------------------------------------------------------------------------- ImportError Traceback (most recent call last) /media/Data/Dropbox/code/ in () ----> 1 import scipy.linalg as spl /usr/local/lib/python2.7/dist-packages/scipy/linalg/__init__.py in () 131 from linalg_version import linalg_version as __version__ 132 --> 133 from misc import * 134 from basic import * 135 from decomp import * /usr/local/lib/python2.7/dist-packages/scipy/linalg/misc.py in () 1 import numpy as np 2 from numpy.linalg import LinAlgError ----> 3 import fblas 4 5 __all__ = ['LinAlgError', 'norm'] ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory If I use scipy.show_config(), everything seems to be hooked up fine: In [6]: sp.show_config() blas_info: libraries = ['openblas'] library_dirs = ['/opt/OpenBLAS/lib'] language = f77 lapack_info: libraries = ['openblas'] library_dirs = ['/opt/OpenBLAS/lib'] language = f77 atlas_threads_info: NOT AVAILABLE blas_opt_info: libraries = ['openblas'] library_dirs = ['/opt/OpenBLAS/lib'] language = f77 define_macros = [('NO_ATLAS_INFO', 1)] atlas_blas_threads_info: NOT AVAILABLE umfpack_info: NOT AVAILABLE lapack_opt_info: libraries = ['openblas', 'openblas'] library_dirs = ['/opt/OpenBLAS/lib'] language = f77 define_macros = [('NO_ATLAS_INFO', 1)] atlas_info: NOT AVAILABLE lapack_mkl_info: NOT AVAILABLE blas_mkl_info: NOT AVAILABLE atlas_blas_info: NOT AVAILABLE mkl_info: NOT AVAILABLE Does anyone have any ideas as to why this is happening, or how I can go about fixing it? Why is scipy.linalg not looking in the same place as scipy seems to be, and how can I figure out where it is looking? Best, Eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From andreas at hilboll.de Mon Nov 26 09:20:54 2012 From: andreas at hilboll.de (Andreas Hilboll) Date: Mon, 26 Nov 2012 15:20:54 +0100 Subject: [SciPy-User] New Scipy website In-Reply-To: References: Message-ID: <50B37AC6.1050603@hilboll.de> > Unfortunately, Github pages can't do proper > HTTP-based redirects. So we can either use hackish HTML/JS redirects, or > host the new site on a server where we can set up proper redirects. Wouldn't it be possible to put "empty" HTML files into github pages with a header content of While being inefficient and ugly, I think it should work in principle, and there's no JS involved. Cheers, Andreas. From helmrp at yahoo.com Mon Nov 26 10:39:06 2012 From: helmrp at yahoo.com (The Helmbolds) Date: Mon, 26 Nov 2012 07:39:06 -0800 (PST) Subject: [SciPy-User] 'brute' function signature error In-Reply-To: References: Message-ID: <1353944346.62594.YahooMailNeo@web31805.mail.mud.yahoo.com> There's an error in the 'brute' function signature. ? Replace ????finish=fmin with ????finish='fmin' ? Bob H -------------- next part -------------- An HTML attachment was scrubbed... URL: From beamesleach at gmail.com Mon Nov 26 11:03:12 2012 From: beamesleach at gmail.com (Alex Leach) Date: Mon, 26 Nov 2012 16:03:12 +0000 Subject: [SciPy-User] scipy.linalg cannot find OpenBLAS In-Reply-To: References: Message-ID: <1596490.Vvk2PUC53E@metabuntu> On Saturday 24 Nov 2012 10:36:57 Eric Hunsberger wrote: > I recently compiled my numpy/scipy to use OpenBLAS, and it seemed to be > working fine (i.e., normal numpy functions seem to use it). However, when I > try to import scipy.linalg, I get an error that libopenblas.so.0 cannot be > found: > > > /usr/local/lib/python2.7/dist-packages/scipy/linalg/misc.py in () > 1 import numpy as np > 2 from numpy.linalg import LinAlgError > ----> 3 import fblas > 4 > 5 __all__ = ['LinAlgError', 'norm'] > > ImportError: libopenblas.so.0: cannot open shared object file: No such file > or directory This error's coming from scipy/linalg/fblas.so, so looks like this module was linked incorrectly. (I assume you're on Linux). Can you run:- ldd /usr/local/lib/python2.7/dist-packages/scipy/linalg/fblas.so On Mac OS X, it's:- otool -L .../fblas.so If there is a line saying 'not found', then you probably want to add /opt/OpenBLAS/lib to your LD_LIBRARY_PATH, (or to a file in /etc/ld.so.conf.d/ and run `sudo ldconfig`). Cheers, Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From jason-sage at creativetrax.com Mon Nov 26 12:21:37 2012 From: jason-sage at creativetrax.com (Jason Grout) Date: Mon, 26 Nov 2012 11:21:37 -0600 Subject: [SciPy-User] permutation matrix in LU decomposition Message-ID: <50B3A521.3080206@creativetrax.com> I just noticed that the permutation matrix P returned in scipy's LU decomposition is the inverse (i.e., transpose) of the permutation matrix returned from Matlab's LU function. In scipy, A = PLU, but in Matlab, PA=LU [1]. It's just a convention, of course, but perhaps a note about this could be added to the relevant table row of http://www.scipy.org/NumPy_for_Matlab_Users? Thanks, Jason [1] http://www.mathworks.com/help/matlab/ref/lu.html From ralf.gommers at gmail.com Mon Nov 26 15:53:12 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 26 Nov 2012 21:53:12 +0100 Subject: [SciPy-User] 'brute' function signature error In-Reply-To: <1353944346.62594.YahooMailNeo@web31805.mail.mud.yahoo.com> References: <1353944346.62594.YahooMailNeo@web31805.mail.mud.yahoo.com> Message-ID: On Mon, Nov 26, 2012 at 4:39 PM, The Helmbolds wrote: > There's an error in the 'brute' function signature. > > Replace > finish=fmin > with > finish='fmin' > No, it's correct. `finish` is a callable, not a string. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ecarlson at eng.ua.edu Mon Nov 26 18:45:13 2012 From: ecarlson at eng.ua.edu (Eric Carlson) Date: Mon, 26 Nov 2012 17:45:13 -0600 Subject: [SciPy-User] cubic spline interpolation - derivative value in an end point In-Reply-To: References: Message-ID: On 11/26/2012 2:34 AM, Maxim wrote: > Thank you, Eric. > But there is a problem with the method you proposed (maybe you have overcome > it...): > I use 'smooth' parameter in splrep function, and there is a jump between first > and second node which is smoothed out by splrep. > If I add another node between the two, it will be just smoothed out again and > thus it makes no impact on the derivative. > > Cheers, > Maxim > If you are smoothing, then my original hack will definitely not work. Given (xdata,ydata), instead of a direct fit maybe something like the following to brute force a zero slope: import scipy as sp import scipy.optimize as optimize from numpy.random import randn import scipy.interpolate as interpolate ### xdata=sp.linspace(0,sp.pi,51) ydata=sp.cos(xdata)+ randn(51)/10 ## smooth = 1.0 y_prime_target = 0.0 x_hack = -.2 x_for_yprime = 0 xfit=sp.insert(xdata,0,x_hack)# or something near x=0 def slope_hack(y_hack): yfit=sp.insert(ydata,0,y_hack)#just a placeholder f=interpolate.UnivariateSpline(xfit, yfit, k=3, s=smooth) return f(x_for_yprime,1)-y_prime_target y_hack = 1.0 y_hack = optimize.fsolve(slope_hack, y_hack) yfit=sp.insert(ydata,0,y_hack) f_clamp=interpolate.UnivariateSpline(xfit, yfit, k=3, s=smooth) f=interpolate.UnivariateSpline(xdata, ydata, k=3, s=smooth) print 'derivative at x=0 for clamped',f_clamp(0,1), '\nderivative at x=0 for regular', f(0,1) import matplotlib.pyplot as plt plt.plot(xdata,ydata, 'r.', label='data') xplot=sp.linspace(0,sp.pi,201) plt.plot(xplot, f_clamp(xplot),'-b', label='clamp fit') plt.plot(xplot, f(xplot), '--g', label='normal fit') plt.legend(loc=0) plt.show() From spamaxx at gmail.com Tue Nov 27 03:58:36 2012 From: spamaxx at gmail.com (Maxim) Date: Tue, 27 Nov 2012 08:58:36 +0000 (UTC) Subject: [SciPy-User] =?utf-8?q?cubic_spline_interpolation_-_derivative_va?= =?utf-8?q?lue_in=09an_end_point?= References: Message-ID: Thank you a lot, Eric! From pawel.kw at gmail.com Tue Nov 27 07:10:39 2012 From: pawel.kw at gmail.com (=?ISO-8859-2?Q?Pawe=B3_Kwa=B6niewski?=) Date: Tue, 27 Nov 2012 13:10:39 +0100 Subject: [SciPy-User] New Scipy website In-Reply-To: References: Message-ID: Hi Thomas, I'm currently using Fedora 17 (x86_64), and here the set of packages listed on the scipy installation instructions site can be installed using the following command: > sudo yum install numpy scipy python-matplotlib ipython python-pandas sympy python-nose The versions available in the repositories can be viewed by running: > sudo yum info numpy scipy python-matplotlib ipython python-pandas sympy python-nose The above command will not install anything, just print the info. I attach the output of this command run today. I hope this helps. As for myself, I prefer to use the pip installer, since the repository versions seem to have quite a lag in Fedora. This could also be listed as a way to install different python packages, as it's quite simple: > sudo pip install Cheers, Pawe? 2012/11/26 Thomas Kluyver > I'd like to discuss how we can move the new SciPy website (currently at > http://scipy.github.com/ ) to be the homepage of scipy.org. > > The new site is still something of a work in progress, but it's been there > for years, and we can't continue to maintain two sites indefinitely. We've > already been bitten by people referring to out of date FAQs, and I think > the aesthetics of the new site are a marked improvement. So, first, what > does the new site need before it can go live?* > > *Two things I'm already aware of: > > - Installation instructions for other package managers (so far we cover > apt and macports). Please can users of other Linux distros send me the > corresponding installation commands? > http://scipy.github.com/install.html > > - The 'Getting started' page should be rewritten to describe the Scipy > stack, and from a more newcomer-ish perspective. > > Of course, there's a lot of useful information on the existing site - such > as the Cookbook - that it's impractical to move to the new site. So, I > suggest that we move the current MoinMoin-based site to wiki.scipy.org, > and provide redirects so we don't break existing links. Unfortunately, > Github pages can't do proper HTTP-based redirects. So we can either use > hackish HTML/JS redirects, or host the new site on a server where we can > set up proper redirects. > > Thanks, > Thomas > > > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: scipy.info Type: application/octet-stream Size: 7651 bytes Desc: not available URL: From takowl at gmail.com Tue Nov 27 07:28:32 2012 From: takowl at gmail.com (Thomas Kluyver) Date: Tue, 27 Nov 2012 12:28:32 +0000 Subject: [SciPy-User] New Scipy website In-Reply-To: References: Message-ID: Thanks Pawe?, I'll add that to the page. Of those, I think only IPython and matplotlib fall short of the specified minimum versions (0.13 and 1.1, respectively). For IPython, I don't mind pointing users to pip, but I'd rather not go down that route for matplotlib, because the user would have to start thinking about build dependencies. I think matplotlib 1.0.1 should tide most users over fine until they get an upgraded version. Best wishes, Thomas On 27 November 2012 12:10, Pawe? Kwa?niewski wrote: > Hi Thomas, > > I'm currently using Fedora 17 (x86_64), and here the set of packages > listed on the scipy installation instructions site can be installed using > the following command: > > > sudo yum install numpy scipy python-matplotlib ipython python-pandas > sympy python-nose > > The versions available in the repositories can be viewed by running: > > > sudo yum info numpy scipy python-matplotlib ipython python-pandas sympy > python-nose > > The above command will not install anything, just print the info. I attach > the output of this command run today. I hope this helps. > > As for myself, I prefer to use the pip installer, since the repository > versions seem to have quite a lag in Fedora. This could also be listed as a > way to install different python packages, as it's quite simple: > > > sudo pip install > > > Cheers, > > Pawe? > > > 2012/11/26 Thomas Kluyver > >> I'd like to discuss how we can move the new SciPy website (currently at >> http://scipy.github.com/ ) to be the homepage of scipy.org. >> >> The new site is still something of a work in progress, but it's been >> there for years, and we can't continue to maintain two sites indefinitely. >> We've already been bitten by people referring to out of date FAQs, and I >> think the aesthetics of the new site are a marked improvement. So, first, >> what does the new site need before it can go live?* >> >> *Two things I'm already aware of: >> >> - Installation instructions for other package managers (so far we cover >> apt and macports). Please can users of other Linux distros send me the >> corresponding installation commands? >> http://scipy.github.com/install.html >> >> - The 'Getting started' page should be rewritten to describe the Scipy >> stack, and from a more newcomer-ish perspective. >> >> Of course, there's a lot of useful information on the existing site - >> such as the Cookbook - that it's impractical to move to the new site. So, I >> suggest that we move the current MoinMoin-based site to wiki.scipy.org, >> and provide redirects so we don't break existing links. Unfortunately, >> Github pages can't do proper HTTP-based redirects. So we can either use >> hackish HTML/JS redirects, or host the new site on a server where we can >> set up proper redirects. >> >> Thanks, >> Thomas >> >> >> >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at hilboll.de Tue Nov 27 09:37:01 2012 From: lists at hilboll.de (Andreas Hilboll) Date: Tue, 27 Nov 2012 15:37:01 +0100 Subject: [SciPy-User] New Scipy website In-Reply-To: References: Message-ID: <50B4D00D.7060707@hilboll.de> I accidentally replied to Pawel off-list. Here's our exchange: Hi Andreas, You're right, yum info does not require a sudo. To be honest, I was lazy and didn't even check this - I just copy-pasted the install command and changed the 'install' to 'info' in my previous e-mail. But that's a good point - not reason for using sudo where it's not necessary. Cheers, Pawe? 2012/11/27 Andreas Hilboll Hi Pawel, > The versions available in the repositories can be viewed by running: > >> sudo yum info numpy scipy python-matplotlib ipython python-pandas > sympy python-nose I'm not a Fedora user at all, but are you sure that yum really needs to be run as sudo in order to print information? If not, I would vote for leaving out the `sudo` for the `yum info` call. Cheers, Andreas. From lserinol at gmail.com Tue Nov 27 10:40:36 2012 From: lserinol at gmail.com (Levent Serinol) Date: Tue, 27 Nov 2012 17:40:36 +0200 Subject: [SciPy-User] SVD and SVDS Message-ID: Hi, I'm wondering why SVD is removed in new scipy versions. Now, there is only SVDS function in package. What's the difference between SVD and SVDS mainly ? and is there any difference between SVD and SVDS output U,S,V values ? Thanks, -------------- next part -------------- An HTML attachment was scrubbed... URL: From jason-sage at creativetrax.com Tue Nov 27 11:43:59 2012 From: jason-sage at creativetrax.com (Jason Grout) Date: Tue, 27 Nov 2012 10:43:59 -0600 Subject: [SciPy-User] SVD and SVDS In-Reply-To: References: Message-ID: <50B4EDCF.6050605@creativetrax.com> On 11/27/12 9:40 AM, Levent Serinol wrote: > Hi, > > I'm wondering why SVD is removed in new scipy versions. Now, there is > only SVDS function in package. > > What's the difference between SVD and SVDS mainly ? > and is there any difference between SVD and SVDS output U,S,V values ? > Just glancing at the docs, it seems that svds [1] is for sparse matrices and svd [2] is for dense matrices. It doesn't look like the svd function is removed. Thanks, Jason [1] http://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.svds.html [2] http://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.svd.html#scipy.linalg.svd From pav at iki.fi Tue Nov 27 11:43:55 2012 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 27 Nov 2012 18:43:55 +0200 Subject: [SciPy-User] SVD and SVDS In-Reply-To: References: Message-ID: 27.11.2012 17:40, Levent Serinol kirjoitti: > I'm wondering why SVD is removed in new scipy versions. Now, there is > only SVDS function in package. Maybe I don't understand what you exactly mean, but it's not removed: http://docs.scipy.org/doc/scipy/reference/genindex.html#S -- Pauli Virtanen From pav at iki.fi Tue Nov 27 12:03:39 2012 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 27 Nov 2012 19:03:39 +0200 Subject: [SciPy-User] SVD and SVDS In-Reply-To: References: Message-ID: 27.11.2012 18:43, Pauli Virtanen kirjoitti: > 27.11.2012 17:40, Levent Serinol kirjoitti: >> I'm wondering why SVD is removed in new scipy versions. Now, there is >> only SVDS function in package. > > Maybe I don't understand what you exactly mean, but it's not removed: > > http://docs.scipy.org/doc/scipy/reference/genindex.html#S And maybe also this: http://docs.scipy.org/doc/scipy/reference/release.0.9.0.html#arpack-interface-changes From andreas at hilboll.de Tue Nov 27 08:15:39 2012 From: andreas at hilboll.de (Andreas Hilboll) Date: Tue, 27 Nov 2012 14:15:39 +0100 Subject: [SciPy-User] New Scipy website In-Reply-To: References: Message-ID: <50B4BCFB.4080605@hilboll.de> Hi Pawel, > The versions available in the repositories can be viewed by running: > >> sudo yum info numpy scipy python-matplotlib ipython python-pandas > sympy python-nose I'm not a Fedora user at all, but are you sure that yum really needs to be run as sudo in order to print information? If not, I would vote for leaving out the `sudo` for the `yum info` call. Cheers, Andreas. From andreas at hilboll.de Tue Nov 27 09:36:19 2012 From: andreas at hilboll.de (Andreas Hilboll) Date: Tue, 27 Nov 2012 15:36:19 +0100 Subject: [SciPy-User] New Scipy website In-Reply-To: References: Message-ID: <50B4CFE3.2020103@hilboll.de> I accidentally replied to Pawel off-list. Here's our exchange: Hi Andreas, You're right, yum info does not require a sudo. To be honest, I was lazy and didn't even check this - I just copy-pasted the install command and changed the 'install' to 'info' in my previous e-mail. But that's a good point - not reason for using sudo where it's not necessary. Cheers, Pawe? 2012/11/27 Andreas Hilboll Hi Pawel, > The versions available in the repositories can be viewed by running: > >> sudo yum info numpy scipy python-matplotlib ipython python-pandas > sympy python-nose I'm not a Fedora user at all, but are you sure that yum really needs to be run as sudo in order to print information? If not, I would vote for leaving out the `sudo` for the `yum info` call. Cheers, Andreas. From jrocher at enthought.com Wed Nov 28 13:21:21 2012 From: jrocher at enthought.com (Jonathan Rocher) Date: Wed, 28 Nov 2012 12:21:21 -0600 Subject: [SciPy-User] Request for Feedback/Authorization: a new public repository for the conference Message-ID: Dear all, Andy Terrel and I are the co-chairs of the Scipy2013 conference in Austin next June in Austin. We are very excited about this coming edition and will post regularly to these lists for feedback and announcements. But for now, we would like to consolidate its organization under a public open source project inside the Scipy organization on github. We propose that the project be called scipy-conference. The repo would host documents useful to run the conference as well as code to set up the website and the backend. I write here to ask if the community is ok with that and to ask for the rights to create/manage that new repository. Thanks in advance, Jonathan -- Jonathan Rocher, PhD Scientific software developer Enthought, Inc. jrocher at enthought.com 1-512-536-1057 http://www.enthought.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Nov 28 13:28:09 2012 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 28 Nov 2012 19:28:09 +0100 Subject: [SciPy-User] [SciPy-Dev] Request for Feedback/Authorization: a new public repository for the conference In-Reply-To: References: Message-ID: On Wed, Nov 28, 2012 at 7:21 PM, Jonathan Rocher wrote: > Dear all, > > Andy Terrel and I are the co-chairs of the Scipy2013 conference in Austin > next June in Austin. We are very excited about this coming edition and will > post regularly to these lists for feedback and announcements. > > But for now, we would like to consolidate its organization under a public > open source project inside the Scipy organization on github. We propose that > the project be called scipy-conference. The repo would host documents useful > to run the conference as well as code to set up the website and the backend. > > I write here to ask if the community is ok with that and to ask for the > rights to create/manage that new repository. +1. If there is no strenuous objection in short order, I will make a "SciPy Conference" team under the scipy organization and add you two to it. -- Robert Kern From robert.kern at gmail.com Wed Nov 28 13:37:32 2012 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 28 Nov 2012 19:37:32 +0100 Subject: [SciPy-User] [SciPy-Dev] Request for Feedback/Authorization: a new public repository for the conference In-Reply-To: References: Message-ID: On Wed, Nov 28, 2012 at 7:30 PM, Pauli Virtanen wrote: > 28.11.2012 20:28, Robert Kern kirjoitti: > [clip] >> +1. If there is no strenuous objection in short order, I will make a >> "SciPy Conference" team under the scipy organization and add you two >> to it. > > +1 also from me. On the basis of exit polls, I'm calling this election early. Done. -- Robert Kern From pav at iki.fi Wed Nov 28 13:30:53 2012 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 28 Nov 2012 20:30:53 +0200 Subject: [SciPy-User] [SciPy-Dev] Request for Feedback/Authorization: a new public repository for the conference In-Reply-To: References: Message-ID: 28.11.2012 20:28, Robert Kern kirjoitti: [clip] > +1. If there is no strenuous objection in short order, I will make a > "SciPy Conference" team under the scipy organization and add you two > to it. +1 also from me. -- Pauli Virtanen From pav at iki.fi Wed Nov 28 14:17:18 2012 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 28 Nov 2012 21:17:18 +0200 Subject: [SciPy-User] [SciPy-Dev] Request for Feedback/Authorization: a new public repository for the conference In-Reply-To: References: Message-ID: 28.11.2012 20:37, Robert Kern kirjoitti: > On Wed, Nov 28, 2012 at 7:30 PM, Pauli Virtanen wrote: >> 28.11.2012 20:28, Robert Kern kirjoitti: >> [clip] >>> +1. If there is no strenuous objection in short order, I will make a >>> "SciPy Conference" team under the scipy organization and add you two >>> to it. >> >> +1 also from me. > > On the basis of exit polls, I'm calling this election early. Done. The repo is now also there, so things should be ready to roll. -- Pauli Virtanen From jrocher at enthought.com Wed Nov 28 14:33:40 2012 From: jrocher at enthought.com (Jonathan Rocher) Date: Wed, 28 Nov 2012 13:33:40 -0600 Subject: [SciPy-User] [SciPy-Dev] Request for Feedback/Authorization: a new public repository for the conference In-Reply-To: References: Message-ID: Thanks to all. Jonathan On Wed, Nov 28, 2012 at 1:17 PM, Pauli Virtanen wrote: > 28.11.2012 20:37, Robert Kern kirjoitti: > > On Wed, Nov 28, 2012 at 7:30 PM, Pauli Virtanen wrote: > >> 28.11.2012 20:28, Robert Kern kirjoitti: > >> [clip] > >>> +1. If there is no strenuous objection in short order, I will make a > >>> "SciPy Conference" team under the scipy organization and add you two > >>> to it. > >> > >> +1 also from me. > > > > On the basis of exit polls, I'm calling this election early. Done. > > The repo is now also there, so things should be ready to roll. > > -- > Pauli Virtanen > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Jonathan Rocher, PhD Scientific software developer Enthought, Inc. jrocher at enthought.com 1-512-536-1057 http://www.enthought.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Nov 28 15:00:32 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 28 Nov 2012 21:00:32 +0100 Subject: [SciPy-User] permutation matrix in LU decomposition In-Reply-To: <50B3A521.3080206@creativetrax.com> References: <50B3A521.3080206@creativetrax.com> Message-ID: On Mon, Nov 26, 2012 at 6:21 PM, Jason Grout wrote: > I just noticed that the permutation matrix P returned in scipy's LU > decomposition is the inverse (i.e., transpose) of the permutation matrix > returned from Matlab's LU function. In scipy, A = PLU, but in Matlab, > PA=LU [1]. It's just a convention, of course, but perhaps a note about > this could be added to the relevant table row of > http://www.scipy.org/NumPy_for_Matlab_Users? > Sure, done. Note that that page is a wiki, so feel free to make more updates. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From jason-sage at creativetrax.com Wed Nov 28 15:22:00 2012 From: jason-sage at creativetrax.com (Jason Grout) Date: Wed, 28 Nov 2012 14:22:00 -0600 Subject: [SciPy-User] permutation matrix in LU decomposition In-Reply-To: References: <50B3A521.3080206@creativetrax.com> Message-ID: <50B67268.5070804@creativetrax.com> On 11/28/12 2:00 PM, Ralf Gommers wrote: > > > > On Mon, Nov 26, 2012 at 6:21 PM, Jason Grout > > wrote: > > I just noticed that the permutation matrix P returned in scipy's LU > decomposition is the inverse (i.e., transpose) of the permutation matrix > returned from Matlab's LU function. In scipy, A = PLU, but in Matlab, > PA=LU [1]. It's just a convention, of course, but perhaps a note about > this could be added to the relevant table row of > http://www.scipy.org/NumPy_for_Matlab_Users? > > > Sure, done. Note that that page is a wiki, so feel free to make more > updates. Ah, right. Had I realized that, I would have just made the update myself. Thanks, Jason From jason-sage at creativetrax.com Wed Nov 28 15:31:49 2012 From: jason-sage at creativetrax.com (Jason Grout) Date: Wed, 28 Nov 2012 14:31:49 -0600 Subject: [SciPy-User] permutation matrix in LU decomposition In-Reply-To: References: <50B3A521.3080206@creativetrax.com> Message-ID: <50B674B5.1010404@creativetrax.com> On 11/28/12 2:00 PM, Ralf Gommers wrote: > > > > On Mon, Nov 26, 2012 at 6:21 PM, Jason Grout > > wrote: > > I just noticed that the permutation matrix P returned in scipy's LU > decomposition is the inverse (i.e., transpose) of the permutation matrix > returned from Matlab's LU function. In scipy, A = PLU, but in Matlab, > PA=LU [1]. It's just a convention, of course, but perhaps a note about > this could be added to the relevant table row of > http://www.scipy.org/NumPy_for_Matlab_Users? > > > Sure, done. Note that that page is a wiki, so feel free to make more > updates. I just tried tweaking it a bit, and got this error when I tried to submit my edits: [Errno 31] Too many links: '/home/scipy/wiki/data/pages/JasonGrout(2f)MoinEditorBackup' See http://sage.math.washington.edu/home/jason/numpywikierror.html for more debugging output. Thanks, Jason From ralf.gommers at gmail.com Wed Nov 28 15:35:48 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 28 Nov 2012 21:35:48 +0100 Subject: [SciPy-User] permutation matrix in LU decomposition In-Reply-To: <50B674B5.1010404@creativetrax.com> References: <50B3A521.3080206@creativetrax.com> <50B674B5.1010404@creativetrax.com> Message-ID: On Wed, Nov 28, 2012 at 9:31 PM, Jason Grout wrote: > On 11/28/12 2:00 PM, Ralf Gommers wrote: > > > > > > > > On Mon, Nov 26, 2012 at 6:21 PM, Jason Grout > > > > wrote: > > > > I just noticed that the permutation matrix P returned in scipy's LU > > decomposition is the inverse (i.e., transpose) of the permutation > matrix > > returned from Matlab's LU function. In scipy, A = PLU, but in > Matlab, > > PA=LU [1]. It's just a convention, of course, but perhaps a note > about > > this could be added to the relevant table row of > > http://www.scipy.org/NumPy_for_Matlab_Users? > > > > > > Sure, done. Note that that page is a wiki, so feel free to make more > > updates. > > I just tried tweaking it a bit, and got this error when I tried to > submit my edits: > > [Errno 31] Too many links: > '/home/scipy/wiki/data/pages/JasonGrout(2f)MoinEditorBackup' > > See http://sage.math.washington.edu/home/jason/numpywikierror.html for > more debugging output. > Seen that before, I think it's an intermittent error. I just managed to save the same page again. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From monocongo at gmail.com Wed Nov 28 17:28:57 2012 From: monocongo at gmail.com (James Adams) Date: Wed, 28 Nov 2012 17:28:57 -0500 Subject: [SciPy-User] Which stats package am I missing for a function to do L-moments to Pearson 3 parameters calculation? Message-ID: I'm writing a Python program which needs to compute Pearson Type III parameters from L-moments, and then calculate the cumulative distribution function (CDF) using those parameters. For going from L-moments to Pearson parameters I've found a function in the source for hydroclimpy (the _lmparams_pearson3 function in trunk/hydroclimpy/scikits/hydroclimpy/stats/lmoments.py) which might do the trick, although it uses L-statistics rather than L-moments as its input, but I assume that that is an easy conversion. Is this a good approach, or is there a better/more standard way of doing this? The next step is to find a function that calculates the CDF for a Pearson Type III distribution. It looks like I'll need to make sure that I have a Pearson Type III continuous distribution installed, is this correct? If so then where can I find these distribution classes for installation, and/or how can I confirm that these are properly installed? Thanks in advance for any suggestions or comments. I'm a new Python programmer and new to doing this kind of math; my apologies if this is too much of an open-ended/beginner's question. --James From josef.pktd at gmail.com Wed Nov 28 17:48:49 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 28 Nov 2012 17:48:49 -0500 Subject: [SciPy-User] Which stats package am I missing for a function to do L-moments to Pearson 3 parameters calculation? In-Reply-To: References: Message-ID: On Wed, Nov 28, 2012 at 5:28 PM, James Adams wrote: > I'm writing a Python program which needs to compute Pearson Type III > parameters from L-moments, and then calculate the cumulative > distribution function (CDF) using those parameters. > > For going from L-moments to Pearson parameters I've found a function > in the source for hydroclimpy (the _lmparams_pearson3 function in > trunk/hydroclimpy/scikits/hydroclimpy/stats/lmoments.py) which might > do the trick, although it uses L-statistics rather than L-moments as > its input, but I assume that that is an easy conversion. Is this a > good approach, or is there a better/more standard way of doing this? hydroclimpy is still the only package that has L-statistics/ L-moments in python (AFAIK) Given that the fortran source is from the main L-moment person, I assume it works. I don't remember whether I read anything about fitting a pearson3 distribution, but for other distributions there are alternative estimators (not L-moment based) that do at least as well in simulation studies. (But I haven't looked at this in a while. And I also never figured out whether it's worth keeping L-moments in Fortran or whether it could just be translated into numpython) > > The next step is to find a function that calculates the CDF for a > Pearson Type III distribution. It looks like I'll need to make sure > that I have a Pearson Type III continuous distribution installed, is > this correct? If so then where can I find these distribution classes > for installation, and/or how can I confirm that these are properly > installed? As far as I could figure out Pearson 3 is just a reparameterized generalized gamma distribution, and it should be possible to use scipy.stats.distributions. An explicit Pearson 3 will come to scipy ( I assume ) https://github.com/scipy/scipy/pull/336 (I think I lost track of this one, and never checked the latest changes) Hopefully someone has more specific information about using L-moments and Pearson 3. Josef > > Thanks in advance for any suggestions or comments. I'm a new Python > programmer and new to doing this kind of math; my apologies if this is > too much of an open-ended/beginner's question. > > --James > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From monocongo at gmail.com Wed Nov 28 18:08:59 2012 From: monocongo at gmail.com (James Adams) Date: Wed, 28 Nov 2012 18:08:59 -0500 Subject: [SciPy-User] Which stats package am I missing for a function to do L-moments to Pearson 3 parameters calculation? Message-ID: I have also found Python code in CDAT which appears to provide this here: http://www2-pcmdi.llnl.gov/cdat/contrib/lmomentsdoc (specifically the PELPE3 function). Does this look reasonable? Has anyone else ever used CDAT, is it solid, etc.? It looks like it'll be a bear to install, so I don't want to start down that path unless it'll probably be worth it. --James From josef.pktd at gmail.com Wed Nov 28 18:31:52 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 28 Nov 2012 18:31:52 -0500 Subject: [SciPy-User] Which stats package am I missing for a function to do L-moments to Pearson 3 parameters calculation? In-Reply-To: References: Message-ID: On Wed, Nov 28, 2012 at 6:08 PM, James Adams wrote: > I have also found Python code in CDAT which appears to provide this > here: http://www2-pcmdi.llnl.gov/cdat/contrib/lmomentsdoc > (specifically the PELPE3 function). Does this look reasonable? Has > anyone else ever used CDAT, is it solid, etc.? It looks like it'll be > a bear to install, so I don't want to start down that path unless > it'll probably be worth it. It looks like another wrapper of J. R. M. HOSKING's fortran code, the wrapper looks like just a thin f2py wrapper. Maybe the same is available in the hydroclimpy wrapper. I never checked how complete the wrapping is. Josef > > --James > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From josef.pktd at gmail.com Wed Nov 28 18:59:31 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 28 Nov 2012 18:59:31 -0500 Subject: [SciPy-User] Which stats package am I missing for a function to do L-moments to Pearson 3 parameters calculation? In-Reply-To: References: Message-ID: On Wed, Nov 28, 2012 at 6:31 PM, wrote: > On Wed, Nov 28, 2012 at 6:08 PM, James Adams wrote: >> I have also found Python code in CDAT which appears to provide this >> here: http://www2-pcmdi.llnl.gov/cdat/contrib/lmomentsdoc >> (specifically the PELPE3 function). Does this look reasonable? Has >> anyone else ever used CDAT, is it solid, etc.? It looks like it'll be >> a bear to install, so I don't want to start down that path unless >> it'll probably be worth it. > > It looks like another wrapper of J. R. M. HOSKING's fortran code, the > wrapper looks like just a thin f2py wrapper. > Maybe the same is available in the hydroclimpy wrapper. I never > checked how complete the wrapping is. hydroclimpy also has Pearson 3 distribution, including cdf https://github.com/pierregm/scikits.hydroclimpy/blob/master/scikits/hydroclimpy/stats/extradistributions.py#L306 I didn't find PELPE3 in hydroclimpy, only estimation for kappa seems to be there. on the other hand https://www2-pcmdi.llnl.gov/trac/browser/cdat/trunk/contrib/lmoments/Src/pelpe3.for is only a few lines, which shouldn't be too difficult to translate into python. so the CDAT wrapper contains more functions Josef > > Josef > > >> >> --James >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user From scopatz at gmail.com Wed Nov 28 13:33:41 2012 From: scopatz at gmail.com (Anthony Scopatz) Date: Wed, 28 Nov 2012 12:33:41 -0600 Subject: [SciPy-User] [SciPy-Dev] Request for Feedback/Authorization: a new public repository for the conference In-Reply-To: References: Message-ID: On Wed, Nov 28, 2012 at 12:28 PM, Robert Kern wrote: > On Wed, Nov 28, 2012 at 7:21 PM, Jonathan Rocher > wrote: > > Dear all, > > > > Andy Terrel and I are the co-chairs of the Scipy2013 conference in Austin > > next June in Austin. We are very excited about this coming edition and > will > > post regularly to these lists for feedback and announcements. > > > > But for now, we would like to consolidate its organization under a public > > open source project inside the Scipy organization on github. We propose > that > > the project be called scipy-conference. The repo would host documents > useful > > to run the conference as well as code to set up the website and the > backend. > > > > I write here to ask if the community is ok with that and to ask for the > > rights to create/manage that new repository. > > +1. If there is no strenuous objection in short order, I will make a > "SciPy Conference" team under the scipy organization and add you two > to it. > +1 as well. > > -- > Robert Kern > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rainexpected at theo.to Fri Nov 30 12:25:33 2012 From: rainexpected at theo.to (Ted To) Date: Fri, 30 Nov 2012 12:25:33 -0500 Subject: [SciPy-User] Weird randomization issue with multiprocessing module Message-ID: <50B8EC0D.7040701@theo.to> Hi, I have a very strange problem that maybe someone might have some ideas as to what is going on. I'm making a series of random draws across multiple processor threads on a Debian Wheezy, dual core AMD Phenom box with scipy 0.10.1 and numpy 1.6.2 (once I debug my code, I will run it on a more powerful workstation). Since each series of draws is in a separate thread, I would have presumed that the draws are completely independent of one another. But strangely, for a total of 1000 threads (managed by multiprocessing.Pool), early in the series of threads, I see precisely the same numbers in some of the variables produced by two adjacent threads. The probability of this happening should be vanishingly small but strangely happens twice within the first 25 threads and by cursory examination not at all in the remaining 975. The numbers are being drawn using scipy.stats.norm and random.random. Any ideas? Thanks in advance, Ted To From takowl at gmail.com Fri Nov 30 12:43:48 2012 From: takowl at gmail.com (Thomas Kluyver) Date: Fri, 30 Nov 2012 17:43:48 +0000 Subject: [SciPy-User] Weird randomization issue with multiprocessing module In-Reply-To: <50B8EC0D.7040701@theo.to> References: <50B8EC0D.7040701@theo.to> Message-ID: On 30 November 2012 17:25, Ted To wrote: > The > numbers are being drawn using scipy.stats.norm and random.random. > I think the default seed for the random number generator uses the system time, so maybe if your threads/processes spin up too quickly, some of them could get the same seed. But it says it should also be using /dev/urandom in the seed, which should negate that. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Fri Nov 30 12:57:12 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 30 Nov 2012 12:57:12 -0500 Subject: [SciPy-User] Weird randomization issue with multiprocessing module In-Reply-To: References: <50B8EC0D.7040701@theo.to> Message-ID: On Fri, Nov 30, 2012 at 12:43 PM, Thomas Kluyver wrote: > On 30 November 2012 17:25, Ted To wrote: >> >> The >> numbers are being drawn using scipy.stats.norm and random.random. > > > I think the default seed for the random number generator uses the system > time, so maybe if your threads/processes spin up too quickly, some of them > could get the same seed. But it says it should also be using /dev/urandom in > the seed, which should negate that. for some discussion http://mail.scipy.org/pipermail/numpy-discussion/2008-December/039179.html Josef > > Thomas > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From rainexpected at theo.to Fri Nov 30 13:19:32 2012 From: rainexpected at theo.to (Ted To) Date: Fri, 30 Nov 2012 13:19:32 -0500 Subject: [SciPy-User] Weird randomization issue with multiprocessing module In-Reply-To: References: <50B8EC0D.7040701@theo.to> Message-ID: <50B8F8B4.7010006@theo.to> On 11/30/2012 12:57 PM, josef.pktd at gmail.com wrote: > On Fri, Nov 30, 2012 at 12:43 PM, Thomas Kluyver wrote: >> On 30 November 2012 17:25, Ted To wrote: >>> >>> The >>> numbers are being drawn using scipy.stats.norm and random.random. >> >> >> I think the default seed for the random number generator uses the system >> time, so maybe if your threads/processes spin up too quickly, some of them >> could get the same seed. But it says it should also be using /dev/urandom in >> the seed, which should negate that. > > for some discussion > http://mail.scipy.org/pipermail/numpy-discussion/2008-December/039179.html Many thanks -- adding a seed() at the beginning of the routine did the trick!